[SERVER-24893] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data Created: 05/Jul/16  Updated: 05/Jul/16  Resolved: 05/Jul/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Manish Gupta Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu server 14.04 LTS 64 bit


Attachments: Text File mongod.log    
Operating System: ALL
Participants:

 Description   

I am running mongodb 3.2.7 on ubuntu 14.04 64 bit server. The mongodb server keeps crashing every 2-3 hours. On restart it says this:

stop: Unknown instance:
mongod start/running, process 25687

When I open the mongo shell after this, I see thew following error and the shell is closed automatically.

src/third_party/gperftools-2.2/src/page_heap_allocator.h:74] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48

This error was not there 2 days ago. So how can I correct this error?



 Comments   
Comment by Ramon Fernandez Marina [ 05/Jul/16 ]

manish2492, thanks for uploading the log file; I see the following in it:

2016-07-04T10:23:26.660-0400 I NETWORK  [conn3069] end connection 127.0.0.1:56978 (6 connections now open)
2016-07-04T10:23:26.662-0400 I NETWORK  [conn3070] end connection 127.0.0.1:56980 (4 connections now open)
2016-07-04T10:23:26.662-0400 I NETWORK  [conn3071] end connection 127.0.0.1:56982 (3 connections now open)
2016-07-05T02:24:55.796-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] MongoDB starting : pid=25671 port=27017 dbpath=/var/lib/mongodb 64-bit host=sellerhub
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] db version v3.2.7
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] allocator: tcmalloc
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] modules: none
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] build environment:
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten]     distmod: ubuntu1404
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten]     distarch: x86_64
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten]     target_arch: x86_64
2016-07-05T02:24:55.813-0400 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "*", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-07-05T02:24:55.842-0400 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-07-05T02:24:55.843-0400 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2016-07-05T02:24:55.843-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-07-05T02:24:55.843-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-07-05T02:24:56.010-0400 E STORAGE  [initandlisten] WiredTiger (12) [1467699896:7534][25671:0x7f6e97111cc0], eviction-server: pthread_create: Cannot allocate memory
2016-07-05T02:24:56.011-0400 E STORAGE  [initandlisten] WiredTiger (12) [1467699896:11963][25671:0x7f6e97111cc0], txn-recover: Recovery failed: Cannot allocate memory
2016-07-05T02:24:56.031-0400 I -        [initandlisten] Assertion: 28595:12: Cannot allocate memory
2016-07-05T02:24:56.037-0400 I STORAGE  [initandlisten] exception in initAndListen: 28595 12: Cannot allocate memory, terminating
2016-07-05T02:24:56.037-0400 I CONTROL  [initandlisten] dbexit:  rc: 100

The SERVER RESTARTED marker without a proper shutdown indicates that mongod was terminated by the OOM killer. Combined with the subsequent out of memory errors, I would say that this machine is underpowered – specifically it needs more memory.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Comment by Manish Gupta [ 05/Jul/16 ]

Mongodb log file

Generated at Thu Feb 08 04:07:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.