[SERVER-5174] Heap memory error in mongod Created: 02/Mar/12  Updated: 28/Dec/13  Resolved: 07/Mar/12

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Barrie Segal Assignee: Randolph Tan
Resolution: Won't Fix Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Amazon EC2 Linux, Mongo 2.0.2


Issue Links:
Duplicate
is duplicated by SERVER-11823 invalid fastbin entry (free) Closed
Related
is related to DOCS-396 Document error caused by SERVER-5174 Closed
Operating System: Linux
Participants:

 Description   

excerpt from mongod logs:

Fri Mar  2 02:39:39 [conn32354] info DFM::findAll(): extent 0:3000 was empty, skipping ahead. ns:data.stream
*** glibc detected *** /usr/local/bin/mongod: invalid fastbin entry (free): 0x00007f68505d41a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x750c6)[0x7f75ec6090c6]
/usr/local/bin/mongod(_ZN5mongo5CurOp5leaveEPNS_6Client7ContextE+0xe4)[0x8900e4]
/usr/local/bin/mongod(_ZN5mongo6Client7ContextD1Ev+0x2a)[0x89015a]
/usr/local/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xe8e)[0x965e1e]
/usr/local/bin/mongod[0x883877]
/usr/local/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c)[0x88a09c]
/usr/local/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76)[0xaa00c6]
/usr/local/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)[0x635bb7]
/lib64/libpthread.so.0(+0x77f1)[0x7f75ed0dc7f1]
/lib64/libc.so.6(clone+0x6d)[0x7f75ec67992d]
Fri Mar  2 02:39:39 Backtrace:
0xa8d669 0x7f75ec5c6900 0x7f75ec5c6885 0x7f75ec5c8065 0x7f75ec6037a7 0x7f75ec6090c6 0x8900e4 0x89015a 0x965e1e 0x883877 0x88a09c 0xaa00c6 0x635bb7 0x7f75ed0dc7f1 0x7f75ec67992d 
 /usr/local/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa8d669]
 /lib64/libc.so.6(+0x32900) [0x7f75ec5c6900]
 /lib64/libc.so.6(gsignal+0x35) [0x7f75ec5c6885]
 /lib64/libc.so.6(abort+0x175) [0x7f75ec5c8065]
 /lib64/libc.so.6(+0x6f7a7) [0x7f75ec6037a7]
 /lib64/libc.so.6(+0x750c6) [0x7f75ec6090c6]
 /usr/local/bin/mongod(_ZN5mongo5CurOp5leaveEPNS_6Client7ContextE+0xe4) [0x8900e4]
 /usr/local/bin/mongod(_ZN5mongo6Client7ContextD1Ev+0x2a) [0x89015a]
 /usr/local/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xe8e) [0x965e1e]
 /usr/local/bin/mongod() [0x883877]
 /usr/local/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88a09c]
 /usr/local/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6]
 /usr/local/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7]
 /lib64/libpthread.so.0(+0x77f1) [0x7f75ed0dc7f1]
 /lib64/libc.so.6(clone+0x6d) [0x7f75ec67992d]
 
...
 
Logstream::get called in uninitialized state
Fri Mar  2 02:39:39 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Fri Mar  2 02:39:39 ERROR: Client::shutdown not called: conn



 Comments   
Comment by ccokme [ 12/Nov/13 ]

Hi guys,

We had this issue too, when this issue happened, the mongodb process was still alive and capable to receive new connections, but the query is halt, there's no data return to the client, will this exception cause this issue and why mongodb process didn't exit when this exception happened?

Comment by Damon Cortesi [ 18/Sep/12 ]

I just ran into this with version 2.15 of glibc.

$ lsof -p $(pgrep mongod) | grep libc
mongod  8264 mongodb  mem       REG              253,1   1802936    655380 /lib/x86_64-linux-gnu/libc-2.15.so

Relevant stack trace:

======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7e626)[0x7f172a126626]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSsD1Ev+0x23)[0x7f172aa13c13]
/usr/bin/mongod(_ZN5mongo9QueryPlanC1EPNS_16NamespaceDetailsEiRKNS_17FieldRangeSetPairEPS4_RKNS_7BSONObjES9_bS9_S9_Ss+0xd01)[0x8d2ce1]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet4initEv+0xca6)[0x8dadd6]
/usr/bin/mongod(_ZN5mongo12QueryPlanSetC1EPKcSt8auto_ptrINS_17FieldRangeSetPairEES5_RKNS_7BSONObjES8_bPKNS_11BSONElementEbS8_S8_bb+0x1f7)[0x8dd977]
/usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x2bd)[0x8e080d]
/usr/bin/mongod(_ZN5mongo11MultiCursorC1EPKcRKNS_7BSONObjES5_N5boost10shared_ptrINS0_8CursorOpEEEb+0xda)[0x8e12ca]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverE+0x372)[0x95f092]
/usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugE+0x125)[0x963e25]
/usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x46c)[0x88cf4c]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x116f)[0x88e8ef]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78)[0xaa0bc8]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)[0x6389f7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f172ac7ce9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f172a19a4bd]

$ ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Apr 19 17:09 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.15.so

Comment by Randolph Tan [ 07/Mar/12 ]

This is an issue with glibc. Ben pointed out that it was possibly fixed in the later versions, but since the function is called from the shared library of the local machine running mongod, we have no control over it.

Comment by Randolph Tan [ 06/Mar/12 ]

Just to add a little bit of context for documentation purposes:

According to the logs, the error occurred when the deconstructor for the string object was called. The string was most likely the temporary string made to implicitly covert _ns (which is a char array) to string when it was passed to the Top::global.record method inside CurOp::leave.

Comment by Ben Becker [ 05/Mar/12 ]

I think this was a glibc bug fixed here: http://sourceware.org/git/?p=glibc.git;a=commitdiff;h=5f24d53acb2cd02ffb4ac925c0244e4f1f854499;hp=d84acf388ed8b3f162dda0512139095bfc268beb (see malloc.c diff)

Generated at Thu Feb 08 03:08:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.