[SERVER-4054] mongod 2.0.0 -- segmentation fault Created: 11/Oct/11  Updated: 30/Mar/12  Resolved: 16/Oct/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.0.0
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Harald Lapp Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: FRVIa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 2.6.32-21-server #32
Ubuntu SMP
x86_64 GNU/Linux


Operating System: Linux
Participants:

 Description   

If filed a new issue, because my backtrace looks quite different from the backtrace that was already reported by other people. The only thing i can tell is: heavy reads, some writes. MongoDB 2.0.0 went into production on our server yesterday in the evening and the crash occured now two times in two hours. Here are the backtraces:

Tue Oct 11 09:11:52 [initandlisten] connection accepted from 10.10.0.12:58806 #68483
Tue Oct 11 09:11:53 Invalid access at address: 0x68b9000

Tue Oct 11 09:11:53 Got signal: 11 (Segmentation fault).

Tue Oct 11 09:11:53 [initandlisten] connection accepted from 10.10.0.12:58828 #68484
Tue Oct 11 09:11:53 Backtrace:
0xa83fc9 0xa845a0 0x7fcb5c6828f0 0x7fcb5bbc75e5 0x51d300 0x54cd98 0x54d0ac 0x55a617 0x55b501 0x55bdd4 0x8d54a6 0x8e3475 0x99a242 0x99cc9c 0x973fe9 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib/libpthread.so.0(+0xf8f0) [0x7fcb5c6828f0]
/lib/libc.so.6(+0x855e5) [0x7fcb5bbc75e5]
/opt/mongodb/bin/mongod(ZN5mongo20compareElementValuesERKNS_11BSONElementES2+0x2d0) [0x51d300]
/opt/mongodb/bin/mongod(ZN5mongo20fieldIntervalOverlapERKNS_13FieldIntervalES2_RS0+0x228) [0x54cd98]
/opt/mongodb/bin/mongod(ZN5mongo10FieldRangeaNERKS0+0xac) [0x54d0ac]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSet16processOpElementEPKcRKNS_11BSONElementEbb+0x97) [0x55a617]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x4a1) [0x55b501]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55bdd4]
/opt/mongodb/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f6) [0x8d54a6]
/opt/mongodb/bin/mongod(ZN5mongo25NamespaceDetailsTransient9getCursorEPKcRKNS_7BSONObjES5+0x5e5) [0x8e3475]
/opt/mongodb/bin/mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x332) [0x99a242]
/opt/mongodb/bin/mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x11ec) [0x99cc9c]
/opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xb49) [0x973fe9]
/opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97512f]
/opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x95d725]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xee4) [0x9607b4]
/opt/mongodb/bin/mongod() [0x87e037]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]

Logstream::get called in uninitialized state
Tue Oct 11 09:11:54 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Tue Oct 11 09:11:54 ERROR: Client::shutdown not called: conn
Tue Oct 11 09:11:54 Got signal: 6 (Aborted).

Tue Oct 11 09:11:54 Backtrace:
0xa83fc9 0x7fcb5bb75af0 0x7fcb5bb75a75 0x7fcb5bb795c0 0x7fcb5bb6e941 0x67a48f 0x794b48 0x79aedc 0x57756f 0x579624 0xa9ea30 0x7fcb5c6799ca 0x7fcb5bc2870d
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/lib/libc.so.6(+0x33af0) [0x7fcb5bb75af0]
/lib/libc.so.6(gsignal+0x35) [0x7fcb5bb75a75]
/lib/libc.so.6(abort+0x180) [0x7fcb5bb795c0]
/lib/libc.so.6(__assert_fail+0xf1) [0x7fcb5bb6e941]
/opt/mongodb/bin/mongod(_ZN5mongo9MongoFile17totalMappedLengthEv+0x20f) [0x67a48f]
/opt/mongodb/bin/mongod(_ZN5mongo15sayMemoryStatusEv+0xd8) [0x794b48]
/opt/mongodb/bin/mongod(_ZN5mongo19ClientCursorMonitor3runEv+0x11c) [0x79aedc]
/opt/mongodb/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xbf) [0x57756f]
/opt/mongodb/bin/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x579624]
/opt/mongodb/bin/mongod(thread_proxy+0x80) [0xa9ea30]
/lib/libpthread.so.0(+0x69ca) [0x7fcb5c6799ca]
/lib/libc.so.6(clone+0x6d) [0x7fcb5bc2870d]

--->%--snip--->%----

Tue Oct 11 10:51:15 Got signal: 11 (Segmentation fault).

Tue Oct 11 10:51:16 Backtrace:
0xa83fc9 0xa845a0 0x7fbb46c348f0 0x7fbb461795e5 0x51d300 0x54cd98 0x54d0ac 0x55a617 0x55b501 0x55bdd4 0x8d54a6 0x8e3475 0x99a242 0x99cc9c 0x973fe9 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib/libpthread.so.0(+0xf8f0) [0x7fbb46c348f0]
/lib/libc.so.6(+0x855e5) [0x7fbb461795e5]
/opt/mongodb/bin/mongod(ZN5mongo20compareElementValuesERKNS_11BSONElementES2+0x2d0) [0x51d300]
/opt/mongodb/bin/mongod(ZN5mongo20fieldIntervalOverlapERKNS_13FieldIntervalES2_RS0+0x228) [0x54cd98]
/opt/mongodb/bin/mongod(ZN5mongo10FieldRangeaNERKS0+0xac) [0x54d0ac]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSet16processOpElementEPKcRKNS_11BSONElementEbb+0x97) [0x55a617]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x4a1) [0x55b501]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55bdd4]
/opt/mongodb/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f6) [0x8d54a6]
/opt/mongodb/bin/mongod(ZN5mongo25NamespaceDetailsTransient9getCursorEPKcRKNS_7BSONObjES5+0x5e5) [0x8e3475]
/opt/mongodb/bin/mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x332) [0x99a242]
/opt/mongodb/bin/mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x11ec) [0x99cc9c]
/opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xb49) [0x973fe9]
/opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97512f]
/opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x95d725]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xee4) [0x9607b4]
/opt/mongodb/bin/mongod() [0x87e037]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]

Logstream::get called in uninitialized state
Tue Oct 11 10:51:16 [initandlisten] connection accepted from 10.10.0.12:36644 #9736
Tue Oct 11 10:51:16 Invalid access at address: 0x4

Tue Oct 11 10:51:16 Got signal: 11 (Segmentation fault).

Tue Oct 11 10:51:16 Backtrace:
0xa83fc9 0xa845a0 0x7fbb46c348f0 0x54f7c9 0x55b0e4 0x55bdd4 0x8d54a6 0x960522 0x87e037 0x88485c 0xa96a46 0x635dd7 0x7fbb46c2b9ca 0x7fbb461da70d
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib/libpthread.so.0(+0xf8f0) [0x7fbb46c348f0]
/opt/mongodb/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x199) [0x54f7c9]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55b0e4]
/opt/mongodb/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55bdd4]
/opt/mongodb/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f6) [0x8d54a6]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xc52) [0x960522]
/opt/mongodb/bin/mongod() [0x87e037]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/opt/mongodb/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/opt/mongodb/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib/libpthread.so.0(+0x69ca) [0x7fbb46c2b9ca]
/lib/libc.so.6(clone+0x6d) [0x7fbb461da70d]



 Comments   
Comment by Eliot Horowitz (Inactive) [ 16/Oct/11 ]

See SERVER-3961

Comment by Harald Lapp [ 12/Oct/11 ]

sorry, just got the information, that we indeed have a group command in a query. but as far as i can see, it's very simple and it doesn't get executed very often (about two per minute). i'll post details tomorrow ...

Comment by Harald Lapp [ 11/Oct/11 ]

No, the nightly crashed, too, with similar backtrace. We've downgraded to 1.8.3 now, which we are using successfully with some other project for quite a while now. Is there anything i can do to help tracking down the problem?

Comment by Harald Lapp [ 11/Oct/11 ]

Thanks for your reply. We do not do any special things here. Almost normal query with some limit. I've now installed the nightly. It's running for over an hour now without a crash, which is quite good compared to the 2.0.0 stable ... but let's see, if it will stay this stable for the next hours ...

Comment by Scott Hernandez (Inactive) [ 11/Oct/11 ]

What is the group command you are using? Is it possible to try the latest 2.0.0 (nightly) which is very close to what the 2.0.1 release will be?

Generated at Thu Feb 08 03:04:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.