[SERVER-3961] Segmentation Fault when creating new indexes Created: 27/Sep/11  Updated: 16/Nov/21  Resolved: 16/Oct/11

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.0.0
Fix Version/s: 2.0.1, 2.1.0

Type: Bug Priority: Major - P3
Reporter: Reid Morrison Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: FRVIa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Red Hat Enterprise Linux 5, 64 bit


Issue Links:
Depends
Duplicate
is duplicated by SERVER-5770 mongod crash Closed
is duplicated by SERVER-4655 Segmentation fault Closed
Operating System: Linux
Participants:

 Description   

Using a sharded cluster consisting of 5 shards. The collection started off empty and using a batch process to load the data from MySQL into Mongo using Mongoid.

During the run we decided to create multiple indexes to reduce CPU utilization. When creating the indexes the primary server in the first shard crashed with a Segmentation fault. After restarting the mongod we then re-ran the steps to create the indexes, but this time included the background: true option. The second time around it took about a minute and only then crashed with a Segmentation fault. The key is that these collections were queried and inserted into while the indexes were being created. Did not see this problem when running 1.8.3.

Below are both stack traces from the log file. The first was without the background option and mongod crashed almost immediately:

Tue Sep 27 14:23:03 Invalid access at address: 0x10

Tue Sep 27 14:23:03 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:23:03 Backtrace:
0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0 [0x3cc340eb10]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]

pure virtual method called
Tue Sep 27 14:23:03 terminate() called, printing stack:
0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
Logstream::get called in uninitialized state
Tue Sep 27 14:23:03 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Tue Sep 27 14:23:03 ERROR: Client::shutdown not called: conn
pure virtual method called
Tue Sep 27 14:23:03 terminate() called, printing stack:
0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
pure virtual method called
Tue Sep 27 14:23:03 terminate() called, printing stack:
0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
Logstream::get called in uninitialized state
Tue Sep 27 14:23:03 [snapshotthread] ERROR in SnapshotThread: St9bad_alloc
Tue Sep 27 14:23:03 Got signal: 6 (Aborted).

/usr/bin/mongod(_ZN5mongo11myterminateEv+0x4d) [0xa834ed]
/usr/lib64/libstdc++.so.6 [0x3cc2cbce16]
/usr/lib64/libstdc++.so.6 [0x3cc2cbce43]
/usr/lib64/libstdc++.so.6 [0x3cc2cbd34f]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x637) [0x973ad7]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97512f]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x95d725]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xee4) [0x9607b4]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]
/lib64/libc.so.6(clone+0x6d) [0x3cc24d44bd]
Tue Sep 27 14:23:03 Got signal: 6 (Aborted).

Building indexes using background: true
========================

Tue Sep 27 14:31:05 [conn63] build index done 38913 records 9.878 secs
Tue Sep 27 14:31:05 [conn63] insert clarity_production.system.indexes 9879ms
Tue Sep 27 14:31:05 Invalid access at address: 0

Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:31:05 Invalid access at address: 0

Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:31:05 Invalid access at address: 0

Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:31:05 Backtrace:
0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0 [0x3cc340eb10]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]

Logstream::get called in uninitialized state
Tue Sep 27 14:31:05 [conn43] query clarity_production.clarity_raw_xmls ntoreturn:1 nscanned:39157 reslen:20 400ms
Logstream::get called in uninitialized state
Tue Sep 27 14:31:05 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Tue Sep 27 14:31:05 ERROR: Client::shutdown not called: conn
Tue Sep 27 14:31:05 Backtrace:
0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0 [0x3cc340eb10]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]



 Comments   
Comment by auto [ 04/Dec/11 ]

Author:

{u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-3961 clear BtreeCursor's IndexSpec when no longer needed to avoid keeping a reference that may become invalid
Branch: master
https://github.com/mongodb/mongo/commit/e5c440ac103fd6f8836402cbcf7caf4fb09002a7

Comment by auto [ 04/Dec/11 ]

Author:

{u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-3961 make FieldRangeVector's IndexSpec const
Branch: master
https://github.com/mongodb/mongo/commit/16e8ae73ca7f0cdb02ed4f80daf4f8a8c66732cc

Comment by auto [ 16/Oct/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: fix segfault after modifying indexes during yield SERVER-3961 3
Branch: v2.0
https://github.com/mongodb/mongo/commit/858d2ad0cd3ae6599ac362aa0beaa12adfe88032

Comment by auto [ 16/Oct/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: threaded index test mod for SERVER-3961
Branch: master
https://github.com/mongodb/mongo/commit/b45634307b09030fe8e157120d616464ed947129

Comment by auto [ 16/Oct/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: fix segfault after modifying indexes during yield SERVER-3961 3
Branch: master
https://github.com/mongodb/mongo/commit/ef03c91415753ee793c5512bf8588f3e60ff8d40

Comment by Wijnand Wiersma [ 04/Oct/11 ]

I can confirm this, but in our case it happens on a single server instance (in case that helps).
We are using the official 10gen 64bit ubuntu packages.

Comment by Reid Morrison [ 27/Sep/11 ]

It is the official binaries distributed in the 10gen RPM package. We just repackaged the binaries into new RPM files to match: https://jira.mongodb.org/browse/SERVER-3748
Per pull request: https://github.com/mongodb/mongo/pull/91 to support RPM packages in a sharded environment.

Comment by Eliot Horowitz (Inactive) [ 27/Sep/11 ]

Was this an official binary or one you built yourself?

Generated at Thu Feb 08 03:04:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.