Using a sharded cluster consisting of 5 shards. The collection started off empty and using a batch process to load the data from MySQL into Mongo using Mongoid.

During the run we decided to create multiple indexes to reduce CPU utilization. When creating the indexes the primary server in the first shard crashed with a Segmentation fault. After restarting the mongod we then re-ran the steps to create the indexes, but this time included the background: true option. The second time around it took about a minute and only then crashed with a Segmentation fault. The key is that these collections were queried and inserted into while the indexes were being created. Did not see this problem when running 1.8.3.

Below are both stack traces from the log file. The first was without the background option and mongod crashed almost immediately:

Tue Sep 27 14:23:03 Invalid access at address: 0x10

Tue Sep 27 14:23:03 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:23:03 Backtrace:
0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0 [0x3cc340eb10]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]

pure virtual method called
Tue Sep 27 14:23:03 terminate() called, printing stack:
0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
Logstream::get called in uninitialized state
Tue Sep 27 14:23:03 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Tue Sep 27 14:23:03 ERROR: Client::shutdown not called: conn
pure virtual method called
Tue Sep 27 14:23:03 terminate() called, printing stack:
0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
pure virtual method called
Tue Sep 27 14:23:03 terminate() called, printing stack:
0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
Logstream::get called in uninitialized state
Tue Sep 27 14:23:03 [snapshotthread] ERROR in SnapshotThread: St9bad_alloc
Tue Sep 27 14:23:03 Got signal: 6 (Aborted).

/usr/bin/mongod(_ZN5mongo11myterminateEv+0x4d) [0xa834ed]
/usr/lib64/libstdc++.so.6 [0x3cc2cbce16]
/usr/lib64/libstdc++.so.6 [0x3cc2cbce43]
/usr/lib64/libstdc++.so.6 [0x3cc2cbd34f]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x637) [0x973ad7]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97512f]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x95d725]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xee4) [0x9607b4]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]
/lib64/libc.so.6(clone+0x6d) [0x3cc24d44bd]
Tue Sep 27 14:23:03 Got signal: 6 (Aborted).

Building indexes using background: true
========================

Tue Sep 27 14:31:05 [conn63] build index done 38913 records 9.878 secs
Tue Sep 27 14:31:05 [conn63] insert clarity_production.system.indexes 9879ms
Tue Sep 27 14:31:05 Invalid access at address: 0

Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:31:05 Invalid access at address: 0

Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:31:05 Invalid access at address: 0

Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

Tue Sep 27 14:31:05 Backtrace:
0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0 [0x3cc340eb10]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]

Logstream::get called in uninitialized state
Tue Sep 27 14:31:05 [conn43] query clarity_production.clarity_raw_xmls ntoreturn:1 nscanned:39157 reslen:20 400ms
Logstream::get called in uninitialized state
Tue Sep 27 14:31:05 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Tue Sep 27 14:31:05 ERROR: Client::shutdown not called: conn
Tue Sep 27 14:31:05 Backtrace:
0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0 [0x3cc340eb10]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
/usr/bin/mongod [0x87e037]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
/lib64/libpthread.so.0 [0x3cc340673d]