Crash on slave (replica sets) re-sync

master:

Thu Mar 3 05:46:30 [conn143] query admin.$cmd ntoreturn:1 command:

{ logRotate: 1.0 }

reslen:53 155ms
Thu Mar 3 05:46:43 [conn123] insert postbacks.postbacks 1526ms
Thu Mar 3 05:46:44 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000cd') }

21844ms
Thu Mar 3 05:46:50 [dur] lsn set 2744873
Thu Mar 3 05:46:50 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc63000218') }

20758ms
Thu Mar 3 05:47:00 [conn140] getmore metrics.metrics cid:2573549354707021591 getMore: {} bytes:4194370 nreturned:26340 exhaust 133ms
Thu Mar 3 05:47:00 [conn145] query admin.$cmd ntoreturn:1 command:

{ listDatabases: 1 }

reslen:358 152ms
Thu Mar 3 05:47:01 [conn145] query metrics.$cmd ntoreturn:1 command:

{ dbstats: 1 }

reslen:220 592ms
Thu Mar 3 05:47:01 [conn145] query postbacks.$cmd ntoreturn:1 command:

{ dbstats: 1 }

reslen:218 278ms
Thu Mar 3 05:47:05 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000ce') }

20801ms
Thu Mar 3 05:47:09 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc63000219') }

18986ms
Thu Mar 3 05:47:25 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000cf') }

20020ms
Thu Mar 3 05:47:27 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc6300021a') }

18073ms
Thu Mar 3 05:47:35 [conn144] query metrics.$cmd ntoreturn:1 command: { count: "metrics", query:

{ p: false }

, fields: {} } reslen:64 3307ms
Thu Mar 3 05:47:46 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d0') }

20665ms
Thu Mar 3 05:47:47 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc6300021b') }

19401ms
Thu Mar 3 05:47:50 [dur] lsn set 2802607
Thu Mar 3 05:48:04 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc6300021c') }

17236ms
Thu Mar 3 05:48:05 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d1') }

19011ms
Thu Mar 3 05:48:21 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc6300021d') }

17114ms
Thu Mar 3 05:48:22 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d2') }

17298ms
Thu Mar 3 05:48:38 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc6300021e') }

16917ms
Thu Mar 3 05:48:39 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d3') }

17233ms
Thu Mar 3 05:48:50 [dur] lsn set 2861698
Thu Mar 3 05:48:55 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc6300021f') }

17471ms
Thu Mar 3 05:48:56 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d4') }

17178ms
Thu Mar 3 05:49:12 [dur] old journal file will be removed: /mongodbdata/journal/j._1
Thu Mar 3 05:49:12 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc63000220') }

17212ms
Thu Mar 3 05:49:14 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d5') }

17988ms
Thu Mar 3 05:49:29 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3150ba0bc63000221') }

16986ms
Thu Mar 3 05:49:32 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2053140ba064550000d6') }

17639ms
Thu Mar 3 05:49:34 [conn6] insert postbacks.postbacks 134ms
Thu Mar 3 05:49:48 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002db') }

18290ms
Thu Mar 3 05:49:50 [dur] lsn set 2920533
Thu Mar 3 05:49:51 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000eb') }

19466ms
Thu Mar 3 05:50:01 [conn46] insert metrics.metrics 169ms
Thu Mar 3 05:50:07 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002dc') }

18959ms
Thu Mar 3 05:50:12 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000ec') }

20261ms
Thu Mar 3 05:50:13 [conn144] query metrics.$cmd ntoreturn:1 command: { count: "metrics", query:

{ p: false }

, fields: {} } reslen:64 3644ms
Thu Mar 3 05:50:21 [ReplSetHealthPollTask] DBClientCursor::init call() failed
Thu Mar 3 05:50:21 [ReplSetHealthPollTask] replSet info pb1b:27017 is down (or slow to respond): DBClientBase::findOne: transport error: pb1b:27017 query:

{ replSetHeartbeat: "pb1", v: 2, pv: 1, checkEmpty: false, from: "pb1a:27017" }

Thu Mar 3 05:50:26 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002dd') }

19259ms
Thu Mar 3 05:50:32 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000ed') }

20498ms
Thu Mar 3 05:50:44 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002de') }

17776ms
Thu Mar 3 05:50:50 [dur] lsn set 2979114
Thu Mar 3 05:50:53 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000ee') }

20448ms
Thu Mar 3 05:51:03 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002df') }

19533ms
Thu Mar 3 05:51:21 [conn152] insert postbacks.postbacks 234ms
Thu Mar 3 05:51:21 [conn12] insert postbacks.postbacks 421ms
Thu Mar 3 05:51:29 [conn69] insert postbacks.postbacks 422ms
Thu Mar 3 05:51:32 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000ef') }

39224ms
Thu Mar 3 05:51:38 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002e0') }

34554ms
Thu Mar 3 05:51:41 [conn157] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000ef') }

34388ms
Thu Mar 3 05:51:42 [conn60] insert postbacks.postbacks 190ms
Thu Mar 3 05:51:50 [dur] lsn set 3036858
Thu Mar 3 05:52:03 [conn60] insert postbacks.postbacks 120ms
Thu Mar 3 05:52:03 [ReplSetHealthPollTask] replSet info pb1b:27017 is up
Thu Mar 3 05:52:03 [ReplSetHealthPollTask] replSet member pb1b:27017 STARTUP2
Thu Mar 3 05:52:03 [conn30] update metrics.metrics query:

{ _id: ObjectId('4d6f2054150ba0570d0000f0') }

31656ms
Thu Mar 3 05:52:05 [conn8] update metrics.metrics query:

{ _id: ObjectId('4d6f07d3160ba049640002e1') }

26837ms
Thu Mar 3 05:52:05 [ReplSetHealthPollTask] replSet member pb1b:27017 RECOVERING
Thu Mar 3 05:52:09 [conn161] query postbacks.$cmd ntoreturn:1 command:

{ dbstats: 1 }

reslen:222 534ms

slave:

Thu Mar 3 05:44:29 [initandlisten] MongoDB starting : pid=12617 port=27017 dbpath=/mongodbdata/ 64-bit
Thu Mar 3 05:44:29 [initandlisten] db version v1.8.0-rc0, pdfile version 4.5
Thu Mar 3 05:44:29 [initandlisten] git version: 65a7e81df0747b6bc9380b78e0182192bacdb4d0
Thu Mar 3 05:44:29 [initandlisten] build sys info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Thu Mar 3 05:44:29 [initandlisten] journal dir=/mongodbdata/journal
Thu Mar 3 05:44:29 [initandlisten] recover : no journal files present, no recovery needed
Thu Mar 3 05:44:29 [initandlisten] waiting for connections on port 27017
Thu Mar 3 05:44:29 [websvr] web admin interface listening on port 28017
Thu Mar 3 05:44:29 [startReplSets] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Thu Mar 3 05:44:29 [startReplSets] replSet info you may need to run replSetInitiate – rs.initiate() in the shell – if that is not already done
Thu Mar 3 05:44:39 [startReplSets] trying to contact pb1arbiter:27017
Thu Mar 3 05:44:39 [startReplSets] replSet got config version 2 from a remote, saving locally
Thu Mar 3 05:44:39 [startReplSets] replSet info saving a newer config version to local.system.replset
Thu Mar 3 05:44:39 [FileAllocator] allocating new datafile /mongodbdata/local.ns, filling with zeroes...
Thu Mar 3 05:44:39 [FileAllocator] done allocating datafile /mongodbdata/local.ns, size: 16MB, took 0.052 secs
Thu Mar 3 05:44:39 [FileAllocator] allocating new datafile /mongodbdata/local.0, filling with zeroes...
Thu Mar 3 05:44:39 [FileAllocator] done allocating datafile /mongodbdata/local.0, size: 64MB, took 0.102 secs
Thu Mar 3 05:44:39 [FileAllocator] allocating new datafile /mongodbdata/local.1, filling with zeroes...
Thu Mar 3 05:44:39 [startReplSets] replSet STARTUP2
Thu Mar 3 05:44:39 [rs Manager] replSet can't see a majority, will not try to elect self
Thu Mar 3 05:44:39 [replica set sync] ******
Thu Mar 3 05:44:39 [replica set sync] creating replication oplog of size: 2551MB...
Thu Mar 3 05:44:40 [FileAllocator] done allocating datafile /mongodbdata/local.1, size: 128MB, took 0.248 secs
Thu Mar 3 05:44:40 [FileAllocator] allocating new datafile /mongodbdata/local.2, filling with zeroes...
Thu Mar 3 05:44:41 [ReplSetHealthPollTask] replSet info pb1a:27017 is up
Thu Mar 3 05:44:41 [ReplSetHealthPollTask] replSet member pb1a:27017 PRIMARY
Thu Mar 3 05:44:41 [ReplSetHealthPollTask] replSet info pb1arbiter:27017 is up
Thu Mar 3 05:44:41 [ReplSetHealthPollTask] replSet member pb1arbiter:27017 ARBITER
Thu Mar 3 05:44:46 [FileAllocator] done allocating datafile /mongodbdata/local.2, size: 2047MB, took 6.391 secs
Thu Mar 3 05:44:46 [FileAllocator] allocating new datafile /mongodbdata/local.3, filling with zeroes...
Thu Mar 3 05:44:53 [FileAllocator] done allocating datafile /mongodbdata/local.3, size: 2047MB, took 6.777 secs
Thu Mar 3 05:44:53 [replica set sync] ******
Thu Mar 3 05:44:53 [replica set sync] replSet initial sync pending
Thu Mar 3 05:44:53 [replica set sync] replSet syncing to: pb1a:27017
Thu Mar 3 05:44:53 [dur] lsn set 19664
Thu Mar 3 05:44:53 [replica set sync] building new index on

{ _id: 1 }

for local.me
Thu Mar 3 05:44:53 [replica set sync] done for 0 records 0.07secs
Thu Mar 3 05:44:54 [replica set sync] replSet initial sync drop all databases
Thu Mar 3 05:44:54 [replica set sync] dropAllDatabasesExceptLocal 1
Thu Mar 3 05:44:54 [replica set sync] replSet initial sync clone all databases
Thu Mar 3 05:44:54 [replica set sync] replSet initial sync cloning db: metrics
Thu Mar 3 05:44:54 [FileAllocator] allocating new datafile /mongodbdata/metrics.ns, filling with zeroes...
Thu Mar 3 05:44:54 [FileAllocator] done allocating datafile /mongodbdata/metrics.ns, size: 16MB, took 0.019 secs
Thu Mar 3 05:44:54 [FileAllocator] allocating new datafile /mongodbdata/metrics.0, filling with zeroes...
Thu Mar 3 05:44:54 [FileAllocator] done allocating datafile /mongodbdata/metrics.0, size: 64MB, took 0.075 secs
Thu Mar 3 05:44:54 [FileAllocator] allocating new datafile /mongodbdata/metrics.1, filling with zeroes...
Thu Mar 3 05:44:54 [FileAllocator] done allocating datafile /mongodbdata/metrics.1, size: 128MB, took 0.158 secs
Thu Mar 3 05:44:54 [FileAllocator] allocating new datafile /mongodbdata/metrics.2, filling with zeroes...
Thu Mar 3 05:45:00 [FileAllocator] done allocating datafile /mongodbdata/metrics.2, size: 2047MB, took 6.455 secs
Thu Mar 3 05:45:02 [conn5] query admin.$cmd ntoreturn:1 command:

{ listDatabases: 1 }

reslen:246 234ms
Thu Mar 3 05:45:02 [conn5] query metrics.$cmd ntoreturn:1 command: { count: "system.namespaces", query: {}, fields: null } reslen:64 267ms
Thu Mar 3 05:45:32 [dur] lsn set 52635
Thu Mar 3 05:46:01 [replica set sync] replauthenticate: no user in local.system.users to use for authentication
4407028 objects cloned so far from collection metrics.metrics
Thu Mar 3 05:46:04 [conn6] query metrics.$cmd ntoreturn:1 command:

{ dbstats: 1 }

reslen:216 218ms
Thu Mar 3 05:46:32 [dur] lsn set 111046
Thu Mar 3 05:47:02 [replica set sync] 8660385 objects cloned so far from collection metrics.metrics
Thu Mar 3 05:47:08 [conn7] query admin.$cmd ntoreturn:1 command:

{ listDatabases: 1 }

reslen:246 270ms
Thu Mar 3 05:47:08 [conn7] query metrics.$cmd ntoreturn:1 command:

{ dbstats: 1 }

reslen:220 881ms
Thu Mar 3 05:47:09 [conn7] query metrics.$cmd ntoreturn:1 command: { count: "system.namespaces", query: {}, fields: null } reslen:64 180ms
Thu Mar 3 05:47:31 [dur] lsn set 169647
Thu Mar 3 05:47:43 [dur] old journal file will be removed: /mongodbdata/journal/j._0
Thu Mar 3 05:47:53 [replica set sync] info: indexing in foreground on this replica; was a background index build on the primary
Thu Mar 3 05:47:53 [replica set sync] building new index on

{ p: 1.0 }

for metrics.metrics
1000000/12243150 8%
2000000/12243150 16%
3316700/12243150 27%
4579200/12243150 37%
5839400/12243150 47%
6792100/12243150 55%
7869200/12243150 64%
9000000/12243150 73%
10169200/12243150 83%
11521900/12243150 94%
Thu Mar 3 05:49:41 [replica set sync] external sort used : 13 files in 107 secs
3583600/12243150 29%
Thu Mar 3 05:49:57 [FileAllocator] allocating new datafile /mongodbdata/metrics.3, filling with zeroes...
Thu Mar 3 05:50:03 [FileAllocator] done allocating datafile /mongodbdata/metrics.3, size: 2047MB, took 6.105 secs
Thu Mar 3 05:50:03 [FileAllocator] allocating new datafile /mongodbdata/metrics.4, filling with zeroes...
6755500/12243150 55%
Thu Mar 3 05:50:11 [FileAllocator] done allocating datafile /mongodbdata/metrics.4, size: 2047MB, took 7.138 secs
7934700/12243150 64%
Thu Mar 3 05:50:20 [replica set sync] done building bottom layer, going to commit
Thu Mar 3 05:50:20 terminate() called, printing stack:
0x8a241d 0x3a7d4bcdb6 0x3a7d4bc16b 0x3a7d4bcc43 0x3a7a408978 0x3a7a408cea 0x67ebae 0x776dbc 0x777318 0x778d70 0x77a1ac 0x724b54 0x726471 0x7273fd 0x70d822 0x70f298 0x7092ac 0x70939c 0x709835 0x8b6a30
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11myterminateEv+0x4d) [0x8a241d]
/usr/lib64/libstdc++.so.6 [0x3a7d4bcdb6]
/usr/lib64/libstdc++.so.6 [0x3a7d4bc16b]
/usr/lib64/libstdc++.so.6(__gxx_personality_v0+0x353) [0x3a7d4bcc43]
/lib64/libgcc_s.so.1 [0x3a7a408978]
/lib64/libgcc_s.so.1(_Unwind_Resume+0x9a) [0x3a7a408cea]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo12BtreeBuilderD1Ev+0x63e) [0x67ebae]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x17bc) [0x776dbc]
/home/david/mongodb/latest/bin/mongod [0x777318]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf60) [0x778d70]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x4c) [0x77a1ac]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x3b4) [0x724b54]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x12b1) [0x726471]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x7273fd]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0xf12) [0x70d822]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x28) [0x70f298]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x5c) [0x7092ac]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70939c]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo15startSyncThreadEv+0x215) [0x709835]
/home/david/mongodb/latest/bin/mongod(thread_proxy+0x80) [0x8b6a30]
Thu Mar 3 05:50:21 Got signal: 6 (Aborted).

Thu Mar 3 05:50:21 Backtrace:
0x8a3189 0x3a768302d0 0x3a76830265 0x3a76831d10 0x8a24eb 0x3a7d4bcdb6 0x3a7d4bc16b 0x3a7d4bcc43 0x3a7a408978 0x3a7a408cea 0x67ebae 0x776dbc 0x777318 0x778d70 0x77a1ac 0x724b54 0x726471 0x7273fd 0x70d822 0x70f298
/home/david/mongodb/latest/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a3189]
/lib64/libc.so.6 [0x3a768302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3a76830265]
/lib64/libc.so.6(abort+0x110) [0x3a76831d10]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11myterminateEv+0x11b) [0x8a24eb]
/usr/lib64/libstdc++.so.6 [0x3a7d4bcdb6]
/usr/lib64/libstdc++.so.6 [0x3a7d4bc16b]
/usr/lib64/libstdc++.so.6(__gxx_personality_v0+0x353) [0x3a7d4bcc43]
/lib64/libgcc_s.so.1 [0x3a7a408978]
/lib64/libgcc_s.so.1(_Unwind_Resume+0x9a) [0x3a7a408cea]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo12BtreeBuilderD1Ev+0x63e) [0x67ebae]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x17bc) [0x776dbc]
/home/david/mongodb/latest/bin/mongod [0x777318]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf60) [0x778d70]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x4c) [0x77a1ac]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x3b4) [0x724b54]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x12b1) [0x726471]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x7273fd]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0xf12) [0x70d822]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x28) [0x70f298]

Thu Mar 3 05:50:21 dbexit:
Thu Mar 3 05:50:21 [replica set sync] shutdown: going to close listening sockets...
Thu Mar 3 05:50:21 [replica set sync] closing listening socket: 5
Thu Mar 3 05:50:21 [replica set sync] closing listening socket: 6
Thu Mar 3 05:50:21 [replica set sync] closing listening socket: 8
Thu Mar 3 05:50:21 [replica set sync] closing listening socket: 9
Thu Mar 3 05:50:21 [replica set sync] removing socket file: /tmp/mongodb-27017.sock
Thu Mar 3 05:50:21 [replica set sync] removing socket file: /tmp/mongodb-28017.sock
Thu Mar 3 05:50:21 [replica set sync] shutdown: going to flush diaglog...
Thu Mar 3 05:50:21 [replica set sync] shutdown: going to close sockets...
Thu Mar 3 05:50:21 [replica set sync] shutdown: waiting for fs preallocator...
Thu Mar 3 05:50:21 [replica set sync] shutdown: lock for final commit...
Thu Mar 3 05:50:21 [replica set sync] shutdown: final commit...
Thu Mar 3 05:50:21 [replica set sync] metrics Assertion failure a <= 128*1024*1024 util/alignedbuilder.cpp 51
0x55125e 0x5628af 0x629154 0x636d1a 0x637574 0x637729 0x6378fb 0x62e1bf 0x62eb48 0x753b86 0x754454 0x8a32d5 0x3a768302d0 0x3a76830265 0x3a76831d10 0x8a24eb 0x3a7d4bcdb6 0x3a7d4bc16b 0x3a7d4bcc43 0x3a7a408978
/home/david/mongodb/latest/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x55125e]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x10f) [0x5628af]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x44) [0x629154]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2aa) [0x636d1a]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x74) [0x637574]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo3dur14_PREPLOGBUFFEREv+0x79) [0x637729]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFEREv+0x9b) [0x6378fb]
/home/david/mongodb/latest/bin/mongod [0x62e1bf]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x18) [0x62eb48]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo14shutdownServerEv+0x466) [0x753b86]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKcb+0x1f4) [0x754454]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo10abruptQuitEi+0x4e5) [0x8a32d5]
/lib64/libc.so.6 [0x3a768302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3a76830265]
/lib64/libc.so.6(abort+0x110) [0x3a76831d10]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo11myterminateEv+0x11b) [0x8a24eb]
/usr/lib64/libstdc++.so.6 [0x3a7d4bcdb6]
/usr/lib64/libstdc++.so.6 [0x3a7d4bc16b]
/usr/lib64/libstdc++.so.6(__gxx_personality_v0+0x353) [0x3a7d4bcc43]
/lib64/libgcc_s.so.1 [0x3a7a408978]
Thu Mar 3 05:50:24 [replica set sync] shutdown: closing all files...
Thu Mar 3 05:50:24 closeAllFiles() finished
Thu Mar 3 05:50:24 [replica set sync] shutdown: journalCleanup...
Thu Mar 3 05:50:24 [replica set sync] removeJournalFiles
Thu Mar 3 05:50:24 [replica set sync] shutdown: removing fs lock...
Thu Mar 3 05:50:24 dbexit: really exiting now
Thu Mar 3 05:50:24 ERROR: Client::~Client _context should be null but is not; client:replica set sync