-
Type: Bug
-
Resolution: Done
-
Priority: Blocker - P1
-
Affects Version/s: 2.2.0
-
Component/s: Logging, Replication
-
None
-
ALL
Running a replica set on 2x hi1.4xlarge amazon boxes + an arbiter all in amazon's us-east-1 datacenter. This is in production doing ~1500 inserts/sec on the primary. I'm seeing this over and over in the secondary's log (and it won't catch up to the primary):
Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.577MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x62dd73 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x113) [0x62dd73] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.581MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60775 0xa61980 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x125) [0xa60775] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x340) [0xa61980] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.585MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.589MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.593MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.597MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.601MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.606MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.61MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.614MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.618MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.622MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.626MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.63MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.634MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.638MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.642MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.647MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.651MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.655MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0x62de4b 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x1eb) [0x62de4b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.659MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.663MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.667MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.671MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.675MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.679MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.683MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.687MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.692MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.696MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60775 0xa61980 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x125) [0xa60775] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x340) [0xa61980] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.7MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.704MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.708MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.712MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.716MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.72MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.724MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.728MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.733MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.737MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x625fe7 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x27) [0x625fe7] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.741MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.745MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.749MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.753MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.757MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.761MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.765MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.769MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.774MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.778MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60bc9 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0xe9) [0xa60bc9] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.782MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.786MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.79MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.794MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.798MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.802MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.806MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.81MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.814MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.819MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60775 0xa61980 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x125) [0xa60775] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x340) [0xa61980] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.823MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.827MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.831MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.835MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.839MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.843MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.847MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.851MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.855MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.86MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60dcb 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x2eb) [0xa60dcb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.864MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.868MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.872MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.876MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.88MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.884MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.888MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.892MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.896MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.9MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x62ddf1 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x191) [0x62ddf1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.905MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.909MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.913MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.917MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.921MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.925MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.929MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.933MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.937MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.941MB 0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.946MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.95MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.954MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.958MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.962MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.966MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.97MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.974MB Tue Sep 4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.978MB Tue Sep 4 03:27:04 [DataFileSync] flushing mmaps took 14012ms for 250 files Tue Sep 4 03:27:10 [TTLMonitor] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:ge_raw.system.indexes query:{ expireAfterSeconds: { $exists: true } } Tue Sep 4 03:27:10 [TTLMonitor] problem detected during query over ge_raw.system.indexes : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 } Tue Sep 4 03:27:10 [TTLMonitor] ERROR: error processing ttl for db: ge_raw 10065 invalid parameter: expected an object () Tue Sep 4 03:27:14 [journal] warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90 0xade6e1 0x802c5a 0x77dc73 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x77dc73] /ebs/mongo/LATEST/bin/mongod() [0x753da5] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950] /ebs/mongo/LATEST/bin/mongod() [0xa0a779] /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24] /ebs/mongo/LATEST/bin/mongod() [0x7c3659] /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca] /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d]
- depends on
-
SERVER-6816 Improve journal data handling after multithreaded batch writing
- Closed
- is related to
-
SERVER-16588 [rsSync] warning: DR102 too much data written uncommitted 314.577MB
- Closed