[SERVER-6925] DR102 too much data written uncommitted Created: 04/Sep/12  Updated: 16/Nov/21  Resolved: 15/Oct/12

Status: Closed
Project: Core Server
Component/s: Logging, Replication
Affects Version/s: 2.2.0
Fix Version/s: 2.2.1, 2.3.0

Type: Bug Priority: Blocker - P1
Reporter: Mike Khristo Assignee: Eric Milkie
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-6816 Improve journal data handling after m... Closed
Related
is related to SERVER-16588 [rsSync] warning: DR102 too much data... Closed
Operating System: ALL
Participants:

 Description   

Running a replica set on 2x hi1.4xlarge amazon boxes + an arbiter all in amazon's us-east-1 datacenter. This is in production doing ~1500 inserts/sec on the primary. I'm seeing this over and over in the secondary's log (and it won't catch up to the primary):

Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.577MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x62dd73 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x113) [0x62dd73]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
 /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.581MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60775 0xa61980 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x125) [0xa60775]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x340) [0xa61980]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.585MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.589MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.593MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.597MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.601MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.606MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.61MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.614MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.618MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.622MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.626MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.63MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.634MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.638MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.642MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.647MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.651MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.655MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0x62de4b 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x1eb) [0x62de4b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
 /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.659MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.663MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.667MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.671MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.675MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.679MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.683MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.687MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.692MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.696MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60775 0xa61980 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x125) [0xa60775]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x340) [0xa61980]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.7MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.704MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.708MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.712MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.716MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.72MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.724MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.728MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.733MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.737MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x625fe7 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x27) [0x625fe7]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.741MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.745MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.749MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.753MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.757MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.761MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.765MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.769MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.774MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.778MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60bc9 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0xe9) [0xa60bc9]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.782MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.786MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.79MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.794MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.798MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.802MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.806MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.81MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.814MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.819MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xa60775 0xa61980 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x125) [0xa60775]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x340) [0xa61980]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.823MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.827MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.831MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.835MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.839MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.843MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.847MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.851MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.855MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.86MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60dcb 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x2eb) [0xa60dcb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.864MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.868MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.872MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.876MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.88MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.884MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.888MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.892MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.896MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.9MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x62ddf1 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x191) [0x62ddf1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
 /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.905MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.909MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.913MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.917MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.921MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.925MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.929MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.933MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.937MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.941MB
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x626166 0xa60e16 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x1a6) [0x626166]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x336) [0xa60e16]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.946MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.95MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.954MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.958MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.962MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.966MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.97MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.974MB
Tue Sep  4 03:26:25 [rsSync] warning: DR102 too much data written uncommitted 314.978MB
Tue Sep  4 03:27:04 [DataFileSync] flushing mmaps took 14012ms  for 250 files
Tue Sep  4 03:27:10 [TTLMonitor] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:ge_raw.system.indexes query:{ expireAfterSeconds: { $exists: true } }
Tue Sep  4 03:27:10 [TTLMonitor] problem detected during query over ge_raw.system.indexes : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }
Tue Sep  4 03:27:10 [TTLMonitor] ERROR: error processing ttl for db: ge_raw 10065 invalid parameter: expected an object ()
Tue Sep  4 03:27:14 [journal]   warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
0xade6e1 0x802c5a 0x77dc73 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f0baacfc9ca 0x7f0baa0a370d 
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x77dc73]
 /ebs/mongo/LATEST/bin/mongod() [0x753da5]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950]
 /ebs/mongo/LATEST/bin/mongod() [0xa0a779]
 /ebs/mongo/LATEST/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
 /ebs/mongo/LATEST/bin/mongod() [0x7c3659]
 /lib/libpthread.so.0(+0x69ca) [0x7f0baacfc9ca]
 /lib/libc.so.6(clone+0x6d) [0x7f0baa0a370d]



 Comments   
Comment by auto [ 12/Oct/12 ]

Author:

{u'date': u'2012-10-12T15:12:12-07:00', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-6925 use the size of the queue in bytes rather than number of objects when calculating cap
Branch: v2.2
https://github.com/mongodb/mongo/commit/c0d27d82dd8212d4af5086bad9e2993a46de56eb

Comment by auto [ 12/Oct/12 ]

Author:

{u'date': u'2012-10-12T15:12:12-07:00', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-6925 use the size of the queue in bytes rather than number of objects when calculating cap
Branch: master
https://github.com/mongodb/mongo/commit/3a91169190b1876f6a5f74bbfb7ef1908e6cc5cd

Comment by Eric Milkie [ 12/Oct/12 ]

I found the bug that is breaking the BlockingQueue cap. The fix is simple and will go into version 2.2.1.

Comment by Brent Miller [ 12/Oct/12 ]

We'll actually see the bgsync buffer go well over 256 megs. Here's the lines from the logfile right before the os kills mongod for oom. Here you can see that the bgsync buffer is almost 4.5 gigs:

Thu Oct 11 13:09:56 [rsBackgroundSync] bgsync buffer has 4622617654 bytes
Thu Oct 11 13:09:56 [rsBackgroundSync] bgsync buffer has 4622763723 bytes
Thu Oct 11 13:09:56 [rsBackgroundSync] bgsync buffer has 4622861100 bytes

The secondaries that we're using at the moment only have 4G of ram; I know that's not a lot for mongo, but we're just testing on what we have lying around and all of our indexes are less than 1G so we assumed that this would be ok for the occasional fall-over and running aggregate queries.

It could be due to page faults, but it might be worth noting that we've only see the bgsync buffer grow this large after we've taken the secondaries offline for a while (~15 mins) and then turn them back on (and some times after their initial sync.) It's when the secondaries are catching back up to the primary that we see this. It's also probably worth noting that we have a pretty write-heavy load where we're making a lot of updates to documents that are on the order of a megabyte.

-Brent

Comment by Eric Milkie [ 12/Oct/12 ]

The bgsync buffer is capped at 256 MB. Could it be that the secondary is initially applying ops slower than the primary due to page faults (e.g. the data being updated is not yet paged in on the secondary?) – and that is why the bgsync buffer is filling so much?
How much RAM and swap does your secondary have? I chose 256 MB because it was a reasonable way to store several documents if they were of the maximum size, 16 MB. Did you see the bgsync buffer go over 268435456?

Comment by Brent Miller [ 12/Oct/12 ]

Hi Mike,

We never see any of the "[rsSync] replication batch size is x" lines go above 5000, but where we do see the DR102 warning messages is when we see a lot of messages regarding the bgsync buffer growing large:

Thu Oct 11 13:00:20 [rsSync] warning: DR102 too much data written uncommitted 314.577MB
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 183622568 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 183757061 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 183894478 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184020225 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184131226 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184271473 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184386664 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184517681 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184627968 bytes
Thu Oct 11 13:00:20 [rsBackgroundSync] bgsync buffer has 184771139 bytes
0xade6e1 0x78c421 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x73ddf4 0x7474fe 0xa61af4 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64
b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 
 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x78c421]
 mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0xf4) [0x73ddf4]
 mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x21e) [0x7474fe]
 mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4]
 mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 mongod() [0x7c3659]

It seems that the bgsync buffer will grow to the point where it will consume all ram and swap, which is what is causing the out of memory error. We first notice the DR102 warnings when the os starts swapping, so I'm guessing this is what is causing the data to be written uncommitted.

-Brent

Comment by Eric Milkie [ 04/Oct/12 ]

Hi Brent,
In the log snippit that you posted, there is no exception (it is just printing a stack trace for debugging purposes). Can you try running with log level 2? (-vv on the command line) I'm interested in the log lines that show how large the replication batches are. In 2.2.1, they ought to be capped at 100MB, or 5000 operations, whichever occurs first. While the cap doesn't directly correspond to the amount of journaling data, it would be difficult but not impossible to achieve 300MB of changed pages in a single journal commit.

Comment by Brent Miller [ 02/Oct/12 ]

FWIW, I'm still seeing the same exception that the original poster had in 2.2.1-rc0

If we set "nojournal = true" we don't have any issue syncing, but we'd rather leave journaling on. Is there any more info we can provide to help diagnose this issue?

Tue Oct 2 14:30:22 [rsSync] warning: DR102 too much data written uncommitted 314.991MB
Tue Oct 2 14:30:22 [rsSync] warning: DR102 too much data written uncommitted 314.995MB
Tue Oct 2 14:30:22 [rsSync] warning: DR102 too much data written uncommitted 314.999MB
0xaf9261 0x730c31 0x730d0f 0x730fe2 0x7310fd 0x73116b 0x72afea 0x85a9f8 0x81b49a 0x99821b 0x99ebac 0x9810cd 0x98324c 0x984737 0x99e128 0x99e1ca 0x99e5da 0xb3f299 0x7f53c5056e9a 0x7f
53c436c4bd
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf9261]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x730c31]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x730d0f]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x730fe2]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x7310fd]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x73116b]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x72afea]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x228) [0x85a9f8]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0x81b49a]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x99821b]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4+0x55c) [0x99ebac]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset11InitialSyncEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9+0x41d) [0x9810cd]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x83c) [0x98324c]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x984737]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x99e128]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x99e1ca]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod(_ZN5mongo15startSyncThreadEv+0xaa) [0x99e5da]
/opt/mongodb-linux-x86_64-2.2.1-rc0/bin/mongod() [0xb3f299]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f53c5056e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f53c436c4bd]

Comment by Eric Milkie [ 01/Oct/12 ]

Hi David, it may be too late for this answer but you can use 2.2.1rc0 to avoid the issue and finish syncing. If you restart mongod when the node is in RECOVERING stage, it shouldn't start over by recloning the data again; it should just start applying ops until it catches up.

Comment by David Snelling [ 24/Sep/12 ]

I am in the middle of this currently, is there anything I can do to get it to finish syncing? It is at the end of a sync to a new secondary that takes over 24 hours to sync. It is at the oplog stage and if I restart, it will start from complete scratch.

Comment by Eric Milkie [ 17/Sep/12 ]

The fixes in SERVER-6816, to be released in 2.2.1, will stop the crashes and ought to improve performance on secondaries as well. However, it may still be possible for secondaries to fall behind primaries for certain workloads. For certain types of writes, especially small updates to large documents, it could help to turn off index prefetches on secondaries: http://docs.mongodb.org/manual/reference/mongod/#cmdoption-mongod--replIndexPrefetch

Comment by Brent Miller [ 16/Sep/12 ]

We have the exact same problem here. This seemed to creep up a couple days ago even though we've been running 2.2 since its release. We had two secondaries and both fell behind to the point where they were unable to keep up, then they started to crash. Wiping the data dir on the secondaries and resyncing didn't help. We've had to demote one of them to an arbiter and disable the other just to keep the the majority up. When the secondaries crash, we'll see out of memory errors like:

Sat Sep 15 22:26:29 [rsSync] warning: DR102 too much data written uncommitted 314.593MB
Sat Sep 15 22:26:30 [journal] warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
0xade6e1 0x802c5a 0x77dc73 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f5cfd3bfe9a 0x7f5cfc6d54bd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
/usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x77dc73]
/usr/bin/mongod() [0x753da5]
/usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f5cfd3bfe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5cfc6d54bd]
Sat Sep 15 22:26:31 [journal] Assertion: 13524:out of memory AlignedBuilder
0xade6e1 0x8036eb 0x77dadc 0x77db09 0x62284e 0x622c12 0xa099f1 0xa0a779 0xa0ae24 0x7c3659 0x7f5cfd3bfe9a 0x7f5cfc6d54bd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0x8036eb]
/usr/bin/mongod() [0x77dadc]
/usr/bin/mongod(_ZN5mongo14AlignedBuilder8_reallocEjj+0x29) [0x77db09]
/usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x4e) [0x62284e]
/usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x622c12]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x141) [0xa099f1]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f5cfd3bfe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5cfc6d54bd]
Sat Sep 15 22:26:31 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder
Sat Sep 15 22:26:31 dur1
Sat Sep 15 22:26:31 Got signal: 6 (Aborted).

Sat Sep 15 22:26:31 Backtrace:
0xade6e1 0x5582d9 0x7f5cfc6194c0 0x7f5cfc619445 0x7f5cfc61cbab 0xb503f7 0xa09e1f 0xa0a779 0xa0ae24 0x7c3659 0x7f5cfd3bfe9a 0x7f5cfc6
d54bd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7f5cfc6194c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f5cfc619445]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f5cfc61cbab]
/usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0xb503f7]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0xa09e1f]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f5cfd3bfe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5cfc6d54bd]

It may be worth noting that the secondaries have much less RAM then the primary (4G vs 48G) and are only really used for running aggregate queries and fall-over.

Comment by Eric Milkie [ 06/Sep/12 ]

When you say a single insert will effectively lock up the box, do you mean the primary, the secondary, or both? (The messages above are only affecting a secondary)

Comment by Mike Khristo [ 06/Sep/12 ]

Small inserts. 1 collection has the most volume (up to 250M docs now). The other only has a few hundred thousand docs. All small documents - think of it like a hit tracker.

Things are working great, then all of a sudden, a single insert will effectively lock up the box. With most inserts taking <100ms, every hour, there are probably 2 that take over a 20seconds. All the docs are similar in size/scope.

Comment by Eric Milkie [ 04/Sep/12 ]

The messages are annoying and I'm working on suppressing them (see linked SERVER ticket).
They shouldn't be a direct cause of why your replication is falling behind, however.

What version did you use before you upgraded, and what was the performance like? Are you writing to multiple databases or collections? Are the documents large or small, and are the writes upserts, updates, or inserts?

Generated at Thu Feb 08 03:13:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.