Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7607

Delay on replica resynchronization

    XMLWordPrintable

    Details

    • Operating System:
      Linux

      Description

      I have a mongo instance used primarily for backup, running LVM on loop filesystems (losetup /dev/loop0 /lvmdata/data0). The LVM kernel module crashes occasionally, and once I bring the mongo instance up, it joins as a secondary (as expected), only it doesn't start catching up.

      After an hour or so, it begins to catch up. I noticed this in the logs after it was in sync. I am not sure of the timing of events, as I looked at the logs after it caught up.

      Fri Nov 9 20:47:34 [rsSync] warning: DR102 too much data written uncommitted 314.577MB
      Fri Nov 9 20:47:36 [rsSync] local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101
      0xade6e1 0x802c5a 0x78c4a0 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x739675 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f22b7718e9a 0
      x7f22b6a2ddbd
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
      /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
      /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x280) [0x78c4a0]
      /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
      /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
      /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
      /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
      /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x195) [0x739675]
      /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
      /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
      /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
      /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
      /usr/bin/mongod() [0x7c3659]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f22b7718e9a]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f22b6a2ddbd]
      Fri Nov 9 20:47:41 [rsSync] local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101
      0xade6e1 0x802c5a 0x78c4a0 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xb670ba 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f22b7718e9a 0x7f22b6a2
      ddbd
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
      /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
      /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x280) [0x78c4a0]
      /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
      /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
      /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
      /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
      /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
      /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x2aa) [0xb670ba]
      /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
      /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
      /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
      /usr/bin/mongod() [0x7c3659]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f22b7718e9a]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f22b6a2ddbd]

        Attachments

          Activity

            People

            Assignee:
            jantje.daniel Ian Daniel
            Reporter:
            fumduq Eric Sabban
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: