Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3283

[replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.8.2
    • Component/s: Replication
    • Labels:
      None
    • Environment:
      Ubuntu on EC2
    • Linux

      I started seeing this assertion error on one of the slaves in one of my replica sets. It seems to have caused replication to stop.

      Fri Jun 17 04:15:24 [conn1059] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded68a18601f9f32f826c44') }

      reslen:60 300009ms
      Fri Jun 17 04:15:25 [conn924] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded637aef4335230f35faf2') }

      reslen:60 300007ms
      Fri Jun 17 04:15:25 [conn846] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded63529d03464f291491fd') }

      reslen:60 300009ms
      Fri Jun 17 04:15:25 [conn841] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded633d2179e8c686856149') }

      reslen:60 300009ms
      Fri Jun 17 04:15:27 [conn837] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded6336eb7379566e2c7ec9') }

      reslen:60 300010ms
      Fri Jun 17 04:15:52 [conn7822] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4dee9a6cd4822fa37bc008fe') }

      reslen:60 300007ms
      Fri Jun 17 04:15:56 [conn551] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded55b540adf833f570a90d') }

      reslen:60 300008ms
      Fri Jun 17 04:16:15 [dur] lsn set 445741544
      Fri Jun 17 04:16:27 [conn850] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded635aca11c4f8c54ea25d') }

      reslen:60 300009ms
      Fri Jun 17 04:17:06 [conn29103] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4df25181d059bd052edf5abd') }

      reslen:60 300008ms
      Fri Jun 17 04:17:15 [dur] lsn set 445771580
      Fri Jun 17 04:17:16 [conn919] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded63638a912a64992a16e3') }

      reslen:60 300008ms
      Fri Jun 17 04:17:18 [conn840] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded6345ccacebb71829396e') }

      reslen:60 300008ms
      Fri Jun 17 04:17:18 [conn917] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded636c6b00c1f2c76d1dbe') }

      reslen:60 300008ms
      Fri Jun 17 04:17:23 [conn833] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4ded632f2bbe6d881ab63b90') }

      reslen:60 300008ms
      Fri Jun 17 04:17:44 [replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259
      0x54f08e 0x5606dd 0x64dbbe 0x76f563 0x772768 0x73dc97 0x732135 0x73217a 0x76fe5d 0x710f23 0x7080e5 0x7084a8 0x70852c 0x70b342 0x8baea0 0x7ff247ff69ca 0x7ff2475a570d
      /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f08e]
      /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x5606dd]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x8e) [0x64dbbe]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0x113) [0x76f563]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbb+0x198) [0x772768]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x377) [0x73dc97]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x732135]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x73217a]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x18d) [0x76fe5d]
      /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x253) [0x710f23]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x1b25) [0x7080e5]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc8) [0x7084a8]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70852c]
      /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x252) [0x70b342]
      /usr/bin/mongod(thread_proxy+0x80) [0x8baea0]
      /lib/libpthread.so.0(+0x69ca) [0x7ff247ff69ca]
      /lib/libc.so.6(clone+0x6d) [0x7ff2475a570d]
      Fri Jun 17 04:17:44 [replica set sync] replSet syncThread: 0 assertion db/pdfile.h:259
      Fri Jun 17 04:17:55 [replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259
      0x54f08e 0x5606dd 0x64dbbe 0x76f563 0x772768 0x73dc97 0x732135 0x73217a 0x76fe5d 0x710f23 0x7080e5 0x7084a8 0x70852c 0x70b342 0x8baea0 0x7ff247ff69ca 0x7ff2475a570d
      /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f08e]
      /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x5606dd]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x8e) [0x64dbbe]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0x113) [0x76f563]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbb+0x198) [0x772768]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x377) [0x73dc97]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x732135]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x73217a]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x18d) [0x76fe5d]
      /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x253) [0x710f23]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x1b25) [0x7080e5]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc8) [0x7084a8]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70852c]
      /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x252) [0x70b342]
      /usr/bin/mongod(thread_proxy+0x80) [0x8baea0]
      /lib/libpthread.so.0(+0x69ca) [0x7ff247ff69ca]
      /lib/libc.so.6(clone+0x6d) [0x7ff2475a570d]
      Fri Jun 17 04:17:55 [replica set sync] replSet syncThread: 0 assertion db/pdfile.h:259

      Restarting the slave has had no effect, as you can see:

      Fri Jun 17 15:26:24 [initandlisten] MongoDB starting : pid=27662 port=27018 dbpath=/var/lib/mongodb 64-bit
      Fri Jun 17 15:26:24 [initandlisten] db version v1.8.2-rc3, pdfile version 4.5
      Fri Jun 17 15:26:24 [initandlisten] git version: 2d7719228787c9c8100456bc70bf860ec2885732
      Fri Jun 17 15:26:24 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
      Fri Jun 17 15:26:24 [initandlisten] journal dir=/var/lib/mongodb/journal
      Fri Jun 17 15:26:24 [initandlisten] recover : no journal files present, no recovery needed
      Fri Jun 17 15:26:24 [initandlisten] preallocateIsFaster=true 2.44
      Fri Jun 17 15:26:24 [initandlisten] preallocateIsFaster=true 2.6
      Fri Jun 17 15:26:24 [initandlisten] preallocateIsFaster=true 2.5
      Fri Jun 17 15:26:24 [initandlisten] waiting for connections on port 27018
      Fri Jun 17 15:26:24 [websvr] web admin interface listening on port 28018
      Fri Jun 17 15:26:24 [startReplSets] replSet STARTUP2
      Fri Jun 17 15:26:24 [rs Manager] replSet can't see a majority, will not try to elect self
      Fri Jun 17 15:26:24 [replica set sync] replSet SECONDARY
      Fri Jun 17 15:26:24 [ReplSetHealthPollTask] replSet info mongodb09.example.com:27018 is up
      Fri Jun 17 15:26:24 [ReplSetHealthPollTask] replSet member mongodb09.example.com:27018 ARBITER
      Fri Jun 17 15:26:24 [rs Manager] replSet info not electing self, we are not freshest
      Fri Jun 17 15:26:26 [ReplSetHealthPollTask] replSet info mongodb07.example.com:27018 is up
      Fri Jun 17 15:26:26 [ReplSetHealthPollTask] replSet member mongodb07.example.com:27018 PRIMARY
      Fri Jun 17 15:26:26 [replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259
      0x54f08e 0x5606dd 0x64dbbe 0x76f563 0x772768 0x73dc97 0x732135 0x73217a 0x76fe5d 0x710f23 0x7080e5 0x7084a8 0x70852c 0x70b342 0x8baea0 0x7f2a569f89ca 0x7f2a55fa770d
      /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f08e]
      /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x5606dd]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x8e) [0x64dbbe]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0x113) [0x76f563]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbb+0x198) [0x772768]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x377) [0x73dc97]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x732135]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x73217a]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x18d) [0x76fe5d]
      /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x253) [0x710f23]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x1b25) [0x7080e5]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc8) [0x7084a8]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70852c]
      /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x252) [0x70b342]
      /usr/bin/mongod(thread_proxy+0x80) [0x8baea0]
      /lib/libpthread.so.0(+0x69ca) [0x7f2a569f89ca]
      /lib/libc.so.6(clone+0x6d) [0x7f2a55fa770d]
      Fri Jun 17 15:26:26 [replica set sync] replSet syncThread: 0 assertion db/pdfile.h:259

            Assignee:
            kristina Kristina Chodorow (Inactive)
            Reporter:
            mconigliaro Michael Conigliaro
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: