-
Type: Bug
-
Resolution: Won't Fix
-
Priority: Major - P3
-
None
-
Affects Version/s: 1.8.2
-
Component/s: Replication
-
None
-
Environment:Ubuntu on EC2
-
Linux
I started seeing this assertion error on one of the slaves in one of my replica sets. It seems to have caused replication to stop.
Fri Jun 17 04:15:24 [conn1059] query admin.$cmd ntoreturn:1 command:
{ writebacklisten: ObjectId('4ded68a18601f9f32f826c44') } reslen:60 300009ms
Fri Jun 17 04:15:25 [conn924] query admin.$cmd ntoreturn:1 command:
reslen:60 300007ms
Fri Jun 17 04:15:25 [conn846] query admin.$cmd ntoreturn:1 command:
reslen:60 300009ms
Fri Jun 17 04:15:25 [conn841] query admin.$cmd ntoreturn:1 command:
reslen:60 300009ms
Fri Jun 17 04:15:27 [conn837] query admin.$cmd ntoreturn:1 command:
reslen:60 300010ms
Fri Jun 17 04:15:52 [conn7822] query admin.$cmd ntoreturn:1 command:
reslen:60 300007ms
Fri Jun 17 04:15:56 [conn551] query admin.$cmd ntoreturn:1 command:
reslen:60 300008ms
Fri Jun 17 04:16:15 [dur] lsn set 445741544
Fri Jun 17 04:16:27 [conn850] query admin.$cmd ntoreturn:1 command:
reslen:60 300009ms
Fri Jun 17 04:17:06 [conn29103] query admin.$cmd ntoreturn:1 command:
reslen:60 300008ms
Fri Jun 17 04:17:15 [dur] lsn set 445771580
Fri Jun 17 04:17:16 [conn919] query admin.$cmd ntoreturn:1 command:
reslen:60 300008ms
Fri Jun 17 04:17:18 [conn840] query admin.$cmd ntoreturn:1 command:
reslen:60 300008ms
Fri Jun 17 04:17:18 [conn917] query admin.$cmd ntoreturn:1 command:
reslen:60 300008ms
Fri Jun 17 04:17:23 [conn833] query admin.$cmd ntoreturn:1 command:
reslen:60 300008ms
Fri Jun 17 04:17:44 [replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259
0x54f08e 0x5606dd 0x64dbbe 0x76f563 0x772768 0x73dc97 0x732135 0x73217a 0x76fe5d 0x710f23 0x7080e5 0x7084a8 0x70852c 0x70b342 0x8baea0 0x7ff247ff69ca 0x7ff2475a570d
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f08e]
/usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x5606dd]
/usr/bin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x8e) [0x64dbbe]
/usr/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0x113) [0x76f563]
/usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbb+0x198) [0x772768]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x377) [0x73dc97]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x732135]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x73217a]
/usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x18d) [0x76fe5d]
/usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x253) [0x710f23]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x1b25) [0x7080e5]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc8) [0x7084a8]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70852c]
/usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x252) [0x70b342]
/usr/bin/mongod(thread_proxy+0x80) [0x8baea0]
/lib/libpthread.so.0(+0x69ca) [0x7ff247ff69ca]
/lib/libc.so.6(clone+0x6d) [0x7ff2475a570d]
Fri Jun 17 04:17:44 [replica set sync] replSet syncThread: 0 assertion db/pdfile.h:259
Fri Jun 17 04:17:55 [replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259
0x54f08e 0x5606dd 0x64dbbe 0x76f563 0x772768 0x73dc97 0x732135 0x73217a 0x76fe5d 0x710f23 0x7080e5 0x7084a8 0x70852c 0x70b342 0x8baea0 0x7ff247ff69ca 0x7ff2475a570d
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f08e]
/usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x5606dd]
/usr/bin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x8e) [0x64dbbe]
/usr/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0x113) [0x76f563]
/usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbb+0x198) [0x772768]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x377) [0x73dc97]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x732135]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x73217a]
/usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x18d) [0x76fe5d]
/usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x253) [0x710f23]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x1b25) [0x7080e5]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc8) [0x7084a8]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70852c]
/usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x252) [0x70b342]
/usr/bin/mongod(thread_proxy+0x80) [0x8baea0]
/lib/libpthread.so.0(+0x69ca) [0x7ff247ff69ca]
/lib/libc.so.6(clone+0x6d) [0x7ff2475a570d]
Fri Jun 17 04:17:55 [replica set sync] replSet syncThread: 0 assertion db/pdfile.h:259
Restarting the slave has had no effect, as you can see:
Fri Jun 17 15:26:24 [initandlisten] MongoDB starting : pid=27662 port=27018 dbpath=/var/lib/mongodb 64-bit
Fri Jun 17 15:26:24 [initandlisten] db version v1.8.2-rc3, pdfile version 4.5
Fri Jun 17 15:26:24 [initandlisten] git version: 2d7719228787c9c8100456bc70bf860ec2885732
Fri Jun 17 15:26:24 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Fri Jun 17 15:26:24 [initandlisten] journal dir=/var/lib/mongodb/journal
Fri Jun 17 15:26:24 [initandlisten] recover : no journal files present, no recovery needed
Fri Jun 17 15:26:24 [initandlisten] preallocateIsFaster=true 2.44
Fri Jun 17 15:26:24 [initandlisten] preallocateIsFaster=true 2.6
Fri Jun 17 15:26:24 [initandlisten] preallocateIsFaster=true 2.5
Fri Jun 17 15:26:24 [initandlisten] waiting for connections on port 27018
Fri Jun 17 15:26:24 [websvr] web admin interface listening on port 28018
Fri Jun 17 15:26:24 [startReplSets] replSet STARTUP2
Fri Jun 17 15:26:24 [rs Manager] replSet can't see a majority, will not try to elect self
Fri Jun 17 15:26:24 [replica set sync] replSet SECONDARY
Fri Jun 17 15:26:24 [ReplSetHealthPollTask] replSet info mongodb09.example.com:27018 is up
Fri Jun 17 15:26:24 [ReplSetHealthPollTask] replSet member mongodb09.example.com:27018 ARBITER
Fri Jun 17 15:26:24 [rs Manager] replSet info not electing self, we are not freshest
Fri Jun 17 15:26:26 [ReplSetHealthPollTask] replSet info mongodb07.example.com:27018 is up
Fri Jun 17 15:26:26 [ReplSetHealthPollTask] replSet member mongodb07.example.com:27018 PRIMARY
Fri Jun 17 15:26:26 [replica set sync] local.oplog.rs Assertion failure isOk() db/pdfile.h 259
0x54f08e 0x5606dd 0x64dbbe 0x76f563 0x772768 0x73dc97 0x732135 0x73217a 0x76fe5d 0x710f23 0x7080e5 0x7084a8 0x70852c 0x70b342 0x8baea0 0x7f2a569f89ca 0x7f2a55fa770d
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f08e]
/usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x5606dd]
/usr/bin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x8e) [0x64dbbe]
/usr/bin/mongod(_ZN5mongo11DataFileMgr13_deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocE+0x113) [0x76f563]
/usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbb+0x198) [0x772768]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x377) [0x73dc97]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x732135]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x73217a]
/usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x18d) [0x76fe5d]
/usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x253) [0x710f23]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x1b25) [0x7080e5]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc8) [0x7084a8]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x3c) [0x70852c]
/usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x252) [0x70b342]
/usr/bin/mongod(thread_proxy+0x80) [0x8baea0]
/lib/libpthread.so.0(+0x69ca) [0x7f2a569f89ca]
/lib/libc.so.6(clone+0x6d) [0x7f2a55fa770d]
Fri Jun 17 15:26:26 [replica set sync] replSet syncThread: 0 assertion db/pdfile.h:259