-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 1.4.0
-
Component/s: Replication
-
None
-
Environment:64-bit CentOS Linux. Xeon X5272 @ 3.40GHz.
I'm starting a MongoDB slave with the following options:
/usr/local/mongodb/bin/mongod --dbpath /var/lib/mongodb --slave --source $MASTER --autoresync --profile 1 --quiet run
It start syncing files, but ultimately fails and then autorsyncs thanks to the --autoresync option.
[root@mongo1 ~]# grep "An earlier initial" /var/log/mongodb/mongodb.log
Fri Mar 26 12:29:48 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
Fri Mar 26 12:39:32 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
Fri Mar 26 12:49:12 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
Fri Mar 26 12:58:54 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
Fri Mar 26 13:08:40 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
...
It is repeated 24 times in ~4 hours.
Here's the errors in the log prior to one instance if the above error line (the same error occurs each time it gets this far in the replication process):
Fri Mar 26 12:57:54 building new index on
for mydb_loadtest.fields_current
Fri Mar 26 12:57:54 mydb_loadtest. Assertion failure ! _progressMeter.isActive() db/curop.h 247
0x4fe216 0x506914 0x61b2b1 0x60f887 0x6109a2 0x6125c6 0x5be31d 0x5c2286 0x5c30cd 0x59754e 0x5a6cea 0x5a89d7 0x5ab360 0x5ab56e 0x5abbac 0x5ac3e8 0x6bbde0 0x366ea06617 0x366e2d3c2d
/usr/local/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4fe216]
/usr/local/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x506914]
/usr/local/mongodb/bin/mongod(_ZN5mongo23BackgroundIndexBuildJob18addExistingToIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x5d1) [0x61b2b1]
/usr/local/mongodb/bin/mongod [0x60f887]
/usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x8a2) [0x6109a2]
/usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x6125c6]
/usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0xd5d) [0x5be31d]
/usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1346) [0x5c2286]
/usr/local/mongodb/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x5c30cd]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource6resyncESs+0x12e) [0x59754e]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource29sync_pullOpLog_applyOperationERNS_7BSONObjEPNS_6OpTimeE+0x77a) [0x5a6cea]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource14sync_pullOpLogERi+0x11c7) [0x5a89d7]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource4syncERi+0x3d0) [0x5ab360]
/usr/local/mongodb/bin/mongod(_ZN5mongo9_replMainERSt6vectorIN5boost10shared_ptrINS_10ReplSourceEEESaIS4_EERi+0xde) [0x5ab56e]
/usr/local/mongodb/bin/mongod(_ZN5mongo8replMainEv+0xcc) [0x5abbac]
/usr/local/mongodb/bin/mongod(_ZN5mongo15replSlaveThreadEv+0x2d8) [0x5ac3e8]
/usr/local/mongodb/bin/mongod(thread_proxy+0x80) [0x6bbde0]
/lib64/libpthread.so.0 [0x366ea06617]
/lib64/libc.so.6(clone+0x6d) [0x366e2d3c2d]
Fri Mar 26 12:57:54 mydb_loadtest. Assertion failure le && !saveerrmsg.empty() db/pdfile.cpp 1494
0x4fe216 0x506914 0x611f0f 0x6125c6 0x5be31d 0x5c2286 0x5c30cd 0x59754e 0x5a6cea 0x5a89d7 0x5ab360 0x5ab56e 0x5abbac 0x5ac3e8 0x6bbde0 0x366ea06617 0x366e2d3c2d
/usr/local/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4fe216]
/usr/local/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x506914]
/usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x1e0f) [0x611f0f]
/usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x6125c6]
/usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0xd5d) [0x5be31d]
/usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1346) [0x5c2286]
/usr/local/mongodb/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x5c30cd]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource6resyncESs+0x12e) [0x59754e]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource29sync_pullOpLog_applyOperationERNS_7BSONObjEPNS_6OpTimeE+0x77a) [0x5a6cea]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource14sync_pullOpLogERi+0x11c7) [0x5a89d7]
/usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource4syncERi+0x3d0) [0x5ab360]
/usr/local/mongodb/bin/mongod(_ZN5mongo9_replMainERSt6vectorIN5boost10shared_ptrINS_10ReplSourceEEESaIS4_EERi+0xde) [0x5ab56e]
/usr/local/mongodb/bin/mongod(_ZN5mongo8replMainEv+0xcc) [0x5abbac]
/usr/local/mongodb/bin/mongod(_ZN5mongo15replSlaveThreadEv+0x2d8) [0x5ac3e8]
/usr/local/mongodb/bin/mongod(thread_proxy+0x80) [0x6bbde0]
/lib64/libpthread.so.0 [0x366ea06617]
/lib64/libc.so.6(clone+0x6d) [0x366e2d3c2d]
Fri Mar 26 12:57:54 replMain AssertionException assertion db/pdfile.cpp:1494
Fri Mar 26 12:58:54 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
Fri Mar 26 12:58:54 resync: dropping database mydb_loadtest
I was originally running 1.3.4 on the master and 1.3.3 on the slave. I upgraded both to 1.4.0 today, removed the local data directory, and tried again but ran into the same identical problem.