Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-837

Assertion failure ! _progressMeter.isActive() db/curop.h 247, replication restarts over and over

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 1.4.1, 1.5.0
    • Affects Version/s: 1.4.0
    • Component/s: Replication
    • Labels:
      None
    • Environment:
      64-bit CentOS Linux. Xeon X5272 @ 3.40GHz.

      I'm starting a MongoDB slave with the following options:
      /usr/local/mongodb/bin/mongod --dbpath /var/lib/mongodb --slave --source $MASTER --autoresync --profile 1 --quiet run

      It start syncing files, but ultimately fails and then autorsyncs thanks to the --autoresync option.

      [root@mongo1 ~]# grep "An earlier initial" /var/log/mongodb/mongodb.log
      Fri Mar 26 12:29:48 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:39:32 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:49:12 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:58:54 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 13:08:40 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      ...
      It is repeated 24 times in ~4 hours.

      Here's the errors in the log prior to one instance if the above error line (the same error occurs each time it gets this far in the replication process):
      Fri Mar 26 12:57:54 building new index on

      { _bundle: 1, status: 1 }

      for mydb_loadtest.fields_current
      Fri Mar 26 12:57:54 mydb_loadtest. Assertion failure ! _progressMeter.isActive() db/curop.h 247
      0x4fe216 0x506914 0x61b2b1 0x60f887 0x6109a2 0x6125c6 0x5be31d 0x5c2286 0x5c30cd 0x59754e 0x5a6cea 0x5a89d7 0x5ab360 0x5ab56e 0x5abbac 0x5ac3e8 0x6bbde0 0x366ea06617 0x366e2d3c2d
      /usr/local/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4fe216]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x506914]
      /usr/local/mongodb/bin/mongod(_ZN5mongo23BackgroundIndexBuildJob18addExistingToIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x5d1) [0x61b2b1]
      /usr/local/mongodb/bin/mongod [0x60f887]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x8a2) [0x6109a2]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x6125c6]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0xd5d) [0x5be31d]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1346) [0x5c2286]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x5c30cd]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource6resyncESs+0x12e) [0x59754e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource29sync_pullOpLog_applyOperationERNS_7BSONObjEPNS_6OpTimeE+0x77a) [0x5a6cea]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource14sync_pullOpLogERi+0x11c7) [0x5a89d7]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource4syncERi+0x3d0) [0x5ab360]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9_replMainERSt6vectorIN5boost10shared_ptrINS_10ReplSourceEEESaIS4_EERi+0xde) [0x5ab56e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8replMainEv+0xcc) [0x5abbac]
      /usr/local/mongodb/bin/mongod(_ZN5mongo15replSlaveThreadEv+0x2d8) [0x5ac3e8]
      /usr/local/mongodb/bin/mongod(thread_proxy+0x80) [0x6bbde0]
      /lib64/libpthread.so.0 [0x366ea06617]
      /lib64/libc.so.6(clone+0x6d) [0x366e2d3c2d]
      Fri Mar 26 12:57:54 mydb_loadtest. Assertion failure le && !saveerrmsg.empty() db/pdfile.cpp 1494
      0x4fe216 0x506914 0x611f0f 0x6125c6 0x5be31d 0x5c2286 0x5c30cd 0x59754e 0x5a6cea 0x5a89d7 0x5ab360 0x5ab56e 0x5abbac 0x5ac3e8 0x6bbde0 0x366ea06617 0x366e2d3c2d
      /usr/local/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4fe216]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x506914]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x1e0f) [0x611f0f]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x6125c6]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0xd5d) [0x5be31d]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1346) [0x5c2286]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x5c30cd]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource6resyncESs+0x12e) [0x59754e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource29sync_pullOpLog_applyOperationERNS_7BSONObjEPNS_6OpTimeE+0x77a) [0x5a6cea]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource14sync_pullOpLogERi+0x11c7) [0x5a89d7]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource4syncERi+0x3d0) [0x5ab360]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9_replMainERSt6vectorIN5boost10shared_ptrINS_10ReplSourceEEESaIS4_EERi+0xde) [0x5ab56e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8replMainEv+0xcc) [0x5abbac]
      /usr/local/mongodb/bin/mongod(_ZN5mongo15replSlaveThreadEv+0x2d8) [0x5ac3e8]
      /usr/local/mongodb/bin/mongod(thread_proxy+0x80) [0x6bbde0]
      /lib64/libpthread.so.0 [0x366ea06617]
      /lib64/libc.so.6(clone+0x6d) [0x366e2d3c2d]
      Fri Mar 26 12:57:54 replMain AssertionException assertion db/pdfile.cpp:1494
      Fri Mar 26 12:58:54 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:58:54 resync: dropping database mydb_loadtest

      I was originally running 1.3.4 on the master and 1.3.3 on the slave. I upgraded both to 1.4.0 today, removed the local data directory, and tried again but ran into the same identical problem.

            Assignee:
            eliot Eliot Horowitz (Inactive)
            Reporter:
            jeremy Jeremy Andrews
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: