Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-837

Assertion failure ! _progressMeter.isActive() db/curop.h 247, replication restarts over and over

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • 1.4.1, 1.5.0
    • 1.4.0
    • Replication
    • None
    • 64-bit CentOS Linux. Xeon X5272 @ 3.40GHz.

    Description

      I'm starting a MongoDB slave with the following options:
      /usr/local/mongodb/bin/mongod --dbpath /var/lib/mongodb --slave --source $MASTER --autoresync --profile 1 --quiet run

      It start syncing files, but ultimately fails and then autorsyncs thanks to the --autoresync option.

      [root@mongo1 ~]# grep "An earlier initial" /var/log/mongodb/mongodb.log
      Fri Mar 26 12:29:48 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:39:32 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:49:12 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:58:54 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 13:08:40 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      ...
      It is repeated 24 times in ~4 hours.

      Here's the errors in the log prior to one instance if the above error line (the same error occurs each time it gets this far in the replication process):
      Fri Mar 26 12:57:54 building new index on

      { _bundle: 1, status: 1 }

      for mydb_loadtest.fields_current
      Fri Mar 26 12:57:54 mydb_loadtest. Assertion failure ! _progressMeter.isActive() db/curop.h 247
      0x4fe216 0x506914 0x61b2b1 0x60f887 0x6109a2 0x6125c6 0x5be31d 0x5c2286 0x5c30cd 0x59754e 0x5a6cea 0x5a89d7 0x5ab360 0x5ab56e 0x5abbac 0x5ac3e8 0x6bbde0 0x366ea06617 0x366e2d3c2d
      /usr/local/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4fe216]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x506914]
      /usr/local/mongodb/bin/mongod(_ZN5mongo23BackgroundIndexBuildJob18addExistingToIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x5d1) [0x61b2b1]
      /usr/local/mongodb/bin/mongod [0x60f887]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x8a2) [0x6109a2]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x6125c6]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0xd5d) [0x5be31d]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1346) [0x5c2286]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x5c30cd]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource6resyncESs+0x12e) [0x59754e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource29sync_pullOpLog_applyOperationERNS_7BSONObjEPNS_6OpTimeE+0x77a) [0x5a6cea]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource14sync_pullOpLogERi+0x11c7) [0x5a89d7]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource4syncERi+0x3d0) [0x5ab360]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9_replMainERSt6vectorIN5boost10shared_ptrINS_10ReplSourceEEESaIS4_EERi+0xde) [0x5ab56e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8replMainEv+0xcc) [0x5abbac]
      /usr/local/mongodb/bin/mongod(_ZN5mongo15replSlaveThreadEv+0x2d8) [0x5ac3e8]
      /usr/local/mongodb/bin/mongod(thread_proxy+0x80) [0x6bbde0]
      /lib64/libpthread.so.0 [0x366ea06617]
      /lib64/libc.so.6(clone+0x6d) [0x366e2d3c2d]
      Fri Mar 26 12:57:54 mydb_loadtest. Assertion failure le && !saveerrmsg.empty() db/pdfile.cpp 1494
      0x4fe216 0x506914 0x611f0f 0x6125c6 0x5be31d 0x5c2286 0x5c30cd 0x59754e 0x5a6cea 0x5a89d7 0x5ab360 0x5ab56e 0x5abbac 0x5ac3e8 0x6bbde0 0x366ea06617 0x366e2d3c2d
      /usr/local/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4fe216]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x506914]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x1e0f) [0x611f0f]
      /usr/local/mongodb/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x6125c6]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0xd5d) [0x5be31d]
      /usr/local/mongodb/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1346) [0x5c2286]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x5c30cd]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource6resyncESs+0x12e) [0x59754e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource29sync_pullOpLog_applyOperationERNS_7BSONObjEPNS_6OpTimeE+0x77a) [0x5a6cea]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource14sync_pullOpLogERi+0x11c7) [0x5a89d7]
      /usr/local/mongodb/bin/mongod(_ZN5mongo10ReplSource4syncERi+0x3d0) [0x5ab360]
      /usr/local/mongodb/bin/mongod(_ZN5mongo9_replMainERSt6vectorIN5boost10shared_ptrINS_10ReplSourceEEESaIS4_EERi+0xde) [0x5ab56e]
      /usr/local/mongodb/bin/mongod(_ZN5mongo8replMainEv+0xcc) [0x5abbac]
      /usr/local/mongodb/bin/mongod(_ZN5mongo15replSlaveThreadEv+0x2d8) [0x5ac3e8]
      /usr/local/mongodb/bin/mongod(thread_proxy+0x80) [0x6bbde0]
      /lib64/libpthread.so.0 [0x366ea06617]
      /lib64/libc.so.6(clone+0x6d) [0x366e2d3c2d]
      Fri Mar 26 12:57:54 replMain AssertionException assertion db/pdfile.cpp:1494
      Fri Mar 26 12:58:54 An earlier initial clone of 'mydb_loadtest' did not complete, now resyncing.
      Fri Mar 26 12:58:54 resync: dropping database mydb_loadtest

      I was originally running 1.3.4 on the master and 1.3.3 on the slave. I upgraded both to 1.4.0 today, removed the local data directory, and tried again but ran into the same identical problem.

      Attachments

        Activity

          People

            eliot Eliot Horowitz (Inactive)
            jeremy Jeremy Andrews
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: