index build failed, ! _progressMeter.isActive()

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 2.6.8
    • Component/s: Replication
    • None
    • Fully Compatible
    • Hide

      I'm not sure that you can repro, as this probably relates to specific DB layout.

      Here's the setup we're running on though:

      • Mongo 2.6.8
      • ubuntu 14.04 LTS
      • uname -a -> Linux production-db-b1v0 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      • Running on AWS, m3.large, with 2 1T SSD ebs volumes running raid 0
      Show
      I'm not sure that you can repro, as this probably relates to specific DB layout. Here's the setup we're running on though: Mongo 2.6.8 ubuntu 14.04 LTS uname -a -> Linux production-db-b1v0 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Running on AWS, m3.large, with 2 1T SSD ebs volumes running raid 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Most of this leadup shouldn't matter, but I'm adding it for completeness.

      We were running mongo 2.4.10 until recently.
      We have 3 mongo DB machines and 2 arbiters running in a replication set.
      I upgraded each machine to 2.6.8.
      When I went to upgrade the 3'rd Db (after all other machines were upgraded) it ran into some problems, likely related to ulimits etc.

      So, I removed it from the replica set.
      I upgraded the auth data as per: http://docs.mongodb.org/manual/release-notes/2.6-upgrade-authorization/
      I then got a new fresh machine, installed 2.6.8 on it, and tried to resync it.
      We ran in to some problems with various limits (ulimits, and total system memory mapping limit), but eventually resolved those.

      Now, when the replica tries to resync, the log has errors all over, similar to those at the bottom of the post and it never finishes syncing. It might make it up to 26% of the 2T disk, but then drops to 6% again.

      2015-03-22T06:57:19.526+0000 [FileAllocator] allocating new datafile /db/mongodb/worplay-for-waterlink_meteor_com.0, filling with zeroes...
      2015-03-22T06:57:19.529+0000 [FileAllocator] done allocating datafile /db/mongodb/worplay-for-waterlink_meteor_com.0, size: 16MB, took 0.002 secs
      2015-03-22T06:57:19.535+0000 [rsSync] build index on: worplay-for-waterlink_meteor_com.system.users properties: { v: 1, key:

      { _id: 1 }

      , name: "id", ns: "worplay-for-waterlink_meteor_com.system.users" }
      2015-03-22T06:57:19.535+0000 [rsSync] building index using bulk method
      2015-03-22T06:57:19.535+0000 [rsSync] worplay-for-waterlink_meteor_com Assertion failure ! _progressMeter.isActive() src/mongo/db/curop.cpp 154
      2015-03-22T06:57:19.542+0000 [rsSync] worplay-for-waterlink_meteor_com 0x1205431 0x11a7229 0x118b53e 0xa1a460 0x8eea05 0x8f13c2 0x8e70f2 0x8e8d47 0x910a62 0xea0d28 0xea1dad 0xea3177 0xec3844 0xec38f0 0xec3c0a 0x1249dc9 0x7f3e79b1a182 0x7f3e78e1efbd
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x1205431]
      /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x11a7229]
      /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x118b53e]
      /usr/bin/mongod(_ZN5mongo5CurOp10setMessageEPKcSsyi+0x1c0) [0xa1a460]
      /usr/bin/mongod(_ZN5mongo18addExistingToIndexEPNS_10CollectionEPKNS_15IndexDescriptorEPNS_17IndexAccessMethodEb+0x1a5) [0x8eea05]
      /usr/bin/mongod(_ZN5mongo12buildAnIndexEPNS_10CollectionEPNS_17IndexCatalogEntryEb+0x8e2) [0x8f13c2]
      /usr/bin/mongod(_ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE+0x922) [0x8e70f2]
      /usr/bin/mongod(_ZN5mongo12IndexCatalog17ensureHaveIdIndexEv+0x427) [0x8e8d47]
      /usr/bin/mongod(_ZN5mongo6Cloner2goERNS_6Client7ContextERKSsRKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0x1ad2) [0x910a62]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl17_initialSyncCloneERNS_6ClonerERKSsRKSt4listISsSaISsEEb+0x318) [0xea0d28]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl12_initialSyncEv+0x7ad) [0xea1dad]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0xea3177]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x124) [0xec3844]
      /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x30) [0xec38f0]
      /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xaa) [0xec3c0a]
      /usr/bin/mongod() [0x1249dc9]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f3e79b1a182]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f3e78e1efbd]
      2015-03-22T06:57:19.542+0000 [rsSync] index build failed. spec: { v: 1, key:

      { _id: 1 }

      , name: "id", ns: "worplay-for-waterlink_meteor_com.system.users" } error: 0 assertion src/mongo/db/curop.cpp:154

      Thanks... happy to add more detail if it's needed.

            Assignee:
            Sam Kleinman (Inactive)
            Reporter:
            Matthew Brewer
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: