Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6790

'[Balancer] Assertion: 13141:Chunk map pointed to incorrect chunk' error in the mongos log

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • 2.2.0-rc1
    • None
    • Sharding
    • None
    • All servers are EC2 hosts, Ubuntu 11.04 (GNU/Linux 2.6.38-13-virtual x86_64)
      9 Shard hosts (m2.4xlarge), 3 Replica sets, 3 servers in each
      3 config servers (m1.small)
      2 mongos hosts (m1.medium)
    • Linux

    Description

      Recently we sharded few big collections in our database (2.2.0-rc1)

      First collection was balanced just fine:
      oggiReporting.site.mainStats chunks:
      RS02 238
      RS03 238
      RS01 239

      But for second collection we get a balancer error in mongos log every few seconds:

      Fri Aug 17 00:56:27 [Balancer] ns: oggiReporting.playlist.mainStats going to move { _id: "oggiReporting.playlist.mainStats-campaignId_MinKeyplaylistId_MinKeydate_MinKey", lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('502be6c21d848a61a13d51eb'), ns: "oggiReporting.playlist.mainStats", min:

      { campaignId: MinKey, playlistId: MinKey, date: MinKey }

      , max:

      { campaignId: "014869ee-5d62-4212-a609-40edcc2169d3", playlistId: "0c5bb01a-5780-413d-8d7c-b22c722e6ad4", date: new Date(1309309200000) }

      , shard: "RS01" } from: RS01 to: RS02 tag []
      foo:

      { campaignId: "0095af50-203b-4e1a-9337-6dad60a46688", playlistId: "09038ddd-ac89-4818-b6c3-90c3c43cdce9", domain: "biography.com", date: new Date(1311120000000), partition: "2011-07" }

      *c: ns:oggiReporting.domain.mainStats at: RS02:RS02/RPTDB-RS02-Zuse1b-S01.oggifinogi.com:27018,RPTDB-RS02-Zuse1c-S01.oggifinogi.com:27018,RPTDB-RS02-Zuse1d-S01.oggifinogi.com:27018 lastmod: 2|0||000000000000000000000000 min:

      { campaignId: MinKey, playlistId: MinKey, domain: MinKey, date: MinKey }

      max:

      { campaignId: "0095af50-203b-4e1a-9337-6dad60a46688", playlistId: "09038ddd-ac89-4818-b6c3-90c3c43cdce9", domain: "biography.com", date: new Date(1311120000000), partition: "2011-07" }

      key:

      { campaignId: "0095af50-203b-4e1a-9337-6dad60a46688", playlistId: "09038ddd-ac89-4818-b6c3-90c3c43cdce9", domain: "biography.com", date: new Date(1311120000000) }

      Fri Aug 17 00:56:27 [Balancer] Assertion: 13141:Chunk map pointed to incorrect chunk
      0x5389f1 0x6ed1cb 0x57d090 0x5312ec 0x532d66 0x670b9e 0x6724e4 0x72ab79 0x7f4a00f4cd8c 0x7f4a002eec2d
      /usr/bin/mongos(_ZN5mongo15printStackTraceERSo+0x21) [0x5389f1]
      /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x9b) [0x6ed1cb]
      /usr/bin/mongos(_ZNK5mongo12ChunkManager9findChunkERKNS_7BSONObjE+0x540) [0x57d090]
      /usr/bin/mongos(_ZN5mongo8Balancer11_moveChunksEPKSt6vectorIN5boost10shared_ptrINS_11MigrateInfoEEESaIS5_EEb+0x5ac) [0x5312ec]
      /usr/bin/mongos(_ZN5mongo8Balancer3runEv+0x986) [0x532d66]
      /usr/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xbe) [0x670b9e]
      /usr/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x6724e4]
      /usr/bin/mongos() [0x72ab79]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x7f4a00f4cd8c]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f4a002eec2d]
      Fri Aug 17 00:56:27 [Balancer] distributed lock 'balancer/RPTSRVC-Zuse1b-S01.oggifinogi.com:27017:1345162100:1804289383' unlocked.
      Fri Aug 17 00:56:27 [Balancer] scoped connection to RPTDB-CFG-Zuse1b-S01.oggifinogi.com:27019,RPTDB-CFG-Zuse1c-S01.oggifinogi.com:27019,RPTDB-CFG-Zuse1d-S01.oggifinogi.com:27019 not being returned to the pool
      Fri Aug 17 00:56:27 [Balancer] caught exception while doing balance: Chunk map pointed to incorrect chunk

      I tried to reboot mongos as well as config and replica set servers - no luck

      Attachments

        1. configdb.dmp.tgz
          300 kB
        2. configdb.dmp.tgz
          236 kB
        3. logs.zip
          1.46 MB

        Issue Links

          Activity

            People

              spencer@mongodb.com Spencer Brody (Inactive)
              vladimir.poluyaktov Vladimir Poluyaktov
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: