Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29397

Invariant failure on config server when inserting tag into config.tags

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.4.11, 3.6.1, 3.7.1
    • Affects Version/s: 3.4.4, 3.5.8
    • Component/s: Sharding
    • Labels:
      None
    • Fully Compatible
    • ALL
    • v3.6, v3.4
    • Hide
      1. Ensure MongoDB 3.4 installed and available from "C:\Program Files\MongoDB\Server\3.4\bin
      2. Unzip attached LocalMongoDB.zip to e.g. C:\LocalFolder\temp\MongoDBUpgrade\3.4
      3. In all bat files update MONGODB_VERSION to 3.4
      4. Start all bat files
      5. (I used Robomongo 0.9.0-RC7 to run the following commands)
      6. Open mongo shell on localhost:27001 (LN1 mongod), and run:
      7. config = {
        _id : "LN1",
        members : [
        {_id : 1, host : "localhost:27001"},
        {_id : 2, host : "localhost:27002"},
        {_id : 3, host : "localhost:27003", arbiterOnly: true}
        ]
        }
        rs.initiate(config)
      8. Open mongo shell on localhost:27004 (NY1 mongod), and run:
      9. config = {
        _id : "NY1",
        members : [
        {_id : 1, host : "localhost:27004"},
        {_id : 2, host : "localhost:27005"},
        {_id : 3, host : "localhost:27006", arbiterOnly: true}
        ]
        }
        rs.initiate(config)
      10. Open mongo shell on localhost:27007 (LN1 config server mongod), and run:
      11. config = {
        _id : "conf",
        members : [
        {_id : 1, host : "localhost:27007"},
        {_id : 2, host : "localhost:27008"}
        ]
        }
        rs.initiate(config)
      12. Open shell on localhost:27009 (LN1 mongos) and run the following:
        1. sh.addShard("LN1/localhost:27001")
        2. sh.addShard("NY1/localhost:27004")
        3. sh.addShardTag("LN1", "LN1")
        4. sh.addShardTag("NY1", "NY1")
        5. sh.enableSharding("ddp")
      13. Shard collection datasources:
        1. sh.shardCollection("ddp.datasources", { "location" : 1, "shard" : 1 }

          )

        2. db.datasources.createIndex( { "location" : 1, "shard" : 1 }

          )

      14. Now insert tag:
        1. use config
        2. db.tags.insertOne({ "_id" :
          Unknown macro: { "ns" }

          , "ns" : "ddp.datasources", "min" :

          { "location" : "LN", "shard" : "LN1" }

          , "max" :

          { "location" : "LM", "shard" : "LN1" }

          , "tag" : "LN1" })

      15. Wait and will see that output from cmd window running LN1-config.bat fails with stack trace given in the description.

      I have repeated the steps using 3.2 without any issues.

      Can someone please advise? I can provide a Java program that does all the manual steps above and gets the same error if this is helpful.

      Show
      Ensure MongoDB 3.4 installed and available from "C:\Program Files\MongoDB\Server\3.4\bin Unzip attached LocalMongoDB.zip to e.g. C:\LocalFolder\temp\MongoDBUpgrade\3.4 In all bat files update MONGODB_VERSION to 3.4 Start all bat files (I used Robomongo 0.9.0-RC7 to run the following commands) Open mongo shell on localhost:27001 (LN1 mongod), and run: config = { _id : "LN1", members : [ {_id : 1, host : "localhost:27001"}, {_id : 2, host : "localhost:27002"}, {_id : 3, host : "localhost:27003", arbiterOnly: true} ] } rs.initiate(config) Open mongo shell on localhost:27004 (NY1 mongod), and run: config = { _id : "NY1", members : [ {_id : 1, host : "localhost:27004"}, {_id : 2, host : "localhost:27005"}, {_id : 3, host : "localhost:27006", arbiterOnly: true} ] } rs.initiate(config) Open mongo shell on localhost:27007 (LN1 config server mongod), and run: config = { _id : "conf", members : [ {_id : 1, host : "localhost:27007"}, {_id : 2, host : "localhost:27008"} ] } rs.initiate(config) Open shell on localhost:27009 (LN1 mongos) and run the following: sh.addShard("LN1/localhost:27001") sh.addShard("NY1/localhost:27004") sh.addShardTag("LN1", "LN1") sh.addShardTag("NY1", "NY1") sh.enableSharding("ddp") Shard collection datasources: sh.shardCollection("ddp.datasources", { "location" : 1, "shard" : 1 } ) db.datasources.createIndex( { "location" : 1, "shard" : 1 } ) Now insert tag: use config db.tags.insertOne({ "_id" : Unknown macro: { "ns" } , "ns" : "ddp.datasources", "min" : { "location" : "LN", "shard" : "LN1" } , "max" : { "location" : "LM", "shard" : "LN1" } , "tag" : "LN1" }) Wait and will see that output from cmd window running LN1-config.bat fails with stack trace given in the description. I have repeated the steps using 3.2 without any issues. Can someone please advise? I can provide a Java program that does all the manual steps above and gets the same error if this is helpful.
    • Sharding 2017-12-04

      I am writing a Java program to test various aspects of MongoDb so that we can upgrade in confidence in future. As part of this, I test aspects of sharding and shard tags.

      In MongoDB 3.2 (3.2.9) the sharding steps work correctly. In MongoDB 3.4.4 (also seen in 3.4.0), after inserting the following into tags collection on config database via mongos:

      db.tags.insertOne({ "_id" : { "ns" : "ddp.datasources", "min" :

      { "location" : "LN", "shard" : "LN1" }

      }, "ns" : "ddp.datasources", "min" :

      { "location" : "LN", "shard" : "LN1" }

      , "max" :

      { "location" : "LM", "shard" : "LN1" }

      , "tag" : "LN1" })

      I get the following form the config server (I have attached mdmp):

      2017-05-30T12:50:43.588+0100 I -        [Balancer] Invariant failure splitPoint.woCompare(it->second.splitKeys.back()) == 0 src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp 154
2017-05-30T12:50:43.589+0100 I -        [Balancer]

***aborting after invariant() failure


2017-05-30T12:50:43.692+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)                                   mongo::printStackTrace+0x43
2017-05-30T12:50:43.693+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)                          mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
2017-05-30T12:50:43.695+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)                          mongo::`anonymous namespace'::abruptQuit+0x85
2017-05-30T12:50:43.696+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   raise+0x1e8
2017-05-30T12:50:43.697+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   abort+0x31
2017-05-30T12:50:43.698+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\assert_util.cpp(154)                                          mongo::invariantFailed+0x19a
2017-05-30T12:50:43.700+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(154)        mongo::`anonymous namespace'::SplitCandidatesBuffer::addSplitPoint+0x32e
2017-05-30T12:50:43.701+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(408)        mongo::BalancerChunkSelectionPolicyImpl::_getSplitCandidatesForCollection+0x4c6
2017-05-30T12:50:43.703+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(217)        mongo::BalancerChunkSelectionPolicyImpl::selectChunksToSplit+0x3d0
2017-05-30T12:50:43.704+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer.cpp(546)                                    mongo::Balancer::_enforceTagRanges+0x65
2017-05-30T12:50:43.705+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer.cpp(387)                                    mongo::Balancer::_mainThread+0xa41
2017-05-30T12:50:43.707+0100 I CONTROL  [Balancer] mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81e7d4> >,std::default_delete<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81
e7d4> > > > >::_Run+0x77
2017-05-30T12:50:43.709+0100 I CONTROL  [Balancer] mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9
2017-05-30T12:50:43.710+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   crt_at_quick_exit+0x7d
2017-05-30T12:50:43.712+0100 I CONTROL  [Balancer] kernel32.dll                                                                                   BaseThreadInitThunk+0xd
2017-05-30T12:50:43.713+0100 F -        [Balancer] Got signal: 22 (SIGABRT).
2017-05-30T12:50:43.715+0100 I CONTROL  [Balancer] *** unhandled exception 0x0000000E at 0x000007FEFD3BA06D, terminating
2017-05-30T12:50:43.716+0100 I CONTROL  [Balancer] *** stack trace for unhandled exception:
2017-05-30T12:50:43.723+0100 I CONTROL  [Balancer] KERNELBASE.dll                                                                                   RaiseException+0x3d
2017-05-30T12:50:43.724+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)                          mongo::`anonymous namespace'::abruptQuit+0x9d
2017-05-30T12:50:43.726+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     raise+0x1e8
2017-05-30T12:50:43.727+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     abort+0x31
2017-05-30T12:50:43.728+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\util\assert_util.cpp(154)                                          mongo::invariantFailed+0x19a
2017-05-30T12:50:43.730+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(154)        mongo::`anonymous namespace'::SplitCandidatesBuffer::addSplitPoint+0x32e
2017-05-30T12:50:43.732+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(408)        mongo::BalancerChunkSelectionPolicyImpl::_getSplitCandidatesForCollection+0x4c6
2017-05-30T12:50:43.733+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(217)        mongo::BalancerChunkSelectionPolicyImpl::selectChunksToSplit+0x3d0
2017-05-30T12:50:43.734+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer.cpp(546)                                    mongo::Balancer::_enforceTagRanges+0x65
2017-05-30T12:50:43.736+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer.cpp(387)                                    mongo::Balancer::_mainThread+0xa41
2017-05-30T12:50:43.737+0100 I CONTROL  [Balancer] mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81e7d4> >,std::default_delete<std::tuple<<lambda_d43e9bbb1383790602f0e36fad
81e7d4> > > > >::_Run+0x77
2017-05-30T12:50:43.739+0100 I CONTROL  [Balancer] mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9
2017-05-30T12:50:43.741+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     crt_at_quick_exit+0x7d
2017-05-30T12:50:43.742+0100 I CONTROL  [Balancer] kernel32.dll                                                                                     BaseThreadInitThunk+0xd
2017-05-30T12:50:43.744+0100 I -        [Balancer]
2017-05-30T12:50:43.746+0100 I CONTROL  [Balancer] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-05-30T11-50-43.mdmp
2017-05-30T12:50:44.017+0100 I CONTROL  [Balancer] *** immediate exit due to unhandled exception

        1. 3.2017-05-30T11-50-43.mdmp
          2.29 MB
        2. LocalMongoDB.zip
          4 kB
        3. UpgradeTester.zip
          18 kB

            Assignee:
            dianna.hohensee@mongodb.com Dianna Hohensee (Inactive)
            Reporter:
            EvilChill Clive Hill
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: