Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29397

Invariant failure on config server when inserting tag into config.tags

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.4.4, 3.5.8
    • Fix Version/s: 3.4.11, 3.6.1, 3.7.1
    • Component/s: Sharding
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v3.6, v3.4
    • Steps To Reproduce:
      Hide
      1. Ensure MongoDB 3.4 installed and available from "C:\Program Files\MongoDB\Server\3.4\bin
      2. Unzip attached LocalMongoDB.zip to e.g. C:\LocalFolder\temp\MongoDBUpgrade\3.4
      3. In all bat files update MONGODB_VERSION to 3.4
      4. Start all bat files
      5. (I used Robomongo 0.9.0-RC7 to run the following commands)
      6. Open mongo shell on localhost:27001 (LN1 mongod), and run:
      7. config = {
        _id : "LN1",
        members : [
        {_id : 1, host : "localhost:27001"},
        {_id : 2, host : "localhost:27002"},
        {_id : 3, host : "localhost:27003", arbiterOnly: true}
        ]
        }
        rs.initiate(config)
      8. Open mongo shell on localhost:27004 (NY1 mongod), and run:
      9. config = {
        _id : "NY1",
        members : [
        {_id : 1, host : "localhost:27004"},
        {_id : 2, host : "localhost:27005"},
        {_id : 3, host : "localhost:27006", arbiterOnly: true}
        ]
        }
        rs.initiate(config)
      10. Open mongo shell on localhost:27007 (LN1 config server mongod), and run:
      11. config = {
        _id : "conf",
        members : [
        {_id : 1, host : "localhost:27007"},
        {_id : 2, host : "localhost:27008"}
        ]
        }
        rs.initiate(config)
      12. Open shell on localhost:27009 (LN1 mongos) and run the following:
        1. sh.addShard("LN1/localhost:27001")
        2. sh.addShard("NY1/localhost:27004")
        3. sh.addShardTag("LN1", "LN1")
        4. sh.addShardTag("NY1", "NY1")
        5. sh.enableSharding("ddp")
      13. Shard collection datasources:
        1. sh.shardCollection("ddp.datasources", { "location" : 1, "shard" : 1 }

          )

        2. db.datasources.createIndex( { "location" : 1, "shard" : 1 }

          )

      14. Now insert tag:
        1. use config
        2. db.tags.insertOne({ "_id" :
          Unknown macro: { "ns" }

          , "ns" : "ddp.datasources", "min" :

          { "location" : "LN", "shard" : "LN1" }

          , "max" :

          { "location" : "LM", "shard" : "LN1" }

          , "tag" : "LN1" })

      15. Wait and will see that output from cmd window running LN1-config.bat fails with stack trace given in the description.

      I have repeated the steps using 3.2 without any issues.

      Can someone please advise? I can provide a Java program that does all the manual steps above and gets the same error if this is helpful.

      Show
      Ensure MongoDB 3.4 installed and available from "C:\Program Files\MongoDB\Server\3.4\bin Unzip attached LocalMongoDB.zip to e.g. C:\LocalFolder\temp\MongoDBUpgrade\3.4 In all bat files update MONGODB_VERSION to 3.4 Start all bat files (I used Robomongo 0.9.0-RC7 to run the following commands) Open mongo shell on localhost:27001 (LN1 mongod), and run: config = { _id : "LN1", members : [ {_id : 1, host : "localhost:27001"}, {_id : 2, host : "localhost:27002"}, {_id : 3, host : "localhost:27003", arbiterOnly: true} ] } rs.initiate(config) Open mongo shell on localhost:27004 (NY1 mongod), and run: config = { _id : "NY1", members : [ {_id : 1, host : "localhost:27004"}, {_id : 2, host : "localhost:27005"}, {_id : 3, host : "localhost:27006", arbiterOnly: true} ] } rs.initiate(config) Open mongo shell on localhost:27007 (LN1 config server mongod), and run: config = { _id : "conf", members : [ {_id : 1, host : "localhost:27007"}, {_id : 2, host : "localhost:27008"} ] } rs.initiate(config) Open shell on localhost:27009 (LN1 mongos) and run the following: sh.addShard("LN1/localhost:27001") sh.addShard("NY1/localhost:27004") sh.addShardTag("LN1", "LN1") sh.addShardTag("NY1", "NY1") sh.enableSharding("ddp") Shard collection datasources: sh.shardCollection("ddp.datasources", { "location" : 1, "shard" : 1 } ) db.datasources.createIndex( { "location" : 1, "shard" : 1 } ) Now insert tag: use config db.tags.insertOne({ "_id" : Unknown macro: { "ns" } , "ns" : "ddp.datasources", "min" : { "location" : "LN", "shard" : "LN1" } , "max" : { "location" : "LM", "shard" : "LN1" } , "tag" : "LN1" }) Wait and will see that output from cmd window running LN1-config.bat fails with stack trace given in the description. I have repeated the steps using 3.2 without any issues. Can someone please advise? I can provide a Java program that does all the manual steps above and gets the same error if this is helpful.
    • Sprint:
      Sharding 2017-12-04

      Description

      I am writing a Java program to test various aspects of MongoDb so that we can upgrade in confidence in future. As part of this, I test aspects of sharding and shard tags.

      In MongoDB 3.2 (3.2.9) the sharding steps work correctly. In MongoDB 3.4.4 (also seen in 3.4.0), after inserting the following into tags collection on config database via mongos:

      db.tags.insertOne({ "_id" : { "ns" : "ddp.datasources", "min" :

      { "location" : "LN", "shard" : "LN1" }

      }, "ns" : "ddp.datasources", "min" :

      { "location" : "LN", "shard" : "LN1" }

      , "max" :

      { "location" : "LM", "shard" : "LN1" }

      , "tag" : "LN1" })

      I get the following form the config server (I have attached mdmp):

      2017-05-30T12:50:43.588+0100 I -        [Balancer] Invariant failure splitPoint.woCompare(it->second.splitKeys.back()) == 0 src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp 154
      2017-05-30T12:50:43.589+0100 I -        [Balancer]
       
      ***aborting after invariant() failure
       
       
      2017-05-30T12:50:43.692+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)                                   mongo::printStackTrace+0x43
      2017-05-30T12:50:43.693+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)                          mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
      2017-05-30T12:50:43.695+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)                          mongo::`anonymous namespace'::abruptQuit+0x85
      2017-05-30T12:50:43.696+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   raise+0x1e8
      2017-05-30T12:50:43.697+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   abort+0x31
      2017-05-30T12:50:43.698+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\assert_util.cpp(154)                                          mongo::invariantFailed+0x19a
      2017-05-30T12:50:43.700+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(154)        mongo::`anonymous namespace'::SplitCandidatesBuffer::addSplitPoint+0x32e
      2017-05-30T12:50:43.701+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(408)        mongo::BalancerChunkSelectionPolicyImpl::_getSplitCandidatesForCollection+0x4c6
      2017-05-30T12:50:43.703+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(217)        mongo::BalancerChunkSelectionPolicyImpl::selectChunksToSplit+0x3d0
      2017-05-30T12:50:43.704+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer.cpp(546)                                    mongo::Balancer::_enforceTagRanges+0x65
      2017-05-30T12:50:43.705+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer.cpp(387)                                    mongo::Balancer::_mainThread+0xa41
      2017-05-30T12:50:43.707+0100 I CONTROL  [Balancer] mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81e7d4> >,std::default_delete<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81
      e7d4> > > > >::_Run+0x77
      2017-05-30T12:50:43.709+0100 I CONTROL  [Balancer] mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9
      2017-05-30T12:50:43.710+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   crt_at_quick_exit+0x7d
      2017-05-30T12:50:43.712+0100 I CONTROL  [Balancer] kernel32.dll                                                                                   BaseThreadInitThunk+0xd
      2017-05-30T12:50:43.713+0100 F -        [Balancer] Got signal: 22 (SIGABRT).
      2017-05-30T12:50:43.715+0100 I CONTROL  [Balancer] *** unhandled exception 0x0000000E at 0x000007FEFD3BA06D, terminating
      2017-05-30T12:50:43.716+0100 I CONTROL  [Balancer] *** stack trace for unhandled exception:
      2017-05-30T12:50:43.723+0100 I CONTROL  [Balancer] KERNELBASE.dll                                                                                   RaiseException+0x3d
      2017-05-30T12:50:43.724+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)                          mongo::`anonymous namespace'::abruptQuit+0x9d
      2017-05-30T12:50:43.726+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     raise+0x1e8
      2017-05-30T12:50:43.727+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     abort+0x31
      2017-05-30T12:50:43.728+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\util\assert_util.cpp(154)                                          mongo::invariantFailed+0x19a
      2017-05-30T12:50:43.730+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(154)        mongo::`anonymous namespace'::SplitCandidatesBuffer::addSplitPoint+0x32e
      2017-05-30T12:50:43.732+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(408)        mongo::BalancerChunkSelectionPolicyImpl::_getSplitCandidatesForCollection+0x4c6
      2017-05-30T12:50:43.733+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(217)        mongo::BalancerChunkSelectionPolicyImpl::selectChunksToSplit+0x3d0
      2017-05-30T12:50:43.734+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer.cpp(546)                                    mongo::Balancer::_enforceTagRanges+0x65
      2017-05-30T12:50:43.736+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer.cpp(387)                                    mongo::Balancer::_mainThread+0xa41
      2017-05-30T12:50:43.737+0100 I CONTROL  [Balancer] mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81e7d4> >,std::default_delete<std::tuple<<lambda_d43e9bbb1383790602f0e36fad
      81e7d4> > > > >::_Run+0x77
      2017-05-30T12:50:43.739+0100 I CONTROL  [Balancer] mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9
      2017-05-30T12:50:43.741+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     crt_at_quick_exit+0x7d
      2017-05-30T12:50:43.742+0100 I CONTROL  [Balancer] kernel32.dll                                                                                     BaseThreadInitThunk+0xd
      2017-05-30T12:50:43.744+0100 I -        [Balancer]
      2017-05-30T12:50:43.746+0100 I CONTROL  [Balancer] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-05-30T11-50-43.mdmp
      2017-05-30T12:50:44.017+0100 I CONTROL  [Balancer] *** immediate exit due to unhandled exception
      

        Attachments

        1. 3.2017-05-30T11-50-43.mdmp
          2.29 MB
        2. LocalMongoDB.zip
          4 kB
        3. UpgradeTester.zip
          18 kB

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: