Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29397

Invariant failure on config server when inserting tag into config.tags

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • 3.4.4, 3.5.8
    • 3.4.11, 3.6.1, 3.7.1
    • Sharding
    • None
    • Fully Compatible
    • ALL
    • v3.6, v3.4
    • Hide
      1. Ensure MongoDB 3.4 installed and available from "C:\Program Files\MongoDB\Server\3.4\bin
      2. Unzip attached LocalMongoDB.zip to e.g. C:\LocalFolder\temp\MongoDBUpgrade\3.4
      3. In all bat files update MONGODB_VERSION to 3.4
      4. Start all bat files
      5. (I used Robomongo 0.9.0-RC7 to run the following commands)
      6. Open mongo shell on localhost:27001 (LN1 mongod), and run:
      7. config = {
        _id : "LN1",
        members : [
        {_id : 1, host : "localhost:27001"},
        {_id : 2, host : "localhost:27002"},
        {_id : 3, host : "localhost:27003", arbiterOnly: true}
        ]
        }
        rs.initiate(config)
      8. Open mongo shell on localhost:27004 (NY1 mongod), and run:
      9. config = {
        _id : "NY1",
        members : [
        {_id : 1, host : "localhost:27004"},
        {_id : 2, host : "localhost:27005"},
        {_id : 3, host : "localhost:27006", arbiterOnly: true}
        ]
        }
        rs.initiate(config)
      10. Open mongo shell on localhost:27007 (LN1 config server mongod), and run:
      11. config = {
        _id : "conf",
        members : [
        {_id : 1, host : "localhost:27007"},
        {_id : 2, host : "localhost:27008"}
        ]
        }
        rs.initiate(config)
      12. Open shell on localhost:27009 (LN1 mongos) and run the following:
        1. sh.addShard("LN1/localhost:27001")
        2. sh.addShard("NY1/localhost:27004")
        3. sh.addShardTag("LN1", "LN1")
        4. sh.addShardTag("NY1", "NY1")
        5. sh.enableSharding("ddp")
      13. Shard collection datasources:
        1. sh.shardCollection("ddp.datasources", { "location" : 1, "shard" : 1 }

          )

        2. db.datasources.createIndex( { "location" : 1, "shard" : 1 }

          )

      14. Now insert tag:
        1. use config
        2. db.tags.insertOne({ "_id" :
          Unknown macro: { "ns" }

          , "ns" : "ddp.datasources", "min" :

          { "location" : "LN", "shard" : "LN1" }

          , "max" :

          { "location" : "LM", "shard" : "LN1" }

          , "tag" : "LN1" })

      15. Wait and will see that output from cmd window running LN1-config.bat fails with stack trace given in the description.

      I have repeated the steps using 3.2 without any issues.

      Can someone please advise? I can provide a Java program that does all the manual steps above and gets the same error if this is helpful.

      Show
      Ensure MongoDB 3.4 installed and available from "C:\Program Files\MongoDB\Server\3.4\bin Unzip attached LocalMongoDB.zip to e.g. C:\LocalFolder\temp\MongoDBUpgrade\3.4 In all bat files update MONGODB_VERSION to 3.4 Start all bat files (I used Robomongo 0.9.0-RC7 to run the following commands) Open mongo shell on localhost:27001 (LN1 mongod), and run: config = { _id : "LN1", members : [ {_id : 1, host : "localhost:27001"}, {_id : 2, host : "localhost:27002"}, {_id : 3, host : "localhost:27003", arbiterOnly: true} ] } rs.initiate(config) Open mongo shell on localhost:27004 (NY1 mongod), and run: config = { _id : "NY1", members : [ {_id : 1, host : "localhost:27004"}, {_id : 2, host : "localhost:27005"}, {_id : 3, host : "localhost:27006", arbiterOnly: true} ] } rs.initiate(config) Open mongo shell on localhost:27007 (LN1 config server mongod), and run: config = { _id : "conf", members : [ {_id : 1, host : "localhost:27007"}, {_id : 2, host : "localhost:27008"} ] } rs.initiate(config) Open shell on localhost:27009 (LN1 mongos) and run the following: sh.addShard("LN1/localhost:27001") sh.addShard("NY1/localhost:27004") sh.addShardTag("LN1", "LN1") sh.addShardTag("NY1", "NY1") sh.enableSharding("ddp") Shard collection datasources: sh.shardCollection("ddp.datasources", { "location" : 1, "shard" : 1 } ) db.datasources.createIndex( { "location" : 1, "shard" : 1 } ) Now insert tag: use config db.tags.insertOne({ "_id" : Unknown macro: { "ns" } , "ns" : "ddp.datasources", "min" : { "location" : "LN", "shard" : "LN1" } , "max" : { "location" : "LM", "shard" : "LN1" } , "tag" : "LN1" }) Wait and will see that output from cmd window running LN1-config.bat fails with stack trace given in the description. I have repeated the steps using 3.2 without any issues. Can someone please advise? I can provide a Java program that does all the manual steps above and gets the same error if this is helpful.
    • Sharding 2017-12-04

    Description

      I am writing a Java program to test various aspects of MongoDb so that we can upgrade in confidence in future. As part of this, I test aspects of sharding and shard tags.

      In MongoDB 3.2 (3.2.9) the sharding steps work correctly. In MongoDB 3.4.4 (also seen in 3.4.0), after inserting the following into tags collection on config database via mongos:

      db.tags.insertOne({ "_id" : { "ns" : "ddp.datasources", "min" :

      { "location" : "LN", "shard" : "LN1" }

      }, "ns" : "ddp.datasources", "min" :

      { "location" : "LN", "shard" : "LN1" }

      , "max" :

      { "location" : "LM", "shard" : "LN1" }

      , "tag" : "LN1" })

      I get the following form the config server (I have attached mdmp):

      2017-05-30T12:50:43.588+0100 I -        [Balancer] Invariant failure splitPoint.woCompare(it->second.splitKeys.back()) == 0 src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp 154
      		 
      2017-05-30T12:50:43.589+0100 I -        [Balancer]
      		 
       
      		 
      ***aborting after invariant() failure
      		 
       
      		 
       
      		 
      2017-05-30T12:50:43.692+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)                                   mongo::printStackTrace+0x43
      		 
      2017-05-30T12:50:43.693+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)                          mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
      		 
      2017-05-30T12:50:43.695+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)                          mongo::`anonymous namespace'::abruptQuit+0x85
      		 
      2017-05-30T12:50:43.696+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   raise+0x1e8
      		 
      2017-05-30T12:50:43.697+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   abort+0x31
      		 
      2017-05-30T12:50:43.698+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\util\assert_util.cpp(154)                                          mongo::invariantFailed+0x19a
      		 
      2017-05-30T12:50:43.700+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(154)        mongo::`anonymous namespace'::SplitCandidatesBuffer::addSplitPoint+0x32e
      		 
      2017-05-30T12:50:43.701+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(408)        mongo::BalancerChunkSelectionPolicyImpl::_getSplitCandidatesForCollection+0x4c6
      		 
      2017-05-30T12:50:43.703+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(217)        mongo::BalancerChunkSelectionPolicyImpl::selectChunksToSplit+0x3d0
      		 
      2017-05-30T12:50:43.704+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer.cpp(546)                                    mongo::Balancer::_enforceTagRanges+0x65
      		 
      2017-05-30T12:50:43.705+0100 I CONTROL  [Balancer] mongod.exe    ...\src\mongo\db\s\balancer\balancer.cpp(387)                                    mongo::Balancer::_mainThread+0xa41
      		 
      2017-05-30T12:50:43.707+0100 I CONTROL  [Balancer] mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81e7d4> >,std::default_delete<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81
      		 
      e7d4> > > > >::_Run+0x77
      		 
      2017-05-30T12:50:43.709+0100 I CONTROL  [Balancer] mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9
      		 
      2017-05-30T12:50:43.710+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                   crt_at_quick_exit+0x7d
      		 
      2017-05-30T12:50:43.712+0100 I CONTROL  [Balancer] kernel32.dll                                                                                   BaseThreadInitThunk+0xd
      		 
      2017-05-30T12:50:43.713+0100 F -        [Balancer] Got signal: 22 (SIGABRT).
      		 
      2017-05-30T12:50:43.715+0100 I CONTROL  [Balancer] *** unhandled exception 0x0000000E at 0x000007FEFD3BA06D, terminating
      		 
      2017-05-30T12:50:43.716+0100 I CONTROL  [Balancer] *** stack trace for unhandled exception:
      		 
      2017-05-30T12:50:43.723+0100 I CONTROL  [Balancer] KERNELBASE.dll                                                                                   RaiseException+0x3d
      		 
      2017-05-30T12:50:43.724+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)                          mongo::`anonymous namespace'::abruptQuit+0x9d
      		 
      2017-05-30T12:50:43.726+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     raise+0x1e8
      		 
      2017-05-30T12:50:43.727+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     abort+0x31
      		 
      2017-05-30T12:50:43.728+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\util\assert_util.cpp(154)                                          mongo::invariantFailed+0x19a
      		 
      2017-05-30T12:50:43.730+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(154)        mongo::`anonymous namespace'::SplitCandidatesBuffer::addSplitPoint+0x32e
      		 
      2017-05-30T12:50:43.732+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(408)        mongo::BalancerChunkSelectionPolicyImpl::_getSplitCandidatesForCollection+0x4c6
      		 
      2017-05-30T12:50:43.733+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer_chunk_selection_policy_impl.cpp(217)        mongo::BalancerChunkSelectionPolicyImpl::selectChunksToSplit+0x3d0
      		 
      2017-05-30T12:50:43.734+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer.cpp(546)                                    mongo::Balancer::_enforceTagRanges+0x65
      		 
      2017-05-30T12:50:43.736+0100 I CONTROL  [Balancer] mongod.exe      ...\src\mongo\db\s\balancer\balancer.cpp(387)                                    mongo::Balancer::_mainThread+0xa41
      		 
      2017-05-30T12:50:43.737+0100 I CONTROL  [Balancer] mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_d43e9bbb1383790602f0e36fad81e7d4> >,std::default_delete<std::tuple<<lambda_d43e9bbb1383790602f0e36fad
      		 
      81e7d4> > > > >::_Run+0x77
      		 
      2017-05-30T12:50:43.739+0100 I CONTROL  [Balancer] mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9
      		 
      2017-05-30T12:50:43.741+0100 I CONTROL  [Balancer] ucrtbase.DLL                                                                                     crt_at_quick_exit+0x7d
      		 
      2017-05-30T12:50:43.742+0100 I CONTROL  [Balancer] kernel32.dll                                                                                     BaseThreadInitThunk+0xd
      		 
      2017-05-30T12:50:43.744+0100 I -        [Balancer]
      		 
      2017-05-30T12:50:43.746+0100 I CONTROL  [Balancer] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-05-30T11-50-43.mdmp
      		 
      2017-05-30T12:50:44.017+0100 I CONTROL  [Balancer] *** immediate exit due to unhandled exception

      Attachments

        1. 3.2017-05-30T11-50-43.mdmp
          2.29 MB
        2. LocalMongoDB.zip
          4 kB
        3. UpgradeTester.zip
          18 kB

        Activity

          People

            dianna.hohensee@mongodb.com Dianna Hohensee
            EvilChill Clive Hill
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: