Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7857

moveChunk fails after removing an unrelated shard (replset shards)

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 2.2.1
    • Sharding
    • None
    • Sharding
    • ALL

    Description

      I've created a 4-shard cluster, each consisting of a replica set (3 nodes).
      After enableSharding and shardCollection, I remove shard02. I then try to moveChunk the only existing chunk from shard01 to shard04. An error is thrown and the move fails:

      {
      	"errmsg" : "exception: No replica set monitor active and no cached seed found for set: shard02",
      	"code" : 16340,
      	"ok" : 0
      }

      After quitting the mongo shell and restarting it, the moveChunk command works fine. Note: Nothing else (mongod, mongos) was restarted, only the shell reconnected.

      Below is the history to reproduce the issue. It was tested on 2.2.0 and 2.2.1. Both versions are affected.

      tr@capslock:~/Documents/code/mtools$ mongo --port 27030
      MongoDB shell version: 2.2.1
      connecting to: 127.0.0.1:27030/test
      mongos> sh.status()
      --- Sharding Status --- 
        sharding version: { "_id" : 1, "version" : 3 }
        shards:
      	{  "_id" : "shard01",  "host" : "shard01/capslock.local:27017,capslock.local:27018,capslock.local:27019" }
      	{  "_id" : "shard02",  "host" : "shard02/capslock.local:27020,capslock.local:27021,capslock.local:27022" }
      	{  "_id" : "shard03",  "host" : "shard03/capslock.local:27023,capslock.local:27024,capslock.local:27025" }
      	{  "_id" : "shard04",  "host" : "shard04/capslock.local:27026,capslock.local:27027,capslock.local:27028" }
        databases:
      	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
       
      mongos> sh.enableSharding('test')
      { "ok" : 1 }
      mongos> sh.shardCollection('test.docs', {shardkey: 1})
      { "collectionsharded" : "test.docs", "ok" : 1 }
      mongos> sh.status()
      --- Sharding Status --- 
        sharding version: { "_id" : 1, "version" : 3 }
        shards:
      	{  "_id" : "shard01",  "host" : "shard01/capslock.local:27017,capslock.local:27018,capslock.local:27019" }
      	{  "_id" : "shard02",  "host" : "shard02/capslock.local:27020,capslock.local:27021,capslock.local:27022" }
      	{  "_id" : "shard03",  "host" : "shard03/capslock.local:27023,capslock.local:27024,capslock.local:27025" }
      	{  "_id" : "shard04",  "host" : "shard04/capslock.local:27026,capslock.local:27027,capslock.local:27028" }
        databases:
      	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
      	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard01" }
      		test.docs chunks:
      				shard01	1
      			{ "shardkey" : { $minKey : 1 } } -->> { "shardkey" : { $maxKey : 1 } } on : shard01 Timestamp(1000, 0) 
       
      mongos> db.adminCommand({removeShard: 'shard02'})
      {
      	"msg" : "draining started successfully",
      	"state" : "started",
      	"shard" : "shard02",
      	"ok" : 1
      }
      mongos> db.adminCommand({removeShard: 'shard02'})
      {
      	"msg" : "removeshard completed successfully",
      	"state" : "completed",
      	"shard" : "shard02",
      	"ok" : 1
      }
      mongos> db.adminCommand({removeShard: 'shard02'})
      {
      	"errmsg" : "exception: can't find shard for: shard02",
      	"code" : 13129,
      	"ok" : 0
      }
      mongos> sh.status()
      --- Sharding Status --- 
        sharding version: { "_id" : 1, "version" : 3 }
        shards:
      	{  "_id" : "shard01",  "host" : "shard01/capslock.local:27017,capslock.local:27018,capslock.local:27019" }
      	{  "_id" : "shard03",  "host" : "shard03/capslock.local:27023,capslock.local:27024,capslock.local:27025" }
      	{  "_id" : "shard04",  "host" : "shard04/capslock.local:27026,capslock.local:27027,capslock.local:27028" }
        databases:
      	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
      	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard01" }
      		test.docs chunks:
      				shard01	1
      			{ "shardkey" : { $minKey : 1 } } -->> { "shardkey" : { $maxKey : 1 } } on : shard01 Timestamp(1000, 0) 
       
      mongos> sh.moveChunk('test.docs', {shardkey: 0}, 'shard04')
      {
      	"errmsg" : "exception: No replica set monitor active and no cached seed found for set: shard02",
      	"code" : 16340,
      	"ok" : 0
      }
      mongos> ^C
      bye
      tr@capslock:~/Documents/code/mtools$ mongo --port 27030
      MongoDB shell version: 2.2.1
      connecting to: 127.0.0.1:27030/test
      mongos> sh.moveChunk('test.docs', {shardkey: 0}, 'shard04')
      { "millis" : 2236, "ok" : 1 }
      mongos> sh.status()
      --- Sharding Status --- 
        sharding version: { "_id" : 1, "version" : 3 }
        shards:
      	{  "_id" : "shard01",  "host" : "shard01/capslock.local:27017,capslock.local:27018,capslock.local:27019" }
      	{  "_id" : "shard03",  "host" : "shard03/capslock.local:27023,capslock.local:27024,capslock.local:27025" }
      	{  "_id" : "shard04",  "host" : "shard04/capslock.local:27026,capslock.local:27027,capslock.local:27028" }
        databases:
      	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
      	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard01" }
      		test.docs chunks:
      				shard04	1
      			{ "shardkey" : { $minKey : 1 } } -->> { "shardkey" : { $maxKey : 1 } } on : shard04 Timestamp(2000, 0) 
       
      mongos> 

      Attachments

        1. bug_repro.js
          0.7 kB
        2. logfiles.zip
          37 kB

        Activity

          People

            backlog-server-sharding [DO NOT USE] Backlog - Sharding Team
            thomas.rueckstiess@mongodb.com Thomas Rueckstiess
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: