[SERVER-14688] Databases are created on a shard node even if it's in draining state Created: 25/Jul/14  Updated: 06/Dec/22  Resolved: 07/Nov/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.6.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Done Votes: 1
Labels: 28qa, pm-1051-legacy-tickets
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-14916 Draining shard shouldn't receive new ... Closed
Assigned Teams:
Sharding
Operating System: ALL
Steps To Reproduce:
  • Create a 3 shard set, sh1, sh2, sh3
  • Start a mongos & config server
  • Connect to mongos
  • Create 4 DBs and insert data. Note that the DBs are created in a round-robin fashion across the shard nodes, i.e.,
    • sh1 - db1& db4
    • sh2 - db2
    • sh3 - db3
  • Remove sh2: db.adminCommand( { removeShard: "sh2" } )
    • The state will go to "draining", but will not complete because the primary database(s) must be moved (movePrimary) or dropped
  • Create another DB and insert data
  • Run the db.adminCommand( { removeShard: "sh2" } ) command
    • The new DB is now created on this shard node
    • Note same will happen if using mongorestore
Participants:

 Description   

A node in a shard set can have a DB created on it, even if it's been set to a draining state (from removeShard).



 Comments   
Comment by Sheeri Cabral (Inactive) [ 07/Nov/19 ]

To see if this is still an issue on recent versions, I created a sharded cluster (2 shards), made 4 databases:

 

MongoDB Enterprise mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
  	"_id" : 1,
  	"minCompatibleVersion" : 5,
  	"currentVersion" : 6,
  	"clusterId" : ObjectId("5dc481fcf8e17bc595a56e87")
  }
  shards:
        {  "_id" : "rs0""host" : "rs0/192.168.103.100:27001,192.168.103.100:27002,192.168.103.100:27003""state" : 1 }
        {  "_id" : "rs1""host" : "rs1/192.168.103.100:27004""state" : 1 }
  active mongoses:
        "3.6.15" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "config""primary" : "config""partitioned" : true }
        {  "_id" : "shardTest0""primary" : "rs1""partitioned" : true }
                shardTest0.products
                        shard key: { "sku" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                rs1	1
                        { "sku" : { "$minKey" : 1 } } -->> { "sku" : { "$maxKey" : 1 } } on : rs1 Timestamp(1, 0) 
        {  "_id" : "shardTest1""primary" : "rs0""partitioned" : true }
                shardTest1.products
                        shard key: { "sku" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                rs0	1
                        { "sku" : { "$minKey" : 1 } } -->> { "sku" : { "$maxKey" : 1 } } on : rs0 Timestamp(1, 0) 
        {  "_id" : "shardTest2""primary" : "rs0""partitioned" : true }
                shardTest2.products
                        shard key: { "sku" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                rs0	1
                        { "sku" : { "$minKey" : 1 } } -->> { "sku" : { "$maxKey" : 1 } } on : rs0 Timestamp(1, 0) 
        {  "_id" : "shardTest3""primary" : "rs1""partitioned" : true }
                shardTest3.products
                        shard key: { "sku" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                rs1	1
                        { "sku" : { "$minKey" : 1 } } -->> { "sku" : { "$maxKey" : 1 } } on : rs1 Timestamp(1, 0) 

And got the expected error "you need to drop or movePrimary" for the databases:

 

MongoDB Enterprise mongos> db.adminCommand( { removeShard: "rs1" } )
{
	"msg" : "draining started successfully",
	"state" : "started",
	"shard" : "rs1",
	"note" : "you need to drop or movePrimary these databases",
	"dbsToMove" : [
		"shardTest0",
		"shardTest3"
	],
	"ok" : 1,
	"operationTime" : Timestamp(1573159656, 3),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1573159656, 3),
		"signature" : {
			"hash" : BinData(0,"3OMeRkCfZkaGl3UlrF/cfDH+siQ="),
			"keyId" : NumberLong("6756668260294328346")
		}
	}
}

 

Generated at Thu Feb 08 03:35:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.