[SERVER-6546] All of config servers go down. Just restart and not recover from backup, config server's metadata of sharding chunks information can resync up to date as APP request is still writing to mongod Created: 21/Jul/12  Updated: 15/Aug/12  Resolved: 22/Jul/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.6
Fix Version/s: None

Type: Question Priority: Critical - P2
Reporter: Jianfeng Xu Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

There is a MongoDB Sharding cluster:

3 mongos nodes: A/B/C, 3 config server nodes: X/Y/Z, 2 shards with replica sets: s1/s2.

— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "s1", "host" : "shard1/host1:27032,host2:27032" } { "_id" : "s2", "host" : "shard2/host3:27032,host4:27032" }

databases:

{ "_id" : "test", "partitioned" : false, "primary" : "s1" } { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d", "partitioned" : true, "primary" : "s1" }

d.t chunks:
s1 139
s2 137
too many chunks to print, use verbose if you want to force print

From document, if any of the config servers is down (X or Y or Z), the cluster's metadata goes read-only. However, even in such a failure state, the MongoDB cluster can still be read from and written to. Shutdown all of config server nodes as one by one, app request is still writing to mongod from replica sets.

But because of app writing, lots of chunks are created. I am not sure that they are on s1 or s2? Can split or move chunks to banlance between s1/s2? Can the config servers metadata keep up to date?

After all of config servers go down. To solve, do just restart all of config servers or recover config server node from old backup? If restart all of config servers without recover from backup, is the cluster available and is need to reconfig the cluster? The config server's metadata will resync up to date?

Thanks



 Comments   
Comment by Jianfeng Xu [ 22/Jul/12 ]

Can u explain more details when all config servers are down. Thanks!

Comment by Eliot Horowitz (Inactive) [ 22/Jul/12 ]

If one or two is down, no problems.
If all 3 are down, it could be a problem.

Comment by Jianfeng Xu [ 22/Jul/12 ]

Thanks for your quick reply.

Whatever one or all of config servers go down, the cluster is available and no need to reconfig?

Comment by Eliot Horowitz (Inactive) [ 22/Jul/12 ]

If all the config servers go down, then when you bring them back up they should be consistent as no writes are allowed in the interim.
When a mongos restarts it validates that they are in sync.
No splits or migrates happen if one is down, so nothing to resync.

Generated at Thu Feb 08 03:12:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.