[SERVER-1922] need to restart mongod to clear stale shard mete-data Created: 11/Oct/10  Updated: 12/Jul/16  Resolved: 11/Oct/10

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.7.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alvin Richards (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Problem:
After dropping a shard and re-creating a shard with the same name, the following was seen in the logs

Sun Oct 10 20:37:43 [conn173932] DBException in process: setShardVersion failed!

{ "errmsg" : "exception: gotShardHost different than what i had before before [set3/rs3a:27018] got [set3/rs3a:27018,rs3b:27018] ", "code" : 13299, "ok" : 0 }

Reproduce:

  • turn the balancer off
    db.settings.update( { _id : "balancer" }

    ,

    Unknown macro: { $set }

    , true )

  • create a 2 member replset, "foo"
  • add the shard with a single member
    db.runCommand( { addshard : "foo/node1", maxSize: 409600, name : "shard1" }

    );

  • remove the shard
    db.runCommand( { removeshard : "foo/node1" }

    );

  • add the shard again, but with both nodes
    db.runCommand( { addshard : "foo/node1,node2", maxSize: 409600, name : "shard1" }

    );

Workaround:
Since the members of the shard were part of a replset, the following was performed to clear the error

  • find the current master (through looking at the rs.status()
  • for the current master, do a rs.stepDown()
  • restart that mongod process
  • repeat until all members of the replset had been re-started

Business Case:

  • Reliability
    Need to deal with stale meta-data more gracefully and automatically


 Comments   
Comment by Azat Khuzhin [ 20/Jul/12 ]

This is not helps for me, because before I have 2 single shards, after I have 2 shards with replSet for each

For how I migrate from single to replSet in sharding see #SERVER-6424

Comment by Eliot Horowitz (Inactive) [ 11/Oct/10 ]

http://github.com/mongodb/mongo/commit/f6141d9bac74be42762c234769b44a661cc1f77c

Generated at Thu Feb 08 02:58:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.