Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-95717

Make removeShardFromZone resilient to config server replica set transitions

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 5.0.0, 6.0.0, 7.0.0, 8.1.0-rc0, 8.0.0
    • Component/s: None
    • None
    • Catalog and Routing
    • ALL
    • CAR Team 2024-10-28

      An unlikely sequence of config server stepdowns and concurrent operations during the execution of the removeShardFromZone command can create an inconsistency by leaving a zone without any shards associated (usually, a ZoneStillInUse error would be thrown).

      Reproduction steps:

      1. Add a fail point hangRemoveShardFromZoneBeforeCommit in the following line of code.
      2. Start a sharded cluster with at least two nodes in the config server replica set and --setParameter enableTestCommands=1.
      3. On the mongos router, add a shard to a zone: sh.addShardToZone("shard01", "NYC")
      4. On the CSRS primary, enable the fail point: db.adminCommand({configureFailPoint: 'hangRemoveShardFromZoneBeforeCommit', mode: "alwaysOn"})
      5. On the mongos router, remove the shard from the zone: sh.removeShardFromZone("shard01", "NYC"). The command will hang.
      6. Step down the CSRS primary: rs.stepDown()
      7. From the mongos router (on another shell), associate the zone to a key range: sh.updateZoneKeyRange("records.users", { zipcode: "10001" }, { zipcode: "10281" }, "NYC")
      8. Step down CSRS primary(es) until the original config server primary gets elected primary again: rs.stepDown()
      9. From the CSRS primary, disable the fail point: db.adminCommand({configureFailPoint: 'hangRemoveShardFromZoneBeforeCommit', mode: "off"})

       

      At this point, the hung removeShardFromZone command will complete successfully and the zone will be orphaned.

      > db.getSiblingDB("config").tags.find()
      { "_id" : ObjectId("67082f58eee0b05d5418e785"), "ns" : "records.users", "min" : { "zipcode" : "10001" }, "max" : { "zipcode" : "10281" }, "tag" : "NYC" }
      > db.getSiblingDB("config").shards.find()
      { "_id" : "shard01", "host" : "shard01/localhost:27018", "state" : 1, "topologyTime" : Timestamp(1728589374, 13), "replSetConfigVersion" : NumberLong(-1), "tags" : [ ] }
      

       

      A fix possibility is adding a call to opCtx->setAlwaysInterruptAtStepDownOrUp_UNSAFE(); at the beginning of the command.

            Assignee:
            joan.mico@mongodb.com Joan Bruguera Micó (Inactive)
            Reporter:
            joan.mico@mongodb.com Joan Bruguera Micó (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: