Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34672

Unable to add shard on 3.7.5 sharded cluster with mmapv1 shard

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide
      1. Start up a 3-node shard on mmapv1
      2. Initiate repl set:

        > rs.initiate({_id: "a", "members" : [ {"_id" : 0, host : "louisamac:9001"}, {_id: 1, host: "louisamac:9002"}, {_id: 2, host: "louisamac:9003"} ] })
        {
        	"ok" : 1,
        	"operationTime" : Timestamp(1524673861, 1),
        	"$clusterTime" : {
        		"clusterTime" : Timestamp(1524673861, 1),
        		"signature" : {
        			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
        			"keyId" : NumberLong(0)
        		}
        	}
        }
        

      3. Start up a 3-node csrs
      4. Initiate csrs

        > rs.initiate({configsvr: true, protocolVersion:1, _id: "csrs", "members" : [ {"_id" : 0, host : "louisamac:9007"}, {_id: 1, host: "louisamac:9008"}, {_id: 2, host: "louisamac:9009"} ] })
        {
        	"ok" : 1,
        	"operationTime" : Timestamp(1524674021, 1),
        	"$gleStats" : {
        		"lastOpTime" : Timestamp(1524674021, 1),
        		"electionId" : ObjectId("000000000000000000000000")
        	},
        	"$clusterTime" : {
        		"clusterTime" : Timestamp(1524674021, 1),
        		"signature" : {
        			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
        			"keyId" : NumberLong(0)
        		}
        	},
        	"lastCommittedOpTime" : Timestamp(0, 0)
        }
        

      5. Start up mongos
      6. Call {{db.runCommand( {"setFeatureCompatibilityVersion" : "4.0"}

        )}} on mongos

      7. Attempt to add a shard

        mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"})
        {
        	"ok" : 0,
        	"errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out",
        	"code" : 96,
        	"codeName" : "OperationFailed",
        	"$clusterTime" : {
        		"clusterTime" : Timestamp(1524674234, 1),
        		"signature" : {
        			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
        			"keyId" : NumberLong(0)
        		}
        	},
        	"operationTime" : Timestamp(1524674234, 1)
        }
        

      Show
      Start up a 3-node shard on mmapv1 Initiate repl set: > rs.initiate({_id: "a", "members" : [ {"_id" : 0, host : "louisamac:9001"}, {_id: 1, host: "louisamac:9002"}, {_id: 2, host: "louisamac:9003"} ] }) { "ok" : 1, "operationTime" : Timestamp(1524673861, 1), "$clusterTime" : { "clusterTime" : Timestamp(1524673861, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } Start up a 3-node csrs Initiate csrs > rs.initiate({configsvr: true, protocolVersion:1, _id: "csrs", "members" : [ {"_id" : 0, host : "louisamac:9007"}, {_id: 1, host: "louisamac:9008"}, {_id: 2, host: "louisamac:9009"} ] }) { "ok" : 1, "operationTime" : Timestamp(1524674021, 1), "$gleStats" : { "lastOpTime" : Timestamp(1524674021, 1), "electionId" : ObjectId("000000000000000000000000") }, "$clusterTime" : { "clusterTime" : Timestamp(1524674021, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "lastCommittedOpTime" : Timestamp(0, 0) } Start up mongos Call {{db.runCommand( {"setFeatureCompatibilityVersion" : "4.0"} )}} on mongos Attempt to add a shard mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"}) { "ok" : 0, "errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out", "code" : 96, "codeName" : "OperationFailed", "$clusterTime" : { "clusterTime" : Timestamp(1524674234, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1524674234, 1) }
    • Linked BF Score:
      0

      Description

      If you bring up a sharded cluster on 3.7.5 with a mmapv1 multi-node shard, you get a timeout error from trying to add the shard on the mongos:

      mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"})
      {
      	"ok" : 0,
      	"errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out",
      	"code" : 96,
      	"codeName" : "OperationFailed",
      	"$clusterTime" : {
      		"clusterTime" : Timestamp(1524674234, 1),
      		"signature" : {
      			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
      			"keyId" : NumberLong(0)
      		}
      	},
      	"operationTime" : Timestamp(1524674234, 1)
      }
      

      Note: I tried this on WT and it worked fine. Assigning to storage for that reason, but may need to be reassigned.
      Second note: Also did not have this problem with a 1-node shard.

      Attached config files and log files.

        Attachments

        1. cfg1.json
          0.4 kB
        2. cfg2.json
          0.4 kB
        3. cfg3.json
          0.4 kB
        4. csrs_primary.log
          41 kB
        5. csrs_secondary.log
          29 kB
        6. mongos.json
          0.2 kB
        7. mongos.log
          7 kB
        8. shard_primary.log
          37 kB
        9. shard_secondary.log
          37 kB
        10. shard1.json
          0.4 kB
        11. shard2.json
          0.4 kB
        12. shard3.json
          0.4 kB

          Issue Links

            Activity

              People

              Assignee:
              esha.maharishi Esha Maharishi
              Reporter:
              louisa.berger Louisa Berger
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: