Unable to add shard on 3.7.5 sharded cluster with mmapv1 shard

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • None
    • ALL
    • Hide
      1. Start up a 3-node shard on mmapv1
      2. Initiate repl set: 
        > rs.initiate({_id: "a", "members" : [ {"_id" : 0, host : "louisamac:9001"}, {_id: 1, host: "louisamac:9002"}, {_id: 2, host: "louisamac:9003"} ] })
{
	"ok" : 1,
	"operationTime" : Timestamp(1524673861, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524673861, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}
      3. Start up a 3-node csrs
      4. Initiate csrs 
        > rs.initiate({configsvr: true, protocolVersion:1, _id: "csrs", "members" : [ {"_id" : 0, host : "louisamac:9007"}, {_id: 1, host: "louisamac:9008"}, {_id: 2, host: "louisamac:9009"} ] })
{
	"ok" : 1,
	"operationTime" : Timestamp(1524674021, 1),
	"$gleStats" : {
		"lastOpTime" : Timestamp(1524674021, 1),
		"electionId" : ObjectId("000000000000000000000000")
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524674021, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"lastCommittedOpTime" : Timestamp(0, 0)
}
      5. Start up mongos
      6. Call {{db.runCommand( {"setFeatureCompatibilityVersion" : "4.0"}

        )}} on mongos

      7. Attempt to add a shard 
        mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"})
{
	"ok" : 0,
	"errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out",
	"code" : 96,
	"codeName" : "OperationFailed",
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524674234, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"operationTime" : Timestamp(1524674234, 1)
}
      Show
      Start up a 3-node shard on mmapv1 Initiate repl set: > rs.initiate({_id: "a" , "members" : [ { "_id" : 0, host : "louisamac:9001" }, {_id: 1, host: "louisamac:9002" }, {_id: 2, host: "louisamac:9003" } ] }) { "ok" : 1, "operationTime" : Timestamp(1524673861, 1), "$clusterTime" : { "clusterTime" : Timestamp(1524673861, 1), "signature" : { "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA=" ), "keyId" : NumberLong(0) } } } Start up a 3-node csrs Initiate csrs > rs.initiate({configsvr: true , protocolVersion:1, _id: "csrs" , "members" : [ { "_id" : 0, host : "louisamac:9007" }, {_id: 1, host: "louisamac:9008" }, {_id: 2, host: "louisamac:9009" } ] }) { "ok" : 1, "operationTime" : Timestamp(1524674021, 1), "$gleStats" : { "lastOpTime" : Timestamp(1524674021, 1), "electionId" : ObjectId( "000000000000000000000000" ) }, "$clusterTime" : { "clusterTime" : Timestamp(1524674021, 1), "signature" : { "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA=" ), "keyId" : NumberLong(0) } }, "lastCommittedOpTime" : Timestamp(0, 0) } Start up mongos Call {{db.runCommand( {"setFeatureCompatibilityVersion" : "4.0"} )}} on mongos Attempt to add a shard mongos> db.runCommand({ "addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003" , name: "shard0" }) { "ok" : 0, "errmsg" : "failed to run command { setFeatureCompatibilityVersion: \" 4.0\ " } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out" , "code" : 96, "codeName" : "OperationFailed" , "$clusterTime" : { "clusterTime" : Timestamp(1524674234, 1), "signature" : { "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA=" ), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1524674234, 1) }
    • 0
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      If you bring up a sharded cluster on 3.7.5 with a mmapv1 multi-node shard, you get a timeout error from trying to add the shard on the mongos: 

      mongos> db.runCommand({"addShard" : "a/louisamac:9001,louisamac:9002,louisamac:9003", name: "shard0"})
{
	"ok" : 0,
	"errmsg" : "failed to run command { setFeatureCompatibilityVersion: \"4.0\" } when attempting to add shard a/louisamac:9001,louisamac:9002,louisamac:9003 :: caused by :: NetworkInterfaceExceededTimeLimit: timed out",
	"code" : 96,
	"codeName" : "OperationFailed",
	"$clusterTime" : {
		"clusterTime" : Timestamp(1524674234, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"operationTime" : Timestamp(1524674234, 1)
}

      Note: I tried this on WT and it worked fine. Assigning to storage for that reason, but may need to be reassigned.
      Second note: Also did not have this problem with a 1-node shard.

      Attached config files and log files.

        1. cfg1.json
          0.4 kB
        2. cfg2.json
          0.4 kB
        3. cfg3.json
          0.4 kB
        4. csrs_primary.log
          41 kB
        5. csrs_secondary.log
          29 kB
        6. mongos.json
          0.2 kB
        7. mongos.log
          7 kB
        8. shard_primary.log
          37 kB
        9. shard_secondary.log
          37 kB
        10. shard1.json
          0.4 kB
        11. shard2.json
          0.4 kB
        12. shard3.json
          0.4 kB

              Assignee:
              Esha Maharishi (Inactive)
              Reporter:
              Louisa Berger
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: