-
Type:
Question
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.2.10, 3.4.0-rc2
-
Component/s: Replication
Hey Mongoids,
I'm having some difficulty creating a replica set on a Docker Swarm cluster, and I was hoping that you could help me figure out why. I've created three separate services--mongo1, mongo2, and mongo3--from the mongo:latest Docker image using the following command:
docker service create --name=mongo1 --mode=replicated --replicas=1 --publish=27017 --network=ignotae-private mongo:latest mongod --replSet=ignotae --storageEngine=wiredTiger
The three containers come up successfully, usually on hosts that are coincidentally physically separate. From each container, I'm able to ping the others (in my configuration, they typically resolve to the IP addresses 10.0.72.2, 10.0.72.4, and 10.0.72.6) and connect to them with the Mongo shell using their hostnames.
In mongo1, I initiate a replica set and update the first member's hostname:
> rs.initiate()
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "1e0b80e3f356:27017",
"ok" : 1
}
ignotae:SECONDARY> cfg = rs.config()
{
"_id" : "ignotae",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "1e0b80e3f356:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("582806f337c79bc4a39690c2")
}
}
ignotae:PRIMARY> cfg.members[0].host = 'mongo1:27017'
mongo1:27017
ignotae:PRIMARY> rs.reconfig(cfg)
{ "ok" : 1 }
All seems to be well, but begins to go awry when I attempt to add another service to the set:
ignotae:PRIMARY> rs.add('mongo2:27017') { "ok" : 0, "errmsg" : "The hosts mongo1:27017 and mongo2:27017 all map to this node in new configuration version 3 for replica set ignotae", "code" : 103 }
The only log line that seemed pertinent was this one:
2016-11-13T06:58:42.869+0000 E REPL [conn2] replSetReconfig got DuplicateKey: The hosts mongo1:27017 and mongo2:27017 all map to this node in new configuration version 3 for replica set ignotae while validating { _id: "ignotae", version: 3, protocolVersion: 1, members: [ { _id: 0, host: "mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1.0, host: "mongo2:27017" } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('58280e35059ee85bc11db099') } }
For the life of me, I can't figure out what causes this check to fail---the network seems functional and the services are isolated and distinct in every conceivable way. I'm working on rolling back to Docker 1.12 instead of the 1.13 release candidate to see if some change in their networking stack caused this, but I'd be grateful to know if you had any insight into what might cause MongoDB to believe that hosts are duplicates of one another.
Thanks so much, in advance, for your thoughts!