[SERVER-24778] Kubernetes service not recognized as self in 3.2.7 Created: 24/Jun/16  Updated: 25/Aug/22  Resolved: 06/Jul/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Peter Kriens Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive objects.zip     PNG File screenshot-1.png    
Operating System: ALL
Steps To Reproduce:

Running Mongodb 3.2.7 on Kubernetes with a 2 member replicata set. Defined 2 services for this on mongo-r0 and mongo-r2. Mongodb replicate set initialized to have memberp[0] be "mongo-r0:27017" and [1] "mongo-r1:27017". At startup mongodb complains that _isSelf fails to recognize either as itself.

After replacing the members[x].host with their respective IP number I got the replica up and running again (using force:true).

To verify, I changed this again, to verify it was transient but this removed the replicates from the set and everything came to another standstill. Then repaired the replica set.

The I verified that running on the docker image where mongo runs I can connect to the database with the Kubernetes service name mongo-1 and mongo-0:

  root@mongo-r0-iw4q4:/#  mongo --host mongo-r1
    MongoDB shell version: 3.2.7
    connecting to: mongo-r1:27017/test
 
   root@mongo-r0-iw4q4:/# mongo --host mongo-r0
   MongoDB shell version: 3.2.7
   connecting to: mongo-r0:27017/test

This is the rs.conf() on the primary after it was fixed again:

{
        "_id" : "jpm",
        "version" : 95740,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "10.0.2.9:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "10.0.0.4:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "getLastErrorModes" : {
 
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                }
        }
}

And status

        "set" : "jpm",
        "date" : ISODate("2016-06-24T13:18:31.464Z"),
        "myState" : 2,
        "term" : NumberLong(-1),
        "syncingTo" : "10.0.0.4:27017",
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "10.0.2.11:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 2298,
                        "optime" : Timestamp(1466774311, 1),
                        "optimeDate" : ISODate("2016-06-24T13:18:31Z"),
                        "syncingTo" : "10.0.0.4:27017",
                        "configVersion" : 133491,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "10.0.0.4:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 2067,
                        "optime" : Timestamp(1466774309, 3),
                        "optimeDate" : ISODate("2016-06-24T13:18:29Z"),
                        "lastHeartbeat" : ISODate("2016-06-24T13:18:29.797Z"),
                        "lastHeartbeatRecv" : ISODate("2016-06-24T13:18:30.222Z"),
                        "pingMs" : NumberLong(0),
                        "electionTime" : Timestamp(1466772243, 1),
                        "electionDate" : ISODate("2016-06-24T12:44:03Z"),
                        "configVersion" : 133491
                }
        ],
        "ok" : 1
}

Changing the host in members[0&1] to mongo-rx:27017 makes the

Participants:

 Description   

Mongodb fails to recognize itself at startup on Kubernetes when a Kubernetes service name is used for the database access host/port. This worked on the (unknown) previous version but fails on 3.2.7

rs.status()
{
        "state" : 10,
        "stateStr" : "REMOVED",
        "uptime" : 2035,
        "optime" : Timestamp(1466516697, 2),
        "optimeDate" : ISODate("2016-06-21T13:44:57Z"),
        "ok" : 0,
        "errmsg" : "Our replica set config is invalid or we are not a member of it",
        "code" : 93
}

Verified that the Kubernetes service host name is present in the local DNS and that it properly maps to the correct endpoint by opening a mongo shell via that DNS name.

This happened after my Kubernetes pod was reset due to problems with the application. It is likely that this updated mongodb to the latest version 3.2.7 since the pod uses the latest mongodb image.

The fix was to hard code the IP number but this is very fragile in Kubernetes because the IP number changes when the pod gets deleted, which can happen randomly.



 Comments   
Comment by V _ [ 25/Aug/22 ]

thank you @Immanuel P

`publishNotReadyAddresses: true` fixes my problem!

Sorry to comment on this closed issue.  I think it can save others some headache if we document this K8s behavior somewhere?

Comment by Immanuel P. [ 13/Mar/19 ]

For anyone ending up here from google like me: Setting `publishNotReadyAddresses: true` in the Service spec also fixed this issue for me. Please note that this is a Headless service used in a StatefulSet

kind: Service
apiVersion: v1
metadata:
  name: shard-1
  labels:
    app: shard-1
spec:
  selector:
    app: shard-1
  ports:
  - protocol: TCP
    port: 27018
    targetPort: 27018
  clusterIP: None # Headless service
  publishNotReadyAddresses: true # Fix MongoDB hostname issue

Comment by Andy Schwerin [ 06/Jul/16 ]

pkriens, I'm going to close this ticket as "community answered", as it looks like there is no required change to MongoDB at present. Re-open if you disagree. Thanks!

Comment by Peter Kriens [ 05/Jul/16 ]

Yup that does the yucky trick I added a 5 second delay and then my test server started up.

I had focused on the DNS aspect but k8s of course needs to initialize the IP tables once the service is up. That is a pretty nasty race condition.

Thanks for the excellent support here! And sorry to have bothered/blamed you

Comment by LaurieTurner [ 05/Jul/16 ]

@Peter - in my env. my cluster IP address never changes because I deploy a service for the cluster member. There are two aspects here 1> I can see that it takes some period of time for kubernetes to bind that ip address to the port at the endpoint when the controller is re-scaled. You can use command "kubectl describe endpoint <controller>" and you'll see the Endpoints isn't set for a period of time. 2> I also have a readiness probe which is sort of a catch 22 here. k8s will not turn on my service address until my readiness probe starts. Without that address, the Cluster IP defined by the service is not viable.

I plan to use the rest api provided by k8s to determine when the endpoint is bound (v1/api/endpoints). For now, my env. is not setup , security wise for that, so i put a simply delay in the entrypoint script , and the problem does not happen.

I

Comment by Peter Kriens [ 04/Jul/16 ]

@Andy: a full log on a 1 member replica set that exhibits this behavior

$ kubectl create -f objects/mongo-r9-controller.yaml
replicationcontrollers/mongo-r9
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
jpm-u7tqg 1/1 Running 0 9d
mongo-r0-iw4q4 1/1 Running 0 9d
mongo-r1-tq2yn 1/1 Running 0 9d
mongo-r9-4yki6 0/1 Pending 0 5s
$ kubectl logs mongo-r9-4yki6
2016-07-04T10:03:19.169+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=mongo-r9-4yki6
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] db version v3.2.7
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] modules: none
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] build environment:
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] distmod: debian71
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] distarch: x86_64
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-07-04T10:03:19.170+0000 I CONTROL [initandlisten] options: { replication:

{ replSet: "jpm" }

}
2016-07-04T10:03:19.176+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 29981 processes, 1000000 files. Number of processes should be at least 500000 : 0.5 times number of files.
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.440+0000 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument: Did not find replica set lastVote document in local.replset.election
2016-07-04T10:03:19.440+0000 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2016-07-04T10:03:19.441+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2016-07-04T10:03:19.441+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-07-04T10:03:19.484+0000 I NETWORK [initandlisten] waiting for connections on port 27017

$ kubectl exec -it mongo-r9-4yki6 mongo
MongoDB shell version: 3.2.7
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 29981 processes, 1000000 files. Number of processes should be at least 500000 : 0.5 times number of files.
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
>
> rs.config()
2016-07-04T10:04:47.039+0000 E QUERY [thread1] Error: Could not retrieve replica set config:

{ "info" : "run rs.initiate(...) if not yet done for the set", "ok" : 0, "errmsg" : "no replset config has been received", "code" : 94 }

:
rs.conf@src/mongo/shell/utils.js:1196:11
@(shell):1:1

> rs.initiate()

{ "info2" : "no configuration specified. Using a default configuration for the set", "me" : "mongo-r9-4yki6:27017", "ok" : 1 }

jpm:OTHER> rs.add( "mongo-r9:27017")

{ "ok" : 0, "errmsg" : "The hosts mongo-r9-4yki6:27017 and mongo-r9:27017 all map to this node in new configuration version 2 for replica set jpm", #pkr# #pkr# this is true but for this to work the name 'mongo-r9-4yki6' is obviously wrong since it will change on restart #pkr# "code" : 103 }

jpm:PRIMARY> rs.config()
{
"_id" : "jpm",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "mongo-r9-4yki6:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" :

{ "w" : 1, "wtimeout" : 0 }

,
"replicaSetId" : ObjectId("577a34deb55abf961fc3d973")
}
}
jpm:PRIMARY> exit
bye
$ kubectl exec -it mongo-r9-4yki6 mongo
MongoDB shell version: 3.2.7
connecting to: test
Server has startup warnings:
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 29981 processes, 1000000 files. Number of processes should be at least 500000 : 0.5 times number of files.
2016-07-04T10:03:19.380+0000 I CONTROL [initandlisten]
jpm:PRIMARY> x=rs.config()
{
"_id" : "jpm",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "mongo-r9-4yki6:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" :

{ "w" : 1, "wtimeout" : 0 }

,
"replicaSetId" : ObjectId("577a34deb55abf961fc3d973")
}
}
#pkr#
#pkr# force the member to use the service name
#pkr#
jpm:PRIMARY> x.members[0].host="mongo-r9:27017"
mongo-r9:27017
jpm:PRIMARY> rs.reconfig

{ "ok" : 1 }

jpm:PRIMARY> rs.config()
{
"_id" : "jpm",
"version" : 2,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "mongo-r9:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" :

{ "w" : 1, "wtimeout" : 0 }

,
"replicaSetId" : ObjectId("577a34deb55abf961fc3d973")
}
}
jpm:PRIMARY> exit
bye

#pkr#
#pkr# restart
#pkr#
$ kubectl delete -f objects/mongo-r9-controller.yaml
replicationcontrollers/mongo-r9
$ kubectl create -f objects/mongo-r9-controller.yaml
replicationcontrollers/mongo-r9
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
jpm-u7tqg 1/1 Running 0 9d
mongo-r0-iw4q4 1/1 Running 0 9d
mongo-r1-tq2yn 1/1 Running 0 9d
mongo-r9-83opw 0/1 Running 0 7s
$ kubectl logs mongo-r9-83opw
2016-07-04T10:08:56.838+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=mongo-r9-83opw
2016-07-04T10:08:56.838+0000 I CONTROL [initandlisten] db version v3.2.7
2016-07-04T10:08:56.838+0000 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2016-07-04T10:08:56.838+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2016-07-04T10:08:56.839+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-07-04T10:08:56.840+0000 I CONTROL [initandlisten] modules: none
2016-07-04T10:08:56.840+0000 I CONTROL [initandlisten] build environment:
2016-07-04T10:08:56.840+0000 I CONTROL [initandlisten] distmod: debian71
2016-07-04T10:08:56.840+0000 I CONTROL [initandlisten] distarch: x86_64
2016-07-04T10:08:56.840+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-07-04T10:08:56.840+0000 I CONTROL [initandlisten] options: { replication:

{ replSet: "jpm" }

}
2016-07-04T10:08:56.847+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-07-04T10:08:56.847+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-07-04T10:08:57.648+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2016-07-04T10:08:57.648+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 3 records totaling to 288 bytes
2016-07-04T10:08:57.648+0000 I STORAGE [initandlisten] Scanning the oplog to determine where to place markers for truncation
2016-07-04T10:08:57.695+0000 I CONTROL [initandlisten]
2016-07-04T10:08:57.696+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-07-04T10:08:57.696+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-07-04T10:08:57.696+0000 I CONTROL [initandlisten]
2016-07-04T10:08:57.696+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 29981 processes, 1000000 files. Number of processes should be at least 500000 : 0.5 times number of files.
2016-07-04T10:08:57.696+0000 I CONTROL [initandlisten]
2016-07-04T10:08:57.700+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2016-07-04T10:08:57.701+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-07-04T10:08:57.703+0000 I NETWORK [initandlisten] waiting for connections on port 27017
#pkr#
#pkr#
2016-07-04T10:08:57.740+0000 W NETWORK [ReplicationExecutor] couldn't check isSelf (mongo-r9:27017) network error while attempting to run command '_isSelf' on host 'mongo-r9:27017'
#pkr#
#pkr#
2016-07-04T10:08:57.740+0000 W REPL [ReplicationExecutor] Locally stored replica set configuration does not have a valid entry for the current node; waiting for reconfig or remote heartbeat; Got "NodeNotFound: No host described in new configuration 2 for replica set jpm maps to this node" while validating { _id: "jpm", version: 2, protocolVersion: 1, members: [ { _id: 0, host: "mongo-r9:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults:

{ w: 1, wtimeout: 0 }

, replicaSetId: ObjectId('577a34deb55abf961fc3d973') } }
2016-07-04T10:08:57.740+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "jpm", version: 2, protocolVersion: 1, members: [ { _id: 0, host: "mongo-r9:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults:

{ w: 1, wtimeout: 0 }

, replicaSetId: ObjectId('577a34deb55abf961fc3d973') } }
2016-07-04T10:08:57.740+0000 I REPL [ReplicationExecutor] This node is not a member of the config
2016-07-04T10:08:57.740+0000 I REPL [ReplicationExecutor] transition to REMOVED
2016-07-04T10:08:57.740+0000 I REPL [ReplicationExecutor] Starting replication applier threads
$

Comment by Peter Kriens [ 04/Jul/16 ]

Could you explain your hypothesis? I do not want to disrupt the existing setup unnecessary because every time I delete a pod I need to update the database with the new IP numbers

I think the examples I included show that that the DNS record is valid for the container, I can run the mongo shell in the container using the service DNS name of mongo-r0 & mongo-r1? I tried to see if this was a race condition so I created a mongo-r9 and changed the command to

command:

  • /bin/ping
  • mongo-r9

This worked (there was nog log output but changing mongo-r9 to foo created a log entry with an error) Having a race condition here seems also VERY unlikely because the service is never deleted or created. The pod is created long after the service is created.

Thanks for your help trying to locate this problem.

Comment by LaurieTurner [ 30/Jun/16 ]

@Peter - I see your service ip setup. It doesn't really apply in my case because I never delete my service interface, just the controller.

For this problem, I think I might be onto something...I would appreciate it you could try something in your env. for "this" problem. Can you hang the entrypoint script after restart...I added this to the top of mine.

if [ ! -z "$DEBUG" ]; then
tail -f /dev/null
fi

Next, at your master, execute this command "kubectl describe service mongo-r0" . After you see the endpoint definition for the service

(i.e. Endpoints: 172.16.101.4:27000)

go back to the mongo-rc0 pod and manually start mongo (i.e. /usr/bin/mongod --replSet ....)

Comment by Peter Kriens [ 29/Jun/16 ]

@Laurie: 'this problem' -> mongo not being able to self identify, i.e. this issue

I think you're missing the service setup I did. This is depicted in the screenshot-1. The services create a constant IP:port with a DNS record. It is this DNS that I use in the application jpm. I've included the setup files.

Comment by Peter Kriens [ 29/Jun/16 ]

Setup files for jpm

Comment by LaurieTurner [ 28/Jun/16 ]

@andy - Andy - I think my comments should be directed to you. What else do you need to identify this as a Mongo JIRA defect?

Comment by LaurieTurner [ 28/Jun/16 ]

Hi Peter

I'm not sure I follow.

Can you explain what "this" problem is? Is it the issue we are discussing where the node cannot rejoin and does not understand the Cluster IP addresses defined in the replica set configuration?

Would you trying setting up the replica set following the white paper identified in the link - 3 nodes and use the Cluster IP address.

I ask this because - DNS names are not persisted when the pod is rescaled.

for example:

[root@norbert ~]# Laurie/go.sh mongo
Using container c2a4d9372918
root@mongo-2-syrin:/# [root@norbert ~]#

At my kube master:

[root@gru Laurie]# kubectl delete -f /root/kube-meta/mongo2-controller-norbert.yml
replicationcontroller "mongo-2" deleted
[root@gru Laurie]# kubectl create -f /root/kube-meta/mongo2-controller-norbert.yml
replicationcontroller "mongo-2" created

Back at my minion the hostname changed.

[root@norbert ~]# Laurie/go.sh mongo
Using container 403d8bfa57cf
root@mongo-2-oiwwg:/#

So as described in the white paper - the Service address must be used to build the cluster - it is the k8s way to maintain "addressability" of a service that can stop, move , restart, etc...

Regarding the questions of issues with Mongo 3.2.7 and k8s - hopefully Mongo development may have these recorded. Are you asking me to uplift my version to see if this problem is resolved?

Thanks for you attention

Laurie

Comment by Peter Kriens [ 28/Jun/16 ]

My setup is as follows:

mongo-r0-controller
mongo-r0-service
mongo-r1-controller
mongo-r1-service

I've got 2 nodes that have label mongo-r1 and mongo-r2. There is a dynamic DNS name for the IP number of the selected mongo-r0-service and mongo-r1-service. This DNS name (mongo-r0/mongo-r1) is put in the database for the replica set. This works fine for the running the shell. If you're interested I can share the files.

My mongo version was accidentally updated to 3.2.7 because I ran 'latest' and then this problem occurred. I might have to revert to an earlier version to see if the problem goes away. However, I hope we can solve this potential regression in mongod to prevent other people from running in the same problem.

Are there any issues around 3.2.7 for K18s?

Comment by LaurieTurner [ 28/Jun/16 ]

Peter -

Thanks for responding.

Let me clarify my usage and also, another couple questions.

We're not using a Load Balancer. From the white paper, there lots of benefits with Kubernetes/docker are listed below, and you're right, load balancing is not really a benefit since a Mongo replica is not a peer based cluster. Load Balancing is for simpler types of micro-services. But the other benefits listed can be realized.
• Automated deployment and replication of containers
• Online scale-in or scale-out of container clusters
• Load balancing over groups of containers
• Rolling upgrades of application containers
• Resilience, with automated rescheduling of failed containers
• Controlled exposure of network ports to systems outside of the cluster

One more note - you're original question mentioned addressing. The only "persistent" name/ip address in a Kubernetes env. is the Service Host. Both the pod name and the flannel address are regenerated when a docker container is re-deployed.

Lastly, you mention something about the setup working on a previous revision.. Can you clarify? We are using Mongo 3.0.6. Is this not the best version for our goal of Kubernetes environment?

Thanks again,

Laurie

Comment by Peter Kriens [ 28/Jun/16 ]

@Andy: I am on the road right now, will try next week
@Laurie: I read the article but I do not understand why the Load Balancer is needed. They are rather expensive and expose a public IP nr? This setup did work on the previous revision. But thanks for the link, it was an interesting read.

Comment by LaurieTurner [ 28/Jun/16 ]

Peter - The link below describes how to create a replicaset in Kubernetes.
https://www.mongodb.com/blog/post/running-mongodb-as-a-microservice-with-docker-and-kubernetes

It inherently does not want us to use the "container name" as the addressing.

This issue happens 95% of time time . Sometimes, not often, the node does come back.

I can connect the Mongo console inside the container - and it's clear the instance cannot re-join the replicaset , the prompt shows no replica role:.

Using container c2a4d9372918
root@mongo-2-syrin:/# hostname
mongo-2-syrin
root@mongo-2-syrin:/# mongo mongo-2-syrin:27000
MongoDB shell version: 3.0.6
connecting to: mongo-2-syrin:27000/test
>

and as localhost:

root@mongo-2-syrin:/# mongo localhost:27000
MongoDB shell version: 3.0.6
connecting to: localhost:27000/test
>

I can expose a service port and connect a console to the mongo replica however this uses the physical address and not the cluster address .

For example: kubectl expose service mongo-1 --port 9098 --target-port 27000 --name mongo-1-port --external-ip x.x.x.x

MongoDB shell version: 3.2.6
connecting to: x.x.x.x:9098/test
>

I have no problem with a single node replicaset. It restarts fine.

Equally distressing, when the primary node in the 3-node replica won't restart, the other 2 nodes, secondaries in this case, will not/do not have a successful election. Raising priority of a secondary, removing the configuration with force, taking down all the nodes but one and then trying to force the configuration - nothings works.. This should likely be a new issue but I think it's inherently related to the addressing issue here.

Laurie

Comment by Andy Schwerin [ 27/Jun/16 ]

You can run the _isSelf command from the shell, but that won't really tell you much.

If you attempt to create a 1-node replica set, can you see this problem? If so, it would be helpful if you could upload the logs and the replica set configuration. If you need multiple nodes, the logs from all nodes and the replica set configuration would be helpful.

Comment by Peter Kriens [ 25/Jun/16 ]

I did that (well, from one of them but the error was the same on both)

root@mongo-r0-iw4q4:/# mongo --host mongo-r1
MongoDB shell version: 3.2.7
connecting to: mongo-r1:27017/test

root@mongo-r0-iw4q4:/# mongo --host mongo-r0
MongoDB shell version: 3.2.7
connecting to: mongo-r0:27017/test

It could be that my previous version was 3.0

Is there a command in the mongo shell that allows me to exercise the _isSelf logic?

Comment by Andy Schwerin [ 24/Jun/16 ]

Hi, Peter. There haven't been major changes to the isself logic since v3.0. The way isself checks work is that the node attempts to contact each member listed in the replica set config, and looks to see if the remote party's random node id is the same as the local party's. If so, then the remote node is the same as the local node. Prior to 3.0, some shortcuts were taken to avoid making the remote call, but they were error prone and have been removed.

To qualify this as a mongodb bug, I'd like to know that it is possible to make this connection with the container's network configuration. Given that you could do it by IP address, I guess I'd like to know if you can connect a mongo shell in the container to the mongod in the container via the host name listed in the replica set config. Does that make sense?

Comment by Peter Kriens [ 24/Jun/16 ]

If there is anything I can do t o help, don't hesitate to contact me. On Skype: pkriens

Generated at Thu Feb 08 04:07:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.