Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-24778

Kubernetes service not recognized as self in 3.2.7

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.7
    • Component/s: Replication
    • Labels:
      None
    • ALL
    • Hide

      Running Mongodb 3.2.7 on Kubernetes with a 2 member replicata set. Defined 2 services for this on mongo-r0 and mongo-r2. Mongodb replicate set initialized to have memberp[0] be "mongo-r0:27017" and [1] "mongo-r1:27017". At startup mongodb complains that _isSelf fails to recognize either as itself.

      After replacing the members[x].host with their respective IP number I got the replica up and running again (using force:true).

      To verify, I changed this again, to verify it was transient but this removed the replicates from the set and everything came to another standstill. Then repaired the replica set.

      The I verified that running on the docker image where mongo runs I can connect to the database with the Kubernetes service name mongo-1 and mongo-0:

        root@mongo-r0-iw4q4:/#  mongo --host mongo-r1
          MongoDB shell version: 3.2.7
          connecting to: mongo-r1:27017/test
      
         root@mongo-r0-iw4q4:/# mongo --host mongo-r0
         MongoDB shell version: 3.2.7
         connecting to: mongo-r0:27017/test
      

      This is the rs.conf() on the primary after it was fixed again:

      {
              "_id" : "jpm",
              "version" : 95740,
              "members" : [
                      {
                              "_id" : 0,
                              "host" : "10.0.2.9:27017",
                              "arbiterOnly" : false,
                              "buildIndexes" : true,
                              "hidden" : false,
                              "priority" : 1,
                              "tags" : {
                              },
                              "slaveDelay" : NumberLong(0),
                              "votes" : 1
                      },
                      {
                              "_id" : 1,
                              "host" : "10.0.0.4:27017",
                              "arbiterOnly" : false,
                              "buildIndexes" : true,
                              "hidden" : false,
                              "priority" : 1,
                              "tags" : {
                              },
                              "slaveDelay" : NumberLong(0),
                              "votes" : 1
                      }
              ],
              "settings" : {
                      "chainingAllowed" : true,
                      "heartbeatIntervalMillis" : 2000,
                      "heartbeatTimeoutSecs" : 10,
                      "electionTimeoutMillis" : 10000,
                      "getLastErrorModes" : {
      
                      },
                      "getLastErrorDefaults" : {
                              "w" : 1,
                              "wtimeout" : 0
                      }
              }
      }
      

      And status

              "set" : "jpm",
              "date" : ISODate("2016-06-24T13:18:31.464Z"),
              "myState" : 2,
              "term" : NumberLong(-1),
              "syncingTo" : "10.0.0.4:27017",
              "heartbeatIntervalMillis" : NumberLong(2000),
              "members" : [
                      {
                              "_id" : 0,
                              "name" : "10.0.2.11:27017",
                              "health" : 1,
                              "state" : 2,
                              "stateStr" : "SECONDARY",
                              "uptime" : 2298,
                              "optime" : Timestamp(1466774311, 1),
                              "optimeDate" : ISODate("2016-06-24T13:18:31Z"),
                              "syncingTo" : "10.0.0.4:27017",
                              "configVersion" : 133491,
                              "self" : true
                      },
                      {
                              "_id" : 1,
                              "name" : "10.0.0.4:27017",
                              "health" : 1,
                              "state" : 1,
                              "stateStr" : "PRIMARY",
                              "uptime" : 2067,
                              "optime" : Timestamp(1466774309, 3),
                              "optimeDate" : ISODate("2016-06-24T13:18:29Z"),
                              "lastHeartbeat" : ISODate("2016-06-24T13:18:29.797Z"),
                              "lastHeartbeatRecv" : ISODate("2016-06-24T13:18:30.222Z"),
                              "pingMs" : NumberLong(0),
                              "electionTime" : Timestamp(1466772243, 1),
                              "electionDate" : ISODate("2016-06-24T12:44:03Z"),
                              "configVersion" : 133491
                      }
              ],
              "ok" : 1
      }
      

      Changing the host in members[0&1] to mongo-rx:27017 makes the

      Show
      Running Mongodb 3.2.7 on Kubernetes with a 2 member replicata set. Defined 2 services for this on mongo-r0 and mongo-r2. Mongodb replicate set initialized to have memberp[0] be "mongo-r0:27017" and [1] "mongo-r1:27017". At startup mongodb complains that _isSelf fails to recognize either as itself. After replacing the members[x].host with their respective IP number I got the replica up and running again (using force:true). To verify, I changed this again, to verify it was transient but this removed the replicates from the set and everything came to another standstill. Then repaired the replica set. The I verified that running on the docker image where mongo runs I can connect to the database with the Kubernetes service name mongo-1 and mongo-0: root@mongo-r0-iw4q4:/# mongo --host mongo-r1 MongoDB shell version: 3.2.7 connecting to: mongo-r1:27017/test root@mongo-r0-iw4q4:/# mongo --host mongo-r0 MongoDB shell version: 3.2.7 connecting to: mongo-r0:27017/test This is the rs.conf() on the primary after it was fixed again: { "_id" : "jpm", "version" : 95740, "members" : [ { "_id" : 0, "host" : "10.0.2.9:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 1, "host" : "10.0.0.4:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatIntervalMillis" : 2000, "heartbeatTimeoutSecs" : 10, "electionTimeoutMillis" : 10000, "getLastErrorModes" : { }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 } } } And status "set" : "jpm", "date" : ISODate("2016-06-24T13:18:31.464Z"), "myState" : 2, "term" : NumberLong(-1), "syncingTo" : "10.0.0.4:27017", "heartbeatIntervalMillis" : NumberLong(2000), "members" : [ { "_id" : 0, "name" : "10.0.2.11:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2298, "optime" : Timestamp(1466774311, 1), "optimeDate" : ISODate("2016-06-24T13:18:31Z"), "syncingTo" : "10.0.0.4:27017", "configVersion" : 133491, "self" : true }, { "_id" : 1, "name" : "10.0.0.4:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2067, "optime" : Timestamp(1466774309, 3), "optimeDate" : ISODate("2016-06-24T13:18:29Z"), "lastHeartbeat" : ISODate("2016-06-24T13:18:29.797Z"), "lastHeartbeatRecv" : ISODate("2016-06-24T13:18:30.222Z"), "pingMs" : NumberLong(0), "electionTime" : Timestamp(1466772243, 1), "electionDate" : ISODate("2016-06-24T12:44:03Z"), "configVersion" : 133491 } ], "ok" : 1 } Changing the host in members [0&1] to mongo-rx:27017 makes the

      Mongodb fails to recognize itself at startup on Kubernetes when a Kubernetes service name is used for the database access host/port. This worked on the (unknown) previous version but fails on 3.2.7

      rs.status()
      {
              "state" : 10,
              "stateStr" : "REMOVED",
              "uptime" : 2035,
              "optime" : Timestamp(1466516697, 2),
              "optimeDate" : ISODate("2016-06-21T13:44:57Z"),
              "ok" : 0,
              "errmsg" : "Our replica set config is invalid or we are not a member of it",
              "code" : 93
      }
      

      Verified that the Kubernetes service host name is present in the local DNS and that it properly maps to the correct endpoint by opening a mongo shell via that DNS name.

      This happened after my Kubernetes pod was reset due to problems with the application. It is likely that this updated mongodb to the latest version 3.2.7 since the pod uses the latest mongodb image.

      The fix was to hard code the IP number but this is very fragile in Kubernetes because the IP number changes when the pod gets deleted, which can happen randomly.

        1. objects.zip
          6 kB
        2. screenshot-1.png
          screenshot-1.png
          39 kB

            Assignee:
            schwerin@mongodb.com Andy Schwerin
            Reporter:
            pkriens Peter Kriens
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: