Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-25432

Migrating using Replication - Issue with db authentication on new Replica Member

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Done
    • Icon: Critical - P2 Critical - P2
    • None
    • None
    • Admin, Replication
    • None

    Description

      Our team is working on a migration from a clustered replica set mongo on AWS EC2 to a replica set cluster on Google's Compute Engine.

      We were able to add one of the new Google mongo instances as a member to the AWS cluster with a priority 0, however it appears that there is an issue with authentication which won't allow the new mongo to sync.

      Google Compute Instance:

      INFRA-GENERAL-00:SECONDARY> rs.status()
      {
      	"set" : "INFRA-GENERAL-00",
      	"date" : ISODate("2016-08-03T00:22:37.275Z"),
      	"myState" : 2,
      	"term" : NumberLong(2),
      	"heartbeatIntervalMillis" : NumberLong(2000),
      	"members" : [
      		{
      			"_id" : 0,
      			"name" : "****:27017",
      			"health" : 0,
      			"state" : 8,
      			"stateStr" : "(not reachable/healthy)",
      			"uptime" : 0,
      			"optime" : {
      				"ts" : Timestamp(0, 0),
      				"t" : NumberLong(-1)
      			},
      			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
      			"lastHeartbeat" : ISODate("2016-08-03T00:22:35.205Z"),
      			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
      			"pingMs" : NumberLong(0),
      			"lastHeartbeatMessage" : "exception: field not found, expected type 16",
      			"configVersion" : -1
      		},
      		{
      			"_id" : 1,
      			"name" : "****:27017",
      			"health" : 0,
      			"state" : 8,
      			"stateStr" : "(not reachable/healthy)",
      			"uptime" : 0,
      			"optime" : {
      				"ts" : Timestamp(0, 0),
      				"t" : NumberLong(-1)
      			},
      			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
      			"lastHeartbeat" : ISODate("2016-08-03T00:22:34.728Z"),
      			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
      			"pingMs" : NumberLong(0),
      			"lastHeartbeatMessage" : "exception: field not found, expected type 16",
      			"configVersion" : -1
      		},
      		{
      			"_id" : 2,
      			"name" : "****:27017",
      			"health" : 0,
      			"state" : 8,
      			"stateStr" : "(not reachable/healthy)",
      			"uptime" : 0,
      			"optime" : {
      				"ts" : Timestamp(0, 0),
      				"t" : NumberLong(-1)
      			},
      			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
      			"lastHeartbeat" : ISODate("2016-08-03T00:22:35.248Z"),
      			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
      			"pingMs" : NumberLong(0),
      			"lastHeartbeatMessage" : "exception: field not found, expected type 16",
      			"configVersion" : -1
      		},
      		{
      			"_id" : 3,
      			"name" : "****:27017",
      			"health" : 1,
      			"state" : 2,
      			"stateStr" : "SECONDARY",
      			"uptime" : 338,
      			"optime" : {
      				"ts" : Timestamp(1470169610, 1),
      				"t" : NumberLong(2)
      			},
      			"optimeDate" : ISODate("2016-08-02T20:26:50Z"),
      			"configVersion" : 175485,
      			"self" : true
      		}
      	],
      	"ok" : 1
      }
      

      AWS Primary Instance:

      INFRA-GENERAL-00:PRIMARY> rs.status()
      {
      	"set" : "INFRA-GENERAL-00",
      	"date" : ISODate("2016-08-03T16:00:59Z"),
      	"myState" : 1,
      	"members" : [
      		{
      			"_id" : 0,
      			"name" : "****:27017",
      			"health" : 1,
      			"state" : 1,
      			"stateStr" : "PRIMARY",
      			"uptime" : 690001,
      			"optime" : Timestamp(1470240059, 4),
      			"optimeDate" : ISODate("2016-08-03T16:00:59Z"),
      			"electionTime" : Timestamp(1470167971, 1),
      			"electionDate" : ISODate("2016-08-02T19:59:31Z"),
      			"self" : true
      		},
      		{
      			"_id" : 1,
      			"name" : "****:27017",
      			"health" : 1,
      			"state" : 2,
      			"stateStr" : "SECONDARY",
      			"uptime" : 72094,
      			"optime" : Timestamp(1470240058, 29),
      			"optimeDate" : ISODate("2016-08-03T16:00:58Z"),
      			"lastHeartbeat" : ISODate("2016-08-03T16:00:58Z"),
      			"lastHeartbeatRecv" : ISODate("2016-08-03T16:00:59Z"),
      			"pingMs" : 1,
      			"syncingTo" : "mongorep01.restdev.com:27017"
      		},
      		{
      			"_id" : 2,
      			"name" : "****:27017",
      			"health" : 1,
      			"state" : 2,
      			"stateStr" : "SECONDARY",
      			"uptime" : 72094,
      			"optime" : Timestamp(1470240057, 215),
      			"optimeDate" : ISODate("2016-08-03T16:00:57Z"),
      			"lastHeartbeat" : ISODate("2016-08-03T16:00:57Z"),
      			"lastHeartbeatRecv" : ISODate("2016-08-03T16:00:58Z"),
      			"pingMs" : 0,
      			"syncingTo" : "mongorep01.restdev.com:27017"
      		},
      		{
      			"_id" : 3,
      			"name" : "****:27017",
      			"health" : 1,
      			"state" : 2,
      			"stateStr" : "SECONDARY",
      			"uptime" : 54470,
      			"optime" : Timestamp(1470169610, 1),
      			"optimeDate" : ISODate("2016-08-02T20:26:50Z"),
      			"lastHeartbeat" : ISODate("2016-08-03T16:00:58Z"),
      			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
      			"pingMs" : 37
      		}
      	],
      	"ok" : 1
      }
      

      While checking the Google Compute Instance's mongo log, I see the following:

      2016-08-03T00:45:31.801+0000 I REPL     [ReplicationExecutor] waiting for 6 pings from other members before syncing
      2016-08-03T00:45:32.251+0000 I NETWORK  [conn272] end connection ***:60176 (4 connections now open)
      2016-08-03T00:45:32.288+0000 I NETWORK  [initandlisten] connection accepted from ***:60179 #275 (5 connections now open)
      2016-08-03T00:45:32.325+0000 I ACCESS   [conn275]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
      2016-08-03T00:45:32.809+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to ***:27017; Location13111: exception: field not found, expected type 16
      2016-08-03T00:45:32.850+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to ***:27017; Location13111: exception: field not found, expected type 16
      2016-08-03T00:45:32.890+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to ***:27017; Location13111: exception: field not found, expected type 16
      

      The Primary and other AWS instances seem to require authentication to be done through the 'admin' db, not 'local'. I'm assuming this is where the problem lies.

      Is there a way to have the new replica set member authenticate against admin, not local?

      Attachments

        Activity

          People

            Unassigned Unassigned
            mitchell@restorationmedia.com Mitchell Harding
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: