Migrating using Replication - Issue with db authentication on new Replica Member

XMLWordPrintableJSON

    • Type: Question
    • Resolution: Done
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: Admin, Replication
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Our team is working on a migration from a clustered replica set mongo on AWS EC2 to a replica set cluster on Google's Compute Engine.

      We were able to add one of the new Google mongo instances as a member to the AWS cluster with a priority 0, however it appears that there is an issue with authentication which won't allow the new mongo to sync.

      Google Compute Instance: 

      INFRA-GENERAL-00:SECONDARY> rs.status()
{
	"set" : "INFRA-GENERAL-00",
	"date" : ISODate("2016-08-03T00:22:37.275Z"),
	"myState" : 2,
	"term" : NumberLong(2),
	"heartbeatIntervalMillis" : NumberLong(2000),
	"members" : [
		{
			"_id" : 0,
			"name" : "****:27017",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2016-08-03T00:22:35.205Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "exception: field not found, expected type 16",
			"configVersion" : -1
		},
		{
			"_id" : 1,
			"name" : "****:27017",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2016-08-03T00:22:34.728Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "exception: field not found, expected type 16",
			"configVersion" : -1
		},
		{
			"_id" : 2,
			"name" : "****:27017",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2016-08-03T00:22:35.248Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "exception: field not found, expected type 16",
			"configVersion" : -1
		},
		{
			"_id" : 3,
			"name" : "****:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 338,
			"optime" : {
				"ts" : Timestamp(1470169610, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2016-08-02T20:26:50Z"),
			"configVersion" : 175485,
			"self" : true
		}
	],
	"ok" : 1
}

      AWS Primary Instance:

      INFRA-GENERAL-00:PRIMARY> rs.status()
{
	"set" : "INFRA-GENERAL-00",
	"date" : ISODate("2016-08-03T16:00:59Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "****:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 690001,
			"optime" : Timestamp(1470240059, 4),
			"optimeDate" : ISODate("2016-08-03T16:00:59Z"),
			"electionTime" : Timestamp(1470167971, 1),
			"electionDate" : ISODate("2016-08-02T19:59:31Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "****:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 72094,
			"optime" : Timestamp(1470240058, 29),
			"optimeDate" : ISODate("2016-08-03T16:00:58Z"),
			"lastHeartbeat" : ISODate("2016-08-03T16:00:58Z"),
			"lastHeartbeatRecv" : ISODate("2016-08-03T16:00:59Z"),
			"pingMs" : 1,
			"syncingTo" : "mongorep01.restdev.com:27017"
		},
		{
			"_id" : 2,
			"name" : "****:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 72094,
			"optime" : Timestamp(1470240057, 215),
			"optimeDate" : ISODate("2016-08-03T16:00:57Z"),
			"lastHeartbeat" : ISODate("2016-08-03T16:00:57Z"),
			"lastHeartbeatRecv" : ISODate("2016-08-03T16:00:58Z"),
			"pingMs" : 0,
			"syncingTo" : "mongorep01.restdev.com:27017"
		},
		{
			"_id" : 3,
			"name" : "****:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 54470,
			"optime" : Timestamp(1470169610, 1),
			"optimeDate" : ISODate("2016-08-02T20:26:50Z"),
			"lastHeartbeat" : ISODate("2016-08-03T16:00:58Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : 37
		}
	],
	"ok" : 1
}

      While checking the Google Compute Instance's mongo log, I see the following:

      2016-08-03T00:45:31.801+0000 I REPL     [ReplicationExecutor] waiting for 6 pings from other members before syncing
2016-08-03T00:45:32.251+0000 I NETWORK  [conn272] end connection ***:60176 (4 connections now open)
2016-08-03T00:45:32.288+0000 I NETWORK  [initandlisten] connection accepted from ***:60179 #275 (5 connections now open)
2016-08-03T00:45:32.325+0000 I ACCESS   [conn275]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2016-08-03T00:45:32.809+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to ***:27017; Location13111: exception: field not found, expected type 16
2016-08-03T00:45:32.850+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to ***:27017; Location13111: exception: field not found, expected type 16
2016-08-03T00:45:32.890+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to ***:27017; Location13111: exception: field not found, expected type 16

      The Primary and other AWS instances seem to require authentication to be done through the 'admin' db, not 'local'. I'm assuming this is where the problem lies.

      Is there a way to have the new replica set member authenticate against admin, not local?

            Assignee:
            Unassigned
            Reporter:
            Mitchell Harding
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: