[SERVER-6897] 2.2.0 upgrade of secondary of replica set with --keyfile authentication produces authentication errors - cannot upgrade without downtime Created: 30/Aug/12  Updated: 04/Feb/15  Resolved: 05/Sep/12

Status: Closed
Project: Core Server
Component/s: Replication, Write Ops
Affects Version/s: 2.2.0
Fix Version/s: 2.2.1

Type: Bug Priority: Major - P3
Reporter: Ian Daniel Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

upgrading 3-node replica sets from 2.0.5 to 2.2.0


Issue Links:
Depends
Related
is related to DOCS-465 2.2.0 Release Notes Require an Update... Closed
Operating System: ALL
Participants:

 Description   

Problem Description

The 2.2 release notes instructions for upgrading a replica set do not have any particular instructions for when using authentication (i.e. running with --keyfile). Given that the instructions for upgrading a shard cluster that immediately follow mention specific instructions for authentication, the strong impression is that there is nothing special to do for a replica set.

In a 2.0.x replica set, not in a sharded cluster and using authentication, when the secondary is dropped and then 2.2.0 started instead, the following error occurs (in the log file):

Thu Aug 30 14:47:37 [conn30] command denied: { replSetHeartbeat: "MyRS", v: 3, pv: 1, checkEmpty: false, from: "ian.local:30003" }
Thu Aug 30 14:47:37 [conn31] command denied: { replSetHeartbeat: "MyRS", v: 3, pv: 1, checkEmpty: false, from: "ian.local:30001" }

The entry for the secondary in rs.status() contains the following error:

"errmsg" : "unauthorized: no auth credentials provided for command and authenticated using internal user.  This is most likely because you are using an old version of mongos

The full rs.status() is:

MyRS:PRIMARY> rs.status()
{
	"set" : "MyRS",
	"date" : ISODate("2012-08-30T04:48:56Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ian.local:30001",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"optime" : Timestamp(1346301568000, 1),
			"optimeDate" : ISODate("2012-08-30T04:39:28Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ian.local:30002",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : Timestamp(1346301568000, 1),
			"optimeDate" : ISODate("2012-08-30T04:39:28Z"),
			"lastHeartbeat" : ISODate("2012-08-30T04:48:55Z"),
			"pingMs" : 0,
			"errmsg" : "unauthorized: no auth credentials provided for command and authenticated using internal user.  This is most likely because you are using an old version of mongos"
		},
		{
			"_id" : 2,
			"name" : "ian.local:30003",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 568,
			"optime" : Timestamp(0, 0),
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2012-08-30T04:48:55Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}

Note that the error message in the secondary of the rs.status() mentions mongos, which implies a sharded cluster. There is no sharded cluster; just a single replica set.

Consequently if you proceed with dropping the other secondary nodes, as per our upgrade instructions, eventually the primary will demote itself to a secondary once there is no longer a majority. Thus there is downtime. You have to shutdown the primary then start it up again with the new version before the problem is resolved.

Workaround for 2.2.0

You cannot perform a rolling upgrade of a replica set using authentication from an earlier version to 2.2.0. You must shut down all nodes of the replica set first, then start each node with the 2.2.0 mongod binary.

Fix in 2.2.1

You will be able to perform a rolling upgrade of a replica set using authentication to 2.2.1 (and later versions). You will be able to shutdown each secondary one at a time and re-start it with the 2.2.1 (or later) binary. Then perform an rs.stepDown() on the primary, then shut it down and re-start it with the 2.2.1 (or later) binary.

Looking forward, it will be possible to do a rolling upgrade of a replica set using authentication from 2.0.x to 2.4, however only by first doing a rolling upgrade to 2.2.1 (or a later 2.2.x release) and then a rolling upgrade from 2.2.x to 2.4.x.



 Comments   
Comment by Spencer Brody (Inactive) [ 05/Sep/12 ]

The fix that has been pushed - and will be included in 2.2.1 - will allow a rolling upgrade from 2.0.x to 2.2.1 running auth without downtime.

In the future, if someone wants to upgrade from 2.0 to 2.4, they will have to first upgrade from 2.0 to 2.2, then from 2.2 to 2.4, rather than going from 2.0 to 2.4 directly, to do the upgrade without downtime.

Comment by Spencer Brody (Inactive) [ 05/Sep/12 ]

Fix has been pushed to the 2.2 branch. It has purposefully been left out of master as it will not be going into 2.3/2.4. This means that to upgrade a replica set with auth from 2.0 to 2.4 will require upgrading to 2.2 first.

Comment by auto [ 05/Sep/12 ]

Author:

{u'date': u'2012-08-30T10:38:14-07:00', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}

Message: Allow commands sent by system user to run even without an $auth table. SERVER-6897

Upgrading a replica set from 2.0 to 2.2 using auth is impossible without downtime
otherwise, as the heartbeats sent from the 2.0 nodes won't contain an auth table.
Branch: v2.2
https://github.com/mongodb/mongo/commit/a9c7cbec42c94bba604c67d279c9f65ee27efb6a

Comment by Ian Daniel [ 30/Aug/12 ]

For completeness, I can confirm that the replica set upgrade instructions work fine if you are not running authentication in the replica set.

Generated at Thu Feb 08 03:13:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.