[SERVER-6897] 2.2.0 upgrade of secondary of replica set with --keyfile authentication produces authentication errors - cannot upgrade without downtime Created: 30/Aug/12 Updated: 04/Feb/15 Resolved: 05/Sep/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Write Ops |
| Affects Version/s: | 2.2.0 |
| Fix Version/s: | 2.2.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ian Daniel | Assignee: | Spencer Brody (Inactive) |
| Resolution: | Done | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
upgrading 3-node replica sets from 2.0.5 to 2.2.0 |
||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Problem Description The 2.2 release notes instructions for upgrading a replica set do not have any particular instructions for when using authentication (i.e. running with --keyfile). Given that the instructions for upgrading a shard cluster that immediately follow mention specific instructions for authentication, the strong impression is that there is nothing special to do for a replica set. In a 2.0.x replica set, not in a sharded cluster and using authentication, when the secondary is dropped and then 2.2.0 started instead, the following error occurs (in the log file):
The entry for the secondary in rs.status() contains the following error:
The full rs.status() is:
Note that the error message in the secondary of the rs.status() mentions mongos, which implies a sharded cluster. There is no sharded cluster; just a single replica set. Consequently if you proceed with dropping the other secondary nodes, as per our upgrade instructions, eventually the primary will demote itself to a secondary once there is no longer a majority. Thus there is downtime. You have to shutdown the primary then start it up again with the new version before the problem is resolved. Workaround for 2.2.0 You cannot perform a rolling upgrade of a replica set using authentication from an earlier version to 2.2.0. You must shut down all nodes of the replica set first, then start each node with the 2.2.0 mongod binary. Fix in 2.2.1 You will be able to perform a rolling upgrade of a replica set using authentication to 2.2.1 (and later versions). You will be able to shutdown each secondary one at a time and re-start it with the 2.2.1 (or later) binary. Then perform an rs.stepDown() on the primary, then shut it down and re-start it with the 2.2.1 (or later) binary. Looking forward, it will be possible to do a rolling upgrade of a replica set using authentication from 2.0.x to 2.4, however only by first doing a rolling upgrade to 2.2.1 (or a later 2.2.x release) and then a rolling upgrade from 2.2.x to 2.4.x. |
| Comments |
| Comment by Spencer Brody (Inactive) [ 05/Sep/12 ] |
|
The fix that has been pushed - and will be included in 2.2.1 - will allow a rolling upgrade from 2.0.x to 2.2.1 running auth without downtime. In the future, if someone wants to upgrade from 2.0 to 2.4, they will have to first upgrade from 2.0 to 2.2, then from 2.2 to 2.4, rather than going from 2.0 to 2.4 directly, to do the upgrade without downtime. |
| Comment by Spencer Brody (Inactive) [ 05/Sep/12 ] |
|
Fix has been pushed to the 2.2 branch. It has purposefully been left out of master as it will not be going into 2.3/2.4. This means that to upgrade a replica set with auth from 2.0 to 2.4 will require upgrading to 2.2 first. |
| Comment by auto [ 05/Sep/12 ] |
|
Author: {u'date': u'2012-08-30T10:38:14-07:00', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}Message: Allow commands sent by system user to run even without an $auth table. Upgrading a replica set from 2.0 to 2.2 using auth is impossible without downtime |
| Comment by Ian Daniel [ 30/Aug/12 ] |
|
For completeness, I can confirm that the replica set upgrade instructions work fine if you are not running authentication in the replica set. |