[SERVER-12077] authSchemaUpgradeStep error message when running on 2.5.4 mongos with 2.4 config server could be better Created: 12/Dec/13  Updated: 11/Jul/16  Resolved: 19/Feb/14

Status: Closed
Project: Core Server
Component/s: Security, Sharding
Affects Version/s: None
Fix Version/s: 2.6.0-rc0

Type: Improvement Priority: Minor - P4
Reporter: Valeri Karpov Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 0
Labels: 26qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-12654 append additional information to inco... Closed
Participants:

 Description   

Output now looks like:

Auth schema upgraded failed with UnknownError cannot update system collection: admin.system.version q:

{ _id: "authSchema" }

u: { $set:

{ currentVersion: 1 }

}

This could be more clear and indicate that there's a config server that you forgot to upgrade.



 Comments   
Comment by Valeri Karpov [ 18/Feb/14 ]

spencer LGTM!

----------
Doing upgrade

m30999| 2014-02-18T18:17:14.946-0500 [conn1] distributed lock 'authorizationData/specter.local:30999:1392765432:16807' acquired, ts : 5303e9fada09571b96365e03
m30999| 2014-02-18T18:17:14.947-0500 [conn1] checking that version of host localhost:30000 is compatible with 2.5.4
m30999| 2014-02-18T18:17:14.947-0500 [conn1] checking that version of host localhost:30001 is compatible with 2.5.4
m30999| 2014-02-18T18:17:14.947-0500 [conn1] checking that version of host localhost:29000 is compatible with 2.5.4
m30999| 2014-02-18T18:17:14.947-0500 [conn1] Auth schema upgrade failed: RemoteValidationError version 2.4.8 detected on mongo server at localhost:29000, but version >= 2.5.4 required
m30999| 2014-02-18T18:17:14.948-0500 [conn1] distributed lock 'authorizationData/specter.local:30999:1392765432:16807' unlocked.

----------
Upgrade done!

Comment by Githook User [ 13/Feb/14 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-12077 Log when auth schema upgrade fails due to out of date server
Branch: master
https://github.com/mongodb/mongo/commit/66f564bf028cfd44193a464daec890fd0de5929f

Comment by Spencer Brody (Inactive) [ 13/Feb/14 ]

Looks like the command is returning the right result object but nothing is being logged by the server to indicate that the upgrade failed.

Comment by Valeri Karpov [ 13/Feb/14 ]

So I'm running https://github.com/10gen/QA/blob/master/QA-424/sharded_upgrade_with_24_configsvr.js against the binary from http://mci.10gen.com/ui/build/mongodb_mongo_master_osx_108_db7b5a93e9bdfee1826a91b94a9f8b904be89813_14_02_13_03_03_10 and the only output that I get from the authSchemaUpgrade is:

m30999| 2014-02-13T16:17:33.594-0500 [conn1] distributed lock 'authorizationData/specter.local:30999:1392326251:16807' acquired, ts : 52fd366dc7de95733d8d1d70
m30999| 2014-02-13T16:17:33.594-0500 [conn1] checking that version of host localhost:30000 is compatible with 2.5.4
m30999| 2014-02-13T16:17:33.594-0500 [conn1] checking that version of host localhost:30001 is compatible with 2.5.4
m30999| 2014-02-13T16:17:33.595-0500 [conn1] checking that version of host localhost:29000 is compatible with 2.5.4
m30999| 2014-02-13T16:17:33.595-0500 [conn1] distributed lock 'authorizationData/specter.local:30999:1392326251:16807' unlocked.

The command does fail, which is good, but I don't see any indication that I'm doing something wrong in the logs.

Comment by Githook User [ 13/Feb/14 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-12077 Allow custom ConnectionStrings to unbreak unit tests
Branch: master
https://github.com/mongodb/mongo/commit/b40b20db5f5770cb4c1d0bc2899358a4d1860f7a

Comment by Spencer Brody (Inactive) [ 12/Feb/14 ]

With the change I just pushed, now when I try this I get:

db.adminCommand("authSchemaUpgrade")
{
	"ok": 0,
	"errmsg": "version 2.4.4 detected on mongo server at ubuntu:30002, but version >= 2.5.4 required",
	"code": 25
}

valeri.karpov, can you test again with the change I just put in?

Comment by Githook User [ 12/Feb/14 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-12077 Include config servers in minimum server version checking code

This improves the error message when you try to run the authSchemaUpgrade on a system with an
outdated config server
Branch: master
https://github.com/mongodb/mongo/commit/51d9c6e2f92001f9a6fd988548f86c1c9ed0c4e4

Comment by Githook User [ 12/Feb/14 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-12077 Include config servers in minimum server version checking code

This improves the error message when you try to run the authSchemaUpgrade on a system with an
outdated config server
Branch: mastedr
https://github.com/mongodb/mongo/commit/51d9c6e2f92001f9a6fd988548f86c1c9ed0c4e4

Comment by Valeri Karpov [ 12/Feb/14 ]

I re-ran my tests and right now I don't get any error message or other indication that something went wrong, other than authSchemaVersion is still 1. The only errors I see are:

m30999| 2014-02-12T10:41:50.693-0500 [conn1] warning: Auth schema upgrade failed to drop indexes on admin.system.new_users (Unknown error code dropIndexes failed)

Comment by Spencer Brody (Inactive) [ 07/Feb/14 ]

Taking back to update the function that checks the versions of the cluster before doing an upgrade to include checking the config servers.

Comment by Greg Studer [ 07/Feb/14 ]

In general it's not an error to write to different versions of config servers at the write level - it shouldn't/didn't fail because one of the servers was 2.4, it failed because the write result was different on one of the servers.

We could attempt to make it an option to only write to particular versions and fail if that's not possible - I see this as having limited general value right now though (maybe later). It seems like this is something that should be caught when you check the versions of everything in the cluster before doing the final upgrade step?

Comment by Spencer Brody (Inactive) [ 06/Feb/14 ]

The fact that the message says the write was not consistent makes me a bit nervous... Also, it'd be nice if the clusterWrite operation could indicate that the reason it failed was because one of the config servers was still running 2.4. Assigning to Greg for triage.

Comment by Spencer Brody (Inactive) [ 05/Feb/14 ]

I believe this behavior has changed since this ticket was filed due to the switch to using cluster write operations. The new return value is:

> db.runCommand('authSchemaUpgrade')
{
	"ok": 0,
	"errmsg": "config write was not consistent, manual intervention may be required",
	"code": 51
}

I think any more detailed error message would have to come from the clusterWrite code, not the auth code.

valeri.karpov, can you confirm that the behavior has changed from what you initially observed when this was filed?

Comment by Andy Schwerin [ 12/Dec/13 ]

The AuthzManagerExternalStateS should be extracting codes from failed operations, when available, rather than using UnknownError. This would allow us to check the code and put a custom message in at a higher level.

Generated at Thu Feb 08 03:27:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.