-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 8.2.12
-
Component/s: None
-
None
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Hello,
I started upgrading a sharded cluster from 8.0.20 to 8.2.12 in production this morning, and it failed badly. I am unsure of the root cause but have a lead.
I followed the usual procedure as described on https://www.mongodb.com/docs/manual/release-notes/8.2-upgrade-sharded-cluster/ :
- upgrade config server secondaries
- stepdown the primary config server, upgrade it
- same with shards
- and finally the mongos
The primary server stepped down at 09:05:22, and all instance of our applications, connected to 15 different mongos, and using the Java driver 5.3.1, started logging "com.mongodb.MongoQueryException: Command failed with error 13 (Unauthorized): 'Command find requires authentication' on server <mongos_server>" at 09:05:28. Every query was impacted and nothing was working.
I stopped the upgrade, and rolled back to config servers to 8.0.20; but the applications kept logging "Command failed with error 13".
The problem went away after restarting all applications.
I investigated the issue on preproduction (and especially why I had not seen that problem in preproduction).
It seems that one difference between the two platforms is that preprod was redeployed more recently, from scratch, including new databases and users, when we were at mongo 7.0. In particular, the admin db.system.users have an userId field in preproduction, and not in production.
I tried, in preproduction, to drop that userId field:
```
[direct: mongos] admin> db.system.users.updateOne( { _id: 'test-db.test-user' }, { $unset:
{ userId: "" }[direct: mongos] admin> db.adminCommand({ invalidateUserCache: 1 })
```
All preproduction applications started failing with "Command failed with error 13 (Unauthorized)".
I let the applications in this failing state, and set back the userId as it was:
```
[direct: mongos] admin> db.system.users.updateOne( { _id: "test-db.test-user" }, { $set:
{ userId: UUID("7ea21ff8-d8a2-44f8-9c78-f1a6f914b590") }} )
[direct: mongos] admin> db.adminCommand({ invalidateUserCache: 1 })
```
The applications were still failing.
More tests showed that changing userId (either setting it, unsetting it, changing it) require a restart of all applications (or all mongos instances to force reconnections).
I suppose that manually setting a userId field in the production's system.users collection would trigger the same problem.
I also suppose deleting and recreating users in the production cluster would lead to downtime.
I don't think I missed a release note informing users of this potential issue, but most importantly, I find no information on how to avoid that problem the next time I try to upgrade my cluster from 8.0 to 8.2.
Thanks in advance,
Colin