[SERVER-78462] Upgrade from 4.4 -> 5.0 -> 6.0 results in inconsitent config.databases information Created: 26/Jun/23  Updated: 27/Oct/23  Resolved: 25/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Vinicius Grippa Assignee: Edwin Zhou
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Triage
Operating System: ALL
Participants:

 Description   

This seems to be related to the following:
SERVER-68511

After upgrading from 4.4.9 -> 5.0.12 -> 6.0.1, the config.databases collections is not in the expected state, and this can lead to errors.

See the following outputs:

5.0.12 FCV 4.4
mongos> db.getSiblingDB("config").getCollection("databases").find()
 
{ "_id" : "percona", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("064c5edc-6f5b-4e75-b47c-b58ec6ef2b02"), "lastMod" : 1 } }
{ "_id" : "percona10", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("cf5062e8-5264-4b02-9ac8-a42ce4bc9158"), "lastMod" : 1 } }
{ "_id" : "percona9", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("e9d64e96-b0a4-48c5-91bd-c04e56afc131"), "lastMod" : 1 } }
{ "_id" : "percona8", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("154f030b-0e46-4a98-9a7d-07395232604d"), "lastMod" : 1 } }
{ "_id" : "percona7", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("5df7095f-d14b-4460-95ff-8f06d61108bf"), "lastMod" : 1 } }
{ "_id" : "percona6", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("64d923a2-ca80-4d5c-8464-883d5373fb3b"), "lastMod" : 1 } }
{ "_id" : "percona5", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("b09adf8b-2367-428e-8d4b-f7a1c1e76b52"), "lastMod" : 1 } }
{ "_id" : "percona4", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("fbad9f30-7153-45c1-b85b-8c582904f5f8"), "lastMod" : 1 } }
{ "_id" : "percona3", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("34707ac1-e7d0-42a9-aa19-6b77ba73e590"), "lastMod" : 1 } }
{ "_id" : "percona2", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("d7cc77c3-fe3d-48ef-aa1b-d388b93db7ea"), "lastMod" : 1 } }
{ "_id" : "percona1", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("15bc26e3-1f98-4282-bc25-e0bb9ba2e060"), "lastMod" : 1 } }

And:

 

5.0.12 FCV 5.0
mongos> db.getSiblingDB("config").getCollection("databases").find()
{ "_id" : "percona", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("064c5edc-6f5b-4e75-b47c-b58ec6ef2b02"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 1) } }
{ "_id" : "percona10", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("cf5062e8-5264-4b02-9ac8-a42ce4bc9158"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 2) } }
{ "_id" : "percona9", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("e9d64e96-b0a4-48c5-91bd-c04e56afc131"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 3) } }
{ "_id" : "percona8", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("154f030b-0e46-4a98-9a7d-07395232604d"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 4) } }
{ "_id" : "percona7", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("5df7095f-d14b-4460-95ff-8f06d61108bf"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 5) } }
{ "_id" : "percona6", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("64d923a2-ca80-4d5c-8464-883d5373fb3b"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 6) } }
{ "_id" : "percona5", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("b09adf8b-2367-428e-8d4b-f7a1c1e76b52"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 7) } }
{ "_id" : "percona4", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("fbad9f30-7153-45c1-b85b-8c582904f5f8"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 8) } }
{ "_id" : "percona3", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("34707ac1-e7d0-42a9-aa19-6b77ba73e590"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 9) } }
{ "_id" : "percona2", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("d7cc77c3-fe3d-48ef-aa1b-d388b93db7ea"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 10) } }
{ "_id" : "percona1", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("15bc26e3-1f98-4282-bc25-e0bb9ba2e060"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 11) } } 

Lastly:

 

 

6.0.1  FCV 6.0
mongos> db.getSiblingDB("config").getCollection("databases").find()
{ "_id" : "percona", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("064c5edc-6f5b-4e75-b47c-b58ec6ef2b02"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 1) } }
{ "_id" : "percona10", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("cf5062e8-5264-4b02-9ac8-a42ce4bc9158"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 2) } }
{ "_id" : "percona9", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("e9d64e96-b0a4-48c5-91bd-c04e56afc131"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 3) } }
{ "_id" : "percona8", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("154f030b-0e46-4a98-9a7d-07395232604d"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 4) } }
{ "_id" : "percona7", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("5df7095f-d14b-4460-95ff-8f06d61108bf"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 5) } }
{ "_id" : "percona6", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("64d923a2-ca80-4d5c-8464-883d5373fb3b"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 6) } }
{ "_id" : "percona5", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("b09adf8b-2367-428e-8d4b-f7a1c1e76b52"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 7) } }
{ "_id" : "percona4", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("fbad9f30-7153-45c1-b85b-8c582904f5f8"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 8) } }
{ "_id" : "percona3", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("34707ac1-e7d0-42a9-aa19-6b77ba73e590"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 9) } }
{ "_id" : "percona2", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("d7cc77c3-fe3d-48ef-aa1b-d388b93db7ea"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 10) } }
{ "_id" : "percona1", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("15bc26e3-1f98-4282-bc25-e0bb9ba2e060"), "lastMod" : 1, "timestamp" : Timestamp(1687810800, 11) } } 

 

The only way to fix this is to follow the workaround suggested in SERVER-68511.

 

Once we apply the update, we have the following:

mongos> db.getSiblingDB("config").getCollection("databases").find()
{ "_id" : "percona", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("064c5edc-6f5b-4e75-b47c-b58ec6ef2b02"), "timestamp" : Timestamp(1687810800, 1), "lastMod" : 1 } }
{ "_id" : "percona10", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("cf5062e8-5264-4b02-9ac8-a42ce4bc9158"), "timestamp" : Timestamp(1687810800, 2), "lastMod" : 1 } }
{ "_id" : "percona9", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("e9d64e96-b0a4-48c5-91bd-c04e56afc131"), "timestamp" : Timestamp(1687810800, 3), "lastMod" : 1 } }
{ "_id" : "percona8", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("154f030b-0e46-4a98-9a7d-07395232604d"), "timestamp" : Timestamp(1687810800, 4), "lastMod" : 1 } }
{ "_id" : "percona7", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("5df7095f-d14b-4460-95ff-8f06d61108bf"), "timestamp" : Timestamp(1687810800, 5), "lastMod" : 1 } }
{ "_id" : "percona6", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("64d923a2-ca80-4d5c-8464-883d5373fb3b"), "timestamp" : Timestamp(1687810800, 6), "lastMod" : 1 } }
{ "_id" : "percona5", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("b09adf8b-2367-428e-8d4b-f7a1c1e76b52"), "timestamp" : Timestamp(1687810800, 7), "lastMod" : 1 } }
{ "_id" : "percona4", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("fbad9f30-7153-45c1-b85b-8c582904f5f8"), "timestamp" : Timestamp(1687810800, 8), "lastMod" : 1 } }
{ "_id" : "percona3", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("34707ac1-e7d0-42a9-aa19-6b77ba73e590"), "timestamp" : Timestamp(1687810800, 9), "lastMod" : 1 } }
{ "_id" : "percona2", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("d7cc77c3-fe3d-48ef-aa1b-d388b93db7ea"), "timestamp" : Timestamp(1687810800, 10), "lastMod" : 1 } }
{ "_id" : "percona1", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("15bc26e3-1f98-4282-bc25-e0bb9ba2e060"), "timestamp" : Timestamp(1687810800, 11), "lastMod" : 1 } } 

The previous Jira ticket suggests its been fixed when using the proper binaries, but that does not seem true.

 



 Comments   
Comment by Edwin Zhou [ 05/Sep/23 ]

Hi vgrippa@gmail.com,

Thank you for your patience while I look further into this issue.

Per our Sharding team, our MongoDB drivers do not access config.databases, and only the mongos and mongod processes are accessing that collection for internal metadata management.

SERVER-68511 improves the movePrimary command to be independent of the field ordering. If your application is depending on the ordering of internal metadata collections, your application has the same bug reported in SERVER-68511.

Please be advised that users must never rely on field ordering for internal metadata collections, and if you do choose to read or write to those collections, we advise you do so independent of field ordering. Otherwise, you will need to reorder the fields using the mitigation described in SERVER-68511.

I will now close this as works as designed. If you have further questions, we'd like to encourage you to ask our community for help by posting on the MongoDB Developer Community Forums.

Kind regards,
Edwin

Comment by Vinicius Grippa [ 25/Jul/23 ]

Hi edwin.zhou@mongodb.com,

 

Ok, that explains. I expected the upgrade to reorder the collection entirely once the upgrade was done. Still, we might have a driver using ordering fields and they would end up failing. It would be nice if Mongo could do the proper fix (reorder in this case) on his own.

 

Otherwise, it would be more secure to perform a logical export/import each time a new major version is released (which is a very tedious process).

 

Comment by Edwin Zhou [ 25/Jul/23 ]

Hi vgrippa@gmail.com,

Thank you for your submission and providing a clear output of config.databases across versions. To clarify SERVER-68511, this ticket addresses the movePrimary command causing sharding metadata to be inconsistent if the database was created in 4.4, as movePrimary was sensitive to the ordering of certain fields in the config.databases document. SERVER-68511 fixes the fragility that movePrimary had on the ordering of the fields.

The output that you provided seems to have expected behavior; upgrading FCV from 4.4 to 5.0 will append the timestamp field to the document. The workaround you applied reorders the lastMod and timestamp field to allow movePrimary to work on MongoDB <=5.0.10. After SERVER-68511, movePrimary is no longer dependent on field ordering.

Does that explain this behavior?

Kind regards,
Edwin

Generated at Thu Feb 08 06:38:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.