[SERVER-32994] Create a script to rebuild unique indexes in right format after a downgrade Created: 30/Jan/18  Updated: 29/Oct/23  Resolved: 06/Aug/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.0.2, 4.1.2

Type: Improvement Priority: Major - P3
Reporter: Neha Khatri Assignee: Luke Chen
Resolution: Fixed Votes: 0
Labels: nonnyc, storage-engines
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File script_rev1.js    
Issue Links:
Backports
Depends
depends on DOCS-11933 4.2 Downgrade consideration: Unique I... Closed
Documented
is documented by DOCS-11933 4.2 Downgrade consideration: Unique I... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.0
Sprint: Storage Non-NYC 2018-04-09, Storage Engines 2018-07-30, Storage Engines 2018-08-13
Participants:

 Description   

When a MongoDB 4.2 database is downgraded to FCV=4.0, the existing unique indexes in data format 11 and 12 should continue to operate in this format. Create a script that users can run that will identify and rebuild all of these indexes in the older format of 6 and 8.

Coordinate with docs team regarding publishing of this script.



 Comments   
Comment by Luke Chen [ 21/Oct/18 ]

Hi kay.kim, The docs changes in the link look good to me. To answer your questions, yes the script will rebuild all unique indexes, no matter they are initially created with 4.0 or 4.2 binary. There could be ways to tell which ones are with 4.0-format that could be excluded from the recreation process (like what milkie mentioned above), but it's likely not trivial/straightforward to implement and communicate to users. 

Comment by Githook User [ 06/Aug/18 ]

Author:

{'username': 'lukech', 'name': 'Luke Chen', 'email': 'luke.chen@mongodb.com'}

Message: SERVER-32994 Update reference link for Unique Index 4.2 downgrading error

(cherry picked from commit e49247580c76ccdd4013ff0357e03d0b1c95cbb6)
Branch: v4.0
https://github.com/mongodb/mongo/commit/349140f8d0a5457524437a101b923853a8b0ee9c

Comment by Luke Chen [ 06/Aug/18 ]

alexander.gorrod and kay.kim, I just pushed the change of updating error message with the new dochub.mongodb.org/core/4.2-downgrade-index link. Will backport the change to v4.0 to get the desired effect - error out by pointing to the new link while starting data files with 4.2-format unique indexes using a 4.0 binary.  

Comment by Githook User [ 06/Aug/18 ]

Author:

{'name': 'Luke Chen', 'email': 'luke.chen@mongodb.com', 'username': 'lukech'}

Message: SERVER-32994 Update reference link for Unique Index 4.2 downgrading error
Branch: master
https://github.com/mongodb/mongo/commit/e49247580c76ccdd4013ff0357e03d0b1c95cbb6

Comment by Alexander Gorrod [ 02/Aug/18 ]

Thanks kay.kim - that's very useful!

luke.chen could you work on the change to add that message into MongoDB?

Comment by Kay Kim (Inactive) [ 02/Aug/18 ]

alexander.gorrod Will do.  In the meantime, I can give you a dochub link  dochub.mongodb.org/core/4.2-downgrade-index  which you can add to MongoDB and you all can finish up on your side since you don't have to worry about where it points.

That link just redirects to whatever page we tell it to. So until the upcoming downgrade page exists, I can have it point to some page (like the unique index page or whatnot). When the 4.2 downgrade page/section exists, I'll have it point to the correct page but you don't need to change the link on your side. The dochub link remains the same, we on our side just tell it to point somewhere else.

I haven't created the dochub.mongodb.org/core/4.2-downgrade-index  link yet – so it will 404. But let me know if you want to proceed like that.

Comment by Alexander Gorrod [ 02/Aug/18 ]

luke.chen I just re-opened this ticket, and put it on the backlog.

kay.kim it would be great if you can let us know when the docs are done - so we can add a message to MongoDB that includes a link to the documentation.

Comment by Kay Kim (Inactive) [ 01/Aug/18 ]

luke.chen – your script looks great. Should be enough. Thanks!

Comment by Luke Chen [ 01/Aug/18 ]

Hi kay.kim, I got this ticket that may need documentation update regarding 4.2 --> 4.0 downgrade. Could you please have a check and let me know if you need more information? Thanks.

A new set of internal formats (i.e. 11 and 12) will be implemented for unique indexes in 4.2 (via PM-934), replacing the old set of internal unique index formats (i.e. 6 and 8) of 4.0 (and prior releases). The corresponding external unique index versions are v:1 and v:2 which are visible and customisable to MongoDB users. When users try to downgrade mongod from 4.2 to 4.0, after setting FCV to 4.0, all unique indexes (both v:1 and v:2) need to be dropped and recreated with the same external version, so that all recreated unique indexes have the old set of formats (i.e. 6 and 8) that 4.0 is able to handle. Below script (code snippet) can be used to drop and recreate unique indexes during 4.2 --> 4.0 downgrading. It had been verified in our test environment with the co-existence of unique index external/internal version permutations (i.e. v:1/6, v:2/8, v:1/11, v:2/12) and a 4.2 --> 4.0 downgrade procedure. 

// A script to rebuild unique indexes in right format after a downgrade.
// This is a helper script meant to be used while downgrading from 4.2 to 4.0
// Steps:
// 1. Set FCV to 4.0 by running "db.adminCommand( { setFeatureCompatibilityVersion: <version> } )"
// 2. Run this script to drop unique indexes for both MongoDB versions - "v: 1" and "v: 2"
//    (correspond to internal format versions 11 & 12 or 6 & 8, depending on the FCV setting),
//    and recreate them with the same MongoDB versions - "v: 1: and "v: 2" (correspond to internal format versions 6 & 8).
//    The aim is to have internal format version lined up to 6 & 8 for all unique indexes that 4.0 supports.
// 3. Run with the downgraded 4.0 binary
 
// Obtain a list of unique indexes for both v:1 and v:2
var unique_idx_v1 = [];
var unique_idx_v2 = [];
db.adminCommand("listDatabases").databases.forEach(function(d){
   let mdb = db.getSiblingDB(d.name);
   mdb.getCollectionInfos().forEach(function(c){
      let currentCollection = mdb.getCollection(c.name);
      currentCollection.getIndexes().forEach(function(i){
         if (i.unique){
            if (i.v === 1) {
               unique_idx_v1.push(i);
            }
            else {
               unique_idx_v2.push(i);
            }
            return;
         }
      });
   });
});
 
printjson(unique_idx_v1);
printjson(unique_idx_v2);
 
// Drop and recreate all v:1 indexes
for (let idx of unique_idx_v1) {
   let [dbName, collName] = idx.ns.split(".");
   let res = db.getSiblingDB(dbName).runCommand({dropIndexes: collName, index: idx.name});
   assert.commandWorked(res);
   res = db.getSiblingDB(dbName).runCommand({
      createIndexes: collName,
      indexes: [{"key": idx.key, "name": idx.name, "unique": true, "v": 1}]
   });
   assert.commandWorked(res);
}
 
// Drop and recreate all v:2 indexes
for (let idx of unique_idx_v2) {
   let [dbName, collName] = idx.ns.split(".");
   let res = db.getSiblingDB(dbName).runCommand({dropIndexes: collName, index: idx.name});
   assert.commandWorked(res);
   res = db.getSiblingDB(dbName).runCommand({
      createIndexes: collName,
      indexes: [{"key": idx.key, "name": idx.name, "unique": true, "v": 2}]
   });
   assert.commandWorked(res);
}

Comment by Alexander Gorrod [ 30/Jul/18 ]

An example of such a script and associated documentation from a previous backward breaking change is:
https://docs.mongodb.com/manual/release-notes/3.4-downgrade/

Comment by Alexander Gorrod [ 07/May/18 ]

Guess is 5 days to complete.

Generated at Thu Feb 08 04:31:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.