[SERVER-55313] Unable to start standalone mongod after downgrade from 4.4 -> 4.2 -> 4.0 Created: 18/Mar/21  Updated: 08/Mar/22  Resolved: 24/Mar/21

Status: Closed
Project: Core Server
Component/s: Upgrade/Downgrade
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Cayden Dagnall Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHEL 8


Operating System: ALL
Steps To Reproduce:
  1.  Create a new database with version 4.4
  2.  Create user
  3.  Set FCV to 4.2
  4. Shutdown 4.4
  5. Start 4.2
  6. Set fcv to 4.0
  7. Run commands:
  8. db.adminCommand("listDatabases").databases.forEach(function(d){
       let mdb = db.getSiblingDB(d.name);   mdb.getCollectionInfos( { type: "collection" } ).forEach(function(c){
          let currentCollection = mdb.getCollection(c.name);      currentCollection.getIndexes().forEach(function(idx){
             if (idx.unique){
                print("Dropping and recreating the following index:" + tojson(idx))            assert.commandWorked(mdb.runCommand({dropIndexes: c.name, index: idx.name}));            let res = mdb.runCommand({ createIndexes: c.name, indexes: [idx] });
                if (res.ok !== 1)
                   assert.commandWorked(res);
             }
          });
       });
    });
     
    db.getSiblingDB("admin").getCollection("system.users").dropIndex("user_1_db_1")
     
    db.adminCommand( { shutdown: 1 } )

  1. Start 4.0
Participants:
Case:

 Description   

When attempting to down a standalone mongod from 4.4 -> 4.2 -> 4.0, the 4.0 mongod fails to start.

Our specific use case is upgrade from 3.6 -> 4.4 and then downgrade from 4.4 -> 3.6, but this error also happens without upgrading an existing database and simple starting from 4.4.

Downgrade from 4.2 -> 4.0 works when starting at 4.2.  Downgrade also works when performing a mongodump/mongorestore between 4.2 -> 4.0 when starting from 4.4.

 

2021-03-18T18:51:20.076+0000 I STORAGE [initandlisten] Detected data files in /mongo/db/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2021-03-18T18:51:20.090+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3467M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-03-18T18:51:21.332+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1616093481:332154][1553676:0x7f4f2d5fab40], connection: __log_open_verify, 985: unsupported WiredTiger file version: this build only supports versions up to 3, and the file is version 4: WT_ERROR: non-specific WiredTiger error Raw: [1616093481:332154][1553676:0x7f4f2d5fab40], connection: __log_open_verify, 985: unsupported WiredTiger file version: this build only supports versions up to 3, and the file is version 4: WT_ERROR: non-specific WiredTiger error
2021-03-18T18:51:21.373+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1616093481:373170][1553676:0x7f4f2d5fab40], connection: __log_open_verify, 985: unsupported WiredTiger file version: this build only supports versions up to 3, and the file is version 4: WT_ERROR: non-specific WiredTiger error Raw: [1616093481:373170][1553676:0x7f4f2d5fab40], connection: __log_open_verify, 985: unsupported WiredTiger file version: this build only supports versions up to 3, and the file is version 4: WT_ERROR: non-specific WiredTiger error
2021-03-18T18:51:21.419+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1616093481:419674][1553676:0x7f4f2d5fab40], connection: __log_open_verify, 985: unsupported WiredTiger file version: this build only supports versions up to 3, and the file is version 4: WT_ERROR: non-specific WiredTiger error Raw: [1616093481:419674][1553676:0x7f4f2d5fab40], connection: __log_open_verify, 985: unsupported WiredTiger file version: this build only supports versions up to 3, and the file is version 4: WT_ERROR: non-specific WiredTiger error
2021-03-18T18:51:21.426+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2021-03-18T18:51:21.427+0000 F STORAGE [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error
2021-03-18T18:51:21.427+0000 F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 711
2021-03-18T18:51:21.429+0000 F - [initandlisten] \n\n***aborting after fassert() failure\n\n



 Comments   
Comment by Eric Sedor [ 24/Mar/21 ]

Thanks cayden.dagnall@leidos.com. In reviewing the documentation I've requested some clarifications in DOCS-14312. Feel free to comment there if you have additional suggestions that you think may have helped.

I'll close this ticket now. Be well!

Comment by Cayden Dagnall [ 18/Mar/21 ]

Thanks, I missed that in the documentation.  

Comment by Daniel Gottlieb (Inactive) [ 18/Mar/21 ]

Once a MongoDB 4.4 binary starts up on data files, those data files can never be used by a MongoDB version 4.2.5 or earlier.

If you need to go from MongoDB 4.2.6+ to an earlier version, you'll have to perform an initial sync (for replica sets) or a mongodump + mongorestore (for standalones) into a 4.2 binary on a fresh dbpath. Then continuing your downgrade procedure should work.

See the documentation on the downgrade floor here.

Comment by Cayden Dagnall [ 18/Mar/21 ]

Versions reproduced on:

4.4.1

4.2.12/4.2.13-rc2

4.0.22/4.0.23

Generated at Thu Feb 08 05:36:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.