[SERVER-31611] Unknown FCVs to 3.6 must go through clean shutdown. Created: 17/Oct/17  Updated: 30/Oct/23  Resolved: 27/Oct/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: 3.6.0-rc2

Type: Bug Priority: Major - P3
Reporter: Ramon Fernandez Marina Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Storage 2017-10-23, Storage 2017-11-13
Participants:

 Description   

To simulate an upgrade from 3.2 to 3.4, I started a 3.4.9 node and set it to 3.2 compatibility mode:

2017-10-17T18:47:04.090-0400 I CONTROL  [initandlisten] MongoDB starting : pid=18532 port=27017 dbpath=/home/rf/tmp/BUILD-3850/1/data/db 64-bit host=tab
2017-10-17T18:47:04.090-0400 I CONTROL  [initandlisten] db version v3.4.9
...
2017-10-17T18:47:13.070-0400 I COMMAND  [conn5] setting featureCompatibilityVersion to 3.2

I then shut down this node, and tried to start it with 3.6.0-rc0, but mongod rightfully complained about the value of featureCompatibilityVersion:

2017-10-17T18:47:47.751-0400 I CONTROL  [initandlisten] MongoDB starting : pid=18608 port=27017 dbpath=/home/rf/tmp/BUILD-3850/1/data/db 64-bit host=tab
2017-10-17T18:47:47.751-0400 I CONTROL  [initandlisten] db version v3.6.0-rc0
...
2017-10-17T18:47:48.251-0400 F STORAGE  [initandlisten] BadValue: Invalid value for version, found 3.2, expected '3.6' or '3.4'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. See http://dochub.mongodb.org/core/3.6-feature-compatibility.
2017-10-17T18:47:48.251-0400 F -        [initandlisten] Fatal Assertion 40283 at src/mongo/db/db.cpp 411
2017-10-17T18:47:48.251-0400 F -        [initandlisten]
 
***aborting after fassert() failure

I then tried to go back to 3.4 to do a proper upgrade to 3.4, but looks like WiredTiger had already upgraded the file format:

2017-10-17T18:47:57.496-0400 I CONTROL  [initandlisten] MongoDB starting : pid=18650 port=27017 dbpath=/home/rf/tmp/BUILD-3850/1/data/db 64-bit host=tab
2017-10-17T18:47:57.496-0400 I CONTROL  [initandlisten] db version v3.4.9
...
2017-10-17T18:47:57.540-0400 E STORAGE  [initandlisten] WiredTiger error (-31802) [1508280477:540915][18650:0x7fc56c2e9d00], txn-recover: unsupported WiredTiger file version: this build  only supports major/minor versions up to 1/0,  and the file is version 2/0: WT_ERROR: non-specific WiredTiger error
2017-10-17T18:47:57.540-0400 E STORAGE  [initandlisten] WiredTiger error (0) [1508280477:540956][18650:0x7fc56c2e9d00], txn-recover: WiredTiger is unable to read the recovery log.
2017-10-17T18:47:57.540-0400 E STORAGE  [initandlisten] WiredTiger error (0) [1508280477:540968][18650:0x7fc56c2e9d00], txn-recover: This may be due to the log files being encrypted, being from an older version or due to corruption on disk
2017-10-17T18:47:57.540-0400 E STORAGE  [initandlisten] WiredTiger error (0) [1508280477:540979][18650:0x7fc56c2e9d00], txn-recover: You should confirm that you have opened the database with the correct options including all encryption and compression options
2017-10-17T18:47:57.540-0400 E STORAGE  [initandlisten] WiredTiger error (-31802) [1508280477:540990][18650:0x7fc56c2e9d00], txn-recover: Recovery failed: WT_ERROR: non-specific WiredTiger error
2017-10-17T18:47:57.545-0400 I -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 269
2017-10-17T18:47:57.545-0400 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2017-10-17T18:47:57.545-0400 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2017-10-17T18:47:57.545-0400 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2017-10-17T18:47:57.545-0400 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2017-10-17T18:47:57.546-0400 I CONTROL  [initandlisten] now exiting
2017-10-17T18:47:57.546-0400 I CONTROL  [initandlisten] shutting down with code:100



 Comments   
Comment by Githook User [ 27/Oct/17 ]

Author:

{'email': 'daniel.gottlieb@mongodb.com', 'name': 'Daniel Gottlieb', 'username': 'dgottlieb'}

Message: SERVER-31611: Perform a clean shutdown and downgrade on an interpreted FCV 3.2.

There are a few cases where a 3.6 binary will refuse to start up because it
concludes the datafiles belong to a 3.2 mongod. In those cases, the server
must perform a clean shutdown that ensures datafiles are downgraded to a
format that older binaries can understand.
Branch: master
https://github.com/mongodb/mongo/commit/09388643a69af34af0d123c7ccb814548454f35e

Comment by Susan LoVerso [ 18/Oct/17 ]

The FCV is in a WiredTiger table. Starting a 3.6 binary gives you 3.6 file formats until a downgrade is performed (on a clean shutdown). So daniel.gottlieb and I agree there are two changes needed.

  1. As milkie said, detecting this should go through the shutdown path. All downgrades to earlier file formats need a clean shutdown.
  2. The WiredTigerKVEngine::cleanShutdown code needs to loosen its conditions for needsDowngrade to anything < 3.6.
Comment by Eric Milkie [ 18/Oct/17 ]

To fix this, it might be as easy as not fasserting at db.cpp:411, but instead following the normal shutdown path, which would downgrade the data format back to 3.4 (presumably).

Generated at Thu Feb 08 04:27:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.