[SERVER-54725] Failure on downgrade from 4.4 to 4.2 after unclean shutdown Created: 23/Feb/21  Updated: 03/Mar/21  Resolved: 03/Mar/21

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 4.4.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Orgad Shaneh Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

docker run -d --name mon --rm -v mongodb:/data/db mongo:4.2
sleep 5
docker stop mon
docker run -d --name mon --rm -v mongodb:/data/db mongo:4.4
sleep 5
docker kill mon
docker run --name mon --rm -v mongodb:/data/db mongo:4.2

Participants:

 Description   

If you upgrade from 4.2 to 4.4, then kill 4.4 and run 4.2 again, it fails to start and cannot recover.
Output:

2021-02-23T14:33:58.753+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2021-02-23T14:33:58.756+0000 W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
2021-02-23T14:33:58.757+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=c30e2eb88c85
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] db version v4.2.12
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] git version: 5593fd8e33b60c75802edab304e23998fa0ce8a5
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] modules: none
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] build environment:
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2021-02-23T14:33:58.758+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: "*" } }
2021-02-23T14:33:58.761+0000 W  STORAGE  [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2021-02-23T14:33:58.761+0000 I  STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2021-02-23T14:33:58.761+0000 W  STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2021-02-23T14:33:58.761+0000 I  STORAGE  [initandlisten]
2021-02-23T14:33:58.761+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2021-02-23T14:33:58.761+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2021-02-23T14:33:58.761+0000 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3468M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2021-02-23T14:33:59.424+0000 E  STORAGE  [initandlisten] WiredTiger error (-31802) [1614090839:423987][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1614090839:423987][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error
2021-02-23T14:34:00.758+0000 E  STORAGE  [initandlisten] WiredTiger error (-31802) [1614090840:758579][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1614090840:758579][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error
2021-02-23T14:34:00.774+0000 E  STORAGE  [initandlisten] WiredTiger error (-31802) [1614090840:774796][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1614090840:774796][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error
2021-02-23T14:34:00.786+0000 E  STORAGE  [initandlisten] WiredTiger error (-31802) [1614090840:786545][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1614090840:786545][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error
2021-02-23T14:34:00.801+0000 E  STORAGE  [initandlisten] WiredTiger error (-31802) [1614090840:801601][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1614090840:801601][1:0x7fb808665b00], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error
2021-02-23T14:34:00.808+0000 W  STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
2021-02-23T14:34:00.808+0000 F  STORAGE  [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error
2021-02-23T14:34:00.808+0000 F  -        [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 925
2021-02-23T14:34:00.808+0000 F  -        [initandlisten] \n\n***aborting after fassert() failure\n\n
 



 Comments   
Comment by Eric Sedor [ 03/Mar/21 ]

I agree it's not clear that a clean shutdown is required and have opened DOCS-14264 to improve this. Thank you orgads@gmail.com!

Comment by Orgad Shaneh [ 02/Mar/21 ]

Hi,

The downgrade instructions you linked are for replica set. On the standalone downgrade instructions it is not clear that a clean shutdown is mandatory for downgrading.

In our case, we faced this problem because we have a customized docker image with an alternative entrypoint that executes the default one as a separate process. For some reason, the new process doesn't receive SIGSTOP or whatever signal docker sends on stop, so it didn't terminate gracefully.

We solved it by calling the default entrypoint with exec, to reuse PID 1. This way the process receives the signal and shuts down cleanly.

But I wasn't sure if this behavior is expected, so I reported it.

If this is expected, you should emphasize it in the documentation.

Thanks!

Comment by Eric Sedor [ 02/Mar/21 ]

Hi orgads@gmail.com,

Thanks for your patience. The instructions to perform a clean shutdown are included in downgrade instructions because a clean shutdown process ensures a WiredTiger file version that MongoDB 4.2 supports. You should be able to restart on 4.4, ensure a clean shutdown, and then restart on 4.2.

If that's not the case, can you let us know?

Eric

Generated at Thu Feb 08 05:34:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.