[SERVER-62692] Unexpected reboot corrupted wiredTiger files Created: 17/Jan/22  Updated: 02/Jun/22  Resolved: 22/Feb/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ayman Mostafa Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File wiredTiger.7z    
Operating System: ALL
Participants:

 Description   

Hello,

There was an unexpectedly reboot on the server. And I don't have backups. and I tried to use "-repair" command but I got the same error that mentioned below, I also attached my wiredTiger files.

Windows server 2016

mongo version: v4.0.2

2022-01-17T13:55:32.432+0200 I CONTROL [main] Trying to start Windows service 'MongoDB'
2022-01-17T13:55:32.474+0200 I CONTROL [initandlisten] MongoDB starting : pid=11400 port=27017 dbpath=C:\data\db\ 64-bit host=CAI1-SV00075
2022-01-17T13:55:32.483+0200 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2022-01-17T13:55:32.484+0200 I CONTROL [initandlisten] db version v4.0.2
2022-01-17T13:55:32.496+0200 I CONTROL [initandlisten] git version: fc1573ba18aee42f97a3bb13b67af7d837826b47
2022-01-17T13:55:32.505+0200 I CONTROL [initandlisten] allocator: tcmalloc
2022-01-17T13:55:32.518+0200 I CONTROL [initandlisten] modules: none
2022-01-17T13:55:32.519+0200 I CONTROL [initandlisten] build environment:
2022-01-17T13:55:32.522+0200 I CONTROL [initandlisten] distmod: 2008plus-ssl
2022-01-17T13:55:32.529+0200 I CONTROL [initandlisten] distarch: x86_64
2022-01-17T13:55:32.533+0200 I CONTROL [initandlisten] target_arch: x86_64
2022-01-17T13:55:32.534+0200 I CONTROL [initandlisten] options: { service: true, systemLog:

{ destination: "file", path: "C:\Program Files\MongoDB\Server\4.0\loginstall.txt" }

}
2022-01-17T13:55:32.586+0200 I STORAGE [initandlisten] Detected data files in C:\data\db\ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2022-01-17T13:55:32.599+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3582M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2022-01-17T13:55:32.838+0200 E STORAGE [initandlisten] WiredTiger error (-31803) [1642420532:837859][11400:140721342077392], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle file read error: WT_NOTFOUND: item not found Raw: [1642420532:837859][11400:140721342077392], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle file read error: WT_NOTFOUND: item not found
2022-01-17T13:55:32.839+0200 E STORAGE [initandlisten] WiredTiger error (-31804) [1642420532:838416][11400:140721342077392], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1642420532:838416][11400:140721342077392], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic
2022-01-17T13:55:32.848+0200 F - [initandlisten] Fatal Assertion 50853 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 401
2022-01-17T13:55:32.863+0200 F - [initandlisten]

***aborting after fassert() failure

2022-01-17T13:56:06.575+0200 F - [initandlisten] Got signal: 22 (SIGABRT).
mongod.exe ...\src\mongo\util\stacktrace_windows.cpp(244) mongo::printStackTrace+0x67x
mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(182) mongo::`anonymous namespace'::printSignalAndBacktrace+0x116x
mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(238) mongo::`anonymous namespace'::abruptQuit+0x133x
ucrtbase.dll raise+0x487x
ucrtbase.dll abort+0x49x
mongod.exe ...\src\mongo\util\assert_util.cpp(145) mongo::fassertFailedWithLocation+0x385x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(401) mongo::`anonymous namespace'::mdb_handle_error_with_startup_suppression+0x503x
mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(277) __eventv+0x897x
mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(331) __wt_err_func+0x42x
mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(538) __wt_panic+0x69x
mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_turtle.c(335) __wt_turtle_read+0x766x
mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(304) __wt_metadata_search+0x207x
mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(48) __conn_dhandle_config_set+0x34x
mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(443) __wt_conn_dhandle_open+0x117x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(546) __wt_session_get_dhandle+0x278x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(538) __wt_session_get_dhandle+0x475x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(351) __wt_session_get_btree_ckpt+0x196x
mongod.exe ...\src\third_party\wiredtiger\src\cursor\cur_file.c(801) __wt_curfile_open+0x462x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(516) __session_open_cursor_int+0x697x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(591) __wt_open_cursor+0x96x
mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(115) __wt_metadata_cursor+0x153x
mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_api.c(2737) wiredtiger_open+0x3462x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(168) mongo::`anonymous namespace'::openWiredTiger+0x130x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(545) mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x2423x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(106) mongo::`anonymous namespace'::WiredTigerFactory::create+0x649x
mongod.exe ...\src\mongo\db\storage\storage_engine_init.cpp(164) mongo::initializeStorageEngine+0x2485x
mongod.exe ...\src\mongo\db\db.cpp(344) mongo::`anonymous namespace'::_initAndListen+0x1788x
mongod.exe ...\src\mongo\db\db.cpp(683) mongo::`anonymous namespace'::initAndListen+0x39x
mongod.exe ...\src\mongo\util\ntservice.cpp(560) mongo::ntservice::initService+0x83x
sechost.dll LsaLookupFreeMemory+0x434x
KERNEL32.DLL BaseThreadInitThunk+0x20x
2022-01-17T13:56:06.599+0200 F CONTROL [initandlisten] *** unhandled exception 0x0000000E at 0x00007FFC3D7B4F38, terminating
2022-01-17T13:56:06.600+0200 F CONTROL [initandlisten] *** stack trace for unhandled exception:
2022-01-17T13:56:06.917+0200 I - [initandlisten] KERNELBASE.dll RaiseException+0x104x
mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(239) mongo::`anonymous namespace'::abruptQuit+0x157x
ucrtbase.dll raise+0x487x
ucrtbase.dll abort+0x49x
mongod.exe ...\src\mongo\util\assert_util.cpp(145) mongo::fassertFailedWithLocation+0x385x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(401) mongo::`anonymous namespace'::mdb_handle_error_with_startup_suppression+0x503x
mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(277) __eventv+0x897x
mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(331) __wt_err_func+0x42x
mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(538) __wt_panic+0x69x
mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_turtle.c(335) __wt_turtle_read+0x766x
mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(304) __wt_metadata_search+0x207x
mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(48) __conn_dhandle_config_set+0x34x
mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(443) __wt_conn_dhandle_open+0x117x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(546) __wt_session_get_dhandle+0x278x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(538) __wt_session_get_dhandle+0x475x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(351) __wt_session_get_btree_ckpt+0x196x
mongod.exe ...\src\third_party\wiredtiger\src\cursor\cur_file.c(801) __wt_curfile_open+0x462x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(516) __session_open_cursor_int+0x697x
mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(591) __wt_open_cursor+0x96x
mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(115) __wt_metadata_cursor+0x153x
mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_api.c(2737) wiredtiger_open+0x3462x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(168) mongo::`anonymous namespace'::openWiredTiger+0x130x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(545) mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x2423x
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(106) mongo::`anonymous namespace'::WiredTigerFactory::create+0x649x
mongod.exe ...\src\mongo\db\storage\storage_engine_init.cpp(164) mongo::initializeStorageEngine+0x2485x
mongod.exe ...\src\mongo\db\db.cpp(344) mongo::`anonymous namespace'::_initAndListen+0x1788x
mongod.exe ...\src\mongo\db\db.cpp(683) mongo::`anonymous namespace'::initAndListen+0x39x
mongod.exe ...\src\mongo\util\ntservice.cpp(560) mongo::ntservice::initService+0x83x
sechost.dll LsaLookupFreeMemory+0x434x
KERNEL32.DLL BaseThreadInitThunk+0x20x
2022-01-17T13:56:06.926+0200 I CONTROL [initandlisten] writing minidump diagnostic file C:\Program Files\MongoDB\Server\4.2022-01-17T11-56-06.mdmp
2022-01-17T13:56:15.015+0200 F CONTROL [initandlisten] *** immediate exit due to unhandled exception



 Comments   
Comment by Edwin Zhou [ 22/Feb/22 ]

Hi ayman.mostafa@valeo.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Best,
Edwin

Comment by Edwin Zhou [ 14/Feb/22 ]

Hi ayman.mostafa@valeo.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please let us know if you've been able to perform --repair on your node using the latest version of MongoDB?

Best,
Edwin

Comment by Edwin Zhou [ 31/Jan/22 ]

Hi ayman.mostafa@valeo.com,

If you are working off a replica set, the ideal resolution is to perform a clean resync from an unaffected node.

Otherwise, first make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath. Please note that MongoDB 4.0.2 uses an outdated version of the --repair command. Please reattempt mongod --repair using the latest version of MongoDB.

In the event that a --repair operation is unsuccessful, then please also provide:

  • The logs leading up to the first occurrence of any issue
  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

Best,
Edwin

Generated at Thu Feb 08 05:55:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.