[SERVER-41449] WiredTiger error (-31804) Created: 01/Jun/19  Updated: 06/Jun/19  Resolved: 06/Jun/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.6.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zhu Zi Qing Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: wt-repair-success, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-41449_repair_attempt.tgz     File WiredTiger.turtle     File WiredTiger.wt     Text File mongo.log    
Operating System: ALL
Participants:

 Description   

Hi! My mongoDB can not  start after my computer suddenly loses power, and I have no idea how to fix it, when I tried to restart it, the error appears:

2019-06-01T14:26:02.393+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] MongoDB starting : pid=3768 port=27017 dbpath=D:\MongoDB\data 64-bit host=zzq-PC
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] db version v3.6.3
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1u-fips 22 Sep 2016
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] modules: none
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] build environment:
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] distmod: 2008plus-ssl
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] distarch: x86_64
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-06-01T14:26:04.250+0800 I CONTROL [initandlisten] options: { config: "C:\Program Files\MongoDB\Server\3.6\config\mongod.cfg", net:

{ bindIp: "0.0.0.0", port: 27017 }

, processManagement: { windowsService:

{ description: "Hahahaaa,MongoDB!", displayName: "MongoDB Service", serviceName: "MongoDB" }

}, storage: { dbPath: "D:\MongoDB\data", directoryPerDB: true, wiredTiger: { engineConfig:

{ cacheSizeGB: 4.0 }

} }, systemLog: { destination: "file", logAppend: true, path: "D:\MongoDB\logs\mongo.log" } }
2019-06-01T14:26:04.251+0800 W - [initandlisten] Detected unclean shutdown - D:\MongoDB\data\mongod.lock is not empty.
2019-06-01T14:26:04.251+0800 I - [initandlisten] Detected data files in D:\MongoDB\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-06-01T14:26:04.251+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-06-01T14:26:04.252+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-06-01T14:26:04.265+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559370364:265216][3768:2001878752], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_read, 291)
2019-06-01T14:26:04.265+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1559370364:265216][3768:2001878752], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-06-01T14:26:04.265+0800 F - [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2019-06-01T14:26:04.265+0800 F - [initandlisten]

***aborting after fassert() failure

2019-06-01T14:26:04.588+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\util\stacktrace_windows.cpp(244) mongo::printStackTrace+0x67x
2019-06-01T14:26:04.589+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(182) mongo::`anonymous namespace'::printSignalAndBacktrace+0x116x
2019-06-01T14:26:04.589+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(238) mongo::`anonymous namespace'::abruptQuit+0x133x
2019-06-01T14:26:04.589+0800 I CONTROL [initandlisten] ucrtbase.DLL raise+0x488x
2019-06-01T14:26:04.591+0800 I CONTROL [initandlisten] ucrtbase.DLL abort+0x49x
2019-06-01T14:26:04.591+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\util\assert_util.cpp(148) mongo::fassertFailedWithLocation+0x385x
2019-06-01T14:26:04.591+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361) mongo::`anonymous namespace'::mdb_handle_error+0x332x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(277) __eventv+0x897x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(330) __wt_err+0x50x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(578) __wt_illegal_value_func+0x126x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_turtle.c(291) __wt_turtle_read+0x719x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(281) __wt_metadata_search+0x663x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(48) __conn_dhandle_config_set+0x34x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(437) __wt_conn_dhandle_open+0x117x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(546) __wt_session_get_dhandle+0x278x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(538) __wt_session_get_dhandle+0x475x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(351) __wt_session_get_btree_ckpt+0x196x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\cursor\cur_file.c(692) __wt_curfile_open+0x441x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(389) __session_open_cursor_int+0x759x
2019-06-01T14:26:04.592+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(444) __wt_open_cursor+0x27x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(92) __wt_metadata_cursor+0x153x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_api.c(2676) wiredtiger_open+0x3050x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(409) mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x2490x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(91) mongo::`anonymous namespace'::WiredTigerFactory::create+0x296x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\service_context_d.cpp(202) mongo::ServiceContextMongoD::initializeGlobalStorageEngine+0x1362x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\db.cpp(745) mongo::`anonymous namespace'::_initAndListen+0x1716x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\db.cpp(1051) mongo::`anonymous namespace'::initAndListen+0x39x
2019-06-01T14:26:04.593+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\db.cpp(1404) mongo::mongoDbMain+0x414x
2019-06-01T14:26:04.594+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\dbmain.cpp(45) wmain+0x53x
2019-06-01T14:26:04.594+0800 I CONTROL [initandlisten] mongod.exe f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(253) __scrt_common_main_seh+0x285x
2019-06-01T14:26:04.594+0800 I CONTROL [initandlisten] kernel32.dll BaseThreadInitThunk+0x13x
2019-06-01T14:26:04.594+0800 F - [initandlisten] Got signal: 22 (SIGABRT).
2019-06-01T14:26:04.594+0800 I CONTROL [initandlisten] *** unhandled exception 0x0000000E at 0x000007FEFD7FA49D, terminating
2019-06-01T14:26:04.594+0800 I CONTROL [initandlisten] *** stack trace for unhandled exception:
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] KERNELBASE.dll RaiseException+0x61x
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(239) mongo::`anonymous namespace'::abruptQuit+0x157x
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] ucrtbase.DLL raise+0x488x
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] ucrtbase.DLL abort+0x49x
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\util\assert_util.cpp(148) mongo::fassertFailedWithLocation+0x385x
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361) mongo::`anonymous namespace'::mdb_handle_error+0x332x
2019-06-01T14:26:04.629+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(277) __eventv+0x897x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(330) __wt_err+0x50x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(578) __wt_illegal_value_func+0x126x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_turtle.c(291) __wt_turtle_read+0x719x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(281) __wt_metadata_search+0x663x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(48) __conn_dhandle_config_set+0x34x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(437) __wt_conn_dhandle_open+0x117x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(546) __wt_session_get_dhandle+0x278x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(538) __wt_session_get_dhandle+0x475x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_dhandle.c(351) __wt_session_get_btree_ckpt+0x196x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\cursor\cur_file.c(692) __wt_curfile_open+0x441x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(389) __session_open_cursor_int+0x759x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(444) __wt_open_cursor+0x27x
2019-06-01T14:26:04.630+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\meta\meta_table.c(92) __wt_metadata_cursor+0x153x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_api.c(2676) wiredtiger_open+0x3050x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(409) mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x2490x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(91) mongo::`anonymous namespace'::WiredTigerFactory::create+0x296x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\service_context_d.cpp(202) mongo::ServiceContextMongoD::initializeGlobalStorageEngine+0x1362x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\db.cpp(745) mongo::`anonymous namespace'::_initAndListen+0x1716x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\db.cpp(1051) mongo::`anonymous namespace'::initAndListen+0x39x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\db.cpp(1404) mongo::mongoDbMain+0x414x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe ...\src\mongo\db\dbmain.cpp(45) wmain+0x53x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] mongod.exe f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(253) __scrt_common_main_seh+0x285x
2019-06-01T14:26:04.631+0800 I CONTROL [initandlisten] kernel32.dll BaseThreadInitThunk+0x13x
2019-06-01T14:26:04.631+0800 I - [initandlisten]
2019-06-01T14:26:04.632+0800 I CONTROL [initandlisten] failed to open minidump file C:\Program Files\MongoDB\Server\3.2019-06-01T06-26-04.mdmp : 拒绝访问。
2019-06-01T14:26:04.632+0800 I CONTROL [initandlisten] *** immediate exit due to unhandled exception

 I have uploaded the WiredTiger related files in attachment.

 



 Comments   
Comment by Eric Sedor [ 06/Jun/19 ]

Thanks zzqxztc; we're going to close this ticket but will keep this information on hand.

Comment by Zhu Zi Qing [ 06/Jun/19 ]
  1. A description of the underlying storage mechanism in use, including details like:
    1. What file system and/or volume management system is in use? windows 10, ntfs
    2. Is data storage locally attached or network-attached? locally
    3. Are disks RAIDed and if so how? No,only one disk
    4. Are disks SSDs or HDDs? HDDs
  2. A description of your backup method, if any. using MongoDB "dump" command
  3. A description of your disks have been recently checked for integrity? Normal, i think it has no "bad track".
  4. A history of the deployment, including:
    1. a timeline of version changes 3.6.3->4.0.1
    2. a timeline of hardware upgrade/downgrade cycles or configuration changes ram 16GB->32GB
    3. a timeline of disaster recovery or backup restoration activities my PC lost power suddenly. 
    4. a timeline of any manipulations of the underlying database files, including copies or moves, and information about whether mongod was running during each manipulation. As above mentioned, i used "dump" command to produce backup files sometimes and only move or copy these files. 
Comment by Eric Sedor [ 05/Jun/19 ]

We know you are eager to get back to work but the following information may reveal the original cause of corruption and could be helpful to us in the future:

  1. A description of the underlying storage mechanism in use, including details like:
    1. What file system and/or volume management system is in use?
    2. Is data storage locally attached or network-attached?
    3. Are disks RAIDed and if so how?
    4. Are disks SSDs or HDDs?
  2. A description of your backup method, if any.
  3. A description of your disks have been recently checked for integrity?
  4. A history of the deployment, including:
    1. a timeline of version changes
    2. a timeline of hardware upgrade/downgrade cycles or configuration changes
    3. a timeline of disaster recovery or backup restoration activities
    4. a timeline of any manipulations of the underlying database files, including copies or moves, and information about whether mongod was running during each manipulation.
Comment by Eric Sedor [ 05/Jun/19 ]

We are glad to hear that, zzqxztc!

Comment by Zhu Zi Qing [ 05/Jun/19 ]

Thanks for your reply @eric.sedor.

I have put these wiredTiger.wt files in my  $dbpath directory, and I try to run MongoDB again on my computer. However, I still get the error message as

2019-06-04T11:05:21.232+0800 I CONTROL [initandlisten] MongoDB starting : pid=45596 port=27017 dbpath=Z:\study\data\data 64-bit host=zzq-LAB
2019-06-04T11:05:21.234+0800 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2019-06-04T11:05:21.234+0800 I CONTROL [initandlisten] db version v3.6.3
2019-06-04T11:05:21.234+0800 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1u-fips 22 Sep 2016
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] modules: none
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] build environment:
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] distmod: 2008plus-ssl
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] distarch: x86_64
2019-06-04T11:05:21.235+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-06-04T11:05:21.236+0800 I CONTROL [initandlisten] options: { config: "D:\Softwares\MongoDB\Server\3.6\mongod.cfg", net:

Unknown macro: { bindIp}

, processManagement: { windowsService:

Unknown macro: { description}

}, storage: { dbPath: "Z:\study\data\data", directoryPerDB: true, wiredTiger: { engineConfig:

Unknown macro: { cacheSizeGB}

} }, systemLog: { destination: "file", logAppend: true, path: "D:\Softwares\MongoDB\logs\mongo.log" } }
2019-06-04T11:05:21.250+0800 W - [initandlisten] Detected unclean shutdown - Z:\study\data\data\mongod.lock is not empty.
2019-06-04T11:05:21.256+0800 I - [initandlisten] Detected data files in Z:\study\data\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-06-04T11:05:21.257+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-06-04T11:05:21.266+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-06-04T11:05:29.743+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559617529:743099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 16150528: block header checksum of 1841193149 doesn't match expected checksum of 2489320858
2019-06-04T11:05:29.743+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559617529:743099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: {16150528, 4096, 2489320858}: (chunk 1 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 de 00 00 00 b6 00 00 00 01 00 00 00 00 10 00 00 bd 60 be 6d 01 00 00 00 e2 f5 1a 80 e3 f6 4f c0 cf c0 e3 f6 9f c0 cf c0 e3 fa 6f c0 cf c0 e4 01 02 6f c0 cf c0 e4 01 07 3f c0 cf c0 e4 01 0d 3f c0 cf c0 e4 01 1a 5f c0 cf c0 e4 01 25 1f c0 cf c0 e4 01 29 af c0 cf c0 e4 01 31 1f c0 cf c0 e4 01 34 af c0 cf c0 e4 01 37 8f c0 cf c0 e4 01 3d ff c0 cf c0 e4 01 41 4f c0 cf c0 e4 01 43 bf c0 cf c0 e4 01 48 4f c0 cf c0 e4 01 4a 6f c0 cf c0 e4 01 4c 2f c0 cf c0 e4 01 4c 8f c0 cf c0 e4 01 50 af c0 cf c0 e3 17 3f c0 cf c0 e3 17 7f c0 cf c0 e3 17 ef c0 cf c0 e3 40 6f c0 cf c0 e4 01 50 8f c0 df c0 e4 01 50 ef c0 e2 0f c0 80 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-06-04T11:05:29.743+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559617529:743099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: {16150528, 4096, 2489320858}: (chunk 2 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-06-04T11:05:29.744+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559617529:744099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: {16150528, 4096, 2489320858}: (chunk 3 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-06-04T11:05:29.744+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559617529:744099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: {16150528, 4096, 2489320858}: (chunk 4 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-06-04T11:05:29.744+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559617529:744099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: instagram_Media_comments/collection-7768-4208642328751036629.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2019-06-04T11:05:29.744+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1559617529:744099][45596:140703474012768], file:instagram_Media_comments/collection-7768-4208642328751036629.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-06-04T11:05:29.744+0800 F - [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2019-06-04T11:05:29.744+0800 F - [initandlisten]

***aborting after fassert() failure

2019-06-04T11:05:29.754+0800 F - [WTJournalFlusher] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64
2019-06-04T11:05:29.754+0800 F - [WTJournalFlusher]

***aborting after fassert() failure

I have tried to use the command "mongo -repair" to fix my MongoDB(version=3.6.3 ), But the mongoDB can not complete repair process.

You  mention that "MongoDB 4.0 offers improved repair capabilities". I use "repair" command again to fix MongoDB(version=4.0.1). It works!

MongoDB successfully starts fix itself automatically.  Though  i consider some data may lost due to db engine corruption.  My most data are recovery. Thanks for your technical work.

Comment by Eric Sedor [ 03/Jun/19 ]

zzqxztc, this error message leads us to suspect corruption in the wiredTiger.wt file. Please make a complete copy of the database's $dbpath directory to work off of and safeguard the current $dbpath.

The ideal resolution is to perform a clean resync from an unaffected node.

However, I've attached a repair attempt of the files you provided as SERVER-41449_repair_attempt.tgz. Please extract these files, replace them in your $dbpath, and let us know if it resolves the issue.

Please keep in mind that this repair effort may not be successful, and that diagnosing corruption issues requires significant information and effort.

To avoid a problem like in the future, it is our strong recommendation to:

  • Have a process in place to restore backups.
  • Schedule and perform regular checks of the integrity of your filesystems and disks.
  • Never manipulate the underlying database files in any way.
Generated at Thu Feb 08 04:57:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.