[SERVER-40458] Unexpected server shutdown and corrupted data Created: 03/Apr/19  Updated: 16/Apr/19  Resolved: 16/Apr/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.20
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Raivis Ginters Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: wtc, wtru
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-40458_repair_attempt.tgz     File WiredTiger.turtle     File WiredTiger.wt     File collection-2--9014212594577530165.wt    
Participants:

 Description   

Hello!
We had unexpected server shutdown and it made our database corrupt, we wanted to be sure if it is possible to get back at least some data out of corruption?
We had no luck with  --repair and other commands this far.
We are using mongodb v.3.4.2, here is our log output, i have replaced db path and my hostname with ***** -

2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] MongoDB starting : pid=931 port=27017 dbpath=/*****/ 64-bit host=*****
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] db version v3.4.2
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] allocator: tcmalloc
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] modules: none
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] build environment:
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] distmod: ubuntu1604
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] distarch: x86_64
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] target_arch: x86_64
2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] options: { net: { port: 27017 }, replication: { oplogSizeMB: 128, replSet: "rs0" }, storage: { dbPath: "/*****/", mmapv1: { smallFiles: true } } }
2019-04-03T14:24:25.632+0300 I - [initandlisten] Detected data files in /*****/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-04-03T14:24:25.632+0300 I STORAGE [initandlisten]
2019-04-03T14:24:25.632+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-04-03T14:24:25.632+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-04-03T14:24:25.633+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7453M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2019-04-03T14:24:25.703+0300 E STORAGE [initandlisten] WiredTiger error (0) [1554290665:703634][931:0x7fe3ea5a9cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 2396726276 doesn't match expected checksum of 1875689742
2019-04-03T14:24:25.703+0300 E STORAGE [initandlisten] WiredTiger error (0) [1554290665:703692][931:0x7fe3ea5a9cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2019-04-03T14:24:25.703+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1554290665:703699][931:0x7fe3ea5a9cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-04-03T14:24:25.703+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-04-03T14:24:25.703+0300 I - [initandlisten]
***aborting after fassert() failure
 
2019-04-03T14:24:25.724+0300 F - [initandlisten] Got signal: 6 (Aborted).
0x5652eae43b81 0x5652eae42c79 0x5652eae4315d 0x7fe3e91cd390 0x7fe3e8e28428 0x7fe3e8e2a02a 0x5652ea0d2293 0x5652eab536c6 0x5652ea0dc505 0x5652ea0dc5f9 0x5652ea0dc851 0x5652eb7442bf 0x5652eb74480b 0x5652eb740f1d 0x5652eb7459e7 0x5652eb763126 0x5652eb79988b 0x5652eb82668d 0x5652eb826d88 0x5652eb82729c 0x5652eb7a9681 0x5652eb81c0c0 0x5652eb7e653e 0x5652eb7e662b 0x5652eb795d13 0x5652eab36cf6 0x5652eab2fb8c 0x5652eaa1f3d0 0x5652ea0bdb7e 0x5652ea0ddc3f 0x7fe3e8e13830 0x5652ea13b979
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5652E98C1000","o":"1582B81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5652E98C1000","o":"1581C79"},{"b":"5652E98C1000","o":"158215D"},{"b":"7FE3E91BC000","o":"11390"},{"b":"7FE3E8DF3000","o":"35428","s":"gsignal"},{"b":"7FE3E8DF3000","o":"3702A","s":"abort"},{"b":"5652E98C1000","o":"811293","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5652E98C1000","o":"12926C6"},{"b":"5652E98C1000","o":"81B505"},{"b":"5652E98C1000","o":"81B5F9","s":"__wt_err"},{"b":"5652E98C1000","o":"81B851","s":"__wt_panic"},{"b":"5652E98C1000","o":"1E832BF"},{"b":"5652E98C1000","o":"1E8380B"},{"b":"5652E98C1000","o":"1E7FF1D"},{"b":"5652E98C1000","o":"1E849E7"},{"b":"5652E98C1000","o":"1EA2126"},{"b":"5652E98C1000","o":"1ED888B"},{"b":"5652E98C1000","o":"1F6568D"},{"b":"5652E98C1000","o":"1F65D88"},{"b":"5652E98C1000","o":"1F6629C"},{"b":"5652E98C1000","o":"1EE8681"},{"b":"5652E98C1000","o":"1F5B0C0"},{"b":"5652E98C1000","o":"1F2553E"},{"b":"5652E98C1000","o":"1F2562B"},{"b":"5652E98C1000","o":"1ED4D13","s":"wiredtiger_open"},{"b":"5652E98C1000","o":"1275CF6","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5652E98C1000","o":"126EB8C"},{"b":"5652E98C1000","o":"115E3D0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5652E98C1000","o":"7FCB7E"},{"b":"5652E98C1000","o":"81CC3F","s":"main"},{"b":"7FE3E8DF3000","o":"20830","s":"__libc_start_main"},{"b":"5652E98C1000","o":"87A979","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-53-generic", "version" : "#74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "5652E98C1000", "elfType" : 3, "buildId" : "5369A3377E676C284ECBB17624E909071B0846FC" }, { "b" : "7FFF7FBE5000", "elfType" : 3, "buildId" : "E3616EB2EA9FD27AC9DF53212C1C57A8A091F03D" }, { "b" : "7FE3EA148000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "99E596AA738A74440C0300BFF1086C8C15455279" }, { "b" : "7FE3E9D04000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3F3A35F9ABBE347B08E59845A1FC8EB5AE226797" }, { "b" : "7FE3E9AFC000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "0DBB8C21FC5D977098CA718BA2BFD6C4C21172E9" }, { "b" : "7FE3E98F8000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C0C5B7F18348654040534B050B110D32A19EA38D" }, { "b" : "7FE3E95EF000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "05451CB4D66C321691F64F253880B7CE5B8812A6" }, { "b" : "7FE3E93D9000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FE3E91BC000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "84538E3C6CFCD5D4E3C0D2B6C3373F802915A498" }, { "b" : "7FE3E8DF3000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CBFA941A8EB7A11E4F90E81B66FCD5A820995D7C" }, { "b" : "7FE3EA3B1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A7D5A820B802049276B1FC26C8E845A3E194EB6B" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5652eae43b81]
 mongod(+0x1581C79) [0x5652eae42c79]
 mongod(+0x158215D) [0x5652eae4315d]
 libpthread.so.0(+0x11390) [0x7fe3e91cd390]
 libc.so.6(gsignal+0x38) [0x7fe3e8e28428]
 libc.so.6(abort+0x16A) [0x7fe3e8e2a02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5652ea0d2293]
 mongod(+0x12926C6) [0x5652eab536c6]
 mongod(+0x81B505) [0x5652ea0dc505]
 mongod(__wt_err+0x9D) [0x5652ea0dc5f9]
 mongod(__wt_panic+0x24) [0x5652ea0dc851]
 mongod(+0x1E832BF) [0x5652eb7442bf]
 mongod(+0x1E8380B) [0x5652eb74480b]
 mongod(+0x1E7FF1D) [0x5652eb740f1d]
 mongod(+0x1E849E7) [0x5652eb7459e7]
 mongod(+0x1EA2126) [0x5652eb763126]
 mongod(+0x1ED888B) [0x5652eb79988b]
 mongod(+0x1F6568D) [0x5652eb82668d]
 mongod(+0x1F65D88) [0x5652eb826d88]
 mongod(+0x1F6629C) [0x5652eb82729c]
 mongod(+0x1EE8681) [0x5652eb7a9681]
 mongod(+0x1F5B0C0) [0x5652eb81c0c0]
 mongod(+0x1F2553E) [0x5652eb7e653e]
 mongod(+0x1F2562B) [0x5652eb7e662b]
 mongod(wiredtiger_open+0x1713) [0x5652eb795d13]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xA76) [0x5652eab36cf6]
 mongod(+0x126EB8C) [0x5652eab2fb8c]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6E0) [0x5652eaa1f3d0]
 mongod(+0x7FCB7E) [0x5652ea0bdb7e]
 mongod(main+0x95F) [0x5652ea0ddc3f]
 libc.so.6(__libc_start_main+0xF0) [0x7fe3e8e13830]
 mongod(_start+0x29) [0x5652ea13b979]
----- END BACKTRACE -----
Aborted

I have also attached .turtle and .wt WiredTiger files.

Hope to hear from You soon



 Comments   
Comment by Eric Sedor [ 05/Apr/19 ]

We're sorry to hear that. It's possible that 4.0.8 --repair will help, but unfortunately we aren't able to repair this corruption. To avoid a problem like in the future, it is our strong recommendation to:

Comment by Eric Sedor [ 03/Apr/19 ]

Thank you for your report. I've attached a repair attempt of the files you provided ( SERVER-40458_repair_attempt.tgz ). Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If not, can you also confirm you have attempted --repair using the latest release of MongoDB (4.0.8)?

If you are able, can you please provide:

  1. # The complete logs for the affected node, including before, leading up to, and after the first sign of corruption.
  2. A complete description of the underlying storage mechanism in use. Please address questions such as whether storage is locally attached or network-attached, whether disks are SSDs or HDDs, whether disks are Are the disks SSDs or HDDs, whether RAID is in use and if so how it is configured, and what file system and/or volume management system is in use.
  3. A description of your backup method, if any.
  4. A history of the deployment, including:
    1. a timeline of version changes
    2. a timeline of hardware upgrade/downgrade cycles or configuration changes
    3. a timeline of disaster recovery or backup restoration activities
    4. a timeline of any manipulations of the underlying database files, including copies or moves, and information about whether mongod was running during each manipulation.
  5. Finally, can you provide assurances that you have not manipulated (copied or moved) the underlying database files while mongod was running, and that your disks have been recently checked for integrity?
Generated at Thu Feb 08 04:55:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.