[SERVER-51515] Mongodb cannot start after being accidentally shut down -- Repair parameter repair reports the following exception: Please help me!!Version is V3.4.24 Created: 13/Oct/20  Updated: 27/Oct/23  Resolved: 19/Oct/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.22
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: kun zhang Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     Zip Archive file1.zip     HTML File log     Text File mongod.log     Text File repair.log     Zip Archive repair_attempt_SERVER-51515.zip     Zip Archive repair_attempt_SERVER-51515_take2.zip     Zip Archive task2.log.zip    
Operating System: ALL
Steps To Reproduce:

 I have checked SERVER-19815 and need to repair the file. I have uploaded it. Please help me repair it, thank you!

Participants:

 Description   

2020-10-13T17:38:23.575 + 0800 I CONTROL [initandlisten] MongoDB开始:pid = 12114端口= 27017 dbpath = / data / mongo_27018 64位主机= localhost.localdomain
2020-10-13T17:38:23.575 + 0800我控制[initandlisten]数据库版本v3.4.24
2020-10-13T17:38:23.575 + 0800我控制[initandlisten] git版本:865b4f6a96d0f5425e39a18337105f33e8db504d
2020-10-13T17:38:23.575 + 0800我控制了[initandlisten] OpenSSL版本1.0.1e-fips 2013年2月11日
2020-10-13T17:38:23.575 + 0800 I控制[initandlisten]分配器:tcmalloc
2020-10-13T17:38:23.575 + 0800 I控制[initandlisten]模块:无
2020-10- 13T17:38:23.575 + 0800我控制[initandlisten]构建环境:
2020-10-13T17:38:23.575 + 0800我控制[initandlisten] distmod:rhel70
2020-10-13T17:38:23.575+0800 I CONTROL [initandlisten] distarch: x86_64
2020-10-13T17:38:23.575+0800 I CONTROL [initandlisten] target_arch: x86_64
2020-10-13T17:38:23.575+0800 I CONTROL [initandlisten] options: { repair: true, storage:

{ dbPath: "/data/mongo_27018" }

}
2020-10-13T17:38:23.592+0800 I - [initandlisten] Detected data files in /data/mongo_27018 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-10-13T17:38:23.592+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2020-10-13T17:38:23.592+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=15440M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-10-13T17:38:23.969+0800 E STORAGE [initandlisten] WiredTiger error (0) [1602581903:969973][12114:0x7fc92f142e80], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 20480B block at offset 49692672: block header checksum of 3298646904 doesn't match expected checksum of 3987027328
2020-10-13T17:38:23.970+0800 E STORAGE [initandlisten] WiredTiger error (0) [1602581903:970037][12114:0x7fc92f142e80], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value
2020-10-13T17:38:23.970+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1602581903:970049][12114:0x7fc92f142e80], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-10-13T17:38:23.970+0800 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
2020-10-13T17:38:23.970+0800 I - [initandlisten]

***aborting after fassert() failure

2020-10-13T17:38:23.986+0800 F - [initandlisten] Got signal: 6 (Aborted).

0x559c9c31fa61 0x559c9c31ec99 0x559c9c31f17d 0x7fc92dd2f5d0 0x7fc92d989207 0x7fc92d98a8f8 0x559c9b5afdd1 0x559c9c0275a6 0x559c9b5ba4fe 0x559c9b5ba71a 0x559c9b5ba97c 0x559c9cc333d5 0x559c9cc4cf9b 0x559c9cc53e8d 0x559c9cc6cc41 0x559c9cc388c9 0x559c9cc91d53 0x559c9cd238a1 0x559c9cd23d87 0x559c9cc88a17 0x559c9cc80e44 0x559c9c00be0a 0x559c9c004105 0x559c9bef64e7 0x559c9b59ae6e 0x559c9b5bbd7b 0x7fc92d9753d5 0x559c9b61c74f
----- BEGIN -----回溯

{“ backtrace”:[\{“ b”:“ 559C9AD5D000”,“ o”:“ 15C2A61”,“ s”:“ _ ZN5mongo15printStackTraceERSo”}

,{“ b”:“ 559C9AD5D000”,“ o”:“ 15C1C99”}, {“ b”:“ 559C9AD5D000”,“ o”:“ 15C217D”},{“ b”:“ 7FC92DD20000”,“ o”:“ F5D0”},{“ b”:“ 7FC92D953000”,“ o”:“ 36207“,” s“:” gsignal“},{” b“:” 7FC92D953000“,” o“:” 378F8“,” s“:”中止“},{” b“:” 559C9AD5D000“,” o“ :“ 852DD1”,“ s”:“ _ ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj”},{“ b”:“ 559C9AD5D000”,“ o”:“ 12CA5A6”},{“ b”:“ 559C9AD5D000”,“ o”:“ 85D4FE”,“ s”:“ __ wt_eventv”},{“ b”:“ 559C9AD5D000”,“ o”:“ 85D71A”,“ s”:“ __ wt_err”},{“ b”:“ 559C9AD5D000 “,” o“:” 85D97C“,” s“:” __ wt_panic“},{” b“:” 559C9AD5D000“,” o“:” 1ED63D5“,” s“:” __ wt_bm_read“},{” b“: “ 559C9AD5D000”,“ o”:“ 1EEFF9B”,“ s”:“ __ wt_bt_read”},{“ b”:“ 559C9AD5D000”,“ o”:“ 1EF6E8D”,“ s”:“ __ wt_page_in_func”},{“ b “:” 559C9AD5D000“,” o“:” 1F0FC41“},{” b“:” 559C9AD5D000“,” o“:” 1EDB8C9“,” s“:” __ wt_btcur_next“},{” b“:”559C9AD5D000“,” o“:” 1F34D53“},{” b“:” 559C9AD5D000“,” o“:” 1FC68A1“},{” b“:” 559C9AD5D000“,” o“:” 1FC6D87“,” s“ :“ __ wt_txn_recover”},{“ b”:“ 559C9AD5D000”,“ o”:“ 1F2BA17”,“ s”:“ __ wt_connection_workers”},{“ b”:“ 559C9AD5D000”,“ o”:“ 1F23E44”,“ s“:” wiredtiger_open“},{” b“:” 559C9AD5D000“,” o“:” 12AEE0A“,” s“:” _ ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_b“ B” B“ 9”“ B” },{“ b”:“ 559C9AD5D000”,“ o”:“ 11994E7”,“ s”:“ _ ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv”},{“ b”:“ 559C9AD5D000”,“ o”:“ 83DE6E”},{“ b”:“ 559C9AD5D000”,“ o”:“ 85ED7B”,“ s”:“ main “},{” b“:” 7FC92D953000“,” o“:” 223D5“,” s“:” __ libc_start_main“},{” b“:” 559C9AD5D000“,” o“:” 8BF74F“}}],” processInfo “:{” mongodbVersion“:” 3.4.24“,” gitVersion“:” 865b4f6a96d0f5425e39a18337105f33e8db504d“,” compiledModules“:[],” uname“:

{” sysname“:” Linux“,” release“:” 3.10.0- 957.el7.x86_64“,” version“:”#1 SMP Thu Nov 8 23:39:32 UTC 2018“,” machine“:“ x86_64”}

,“ somap”:[\{“ b”:“ 559C9AD5D000”,“ elfType”:3,“ buildId”:“ 65C0AAD3BBB742F05AE87A9FDCADE7BF5F94FBE3”},\{“ b”:“ 7FFE960D8000”,“ elfType”:3, “ buildId”:“ 163C2DC43405427478788BAD0AFD537A7ACF7A13”},\{“ b”:“ 7FC92ECC1000”,“ path”:“ /lib64/libssl.so.10”,“ elfType”:3,“ buildId”:“ AEF5E6F2240B55F90E9DF76CFBB8B9658” b“:” 7FC92E860000“,” path“:” / lib64 / libcrypto.so.10“,” elfType“:3,” buildId“:” 8BD89856B64DD5189BF075EF574EDF203F93D44A“},\{” b“:” 7FC92E658000“,” path“: “ /lib64/librt.so.1”,“ elfType”:3,“ buildId”:“ EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2”},\{“ b”:“ 7FC92E454000”,“ path”:“ /lib64/libdl.so.2”,“ elfType”:3,“ buildId”:“ 67AD3498AC7DE3EB952A24321DF b“:” 7FC92E152000“,”路径“:” / lib64 / libm.so.6“,” elfType“:3,” buildId“:” 918D3696BF321AA8D32950AB2AB8D0F1B21AC907“},\{” b“:” 7FC92DF3C000“,”路径“: “ /lib64/libgcc_s.so.1”,“ elfType”:3,“ buildId”:“ 6B4F3D896CD0F06FCB3DEF0245F204ECE3220D7E”},\{“ b”:“ 7FC92DD20000”,“ path”:“ /lib64/libpthread.so.0”, “ elfType”:3,“ buildId”:“3D9441083D079DC2977F1BD50C8068D11767232D“},\{” b“:” 7FC92D953000“,” path“:” /lib64/libc.so.6“,” elfType“:3,” buildId“:” 3C61131D1DAC9DA79B73188E7702BEF786C2AD54“} “,” path“:” /lib64/ld-linux-x86-64.so.2“,” elfType“:3,” buildId“:” 5DA2D47925497B2F5875A7D8D1799A1227E2FDE4“},\{” b“:” 7FC92D706000“,” path“ :“ /lib64/libgssapi_krb5.so.2”,“ elfType”:3,“ buildId”:“ B5C83BDE7ED7026835B779FA0F957FCCCD599F40”},\{“ b”:“ 7FC92D41D000”,“路径”:“ /lib64/libkrb5.so.3” ,“ elfType”:3,“ buildId”:“ 8B63976509135BA73A12153D6FDF7B3B9E5D2A54”},\{“ b”:“ 7FC92D219000”,“ path”:“ /lib64/libcom_err.so.2”,“ elfType”:3,“ buildId”:“ B4BE1023D9606A88169DF411BF94AF417D7BA1A0”},\{“ b”:“ 7FC92CFFE000”,“路径“:” /lib64/libk5crypto.so.3“,” elfType“:3,” buildId“:” 6183129B5F29CA14580E517DF94EF317761FA6C9“},\{” b“:” 7FC92CDE8000“,” path“:” /lib64/libz.so.1 “,” elfType“:3,” buildId“:” B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745“},\{” b“:” 7FC92CBD9000“,” path“:” /lib64/libkrb5support.so.0“,” elfType“:3,” buildId“ :“ 98F619035053EF68358099CE7CF1AA528B3B229D”},\{“ b”:“ 7FC92C9D5000”,“路径”:“ / lib64 / libkeyutils.so.1”,“ elfType”:3,“ buildId”:“ 2E01D5AC08C1280D013AAB96B292AC58BC30A263”},\{“ b”:“ 7FC92C7BC000”,“路径”:“ / lib64 / libresolv.so。 2“,” elfType“:3,” buildId“:” 4C488F6E7044BB966162C1F7081ABBA6EBB2B485“},\{” b“:” 7FC92C595000“,” path“:” /lib64/libselinux.so.1“,” elfType“:3,” buildId“ “:”“ D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179”“},\{” b“:” 7FC92C333000“,” path“:” /lib64/libpcre.so.1“,” elfType“:3,” buildId“:” 9CA3D11F018BEEB719CDB34BE800BF16413}2E01D5AC08C1280D013AAB96B292AC58BC30A263“},\{” b“:” 7FC92C7BC000“,” path“:” /lib64/libresolv.so.2“,” elfType“:3,” buildId“:” 4C488F6E7044BB966162C1F7081ABBA6“” 5FC2B4B “,” path“:” /lib64/libselinux.so.1“,” elfType“:3,” buildId“:” D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179“},\{” b“:” 7FC92C333000“,” path“:” / lib64 / libpcre .so.1“,” elfType“:3,” buildId“:” 9CA3D11F018BEEB719CDB34BE800BF1641350D0A“}]}}2E01D5AC08C1280D013AAB96B292AC58BC30A263“},{” b“:” 7FC92C7BC000“,” path“:” /lib64/libresolv.so.2“,” elfType“:3,” buildId“:” 4C488F6E7044BB966162C1F7081ABBA6“” 5BFC2B5250“ “,” path“:” /lib64/libselinux.so.1“,” elfType“:3,” buildId“:” D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179“},{” b“:” 7FC92C333000“,” path“:” / lib64 / libpcre .so.1“,” elfType“:3,” buildId“:” 9CA3D11F018BEEB719CDB34BE800BF1641350D0A“}]}}4C488F6E7044BB966162C1F7081ABBA6EBB2B485“},{” b“:” 7FC92C595000“,” path“:” /lib64/libselinux.so.1“,” elfType“:3,” buildId“:” D2DD4DA3FDE1477D25BFFF80F3A25“ B” 7,7“ 92 “,” path“:” /lib64/libpcre.so.1“,” elfType“:3,” buildId“:” 9CA3D11F018BEEB719CDB34BE800BF1641350D0A“}]}}4C488F6E7044BB966162C1F7081ABBA6EBB2B485“},{” b“:” 7FC92C595000“,”路径“:” /lib64/libselinux.so.1“,” elfType“:3,” buildId“:” D2DD4DA3FDE1477D25BFFF80F3A25“ C:” 7,7“ 92 “,” path“:” /lib64/libpcre.so.1“,” elfType“:3,” buildId“:” 9CA3D11F018BEEB719CDB34BE800BF1641350D0A“}]}}9CA3D11F018BEEB719CDB34BE800BF1641350D0A“}]}}9CA3D11F018BEEB719CDB34BE800BF1641350D0A“}]}}
mongod(_ZN5mongo15printStackTraceERSo + 0x41)[0x559c9c31fa61]
mongod(+ 0x15C1C99)[0x559c9c31ec99]
mongod(+ 0x15C217D)[0x559c9c31f17d]
libpthread.so.0(+ 0xF5D0)(c0c6f09
(00F6D0)(00F6D0)
libc.so.6的(中止+ 0x148)[0x7fc92d98a8f8]
的mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj +为0x0)[0x559c9b5afdd1]
mongod的(+ 0x12CA5A6)[0x559c9c0275a6]
的mongod(__ wt_eventv + 0x3D7)[0x559c9b5ba4fe]
的mongod(__ wt_err + 0x9D)[0x559c9b5ba71a]
的mongod( __wt_panic + 0x2E)[0x559c9b5ba97c]
mongod(__ wt_bm_read + 0x135)[0x559c9cc333d5]
mongod(__ wt_bt_read + 0x1FB)[0x559c9cc4cf9b]
mongod(__ wt_page_in_func + 0x11DD)
的mongod(+ 0x1F0FC41)[0x559c9cc6cc41]
的mongod(__ wt_btcur_next + 0x399)[0x559c9cc388c9]
mongod的(+ 0x1F34D53)[0x559c9cc91d53]
mongod的(+ 0x1FC68A1)[0x559c9cd238a1]
的mongod(__ wt_txn_recover + 0x487)[0x559c9cd23d87]
的mongod(__ wt_connection_workers + 0×37)[0x559c9cc88a17 ]
的mongod(wiredtiger_open + 0x1984)[0x559c9cc80e44]
的mongod(ZN5mongo18WiredTigerKVEngineC1ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb + 0xAFA)[0x559c9c00be0a]
mongod的(+ 0x12A7105)[0x559c9c004105]
的mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv + 0x697)[0x559c9bef64e7]
mongod的(+ 0x83DE6E)[0x559c9b59ae6e]
的mongod(主+ 0x96B) [0x559c9b5bbd7b]
libc.so.6(__ libc_start_main + 0xF5)[0x7fc92d9753d5]
mongod(+ 0x8BF74F)[0x559c9b61c74f]
----- END BACKTRACE -----
中止(核心转储)



 Comments   
Comment by kun zhang [ 20/Oct/20 ]

Hi Dima,

     Can corrupt files be ignored to start MongoDB?

Comment by Dmitry Agranat [ 19/Oct/20 ]

Hi wozhangkun@hotmail.com,

Unfortunately we aren't able to repair this corruption.

To avoid a problem like this in the future, it is our strong recommendation to:

Regards,
Dima

Comment by kun zhang [ 19/Oct/20 ]

XFS file system based on Centos-7.6-X86_64

Comment by Dmitry Agranat [ 16/Oct/20 ]

Hi wozhangkun@hotmail.com, what file system do you use, is it a compressed NTFS file system?

Comment by kun zhang [ 16/Oct/20 ]

Hi  Dmitry Agranat,

Thank you for helping me, Sorry to trouble you again!

 I replace the file with $dbpath.I couldn't start MongoDB correctly by executing the commands "systemctl start mongod. service" and "mongod --repair --dbpath = / data/mongo_27018 /".  latest WiredTiger.wt and WiredTiger.turtle and log files :task2.log.zip.Please help me!!!Thank you very much.

Comment by Dmitry Agranat [ 15/Oct/20 ]

Hi wozhangkun@hotmail.com,

I've attached a second repair attempt of the files you provided as repair_attempt_SERVER-51515_take2.zip. Please extract these files, replace them in your $dbpath, and let us know if it resolves the issue.

Thanks,
Dima

Comment by kun zhang [ 15/Oct/20 ]

Upload attachment:file1.zip

Comment by Dmitry Agranat [ 15/Oct/20 ]

Hi wozhangkun@hotmail.com,

Please upload the latest WiredTiger.wt and WiredTiger.turtle files and we'll try to repair it again.

Thanks,
Dima

Comment by kun zhang [ 15/Oct/20 ]

Hi, Dmitry Agranat, thanks for your help.I replace the file with $dbpath.I couldn't start MongoDB correctly by executing the commands "systemctl start mongod. service" and "mongod --repair --dbpath = / data/mongo_27018 /".The logs indicate that there should be two file exceptions, and we seem to be one step away from success.I can tolerate ignoring fewer exception files to start MongoDB properly.see log repair.log and mongod.log.Please help me!!!Thank you very much.

 

All the files mentioned in the logs exist as follows:

[root@localhost mongo_27018]# ll index-4829-5116004730064471429.wt
rw-rr- 1 mongod mongod 3645440 Jun 25 19:19 index-4829-5116004730064471429.wt
[root@localhost mongo_27018]# ll collection-28--4225635139039664472.wt
rw-rr- 1 mongod mongod 75055104 Oct 15 10:06 collection-28--4225635139039664472.wt

Comment by Dmitry Agranat [ 14/Oct/20 ]

Hi wozhangkun@hotmail.com,

As MongoDB 3.4 has reached EOL, we can try to assist you as a one-time exception.

I've attached a repair attempt of the files you provided as repair_attempt_SERVER-51515.zip. Please extract these files, replace them in your $dbpath, and let us know if it resolves the issue.

Thanks,
Dima

Generated at Thu Feb 08 05:25:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.