Details
-
Question
-
Resolution: Done
-
Critical - P2
-
None
-
None
-
None
-
None
Description
Hi, unfortuneatly I am having issues getting a mongodb started back up after a system failure.
I have tried the mongod --repair, as well as using the wt tool to check, however they both report an invalid checksuom on the WiredTiger.wt file.
I was hoping you ad a better way of attempting recovery on these?
Thank you, logs below:
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] MongoDB starting : pid=29 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodb-1-wnfnn
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] db version v3.4.9
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] modules: none
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] build environment:
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] distarch: x86_64
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
, replication: { oplogSizeMB: 64 }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb/data", wiredTiger: { engineConfig:
{ cacheSizeGB: 1.0 } } }, systemLog: { quiet: true } }
MongoDB shell version v3.4.9
2020-07-09T21:16:03.409+0000 I - [initandlisten] Detected data files in /var/lib/mongodb/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-07-09T21:16:03.411+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1024M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
connecting to: mongodb://127.0.0.1:27017/localhost
2020-07-09T21:16:03.458+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2020-07-09T21:16:03.458+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:237:13
@(connect):1:6
2020-07-09T21:16:03.465+0000 E STORAGE [initandlisten] WiredTiger error (0) [1594329363:465773][29:0x7f47e1b69e80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 57344: block header checksum of 979725410 doesn't match expected checksum of 793883532
2020-07-09T21:16:03.465+0000 E STORAGE [initandlisten] WiredTiger error (0) [1594329363:465887][29:0x7f47e1b69e80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2020-07-09T21:16:03.465+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1594329363:465909][29:0x7f47e1b69e80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-07-09T21:16:03.465+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2020-07-09T21:16:03.465+0000 I - [initandlisten]
***aborting after fassert() failure
exception: connect failed
2020-07-09T21:16:03.470+0000 F - [initandlisten] Got signal: 6 (Aborted).
0x56354b05735a 0x56354b056699 0x56354b056b66 0x7f47de49e6d0 0x7f47de0f8277 0x7f47de0f9968 0x56354a2b745b 0x56354ad5cc26 0x56354a2c077a 0x56354a2c0991 0x56354a2c0bef 0x56354b84887f 0x56354b848ddb 0x56354b845435 0x56354b849fa7 0x56354b863a35 0x56354b89d3f3 0x56354b928747 0x56354b928c96 0x56354b928f5c 0x56354b8abb31 0x56354b91e9f8 0x56354b8e82be 0x56354b8e83ab 0x56354b899777 0x56354ad42031 0x56354ad39fc6 0x56354ac20517 0x56354a29d984 0x56354a2c1e44 0x7f47de0e4445 0x56354a31f615
----- BEGIN BACKTRACE -----
,{"b":"56354A0B4000","o":"FA2699"},{"b":"56354A0B4000","o":"FA2B66"},{"b":"7F47DE48F000","o":"F6D0"},{"b":"7F47DE0C2000","o":"36277","s":"gsignal"},{"b":"7F47DE0C2000","o":"37968","s":"abort"},{"b":"56354A0B4000","o":"20345B"},{"b":"56354A0B4000","o":"CA8C26"},{"b":"56354A0B4000","o":"20C77A"},{"b":"56354A0B4000","o":"20C991","s":"_wt_err"},{"b":"56354A0B4000","o":"20CBEF","s":"wt_panic"},{"b":"56354A0B4000","o":"179487F"},{"b":"56354A0B4000","o":"1794DDB"},{"b":"56354A0B4000","o":"1791435"},{"b":"56354A0B4000","o":"1795FA7"},{"b":"56354A0B4000","o":"17AFA35"},{"b":"56354A0B4000","o":"17E93F3"},{"b":"56354A0B4000","o":"1874747"},{"b":"56354A0B4000","o":"1874C96"},{"b":"56354A0B4000","o":"1874F5C"},{"b":"56354A0B4000","o":"17F7B31"},{"b":"56354A0B4000","o":"186A9F8"},{"b":"56354A0B4000","o":"18342BE"},{"b":"56354A0B4000","o":"18343AB"},{"b":"56354A0B4000","o":"17E5777","s":"wiredtiger_open"},{"b":"56354A0B4000","o":"C8E031"},{"b":"56354A0B4000","o":"C85FC6"},{"b":"56354A0B4000","o":"B6C517"},{"b":"56354A0B4000","o":"1E9984"},{"b":"56354A0B4000","o":"20DE44"},{"b":"7F47DE0C2000","o":"22445","s":"_libc_start_main"},{"b":"56354A0B4000","o":"26B615"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" :
{ "sysname" : "Linux", "release" : "3.10.0-1127.13.1.el7.x86_64", "version" : "#1 SMP Tue Jun 23 15:46:38 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "56354A0B4000", "elfType" : 3, "buildId" : "28974745B6B8764D64F97F6BF19D04AC5EDD5B65" }, { "b" : "7FFE6449A000", "elfType" : 3, "buildId" : "B1962B2DF9C13102C04B311EBA759764C1B8E9D8" }, { "b" : "7F47E1749000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F47E14C0000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libyaml-cpp.so.rh-mongodb34-0.5", "elfType" : 3, "buildId" : "FE3E65529D3D5623ECF32A331FFBA7ACD02D40A6" }, { "b" : "7F47E12BA000", "path" : "/lib64/libsnappy.so.1", "elfType" : 3, "buildId" : "3CEB901120465B031FA92FA449A9AD981CEC1659" }, { "b" : "7F47E1048000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libtcmalloc.so.rh-mongodb34-4", "elfType" : 3, "buildId" : "9B8AE99A244389054FED1DC74C0F7B79CD6920CE" }, { "b" : "7F47E0DE6000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F47E0BDD000", "path" : "/lib64/libpcrecpp.so.0", "elfType" : 3, "buildId" : "8BBAA8A5638DCB0C4B523FCA4B54613AE543BDF1" }, { "b" : "7F47E096A000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_program_options.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "51F7B5995F6FD852D84CE889AED4DF5807C588FD" }, { "b" : "7F47E0753000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_filesystem.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "BC537D140DFB358B75B19A985EDE485EB9D122EE" }, { "b" : "7F47E0530000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_thread.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "66EB1D274A8B36597AE3AEDB3A86730FD37F5317" }, { "b" : "7F47E032C000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_system.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "5BCE91B1A7B7AAA44098CF23DA1B795F62F2B1D4" }, { "b" : "7F47E0114000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_iostreams.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "5FF6A64EF26564056ECEB54D3BD7923CEDB27112" }, { "b" : "7F47DFF0C000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_chrono.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "413AD9DB5A13FFC6E5252AD320DC2C84D17753DF" }, { "b" : "7F47DFBFC000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_regex.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "C581BE7B9B2C0A59BC05D34F6922A66C5D89D475" }, { "b" : "7F47DF9A9000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libstemmer.so.rh-mongodb34-0", "elfType" : 3, "buildId" : "EAAFF5B4F644622B2E88F23A70098C539A7DA9E8" }, { "b" : "7F47DF737000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, { "b" : "7F47DF2D6000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, { "b" : "7F47DF0CE000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F47DEECA000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F47DEBC3000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "9589AE0FDA6AEB1183EBA1C62A328F933E7817FD" }, { "b" : "7F47DE8C1000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F47DE6AB000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F47DE48F000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F47DE0C2000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F47DDEA8000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libunwind.so.rh-mongodb34-8", "elfType" : 3, "buildId" : "CE5996C64EA0376E71D6F37FCEC1FB95F860972F" }, { "b" : "7F47E195F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F47DDC98000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0C85C0386F0CF41EA39969CF7F58A558D1AD3235" }, { "b" : "7F47DC6C4000", "path" : "/lib64/libicudata.so.50", "elfType" : 3, "buildId" : "04813117A9FB30F2DB02B8C546C4FC38BF4890A0" }, { "b" : "7F47DC2C5000", "path" : "/lib64/libicui18n.so.50", "elfType" : 3, "buildId" : "7CAA9714B33AA18B16FAFDC88FA43185238C6C86" }, { "b" : "7F47DBF4C000", "path" : "/lib64/libicuuc.so.50", "elfType" : 3, "buildId" : "E9EA46B17EEE2AE15094650C00B891C094CB802D" }, { "b" : "7F47DBCFF000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "16FE0DC6CEFC5F444BC876516D02EFE9CC2D432F" }, { "b" : "7F47DBA17000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D1CD1B94855A85FBC735C745DB39BC096F7D8CC3" }, { "b" : "7F47DB813000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "2C7EF64EF0C5AF8BCFA8F9E628E5605A7D8C52D3" }, { "b" : "7F47DB5E0000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A20F715C514B3EA873F4CC77D585A50CB670E266" }, { "b" : "7F47DB3D2000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "9F82B06CE44724A8ACE827A2C95A9A76619EA314" }, { "b" : "7F47DB1CE000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F47DAFB5000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F47DAD8E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "1B0C0DC4F9054C5922082921C1447739F27AC8F4" } ] }}
mongod(+0xFA335A) [0x56354b05735a]
mongod(+0xFA2699) [0x56354b056699]
mongod(+0xFA2B66) [0x56354b056b66]
libpthread.so.0(+0xF6D0) [0x7f47de49e6d0]
libc.so.6(gsignal+0x37) [0x7f47de0f8277]
libc.so.6(abort+0x148) [0x7f47de0f9968]
mongod(+0x20345B) [0x56354a2b745b]
mongod(+0xCA8C26) [0x56354ad5cc26]
mongod(+0x20C77A) [0x56354a2c077a]
mongod(__wt_err+0xA0) [0x56354a2c0991]
mongod(__wt_panic+0x24) [0x56354a2c0bef]
mongod(+0x179487F) [0x56354b84887f]
mongod(+0x1794DDB) [0x56354b848ddb]
mongod(+0x1791435) [0x56354b845435]
mongod(+0x1795FA7) [0x56354b849fa7]
mongod(+0x17AFA35) [0x56354b863a35]
mongod(+0x17E93F3) [0x56354b89d3f3]
mongod(+0x1874747) [0x56354b928747]
mongod(+0x1874C96) [0x56354b928c96]
mongod(+0x1874F5C) [0x56354b928f5c]
mongod(+0x17F7B31) [0x56354b8abb31]
mongod(+0x186A9F8) [0x56354b91e9f8]
mongod(+0x18342BE) [0x56354b8e82be]
mongod(+0x18343AB) [0x56354b8e83ab]
mongod(wiredtiger_open+0x1767) [0x56354b899777]
mongod(+0xC8E031) [0x56354ad42031]
mongod(+0xC85FC6) [0x56354ad39fc6]
mongod(+0xB6C517) [0x56354ac20517]
mongod(+0x1E9984) [0x56354a29d984]
mongod(+0x20DE44) [0x56354a2c1e44]
libc.so.6(__libc_start_main+0xF5) [0x7f47de0e4445]
mongod(+0x26B615) [0x56354a31f615]
----- END BACKTRACE -----