-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.2.12
-
Component/s: WiredTiger
-
Labels:None
-
ALL
Hello, we have a problem with Mongo after a power outage. Mongo did not start the status this:
● mongodb.service - High-performance, schema-free document-oriented database
Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Fri 2018-04-06 08:02:31 UTC; 4s ago
Process: 12806 ExecStart = / usr / bin / mongod --quiet --config /etc/mongod.conf (code = dumped, signal
Main PID: 12806 (code = dumped, signal = ABRT)
Apr 06 08:02:30 nondb02.mm4.com systemd [1]: Started High-performance, schema-free document-orient
Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Main process exited, code = dumped, st
Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Unit entered failed state.
Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Failed with result 'core-dump'.
In the logs there was an error by the WiredTiger.wt checksum, we fixed it by replacing the files from adjacent ticket.
And now we have next trouble:
service status exactly the same.
the log is written below:
2017-12-26T09:15:01.616+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] MongoDB starting : pid=21247 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] db version v3.2.12 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] modules: none 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] build environment: 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] distmod: ubuntu1604 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] distarch: x86_64 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-12-26T09:15:01.624+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var /lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } } 2017-12-26T09:15:01.646+0000 E NETWORK [initandlisten] listen(): bind() failed errno:98 Address already in use for socket: 172.21.23.24:27017 2017-12-26T09:15:01.646+0000 E NETWORK [initandlisten] addr already in use 2017-12-26T09:15:01.646+0000 E STORAGE [initandlisten] Failed to set up sockets during startup. 2017-12-26T09:15:01.646+0000 I CONTROL [initandlisten] dbexit: rc: 48 2018-04-06T07:32:03.493+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] MongoDB starting : pid=1435 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] db version v3.2.12 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] allocator: tcmalloc 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] modules: none 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] build environment: 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] distmod: ubuntu1604 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] distarch: x86_64 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] target_arch: x86_64 2018-04-06T07:32:03.503+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var /lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } } 2018-04-06T07:32:03.531+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-04-06T07:32:03.531+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2018-04-06T07:32:03.531+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2018-04-06T07:32:03.531+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true, path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2018-04-06T07:32:03.916+0000 E STORAGE [initandlisten] WiredTiger (0) [1522999923:916176][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header c hecksum of 758591789 doesn't match expected checksum of 1262014029 2018-04-06T07:32:03.916+0000 E STORAGE [initandlisten] WiredTiger (0) [1522999923:916233][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2018-04-06T07:32:03.916+0000 E STORAGE [initandlisten] WiredTiger (-31804) [1522999923:916242][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTige r library panic 2018-04-06T07:32:03.916+0000 I - [initandlisten] Fatal Assertion 28558 2018-04-06T07:32:03.916+0000 I - [initandlisten] ***aborting after fassert() failure 2018-04-06T07:32:04.040+0000 F - [initandlisten] Got signal: 6 (Aborted). 0x1533852 0x15327f9 0x1533062 0x7fdd53c7c390 0x7fdd538d7428 0x7fdd538d902a 0x14b57f3 0x125b4d2 0x96f272 0x96f343 0x96f509 0x1b7681f 0x1b76d7b 0x1b7371d 0x1b77eb7 0x1b94e6c 0x1bca33b 0x1c52eb2 0x1c53585 0x1c53a7c 0x1bd9ac1 0x1c48d68 0x1c148df 0x1c149ac 0x1bc6816 0x123f568 0x1238735 0x1133f38 0x9bba60 0x970392 0x7fdd538c2830 0x9b6089 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"1133852","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11327F9"},{"b":"400000","o":"1133062"},{"b":"7FDD53C6B000","o":"11390"},{"b":"7FDD538A2000","o":"35428","s":"gsignal"},{"b":"7FDD538A2000","o":"3702A","s":"abort"},{"b":"400000","o":"10B57F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E5B4D2"},{"b":"400000","o":"56F272"},{"b":"400000","o":"56F343","s":"__wt_err"},{"b":"400000","o":"56F509","s":"__wt_panic"},{"b":"400000","o":"177681F"},{"b":"400000","o":"1776D7B"},{"b":"400000","o":"177371D"},{"b":"400000","o":"1777EB7"},{"b":"400000","o":"1794E6C"},{"b":"400000","o":"17CA33B"},{"b":"400000","o":"1852EB2"},{"b":"400000","o":"1853585"},{"b":"400000","o":"1853A7C"},{"b":"400000","o":"17D9AC1"},{"b":"400000","o":"1848D68"},{"b":"400000","o":"18148DF"},{"b":"400000","o":"18149AC"},{"b":"400000","o":"17C6816","s":"wiredtiger_open"},{"b":"400000","o":"E3F568","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E38735"},{"b":"400000","o":"D33F38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5BBA60","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"570392","s":"main"},{"b":"7FDD538A2000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B6089","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "u ame" : { "sysname" : "Linux", "release" : "4.4.0-66-generic", "version" : "#87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7877295E4 A95F29C74796963664BD1115CFE251E" }, { "b" : "7FFEDABCB000", "elfType" : 3, "buildId" : "5CCDAB3B3062DC906D26EBB5A688D41BFE5EFD75" }, { "b" : "7FDD54BF7000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfT ype" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FDD547B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4 F" }, { "b" : "7FDD545AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FDD543A7000", "path" : "/lib/x86_64-linux-gnu/libdl.so .2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FDD5409E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B14 89" }, { "b" : "7FDD53E88000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FDD53C6B000", "path" : "/lib/x86_64-linux-gnu/libp thread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FDD538A2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6 E4E68692053" }, { "b" : "7FDD54E60000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1533852] mongod(+0x11327F9) [0x15327f9] mongod(+0x1133062) [0x1533062] libpthread.so.0(+0x11390) [0x7fdd53c7c390] libc.so.6(gsignal+0x38) [0x7fdd538d7428] libc.so.6(abort+0x16A) [0x7fdd538d902a] mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14b57f3] mongod(+0xE5B4D2) [0x125b4d2] mongod(+0x56F272) [0x96f272] mongod(__wt_err+0x8B) [0x96f343] mongod(__wt_panic+0x24) [0x96f509] mongod(+0x177681F) [0x1b7681f] mongod(+0x1776D7B) [0x1b76d7b] mongod(+0x177371D) [0x1b7371d] mongod(+0x1777EB7) [0x1b77eb7] mongod(+0x1794E6C) [0x1b94e6c] mongod(+0x17CA33B) [0x1bca33b] mongod(+0x1852EB2) [0x1c52eb2] mongod(+0x1853585) [0x1c53585] mongod(+0x1853A7C) [0x1c53a7c] mongod(+0x17D9AC1) [0x1bd9ac1] mongod(+0x1848D68) [0x1c48d68] mongod(+0x18148DF) [0x1c148df] mongod(+0x18149AC) [0x1c149ac] mongod(wiredtiger_open+0x15D6) [0x1bc6816] mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xDD8) [0x123f568] mongod(+0xE38735) [0x1238735] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1133f38] mongod(_ZN5mongo13initAndListenEi+0x480) [0x9bba60] mongod(main+0x712) [0x970392] libc.so.6(__libc_start_main+0xF0) [0x7fdd538c2830] mongod(_start+0x29) [0x9b6089] ----- END BACKTRACE -----
In the logs there was an error by the BT checkbox, we fixed it by replacing the files 1