[SERVER-34354] Mongodb do not start up again after a power outage Created: 06/Apr/18  Updated: 24/Apr/18  Resolved: 10/Apr/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.12
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Pavel [X] Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tar.gz    
Operating System: ALL
Participants:

 Description   

Hello, we have a problem with Mongo after a power outage. Mongo did not start the status this:
● mongodb.service - High-performance, schema-free document-oriented database
Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Fri 2018-04-06 08:02:31 UTC; 4s ago
Process: 12806 ExecStart = / usr / bin / mongod --quiet --config /etc/mongod.conf (code = dumped, signal
Main PID: 12806 (code = dumped, signal = ABRT)
Apr 06 08:02:30 nondb02.mm4.com systemd [1]: Started High-performance, schema-free document-orient
Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Main process exited, code = dumped, st
Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Unit entered failed state.
Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Failed with result 'core-dump'.

In the logs there was an error by the WiredTiger.wt checksum, we fixed it by replacing the files from adjacent ticket.

And now we have next trouble:
service status exactly the same.
the log is written below:

2017-12-26T09:15:01.616+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] MongoDB starting : pid=21247 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] db version v3.2.12
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] modules: none
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] build environment:
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var
/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2017-12-26T09:15:01.646+0000 E NETWORK  [initandlisten] listen(): bind() failed errno:98 Address already in use for socket: 172.21.23.24:27017
2017-12-26T09:15:01.646+0000 E NETWORK  [initandlisten]   addr already in use
2017-12-26T09:15:01.646+0000 E STORAGE  [initandlisten] Failed to set up sockets during startup.
2017-12-26T09:15:01.646+0000 I CONTROL  [initandlisten] dbexit:  rc: 48
2018-04-06T07:32:03.493+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1435 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] db version v3.2.12
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] modules: none
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] build environment:
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var
/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2018-04-06T07:32:03.531+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-06T07:32:03.531+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-06T07:32:03.531+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-06T07:32:03.531+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,
path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-06T07:32:03.916+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522999923:916176][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header c
hecksum of 758591789 doesn't match expected checksum of 1262014029
2018-04-06T07:32:03.916+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522999923:916233][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2018-04-06T07:32:03.916+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1522999923:916242][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTige
r library panic
2018-04-06T07:32:03.916+0000 I -        [initandlisten] Fatal Assertion 28558
2018-04-06T07:32:03.916+0000 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-04-06T07:32:04.040+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1533852 0x15327f9 0x1533062 0x7fdd53c7c390 0x7fdd538d7428 0x7fdd538d902a 0x14b57f3 0x125b4d2 0x96f272 0x96f343 0x96f509 0x1b7681f 0x1b76d7b 0x1b7371d 0x1b77eb7 0x1b94e6c 0x1bca33b 0x1c52eb2 0x1c53585 0x1c53a7c 0x1bd9ac1 0x1c48d68 0x1c148df 0x1c149ac 0x1bc6816 0x123f568 0x1238735 0x1133f38 0x9bba60 0x970392 0x7fdd538c2830 0x9b6089
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1133852","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11327F9"},{"b":"400000","o":"1133062"},{"b":"7FDD53C6B000","o":"11390"},{"b":"7FDD538A2000","o":"35428","s":"gsignal"},{"b":"7FDD538A2000","o":"3702A","s":"abort"},{"b":"400000","o":"10B57F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E5B4D2"},{"b":"400000","o":"56F272"},{"b":"400000","o":"56F343","s":"__wt_err"},{"b":"400000","o":"56F509","s":"__wt_panic"},{"b":"400000","o":"177681F"},{"b":"400000","o":"1776D7B"},{"b":"400000","o":"177371D"},{"b":"400000","o":"1777EB7"},{"b":"400000","o":"1794E6C"},{"b":"400000","o":"17CA33B"},{"b":"400000","o":"1852EB2"},{"b":"400000","o":"1853585"},{"b":"400000","o":"1853A7C"},{"b":"400000","o":"17D9AC1"},{"b":"400000","o":"1848D68"},{"b":"400000","o":"18148DF"},{"b":"400000","o":"18149AC"},{"b":"400000","o":"17C6816","s":"wiredtiger_open"},{"b":"400000","o":"E3F568","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E38735"},{"b":"400000","o":"D33F38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5BBA60","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"570392","s":"main"},{"b":"7FDD538A2000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B6089","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "u
ame" : { "sysname" : "Linux", "release" : "4.4.0-66-generic", "version" : "#87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7877295E4
A95F29C74796963664BD1115CFE251E" }, { "b" : "7FFEDABCB000", "elfType" : 3, "buildId" : "5CCDAB3B3062DC906D26EBB5A688D41BFE5EFD75" }, { "b" : "7FDD54BF7000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfT
ype" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FDD547B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4
F" }, { "b" : "7FDD545AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FDD543A7000", "path" : "/lib/x86_64-linux-gnu/libdl.so
.2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FDD5409E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B14
89" }, { "b" : "7FDD53E88000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FDD53C6B000", "path" : "/lib/x86_64-linux-gnu/libp
thread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FDD538A2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6
E4E68692053" }, { "b" : "7FDD54E60000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1533852]
 mongod(+0x11327F9) [0x15327f9]
 mongod(+0x1133062) [0x1533062]
 libpthread.so.0(+0x11390) [0x7fdd53c7c390]
 libc.so.6(gsignal+0x38) [0x7fdd538d7428]
 libc.so.6(abort+0x16A) [0x7fdd538d902a]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14b57f3]
 mongod(+0xE5B4D2) [0x125b4d2]
 mongod(+0x56F272) [0x96f272]
 mongod(__wt_err+0x8B) [0x96f343]
 mongod(__wt_panic+0x24) [0x96f509]
 mongod(+0x177681F) [0x1b7681f]
 mongod(+0x1776D7B) [0x1b76d7b]
 mongod(+0x177371D) [0x1b7371d]
 mongod(+0x1777EB7) [0x1b77eb7]
 mongod(+0x1794E6C) [0x1b94e6c]
 mongod(+0x17CA33B) [0x1bca33b]
 mongod(+0x1852EB2) [0x1c52eb2]
 mongod(+0x1853585) [0x1c53585]
 mongod(+0x1853A7C) [0x1c53a7c]
 mongod(+0x17D9AC1) [0x1bd9ac1]
 mongod(+0x1848D68) [0x1c48d68]
 mongod(+0x18148DF) [0x1c148df]
 mongod(+0x18149AC) [0x1c149ac]
 mongod(wiredtiger_open+0x15D6) [0x1bc6816]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xDD8) [0x123f568]
 mongod(+0xE38735) [0x1238735]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1133f38]
 mongod(_ZN5mongo13initAndListenEi+0x480) [0x9bba60]
 mongod(main+0x712) [0x970392]
 libc.so.6(__libc_start_main+0xF0) [0x7fdd538c2830]
 mongod(_start+0x29) [0x9b6089]
-----  END BACKTRACE  -----

In the logs there was an error by the BT checkbox, we fixed it by replacing the files 1



 Comments   
Comment by Pavel [X] [ 11/Apr/18 ]

Thank you, for your time, and your help!

Comment by Kelsey Schubert [ 10/Apr/18 ]

Hi iArcher,

Unfortunately, this error indicates that there was corruption on the disk, most often caused by a faulty storage layer. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thank you,
Kelsey

Comment by Pavel [X] [ 10/Apr/18 ]

Your files helped solve the problem with the error of the checks. But there is a new bug in the logs associated with the collection file.

2018-04-10T11:10:08.227+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] MongoDB starting : pid=6650 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb01.mm4.com
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] db version v3.2.12
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] modules: none
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] build environment:
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-10T11:10:08.234+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.21.176,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/va
r/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2018-04-10T11:10:08.258+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-10T11:10:08.258+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-10T11:10:08.258+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-10T11:10:08.258+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,p
ath=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-10T11:10:08.617+0000 E STORAGE  [initandlisten] WiredTiger (0) [1523358608:616997][6650:0x7fe3d8e8bc80], file:collection-6--595897270412410555.wt, txn-recover: read checksum error for 4096B block at offse
t 28721152: block header checksum of 2847978445 doesn't match expected checksum of 2937228681
2018-04-10T11:10:08.617+0000 E STORAGE  [initandlisten] WiredTiger (0) [1523358608:617051][6650:0x7fe3d8e8bc80], file:collection-6--595897270412410555.wt, txn-recover: collection-6--595897270412410555.wt: encount
ered an illegal file format or internal value
2018-04-10T11:10:08.617+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1523358608:617061][6650:0x7fe3d8e8bc80], file:collection-6--595897270412410555.wt, txn-recover: the process must exit and restart: WT_P
ANIC: WiredTiger library panic
2018-04-10T11:10:08.617+0000 I -        [initandlisten] Fatal Assertion 28558
2018-04-10T11:10:08.617+0000 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-04-10T11:10:08.641+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1533852 0x15327f9 0x1533062 0x7fe3d7a92390 0x7fe3d76ed428 0x7fe3d76ef02a 0x14b57f3 0x125b4d2 0x96f272 0x96f343 0x96f509 0x1b7681f 0x1b76d7b 0x1b7371d 0x1b77eb7 0x1b94e6c 0x1bca33b 0x1c52eb2 0x1c53585 0x1c53a7c 0x1bd9ac1 0x1c48d68 0x1c65fc0 0x1c664cf 0x1bfd827 0x1c66f07 0x1bce287 0x1bc682f 0x123f568 0x1238735 0x1133f38 0x9bba60 0x970392 0x7fe3d76d8830 0x9b6089
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1133852","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11327F9"},{"b":"400000","o":"1133062"},{"b":"7FE3D7A81000","o":"11390"},{"b":"7FE3D76B8000","o":"35428","s":"gsignal"},{"b":"7FE3D76B8000","o":"3702A","s":"abort"},{"b":"400000","o":"10B57F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E5B4D2"},{"b":"400000","o":"56F272"},{"b":"400000","o":"56F343","s":"__wt_err"},{"b":"400000","o":"56F509","s":"__wt_panic"},{"b":"400000","o":"177681F"},{"b":"400000","o":"1776D7B"},{"b":"400000","o":"177371D"},{"b":"400000","o":"1777EB7"},{"b":"400000","o":"1794E6C"},{"b":"400000","o":"17CA33B"},{"b":"400000","o":"1852EB2"},{"b":"400000","o":"1853585"},{"b":"400000","o":"1853A7C"},{"b":"400000","o":"17D9AC1"},{"b":"400000","o":"1848D68"},{"b":"400000","o":"1865FC0"},{"b":"400000","o":"18664CF"},{"b":"400000","o":"17FD827"},{"b":"400000","o":"1866F07"},{"b":"400000","o":"17CE287"},{"b":"400000","o":"17C682F","s":"wiredtiger_open"},{"b":"400000","o":"E3F568","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E38735"},{"b":"400000","o":"D33F38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5BBA60","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"570392","s":"main"},{"b":"7FE3D76B8000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B6089","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-66-generic", "version" : "#87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7877295E4A95F29C74796963664BD1115CFE251E" }, { "b" : "7FFC3E9E6000", "elfType" : 3, "buildId" : "5CCDAB3B3062DC906D26EBB5A688D41BFE5EFD75" }, { "b" : "7FE3D8A0D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FE3D85C9000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7FE3D83C1000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FE3D81BD000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FE3D7EB4000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B1489" }, { "b" : "7FE3D7C9E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FE3D7A81000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FE3D76B8000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6E4E68692053" }, { "b" : "7FE3D8C76000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1533852]
 mongod(+0x11327F9) [0x15327f9]
 mongod(+0x1133062) [0x1533062]
 libpthread.so.0(+0x11390) [0x7fe3d7a92390]
 libc.so.6(gsignal+0x38) [0x7fe3d76ed428]
 libc.so.6(abort+0x16A) [0x7fe3d76ef02a]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14b57f3]
 mongod(+0xE5B4D2) [0x125b4d2]
 mongod(+0x56F272) [0x96f272]
 mongod(__wt_err+0x8B) [0x96f343]
 mongod(__wt_panic+0x24) [0x96f509]
 mongod(+0x177681F) [0x1b7681f]
 mongod(+0x1776D7B) [0x1b76d7b]
 mongod(+0x177371D) [0x1b7371d]
 mongod(+0x1777EB7) [0x1b77eb7]
 mongod(+0x1794E6C) [0x1b94e6c]
 mongod(+0x17CA33B) [0x1bca33b]
 mongod(+0x1852EB2) [0x1c52eb2]
 mongod(+0x1853585) [0x1c53585]
 mongod(+0x1853A7C) [0x1c53a7c]
 mongod(+0x17D9AC1) [0x1bd9ac1]
 mongod(+0x1848D68) [0x1c48d68]
 mongod(+0x1865FC0) [0x1c65fc0]
 mongod(+0x18664CF) [0x1c664cf]
 mongod(+0x17FD827) [0x1bfd827]
 mongod(+0x1866F07) [0x1c66f07]
 mongod(+0x17CE287) [0x1bce287]
 mongod(wiredtiger_open+0x15EF) [0x1bc682f]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xDD8) [0x123f568]
 mongod(+0xE38735) [0x1238735]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1133f38]
 mongod(_ZN5mongo13initAndListenEi+0x480) [0x9bba60]
 mongod(main+0x712) [0x970392]
 libc.so.6(__libc_start_main+0xF0) [0x7fe3d76d8830]
 mongod(_start+0x29) [0x9b6089]
-----  END BACKTRACE  -----

Comment by Kelsey Schubert [ 09/Apr/18 ]

Hi iArcher,

I've attached a repair attempt, repair_attempt.tar.gz, of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node so that we can further investigate.

Thank you,
Kelsey

Comment by Pavel [X] [ 09/Apr/18 ]

We made copies of our files before they were replaced with new ones.
Here are our original files, thank you, Ramon.

WiredTiger.turtle WiredTiger.wt

Comment by Ramon Fernandez Marina [ 09/Apr/18 ]

iArcher, what do you mean by

we fixed it by replacing the files from adjacent ticket

If you took WiredTiger.wt and WiredTiger.turtle from a different SERVER ticket and overwrote yours then your dataset is beyond repair, and at this stage you should start a new cluster and restore from backups (assuming you don't have a healthy replica set node).

If you still have the original WiredTiger.wt and WiredTiger.turtle files from when you first had this issue, please upload them to this ticket so we can run a repair attempt, but please there are no guarantees of success.

Thanks,
Ramón.

Comment by Pavel [X] [ 06/Apr/18 ]

When I was try repair mongo, i have next

mongod --dbpath /var/lib/mongodb --repair

Show all

2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] MongoDB starting : pid=6702 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] db version v3.2.12
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] modules: none
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] build environment:
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-06T00:56:27.757+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
2018-04-06T00:56:27.785+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-06T00:56:27.785+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-06T00:56:27.785+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-06T00:56:27.785+0000 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-04-06T00:56:27.785+0000 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-06T00:56:27.827+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522976187:827578][6702:0x7fd30dac7c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 118784: block header checksum of 1667457286 doesn't match expected checksum of 4182632751
2018-04-06T00:56:27.827+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522976187:827651][6702:0x7fd30dac7c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2018-04-06T00:56:27.827+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1522976187:827660][6702:0x7fd30dac7c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-06T00:56:27.827+0000 I -        [initandlisten] Fatal Assertion 28558
2018-04-06T00:56:27.827+0000 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-04-06T00:56:27.849+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1533852 0x15327f9 0x1533062 0x7fd30c6cf390 0x7fd30c32a428 0x7fd30c32c02a 0x14b57f3 0x125b4d2 0x96f272 0x96f343 0x96f509 0x1b7681f 0x1b76d7b 0x1b7371d 0x1b77eb7 0x1b94e6c 0x1bca33b 0x1c52eb2 0x1c53585 0x1c53a7c 0x1bd9ac1 0x1c48d68 0x1c148df 0x1c149ac 0x1bc6816 0x123f35e 0x1238735 0x1133f38 0x9bba60 0x970392 0x7fd30c315830 0x9b6089
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1133852","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11327F9"},{"b":"400000","o":"1133062"},{"b":"7FD30C6BE000","o":"11390"},{"b":"7FD30C2F5000","o":"35428","s":"gsignal"},{"b":"7FD30C2F5000","o":"3702A","s":"abort"},{"b":"400000","o":"10B57F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E5B4D2"},{"b":"400000","o":"56F272"},{"b":"400000","o":"56F343","s":"__wt_err"},{"b":"400000","o":"56F509","s":"__wt_panic"},{"b":"400000","o":"177681F"},{"b":"400000","o":"1776D7B"},{"b":"400000","o":"177371D"},{"b":"400000","o":"1777EB7"},{"b":"400000","o":"1794E6C"},{"b":"400000","o":"17CA33B"},{"b":"400000","o":"1852EB2"},{"b":"400000","o":"1853585"},{"b":"400000","o":"1853A7C"},{"b":"400000","o":"17D9AC1"},{"b":"400000","o":"1848D68"},{"b":"400000","o":"18148DF"},{"b":"400000","o":"18149AC"},{"b":"400000","o":"17C6816","s":"wiredtiger_open"},{"b":"400000","o":"E3F35E","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E38735"},{"b":"400000","o":"D33F38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5BBA60","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"570392","s":"main"},{"b":"7FD30C2F5000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B6089","s":"_start"}],"processInfo":{
 "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-66-generic", "version" : "#87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7877295E4A95F29C74796963664BD1115CFE251E" }, { "b" : "7FFE7C49E000", "elfType" : 3, "buildId" : "5CCDAB3B3062DC906D26EBB5A688D41BFE5EFD75" }, { "b" : "7FD30D64A000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FD30D206000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7FD30CFFE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FD30CDFA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FD30CAF1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B1489" }, { "b" : "7FD30C8DB000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FD30C6BE000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FD30C2F5000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6E4E68692053" }, { "b" : "7FD30D8B3000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1533852]
 mongod(+0x11327F9) [0x15327f9]
 mongod(+0x1133062) [0x1533062]
 libpthread.so.0(+0x11390) [0x7fd30c6cf390]
 libc.so.6(gsignal+0x38) [0x7fd30c32a428]
 libc.so.6(abort+0x16A) [0x7fd30c32c02a]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14b57f3]
 mongod(+0xE5B4D2) [0x125b4d2]
 mongod(+0x56F272) [0x96f272]
 mongod(__wt_err+0x8B) [0x96f343]
 mongod(__wt_panic+0x24) [0x96f509]
 mongod(+0x177681F) [0x1b7681f]
 mongod(+0x1776D7B) [0x1b76d7b]
 mongod(+0x177371D) [0x1b7371d]
 mongod(+0x1777EB7) [0x1b77eb7]
 mongod(+0x1794E6C) [0x1b94e6c]
 mongod(+0x17CA33B) [0x1bca33b]
 mongod(+0x1852EB2) [0x1c52eb2]
 mongod(+0x1853585) [0x1c53585]
 mongod(+0x1853A7C) [0x1c53a7c]
 mongod(+0x17D9AC1) [0x1bd9ac1]
 mongod(+0x1848D68) [0x1c48d68]
 mongod(+0x18148DF) [0x1c148df]
 mongod(+0x18149AC) [0x1c149ac]
 mongod(wiredtiger_open+0x15D6) [0x1bc6816]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xBCE) [0x123f35e]
 mongod(+0xE38735) [0x1238735]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1133f38]
 mongod(_ZN5mongo13initAndListenEi+0x480) [0x9bba60]
 mongod(main+0x712) [0x970392]
 libc.so.6(__libc_start_main+0xF0) [0x7fd30c315830]
 mongod(_start+0x29) [0x9b6089]
-----  END BACKTRACE  -----
Aborted (core dumped)

Comment by Pavel [X] [ 06/Apr/18 ]

Sorry for first comment, con`t find where edit button.

After replace WiredToger.wt and WiredToger.turtle we have log with no specify error


Show all

{noformat}
2018-04-06T08:30:58.117+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] MongoDB starting : pid=25397 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] db version v3.2.12
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] modules: none
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] build environment:
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-06T08:30:58.122+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2018-04-06T08:30:58.147+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-06T08:30:58.147+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-06T08:30:58.147+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-06T08:30:58.147+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-06T08:30:58.188+0000 E STORAGE  [initandlisten] WiredTiger (-31803) [1523003458:188322][25397:0x7fe7e5a4fc80], txn-recover: Recovery failed: WT_NOTFOUND: item not found
2018-04-06T08:30:58.189+0000 I -        [initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
2018-04-06T08:30:58.193+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating
2018-04-06T08:30:58.193+0000 I CONTROL  [initandlisten] dbexit:  rc: 100
{noformat}

Generated at Thu Feb 08 04:36:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.