[SERVER-32998] Mongodb Crashed with the Got signal: 6 (Aborted) Created: 30/Jan/18  Updated: 14/Aug/18  Resolved: 01/Feb/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dingfu ye Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envc, rpo, rpu, trcf, vz, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt.7z     File repair-SERVER-32998.tar.gz    
Operating System: Linux
Steps To Reproduce:

The server is powered off and then mongo can't startup.

Participants:

 Description   

2018-01-30T12:51:40.475+0800 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] MongoDB starting : pid=2621 port=27017 dbpath=/var/lib/mongodb 64-bit host=db
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] db version v3.2.7
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] modules: none
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] build environment:
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     distmod: debian71
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     distarch: x86_64
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, syst
emLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-01-30T12:51:40.503+0800 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTige
r'.
2018-01-30T12:51:40.503+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabl
ed=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (0) [1517287900:511844][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: read checksum error for 8192B block at offset
 15413248: block header checksum of 3970207674 doesn't match expected checksum of 1545588312
2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (0) [1517287900:511873][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file fo
rmat or internal value
2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1517287900:511888][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PA
NIC: WiredTiger library panic
2018-01-30T12:51:40.511+0800 I -        [initandlisten] Fatal Assertion 28558
2018-01-30T12:51:40.511+0800 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-01-30T12:51:40.533+0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x13043c2 0x13032e9 0x1303af2 0x7ff38993b0a0 0x7ff3895d1125 0x7ff3895d43a0 0x128a3b2 0x1072af3 0x1a7938c 0x1a7984d 0x1a79c34 0x19aa04c 0x19aa5c3 0x19a7555 0x19ab609 0x19c807c 0x19ff3d0 0x1a
780b7 0x1a785f9 0x1a7871b 0x1a0da08 0x1a75275 0x1a3fdbf 0x1a3febe 0x19fb9c1 0x105a6c7 0x1056a43 0xf7c258 0x949b1d 0x94d71d 0x7ff3895bdead 0x946279
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F043C2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F032E9"},{"b":"400000","o":"F03AF2"},{"b":"7FF38992C000","o":"F0A0"},{"b":"7FF38959F000","o"
:"32125","s":"gsignal"},{"b":"7FF38959F000","o":"353A0","s":"abort"},{"b":"400000","o":"E8A3B2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C72AF3"},{"b":"400000","o":"167938C","s":
"__wt_eventv"},{"b":"400000","o":"167984D","s":"__wt_err"},{"b":"400000","o":"1679C34","s":"__wt_panic"},{"b":"400000","o":"15AA04C","s":"__wt_block_extlist_read"},{"b":"400000","o":"15AA5C3
","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15A7555","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15AB609"},{"b":"400000","o":"15C807C","s":"__wt_btree_open"},{"b":"400000","o":"15FF3D0","s":"__wt_conn_btree_open"},{"b":"400000","o":"16780B7","s":"__wt_session_get_btree"},{"b":"400000","o":"16785F9","s":"__wt_session_get_btree"},{"b":"400000","o":"167871B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"160DA08","s":"__wt_curfile_open"},{"b":"400000","o":"1675275"},{"b":"400000","o":"163FDBF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"163FEBE","s":"__wt_metadata_cursor"},{"b":"400000","o":"15FB9C1","s":"wiredtiger_open"},{"b":"400000","o":"C5A6C7","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C56A43"},{"b":"400000","o":"B7C258","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"549B1D","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54D71D","s":"main"},{"b":"7FF38959F000","o":"1EEAD","s":"__libc_start_main"},{"b":"400000","o":"546279"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-13-pve", "version" : "#1 SMP Mon Jul 9 08:39:20 CEST 2012", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "290161E49FF073654474160230025167BE1051AF" }, { "b" : "7FFF098FE000", "elfType" : 3, "buildId" : "1017EA1FFF5B14575F12B48EB997CEC1B048763C" }, { "b" : "7FF38A7E5000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "16DA62C831D8080E3BCB9A1E7738D399EA3A89C0" }, { "b" : "7FF38A3EC000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "079D784F9F4A88B6D87FC3E6B0B07BFBE209540E" }, { "b" : "7FF38A1E4000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "07CC333722DC4237B5F88E47A38C214FDA0B9B59" }, { "b" : "7FF389FE0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "028546D6F23D0FD259928FED6EFE53E64782EF2D" }, { "b" : "7FF389D5E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "2C5446FA41E784A1C6FDD528AB2BCC7BCA9EF06C" }, { "b" : "7FF389B48000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7FF38992C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "3F3BFFEDAF77C955862785C6CBA7670F82A05563" }, { "b" : "7FF38959F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "50B9FD83D874F18E3B385E0349FBD5D1281A19E3" }, { "b" : "7FF38AA44000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "37BC6FC6FF3FA04B0A3EC20DFF215AB55EA5F5CB" }, { "b" : "7FF389388000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13043c2]
 mongod(+0xF032E9) [0x13032e9]
 mongod(+0xF03AF2) [0x1303af2]
 libpthread.so.0(+0xF0A0) [0x7ff38993b0a0]
 libc.so.6(gsignal+0x35) [0x7ff3895d1125]
 libc.so.6(abort+0x180) [0x7ff3895d43a0]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x128a3b2]
 mongod(+0xC72AF3) [0x1072af3]
 mongod(__wt_eventv+0x42C) [0x1a7938c]
 mongod(__wt_err+0x8D) [0x1a7984d]
 mongod(__wt_panic+0x24) [0x1a79c34]
 mongod(__wt_block_extlist_read+0x6C) [0x19aa04c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19aa5c3]
 mongod(__wt_block_checkpoint_load+0x3C5) [0x19a7555]
 mongod(+0x15AB609) [0x19ab609]
 mongod(__wt_btree_open+0xC7C) [0x19c807c]
 mongod(__wt_conn_btree_open+0x140) [0x19ff3d0]
 mongod(__wt_session_get_btree+0xE7) [0x1a780b7]
 mongod(__wt_session_get_btree+0x629) [0x1a785f9]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7871b]
 mongod(__wt_curfile_open+0x218) [0x1a0da08]
 mongod(+0x1675275) [0x1a75275]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a3fdbf]
 mongod(__wt_metadata_cursor+0x7E) [0x1a3febe]
 mongod(wiredtiger_open+0x1541) [0x19fb9c1]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x567) [0x105a6c7]
 mongod(+0xC56A43) [0x1056a43]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf7c258]
 mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x949b1d]
 mongod(main+0x15D) [0x94d71d]
 libc.so.6(__libc_start_main+0xFD) [0x7ff3895bdead]
 mongod(+0x546279) [0x946279]
-----  END BACKTRACE  -----



 Comments   
Comment by Mark Agarunov [ 01/Feb/18 ]

Hello rory,

Unfortunately, this indicates that there was corruption on the disk, most often cause by a faulty storage layer. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thanks,
Mark

Comment by Dingfu ye [ 01/Feb/18 ]

Mark Agarunov, Thanks.

But no lucky.

Openvz is not recommended?

Comment by Mark Agarunov [ 31/Jan/18 ]

Hello rory,

Thank you for your report. I've attached a repair attempt of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node(s) so that we can further investigate.

Thanks,
Mark

Comment by Dingfu ye [ 31/Jan/18 ]

@mark.agarunov Thanks, please check the WiredTiger.wt.7z file.

Comment by Mark Agarunov [ 30/Jan/18 ]

Hello rory,

Thank you for the report. So that we can attempt a repair, could you please provide the WiredTiger.wt file in addition to the WiredTiger.turtle file you've already provided?

Thanks,
Mark

Comment by Dingfu ye [ 30/Jan/18 ]

root@db:~# ulimit -n
30000
root@db:~# cat /etc/issue
Debian GNU/Linux 7 \n \l

It's a OpenVZ container.

Comment by Dingfu ye [ 30/Jan/18 ]

root@db:~# uname -a
Linux db 2.6.32-13-pve #1 SMP Mon Jul 9 08:39:20 CEST 2012 x86_64 GNU/Linux
root@db:~# mongo --version
MongoDB shell version: 3.2.7

Generated at Thu Feb 08 04:31:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.