[SERVER-43799] WiredTiger library panic Created: 03/Oct/19  Updated: 04/Oct/19  Resolved: 04/Oct/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.22
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Vlad Kovalchuk Assignee: Carl Champain (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File repair_attempt.tgz    
Operating System: ALL
Participants:

 Description   

Hello!

After backuping of my ubuntu 18 server i can't run my mongo db cause of WiredTiger error i think

 

running:

mongod --repair --dbpath /var/lib/mongodb/

output:

2019-10-03T14:25:40.981+0000 I CONTROL  [initandlisten] MongoDB starting : pid=2539 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=Dobroplat
2019-10-03T14:25:40.981+0000 I CONTROL  [initandlisten] db version v3.2.22
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] git version: 105acca0d443f9a47c1a5bd608fd7133840a58dd
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2n  7 Dec 2017
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] modules: none
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] build environment:
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } }
2019-10-03T14:25:41.004+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-10-03T14:25:41.004+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2019-10-03T14:25:41.005+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-10-03T14:25:41.005+0000 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2019-10-03T14:25:41.005+0000 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2019-10-03T14:25:41.026+0000 E STORAGE  [initandlisten] WiredTiger (0) [1570112741:26536][2539:0x7f71c26d6100], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 114688: block header checksum of 1635017060 doesn't match expected checksum of 850816007
2019-10-03T14:25:41.026+0000 E STORAGE  [initandlisten] WiredTiger (0) [1570112741:26733][2539:0x7f71c26d6100], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2019-10-03T14:25:41.026+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1570112741:26825][2539:0x7f71c26d6100], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-10-03T14:25:41.026+0000 I -        [initandlisten] Fatal Assertion 28558
2019-10-03T14:25:41.026+0000 I -        [initandlisten]***aborting after fassert() failure
2019-10-03T14:25:41.044+0000 F -        [initandlisten] Got signal: 6 (Aborted). 0x155c5e2 0x155b589 0x155bdf2 0x7f71c1241890 0x7f71c0e7ce97 0x7f71c0e7e801 0x14d85f3 0x12787aa 0x97e457 0x97e63d 0x97e80d 0x1ba0caf 0x1ba120b 0x1b9daad 0x1ba2367 0x1bbb6cd 0x1bf3a9b 0x1c79bcd 0x1c7a0f9 0x1c7a39c 0x1c01ae1 0x1c76a68 0x1c3bbdf 0x1c3bc9c 0x1bf012c 0x125d1ca 0x1255cb5 0x1150ebe 0x9cbb91 0x97f5ea 0x7f71c0e5fb97 0x9c6e89
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"115C5E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"115B589"},{"b":"400000","o":"115BDF2"},{"b":"7F71C122F000","o":"12890"},{"b":"7F71C0E3E000","o":"3EE97","s":"gsignal"},{"b":"7F71C0E3E000","o":"40801","s":"abort"},{"b":"400000","o":"10D85F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E787AA"},{"b":"400000","o":"57E457","s":"__wt_eventv"},{"b":"400000","o":"57E63D","s":"__wt_err"},{"b":"400000","o":"57E80D","s":"__wt_panic"},{"b":"400000","o":"17A0CAF","s":"__wt_block_extlist_read"},{"b":"400000","o":"17A120B","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"179DAAD","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"17A2367"},{"b":"400000","o":"17BB6CD","s":"__wt_btree_open"},{"b":"400000","o":"17F3A9B","s":"__wt_conn_btree_open"},{"b":"400000","o":"1879BCD","s":"__wt_session_get_btree"},{"b":"400000","o":"187A0F9","s":"__wt_session_get_btree"},{"b":"400000","o":"187A39C","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1801AE1","s":"__wt_curfile_open"},{"b":"400000","o":"1876A68"},{"b":"400000","o":"183BBDF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"183BC9C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17F012C","s":"wiredtiger_open"},{"b":"400000","o":"E5D1CA","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E55CB5"},{"b":"400000","o":"D50EBE","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5CBB91"},{"b":"400000","o":"57F5EA","s":"main"},{"b":"7F71C0E3E000","o":"21B97","s":"__libc_start_main"},{"b":"400000","o":"5C6E89","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.22", "gitVersion" : "105acca0d443f9a47c1a5bd608fd7133840a58dd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-62-generic", "version" : "#69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C2070FF92CF0E7C7AF25D84027F691037262CEA2" }, { "b" : "7FFFCDEDE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "17ABBA294C8661C069F5408AC1D74F121AD71734" }, { "b" : "7F71C2253000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0D054641049B9747C05D030262295DFDFDD3055D" }, { "b" : "7F71C1E10000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "9C228817BA6E0730F4FCCFAC6E033BD1E0C5620A" }, { "b" : "7F71C1C08000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F71C1A04000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F71C1666000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F71C144E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F71C122F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F71C0E3E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F71C24BB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x155c5e2]
 mongod(+0x115B589) [0x155b589]
 mongod(+0x115BDF2) [0x155bdf2]
 libpthread.so.0(+0x12890) [0x7f71c1241890]
 libc.so.6(gsignal+0xC7) [0x7f71c0e7ce97]
 libc.so.6(abort+0x141) [0x7f71c0e7e801]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14d85f3]
 mongod(+0xE787AA) [0x12787aa]
 mongod(__wt_eventv+0x3BA) [0x97e457]
 mongod(__wt_err+0x8B) [0x97e63d]
 mongod(__wt_panic+0x2E) [0x97e80d]
 mongod(__wt_block_extlist_read+0x6F) [0x1ba0caf]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x1ba120b]
 mongod(__wt_block_checkpoint_load+0x24D) [0x1b9daad]
 mongod(+0x17A2367) [0x1ba2367]
 mongod(__wt_btree_open+0xB0D) [0x1bbb6cd]
 mongod(__wt_conn_btree_open+0x16B) [0x1bf3a9b]
 mongod(__wt_session_get_btree+0xDD) [0x1c79bcd]
 mongod(__wt_session_get_btree+0x609) [0x1c7a0f9]
 mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c7a39c]
 mongod(__wt_curfile_open+0x151) [0x1c01ae1]
 mongod(+0x1876A68) [0x1c76a68]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1c3bbdf]
 mongod(__wt_metadata_cursor+0x3C) [0x1c3bc9c]
 mongod(wiredtiger_open+0x18DC) [0x1bf012c]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0x86A) [0x125d1ca]
 mongod(+0xE55CB5) [0x1255cb5]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x1150ebe]
 mongod(+0x5CBB91) [0x9cbb91]
 mongod(main+0x73A) [0x97f5ea]
 libc.so.6(__libc_start_main+0xE7) [0x7f71c0e5fb97]
 mongod(_start+0x29) [0x9c6e89]
-----  END BACKTRACE  -----
Aborted (core dumped)

 

Thanks!



 Comments   
Comment by Vlad Kovalchuk [ 04/Oct/19 ]

Oh it works now!
Really thank you, Carl!

Comment by Carl Champain (Inactive) [ 04/Oct/19 ]

Hi dobro322@yandex.ru,

Thanks for the report.

I've attached a repair attempt of the files you provided, repair_attempt.tgz. Please extract these files and replace them in the $dbpath. I hope this will resolve your issue.

Importantly, 3.2.22 version of MongoDB is already end-of-lifed, you should upgrade to the latest 4.2.0 or 4.0.12. I am going to close this ticket as your version of MongoDB is end-of-lifed.

Kind regards,
Carl

 

 

Generated at Thu Feb 08 05:04:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.