[SERVER-38665] Mongo WiredTiger.turtle: encountered an illegal file format or internal value on startup after unexpected shutdown Created: 17/Dec/18  Updated: 18/Dec/18  Resolved: 18/Dec/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Schell Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File _mdb_catalog.wt     File repair_attempt.tar     File sizeStorer.wt    
Operating System: ALL
Participants:

 Description   

After the a Hyper-V virtual machine running RHEL and mongod (v3.4.18) shutdown expectantly when the host was patched the database can no longer be started. 

The last backup was a couple of weeks ago, and was done via Mongodump so I'm unable to recover the original metadata files to attempt to the wt restore method.
Looking online this appears to occur when the database metadata files have become corrupt, and in the past it's been recommended to upload the attached files incase the developers are able to manually repair them.

If this is possible it would be much appreciated.



 Comments   
Comment by Danny Hatcher (Inactive) [ 18/Dec/18 ]

Hello Daniel,

Unfortunately, it does look like you need to recover from your backup. In order to avoid issues like this in the future:

  • Ensure that the mongod process is cleanly shutdown before performing any maintenance on the VM that holds the process.
  • Use a replica set as you will still have intact copies of your data even if one node gets corrupted.

Thank you,

Danny

Comment by Daniel Schell [ 18/Dec/18 ]

Thanks for trying, it looks like the repair hasn't worked. We'll have to restore the previous database.

 

mongod --dbpath /data/db/ -directoryperdb --repair
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] MongoDB starting : pid=1572 port=27017 dbpath=/data/db/ 64-bit host=localhost.localdomain
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] db version v3.4.18
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] git version: 4410706bef6463369ea2f42399e9843903b31923
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] allocator: tcmalloc
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] modules: none
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] build environment:
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] distmod: rhel70
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] distarch: x86_64
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] target_arch: x86_64
2018-12-18T11:12:52.773+1030 I CONTROL [initandlisten] options: { repair: true, storage:

{ dbPath: "/data/db/", directoryPerDB: true }

}
2018-12-18T11:12:52.788+1030 I - [initandlisten] Detected data files in /data/db/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-12-18T11:12:52.788+1030 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2018-12-18T11:12:52.788+1030 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-12-18T11:12:52.807+1030 E STORAGE [initandlisten] WiredTiger error (0) [1545093772:807666][1572:0x7fc25cef8e80], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-12-18T11:12:52.807+1030 E STORAGE [initandlisten] WiredTiger error (-31804) [1545093772:807699][1572:0x7fc25cef8e80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-12-18T11:12:52.807+1030 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
2018-12-18T11:12:52.807+1030 I - [initandlisten]

***aborting after fassert() failure

2018-12-18T11:12:52.819+1030 F - [initandlisten] Got signal: 6 (Aborted).

0x555f044b2cb1 0x555f044b1ec9 0x555f044b23ad 0x7fc25bae45d0 0x7fc25b73e207 0x7fc25b73f8f8 0x555f037432d5 0x555f041b6456 0x555f0374da2a 0x555f0374dc46 0x555f0374dea8 0x555f04e6253c 0x555f04e60b38 0x555f04e15503 0x555f04ea001b 0x555f04ea055d 0x555f04ea081c 0x555f04e23d81 0x555f04e96308 0x555f04e601ee 0x555f04e602cb 0x555f04e11abb 0x555f0419ad8a 0x555f04193075 0x555f04084757 0x555f0372f66e 0x555f0374f25b 0x7fc25b72a3d5 0x555f037af1df
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"555F02F24000","o":"158ECB1","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"555F02F24000","o":"158DEC9"},{"b":"555F02F24000","o":"158E3AD"},{"b":"7FC25BAD5000","o":"F5D0"},{"b":"7FC25B708000","o":"36207","s":"gsignal"},{"b":"7FC25B708000","o":"378F8","s":"abort"},{"b":"555F02F24000","o":"81F2D5","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"555F02F24000","o":"1292456"},{"b":"555F02F24000","o":"829A2A","s":"wt_eventv"},{"b":"555F02F24000","o":"829C46","s":"wt_err"},{"b":"555F02F24000","o":"829EA8","s":"wt_panic"},{"b":"555F02F24000","o":"1F3E53C","s":"wt_turtle_read"},{"b":"555F02F24000","o":"1F3CB38","s":"wt_metadata_search"},{"b":"555F02F24000","o":"1EF1503","s":"wt_conn_btree_open"},{"b":"555F02F24000","o":"1F7C01B","s":"wt_session_get_btree"},{"b":"555F02F24000","o":"1F7C55D","s":"wt_session_get_btree"},{"b":"555F02F24000","o":"1F7C81C","s":"wt_session_get_btree_ckpt"},{"b":"555F02F24000","o":"1EFFD81","s":"wt_curfile_open"},{"b":"555F02F24000","o":"1F72308"},{"b":"555F02F24000","o":"1F3C1EE","s":"wt_metadata_cursor_open"},{"b":"555F02F24000","o":"1F3C2CB","s":"wt_metadata_cursor"},{"b":"555F02F24000","o":"1EEDABB","s":"wiredtiger_open"},{"b":"555F02F24000","o":"1276D8A","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"555F02F24000","o":"126F075"},{"b":"555F02F24000","o":"1160757","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"555F02F24000","o":"80B66E"},{"b":"555F02F24000","o":"82B25B","s":"main"},{"b":"7FC25B708000","o":"223D5","s":"_libc_start_main"},{"b":"555F02F24000","o":"88B1DF"}],"processInfo":{ "mongodbVersion" : "3.4.18", "gitVersion" : "4410706bef6463369ea2f42399e9843903b31923", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "3.10.0-862.el7.x86_64", "version" : "#1 SMP Fri Apr 20 16:44:24 UTC 2018", "machine" : "x86_64" }

, "somap" : [ { "b" : "555F02F24000", "elfType" : 3, "buildId" : "2346B88195258DF708C923592FA0F13714798067" }, { "b" : "7FFD0D4D9000", "elfType" : 3, "buildId" : "463512F624A152723F0EFEED311DB2A05ACEF3D6" }, { "b" : "7FC25CA76000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, { "b" : "7FC25C615000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, { "b" : "7FC25C40D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7FC25C209000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "67AD3498AC7DE3EAB952A243094DF5C12A21CD7D" }, { "b" : "7FC25BF07000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "918D3696BF321AA8D32950AB2AB8D0F1B21AC907" }, { "b" : "7FC25BCF1000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6B4F3D896CD0F06FCB3DEF0245F204ECE3220D7E" }, { "b" : "7FC25BAD5000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3D9441083D079DC2977F1BD50C8068D11767232D" }, { "b" : "7FC25B708000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3C61131D1DAC9DA79B73188E7702BEF786C2AD54" }, { "b" : "7FC25CCE8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5DA2D47925497B2F5875A7D8D1799A1227E2FDE4" }, { "b" : "7FC25B4BB000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "39E6427542C05B1019EC312345239164CCFE84DF" }, { "b" : "7FC25B1D3000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "36971F0AF2B69B25C7F9589CDE3A82D34A60C591" }, { "b" : "7FC25AFCF000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5025A7A2E8AECA9ACB996121370821B9CAA9E397" }, { "b" : "7FC25AD9C000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "9891D2B285C6D5E761DFE7AEE8BAD629EA73D061" }, { "b" : "7FC25AB86000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7FC25A978000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AA19B311D788A700456C8A7E1C727F8609CFB302" }, { "b" : "7FC25A774000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FC25A55B000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4C488F6E7044BB966162C1F7081ABBA6EBB2B485" }, { "b" : "7FC25A334000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "1B0C0DC4F9054C5922082921C1447739F27AC8F4" }, { "b" : "7FC25A0D2000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x555f044b2cb1]
mongod(+0x158DEC9) [0x555f044b1ec9]
mongod(+0x158E3AD) [0x555f044b23ad]
libpthread.so.0(+0xF5D0) [0x7fc25bae45d0]
libc.so.6(gsignal+0x37) [0x7fc25b73e207]
libc.so.6(abort+0x148) [0x7fc25b73f8f8]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x555f037432d5]
mongod(+0x1292456) [0x555f041b6456]
mongod(__wt_eventv+0x3D7) [0x555f0374da2a]
mongod(__wt_err+0x9D) [0x555f0374dc46]
mongod(__wt_panic+0x2E) [0x555f0374dea8]
mongod(__wt_turtle_read+0x26C) [0x555f04e6253c]
mongod(__wt_metadata_search+0xE8) [0x555f04e60b38]
mongod(__wt_conn_btree_open+0x73) [0x555f04e15503]
mongod(__wt_session_get_btree+0xFB) [0x555f04ea001b]
mongod(__wt_session_get_btree+0x63D) [0x555f04ea055d]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x555f04ea081c]
mongod(__wt_curfile_open+0x161) [0x555f04e23d81]
mongod(+0x1F72308) [0x555f04e96308]
mongod(__wt_metadata_cursor_open+0x6E) [0x555f04e601ee]
mongod(__wt_metadata_cursor+0x4B) [0x555f04e602cb]
mongod(wiredtiger_open+0x196B) [0x555f04e11abb]
mongod(ZN5mongo18WiredTigerKVEngineC2ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xAFA) [0x555f0419ad8a]
mongod(+0x126F075) [0x555f04193075]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x697) [0x555f04084757]
mongod(+0x80B66E) [0x555f0372f66e]
mongod(main+0x96B) [0x555f0374f25b]
libc.so.6(__libc_start_main+0xF5) [0x7fc25b72a3d5]
mongod(+0x88B1DF) [0x555f037af1df]
----- END BACKTRACE -----
Aborted

Comment by Danny Hatcher (Inactive) [ 17/Dec/18 ]

Hello Daniel,

Do you have other healthy replica set members? If so, that is the recommended method to recover from this problem. If you do not, I have attached repair_attempt.tar. Please extract these files and replace the equivalent files under your $dbpath.

Thank you,

Danny

Comment by Daniel Schell [ 17/Dec/18 ]

[root@localhost db]# mongod --dbpath /data/db/ -directoryperdb --repair
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] MongoDB starting : pid=1296 port=27017 dbpath=/data/db/ 64-bit host=localhost.localdomain
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] db version v3.4.18
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] git version: 4410706bef6463369ea2f42399e9843903b31923
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] allocator: tcmalloc
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] modules: none
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] build environment:
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] distmod: rhel70
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] distarch: x86_64
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] target_arch: x86_64
2018-12-17T14:52:07.005+1030 I CONTROL [initandlisten] options: \{ repair: true, storage: { dbPath: "/data/db/", directoryPerDB: true } }
2018-12-17T14:52:07.019+1030 I - [initandlisten] Detected data files in /data/db/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-12-17T14:52:07.019+1030 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2018-12-17T14:52:07.019+1030 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-12-17T14:52:07.023+1030 E STORAGE [initandlisten] WiredTiger error (0) [1545020527:23397][1296:0x7f3dd7f8de80], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-12-17T14:52:07.023+1030 E STORAGE [initandlisten] WiredTiger error (-31804) [1545020527:23434][1296:0x7f3dd7f8de80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-12-17T14:52:07.023+1030 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
2018-12-17T14:52:07.023+1030 I - [initandlisten]
 
***aborting after fassert() failure
 
 
2018-12-17T14:52:07.035+1030 F - [initandlisten] Got signal: 6 (Aborted).
 
0x560afaa90cb1 0x560afaa8fec9 0x560afaa903ad 0x7f3dd6b795d0 0x7f3dd67d3207 0x7f3dd67d48f8 0x560af9d212d5 0x560afa794456 0x560af9d2ba2a 0x560af9d2bc46 0x560af9d2bea8 0x560afb44053c 0x560afb43eb38 0x560afb3f3503 0x560afb47e01b 0x560afb47e55d 0x560afb47e81c 0x560afb401d81 0x560afb474308 0x560afb43e1ee 0x560afb43e2cb 0x560afb3efabb 0x560afa778d8a 0x560afa771075 0x560afa662757 0x560af9d0d66e 0x560af9d2d25b 0x7f3dd67bf3d5 0x560af9d8d1df
----- BEGIN BACKTRACE -----
{"backtrace":[\{"b":"560AF9502000","o":"158ECB1","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"560AF9502000","o":"158DEC9"},\{"b":"560AF9502000","o":"158E3AD"},\{"b":"7F3DD6B6A000","o":"F5D0"},\{"b":"7F3DD679D000","o":"36207","s":"gsignal"},\{"b":"7F3DD679D000","o":"378F8","s":"abort"},\{"b":"560AF9502000","o":"81F2D5","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"560AF9502000","o":"1292456"},\{"b":"560AF9502000","o":"829A2A","s":"__wt_eventv"},\{"b":"560AF9502000","o":"829C46","s":"__wt_err"},\{"b":"560AF9502000","o":"829EA8","s":"__wt_panic"},\{"b":"560AF9502000","o":"1F3E53C","s":"__wt_turtle_read"},\{"b":"560AF9502000","o":"1F3CB38","s":"__wt_metadata_search"},\{"b":"560AF9502000","o":"1EF1503","s":"__wt_conn_btree_open"},\{"b":"560AF9502000","o":"1F7C01B","s":"__wt_session_get_btree"},\{"b":"560AF9502000","o":"1F7C55D","s":"__wt_session_get_btree"},\{"b":"560AF9502000","o":"1F7C81C","s":"__wt_session_get_btree_ckpt"},\{"b":"560AF9502000","o":"1EFFD81","s":"__wt_curfile_open"},\{"b":"560AF9502000","o":"1F72308"},\{"b":"560AF9502000","o":"1F3C1EE","s":"__wt_metadata_cursor_open"},\{"b":"560AF9502000","o":"1F3C2CB","s":"__wt_metadata_cursor"},\{"b":"560AF9502000","o":"1EEDABB","s":"wiredtiger_open"},\{"b":"560AF9502000","o":"1276D8A","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},\{"b":"560AF9502000","o":"126F075"},\{"b":"560AF9502000","o":"1160757","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},\{"b":"560AF9502000","o":"80B66E"},\{"b":"560AF9502000","o":"82B25B","s":"main"},\{"b":"7F3DD679D000","o":"223D5","s":"__libc_start_main"},\{"b":"560AF9502000","o":"88B1DF"}],"processInfo":\{ "mongodbVersion" : "3.4.18", "gitVersion" : "4410706bef6463369ea2f42399e9843903b31923", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-862.el7.x86_64", "version" : "#1 SMP Fri Apr 20 16:44:24 UTC 2018", "machine" : "x86_64" }, "somap" : [ \{ "b" : "560AF9502000", "elfType" : 3, "buildId" : "2346B88195258DF708C923592FA0F13714798067" }, \{ "b" : "7FFC4F8F1000", "elfType" : 3, "buildId" : "463512F624A152723F0EFEED311DB2A05ACEF3D6" }, \{ "b" : "7F3DD7B0B000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, \{ "b" : "7F3DD76AA000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, \{ "b" : "7F3DD74A2000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, \{ "b" : "7F3DD729E000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "67AD3498AC7DE3EAB952A243094DF5C12A21CD7D" }, \{ "b" : "7F3DD6F9C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "918D3696BF321AA8D32950AB2AB8D0F1B21AC907" }, \{ "b" : "7F3DD6D86000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6B4F3D896CD0F06FCB3DEF0245F204ECE3220D7E" }, \{ "b" : "7F3DD6B6A000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3D9441083D079DC2977F1BD50C8068D11767232D" }, \{ "b" : "7F3DD679D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3C61131D1DAC9DA79B73188E7702BEF786C2AD54" }, \{ "b" : "7F3DD7D7D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5DA2D47925497B2F5875A7D8D1799A1227E2FDE4" }, \{ "b" : "7F3DD6550000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "39E6427542C05B1019EC312345239164CCFE84DF" }, \{ "b" : "7F3DD6268000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "36971F0AF2B69B25C7F9589CDE3A82D34A60C591" }, \{ "b" : "7F3DD6064000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5025A7A2E8AECA9ACB996121370821B9CAA9E397" }, \{ "b" : "7F3DD5E31000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "9891D2B285C6D5E761DFE7AEE8BAD629EA73D061" }, \{ "b" : "7F3DD5C1B000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, \{ "b" : "7F3DD5A0D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AA19B311D788A700456C8A7E1C727F8609CFB302" }, \{ "b" : "7F3DD5809000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, \{ "b" : "7F3DD55F0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4C488F6E7044BB966162C1F7081ABBA6EBB2B485" }, \{ "b" : "7F3DD53C9000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "1B0C0DC4F9054C5922082921C1447739F27AC8F4" }, \{ "b" : "7F3DD5167000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x560afaa90cb1]
 mongod(+0x158DEC9) [0x560afaa8fec9]
 mongod(+0x158E3AD) [0x560afaa903ad]
 libpthread.so.0(+0xF5D0) [0x7f3dd6b795d0]
 libc.so.6(gsignal+0x37) [0x7f3dd67d3207]
 libc.so.6(abort+0x148) [0x7f3dd67d48f8]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x560af9d212d5]
 mongod(+0x1292456) [0x560afa794456]
 mongod(__wt_eventv+0x3D7) [0x560af9d2ba2a]
 mongod(__wt_err+0x9D) [0x560af9d2bc46]
 mongod(__wt_panic+0x2E) [0x560af9d2bea8]
 mongod(__wt_turtle_read+0x26C) [0x560afb44053c]
 mongod(__wt_metadata_search+0xE8) [0x560afb43eb38]
 mongod(__wt_conn_btree_open+0x73) [0x560afb3f3503]
 mongod(__wt_session_get_btree+0xFB) [0x560afb47e01b]
 mongod(__wt_session_get_btree+0x63D) [0x560afb47e55d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x560afb47e81c]
 mongod(__wt_curfile_open+0x161) [0x560afb401d81]
 mongod(+0x1F72308) [0x560afb474308]
 mongod(__wt_metadata_cursor_open+0x6E) [0x560afb43e1ee]
 mongod(__wt_metadata_cursor+0x4B) [0x560afb43e2cb]
 mongod(wiredtiger_open+0x196B) [0x560afb3efabb]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xAFA) [0x560afa778d8a]
 mongod(+0x126F075) [0x560afa771075]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x697) [0x560afa662757]
 mongod(+0x80B66E) [0x560af9d0d66e]
 mongod(main+0x96B) [0x560af9d2d25b]
 libc.so.6(__libc_start_main+0xF5) [0x7f3dd67bf3d5]
 mongod(+0x88B1DF) [0x560af9d8d1df]
----- END BACKTRACE -----

Generated at Thu Feb 08 04:49:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.