[SERVER-48009] WiredTiger Error Created: 07/May/20  Updated: 06/Dec/22  Resolved: 11/May/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Sts BMU Assignee: Backlog - Triage Team
Resolution: Done Votes: 0
Labels: repair, repairDatabase
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu, Docker


Attachments: Text File DockerMongoDB.txt     HTML File ErrorLog    
Assigned Teams:
Server Triage
Participants:

 Description   

Hello,

I have some error with MongoDB starting on Docker following server restart. Could you suggest me about fix?

Here is the logs from system:

 

// code placeholder 
LOGS:
 
2020-05-07T04:41:56.388+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=35d6588e67c8
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] db version v4.2.0
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] modules: none
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] build environment:
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2020-05-07T04:41:56.394+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: "*" }, security: { authorization: "enabled" } }
2020-05-07T04:41:56.395+0000 W  STORAGE  [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2020-05-07T04:41:56.396+0000 I  STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-05-07T04:41:56.396+0000 W  STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2020-05-07T04:41:56.396+0000 I  STORAGE  [initandlisten]
2020-05-07T04:41:56.396+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-05-07T04:41:56.396+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-05-07T04:41:56.396+0000 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15560M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (0) [1588826517:268321][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_block_read_off, 299: WiredTiger.wt: read checksum error for 4096B block at offset 24576: block header checksum of 0x6f665f79 doesn't match expected checksum of 0xca2ef9d1 Raw: [1588826517:268321][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_block_read_off, 299: WiredTiger.wt: read checksum error for 4096B block at offset 24576: block header checksum of 0x6f665f79 doesn't match expected checksum of 0xca2ef9d12020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (0) [1588826517:268489][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 1 of 4): 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 69 67 6e 6f 72 65  Raw: [1588826517:268489][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 1 of 4): 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 
2020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (0) [1588826517:268626][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 2 of 4): 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61   Raw: [1588826517:268626][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 2 of 4): 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 
2020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (0) [1588826517:268748][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 3 of 4): 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 33 34 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 2c 65 6e 63 72 79 70  Raw: [1588826517:268748][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 3 of 4): 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 33 34 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 
2020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (0) [1588826517:268881][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 4 of 4): 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 2d 31 31 2c 77 72 69  Raw: [1588826517:268881][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_bm_corrupt_dump, 145: {24576, 4096, 0xca2ef9d1}: (chunk 4 of 4): 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 
2020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (-31802) [1588826517:268922][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_block_read_off, 309: WiredTiger.wt: fatal read error: WT_ERROR: non-specific WiredTiger error Raw: [1588826517:268922][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_block_read_off, 309: WiredTiger.wt: fatal read error: WT_ERROR: non-specific WiredTiger error
2020-05-07T04:41:57.268+0000 E  STORAGE  [initandlisten] WiredTiger error (-31804) [1588826517:268932][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_panic, 520: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1588826517:268932][1:0x7f9cf606cb00], file:WiredTiger.wt, connection: __wt_panic, 520: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-05-07T04:41:57.268+0000 F  -        [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403
2020-05-07T04:41:57.268+0000 F  -        [initandlisten]***aborting after fassert() failure
2020-05-07T04:41:57.296+0000 F  -        [initandlisten] Got signal: 6 (Aborted).
 0x55b121385b31 0x55b12138532e 0x55b1213853c6 0x7f9cf469e890 0x7f9cf42d9e97 0x7f9cf42db801 0x55b11f8be95f 0x55b11f60ffd7 0x55b11fa1e6ca 0x55b11f61bbaf 0x55b11f61c013 0x55b11f629660 0x55b11fb2745d 0x55b11fb2796b 0x55b11fb228fc 0x55b11fb0aec7 0x55b11fa4fca1 0x55b11f9ca3b5 0x55b11fa1cff6 0x55b11fa1d56b 0x55b11fa1d744 0x55b11fab0f2a 0x55b11fa18c1f 0x55b11f9e95ae 0x55b11f9e969b 0x55b11f9c90d8 0x55b11f98f02e 0x55b11f990b26 0x55b11f965243 0x55b12014558f 0x55b11f93fbfa 0x55b11f94325d 0x55b11f8c98f9 0x7f9cf42bcb97 0x55b11f93e81a
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55B11EC05000","o":"2780B31","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B11EC05000","o":"278032E"},{"b":"55B11EC05000","o":"27803C6"},{"b":"7F9CF468C000","o":"12890"},{"b":"7F9CF429B000","o":"3EE97","s":"gsignal"},{"b":"7F9CF429B000","o":"40801","s":"abort"},{"b":"55B11EC05000","o":"CB995F","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B11EC05000","o":"A0AFD7"},{"b":"55B11EC05000","o":"E196CA"},{"b":"55B11EC05000","o":"A16BAF","s":"__wt_err_func"},{"b":"55B11EC05000","o":"A17013","s":"__wt_panic"},{"b":"55B11EC05000","o":"A24660"},{"b":"55B11EC05000","o":"F2245D","s":"__wt_block_extlist_read"},{"b":"55B11EC05000","o":"F2296B","s":"__wt_block_extlist_read_avail"},{"b":"55B11EC05000","o":"F1D8FC","s":"__wt_block_checkpoint_load"},{"b":"55B11EC05000","o":"F05EC7"},{"b":"55B11EC05000","o":"E4ACA1","s":"__wt_btree_open"},{"b":"55B11EC05000","o":"DC53B5","s":"__wt_conn_dhandle_open"},{"b":"55B11EC05000","o":"E17FF6","s":"__wt_session_get_dhandle"},{"b":"55B11EC05000","o":"E1856B","s":"__wt_session_get_dhandle"},{"b":"55B11EC05000","o":"E18744","s":"__wt_session_get_btree_ckpt"},{"b":"55B11EC05000","o":"EABF2A","s":"__wt_curfile_open"},{"b":"55B11EC05000","o":"E13C1F"},{"b":"55B11EC05000","o":"DE45AE","s":"__wt_metadata_cursor_open"},{"b":"55B11EC05000","o":"DE469B","s":"__wt_metadata_cursor"},{"b":"55B11EC05000","o":"DC40D8","s":"wiredtiger_open"},{"b":"55B11EC05000","o":"D8A02E","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_"},{"b":"55B11EC05000","o":"D8BB26","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb"},{"b":"55B11EC05000","o":"D60243"},{"b":"55B11EC05000","o":"154058F","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},{"b":"55B11EC05000","o":"D3ABFA"},{"b":"55B11EC05000","o":"D3E25D"},{"b":"55B11EC05000","o":"CC48F9"},{"b":"7F9CF429B000","o":"21B97","s":"__libc_start_main"},{"b":"55B11EC05000","o":"D3981A","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.2.0", "gitVersion" : "a4b751dcf51dd249c5865812b390cfd1c0129c30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-99-generic", "version" : "#100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "55B11EC05000", "elfType" : 3, "buildId" : "D849CC14858F969B5888F892C9E16777E4C6FF32" }, { "b" : "7FFD45BF4000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "03C9E629B1AFDF4613CE0B5DCFF2BE61A21CC53A" }, { "b" : "7F9CF5BE0000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "1C6BC2C0699CE0F7E848CA0B267E0CF07553F6AB" }, { "b" : "7F9CF59C5000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F9CF54FA000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "CB6876717C83B0CC01C3C919B9B6E86D8554F546" }, { "b" : "7F9CF526D000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "439A262CC0127BA401707DEC7A28884D617550E0" }, { "b" : "7F9CF5069000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F9CF4E61000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F9CF4AC3000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F9CF48AB000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F9CF468C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F9CF429B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F9CF5E5F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F9CF4076000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F9CF3E59000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F9CF3C3D000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F9CF3A2F000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F9CF37E4000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F9CF3592000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "70EEF126558D1559A0A4E334FB68E4E9AABE90CB" }, { "b" : "7F9CF3384000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "C14042EC7BD22B9A07D2C16563FE3C2606F52AB7" }, { "b" : "7F9CF3167000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F9CF2DE9000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F9CF2A84000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "E5AE5C31F804BE96532D0DB2091F19E472F2D4A0" }, { "b" : "7F9CF2850000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F9CF261A000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F9CF2399000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F9CF20C3000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F9CF1E91000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F9CF1C8D000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "C0CB7E35A4566A443F99DFBC1A54D3A0677C8A10" }, { "b" : "7F9CF1A82000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F9CF1867000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F9CF1626000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F9CF12F7000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F9CF10E4000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F9CF0EE0000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F9CF0CD7000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F9CF0A4A000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F9CF07A8000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F9CF0572000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F9CF035C000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F9CF0154000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F9CEFF2B000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F9CEFD1C000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F9CEFAD2000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F9CEF7C9000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "BCE351987CF42B3D258B09F0CAC867758D935086" }, { "b" : "7F9CEF591000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b121385b31]
 mongod(+0x278032E) [0x55b12138532e]
 mongod(+0x27803C6) [0x55b1213853c6]
 libpthread.so.0(+0x12890) [0x7f9cf469e890]
 libc.so.6(gsignal+0xC7) [0x7f9cf42d9e97]
 libc.so.6(abort+0x141) [0x7f9cf42db801]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b11f8be95f]
 mongod(+0xA0AFD7) [0x55b11f60ffd7]
 mongod(+0xE196CA) [0x55b11fa1e6ca]
 mongod(__wt_err_func+0x90) [0x55b11f61bbaf]
 mongod(__wt_panic+0x39) [0x55b11f61c013]
 mongod(+0xA24660) [0x55b11f629660]
 mongod(__wt_block_extlist_read+0x8D) [0x55b11fb2745d]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x55b11fb2796b]
 mongod(__wt_block_checkpoint_load+0x17C) [0x55b11fb228fc]
 mongod(+0xF05EC7) [0x55b11fb0aec7]
 mongod(__wt_btree_open+0xD61) [0x55b11fa4fca1]
 mongod(__wt_conn_dhandle_open+0x285) [0x55b11f9ca3b5]
 mongod(__wt_session_get_dhandle+0xE6) [0x55b11fa1cff6]
 mongod(__wt_session_get_dhandle+0x65B) [0x55b11fa1d56b]
 mongod(__wt_session_get_btree_ckpt+0x154) [0x55b11fa1d744]
 mongod(__wt_curfile_open+0x5A) [0x55b11fab0f2a]
 mongod(+0xE13C1F) [0x55b11fa18c1f]
 mongod(__wt_metadata_cursor_open+0x6E) [0x55b11f9e95ae]
 mongod(__wt_metadata_cursor+0x4B) [0x55b11f9e969b]
 mongod(wiredtiger_open+0x22B8) [0x55b11f9c90d8]
 mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0xAE) [0x55b11f98f02e]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb+0x766) [0x55b11f990b26]
 mongod(+0xD60243) [0x55b11f965243]
 mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x52F) [0x55b12014558f]
 mongod(+0xD3ABFA) [0x55b11f93fbfa]
 mongod(+0xD3E25D) [0x55b11f94325d]
 mongod(+0xCC48F9) [0x55b11f8c98f9]
 libc.so.6(__libc_start_main+0xE7) [0x7f9cf42bcb97]
 mongod(_start+0x2A) [0x55b11f93e81a]
-----  END BACKTRACE  -----

 



 Comments   
Comment by Carl Champain (Inactive) [ 11/May/20 ]

Glad it worked! I will close this ticket.

Comment by Sts BMU [ 09/May/20 ]

Hi Carl,

I have managed to run repair from correctly by moving data folder to host. So basic solution for this issue was mongod --repair. 

Overall, I still need to check for completeness of data but repair did its job.

Thank you for your time!

Regards,

Comment by Sts BMU [ 09/May/20 ]

I have managed to run repair and start the MongoDB container but all my collections are gone. Could you suggest me what to do?

 

// code placeholder
root@qm:/home/qm# docker run it -v data_volume:/data/db mongo mongod --repair
Unable to find image 'it:latest' locally
docker: Error response from daemon: pull access denied for it, repository does n                                                                                                                                                             ot exist or may require 'docker login': denied: requested access to the resource                                                                                                                                                              is denied.
See 'docker run --help'.
root@qm:/home/qm# docker run -it -v data_volume:/data/db mongo mongod --repair
2020-05-09T04:27:18.260+0000 I  CONTROL  [main] Automatically disabling TLS 1.0,                                                                                                                                                              to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=                                                                                                                                                             1 port=27017 dbpath=/data/db 64-bit host=169f1521d4a0
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] db version v4.2.0
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] git version: a4b751dcf5                                                                                                                                                             1dd249c5865812b390cfd1c0129c30
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSS                                                                                                                                                             L 1.1.1  11 Sep 2018
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] modules: none
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] build environment:
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2020-05-09T04:27:18.265+0000 I  CONTROL  [initandlisten] options: { net: { bindI                                                                                                                                                             p: "*" }, repair: true }
2020-05-09T04:27:18.266+0000 I  STORAGE  [initandlisten]
2020-05-09T04:27:18.266+0000 I  STORAGE  [initandlisten] ** WARNING: Using the X                                                                                                                                                             FS filesystem is strongly recommended with the WiredTiger storage engine
2020-05-09T04:27:18.266+0000 I  STORAGE  [initandlisten] **          See http://                                                                                                                                                             dochub.mongodb.org/core/prodnotes-filesystem
2020-05-09T04:27:18.266+0000 I  STORAGE  [initandlisten] wiredtiger_open config:                                                                                                                                                              create,cache_size=15560M,cache_overflow=(file_max=0M),session_max=33000,evictio                                                                                                                                                             n=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled                                                                                                                                                             =true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time                                                                                                                                                             =100000),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress]                                                                                                                                                             ,
2020-05-09T04:27:21.411+0000 I  STORAGE  [initandlisten] WiredTiger message [1588998441:411776][1:0x7fe8ff877b00], txn-recover: Main recovery loop: starting at 1/0 to 2/256
2020-05-09T04:27:21.519+0000 I  STORAGE  [initandlisten] WiredTiger message [1588998441:519461][1:0x7fe8ff877b00], txn-recover: Recovering log 1 through 2
2020-05-09T04:27:21.580+0000 I  STORAGE  [initandlisten] WiredTiger message [1588998441:580767][1:0x7fe8ff877b00], txn-recover: Recovering log 2 through 2
2020-05-09T04:27:21.638+0000 I  STORAGE  [initandlisten] WiredTiger message [1588998441:638435][1:0x7fe8ff877b00], txn-recover: Set global recovery timestamp: (0,0)
2020-05-09T04:27:21.683+0000 I  RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2020-05-09T04:27:21.734+0000 I  STORAGE  [initandlisten] Timestamp monitor starting
2020-05-09T04:27:21.748+0000 I  CONTROL  [initandlisten]
2020-05-09T04:27:21.748+0000 I  CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-05-09T04:27:21.748+0000 I  CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2020-05-09T04:27:21.748+0000 I  CONTROL  [initandlisten]
2020-05-09T04:27:21.749+0000 I  STORAGE  [initandlisten] createCollection: admin.system.version with provided UUID: 0f592add-faa7-461a-ba02-1de65b764b31 and options: { uuid: UUID("0f592add-faa7-461a-ba02-1de65b764b31") }
2020-05-09T04:27:21.793+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns admin.system.version
2020-05-09T04:27:21.794+0000 W  ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2020-05-09T04:27:21.794+0000 I  SHARDING [initandlisten] Marking collection admin.system.version as collection version: <unsharded>
2020-05-09T04:27:21.794+0000 I  COMMAND  [initandlisten] setting featureCompatibilityVersion to 4.2
2020-05-09T04:27:21.794+0000 I  STORAGE  [initandlisten] repairDatabase admin
2020-05-09T04:27:21.795+0000 I  STORAGE  [initandlisten] Repairing collection admin.system.version
2020-05-09T04:27:21.807+0000 I  STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-3922263813400813416. Not salvaging.
2020-05-09T04:27:21.877+0000 I  INDEX    [initandlisten] index build: starting on admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" } using method: Hybrid
2020-05-09T04:27:21.878+0000 I  INDEX    [initandlisten] build may temporarily use up to 500 megabytes of RAM
2020-05-09T04:27:21.878+0000 I  STORAGE  [initandlisten] Index build initialized: b042b437-0cf0-4061-9f10-82bc1357ae4b: admin.system.version: indexes: 1
2020-05-09T04:27:21.885+0000 I  STORAGE  [initandlisten] Index builds manager starting: b042b437-0cf0-4061-9f10-82bc1357ae4b: admin.system.version
2020-05-09T04:27:21.886+0000 I  INDEX    [initandlisten] index build: inserted 1 keys from external sorter into index in 0 seconds
2020-05-09T04:27:21.890+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns admin.system.version
2020-05-09T04:27:21.891+0000 I  STORAGE  [initandlisten] Index builds manager completed successfully: b042b437-0cf0-4061-9f10-82bc1357ae4b: admin.system.version. Index specs requested: 1. Indexes in catalog before build: 1. Indexes in catalog after build: 1
2020-05-09T04:27:21.950+0000 I  SHARDING [initandlisten] Marking collection local.system.replset as collection version: <unsharded>
2020-05-09T04:27:21.950+0000 I  STORAGE  [initandlisten] Flow Control is enabled on this deployment.
2020-05-09T04:27:21.950+0000 I  STORAGE  [initandlisten] finished checking dbs
2020-05-09T04:27:21.951+0000 I  -        [initandlisten] Stopping further Flow Control ticket acquisitions.
2020-05-09T04:27:21.951+0000 I  STORAGE  [initandlisten] Deregistering all the collections
2020-05-09T04:27:21.951+0000 I  STORAGE  [initandlisten] Timestamp monitor shutting down
2020-05-09T04:27:21.951+0000 I  STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2020-05-09T04:27:21.951+0000 I  STORAGE  [initandlisten] Shutting down session sweeper thread
2020-05-09T04:27:21.951+0000 I  STORAGE  [initandlisten] Finished shutting down session sweeper thread
2020-05-09T04:27:21.952+0000 I  STORAGE  [initandlisten] Shutting down journal flusher thread
2020-05-09T04:27:22.048+0000 I  STORAGE  [initandlisten] Finished shutting down journal flusher thread
2020-05-09T04:27:22.048+0000 I  STORAGE  [initandlisten] Shutting down checkpoint thread
2020-05-09T04:27:22.048+0000 I  STORAGE  [initandlisten] Finished shutting down checkpoint thread
2020-05-09T04:27:22.078+0000 I  STORAGE  [initandlisten] shutdown: removing fs lock...
2020-05-09T04:27:22.079+0000 I  CONTROL  [initandlisten] now exiting
2020-05-09T04:27:22.079+0000 I  CONTROL  [initandlisten] shutting down with code:0

Comment by Carl Champain (Inactive) [ 08/May/20 ]

Did you do mongo --repair or mongod --repair?
Can you please provide:

  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

Thank you,
Carl

Comment by Sts BMU [ 08/May/20 ]

Hi Carl,

 

Thanks for response!

I have tried to run repair with following command:

docker run --rm --volumes-from mydatabase mongo --repair

But system returns exactly same logs as for startup. What do you think?

Comment by Carl Champain (Inactive) [ 07/May/20 ]

Hi setsenwn@gmail.com,

This error message leads us to suspect some form of physical corruption. Please make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath.

Can you please try mongod --repair using the latest version of MongoDB?

Thank you,
Carl
 

Generated at Thu Feb 08 05:15:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.