Details
-
Bug
-
Status: Closed
-
Major - P3
-
Resolution: Done
-
4.0.17
-
None
-
ALL
Description
I have installed Mongo container for my product to log data.
Here is the version of Mongo for your reference.
MongoDB shell version v4.0.17
git version: 0bc918c73390f6e4d6349660e4cd233f5900b69a
OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1604
distarch: x86_64
target_arch: x86_64
root@9e31c7f2d640:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 38G 3.1G 33G 9% /
tmpfs 64M 0 64M 0% /dev
tmpfs 2.6G 0 2.6G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/sda1 38G 3.1G 33G 9% /data/db
tmpfs 2.6G 0 2.6G 0% /proc/asound
tmpfs 2.6G 0 2.6G 0% /proc/acpi
tmpfs 2.6G 0 2.6G 0% /proc/scsi
tmpfs 2.6G 0 2.6G 0% /sys/firmware
After logging data for a couple of days I am getting data corruption issues. This is resulting in data loss for me.
|
2020-05-17T06:44:22.673+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
|
2020-05-17T06:44:22.681+0000 I CONTROL [initandlisten] MongoDB starting : pid=6 port=27017 dbpath=/data/db 64-bit host=9e31c7f2d640
|
2020-05-17T06:44:22.681+0000 I CONTROL [initandlisten] db version v4.0.18
|
2020-05-17T06:44:22.681+0000 I CONTROL [initandlisten] git version: 6883bdfb8b8cff32176b1fd176df04da9165fd67
|
2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
|
2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] allocator: tcmalloc
|
2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] modules: none
|
2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] build environment:
|
2020-05-17T06:44:22.682+0000 I CONTROL [initandlisten] distmod: ubuntu1604
|
2020-05-17T06:44:22.683+0000 I CONTROL [initandlisten] distarch: x86_64
|
2020-05-17T06:44:22.683+0000 I CONTROL [initandlisten] target_arch: x86_64
|
2020-05-17T06:44:22.683+0000 I CONTROL [initandlisten] options: {}
|
2020-05-17T06:44:22.684+0000 W STORAGE [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
|
2020-05-17T06:44:22.684+0000 I STORAGE [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
|
2020-05-17T06:44:22.684+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
|
2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten]
|
2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
|
2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] ** See [http://dochub.mongodb.org/core/prodnotes-filesystem]
|
2020-05-17T06:44:22.685+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=2148M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
|
2020-05-17T06:44:24.298+0000 E STORAGE [initandlisten] WiredTiger error (-31803) [1589697864:298835][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_turtle_read, 341: WiredTiger.turtle: fatal turtle file read error: WT_NOTFOUND: item not found Raw: [1589697864:298835][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_turtle_read, 341: WiredTiger.turtle: fatal turtle file read error: WT_NOTFOUND: item not found
|
2020-05-17T06:44:24.299+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1589697864:299514][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1589697864:299514][6:0x7f8a9bbaaa80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2020-05-17T06:44:24.299+0000 F - [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 420
|
2020-05-17T06:44:24.299+0000 F - [initandlisten]
|
|
***aborting after fassert() failure
|
|
2020-05-17T06:44:24.369+0000 F - [initandlisten] Got signal: 6 (Aborted).
|
0x55a2f0532031 0x55a2f0531249 0x55a2f053172d 0x7f8a9a32e390 0x7f8a99f88428 0x7f8a99f8a02a 0x55a2eeae8761 0x55a2eebec4d6 0x55a2eec56a99 0x55a2eea6fac0 0x55a2eea6feda 0x55a2eec22bb0 0x55a2eec20bdc 0x55a2eec00580 0x55a2eec5532d 0x55a2eec5593f 0x55a2eec55bcc 0x55a2eecd4e16 0x55a2eec50ca8 0x55a2eec200ce 0x55a2eec201ab 0x55a2eebfee83 0x55a2eebc03fa 0x55a2eebc6594 0x55a2eeba63d8 0x55a2ef2c7452 0x55a2eeb56986 0x55a2eeb5ab45 0x55a2eeaea2f9 0x7f8a99f73830 0x55a2eeb55299
|
----- BEGIN BACKTRACE -----
|
|
{"backtrace":[\\{"b":"55A2EE0C0000","o":"2472031","s":"_ZN5mongo15printStackTraceERSo"}
|
|
,\{"b":"55A2EE0C0000","o":"2471249"},\{"b":"55A2EE0C0000","o":"247172D"},\{"b":"7F8A9A31D000","o":"11390"},\{"b":"7F8A99F53000","o":"35428","s":"gsignal"},\{"b":"7F8A99F53000","o":"3702A","s":"abort"},{"b":"55A2EE0C0000","o":"A28761","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"55A2EE0C0000","o":"B2C4D6"},\{"b":"55A2EE0C0000","o":"B96A99"},{"b":"55A2EE0C0000","o":"9AFAC0","s":"__wt_err_func"},{"b":"55A2EE0C0000","o":"9AFEDA","s":"__wt_panic"},{"b":"55A2EE0C0000","o":"B62BB0","s":"__wt_turtle_read"},{"b":"55A2EE0C0000","o":"B60BDC","s":"__wt_metadata_search"},{"b":"55A2EE0C0000","o":"B40580","s":"__wt_conn_dhandle_open"},{"b":"55A2EE0C0000","o":"B9532D","s":"__wt_session_get_dhandle"},{"b":"55A2EE0C0000","o":"B9593F","s":"__wt_session_get_dhandle"},{"b":"55A2EE0C0000","o":"B95BCC","s":"__wt_session_get_btree_ckpt"},{"b":"55A2EE0C0000","o":"C14E16","s":"__wt_curfile_open"},\{"b":"55A2EE0C0000","o":"B90CA8"},{"b":"55A2EE0C0000","o":"B600CE","s":"__wt_metadata_cursor_open"},{"b":"55A2EE0C0000","o":"B601AB","s":"__wt_metadata_cursor"},\{"b":"55A2EE0C0000","o":"B3EE83","s":"wiredtiger_open"},{"b":"55A2EE0C0000","o":"B003FA","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_"},{"b":"55A2EE0C0000","o":"B06594","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb"},\{"b":"55A2EE0C0000","o":"AE63D8"},\{"b":"55A2EE0C0000","o":"1207452","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},\{"b":"55A2EE0C0000","o":"A96986"},{"b":"55A2EE0C0000","o":"A9AB45","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},\{"b":"55A2EE0C0000","o":"A2A2F9","s":"main"},\{"b":"7F8A99F53000","o":"20830","s":"__libc_start_main"},\{"b":"55A2EE0C0000","o":"A95299","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.0.18", "gitVersion" : "6883bdfb8b8cff32176b1fd176df04da9165fd67", "compiledModules" : [], "uname" :
|
|
{ "sysname" : "Linux", "release" : "4.14.154-boot2docker", "version" : "#1 SMP Thu Nov 14 19:19:08 UTC 2019", "machine" : "x86_64" }
|
|
, "somap" : [ \{ "b" : "55A2EE0C0000", "elfType" : 3, "buildId" : "9BAA54C4A92F5506E39F152D9D3F90BD0DA0526F" }, \{ "b" : "7FFDF51B0000", "elfType" : 3, "buildId" : "ACA410FCB9C0B0653F00BCE6E8284FD17E9006CF" }, \{ "b" : "7F8A9B72E000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "03F16B86D21AA657A636EB6D7977D285E5B111E4" }, \{ "b" : "7F8A9B513000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, \{ "b" : "7F8A9B0CE000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, \{ "b" : "7F8A9AE65000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, \{ "b" : "7F8A9AC61000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, \{ "b" : "7F8A9AA59000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, \{ "b" : "7F8A9A750000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, \{ "b" : "7F8A9A53A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, \{ "b" : "7F8A9A31D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, \{ "b" : "7F8A99F53000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, \{ "b" : "7F8A9B99D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, \{ "b" : "7F8A99D20000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, \{ "b" : "7F8A99B04000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, \{ "b" : "7F8A998BA000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, \{ "b" : "7F8A996AB000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E150CC35D09A20D93B4D936D72AA7637DABC978D" }, \{ "b" : "7F8A9945A000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3D51E97C8DDD5F9EA1674954F8FA8BB2712E64B5" }, \{ "b" : "7F8A99240000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "E2357F88F4E53F0DE76266532DDDCB7FEB6F87A7" }, \{ "b" : "7F8A98F10000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "85FA1D993E14AC403E56E2D45CAE00ACFBD5766C" }, \{ "b" : "7F8A98CDD000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, \{ "b" : "7F8A98AA7000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, \{ "b" : "7F8A98827000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, \{ "b" : "7F8A98555000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, \{ "b" : "7F8A98326000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, \{ "b" : "7F8A98122000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "E6F8CA010177433A1298DAB8912203E5A5EBB9BA" }, \{ "b" : "7F8A97F17000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, \{ "b" : "7F8A97CFC000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "4FE2C629045AB0AF26EC5D7F8A3D64B9F8CB21B2" }, \{ "b" : "7F8A97ABB000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, \{ "b" : "7F8A97857000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, \{ "b" : "7F8A97644000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, \{ "b" : "7F8A97440000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, \{ "b" : "7F8A97237000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, \{ "b" : "7F8A96FAD000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, \{ "b" : "7F8A96D0B000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, \{ "b" : "7F8A96AD8000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, \{ "b" : "7F8A968C2000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, \{ "b" : "7F8A966BA000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, \{ "b" : "7F8A96491000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, \{ "b" : "7F8A96282000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, \{ "b" : "7F8A96037000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, \{ "b" : "7F8A95D62000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "F0615D615CC5E394B0A73D12AA1382AA872DD1F0" }, \{ "b" : "7F8A95B2A000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FD61CA7A6D603E94E5EFD5C88D8810AE104FCF40" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55a2f0532031]
|
mongod(+0x2471249) [0x55a2f0531249]
|
mongod(+0x247172D) [0x55a2f053172d]
|
libpthread.so.0(+0x11390) [0x7f8a9a32e390]
|
libc.so.6(gsignal+0x38) [0x7f8a99f88428]
|
libc.so.6(abort+0x16A) [0x7f8a99f8a02a]
|
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55a2eeae8761]
|
mongod(+0xB2C4D6) [0x55a2eebec4d6]
|
mongod(+0xB96A99) [0x55a2eec56a99]
|
mongod(__wt_err_func+0x90) [0x55a2eea6fac0]
|
mongod(__wt_panic+0x39) [0x55a2eea6feda]
|
mongod(__wt_turtle_read+0x2B0) [0x55a2eec22bb0]
|
mongod(__wt_metadata_search+0x1CC) [0x55a2eec20bdc]
|
mongod(__wt_conn_dhandle_open+0xA0) [0x55a2eec00580]
|
mongod(__wt_session_get_dhandle+0xED) [0x55a2eec5532d]
|
mongod(__wt_session_get_dhandle+0x6FF) [0x55a2eec5593f]
|
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55a2eec55bcc]
|
mongod(__wt_curfile_open+0x56) [0x55a2eecd4e16]
|
mongod(+0xB90CA8) [0x55a2eec50ca8]
|
mongod(__wt_metadata_cursor_open+0x6E) [0x55a2eec200ce]
|
mongod(__wt_metadata_cursor+0x4B) [0x55a2eec201ab]
|
mongod(wiredtiger_open+0x1A13) [0x55a2eebfee83]
|
mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0xAA) [0x55a2eebc03fa]
|
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb+0x674) [0x55a2eebc6594]
|
mongod(+0xAE63D8) [0x55a2eeba63d8]
|
mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x552) [0x55a2ef2c7452]
|
mongod(+0xA96986) [0x55a2eeb56986]
|
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0xF95) [0x55a2eeb5ab45]
|
mongod(main+0x9) [0x55a2eeaea2f9]
|
libc.so.6(__libc_start_main+0xF0) [0x7f8a99f73830]
|
mongod(_start+0x29) [0x55a2eeb55299]
|
----- END BACKTRACE -----
|
|
I would like to know the reason this might happen. Is there any thing which I am missing in configuration ?
Attached the complete error log & diagnostic data for reference.
Thanks
-Harshad