[SERVER-36365] MongoDB 4.0 Crashed Multiple Times During Insert Operations Created: 30/Jul/18  Updated: 04/Nov/18  Resolved: 01/Oct/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 4.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kelvin Lu Assignee: Nick Brewer
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2018-07-30-16-14-46-412.png     PNG File image-2018-07-31-16-21-39-186.png     PNG File image-2018-07-31-16-22-10-296.png     Text File log.txt     Zip Archive src.zip    
Operating System: ALL
Steps To Reproduce:
  1. Go to http://openaddresses.io download address data and extract to folder as testing data.
  2. Use attached source code in src.zip to run in debug model for 4-5 hours to reproduce this issue. (Use MongoDB 2.7 C# Driver)
  3. Please change following in program.cs file to your local location

        const string ROOT_PATH  = @"E:\temp\Address\us\tx";
        const string ARCHIVE    = @"E:\temp\Address\archive";
        const string INDEX_PATH = @"E:\temp\Address\index";

Participants:

 Description   

Hello Team,

I am testing MongoDB 4.0 on my local Ubuntu 18.04.1 and mongoDB crashed when I insert document with following error. Log file attached

 

2018-07-30T20:15:48.153+0000 E STORAGE [thread3] WiredTiger error (0) [1532981748:153737][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: encountered an illegal file format or internal value: (__inmem_row_leaf, 617) Raw: [1532981748:153737][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: encountered an illegal file format or internal value: (__inmem_row_leaf, 617)
2018-07-30T20:15:48.153+0000 E STORAGE [thread3] WiredTiger error (-31804) [1532981748:153796][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1532981748:153796][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-07-30T20:15:48.153+0000 F - [thread3] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403
2018-07-30T20:15:48.153+0000 F - [thread3]
 
***aborting after fassert() failure
 
 
2018-07-30T20:15:48.153+0000 F - [conn2] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 88
2018-07-30T20:15:48.153+0000 F - [conn2]
 
***aborting after fassert() failure
 
 
2018-07-30T20:15:48.154+0000 E STORAGE [thread4] WiredTiger error (-31804) [1532981748:154320][2126:0x7f81809ed700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic Raw: [1532981748:154320][2126:0x7f81809ed700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic
2018-07-30T20:15:48.154+0000 F - [thread4] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403
2018-07-30T20:15:48.154+0000 F - [thread4]
 
***aborting after fassert() failure
 
 
2018-07-30T20:15:48.172+0000 E STORAGE [thread5] WiredTiger error (-31804) [1532981748:172184][2126:0x7f81819ef700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic Raw: [1532981748:172184][2126:0x7f81819ef700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic
2018-07-30T20:15:48.172+0000 F - [thread5] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403
2018-07-30T20:15:48.172+0000 F - [thread5]
 
***aborting after fassert() failure
 
 
2018-07-30T20:15:48.203+0000 E STORAGE [thread6] WiredTiger error (-31804) [1532981748:203320][2126:0x7f81829f1700], log-server: log server error: WT_PANIC: WiredTiger library panic Raw: [1532981748:203320][2126:0x7f81829f1700], log-server: log server error: WT_PANIC: WiredTiger library panic
2018-07-30T20:15:48.203+0000 F - [thread6] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403
2018-07-30T20:15:48.203+0000 F - [thread6]
 
***aborting after fassert() failure
 
 
2018-07-30T20:15:48.485+0000 F - [thread3] Got signal: 6 (Aborted).
 0x560f086dd141 0x560f086dc359 0x560f086dc83d 0x7f8189b03890 0x7f818973ee97 0x7f8189740801 0x560f06d5d22f 0x560f06e4e726 0x560f06ebc081 0x560f06ce8119 0x560f06ce8442 0x560f06ee710d 0x560f06ef3af2 0x560f06ef8cf0 0x560f06e7902a 0x560f06e716a3 0x560f06e71bd3 0x560f06e74731 0x560f06ec85e9 0x7f8189af86db 0x7f818982188f
----- BEGIN BACKTRACE -----
{"backtrace":[\{"b":"560F06365000","o":"2378141","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"560F06365000","o":"2377359"},\{"b":"560F06365000","o":"237783D"},\{"b":"7F8189AF1000","o":"12890"},\{"b":"7F8189700000","o":"3EE97","s":"gsignal"},\{"b":"7F8189700000","o":"40801","s":"abort"},\{"b":"560F06365000","o":"9F822F","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"560F06365000","o":"AE9726"},\{"b":"560F06365000","o":"B57081"},\{"b":"560F06365000","o":"983119","s":"__wt_err"},\{"b":"560F06365000","o":"983442","s":"__wt_panic"},\{"b":"560F06365000","o":"B8210D","s":"__wt_page_inmem"},\{"b":"560F06365000","o":"B8EAF2"},\{"b":"560F06365000","o":"B93CF0","s":"__wt_split_rewrite"},\{"b":"560F06365000","o":"B1402A","s":"__wt_evict"},\{"b":"560F06365000","o":"B0C6A3"},\{"b":"560F06365000","o":"B0CBD3"},\{"b":"560F06365000","o":"B0F731","s":"__wt_evict_thread_run"},\{"b":"560F06365000","o":"B635E9"},\{"b":"7F8189AF1000","o":"76DB"},\{"b":"7F8189700000","o":"12188F","s":"clone"}],"processInfo":\{ "mongodbVersion" : "4.0.0", "gitVersion" : "3b07af3d4f471ae89e8186d33bbb1d5259597d51", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-29-generic", "version" : "#31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018", "machine" : "x86_64" }, "somap" : [ \{ "b" : "560F06365000", "elfType" : 3, "buildId" : "365C0577E1D4FD5CD19A9455CB05F9122B9CF331" }, \{ "b" : "7FFFE4FF4000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "790546AE562C218C1F03B75290F2E817B4222315" }, \{ "b" : "7F818AF98000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "683A1F6A8C6401C61D29715C7F38B4586A175873" }, \{ "b" : "7F818AD7D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, \{ "b" : "7F818A93A000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "5D934D586011BB3BD74719F877408B61FBD0AD3D" }, \{ "b" : "7F818A6D2000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "13EB7A057F4E3863403208D789D74A7CF58D66D5" }, \{ "b" : "7F818A4CE000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, \{ "b" : "7F818A2C6000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, \{ "b" : "7F8189F28000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, \{ "b" : "7F8189D10000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "039AE85FEF075EC14FE3528762A0645C8CF73B29" }, \{ "b" : "7F8189AF1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, \{ "b" : "7F8189700000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, \{ "b" : "7F818B217000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, \{ "b" : "7F81894DB000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, \{ "b" : "7F81892BE000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, \{ "b" : "7F81890A2000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, \{ "b" : "7F8188E94000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, \{ "b" : "7F8188C49000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "451E2EC02C8F974B21D1CE47F5D2C21A21C16A9F" }, \{ "b" : "7F81889F7000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "0CDDF9D1F87FE7FF07D73DBE1D7CB57EE5BA802C" }, \{ "b" : "7F81887E9000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "DD2893FC0AA3C58798F805D4C88EE086564D2EEB" }, \{ "b" : "7F81885CC000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, \{ "b" : "7F818824E000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, \{ "b" : "7F8187EE9000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, \{ "b" : "7F8187CB5000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, \{ "b" : "7F8187A7F000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, \{ "b" : "7F81877FE000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, \{ "b" : "7F8187528000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "AFF9A2A2B6ACB066BF450CC4A0772B9DC04299CB" }, \{ "b" : "7F81872F6000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "04AADCB371B97A570756D69A0D5B3B0E6737E349" }, \{ "b" : "7F81870F2000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "9E885FB7F54DA464A3D2A978C71C1178A6EC7512" }, \{ "b" : "7F8186EE7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "27A5983855C50A37EFA526BB12EA48CDAE3B5FCA" }, \{ "b" : "7F8186CCC000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, \{ "b" : "7F8186A8B000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, \{ "b" : "7F818675C000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, \{ "b" : "7F8186549000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, \{ "b" : "7F8186345000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, \{ "b" : "7F818613C000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, \{ "b" : "7F8185EAF000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, \{ "b" : "7F8185C0D000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, \{ "b" : "7F81859D7000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, \{ "b" : "7F81857C1000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, \{ "b" : "7F81855B9000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, \{ "b" : "7F8185390000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, \{ "b" : "7F8185181000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, \{ "b" : "7F8184F37000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, \{ "b" : "7F8184C2E000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, \{ "b" : "7F81849F6000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x560f086dd141]
 mongod(+0x2377359) [0x560f086dc359]
 mongod(+0x237783D) [0x560f086dc83d]
 libpthread.so.0(+0x12890) [0x7f8189b03890]
 libc.so.6(gsignal+0xC7) [0x7f818973ee97]
 libc.so.6(abort+0x141) [0x7f8189740801]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x560f06d5d22f]
 mongod(+0xAE9726) [0x560f06e4e726]
 mongod(+0xB57081) [0x560f06ebc081]
 mongod(__wt_err+0x9D) [0x560f06ce8119]
 mongod(__wt_panic+0x33) [0x560f06ce8442]
 mongod(__wt_page_inmem+0x93D) [0x560f06ee710d]
 mongod(+0xB8EAF2) [0x560f06ef3af2]
 mongod(__wt_split_rewrite+0xA0) [0x560f06ef8cf0]
 mongod(__wt_evict+0x106A) [0x560f06e7902a]
 mongod(+0xB0C6A3) [0x560f06e716a3]
 mongod(+0xB0CBD3) [0x560f06e71bd3]
 mongod(__wt_evict_thread_run+0x71) [0x560f06e74731]
 mongod(+0xB635E9) [0x560f06ec85e9]
 libpthread.so.0(+0x76DB) [0x7f8189af86db]
 libc.so.6(clone+0x3F) [0x7f818982188f]
----- END BACKTRACE -----



 Comments   
Comment by Nick Brewer [ 01/Oct/18 ]

kelvinl Since there's been no activity on this ticket in some time, I'm going to close it. Feel free to update this ticket with the requested information, and we'll reopen it for you.

Thanks,
-Nick

Comment by Nick Brewer [ 20/Aug/18 ]

kelvinl Could provide the following information, if possible:

  • The full mongod logs leading up to the crash
  • Any details required to recreate this behavior (a specific query, etc)
  • An archive (tar or zip) of the dbpath/diagnostic.data directory

If you'd prefer to keep any of this information private, we can generate a secure upload portal for you.

One other thing I would note: as I mentioned previously, mongod needs to be able to perform fsync on directories - I would be curious to know if you can recreate this behavior when not using a shared drive for your dbpath
Thanks,
Nick

Comment by Kelvin Lu [ 20/Aug/18 ]

Nick,

I was out for vacation and did not reply your comment. 

I re-installed MongoDB with most recent version 4.01 and reload data. I do not see that bug again. However, I experienced following error from time to time with 

{ driver: { name: "mongo-csharp-driver", version: "2.7.0.0" }, os: { type: "Windows", name: "Microsoft Windows 10.0.10240", architecture: "x86_64", version: "10.0.10240" }, platform: ".NET Core 4.6.26515.07" }

2018-08-20T22:11:03.841+0000 I NETWORK  [conn228] received client metadata from 10.10.25.26:60642 conn228: { driver: { name: "mongo-csharp-driver", version: "2.7.0.0" }, os: { type: "Windows", name: "Microsoft Windows 10.0.10240", architecture: "x86_64", version: "10.0.10240" }, platform: ".NET Core 4.6.26515.07" }
2018-08-20T22:32:40.614+0000 F -        [WTCheckpointThread] Invalid access at address: 0x4
2018-08-20T22:32:40.745+0000 F -        [WTCheckpointThread] Got signal: 11 (Segmentation fault).
 0x55fcfa781bf1 0x55fcfa780e09 0x55fcfa781476 0x7f342dec2890 0x55fcf8f3caa9 0x55fcf8f3dad6 0x55fcf8f42feb 0x55fcf8f9ef04 0x55fcf8f6f06a 0x55fcf8f714e6 0x55fcf8f72503 0x55fcf8f580ba 0x55fcf8ed0aef 0x55fcfa2c2771 0x55fcfa8910b0 0x7f342deb76db 0x7f342dbe088f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55FCF8404000","o":"237DBF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55FCF8404000","o":"237CE09"},{"b":"55FCF8404000","o":"237D476"},{"b":"7F342DEB0000","o":"12890"},{"b":"55FCF8404000","o":"B38AA9"},{"b":"55FCF8404000","o":"B39AD6"},{"b":"55FCF8404000","o":"B3EFEB","s":"__wt_reconcile"},{"b":"55FCF8404000","o":"B9AF04","s":"__wt_cache_op"},{"b":"55FCF8404000","o":"B6B06A"},{"b":"55FCF8404000","o":"B6D4E6"},{"b":"55FCF8404000","o":"B6E503","s":"__wt_txn_checkpoint"},{"b":"55FCF8404000","o":"B540BA"},{"b":"55FCF8404000","o":"ACCAEF","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"55FCF8404000","o":"1EBE771","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"55FCF8404000","o":"248D0B0"},{"b":"7F342DEB0000","o":"76DB"},{"b":"7F342DABF000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.1", "gitVersion" : "54f1582fc6eb01de4d4c42f26fc133e623f065fb", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-30-generic", "version" : "#32-Ubuntu SMP Thu Jul 26 17:42:43 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55FCF8404000", "elfType" : 3, "buildId" : "277667F916B0411F06935B33EDB63802E236E5DA" }, { "b" : "7FFD1A4F7000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "C65E0B5983446DCDF584BC5707F2CCBF32AA1884" }, { "b" : "7F342F38E000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "6AF728F07FFCDB4DAABE4F4CFC6EDB1401755DC7" }, { "b" : "7F342F173000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F342ECFB000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "DA6345B41E06A9001B5C51DEAF0784BE9D49810D" }, { "b" : "7F342EA91000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "9787FF1FFFE8D82268D89F6BFA3AF07B3E9D140F" }, { "b" : "7F342E88D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F342E685000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F342E2E7000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F342E0CF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "039AE85FEF075EC14FE3528762A0645C8CF73B29" }, { "b" : "7F342DEB0000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F342DABF000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F342F60D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F342D89A000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F342D67D000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F342D461000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F342D253000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F342D008000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "451E2EC02C8F974B21D1CE47F5D2C21A21C16A9F" }, { "b" : "7F342CDB6000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "0CDDF9D1F87FE7FF07D73DBE1D7CB57EE5BA802C" }, { "b" : "7F342CBA8000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "DD2893FC0AA3C58798F805D4C88EE086564D2EEB" }, { "b" : "7F342C98B000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F342C60D000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, { "b" : "7F342C2A8000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, { "b" : "7F342C074000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F342BE3E000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F342BBBD000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F342B8E7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "AFF9A2A2B6ACB066BF450CC4A0772B9DC04299CB" }, { "b" : "7F342B6B5000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "04AADCB371B97A570756D69A0D5B3B0E6737E349" }, { "b" : "7F342B4B1000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "9E885FB7F54DA464A3D2A978C71C1178A6EC7512" }, { "b" : "7F342B2A6000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "27A5983855C50A37EFA526BB12EA48CDAE3B5FCA" }, { "b" : "7F342B08B000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F342AE4A000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F342AB1B000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F342A908000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F342A704000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F342A4FB000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F342A26E000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F3429FCC000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F3429D96000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F3429B80000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F3429978000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F342974F000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F3429540000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F34292F6000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F3428FED000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, { "b" : "7F3428DB5000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55fcfa781bf1]
 mongod(+0x237CE09) [0x55fcfa780e09]
 mongod(+0x237D476) [0x55fcfa781476]
 libpthread.so.0(+0x12890) [0x7f342dec2890]
 mongod(+0xB38AA9) [0x55fcf8f3caa9]
 mongod(+0xB39AD6) [0x55fcf8f3dad6]
 mongod(__wt_reconcile+0x297B) [0x55fcf8f42feb]
 mongod(__wt_cache_op+0x594) [0x55fcf8f9ef04]
 mongod(+0xB6B06A) [0x55fcf8f6f06a]
 mongod(+0xB6D4E6) [0x55fcf8f714e6]
 mongod(__wt_txn_checkpoint+0x1C3) [0x55fcf8f72503]
 mongod(+0xB540BA) [0x55fcf8f580ba]
 mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x59F) [0x55fcf8ed0aef]
 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x55fcfa2c2771]
 mongod(+0x248D0B0) [0x55fcfa8910b0]
 libpthread.so.0(+0x76DB) [0x7f342deb76db]
 libc.so.6(clone+0x3F) [0x7f342dbe088f]
-----  END BACKTRACE  -----

 

 

Comment by Nick Brewer [ 20/Aug/18 ]

kelvinl Since we haven't heard back from you, I'm going to close this issue for now. One thing I'll note is that it's possible you were running into an fsync issue due to the use of the shared Virtualbox drive.

-Nick

Comment by Nick Brewer [ 01/Aug/18 ]

kelvinl Thanks for the info. We'll await your results.

-Nick

Comment by Kelvin Lu [ 31/Jul/18 ]

I use Ubuntu 18.04 on Oracle Virtual Box and dbpath point to /mnt is just another mount point from different logic drive file.

 

I will use your latest binary to build a new mongoDB 4.0 Server with Ubuntu 18.04 and start over again and will let you know if I run to any issues later

 

 

Comment by Nick Brewer [ 31/Jul/18 ]

kelvinl Sorry, this is the correct set of binaries to use for your testing.

From your log, I noticed that the dbpath is pointing to a directory with /mnt - is this an external device? And if so, what kind? It would be helpful to get the device's corresponding entry in /etc/fstab, if it has one.

Thanks,
Nick

Comment by Nick Brewer [ 31/Jul/18 ]

kelvinl MongoDB 4.0.0 is not officially supported on Ubuntu 18.04 - were you using a binary for an older release?

An 18.04 binary is available for our current development release, which you can find here. Could you try recreating this on the latest version? Note that you'll need to test on a new database (starting over from a clean dbpath).

Thanks,
Nick

Generated at Thu Feb 08 04:42:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.