Suddenly could not start - Failed with result 'core-dump'

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 4.0.2
    • Component/s: None
    • None
    • ALL
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Today when I start my computer, MongoDB does't not start any more. Manually runing its related systemd service (I would describe later), the error logs said:

      sudo systemctl status mongo.service 
● mongo.service - MongoDB
   Loaded: loaded (/etc/systemd/system/mongo.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Tue 2018-10-02 09:49:41 +07; 424ms ago
  Process: 6022 ExecStart=/storage/mongodb/bin/mongod --dbpath /storage/mongodb/data (code=dumped, signal=SEGV)
 Main PID: 6022 (code=dumped, signal=SEGV)Oct 02 09:49:41 u1804d mongod[6022]:  mongod(+0xAB80BD) [0x5568f56e30bd]
Oct 02 09:49:41 u1804d mongod[6022]:  mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0xBF5) [0x5568f5d2d0b5]
Oct 02 09:49:41 u1804d mongod[6022]:  mongod(+0x988F56) [0x5568f55b3f56]
Oct 02 09:49:41 u1804d mongod[6022]:  mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x873) [0x5568f5699f73]
Oct 02 09:49:41 u1804d mongod[6022]:  mongod(main+0x9) [0x5568f5630f99]
Oct 02 09:49:41 u1804d mongod[6022]:  libc.so.6(__libc_start_main+0xE7) [0x7f6ac0835b97]
Oct 02 09:49:41 u1804d mongod[6022]:  mongod(_start+0x2A) [0x5568f5698a2a]
Oct 02 09:49:41 u1804d mongod[6022]: -----  END BACKTRACE  -----
Oct 02 09:49:41 u1804d systemd[1]: mongo.service: Main process exited, code=dumped, status=11/SEGV
Oct 02 09:49:41 u1804d systemd[1]: mongo.service: Failed with result 'core-dump'.

      If I try to run it directy, it says:

      /storage/mongodb/bin/mongod --dbpath /storage/mongodb/data
2018-10-02T09:53:15.992+0700 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] MongoDB starting : pid=6111 port=27017 dbpath=/storage/mongodb/data 64-bit host=u1804d
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] db version v4.0.2
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] git version: fc1573ba18aee42f97a3bb13b67af7d837826b47
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] allocator: tcmalloc
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] modules: none
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] build environment:
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten]     distmod: ubuntu1804
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten]     distarch: x86_64
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten]     target_arch: x86_64
2018-10-02T09:53:15.999+0700 I CONTROL  [initandlisten] options: { storage: { dbPath: "/storage/mongodb/data" } }
2018-10-02T09:53:15.999+0700 I STORAGE  [initandlisten] Detected data files in /storage/mongodb/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-10-02T09:53:15.999+0700 I STORAGE  [initandlisten] 
2018-10-02T09:53:15.999+0700 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-10-02T09:53:15.999+0700 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-10-02T09:53:15.999+0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3436M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-10-02T09:53:16.683+0700 I STORAGE  [initandlisten] WiredTiger message [1538448796:683576][6111:0x7f3b99134a00], txn-recover: Main recovery loop: starting at 487/256
2018-10-02T09:53:16.683+0700 I STORAGE  [initandlisten] WiredTiger message [1538448796:683829][6111:0x7f3b99134a00], txn-recover: Recovering log 487 through 488
2018-10-02T09:53:16.760+0700 I STORAGE  [initandlisten] WiredTiger message [1538448796:760168][6111:0x7f3b99134a00], txn-recover: Recovering log 488 through 488
2018-10-02T09:53:16.807+0700 I STORAGE  [initandlisten] WiredTiger message [1538448796:807688][6111:0x7f3b99134a00], txn-recover: Set global recovery timestamp: 0
2018-10-02T09:53:17.026+0700 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2018-10-02T09:53:17.063+0700 E STORAGE  [initandlisten] WiredTiger error (2) [1538448797:63312][6111:0x7f3b99134a00], file:collection-30-2623056915678715117.wt, WT_SESSION.open_cursor: __posix_open_file, 711: /storage/mongodb/data/collection-30-2623056915678715117.wt: handle-open: open: No such file or directory Raw: [1538448797:63312][6111:0x7f3b99134a00], file:collection-30-2623056915678715117.wt, WT_SESSION.open_cursor: __posix_open_file, 711: /storage/mongodb/data/collection-30-2623056915678715117.wt: handle-open: open: No such file or directory
2018-10-02T09:53:17.063+0700 E STORAGE  [initandlisten] no cursor for uri: table:collection-30-2623056915678715117
2018-10-02T09:53:17.063+0700 F -        [initandlisten] Invalid access at address: 0x58
2018-10-02T09:53:17.072+0700 F -        [initandlisten] Got signal: 11 (Segmentation fault).
 0x55cb0a04fae1 0x55cb0a04ecf9 0x55cb0a04f366 0x7f3b977ef890 0x55cb08772418 0x55cb0877daad 0x55cb0876488c 0x55cb088e8108 0x55cb088e2885 0x55cb088e354d 0x55cb0874f0bd 0x55cb08d990b5 0x55cb0861ff56 0x55cb08705f73 0x55cb0869cf99 0x7f3b9740db97 0x55cb08704a2a
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55CB07C97000","o":"23B8AE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55CB07C97000","o":"23B7CF9"},{"b":"55CB07C97000","o":"23B8366"},{"b":"7F3B977DD000","o":"12890"},{"b":"55CB07C97000","o":"ADB418","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv"},{"b":"55CB07C97000","o":"AE6AAD","s":"_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE"},{"b":"55CB07C97000","o":"ACD88C","s":"_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE"},{"b":"55CB07C97000","o":"C51108","s":"_ZN5mongo26KVDatabaseCatalogEntryBase14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb"},{"b":"55CB07C97000","o":"C4B885","s":"_ZN5mongo15KVStorageEngine11loadCatalogEPNS_16OperationContextE"},{"b":"55CB07C97000","o":"C4C54D","s":"_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE"},{"b":"55CB07C97000","o":"AB80BD"},{"b":"55CB07C97000","o":"11020B5","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},{"b":"55CB07C97000","o":"988F56"},{"b":"55CB07C97000","o":"A6EF73","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55CB07C97000","o":"A05F99","s":"main"},{"b":"7F3B973EC000","o":"21B97","s":"__libc_start_main"},{"b":"55CB07C97000","o":"A6DA2A","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.0.2", "gitVersion" : "fc1573ba18aee42f97a3bb13b67af7d837826b47", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-33-generic", "version" : "#36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55CB07C97000", "elfType" : 3, "buildId" : "8F771DD94C444FE18EFA84CA664CE8A0E6100804" }, { "b" : "7FFEC19E6000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FFC34FD287857F2EB9923E5FF16B73F3BF3F7B39" }, { "b" : "7F3B98CBB000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "33F6C90EE440B440A9ADBD0124BDCAE2FFE7C47B" }, { "b" : "7F3B98AA0000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F3B98628000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "DA6345B41E06A9001B5C51DEAF0784BE9D49810D" }, { "b" : "7F3B983BE000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "9787FF1FFFE8D82268D89F6BFA3AF07B3E9D140F" }, { "b" : "7F3B981BA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F3B97FB2000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F3B97C14000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F3B979FC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "92E0BE1929D28508CF9C6D5754C7EB48C12255B3" }, { "b" : "7F3B977DD000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F3B973EC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F3B98F3A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F3B971C7000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F3B96FAA000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F3B96D8E000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F3B96B80000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F3B96935000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "451E2EC02C8F974B21D1CE47F5D2C21A21C16A9F" }, { "b" : "7F3B966E3000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "0CDDF9D1F87FE7FF07D73DBE1D7CB57EE5BA802C" }, { "b" : "7F3B964D5000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "DD2893FC0AA3C58798F805D4C88EE086564D2EEB" }, { "b" : "7F3B962B8000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F3B95F3A000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, { "b" : "7F3B95BD5000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, { "b" : "7F3B959A1000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F3B9576B000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F3B954EA000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F3B95214000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "AFF9A2A2B6ACB066BF450CC4A0772B9DC04299CB" }, { "b" : "7F3B94FE2000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "04AADCB371B97A570756D69A0D5B3B0E6737E349" }, { "b" : "7F3B94DDE000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "9E885FB7F54DA464A3D2A978C71C1178A6EC7512" }, { "b" : "7F3B94BD3000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "27A5983855C50A37EFA526BB12EA48CDAE3B5FCA" }, { "b" : "7F3B949B8000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F3B94777000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F3B94448000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F3B94235000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F3B94031000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F3B93E28000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F3B93B9B000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F3B938F9000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F3B936C3000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F3B934AD000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F3B932A5000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F3B9307C000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F3B92E6D000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F3B92C23000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F3B9291A000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, { "b" : "7F3B926E2000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55cb0a04fae1]
 mongod(+0x23B7CF9) [0x55cb0a04ecf9]
 mongod(+0x23B8366) [0x55cb0a04f366]
 libpthread.so.0(+0x12890) [0x7f3b977ef890]
 mongod(_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv+0x228) [0x55cb08772418]
 mongod(_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE+0xCD) [0x55cb0877daad]
 mongod(_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE+0x27C) [0x55cb0876488c]
 mongod(_ZN5mongo26KVDatabaseCatalogEntryBase14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x568) [0x55cb088e8108]
 mongod(_ZN5mongo15KVStorageEngine11loadCatalogEPNS_16OperationContextE+0xA95) [0x55cb088e2885]
 mongod(_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE+0x22D) [0x55cb088e354d]
 mongod(+0xAB80BD) [0x55cb0874f0bd]
 mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0xBF5) [0x55cb08d990b5]
 mongod(+0x988F56) [0x55cb0861ff56]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x873) [0x55cb08705f73]
 mongod(main+0x9) [0x55cb0869cf99]
 libc.so.6(__libc_start_main+0xE7) [0x7f3b9740db97]
 mongod(_start+0x2A) [0x55cb08704a2a]
-----  END BACKTRACE  -----
Segmentation fault (core dumped)

      Sorry for the very long stack trace inserted, I hope they make some sense.

      For setting up MongoDB, I often do the same steps as below:

      • Create a location for database engine/storage
      • Download .tgz file, extract into its location
      • Create systemd service file to run it as service

      I work on Ubuntu 18.04, minimal version with XFCE DE, gcc/g++ v7.3.0

      Here is my installation script for MongoDB 4.0.2 that is occurring issue:

      export MONGO_VERSION=4.0.2
export MONGO_DOWNLOAD_URL=https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1804-$MONGO_VERSION.tgz
export MONGO_DIR=/storage/mongodb
sudo apt install libcurl3 openssl
sudo mkdir -p $MONGO_DIR/data
sudo chown ${USER} -R $MONGO_DIR
wget "$MONGO_DOWNLOAD_URL" -O mongo.tar.tgz
tar -zxvf mongo.tar.tgz -C $MONGO_DIR --strip-components=1

      And mongo.service in /etc/systemd/system

      [Unit]
Description=MongoDB
After=network.target[Service]
User=ndaidong
WorkingDirectory=/storage/mongodb
ExecStart=/storage/mongodb/bin/mongod --dbpath /storage/mongodb/data
Restart=always
RestartSec=500ms
StartLimitInterval=0[Install]
WantedBy=multi-user.target

      It works like as charm until today. Nothing special changes in my system.

      What's wrong with my configuration? And how to get it works again?

      Thank you for any advice.

        1. mongo fail at starting.png
          559 kB
          Dong Nguyen
        2. system info.png
          155 kB
          Dong Nguyen

            Assignee:
            Nick Brewer (Inactive)
            Reporter:
            Dong Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: