Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-43705

Mongodb Crashed with the Got signal: 6 (Aborted)

    • Type: Icon: Question Question
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.22
    • Component/s: WiredTiger
    • Labels:
      None

      I've stop and mount my data volume into a docker container running to new storage. After that I could not restart my mongod using the same data.

      I have attachment file Wiretiger.wt and Wiretiger.turtle. Please help me repair it.

      Thank you in advance.

      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=8a673aab6e94
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] db version v3.4.22
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] git version: aa313e18da568c4afd120f99e56d2da7d7fcdd58
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] allocator: tcmalloc
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] modules: none
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] build environment:
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] distmod: ubuntu1604
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] distarch: x86_64
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] target_arch: x86_64
      2019-09-29T02:15:43.086+0700 I CONTROL [initandlisten] options: {}
      2019-09-29T02:15:43.089+0700 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
      2019-09-29T02:15:43.094+0700 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2019-09-29T02:15:43.094+0700 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
      2019-09-29T02:15:43.095+0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15460M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
      2019-09-29T02:15:44.400+0700 I STORAGE [initandlisten] WiredTiger message [1569698144:400387][1:0x7faed3d5bd40], txn-recover: Main recovery loop: starting at 63/128
      2019-09-29T02:15:44.408+0700 I STORAGE [initandlisten] WiredTiger message [1569698144:408627][1:0x7faed3d5bd40], txn-recover: Recovering log 63 through 64
      2019-09-29T02:15:44.549+0700 I STORAGE [initandlisten] WiredTiger message [1569698144:549893][1:0x7faed3d5bd40], txn-recover: Recovering log 64 through 64
      2019-09-29T02:15:44.917+0700 E STORAGE [initandlisten] WiredTiger error (0) [1569698144:917503][1:0x7faed3d5bd40], file:collection-53--4208631159875725994.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 12288: block header checksum of 3011857163 doesn't match expected checksum of 1083219503
      2019-09-29T02:15:44.917+0700 E STORAGE [initandlisten] WiredTiger error (0) [1569698144:917577][1:0x7faed3d5bd40], file:collection-53--4208631159875725994.wt, WT_SESSION.open_cursor: collection-53--4208631159875725994.wt: encountered an illegal file format or internal value
      2019-09-29T02:15:44.917+0700 E STORAGE [initandlisten] WiredTiger error (-31804) [1569698144:917603][1:0x7faed3d5bd40], file:collection-53--4208631159875725994.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2019-09-29T02:15:44.917+0700 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
      2019-09-29T02:15:44.917+0700 I - [initandlisten]
      ***aborting after fassert() failure
      
      2019-09-29T02:15:44.925+0700 I - [WTJournalFlusher] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
      2019-09-29T02:15:44.925+0700 I - [WTJournalFlusher]
      ***aborting after fassert() failure
      
      2019-09-29T02:15:44.961+0700 F - [initandlisten] Got signal: 6 (Aborted).
      0x5600bc250d41 0x5600bc24ff59 0x5600bc25043d 0x7faed295b390 0x7faed25b5428 0x7faed25b702a 0x5600bb4e14e7 0x5600bbf55266 0x5600bb4ebb64 0x5600bb4ebd89 0x5600bb4ebfeb 0x5600bcb5d8cf 0x5600bcb5de1b 0x5600bcb5a40d 0x5600bcb5efe7 0x5600bcb78a03 0x5600bcbb1f8b 0x5600bcc3caab 0x5600bcc3cfed 0x5600bcc3d2ac 0x5600bcbc0851 0x5600bcc32d98 0x5600bcbd9c6b 0x5600bcc32b34 0x5600bcc33385 0x5600bbf4e517 0x5600bbf4c29c 0x5600bbf487b4 0x5600bbf46ddc 0x5600bbf342db 0x5600bbe75c23 0x5600bbe7a9e4 0x5600bbf31e29 0x5600bbe249b0 0x5600bb4cc3a3 0x5600bb4ed3d6 0x7faed25a0830 0x5600bb54d7b9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"5600BAC9E000","o":"15B2D41","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5600BAC9E000","o":"15B1F59"},{"b":"5600BAC9E000","o":"15B243D"},{"b":"7FAED294A000","o":"11390"},{"b":"7FAED2580000","o":"35428","s":"gsignal"},{"b":"7FAED2580000","o":"3702A","s":"abort"},{"b":"5600BAC9E000","o":"8434E7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5600BAC9E000","o":"12B7266"},{"b":"5600BAC9E000","o":"84DB64","s":"__wt_eventv"},{"b":"5600BAC9E000","o":"84DD89","s":"__wt_err"},{"b":"5600BAC9E000","o":"84DFEB","s":"__wt_panic"},{"b":"5600BAC9E000","o":"1EBF8CF","s":"__wt_block_extlist_read"},{"b":"5600BAC9E000","o":"1EBFE1B","s":"__wt_block_extlist_read_avail"},{"b":"5600BAC9E000","o":"1EBC40D","s":"__wt_block_checkpoint_load"},{"b":"5600BAC9E000","o":"1EC0FE7"},{"b":"5600BAC9E000","o":"1EDAA03","s":"__wt_btree_open"},{"b":"5600BAC9E000","o":"1F13F8B","s":"__wt_conn_btree_open"},{"b":"5600BAC9E000","o":"1F9EAAB","s":"__wt_session_get_btree"},{"b":"5600BAC9E000","o":"1F9EFED","s":"__wt_session_get_btree"},{"b":"5600BAC9E000","o":"1F9F2AC","s":"__wt_session_get_btree_ckpt"},{"b":"5600BAC9E000","o":"1F22851","s":"__wt_curfile_open"},{"b":"5600BAC9E000","o":"1F94D98"},{"b":"5600BAC9E000","o":"1F3BC6B","s":"__wt_curtable_open"},{"b":"5600BAC9E000","o":"1F94B34"},{"b":"5600BAC9E000","o":"1F95385"},{"b":"5600BAC9E000","o":"12B0517","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"5600BAC9E000","o":"12AE29C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"5600BAC9E000","o":"12AA7B4","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"5600BAC9E000","o":"12A8DDC","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"5600BAC9E000","o":"12962DB","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"5600BAC9E000","o":"11D7C23","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb"},{"b":"5600BAC9E000","o":"11DC9E4","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"5600BAC9E000","o":"1293E29"},{"b":"5600BAC9E000","o":"11869B0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5600BAC9E000","o":"82E3A3"},{"b":"5600BAC9E000","o":"84F3D6","s":"main"},{"b":"7FAED2580000","o":"20830","s":"__libc_start_main"},{"b":"5600BAC9E000","o":"8AF7B9","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.22", "gitVersion" : "aa313e18da568c4afd120f99e56d2da7d7fcdd58", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.10.1.el7.x86_64", "version" : "#1 SMP Mon Mar 18 15:06:45 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "5600BAC9E000", "elfType" : 3, "buildId" : "847B3AF37716C2247DA289A43D9854E98053D28B" }, { "b" : "7FFDF9AB8000", "elfType" : 3, "buildId" : "63023E5151C4D5BBDB7F6F85A6F95A6B95DC780E" }, { "b" : "7FAED38D7000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7FAED3492000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7FAED328A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7FAED3086000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7FAED2D7D000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7FAED2B67000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FAED294A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7FAED2580000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7FAED3B40000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5600bc250d41]
       mongod(+0x15B1F59) [0x5600bc24ff59]
       mongod(+0x15B243D) [0x5600bc25043d]
       libpthread.so.0(+0x11390) [0x7faed295b390]
       libc.so.6(gsignal+0x38) [0x7faed25b5428]
       libc.so.6(abort+0x16A) [0x7faed25b702a]
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5600bb4e14e7]
       mongod(+0x12B7266) [0x5600bbf55266]
       mongod(__wt_eventv+0x3D7) [0x5600bb4ebb64]
       mongod(__wt_err+0x9D) [0x5600bb4ebd89]
       mongod(__wt_panic+0x2E) [0x5600bb4ebfeb]
       mongod(__wt_block_extlist_read+0x8F) [0x5600bcb5d8cf]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x5600bcb5de1b]
       mongod(__wt_block_checkpoint_load+0x26D) [0x5600bcb5a40d]
       mongod(+0x1EC0FE7) [0x5600bcb5efe7]
       mongod(__wt_btree_open+0xB53) [0x5600bcb78a03]
       mongod(__wt_conn_btree_open+0x18B) [0x5600bcbb1f8b]
       mongod(__wt_session_get_btree+0xFB) [0x5600bcc3caab]
       mongod(__wt_session_get_btree+0x63D) [0x5600bcc3cfed]
       mongod(__wt_session_get_btree_ckpt+0x14C) [0x5600bcc3d2ac]
       mongod(__wt_curfile_open+0x161) [0x5600bcbc0851]
       mongod(+0x1F94D98) [0x5600bcc32d98]
       mongod(__wt_curtable_open+0x12B) [0x5600bcbd9c6b]
       mongod(+0x1F94B34) [0x5600bcc32b34]
       mongod(+0x1F95385) [0x5600bcc33385]
       mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x77) [0x5600bbf4e517]
       mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x5600bbf4c29c]
       mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x64) [0x5600bbf487b4]
       mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x47C) [0x5600bbf46ddc]
       mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x23B) [0x5600bbf342db]
       mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x2C3) [0x5600bbe75c23]
       mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0xAA4) [0x5600bbe7a9e4]
       mongod(+0x1293E29) [0x5600bbf31e29]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5600bbe249b0]
       mongod(+0x82E3A3) [0x5600bb4cc3a3]
       mongod(main+0x966) [0x5600bb4ed3d6]
       libc.so.6(__libc_start_main+0xF0) [0x7faed25a0830]
       mongod(_start+0x29) [0x5600bb54d7b9]
      ----- END BACKTRACE -----// code placeholder
      

        1. repair_attempt.tgz
          5 kB
        2. WiredTiger.turtle
          1.0 kB
        3. WiredTiger.wt
          124 kB

            Assignee:
            daniel.hatcher@mongodb.com Danny Hatcher (Inactive)
            Reporter:
            minhhoanghcm96@gmail.com Minh Hoàng Lê Anh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: