Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26237

Mongodb failed to start after a disk 100% full problem

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Duplicate
    • 3.2.3
    • None
    • WiredTiger
    • Linux

    Description

      My server had a 'No space left on device' problem because our disk was running out of inode early today.

      But after we fixed the disk problem, our mongodb was still unable to start or repair.

      I desperately need your help : (

      And the log information as follow:

      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] MongoDB starting : pid=10992 port=27017 dbpath=/ick/data2 64-bit host=localhost.localdomain
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] db version v3.2.3
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] allocator: tcmalloc
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] modules: none
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] build environment:
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     distmod: rhel62
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     distarch: x86_64
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     target_arch: x86_64
      2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/ick/data2" } }
      2016-07-25T17:29:53.847+0800 I -        [initandlisten] Detected data files in /ick/data2 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898124][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 196608: block header checksum of 675114611 doesn't match expected checksum of 1687733155
      2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898190][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1469438993:898209][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2016-07-25T17:29:53.898+0800 I -        [initandlisten] Fatal Assertion 28558
      2016-07-25T17:29:53.898+0800 I -        [initandlisten] 
       
      ***aborting after fassert() failure
       
       
      2016-07-25T17:29:53.946+0800 F -        [initandlisten] Got signal: 6 (Aborted).
       
       0x130e1e2 0x130d119 0x130d922 0x3ee060f7e0 0x3ee0232625 0x3ee0233e05 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x199067c 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a20b7f 0x1a20c5e 0x19df078 0x107322e 0x106f6a3 0xf99268 0x974701 0x97815d 0x3ee021ed5d 0x972279
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F0D119"},{"b":"400000","o":"F0D922"},{"b":"3EE0600000","o":"F7E0"},{"b":"3EE0200000","o":"32625","s":"gsignal"},{"b":"3EE0200000","o":"33E05","s":"abort"},{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B3E3"},{"b":"400000","o":"1657E3C","s":"__wt_eventv"},{"b":"400000","o":"1657FDD","s":"__wt_err"},{"b":"400000","o":"16583C4","s":"__wt_panic"},{"b":"400000","o":"159067C","s":"__wt_block_extlist_read"},{"b":"400000","o":"1590BF3","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15919DE"},{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"},{"b":"400000","o":"15E2140","s":"__wt_conn_btree_open"},{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"},{"b":"400000","o":"1657117","s":"__wt_session_get_btree"},{"b":"400000","o":"165726B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"},{"b":"400000","o":"1654415"},{"b":"400000","o":"1620B7F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1620C5E","s":"__wt_metadata_cursor"},{"b":"400000","o":"15DF078","s":"wiredtiger_open"},{"b":"400000","o":"C7322E","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6F6A3"},{"b":"400000","o":"B99268","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"574701","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57815D","s":"main"},{"b":"3EE0200000","o":"1ED5D","s":"__libc_start_main"},{"b":"400000","o":"572279"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }, { "b" : "7FFEC8C76000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "5D42D8781F5FF4C06B4F839B85B21786AA44488D" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2]
       mongod(+0xF0D119) [0x130d119]
       mongod(+0xF0D922) [0x130d922]
       libpthread.so.0(+0xF7E0) [0x3ee060f7e0]
       libc.so.6(gsignal+0x35) [0x3ee0232625]
       libc.so.6(abort+0x175) [0x3ee0233e05]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32]
       mongod(+0xC8B3E3) [0x108b3e3]
       mongod(__wt_eventv+0x40C) [0x1a57e3c]
       mongod(__wt_err+0x8D) [0x1a57fdd]
       mongod(__wt_panic+0x24) [0x1a583c4]
       mongod(__wt_block_extlist_read+0x6C) [0x199067c]
       mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3]
       mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7]
       mongod(+0x15919DE) [0x19919de]
       mongod(__wt_btree_open+0xC68) [0x19ae1a8]
       mongod(__wt_conn_btree_open+0x140) [0x19e2140]
       mongod(__wt_session_get_btree+0xEB) [0x1a56bfb]
       mongod(__wt_session_get_btree+0x607) [0x1a57117]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b]
       mongod(__wt_curfile_open+0x1E8) [0x19f02f8]
       mongod(+0x1654415) [0x1a54415]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a20b7f]
       mongod(__wt_metadata_cursor+0x7E) [0x1a20c5e]
       mongod(wiredtiger_open+0x15F8) [0x19df078]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x45E) [0x107322e]
       mongod(+0xC6F6A3) [0x106f6a3]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf99268]
       mongod(_ZN5mongo13initAndListenEi+0x321) [0x974701]
       mongod(main+0x15D) [0x97815d]
       libc.so.6(__libc_start_main+0xFD) [0x3ee021ed5d]
       mongod(+0x572279) [0x972279]
      -----  END BACKTRACE  -----
      

      Attachments

        1. mongodb.log
          17 kB
        2. repair_attempt.tgz
          12 kB
        3. WiredTiger.turtle
          0.9 kB
        4. WiredTiger.wt
          244 kB

        Activity

          People

            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Paddyzhu Paddy Zhu [X]
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: