Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34560

file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.7
    • Component/s: WiredTiger
    • Linux

      service mongod unexpectedly changed state from "active" to "failed". repair tool did not help.

      log file output:

      2018-04-19T05:24:30.688+0300 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-19T05:24:30.696+0300 I CONTROL  [initandlisten] MongoDB starting : pid=2648 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] modules: none
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] build environment:
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-04-19T05:24:30.697+0300 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-19T05:24:30.751+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-19T05:24:30.751+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] 
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-19T05:24:30.762+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1524104670:761987][2648:0x7f15a3c44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-19T05:24:30.762+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1524104670:762051][2648:0x7f15a3c44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-19T05:24:30.762+0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-19T05:24:30.762+0300 I -        [initandlisten] 

***aborting after fassert() failure


2018-04-19T05:24:30.794+0300 F -        [initandlisten] Got signal: 6 (Aborted).

 0x556257693ea1 0x5562576930b9 0x55625769359d 0x7f15a2849390 0x7f15a24a3428 0x7f15a24a502a 0x556256942733 0x5562573a7c26 0x55625694c9cc 0x55625694cbf1 0x55625694ce49 0x55625803aafc 0x556258039269 0x556257fedd03 0x55625807865b 0x556258078b9d 0x556258078e5c 0x556257ffc4f1 0x55625806ea08 0x5562580389ae 0x556258038a8b 0x556257fea05d 0x55625738ca6f 0x5562573852ec 0x556257278a00 0x55625692e95e 0x55625694e1f6 0x7f15a248e830 0x5562569ace09
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"556256128000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"556256128000","o":"156B0B9"},{"b":"556256128000","o":"156B59D"},{"b":"7F15A2838000","o":"11390"},{"b":"7F15A246E000","o":"35428","s":"gsignal"},{"b":"7F15A246E000","o":"3702A","s":"abort"},{"b":"556256128000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"556256128000","o":"127FC26"},{"b":"556256128000","o":"8249CC","s":"__wt_eventv"},{"b":"556256128000","o":"824BF1","s":"__wt_err"},{"b":"556256128000","o":"824E49","s":"__wt_panic"},{"b":"556256128000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"556256128000","o":"1F11269","s":"__wt_metadata_search"},{"b":"556256128000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"556256128000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"556256128000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"556256128000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"556256128000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"556256128000","o":"1F46A08"},{"b":"556256128000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"556256128000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"556256128000","o":"1EC205D","s":"wiredtiger_open"},{"b":"556256128000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"556256128000","o":"125D2EC"},{"b":"556256128000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"556256128000","o":"80695E"},{"b":"556256128000","o":"8261F6","s":"main"},{"b":"7F15A246E000","o":"20830","s":"__libc_start_main"},{"b":"556256128000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "556256128000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFD781FB000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7F15A37C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F15A3380000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F15A3178000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F15A2F74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F15A2C6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F15A2A55000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F15A2838000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F15A246E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F15A3A2D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x556257693ea1]
 mongod(+0x156B0B9) [0x5562576930b9]
 mongod(+0x156B59D) [0x55625769359d]
 libpthread.so.0(+0x11390) [0x7f15a2849390]
 libc.so.6(gsignal+0x38) [0x7f15a24a3428]
 libc.so.6(abort+0x16A) [0x7f15a24a502a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x556256942733]
 mongod(+0x127FC26) [0x5562573a7c26]
 mongod(__wt_eventv+0x3D7) [0x55625694c9cc]
 mongod(__wt_err+0x9D) [0x55625694cbf1]
 mongod(__wt_panic+0x24) [0x55625694ce49]
 mongod(__wt_turtle_read+0x26C) [0x55625803aafc]
 mongod(__wt_metadata_search+0x99) [0x556258039269]
 mongod(__wt_conn_btree_open+0x73) [0x556257fedd03]
 mongod(__wt_session_get_btree+0xFB) [0x55625807865b]
 mongod(__wt_session_get_btree+0x63D) [0x556258078b9d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x556258078e5c]
 mongod(__wt_curfile_open+0x161) [0x556257ffc4f1]
 mongod(+0x1F46A08) [0x55625806ea08]
 mongod(__wt_metadata_cursor_open+0x6E) [0x5562580389ae]
 mongod(__wt_metadata_cursor+0x4B) [0x556258038a8b]
 mongod(wiredtiger_open+0x164D) [0x556257fea05d]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55625738ca6f]
 mongod(+0x125D2EC) [0x5562573852ec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x556257278a00]
 mongod(+0x80695E) [0x55625692e95e]
 mongod(main+0x966) [0x55625694e1f6]
 libc.so.6(__libc_start_main+0xF0) [0x7f15a248e830]
 mongod(_start+0x29) [0x5562569ace09]
-----  END BACKTRACE  -----

      (sudo mongod --repair --dbpath /var/log/mongodb/ --storageEngine wiredTiger)
      --reapir output:

      2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] MongoDB starting : pid=2675 port=27017 dbpath=/var/log/mongodb/ 64-bit host=ColeteElastic
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] modules: none
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] build environment:
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/log/mongodb/", engine: "wiredTiger" } }
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] 
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2018-04-19T05:48:43.748+0300 I STORAGE  [initandlisten] Repairing size cache
2018-04-19T05:48:43.749+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2018-04-19T05:48:43.749+0300 I STORAGE  [initandlisten] Repairing catalog metadata
2018-04-19T05:48:43.750+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I STORAGE  [initandlisten] finished checking dbs
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-04-19T05:48:43.766+0300 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-04-19T05:48:43.793+0300 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-04-19T05:48:43.793+0300 I CONTROL  [initandlisten] now exiting
2018-04-19T05:48:43.793+0300 I CONTROL  [initandlisten] shutting down with code:0

      more info:
      What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?

      • storage device attached over network, HDD, RAID (NetApp native)

      Would you please check the integrity of your disks?

      • yes

      Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
      -yes, always this version of mongo

      Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?

      • no

      Have you ever restored this instance from backups?

      • no

      What method do you use to create backups?

      • there was no back-ups at the moment.

      When was the underlying filesystem last checked and is it currently marked clean?

      • never

        1. WiredTigerLAS.wt
          4 kB
        2. WiredTiger.wt
          244 kB
        3. WiredTiger.turtle
          1.0 kB
        4. WiredTiger.lock
          0.0 kB
        5. WiredTiger
          0.0 kB
        6. repair_attempt.tar.gz
          15 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            Durbala Maxim [X]
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: