[SERVER-29023]  file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 30/Apr/17  Updated: 27/Jul/18  Resolved: 14/Sep/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.11
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: jonny da Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envh, rpo, rpu, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File repair_attempt.tar.gz     Zip Archive wiredTiger.zip    
Operating System: Linux
Participants:

 Description   

Hello,
my mongod sudden shutdown , I am unable to restart Mongod.

This is my restart log

2017-04-30T05:56:11.884+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] MongoDB starting : pid=61275 port=27017 dbpath=/var/lib/mongo 64-bit host=download
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] db version v3.2.11
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] modules: none
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] build environment:
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten]     distmod: rhel70
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten]     distarch: x86_64
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-04-30T05:56:11.932+0200 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-04-30T05:56:11.932+0200 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2017-04-30T05:56:11.932+0200 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-04-30T05:56:11.932+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-04-30T05:56:11.937+0200 E STORAGE  [initandlisten] WiredTiger (0) [1493524571:937885][61275:0x7f9caa384dc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2017-04-30T05:56:11.937+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1493524571:937937][61275:0x7f9caa384dc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-04-30T05:56:11.937+0200 I -        [initandlisten] Fatal Assertion 28558
2017-04-30T05:56:11.937+0200 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-04-30T05:56:11.956+0200 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x132fa32 0x132eb89 0x132f392 0x7f9ca8ffc370 0x7f9ca8c611d7 0x7f9ca8c628c8 0x12b58b2 0x10af973 0x97194c 0x971a21 0x971be5 0x1a64c46 0x1a63696 0x1a18921 0x1aa1f7e 0x1aa2625 0x1aa2aab 0x1a27586 0x1a9d3cd 0x1a62cff 0x1a62e0e 0x1a14e63 0x1096e0f 0x10930c3 0xfbbd98 0x9b9cde 0x9bc250 0x97251d 0x7f9ca8c4db35 0x9b6607
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F2FA32","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F2EB89"},{"b":"400000","o":"F2F392"},{"b":"7F9CA8FED000","o":"F370"},{"b":"7F9CA8C2C000","o":"351D7","s":"gsignal"},{"b":"7F9CA8C2C000","o":"368C8","s":"abort"},{"b":"400000","o":"EB58B2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAF973"},{"b":"400000","o":"57194C","s":"__wt_eventv"},{"b":"400000","o":"571A21","s":"__wt_err"},{"b":"400000","o":"571BE5","s":"__wt_panic"},{"b":"400000","o":"1664C46","s":"__wt_turtle_read"},{"b":"400000","o":"1663696","s":"__wt_metadata_search"},{"b":"400000","o":"1618921","s":"__wt_conn_btree_open"},{"b":"400000","o":"16A1F7E","s":"__wt_session_get_btree"},{"b":"400000","o":"16A2625","s":"__wt_session_get_btree"},{"b":"400000","o":"16A2AAB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1627586","s":"__wt_curfile_open"},{"b":"400000","o":"169D3CD"},{"b":"400000","o":"1662CFF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1662E0E","s":"__wt_metadata_cursor"},{"b":"400000","o":"1614E63","s":"wiredtiger_open"},{"b":"400000","o":"C96E0F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C930C3"},{"b":"400000","o":"BBBD98","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B9CDE"},{"b":"400000","o":"5BC250","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57251D","s":"main"},{"b":"7F9CA8C2C000","o":"21B35","s":"__libc_start_main"},{"b":"400000","o":"5B6607"}],"processInfo":{ "mongodbVersion" : "3.2.11", "gitVersion" : "009580ad490190ba33d1c6253ebd8d91808923e4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-514.2.2.el7.x86_64", "version" : "#1 SMP Tue Dec 6 23:06:41 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "88BEAACF5E9DF7E467AF167F9F4F31D20015315F" }, { "b" : "7FFDC2AC2000", "elfType" : 3, "buildId" : "183CE4B56A9471419F233CCEF078E0504837ABF5" }, { "b" : "7F9CA9F17000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D0018CA5E24522ED0DC1844556FA8DBC4B39D5C3" }, { "b" : "7F9CA9B2D000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8756D2315BF50F8610875B1AFF128198FB9D202D" }, { "b" : "7F9CA9925000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "82E77ADE22BC9FFF8D3458BD37331E7EDF174C28" }, { "b" : "7F9CA9721000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "C5F560504E1AF52E29679C3B52FF11121015D6BB" }, { "b" : "7F9CA941F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "721C7CC9488EFA25F83B48AF713AB27DBE48EF3E" }, { "b" : "7F9CA9209000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7F9CA8FED000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C3DEB1FA27CD0C1C3CC575B944ABACBA0698B0F2" }, { "b" : "7F9CA8C2C000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8B2C421716985B927AA0CAF2A05D0B1F452367F7" }, { "b" : "7F9CAA185000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "8F3E366E2DB73C330A3791DEAE31AE9579099B44" }, { "b" : "7F9CA89DE000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "A2499C359AA179EE23324ED949C0E508E4434F10" }, { "b" : "7F9CA86F7000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "E09A34D9083DC6FEAF7018C09D55631DEEE2836D" }, { "b" : "7F9CA84F3000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "BF54B7C8932E450769FBBB8B18864D1DD70BBC67" }, { "b" : "7F9CA82C1000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "BF8F00D7CB849ADB0B7A4703BC7B8D66AEE6A49C" }, { "b" : "7F9CA80AB000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F9CA7E9C000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "1E7A92FDD6FB3871DA97F4BCA2E147E72B6B6E1F" }, { "b" : "7F9CA7C98000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F9CA7A7E000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FE7AE845A123A3DFC0FDC2408BCBC2BA8B61B158" }, { "b" : "7F9CA7857000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "76687CA31A406854DF3BCF8D03055656F56E6892" }, { "b" : "7F9CA75F6000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "AE64AA461A26E01F60408013D361749D56DD0AE1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x132fa32]
 mongod(+0xF2EB89) [0x132eb89]
 mongod(+0xF2F392) [0x132f392]
 libpthread.so.0(+0xF370) [0x7f9ca8ffc370]
 libc.so.6(gsignal+0x37) [0x7f9ca8c611d7]
 libc.so.6(abort+0x148) [0x7f9ca8c628c8]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12b58b2]
 mongod(+0xCAF973) [0x10af973]
 mongod(__wt_eventv+0x388) [0x97194c]
 mongod(__wt_err+0x8D) [0x971a21]
 mongod(__wt_panic+0x24) [0x971be5]
 mongod(__wt_turtle_read+0x2D6) [0x1a64c46]
 mongod(__wt_metadata_search+0x1F6) [0x1a63696]
 mongod(__wt_conn_btree_open+0x61) [0x1a18921]
 mongod(__wt_session_get_btree+0xDE) [0x1aa1f7e]
 mongod(__wt_session_get_btree+0x785) [0x1aa2625]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1aa2aab]
 mongod(__wt_curfile_open+0x86) [0x1a27586]
 mongod(+0x169D3CD) [0x1a9d3cd]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a62cff]
 mongod(__wt_metadata_cursor+0x7E) [0x1a62e0e]
 mongod(wiredtiger_open+0x14B3) [0x1a14e63]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x1096e0f]
 mongod(+0xC930C3) [0x10930c3]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfbbd98]
 mongod(+0x5B9CDE) [0x9b9cde]
 mongod(_ZN5mongo13initAndListenEi+0x10) [0x9bc250]
 mongod(main+0x15D) [0x97251d]
 libc.so.6(__libc_start_main+0xF5) [0x7f9ca8c4db35]
 mongod(+0x5B6607) [0x9b6607]
-----  END BACKTRACE  -----

I've uploaded my WiredTiger.* files in the hopes that someone can respond with fixed WiredTiger.* files. I would be extremely grateful...thank you!

If anyone has any further questions, feel free to ask. Thank you!!



 Comments   
Comment by Kelsey Schubert [ 14/Sep/17 ]

Hi jonnyda,

Sorry for the delay getting back to you. I'm glad that you're no longer encountering issues. Since mongodump is not a file-level backup, encountering issues after restoring suggests an issue external to MongoDB since it would effectively be applying a series of inserts.

Kind regards,
Kelsey

Comment by jonny da [ 02/May/17 ]

Hello Thomas,
Thanks for your help.
I had backup my database (losing 1 day data) but still seeing it errors, then i decided to update version 3.4 and changed storage engine to mmapv1 and now it not crash.
pls see my server info below ;
1 . Our server using local HDDs (raid 1, ext4) with 1TB ,
2. I have not check filesystem before, but no have any error though
3.Our server always run same verison of MongoDB
4. I have not copy and move database file
5.I have tried backup but sitll got same error after few minutes
6. I have backup database daily using mongodump

Hope this info can help you determine our issue
Thank you

Comment by Kelsey Schubert [ 01/May/17 ]

Hello jonnyda,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Determining the root cause of corruption without a straightforward reproduction is very challenging, in this case, it is likely the result of a fault at the disk layer. To better understand what has occurred, would you please answer the following questions?

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. When was the underlying filesystem last checked and is it currently marked clean?
  3. Would you please check the integrity of your disks?
  4. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  5. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  6. Have you ever restored this instance from backups?
  7. What method do you use to create backups?

If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate.

Thank you,
Thomas

Generated at Thu Feb 08 04:19:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.