[SERVER-31377] WiredTiger.wt corrupt Created: 04/Oct/17  Updated: 27/Jul/18  Resolved: 04/Oct/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: James Groat Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rge, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-31377-repair.tar.gz     Zip Archive WiredTiger-files.zip    
Operating System: Linux
Participants:

 Description   

Hello,

I have seen other issues with similar corruption as I'm seeing on my mongo instance. I've attached my wiredtiger files in the hopes that you can repair them as you have others. My server suffered a

Thanks!

2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] MongoDB starting : pid=13877 port=27017 dbpath=/data/db 64-bit host=288259c1abe3
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] db version v3.4.9
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] modules: none
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] build environment:
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten]     distmod: debian81
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-04T03:57:15.471+0000 I CONTROL  [initandlisten] options: {}
2017-10-04T03:57:15.496+0000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-10-04T03:57:15.496+0000 I STORAGE  [initandlisten]
2017-10-04T03:57:15.496+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-04T03:57:15.496+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-04T03:57:15.496+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1459M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-04T03:57:15.546+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1507089435:546504][13877:0x7f5e8be59d40], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2017-10-04T03:57:15.546+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507089435:546613][13877:0x7f5e8be59d40], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2017-10-04T03:57:15.546+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507089435:546637][13877:0x7f5e8be59d40], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2017-10-04T03:57:15.546+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507089435:546676][13877:0x7f5e8be59d40], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2017-10-04T03:57:15.557+0000 I -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 269
2017-10-04T03:57:15.572+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2017-10-04T03:57:15.572+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2017-10-04T03:57:15.572+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2017-10-04T03:57:15.572+0000 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2017-10-04T03:57:15.572+0000 I CONTROL  [initandlisten] now exiting
2017-10-04T03:57:15.572+0000 I CONTROL  [initandlisten] shutting down with code:100



 Comments   
Comment by Mark Agarunov [ 04/Oct/17 ]

Hello jamesgroat,

Thanks for your response. I'm glad to hear that this fixed the issue and everything is working again. To prevent this type of problem in the future, we recommend implementing regular backups and/or replication to mitigate any issues related to unreliable storage layers or server failures.

Thanks,
Mark

Comment by James Groat [ 04/Oct/17 ]

Mark,

Thank you! This helped and mongod was able to repair the database.

Thanks,

James

Comment by Mark Agarunov [ 04/Oct/17 ]

Hello jamesgroat,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Generated at Thu Feb 08 04:26:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.