[SERVER-31124] WiredTiger (-31802) [1505722563:325909][1:0x7f5e4e884cc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Created: 18/Sep/17  Updated: 10/Oct/17  Resolved: 18/Sep/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Shalva Usubov [X] Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-31124-repair.tar.gz     Zip Archive mongodb.zip    
Issue Links:
Duplicate
is duplicated by SERVER-31125 file:WiredTiger.wt, connection: unabl... Closed
Operating System: ALL
Participants:

 Description   

I am running MongoDB v3.2.16 from official docker image.

I can't start mongo after space on the server was the end. Now I added few GB, but the MongoDB can't start.

2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] db version v3.2.16
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] git version: 056bf45128114e44c5358c7a8776fb582363e094
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] modules: none
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] build environment:
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] distmod: debian81
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] distarch: x86_64
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-09-18T08:16:03.311+0000 I CONTROL [initandlisten] options: { storage: { journal:

{ enabled: false }

} }
2017-09-18T08:16:03.315+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-09-18T08:16:03.315+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2017-09-18T08:16:03.315+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-09-18T08:16:03.325+0000 E STORAGE [initandlisten] WiredTiger (-31802) [1505722563:325909][1:0x7f5e4e884cc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2017-09-18T08:16:03.325+0000 E STORAGE [initandlisten] WiredTiger (0) [1505722563:325939][1:0x7f5e4e884cc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2017-09-18T08:16:03.325+0000 E STORAGE [initandlisten] WiredTiger (0) [1505722563:325945][1:0x7f5e4e884cc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2017-09-18T08:16:03.326+0000 E STORAGE [initandlisten] WiredTiger (0) [1505722563:325985][1:0x7f5e4e884cc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2017-09-18T08:16:03.326+0000 I - [initandlisten] Assertion: 28718:-31802: WT_ERROR: non-specific WiredTiger error
2017-09-18T08:16:03.326+0000 I STORAGE [initandlisten] exception in initAndListen: 28718 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2017-09-18T08:16:03.326+0000 I CONTROL [initandlisten] dbexit: rc: 100



 Comments   
Comment by Mark Agarunov [ 18/Sep/17 ]

Hello Shalva,

Unfortunately, this error indicates that there was corruption on the disk. In this situation, my best recommendation would be to resync the affected node or restore from a backup if possible.

Thanks,
Mark

Comment by Shalva Usubov [X] [ 18/Sep/17 ]

Hello Mark,

Unfortunately oi not worked, I provided output: https://gist.github.com/shaliko/994ba1532a2d43288f2ba2b06b1c4f06

Before running this script I copy all $dbpath.

Comment by Mark Agarunov [ 18/Sep/17 ]

Hello Shalva,

I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Comment by Shalva Usubov [X] [ 18/Sep/17 ]

Hello Mark,

Uploaded WiredTiger.wt and WiredTiger.turtle.

Thanks for the help!

Comment by Mark Agarunov [ 18/Sep/17 ]

Hello Shalva,

Thank you for the report. Would you please upload the WiredTiger.wt and WiredTiger.turtle files so we can attempt a repair?

Thanks,
Mark

Generated at Thu Feb 08 04:26:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.