[SERVER-34607]  WT_ERROR non-specific WiredTiger error Created: 23/Apr/18  Updated: 14/Aug/18  Resolved: 25/Apr/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.16
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Chunfeng Zhao Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: docker, envc, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

After an unclean shutdown of a docker container running mongo:3.2 tag, mongod fails to start. I have tried a --repair with :3.2 and :latest tags with no success.

I've attached my WiredTiger.wt and WiredTiger.turtle files for your review.

Any ideas? Thanks so much!

[1524451802:484055][30835:0x7fcd93ba7740], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
[1524451802:484080][30835:0x7fcd93ba7740], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
[1524451802:484085][30835:0x7fcd93ba7740], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
[1524451802:484090][30835:0x7fcd93ba7740], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
lt-wt: WT_ERROR: non-specific WiredTiger error



 Comments   
Comment by Kelsey Schubert [ 25/Apr/18 ]

Hi twinklezcf,

Thanks for your response. I'm glad to hear that this fixed the issue and everything is working again. To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures:

Thank you,
Kelsey

Comment by Chunfeng Zhao [ 24/Apr/18 ]

Hi Kelsey,
Too much thanks for your quickly response and fixed the issue. We can use the mongodb now. Thank you very much again.

2018-04-24T10:00:17.904+0800 I NETWORK [initandlisten] waiting for connections on port 27017
2018-04-24T10:00:17.906+0800 I REPL [ReplicationExecutor] New replica set config in use: { _id: "rs0", version: 40476, protocolVersion: 1, members: [ { _id: 0, host: "localhost:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults:

{ w: 1, wtimeout: 0 }

, replicaSetId: ObjectId('59f1f68c1d74adcbc93e11d8') } }
2018-04-24T10:00:17.906+0800 I REPL [ReplicationExecutor] This node is localhost:27017 in the config
2018-04-24T10:00:17.906+0800 I REPL [ReplicationExecutor] transition to STARTUP2
2018-04-24T10:00:17.906+0800 I REPL [ReplicationExecutor] Starting replication applier threads
2018-04-24T10:00:17.906+0800 I REPL [ReplicationExecutor] transition to RECOVERING
2018-04-24T10:00:17.907+0800 I REPL [ReplicationExecutor] transition to SECONDARY
2018-04-24T10:00:17.907+0800 I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected
2018-04-24T10:00:17.907+0800 I REPL [ReplicationExecutor] dry election run succeeded, running for election
2018-04-24T10:00:17.929+0800 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 15
2018-04-24T10:00:17.929+0800 I REPL [ReplicationExecutor] transition to PRIMARY
2018-04-24T10:00:18.019+0800 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2018-04-24T10:00:18.907+0800 I REPL [rsSync] transition to primary complete; database writes are now permitted

Comment by Kelsey Schubert [ 23/Apr/18 ]

Hi twinklezcf,

Thank you for your report. I've attached a repair attempt, repair_attempt.tar.gz, of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node so that we can further investigate.

Thank you,
Kelsey

Comment by Chunfeng Zhao [ 23/Apr/18 ]

mongod --dbpath=/data/mongo --smallfiles --replSet rs0
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] MongoDB starting : pid=30872 port=27017 dbpath=/data/mongo 64-bit host=089bb18d044b
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] db version v3.2.16
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] git version: 056bf45128114e44c5358c7a8776fb582363e094
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] allocator: tcmalloc
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] modules: none
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] build environment:
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] distmod: debian81
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] distarch: x86_64
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] target_arch: x86_64
2018-04-23T03:13:11.425+0000 I CONTROL [initandlisten] options: { replication:

{ replSet: "rs0" }

, storage: { dbPath: "/data/mongo", mmapv1:

{ smallFiles: true }

} }
2018-04-23T03:13:11.446+0000 I - [initandlisten] Detected data files in /data/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-23T03:13:11.446+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-23T03:13:11.456+0000 E STORAGE [initandlisten] WiredTiger (-31802) [1524453191:456252][30872:0x7f9acb95dcc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2018-04-23T03:13:11.456+0000 E STORAGE [initandlisten] WiredTiger (0) [1524453191:456307][30872:0x7f9acb95dcc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2018-04-23T03:13:11.456+0000 E STORAGE [initandlisten] WiredTiger (0) [1524453191:456320][30872:0x7f9acb95dcc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2018-04-23T03:13:11.456+0000 E STORAGE [initandlisten] WiredTiger (0) [1524453191:456335][30872:0x7f9acb95dcc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2018-04-23T03:13:11.456+0000 I - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error
2018-04-23T03:13:11.456+0000 I STORAGE [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2018-04-23T03:13:11.456+0000 I CONTROL [initandlisten] dbexit: rc: 100

We can not access the mongodb now, Would you please help us to fix this? Thank you very much.

Generated at Thu Feb 08 04:37:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.