[SERVER-38137] WT_ERROR non-specific WiredTiger error Created: 14/Nov/18  Updated: 21/Nov/18  Resolved: 21/Nov/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: A R Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.tar     File WiredTiger.turtle     File WiredTiger.wt    
Operating System: ALL
Participants:

 Description   

After an unclean shutdown of our swarm and our docker container running mongo:3.2 tag, mongod fails to start. I have tried a --repair with no success.

I've attached my WiredTiger.wt and WiredTiger.turtle files for your review as I have seen other do here (for example): https://jira.mongodb.org/browse/SERVER-34607

Is there a way for me to fix this? Or is there someone on the Mongo team that can fix this?

Many Thanks!



 Comments   
Comment by A R [ 21/Nov/18 ]

Thanks for the heads up.  Will do!

Comment by Danny Hatcher (Inactive) [ 21/Nov/18 ]

I'm glad to hear it! One last thing to note is that MongoDB 3.2 has reached end-of-life. You should upgrade to 3.4, 3.6, or 4.0 when possible.

Have a great day,

Danny

Comment by A R [ 20/Nov/18 ]

Thank you Danny!  The repair worked. 

 

We have a good back up plan implemented now.   This was mistakenly omitted during the stand up of Rocket Chat.

 

You saved us.  Thanks again.

Comment by Danny Hatcher (Inactive) [ 19/Nov/18 ]

Hello,

I've attached a repair attempt (WiredTiger.tar) of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node so that we can further investigate.

Thank you,
Danny

Comment by A R [ 15/Nov/18 ]

Please let me know if this is insufficient.  Thanks.

 

Rocketchat_mongodb logs
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=bc106c82f975
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] db version v3.2.21
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] git version: 1ab1010737145ba3761318508ff65ba74dfe8155
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] allocator: tcmalloc
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] modules: none
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] build environment:
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] distmod: debian81
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] distarch: x86_64
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] target_arch: x86_64
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.572+0000 I CONTROL [initandlisten] options: { replication:

{ oplogSizeMB: 128, replSet: "rs0" }

, storage: { mmapv1:

{ smallFiles: true }

} }
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.597+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.604+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.612+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.618+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=17G,session_max=20000,eviction=(threads_min=4,threads_max=4,config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.678+0000 E STORAGE [initandlisten] WiredTiger (-31802) [1541178164:678851][1:0x7f2c8352acc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.678+0000 E STORAGE [initandlisten] WiredTiger (0) [1541178164:678916][1:0x7f2c8352acc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.678+0000 E STORAGE [initandlisten] WiredTiger (0) [1541178164:678932][1:0x7f2c8352acc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.678+0000 E STORAGE [initandlisten] WiredTiger (0) [1541178164:678946][1:0x7f2c8352acc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.688+0000 I - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.689+0000 I STORAGE [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
rocketchat_mongo.1.hash@comp.dom.com | 2018-11-02T17:02:44.689+0000 I CONTROL [initandlisten] dbexit: rc: 100

Rocketchat_rocketchat Logs
rocketchat_rocketchat.1.hash@comp.dom.com |
rocketchat_rocketchat.1.hash@comp.dom.com | /app/bundle/programs/server/node_modules/fibers/future.js:313
rocketchat_rocketchat.1.hash@comp.dom.com | throw(ex);
rocketchat_rocketchat.1.hash@comp.dom.com | ^
rocketchat_rocketchat.1.hash@comp.dom.com | MongoError: failed to connect to server [mongo:27017] on first connect [MongoError: getaddrinfo ENOTFOUND mongo mongo:27017]
rocketchat_rocketchat.1.hash@comp.dom.com | at Pool.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/server.js:336:35)
rocketchat_rocketchat.1.hash@comp.dom.com | at emitOne (events.js:116:13)
rocketchat_rocketchat.1.hash@comp.dom.com | at Pool.emit (events.js:211:7)
rocketchat_rocketchat.1.hash@comp.dom.com | at Connection.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:280:12)
rocketchat_rocketchat.1.hash@comp.dom.com | at Object.onceWrapper (events.js:317:30)
rocketchat_rocketchat.1.hash@comp.dom.com | at emitTwo (events.js:126:13)
rocketchat_rocketchat.1.hash@comp.dom.com | at Connection.emit (events.js:214:7)
rocketchat_rocketchat.1.hash@comp.dom.com | at Socket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:189:49)
rocketchat_rocketchat.1.hash@comp.dom.com | at Object.onceWrapper (events.js:315:30)
rocketchat_rocketchat.1.hash@comp.dom.com | at emitOne (events.js:116:13)
rocketchat_rocketchat.1.hash@comp.dom.com | at Socket.emit (events.js:211:7)
rocketchat_rocketchat.1.hash@comp.dom.com | at emitErrorNT (internal/streams/destroy.js:64:8)
rocketchat_rocketchat.1.hash@comp.dom.com | at _combinedTickCallback (internal/process/next_tick.js:138:11)
rocketchat_rocketchat.1.hash@comp.dom.com | at process._tickCallback (internal/process/next_tick.js:180:9)

Comment by Danny Hatcher (Inactive) [ 15/Nov/18 ]

Hello,

Please provide the error that you are experiencing when you attempt to start the process.

Thank you,

Danny

Generated at Thu Feb 08 04:48:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.