[SERVER-24343] When PC reboot, mongodb is not shutdown cleanly, then mongodb failed to recover. Using WT storage engine Created: 01/Jun/16  Updated: 14/Aug/18  Resolved: 02/Jun/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: tianlei.shi Assignee: Unassigned
Resolution: Done Votes: 0
Labels: envns, rpo, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

db version v3.2.0
git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
distmod: rhel70
distarch: x86_64
target_arch: x86_64


Attachments: Text File mongodb.log    
Operating System: Linux
Participants:

 Description   

Mongodb works fine at first.
Then PC reboots without shutting down the mongodb cleanly.
Then we try to start the mongod service again by the same command, it fails and return "dbexit: rc: 100".
Also we're not able to repair the mongodb database.
We're using WiredTiger storage engine.
Attachment is the mongodb log containing the error messages.

2016-06-01T16:55:23.838+0800 I CONTROL  [main] ***** SERVER RESTARTED *****
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] MongoDB starting : pid=2493 port=27017 dbpath=/home/mongodb/data/db 64-bit host=localhost.localdomain
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] db version v3.2.0
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] modules: none
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] build environment:
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten]     distmod: rhel70
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-06-01T16:55:23.948+0800 I CONTROL  [initandlisten] options: { net: { port: 27017 }, processManagement: { fork: true }, storage: { dbPath: "/home/mongodb/data/db" }, systemLog: { destination: "file", logAppend: true, path: "/home/mongodb/log/mongodb.log" } }
2016-06-01T16:55:24.029+0800 I -        [initandlisten] Detected data files in /home/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-06-01T16:55:24.029+0800 W -        [initandlisten] Detected unclean shutdown - /home/mongodb/data/db/mongod.lock is not empty.
2016-06-01T16:55:24.029+0800 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-06-01T16:55:24.029+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=37G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-06-01T16:55:24.163+0800 E STORAGE  [initandlisten] WiredTiger (-31802) [1464771324:163295][2493:0x7f5b28760dc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2016-06-01T16:55:24.163+0800 E STORAGE  [initandlisten] WiredTiger (0) [1464771324:163393][2493:0x7f5b28760dc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2016-06-01T16:55:24.163+0800 E STORAGE  [initandlisten] WiredTiger (0) [1464771324:163408][2493:0x7f5b28760dc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2016-06-01T16:55:24.163+0800 E STORAGE  [initandlisten] WiredTiger (0) [1464771324:163421][2493:0x7f5b28760dc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2016-06-01T16:55:24.164+0800 I -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error
2016-06-01T16:55:24.277+0800 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2016-06-01T16:55:24.277+0800 I CONTROL  [initandlisten] dbexit:  rc: 100



 Comments   
Comment by Kelsey Schubert [ 02/Jun/16 ]

Additionally, our recommendations for stopping MongoDB can be found here. To ensure a clean shutdown, please follow one of the methods listed.

Thanks,
Thomas

Comment by Ramon Fernandez Marina [ 02/Jun/16 ]

The underlying storage layer, from the filesystem to the physical devices, need to offer certain durability guarantees for MongoDB to work properly. For more details please see the MongoDB Production Notes.

Regards,
Ramón.

Comment by tianlei.shi [ 02/Jun/16 ]

I'll use a backup then.

According to your comments, database will be probably corrupted in the event of a power loss for XFS.
What about the manual reboot?

Thanks,

Comment by Ramon Fernandez Marina [ 01/Jun/16 ]

litmus, you may try to run mongod --repair, but please note the following:

  • I'd strongly suggest you make a backup copy of your corrupt files before you try any operations
  • While I can't make a general statement, in my experience XFS may not behave well in the event of a power loss and using a UPS is highly recommended – specially if write barriers are disabled. Is this your case?
  • If you upload your WiredTiger.wt and WiredTiger.turtle files we can do a repair attempt, but the signs point to unrecoverable corruption so you should consider recovering from backups and using a replica set

That being said, please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience.

I'll keep the ticket open a bit longer if you want to explore the repair attempt but, in the future, a question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Thanks,
Ramón.

Comment by tianlei.shi [ 01/Jun/16 ]

Our file system is XFS, we're using RAID-5.

Comment by tianlei.shi [ 01/Jun/16 ]

We don't have a replica.
The data means a lot to us, is there a way to recover the database? Thanks.

Generated at Thu Feb 08 04:06:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.