[SERVER-25525] the machine reboot so that mongodb can't gain the correct data and the reason is about WiredTiger.wt Created: 10/Aug/16  Updated: 23/Aug/16  Resolved: 23/Aug/16

Status: Closed
Project: Core Server
Component/s: Admin, WiredTiger
Affects Version/s: 3.0.4
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: ao.zhang Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

centos 6.4


Attachments: File WiredTiger.tar.gz     File WiredTiger.turtle     File WiredTiger.wt     File mongod.conf     Text File mongod.log     Text File new_mongod.log     File repair_attempt-2.tar.gz     File repair_attempt.tar    
Operating System: Linux
Participants:

 Description   

hello:
my machine system is centos6.4,after an unexpected reboot my mongodb can't be start correctly.and the data is 285G,I am repair two days and detect it can't start correctly may because of WiredTiger.wt was damaged.the following is my manage process!

[work@bjs-ihotel-ichartmongodb-01 mongodb]$ ./bin/mongod -f ./conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 26446
ERROR: child process failed, exited with error number 14
[work@bjs-ihotel-ichartmongodb-01 data]$ rm -fr /home/work/ichartmongodb/mongodb/data/mongod.lock 
[work@bjs-ihotel-ichartmongodb-01 data]$ rm -fr /home/work/ichartmongodb/mongodb/data/mongod/mongod.lock 
[work@bjs-ihotel-ichartmongodb-01 data]$ rm -fr /home/work/ichartmongodb/mongodb/data/mongod/WiredTiger.lock 
[work@bjs-ihotel-ichartmongodb-01 data]$ /home/work/ichartmongodb/mongodb/bin/mongod --dbpath=/home/work/ichartmongodb/mongodb/data --port=27017 --logpath=/home/work/ichartmongodb/mongodb/log/mongodb.log --repair --fork   
about to fork child process, waiting until server is ready for connections.
forked process: 27873
child process started successfully, parent exiting
 
[work@bjs-ihotel-ichartmongodb-01 data]$ /home/work/ichartmongodb/mongodb/bin/mongod -f /home/work/ichartmongodb/mongodb/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 28042
ERROR: child process failed, exited with error number 14
 
[work@bjs-ihotel-ichartmongodb-01 ~]$ mv /home/work/ichartmongodb/mongodb/data/mongod/WiredTiger* /home/work/lei.zhang/new/now
 
[work@bjs-ihotel-ichartmongodb-01 ~]$ du -sh /home/work/lei.zhang/new/now/WiredTiger.wt 
16M     /home/work/lei.zhang/new/now/WiredTiger.wt
 
[work@bjs-ihotel-ichartmongodb-01 ~]$ /home/work/ichartmongodb/mongodb/bin/mongod --dbpath=/home/work/ichartmongodb/mongodb/data --port=27017 --logpath=/home/work/ichartmongodb/mongodb/log/mongodb.log --repair --fork
about to fork child process, waiting until server is ready for connections.
forked process: 29078
child process started successfully, parent exiting
 
[work@bjs-ihotel-ichartmongodb-01 ~]$ /home/work/ichartmongodb/mongodb/bin/mongod -f /home/work/ichartmongodb/mongodb/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 29206
child process started successfully, parent exiting
 
[work@bjs-ihotel-ichartmongodb-01 ~]$ /home/work/ichartmongodb/mongodb/bin/mongod -f /home/work/ichartmongodb/mongodb/conf/arbiter.conf 
about to fork child process, waiting until server is ready for connections.
forked process: 30051
child process started successfully, parent exiting
 
[work@bjs-ihotel-ichartmongodb-01 ~]$ du -sh /home/work/ichartmongodb/mongodb/data/mongod/WiredTiger.wt 
76K     /home/work/ichartmongodb/mongodb/data/mongod/WiredTiger.wt

but my page show

{"errno":505,"timestamp":1470813472,"errmsg":"Failed to connect to: 192.168.110.221:27017: ismaster: got unknown node type"}

and i think this because WiredTiger.wt damaged because machine reboot,and i need to recover WiredTiger.wt. please help me ,i had try my best to repair it ,but all is useless。thank you,the attachment is the old WiredTiger.wt



 Comments   
Comment by Kelsey Schubert [ 23/Aug/16 ]

Hi ao.zhang,

I have attached repair_attempt-2.tar.gz for the backup files you uploaded. As before, please place these files in your backup's $dbpath and restart the mongod.

If this repair attempt does not resolve the issue, I would recommend posting on the mongodb-users group, and providing the latest logs after starting with --repair using the files I have provided. The mongodb-users group provides a forum for MongoDB-related support discussion, and users there may be able to suggest additional steps to recover data. In this situation, please understand that a complete recovery would be unlikely.

Kind regards,
Thomas

Comment by ao.zhang [ 22/Aug/16 ]

it's my backup machine's WiredTiger.wt ,and it's error the same as the master machine.

Comment by ao.zhang [ 22/Aug/16 ]

@Thomas Schubert :I'm sorry for not replying to your message,thanks your help.
i use your repair_attempt.tar,but it's not worked.The error is as follows

./bin/mongod -f ./conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 2552
ERROR: child process failed, exited with error number 14.

the attachment is the my backup machine's WiredTiger file,Can you fix this file for me?
My last year's data are kept in the mongodb,please help us, thanks !

Comment by Kelsey Schubert [ 18/Aug/16 ]

Hi ao.zhang,

I've attached repair_attempt.tar with the WiredTiger.wt and WiredTiger.turtle files after a repair attempt. Please extract these files in your dbpath and try restarting your mongod. If there is no other corruption, MongoDB should start successfully.

Kind regards,
Thomas

Comment by ao.zhang [ 11/Aug/16 ]

@Romon Fernandez:
Thanks very much!
1:the log is quite big(35G).last day the log which i upload is a little , and the new_mongod.log is the log i have selected from the 35G .
2:yes , i mv WiredTiger.wt to other place , and i restart the service(./bin/mongod/ --dbpath=... --logpath=... --port=27017 --fork), and it's start successly. and produce a new WiredTiger.wt WiredTiger.turtle in the dbpath. use the old WiredTiger.wt(16M) i can't start the service, so i think the Service exception is about WiredTiger.wt。
3:yes, we have a backup system,but the backup system also reboot, and it's also can't be start correctly! (sorry,my Colleague and i not familiar with mongodb,maybe Some mistakes were made in the process).
4.the attachment is the mongod.conf,and WiredTiger.turtle,the WiredTiger.wt(16 M i already uploaded!)
5: i want your guys help us repair WiredTiger.wt WiredTiger.turtle , thank you !
finally , your guys so nice!

Comment by Ramon Fernandez Marina [ 10/Aug/16 ]

ao.zhang, sorry you've run into this issue. From the list of steps you tried above and the logs you uploaded I'm not sure this is a recoverable situation, but I'd like to ask some more questions to be sure:

  • Are these all the logs you have? The logs you uploaded seem to be missing important pieces, like messages from the startup sequence or information about some of the steps you described above
  • The WiredTiger.wt file is an important one, and it should not be removed from the dbpath. Did you move/delete any other files in the dbpath
  • Do you have a backup of your data prior to the unexpected shutdown? By removing the WiredTiger.wt the repair operation may have further damaged your data. At this stage the easiest way forward is to restore this node from a healthy primary or from a data backup.
  • Can you please upload your /home/work/ichartmongodb/mongodb/conf/mongod.conf file?

If I'm not mistaken, the only thing we can try is to attempt a repair of your original WiredTiger.wt file (the 16MB one) and the WiredTiger.turtle file (the ones you moved to /home/work/lei.zhang/new/now). If you upload them here we can attempt a repair, but there's no guarantee that it will work.

Please let us know how you want to proceed.

Thanks,
Ramón.

Comment by ao.zhang [ 10/Aug/16 ]

Alexander Gorrod :thanks you ,谢谢您!the attachment is the log.

Comment by Alexander Gorrod [ 10/Aug/16 ]

ao.zhang Sorry you are having problems with your MongoDB database. We will need more information before we can help. The first thing we will need is the full mongod log file from the database.

Comment by ao.zhang [ 10/Aug/16 ]

help me,please me ,thank you !

Generated at Thu Feb 08 04:09:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.