[SERVER-29563] mongod server fails to start and hangs Created: 12/Jun/17 Updated: 20/Sep/17 Resolved: 28/Aug/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Abhishek Vaid | Assignee: | Mark Agarunov |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Few days back my mongod crashed. It's a single node machine storing about 1 TB of data. The machine is ubuntu 16.04 and mongodb version is 3.4.2. The directory has all files intact. When I try to start mongod again, I see these log entries:
|
| Comments |
| Comment by Kelsey Schubert [ 28/Aug/17 ] | ||||||||||||||||||||
|
Hi vaidabhishek, We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket. Regards, | ||||||||||||||||||||
| Comment by Mark Agarunov [ 15/Jun/17 ] | ||||||||||||||||||||
|
Hello vaidabhishek, Looking at the output you've provided, it does appear that this hang is due to recovery running:
Has the workaround by disabling time synchroniztion or leaving mongod running to finish the recovery had any success? Thanks, | ||||||||||||||||||||
| Comment by Abhishek Vaid [ 15/Jun/17 ] | ||||||||||||||||||||
|
Any updates on this ? | ||||||||||||||||||||
| Comment by Abhishek Vaid [ 13/Jun/17 ] | ||||||||||||||||||||
|
Mark, I upgraded mongo to 3.4.4 and tried to bring up mongod again. Here is how logs are looking. I'm running it in a screen session with log output coming on stdout. I've copied this output after 10 minutes of server restart. As you can see, nothing seems to be happening. *The first line is the command I ran. *
| ||||||||||||||||||||
| Comment by Mark Agarunov [ 13/Jun/17 ] | ||||||||||||||||||||
|
Hello vaidabhishek, Thank you for providing these files. Looking over the diagnostic data and the description of the behavior you've given, the hang during startup may be due to the recovery process. It seems that a checkpoint hadn't completed in a long time before mongod shut down, and this is now causing the delay. Unfortunately the version of MongoDB you are using (3.4.2) does not log the recovery process, however newer versions, as of 3.4.3, do. My recommendation would be to upgrade if possible and attempt to start mongod with verbose logging enabled again and provide us with the logs that are generated. This should let us know exactly what is happening during the recovery. Additionally, the reason a checkpoint hadn't completed before shutdown may be due to a time sync issue if you are using Hyper-V. Until there is a fix for this in a newer release, you may be able to work around the issue by disabling time synchronization. Thanks, | ||||||||||||||||||||
| Comment by Abhishek Vaid [ 13/Jun/17 ] | ||||||||||||||||||||
|
These are the files you asked. I started my mongod with following command: sudo mongod --directoryperdb --dbpath /data-raid/mongo/data --port 28018 --logpath /data-raid/mongo/logs/mongodb.log -vvvvv | ||||||||||||||||||||
| Comment by Mark Agarunov [ 12/Jun/17 ] | ||||||||||||||||||||
|
Hello vaidabhishek, Thank you for the report. To get a better idea of what may be causing the behavior you're seeing, I'd like to request the following:
This should give us some more information to better investigate the issue. Thanks, |