[SERVER-2833] mongod.exe does not shut down correctly as a Windows service Created: 25/Mar/11 Updated: 12/Jul/16 Resolved: 09/Jun/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Usability |
| Affects Version/s: | 1.8.0 |
| Fix Version/s: | 2.0.7, 2.1.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Daniel Weber | Assignee: | Tad Marshall |
| Resolution: | Done | Votes: | 5 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Windows 7 64-bit |
||
| Issue Links: |
|
||||||||||||||||||||
| Operating System: | Windows | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Rewritten description: When a MongoDB server is stopped by the Windows Service Control Manager (with "net stop mongodb" or during system shutdown or restart), mongo wasn't notifying the Service Control Manager of its steps to shutdown and was ignoring a direct shutdown notification, leading to an unclean shutdown. Original description: I've created a service on my pc to start mongod on every start-up of the system. We figured out that the problem is that mongod didn't start properly. When I have a look at the service list the status is always "starting" and didn't change. The service isn't able to start up properly. When I go to the data folder where all my db data are stored and remove the mongod.lock file. The service is able to start and I can connect to mongoDB. When I reboot my pc the problem is going to occur again. Removing the mongod.lock file resolves the problem. Therefore I guess there is a problem that the service didn't shutdown and remove the mongod.lock file when you shutdown your pc. We were able to reproduce the problem on different computers. |
| Comments |
| Comment by Tad Marshall [ 10/Jun/12 ] | |
|
Backported to 2.0 branch for 2.0.7. | |
| Comment by auto [ 10/Jun/12 ] | |
|
Author: {u'login': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}Message: Added code to log shutdown events from the Windows Service Controller | |
| Comment by Andy Schwerin [ 09/Jun/12 ] | |
|
Tad to backport for 2.0.7. | |
| Comment by Tad Marshall [ 08/Jun/12 ] | |
|
Changed title and description to better describe the issue, rather than the symptoms. | |
| Comment by auto [ 02/Dec/11 ] | |
|
Author: {u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}Message: Added code to log shutdown events from the Windows Service Controller | |
| Comment by Tad Marshall [ 02/Dec/11 ] | |
|
After reworking the code, my best guess for what is happening is that on the customers' machines where this problem is seen, there are services running that take a little while to shut down. This introduces a race condition in which while Windows is waiting for the Service Controller to respond to the shutdown message, it starts to kill individual processes itself. Because we were ignoring the direct shutdown signal, we just get killed and don't shutdown cleanly. I (hopefully) fixed this by adding code to respond correctly to both the callback from the Service Controller and also to the direct message from the OS. I also added code to log the events from the Service Controller and do the shutdown on a named thread, so it should be clear from the log whether we got the message and how we responded to it. | |
| Comment by Tad Marshall [ 01/Dec/11 ] | |
|
Like Robert Stam, I am unable to reproduce the "no clean shutdown of Windows service", but I don't like the messages that are generated when we shut down in this mode.
None of this covers the fact that I can't reproduce the reported problem. But I can fix the problems that I can reproduce and we can see if that makes the original problem go away. | |
| Comment by Tad Marshall [ 01/Dec/11 ] | |
|
@Mayur Ahir – please create a separate Jira ticket for your issues on Ubuntu. This Jira ticket is specific to problems with the Windows version when installed as a Windows Service. Thank you! | |
| Comment by Mayur Ahir [ 06/Sep/11 ] | |
|
I am using v1.8.3 with Ubuntu 11.04 and still getting same error after reboot. $ mongo | |
| Comment by Michael Catanzariti [ 21/Jun/11 ] | |
|
net start mongodb === BEGINING OF FILE Tue Jun 21 09:48:09 [initandlisten] MongoDB starting : pid=6088 port=27017 dbpath=D:\mongoDB\data 64-bit === END OF FILE | |
| Comment by Michael Catanzariti [ 21/Jun/11 ] | |
|
net stop mongodb === BEGINING OF FILE Tue Jun 21 09:47:14 shutdown: going to close listening sockets... === END OF FILE | |
| Comment by Michael Catanzariti [ 21/Jun/11 ] | |
|
There's nothing noticeable at the time of the shutdown... | |
| Comment by Robert Stam [ 20/Jun/11 ] | |
|
It's a long shot, but is there anything in the Windows event viewer at the time of the shutdown? | |
| Comment by Robert Stam [ 20/Jun/11 ] | |
|
Does "net stop mongodb" from the command prompt work? What about "net start mongodb" after that? Can you post the log file for that to compare against? | |
| Comment by Michael Catanzariti [ 20/Jun/11 ] | |
|
Hi Robert, Yes, all the machine runs Windows 7 Enterprise with MongoDB 1.8.2 | |
| Comment by Robert Stam [ 20/Jun/11 ] | |
|
I realize you have already given me your log file. I was just adding my log file for comparison. I tried to follow your steps as closely as possible and still was unable to reproduce. The only thing I see that is different is that you are using Windows 7 Enterprise and I am using Windows 7 Ultimate. Also, this time I tested using MongoDB 1.8.2, but last time I was unable to reproduce I was using MongoDB 1.8.1, so I don't think anything changed between these two versions. Are all the machines you are able to reproduce this on running Windows 7 Enterprise? | |
| Comment by Michael Catanzariti [ 20/Jun/11 ] | |
|
Hi Robert, I just installed version 1.8.2 and still reproduce the problem... === BEGINING OF FILE Mon Jun 20 18:08:04 CTRL_SHUTDOWN_EVENT signal (ignored) === END OF FILE | |
| Comment by Michael Catanzariti [ 20/Jun/11 ] | |
|
Robert, I already gave you my config and the mongodb log about this problem. How can I help you to reproduce the problem ? | |
| Comment by Robert Stam [ 20/Jun/11 ] | |
|
I am unable to reproduce this. Just tried again with the latest 1.8.2 release. Here's the content of my log file after 1. installing MongoDB as a service The last few lines of the log file show MongoDB starting up cleanly after the reboot. =====log file===== C:\data\db2> | |
| Comment by Michael Catanzariti [ 20/Jun/11 ] | |
|
Hi Robert, Is there any progress about this bug ? | |
| Comment by Robert Stam [ 10/May/11 ] | |
|
Thanks for the log. That's definitely different than my log. After CTRL_SHUTDOWN_EVENT I have another 20 or lines culminating with a clean exit. We'll have to figure out what's different about your environment than mine. One difference is that I'm using Windows 7 Ultimate instead of Windows 7 Enterprise, but that shouldn't make any difference. | |
| Comment by Michael Catanzariti [ 09/May/11 ] | |
|
I just installed a fresh version of mongo with no data, started it and shutowned windows 7. So there is no load & no data at all. This is my hardware config : This is the "Path to executable" windows service setting : "C:\Program Files\MongoDB\bin\mongod" --dbpath "D:\mongoDB\data" --logpath "D:\mongoDB\logs\log.txt" --logappend --service This is the content of my log.txt file === BEGINING OF FILE | |
| Comment by Robert Stam [ 09/May/11 ] | |
|
Reopening because user reports they can reliably reproduce the issue. Am waiting for log files. | |
| Comment by Robert Stam [ 09/May/11 ] | |
|
Something must be different on our systems then. Do you have any log files we could look at? One problem with the infinite restarts when booting is that it creates very large log files, so the first step I would recommend is manually configuring the MongoDB service to only restart on failure the first or second time. And then perhaps start with a new empty log file and reproduce. Windows only allows a process about 30 seconds to shutdown, so it's possible that something is preventing mongod from shutting down within 30 seconds and is then being killed by Windows for taking too long. Is your mongod under heavy load when you reboot the system? Are there any long running operations that take more than 30 seconds? | |
| Comment by Michael Catanzariti [ 09/May/11 ] | |
|
Sorry, but on several workstations of my company, we use mongodb 1.8.1 on Windows 7 Enterprise 64 bit. | |
| Comment by Robert Stam [ 09/May/11 ] | |
|
Works as designed. See previous comment. | |
| Comment by Robert Stam [ 09/May/11 ] | |
|
I just retested this using MongoDB 1.8.1 on a Windows 7 64-bit system, and it worked as designed. Just to clarify, this JIRA ticket had to do with MongoDB not shutting down cleanly when Windows was shutdown. The result of a Windows shutdown prior to this fix was that a mongod.lock file was left behind. The presence of a mongod.lock file is evidence that mongod did not shut down cleanly and that a repair should be attempted before restarting mongod. I mongod crashes for any other reason it will also leave a mongod.lock file behind, but that is unrelated to this JIRA ticket. Deleting a mongod.lock file without running repair is tempting fate a bit. I have created a new JIRA ticket regarding Windows service startup options related to mongod failing to start and how many attempts Windows will automatically make to restart the service (as opposed to this JIRA ticket which is related to a clean shutdown). See: | |
| Comment by Frank Henard [ 09/May/11 ] | |
|
Likewise, as you see in my comment above. I am using 1.8.1 with Ubuntu Server 11.04, and mongo is not stopping at shutdown. | |
| Comment by Michael Catanzariti [ 09/May/11 ] | |
|
The fix does not seem to work on Windows 7, with the 1.8.1 release. | |
| Comment by Robert Stam [ 05/May/11 ] | |
|
The fix for this didn't make in to the 1.8.0 release. It is in the 1.8.1 release. | |
| Comment by Robert Stam [ 05/May/11 ] | |
|
Fixed in 1.8.1 (not 1.8.0). | |
| Comment by Frank Henard [ 05/May/11 ] | |
|
I'm also having this problem - Ubuntu Server 11.04, Mongo 1.8.1 EDIT 6/20/11 - It turns out that I was not shutting down the server correctly. Improperly shutting down the server did not impact apache, but was a problem for mongo because it would not start on the server restart. |