[SERVER-18977] Initscript does not stop a running mongod daemon Created: 15/Jun/15  Updated: 25/Nov/15  Resolved: 25/Jun/15

Status: Closed
Project: Core Server
Component/s: Packaging
Affects Version/s: 3.0.3
Fix Version/s: 3.0.5, 3.1.5

Type: Bug Priority: Major - P3
Reporter: James Laska Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-21668 Mongo not stopping on 3.0.7 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

1. Start mongod

# service mongod start
# service mongod status
mongod (pid 22345) is running...

2. Remove the mongod pid file

rm -f /var/run/mongod/mongod.pid

3. Stop mongod

# service mongod stop

4. Observe mongod is still running

# service mongod status
mongod (pid 22345) is running...

Sprint: Build 5 06/26/16
Participants:

 Description   

I believe this problem may have been misdiagnosed. I'm seeing this issue on CentOS-6.5 with `mongodb-org-server-3.0.3-1.el6.x86_64`

The problem is that `service mongod stop` reports that mongo was successfully stopped.

# service mongod stop
Stopping mongod:                                           [  OK  ]

However, hours later `mongod` is still running:

# service mongod status
mongod (pid 22345) is running...

Repeated attempts to stop mongod are unsuccessful:

# service mongod stop
Stopping mongod:                                           [  OK  ]
# service mongod status
mongod (pid 22345) is running...

The problem is that the pidfile has been removed, but mongod is still running. The current initscript uses a shell function mongo_killproc which relies on the function pidofproc to determine the pid of a running mongo. This function does not work when the pid file has been removed, but mongo is still running.

# service mongod status
mongod (pid 22345) is running...
 
# cat /var/run/mongodb/mongod.pid
cat: /var/run/mongodb/mongod.pid: No such file or directory
 
# pidof mongod ; echo $?
22345
0
 
# pidofproc -p /var/run/mongod/mongod.pid mongod ; echo $?
3

The solution (for CentOS-6.5) appears to be using pidof rather than pidofproc in mongo_killproc



 Comments   
Comment by Sam Kleinman (Inactive) [ 09/Jul/15 ]

This will be included in the 3.0.5 release.

Comment by James Laska [ 06/Jul/15 ]

samk Thanks! How do I know when this change will be released?

Comment by Sam Kleinman (Inactive) [ 25/Jun/15 ]

Thanks for the catch. I've committed a fix that should only remove the file if the process is not currently active.

Comment by Githook User [ 25/Jun/15 ]

Author:

{u'username': u'tychoish', u'name': u'Sam Kleinman', u'email': u'samk@10gen.com'}

Message: SERVER-18977: remove pidfiles only when process does not exist
Branch: master
https://github.com/mongodb/mongo/commit/a89a8c59dc663c6abcc96431bf4aa53343441e6a

Comment by Githook User [ 25/Jun/15 ]

Author:

{u'username': u'tychoish', u'name': u'Sam Kleinman', u'email': u'samk@10gen.com'}

Message: SERVER-18977: remove pidfiles only when process does not exist
Branch: v3.0
https://github.com/mongodb/mongo/commit/60f63b37466e1e03a61e878eb5d8fa4c8b496adf

Comment by James Laska [ 22/Jun/15 ]

> Sorry for the delay in getting back to you. I'm curious to know if you've run into this issue in a situation where you did not remove the pidfile manually?

Yes, I reduced the problem to this issue with the pid file. The problem manifested by issuing the ` service mongod stop/start` requests in rapid succession. I eventually found that the service remained online after issuing a stop. After review, the problem appears to be caused by the pidfile being removed, while mongod is still running.

Comment by Sam Kleinman (Inactive) [ 22/Jun/15 ]

Sorry for the delay in getting back to you. I'm curious to know if you've run into this issue in a situation where you did not remove the pidfile manually?

Regards,
sam

Generated at Thu Feb 08 03:49:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.