[SERVER-12577] mongod hangs on shutdown Created: 03/Feb/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.4.9
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Yev Assignee: Thomas Rueckstiess
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final

Linux db-server 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Mongo 2.4.9 standalone host with user authentication.


Issue Links:
Duplicate
duplicates SERVER-1423 reads often aren't possible while in ... Closed
Operating System: Linux
Steps To Reproduce:

Not sure what caused this. Happened after few hours of starting the service.

Participants:

 Description   

After executing $sudo service mongod stop the stop process got stuck.

The last line in mongod.log says:
Mon Feb 3 14:59:33.237 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends

The process is not responding to kill command (not kill -9).
Mongo client tool is able to connect but is not responding to any input. The connection is logged in the log file.

$mongod -f /etc/mongod.conf --shutdown gets stuck too and nothing good is happening.

netstat shows that it's listening on tcp/27017 and tcp/28017.

Eventually executed $kill -9 and started the service with no problem.
Stopped it again cleanly.



 Comments   
Comment by Thomas Rueckstiess [ 15/Apr/14 ]

I'm marking this a duplicate of SERVER-1423, and I added a comment about how signal handling is affected on a fsyncLocked mongod with auth, preventing kill -15.

Comment by Yev [ 20/Mar/14 ]

Hi Stephen,

It does seem to be caused by fSyncLock. The lock causes Mongo to stop responding to any command.
I removed the fSyncLock from the backup script and it works now and the server doesn't hang.

Might also be related to SERVER-1423.

Comment by Stennie Steneker (Inactive) [ 17/Mar/14 ]

Hi Yev,

Are you still experiencing this issue? If so, would you be able to attach a copy of your backup script?

If you are having issues with fSyncLock/fSyncUnlock, this may be related to SERVER-6302.

Regards,
Stephen

Comment by Yev [ 04/Feb/14 ]

Seems to be happening because of a lock.
I'm testing a new backup script that locks and unlocks the database.

It seems that the unlock operation gets stuck and prevents Mongo from shutting down.
The server has no traffic or connections, so it can't be because of load. And even after few hours, the unlock operation doesn't complete.

Generated at Thu Feb 08 03:28:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.