[SERVER-14031] Unexpected Shutdown in Replica Set Created: 22/May/14  Updated: 10/Dec/14  Resolved: 22/May/14

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: morteza irvani Assignee: Asya Kamsky
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo.log.2014-05-22T16-20-16    
Participants:

 Description   

Hi

There is a replica set with 3 member (primary, secondary, arbiter ). in 2 month ago, this is the third time secondary shut down unexpectedly

all three times [signalProcessingThread] caused shut down, i dont know what this means!!!!

Server sources are very high, and i am not concern about
I attachment log file. may someone could find the answer



 Comments   
Comment by Eric Milkie [ 23/May/14 ]

Bash resends a SIGHUP to all jobs if it receives a SIGHUP itself; this would include mongod.
Also, if you happen to have huponexit turned on, every job will receive a SIGHUP when bash exits.
https://www.gnu.org/software/bash/manual/html_node/Signals.html

Comment by morteza irvani [ 23/May/14 ]

Thanks Asya
you're right. it not using with fork (now, i enabled it). but for months, i using linux command to run mongod as background:

/mnt/mongodb/bin/mongod --config /mnt/mongodb/conf/mongo.conf &> /mnt/mongodb/log/linuxshell.log &

it seems works.

you said someone send signal to stop process!
1. If someone use db.shutdownServer(), It create log lines that related to its connection number.

Wed May 21 14:44:33.930 [conn1093] terminating, shutdown command received
Wed May 21 14:44:33.930 dbexit: shutdown called
Wed May 21 14:44:33.930 [conn1093] shutdown: going to close listening sockets...

so its not cause.

2. if someone use linux command "kill processId", It create such log lines

[signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends

so, it may be the cause, but our NetWork Environment is very private and no one can access to linux shell and call kill command.

3.finally, remain one cause: linux automatically close the daemon!!!!!
If its cause. The important questions is WHY? is it ordinary?

Comment by Asya Kamsky [ 22/May/14 ]

From your log file, I see that you are not starting your mongod process with the --fork option.

options: { auth: "true", config: "/mnt/mongodb/conf/mongo.conf", dbpath: "/mnt/mongodb/dbdata/db/", keyFile: "/mnt/mongodb/key", logpath: "/mnt/mongodb/log/mongo.log", oplogSize: 51200, replSet: "rs0" }

Since it's not forked, it's likely still attached to the tty of the shell that it was started in. This can cause it to be terminated when you log out.

Starting it with fork option would resolve this issue.

Comment by Asya Kamsky [ 22/May/14 ]

 [signalProcessingThread] got signal 1 (Hangup), will terminate after current cmd ends 

Usually this means that someone sent a signal to the process to stop it.

Do you have any automation that shuts down processes that match some set of rules or thresholds?

Generated at Thu Feb 08 03:33:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.