[SERVER-23528] Primary restarts itself unexpectedly Created: 05/Apr/16  Updated: 07/Apr/16  Resolved: 07/Apr/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.6.4
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Kroll Ontrack Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive logs.zip    
Operating System: ALL
Participants:

 Description   

Hi,

We have a three node mongodb cluster (with one arbiter) where every couple of days, mongod service on the primary restarts itself without any apparent reason. I have the following entry in the logs:

2016-04-04T03:34:50.130-0500 [signalProcessingThread] now exiting
2016-04-04T03:34:50.130-0500 [signalProcessingThread] dbexit:
2016-04-04T03:34:50.130-0500 [signalProcessingThread] shutdown: going to close listening sockets...
2016-04-04T03:34:50.130-0500 [signalProcessingThread] closing listening socket: 10
2016-04-04T03:34:50.130-0500 [signalProcessingThread] closing listening socket: 11
2016-04-04T03:34:50.130-0500 [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2016-04-04T03:34:50.137-0500 [signalProcessingThread] shutdown: going to flush diaglog...
2016-04-04T03:34:50.137-0500 [signalProcessingThread] shutdown: going to close sockets...
2016-04-04T03:34:50.137-0500 [signalProcessingThread] shutdown: waiting for fs preallocator...
2016-04-04T03:34:50.137-0500 [signalProcessingThread] shutdown: lock for final commit...
2016-04-04T03:34:50.137-0500 [signalProcessingThread] shutdown: final commit...

There is nothing before the shutdown signal in the logs (just normal operation). Nothing weird in SAR logs (processor, memory, disk usage normal), no log messages in /var/log/messages. It just looks like as if somebody restarted it manually but in fact nobody did, there is no indication of that in command history.

Mongodb version: 2.6.4-1
OS: RHEL 6.5

Any idea why this is happening?



 Comments   
Comment by Kelsey Schubert [ 07/Apr/16 ]

Hi koitautomation@altegrity.com

The logs you have uploaded indicate the mongod received a got signal 15 (Terminated) and as a result the mongod is exiting normally. As you note, this signal is most commonly received when someone runs kill <pid-of-mongod> as described in Stop mongoD Processes->Use kill. Please investigate what is sending this signal to the mongod.

Since this issue is not indicative of a bug in MongoDB server, I will be closing this ticket. For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Kroll Ontrack [ 07/Apr/16 ]

I've emailed the password for the zip file to you.

Comment by Kroll Ontrack [ 07/Apr/16 ]

Hi Thomas,

1. Attaching the logs.
2. Yes, occurs everyday between 03:00 - 04:00
3. Yes but we need a new release of our application and this is a work in progress.

Comment by Kelsey Schubert [ 05/Apr/16 ]

Hi koitautomation@altegrity.com,

Thank you for reporting this issue. So we can continue our investigation, please answer the following questions:

  1. Can you please upload the logs for the hour preceding the shutdown?
  2. Does this issue occur only at specific times?
  3. Have you considered upgrading to the latest revision of Mongo 2.6?

Kind regards,
Thomas

Generated at Thu Feb 08 04:03:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.