[SERVER-1533] Replica set SECONDARY crash when terminating other PRIMARY and SECONDARY Created: 01/Aug/10  Updated: 03/Aug/10  Resolved: 03/Aug/10

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.5.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David Mytton Assignee: Dwight Merriman
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-1483 Shutdown / rs.stepDown() of master cr... Closed
Operating System: Linux
Participants:

 Description   

I am unable to reproduce this again and am not totally sure of the circumstances leading to it, but there may be some useful information in the backtrace.

I had 3 members of replica set. I terminate the primary and then immediately after one of the secondaries. The third secondary crashed on its own.

PRIMARY

mongodb-linux-x86_64-1.5.7/bin/mongod --shardsvr --dbpath data/ --replSet set1/domU-12-31-39-06-89-F1.compute-1.internal:27018 --rest

Sun Aug 1 05:16:33 [conn9] getmore local.oplog.rs cid:5805707532161602891 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3007ms
Sun Aug 1 05:16:45 [conn9] getmore local.oplog.rs cid:5805707532161602891 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3003ms
Sun Aug 1 05:16:46 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
Sun Aug 1 05:16:46 [interruptThread] now exiting
Sun Aug 1 05:16:46 dbexit:

Sun Aug 1 05:16:46 [interruptThread] shutdown: going to close listening sockets...
Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 9
Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 10
Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 11
Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 12
Sun Aug 1 05:16:46 [interruptThread] shutdown: going to flush oplog...
Sun Aug 1 05:16:46 [interruptThread] shutdown: going to close sockets...
Sun Aug 1 05:16:46 [interruptThread] shutdown: waiting for fs preallocator...
Sun Aug 1 05:16:46 [interruptThread] shutdown: closing all files...
Sun Aug 1 05:16:46 closeAllFiles() finished

Sun Aug 1 05:16:46 [interruptThread] shutdown: removing fs lock...
Sun Aug 1 05:16:46 [conn1] end connection 127.0.0.1:43135
Sun Aug 1 05:16:46 [conn4] end connection 10.208.46.160:43621
Sun Aug 1 05:16:46 dbexit: really exiting now

SECONDARY 1

mongodb-linux-x86_64-1.5.7/bin/mongod --shardsvr --dbpath data/ --replSet set1/domU-12-31-39-06-29-52.compute-1.internal:27018 --rest

Sun Aug 1 05:17:57 [conn5] getmore local.oplog.rs cid:5040576339790303823 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3007ms
Sun Aug 1 05:17:58 [conn6] getmore local.oplog.rs cid:2586884600444785463 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3007ms
Sun Aug 1 05:18:03 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
Sun Aug 1 05:18:03 [interruptThread] now exiting
Sun Aug 1 05:18:03 dbexit:

Sun Aug 1 05:18:03 [interruptThread] shutdown: going to close listening sockets...
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 10
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 11
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 12
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 13
Sun Aug 1 05:18:03 [interruptThread] shutdown: going to flush oplog...
Sun Aug 1 05:18:03 [interruptThread] shutdown: going to close sockets...
Sun Aug 1 05:18:03 [interruptThread] shutdown: waiting for fs preallocator...
Sun Aug 1 05:18:03 [interruptThread] shutdown: closing all files...
Sun Aug 1 05:18:03 [conn2] end connection 127.0.0.1:48610
Sun Aug 1 05:18:03 closeAllFiles() finished

Sun Aug 1 05:18:03 [interruptThread] shutdown: removing fs lock...
Sun Aug 1 05:18:03 dbexit: really exiting now

SECONDARY 2

mongodb-linux-x86_64-1.5.7/bin/mongod --shardsvr --dbpath data/ --replSet set1/domU-12-31-39-06-29-52.compute-1.internal:27018 --rest

Sun Aug 1 04:38:36 [rs_sync] replSet SECONDARY
Sun Aug 1 05:16:47 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
Sun Aug 1 05:16:47 [conn2] end connection 10.208.46.160:53683
Sun Aug 1 05:16:48 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now down (or slow to respond)
Sun Aug 1 05:16:48 [conn3] replSet info voting yea for 1
Sun Aug 1 05:16:49 [rs Manager] replSet TEMP not voting yea for 2
Sun Aug 1 05:16:49 [rs Manager] replSet TEMP voted for 1 1 secs ago
Sun Aug 1 05:16:49 [rs Manager] replSet not trying to elect self as responded yea to someone else recently
Sun Aug 1 05:16:56 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now up
Sun Aug 1 05:16:57 [initandlisten] connection accepted from 10.208.46.160:50868 #4
Sun Aug 1 05:16:58 [rs_sync] replSet SECONDARY
Sun Aug 1 05:18:03 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
Sun Aug 1 05:18:03 [conn3] end connection 10.208.142.3:45480
Sun Aug 1 05:18:03 [conn4] replSet info voting yea for 0
Sun Aug 1 05:18:04 [rs Manager] should never be called?
Sun Aug 1 05:18:04 [rs Manager] Assertion failure false db/repl/manager.cpp 55
0x5313c3 0x53db81 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb3) [0x5313c3]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x53db81]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]
Sun Aug 1 05:18:04 terminate() called, printing stack:

0x801ab0 0x2aaaaaf9ee76 0x2aaaaaf9eea3 0x2aaaaaf9ef8a 0x53ddfc 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo11myterminateEv+0x50) [0x801ab0]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ee76]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9eea3]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ef8a]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x38c) [0x53ddfc]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]
Sun Aug 1 05:18:04 Got signal: 6 (Aborted).

Sun Aug 1 05:18:04 Backtrace:
0x8025c9 0x2aaaab6a4f30 0x2aaaab6a4ec5 0x2aaaab6a6970 0x801b6f 0x2aaaaaf9ee76 0x2aaaaaf9eea3 0x2aaaaaf9ef8a 0x53ddfc 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8025c9]
/lib64/libc.so.6 [0x2aaaab6a4f30]
/lib64/libc.so.6(gsignal+0x35) [0x2aaaab6a4ec5]
/lib64/libc.so.6(abort+0x110) [0x2aaaab6a6970]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo11myterminateEv+0x10f) [0x801b6f]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ee76]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9eea3]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ef8a]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x38c) [0x53ddfc]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]

Sun Aug 1 05:18:04 dbexit:

Sun Aug 1 05:18:04 [rs Manager] shutdown: going to close listening sockets...
Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 10
Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 11
Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 12
Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 13
Sun Aug 1 05:18:04 [rs Manager] shutdown: going to flush oplog...
Sun Aug 1 05:18:04 [rs Manager] shutdown: going to close sockets...
Sun Aug 1 05:18:04 [rs Manager] shutdown: waiting for fs preallocator...
Sun Aug 1 05:18:04 [rs Manager] shutdown: closing all files...
Sun Aug 1 05:18:04 closeAllFiles() finished

Sun Aug 1 05:18:04 [rs Manager] shutdown: removing fs lock...
Sun Aug 1 05:18:04 dbexit: really exiting now



 Comments   
Comment by Eliot Horowitz (Inactive) [ 03/Aug/10 ]

SEE SERVER-1483

Comment by Dwight Merriman [ 03/Aug/10 ]

i think you are right

we are doing 1.5.8 today, so you might want to wait for that and try it.

Comment by David Mytton [ 03/Aug/10 ]

Is this the right nightly build? It seems to have failed

http://buildbot.mongodb.org/builders/Nightly%20Linux%2064-bit

Comment by Dwight Merriman [ 03/Aug/10 ]

dave can you try the latest code (newer than 1.5.7? maybe tomorrow's daily build?) the newer code at a minimum has better logging diagnostics on the issue.

Comment by David Mytton [ 03/Aug/10 ]

I experienced this again. Does look similar to SERVER-1483

Terminated PRIMARY:

Mon Aug 2 21:34:53 [conn9] getmore local.oplog.rs cid:1283402304456619496 getMore: { ts:

{ $gte: new Date(5500988726827810817) }

} bytes:20 nreturned:0 3013ms
Mon Aug 2 21:34:53 [conn12] getmore local.oplog.rs cid:8915049892226683696 getMore: { ts:

{ $gte: new Date(5500988726827810817) }

} bytes:20 nreturned:0 3013ms
Mon Aug 2 21:35:00 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
Mon Aug 2 21:35:00 [interruptThread] now exiting
Mon Aug 2 21:35:00 dbexit:

Mon Aug 2 21:35:00 [interruptThread] shutdown: going to close listening sockets...
Mon Aug 2 21:35:00 [interruptThread] going to close listening socket: 9
Mon Aug 2 21:35:00 [interruptThread] going to close listening socket: 10
Mon Aug 2 21:35:00 [interruptThread] going to close listening socket: 12
Mon Aug 2 21:35:00 [interruptThread] going to close listening socket: 13
Mon Aug 2 21:35:00 [interruptThread] shutdown: going to flush oplog...
Mon Aug 2 21:35:00 [interruptThread] shutdown: going to close sockets...
Mon Aug 2 21:35:00 [interruptThread] shutdown: waiting for fs preallocator...
Mon Aug 2 21:35:00 [interruptThread] shutdown: closing all files...
Mon Aug 2 21:35:00 closeAllFiles() finished

Mon Aug 2 21:35:00 [interruptThread] shutdown: removing fs lock...
Mon Aug 2 21:35:00 dbexit: really exiting now

SECONDARY 1 took over as PRIMARY

SECONDARY 2 crashed

Mon Aug 2 21:34:42 [initandlisten] connection accepted from 10.255.62.79:43801 #6
Mon Aug 2 21:35:00 [conn2] end connection 10.208.51.111:37013
Mon Aug 2 21:35:00 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
Mon Aug 2 21:35:01 [conn3] replSet info voting yea for 1
Mon Aug 2 21:35:02 [rs Manager] should never be called?
Mon Aug 2 21:35:02 [rs Manager] Assertion failure false db/repl/manager.cpp 55
Mon Aug 2 21:35:02 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-2C-81.compute-1.internal is now down (or slow to respond)
Mon Aug 2 21:35:02 Got signal: 11 (Segmentation fault).

0x5313c3 0x53db81 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb3) [0x5313c3]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x53db81]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]
Mon Aug 2 21:35:02 terminate() called, printing stack:

0x801ab0 0x2aaaaaf9ee76 0x2aaaaaf9eea3 0x2aaaaaf9ef8a 0x53ddfc 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo11myterminateEv+0x50) [0x801ab0]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ee76]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9eea3]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ef8a]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x38c) [0x53ddfc]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]
Mon Aug 2 21:35:02 Got signal: 6 (Aborted).

Mon Aug 2 21:35:02 Backtrace:
0x8025c9 0x2aaaab6a4f30 0x2aaaab6a4ec5 0x2aaaab6a6970 0x801b6f 0x2aaaaaf9ee76 0x2aaaaaf9eea3 0x2aaaaaf9ef8a 0x53ddfc 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8025c9]
/lib64/libc.so.6 [0x2aaaab6a4f30]
/lib64/libc.so.6(gsignal+0x35) [0x2aaaab6a4ec5]
/lib64/libc.so.6(abort+0x110) [0x2aaaab6a6970]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo11myterminateEv+0x10f) [0x801b6f]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ee76]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9eea3]
/usr/lib64/libstdc++.so.6 [0x2aaaaaf9ef8a]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x38c) [0x53ddfc]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]

Mon Aug 2 21:35:02 dbexit:

Mon Aug 2 21:35:02 [rs Manager] shutdown: going to close listening sockets...
Mon Aug 2 21:35:02 [rs Manager] going to close listening socket: 10
Mon Aug 2 21:35:02 [rs Manager] going to close listening socket: 11
Mon Aug 2 21:35:02 [rs Manager] going to close listening socket: 12
Mon Aug 2 21:35:02 [rs Manager] going to close listening socket: 13
Mon Aug 2 21:35:02 [rs Manager] shutdown: going to flush oplog...
Mon Aug 2 21:35:02 [rs Manager] shutdown: going to close sockets...
Mon Aug 2 21:35:02 [rs Manager] shutdown: waiting for fs preallocator...
Mon Aug 2 21:35:02 [rs Manager] shutdown: closing all files...
Mon Aug 2 21:35:02 Backtrace:
0x8025c9 0x2aaaab6a4f30 0x2aaaaaccf343 0x54350a 0x68d3d9 0x5433e3 0x5269e5 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8025c9]
/lib64/libc.so.6 [0x2aaaab6a4f30]
/lib64/libpthread.so.0(pthread_mutex_lock+0x13) [0x2aaaaaccf343]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo4task6Server4sendEN5boost8functionIFvvEEE+0x2a) [0x54350a]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo21ReplSetHealthPollTask6doWorkEv+0x1159) [0x68d3d9]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo4task4Task3runEv+0x33) [0x5433e3]
mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0x85) [0x5269e5]
mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
/lib64/libpthread.so.0 [0x2aaaaaccd407]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]

Mon Aug 2 21:35:02 dbexit: ; exiting immediately

Mon Aug 2 21:35:02 [conn1] end connection 127.0.0.1:47262
mongod: /opt/extra/include/boost/thread/pthread/mutex.hpp:50: void boost::mutex::lock(): Assertion `!pthread_mutex_lock(&m)' failed.
Mon Aug 2 21:35:02 Got signal: 6 (Aborted).

Mon Aug 2 21:35:02 [initandlisten] now exiting

SECONDARY 1 (now PRIMARY) unable to continue as all other members of the replica set are down.

SECONDARY 2 details (the one that crashed)

mongodb-linux-x86_64-1.5.7/bin/mongod --rest --shardsvr --replSet set1/domU-12-31-39-06-2C-81.compute-1.internal --dbpath data/ --port 27017
Mon Aug 2 21:25:04 MongoDB starting : pid=3956 port=27017 dbpath=data/ 64-bit

    • NOTE: This is a development version (1.5.7) of MongoDB.
    • Not recommended for production.

Mon Aug 2 21:25:04 db version v1.5.7, pdfile version 4.5
Mon Aug 2 21:25:04 git version: 5b667e49b1c88f201cdd3912b3d1d1c1098a25b4
Mon Aug 2 21:25:04 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41

Comment by Alex [ 02/Aug/10 ]

This looks like it's connected to issue http://jira.mongodb.org/browse/SERVER-1483

Comment by David Mytton [ 02/Aug/10 ]

Yes, 1.5.7.

Comment by Dwight Merriman [ 02/Aug/10 ]

will investigate.

this was with 1.5.7?

Comment by David Mytton [ 01/Aug/10 ]

Seem to be missing the full SECONDARY 1 log. Here it is:

Sun Aug 1 04:38:24 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now up
Sun Aug 1 04:38:27 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-2C-81.compute-1.internal:27018 is now up
Sun Aug 1 04:38:28 [initandlisten] connection accepted from 10.208.51.111:33702 #3
Sun Aug 1 05:16:47 [conn1] end connection 10.208.46.160:38289
Sun Aug 1 05:16:47 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
Sun Aug 1 05:16:48 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now down (or slow to respond)
Sun Aug 1 05:16:48 [rs Manager] replSet info electSelf 1
Sun Aug 1 05:16:48 [rs Manager] replSet PRIMARY
Sun Aug 1 05:16:55 [initandlisten] connection accepted from 10.208.46.160:48179 #4
Sun Aug 1 05:16:56 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now up
Sun Aug 1 05:16:57 [initandlisten] connection accepted from 10.208.46.160:48181 #5
Sun Aug 1 05:16:58 [initandlisten] connection accepted from 10.208.51.111:34965 #6
Sun Aug 1 05:17:00 [conn5] getmore local.oplog.rs cid:5040576339790303823 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3008ms
Sun Aug 1 05:17:01 [conn6] getmore local.oplog.rs cid:2586884600444785463 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3008ms
Sun Aug 1 05:17:48 [conn5] getmore local.oplog.rs cid:5040576339790303823 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3016ms
Sun Aug 1 05:17:49 [conn6] getmore local.oplog.rs cid:2586884600444785463 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3025ms
Sun Aug 1 05:17:51 [conn5] getmore local.oplog.rs cid:5040576339790303823 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3017ms
Sun Aug 1 05:17:52 [conn6] getmore local.oplog.rs cid:2586884600444785463 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3008ms
Sun Aug 1 05:17:57 [conn5] getmore local.oplog.rs cid:5040576339790303823 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3007ms
Sun Aug 1 05:17:58 [conn6] getmore local.oplog.rs cid:2586884600444785463 getMore: { ts:

{ $gte: new Date(5500358045240131585) }

} bytes:20 nreturned:0 3007ms
Sun Aug 1 05:18:03 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
Sun Aug 1 05:18:03 [interruptThread] now exiting
Sun Aug 1 05:18:03 dbexit:

Sun Aug 1 05:18:03 [interruptThread] shutdown: going to close listening sockets...
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 10
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 11
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 12
Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 13
Sun Aug 1 05:18:03 [interruptThread] shutdown: going to flush oplog...
Sun Aug 1 05:18:03 [interruptThread] shutdown: going to close sockets...
Sun Aug 1 05:18:03 [interruptThread] shutdown: waiting for fs preallocator...
Sun Aug 1 05:18:03 [interruptThread] shutdown: closing all files...
Sun Aug 1 05:18:03 [conn2] end connection 127.0.0.1:48610
Sun Aug 1 05:18:03 closeAllFiles() finished

Sun Aug 1 05:18:03 [interruptThread] shutdown: removing fs lock...
Sun Aug 1 05:18:03 dbexit: really exiting now

Generated at Thu Feb 08 02:57:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.