[SERVER-23462] all memeber become primary in 3 node replicaset Created: 01/Apr/16  Updated: 28/Apr/16  Resolved: 28/Apr/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.6.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: leipei Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 3_primarys.txt     Text File first.log     Text File second.log     Text File third.log    
Operating System: ALL
Participants:

 Description   

we have a replication for developing.
normally, There are a primary node and two secondary nodes .
but yesterday, we found all member become primary.



 Comments   
Comment by Kelsey Schubert [ 28/Apr/16 ]

Hi leipei2352,

Thank you for confirming that your tester changed the date time on the ntp server. Replication in MongoDB 2.6 does not support this form of time manipulation. However, we expect that this issue will not occur in MongoDB 3.2, which utilizes a new default replication protocol.

If this is still an issue after stopping the tests that manipulate the ntp server, please comment and we continue to investigate.

Thank you,
Thomas

Comment by leipei [ 15/Apr/16 ]

hi,Ramon
The three hosts sync time from a same ntp server.
But our tester changed date time on ntp server and force time-sync for several times.

Comment by leipei [ 15/Apr/16 ]

the first host(change time clock second time):
2016-03-29T13:36:26.052+0800 [conn23344] end connection 192.168.208.5:59185 (62 connections now open)
2016-03-29T13:36:26.067+0800 [signalProcessingThread] shutdown: closing all files...
2016-03-29T13:36:27.279+0800 [signalProcessingThread] closeAllFiles() finished
2016-03-29T13:36:27.279+0800 [signalProcessingThread] journalCleanup...
2016-03-29T13:36:27.279+0800 [signalProcessingThread] removeJournalFiles
2016-03-29T13:36:27.284+0800 [signalProcessingThread] shutdown: removing fs lock...
2016-03-29T13:36:27.284+0800 [signalProcessingThread] dbexit: really exiting now
2016-03-14T15:25:55.682+0800 ***** SERVER RESTARTED *****
2016-03-14T15:25:55.687+0800 [initandlisten] MongoDB starting : pid=34459 port=27017 dbpath=/var/ceilometer 64-bit host=mongodb3
2016-03-14T15:25:55.687+0800 [initandlisten] db version v2.6.8
2016-03-14T15:25:55.687+0800 [initandlisten] git version: nogitversion
2016-03-14T15:25:55.687+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1g 7 Apr 2014
2016-03-14T15:25:55.687+0800 [initandlisten] build info: Linux HGH1000017384 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 BOOST_LIB_VERSION=1_49
2016-03-14T15:25:55.687+0800 [initandlisten] allocator: tcmalloc
2016-03-14T15:25:55.687+0800 [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net:

{ bindIp: "192.168.219.140" }

, processManagement:

{ fork: true }

, replication:

{ replSet: "fsp" }

, storage:

{ dbPath: "/var/ceilometer", smallFiles: true }

, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/fusionsphere/component/mongodb/mongodb.log" }

}
2016-03-14T15:25:55.687+0800 [initandlisten]
2016-03-14T15:25:55.687+0800 [initandlisten] ** WARNING: Readahead for /var/ceilometer is set to 512KB
2016-03-14T15:25:55.687+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2016-03-14T15:25:55.687+0800 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2016-03-14T15:25:55.689+0800 [initandlisten] journal dir=/var/ceilometer/journal
2016-03-14T15:25:55.689+0800 [initandlisten] recover : no journal files present, no recovery needed
2016-03-14T15:25:55.802+0800 [initandlisten] waiting for connections on port 27017
2016-03-14T15:25:55.803+0800 [initandlisten] connection accepted from 192.168.208.15:41543 #1 (1 connection now open)
2016-03-14T15:25:55.813+0800 [rsStart] replSet I am 192.168.219.140:27017
2016-03-14T15:25:55.818+0800 [rsStart] replSet STARTUP2
2016-03-14T15:25:55.818+0800 [rsSync] replSet SECONDARY
2016-03-14T15:25:57.053+0800 [initandlisten] connection accepted from 192.168.208.4:58512 #2 (2 connections now open)
2016-03-14T15:25:57.053+0800 [initandlisten] connection accepted from 192.168.208.4:58513 #3 (3 connections now open)
2016-03-14T15:25:57.053+0800 [initandlisten] connection accepted from 192.168.208.4:58511 #4 (4 connections now open)
2016-03-14T15:25:57.053+0800 [conn2] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:57.053+0800 [conn3] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:57.053+0800 [conn4] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:57.461+0800 [initandlisten] connection accepted from 192.168.208.8:55159 #5 (2 connections now open)
2016-03-14T15:25:57.462+0800 [initandlisten] connection accepted from 192.168.208.8:55161 #6 (3 connections now open)
2016-03-14T15:25:57.462+0800 [initandlisten] connection accepted from 192.168.208.8:55160 #7 (4 connections now open)
2016-03-14T15:25:57.462+0800 [conn5] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:57.462+0800 [conn6] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:57.462+0800 [conn7] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:57.647+0800 [initandlisten] connection accepted from 192.168.208.14:48732 #8 (2 connections now open)
2016-03-14T15:25:57.652+0800 [conn8] end connection 192.168.208.14:48732 (1 connection now open)
2016-03-14T15:25:57.652+0800 [initandlisten] connection accepted from 192.168.208.14:48733 #9 (2 connections now open)
2016-03-14T15:25:57.707+0800 [initandlisten] connection accepted from 192.168.208.13:38419 #10 (3 connections now open)
2016-03-14T15:25:57.711+0800 [conn10] end connection 192.168.208.13:38419 (2 connections now open)
2016-03-14T15:25:57.711+0800 [initandlisten] connection accepted from 192.168.208.13:38420 #11 (3 connections now open)
2016-03-14T15:25:57.814+0800 [rsHealthPoll] replSet member 192.168.219.142:27017 is up
2016-03-14T15:25:57.814+0800 [rsHealthPoll] replSet member 192.168.219.141:27017 is up
2016-03-14T15:25:57.814+0800 [rsHealthPoll] replSet member 192.168.219.142:27017 is now in state SECONDARY
2016-03-14T15:25:57.814+0800 [rsHealthPoll] replSet member 192.168.219.141:27017 is now in state SECONDARY
2016-03-14T15:25:57.814+0800 [rsMgr] not electing self, 192.168.219.142:27017 would veto with 'I don't think 192.168.219.140:27017 is electable'
2016-03-14T15:25:57.815+0800 [rsMgr] not electing self, 192.168.219.142:27017 would veto with 'I don't think 192.168.219.140:27017 is electable'
2016-03-14T15:25:58.968+0800 [initandlisten] connection accepted from 192.168.208.6:40681 #12 (4 connections now open)
2016-03-14T15:25:58.978+0800 [initandlisten] connection accepted from 192.168.208.6:40680 #13 (5 connections now open)
2016-03-14T15:25:58.978+0800 [initandlisten] connection accepted from 192.168.208.6:40683 #14 (6 connections now open)
2016-03-14T15:25:58.978+0800 [conn12] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:58.978+0800 [conn13] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:58.978+0800 [conn14] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:59.096+0800 [initandlisten] connection accepted from 192.168.208.5:59793 #15 (4 connections now open)
2016-03-14T15:25:59.096+0800 [initandlisten] connection accepted from 192.168.208.5:59795 #16 (5 connections now open)
2016-03-14T15:25:59.096+0800 [initandlisten] connection accepted from 192.168.208.5:59794 #17 (6 connections now open)
2016-03-14T15:25:59.096+0800 [conn15] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:59.096+0800 [conn16] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:25:59.096+0800 [conn17] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.046+0800 [initandlisten] connection accepted from 192.168.208.4:58520 #18 (4 connections now open)
2016-03-14T15:26:00.047+0800 [conn18] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.047+0800 [initandlisten] connection accepted from 192.168.208.4:58521 #19 (4 connections now open)
2016-03-14T15:26:00.047+0800 [conn19] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.053+0800 [initandlisten] connection accepted from 192.168.208.4:58522 #20 (4 connections now open)
2016-03-14T15:26:00.053+0800 [conn20] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.053+0800 [initandlisten] connection accepted from 192.168.208.4:58523 #21 (4 connections now open)
2016-03-14T15:26:00.054+0800 [initandlisten] connection accepted from 192.168.208.4:58524 #22 (5 connections now open)
2016-03-14T15:26:00.054+0800 [conn22] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.054+0800 [conn21] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.458+0800 [initandlisten] connection accepted from 192.168.208.8:55174 #23 (4 connections now open)
2016-03-14T15:26:00.458+0800 [initandlisten] connection accepted from 192.168.208.8:55173 #24 (5 connections now open)
2016-03-14T15:26:00.458+0800 [initandlisten] connection accepted from 192.168.208.8:55175 #25 (6 connections now open)
2016-03-14T15:26:00.459+0800 [conn24] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.459+0800 [conn23] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.459+0800 [conn25] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.459+0800 [initandlisten] connection accepted from 192.168.208.8:55176 #26 (4 connections now open)
2016-03-14T15:26:00.460+0800 [conn26] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:00.460+0800 [initandlisten] connection accepted from 192.168.208.8:55177 #27 (4 connections now open)
2016-03-14T15:26:00.460+0800 [conn27] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:01.964+0800 [initandlisten] connection accepted from 192.168.208.6:40707 #28 (4 connections now open)
2016-03-14T15:26:01.964+0800 [initandlisten] connection accepted from 192.168.208.6:40709 #29 (5 connections now open)
2016-03-14T15:26:01.965+0800 [conn28] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:01.965+0800 [initandlisten] connection accepted from 192.168.208.6:40710 #30 (5 connections now open)
2016-03-14T15:26:01.965+0800 [conn29] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:01.965+0800 [initandlisten] connection accepted from 192.168.208.6:40711 #31 (5 connections now open)
2016-03-14T15:26:01.965+0800 [conn30] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:01.965+0800 [conn31] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:01.971+0800 [initandlisten] connection accepted from 192.168.208.6:40679 #32 (4 connections now open)
2016-03-14T15:26:01.972+0800 [conn32] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:02.090+0800 [initandlisten] connection accepted from 192.168.208.5:59808 #33 (4 connections now open)
2016-03-14T15:26:02.090+0800 [conn33] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:02.095+0800 [initandlisten] connection accepted from 192.168.208.5:59792 #34 (4 connections now open)
2016-03-14T15:26:02.096+0800 [conn34] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:02.097+0800 [initandlisten] connection accepted from 192.168.208.5:59809 #35 (4 connections now open)
2016-03-14T15:26:02.097+0800 [initandlisten] connection accepted from 192.168.208.5:59810 #36 (5 connections now open)
2016-03-14T15:26:02.097+0800 [conn35] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:02.098+0800 [conn36] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:02.098+0800 [initandlisten] connection accepted from 192.168.208.5:59811 #37 (4 connections now open)
2016-03-14T15:26:02.098+0800 [conn37] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-14T15:26:03.815+0800 [rsMgr] replSet info electSelf 0
2016-03-14T15:26:03.819+0800 [rsMgr] replSet PRIMARY

Comment by leipei [ 15/Apr/16 ]

the first host(change time clock first time):
2016-03-12T09:59:40.807+0800 [conn49] end connection 192.168.208.2:56616 (77 connections now open)
2016-03-12T09:59:40.820+0800 [signalProcessingThread] shutdown: closing all files...
2016-03-12T09:59:40.830+0800 [signalProcessingThread] closeAllFiles() finished
2016-03-12T09:59:40.830+0800 [signalProcessingThread] journalCleanup...
2016-03-12T09:59:40.830+0800 [signalProcessingThread] removeJournalFiles
2016-03-12T09:59:40.831+0800 [signalProcessingThread] shutdown: removing fs lock...
2016-03-12T09:59:40.831+0800 [signalProcessingThread] dbexit: really exiting now
2016-03-27T08:16:45.290+0800 ***** SERVER RESTARTED *****
2016-03-27T08:16:45.296+0800 [initandlisten] MongoDB starting : pid=18895 port=27017 dbpath=/var/ceilometer 64-bit host=mongodb3
2016-03-27T08:16:45.296+0800 [initandlisten] db version v2.6.8
2016-03-27T08:16:45.296+0800 [initandlisten] git version: nogitversion
2016-03-27T08:16:45.296+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1g 7 Apr 2014
2016-03-27T08:16:45.296+0800 [initandlisten] build info: Linux HGH1000017384 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 BOOST_LIB_VERSION=1_49
2016-03-27T08:16:45.296+0800 [initandlisten] allocator: tcmalloc
2016-03-27T08:16:45.296+0800 [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net:

{ bindIp: "192.168.219.140" }

, processManagement:

{ fork: true }

, replication:

{ replSet: "fsp" }

, storage:

{ dbPath: "/var/ceilometer", smallFiles: true }

, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/fusionsphere/component/mongodb/mongodb.log" }

}
2016-03-27T08:16:45.297+0800 [initandlisten]
2016-03-27T08:16:45.297+0800 [initandlisten] ** WARNING: Readahead for /var/ceilometer is set to 512KB
2016-03-27T08:16:45.297+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2016-03-27T08:16:45.297+0800 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2016-03-27T08:16:45.299+0800 [initandlisten] journal dir=/var/ceilometer/journal
2016-03-27T08:16:45.299+0800 [initandlisten] recover : no journal files present, no recovery needed
2016-03-27T08:16:45.354+0800 [initandlisten] waiting for connections on port 27017
2016-03-27T08:16:45.355+0800 [initandlisten] connection accepted from 192.168.208.15:41051 #1 (1 connection now open)
2016-03-27T08:16:45.364+0800 [rsStart] warning: Failed to connect to 192.168.219.142:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.364+0800 [rsStart] replSet I am 192.168.219.140:27017
2016-03-27T08:16:45.364+0800 [rsStart] warning: Failed to connect to 192.168.219.142:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.365+0800 [rsHealthPoll] replset info 192.168.219.141:27017 thinks that we are down
2016-03-27T08:16:45.365+0800 [rsHealthPoll] replSet member 192.168.219.141:27017 is up
2016-03-27T08:16:45.365+0800 [rsHealthPoll] replSet member 192.168.219.141:27017 is now in state SECONDARY
2016-03-27T08:16:45.371+0800 [rsStart] replSet STARTUP2
2016-03-27T08:16:45.371+0800 [rsSync] replSet SECONDARY
2016-03-27T08:16:45.404+0800 [initandlisten] connection accepted from 192.168.208.13:41006 #2 (2 connections now open)
2016-03-27T08:16:45.404+0800 [conn2] end connection 192.168.208.13:41006 (1 connection now open)
2016-03-27T08:16:45.408+0800 [initandlisten] connection accepted from 192.168.208.13:41008 #3 (2 connections now open)
2016-03-27T08:16:46.791+0800 [initandlisten] connection accepted from 192.168.208.4:37475 #4 (3 connections now open)
2016-03-27T08:16:46.792+0800 [initandlisten] connection accepted from 192.168.208.4:37474 #5 (4 connections now open)
2016-03-27T08:16:46.792+0800 [initandlisten] connection accepted from 192.168.208.4:37477 #6 (5 connections now open)
2016-03-27T08:16:46.792+0800 [conn4] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:46.792+0800 [conn5] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:46.792+0800 [conn6] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.301+0800 [initandlisten] connection accepted from 192.168.208.14:49543 #7 (3 connections now open)
2016-03-27T08:16:47.305+0800 [conn7] end connection 192.168.208.14:49543 (2 connections now open)
2016-03-27T08:16:47.305+0800 [initandlisten] connection accepted from 192.168.208.14:49544 #8 (3 connections now open)
2016-03-27T08:16:47.328+0800 [initandlisten] connection accepted from 192.168.208.8:49887 #9 (4 connections now open)
2016-03-27T08:16:47.328+0800 [initandlisten] connection accepted from 192.168.208.8:49888 #10 (5 connections now open)
2016-03-27T08:16:47.328+0800 [initandlisten] connection accepted from 192.168.208.8:49886 #11 (6 connections now open)
2016-03-27T08:16:47.329+0800 [conn9] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.329+0800 [conn10] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.329+0800 [conn11] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.365+0800 [rsHealthPoll] replSet member 192.168.219.142:27017 is up
2016-03-27T08:16:47.365+0800 [rsHealthPoll] replSet member 192.168.219.142:27017 is now in state SECONDARY
2016-03-27T08:16:47.366+0800 [rsMgr] not electing self, 192.168.219.142:27017 would veto with 'I don't think 192.168.219.140:27017 is electable'
2016-03-27T08:16:47.590+0800 [initandlisten] connection accepted from 192.168.208.5:52391 #12 (4 connections now open)
2016-03-27T08:16:47.590+0800 [initandlisten] connection accepted from 192.168.208.5:52392 #13 (5 connections now open)
2016-03-27T08:16:47.590+0800 [initandlisten] connection accepted from 192.168.208.5:52395 #14 (6 connections now open)
2016-03-27T08:16:47.590+0800 [initandlisten] connection accepted from 192.168.208.5:52396 #15 (7 connections now open)
2016-03-27T08:16:47.591+0800 [conn12] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.591+0800 [conn13] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.591+0800 [conn14] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:47.591+0800 [conn15] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:48.651+0800 [initandlisten] connection accepted from 192.168.208.6:45368 #16 (4 connections now open)
2016-03-27T08:16:48.651+0800 [initandlisten] connection accepted from 192.168.208.6:45365 #17 (5 connections now open)
2016-03-27T08:16:48.651+0800 [initandlisten] connection accepted from 192.168.208.6:45367 #18 (6 connections now open)
2016-03-27T08:16:48.652+0800 [conn17] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:48.652+0800 [conn18] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:48.652+0800 [conn16] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:49.785+0800 [initandlisten] connection accepted from 192.168.208.4:37485 #19 (4 connections now open)
2016-03-27T08:16:49.785+0800 [conn19] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:49.785+0800 [initandlisten] connection accepted from 192.168.208.4:37486 #20 (4 connections now open)
2016-03-27T08:16:49.786+0800 [conn20] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:49.791+0800 [initandlisten] connection accepted from 192.168.208.4:37473 #21 (4 connections now open)
2016-03-27T08:16:49.791+0800 [initandlisten] connection accepted from 192.168.208.4:37472 #22 (5 connections now open)
2016-03-27T08:16:49.791+0800 [conn22] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:49.792+0800 [conn21] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:49.792+0800 [initandlisten] connection accepted from 192.168.208.4:37487 #23 (4 connections now open)
2016-03-27T08:16:49.793+0800 [conn23] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.325+0800 [initandlisten] connection accepted from 192.168.208.8:49906 #24 (4 connections now open)
2016-03-27T08:16:50.326+0800 [conn24] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.326+0800 [initandlisten] connection accepted from 192.168.208.8:49907 #25 (5 connections now open)
2016-03-27T08:16:50.326+0800 [initandlisten] connection accepted from 192.168.208.8:49908 #26 (5 connections now open)
2016-03-27T08:16:50.326+0800 [conn25] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.326+0800 [initandlisten] connection accepted from 192.168.208.8:49910 #27 (5 connections now open)
2016-03-27T08:16:50.326+0800 [conn26] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.327+0800 [conn27] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.327+0800 [initandlisten] connection accepted from 192.168.208.8:49911 #28 (4 connections now open)
2016-03-27T08:16:50.327+0800 [conn28] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.587+0800 [initandlisten] connection accepted from 192.168.208.5:52406 #29 (4 connections now open)
2016-03-27T08:16:50.588+0800 [conn29] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.588+0800 [initandlisten] connection accepted from 192.168.208.5:52407 #30 (4 connections now open)
2016-03-27T08:16:50.588+0800 [initandlisten] connection accepted from 192.168.208.5:52408 #31 (5 connections now open)
2016-03-27T08:16:50.588+0800 [conn30] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.588+0800 [conn31] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:50.594+0800 [initandlisten] connection accepted from 192.168.208.5:52390 #32 (4 connections now open)
2016-03-27T08:16:50.594+0800 [conn32] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:51.647+0800 [initandlisten] connection accepted from 192.168.208.6:45385 #33 (4 connections now open)
2016-03-27T08:16:51.647+0800 [initandlisten] connection accepted from 192.168.208.6:45386 #34 (5 connections now open)
2016-03-27T08:16:51.647+0800 [conn33] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:51.648+0800 [conn34] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:51.654+0800 [initandlisten] connection accepted from 192.168.208.6:45364 #35 (4 connections now open)
2016-03-27T08:16:51.654+0800 [initandlisten] connection accepted from 192.168.208.6:45363 #36 (5 connections now open)
2016-03-27T08:16:51.655+0800 [conn35] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:51.655+0800 [initandlisten] connection accepted from 192.168.208.6:45361 #37 (6 connections now open)
2016-03-27T08:16:51.655+0800 [conn36] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:51.656+0800 [conn37] AssertionException handling request, closing client connection: 17132 SSL handshake received but server is started without SSL support
2016-03-27T08:16:53.367+0800 [rsMgr] replSet info electSelf 0
2016-03-27T08:16:53.372+0800 [rsMgr] replSet PRIMARY

Comment by leipei [ 15/Apr/16 ]

the second host(change time clock second time):
2016-03-29T13:36:25.956+0800 [conn367] end connection 192.168.208.4:56855 (418 connections now open)
2016-03-29T13:36:25.975+0800 [signalProcessingThread] shutdown: closing all files...
2016-03-29T13:36:27.008+0800 [signalProcessingThread] File Closing Progress: 280/303 92% (files)
2016-03-29T13:36:27.190+0800 [signalProcessingThread] closeAllFiles() finished
2016-03-29T13:36:27.190+0800 [signalProcessingThread] journalCleanup...
2016-03-29T13:36:27.190+0800 [signalProcessingThread] removeJournalFiles
2016-03-29T13:36:27.195+0800 [signalProcessingThread] shutdown: removing fs lock...
2016-03-29T13:36:27.195+0800 [signalProcessingThread] dbexit: really exiting now
2016-03-14T15:25:55.553+0800 ***** SERVER RESTARTED *****
2016-03-14T15:25:55.559+0800 [initandlisten] MongoDB starting : pid=303 port=27017 dbpath=/var/ceilometer 64-bit host=mongodb2
2016-03-14T15:25:55.559+0800 [initandlisten] db version v2.6.8
2016-03-14T15:25:55.559+0800 [initandlisten] git version: nogitversion
2016-03-14T15:25:55.559+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1g 7 Apr 2014
2016-03-14T15:25:55.559+0800 [initandlisten] build info: Linux HGH1000017384 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 BOOST_LIB_VERSION=1_49
2016-03-14T15:25:55.559+0800 [initandlisten] allocator: tcmalloc
2016-03-14T15:25:55.559+0800 [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net:

{ bindIp: "192.168.219.141" }

, processManagement:

{ fork: true }

, replication:

{ replSet: "fsp" }

, storage:

{ dbPath: "/var/ceilometer", smallFiles: true }

, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/fusionsphere/component/mongodb/mongodb.log" }

}
2016-03-14T15:25:55.559+0800 [initandlisten]
2016-03-14T15:25:55.559+0800 [initandlisten] ** WARNING: Readahead for /var/ceilometer is set to 512KB
2016-03-14T15:25:55.559+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2016-03-14T15:25:55.559+0800 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2016-03-14T15:25:55.561+0800 [initandlisten] journal dir=/var/ceilometer/journal
2016-03-14T15:25:55.561+0800 [initandlisten] recover : no journal files present, no recovery needed
2016-03-14T15:25:55.627+0800 [initandlisten] waiting for connections on port 27017
2016-03-14T15:25:55.628+0800 [initandlisten] connection accepted from 192.168.208.14:55906 #1 (1 connection now open)
2016-03-14T15:25:55.638+0800 [rsStart] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.641+0800 [rsStart] warning: Failed to connect to 192.168.219.142:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.641+0800 [rsStart] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.641+0800 [rsStart] replSet I am 192.168.219.141:27017
2016-03-14T15:25:55.641+0800 [rsStart] warning: Failed to connect to 192.168.219.142:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.642+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.642+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140), connection attempt failed
2016-03-14T15:25:55.642+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.642+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140) failed, connection attempt failed
2016-03-14T15:25:55.643+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.643+0800 [rsHealthPoll] replset info 192.168.219.140:27017 heartbeat failed, retrying
2016-03-14T15:25:55.644+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.644+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140) failed, connection attempt failed
2016-03-14T15:25:55.644+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.644+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140) failed, connection attempt failed
2016-03-14T15:25:55.644+0800 [rsStart] replSet STARTUP2
2016-03-14T15:25:55.644+0800 [rsSync] replSet SECONDARY

Comment by leipei [ 15/Apr/16 ]

the second host(change time clock first time):
2016-03-12T09:59:40.771+0800 [conn41] end connection 192.168.208.15:55583 (85 connections now open)
2016-03-12T09:59:40.772+0800 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: 192.168.219.142:27017
2016-03-12T09:59:40.828+0800 [signalProcessingThread] shutdown: closing all files...
2016-03-12T09:59:40.866+0800 [signalProcessingThread] closeAllFiles() finished
2016-03-12T09:59:40.866+0800 [signalProcessingThread] journalCleanup...
2016-03-12T09:59:40.866+0800 [signalProcessingThread] removeJournalFiles
2016-03-12T09:59:40.867+0800 [signalProcessingThread] shutdown: removing fs lock...
2016-03-12T09:59:40.867+0800 [signalProcessingThread] dbexit: really exiting now
2016-03-27T08:16:45.220+0800 ***** SERVER RESTARTED *****
2016-03-27T08:16:45.225+0800 [initandlisten] MongoDB starting : pid=5029 port=27017 dbpath=/var/ceilometer 64-bit host=mongodb2
2016-03-27T08:16:45.225+0800 [initandlisten] db version v2.6.8
2016-03-27T08:16:45.225+0800 [initandlisten] git version: nogitversion
2016-03-27T08:16:45.225+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1g 7 Apr 2014
2016-03-27T08:16:45.225+0800 [initandlisten] build info: Linux HGH1000017384 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 BOOST_LIB_VERSION=1_49
2016-03-27T08:16:45.225+0800 [initandlisten] allocator: tcmalloc
2016-03-27T08:16:45.226+0800 [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net:

{ bindIp: "192.168.219.141" }

, processManagement:

{ fork: true }

, replication:

{ replSet: "fsp" }

, storage:

{ dbPath: "/var/ceilometer", smallFiles: true }

, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/fusionsphere/component/mongodb/mongodb.log" }

}
2016-03-27T08:16:45.226+0800 [initandlisten]
2016-03-27T08:16:45.226+0800 [initandlisten] ** WARNING: Readahead for /var/ceilometer is set to 512KB
2016-03-27T08:16:45.226+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2016-03-27T08:16:45.226+0800 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2016-03-27T08:16:45.228+0800 [initandlisten] journal dir=/var/ceilometer/journal
2016-03-27T08:16:45.228+0800 [initandlisten] recover : no journal files present, no recovery needed
2016-03-27T08:16:45.283+0800 [initandlisten] waiting for connections on port 27017
2016-03-27T08:16:45.284+0800 [initandlisten] connection accepted from 192.168.208.14:56716 #1 (1 connection now open)
2016-03-27T08:16:45.291+0800 [rsStart] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.294+0800 [rsStart] warning: Failed to connect to 192.168.219.142:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.295+0800 [rsStart] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.295+0800 [rsStart] replSet I am 192.168.219.141:27017
2016-03-27T08:16:45.295+0800 [rsStart] warning: Failed to connect to 192.168.219.142:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.295+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.295+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140), connection attempt failed
2016-03-27T08:16:45.296+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.296+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140) failed, connection attempt failed
2016-03-27T08:16:45.297+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.297+0800 [rsHealthPoll] replset info 192.168.219.140:27017 heartbeat failed, retrying
2016-03-27T08:16:45.297+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.297+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140) failed, connection attempt failed
2016-03-27T08:16:45.298+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-27T08:16:45.298+0800 [rsHealthPoll] couldn't connect to 192.168.219.140:27017: couldn't connect to server 192.168.219.140:27017 (192.168.219.140) failed, connection attempt failed
2016-03-27T08:16:45.299+0800 [rsStart] replSet STARTUP2
2016-03-27T08:16:45.299+0800 [rsSync] replSet SECONDARY

Comment by leipei [ 15/Apr/16 ]

the third host(change time clock second time):
2016-03-29T13:36:26.019+0800 [conn36] end connection 192.168.208.2:57700 (84 connections now open)
2016-03-29T13:36:26.028+0800 [signalProcessingThread] shutdown: closing all files...
2016-03-29T13:36:27.033+0800 [signalProcessingThread] closeAllFiles() finished
2016-03-29T13:36:27.033+0800 [signalProcessingThread] journalCleanup...
2016-03-29T13:36:27.033+0800 [signalProcessingThread] removeJournalFiles
2016-03-29T13:36:27.036+0800 [signalProcessingThread] shutdown: removing fs lock...
2016-03-29T13:36:27.036+0800 [signalProcessingThread] dbexit: really exiting now
2016-03-14T15:25:55.624+0800 ***** SERVER RESTARTED *****
2016-03-14T15:25:55.629+0800 [initandlisten] MongoDB starting : pid=31465 port=27017 dbpath=/var/ceilometer 64-bit host=mongodb1
2016-03-14T15:25:55.630+0800 [initandlisten] db version v2.6.8
2016-03-14T15:25:55.630+0800 [initandlisten] git version: nogitversion
2016-03-14T15:25:55.630+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1g 7 Apr 2014
2016-03-14T15:25:55.630+0800 [initandlisten] build info: Linux HGH1000017384 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 BOOST_LIB_VERSION=1_49
2016-03-14T15:25:55.630+0800 [initandlisten] allocator: tcmalloc
2016-03-14T15:25:55.630+0800 [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net:

{ bindIp: "192.168.219.142" }

, processManagement:

{ fork: true }

, replication:

{ replSet: "fsp" }

, storage:

{ dbPath: "/var/ceilometer", smallFiles: true }

, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/fusionsphere/component/mongodb/mongodb.log" }

}
2016-03-14T15:25:55.630+0800 [initandlisten]
2016-03-14T15:25:55.630+0800 [initandlisten] ** WARNING: Readahead for /var/ceilometer is set to 512KB
2016-03-14T15:25:55.630+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2016-03-14T15:25:55.630+0800 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2016-03-14T15:25:55.631+0800 [initandlisten] journal dir=/var/ceilometer/journal
2016-03-14T15:25:55.631+0800 [initandlisten] recover : no journal files present, no recovery needed
2016-03-14T15:25:55.687+0800 [initandlisten] waiting for connections on port 27017
2016-03-14T15:25:55.688+0800 [initandlisten] connection accepted from 192.168.208.13:43077 #1 (1 connection now open)
2016-03-14T15:25:55.696+0800 [rsStart] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.701+0800 [rsStart] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.701+0800 [rsStart] replSet I am 192.168.219.142:27017
2016-03-14T15:25:55.701+0800 [rsStart] replSet STARTUP2
2016-03-14T15:25:55.701+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:111 Connection refused
2016-03-14T15:25:55.701+0800 [rsSync] replSet SECONDARY

Comment by leipei [ 15/Apr/16 ]

the third host(change time clock first time):
2016-03-12T09:59:40.787+0800 [conn145] end connection 192.168.208.5:34033 (136 connections now open)
2016-03-12T09:59:40.837+0800 [signalProcessingThread] shutdown: closing all files...
2016-03-12T09:59:40.841+0800 [signalProcessingThread] closeAllFiles() finished
2016-03-12T09:59:40.841+0800 [signalProcessingThread] journalCleanup...
2016-03-12T09:59:40.841+0800 [signalProcessingThread] removeJournalFiles
2016-03-12T09:59:40.842+0800 [signalProcessingThread] shutdown: removing fs lock...
2016-03-12T09:59:40.842+0800 [signalProcessingThread] dbexit: really exiting now
2016-03-27T08:16:45.270+0800 ***** SERVER RESTARTED *****
2016-03-27T08:16:45.275+0800 [initandlisten] MongoDB starting : pid=48516 port=27017 dbpath=/var/ceilometer 64-bit host=mongodb1
2016-03-27T08:16:45.275+0800 [initandlisten] db version v2.6.8
2016-03-27T08:16:45.275+0800 [initandlisten] git version: nogitversion
2016-03-27T08:16:45.275+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1g 7 Apr 2014
2016-03-27T08:16:45.275+0800 [initandlisten] build info: Linux HGH1000017384 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 BOOST_LIB_VERSION=1_49
2016-03-27T08:16:45.275+0800 [initandlisten] allocator: tcmalloc
2016-03-27T08:16:45.275+0800 [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net:

{ bindIp: "192.168.219.142" }

, processManagement:

{ fork: true }

, replication:

{ replSet: "fsp" }

, storage:

{ dbPath: "/var/ceilometer", smallFiles: true }

, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/fusionsphere/component/mongodb/mongodb.log" }

}
2016-03-27T08:16:45.275+0800 [initandlisten]
2016-03-27T08:16:45.276+0800 [initandlisten] ** WARNING: Readahead for /var/ceilometer is set to 512KB
2016-03-27T08:16:45.276+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2016-03-27T08:16:45.276+0800 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2016-03-27T08:16:45.277+0800 [initandlisten] journal dir=/var/ceilometer/journal
2016-03-27T08:16:45.277+0800 [initandlisten] recover : no journal files present, no recovery needed
2016-03-27T08:16:45.395+0800 [initandlisten] waiting for connections on port 27017
2016-03-27T08:16:45.396+0800 [initandlisten] connection accepted from 192.168.208.13:45678 #1 (1 connection now open)
2016-03-27T08:16:45.407+0800 [rsStart] replSet I am 192.168.219.142:27017
2016-03-27T08:16:45.407+0800 [rsStart] replSet STARTUP2
2016-03-27T08:16:45.408+0800 [rsSync] replSet SECONDARY
2016-03-27T08:16:45.408+0800 [rsHealthPoll] replset info 192.168.219.140:27017 thinks that we are down
2016-03-27T08:16:45.408+0800 [rsHealthPoll] replSet member 192.168.219.140:27017 is up
2016-03-27T08:16:45.408+0800 [rsHealthPoll] replSet member 192.168.219.140:27017 is now in state SECONDARY

Comment by leipei [ 15/Apr/16 ]

Ramon,I have reviewed previous log and find that Maybe our tester changed the time clock of three hosts.

Comment by Ramon Fernandez Marina [ 14/Apr/16 ]

leipei2352, a quick look at the logs shows warnings about clock skew. Can you please make sure all your nodes are running ntp and their clocks are properly synchronized?

Comment by Ramon Fernandez Marina [ 14/Apr/16 ]

Thanks for sending the logs leipei2352, I've moved them to files attached to the ticket (first.log, second.log and third.log).

Comment by leipei [ 14/Apr/16 ]

sorry, Thomas.
last week, I tried to upload the log file several times but found that I could not upload large file to Internet for some reason.
I will paste the key content of log file below.

Comment by Kelsey Schubert [ 05/Apr/16 ]

Hi leipei2352,

Sorry you are encountering trouble uploading the logs. I have created a secure upload portal here - can you please try uploading the complete logs for the three nodes using this portal?

Thank you for your help,
Thomas

Comment by leipei [ 05/Apr/16 ]

I find the below content in the seond member's mongodb log:
"
2016-03-21T21:10:59.905+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:113 No route to host
2016-03-21T21:11:02.797+0800 [rsBackgroundSync] Socket recv() timeout 192.168.219.140:27017
2016-03-21T21:11:02.797+0800 [rsBackgroundSync] caught exception (socket exception [RECV_TIMEOUT] for 192.168.219.140:27017) in destructor (~PiggyBackData)
2016-03-21T21:11:02.797+0800 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: 192.168.219.140:27017
2016-03-21T21:11:02.797+0800 [rsMgr] replSet PRIMARY
2016-03-21T21:11:03.535+0800 [rsHealthPoll] replSet member 192.168.219.141:27017 is now in state PRIMARY
2016-03-21T21:11:03.535+0800 [rsMgr] another PRIMARY detected but it should step down since it was elected earlier than me
2016-03-21T21:11:03.535+0800 [rsMgr] another PRIMARY detected but it should step down since it was elected earlier than me
2016-03-21T21:11:04.792+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:113 No route to host
2016-03-21T21:11:04.792+0800 [rsHealthPoll] replset info 192.168.219.140:27017 heartbeat failed, retrying
2016-03-21T21:11:05.203+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:05.210+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:05.218+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:05.224+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:05.229+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:05.233+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:05.240+0800 [clientcursormon] mem (MB) res:368 virt:123296
2016-03-21T21:11:05.240+0800 [clientcursormon] mapped (incl journal view):122948
2016-03-21T21:11:05.240+0800 [clientcursormon] connections:73
2016-03-21T21:11:05.240+0800 [clientcursormon] replication threads:32
2016-03-21T21:11:05.240+0800 [TTLMonitor] clock skew detected prev: 1459229783 now: 1458565865
2016-03-21T21:11:07.797+0800 [rsHealthPoll] warning: Failed to connect to 192.168.219.140:27017, reason: errno:113 No route to host
2016-03-21T21:11:09.536+0800 [rsMgr] another PRIMARY detected but it should step down since it was elected earlier than me
2016-03-21T21:11:09.536+0800 [rsMgr] another PRIMARY detected but it should step down since it was elected earlier than me

"

Comment by leipei [ 05/Apr/16 ]

I get an error "Cannot attach file mongodb.log.9: Unknown server error (403)." when I try to upload the mongodb log

Comment by Kelsey Schubert [ 01/Apr/16 ]

Hi leipei2352,

Thank you for reporting this behavior. So we can continue to investigate, can you please upload the complete logs for the three nodes in your replica set?

Thank you,
Thomas

Generated at Thu Feb 08 04:03:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.