-
Type:
Bug
-
Resolution: Cannot Reproduce
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.13.0
-
Component/s: Cluster Management, Connection Management
-
Environment:App Servers: Scala code,in jetty app server on on linux debian 7.8
DB: mongo 2.6.10 on primary, 2.6.8 on secondary that got elected as primary when original primary rebooted.
After a primary db machine did a reboot, a secondary node, next in priority, got elected as primary, and when the original primary came up and got re-elected as primary, the connections from the app servers, that moved first to the secondary node, that became a primary, did not move back to the original primary.
The application got - NullPointerExceptions from all the related DB code, we saw in the secondary node log file errors as below, and new connections were not established on the original primary (which could be accessed by mongo CLI).
Only a re-start of all app servers, caused the app severs to re-connect to the original primary:
Errors from the secondary log file, after it stepped down:
2015-09-21T08:22:23.186+0000 [conn2782255] 172.16.209.10:27017 is now closest at 55ffbe3e:3 2015-09-21T08:22:23.186+0000 [conn2782255] replSet info stepping down as primary secs=1 2015-09-21T08:22:23.187+0000 [conn2782255] replSet relinquishing primary state 2015-09-21T08:22:23.187+0000 [conn2782255] replSet SECONDARY 2015-09-21T08:22:23.187+0000 [conn2782255] replSet closing client sockets after relinquishing primary 2015-09-21T08:22:23.189+0000 [conn2782140] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [172.16.213.148:36694] 2015-09-21T08:22:24.261+0000 [rsMgr] not electing self, 172.16.210.10:27017 would veto with '172.16.144.20:27017 has lower priority than 172.16.209.10:27017' 2015-09-21T08:22:27.259+0000 [conn2782065] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [172.16.144.188:33313] 2015-09-21T08:22:27.259+0000 [conn2782059] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [172.16.22.10:63550] 2015-09-21T08:22:27.259+0000 [conn2782061] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [172.16.210.154:62381] 2015-09-21T08:22:27.259+0000 [conn2782247] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [172.16.209.76:40453] 2015-09-21T08:22:27.285+0000 [conn2782057] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.14.6.10:60497] 2015-09-21T08:22:27.486+0000 [conn2782072] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.12.103.1:52824] 2015-09-21T08:22:29.384+0000 [conn2782256] replSet info voting yea for 172.16.209.10:27017 (8) 2015-09-21T08:22:30.089+0000 [rsMgr] not electing self, 172.16.210.10:27017 would veto with '172.16.144.20:27017 has lower priority than 172.16.209.10:27017' 2015-09-21T08:22:34.324+0000 [rsMgr] not electing self, 172.16.210.10:27017 would veto with '172.16.144.20:27017 is trying to elect itself but 172.16.209.10:27017 is already primary and more up-to-date' 2015-09-21T08:22:34.585+0000 [rsHealthPoll] replSet member 172.16.209.10:27017 is now in state PRIMARY 2015-09-21T08:22:34.688+0000 [rsBackgroundSync] replSet syncing to: 172.16.209.10:27017 2015-09-21T08:22:34.689+0000 [rsBackgroundSync] replset setting syncSourceFeedback to 172.16.209.10:27017