[SERVER-23773] Replica set primary unexpectedly steps down for lower priority secondary Created: 17/Apr/16  Updated: 16/May/16  Resolved: 16/May/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.2.1
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Linar Savion Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongodb_repl_fail_14_04_2016.log     Text File mongodb_repl_fail_14_04_2016_arb.log     Text File mongodb_repl_fail_14_04_2016_prim.log    
Operating System: ALL
Participants:

 Description   

The replica set's lower priority secondary that had network issues suddenly elects itself(even though oplog is 10min old), causing primary to step down for no apparent reason,
Main higher priority node then gets re-elected as primary a few seconds later.

This issue triggered a rollback once the main node received primary, causing data loss at a critical moment.

The setup is 2 replicas (one with priority 1 and the other 0.5) and an arbiter node.

It appears that the arbiter voted for the older secondary even though it is in the same local network as the higher priority node, arbiter show this line in the log:

2016-04-14T14:14:13.544+0000 I ASIO     [ReplicationExecutor] dropping unhealthy pooled connection to 3.3.153.248:27017
2016-04-14T14:14:13.544+0000 I ASIO     [ReplicationExecutor] after drop, pool was empty, going to spawn some connections

build information:

{ "version" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "modules" : [  ], "allocator" : "tcmalloc", "javascriptEngine" : "mozjs", "sysInfo" : "deprecated", "versionArray" : [ 3, 2, 1, 0 ], "openssl" : { "running" : "OpenSSL 1.0.1k 8 Jan 2015", "compiled" : "OpenSSL 1.0.1e 11 Feb 2013" }, "buildEnvironment" : { "distmod" : "debian71", "distarch" : "x86_64", "cc" : "/opt/mongodbtoolchain/bin/gcc: gcc (GCC) 4.8.2", "ccflags" : "-fno-omit-frame-pointer -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fno-builtin-memcmp", "cxx" : "/opt/mongodbtoolchain/bin/g++: g++ (GCC) 4.8.2", "cxxflags" : "-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11", "linkflags" : "-fPIC -pthread -Wl,-z,now -rdynamic -fuse-ld=gold", "target_arch" : "x86_64", "target_os" : "linux" }, "bits" : 64, "debug" : false, "maxBsonObjectSize" : 16777216, "storageEngines" : [ "devnull", "ephemeralForTest", "mmapv1", "wiredTiger" ], "ok" : 1.0 }



 Comments   
Comment by Kelsey Schubert [ 16/May/16 ]

Hi linar-jether,

Thank you for your patience. I have carefully examined the logs and determined that this is expected behavior. In my response I'll be referring to the nodes by their aliases in the table below

I'll describe the election that occurs at 2016-04-14T14:18:52 since it has the most surrounding context. Other elections appear to have the same root cause.

Due to network issues, Node2 successfully connects to the Arbiter, but not Node1. Consequently, when the Arbiter has to drop its pooled connection to Node1, Node2 starts an election since it has not seen a primary (Node1) for over 10 seconds. Since neither Node2 nor the Arbiter are currently connected Node1, Node2 wins the election. Eventually, Node1 is able to connect to Node2 and schedules a priority takeover.

All members of a replica set must be able to connect to every other member of the set to support replication. Since this condition was not met, the failover occurred. Unfortunately, since MongoDB cannot predict these network conditions, it must always follow its replication protocol.

For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. Questions about troubleshooting network connectivity issues would involve more discussion are best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Linar Savion [ 17/Apr/16 ]

May be related to the comments at the end of this issue: SERVER-20985
Talking about a lower priority node being elected for a few seconds

Generated at Thu Feb 08 04:04:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.