-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
None
-
Affects Version/s: 3.2.1
-
Component/s: Replication
-
None
-
ALL
The replica set's lower priority secondary that had network issues suddenly elects itself(even though oplog is 10min old), causing primary to step down for no apparent reason,
Main higher priority node then gets re-elected as primary a few seconds later.
This issue triggered a rollback once the main node received primary, causing data loss at a critical moment.
The setup is 2 replicas (one with priority 1 and the other 0.5) and an arbiter node.
It appears that the arbiter voted for the older secondary even though it is in the same local network as the higher priority node, arbiter show this line in the log:
2016-04-14T14:14:13.544+0000 I ASIO [ReplicationExecutor] dropping unhealthy pooled connection to 3.3.153.248:27017 2016-04-14T14:14:13.544+0000 I ASIO [ReplicationExecutor] after drop, pool was empty, going to spawn some connections
build information:
{ "version" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "modules" : [ ], "allocator" : "tcmalloc", "javascriptEngine" : "mozjs", "sysInfo" : "deprecated", "versionArray" : [ 3, 2, 1, 0 ], "openssl" : { "running" : "OpenSSL 1.0.1k 8 Jan 2015", "compiled" : "OpenSSL 1.0.1e 11 Feb 2013" }, "buildEnvironment" : { "distmod" : "debian71", "distarch" : "x86_64", "cc" : "/opt/mongodbtoolchain/bin/gcc: gcc (GCC) 4.8.2", "ccflags" : "-fno-omit-frame-pointer -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fno-builtin-memcmp", "cxx" : "/opt/mongodbtoolchain/bin/g++: g++ (GCC) 4.8.2", "cxxflags" : "-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11", "linkflags" : "-fPIC -pthread -Wl,-z,now -rdynamic -fuse-ld=gold", "target_arch" : "x86_64", "target_os" : "linux" }, "bits" : 64, "debug" : false, "maxBsonObjectSize" : 16777216, "storageEngines" : [ "devnull", "ephemeralForTest", "mmapv1", "wiredTiger" ], "ok" : 1.0 }