[SERVER-28857] Strange election on network failure Created: 19/Apr/17  Updated: 27/Oct/23  Resolved: 01/May/17

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.4.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aristarkh Zagorodnikov Assignee: Kelsey Schubert
Resolution: Works as Designed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File t1.log.gz     File t2.log.gz     File t3.log.gz    
Operating System: ALL
Participants:

 Description   

The subject replicaset has 3 nodes (see rs.conf() below).
t1 IP address is 10.3.1.12
t2 IP address is 10.3.1.13
t3 IP address is 10.3.1.16

After a transient network failure (switch ports were disabled and enabled back) on the secondary (t3) it became primary, causing rollbacks on the previous primary (t1) and other secondary (t2). All writes are done with w:majority, so this is really strange. Logs from all three machines are attached.

rs.conf()

{
        "_id" : "driveFS-temp-1",
        "version" : 4,
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : false,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "t1.s1.fs.drive.bru:27231",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
 
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "t2.s1.fs.drive.bru:27231",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
 
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "t3.s1.fs.drive.bru:27231",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
 
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 5000,
                "catchUpTimeoutMillis" : 2000,
                "getLastErrorModes" : {
 
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("58c9657b40aba377920b23f2")
        }
}



 Comments   
Comment by Aristarkh Zagorodnikov [ 01/May/17 ]

Thank you for the information. I was more surprised that election allowed a secondary to be elected as primary when primary was available and connected to the other secondary. Well, since this preserves the acknowledged majority writes it's okay.

Comment by Kelsey Schubert [ 01/May/17 ]

Hi onyxmaster,

After reviewing the logs, there is no indication of a bug during this failover. While, w : majority acknowledges writes will not be rolled back, writes written with this write concern that have not been acknowledged to the application are liable to be rolled back on failover. In this case, it appears that writes were completed on the secondary, but the rest of the replicaset (and application by extension) was not yet aware that these writes had been completed. Consequently, the secondary and old primary rolled back on fail over.

Kind regards,
Thomas

Comment by Kelsey Schubert [ 19/Apr/17 ]

Hi onyxmaster,

Thank you for the detailed report and logs, we're investigating this behavior and will update this ticket after we've finished reviewing the logs.

Kind regards,
Thomas

Generated at Thu Feb 08 04:19:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.