[SERVER-11103] replset mutex should be not be held during DB lock attempts Created: 09/Oct/13  Updated: 11/Jul/16  Resolved: 28/Oct/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.4.6
Fix Version/s: 2.8.0-rc0

Type: Bug Priority: Major - P3
Reporter: Alexander Komyagin Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: elections
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File block_hearbeat.js    
Issue Links:
Related
related to SERVER-11059 Elections can be delayed by some locks Closed
Tested
Operating System: ALL
Participants:

 Description   

Result: Elections will not proceed correctly until heartbeats are processed.

Repro:

  1. fsyncLock() secondary (XXX:27019)
  2. raise the priority for the PRIMARY (XXX:27017) and issue rs.reconfig()
  3. shut down the PRIMARY
  4. observe that another SECONDARY (XXX:27018) cannot be elected since the fsyncLock'ed one vetoes it:

    Wed Oct  9 11:58:21.339 [rsMgr] not electing self, XXX:27019 would veto with 'XXX:27018 is trying to elect itself but XXX:27017 is already primary and more up-to-date'

  5. fsyncUnlock() the SECONDARY ( XXX:27019 ), observe the XXX:27018 being elected

Expected behavior:
Write locks should not prevent an election or delay detecting replica set node state changes. This message is an example:

Wed Oct  9 11:51:01.864 [rsHealthPoll] replSet member XXX:27017 is now in state DOWN

Ideally, XXX:27019 should vote for XXX:27018 (see SERVER-11059).



 Comments   
Comment by Githook User [ 28/Oct/14 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-11103 test that fsyncLocked secondaries will still participate in elections
Branch: master
https://github.com/mongodb/mongo/commit/4d8aa08ef8b45a9d62dff13b96b210d30099fa7d

Comment by Scott Hernandez (Inactive) [ 20/Oct/14 ]

This fails to elect primary due to a config version mismatch, as this just repeats:

 m31001| 2014-10-20T15:39:44.764-0400 W NETWORK  Failed to connect to 10.4.100.53:31000 after 5000 milliseconds, giving up.
 m31002| 2014-10-20T15:39:44.764-0400 W NETWORK  Failed to connect to 10.4.100.53:31000 after 5000 milliseconds, giving up.
 m31001| 2014-10-20T15:39:44.764-0400 I REPLSETS Standing for election
 m31001| 2014-10-20T15:39:44.764-0400 I REPLSETS replSet info electSelf
 m31002| 2014-10-20T15:39:44.765-0400 I REPLSETS replSetElect not voting because our config version is stale. Our version: 1, their version: 2
 m31001| 2014-10-20T15:39:44.765-0400 I REPLSETS replSet couldn't elect self, only received 1 votes, but needed at least 2

This test is flawed since it blocks writing the new config version which causes the problem of having an old config version for voting. I put up the start of a better test, but haven't finished... and now mattd@10gen.com is taking over

Comment by Eric Milkie [ 17/Oct/14 ]

The code now works as per Expected Behavior in the description above. However, the provided test still fails because a node is not allowed to reconfig while fsync-and-locked, as per Matt's comment above.

Comment by Eric Milkie [ 28/Aug/14 ]

We need to confirm the attached jstest succeeds after the refactor is complete.

Comment by Matt Dannenberg [ 16/Jan/14 ]

I did a bit of digging around and it looks like the problem is that to save the new config (saveConfigLocally in rs_config.cpp) the node needs to acquire a WriteContext on the "local.system.replset" collection, which it cannot get due to being fsync and locked.

Comment by Alexander Komyagin [ 14/Oct/13 ]

Attaching jstest that reproduces the issue

Generated at Thu Feb 08 03:24:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.