[SERVER-12264] replicaset nodes appear to be down during compact Created: 06/Jan/14  Updated: 11/Jul/16  Resolved: 13/Jan/14

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: 2.4.8, 2.5.4
Fix Version/s: 2.4.10, 2.5.5

Type: Bug Priority: Major - P3
Reporter: Matt Dannenberg Assignee: Matt Dannenberg
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-12359 Compacting blocks "admin" db authenti... Closed
is related to SERVER-12727 index building can make replica set m... Closed
Operating System: ALL
Participants:

 Description   
Issue Status as of March 31, 2014

ISSUE SUMMARY

During a compact operation, heartbeats to and from the node fail and the node can appear to be offline to the rest of the replica set ("... thinks that we are down..." log messages). This is because a global write lock is taken before checking if the node is running a compact.

USER IMPACT

Running compact makes the node appear to be down to other members, which can affect majority concerns in a replica set (write concern, voting, etc). Running compact on several secondaries in one replica set can cause loss of majority and a step-down of the primary node, rendering the replica set read-only.

SOLUTION

To fix the issue, the global write lock is not acquired if the node is running compact or other maintenance tasks.

WORKAROUNDS

Run compact only on one secondary at a time for each replica set, making sure that the quorum is not affected. The secondary can temporarily be replaced with an arbiter node to ensure high availability.

AFFECTED VERSIONS

All recent production release versions up to 2.4.9 are affected.

PATCHES

The fix is included in the 2.4.10 production release and the 2.5.5 development version, which will evolve into the 2.6.0 production release.

Original Description

Heartbeats fail in both directions on the node running compact().

On the send side of a heartbeat, the sender need to up() the receiver node, which calls addToElectable() set which needs the rsbase mutex. The rsbase mutex is held by the syncthread as it tryToGoLiveAsSecondary()s, but the syncthread has to wait on a global write lock which it can't get because compact is holding a write lock.

On the receiving side, the issue is the exact same only the receiver is looking for the rsbase mutex in iAmElectable().

It might be possible to make tryToGoLiveAsSecondary() return prior to the global writeLock request if we are running compact.



 Comments   
Comment by Githook User [ 10/Mar/14 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-12264 prevent from tryToGoLiveAsSecondary() from waiting on global writelock during compact
Branch: v2.4
https://github.com/mongodb/mongo/commit/435f6118c5c3f2515970c65aab53f59bb54f2982

Comment by Githook User [ 13/Jan/14 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-12264 prevent from tryToGoLiveAsSecondary() from waiting on global writelock during compact
Branch: master
https://github.com/mongodb/mongo/commit/939fa9c7460236b7bab9c7704096bcd240503389

Generated at Thu Feb 08 03:28:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.