[SERVER-39831] Never update commit point beyond last applied if learned from sync source Created: 25/Feb/19  Updated: 29/Oct/23  Resolved: 01/Apr/19

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.6.13, 4.1.10, 4.0.10

Type: Bug Priority: Major - P3
Reporter: Siyuan Zhou Assignee: Tess Avitabile (Inactive)
Resolution: Fixed Votes: 0
Labels: RF
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-39367 lastOpCommitted being reset on restar... Closed
is related to SERVER-39626 Majority committed oplog entries may ... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.0, v3.6
Sprint: Repl 2019-03-25, Repl 2019-04-08
Participants:
Linked BF Score: 16

 Description   

SERVER-39367 ensures the correctness of commit point learning protocol, but it can cause stale secondaries to hold large history after restart since its commit point cannot advance until its oplog reaches the latest term.

We can relax the term check when learning from sync source but only update its commit point to min(commit point, my last applied). Given that requiring the term check on learning commit point ensures that the commit point is always on a node's branch, spanning tree ensures the syncing node is on the same branch as the sync source, so the syncing node knows it's on the same branch as the commit point even if they have different terms.

This ticket also fixes the issue where a secondary learns of the commit point in higher term from sync source, then syncs from another node on a diverged branch in lower term and marks the diverged branch committed by mistake. Thus this needs to be backported to earlier versions.



 Comments   
Comment by Githook User [ 17/Apr/19 ]

Author:

{'email': 'tess.avitabile@mongodb.com', 'name': 'Tess Avitabile', 'username': 'tessavitabile'}

Message: SERVER-39831 Never update commit point beyond last applied if learned from sync source

(cherry picked from commit 1c54cae8d57f6ab29c155a2a8edeb12da71c18d5)
Branch: v3.6
https://github.com/mongodb/mongo/commit/16fad378d225c19bafeb6181d532bbca1d117de4

Comment by Githook User [ 17/Apr/19 ]

Author:

{'name': 'Tess Avitabile', 'username': 'tessavitabile', 'email': 'tess.avitabile@mongodb.com'}

Message: SERVER-39831 Never update commit point beyond last applied if learned from sync source
Branch: v4.0
https://github.com/mongodb/mongo/commit/1c54cae8d57f6ab29c155a2a8edeb12da71c18d5

Comment by Githook User [ 04/Apr/19 ]

Author:

{'name': 'Tess Avitabile', 'username': 'tessavitabile', 'email': 'tess.avitabile@mongodb.com'}

Message: SERVER-39831 ephemeralForTest requires majority read concern
Branch: master
https://github.com/mongodb/mongo/commit/54fd726353b8cf6d4ba1a78a1dc7eb035d1fb39f

Comment by Githook User [ 04/Apr/19 ]

Author:

{'name': 'Tess Avitabile', 'username': 'tessavitabile', 'email': 'tess.avitabile@mongodb.com'}

Message: SERVER-39831 Add DEATH_TEST to OplogFetcherTest
Branch: master
https://github.com/mongodb/mongo/commit/4051bd3b3b4eedc8e0eb770b8b8216f28043490c

Comment by Githook User [ 02/Apr/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39831 remove failing test cases from OplogFetcherTest
Branch: master
https://github.com/mongodb/mongo/commit/f871dc86098cde8ab69286c650491cbdd637d724

Comment by Githook User [ 02/Apr/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39831 fix compile
Branch: master
https://github.com/mongodb/mongo/commit/b67030199c958be68cfb49e8d4477db145203d44

Comment by Githook User [ 01/Apr/19 ]

Author:

{'email': 'tess.avitabile@mongodb.com', 'name': 'Tess Avitabile', 'username': 'tessavitabile'}

Message: SERVER-39831 Never update commit point beyond last applied if learned from sync source
Branch: master
https://github.com/mongodb/mongo/commit/6803c64d71c1104634a9dc18e8e9d368ed6be228

Comment by Tess Avitabile (Inactive) [ 26/Feb/19 ]

Oh, got it. SERVER-39626 is about rolling back majority committed operations due to switching to a sync source that is on a different branch of history. I agree that it's more important to ensure we don't incorrectly advance the commit point beyond non-majority-committed operations, and that we don't unnecessarily keep history. SERVER-39626 is not a correctness bug, it can just lead to a crash (which we have never seen). We could consider fixing SERVER-39626 separately by making a change so that A refuses to sync from B.

Comment by Tess Avitabile (Inactive) [ 26/Feb/19 ]

siyuan.zhou, I don't follow why this doesn't fix the problem of SERVER-39626, since as you said, "spanning tree ensures the syncing node is on the same branch as the sync source".

Comment by Siyuan Zhou [ 25/Feb/19 ]

I just realized this won't fix the problem of SERVER-39626. Only term check as in SERVER-39367 can prevent SERVER-39626, but it has the problem of holding history. I feel like this one and SERVER-39626 are two contradictory goals.

If we have to choose, I'd vote for this one, since we've never seen the invariant in SERVER-39626. This is because the windows is small as explained in SERVER-39626.

Comment by Tess Avitabile (Inactive) [ 25/Feb/19 ]

siyuan.zhou, do you think I should close SERVER-39626 as a duplicate of this ticket? I intended that SERVER-39626 would be the fix on 3.4 and 3.6 and this ticket would be the fix on master and 4.0 to ensure we don't keep too much history, but since they will be the same solution, I can just close SERVER-39626.

Generated at Thu Feb 08 04:53:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.