[SERVER-35113] Stable timestamp does not advance if lastApplied does not move forward, but all committed timestamp does, on single node RS Created: 21/May/18  Updated: 29/Oct/23  Resolved: 22/May/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.6.7, 4.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Judah Schvimer Assignee: Judah Schvimer
Resolution: Fixed Votes: 0
Labels: SWNA
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-35113.diff    
Issue Links:
Backports
Depends
Problem/Incident
causes SERVER-50443 When j:false, performance degradation... Closed
is caused by SERVER-34895 Stable timestamp can be set to timest... Closed
Related
is related to SERVER-35154 Exceptions that escape a ScopedThread... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6
Sprint: Repl 2018-06-04
Participants:
Linked BF Score: 65

 Description   

SERVER-34895 made it so that the stable timestamp would never advance ahead of the all committed timestamp, so that single node primaries never storage-commit a wiredTiger transaction behind the stable timestamp.

Consider 3 operations in flight on a single node replica set, at ts 1, 2, and 3. If 1 commits first, the all committed timestamp and lastApplied and stable timestamp will all be set to 1. If 3 commits next, then the lastApplied will be set to 3, but the all committed timestamp will still be at 1, keeping the stable timestamp at 1. When 2 commits, the all committed timestamp will advance to 3, but the lastApplied will not advance, and so the stable timestamp will not advance either.

This will lead to majority writes not committing until another write comes in.



 Comments   
Comment by Githook User [ 07/Aug/18 ]

Author:

{'username': 'stbrody', 'name': 'Spencer T Brody', 'email': 'spencer@mongodb.com'}

Message: SERVER-35113 Fix storage_commit_out_of_order.js test to work with auth passthrough suites

(cherry picked from commit abb1b353648260175c3dfe02ac8ae54c083956f7)
Branch: v3.6
https://github.com/mongodb/mongo/commit/8c8148e4ed16436c2a41ab9df53ecf39be7fc8a5

Comment by Githook User [ 07/Aug/18 ]

Author:

{'username': 'judahschvimer', 'name': 'Judah Schvimer', 'email': 'judah@mongodb.com'}

Message: SERVER-35113 Allow single voting primaries to advance stable timestamp even when last applied does not advance

(cherry picked from commit 4cee07d8a97bb0663e7bfbc3f2e1fbf539140adf)
Branch: v3.6
https://github.com/mongodb/mongo/commit/57c6f90c4422ec911a4b3b038bd5338b769454aa

Comment by Githook User [ 22/May/18 ]

Author:

{'username': 'stbrody', 'name': 'Spencer T Brody', 'email': 'spencer@mongodb.com'}

Message: SERVER-35113 Fix storage_commit_out_of_order.js test to work with auth passthrough suites
Branch: master
https://github.com/mongodb/mongo/commit/abb1b353648260175c3dfe02ac8ae54c083956f7

Comment by Githook User [ 21/May/18 ]

Author:

{'username': 'judahschvimer', 'name': 'Judah Schvimer', 'email': 'judah@mongodb.com'}

Message: SERVER-35113 Allow single voting primaries to advance stable timestamp even when last applied does not advance
Branch: master
https://github.com/mongodb/mongo/commit/4cee07d8a97bb0663e7bfbc3f2e1fbf539140adf

Comment by Judah Schvimer [ 21/May/18 ]

lastApplied does not advance when 2 commits because it's already at 3, so there's nothing to advance to. It's currently acceptable for lastApplied to be ahead of uncommitted transactions, we could change that and claw lastApplied back to the all committed timestamp, but insert the actual applied optime into the stable timestamp candidates list. I'd have to think more about what potential consequences that would have.

Alternatively, we could always check if the stable timestamp can advance when we apply an operation. This would create a potential performance regression. We could track the all committed timestamp so that we only do this when necessary, or limit this to when the majority vote count is 1. This bug can only manifest on a single voting primary as far as I can tell because otherwise secondaries sending progress will advance the stable timestamp rather than the primary last applied moving forward.

Comment by Eric Milkie [ 21/May/18 ]

Alternatively, is it a bug that lastApplied is set to an invisible operation, when 3 commits?

Comment by Eric Milkie [ 21/May/18 ]

In your scenario, isn't it a bug that lastApplied does not advance when 2 commits?

Comment by Judah Schvimer [ 21/May/18 ]

Repro attached, applies on commit 6ab1592260c9b21d802aa65a11d268c0a97b11a7. SERVER-35113.diff

Generated at Thu Feb 08 04:38:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.