[SERVER-35113] Stable timestamp does not advance if lastApplied does not move forward, but all committed timestamp does, on single node RS Created: 21/May/18 Updated: 29/Oct/23 Resolved: 22/May/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | 3.6.7, 4.0.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Judah Schvimer | Assignee: | Judah Schvimer |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | SWNA | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||
| Backport Requested: |
v3.6
|
||||||||||||||||||||||||||||
| Sprint: | Repl 2018-06-04 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Linked BF Score: | 65 | ||||||||||||||||||||||||||||
| Description |
|
Consider 3 operations in flight on a single node replica set, at ts 1, 2, and 3. If 1 commits first, the all committed timestamp and lastApplied and stable timestamp will all be set to 1. If 3 commits next, then the lastApplied will be set to 3, but the all committed timestamp will still be at 1, keeping the stable timestamp at 1. When 2 commits, the all committed timestamp will advance to 3, but the lastApplied will not advance, and so the stable timestamp will not advance either. This will lead to majority writes not committing until another write comes in. |
| Comments |
| Comment by Githook User [ 07/Aug/18 ] |
|
Author: {'username': 'stbrody', 'name': 'Spencer T Brody', 'email': 'spencer@mongodb.com'}Message: (cherry picked from commit abb1b353648260175c3dfe02ac8ae54c083956f7) |
| Comment by Githook User [ 07/Aug/18 ] |
|
Author: {'username': 'judahschvimer', 'name': 'Judah Schvimer', 'email': 'judah@mongodb.com'}Message: (cherry picked from commit 4cee07d8a97bb0663e7bfbc3f2e1fbf539140adf) |
| Comment by Githook User [ 22/May/18 ] |
|
Author: {'username': 'stbrody', 'name': 'Spencer T Brody', 'email': 'spencer@mongodb.com'}Message: |
| Comment by Githook User [ 21/May/18 ] |
|
Author: {'username': 'judahschvimer', 'name': 'Judah Schvimer', 'email': 'judah@mongodb.com'}Message: |
| Comment by Judah Schvimer [ 21/May/18 ] |
|
lastApplied does not advance when 2 commits because it's already at 3, so there's nothing to advance to. It's currently acceptable for lastApplied to be ahead of uncommitted transactions, we could change that and claw lastApplied back to the all committed timestamp, but insert the actual applied optime into the stable timestamp candidates list. I'd have to think more about what potential consequences that would have. Alternatively, we could always check if the stable timestamp can advance when we apply an operation. This would create a potential performance regression. We could track the all committed timestamp so that we only do this when necessary, or limit this to when the majority vote count is 1. This bug can only manifest on a single voting primary as far as I can tell because otherwise secondaries sending progress will advance the stable timestamp rather than the primary last applied moving forward. |
| Comment by Eric Milkie [ 21/May/18 ] |
|
Alternatively, is it a bug that lastApplied is set to an invisible operation, when 3 commits? |
| Comment by Eric Milkie [ 21/May/18 ] |
|
In your scenario, isn't it a bug that lastApplied does not advance when 2 commits? |
| Comment by Judah Schvimer [ 21/May/18 ] |
|
Repro attached, applies on commit 6ab1592260c9b21d802aa65a11d268c0a97b11a7. SERVER-35113.diff |