[SERVER-56631] Retryable write pre-fetch phase could miss entry from config.transactions when reading from donor secondaries Created: 04/May/21  Updated: 29/Oct/23  Resolved: 08/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.0-rc1, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Lingzhi Deng Assignee: Wenbin Zhu
Resolution: Fixed Votes: 0
Labels: pm-1791_non-cloud-blocking, pm-1791_other_required, post-rc0
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-55214 Resharding txn cloner can miss config... Closed
related to SERVER-55305 Retryable write may execute more than... Closed
related to SERVER-55578 Disallow atClusterTime reads on the c... Closed
is related to SERVER-57345 Retryable write pre-fetch missing doc... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Repl 2021-05-17, Repl 2021-05-31, Repl 2021-06-14
Participants:

 Description   

SessionUpdateTracker::_updateSessionInfo() is used by secondary oplog application to coalesce multiple updates to the same config.transactions record into a single update of the most recent retryable write statement. After SERVER-47844, majority snapshot of a secondary is allowed to exist somewhere in the middle of a completed batch. So secondary majority reads on config.transactions may not reflect committed retryable writes at that majority commit point.

For example, we could have a batch on a donor secondary like this:
TS1 txn1 stmtId 1
TS2 txn1 stmtId 2
TS5 txn1 stmtId 3
TS10 txn1 stmtId 4
After applying this batch, the secondary does a single update to the config.transaction table using TS10 and the transaction entry will have lastWriteOpTime at 10. When the retryable write pre-fetch phase fetches from the donor secondary, the majority committed snapshot could be at TS5. If that's the case, reading with TS5 would not see the config.transactions entry written at TS10. Let's assume the recipient's startFetchingTimestamp is also TS5, the recipient would then end up missing stmtId 1 and 2 after the migration because the pre-fetch phase misses the transaction record written at TS10. (See also SERVER-55305 and SERVER-55578)

This ticket should verify this behavior. One idea to fix this is to read with local read concern (which would always at lastApplied/last batch boundary on secondaries) and then wait for the operationTime returned to be majority committed on the donor, similar to what we did on the cloners.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 09/Jun/21 ]

Author:

{'name': 'Wenbin Zhu', 'email': 'wenbin.zhu@mongodb.com', 'username': 'WenbinZhu'}

Message: SERVER-56631 Make sure retryable write pre-fetch phase can see the config.transactions record when committed snapshot is not at batch boundary.

(cherry picked from commit 18f91c4304086b0334d90c1d94a7d3c7225439bf)
Branch: v5.0
https://github.com/mongodb/mongo/commit/047e62422aeb20f6a5bd476ed970d4deb8235237

Comment by Githook User [ 08/Jun/21 ]

Author:

{'name': 'Wenbin Zhu', 'email': 'wenbin.zhu@mongodb.com', 'username': 'WenbinZhu'}

Message: SERVER-56631 Make sure retryable write pre-fetch phase can see the config.transactions record when committed snapshot is not at batch boundary.
Branch: master
https://github.com/mongodb/mongo/commit/18f91c4304086b0334d90c1d94a7d3c7225439bf

Comment by Wenbin Zhu [ 02/Jun/21 ]

I have verified and reproduced this issue, will start working on the fix.

Generated at Thu Feb 08 05:39:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.