[SERVER-79217] Race in PrimaryOnlyServiceMongoDTest::stepUp causes invariant failure Created: 21/Jul/23  Updated: 16/Nov/23  Resolved: 03/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0, 7.0.5

Type: Bug Priority: Major - P3
Reporter: Patrick Freed Assignee: Patrick Freed
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0
Sprint: Service Arch 2023-07-24, Service Arch 2023-08-07
Participants:
Linked BF Score: 6

 Description   

PrimaryOnlyServiceMongoDTest::stepUp artificially advances the last applied optime by incrementing the term but keeping the same timestamp. Reading the existing timestamp and updating it are done non-atomically, so if there are operations in the background that could modify the last applied op time, then it is possible that stepUp could try to set the op time to an older time, triggering an invariant failure.

PrimaryOnlyServiceTest runs into this race because the _rebuildService implementation in the service it uses (TestService) builds an index. _rebuildService is invoked asynchronously via opStepUpComplete, so its possible that a test case could begin while this index build is happening in the background.

There are several test cases that may fail due to this (e.g. ShutdownDuringStepUp and StateTransitionFromRebuildingShouldWakeUpConditionVariable).



 Comments   
Comment by Githook User [ 11/Nov/23 ]

Author:

{'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}

Message: SERVER-79217 Wait for step up to fully complete in PrimaryOnlyService tests

(cherry picked from commit 220f515a6e6aac4b8d6e79ed917c4816522475a4)
Branch: v7.0
https://github.com/mongodb/mongo/commit/2477e351ad0049c224d0dbaa698d135b53338a32

Comment by Githook User [ 03/Aug/23 ]

Author:

{'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}

Message: SERVER-79217 Wait for step up to fully complete in PrimaryOnlyService tests
Branch: minh.luu-no_compile_sys-perf
https://github.com/mongodb/mongo/commit/220f515a6e6aac4b8d6e79ed917c4816522475a4

Comment by Githook User [ 02/Aug/23 ]

Author:

{'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}

Message: SERVER-79217 Wait for step up to fully complete in PrimaryOnlyService tests
Branch: master
https://github.com/mongodb/mongo/commit/220f515a6e6aac4b8d6e79ed917c4816522475a4

Generated at Thu Feb 08 06:40:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.