[SERVER-48327] [4.4] Index build abort can set ghost timestamps on primaries Created: 20/May/20  Updated: 29/Jan/21  Resolved: 29/Jan/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Louis Williams
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-47867 Remove ghost timestamping code Closed
Operating System: ALL
Steps To Reproduce:

Apply this patch:

diff --git a/src/mongo/db/catalog/multi_index_block.cpp b/src/mongo/db/catalog/multi_index_block.cpp
index f08446efb6..d2a94dac78 100644
--- a/src/mongo/db/catalog/multi_index_block.cpp
+++ b/src/mongo/db/catalog/multi_index_block.cpp
@@ -115,6 +115,8 @@ void MultiIndexBlock::abortIndexBuild(OperationContext* opCtx,
                 repl::UnreplicatedWritesBlock uwb(opCtx);
                 if (!IndexTimestampHelper::setGhostCommitTimestampForCatalogWrite(opCtx, nss)) {
                     LOGV2(20382, "Did not timestamp index abort write.");
+                } else {
+                    LOGV2_FATAL(0, "Wrote ghost timestamp for index abort write.");
                 }
             }

And run:

resmoke run --suite=no_passthrough jstests/noPassthrough/index_stepdown_abort_prepare_conflict.js

Sprint: Execution Team 2020-06-01, Execution Team 2020-06-15, Execution Team 2021-02-08
Participants:
Linked BF Score: 22

 Description   

When aborting an index build, we determine whether or not to write a ghost timestamp through a call to "getCommitTimestamp" here, however this only returns a timestamp on secondaries.

On primaries, this is always false, and results in a call to RecoveryUnit::setTimestamp.

This is immediately followed by a call to write an abortIndexBuild oplog entry via the onCleanUp handler. This overwrites and updates the commit timestamp, so the catalog write ends up being timestamped correctly.

TLDR: I believe this can temporarily cause the WiredTiger "all_durable" value to move backwards, which violates the contract "all_durable" on primaries.

Say, for example, there are no open WT transactions. Both lastApplied and "all_durable" all 10. If I open a WT transaction and set its commit timestamp to 10 (a timestamp that has already been committed), "all_durable" will move backwards to 9. After the transaction commits, "all_durable" can advance back to 10.



 Comments   
Comment by Louis Williams [ 03/Jun/20 ]

I am closing this ticket in favor of SERVER-48565, which would prevent readers from observing the all_durable timestamp move backward.

At this point, ghost timestamping on primaries is only known to cause this one bug. In 4.5 we plan on removing ghost timestamps, and in 4.4, it will be hard to entirely remove ghost timestamps because of single-phase behavior in FCV 4.2.

Comment by Louis Williams [ 02/Jun/20 ]

Because FCV 4.2 single-phase index builds require ghost timestamps, this bug is not easily fixable. This ticket will only address the ghost timestamping issues for 2-phase index builds in FCV 4.4.

Comment by Louis Williams [ 02/Jun/20 ]

This ticket made unnecessary on 4.5+ by SERVER-47867.

Comment by Louis Williams [ 20/May/20 ]

This ticket should also backport this invariant to the 4.4 branch.

Generated at Thu Feb 08 05:16:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.