[SERVER-40482] incorrect fastcount for a majority committed prepared transaction that is in prepare after a rollback and then committed Created: 04/Apr/19  Updated: 29/Oct/23  Resolved: 12/Apr/19

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.1.11

Type: Bug Priority: Major - P3
Reporter: Pavithra Vetriselvan Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: prepare_durability, txn_storage
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-40566 Fix fastcount for large transaction f... Closed
Related
related to SERVER-40517 Fastcount isn't adjusted correctly wh... Closed
is related to SERVER-39762 Fix fastcount after rollback recovery... Closed
is related to SERVER-40614 Rollback errors should be fatal while... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

(function() {
    "use strict";
    load("jstests/aggregation/extras/utils.js");
    load("jstests/core/txns/libs/prepare_helpers.js");
    load("jstests/replsets/libs/rollback_test.js");
 
    const dbName = "test";
    const collName = "rollback_reconstructs_transactions_prepared_before_stable";
 
    const rollbackTest =
        new RollbackTest(dbName, undefined, true /* expect prepared transaction after rollback */);
    let primary = rollbackTest.getPrimary();
 
    // Create collection we're using beforehand.
    const testDB = primary.getDB(dbName);
    const testColl = testDB.getCollection(collName);
 
    testDB.runCommand({drop: collName});
    assert.commandWorked(testDB.runCommand({create: collName}));
 
    // Start a session on the primary.
    let session = primary.startSession();
    const sessionID = session.getSessionId();
    let sessionDB = session.getDatabase(dbName);
    let sessionColl = sessionDB.getCollection(collName);
 
    assert.commandWorked(sessionColl.insert({_id: 0}));
 
    // Prepare the transaction on the session.
    session.startTransaction();
    assert.commandWorked(sessionColl.insert({_id: 1}));
    assert.commandWorked(sessionColl.update({_id: 0}, {$set: {a: 1}}));
    const prepareTimestamp = PrepareHelpers.prepareTransaction(session);
 
    // Fastcount reflects the insert of a prepared transaction.
    assert.eq(testColl.count(), 2);
 
    jsTestLog("Do a majority write to advance the stable timestamp past the prepareTimestamp");
    // Doing a majority write after preparing the transaction ensures that the stable timestamp is
    // past the prepare timestamp because this write must be in the committed snapshot.
    assert.commandWorked(
        testColl.runCommand("insert", {documents: [{_id: 2}]}, {writeConcern: {w: "majority"}}));
 
    // Fastcount reflects the insert of a prepared transaction.
    assert.eq(testColl.count(), 3);
 
    // Check that we have one transaction in the transactions table.
    assert.eq(primary.getDB('config')['transactions'].find().itcount(), 1);
 
    // The transaction should still be prepared after going through rollback.
    rollbackTest.transitionToRollbackOperations();
    rollbackTest.transitionToSyncSourceOperationsBeforeRollback();
    rollbackTest.transitionToSyncSourceOperationsDuringRollback();
    rollbackTest.transitionToSteadyStateOperations();
 
    // Make sure there is still one transactions in the transactions table. This is because the
    // entry in the transactions table is made durable when a transaction is prepared.
    assert.eq(primary.getDB('config')['transactions'].find().itcount(), 1);
 
    // TODO: Fix fastcount. Fastcount will return 2 instead of 3. This is inconsistent with the
    // behavior of fastcount observing the insert at line 38.
    assert.eq(testColl.count(), 3);
 
    // Make sure we cannot see the writes from the prepared transaction yet.
    arrayEq(testColl.find().toArray(), [{_id: 0}, {_id: 2}]);
 
    // Get the correct primary after the topology changes.
    primary = rollbackTest.getPrimary();
    rollbackTest.awaitReplication();
 
    // Make sure we can successfully commit the recovered prepared transaction.
    session =
        PrepareHelpers.createSessionWithGivenId(primary, sessionID, {causalConsistency: false});
    sessionDB = session.getDatabase(dbName);
    // The transaction on this session should have a txnNumber of 0. We explicitly set this
    // since createSessionWithGivenId does not restore the current txnNumber in the shell.
    session.setTxnNumber_forTesting(0);
    const txnNumber = session.getTxnNumber_forTesting();
 
    // Make sure we cannot add any operations to a prepared transaction.
    assert.commandFailedWithCode(sessionDB.runCommand({
        insert: collName,
        txnNumber: NumberLong(txnNumber),
        documents: [{_id: 10}],
        autocommit: false,
    }),
                                 ErrorCodes.PreparedTransactionInProgress);
 
    // Make sure that writing to a document that was updated in the prepared transaction causes
    // a write conflict.
    assert.commandFailedWithCode(
        sessionDB.runCommand(
            {update: collName, updates: [{q: {_id: 0}, u: {$set: {a: 2}}}], maxTimeMS: 5 * 1000}),
        ErrorCodes.MaxTimeMSExpired);
 
    // Commit the transaction.
    assert.commandWorked(sessionDB.adminCommand({
        commitTransaction: 1,
        commitTimestamp: prepareTimestamp,
        txnNumber: NumberLong(txnNumber),
        autocommit: false,
    }));
 
    // Make sure we can see the effects of the prepared transaction.
    arrayEq(testColl.find().toArray(), [{_id: 0, a: 1}, {_id: 1}, {_id: 2}]);
    // TODO: Fix fastcount. Fastcount will return 2 instead of 3.
    assert.eq(testColl.count(), 3);
 
    rollbackTest.stop();
}());

Sprint: Storage NYC 2019-04-08, Storage NYC 2019-04-22
Participants:
Linked BF Score: 0

 Description   

This came out of SERVER-39689, where we test that a transaction prepared before the stable timestamp will still be in prepare after a rollback. When we commit this transaction, the counts do not reflect the insert of the prepared transaction.

My understanding is that we add to the count when we put a transaction into prepare. We then subtract from the count when we abort the storage transaction of a prepared transaction during rollback.

We do not change the counts when reconstructing the prepared transaction during recovery. This is not a problem if the prepared transaction is eventually aborted. However, if this transaction is eventually committed, the counts are incorrect.



 Comments   
Comment by Githook User [ 12/Apr/19 ]

Author:

{'name': 'Louis Williams', 'username': 'louiswilliams', 'email': 'louis.williams@mongodb.com'}

Message: SERVER-40482 fix dbtests
Branch: master
https://github.com/mongodb/mongo/commit/afcb670f36b9cabda26d719c81f3991b9b2ed5b8

Comment by Githook User [ 12/Apr/19 ]

Author:

{'name': 'Louis Williams', 'username': 'louiswilliams', 'email': 'louis.williams@mongodb.com'}

Message: SERVER-40482 SERVER-40517 Fix fastcount algorithm for rollback of prepared transactions

This fixes two bugs, both related the correctness of the algorithm for adjusting collection counts
during rollback. The first bug is that rolled-back non-majority confirmed "prepare" oplog entries
may rollback and incorrectly adjust collection fastcounts. The second bug is that a prepared and
committed transaction will have incorrect collection counts after rollback.

The new high-level order of operations during replication rollback are as follows:
1. Abort all active prepared transactions, rolling back any in-memory counts
2. Calculate collection count adjustments by scanning rolled-back oplog entries
3. If a 'commitTransaction' oplog entry is rolled-back, find the associated 'prepare' to calculate
size adjustments
4. Rollback to the stable timestamp. Replay oplog to common point. This makes no collection count
adjustments.
5. Set collection counts to previously calculated values
6. Reconstruct prepared transactions, which updates in-memory fastcounts
Branch: master
https://github.com/mongodb/mongo/commit/8c2ef8757dfe625e7fc06c3cdef6b7692764d00c

Comment by Louis Williams [ 08/Apr/19 ]

By "after rollback" do you mean after "recover to a timestamp" but before we reconstruct prepared transactions?

Yes, the idea is that after recovery to the stable timestamp, all prepared transactions regardless of the commit/abort decision have rolled-back counts. From that point, the in-memory fastcounts are reconstructed with each prepared transaction and updated accordingly as they commit or abort.

As far as SERVER-40517 goes, since that requires reordering certain operations, I'm going to shift my focus there first.

Comment by Judah Schvimer [ 08/Apr/19 ]

Rollback aborts and rolls-back a prepared transaction that not been committed (but will be).

If prepare is rolled back, then we're fine since we aborted the prepared transaction. I think the problem here is if the stable timestamp is behind the prepare oplog entry, but the common point is after the prepare oplog entry (prepare does not get rolled back) so the transaction gets re-prepared without changing fastcount.

When rolling-back "commitTransaction" oplog entries, find the previous "prepare" entry and use that to subtract the correct collection counts. After rollback, all prepared transactions will be in the same state, with rolled-back fastcounts.

I'm confused by this solution. By "after rollback" do you mean after "recover to a timestamp" but before we reconstruct prepared transactions? From your previous sentence it seemed to me that after prepared transactions are reconstructed they would have fastcounts in them.

I do think that if my understanding is correct that this (1) is the simpler solution. It returns us to a state where _countDiffs is complete and the only logical change from 4.0 lives in the "reconstructing prepared transactions" logic, which is already new.

Before moving ahead, I'm interested in if SERVER-40517's solution will play well with this.

Comment by Louis Williams [ 05/Apr/19 ]

The issue here describes incorrect counts in the following cases:

  • Rollback aborts but does not roll-back a prepared transaction that has not been committed (but will be).
    • i.e. the stable timestamp is ahead of the prepare oplog entry
  • Rollback aborts and rolls-back a prepared transaction that not been committed (but will be).
    • i.e. the stable timestamp is behind the prepare oplog entry

In either case, aborting a prepared transaction before a commit or abort decision has been received means that a prepared transaction will roll-back counts even though the commit/abort decision is unknown. During recovery, no new in-memory fastcount is added, so a future commit will not update appropriately either.

The solution I have is this: while prepared transactions are reconstructed after oplog replay, make the operations count toward collection size adjustments by setting this flag to "false". It will then be necessary to also do one of the following beforehand:

  1. When rolling-back "commitTransaction" oplog entries, find the previous "prepare" entry and use that to subtract the correct collection counts. After rollback, all prepared transactions will be in the same state, with rolled-back fastcounts.
  2. Record when "commitTransaction" oplog entries are rolled-back for prepared transactions (a set/map of some sort), which requires finding a previous oplog entry. Don't apply collection count adjustment if the transaction has already been committed (by checking this set/map). This will prevent double-counting transactions that have been committed.

Both solutions require processing "commitTransaction" operations and traversing back in the oplog to find the associated "prepare" entry. I think the first is a little more simple and doesn't require an additional data structure. judah.schvimer let me know what you think of this approach or if there is something I've missed.

Comment by Judah Schvimer [ 04/Apr/19 ]

SERVER-39762 did not account for the case where a commit is not actually rolled back, but is simply replayed during replication recovery.

Generated at Thu Feb 08 04:55:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.