[SERVER-39021] Switch migrations to observe multi-statement transaction CRUD statements onCommit instead of onCRUD Created: 15/Jan/19  Updated: 29/Oct/23  Resolved: 21/Feb/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.1.6
Fix Version/s: 4.1.9

Type: Task Priority: Major - P3
Reporter: Blake Oler Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-39017 Allow prepared transaction statements... Closed
is depended on by SERVER-38284 Remove donor collection X-lock acquis... Closed
is depended on by SERVER-38828 Replace uses of UninterruptibleLockGu... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2019-01-28, Sharding 2019-02-11, Sharding 2019-02-25
Participants:

 Description   

Problem Summary

We are proposing removing a collection X-lock in the MigrationSourceManager. This X-lock creates a barrier after which all active transactions on the collection have been committed. This barrier ensures that the migration's chunk cloner will not miss writes that started before the migration.

Without this barrier, the cloner will miss transaction writes that start before a migration starts. These writes will not be registered under the LogOpForShardingHandler.

Proposed Approach

Because of this inconsistency, we would like to switch how migrations handle observing CRUD operations in multi-document transactions.

After the change in SERVER-39017, we will be able to observe all transaction statements on commit.Accordingly, on the op observer onCommit handler, we can pass these statements to a new method on the OpObserverShardingImpl class. This new method is defined in the next paragraph.

A new method shardObserveTransactionCommit placed on the OpObserverShardingImpl class will do the following:

  • Verify shard version (parity with other CRUD op observers).
  • Lock the CollectionShardingRuntimeLock (parity with other CRUD op observers).
  • Verify the existence of a migration (parity with other CRUD op observers).
  • Iterate over transaction statements and apply each CRUD op individually.
  • Verify that the chunk has not yet moved if in transaction (parity with other CRUD op observers).

The previously stated iteration process will do the following with a vector of ReplOperations.

NOTE: We previously stated that we needed an iterator class to analyze the list of replOperations. However, the iteration process is actually very simple – for now, I think an anonymous function in the same file as the OpObserverShardingImpl class will be sufficient.

As a result of the previously stated work, we will now be duplicating ops sent to the shard observer – we will be observing every individual CRUD operation, as well as those same operations if they are also inside a multi statement transaction. We don't want the migration to see the same write twice (since these updates are idempotent, it wouldn't be incorrect, just unnecessary). We should only send writes to the cloner if we are not in a multi statement transaction, and a simple check of an already-given boolean will do that.



 Comments   
Comment by Githook User [ 21/Feb/19 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-39021 Switch migrations to observe multi-statement transaction CRUD statements onCommit instead of onCRUD
Branch: master
https://github.com/mongodb/mongo/commit/42999ae47bacedf260be2d6dd5e377ff0ae63744

Comment by Pavithra Vetriselvan [ 06/Feb/19 ]

blake.oler Yes, feel free to CC me on this review!

Comment by Blake Oler [ 06/Feb/19 ]

As part of this ticket, we'd like to change prepared transactions from:

  1. Committing to storage
  2. Observing transaction commit

to

  1. Observing transaction commit
  2. Committing to storage

So that we can guarantee that we will be in a WriteUnitOfWork during observing the transaction commit. This will match the contract for op observers that observing always happens before storage commits. pavithra.vetriselvan gave the okay to change this ordering.

pavithra.vetriselvan@mongodb.com shall I add you as a reviewer to my change for this ticket?

Comment by Randolph Tan [ 15/Jan/19 ]

lgtm

Comment by Blake Oler [ 15/Jan/19 ]

kaloian.manassiev or renctan, could I get an LGTM on this new approach?

Generated at Thu Feb 08 04:50:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.