[SERVER-65295] Handle direct CRUD ops to shards as part of a txn introducing orphan documents Created: 06/Apr/22  Updated: 29/Oct/23  Resolved: 17/May/22

Status: Closed
Project: Core Server
Component/s: Replication, Sharding
Affects Version/s: None
Fix Version/s: 6.0.0-rc6, 6.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Sergi Mateo Bellido Assignee: Antonio Fuschetto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-65859 Enable filtering of change steam even... Closed
depends on SERVER-65858 Mark the applyOps entries when the op... Closed
Related
Backwards Compatibility: Fully Compatible
Sprint: Sharding EMEA 2022-04-18, Sharding EMEA 2022-05-02, Sharding EMEA 2022-05-16, Sharding EMEA 2022-05-30
Participants:

 Description   

The code introduced in SERVER-62581 was taking care of direct inserts to shards but we forgot to cover the case in which those inserts are part of a transaction. It looks like under that scenario we take a different insert path

antonio.fuschetto@mongodb.com  discovered that the problem is not just for TXN doing inserts but for TXN doing any op BUT always directly against the shard, without going through the mongos).



 Comments   
Comment by Josef Ahmad [ 26/Apr/22 ]

The batched delete query exec stage also depends on SERVER-65858 and SERVER-65859 to avoid emitting change stream events for orphaned documents, because it groups deletes into applyOps statements much like multi-doc transactions (see SERVER-63047).

Comment by Antonio Fuschetto [ 21/Apr/22 ]

Problem description

As a consequence of PM-2423, direct write operations (performed from a client directly connected to a shard) on orphaned documents are allowed but must not generate events on change streams.

Multi-document transactions should have the same behavior, but at the moment this use case is bugged as events corresponding to individual statements are notified on change streams.

Agreed solution

The logic to skip notifications on changes stream leverages the existing fromMigrate flag of the oplog entry (that is, do not generate events when the flag is set). Unfortunately, the oplog entry corresponding to individual statements of a transaction (applyOps entries) don't have the fromMigrate flag and the change stream workflow is currently unable to skip related events.

Following a discussion with bernard.gorman@mongodb.com, we have agreed that the Sharding EMEA team will extend the applyOps entry with the fromMigrate flag where appropriate (see SERVER-65858), whereas the Query Execution team will modify the filtering as necessary across change streams to filter out these entries (see SERVER-65859).

Generated at Thu Feb 08 06:02:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.