[SERVER-46373] Allow an aggregation sub-operation on a shard to target itself when running in a transaction Created: 24/Feb/20  Updated: 02/Jan/24

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Nicholas Zolnierz Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 0
Labels: pm3229-m1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-46378 Support continuing a transaction star... Closed
is depended on by SERVER-48122 Support $unionWith in a transaction Backlog
is depended on by SERVER-45542 Support and test $unionWith aggregati... Backlog
Related
related to SERVER-84470 Allow $lookup to target an unsplittab... Blocked
related to SERVER-39162 Allow $lookup in sharded transaction Backlog
Assigned Teams:
Query Optimization
Participants:

 Description   

Certain aggregation stages involve sub-operations or sub-pipelines, which may involve targetting and reading from sharded collections when running on a shard (e.g. $lookup, $unionWith). Supporting such stages in a transaction would cause a deadlock on the session mutex if the sub-operation targets the same node its running on, similar to what is described in SERVER-33683. Note that SERVER-33683 allows a $mergeCursors to run on a shard in a txn, but does not support the case when a stage starts its own sub-operation.

For this ticket, one possible solution that was discussed would be to use the resource yielder that was built in SERVER-33683 within the MultiStatementTransactionRequestsSender if one of the remote shards involved happens to be itself.



 Comments   
Comment by Jordi Serra Torrens [ 14/Nov/23 ]

One thing to note is that the original issue described in this ticket (a shard deadlocking when remote-targeting itself) has been almost solved by SERVER-63495, that made MultiStatementTransactionRequestsSender use a ResourceYielder when sending remote commands. I believe we only need to tweak it so that when executing on a shard, it uses a TransactionParticipantResourceYielder (which also stashes the participant's transaction resources) instead of a TransactionRouterResourceYielder (which only yields the checked out session).

However, the problem that remains is how to make a shard be able target other shards which may not yet be a participant of the transactions. The shard must make sure to start the transaction on the new participant passing it the proper transaction metadata (readConcern, placementConflictConcern). Also, the shard must report back the new additional participant to the TransactionRouter (on mongos, typically). This work is expected to be done by PM-2844.

Comment by David Storch [ 04/Aug/23 ]

This is something that we believe we will need to do as part of "remove query reliance on db primary". As part of this project, we will create situations where the merging pipeline has a $lookup that needs to target a shard which is already participating in the transaction.

Comment by Charlie Swanson [ 30/Mar/20 ]

Marking this as depending on SERVER-46378 because that ticket is believed to be much larger and both are dependencies for SERVER-45542. The query team can pick this one up if/when SERVER-46378 is resolved so we can unblock $unionWith in a transaction.

Generated at Thu Feb 08 05:11:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.