[SERVER-44771] Allow operations in transactions to safely consult the CatalogCache on mongod Created: 21/Nov/19  Updated: 29/Oct/23  Resolved: 14/Feb/20

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Sharding
Affects Version/s: None
Fix Version/s: 4.3.4

Type: Improvement Priority: Major - P3
Reporter: Charlie Swanson Assignee: Tommaso Tocci
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-46131 Fix race in retry loops in MigrationS... Closed
is depended on by SERVER-45542 Support and test $unionWith aggregati... Backlog
Duplicate
is duplicated by SERVER-32724 Error if $lookup or $out needs to con... Closed
Problem/Incident
Related
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2019-12-30, Sharding 2020-01-13, Sharding 2020-01-27, Sharding 2020-02-10, Sharding 2020-02-24
Participants:
Linked BF Score: 0

 Description   

When brainstorming the design for $unionWith, we are concerned about an issue we have previously run into in such a scenario. esha.maharishi sums up a possible deadlock that can occur when a shard tries to look at the CatalogCache while still holding a lock in this comment. It's also described in SERVER-37398 where we added this invariant to ensure this never happens. This was discovered when working with improved $out (now branded as $merge), but it has implications for sharded $lookup and now $unionWith.

This ticket tracks a feature request to remove that invariant. It should be possible to query a different shard using the CatalogCache while you are in a transaction (and thus hold some locks which you cannot drop). Currently you cannot do this because it may deadlock.



 Comments   
Comment by Githook User [ 14/Feb/20 ]

Author:

{'username': 'toto-dev', 'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com'}

Message: SERVER-44771 Introduce 2nd CatalogCache for filtering only

Only on ShardServer (not ReadOnly) use a second catalog cache for
filtering only in order to allow operations in transactions to safely
consult the CatalogCache.
Branch: master
https://github.com/mongodb/mongo/commit/c25f0bc16a1f515f20972f30abf6bd569235a00f

Comment by Tommaso Tocci [ 13/Feb/20 ]

Blocked by SERVER-46131

Comment by Githook User [ 08/Feb/20 ]

Author:

{'username': 'will62794', 'name': 'William Schultz', 'email': 'william.schultz@mongodb.com'}

Message: Revert "SERVER-44771 Introduce 2nd CatalogCache for filtering only"

This reverts commit 3473d09fa9b5e418e119941b01c88863744689cc.
Branch: master
https://github.com/mongodb/mongo/commit/4ae67479e73174a75f2d7141360edb8a8ef90be8

Comment by Githook User [ 07/Feb/20 ]

Author:

{'name': 'Tommaso Tocci', 'username': 'toto-dev', 'email': 'tommaso.tocci@mongodb.com'}

Message: SERVER-44771 Introduce 2nd CatalogCache for filtering only

Only on ShardServer (not ReadOnly) use a second catalog cache for
filtering only in order to allow operations in transactions to safely
consult the CatalogCache.
Branch: master
https://github.com/mongodb/mongo/commit/3473d09fa9b5e418e119941b01c88863744689cc

Comment by Githook User [ 06/Feb/20 ]

Author:

{'name': 'Tommaso Tocci', 'username': 'toto-dev', 'email': 'tommaso.tocci@mongodb.com'}

Message: Revert "SERVER-44771 Introduce 2nd CatalogCache for filtering only"

This reverts commit 84bdc9478401355656ab7eaec745dbae2e4294cb.
Branch: master
https://github.com/mongodb/mongo/commit/7a4ea53a34703a51186c9d1ffffa31cf566ef11c

Comment by Githook User [ 06/Feb/20 ]

Author:

{'username': 'toto-dev', 'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com'}

Message: SERVER-44771 Introduce 2nd CatalogCache for filtering only

Only on ShardServer (not ReadOnly) use a second catalog cache for
filtering only in order to allow operations in transactions to safely
consult the CatalogCache.
Branch: master
https://github.com/mongodb/mongo/commit/84bdc9478401355656ab7eaec745dbae2e4294cb

Generated at Thu Feb 08 05:06:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.