[SERVER-46105] Allow new collection creation inside prepared transactions. Created: 12/Feb/20  Updated: 08/Nov/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Maria van Keulen Assignee: Backlog - Catalog and Routing
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-81037 Cluster the local durable catalog or ... Backlog
Related
is related to SERVER-45945 Prohibit collection/index creation in... Closed
is related to SERVER-81498 Internally allow creation of new coll... Closed
Assigned Teams:
Catalog and Routing
Sprint: Execution Team 2021-01-11, Execution EMEA Team 2023-08-07, Execution EMEA Team 2023-08-21, Execution EMEA Team 2023-09-04, Execution EMEA Team 2023-09-18, Execution EMEA Team 2023-10-02, Execution EMEA Team 2023-10-16, Execution EMEA Team 2023-10-30
Participants:

 Description   

SERVER-45945 describes known wrinkles with this presently. The solution should take these into consideration.



 Comments   
Comment by Josef Ahmad [ 14/Sep/23 ]

I am marking this ticket as dependent on SERVER-81037, which is a placeholder for a proposal to migrate most of the catalog concurrency control to the storage engine.

Comment by Josef Ahmad [ 30/Aug/23 ]

After consulting with gregory.wlodarek@mongodb.com, it turns out that creating a collection inside a prepared transaction needs additional local catalog work. Today, the catalog only detects conflicting uncommitted catalog updates for unprepared transactions. For prepared transactions, the conflict detection must happen at prepare time rather than at commit time; that also requires handling prepare conflicts with concurrent uncommitted catalog updates, which I'm unsure the catalog is capable of today.

Comment by Josef Ahmad [ 28/Jul/23 ]

I managed to POC an initial cut of implicit collection creation, modulo createIndexes support which needs a redesign of the concurrency model on secondaries.

Comment by Josef Ahmad [ 27/Jul/23 ]

I may have jumped the gun. The issue on secondaries described in SERVER-45945 seems to be still present.

Comment by Josef Ahmad [ 26/Jul/23 ]

I've experimented with removing the guard rails that prevent an end user from implicitly creating a collection in a distributed transaction, and I was able to create the collection implicitly. It looks like we've (perhaps inadvertently) resolved the issue described in SERVER-45945. Here's a snippet of the oplog entries on one shard, demonstrating the implicit collection creation:

{ "lsid" : { "id" : UUID("aef5f2d4-ab8d-4144-8d6a-ab585bf66ec8"), "uid" : BinData(0,"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=") }, "txnNumber" : NumberLong(0), "op" : "c", "ns" : "admin.$cmd", "o" : { "commitTransaction" : 1, "commitTimestamp" : Timestamp(1690366129, 7) }, "ts" : Timestamp(1690366129, 9), "t" : NumberLong(1), "v" : NumberLong(2), "wall" : ISODate("2023-07-26T10:08:49.732Z"), "prevOpTime" : { "ts" : Timestamp(1690366129, 7), "t" : NumberLong(1) } }
{ "lsid" : { "id" : UUID("aef5f2d4-ab8d-4144-8d6a-ab585bf66ec8"), "uid" : BinData(0,"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=") }, "txnNumber" : NumberLong(0), "op" : "c", "ns" : "admin.$cmd", "o" : { "applyOps" : [ { "op" : "i", "ns" : "test.c", "ui" : UUID("84d6dfd0-60fc-4bf4-84ab-5cbb15b57543"), "o" : { "_id" : ObjectId("64c0f0ab0403d5d5573ba686") }, "o2" : { "_id" : ObjectId("64c0f0ab0403d5d5573ba686") } }, { "op" : "c", "ns" : "test.$cmd", "ui" : UUID("5cc56dd9-f8b1-4745-aefa-8b5b7cc949b7"), "o" : { "create" : "c2", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } } }, { "op" : "i", "ns" : "test.c2", "ui" : UUID("5cc56dd9-f8b1-4745-aefa-8b5b7cc949b7"), "o" : { "_id" : ObjectId("64c0f0ac0403d5d5573ba696") }, "o2" : { "_id" : ObjectId("64c0f0ac0403d5d5573ba696") } } ], "prepare" : true }, "ts" : Timestamp(1690366129, 7), "t" : NumberLong(1), "v" : NumberLong(2), "wall" : ISODate("2023-07-26T10:08:49.703Z"), "prevOpTime" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) } }

The evergreen patch is green, except for the tests that verified that implicit collection creation in distributed transactions is disallowed.

The feature needs comprehensive validation coverage, and more work will likely surface.

Comment by Eric Milkie [ 13/Feb/20 ]

We will consider backporting this work to 4.4

Generated at Thu Feb 08 05:10:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.