[JAVA-4890] Multidocument transaction support in GridFS Created: 21/Feb/23  Updated: 27/Sep/23

Status: Backlog
Project: Java Driver
Component/s: GridFS
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Unknown
Reporter: Ashni Mehta Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: roadmap
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Issue split
split from DRIVERS-2062 Modify GridFS spec to support session... Backlog
Related
is related to JAVA-4887 GridFS doesn't support multi-document... Closed
is related to DRIVERS-559 GridFS with multi-document transactio... Backlog
Quarter: FY24Q4
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

Users have asked for multidocument transaction support in GridFS. Some requests tracked here:



 Comments   
Comment by Jeffrey Yemin [ 07/Mar/23 ]

One easy fix for this could be to just not use the ClientSession for the index check/creation. Sending back to Needs Triage.

Comment by PM Bot [ 07/Mar/23 ]

Hey ashni.mehta@mongodb.com, We need additional details to investigate the problem. If this is still an issue for you, please provide the requested information.

Comment by Jeffrey Yemin [ 22/Feb/23 ]

Thanks. It does seem that listIndexes is not supported in a transaction, although createIndexes is.

While it's not the prettiest, I can suggest a workaround: The driver only executes listIndexes if the "files" collection is empty. So if you add a dummy file to that collection outside of a transaction, it should work. Something like this (though of course you'll have to structure it in a way that makes sense for your application):

            MongoDatabase database = client.getDatabase("JAVA4890");
            String bucketName = "myBucket";
            var bucket = GridFSBuckets.create(database, bucketName);
 
            // create dummy file if the "files" collection is empty for the GridFS bucket
            // if possible do this once at startup.
            MongoCollection<Document> filesCollection = database.getCollection(bucketName + ".files");
            if (filesCollection.estimatedDocumentCount() == 0) {
                bucket.uploadFromStream("dummy_file", new ByteArrayInputStream(new byte[0]));
            }
 
            // now insert the real file.  Since the files collection is not empty, it won't result in a call to listIndexes
            try (var clientSession = client.startSession()) {
                clientSession.withTransaction(() -> bucket.uploadFromStream(clientSession, "JAVA4890_file", 
                         new ByteArrayInputStream(new byte[42])));
            }

Comment by Greg Stewart [ 22/Feb/23 ]

@Jeffrey Yemin The docker container I am using for testing is `mongo:5.0.14`

Comment by Jeffrey Yemin [ 22/Feb/23 ]

contact@gjstewart.net what version of MongoDB are you running? According to https://www.mongodb.com/docs/manual/core/transactions/#std-label-transactions-create-collections-indexes, it seems that listIndexes, etc are supported in transactions as of MongoDB 4.4.

Comment by Greg Stewart [ 21/Feb/23 ]

Hey guys, thanks for the ticket!

Multidocument support is really important to me in my application. Without it, it is very hard to keep some form of ACID compliance.

This affects me in two ways;

The implementation I want to use uses a regular collection to keep track of all files that are stored, and then the gridfs to actually store them and handle revisions. This makes it so any modification operation requires  a transactional approach that spans collections and buckets. Understand this could be antipattern in a strict sense, but I wanted more contexual information stored along with a central id to manage and track a given file/ revisions.

Secondly, concerning bulk operations, i.e, import/export. My application needs to support import/export of all data held. For importing all the data across many collections and Gridfs buckets, this requires a transaction to ensure if/when errors occur to not leave the database in an inconsistent state.

For now, I have left file support on my app on the backburner but would like to finish implementing.

Generated at Thu Feb 08 09:03:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.