[JAVA-1541] unordered bulk writes should continue sending batches after an error Created: 10/Nov/14  Updated: 01/Apr/16  Resolved: 10/Nov/14

Status: Closed
Project: Java Driver
Component/s: Write Operations
Affects Version/s: 2.12.0
Fix Version/s: 2.12.5

Type: Bug Priority: Critical - P2
Reporter: Alexander Komyagin Assignee: Jeffrey Yemin
Resolution: Done Votes: 0
Labels: cs
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related

 Description   

The bulk write API added in 2.12 supports sending inserts, updates, and deletes in bulk to the server. If the number of writes requests exceeds 1000, or the total size of the write requests exceed 16MB, the driver will split the write requests into batches of 1000 and/or batches whose size does not exceed 16MB.

For ordered bulk writes, the driver stops sending more batches to the server if there were any errors reported in the current batch.

For unordered bulk writes, the driver should keep sending the rest of the batches to the server, even if errors were reported in the current batch. There is a bug which is preventing that from happening. Instead, the driver stops sending more batches and immediately reports the first batch of errors to the application, ignoring any subsequent batches.

An application will be effected by this bug in the following circumstances:

  • It is using the method DBCollection.initializeUnorderedBulkWriteOperation
  • The number of writes added to the operation exceeds 1000 or 16MB of total data
  • The possibility of an error (most likely a duplicate key error) exists

Workaround:

Limit the number of writes in a single bulk write operation to less than 1000 and less than 16 MB



 Comments   
Comment by Alexander L. [ 20/Nov/14 ]

I was under the impression that a BulkWriteException is never thrown with an unordered execution and for checking any errors, I need to check the return from execute() if I want to be informed about any errors. If that's not the case, from what it seems, maybe it would be nice if it's a bit more clear on the documentation for the unordered operation.

Thanks for your help though, I wrapped my calls in try-catch and everything works fine now

Alexander

Comment by Jeffrey Yemin [ 20/Nov/14 ]

Tell me what you are trying to demonstrate with this change. I don't see that it changes anything. Everything after the catch is just to prove that execute worked as expected.

Comment by Alexander L. [ 20/Nov/14 ]

Hi Jeff,

Can you run it again without the try-catch for o.execute() ?

Alexander

Comment by Jeffrey Yemin [ 20/Nov/14 ]

This works as expected in my equivalent Java test:

       MongoClient mc = new MongoClient();
        DBCollection c = mc.getDB("JAVA1541").getCollection("test");
        c.drop();
 
        BulkWriteOperation o = c.initializeUnorderedBulkOperation();
        o.insert(new BasicDBObject("_id", 1));
        o.insert(new BasicDBObject("_id", 1));
        o.insert(new BasicDBObject("_id", 2));
 
        try {
            o.execute();
        } catch (BulkWriteException e) {
            System.out.println(e.getWriteResult());
        }
 
        System.out.println("Count: " + c.count());
        System.out.println("_ids: " + c.distinct("_id"))

Comment by Alexander L. [ 20/Nov/14 ]

Hi Jeff,

I am using Mongo v2.6.5 and I use mongo-java-driver 2.13.0-rc0 from within a clojure v1.6.0 application.

I am doing the following simple test in clojure REPL:

(import '(com.mongodb MongoClient WriteConcern BasicDBObject))
(def client (MongoClient.))
(def db (.getDB client "test"))
(def coll (.getCollection db "items"))
(def bulk (.initializeUnorderedBulkOperation coll))
(.insert bulk (BasicDBObject. "_id" "a"))
(.insert bulk (BasicDBObject. "_id" "a")) ;; My dupe key
(.insert bulk (BasicDBObject. "_id" "b"))
(.execute bulk)

BulkWriteException Bulk write operation error on server 127.0.0.1:27017. Write errors: [BulkWriteError{index=1, code=11000, message='insertDocument :: caused by :: 11000 E11000 duplicate key error index: test.items.$_id_ dup key: { : "a" }', details={ }}]. com.mongodb.BulkWriteBatchCombiner.throwOnError (BulkWriteBatchCombiner.java:125)

Do you see anything I am doing wrong on the above test?
Is there a method in mongo-java-driver which I can call and verify the version? I couldn't find anything in the documentation.

Alexander

Comment by Jeffrey Yemin [ 19/Nov/14 ]

Hi Alexander,

Yes, the fix is in 2.13.0-rc0. This is the test that verifies it: https://github.com/mongodb/mongo-java-driver/blob/r2.13.0-rc0/src/test/com/mongodb/BulkWriteOperationSpecification.groovy#L807-824. Can you take a look at it and let us know what you're application is doing that is different?

Thanks,
Jeff

Comment by Alexander L. [ 19/Nov/14 ]

Is this fixed in 2.13.0-rc0? Because I got it from maven today and the bug still seems to be there.
There is no 2.12.5 in maven and checking out the above commit and manually compiling it didn't seem to fix the problem either.

Can someone confirm the fix please?

Comment by Githook User [ 10/Nov/14 ]

Author:

{u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}

Message: Fixed unordered bulk write operation to keep sending batches to the server even after an error in the current batch.

JAVA-1541
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/7a970716d359078438a933c0093d20e718a13a0f

Comment by Githook User [ 10/Nov/14 ]

Author:

{u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}

Message: Fixed unordered bulk write operation to keep sending batches to the server even after an error in the current batch.

JAVA-1541
Branch: 2.12.x
https://github.com/mongodb/mongo-java-driver/commit/988455002d6d30b07f4a14c75c5d1428234bc6b8

Generated at Thu Feb 08 08:54:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.