Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-4403

Trying to bulk load documents with complex _id gives error

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.4.0
    • Affects Version/s: 4.0.0
    • Component/s: API
    • Labels:

      Versions:

      • Spring Boot 2.5.6
      • Spring Data Mongo: 3.2.6
      • Mongo Driver: 4.2.3

      I am trying to upgrade my application to latest Spring/Mongo APIs.

      I am getting an error when I try to bulk load data saved using mongoexport.

      Attached is one of the files that gives the error.

      The error is:

       

      2021-11-10 15:50:00.668 DEBUG 84853 --- [           main] org.mongodb.driver.protocol.command      : Sending command '{"insert": "client_counts_by_hour", "ordered": false, "bypassDocumentValidation": true, "$db": "jameson", "lsid": {"id": {"$binary": {"base64": "y1q10S0HRKaUCzpnqxLHhg==", "subType": "04"}}}, "documents": [{"_id": {"$oid": "618bea287d91ca1961b3e1cb"}, "attributes": {"5a04d33ea09f961635c03580": {"attribute": "5a04d33ea09f961635c03580", "count": 2, "sessionCount": 2, "totalAvgDwell": 240147.0, "totalDwell": 240147.0}, "all": {"attribute": "all", "count": 2, "sessionCount": 2, "totalAvgDwell": 240147.0, "totalDwell": 240147.0}, "frequent": {"attribute": "frequent", "count": 2, "sessionCount": 2, "totalAvgDwell": 240147.0, "totalDwell": 240147.0}, "probing": {"attribute": "probing", "count": 2, "sessionCount": 2, "totalAvgDwell": 240147.0, "totalDwell": 240147.0}, "returned": {"attribute": "returned", "count": 2, "sessionCount": 2, "totalAvgDwell": 240147.0, "totalDwell": 240147.0}}, "locations": {"5d418961c678980007efe353": {"attributes": {"5a04d33ea09f961635c03580": {"attribute": "5a04d3 ...' with request id 44 to database jameson on connection [connectionId{localValue:3, serverValue:3}] to server localhost:58134
2021-11-10 15:50:00.677 DEBUG 84853 --- [           main] org.mongodb.driver.protocol.command      : Execution of command with request id 44 completed successfully in 10.60 ms on connection [connectionId{localValue:3, serverValue:3}] to server localhost:58134
Error writing to client_counts_by_hour: Bulk write operation error on server localhost:58134. Write errors: [BulkWriteError{index=0, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=1, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=2, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=3, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=4, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=5, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=6, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=7, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=8, code=2, message='can't have multiple _id fields in one document', details={}}, BulkWriteError{index=9, code=2, message='can't have multiple _id fields in one document', details={}}]. 

       

      Note that an extra _id is being generated/added by the driver and can be seen in the trace. So the server side error makes sense. I have been tracing through the code but so far haven't found where it gets added.

      This is my code for reference. I am accessing the Mongo driver via the spring template..

         Scanner scanner = new Scanner(is);
    List<InsertOneModel<Document>> documents = new ArrayList<>();
    int count = 0;


    while (scanner.hasNext()) {
      documents.add(new InsertOneModel<>(Document.parse(scanner.nextLine())));
      count++;
      if (documents.size() > 1000) {
        try {
          template.getCollection(collectionName).bulkWrite(documents,new BulkWriteOptions().bypassDocumentValidation(true).ordered(false));
        } catch (Exception e) {
          System.err.println("Error writing to "+collectionName+ ": "+e.getLocalizedMessage());
        } finally {
          documents.clear();
        }
      }
    }
    try {
        template.getCollection(collectionName).bulkWrite(documents,new BulkWriteOptions().bypassDocumentValidation(true).ordered(false));
    } catch (Exception e) {
        System.err.println("Error writing to "+collectionName+ ": "+e.getLocalizedMessage());
        throw e;
    } finally {
      documents.clear();
    }
    System.out.println(count + " documents loaded for " + collectionName + " collection.");

       

       

       

        1. client_counts_by_hour.json
          52 kB

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            steve@kloudspot.com Steve Jerman
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: