Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-716

Improved Bulk Write API

    • Type: Icon: Epic Epic
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Component/s: CRUD
    • None
    • Hide

      Summary of necessary driver changes

      • Implement the new bulk write API and tests defined in the specification.

      Commits for syncing spec/prose tests

      • Sync spec tests and implement prose tests from this commit.
        • 2024-08-28 include skips on serverless from 0984b09
        • 2024-08-29 include missing skip on serverless from e9f02f3
      • Sync the unacknowledged-client-bulkWrite command monitoring test files from this commit. The outdated version of this test in the commit from the previous bullet has a potential race.
      • See DRIVERS-2993 for a fix to unacknowledged write handling.

      Reference implementations

      Show
      Summary of necessary driver changes Implement the new bulk write API and tests defined in the specification . Commits for syncing spec/prose tests Sync spec tests and implement prose tests from this commit . 2024-08-28 include skips on serverless from  0984b09 2024-08-29 include missing skip on serverless from e9f02f3 Sync the unacknowledged-client-bulkWrite command monitoring test files from this commit . The outdated version of this test in the commit from the previous bullet has a potential race. See DRIVERS-2993 for a fix to unacknowledged write handling. Reference implementations Rust C
    • To Do
    • Improved Bulk Write API
    • Hide

      Engineer: Isabel Atkinson
      Summary: We will add driver support for the new bulkWrite command, and improve the usability and performance of the existing bulk write API.

      2024-05-10:

      What was accomplished since the last update?

      • Spec merged.

      What goals are we targeting for the next two weeks?

      • Update "Downstream Changes Summary" and move to "Teams Implementing"

      2024-04-29:

      What was accomplished since the last update?

      • Continued effort on spec PR:
        • Added tests for test parity with collection bulk write.
          **Removed variation in batch splitting calculation.

      What goals are we targeting for the next two weeks?

      • Get approval by Jeremy + Kevin + Jeff. Merge spec.

      2024-04-12:

      What was accomplished since the last update?

      • Continued effort on spec PR:
        • Added tests for test parity with collection bulk write.
        • Removed variation in batch splitting calculation.

      What goals are we targeting for the next two weeks?

      • Get approval by Jeremy + Kevin. Request review from a driver language with exceptions (Python/Java/.NET?). Merge spec.

      2024-04-12:

      What was accomplished since the last update?

      • Continued effort on spec PR. Added many spec tests.

      What goals are we targeting for the next two weeks?

      • Get approval by Jeremy + Kevin. Request review from a driver language with exceptions (Python/Java?). Merge spec.

      2024-03-28:

      What was accomplished since the last update?

      • Spec PR has been reviewed. C implementation passes all tests. Several new test cases were suggested.

      What goals are we targeting for the next two weeks?

      • Get approval by Jeremy + Kevin. Request review from broader audience. Merge spec.

      2024-03-14:

      What was accomplished since the last update?

      • Spec PR taken out of draft. Has been reviewed.

      What goals are we targeting for the next two weeks?

      • Merge spec.

      2024-03-01: 
      What was completed over the last two weeks?

      • Draft PRs are up for the spec changes and Rust implementation.

      What's the focus over the next two weeks?

      • Gather feedback from reviewers on spec changes and Rust implementation.

      Impediments encountered over the last two weeks:
      None

      2024-02-16: 
      What was completed over the last two weeks?

      • libmongocrypt support of bulkWrite put into review.

      What's the focus over the next two weeks?

      • To put spec, spec tests, and Rust implementation in review.

      Impediments encountered over the last two weeks:

      • Implementation difficulties w.r.t. Rust driver retry for writeConcernErrors.

       


      2024-02-02

      • Spec, spec tests, and Rust implementation in progress.
      • libmongocrypt support of bulkWrite in progress.

      2023-12-23

      • BulkWrite Syntax finalized
      • Rust POC in progress
      1. Discuss end date with Isabel post-New Year
      • Second implementation will likely be in C
      1. Not feasible to be prototype this in PHP
      2. To be decided post-New Year during team QP prep
      • Server project expected completion is 2024-02-05

      2023-12-08

      • Syntax document now in Final Review
      • Server project expected completion is now set to 2024-02-05

      2023-11-10

      • Syntax document in review
      • Coordinating new server-side API with server team; command structure will change to improve performance

      2023-10-27

      • Working on Syntax document
      • Meeting with Server team on 10-31 to discuss potential blockers and timeline

      2023-10-13

      • Working on proposed Full Results API design

      2023-09-15

      • Designing and collecting feedback on driver API response

      2023-09-01

      • Work scheduled to resume on 9/15
      • Rust prototype will be used to finalize design, currently in-progress

      2023-08-18

      • Work scheduled to resume on 8/28
      • Rust prototype will be used to finalise design

      2023-08-04

      • Scope reduction approved
        • findAndModify support cut
        • autocommit with bulkWrite cut

      2023-07-07

      • Scope reduction currently in team review

      2023-06-23

      • Discussing scope reduction with server
      • findAndModify support cut from initial API
      • Retried bulk writes with autocommit: true will not return detailed result (20+ week effort in server)
      • Cursor response to return detailed results

      2023-06-07

      • Drivers work is on hold while the scope of the Server feature is being reworked, with planned reductions in scope
      • BulkWrite server command is ready for POC in Rust, hope to pick up after more conversations with Server next week

      2023-05-12

      • Document in progress and first review stages
      • Need to change return type handling due to command splitting (e.g. when sending a large batch)

      2023-04-28

      • Assembling review crew, document will move to team review soon
      • Implementation work still blocked on Server until ~August

      2023-04-14

      • Andreas has completed initial review of the syntax document
      • Feedback to be incorporated, Syntax document moving to Review soon
      • Implementation work still blocked on Server until ~August

      2023-03-31

      • Waiting for feedback on syntax document.
      • No updates in the last 2 weeks

      2023-03-17

      • Waiting for feedback on syntax document.
      • Server team has pushed off impl by a few more weeks.

      2023-03-03

      • Continued design meetings and the syntax document is in review.

      2023-02-17

      • Have held several design meetings and working on wrapping up the syntax document.

      2023-02-03

      • Continuing working on syntax document and addressing design questions.

      2023-01-20

      • Working on writing the syntax document and an API POC in the Rust driver.

      2023-01-06

      • Design questions document is in review.
      • Next up is the syntax document for the public API.
      Show
      Engineer: Isabel Atkinson Summary: We will add driver support for the new bulkWrite command, and improve the usability and performance of the existing bulk write API. 2024-05-10: What was accomplished since the last update? Spec merged. What goals are we targeting for the next two weeks? Update "Downstream Changes Summary" and move to "Teams Implementing" 2024-04-29: What was accomplished since the last update? Continued effort on spec PR: Added tests for test parity with collection bulk write. **Removed variation in batch splitting calculation. What goals are we targeting for the next two weeks? Get approval by Jeremy + Kevin + Jeff. Merge spec. 2024-04-12: What was accomplished since the last update? Continued effort on spec PR: Added tests for test parity with collection bulk write. Removed variation in batch splitting calculation. What goals are we targeting for the next two weeks? Get approval by Jeremy + Kevin. Request review from a driver language with exceptions (Python/Java/.NET?). Merge spec. 2024-04-12: What was accomplished since the last update? Continued effort on spec PR. Added many spec tests. What goals are we targeting for the next two weeks? Get approval by Jeremy + Kevin. Request review from a driver language with exceptions (Python/Java?). Merge spec. 2024-03-28: What was accomplished since the last update? Spec PR has been reviewed. C implementation passes all tests. Several new test cases were suggested. What goals are we targeting for the next two weeks? Get approval by Jeremy + Kevin. Request review from broader audience. Merge spec. 2024-03-14: What was accomplished since the last update? Spec PR taken out of draft. Has been reviewed. What goals are we targeting for the next two weeks? Merge spec. 2024-03-01:  What was completed over the last two weeks? Draft PRs are up for the spec changes and Rust implementation. What's the focus over the next two weeks? Gather feedback from reviewers on spec changes and Rust implementation. Impediments encountered over the last two weeks: None 2024-02-16:  What was completed over the last two weeks? libmongocrypt support of bulkWrite put into review. What's the focus over the next two weeks? To put spec, spec tests, and Rust implementation in review. Impediments encountered over the last two weeks: Implementation difficulties w.r.t. Rust driver retry for writeConcernErrors.   2024-02-02 Spec, spec tests, and Rust implementation in progress. libmongocrypt support of bulkWrite in progress. 2023-12-23 BulkWrite Syntax finalized Rust POC in progress Discuss end date with Isabel post-New Year Second implementation will likely be in C Not feasible to be prototype this in PHP To be decided post-New Year during team QP prep Server project expected completion is 2024-02-05 2023-12-08 Syntax document now in Final Review Server project expected completion is now set to 2024-02-05 2023-11-10 Syntax document in review Coordinating new server-side API with server team; command structure will change to improve performance 2023-10-27 Working on Syntax document Meeting with Server team on 10-31 to discuss potential blockers and timeline 2023-10-13 Working on proposed Full Results API design 2023-09-15 Designing and collecting feedback on driver API response 2023-09-01 Work scheduled to resume on 9/15 Rust prototype will be used to finalize design, currently in-progress 2023-08-18 Work scheduled to resume on 8/28 Rust prototype will be used to finalise design 2023-08-04 Scope reduction approved findAndModify support cut autocommit with bulkWrite cut 2023-07-07 Scope reduction currently in team review 2023-06-23 Discussing scope reduction with server findAndModify support cut from initial API Retried bulk writes with autocommit: true will not return detailed result (20+ week effort in server) Cursor response to return detailed results 2023-06-07 Drivers work is on hold while the scope of the Server feature is being reworked, with planned reductions in scope BulkWrite server command is ready for POC in Rust, hope to pick up after more conversations with Server next week 2023-05-12 Document in progress and first review stages Need to change return type handling due to command splitting (e.g. when sending a large batch) 2023-04-28 Assembling review crew, document will move to team review soon Implementation work still blocked on Server until ~August 2023-04-14 Andreas has completed initial review of the syntax document Feedback to be incorporated, Syntax document moving to Review soon Implementation work still blocked on Server until ~August 2023-03-31 Waiting for feedback on syntax document. No updates in the last 2 weeks 2023-03-17 Waiting for feedback on syntax document. Server team has pushed off impl by a few more weeks. 2023-03-03 Continued design meetings and the syntax document is in review. 2023-02-17 Have held several design meetings and working on wrapping up the syntax document. 2023-02-03 Continuing working on syntax document and addressing design questions. 2023-01-20 Working on writing the syntax document and an API POC in the Rust driver. 2023-01-06 Design questions document is in review. Next up is the syntax document for the public API.
    • Needed
    • Hide

      Drivers:
      Introduce a new bulk write command to Stable API V1 which enhances the current way drivers expose bulkWrite() to applications. Specifically, enable drivers to offer the following capabilities in their bulkWrite() API:

      1. Support mixing inserts, updates, and deletes in a single request.
      2. Support writing to multiple namespaces in a single request.

      Docs:
      Create documentation for the new server bulkWrite command and its limitations. Document the autocommit:true flag and its limitations in bulkWrite.

      Show
      Drivers: Introduce a new bulk write command to Stable API V1 which enhances the current way drivers expose bulkWrite() to applications. Specifically, enable drivers to offer the following capabilities in their bulkWrite() API: 1. Support mixing inserts, updates, and deletes in a single request. 2. Support writing to multiple namespaces in a single request. Docs: Create documentation for the new server bulkWrite command and its limitations. Document the autocommit:true flag and its limitations in bulkWrite.
    • $i18n.getText("admin.common.words.hide")
      Key Status/Resolution FixVersion
      CDRIVER-4363 Fixed 1.28.0
      CXX-2494 Backlog
      CSHARP-4145 Fixed 3.0.0
      GODRIVER-2388 In Progress 2.1.0
      JAVA-4586 In Progress 5.3.0
      NODE-4197 Fixed 6.10.0
      MOTOR-940 Fixed 3.6
      PYTHON-3233 Fixed 4.9
      PHPLIB-847 In Progress
      RUBY-2964 Backlog
      RUST-1282 Fixed 3.0.0
      SWIFT-1549 Won't Do
      $i18n.getText("admin.common.words.show")
      #scriptField, #scriptField *{ border: 1px solid black; } #scriptField{ border-collapse: collapse; } #scriptField td { text-align: center; /* Center-align text in table cells */ } #scriptField td.key { text-align: left; /* Left-align text in the Key column */ } #scriptField a { text-decoration: none; /* Remove underlines from links */ border: none; /* Remove border from links */ } /* Add green background color to cells with FixVersion */ #scriptField td.hasFixVersion { background-color: #00FF00; /* Green color code */ } /* Center-align the first row headers */ #scriptField th { text-align: center; } Key Status/Resolution FixVersion CDRIVER-4363 Fixed 1.28.0 CXX-2494 Backlog CSHARP-4145 Fixed 3.0.0 GODRIVER-2388 In Progress 2.1.0 JAVA-4586 In Progress 5.3.0 NODE-4197 Fixed 6.10.0 MOTOR-940 Fixed 3.6 PYTHON-3233 Fixed 4.9 PHPLIB-847 In Progress RUBY-2964 Backlog RUST-1282 Fixed 3.0.0 SWIFT-1549 Won't Do

      Epic Summary

      Summary

      We will add driver support for performing write-only one-shot transactions, and improve the usability and performance of the existing bulk write API. We will accomplish this by expanding the bulk write API to allow for bulk writes to be executed across multiple collections and databases, and to make mixed operation and cross-namespace bulk writes more efficient by executing bulk write operations in one batch.

      Lead : Jeremy Mikola
      Spec Author : Isabel Atkinson
      POCs : Rust & TBD
      Product Manager: Rachelle Palmer

      Spec Update

            Assignee:
            isabel.atkinson@mongodb.com Isabel Atkinson
            Reporter:
            esha.bhargava@mongodb.com Esha Bhargava
            Jeremy Mikola Jeremy Mikola
            Tom Selander Tom Selander
            Rishabh Bisht Rishabh Bisht
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated: