Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-716

Improved Bulk Write API

    • Type: Icon: Epic Epic
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Component/s: CRUD
    • Labels:
      None
    • Needed
    • Hide

      Drivers:
      Introduce a new bulk write command to Stable API V1 which enhances the current way drivers expose bulkWrite() to applications. Specifically, enable drivers to offer the following capabilities in their bulkWrite() API:

      1. Support mixing inserts, updates, and deletes in a single request.
      2. Support writing to multiple namespaces in a single request.

      Docs:
      Create documentation for the new server bulkWrite command and its limitations. Document the autocommit:true flag and its limitations in bulkWrite.

      Show
      Drivers: Introduce a new bulk write command to Stable API V1 which enhances the current way drivers expose bulkWrite() to applications. Specifically, enable drivers to offer the following capabilities in their bulkWrite() API: 1. Support mixing inserts, updates, and deletes in a single request. 2. Support writing to multiple namespaces in a single request. Docs: Create documentation for the new server bulkWrite command and its limitations. Document the autocommit:true flag and its limitations in bulkWrite.
    • Hide

      Summary of necessary driver changes

      •  

      Commits for syncing spec/prose tests
      (and/or refer to an existing language POC if needed)

      •  

      Context for other referenced/linked tickets

      •  
      Show
      Summary of necessary driver changes   Commits for syncing spec/prose tests (and/or refer to an existing language POC if needed)   Context for other referenced/linked tickets  
    • Hide

      Engineer: Isabel Atkinson
      Summary: We will add driver support for the new bulkWrite command, and improve the usability and performance of the existing bulk write API.

      2024-04-12:

      What was accomplished since the last update?

      • Continued effort on spec PR. Added many spec tests.

      What goals are we targeting for the next two weeks?

      • Get approval by Jeremy + Kevin. Request review from a driver language with exceptions (Python/Java?). Merge spec.

      2024-03-28:

      What was accomplished since the last update?

      • Spec PR has been reviewed. C implementation passes all tests. Several new test cases were suggested.

      What goals are we targeting for the next two weeks?

      • Get approval by Jeremy + Kevin. Request review from broader audience. Merge spec.

      2024-03-14:

      What was accomplished since the last update?

      • Spec PR taken out of draft. Has been reviewed.

      What goals are we targeting for the next two weeks?

      • Merge spec.

      2024-03-01: 
      What was completed over the last two weeks?

      • Draft PRs are up for the spec changes and Rust implementation.

      What's the focus over the next two weeks?

      • Gather feedback from reviewers on spec changes and Rust implementation.

      Impediments encountered over the last two weeks:
      None

      2024-02-16: 
      What was completed over the last two weeks?

      • libmongocrypt support of bulkWrite put into review.

      What's the focus over the next two weeks?

      • To put spec, spec tests, and Rust implementation in review.

      Impediments encountered over the last two weeks:

      • Implementation difficulties w.r.t. Rust driver retry for writeConcernErrors.

       


      2024-02-02

      • Spec, spec tests, and Rust implementation in progress.
      • libmongocrypt support of bulkWrite in progress.

      2023-12-23

      • BulkWrite Syntax finalized
      • Rust POC in progress
      1. Discuss end date with Isabel post-New Year
      • Second implementation will likely be in C
      1. Not feasible to be prototype this in PHP
      2. To be decided post-New Year during team QP prep
      • Server project expected completion is 2024-02-05

      2023-12-08

      • Syntax document now in Final Review
      • Server project expected completion is now set to 2024-02-05

      2023-11-10

      • Syntax document in review
      • Coordinating new server-side API with server team; command structure will change to improve performance

      2023-10-27

      • Working on Syntax document
      • Meeting with Server team on 10-31 to discuss potential blockers and timeline

      2023-10-13

      • Working on proposed Full Results API design

      2023-09-15

      • Designing and collecting feedback on driver API response

      2023-09-01

      • Work scheduled to resume on 9/15
      • Rust prototype will be used to finalize design, currently in-progress

      2023-08-18

      • Work scheduled to resume on 8/28
      • Rust prototype will be used to finalise design

      2023-08-04

      • Scope reduction approved
        • findAndModify support cut
        • autocommit with bulkWrite cut

      2023-07-07

      • Scope reduction currently in team review

      2023-06-23

      • Discussing scope reduction with server
      • findAndModify support cut from initial API
      • Retried bulk writes with autocommit: true will not return detailed result (20+ week effort in server)
      • Cursor response to return detailed results

      2023-06-07

      • Drivers work is on hold while the scope of the Server feature is being reworked, with planned reductions in scope
      • BulkWrite server command is ready for POC in Rust, hope to pick up after more conversations with Server next week

      2023-05-12

      • Document in progress and first review stages
      • Need to change return type handling due to command splitting (e.g. when sending a large batch)

      2023-04-28

      • Assembling review crew, document will move to team review soon
      • Implementation work still blocked on Server until ~August

      2023-04-14

      • Andreas has completed initial review of the syntax document
      • Feedback to be incorporated, Syntax document moving to Review soon
      • Implementation work still blocked on Server until ~August

      2023-03-31

      • Waiting for feedback on syntax document.
      • No updates in the last 2 weeks

      2023-03-17

      • Waiting for feedback on syntax document.
      • Server team has pushed off impl by a few more weeks.

      2023-03-03

      • Continued design meetings and the syntax document is in review.

      2023-02-17

      • Have held several design meetings and working on wrapping up the syntax document.

      2023-02-03

      • Continuing working on syntax document and addressing design questions.

      2023-01-20

      • Working on writing the syntax document and an API POC in the Rust driver.

      2023-01-06

      • Design questions document is in review.
      • Next up is the syntax document for the public API.
      Show
      Engineer: Isabel Atkinson Summary: We will add driver support for the new bulkWrite command, and improve the usability and performance of the existing bulk write API. 2024-04-12: What was accomplished since the last update? Continued effort on spec PR. Added many spec tests. What goals are we targeting for the next two weeks? Get approval by Jeremy + Kevin. Request review from a driver language with exceptions (Python/Java?). Merge spec. 2024-03-28: What was accomplished since the last update? Spec PR has been reviewed. C implementation passes all tests. Several new test cases were suggested. What goals are we targeting for the next two weeks? Get approval by Jeremy + Kevin. Request review from broader audience. Merge spec. 2024-03-14: What was accomplished since the last update? Spec PR taken out of draft. Has been reviewed. What goals are we targeting for the next two weeks? Merge spec. 2024-03-01:  What was completed over the last two weeks? Draft PRs are up for the spec changes and Rust implementation. What's the focus over the next two weeks? Gather feedback from reviewers on spec changes and Rust implementation. Impediments encountered over the last two weeks: None 2024-02-16:  What was completed over the last two weeks? libmongocrypt support of bulkWrite put into review. What's the focus over the next two weeks? To put spec, spec tests, and Rust implementation in review. Impediments encountered over the last two weeks: Implementation difficulties w.r.t. Rust driver retry for writeConcernErrors.   2024-02-02 Spec, spec tests, and Rust implementation in progress. libmongocrypt support of bulkWrite in progress. 2023-12-23 BulkWrite Syntax finalized Rust POC in progress Discuss end date with Isabel post-New Year Second implementation will likely be in C Not feasible to be prototype this in PHP To be decided post-New Year during team QP prep Server project expected completion is 2024-02-05 2023-12-08 Syntax document now in Final Review Server project expected completion is now set to 2024-02-05 2023-11-10 Syntax document in review Coordinating new server-side API with server team; command structure will change to improve performance 2023-10-27 Working on Syntax document Meeting with Server team on 10-31 to discuss potential blockers and timeline 2023-10-13 Working on proposed Full Results API design 2023-09-15 Designing and collecting feedback on driver API response 2023-09-01 Work scheduled to resume on 9/15 Rust prototype will be used to finalize design, currently in-progress 2023-08-18 Work scheduled to resume on 8/28 Rust prototype will be used to finalise design 2023-08-04 Scope reduction approved findAndModify support cut autocommit with bulkWrite cut 2023-07-07 Scope reduction currently in team review 2023-06-23 Discussing scope reduction with server findAndModify support cut from initial API Retried bulk writes with autocommit: true will not return detailed result (20+ week effort in server) Cursor response to return detailed results 2023-06-07 Drivers work is on hold while the scope of the Server feature is being reworked, with planned reductions in scope BulkWrite server command is ready for POC in Rust, hope to pick up after more conversations with Server next week 2023-05-12 Document in progress and first review stages Need to change return type handling due to command splitting (e.g. when sending a large batch) 2023-04-28 Assembling review crew, document will move to team review soon Implementation work still blocked on Server until ~August 2023-04-14 Andreas has completed initial review of the syntax document Feedback to be incorporated, Syntax document moving to Review soon Implementation work still blocked on Server until ~August 2023-03-31 Waiting for feedback on syntax document. No updates in the last 2 weeks 2023-03-17 Waiting for feedback on syntax document. Server team has pushed off impl by a few more weeks. 2023-03-03 Continued design meetings and the syntax document is in review. 2023-02-17 Have held several design meetings and working on wrapping up the syntax document. 2023-02-03 Continuing working on syntax document and addressing design questions. 2023-01-20 Working on writing the syntax document and an API POC in the Rust driver. 2023-01-06 Design questions document is in review. Next up is the syntax document for the public API.
    • $i18n.getText("admin.common.words.hide")
      Key Status/Resolution FixVersion
      CDRIVER-4363 Investigating
      CXX-2494 Blocked
      CSHARP-4145 Blocked
      GODRIVER-2388 Blocked
      JAVA-4586 Blocked
      NODE-4197 Blocked
      MOTOR-940 Blocked
      PYTHON-3233 Blocked
      PHPLIB-847 Blocked
      RUBY-2964 Blocked
      RUST-1282 Implementing
      SWIFT-1549 Won't Do
      $i18n.getText("admin.common.words.show")
      #scriptField, #scriptField *{ border: 1px solid black; } #scriptField{ border-collapse: collapse; } #scriptField td { text-align: center; /* Center-align text in table cells */ } #scriptField td.key { text-align: left; /* Left-align text in the Key column */ } #scriptField a { text-decoration: none; /* Remove underlines from links */ border: none; /* Remove border from links */ } /* Add green background color to cells with FixVersion */ #scriptField td.hasFixVersion { background-color: #00FF00; /* Green color code */ } /* Center-align the first row headers */ #scriptField th { text-align: center; } Key Status/Resolution FixVersion CDRIVER-4363 Investigating CXX-2494 Blocked CSHARP-4145 Blocked GODRIVER-2388 Blocked JAVA-4586 Blocked NODE-4197 Blocked MOTOR-940 Blocked PYTHON-3233 Blocked PHPLIB-847 Blocked RUBY-2964 Blocked RUST-1282 Implementing SWIFT-1549 Won't Do

      Epic Summary

      Summary

      We will add driver support for performing write-only one-shot transactions, and improve the usability and performance of the existing bulk write API. We will accomplish this by expanding the bulk write API to allow for bulk writes to be executed across multiple collections and databases, and to make mixed operation and cross-namespace bulk writes more efficient by executing bulk write operations in one batch.

      Lead : Jeremy Mikola
      Spec Author : Isabel Atkinson
      POCs : Rust & TBD
      Product Manager: Rachelle Palmer

      Spec Update

            Assignee:
            isabel.atkinson@mongodb.com Isabel Atkinson
            Reporter:
            esha.bhargava@mongodb.com Esha Bhargava
            Jeremy Mikola Jeremy Mikola
            Rishabh Bisht Rishabh Bisht
            Tom Selander Tom Selander
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: