[DRIVERS-716] Improved Bulk Write API Created: 19/Aug/19  Updated: 02/Feb/24

Status: Designing
Project: Drivers
Component/s: CRUD
Fix Version/s: None

Type: Epic Priority: Major - P3
Reporter: Esha Bhargava Assignee: Isabel Atkinson
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by MONGOSH-1100 Support Bulk Write API 2.0 Waiting (Blocked)
is depended on by MONGOID-5026 Coalesce updates where possible Closed
Duplicate
is duplicated by DRIVERS-717 One Shot Transactions Closed
Issue split
split to CSHARP-4145 Improved Bulk Write API Blocked
split to CXX-2494 Improved Bulk Write API Blocked
split to GODRIVER-2388 Improved Bulk Write API Blocked
split to JAVA-4586 Improved Bulk Write API Blocked
split to MOTOR-940 Improved Bulk Write API Blocked
split to NODE-4197 Improved Bulk Write API Blocked
split to PHPLIB-847 Improved Bulk Write API Blocked
split to PYTHON-3233 Improved Bulk Write API Blocked
split to RUBY-2964 Improved Bulk Write API Blocked
split to CDRIVER-4363 Improved Bulk Write API Scheduled
split to RUST-1282 Improved Bulk Write API Implementing
Related
related to DRIVERS-717 One Shot Transactions Closed
related to DRIVERS-2725 BulkWriteResult.insertedIds does not ... Backlog
related to DRIVERS-2763 Reorganize CRUD spec headings Scheduled
related to DRIVERS-2470 Extra write and write concern error f... Backlog
is related to DRIVERS-2582 Auto-batching of write operations Closed
is related to DRIVERS-2215 Add specs and tests for grouping bulk... Backlog
is related to DRIVERS-2090 Discuss errors that may interrupt a b... Backlog
is related to DRIVERS-2093 How should drivers handle multiple Wr... Backlog
is related to DRIVERS-2159 How should drivers handle non-write e... Backlog
Driver Changes: Needed
Server Compat: 8.0
Quarter: FY23Q3, FY23Q4, FY24Q1, FY24Q3, FY24Q4, FY25Q1
Upstream Changes Summary:

Drivers:
Introduce a new bulk write command to Stable API V1 which enhances the current way drivers expose bulkWrite() to applications. Specifically, enable drivers to offer the following capabilities in their bulkWrite() API:

1. Support mixing inserts, updates, and deletes in a single request.
2. Support writing to multiple namespaces in a single request.

Docs:
Create documentation for the new server bulkWrite command and its limitations. Document the autocommit:true flag and its limitations in bulkWrite.

Downstream Changes Summary:

Summary of necessary driver changes

  •  

Commits for syncing spec/prose tests
(and/or refer to an existing language POC if needed)

  •  

Context for other referenced/linked tickets

  •  
Engineering Lead: Jeremy Mikola Jeremy Mikola
Product Manager: Rachelle Palmer Rachelle Palmer
Program Manager: Tom Selander Tom Selander
Start date:
Detailed Project Statuses:

Engineer: Isabel Atkinson
Summary: We will add driver support for the new bulkWrite command, and improve the usability and performance of the existing bulk write API.

2024-02-02

  • Spec, spec tests, and Rust implementation in progress.
  • libmongocrypt support of bulkWrite in progress.

2023-12-23

  • BulkWrite Syntax finalized
  • Rust POC in progress
  1. Discuss end date with Isabel post-New Year
  • Second implementation will likely be in C
  1. Not feasible to be prototype this in PHP
  2. To be decided post-New Year during team QP prep
  • Server project expected completion is 2024-02-05

2023-12-08

  • Syntax document now in Final Review
  • Server project expected completion is now set to 2024-02-05

2023-11-10

  • Syntax document in review
  • Coordinating new server-side API with server team; command structure will change to improve performance

2023-10-27

  • Working on Syntax document
  • Meeting with Server team on 10-31 to discuss potential blockers and timeline

2023-10-13

  • Working on proposed Full Results API design

2023-09-15

  • Designing and collecting feedback on driver API response

2023-09-01

  • Work scheduled to resume on 9/15
  • Rust prototype will be used to finalize design, currently in-progress

2023-08-18

  • Work scheduled to resume on 8/28
  • Rust prototype will be used to finalise design

2023-08-04

  • Scope reduction approved
    • findAndModify support cut
    • autocommit with bulkWrite cut

2023-07-07

  • Scope reduction currently in team review

2023-06-23

  • Discussing scope reduction with server
  • findAndModify support cut from initial API
  • Retried bulk writes with autocommit: true will not return detailed result (20+ week effort in server)
  • Cursor response to return detailed results

2023-06-07

  • Drivers work is on hold while the scope of the Server feature is being reworked, with planned reductions in scope
  • BulkWrite server command is ready for POC in Rust, hope to pick up after more conversations with Server next week

2023-05-12

  • Document in progress and first review stages
  • Need to change return type handling due to command splitting (e.g. when sending a large batch)

2023-04-28

  • Assembling review crew, document will move to team review soon
  • Implementation work still blocked on Server until ~August

2023-04-14

  • Andreas has completed initial review of the syntax document
  • Feedback to be incorporated, Syntax document moving to Review soon
  • Implementation work still blocked on Server until ~August

2023-03-31

  • Waiting for feedback on syntax document.
  • No updates in the last 2 weeks

2023-03-17

  • Waiting for feedback on syntax document.
  • Server team has pushed off impl by a few more weeks.

2023-03-03

  • Continued design meetings and the syntax document is in review.

2023-02-17

  • Have held several design meetings and working on wrapping up the syntax document.

2023-02-03

  • Continuing working on syntax document and addressing design questions.

2023-01-20

  • Working on writing the syntax document and an API POC in the Rust driver.

2023-01-06

  • Design questions document is in review.
  • Next up is the syntax document for the public API.
Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4363 Scheduled
CXX-2494 Blocked
CSHARP-4145 Blocked
GODRIVER-2388 Blocked
JAVA-4586 Blocked
NODE-4197 Blocked
MOTOR-940 Blocked
PYTHON-3233 Blocked
PHPLIB-847 Blocked
RUBY-2964 Blocked
RUST-1282 Implementing
SWIFT-1549 Won't Do

 Description   
Epic Summary

Summary

We will add driver support for performing write-only one-shot transactions, and improve the usability and performance of the existing bulk write API. We will accomplish this by expanding the bulk write API to allow for bulk writes to be executed across multiple collections and databases, and to make mixed operation and cross-namespace bulk writes more efficient by executing bulk write operations in one batch.

Lead : Jeremy Mikola
Spec Author : Isabel Atkinson
POCs : Rust & TBD
Product Manager: Rachelle Palmer

Spec Update



 Comments   
Comment by PM Bot [ 04/Oct/22 ]

Moved to Needs Triage because a linked PM issue PM-1452 was moved to Ready for Work.

Comment by PM Bot [ 15/Feb/22 ]

If you are not logged in, you can view the tickets in this epic by following this link.

Comment by PM Bot [ 14/Jan/22 ]

If you are not logged in, you can view the tickets in this epic by following this link.

Generated at Thu Feb 08 08:22:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.