[SERVER-19496] Latency optimized write API Created: 20/Jul/15  Updated: 26/Apr/19  Resolved: 02/Oct/17

Status: Closed
Project: Core Server
Component/s: Networking, Write Ops
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Bernie Hackett Assignee: Backlog - Tuning Team
Resolution: Done Votes: 1
Labels: PM-315
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File batch_vs_update.py    
Issue Links:
Depends
depends on SERVER-28510 Make Ingress and Egress networking la... Closed
Related
related to DRIVERS-266 Ensure spec compliance for unacknowle... Closed
related to CDRIVER-786 Using w=-1 write concern causes "inva... Closed
is related to SERVER-19287 Support unacknowledged operations via... Closed
Participants:

 Description   

MongoDB's legacy OP_INSERT, OP_UPDATE, and OP_REMOVE wire protocol operations don't return a response. This allows users to write applications that provide good write throughput with very low latency, assuming they aren't concerned with individual failed writes (for example, storing clickstream data for later aggregate analysis). The write commands introduced in MongoDB 2.6 always return a response regardless of the w value passed for writeConcern. If / when legacy write operations are removed from the server, application developers will no longer have a way to achieve the same low latency write performance. This is particularly a problem for client languages that provide no real parallelism (CPython / pypy, Ruby MRI), no usable threading support (Perl) or no threads of any kind (PHP).

One solution to this problem could be to provide a streaming write API, in addition to the existing RPC model. Clients could start a write session with the server, pushing mixed write operations (rather than batches of the same type of write operation required by existing write commands), then receive a write result when the session is closed.



 Comments   
Comment by Ian Whalen (Inactive) [ 26/Apr/19 ]

Switching "Drivers Changes Needed" from "Maybe" to "Not Needed" since this was closed as something other than Fixed.

Comment by Mathias Stearn [ 02/Oct/17 ]

Implemented in SERVER-28510

Comment by James Blackburn [ 21/Jul/15 ]

As some background we see that bulk inserts w=1 are half the speed of the deprecated w=0 followed by a query.

Test case attached.

* python batch_vs_update.py 
2015-05-20 19:51:02,767 Connecting
2015-05-20 19:51:04,577 Starting up
2015-05-20 19:51:04,577 3.0.2.dev0
2015-05-20 19:51:04,579 Removed: {u'ok': 1, u'n': 0}
2015-05-20 19:51:04,609 Executing batch WRITE_CONCERN: 1
2015-05-20 19:51:43,893 Documents 500 found
2015-05-20 19:51:43,893 Done: 0:00:39.314308
2015-05-20 19:52:01,138 Removed: {u'ok': 1, u'n': 500}
2015-05-20 19:52:01,168 Executing batch WRITE_CONCERN: 0
2015-05-20 19:52:19,053 Documents 500 found
2015-05-20 19:52:19,053 Done: 0:00:17.914555
2015-05-20 19:52:36,971 Removed: {u'ok': 1, u'n': 500}
2015-05-20 19:52:36,972 Executing update
2015-05-20 19:52:55,189 Documents 500 found
2015-05-20 19:52:55,189 Done: 0:00:18.217650
2015-05-20 19:53:12,598 Removed: {u'ok': 1, u'n': 500}

So we see, for large documents, w=1 taxes 2x (37s) as long as w=0 + find()/count() (18s) .

Generated at Thu Feb 08 03:51:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.