[DRIVERS-1340] Allow hint for unacknowledged writes using OP_MSG when supported by the server Created: 20/Jul/20  Updated: 22/Jun/22

Status: Implementing
Project: Drivers
Component/s: CRUD
Fix Version/s: None

Type: Spec Change Priority: Minor - P4
Reporter: Jeremy Mikola Assignee: Jeremy Mikola
Resolution: Unresolved Votes: 0
Labels: leads-triage
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on DRIVERS-681 Allow passing hint to findAndModify u... Closed
depends on DRIVERS-751 support ability to pass hint to update Closed
Issue split
split to NODE-3541 Allow hint for unacknowledged writes ... Backlog
split to JAVA-4275 Allow hint for unacknowledged writes ... Closed
split to PHPLIB-712 Allow hint for unacknowledged writes ... Closed
split to CSHARP-3791 Allow hint for unacknowledged writes ... Backlog
split to GODRIVER-2125 Allow hint for unacknowledged writes ... Backlog
split to CDRIVER-4127 Allow hint for unacknowledged writes ... Closed
split to CXX-2347 Allow hint for unacknowledged writes ... Closed
split to MOTOR-806 Allow hint for unacknowledged writes ... Closed
split to PYTHON-2863 Allow hint for unacknowledged writes ... Closed
split to RUBY-2766 Allow hint for unacknowledged writes ... Closed
split to RUST-974 Allow hint for unacknowledged writes ... Closed
Related
related to DRIVERS-1899 Write results should be optional to a... Closed
is related to DRIVERS-790 Raise error if hint specified for una... Closed
Driver Changes: Needed
Downstream Changes Summary:

Drivers should no longer prohibit using hint with an unacknowledged write concern for update, delete, and findAndModify commands (issued via OP_MSG) and related helpers (e.g. UpdateOne, bulkWrite, findOneAndDelete) if hint is known to be supported by the server executing the operation (i.e. 4.2+ for update commands, 4.4+ for delete and findAndModify). Drivers should sync unified CRUD tests with mongodb/specifications@59a5dad

Drivers (like Java, for example) that ignore unacknowledged write concern for findOneAndReplace/Update/Delete helpers should skip the following tests:

  • Unacknowledged findOneAndReplace with hint string on 4.4+ server
  • Unacknowledged findOneAndReplace with hint document on 4.4+ server
  • Unacknowledged findOneAndUpdate with hint string on 4.4+ server
  • Unacknowledged findOneAndUpdate with hint document on 4.4+ server
  • Unacknowledged findOneAndDelete with hint string on 4.4+ server
  • Unacknowledged findOneAndDelete with hint document on 4.4+ server

because those tests are assuming that the driver includes writeConcern: {w : 0 } in the findAndModify command and that, as a result, the server response with only ok: 1 and therefore value is unset.

Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4127 Fixed 1.22.0
CXX-2347 Fixed 3.7.0
CSHARP-3791 Backlog
GODRIVER-2125 Backlog
JAVA-4275 Fixed 4.5.0
NODE-3541 Backlog
MOTOR-806 Duplicate
PYTHON-2863 Fixed 4.1
PHPLIB-712 Fixed 1.10.0
RUBY-2766 Fixed 2.18.0
RUST-974 Duplicate
SWIFT-1304 Fixed 1.4.0

 Description   

This reverts DRIVERS-991, which instructed drivers to raise a client-side error when the hint option is provided with an unacknowledged write issued against any server version.

When the the server is known to support hint (i.e. 4.2+ for update, 4.4+ for delete), drivers should allow the option to be specified. They should only report an error for earlier server versions where the server is known to not support the option and an error would otherwise not be returned because of the nature of unacknowledged writes.

Previous arguments relating hint to existing treatment of arrayFilters and collation aren't relevant, because their documentation refers to "unacknowledged writes using OP_UPDATE" (where the option cannot be expressed). Those options don't refer to OP_MSG at all, because they were both supported by all server versions that support OP_MSG. hint can retain the language about OP_UPDATE, but its OP_MSG language should be revised to only mandate an error on unsupported server versions.

Lastly, the following Q&A should be revised:

Q: Why are client-side errors raised when options are provided for unacknowledged write operations, even on server versions that support those options?
Even if the server supports an option, that option could be incorrectly formatted or contain outdated information (e.g. a "hint" option referencing an index that no longer exists). This is a problem during unacknowledged writes because the server cannot return an error response to alert the client of an incorrect option. By always raising a client-side error when options are specified for unacknowledged writes, users are prevented from performing operations with options that do not work.

This is a poor argument as incorrectly formatted options can just as easily apply to the filter or update/replacement arguments (e.g. using an invalid query operator). It's one thing to raise a client-error when the driver knows for certain that a server option is not supported, but the current strategy is presumptive.

We should also update the language for findAndModify in the CRUD spec (since it does refer to behavior of unacknowledged write concerns). findAndModify support for hint requires 4.4+ (see: SERVER-42099, SERVER-44367). Although findAndModify always returns a response, an unacknowledged write concern seems to force an {{

{ok:1}

}} response, which hides any error if hint is unsupported or invalid (e.g. no such index). With respect to this issue, we need only concern ourselves with whether hint is supported, so raising an error if hint is used on a pre-4.4 server with an unacknowledged write concern is sufficient.


Summary

This issue was originally reported after starting work on language tickets for DRIVERS-991, which asked drivers to raise client-side errors for any use of hint with unacknowledged writes. This ticket seeks to revert that change and make an allowance for cases where drivers know that the server does support the hint option. Since DRIVERS-991 was filed, libmongoc and dependent drivers have chosen not to implement that ticket and are instead waiting on this ticket to be resolved. This ticket has languished for more than a year since and remains outstanding.

Motivation

Who is the affected end user?

Applications that would like to utilize hint options with unacknowledged writes.

How does this affect the end user?

The behavior is currently prohibited by our specifications despite being possible to implement. In hindsight, it was premature to add the blanket prohibition.

How likely is it that this problem or use case will occur?

A minority of applications make individual use of hint and unacknowledged writes. The intersection of applications using both features together is even rarer.

If the problem does occur, what are the consequences and how severe are they?

Inability to execute a supported operation.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

This ticket affects both driver functionality and our spec tests.



 Comments   
Comment by Jeremy Mikola [ 06/Dec/21 ]

Documentation for the value result field only talks about null and document values. It does not mention that the value field will be entirely omitted if findAndModify specifies an unacknowledged write concern, as shown below:

> db.foo.drop()
false
> db.foo.insertOne({x:1})
{
	"acknowledged" : true,
	"insertedId" : ObjectId("61ae3844a4b363eaef266d3d")
}
> db.runCommand({findAndModify:"foo",query:{x:1},update:{$inc:{y:1}},new:true,writeConcern:{w:0}})
{ "ok" : 1 }
> db.foo.findOne()
{ "_id" : ObjectId("61ae3844a4b363eaef266d3d"), "x" : 1, "y" : 1 }
> db.runCommand({findAndModify:"foo",query:{x:1},update:{$inc:{y:1}},new:false,writeConcern:{w:0}})
{ "ok" : 1 }
> db.foo.findOne()
{ "_id" : ObjectId("61ae3844a4b363eaef266d3d"), "x" : 1, "y" : 2 }

I'll note that the CRUD spec currently documents the return value of findAndModify helpers as "Document" and doesn't seem to discuss a possible null return value at all. SPEC-148 is an old ticket that talked about that inconsistency and claims to have addressed it but it's not cross-referenced with any commit that I could find.

As far as handling non-existent results (i.e. value is omitted instead of being null), I created DRIVERS-1899 while working on this issue (DRIVERS-1340) to propose that we use "Optional<>" syntax for all results of CRUD methods that might utilize an unacknowledged write concern. That would only entail changes to the spec itself (not tests), which is why I used $$unsetOrMatches in the existing tests for DRIVERS-1340.


I expect many drivers to fail this assertion because it would require that the moreToCome bit is set for the message (or the response is dropped by the driver itself). But it's not clear that is the correct behavior. Consider that:

  • findAndModify is a query, so a result is expected by the application, as with any other query.
  • findAndModify can return null for the (possibly) modified document, and it would be odd if that could mean both that no document was modified or that a document was possibly modified but the write was unacknowledged.

As one reference, the Java driver's current behavior is the following:

  • If the write concern is unacknowledged and hint is specified, an exception is thrown.
  • If the write concern is unacknowledged, the write concern is not included in the findAndModify command, and the moreToCome bit remains unset.
  • If the write concern is set and is not unacknowledged, it's included in the findAndModify command.

I'm not aware of any guidance that drivers set moreToCome for findAndModify commands. With respect to unacknowledged write concerns, I've only seen moreToCome used with the insert, update, and delete commands. Like most other commands, findAndModify always returns a result (even if just {ok:1}).

The CRUD spec itself never addresses the wire protocol for the underlying commands, which is why I didn't bother to talk about this in the spec changes associated with this ticket.

My recommendation is to specify as standard the latter two behaviors, and entirely remove any restrictions on use of hint with unacknowledged write concern, since that write concern will be ignored according to the latter two behaviors.

I think it's still sensible to prohibit hint and {w:0} when we know hint is unsupported. This was originally requested as a safeguard for users since they might otherwise never realize their command failed to execute, and the purpose of this issue was simply to relax that restriction when drivers know that hint is supported by the server.

With all that said, I'm failing to see how $$unsetOrMatches would be problematic for drivers. Whether a driver decides to return null or nothing at all (for languages that make such a distinction), that assertion should be satisfied.

Comment by Githook User [ 31/Aug/21 ]

Author:

{'name': 'Jeremy Mikola', 'email': 'jmikola@gmail.com', 'username': 'jmikola'}

Message: DRIVERS-1340: Allow w:0 and hint if supported by server (#1057)

update supports hint on 4.2+. delete and findAndModify support hint on 4.4+.

Add APM assertions for successful commands to ensure that w:0 is actually used.

Co-authored-by: Benjamin Rewis <32186188+benjirewis@users.noreply.github.com>
Branch: master
https://github.com/mongodb/specifications/commit/59a5dadb2eeab8bba5a7d9a9acd47aa8c1d594cf

Comment by Jeremy Mikola [ 26/Aug/21 ]

Cross-referencing with the related tickets for introducing support for hint on update, delete, and findAndModify comands. The spec tests for this issue will depend on those features being implemented, so it makes sense to model this as an issue dependency.

Comment by Jeremy Mikola [ 26/Aug/21 ]

https://github.com/mongodb/specifications/pull/1057

Generated at Thu Feb 08 08:23:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.