[DRIVERS-1995] Do not error when parsing change stream event documents Created: 29/Nov/21  Updated: 21/May/22

Status: Implementing
Project: Drivers
Component/s: Change Streams
Fix Version/s: None

Type: Spec Change Priority: Major - P3
Reporter: Kevin Albertson Assignee: James Kovacs
Resolution: Unresolved Votes: 0
Labels: spec-change
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt Dependency
has to be done after SERVER-65497 Server Error when using a new structu... Closed
has to be done after DRIVERS-2231 ChangeStream Spec: fullDocument field... Closed
Issue split
split to PHPLIB-829 Do not error when parsing change stre... Closed
split to RUST-1168 Do not error when parsing change stre... Closed
split to JAVA-4470 Do not error when parsing change stre... Closed
split to CDRIVER-4279 Do not error when parsing change stre... Backlog
split to MOTOR-880 Do not error when parsing change stre... Closed
split to NODE-3940 Do not error when parsing change stre... Closed
split to PYTHON-3095 Do not error when parsing change stre... Closed
split to RUBY-2893 Do not error when parsing change stre... Closed
split to CSHARP-4036 Do not error when parsing change stre... Closed
split to CXX-2440 Do not error when parsing change stre... Closed
split to GODRIVER-2296 Do not error when parsing change stre... Closed
Related
related to DRIVERS-2261 Remove $$unsetOrMatches in poc-change... Backlog
Driver Changes: Needed
Quarter: FY23Q1
Downstream Changes Summary:

Updated the Change Streams spec and added new unified tests. Drivers should sync unified change stream tests with dbed22d and 3cb20c1.

Note: drivers that have yet to implement DRIVERS-2231 should do so before this ticket; otherwise, commandStartedEvent assertions may fail due to an unexpected fullDocument field appearing in observed events.

Engineering Lead: James Kovacs James Kovacs
Product Manager: Rachelle Palmer Rachelle Palmer
Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4279 Backlog
CXX-2440 Fixed 3.7.0
CSHARP-4036 Done 2.16.0
GODRIVER-2296 Done
JAVA-4470 Fixed 4.6.0
NODE-3940 Done
MOTOR-880 Duplicate
PYTHON-3095 Duplicate
PHPLIB-829 Fixed 1.13.0-beta1, 1.13.0
RUBY-2893 Fixed 2.18.0
RUST-1168 Fixed 2.3.0
SWIFT-1474 Won't Do

 Description   

Summary

Some drivers parse change stream documents into language types.

Proposed change is to avoid parsing errors of change stream events that prevent:

  • Server from adding new values to operationType.
  • Server from adding fields.
  • Users from applying projection.

Motivation

Some drivers parse operationType as an enumeration and throw an error if parsing an unrecognized operationType. See this spreadsheet for a list of affected drivers.

That behavior necessitated an opt-in flag for adding new change stream events in PM-1950. PM-1950 makes the assumption that introducing new change stream event types does not break user applications.

Drivers should permit new fields to allow extension. C# throws on extra fields in the ns subdocument. The ns document could have fields added in the future. E.g. viewOn.

One motivation is to avoid opt-in flags when adding events in the future.

Another motivation is to allow users to apply a $project stage. The server errors if _id is projected out. Users should be free to add, remove or modify any other fields in the change stream event as they see fit.

Who is the affected end user?

Permitting unrecognized operationType permits the server to add change stream events with a new operationType without breaking existing user applications. Allowing extra fields permits the server to extend change stream events in the future. Permitting missing / modified fields permits the user to apply $project.

How does this affect the end user?

If new operationType values result in an error, it requires the server to add new change stream events behind an "opt-in" flag.

If there is no "opt-in" flag, user applications will error when a server upgrade occurs before a driver upgrade. Example:

  • Atlas upgrades the backing server
  • User is stuck on an old driver that does not recognize the new operationType
  • Change stream iteration errors when the new event is produced.

How likely is it that this problem or use case will occur?

The query team believes it is likely that new change stream events will be added in the future.

If the problem does occur, what are the consequences and how severe are they?

If this is not done before PM-1950, the server will need to continue to add change stream events behind additional "opt-in" flags.

Is this issue urgent?

This ticket should be done on the timeline of PM-1950. That will enable all future events to be behind the one "opt-in" flag introduced in PM-1950.

Is this ticket required by a downstream team?

Possibly Shell.

Is this ticket only for tests?

No. This has functional impact.



 Comments   
Comment by Githook User [ 09/May/22 ]

Author:

{'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}

Message: DRIVERS-1995 Skip failing test on Server Version 5.3 (#1209)
Branch: master
https://github.com/mongodb/specifications/commit/92d77a6d1b9d54e60f261d56b7eb85ae295523b0

Comment by Steve Silvester [ 12/Apr/22 ]

Thanks jmikola@mongodb.com, I opened SERVER-65497

Comment by Jeremy Mikola [ 12/Apr/22 ]

We're not validating the operationType as expected, but with the latest and rapid versions of the server we're getting an error back:

steve.silvester@mongodb.com: Can you clarify exactly what test case(s) reproduce this server assertion failure? If it's only Test unknown operationType MUST NOT err, I might suggest opening a SERVER ticket to inform them of a possible regression and confirm whether the error is expected or not. Since these spec tests are mostly executing watch() operations, you should be able to create a [minimal reproducible example]|https://stackoverflow.com/help/minimal-reproducible-example] using the shell (e.g. db.collection.watch()) to make it easier for the server team to triage/diagnose.

Even if the server did add some extra validation that prohibits projecting an unsupported value into the operationType field, I'd expect a more straightforward error message than what you're seeing. A blanket prohibition on projecting that field could even be preferable (similar to the prohibition on projecting out the _id field IIRC).

Comment by Steve Silvester [ 12/Apr/22 ]

james.kovacs@mongodb.com, I'm trying to implement this for the Python driver in pymongo. We're not validating the operationType as expected, but with the latest and rapid versions of the server we're getting an error back:

pymongo.errors.OperationFailure: assertion src/mongo/db/exec/document_value/value.h:478, full error: {'ok': 0.0, 'errmsg': 'assertion src/mongo/db/exec/document_value/value.h:478', 'code': 8, 'codeName': 'UnknownError', '$clusterTime': {'clusterTime': Timestamp(1649448273, 1), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}, 'operationTime': Timestamp(1649448273, 1)}

Comment by Githook User [ 06/Apr/22 ]

Author:

{'name': 'BorisDog', 'email': 'BorisDog@users.noreply.github.com', 'username': 'BorisDog'}

Message: DRIVERS-1995: Restrict "test server error on projecting out _id" test to 4.2 an higher (#1172)
Branch: master
https://github.com/mongodb/specifications/commit/3cb20c18bd987096ac404aefb201a3b69a3c2585

Comment by Jeremy Mikola [ 31/Mar/22 ]

Adding "has to be done after" DRIVERS-2231, since drivers that have yet to implement that ticket may run into trouble with the commandStartedEvent assertions. See DRIVERS-2261 for more details.

Comment by Githook User [ 31/Mar/22 ]

Author:

{'name': 'James Kovacs', 'email': 'jkovacs@post.harvard.edu', 'username': 'JamesKovacs'}

Message: DRIVERS-1995: Do not error when parsing change stream event documents (#1164)
Branch: master
https://github.com/mongodb/specifications/commit/dbed22d67ef3b005c73a0a4b0ce576a5ee81b170

Generated at Thu Feb 08 08:24:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.