[DRIVERS-1995] Do not error when parsing change stream event documents Created: 29/Nov/21 Updated: 21/May/22 |
|
| Status: | Implementing |
| Project: | Drivers |
| Component/s: | Change Streams |
| Fix Version/s: | None |
| Type: | Spec Change | Priority: | Major - P3 |
| Reporter: | Kevin Albertson | Assignee: | James Kovacs |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | spec-change | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Driver Changes: | Needed | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Quarter: | FY23Q1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Downstream Changes Summary: | Updated the Change Streams spec and added new unified tests. Drivers should sync unified change stream tests with dbed22d and 3cb20c1. Note: drivers that have yet to implement |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Engineering Lead: | |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Product Manager: | |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Driver Compliance: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
SummarySome drivers parse change stream documents into language types. Proposed change is to avoid parsing errors of change stream events that prevent:
MotivationSome drivers parse operationType as an enumeration and throw an error if parsing an unrecognized operationType. See this spreadsheet for a list of affected drivers. That behavior necessitated an opt-in flag for adding new change stream events in PM-1950. PM-1950 makes the assumption that introducing new change stream event types does not break user applications. Drivers should permit new fields to allow extension. C# throws on extra fields in the ns subdocument. The ns document could have fields added in the future. E.g. viewOn. One motivation is to avoid opt-in flags when adding events in the future. Another motivation is to allow users to apply a $project stage. The server errors if _id is projected out. Users should be free to add, remove or modify any other fields in the change stream event as they see fit. Who is the affected end user?Permitting unrecognized operationType permits the server to add change stream events with a new operationType without breaking existing user applications. Allowing extra fields permits the server to extend change stream events in the future. Permitting missing / modified fields permits the user to apply $project. How does this affect the end user?If new operationType values result in an error, it requires the server to add new change stream events behind an "opt-in" flag. If there is no "opt-in" flag, user applications will error when a server upgrade occurs before a driver upgrade. Example:
How likely is it that this problem or use case will occur?The query team believes it is likely that new change stream events will be added in the future. If the problem does occur, what are the consequences and how severe are they?If this is not done before PM-1950, the server will need to continue to add change stream events behind additional "opt-in" flags. Is this issue urgent?This ticket should be done on the timeline of PM-1950. That will enable all future events to be behind the one "opt-in" flag introduced in PM-1950. Is this ticket required by a downstream team?Possibly Shell. Is this ticket only for tests?No. This has functional impact. |
| Comments |
| Comment by Githook User [ 09/May/22 ] | |
|
Author: {'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}Message: DRIVERS-1995 Skip failing test on Server Version 5.3 (#1209) | |
| Comment by Steve Silvester [ 12/Apr/22 ] | |
|
Thanks jmikola@mongodb.com, I opened | |
| Comment by Jeremy Mikola [ 12/Apr/22 ] | |
steve.silvester@mongodb.com: Can you clarify exactly what test case(s) reproduce this server assertion failure? If it's only Test unknown operationType MUST NOT err, I might suggest opening a SERVER ticket to inform them of a possible regression and confirm whether the error is expected or not. Since these spec tests are mostly executing watch() operations, you should be able to create a [minimal reproducible example]|https://stackoverflow.com/help/minimal-reproducible-example] using the shell (e.g. db.collection.watch()) to make it easier for the server team to triage/diagnose. Even if the server did add some extra validation that prohibits projecting an unsupported value into the operationType field, I'd expect a more straightforward error message than what you're seeing. A blanket prohibition on projecting that field could even be preferable (similar to the prohibition on projecting out the _id field IIRC). | |
| Comment by Steve Silvester [ 12/Apr/22 ] | |
|
james.kovacs@mongodb.com, I'm trying to implement this for the Python driver in pymongo. We're not validating the operationType as expected, but with the latest and rapid versions of the server we're getting an error back:
| |
| Comment by Githook User [ 06/Apr/22 ] | |
|
Author: {'name': 'BorisDog', 'email': 'BorisDog@users.noreply.github.com', 'username': 'BorisDog'}Message: DRIVERS-1995: Restrict "test server error on projecting out _id" test to 4.2 an higher (#1172) | |
| Comment by Jeremy Mikola [ 31/Mar/22 ] | |
|
Adding "has to be done after" | |
| Comment by Githook User [ 31/Mar/22 ] | |
|
Author: {'name': 'James Kovacs', 'email': 'jkovacs@post.harvard.edu', 'username': 'JamesKovacs'}Message: DRIVERS-1995: Do not error when parsing change stream event documents (#1164) |