Summary

Add option to resume change stream after receiving an "invalidate" event.

Motivation

After a change stream receives an "invalidate" event, the server closes the change stream.

Callers wanting to continue receiving events are required to create a new change stream with the `startAfter` option set to the most recent `resumeToken`.

An opt-in option (e.g. `ChangeStreamOptions.resumeAfterInvalidate`?) to resume using `startAfter` may provide a convenience.

Who is the affected end user?

Change stream users who want to continue listening for events after "invalidate".

How does this affect the end user?

To achieve this behavior, it requires a workaround: Track the resume token. If an "invalidate" event is received, recreate the change stream with the `startAfter` option.

How likely is it that this problem or use case will occur?

Not sure.

If the problem does occur, what are the consequences and how severe are they?

Requires a workaround.

Is this issue urgent?

No. There is a workaround.

Is this ticket required by a downstream team?

Requested by internal team described in this comment.

Is this ticket only for tests?

No.

Acceptance Criteria

Enable change streams to continue listening after receiving an "invalidate" event.

 

---

Original ticket description:

Modifying the example, this behavior reproduces:

• Insert a document to db.coll to create the collection.
• Watch on db.coll.
• Drop db.coll.
• Iterate. Get a “drop” event.
• Iterate. Get an “invalidate” event.
• Iterate. Get an error: “Attempting to resume a change stream using ‘resumeAfter’ is not allowed from an invalidate notification”.The change stream specification describes the expected protocol. Drivers do not inspect events or handle “invalidate” specially.To have a change stream survive after an “invalidate”, I expect it is needed to recreate the change stream with the “startAfter” option.
   
  Kevin Albertson  2 hours ago

I think there is a related bug in the C driver. I do not expect the resume after “invalidate”. Comparing with the Go driver, I expect iterating after “invalidate” should result in no event returned and no error. I plan to file a ticket.
 
matthew.normyle  37 minutes ago

Thank you so much for investigating.

To have a change stream survive after an “invalidate”, I expect it is needed to recreate the change stream with the “startAfter” option.

Yep, for what it’s worth, this problem goes away if I change this  line to startAfter instead of resumeAfter 
 
mongoc-change-stream.c
              &change_stream_doc, "resumeAfter", &stream->resume_token);
mongodb/mongo-c-driver | Added by GitHub
 
Kevin Albertson  25 minutes ago

I expect the Resume Process using resumeAfter rather than startAfter is intentional. After “invalidate”, later events may not be related to previous events. E.g. the collection may have been dropped, and recreated with a different schema. It may no longer be the “same collection”.
 
matthew.normyle  24 minutes ago

Yep, totally makes sense. Perhaps the fix here is to expose an option for this, so the customer can decide what happens. In our case we’d like to keep reading after a drop and invalidate event (startAfter semantics).

matthew.normyle  24 minutes ago

(Also I think startAfter only works for MongoDB >= 4.X)

Kevin Albertson  22 minutes ago

That seems reasonable. If it is wanted, I suggest filing a DRIVERS ticket with the feature request. That may result in a specification change to support that use case.