-
Type: Improvement
-
Resolution: Unresolved
-
Priority: Minor - P4
-
None
-
Affects Version/s: 3.6.0, 3.6.1
-
Component/s: Aggregation Framework
-
Query Execution
-
Minor Change
-
(copied to CRM)
As far as I understand currently, change streams allows to tail either from resumeAfter token or from the tail of the stream.
For many uses that's not enough, we may want to resume from the beginning if a client is not in possession of a resume token.
Something like {{
{resumeAfter: 'oldest'}}} for the $changeStream would be great. Although the post-update lookup would lack proper semantic for old events, streaming ETL with aggregation in external pipelines would definitely benefit from that.
It's also possible to return always the currently available version of the document in the lookup field.
I believe it would simplify the initial ingestion of events without resorting to other means (like starting the $changeStream the first time and touching all the documents).
In case the $changeStream is relying on oplog.rs then ideally the same kind of option should be applied as a fallback for a token referring to an operation that has been truncated from the oplog (restart from the first operation found or from the tail?).
Such a feature is quite important to build a reliable ETL that ensures at-least-once semantic.
- is duplicated by
-
SERVER-45126 Allow change stream to resume from the earliest oplog entry
- Closed
- is related to
-
SERVER-50903 No error when change stream's startAtOperationTime is not in oplog
- Closed
- related to
-
CSHARP-2146 System.IO.EndOfStreamException when iterating change stream cursor
- Closed