Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32496

Start tailing from the beginning of changes history

    • Query Execution
    • Minor Change

      As far as I understand currently, change streams allows to tail either from resumeAfter token or from the tail of the stream.

      For many uses that's not enough, we may want to resume from the beginning if a client is not in possession of a resume token.

      Something like {{

      {resumeAfter: 'oldest'}

      }} for the $changeStream would be great. Although the post-update lookup would lack proper semantic for old events, streaming ETL with aggregation in external pipelines would definitely benefit from that.

      It's also possible to return always the currently available version of the document in the lookup field.

      I believe it would simplify the initial ingestion of events without resorting to other means (like starting the $changeStream the first time and touching all the documents).

      In case the $changeStream is relying on oplog.rs then ideally the same kind of option should be applied as a fallback for a token referring to an operation that has been truncated from the oplog (restart from the first operation found or from the tail?).

      Such a feature is quite important to build a reliable ETL that ensures at-least-once semantic.

            Assignee:
            backlog-query-execution [DO NOT USE] Backlog - Query Execution
            Reporter:
            mcasimirri Maurizio Casimirri
            Votes:
            8 Vote for this issue
            Watchers:
            28 Start watching this issue

              Created:
              Updated: