Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-16641

Investigate changes in SERVER-82436: Add support for $documents in streams pipelines

      Original Downstream Change Summary

      See https://jira.mongodb.org/browse/DOCSP-36628

      Description of Linked Ticket

      The goal in this ticket is to support the equivalent of the $documents stage in stream processor pipelines. A pipeline like below should work:

      [        { $source: { documents: [ {a: 1, timestamp: ...}, ... ], timeField: {$toDate: "$timestamp"}, tsFieldOverride: "_foo" } },
        { $tumblingWindow: ... },
        { $merge: ... }
      ] 

      We want a syntax like

       { $source: { documents: [ {a: 1, timestamp: ...}, ... ], timeField: {$toDate: "$timestamp"}, tsFieldOverride: "_foo" } },  

      Internally, the $documents stage should parse into a new streams Operator that derives SourceOperator. This will require changes in planner.cpp.

      This new Operator should do the other things SourceOperators do: sending data and watermarks to subsequent operators in the streaming dag.

      The new Operator can likely be similar to SampleDataSourceOperator, and hopefully it can derived from GeneratedDataSourceOperator.

      We should add jstests for the new functionality in the streams suite. Feel free to also add unit tests. 

       

            Assignee:
            Unassigned Unassigned
            Reporter:
            backlog-server-pm Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:
              10 weeks, 5 days ago