-
Type: Task
-
Resolution: Duplicate
-
Priority: Minor - P4
-
Affects Version/s: None
-
Component/s: Server
-
Labels:None
Original Downstream Change Summary
See https://jira.mongodb.org/browse/DOCSP-36628
Description of Linked Ticket
The goal in this ticket is to support the equivalent of the $documents stage in stream processor pipelines. A pipeline like below should work:
[ { $source: { documents: [ {a: 1, timestamp: ...}, ... ], timeField: {$toDate: "$timestamp"}, tsFieldOverride: "_foo" } }, { $tumblingWindow: ... }, { $merge: ... } ]
We want a syntax like
{ $source: { documents: [ {a: 1, timestamp: ...}, ... ], timeField: {$toDate: "$timestamp"}, tsFieldOverride: "_foo" } },
Internally, the $documents stage should parse into a new streams Operator that derives SourceOperator. This will require changes in planner.cpp.
This new Operator should do the other things SourceOperators do: sending data and watermarks to subsequent operators in the streaming dag.
The new Operator can likely be similar to SampleDataSourceOperator, and hopefully it can derived from GeneratedDataSourceOperator.
We should add jstests for the new functionality in the streams suite. Feel free to also add unit tests.