[SERVER-51889] Introduce new stage to peek / debug a pipeline stage's output Created: 30/Oct/20 Updated: 06/Dec/22 Resolved: 03/Nov/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Minor - P4 |
| Reporter: | Prasad Saya | Assignee: | Backlog - Query Execution |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Assigned Teams: |
Query Execution
|
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Compass has a feature to build an aggregation pipeline. In that interface/tab, as you build each stage of the pipeline - you can preview the transformed document samples in the adjacent part of the screen. A similar feature with the aggregation pipeline in the mongo shell will be useful. Suppose there is a stage called $peek, and it can be placed at any stage of the pipeline and outputs sample transformed documents (one, more than one - perhaps configurable) along with the query output. This will be helpful in debugging and while working with longer pipelines and for the folks who prefer using the command-line shell interface to the Compass GUI. There is a similar feature in Java programming language - in Java Streams API. As the data flows thru a stream's pipeline, the peek method has such a function. For example (as shown in the https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/stream/Stream.html#peek(java.util.function.Consumer) ):
In the above Java code, the peek method outputs the elements after the filter and map operations. You get to see what is happening as the data flows thru the pipeline. Similarly, with the aggregation pipeline:
|
| Comments |
| Comment by Prasad Saya [ 04/Nov/20 ] |
|
From the ticket SERVER-28954 - "I would like to be able to use the $out pipeline embedded into the aggregation in multiple locations to snapshot the current output.". It is the same idea. |
| Comment by Asya Kamsky [ 03/Nov/20 ] |
|
Is this not basically the same as request for $tee and a duplicate of SERVER-28954 ticket? |
| Comment by Craig Homa [ 03/Nov/20 ] |
|
Hey massimiliano.marcon, would this be a feature that you would consider adding to the new shell? |
| Comment by Prasad Saya [ 03/Nov/20 ] |
|
The SERVER-40117 ("$exit" aggregation stage (with $cond operator support)) is about a functionality. That is, it drives the logic of the query. $peek doesn't affect the functionality or the logic of the query - its a debugging aid. Both are of different category. As such, $peek can be used with any aggregation query where one wants to do some debugging. Maybe the $peek name can be replaced with $debug. |
| Comment by Edwin Zhou [ 02/Nov/20 ] |
|
Does SERVER-40117 fit your use case? That ticket's feature request of $exit attempts to remedy a similar issue of debugging or assessing more complicated aggregation pipelines. However, $exit will exit the pipeline if the condition is true, whereas $peek limits a sample and allows the pipeline to continue. Best, Edwin |
| Comment by Edwin Zhou [ 02/Nov/20 ] |
|
Thank you for the detailed example, it sounds like a great idea! We're assigning this ticket to the appropriate team to be evaluated against our currently planned work. Updates will be posted on this ticket as they happen. Best, |