[COMPASS-7012] 🚨 Switching aggregation pipeline to text view causes it to run! Created: 06/Jul/23 Updated: 29/Oct/23 Resolved: 12/Sep/23 |
|
| Status: | Closed |
| Project: | Compass |
| Component/s: | Explain |
| Affects Version/s: | 1.38.2 |
| Fix Version/s: | 1.40.0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Jack Newberry | Assignee: | Sergey Petushkov |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
OS: Mac, m2 |
||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Story Points: | 3 | ||||
| Documentation Changes: | Needed | ||||
| Documentation Changes Summary: | When requesting explain plan for aggregation pipelines, out stages like $merge and $out will be ignored and will not show in explain plan |
||||
| Sprint: | Iteration Juravenator | ||||
| Description |
Problem Statement/Rationale
Please be sure to attach relevant logs with any sensitive data redacted. Steps to Reproduce
Expected ResultsWhat do you expect to happen? Aggregation pipelines are high-risk data transformations. Compass should only run aggregation pipelines after an explicit user action. Actual ResultsWhat do you observe is happening? Compass runs aggregation pipelines (including $merge) without the user pressing 'run'. Additional NotesI have noticed this for several pipelines across the last 1-2 weeks. This when I started using the code view. // Pipeline on a collection called "integrations" [ ``` |
| Comments |
| Comment by Githook User [ 25/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 18/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 14/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 14/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 13/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 13/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 12/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Githook User [ 12/Sep/23 ] |
|
Author: {'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}Message: fix(aggregations, explain-plan): remove out stages before running explain plan fix(aggregations, explain-plan): remove out stages before running explain plan |
| Comment by Sergey Petushkov [ 11/Sep/23 ] |
|
Hey jack@healthtech1.uk! In this particular case what happens is Compass is trying to proactively get information about index usage for aggregation to show a hint when working with aggregations and suggest that index might improve the performance for a particular query. For that we're running explain plan on aggregation in queryPlanner mode. This explain plan mode exists specifically so that query / aggregation is not executed and only planned way of executing the query is returned, but it doesn't look like CosmosDB respects that part of the spec. Even though it's not correct that CosmosDB is executing the pipeline when getting explain in queryPlanner mode, we will work on addressing the issue on our side to prevent potential data corruption from happening, thanks for bringing this to our attention! |
| Comment by Jack Newberry [ 06/Sep/23 ] |
|
Thank you for investigating!
Do you think the explain runs the merge because of how Compass orchestrates the explain, or because of a bug in CosmosDB? i.e. is this something which would be in your power to fix?
I am using Compass 1.39.3 (1.39.3). I always accept automatic updates. |
| Comment by Le Roux Bodenstein [ 01/Sep/23 ] |
|
OK we think we know what's happening. Compass does an explain on the aggregation for proactive performance insights and cosmosdb runs the merge when performing the explain. |
| Comment by Le Roux Bodenstein [ 01/Sep/23 ] |
|
I watched your video and followed your instructions to the letter and cannot get mongodb to update the collection. I've traced the pipelines that get run everywhere in the process and it never sends a $merge stage. Is this not some cosmosdb bug? Do you mind trying to recreate this with a mongodb server? I'm wondering if $merge even has anything to do with it. Do you mind confirming which version of compass you're running? |
| Comment by Jack Newberry [ 01/Sep/23 ] |
|
Hi Le Roux, I have recorded a video demonstrating the issue: https://www.loom.com/share/bf9cf7eb0f8c407cb11f124d8c8c06ad?sid=e0286205-0a51-4802-b08f-6b8469524160.
Steps to reproduce:
Please let me know if you want me to try any other actions, or try any actions and send you the logs from them. I no longer think this is related to switching views from text to stages view. It seems that the pipeline runs as soon as a valid $merge is added. |
| Comment by Le Roux Bodenstein [ 01/Sep/23 ] |
|
Hi Jack The logs don't show that the $merge ran, just $set. Compass detects $merge and $out and handles that specially. $merge can only run as the last stage and any aggregation we build for updating the preview would have a $limit stage at the end anyway. By default the preview gets updated automatically by running an aggregation, are you sure the preview wasn't just turned on and that's what you're seeing? |
| Comment by PM Bot [ 06/Jul/23 ] |
|
Hello jack@healthtech1.uk, thank you for reaching out to us! The team will review your issue and get back to you soon as soon as possible. Please review your issue to ensure you've included your environment details and have attached relevant logs (with any sensitive data redacted), so that we're best able to provide you a timely and thorough response. Thanks again! |