[COMPASS-7012] 🚨 Switching aggregation pipeline to text view causes it to run! Created: 06/Jul/23  Updated: 29/Oct/23  Resolved: 12/Sep/23

Status: Closed
Project: Compass
Component/s: Explain
Affects Version/s: 1.38.2
Fix Version/s: 1.40.0

Type: Bug Priority: Critical - P2
Reporter: Jack Newberry Assignee: Sergey Petushkov
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: Mac, m2
node.js / npm versions: 18.16.0, 9.5.1 (Not sure this is relevant)
Compass: Version 1.38.2 (1.38.2)
Additional info:
Running against CosmosDB v4.2 rather than real Mongo (sorry).


Attachments: Text File mongo.txt    
Issue Links:
Documented
Story Points: 3
Documentation Changes: Needed
Documentation Changes Summary:

When requesting explain plan for aggregation pipelines, out stages like $merge and $out will be ignored and will not show in explain plan

Sprint: Iteration Juravenator

 Description   

Problem Statement/Rationale

  • 🚨 Compass sometimes runs aggregation pipelines without the user clicking 'run'.
  • This puts users at significant risk of running pipelines prematurely, this could easily lead to destroying data unintentionally.
  • (I noticed this bug with a dev db, I have not lost data 🤞🏽)

Please be sure to attach relevant logs with any sensitive data redacted.
How to retrieve logs for: Compass; Shell

Steps to Reproduce

  • Creating an aggregation pipeline using the 'stages' view
  • Add some stages, including a final merge step, back into the original collection
  • Take note of what the intermediate preview docs
  • Click to switch to the text view
  • Click back to the stages view
  • Notice that the pipeline has been executed!
    • I think execution occurs when first clicking to switch to code view, but it is possible it occurs when switching back.
  • I never clicked "Run". There was no confirmation modal.

Expected Results

What do you expect to happen?

Aggregation pipelines are high-risk data transformations. Compass should only run aggregation pipelines after an explicit user action.

Actual Results

What do you observe is happening?

Compass runs aggregation pipelines (including $merge) without the user pressing 'run'.

Additional Notes

I have noticed this for several pipelines across the last 1-2 weeks. This when I started using the code view.

This is one example. Notice that the pipeline filtering docs then altering them so they become excluded from the initial $match. This means that looking at the Compass preview documents it is obvious when the pipeline has been executed because all the previews disappear.

// Pipeline on a collection called "integrations"
```

[
  {
    $match:
      /**
       * query: The query in MQL.
       */
      {
        roles: {           $exists: false,         },
      },
  },
  {
    $set:
      /**
       * field: The field name
       * expression: The expression.
       */
      {
        roles: [
          {             tenantId: "*",             role: "member",           },
        ],
      },
  },
  {
    $merge:
      /**
       * into: The target collection.
       * on: Fields to  identify.
       * let: Defined variables.
       * whenMatched: Action for matching docs.
       * whenNotMatched: Action for non-matching docs.
       */
      {         into: "integrations",         on: "_id",         whenMatched: "merge",         whenNotMatched: "fail",       },
  }
]

```



 Comments   
Comment by Githook User [ 25/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: ga-releases
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 18/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: beta-releases
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 14/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: COMPASS-7171-search-e2e-tests
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 14/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: export-to-language-v6
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 13/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: use-user-data-in-pipelines
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 13/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: search-indexes-list
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 12/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: feature/COMPASS-7165
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Githook User [ 12/Sep/23 ]

Author:

{'name': 'Sergey Petushkov', 'email': 'petushkov.sergey@gmail.com', 'username': 'gribnoysup'}

Message: fix(aggregations, explain-plan): remove out stages before running explain plan COMPASS-7012 (#4830)

fix(aggregations, explain-plan): remove out stages before running explain plan
Branch: main
https://github.com/mongodb-js/compass/commit/a390314895b5775ce5b6013d339bf1b26528236d

Comment by Sergey Petushkov [ 11/Sep/23 ]

Hey jack@healthtech1.uk! In this particular case what happens is Compass is trying to proactively get information about index usage for aggregation to show a hint when working with aggregations and suggest that index might improve the performance for a particular query. For that we're running explain plan on aggregation in queryPlanner mode. This explain plan mode exists specifically so that query / aggregation is not executed and only planned way of executing the query is returned, but it doesn't look like CosmosDB respects that part of the spec. Even though it's not correct that CosmosDB is executing the pipeline when getting explain in queryPlanner mode, we will work on addressing the issue on our side to prevent potential data corruption from happening, thanks for bringing this to our attention!

Comment by Jack Newberry [ 06/Sep/23 ]

Thank you for investigating!

 

Do you think the explain runs the merge because of how Compass orchestrates the explain, or because of a bug in CosmosDB? 

i.e. is this something which would be in your power to fix?

 

I am using Compass 1.39.3 (1.39.3). I always accept automatic updates.

Comment by Le Roux Bodenstein [ 01/Sep/23 ]

OK we think we know what's happening. Compass does an explain on the aggregation for proactive performance insights and cosmosdb runs the merge when performing the explain.

Comment by Le Roux Bodenstein [ 01/Sep/23 ]

I watched your video and followed your instructions to the letter and cannot get mongodb to update the collection. I've traced the pipelines that get run everywhere in the process and it never sends a $merge stage.

Is this not some cosmosdb bug? Do you mind trying to recreate this with a mongodb server?

I'm wondering if $merge even has anything to do with it.

Do you mind confirming which version of compass you're running?

Comment by Jack Newberry [ 01/Sep/23 ]

Hi Le Roux,

I have recorded a video demonstrating the issue: https://www.loom.com/share/bf9cf7eb0f8c407cb11f124d8c8c06ad?sid=e0286205-0a51-4802-b08f-6b8469524160

 

Steps to reproduce:

  • record the contents of a collection to check against later
  • create an aggregation pipeline with a $set followed by a $merge stage.
    • I have only tried merge into the same collection, on `_id`
  • do not press "Run"
  • query the contents of the collection
  • see that the $merge has been run, and the change from $set persisted to the collection.

 

Please let me know if you want me to try any other actions, or try any actions and send you the logs from them.

I no longer think this is related to switching views from text to stages view. It seems that the pipeline runs as soon as a valid $merge is added.

Comment by Le Roux Bodenstein [ 01/Sep/23 ]

Hi Jack

The logs don't show that the $merge ran, just $set. Compass detects $merge and $out and handles that specially. $merge can only run as the last stage and any aggregation we build for updating the preview would have a $limit stage at the end anyway.

By default the preview gets updated automatically by running an aggregation, are you sure the preview wasn't just turned on and that's what you're seeing?

Comment by PM Bot [ 06/Jul/23 ]

Hello jack@healthtech1.uk, thank you for reaching out to us! The team will review your issue and get back to you soon as soon as possible.

Please review your issue to ensure you've included your environment details and have attached relevant logs (with any sensitive data redacted), so that we're best able to provide you a timely and thorough response. Thanks again!

Generated at Wed Feb 07 22:45:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.