[SERVER-43350] The server crashes when trying to join collections ($ lookup with pipeline). Created: 16/Sep/19  Updated: 29/Oct/23  Resolved: 02/Oct/19

Status: Closed
Project: Core Server
Component/s: Usability
Affects Version/s: 3.6.14, 4.0.12, 4.2.0
Fix Version/s: 3.6.15, 4.0.13, 4.2.1, 4.3.1

Type: Bug Priority: Critical - P2
Reporter: Alexey Glukhov Assignee: Bernard Gorman
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File col info.txt     Text File mongod.log    
Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2, v4.0, v3.6
Sprint: Query 2019-10-07
Participants:

 Description   
CVE-2019-2393

Title: Crash while joining collections with $lookup

Description:
A user authorized to perform database queries may trigger denial of service by issuing specially crafted queries, which use $lookup and collations. This issue affects: MongoDB Inc. MongoDB Server v4.2 versions prior to 4.2.1; v4.0 versions prior to 4.0.13; v3.6 versions prior to 3.6.15.

CVSS score:
This issue's CVSS:3.1 severity is scored at 6.5 using the following scoring metrics:
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H

Affected versions:
MongoDB Inc. MongoDB Server v4.2 versions prior to 4.2.1; v4.0 versions prior to 4.0.13; v3.6 versions prior to 3.6.15.

CWE: CWE-416: Use After Free



 Comments   
Comment by Githook User [ 03/Oct/19 ]

Author:

{'name': 'Bernard Gorman', 'username': 'gormanb', 'email': 'bernard.gorman@mongodb.com'}

Message: SERVER-43350 $lookup with no local default or user-specified collation should explicitly set the simple collation on the foreign expression context

(cherry picked from commit d6133a3a5464fac202f512b0310dfeb200c126f9)
Branch: v3.6
https://github.com/mongodb/mongo/commit/785b41740a216429573a89a5df82f96064965559

Comment by Githook User [ 02/Oct/19 ]

Author:

{'name': 'Bernard Gorman', 'username': 'gormanb', 'email': 'bernard.gorman@mongodb.com'}

Message: SERVER-43350 $lookup with no local default or user-specified collation should explicitly set the simple collation on the foreign expression context

(cherry picked from commit d6133a3a5464fac202f512b0310dfeb200c126f9)
Branch: v4.0
https://github.com/mongodb/mongo/commit/760f653d1ea2f370357e42ffbb8a525c07b56f48

Comment by Githook User [ 02/Oct/19 ]

Author:

{'name': 'Bernard Gorman', 'username': 'gormanb', 'email': 'bernard.gorman@mongodb.com'}

Message: SERVER-43350 $lookup with no local default or user-specified collation should explicitly set the simple collation on the foreign expression context

(cherry picked from commit d6133a3a5464fac202f512b0310dfeb200c126f9)
Branch: v4.2
https://github.com/mongodb/mongo/commit/3970aa26552a170f55ccf1d9d51607e0a911d8c6

Comment by Githook User [ 01/Oct/19 ]

Author:

{'name': 'Bernard Gorman', 'username': 'gormanb', 'email': 'bernard.gorman@mongodb.com'}

Message: SERVER-43350 $lookup with no local default or user-specified collation should explicitly set the simple collation on the foreign expression context
Branch: master
https://github.com/mongodb/mongo/commit/d6133a3a5464fac202f512b0310dfeb200c126f9

Comment by Bernard Gorman [ 27/Sep/19 ]

In the case where we have a pipeline on a collection with no user-defined collation and no default collation, the ExpressionContext will be initialized with a nullptr collator and an empty BSONObj collation spec. When we create the $lookup pipeline stage, we make a copy of this ExpressionContext for use in the foreign pipeline. When we come to actually build the execution machinery, we find our way down to prepareExecution, where the lack of a defined collation spec causes us to adopt the foreign collection's default collation by setting an owned clone on the CanonicalQuery, which in turn is propagated back into the ExpressionContext as an unowned pointer. This is OK as long as there is only one document in the local collection. But if the pipeline is torn down and rebuilt for a second local document, the CanonicalQuery and its owned collator are destroyed, leaving the unowned pointer in the ExpressionContext dangling.

This does not happen if an explicit collation spec is provided by the user - or if we inherited a collator from the local collection, because we serialize the collator into its equivalent BSONObj spec here before creating the CanonicalQuery. The solution is to have $lookup explicitly request the simple collation on the foreign ExpressionContext in the case where the local collator is null and the user did not specify a collation.

Comment by Danny Hatcher (Inactive) [ 16/Sep/19 ]

Thank you Alexey I was able to retrieve the files and I've reproduced your issue.

When the collection in the from portion of an aggregation has a collation but the collection running the aggregation command does not and there are multiple records in the latter, the invariant will occur.

> db.createCollection("orders", {collation:{locale:"en_US"}})
{ "ok" : 1 }
> db.items.insert({a:1})
WriteResult({ "nInserted" : 1 })
> db.items.insert({a:2})
WriteResult({ "nInserted" : 1 })
> db.items.aggregate( [{   $lookup:          {            from: "orders",            let: {id: "$_id"},            pipeline: [               { $match:                  { $expr:                     {$and:                        [                          {$eq: [ "$item_id",  "$$id" ]},                        ]                     }                  }               }            ],            as: "orders"          } }] , {"allowDiskUse": true} )
2019-09-16T16:43:34.228-0400 I  NETWORK  [js] DBClientConnection failed to receive message from 127.0.0.1:27017 - HostUnreachable: Connection closed by peer
2019-09-16T16:43:34.229-0400 E  QUERY    [js] uncaught exception: Error: error doing query: failed: network error while attempting to run command 'aggregate' on host '127.0.0.1:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:169:19
DB.prototype.runReadCommand@src/mongo/shell/db.js:141:12
DB.prototype._runAggregate/doAgg<@src/mongo/shell/db.js:245:60
DB.prototype._runAggregate@src/mongo/shell/db.js:248:17
DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1012:12
@(shell):1:1
2019-09-16T16:43:34.230-0400 I  NETWORK  [js] trying reconnect to 127.0.0.1:27017 failed
2019-09-16T16:43:34.230-0400 I  NETWORK  [js] reconnect 127.0.0.1:27017 failed failed

I will forward this on to the appropriate team to take a closer look.

Comment by Alexey Glukhov [ 16/Sep/19 ]

I`m uploaded two files:
freshdatabase.items.json
freshdatabase.orders.json
but there is no link

Comment by Danny Hatcher (Inactive) [ 16/Sep/19 ]

You can upload everything to our Secure Uploader where things can only be viewed by MongoDB Engineers.

Comment by Alexey Glukhov [ 16/Sep/19 ]

Ок, getCollectionInfos() result uploaded. Can I somehow give you data not for public viewing?

Comment by Danny Hatcher (Inactive) [ 16/Sep/19 ]

Thanks for the repeated attempts in the log; it provides a lot of context. It appears to be related to our concept of Collation which I notice you are not specifying in the sample aggregation you provided. I assume that you have different collations specified at the collection level for both "items" and "orders"? Could you please provide the output of the following command run against the appropriate logical database?

db.getCollectionInfos()

Additionally, if you could provide a few sample documents from each collection I can emulate them in my own testing.

Comment by Alexey Glukhov [ 16/Sep/19 ]

It is also reproduced in version 4.0, in the enterprise and community. In 3.x did not try.
File with today's logs uploaded

Comment by Danny Hatcher (Inactive) [ 16/Sep/19 ]

Have you seen this issue only once or is it reproducible? Have you run the same queries on earlier versions of MongoDB 4.0 and seen the queries succeed? Please provide the full mongod log for the server from the last restart until the crash so that we can investigate.

Generated at Thu Feb 08 05:02:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.