[SERVER-40317] $facet execution has no limit on how much memory it can consume Created: 22/Mar/19  Updated: 29/Oct/23  Resolved: 14/Aug/20

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 3.6.20, 4.7.0, 4.2.10, 4.4.2, 4.0.21

Type: Bug Priority: Major - P3
Reporter: Justin Seyster Assignee: David Storch
Resolution: Fixed Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-49506 Excessive memory being used in pipeli... Closed
Related
related to SERVER-49506 Excessive memory being used in pipeli... Closed
related to SERVER-59837 Improve $facet error message when exc... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4, v4.2, v4.0, v3.6
Sprint: Query 2020-08-24
Participants:
Case:
Linked BF Score: 21

 Description   

A $facet always outputs one document but it completely executes all of its pipelines no matter how large they are. To avoid unreasonable memory consumption, the $facet should return an error as soon as it has >100MB of results, rather than waiting until it's computed all of its data.

Note that the $facet's output document is allowed to be up to 100MB large if it is an intermediate result. However, all documents produced by an aggregation pipeline's result set must be 16MB or less due to the BSON size limit. So the 100MB limit applies to the output document produced by $facet, but the pipeline will still fail unless the size of this document is subsequently reduced to 16MB or less.

Also, note that the value of allowDiskUse does not affect the behavior implemented under this ticket, since the $facet stage's current implementation cannot spill to disk.



 Comments   
Comment by Githook User [ 11/Sep/20 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-40317 Fail query when $facet intermediate output exceeds 100MB

Co-authored-by: Justin Seyster <justin.seyster@mongodb.com>
Co-authored-by: Jacob Evans <jacob.evans@10gen.com>
(cherry picked from commit e0bbfe119331514119c6f573eac9998ccf2a3dbe)
Branch: v4.4
https://github.com/mongodb/mongo/commit/05cb6dc742fe17b55cf2851f67d8c88e00868efb

Comment by Githook User [ 25/Aug/20 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-40317 Fail query when $facet intermediate output exceeds 100MB

Co-authored-by: Justin Seyster <justin.seyster@mongodb.com>
Co-authored-by: Jacob Evans <jacob.evans@10gen.com>
(cherry picked from commit 1e62dde76309c37f8aae937f8782431177b90477)
Branch: v3.6
https://github.com/mongodb/mongo/commit/bfecd6ba7c6bd490c19a7bc6bd0e97b577711a6b

Comment by Githook User [ 25/Aug/20 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-40317 Fail query when $facet intermediate output exceeds 100MB

Co-authored-by: Justin Seyster <justin.seyster@mongodb.com>
Co-authored-by: Jacob Evans <jacob.evans@10gen.com>
(cherry picked from commit 6b6e686be20ed63446111445982513ebfb94a8cb)
Branch: v4.0
https://github.com/mongodb/mongo/commit/1e62dde76309c37f8aae937f8782431177b90477

Comment by Githook User [ 24/Aug/20 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-40317 Fail query when $facet intermediate output exceeds 100MB

Co-authored-by: Justin Seyster <justin.seyster@mongodb.com>
Co-authored-by: Jacob Evans <jacob.evans@10gen.com>
(cherry picked from commit e0bbfe119331514119c6f573eac9998ccf2a3dbe)
Branch: v4.2
https://github.com/mongodb/mongo/commit/6b6e686be20ed63446111445982513ebfb94a8cb

Comment by Githook User [ 14/Aug/20 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-40317 Fail query when $facet intermediate output exceeds 100MB

Co-authored-by: Justin Seyster <justin.seyster@mongodb.com>
Co-authored-by: Jacob Evans <jacob.evans@10gen.com>
Branch: master
https://github.com/mongodb/mongo/commit/e0bbfe119331514119c6f573eac9998ccf2a3dbe

Comment by Jacob Evans [ 06/Jun/19 ]

We decided that this was lower value than rethinking our $facet passthrough which this would necessitate. The passthrough would hit the proposed limit many times and require engineering effort to manually disable with blacklisting.

Generated at Thu Feb 08 04:54:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.