[SERVER-44174] $push and $addToSet should restrict memory usage Created: 23/Oct/19  Updated: 29/Oct/23  Resolved: 09/Nov/19

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 3.6.17, 4.3.1, 4.2.3, 4.0.14

Type: Bug Priority: Major - P3
Reporter: Charlie Swanson Assignee: Charlie Swanson
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Documented
is documented by DOCS-13213 Investigate changes in SERVER-44174: ... Closed
Related
related to SERVER-43944 Accumulators (and possibly expression... Backlog
related to SERVER-73315 Guardrail to prevent an accumulator f... Closed
related to SERVER-44869 Add query knob to control memory limi... Closed
related to SERVER-49027 Extend $group explain execution stats... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2, v4.0, v3.6, v3.4
Sprint: Query 2019-11-18
Participants:
Case:

 Description   

SERVER-43944 describes a more pervasive problem, but it's probably the case that $push and $addToSet are the worst offenders and not too difficult to fix. Let's start with those two and leave SERVER-43944 to track the more general problem which is likely more complex.



 Comments   
Comment by Charlie Swanson [ 29/Jan/20 ]

Hi andrii.litvinov@gmail.com. Yes, in SERVER-44869 we added two server parameters that can be used to control this limit: internalQueryMaxPushBytes and internalQueryMaxAddToSetBytes. Using the mongo shell, you should be able to configure it like so:

> newLimit = 100 * 1024 * 1024;
> db.adminCommand({setParameter: 1, internalQueryMaxAddToSetBytes: newLimit})

Comment by Andrii Litvinov [ 29/Jan/20 ]

Hi, we upgraded to MongoDB 4.2.3 and now facing issue:

Command aggregate failed: $addToSet used too much memory and cannot spill to disk. Memory limit: 104857600 bytes.

even though we use aggregation

{AllowDiskUse = true}

Is there any way to increase the limit so that our aggregation can run successfully as it was before?

Comment by Githook User [ 10/Dec/19 ]

Author:

{'email': 'charlie.swanson@mongodb.com', 'name': 'Charlie Swanson', 'username': 'cswanson310'}

Message: SERVER-44174 $push and $addToSet should restrict memory usage

(cherry picked from commit 504b518b9bd432a1d614d06f004712e70a1a754b)
(cherry picked from commit 2d0ad29a5fa8b328610e69f34aa26802b5ec7cc9)
(cherry picked from commit 83b3c795e154b7456f43269b5d95ac8e2e00d96e)
Branch: v3.6
https://github.com/mongodb/mongo/commit/fba2959a87c424463fdc39ea256f2feee20fa34c

Comment by Githook User [ 09/Dec/19 ]

Author:

{'email': 'charlie.swanson@mongodb.com', 'name': 'Charlie Swanson', 'username': 'cswanson310'}

Message: SERVER-44174 $push and $addToSet should restrict memory usage

(cherry picked from commit 504b518b9bd432a1d614d06f004712e70a1a754b)
Branch: v4.2
https://github.com/mongodb/mongo/commit/9ff501164cd554019dca63f8c5968b6f0cf49a25

Comment by Githook User [ 09/Dec/19 ]

Author:

{'name': 'Charlie Swanson', 'username': 'cswanson310', 'email': 'charlie.swanson@mongodb.com'}

Message: SERVER-44174 $push and $addToSet should restrict memory usage

(cherry picked from commit 504b518b9bd432a1d614d06f004712e70a1a754b)
(cherry picked from commit 2d0ad29a5fa8b328610e69f34aa26802b5ec7cc9)
Branch: v4.0
https://github.com/mongodb/mongo/commit/83b3c795e154b7456f43269b5d95ac8e2e00d96e

Comment by Githook User [ 09/Nov/19 ]

Author:

{'username': 'cswanson310', 'email': 'charlie.swanson@mongodb.com', 'name': 'Charlie Swanson'}

Message: SERVER-44174 $push and $addToSet should restrict memory usage
Branch: master
https://github.com/mongodb/mongo/commit/504b518b9bd432a1d614d06f004712e70a1a754b

Generated at Thu Feb 08 05:05:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.