[SERVER-73526] DocumentSourceGroup should spill in debug builds when allowDiskUse is true, not when allowDiskUse is false Created: 01/Feb/23  Updated: 29/Oct/23  Resolved: 02/May/23

Status: Closed
Project: Core Server
Component/s: Query Execution
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: David Storch Assignee: David Storch
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: QE 2023-05-15
Participants:

 Description   

There is longstanding behavior in which DocumentSourceGroup will spill eagerly in debug builds, even when the memory limit is not yet exhausted. (Similar behavior was added in SBE's HashAggStage as part of SERVER-70395.) The goal is to gain extra coverage of the spilling logic in test environments. We currently perform this test-only extraneous spilling only when allowDiskUse is false. See here:

https://github.com/mongodb/mongo/blob/8a1ce1c8c1b902c60b421e14feca681bee735547/src/mongo/db/pipeline/document_source_group_base.cpp#L548

This dates from the days when disk use was disallowed by default. The rationale was that when disk use is allowed, this means that the client has explicitly enabled spilling. and the query is likely trying to test spilling behavior explicitly. In such situations, the logic to spill eagerly did not kick in, as it could interfere with spilling-related tests. On the other hand, when disk use was not explicitly enabled, we would spill in the background, thus ensuring that all of our $group tests unrelated to spilling still work as expected even if spilling occurs.

SERVER-63208 recently made queries allow disk use by default, which rendered the debug build behavior described above stale. We are now only getting additional spilling test coverage in debug builds for queries that pass allowDiskUse:false explicitly. This is presumably quite rare, and therefore we have lost useful test coverage. It is also arguably incorrect to spill to disk in debug builds even if spilling was explicitly prohibited by the client.

The work for this ticket is to invert the logic so that the extra spilling in debug builds kicks in when allowDiskUse is true and not when allowDiskUse is false. This may require adjusting tests that make assertions about spill-related execution stats to either not run against debug builds, or to expect different spill behavior when running against a debug build.



 Comments   
Comment by Githook User [ 02/May/23 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-73526 Invert logic for when DocumentSourceGroup does extra spilling in debug builds

This behavior is intended to provide extra test coverage for
$group spilling. In the past, the extra spilling happened
unless disk use was explicitly enabled. Now that disk use is
allowed by default, it makes more sense to spill unless disk
use is explicitly disallowed.
Branch: master
https://github.com/mongodb/mongo/commit/c4c70867481f1e0a004fe7dd879b4810ddeb29b8

Generated at Thu Feb 08 06:24:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.