[SERVER-57483] Results from $lookup stage are not cached for a resharding operation's sub-pipeline Created: 07/Jun/21 Updated: 29/Oct/23 Resolved: 10/Jun/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 5.0.0-rc2, 5.1.0-rc0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Max Hirschhorn | Assignee: | Max Hirschhorn |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | PM-234-M3, PM-234-T-data-clone, post-rc0 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v5.0
|
||||||||||||||||||||||||||||||||||||||||
| Sprint: | Sharding 2021-06-14 | ||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||
| Story Points: | 1 | ||||||||||||||||||||||||||||||||||||||||
| Description |
|
DocumentSourceLookUp::buildPipeline() calls ShardServerProcessInterface::attachCursorSourceToPipeline() which calls sharded_agg_helpers::attachCursorToPipeline() and always serializes and re-parses, even when the pipeline may execute locally on the shard via ShardServerProcessInterface::attachCursorSourceToPipelineForLocalRead(). Because the $sequentialCache stage omits itself from the serialized pipeline, the parsed pipeline which executes locally on the shard won't ever use the cache. |
| Comments |
| Comment by Vivian Ge (Inactive) [ 06/Oct/21 ] | ||||||||||||||||||||||||||||||
|
Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you! | ||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 15/Jun/21 ] | ||||||||||||||||||||||||||||||
|
Note for posterity: ted.tuckman, pointed out that the $match stages in the ReshardingCollectionCloner's $lookup pipeline are being coalesced from the call to Pipeline::optimizePipeline() and causing the $sequentialCache to be removed (SERVER-57705). I hadn't noticed this in my performance testing because I had also been including my changes for | ||||||||||||||||||||||||||||||
| Comment by Githook User [ 10/Jun/21 ] | ||||||||||||||||||||||||||||||
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: (cherry picked from commit 0c046b785bd8d48114f627e105c7215e4734c6db) | ||||||||||||||||||||||||||||||
| Comment by Githook User [ 10/Jun/21 ] | ||||||||||||||||||||||||||||||
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: | ||||||||||||||||||||||||||||||
| Comment by Charlie Swanson [ 09/Jun/21 ] | ||||||||||||||||||||||||||||||
|
max.hirschhorn we discussed you sending us a CR for a patch which is similar to what you outlined in your comment, so I'm assigning this to you. I filed | ||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 07/Jun/21 ] | ||||||||||||||||||||||||||||||
|
One solution could be to avoid calling Pipeline::clone() when the executes locally on the shard. This wouldn't help when the server is running with --setParameter internalQueryAllowShardedLookup=true but that isn't an officially supported configuration anyhow.
|