[SERVER-54082] Explore unpacking of buckets on mongos in a sharded environment Created: 27/Jan/21 Updated: 27/Dec/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Eric Cox (Inactive) | Assignee: | Backlog - Query Integration |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | qi-timeseries, qopt-team, query-skunkworks | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Query Integration
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Under PM-1945 the simplest approach to allow for $_internalUnpackBucket to work in a sharded environment was to push $_internalUnpackBucket onto the shards and send the unpacked measurements to mongos. This ticket encapsulates work that could be a good optimization to reduce network traffic between mongod and mongos. We should explore when the tipping point would be given metadata size, the number of included/excluded projections when it would be cheaper to send buckets between mongos and mongod and unpack on the merging node. |
| Comments |
| Comment by David Percy [ 04/Nov/21 ] |
|
Maybe this should wait until we have sharded time-series benchmarks running in CI ( |
| Comment by David Percy [ 02/Nov/21 ] |
We might want to move other stages to the mongos too. If you have [$match, $_internalUnpackBucket, $set] then you might want to run [$_internalUnpackBucket, $set] on mongos, even though $_internalUnpackBucket isn't the last stage. Moving a stage from the shards to mongos would be bad if that stage's output will be much bigger than its input. We might not want to move $match in case it's very selective. |
| Comment by David Percy [ 15/Oct/21 ] |
In the future, if we create a more compressed bucket format, it would be even more beneficial to do the unpacking on mongos. |
| Comment by Charlie Swanson [ 04/Feb/21 ] |
|
For more details, I was imagining we would do something similar to this $unwind optimization. |
| Comment by Eric Cox (Inactive) [ 28/Jan/21 ] |
|
david.storch Yes indeed this is future work, sounds good about moving it. I only put it here because I wasn't absolutely sure about what epic to use or if it should go only in the backlog. |
| Comment by David Storch [ 28/Jan/21 ] |
|
eric.cox I think we agreed that this would be future work, right? For that reason, I'm going to remove this ticket from our current project and put it into the team's triage queue. |