[SERVER-54082] Explore unpacking of buckets on mongos in a sharded environment Created: 27/Jan/21  Updated: 27/Dec/23

Status: Backlog
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Eric Cox (Inactive) Assignee: Backlog - Query Integration
Resolution: Unresolved Votes: 0
Labels: qi-timeseries, qopt-team, query-skunkworks
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-61187 Add evergreen targets for TSBS on sha... Closed
Related
Assigned Teams:
Query Integration
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Under PM-1945 the simplest approach to allow for $_internalUnpackBucket to work in a sharded environment was to push $_internalUnpackBucket onto the shards and send the unpacked measurements to mongos. This ticket encapsulates work that could be a good optimization to reduce network traffic between mongod and mongos. We should explore when the tipping point would be given metadata size, the number of included/excluded projections when it would be cheaper to send buckets between mongos and mongod and unpack on the merging node.



 Comments   
Comment by David Percy [ 04/Nov/21 ]

Maybe this should wait until we have sharded time-series benchmarks running in CI (SERVER-61187).

Comment by David Percy [ 02/Nov/21 ]

I was imagining we would do something similar to this $unwind optimization.

We might want to move other stages to the mongos too. If you have [$match, $_internalUnpackBucket, $set] then you might want to run [$_internalUnpackBucket, $set] on mongos, even though $_internalUnpackBucket isn't the last stage.

Moving a stage from the shards to mongos would be bad if that stage's output will be much bigger than its input. We might not want to move $match in case it's very selective.

Comment by David Percy [ 15/Oct/21 ]

when the tipping point would be given metadata size, the number of included/excluded projections

In the future, if we create a more compressed bucket format, it would be even more beneficial to do the unpacking on mongos.

Comment by Charlie Swanson [ 04/Feb/21 ]

For more details, I was imagining we would do something similar to this $unwind optimization.

Comment by Eric Cox (Inactive) [ 28/Jan/21 ]

david.storch Yes indeed this is future work, sounds good about moving it. I only put it here because I wasn't absolutely sure about what epic to use or if it should go only in the backlog.

Comment by David Storch [ 28/Jan/21 ]

eric.cox I think we agreed that this would be future work, right? For that reason, I'm going to remove this ticket from our current project and put it into the team's triage queue.

Generated at Thu Feb 08 05:32:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.