[SERVER-68000] Optimize index used to support query-based reopening Created: 12/Jul/22 Updated: 21/Mar/23 Resolved: 14/Dec/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Dan Larkin-York | Assignee: | Dan Larkin-York |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Sprint: | Execution Team 2022-11-28, Execution Team 2022-12-26 | ||||||||||||
| Participants: | |||||||||||||
| Linked BF Score: | 5 | ||||||||||||
| Description |
|
The index we build by default to support query-based reopening can lead to about a 10% decrease in throughput for bulk insert workloads, and up to 30% increase in median latency for individual inserts. We can probably improve this by building the index directly on the system.buckets collection over {meta: 1, control.min.time: 1} rather than using the standard rewrite ({meta: 1, control.min.time: 1, control.max.time: 1}) to decrease the frequency of index key updates. Additionally, we could potentially skip calculating index keys if we know the underlying fields haven't changed. |
| Comments |
| Comment by Dan Larkin-York [ 14/Dec/22 ] |
|
Using the additional index as-is enables a number of query optimizations that may go away if we were to change the index definition. Additionally, calculating the index keys has at most a modest effect on the write throughput for time series collections, so we will not be investigating any special-cased optimizations at this time. |