[SERVER-19745] Throttle concurrent index builds in WiredTiger Created: 04/Aug/15  Updated: 06/Dec/22  Resolved: 23/May/18

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage, WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Andrew Ryder (Inactive) Assignee: Backlog - Storage Execution Team
Resolution: Won't Fix Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-30939 Multiple index builds on same collect... Closed
Assigned Teams:
Storage Execution
Participants:

 Description   

Whenever the case occurs that an index build could occur simultaneously (initial sync, or via oplog using background indexes) applying to the same collection, MMAP builds the indexes in serial, while WiredTiger builds the indexes in parallel (no observed limit).

This has performance implications in both directions.

  • MMAP does not make use of available hardware beyond the disk IO for a single index (despite the other indexes potentially requiring the same data).
  • WiredTiger can overrun the capacity of a lower powered machine (a hidden secondary for backup or analytic purposes) and end up thrashing to cause significant stalling.

This ticket is to request a review of the WiredTiger behavior to consider if some feedback on the hardware can be taken into consideration when building indexes in parallel.



 Comments   
Comment by Eric Milkie [ 22/May/18 ]

In the past three years we've made strides at reducing thrashing and stalling when WiredTiger is under load. We're not planning on doing anything specific for throttling index builds versus any other heavy operations such as collection scans, index scans, or heavy concurrent write workloads.

Generated at Thu Feb 08 03:51:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.