[SERVER-45201] Implicit collection creation from createIndexes can stall replication on secondaries by circumventing thread pool on primary Created: 17/Dec/19  Updated: 29/Oct/23  Resolved: 14/Jan/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.3.3
Fix Version/s: 4.3.3

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
Related
related to SERVER-45347 2-phase index build on empty collecti... Closed
related to SERVER-46668 createIndexes command does not honor ... Closed
related to SERVER-38961 Remove override for causally consiste... Closed
is related to SERVER-45718 Index builder can skip index building... Closed
is related to SERVER-46603 disallow empty collection index build... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2020-01-13, Execution Team 2020-01-27, Execution Team 2019-12-30
Participants:
Linked BF Score: 43

 Description   

Implicit collection creation bypasses the IndexBuildsCoordinator thread pool on the primary and replicates "startIndexBuild" oplog entries directly. The thread pool size is 10 on both primaries and secondaries, and is used to limit resource usage. Each additional index build will block until a thread resource is available.

This is problematic in the following scenario:

  • Start, but do not complete 10 (or the current thread pool size) index builds on the primary.
  • Using createIndexes, implicitly create a collection as of SERVER-44405. This will replicate a "startIndexBuild" oplog entry, but will block on the secondary because all thread pool resources are in use. None of the in-progress index builds can commit because replication is now blocked.

This results in a deadlock, which stalls replication on the secondary



 Comments   
Comment by Githook User [ 14/Jan/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-45201 add createIndexes fast path for index builds on empty collections
Branch: master
https://github.com/mongodb/mongo/commit/458857bdc28f8e60f62992d0c66b0ab468c179eb

Comment by Githook User [ 13/Jan/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-45201 fix index tests to avoid empty collection optimization

These tests rely on fail points that work only with index builds on non-empty collections.
Branch: master
https://github.com/mongodb/mongo/commit/b7d82b16595c61e2a97efe0588ba0b584084d55f

Comment by Benety Goh [ 18/Dec/19 ]

There's a potential for optimizing the empty collection case in this part of IndexBuildsCorodinator::_filterSpecsAndRegisterBuild(). This will also improve the behavior for the createIndexes command on a primary node on an empty collection.

Comment by Louis Williams [ 17/Dec/19 ]

One approach could be to use the same logic on the secondary to decide whether or not to circumvent the thread pool on the primary.

Generated at Thu Feb 08 05:08:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.