[SERVER-21307] Replicated DDL (catalog) operation during background index build blocks reads/writes Created: 05/Nov/15  Updated: 20/Mar/23  Resolved: 27/Feb/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.0.7, 3.2.0-rc2
Fix Version/s: 4.3.4

Type: Bug Priority: Major - P3
Reporter: James Wahlin Assignee: Benety Goh
Resolution: Done Votes: 16
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File repro.js    
Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-25841 Dropping an index while another index... Closed
is duplicated by SERVER-32398 Primary freezes during background ind... Closed
is duplicated by SERVER-34686 `dropIndex` blocks secondary if it is... Closed
is duplicated by SERVER-41933 secondary block during background ind... Closed
is duplicated by SERVER-31546 Write lock on one database causes loc... Closed
Related
related to SERVER-37726 Make dropIndexes abort in-progress in... Closed
related to SERVER-32033 Allow dropIndex to succeed when index... Closed
is related to SERVER-34192 Secondary reads during batch applicat... Closed
is related to SERVER-34383 FSM test of secondary reads during op... Closed
is related to SERVER-37726 Make dropIndexes abort in-progress in... Closed
is related to SERVER-43692 enable two phase index builds by default Closed
is related to DOCS-14687 Remove the warning added in DOCSP-941... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Reproduction script attached. Tested on MMAPv1.

Sprint: Execution Team 2020-03-09
Participants:
Case:

 Description   

A replicated dropIndexes operation, performed on a collection that is currently performing a background index build, will block replication and all read operations to that member for the duration of the index build.

  1. Perform a background index build.
  2. Once completed on the primary and once the background build has started on a secondary, run dropIndexes against the primary.
  3. The dropIndexes being a quick operation will replicate to the secondary while the index build is in progress (for any long running index build).
  4. Once the secondary receives the dropIndexes via replication, reads will be blocked across all namespaces and replication will halt. This persists until the background index build completes and the dropIndexes is allowed to execute.


 Comments   
Comment by Benety Goh [ 27/Feb/20 ]

Underlying issues fixed in other SERVER tickets.

The test referenced in git commit has been updated to match the user commands in the reproduction script attached to this ticket.

Comment by Githook User [ 27/Feb/20 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-21307 add secondary reads to dropIndexes and index builds test
Branch: master
https://github.com/mongodb/mongo/commit/5e57c0b0f7505035c37179d100fdd43ef2b6cc36

Comment by Benety Goh [ 25/Feb/20 ]

The scenario described in the attached repro.js is no longer applicable:

  • var ps = startParallelShell... The reads on the secondary will no longer be blocked due to SERVER-34192.
  • pcol.dropIndexes(); This will abort the index build and complete successfully without any blocking replication on the secondary. See SERVER-37726.
Comment by Benety Goh [ 25/Feb/20 ]

With SERVER-43692, by replicating two oplog entries (startIndexBuild and commitIndexBuild/abortIndexBuild), we no longer have to wait for the index build to complete on the primary before starting it on the secondary.

This change in replica set semantics for index builds directly supports the work in SERVER-37726 which allows the dropIndexes command to interrupt and abort in-progress index builds across the replica set.

Comment by Eric Milkie [ 26/Apr/18 ]

After the completion of SERVER-34192, this problem only affects replication writes on secondaries; reads are no longer affected.

Comment by Asya Kamsky [ 07/Dec/17 ]

We are working on a larger project to improve handling of index builds/drops on all nodes which should resolve this issue.

Comment by Eric Milkie [ 16/Oct/17 ]

Some metadata operations require no background index builds in any collection in the database, while others only require no background index build in the collection of the operation.
The dropIndex command checks for in-progress background index builds in the same collection only.

Comment by Chuy Martinez [ 12/Oct/17 ]

Does this problem only happen when running an index build and dropping an index (or any other metadata update as per above) in the same collection? Or would this affect any simultaneous createIndex/dropIndex regardless of the collection?

Comment by Eric Milkie [ 05/Nov/15 ]

I believe this is actually true for any metadata operation that affects the index build, not just dropIndexes.

Generated at Thu Feb 08 03:56:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.