[SERVER-37720] Allow active index builds during renameCollection Created: 24/Oct/18  Updated: 04/Dec/23

Status: Blocked
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Dianna Hohensee (Inactive) Assignee: Backlog - Storage Execution Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-42242 Acquire locks by collection UUID duri... Closed
depends on SERVER-42487 Acquire locks by UUID in IndexBuildsC... Closed
depends on SERVER-31695 Support queries across collection ren... Backlog
Related
related to SERVER-42494 Deadlock between aggregation pipeline... Closed
Assigned Teams:
Storage Execution
Sprint: Execution Team 2019-06-17, Execution Team 2019-07-01, Execution Team 2019-07-15, Execution Team 2019-07-29, Execution Team 2019-08-12, Execution Team 2019-12-02, Execution Team 2019-12-16, Execution Team 2020-01-13, Execution Team 2020-01-27, Execution Team 2020-02-10, Execution Team 2019-12-30
Participants:
Linked BF Score: 19

 Description   

We expect allowing index builds during rename collection to be fairly straightforward now that SERVER-37443 has made catalog objects survive rename collection. Conceptually, either we should only need to add {ready:false} index catalog entries to those that get updated on rename or it might just might work if the assert that no index builds are running is removed.

This task will be a little exploratory as no has taken a look at the code.



 Comments   
Comment by Gregory Wlodarek [ 27/May/20 ]

We aren’t able to do this ticket just yet as queries do not survive across collection renames. Once SERVER-31695 is finished, we can revisit this ticket.

If we were able to implement this today anyway, it’d probably have to be limited to only allow renaming collections within the same database if active index builds are present on the source collection. This is because index builds are pinned to a collection UUID and renaming collections across databases creates a new collection in the target database with a new collection UUID.

I was able to make the changes to allow renaming collections with active index builds but eventually ran into the problem where the active index builds MultiIndexBlock PlanExecutor was unusable.

I observed the following:

  1. An index build starts on the collection.
  2. The index build creates the PlanExecutor in its MultiIndexBlock block.
  3. A concurrent operation renames the collection.
  4. The index build tries to call getNext() on its PlanExecutor but ends up getting the error below as the query didn't survive the collection rename.

{
    "state": ExecState::FAILURE,
    "err": {
        "ok":0,
        "code":QueryPlanKilled,
        "errmsg":"collection renamed from test.a to test.b. UUID 87131bb9-572b-4c3e-9e99-f7f749ce7f68"
    }
}

Comment by Dianna Hohensee (Inactive) [ 05/Dec/19 ]

A renameCollection cmd with a target that already exists will fail unless the dropTarget is set to true. Therefore, renameCollection with dropTarget set to false should never run into a BackgroundOperationInProgress error. We will continue to retry renameCollection if a BackgroundOperationInProgress is encountered in a different form of the cmd.

I was going to try to JS test that rename across databases, etc., failed with BackgroundOperationInProgress, but I think I'll try to unit test it instead.

Comment by Dianna Hohensee (Inactive) [ 05/Dec/19 ]

This task poses the question of how to handle renameCollection in the testing infrastructure.

Right now, implicitly_retry_background_op_in_progress.js will endlessly retry renameCollection if a background operation in progress error code is returned.

This patch will make it so that renameCollection within a database, where the target collection does not exist, should never run into a background op in prog error. However, renameCollection across databases, or when the target collection exists, can still run into background op in progress errors. The first because we create a new Collection object for rename across databases. The second because a target collection that already exists can have an index build in progress that prevents dropping it.

Comment by Dianna Hohensee (Inactive) [ 06/Aug/19 ]

gregory.wlodarek there's a caveat to this ticket: we are only allowing rename during an index build if the rename is within the same database, and there are no index builds in progress on the target collection namespace.

Comment by Gregory Wlodarek [ 05/Aug/19 ]

dianna.hohensee a BF I diagnosed will be fixed with the work being done here that deadlocks during renameCollection(). The check that asserts no index builds are in progress during rename will go away with the work being done here, fixing the deadlock.

Generated at Thu Feb 08 04:46:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.