[SERVER-35671] DatabaseHolderImpl::closeAll can leave catalog in an incomplete state Created: 19/Jun/18  Updated: 29/Oct/23  Resolved: 05/Jul/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.0.1, 4.1.1

Type: Bug Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Ben Judd
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Steps To Reproduce:

Running a replica set with --setParameter enableTestCommands=true:

Shell 1:

function setFailpointBool(failpointName, alwaysOn, times) {
    if (times) {
        return db.adminCommand({configureFailPoint: failpointName, mode: {"times": times}});
    } else if (alwaysOn) {
        return db.adminCommand({configureFailPoint: failpointName, mode: "alwaysOn"});
    } else {
        return db.adminCommand({configureFailPoint: failpointName, mode: "off"});
    }
}
rs.initiate()
setFailpointBool("hangAfterStartingIndexBuildUnlocked", true)
db.coll.ensureIndex({a: 1, b: 1}, {background: true}) // Blocks because of failpoint

Shell 2:

db.adminCommand({restartCatalog: 1}) // fails with message: cannot perform operation: a background operation is currently running for database test
db.adminCommand({restartCatalog: 1})

Sprint: Storage NYC 2018-07-02, Storage NYC 2018-07-16
Participants:
Linked BF Score: 60

 Description   

As `closeAll` iterates and closes each individual database, it checks that the database has no background operations. If that check fails, a uassert is thrown. That uassert reaching the user implies the database is left in a state that is partially closed. In particular the databases returned by `StorageEngine::listDatabases` may not exist in a call to `DatabaseHolder::get`.

Suggested fix: Move the `BackgroundOperation::assertNoBgOpInProgForDb` call to the previous scan that grabs the database names to close. I.e: assert the complete precondition before taking any actions.



 Comments   
Comment by Githook User [ 19/Jul/18 ]

Author:

{'name': 'Ben Judd', 'email': 'ben.judd@10gen.com', 'username': 'Icantjuddle'}

Message: SERVER-35671 Add handling so closeAll, on failure, leaves UUIDCatalog in
consistent state

(cherry picked from commit c0fecea1c3eb40fe6b5affe7cd505b5ce7dc2faa)
Branch: v4.0
https://github.com/mongodb/mongo/commit/a4bda50da86c972e0bb65031eaf5a2b03c03da85

Comment by Githook User [ 05/Jul/18 ]

Author:

{'username': 'Icantjuddle', 'name': 'Ben Judd', 'email': 'ben.judd@10gen.com'}

Message: SERVER-35671 Add handling so closeAll, on failure, leaves UUIDCatalog in
consistent state
Branch: master
https://github.com/mongodb/mongo/commit/c0fecea1c3eb40fe6b5affe7cd505b5ce7dc2faa

Generated at Thu Feb 08 04:40:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.