[SERVER-33272] The DatabaseHolder::close() function no longer requires a global write lock and neither does the dropDatabase command. Created: 12/Feb/18  Updated: 29/Oct/23  Resolved: 09/Oct/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.3.1, 4.2.2

Type: Improvement Priority: Major - P3
Reporter: Eric Milkie Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-34431 Move DatabaseShardingState into its o... Closed
depends on SERVER-43890 Remove two-phase database drops Backlog
is depended on by SERVER-37552 Illegal concurrent access to KVDataba... Closed
is depended on by SERVER-35283 KVStorageEngine::listDatabases() omit... Closed
is depended on by SERVER-42053 proactively drop newly empty database... Closed
Documented
is documented by DOCS-12980 Investigate changes in SERVER-33272: ... Closed
Issue split
split to SERVER-43925 Proactively close newly empty databases Backlog
split from SERVER-43242 Deadlock involving commands acquiring... Closed
Problem/Incident
Related
related to SERVER-42053 proactively drop newly empty database... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.2
Sprint: Execution Team 2019-08-12, Execution Team 2019-08-26, Execution Team 2019-09-09, Execution Team 2019-09-23, Execution Team 2019-10-07, Execution Team 2019-10-21
Participants:
Case:
Linked BF Score: 0

 Description   

We can demote the lock needed for DatabaseHolder::close() from an exclusive global lock to a database exclusive lock as the DatabaseHolder is protected by a mutex. With this, we can also remove the exclusive global locks needed in the dropDatabase command.

Old Description:
Currently, the kv engine does not check to see if a database is empty after a collection drop. Thus, empty databases can persist until mongod restarts (or the catalog is reloaded via replication rollback to stable timestamp). This situation is usually benign, but a database entry's presence does do one thing: it enforces that no other databases that are spelled the same but differ in capitalization can be created.
Because recover-to-timestamp exists, it can result in some replica set nodes having a catalog entry for an empty database while other nodes do not have such an entry. This can result in the ability to crash secondary nodes that have an empty database entry that a primary node does not have - by attempting to create a database with a different case than the existing empty database entry.

To fix this, we can check to see if a database is empty (i.e., no collection entries remain in the catalog) after a collection drop. If it is indeed empty, we can attempt to close the database by locking it in X mode and calling DatabaseHolder::close().



 Comments   
Comment by Githook User [ 16/Oct/19 ]

Author:

{'name': 'Gregory Wlodarek', 'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com'}

Message: SERVER-33272 The DatabaseHolder::close() function no longer requires a global write lock and neither does the dropDatabase command.
Branch: v4.2
https://github.com/mongodb/mongo/commit/1a42e98facd32e981104074d0087ddec55758425

Comment by Gregory Wlodarek [ 09/Oct/19 ]

The work for this ticket has been committed here.

Comment by Gregory Wlodarek [ 09/Oct/19 ]

Please see SERVER-43925 for the continuation of the ticket for proactively close newly empty databases.

Comment by Gregory Wlodarek [ 08/Oct/19 ]

After discussing with the team, we're going to set this ticket aside as there's no urgency for it right now and needs more work to be done before tackling it.
To be able to do this ticket we'll need to do the following beforehand in order:

  • Get rid of readConcernMajority=false.
  • Get rid of two-phase database drops. The database object today doesn't contain any special state except for the isDropPending flag. We can remove the dropDatabase oplog entry and rollback handler.

This is problematic to do today because if we remove the Database in the DatabaseHolder when the last collection is dropped, then we lose our isDropPending flag (a member of the Database object class). This flag prevents new collections from being created on the database while waiting for the second phase of collection drops to be majority committed when using majority read concern.
While waiting for the majority point to be reached, a new collection can be created, which will implicitly open the database with the same name while there are collections still drop pending for that database name. Once the majority point is reached, the newly created collection would be forcefully one phase dropped.

Timeline of events if we implicitly close empty databases today

Client 1:

  • dropDatabase command executed, isDropPending flag set.
  • First phase drop for Coll A
    • Replicate...
  • First phase drop for Coll B
    • Replicate...
    • Close Database in DatabaseHolder since it's empty now.
    • Replicate...

Client 2:

  • Create collection C on the same database (the isDropPending flag would've prevented this).
    • Replicate openDB() and createCollection()...

Client 1:

  • Majority point reached, second phase of drops executed and committed across all nodes.
  • Forcefully one phase drops collection C created by 'Client 2'.

Another issue that would occur is during rollback. When the last collection is first phase dropped and rollback occurs, the Database object will no longer exist in the DatabaseHolder.

Comment by Githook User [ 27/Aug/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: Revert "SERVER-33272 Proactively close newly empty databases"

This reverts commit 40f226b5a9bfb4863268334d287a46fb226a22cf.
Branch: master
https://github.com/mongodb/mongo/commit/077f247b29e9f6c5afecec11bb41858ae95067b6

Comment by Githook User [ 27/Aug/19 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-33272 Proactively close newly empty databases
Branch: master
https://github.com/mongodb/mongo/commit/40f226b5a9bfb4863268334d287a46fb226a22cf

Comment by Githook User [ 23/Aug/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: Revert "SERVER-33272 Proactively close newly empty databases"

This reverts commit b6b81f34516ba7b1472cb1dd319da8785f24ae58.
Branch: master
https://github.com/mongodb/mongo/commit/e51927f743d0b5bf13e13873f7cfcc1cf07c967a

Comment by Githook User [ 23/Aug/19 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-33272 Proactively close newly empty databases
Branch: master
https://github.com/mongodb/mongo/commit/b6b81f34516ba7b1472cb1dd319da8785f24ae58

Comment by Githook User [ 23/Aug/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-33272 The DatabaseHolder::close() function no longer requires a global write lock and neither does the dropDatabase command.
Branch: master
https://github.com/mongodb/mongo/commit/d743246d3c0e35c21b4f1d954bc138df60c47a5a

Comment by Eric Milkie [ 02/Jan/19 ]

After some discussion with Esha, I have decided to pull this QW ticket from the project, as it will exacerbate an existing problem with unsharded databases. See SERVER-34431.
The problem today is that certain sharding refresh logic is currently hooked into the dropDatabase command; it is missed when databases are implicitly dropped by individual drops of all collections. Once SERVER-34431 is fixed, we can proceed with the work in this ticket.

Comment by Esha Maharishi (Inactive) [ 02/Jan/19 ]

This may have implications for sharding, since the list of databases is also stored in the routing table on the config server.

Generated at Thu Feb 08 04:32:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.