[SERVER-80123] Enable background compaction by default in appropriate test suites Created: 16/Aug/23  Updated: 07/Feb/24

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Backlog - Storage Execution Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on WT-11342 Implement the background compaction s... Closed
depends on SERVER-84706 Investigate if setting the oldest tim... In Code Review
depends on SERVER-83734 Create a server parameter to enable b... Closed
Gantt Dependency
has to be done after SERVER-84706 Investigate if setting the oldest tim... In Code Review
has to be done after WT-12032 Investigate an out-of-order timestamp... Closed
Problem/Incident
Related
is related to WT-11678 Investigate issues generated by backg... Closed
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Sprint: StorEng - Defined Pipeline, 2023-12-12 - Heisenbug
Participants:
Linked BF Score: 154
Story Points: 3

 Description   

Part of the work will be determining which suites would be appropriate for this. Perhaps the concurrency suite? This will help identify any issues over the long term with background compaction.



 Comments   
Comment by Etienne Petrel [ 09/Jan/24 ]

Before enabling background compaction by default, we want to make sure it is safe to do so.
Since background compaction creates internal checkpoints that may cause trouble in the server layer (see those tickets as examples WT-12032/BF-31107/BF-31367), it is better, for now, to hold off with this work.

Comment by Githook User [ 15/Dec/23 ]

Author:

{'name': 'Etienne Petrel', 'email': 'etienne.petrel@mongodb.com', 'username': 'etienneptl'}

Message: Revert "SERVER-80123 Enable background compaction in suites/core.yml"

This reverts commit 1d5fdb4ae9459a2852132d9db4a558d27bca322a.

GitOrigin-RevId: 2dfe5e03f4f1d50020225e0cff0c198a3b6ef309
Branch: master
https://github.com/mongodb/mongo/commit/dda591fa968b92faf4fa5fd8b79d667af48a59f5

Comment by Githook User [ 13/Dec/23 ]

Author:

{'name': 'Etienne Petrel', 'email': 'etienne.petrel@mongodb.com', 'username': 'etienneptl'}

Message: SERVER-80123 Enable background compaction in suites/core.yml

GitOrigin-RevId: 1d5fdb4ae9459a2852132d9db4a558d27bca322a
Branch: master
https://github.com/mongodb/mongo/commit/1236a44479aa921b3c89b0557843c7791be194e3

Comment by Etienne Petrel [ 13/Dec/23 ]

After several patch builds, it seems that the suites/core.yml is suitable to have background compaction enabled by default.
Latest patch build: link.

Comment by Etienne Petrel [ 13/Dec/23 ]

Yes Etienne Petrel, we should take the IX global lock there. Nice catch.

Thank you, created SERVER-83998 to address this.

Note that collection_truncate.js uses the emptycapped test-only command, which calls WT_SESSION::truncate(). It failed on a replica set, which uses timestamps and truncate does not support timestamps. Might be related to this?

It could be, nonetheless, I don't see how background compaction is involved.

Comment by Gregory Wlodarek [ 07/Dec/23 ]

Yes etienne.petrel@mongodb.com, we should take the IX global lock there. Nice catch.

Comment by Etienne Petrel [ 07/Dec/23 ]

Created a patch build enabling background compaction in the aforementioned suites and they fail here:

    dassert(shard_role_details::getLocker(opCtx)->isWriteLocked());

When we call autoCompact from notifyStartupComplete, we don't hold the IX lock, should we, gregory.wlodarek@mongodb.com? I am thinking of the following:

diff --git a/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp b/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp
index 5cdcfabc190..43e2e7a0ea2 100644
--- a/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp
+++ b/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp
@@ -632,6 +632,7 @@ void WiredTigerKVEngine::notifyStartupComplete(OperationContext* opCtx) {
     StorageEngine::AutoCompactOptions options{/*enable=*/true,
                                               /*freeSpaceTargetMB=*/boost::none,
                                               /*excludedIdents*/ std::vector<StringData>()};
+    Lock::GlobalLock lk(opCtx, MODE_IX);
     auto status = autoCompact(opCtx, options);
     uassert(8373401, "Failed to execute autoCompact.", status.isOK());
 }

Comment by Etienne Petrel [ 06/Dec/23 ]

Potential suites:

https://github.com/mongodb/mongo/blob/master/buildscripts/resmokeconfig/suites/replica_sets_jscore_passthrough.yml
https://github.com/mongodb/mongo/blob/master/buildscripts/resmokeconfig/suites/sharding_jscore_passthrough.yml
https://github.com/mongodb/mongo/blob/master/buildscripts/resmokeconfig/suites/core.yml

Comment by Etienne Petrel [ 29/Nov/23 ]

After a discussion with gregory.wlodarek@mongodb.com on Slack, here is what we can do:

  • Create a new server parameter, enableAutoCompaction and enable it by default in a handful of suites. For instance, in core.yml, it would look like this:

    diff --git a/buildscripts/resmokeconfig/suites/core.yml b/buildscripts/resmokeconfig/suites/core.yml
    index 894e268994f..b9312332972 100644
    --- a/buildscripts/resmokeconfig/suites/core.yml
    +++ b/buildscripts/resmokeconfig/suites/core.yml
    @@ -32,3 +32,4 @@ executor:
         mongod_options:
           set_parameters:
             enableTestCommands: 1
    +        enableAutoCompaction: 1
    

  • The new parameter needs to be added to storage_parameters.idl
  • When enableAutoCompaction is set, we would start the background compaction server like this:

    diff --git a/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp b/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp
    index 7a661e78c61..f9b84e9bb6c 100644
    --- a/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp
    +++ b/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp
    @@ -607,6 +607,25 @@ WiredTigerKVEngine::WiredTigerKVEngine(OperationContext* opCtx,
         _runTimeConfigParam.reset(makeServerParameter<WiredTigerEngineRuntimeConfigParameter>(
             "wiredTigerEngineRuntimeConfig", ServerParameterType::kRuntimeOnly));
         _runTimeConfigParam->_data.second = this;
    +    if (opCtx->getServiceContext()->userWritesAllowed() && !storageGlobalParams.repair && storageGlobalParams.syncdelay > 0) {
    +        WT_SESSION* s = session.getSession();
    +        int ret = s->compact(s, nullptr, "background=true");
    +        auto status = wtRCToStatus(ret, nullptr, "WiredTigerKVEngine::compact()");
    +        if (!status.isOK()) {
    +            // Background compaction failed to start
    +    }
     }
     
     WiredTigerKVEngine::~WiredTigerKVEngine() {
    

This work will be done in a separate ticket: SERVER-83734

Comment by Etienne Petrel [ 29/Nov/23 ]

gregory.wlodarek@mongodb.com, in WT-11678, I was enabling background compaction by default using the WiredTiger API as shown in this patch build. Now we have the autoCompact command ready for use, what would be a clean way to trigger that command by default?

Generated at Thu Feb 08 06:42:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.