[SERVER-83548] Exclude the oplog when executing the autoCompact command Created: 22/Nov/23  Updated: 07/Jan/24  Resolved: 06/Dec/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.3.0-rc0

Type: Task Priority: Major - P3
Reporter: Etienne Petrel Assignee: Etienne Petrel
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-80121 Create the background compaction API Closed
Related
related to SERVER-84357 Exclude the oplog when auto compactio... Open
Assigned Teams:
Storage Engines
Backwards Compatibility: Fully Compatible
Sprint: 2023-12-12 - Heisenbug
Participants:
Story Points: 3

 Comments   
Comment by Githook User [ 06/Dec/23 ]

Author:

{'name': 'Etienne Petrel', 'email': 'etienne.petrel@mongodb.com', 'username': 'etienneptl'}

Message: SERVER-83548 Always exclude the oplog when enabling background compaction

GitOrigin-RevId: 74fcb07147d951e049f194904d96c237634d859e
Branch: master
https://github.com/mongodb/mongo/commit/89dde5b46155fa8e65dbc3c48e1ada0755159291

Comment by Etienne Petrel [ 05/Dec/23 ]

Patch build.

Comment by Gregory Noma [ 29/Nov/23 ]

etienne.petrel@mongodb.com I would suggest doing it here

Comment by Etienne Petrel [ 29/Nov/23 ]

gregory.noma@mongodb.com, oh I see, thanks for clarifying. From which layer is it ok to retrieve the oplog URI?

Comment by Gregory Wlodarek [ 29/Nov/23 ]

When does the collection not exist? When do we not create the oplog?

The oplog collection is created once a replica set has been initialized. It will not exist if the node has only been in standalone mode since the beginning.

Comment by Gregory Noma [ 29/Nov/23 ]

However, we do not want to expose it for now in the MDB command, we want it to be used internally.

Oh sorry if I wasn't clear etienne.petrel@mongodb.com, I didn't mean exposing it in the MDB command. What I meant was just exposing it as a parameter to the WiredTigerKVEngine::autoCompact function, since it may be a layering violation for that function to internally fetch the URI of the oplog table.

Comment by Etienne Petrel [ 29/Nov/23 ]

Thank you both!

Taking a step back to think about the WiredTigerKVEngine::autoCompact API, would it make sense to have it just take as a parameter a set of idents that should be excluded?

We do have this feature ready at a WiredTiger level, see WT-11752. However, we do not want to expose it for now in the MDB command, we want it to be used internally.

Thank you, gregory.wlodarek@mongodb.com for the suggested piece of code. When does the collection not exist? When do we not create the oplog?

Comment by Gregory Wlodarek [ 28/Nov/23 ]

I like the idea of adding a new exclude list parameter to WiredTigerKVEngine::autoCompact.

In the auto_compact.cpp code we can fetch the ident like so:

    // Global MODE_IX lock must be held prior to this point
    std::shared_ptr<const CollectionCatalog> catalog = CollectionCatalog::get(opCtx);
    auto collection = catalog.lookupCollectionByNamespace(opCtx, NamespaceString::kRsOplogNamespace);
    if (!collection) {
        // Collection doesn't exist
    }
    const std::string uri = collection->getSharedIdent()->getIdent();

Comment by Gregory Noma [ 28/Nov/23 ]

etienne.petrel@mongodb.com if you're trying to get the oplog table's URI from within WiredTigerKVEngine::autoCompact, I see that we have WiredTigerKVEngine::_oplogRecordStore so in theory I think you may be able to do something like this:

auto oplogUri = _uri(_oplogRecordStore->getIdent());

But with that being said, _oplogRecordStore is only set by WiredTigerKVEngine::startOplogManager/WiredTigerKVEngine::haltOplogManager and is currently only used for equality checks. So there isn't a precedent so far for using it for anything else, but also I don't know the history of its current semantics and (limited) usage. gregory.wlodarek@mongodb.com do you know anything more about this?

Taking a step back to think about the WiredTigerKVEngine::autoCompact API, would it make sense to have it just take as a parameter a set of idents that should be excluded? And then the caller (presumably the autoCompact command) can fetch the oplog table URI at a higher level and pass it in to the function.

Comment by Etienne Petrel [ 28/Nov/23 ]

The function to be updated should be WiredTigerKVEngine::autoCompact and specifically, the compact configuration string with the exclude keyword.
gregory.wlodarek@mongodb.com, gregory.noma@mongodb.com, how do we get the corresponding WiredTiger table name corresponding to the oplog collection?

Generated at Thu Feb 08 06:52:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.