[SERVER-83548] Exclude the oplog when executing the autoCompact command Created: 22/Nov/23 Updated: 07/Jan/24 Resolved: 06/Dec/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.3.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Etienne Petrel | Assignee: | Etienne Petrel |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Storage Engines
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | 2023-12-12 - Heisenbug | ||||||||||||||||
| Participants: | |||||||||||||||||
| Story Points: | 3 | ||||||||||||||||
| Comments |
| Comment by Githook User [ 06/Dec/23 ] | |||||||
|
Author: {'name': 'Etienne Petrel', 'email': 'etienne.petrel@mongodb.com', 'username': 'etienneptl'}Message: GitOrigin-RevId: 74fcb07147d951e049f194904d96c237634d859e | |||||||
| Comment by Etienne Petrel [ 05/Dec/23 ] | |||||||
| Comment by Gregory Noma [ 29/Nov/23 ] | |||||||
|
etienne.petrel@mongodb.com I would suggest doing it here | |||||||
| Comment by Etienne Petrel [ 29/Nov/23 ] | |||||||
|
gregory.noma@mongodb.com, oh I see, thanks for clarifying. From which layer is it ok to retrieve the oplog URI? | |||||||
| Comment by Gregory Wlodarek [ 29/Nov/23 ] | |||||||
The oplog collection is created once a replica set has been initialized. It will not exist if the node has only been in standalone mode since the beginning. | |||||||
| Comment by Gregory Noma [ 29/Nov/23 ] | |||||||
Oh sorry if I wasn't clear etienne.petrel@mongodb.com, I didn't mean exposing it in the MDB command. What I meant was just exposing it as a parameter to the WiredTigerKVEngine::autoCompact function, since it may be a layering violation for that function to internally fetch the URI of the oplog table. | |||||||
| Comment by Etienne Petrel [ 29/Nov/23 ] | |||||||
|
Thank you both!
We do have this feature ready at a WiredTiger level, see Thank you, gregory.wlodarek@mongodb.com for the suggested piece of code. When does the collection not exist? When do we not create the oplog? | |||||||
| Comment by Gregory Wlodarek [ 28/Nov/23 ] | |||||||
|
I like the idea of adding a new exclude list parameter to WiredTigerKVEngine::autoCompact. In the auto_compact.cpp code we can fetch the ident like so:
| |||||||
| Comment by Gregory Noma [ 28/Nov/23 ] | |||||||
|
etienne.petrel@mongodb.com if you're trying to get the oplog table's URI from within WiredTigerKVEngine::autoCompact, I see that we have WiredTigerKVEngine::_oplogRecordStore so in theory I think you may be able to do something like this:
But with that being said, _oplogRecordStore is only set by WiredTigerKVEngine::startOplogManager/WiredTigerKVEngine::haltOplogManager and is currently only used for equality checks. So there isn't a precedent so far for using it for anything else, but also I don't know the history of its current semantics and (limited) usage. gregory.wlodarek@mongodb.com do you know anything more about this? Taking a step back to think about the WiredTigerKVEngine::autoCompact API, would it make sense to have it just take as a parameter a set of idents that should be excluded? And then the caller (presumably the autoCompact command) can fetch the oplog table URI at a higher level and pass it in to the function. | |||||||
| Comment by Etienne Petrel [ 28/Nov/23 ] | |||||||
|
The function to be updated should be WiredTigerKVEngine::autoCompact and specifically, the compact configuration string with the exclude keyword. |