[SERVER-29495] Change DBDirectClient to access the query system interface directly Created: 07/Jun/17 Updated: 25/Oct/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Internal Code, Networking, Storage |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Spencer Brody (Inactive) | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | techdebt | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||
| Participants: | |||||||||||||
| Description |
|
Currently all methods on DBDirectClient go through the full networking and command execution layers to perform basic i/o against the database. This has caused many issues in the past including but not limited to problems with access control, curop tracking/profiling, write concern and maxTimeMs handling, command metadata management, etc. It also means that any code using DBDirectClient requires linking in a huge amount of the mongoDB codebase. The public API of DBDirectClient isn't bad, it's the implementation that goes through networking and command execution just to do basic reads and writes to documents in the database. If instead DBDirectClient bypassed those layers and called directly into the query API, it would be much easier to use correctly and much more useful. |
| Comments |
| Comment by A. Jesse Jiryu Davis [ 09/Sep/20 ] |
|
I struggled with DBDirectClient while implementing I solved this for the APIParameters class specifically (it decorates OperationContext), at a specific DBDirectClient method call site. I yearn for a general solution to save/restore the OperationContext when calling into DBDirectClient methods. Or get rid of DBDirectClient entirely. |
| Comment by Eric Milkie [ 19/Jun/17 ] |
|
Based on the discussion and refinement of the goals of this ticket, I have changed the title and description a bit and assigned this to Query for triage. |
| Comment by Esha Maharishi (Inactive) [ 09/Jun/17 ] |
|
Two things I'd add: 1) There hasn't been a good set of functions for writes, queries, and commands written on top of ASIO. The closest that I'm aware of (at least in mongos) is the Shard class. The Shard class has a runCommand, runBatchWriteCommand which wraps runCommand but parses the response as a write response, and exhaustiveFindOnConfig (which internally exhausts a cursor, and invariants that it's only run against the config database). So, there is no function that returns a cursor. Also, runCommand is confusingly used to do reads or writes (for example, a count is a read). This interface has been growing ad-hoc. 2) the Shard interface has a ShardRemote and ShardLocal implementation, somewhat analogously to DBClient and DBDirectClient. From working with these classes, we've realized that it's almost always better to use ShardLocal or ShardRemote explicitly, and not much reason for them to have an identical interface. (They cannot currently be separated because some components, such as the DistLockManager, that exist on mongos as well as mongod simply reference Shard, and depending on where it's running, a ShardLocal or ShardRemote is actually bound to the reference). |
| Comment by Andrew Morrow (Inactive) [ 09/Jun/17 ] |
|
Seeing an end to the DBClientInterface class hierarchy has been a long term goal. However, we are still fairly far away from it, as I see things. There are several aspects to consider.
|
| Comment by Eric Milkie [ 09/Jun/17 ] |
|
I disagree with the Description's premise that the "public API of DBDirectClient isn't bad"; it's missing the fact that it really does implement all of the old C++ driver's interface, include things like "call", "say", and "getMaxWireVersion". It sounds like what we really want to do is rip the lower-level parts of this driver away (which presumably nothing is using anymore), and just leave the higher-level wrapper-like helper functions like "query". And still support a cursor-like interface. |
| Comment by Spencer Brody (Inactive) [ 08/Jun/17 ] |
|
Ah yes, good call, that's definitely better |
| Comment by Andy Schwerin [ 07/Jun/17 ] |
|
To spencer's comment about invoking the command interface directly, I think that if you're inside the server, you should be making function calls, not running mongodb commands. The command should delegate to a function, and internally, if you need that functionality, you call the function. |
| Comment by Spencer Brody (Inactive) [ 07/Jun/17 ] |
|
An alternative to doing this ticket would be to try to turn the repl StorageInterface into something generic and give it cursor support. Then we could start switching users of DBDirectClient over to the StorageInterface instead and eventually kill DBDirectClient. I'm not really sure which would be more work at this point. |
| Comment by Spencer Brody (Inactive) [ 07/Jun/17 ] |
|
I feel like that shouldn't really be under the purview of DBDirectClient. I think if you want to run an arbitrary command there should be a way to call directly into the Command interface. I think part of the problem with DBDirectClient is it's doing too much - I'd rather have it focus on being a good way to do local database i/o, since we still don't really have a good general-purpose solution for that. The repl team has been building a StorageInterface that does much of that, but one thing it still doesn't do that DBDirectClient does is provide a way to do a query against the local database contents and get back a cursor-like thing that can be used to process the results one at a time without pulling the entire result set into memory. |
| Comment by Esha Maharishi (Inactive) [ 07/Jun/17 ] |
|
Question about this - what about uses of DBDirectClient that run a command that doesn't necessarily touch the storage engine, or do more than touch the storage engine? For example, we plan to use DBDirectClient to make the config server run setFeatureCompatibilityVersion against itself inside of _configsvrSetFeatureCompatibilityVersion. Will DBDirectClient still go through assembleResponse etc? |