[SERVER-46696] mongo shell can end up creating multiple implicit sessions on the same replica set connection Created: 06/Mar/20 Updated: 08/Jan/24 Resolved: 17/Mar/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 4.4.0-rc0, 4.7.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Janna Golden | Assignee: | Randolph Tan |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | sharding-4.4-stabilization | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v4.4
|
||||||||
| Sprint: | Sharding 2020-03-23 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 34 | ||||||||
| Description |
|
The issue is that the mongo shell creates new _mongo variable every query (via setHiddenMongo) and this is the variable ends up being passed to _DelegatingDriverSession. Note that setHiddenMongo ends up creating new mongo variables because the connection the connection is actually a DBClientRS so runCommandWithTarget will return the connection to the primary. Thus, the comparison here will always fail. The variable also uses the same connection object, but since it is a new variable, it "forgets" everything about the connection, leaving the "authenticated" and "_defaulSession" variables undefined. This means that when getDB is called, it will end up trying to authenticate on the same c++ connection, which in turn will try to get the default session. But since it is a new variable, it will not have the _defaultSession variable set and end up creating a new one on the same cpp connection. |
| Comments |
| Comment by Githook User [ 26/Mar/20 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}Message: (cherry picked from commit ce6a53e5dd0ea28b3ed71f647d4013cb9a9ad43e) | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 17/Mar/20 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}Message: | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 17/Mar/20 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}Message: | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 13/Mar/20 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for the thorough analysis renctan! When Janna and I had traced where the calls to Mongo.prototype.startSession() were coming from with a patch similar to the following, we had missed that it was due to check_orphans_are_deleted_helpers.js authenticating rather than from it creating a new connection.
It seems like a bug that the DBClientConnection from the DBClientReplicaSet would use a different logical session than the one being used by default for the replica set connection. We had addressed this specifically for cursors in the mongo shell with the changes to create an _DelegatingDriverSession in DBCommandCursor as part of
| ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Randolph Tan [ 12/Mar/20 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Looks like there's something much more to this issue. The hook only creates the same number of connections as the number of shards (four). The issue is that the mongo shell creates new _mongo variable every query (via setHiddenMongo) and this is the variable ends up being passed to _DelegatingDriverSession. Note that setHiddenMongo ends up creating new mongo variables because the connection the connection is actually a DBClientRS so runCommandWithTarget will return the connection to the primary. Thus, the comparison here will always fail. The variable also uses the same connection object, but since it is a new variable, it "forgets" everything about the connection, leaving the "authenticated" and "_defaulSession" variables undefined. This means that when getDB is called, it will end up trying to authenticate on the same c++ connection, which in turn will try to get the default session. But since it is a new variable, it will not have the _defaultSession variable set and end up creating a new one on the same cpp connection. The problem with accumulating so many sessions on the same connection is that it will try to run commands on the connection on destruction. This occurs after when all the servers have already been torn down. Since the server is already down, the shell will have trouble connecting to the server, and the connection will attempt reconnecting everytime this happens, and since they are bound to the same connection, the sleep backoff will get bigger and bigger. I'm not sure how to fix this right now, but as a temporary measure I propose that we black list this test in auth suites. What do you guys think? mira.carey@mongodb.com, max.hirschhorn |