[SERVER-46696] mongo shell can end up creating multiple implicit sessions on the same replica set connection Created: 06/Mar/20  Updated: 08/Jan/24  Resolved: 17/Mar/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Janna Golden Assignee: Randolph Tan
Resolution: Fixed Votes: 0
Labels: sharding-4.4-stabilization
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Sharding 2020-03-23
Participants:
Linked BF Score: 34

 Description   

The issue is that the mongo shell creates new _mongo variable every query (via setHiddenMongo) and this is the variable ends up being passed to _DelegatingDriverSession. Note that setHiddenMongo ends up creating new mongo variables because the connection the connection is actually a DBClientRS so runCommandWithTarget will return the connection to the primary. Thus, the comparison here will always fail. The variable also uses the same connection object, but since it is a new variable, it "forgets" everything about the connection, leaving the "authenticated" and "_defaulSession" variables undefined.

This means that when getDB is called, it will end up trying to authenticate on the same c++ connection, which in turn will try to get the default session. But since it is a new variable, it will not have the _defaultSession variable set and end up creating a new one on the same cpp connection.



 Comments   
Comment by Githook User [ 26/Mar/20 ]

Author:

{'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}

Message: SERVER-46696 mongo shell can end up creating multiple implicit sessions on the same replica set connection

(cherry picked from commit ce6a53e5dd0ea28b3ed71f647d4013cb9a9ad43e)
(cherry picked from commit 9af76322fd8e69ebdb9aa443f1d18c44efc71684)
Branch: v4.4
https://github.com/mongodb/mongo/commit/38e017530f4a317c109524bde9bf1f1366896bc5

Comment by Githook User [ 17/Mar/20 ]

Author:

{'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}

Message: SERVER-46696 Sort interned authenticated js string
Branch: master
https://github.com/mongodb/mongo/commit/9af76322fd8e69ebdb9aa443f1d18c44efc71684

Comment by Githook User [ 17/Mar/20 ]

Author:

{'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}

Message: SERVER-46696 mongo shell can end up creating multiple implicit sessions on the same replica set connection
Branch: master
https://github.com/mongodb/mongo/commit/ce6a53e5dd0ea28b3ed71f647d4013cb9a9ad43e

Comment by Max Hirschhorn [ 13/Mar/20 ]

Thanks for the thorough analysis renctan! When Janna and I had traced where the calls to Mongo.prototype.startSession() were coming from with a patch similar to the following, we had missed that it was due to check_orphans_are_deleted_helpers.js authenticating rather than from it creating a new connection.

diff --git a/jstests/sharding/bulk_shard_insert.js b/jstests/sharding/bulk_shard_insert.js
index a8bea4e3fc..e419f7e9c4 100644
--- a/jstests/sharding/bulk_shard_insert.js
+++ b/jstests/sharding/bulk_shard_insert.js
@@ -9,6 +9,13 @@
 (function() {
 'use strict';
 
+Mongo.prototype.startSession = (function(original) {
+    return function() {
+        print(`@@@ session being created from ${new Error().stack}`);
+        return original.apply(this, arguments);
+    };
+})(Mongo.prototype.startSession);
+
 var st = new ShardingTest({shards: 4, chunkSize: 1});
 
 // Double the balancer interval to produce fewer migrations per unit time so that the test does not

It seems like a bug that the DBClientConnection from the DBClientReplicaSet would use a different logical session than the one being used by default for the replica set connection. We had addressed this specifically for cursors in the mongo shell with the changes to create an _DelegatingDriverSession in DBCommandCursor as part of SERVER-30848. I suspect moving that logic to C++ as part of the setHiddenMongo() function and setting the "_defaultSession" property (similar to but slightly more involved than what we do for the "slaveOK" and "defaultDB" properties already) to be a _DelegatingDriverSession object would address this issue.


[js_test:bulk_shard_insert] 2020-03-13T13:51:53.163-0400 @@@ session being created from Mongo.prototype.startSession@jstests/sharding/bulk_shard_insert.js:14:49
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.164-0400 getDefaultSession@src/mongo/shell/mongo.js:592:40
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.164-0400 DB.prototype.getSession@src/mongo/shell/db.js:1841:25
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.164-0400 DB.prototype._runCommandImpl@src/mongo/shell/db.js:146:21
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.164-0400 DB.prototype.runCommand@src/mongo/shell/db.js:162:16
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.164-0400 DB.prototype._getDefaultAuthenticationMechanism@src/mongo/shell/db.js:1475:24
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.164-0400 DB.prototype._authOrThrow@src/mongo/shell/db.js:1528:28
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.165-0400 DB.prototype.auth@src/mongo/shell/db.js:1554:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.165-0400 jsTest.authenticate/<@src/mongo/shell/utils.js:399:34
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.165-0400 assert.soon@src/mongo/shell/assert.js:340:21
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.165-0400 jsTest.authenticate@src/mongo/shell/utils.js:393:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.165-0400 Mongo.prototype.getDB@src/mongo/shell/mongo.js:43:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.165-0400 DelegatingDriverSession/this.getDatabase@src/mongo/shell/session.js:1023:24
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.166-0400 DBCommandCursor@src/mongo/shell/query.js:699:14
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.166-0400 DBQuery.prototype._exec@src/mongo/shell/query.js:112:28
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.166-0400 DBQuery.prototype.hasNext@src/mongo/shell/query.js:287:5
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.166-0400 DBQuery.prototype.toArray@src/mongo/shell/query.js:341:12
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.166-0400 runCheck/<@jstests/libs/check_orphans_are_deleted_helpers.js:11:48
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.167-0400 assert.soon@src/mongo/shell/assert.js:340:21
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.167-0400 runCheck@jstests/libs/check_orphans_are_deleted_helpers.js:8:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.167-0400 ShardingTest.prototype.checkOrphansAreDeleted/<@jstests/libs/override_methods/check_orphans_are_deleted.js:60:17
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.167-0400 DBQuery.prototype.forEach@src/mongo/shell/query.js:512:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.167-0400 ShardingTest.prototype.checkOrphansAreDeleted@jstests/libs/override_methods/check_orphans_are_deleted.js:55:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.167-0400 ShardingTest/this.stop@src/mongo/shell/shardingtest.js:392:9
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.168-0400 @jstests/sharding/bulk_shard_insert.js:114:1
[js_test:bulk_shard_insert] 2020-03-13T13:51:53.168-0400 @jstests/sharding/bulk_shard_insert.js:9:2

Comment by Randolph Tan [ 12/Mar/20 ]

Looks like there's something much more to this issue. The hook only creates the same number of connections as the number of shards (four). The issue is that the mongo shell creates new _mongo variable every query (via setHiddenMongo) and this is the variable ends up being passed to _DelegatingDriverSession. Note that setHiddenMongo ends up creating new mongo variables because the connection the connection is actually a DBClientRS so runCommandWithTarget will return the connection to the primary. Thus, the comparison here will always fail. The variable also uses the same connection object, but since it is a new variable, it "forgets" everything about the connection, leaving the "authenticated" and "_defaulSession" variables undefined.

This means that when getDB is called, it will end up trying to authenticate on the same c++ connection, which in turn will try to get the default session. But since it is a new variable, it will not have the _defaultSession variable set and end up creating a new one on the same cpp connection.

The problem with accumulating so many sessions on the same connection is that it will try to run commands on the connection on destruction. This occurs after when all the servers have already been torn down. Since the server is already down, the shell will have trouble connecting to the server, and the connection will attempt reconnecting everytime this happens, and since they are bound to the same connection, the sleep backoff will get bigger and bigger.

I'm not sure how to fix this right now, but as a temporary measure I propose that we black list this test in auth suites. What do you guys think? mira.carey@mongodb.com, max.hirschhorn

Generated at Thu Feb 08 05:12:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.