[COMPASS-7009] Frequent "Connection pool cleared" error Created: 06/Jul/23  Updated: 04/Aug/23

Status: Open
Project: Compass
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: D V Assignee: Julia Oppenheim
Resolution: Unresolved Votes: 0
Labels: needs-prioritization
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: macOS Ventura 13.4.1
node.js / npm versions:
Additional info: Macbook Pro 16-inch, 2021, Apple M1 Pro, 16GB RAM. MongoDB Compass 1.38.2 (but issue has been happening in several past major versions at least as far back as 1.33)


Attachments: PNG File SCR-20230706-nclu.png     File compass-logs.json    
Issue Links:
Related
related to SERVER-61585 Mongosh triggering an okta 2fa on eve... Open
related to COMPASS-4527 Multiple mongos connections Closed
Story Points: 3

 Description   

Problem Statement/Rationale

Experiencing a frequent error when trying to connect to mongos via GCP IAP tunnel:
 
An error occurred while loading instance info: Connection pool for localhost:27027 was cleared because another operation failed with: "connection <monitor> to 127.0.0.1:27027 closed" * This happens about 9 out of 10 times until it successfully connects.

  • Connecting on the command line via mongosh with the same connection string succeeds 100% of the time. Same for Studio 3T and JetBrains Goland (DB plugin).
  • Once successfully connected the error does not appear for the duration of the session.
  • I've tried changing a bunch of different connection settings to no effect. e.g. direction connection, socket/connect timeouts, wait queue time, heartbeat frequency, etc.

Please be sure to attach relevant logs with any sensitive data redacted.

Compass logs attached.

Steps to Reproduce

  1. Set up Mongo on a GCP VM such that it's accessible via IAP, mongos is running on port 27018, and LDAP authentication is required.
  2. Set up the tunnel. I used something like this: gcloud --project my-project compute start-iap-tunnel mongo-vm 27018 --local-host-port=127.0.0.1:27027 --zone=northamerica-northeast1-b
  3. Use a connection string like the following: mongodb://myusername%40example.com:*****@localhost:27027/?readPreference=primary&ssl=false&authMechanism=PLAIN&authSource=%24external&directConnection=true
  4. Connect a few times

Expected Results

Successful connection

Actual Results

Most of the time it errors out.

Additional Notes

My workaround right now is to quickly open a bunch of Compass windows and flip through them all, connecting in each one, and then closing any that have errored out. This is annoying enough that most of my team have moved off Compass.



 Comments   
Comment by D V [ 14/Jul/23 ]

anna.henningsen@mongodb.com Thanks for the context! I can definitely see how the "proper" solution is difficult.

As far as I know there're no TOTP/OTP on my side. At least, I don't have another auth factor that I need to provide in order to connect via LDAP. I just run the gcloud start-iap-tunnel command.

I agree that updating the error message could be sufficient, especially if it has the workaround of adding maxPool=1 as an explicit suggestion within it, since it's totally non-obvious. My entire team of 50+ engineers essentially gave up on Compass, because it was the only client that showed that error, so it felt like a Compass-specific bug.

Comment by Anna Henningsen [ 12/Jul/23 ]

dv@glyphy.com It’s not straightforward, unfortunately. I assume you’re using LDAP with some sort of TOTP mechanism, where the OTP is part of the password you pass to Compass?

The reason why that fails is that Compass opens (and this might be obvious from the error message here) a connection pool, but uses the same OTP for all of those connections, so some of them will successfully authenticate and some not. This is a very specific use case, though, and not one that we can reliably detect at that, and in general, Compass does utilize the connection pool to improve application performance.

So, yes, we are aware of this pain point. We might be able to improve the error message here, but a “proper” solution is unfortunately not trivial.

(As a side note, we announced upcoming preview support for OpenID Connect authentication for MongoDB 7.0+ databases just last month. Switching authentication mechanisms is obviously a bigger change for users, but that’s part of our long-term story for better support for “modern” human-facing authentication.)

Comment by D V [ 11/Jul/23 ]

anna.henningsen@mongodb.com Thank you, that entirely solves the issue! Should this setting be enabled by default?

Comment by Anna Henningsen [ 11/Jul/23 ]

dv@glyphy.com What happens if you add maxPoolSize=1 to your connection string? Does that make any difference?

Comment by PM Bot [ 06/Jul/23 ]

Hello dv@glyphy.com, thank you for reaching out to us! The team will review your issue and get back to you soon as soon as possible.

Please review your issue to ensure you've included your environment details and have attached relevant logs (with any sensitive data redacted), so that we're best able to provide you a timely and thorough response. Thanks again!

Generated at Wed Feb 07 22:45:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.