Detailed steps to reproduce the problem?

I have tried to reproduce the problem in our staging environment, but was unable to do so after a day of load testing. Our staging and production environments use separate DB servers, but both are hosted in Atlas and are running version 5.0.23.

Definition of done: what must be done to consider the task complete?

The exact Python version used, with patch level:

3.9.6 (default, Aug 17 2021, 15:37:33)

[GCC 10.2.1 20210110]

The exact version of PyMongo used, with patch level:

4.6.0

True

Describe how MongoDB is set up. Local vs Hosted, version, topology, load balanced, etc.

MongoDB is setup using the Flask Mongo Engine client. Below are the applicable settings.
 
db_settings =

{ "host": app.config.get("DB_HOST"), "port": app.config.get("DB_PORT"), "db": app.config.get("DB"), "alias": "default", "uuidRepresentation": "standard", "minPoolSize": 1, "compressors": ["snappy", "zstd"], "retryReads": True, "retryWrites": True, }

db_settings["socketTimeoutMS"] = 60 * 1000
db_settings["timeoutMS"] = 30 * 1000
db_settings["serverSelectionTimeoutMS"] = 10 * 1000
db_settings["connectTimeoutMS"] = 10 * 1000
db_settings["waitQueueTimeoutMS"] = 10 * 1000

app.config.update({"MONGODB_SETTINGS": [db_settings]})

The operating system and version (e.g. Windows 7, OSX 10.8, ...)

Amazon Linux 2 AMI for EKS on Kubernetes 1.23.

Web framework or asynchronous network library used, if any, with version (e.g. Django 1.7, mod_wsgi 4.3.0, gevent 1.0.1, Tornado 4.0.2, ...)

gunicorn==21.2.0
gevent==23.9.1

We have tried upgrade gevent and pymongo and neither have addresses the issue. We added all Pymongo logging options. We have tried adding, removing, and removing the pymongo timeout settings. I have tried running with and without preload and have tried adjusting multiple Gunicorn settings to try to alleviate the issue and nothing has worked.

Something of note is pymongo stops logging anything at all after showing:
[2023-11-30 22:24:23 +0000] [10] [INFO] Heartbeat to server {0.connection_id} succeeded

What is strange is that this log is missing at the end of the log file:
[2023-11-30 22:24:23 +0000] [10] [INFO] Heartbeat sent to server ('cluster0-shard-00-00.o0jn3.mongodb.net', 27017)