-
Type: Bug
-
Resolution: Cannot Reproduce
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
-
(copied to CRM)
This is a devastating intermittent issue that renders our application unusable when it happens.
Our application is deployed on AWS lambda, and once every few weeks, the following error happens when calling find or find_one methods, for example:
result = self.__database[......].find_one(condition)
The error happens in the following call stack (sorry for repetition in the end, but this is what I get from python, I thought I keep it, it might signal some sort of retries?), I tried to sanitise and make as readable as possible:
File "/var/task/pymongo/collection.py", line 1319, in find_one92
for result in cursor.limit(-1):10
{{}}
File "/var/task/pymongo/cursor.py", line 1207, in next11
if len(self.__data) or self._refresh():12
{{}}
File "/var/task/pymongo/cursor.py", line 1100, in _refresh13
self._session = self._collection.database.client._ensure_session()14
{{}}
File "/var/task/pymongo/mongo_client.py", line 1816, in _ensure_session15
return self.__start_session(True, causal_consistency=False)16
File "/var/task/pymongo/mongo_client.py", line 1766, in __start_session 17
server_session = self._get_server_session()18
File "/var/task/pymongo/mongo_client.py", line 1802, in _get_server_session19
return self._topology.get_server_session()20
File "/var/task/pymongo/topology.py", line 504, in get_server_session21
None)22
File "/var/task/pymongo/topology.py", line 217, in _select_servers_loop23
(self._error_message(selector), timeout, self.description))24
pymongo.errors.ServerSelectionTimeoutError: clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091),clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091),clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091), Timeout: 30s, Topology Description: <TopologyDescription id: 60e434cdd9d938xxxxxxxxxx, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('clusterx-shard-xx-xx.xxxxx.mongodb.net', xxxxx) server_type: Unknown, rtt: None, error=AutoReconnect('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')>, <ServerDescription ('clusterx-shard-xx-xx.xxxxx.mongodb.net', xxxxx) server_type: Unknown, rtt: None, error=AutoReconnect('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')>, <ServerDescription ('clusterx-shard-xx-xx.xxxxx.mongodb.net', xxxxx) server_type: Unknown, rtt: None, error=AutoReconnect('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')>]>25
{{}}
None26
clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091),clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091),clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091), Timeout: 30s, Topology Description: <TopologyDescription id: 60e434cdd9d93814dxxxxxxx, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx', xxxxx) server_type: Unknown, rtt: None, error=AutoReconnect('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')>, <ServerDescription ('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx', xxxxx) server_type: Unknown, rtt: None, error=AutoReconnect('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')>, <ServerDescription ('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx', xxxxx) server_type: Unknown, rtt: None, error=AutoReconnect('clusterx-shard-xx-xx.xxxxx.mongodb.net:xxxxx:xxxxx: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')>]>
Once this error happens, every request to the lambda fails, and the whole application is brought to its knees, the only way to get rid of it is to deploy a new lambda (which will trigger restart for running instances).