-
Type: Build Failure
-
Resolution: Fixed
-
Priority: Unknown
-
Affects Version/s: None
-
Component/s: None
-
None
-
Not Needed
Tests fail because killAllSessions sometimes raises network error on 8.0.
[2024/11/18 16:39:21.595] FAILURE: pymongo.errors.AutoReconnect: localhost:27017: connection closed (configured timeouts: connectTimeoutMS: 20000.0ms) () [2024/11/18 16:39:21.595] self = AsyncConnection(<ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>) CLOSED at 84527384 [2024/11/18 16:39:21.595] dbname = 'admin' [2024/11/18 16:39:21.595] spec = {'$clusterTime': {'clusterTime': Timestamp(1731976014, 1), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x0...'$db': 'admin', 'killAllSessions': [], 'lsid': {'id': Binary(b'/\xdf\xe2,gTA\xe0\x95\x01\x04\xf3\xa7\x9b\xf3\xd6', 4)}} [2024/11/18 16:39:21.595] read_preference = Primary() [2024/11/18 16:39:21.595] codec_options = CodecOptions(document_class=dict, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_e...ne, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME) [2024/11/18 16:39:21.595] check = True, allowable_errors = None, read_concern = None, write_concern = None [2024/11/18 16:39:21.595] parse_write_concern_error = False, collation = None [2024/11/18 16:39:21.595] session = <pymongo.asynchronous.client_session.AsyncClientSession object at 0x000000000fc39980> [2024/11/18 16:39:21.595] client = AsyncMongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='repl0', tls=...en/x509gen/client.pem', tlscafile='/data/mci/53b7807baf70eb132ca46bcc0bac15a4/drivers-tools/.evergreen/x509gen/ca.pem') [2024/11/18 16:39:21.595] retryable_write = False, publish_events = True, user_fields = None [2024/11/18 16:39:21.595] exhaust_allowed = False [2024/11/18 16:39:21.595] @_handle_reauth [2024/11/18 16:39:21.595] async def command( [2024/11/18 16:39:21.595] self, [2024/11/18 16:39:21.595] dbname: str, [2024/11/18 16:39:21.595] spec: MutableMapping[str, Any], [2024/11/18 16:39:21.595] read_preference: _ServerMode = ReadPreference.PRIMARY, [2024/11/18 16:39:21.595] codec_options: CodecOptions = DEFAULT_CODEC_OPTIONS, [2024/11/18 16:39:21.595] check: bool = True, [2024/11/18 16:39:21.595] allowable_errors: Optional[Sequence[Union[str, int]]] = None, [2024/11/18 16:39:21.595] read_concern: Optional[ReadConcern] = None, [2024/11/18 16:39:21.595] write_concern: Optional[WriteConcern] = None, [2024/11/18 16:39:21.595] parse_write_concern_error: bool = False, [2024/11/18 16:39:21.595] collation: Optional[_CollationIn] = None, [2024/11/18 16:39:21.595] session: Optional[AsyncClientSession] = None, [2024/11/18 16:39:21.595] client: Optional[AsyncMongoClient] = None, [2024/11/18 16:39:21.595] retryable_write: bool = False, [2024/11/18 16:39:21.595] publish_events: bool = True, [2024/11/18 16:39:21.595] user_fields: Optional[Mapping[str, Any]] = None, [2024/11/18 16:39:21.595] exhaust_allowed: bool = False, [2024/11/18 16:39:21.595] ) -> dict[str, Any]: [2024/11/18 16:39:21.595] """Execute a command or raise an error. [2024/11/18 16:39:21.595] [2024/11/18 16:39:21.595] :param dbname: name of the database on which to run the command [2024/11/18 16:39:21.595] :param spec: a command document as a dict, SON, or mapping object [2024/11/18 16:39:21.595] :param read_preference: a read preference [2024/11/18 16:39:21.595] :param codec_options: a CodecOptions instance [2024/11/18 16:39:21.595] :param check: raise OperationFailure if there are errors [2024/11/18 16:39:21.595] :param allowable_errors: errors to ignore if `check` is True [2024/11/18 16:39:21.595] :param read_concern: The read concern for this command. [2024/11/18 16:39:21.595] :param write_concern: The write concern for this command. [2024/11/18 16:39:21.595] :param parse_write_concern_error: Whether to parse the [2024/11/18 16:39:21.595] ``writeConcernError`` field in the command response. [2024/11/18 16:39:21.595] :param collation: The collation for this command. [2024/11/18 16:39:21.595] :param session: optional AsyncClientSession instance. [2024/11/18 16:39:21.595] :param client: optional AsyncMongoClient for gossipping $clusterTime. [2024/11/18 16:39:21.595] :param retryable_write: True if this command is a retryable write. [2024/11/18 16:39:21.595] :param publish_events: Should we publish events for this command? [2024/11/18 16:39:21.595] :param user_fields: Response fields that should be decoded [2024/11/18 16:39:21.595] using the TypeDecoders from codec_options, passed to [2024/11/18 16:39:21.595] bson._decode_all_selective. [2024/11/18 16:39:21.595] """ [2024/11/18 16:39:21.595] self.validate_session(client, session) [2024/11/18 16:39:21.595] session = _validate_session_write_concern(session, write_concern) [2024/11/18 16:39:21.595] [2024/11/18 16:39:21.595] # Ensure command name remains in first place. [2024/11/18 16:39:21.595] if not isinstance(spec, ORDERED_TYPES): # type:ignore[arg-type] [2024/11/18 16:39:21.595] spec = dict(spec) [2024/11/18 16:39:21.595] [2024/11/18 16:39:21.595] if not (write_concern is None or write_concern.acknowledged or collation is None): [2024/11/18 16:39:21.595] raise ConfigurationError("Collation is unsupported for unacknowledged writes.") [2024/11/18 16:39:21.595] [2024/11/18 16:39:21.595] self.add_server_api(spec) [2024/11/18 16:39:21.595] if session: [2024/11/18 16:39:21.595] session._apply_to(spec, retryable_write, read_preference, self) [2024/11/18 16:39:21.595] self.send_cluster_time(spec, session, client) [2024/11/18 16:39:21.595] listeners = self.listeners if publish_events else None [2024/11/18 16:39:21.595] unacknowledged = bool(write_concern and not write_concern.acknowledged) [2024/11/18 16:39:21.595] if self.op_msg_enabled: [2024/11/18 16:39:21.595] self._raise_if_not_writable(unacknowledged) [2024/11/18 16:39:21.595] try: [2024/11/18 16:39:21.595] > return await command( [2024/11/18 16:39:21.595] self, [2024/11/18 16:39:21.595] dbname, [2024/11/18 16:39:21.595] spec, [2024/11/18 16:39:21.595] self.is_mongos, [2024/11/18 16:39:21.595] read_preference, [2024/11/18 16:39:21.595] codec_options, [2024/11/18 16:39:21.595] session, [2024/11/18 16:39:21.595] client, [2024/11/18 16:39:21.595] check, [2024/11/18 16:39:21.595] allowable_errors, [2024/11/18 16:39:21.595] self.address, [2024/11/18 16:39:21.595] listeners, [2024/11/18 16:39:21.595] self.max_bson_size, [2024/11/18 16:39:21.595] read_concern, [2024/11/18 16:39:21.595] parse_write_concern_error=parse_write_concern_error, [2024/11/18 16:39:21.595] collation=collation, [2024/11/18 16:39:21.595] compression_ctx=self.compression_context, [2024/11/18 16:39:21.595] use_op_msg=self.op_msg_enabled, [2024/11/18 16:39:21.595] unacknowledged=unacknowledged, [2024/11/18 16:39:21.595] user_fields=user_fields, [2024/11/18 16:39:21.595] exhaust_allowed=exhaust_allowed, [2024/11/18 16:39:21.595] write_concern=write_concern, [2024/11/18 16:39:21.595] ) [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:538: [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [2024/11/18 16:39:21.595] pymongo/asynchronous/network.py:203: in command [2024/11/18 16:39:21.595] reply = await receive_message(conn, request_id) [2024/11/18 16:39:21.595] pymongo/asynchronous/network.py:315: in receive_message [2024/11/18 16:39:21.595] length, _, response_to, op_code = _UNPACK_HEADER(await async_receive_data(conn, 16, deadline)) [2024/11/18 16:39:21.595] pymongo/network_layer.py:278: in async_receive_data [2024/11/18 16:39:21.595] return read_task.result() [2024/11/18 16:39:21.595] /opt/python/pypy3.10/lib/pypy3.10/asyncio/futures.py:201: in result [2024/11/18 16:39:21.595] raise self._exception.with_traceback(self._exception_tb) [2024/11/18 16:39:21.595] /opt/python/pypy3.10/lib/pypy3.10/asyncio/tasks.py:232: in __step [2024/11/18 16:39:21.595] result = coro.send(None) [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [2024/11/18 16:39:21.595] conn = <ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6> [2024/11/18 16:39:21.595] length = 16 [2024/11/18 16:39:21.595] loop = <_UnixSelectorEventLoop running=False closed=True debug=True> [2024/11/18 16:39:21.595] once = False [2024/11/18 16:39:21.595] async def _async_receive_ssl( [2024/11/18 16:39:21.595] conn: _sslConn, length: int, loop: AbstractEventLoop, once: Optional[bool] = False [2024/11/18 16:39:21.595] ) -> memoryview: [2024/11/18 16:39:21.595] mv = memoryview(bytearray(length)) [2024/11/18 16:39:21.595] total_read = 0 [2024/11/18 16:39:21.595] [2024/11/18 16:39:21.595] def _is_ready(fut: Future) -> None: [2024/11/18 16:39:21.595] if fut.done(): [2024/11/18 16:39:21.595] return [2024/11/18 16:39:21.595] fut.set_result(None) [2024/11/18 16:39:21.595] [2024/11/18 16:39:21.595] while total_read < length: [2024/11/18 16:39:21.595] try: [2024/11/18 16:39:21.595] read = conn.recv_into(mv[total_read:]) [2024/11/18 16:39:21.595] if read == 0: [2024/11/18 16:39:21.595] > raise OSError("connection closed") [2024/11/18 16:39:21.595] E OSError: connection closed [2024/11/18 16:39:21.595] pymongo/network_layer.py:147: OSError [2024/11/18 16:39:21.595] The above exception was the direct cause of the following exception: [2024/11/18 16:39:21.595] self = <test.asynchronous.test_command_logging.TestCommandLoggingRedactedCommands testMethod=test_network_error_in_response_to_copydb_is_not_redacted> [2024/11/18 16:39:21.595] async def test_case(self): [2024/11/18 16:39:21.595] > await self.run_scenario(spec) [2024/11/18 16:39:21.595] test/asynchronous/unified_format.py:1482: [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [2024/11/18 16:39:21.595] test/asynchronous/unified_format.py:1403: in run_scenario [2024/11/18 16:39:21.595] await self.kill_all_sessions() [2024/11/18 16:39:21.595] test/asynchronous/unified_format.py:766: in kill_all_sessions [2024/11/18 16:39:21.595] await client.admin.command("killAllSessions", []) [2024/11/18 16:39:21.595] pymongo/_csot.py:109: in csot_wrapper [2024/11/18 16:39:21.595] return await func(self, *args, **kwargs) [2024/11/18 16:39:21.595] pymongo/asynchronous/database.py:932: in command [2024/11/18 16:39:21.595] return await self._command( [2024/11/18 16:39:21.595] pymongo/asynchronous/database.py:770: in _command [2024/11/18 16:39:21.595] return await conn.command( [2024/11/18 16:39:21.595] pymongo/asynchronous/helpers.py:45: in inner [2024/11/18 16:39:21.595] return await func(*args, **kwargs) [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:566: in command [2024/11/18 16:39:21.595] self._raise_connection_failure(error) [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:765: in _raise_connection_failure [2024/11/18 16:39:21.595] _raise_connection_failure(self.address, error, timeout_details=details) [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [2024/11/18 16:39:21.595] address = ('localhost', 27017), error = OSError('connection closed') [2024/11/18 16:39:21.595] msg_prefix = None, timeout_details = {'connectTimeoutMS': 20000.0} [2024/11/18 16:39:21.595] def _raise_connection_failure( [2024/11/18 16:39:21.595] address: Any, [2024/11/18 16:39:21.595] error: Exception, [2024/11/18 16:39:21.595] msg_prefix: Optional[str] = None, [2024/11/18 16:39:21.595] timeout_details: Optional[dict[str, float]] = None, [2024/11/18 16:39:21.595] ) -> NoReturn: [2024/11/18 16:39:21.595] """Convert a socket.error to ConnectionFailure and raise it.""" [2024/11/18 16:39:21.595] host, port = address [2024/11/18 16:39:21.595] # If connecting to a Unix socket, port will be None. [2024/11/18 16:39:21.595] if port is not None: [2024/11/18 16:39:21.595] msg = "%s:%d: %s" % (host, port, error) [2024/11/18 16:39:21.595] else: [2024/11/18 16:39:21.595] msg = f"{host}: {error}" [2024/11/18 16:39:21.595] if msg_prefix: [2024/11/18 16:39:21.595] msg = msg_prefix + msg [2024/11/18 16:39:21.595] if "configured timeouts" not in msg: [2024/11/18 16:39:21.595] msg += format_timeout_details(timeout_details) [2024/11/18 16:39:21.595] if isinstance(error, socket.timeout): [2024/11/18 16:39:21.595] raise NetworkTimeout(msg) from error [2024/11/18 16:39:21.595] elif isinstance(error, SSLError) and "timed out" in str(error): [2024/11/18 16:39:21.595] # Eventlet does not distinguish TLS network timeouts from other [2024/11/18 16:39:21.595] # SSLErrors (https://github.com/eventlet/eventlet/issues/692). [2024/11/18 16:39:21.595] # Luckily, we can work around this limitation because the phrase [2024/11/18 16:39:21.595] # 'timed out' appears in all the timeout related SSLErrors raised. [2024/11/18 16:39:21.595] raise NetworkTimeout(msg) from error [2024/11/18 16:39:21.595] else: [2024/11/18 16:39:21.595] > raise AutoReconnect(msg) from error [2024/11/18 16:39:21.595] E pymongo.errors.AutoReconnect: localhost:27017: connection closed (configured timeouts: connectTimeoutMS: 20000.0ms) [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:208: AutoReconnect
This could be a bug in the server but for now we can work around it by ignoring the network error.
- is related to
-
PYTHON-4982 Test Failure - network errors running configureFailPoint
- Backlog