Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-4980

Tests fail now that killAllSessions raises network error on 8.0

    • Type: Icon: Build Failure Build Failure
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 4.11
    • Affects Version/s: None
    • Component/s: None
    • None
    • Not Needed

      Tests fail because killAllSessions sometimes raises network error on 8.0.

       [2024/11/18 16:39:21.595] FAILURE: pymongo.errors.AutoReconnect: localhost:27017: connection closed (configured timeouts: connectTimeoutMS: 20000.0ms) ()
       [2024/11/18 16:39:21.595] self = AsyncConnection(<ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>) CLOSED at 84527384
       [2024/11/18 16:39:21.595] dbname = 'admin'
       [2024/11/18 16:39:21.595] spec = {'$clusterTime': {'clusterTime': Timestamp(1731976014, 1), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x0...'$db': 'admin', 'killAllSessions': [], 'lsid': {'id': Binary(b'/\xdf\xe2,gTA\xe0\x95\x01\x04\xf3\xa7\x9b\xf3\xd6', 4)}}
       [2024/11/18 16:39:21.595] read_preference = Primary()
       [2024/11/18 16:39:21.595] codec_options = CodecOptions(document_class=dict, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_e...ne, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME)
       [2024/11/18 16:39:21.595] check = True, allowable_errors = None, read_concern = None, write_concern = None
       [2024/11/18 16:39:21.595] parse_write_concern_error = False, collation = None
       [2024/11/18 16:39:21.595] session = <pymongo.asynchronous.client_session.AsyncClientSession object at 0x000000000fc39980>
       [2024/11/18 16:39:21.595] client = AsyncMongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='repl0', tls=...en/x509gen/client.pem', tlscafile='/data/mci/53b7807baf70eb132ca46bcc0bac15a4/drivers-tools/.evergreen/x509gen/ca.pem')
       [2024/11/18 16:39:21.595] retryable_write = False, publish_events = True, user_fields = None
       [2024/11/18 16:39:21.595] exhaust_allowed = False
       [2024/11/18 16:39:21.595]     @_handle_reauth
       [2024/11/18 16:39:21.595]     async def command(
       [2024/11/18 16:39:21.595]         self,
       [2024/11/18 16:39:21.595]         dbname: str,
       [2024/11/18 16:39:21.595]         spec: MutableMapping[str, Any],
       [2024/11/18 16:39:21.595]         read_preference: _ServerMode = ReadPreference.PRIMARY,
       [2024/11/18 16:39:21.595]         codec_options: CodecOptions = DEFAULT_CODEC_OPTIONS,
       [2024/11/18 16:39:21.595]         check: bool = True,
       [2024/11/18 16:39:21.595]         allowable_errors: Optional[Sequence[Union[str, int]]] = None,
       [2024/11/18 16:39:21.595]         read_concern: Optional[ReadConcern] = None,
       [2024/11/18 16:39:21.595]         write_concern: Optional[WriteConcern] = None,
       [2024/11/18 16:39:21.595]         parse_write_concern_error: bool = False,
       [2024/11/18 16:39:21.595]         collation: Optional[_CollationIn] = None,
       [2024/11/18 16:39:21.595]         session: Optional[AsyncClientSession] = None,
       [2024/11/18 16:39:21.595]         client: Optional[AsyncMongoClient] = None,
       [2024/11/18 16:39:21.595]         retryable_write: bool = False,
       [2024/11/18 16:39:21.595]         publish_events: bool = True,
       [2024/11/18 16:39:21.595]         user_fields: Optional[Mapping[str, Any]] = None,
       [2024/11/18 16:39:21.595]         exhaust_allowed: bool = False,
       [2024/11/18 16:39:21.595]     ) -> dict[str, Any]:
       [2024/11/18 16:39:21.595]         """Execute a command or raise an error.
       [2024/11/18 16:39:21.595]     
       [2024/11/18 16:39:21.595]         :param dbname: name of the database on which to run the command
       [2024/11/18 16:39:21.595]         :param spec: a command document as a dict, SON, or mapping object
       [2024/11/18 16:39:21.595]         :param read_preference: a read preference
       [2024/11/18 16:39:21.595]         :param codec_options: a CodecOptions instance
       [2024/11/18 16:39:21.595]         :param check: raise OperationFailure if there are errors
       [2024/11/18 16:39:21.595]         :param allowable_errors: errors to ignore if `check` is True
       [2024/11/18 16:39:21.595]         :param read_concern: The read concern for this command.
       [2024/11/18 16:39:21.595]         :param write_concern: The write concern for this command.
       [2024/11/18 16:39:21.595]         :param parse_write_concern_error: Whether to parse the
       [2024/11/18 16:39:21.595]             ``writeConcernError`` field in the command response.
       [2024/11/18 16:39:21.595]         :param collation: The collation for this command.
       [2024/11/18 16:39:21.595]         :param session: optional AsyncClientSession instance.
       [2024/11/18 16:39:21.595]         :param client: optional AsyncMongoClient for gossipping $clusterTime.
       [2024/11/18 16:39:21.595]         :param retryable_write: True if this command is a retryable write.
       [2024/11/18 16:39:21.595]         :param publish_events: Should we publish events for this command?
       [2024/11/18 16:39:21.595]         :param user_fields: Response fields that should be decoded
       [2024/11/18 16:39:21.595]             using the TypeDecoders from codec_options, passed to
       [2024/11/18 16:39:21.595]             bson._decode_all_selective.
       [2024/11/18 16:39:21.595]         """
       [2024/11/18 16:39:21.595]         self.validate_session(client, session)
       [2024/11/18 16:39:21.595]         session = _validate_session_write_concern(session, write_concern)
       [2024/11/18 16:39:21.595]     
       [2024/11/18 16:39:21.595]         # Ensure command name remains in first place.
       [2024/11/18 16:39:21.595]         if not isinstance(spec, ORDERED_TYPES):  # type:ignore[arg-type]
       [2024/11/18 16:39:21.595]             spec = dict(spec)
       [2024/11/18 16:39:21.595]     
       [2024/11/18 16:39:21.595]         if not (write_concern is None or write_concern.acknowledged or collation is None):
       [2024/11/18 16:39:21.595]             raise ConfigurationError("Collation is unsupported for unacknowledged writes.")
       [2024/11/18 16:39:21.595]     
       [2024/11/18 16:39:21.595]         self.add_server_api(spec)
       [2024/11/18 16:39:21.595]         if session:
       [2024/11/18 16:39:21.595]             session._apply_to(spec, retryable_write, read_preference, self)
       [2024/11/18 16:39:21.595]         self.send_cluster_time(spec, session, client)
       [2024/11/18 16:39:21.595]         listeners = self.listeners if publish_events else None
       [2024/11/18 16:39:21.595]         unacknowledged = bool(write_concern and not write_concern.acknowledged)
       [2024/11/18 16:39:21.595]         if self.op_msg_enabled:
       [2024/11/18 16:39:21.595]             self._raise_if_not_writable(unacknowledged)
       [2024/11/18 16:39:21.595]         try:
       [2024/11/18 16:39:21.595] >           return await command(
       [2024/11/18 16:39:21.595]                 self,
       [2024/11/18 16:39:21.595]                 dbname,
       [2024/11/18 16:39:21.595]                 spec,
       [2024/11/18 16:39:21.595]                 self.is_mongos,
       [2024/11/18 16:39:21.595]                 read_preference,
       [2024/11/18 16:39:21.595]                 codec_options,
       [2024/11/18 16:39:21.595]                 session,
       [2024/11/18 16:39:21.595]                 client,
       [2024/11/18 16:39:21.595]                 check,
       [2024/11/18 16:39:21.595]                 allowable_errors,
       [2024/11/18 16:39:21.595]                 self.address,
       [2024/11/18 16:39:21.595]                 listeners,
       [2024/11/18 16:39:21.595]                 self.max_bson_size,
       [2024/11/18 16:39:21.595]                 read_concern,
       [2024/11/18 16:39:21.595]                 parse_write_concern_error=parse_write_concern_error,
       [2024/11/18 16:39:21.595]                 collation=collation,
       [2024/11/18 16:39:21.595]                 compression_ctx=self.compression_context,
       [2024/11/18 16:39:21.595]                 use_op_msg=self.op_msg_enabled,
       [2024/11/18 16:39:21.595]                 unacknowledged=unacknowledged,
       [2024/11/18 16:39:21.595]                 user_fields=user_fields,
       [2024/11/18 16:39:21.595]                 exhaust_allowed=exhaust_allowed,
       [2024/11/18 16:39:21.595]                 write_concern=write_concern,
       [2024/11/18 16:39:21.595]             )
       [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:538: 
       [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
       [2024/11/18 16:39:21.595] pymongo/asynchronous/network.py:203: in command
       [2024/11/18 16:39:21.595]     reply = await receive_message(conn, request_id)
       [2024/11/18 16:39:21.595] pymongo/asynchronous/network.py:315: in receive_message
       [2024/11/18 16:39:21.595]     length, _, response_to, op_code = _UNPACK_HEADER(await async_receive_data(conn, 16, deadline))
       [2024/11/18 16:39:21.595] pymongo/network_layer.py:278: in async_receive_data
       [2024/11/18 16:39:21.595]     return read_task.result()
       [2024/11/18 16:39:21.595] /opt/python/pypy3.10/lib/pypy3.10/asyncio/futures.py:201: in result
       [2024/11/18 16:39:21.595]     raise self._exception.with_traceback(self._exception_tb)
       [2024/11/18 16:39:21.595] /opt/python/pypy3.10/lib/pypy3.10/asyncio/tasks.py:232: in __step
       [2024/11/18 16:39:21.595]     result = coro.send(None)
       [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
       [2024/11/18 16:39:21.595] conn = <ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
       [2024/11/18 16:39:21.595] length = 16
       [2024/11/18 16:39:21.595] loop = <_UnixSelectorEventLoop running=False closed=True debug=True>
       [2024/11/18 16:39:21.595] once = False
       [2024/11/18 16:39:21.595]     async def _async_receive_ssl(
       [2024/11/18 16:39:21.595]         conn: _sslConn, length: int, loop: AbstractEventLoop, once: Optional[bool] = False
       [2024/11/18 16:39:21.595]     ) -> memoryview:
       [2024/11/18 16:39:21.595]         mv = memoryview(bytearray(length))
       [2024/11/18 16:39:21.595]         total_read = 0
       [2024/11/18 16:39:21.595]     
       [2024/11/18 16:39:21.595]         def _is_ready(fut: Future) -> None:
       [2024/11/18 16:39:21.595]             if fut.done():
       [2024/11/18 16:39:21.595]                 return
       [2024/11/18 16:39:21.595]             fut.set_result(None)
       [2024/11/18 16:39:21.595]     
       [2024/11/18 16:39:21.595]         while total_read < length:
       [2024/11/18 16:39:21.595]             try:
       [2024/11/18 16:39:21.595]                 read = conn.recv_into(mv[total_read:])
       [2024/11/18 16:39:21.595]                 if read == 0:
       [2024/11/18 16:39:21.595] >                   raise OSError("connection closed")
       [2024/11/18 16:39:21.595] E                   OSError: connection closed
       [2024/11/18 16:39:21.595] pymongo/network_layer.py:147: OSError
       [2024/11/18 16:39:21.595] The above exception was the direct cause of the following exception:
       [2024/11/18 16:39:21.595] self = <test.asynchronous.test_command_logging.TestCommandLoggingRedactedCommands testMethod=test_network_error_in_response_to_copydb_is_not_redacted>
       [2024/11/18 16:39:21.595]     async def test_case(self):
       [2024/11/18 16:39:21.595] >       await self.run_scenario(spec)
       [2024/11/18 16:39:21.595] test/asynchronous/unified_format.py:1482: 
       [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
       [2024/11/18 16:39:21.595] test/asynchronous/unified_format.py:1403: in run_scenario
       [2024/11/18 16:39:21.595]     await self.kill_all_sessions()
       [2024/11/18 16:39:21.595] test/asynchronous/unified_format.py:766: in kill_all_sessions
       [2024/11/18 16:39:21.595]     await client.admin.command("killAllSessions", [])
       [2024/11/18 16:39:21.595] pymongo/_csot.py:109: in csot_wrapper
       [2024/11/18 16:39:21.595]     return await func(self, *args, **kwargs)
       [2024/11/18 16:39:21.595] pymongo/asynchronous/database.py:932: in command
       [2024/11/18 16:39:21.595]     return await self._command(
       [2024/11/18 16:39:21.595] pymongo/asynchronous/database.py:770: in _command
       [2024/11/18 16:39:21.595]     return await conn.command(
       [2024/11/18 16:39:21.595] pymongo/asynchronous/helpers.py:45: in inner
       [2024/11/18 16:39:21.595]     return await func(*args, **kwargs)
       [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:566: in command
       [2024/11/18 16:39:21.595]     self._raise_connection_failure(error)
       [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:765: in _raise_connection_failure
       [2024/11/18 16:39:21.595]     _raise_connection_failure(self.address, error, timeout_details=details)
       [2024/11/18 16:39:21.595] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
       [2024/11/18 16:39:21.595] address = ('localhost', 27017), error = OSError('connection closed')
       [2024/11/18 16:39:21.595] msg_prefix = None, timeout_details = {'connectTimeoutMS': 20000.0}
       [2024/11/18 16:39:21.595]     def _raise_connection_failure(
       [2024/11/18 16:39:21.595]         address: Any,
       [2024/11/18 16:39:21.595]         error: Exception,
       [2024/11/18 16:39:21.595]         msg_prefix: Optional[str] = None,
       [2024/11/18 16:39:21.595]         timeout_details: Optional[dict[str, float]] = None,
       [2024/11/18 16:39:21.595]     ) -> NoReturn:
       [2024/11/18 16:39:21.595]         """Convert a socket.error to ConnectionFailure and raise it."""
       [2024/11/18 16:39:21.595]         host, port = address
       [2024/11/18 16:39:21.595]         # If connecting to a Unix socket, port will be None.
       [2024/11/18 16:39:21.595]         if port is not None:
       [2024/11/18 16:39:21.595]             msg = "%s:%d: %s" % (host, port, error)
       [2024/11/18 16:39:21.595]         else:
       [2024/11/18 16:39:21.595]             msg = f"{host}: {error}"
       [2024/11/18 16:39:21.595]         if msg_prefix:
       [2024/11/18 16:39:21.595]             msg = msg_prefix + msg
       [2024/11/18 16:39:21.595]         if "configured timeouts" not in msg:
       [2024/11/18 16:39:21.595]             msg += format_timeout_details(timeout_details)
       [2024/11/18 16:39:21.595]         if isinstance(error, socket.timeout):
       [2024/11/18 16:39:21.595]             raise NetworkTimeout(msg) from error
       [2024/11/18 16:39:21.595]         elif isinstance(error, SSLError) and "timed out" in str(error):
       [2024/11/18 16:39:21.595]             # Eventlet does not distinguish TLS network timeouts from other
       [2024/11/18 16:39:21.595]             # SSLErrors (https://github.com/eventlet/eventlet/issues/692).
       [2024/11/18 16:39:21.595]             # Luckily, we can work around this limitation because the phrase
       [2024/11/18 16:39:21.595]             # 'timed out' appears in all the timeout related SSLErrors raised.
       [2024/11/18 16:39:21.595]             raise NetworkTimeout(msg) from error
       [2024/11/18 16:39:21.595]         else:
       [2024/11/18 16:39:21.595] >           raise AutoReconnect(msg) from error
       [2024/11/18 16:39:21.595] E           pymongo.errors.AutoReconnect: localhost:27017: connection closed (configured timeouts: connectTimeoutMS: 20000.0ms)
       [2024/11/18 16:39:21.595] pymongo/asynchronous/pool.py:208: AutoReconnect
      

      https://spruce.mongodb.com/task/mongo_python_driver_test_rhel8_pypy3.10_cov_test_8.0_replica_set_noauth_ssl_sync_async_patch_a3bdc133ca497c2e966e41a283c1b712d045f7fe_673bda62a998a40007f45e15_24_11_19_00_22_59?execution=0&sortBy=STATUS&sortDir=ASC

      This could be a bug in the server but for now we can work around it by ignoring the network error.

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: