Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-3345

CSOT rtt heuristic is incompatible with loadBalanced=True mode

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 4.2
    • Affects Version/s: None
    • Component/s: None
    • None

      The CSOT rtt heuristic is incompatible with loadBalanced=True mode:

       [2022/07/04 20:47:34.387] ERROR [0.216s]: test_maxTimeMS_value_in_the_command_is_less_than_timeoutMS (test_csot.TestCsotCommandExecution)
       [2022/07/04 20:47:34.387] ----------------------------------------------------------------------
       [2022/07/04 20:47:34.387] Traceback (most recent call last):
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1472, in test_case
       [2022/07/04 20:47:34.387]     self.run_scenario(spec)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1449, in run_scenario
       [2022/07/04 20:47:34.387]     self.run_operations(spec["operations"])
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1376, in run_operations
       [2022/07/04 20:47:34.387]     self.run_entity_operation(op)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1199, in run_entity_operation
       [2022/07/04 20:47:34.387]     result = cmd(**dict(arguments))
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/collection.py", line 626, in insert_one
       [2022/07/04 20:47:34.387]     self._insert_one(
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/collection.py", line 567, in _insert_one
       [2022/07/04 20:47:34.387]     self.__database.client._retryable_write(acknowledged, _insert_command, session)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/mongo_client.py", line 1433, in _retryable_write
       [2022/07/04 20:47:34.387]     return self._retry_with_session(retryable, func, s, None)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/mongo_client.py", line 1318, in _retry_with_session
       [2022/07/04 20:47:34.387]     return self._retry_internal(retryable, func, session, bulk)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/mongo_client.py", line 1353, in _retry_internal
       [2022/07/04 20:47:34.387]     return func(session, sock_info, retryable)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/collection.py", line 555, in _insert_command
       [2022/07/04 20:47:34.387]     result = sock_info.command(
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/pool.py", line 793, in command
       [2022/07/04 20:47:34.387]     self._raise_connection_failure(error)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/pool.py", line 765, in command
       [2022/07/04 20:47:34.387]     return command(
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/network.py", line 121, in command
       [2022/07/04 20:47:34.387]     sock_info.apply_timeout(client, spec, write_concern)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/pool.py", line 583, in apply_timeout
       [2022/07/04 20:47:34.387]     max_time_ms = timeout - rtt
       [2022/07/04 20:47:34.387] TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'
       [2022/07/04 20:47:34.387] ======================================================================
       [2022/07/04 20:47:34.387] FAIL [0.057s]: test_timeoutMS_applied_to_entire_bulkWrite,_not_individual_commands (test_csot.TestCsotBulkWrite)
       [2022/07/04 20:47:34.387] ----------------------------------------------------------------------
       [2022/07/04 20:47:34.387] Traceback (most recent call last):
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1199, in run_entity_operation
       [2022/07/04 20:47:34.387]     result = cmd(**dict(arguments))
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/collection.py", line 534, in bulk_write
       [2022/07/04 20:47:34.387]     bulk_api_result = blk.execute(write_concern, session)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/bulk.py", line 514, in execute
       [2022/07/04 20:47:34.387]     return self.execute_command(generator, write_concern, session)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/bulk.py", line 392, in execute_command
       [2022/07/04 20:47:34.387]     client._retry_with_session(self.is_retryable, retryable_bulk, s, self)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/mongo_client.py", line 1318, in _retry_with_session
       [2022/07/04 20:47:34.387]     return self._retry_internal(retryable, func, session, bulk)
       [2022/07/04 20:47:34.387] TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'
       [2022/07/04 20:47:34.387] During handling of the above exception, another exception occurred:
       [2022/07/04 20:47:34.387] Traceback (most recent call last):
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1472, in test_case
       [2022/07/04 20:47:34.387]     self.run_scenario(spec)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1449, in run_scenario
       [2022/07/04 20:47:34.387]     self.run_operations(spec["operations"])
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1376, in run_operations
       [2022/07/04 20:47:34.387]     self.run_entity_operation(op)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1208, in run_entity_operation
       [2022/07/04 20:47:34.387]     return self.process_error(exc, expect_error)
       [2022/07/04 20:47:34.387]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 955, in process_error
       [2022/07/04 20:47:34.388]     self.assertIsInstance(
       [2022/07/04 20:47:34.388] AssertionError: TypeError("unsupported operand type(s) for -: 'float' and 'NoneType'") is not an instance of (<class 'pymongo.errors.NetworkTimeout'>, <class 'pymongo.errors.ExecutionTimeout'>, <class 'pymongo.errors.ServerSelectionTimeoutError'>)
       [2022/07/04 20:47:34.388] ======================================================================
       [2022/07/04 20:47:34.388] FAIL [0.219s]: test_command_is_not_sent_if_RTT_is_greater_than_timeoutMS (test_csot.TestCsotCommandExecution)
       [2022/07/04 20:47:34.388] ----------------------------------------------------------------------
       [2022/07/04 20:47:34.388] Traceback (most recent call last):
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1199, in run_entity_operation
       [2022/07/04 20:47:34.388]     result = cmd(**dict(arguments))
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/collection.py", line 626, in insert_one
       [2022/07/04 20:47:34.388]     self._insert_one(
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/collection.py", line 567, in _insert_one
       [2022/07/04 20:47:34.388]     self.__database.client._retryable_write(acknowledged, _insert_command, session)
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/mongo_client.py", line 1433, in _retryable_write
       [2022/07/04 20:47:34.388]     return self._retry_with_session(retryable, func, s, None)
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/pymongo/mongo_client.py", line 1318, in _retry_with_session
       [2022/07/04 20:47:34.388]     return self._retry_internal(retryable, func, session, bulk)
       [2022/07/04 20:47:34.388] TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'
       [2022/07/04 20:47:34.388] During handling of the above exception, another exception occurred:
       [2022/07/04 20:47:34.388] Traceback (most recent call last):
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1472, in test_case
       [2022/07/04 20:47:34.388]     self.run_scenario(spec)
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1449, in run_scenario
       [2022/07/04 20:47:34.388]     self.run_operations(spec["operations"])
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1376, in run_operations
       [2022/07/04 20:47:34.388]     self.run_entity_operation(op)
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 1208, in run_entity_operation
       [2022/07/04 20:47:34.388]     return self.process_error(exc, expect_error)
       [2022/07/04 20:47:34.388]   File "/data/mci/7e45ef05a52098a087c4d4c579126acd/src/test/unified_format.py", line 955, in process_error
       [2022/07/04 20:47:34.388]     self.assertIsInstance(
       [2022/07/04 20:47:34.388] AssertionError: TypeError("unsupported operand type(s) for -: 'float' and 'NoneType'") is not an instance of (<class 'pymongo.errors.NetworkTimeout'>, <class 'pymongo.errors.ExecutionTimeout'>, <class 'pymongo.errors.ServerSelectionTimeoutError'>)
      

      https://evergreen.mongodb.com/task/mongo_python_driver_load_balancer__platform~ubuntu_18.04_auth_ssl~noauth_nossl_mongodb_version~6.0_python_version~3.10_loadbalancer~enabled_load_balancer_test_6d916d68c2db341847b46fabf961f3ad4ba045e4_22_07_01_19_36_12

      The problem is that load balanced mode does not run SDAM and hence does not have an rtt at all. Some ways to fix this:

      • use the rtt from the connection handshake for the lifetime of the connection.
      • use a fixed value for the rtt, eg 100ms

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: