Large data sizes cause MyModel.objects.filter() evaluation to raise AutoReconnect: connection pool paused

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Cannot Reproduce
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: django
    • None
    • 🔵 Done
    • Python Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      With 1 million documents, the following query raises a pymongo.errors.AutoReconnect: connection pool paused error:

      list(book_class.objects.filter(publish_date__range=["1980-01-01", "2000-01-01"])) 

      Stack trace:

      pymongo.errors.AutoReconnect: connection pool paused
      
      The above exception was the direct cause of the following exception:Traceback (most recent call last):
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django/core/handlers/exception.py", line 55, in inner
          response = get_response(request)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django/core/handlers/base.py", line 197, in _get_response
          response = wrapped_callback(request, *callback_args, **callback_kwargs)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/benchmarking/filter_operations/views.py", line 125, in benchmark_filter_operations
          run_stats = _run_benchmark(benchmark_function, setup_function, 10)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/benchmarking/filter_operations/views.py", line 72, in _run_benchmark
          trial_stats["cpu_percent"], trial_stats["memory_usage"] = _measure_resource_utilization(benchmark_function)
                                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/benchmarking/filter_operations/views.py", line 47, in _measure_resource_utilization
          func()
          ~~~~^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/benchmarking/filter_operations/views.py", line 122, in benchmark_function
          list(book_class.objects.filter(
          ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
              publish_date__range=["1980-01-01", "2000-01-01"]))  # use list() to force evaluation
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django/db/models/query.py", line 384, in __iter__
          self._fetch_all()
          ~~~~~~~~~~~~~~~^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django/db/models/query.py", line 1949, in _fetch_all
          self._result_cache = list(self._iterable_class(self))
                               ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django/db/models/query.py", line 91, in __iter__
          results = compiler.execute_sql(
              chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size
          )
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django_mongodb_backend/compiler.py", line 266, in execute_sql
          return list(result)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/django_mongodb_backend/compiler.py", line 325, in cursor_iter
          for row in cursor:
                     ^^^^^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/command_cursor.py", line 342, in __next__
          return self.next()
                 ~~~~~~~~~^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/command_cursor.py", line 335, in next
          doc = self._try_next(True)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/command_cursor.py", line 347, in _try_next
          self._refresh()
          ~~~~~~~~~~~~~^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/command_cursor.py", line 307, in _refresh
          self._send_message(
          ~~~~~~~~~~~~~~~~~~^
              self._getmore_class(
              ^^^^^^^^^^^^^^^^^^^^
          ...<12 lines>...
              )
              ^
          )
          ^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/command_cursor.py", line 254, in _send_message
          response = client._run_operation(
              operation, self._unpack_response, address=self._address
          )
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
          return func(self, *args, **kwargs)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 1917, in _run_operation
          return self._retryable_read(
                 ~~~~~~~~~~~~~~~~~~~~^
              _cmd,
              ^^^^^
          ...<4 lines>...
              operation=operation.name,
              ^^^^^^^^^^^^^^^^^^^^^^^^^
          )
          ^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 2026, in _retryable_read
          return self._retry_internal(
                 ~~~~~~~~~~~~~~~~~~~~^
              func,
              ^^^^^
          ...<7 lines>...
              operation_id=operation_id,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
          )
          ^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
          return func(self, *args, **kwargs)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 1993, in _retry_internal
          ).run()
            ~~~^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 2730, in run
          return self._read() if self._is_read else self._write()
                 ~~~~~~~~~~^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 2877, in _read
          with self._client._conn_from_server(self._read_pref, self._server, self._session) as (
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/.pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 141, in __enter__
          return next(self.gen)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 1846, in _conn_from_server
          with self._checkout(server, session) as conn:
               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/.pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 141, in __enter__
          return next(self.gen)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/mongo_client.py", line 1756, in _checkout
          with server.checkout(handler=err_handler) as conn:
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/.pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 141, in __enter__
          return next(self.gen)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/pool.py", line 1119, in checkout
          conn = self._get_conn(checkout_started_time, handler=handler)
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/pool.py", line 1235, in _get_conn
          self._raise_if_not_ready(checkout_started_time, emit_event=True)
          ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/synchronous/pool.py", line 1187, in _raise_if_not_ready
          _raise_connection_failure(
          ~~~~~~~~~~~~~~~~~~~~~~~~~^
              self.address, AutoReconnect("connection pool paused"), timeout_details=details
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          )
          ^
        File "/Users/nstapp/Github/django-mongodb-backend-benchmark/.venv/lib/python3.13/site-packages/pymongo/pool_shared.py", line 149, in _raise_connection_failure
          raise AutoReconnect(msg) from error
      pymongo.errors.AutoReconnect: djangobenchmarking-shard-00-01.oztdp.mongodb-dev.net:27017: connection pool paused (configured timeouts: connectTimeoutMS: 20000.0ms) 

      Should we explicitly give guidance to users on when evaluating a large query in this manner may fail?

            Assignee:
            Unassigned
            Reporter:
            Noah Stapp
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: