Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8025

Give the replica set more time to be initialized in many-collection-test

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • 3
    • Storage - Ra 2021-09-06

      As part of upgrading our Ubuntu distribution from 18.04 to 20.04, an issue we are running into is that many-collection-test fails in evergreen due to ServerSelectionTimeoutError.

      Here is the link to the failure: https://evergreen.mongodb.com/task_log_raw/wiredtiger_large_scale_tests_many_collection_test_patch_f25c636e0d9d6559386b732d71f943504e01870b_61282c6f306615453ff1a2a2_21_08_27_00_06_07/0?type=T

      Traceback:

       [2021/08/27 01:08:26.656] Traceback (most recent call last):
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1458, in _retry_internal
       [2021/08/27 01:08:26.656]     server = self._select_server(writable_server_selector, session)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1346, in _select_server
       [2021/08/27 01:08:26.656]     server = topology.select_server(server_selector)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/topology.py", line 244, in select_server
       [2021/08/27 01:08:26.656]     return random.choice(self.select_servers(selector,
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/topology.py", line 202, in select_servers
       [2021/08/27 01:08:26.656]     server_descriptions = self._select_servers_loop(
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/topology.py", line 218, in _select_servers_loop
       [2021/08/27 01:08:26.656]     raise ServerSelectionTimeoutError(
       [2021/08/27 01:08:26.656] pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 612838eec6cc03a2fac91640, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>
       [2021/08/27 01:08:26.656] During handling of the above exception, another exception occurred:
       [2021/08/27 01:08:26.656] Traceback (most recent call last):
       [2021/08/27 01:08:26.656]   File "/opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/python3-v3.KL4/lib/python3.9/multiprocessing/pool.py", line 125, in worker
       [2021/08/27 01:08:26.656]     result = (True, func(*args, **kwds))
       [2021/08/27 01:08:26.656]   File "/opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/python3-v3.KL4/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
       [2021/08/27 01:08:26.656]     return list(map(*args))
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/many-collection/../many-collection-test.py", line 624, in populate_collections_worker
       [2021/08/27 01:08:26.656]     bulk.execute()
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/bulk.py", line 709, in execute
       [2021/08/27 01:08:26.656]     return self.__bulk.execute(write_concern, session=None)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/bulk.py", line 529, in execute
       [2021/08/27 01:08:26.656]     return self.execute_command(generator, write_concern, session)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/bulk.py", line 360, in execute_command
       [2021/08/27 01:08:26.656]     client._retry_with_session(
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
       [2021/08/27 01:08:26.656]     return self._retry_internal(retryable, func, session, bulk)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1476, in _retry_internal
       [2021/08/27 01:08:26.656]     raise last_error
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
       [2021/08/27 01:08:26.656]     return func(session, sock_info, retryable)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/bulk.py", line 354, in retryable_bulk
       [2021/08/27 01:08:26.656]     self._execute_command(
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/bulk.py", line 310, in _execute_command
       [2021/08/27 01:08:26.656]     result, to_send = bwc.execute(ops, client)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/message.py", line 950, in execute
       [2021/08/27 01:08:26.656]     result = self.write_command(request_id, msg, to_send)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/message.py", line 1042, in write_command
       [2021/08/27 01:08:26.656]     reply = self.sock_info.write_command(request_id, msg)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/pool.py", line 794, in write_command
       [2021/08/27 01:08:26.656]     reply = self.receive_message(request_id)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/pool.py", line 753, in receive_message
       [2021/08/27 01:08:26.656]     self._raise_connection_failure(error)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/pool.py", line 751, in receive_message
       [2021/08/27 01:08:26.656]     return receive_message(self, request_id, self.max_message_size)
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/network.py", line 196, in receive_message
       [2021/08/27 01:08:26.656]     _receive_data_on_socket(sock_info, 16, deadline))
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/venv/lib/python3.9/site-packages/pymongo/network.py", line 293, in _receive_data_on_socket
       [2021/08/27 01:08:26.656]     raise AutoReconnect("connection closed")
       [2021/08/27 01:08:26.656] pymongo.errors.AutoReconnect: connection closed
       [2021/08/27 01:08:26.656] """
       [2021/08/27 01:08:26.656] The above exception was the direct cause of the following exception:
       [2021/08/27 01:08:26.656] Traceback (most recent call last):
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/many-collection/../many-collection-test.py", line 798, in <module>
       [2021/08/27 01:08:26.656]     populate_collections(num_collections, docs_per, multiprocessing.cpu_count())
       [2021/08/27 01:08:26.656]   File "/data/mci/a4dd170b398cba0dc39c1fa65c45f7a5/mongo-tests/largescale/many-collection/../many-collection-test.py", line 649, in populate_collections
       [2021/08/27 01:08:26.656]     p.map(populate_collections_worker, pop_workers)
       [2021/08/27 01:08:26.656]   File "/opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/python3-v3.KL4/lib/python3.9/multiprocessing/pool.py", line 364, in map
       [2021/08/27 01:08:26.656]     return self._map_async(func, iterable, mapstar, chunksize).get()
       [2021/08/27 01:08:26.656]   File "/opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/python3-v3.KL4/lib/python3.9/multiprocessing/pool.py", line 771, in get
       [2021/08/27 01:08:27.140]     raise self._value
       [2021/08/27 01:08:27.140] pymongo.errors.AutoReconnect: connection closed
       [2021/08/27 01:08:27.140] Command failed: command encountered problem: error waiting on process '50e9286c-ab42-4727-ab5f-62e97f15d19f': exit status 1
      

       

            Assignee:
            etienne.petrel@mongodb.com Etienne Petrel
            Reporter:
            siddhartha.mahajan@mongodb.com Sid Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: