For the mongod processes that resmoke.py creates, an unused port is requested from the OS

self.__socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.__socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.__socket.bind(("0.0.0.0", 0))

by creating a socket, closing it, and then handing the port number that was received from the OS to the mongod command line. Note that this is racy in a way that is not easy to fix.

However, another issue arises from the way resmoke.py starts multiple mongod processes when the --jobs=N flag is specified. The MongoDFixture.setup() method starts the process, but returns before attempting to establish a connection to the mongod. Instead, resmoke.py starts a mongod processes for each job without waiting to see if a connection can be established before starting the next. After all of the mongod processes are started, MongoDFixture.await_ready() is called on each to ensure a connection can be established. This was done so that the initialization of mongod processes can happen concurrently, which decreases the overall execution time.

The following sequence of events describes the problem:

Job 1 gets port 42000
(Job 1 closes its socket)
Job 1 starts a mongod process on port 42000
Job 2 gets port 42000
(Job 2 closes its socket)
The mongod process of Job 1 calls bind() on port 42000 (ok)
Job 2 starts a mongod process on port 42000
The mongod process of Job 2 calls bind() on port 42000 (fails because the address is already in use)

The proposal is to have the UnusedPort class keep track of the set of ports that have been acquired by each instance and to request another port from the OS if a duplicate is received.