-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
None
-
Component/s: Mongo Orchestration
-
None
-
Not Needed
Summary
mongo-orchestration returns a 500 "internal server error" when asked to create multiple servers with unique ports (by leaving the port empty) the first time this is requested. Making the same request again will succeed, so this appears to be a race condition on the first time mongo-orchestration populates its list of available ports. I can reproduce this with the following request for a basic replica set with two nodes:
$ curl -XPOST --data "{\"id\": \"rs0\", \"members\": [{}, {}]}" "http://localhost:8889/v1/replica_sets" Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.10/site-packages/mongo_orchestration/apps/__init__.py", line 66, in wrap return f(*arg, **kwd) File "/home/ubuntu/.local/lib/python3.10/site-packages/mongo_orchestration/apps/replica_sets.py", line 80, in rs_create result = _rs_create(data) File "/home/ubuntu/.local/lib/python3.10/site-packages/mongo_orchestration/apps/replica_sets.py", line 37, in _rs_create rs_id = ReplicaSets().create(params) File "/home/ubuntu/.local/lib/python3.10/site-packages/mongo_orchestration/replica_sets.py", line 658, in create repl = ReplicaSet(rs_params) File "/home/ubuntu/.local/lib/python3.10/site-packages/mongo_orchestration/replica_sets.py", line 95, in __init__ if not self.repl_init(config): File "/home/ubuntu/.local/lib/python3.10/site-packages/mongo_orchestration/replica_sets.py", line 210, in repl_init result = self.connection(init_server).admin.command("replSetInitiate", config) File "/usr/local/lib/python3.10/dist-packages/pymongo/_csot.py", line 119, in csot_wrapper return func(self, *args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/pymongo/synchronous/database.py", line 930, in command return self._command( File "/usr/local/lib/python3.10/dist-packages/pymongo/synchronous/database.py", line 770, in _command return conn.command( File "/usr/local/lib/python3.10/dist-packages/pymongo/synchronous/helpers.py", line 47, in inner return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/pymongo/synchronous/pool.py", line 536, in command return command( File "/usr/local/lib/python3.10/dist-packages/pymongo/synchronous/network.py", line 213, in command helpers_shared._check_command_response( File "/usr/local/lib/python3.10/dist-packages/pymongo/helpers_shared.py", line 247, in _check_command_response raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: Found two member configurations with same host field, members.0.host == members.1.host == localhost:1030, full error: {'ok': 0.0, 'errmsg': 'Found two member configurations with same host field, members.0.host == members.1.host == localhost:1030', 'code': 93, 'codeName': 'InvalidReplicaSetConfig'}
Motivation
Who is the affected end user?
Developers may experience this the first time an integration test is run. See https://jira.mongodb.org/browse/TUNE-197
We can work around this issue in our CI by requesting mongo-orchestration to create a single server (with no port assigned). This ensures that the port pool is initialized and subsequent requests succeed.
How does this affect the end user?
Developers will be confused by this error and may spend time trying to track down what went wrong in a test when it is really mongo-orchestration at fault.
How likely is it that this problem or use case will occur?
Every first run of an integration test on a new mongo-orchestration instance is affected.
If the problem does occur, what are the consequences and how severe are they?
Annoyance/distraction/time-sink.
Is this issue urgent?
It would be ideal to fix this ASAP, but the workaround of retrying the request makes this technically non-blocking.
Is this ticket required by a downstream team?
Mongotune.
Is this ticket only for tests?
Just tests / dev workflow.
Acceptance Criteria
Basic request of mongo-orchestration assigning ports automatically should succeed on the first try.