Major - P3 src/mongoc/mongoc-topology-scanner.c:671 mongoc_topology_scanner_start(): precondition failed: !node->cmd
In the test configuration that caused the assert, we have a 3-node replica set: primary on port 27017, secondary on 27018, arbiter on 27019.
The test creates a client pool with this connection string:
"mongodb://localhost:27017,localhost:27018/?replicaset=replset&serverselectiontryonce=false"
Note that the arbiter on port 27019 is missing from the seed list.
The test pops a client from the pool. Popping the client triggers the pool to begin the background scanner thread (CDRIVER-1333).
The test creates a cursor and sets the cursor's id to 1. (In this setup, server 1 is usually primary.) The test calls mongoc_cursor_next, which calls down to mongoc_cluster_fetch_stream_pooled, which creates a stream connected to the primary and calls _mongoc_cluster_add_node, which calls _mongoc_cluster_run_ismaster.
Once the isMaster completes, the driver calls _mongoc_topology_update_from_handshake to update its topology description (CDRIVER-562).
If (rarely, apparently) _mongoc_topology_update_from_handshake beats the background scanner thread to the first ismaster, then it's the first function to discover the arbiter and add it to the topology scanner. It does so, and calls mongoc_topology_scanner_add_and_scan, which creates a scanner node for the arbiter and initializes node->cmd, the scanner node's async_cmd_t representing the isMaster command it wants to execute.
Then (again, this is a rare race outcome) the background thread begins and sets up scanner nodes for all known servers. By now the arbiter is in the topology description so the thread attempts to call isMaster on it. But first, it asserts that the arbiter's scanner node doesn't already have an async_cmd_t. In this sequence, the scanner's assert fails.
Solution: when discovering a node from _mongoc_topology_update_from_handshake, don't set its async_cmd_t. That's the job of the topology scanner.