beacuse of a mixture of hardware-servers and VMs, we are running a replicaset with 4 active members and 2 arbiters.
Host1: Primary + arbiter
Host2: Secondary + arbiter
As soon as we configure a secondary to be a delayed and hidden replica, queries using the
changeStream features clogg up the system until its unusable.
Without the delayed replica these queries take < 100ms, with the delayed replica deployed these timings increase significantely up to multiple thousands of ms.
Also the systemload is increasing heavily, from ~0.5 without delayed replica, up to observed 6.
The documentation offers the following advice for PSA-Configurations (only if one has only 3 members in the replicaset: master-slave-arbiter):
But this should have no effect in our setup, as we still do have a fully functional PSS replication.
After removing the delay and setting the replica to hidden=false the problems vanish and the system is stable and performant again.
This is reproducible on our setup.