[SERVER-42732] delayed replica performance downgrade Created: 09/Aug/19 Updated: 06/Dec/22 Resolved: 15/Aug/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, Querying, Replication, Usability |
| Affects Version/s: | 3.6.13 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Markus Kral | Assignee: | Backlog - Triage Team |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Server Triage
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Hi,
beacuse of a mixture of hardware-servers and VMs, we are running a replicaset with 4 active members and 2 arbiters.
Host1: Primary + arbiter Host2: Secondary + arbiter Host3: Secondary Host4: Secondary
As soon as we configure a secondary to be a delayed and hidden replica, queries using the changeStream features clogg up the system until its unusable. Without the delayed replica these queries take < 100ms, with the delayed replica deployed these timings increase significantely up to multiple thousands of ms. Also the systemload is increasing heavily, from ~0.5 without delayed replica, up to observed 6.
The documentation offers the following advice for PSA-Configurations (only if one has only 3 members in the replicaset: master-slave-arbiter): https://docs.mongodb.com/v3.6/reference/read-concern-majority/#disable-read-concern-majority
But this should have no effect in our setup, as we still do have a fully functional PSS replication.
After removing the delay and setting the replica to hidden=false the problems vanish and the system is stable and performant again.
This is reproducible on our setup.
We can |
| Comments |
| Comment by Danny Hatcher (Inactive) [ 15/Aug/19 ] |
|
Using more than one arbiter in a replica set configuration has the opportunity to cause various issues due to having two nodes not actually copying data. The existence of nodes in your replica set still can effect the definitions of "majority"; in your proposed configuration you have a PSSAAH setup which is vastly different from PSS. I recommend refining your environment to only have one arbiter at most (none is ideal) and ensuring that the hidden node cannot vote. The SERVER project is for bugs and feature suggestions for the MongoDB server and this seems to be an environmental issue. If you need further assistance troubleshooting, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag. |