[SERVER-6027] Replication suffers on a server with page faults and degraded hardware Created: 07/Jun/12 Updated: 06/Dec/22 Resolved: 03/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.0.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Colin Howe | Assignee: | Backlog - Replication Team |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Replication
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Our primary server had a hard drive failure that resulted in read/write performance being ~10x slower than normal. We normally have a low number of page faults but this became worse as reads started to pile up. We wanted to failover to a server with good hardware but we couldn't as the secondaries replication had both fallen behind - and they were getting further behind. After some digging through logs we discovered that the oplog was being sent to the secondaries extremely slowly (~5 minutes for a single query). I then spoke to Scott Hernandez on IRC and he confirmed my suspicion - the parts of the oplog we needed had been paged out and were being read from disk. Due to the degraded hardware these reads were incredibly slow. We had to shutdown our entire service to allow the server to dedicate its poorly performing disks to serving the oplog so we could fail over to better hardware. This isn't ideal. In my opinion, reading the oplog should always get priority over other reads - if replication falls behind and you have to hit disk to get the oplog then replication will likely carry on falling further and further behind. I'd much rather see the reads starting to fail and know I can failover (whilst keeping data). |
| Comments |
| Comment by Judah Schvimer [ 03/Jan/20 ] |
|
Closing "Gone Away" since the server has changed significantly, including in the locking system, since this was originally described. If this is still a problem on a supported version of MongoDB, please re-open with an explanation of what is occurring. |
| Comment by Eric Milkie [ 05/Mar/13 ] |
|
Unfortunately, other users want reads and writes to never fail and would rather have replication be down for a bit, especially if they are only using replication for backup and failover. Finding a happy medium is difficult here. A new lock scheduling algorithm for the server is planned as future work, after which time we might be able to prioritize replication machinery at the expense of client read and write performance. |