A couple times recently the Evergreen database has held an election. This is unusual, and in the second case caused a brief outage. This might either indicate some bug in our code, or potentially some task is putting unusually high load on the database.
The first was 4/26 around 3pm ET. The second was 4/27 at 9:42pm ET.
The questions we should investigate include:
- Is there some task that was running during both times?
- Is there some increase in access to some API endpoint?
- Is there anything useful on the APM dashboard that might be a clue?
- Do the logs indicate some kind of error with the database itself?