Cluster state

• Configuration
  • 1 Primary
  • 2 Secondaries (one secondary does not participate in voting and acts like a standby)
  • 1 Arbiter
• Data size
  • 1.9TB of data in one collection
  • 30GB index size

Incident timeline (all times are in IST)

Incident date: 29th May 2022

• 12:12: One of the secondaries crashed due to a segmentation fault.
• 12:12: Increase in Pages queued for eviction along with errors in page eviction
• 12:12 - 14:20: Increase in pages queued for eviction as well as page eviction errors still ongoing with some occasional drops in the increase. These drops correspond to checkpoint activities.
• 14:30 - 14:52: Secondary comes back up and starts syncing from primary. Again seeing increase in page queued for eviction as well as page eviction errors.
• 14:52: All metrics normalized.
• 16:00 - now: Increase in page eviction queues and page eviction errors, not subsided yet. Happening as of now.