Investigate queued-write behavior on failover and add test coverage

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Workload Resilience
    • WR Prioritized list
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When a replica set primary steps down, operations already waiting in the IngressRequestRateLimiter have passed the “can this run here” check but have not taken write locks yet. Stepdown only interrupts those ops if they are already retryable writes (txnNumber set) which fail immediately with InterruptedDueToReplStateChange so the driver can retry on the new primary. Non-retryable writes (retryWrites: false, multi-document updates/deletes, most DDL) stay in the queue with no error, so the client cannot retarget and only fails later when the request queue is drained. We should decide whether interrupting those writes is the right product behavior, or whether leaving them queued is preferable. The opportunity cost here is that the queue drain time becomes added latency to those writes.

      The attached ingress_request_rate_limiter_stepdown.js test covers both sides of that split and should be committed with this ticket. While landing it, audit other admission and rate-limit queues for the same gap: a queued primary-only command that is not a retryable write may similarly survive failover without being interrupted or retried.

            Assignee:
            Unassigned
            Reporter:
            Matt Broadstone
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: