Problem

jstests/concurrency/fsm_workloads/timeseries/timeseries_reads_setFCV.js fails in the concurrency_sharded_replication_rate_limited suite when one of its reads is shed by the ingress request rate limiter with error 462 IngressRequestRateLimitExceeded.

This is the same class of issue as SERVER-131819, which hardened the sibling workload timeseries_upgrade_downgrade_with_crud.js. That fix did not cover this file.

Observed on mongodb-mongo-v9.0-staging @ 4a566a41, variant suse15, task concurrency_sharded_replication_rate_limited_2-linux. The occurrence was misfiled against BF-44699, which tracks the (already fixed) timeseries_upgrade_downgrade_with_crud.js failure.

Root cause

The workload routes all of its read states through a single retry helper that accepts exactly one error code:

// Runs `func` and retries if it is interrupted with a transient timeseries upgrade/downgrade error.
function withRetryOnTimeseriesUpgradeDowngradeError(func) {
    let result;
    assert.soonRetryOnAcceptableErrors(
        () => {
            result = func();
            return true;
        },
        ErrorCodes.InterruptedDueToTimeseriesUpgradeDowngrade,
        "Timed out waiting for timeseries operation to succeed without upgrade/downgrade error",
    );
    return result;
}

IngressRequestRateLimitExceeded is not in the accepted list, so assert.soonRetryOnAcceptableErrors rethrows immediately and the FSM thread dies.

The rate limiter rejection is expected behavior in this suite: it sets failpoint.failIngressRequestRateLimiting with activationProbability: 0.25 on the shards. The error is returned with errorLabels: ["SystemOverloadedError"] and without RetryableError, so mongos does not retry it internally (reads are not marked idempotent; SERVER-108898 is not planned) and it surfaces directly to the shell.

Failure analysis

From the failing run, the whole event is one client operation (conn8697 on mongos [j0:s1]) spanning 2963 ms:

• 23:33:31.165 - Unable to establish remote cursors: StaleConfig (13388) from shard-rs1, "timestamp mismatch detected for test24_fsmdb0.system.buckets.timeseries_reads_setFCV_5". mongos refreshes routing and retries.
• 23:33:33.680 - StaleConfig again, this time with vWanted epoch/timestamp all zeros, i.e. the buckets namespace is no longer tracked (the viewless timeseries conversion replaced it). mongos retries again.
• 23:33:34.201 - final attempt rejected with 462 IngressRequestRateLimitExceeded, durationMillis: 2963.

The rejected command is the findOne state:

{"find": "timeseries_reads_setFCV_5", "filter": {"t": {"$date": "2024-01-01T00:00:00.000Z"}},
 "limit": 1, "singleBatch": true, "projection": {"_id": 0}}

Notably there is exactly one 462 in the entire 33 MB task log. The concurrent FCV transition is what made this reachable: each StaleConfig-forced retry is another roll of the 0.25 probability against the shard's limiter.

Proposed fix

Add ErrorCodes.IngressRequestRateLimitExceeded to the accepted list in withRetryOnTimeseriesUpgradeDowngradeError. Because every read state already funnels through this helper, the one change covers all seven of them: find, findWithMajority, findOne, aggregate, countDocuments, collStatsCmd, collStatsAgg.

Retry, do not swallow. SERVER-131819 swallowed the error in the sibling workload, which was correct there because those states assert nothing about the result. Here the states assert on results afterwards (assert.sameMembers, assert.eq(doc, expectedDocs[0]), assert.hasFields), so swallowing would yield undefined and cascade into a confusing assertion failure. The existing helper already retries, which is the right treatment.

Scope note

grep -rln "IngressRequestRateLimitExceeded" jstests/concurrency/fsm_workloads/ currently matches only timeseries_upgrade_downgrade_with_crud.js. Other workloads in this suite may have the same latent gap and are worth a sweep.

Related files

• jstests/concurrency/fsm_workloads/timeseries/timeseries_reads_setFCV.js
• Prior art: jstests/concurrency/fsm_workloads/timeseries/timeseries_upgrade_downgrade_with_crud.js (SERVER-131819)