-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Workload Resilience
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
The ingress rate limiter matches ingressRequestRateLimiterApplicationExemptions against the app name of the connection, not of the request. A request routed through mongos reaches the shard mongod on a mongos pool connection named NetworkInterfaceTL-TaskExecutorPool-<n>, so the end client's app name is never seen.
As a result, an exempt client such as the Atlas canary (MongoDB Automation Agent) is exempt at mongos and on direct mongod connections, but is rate limited on the mongos -> mongod hop.
Exempting NetworkInterfaceTL-TaskExecutorPool is not an option: that pool carries the main CRUD path, so it would turn off shard rate limiting for all routed traffic.
Impact
- Canary availability on sharded clusters diverges from replica sets. In BF-46676 (disk_bandwidth_exhaustion_sharded_locust, 3 shards), the mongos -> config canary probe dropped from 99.0% to 78.5% during degradation. Every new failure was code 462 IngressRequestRateLimitExceeded, returned by the shard and config mongods. Direct probes stayed around 98%.
- An earlier experiment that checked the forwarded client metadata on the mongod raised sharded canary availability from 40% to 98%.
- Automation Agent sharding commands (e.g. disable balancer, remove shard) also go through mongos. Many maintenance paths use NetworkInterfaceTL-Sharding-Fixed, which is already exempt.
Why it happens
- SessionWorkflow::Impl::_dispatchWork calls _rateLimit() before the request is parsed and before an operation context exists (src/mongo/transport/session_workflow.cpp). The exemption check can only see the connection's hello metadata.
- Mongos already forwards the end client's metadata on every request as $client (src/mongo/s/client_metadata_propagation_egress_hook.cpp), but the mongod reads it only after admission.
- Execution control does not have this gap: mongos forwards a per-operation admission type to the shard, and the deprioritization exemption is checked after $client is applied.
Possible fixes
- Forward an "exempt" bit in the OP_MSG header. If mongos admitted the original request as exempt, it sets a new optional flag bit on the request to the shard. The mongod honors it only on connections authenticated as internal cluster members. No parsing is needed, and older binaries ignore optional flag bits.
- Dedicated exempt connection pool on mongos. Send requests from exempt clients through a separate executor whose connections are exempt on the mongod (priority port or an exempt pool name).
- Late admission for internal connections. For internal cluster connections, defer the check until $client is parsed and match the forwarded app name. Smallest change, but adds parse cost before rejection for all routed traffic.
Option 1 looks like the best starting point. A non-server alternative raised in discussion is to rate limit only at mongos in colocated mongos + mongod deployments.