-
Type: Task
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Atlas Streams
-
Fully Compatible
-
Sprint 62
-
200
What
We'd like to make a new private method in ExternalApiOperator that looks like the following:
throttleInfoLog(LOG_ID, msg, ...attr);
LOG_ID: the normal log id we use in normal LOGV2 calls. Token bucket rate limiters will be keyed on this field.
msg: the message
...attr: the attributes associated with the log.
We should use this log statement to control the amount of logs we produce in catch blocks within the processStreamDocument function.
We'd like to use these logs to gain insight into what customers are doing to produce failures
Suggested Implementation
- Add a stdx::unordered_map that maps the log id key to a token bucket rate limiter for a given ExternalApiOperator instance
- Implement a private ExternalApiOperator::throttleInfoLog(LOG_ID, [&] {}) method
- when creating a token bucket, hard code the cadence to be 1 per minute for all usages of throttleInfoLog
Suggested logs:
- Log whenever we produce a DLQ doc when trying to make a request to the external API endpoint. (catch block)
- Log whenever we fail to read or build a StreamDocument from response of the external API endpoint (catch block)
- depends on
-
SERVER-97052 Add decimal token rate support to rate limiter
- Closed