Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-96048

Create a "log once per X occurrences per Y time period" functionality for $externalApi

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • Fully Compatible
    • Sprint 62
    • 200

      What

       

      We'd like to make a new private method in ExternalApiOperator that looks like the following:

       

      throttleInfoLog(LOG_ID, msg, ...attr);

      LOG_ID: the normal log id we use in normal LOGV2 calls. Token bucket rate limiters will be keyed on this field.

      msg: the message

      ...attr: the attributes associated with the log.

       

      We should use this log statement to control the amount of logs we produce in catch blocks within the processStreamDocument function.

       

      We'd like to use these logs to gain insight into what customers are doing to produce failures

      Suggested Implementation

       

      1. Add a stdx::unordered_map that maps the log id key to a token bucket rate limiter for a given ExternalApiOperator instance
      2. Implement a private ExternalApiOperator::throttleInfoLog(LOG_ID, [&] {}) method
      3. when creating a token bucket, hard code the cadence to be 1 per minute for all usages of throttleInfoLog

       

      Suggested logs:

       

      1. Log whenever we produce a DLQ doc when trying to make a request to the external API endpoint. (catch block)
      2. Log whenever we fail to read or build a StreamDocument from response of the external API endpoint (catch block)
      3.  

            Assignee:
            guyjacques.isombe@mongodb.com Guy-Jacques Isombe
            Reporter:
            andrew.chen@mongodb.com Andrew Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: