Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-59858

Add observability for tasks scheduled on the reactor thread

    XMLWordPrintable

    Details

    • Backport Requested:
      v5.0, v4.4, v4.2
    • Sprint:
      Service Arch 2021-10-04, Service Arch 2021-10-18, Service Arch 2021-11-01
    • Case:
    • Linked BF Score:
      37
    • Story Points:
      3

      Description

      Extend the implementation of ASIO reactor (defined here) to add a section to serverStatus. This section should include histograms that track the latency breakdown for tasks scheduled on the rector thread (created here). In particular, we are interested in collecting the waiting and execution time for each task:

      void ASIOReactor::schedule(Task task) {
          Date_t scheduled = now();
          asio::post(_ioContext, [this, scheduled, task = std::move(task)] {        
              Date_t started = now();
              const auto waitingTime = started - scheduled;
              task(Status::OK());
              const auto executionTime = now() - started;
              // TODO update the histograms and log if necessary.
          });
      }
      

      The histograms could have 21 buckets, allocating 10 buckets for microsecond latencies (e.g., 0 to 100, 100 to 200, and so on), another 10 for millisecond latencies, and a final bucket for latencies over 1 second.

      Additionally, large latencies should be logged at debug, info, or warning levels, depending on predefined thresholds (e.g., 50, 500, and 1000 ms). It'd be helpful to also log the stack-trace for very large latencies.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              amirsaman.memaripour Amirsaman Memaripour
              Reporter:
              amirsaman.memaripour Amirsaman Memaripour
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated: