Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-59858

Add observability for tasks scheduled on the reactor thread

    • Fully Compatible
    • v5.0, v4.4, v4.2
    • Service Arch 2021-10-04, Service Arch 2021-10-18, Service Arch 2021-11-01, Service Arch 2021-11-15
    • 37
    • 3

      Extend the implementation of ASIO reactor (defined here) to add a section to serverStatus. This section should include histograms that track the latency breakdown for tasks scheduled on the rector thread (created here). In particular, we are interested in collecting the waiting and execution time for each task:

      void ASIOReactor::schedule(Task task) {
          Date_t scheduled = now();
          asio::post(_ioContext, [this, scheduled, task = std::move(task)] {        
              Date_t started = now();
              const auto waitingTime = started - scheduled;
              task(Status::OK());
              const auto executionTime = now() - started;
              // TODO update the histograms and log if necessary.
          });
      }
      

      The histograms could have 21 buckets, allocating 10 buckets for microsecond latencies (e.g., 0 to 100, 100 to 200, and so on), another 10 for millisecond latencies, and a final bucket for latencies over 1 second.

      Additionally, large latencies should be logged at debug, info, or warning levels, depending on predefined thresholds (e.g., 50, 500, and 1000 ms). It'd be helpful to also log the stack-trace for very large latencies.

            Assignee:
            amirsaman.memaripour@mongodb.com Amirsaman Memaripour
            Reporter:
            amirsaman.memaripour@mongodb.com Amirsaman Memaripour
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: