Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-59858

Add observability for tasks scheduled on the reactor thread

    XMLWordPrintableJSON

Details

    • Fully Compatible
    • v5.0, v4.4, v4.2
    • Service Arch 2021-10-04, Service Arch 2021-10-18, Service Arch 2021-11-01, Service Arch 2021-11-15
    • 37
    • 3

    Description

      Extend the implementation of ASIO reactor (defined here) to add a section to serverStatus. This section should include histograms that track the latency breakdown for tasks scheduled on the rector thread (created here). In particular, we are interested in collecting the waiting and execution time for each task:

      void ASIOReactor::schedule(Task task) {
          Date_t scheduled = now();
          asio::post(_ioContext, [this, scheduled, task = std::move(task)] {        
              Date_t started = now();
              const auto waitingTime = started - scheduled;
              task(Status::OK());
              const auto executionTime = now() - started;
              // TODO update the histograms and log if necessary.
          });
      }
      

      The histograms could have 21 buckets, allocating 10 buckets for microsecond latencies (e.g., 0 to 100, 100 to 200, and so on), another 10 for millisecond latencies, and a final bucket for latencies over 1 second.

      Additionally, large latencies should be logged at debug, info, or warning levels, depending on predefined thresholds (e.g., 50, 500, and 1000 ms). It'd be helpful to also log the stack-trace for very large latencies.

      Attachments

        Issue Links

          Activity

            People

              amirsaman.memaripour@mongodb.com Amirsaman Memaripour
              amirsaman.memaripour@mongodb.com Amirsaman Memaripour
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: