[SERVER-59858] Add observability for tasks scheduled on the reactor thread Created: 09/Sep/21  Updated: 29/Oct/23  Resolved: 28/Oct/21

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: 4.4.11, 5.1.0-rc3, 5.0.5

Type: Improvement Priority: Major - P3
Reporter: Amirsaman Memaripour Assignee: Amirsaman Memaripour
Resolution: Fixed Votes: 0
Labels: servicearch-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-61018 Create a generic histogram type Closed
is related to SERVER-58408 Improve diagnostics for networking re... Backlog
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.0, v4.4, v4.2
Sprint: Service Arch 2021-10-04, Service Arch 2021-10-18, Service Arch 2021-11-01, Service Arch 2021-11-15
Participants:
Case:
Linked BF Score: 37
Story Points: 3

 Description   

Extend the implementation of ASIO reactor (defined here) to add a section to serverStatus. This section should include histograms that track the latency breakdown for tasks scheduled on the rector thread (created here). In particular, we are interested in collecting the waiting and execution time for each task:

void ASIOReactor::schedule(Task task) {
    Date_t scheduled = now();
    asio::post(_ioContext, [this, scheduled, task = std::move(task)] {        
        Date_t started = now();
        const auto waitingTime = started - scheduled;
        task(Status::OK());
        const auto executionTime = now() - started;
        // TODO update the histograms and log if necessary.
    });
}

The histograms could have 21 buckets, allocating 10 buckets for microsecond latencies (e.g., 0 to 100, 100 to 200, and so on), another 10 for millisecond latencies, and a final bucket for latencies over 1 second.

Additionally, large latencies should be logged at debug, info, or warning levels, depending on predefined thresholds (e.g., 50, 500, and 1000 ms). It'd be helpful to also log the stack-trace for very large latencies.



 Comments   
Comment by Githook User [ 10/Nov/21 ]

Author:

{'name': 'Amirsaman Memaripour', 'email': 'amirsaman.memaripour@mongodb.com', 'username': 'samanca'}

Message: SERVER-59858 Add observability for tasks scheduled on reactor threads

(cherry picked from commit 5df159a364ec3a94d1e1ae01c70e3ca33cb10b4f)
Branch: v4.4
https://github.com/mongodb/mongo/commit/b89585f52685ca1d2d2442b09f49e284cd1cabcd

Comment by Githook User [ 09/Nov/21 ]

Author:

{'name': 'Amirsaman Memaripour', 'email': 'amirsaman.memaripour@mongodb.com', 'username': 'samanca'}

Message: SERVER-59858 Add observability for tasks scheduled on reactor threads

(cherry picked from commit 5df159a364ec3a94d1e1ae01c70e3ca33cb10b4f)
Branch: v5.0
https://github.com/mongodb/mongo/commit/b91c9dc3564288761624606af6371bfb02890883

Comment by Githook User [ 28/Oct/21 ]

Author:

{'name': 'Amirsaman Memaripour', 'email': 'amirsaman.memaripour@mongodb.com', 'username': 'samanca'}

Message: SERVER-59858 Add observability for tasks scheduled on reactor threads
Branch: master
https://github.com/mongodb/mongo/commit/5df159a364ec3a94d1e1ae01c70e3ca33cb10b4f

Generated at Thu Feb 08 05:48:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.