Reduce MirrorMaestro dispatch blocking on connection threads via thread-local batched dispatch

XMLWordPrintableJSON

    • Product Performance
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      Off-CPU profiling on YCSB 100read at 128 threads shows connection threads blocking on pthread_cond_signal when dispatching mirror requests, with 10–16ms of off-CPU time per occurrence. MirrorMaestroImpl::tryMirror is invoked on every mirror-eligible command from the service executor thread, and it calls ExecutorFuture(_executor).getAsync(...) which goes through GuaranteedExecutor::scheduleThreadPoolTaskExecutor::scheduleWorkThreadPool::Impl::schedule. Inside that last call the path acquires the ThreadPool task-queue mutex, enqueues the task, and calls _workAvailable.notify_one() — with 128 threads concurrently scheduling mirror requests, the ThreadPool mutex and condition variable become contention points and the per-request pthread_cond_signal cost dominates the dispatch path.

      Solution

      Replace the synchronous one-mirror-per-getAsync dispatch in MirrorMaestroImpl::tryMirror (src/mongo/db/mirror_maestro.cpp) with a thread-local batch queue. Each connection thread accumulates its mirror requests in a per-thread PerThreadBatch (held by shared_ptr so trailing entries survive the thread's exit), and when the batch reaches kBatchSize=16 the entire batch is moved into a single getAsync task that re-enters the existing _mirror loop for every accumulated request. A 100ms periodic flusher (registered via ServiceContext::getPeriodicRunner()->makeJob and held in a PeriodicJobAnchor) walks an _batchRegistry of all per-thread batches and drains any partial batches so that mirroring metrics (gMirroredReadsSection.sent, .pending) converge within the existing assert.soon budgets used by mirror-reads jstests. Per-instance batch ownership is tracked by a monotonic _instanceId so test fixtures that destroy and recreate MirrorMaestroImpl on the same thread re-register cleanly across address-reuse boundaries on debug/TSAN builds.

            Assignee:
            Jawwad Asghar
            Reporter:
            Jawwad Asghar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: