ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This ticket decomposes pipelined-apply latency into per-stage counters under serverStatus().repl.pipelinedApply.

      This ticket should include:

      • Micros counters: batcherWaitMicros, dispatchMicros, backpressureMicros, drainMicros, inlineApplyMicros, publishWaitMicros, workerApplyMicros, workerIdleMicros, advancerPublishMicros, advancerMultikeyMicros.
      • Backpressure trigger counter: backpressure.inflight
      • A workerIdleMicros hook in the PipelinedApplierWorkerPool — the pool has no idle callback today, so one must be added.

      edit: updated list of counters:

      dispatcher loop:

      • request batch (batcherWaitMicros)
      • if must serialize:
        • drain (see below)
        • dispatch to one worker (serializedApplyMicros, see below)
        • drain again (see below)
      • else, if can parallelize:
      • dispatch to workers (parallelApplyMicros, see below)

       
      drain:

      • start timer for drainWorkers
        • wait for worker pool to become idle
        • wait for advancer to become idle
          • (every time the advancer waits for a new batch from the tracker, it updates publishWaitMicros)
          • (every time the advancer publishes, it updates advancerPublishMicros and/or advancerMultikeyMicros)

       
      dispatch:

      • start dispatch timer for dispatchMicros
        • pause timer
        • add batch to batch tracker
          • check if backpressure necessary (backpressureMicros)
        • assign batches to workers (see below)
        • resume timer

       
      worker loop:

      • wait for a batch
      • process the batch (workerApplyMicros)

            Assignee:
            Joseph Obaraye
            Reporter:
            Ali Mir
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: