ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DB Integration & Observability
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      Recording a span for a traced request costs far more than the 10% latency budget in SPM-3868's scope doc allows: on crud_bm find/insert, tracing every request added about 28%/27% CPU. About half of that was the work of recording the span itself, both in our own span bookkeeping and in the OpenTelemetry C++ SDK's per-span machinery. The other half came from that work's memory and code footprint slowing the rest of the request. This ticket reduces both:

      1. Part 1, span bookkeeping: make our own per-span bookkeeping cheap.
      2. Part 2, lightweight span recorder: record spans with a lightweight recorder that keeps the SDK's exporters but skips its per-span layers.

      Background: how a traced request records a span today

      Server code creates spans with mongo::otel::traces::Span::start*() (src/mongo/otel/traces/span/span.cpp). Once the sampler has decided to trace (see SERVER-137029), Span::_start:

      1. Gets a tracer: getGlobalTracerProviderService()>getTracerProvider()>GetTracer("mongodb"). GetTracer takes a lock in the SDK.
      2. Calls tracer->StartSpan(name, options) with the parent span from the operation's SpanTelemetryContextImpl (span_telemetry_context_impl.h), which tracks the operation's current span.
      3. Wraps the result in a Span::SpanImpl, which makes the new span current on the context, and restores the parent when it ends. SpanImpl::setStatus sets errorCode/errorCodeString attributes on every request.
        Inside the SDK, Tracer::StartSpan (src/third_party/opentelemetry-cpp/sdk/src/trace/tracer.cc):
      • looks up the thread-local active context, even though we always pass the parent explicitly;
      • runs the SDK's own sampler, which duplicates ours;
      • heap-allocates a SpanContext and a Span;
      • creates a recordable through a MultiSpanProcessor/MultiRecordable layer that is always present, even with one processor. It adds a map lookup and an extra virtual call to every field set, including about 8 at span start.

      Part 1: span bookkeeping

      Problems:

      • SpanTelemetryContextImpl stored the current span inside an OTel Context. Context is immutable, so each setSpan allocated a new node (Context::SetValue), and each getSpan did a string-keyed lookup that allocated a placeholder span when empty.
      • Each call took a std::mutex. The lock is needed (SERVER-131480: spans can be started or ended on two threads, or the context serialized while another thread changes it), but a mutex is heavier than needed for a pointer copy.
      • The tracer was fetched (with its lock) on every span, and the provider was looked up before the sampling decision.
        Changes:
      1. SpanTelemetryContextImpl stores the current span (ScopedSpan) directly. It builds an OTel Context only in propagate(), i.e. when sending a sampled request onward. When there is no span, getSpan() returns one shared immortal invalid span instead of allocating.
      2. The lock becomes mongo::SpinLock (src/mongo/util/concurrency/spin_lock.h); keep the TSAN test from SERVER-131480.
      3. TracerProviderService::getTracer() (tracer_provider_service.{h,cpp}) caches the tracer per thread, next to the per-thread provider snapshot it already keeps. It is keyed by service id and snapshot version, so a replaced provider never hands out a stale tracer.
      4. Span::_start gets the tracer only after the sampling decision.
        Measured (profiling build, cycles per traced request): Context::SetValue 2,510 -> 416; parent restore 2,304 -> 333; context release 2,358 -> 368; GetSpan 758 -> 282; tracer lookup 729 -> 184. More than half of the total gain came from non-tracing code running faster again (less cache pollution). Native, with the SDK recorder: about 1.3 us less per traced request.

      Part 2: lightweight span recorder

      Design: LeanTracerProvider (src/mongo/otel/traces/lean_tracer.{h,cpp}) implements the OpenTelemetry API interfaces TracerProvider, Tracer and Span:

      • StartSpan generates ids, builds the SpanContext (continuing the parent's trace if any), asks the span processor for a recordable, and writes the start fields into it directly, with no multi-processor layer.
      • Spans write attributes, events and status straight into the recordable. End() sets the duration and hands the recordable to the processor.
      • The SDK's span processors and exporters are used unchanged, so transports (HTTP, file, gRPC), batching and retries keep coming from SDK upgrades. Call sites and TracerProviderService don't change: it's selected by which provider the factory installs.
      • Every span it starts is sampled, because TracingSampler already decided before StartSpan is called.
      • One allocation per span (make_shared). Spans keep a raw pointer to their tracer: the provider must outlive spans and their export, which the recordables' pointers to the resource and scope already require.
        Productionization:
      1. Install it in createHttpTracerProviderService()/createFileTracerProviderService() (tracer_provider_service_factory.cpp), behind a startup parameter (default lean, fallback SDK).
      2. Handle LeanTracerProvider in TracerProviderService::shutdown(), which today only knows the SDK and no-op providers.
      3. Use the same resource attributes as today (service.name, service.instance.id, user-configured attributes).
      4. Add a conformance test that records the same spans through the SDK recorder and the lightweight one and compares the exported data, so SDK upgrades are checked automatically.
      5. Document the contract that a span is used by one thread at a time. Our Span wrapper already guarantees it; the SDK's span doesn't require it.

      Where to start

      • Prototype: everything above is implemented and unit-tested on <PROTOTYPE_BRANCH>:
        • Part 1: span_telemetry_context_impl.{h,cpp}, tracer_provider_service.{h,cpp}, { Unknown macro: {span.cpp}
          }
        • Part 2: lean_tracer.{h,cpp}, { Unknown macro: {lean_tracer_test.cpp}
          }
      • Order: land Part 1 first; it's small, low-risk, and helps the SDK recorder too.
      • Tests: bazel run //src/mongo/otel/traces/span:span_telemetry_context_impl_test, //src/mongo/otel/traces:tracer_provider_service_test, //src/mongo/otel/traces:lean_tracer_test, //src/mongo/otel/traces/span:span_test, //src/mongo/otel/traces:tracing_test (propagation and serialization); trace jstests under jstests/noPassthrough/observability/.
      • Measuring: crud_bm BM_FIND_ONE/BM_INSERT_ONE with the tracing fixtures from SERVER-137030 (prototype modes kTraced = SDK recorder, kTracedLean = lightweight recorder). Compare within one binary: code layout alone moves baselines a few % between builds. For profiles on ARM64, build with --copt=-fno-optimize-sibling-calls --copt=-fno-omit-frame-pointer and use perf record --call-graph fp; callgrind's call graph is unreliable on optimized builds because of identical code folding.

      Acceptance criteria

      • Exported trace data is unchanged with the lightweight recorder (conformance test; trace jstests).
      • crud_bm traced find/insert overhead is reduced as below, measured in one binary.
      • The SDK recorder remains available behind the startup parameter.
      • Shutdown with the lightweight recorder is clean: the existing tracer_provider_service_test shutdown tests, run with it.

      Risks and notes

      • Feature parity: the lightweight recorder covers what server code uses (attributes, status, events, links, rename, propagation). If server code starts using SDK features such as additional span processors' OnStart hooks, it needs to keep up. The conformance test is the guard.
      • Lifetime: spans hold a raw tracer pointer; check the shutdown path while operations are in flight.
      • Out of scope: storing the telemetry context inline on the opCtx instead of allocating it per traced request. It saves ~200 cycles (~0.1%) per traced request, but would mean handing out non-owning pointers to opCtx-owned storage.

       

      ------------

       

      Part 3: lazy span recording (decide after the operation, not before)

      Problem: even with Parts 1 and 2, recording a span costs ~5 us per traced command on crud_bm (find +11%, insert +10%), so recording every command (100% capture, exporting only the interesting ones) would cost far more than SPM-3868's 2% CPU budget. The cost is building spans as OpenTelemetry objects while the operation runs (span and context objects, shared pointers, attribute strings, locks), not exporting them.

      Design: record spans as plain values and build OpenTelemetry spans only for traces that are kept.

      1. Span::_start writes a small record (name, start time, parent index, kind) into a per-operation trace (detail::LazyTrace, reused from a per-thread cache, so a dropped trace allocates nothing). setAttribute/setStatus write into the record; ending a span stores its end time.
      2. When the operation's root span ends, triggers decide: keep if the remote parent was sampled (-01), the operation failed, or it ran longer than a threshold; otherwise drop.
      3. A kept trace is turned into OpenTelemetry spans through Part 2's LeanTracerProvider, with the recorded start/end times and parent links, and exported through the usual batch processor (and SERVER-137203's deferred payloads). A dropped trace is reset and returned to the cache.
        Instrumentation call sites don't change: they use our Span wrapper, never the OTel API.

      Prototype results (crud_bm, one binary, vs untraced; modes kLazyCapture = every trace dropped, kLazyKeepAll = every trace kept):

        find, 1 thread find, 16 threads insert, 1 thread insert, 16 threads
      Lazy, all dropped (0% export) +0.7% +1.2% -1.4% (noise) +0.3%
      Lazy, all kept and exported +8.4% +8.5% +3.5% +5.9%
      Eager (Parts 1-2 + SERVER-137203) +11.6% +19.8% +7.7% +12.2%

      Capture costs ~0.4 us or less per command; keeping a trace costs ~2-4 us on the request thread plus ~0.9 us on the export thread.

      Productionization:

      1. Identity and propagation: generate span ids when a span is propagated (egress) or kept; build the TelemetryContext sent to shards from the lazy record. Today egress spans on network threads (AsyncDBClient::startEgressSpan) have no opCtx and fall back to eager recording.
      2. Lifetime and threads: attach the trace to the telemetry context rather than only the opCtx, so spans that outlive the operation or cross threads (executors, futures, SERVER-131480) stay valid; protect it with the existing SpinLock.
      3. Limits: replace the fixed 16-span cap with growth plus a hard cap, and record how many spans were dropped as an attribute.
      4. Attributes: keys get a static-lifetime type (like SpanName); add a lazy attribute form (a callback evaluated only if the trace is kept) for expensive values such as query shape or plan summary.
      5. Triggers: server parameters for the duration threshold (consider defaulting to slowms), error codes, and a random baseline ratio; TracingSampler's head rules and rate limits apply to kept traces, i.e. they limit export, not capture. Keep the openTelemetryExternalTracing.enabled kill switch.
      6. Correctness: a conformance test that the exported data of a lazily recorded trace equals the eagerly recorded one (ids aside); refresh the steady-to-system clock offset periodically; support span events and links if server code uses them.
      7. Optional: move building kept spans from the request thread to the export thread by handing it the record.
      8. Observability: serverStatus counters for traces captured, kept (by trigger), dropped and overflowed.
      9. Tests and perf gates: lazy variants of span_test, a TSAN test, trace jstests in lazy mode, and a SERVER-137030 gate that kLazyCapture stays within noise of untraced.
      10. Rollout: behind a feature flag, default off; a sys-perf run like R7 with lazy recording on.

      Out of scope (follow-ups):

      • Cross-node retention, with mongos as the source of truth: mongod keeps its pieces of every trace in a retention buffer, indexed by trace id, under a lease equal to mongos's retention window; mongos sends "keep trace X" to the shards it called when it keeps a trace; pieces expire with the lease. Needs a design review (keep message transport, lease parameter, memory accounting, behavior when the buffer fills before the lease ends). Lazy records are ~140 bytes per span, against 577 for SERVER-137203's records.
      • Hang and timeout dumps: build the in-progress trees of running operations on demand (e.g. from the hang analyzer or a timeout), which neither tail spans from CurOp nor the SDK can do.

      Where to start: Span::_startLazy/_endLazy and detail::LazyTrace in src/mongo/otel/traces/span/span.cpp; benchmark modes kLazyCapture/kLazyKeepAll in benchmark_tracing.{h,cpp}; fixtures in src/mongo/db/crud_bm.cpp. Prototype limitations: no span ids until keep, attribute keys assumed static, one thread per trace, 16-span cap, no retention.

            Assignee:
            Calvin Nguyen
            Reporter:
            Calvin Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: