• Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DB Integration & Observability
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      When a traced request records a span, the OTLP exporter's representation of that span (a protobuf message) is built field by field on the operation's thread, even though the span is serialized and sent later on a background thread. Build a small plain-value record on the operation thread instead, and convert it to the protobuf on the background export thread. This removes ~1.7% of a request's CPU from the request path of every traced request, toward SPM-3868's goal of <10% added latency on traced requests.

      Background: how span export works today

      • Server code creates spans through our wrapper mongo::otel::traces::Span (src/mongo/otel/traces/span/span.cpp), on the operation's thread.
      • The export pipeline is built at startup (src/mongo/otel/traces/trace_initialization.cpp) by createHttpTracerProviderService() or createFileTracerProviderService() in src/mongo/otel/traces/tracer_provider_service_factory.cpp: an OTLP exporter (OtlpHttpExporter or OtlpFileExporter) wrapped in a BatchSpanProcessor, given to the SDK's TracerProvider.
      • In the OpenTelemetry C++ SDK, a span's data lives in a Recordable that the exporter creates:
        1. When a span starts, the span processor calls exporter->MakeRecordable() and the span writes its fields (name, ids, kind, times, attributes, status, ...) into it as it runs.
        2. When the span ends, the recordable goes to BatchSpanProcessor::OnEnd(), which queues it.
        3. The batch processor's background thread calls exporter->Export(batch), which serializes and sends.
      • For OTLP exporters the recordable is OtlpRecordable, which is a protobuf opentelemetry::proto::trace::v1::Span. Steps 1-2, including string copies and an attribute message per attribute, happen on the operation thread.

      Problem

      On crud_bm BM_FIND_ONE with every request traced, filling the OTLP protobuf on the operation thread costs ~1.7% of an untraced request's CPU (measured with perf). Every traced request pays it, even with background export.

      Proposed design

      A SpanExporter wrapper, DeferredRecordingExporter, around any SDK exporter:

      • MakeRecordable() returns a DeferredRecordable: a small struct of plain values (span context, parent span id, flags, kind, name, start time, duration, status, pointers to the resource and instrumentation scope, attributes with up to 4 stored inline, events, links).
      • Export(batch), which the BatchSpanProcessor calls only from its background thread, replays each record into a recordable from the wrapped exporter (inner->MakeRecordable(), setting fields in the order the SDK's span does) and passes those to inner->Export().
      • ForceFlush()/Shutdown() delegate to the wrapped exporter.
      • It works with any exporter (HTTP, file, and gRPC once SERVER-127722 lands) and with either span recorder (the SDK's, or the lightweight one from SERVER-137201): both obtain recordables through MakeRecordable().

      Where to start

      • Prototype: src/mongo/otel/traces/deferred_recording_exporter.{h,cpp} and deferred_recording_exporter_test.cpp on <PROTOTYPE_BRANCH>. It is complete and tested; what remains is production wiring and the items below.
      • Wiring: in tracer_provider_service_factory.cpp, wrap the exporter in both create*TracerProviderService() functions before BatchSpanProcessorFactory::Create(). Consider a startup parameter to disable it as a fallback.
      • Build: give it its own mongo_cc_library in src/mongo/otel/traces/BUILD.bazel (in the prototype it shares the lean_tracer target), with deps on //src/third_party/opentelemetry-cpp/api and .../sdk/src/trace; tracer_provider_service_factory depends on it.
      • Conformance test: the prototype test records the same representative span (every attribute type including arrays, events, status, rename) through a capturing exporter directly and through DeferredRecordingExporter, then compares the resulting SpanData. Change it to produce spans with the SDK's TracerProvider (TracerProviderFactory::Create(SimpleSpanProcessorFactory::Create(...))) instead of LeanTracerProvider, so this ticket doesn't depend on SERVER-137201.
      • Measuring: crud_bm with the tracing fixtures from SERVER-137030 (prototype modes kTracedBatch and kTracedLeanDeferred). Compare modes within one binary; code layout alone moves baselines a few % between builds. Build with bazel build -config=opt //src/mongo/db:crud_bm, run with -benchmark_filter='^CrudTraced.*BM_FIND_ONE/threads:1$' --benchmark_repetitions=5.

      Acceptance criteria

      • The HTTP and file exporters are wrapped; exported data is unchanged (conformance test; existing trace jstests, e.g. jstests/noPassthrough/observability/otel_*span*export.js and otel_traces_export_headers.js).
      • Traced crud_bm find/insert with background export: request-thread CPU drops by about the protobuf-fill cost. Report export-thread CPU per span too, which goes up slightly.
      • No spans dropped at the default batch settings (queue 2048, batch 512, 1 s).

      Risks and notes

      • The record keeps raw pointers to the provider's resource and instrumentation scope. They must outlive export. The batch processor flushes on shutdown before the provider is destroyed, but check TracerProviderService::shutdown().
      • Array attributes need temporary storage while replaying (std::vector<bool> has no contiguous storage; string arrays need string_view arrays). The prototype's AttributeViews handles both.
      • Records wait in the batch queue (up to 2048), so their size matters for memory; they are smaller than the protobufs they replace.

            Assignee:
            Unassigned
            Reporter:
            Calvin Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: