-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
DB Integration & Observability
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Summary
When a traced request records a span, the OTLP exporter's representation of that span (a protobuf message) is built field by field on the operation's thread, even though the span is serialized and sent later on a background thread. Build a small plain-value record on the operation thread instead, and convert it to the protobuf on the background export thread. This removes ~1.7% of a request's CPU from the request path of every traced request, toward SPM-3868's goal of <10% added latency on traced requests.
Background: how span export works today
- Server code creates spans through our wrapper mongo::otel::traces::Span (src/mongo/otel/traces/span/span.cpp), on the operation's thread.
- The export pipeline is built at startup (src/mongo/otel/traces/trace_initialization.cpp) by createHttpTracerProviderService() or createFileTracerProviderService() in src/mongo/otel/traces/tracer_provider_service_factory.cpp: an OTLP exporter (OtlpHttpExporter or OtlpFileExporter) wrapped in a BatchSpanProcessor, given to the SDK's TracerProvider.
- In the OpenTelemetry C++ SDK, a span's data lives in a Recordable that the exporter creates:
- When a span starts, the span processor calls exporter->MakeRecordable() and the span writes its fields (name, ids, kind, times, attributes, status, ...) into it as it runs.
- When the span ends, the recordable goes to BatchSpanProcessor::OnEnd(), which queues it.
- The batch processor's background thread calls exporter->Export(batch), which serializes and sends.
- For OTLP exporters the recordable is OtlpRecordable, which is a protobuf opentelemetry::proto::trace::v1::Span. Steps 1-2, including string copies and an attribute message per attribute, happen on the operation thread.
Problem
On crud_bm BM_FIND_ONE with every request traced, filling the OTLP protobuf on the operation thread costs ~1.7% of an untraced request's CPU (measured with perf). Every traced request pays it, even with background export.
Proposed design
A SpanExporter wrapper, DeferredRecordingExporter, around any SDK exporter:
- MakeRecordable() returns a DeferredRecordable: a small struct of plain values (span context, parent span id, flags, kind, name, start time, duration, status, pointers to the resource and instrumentation scope, attributes with up to 4 stored inline, events, links).
- Export(batch), which the BatchSpanProcessor calls only from its background thread, replays each record into a recordable from the wrapped exporter (inner->MakeRecordable(), setting fields in the order the SDK's span does) and passes those to inner->Export().
- ForceFlush()/Shutdown() delegate to the wrapped exporter.
- It works with any exporter (HTTP, file, and gRPC once SERVER-127722 lands) and with either span recorder (the SDK's, or the lightweight one from SERVER-137201): both obtain recordables through MakeRecordable().
Where to start
- Prototype: src/mongo/otel/traces/deferred_recording_exporter.{h,cpp} and deferred_recording_exporter_test.cpp on <PROTOTYPE_BRANCH>. It is complete and tested; what remains is production wiring and the items below.
- Wiring: in tracer_provider_service_factory.cpp, wrap the exporter in both create*TracerProviderService() functions before BatchSpanProcessorFactory::Create(). Consider a startup parameter to disable it as a fallback.
- Build: give it its own mongo_cc_library in src/mongo/otel/traces/BUILD.bazel (in the prototype it shares the lean_tracer target), with deps on //src/third_party/opentelemetry-cpp/api and .../sdk/src/trace; tracer_provider_service_factory depends on it.
- Conformance test: the prototype test records the same representative span (every attribute type including arrays, events, status, rename) through a capturing exporter directly and through DeferredRecordingExporter, then compares the resulting SpanData. Change it to produce spans with the SDK's TracerProvider (TracerProviderFactory::Create(SimpleSpanProcessorFactory::Create(...))) instead of LeanTracerProvider, so this ticket doesn't depend on SERVER-137201.
- Measuring: crud_bm with the tracing fixtures from
SERVER-137030(prototype modes kTracedBatch and kTracedLeanDeferred). Compare modes within one binary; code layout alone moves baselines a few % between builds. Build with bazel build -config=opt //src/mongo/db:crud_bm, run with -benchmark_filter='^CrudTraced.*BM_FIND_ONE/threads:1$' --benchmark_repetitions=5.
Acceptance criteria
- The HTTP and file exporters are wrapped; exported data is unchanged (conformance test; existing trace jstests, e.g. jstests/noPassthrough/observability/otel_*span*export.js and otel_traces_export_headers.js).
- Traced crud_bm find/insert with background export: request-thread CPU drops by about the protobuf-fill cost. Report export-thread CPU per span too, which goes up slightly.
- No spans dropped at the default batch settings (queue 2048, batch 512, 1 s).
Risks and notes
- The record keeps raw pointers to the provider's resource and instrumentation scope. They must outlive export. The batch processor flushes on shutdown before the provider is destroyed, but check TracerProviderService::shutdown().
- Array attributes need temporary storage while replaying (std::vector<bool> has no contiguous storage; string arrays need string_view arrays). The prototype's AttributeViews handles both.
- Records wait in the batch queue (up to 2048), so their size matters for memory; they are smaller than the protobufs they replace.
- related to
-
SERVER-137029 Minimize the cost of trace sampling decisions
-
- Investigating
-