-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
DB Integration & Observability
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Goal
Make the decision "should this request be traced?" cheap: free when nothing can be sampled, and near-free at the default sampling rate, where nearly every request is still rejected. The scope doc's target is under 2% CPU at the default level.
Changes
1. Fast sampling roll (land first)
Every sampleable root span draws a roll with PseudoRandom::nextCanonicalDouble(), i.e. libstdc++'s std::generate_canonical, which computes in long double with two logl calls per call. ARM64 has no hardware long double, so each roll ran software quad-precision arithmetic: about 6,600 instructions, nearly half of the default-sampling overhead.
- Draw the roll as the top 53 bits of one 64-bit draw, scaled: (uint64_t(prng.nextInt64()) >> 11) * 0x1.0p-53 (SpanTelemetryContextImpl::drawSamplingRoll).
2. Skip all sampling work when nothing can be sampled
- TracingSampler::mightSampleAny(), a single relaxed atomic load, lets root spans exit before looking up the tracer provider, feature flags or sampler.
- The sampling roll is drawn lazily.
3. Decide before allocating
At a non-zero factor, root spans allocated a SpanTelemetryContextImpl before the sampler rejected them, and stored it on the opCtx, putting the rest of the operation on the slower has-context path.
- Draw the roll first; allocate a context only if the request is sampled.
- The roll is stored per operation (TelemetryContextHolder::samplingRoll()), keeping one roll per operation (e.g. reshardCollection has two sampleable spans).
- New Span::startIngressSpan(OperationContext*, ...), used by service_entry_point_shard_role.cpp and s/commands/strategy.cpp.
4. One lookup per decision
canSample and shouldSample each refreshed the config snapshot and hashed the span name; an admitted span then did a second versioned-map lookup for its admission::RateLimiter.
- TracingSampler::sample(spanName, function_ref<double()> drawRoll): one lookup; the roll is drawn only for eligible spans. The base class implements it with canSample + shouldSample, so test samplers keep working.
5. Lock-free token bucket
- SamplingTokenBucket: generic cell rate algorithm, one atomic "theoretical arrival time", one CAS per admission. Stored in the span's config entry (ResolvedSampling), so no second lookup. Replaces RateLimiterMap/_getOrCreateRateLimiter for internal spans; the external-trace limiter is unchanged.
- Buckets are carried over by span name across reconfiguration, keeping available tokens clamped to the new burst (as folly's TokenBucket::reset does). Admission metrics are still recorded.
- Behavior change: buckets are created with the config, one per sampleable span name, not lazily.
Results
- Untraced find/insert (crud_bm, callgrind): 246/251 -> 75/71 tracing instructions per request, the same as tracing off.
- Default sampling (callgrind): 870 -> 542 tracing instructions and 1 -> 0 allocations per request, from (3).
- Default sampling (native): find +7.3% -> +0.8%, insert +6.1% -> +1.1%.
- sys-perf at the documented default (0.045%), primary CPU spent in tracing: 2.95%
> 0.81%(ycsb in_cache), 1.84-1.94% -> 0.59-0.64% (sharded rate-limited), from (1)(3). - Sampler cost per traced request (profiling build): ~3,000 -> ~1,200 cycles.
Tests
- span_telemetry_context_impl_test: roll formula, one draw per roll, range.
- span_test: a rejected root span creates no context; one roll per operation; a later sampled span keeps it; the opCtx ingress overload.
- sampler_test: mightSampleAny; sample() equivalence and lazy roll; base-class default; SamplingTokenBucket burst/refill, reconfiguration, concurrency. Existing stats and metrics tests unchanged.
- Trace jstests.
- is related to
-
SERVER-137201 Switch from using SDK-provided Spans to using a lightweight implementation
-
- Investigating
-
-
SERVER-137203 Build span payloads on the export thread instead of the operation thread
-
- Needs Scheduling
-