ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DB Integration & Observability
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Goal

      Make the decision "should this request be traced?" cheap: free when nothing can be sampled, and near-free at the default sampling rate, where nearly every request is still rejected. The scope doc's target is under 2% CPU at the default level.

      Changes

      1. Fast sampling roll (land first)

      Every sampleable root span draws a roll with PseudoRandom::nextCanonicalDouble(), i.e. libstdc++'s std::generate_canonical, which computes in long double with two logl calls per call. ARM64 has no hardware long double, so each roll ran software quad-precision arithmetic: about 6,600 instructions, nearly half of the default-sampling overhead.

      • Draw the roll as the top 53 bits of one 64-bit draw, scaled: (uint64_t(prng.nextInt64()) >> 11) * 0x1.0p-53 (SpanTelemetryContextImpl::drawSamplingRoll).

      2. Skip all sampling work when nothing can be sampled

      • TracingSampler::mightSampleAny(), a single relaxed atomic load, lets root spans exit before looking up the tracer provider, feature flags or sampler.
      • The sampling roll is drawn lazily.

      3. Decide before allocating

      At a non-zero factor, root spans allocated a SpanTelemetryContextImpl before the sampler rejected them, and stored it on the opCtx, putting the rest of the operation on the slower has-context path.

      • Draw the roll first; allocate a context only if the request is sampled.
      • The roll is stored per operation (TelemetryContextHolder::samplingRoll()), keeping one roll per operation (e.g. reshardCollection has two sampleable spans).
      • New Span::startIngressSpan(OperationContext*, ...), used by service_entry_point_shard_role.cpp and s/commands/strategy.cpp.

      4. One lookup per decision

      canSample and shouldSample each refreshed the config snapshot and hashed the span name; an admitted span then did a second versioned-map lookup for its admission::RateLimiter.

      • TracingSampler::sample(spanName, function_ref<double()> drawRoll): one lookup; the roll is drawn only for eligible spans. The base class implements it with canSample + shouldSample, so test samplers keep working.

      5. Lock-free token bucket

      • SamplingTokenBucket: generic cell rate algorithm, one atomic "theoretical arrival time", one CAS per admission. Stored in the span's config entry (ResolvedSampling), so no second lookup. Replaces RateLimiterMap/_getOrCreateRateLimiter for internal spans; the external-trace limiter is unchanged.
      • Buckets are carried over by span name across reconfiguration, keeping available tokens clamped to the new burst (as folly's TokenBucket::reset does). Admission metrics are still recorded.
      • Behavior change: buckets are created with the config, one per sampleable span name, not lazily.

      Results

      • Untraced find/insert (crud_bm, callgrind): 246/251 -> 75/71 tracing instructions per request, the same as tracing off.
      • Default sampling (callgrind): 870 -> 542 tracing instructions and 1 -> 0 allocations per request, from (3).
      • Default sampling (native): find +7.3% -> +0.8%, insert +6.1% -> +1.1%.
      • sys-perf at the documented default (0.045%), primary CPU spent in tracing: 2.95% > 0.81% (ycsb in_cache), 1.84-1.94% -> 0.59-0.64% (sharded rate-limited), from (1)(3).
      • Sampler cost per traced request (profiling build): ~3,000 -> ~1,200 cycles.

      Tests

      • span_telemetry_context_impl_test: roll formula, one draw per roll, range.
      • span_test: a rejected root span creates no context; one roll per operation; a later sampled span keeps it; the opCtx ingress overload.
      • sampler_test: mightSampleAny; sample() equivalence and lazy roll; base-class default; SamplingTokenBucket burst/refill, reconfiguration, concurrency. Existing stats and metrics tests unchanged.
      • Trace jstests.

            Assignee:
            Calvin Nguyen
            Reporter:
            Calvin Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: