-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
DB Integration & Observability
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Right now, commands with trace info are subject to sampling rates and rate limits. We should consider how to ensure that commands with trace info are always traced by default. Also add the ability to disable it as an emergency.
Implemented on calvin.nguyen/otel-trace-perf (commit 9c616f6d1fe)
- An incoming trace is continued only if its traceparent is marked sampled (-01). Drivers propagate a traceparent whether or not they are tracing, so continuing every incoming trace made every request pay for tracing.
- New openTelemetryExternalTracing.enabled parameter: the emergency off switch this ticket asks for.
- canSample is checked before any per-request allocation.
Results
- BM_SEP_PING with an exporter configured: +25% -> +2.7% (1 thread) / +0.6% (16 threads).
- HVW with the flag on and nothing sampled (perf-required + perf-required-shard): within noise.
Remaining
- Decide whether sampled incoming traces should bypass the external rate limit by default, per this ticket's ask.