Summary

The trace-context propagation spec (mongodb/specifications#1966; OP_MSG
telemetry section, MongoDB 9.0+ / maxWireVersion >= 29) requires a prose test
that verifies server-created spans: correct parent linkage to the driver's
span, one span per retry attempt, and none for auth/monitoring commands.
Server spans are only observable when mongod's OpenTelemetry file exporter is
configured at startup — there is no wire protocol or command to fetch them —
and driver CI orchestration does not currently provision this, so no driver
can implement the prose test.

This ticket adds an opt-in --otel / OTEL=1 configuration to
drivers-evergreen-tools orchestration that starts every mongod/mongos with the
required setParameters (opentelemetryTraceDirectory per member,
featureFlagOtelTraceSampling, raised sampling and external-tracing token
buckets for deterministic assertions), exports the trace directory to driver
test suites as OTEL_TRACE_DIR via the existing mo-expansion mechanism, and
fails fast on incompatible setups (server < 9.0, DOCKER_RUNNING,
-local-atlas, -mongodb-runner).

Motivation

Driver engineers implementing the trace-context propagation spec — every
driver team that must ship the spec's prose test. The immediate consumer is
the Java driver reference implementation (mongo-java-driver#2022).

Who are the stakeholders?

The DBX drivers teams (spec authors and implementers of
mongodb/specifications#1966), drivers-evergreen-tools maintainers, and the
server observability team whose driver→server trace propagation feature
(MongoDB 9.0+) this test coverage validates.

Are they blocked? Are they annoyed? Are they confused?

Blocked. Without orchestration provisioning the OTel exporter, the prose test
cannot be run in any driver's CI: server spans are unobservable, so the
spec's review requirement (verify parent linkage, per-retry spans, exclusion
of auth/monitoring commands) cannot be met. Each driver team would otherwise
have to hand-roll its own server-launch configuration, duplicating effort and
diverging from the shared orchestration path.

Main path? Edge case?

Main path for the spec rollout: every driver implementing
mongodb/specifications#1966 hits this prerequisite. It is opt-in, so it does
not affect any existing CI variant that does not enable it.

Minor annoyance at a log message? Performance concern? Outage/unavailability? Failover can't complete?

No outage or production impact — this is CI test infrastructure. The
consequence of not doing it is a permanent test-coverage gap: driver→server
span linkage would ship unverified across all drivers, and regressions in
trace-context propagation (wrong parentage, spans for monitoring/auth
commands, missing per-retry spans) would go undetected. A secondary
consequence addressed by the design: naive configuration (default sampling
factor 0.000045, default 1/s-10-token rate limits) makes span assertions
probabilistic, i.e. a flaky test in every driver's CI.

Does this ticket have a required timeline? What is it?

It is the prerequisite blocking the prose test in mongodb/specifications#1966,
which is in review now with a working reference implementation waiting on it
(mongo-java-driver#2022). No hard calendar deadline, but it gates merging the
spec's test requirements and should land while MongoDB 9.0 (the first server
version with these parameters) is current.

Needed by e.g. Atlas, Shell, Compass?

Yes — by driver teams (Java first, then all drivers implementing the spec).
Not needed by Atlas/Shell/Compass

Does this ticket have any functional impact, or is it just test improvements?

Test infrastructure only. It changes no shipped driver or server behavior;
it adds an opt-in flag to CI orchestration tooling. With the flag unset,
orchestration behavior is byte-identical to today.

Acceptance Criteria

• OTEL=1 MONGODB_VERSION=latest bash .evergreen/run-orchestration.sh starts
  standalone, replica set, and sharded clusters (mongos included) with the
  OTel setParameters applied — verifiable via getParameter — using the
  two-parameter shape from the server IDL:
  `openTelemetryTracingSampling` (defaultSampling samplingFactor 1.0 with a
  raised token bucket) plus `openTelemetryExternalTracing` (raised token
  bucket for externally-propagated driver traceparents).
• Each cluster member writes to its own per-port directory under
  `$DRIVERS_TOOLS/otel/`, created before server start, writable by the mongod
  user, wiped between executions.
• `OTEL_TRACE_DIR` is exported via `mo-expansion.sh`/`mo-expansion.yml`;
  drivers gate the prose test on its presence (absent everywhere the flag is
  not enabled).
• Opt-in guarantee: without `OTEL`, no otel directory, no expansion entry, and
  no behavioral change to any existing configuration.
• Fail-fast with a clear error, before any download or deployment, when
  `OTEL` is combined with server < 9.0, `DOCKER_RUNNING`, `--local-atlas`, or
  `--mongodb-runner` (same-host filesystem requirement).
• Unit tests cover parameter injection across all three topologies (including
  mongos routers), preservation of pre-existing `setParameter` keys, the
  pinned parameter values, and every fail-fast path.
• README documents usage, the directory layout, the OTel-enabled-build
  requirement (`requires_otel_build` — standard archive builds accept the
  parameters but export nothing), flush-interval polling guidance for tests,
  and an Evergreen artifact-upload snippet for capturing traces on failure.