Deterministic locality-aware mongos selection for multi-region sharded clusters

XMLWordPrintableJSON

    • Type: Epic
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Component/s: SDAM, Server Selection
    • Locality-Aware Mongos Selection
    • None
    • Hide

      Summary of necessary driver changes

      •  

      Commits for syncing spec/prose tests
      (and/or refer to an existing language POC if needed)

      •  

      Context for other referenced/linked tickets

      •  
      Show
      Summary of necessary driver changes   Commits for syncing spec/prose tests (and/or refer to an existing language POC if needed)   Context for other referenced/linked tickets  
    • In Progress
    • 0
    • 0
    • 0
    • 100
    • None
    • None
    • None
    • Needed
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      Drivers have no supported way to deterministically prefer mongos endpoints in a designated region during normal operations while automatically and transparently failing over to another region's mongos endpoints when the preferred region becomes unavailable; without any connection string change, application restart, or intervention at the moment of failover.

      localThresholdMS is the closest existing mechanism, but it is probabilistic: the driver builds a latency window around the fastest host and randomly load-balances across every candidate inside that window, so a non-preferred-region mongos can still be selected on any given operation. Read preference `tagSets` don't help either, since they govern replica set member selection and have no effect on which mongos a driver picks in a sharded topology.

      We are trying to achieve deterministic, driver-level regional affinity for mongos selection: 100% of traffic to the preferred region while it's healthy, automatic tiered fallback when it isn't, and automatic recovery once it's healthy again - with zero manual or automated intervention required at the moment of failover.

      Motivation

      Who is the affected end user?

      Operators of multi-region sharded clusters with an active/DR (or multi-cloud, multi-region) topology - both self-hosted Enterprise Advanced deployments and Atlas. The pattern is common in regulated industries (e.g. financial services) running mission-critical, latency-sensitive workloads across two or more datacenters/regions.

      How does this affect the end user?

      Affected users face a forced choice between two unacceptable options: (1) include all-region mongos endpoints and accept non-deterministic cross-region routing and latency variance in steady state, or (2) exclude non-preferred-region endpoints and accept a connection-string change as an application-impacting step at DR time. Neither preserves an expectation of seamless, zero-downtime failover for sharded clusters. This is more than annoyance - for latency-sensitive, mission-critical workloads it undermines a core availability commitment made to the business running on top of the cluster.

      How likely is it that this problem or use case will occur?

      Main path. Any sharded cluster deployed across more than one region/datacenter with multiple mongos instances is exposed to this by default - cross-region routing is the expected outcome of current latency-window selection whenever two regions fall within (or near) the same latency window, not a rare misconfiguration.

      If the problem does occur, what are the consequences and how severe are they?

      Primarily a performance concern in steady state (added latency, latency variance, and in multi-cloud cases, cross-cloud egress cost) rather than an outage - operations still complete. However, for workloads with strict latency SLAs (e.g. payments processing), sustained cross-region routing can translate into SLA violations. Separately, the DR-time workaround (connection string change) introduces a real availability regression risk: if that step is missed, delayed, or mishandled, failover can be materially slower or require an application restart, which is the specific outcome mission-critical users are trying to avoid entirely.

      Is this issue urgent?

      Yes. This is a re-escalation of a previously closed request (DRIVERS-3191, Won't Do, P2), now raised again due to renewed field pressure from enterprise accounts and at least one case where latency-sensitive workloads are scheduled to move onto affected clusters in the near term. There is no hard contractual deadline captured yet, but the business context has materially increased priority relative to the original closure.

      Is this ticket required by a downstream team?

      Yes, confirmed - this is a hard blocking dependency, not a coordination nicety. The tiered selection logic depends on a per-host (or per-cluster) DNS TXT record encoding locality metadata. For Atlas clusters, that record does not exist today and must be created and kept in sync by the Cloud/DBaaS team as clusters are provisioned or topology changes - this is new infrastructure work, not a config change. This ticket is currently Blocked specifically on securing that commitment from Cloud; Drivers-side design and implementation cannot deliver a usable Atlas experience without it. For self-hosted (EA) deployments, no MongoDB-side engineering work is required - the same TXT record format can be published directly by the operator in their own DNS zone, with the one additional prerequisite that the deployment must use mongodb+srv:// discovery, since per-host TXT lookup is confirmed to only occur as part of SRV-based seedlist discovery today.

      Design should treat self-hosted (EA) as a first-class path, not an Atlas-derivative afterthought - the mechanism, prerequisites, and constraints for self-hosted operators (SRV adoption, manual TXT record publication) should be documented with the same rigor as the Atlas path, given active field demand from self-hosted enterprise accounts.

      Is this ticket only for tests?

      No. This has direct functional impact on driver server-selection (SDAM) behavior — new client-configurable options and new tiered mongos selection logic are required, not just test coverage improvements.

      Dependency to resolve during design
      The proposed spec's per-host locality (DNS TXT) lookup is confirmed to only occur today as part of mongodb+srv:// seedlist discovery - there is no existing per-host TXT lookup path for hosts supplied via a plain mongodb:// connection string. Since direct/seed-list connection strings are a common pattern for self-hosted deployments, design must explicitly resolve whether this lookup can be generalized to non-SRV connections, or whether SRV adoption becomes a documented hard prerequisite for using this feature. This should not be left ambiguous.

      Acceptance Criteria

      • Determinism is defined and testable. The design specifies exactly how 100% of traffic is routed to preferred-region mongos instances while healthy, distinguishing this precisely from the probabilistic bias localThresholdMS already provides.
      • Failover and recovery behavior is fully specified. The design describes how the driver falls through to a non-preferred region when the preferred region is unreachable, and how it resumes preferring that region once healthy again -both without application code changes, connection string changes, or restarts.
      • Locality metadata is static by design. The design confirms that failover is driven by existing SDAM health/heartbeat monitoring, not by creating, updating, or removing locality metadata (e.g. DNS records) at the moment of a failover event.
      • The SRV dependency is explicitly resolved. The design states plainly whether direct (non-SRV) connection strings are supported, and if not, documents SRV adoption as a required prerequisite rather than leaving it implicit.
      • Atlas / self-hosted parity is addressed. The design confirms whether any mechanism required on the Atlas side (e.g. published DNS records) can be equivalently produced by a self-hosted operator with no dependency on Atlas-managed infrastructure or mongod/mongos colocation.
      • Scope boundary with read preference is explicit. The design confirms this feature is limited to mongos selection in sharded topologies and does not alter read preference tagSet or replica set member selection behavior.
      • Backward-compatible degradation is defined. The design specifies how a mongos host with no locality metadata is treated (fallback candidate, not an error), preserving compatibility with existing deployments that don't opt in.
      • Cross-driver consistency plan exists. The design defines a single spec-level behavior and option naming convention intended for uniform implementation across all official drivers, rather than a driver-specific mechanism.
      • Observability is addressed. The design specifies what, if anything, drivers surface (logs/events) when active mongos selection shifts between preferred and fallback regions.
      • Cloud/DBaaS commitment secured for Atlas. Design is not complete until the Cloud team has explicitly committed to implementing and owning ongoing publication of the locality TXT record(s) for Atlas clusters. Without this, the feature cannot function for Atlas customers regardless of driver-side readiness, and the ticket should remain Blocked until this commitment is on record.
      • API shape reflects tagSet-like configuration. Per design discussion, the client option should be evaluated as a list of mappings (e.g. [{"AWS": ["us-east-1", "us-central-1"]}, {"GCP": [...]}]) rather than a single localCloud/localRegion pair, both for consistency with existing tagSet configuration patterns and to support customers with more than one acceptable preferred region.
      • DNS call volume is bounded for large clusters. Design should avoid a per-host TXT lookup for every host in the seedlist; a single cluster-level TXT record encoding locality metadata for all hosts is preferred to keep DNS resolution overhead low for large multi-region deployments (100+ node clusters).
      • EA is treated as a first-class design path, not an Atlas afterthought. The design doc documents the self-hosted mechanism, prerequisites (SRV adoption, manual TXT record publication), and constraints with the same rigor as the Atlas path, reflecting active field demand from self-hosted enterprise accounts.

            Assignee:
            Unassigned
            Reporter:
            Rishit Bhatia
            None
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              None
              None
              None
              None
              None