Disagg SLS setup flakes with init-cloud-resources exit 1 (retry budget exhausted under host load); compose up failures are undiagnosable

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Multiple no_passthrough_disagg_override_replsets tasks failed at commit a9ff3c5a (2026-08-18) because the SLS multi-cell fixture could not be set up. None of the affected tests ran any test logic:

      Container ...-init-cloud-resources-1 Error service "init-cloud-resources" didn't complete successfully: exit 1
      uncaught exception: Error: docker-compose up failed with exit code: 1
      failed to load: jstests/... -> exiting with code -3
      

      Affected (same build, 5 tests across 3 tasks):

      • disagg_override_replsets_1: jstests/noPassthrough/query/cbr/ndv_estimate_multikey_race.js
      • disagg_override_replsets_5: jstests/noPassthrough/query/unionWith_facet_view_stage_params.js
      • disagg_override_replsets_7 (jobs 0, 3, 5 — same host, within ~90s): internal_query_max_memory_per_operation_default.js, out_merge_on_secondary_batch_write_sharded_merge_keep_existing.js, command_diagnostics_subpipeline.js

      This is a recurrence of the SERVER-132918 flake class. That fix wrapped the three AWS setup calls in the init-cloud-resources one-shot container (create-bucket, create_sqs_queue, update_s3_notification_config) in a retry() helper, because localstack's plain-HTTP healthcheck reports healthy before its HTTPS listener is usable. The fix was present at the failing commit, but:

      • The container now exits 1 (not 101), which is what retry() returns after exhausting its budget of 10 attempts x 2s (~20s).
      • Three of the five failures were concurrent jobs on the same host, i.e. several complete SLS stacks (60+ containers) booting at once. Under that load, localstack takes well over 20s to serve HTTPS, so the retry budget is exhausted.
      • Crucially, the actual reason for the exit is unknowable from the logs: DockerComposeProject.up() (src/mongo/db/modules/atlas/jstests/disagg_storage/libs/docker_compose.js) only starts the streaming log follower after docker-compose up -d succeeds. When up fails, the one-shot container's stderr (the retry attempt messages, or any other failing command under set -e) is silently discarded. All five logs contain nothing but the exit code.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: