Disagg SLS test setup flakes with init-cloud-resources exit 101 (localstack HTTPS not ready)

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Minor - P4
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Disaggregated-storage (SLS) jstests that stand up the SLS layer via Docker Compose intermittently fail during setup, before any test logic runs. The failure surfaces as:

      Container ...-init-cloud-resources-1 Error service "init-cloud-resources"
      didn't complete successfully: exit 101
      uncaught exception: Error: docker-compose up failed with exit code: 1
      failed to load: documents_out_sharded.js
      exiting with code -3
      

      Exit code 101 is a Rust panic. The init-cloud-resources container runs its provisioning script under "set -e", and its first command is the Rust /testdriver binary, so "/testdriver object-storage create-bucket" panicked and aborted the whole setup.

      Root cause. init-cloud-resources provisions the S3 bucket, SQS queue, and bucket-notification config against the localstack mock over HTTPS.

      Its start gate is the localstack container healthcheck: awslocal s3 ls && awslocal sqs list-queues which probes the plain-HTTP internal endpoint. localstack can report "healthy" before its HTTPS listener/TLS is fully ready. The testdriver's first HTTPS call then hits a transient connection/TLS error and unwrap-panics (exit 101). Because the script runs under "set -e", docker-compose up returns 1, up() throws, and the ShardingTest / DisaggReplSetTest constructor fails (exit -3).

      This is a test-harness startup race (healthcheck passes but service not fully ready), not a product bug. The /testdriver binary and its /scripts live inside the SLS image, so the panic itself is not fixable from the mongo repo.

      Sequence of events.

      • docker-compose up -d starts all SLS containers.
      • infra.localstack reports Healthy via its plain-HTTP awslocal probe.
      • init-cloud-resources starts and runs /testdriver object-storage create-bucket over HTTPS.
      • HTTPS listener/TLS is not ready yet -> transient error -> Rust panic (exit 101).
      • set -e aborts the container -> docker-compose up returns 1 -> up() throws (docker_compose.js:101) -> DisaggReplSetTest ctor fails -> test exits -3.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: