-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Optimization
-
Fully Compatible
-
ALL
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Disagg tasks intermittently system-fail in the "fetch module images" setup step (before any test runs) when a transient network error hits one of the docker pull calls for the SLS service images. Add a bounded retry loop with backoff around docker pull in src/mongo/db/modules/atlas/evergreen/fetch_images.sh.
Recurring flake tracked by BF-45594 (Build Baron auto-resolution rule matches 'fetch module images', no_failing_tests: true).
Failure Example
Task: ...tsan_all_feature_flags_sharded_clusters_disagg_change_streams_sharding_fsm_batch_size_1_embedded_watch_collection_3 (2026-08-26):
+ docker pull 664315256653.dkr.ecr.us-east-1.amazonaws.com/disagg-storage/scheduler:0bacaee117b4b5674a463a31a481d11c8b990a58 Error response from daemon: Head "https://664315256653.dkr.ecr.us-east-1.amazonaws.com/v2/disagg-storage/scheduler/manifests/0bacaee...": dial tcp 52.44.174.17:443: i/o timeout Command 'subprocess.exec' ('fetch module images') in function 'run generated tests' (step 3.23 of 3) failed: process encountered problem: exit code 1.
Root Cause
- Every disagg task pulls 5 images (log, page, cellmetadata, scheduler, pagematerializer) fresh from ECR on ephemeral hosts, so each task has a multi-minute window of network exposure.
- Docker automatically retries layer downloads (visible earlier in the same log: 4e09a218a714: Retrying in 5 seconds), but a transient error on the initial manifest request fails the whole docker pull immediately with exit 1, and set -e aborts the script.
- The docker pull loop is the last network operation in this script without retry protection — the curl downloads in the same file were already hardened for the identical flake class (SERVER-125942,
SERVER-132843).
- is related to
-
SERVER-132843 Retry module image dependency downloads after curl connection resets
-
- Closed
-