Summary

SERVER-129815 tests the encryption layer under SLS infrastructure chaos (log leader kills, network partitions between SLS services, etc.) while keeping the KMIP server reachable. This ticket covers the orthogonal scenario: the KMIP server itself becomes unreachable while mongod is running. The two fault domains are kept separate so failures can be attributed clearly — mixing SLS chaos with KMIP faults in one test makes root cause analysis very difficult.

The expected behavior under KMIP unavailability is:

• A KEK or CMK rotation that is in flight when KMIP goes down must fail gracefully: the rotation state machine marks the rotation as failure, restores the keystore snapshot, and leaves the cluster fully operational under the previously active KEK/CMK.
• Reads and writes using an already-active KEK must continue uninterrupted while KMIP is down. The active KEK is cached in memory; mongod must not attempt a KMIP call on every encryption or decryption operation.
• A mongod restart while KMIP is unreachable must succeed. mongod must be able to load its KEK from the WiredTiger keystore without contacting KMIP at startup.

This test reuses the KMIP Docker container built in SERVER-129815 (Work Item 1) but inverts the IP assignment: the KMIP container is placed below the 10.20.20.130 threshold so Antithesis injects network faults on it, while the SLS service containers are placed above the threshold to keep the storage layer healthy.

Work Items

1. Create a new docker-compose configuration for this topology (coordinate with SLS team). It differs from the SERVER-129815 topology in IP assignment only. The KMIP Dockerfile itself is unchanged from SERVER-129815 Work Item 1:
   • KMIP container: assign an IP < 10.20.20.130 so Antithesis injects network faults on it.
   • SLS service containers (logd, pagematd, CMS): assign IPs >= 10.20.20.130 so they are not faulted. The storage layer must remain healthy; its failure would confound results.
   • mongod and the workload container: no change from SERVER-129815
2. Create the template directory buildscripts/antithesis/test_composer/disagg_kmip_unavailability/. All scripts must be executable (chmod +x). Shared logic goes in a helper_*.sh file or subdirectory so Test Composer ignores it.
3. Write first_setup.sh (one-time setup, runs before any driver starts):
   • Create and activate two KMIP keys via kmip_manage_key.py (same as SERVER-129815 Work Item 3). KMIP is reachable at this point; Antithesis has not started faulting yet.
   • Wait for mongod to become primary and confirm KEK initialization completed (getESERotateActiveKEKStatus returns activeKEKId >= 1).
   • Insert a seed collection of documents for post-fault consistency checks.
4. Write parallel_driver_crud.sh (continuous CRUD workload, runs during fault injection):
   • Same as SERVER-129815Work Item 4: continuous inserts, updates, finds, and deletes with retries on NotWritablePrimary and NetworkError.
   • This driver must continue producing successful operations even while KMIP is faulted, because the active KEK is cached in memory. Sustained write failures during a KMIP fault window indicate mongod is incorrectly blocking operations on KMIP availability.
5. Write parallel_driver_kek_rotation.sh (KEK rotation driver, runs during fault injection):
   • Same loop structure as antithesis_kek.txt Work Item 5: sleep a random interval (10-40 s), attempt eseRotateActiveKEK, tolerate transient errors.
   • Unlike SERVER-129815, rotation failures are expected here: when KMIP is faulted, the rotation cannot contact KMIP to encrypt the new KEK and must fail. The driver must not assert on failure status; it logs both completed and failure outcomes and continues.
6. Write parallel_driver_cmk_rotation.sh (CMK rotation driver, runs during fault injection):
   • Same loop structure as SERVER-129815 Work Item 6.
   • Same tolerance for failure outcomes as Work Item 5 above.
7. Write eventually_check_rotation_state.sh (post-recovery check, runs after faults stop):
   • Poll getESERotateActiveKEKStatus until status is not pending (timeout 60 s). Both completed and failure are acceptable outcomes here, since the rotation may have been in flight when KMIP was faulted. Assert only that the status is not still pending after fault injection ends and KMIP has recovered.
   • Poll getESECMKIdentifierListStatus with the same timeout and same acceptance of both completed and failure.
   • Verify the cluster is still serving writes: insert a document and read it back. A failure here means mongod lost the ability to encrypt/decrypt with the active KEK during or after the KMIP fault window, which is a server-side bug.
   • Run validate on the seed collection.
8. Write eventually_check_restart_survives_kmip_down.sh (restart-under-fault check). Confirm with the SLS team whether restarting mongod mid-fault is feasible in the Antithesis topology before committing to this work item.
   • This script runs while KMIP is still faulted (Antithesis has not healed the network). Use an eventually_* prefix so it runs in a quiescent window, but do not wait for faults to stop.
   • Gracefully restart mongod.
   • Poll hello until isWritablePrimary (timeout 60 s). A failure here means mongod cannot start without KMIP, which is a server-side bug: startup must succeed using the KEK already persisted in the WiredTiger keystore.
   • Insert a document and read it back to confirm encryption is functional post-restart.
9. Wire the new template into the Evergreen task in atlas_dev.yml. Add a task to the amazon-linux-2023-x86-disagg-storage-services-antithesis-integration-cron-only variant:

     - <<: *antithesis_task_template
       name: antithesis_disagg_kmip_unavailability
       commands:
         - func: "do setup for antithesis"
         - func: "antithesis image build and push"
           vars:
             suite: <disagg resmoke suite — same as antithesis_kek.txt>
             antithesis_test_composer_dir: disagg_kmip_unavailability 

Acceptance Criteria

The acceptance criteria for merging this ticket are limited to test correctness. If the scripts expose server-side bugs when run, each bug should be filed as a separate Jira ticket and the scripts should be merged regardless. The test must not be held back waiting for those bugs to be fixed; it is a diagnostic tool, and other team members need access to it to reproduce and track the failures.

Required to merge

• All scripts are syntactically valid and executable; Test Composer can invoke each one without a launch error.
• The docker-compose topology starts cleanly with the KMIP container at an IP below 10.20.20.130 and SLS service containers at or above it.
• first_setup.sh exits 0 against the local topology: KEK initialization is confirmed and the seed collection is in place before any driver fires.
• Running the topology locally via docker compose up starts all containers cleanly.

Expected behavior when the server has no bugs (not required to merge)

• parallel_driver_crud.sh continues to produce successful operations during KMIP fault injection windows, confirming the active KEK is cached and mongod does not block writes on KMIP availability.
• After a KMIP fault heals, eventually_check_rotation_state.sh confirms both rotation statuses are terminal (completed or failure, not pending) and the cluster accepts new writes.
• Any rotation that was in flight when KMIP became unreachable completes with failure status (not completed), confirming the graceful-failure path fires and the keystore snapshot is restored.
• eventually_check_restart_survives_kmip_down.sh: mongod restarts successfully while KMIP is faulted and resumes serving writes, confirming startup does not require KMIP.

Notes

• The TSAN variant should pick up this task once the standard variant is validated, for data-race coverage of the KMIP error-handling paths in the rotation state machine.
• Do not place SLS service containers below 10.20.20.130 in this topology. The point of this test is to isolate KMIP as the fault domain; SLS chaos is covered by SERVER-129815.
• The restart-under-fault check (Work Item 8) depends on how the SLS team's topology handles mongod restarts. If restarting mongod in Antithesis is not straightforward, scope this as a follow-up ticket rather than blocking the rest of the work.