Summary

The existing disagg-storage Antithesis Evergreen variants (amazon-linux-2023-x86-disagg-storage-services-antithesis-integration-cron-only and its TSAN
counterpart) build mongod and push it to the SLS team's ECR via push_mongod_to_ecr. The SLS team's Antithesis environment then runs tests using that binary. Currently neither the MongoDB repo nor the SLS team's Antithesis topology has any test_composer scripts that exercise dynamic KEK generation, eseRotateActiveKEK, or updateESECMKIdentifierList. This ticket adds that coverage by writing Test Composer scripts in a new template directory and wiring them
into the existing Antithesis task in atlas_dev.yml.

There is no existing KMIP container in the disagg Antithesis topology. One must be built as part of this ticket (Work Item 1). Coordinate with the SLS/PALI team to confirm the structure of their existing Docker topology (which containers are present, which images are used, and where the workload container executes scripts) before wiring in the new KMIP Docker container listed below as the first work item.

Work Items

1. Build the KMIP Docker container. The Antithesis topology has no KMIP server. Create a purpose-built container based on ubuntu:22.04 x86-64:
   • Write buildscripts/antithesis/base_images/kmip/Dockerfile:
     • FROM ubuntu:22.04
     • Install Python 3.13 via the deadsnakes PPA (mirror the pattern in the existing workload/Dockerfile).
     • pip install the kmip package. The venv at python3-venv/ carries kmip 0.11.0.dev1; install from that local copy rather than PyPI to ensure the same version used in the existing encryption JS tests is used here.
     • Copy src/mongo/db/modules/enterprise/jstests/encryptdb/kmip_server.py and kmip_manage_key.py into the image. These scripts contain critical monkey patches to the standard PyKMIP server (default encrypt/decrypt parameters, socket timeout removal, connection-error handling) that are required for compatibility with mongod's  KMIP client. Do NOT use the bare pykmip-server entry point.
     • Copy the TLS certificates (trusted-ca.pem, trusted-server.pem, trusted-client.pem) from jstests/libs/ into the image. Set the container command to: python kmip_server.py <ca_path> <cert_path> --kmipPort 5696
   • Add the kmip service to the disagg docker-compose.yml (coordinate with SLS team on exact placement):
     • Assign an IP >= 10.20.20.130 so Antithesis does not inject network faults on it. Faulting the KMIP server would test KMIP unavailability, not encryption under SLS chaos; that is a separate, explicitly scoped scenario.
     • The mongod container depends_on the kmip service.
   • Update the mongod startup config in the topology to include:
     • kmipPort, kmipServerName, kmipServerCAFile, kmipClientCertificateFile pointing at the kmip container.
     • initialDisaggESECMKIdentifierList set to a CMK entry whose cmkId matches the UUID of the key created by first_setup.sh
     • featureFlagKEKPushMode: true and featureFlagCMKRotation: true in setParameters.
2. Create the template directory buildscripts/antithesis/test_composer/disagg_encryption/. All scripts in this directory must be executable (chmod +x). Any shared logic goes in a helper_*.sh file or a subdirectory so Test Composer ignores it.
3. Write first_setup.sh (one-time cluster setup, runs before any driver starts):
   • Use kmip_manage_key.py to create and activate a KMIP key; record its UUID for use by the CMK rotation driver (Work Item 6) and for the initialDisaggESECMKIdentifierList value baked into the mongod config (Work Item 1). Create a second key and activate it for the two-entry CMK list used during CMK rotations.
   • Wait for mongod to become primary (poll hello until isWritablePrimary).
   • Verify that KEK initialization completed: poll getESERotateActiveKEKStatus until the active KEK ID is >= 1 (fresh-cluster KEK generation is done at step-up).
   • Verify getESECMKIdentifierListStatus returns a non-empty activeCMKIdentifierList.
   • Insert a seed collection of documents that post-rotation consistency checks can query.
4. Write parallel_driver_crud.sh (continuous CRUD workload, runs concurrently during fault injection):
   • Continuously insert, update, find, and delete documents against the primary, with retries on NotWritablePrimary and NetworkError.
   • Mirror the retry wrapper pattern from buildscripts/antithesis/test_composer/basic_js_commands/js/commands.js.
5. Write parallel_driver_kek_rotation.sh (KEK rotation driver, runs concurrently during fault injection):
   • In a loop: sleep a random interval (10-40 s), attempt eseRotateActiveKEK on the primary.
   • Tolerate transient errors (NotWritablePrimary, ConflictingOperationInProgress, NetworkError) — log and retry on the next iteration.
   • On success, poll getESERotateActiveKEKStatus until the status is completed or failure (not pending), then log the outcome and continue.
6. Write parallel_driver_cmk_rotation.sh (CMK rotation driver, runs concurrently during fault injection):
   • In a loop: sleep a random interval (15-60 s), attempt updateESECMKIdentifierList with a two-entry CMK list (the two UUIDs created in first_setup.sh).
   • Tolerate the same transient errors as Work Item 5.
   • On success, poll getESECMKIdentifierListStatus until rotationStatus.status is not pending, then log and continue.
7. Write eventually_check_rotation_state.sh (post-recovery consistency check, runs after drivers are halted and faults are stopped):
   • Poll getESERotateActiveKEKStatus until status is not pending (timeout 60 s). Assert that the status is completed or failure, never still pending. A rotation stuck in pending after fault injection ends indicates a server-side bug in the rotation state machine.
   • Poll getESECMKIdentifierListStatus with the same timeout. Assert not pending.
   • Write one document and read it back to confirm the encryption layer is functional.
   • Run validate on the seed collection.
8. Wire the new template into the Evergreen task in atlas_dev.yml - Add an antithesis task (following the antithesis_task_template pattern in etc/evergreen_yml_components/tasks/misc_tasks.yml) to the amazon-linux-2023-x86-disagg-storage-services-antithesis-integration-cron-only variant:

     - <<: *antithesis_task_template
       name: antithesis_disagg_encryption
       commands:
         - func: "do setup for antithesis"
         - func: "antithesis image build and push"
           vars:
             suite: <disagg resmoke suite that brings up the cluster topology>
             antithesis_test_composer_dir: disagg_encryption 

The exact suite name must be confirmed with the SLS team (it controls the cluster topology used to generate the docker-compose.yml).

Acceptance Criteria

The acceptance criteria for merging this ticket are limited to test correctness. If the
scripts expose server-side bugs when run, each bug should be filed as a separate Jira ticket and the scripts should be merged regardless. The test must not be held back waiting for those bugs to be fixed; it is a diagnostic tool, and other team members need access to it to reproduce and track the failures.

Required to merge

1. The KMIP container builds and starts cleanly; first_setup.sh exits 0 against the local topology (docker compose up), confirming the container, certificates, and key creation steps are all correct.
2. All scripts are syntactically valid and executable; Test Composer can invoke each one without a launch error.
3.  first_setup.sh confirms KEK initialization (activeKEKId >= 1) and inserts the seed collection before any driver fires.
4. Running the topology locally via docker compose up starts all containers cleanly.

Expected behavior when the server has no bugs (not required to merge)

• Under Antithesis network fault injection, at least one KEK rotation and one CMK rotation each reach a terminal status (completed or failure) without getting stuck in pending after a fault heals.
• eventually_check_rotation_state.sh passes: both rotation statuses are terminal, the encryption layer accepts new writes, and validate succeeds.
• parallel_driver_crud.sh produces no assertion errors due to rotation activity; documents written before and after a rotation are readable.

Notes

• The TSAN variant (amazon-linux-2023-x86-disagg-storage-services-antithesis-tsan-integration-cron-only) should also pick up the new task once the standard variant is validated, for data-race coverage across KEK and CMK rotation paths.
• PyKMIP 0.11.0.dev1 is the version in python3-venv/. This is a development version not available on PyPI; it must be installed from the local copy in the repo. Do not substitute the PyPI 0.10.0 release — it lacks the monkey patches in kmip_server.py and will misbehave with mongod's KMIP client.
• Do not assign an IP below 10.20.20.130 to the KMIP container. Antithesis injects network faults only on containers below that threshold; a faulted KMIP server would change the test's character from "encryption under SLS chaos" to "KMIP unavailability handling", which is covered in SERVER-xxx.