ExportXMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Done
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Go Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Name of Failure:

      libssl.so.1.1: cannot open shared object file

      Link to task:

      https://spruce.corp.mongodb.com/task/mongo_go_driver_testazurekms_variant_testazurekms_task_3c254894621703106972d659cfeea33e9e55c103_26_11_05_16_26_25/logs?execution=1

      Context of when and why the failure occurred:

      Seems to have started failing Sept 3 in this build. However, it's not clear that the change caused the failure. It seems more likely that an environmental change caused the failure (e.g. something about the OS image changed).

      Analysis by Claude:

      *Most likely cause: the binary runs on a different machine from the one that built it*

      • `run-gcpkms-test.sh` builds on the Evergreen host (`rhel8.7-small`). It then tars only `./testkms` and `libmongocrypt.*` and runs them on the GCE VM via `copy-file.sh` and `run-command.sh`.
      • Shared dependencies such as `libcrypto` and `libssl` are not copied. Whatever the VM image has is what gets loaded.
      • RHEL 8.x ships OpenSSL 1.1.1, which provides `libssl.so.1.1`. So a RHEL 8.7 host can't be missing that library. A VM image that has moved to OpenSSL 3 (Debian 12+, Ubuntu 22.04+, RHEL 9) would be. That fits your error exactly: the binary is linked against a 1.1 soname that the VM doesn't have.
      • The reverse can also happen. If the build host had OpenSSL 3, the VM would fail on `libssl.so.3`, so the 1.1 soname points to a build host that links against 1.1.

      Confirmed that both GCP and Azure seem to launch VMs with Debian 12. Running the build on RHEL 9.3 seems to fix the OpenSSL 1.1 linking failures (presumably because it's building against OpenSSL 3.0 instead).

      The Azure KMS fails with a new error:

      [2026/11/05 16:11:10.840] az not detected. See https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/csfle/azurekms/README.md for supported distrosWhich is saying the az CLI isn't available.
      

      Stack trace:

      [2026/11/05 13:11:57.795] Running 'LD_LIBRARY_PATH=./install/libmongocrypt/lib64 MONGODB_URI='mongodb://localhost:27017' PROVIDER='azure' AZUREKMS_KEY_NAME=<REDACTED:AZUREKMS_KEY_NAME> AZUREKMS_KEY_VAULT_ENDPOINT=https://drivers-2411-keyvault.vault.azure.net ./testkms' on Azure Virtual Machine ... begin
      [2026/11/05 13:11:58.931] go: downloading golang.org/x/text v0.43.0
      [2026/11/05 13:11:58.931] ./testkms: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
      [2026/11/05 13:11:58.934] task: Failed to run task "test-azurekms": exit status 127
      [2026/11/05 13:11:58.934] Command 'subprocess.exec' (step 1 of 1) failed: process encountered problem: exit code 201.
      

      Seems reasonable to conclude that the Azure and GCP instances have stopped supporting libssl 1.1.

            Assignee:
            Matt Dale
            Reporter:
            Matt Dale
            Preston Vasquez
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: