s_clang_format fails on systems without python3-venv (ensurepip), breaking s_all and all code generators

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines
    • 17.726
    • None
    • None

      WT-17626 changed dist/s_clang_format to obtain clang-format by creating a Python virtual environment and pip-installing the clang-format wheel from PyPI:
      venv="$TOP_DIR/.venv"
      download_clang_format()

      { version=$1 python3 -m venv "$venv" "$venv/bin/pip" install --quiet --disable-pip-version-check "clang-format==$version" }

       
      This introduces a hard dependency on ensurepip (the Debian/Ubuntu python3.10-venv package). On a stock Ubuntu box where that package isn't installed, plain python3 -m venv creates a pip-less venv and the next line fails.

      Steps to reproduce:

      1. On a system without python3-venv/ensurepip, with a clang-format other than 12.0.1 on PATH (or none).
      2. Run dist/s_all (or dist/s_fast, or any generator such as ./api_config_gen.py).

      Actual result:
      The virtual environment was not created successfully because ensurepip is not
      available. On Debian/Ubuntu systems, you need to install the python3-venv
      package using the following command.
       
      apt install python3.10-venv
      ...
      ./s_clang_format: line 14: /home/.../.venv/bin/pip: No such file or directory
      Command '['./s_clang_format', '/home/.../src/include/config.h.MOD']' returned non-zero exit status 1.
       
      Impact:

      • Regression: the previous prebuilt-binary path worked on these machines without python3-venv.
      • Repository is polluted with changes as a result of the s_all failuer:
        (use "git restore <file>..." to discard changes in working directory)
              modified:   ../src/checkpoint/checkpoint.h
              modified:   ../src/checkpoint/checkpoint_private.h
              modified:   ../src/evict/evict.h
              modified:   ../src/evict/evict_private.h
              modified:   ../src/include/extern.h
              modified:   ../src/include/extern_darwin.h
              modified:   ../src/include/extern_linux.h
              modified:   ../src/include/extern_posix.h
              modified:   ../src/include/extern_win.h
              modified:   ../src/live_restore/live_restore.h
              modified:   ../src/live_restore/live_restore_private.h
              modified:   ../src/log/log.h
              modified:   ../src/log/log_auto.c
              modified:   ../src/log/log_private.h
              modified:   ../src/reconcile/reconcile.h
              modified:   ../src/reconcile/reconcile_private.h
              modified:   ../test/format/format_config.h
              modified:   ../test/format/format_config_def.c 
      • Not isolated to the dedicated clang-format step. Every generator that formats its output via dist.py:format_srcfile() shells out to s_clang_format (api_config_gen.pystat.pylog.pyprototypes.pyapi_err.pytype_to_str.py), so they all hit this. Because format_srcfile catches the error and only prints it, generators can "succeed" while emitting unformatted C.
      • Clobber risk: the venv is built at $TOP_DIR/.venv (the repo root), where many developers keep their own virtualenv. Running s_all does python3 -m venv over it and pip-installs into it; on an affected box it leaves a broken, pip-less .venv in the tree.

      Suggested fix:

      • Don't hard-depend on ensurepip/python3-venv. Use a tool that bundles pip (virtualenv/uv), or python3 -m venv --without-pip + bootstrap pip, or fall back to user pip/pipx; at minimum print an actionable error and abort the s_fast 
      • Build the venv in a dedicated, gitignored location (e.g. dist/.clang-format-venv or a cache dir) instead of $TOP_DIR/.venv to avoid clobbering developers' environments.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Luke Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: