[DRIVERS-2497] Remove use of activate_venv.sh and utils.sh Created: 04/Nov/22  Updated: 22/Jan/24  Resolved: 17/Nov/23

Status: Closed
Project: Drivers
Component/s: None
Fix Version/s: None

Type: Task Priority: Unknown
Reporter: Ezra Chung Assignee: Ezra Chung
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Issue split
split to MOTOR-1064 Remove use of activate_venv.sh and ut... Closed
split to CDRIVER-4519 Remove use of activate_venv.sh and ut... Closed
split to CSHARP-4399 Remove use of activate_venv.sh and ut... Closed
split to CXX-2612 Remove use of activate_venv.sh and ut... Closed
split to GODRIVER-2643 Remove use of activate_venv.sh and ut... Closed
split to JAVA-4806 Remove use of activate_venv.sh and ut... Closed
split to NODE-4800 Remove use of activate_venv.sh and ut... Closed
split to PHPC-2164 Refactor run-ocsp-responder.sh to use... Closed
split to PHPLIB-1038 Remove use of activate_venv.sh and ut... Closed
split to PYTHON-3507 Remove use of activate_venv.sh and ut... Closed
split to RUBY-3175 Remove use of activate_venv.sh and ut... Closed
split to RUST-1537 Remove use of activate_venv.sh and ut... Closed
Related
related to DRIVERS-2519 Utilize find-python3.sh in create-ser... Closed
Driver Changes: Needed
Downstream Changes Summary:

Note: the new venv-utils.sh, activate-kmstlsvenv.sh, activate-authawsvenv.sh and activate-ocspvenv.sh scripts MUST be sourced in a Bash shell. See DRIVERS-2497 description and comments for details.

Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4519 Fixed 1.24.0
CXX-2612 Fixed 3.8.0
CSHARP-4399 Fixed 2.19.0
GODRIVER-2643 Done
JAVA-4806 Fixed 4.9.0
NODE-4800 Fixed 4.13.0
MOTOR-1064 Gone away
PYTHON-3507 Gone away
PHPLIB-1038 Fixed 1.16.0
RUBY-3175 Fixed 2.19.0
RUST-1537 Fixed 2.4.0
SWIFT-1678 Won't Do
PHPC-2164 Fixed 1.16.0

 Description   

Summary

The utils.sh and activate_venv.sh scripts provided by Drivers Evergreen Tools are deprecated and will eventually be removed. Drivers must replace their use with the new venv-utils.sh, activate-kmstlsvenv.sh, and find-python3.sh scripts instead.

The new venv-utils.sh and activate-kmstlsvenv.sh are hygienic alternatives to utils.sh and activate_venv.sh that do not leak variables into the environment and do not depend on environment variables for their behavior. Both of these old script behaviors has been observed to cause unexpected and often difficult-to-diagnose distro-specific task failures.

Furthermore, the existing built-in logic to detect a viable python binary capable of creating a virtual environment has been observed to fail on several distros both old and new. To facilitate a more consistent and comprehensive method for finding a viable python binary, a new find-python3.sh script has been provided to replace the current ad hoc implementations both in Drivers and in Drivers Evergreen Tools.

Before:

cd drivers-evergreen-tools/.evergreen/csfle
. ./activate_venv.sh # Leaks PYTHON set prior to venv.
. ./set-temp-creds.sh # Uses leaked non-venv PYTHON.

After:

cd drivers-evergreen-tools/.evergreen/csfle
. ./activate-kmstlsvenv.sh # No leaks.
. ./set-temp-creds.sh # Uses venv python.

Before:

. drivers-evergreen-tools/.evergreen/utils.sh
PYTHON="path/to/python" # Ad hoc, complicated, and inconsistent.
venvcreate "$PYTHON" venv # Leaks PYTHON, depends on undocumented OS env var.

After:

. drivers-evergreen-tools/.evergreen/venv-utils.sh
. drivers-evergreen-tools/.evergreen/find-python3.sh
PYTHON="$(find_python3)" # Uniform python lookup utility.
venvcreate "$PYTHON" venv # No leaks or undocumented dependencies.

Motivation

Who is the affected end user?

Drivers that are using the utils.sh or activate_venv.sh scripts provided by Drivers Evergreen Tools.

How does this affect the end user?

Drivers must update their scripts to replace usage of utils.sh or activate_venv.sh as described above. This should only affect Evergreen configurations and scripts; Drivers implementations should not require any modifications.

How likely is it that this problem or use case will occur?

Depends on the Driver's use of the scripts and related environment variables such as PYTHON, PYTHON_BINARY, and OS.

If the problem does occur, what are the consequences and how severe are they?

Time lost to diagnosing unexpected and problematic Python virtual environment creation or activation behavior.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

Yes. Evergreen configuration and scripts only.



 Comments   
Comment by Githook User [ 22/Jan/24 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: DRIVERS-2497 remove deprecated venv scripts (#372)
Branch: dependabot/pip/dot-evergreen/ocsp/flask-2.2.5
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/d3f0e400282eff866cca7e21da0b30a86fae0d3c

Comment by Githook User [ 17/Nov/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: DRIVERS-2497 remove deprecated venv scripts (#372)
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/d3f0e400282eff866cca7e21da0b30a86fae0d3c

Comment by Ross Lawley [ 21/Nov/22 ]

Note: Also added scripts for auth_aws and ocsp testing.

Comment by Githook User [ 21/Nov/22 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Added .evergreen/ocsp/activate-ocspvenv.sh

For a consistent way to manage the ocsp venv

DRIVERS-2497
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/a3a17f1185643135a6b24f318bf0ce1e58fdea94

Comment by Githook User [ 21/Nov/22 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Added activate-authawsvenv.sh for auth_aws

Replace usage of .evergreen/auth_aws/

  • activate_venv.sh with
  • activate-authawsvenv.sh

DRIVERS-2497
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/2180c6c21fb5ef47fc9dfaceb96b9162b89c9e5f

Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: DRIVERS-2497 Fix paths on Cygwin and Python package dependencies (#244)

  • Use absolute Windows form for venv path in Cygwin
  • Remove unnecessary --no-cache-dir
  • Correctly ensure -p is used with virtualenv module
  • Ensure all seed packages are installed in virtual environment
  • Add --system-site-packages when creating virtual environment
  • Address python package build failures
  • Improve documentation justifying use of -p with virtualenv
Comment by Ezra Chung [ 07/Nov/22 ]

Note: the new scripts venv-utils.sh and activate-kmstlsvenv.sh require being sourced in a Bash shell. If converting a given task or script to use Bash is non-trivial, the new scripts may be invoked in an explicit bash (sub)shell to minimize the changes necessary to the rest of the task or script.

Before (non-Bash shell):

. drivers-evergreen-tools/.evergreen/utils.sh
PYTHON="path/to/python" # ad hoc.
venvcreate "$PYTHON" venv
# Commands requiring virtual environment.

After (still non-Bash shell):

bash -s <<-CMD
  . drivers-evergreen-tools/.evergreen/utils.sh
  . drivers-evergreen-tools/.evergreen/find-python3.sh
  PYTHON="$(find_python3)"
  venvcreate "$PYTHON" venv
CMD
 
# (Re-)activate virtual environment in non-Bash shell.
if [ -f venv/bin/activate ]; then
  . venv/bin/activate
else
  . venv/Scripts/activate
fi
 
# Commands requiring virtual environment.

Before (non-Bash shell):

. ./activate_venv.sh
. ./set-temp-creds.sh
# Commands requiring temp creds.
# Other commands.

After (still non-Bash shell):

bash -s <<-CMD
. ./activate-kmstlsvenv.sh
. ./set-temp-cred.sh
# Commands requiring temp creds.
CMD
 
# Other commands.

Generated at Thu Feb 08 08:25:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.