[SERVER-76889] Add version range for setuptools Created: 05/May/23  Updated: 27/Oct/23  Resolved: 27/Oct/23

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 7.0 Required
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andy Artemev Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Won't Do Votes: 0
Labels: Bug, neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 20.04.01


Assigned Teams:
Server Development Platform
Operating System: ALL
Steps To Reproduce:

// Install setuptools (version that is out of mentioned range)  
 
// Try to build Mongo Server
user@ubuntu20.04:~/mongo$ sudo python3 buildscripts/scons.py DESTDIR=/opt/mongo_uttam CCFLAGS="" SHCCFLAGS="" --linker=gold LINKFLAGS="" install-mongod
 
// Try to check & [re]install improper packages as suggested by failed installation output
user@ubuntu20.04:~/mongo$ /usr/bin/python3 -m pip install -r etc/pip/compile-requirements.txt

Participants:

 Description   

Consider including a particular versions range into pip components specification for 

setuptools

 https://github.com/mongodb/mongo/blob/v7.0/etc/pip/components/compile.req#L6

 

because

mongo-tooling-metrics == 1.0.7 

https://github.com/mongodb/mongo/blob/v7.0/etc/pip/components/tooling_metrics.req#L1

requires

 

setuptools >= 58.1.0, < 59.0.0 

 

 

Without that restrictions, if an incompatible (with mongo-tooling-metrics) version of setuptools (e.g. 45.2.0) is already installed on the system, we observe such behavior:

user@ubuntu20.04:~/mongo$ sudo python3 buildscripts/scons.py DESTDIR=/opt/mongo_uttam CCFLAGS="" SHCCFLAGS="" --linker=gold LINKFLAGS="" install-mongod

Checking required python packages...

Requirements list:

    Cheetah3<=3.2.6.post1

    PyYAML<=6.0.0,>=3.0.0

    cryptography==2.3; platform_machine == "s390x" or platform_machine == "ppc64le"

    cryptography==36.0.2; platform_machine != "s390x" and platform_machine != "ppc64le"

    mongo-ninja-python==1.11.1.4; platform_machine == "x86_64" and sys_platform == "linux"

    mongo-tooling-metrics==1.0.7

    ninja>=1.10.0; platform_machine != "x86_64" or sys_platform != "linux"

    packaging<=21.3

    psutil<=5.8.0

    pymongo==4.3.3

    pypiwin32>=223; sys_platform == "win32" and python_version > "3"

    pywin32>=225; sys_platform == "win32" and python_version > "3"

    regex<=2021.11.10

    requests<=2.26.0,>=2.0.0

    requirements_parser<=0.3.1

    setuptools

    types-PyYAML~=6.0.5

    typing-extensions>=3.7.4

(setuptools 45.2.0 (/usr/lib/python3/dist-packages), Requirement.parse('setuptools<59.0.0,>=58.1.0'), {'mongo-tooling-metrics'})

Try running:

    /usr/bin/python3 -m pip install -r etc/pip/compile-requirements.txt

 

user@ubuntu20.04:~/mongo$ /usr/bin/python3 -m pip install -r etc/pip/compile-requirements.txt

/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead

  from cryptography.utils import int_from_bytes

/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead

  from cryptography.utils import int_from_bytes

Ignoring pypiwin32: markers 'sys_platform == "win32" and python_version > "3"' don't match your environment

Ignoring pywin32: markers 'sys_platform == "win32" and python_version > "3"' don't match your environment

Ignoring cryptography: markers 'platform_machine == "s390x" or platform_machine == "ppc64le"' don't match your environment

Ignoring ninja: markers 'platform_machine != "x86_64" or sys_platform != "linux"' don't match your environment

Requirement already satisfied: cryptography==36.0.2 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/platform.req (line 6)) (36.0.2)

Requirement already satisfied: mongo-ninja-python==1.11.1.4 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/platform.req (line 8)) (1.11.1.4)

Requirement already satisfied: psutil<=5.8.0 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/core.req (line 2)) (5.8.0)

Requirement already satisfied: pymongo==4.3.3 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/core.req (line 3)) (4.3.3)

Requirement already satisfied: PyYAML<=6.0.0,>=3.0.0 in /usr/lib/python3/dist-packages (from -r etc/pip/components/core.req (line 4)) (5.3.1)

Requirement already satisfied: types-PyYAML~=6.0.5 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/core.req (line 5)) (6.0.12.9)

Requirement already satisfied: requests<=2.26.0,>=2.0.0 in /usr/lib/python3/dist-packages (from -r etc/pip/components/core.req (line 6)) (2.22.0)

Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/core.req (line 7)) (4.5.0)

Requirement already satisfied: Cheetah3<=3.2.6.post1 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/compile.req (line 2)) (3.2.6.post1)

Requirement already satisfied: packaging<=21.3 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/compile.req (line 3)) (21.3)

Requirement already satisfied: regex<=2021.11.10 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/compile.req (line 4)) (2021.11.10)

Requirement already satisfied: requirements_parser<=0.3.1 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/compile.req (line 5)) (0.3.1)

Requirement already satisfied: setuptools in /home/ubuntu/.local/lib/python3.8/site-packages (from -r etc/pip/components/compile.req (line 6)) (45.2.0)

Requirement already satisfied: mongo-tooling-metrics==1.0.7 in /usr/local/lib/python3.8/dist-packages (from -r etc/pip/components/tooling_metrics.req (line 1)) (1.0.7)

Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.8/dist-packages (from cryptography==36.0.2->-r etc/pip/components/platform.req (line 6)) (1.15.1)

Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in /usr/local/lib/python3.8/dist-packages (from pymongo==4.3.3->-r etc/pip/components/core.req (line 3)) (2.3.0)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging<=21.3->-r etc/pip/components/compile.req (line 3)) (3.0.9)

Requirement already satisfied: types-setuptools<58.0.0,>=57.4.4 in /usr/local/lib/python3.8/dist-packages (from requirements_parser<=0.3.1->-r etc/pip/components/compile.req (line 5)) (57.4.18)

Requirement already satisfied: pydantic<2.0.0,>=1.8.2 in /usr/local/lib/python3.8/dist-packages (from mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (1.10.7)

Requirement already satisfied: gitpython<4.0.0,>=3.1.29 in /usr/local/lib/python3.8/dist-packages (from mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (3.1.31)

Requirement already satisfied: distro<2.0.0,>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (1.8.0)

Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.12->cryptography==36.0.2->-r etc/pip/components/platform.req (line 6)) (2.21)

Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.8/dist-packages (from gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (4.0.10)

Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.8/dist-packages (from gitdb<5,>=4.0.1->gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (5.0.0)

 



 Comments   
Comment by Andy Artemev [ 11/May/23 ]

Hey tausif.rahman@mongodb.com, thank you for your prompt response!

 

  • Sure, on my setup I solved that issue by manually upgrading setuptools when `pip install -r etc/pip/compile-requirements.txt` failed to do that;

 

  • My setup is: python v3.8.10, pip v20.0.2, no virtualenv;

 

  • After your comment I tried to run `pip install -r etc/pip/compile-requirements.txt` using virtualenv v20.0.17 with the same "bad" results:

Requirement already satisfied: pycparser in /home/user/test/venv/lib/python3.8/site-packages (from cffi>=1.12->cryptography==36.0.2->-r etc/pip/components/platform.req (line 6)) (2.21)

Requirement already satisfied: gitdb<5,>=4.0.1 in /home/user/test/venv/lib/python3.8/site-packages (from gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (4.0.10)

Requirement already satisfied: smmap<6,>=3.0.1 in /home/user/test/venv/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (5.0.0)

ERROR: mongo-tooling-metrics 1.0.7 has requirement setuptools<59.0.0,>=58.1.0, but you'll have setuptools 45.0.0 which is incompatible.

Installing collected packages: mongo-tooling-metrics

Successfully installed mongo-tooling-metrics-1.0.7

 

  • But after upgrading pip from v20.0.2 to your v23.1.2, I ended up with your "good" results:

Requirement already satisfied: pycparser in /home/user/test/venv/lib/python3.8/site-packages (from cffi>=1.12->cryptography==36.0.2->-r etc/pip/components/platform.req (line 6)) (2.21)

Requirement already satisfied: gitdb<5,>=4.0.1 in /home/user/test/venv/lib/python3.8/site-packages (from gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (4.0.10)

Requirement already satisfied: smmap<6,>=3.0.1 in /home/user/test/venv/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (5.0.0)

Installing collected packages: setuptools

  Attempting uninstall: setuptools

    Found existing installation: setuptools 45.0.0

    Uninstalling setuptools-45.0.0:

      Successfully uninstalled setuptools-45.0.0

Successfully installed mongo-tooling-metrics-1.0.7 setuptools-58.5.3

 

  • So the difference in behavior is in pip versions (v20.0.2 vs v23.1.2);

 

  • Regarding the boundaries of the dependency’s versions range – I suppose the wider the range will be (of course, without regression and side effects), the more users setups it will cover. About the upper bound - currently the latest version of setuptools is 67.7.2, which also has the `get_distribution` function.

 

Comment by Tausif Rahman (Inactive) [ 10/May/23 ]

We have the dependency in mongo-tooling metrics because it uses the setuptools `get_distribution` function. I kind of just chose that setuptools version, but we haven't had any issues for the past few months.

  1. Can you upgrade your setuptools vesrion?
  2. I tried to repro your issue w/ python3.8, pip 23.1.2, virtualenv, & all packages installed except mongo-tooling-metrics. I also downgraded setuptools to 45. When I run pip install -r etc/pip/compile-requirements.txt, it downloads mongo-tooling-metrics and upgrade setuptools to 58. I see this at the end of the output:
    1. Requirement already satisfied: pycparser in ./test-venv/lib/python3.8/site-packages (from cffi>=1.12->cryptography==36.0.2->-r etc/pip/components/platform.req (line 6)) (2.21)
      Requirement already satisfied: gitdb<5,>=4.0.1 in ./test-venv/lib/python3.8/site-packages (from gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (4.0.10)
      Requirement already satisfied: smmap<6,>=3.0.1 in ./test-venv/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->gitpython<4.0.0,>=3.1.29->mongo-tooling-metrics==1.0.7->-r etc/pip/components/tooling_metrics.req (line 1)) (5.0.0)
      Installing collected packages: setuptools
        Attempting uninstall: setuptools
          Found existing installation: setuptools 45.0.0
          Uninstalling setuptools-45.0.0:
            Successfully uninstalled setuptools-45.0.0
      Successfully installed setuptools-58.5.3

We can loosen the dependency as well – I've tried with as old as setuptools 20 & it looks like this works. But this will require changes across 3 repos (mongo-tooling-metrics, mongo-ninja-python [uses mongo-tooling-metrics] & mongo[uses mongo-tooling-metrics]).

Generated at Thu Feb 08 06:33:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.