[SERVER-34438] Consolidate requirements.txt Created: 12/Apr/18  Updated: 27/Oct/23  Resolved: 28/Sep/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 3.7.3
Fix Version/s: None

Type: Task Priority: Trivial - P5
Reporter: Robert Guo (Inactive) Assignee: DO NOT USE - Backlog - Test Infrastructure Group (TIG)
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-36152 Update requirements.txt Closed
Participants:

 Description   

There are currently a number of requirements.txt files used by various Python components of the server codebase. Since the server is logically a single project that is typically run under the same virtualenv, we should have a single place to specify requirements to avoid various issues like conflicting dependencies (even with proper dependency resolution) and missing dependencies.



 Comments   
Comment by Robert Guo (Inactive) [ 28/Sep/18 ]

Definitely! I will close this ticket.

Comment by Benjamin Caimano (Inactive) [ 28/Sep/18 ]

robert.guo, do you think SERVER-36152 has addressed this ticket?

Comment by Robert Guo (Inactive) [ 12/Apr/18 ]

I don't think there are missing dependencies in there. pytests/requirements.txt was what I had in mind when saying "hunt down all the requirement files", it is currently independent of buildscripts.

Comment by Max Hirschhorn [ 12/Apr/18 ]

Max Hirschhorn what do you think of adding a top-level requirements.txt file to include all the other requirements.

I think that's what buildscripts/requirements.txt should be doing already. If there are dependencies which are missing then we should probably address that. The boto3 package was an interesting case of we didn't want local users of resmoke.py to need to bother installing it since archival is only enabled in Evergreen when we have AWS credentials to use. https://www.python.org/dev/peps/pep-0508/#extras may be a better way of expressing the behavior we want.

Comment by Robert Guo (Inactive) [ 12/Apr/18 ]

max.hirschhorn what do you think of adding a top-level requirements.txt file to include all the other requirements. This would allow us to catch conflicting dependencies across multiple sub-projects and not force users to hunt down all the requirement files just to run pylinters.py lint. (The pylinters issue is temporarily worsened by a subtle dependency of boto3 in pytests/requirements by buildscripts/ because of boto3 not being in all the necessary requirements files due to external factors.)

On an unrelated note, I had misinterpreted 1a in the dependency resolution issue: https://github.com/pypa/pip/issues/988

only one spec allowed per project, regardless of conflicts or not

This isn't the case for my local testing. So fixing the "missing dependencies" benefit in this ticket description is incorrect

Comment by Max Hirschhorn [ 12/Apr/18 ]

I think individual Python packages should declare their own requirements. This is why -r resmokelib/requirements.txt is used to reference resmoke.py's dependencies in buildscripts/requirements.txt. I'd be happy to change it to requirements/resmoke.txt, etc. but I'd still like them in separate files.

Generated at Thu Feb 08 04:36:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.