[SERVER-62686] Can't build due to pinned dependencies Created: 17/Jan/22  Updated: 01/Sep/23  Resolved: 01/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ryan Schmidt Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-63004 Rename resmoke.req to testing.req Closed
Related
related to SERVER-63004 Rename resmoke.req to testing.req Closed
Assigned Teams:
Server Development Platform
Operating System: ALL
Participants:

 Description   

I'm working on updating mongodb-devel in MacPorts to 5.2.0 and it fails because of the resolution of SERVER-57312:

Checking required python packages...
Requirements list:
    Cheetah3<=3.2.6.post1
    PyYAML<=6.0.0,>=3.0.0
    oauthlib==3.1.1
    packaging<=21.3
    pkce==1.0.3
    psutil<=5.8.0
    pymongo<4.0,>=3.9
    pypiwin32==223; sys_platform == "win32" and python_version > "3"
    pywin32==225; sys_platform == "win32" and python_version > "3"
    regex<=2021.11.10
    requests-oauthlib==1.3.0
    requests<=2.26.0,>=2.0.0
    requirements_parser<=0.3.1
    setuptools
(Cheetah3 3.2.6.post2 (/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages), Requirement.parse('Cheetah3<=3.2.6.post1'))

Reverting b6d30f82753814220e360b82d0b5d87e834b279c allows me to get past that but I don't yet know if the build would be successful because of the new dependency on [pkce which is not yet in MacPorts|https://trac.macports.org/ticket/64459].



 Comments   
Comment by Andrew Morrow (Inactive) [ 19/Mar/22 ]

ryandesign - Is there anything further we can help with on this ticket? If not, I'm planning to close it as works-as-designed. While I agree that it is unfortunate that the pinning has made it more difficult to build with macports, we aren't currently in a position to revisit the decision, and using a virtualenv will isolate these dependencies from the macports versions. Note that the final package that we build contains only C++ binaries, so there is no dependency after build time.

Comment by Andrew Morrow (Inactive) [ 02/Mar/22 ]

ryandesign - I asked that this ticket be re-opened because I don't think the prior answer particularly clarified how the build documentation addresses the issue you were experiencing, but I was OOO and not following along until recently. The version pinning, for better or for worse, is required for us because without it we lose the ability to reproduce exact build and test environments at a given commit for CI. The intended usage is that the packages from the requirements files should be installed within a dedicated virtualenv used for building (and testing) mongodb. The build documentation dmitry.agranat linked to does mention that use of a virtualenv is recommended, but without making that connection clear in the reply, I don't think linking to it was helpful on its own. I expect you can sidestep the issues you are encountering by updating your packaging script for the macports mongodb build to create a temporary virtualenv and then install the requirements files within the virtualenv. That will allow macports to continue to release newer versions of packages without disrupting the finicky reliance on specific versions that the mongodb build exhibits. Please let me know if you have any further questions or thoughts on that proposed resolution.

Comment by Dmitry Agranat [ 23/Feb/22 ]

Following our current recommendation provided in my last comment, I will go ahead and close this ticket.

Comment by Dmitry Agranat [ 08/Feb/22 ]

Hi ryandesign, after discussing this internally, our current recommendation is to use these instructions https://github.com/mongodb/mongo/blob/master/docs/building.md when building MongoDB

Comment by Ryan Schmidt [ 03/Feb/22 ]

Please reopen. The latest build failure report from a MacPorts user is:

Checking required python packages...
Requirements list:
Cheetah3
PyYAML>=3.0.0
oauthlib==3.1.1
pkce==1.0.3
psutil
pymongo!=3.6.0,>=3.9
pypiwin32==223; sys_platform == "win32" and python_version > "3"
pywin32==225; sys_platform == "win32" and python_version > "3"
regex
requests-oauthlib==1.3.0
requests>=2.0.0
requirements_parser
setuptools
(requests-oauthlib 1.3.1 (/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages), Requirement.parse('requests-oauthlib==1.3.0'))

You require requests-oauthlib 1.3.0 exactly; MacPorts now has requests-oauthlib 1.3.1.

You also say you require oauthlib 3.1.1. oauthlib 3.2.0 was released a few days ago and it's only a matter of time before its maintainer in MacPorts decides to update it to the latest version.

Pinning dependencies to specific versions is the antithesis of what package managers offer. In a package manager like MacPorts, what's supposed to happen is that projects like mongodb might depend on requests-oauthlib >= 1.3.0, and then when the package manager updates to requests-oauthlib 1.3.1, that's supposed to be good for mongodb and everyone else who uses requests-oauthlib since they will all benefit from whatever bugfixes that version included.

Please consider unpinning all your dependencies, except in the hopefully rare to nonexistent cases where you are aware of a specific problem in a new version of a dependency that adversely affects mongodb. Your default assumption should be that new versions of dependencies are good, not bad.

Comment by Ryan Schmidt [ 28/Jan/22 ]

Please reopen. This is not a duplicate.

Yes, the build is successful without pkce and I was going to suggest separately that you remove that requirement.

However this bug report was about the fact that you have pinned your python module dependencies to specific versions. Note the error message was about Cheetah3: 

(Cheetah3 3.2.6.post2 (/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages), Requirement.parse('Cheetah3<=3.2.6.post1')) 

You now specify that you require Cheetah3<=3.2.6.post1 but MacPorts has 3.2.6.post2 so the build fails. When I reverted to unpinned dependencies it built fine. Why do you require Cheetah3<=3.2.6.post1? What's wrong with 3.2.6.post2?

The pinned dependencies made their way into stable mongodb 5.0.6 so now I'll have to revert the pinning there as well.

I see now that additional pinning of dependencies happened in SERVER-61791. Because of that, and the fact that MacPorts has upgraded to requests 2.27.1, I now get this error trying to build mongodb: 

(requests 2.27.1 (/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages), Requirement.parse('requests<=2.26.0,>=2.0.0')) 

Why do you require requests<=2.26.0? What's wrong with 2.27.1?

Comment by Dmitry Agranat [ 26/Jan/22 ]

Thanks for reporting this ryandesign, we've filed SERVER-63004 to address this but in the meantime, you can take out pkce to build the server

Generated at Thu Feb 08 05:55:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.