[SERVER-53579] python's dev-requirements.txt is incompatible with pip 20.3.3 Created: 05/Jan/21  Updated: 29/Oct/23  Resolved: 27/Jan/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.9.0, 4.2.13, 4.4.5

Type: Bug Priority: Major - P3
Reporter: Igor Solodovnikov Assignee: Ryan Egesdahl (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
is related to SERVER-53250 Evaluate consequences of using upgrad... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4, v4.2
Steps To Reproduce:

Run

 
pip install -r etc/pip/dev-requirements.txt

with pip version 20.3.3

Sprint: Dev Platform 2021-01-25, Dev Platform 2021-02-08
Participants:

 Description   

Trying to build v4.4 from sources as described in https://github.com/mongodb/mongo/blob/v4.4/docs/building.md

When I run following command with latest pip version (20.3.3) it never finishes:

 
pip install -r etc/pip/dev-requirements.txt

With previous pip version (20.2.2) the same command finishes in few seconds.

I believe this behavior is caused by this issue in the new pip's dependency resolver implementation: https://github.com/pypa/pip/issues/9215



 Comments   
Comment by Githook User [ 18/Feb/21 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-53579 Update mypy version

The version of mypy we were depending on was very old and was causing a
version conflict on typed-ast with another module. On newer pip versions,
the result was a very slow process of walking back compatible versions
of multiple packages. This change updates mypy to the latest version
available.

(cherry picked from commit e4bd2ae1c3e3b9c8acd175472239134ffde891de)
(cherry picked from commit 46962af825bc3165055dd75f36933555adc09ce9)
Branch: v4.2
https://github.com/mongodb/mongo/commit/0e9c675a0f476a9334f36d98a91f23f1cf46d7d5

Comment by Githook User [ 16/Feb/21 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-53579 Update mypy version

The version of mypy we were depending on was very old and was causing a
version conflict on typed-ast with another module. On newer pip versions,
the result was a very slow process of walking back compatible versions
of multiple packages. This change updates mypy to the latest version
available.

(cherry picked from commit e4bd2ae1c3e3b9c8acd175472239134ffde891de)
Branch: v4.4
https://github.com/mongodb/mongo/commit/46962af825bc3165055dd75f36933555adc09ce9

Comment by Andrew Morrow (Inactive) [ 08/Feb/21 ]

ryan.egesdahl - Can we backport this to v4.4 and v4.2?

Comment by Githook User [ 27/Jan/21 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-53579 Update mypy version

The version of mypy we were depending on was very old and was causing a
version conflict on typed-ast with another module. On newer pip versions,
the result was a very slow process of walking back compatible versions
of multiple packages. This change updates mypy to the latest version
available.
Branch: master
https://github.com/mongodb/mongo/commit/e4bd2ae1c3e3b9c8acd175472239134ffde891de

Comment by Ryan Egesdahl (Inactive) [ 05/Jan/21 ]

igorsol I fixed exactly the same thing for exactly the same reasons and in almost exactly the same way back in SERVER-50592, but it looks like we may need to do it again. I could wish the new dependency resolver had been a bit more specific about why it was choking, though. Thank you very much for the extra effort!

Comment by Igor Solodovnikov [ 05/Jan/21 ]

@Ryan Egesdahl

FWIW I found another workaround: I have noticed outdated version of 'mypy' is specified in 'etc/pip/components/lint.req' and this is why outdated version of typed-ast is installed.

So I replaced mypy version 0.580 with 0.790 (in etc/pip/components/lint.req). With this change pip 20.3.3 (without 'legacy-resolver' option) successfully finished in about 10 seconds without any warnings or errors.

Comment by Igor Solodovnikov [ 05/Jan/21 ]

@Ryan Egesdahl

Yes, the workaround with `legacy-resolver` works for me. But this is only a temporary solution because pip developers have plans to remove this option in the pip 21.0.0.

BTW when I run pip with `legacy-resolver` it prints this error message:

 
ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
astroid 2.4.2 requires typed-ast<1.5,>=1.4.0; implementation_name == "cpython" and python_version < "3.8", but you'll have typed-ast 1.1.2 which is incompatible.

but everything else seems to be ok.

Comment by Ryan Egesdahl (Inactive) [ 05/Jan/21 ]

igorsol - Ah, sorry, I was thinking of another issue we've been seeing with the new dependency resolver. We've seen this one as well, and I think people have been simply pinning the pip version. From what I can tell of reading the issues, I am not sure I could tell you whether the problem is actually etc/pip/dev-requirements.txt or a bug in the new resolver - and I don't think the pip developers could at this point, either. It could even be a problem caused by dependencies set by two packages we depend on. Here are the two issues I found that seem most relevant:

https://github.com/pypa/pip/issues/9187
https://github.com/pypa/pip/issues/6536

From what I am reading, it seems like you might be able to work around this by specifying the legacy resolver, like so:

python3 -m pip install --use-deprecated=legacy-resolver -r etc/pip/dev-requirements.txt

Please try that and let us know if it works for you.

Comment by Igor Solodovnikov [ 05/Jan/21 ]

@Ryan

Yes, I tried this inside clean virtual environment.

No, I don't see pip exit. Pip just continues to download more and more versions of packages and then stucks on some version of PyMongo if I remember it correctly.

Comment by Ryan Egesdahl (Inactive) [ 05/Jan/21 ]

igorsol - Have you tried installing into a clean virtual environment? I ask because that seems to have done the trick for most people who have run into this problem, including me. Also, are you seeing pip exit with an error message? If so, would you please attach the error and traceback for us?

Generated at Thu Feb 08 05:31:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.