[SERVER-76751] Add poetry as a way to pin python deps in mongo master Created: 02/May/23  Updated: 17/Jan/24  Resolved: 05/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Alex Neben Assignee: Alex Neben
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt Dependency
has to be done after SERVER-72262 Update minimum python support to pyth... Closed
Problem/Incident
causes SERVER-81122 Investigate bug in which pretty print... Open
Related
is related to SERVER-70711 Improve performance of venv setup Open
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Participants:
Linked BF Score: 35

 Description   

https://github.com/pantsbuild/pex

 

We are consistently having problems like BF-29350 that are due to unpinned python dependencies. We should use poetry lock file to pin all our python dependencies.



 Comments   
Comment by Alex Neben [ 15/Sep/23 ]

There is no plans to backport this as the lift required would be enormous . Thank you for the ping in case this props up elsewhere. I am surprised that running more parallel fails? Does that mean we have a bug in our dependency graph or something?

Comment by Simon Eismann [ 15/Sep/23 ]

alex.neben@mongodb.com FYI: The switch to poetry ended up causing some intermittent failures in sys-perf as it parallelizes the package installation based on the number of cores of the underlying instance (BF-29897). We have fixed this on sys-perf by limiting the number of workers used by poetry, but I wanted to raise this in case this issue crops up on other waterfalls as well.

On a side note: Are there plans to backport this? Keeping the compile identical across all branches seems preferable.

Comment by Githook User [ 05/Sep/23 ]

Author:

{'name': 'Alex Neben', 'email': 'alex.neben@mongodb.com', 'username': 'IamXander'}

Message: SERVER-76751 Pinning all python deps with poetry
Branch: master
https://github.com/mongodb/mongo/commit/13f208e8c4efcce662da423088241a4663a5f6f1

Comment by Alex Neben [ 19/Jun/23 ]

Looks like poetry might have a bug when trying to export all pip dependencies. When trying to run "poetry export --output="requirements.txt" --format="requirements.txt" --without-hashes" it gets stuck on pip libraries that have multiple versions and are separated by markers.

Comment by Alex Neben [ 19/Jun/23 ]

See https://github.com/10gen/mongo/pull/new/IamXander/pex_rewrite for full example. Making a smaller version to get pinning before we take a bigger bite if this problem.

Generated at Thu Feb 08 06:33:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.