[SERVER-76796] Upgrade python linters Created: 03/May/23  Updated: 29/Oct/23  Resolved: 17/Jul/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Alex Neben Assignee: Steve Gross
Resolution: Fixed Votes: 0
Labels: auto-reverted
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Participants:
Linked BF Score: 154

 Description   

etc/pip/components/lint.req

These linters are very out of date - we should try to upgrade them

 

You should also update https://wiki.corp.mongodb.com/display/HGTC/Remote+Development+with+Visual+Studio+Code+%28vscode%29+for+MongoDB as part of this ticket.



 Comments   
Comment by Githook User [ 16/Jul/23 ]

Author:

{'name': 'Steve Gross', 'email': 'steve.gross@mongodb.com', 'username': 'stevegrossmongodb'}

Message: SERVER-76796 Enable more lint warnings and fix them
Branch: master
https://github.com/mongodb/mongo/commit/babf1606540761a450c1d8f7770757c3f3361a48

Comment by xgen-buildbaron-user [ 13/Jul/23 ]

Ticket re-opened due to revert. buildscripts_test began a consistent failure of buildscripts\tests\test_exception_exctractor.py,buildscripts\tests\resmokelib\testing\fixtures\test_api_adherence.py

Comment by Githook User [ 13/Jul/23 ]

Author:

{'name': 'auto-revert-processor', 'email': 'dev-prod-dag@mongodb.com', 'username': ''}

Message: Revert "SERVER-76796 Enable more lint warnings and fix them"

This reverts commit 7e0887b55daf8cff8fd9552da38ce95d3faa73a9.
Branch: master
https://github.com/mongodb/mongo/commit/8837a2dfe2bf2859e481fa18ca22cda36a44f1ca

Comment by Githook User [ 13/Jul/23 ]

Author:

{'name': 'Steve Gross', 'email': 'steve.gross@mongodb.com', 'username': 'stevegrossmongodb'}

Message: SERVER-76796 Enable more lint warnings and fix them
Branch: master
https://github.com/mongodb/mongo/commit/7e0887b55daf8cff8fd9552da38ce95d3faa73a9

Comment by Githook User [ 12/Jul/23 ]

Author:

{'name': 'Steve Gross', 'email': 'steve.gross@mongodb.com', 'username': 'stevegrossmongodb'}

Message: SERVER-76796 Upgrade pylint and suppress all new warnings
Branch: master
https://github.com/mongodb/mongo/commit/691ec726448bdbe0b977cbcfac28ee3767e7335a

Comment by Steve Gross [ 03/Jul/23 ]

Closing out this issue; remaining linter upgrades are tracked here

Comment by Githook User [ 03/Jul/23 ]

Author:

{'name': 'stevegrossmongodb', 'email': '136485666+stevegrossmongodb@users.noreply.github.com', 'username': 'stevegrossmongodb'}

Message: SERVER-76796 Upgrade several linters
Branch: master
https://github.com/mongodb/mongo/commit/373aff474e1e408ce9cdec6c6222a0bd63865960

Comment by Steve Gross [ 27/Jun/23 ]

Per this guidance, I examined the "failure details" for the 3 failures. In each case, I see that the failure is likely UNRELATED to the change itself. So, I'll send the PR for review!

Comment by Steve Gross [ 27/Jun/23 ]

The latest PB shows only a few failures:

I think these are unrelated to my change. I'll consult my SDP peers for guidance to make sure.

Comment by Steve Gross [ 26/Jun/23 ]

Darn, a few more linter warnings. Fixed 'em up, retriggered PB

Comment by Steve Gross [ 26/Jun/23 ]

PB in flight

Comment by Steve Gross [ 26/Jun/23 ]

I agree that the path-of-least-resistance here is to not upgrade yapf, see if that passes PB, and get the PR merged. Then I can take a fresh look at the yapf upgrade and see if it's feasible. I'll do that.

Comment by Alex Neben [ 26/Jun/23 ]

I think this means there is a conflict between pydocstyle and yapf. pydocstyle for python 3.8 looks to depend on importlib-metadata<5.0.0 whereas yapf depends on importlib-metadata>=6.6.0. The solution is to either upgrade python on windows or to downgrade yapf till you find a version that does not require importlib-metadata>=5.0.0. I would choose the latter and we can always revisit if I can land the windows python upgrade.
[2023/06/23 19:38:04.413] 2023-06-23T20:38:03,483 The conflict is caused by:
 
[2023/06/23 19:38:04.413] 2023-06-23T20:38:03,483 pydocstyle 6.3.0 depends on importlib-metadata<5.0.0 and >=2.0.0; python_version < "3.8"
 
[2023/06/23 19:38:04.413] 2023-06-23T20:38:03,483 yapf 0.40.1 depends on importlib-metadata>=6.6.0
 

Comment by Steve Gross [ 23/Jun/23 ]

I'm seeing at least one failed task with an error related to this change:

ERROR: Cannot install -r C:\data\mci\824a1eecb4fedd4bcedcacb33c0f8cf6/src/etc/pip\components/lint.req (line 5) and -r C:\data\mci\824a1eecb4fedd4bcedcacb33c0f8cf6/src/etc/pip\components/lint.req (line 9) because these package versions have conflicting dependencies.

The lines in question are ({pydocstyle == 6.1.1}} and yapf == 0.26.0. I'll continue to investigate.

Note that I did successfully run python3 buildscripts/pylinters.py lint-all in my workspace, so there must be some problem not detected by simply running the linter.

Comment by Steve Gross [ 23/Jun/23 ]

The latest PB succeeded. Now I'll trigger a full PB.

Comment by Steve Gross [ 23/Jun/23 ]

Curiously, the PB failed (log). The relevant portion of the error log suggests that there is an incompatibility in structlog versions:

[2023/06/23 18:58:51.841] 2023-06-23T18:58:51,601 ERROR: Cannot install structlog~=19.2.0 and structlog~=23.1.0 because these package versions have conflicting dependencies.
[2023/06/23 18:58:51.841] 2023-06-23T18:58:51,602
[2023/06/23 18:58:51.841] 2023-06-23T18:58:51,602 The conflict is caused by:
[2023/06/23 18:58:51.841] 2023-06-23T18:58:51,602 The user requested structlog~=19.2.0
[2023/06/23 18:58:51.841] 2023-06-23T18:58:51,602 The user requested structlog~=23.1.0
{{}}

I see that evergreen.req also specifies a structlog version. I'll try amending that to 23.1.0 and retrigger the PB.

Comment by Steve Gross [ 23/Jun/23 ]

Finished most of the linter upgrade code, and set a PB running

Comment by Steve Gross [ 22/Jun/23 ]

Python linter is updated. Now I'll tackle the remaining ones.

Comment by Githook User [ 22/Jun/23 ]

Author:

{'name': 'Steve Gross', 'email': 'steve.gross@mongodb.com', 'username': 'stevegrossmongodb'}

Message: SERVER-76796 Upgrade python linter
Branch: master
https://github.com/mongodb/mongo/commit/4535a5b80155e507c7d5db436c81e748a9649e38

Comment by Steve Gross [ 21/Jun/23 ]

I'm taking a look at this request.

Generated at Thu Feb 08 06:33:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.