[SERVER-37588] Python linters version incompatability Created: 12/Oct/18  Updated: 29/Oct/23  Resolved: 15/Oct/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.1.5

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: Benjamin Caimano (Inactive)
Resolution: Fixed Votes: 0
Labels: tig-pylint
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-37602 Upgrade yapf to 0.24.0 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Service Arch 2018-10-22
Participants:

 Description   

The work for SERVER-36152 introduced etc/pip/constraints.txt which defines the versions for the various pip components. There are some conflicts with the versions that are defined in the Python linters in buildscripts/linter/:

  • yapf 0.21.0
  • pylint 1.8.3
  • pydocstyle 2.1.1
  • mypy 0.580


 Comments   
Comment by Githook User [ 15/Oct/18 ]

Author:

{'name': 'Ben Caimano', 'email': 'ben.caimano@10gen.com'}

Message: SERVER-37588 Python linters version incompatability
Branch: master
https://github.com/mongodb/mongo/commit/afc927262eaa1ca4957a2fed5cfc7cb8c9e161f9

Comment by Max Hirschhorn [ 12/Oct/18 ]

Great - this means the ticket needs to be fixed before the Build team can roll out the new version of the toolchain because otherwise the lint task will start to fail.

Comment by Benjamin Caimano (Inactive) [ 12/Oct/18 ]

Apparently these checks fall back to toolchain locations without erroring. I completely missed that this was happening. ( jonathan.abrahams let me know about this Wednesday I think?) I really don't like this version checking pattern but it is what it is. We need to strictly fix the linter versions as long as it is so. I'd also have liked this to come up during the long code review period but it was a big review.

About potentially upgraded versions:

  • mypy should be locked at the existing version because there are some logic changes in idl that need to happen for it to pass the newer more thorough static checking.
  • Pylint is fine being upgraded.
  • Pydocstyle hasn't moved in a year.
  • Yapf did something odd with its function argument rules. I'm looking into it.

This also revealed a surprising and unfortunate interaction between virtualenv and pip upgrades. I'll probably bug you two offline about my proposed fix.

Comment by Jonathan Abrahams [ 12/Oct/18 ]

Per the comment in buildscripts/linter/runner.py, the linter is checked in different locations.

On rhel2-small, the toolchain has not yet been updated:

[root@ip-10-122-18-168 ~]# /opt/mongodbtoolchain/v2/bin/yapf --version
yapf 0.21.0
[root@ip-10-122-18-168 ~]# /opt/mongodbtoolchain/v2/bin/pylint --version
No config file found, using default configuration
pylint 1.8.3,
astroid 1.6.3
Python 3.6.0 (default, Jun  5 2018, 17:58:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
[root@ip-10-122-18-168 ~]# /opt/mongodbtoolchain/v2/bin/pydocstyle --version
2.1.1
[root@ip-10-122-18-168 ~]# /opt/mongodbtoolchain/v2/bin/mypy --version
mypy 0.580

Comment by Max Hirschhorn [ 12/Oct/18 ]

Eventually the working linter is found.

jonathan.abrahams, where are the working linters found if not in the virtual environment?

Comment by Jonathan Abrahams [ 12/Oct/18 ]

The version checks aren't failing, per se, it's the order in which the linters are searched. Eventually the working linter is found.

Comment by Max Hirschhorn [ 12/Oct/18 ]

It looks like all of the version checks are failing but the lint task continues to run regardless. CC ben.caimano

[2018/10/12 11:16:08.571] 2018-10-12 11:16:08,570 INFO ['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/yapf', '--version']
[2018/10/12 11:16:08.585] 2018-10-12 11:16:08,585 INFO Version check failed for [['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/yapf', '--version']], return code '2'.Standard Output:
[2018/10/12 11:16:08.585] Standard Error:
[2018/10/12 11:16:08.585] /data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2: can't open file '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/yapf': [Errno 2] No such file or directory
[2018/10/12 11:16:08.586] 2018-10-12 11:16:08,585 INFO Linter yapf has wrong version for '['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/yapf', '--version']'. Expected 'yapf\ 0\.21\.0',Standard Output:
[2018/10/12 11:16:08.586] ''
[2018/10/12 11:16:08.586] Standard Error:
[2018/10/12 11:16:08.586] /data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2: can't open file '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/yapf': [Errno 2] No such file or directory
[2018/10/12 11:16:08.586] 2018-10-12 11:16:08,585 INFO First version check failed for linter 'yapf', trying a different location.
[2018/10/12 11:16:08.595] 2018-10-12 11:16:08,586 INFO ['yapf', '--version']
[2018/10/12 11:16:08.595] 2018-10-12 11:16:08,594 INFO Version check command [['yapf', '--version']] failed: [Errno 2] No such file or directory
[2018/10/12 11:16:08.595] 2018-10-12 11:16:08,594 INFO ['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/opt/mongodbtoolchain/v2/bin/yapf', '--version']
[2018/10/12 11:16:08.757] 2018-10-12 11:16:08,756 INFO ['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pylint', '--version']
[2018/10/12 11:16:08.770] 2018-10-12 11:16:08,770 INFO Version check failed for [['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pylint', '--version']], return code '2'.Standard Output:
[2018/10/12 11:16:08.770] Standard Error:
[2018/10/12 11:16:08.771] /data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2: can't open file '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pylint': [Errno 2] No such file or directory
[2018/10/12 11:16:08.771] 2018-10-12 11:16:08,771 INFO Linter pylint has wrong version for '['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pylint', '--version']'. Expected 'pylint\ 1\.8\.3',Standard Output:
[2018/10/12 11:16:08.771] ''
[2018/10/12 11:16:08.771] Standard Error:
[2018/10/12 11:16:08.771] /data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2: can't open file '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pylint': [Errno 2] No such file or directory
[2018/10/12 11:16:08.771] 2018-10-12 11:16:08,771 INFO First version check failed for linter 'pylint', trying a different location.
[2018/10/12 11:16:08.777] 2018-10-12 11:16:08,771 INFO ['pylint', '--version']
[2018/10/12 11:16:08.777] 2018-10-12 11:16:08,777 INFO Version check command [['pylint', '--version']] failed: [Errno 2] No such file or directory
[2018/10/12 11:16:08.777] 2018-10-12 11:16:08,777 INFO ['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/opt/mongodbtoolchain/v2/bin/pylint', '--version']
[2018/10/12 11:16:09.412] 2018-10-12 11:16:09,412 INFO ['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pydocstyle', '--version']
[2018/10/12 11:16:09.426] 2018-10-12 11:16:09,426 INFO Version check failed for [['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pydocstyle', '--version']], return code '2'.Standard Output:
[2018/10/12 11:16:09.426] Standard Error:
[2018/10/12 11:16:09.426] /data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2: can't open file '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pydocstyle': [Errno 2] No such file or directory
[2018/10/12 11:16:09.427] 2018-10-12 11:16:09,426 INFO Linter pydocstyle has wrong version for '['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pydocstyle', '--version']'. Expected '2\.1\.1',Standard Output:
[2018/10/12 11:16:09.427] ''
[2018/10/12 11:16:09.427] Standard Error:
[2018/10/12 11:16:09.427] /data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2: can't open file '/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/pydocstyle': [Errno 2] No such file or directory
[2018/10/12 11:16:09.427] 2018-10-12 11:16:09,426 INFO First version check failed for linter 'pydocstyle', trying a different location.
[2018/10/12 11:16:09.433] 2018-10-12 11:16:09,427 INFO ['pydocstyle', '--version']
[2018/10/12 11:16:09.433] 2018-10-12 11:16:09,433 INFO Version check command [['pydocstyle', '--version']] failed: [Errno 2] No such file or directory
[2018/10/12 11:16:09.433] 2018-10-12 11:16:09,433 INFO ['/data/mci/497973c0ef906ae3f4a28aae017f12d0/venv/bin/python2', '/opt/mongodbtoolchain/v2/bin/pydocstyle', '--version']
[2018/10/12 11:16:40.094] 2018-10-12 11:16:09,557 INFO ['/opt/mongodbtoolchain/v2/bin/mypy', '--version']

https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_lint_ae2b60fce624cd9f4a95037316033b53846be176_18_10_12_11_10_58

Generated at Thu Feb 08 04:46:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.