[SERVER-49352] 4.0 does not build with python 3.8 Created: 08/Jul/20  Updated: 29/Oct/23  Resolved: 08/Aug/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.19
Fix Version/s: 3.6.20, 4.0.21

Type: Bug Priority: Major - P3
Reporter: Oleg Pudeyev (Inactive) Assignee: Ryan Egesdahl (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6
Sprint: Dev Platform 2020-08-10
Participants:

 Description   

When I run, in 4.0.19 tree,

  pip install --user -r buildscripts/requirements.txt

I receive the following error:

    Running setup.py install for typed-ast ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xvnr8otx/typed-ast/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xvnr8otx/typed-ast/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9z5dkcdx/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/build/.local/include/python3.8/typed-ast
         cwd: /tmp/pip-install-xvnr8otx/typed-ast/
    Complete output (23 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.8/typed_ast
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/ast27
    creating build/temp.linux-x86_64-3.8/ast27/Parser
    creating build/temp.linux-x86_64-3.8/ast27/Python
    creating build/temp.linux-x86_64-3.8/ast27/Custom
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iast27/Include -I/usr/include/python3.8 -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.8/ast27/Parser/acceler.o
    ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
       13 | #include "pgenheaders.h"
          |          ^~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

4.0 build transcript: https://gist.github.com/p-mongo/2804e19d6862d08f4192faa45ae1e159

I also get this error if I run the same command in 4.4.0-rc12 tree:

    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-262v9yio/typed-ast/setup.py'"'"'; __file__='"'"'/tmp/pip-install-262v9yio/typed-ast/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8z8zcirp/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/build/.local/include/python3.8/typed-ast
         cwd: /tmp/pip-install-262v9yio/typed-ast/
    Complete output (23 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.8/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.8/typed_ast
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/ast27
    creating build/temp.linux-x86_64-3.8/ast27/Parser
    creating build/temp.linux-x86_64-3.8/ast27/Python
    creating build/temp.linux-x86_64-3.8/ast27/Custom
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iast27/Include -I/usr/include/python3.8 -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.8/ast27/Parser/acceler.o
    ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
       13 | #include "pgenheaders.h"
          |          ^~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------

4.4 transcript: https://gist.github.com/p-mongo/f6748100d02e999d4315fd3609153809

The issue appears to be described here: https://github.com/python/typed_ast/issues/126

The solution appears to be updating to a newer version of typed_ast which appears to not be a direct dependency listed in build requirements.txt.

On 4.4 however, the instructions I have call for running instead the following which does work:

pip3 install --user -r etc/pip/compile-requirements.txt

So the issue seems to affect servers like 4.0 that do not have the etc/pip/compile-requirements.txt file, and whoever uses the buildscripts/requirements.txt file in newer servers which apparently isn't needed for building by users.



 Comments   
Comment by Githook User [ 10/Aug/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-49352 Clarify build documentation for Python 2.7

Older versions of MongoDB only build with Python 2.7.x and had a comment
that running `pip` would work to install modules. However, on newer
systems, a symlink named `pip2` may not exist even though Python 2.7.x
is installed. In such cases, users would install the modules for the
wrong Python version and become confused when the build failed. The build
documentation is clarified to use a different and more reliable way to
execute `pip` without accidentally executing the wrong Python version.

(cherry picked from commit 936ebad6b67325f068e2ae83c9e5c143778964ef)
Branch: v3.6
https://github.com/mongodb/mongo/commit/dd70c35c560dd939dd4a9320e3ac7a619237068b

Comment by Githook User [ 08/Aug/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-49352 Clarify build documentation for Python 2.7

Older versions of MongoDB only build with Python 2.7.x and had a comment
that running `pip` would work to install modules. However, on newer
systems, a symlink named `pip2` may not exist even though Python 2.7.x
is installed. In such cases, users would install the modules for the
wrong Python version and become confused when the build failed. The build
documentation is clarified to use a different and more reliable way to
execute `pip` without accidentally executing the wrong Python version.
Branch: v4.0
https://github.com/mongodb/mongo/commit/936ebad6b67325f068e2ae83c9e5c143778964ef

Comment by April Schoffer [ 30/Jul/20 ]

We'll make a documentation update to clarify the steps to run here.

Comment by Oleg Pudeyev (Inactive) [ 30/Jul/20 ]

My OS no longer ships a python2-using pip in its packages, but I was able to get past the issue by using a python 2.7 virtualenv. (https://github.com/p-mongo/dev/commit/35f077370397e170dd8bef769896a10a4f3481df)

docs/building.md currently contains this passage:

> $ pip2 install -r buildscripts/requirements.txt

> Note: If the `pip2` command is not available, `pip` without a suffix may be the pip command
associated with Python 2.7.x.

On my system there is no pip2 and pip invokes python 3.

I would suggest adding the following two notes to this section:

1. Explicitly stating that python 3 is not suitable for building the server. The document already explicitly states that python 2.7.x is required, but since this version requirement is apparently not enforced (e.g. by the requirements file itself) I think it won't hurt to explicitly state that none of the maintained python versions are usable.

2. Add a note suggesting using a python 2 virtualenv to install pip for modern operating systems that are retiring python 2 packages. Example proposed language:

> Note: If the `pip2` command is not available, `pip` without a suffix may be the pip command
associated with Python 2.7.x. On some systems, `pip` may only be available with Python 3; on systems like this, a viable workaround may be to create a virtualenv using Python 2.7 and install pip in this virtualenv.

Comment by Ryan Egesdahl (Inactive) [ 29/Jul/20 ]

You can give that a try, and please let me know here if there are any errors installing the modules that prevents you from building.

Comment by Oleg Pudeyev (Inactive) [ 29/Jul/20 ]

My pip apparently uses python 3:

serene% pip --version
pip 20.1.1 from /usr/lib/python3/dist-packages/pip (python 3.8)

Perhaps specifying `python2.7 -m pip ...` instead of `pip` would fix the issue, I can try this if desired.

Comment by Ryan Egesdahl (Inactive) [ 29/Jul/20 ]

I am confused. The 4.0 branch only builds with Python 2.7 right now, and you will get an error message if you try building with anything else. How are you encountering this problem?

Generated at Thu Feb 08 05:19:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.