[SERVER-42911] Rebuild mongodb by building.md and failed to build due to ModuleNotFoundError: No module named 'Cheetah' with MSVC on windows Created: 20/Aug/19  Updated: 29/Oct/23  Resolved: 03/Sep/19

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 3.6.15, 4.0.13, 3.4.23, 4.2.1, 4.3.1

Type: Bug Priority: Major - P3
Reporter: LinGao Assignee: Mathew Robinson (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

VS 2017 + Windows Server 2016 + MongoDB master branch latest revision


Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.2, v4.0, v3.6, v3.4
Sprint: Dev Tools 2019-08-26, Dev Tools 2019-09-09
Participants:

 Description   

We tried to rebuild mongodb by building.md, after executing the python.exe buildscripts/scons.py all -j4 command, the error will always be reported as below: 

C:\Python\Python37\python.exe src\mongo\base\generate_error_codes.py src\mongo\base\error_codes.yml src\mongo\base\error
_codes.tpl.h build\opt\mongo\base\error_codes.h
Traceback (most recent call last):
  File "src\mongo\base\generate_error_codes.py", line 31, in <module>
    from Cheetah.Template import Template
ModuleNotFoundError: No module named 'Cheetah'

 

And we checked the 'src\mongo\base\generate_error_codes.py' file, but I don't understand why this error was reported: ModuleNotFoundError: No module named 'Cheetah' . Could you please help to take a look at this issue? Is there any configuration missing? Thanks in advance!

 
Reproduce steps:
1. git clone -c core.autocrlf=true https://github.com/mongodb/mongo   F:\git_projects\mongodb\mongo
2. Open a VS 2017 x64 command prompt and browse to F:\git_projects\mongodb\mongo
3. pip.exe install --user -r .\etc\pip\dev-requirements.txt
4. python.exe buildscripts/scons.py all -j4

 

Note: 

F:\git_projects\mongodb\mongo> python --version
Python 3.7.4
Acutal result:

scons: Reading SConscript files ...
scons: running with args C:\Python\Python37\python.exe buildscripts/scons.py all -j4
scons version: 3.1.1
python version: 3 7 4 'final' 0
CC is cl
cl was not found in $PATH
cl resolves to F:\git_projects\mongodb\mongo\cl
CXX is $CC
$CC was not found in $PATH
$CC resolves to F:\git_projects\mongodb\mongo\$CC
Checking whether the C compiler works... (cached) yes
Checking whether the C++ compiler works... (cached) yes
Checking that the C++ compiler can link a C++ program... (cached) yes
Checking if C++ compiler "$CC" is MSVC... (cached) yes
Checking if C compiler "cl" is MSVC... (cached) yes
Detected a x86_64 processor
Checking if target OS windows is supported by the toolchain... (cached) yes
Checking if C compiler is Microsoft Visual Studio 2017 15.9 or newer...(cached) yes
Checking if C++ compiler is Microsoft Visual Studio 2017 15.9 or newer...(cached) yes
Checking if we are using libstdc++... (cached) no
Checking for C++17... (cached) yes
Checking for memset_s... (cached) no
Checking for C function strnlen()... (cached) yes
Checking Windows SDK is 8.1 or newer... (cached) yes
Checking if we are on a POSIX system... (cached) no
Checking for storage class thread_local (cached) yes
Checking for C++14 std::enable_if_t support...(cached) yes
Using SSL Provider: windows
Checking for C++ header file execinfo.h... (cached) no
Checking for C library pcap... (cached) no
Checking for C library wpcap... (cached) no
Checking if std::atomic<int64_t> works... (cached) yes
Checking if std::atomic<uint64_t> works... (cached) yes
Checking if std::atomic<int32_t> works... (cached) yes
Checking if std::atomic<uint32_t> works... (cached) yes
Checking for extended alignment 64 for concurrency types... (cached) yes
Checking for mongoc_get_major_version() in C library mongoc-1.0... (cached) no
Checking for C function fallocate()... (cached) no
Checking for C function sync_file_range()... (cached) no
Checking for C header file x86intrin.h... (cached) no
Checking for C header file arm_neon.h... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
C:\Python\Python37\python.exe buildscripts/idl/idlc.py --include src --base_dir build/opt --target_arch x86_64 --header
build\opt\mongo\db\repl\rollback_gen.h --output build\opt\mongo\db\repl\rollback_gen.cpp src\mongo\db\repl\rollback.idl
C:\Python\Python37\python.exe src\mongo\base\generate_error_codes.py src\mongo\base\error_codes.yml src\mongo\base\error
_codes.tpl.h build\opt\mongo\base\error_codes.h
Traceback (most recent call last):
  File "src\mongo\base\generate_error_codes.py", line 31, in <module>
    from Cheetah.Template import Template
ModuleNotFoundError: No module named 'Cheetah'
C:\Python\Python37\python.exe buildscripts/idl/idlc.py --include src --base_dir build/opt --target_arch x86_64 --header
build\opt\mongo\rpc\metadata\impersonated_user_metadata_gen.h --output build\opt\mongo\rpc\metadata\impersonated_user_me
tadata_gen.cpp src\mongo\rpc\metadata\impersonated_user_metadata.idl
C:\Python\Python37\python.exe buildscripts/idl/idlc.py --include src --base_dir build/opt --target_arch x86_64 --header
build\opt\mongo\db\repl\repl_settings_gen.h --output build\opt\mongo\db\repl\repl_settings_gen.cpp src\mongo\db\repl\rep
l_settings.idl
scons: *** [build\opt\mongo\base\error_codes.h] Error 1
scons: building terminated because of errors.
build\opt\mongo\base\error_codes.h failed: Error 1



 Comments   
Comment by Githook User [ 09/Sep/19 ]

Author:

{'username': 'chasinglogic', 'email': 'mathew.robinson@mongodb.com', 'name': 'Mathew Robinson'}

Message: SERVER-42911 Pass %APPDATA% into ENV for python subprocesses on Windows

(cherry picked from commit 89b3607e30edefe76ac84052cfe411ee1fda3b95)
Branch: v3.4
https://github.com/mongodb/mongo/commit/dd0bc4758718abc47050f0b91dfc777a54b67899

Comment by Githook User [ 09/Sep/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'mathew.robinson@mongodb.com'}

Message: SERVER-42911 Pass %APPDATA% into ENV for python subprocesses on Windows

(cherry picked from commit 89b3607e30edefe76ac84052cfe411ee1fda3b95)
Branch: v3.6
https://github.com/mongodb/mongo/commit/05cddb7914362e3986d5e20d7a37adca105503ac

Comment by Githook User [ 09/Sep/19 ]

Author:

{'username': 'chasinglogic', 'email': 'mathew.robinson@mongodb.com', 'name': 'Mathew Robinson'}

Message: SERVER-42911 Pass %APPDATA% into ENV for python subprocesses on Windows

(cherry picked from commit 89b3607e30edefe76ac84052cfe411ee1fda3b95)
Branch: v4.0
https://github.com/mongodb/mongo/commit/2eeba9bf12fa5c83f19ca36bc0d20914c22a25bb

Comment by Githook User [ 09/Sep/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'mathew.robinson@mongodb.com'}

Message: SERVER-42911 Pass %APPDATA% into ENV for python subprocesses on Windows

(cherry picked from commit 89b3607e30edefe76ac84052cfe411ee1fda3b95)
Branch: v4.2
https://github.com/mongodb/mongo/commit/8caf0e6c0dd80689bb79e02b977d0e76123285bc

Comment by Mathew Robinson (Inactive) [ 03/Sep/19 ]

spacelg I'm going to close this ticket, if you pull in the commit from above you should no longer experience this issue.

Feel free to reopen if the above commit does not resolve this for you.

Comment by Githook User [ 28/Aug/19 ]

Author:

{'email': 'mathew.robinson@mongodb.com', 'name': 'Mathew Robinson', 'username': 'chasinglogic'}

Message: SERVER-42911 Pass %APPDATA% into ENV for python subprocesses on Windows
Branch: master
https://github.com/mongodb/mongo/commit/89b3607e30edefe76ac84052cfe411ee1fda3b95

Comment by Mathew Robinson (Inactive) [ 28/Aug/19 ]

Hey spacelg,

Thanks for bringing this to our attention. This is actually a weird interaction between how SCons creates subprocesses, pip with the --user flag, and how Python does module lookups.

SCons does not, by design, automatically forward all parent environment variables down to subprocesses. This includes variables like %APPDATA% on WIndows which tells Python where to look for user site packages. Since you installed the requirements.txt (correctly) using the --user option it installed the packages into this user site-packages directory. This is why SCons was able to import those packages, like psutil, but when it ran python subprocesses like generate_error_codes.py they could not. I've made the change to our SConstruct to pass %APPDATA% down to subprocesses to fix this issue on Windows.

If you need to build an older branch or do not want to wait for this fix to land you can work around it by running your pip install command without the --user flag.

Thanks.

Comment by Kelsey Schubert [ 21/Aug/19 ]

Hi spacelg,

I've assigned this issue to the Developer Tools team to investigate — we'll look into it.

Kind regards,
Kelsey

Comment by LinGao [ 21/Aug/19 ]

Hi,

Is there anyone looking at this issue ? 

Thanks,

Lin

Generated at Thu Feb 08 05:01:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.