[SERVER-42787] 4.2.0 regression (vs 4.0.12): Build failure with various system packages Created: 13/Aug/19  Updated: 29/Oct/23  Resolved: 14/Aug/19

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 4.2.0
Fix Version/s: 4.2.1, 4.3.1

Type: Bug Priority: Major - P3
Reporter: James Harvey Assignee: Andrew Morrow (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2
Steps To Reproduce:

Have some or all of the mentioned system packages installed, and give scons the argument(s) telling it to use them instead of bringing in their source trees and building them as part of mongoDB.

Sprint: Dev Tools 2019-08-26
Participants:

 Description   

As of 4.0.12, it was possible to build mongoDB using system packages of libstemmer 0+337, pcre 8.43, wiredtiger 48bf8d (the commit specified in the mongodb git repo for this version), yaml-cpp 0.6.2, snappy 1.1.7, and zlib 1.2.11.  This was done by giving scons the arguments: --use-system-pcre --use-system-snappy --use-system-yaml --use-system-zlib --use-system-wiredtiger --use-system-stemmer.

As of 4.2.0, each of these now fail with errors like that given below.  Is this an oversight, or is it intentional that 4.2.0 no longer works with these system packages?

(For some time, system package versions of asio, icu, boost, and mongo-c haven't worked, and some of these have their own tickets open with answers of mongoDB having required custom patches in these.)

 

scons: Reading SConscript files ...
scons: warning: EnsureSConsVersion is ignored for development version
File "/build/mongodb/src/mongodb-src-r4.2.0/SConstruct", line 31, in <module>
scons: running with args /usr/bin/python /usr/bin/scons core --use-system-pcre --use-sasl-client --ssl --disable-warnings-as-errors --use-system-tcmalloc
...
Checking for C function fallocate()... yes
Checking for C function sync_file_range()... yes
Checking for C header file x86intrin.h... yes
Checking for C header file arm_neon.h... no
KeyError: 'pcre':
 File "/build/mongodb/src/mongodb-src-r4.2.0/SConstruct", line 3901:
 variant_dir='$BUILD_DIR',
 File "/usr/lib/python3.7/site-packages/SCons/Script/SConscript.py", line 605:
 return _SConscript(self.fs, *files, **subst_kw)
 File "/usr/lib/python3.7/site-packages/SCons/Script/SConscript.py", line 286:
 exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
 File "/build/mongodb/src/mongodb-src-r4.2.0/src/SConscript", line 23:
 'safeint',
 File "/usr/lib/python3.7/site-packages/SCons/Environment.py", line 224:
 return self.method(*nargs, **kwargs)
 File "/build/mongodb/src/mongodb-src-r4.2.0/src/third_party/SConscript", line 191:
 mods = thirdPartyEnvironmentModifications[lib]

 



 Comments   
Comment by Andrew Morrow (Inactive) [ 16/Aug/19 ]

jamespharvey20@gmail.com - The fix has been backported and will be available in 4.2.1. If you would like to apply the above commit locally in your build of 4.2.0 it should work fine until 4.2.1 is released.

Comment by Githook User [ 16/Aug/19 ]

Author:

{'email': 'andrew.morrow@10gen.com', 'name': 'Andrew Morrow'}

Message: SERVER-42787 Handle missing third party modifications more gracefully

(cherry picked from commit dc1afe419ba54d012e20a952fd565072b150e6ba)
Branch: v4.2
https://github.com/mongodb/mongo/commit/18ce5c9f56a01f2c933a720d64707f3456f81c8b

Comment by Andrew Morrow (Inactive) [ 14/Aug/19 ]

jamespharvey20@gmail.com - This has been fixed on master and will get backported to the v4.2 branch soon.

Comment by Githook User [ 14/Aug/19 ]

Author:

{'name': 'Andrew Morrow', 'email': 'andrew.morrow@10gen.com'}

Message: SERVER-42787 Handle missing third party modifications more gracefully
Branch: master
https://github.com/mongodb/mongo/commit/dc1afe419ba54d012e20a952fd565072b150e6ba

Comment by James Harvey [ 13/Aug/19 ]

Andrew Morris - Yes, thanks, your patch resolves the original issue I posted, of the initial configuration giving an error of "KeyError".  Your patch allows using all of the "--use-system-X" flags I mentioned.  I run into a much later compilation error with "--use-system-wiredtiger" which should be completely unrelated to this issue, so I'm creating a new issue for that.  But, this issue seems resolved.

EDIT: Apparently your system interprets the two "--" prefixes I typed as a section to strikethrough. The visual preview before posting didn't show it this way. Not sure how to prevent it from doing that. Using noformat makes it all appear on one line, so I'm not using that.

Comment by Andrew Morrow (Inactive) [ 13/Aug/19 ]

jamespharvey20@gmail.com - Could you give the following patch a quick test and see if it resolves the issue for you:

diff --git a/src/third_party/SConscript b/src/third_party/SConscript
index 5c0c129b30..6531300fee 100644
--- a/src/third_party/SConscript
+++ b/src/third_party/SConscript
@@ -206,7 +206,9 @@ def injectThirdParty(thisEnv, libraries=[], parts=[]):
     libraries = thisEnv.Flatten([libraries])
     parts = thisEnv.Flatten([parts])
     for lib in libraries:
-        mods = thirdPartyEnvironmentModifications[lib]
+        mods = thirdPartyEnvironmentModifications.get(lib, None)
+        if not mods:
+            continue
         if not parts:
             thisEnv.PrependUnique(**mods)
         else:

Comment by Andrew Morrow (Inactive) [ 13/Aug/19 ]

jamespharvey20@gmail.com - Thanks for the bug report. This is almost certainly a regression introduced during 4.2 development. We don't test those flags ourselves, though we arguably should, because none of the builds that we ship actually use them. I've wanted for a while to overhaul this part of the build system so it would actually do smart things like enforce minimum versions when selecting system packages, and require some sort of an override flag to enable the use-system options for packages that we have modified in some way. But for now, support is best effort. We will take a look at getting this fixed for 4.2.1.

Generated at Thu Feb 08 05:01:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.