[SERVER-61304] Unable to use --use-system-{pcre,boost,...} with r5.1.0 Created: 07/Nov/21  Updated: 29/Oct/23  Resolved: 11/Nov/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.2.0, 5.1.1

Type: Bug Priority: Major - P3
Reporter: Nehal J Wani Assignee: Andrew Morrow (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.1
Sprint: Dev Platform 2021-11-15
Participants:

 Description   

Old behavior:

 

# Checkout to the last known good version
git reset --hard r5.0.3
 
# Build with --use-system-pcre
git clean -f -x -d && python buildscripts/scons.py --dbg=off --disable-warnings-as-errors --enable-free-mon=on --enable-http-client=on --opt=on --release --server-js=on --ssl=on --wiredtiger=on --ninja=enabled --use-system-pcre generate-ninja
 
# Check if the compile and link commands have -lpcre in them? Yes!
grep 'lpcre' build.ninja | wc -l
132

Current behavior:

 

# Checkout to the latest version
git reset --hard r5.1.0
# Build with --use-system-pcre
git clean -f -x -d && python buildscripts/scons.py --dbg=off --disable-warnings-as-errors --enable-free-mon=on --enable-http-client=on --opt=on --release --server-js=on --ssl=on --wiredtiger=on --ninja=enabled --use-system-pcre generate-ninja
# Check if the compile and link commands have -lpcre in them? No!
grep lpcre build.ninja | wc -l
0

A git bisect reveals that the following commit is the cause:

 

commit b7aeb64343266da73e29632ef96ebfad5df91495 (refs/bisect/bad)
Author: Andrew Morrow <acm@mongodb.com>
Date: Tue Aug 10 13:57:18 2021 -0400
SERVER-54487 Third-party shim libraries do not need to be linked to or installed

 

 



 Comments   
Comment by Andrew Morrow (Inactive) [ 19/Nov/21 ]

nehaljw.kkd1@gmail.com - The fix has been backported to the v5.1 branch. The next release from that branch will contain the fix.

Comment by Githook User [ 19/Nov/21 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-61304 Handle virtual libdeps by excluding them on the link line rather than at graph walk time

(cherry picked from commit b3eefe3ab3a9cf8cce299c1ad8664258b6c8190a)
Branch: v5.1
https://github.com/mongodb/mongo/commit/32ff83b883745b9fe7ec4de2369ce96a54d876fd

Comment by Andrew Morrow (Inactive) [ 11/Nov/21 ]

nehaljw.kkd1@gmail.com - This is now fixed on master and we will get it backported to the v5.1 branch after a bit of burn-in time. The fixVersion on this ticket will be updated to reflect the 5.1 release in which the fix will land.

Comment by Nehal J Wani [ 11/Nov/21 ]

I've tested this patch and it works for my use case.

Comment by Githook User [ 11/Nov/21 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-61304 Handle virtual libdeps by excluding them on the link line rather than at graph walk time
Branch: master
https://github.com/mongodb/mongo/commit/b3eefe3ab3a9cf8cce299c1ad8664258b6c8190a

Comment by Andrew Morrow (Inactive) [ 10/Nov/21 ]

nehaljw.kkd1@gmail.com - If you are interested, can you see if the following patch fixes the issue for you? We are still reviewing/validating so I wouldn't use this for anything other than testing, but it would be good to know if it appears to fix the issue for you.

 

diff --git a/site_scons/libdeps.py b/site_scons/libdeps.py
index 1b58934e58..aa29f1dd82 100644
--- a/site_scons/libdeps.py
+++ b/site_scons/libdeps.py
@@ -730,13 +730,7 @@ def _libdeps_visit(n, tsorted, marked, walking, debug=False):
                 _libdeps_visit_private(child, marked, walking, debug)
 
         marked[n.target_node] = LibdepsVisitationMark.MARKED_PUBLIC
-
-        # If the node has been marked as a virtual libdep in the tags,
-        # we don't add it to tsorted because it isn't a real
-        # dependency, just a node that adds further transitive
-        # dependencies.
-        if not 'virtual-libdep' in n.target_node.get_env().get('LIBDEPS_TAGS', []):
-            tsorted.append(n.target_node)
+        tsorted.append(n.target_node)
 
     except DependencyCycleError as e:
         if len(e.cycle_nodes) == 1 or e.cycle_nodes[0] != e.cycle_nodes[-1]:
@@ -903,13 +897,17 @@ def _append_direct_libdeps(node, prereq_nodes):
     node.attributes.libdeps_direct.extend(prereq_nodes)
 
 
-def _get_flagged_libdeps(source, target, env, for_signature):
+def _get_libdeps_with_link_flags(source, target, env, for_signature):
     for lib in get_libdeps(source, target, env, for_signature):
         # Make sure lib is a Node so we can get the env to check for flags.
         libnode = lib
         if not isinstance(lib, (str, SCons.Node.FS.File, SCons.Node.FS.Entry)):
             libnode = env.File(lib)
 
+        # Virtual libdeps don't appear on the link line
+        if 'virtual-libdep' in libnode.get_env().get('LIBDEPS_TAGS', []):
+            continue
+
         # Create a libdep and parse the prefix and postfix (and separators if any)
         # flags from the environment.
         cur_lib = FlaggedLibdep(libnode, env)
@@ -1131,7 +1129,7 @@ def expand_libdeps_tags(source, target, env, for_signature):
     return results
 
 
-def expand_libdeps_with_flags(source, target, env, for_signature):
+def expand_libdeps_for_link(source, target, env, for_signature):
 
     libdeps_with_flags = []
 
@@ -1140,7 +1138,7 @@ def expand_libdeps_with_flags(source, target, env, for_signature):
     # below a bit cleaner.
     prev_libdep = None
 
-    for flagged_libdep in _get_flagged_libdeps(source, target, env, for_signature):
+    for flagged_libdep in _get_libdeps_with_link_flags(source, target, env, for_signature):
 
         # If there are no flags to process we can move on to the next lib.
         # start_index wont mater in the case because if there are no flags
@@ -1404,11 +1402,11 @@ def setup_environment(env, emitting_shared=False, debug='off', linting='on'):
         PROGEMITTER=lambda target, source, env: env["LIBDEPS_PROGEMITTER"](target, source, env),
     )
 
-    env["_LIBDEPS_LIBS_WITH_TAGS"] = expand_libdeps_with_flags
+    env["_LIBDEPS_LIBS_FOR_LINK"] = expand_libdeps_for_link
 
     env["_LIBDEPS_LIBS"] = (
         "$LINK_LIBGROUP_START "
-        "$_LIBDEPS_LIBS_WITH_TAGS "
+        "$_LIBDEPS_LIBS_FOR_LINK "
         "$LINK_LIBGROUP_END "
     )

Comment by Andrew Morrow (Inactive) [ 09/Nov/21 ]

nehaljw.kkd1@gmail.com - I've reproduced this and have diagnosed why that change broke it. I have some ideas about how we might fix it, rather than reverting the virtual libdep feature.

Comment by Andrew Morrow (Inactive) [ 07/Nov/21 ]

nehaljw.kkd1@gmail.com - Thanks for the ticket, I'll follow up when I have more details.

Comment by Nehal J Wani [ 07/Nov/21 ]

To provide some context: I'm the maintainer of the MongoDB feedstock in the Conda Forge community. We try to use packages provided by the ecosystem instead of building them and linking them statically. I noticed lots of linker errors when building the r5.1.0 release, which get resolved on reverting the commit mentioned above.

Generated at Thu Feb 08 05:52:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.