[SERVER-79122] Support building the GDB pretty printer tests using ninja Created: 19/Jul/23  Updated: 29/Oct/23  Resolved: 06/Sep/23

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Trevor Guidry
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Only supporting building the pretty_printer_test_launcher*.py scripts using SCons is inconvenient for local development because Server engineers are primarily compiling using ninja and the icecream cluster. The changes from b334631 as part of SERVER-77339 are misleading because the ninja target for install-pretty-printer-tests is defined but does not work as expected. Instead the build/install/pretty_printer_tests.txt file is left empty.

Moreover, while the install-pretty-printer-tests ninja target will generate a build/opt/mongo/util/pretty_printer_test_launcher_pretty_printer_test.py file, it won't install the file into the build/install/ directory. The install-pretty-printer-tests ninja target won't even generate the build/opt/mongo/db/concurrency/pretty_printer_test_launcher_lock_gdb_test.py or build/opt/mongo/db/query/optimizer/pretty_printer_test_launcher_optimizer_gdb_test.py files and so it isn't possible to build all of the GDB pretty printers testing code without running SCons.

We should make it simple for Server engineers to build and run the GDB pretty printer tests locally.



 Comments   
Comment by Githook User [ 06/Sep/23 ]

Author:

{'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com', 'username': ''}

Message: SERVER-79122 Support building the GDB pretty printer tests using ninja
Branch: master
https://github.com/mongodb/mongo/commit/33e99bb65cfaa116780b466a9191b09a8a9a7157

Comment by Alex Neben [ 22/Aug/23 ]

I think this is a great path forward. Especially, while we are in the middle of the scons transition. Thank you for doing this. We will take your diff, test it, and merge it in.

Comment by Max Hirschhorn [ 21/Aug/23 ]

I spent some time on my idea to create a new SCons target which builds only the non C++ targets. It works by removing test_program (== mongod) from the python program's dependency list when GDB_PPTEST_PYONLY=1 is specified.

ninja -f build.ninja -j 100 install-core install-dist-test
python buildscripts/scons.py --dbg=off --opt=on --link-model=dynamic --variables-files=./etc/scons/mongodbtoolchain_stable_gcc.vars GDB_PPTEST_PYONLY=1 install-pretty-printer-tests-pyonly
python buildscripts/resmoke.py run --suite=pretty-printer-tests

diff --git a/SConstruct b/SConstruct
index 1f75cd522e9..49a4edd43a4 100644
--- a/SConstruct
+++ b/SConstruct
@@ -1470,6 +1470,17 @@ env_vars.Add(
     default="auto",
 )
 
+env_vars.Add(
+    'GDB_PPTEST_PYONLY',
+    help='''Set the boolean (on/off true/false 1/0) to enable SCons to only emit the .py files
+    needed for testing GDB pretty printers.
+
+    Useful for when the executable files for testing GDB pretty printers are more rapidly built
+    using Ninja and the install-dist-test target (separately from SCons).''',
+    converter=functools.partial(bool_var_converter, var='GDB_PPTEST_PYONLY'),
+    default='False',
+)
+
 env_vars.Add(
     'ENABLE_OOM_RETRY',
     help=
diff --git a/site_scons/site_tools/mongo_pretty_printer_tests.py b/site_scons/site_tools/mongo_pretty_printer_tests.py
index b45e786a7ff..f5ecaff78d4 100644
--- a/site_scons/site_tools/mongo_pretty_printer_tests.py
+++ b/site_scons/site_tools/mongo_pretty_printer_tests.py
@@ -63,7 +63,7 @@ def build_pretty_printer_test(env, target, **kwargs):
         print_warning("Can't find gdb, not building pretty printer tests.")
         return []
 
-    test_component = {"dist-test"}
+    test_component = {"dist-test", "pretty-printer-tests-pyonly"}
 
     if "AIB_COMPONENTS_EXTRA" in kwargs:
         kwargs["AIB_COMPONENTS_EXTRA"] = set(kwargs["AIB_COMPONENTS_EXTRA"]).union(test_component)
@@ -153,8 +153,7 @@ def build_pretty_printer_test(env, target, **kwargs):
         AIB_COMPONENTS_EXTRA=kwargs["AIB_COMPONENTS_EXTRA"])
     env.Depends(
         pretty_printer_test_launcher[0],
-        [
-            test_program,
+        [test_program] if not env.get('GDB_PPTEST_PYONLY') else [] + [
             gen_test_script_install,
         ],
     )
@@ -196,7 +195,12 @@ def build_pretty_printer_test(env, target, **kwargs):
                            target_scanner=scanner)
     env.Pseudo(run_test)
     env.Alias('+' + os.path.splitext(os.path.basename(gdb_test_script))[0], run_test)
-    env.Depends(pretty_printer_test_launcher_install, [gen_test_script_install, test_program])
+    env.Depends(
+        pretty_printer_test_launcher_install,
+        [test_program] if not env.get('GDB_PPTEST_PYONLY') else [] + [
+            gen_test_script_install,
+        ],
+    )
 
     env.RegisterTest('$PRETTY_PRINTER_TEST_LIST', pretty_printer_test_launcher_install[0])
     env.Alias("$PRETTY_PRINTER_TEST_ALIAS", pretty_printer_test_launcher_install[0])

Comment by Steve Gross [ 10/Aug/23 ]

Per conversation w/ max.hirschhorn@mongodb.com : We agree that this problem warrants attention, and that the Bazel adoption is sufficiently far off in the calendar that we should find a way to address this sooner. However, we're staffing-constrained (that is, we have a variety of other projects on the docket), so it's difficult to figure out where this fits in. With that said, max.hirschhorn@mongodb.com agreed to investigate the issue and make a recommendation about how it might be solved. max.hirschhorn@mongodb.com , have I got that right?

Comment by Daniel Moody [ 19/Jul/23 ]

ninja can not install the pretty printer test files correctly, so it does not. The problem is a chmod command, which the input and output are the same file, which ninja does not support.

Also ninja can not support it because of the use of SCons substfile which our ninja tool does not support.

In spite of these issues we could workaround it with enough effort.

Generated at Thu Feb 08 06:40:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.