[SERVER-46744] unittest debug symbols not uploaded to evergreen Created: 10/Mar/20  Updated: 29/Oct/23  Resolved: 17/Apr/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.4.0-rc2, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Kevin Pulo Assignee: Andrew Morrow (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
Related
related to SERVER-49716 "gather_failed_unittests" does not wo... Closed
related to SERVER-45048 Cleanup artifacts.tgz and remove brid... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Dev Platform 2020-04-20
Participants:
Linked BF Score: 55

 Description   

SERVER-45048 added --separate-debug for unittests, and this can be seen in the unittest compilation logs, eg:

[2020/03/10 07:34:48.693] Generating debug info for build/cached/mongo/db/repl/db_repl_test into build/cached/mongo/db/repl/db_repl_test.debug
[2020/03/10 07:35:58.410] Stripping debug info from build/cached/mongo/db/repl/db_repl_test and adding .gnu.debuglink to build/cached/mongo/db/repl/db_repl_test.debug

However, "gather failed unittests" does not expect to have to collect separate foo.debug files, which means that the uploaded unittest files contain only the stripped binary, and not the (now separate) debug symbols. This isn't great for debugging, since the debug symbols are almost always needed for unittests.

Presumably the reason for adding --separate-debug, when switching to hygenic builds, was to avoid copying very large unittests binaries, and this concern is likely still valid. Therefore "gather failed unittests" should be updated to also grab the .debug files (which are likely still in the build tree, not the install tree).



 Comments   
Comment by Githook User [ 20/Apr/20 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-46744 Include mqrlun in dist-test
Branch: v4.4
https://github.com/10gen/mongo-enterprise-modules/commit/a7e80ea200f0af3aa01bdc05f8525989e93a2eeb

Comment by Githook User [ 20/Apr/20 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-46744 Install unit test debug info without increasing disk utilization

(cherry picked from commit cfa89fbaf0b397f07d8f9c884a04776224b4e918)
Branch: v4.4
https://github.com/mongodb/mongo/commit/ce0f3f551dd83dbd844beb8779a462b1df1fcbf6

Comment by Githook User [ 17/Apr/20 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-46744 Install unit test debug info without increasing disk utilization
Branch: master
https://github.com/mongodb/mongo/commit/cfa89fbaf0b397f07d8f9c884a04776224b4e918

Comment by Githook User [ 17/Apr/20 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-46744 Include mqrlun in dist-test
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/df5561f3331e415ae943044aa3758a06bc490aa1

Comment by Andrew Morrow (Inactive) [ 07/Apr/20 ]

kevin.pulo - The whole point of the hygienic builds program is to stop doing things like that, and let the build system ensure that the right files are in the right places, so I'd really prefer to leverage that, rather than make this already overly complex and fragile script worse.

Comment by Kevin Pulo [ 07/Apr/20 ]

Can we not just use find to find them in the build tree, and grab them from there? eg. something like this:

diff --git a/etc/evergreen.yml b/etc/evergreen.yml
index 5d5b2c26eb0..7294dc0a4fa 100644
--- a/etc/evergreen.yml
+++ b/etc/evergreen.yml
@@ -3472,16 +3472,21 @@ functions:
           fi
           for binary_file_location in $binary_file_locations
           do
-            new_binary_file=unittest_binaries/$(echo $binary_file_location | sed "s/.*\///")
+            binary_file=$(echo $binary_file_location | sed "s/.*\///")
+            new_binary_file=unittest_binaries/$binary_file
             if [ ! -f $new_binary_file ]; then
               mv $binary_file_location $new_binary_file
             fi
-            # On Windows if a .pdb symbol file exists, include it in the archive.
-            pdb_file=$(echo $binary_file_location | sed "s/\.exe/.pdb/")
-            if [ -f $pdb_file ]; then
-              new_pdb_file=unittest_binaries/$(echo $pdb_file | sed "s/.*\///")
-              mv $pdb_file $new_pdb_file
-            fi
+            # Include any corresponding .debug, .dSYM, or .pdb file from the build tree into the archive.
+            pdb_file=$(echo $binary_file | sed "s/\.exe/.pdb/")
+            debugsymbol_file_locations=$(/usr/bin/find -H build/ \( -name "$binary_file.debug" -o -name "$binary_file.dSYM" -o -name "$pdb_file" \) 2> /dev/null)
+            for debugsymbol_file_location in $debugsymbol_file_locations
+            do
+                new_debugsymbol_file=unittest_binaries/$(echo $debugsymbol_file_location | sed "s/.*\///")
+                if [ ! -f $new_debugsymbol_file ]; then
+                  mv $debugsymbol_file_location $new_debugsymbol_file
+                fi
+            done
           done
         done

Comment by Andrew Morrow (Inactive) [ 06/Apr/20 ]

Well, actually, it definitely doesn't work for PDBs because for the script to do anything you would need to run toolchain GDB, and that definitely doesn't exist on Windows. There is also no handling for dSYM.

Comment by Andrew Morrow (Inactive) [ 06/Apr/20 ]

And on some further reflection, I doubt this works for PDBs right now, since the execution directory will be build/install/bin, and since we haven't run install-unittests-debug, the PDB files won't have been copied up either.

Comment by Andrew Morrow (Inactive) [ 06/Apr/20 ]

Ah, I see. Actually we are just building install-unittests, not install-unittests-debug, which would be required to get the unit test debug symbols copied up to the install directory. The problem with this is that the debug information is massive, and this would effectively double the space required to build the unittests, which is already outrageous (hundreds of GB due to static linking). I will think a bit about what the right way to do this might look like.

Comment by Andrew Morrow (Inactive) [ 06/Apr/20 ]

kevin.pulo - Actually the debug files should be being copied into the install directory, so this as easy as updating that "gather failed unittests" script to also look for .debug files or .dSYM files.

Generated at Thu Feb 08 05:12:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.