Context here.

Benchmarks are built with separate-debug=on and then the debug symbols are lost. We should turn it off or upload the symbols at least for the failing executables.

cc steve.gross@mongodb.com since the assignee recommended is labeled [DO NOT ASSIGN].

The tarball for this task is somewhat incorrectly created. The location of the debug symbols will not be useable in the install directory. The compile_upload_benchmarks task should be updated to also build the the install-benchmarks-debug target which will move the .debug files to the correct location in the install dir. Also instead of the task taring it up itself, the task could just use the archive targets scons provides (archive-benchmarks archive-benchmarks-debug)(this also leverage the faster pigz usage we implemented in scons recently).

Regarding the size of the tarball, if we switch to dynamic link, it will be much smaller, but since benchmarks imply perf, dynamic link may not be an option, as dynamic linking can reduce perf. Aside from dynamic link, the other option would be binary consolidation. Instead of build N number of benchmark binaries, combine the benchmark tests into a single benchmark binary (or just a 2-3 binaries).

Binary consolidation may require some tweaks to the c++ code and structure and is a larger undertaking, but certainly possible (and has been done before with several unittests). Reducing the size of the tarball has been split out to SERVER-86426.