Currently we build many different benchmark binaries. Considering that benchmarks binaries are generally going to be static to improve and match release configurations, this means the debug info for these numerous static binaries is going to be significant. As a result, each benchmark binary itself is large (2+G).

Our current build process tars up all those binaries into a single tarball, which is getting pretty close to S3's current file size limit of 50G. This results in occasional failures of tasks due to exceeding S3 limitations.

We should explore options for how to remediate the situation. Options thus far discussed include:

1. Refactor the benchmark binary build process to instead build a single "collection" binary of all benchmarks. This would eliminate the duplication-of-libraries issue, which should theoretically reduce total tarball size.
2. Eliminate some benchmark binaries entirely
3. Stop including debug info in the benchmark binaries by default (which will significantly reduce the tarball size). Additionally, provide instructions to engineers on how to re-create debug info when needed. (This is possible currently, but requires expert knowledge).
4. Shard the benchmark tarball itself into multiple tarballs (and update all call sites to obtain N tarballs accordingly).

Note: There's a thread (https://mongodb.slack.com/archives/C0V896UV8/p1713886468742869 ) discussing what to do about this.