In SERVER-44017, the "do setup" task was changed to unconditionally download the debug symbols at the setup for ever task, in furtherance of improvements to the hang analyzer.

This has the unfortunate effect of adding bandwidth use and test startup latency to every task, because the debug symbols are enormous.

Additionally, it means that we can't defer aggregation, archiving, and uploading of the debug symbols into a separate "post compile" phase, which was one of the long term aims of the hygienic builds project. Doing so would cut several minutes off the runtime of the "compile" phase, allowing all tests to start sooner, and run concurrently with the work to produce and upload the debugsymbols archive.

As most tests don't actually hang, we should revise this such that the debug symbols are only downloaded on demand. This would have been straightforward, by adding them to the timeout steps, or directly to the run hang analyzer task.

Unfortunately, the introduction of launching the hang analyzer from assert.soon in SERVER-26867 has prevented this strategy from being viable.

In order for us to be able to defer the production of the debug symbols and shorten the latency between compile starting and tests running, we need to find a way to make the hang analyzer itself download the debug symbols only when needed, rather than obtaining them unconditionally as part of setup.

Note that moving the production of debug symbols out of the 'compile' task, but allowing testing to proceed in parallel with the production of the debug symbols, means that a task that hangs very quickly, before the symbols are uploaded, won't be able to obtain them immediately. The script that the hang analyzer uses to obtain the symbols should incorporate a wait and backoff strategy so that it can obtain the symbols once they are ready.

CC robert.guo and ryan.timmons