Switch remote_download_regex to include object files

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Critical - P2
    • 8.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • DevProd Build
    • Fully Compatible
    • None
    • 1
    • Hour
    • None
    • None
    • None
    • None
    • None
    • None

      The current filter forces all of the downloads to the end rather than interleaving them throughout the build, which ends up bottlenecking the link step:

      from mathias@mongodb.com :

      I noticed something interesting. This may be particular to my workflow where I am running bazel locally on a network with ~400mbps bandwidth, but it may affect others as well. I noticed that I frequently ended up doing a lot of downloading at the end of the build when the link was trying to start with a fairly idle-ish (or spikey) network usage throughout the earlier phases of the build. My hypothosis is that rather than eagerly fetching any files that bazel determines it will eventually need, it instead lazily waits until a task actually needs it. That means that it is waiting to fetch at least some .o files until the link needs them, rather than taking advantage of any available network capacity to try to fetch the files as soon as the compiles that produce them finish. I tried another large build with --remote_download_all and the middle parts of the build consumed more networking bandwidth, and the link started basically immediately after the last compile finished. I think that confirms the hypothosis. To me, that implies that we should strongly consider using --remote_download_all for any build that does linking locally. Alternatively, we should add .o (and other platform equivalents) to our --remote_download_regex. I haven't tested if that flag has the same eager-download behavior, but I suspect it does since my theory is that the network activity seen during the compile phase of the build is fetching the .dwo files. 

      Fixing this should greatly speed up the first link during a large re-build or first build

            Assignee:
            Andrew Bradshaw
            Reporter:
            Zack Winter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: