Tasks using an old base commit frequently timeout when the task runtime is significantly reduced

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DevProd Correctness
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When a task is made significantly quicker, any patch based on an older commit can frequently timeout. The timeout calculation always uses the latest runtimes.

      It is a simple workaround to rebase/merge the latest changes.

       

      For example, unittest_shell_hang_analyzer was significantly sped up by https://github.com/10gen/mongo/commit/640931caeab20de94945e8ce9fabb1bbaedfacd5#diff-0a36fcfdb73c149a6bfd9ca02c3d7d35493172639e211637fb1081accd1f192aL194-L199. Circa a week after this change, the calculated idle timeout dropped from 960 seconds to 300 seconds, lower than the typical runtime of the jstest in the task prior to the change. Patches with base commits prior to this change were almost guaranteed to fail:

      https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2023_arm64_static_compile_display_unittest_shell_hang_analyzer_patch_c283bc5f68a8450886fe4675e99bb04065dc3c17_68c081ebb8760e0007fb4fdb_25_09_09_19_38_21/tests

      https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2023_arm64_static_compile_unittest_shell_hang_analyzer_0_linux_enterprise_patch_09f45927889c643fb73f9d9397308e7618c4311a_68c0a2b83f77d50007014746_25_09_09_21_57_22/logs?execution=1

      Metrics on this task's times: https://ui.honeycomb.io/mongodb-4b/environments/production/datasets/evergreen-agent/result/b15a5fcDA91?vs=hideCompare&cs_0=omitMissingValues&cs_1=omitMissingValues

            Assignee:
            Unassigned
            Reporter:
            Sean Lyons
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: