Uploaded image for project: 'Evergreen'
  1. Evergreen
  2. EVG-6314

Timeout phase in Evergreen appears to run with stale values for expansions

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • None
    • v2019.11.01
    • app

    Description

      https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_rollback_fuzzer_unclean_shutdowns_30_enterprise_rhel_62_64_bit_patch_a368e90685fdaa5d855ecd54769f5ca0c09d4378_5d04308d850e6160f69b880e_19_06_14_23_46_06/6 was part of a patch build which had bypassed compile (due to the compile task having succeeded in the base commit) but had attempted to download the debug symbols as if the compile task wasn't bypassed when the task timed out. This led to the hang_analyzer.py script running without having the debug symbols available, which severely limits the utility of its output.

      [2019/06/15 23:57:30.785] Running command 'expansions.update' in "set task expansion macros" (step 3 of 4)
      ...
      [2019/06/15 23:57:30.795] Finished running pre-task commands.
      [2019/06/15 23:57:30.796] Running command 's3.get' in "do setup" (step 1.1 of 4)
      [2019/06/15 23:57:30.822] fetching mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/artifacts/mongodb_mongo_master_enterprise_rhel_62_64_bit_patch_a368e90685fdaa5d855ecd54769f5ca0c09d4378_5d04308d850e6160f69b880e_19_06_14_23_46_06.tgz from s3 bucket mciuploads (attempt 1 of 10)
      [2019/06/15 23:57:33.426] Running command 'expansions.update' in "do setup" (step 1.2 of 4)
      [2019/06/15 23:57:33.426] Updating expansions with keys from file: /data/mci/89c9d9b73d96b2d10839a88ae9ef2993/src/bypass_compile_expansions.yml
      [2019/06/15 23:57:33.426] Running command 's3.get' in "do setup" (step 1.3 of 4)
      [2019/06/15 23:57:33.455] fetching mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/binaries/mongo-mongodb_mongo_master_enterprise_rhel_62_64_bit_a368e90685fdaa5d855ecd54769f5ca0c09d4378_19_06_14_21_59_58.tgz from s3 bucket mciuploads (attempt 1 of 10)
      ...
      [2019/06/16 00:28:27.178] Running task-timeout commands.
      [2019/06/16 00:28:27.178] Command stopped early: context canceled
      ...
      [2019/06/16 00:28:27.369] Running command 's3.get' in "fetch debugsymbols archive" (step 1 of 5)
      [2019/06/16 00:28:27.401] fetching mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/debugsymbols/debugsymbols-mongodb_mongo_master_enterprise_rhel_62_64_bit_patch_a368e90685fdaa5d855ecd54769f5ca0c09d4378_5d04308d850e6160f69b880e_19_06_14_23_46_06.tgz from s3 bucket mciuploads (attempt 1 of 10)
      ...
      [2019/06/16 00:29:38.973] fetching mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/debugsymbols/debugsymbols-mongodb_mongo_master_enterprise_rhel_62_64_bit_patch_a368e90685fdaa5d855ecd54769f5ca0c09d4378_5d04308d850e6160f69b880e_19_06_14_23_46_06.tgz from s3 bucket mciuploads (attempt 10 of 10)
      [2019/06/16 00:29:54.992] Command failed: S3 get failed after 10 attempts
      

      The bypass_compile_expansions.yml file in the artifacts tarball of the associated compile task has an entry for mongo_debugsymbols which is the S3 location getting fetched.

      bypass_compile_expansions.yml

      bypass_compile: true
      mongo_binaries: mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/binaries/mongo-mongodb_mongo_master_enterprise_rhel_62_64_bit_a368e90685fdaa5d855ecd54769f5ca0c09d4378_19_06_14_21_59_58.tgz
      mongo_debugsymbols: mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/debugsymbols/debugsymbols-mongodb_mongo_master_enterprise_rhel_62_64_bit_a368e90685fdaa5d855ecd54769f5ca0c09d4378_19_06_14_21_59_58.tgz
      mongo_shell: mongodb-mongo-master/enterprise-rhel-62-64-bit/a368e90685fdaa5d855ecd54769f5ca0c09d4378/binaries/mongo-shell-mongodb_mongo_master_enterprise_rhel_62_64_bit_a368e90685fdaa5d855ecd54769f5ca0c09d4378_19_06_14_21_59_58.tgz
      

      Attachments

        Issue Links

          Activity

            People

              backlog-server-evg Backlog - Evergreen Team
              max.hirschhorn@mongodb.com Max Hirschhorn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: