Uploaded image for project: 'Evergreen'
  1. Evergreen
  2. EVG-759

A task and its host's running_task field will sometimes go out of sync

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1
    • Component/s: None
    • Labels:
      None

      Description

      Every once in a while, a task will run seemingly run without knowing what host it is running on. Similarly, its host will know that it ran the task, but will never know when the task finishes. This is pretty dangerous, as it locks up the host indefinitely.

      There are a few angles to this problem:
      1. If an agent start cmd times out when we try to launch it on a host, it could still actually start, as happened in the tasks attached to this ticket. We could try ensuring the agent is actually dead.
      2. We do nothing to ensure that two hosts aren't running the same task, it seems.


      Here are a few example pairings:
      https://evergreen.mongodb.com/host/sir-023gt8cx
      https://evergreen.mongodb.com/task/mongodb_mongo_master_linux_64_jsCore_auth_45d947729a0315accb6d4f15a6b06be6d9c19fe7_15_12_02_19_04_24

      https://evergreen.mongodb.com/host/sir-023kvd5t
      https://evergreen.mongodb.com/task/mongodb_mongo_master_linux_64_jsCore_auth_45d947729a0315accb6d4f15a6b06be6d9c19fe7_15_12_02_19_04_24

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kyle.erf Kyle Erf
                Reporter:
                kyle.erf Kyle Erf
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: