Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33696

Set timeout_secs for powercycle* task to restrict the duration of a single powercycle loop

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Backport Requested:
      v3.6
    • Sprint:
      TIG 2018-03-26
    • Linked BF Score:
      19

      Description

      We've occasionally seen NetworkTimeout and ServerSelectionTimeoutError exceptions when attempting to insert the canary document as a result of mongod having crashed or hung. If the mongod process is hung, it'd be beneficial to trigger the hang analyzer script. Given how the powertest.py script writes to its stdout and stderr, we should be able to set timeout_secs for the "run powercycle test" function in each of the powercycle* Evergreen tasks and have that correspond to the time limit for a single powercycle loop.

      We should raise the "serverSelectionTimeoutMS" and "socketTimeoutMS" parameters used when constructing a pymongo.MongoClient instance to be well above whatever we use for the timeout_secs value to ensure the script doesn't exit before Evergreen has a chance to trigger the hang analyzer.

      # Set the serverSelectionTimeoutMS & socketTimeoutMS to 10 minutes
      mongo_args = {
          "serverSelectionTimeoutMS": 600000,
          "socketTimeoutMS": 600000
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: