Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33696

Set timeout_secs for powercycle* task to restrict the duration of a single powercycle loop

    XMLWordPrintableJSON

Details

    • Fully Compatible
    • v3.6
    • TIG 2018-03-26
    • 19

    Description

      We've occasionally seen NetworkTimeout and ServerSelectionTimeoutError exceptions when attempting to insert the canary document as a result of mongod having crashed or hung. If the mongod process is hung, it'd be beneficial to trigger the hang analyzer script. Given how the powertest.py script writes to its stdout and stderr, we should be able to set timeout_secs for the "run powercycle test" function in each of the powercycle* Evergreen tasks and have that correspond to the time limit for a single powercycle loop.

      We should raise the "serverSelectionTimeoutMS" and "socketTimeoutMS" parameters used when constructing a pymongo.MongoClient instance to be well above whatever we use for the timeout_secs value to ensure the script doesn't exit before Evergreen has a chance to trigger the hang analyzer.

      # Set the serverSelectionTimeoutMS & socketTimeoutMS to 10 minutes
      mongo_args = {
          "serverSelectionTimeoutMS": 600000,
          "socketTimeoutMS": 600000
      }
      

      Attachments

        Issue Links

          Activity

            People

              jonathan.abrahams Jonathan Abrahams
              max.hirschhorn@mongodb.com Max Hirschhorn
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: