[SERVER-33696] Set timeout_secs for powercycle* task to restrict the duration of a single powercycle loop Created: 06/Mar/18  Updated: 29/Oct/23  Resolved: 09/Mar/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.6.4, 3.7.3

Type: Improvement Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Fixed Votes: 0
Labels: tig-powercycle
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Gantt Dependency
has to be done after SERVER-33691 Remote EC2 instances should have a lo... Closed
Problem/Incident
causes SERVER-34412 powercycle* tasks set timeout_secs in... Closed
Related
related to SERVER-34318 Set timeout for canary insert to 1 ho... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6
Sprint: TIG 2018-03-26
Participants:
Linked BF Score: 19

 Description   

We've occasionally seen NetworkTimeout and ServerSelectionTimeoutError exceptions when attempting to insert the canary document as a result of mongod having crashed or hung. If the mongod process is hung, it'd be beneficial to trigger the hang analyzer script. Given how the powertest.py script writes to its stdout and stderr, we should be able to set timeout_secs for the "run powercycle test" function in each of the powercycle* Evergreen tasks and have that correspond to the time limit for a single powercycle loop.

We should raise the "serverSelectionTimeoutMS" and "socketTimeoutMS" parameters used when constructing a pymongo.MongoClient instance to be well above whatever we use for the timeout_secs value to ensure the script doesn't exit before Evergreen has a chance to trigger the hang analyzer.

# Set the serverSelectionTimeoutMS & socketTimeoutMS to 10 minutes
mongo_args = {
    "serverSelectionTimeoutMS": 600000,
    "socketTimeoutMS": 600000
}



 Comments   
Comment by Githook User [ 23/Mar/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}

Message: SERVER-33696 Set timeout_secs for powercycle* task to restrict the duration of a single powercycle loop

(cherry picked from commit 3dbea83df2a71384097a42f0870037c8e364efff)
Branch: v3.6
https://github.com/mongodb/mongo/commit/4bb6f1da8c89567c6038b3390e7b034fcbaf18d0

Comment by Githook User [ 09/Mar/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}

Message: SERVER-33696 Set timeout_secs for powercycle* task to restrict the duration of a single powercycle loop
Branch: master
https://github.com/mongodb/mongo/commit/3dbea83df2a71384097a42f0870037c8e364efff

Comment by Max Hirschhorn [ 06/Mar/18 ]

Without the changes from SERVER-33691, the "timeout" section of the etc/evergreen.yml project configuration file will fail to copy across the debug symbols.

Generated at Thu Feb 08 04:34:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.