[SERVER-33696] Set timeout_secs for powercycle* task to restrict the duration of a single powercycle loop Created: 06/Mar/18 Updated: 29/Oct/23 Resolved: 09/Mar/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 3.6.4, 3.7.3 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Jonathan Abrahams |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | tig-powercycle | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Backport Requested: |
v3.6
|
||||||||||||||||||||||||||||
| Sprint: | TIG 2018-03-26 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Linked BF Score: | 19 | ||||||||||||||||||||||||||||
| Description |
|
We've occasionally seen NetworkTimeout and ServerSelectionTimeoutError exceptions when attempting to insert the canary document as a result of mongod having crashed or hung. If the mongod process is hung, it'd be beneficial to trigger the hang analyzer script. Given how the powertest.py script writes to its stdout and stderr, we should be able to set timeout_secs for the "run powercycle test" function in each of the powercycle* Evergreen tasks and have that correspond to the time limit for a single powercycle loop. We should raise the "serverSelectionTimeoutMS" and "socketTimeoutMS" parameters used when constructing a pymongo.MongoClient instance to be well above whatever we use for the timeout_secs value to ensure the script doesn't exit before Evergreen has a chance to trigger the hang analyzer.
|
| Comments |
| Comment by Githook User [ 23/Mar/18 ] |
|
Author: {'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}Message: (cherry picked from commit 3dbea83df2a71384097a42f0870037c8e364efff) |
| Comment by Githook User [ 09/Mar/18 ] |
|
Author: {'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}Message: |
| Comment by Max Hirschhorn [ 06/Mar/18 ] |
|
Without the changes from |