[SERVER-63827] Perform dynamic timeout calculation at the start of task execution Created: 18/Feb/22  Updated: 29/Oct/23  Resolved: 04/Mar/22

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: Improvement Priority: Major - P3
Reporter: David Bradford (Inactive) Assignee: David Bradford (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: DAG 2022-03-07
Participants:
Story Points: 3

 Description   

Currently, when generating tasks, we will calculate and set a dynamic timeouts value based on the historic runtimes of the tests in each sub-suite (as long as we have runtime information for all the tasks). The dynamic runtimes are useful in keeping the timeouts fairly close to the expected runtimes. This way if a hang occurs, we can wait less time for the task to hit the timeout and hopefully have smaller logs we need to dig through to find the problem.

Two of the big drawbacks to this approach are: (1) it only works for generated tasks, tasks that have not been generated cannot take advantage of it, and (2) the generated tasks cannot be shared between build variants, since runtimes on each build variant can be different, they each need to generate their own timeouts.

To overcome these limitation, we should move the calculation of the timeouts to the start of task execution, instead of it being part of generated tasks. The runtime of calculating the runtime of a single task should be relatively small (a few seconds), so the impact on the overall task runtime should be negligible. This approach would, however, provide a consistent way of calculating timeouts for both generated and non-generated tasks and allow generated tasks to be shared across build variants.



 Comments   
Comment by Githook User [ 04/Mar/22 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-63827: Determine task timeout at the start of task execution
Branch: master
https://github.com/mongodb/mongo/commit/0a079df588d9d9ecdfd6898a74b65526bcb339a6

Comment by Githook User [ 02/Mar/22 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: Revert "SERVER-63827: Determine task timeout at the start of task execution"

This reverts commit 4080bd69384735eb49eb1799734d1bf8c93feb09.
Branch: master
https://github.com/mongodb/mongo/commit/d883e3e0c216f10f7a1368bdd9d59316c32d2cd6

Comment by Githook User [ 02/Mar/22 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-63827: Determine task timeout at the start of task execution
Branch: master
https://github.com/mongodb/mongo/commit/4080bd69384735eb49eb1799734d1bf8c93feb09

Generated at Thu Feb 08 05:58:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.