[SERVER-45644] Reevaluate timeouts used by burn_in_test Created: 17/Jan/20  Updated: 29/Oct/23  Resolved: 24/Jan/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.2.4, 4.3.3

Type: Improvement Priority: Major - P3
Reporter: David Bradford (Inactive) Assignee: Alexander Costas (Inactive)
Resolution: Fixed Votes: 0
Labels: tig-burnin
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.2
Sprint: DAG 2020-01-27
Participants:
Story Points: 3

 Description   

In burn_in_tests, we will dynamically set timeouts based on the expected runtime of the test being run. There have, however, been some issues with the timeouts being used. We should investigate if we can improve the timeout calculations.


As a Server engineer,
I don't want my burn_in_tests runs to timeout if there are not issues,
So that I do not have to spend time investigating non-issues.


AC:

  • burn_in_timeouts are adjusted to generate less false-timeouts.


 Comments   
Comment by Githook User [ 10/Feb/20 ]

Author:

{'name': 'Alexander Costas', 'username': 'ancostas', 'email': 'alexander.costas@mongodb.com'}

Message: SERVER-45644: Reevaluate timeouts used by burn_in_test
Branch: v4.2
https://github.com/mongodb/mongo/commit/379dd0ca8eb89a94cbe373de05000bd88c999268

Comment by Githook User [ 24/Jan/20 ]

Author:

{'email': 'alexander.costas@mongodb.com', 'username': 'ancostas', 'name': 'Alexander Costas'}

Message: SERVER-45644: Reevaluate timeouts used by burn_in_test
Branch: master
https://github.com/mongodb/mongo/commit/aed4edcbf74f914fbcdd831937fa581aa436c5c6

Comment by David Bradford (Inactive) [ 23/Jan/20 ]

For the near term, we should probably just add a ~4 minute buffer to the exec timeout for test setup. It could be added to the value returned by calculate_exec_timeout

Comment by David Bradford (Inactive) [ 22/Jan/20 ]

I think this might explain what we are seeing .

When we set the timeouts in burn_in_tests, we are setting them based on the test execution time. However, the timeouts are for the entire task. We spend some amount of time downloading artifacts and doing setup. If this ends up being larger than the buffer we include in the timeout we will end up timing out.

We need someway of including the setup time in the timeouts. Since the setup time can be different across different build variants this could be tricky.

Comment by Lydia Stepanek (Inactive) [ 21/Jan/20 ]

Here is another patch in which burn_in_tests had some tests timeout: https://evergreen.mongodb.com/version/5e2749671e2d171abfe245dc

Generated at Thu Feb 08 05:09:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.