[SERVER-32121] resmoke.py should cause the Evergreen task to system fail if the EC2 instance is terminated Created: 30/Nov/17  Updated: 30/Oct/23  Resolved: 11/Dec/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.7.1

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Kevin Albertson
Resolution: Fixed Votes: 0
Labels: tig-evgconfig, tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-35472 resmoke.py shouldn't fall back to std... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-12-18
Participants:

 Description   

The changes from EVG-1222 added an event to the "host logs" to indicate if the EC2 instance was terminated. This has been known to happen (most often on the Windows 2008R2 DEBUG builder) due to increases in the spot price. When the EC2 instance is terminated, a CTRL_SHUTDOWN_EVENT is sent to the mongod processes causing them to exit and for tests communicting with those servers to fail. Additionally, an IOError "Interrupted function call" exception is raised to indicate that waiting for the test queue to become empty was interrupted. This leads to a race where (1) resmoke.py may exit with a nonzero code and (2) the Evergreen agent runs the attach.results command to indicate the Evergreen task has failed prior to the EC2 being terminated.

We should instead capture the IOError exception and have resmoke.py exit with a specific return code. The "run tests" function in etc/evergreen.yml should then check that return code and defer causing the task to fail in a subsequent shell.exec command of type=system. See here for how this is done with existing Jepsen tasks.



 Comments   
Comment by Githook User [ 11/Dec/17 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@10gen.com', 'username': 'kevinAlbs'}

Message: SERVER-32121 sys fail on IOError
Branch: master
https://github.com/mongodb/mongo/commit/53b76274a1b25bc3c505e4e48d9cf2a5ac44f809

Generated at Thu Feb 08 04:29:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.