[SERVER-53062] Add shutdown cleanness checking when running resmoke tests Created: 23/Nov/20 Updated: 29/Oct/23 Resolved: 23/Jun/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Security |
| Affects Version/s: | None |
| Fix Version/s: | 5.1.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Luke Chen | Assignee: | Spencer Jackson |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Sprint: | Security 2020-12-14, Security 2021-01-11, Security 2021-01-25, Security 2021-02-08, Security 2021-02-22, Security 2021-03-08, Security 2021-03-22, Security 2021-04-05, Security 2021-05-03, Security 2021-05-17, Security 2021-05-31, Security 2021-06-14, Security 2021-06-28 | ||||
| Participants: | |||||
| Description |
|
A recent issue reproduction experience running a resmoke ESE test yielded a positive testing result, though there were BACKTRACE's reported in the logs during the shutdown phase. It would be useful to add a shutdown cleanness checking, as unclean shutdowns could sometimes cause serious consequences like data corruption, which should be captured and reflected on the testing result. It may also worth considering how widespread this improvement should be applied, e.g. certain resmoke tests, all resmoke tests, other tests running on Evergreen. |
| Comments |
| Comment by Vivian Ge (Inactive) [ 06/Oct/21 ] |
|
Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you! |
| Comment by Githook User [ 23/Jun/21 ] |
|
Author: {'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com', 'username': 'spencerjackson'}Message: |
| Comment by Githook User [ 23/Jun/21 ] |
|
Author: {'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com', 'username': 'spencerjackson'}Message: |
| Comment by Robert Guo (Inactive) [ 16/Feb/21 ] |
|
It should be possible to do this check for all tests that use shell-spawned fixtures like kmip.js. An example of such a check is validate collection. The shutdown backtrace can be next to it, either pre or post the shutdown action itself. resmoke fixtures used by most "passthrough" suites will not be able to benefit from this change but they also start and stop far less frequently; a few times per test suite, compared with a few times per test for shell fixtures. A check can likely be added to the CleanEveryN hook (in cleanup.py) but I think it'll be lower value. |
| Comment by Spencer Jackson [ 08/Feb/21 ] |
|
Unfortunately, I think that kmip.js has to use MongoRunner.runMongod for managing server processes. While running in ESE mode, the server supports a startup option called --kmipRotateMasterKey, which when set causes the server to create a new key encryption key in KMIP, re-encrypt its keystore using the new key, and shutdown. Because kmip.js is testing this functionality, it needs to retain control over process lifecycle and repeatedly start and stop the server.
robert.guo, is my understanding of the usecase for fixtures correct, or is there some way to use them while still retaining control over the lifecycle? |
| Comment by Brooke Miller [ 24/Nov/20 ] |
|
robert.guo mentioned that we don't use resmoke fixtures for the kmip.js test (context from this comment), so this would end up being a change in the kmip test. Therefore, we're re-assigning to security.
|