[SERVER-53062] Add shutdown cleanness checking when running resmoke tests Created: 23/Nov/20  Updated: 29/Oct/23  Resolved: 23/Jun/21

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Luke Chen Assignee: Spencer Jackson
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Fully Compatible
Sprint: Security 2020-12-14, Security 2021-01-11, Security 2021-01-25, Security 2021-02-08, Security 2021-02-22, Security 2021-03-08, Security 2021-03-22, Security 2021-04-05, Security 2021-05-03, Security 2021-05-17, Security 2021-05-31, Security 2021-06-14, Security 2021-06-28
Participants:

 Description   

A recent issue reproduction experience running a resmoke ESE test yielded a positive testing result, though there were BACKTRACE's reported in the logs during the shutdown phase. 

It would be useful to add a shutdown cleanness checking, as unclean shutdowns could sometimes cause serious consequences like data corruption, which should be captured and reflected on the testing result.

It may also worth considering how widespread this improvement should be applied, e.g. certain resmoke tests, all resmoke tests, other tests running on Evergreen. 



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 23/Jun/21 ]

Author:

{'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com', 'username': 'spencerjackson'}

Message: SERVER-53062 Throw an exception in tests if server startup fails
Branch: master
https://github.com/mongodb/mongo/commit/09f99b41e373768c5c1645d15733d1148b891176

Comment by Githook User [ 23/Jun/21 ]

Author:

{'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com', 'username': 'spencerjackson'}

Message: SERVER-53062 Throw an exception in tests if server startup fails
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/1d5cf0a1ab3dde3edb5ff2a64c901c9e84685d0b

Comment by Robert Guo (Inactive) [ 16/Feb/21 ]

It should be possible to do this check for all tests that use shell-spawned fixtures like kmip.js. An example of such a check is validate collection. The shutdown backtrace can be next to it, either pre or post the shutdown action itself.

resmoke fixtures used by most "passthrough" suites will not be able to benefit from this change but they also start and stop far less frequently; a few times per test suite, compared with a few times per test for shell fixtures. A check can likely be added to the CleanEveryN hook (in cleanup.py) but I think it'll be lower value.

Comment by Spencer Jackson [ 08/Feb/21 ]

Unfortunately, I think that kmip.js has to use MongoRunner.runMongod for managing server processes. While running in ESE mode, the server supports a startup option called --kmipRotateMasterKey, which when set causes the server to create a new key encryption key in KMIP, re-encrypt its keystore using the new key, and shutdown. Because kmip.js is testing this functionality, it needs to retain control over process lifecycle and repeatedly start and stop the server.

 

robert.guo, is my understanding of the usecase for fixtures correct, or is there some way to use them while still retaining control over the lifecycle? 

Comment by Brooke Miller [ 24/Nov/20 ]

robert.guo mentioned that we don't use resmoke fixtures for the kmip.js test (context from this comment), so this would end up being a change in the kmip test. Therefore, we're re-assigning to security.

 

Generated at Thu Feb 08 05:29:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.