[SERVER-37664] Add support for doing resmoke.py process management through jasper Created: 19/Oct/18  Updated: 29/Oct/23  Resolved: 14/Nov/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.1.6

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Shreyas Kalyan
Resolution: Fixed Votes: 0
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-38017 Return the actual exit code that jasp... Closed
Backwards Compatibility: Fully Compatible
Sprint: STM 2018-11-05, STM 2018-11-19
Participants:
Story Points: 5

 Description   

https://github.com/mongodb/jasper is a library for doing process management through commands over a socket. Having process management available as a service means (1) we can consolidate the various implementations we have through the subprocess / subprocess32 Python packages and the mongo shell's shell_utils_launcher.cpp C++ code, and (2) we can allow tests to interact with the cluster in a potentially destructive way. #2 enables tools such as genny to be able to run performance workloads that measure the latency of operations after restarting a mongod or mongos process.

MAKE-497 exposed jasper through the curator binary.



 Comments   
Comment by Githook User [ 14/Nov/18 ]

Author:

{'name': 'Shreyas Kalyan', 'email': 'shreyaskalyan@gmail.com', 'username': 'shreyaskal'}

Message: SERVER-37664 Add support for doing resmoke.py process management through jasper
Branch: master
https://github.com/mongodb/mongo/commit/ca5e2e4e3334c92062cad841a53da7f69711cdcc

Comment by Max Hirschhorn [ 26/Oct/18 ]

shreyas.kalyan, I would say at a high level, this ticket is made of the following pieces:

  1. Introducing a new command line option --spawnUsing=python|jasper with --spawnUsing=python being the default and identical to the existing behavior.
  2. Creating a version of the resmokelib.core.Process class that sends an HTTP request to the jasper service rather than using the subprocess package, but otherwise has an identical API. There's no API documentation for jasper but the appropriate endpoints to use for spawning a process, sending it a signal, and waiting for it to exit should hopefully be self-evident from their names in this list. It should be fine for now to have their logs go to separate files in Fixture.get_dbpath_prefix() rather than being logged to the console.
  3. When --spawnUsing=jasper, resmoke.py should download a pinned version of the curator binary. You can use https://s3.amazonaws.com/boxes.10gen.com/build/curator/curator-dist-<variant>-<githash>.tar.gz as the URL, where variant could be "ubuntu1604", "macos", or "windows-64" and should be determined at runtime via sys.platform. Take a look at https://evergreen.mongodb.com/waterfall/curator?task_filter=push to see the other build variants if you'd like, but I'd say it isn't worth supporting the ability to download zSeries, ARM, or PowerPC binaries because we'll never run those locally for genny. The binary should be downloaded into a temporary directory and extracted into the build/ directory like with do with clang-format-3.8. It should be checked for at the beginning to avoid downloading the curator binary if it is already present and of the same version.
  4. When --spawnUsing=jasper, resmoke.py should run curator jasper rest --port=<basePort - 1 = 19999 by default> at the beginning to start the service and terminate the process before exiting. This means spawning the curator process should be the only usage of the subprocess package (really the subprocess32 package) when --spawnUsing=jasper. The reasoning behind having the usage of jasper be aware of resmoke.py base port is to ensure it is possible to run resmoke.py with --spawnUsing=jasper on a shared Linux workstation. (Longer term we would want there to be a single instance of jasper running on the shared Linux workstation but that would mean resmoke.py could run processes as a different user than the resmoke.py Python process, which we haven't instruction Server engineers how to protect themselves against yet.)
  5. Every usage of the resmokelib.core.Process class should be replaced by a means of using resmokelib.core.Process when --spawnUsing=python and using the new class when --spawnUsing=jasper. One way to achieve this would be to add a make_process() function that constructs an instance of the appropriate class after consulting a variable defined in the config.py module.
Generated at Thu Feb 08 04:46:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.