[SERVER-36076] Create new resmoke.py test suite for running mongoebench on a desktop Created: 11/Jul/18  Updated: 29/Oct/23  Resolved: 13/Aug/18

Status: Closed
Project: Core Server
Component/s: Performance, Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.0.3, 4.1.2

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-36069 Vendor mongoebench-compatible JSON co... Closed
depends on SERVER-36073 Save stats from BenchRunner::finish()... Closed
is depended on by SERVER-36077 Create new resmoke.py test suite for ... Closed
Problem/Incident
causes SERVER-37060 RHEL 6.2 (mobile) build variant attem... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.0
Sprint: TIG 2018-08-27, TIG 2018-08-13
Participants:
Story Points: 5

 Description   

It should run mongoebench with the various JSON config files that live in the src/third_party/mongo-perf/mongoebench/ directory that have been vendored into the source tree as part of the changes from SERVER-36069.

This involves creating a new buildscripts/resmokelib/testing/testcases/mongoebench_test.py test case that executes mongoebench with the appropriate arguments. For example, the value for the --benchmarkMinTimeSecs command line option should be forwarded as the --time command line option to mongoebench.

This also involves creating a new hook similar to the CombineBenchmarkResults hook that parses the JSON stats file specified as the --output command line option (from SERVER-36073) to mongoebench. The new hook should accumulate benchmark results of all the test cases we run as part of the test suite and serialize them as a JSON file (taking its name from the
--perfReportFile
command line option) that can be used for the json.send Evergreen command to display the performance results. The test case should also handle the --benchmarkRepetitions command line option (in Python, as there is no equivalent option to forward to mongoebench) and accumulate the benchmark results of multiple executions.

We may find it beneficial to define separate test suites that each run a subset of the test cases similar to what is done in the performance Evergreen project when these test cases are run with benchrun.py to avoid having an Evergreen task run for a long time.



 Comments   
Comment by Githook User [ 22/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36076 Create new resmoke.py test suite for running mongoebench on a desktop

(cherry picked from commit a444720202d97795b71cf2daaaa2bea94b430ef9)
Branch: v4.0
https://github.com/mongodb/mongo/commit/1b0a962b03696784741ff9f4e83b2f084d1d88d7

Comment by Githook User [ 13/Aug/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36076 Create new resmoke.py test suite for running mongoebench on a desktop
Branch: master
https://github.com/mongodb/mongo/commit/a444720202d97795b71cf2daaaa2bea94b430ef9

Comment by Max Hirschhorn [ 19/Jul/18 ]

I think we should just run these on the centos6-perf distro as part of a daily or non-required build variant.

Generated at Thu Feb 08 04:41:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.