[SERVER-33201] Add Benchmark resmoke testcase Created: 08/Feb/18  Updated: 29/Oct/23  Resolved: 07/Mar/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 3.7.1
Fix Version/s: 3.6.5, 3.7.3

Type: Task Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Robert Guo (Inactive)
Resolution: Fixed Votes: 0
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6
Sprint: TIG 2018-02-26, TIG 2018-03-12
Participants:

 Description   

Add new resmoke testcase for Benchmark, and pipe in the following options:

--benchmark_repetitions=6, also user configuration from resmoke. if --repeat is used, error out
--benchmark_format=json
--benchmark_out=<filename>=TEST_NAME.json
--benchmark_min_time=5 # 5 seconds (or a different number, depending on code review feedback)



 Comments   
Comment by Githook User [ 26/Apr/18 ]

Author:

{'email': 'robert.guo@10gen.com', 'username': 'guoyr', 'name': 'Robert Guo'}

Message: SERVER-33201 add resmoke.py testcase for Benchmark tests

(cherry picked from commit a47c30b73e686fb4b5743a969e4c79386bd26c7b)
Branch: v3.6
https://github.com/mongodb/mongo/commit/ea6f63582215843b984ba1ce0d25174ee85cc84d

Comment by Githook User [ 07/Mar/18 ]

Author:

{'email': 'robert.guo@10gen.com', 'name': 'Robert Guo', 'username': 'guoyr'}

Message: SERVER-33201 add resmoke.py testcase for Benchmark tests
Branch: master
https://github.com/mongodb/mongo/commit/a47c30b73e686fb4b5743a969e4c79386bd26c7b

Comment by Max Hirschhorn [ 20/Feb/18 ]

Add new resmoke testcase for Benchmark, and pipe in the following options:

--benchmark_repetitions=6, also user configuration from resmoke. if --repeat is used, error out
--benchmark_format=json
--benchmark_out=<filename>=TEST_NAME.json

robert.guo, resmoke.py uses --camelCase command line options to match the style used by mongod. I think that even though google/benchmark uses --snake_case command line options, we should continue to respect resmoke.py's established convention.

On 2018/02/17 16:07:09, robert.guo wrote:
> On 2018/02/16 00:03:31, maxh wrote:
> > On 2018/02/15 18:52:06, robert.guo wrote:
> > > I was thinking of having aliases. Since most people haven't used Benchmark
> > > before, providing a resmoke interface will provide a lower learning curve. For
> > > people who have used benchmark, they can always just run the Benchmark binaries
> > > directly.
> > >
> > > On 2018/02/15 00:14:43, maxh wrote:
> > > > "for Benchmark tests" -> "for google/benchmark tests"? It is marginally more
> > > > informative as to where the supported command line options come from. We should
> > > > consider whether we want to have the snake_case options be blindly forwarded to
> > > > the executable or offer most consistent aliases that benchmark_test.py maps for
> > > > the user.
> > >
> >
> > On one hand, resmoke.py has used snake_case options to represent those which are
> > "internal" and not blindly forwarded to the spawned process. On the other hand,
> > I consider it a feature that users shouldn't need to look at the TestCase to
> > understand how to make resmoke.py do a particular thing. I suspect we may want
> > to change resmoke.py to use a different convention to support a wider range of
> > command line option structures.
>
> Acknowledged. Could we continue this discussion on the "Adding Benchmark
> testcase" ticket that adds these options? I'm in favor of not blindly forwarding
> anything for benchmarks, and only give users 3 knobs described in that ticket:
> https://jira.mongodb.org/browse/SERVER-33201

Continuing our conversation on the code review from SERVER-33199 here. I think that it is reasonable to only expose a few google/benchmark options as resmoke.py command line options. This is consistent with resmoke.py's existing philosophy of anyone trying to do something more sophisticated must be willing to edit at least one YAML file. However, to that point, I would say that blindly forwarding any additional options must be possible.

Generated at Thu Feb 08 04:32:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.