[CDRIVER-1565] JSON output from mongo-c-driver-performance suite for Evergreen perf stats Created: 27/Sep/16  Updated: 03/May/17  Resolved: 05/Oct/16

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: TBD

Type: Improvement Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on CDRIVER-1212 C Driver Performance Benchmarking Closed
Epic Link: BUILD-17113

 Description   

The C Driver's performance benchmark is here:

https://github.com/mongodb/mongo-c-driver-performance

Right now it outputs comma-delimited stats. Update it to output JSON to a file, in the format consumed by Evergreen's "performance visualization" plugin.



 Comments   
Comment by Brian McCarthy [ 04/Oct/16 ]

Just tested it out in evergreen and it works.

Comment by Brian McCarthy [ 04/Oct/16 ]

I just ran a couple of the tests and it seems to be formatting correctly:

{
  "results": [
    {
      "name": "TestFlatEncoding",
      "results": {
        "1": {
          "ops_per_sec": 155196112.575398
        }
      }
    },
    {
      "name": "TestDeepEncoding",
      "results": {
        "1": {
          "ops_per_sec": 108160500.490137
        }
      }
    }
  ]
}

I'm going to run a full test in the perf dashboard.

Comment by A. Jesse Jiryu Davis [ 04/Oct/16 ]

How's this?:

https://github.com/mongodb/mongo-c-driver-performance/commit/b70ffb964938a6148c1593e819ca9a725ed3cfdd

Comment by A. Jesse Jiryu Davis [ 30/Sep/16 ]

This writes a JSON document like the one shown above to "results.json" in the current directory:

https://github.com/mongodb/mongo-c-driver-performance/commit/f90e05c20

The driver benchmark spec talks about performance in terms of MB/sec of data processed, so I calculate the median value for that number for each benchmark and use that as "ops_per_sec".

Comment by Brian McCarthy [ 30/Sep/16 ]

I just tested that format out locally and I can see it in the dashboard.

Comment by David Daly [ 30/Sep/16 ]

Yes, I think that should do it.

Comment by A. Jesse Jiryu Davis [ 30/Sep/16 ]

Thanks, so my code should store one file like this?:

{
  "results": [
    {
      "name": "Benchmark One",
      "results": {
        "1": {
          "ops_per_sec": 12345
        }
      },
    {
      "name": "Benchmark Two",
      "results": {
        "1": {
          "ops_per_sec": 42
        }
      }
    ]
  }

... then the perf visualization tools can take it from there?

Comment by David Daly [ 30/Sep/16 ]

Brian has the correct names on there. The initial support was based off of the results mongo-perf made. I believe evergreen is filling in the top level fields, and you want to create the sub document rooted in "data".

The visualization is mainly using the test name, an ops_per_second value.

Comment by Brian McCarthy [ 30/Sep/16 ]

I believe the JSON file can be named whatever we like and saved in any directory. Then we specify the filename in the command in the evergreen yaml file:

    - command: "json.send"
      params:
         name: "perf"
         file: "src/dashboard.json"

For the other questions, in addition to David, we might ask mpobrien or kyle.erf

Comment by A. Jesse Jiryu Davis [ 30/Sep/16 ]

questions for brian.mccarthy or david.daly:

  • who knows the performance visualization evergreen plugin best? who maintains it?
  • what fields are required in the top-level JSON document, and in each "results" subdocument?
  • what additional fields are supported in the top-level JSON document, and in each "results" subdocument?
  • what should the JSON file be named and where should it be saved to during the build task?
Comment by Brian McCarthy [ 29/Sep/16 ]

Excerpted example from https://evergreen.mongodb.com/plugin/json/task/sys_perf_linux_1_node_replSet_core_workloads_WT_7f6eb7f2361addf3f75308bf1603caa8cdc71b68_16_09_28_21_53_17/perf/

{
  "name": "perf",
  "task_name": "core_workloads_WT",
  "project_id": "sys-perf",
  "task_id": "sys_perf_linux_1_node_replSet_core_workloads_WT_7f6eb7f2361addf3f75308bf1603caa8cdc71b68_16_09_28_21_53_17",
  "build_id": "sys_perf_linux_1_node_replSet_7f6eb7f2361addf3f75308bf1603caa8cdc71b68_16_09_28_21_53_17",
  "variant": "linux-1-node-replSet",
  "version_id": "sys_perf_7f6eb7f2361addf3f75308bf1603caa8cdc71b68",
  "create_time": "2016-09-28T21:53:17.417Z",
  "is_patch": false,
  "order": 5035,
  "revision": "7f6eb7f2361addf3f75308bf1603caa8cdc71b68",
  "data": {
  "results": [
      {
        "end": 1.475112711e+09,
        "name": "contended_update-wiredTiger",
        "results": {
          "32": {
            "ops_per_sec": 21440.55310319377
          },
          "64": {
            "ops_per_sec": 14667.715142320709
          }
        },
        "start": 1.475108858e+09,
        "workload": "shell/benchrun"
      }
    ]
  },
  "tag": ""
}

Generated at Wed Feb 07 21:12:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.