[SERVER-74114] Investigate 'snapshot_reads_with_currentOp' and 'snapshot_reads_currentOp_throughput' regressions Created: 16/Feb/23  Updated: 27/Mar/23

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Mihai Andrei Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Query Execution
Participants:

 Description   

db.sbe_perf_latest.aggregate([{ $match: { tags: {$ne: "ignore"}, test: "snapshot_reads_with_currentOp" } }, {$project: {signed_percent_change_stable: 1, test: 1, args: 1, _id: 0, measurement: 1}}]).pretty()
[
  {
    args: { thread_level: 32 },
    measurement: 'ops_per_sec',
    signed_percent_change_stable: -12.6343820837036,
    test: 'snapshot_reads_with_currentOp'
  },
  {
    args: { thread_level: 1 },
    measurement: 'ops_per_sec',
    signed_percent_change_stable: -4.802311748749716,
    test: 'snapshot_reads_with_currentOp'
  },
  {
    args: { thread_level: 64 },
    measurement: 'ops_per_sec',
    signed_percent_change_stable: 0.08377862328757146,
    test: 'snapshot_reads_with_currentOp'
  }
] 



 Comments   
Comment by David Storch [ 27/Mar/23 ]

Here is the latest data:

[
  {
    args_str: "{'thread_level': 64}",
    classification: 'not statistically significant noise',
    'meanpercent adjusted': -7.401655234557514,
    measurement: 'ops_per_sec',
    sign: 1,
    signed_percent_change_stable: -0.6510832046425321,
    signed_z_score_change_stable: -0.19897378296047885,
    stable_mean_baseline: 79529.95388957919,
    stable_mean_measure: 73643.42089446998,
    test: 'snapshot_reads_with_currentOp',
    value_measure: 79012.1477171442,
    stable_to_stable_signed: -7.401655234557514
  },
  {
    args_str: "{'thread_level': 32}",
    classification: 'not statistically significant noise',
    'meanpercent adjusted': -3.1273575403617637,
    measurement: 'ops_per_sec',
    sign: 1,
    signed_percent_change_stable: 1.3838955248721732,
    signed_z_score_change_stable: 0.5320434747566567,
    stable_mean_baseline: 69235.87210162004,
    stable_mean_measure: 67070.6188348148,
    test: 'snapshot_reads_with_currentOp',
    value_measure: 70194.02423724058,
    stable_to_stable_signed: -3.1273575403617637
  },
  {
    args_str: "{'thread_level': 1}",
    classification: 'not statistically significant noise',
    'meanpercent adjusted': -1.1525635949785173,
    measurement: 'ops_per_sec',
    sign: 1,
    signed_percent_change_stable: 3.367256107897518,
    signed_z_score_change_stable: 0.30413468138428856,
    stable_mean_baseline: 9187.623379425217,
    stable_mean_measure: 9081.730177110227,
    test: 'snapshot_reads_with_currentOp',
    value_measure: 9496.994188839533,
    stable_to_stable_signed: -1.1525635949785173
  },
  {
    args_str: "{'thread_level': 1}",
    classification: 'not statistically significant noise',
    'meanpercent adjusted': 1.3935552903332482,
    measurement: 'ops_per_sec',
    sign: 1,
    signed_percent_change_stable: 2.2451442927723377,
    signed_z_score_change_stable: 0.6258752453659838,
    stable_mean_baseline: 1189.8021356452002,
    stable_mean_measure: 1206.3826862509818,
    test: 'snapshot_reads_currentOp_throughput',
    value_measure: 1216.5149103889219,
    stable_to_stable_signed: 1.3935552903332482
  }
]

I'm focusing here on the stable_to_stable_signed numbers, which compare the stable mean for SBE against the stable mean for the classic engine. There are no critical regressions here about the 10% threshold, but it does look like SBE might be a bit slower for the 64 thread variant of "stable_to_stable_signed". While we do not have time to investigate this more deeply as part of the current SBE perf project, we may want to look at it again as part of a follow-up project.

Comment by Kyle Suarez [ 14/Mar/23 ]

david.storch@mongodb.com to investigate this after writing up the report in PERF-3334

Generated at Thu Feb 08 06:26:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.