Reduce workload client CPU in heavy_cursor_memory_usage_locust

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Minor - P4
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The heavy_cursor_memory_usage_locust sys-perf task (variant mongotune-perf-atlas-real-ec-dynamic.availability.arm.aws) intermittently fails with:

      CPUUsageError('Workload client CPU usage exceeded threshold: 92.3% > 90.0%')
      

      This is Weta's client-side safety check firing: the Locust workload client (not mongod) is overloaded, which would make the collected performance results unreliable. Failure tracked in BF-45764 (Build Baron auto-matches recurrences via search term CPUUsageError).

      Root cause

      In workloads/availability/heavy-cursor-memory-usage/src/heavy_cursor_memory_usage_workload.py, every invocation of open_cursor_task issues a $currentOp (idleCursors: true) aggregation to check whether the max_open_cursors cap (1000) has been reached, before running the actual measured find.

      With locust_users: 350 across locust_processes: 7 at ~4 rps/user, this bookkeeping check alone generates up to ~1400 extra ops/sec of client-side pipeline construction and BSON decoding, roughly doubling the client request volume. This pushes workload client CPU just over the default 90% threshold (observed 92.3%). It also runs inside the measure("find_cursor") block, skewing
      the reported latency metric.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: