Loading...

Type: Task
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- perf-ongoing

Assigned Teams:

Workload Scheduling
Linked BF Score:
0
Confidence Status:
None
Work Order:
3
Size Category:
TBD
CAR Domain/s:
None

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name(s):
None
Goal Link:
None

Estimated Weeks:
0

many_memory_heavy_ops_locust failed on 3-Node ReplSet Availability ARM AWS 2024-05

Host: N/A
Project: System Performance (master)
Commit: diff: ~~SERVER-104491~~ Increase sleep between bazel compile retries (#35552)

This may mitigate
[DEVPROD-17274](https://jira.mongodb.org/browse/DEVPROD-17274) and is
more likely to resolve issues from general network outages | 29 Apr 25 23:07 UTC
Evergreen Subscription: ; Evergreen Event:

Task Logs (many_memory_heavy_ops_locust)

 [2025/04/29 21:19:05.999] [info   ] [dsi.libanalysis.resilience_metrics.time_to_failover] Calculating time to failover
 [2025/04/29 21:19:06.005] [error  ] [dsi.error                   ] 
 [2025/04/29 21:19:06.005]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.005]         Error Details:
 [2025/04/29 21:19:06.005]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.005]         Error Type:     RuntimeError
 [2025/04/29 21:19:06.005]         Error Message:  No failover log found in mongod
 [2025/04/29 21:19:06.005]         File:           /data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/resilience_metrics/time_to_failover.py
 [2025/04/29 21:19:06.005]         Line Number:    176
 [2025/04/29 21:19:06.005]         Function:       calculate_metric
 [2025/04/29 21:19:06.005]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.005]         Stack Trace:
 [2025/04/29 21:19:06.006]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.006]           File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/opentelemetry/context_tools.py", line 45, in wrapper
 [2025/04/29 21:19:06.006]     return func(*args, **kwargs)
 [2025/04/29 21:19:06.006]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/analysis.py", line 255, in main
 [2025/04/29 21:19:06.006]     cedar_outcome, timers, report = _generate_cedar_json(config, reference_run)
 [2025/04/29 21:19:06.006]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/analysis.py", line 173, in _generate_cedar_json
 [2025/04/29 21:19:06.006]     this_passed, cedar_tests = parser.parse_and_save_log_file()
 [2025/04/29 21:19:06.006]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/workload_output_parser.py", line 293, in parse_and_save_log_file
 [2025/04/29 21:19:06.006]     passed = self.parse()
 [2025/04/29 21:19:06.006]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/workload_output_parser.py", line 343, in parse
 [2025/04/29 21:19:06.006]     self._parse()
 [2025/04/29 21:19:06.006]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/workload_output_parser.py", line 767, in _parse
 [2025/04/29 21:19:06.006]     cedar_test = time_to_failover.calculate_metric(self.test_id)
 [2025/04/29 21:19:06.006]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/resilience_metrics/time_to_failover.py", line 176, in calculate_metric
 [2025/04/29 21:19:06.006]     handle_fatal_error(RuntimeError("No failover log found in mongod"))
 [2025/04/29 21:19:06.006]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.006]         Exception Chain:
 [2025/04/29 21:19:06.006]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.006]         RuntimeError: No failover log found in mongod
 [2025/04/29 21:19:06.006]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.006]         
 [2025/04/29 21:19:06.007] [error  ] [dsi.libanalysis.workload_output_parser] ResultParser.parse() encountered an error. At least some results are likely missing. I will now print the error and then try to gracefully continue to the end.
 [2025/04/29 21:19:06.007] Traceback (most recent call last):
 [2025/04/29 21:19:06.007]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/workload_output_parser.py", line 343, in parse
 [2025/04/29 21:19:06.007]     self._parse()
 [2025/04/29 21:19:06.007]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/workload_output_parser.py", line 767, in _parse
 [2025/04/29 21:19:06.007]     cedar_test = time_to_failover.calculate_metric(self.test_id)
 [2025/04/29 21:19:06.007]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/resilience_metrics/time_to_failover.py", line 176, in calculate_metric
 [2025/04/29 21:19:06.007]     handle_fatal_error(RuntimeError("No failover log found in mongod"))
 [2025/04/29 21:19:06.007]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/contextlib.py", line 79, in inner
 [2025/04/29 21:19:06.007]     return func(*args, **kwds)
 [2025/04/29 21:19:06.007]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/error/error.py", line 110, in handle_fatal_error
 [2025/04/29 21:19:06.007]     raise exception
 [2025/04/29 21:19:06.007] RuntimeError: No failover log found in mongod
 [2025/04/29 21:19:06.007] [info   ] [dsi.libanalysis.workload_output_parser] Saved perf.json cwd=/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755 path=./build/LegacyPerfJson/perf.json
 [2025/04/29 21:19:06.015] [error  ] [dsi.libanalysis.system_metrics.end_state_metrics] build/WorkloadOutput/reports/many_memory_heavy_ops/mongod.0/pre_post_task_output/cpuid.txt file not found
 [2025/04/29 21:19:06.015] [error  ] [dsi.libanalysis.system_metrics.end_state_metrics] build/WorkloadOutput/reports/many_memory_heavy_ops/mongod.1/pre_post_task_output/cpuid.txt file not found
 [2025/04/29 21:19:06.015] [error  ] [dsi.libanalysis.system_metrics.end_state_metrics] build/WorkloadOutput/reports/many_memory_heavy_ops/mongod.2/pre_post_task_output/cpuid.txt file not found
 [2025/04/29 21:19:06.016] [error  ] [dsi.libanalysis.system_metrics.end_state_metrics] build/WorkloadOutput/reports/many_memory_heavy_ops/workload_client.1/pre_post_task_output/cpuid.txt file not found
 [2025/04/29 21:19:06.016] [error  ] [dsi.libanalysis.system_metrics.end_state_metrics] build/WorkloadOutput/reports/many_memory_heavy_ops/workload_client.0/pre_post_task_output/cpuid.txt file not found
 [2025/04/29 21:19:06.025] [error  ] [dsi.error                   ] 
 [2025/04/29 21:19:06.025]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.025]         Error Details:
 [2025/04/29 21:19:06.025]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.025]         Error Type:     FileNotFoundError
 [2025/04/29 21:19:06.025]         Error Message:  [Errno 2] No such file or directory: './build/WorkloadOutput/reports/many_memory_heavy_ops/mongod.0/pre_post_task_output/du.end'
 [2025/04/29 21:19:06.025]         File:           /data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py
 [2025/04/29 21:19:06.025]         Line Number:    263
 [2025/04/29 21:19:06.025]         Function:       parallel_operations
 [2025/04/29 21:19:06.025]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.025]         Stack Trace:
 [2025/04/29 21:19:06.025]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.025]           File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/.venv/bin/dsi", line 6, in <module>
 [2025/04/29 21:19:06.025]     sys.exit(cli())
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 1098, in run_workload
 [2025/04/29 21:19:06.025]     _run_workload(
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 464, in _run_workload
 [2025/04/29 21:19:06.025]     _run_infrastructure_teardown_and_analysis()
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 155, in wrapper
 [2025/04/29 21:19:06.025]     result = func(*args, **kwargs)
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 285, in _run_infrastructure_teardown_and_analysis
 [2025/04/29 21:19:06.025]     return CLIOperation(op, "infrastructure_teardown_and_analysis", FailureMode.TEST).run()
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 84, in run
 [2025/04/29 21:19:06.025]     outcome = self.op()
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 263, in parallel_operations
 [2025/04/29 21:19:06.025]     handle_fatal_error(exception)
 [2025/04/29 21:19:06.025]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.025]         Exception Chain:
 [2025/04/29 21:19:06.025]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.025]         Traceback (most recent call last):
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/cli/cli.py", line 260, in parallel_operations
 [2025/04/29 21:19:06.025]     if future.result() == 1:
 [2025/04/29 21:19:06.025]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/concurrent/futures/_base.py", line 439, in result
 [2025/04/29 21:19:06.025]     return self.__get_result()
 [2025/04/29 21:19:06.025]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
 [2025/04/29 21:19:06.025]     raise self._exception
 [2025/04/29 21:19:06.025]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/concurrent/futures/thread.py", line 58, in run
 [2025/04/29 21:19:06.025]     result = self.fn(*self.args, **self.kwargs)
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/opentelemetry/context_tools.py", line 45, in wrapper
 [2025/04/29 21:19:06.025]     return func(*args, **kwargs)
 [2025/04/29 21:19:06.025]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/contextlib.py", line 79, in inner
 [2025/04/29 21:19:06.025]     return func(*args, **kwds)
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/analysis.py", line 255, in main
 [2025/04/29 21:19:06.025]     cedar_outcome, timers, report = _generate_cedar_json(config, reference_run)
 [2025/04/29 21:19:06.025]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/contextlib.py", line 79, in inner
 [2025/04/29 21:19:06.025]     return func(*args, **kwds)
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/analysis.py", line 186, in _generate_cedar_json
 [2025/04/29 21:19:06.025]     for test in analyze_all_metrics(
 [2025/04/29 21:19:06.025]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/contextlib.py", line 79, in inner
 [2025/04/29 21:19:06.025]     return func(*args, **kwds)
 [2025/04/29 21:19:06.025]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/system_metrics/__init__.py", line 63, in analyze_all_metrics
 [2025/04/29 21:19:06.025]     results += analyze_phase(config, metrics, root_phase, timer_manager_filename)
 [2025/04/29 21:19:06.026]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/contextlib.py", line 79, in inner
 [2025/04/29 21:19:06.026]     return func(*args, **kwds)
 [2025/04/29 21:19:06.026]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/system_metrics/__init__.py", line 97, in analyze_phase
 [2025/04/29 21:19:06.026]     results += analyze_root_phase(config, root_phase, timer_manager_filename)
 [2025/04/29 21:19:06.026]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/system_metrics/__init__.py", line 175, in analyze_root_phase
 [2025/04/29 21:19:06.026]     metrics = analyze_directory_growth_metrics(path, test_phase)
 [2025/04/29 21:19:06.026]   File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CA4/lib/python3.10/contextlib.py", line 79, in inner
 [2025/04/29 21:19:06.026]     return func(*args, **kwds)
 [2025/04/29 21:19:06.026]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/system_metrics/end_state_metrics.py", line 62, in analyze_directory_growth_metrics
 [2025/04/29 21:19:06.026]     end_results = read_du(os.path.join(mongod_path, DU_END_FILE))
 [2025/04/29 21:19:06.026]   File "/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/src/dsi/libanalysis/system_metrics/end_state_metrics.py", line 164, in read_du
 [2025/04/29 21:19:06.026]     with open(path) as handle:
 [2025/04/29 21:19:06.026] FileNotFoundError: [Errno 2] No such file or directory: './build/WorkloadOutput/reports/many_memory_heavy_ops/mongod.0/pre_post_task_output/du.end'
 [2025/04/29 21:19:06.026]         --------------------------------------------------------------------------------
 [2025/04/29 21:19:06.026]         
 [2025/04/29 21:20:50.367] [info   ] [dsi.config.config           ] Loaded DSI config config_root=/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755 loaded_files=['/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/bootstrap.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/configurations/bootstrap/defaults.original.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/runtime.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/runtime_secret.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/infrastructure_provisioning.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/configurations/infrastructure_provisioning/defaults.2025-01.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/workload_setup.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/configurations/workload_setup/defaults.original.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/mongodb_setup.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/configurations/mongodb_setup/defaults.2025-01-mongotune.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/test_control.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/configurations/test_control/defaults.original.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/analysis.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/configurations/analysis/defaults.original.yml', '/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/overrides.yml']
 [2025/04/29 21:20:50.373] [info   ] [dsi.config.config           ] Saved config file file_name=/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/mongodb_setup.out.yml
 [2025/04/29 21:20:50.374] [info   ] [dsi.infrastructure_teardown ] Successfully completed infrastructure_teardown. outcome_code=0
 [2025/04/29 21:20:50.375] [info   ] [dsi.cli                     ] Finished running DSI operation exit_code=1 invocation=['/data/mci/4c7c34f757f9c6e1ba5d6f7aaf186755/src/dsi/.venv/bin/dsi', 'run_workload', '--parallelize-phases'] op_uuid=619b1a6f operation=infrastructure_teardown_and_analysis
 [2025/04/29 21:20:50.377] [info   ] [dsi.cli                     ] 
 [2025/04/29 21:20:50.377] ------------------------------------------------------------------------------------------
 [2025/04/29 21:20:50.377] |              infrastructure_teardown_and_analysis completed with failures              |

is related to

SERVER-104491 Increase sleep between bazel compile retries

Closed

Details

Description

many_memory_heavy_ops_locust failed on 3-Node ReplSet Availability ARM AWS 2024-05

Attachments

Issue Links

Activity

People

Dates