[SERVER-74229] Resmoke.py Invocation for Local Usage is incomplete Created: 17/Feb/23  Updated: 09/May/23  Resolved: 09/May/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kevin Cherkauer Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-72923 Upload db-contrib-tool invocation Closed
Assigned Teams:
Server Development Platform
Participants:

 Description   

We need better output from Evergreen on how to run a failing test locally (i.e. on a VWS).

Trying to reproduce this failure locally:

https://spruce.mongodb.com/task/mongodb_mongo_v6.0_rhel80_debug_asan_sharding_jscore_passthrough_last_lts_new_old_old_new_0_linux_enterprise_4d4772c246e440b577218ff40de892c2c1aa66ab_23_01_26_23_37_04/files?execution=0&sortBy=STATUS&sortDir=ASC

using instructions from this wiki:

https://github.com/mongodb/mongo/wiki/Running-Tests-from-Evergreen-Tasks-Locally#building-up-the-resmokepy-invocation

I found that the information from Evergreen on how to execute the correct resmoke.py command locally is incomplete. The wiki plus the "Resmoke.py Invocation for Local Usage" link on the Files tab:

https://mciuploads.s3.amazonaws.com/mongodb-mongo-v6.0/rhel80-debug-asan/4d4772c246e440b577218ff40de892c2c1aa66ab/local-resmoke-invocation-mongodb_mongo_v6.0_rhel80_debug_asan_sharding_jscore_passthrough_last_lts_new_old_old_new_0_linux_enterprise_4d4772c246e440b577218ff40de892c2c1aa66ab_23_01_26_23_37_04-0.txt

indicate the command should be

buildscripts/resmoke.py run --suites=sharding_jscore_passthrough_last_lts_new_old_old_new --installDir=dist-test/bin --continueOnFailure --excludeWithAnyTags=multiversion_incompatible,backport_required_multiversion,sharding_jscore_multiversion_backport_required_multiversion,requires_fcv_51,requires_fcv_52,requires_fcv_53,requires_fcv_60 --excludeWithAnyTags=requires_fast_memory,requires_ocsp_stapling --jobs=8 --shuffle --runNoFeatureFlagTests --storageEngineCacheSizeGB=1 jstests/core/index_filter_collation.js 

But buildscripts/resmokelib/configure_resmoke.py itself fails with the following error:

FileNotFoundError: [Errno 2] No such file or directory: 'all_feature_flags.txt' 

Full stack of this first failure:

Traceback (most recent call last):
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmoke.py", line 14, in <module>
    cli.main(sys.argv)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/cli.py", line 16, in main
    subcommand = parser.parse_command_line(
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/parser.py", line 53, in parse_command_line
    subcommand_obj = plugin.parse(subcommand, parser, parsed_args, **kwargs)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/run/__init__.py", line 537, in parse
    configure_resmoke.validate_and_update_config(parser, parsed_args)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 30, in validate_and_update_config
    _update_config_vars(args)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 222, in _update_config_vars
    _config.ENABLED_FEATURE_FLAGS, all_feature_flags = setup_feature_flags()
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 200, in setup_feature_flags
    all_ff = process_feature_flag_file(ALL_FEATURE_FLAG_FILE)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 185, in process_feature_flag_file
    with open(path) as fd:
FileNotFoundError: [Errno 2] No such file or directory: 'all_feature_flags.txt' 

vishnu.kaushik@mongodb.com told me, via the #evergreen-users Slack channel, that to generate the all_feature_flags.txt file I should run

 python buildscripts/idl/gen_all_feature_flag_list.py

This succeeded in creating file mongo/all_feature_flags.txt, but this was still not sufficient to run the test locally, as after that the resmoke tooling failed with a different error:

  File "/opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/python3-v4.8tW/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'mongo-5.0 --version' returned non-zero exit status 127.

Full stack for the second error:

 [executor] 22:21:57.753Z Shuffling order of tests for js_tests in suite sharding_jscore_passthrough_last_lts_new_old_old_new. The seed is 429228164449.
/bin/sh: 1: mongo-5.0: not found
[executor] 22:21:57.823Z Encountered an error when running js_tests of suite sharding_jscore_passthrough_last_lts_new_old_old_new.
Traceback (most recent call last):
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/run/__init__.py", line 300, in _execute_suite
    executor = testing.executor.TestSuiteExecutor(
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 59, in __init__
    self._jobs = self._create_jobs(suite.get_num_jobs_to_start())
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 67, in _create_jobs
    return [self._make_job(job_num) for job_num in range(num_jobs)]
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 67, in <listcomp>
    return [self._make_job(job_num) for job_num in range(num_jobs)]
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 263, in _make_job
    fixture = self._make_fixture(job_num)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 237, in _make_fixture
    return fixtures.make_fixture(fixture_class, fixture_logger, job_num, **fixture_config)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 31, in make_fixture
    return builder.build_fixture(logger, job_num, fixturelib, *args, **kwargs)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 341, in build_fixture
    config_svr = self._new_configsvr(sharded_cluster, is_multiversion, old_bin_version)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 368, in _new_configsvr
    return make_fixture("ReplicaSetFixture", configsvr_logger, sharded_cluster.job_num,
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 31, in make_fixture
    return builder.build_fixture(logger, job_num, fixturelib, *args, **kwargs)
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 132, in build_fixture
    load_version(version_path_suffix=self.multiversion_class_suffix,
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 211, in load_version
    commit = gen_tests.get_backports_required_hash_for_shell_version(
  File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/run/generate_multiversion_exclude_tags.py", line 35, in get_backports_required_hash_for_shell_version
    shell_version = check_output(f"{mongo_shell} --version", shell=True,
  File "/opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/python3-v4.8tW/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/python3-v4.8tW/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'mongo-5.0 --version' returned non-zero exit status 127.

FYI: kyle.suarez@mongodb.com arun.banala@mongodb.com amr.elhelw@mongodb.com 



 Comments   
Comment by Alex Neben [ 09/May/23 ]

The PR to fix this is currently open. Making this as duplicate.

Comment by Kevin Cherkauer [ 21/Feb/23 ]

Thank you for correcting the target team.

Comment by John Daniels [ 21/Feb/23 ]

annie.black@mongodb.com I agree with your logic there.

kevin.cherkauer@mongodb.com We do not produce that page. Evergreen allows scripts to upload files and the server deploy scripts generate and upload that file when they detect failure. Here is the code that uploads that file and here is the code that writes it.

Comment by Annie Black [ 21/Feb/23 ]

kevin.cherkauer@mongodb.com , what goes into the Files tab for Evergreen tasks is configured by users using the attach.artifacts command. If there was an issue with attaching the file then this would be an Evergreen issue, but it sounds instead like the contents of the file aren't what you're looking for, which isn't owned by us.

Comment by Kevin Cherkauer [ 21/Feb/23 ]

annie.black@mongodb.com Isn't it Evergreen dumping the info in the "Files / Resmoke.py Invocation for Local Usage" page? This is supposed to be the command Evergreen ran that a human can therefore also run to execute the same test Evergreen did. The wiki is mainly pointing this out to the user. But Evergreen is not dumping the full set of commands a human needs to run to reproduce an Evergreen run, thus this seems like Evergreen territory.

Comment by Annie Black [ 21/Feb/23 ]

I'm not certain this is something Evergreen can help with, since we don't own that wiki or the server setup. john.daniels@mongodb.com can you confirm this isn't something PLT can assist with? If that is the case then you may want to ask in #server-testing who should own this kevin.cherkauer@mongodb.com .

Generated at Thu Feb 08 06:26:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.