Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-74229

Resmoke.py Invocation for Local Usage is incomplete

    • Type: Icon: Improvement Improvement
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Server Development Platform

      We need better output from Evergreen on how to run a failing test locally (i.e. on a VWS).

      Trying to reproduce this failure locally:

      https://spruce.mongodb.com/task/mongodb_mongo_v6.0_rhel80_debug_asan_sharding_jscore_passthrough_last_lts_new_old_old_new_0_linux_enterprise_4d4772c246e440b577218ff40de892c2c1aa66ab_23_01_26_23_37_04/files?execution=0&sortBy=STATUS&sortDir=ASC

      using instructions from this wiki:

      https://github.com/mongodb/mongo/wiki/Running-Tests-from-Evergreen-Tasks-Locally#building-up-the-resmokepy-invocation

      I found that the information from Evergreen on how to execute the correct resmoke.py command locally is incomplete. The wiki plus the "Resmoke.py Invocation for Local Usage" link on the Files tab:

      https://mciuploads.s3.amazonaws.com/mongodb-mongo-v6.0/rhel80-debug-asan/4d4772c246e440b577218ff40de892c2c1aa66ab/local-resmoke-invocation-mongodb_mongo_v6.0_rhel80_debug_asan_sharding_jscore_passthrough_last_lts_new_old_old_new_0_linux_enterprise_4d4772c246e440b577218ff40de892c2c1aa66ab_23_01_26_23_37_04-0.txt

      indicate the command should be

      buildscripts/resmoke.py run --suites=sharding_jscore_passthrough_last_lts_new_old_old_new --installDir=dist-test/bin --continueOnFailure --excludeWithAnyTags=multiversion_incompatible,backport_required_multiversion,sharding_jscore_multiversion_backport_required_multiversion,requires_fcv_51,requires_fcv_52,requires_fcv_53,requires_fcv_60 --excludeWithAnyTags=requires_fast_memory,requires_ocsp_stapling --jobs=8 --shuffle --runNoFeatureFlagTests --storageEngineCacheSizeGB=1 jstests/core/index_filter_collation.js 

      But buildscripts/resmokelib/configure_resmoke.py itself fails with the following error:

      FileNotFoundError: [Errno 2] No such file or directory: 'all_feature_flags.txt' 

      Full stack of this first failure:

      Traceback (most recent call last):
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmoke.py", line 14, in <module>
          cli.main(sys.argv)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/cli.py", line 16, in main
          subcommand = parser.parse_command_line(
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/parser.py", line 53, in parse_command_line
          subcommand_obj = plugin.parse(subcommand, parser, parsed_args, **kwargs)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/run/__init__.py", line 537, in parse
          configure_resmoke.validate_and_update_config(parser, parsed_args)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 30, in validate_and_update_config
          _update_config_vars(args)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 222, in _update_config_vars
          _config.ENABLED_FEATURE_FLAGS, all_feature_flags = setup_feature_flags()
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 200, in setup_feature_flags
          all_ff = process_feature_flag_file(ALL_FEATURE_FLAG_FILE)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/configure_resmoke.py", line 185, in process_feature_flag_file
          with open(path) as fd:
      FileNotFoundError: [Errno 2] No such file or directory: 'all_feature_flags.txt' 

      vishnu.kaushik@mongodb.com told me, via the #evergreen-users Slack channel, that to generate the all_feature_flags.txt file I should run

       python buildscripts/idl/gen_all_feature_flag_list.py

      This succeeded in creating file mongo/all_feature_flags.txt, but this was still not sufficient to run the test locally, as after that the resmoke tooling failed with a different error:

        File "/opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/python3-v4.8tW/lib/python3.10/subprocess.py", line 524, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'mongo-5.0 --version' returned non-zero exit status 127.
      

      Full stack for the second error:

       [executor] 22:21:57.753Z Shuffling order of tests for js_tests in suite sharding_jscore_passthrough_last_lts_new_old_old_new. The seed is 429228164449.
      /bin/sh: 1: mongo-5.0: not found
      [executor] 22:21:57.823Z Encountered an error when running js_tests of suite sharding_jscore_passthrough_last_lts_new_old_old_new.
      Traceback (most recent call last):
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/run/__init__.py", line 300, in _execute_suite
          executor = testing.executor.TestSuiteExecutor(
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 59, in __init__
          self._jobs = self._create_jobs(suite.get_num_jobs_to_start())
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 67, in _create_jobs
          return [self._make_job(job_num) for job_num in range(num_jobs)]
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 67, in <listcomp>
          return [self._make_job(job_num) for job_num in range(num_jobs)]
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 263, in _make_job
          fixture = self._make_fixture(job_num)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/executor.py", line 237, in _make_fixture
          return fixtures.make_fixture(fixture_class, fixture_logger, job_num, **fixture_config)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 31, in make_fixture
          return builder.build_fixture(logger, job_num, fixturelib, *args, **kwargs)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 341, in build_fixture
          config_svr = self._new_configsvr(sharded_cluster, is_multiversion, old_bin_version)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 368, in _new_configsvr
          return make_fixture("ReplicaSetFixture", configsvr_logger, sharded_cluster.job_num,
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 31, in make_fixture
          return builder.build_fixture(logger, job_num, fixturelib, *args, **kwargs)
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 132, in build_fixture
          load_version(version_path_suffix=self.multiversion_class_suffix,
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/testing/fixtures/_builder.py", line 211, in load_version
          commit = gen_tests.get_backports_required_hash_for_shell_version(
        File "/home/ubuntu/mongo-v60/mongo/buildscripts/resmokelib/run/generate_multiversion_exclude_tags.py", line 35, in get_backports_required_hash_for_shell_version
          shell_version = check_output(f"{mongo_shell} --version", shell=True,
        File "/opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/python3-v4.8tW/lib/python3.10/subprocess.py", line 420, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
        File "/opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/python3-v4.8tW/lib/python3.10/subprocess.py", line 524, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'mongo-5.0 --version' returned non-zero exit status 127.
      

      FYI: kyle.suarez@mongodb.com arun.banala@mongodb.com amr.elhelw@mongodb.com 

            Assignee:
            backlog-server-devplatform [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
            Reporter:
            kevin.cherkauer@mongodb.com Kevin Cherkauer
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: