[CDRIVER-3620] Audit evergreen matrix Created: 14/Apr/20  Updated: 28/Oct/23  Resolved: 14/Feb/23

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.24.0

Type: Task Priority: Major - P3
Reporter: Kevin Albertson Assignee: Ezra Chung
Resolution: Fixed Votes: 0
Labels: size-large
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to CDRIVER-3680 Valgrind OpenSSL tasks take over thre... Closed
related to CDRIVER-4033 power8-rhel81 OpenSSL linking failure Closed
related to CDRIVER-4562 Auth tests fail with OpenSSL 1.1.0 Closed
related to CDRIVER-4571 Finish relocating Evergreen config in... Backlog
related to CDRIVER-4347 CSFLE tests failing with certificate ... Closed
related to CDRIVER-3317 Replace valgrind variant with Memory ... Closed
related to CXX-2693 Remove valgrind tests from Evergreen Closed
related to CDRIVER-3335 Run authentication tests on ASAN Closed
related to CDRIVER-4655 Add support for C17/C18 standard Closed
Quarter: FY23Q3

 Description   

There are redundant tasks. For example, testing with sasl support but without authentication seems redundant, but we have many "noauth-sasl tasks. Many tasks  test with nossl, when that is likely an uncommon use case.

We're not sure that all tests are run. Many tests skip if the environment does not support the test, but we have no global information on whether all tests ran in at least one task.

The config.yml file is ever growing. We should consider using shrub.py.

In some discussion with roberto.sanchez, one approach may be to build/test tasks with all features (ssl, auth, sasl, etc.) enabled, all disabled, and then combinations which have dependencies.



 Comments   
Comment by Githook User [ 14/Feb/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Add SASL, CSE, and Sanitizers matrices (#1201)

  • Remove unused _hash_ from Distro class
  • Relocate DarwinSSL SASL tests to config_generator
  • Relocate WinSSL SASL tests to config_generator
  • Relocate Static OpenSSL SASL tests to config_generator
  • Relocate OpenSSL SASL tests to config_generator
  • Relocate NoSSL SASL tests to config_generator
  • Relocate DarwinSSL CSE tests to config_generator
  • Relocate WinSSL CSE tests to config_generator
  • Relocate Static OpenSSL CSE tests to config_generator
  • Relocate OpenSSL CSE tests to config_generator
  • Relocate ASAN tests to config_generator
  • Relocate TSAN tests to config_generator
  • Remove UBSAN tasks in favor of ASAN+UBSAN
  • Clean up IntegrationTask -> CoverageTask
  • Relocate C11 compatibility tests to config_generator
  • Merge macos-1100-arm64 compile task into sasl.darwinssl
  • Remove compile tasks without dependent test task
  • Also ensure EXTRA_CONFIGURE_FLAGS are being used when compiling on Windows
  • Remove unused SSPI compile functions
  • Fix missing handling of MARCH variable in compile-unix.sh
  • Move Static OpenSSL compile coverage into seperate component
  • Update cse.openssl.TEST_MATRIX as suggested
  • Update cse.winssl.TEST_MATRIX as suggested
  • Update cse.darwinssl.TEST_MATRIX as suggested
  • Update sasl.nossl.TEST_MATRIX as suggested
  • Remove noauth from test matrices
  • Update sasl.openssl.TEST_MATRIX as suggested
  • Update sasl.winssl.TEST_MATRIX as suggested
  • Update std.std_c11.COMPILE_MATRIX as suggested
  • Add C17 compile coverage
  • Silence file exists warning in upload-test-results
  • Limit tests executed to /client_side_encryption for CSE tasks
  • Remove 4.2 and 4.4 tests on rhel83-zseries
  • CDRIVER-4347 Remove skipped tests due to OCSP verification failure
  • Revenge of scan-build warnings of tmp->tm_mon
  • Explicitly forbid SASL for SaslOff tasks for compile-unix.sh vs. compile-windows.sh consistency
  • Move cse.darwinssl tasks from macos-1014 to macos-1015
  • Rename EvgTaskWithRunOn to Task and add field disable
Comment by Githook User [ 02/Feb/23 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: CDRIVER-3620 Unskip `-cse` tests on Windows (#1199)

  • use Windows paths for libmongocrypt
  • remove extra CMAKE_PREFIX_PATH arg

It is already set in configure_flags

  • fail if libmongoc compile does not find expected libmongocrypt
  • use relative paths for CSFLE_TLS vars
  • remove duplicate CSFLE_TLS vars
  • return error on failure to create tls stream

Fixes an observed assert on failure to load the PEM file

---------

Co-authored-by: Ezra Chung <88335979+eramongodb@users.noreply.github.com>
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/8021da63e76c895419d6f49f3d5ba3ee0e177618

Comment by Githook User [ 01/Feb/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Add new config_generator (#1193)

  • Add new config_generator scripts
  • Move top-level Evergreen configuration into config.yml
  • Add early termination handler to top-level config
  • Relocate pre commands into config_generator and tasks
  • Relocate abi-compliance-check task into config_generator
  • Relocate upload build function to config_generator
  • Relocate make-release-archive into config_generator
  • Relocate backtrace into config_generator
  • Relocate timeout hook into config.yml
  • Relocate prepare kerberos into config_generator
  • Relocate bootstrap mongo-orchestration into config_generator
  • Relocate check-headers into config_generator
  • Relocate fetch build into config_generator
  • Relocate clone drivers-evergreen-tools into config_generator
  • Relocate run kms servers into config_generator
  • Relocate run tests to config_generator
  • Relocate stop load balancer into config_generator
  • Move cleanup into config_generator
  • Relocate upload test results into config_generator
  • Relocate upload mo artifacts into config_generator
Comment by Githook User [ 26/Jan/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Relocate legacy-config.yml into .evergreen/generated_configs (#1191)
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/028695fb895a432f2423065b688de08219a27dfe

Comment by Githook User [ 25/Jan/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Audit compile and test scripts (#1187)

  • Reorder maintainer flags to permit manual disabling of warnings
  • Improve behavior of bypass_dlclose()
  • Add support for add_expansions_to_env to shell_exec
  • Format .codecov.yml
  • Format scripts
  • Sanitize abi-compliance-check.sh
  • Sanitize add-build-dirs-to-paths.sh
  • Sanitize build-and-test-with-toolchain.sh
  • Sanitize compile*.sh scripts
  • Sanitize install-ssl.sh and run*.sh scripts
  • Do not run tests in compile tasks
  • Move all code coverage test-coverage-* tasks
  • Add env-var-utils.sh
  • Use env-var-utils.sh
  • Use nproc to specify Make job count
  • Support linking with custom OpenSSL library without system conflicts
  • CDRIVER-4562 Add SSL_CERT_DIR to resolve cert verify failure with 1.1.0
Comment by Githook User [ 20/Jan/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Remove authentication tests from Ubuntu 14.04 (#1186)

Related to CDRIVER-4541.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e401a5b5b73dc5ca91dea48e0309eca28822d583

Comment by Githook User [ 13/Jan/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Remove RHEL 6.2 tests and RHEL 7.0 CSE tests from Evergreen (#1180)

Comment by Githook User [ 13/Jan/23 ]

Author:

{'name': 'Ezra Chung', 'email': '88335979+eramongodb@users.noreply.github.com', 'username': 'eramongodb'}

Message: CDRIVER-3620 Remove Valgrind tests from Evergreen (#1177)

  • Remove Valgrind tests from Evergreen
  • Add bypass-dlclose.sh
  • Silence LSAN "unknown module" leak warnings when running test executables
  • Ensure ENABLE_EXTRA_ALIGNMENT=OFF for authentication-tests-memcheck
  • Move authentication-tests-memcheck with Clang from 3.8 to 6.0
Comment by Ezra Chung [ 11/Jan/23 ]

Related PR: https://github.com/mongodb/mongo-c-driver/pull/1174
Related Commit: https://github.com/mongodb/mongo-c-driver/commit/8409087f3eb054a80a0c571573730d6d53b01039

Remove debug-compile-coverage from Apple LLVM variant

Comment by Kevin Albertson [ 08/Jun/20 ]

If we do make large changes, it may be worth transitioning to https://github.com/evergreen-ci/evergreen/wiki/Project-Commands#generate-tasks to generate the matrix instead of our scripts, or not using a generator at all.

Generated at Wed Feb 07 21:18:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.