[CDRIVER-2479] Split up Evergreen config.yml Created: 30/Jan/18  Updated: 28/Oct/23  Resolved: 28/Nov/18

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.13.0, 1.14.0

Type: Improvement Priority: Minor - P4
Reporter: Kevin Albertson Assignee: A. Jesse Jiryu Davis
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Link: CDRIVER-2733

 Description   

Our config.yml is over 8k lines, which makes it a bit painful to navigate and make changes.

It might be worth taking advantage of the Evergreen matrix variant definition to remove duplicate definitions of tasks that are common to many variants.

Consider splitting config.yml into multiple files.

Consider generating parts of the config.yml with scripts.

Consider defining a RELEASED variable that is set to the last released MongoDB version. It will save us the pain of replacing "4.0" with "4.2" next year for every task we want to run on only the last release.

Consider using features of YAML to avoid duplication, e.g. anchors and references.

------------

Step 1: generate the list of tasks from a Python file and Jinja2 template, so that the tasks are defined by a short Python data structure instead of handwritten.

1. Write a Python script and some templates to generate config.yml identical to the current one.
2. Simplify the script by allowing its output to differ slightly to the current config.yml.



 Comments   
Comment by A. Jesse Jiryu Davis [ 28/Nov/18 ]

Done. Common script code is moved to https://github.com/mongodb-labs/drivers-evergreen-tools/tree/master/evergreen_config_generator for reuse in the C++ Driver.

Comment by Githook User [ 28/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 factor common code from generate-evergreen-config
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/6c45f7f545e7f6d7c31d8253cdd9261e8f5897b4

Comment by Githook User [ 28/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 unused import
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/31c7f5c729baecffe5eed4f29802c8f3103c88e9

Comment by Githook User [ 20/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate evergreen config w/o Jinja
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/7ee37b01ac65b59223b791f02a9d1979a873e4be

Comment by Githook User [ 20/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 unused import
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/af2d33fdb560755dc7b3a4cfdcba3bac063582cf

Comment by Githook User [ 15/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate evergreen functions
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/d693f2127d9c85194408a9f08e5e8701b16f196b

Comment by Githook User [ 15/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 no shebang in python module files
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/f6235102cb67aa27fb97fba3b9102766abbfa814

Comment by Githook User [ 15/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 don't add ldaptest to /etc/hosts

The host entry is preinstalled on all machines (BUILD-2557).
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/3e778579de51cbddea67aaf39fd8cc7e251e4557

Comment by Githook User [ 15/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 unused task "getdata"
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/8606a1a2da6ec19b8ba5d6f1e0feabe2511652ab

Comment by Githook User [ 02/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate build variants
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/62dffa714abb9a5b3ef0cc010905d15c1450c32a

Comment by Githook User [ 02/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate IPV4-IPV6 tests
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/5dbb5e686bf397ecc532fe8220b180068ae198ec

Comment by Githook User [ 02/Nov/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate compile-libmongocapi
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/b960a96c3e0447964ef0146380e05a7bc4623beb

Comment by Githook User [ 26/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 factor errexit/xtrace commands
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e30479111fadadf90ad9a8a2d425325be11312b6

Comment by Githook User [ 26/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate OpenSSL version tasks
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/60f8f3a04b1cb5c765eecee383567519bc157fcc

Comment by Githook User [ 26/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate auth-tests-memcheck task

Also fix it to actually use valgrind. Before, it was compiled with
memcheck but it didn't actually check for leaks.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e88f03f0798f81baaeb544ace6d57bd640027e9d

Comment by Githook User [ 26/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate mock server and asan tasks
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/ecd37361e13d5dd4c6b6912cbd23a68ce3254a80

Comment by Githook User [ 24/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate retryWrites tasks
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/f3ac9b33002248aab52ab652244269d43c79a3a3

Comment by Githook User [ 24/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate compression tasks
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/7fa6fc5e64701be47d1ff3b376cbde511cdc298c

Comment by Githook User [ 24/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate DNS tasks
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/81cb7b19008040559ef1cae159ce34508b930369

Comment by Githook User [ 24/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 refactor Task classes
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/fc7c1fe59ac58fe6f7baca21892e8a1c0911cb7f

Comment by Githook User [ 24/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 rational task order
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/d768ce931a56f006af276f68f575e5fbb28cc0f4

Comment by Githook User [ 22/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate uninstall tests
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/73afd886b7a590aecb7033c87530bbe5bfeb566c

Comment by Githook User [ 22/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate debian-package-build task
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/0ea7aa88dff5594c932305f779f1bbd431a48ca9

Comment by Githook User [ 18/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate more Evergreen tasks
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/3166d61d195a0b4c660d9bbba9c32209b9e70853

Comment by Githook User [ 10/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate link tasks with Python
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/7f1f18c6fe97aec9d11ad3de7e4d7bd87a7b6508

Comment by Githook User [ 08/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate all compile tasks w/ python

Also fix RDTSCP compile task.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/4a26e2526f659acf44e3c065c28da7be73bbd5d7

Comment by Githook User [ 03/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 fix address sanitizer build task
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/fc6bf710dc7b3ea250eeb6958860de12fe1efe11

Comment by Githook User [ 03/Oct/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate some compile tasks in Python

Also fix ENABLE_RDTSCP test.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e5dbb4ea3aef4f3f592d65280c940c3f73df0e2f

Comment by Githook User [ 28/Sep/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 rearrange config.yml

Put all compile steps at the top, release-engineering tests next, and
integration tests next.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/d9f8e19dbaba3836f86997fc3fa611646218923b

Comment by Githook User [ 28/Sep/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 remove debug-compile-valgrind

All compilation for valgrind tests includes BSON_MEMCHECK, so merge
the debug-compile-valgrind-memcheck task into debug-compile-valgrind.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/cac5b99ad5e6032800ad4d9d565d7436ce74f0de

Comment by Githook User [ 20/Sep/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 simpler config.yml.template

Move handwritten tests from include files into the template, generate
some auth tests from a matrix defined in Python, and format tasks with a
YAML formatter instead of a template.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/42c9daa16fe52083a353a73287e2904389552acb

Comment by Githook User [ 20/Sep/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 use SSL build to test ASAN with SSL
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/0e4accd87bef635191e4ab8675869893d5016bb1

Comment by Githook User [ 10/Sep/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 more replica set and memcheck tests

Always set BSON_MEMCHECK for valgrind and ASAN tests, and add replica
set auth tests to the valgrind, ASAN, and code-coverage matrixes. Reduce
the number of special rules and exceptions in the matrix.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/a55f91d75c564344aa9ce0cd74ecca46b73788ff

Comment by Githook User [ 08/Aug/18 ]

Author:

{'username': 'ajdavis', 'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com'}

Message: CDRIVER-2479 generate task order and names
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/f12879625ff8582868355ff39fe50ff683ae26c3

Comment by Githook User [ 03/Aug/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2479 generate evergreen tasks w/ script

Begin to generate repetitive task definitions from a shorter Python
script. In order to create an identical config.yml the Python script
is full of hacks and reproduces some unintentional inconsistencies in
the handwritten config.yml.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/ea29177af4d347616b2213016dac59c59e2b0eb7

Generated at Wed Feb 07 21:15:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.