[SERVER-34330] Split up long-running Evergreen tasks into shorter execution units Created: 05/Apr/18  Updated: 29/Oct/23  Resolved: 20/Apr/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.7.6

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: David Bradford (Inactive)
Resolution: Fixed Votes: 0
Labels: tig-evgconfig
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-34296 Create Python script to generate resm... Closed
Gantt Dependency
Related
related to SERVER-34862 Avoid repeating list of execution tas... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2018-04-23, TIG 2018-05-07
Participants:
Linked BF Score: 0

 Description   

The following test suites when run on the Enterprise RHEL 6.2, Linux DEBUG, and Enterprise Windows 2008R2 cause the makespan to exceed 1 hour. We should use the script from SERVER-34296 to split them into test suites that take no longer than 20 minutes in order to reap concurrency benefits by running on separate EC2 instances. The Evergreen tasks added to the etc/evergreen.yml project configuration should be done under a display task of the original name in order to limit the Evergreen UI differences Server team engineers would otherwise encounter.

  • replica_sets_auth.yml (~1h on Enterprise Windows 2008R2)
  • replica_sets_ese.yml (~1h on Enterprise Windows 2008R2)
  • sharding.yml (~50m on Enterprise RHEL 6.2, ~1h20m on Linux DEBUG)
  • sharding_auth.yml (~50m on Enterprise RHEL 6.2, ~1h10m on Linux DEBUG, ~1h40m on Enterprise Windows 2008R2)
  • sharding_ese.yml (~50m on Enterprise RHEL 6.2, ~1h50m on Enterprise Windows 2008R2)
  • sharding_last_stable_mongos_and_mixed_shards.yml (~50m on Enterprise RHEL 6.2)
  • sharding_op_query.yml (~50m on Enterprise RHEL 6.2)
  • sharding_auth_audit.yml (~45m on Enterprise RHEL 6.2, ~1h45m on Enterprise Windows 2008R2, but only runs after sharding_auth Evergreen task completes)

https://evergreen.mongodb.com/build/mongodb_mongo_master_enterprise_rhel_62_64_bit_c3badcbbcf069f428a765ba5937106d1da814076_18_04_05_00_46_39
https://evergreen.mongodb.com/build/mongodb_mongo_master_linux_64_debug_f00b908d2bf6cca4c2527eaa88b0ae79d745fd0b_18_04_05_02_02_23
https://evergreen.mongodb.com/build/mongodb_mongo_master_enterprise_windows_64_2k8_2e31167d0e2cf179a228afeec71853cfb0f0683f_18_04_05_00_06_12

Note: The write_concern_majority_passthrough Evergreen task takes ~50m on Linux DEBUG; however, that seems more likely to be due to the removeb.js, mr1.js, remove9.js, geo_s2sparse.js, and queryoptimizer3.js tests taking longer than 15 minutes. We should look to modify these tests in a similar fashion to the changes in SERVER-33645 under a separate ticket.



 Comments   
Comment by Githook User [ 20/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: split up sharding_auth and sharding_auth_audit suites
Branch: master
https://github.com/mongodb/mongo/commit/b0a3504b94d7a940edffc679d62c96a8cfe07811

Comment by Githook User [ 19/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: Split sharding_op_query suite into multiple units
Branch: master
https://github.com/mongodb/mongo/commit/85a811117b9d0bfd6d3d147ce773e0abc5b0c8a2

Comment by Githook User [ 19/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: Split sharding_last_stable_mongos_and_mixed_shards into smaller units
Branch: master
https://github.com/mongodb/mongo/commit/f4652f501d7b5642a1ad965bc8eefad7803bd120

Comment by Githook User [ 19/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: Split sharding_ese suite into smaller units
Branch: master
https://github.com/mongodb/mongo/commit/f769ba3ff8815d4264aa50d5196d450a7b3f9f87

Comment by Githook User [ 19/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: Add excluded test to replica-sets-auth-misc suite
Branch: master
https://github.com/mongodb/mongo/commit/8c2515904ee5bd8a592fde85fa0295c8ccb96510

Comment by Githook User [ 19/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: Split up replica_sets_ese suite into smaller suites
Branch: master
https://github.com/mongodb/mongo/commit/eeded592270082dba3dd59d65c8e94128030af18

Comment by Githook User [ 19/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'username': 'dbradf', 'name': 'David Bradford'}

Message: SERVER-34330: Split up replica_sets_auth suite
Branch: master
https://github.com/mongodb/mongo/commit/d4a0c4547d1f83b13d9f6c7909d3120525f18d4f

Comment by Githook User [ 18/Apr/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'name': 'David Bradford', 'username': 'dbradf'}

Message: SERVER-34330: Split up sharding test suite into smaller units
Branch: master
https://github.com/mongodb/mongo/commit/32b1be84310f2bdb617f79d50b5a5b8b44a99bed

Comment by Githook User [ 16/Apr/18 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-34330: Include a misc test suite to catch new tests
Branch: master
https://github.com/mongodb/mongo/commit/301affcaa17d18c89e735d83c72ca9bdf8f76f1e

Comment by Githook User [ 16/Apr/18 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-34330: Fix python lint errors
Branch: master
https://github.com/mongodb/mongo/commit/da8b671ce73cc61f6b4f5ebbcefdc76977324c6b

Comment by Max Hirschhorn [ 13/Apr/18 ]

david.bradford, I realized that I omitted an important detail from the description of this ticket that had been brought up in the scope document. In addition to generating the sharding_000.yml, etc. test suites, the script should also generate a sharding_misc.yml test suite that excludes all JavaScript tests being run elsewhere. This ensures that new JavaScript tests added the to jstests/sharding/ directory after SERVER-34330 are still run in Evergreen.

Comment by Max Hirschhorn [ 05/Apr/18 ]

kyle.suarez, we're going to remove the compile_all task from the "bang builder" version of the Enterprise Windows 2008R2 build variant. It'll still run once an hour but you won't be required to patch build on it.

Comment by Kyle Suarez [ 05/Apr/18 ]

Great for jstests, though I'm frequently annoyed by compile_all on Enterprise Windows, which consistently takes over two hours.

Generated at Thu Feb 08 04:36:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.