[SERVER-31140] Add RHEL 7 shadow builders to validate memory correctness Created: 18/Sep/17  Updated: 30/Oct/23  Resolved: 03/Oct/17

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 3.6.0-rc0

Type: Task Priority: Major - P3
Reporter: Andrew Morrow (Inactive) Assignee: Eddie Louie
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-31279 Adjust testing frequency of Windows 2... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-10-23
Participants:
Linked BF Score: 25

 Description   

We have two longstanding memory corruption issues that reoccur with low frequency. As it happens, both builders where we observe this corruption run on RHEL 6.2.

In the interest of eliminating the hypothesis that this is a RHEL 6.2 specific bug, we would like to make some changes to the set of builders we run.

Specifically, we would like to:

  • Increase the frequency of the Linux DEBUG builder by adding it to the "star" builders, with the same cadence as the existing star builders.
  • Create a new "star" builder that is the same as Linux DEBUG, but runs on RHEL 7 rather than RHEL 6, again with same cadence.
  • Increase the frequency of the Enterprise RHEL 7 by adding it to the "star" builders, again, same cadence as others.


 Comments   
Comment by Githook User [ 03/Oct/17 ]

Author:

{'email': 'eddie.louie@mongodb.com', 'name': 'Eddie Louie', 'username': 'elouie99'}

Message: SERVER-31140 Add RHEL 7 shadow builder to validate memory correctness
Branch: master
https://github.com/mongodb/mongo/commit/3a2666842f2a6ae57fdb09ccaedc629ca5871e67

Comment by Max Hirschhorn [ 28/Sep/17 ]

This code will get the tasks that are in the Linux builder and not in the Windows builder. Did you mean to do the opposite? i.e get the tasks in the Windows builder that are not in the Linux builder?

Oops, I meant to do the set difference in the opposite order. I've fixed my earlier comment.

Comment by Eddie Louie [ 28/Sep/17 ]

max.hirschhorn Need clarification of your statement above:

Update the "Linux DEBUG" builder to include all of the Evergreen tasks that also run on the "Windows 2008R2 DEBUG" builder. The output below shows which ones are missing.

Followed by this script:
python -c 'from buildscripts.ciconfig.evergreen import EvergreenProjectConfig; config = EvergreenProjectConfig("etc/evergreen.yml"); print("\n".join(sorted(set(config.get_variant("linux-64-debug").task_names) - set(config.get_variant("windows-64-2k8-debug").task_names))))'

This code will get the tasks that are in the Linux builder and not in the Windows builder. Did you mean to do the opposite? i.e get the tasks in the Windows builder that are not in the Linux builder?

Comment by Eric Milkie [ 27/Sep/17 ]

Ah. To be clear, the only part of the original description to be superceded is the first bullet, regarding moving the Linux DEBUG builder to the star list. It will instead move to the bang list. SGTM

Comment by Andrew Morrow (Inactive) [ 27/Sep/17 ]

milkie - Max's steps don't supersede my requests here. We have repeated instances of memory corruption that have only ever happened on RHEL 6.2 builders. We need the new RHEL 7 DEBUG builder to do differential diagnosis of the DEBUG builds, and we need the Enterprise RHEL 7 build running with more frequency to see if we also observe the credential mismatch issue there. I propose that the list should look like:

  • ! Enterprise RHEL 6.2
  • ! Enterprise Ubuntu 16.04 (Clang 3.8)
  • ! Enterprise Windows 2008R2
  • ! {A,UB}SAN Enterprise SSL Ubuntu 16.04 DEBUG
  • ! Linux DEBUG (Change)
  • * OS X 10.10 DEBUG
  • * Shared Library Enterprise Ubuntu 16.04
  • * Windows 2008R2 DEBUG
  • * RHEL 7 DEBUG (Change)
  • * Enterprise RHEL 7.0 (Change)
Comment by Eric Milkie [ 27/Sep/17 ]

While we're discussing this, I'd like to bring up the possibility of making the Debug builder build the enterprise module and associated test suites – perhaps it's a good opportunity to add this coverage?

Comment by Andrew Morrow (Inactive) [ 27/Sep/17 ]

If we are going to do this work can we please add the "* RHEL 7 Debug" builder, and move "Enterprise RHEL 7" to "*" status?

Comment by Eric Milkie [ 27/Sep/17 ]

I think we should specify exactly how Max's steps 1 and 2 supercede (if at all) the three bullets in the original Description. Are we going to have Linux DEBUG run on RHEL 6 or 7 or both?

Comment by Max Hirschhorn [ 27/Sep/17 ]

Given the changes from SERVER-31279 which removed the "Windows 2008R2 DEBUG" builder from the required builders list, I think we'll want to

  1. Add the "Linux DEBUG" builder to the required builders list. (by prefixing its display name with "!")
  2. Update the "Linux DEBUG" builder to include all of the Evergreen tasks that also run on the "Windows 2008R2 DEBUG" builder. The output below shows which ones are missing.

$ python -c 'from buildscripts.ciconfig.evergreen import EvergreenProjectConfig; config = EvergreenProjectConfig("etc/evergreen.yml"); print("\n".join(sorted(set(config.get_variant("windows-64-2k8-debug").task_names) - set(config.get_variant("linux-64-debug").task_names))))'
aggregation_facet_unwind_passthrough_WT
causally_consistent_jscore_passthrough_WT
causally_consistent_jscore_passthrough_auth_WT
change_streams_WT
change_streams_mongos_passthrough_WT
concurrency_sharded_causal_consistency
concurrency_sharded_causal_consistency_WT
concurrency_sharded_causal_consistency_and_balancer
concurrency_sharded_causal_consistency_and_balancer_WT
concurrency_sharded_with_stepdowns
concurrency_sharded_with_stepdowns_WT
concurrency_sharded_with_stepdowns_and_balancer
concurrency_sharded_with_stepdowns_and_balancer_WT
integration_tests_replset
integration_tests_sharded
integration_tests_standalone
jsonSchema
read_only
read_only_WT
read_only_sharded
read_only_sharded_WT
replica_sets_initsync_jscore_passthrough
replica_sets_initsync_jscore_passthrough_WT
replica_sets_initsync_static_jscore_passthrough
replica_sets_initsync_static_jscore_passthrough_WT
replica_sets_kill_secondaries_jscore_passthrough_WT
replica_sets_resync_static_jscore_passthrough
replica_sets_resync_static_jscore_passthrough_WT
retryable_writes_jscore_passthrough_WT
session_jscore_passthrough_WT
sharded_causally_consistent_jscore_passthrough_WT

CC ramon.fernandez, acm, milkie

Generated at Thu Feb 08 04:26:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.