[SERVER-71954] Evergreen tasks do not always depend on archive_dist_test_debug Created: 07/Dec/22  Updated: 29/Oct/23  Resolved: 19/Jan/23

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Bug Priority: Critical - P2
Reporter: George Wangensteen Assignee: Robert Guo (Inactive)
Resolution: Fixed Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
is caused by SERVER-65040 Reuse build artifacts across builders... Closed
Related
related to SERVER-71113 Tasks generated do not have access to... Closed
related to SERVER-79904 Investigate changing resmoke dependen... Closed
related to SERVER-73006 consider disabling separate debug for... Closed
is related to SERVER-59605 setup_spawnhost_coredump no longer ha... Closed
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Some evergreen tasks that run server tests (i.e. https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel80_dynamic_v4gcc_debug_experimental_display_tenant_migration_stepdown_jscore_passthrough_1761823f3a34b56a39816b774f5d632e0e67ca91_22_12_07_03_54_19/execution-tasks?execution=0&sorts=STATUS%3AASC ) do not depend on "archive_dist_test_debug". This means that when `evergreen fetch` is used to fetch artifacts associated with such a task, debug symbols are not gathered. 

In practice, this means that when server engineers spawn hosts from tasks to investigate failures, any core dumps are effectively unusable because the debug symbols for the binaries and shared libraries are not fetched to the machine. This also means the setup_spawnhost_coredump script cannot setup the machine for easy debugging, as the debug symbols are not on the machine when the script starts running. The workaround is for server engineers to manually find the archive_dist_test_debug task matching their build and manually move the symbols (using curl or similar) to the spawned host, unpack them, put them in the appropriate directory, and configure gdb to find the symbols.

Since these steps were previously automated by the setup_spawnhost_coredump script and evergreen fetch, this has added a lot of friction to the core-dump debugging workflow for server engineers. 



 Comments   
Comment by Githook User [ 19/Jan/23 ]

Author:

{'name': 'Rob Guo', 'email': 'robert.guo@mongodb.com'}

Message: SERVER-71954 add missing archive_dist_test_debug dependency
Branch: master
https://github.com/mongodb/mongo/commit/cc87678aa5904d324f34f0d7884721bab058df4b

Comment by Robert Guo (Inactive) [ 10/Jan/23 ]

Quick update: For getting binaries set up, please consider using db-contrib-tool setup-repro-env in lieu of evergreen fetch as it is more customizable for Server needs. Install instructions are here. Please let me know of any issues.

Additionally, I'm working on updating setup_spawnhost_coredump to support the new task dependencies. Apologies for the friction in the initial rollout here. We'd like to work through the breakages to help Server's CI experience scale by eliminating stubbornly serial portions.

Comment by Alex Neben [ 09/Jan/23 ]

Now that robert.guo@mongodb.com is back I am assigning this to him.

Comment by Alex Neben [ 08/Dec/22 ]

I think this is just a variant I missed when I had originally reverted these changes. v4gcc builders are going to gone soon anyway so we might be able to just mark this one as resolved once v4 merges (hopefully this week or next week).

Generated at Thu Feb 08 06:20:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.