[SERVER-71954] Evergreen tasks do not always depend on archive_dist_test_debug Created: 07/Dec/22 Updated: 29/Oct/23 Resolved: 19/Jan/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 6.3.0-rc0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | George Wangensteen | Assignee: | Robert Guo (Inactive) |
| Resolution: | Fixed | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Assigned Teams: |
Server Development Platform
|
||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Description |
|
Some evergreen tasks that run server tests (i.e. https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel80_dynamic_v4gcc_debug_experimental_display_tenant_migration_stepdown_jscore_passthrough_1761823f3a34b56a39816b774f5d632e0e67ca91_22_12_07_03_54_19/execution-tasks?execution=0&sorts=STATUS%3AASC ) do not depend on "archive_dist_test_debug". This means that when `evergreen fetch` is used to fetch artifacts associated with such a task, debug symbols are not gathered. In practice, this means that when server engineers spawn hosts from tasks to investigate failures, any core dumps are effectively unusable because the debug symbols for the binaries and shared libraries are not fetched to the machine. This also means the setup_spawnhost_coredump script cannot setup the machine for easy debugging, as the debug symbols are not on the machine when the script starts running. The workaround is for server engineers to manually find the archive_dist_test_debug task matching their build and manually move the symbols (using curl or similar) to the spawned host, unpack them, put them in the appropriate directory, and configure gdb to find the symbols. Since these steps were previously automated by the setup_spawnhost_coredump script and evergreen fetch, this has added a lot of friction to the core-dump debugging workflow for server engineers. |
| Comments |
| Comment by Githook User [ 19/Jan/23 ] |
|
Author: {'name': 'Rob Guo', 'email': 'robert.guo@mongodb.com'}Message: |
| Comment by Robert Guo (Inactive) [ 10/Jan/23 ] |
|
Quick update: For getting binaries set up, please consider using db-contrib-tool setup-repro-env in lieu of evergreen fetch as it is more customizable for Server needs. Install instructions are here. Please let me know of any issues. Additionally, I'm working on updating setup_spawnhost_coredump to support the new task dependencies. Apologies for the friction in the initial rollout here. We'd like to work through the breakages to help Server's CI experience scale by eliminating stubbornly serial portions. |
| Comment by Alex Neben [ 09/Jan/23 ] |
|
Now that robert.guo@mongodb.com is back I am assigning this to him. |
| Comment by Alex Neben [ 08/Dec/22 ] |
|
I think this is just a variant I missed when I had originally reverted these changes. v4gcc builders are going to gone soon anyway so we might be able to just mark this one as resolved once v4 merges (hopefully this week or next week). |