[SERVER-63191] Create Task that Generates Version Metadata Earlier Created: 03/Dec/21 Updated: 29/Oct/23 Resolved: 13/Jun/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.0.2, 6.1.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Vivian Ge (Inactive) | Assignee: | Ryan Egesdahl (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v6.0, v5.0, v4.4, v4.2
|
||||||||||||||||||||||||||||||||||||
| Sprint: | Dev Platform 2022-05-16, Dev Platform 2022-05-30, Dev Platform 2022-06-13 | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 135 | ||||||||||||||||||||||||||||||||||||
| Description |
|
As an outcome of the post-mortem from November 29th regarding Incorrect 5.1.0 GA Binaries, the following action item has been identified below. Please note, the 5.2.0 GA release is scheduled for January 19th.
|
| Comments |
| Comment by Githook User [ 12/Aug/22 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: (cherry picked from commit 131ea34b9d629fc0f74dad8d664d724b569363e1) |
| Comment by Githook User [ 12/Aug/22 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: (cherry picked from commit 2836afbf9ae9244cae0a77e39ae9e59f46441ce8) |
| Comment by Githook User [ 09/Jun/22 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: |
| Comment by Githook User [ 09/Jun/22 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: |
| Comment by Githook User [ 01/Jun/22 ] |
|
Author: {'name': 'Uladzimir Makouski', 'email': 'uladzimir.makouski@mongodb.com', 'username': 'umakouski'}Message: Revert " This reverts commit 79c61c07d61102ca7ca387880a23543faf156ca3. |
| Comment by Githook User [ 31/May/22 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: |
| Comment by David Bradford (Inactive) [ 05/Jan/22 ] |
|
I have been experimenting with having cross-build variant dependencies as described here. I don't have anything that I would expect to get merged in the short-term, but I have run a few patch builds. One thing to watch out for is EVG-16026, which I hit in my patch builds. I was seeing all dependent tasks that were part of a cross-build-variant dependency get activated in run, which I would not expect. I don't know if it was something strange in my patch or an evergreen issue, but it might be something to watch out for. |
| Comment by Andrew Morrow (Inactive) [ 05/Jan/22 ] |
|
ryan.egesdahl - That sounds like the right approach, to have one dependency that is shared across all variants. I believe david.bradford is pursuing something similar w.r.t. task generation so it may be helpful to sync up with him too. I also wonder whether it might be possible to avoid needing to transmogrify the expansions (which sounds tricky and possessed of low observability) by instead creating and using multiple expansions files: one that gets generated per commit/build, and another that gets generated within each variant. Potentially, there could be yet another level at the task level, if we ever needed task specific expansions like that. I also like the idea that the expansions would be pushed to s3 as a task artifact, because it would make it much easier to see what was actually generated. |
| Comment by Ryan Egesdahl (Inactive) [ 05/Jan/22 ] |
|
acm, robert.guo After more thought and some experimentation, I think there's a general solution to this problem, but it might require some fairly extensive plumbing to accomplish. What we would have to do is generate an initial expansions.yml with global build data and upload it to S3 as a build artifact in a task depended upon by all build variants. We would then need to update those expansions for each build variant by downloading expansions.yml first, running a script to add specific platform-specific expansions, and then applying expansions.update to the file. I'll see if I can run an experiment on this method at some point this week to verify that it would work. I'll let you know what happens. |
| Comment by Ryan Egesdahl (Inactive) [ 20/Dec/21 ] |
|
acm robert.guo If I am reading this correctly, I think that in the specific case of MONGO_VERSION, we should be able to just pre-generate a version.json and upload it to S3 as a build artifact. We need it for the tarball later on anyway, so we can just treat it as a target we hit before we actually compile any code and then try to download it from S3 before we generate it. I'll just have to set up a target for it and then somehow make sure it gets done as part of the default target as well. There will also need to be some wiring up done with all the build variants as well so they all end up sharing this one build artifact, too. One problem I can predict with this approach is that there could be a (very) brief window for a race if two build variants start at almost the same time, both have to generate `version.json`, and yet a new tag gets added in the probably few seconds or minutes it takes to generate the file and upload it to S3. I don't think it's a very high risk, though. It's probably not something I could solve here anyway, and it does at least drastically narrow the window where something awful could happen. I don't think I know how to do the broader ask to generate all the compile expansions earlier yet. I'd vote to consider that question a bit more deeply, though, since it seems we're starting to ask Evergreen to share singleton data between build variants, which is something it's not really designed to do. |
| Comment by Robert Guo (Inactive) [ 07/Dec/21 ] |
|
acm I think that's very reasonable and should get us a step closer to having a single compile task for multiple variants. One thing that came to mind wrt implementation is that there're may be some edge-case behavior that's different for task groups (used by compile); E.g. EVG-15188. You might encounter something here. An aside, I don't have much context on the issue here beyond what's in your comment, which is very clear btw. But I'm not sure how/why this version metadata issue would affect downstream and wonder if it's worth adding some checks there for production releases. No need to respond if this doesn't seem relevant or if you've already considered it. |