[SERVER-63191] Create Task that Generates Version Metadata Earlier Created: 03/Dec/21  Updated: 29/Oct/23  Resolved: 13/Jun/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.2, 6.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Vivian Ge (Inactive) Assignee: Ryan Egesdahl (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
causes SERVER-73368 Evergreen push task is failing to upd... Closed
Related
related to SERVER-68568 future_git_tag should override MONGO_... Closed
related to SERVER-69127 Missing expansion for curator --versi... Closed
is related to SERVER-67717 ${suffix} is not set in push task Closed
is related to SERVER-66804 Remove check_binary_version from test... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0, v5.0, v4.4, v4.2
Sprint: Dev Platform 2022-05-16, Dev Platform 2022-05-30, Dev Platform 2022-06-13
Participants:
Linked BF Score: 135

 Description   

As an outcome of the post-mortem from November 29th regarding Incorrect 5.1.0 GA Binaries, the following action item has been identified below. Please note, the 5.2.0 GA release is scheduled for January 19th. 

  • To eliminate the problem of having an overwrite, derive the release version earlier/uniformly when the task is run by having an earlier task that generates version metadata that all the subsequent compile tasks download.


 Comments   
Comment by Githook User [ 12/Aug/22 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-63191 Fix Windows builds with generated version expansions

(cherry picked from commit 131ea34b9d629fc0f74dad8d664d724b569363e1)
Branch: v6.0
https://github.com/mongodb/mongo/commit/713e21b3bef3353c888e5b0006a5d2d6aad62692

Comment by Githook User [ 12/Aug/22 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-63191 Generate build expansions early

(cherry picked from commit 2836afbf9ae9244cae0a77e39ae9e59f46441ce8)
Branch: v6.0
https://github.com/mongodb/mongo/commit/6fdbde504f6f699eadedb9c26e96f6fce8bafcc0

Comment by Githook User [ 09/Jun/22 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-63191 Fix Windows builds with generated version expansions
Branch: master
https://github.com/mongodb/mongo/commit/131ea34b9d629fc0f74dad8d664d724b569363e1

Comment by Githook User [ 09/Jun/22 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-63191 Generate build expansions early
Branch: master
https://github.com/mongodb/mongo/commit/2836afbf9ae9244cae0a77e39ae9e59f46441ce8

Comment by Githook User [ 01/Jun/22 ]

Author:

{'name': 'Uladzimir Makouski', 'email': 'uladzimir.makouski@mongodb.com', 'username': 'umakouski'}

Message: Revert "SERVER-63191 Generate build expansions early"

This reverts commit 79c61c07d61102ca7ca387880a23543faf156ca3.
Branch: master
https://github.com/mongodb/mongo/commit/be6bf2e5c2bfda8bdfec13419afabbfded6b432b

Comment by Githook User [ 31/May/22 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-63191 Generate build expansions early
Branch: master
https://github.com/mongodb/mongo/commit/79c61c07d61102ca7ca387880a23543faf156ca3

Comment by David Bradford (Inactive) [ 05/Jan/22 ]

I have been experimenting with having cross-build variant dependencies as described here. I don't have anything that I would expect to get merged in the short-term, but I have run a few patch builds. One thing to watch out for is EVG-16026, which I hit in my patch builds. I was seeing all dependent tasks that were part of a cross-build-variant dependency get activated in run, which I would not expect. I don't know if it was something strange in my patch or an evergreen issue, but it might be something to watch out for.

Comment by Andrew Morrow (Inactive) [ 05/Jan/22 ]

ryan.egesdahl - That sounds like the right approach, to have one dependency that is shared across all variants. I believe david.bradford is pursuing something similar w.r.t. task generation so it may be helpful to sync up with him too. I also wonder whether it might be possible to avoid needing to transmogrify the expansions (which sounds tricky and possessed of low observability) by instead creating and using multiple expansions files: one that gets generated per commit/build, and another that gets generated within each variant. Potentially, there could be yet another level at the task level, if we ever needed task specific expansions like that. I also like the idea that the expansions would be pushed to s3 as a task artifact, because it would make it much easier to see what was actually generated.

Comment by Ryan Egesdahl (Inactive) [ 05/Jan/22 ]

acm, robert.guo After more thought and some experimentation, I think there's a general solution to this problem, but it might require some fairly extensive plumbing to accomplish. What we would have to do is generate an initial expansions.yml with global build data and upload it to S3 as a build artifact in a task depended upon by all build variants. We would then need to update those expansions for each build variant by downloading expansions.yml first, running a script to add specific platform-specific expansions, and then applying expansions.update to the file.

I'll see if I can run an experiment on this method at some point this week to verify that it would work. I'll let you know what happens.

Comment by Ryan Egesdahl (Inactive) [ 20/Dec/21 ]

acm robert.guo If I am reading this correctly, I think that in the specific case of MONGO_VERSION, we should be able to just pre-generate a version.json and upload it to S3 as a build artifact. We need it for the tarball later on anyway, so we can just treat it as a target we hit before we actually compile any code and then try to download it from S3 before we generate it. I'll just have to set up a target for it and then somehow make sure it gets done as part of the default target as well. There will also need to be some wiring up done with all the build variants as well so they all end up sharing this one build artifact, too.

One problem I can predict with this approach is that there could be a (very) brief window for a race if two build variants start at almost the same time, both have to generate `version.json`, and yet a new tag gets added in the probably few seconds or minutes it takes to generate the file and upload it to S3. I don't think it's a very high risk, though. It's probably not something I could solve here anyway, and it does at least drastically narrow the window where something awful could happen.

I don't think I know how to do the broader ask to generate all the compile expansions earlier yet. I'd vote to consider that question a bit more deeply, though, since it seems we're starting to ask Evergreen to share singleton data between build variants, which is something it's not really designed to do.

Comment by Robert Guo (Inactive) [ 07/Dec/21 ]

acm I think that's very reasonable and should get us a step closer to having a single compile task for multiple variants. One thing that came to mind wrt implementation is that there're may be some edge-case behavior that's different for task groups (used by compile); E.g. EVG-15188. You might encounter something here.

An aside, I don't have much context on the issue here beyond what's in your comment, which is very clear btw. But I'm not sure how/why this version metadata issue would affect downstream and wonder if it's worth adding some checks there for production releases. No need to respond if this doesn't seem relevant or if you've already considered it.

Generated at Thu Feb 08 05:57:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.