[SERVER-74523] Upgrade abseil to 2023-08-02 LTS Created: 01/Mar/23 Updated: 08/Dec/23 Resolved: 07/Dec/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.3.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Billy Donahue | Assignee: | Patrick Freed |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||
| Sprint: | Service Arch 2023-03-06 | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Description |
|
We are using an old version. Motivation (from Slack thread in pm-1812 channel):
Staying up to date is no longer optional, as Google's other upstream products that we need (tcmalloc, grpc) are going to be moving forward with dependencies on "head" abseil. We'd need a strong reason to defer upgrading. About 1 month ago, a revision tag was introduced upstream.
The upstream is tracked by our https://github.com/mongodb-forks/abseil-cpp/ repo, which is the direct origin of our third_party lib. Fortunately, we no longer maintain site-specific customizations. (depended on by All LTS branches as of right now:
Minimum required Abseil for TCMalloc hasn't been figured out (not provided by upstream docs), but we can proceed without knowing it. |
| Comments |
| Comment by Githook User [ 08/Dec/23 ] |
|
Author: {'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}Message: GitOrigin-RevId: a132fa5e364fbee6b9ca3dc8b795b90266451fd9 |
| Comment by Githook User [ 07/Dec/23 ] |
|
Author: {'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}Message: GitOrigin-RevId: e67cba3fc97dbc3a877a75c8863c62f09fc3666d |
| Comment by Githook User [ 06/Mar/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: |
| Comment by Billy Donahue [ 02/Mar/23 ] |
|
Daniel, as discussed in our offline conversation I'll hand this ticket off to you. I can pair with you and take on some of these tasks and do expedited reviews if needed. It's only the abseil upgrade. You've gotten further with abseil vendoring reimplementation in your grpc work than I have from the TCMalloc side. We'll break off at least the Abseil part of the grpc ticket into a separate commit, forming an up-to-date abseil baseline from which to make grpc and tcmalloc branches. This would be much smaller than the full monolithic grpc PR, which brings in all of grpc's third party deps in one big leap. We discussed possibly vendor grpc's other deps separately as well (but I'm not involved in those), even if they are unused "dead code" at first. This makes a trail of separable revertible and reviewable changes. I'll try to capture the plan we discussed, in order of execution.
(this comment raced with your last comment above, and it sounds like they are in agreement). |
| Comment by Daniel Moody [ 02/Mar/23 ] |
|
Billy, in our meeting we talked about me picking up the abseil upgrade and and doing to PRs (commits) the first being upgrading the structure and process of abseil maintaining the existing version. Then a follow on PR which upgrades abseil to the latest. I need to test the latest abseil with the GRPC we selected and verify the API all works. It seemed you verified the API of the latest abseil was compat with the mongo server so that gives me some hope it will be the same case for GRPC. I'll assign this to myself and should have something in the next few days (most of the heavy lifting work is already done via previous grpc work). |
| Comment by Billy Donahue [ 02/Mar/23 ] |
|
Daniel, first off, I love the new ninja->SConscript generator script. Needs docs but it's a great idea and should help us stay up to date more easily. I have been working on a manual version of that process for a few days here and it's a drag sorting through the 8000 total distributed lines of Blaze and reorganizing it into a graph. This manual work would have to be repeated on every upgrade. I have some thoughts about the process that we might want to discuss in person. I see that your change is doing several things at once and I think it would be easier to work around it (and commit it) if it was separated into separate commits. |
| Comment by Daniel Moody [ 02/Mar/23 ] |
|
btw I mentioned "it should be merging soon", we have been working on improving the evergreen scons cache situation and feel it is near ready for another attempt to merge it. Unfortunately the cache only failed when it was subjected to scale after the merge. I did a lot of patch builds prior to merge and never saw any cache issues. |
| Comment by Daniel Moody [ 02/Mar/23 ] |
|
There are two tickets, I did the original addition of grpc with an older version than was finally settled on in https://jira.mongodb.org/browse/PM-2922. Original GRPC addition (merged then reverted due to evergreen cache capacity limitations): Right behind that was the update of GRPC (which never made it to review because of the revert of the other ticket): GRPC needs alot more of abseil than we currently use, so you may notice I am generating the abseil SConscript from the script which parses the generated ninja file from the abseil cmake build. |
| Comment by Billy Donahue [ 02/Mar/23 ] |
|
Daniel do you have a ticket for the grpc work you're referring to? Can you link it here? I don't see a code review for it on the 10gen repo. I need to see if it will conflict with what I'm trying to do here, and I want to avoid interference with each other's efforts. I'm almost done with the change to upgrade only abseil, which I would want to keep as a separate branch here until we're ready to commit it to master. |
| Comment by Daniel Moody [ 02/Mar/23 ] |
|
we will need to upgrade abseil for use with grpc 1.46.6, which requires at least 20211102.0. It should be merging soon. |
| Comment by Billy Donahue [ 01/Mar/23 ] |
|
SERVER-54747 involved a prior effort here. That project was focused on TCMalloc, but had to handle the mutual dependency between TCMalloc and Abseil, requiring an Abseil upgrade and extensive SConscript hacking. |
| Comment by Billy Donahue [ 01/Mar/23 ] |
|
I'm creating this ticket as a placeholder for the upgrade we'll eventually need to do. I also need this place to attach a persistent discoverable record of efforts to track the vendoring of this newer abseil release, as its internal layout has changed. The artifacts created in the translation of of upstream's Bazel files to mongo third_party Sconscript files can be tracked here. |