[SERVER-74523] Upgrade abseil to 2023-08-02 LTS Created: 01/Mar/23  Updated: 08/Dec/23  Resolved: 07/Dec/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Billy Donahue Assignee: Patrick Freed
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-75761 Upgrade gRPC third-party libraries Closed
is depended on by SERVER-74420 Set up the import of tcmalloc Closed
Fix
fixes SERVER-83522 absl::hash doesn't guarantee hashing ... Closed
Related
is related to SERVER-54747 Vendor tcmalloc into MongoDB's third ... Blocked
is related to SERVER-51476 Upgrade Abseil to 20210324.1 Closed
Backwards Compatibility: Fully Compatible
Sprint: Service Arch 2023-03-06
Participants:

 Description   

We are using an old version.
Reading changelogs, the upstream has made significant improvements in the past 2 years.

Motivation (from Slack thread in pm-1812 channel):

Google's open source offerings (tcmalloc, abseil, grpc) have basically discarded semantic versioning and are going with a "live at head" initiative, which is directly harmful to non-google versioned projects like ours. We are going to have a hard time tracking projects like "evaluate tcmalloc upgrade" or "evaluate grpc" without considering holistic effects of upgrading everything everygoogle all at once. We'll have to adapt to stress on "evergreen cache capacity" and incur cognitive load in interpreting experimental branch perf results as well.

Staying up to date is no longer optional, as Google's other upstream products that we need (tcmalloc, grpc) are going to be moving forward with dependencies on "head" abseil. We'd need a strong reason to defer upgrading.

About 1 month ago, a revision tag was introduced upstream.

- LIB_GIT_REVISION=20210324.1
+ LIB_GIT_REVISION=20230125.0

The upstream is tracked by our https://github.com/mongodb-forks/abseil-cpp/ repo, which is the direct origin of our third_party lib.

Fortunately, we no longer maintain site-specific customizations.

(depended on by SERVER-74420)
(related to SERVER-51476)

All LTS branches as of right now:

  remotes/google/lts_2018_06_20
  remotes/google/lts_2018_12_18
  remotes/google/lts_2019_08_08
  remotes/google/lts_2020_02_25
  remotes/google/lts_2020_09_23
  remotes/google/lts_2021_03_24   <= we are here (but a patch release behind the tip of this branch)
  remotes/google/lts_2021_11_02   <= minimum gRPC requirement
  remotes/google/lts_2022_06_23
  remotes/google/lts_2023_01_25

Minimum required Abseil for TCMalloc hasn't been figured out (not provided by upstream docs), but we can proceed without knowing it.



 Comments   
Comment by Githook User [ 08/Dec/23 ]

Author:

{'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}

Message: SERVER-74523 Upgrade gRPC to 1.59.2, protobuf to 4.25.0

GitOrigin-RevId: a132fa5e364fbee6b9ca3dc8b795b90266451fd9
Branch: master
https://github.com/mongodb/mongo/commit/767fc4390da4135223579c1fd93eb7d274241af4

Comment by Githook User [ 07/Dec/23 ]

Author:

{'name': 'Patrick Freed', 'email': 'patrick.freed@mongodb.com', 'username': 'patrickfreed'}

Message: SERVER-74523 Upgrade abseil to v20230802.1

GitOrigin-RevId: e67cba3fc97dbc3a877a75c8863c62f09fc3666d
Branch: master
https://github.com/mongodb/mongo/commit/8bac9845656c694f8f76895e2dc51f7e019fd72f

Comment by Githook User [ 06/Mar/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-74523 fix abseil structure and improve import script.
Branch: master
https://github.com/mongodb/mongo/commit/21d24e6abde15d8c00a69522605706c1d19a00c3

Comment by Billy Donahue [ 02/Mar/23 ]

Daniel, as discussed in our offline conversation I'll hand this ticket off to you. I can pair with you and take on some of these tasks and do expedited reviews if needed. It's only the abseil upgrade. You've gotten further with abseil vendoring reimplementation in your grpc work than I have from the TCMalloc side.

We'll break off at least the Abseil part of the grpc ticket into a separate commit, forming an up-to-date abseil baseline from which to make grpc and tcmalloc branches. This would be much smaller than the full monolithic grpc PR, which brings in all of grpc's third party deps in one big leap. We discussed possibly vendor grpc's other deps separately as well (but I'm not involved in those), even if they are unused "dead code" at first. This makes a trail of separable revertible and reviewable changes.

I'll try to capture the plan we discussed, in order of execution.

  • Directory reorganization: Implement the reorganization of vendored Abseil's layout without changing its version. third_party/abseil-cpp-master to third_party/abseil-cpp, placing source into a dist/ folder, change the import script at the same time so that it produces the new layout, conforming to the mongodb-forks vendoring idiom. This will be compact and history-preserving as git should recognize that everything under abseil is moving but not changing. The current monolithic grpc change shows such files as hundreds of new files and file deletions and we'll avoid that problem with this prerequisite step.
  • SConscript generator: Change the abseil SConscript to be generated from ninja files as part of the vendoring process going forward, and commit the generated SConscript. Have mongo server adapt and depend on the generated abseil library names if they've changed. Optionally provide abstract metalibrary aliases that mongo server depends on, so that server doesn't need to care about abseil's library structure, because this changes between versions.
  • Abseil upgrade: Edit the revision tag in the import.sh script, run the import script, and regenerate the SConscript from the new abseil's ninja artifacts. This should be compact and history-preserving as git will see most files being changed in place but not moving. We discussed using the most up to date LTS Abseil release tag: 20230125.1, instead of the 18-month old version currently slated for the gRPC branch. I've verified that mongo can use it today, and we are going to have to keep up with Abseil's latest no matter what we do, so we should take their best code if we can. The upstream changelogs show very active development and we will want those gains and fixes. If there's a problem with the new version we can fall back to the 18-month old LTS originally planned for the grpc project.
  • gRPC and TCMalloc proceed independently from the upgraded Abseil version as a baseline. Like gRPC, TCMalloc also needs more of Abseil than mongo needs today. With your ninja importing script, it should become super easy to just add the sublibraries that tcmalloc needs to the text file and rerun the ninja->SConscript process.

(this comment raced with your last comment above, and it sounds like they are in agreement).

Comment by Daniel Moody [ 02/Mar/23 ]

Billy, in our meeting we talked about me picking up the abseil upgrade and and doing to PRs (commits) the first being upgrading the structure and process of abseil maintaining the existing version. Then a follow on PR which upgrades abseil to the latest.

I need to test the latest abseil with the GRPC we selected and verify the API all works. It seemed you verified the API of the latest abseil was compat with the mongo server so that gives me some hope it will be the same case for GRPC. I'll assign this to myself and should have something in the next few days (most of the heavy lifting work is already done via previous grpc work).

Comment by Billy Donahue [ 02/Mar/23 ]

Daniel, first off, I love the new ninja->SConscript generator script. Needs docs but it's a great idea and should help us stay up to date more easily. I have been working on a manual version of that process for a few days here and it's a drag sorting through the 8000 total distributed lines of Blaze and reorganizing it into a graph. This manual work would have to be repeated on every upgrade.

I have some thoughts about the process that we might want to discuss in person. I see that your change is doing several things at once and I think it would be easier to work around it (and commit it) if it was separated into separate commits.

Comment by Daniel Moody [ 02/Mar/23 ]

btw I mentioned "it should be merging soon", we have been working on improving the evergreen scons cache situation and feel it is near ready for another attempt to merge it. Unfortunately the cache only failed when it was subjected to scale after the merge. I did a lot of patch builds prior to merge and never saw any cache issues.

Comment by Daniel Moody [ 02/Mar/23 ]

There are two tickets, I did the original addition of grpc with an older version than was finally settled on in https://jira.mongodb.org/browse/PM-2922.

Original GRPC addition (merged then reverted due to evergreen cache capacity limitations):
https://jira.mongodb.org/browse/SERVER-68365

Right behind that was the update of GRPC (which never made it to review because of the revert of the other ticket):
https://jira.mongodb.org/browse/SERVER-73467

GRPC needs alot more of abseil than we currently use, so you may notice I am generating the abseil SConscript from the script which parses the generated ninja file from the abseil cmake build.

Comment by Billy Donahue [ 02/Mar/23 ]

Daniel do you have a ticket for the grpc work you're referring to? Can you link it here?

I don't see a code review for it on the 10gen repo.

I need to see if it will conflict with what I'm trying to do here, and I want to avoid interference with each other's efforts.

I'm almost done with the change to upgrade only abseil, which I would want to keep as a separate branch here until we're ready to commit it to master.

https://github.com/10gen/mongo/pull/10994

Comment by Daniel Moody [ 02/Mar/23 ]

we will need to upgrade abseil for use with grpc 1.46.6, which requires at least 20211102.0. It should be merging soon.

Comment by Billy Donahue [ 01/Mar/23 ]

SERVER-54747 involved a prior effort here. That project was focused on TCMalloc, but had to handle the mutual dependency between TCMalloc and Abseil, requiring an Abseil upgrade and extensive SConscript hacking.

Comment by Billy Donahue [ 01/Mar/23 ]

I'm creating this ticket as a placeholder for the upgrade we'll eventually need to do.

I also need this place to attach a persistent discoverable record of efforts to track the vendoring of this newer abseil release, as its internal layout has changed. The artifacts created in the translation of of upstream's Bazel files to mongo third_party Sconscript files can be tracked here.

Generated at Thu Feb 08 06:27:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.