[SERVER-68475] Find solution to relocation overflow in static builds Created: 01/Aug/22 Updated: 29/Oct/23 Resolved: 01/Sep/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.0.2, 6.1.0-rc1, 6.2.0-rc0 |
| Type: | Improvement | Priority: | Blocker - P1 |
| Reporter: | Daniel Moody | Assignee: | Daniel Moody |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | dp-qol | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Backport Requested: |
v6.1
|
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Linked BF Score: | 160 | ||||||||||||||||||||||||||||
| Description |
|
https://jira.mongodb.org/browse/BF-25986 was a hot issue which we implemented a temporary fix in
This issue should block the 6.1 release because the current temporary fix breaks the ability to easily debug the binaries. |
| Comments |
| Comment by Githook User [ 01/Sep/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit b41c85c2e798f11db81b6c0ba3ca400bbf01f063) | |||||||||||||||||||||
| Comment by Githook User [ 30/Aug/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: Revert " This reverts commit 26054e10384bd0d46de6e047a36083181d872be1. | |||||||||||||||||||||
| Comment by Githook User [ 29/Aug/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit b41c85c2e798f11db81b6c0ba3ca400bbf01f063) | |||||||||||||||||||||
| Comment by Githook User [ 29/Aug/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit 53aa9bec47e34bd010b5dc6b751474c74c3a1797) | |||||||||||||||||||||
| Comment by Githook User [ 29/Aug/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit b41c85c2e798f11db81b6c0ba3ca400bbf01f063) | |||||||||||||||||||||
| Comment by Githook User [ 29/Aug/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: | |||||||||||||||||||||
| Comment by Githook User [ 26/Aug/22 ] | |||||||||||||||||||||
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: | |||||||||||||||||||||
| Comment by Daniel Moody [ 25/Aug/22 ] | |||||||||||||||||||||
|
I tested adding this option with v3 gcc:
and looks like it also solves the problem:
This solution would be better than dwp because it would save EDIT: this really wont save any time, actually split-dwarf with dwp is the same time as no split-dwarf no dwp, so just potentially diskspace and the baggage would be saved. | |||||||||||||||||||||
| Comment by Daniel Moody [ 17/Aug/22 ] | |||||||||||||||||||||
|
yes v4 gcc is much better about debug info and section size here is a mongod (recent master) bloaty from v4:
Here is v3 right before we started getting relocation overflows:
So switching to v4 frees up about a GB of available section size | |||||||||||||||||||||
| Comment by Daniel Moody [ 17/Aug/22 ] | |||||||||||||||||||||
Yeah we don't really have any static builders for gcc v4, but I tested locally without split-dwarf and it looks like the newer gcc does not face the same relocation overflow issue, so maybe its become more efficient in regards to section data. I'll see if I can verify its close to the limit. | |||||||||||||||||||||
| Comment by Andy Schwerin [ 17/Aug/22 ] | |||||||||||||||||||||
|
Indeed, I do. I think there's a good observation from Daniel, though, that we don't need to have "statically linked" debug information. As long as we had some easy-to-acquire-and-move bundle with the debug information, and it was easy to make that information available to GDB and the stack symbolizer, we could probably defer the "linking" of the debug information that happens today until it was actually required. | |||||||||||||||||||||
| Comment by Andy Schwerin [ 17/Aug/22 ] | |||||||||||||||||||||
|
Are the split-dwarf problems we're experiencing also problems on newer toolchains? I'm curious if we're stuck here partly because we couldn't complete the last toolchain upgrade. | |||||||||||||||||||||
| Comment by Daniel Moody [ 15/Aug/22 ] | |||||||||||||||||||||
|
another possible solution: Currently, we split the debug info out after the link step, which is were we end up running out of section space. What if we split the debug info out at the object file level using objcopy after each compile step. This is essentially almost the same thing as gsplit-dwarf, but it may get around some of the dwarf format issues we're seeing. I am not sure how to then combine the object files split debug into a single binary to produce the same .debug file we make during our link step, but I am pretty sure you can link the object file split debugs similar to just linking a binary, but the thing I am unsure about is how the linker would determine what symbols it really needs to put in the final .debug if it was separated out earlier at the object level? Maybe it doesn't matter if the .debug has all the object debug symbols? | |||||||||||||||||||||
| Comment by Daniel Moody [ 12/Aug/22 ] | |||||||||||||||||||||
|
Definitely a fair amount of regression with the large code model: | |||||||||||||||||||||
| Comment by Daniel Moody [ 12/Aug/22 ] | |||||||||||||||||||||
|
Here is some perf builds with the different -mcmodel options: -mcmodel=medium (failed from internal compiler error):
-mcmodel=large: | |||||||||||||||||||||
| Comment by Daniel Moody [ 11/Aug/22 ] | |||||||||||||||||||||
|
We should consider impacts and edge cases around mongosymb.py. This tool use llvm-symbolizer under the hood, which llvm-symbolizer appears to automatically use the .dwp if the .dwp is next to the .debug file. There might be other cases where the mongosymb.py tool fetches symbols remotely and doesn't download the .dwp file? | |||||||||||||||||||||
| Comment by Daniel Moody [ 09/Aug/22 ] | |||||||||||||||||||||
|
alexander.neben@mongodb.com it is not an issue for clang. | |||||||||||||||||||||
| Comment by Alex Neben [ 09/Aug/22 ] | |||||||||||||||||||||
|
We should also confirm this affects both clang and gcc builds | |||||||||||||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 08/Aug/22 ] | |||||||||||||||||||||
|
I think that if we spend some time focusing on work like SERVER-68657 and | |||||||||||||||||||||
| Comment by Alex Neben [ 08/Aug/22 ] | |||||||||||||||||||||
|
I updated my above list of ideas. I see a lot of watchers on this ticket and would love to hear feedback from anyone about next steps here. | |||||||||||||||||||||
| Comment by Daniel Moody [ 08/Aug/22 ] | |||||||||||||||||||||
|
ryan.egesdahl@mongodb.com Just want to clarify a few things in regard to the last comment. So we are still doing --separate-debug, that has not changed at all. Both the .debug and the .dwp are needed to debug with gdb. | |||||||||||||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 08/Aug/22 ] | |||||||||||||||||||||
|
daniel.moody@mongodb.com I think releasing with .dwp instead of the current --separate-debug mechanism using objcopy is going to have user-facing changes because we won't be able to install debug symbols into a distro-specific path anymore. I don't disagree with doing it that way, necessarily, but we will have to have some conversations with Product and Release teams about it and update our documentation accordingly. I'll do a little research to see if there's some way we can do some similar linking mechanism that we do with --separate-debug. | |||||||||||||||||||||
| Comment by Iryna Zhuravlova [ 08/Aug/22 ] | |||||||||||||||||||||
|
ryan.egesdahl@mongodb.com feel free to add your comments here | |||||||||||||||||||||
| Comment by Alex Neben [ 04/Aug/22 ] | |||||||||||||||||||||
|
Here are a few ideas that can act as strawmen in no particular order.
Let me know thoughts if anyone here has them. I think the best solution is to go with shared objects since that is a very standard linux/windows/mac pattern. |