-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Critical - P2
-
Affects Version/s: None
-
Component/s: None
-
None
-
DevProd Build
-
ALL
-
v9.0
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Stack traces all in crashes, asserts, and the heap profiler are mis-symbolized due to BOLT. Frames will resolve to unrelated data symbols and wrong function names. The bug is that both abseil and libunwind assume a load relocation for program counters that is now incorrect due to BOLT adding a second LOAD segment. Both abseil and libunwind assume that the first load segment is the only load segment, and look up symbol names relative to that segment.
The impact is that emitted stack traces both in crashes and heap profiling will have incorrect symbols/function names whenever they relate to a pathway that BOLT deems cold (AKA stems from the second load segment). See an example here derived from a local jsTest. I've also verified this happens with the heap profiler – almost half of the stacks in a performance test run were poisoned due to this bug.

There is a silver lining in this cloud – with crashes, we also emit the address at each stack. So an external symbolizer tool can always derive the correct stack after the fact. However, this still means that the emitted stack trace will still show incorrect symbols, and will mislead somebody who isn't aware that they need to re-derive the stack.
And for the heap profiler, it's impossible to get the correct stacktrace without a SERVER ticket to emit stack addresses.
This is definitely happening in production, but I haven't been able to verify the impact. Hopefully whoever picks this up can do so.
- related to
-
SERVER-138313 Log raw program counters in the 8.0+ heap profiler path in case of stack symbolization corruption
-
- Open
-