[SERVER-62227] Perf annotate tool on Linux doesn't show sources for dynamically linked builds Created: 22/Dec/21  Updated: 17/Feb/22  Resolved: 17/Feb/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Irina Yatsenko (Inactive) Assignee: Daniel Moody
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-59243 icecream 1.1 remotes fail to handle s... Closed
is related to SERVER-62313 Compiling pre-processed sourcecode wi... Closed
Sprint: Dev Platform 2022-01-10, Dev Platform 2022-01-24, Dev Platform 2022-02-07, Dev Platform 2022-02-21
Participants:

 Description   

After recording profiles with "perf record" it's possible later to view per-instruction counter stats with "perf annotate" (or interactively from "perf report"). However, when building with `--link-model=dynamic` the report only shows assembly and doesn't include source lines.

Full build command: `./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_stable_clang.vars --ninja generate-ninja ICECC=icecc CCACHE=ccache --dbg=off --opt=on --link-model=dynamic`

Running perf tool:
1. Start mongod
2. $ perf record -p $(pidof mongod) -e cpu-cycles
3. Execute a query against mongod
4. stop perf (Ctrl+C), it will create perf.data file in the working dir
5. $ perf annotate (by default it will use perf.data file)
The tool should show information similar to this:

switch (conf->go[*(const uint8_t *)conf->cur]) {
  4.0080:   movzbl (%rbx),%r10d
  2.00 │       movsbl (%rax,%r10,1),%eax
 20.64 │       add    $0xffffffff,%eax

But when using dynamic linking it only shows disassembly and "s" command doesn't toggle the sources on/off (only recolors the line numbers).



 Comments   
Comment by Daniel Moody [ 17/Feb/22 ]

Closing as duplicate to the two related issues. Icecream and ccache have been having issues with source paths as described with the tickets

Comment by Irina Yatsenko (Inactive) [ 17/Feb/22 ]

After rebuilding without using ICECC and CCACHE ("./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_stable_clang.vars --ninja generate-ninja --dbg=off --opt=on") the sources do show correctly in the perf tool.

Comment by Irina Yatsenko (Inactive) [ 17/Feb/22 ]

I've installed the newer tools package per Daniel's instructions above but the perf tool was still showing version "4.15.18" after that and the issue continues to repro. The machine I'm using is an Evergreen VM from "ubuntu1804-metal" distro.

Comment by Iryna Zhuravlova [ 17/Feb/22 ]

Hi irina.yatsenko, I was wondering if there are any updates in reproducing this? Thank you!

Comment by Irina Yatsenko (Inactive) [ 15/Feb/22 ]

I don't remember installing perf explicitly. I believe it came with the distro (ubuntu1804-metal). I'll try to repro on my side again and then upgrade perf to see if the problem goes away. Let me know if you'd like me to do anything else/different.

Comment by Daniel Moody [ 15/Feb/22 ]

the perf on my system is 5.4. I didn't do any special configuration to get perf to show me the source code. I suspect the symbols are there and there is just something missing from the configuration of perf. For my system, I did need to install it, which I did via: 

sudo apt-get install linux-tools-common linux-tools-5.4.0-1056-aws

Comment by Irina Yatsenko (Inactive) [ 15/Feb/22 ]

perf version 4.15.18
I usually run mongod something like this: numactl --physcpubind=4,5,6,7 ./build/opt/install/bin/mongod --fork --logpath ~/mongod.log --dbpath ~/data --wiredTigerCacheSizeGB 16 --oplogSize 100000 --syncdelay 3600 --nojournal --setParameter ttlMonitorEnabled=false

Comment by Daniel Moody [ 14/Feb/22 ]

I couldn't reproduce this, when I build with the same command, I can perf record and then perf report and see the source lines. Where you using the mongod in the build/opt/install location? What version of perf does your system have?

Generated at Thu Feb 08 05:54:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.