[SERVER-13683] all threads must call ProfilerRegisterThread() for gperftools Created: 22/Apr/14 Updated: 10/Dec/14 Resolved: 24/Jul/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Diagnostics |
| Affects Version/s: | 2.6.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Mark Callaghan | Assignee: | Andrew Morrow (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | see https://code.google.com/p/gperftools/issues/detail?id=335 where gperftool maintainers admit this is probably needed and the thread at https://groups.google.com/forum/#!topic/google-perftools/gEpE7_AdibI. This is a regression in gperftools as it wasn't needed in the near past. But many, many years ago it was needed when I was profiling mysqld. I added the call to ProfilerRegisterThread to to handleIncomingMsg. In this case I compiled the server with --user-cpu-profiler, called _cpuProfilerStart, ran a workload, called _cpuProfilerStop |
| Participants: |
| Description |
|
To get CPU profiling from gperftools on recent Linux systems/kernels, all threads should call ProfilerRegisterThread. This issue is forked from https://jira.mongodb.org/browse/SERVER-6628 Without adding the calls to ProfilerRegisterThread the gperftools profiler output file has nothing useful... |
| Comments |
| Comment by Ramon Fernandez Marina [ 14/Jul/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
mdcallag, we have not been able to reproduce the behavior you describe on this ticket, so we're going to mark it as resolved. I'll post the detailed steps we tried in a separate comment, in case you're interested in following up. – | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 02/May/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Mark - I just spun up a clean Ubuntu 13.10 image on Digital Ocean, and I was able to use the cpu profiler:
I set up the packages necessary to build. You need to install google-perftools to get a copy of google-pprof since we don't build it. Then cloned mongo, checked out 2.6.0, built it using your options, and started mongod.
Then in the mongo shell in another session:
Then run some update load against mongod, then, in the mongo shell
I now have a foo.prof file with contents:
Running google-pprof (from the ubuntu google-perftools package) against this file gets me useful output:
So it looks to me like this works fine on Ubuntu 13.10. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 02/May/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Switched back to Ubuntu 13.10 on my VM using the build command below. The gperftools output file created by mongod is ~160kb but pprof --text prints nothing and pprof --gv prints "No nodes to print". This is the same as I experienced before. Given that MongoDB already has a few lines of code compiled when gperftools is enabled, this only requires 1 or 2 extra lines to avoid uncertainty from problems with certain versions of Linux.
| |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 02/May/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks Mark - Your scons invocation looks reasonable to me, though I wonder if you actually need --extralib=unwind, since we should be adding it for you when building with --use-cpu-profiler: https://github.com/mongodb/mongo/blob/v2.6/src/third_party/gperftools-2.0/SConscript#L82-L84. In an effort to rule out one potential source of variation between your system and ours, could you perhaps try building against the vanilla system compiler and runtime libraries, rather than the ones from your production compatible toolchain? Thanks, | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 02/May/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
I use the following and don't show all of the paths (BASE_CXX, BASE_CC, UWPATH) but trust me that it makes builds complicated because we have compiler/library toolchains so that builds done on my devel server will also work in production.
From the build output libtcmalloc gets built from provided code: From "nm mongod | grep -i tcmalloc" I see tcmalloc but I do not see it linked with libtcmalloc in the build output From the mongod command line I see this, and there is more but command line output is huge (all object files, few libraries). So I think I have included the right version of gperftools into mongod | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 29/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Mark - A few of us have recently tested this on our dev machines, and in each case (after making some local edit to fix or work around I see that you confirm above that you are using --allocator=tcmalloc, however that only enables using tcmalloc: it does not select which version (vendored vs system) of the tcmalloc allocator you are using. That choice is controlled by the --use-system-tcmalloc and the --use-system-all flag. Can you confirm that you are also not using --use-system-tcmalloc or --use-system-all flags, or doing anything with the include or library search path which would cause the build system to select something other than the vendored gperftools? As an aside, another approach to profiling would be to use callgrind, though of course that comes with significantly higher overhead than the stack sampling profiler. Please let me know how else we can help. Thanks, | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 22/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
I use the version shipped with MongoDB (--allocator=tcmalloc) and confirmed by looking at scons build output. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 22/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Are you using the system version of tcmalloc or the version shipped in the mongodb source? This comes at an opportune time, as we're investigating upgrading our shipped tcmalloc. |