[SERVER-17265] thread convoys from WiredTigerRecoveryUnit::registerChange Created: 12/Feb/15 Updated: 26/Sep/15 Resolved: 26/Sep/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage, WiredTiger |
| Affects Version/s: | 3.0.0-rc8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Mark Callaghan | Assignee: | Unassigned |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Steps To Reproduce: | See the files I uploaded for https://jira.mongodb.org/browse/SERVER-17141 |
||||||||||||||||
| Sprint: | Quint 9 09/18/15 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
This occurs with the WiredTiger btree with 10 user threads. I see frequent cases where most threads are stalled in pthread_mutex_timedlock which appears to be in g++/STL code. AFAIK, STL has something to cache allocations that the environment variable GLIBCXXX_FORCE_NEW=1 might disable. Can WiredTiger do something to reduce pressure on the STL allocator from the use of WiredTigerRecoverUnit::_changes which is: I have yet to notice this problem in RocksDB, but maybe pressure on the allocator from other code triggers the problems for WiredTiger. The common thread stack with 3.0 rc8 is:
|
| Comments |
| Comment by Ramon Fernandez Marina [ 26/Sep/15 ] | |||||||||||||||||||||||||||||||
|
Hi mdcallag, we believe Thanks, | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 15/May/15 ] | |||||||||||||||||||||||||||||||
|
Hi mdcallag, have you had a chance to re-test with a version that includes Thanks, | |||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 22/Mar/15 ] | |||||||||||||||||||||||||||||||
|
Backporting the logop rollback safety changes from | |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 22/Mar/15 ] | |||||||||||||||||||||||||||||||
|
Running test for 3.0.1. Problem is still there for zlib and snappy block compressors. I am using gcc 4.9.x compiler. PMP shows the problem call stack as:
| |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 17/Feb/15 ] | |||||||||||||||||||||||||||||||
|
I get these thread stacks after changing the build from -O3 to -O2 or -O1 and it repeats for tcmalloc and glibc malloc. It doesn't reproduce when -fno-omit-frame-pointer is used with -O2 as I get different stacks with that – still thread convoys, just elsewhere. | |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 17/Feb/15 ] | |||||||||||||||||||||||||||||||
|
Given this comment I think this task should be closed. I am sure there is a lot of mutex contention that should be made better but I don't have anything concrete in what I provided here. I will repeat the 4.9.x tests with a lower level of optimization to see if I can get a better stack trace. Repeated tests for 4.8.1 and:
| |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
Reproduces with tcmalloc. Next I will try tcmalloc and a gcc 4.8.1 toolchain. I have been using 4.9.x. | |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
I will repeat the tests using a binary linked with tcmalloc. Thanks for your feedback. | |||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
The deallocate method of the allocator in new_allocator.h just delegates to operator delete, which in turn delegates to your malloc/free implementation. Do any paths through jemalloc's free() implementation ever take a mutex? The thread-per-connection architecture is pretty cruel to allocators with per-thread caches when the connection count gets even modestly high; we've had to do some tuning on tcmalloc to deal with similar issues. | |||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
Yeah, my comment about the relation to | |||||||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
My understanding (and a brief look through the GCC sources seems to confirm) is that GLIBCXX_FORCE_NEW facility only applies to the extension allocators defined in "ext/mt_allocator.h" and "ext/pool_allocator.h" allocators shipped with libstdc++. The call stack that you have provided shows that you are calling into new_allocator.h, which does not reference GLIBCXX_FORCE_NEW. Typically, the choice of default allocator is made at GCC configure time, via the --enable-libstdcxx-allocator[=KIND] flag, but it defaults to 'auto', which I believe is the same as 'new'. | |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
FYI, I compiled with gcc 4.9, but don't know the minor version: This also uses glibc 2.20 and jemalloc. But the problem stacks don't show jemalloc, so I think this is g++/STL. I am also far from an expert with modern c++ toolchains, but I am seeking advice from co-workers including the jemalloc author. | |||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
The solution planned for | |||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
acm , I'm surprised to hear that libstdc++ might still have it's own caching allocator. Mark, I think new vanilla gcc toolchains don't have a caching allocator inside the stl anymore. Can you find out if your toolchain is special in this regard? Also, are you using the bundled tcmalloc or another malloc implementation? | |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
For the data above, I took 23 thread stacks over 15 minutes and 5 of them had this stall | |||||||||||||||||||||||||||||||
| Comment by Mark Callaghan [ 12/Feb/15 ] | |||||||||||||||||||||||||||||||
|
The other thread stacks from that example don't show anything interesting. Perhaps the lock holder has moved on and the waiters have yet to wake and run.
And another example using PoorMansProfiler (PMP) style output, where the lock holder isn't obvious
|