[SERVER-16551] tcmalloc ReleaseToCentralCache account > 14% usage with long insert traffic Created: 15/Dec/14  Updated: 24/Jan/15  Resolved: 07/Jan/15

Status: Closed
Project: Core Server
Component/s: Performance, Storage
Affects Version/s: 2.8.0-rc2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rui Zhang (Inactive) Assignee: Geert Bosch
Resolution: Won't Fix Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File perf-kernel-stable-after-bad3.svg     File perf.data.22ksec.gz     File perf.data.22ksec.svg     File perf.data.4000sec.gz     File perf.data.4000sec.svg     File perf.data.stable-after-bad3.gz     PNG File thread_cache_64mb_vs_32mb.png    
Issue Links:
Related
is related to SERVER-16517 Secondary on 2.8.0rc2 cannot keep up ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

during long insert with 64 threads, after about 2500 sec, perf event show tcmalloc's ReleaseToCentralCache is hot, sampled at 14.59% in the following case

This test is done with wiredtiger, haven;t check mmapv1 yet.

14.59%    mongod  mongod     [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)

it seems we can adjust max thread cache size to alleviate this situation. The default size is 32M per thread, did test with 64M, which seems improve the throughput

and here is perf even flame graph, you can check usage interactively. (please download and open in a new window for better view)



 Comments   
Comment by Githook User [ 13/Jan/15 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: SERVER-16829 Expose tcmalloc configuration options for per-thread cache size and decommit behavior.

Also, increases the default cap on the sum of all per-thread cache sizes from
32MB to 64MB, as advised by SERVER-16551. Should improve behavior of
SERVER-16517, as well.
Branch: master
https://github.com/mongodb/mongo/commit/a6ca16f211fe01d2870593d9ab9bfaaa88097a14

Comment by Geert Bosch [ 07/Jan/15 ]

In the end, this is not performance-critical here, because the bottle-neck is in the journaling: less time in malloc would just mean more time in yield. As we're not going to change tcmalloc, I'm closing this.

Generated at Thu Feb 08 03:41:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.