Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1634

Opening a cursor with many tables is slow

    • Type: Icon: Task Task
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      This references https://jira.mongodb.org/browse/SERVER-17078. Similar to the earlier issue, things get much slower as there are more tables to open. I (again) modified ex_stat to create many tables and then open a stat cursor on each one. On my Mac it takes 17 seconds to open/get a stat/close with a stat cursor on 10000 tables. As the Jira issue point out, it appears that the three loops in _wt_open (twice walking the fh list) and _wt_block_open eat a lot of time.

      I added name hashes for both of those structures to avoid the strcmp and that cuts it down to 15.5 seconds. Zoom shows us spending, overall, 12.3 seconds in *wt_curstat_open of which 2.8 seconds (23%) is *wt_block_open (1.5 seconds) and __wt_open (1.3 seconds) and I've confirmed the line numbers are the hash comparison. (Another 2.1 seconds (17%) is spent in config_next.)

      Running with a statistics logging on and adding a stat counter in those loops shows that for 17084 curstat open calls, we were in the block name loop, and both open loops 36 million times each or an average of 2200 iterations through each loop for every curstat open call. It looks like both of those structures may need the hash buckets like dhandles.

            Unassigned Unassigned
            sue.loverso@mongodb.com Susan LoVerso
            0 Vote for this issue
            3 Start watching this issue