Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1865

Performance degrades as more files are open.

XMLWordPrintableJSON

    • Type: Icon: Task Task
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      I made a change to the wtperf workload generator to have it open additional tables. The change causes handles to be created at the populate phase, and then a cursor is opened/closed when the workload begins.

      I run the following configuration file:

      conn_config="cache_size=1G,file_manager=(close_idle_time=1000)"
table_config="type=file"
icount=50000000
populate_threads=1
compact=false
threads=((count=8,read=1),(count=8,update=1))
run_time=120
report_interval=5
table_count=1
table_count_idle=10240

      Switching between

      Unable to find source-code formatter for language: table_count_idle. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      =0

      and

      Unable to find source-code formatter for language: table_count_idle. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      =10240

      . The performance results are:

      Phase No Idle 10k idle
      — — — – –
      Load (seconds) 27s 28s
      Read rate (ops/sec) 494324 126487
      Update rate (ops/sec) 494324 126487

      That means that having 10k tables open causes performance to drop by 4x - even when only a single table is being actively used.

      This could become a bigger problem when we make changes to the sweep server so that it is much less aggressive at closing idle handles.

      The changes to wtperf are in pull request WT-1864

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: