Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1865

Performance degrades as more files are open.

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Gone away
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      I made a change to the wtperf workload generator to have it open additional tables. The change causes handles to be created at the populate phase, and then a cursor is opened/closed when the workload begins.

      I run the following configuration file:

      conn_config="cache_size=1G,file_manager=(close_idle_time=1000)"
      table_config="type=file"
      icount=50000000
      populate_threads=1
      compact=false
      threads=((count=8,read=1),(count=8,update=1))
      run_time=120
      report_interval=5
      table_count=1
      table_count_idle=10240
      

      Switching between

      =0

      and

      =10240

      . The performance results are:

      Phase No Idle 10k idle
      Load (seconds) 27s 28s
      Read rate (ops/sec) 494324 126487
      Update rate (ops/sec) 494324 126487

      That means that having 10k tables open causes performance to drop by 4x - even when only a single table is being actively used.

      This could become a bigger problem when we make changes to the sweep server so that it is much less aggressive at closing idle handles.

      The changes to wtperf are in pull request WT-1864

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                backlog-server-execution Backlog - Execution Team
                Reporter:
                alexander.gorrod Alexander Gorrod
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: