Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21792

Performance regression with directio enabled on Windows with WiredTiger

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.1, 3.3.0
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Sprint:
      Platforms E (01/08/16)

      Description

      Issue Status as of Jan 05, 2016

      ISSUE SUMMARY
      In SERVER-20991, direct_io was enabled by default on Windows machines. This change reduced the memory consumption of mongod processes, but may cause a degradation in performance. This change to the default setting of direct_io was reverted in this ticket.

      USER IMPACT
      With direct_io enabled, when the WiredTiger cache becomes full, the insertion rate may drop significantly.

      WORKAROUNDS
      To correct the performance regression in MongoDB 3.2.0, start mongod with

      --wiredTigerEngineConfigString="direct_io="

      To maintain to the performance characteristics of MongoDB 3.2.0 in later versions, start mongod with

      --wiredTigerEngineConfigString="direct_io=(data)"

      This setting may also be configured in the mongod configuration file:

      storage:
          wiredTiger:
              engineConfig:
                  configString: direct_io=(data)
      

      AFFECTED VERSIONS
      MongoDB 3.2.0

      FIX VERSION
      The fix is included in the 3.2.1 production release.

      Original description
      • AWS Instance - c4.4xlarge (16 CPU, 30GB RAM)
      • journal and data files configured on separate drives
      • 5 threads inserting small records in batches of 1000
      • default cache settings
      • syncdelay 10 (probably not relevant as checkpoints are very long anyway)
      • test run under rc6, same code as ga

      3.0.7

      3.2.0

      • initially 3.2.0 has somewhat higher insert rate
      • after cache fills 3.0.7 insert rate remains constant but 3.2.0 drops by about 5x, to about 25% of rate under 3.0.7

      Averages insert rates during initial and cache full portions of run:

                  initial   cache full
       
      3.0.7       152 k/s    135 k/s
      3.2.0       178 k/s     33 k/s
      

      1. 3.0.7.png
        80 kB
      2. 3.2.0-rc6.png
        79 kB
      3. 3.2.0-rc6-nodirectio.png
        68 kB
      4. 3.2.0-rc6-perf.png
        435 kB

        Activity

        Hide
        pasette Dan Pasette added a comment -

        The nightly development releases can be found here:
        https://www.mongodb.org/downloads/#development

        From there, you have to select your os version and mongodb version

        Show
        pasette Dan Pasette added a comment - The nightly development releases can be found here: https://www.mongodb.org/downloads/#development From there, you have to select your os version and mongodb version
        Hide
        michael@aorato.com Michael Dolinsky added a comment -

        So with 3.2.1 if there are a lot of writes (more then the system can handle) essentially the system cache will take up all memory and would cause an OS issue / MongoDB issue
        Have you run this stress test to see this for yourself?

        Show
        michael@aorato.com Michael Dolinsky added a comment - So with 3.2.1 if there are a lot of writes (more then the system can handle) essentially the system cache will take up all memory and would cause an OS issue / MongoDB issue Have you run this stress test to see this for yourself?
        Hide
        nick@innsenroute.com Nick Judson added a comment -

        @Michael Dolinsky - the original fix (which this reverts) was to fix that exact problem. Running with direct_io gives poor query performance, running without direct_io will cause all the system memory to be consumed. In 3.2, this wasn't the case (for my workload). I had good query performance and constant memory usage. I didn't have any regressions as described above.

        Now it sounds like the caching strategy for windows may be revisited for the next major release.

        Show
        nick@innsenroute.com Nick Judson added a comment - @Michael Dolinsky - the original fix (which this reverts) was to fix that exact problem. Running with direct_io gives poor query performance, running without direct_io will cause all the system memory to be consumed. In 3.2, this wasn't the case (for my workload). I had good query performance and constant memory usage. I didn't have any regressions as described above. Now it sounds like the caching strategy for windows may be revisited for the next major release.
        Hide
        michael@aorato.com Michael Dolinsky added a comment -

        How can we configure this setting: wiredTigerEngineConfigString="direct_io=(data)" in the cfg file?

        Show
        michael@aorato.com Michael Dolinsky added a comment - How can we configure this setting: wiredTigerEngineConfigString="direct_io=(data)" in the cfg file?
        Hide
        mark.benvenuto Mark Benvenuto added a comment -

        The Yaml configuration setting is hidden:

        storage:
            wiredTiger:
                engineConfig:
                    configString: direct_io=(data)
        

        Show
        mark.benvenuto Mark Benvenuto added a comment - The Yaml configuration setting is hidden : storage: wiredTiger: engineConfig: configString: direct_io=(data)

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            21 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

                Agile