Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37233

Increase in disk i/o for writes to replica set

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Storage
    • Labels:
    • Operating System:
      ALL
    • Sprint:
      Storage NYC 2018-10-08, Storage NYC 2018-10-22, Storage NYC 2018-11-05

      Description

      Issue Summary as of Dec 23, 2018

      ISSUE SUMMARY
      Beginning in MongoDB 3.6, administrators may observe an increase in disk i/o on mongod primaries.

      ISSUE IMPACT
      This increase in disk i/o should not generally be cause for concern.

      To ensure writes received by secondaries are durable on the primary, oplog entries are journaled and written to disk before being replicated. As a result, it is beneficial for mongod to flush the journal more rapidly, and consequently more heavily utilize the disk, to make these oplog entries available for replication as soon as possible.

      If the disk is fully utilized, the frequency of journal flushes will decrease to provide the same overall throughput of the node as compared to the performance of disk-bound workloads in earlier versions of MongoDB.

      AFFECTED VERSIONS
      MongoDB 3.6.x and subsequent major releases exhibit this behavior.

      Original description

      This is a continuation of SERVER-31679. It appears that the fix to that issue reduced the rate of journal flushes, but it is still significantly (about 30x) higher than in 3.4.

      Simple insert workload:

      function repro() {
          db.c.insert({_id: 0, i: 0})
          for (i = 0; i < 100000; i++) {
              db.c.update({_id: 0}, {$inc: {i: 1}})
          }
      }
      

      Results on 3.4.17, 3.6.5, and 3.6.6 respectively:

      Note that while SERVER-31679 has reduced the number of journal flush operations significantly between 3.6.5 and 3.6.6, at about 300/s is still about 30x larger than in 3.4.

      This becomes clearer if we simulate an application that is doing a low rate of inserts:

      function repro() {
          db.c.insert({_id: 0, i: 0})
          for (i = 0; i < 10000; i++) {
              db.c.update({_id: 0}, {$inc: {i: 1}})
              sleep(3)
          }
      }
      

      Results on 3.4.17, 3.6.5, and 3.6.6 respectively:

      For this workload SERVER-31679 has not made any difference in the number of journal flush operations - it is still about 30x larger than in 3.4.

        Attachments

        1. fast.png
          fast.png
          107 kB
        2. slow.png
          slow.png
          99 kB

          Issue Links

            Activity

              People

              Assignee:
              kelsey.schubert Kelsey T Schubert
              Reporter:
              bruce.lucas Bruce Lucas
              Participants:
              Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: