Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19975

30% performance regression on simple insert workload on primary

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.1.5, 3.1.6
    • Fix Version/s: 3.1.8
    • Component/s: Replication
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      Saw a 30% performance decline between 3.1.4 and 3.1.5 in a simple insert workload - 8 threads of the following:

          every = 10000
          for (var i=0; i<500000; ) {
              var bulk = db.c.initializeUnorderedBulkOp();
              for (var j=0; j<every; j++, i++)
                  bulk.insert({x:0})
              bulk.execute()
          }
      

      Execution time in seconds:

              1-node rs    standalone     1-node rs
              wiredtiger   wiredtiger      mmapv1
      3.0.5     19            11             90
      3.1.4     20            12             91
      3.1.5     28            12            109
      

      No performance regression on standalone node, and regression is the primary, so presumably related to oplog insertion. Similar performance regression for both WT and mmapv1.

      Did a git bisect and identified f302737346da1bfb46f5c3b295bfb7896daa1508 as the first bad commit, and this change is indeed related to oplog processing.

      Even though this is a bit old at this point, more recent builds show similar (or worse) performance, so presumably this is still relevant.

        Issue Links

          Activity

          Hide
          bruce.lucas Bruce Lucas added a comment -

          Added timing above for 3.0.5 to clarify that this is a significant performance regression relative to 3.0.

          Show
          bruce.lucas Bruce Lucas added a comment - Added timing above for 3.0.5 to clarify that this is a significant performance regression relative to 3.0.
          Hide
          scotthernandez Scott Hernandez (Inactive) added a comment -

          I believe this may be due to the additional checks on protocol version and have a patch to address this which will be included in SERVER-19950.

          Show
          scotthernandez Scott Hernandez (Inactive) added a comment - I believe this may be due to the additional checks on protocol version and have a patch to address this which will be included in SERVER-19950 .
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

          Message: SERVER-19975: use cached protocol version (atomic)
          Branch: master
          https://github.com/mongodb/mongo/commit/9ad561b8a31f3b8233ac5f905e2b2a089c5b1302

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'} Message: SERVER-19975 : use cached protocol version (atomic) Branch: master https://github.com/mongodb/mongo/commit/9ad561b8a31f3b8233ac5f905e2b2a089c5b1302
          Hide
          scotthernandez Scott Hernandez (Inactive) added a comment -

          Other commits may have contributed to restoring throughput other than the ones included here.

          Show
          scotthernandez Scott Hernandez (Inactive) added a comment - Other commits may have contributed to restoring throughput other than the ones included here.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: