Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18079

Large performance drop with documents > 16k on Windows

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.0.4, 3.1.3
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      Windows
    • Backport Completed:
    • Sprint:
      Platform 3 05/15/15

      Description

      The Windows Heap is implemented in user-space as part of kernel32.dll. It divides the heap into two portions:

      • the Low-Fragmentation Heap (LFH) which is optimized for allocations < 16 KB
      • the large allocation portion which takes a CriticalSection for each allocation and free.

      When WiredTiger needs to make many allocations >= 16KB performance on Windows will to suffer once the WT cache fills up. This is easily triggered by workloads with large documents.
      The solution we're pursuing is to use tc_malloc in place of the system allocator for WT on windows.
      See below for more details and the original bug report.

      Original Description

      • At D performance drops and remains low for the second half of the run (first row).
      • This coincides with the oplog reaching capacity (second row).
      • It also coincides with an increase in heap lock contention (third row), presumably related to the increased memory allocator activity caused by deleting entries from the oplog. Note that heap lock contention has about doubled in spite of the operation rate having fallen by about 4x.
      • A couple of other momentary dips in performance (A, B, C) appear to be related to disk activity (last two rows), and in fact heap lock contention fell during those dips because of the reduced op rate.

      Interestingly, the drop is much more pronounced with a 2-node replica set than it is with a 1-node replica set (write concern 1 in both cases). The stack traces confirm the contention and show why a 2-node replica set is worse:

      During the second half of the run there are significant periods when all 10 threads are waiting for the oplog deleter thread to reduce the size of the oplog:

      The oplog deleter thread is held up by significant contention in the allocator, in 3 different places:

      A primary source of contention seems to be the getmore that is tailing the oplog, which explains why a 2-node replica set is worse. Note that the oplog tail getmore is having to page in the entries even though they are brand new, which explains why it generates contention, but that seems non-optimal in itself, independent of the allocator issues.

      Repro takes about 3 minutes (Windows on VMware instance, 6 cpus, 12 GB memory). Initialize 2-node replica set with these options:

          mongod --storageEngine wiredTiger --wiredTigerCacheSizeGB 5 --oplogSize 6000 ...
      

      Then run 10 threads of the following, which runs twice as long as it takes for the oplog to fill in order to generate a good comparison.

      size = 16000
      w = 1
      batch = 10
       
      function insert() {
          
          x = ''
          for (var i=0; i<size; i++)
              x = x + 'x'
       
          finish = 0
          start = t = ISODate().getTime()
          for (var i=0; finish==0 || t<finish; ) {
              var bulk = db.c.initializeUnorderedBulkOp();
              for (var j=0; j<batch; j++, i++)
                  bulk.insert({x:x})
              bulk.execute({w:w});
              t = ISODate().getTime()
              if (i>0 && i%1000==0) {
                  var s = db.getSiblingDB('local').oplog.rs.stats(1024*1024)
                  print(i, s.size)
                  if (finish==0 && s.size>s.maxSize*0.9) {
                      finish = start + 2*(t-start)
                      print('finishing in', finish-t, 'ms')
                  }
              }
          }
      }
      

      1. drop.png
        127 kB
      2. DualWinHeap.png
        22 kB
      3. stacks-deleter.png
        464 kB
      4. stacks-oplog-tail.png
        215 kB
      5. stacks-waiters.png
        286 kB
      6. sysbench.png
        54 kB
      7. WinHeapBigDocs.png
        36 kB
      8. WinHeapVariousDocSizes.png
        33 kB
      9. wt_mem_alloc.png
        226 kB
      10. ycsb_1k.png
        41 kB
      11. ycsb_4k.png
        52 kB

        Issue Links

          Activity

          Hide
          sallgeud Chad Kreimendahl added a comment -

          Is there no performance benefit or value in using TCMalloc for mongo's heap, too?

          Show
          sallgeud Chad Kreimendahl added a comment - Is there no performance benefit or value in using TCMalloc for mongo's heap, too?
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

          Message: SERVER-18079: Use TCMalloc for WT on Windows

          (cherry picked from commit b43f9663dd08d3f585a9ae9aa67476b5e1a9d07c)
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/99767cdffb16203c5f3190560614b1a166c16bb3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'} Message: SERVER-18079 : Use TCMalloc for WT on Windows (cherry picked from commit b43f9663dd08d3f585a9ae9aa67476b5e1a9d07c) Branch: v3.0 https://github.com/mongodb/mongo/commit/99767cdffb16203c5f3190560614b1a166c16bb3
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

          Message: SERVER-18079: Use TCMalloc for WT on Windows

          (cherry picked from commit 263a5d5b39091a87a553420dba5fb393902a2166)
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/7662238b18a136fbad09e45163adff20d32fd1be

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'} Message: SERVER-18079 : Use TCMalloc for WT on Windows (cherry picked from commit 263a5d5b39091a87a553420dba5fb393902a2166) Branch: v3.0 https://github.com/mongodb/mongo/commit/7662238b18a136fbad09e45163adff20d32fd1be
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

          Message: SERVER-18079: Use TCMalloc for WT on Windows

          (cherry picked from commit 0a2be248888a1c11fda2848682b54fd314ab162c)
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/b677e49bed78c415498102a6d7d1cfbed43e76f7

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'} Message: SERVER-18079 : Use TCMalloc for WT on Windows (cherry picked from commit 0a2be248888a1c11fda2848682b54fd314ab162c) Branch: v3.0 https://github.com/mongodb/mongo/commit/b677e49bed78c415498102a6d7d1cfbed43e76f7
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

          Message: SERVER-18079: Use TCMalloc for WT on Windows
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/7e62741800a6e458ab02b9da91e5c37d389c8eae

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'} Message: SERVER-18079 : Use TCMalloc for WT on Windows Branch: v3.0 https://github.com/mongodb/mongo/commit/7e62741800a6e458ab02b9da91e5c37d389c8eae

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile