Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-23622

Inconsistent throughput during insert workload

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      With a heavy insert load and a secondary that is delayed the cache on the primary fills to 100% and operation rates drop. Here's a run showing behavior on the primary with the secondary delayed due to lag, but a similar effect is seen if the secondary is intentionally delayed using slaveDelay.

      • from D-E the cache is 95% full and insert rate drops considerably, possibly due to application threads doing evictions?
      • F-G and H-I seem to be seem to be related to checkpoints, possibly also in combination with the full cache?
      • the rate of pages walked for eviction is generally very high, about 6k times the rate of pages actually evicted, suggesting that the issue is difficulty finding pages to evict to keep the cache at target levels

      The high rate of pages walked for eviction suggests a connection to SERVER-22831, which also showed that symptom in connection with a full cache; however the above run was on 3.2.5-rc1 where SERVER-22831 was fixed, so it seems there is a different issue here.

      The above test involved

      • 3.2.5-rc1
      • 2-node replica set
      • 25 GB cache
      • 100 GB oplog
      • 5 threads inserting 800 byte documents into 5 separate collections

      for t in $(seq 5); do
          mongo --eval "
              x = ''
              for (var i=0; i<800; i++)
                  x += 'x'
              docs = []
              for (var i=0; i<1000; i++)
                  docs.push({x:x})
              ops = [{
                  op: 'insert',
                  ns: 'test.c' + '$t',
                  doc: docs,
              }]
              res = benchRun({
                  ops: ops,
                  seconds: 10000,
                  parallel: 1
              })
          " &
      done
       
      wait
      

        Attachments

        1. 23622.0901.tar.gz
          728 kB
        2. 23622.20.tar.gz
          1013 kB
        3. 23622.nolog.tar.gz
          428 kB
        4. 23622.results.tar.gz
          868 kB
        5. 2816.20.tar.gz
          392 kB
        6. aggressive.43min.tar.gz
          671 kB
        7. cs29611.png
          cs29611.png
          127 kB
        8. diagnostic.data.tar
          146 kB
        9. exp.tar.gz
          414 kB
        10. lagging.png
          lagging.png
          370 kB
        11. patch.png
          patch.png
          223 kB
        12. patch-diagnostic.data.tar
          532 kB
        13. run8.tar.gz
          331 kB
        14. run8-zoom.png
          run8-zoom.png
          229 kB
        15. run-branch.png
          run-branch.png
          274 kB
        16. stalls.png
          stalls.png
          104 kB

          Issue Links

            Activity

              People

              Assignee:
              sue.loverso Susan LoVerso
              Reporter:
              bruce.lucas Bruce Lucas
              Participants:
              Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: