Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21652

Stalls during checkpoints

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0-rc5
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      • single-node replica set
      • 2 threads inserting 1 kB compressible documents
      • 10 GB oplog, 10 GB cache
      • ssd for data files, separate hdd for journal
      • using this build to avoid issues with LAS file from SERVER-21585

      • once cache fills inserts stall during latter part of checkpoint
      • ops with latencies up to 18779 ms are reported in mongod log
      • ftdc also stalls

      A similar test on 3.0.7 shows some slowdown during checkpoints but no complete stalls, and max latency seen in a similar run was 7s. (The 3.0.7 test used 15 threads instead of 2 to get comparable throughput).

      function repro(threads) {
       
          Random.setRandomSeed();
       
          db.c.ensureIndex({y:1})
       
          var docs = [];
          for (var i = 0; i < 1000; i++) {
              var x = '';
              while (x.length < 1000)
                  x += 'x'
              docs.push({x: x, y: i, z: 0});
          }
       
          var insert_ops = [{
              op: 'insert',
              ns: 'test.c',
              doc: docs,
              safe: true,
          }];
       
          var res = benchRun({
              ops: insert_ops,
              seconds: 10000,
              parallel: threads
          });
          printjson(res);
       
      }
      

        Attachments

        1. bigoplog.png
          bigoplog.png
          126 kB
        2. diagnostic.data.tar
          411 kB
        3. gdb.png
          gdb.png
          191 kB
        4. gdb2.png
          gdb2.png
          49 kB
        5. incident1.png
          incident1.png
          126 kB
        6. incident1-checkpoint.png
          incident1-checkpoint.png
          150 kB
        7. incident1-evict-server.png
          incident1-evict-server.png
          38 kB
        8. incident1-inserts.png
          incident1-inserts.png
          169 kB
        9. incident1-overview.png
          incident1-overview.png
          150 kB
        10. stalls.png
          stalls.png
          131 kB
        11. standalone.png
          standalone.png
          135 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: