Stalls during checkpoints

XMLWordPrintableJSON

    • Fully Compatible
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      • single-node replica set
      • 2 threads inserting 1 kB compressible documents
      • 10 GB oplog, 10 GB cache
      • ssd for data files, separate hdd for journal
      • using this build to avoid issues with LAS file from SERVER-21585

      • once cache fills inserts stall during latter part of checkpoint
      • ops with latencies up to 18779 ms are reported in mongod log
      • ftdc also stalls

      A similar test on 3.0.7 shows some slowdown during checkpoints but no complete stalls, and max latency seen in a similar run was 7s. (The 3.0.7 test used 15 threads instead of 2 to get comparable throughput).

      function repro(threads) {
      
          Random.setRandomSeed();
      
          db.c.ensureIndex({y:1})
      
          var docs = [];
          for (var i = 0; i < 1000; i++) {
              var x = '';
              while (x.length < 1000)
                  x += 'x'
              docs.push({x: x, y: i, z: 0});
          }
      
          var insert_ops = [{
              op: 'insert',
              ns: 'test.c',
              doc: docs,
              safe: true,
          }];
      
          var res = benchRun({
              ops: insert_ops,
              seconds: 10000,
              parallel: threads
          });
          printjson(res);
      
      }
      

        1. incident1-checkpoint.png
          incident1-checkpoint.png
          150 kB
        2. incident1-evict-server.png
          incident1-evict-server.png
          38 kB
        3. incident1-inserts.png
          incident1-inserts.png
          169 kB
        4. incident1-overview.png
          incident1-overview.png
          150 kB
        5. incident1.png
          incident1.png
          126 kB
        6. gdb2.png
          gdb2.png
          49 kB
        7. gdb.png
          gdb.png
          191 kB
        8. standalone.png
          standalone.png
          135 kB
        9. bigoplog.png
          bigoplog.png
          126 kB
        10. stalls.png
          stalls.png
          131 kB
        11. diagnostic.data.tar
          411 kB

            Assignee:
            Michael Cahill (Inactive)
            Reporter:
            Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: