Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21652

Stalls during checkpoints

    • Fully Compatible
    • ALL

      • single-node replica set
      • 2 threads inserting 1 kB compressible documents
      • 10 GB oplog, 10 GB cache
      • ssd for data files, separate hdd for journal
      • using this build to avoid issues with LAS file from SERVER-21585

      • once cache fills inserts stall during latter part of checkpoint
      • ops with latencies up to 18779 ms are reported in mongod log
      • ftdc also stalls

      A similar test on 3.0.7 shows some slowdown during checkpoints but no complete stalls, and max latency seen in a similar run was 7s. (The 3.0.7 test used 15 threads instead of 2 to get comparable throughput).

      function repro(threads) {
      
          Random.setRandomSeed();
      
          db.c.ensureIndex({y:1})
      
          var docs = [];
          for (var i = 0; i < 1000; i++) {
              var x = '';
              while (x.length < 1000)
                  x += 'x'
              docs.push({x: x, y: i, z: 0});
          }
      
          var insert_ops = [{
              op: 'insert',
              ns: 'test.c',
              doc: docs,
              safe: true,
          }];
      
          var res = benchRun({
              ops: insert_ops,
              seconds: 10000,
              parallel: threads
          });
          printjson(res);
      
      }
      

        1. standalone.png
          standalone.png
          135 kB
        2. stalls.png
          stalls.png
          131 kB
        3. incident1-overview.png
          incident1-overview.png
          150 kB
        4. incident1-inserts.png
          incident1-inserts.png
          169 kB
        5. incident1-evict-server.png
          incident1-evict-server.png
          38 kB
        6. incident1-checkpoint.png
          incident1-checkpoint.png
          150 kB
        7. incident1.png
          incident1.png
          126 kB
        8. gdb2.png
          gdb2.png
          49 kB
        9. gdb.png
          gdb.png
          191 kB
        10. diagnostic.data.tar
          411 kB
        11. bigoplog.png
          bigoplog.png
          126 kB

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: