Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21652

Stalls during checkpoints

    • Fully Compatible
    • ALL

      • single-node replica set
      • 2 threads inserting 1 kB compressible documents
      • 10 GB oplog, 10 GB cache
      • ssd for data files, separate hdd for journal
      • using this build to avoid issues with LAS file from SERVER-21585

      • once cache fills inserts stall during latter part of checkpoint
      • ops with latencies up to 18779 ms are reported in mongod log
      • ftdc also stalls

      A similar test on 3.0.7 shows some slowdown during checkpoints but no complete stalls, and max latency seen in a similar run was 7s. (The 3.0.7 test used 15 threads instead of 2 to get comparable throughput).

      function repro(threads) {
      
          Random.setRandomSeed();
      
          db.c.ensureIndex({y:1})
      
          var docs = [];
          for (var i = 0; i < 1000; i++) {
              var x = '';
              while (x.length < 1000)
                  x += 'x'
              docs.push({x: x, y: i, z: 0});
          }
      
          var insert_ops = [{
              op: 'insert',
              ns: 'test.c',
              doc: docs,
              safe: true,
          }];
      
          var res = benchRun({
              ops: insert_ops,
              seconds: 10000,
              parallel: threads
          });
          printjson(res);
      
      }
      

        1. bigoplog.png
          126 kB
          Bruce Lucas
        2. diagnostic.data.tar
          411 kB
          Bruce Lucas
        3. gdb.png
          191 kB
          Martin Bligh
        4. gdb2.png
          49 kB
          Martin Bligh
        5. incident1.png
          126 kB
          Bruce Lucas
        6. incident1-checkpoint.png
          150 kB
          Bruce Lucas
        7. incident1-evict-server.png
          38 kB
          Bruce Lucas
        8. incident1-inserts.png
          169 kB
          Bruce Lucas
        9. incident1-overview.png
          150 kB
          Bruce Lucas
        10. stalls.png
          131 kB
          Bruce Lucas
        11. standalone.png
          135 kB
          Bruce Lucas

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: