Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16651

Mass deletions and updates (including TTL) completely starve insertions in replica set under WiredTiger

    • Fully Compatible
    • ALL

      Set up a ttl collection with documents expiring after 30 seconds, and then start inserting:

          var count = 10000000
          var every = 10000
      
          db.c.drop();
          db.c.ensureIndex({ttl: 1}, {expireAfterSeconds: 30});
          db.c.ensureIndex({i:1})
       
          for (var i=0; i<count; ) {
              var bulk = db.c.initializeUnorderedBulkOp();
              for (var j=0; j<every; i++, j++)
                  bulk.insert({i: Math.random(), ttl: new Date()});
              bulk.execute();
              print(i)
          }
      

      In a single-node replica set the TTL deletion passes almost entirely suppress the insertions - the spikes on the second graph indicate the start of TTL deletion passes.

      Whereas with a standalone node, the TTL deletions have almost no noticeable effect on the insertions.

      Gdb sampling profile shows the insertions being blocked and the TTL thread active, but does not immediately make it clear why the insertion thread is blocked the whole time - the TTL thread is doing a variety of things, many of which it will also be doing in the case of a standalone instance, so why should inserts be blocked in the case of a replica set and not standalone?

      Under mmapv1 the TTL deletions interfere with the insertions but do not completely suppress them, both standalone and with replica set.

        1. blocking-updates.png
          blocking-updates.png
          499 kB
        2. profile.png
          profile.png
          455 kB
        3. wt-rc3-del-rs0.png
          wt-rc3-del-rs0.png
          76 kB
        4. wt-rc3-del-rs1.png
          wt-rc3-del-rs1.png
          85 kB
        5. wt-rc3-rs0.png
          wt-rc3-rs0.png
          59 kB
        6. wt-rc3-rs1.png
          wt-rc3-rs1.png
          58 kB

            Assignee:
            schwerin@mongodb.com Andy Schwerin
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: