Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7287

Urgently Sweep Checkpoint dhandles

    • Type: Icon: Improvement Improvement
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • 5
    • Storage - Ra 2021-05-17

      Currently, for a checkpoint, we allocate and insert a dhandle into the connection list and hash buckets for the URI:WiredTigerCheckpoint### name. This handle is likely never going to be used again. Currently it exists in the list because we assume that any dhandle recently created is also likely to be used soon. In the case of these checkpoint dhandles it is so that if anyone wants to open a cursor on the checkpoint we have that dhandle cached. MongoDB does not currently open cursors on checkpoints. It used to, but does not any longer.

      In sue.loverso’s tests, which have 15000 tables, and the workload modifies half of the tables only, she ended up with over 37500 handles in the list—15000 table: dhandles, 15000 file: dhandles and 7500 file:checkpoint dhandles.

      Since it is unlikely that we will use those dhandles again after the checkpoint completes we should either remove them from the list or mark them so that sweep aggressively closes them. NOTE: it is possible we already do this because I don’t see the list growing with each successive checkpoint. Another possibility would be to explore not putting checkpoint dhandles in the list at all but using something like WT_WITH_DHANDLE.

      Note that sue.loverso’s testing has shown that walking the list is not a major consumer of time but when you’re multiplying by many thousands, even small amounts of time add up to be noticeable.

            Assignee:
            ravi.giri@mongodb.com Ravi Giri
            Reporter:
            sulabh.mahajan@mongodb.com Sulabh Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: