Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35103

Checkpoint creates unevictable clean content

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 3.6.3, 3.6.4
    • Fix Version/s: None
    • Component/s: Storage
    • Operating System:
      ALL
    • Sprint:
      Storage Non-NYC 2018-06-04

      Description

      Attached repro script does the following

      • Inserts 1M documents with 10 indexes over fields that are random strings
      • Stops one secondary then removes all documents via one of the indexes, therefore in random order.
      • Restarts that secondary. While it is catching up (after the script completes), due to SERVER-34938, batches will be large and take several minutes to complete, so multiple checkpoints will run while timestamps are pinning content in cache. It appears that additional unevictable clean content is generated at each checkpoint, exacerbating the effects of SERVER-34938.

      • batch boundaries are at A, E, F
      • at each checkpoint (e.g B, C, D) the amount of clean content is increased, and this is presumably unevictable because the oldest timestamp is only updated between batches
      • at batch boundaries when oldest timestamp is advanced the cache content is reduced, presumably because it becomes evictable

      Above was run on a machine with 24 cpus and a lot of memory, but the script uses numactl to limit each instance to two cpus and cache to 4 GB (during setup) and 1 GB (during recovery), so this should be runnable on a more modest machine.

        Attachments

        1. metrics.2018-05-19T13-40-35Z-00000
          789 kB
        2. repro.png
          repro.png
          133 kB
        3. repro.sh
          3 kB
        4. s35103_high_dirty.png
          s35103_high_dirty.png
          101 kB
        5. s35103.png
          s35103.png
          191 kB

          Issue Links

            Activity

              People

              Assignee:
              bruce.lucas Bruce Lucas
              Reporter:
              bruce.lucas Bruce Lucas
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: