Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16479

Pauses due to large page eviction during fetches under WiredTiger

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.8.0-rc2
    • Component/s: Performance, Storage
    • Labels:
    • Fully Compatible
    • ALL

      Test consists of a single thread inserting about 800 MB worth of documents then fetching them back. Under rc1 significant pauses would be observed during the inserts; rc2 has eliminated the pauses during inserts, but at the expense of comparable pauses during the fetches.

      • From A to B app is doing inserts and is seeing relatively steady insert rates without the significant pauses that were seen in rc1 (see below). No evictions occur at any time during this phase.
      • App then issues three find().hint({_id:1}) queries, at C, F, and G.
      • First query experiences about half a dozen regularly spaced significant pauses (e.g. D and E), coinciding with page evictions due to the large in-memory pages created by the insertions, as indicated by the coincident peaks in pages written from cache.
      • Immediately following each eviction the small on-disk pages are read back into cache, as indicated by the peaks in pages read into cache.
      • Subsequent queries (F-G and G-H) go faster because there are no evictions and the disk pages are now in cache.

      For comparison here is the same test on rc1:

      • The pauses that were seen during insertion on rc1 (e.g. B and C) are the same pauses that are now seen during fetches on rc2.

      Profile was obtained by increasing memory_page_max to 1GB to move all the pauses to a single long pause as the 1GB page is evicted, and then sampling periodically with gdb. The eviction shows up as a pause between the first query and the getmores from the app, coinciding with pages being written, marked by blue lines below. The gdb samples coinciding with this interval show that the eviction is occuring in __wt_btcur_reset called from WiredTigerSession::releaseCursor during the traversal of the collection.

        1. rc1-wt.png
          rc1-wt.png
          278 kB
        2. rc2-wt.png
          rc2-wt.png
          277 kB
        3. rc2-wt-prof.png
          rc2-wt-prof.png
          548 kB

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: