Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27546

Wiredtiger checkpoint is placed too often

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 3.2.11
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Operating System:
      ALL

      Description

      We have small data-set mongodb. All data is processed in memory.
      Total data size is only about 10GB. User data size is only 40MB and oplog.rs is 10GB.
      And wiredtiger cache size is 20GB. And all user documents are changed once in 30 seconds.
      There's 50 ~ 300 queries / second of geowithin query only on primary.

      The weired thing is primary's checkpoint is placed too often.
      "transaction checkpoint currently running" metric is always 1 of "db.serverStatus().wiredTiger.transaction. Attached graphs are "transaction checkpoint currently running" metric's.

      PRIMARY> db.serverStatus().wiredTiger.transaction
      {
      	"number of named snapshots created" : 0,
      	"number of named snapshots dropped" : 0,
      	"transaction begins" : 1466266063,
      	"transaction checkpoint currently running" : 1,
      	"transaction checkpoint generation" : 89630,
      	"transaction checkpoint max time (msecs)" : 6884,
      	"transaction checkpoint min time (msecs)" : 399,
      	"transaction checkpoint most recent time (msecs)" : 2220,
      	"transaction checkpoint scrub dirty target" : 0,
      	"transaction checkpoint scrub time (msecs)" : 0,
      	"transaction checkpoint total time (msecs)" : 163694262,
      	"transaction checkpoints" : 89630,
      	"transaction checkpoints skipped because database was clean" : 1,
      	"transaction failures due to cache overflow" : 0,
      	"transaction fsync calls for checkpoint after allocating the transaction ID" : 89629,
      	"transaction fsync duration for checkpoint after allocating the transaction ID (usecs)" : 7725,
      	"transaction range of IDs currently pinned" : 648,
      	"transaction range of IDs currently pinned by a checkpoint" : 2295,
      	"transaction range of IDs currently pinned by named snapshots" : 0,
      	"transaction sync calls" : 0,
      	"transactions committed" : 506988831,
      	"transactions rolled back" : 958130556
      }
       
      SECONDARY> db.serverStatus().wiredTiger.transaction
      {
      	"number of named snapshots created" : 0,
      	"number of named snapshots dropped" : 0,
      	"transaction begins" : 786350822,
      	"transaction checkpoint currently running" : 0,
      	"transaction checkpoint generation" : 2574,
      	"transaction checkpoint max time (msecs)" : 4007,
      	"transaction checkpoint min time (msecs)" : 527,
      	"transaction checkpoint most recent time (msecs)" : 3104,
      	"transaction checkpoint scrub dirty target" : 0,
      	"transaction checkpoint scrub time (msecs)" : 0,
      	"transaction checkpoint total time (msecs)" : 8154109,
      	"transaction checkpoints" : 2574,
      	"transaction checkpoints skipped because database was clean" : 0,
      	"transaction failures due to cache overflow" : 0,
      	"transaction fsync calls for checkpoint after allocating the transaction ID" : 2574,
      	"transaction fsync duration for checkpoint after allocating the transaction ID (usecs)" : 32661,
      	"transaction range of IDs currently pinned" : 0,
      	"transaction range of IDs currently pinned by a checkpoint" : 0,
      	"transaction range of IDs currently pinned by named snapshots" : 0,
      	"transaction sync calls" : 0,
      	"transactions committed" : 809047207,
      	"transactions rolled back" : 454837
      }
      

      And disk writes are happend every 2 seconds on Primary but not secondary.

      PRIMARY
      Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
      sda               0.00    24.00    0.00  417.00     0.00 18656.50    89.48     0.26    0.63    0.00    0.63   0.02   0.90
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    23.00    0.00  414.00     0.00 18783.00    90.74     0.36    0.87    0.00    0.87   0.02   0.90
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    23.00    0.00  408.00     0.00 18630.00    91.32     0.30    0.74    0.00    0.74   0.02   0.70
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    19.00    0.00  415.00     0.00 19192.50    92.49     0.31    0.75    0.00    0.75   0.02   0.80
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    28.00    0.00  399.00     0.00 18652.50    93.50     0.30    0.74    0.00    0.74   0.03   1.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    19.00    0.00  395.00     0.00 18685.00    94.61     0.30    0.75    0.00    0.75   0.02   0.70
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    27.00    0.00  399.00     0.00 18653.00    93.50     0.32    0.79    0.00    0.79   0.02   0.70
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    20.00    0.00  395.00     0.00 18951.00    95.95     0.31    0.78    0.00    0.78   0.02   0.80
      sda               0.00     1.00    0.00    7.00     0.00    39.50    11.29     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    28.00    0.00  408.00     0.00 18655.50    91.45     0.32    0.79    0.00    0.79   0.02   0.80
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    19.00    0.00  414.00     0.00 18833.50    90.98     0.27    0.64    0.00    0.64   0.02   0.80
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     4.00    0.00   91.00     0.00  1468.00    32.26     0.03    0.29    0.00    0.29   0.01   0.10
      sda               0.00    26.00    0.00  310.00     0.00 17557.00   113.27     0.25    0.81    0.00    0.81   0.02   0.70
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    18.00    0.00  400.00     0.00 18834.00    94.17     0.29    0.72    0.00    0.72   0.02   0.90
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    27.00    0.00  392.00     0.00 18665.00    95.23     0.32    0.82    0.00    0.82   0.02   0.80
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    18.00    0.00  397.00     0.00 18595.00    93.68     0.27    0.67    0.00    0.67   0.03   1.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    27.00    0.00  386.00     0.00 18501.00    95.86     0.30    0.77    0.00    0.77   0.02   0.80
      sda               0.00     1.00    0.00    7.00     0.00    32.00     9.14     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    19.00    0.00  427.00     0.00 18889.50    88.48     0.30    0.70    0.00    0.70   0.02   0.90
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    26.00    0.00  413.00     0.00 18801.00    91.05     0.31    0.75    0.00    0.75   0.02   0.80
      sda               0.00     0.00    0.00    2.00     0.00    16.00    16.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    24.00    0.00  450.00     0.00 19001.00    84.45     0.38    0.84    0.00    0.84   0.02   0.90
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
       
      SECONDARY
      Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    4.00     0.00    28.00    14.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     1.00    0.00    3.00     0.00    10.00     6.67     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    1.00     0.00     6.50    13.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    2.00     0.00     8.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00    23.00    0.00 2396.00     0.00 61823.00    51.61     8.10    3.38    0.00    3.38   0.01   3.20
      sda               0.00     0.00    0.00    2.00     0.00     5.00     5.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    1.00     0.00     4.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
      

        Attachments

        1. diagnostics.data.tar.gz
          38.73 MB
        2. PRIMARY-CheckpointRunning.png
          PRIMARY-CheckpointRunning.png
          47 kB
        3. SECONDARY-CheckpointRunning.png
          SECONDARY-CheckpointRunning.png
          58 kB

          Issue Links

            Activity

              People

              Assignee:
              kelsey.schubert Kelsey T Schubert
              Reporter:
              matt.lee Matt SeongUck Lee
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: