Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29024

Collection drop slows/stalls down database operations

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.4.4
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Environment:
      Single-socket physical machine
      XFS on a SATA SSD
      Ubuntu 14.04 x64, 4.4 (HWE) kernel
      MongoDB 3.4.4
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      Dropping collection that is "active" (I'm guessing that used means not checkpointed), possibly if it's concurrent with an active checkpoint, leads to a visible slowdown (queries that took tens of milliseconds start taking hundreds an thousands of milliseconds) and bumps up the checkpoint time. We were hit by this since we upgraded to 3.4, and while the upgrade to 3.4.4 (WT-3207) helped with "inactive" collections (we create/drop collections that are grouped per-day), but dropping "active" collections still leads to latency spikes, and since the system is sized to latency N, when latency becomes 100*N it just can't cope with the load.
      The worst part of it that entire database instance is affected, not just the collection or database in question.

      SERVER-22199 + WT-2333 and WT-2334 seem to be related.

      Attaching the subject log. The WT checkpoint time which is 3.5 sec average for an hour before/after the event spiked to 25 sec (we've seen 45 sec checkpoints in previous days).

        Attachments

        1. diagnostic.data.tar
          110.70 MB
        2. diagnostic.data-db5.tar
          24.40 MB
        3. event-diagnostics.png
          event-diagnostics.png
          358 kB
        4. mongodb-db2.log.gz
          1.01 MB
        5. mongodb-drive2-db5.log.gz
          121 kB
        6. Screen Shot 2017-05-01 at 3.26.51 pm.png
          Screen Shot 2017-05-01 at 3.26.51 pm.png
          71 kB
        7. Screen Shot 2017-05-01 at 3.27.47 pm.png
          Screen Shot 2017-05-01 at 3.27.47 pm.png
          72 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: