Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16914

15-second stall associated with "sched: RT throttling activated" under WiredTiger

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 2.8.0-rc5
    • Fix Version/s: 3.0.1
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      • heavy mixed workload
      • Ubuntu 14.04.1 LTS, 3.13.0-32-generic
      • VMware, 6 cores

      5-second pause in db ops was seen following this message in syslog:

      Jan 16 14:39:15 ubuntu kernel: [20023.738805] [sched_delayed] sched: RT throttling activated

      • For a second or so from A to B no samples were reported by the external monitoring processes (a mongo shell process calling serverStatus and a python process monitoring system stats.)
      • An extremely high context switch rate was reported at A.
      • At B the monitoring processes resumed.
      • At around B (to within syslog resolution of 1 second) the above message appeared in syslog.
      • Starting at B for about 5 seconds db op rate dropped to 0
      • However not all activity was blocked: evictions appeared to be continuing, serverStatus was being processed.
      • "slots selected for switching that were unavailable" was high from B to C.

      It appears that some behavior involving extreme CPU utilization at high priority from A to B, possibly involving context switches, caused the kernel to suspend those threads for 5 seconds. The threads involved with eviction apparently weren't suspended.

        Attachments

        1. fixed.png
          fixed.png
          172 kB
        2. repro.html
          2.07 MB
        3. Screenshot 2015-02-16 12.45.11.png
          Screenshot 2015-02-16 12.45.11.png
          650 kB
        4. throttled.png
          throttled.png
          295 kB
        5. throttled-15s.png
          throttled-15s.png
          146 kB
        6. throttle-pre9-10.html
          479 kB
        7. throttle-pre9-10.png
          throttle-pre9-10.png
          174 kB
        8. throttle-pre9-10-gdbmon.html
          6.33 MB
        9. throttle-pre9-15.png
          throttle-pre9-15.png
          215 kB

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: