Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16914

15-second stall associated with "sched: RT throttling activated" under WiredTiger



    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 2.8.0-rc5
    • Fix Version/s: 3.0.1
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:


      • heavy mixed workload
      • Ubuntu 14.04.1 LTS, 3.13.0-32-generic
      • VMware, 6 cores

      5-second pause in db ops was seen following this message in syslog:

      Jan 16 14:39:15 ubuntu kernel: [20023.738805] [sched_delayed] sched: RT throttling activated

      • For a second or so from A to B no samples were reported by the external monitoring processes (a mongo shell process calling serverStatus and a python process monitoring system stats.)
      • An extremely high context switch rate was reported at A.
      • At B the monitoring processes resumed.
      • At around B (to within syslog resolution of 1 second) the above message appeared in syslog.
      • Starting at B for about 5 seconds db op rate dropped to 0
      • However not all activity was blocked: evictions appeared to be continuing, serverStatus was being processed.
      • "slots selected for switching that were unavailable" was high from B to C.

      It appears that some behavior involving extreme CPU utilization at high priority from A to B, possibly involving context switches, caused the kernel to suspend those threads for 5 seconds. The threads involved with eviction apparently weren't suspended.


        1. fixed.png
          172 kB
          Bruce Lucas
        2. repro.html
          2.07 MB
          Asya Kamsky
        3. Screenshot 2015-02-16 12.45.11.png
          650 kB
          Asya Kamsky
        4. throttled.png
          295 kB
          Bruce Lucas
        5. throttled-15s.png
          146 kB
          Bruce Lucas
        6. throttle-pre9-10.html
          479 kB
          Bruce Lucas
        7. throttle-pre9-10.png
          174 kB
          Bruce Lucas
        8. throttle-pre9-10-gdbmon.html
          6.33 MB
          Bruce Lucas
        9. throttle-pre9-15.png
          215 kB
          Bruce Lucas



            • Votes:
              0 Vote for this issue
              13 Start watching this issue


              • Created: