Fix non-determinism in service_executor_adaptive_test

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Won't Do
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Service Arch
    • ALL
    • v4.0, v3.6
    • Platforms 2017-09-11, Platforms 2017-11-13, Platforms 2017-12-04, Platforms 2017-12-18, Platforms 2018-01-01, Platforms 2018-04-23
    • 15
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The service_executor_adaptive_test unittest's StuckThreadTest is flakey on windows - stuck thread detection does appear to happen, but it doesn't launch the right number of threads in time:

      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] 	 going to run test: TestStuckThreads
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] wait for executor to finish starting
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I EXECUTOR [worker-1] Starting new database worker thread 1
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [worker-1] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] Scheduling 6 blocked tasks
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [worker-1] waiting on blocked mutex
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] Waiting for executor to start new threads
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [worker-1] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I EXECUTOR [worker-controller] Starting worker thread to avoid starvation.
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I EXECUTOR [worker-2] Starting new database worker thread 2
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I -        [worker-2] waiting on blocked mutex
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I -        [worker-2] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I EXECUTOR [worker-controller] Detected blocked worker threads, starting new reserve threads to unblock service executor
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I EXECUTOR [worker-3] Starting new database worker thread 3
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I -        [worker-3] waiting on blocked mutex
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I -        [worker-3] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 E -        [main] Throwing exception: Expected exec->threadsRunning() == waitFor + 1 (3 == 4) @src\mongo\transport\service_executor_adaptive_test.cpp:197
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 I -        [worker-1] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 I -        [worker-2] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 I -        [worker-3] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.920+0000 2017-08-02T07:43:56.920+0000 I -        [main] FAIL: TestStuckThreads	Expected exec->threadsRunning() == waitFor + 1 (3 == 4) @src\mongo\transport\service_executor_adaptive_test.cpp:197
      

      Currently the test ifdef's the failing assertion out on Windows. We need to figure out why this timing issue occurs and fix it.

            Assignee:
            [DO NOT USE] Backlog - Service Architecture
            Reporter:
            Jonathan Reams
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: