Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30475

Fix non-determinism in service_executor_adaptive_test

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • Service Arch
    • ALL
    • v4.0, v3.6
    • Platforms 2017-09-11, Platforms 2017-11-13, Platforms 2017-12-04, Platforms 2017-12-18, Platforms 2018-01-01, Platforms 2018-04-23
    • 15

    Description

      The service_executor_adaptive_test unittest's StuckThreadTest is flakey on windows - stuck thread detection does appear to happen, but it doesn't launch the right number of threads in time:

      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] 	 going to run test: TestStuckThreads
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] wait for executor to finish starting
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I EXECUTOR [worker-1] Starting new database worker thread 1
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [worker-1] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] Scheduling 6 blocked tasks
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [worker-1] waiting on blocked mutex
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [main] Waiting for executor to start new threads
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.918+0000 2017-08-02T07:43:55.919+0000 I -        [worker-1] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I EXECUTOR [worker-controller] Starting worker thread to avoid starvation.
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I EXECUTOR [worker-2] Starting new database worker thread 2
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I -        [worker-2] waiting on blocked mutex
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:55.924+0000 2017-08-02T07:43:55.924+0000 I -        [worker-2] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I EXECUTOR [worker-controller] Detected blocked worker threads, starting new reserve threads to unblock service executor
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I EXECUTOR [worker-3] Starting new database worker thread 3
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I -        [worker-3] waiting on blocked mutex
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.233+0000 2017-08-02T07:43:56.234+0000 I -        [worker-3] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 E -        [main] Throwing exception: Expected exec->threadsRunning() == waitFor + 1 (3 == 4) @src\mongo\transport\service_executor_adaptive_test.cpp:197
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 I -        [worker-1] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 I -        [worker-2] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.434+0000 2017-08-02T07:43:56.434+0000 I -        [worker-3] Ran callback
      [cpp_unit_test:service_executor_adaptive_test] 2017-08-02T07:43:56.920+0000 2017-08-02T07:43:56.920+0000 I -        [main] FAIL: TestStuckThreads	Expected exec->threadsRunning() == waitFor + 1 (3 == 4) @src\mongo\transport\service_executor_adaptive_test.cpp:197
      

      Currently the test ifdef's the failing assertion out on Windows. We need to figure out why this timing issue occurs and fix it.

      Attachments

        Activity

          People

            backlog-server-servicearch Backlog - Service Architecture
            jonathan.reams@mongodb.com Jonathan Reams
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: