-
Type:
Task
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Developer Productivity
-
None
-
Storage Engines
-
5
-
2024-08-06 - Withholding Tax
Currently the code coverage report creates parallel build directories. After creating the build directories, it sections out all of the tests from the code_coverage.json file, into equal parts down the list. This causes a huge inefficiency in between each other, e.g.
[2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_1 : took 93.11265 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_1 : took 93.11265 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_2 : took 66.48227 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_3 : took 73.530605 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_4 : took 3.470759 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_5 : took 9.090019 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_6 : took 6.485636 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_7 : took 4.241151 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_8 : took 67.636908 seconds [2024/07/10 15:09:19.381] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_9 : took 70.906561 seconds [2024/07/10 15:09:31.454] DEBUG:root:Finished task test/cppsuite/run -t hs_cleanup -f test/cppsuite/configs/hs_cleanup_default.txt in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_10 : took 101.543348 seconds [2024/07/10 15:09:31.454] DEBUG:root:Running task test/csuite/wt2592_join_schema/test_wt2592_join_schema in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_10 [2024/07/10 15:09:32.236] Running test command: test/csuite/wt2592_join_schema/test_wt2592_join_schema [2024/07/10 15:09:32.236] ID 2: country AU, year 1950, population 8267337 [2024/07/10 15:09:32.236] ID 3: country AU, year 2000, population 19053186 [2024/07/10 15:09:32.236] ID 2: country AU, year 1950, population 8267337 [2024/07/10 15:09:32.236] ID 3: country AU, year 2000, population 19053186 [2024/07/10 15:09:32.236] ID 8: country UK, year 1950, population 50127000 [2024/07/10 15:09:32.236] ID 9: country UK, year 2000, population 59522468 [2024/07/10 15:09:32.236] DEBUG:root:Finished task test/csuite/wt2592_join_schema/test_wt2592_join_schema in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_10 : took 0.782423 seconds [2024/07/10 15:09:32.236] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_10 : took 105.95558 seconds [2024/07/10 15:09:32.237] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_10 : took 105.95558 seconds [2024/07/10 15:09:32.237] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_11 : took 31.776144 seconds [2024/07/10 15:09:32.237] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_12 : took 8.053 seconds [2024/07/10 15:09:32.237] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_13 : took 5.746886 seconds [2024/07/10 15:09:32.237] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_14 : took 2.704619 seconds [2024/07/10 15:09:32.237] DEBUG:root:Completed task list in /data/mci/fa445c1936bac8b45596cdadcd242b8f/wiredtiger/build_15 : took 3.818003 seconds
If we see here, we have builds performing 3 seconds while other builds taking overall 100 seconds.
This ticket involves creating shared queue mechanism between all of the threads that are running tests. The idea is that it takes from the queue when it needs a test command to run. This will equally distributed the load.