Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7860

Improve code coverage reporting

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT10.0.1, 4.4.9, 5.0.3, 5.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • 3
    • Storage - Ra 2021-07-26, Storage - Ra 2021-08-09

      There is an existing WT Evergreen job that measures code coverage for a selection of WT tests. This is great as measuring code coverage is a very useful component of software QA.

      While high coverage doesn't guarantee that testing is sufficient (as it may be necessary to exercise code paths repeatedly with multiple different inputs to test the code comprehensively, for example for numerical algorithms), low coverage typically indicates insufficient testing.

      Branch coverage is a much more useful measure of coverage than line coverage, and 80% branch coverage is typically the minimum coverage that should be maintained.

      The WT test coverage results from 15th July are attached and the details can be found here: https://evergreen.mongodb.com/task/wiredtiger_code_statistics_coverage_report_c85ab8d5c2ea820f709c5750b7c92ca2e62f3adc_21_07_15_03_24_14 

      The overall coverage reported on 15th July is 74.8% line coverage and 49.9% branch coverage. The focus of code coverage measurement should be on the main source code (eg not example or test code). For WT that is the src directory.

      The report does not show the average figures across the `src` directory, but it does show individual files, and some of them have limited or no testing shown. For example, the report show that src/cursor/cur_ds.c has no lines or branches covered. 

      It is likely that the low coverage figures are at least in part caused by the limited selection of tests to run with code coverage, so that is something we need to examine.

      Currently, the WT code coverage reporting has a significant limitation. The report is a summary page with one line per source code file. It would be much more useful to also have colour coded reporting per line of each file, as developers can then identify uncovered lines of code that need to be tested. WT uses gcovr for generating the reports, and gcovr has an option (--html-details) to generate such detailed reports - see https://gcovr.github.io/guide.html#html-output for an example.

      I attempted to create create detailed code coverage stats using the --html-details option on gcovr and got some strange warnings about the coverage data for some files having more lines of code than the source file itself. This needs further investigation. 

      The goal for this ticket is to modify the WT code coverage measurement to generate detailed (per line) code coverage data. Any required increase to the test coverage will be addressed in future Jira tickets.

            jeremy.thorp@mongodb.com Jeremy Thorp
            jeremy.thorp@mongodb.com Jeremy Thorp
            0 Vote for this issue
            5 Start watching this issue