What I want, from a .NET coverage tool, and haven't been able to find...
- Ability to run unit tests, and to track which lines of code have been executed by which unit tests, and which lines of code have not been executed at all.
- Ability to generate interactive reports, that allow the user to view statistical summaries of coverage by assembly, file, namespace, classe, etc;, and to drill down to the source code level to see exactly what lines were and weren't covered.
- Ability to run the tests and generate the reports from Jenkins, and to include the reports in the information that Jenkins displays for every build.
- Ability to mark source code, by line, as not needing to be covered.
- Ability to have the reports clearly indicate that every line of source code has either been covered, or marked as not needing to be covered.
I've been looking around at the free coverage tools, and haven't been satisfied. I recently began reading up on dotCover, and it certainly looks like it provides more of what I need. But from what I've read in the docs, it provides mechanisms for excluding code from coverage requirements by file, class, or function. But if it provides the capability to exclude code from coverage by line, I've missed it.
What I'm looking for, in essence, is a pass-fail. 90% coverage, or 80% coverage, or 95% coverage, means nothing to me. 80% coverage might mean 20% of the code doesn't need coverage, 99.9% coverage might mean that only one critical function that desperately needs testing hasn't been tested.
What I want is a pass-fail.
There are three categories:
- Code that has been covered by a test
- Code that has been flagged as not needing to be tested
- Code that has not been covered by a test, and has not been flagged as not needing to be tested
And what I want, after a unit test run, is a pass, if there is no code in category 3, or a fail if there is.
And I need to be able to flag code as not needing to be tested by line, and not just by function, because when I look at a function in a coverage report, it's pretty typical for me to see certain flows of control through the function that weren't executed by the tests. Most commonly, exception or error handling. Sometimes, the effort needed to create tests that will generate the errors or exceptions needed to test these flows of control can be significant, and sometimes the likelihood of the test revealing an issue is small.
In cases like these, I would like to be able to flag the code as having been inspected, and having been determined to not need testing. And when I flag the code in this way, I want the fact that the code wasn't covered by a test to not fail the build.
Does dotCover provide this capability?
If not, is it planned for the future?
Does anyone know of any other .NET coverage tool that does?