dotCover continuous testing doesn't work properly with quick code changes
Hi, I've been using the dotCover evaluation as a potential replacement for NCrunch, but one glaringly obvious issue I have with it is the potential to show false positives/negatives when using continuous testing. When making code changes on "On save, build and run dirty tests" mode, it runs beautifully. The issue, however, is if you make a further code change while the test runner is running that test, it will continue to run with the old code and won't compile and run tests with the newly saved code.
To replicate:
- Write a unit test and some code to cover
- Let the continuous runner execute the test
- Make a code change that won't break the test and wait for continuous testing to start
- Make another code change that will break the test before the runner has finished
- The test runner only runs the first test, shows Success and does nothing else
This is a major flaw as waiting for each test runner process to complete before saving a code change isn't really a viable option, and it has the potential to provide some very dangerous results.
I think with continuous testing, it should either terminate the previous test runner process, or use a queue to initiate all further code change checks one after the other.
Please sign in to leave a comment.
My development team are evaluating dotCover at the moment and are experiencing exactly this problem, very frustrating. We were looking at upgrading our licences to ultimate to get the functionality but this problem and the overall performance issues mean we'll probably have to look towards NCrunch instead.
It would be good to get some kind of reply from JetBrains on these issues and when they anticipate having them resolved.
Hello Matt and Simon,
I'm sorry for the delay in response!
Unfortunately, I can't reproduce the behavior described by Matt. By design continuous testing behaves exactly how Matt said it should do: if a new analysis start-event ("save" or "build") occurs while the current analysis is not finished yet, the current analysis gets aborted and the new analysis gets enqueued (in many cases abort can't be finished immediately).
I've made a screencast illustrating the issue: https://www.screencast.com/t/lq1Bn7CO8p
In this video you can see that I have a long-running method Method1 in class JustClass and a test-method that covers Method1. I make several code changes in Method1 while continuous testing analysis is in progress. Every time I see that the current analysis gets aborted and the new one starts.
2Simon: are you evaluating the latest ReSharper Ultimate version? If not, could you please try to update? In ReSharper Ultimate 2017.2 some continuous testing performance issues have been fixed.
@Ekaterina yes my team were using the latest version of ReSharper Ultimate, unfortunately we're now trialing NCrunch as we're finding dotCover too slow and almost anti TDD. Are there plans for a new version of dotCover to address the performance issues at all as I'd prefer my developers to be using the one solution?