Improved performance when dotTrace is running

Hi there,

We are dealing with an interesting performance issue where our multithreaded program is much slower on a better system (almost 4x slower than it should be). In trying to investigate the cause, I've made a little testing program that simply does a lot of sorting of DataFrames using the Deedle library. This program has an even worse performance (up to 10x, depending on the number of threads), but here something odd happens. When I start dotTrace and select the Launch tab, the already running test program gets about twice as fast. This only happens from the moment that the Launch tab is opened, until dotTrace is closed (regardless of the tabs selected in between); nothing else is done with dotTrace.

Does anyone have an idea what could be causing this? Although starting dotTrace doesn't seem to affect our actual program much, this problem might point towards a crucial part in solving that issue as well.

Of course, any suggestions to what could be causing the problem in general would be welcome as well. Other test have indicated that the system itself is performing fine, but that for some reason there is a much stricter synchronisation going on in our program (mostly in Deedle, but that's also just where a lot of the work is done).

Many thanks in advance for any help.

7 comments
Official comment

Hello Muck,

It is a coincidence that launched dotTrace application improves program performance. Probably, several threads are trying to get access to the same resources and it causes idle problems, as a result, application performance becomes worse. Launched dotTrace also accesses to the resources and queue changes so that idle no longer occurs. Furthermore, the same behavior can be caused by any other application, not dotTrace only.

The best performance appears when threads count is no more than number of logical processors. When application has a lot of threads performance can vary by the particular situation, for example, +2 threads can significantly reduce performance, but +4 return it to its previous state (figures are given as an example, it is not a mandatory system behavior).

If dotTrace standalone application prevents you from reproducing performance issue, we can suggest you to profile your application via dotTrace Command-Line Profiler:
https://www.jetbrains.com/help/profiler/10.0/Performance_Profiling__Profiling_Using_the_Command_Line.html

Also you can use RemoteAgent:
https://www.jetbrains.com/help/profiler/10.0/Starting_Remote_Profiling_Session.html

It gives you the opportunity to profile application without starting dotTrace standalone on the computer with profiled application. 

Here's the data I collected for the mentioned test program. The "twice as fast" was clearly only for one case; overall the effect is even stronger.

On the X-axis is the number of threads used, on the Y-axis the time each run took. The blue line is on the old quad-core system, the dotted orange line is the projection based on the blue line for the new octa-core system, the orange line the actual performance on the new system and the yellow line that performance when dotTrace is idling in the background. The yellow line is about what one would expect given the problem in our real program.

0

Thanks for your response, Anna. I'll look into the alternative ways to profile the application.

Not sure I would call the effect of dotTrace a "coincidence", though. It's a very specific and reproducible behaviour which I haven't seen with anything else. So dotTrace is either accessing some very specific resource or perhaps changing some setting, adding some hook, etc. that causes this. Because it is something that only happens from the first time the "Launch" tab is selected, I was hoping you could tell me what exactly happens when selecting that tab that could have this result. I tried to look into it myself with the Logger of "Debugging Tools for Windows", but it doesn't cooperate with dotTrace well (i.e., dotTrace crashes on start).

0

Muck,

When dotTrace is opened on Launches tab, it does not perform different from what happens when you open other tabs. Probably tab interface somehow influences on it, but we can not say why it is happening exactly. It's a really strange behavior.

As a point, remote configurations periodically turn to remote server to maintain the connection. You can try to delete all recent launches nodes except one (it should be a simple local configuration) and check if dotTrace affects performance after that.

0

Even though I didn't have any remote configurations on that system, I tried removing the existing configurations as well as adding one for another program or one for the "target" program. The odd behaviour remains. Some additional experimentation shows that the "Remote" tab has the same effect, but none of the others do.

0

An additional note: the big delay on the octa-core system is almost completely gone when using the server garbage collector. However, it is still slower than the quad-core system and the dotTrace effect is still there (just less big).

0

Another update: the main performance issue we were having has been solved[1] by updating .NET from 4.5 to 4.6. However, for the test program, we are still seeing the increased performance when dotTrace is around. Note that using the console profiler or remote profiling on the program also resulted in the improved performance, so I've not been able to get further information about this problem.

[1] http://stackoverflow.com/a/38392258/4285 

0

Please sign in to leave a comment.