Clearing symbol file locations doesn't make native call-stacks back to unknown or hidden.
Hi,
A few days ago, I checked the option 'Download symbol files' in 'Advanced Options' when start profiling. And as expected, while taking a snapshot from the target process, symbol files for every modules including native code are downloaded. And after that, when I open the captured snapshot(*.dtt), I can see all native call-stacks even inside of KernelBase.dll or ntdll.dll like GetQueuedCompletionStatus which is API for IOCP. That's good.
But, when I tried to check .NET memory allocations, I saw some weird aspect. Every memory allocations are led to ETW::SamplingLog::SendStackTrace, like below.
It made me hard to match hotspots and call tree, a little, so I decied to clear downloaded symbol files for test. But even after deleting all files in downloaded symbol files and clearing the option 'Download symbol files', when I take a snapshot from the target process and open that, every native code stacks are visible as it does before deletion.
I deleted all files from two directories listed above, 'C:\Windows\Temp' and 'C:\Users\***\AppData\Local\Temp', but still native call stacks are visible like after I downloaded symbol files. There's no environment variable _NT_SYMBOL_PATH defined, and searched all disk *.pdb but could not find any candidate.
Clearing caches from Settings tab had no effect, too.
I'm using dotTrace 2022.3.2 on Windows 10 Enterprise x64(22H2, Build 19045.2486).
Is there anything should I check more to make it back to the state before downloading any symbol files? Or, am I misunderstanding something?
Thanks in advance.
Please sign in to leave a comment.
Hi,
Thank you for the feedback.
There are few options of how the symbol downloader works. If `Download symbol files` is set it takes paths provided in the edit box and downloads symbols to specified local symbol cache folders. If the edit box is empty the downloader checks `_NT_SYMBOL_PATH` variable if it is defined and if it doesn't the default symbol search path will be used. The default path should look like:
`srv*C:\Users\<user name>\AppData\Local\Temp\Symbols*https://msdl.microsoft.com/download/symbols`
If `Download symbol files` check box is not set the algorithm is the same except that no symbol downloading will happen and only local paths will be used.
If you clear `Symbol file locations` edit box and then push `Restore Defaults` you wiil see either a content of `_NT_SYMBOL_PATH` variable or the search default path if no such variable defined. Then please make sure that there is no symbol files in all these locations.
If it won't help please specify any random string in the edit box. It shouldn't be necessarily correct path, something like `abc` just to ensure that the edit box is not empty. And please send us logs from both `%temp%\JetLogs` and `%systemroot%\Temp\JetLogs. We are interested in `JetBrains.ETW.Collector` logs.
Thank you in advance!
Hello Synastry,
While we're looking into what might have gone wrong with the "Download symbols" option, can you please explain the difficulties to match hotspots and call tree? Is "Hide system functions" enabled in the hotspots?
Thanks for your feedback.
Alexey Korovin, Thank you for your response.
And I'm sorry, my explanation was unclear. The point what I got confused is about 'ETW::SamplingLog::SendStackTrace' which is appeared in 'Call Tree' while inspecting '.NET Memory Allocations' with captured snapshot, but not actually related with 'Hotspots'. Hotspots tab works well including the option "Hide system functions", you pointed.
The screenshot below shows what I got confused.
In screenshot, 'System.Rune.ColumnWidth' is divided into 3 children whose are green underlined, and all of them are led to 'ETW:SamplingLog:SendStackTrace' at the end of call tree.
In this situation, which method actually allocated memory? As the call tree, all of allocation in ColumnWidth is performed by SendStackTrace. But it makes no sense, because ETW is activated by attaching ETWHost to the target process via dotTrace. Also, if I exclude SendStackTrace method from result, 99.9% percent of allocation is gone from whole snapshot.
Without native stacks that appeared by downloading symbol files, I think I didn't misunderstand Call Tree like this way. So I said I got confused, and decided to clear downloaed symbols to hide native stack frames from Call Tree.
I'm not sure that I explained enough this time. If you need more information or something I can provide, please let me know.
@..., Thank you for your response.
The directories that I cleared, before posting this, are listed below.
- C:\Users\***\AppData\Local\Temp\Symbols (local cache for MS symbol server)
- D:\SymbolCache (another local symbol cache for our internal symbol server)
- C:\Windows\Temp
- C:\Users\***\AppData\Local\Temp
And there's no _NT_SYMBOL_PATH defined.
So I think I did the same with you suggested except pressing 'Restore Defaults'.
And as you noted, when I press 'Restore Defaults' button after make empty 'Symbol file locations' edit box,
srv*C:\Users\***\AppData\Local\Temp\Symbols*https://msdl.microsoft.com/download/symbols
is appeared, and made sure it's empty.
After that I captured a snapshot, and... yes, they're gone. It's weird but it works as expected, no more native stack frames appears. The only difference is that 'D:\SymbolCache' line in 'Symbol file locations', but I can't explain what happened now. Huh...
Anyway, my problem is gone now, even I don't know how it worked. ; )
I really appreciate your advice, and feel so sorry for bothering both of you, Evgeny and Alexey. What a silly situation this is. ; ) Sorry for making you read a long long post.
Thank you again, and have a nice day. : )
I'm glad to hear that everything works fine now.
I would also like to shed some light on what's going on in the call tree. You are absolutely right that ETW events collector works out of a target process. But ETW technology itself assumes interactions between event providers and consumers. Our ETW collector creates a consumer listening to .NET events. And the provider generally lives inside .NET runtime in order to send events to ETW. When you apply `.NET Memory Allocations` filter you see the stacks in which memory allocations happend along with calls of sending the event to ETW: `GCToCLREventSink::FireGCAllocationTick_V4` -> ... -> `ETW::SamplingLog::SendStackTrace`.
It's actually confusing since it looks like `SendStackTrace` function has allocated all this memory, but it actually hasn't.
So I've created an issue on this matter: https://youtrack.jetbrains.com/issue/DTRC-29334.
Thank you very much!
Wow, I didn't expect that the problem is planned to be fixed, but you did! 👍
And your explanation about ETW really helped me to understand what Call Tree is showing me more clearly.
Thank you so much! 😀