Are there any system logs available to confirm why "no snapshots collected" in linux?

Hi!

  If I got it right we are talking about dotTrace clt. dotTrace clt has separate option for taking logs but they usually used analyze issues on our side. In other words I don't think logs will help you.

  If you want to see which steps dotTrace has done during profiling process and probably deduce what went wrong you could turn `--service-output=on`. To see what output service messages dotTrace have and their meaning see stdout messages description in help `dottrace help service-messages`.

i have tried --service-output=on already, but that was not giving me required information. Here the problem is, due to some internal issue, Snapshots are not creating for the profiling session. I can see all the other logs like, "profile started etc, but last statment is "No snapshots created". Looks like something happened before saving the snapshot, but those internal error logs of dottrace are not visible to me. Since the hardware which Im using is customized Linux environment with lot of restrictions, it might be useful for me to rectify, if I can see the logs and causes for not saving the snapshot. My intuition, is might be due to some IPTABLES configuration. But I could not find any logs specific to it.

Have you seen ##dotTrace["connected"],  ##dotTrace["started"], ##dotTrace["snapshot-saved"], ##dotTrace["disconnected"] service messages? If you've seen connected - that means profiler is connected for example. Usually if dottrace gets error (for example cannot save snapshot), it prints an error to console (in red). If you don't see any of them it looks like there is really something unusual happened. But on which state? While connecting or maybe saving? That's why I mentioned --service-output before.

Don't you use profiling api? If not you could also control dottrace clt using input service messages. For example to control through stdin specify `--service-input=stdin`. As soon as profiler "connected" you can type ##dotTrace["start"] command. As a response you receive "started" output message. Then you could try ##dotTrace["get-snapshot"] command, etc. So you will know what doesn't work in your case.

Of course you can enable debug logs and write them into a separate file `--debug-logging --log-file=<path_to_log_file>`. If it would be convenient for you, you could upload such log file to https://uploads.jetbrains.com/ and probably we could help you.

I am still seeing the same problem on dotTrace 2024.1.5. Trying to attach the dottrace clt to a .NET Core 8.0 Linux binary then stopping with Ctrl-C always gives the same “no snapshots collected” message. 

Note that running an earlier 2323 version of dotTrace last year with .NET Core 7.0 Linux binary was working for me in attach mode, as does “dottrace start” with stdin service input control. But this method is not convenient for collecting snapshots from command line apps.

I could supply some logs for debug. I'm profiling arm64 Linux apps.

I think your problem is quite different since it's related to Ctrl-C.

The problem with Ctrl-C on Linux is that both processes will receive the signal and a profiled application will exit (default behavior for signal handler) before the snapshot will be collected.

To avoid this you can:

1. Send the SIGNIT signal directly to the console profiler (`kill -SIGINT <PID>` , make sure the specified PID is a profiler process, not a profiling application).

2. As you mentioned before, you could use `--service-input` option, but instead of stdin, you could specify a path to a file to read commands from (for example `dottrace attach 123 --service-input=/tmp/control_file.txt`). Then you could control profiler writing commands to this file instead of stdin (for instance, from another console run `echo  '##dotTrace["get-snapshot"]' >> /tmp/control_file.txt`).

Also consider using `--timeout` option if you know in advance how many time do you need to profile.

Just done a test to try the workaround you suggested, but it's still not working on any dotnet app. The profiler still exits without collecting any data. Here's the terminal output after attaching then being disconnected via a SIGINT sent from another shell:

~ $ dottrace attach 2869 --service-input=stdin --collect-data-from-start=on
dotTrace command-line profiler 2024.2 build 777.0.20240812.225234. Copyright (C) 2024 JetBrains s.r.o.
Preparing to profile...
Profiling in progress...
##dotTrace["ready"]
You can control profiling by sending command messages to stdin. For details, type ##dotTrace["help"]
Taking a snapshot... To quit without taking the snapshot, press Ctrl+C
Profiling is finished in 17.137 seconds
No snapshots have been collected

It looks like the profiler found the app from PID, but never actually connected properly?

Note that I've just upgraded to 2024.2 but the same problem as 2024.1.

Sure, something went wrong in your case, there must be ##dotTrace["connected"] message when profiler finished attaching procedure and ##dotTrace["started"] when collection is actually started.

For the investigation we need you to collect some logs. Could you please add those options --core-log-mask=0 --core-logs-directory=/tmp/dottrace-logs --debug-logging --log-file=/tmp/dottrace-logs/clt.log and reproduce the issue. Ensure you waited at least a minute for ##dotTrace["connected"] before sending SIGINT.

When the logs are collected please upload all files from the /tmp/dottrace-logs directory to https://uploads.jetbrains.com/ or create an issue in our tracker https://youtrack.jetbrains.com/newIssue?project=DTRC

Note that the logs could contain some personal information.

Whilst creating a test app I found the the scenario that triggers the bug in dotTrace.

It's because my application is published as a single file executable with SelfContained, PublishSingleFile and IncludeAllContentForSelfExtract all set to true. dotTrace fails to attach properly and snapshot apps on linux-arm64 with these properties set to true. 

Windows x64 apps work absolutely fine with the dottrace clt when published with the same parameters, as do linux-arm64 apps that are just framework-dependent. I will create an issue.

I have added a bug for this which has more details.

https://youtrack.jetbrains.com/issue/DTRC-31181/dotTrace-command-line-tool-fails-to-attach-to-Linux-arm64-single-file-.net-core-binaries