.net core 7 support

Hello,

.NET 7 has several known issues relates to profiling API which should be fixed in runtime itself.

Could you please collect dotMemory logs? you can enable logs from command line:

./dotmemory get-snapshot <PID> --log-level=Trace --core-log-mask=Com,Api,Bridge,Snapshot,MetaDataGt,ThreadGt,ClassGt,FieldResolverGt,ClassRecoveryGt,AppDomainRecoveryGt,ThreadRecoveryGt,MemoryDumpGt,GcCb

Log files will be located in the JetLogs folder in user's temp directory. 

Please collect logs on Windows and on Ubuntu machines.

Hi,

We've reproduced this issue in our environment. This is a known issue in dotnet runtime: https://github.com/dotnet/runtime/issues/78831

We hope this fix will be available in the next .NET version (7.0.2).

In some rare occasions it randomly attaches to the process but then it crashes the process before it can get a snapshot. 

What error occurs on getting the snapshot? Could you please provide requested logs for this case?

HI, 

Dotnet runtime already public 7.0.2 version, but the question still stands(multiple runs has a probability of crash).

Please help to locate this problem

Hello,

We’ve verified described issue (https://github.com/dotnet/runtime/issues/78831) and attach works properly in our environment if .net 7.0.2 runtime is installed. Could you please type dotnet –version in command line to check which dotnet version is installed? Please send us console output.

It looks like you experience some other issue. Could you please configure automatic crash dumps collection on your machine:

Windows

• Download the procdump utility: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump

• Unpack it to any folder, e.g. “C:\Program Files”

• Run cmd as administrator and go to procdump folder

• Enter command procdump.exe -i <Path to crash dump folder> -ma

• Attach dotMemory and wait profiled application crashes;

• Find a newly created dump in <Path to crash dump folder> directory and share with us via https://uploads.jetbrains.com/ form.

Linux

1. The location of core dumps is specified in the `/proc/sys/kernel/core_pattern`, please be sure it will not be piped to another program by `|/path/to/program` pattern, otherwise you have to reconfigure it to /tmp directory using echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern (should be executed in root console, type sudo -i to enter), and restore the previous core_pattern later;

2. Type exit to exit root console if you use it;

3. Check the directory specified in your core_pattern file (usually it is /tmp/ or current working directory);

4. Run the command ulimit -c unlimited in user console and run your application from exactly this console;

5. Attach dotMemory and wait profiled application crashes;

6. Find a newly created dump in the appropriate directory and share with us via https://uploads.jetbrains.com/ form.

Hi Anna Guseva:

Thank you for your prompt reply, I'm so sorry for replying to you late.

cat /proc/version
Linux version 3.10.0-693.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017

[root@bogon tmp]# dotnet --info
.NET SDK:
 Version:   7.0.102
 Commit:    4bbdd14480

Runtime Environment:
 OS Name:     centos
 OS Version:  7
 OS Platform: Linux
 RID:         centos.7-x64
 Base Path:   /data/install/.dotnet/sdk/7.0.102/

Host:
  Version:      7.0.2
  Architecture: x64
  Commit:       d037e070eb

.NET SDKs installed:
  7.0.102 [/data/install/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.2 [/data/install/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.2 [/data/install/.dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/data/install/.dotnet]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

dotnet build --debug -r linux-x64 --self-contained true --source https://nuget.cdn.azure.cn/v3/index.json -o /app Server/Server.csproj

Launch app:
    /app/Server --args 

    
The usage of Dotmemory is as follows:

Execute the following "ExecCMD" method via http access, and it will launch the ".sh" script, To expose the problem faster, access http every 2 minutes during testing.

1:
        public static string ExecCMD(string command, string type, string paramA, string paramB)
        {
            try
            {
                string output = String.Empty;
                using (System.Diagnostics.Process pro = new System.Diagnostics.Process())
                {
                    pro.StartInfo.FileName = "cmd.exe";
                    if (Environment.OSVersion.Platform == PlatformID.Unix)
                    {
                        pro.StartInfo.FileName = "/bin/bash";
                        if (command.IndexOf("dotmemory", StringComparison.Ordinal) != -1)
                            command = "/app/dotMemory/" + command;
                        else if (command.IndexOf("sh", StringComparison.Ordinal) != -1)
                        {
                            command = "/app/" + paramB;
                        }
                        else
                            command = "/app/" + command;

                        pro.StartInfo.WorkingDirectory = "/app";
                        pro.StartInfo.Arguments = command;
                    }

                    pro.StartInfo.UseShellExecute = true;
                    pro.StartInfo.CreateNoWindow = true;
                    pro.StartInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;

                    pro.Start();
                    pro.Close();
                    pro.Dispose();
                    Logger.SLog()
                        .Debug($"agent execCMD success: command: {command}, output: {output}", SandwichLogModuleKey
                            .DotNetAgent);
                    return output;
                }
            }
            catch (Exception e)
            {
                Logger.SLog().Error($"agent execCMD get error: command: {command}", e, SandwichLogModuleKey.DotNetAgent,
                    SandwichErrorCode.DotNetAgentError);
                throw;
            }
        }
        



2:
    #!/bin/sh
    rm -f /tmp/dump.dmw
    /app/dotMemory/dotmemory get-snapshot 1 -o -f=/tmp/dump.dmw > /dev/null
    #echo "success"

The error log is as follows:

{ "date": "2023-01-23T23:28:01.581", "log_level": "debug", "caller": "CMD.cs:164", "message": "agent execCMD success: command: /app/dotMemory.sh, output: " }
{ "date": "2023-01-23T23:28:01.581", "log_level": "debug", "caller": "Agent.cs:124", "message": "CMD exec sh dotMemory.sh, tpMsg: " }
cannot create thread specific key for __cxa_get_globals()
  ---
Unable to start profiling. Unable to read beyond the end of the stream.
Aborted (core dumped)

Attach is the coredump file, please review it. 
The file download: https://livetopiam-hot.s3.us-west-2.amazonaws.com/Coredump/core.Server.31854

Thank you for your patience and cooperation, If you have any questions or concerns, don't hesitate to let me know.

Thank you for information, we'll investigate this dump. You're welcome to follow corresponding ticket.

Look forward to your feedbacks and suggestions soon, I tested "dotTrace" again and it worked.

After my repeated tests over the last few days, run the Dotmemory at every 2 mimute like below:

/dotMemory/dotmemory get-snapshot $pid -o -f=/tmp/dump.dmw

there will be a crash in 36 hours. the question is reproducible, the following error is reported:

"cannot create thread specific key for __cxa_get_globals()
  ---
Unable to start profiling. Unable to read beyond the end of the stream."

Hope the above information is useful to you, your kind assistance on this are very much appreciated.

Hi!

As mentioned here, the problem relates to C++ toolset. This task requires huge investigation and won't be solved in a short time. You're welcome to vote for and follow this ticket to get any updates on it.

As a workaround, we can suggest you not to attach every 2 minutes but use triggers to get memory snapshots: 

./dotmemory attach $pid --trigger-timer=2m

You can also specify some other parameters to control profiling session, e.g., profiling session timeout, use ./dotmemory help attach to read help. As a result, you'll get a single workspace file with collected snapshots.

If this use case is not suitable for you, could you please describe your scenario in details? Why do you perform profiling this way (I mean attaching every 2 minutes)?