Home .NET Finding memory leaks in .NET Core applications on Linux

Finding memory leaks in .NET Core applications on Linux

by admin

NET Core is becoming a more and more mature platform.It’s already comfortable enough to develop on it using the same Rider or VS Code.

However, not everything is smooth there either.For example, code debugging on .NET Core 2 only worked in Rider 2017.2, which came out, just the other day (there were more EAP builds). I had to use VS Code. It works debugging, but to make start of tests works it’s necessary to put beta-version on it manually. C# extensions

I think the point is clear that the tool support is still a far cry from that of Windows development.

For some things, there are no ready-to-use tools yet. For example, for profiling.

Of the sources that are available online, the most informative, in my opinion, at the moment are the articles by Sasha Goldstein :

However, I didn’t manage to find a recipe for finding memory leaks. So I decided to describe the method I found.

How we would act under Windows

Personally, without thinking twice, I would connect to a running application with dotMemory, take 2 snapshots after a period of time and compare them, using a nice GUI.

Finding memory leaks in .NET Core applications on Linux

How cool it would be if under Linux we could do something similar.

Let’s give it a try.

The example we are going to consider

There is nothing complicated here. Of course, you don’t have to resort to any tools to find the leak in the next code. But it will do for training purposes.

using System;namespace leak_example{class Program{static void Main(string[] args){Function1();}private static void Function1(){var leakClass = new LeakClass();leakClass.DoWork();}}}

using System.Collections.Concurrent;using System.Threading.Tasks;namespace leak_example{public class LeakClass{private BlockingCollection<string> _collection;public LeakClass(){_collection = new BlockingCollection<string> (new ConcurrentQueue<string> ());}public void DoWork(){while(true) {_collection.Add(System.Guid.NewGuid().ToString());Thread.Sleep(20);}}}}

How to make a snapshot of a working Linux application

It is quite easy to make a core dump under Linux for a running application. The following 2 commands do that :

$ ulimit -c unlimited$ sudo gcore -o dump1 $(pidof dotnet)

And after a while we make a second snapshot

$ sudo gcore -o dump2 $(pidof dotnet)

We received 2 dumps of our application :

$ ls -lah dump*-rw-r--r-- 1 root root 5.7G oct 18 17:01 dump1.13486-rw-r--r-- 1 root root 6.2G oct 18 17:03 dump2.13486

which we can now try to compare.

How to look into the snapshot of a .NET Core application in theory

Microsoft provides a plugin for LLDB which can help us with this. It is a ported SOS extension from WinDBG with a similar command set.

In theory, to see the memory allocations with the above snapshot we would have to run the following commands :

$ lldb $(which dotnet) --core <dump>(lldb) plugin load libsosplugin.so(lldb) sos DumpHeap -stat

In Sasha Goldstein’s articles, the path to the CLR is also set with the command

(lldb) setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0

But I didn’t need this to investigate the Debug build problems of my test application.

Harsh reality

  1. Microsoft supplies libsoplugin.so along with .NET Core. So you probably have it on your system.

  2. As Sasha wrote in his article, unfortunately, this plugin is linked to a specific version of LLVM. Accordingly, you’ll need a specific version of LLDB to use it.

  3. To see the specific version, it is no longer possible as before, using the command ldd :

    $ ldd /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libsosplugin.solinux-vdso.so.1 => (0x00007ffc31dcb000)libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f65b93bb000)libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f65b90b2000)libgcc_s.so.1 => /lib/x86_x86_64-linux-gnu/libgcc_s.so.1 (0x00007f65b8e9c000)libc.so.6 => /lib/x86_x86_64-linux-gnu/libc.so.6 (0x00007f65b8ad2000)/lib64/ld-linux-x86-64.so.2 (0x00007f65b994e000)

    The fact is that liblldd.so was called differently in different distributions and its was removed from the explicit dependencies

  4. Some of the issues on GitHub have information that in .NET Core 2.0 this plugin is built with lldb-3.8, and the version in .NET Core 2.0.1 will already be built with lldb-3.9.

  5. It would seem that now we know the version and we can just put the lldb of the right version into the system. But no. The fact is that lldb before version 4.0 could not load on-demand core dumps. Exactly what we did (while the program was running).

  6. So it turns out that we have 2 ways, either to build a plugin for lldb-4.0, or to patch and build lldb-3.8 by ourselves. I did the first way.

Building libsoplugin with lldb-4.0

Fortunately, we only need the plugin. We won’t have to use the custom .NET Core, etc. We’ll use the plugin directly from the folder where we build it, so the system won’t get littered.

Actually, in the CoreCLR repository there are instructions on how to build on Linux. We just tweak them a bit to take advantage of lldb-4.0.

I am using Ubuntu 16.04. For other distributions the command set may be slightly different.

  1. Putting the necessary tools for assembly :

    $ sudo apt install cmake llvm-4.0 clang-4.0 lldb-4.0 liblldb-4.0-dev libunwind8 libunwind8-dev gettext libicu-dev liblttng-ust-dev libcurl4-openssl-dev libssl-dev uuid-dev libnuma-dev libkrb5-dev

  2. Clone the repository :

    $ git clone https://github.com/dotnet/coreclr.git$ git checkout release/2.0.0

  3. Apply the following patch :

    diff --git a/src/ToolBox/SOS/lldbplugin/CMakeLists.txt b/src/ToolBox/SOS/lldbplugin/CMakeLists.txtindex fe816ab..ef9846d 100644--- a/src/ToolBox/SOS/lldbplugin/CMakeLists.txt+++ b/src/ToolBox/SOS/lldbplugin/CMakeLists.txt@@ -76, 6 +76, 7 @@ endif()find_path(LLDB_H "lldb/API/LLDB.h" PATHS "${WITH_LLDB_INCLUDES}" NO_DEFAULT_PATH)find_path(LLDB_H "lldb/API/LLDB.h")+find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-4.0/include")find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.9/include")find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.8/include")find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.7/include")

  4. Assembling CoreCLR:

    $ ./build.sh clang4.0

Hooray, we have a working plugin for lldb-4.0.

Putting theory into practice

  1. Opening our snapshot in lldb:

    sudo lldb-4.0 $(which dotnet) --core ./dump1.13486

  2. Loading the assembled plugin :

    (lldb) plugin load /home/user/works/coreclr/bin/Product/Linux.x64.Debug/libsosplugin.so

  3. Dumping statistics on heap usage :

    (lldb) sos DumpHeap -statStatistics:MT Count TotalSize Class Name00007fe36870b4c8 1 24 System.Collections.Generic.GenericEqualityComparer`1[[System.Int32, System.Private.CoreLib]]00007fe3686efea8 1 24 System.Threading.AsyncLocalValueMap+EmptyAsyncLocalValueMap...00007fe367cf7038 1 131096 System.Collections.Concurrent.ConcurrentQueue`1+Segment+Slot[[System.Threading.IThreadPoolWorkItem, System.Private.CoreLib]][]00007fe3686dcec8 19898 477552 System.Threading.TimerHolder00007fe3686bfc70 19898 477552 System.Threading.Timer00007fe3686dcd18 16261 650440 System.Threading.QueueUserWorkItemCallback00007fe3686c4430 19898 1751024 System.Threading.TimerQueueTimer00007fe3686bfbb8 19898 2228576 System.Threading.Tasks.Task+DelayPromise00007fe367d08498 19 268435400 UNKNOWN0000000001b465a0 3069079 449192802 Free00007f53bc74b460 13903273 1362548770 System.StringTotal 17068427 objects

    The first column is the address of the method table for objects of the class, the second is the number of allocated objects of the class, the third is the number of allocated bytes, and the fourth is the name of the class.

    It’s not hard to guess from the conclusion who’s leaking.

  4. Get the stack where the object was created (commands can take a very long time):

    (lldb) sos DumpHeap -mt 00007f53bc74b460...00007f539d0712c0 00007f53bc74b460 9800007f539d071420 00007f53bc74b460 9800007f539d071580 00007f53bc74b460 9800007f539d0716e0 00007f53bc74b460 9800007f539d071840 00007f53bc74b460 9800007f539d0719a0 00007f53bc74b460 9800007f539d071b00 00007f53bc74b460 9800007f539d071c60 00007f53bc74b460 9800007f539d071dc0 00007f53bc74b460 9800007f539d071f20 00007f53bc74b460 9800007f539d072080 00007f53bc74b460 9800007f539d0721e0 00007f53bc74b460 9800007f539d072340 00007f53bc74b460 9800007f539d0724a0 00007f53bc74b460 9800007f539d072600 00007f53bc74b460 9800007f539d072760 00007f53bc74b460 9800007f539d0728c0 00007f53bc74b460 9800007f539d072a20 00007f53bc74b460 9800007f539d072b80 00007f53bc74b460 98...

    We get a huge table in which the first column is the instance address of this class, the second is the address of the method table, and the third is the size of the instance in bytes.

    Select an instance and run the command :

    (lldb) sos GCRoot 00007f539d072b80Thread 4303:00007FFC92921910 00007F53BC9C0E30 leak_example.LeakClass.DoWork() [/home/ilya/works/trading/leak-example/LeakClass.cs @ 18]rbp-48: 00007ffc92921918-> 00007F539D072B80 System.String00007FFC92921910 00007F53BC9C0E30 leak_example.LeakClass.DoWork() [/home/ilya/works/trading/leak-example/LeakClass.cs @ 18]rbp-40: 00007ffc92921920-> 00007F5394014878 <error>-> 00007F5394014530 <error>-> 00007F53984B4A10 <error>-> 00007F53A47E35F0 <error>-> 00007F539D072B80 System.StringFound 2 unique roots (run '!GCRoot -all' to see all roots).

    And as we can see, we are pointed to just the line

    _collection.Add(System.Guid.NewGuid().ToString());

    in the original file.

Snapshot comparison

Since I started off talking about comparing snapshots, I’ll have to at least compare them somehow.

By saving to files dump1.txt and dump2.txt command outputs sos DumpHeap -stat for both snapshots (I just copied them from the console), I processed them with this simple script here (I actually wrote directly in the iPython console, so it doesn’t really look like a script)

dump1 = open('dump1.txt')lines1 = dump1.readlines()methodTables = {}for s in lines1:if s.startswith('000'):(mt, cnt, sz, name) = s.split(maxsplit=3)if not mt in methodTables:methodTables[mt] = {'cnt1': cnt, 'sz1': sz, 'name': name}dump2 = open('dump2.txt')lines2 = dump2.readlines()for s in lines2:if s.startswith('000'):(mt, cnt, sz, name) = s.split(maxsplit=3)if not mt in methodTables:methodTables[mt] = {'cnt2': cnt, 'sz2': sz, 'name': name}else:methodTables[mt]['cnt2'] = cntmethodTables[mt]['sz2'] = szfor mt in methodTables.keys():if 'cnt1' in methodTables[mt] and 'cnt2' in methodTables[mt]:cnt1 = int(methodTables[mt]['cnt1'])sz1 = int(methodTables[mt]['sz1'])cnt2 = int(methodTables[mt]['cnt2'])sz2 = int(methodTables[mt]['sz2'])if (cnt2 > cnt1 and cnt2 > 100 and sz2 > 1024 * 1024):print(mt, cnt1, cnt2, methodTables[mt]['name'])

The result is a list of "hotspots" which are worth paying attention to:

Finding memory leaks in .NET Core applications on Linux

Conclusion

I hope this impromptu way to find memory leaks will be useful for someone.

The attentive reader, of course, will notice that it would be nice to automate the whole process, with lldb scripts. But I don’t have time to do that yet. Couldn’t solve the problem with python-lldb-4.0 installed from the repository refuses to load libsoplugin.so Perhaps someone else will take it further.

Thank you for your consideration!

You may also like