perf kmem: Implement page allocation analysis (v1)

tools/perf/builtin-kmem.c +5 -0

+static int parse_slab_opt(const struct option *opt __maybe_unused,
+			  const char *arg __maybe_unused,
+			  int unset __maybe_unused)
+{
+	kmem_slab = (kmem_page + 1);

On Thu, Mar 12, 2015 at 04:32:49PM +0900, Namhyung Kim wrote:

SNIP

Quoted 9 lines

tools/perf/builtin-kmem.c +9 -0

+static int parse_slab_opt(const struct option *opt __maybe_unused,
+			  const char *arg __maybe_unused,
+			  int unset __maybe_unused)
+{
+	kmem_slab = (kmem_page + 1);
+	return 0;
+}
+
+static int parse_page_opt(const struct option *opt __maybe_unused,

tools/perf/builtin-kmem.c static int parse_alloc_opt(const struct option *opt __maybe_unused, · line 897

Quoted 6 lines

tools/perf/builtin-kmem.c +6 -0

+			  const char *arg __maybe_unused,
+			  int unset __maybe_unused)
+{
+	kmem_page = (kmem_slab + 1);
+	return 0;
+}

hum, just curious.. why not just assign 1, I'm missing the magic ;-)

Hello,

Currently perf kmem command only analyzes SLAB memory allocation. And I'd like to introduce page allocation analysis also. Users can use
--slab and/or --page option to select it. If none of these options
are used, it does slab allocation analysis for backward compatibility.

The patch 1-3 are bugfix and cleanups. Patch 4 implements basic support for page allocation analysis, patch 5 deals with the callsite and finally patch 6 implements sorting.

In this patchset, I used two kmem events: kmem:mm_page_alloc and kmem_page_free for analysis as they can track every memory allocation/free path AFAIK. However, unlike slab tracepoint events, those page allocation events don't provide callsite info directly. So I recorded callchains and extracted callsites like below:

Normal page allocation callchains look like this:

360a7e __alloc_pages_nodemask
3a711c alloc_pages_current
357bc7 __page_cache_alloc <-- callsite
357cf6 pagecache_get_page
48b0a prepare_pages
494d3 __btrfs_buffered_write
49cdf btrfs_file_write_iter
3ceb6e new_sync_write
3cf447 vfs_write
3cff99 sys_write
7556e9 system_call

    f880 __write_nocancel

33eb9 cmd_record
4b38e cmd_kmem
7aa23 run_builtin
27a9a main
20800 __libc_start_main

But first two are internal page allocation functions so it should be skipped. To determine such allocation functions, I used following regex:

^_?_?(alloc|get_free|get_zeroed)_pages?

This gave me a following list of functions (you can see this with -v):

alloc func: __get_free_pages
alloc func: get_zeroed_page
alloc func: alloc_pages_exact
alloc func: __alloc_pages_direct_compact
alloc func: __alloc_pages_nodemask
alloc func: alloc_page_interleave
alloc func: alloc_pages_current
alloc func: alloc_pages_vma
alloc func: alloc_page_buffers
alloc func: alloc_pages_exact_nid

After skipping those function, it got '__page_cache_alloc'.

Other information such as allocation order, migration type and gfp flags are provided by tracepoint events.

Basically the output will be sorted by total allocation bytes, but you can change it by using -s/--sort option. The following sort keys are added to support page analysis: page, order, mtype, gfp. Existing 'callsite', 'bytes' and 'hit' sort keys also can be used.

An example follows:

  # perf kmem record --slab --page sleep 1
  [ perf record: Woken up 0 times to write data ]
  [ perf record: Captured and wrote 49.277 MB perf.data (191027 samples) ]

  # perf kmem stat --page --caller -l 10 -s order,hit

--------------------------------------------------------------------------------------------
Total_alloc/Per | Hit | Order | Migrate type | GFP flag | Callsite

--------------------------------------------------------------------------------------------

       65536/16384 |        4 |     2 |  RECLAIMABLE | 00285250 | new_slab
    51347456/4096  |    12536 |     0 |      MOVABLE | 0102005a | __page_cache_alloc
       53248/4096  |       13 |     0 |    UNMOVABLE | 002084d0 | pte_alloc_one
       40960/4096  |       10 |     0 |      MOVABLE | 000280da | handle_mm_fault
       28672/4096  |        7 |     0 |    UNMOVABLE | 000000d0 | __pollwait
       20480/4096  |        5 |     0 |      MOVABLE | 000200da | do_wp_page
       20480/4096  |        5 |     0 |      MOVABLE | 000200da | do_cow_fault
       16384/4096  |        4 |     0 |    UNMOVABLE | 00000200 | __tlb_remove_page
       16384/4096  |        4 |     0 |    UNMOVABLE | 000084d0 | __pmd_alloc
        8192/4096  |        2 |     0 |    UNMOVABLE | 000084d0 | __pud_alloc
   ...             | ...      | ...   | ...          | ...      | ...

--------------------------------------------------------------------------------------------

SUMMARY (page allocator)
========================
Total alloc requested: 12593
Total alloc failure : 0
Total bytes allocated: 51630080
Total free requested: 115
Total free unmatched: 67
Total bytes freed : 471040

Order UNMOVABLE RECLAIMABLE MOVABLE RESERVED CMA/ISOLATE
----- ------------ ------------ ------------ ------------ ------------

      0            32             0         12557             0             0
      1             0             0             0             0             0
      2             0             4             0             0             0
      3             0             0             0             0             0
      4             0             0             0             0             0
      5             0             0             0             0             0
      6             0             0             0             0             0
      7             0             0             0             0             0
      8             0             0             0             0             0
      9             0             0             0             0             0
     10             0             0             0             0             0

I have some idea how to improve it. But I'd also like to hear other idea, suggestion, feedback and so on.

Quoted 11 lines · Namhyung Kim

Hello,

Currently perf kmem command only analyzes SLAB memory allocation. And I'd like to introduce page allocation analysis also. Users can use
--slab and/or --page option to select it. If none of these options
are used, it does slab allocation analysis for backward compatibility.

The patch 1-3 are bugfix and cleanups. Patch 4 implements basic support for page allocation analysis, patch 5 deals with the callsite and finally patch 6 implements sorting.

Quoted 5 lines

In this patchset, I used two kmem events: kmem:mm_page_alloc and kmem_page_free for analysis as they can track every memory allocation/free path AFAIK. However, unlike slab tracepoint events, those page allocation events don't provide callsite info directly. So I recorded callchains and extracted callsites like below:

Really cool features!

I have a couple of output typography observations:

Quoted 50 lines

Normal page allocation callchains look like this:

360a7e __alloc_pages_nodemask
3a711c alloc_pages_current
357bc7 __page_cache_alloc <-- callsite
357cf6 pagecache_get_page
48b0a prepare_pages
494d3 __btrfs_buffered_write
49cdf btrfs_file_write_iter
3ceb6e new_sync_write
3cf447 vfs_write
3cff99 sys_write
7556e9 system_call

    f880 __write_nocancel

33eb9 cmd_record
4b38e cmd_kmem
7aa23 run_builtin
27a9a main
20800 __libc_start_main

But first two are internal page allocation functions so it should be skipped. To determine such allocation functions, I used following regex:

^_?_?(alloc|get_free|get_zeroed)_pages?

This gave me a following list of functions (you can see this with -v):

alloc func: __get_free_pages
alloc func: get_zeroed_page
alloc func: alloc_pages_exact
alloc func: __alloc_pages_direct_compact
alloc func: __alloc_pages_nodemask
alloc func: alloc_page_interleave
alloc func: alloc_pages_current
alloc func: alloc_pages_vma
alloc func: alloc_page_buffers
alloc func: alloc_pages_exact_nid

After skipping those function, it got '__page_cache_alloc'.

Other information such as allocation order, migration type and gfp flags are provided by tracepoint events.

Basically the output will be sorted by total allocation bytes, but you can change it by using -s/--sort option. The following sort keys are added to support page analysis: page, order, mtype, gfp. Existing 'callsite', 'bytes' and 'hit' sort keys also can be used.

An example follows:

Quoted 8 lines

  # perf kmem record --slab --page sleep 1
  [ perf record: Woken up 0 times to write data ]
  [ perf record: Captured and wrote 49.277 MB perf.data (191027 samples) ]

  # perf kmem stat --page --caller -l 10 -s order,hit

--------------------------------------------------------------------------------------------
Total_alloc/Per | Hit | Order | Migrate type | GFP flag | Callsite

s/Per/Size
s/Hit/Hits
s/Migrate type/Migration type
s/GFP flag/GFP flags

?

Quoted 15 lines

--------------------------------------------------------------------------------------------

       65536/16384 |        4 |     2 |  RECLAIMABLE | 00285250 | new_slab
    51347456/4096  |    12536 |     0 |      MOVABLE | 0102005a | __page_cache_alloc
       53248/4096  |       13 |     0 |    UNMOVABLE | 002084d0 | pte_alloc_one
       40960/4096  |       10 |     0 |      MOVABLE | 000280da | handle_mm_fault
       28672/4096  |        7 |     0 |    UNMOVABLE | 000000d0 | __pollwait
       20480/4096  |        5 |     0 |      MOVABLE | 000200da | do_wp_page
       20480/4096  |        5 |     0 |      MOVABLE | 000200da | do_cow_fault
       16384/4096  |        4 |     0 |    UNMOVABLE | 00000200 | __tlb_remove_page
       16384/4096  |        4 |     0 |    UNMOVABLE | 000084d0 | __pmd_alloc
        8192/4096  |        2 |     0 |    UNMOVABLE | 000084d0 | __pud_alloc
   ...             | ...      | ...   | ...          | ...      | ...

--------------------------------------------------------------------------------------------

Quoted 8 lines

SUMMARY (page allocator)
========================
Total alloc requested: 12593
Total alloc failure : 0
Total bytes allocated: 51630080
Total free requested: 115
Total free unmatched: 67
Total bytes freed : 471040

I'd suggest the following changes to the format:

- Collapse stats into 3 groups: 'allocated+freed', 'allocated only', 'freed only', depending on how much of their lifetime we've managed to trace. These groups are really distinct and it makes little sense to mix up their stats.

- Add commas to the numbers, to make it easier to read and compare larger numbers.

- Right-align the numbers, to make them easy to compare when they are placed under each other.

- Merge the 'count' and 'bytes' stats into a single line, so that it's more compact, easier to navigate, but also only comparable type numbers are placed under each other.

I.e. something like this (mockup) output:

SUMMARY (page allocator)
========================

Pages allocated+freed: 12,593 [ 51,630,080 bytes ]

Pages allocated-only: 2,342 [ 1,235,010 bytes ]
Pages freed-only: 67 [ 135,311 bytes ]

Page allocation failures : 0

Quoted 4 lines

Order UNMOVABLE RECLAIMABLE MOVABLE RESERVED CMA/ISOLATE
----- ------------ ------------ ------------ ------------ ------------

Quoted 11 lines

      0            32             0         12557             0             0
      1             0             0             0             0             0
      2             0             4             0             0             0
      3             0             0             0             0             0
      4             0             0             0             0             0
      5             0             0             0             0             0
      6             0             0             0             0             0
      7             0             0             0             0             0
      8             0             0             0             0             0
      9             0             0             0             0             0
     10             0             0             0             0             0

Here I'd suggest the following refinements:

- Use '.' instead of '0', to make actual nonzero values stand out visually, while still keeping a tabular format

- Merge the 'Reserved', 'CMA/Isolate' columns into a single 'Special'
colum: this will be zero in 99.9% of the cases, as those pages mostly deal with driver interfaces, mostly used during init/deinit.

- Capitalize less.

- Use comma-separated numbers for better readability.

So something like this:

Order Unmovable Reclaimable Movable Special
----- ------------ ------------ ------------ ------------

       0            32             .        12,557             .
       1             .             .             .             .
       2             .             4             .             .
       3             .             .             .             .
       4             .             .             .             .
       5             .             .             .             .
       6             .             .             .             .
       7             .             .             .             .
       8             .             .             .             .
       9             .             .             .             .
      10             .             .             .             .

Look for example how easily noticeable the '4' value is now, while it was pretty easy to miss in the original table.

Quoted 3 lines

I have some idea how to improve it. But I'd also like to hear other idea, suggestion, feedback and so on.

So there's one thing that would be useful: to track pages allocated on one node, but freed on another. Those kinds of allocation/free patterns are especially expensive and might make sense to visualize.

Quoted 10 lines

I.e. something like this (mockup) output:

   SUMMARY (page allocator)
   ========================

   Pages allocated+freed:       12,593   [     51,630,080 bytes ]

   Pages allocated-only:         2,342   [      1,235,010 bytes ]
   Pages freed-only:                67   [        135,311 bytes ]

Quoted 1 line

Page allocation failures : 0

Looks a lot better!

One thing I need to tell you is that the numbers are not pages but requests.

Quoted 10 lines · Namhyung Kim

Quoted 10 lines

I.e. something like this (mockup) output:

   SUMMARY (page allocator)
   ========================

   Pages allocated+freed:       12,593   [     51,630,080 bytes ]

   Pages allocated-only:         2,342   [      1,235,010 bytes ]
   Pages freed-only:                67   [        135,311 bytes ]

Quoted 6 lines

Quoted 1 line

Page allocation failures : 0

Looks a lot better!

One thing I need to tell you is that the numbers are not pages but requests.

Yes, but in the MM code we tend to call larger order allocations 'pages' as well: higher order pages, such as a 2MB hugepage, or a 8K order-1 page. So at least in MM-speak it should be OK to call them 'pages'.

But your call!

Hi Ingo,

Quoted 20 lines · Ingo Molnar

Quoted 10 lines · Namhyung Kim

Quoted 10 lines

I.e. something like this (mockup) output:

   SUMMARY (page allocator)
   ========================

   Pages allocated+freed:       12,593   [     51,630,080 bytes ]

   Pages allocated-only:         2,342   [      1,235,010 bytes ]
   Pages freed-only:                67   [        135,311 bytes ]

Quoted 6 lines

Quoted 1 line

Page allocation failures : 0

Looks a lot better!

One thing I need to tell you is that the numbers are not pages but requests.

Quoted 6 lines

Yes, but in the MM code we tend to call larger order allocations 'pages' as well: higher order pages, such as a 2MB hugepage, or a 8K order-1 page. So at least in MM-speak it should be OK to call them 'pages'.

But your call!

How about this?

SUMMARY (page allocator)
========================

  Total allocation requests   :        9,015  [     37,200 Kbytes ]  (A)
  Total free requests         :        8,093  [     33,176 Kbytes ]  (B)

  Total alloc+freed requests  :        7,985  [     32,732 Kbytes ]  (C)
  Total alloc-only requests   :        1,030  [      4,468 Kbytes ]  (D)
  Total free-only requests    :          108  [        444 Kbytes ]  (E)

Total allocation failure : 0 [ 0 Kbytes ]

Quoted 37 lines · Namhyung Kim

Hi Ingo,

Quoted 20 lines · Ingo Molnar

Quoted 10 lines · Namhyung Kim

Quoted 10 lines

I.e. something like this (mockup) output:

   SUMMARY (page allocator)
   ========================

   Pages allocated+freed:       12,593   [     51,630,080 bytes ]

   Pages allocated-only:         2,342   [      1,235,010 bytes ]
   Pages freed-only:                67   [        135,311 bytes ]

Quoted 6 lines

Quoted 1 line

Page allocation failures : 0

Looks a lot better!

One thing I need to tell you is that the numbers are not pages but requests.

Quoted 6 lines

Yes, but in the MM code we tend to call larger order allocations 'pages' as well: higher order pages, such as a 2MB hugepage, or a 8K order-1 page. So at least in MM-speak it should be OK to call them 'pages'.

But your call!

How about this?

SUMMARY (page allocator)
========================

  Total allocation requests   :        9,015  [     37,200 Kbytes ]  (A)
  Total free requests         :        8,093  [     33,176 Kbytes ]  (B)

Quoted 5 lines

  Total alloc+freed requests  :        7,985  [     32,732 Kbytes ]  (C)
  Total alloc-only requests   :        1,030  [      4,468 Kbytes ]  (D)
  Total free-only requests    :          108  [        444 Kbytes ]  (E)

Total allocation failure : 0 [ 0 Kbytes ]

s/failure/failures
s/Kbytes/KB

I'd leave a bit more space for the numbers, for up into billions of requests and terabytes of data. Other than that, sounds good to me!

Quoted 3 lines · Ingo Molnar

So there's one thing that would be useful: to track pages allocated on one node, but freed on another. Those kinds of allocation/free patterns are especially expensive and might make sense to visualize.

I think it can be done easily as slab analysis already contains the info.

Quoted 6 lines · Namhyung Kim

Quoted 3 lines · Ingo Molnar

So there's one thing that would be useful: to track pages allocated on one node, but freed on another. Those kinds of allocation/free patterns are especially expensive and might make sense to visualize.

I think it can be done easily as slab analysis already contains the info.

Hmm.. it seems slab events provide the node info but page events don't. Without it, I don't know which node a page is in so cannot determine such cross-node alloc+free patterns.

Quoted 6 lines · Namhyung Kim

Quoted 5 lines · Namhyung Kim

Quoted 3 lines · Ingo Molnar

So there's one thing that would be useful: to track pages allocated on one node, but freed on another. Those kinds of allocation/free patterns are especially expensive and might make sense to visualize.

Quoted 5 lines

Quoted 1 line

I think it can be done easily as slab analysis already contains the info.

Hmm.. it seems slab events provide the node info but page events don't. Without it, I don't know which node a page is in so cannot determine such cross-node alloc+free patterns.

Well, but we know the CPU that the allocation occured on, and can map the CPU to the node.

Hi Ingo,

Quoted 15 lines · Ingo Molnar

Quoted 11 lines · Namhyung Kim

Quoted 6 lines · Namhyung Kim

Quoted 3 lines · Ingo Molnar

So there's one thing that would be useful: to track pages allocated on one node, but freed on another. Those kinds of allocation/free patterns are especially expensive and might make sense to visualize.

I think it can be done easily as slab analysis already contains the info.

Hmm.. it seems slab events provide the node info but page events don't. Without it, I don't know which node a page is in so cannot determine such cross-node alloc+free patterns.

Quoted 2 lines

Well, but we know the CPU that the allocation occured on, and can map the CPU to the node.

Does it gaurantee that the page always belongs to a same node allocating cpu is in? I can simply map the page with an node using cpu info but was not sure whether that's always correct.

Quoted 12 lines

Hello,

Currently perf kmem command only analyzes SLAB memory allocation. And I'd like to introduce page allocation analysis also. Users can use
--slab and/or --page option to select it. If none of these options
are used, it does slab allocation analysis for backward compatibility.

The patch 1-3 are bugfix and cleanups. Patch 4 implements basic support for page allocation analysis, patch 5 deals with the callsite and finally patch 6 implements sorting.

Quoted 5 lines

In this patchset, I used two kmem events: kmem:mm_page_alloc and kmem_page_free for analysis as they can track every memory allocation/free path AFAIK. However, unlike slab tracepoint events, those page allocation events don't provide callsite info directly. So I recorded callchains and extracted callsites like below:

Really cool features!

I have a couple of output typography observations:

Quoted 50 lines

Normal page allocation callchains look like this:

360a7e __alloc_pages_nodemask
3a711c alloc_pages_current
357bc7 __page_cache_alloc <-- callsite
357cf6 pagecache_get_page
48b0a prepare_pages
494d3 __btrfs_buffered_write
49cdf btrfs_file_write_iter
3ceb6e new_sync_write
3cf447 vfs_write
3cff99 sys_write
7556e9 system_call

    f880 __write_nocancel

33eb9 cmd_record
4b38e cmd_kmem
7aa23 run_builtin
27a9a main
20800 __libc_start_main

But first two are internal page allocation functions so it should be skipped. To determine such allocation functions, I used following regex:

^_?_?(alloc|get_free|get_zeroed)_pages?

This gave me a following list of functions (you can see this with -v):

alloc func: __get_free_pages
alloc func: get_zeroed_page
alloc func: alloc_pages_exact
alloc func: __alloc_pages_direct_compact
alloc func: __alloc_pages_nodemask
alloc func: alloc_page_interleave
alloc func: alloc_pages_current
alloc func: alloc_pages_vma
alloc func: alloc_page_buffers
alloc func: alloc_pages_exact_nid

After skipping those function, it got '__page_cache_alloc'.

Other information such as allocation order, migration type and gfp flags are provided by tracepoint events.

Basically the output will be sorted by total allocation bytes, but you can change it by using -s/--sort option. The following sort keys are added to support page analysis: page, order, mtype, gfp. Existing 'callsite', 'bytes' and 'hit' sort keys also can be used.

An example follows:

Quoted 8 lines

  # perf kmem record --slab --page sleep 1
  [ perf record: Woken up 0 times to write data ]
  [ perf record: Captured and wrote 49.277 MB perf.data (191027 samples) ]

  # perf kmem stat --page --caller -l 10 -s order,hit

--------------------------------------------------------------------------------------------
Total_alloc/Per | Hit | Order | Migrate type | GFP flag | Callsite

s/Per/Size
s/Hit/Hits
s/Migrate type/Migration type
s/GFP flag/GFP flags

?

Quoted 14 lines

--------------------------------------------------------------------------------------------

       65536/16384 |        4 |     2 |  RECLAIMABLE | 00285250 | new_slab
    51347456/4096  |    12536 |     0 |      MOVABLE | 0102005a | __page_cache_alloc
       53248/4096  |       13 |     0 |    UNMOVABLE | 002084d0 | pte_alloc_one
       40960/4096  |       10 |     0 |      MOVABLE | 000280da | handle_mm_fault
       28672/4096  |        7 |     0 |    UNMOVABLE | 000000d0 | __pollwait
       20480/4096  |        5 |     0 |      MOVABLE | 000200da | do_wp_page
       20480/4096  |        5 |     0 |      MOVABLE | 000200da | do_cow_fault
       16384/4096  |        4 |     0 |    UNMOVABLE | 00000200 | __tlb_remove_page
       16384/4096  |        4 |     0 |    UNMOVABLE | 000084d0 | __pmd_alloc
        8192/4096  |        2 |     0 |    UNMOVABLE | 000084d0 | __pud_alloc
   ...             | ...      | ...   | ...          | ...      | ...

--------------------------------------------------------------------------------------------

Quoted 8 lines

SUMMARY (page allocator)
========================
Total alloc requested: 12593
Total alloc failure : 0
Total bytes allocated: 51630080
Total free requested: 115
Total free unmatched: 67
Total bytes freed : 471040

I'd suggest the following changes to the format:

- Collapse stats into 3 groups: 'allocated+freed', 'allocated only', 'freed only', depending on how much of their lifetime we've managed to trace. These groups are really distinct and it makes little sense to mix up their stats.

- Add commas to the numbers, to make it easier to read and compare larger numbers.

- Right-align the numbers, to make them easy to compare when they are placed under each other.

- Merge the 'count' and 'bytes' stats into a single line, so that it's more compact, easier to navigate, but also only comparable type numbers are placed under each other.

I.e. something like this (mockup) output:

SUMMARY (page allocator)
========================

Pages allocated+freed: 12,593 [ 51,630,080 bytes ]

Pages allocated-only: 2,342 [ 1,235,010 bytes ]
Pages freed-only: 67 [ 135,311 bytes ]

Page allocation failures : 0

Quoted 13 lines

Order UNMOVABLE RECLAIMABLE MOVABLE RESERVED CMA/ISOLATE
----- ------------ ------------ ------------ ------------ ------------

      0            32             0         12557             0             0
      1             0             0             0             0             0
      2             0             4             0             0             0
      3             0             0             0             0             0
      4             0             0             0             0             0
      5             0             0             0             0             0
      6             0             0             0             0             0
      7             0             0             0             0             0
      8             0             0             0             0             0
      9             0             0             0             0             0
     10             0             0             0             0             0

Here I'd suggest the following refinements:

- Use '.' instead of '0', to make actual nonzero values stand out visually, while still keeping a tabular format

- Merge the 'Reserved', 'CMA/Isolate' columns into a single 'Special'
colum: this will be zero in 99.9% of the cases, as those pages mostly deal with driver interfaces, mostly used during init/deinit.

- Capitalize less.

- Use comma-separated numbers for better readability.

So something like this:

Order Unmovable Reclaimable Movable Special
----- ------------ ------------ ------------ ------------

       0            32             .        12,557             .
       1             .             .             .             .
       2             .             4             .             .
       3             .             .             .             .
       4             .             .             .             .
       5             .             .             .             .
       6             .             .             .             .
       7             .             .             .             .
       8             .             .             .             .
       9             .             .             .             .
      10             .             .             .             .

Look for example how easily noticeable the '4' value is now, while it was pretty easy to miss in the original table.

Quoted 2 lines

I have some idea how to improve it. But I'd also like to hear other idea, suggestion, feedback and so on.

So there's one thing that would be useful: to track pages allocated on one node, but freed on another. Those kinds of allocation/free patterns are especially expensive and might make sense to visualize.

Thanks,

Ingo