Skip to contentSkip to conversation

mm: add per-order mTHP alloc_success and alloc_fail counters

PATCH Superseded Barry Song 3 participants 17 messages Apr 3, 2024 – Apr 5, 2024 1 message not in the archive

Subsystems (3)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Barry Song

author
#0 lore raw

PATCH v3 1/1 mm: add per-order mTHP alloc_success and alloc_fail counters

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address David's comment;
* drop a redundant "else", Ryan

 include/linux/huge_mm.h | 18 ++++++++++++++
 mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 3 files changed, 74 insertions(+)
include/linux/huge_mm.h +18 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
  * (which is a limitation of the THP implementation).
  */
 #define THP_ORDERS_ALL_ANON	((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER		2
 
 /*
  * Mask of all large folio orders supported for file THP.
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 					  enforce_sysfs, orders);
 }
 
+enum thp_event_item {
+	THP_ANON_ALLOC_SUCCESS,
+	THP_ANON_ALLOC_FAIL,
+	NR_THP_EVENT_ITEMS
+};
+
+struct thp_event_state {
+	unsigned long event[PMD_ORDER + 1 - THP_MIN_ORDER][NR_THP_EVENT_ITEMS];
+};
+
+DECLARE_PER_CPU(struct thp_event_state, thp_event_states);
+
+static inline void count_thp_event(int order, enum thp_event_item item)
+{
+	this_cpu_inc(thp_event_states.event[order - THP_MIN_ORDER][item]);
+}
+
 #define transparent_hugepage_use_zero_page()				\
 	(transparent_hugepage_flags &					\
 	 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +54 -0

74 lines not on this page · raw message

mm/memory.c +2 -0
index 912cd738ec03..40de2b7a73ab 100644
@@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
 			}
 			folio_throttle_swaprate(folio, gfp);
 			clear_huge_page(&folio->page, vmf->address, 1 << order);
+			count_thp_event(order, THP_ANON_ALLOC_SUCCESS);
 			return folio;
 		}
+		count_thp_event(order, THP_ANON_ALLOC_FAIL);
 next:
 		order = next_order(&orders, order);
 	}
Signature · 1 line
2.34.1
#1 lore raw
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

    David's comment;

* drop a redundant "else", Ryan

  include/linux/huge_mm.h | 18 ++++++++++++++
  mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
  mm/memory.c             |  2 ++
  3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
   * (which is a limitation of the THP implementation).
   */
  #define THP_ORDERS_ALL_ANON	((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER		2
 
  /*
   * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 272

Quoted 8 lines
include/linux/huge_mm.h +5 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
  					  enforce_sysfs, orders);
  }
 
+enum thp_event_item {
+	THP_ANON_ALLOC_SUCCESS,
+	THP_ANON_ALLOC_FAIL,
+	NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

enum thp_event {
	THP_EVENT_ANON_ALLOC_SUCCESS,
	THP_EVENT_ANON_ALLOC_FAIL,
	__THP_EVENT_COUNT,
};
Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#2 lore raw
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters.  Incorporating additional counters should now be straightforward as well.

---
  -v3:
  * save some memory as order-0 and order-1 can't be THP, Ryan;
  * rename to anon_alloc as right now we only support anon to address David's comment;
  * drop a redundant "else", Ryan

  include/linux/huge_mm.h | 18 ++++++++++++++ mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++ mm/memory.c             |  2 ++

  3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
    * (which is a limitation of the THP implementation).
    */
   #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
     /*
    * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                         enforce_sysfs, orders);
   }
   +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines

enum thp_event {
    THP_EVENT_ANON_ALLOC_SUCCESS,
    THP_EVENT_ANON_ALLOC_FAIL,
    __THP_EVENT_COUNT,
};

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

#3 lore raw
Quoted 62 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters.  Incorporating additional counters should now be straightforward as well.

---
  -v3:
  * save some memory as order-0 and order-1 can't be THP, Ryan;
  * rename to anon_alloc as right now we only support anon to address David's comment;
  * drop a redundant "else", Ryan

   include/linux/huge_mm.h | 18 ++++++++++++++
   mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
   mm/memory.c             |  2 ++
   3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
    * (which is a limitation of the THP implementation).
    */
   #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
     /*
    * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                         enforce_sysfs, orders);
   }
   +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines

enum thp_event {
    THP_EVENT_ANON_ALLOC_SUCCESS,
    THP_EVENT_ANON_ALLOC_FAIL,
    __THP_EVENT_COUNT,
};

Quoted 6 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

Fully agree!

Signature · 3 lines
Cheers,

David / dhildenb
author
#4 lore raw
In reply to Ryan Roberts
Quoted 63 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

    David's comment;

* drop a redundant "else", Ryan

  include/linux/huge_mm.h | 18 ++++++++++++++
  mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
  mm/memory.c             |  2 ++
  3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
   * (which is a limitation of the THP implementation).
   */
  #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
    /*
   * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                        enforce_sysfs, orders);
  }
  +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
    THP_EVENT_ANON_ALLOC_SUCCESS,
    THP_EVENT_ANON_ALLOC_FAIL,
    __THP_EVENT_COUNT,
};
Quoted 5 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

enum thp_stat {
THP_EVENT_ANON_ALLOC,
THP_EVENT_ANON_ALLOC_FALLBACK,
THP_EVENT_SWPOUT,
THP_EVENT_SWPOUT_FALLBACK,
...
THP_NR_ANON_PAGES,
THP_NR_FILE_PAGES,
...
__THP_STAT_COUNT,
};

THP_EVENT can only increase; THP_NR_ANON/FILE_PAGES can increase or decrease. Thus, their names have no "EVENT".

Quoted 1 line
reviewer
#5 lore raw
Quoted 72 lines · Barry Song
Quoted 63 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

    David's comment;

* drop a redundant "else", Ryan

  include/linux/huge_mm.h | 18 ++++++++++++++
  mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
  mm/memory.c             |  2 ++
  3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
   * (which is a limitation of the THP implementation).
   */
  #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
    /*
   * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                        enforce_sysfs, orders);
  }
  +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
    THP_EVENT_ANON_ALLOC_SUCCESS,
    THP_EVENT_ANON_ALLOC_FAIL,
    __THP_EVENT_COUNT,
};
Quoted 5 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

Quoted 8 lines
enum thp_stat {
   THP_EVENT_ANON_ALLOC,
   THP_EVENT_ANON_ALLOC_FALLBACK,
   THP_EVENT_SWPOUT,
   THP_EVENT_SWPOUT_FALLBACK,
   ...
   THP_NR_ANON_PAGES,
   THP_NR_FILE_PAGES,

I find this ambiguous; Is it the number of THPs or the number of base pages?

I think David made the point about incorporating the enum name into the labels too, so that there can be no namespace confusion. How about:

<enum>_<type>_<name>

So:

THP_STAT_EV_ANON_ALLOC
THP_STAT_EV_ANON_ALLOC_FALLBACK
THP_STAT_EV_ANON_PARTIAL
THP_STAT_EV_SWPOUT
THP_STAT_EV_SWPOUT_FALLBACK
...
THP_STAT_NR_ANON
THP_STAT_NR_FILE
...
__THP_STAT_COUNT

I'm not sure if we need to prefix SWPOUT with ANON? (might we additionally want SHMEM in future?

I'm aware I'm bikeshedding at this point... sorry :)

Quoted 8 lines
   ...
   __THP_STAT_COUNT,
};

THP_EVENT can only increase; THP_NR_ANON/FILE_PAGES can increase or decrease. Thus, their names have no "EVENT".

Quoted 1 line
author
#6 lore raw
Quoted 83 lines · Ryan Roberts
Quoted 72 lines · Barry Song
Quoted 63 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

    David's comment;

* drop a redundant "else", Ryan

  include/linux/huge_mm.h | 18 ++++++++++++++
  mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
  mm/memory.c             |  2 ++
  3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
   * (which is a limitation of the THP implementation).
   */
  #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
    /*
   * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                        enforce_sysfs, orders);
  }
  +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
    THP_EVENT_ANON_ALLOC_SUCCESS,
    THP_EVENT_ANON_ALLOC_FAIL,
    __THP_EVENT_COUNT,
};
Quoted 5 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

Quoted 8 lines
enum thp_stat {
   THP_EVENT_ANON_ALLOC,
   THP_EVENT_ANON_ALLOC_FALLBACK,
   THP_EVENT_SWPOUT,
   THP_EVENT_SWPOUT_FALLBACK,
   ...
   THP_NR_ANON_PAGES,
   THP_NR_FILE_PAGES,
Quoted 1 line

I find this ambiguous; Is it the number of THPs or the number of base pages?

Indeed. I've noticed that the kernel seems rather unpredictable in terms of both the number of THPs and the number of base pages.

for example:

__node_stat_mod_folio(folio, NR_FILE_PAGES, nr);
__node_stat_mod_folio(folio, NR_FILE_PAGES, -nr);

add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);

__lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr);
__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr);

The previous statements pertain to base pages encountered when dealing with a large folio.;

but the below means THP number:
__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped);

My feeling is that when assessing the true memory size, base pages are the primary
consideration.
However, I haven't addressed this aspect yet, and we can defer this discussion until
I delve into it further upon arrival :-)

Currently, I'm exclusively employing it as a demonstration to differentiate between
"event" and "stat", aiming to provide a glimpse into the future perspective.

Quoted 19 lines

I think David made the point about incorporating the enum name into the labels too, so that there can be no namespace confusion. How about:

<enum>_<type>_<name>

So:

THP_STAT_EV_ANON_ALLOC
THP_STAT_EV_ANON_ALLOC_FALLBACK
THP_STAT_EV_ANON_PARTIAL
THP_STAT_EV_SWPOUT
THP_STAT_EV_SWPOUT_FALLBACK
...
THP_STAT_NR_ANON
THP_STAT_NR_FILE
...
__THP_STAT_COUNT

Quoted 4 lines

I'm not sure if we need to prefix SWPOUT with ANON? (might we additionally want SHMEM in future?

I'm aware I'm bikeshedding at this point... sorry :)

No problem at all. I'm open to discussing it further so that I can better prepare for version 4.

Quoted 9 lines
Quoted 9 lines

   ...
   __THP_STAT_COUNT,
};

THP_EVENT can only increase; THP_NR_ANON/FILE_PAGES can increase or decrease. Thus, their names have no "EVENT".

Thanks
Barry

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#8 lore raw
Quoted 70 lines · David Hildenbrand
Quoted 68 lines · Ryan Roberts
Quoted 67 lines · Barry Song
Quoted 66 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

     David's comment;

* drop a redundant "else", Ryan

   include/linux/huge_mm.h | 18 ++++++++++++++
   mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
   mm/memory.c             |  2 ++
   3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
    * (which is a limitation of the THP implementation).
    */
   #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
     /*
    * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                         enforce_sysfs, orders);
   }
   +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
     THP_EVENT_ANON_ALLOC_SUCCESS,
     THP_EVENT_ANON_ALLOC_FAIL,
     __THP_EVENT_COUNT,
};

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

And most importantly, the below looks much better

       enum thp_stat_item {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };

       DECLARE_PER_CPU(struct thp_state, thp_states);

than

       enum thp_stat {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };
Quoted 28 lines
Quoted 17 lines
Quoted 7 lines
    THP_EVENT_ANON_ALLOC,
    THP_EVENT_ANON_ALLOC_FALLBACK,
    THP_EVENT_SWPOUT,
    THP_EVENT_SWPOUT_FALLBACK,
    ...
    THP_NR_ANON_PAGES,
    THP_NR_FILE_PAGES,

I find this ambiguous; Is it the number of THPs or the number of base pages?

I think David made the point about incorporating the enum name into the labels too, so that there can be no namespace confusion. How about:

<enum>_<type>_<name>

So:

Quoted 10 lines

THP_STAT_EV_ANON_ALLOC
THP_STAT_EV_ANON_ALLOC_FALLBACK
THP_STAT_EV_ANON_PARTIAL
THP_STAT_EV_SWPOUT
THP_STAT_EV_SWPOUT_FALLBACK
...
THP_STAT_NR_ANON
THP_STAT_NR_FILE
...
__THP_STAT_COUNT

Quoted 1 line

I'd even drop the "EV". "NR_ANON" vs "ANON_ALLOC" etc. is expressive enough.

ok.

Quoted 5 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#9 lore raw
Quoted 139 lines · Barry Song
Quoted 70 lines · David Hildenbrand
Quoted 68 lines · Ryan Roberts
Quoted 67 lines · Barry Song
Quoted 66 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

     David's comment;

* drop a redundant "else", Ryan

   include/linux/huge_mm.h | 18 ++++++++++++++
   mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
   mm/memory.c             |  2 ++
   3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
    * (which is a limitation of the THP implementation).
    */
   #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
     /*
    * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                         enforce_sysfs, orders);
   }
   +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
     THP_EVENT_ANON_ALLOC_SUCCESS,
     THP_EVENT_ANON_ALLOC_FAIL,
     __THP_EVENT_COUNT,
};

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

And most importantly, the below looks much better

       enum thp_stat_item {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };

       DECLARE_PER_CPU(struct thp_state, thp_states);

than

       enum thp_stat {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };
Quoted 28 lines
Quoted 27 lines
Quoted 7 lines
    THP_EVENT_ANON_ALLOC,
    THP_EVENT_ANON_ALLOC_FALLBACK,
    THP_EVENT_SWPOUT,
    THP_EVENT_SWPOUT_FALLBACK,
    ...
    THP_NR_ANON_PAGES,
    THP_NR_FILE_PAGES,

I find this ambiguous; Is it the number of THPs or the number of base pages?

I think David made the point about incorporating the enum name into the labels too, so that there can be no namespace confusion. How about:

<enum>_<type>_<name>

So:

THP_STAT_EV_ANON_ALLOC
THP_STAT_EV_ANON_ALLOC_FALLBACK
THP_STAT_EV_ANON_PARTIAL
THP_STAT_EV_SWPOUT
THP_STAT_EV_SWPOUT_FALLBACK
...
THP_STAT_NR_ANON
THP_STAT_NR_FILE
...
__THP_STAT_COUNT

Quoted 3 lines
Quoted 1 line

I'd even drop the "EV". "NR_ANON" vs "ANON_ALLOC" etc. is expressive enough.

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct

vm_area_struct *vma,

                                          enforce_sysfs, orders);

}

+enum thp_stat_item {

+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT

+};
+
+struct thp_state {
+ unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
+};
+
+DECLARE_PER_CPU(struct thp_state, thp_states);
+
+static inline void count_thp_state(int order, enum thp_stat_item item) +{
+ this_cpu_inc(thp_states.state[order][item]);
+}
+

 #define transparent_hugepage_use_zero_page()                           \
        (transparent_hugepage_flags &                                   \
         (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +28 -0
index 9d4b2fbf6872..e704b4408181 100644
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
        .sysfs_ops = &kobj_sysfs_ops,
 };
 
+DEFINE_PER_CPU(struct thp_state, thp_states) = {{{0}}};
+
+static unsigned long sum_thp_states(int order, enum thp_stat_item item)
+{
+       unsigned long sum = 0;
+       int cpu;
+
+       for_each_online_cpu(cpu) {
+               struct thp_state *this = &per_cpu(thp_states, cpu);
+
+               sum += this->state[order][item];
+       }
+
+       return sum;
+}
+
+#define THP_STATE_ATTR(_name, _index)                                  \
+static ssize_t _name##_show(struct kobject *kobj,                      \
+                       struct kobj_attribute *attr, char *buf)         \
+{                                                                      \
+       int order = to_thpsize(kobj)->order;                            \
+                                                                       \
+       return sysfs_emit(buf, "%lu\n", sum_thp_states(order, _index)); \
+}                                                                      \
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
+
+THP_STATE_ATTR(anon_alloc, THP_STAT_ANON_ALLOC);
+THP_STATE_ATTR(anon_alloc_fallback, THP_STAT_ANON_ALLOC_FALLBACK);
 
Quoted 6 lines
Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry

#10 lore raw
Quoted 78 lines · Barry Song
Quoted 77 lines · Barry Song
Quoted 75 lines · David Hildenbrand
Quoted 73 lines · Ryan Roberts
Quoted 72 lines · Barry Song
Quoted 63 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address David's comment;
* drop a redundant "else", Ryan

include/linux/huge_mm.h | 18 ++++++++++++++
mm/huge_memory.c | 54 +++++++++++++++++++++++++++++++++++++++++
mm/memory.c | 2 ++
3 files changed, 74 insertions(+)

include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
     * (which is a limitation of the THP implementation).
     */
    #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
      /*
     * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                          enforce_sysfs, orders);
    }
    +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
      THP_EVENT_ANON_ALLOC_SUCCESS,
      THP_EVENT_ANON_ALLOC_FAIL,
      __THP_EVENT_COUNT,
};
Quoted 5 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

Right, we can do that.

[...]

Quoted 12 lines
Quoted 1 line

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 273

Quoted 10 lines
include/linux/huge_mm.h +6 -0
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                           enforce_sysfs, orders);
  }
 
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+

Final bikeshedding: so we want to call all of that MTHP_, mthp_ etc, to make it clearer that this is per mTHP size?

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#11 lore raw
Quoted 111 lines · David Hildenbrand
Quoted 81 lines · Barry Song
Quoted 80 lines · Barry Song
Quoted 78 lines · David Hildenbrand
Quoted 76 lines · Ryan Roberts
Quoted 75 lines · Barry Song
Quoted 64 lines · Ryan Roberts
Quoted 56 lines · David Hildenbrand
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters.  Incorporating additional counters should now be straightforward as well.

---
    -v3:
    * save some memory as order-0 and order-1 can't be THP, Ryan;
    * rename to anon_alloc as right now we only support anon to address David's comment;
    * drop a redundant "else", Ryan

    include/linux/huge_mm.h | 18 ++++++++++++++ mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++ mm/memory.c             |  2 ++

    3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
      * (which is a limitation of the THP implementation).
      */
     #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0)
 | BIT(1)))
+#define THP_MIN_ORDER        2
       /*
      * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                           enforce_sysfs, orders);
     }
     +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines

enum thp_event {
      THP_EVENT_ANON_ALLOC_SUCCESS,
      THP_EVENT_ANON_ALLOC_FAIL,
      __THP_EVENT_COUNT,
};

Quoted 7 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec
stats that will be added in future.

What about the below?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

Right, we can do that.

[...]

Quoted 16 lines
Quoted 1 line

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                            enforce_sysfs, orders);
   }
 
Quoted 10 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 273

Quoted 6 lines
include/linux/huge_mm.h +6 -0
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+

Final bikeshedding: so we want to call all of that MTHP_, mthp_ etc, to make it clearer that this is per mTHP size?

Perviously we concluded that mTHP was only for commit logs, and in the longer term we wanted THP to mean the full set of sizes like it does for hugetlb. Neither "mTHP" nor "multi-size THP" currently exist in the code base.

Personally I'd vote for leaving it as THP.

#12 lore raw
Quoted 123 lines · Ryan Roberts
Quoted 111 lines · David Hildenbrand
Quoted 81 lines · Barry Song
Quoted 80 lines · Barry Song
Quoted 78 lines · David Hildenbrand
Quoted 76 lines · Ryan Roberts
Quoted 75 lines · Barry Song
Quoted 64 lines · Ryan Roberts
Quoted 56 lines · David Hildenbrand
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters.  Incorporating additional counters should now be straightforward as well.

---
    -v3:
    * save some memory as order-0 and order-1 can't be THP, Ryan;
    * rename to anon_alloc as right now we only support anon to address David's comment;
    * drop a redundant "else", Ryan

     include/linux/huge_mm.h | 18 ++++++++++++++
     mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
     mm/memory.c             |  2 ++
     3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
      * (which is a limitation of the THP implementation).
      */
     #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0)
 | BIT(1)))
+#define THP_MIN_ORDER        2
       /*
      * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                           enforce_sysfs, orders);
     }
     +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines

enum thp_event {
      THP_EVENT_ANON_ALLOC_SUCCESS,
      THP_EVENT_ANON_ALLOC_FAIL,
      __THP_EVENT_COUNT,
};

Quoted 7 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec
stats that will be added in future.

What about the below?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

Right, we can do that.

[...]

Quoted 16 lines
Quoted 1 line

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                            enforce_sysfs, orders);
   }
 
Quoted 10 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 273

Quoted 6 lines
include/linux/huge_mm.h +6 -0
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+

Final bikeshedding: so we want to call all of that MTHP_, mthp_ etc, to make it clearer that this is per mTHP size?

Quoted 3 lines

Perviously we concluded that mTHP was only for commit logs, and in the longer term we wanted THP to mean the full set of sizes like it does for hugetlb. Neither "mTHP" nor "multi-size THP" currently exist in the code base.

True, but some indication that these are "per THP size" might be reasonable.

Signature · 3 lines
Cheers,

David / dhildenb
author
#13 lore raw
Quoted 125 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 121 lines · David Hildenbrand
Quoted 81 lines · Barry Song
Quoted 80 lines · Barry Song
Quoted 78 lines · David Hildenbrand
Quoted 76 lines · Ryan Roberts
Quoted 75 lines · Barry Song
Quoted 64 lines · Ryan Roberts
Quoted 56 lines · David Hildenbrand
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address David's comment;
* drop a redundant "else", Ryan

include/linux/huge_mm.h | 18 ++++++++++++++
mm/huge_memory.c | 54 +++++++++++++++++++++++++++++++++++++++++
mm/memory.c | 2 ++
3 files changed, 74 insertions(+)

include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
      * (which is a limitation of the THP implementation).
      */
     #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0)
 | BIT(1)))
+#define THP_MIN_ORDER        2
       /*
      * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                           enforce_sysfs, orders);
     }
     +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
       THP_EVENT_ANON_ALLOC_SUCCESS,
       THP_EVENT_ANON_ALLOC_FAIL,
       __THP_EVENT_COUNT,
};
Quoted 7 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec
stats that will be added in future.

What about the below?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

Right, we can do that.

[...]

Quoted 12 lines
Quoted 1 line

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 273

Quoted 10 lines
include/linux/huge_mm.h +6 -0
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                            enforce_sysfs, orders);
   }
 
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+

Final bikeshedding: so we want to call all of that MTHP_, mthp_ etc, to make it clearer that this is per mTHP size?

Quoted 5 lines
Quoted 3 lines

Perviously we concluded that mTHP was only for commit logs, and in the longer term we wanted THP to mean the full set of sizes like it does for hugetlb. Neither "mTHP" nor "multi-size THP" currently exist in the code base.

True, but some indication that these are "per THP size" might be reasonable.

It seems quite reasonable, especially considering the existence of THP counters. Additionally, there's even an overlap between THP counters and mTHP counters, specifically for PMD_ORDER. This complexity could potentially confuse readers of the code.

I'm perfectly fine with using "M/m" as a prefix, as long as sysfs hasn't exposed it.
Moreover, "M/m" remains a preferable prefix given that "PER_SIZE" is overly long,
and people are already familiar with the concept of mTHP.

Quoted 6 lines

--
Cheers,

David / dhildenb

#14 lore raw
Quoted 137 lines · Barry Song
Quoted 125 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 121 lines · David Hildenbrand
Quoted 81 lines · Barry Song
Quoted 80 lines · Barry Song
Quoted 78 lines · David Hildenbrand
Quoted 76 lines · Ryan Roberts
Quoted 75 lines · Barry Song
Quoted 64 lines · Ryan Roberts
Quoted 56 lines · David Hildenbrand
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address David's comment;
* drop a redundant "else", Ryan

include/linux/huge_mm.h | 18 ++++++++++++++
mm/huge_memory.c | 54 +++++++++++++++++++++++++++++++++++++++++
mm/memory.c | 2 ++
3 files changed, 74 insertions(+)

include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
       * (which is a limitation of the THP implementation).
       */
      #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0)
 | BIT(1)))
+#define THP_MIN_ORDER        2
        /*
       * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                            enforce_sysfs, orders);
      }
      +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
        THP_EVENT_ANON_ALLOC_SUCCESS,
        THP_EVENT_ANON_ALLOC_FAIL,
        __THP_EVENT_COUNT,
};
Quoted 7 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec
stats that will be added in future.

What about the below?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

Right, we can do that.

[...]

Quoted 12 lines
Quoted 1 line

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 273

Quoted 10 lines
include/linux/huge_mm.h +6 -0
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                             enforce_sysfs, orders);
    }
 
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+

Final bikeshedding: so we want to call all of that MTHP_, mthp_ etc, to make it clearer that this is per mTHP size?

Quoted 5 lines
Quoted 3 lines

Perviously we concluded that mTHP was only for commit logs, and in the longer term we wanted THP to mean the full set of sizes like it does for hugetlb. Neither "mTHP" nor "multi-size THP" currently exist in the code base.

True, but some indication that these are "per THP size" might be reasonable.

It seems quite reasonable, especially considering the existence of THP counters. Additionally, there's even an overlap between THP counters and mTHP counters, specifically for PMD_ORDER. This complexity could potentially confuse readers of the code.

Quoted 5 lines

I'm perfectly fine with using "M/m" as a prefix, as long as sysfs hasn't exposed it.
Moreover, "M/m" remains a preferable prefix given that "PER_SIZE" is overly long,
and people are already familiar with the concept of mTHP.

I think the important part is to not use "mTHP" in any of our ABI. As we expose these stats per THP size, using mTHP internally should be ok. But I'm happy to hear alternatives that express "per THP size" in a better way.

Signature · 3 lines
Cheers,

David / dhildenb
author
#15 lore raw
Quoted 145 lines · David Hildenbrand
Quoted 137 lines · Barry Song
Quoted 125 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 121 lines · David Hildenbrand
Quoted 81 lines · Barry Song
Quoted 80 lines · Barry Song
Quoted 78 lines · David Hildenbrand
Quoted 76 lines · Ryan Roberts
Quoted 75 lines · Barry Song
Quoted 64 lines · Ryan Roberts
Quoted 56 lines · David Hildenbrand
Quoted 51 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due
to the lack of visibility into its operations. Presenting the
success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging
performance improvements and regressions without these figures.
It's crucial for us to understand the true effectiveness of
mTHP in real-world scenarios, especially in systems with
fragmented memory.

This patch sets up the framework for per-order mTHP counters,
starting with the introduction of anon_alloc_success and
anon_alloc_fail counters. Incorporating additional counters
should now be straightforward as well.

Signed-off-by: Barry Song <[email protected]>
---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address
David's comment;
* drop a redundant "else", Ryan

include/linux/huge_mm.h | 18 ++++++++++++++
mm/huge_memory.c | 54 +++++++++++++++++++++++++++++++++++++++++
mm/memory.c | 2 ++
3 files changed, 74 insertions(+)

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index e896ca4760f6..5e9af6be9537 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
* (which is a limitation of the THP implementation).
*/
#define THP_ORDERS_ALL_ANON ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0)
| BIT(1)))
+#define THP_MIN_ORDER 2
/*
* Mask of all large folio orders supported for file THP.
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
vm_area_struct *vma,
enforce_sysfs, orders);
}
+enum thp_event_item {
+ THP_ANON_ALLOC_SUCCESS,
+ THP_ANON_ALLOC_FAIL,
+ NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
        THP_EVENT_ANON_ALLOC_SUCCESS,
        THP_EVENT_ANON_ALLOC_FAIL,
        __THP_EVENT_COUNT,
};
Quoted 7 lines

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

You are only adding events for now, but we have identified a need for inc/dec
stats that will be added in future.

What about the below?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

Right, we can do that.

[...]

Quoted 12 lines
Quoted 1 line

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 273

Quoted 10 lines
include/linux/huge_mm.h +6 -0
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                             enforce_sysfs, orders);
    }
 
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+

Final bikeshedding: so we want to call all of that MTHP_, mthp_ etc, to make it clearer that this is per mTHP size?

Quoted 5 lines
Quoted 3 lines

Perviously we concluded that mTHP was only for commit logs, and in the longer term we wanted THP to mean the full set of sizes like it does for hugetlb. Neither "mTHP" nor "multi-size THP" currently exist in the code base.

True, but some indication that these are "per THP size" might be reasonable.

It seems quite reasonable, especially considering the existence of THP counters. Additionally, there's even an overlap between THP counters and mTHP counters, specifically for PMD_ORDER. This complexity could potentially confuse readers of the code.

Quoted 5 lines

I'm perfectly fine with using "M/m" as a prefix, as long as sysfs hasn't exposed it.
Moreover, "M/m" remains a preferable prefix given that "PER_SIZE" is overly long,
and people are already familiar with the concept of mTHP.

Quoted 3 lines

I think the important part is to not use "mTHP" in any of our ABI. As we expose these stats per THP size, using mTHP internally should be ok. But I'm happy to hear alternatives that express "per THP size" in a better way.

Indeed, as long as we maintain stability in the node names, the designation of "mTHP" remains merely an implementation detail. Therefore, in version 4, I've opted to utilize "mthp" for these macros and variables. Once we discover a more suitable approach, we can proceed with additional renaming as necessary :-)

At present, we've implemented mthp allocation within PF and are executing non-split swapout. It's crucial now for us to assess their effectiveness and operational scope.

Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry

reviewer
#16 lore raw
In reply to Barry Song
Quoted 140 lines · Barry Song
Quoted 139 lines · Barry Song
Quoted 70 lines · David Hildenbrand
Quoted 68 lines · Ryan Roberts
Quoted 67 lines · Barry Song
Quoted 66 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

     David's comment;

* drop a redundant "else", Ryan

   include/linux/huge_mm.h | 18 ++++++++++++++
   mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
   mm/memory.c             |  2 ++
   3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
    * (which is a limitation of the THP implementation).
    */
   #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
     /*
    * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                         enforce_sysfs, orders);
   }
   +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
     THP_EVENT_ANON_ALLOC_SUCCESS,
     THP_EVENT_ANON_ALLOC_FAIL,
     __THP_EVENT_COUNT,
};

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

And most importantly, the below looks much better

       enum thp_stat_item {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };

       DECLARE_PER_CPU(struct thp_state, thp_states);

than

       enum thp_stat {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };
Quoted 28 lines
Quoted 27 lines
Quoted 7 lines
    THP_EVENT_ANON_ALLOC,
    THP_EVENT_ANON_ALLOC_FALLBACK,
    THP_EVENT_SWPOUT,
    THP_EVENT_SWPOUT_FALLBACK,
    ...
    THP_NR_ANON_PAGES,
    THP_NR_FILE_PAGES,

I find this ambiguous; Is it the number of THPs or the number of base pages?

I think David made the point about incorporating the enum name into the labels too, so that there can be no namespace confusion. How about:

<enum>_<type>_<name>

So:

THP_STAT_EV_ANON_ALLOC
THP_STAT_EV_ANON_ALLOC_FALLBACK
THP_STAT_EV_ANON_PARTIAL
THP_STAT_EV_SWPOUT
THP_STAT_EV_SWPOUT_FALLBACK
...
THP_STAT_NR_ANON
THP_STAT_NR_FILE
...
__THP_STAT_COUNT

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I'd even drop the "EV". "NR_ANON" vs "ANON_ALLOC" etc. is expressive enough.

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

Please treat all my comments below as optional - I'm just stating my preference!

Quoted 8 lines
include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                          enforce_sysfs, orders);

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 275

Quoted 10 lines
include/linux/huge_mm.h +8 -0
 }
 
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+
+struct thp_state {
+       unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];

Why using "state" here? I think "stats" is more appropriate? (as in short for "statistics") i.e. struct thp_stats and unsigned long stats[][].

Quoted 17 lines
include/linux/huge_mm.h +9 -0
+};
+
+DECLARE_PER_CPU(struct thp_state, thp_states);
+
+static inline void count_thp_state(int order, enum thp_stat_item item)
+{
+       this_cpu_inc(thp_states.state[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()                           \
        (transparent_hugepage_flags &                                   \

(1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))

mm/huge_memory.c +0 -0
index 9d4b2fbf6872..e704b4408181 100644

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 529

Quoted 4 lines
mm/huge_memory.c +1 -0
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
        .sysfs_ops = &kobj_sysfs_ops,
 };
 
+DEFINE_PER_CPU(struct thp_state, thp_states) = {{{0}}};

Does this need to be explicitly zeroed? Won't that be the default initial state, just like for regular globals? Perhaps PER_CPU is special?

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 531

Quoted 2 lines
mm/huge_memory.c +2 -0
+
+static unsigned long sum_thp_states(int order, enum thp_stat_item item)

Again, I'd call it sum_thp_stats().

Quoted 4 lines
mm/huge_memory.c +4 -0
+{
+       unsigned long sum = 0;
+       int cpu;
+

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 545

Quoted 10 lines
mm/huge_memory.c +10 -0
+       for_each_online_cpu(cpu) {
+               struct thp_state *this = &per_cpu(thp_states, cpu);
+
+               sum += this->state[order][item];
+       }
+
+       return sum;
+}
+
+#define THP_STATE_ATTR(_name, _index)                                  \

And THP_STATS_ATTR(); they are going to live in the "stats" directory after all.

Quoted 21 lines
mm/huge_memory.c +11 -0
+static ssize_t _name##_show(struct kobject *kobj,                      \
+                       struct kobj_attribute *attr, char *buf)         \
+{                                                                      \
+       int order = to_thpsize(kobj)->order;                            \
+                                                                       \
+       return sysfs_emit(buf, "%lu\n", sum_thp_states(order, _index)); \
+}                                                                      \
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
+
+THP_STATE_ATTR(anon_alloc, THP_STAT_ANON_ALLOC);
+THP_STATE_ATTR(anon_alloc_fallback, THP_STAT_ANON_ALLOC_FALLBACK);
 
Quoted 6 lines
Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#17 lore raw
Quoted 163 lines · Ryan Roberts
Quoted 140 lines · Barry Song
Quoted 139 lines · Barry Song
Quoted 70 lines · David Hildenbrand
Quoted 68 lines · Ryan Roberts
Quoted 67 lines · Barry Song
Quoted 66 lines · Ryan Roberts
Quoted 55 lines · David Hildenbrand
Quoted 42 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well.

---
-v3:
* save some memory as order-0 and order-1 can't be THP, Ryan;
* rename to anon_alloc as right now we only support anon to address

     David's comment;

* drop a redundant "else", Ryan

   include/linux/huge_mm.h | 18 ++++++++++++++
   mm/huge_memory.c        | 54 +++++++++++++++++++++++++++++++++++++++++
   mm/memory.c             |  2 ++
   3 files changed, 74 insertions(+)
include/linux/huge_mm.h +1 -0
index e896ca4760f6..5e9af6be9537 100644
@@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr;
    * (which is a limitation of the THP implementation).
    */
   #define THP_ORDERS_ALL_ANON    ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
+#define THP_MIN_ORDER        2
     /*
    * Mask of all large folio orders supported for file THP.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 272

Quoted 8 lines
include/linux/huge_mm.h +4 -0
@@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                         enforce_sysfs, orders);
   }
   +enum thp_event_item {
+    THP_ANON_ALLOC_SUCCESS,
+    THP_ANON_ALLOC_FAIL,
+    NR_THP_EVENT_ITEMS
+};

Maybe use a prefix that resembles matches the enum name and is "obviously" different to the ones in vm_event_item.h, like

Quoted 5 lines
enum thp_event {
     THP_EVENT_ANON_ALLOC_SUCCESS,
     THP_EVENT_ANON_ALLOC_FAIL,
     __THP_EVENT_COUNT,
};

FWIW, I'd personally replace "event" with "stat". For me "event" only ever increments, but "stat" can increment and decrement. An event is a type of stat.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

You are only adding events for now, but we have identified a need for inc/dec stats that will be added in future.

What about the below?

enum thp_stat {

It seems we still need to use enum thp_stat_item rather than thp_stat. This follows
enum zone_stat_item
enum numa_stat_item
enum node_stat_item

And most importantly, the below looks much better

       enum thp_stat_item {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };

       DECLARE_PER_CPU(struct thp_state, thp_states);

than

       enum thp_stat {
              THP_STAT_ANON_ALLOC,
              THP_STAT_ANON_ALLOC_FALLBACK,
              __THP_STAT_COUNT
       };

       struct thp_state {
              unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];
       };
Quoted 28 lines
Quoted 27 lines
Quoted 7 lines
    THP_EVENT_ANON_ALLOC,
    THP_EVENT_ANON_ALLOC_FALLBACK,
    THP_EVENT_SWPOUT,
    THP_EVENT_SWPOUT_FALLBACK,
    ...
    THP_NR_ANON_PAGES,
    THP_NR_FILE_PAGES,

I find this ambiguous; Is it the number of THPs or the number of base pages?

I think David made the point about incorporating the enum name into the labels too, so that there can be no namespace confusion. How about:

<enum>_<type>_<name>

So:

THP_STAT_EV_ANON_ALLOC
THP_STAT_EV_ANON_ALLOC_FALLBACK
THP_STAT_EV_ANON_PARTIAL
THP_STAT_EV_SWPOUT
THP_STAT_EV_SWPOUT_FALLBACK
...
THP_STAT_NR_ANON
THP_STAT_NR_FILE
...
__THP_STAT_COUNT

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I'd even drop the "EV". "NR_ANON" vs "ANON_ALLOC" etc. is expressive enough.

ok.

Hi David, Ryan,

I've named everything as follows. Please let me know if you have any further suggestions before I send the updated version :-)

Please treat all my comments below as optional - I'm just stating my preference!

Quoted 10 lines
include/linux/huge_mm.h +0 -0
index e896ca4760f6..cc13fa14aa32 100644
@@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct
 vm_area_struct *vma,
                                          enforce_sysfs, orders);
 }
 
Quoted 11 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct · line 275

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+enum thp_stat_item {
+       THP_STAT_ANON_ALLOC,
+       THP_STAT_ANON_ALLOC_FALLBACK,
+       __THP_STAT_COUNT
+};
+
+struct thp_state {
+       unsigned long state[PMD_ORDER + 1][__THP_STAT_COUNT];

Why using "state" here? I think "stats" is more appropriate? (as in short for "statistics") i.e. struct thp_stats and unsigned long stats[][].

Sounds good.

Quoted 21 lines
Quoted 21 lines
include/linux/huge_mm.h +9 -0
+};
+
+DECLARE_PER_CPU(struct thp_state, thp_states);
+
+static inline void count_thp_state(int order, enum thp_stat_item item)
+{
+       this_cpu_inc(thp_states.state[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()                           \
        (transparent_hugepage_flags &                                   \

(1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))

mm/huge_memory.c +0 -0
index 9d4b2fbf6872..e704b4408181 100644
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
        .sysfs_ops = &kobj_sysfs_ops,
 };
 
Quoted 4 lines

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 529

Quoted 1 line
mm/huge_memory.c +1 -0
+DEFINE_PER_CPU(struct thp_state, thp_states) = {{{0}}};

Does this need to be explicitly zeroed? Won't that be the default initial state, just like for regular globals? Perhaps PER_CPU is special?

I really don't know. uninitialized data is usually put in bss section so kernel will memset them to 0. but uninitialized per-cpu data is still in .data..percpu section and I can't find where memset is done in this area as this section can also contain initialized per-cpu data. Will compilers fill zero for uninitialized per-cpu data automatically so kernel just thinks they are initialized 0 data?

anyway, I copied the code from vmstat DEFINE_PER_CPU(struct vm_event_state, vm_event_states) = {{0}};

Quoted 5 lines

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 531

Quoted 3 lines
mm/huge_memory.c +2 -0
+
+static unsigned long sum_thp_states(int order, enum thp_stat_item item)

Again, I'd call it sum_thp_stats().

sounds good.

Quoted 17 lines
Quoted 5 lines
mm/huge_memory.c +4 -0
+{
+       unsigned long sum = 0;
+       int cpu;
+

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 545

Quoted 10 lines
mm/huge_memory.c +10 -0
+       for_each_online_cpu(cpu) {
+               struct thp_state *this = &per_cpu(thp_states, cpu);
+
+               sum += this->state[order][item];
+       }
+
+       return sum;
+}
+
+#define THP_STATE_ATTR(_name, _index)                                  \

And THP_STATS_ATTR(); they are going to live in the "stats" directory after all.

sounds good.

Quoted 23 lines
Quoted 22 lines
mm/huge_memory.c +11 -0
+static ssize_t _name##_show(struct kobject *kobj,                      \
+                       struct kobj_attribute *attr, char *buf)         \
+{                                                                      \
+       int order = to_thpsize(kobj)->order;                            \
+                                                                       \
+       return sysfs_emit(buf, "%lu\n", sum_thp_states(order, _index)); \
+}                                                                      \
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
+
+THP_STATE_ATTR(anon_alloc, THP_STAT_ANON_ALLOC);
+THP_STATE_ATTR(anon_alloc_fallback, THP_STAT_ANON_ALLOC_FALLBACK);
 
Quoted 6 lines
Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry