Skip to contentSkip to conversation

mm: add per-order mTHP alloc and swpout counters

PATCH Superseded Barry Song 3 participants 12 messages Apr 5, 2024 – Apr 12, 2024

Subsystems (6)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Barry Song

author
#0 lore raw

v4 cover mm: add per-order mTHP alloc and swpout counters

From: Barry Song <[email protected]>

The patchset introduces a framework to facilitate mTHP counters, starting with the allocation and swap-out counters. Currently, only four new nodes are appended to the stats directory for each mTHP size.

/sys/kernel/mm/transparent_hugepage/hugepages-<size>/stats
anon_alloc
anon_alloc_fallback
anon_swpout
anon_swpout_fallback

These four nodes are crucial for us to monitor the fragmentation levels of both the buddy system and the swapfile. In the future, we may consider adding additional nodes for further insights.

-v4:
* Many thanks to David and Ryan for your patience and valuable insights throughout the numerous renaming efforts!
* Guard the case order > PMD_ORDER in count func rather than in callers,

    Ryan;
  * Add swpout counters;
  * Add a helper DEFINE_MTHP_STAT_ATTR to avoid code duplication for various
    counters;

-v3:
https://lore.kernel.org/linux-mm/[email protected]/

Barry Song (2):
mm: add per-order mTHP anon_alloc and anon_alloc_fallback counters
mm: add per-order mTHP anon_swpout and anon_swpout_fallback counters

 include/linux/huge_mm.h | 21 +++++++++++++++++
 mm/huge_memory.c        | 52 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 mm/page_io.c            |  6 ++++-
 mm/vmscan.c             |  3 +++
 5 files changed, 83 insertions(+), 1 deletion(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v4 1/2 mm: add per-order mTHP anon_alloc and anon_alloc_fallback counters

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc and anon_alloc_fallback counters. Incorporating additional counters should now be straightforward as well.

---
 include/linux/huge_mm.h | 19 ++++++++++++++++
 mm/huge_memory.c        | 48 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 3 files changed, 69 insertions(+)
include/linux/huge_mm.h +19 -0
index e896ca4760f6..c5d33017a4dd 100644
@@ -264,6 +264,25 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 					  enforce_sysfs, orders);
 }
 
+enum mthp_stat_item {
+	MTHP_STAT_ANON_ALLOC,
+	MTHP_STAT_ANON_ALLOC_FALLBACK,
+	__MTHP_STAT_COUNT
+};
+
+struct mthp_stat {
+	unsigned long stats[PMD_ORDER + 1][__MTHP_STAT_COUNT];
+};
+
+DECLARE_PER_CPU(struct mthp_stat, mthp_stats);
+
+static inline void count_mthp_stat(int order, enum mthp_stat_item item)
+{
+	if (unlikely(order > PMD_ORDER))
+		return;
+	this_cpu_inc(mthp_stats.stats[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()				\
 	(transparent_hugepage_flags &					\
 	 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +48 -0

68 lines not on this page · raw message

mm/memory.c +2 -0
index 649e3ed94487..1723c8ddf9cb 100644
@@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
 			}
 			folio_throttle_swaprate(folio, gfp);
 			clear_huge_page(&folio->page, vmf->address, 1 << order);
+			count_mthp_stat(order, MTHP_STAT_ANON_ALLOC);
 			return folio;
 		}
+		count_mthp_stat(order, MTHP_STAT_ANON_ALLOC_FALLBACK);
 next:
 		order = next_order(&orders, order);
 	}
Signature · 1 line
2.34.1
reviewer
#2 lore raw

Re: [PATCH v4 1/2] mm: add per-order mTHP anon_alloc and anon_alloc_fallback counters

Quoted 28 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc and anon_alloc_fallback counters. Incorporating additional counters should now be straightforward as well.

---
 include/linux/huge_mm.h | 19 ++++++++++++++++
 mm/huge_memory.c        | 48 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 3 files changed, 69 insertions(+)
include/linux/huge_mm.h +0 -0
index e896ca4760f6..c5d33017a4dd 100644
@@ -264,6 +264,25 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 					  enforce_sysfs, orders);

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 274

Quoted 10 lines
include/linux/huge_mm.h +8 -0
 }
 
+enum mthp_stat_item {
+	MTHP_STAT_ANON_ALLOC,
+	MTHP_STAT_ANON_ALLOC_FALLBACK,
+	__MTHP_STAT_COUNT
+};
+
+struct mthp_stat {
+	unsigned long stats[PMD_ORDER + 1][__MTHP_STAT_COUNT];

I saw a fix for this allocation dynamically due to powerpc PMD_ORDER not being constant. I wonder if ilog2(MAX_PTRS_PER_PTE) would help here?

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 279

Quoted 5 lines
include/linux/huge_mm.h +5 -0
+};
+
+DECLARE_PER_CPU(struct mthp_stat, mthp_stats);
+
+static inline void count_mthp_stat(int order, enum mthp_stat_item item)

I thought we were going to call this always counting up type of stat and event? "count_mthp_event"? But I'm happy with it as is, personally.

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 282

Quoted 3 lines
include/linux/huge_mm.h +3 -0
+{
+	if (unlikely(order > PMD_ORDER))
+		return;

I'm wondering if it also makes sense to ignore order == 0? Although I guess if called for order-0 its safe since the storage exists and sum_mthp_stat() is never be called for 0. Ignore this comment :)

Quoted 12 lines
include/linux/huge_mm.h +3 -0
+	this_cpu_inc(mthp_stats.stats[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()				\
 	(transparent_hugepage_flags &					\
 	 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +0 -0
index 9d4b2fbf6872..5b875f0fc923 100644
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
 	.sysfs_ops = &kobj_sysfs_ops,

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 536

Quoted 10 lines
mm/huge_memory.c +8 -0
 };
 
+DEFINE_PER_CPU(struct mthp_stat, mthp_stats) = {{{0}}};
+
+static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item)
+{
+	unsigned long sum = 0;
+	int cpu;
+
+	for_each_online_cpu(cpu) {

What happens if a cpu that was online and collected a bunch of stats gets offlined? The user will see stats get smaller?

Perhaps this should be for_each_possible_cpu()? Although I'm not sure what happens to percpu data when a cpu goes offline? Is the data preserved? Or wiped, or unmapped? dunno. Might we need to rescue stats into a global counter at offline-time?

Quoted 7 lines
mm/huge_memory.c +7 -0
+		struct mthp_stat *this = &per_cpu(mthp_stats, cpu);
+
+		sum += this->stats[order][item];
+	}
+
+	return sum;
+}

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 553

Quoted 10 lines
mm/huge_memory.c +10 -0
+
+#define DEFINE_MTHP_STAT_ATTR(_name, _index)					\
+static ssize_t _name##_show(struct kobject *kobj,			\
+			struct kobj_attribute *attr, char *buf)		\
+{									\
+	int order = to_thpsize(kobj)->order;				\
+									\
+	return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index));	\
+}									\
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)

Very nice!

Quoted 32 lines
mm/huge_memory.c +21 -0
+
+DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
+DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+
+static struct attribute *stats_attrs[] = {
+	&anon_alloc_attr.attr,
+	&anon_alloc_fallback_attr.attr,
+	NULL,
+};
+
+static struct attribute_group stats_attr_group = {
+	.name = "stats",
+	.attrs = stats_attrs,
+};
+
 static struct thpsize *thpsize_create(int order, struct kobject *parent)
 {
 	unsigned long size = (PAGE_SIZE << order) / SZ_1K;
@@ -549,6 +589,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent)
 		return ERR_PTR(ret);
 	}
 
+	ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group);
+	if (ret) {
+		kobject_put(&thpsize->kobj);
+		return ERR_PTR(ret);
+	}
+
 	thpsize->order = order;
 	return thpsize;
 }

mm/huge_memory.c vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) · line 1099

Quoted 4 lines
mm/huge_memory.c +1 -0
@@ -1050,8 +1096,10 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 	folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true);
 	if (unlikely(!folio)) {
 		count_vm_event(THP_FAULT_FALLBACK);
+		count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC_FALLBACK);

I think we should aim for the PMD-oder MTHP_STAT_ANON_ALLOC and MTHP_STAT_ANON_ALLOC_FALLBACK to match THP_FAULT_ALLOC and THP_FAULT_FALLBACK. Its not currently setup this way...

Quoted 11 lines
mm/huge_memory.c +1 -0
 		return VM_FAULT_FALLBACK;
 	}
+	count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC);
 	return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp);
 }
 
mm/memory.c +0 -0
index 649e3ed94487..1723c8ddf9cb 100644

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4380

Quoted 7 lines
mm/memory.c +2 -0
@@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
 			}
 			folio_throttle_swaprate(folio, gfp);
 			clear_huge_page(&folio->page, vmf->address, 1 << order);
+			count_mthp_stat(order, MTHP_STAT_ANON_ALLOC);
 			return folio;
 		}
+		count_mthp_stat(order, MTHP_STAT_ANON_ALLOC_FALLBACK);

...And we should follow the usage same pattern for the smaller mTHP here too. Which means MTHP_STAT_ANON_ALLOC_FALLBACK should be after the next: label. We could introduce a MTHP_STAT_ANON_ALLOC_FALLBACK_CHARGE which would only trigger on a fallback due to charge failure, just like THP_FAULT_FALLBACK_CHARGE?

Quoted 3 lines
mm/memory.c +0 -0
 next:
 		order = next_order(&orders, order);
 	}
author
#3 lore raw

Re: [PATCH v4 1/2] mm: add per-order mTHP anon_alloc and anon_alloc_fallback counters

Quoted 32 lines · Ryan Roberts
Quoted 30 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc and anon_alloc_fallback counters. Incorporating additional counters should now be straightforward as well.

---
 include/linux/huge_mm.h | 19 ++++++++++++++++
 mm/huge_memory.c        | 48 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 3 files changed, 69 insertions(+)
include/linux/huge_mm.h +0 -0
index e896ca4760f6..c5d33017a4dd 100644
@@ -264,6 +264,25 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
                                        enforce_sysfs, orders);
 }
 
Quoted 12 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 274

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+enum mthp_stat_item {
+     MTHP_STAT_ANON_ALLOC,
+     MTHP_STAT_ANON_ALLOC_FALLBACK,
+     __MTHP_STAT_COUNT
+};
+
+struct mthp_stat {
+     unsigned long stats[PMD_ORDER + 1][__MTHP_STAT_COUNT];

I saw a fix for this allocation dynamically due to powerpc PMD_ORDER not being constant. I wonder if ilog2(MAX_PTRS_PER_PTE) would help here?

It's a possibility. However, since we've passed all the build tests using dynamic
allocation, it might not be worth the effort to attempt static allocation again. Who
knows what will happen next :-)

Quoted 9 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 279

Quoted 5 lines
include/linux/huge_mm.h +5 -0
+};
+
+DECLARE_PER_CPU(struct mthp_stat, mthp_stats);
+
+static inline void count_mthp_stat(int order, enum mthp_stat_item item)

I thought we were going to call this always counting up type of stat and event? "count_mthp_event"? But I'm happy with it as is, personally.

Quoted 7 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 282

Quoted 3 lines
include/linux/huge_mm.h +3 -0
+{
+     if (unlikely(order > PMD_ORDER))
+             return;

I'm wondering if it also makes sense to ignore order == 0? Although I guess if called for order-0 its safe since the storage exists and sum_mthp_stat() is never be called for 0. Ignore this comment :)

Agreed. I'd like to change it to ignore oder 0;

Quoted 27 lines
Quoted 13 lines
include/linux/huge_mm.h +3 -0
+     this_cpu_inc(mthp_stats.stats[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()                         \
      (transparent_hugepage_flags &                                   \
       (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +0 -0
index 9d4b2fbf6872..5b875f0fc923 100644
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
      .sysfs_ops = &kobj_sysfs_ops,

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 536

Quoted 10 lines
mm/huge_memory.c +8 -0
 };
 
+DEFINE_PER_CPU(struct mthp_stat, mthp_stats) = {{{0}}};
+
+static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item)
+{
+     unsigned long sum = 0;
+     int cpu;
+
+     for_each_online_cpu(cpu) {

What happens if a cpu that was online and collected a bunch of stats gets offlined? The user will see stats get smaller?

Quoted 4 lines

Perhaps this should be for_each_possible_cpu()? Although I'm not sure what happens to percpu data when a cpu goes offline? Is the data preserved? Or wiped, or unmapped? dunno. Might we need to rescue stats into a global counter at offline-time?

Good catch. I see /proc/vmstat is always using the for_each_online_cpu() but it doesn't have the issue, but mTHP counters do have the problem.

* step 1: cat the current thp_swpout value before running a test program which does
swpout;

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 0
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 0

* step 2: run the test program on cpu2;

/ # taskset -c 2 /home/barry/develop/linux/swpcache-2m

* step 3: cat the current thp_swpout value after running a test program which does
swpout;

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 98

*step 4: offline cpu2 and read thp_swpout;

/ # echo 0 > /sys/devices/system/cpu/cpu2/online

[  339.058661] psci: CPU2 killed (polled 0 ms)

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 0

*step 5: online cpu2 and read thp_swpout

/ # echo 1 > /sys/devices/system/cpu/cpu2/online

[  791.642058] CPU2: Booted secondary processor 0x0000000002 [0x000f0510]

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 98

As you can see, in step 4, /proc/vmstat is all good but mTHP counters become zero.

The reason is /proc/vmstat will fold the offline cpu to an online cpu but mthp counters lack
it:

/*
* Fold the foreign cpu events into our own.
*
* This is adding to the events on one processor
* but keeps the global counts constant.
*/
void vm_events_fold_cpu(int cpu)

{
        struct vm_event_state *fold_state = &per_cpu(vm_event_states, cpu);
        int i;

        for (i = 0; i < NR_VM_EVENT_ITEMS; i++) {
                count_vm_events(i, fold_state->event[i]);
                fold_state->event[i] = 0;
        }
}

static int page_alloc_cpu_dead(unsigned int cpu)

{
        ...
        /*
         * Spill the event counters of the dead processor
         * into the current processors event counters.
         * This artificially elevates the count of the current
         * processor.
         */
        vm_events_fold_cpu(cpu);
        ...

        return 0;
}

So I will do the same thing for mTHP counters - fold offline cpu counters to online one.

Quoted 12 lines
Quoted 12 lines
mm/huge_memory.c +11 -0
+             struct mthp_stat *this = &per_cpu(mthp_stats, cpu);
+
+             sum += this->stats[order][item];
+     }
+
+     return sum;
+}
+
+#define DEFINE_MTHP_STAT_ATTR(_name, _index)                                 \
+static ssize_t _name##_show(struct kobject *kobj,                    \
+                     struct kobj_attribute *attr, char *buf)         \
Quoted 8 lines

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 553

Quoted 6 lines
mm/huge_memory.c +6 -0
+{                                                                    \
+     int order = to_thpsize(kobj)->order;                            \
+                                                                     \
+     return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index));  \
+}                                                                    \
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)

Very nice!

Right. I got duplicated copy-paste and bad small in code so I wrote this macro.

Quoted 33 lines
Quoted 33 lines
mm/huge_memory.c +21 -0
+
+DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
+DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+
+static struct attribute *stats_attrs[] = {
+     &anon_alloc_attr.attr,
+     &anon_alloc_fallback_attr.attr,
+     NULL,
+};
+
+static struct attribute_group stats_attr_group = {
+     .name = "stats",
+     .attrs = stats_attrs,
+};
+
 static struct thpsize *thpsize_create(int order, struct kobject *parent)
 {
      unsigned long size = (PAGE_SIZE << order) / SZ_1K;
@@ -549,6 +589,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent)
              return ERR_PTR(ret);
      }
 
+     ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group);
+     if (ret) {
+             kobject_put(&thpsize->kobj);
+             return ERR_PTR(ret);
+     }
+
      thpsize->order = order;
      return thpsize;
 }
@@ -1050,8 +1096,10 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
Quoted 8 lines

mm/huge_memory.c vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) · line 1099

Quoted 4 lines
mm/huge_memory.c +1 -0
      folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true);
      if (unlikely(!folio)) {
              count_vm_event(THP_FAULT_FALLBACK);
+             count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC_FALLBACK);

I think we should aim for the PMD-oder MTHP_STAT_ANON_ALLOC and MTHP_STAT_ANON_ALLOC_FALLBACK to match THP_FAULT_ALLOC and THP_FAULT_FALLBACK. Its not currently setup this way...

right. I also realized this and asked for your comments on this in another thread.

Quoted 8 lines
Quoted 8 lines
mm/huge_memory.c +1 -0
              return VM_FAULT_FALLBACK;
      }
+     count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC);
      return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp);
 }
 
Quoted 15 lines

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4380

Quoted 12 lines
mm/memory.c +2 -0
index 649e3ed94487..1723c8ddf9cb 100644
@@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
                      }
                      folio_throttle_swaprate(folio, gfp);
                      clear_huge_page(&folio->page, vmf->address, 1 << order);
+                     count_mthp_stat(order, MTHP_STAT_ANON_ALLOC);
                      return folio;
              }
+             count_mthp_stat(order, MTHP_STAT_ANON_ALLOC_FALLBACK);

...And we should follow the usage same pattern for the smaller mTHP here too. Which means MTHP_STAT_ANON_ALLOC_FALLBACK should be after the next: label. We

The only difference is the case

if (mem_cgroup_charge(folio, vma->vm_mm, gfp))

      goto next;

but vmstat is counting this as fallback so i feel good to move after next,

        if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
                folio_put(folio);
                count_vm_event(THP_FAULT_FALLBACK);
                count_vm_event(THP_FAULT_FALLBACK_CHARGE);
                return VM_FAULT_FALLBACK;
        }
Quoted 2 lines

could introduce a MTHP_STAT_ANON_ALLOC_FALLBACK_CHARGE which would only trigger on a fallback due to charge failure, just like THP_FAULT_FALLBACK_CHARGE?

it is fine to add this THP_FAULT_FALLBACK_CHARGE though it is not that useful for profiling buddy fragmentation.

Quoted 5 lines

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4383

Quoted 4 lines
mm/memory.c +0 -0
 next:
              order = next_order(&orders, order);
      }

Thanks
Barry

reviewer
#4 lore raw

Re: [PATCH v4 1/2] mm: add per-order mTHP anon_alloc and anon_alloc_fallback counters

Quoted 46 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 30 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc and anon_alloc_fallback counters. Incorporating additional counters should now be straightforward as well.

---
 include/linux/huge_mm.h | 19 ++++++++++++++++
 mm/huge_memory.c        | 48 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 3 files changed, 69 insertions(+)
include/linux/huge_mm.h +0 -0
index e896ca4760f6..c5d33017a4dd 100644
@@ -264,6 +264,25 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
                                        enforce_sysfs, orders);
 }
 
Quoted 12 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 274

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+enum mthp_stat_item {
+     MTHP_STAT_ANON_ALLOC,
+     MTHP_STAT_ANON_ALLOC_FALLBACK,
+     __MTHP_STAT_COUNT
+};
+
+struct mthp_stat {
+     unsigned long stats[PMD_ORDER + 1][__MTHP_STAT_COUNT];

I saw a fix for this allocation dynamically due to powerpc PMD_ORDER not being constant. I wonder if ilog2(MAX_PTRS_PER_PTE) would help here?

Quoted 5 lines

It's a possibility. However, since we've passed all the build tests using dynamic
allocation, it might not be worth the effort to attempt static allocation again. Who
knows what will happen next :-)

If the dynamic version is clear and obvious then fair enough. I tried doing something similar for the swap-out series but it turned out a mess, so ended up falling back to static allocation which was much easier to understand.

Quoted 138 lines
Quoted 10 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 279

Quoted 6 lines
include/linux/huge_mm.h +5 -0
+};
+
+DECLARE_PER_CPU(struct mthp_stat, mthp_stats);
+
+static inline void count_mthp_stat(int order, enum mthp_stat_item item)

I thought we were going to call this always counting up type of stat and event? "count_mthp_event"? But I'm happy with it as is, personally.

Quoted 7 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 282

Quoted 3 lines
include/linux/huge_mm.h +3 -0
+{
+     if (unlikely(order > PMD_ORDER))
+             return;

I'm wondering if it also makes sense to ignore order == 0? Although I guess if called for order-0 its safe since the storage exists and sum_mthp_stat() is never be called for 0. Ignore this comment :)

Agreed. I'd like to change it to ignore oder 0;

Quoted 27 lines
Quoted 13 lines
include/linux/huge_mm.h +3 -0
+     this_cpu_inc(mthp_stats.stats[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()                         \
      (transparent_hugepage_flags &                                   \
       (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +0 -0
index 9d4b2fbf6872..5b875f0fc923 100644
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
      .sysfs_ops = &kobj_sysfs_ops,

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 536

Quoted 10 lines
mm/huge_memory.c +8 -0
 };
 
+DEFINE_PER_CPU(struct mthp_stat, mthp_stats) = {{{0}}};
+
+static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item)
+{
+     unsigned long sum = 0;
+     int cpu;
+
+     for_each_online_cpu(cpu) {

What happens if a cpu that was online and collected a bunch of stats gets offlined? The user will see stats get smaller?

Quoted 4 lines

Perhaps this should be for_each_possible_cpu()? Although I'm not sure what happens to percpu data when a cpu goes offline? Is the data preserved? Or wiped, or unmapped? dunno. Might we need to rescue stats into a global counter at offline-time?

Good catch. I see /proc/vmstat is always using the for_each_online_cpu() but it doesn't have the issue, but mTHP counters do have the problem.

* step 1: cat the current thp_swpout value before running a test program which does
swpout;

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 0
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 0

* step 2: run the test program on cpu2;

/ # taskset -c 2 /home/barry/develop/linux/swpcache-2m

* step 3: cat the current thp_swpout value after running a test program which does
swpout;

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 98

*step 4: offline cpu2 and read thp_swpout;

/ # echo 0 > /sys/devices/system/cpu/cpu2/online

[  339.058661] psci: CPU2 killed (polled 0 ms)

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 0

*step 5: online cpu2 and read thp_swpout

/ # echo 1 > /sys/devices/system/cpu/cpu2/online

[  791.642058] CPU2: Booted secondary processor 0x0000000002 [0x000f0510]

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 98

As you can see, in step 4, /proc/vmstat is all good but mTHP counters become zero.

The reason is /proc/vmstat will fold the offline cpu to an online cpu but mthp counters lack
it:

/*
* Fold the foreign cpu events into our own.
*
* This is adding to the events on one processor
* but keeps the global counts constant.
*/
void vm_events_fold_cpu(int cpu)

{
        struct vm_event_state *fold_state = &per_cpu(vm_event_states, cpu);
        int i;

        for (i = 0; i < NR_VM_EVENT_ITEMS; i++) {
                count_vm_events(i, fold_state->event[i]);
                fold_state->event[i] = 0;
        }
}

static int page_alloc_cpu_dead(unsigned int cpu)

{
        ...
        /*
         * Spill the event counters of the dead processor
         * into the current processors event counters.
         * This artificially elevates the count of the current
         * processor.
         */
        vm_events_fold_cpu(cpu);
        ...

        return 0;
}
Quoted 2 lines

So I will do the same thing for mTHP counters - fold offline cpu counters to online one.

That all looks like a complete mess - better avoided if possible! A quick search for "for_each_possible_cpu" shows loads of places where code is iterating over all *possible* cpus and grabbing its per-cpu data. So the data definitely remains accessible when the cpu is offline. Looks like it doesn't get wiped either.

So can't you just change your sum function to iterate over all possible cpus?

Quoted 66 lines
Quoted 13 lines
Quoted 13 lines
mm/huge_memory.c +11 -0
+             struct mthp_stat *this = &per_cpu(mthp_stats, cpu);
+
+             sum += this->stats[order][item];
+     }
+
+     return sum;
+}
+
+#define DEFINE_MTHP_STAT_ATTR(_name, _index)                                 \
+static ssize_t _name##_show(struct kobject *kobj,                    \
+                     struct kobj_attribute *attr, char *buf)         \
Quoted 8 lines

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 553

Quoted 6 lines
mm/huge_memory.c +6 -0
+{                                                                    \
+     int order = to_thpsize(kobj)->order;                            \
+                                                                     \
+     return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index));  \
+}                                                                    \
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)

Very nice!

Right. I got duplicated copy-paste and bad small in code so I wrote this macro.

Quoted 33 lines
Quoted 33 lines
mm/huge_memory.c +21 -0
+
+DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
+DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+
+static struct attribute *stats_attrs[] = {
+     &anon_alloc_attr.attr,
+     &anon_alloc_fallback_attr.attr,
+     NULL,
+};
+
+static struct attribute_group stats_attr_group = {
+     .name = "stats",
+     .attrs = stats_attrs,
+};
+
 static struct thpsize *thpsize_create(int order, struct kobject *parent)
 {
      unsigned long size = (PAGE_SIZE << order) / SZ_1K;
@@ -549,6 +589,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent)
              return ERR_PTR(ret);
      }
 
+     ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group);
+     if (ret) {
+             kobject_put(&thpsize->kobj);
+             return ERR_PTR(ret);
+     }
+
      thpsize->order = order;
      return thpsize;
 }
@@ -1050,8 +1096,10 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
Quoted 8 lines

mm/huge_memory.c vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) · line 1099

Quoted 4 lines
mm/huge_memory.c +1 -0
      folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true);
      if (unlikely(!folio)) {
              count_vm_event(THP_FAULT_FALLBACK);
+             count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC_FALLBACK);

I think we should aim for the PMD-oder MTHP_STAT_ANON_ALLOC and MTHP_STAT_ANON_ALLOC_FALLBACK to match THP_FAULT_ALLOC and THP_FAULT_FALLBACK. Its not currently setup this way...

Quoted 2 lines

right. I also realized this and asked for your comments on this in another thread.

Ahh sorry - must have missed that.

Quoted 39 lines
Quoted 9 lines
Quoted 9 lines
mm/huge_memory.c +1 -0
              return VM_FAULT_FALLBACK;
      }
+     count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC);
      return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp);
 }
 
Quoted 15 lines

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4380

Quoted 12 lines
mm/memory.c +2 -0
index 649e3ed94487..1723c8ddf9cb 100644
@@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
                      }
                      folio_throttle_swaprate(folio, gfp);
                      clear_huge_page(&folio->page, vmf->address, 1 << order);
+                     count_mthp_stat(order, MTHP_STAT_ANON_ALLOC);
                      return folio;
              }
+             count_mthp_stat(order, MTHP_STAT_ANON_ALLOC_FALLBACK);

...And we should follow the usage same pattern for the smaller mTHP here too. Which means MTHP_STAT_ANON_ALLOC_FALLBACK should be after the next: label. We

The only difference is the case

if (mem_cgroup_charge(folio, vma->vm_mm, gfp))

      goto next;

but vmstat is counting this as fallback so i feel good to move after next,

        if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
                folio_put(folio);
                count_vm_event(THP_FAULT_FALLBACK);
                count_vm_event(THP_FAULT_FALLBACK_CHARGE);
                return VM_FAULT_FALLBACK;
        }
Quoted 5 lines
Quoted 2 lines

could introduce a MTHP_STAT_ANON_ALLOC_FALLBACK_CHARGE which would only trigger on a fallback due to charge failure, just like THP_FAULT_FALLBACK_CHARGE?

it is fine to add this THP_FAULT_FALLBACK_CHARGE though it is not that useful for profiling buddy fragmentation.

Well I thought you were interested in isolating fallback due to fragmentation only. You would get that with (FAULT_FALLBACK - FAULT_FALLBACK_CHARGE)? But if you think the latter will be relatively small/unimportant for now, and therefore FAULT_FALLBACK will give good enough approximation on its own, then I'm happy not to add FAULT_FALLBACK_CHARGE for now.

Quoted 9 lines
Quoted 6 lines

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4383

Quoted 5 lines
mm/memory.c +0 -0
 next:
              order = next_order(&orders, order);
      }

Thanks
Barry

author
#5 lore raw

Re: [PATCH v4 1/2] mm: add per-order mTHP anon_alloc and anon_alloc_fallback counters

Quoted 54 lines · Ryan Roberts
Quoted 46 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 30 lines · Barry Song

From: Barry Song <[email protected]>

Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need.

Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory.

This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc and anon_alloc_fallback counters. Incorporating additional counters should now be straightforward as well.

---
 include/linux/huge_mm.h | 19 ++++++++++++++++
 mm/huge_memory.c        | 48 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             |  2 ++
 3 files changed, 69 insertions(+)
include/linux/huge_mm.h +0 -0
index e896ca4760f6..c5d33017a4dd 100644
@@ -264,6 +264,25 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
                                        enforce_sysfs, orders);
 }
 
Quoted 12 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 274

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+enum mthp_stat_item {
+     MTHP_STAT_ANON_ALLOC,
+     MTHP_STAT_ANON_ALLOC_FALLBACK,
+     __MTHP_STAT_COUNT
+};
+
+struct mthp_stat {
+     unsigned long stats[PMD_ORDER + 1][__MTHP_STAT_COUNT];

I saw a fix for this allocation dynamically due to powerpc PMD_ORDER not being constant. I wonder if ilog2(MAX_PTRS_PER_PTE) would help here?

Quoted 5 lines

It's a possibility. However, since we've passed all the build tests using dynamic
allocation, it might not be worth the effort to attempt static allocation again. Who
knows what will happen next :-)

Quoted 3 lines

If the dynamic version is clear and obvious then fair enough. I tried doing something similar for the swap-out series but it turned out a mess, so ended up falling back to static allocation which was much easier to understand.

mthp_stats doesn't need to be dynamically released, as long as hugepage_init() succeeds, we need it forever even if sysfs is released or disabled.

Quoted 142 lines
Quoted 139 lines
Quoted 11 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 279

Quoted 7 lines
include/linux/huge_mm.h +5 -0
+};
+
+DECLARE_PER_CPU(struct mthp_stat, mthp_stats);
+
+static inline void count_mthp_stat(int order, enum mthp_stat_item item)

I thought we were going to call this always counting up type of stat and event? "count_mthp_event"? But I'm happy with it as is, personally.

Quoted 7 lines

include/linux/huge_mm.h unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · line 282

Quoted 3 lines
include/linux/huge_mm.h +3 -0
+{
+     if (unlikely(order > PMD_ORDER))
+             return;

I'm wondering if it also makes sense to ignore order == 0? Although I guess if called for order-0 its safe since the storage exists and sum_mthp_stat() is never be called for 0. Ignore this comment :)

Agreed. I'd like to change it to ignore oder 0;

Quoted 27 lines
Quoted 13 lines
include/linux/huge_mm.h +3 -0
+     this_cpu_inc(mthp_stats.stats[order][item]);
+}
+
 #define transparent_hugepage_use_zero_page()                         \
      (transparent_hugepage_flags &                                   \
       (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG))
mm/huge_memory.c +0 -0
index 9d4b2fbf6872..5b875f0fc923 100644
@@ -526,6 +526,46 @@ static const struct kobj_type thpsize_ktype = {
      .sysfs_ops = &kobj_sysfs_ops,

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 536

Quoted 10 lines
mm/huge_memory.c +8 -0
 };
 
+DEFINE_PER_CPU(struct mthp_stat, mthp_stats) = {{{0}}};
+
+static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item)
+{
+     unsigned long sum = 0;
+     int cpu;
+
+     for_each_online_cpu(cpu) {

What happens if a cpu that was online and collected a bunch of stats gets offlined? The user will see stats get smaller?

Quoted 4 lines

Perhaps this should be for_each_possible_cpu()? Although I'm not sure what happens to percpu data when a cpu goes offline? Is the data preserved? Or wiped, or unmapped? dunno. Might we need to rescue stats into a global counter at offline-time?

Good catch. I see /proc/vmstat is always using the for_each_online_cpu() but it doesn't have the issue, but mTHP counters do have the problem.

* step 1: cat the current thp_swpout value before running a test program which does
swpout;

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 0
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 0

* step 2: run the test program on cpu2;

/ # taskset -c 2 /home/barry/develop/linux/swpcache-2m

* step 3: cat the current thp_swpout value after running a test program which does
swpout;

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 98

*step 4: offline cpu2 and read thp_swpout;

/ # echo 0 > /sys/devices/system/cpu/cpu2/online

[  339.058661] psci: CPU2 killed (polled 0 ms)

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 0

*step 5: online cpu2 and read thp_swpout

/ # echo 1 > /sys/devices/system/cpu/cpu2/online

[  791.642058] CPU2: Booted secondary processor 0x0000000002 [0x000f0510]

/ # cat /proc/vmstat | grep thp_swpout
thp_swpout 98
/ # cat /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/stats/anon_swpout 98

As you can see, in step 4, /proc/vmstat is all good but mTHP counters become zero.

The reason is /proc/vmstat will fold the offline cpu to an online cpu but mthp counters lack
it:

/*
* Fold the foreign cpu events into our own.
*
* This is adding to the events on one processor
* but keeps the global counts constant.
*/
void vm_events_fold_cpu(int cpu)

{
        struct vm_event_state *fold_state = &per_cpu(vm_event_states, cpu);
        int i;

        for (i = 0; i < NR_VM_EVENT_ITEMS; i++) {
                count_vm_events(i, fold_state->event[i]);
                fold_state->event[i] = 0;
        }
}

static int page_alloc_cpu_dead(unsigned int cpu)

{
        ...
        /*
         * Spill the event counters of the dead processor
         * into the current processors event counters.
         * This artificially elevates the count of the current
         * processor.
         */
        vm_events_fold_cpu(cpu);
        ...

        return 0;
}
Quoted 2 lines

So I will do the same thing for mTHP counters - fold offline cpu counters to online one.

Quoted 6 lines

That all looks like a complete mess - better avoided if possible! A quick search for "for_each_possible_cpu" shows loads of places where code is iterating over all *possible* cpus and grabbing its per-cpu data. So the data definitely remains accessible when the cpu is offline. Looks like it doesn't get wiped either.

So can't you just change your sum function to iterate over all possible cpus?().

I don't find why not. but i sent a v5[1] similar to vm_events_fold_cpu. i can move to for_each_possible_cpu() in v6. if it is a better approach, i can even further refine that for vmstat.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 114 lines
Quoted 67 lines
Quoted 14 lines
Quoted 14 lines
mm/huge_memory.c +11 -0
+             struct mthp_stat *this = &per_cpu(mthp_stats, cpu);
+
+             sum += this->stats[order][item];
+     }
+
+     return sum;
+}
+
+#define DEFINE_MTHP_STAT_ATTR(_name, _index)                                 \
+static ssize_t _name##_show(struct kobject *kobj,                    \
+                     struct kobj_attribute *attr, char *buf)         \
Quoted 8 lines

mm/huge_memory.c static const struct kobj_type thpsize_ktype = { · line 553

Quoted 6 lines
mm/huge_memory.c +6 -0
+{                                                                    \
+     int order = to_thpsize(kobj)->order;                            \
+                                                                     \
+     return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index));  \
+}                                                                    \
+static struct kobj_attribute _name##_attr = __ATTR_RO(_name)

Very nice!

Right. I got duplicated copy-paste and bad small in code so I wrote this macro.

Quoted 33 lines
Quoted 33 lines
mm/huge_memory.c +21 -0
+
+DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
+DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+
+static struct attribute *stats_attrs[] = {
+     &anon_alloc_attr.attr,
+     &anon_alloc_fallback_attr.attr,
+     NULL,
+};
+
+static struct attribute_group stats_attr_group = {
+     .name = "stats",
+     .attrs = stats_attrs,
+};
+
 static struct thpsize *thpsize_create(int order, struct kobject *parent)
 {
      unsigned long size = (PAGE_SIZE << order) / SZ_1K;
@@ -549,6 +589,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent)
              return ERR_PTR(ret);
      }
 
+     ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group);
+     if (ret) {
+             kobject_put(&thpsize->kobj);
+             return ERR_PTR(ret);
+     }
+
      thpsize->order = order;
      return thpsize;
 }
@@ -1050,8 +1096,10 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
Quoted 8 lines

mm/huge_memory.c vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) · line 1099

Quoted 4 lines
mm/huge_memory.c +1 -0
      folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true);
      if (unlikely(!folio)) {
              count_vm_event(THP_FAULT_FALLBACK);
+             count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC_FALLBACK);

I think we should aim for the PMD-oder MTHP_STAT_ANON_ALLOC and MTHP_STAT_ANON_ALLOC_FALLBACK to match THP_FAULT_ALLOC and THP_FAULT_FALLBACK. Its not currently setup this way...

Quoted 2 lines

right. I also realized this and asked for your comments on this in another thread.

Ahh sorry - must have missed that.

Quoted 42 lines
Quoted 9 lines
Quoted 9 lines
mm/huge_memory.c +1 -0
              return VM_FAULT_FALLBACK;
      }
+     count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_ALLOC);
      return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp);
 }
 
Quoted 15 lines

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4380

Quoted 12 lines
mm/memory.c +2 -0
index 649e3ed94487..1723c8ddf9cb 100644
@@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
                      }
                      folio_throttle_swaprate(folio, gfp);
                      clear_huge_page(&folio->page, vmf->address, 1 << order);
+                     count_mthp_stat(order, MTHP_STAT_ANON_ALLOC);
                      return folio;
              }
+             count_mthp_stat(order, MTHP_STAT_ANON_ALLOC_FALLBACK);

...And we should follow the usage same pattern for the smaller mTHP here too. Which means MTHP_STAT_ANON_ALLOC_FALLBACK should be after the next: label. We

The only difference is the case

if (mem_cgroup_charge(folio, vma->vm_mm, gfp))

      goto next;

but vmstat is counting this as fallback so i feel good to move after next,

        if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
                folio_put(folio);
                count_vm_event(THP_FAULT_FALLBACK);
                count_vm_event(THP_FAULT_FALLBACK_CHARGE);
                return VM_FAULT_FALLBACK;
        }
Quoted 2 lines

could introduce a MTHP_STAT_ANON_ALLOC_FALLBACK_CHARGE which would only trigger on a fallback due to charge failure, just like THP_FAULT_FALLBACK_CHARGE?

Quoted 8 lines
Quoted 2 lines

it is fine to add this THP_FAULT_FALLBACK_CHARGE though it is not that useful for profiling buddy fragmentation.

Well I thought you were interested in isolating fallback due to fragmentation only. You would get that with (FAULT_FALLBACK - FAULT_FALLBACK_CHARGE)? But if you think the latter will be relatively small/unimportant for now, and therefore FAULT_FALLBACK will give good enough approximation on its own, then I'm happy not to add FAULT_FALLBACK_CHARGE for now.

in v5, i actually added FAULT_FALLBACK_CHARGE. Please take a look at v5 :-)

Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4383

Quoted 6 lines
mm/memory.c +0 -0
 next:
              order = next_order(&orders, order);
      }

Thanks
Barry

author
#6 lore raw

PATCH v4 2/2 mm: add per-order mTHP anon_swpout and anon_swpout_fallback counters

In reply to Barry Song

From: Barry Song <[email protected]>

This helps to display the fragmentation situation of the swapfile, knowing the proportion of how much we haven't split large folios. So far, we only support non-split swapout for anon memory, with the possibility of expanding to shmem in the future. So, we add the "anon" prefix to the counter names.

---
 include/linux/huge_mm.h | 2 ++
 mm/huge_memory.c        | 4 ++++
 mm/page_io.c            | 6 +++++-
 mm/vmscan.c             | 3 +++
 4 files changed, 14 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +2 -0
index c5d33017a4dd..1d893a358df6 100644
@@ -267,6 +267,8 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 enum mthp_stat_item {
 	MTHP_STAT_ANON_ALLOC,
 	MTHP_STAT_ANON_ALLOC_FALLBACK,
+	MTHP_STAT_ANON_SWPOUT,
+	MTHP_STAT_ANON_SWPOUT_FALLBACK,
 	__MTHP_STAT_COUNT
 };
 
mm/huge_memory.c +4 -0
index 5b875f0fc923..28113f8fdf18 100644
@@ -554,10 +554,14 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
 
 DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
 DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT);
+DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK);
 
 static struct attribute *stats_attrs[] = {
 	&anon_alloc_attr.attr,
 	&anon_alloc_fallback_attr.attr,
+	&anon_swpout_attr.attr,
+	&anon_swpout_fallback_attr.attr,
 	NULL,
 };
 
mm/page_io.c +5 -1
index a9a7c236aecc..7669452e8b4d 100644
@@ -212,13 +212,17 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
 
 static inline void count_swpout_vm_event(struct folio *folio)
 {
+	long nr_pages = folio_nr_pages(folio);
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	if (unlikely(folio_test_pmd_mappable(folio))) {
 		count_memcg_folio_events(folio, THP_SWPOUT, 1);
 		count_vm_event(THP_SWPOUT);
 	}
+	if (nr_pages > 0)
+		count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_SWPOUT);
 #endif
-	count_vm_events(PSWPOUT, folio_nr_pages(folio));
+	count_vm_events(PSWPOUT, nr_pages);
 }
 
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
mm/vmscan.c +3 -0
index ffc4553c8615..b30e6294f82a 100644
@@ -1247,6 +1247,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 						count_vm_event(
 							THP_SWPOUT_FALLBACK);
 					}
+					if (nr_pages > 0)
+						count_mthp_stat(get_order(nr_pages * PAGE_SIZE),
+							MTHP_STAT_ANON_SWPOUT_FALLBACK);
 #endif
 					if (!add_to_swap(folio))
 						goto activate_locked_split;
Signature · 1 line
2.34.1
reviewer
#7 lore raw

Re: [PATCH v4 2/2] mm: add per-order mTHP anon_swpout and anon_swpout_fallback counters

Quoted 55 lines · Barry Song

From: Barry Song <[email protected]>

This helps to display the fragmentation situation of the swapfile, knowing the proportion of how much we haven't split large folios. So far, we only support non-split swapout for anon memory, with the possibility of expanding to shmem in the future. So, we add the "anon" prefix to the counter names.

---
 include/linux/huge_mm.h | 2 ++
 mm/huge_memory.c        | 4 ++++
 mm/page_io.c            | 6 +++++-
 mm/vmscan.c             | 3 +++
 4 files changed, 14 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +2 -0
index c5d33017a4dd..1d893a358df6 100644
@@ -267,6 +267,8 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 enum mthp_stat_item {
 	MTHP_STAT_ANON_ALLOC,
 	MTHP_STAT_ANON_ALLOC_FALLBACK,
+	MTHP_STAT_ANON_SWPOUT,
+	MTHP_STAT_ANON_SWPOUT_FALLBACK,
 	__MTHP_STAT_COUNT
 };
 
mm/huge_memory.c +4 -0
index 5b875f0fc923..28113f8fdf18 100644
@@ -554,10 +554,14 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
 
 DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
 DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT);
+DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK);
 
 static struct attribute *stats_attrs[] = {
 	&anon_alloc_attr.attr,
 	&anon_alloc_fallback_attr.attr,
+	&anon_swpout_attr.attr,
+	&anon_swpout_fallback_attr.attr,
 	NULL,
 };
 
mm/page_io.c +0 -0
index a9a7c236aecc..7669452e8b4d 100644
@@ -212,13 +212,17 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
 
 static inline void count_swpout_vm_event(struct folio *folio)

mm/page_io.c int swap_writepage(struct page *page, struct writeback_control *wbc) · line 223

Quoted 10 lines
mm/page_io.c +4 -0
 {
+	long nr_pages = folio_nr_pages(folio);
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	if (unlikely(folio_test_pmd_mappable(folio))) {
 		count_memcg_folio_events(folio, THP_SWPOUT, 1);
 		count_vm_event(THP_SWPOUT);
 	}
+	if (nr_pages > 0)
+		count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_SWPOUT);

I think you mean "nr_pages > 1"? Although as established in my comments against patch 1, its safe to call for order-0, so could just unconditionally call this (as you effectively are at the moment).

Quoted 11 lines
mm/page_io.c +1 -1
 #endif
-	count_vm_events(PSWPOUT, folio_nr_pages(folio));
+	count_vm_events(PSWPOUT, nr_pages);
 }
 
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
mm/vmscan.c +0 -0
index ffc4553c8615..b30e6294f82a 100644

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1252

Quoted 6 lines
mm/vmscan.c +3 -0
@@ -1247,6 +1247,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 						count_vm_event(
 							THP_SWPOUT_FALLBACK);
 					}
+					if (nr_pages > 0)
+						count_mthp_stat(get_order(nr_pages * PAGE_SIZE),
+							MTHP_STAT_ANON_SWPOUT_FALLBACK);

Same comment.

Quoted 3 lines
mm/vmscan.c +0 -0
 #endif
 					if (!add_to_swap(folio))
 						goto activate_locked_split;
author
#8 lore raw

Re: [PATCH v4 2/2] mm: add per-order mTHP anon_swpout and anon_swpout_fallback counters

Quoted 56 lines · Ryan Roberts
Quoted 54 lines · Barry Song

From: Barry Song <[email protected]>

This helps to display the fragmentation situation of the swapfile, knowing the proportion of how much we haven't split large folios. So far, we only support non-split swapout for anon memory, with the possibility of expanding to shmem in the future. So, we add the "anon" prefix to the counter names.

---
 include/linux/huge_mm.h | 2 ++
 mm/huge_memory.c        | 4 ++++
 mm/page_io.c            | 6 +++++-
 mm/vmscan.c             | 3 +++
 4 files changed, 14 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +2 -0
index c5d33017a4dd..1d893a358df6 100644
@@ -267,6 +267,8 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 enum mthp_stat_item {
      MTHP_STAT_ANON_ALLOC,
      MTHP_STAT_ANON_ALLOC_FALLBACK,
+     MTHP_STAT_ANON_SWPOUT,
+     MTHP_STAT_ANON_SWPOUT_FALLBACK,
      __MTHP_STAT_COUNT
 };
 
mm/huge_memory.c +4 -0
index 5b875f0fc923..28113f8fdf18 100644
@@ -554,10 +554,14 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
 
 DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC);
 DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK);
+DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT);
+DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK);
 
 static struct attribute *stats_attrs[] = {
      &anon_alloc_attr.attr,
      &anon_alloc_fallback_attr.attr,
+     &anon_swpout_attr.attr,
+     &anon_swpout_fallback_attr.attr,
      NULL,
 };
 
mm/page_io.c +0 -0
index a9a7c236aecc..7669452e8b4d 100644
@@ -212,13 +212,17 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
 
Quoted 15 lines

mm/page_io.c int swap_writepage(struct page *page, struct writeback_control *wbc) · line 223

Quoted 11 lines
mm/page_io.c +4 -0
 static inline void count_swpout_vm_event(struct folio *folio)
 {
+     long nr_pages = folio_nr_pages(folio);
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
      if (unlikely(folio_test_pmd_mappable(folio))) {
              count_memcg_folio_events(folio, THP_SWPOUT, 1);
              count_vm_event(THP_SWPOUT);
      }
+     if (nr_pages > 0)
+             count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_SWPOUT);

I think you mean "nr_pages > 1"? Although as established in my comments against patch 1, its safe to call for order-0, so could just unconditionally call this (as you effectively are at the moment).

yes. nr_pages > 1 is the correct condition. though it doesn't cause any crash for
nr_pages > 0 as we have memory for order-0. but it is a useless job to count for nr_pages==1 as there is no place to show it in sysfs.

Quoted 6 lines
Quoted 6 lines
mm/page_io.c +1 -1
 #endif
-     count_vm_events(PSWPOUT, folio_nr_pages(folio));
+     count_vm_events(PSWPOUT, nr_pages);
 }
 
Quoted 14 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1252

Quoted 12 lines
mm/page_io.c +0 -0
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
mm/vmscan.c +3 -0
index ffc4553c8615..b30e6294f82a 100644
@@ -1247,6 +1247,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
                                              count_vm_event(
                                                      THP_SWPOUT_FALLBACK);
                                      }
+                                     if (nr_pages > 0)
+                                             count_mthp_stat(get_order(nr_pages * PAGE_SIZE),
+                                                     MTHP_STAT_ANON_SWPOUT_FALLBACK);

Same comment.

agreed. Also, it doesn't cause any crash as we have memory in mthp_stats for order-0.
i will totally remove if (nr_pages > 0) and as we are certainly in large folios cases.

Quoted 5 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1255

Quoted 4 lines
mm/vmscan.c +0 -0
 #endif
                                      if (!add_to_swap(folio))
                                              goto activate_locked_split;

Thanks
Barry

maintainer
#9 lore raw
In reply to Barry Song
Quoted 9 lines · On Fri, 5 Apr 2024 23:27:02 +1300 Barry Song <[email protected]> wrote:

The patchset introduces a framework to facilitate mTHP counters, starting with the allocation and swap-out counters. Currently, only four new nodes are appended to the stats directory for each mTHP size.

/sys/kernel/mm/transparent_hugepage/hugepages-<size>/stats
anon_alloc
anon_alloc_fallback
anon_swpout
anon_swpout_fallback

Please let's get these documented? Documentation/admin-guide/mm/transhuge.rst seems to be the place, and under Documentation/ABI.

author
#10 lore raw
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Fri, 5 Apr 2024 23:27:02 +1300 Barry Song <[email protected]> wrote:

The patchset introduces a framework to facilitate mTHP counters, starting with the allocation and swap-out counters. Currently, only four new nodes are appended to the stats directory for each mTHP size.

Quoted 7 lines
Quoted 5 lines

/sys/kernel/mm/transparent_hugepage/hugepages-<size>/stats
anon_alloc
anon_alloc_fallback
anon_swpout
anon_swpout_fallback

Please let's get these documented?

Sure, I will do it.

Furthermore, after careful consideration, I suggest renaming anon_alloc to anon_fault_alloc and anon_alloc_fallback to anon_fault_fallback. This renaming is consistent with the naming convention observed in thp_fault_alloc and thp_fault_fallback in vmstat. Additionally, this modification will facilitate the handling of large folio swap-ins.

/sys/kernel/mm/transparent_hugepage/hugepages-<size>/stats
anon_fault_alloc,
anon_fault_fallback,
anon_swpin_alloc,
anon_swpin_fallback,

Otherwise, without this renaming, it would be difficult to differentiate fault_alloc
from swpin_alloc.

Hi David, Ryan,
I would also appreciate your input on the modification I am considering.

Quoted 2 lines

Documentation/admin-guide/mm/transhuge.rst seems to be the place, and under Documentation/ABI.

Thanks
Barry

reviewer
#11 lore raw
Quoted 30 lines · Barry Song
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Fri, 5 Apr 2024 23:27:02 +1300 Barry Song <[email protected]> wrote:

The patchset introduces a framework to facilitate mTHP counters, starting with the allocation and swap-out counters. Currently, only four new nodes are appended to the stats directory for each mTHP size.

Quoted 7 lines
Quoted 5 lines

/sys/kernel/mm/transparent_hugepage/hugepages-<size>/stats
anon_alloc
anon_alloc_fallback
anon_swpout
anon_swpout_fallback

Please let's get these documented?

Sure, I will do it.

Furthermore, after careful consideration, I suggest renaming anon_alloc to anon_fault_alloc and anon_alloc_fallback to anon_fault_fallback. This renaming is consistent with the naming convention observed in thp_fault_alloc and thp_fault_fallback in vmstat. Additionally, this modification will facilitate the handling of large folio swap-ins.

/sys/kernel/mm/transparent_hugepage/hugepages-<size>/stats
anon_fault_alloc,
anon_fault_fallback,
anon_swpin_alloc,
anon_swpin_fallback,

Quoted 6 lines

Otherwise, without this renaming, it would be difficult to differentiate fault_alloc
from swpin_alloc.

Hi David, Ryan,
I would also appreciate your input on the modification I am considering.

Yes, this makes sense to me!

Quoted 6 lines
Quoted 3 lines

Documentation/admin-guide/mm/transhuge.rst seems to be the place, and under Documentation/ABI.

Thanks
Barry