Skip to contentSkip to conversation

hugetlb: Add hugetlb.*.numa_stat file

PATCH Superseded Mina Almasry 3 participants 4 messages Nov 17, 2021 – Nov 18, 2021

Subsystems (4)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v7 1/1 hugetlb: Add hugetlb.*.numa_stat file

For hugetlb backed jobs/VMs it's critical to understand the numa information for the memory backing these jobs to deliver optimal performance.

Currently this technically can be queried from /proc/self/numa_maps, but there are significant issues with that. Namely:
1. Memory can be mapped or unmapped.
2. numa_maps are per process and need to be aggregated across all processes in the cgroup. For shared memory this is more involved as the userspace needs to make sure it doesn't double count shared mappings.
3. I believe querying numa_maps needs to hold the mmap_lock which adds to the contention on this lock.

For these reasons I propose simply adding hugetlb.*.numa_stat file, which shows the numa information of the cgroup similarly to memory.numa_stat.

On cgroup-v2:
cat /sys/fs/cgroup/unified/test/hugetlb.2MB.numa_stat total=2097152 N0=2097152 N1=0

On cgroup-v1:
cat /sys/fs/cgroup/hugetlb/test/hugetlb.2MB.numa_stat total=2097152 N0=2097152 N1=0 hierarichal_total=2097152 N0=2097152 N1=0

This patch was tested manually by allocating hugetlb memory and querying the hugetlb.*.numa_stat file of the cgroup and its parents.

Cc: Mike Kravetz <[email protected]>
Cc: Andrew Morton <akpm-de/[email protected]>
Cc: Shuah Khan <[email protected]>
Cc: Miaohe Lin <[email protected]>
Cc: Oscar Salvador <[email protected]>
Cc: Michal Hocko <mhocko-IBi9RG/[email protected]>
Cc: Muchun Song <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: Shakeel Butt <[email protected]>
Cc: Jue Wang <[email protected]>
Cc: Yang Yao <[email protected]>
Cc: Joanna Li <[email protected]>
Cc: Cannon Matthews <[email protected]>
Cc: [email protected]
Cc: [email protected]

---

Changes in v7:
- Converted back usage to unsigned long + READ_ONCE/WRITE_ONCE rather than atomic_long_t

Changes in v6:
- Changed usage from unsigned long to atomic_long_t

Changes in v5:
- Fixed commit message typo.
- Fixed per node usage documentation to be in pages.
- Removed unnecessary h_cg check.

Changes in v4:
- Removed unnecessary braces.
- usage is now counted in pages instead of bytes.
- Reverted unneeded changes to write_to_hugetlbfs.c

Changes in v3:
- Fixed typos (sorry!)
- Used conventional locations for cgroups mount points in docs/commit message.
- Updated docs.
- Handle kzalloc_node failure, and proper deallocation of per node data.
- Use struct_size() to calculate the struct size.
- Use nr_node_ids instead of MAX_NUMNODES.
- Updated comments per multi-line comment pattern.

Changes in v2:
- Fix warning Reported-by: kernel test robot <[email protected]>

---
 .../admin-guide/cgroup-v1/hugetlb.rst         |   4 +
 Documentation/admin-guide/cgroup-v2.rst       |   5 +
 include/linux/hugetlb.h                       |   4 +-
 include/linux/hugetlb_cgroup.h                |   7 +
 mm/hugetlb_cgroup.c                           | 132 ++++++++++++++++--
 5 files changed, 140 insertions(+), 12 deletions(-)
Documentation/admin-guide/cgroup-v1/hugetlb.rst +4 -0
index 338f2c7d7a1c..0fa724d82abb 100644
@@ -29,12 +29,14 @@ Brief summary of control files::
  hugetlb.<hugepagesize>.max_usage_in_bytes             # show max "hugepagesize" hugetlb  usage recorded
  hugetlb.<hugepagesize>.usage_in_bytes                 # show current usage for "hugepagesize" hugetlb
  hugetlb.<hugepagesize>.failcnt                        # show the number of allocation failure due to HugeTLB usage limit
+ hugetlb.<hugepagesize>.numa_stat                      # show the numa information of the hugetlb memory charged to this cgroup
 
 For a system supporting three hugepage sizes (64k, 32M and 1G), the control
 files include::
 
   hugetlb.1GB.limit_in_bytes
   hugetlb.1GB.max_usage_in_bytes
+  hugetlb.1GB.numa_stat
   hugetlb.1GB.usage_in_bytes
   hugetlb.1GB.failcnt
   hugetlb.1GB.rsvd.limit_in_bytes
@@ -43,6 +45,7 @@ files include::
   hugetlb.1GB.rsvd.failcnt
   hugetlb.64KB.limit_in_bytes
   hugetlb.64KB.max_usage_in_bytes
+  hugetlb.64KB.numa_stat
   hugetlb.64KB.usage_in_bytes
   hugetlb.64KB.failcnt
   hugetlb.64KB.rsvd.limit_in_bytes
@@ -51,6 +54,7 @@ files include::
   hugetlb.64KB.rsvd.failcnt
   hugetlb.32MB.limit_in_bytes
   hugetlb.32MB.max_usage_in_bytes
+  hugetlb.32MB.numa_stat
   hugetlb.32MB.usage_in_bytes
   hugetlb.32MB.failcnt
   hugetlb.32MB.rsvd.limit_in_bytes
Documentation/admin-guide/cgroup-v2.rst +5 -0
index 4d8c27eca96b..356847f8f008 100644
@@ -2252,6 +2252,11 @@ HugeTLB Interface Files
 	are local to the cgroup i.e. not hierarchical. The file modified event
 	generated on this file reflects only the local events.
 
+  hugetlb.<hugepagesize>.numa_stat
+	Similar to memory.numa_stat, it shows the numa information of the
+        hugetlb pages of <hugepagesize> in this cgroup.  Only active in
+        use hugetlb pages are included.  The per-node values are in bytes.
+
 Misc
 ----
 
include/linux/hugetlb.h +2 -2
index 1faebe1cd0ed..0445faaa636e 100644
@@ -613,8 +613,8 @@ struct hstate {
 #endif
 #ifdef CONFIG_CGROUP_HUGETLB
 	/* cgroup control files */
-	struct cftype cgroup_files_dfl[7];
-	struct cftype cgroup_files_legacy[9];
+	struct cftype cgroup_files_dfl[8];
+	struct cftype cgroup_files_legacy[10];
 #endif
 	char name[HSTATE_NAME_LEN];
 };
include/linux/hugetlb_cgroup.h +7 -0
index c137396129db..0f6cd28558d7 100644
@@ -36,6 +36,11 @@ enum hugetlb_memory_event {
 	HUGETLB_NR_MEMORY_EVENTS,
 };
 
+struct hugetlb_cgroup_per_node {
+	/* hugetlb usage in pages over all hstates. */
+	unsigned long usage[HUGE_MAX_HSTATE];
+};
+
 struct hugetlb_cgroup {
 	struct cgroup_subsys_state css;
 
@@ -57,6 +62,8 @@ struct hugetlb_cgroup {
 
 	/* Handle for "hugetlb.events.local" */
 	struct cgroup_file events_local_file[HUGE_MAX_HSTATE];
+
+	struct hugetlb_cgroup_per_node *nodeinfo[];
 };
 
 static inline struct hugetlb_cgroup *
mm/hugetlb_cgroup.c +122 -10

198 lines not on this page · raw message

#1 lore raw

On 11/17/21 12:18, Mina Almasry wrote:
...

Quoted 1 line
mm/hugetlb_cgroup.c +0 -0

...

mm/hugetlb_cgroup.c static void __hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages, · line 321

Quoted 6 lines
mm/hugetlb_cgroup.c +2 -0
@@ -288,11 +317,21 @@ static void __hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages,
 					   struct hugetlb_cgroup *h_cg,
 					   struct page *page, bool rsvd)
 {
+	unsigned long *usage;
+

I assume the use of a pointer is just to make the following WRITE_ONCE look better? I prefer the suggestion by Muchun:

unsigned long usage = h_cg->nodeinfo[page_to_nid(page)]->usage[idx];

usage += nr_pages; WRITE_ONCE(h_cg->nodeinfo[page_to_nid(page)]->usage[idx], usage);

I had to think for just a second 'why are we using/passing a pointer?'. Not insisting we use Muchun's suggestion, it just caused me to think a little more than necessary.

In any case, I would move the variable usage inside the 'if (!rsvd)' block.

Quoted 18 lines
mm/hugetlb_cgroup.c +9 -1
 	if (hugetlb_cgroup_disabled() || !h_cg)
 		return;
 
 	__set_hugetlb_cgroup(page, h_cg, rsvd);
-	return;
+	if (!rsvd) {
+		usage = &h_cg->nodeinfo[page_to_nid(page)]->usage[idx];
+		/*
+		 * This write is not atomic due to fetching *usage and writing
+		 * to it, but that's fine because we call this with
+		 * hugetlb_lock held anyway.
+		 */
+		WRITE_ONCE(*usage, *usage + nr_pages);
+	}
 }
 
 void hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages,

mm/hugetlb_cgroup.c static void __hugetlb_cgroup_uncharge_page(int idx, unsigned long nr_pages, · line 358

Quoted 4 lines
mm/hugetlb_cgroup.c +1 -0
@@ -316,6 +355,7 @@ static void __hugetlb_cgroup_uncharge_page(int idx, unsigned long nr_pages,
 					   struct page *page, bool rsvd)
 {
 	struct hugetlb_cgroup *h_cg;
+	unsigned long *usage;

Same here.

Otherwise, looks good to me.

Signature · 1 line
Mike Kravetz
#2 lore raw
Quoted 15 lines · Mike Kravetz

On 11/17/21 12:18, Mina Almasry wrote:
...

Quoted 1 line
mm/hugetlb_cgroup.c +0 -0

...

mm/hugetlb_cgroup.c static void __hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages, · line 321

Quoted 6 lines
mm/hugetlb_cgroup.c +2 -0
@@ -288,11 +317,21 @@ static void __hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages,
                                         struct hugetlb_cgroup *h_cg,
                                         struct page *page, bool rsvd)
 {
+     unsigned long *usage;
+

I assume the use of a pointer is just to make the following WRITE_ONCE look better? I prefer the suggestion by Muchun:

Quoted 8 lines

unsigned long usage = h_cg->nodeinfo[page_to_nid(page)]->usage[idx];

usage += nr_pages; WRITE_ONCE(h_cg->nodeinfo[page_to_nid(page)]->usage[idx], usage);

I had to think for just a second 'why are we using/passing a pointer?'. Not insisting we use Muchun's suggestion, it just caused me to think a little more than necessary.

At least I have the same question here. For me I think it's unnecessary to use a pointer.

Quoted 32 lines

In any case, I would move the variable usage inside the 'if (!rsvd)' block.

Quoted 18 lines
mm/hugetlb_cgroup.c +9 -1
      if (hugetlb_cgroup_disabled() || !h_cg)
              return;
 
      __set_hugetlb_cgroup(page, h_cg, rsvd);
-     return;
+     if (!rsvd) {
+             usage = &h_cg->nodeinfo[page_to_nid(page)]->usage[idx];
+             /*
+              * This write is not atomic due to fetching *usage and writing
+              * to it, but that's fine because we call this with
+              * hugetlb_lock held anyway.
+              */
+             WRITE_ONCE(*usage, *usage + nr_pages);
+     }
 }
 
 void hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages,

mm/hugetlb_cgroup.c static void __hugetlb_cgroup_uncharge_page(int idx, unsigned long nr_pages, · line 358

Quoted 4 lines
mm/hugetlb_cgroup.c +1 -0
@@ -316,6 +355,7 @@ static void __hugetlb_cgroup_uncharge_page(int idx, unsigned long nr_pages,
                                         struct page *page, bool rsvd)
 {
      struct hugetlb_cgroup *h_cg;
+     unsigned long *usage;

Same here.

Otherwise, looks good to me.
--
Mike Kravetz

#3 lore raw
Quoted 22 lines · Muchun Song
Quoted 20 lines · Mike Kravetz

On 11/17/21 12:18, Mina Almasry wrote:
...

Quoted 1 line
mm/hugetlb_cgroup.c +0 -0

...

mm/hugetlb_cgroup.c static void __hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages, · line 321

Quoted 6 lines
mm/hugetlb_cgroup.c +2 -0
@@ -288,11 +317,21 @@ static void __hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages,
                                         struct hugetlb_cgroup *h_cg,
                                         struct page *page, bool rsvd)
 {
+     unsigned long *usage;
+

I assume the use of a pointer is just to make the following WRITE_ONCE look better? I prefer the suggestion by Muchun:

unsigned long usage = h_cg->nodeinfo[page_to_nid(page)]->usage[idx];

usage += nr_pages; WRITE_ONCE(h_cg->nodeinfo[page_to_nid(page)]->usage[idx], usage);

Quoted 7 lines
Quoted 3 lines

I had to think for just a second 'why are we using/passing a pointer?'. Not insisting we use Muchun's suggestion, it just caused me to think a little more than necessary.

At least I have the same question here. For me I think it's unnecessary to use a pointer.

Hmm to be honest I would have not thought it would be preferable to duplicate a long string like h_cg->nodeinfo[page_to_nid(page)]->usage[idx], and then for future code changes to keep them in sync. I think Marco had the same thinking and that was his initial suggestion, but I don't mind much either way. I'll submit another iteration with the change :-)

Quoted 32 lines
Quoted 32 lines

In any case, I would move the variable usage inside the 'if (!rsvd)' block.

Quoted 18 lines
mm/hugetlb_cgroup.c +9 -1
      if (hugetlb_cgroup_disabled() || !h_cg)
              return;
 
      __set_hugetlb_cgroup(page, h_cg, rsvd);
-     return;
+     if (!rsvd) {
+             usage = &h_cg->nodeinfo[page_to_nid(page)]->usage[idx];
+             /*
+              * This write is not atomic due to fetching *usage and writing
+              * to it, but that's fine because we call this with
+              * hugetlb_lock held anyway.
+              */
+             WRITE_ONCE(*usage, *usage + nr_pages);
+     }
 }
 
 void hugetlb_cgroup_commit_charge(int idx, unsigned long nr_pages,

mm/hugetlb_cgroup.c static void __hugetlb_cgroup_uncharge_page(int idx, unsigned long nr_pages, · line 358

Quoted 4 lines
mm/hugetlb_cgroup.c +1 -0
@@ -316,6 +355,7 @@ static void __hugetlb_cgroup_uncharge_page(int idx, unsigned long nr_pages,
                                         struct page *page, bool rsvd)
 {
      struct hugetlb_cgroup *h_cg;
+     unsigned long *usage;

Same here.

Otherwise, looks good to me.
--
Mike Kravetz