mm: show total hugetlb memory consumption in /proc/meminfo

PATCH Applied Roman Gushchin 6 participants 12 messages Nov 15, 2017 – Nov 22, 2017

Subsystems (3)

Memory management Documentation HugeTLB

lore

Share & tools

lore

Roman Gushchin

author

2017-11-15 23:14 UTC #0 lore raw

PATCH v2 1/1 mm: show total hugetlb memory consumption in /proc/meminfo

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).

If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.

To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.

For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):

  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB

Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.

Signed-off-by Roman Gushchin <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Dave Hansen <[email protected]>

Cc David Rientjes <[email protected]>

Cc [email protected]

---
 Documentation/vm/hugetlbpage.txt | 27 ++++++++++++++++++---------
 mm/hugetlb.c                     | 36 ++++++++++++++++++++++++------------
 2 files changed, 42 insertions(+), 21 deletions(-)

Documentation/vm/hugetlbpage.txt +18 -9

index 59cbc803aad6..faf077d50d42 100644
@@ -20,19 +20,20 @@ options.
 
 The /proc/meminfo file provides information about the total number of
 persistent hugetlb pages in the kernel's huge page pool.  It also displays
-information about the number of free, reserved and surplus huge pages and the
-default huge page size.  The huge page size is needed for generating the
-proper alignment and size of the arguments to system calls that map huge page
-regions.
+default huge page size and information about the number of free, reserved
+and surplus huge pages in the pool of huge pages of default size.
+The huge page size is needed for generating the proper alignment and
+size of the arguments to system calls that map huge page regions.
 
 The output of "cat /proc/meminfo" will include lines like:
 
 .....
-HugePages_Total: vvv
-HugePages_Free:  www
-HugePages_Rsvd:  xxx
-HugePages_Surp:  yyy
-Hugepagesize:    zzz kB
+HugePages_Total: uuu
+HugePages_Free:  vvv
+HugePages_Rsvd:  www
+HugePages_Surp:  xxx
+Hugepagesize:    yyy kB
+Hugetlb:         zzz kB
 
 where:
 HugePages_Total is the size of the pool of huge pages.

@@ -47,6 +48,14 @@ HugePages_Surp  is short for "surplus," and is the number of huge pages in
                 the pool above the value in /proc/sys/vm/nr_hugepages. The
                 maximum number of surplus huge pages is controlled by
                 /proc/sys/vm/nr_overcommit_hugepages.
+Hugepagesize    is the default hugepage size (in Kb).
+Hugetlb         is the total amount of memory (in kB), consumed by huge
+                pages of all sizes.
+                If huge pages of different sizes are in use, this number
+                will exceed HugePages_Total * Hugepagesize. To get more
+                detailed information, please, refer to
+                /sys/kernel/mm/hugepages (described below).
+
 
 /proc/filesystems should also show a filesystem of type "hugetlbfs" configured
 in the kernel.

mm/hugetlb.c +24 ~~-12~~

index 4b3bbd2980bb..672377e6de9f 100644
@@ -2973,20 +2973,32 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+	unsigned long total = 0;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,
-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
+			seq_printf(m,
+				   "HugePages_Total:   %5lu\n"
+				   "HugePages_Free:    %5lu\n"
+				   "HugePages_Rsvd:    %5lu\n"
+				   "HugePages_Surp:    %5lu\n"
+				   "Hugepagesize:   %8lu kB\n",
+				   count,
+				   h->free_huge_pages,
+				   h->resv_huge_pages,
+				   h->surplus_huge_pages,
+				   (PAGE_SIZE << huge_page_order(h)) / 1024);
+	}
+
+	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);
 }
 
 int hugetlb_report_node_meminfo(int nid, char *buf)

Signature · 7 lines

2.13.6

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

Acked-by on v2 1/1

2017-11-16 09:11 UTC #1 lore raw

Quoted 33 lines · Roman Gushchin

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.

Quoted 12 lines

Signed-off-by Roman Gushchin <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dave Hansen <[email protected]>
Cc David Rientjes <[email protected]>
Cc [email protected]
Cc [email protected]
Cc [email protected]

Acked-by Michal Hocko <[email protected]>

Thanks!

Quoted 106 lines

---
 Documentation/vm/hugetlbpage.txt | 27 ++++++++++++++++++---------
 mm/hugetlb.c                     | 36 ++++++++++++++++++++++++------------
 2 files changed, 42 insertions(+), 21 deletions(-)

Documentation/vm/hugetlbpage.txt +18 -9

index 59cbc803aad6..faf077d50d42 100644
@@ -20,19 +20,20 @@ options.
 
 The /proc/meminfo file provides information about the total number of
 persistent hugetlb pages in the kernel's huge page pool.  It also displays
-information about the number of free, reserved and surplus huge pages and the
-default huge page size.  The huge page size is needed for generating the
-proper alignment and size of the arguments to system calls that map huge page
-regions.
+default huge page size and information about the number of free, reserved
+and surplus huge pages in the pool of huge pages of default size.
+The huge page size is needed for generating the proper alignment and
+size of the arguments to system calls that map huge page regions.
 
 The output of "cat /proc/meminfo" will include lines like:
 
.....
-HugePages_Total: vvv
-HugePages_Free:  www
-HugePages_Rsvd:  xxx
-HugePages_Surp:  yyy
-Hugepagesize:    zzz kB
+HugePages_Total: uuu
+HugePages_Free:  vvv
+HugePages_Rsvd:  www
+HugePages_Surp:  xxx
+Hugepagesize:    yyy kB
+Hugetlb:         zzz kB
 
 where:
 HugePages_Total is the size of the pool of huge pages.

@@ -47,6 +48,14 @@ HugePages_Surp  is short for "surplus," and is the number of huge pages in
                 the pool above the value in /proc/sys/vm/nr_hugepages. The
                 maximum number of surplus huge pages is controlled by
                 /proc/sys/vm/nr_overcommit_hugepages.
+Hugepagesize    is the default hugepage size (in Kb).
+Hugetlb         is the total amount of memory (in kB), consumed by huge
+                pages of all sizes.
+                If huge pages of different sizes are in use, this number
+                will exceed HugePages_Total * Hugepagesize. To get more
+                detailed information, please, refer to
+                /sys/kernel/mm/hugepages (described below).
+
 
 /proc/filesystems should also show a filesystem of type "hugetlbfs" configured
 in the kernel.

mm/hugetlb.c +24 ~~-12~~

index 4b3bbd2980bb..672377e6de9f 100644
@@ -2973,20 +2973,32 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+	unsigned long total = 0;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,
-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
+			seq_printf(m,
+				   "HugePages_Total:   %5lu\n"
+				   "HugePages_Free:    %5lu\n"
+				   "HugePages_Rsvd:    %5lu\n"
+				   "HugePages_Surp:    %5lu\n"
+				   "Hugepagesize:   %8lu kB\n",
+				   count,
+				   h->free_huge_pages,
+				   h->resv_huge_pages,
+				   h->surplus_huge_pages,
+				   (PAGE_SIZE << huge_page_order(h)) / 1024);
+	}
+
+	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);
 }
 
 int hugetlb_report_node_meminfo(int nid, char *buf)

Signature · 1 line

2.13.6

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Johannes Weiner

Acked-by on v2 1/1

2017-11-16 14:22 UTC #2 lore raw

In reply to Roman Gushchin

Quoted 33 lines · Roman Gushchin

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.

Quoted 12 lines

Signed-off-by Roman Gushchin <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dave Hansen <[email protected]>
Cc David Rientjes <[email protected]>
Cc [email protected]
Cc [email protected]
Cc [email protected]

Acked-by Johannes Weiner <[email protected]>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

Acked-by on v2 1/1

2017-11-17 09:41 UTC #3 lore raw

In reply to Roman Gushchin

Quoted 33 lines · Roman Gushchin

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.

Quoted 12 lines

Signed-off-by Roman Gushchin <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dave Hansen <[email protected]>
Cc David Rientjes <[email protected]>
Cc [email protected]
Cc [email protected]
Cc [email protected]

Acked-by David Rientjes <[email protected]>

Nice!

Andrew Morton

maintainer

2017-11-21 00:51 UTC #4 lore raw

In reply to Roman Gushchin

Quoted 55 lines · On Wed, 15 Nov 2017 23:14:09 +0000 Roman Gushchin <[email protected]> wrote:

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.
...
mm/hugetlb.c +3 -9
@@ -2973,20 +2973,32 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+	unsigned long total = 0;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,

mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 2987

Quoted 10 lines

mm/hugetlb.c +7 -3

-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)

I'm not understanding this test. Are we assuming that default_hstate always refers to the highest-index hstate? If so why, and is that valid?

Quoted 15 lines

mm/hugetlb.c +14 -0

+			seq_printf(m,
+				   "HugePages_Total:   %5lu\n"
+				   "HugePages_Free:    %5lu\n"
+				   "HugePages_Rsvd:    %5lu\n"
+				   "HugePages_Surp:    %5lu\n"
+				   "Hugepagesize:   %8lu kB\n",
+				   count,
+				   h->free_huge_pages,
+				   h->resv_huge_pages,
+				   h->surplus_huge_pages,
+				   (PAGE_SIZE << huge_page_order(h)) / 1024);
+	}
+
+	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);
 }

Mike Kravetz

2017-11-21 01:09 UTC #5 lore raw

Quoted 63 lines · Andrew Morton

Quoted 61 lines · On Wed, 15 Nov 2017 23:14:09 +0000 Roman Gushchin <[email protected]> wrote:
Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.
...
mm/hugetlb.c +6 ~~-12~~
@@ -2973,20 +2973,32 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+	unsigned long total = 0;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,
-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;

Quoted 8 lines

mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 2987
Quoted 4 lines
mm/hugetlb.c +4 -0
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
I'm not understanding this test. Are we assuming that default_hstate always refers to the highest-index hstate? If so why, and is that valid?

Actually default_hstate is defined as:

#define default_hstate (hstates[default_hstate_idx])

default_hstate_idx is set during hugetlb_init based upon default_hstate_size which defaults to HPAGE_SIZE. However, it can be overridden by the kernel command line argument "default_hugepagesz=<size>".

By definition and history /proc/meminfo lists information on the default huge page size. This code is looping through all hstates to get the total memory consumed by hugetlb pages for the new "Hugetlb" field. When it gets to the default huge page size, it prints the historic fields.

Hope that helps,

Signature · 1 line

Mike Kravetz

Quoted 14 lines

mm/hugetlb.c +13 -0

+			seq_printf(m,
+				   "HugePages_Total:   %5lu\n"
+				   "HugePages_Free:    %5lu\n"
+				   "HugePages_Rsvd:    %5lu\n"
+				   "HugePages_Surp:    %5lu\n"
+				   "Hugepagesize:   %8lu kB\n",
+				   count,
+				   h->free_huge_pages,
+				   h->resv_huge_pages,
+				   h->surplus_huge_pages,
+				   (PAGE_SIZE << huge_page_order(h)) / 1024);
+	}
+

Quoted 9 lines

mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 3002
Quoted 2 lines
mm/hugetlb.c +1 -0
+	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);
 }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

2017-11-21 07:01 UTC #6 lore raw

In reply to Andrew Morton

Quoted 63 lines · Andrew Morton

Quoted 61 lines · On Wed, 15 Nov 2017 23:14:09 +0000 Roman Gushchin <[email protected]> wrote:
Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.
...
mm/hugetlb.c +6 ~~-12~~
@@ -2973,20 +2973,32 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+	unsigned long total = 0;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,
-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;

Quoted 8 lines

mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 2987
Quoted 4 lines
mm/hugetlb.c +4 -0
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
I'm not understanding this test. Are we assuming that default_hstate always refers to the highest-index hstate? If so why, and is that valid?

The whole point of this checks is to provide hugetlb detailed stats _only_ for the default hstate because that is what we have been doing traditionally. The loop is there only to gather total amount and display it separately.

Quoted 16 lines

Quoted 15 lines

mm/hugetlb.c +14 -0

+			seq_printf(m,
+				   "HugePages_Total:   %5lu\n"
+				   "HugePages_Free:    %5lu\n"
+				   "HugePages_Rsvd:    %5lu\n"
+				   "HugePages_Surp:    %5lu\n"
+				   "Hugepagesize:   %8lu kB\n",
+				   count,
+				   h->free_huge_pages,
+				   h->resv_huge_pages,
+				   h->surplus_huge_pages,
+				   (PAGE_SIZE << huge_page_order(h)) / 1024);
+	}
+
+	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);
 }

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Roman Gushchin

author

2017-11-21 15:16 UTC #7 lore raw

In reply to Andrew Morton

Quoted 63 lines · Andrew Morton

Quoted 61 lines · On Wed, 15 Nov 2017 23:14:09 +0000 Roman Gushchin <[email protected]> wrote:
Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.
To solve this problem, let's display the total amount of memory, consumed by hugetlb pages of all sized (both free and used). Let's call it "Hugetlb", and display size in kB to match generic /proc/meminfo style.
For example, (1024 2Mb pages and 2 1Gb pages are pre-allocated):
  $ cat /proc/meminfo
  MemTotal:        8168984 kB
  MemFree:         3789276 kB
  <...>
  CmaFree:               0 kB
  HugePages_Total:    1024
  HugePages_Free:     1024
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  Hugetlb:         4194304 kB
  DirectMap4k:       32632 kB
  DirectMap2M:     4161536 kB
  DirectMap1G:     6291456 kB
Also, this patch updates corresponding docs to reflect Hugetlb entry meaning and difference between Hugetlb and HugePages_Total * Hugepagesize.
...
mm/hugetlb.c +6 ~~-12~~
@@ -2973,20 +2973,32 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+	unsigned long total = 0;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,
-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;

Quoted 8 lines

mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 2987
Quoted 4 lines
mm/hugetlb.c +4 -0
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
I'm not understanding this test. Are we assuming that default_hstate always refers to the highest-index hstate? If so why, and is that valid?

As Mike and Michal pointed, default_hstate is defined as
#define default_hstate (hstates[default_hstate_idx]),
where default_hstate_idx can be altered by a boot argument.

We're iterating over all states to calculate total and also print some additional info for the default size. Having a single loop guarantees consistency of these numbers.

Thanks!

Andrew Morton

maintainer

2017-11-21 19:19 UTC #8 lore raw

Quoted 12 lines · On Tue, 21 Nov 2017 15:15:55 +0000 Roman Gushchin <[email protected]> wrote:

Quoted 11 lines
mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 2987
Quoted 7 lines
mm/hugetlb.c +7 -0
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
I'm not understanding this test. Are we assuming that default_hstate always refers to the highest-index hstate? If so why, and is that valid?

Quoted 8 lines

As Mike and Michal pointed, default_hstate is defined as
#define default_hstate (hstates[default_hstate_idx]),
where default_hstate_idx can be altered by a boot argument.
We're iterating over all states to calculate total and also print some additional info for the default size. Having a single loop guarantees consistency of these numbers.

OK, I misread the handling of `count' -> HugePages_Total.

It seems unnecessarily obscure?

	for_each_hstate(h) {
		unsigned long count = h->nr_huge_pages;

		total += (PAGE_SIZE << huge_page_order(h)) * count;

		if (h == &default_hstate)
			seq_printf(m,
				   "HugePages_Total:   %5lu\n"
				   "HugePages_Free:    %5lu\n"
				   "HugePages_Rsvd:    %5lu\n"
				   "HugePages_Surp:    %5lu\n"
				   "Hugepagesize:   %8lu kB\n",
				   count,
				   h->free_huge_pages,
				   h->resv_huge_pages,
				   h->surplus_huge_pages,
				   (PAGE_SIZE << huge_page_order(h)) / 1024);
	}

	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

Why not

	seq_printf(m,
			"HugePages_Total:   %5lu\n"
			"HugePages_Free:    %5lu\n"
			"HugePages_Rsvd:    %5lu\n"
			"HugePages_Surp:    %5lu\n"
			"Hugepagesize:   %8lu kB\n",
			h->nr_huge_pages,
			h->free_huge_pages,
			h->resv_huge_pages,
			h->surplus_huge_pages,
			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));

	for_each_hstate(h)
		total += (PAGE_SIZE << huge_page_order(h)) * h->nr_huge_pages;
	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

Roman Gushchin

author

2017-11-21 20:00 UTC #9 lore raw

Quoted 67 lines · Andrew Morton

Quoted 12 lines · On Tue, 21 Nov 2017 15:15:55 +0000 Roman Gushchin <[email protected]> wrote:
Quoted 11 lines
mm/hugetlb.c int hugetlb_overcommit_handler(struct ctl_table *table, int write, · line 2987
Quoted 7 lines
mm/hugetlb.c +7 -0
+
+	for_each_hstate(h) {
+		unsigned long count = h->nr_huge_pages;
+
+		total += (PAGE_SIZE << huge_page_order(h)) * count;
+
+		if (h == &default_hstate)
I'm not understanding this test. Are we assuming that default_hstate always refers to the highest-index hstate? If so why, and is that valid?
Quoted 8 lines
As Mike and Michal pointed, default_hstate is defined as
#define default_hstate (hstates[default_hstate_idx]),
where default_hstate_idx can be altered by a boot argument.
We're iterating over all states to calculate total and also print some additional info for the default size. Having a single loop guarantees consistency of these numbers.
OK, I misread the handling of `count' -> HugePages_Total.
It seems unnecessarily obscure?
	for_each_hstate(h) {
		unsigned long count = h->nr_huge_pages;

		total += (PAGE_SIZE << huge_page_order(h)) * count;

		if (h == &default_hstate)
			seq_printf(m,
				   "HugePages_Total:   %5lu\n"
				   "HugePages_Free:    %5lu\n"
				   "HugePages_Rsvd:    %5lu\n"
				   "HugePages_Surp:    %5lu\n"
				   "Hugepagesize:   %8lu kB\n",
				   count,
				   h->free_huge_pages,
				   h->resv_huge_pages,
				   h->surplus_huge_pages,
				   (PAGE_SIZE << huge_page_order(h)) / 1024);
	}

	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);
Why not
	seq_printf(m,
			"HugePages_Total:   %5lu\n"
			"HugePages_Free:    %5lu\n"
			"HugePages_Rsvd:    %5lu\n"
			"HugePages_Surp:    %5lu\n"
			"Hugepagesize:   %8lu kB\n",
			h->nr_huge_pages,
			h->free_huge_pages,
			h->resv_huge_pages,
			h->surplus_huge_pages,
			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));

	for_each_hstate(h)
		total += (PAGE_SIZE << huge_page_order(h)) * h->nr_huge_pages;
	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

Quoted 1 line

?

The idea was that the local variable guarantees the consistency between Hugetlb and HugePages_Total numbers. Otherwise we have to take hugetlb_lock.

What we can do, is to rename "count" into "nr_huge_pages", like:

	for_each_hstate(h) {
		unsigned long nr_huge_pages = h->nr_huge_pages;

		total += (PAGE_SIZE << huge_page_order(h)) * nr_huge_pages;

		if (h == &default_hstate)
			seq_printf(m,
				   "HugePages_Total:   %5lu\n"
				   "HugePages_Free:    %5lu\n"
				   "HugePages_Rsvd:    %5lu\n"
				   "HugePages_Surp:    %5lu\n"
				   "Hugepagesize:   %8lu kB\n",
				   nr_huge_pages,
				   h->free_huge_pages,
				   h->resv_huge_pages,
				   h->surplus_huge_pages,
				   (PAGE_SIZE << huge_page_order(h)) / 1024);
	}

	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

But maybe taking a lock is not a bad idea, because it will also guarantee consistency between other numbers (like HugePages_Free) as well, which is not true right now.

Thanks!

Mike Kravetz

2017-11-22 00:27 UTC #10 lore raw

Quoted 20 lines · Roman Gushchin

Quoted 19 lines · Andrew Morton

Why not

	seq_printf(m,
			"HugePages_Total:   %5lu\n"
			"HugePages_Free:    %5lu\n"
			"HugePages_Rsvd:    %5lu\n"
			"HugePages_Surp:    %5lu\n"
			"Hugepagesize:   %8lu kB\n",
			h->nr_huge_pages,
			h->free_huge_pages,
			h->resv_huge_pages,
			h->surplus_huge_pages,
			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));

	for_each_hstate(h)
		total += (PAGE_SIZE << huge_page_order(h)) * h->nr_huge_pages;
	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

Quoted 5 lines

Quoted 1 line
?
The idea was that the local variable guarantees the consistency between Hugetlb and HugePages_Total numbers. Otherwise we have to take hugetlb_lock.

Most important it prevents HugePages_Total from being larger than Hugetlb.

Quoted 23 lines

What we can do, is to rename "count" into "nr_huge_pages", like:

	for_each_hstate(h) {
		unsigned long nr_huge_pages = h->nr_huge_pages;

		total += (PAGE_SIZE << huge_page_order(h)) * nr_huge_pages;

		if (h == &default_hstate)
			seq_printf(m,
				   "HugePages_Total:   %5lu\n"
				   "HugePages_Free:    %5lu\n"
				   "HugePages_Rsvd:    %5lu\n"
				   "HugePages_Surp:    %5lu\n"
				   "Hugepagesize:   %8lu kB\n",
				   nr_huge_pages,
				   h->free_huge_pages,
				   h->resv_huge_pages,
				   h->surplus_huge_pages,
				   (PAGE_SIZE << huge_page_order(h)) / 1024);
	}

	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

Quoted 3 lines

But maybe taking a lock is not a bad idea, because it will also guarantee consistency between other numbers (like HugePages_Free) as well, which is not true right now.

You are correct in that there is no consistency guarantee for the numbers with the default huge page size today. However, I am not really a fan of taking the lock for that guarantee. IMO, the above code is fine.

This discussion reminds me that ideally there should be a per-hstate lock. My guess is that the global lock is a carry over from the days when only a single huge page size was supported. In practice, I don't think this is much of an issue as people typically only use a single huge page size. But, if anyone thinks is/may be an issue I am happy to make the changes.

Signature · 1 line

Mike Kravetz

Quoted 9 lines

Thanks!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

2017-11-22 09:11 UTC #11 lore raw

Quoted 1 line · Mike Kravetz

On 11/21/2017 11:59 AM, Roman Gushchin wrote:

[...]

Quoted 23 lines

What we can do, is to rename "count" into "nr_huge_pages", like:

	for_each_hstate(h) {
		unsigned long nr_huge_pages = h->nr_huge_pages;

		total += (PAGE_SIZE << huge_page_order(h)) * nr_huge_pages;

		if (h == &default_hstate)
			seq_printf(m,
				   "HugePages_Total:   %5lu\n"
				   "HugePages_Free:    %5lu\n"
				   "HugePages_Rsvd:    %5lu\n"
				   "HugePages_Surp:    %5lu\n"
				   "Hugepagesize:   %8lu kB\n",
				   nr_huge_pages,
				   h->free_huge_pages,
				   h->resv_huge_pages,
				   h->surplus_huge_pages,
				   (PAGE_SIZE << huge_page_order(h)) / 1024);
	}

	seq_printf(m, "Hugetlb:        %8lu kB\n", total / 1024);

Quoted 7 lines

Quoted 3 lines
But maybe taking a lock is not a bad idea, because it will also guarantee consistency between other numbers (like HugePages_Free) as well, which is not true right now.
You are correct in that there is no consistency guarantee for the numbers with the default huge page size today. However, I am not really a fan of taking the lock for that guarantee. IMO, the above code is fine.

I agree

Quoted 5 lines

This discussion reminds me that ideally there should be a per-hstate lock. My guess is that the global lock is a carry over from the days when only a single huge page size was supported. In practice, I don't think this is much of an issue as people typically only use a single huge page size. But, if anyone thinks is/may be an issue I am happy to make the changes.

Well, it kind of makes sense but I am not sure it is worth bothering.

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>