mm: show stats for non-default hugepage sizes in /proc/meminfo

PATCH In review Roman Gushchin 5 participants 13 messages Nov 13, 2017 – Nov 14, 2017

Subsystems (2)

Memory management HugeTLB

lore

Share & tools

lore

Roman Gushchin

author

2017-11-13 16:03 UTC #0 lore raw

PATCH 1/1 mm: show stats for non-default hugepage sizes in /proc/meminfo

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).

If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.

To solve this problem, let's display stats for all hugepage sizes. To provide the backward compatibility let's save the existing format for the default size, and add a prefix (e.g. 1G_) for non-default sizes.

For example (100 2Mb pages and 2 1Gb pages are pre-allocated):

  $ cat /proc/meminfo
  MemTotal:        8168976 kB
  MemFree:         5664792 kB
  <...>
  CmaFree:               0 kB
  HugePages_1G_Total:       2
  HugePages_1G_Free:        2
  HugePages_1G_Rsvd:        0
  HugePages_1G_Surp:        0
  Hugepagesize_1G:    1048576 kB
  HugePages_Total:     100
  HugePages_Free:      100
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  DirectMap4k:       30584 kB
  DirectMap2M:     3115008 kB
  DirectMap1G:     7340032 kB

Signed-off-by Roman Gushchin <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc [email protected]

---
 mm/hugetlb.c | 49 +++++++++++++++++++++++++++++++++++++------------
 1 file changed, 37 insertions(+), 12 deletions(-)

mm/hugetlb.c +37 ~~-12~~

index 4b3bbd2980bb..abd37999f5da 100644
@@ -2973,20 +2973,45 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write,
 
 void hugetlb_report_meminfo(struct seq_file *m)
 {
-	struct hstate *h = &default_hstate;
+	struct hstate *h;
+
 	if (!hugepages_supported())
 		return;
-	seq_printf(m,
-			"HugePages_Total:   %5lu\n"
-			"HugePages_Free:    %5lu\n"
-			"HugePages_Rsvd:    %5lu\n"
-			"HugePages_Surp:    %5lu\n"
-			"Hugepagesize:   %8lu kB\n",
-			h->nr_huge_pages,
-			h->free_huge_pages,
-			h->resv_huge_pages,
-			h->surplus_huge_pages,
-			1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+
+	for_each_hstate(h) {
+		char prefix[16] = "";
+
+		if (h != &default_hstate) {
+			unsigned int order = huge_page_order(h) + PAGE_SHIFT;
+			char suffix = '_';
+
+			if (order >= 30) {
+				order -= 30;
+				suffix = 'G';
+			} else if (order >= 20) {
+				order -= 20;
+				suffix = 'M';
+			} else if (order >= 10) {
+				order -= 10;
+				suffix = 'k';
+			}
+
+			snprintf(prefix, sizeof(prefix), "_%lu%c",
+				 1UL << order, suffix);
+		}
+
+		seq_printf(m,
+			"HugePages%s_Total:   %5lu\n"
+			"HugePages%s_Free:    %5lu\n"
+			"HugePages%s_Rsvd:    %5lu\n"
+			"HugePages%s_Surp:    %5lu\n"
+			"Hugepagesize%s:   %8lu kB\n",
+			prefix, h->nr_huge_pages,
+			prefix, h->free_huge_pages,
+			prefix, h->resv_huge_pages,
+			prefix, h->surplus_huge_pages,
+			prefix, 1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
+	}
 }
 
 int hugetlb_report_node_meminfo(int nid, char *buf)

Signature · 7 lines

2.13.6

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

2017-11-13 16:11 UTC #1 lore raw

Quoted 7 lines · Roman Gushchin

Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.

Yes this sucks but we do have per numa node per h-state stats in sysfs already /sys/devices/system/node/node*/hugepages

I know it is another source of the information but is there any reason you cannot use it?

Signature · 2 lines

Michal Hocko
SUSE Labs

Roman Gushchin

author

2017-11-13 16:39 UTC #2 lore raw

Quoted 9 lines · Michal Hocko

Quoted 7 lines · Roman Gushchin
Currently we display some hugepage statistics (total, free, etc) in /proc/meminfo, but only for default hugepage size (e.g. 2Mb).
If hugepages of different sizes are used (like 2Mb and 1Gb on x86-64), /proc/meminfo output can be confusing, as non-default sized hugepages are not reflected at all, and there are no signs that they are existing and consuming system memory.

Quoted 5 lines

Yes this sucks but we do have per numa node per h-state stats in sysfs already /sys/devices/system/node/node*/hugepages
I know it is another source of the information but is there any reason you cannot use it?

Hi, Michal!

In my case, I didn't know in advance, that hugetlb pages are preallocated, and spent some time trying to find "magically disappeared" several Gb of memory, which are not reflected in any /proc/[meminfo,vmstat] counters.

IMO, /proc/meminfo should give a user a high-level overview of memory usage in the system, without a need to look into other places. Of course, we always have some amount of unaccounted memory, but it shouldn't be measured in Gb, as in this case.

If you're worried about adding counters which will be 0 most of the time for most users, we can print them conditionally, only if total number of corresponding pages is not 0.

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

2017-11-14 08:21 UTC #3 lore raw

On Mon 13-11-17 16:33:05, Roman Gushchin wrote:
[...]

Quoted 4 lines

IMO, /proc/meminfo should give a user a high-level overview of memory usage in the system, without a need to look into other places. Of course, we always have some amount of unaccounted memory, but it shouldn't be measured in Gb, as in this case.

Well, this is not so easy. There can be _a lot_ of unaccounted memory. Gbs is not something unheard of (fs metadata directly allocated by the page allocator, network buffers, you name it). Unlike those the hugetlb requires an explicit admin interaction. Especially for the non-default hugetlb page sizes.

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Hansen

2017-11-13 17:06 UTC #4 lore raw

In reply to Roman Gushchin

Quoted 3 lines · Roman Gushchin

To solve this problem, let's display stats for all hugepage sizes. To provide the backward compatibility let's save the existing format for the default size, and add a prefix (e.g. 1G_) for non-default sizes.

Is there something keeping you from using the sysfs version of this information?

Roman Gushchin

author

2017-11-13 18:11 UTC #5 lore raw

Quoted 7 lines · Dave Hansen

Quoted 3 lines · Roman Gushchin
To solve this problem, let's display stats for all hugepage sizes. To provide the backward compatibility let's save the existing format for the default size, and add a prefix (e.g. 1G_) for non-default sizes.
Is there something keeping you from using the sysfs version of this information?

Just answered the same question to Michal.

In two words: it would be nice to have a high-level overview of memory usage in the system in /proc/meminfo.

Thanks!

Dave Hansen

2017-11-13 18:17 UTC #6 lore raw

Quoted 6 lines · Roman Gushchin

Quoted 5 lines · Dave Hansen
Quoted 3 lines · Roman Gushchin
To solve this problem, let's display stats for all hugepage sizes. To provide the backward compatibility let's save the existing format for the default size, and add a prefix (e.g. 1G_) for non-default sizes.

Quoted 7 lines

Quoted 2 lines
Is there something keeping you from using the sysfs version of this information?
Just answered the same question to Michal.
In two words: it would be nice to have a high-level overview of memory usage in the system in /proc/meminfo.

I don't think it's worth cluttering up meminfo for this, imnho.

Mike Kravetz

2017-11-13 18:30 UTC #7 lore raw

Quoted 10 lines · Dave Hansen

Quoted 9 lines · Roman Gushchin
Quoted 7 lines · Dave Hansen
Quoted 3 lines · Roman Gushchin
To solve this problem, let's display stats for all hugepage sizes. To provide the backward compatibility let's save the existing format for the default size, and add a prefix (e.g. 1G_) for non-default sizes.
Is there something keeping you from using the sysfs version of this information?

Quoted 6 lines

Quoted 4 lines
Just answered the same question to Michal.
In two words: it would be nice to have a high-level overview of memory usage in the system in /proc/meminfo.
I don't think it's worth cluttering up meminfo for this, imnho.

I tend to agree that it would be better not to add additional huge page sizes here. It may not seem too intrusive to (potentially) add one extra set of entries for GB huge pages on x86. However, other architectures such as powerpc or sparc have several several huge pages sizes that could potentially be added here as well. Although, in practice one does tend to use a single huge pages size. If you change the default huge page size, then those entries will be in /proc/meminfo.

Signature · 7 lines

Mike Kravetz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Roman Gushchin

author

2017-11-13 18:45 UTC #8 lore raw

Quoted 18 lines · Mike Kravetz

Quoted 10 lines · Dave Hansen
Quoted 9 lines · Roman Gushchin
Quoted 7 lines · Dave Hansen
Quoted 3 lines · Roman Gushchin
To solve this problem, let's display stats for all hugepage sizes. To provide the backward compatibility let's save the existing format for the default size, and add a prefix (e.g. 1G_) for non-default sizes.
Is there something keeping you from using the sysfs version of this information?
Quoted 6 lines
Quoted 4 lines
Just answered the same question to Michal.
In two words: it would be nice to have a high-level overview of memory usage in the system in /proc/meminfo.
I don't think it's worth cluttering up meminfo for this, imnho.

Quoted 7 lines

I tend to agree that it would be better not to add additional huge page sizes here. It may not seem too intrusive to (potentially) add one extra set of entries for GB huge pages on x86. However, other architectures such as powerpc or sparc have several several huge pages sizes that could potentially be added here as well. Although, in practice one does tend to use a single huge pages size. If you change the default huge page size, then those entries will be in /proc/meminfo.

I do agree that it might add some unnecessary verbosity if these sizes are not used, but if they are, this information is super-useful. So, might be a conditional printing will work here?

Or, at least, some total counter, e.g. how much memory is consumed by hugetlb pages?

Thanks!

Johannes Weiner

2017-11-13 19:11 UTC #9 lore raw

Quoted 2 lines · Roman Gushchin

Or, at least, some total counter, e.g. how much memory is consumed by hugetlb pages?

I'm not a big fan of the verbose breakdown for every huge page size. As others have pointed out such detail exists elswhere.

But I do think we should have a summary counter for memory consumed by hugetlb that lets you know how much is missing from MemTotal. This can be large parts of overall memory, and right now /proc/meminfo will give the impression we are leaking those pages.

Maybe a simple summary counter for everything set aside by the hugetlb subsystem - default and non-default page sizes, whether they're used or only reserved etc.?

Hugetlb 12345 kB

Mike Kravetz

2017-11-13 19:25 UTC #10 lore raw

Quoted 12 lines · Johannes Weiner

Quoted 2 lines · Roman Gushchin
Or, at least, some total counter, e.g. how much memory is consumed by hugetlb pages?
I'm not a big fan of the verbose breakdown for every huge page size. As others have pointed out such detail exists elswhere.
But I do think we should have a summary counter for memory consumed by hugetlb that lets you know how much is missing from MemTotal. This can be large parts of overall memory, and right now /proc/meminfo will give the impression we are leaking those pages.

Quoted 5 lines

Maybe a simple summary counter for everything set aside by the hugetlb subsystem - default and non-default page sizes, whether they're used or only reserved etc.?
Hugetlb 12345 kB

I would prefer this approach. The 'trick' is coming up with a name or description that is not confusing. Unfortunately, we have to leave the existing entries. So, this new entry will be greater than or equal to HugePages_Total. :( I guess Hugetlb is as good of a name as any?

Signature · 7 lines

Mike Kravetz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Roman Gushchin

author

2017-11-14 12:48 UTC #11 lore raw

Quoted 17 lines · Mike Kravetz

Quoted 16 lines · Johannes Weiner
Quoted 2 lines · Roman Gushchin
Or, at least, some total counter, e.g. how much memory is consumed by hugetlb pages?
I'm not a big fan of the verbose breakdown for every huge page size. As others have pointed out such detail exists elswhere.
But I do think we should have a summary counter for memory consumed by hugetlb that lets you know how much is missing from MemTotal. This can be large parts of overall memory, and right now /proc/meminfo will give the impression we are leaking those pages.
Maybe a simple summary counter for everything set aside by the hugetlb subsystem - default and non-default page sizes, whether they're used or only reserved etc.?

Quoted 6 lines

Quoted 1 line
Hugetlb 12345 kB
I would prefer this approach. The 'trick' is coming up with a name or description that is not confusing. Unfortunately, we have to leave the existing entries. So, this new entry will be greater than or equal to HugePages_Total. :( I guess Hugetlb is as good of a name as any?

Yes, I like this approach too, and Hugetlb (in kB) sounds reasonable. I'll post a new patch soon.

Dave Hansen

2017-11-13 19:31 UTC #12 lore raw

In reply to Johannes Weiner

Quoted 3 lines · Johannes Weiner

Maybe a simple summary counter for everything set aside by the hugetlb subsystem - default and non-default page sizes, whether they're used or only reserved etc.?

Yeah, one line is a lot more sane than 5 lines times all the extra sizes. It'll just be a matter of bikeshedding the name and whether it should include the default pages being consumed or not. I vote for:

Hugetlb: "/sysfs FTW!" kB