New: Duplicated MemTotal with different values

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

hm, mine does that too. A discrepancy between `totalram_pages' and NODE_DATA(0)->node_present_pages.

I don't know what the reasons are for that but yes, one would expect the per-node MemTotals to sum up to the global one.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'd suspect it has something to do with 9feedc9d831e ("mm: introduce new field "managed_pages" to struct zone") and 3.8 would be the first kernel release with this change. Is it possible to try 3.7 or, better yet, with this patch reverted?

If neither of these are the case, or you aren't comfortable building and booting a custom kernel, please send along your /proc/zoneinfo.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

My desktop machine at google in inconsistent, as is the 2.6.32-based machine, so it obviously predates 9feedc9d831e.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hmm, ok. The question is which one is right: the per-node MemTotal is the amount of present RAM, the spanned range minus holes, and the system MemTotal is the amount of pages released to the buddy allocator by bootmem and discounts not only the memory holes but also reserved pages. Should they both be the amount of RAM present or the amount of unreserved RAM present?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi David,
We have worked out a patch set to address this issue. The first two
patches have been merged into v3.8, and another two patches are queued in Andrew's mm tree for v3.9.
The patch set introduces a new field named managed_pages into struct
zone to distinguish between pages present in a zone and pages managed by the buddy system. So
zone->present_pages = zone->spanned_pages - pages_in_hole; zone->managed_pages = pages_managed_by_buddy_system_in_the_zone;
We have also added a field named "managed" into /proc/zoneinfo, but
haven't touch /proc/meminfo and /sys/devices/system/node/nodex/meminfo yet. If preferred, we could work out another patch to enhance these two files as suggested above.
Regards!
Gerry

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm glad this is a known issue that you're working on, but my question still stands: if MemTotal is going to be consistent throughout /proc/meminfo and /sys/devices/system/node/nodeX/meminfo, which is correct? The present RAM minus holes or the amount available to the buddy allocator not including reserved memory?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

As reported by https://bugzilla.kernel.org/show_bug.cgi?id=53501,
"MemTotal" from /proc/meminfo means memory pages managed by the buddy system (managed_pages), but "MemTotal" from /sys/.../node/nodex/meminfo means phsical pages present (present_pages) within the NUMA node. There's a difference between managed_pages and present_pages due to bootmem allocator and reserved pages.

So introduce a new field "MemManaged" to /sys/.../nodex/meminfo and /proc/meminfo, so that:
MemTotal = present_pages
MemManaged = managed_pages = present_pages - reserved_pages

---
Hi Andrew and David,
How about these draft patches? It just passes compilation. If you
are OK with them, we will conduct tests tomorrow.
Regards!
Gerry

---
 drivers/base/node.c |    2 ++
 fs/proc/meminfo.c   |    2 ++
 mm/page_alloc.c     |   19 ++++++++++++++++++-
 3 files changed, 22 insertions(+), 1 deletion(-)

index fac124a..6508c4d 100644
@@ -66,6 +66,7 @@ static ssize_t node_read_meminfo(struct device *dev,
 	si_meminfo_node(&i, nid);
 	n = sprintf(buf,
 		       "Node %d MemTotal:       %8lu kB\n"
+		       "Node %d MemManaged:     %8lu kB\n"
 		       "Node %d MemFree:        %8lu kB\n"
 		       "Node %d MemUsed:        %8lu kB\n"
 		       "Node %d Active:         %8lu kB\n"

@@ -77,6 +78,7 @@ static ssize_t node_read_meminfo(struct device *dev,
 		       "Node %d Unevictable:    %8lu kB\n"
 		       "Node %d Mlocked:        %8lu kB\n",
 		       nid, K(i.totalram),
+		       nid, K(i.sharedram),
 		       nid, K(i.freeram),
 		       nid, K(i.totalram - i.freeram),
 		       nid, K(node_page_state(nid, NR_ACTIVE_ANON) +

index 80e4645..5d58cbb 100644
@@ -54,6 +54,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 	 */
 	seq_printf(m,
 		"MemTotal:       %8lu kB\n"
+		"MemManaged:     %8lu kB\n"
 		"MemFree:        %8lu kB\n"
 		"Buffers:        %8lu kB\n"
 		"Cached:         %8lu kB\n"

@@ -106,6 +107,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 #endif
 		,
 		K(i.totalram),
+		K(totalram_pages),
 		K(i.freeram),
 		K(i.bufferram),
 		K(cached),

index 595e655..6884dc5 100644
@@ -2830,7 +2830,13 @@ static inline void show_node(struct zone *zone)
 
 void si_meminfo(struct sysinfo *val)
 {
-	val->totalram = totalram_pages;
+	int nid;
+	unsigned long present_pages = 0;
+
+	for_each_node_state(nid, N_MEMORY)
+		present_pages += node_present_pages(nid);
+
+	val->totalram = present_pages;
 	val->sharedram = 0;
 	val->freeram = global_page_state(NR_FREE_PAGES);
 	val->bufferram = nr_blockdev_pages();

@@ -2844,8 +2850,19 @@ EXPORT_SYMBOL(si_meminfo);
 #ifdef CONFIG_NUMA
 void si_meminfo_node(struct sysinfo *val, int nid)
 {
+	int zone_type;
+	unsigned long managed_pages = 0;
 	pg_data_t *pgdat = NODE_DATA(nid);
 
+	for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++)
+		managed_pages += pgdat->node_zones[zone_type].managed_pages;
+
+	/*
+	 * Ugly hack: struct sysinfo is exported to userspace and there's no
+	 * space available for a new field "managedram", so reuse field
+	 * "sharedram".
+	 */
+	val->sharedram = managed_pages;
 	val->totalram = pgdat->node_present_pages;
 	val->freeram = node_page_state(nid, NR_FREE_PAGES);
 #ifdef CONFIG_HIGHMEM

1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

There's typical usage of si_meminfo as below:

	si_meminfo(&si);
	threshold = si->totalram - si.totalhigh;

It may cause underflow if memory hotplug races with si_meminfo() because there's no mechanism to protect si_meminfo() from memory hotplug operations. And some callers expects that si_meminfo() is a lightweight operations. So introduce a lightweight mechanism to protect si_meminfo() from memory hotplug operations.

---
 mm/page_alloc.c |   24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

index 6884dc5..5cf03d4 100644
@@ -2831,18 +2831,34 @@ static inline void show_node(struct zone *zone)
 void si_meminfo(struct sysinfo *val)
 {
 	int nid;
-	unsigned long present_pages = 0;
+	unsigned long present_pages;
 
+	val->sharedram = 0;
+	val->mem_unit = PAGE_SIZE;
+
+restart:
+	present_pages = 0;
 	for_each_node_state(nid, N_MEMORY)
 		present_pages += node_present_pages(nid);
 
 	val->totalram = present_pages;
-	val->sharedram = 0;
 	val->freeram = global_page_state(NR_FREE_PAGES);
 	val->bufferram = nr_blockdev_pages();
 	val->totalhigh = totalhigh_pages;
 	val->freehigh = nr_free_highpages();
-	val->mem_unit = PAGE_SIZE;
+
+	/*
+	 * si_meminfo() may generate invalid results because it isn't protected
+	 * from memory hotplug operaitons. And some callers expect that it's an
+	 * lightweigh operation. So provide minimal protections without heavy
+	 * overhead.
+	 */
+	if (val->totalram < val->freeram ||
+	    val->totalram < val->bufferram ||
+	    val->totalram < val->totalhigh ||
+	    val->totalhigh < val->freehigh ||
+	    val->freeram < val->freehigh)
+		goto restart;
 }
 
 EXPORT_SYMBOL(si_meminfo);

@@ -2854,6 +2870,7 @@ void si_meminfo_node(struct sysinfo *val, int nid)
 	unsigned long managed_pages = 0;
 	pg_data_t *pgdat = NODE_DATA(nid);
 
+	lock_memory_hotplug();
 	for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++)
 		managed_pages += pgdat->node_zones[zone_type].managed_pages;
 

@@ -2874,6 +2891,7 @@ void si_meminfo_node(struct sysinfo *val, int nid)
 	val->freehigh = 0;
 #endif
 	val->mem_unit = PAGE_SIZE;
+	unlock_memory_hotplug();
 }
 #endif
 

1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nobody is asking for a MemManaged field, we're asking for consistency in MemTotal as exported by /proc/meminfo and /sys/devices/system/node/node*/meminfo. In my opinion, this should be the amount of RAM installed on the system regardless of the amount of memory reserved, i.e. the current /sys/devices/system/node/node*/meminfo semantics. There is no implicit guarantee that memory in MemTotal can be allocated by the buddy allocator.

So how about we just make the MemTotal in /proc/meminfo coincide with these semantics?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Instead of this, I think it would be appropriate to add a comment that requires synchronization if two fields are going to be compared, i.e. use {lock,unlock}_memory_hotplug() in the caller to si_meminfo(), or appropriate underflow checking is done upon return.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi David,

What I confuse is why have /proc/meminfo and /proc/vmstat at the same time, they both use to monitor memory subsystem states. What's the root reason?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This has nothing to do with this thread, but /proc/vmstat actually does not include the MemTotal value being discussed in this thread that /proc/meminfo does. /proc/meminfo is typically the interface used by applications, probably mostly for historical purposes since both are present when procfs is configured and mounted, but also to avoid determining the native page size. There's no implicit userspace API exported by /proc/vmstat.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

As reported by https://bugzilla.kernel.org/show_bug.cgi?id=53501,
"MemTotal" from /proc/meminfo means memory pages managed by the buddy system (managed_pages), but "MemTotal" from /sys/.../node/nodex/meminfo means phsical pages present (present_pages) within the NUMA node. There's a difference between managed_pages and present_pages due to bootmem allocator and reserved pages.

So change /proc/meminfo to report physical memory installed as "MemTotal", which is
MemTotal = sum(pgdat->present_pages)

---
Hi David,
How about this simpilified version?
Regards!
Gerry

---
 mm/page_alloc.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

index e4e8bf1..8e53d6e 100644
@@ -2835,7 +2835,13 @@ static inline void show_node(struct zone *zone)
 
 void si_meminfo(struct sysinfo *val)
 {
-	val->totalram = totalram_pages;
+	int nid;
+	unsigned long present_pages = 0;
+
+	for_each_node_state(nid, N_MEMORY)
+		present_pages += node_present_pages(nid);
+
+	val->totalram = present_pages;
 	val->sharedram = 0;
 	val->freeram = global_page_state(NR_FREE_PAGES);
 	val->bufferram = nr_blockdev_pages();

1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Documentation/filesystems/proc.txt says

MemTotal: Total usable ram (i.e. physical ram minus a few reserved
bits and the kernel binary code)

And arguably, that is more useful than "total physical memory".

Presumably the per-node MemTotals are including kernel memory and reserved memory. Maybe they should be fixed instead (sounds hard).

Or maybe we just leave everything as-is and document it carefully.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Andrew,
It's really hard, but I think it deserve it because have reduced
about 460 lines of code when fixing this bug. So how about following patchset?
The first 27 patches introduces some help functions to simplify
free_initmem() and free_initrd_mem() for most arches.

	The 28th patch increases zone->managed_pages when freeing reserved

pages.
The 29th patch change /sys/.../nodex/meminfo to report "available
pages within the node" as MemTatoal.
Regards!
Gerry

Jiang Liu (29):
mm: introduce free_reserved_mem/page() to reduce duplicated code mm/alpha: use common help functions to free reserved pages mm/ARM: use common help functions to free reserved pages mm/avr32: use common help functions to free reserved pages mm/blackfin: use common help functions to free reserved pages mm/c6x: use common help functions to free reserved pages mm/cris: use common help functions to free reserved pages mm/FRV: use common help functions to free reserved pages mm/h8300: use common help functions to free reserved pages mm/IA64: use common help functions to free reserved pages mm/m32r: use common help functions to free reserved pages mm/m68k: use common help functions to free reserved pages mm/microblaze: use common help functions to free reserved pages mm/MIPS: use common help functions to free reserved pages mm/mn10300: use common help functions to free reserved pages mm/openrisc: use common help functions to free reserved pages mm/parisc: use common help functions to free reserved pages mm/ppc: use common help functions to free reserved pages mm/s390: use common help functions to free reserved pages mm/score: use common help functions to free reserved pages mm/SH: use common help functions to free reserved pages mm/SPARC: use common help functions to free reserved pages mm/um: use common help functions to free reserved pages mm/unicore32: use common help functions to free reserved pages mm/x86: use common help functions to free reserved pages mm/xtensa: use common help functions to free reserved pages mm,kexec: use common help functions to free reserved pages
mm: increase zone->managed_pages when freeing reserved pages
mm: report available pages as "MemTotal" for each NUMA node

 arch/alpha/kernel/sys_nautilus.c             |    5 ++-
 arch/alpha/mm/init.c                         |   21 ++----------
 arch/arm/mm/init.c                           |   43 ++++++++++--------------
 arch/arm64/mm/init.c                         |   26 ++-------------
 arch/avr32/mm/init.c                         |   24 ++------------
 arch/blackfin/mm/init.c                      |   20 ++----------
 arch/c6x/mm/init.c                           |   30 ++---------------
 arch/cris/mm/init.c                          |   12 +------
 arch/frv/mm/init.c                           |   26 ++-------------
 arch/h8300/mm/init.c                         |   28 ++--------------
 arch/ia64/mm/init.c                          |   23 +++----------
 arch/m32r/mm/init.c                          |   21 ++----------
 arch/m68k/mm/init.c                          |   24 ++------------
 arch/microblaze/include/asm/setup.h          |    1 -
 arch/microblaze/mm/init.c                    |   28 ++--------------
 arch/mips/mm/init.c                          |   13 ++------
 arch/mn10300/mm/init.c                       |   23 ++-----------
 arch/openrisc/mm/init.c                      |   23 ++-----------
 arch/parisc/mm/init.c                        |   23 ++-----------
 arch/powerpc/kernel/crash_dump.c             |    5 +--
 arch/powerpc/kernel/fadump.c                 |    5 +--
 arch/powerpc/kernel/kvm.c                    |    7 +---
 arch/powerpc/mm/mem.c                        |   29 ++---------------
 arch/powerpc/platforms/512x/mpc512x_shared.c |    5 +--
 arch/s390/mm/init.c                          |   23 ++-----------
 arch/score/mm/init.c                         |   25 ++------------
 arch/sh/mm/init.c                            |   22 ++-----------
 arch/sparc/kernel/leon_smp.c                 |   15 ++-------
 arch/sparc/mm/init_32.c                      |   40 ++++-------------------
 arch/sparc/mm/init_64.c                      |   20 ++----------
 arch/tile/mm/init.c                          |    1 +
 arch/um/kernel/mem.c                         |   10 +-----
 arch/unicore32/mm/init.c                     |   26 ++-------------
 arch/x86/mm/init.c                           |    5 +--
 arch/xtensa/mm/init.c                        |   21 ++----------
 include/linux/mm.h                           |   45 ++++++++++++++++++++++++++
 kernel/kexec.c                               |    8 ++---
 mm/page_alloc.c                              |    8 ++++-
 38 files changed, 139 insertions(+), 595 deletions(-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

yikes.

Let's defer the problem for now. Please send the patches out in the usual fashion after 3.9-rc1 and we'll take a look?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sure, will send out those patches after 3.9-rc1 is out.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Do you mean /proc/vmstat is not used by applications. sar -B 1
pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
I think they are read from /proc/vmstat

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, there is userspace code that parses /proc/vmstat.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Then why both need /proc/meminfo and /proc/vmstat?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Because we do not break userspace.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>