Skip to contentSkip to conversation

memblock, x86: fix big numa system booting

PATCH In review Yinghai Lu 9 participants 25 messages Jan 24, 2014 – Jan 28, 2014

Subsystems (5)
lore
Share & tools
lore

Yinghai Lu

author
#0 lore raw

cover memblock, x86: fix big numa system booting

Big numa system boot get broken while switch API from bootmem to memblock_virt.

Revert the offending patch, and also address swiotlb regression.

Thanks

Yinghai

author
#1 lore raw

PATCH 1/3 memblock, nobootmem: Add memblock_virt_alloc_low()

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Cc Russell King <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
---
 arch/arm/kernel/setup.c |    2 +-
 include/linux/bootmem.h |   37 +++++++++++++++++++++++++++++++++++++
 lib/swiotlb.c           |    4 ++--
 3 files changed, 40 insertions(+), 3 deletions(-)
include/linux/bootmem.h +37 -0
Index: linux-2.6/include/linux/bootmem.h
===================================================================
@@ -175,6 +175,27 @@ static inline void * __init memblock_vir
 						    NUMA_NO_NODE);
 }
 
+#ifndef ARCH_LOW_ADDRESS_LIMIT
+#define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
+#endif
+
+static inline void * __init memblock_virt_alloc_low(
+					phys_addr_t size, phys_addr_t align)
+{
+	return memblock_virt_alloc_try_nid(size, align,
+						   BOOTMEM_LOW_LIMIT,
+						   ARCH_LOW_ADDRESS_LIMIT,
+						   NUMA_NO_NODE);
+}
+static inline void * __init memblock_virt_alloc_low_nopanic(
+					phys_addr_t size, phys_addr_t align)
+{
+	return memblock_virt_alloc_try_nid_nopanic(size, align,
+						   BOOTMEM_LOW_LIMIT,
+						   ARCH_LOW_ADDRESS_LIMIT,
+						   NUMA_NO_NODE);
+}
+
 static inline void * __init memblock_virt_alloc_from_nopanic(
 		phys_addr_t size, phys_addr_t align, phys_addr_t min_addr)
 {
@@ -238,6 +259,22 @@ static inline void * __init memblock_vir
 	return __alloc_bootmem_nopanic(size, align, BOOTMEM_LOW_LIMIT);
 }
 
+static inline void * __init memblock_virt_alloc_low(
+					phys_addr_t size, phys_addr_t align)
+{
+	if (!align)
+		align = SMP_CACHE_BYTES;
+	return __alloc_bootmem_low(size, align, BOOTMEM_LOW_LIMIT);
+}
+
+static inline void * __init memblock_virt_alloc_low_nopanic(
+					phys_addr_t size, phys_addr_t align)
+{
+	if (!align)
+		align = SMP_CACHE_BYTES;
+	return __alloc_bootmem_low_nopanic(size, align, BOOTMEM_LOW_LIMIT);
+}
+
 static inline void * __init memblock_virt_alloc_from_nopanic(
 		phys_addr_t size, phys_addr_t align, phys_addr_t min_addr)
 {
lib/swiotlb.c +2 -2
Index: linux-2.6/lib/swiotlb.c
===================================================================
@@ -172,7 +172,7 @@ int __init swiotlb_init_with_tbl(char *t
 	/*
 	 * Get the overflow emergency buffer
 	 */
-	v_overflow_buffer = memblock_virt_alloc_nopanic(
+	v_overflow_buffer = memblock_virt_alloc_low_nopanic(
 						PAGE_ALIGN(io_tlb_overflow),
 						PAGE_SIZE);
 	if (!v_overflow_buffer)
@@ -220,7 +220,7 @@ swiotlb_init(int verbose)
 	bytes = io_tlb_nslabs << IO_TLB_SHIFT;
 
 	/* Get IO TLB memory from the low pages */
-	vstart = memblock_virt_alloc_nopanic(PAGE_ALIGN(bytes), PAGE_SIZE);
+	vstart = memblock_virt_alloc_low_nopanic(PAGE_ALIGN(bytes), PAGE_SIZE);
 	if (vstart && !swiotlb_init_with_tbl(vstart, io_tlb_nslabs, verbose))
 		return;
 
arch/arm/kernel/setup.c +1 -1
Index: linux-2.6/arch/arm/kernel/setup.c
===================================================================
@@ -717,7 +717,7 @@ static void __init request_standard_reso
 	kernel_data.end     = virt_to_phys(_end - 1);
 
 	for_each_memblock(memory, region) {
-		res = memblock_virt_alloc(sizeof(*res), 0);
+		res = memblock_virt_alloc_low(sizeof(*res), 0);
 		res->name  = "System RAM";
 		res->start = __pfn_to_phys(memblock_region_memory_base_pfn(region));
 		res->end = __pfn_to_phys(memblock_region_memory_end_pfn(region)) - 1;
#2 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 5 lines · Yinghai Lu

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Please include the title of the patch that caused the regression.
I presume it is "mm/lib/swiotlb: Use memblock apis for early memory allocations"

Interestingly enough when I asked about it:

https://lkml.org/lkml/2013/11/9/280

	>> v_overflow_buffer = memblock_virt_alloc_align_nopanic(
	>> +						PAGE_ALIGN(io_tlb_overflow),
	>> +						PAGE_SIZE);
	> 
	> Does this guarantee that the pages will be allocated below 4GB?
	> 
	Yes. The memblock layer still allocates memory from lowmem. As I
	mentioned, there is no change in the behavior than what is today
	apart from just the interface change.

How did that happend? Was there another patch in the series that altered such assumption?

Quoted 101 lines
Cc Russell King <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
---
 arch/arm/kernel/setup.c |    2 +-
 include/linux/bootmem.h |   37 +++++++++++++++++++++++++++++++++++++
 lib/swiotlb.c           |    4 ++--
 3 files changed, 40 insertions(+), 3 deletions(-)

Index: linux-2.6/include/linux/bootmem.h
===================================================================

include/linux/bootmem.h +37 -0
@@ -175,6 +175,27 @@ static inline void * __init memblock_vir
 						    NUMA_NO_NODE);
 }
 
+#ifndef ARCH_LOW_ADDRESS_LIMIT
+#define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
+#endif
+
+static inline void * __init memblock_virt_alloc_low(
+					phys_addr_t size, phys_addr_t align)
+{
+	return memblock_virt_alloc_try_nid(size, align,
+						   BOOTMEM_LOW_LIMIT,
+						   ARCH_LOW_ADDRESS_LIMIT,
+						   NUMA_NO_NODE);
+}
+static inline void * __init memblock_virt_alloc_low_nopanic(
+					phys_addr_t size, phys_addr_t align)
+{
+	return memblock_virt_alloc_try_nid_nopanic(size, align,
+						   BOOTMEM_LOW_LIMIT,
+						   ARCH_LOW_ADDRESS_LIMIT,
+						   NUMA_NO_NODE);
+}
+
 static inline void * __init memblock_virt_alloc_from_nopanic(
 		phys_addr_t size, phys_addr_t align, phys_addr_t min_addr)
 {
@@ -238,6 +259,22 @@ static inline void * __init memblock_vir
 	return __alloc_bootmem_nopanic(size, align, BOOTMEM_LOW_LIMIT);
 }
 
+static inline void * __init memblock_virt_alloc_low(
+					phys_addr_t size, phys_addr_t align)
+{
+	if (!align)
+		align = SMP_CACHE_BYTES;
+	return __alloc_bootmem_low(size, align, BOOTMEM_LOW_LIMIT);
+}
+
+static inline void * __init memblock_virt_alloc_low_nopanic(
+					phys_addr_t size, phys_addr_t align)
+{
+	if (!align)
+		align = SMP_CACHE_BYTES;
+	return __alloc_bootmem_low_nopanic(size, align, BOOTMEM_LOW_LIMIT);
+}
+
 static inline void * __init memblock_virt_alloc_from_nopanic(
 		phys_addr_t size, phys_addr_t align, phys_addr_t min_addr)
 {

Index: linux-2.6/lib/swiotlb.c
===================================================================

lib/swiotlb.c +2 -2
@@ -172,7 +172,7 @@ int __init swiotlb_init_with_tbl(char *t
 	/*
 	 * Get the overflow emergency buffer
 	 */
-	v_overflow_buffer = memblock_virt_alloc_nopanic(
+	v_overflow_buffer = memblock_virt_alloc_low_nopanic(
 						PAGE_ALIGN(io_tlb_overflow),
 						PAGE_SIZE);
 	if (!v_overflow_buffer)
@@ -220,7 +220,7 @@ swiotlb_init(int verbose)
 	bytes = io_tlb_nslabs << IO_TLB_SHIFT;
 
 	/* Get IO TLB memory from the low pages */
-	vstart = memblock_virt_alloc_nopanic(PAGE_ALIGN(bytes), PAGE_SIZE);
+	vstart = memblock_virt_alloc_low_nopanic(PAGE_ALIGN(bytes), PAGE_SIZE);
 	if (vstart && !swiotlb_init_with_tbl(vstart, io_tlb_nslabs, verbose))
 		return;
 

Index: linux-2.6/arch/arm/kernel/setup.c
===================================================================

arch/arm/kernel/setup.c +1 -1
@@ -717,7 +717,7 @@ static void __init request_standard_reso
 	kernel_data.end     = virt_to_phys(_end - 1);
 
 	for_each_memblock(memory, region) {
-		res = memblock_virt_alloc(sizeof(*res), 0);
+		res = memblock_virt_alloc_low(sizeof(*res), 0);
 		res->name  = "System RAM";
 		res->start = __pfn_to_phys(memblock_region_memory_base_pfn(region));
 		res->end = __pfn_to_phys(memblock_region_memory_end_pfn(region)) - 1;
#3 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 25 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Yinghai Lu

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Please include the title of the patch that caused the regression.
I presume it is "mm/lib/swiotlb: Use memblock apis for early memory allocations"

Interestingly enough when I asked about it:

https://lkml.org/lkml/2013/11/9/280

	>> v_overflow_buffer = memblock_virt_alloc_align_nopanic(
	>> +						PAGE_ALIGN(io_tlb_overflow),
	>> +						PAGE_SIZE);
	> 
	> Does this guarantee that the pages will be allocated below 4GB?
	> 
	Yes. The memblock layer still allocates memory from lowmem. As I
	mentioned, there is no change in the behavior than what is today
	apart from just the interface change.
Quoted 3 lines

How did that happend? Was there another patch in the series that altered such assumption?

Actually it didn't. It was the misunderstanding on my side about the low_mem_max_addr being under 4GB, which is not always true especially for 64-bit systems which have no addressing limitations.

Regards,
Santosh

author
#4 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

In reply to Konrad Rzeszutek Wilk
Quoted 25 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Yinghai Lu

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Please include the title of the patch that caused the regression.
I presume it is "mm/lib/swiotlb: Use memblock apis for early memory allocations"

Interestingly enough when I asked about it:

https://lkml.org/lkml/2013/11/9/280

>> v_overflow_buffer = memblock_virt_alloc_align_nopanic(
>> + PAGE_ALIGN(io_tlb_overflow),
>> + PAGE_SIZE);
>
> Does this guarantee that the pages will be allocated below 4GB? >
Yes. The memblock layer still allocates memory from lowmem. As I mentioned, there is no change in the behavior than what is today apart from just the interface change.

Quoted 3 lines

How did that happend? Was there another patch in the series that altered such assumption?

Yes, that is one.

He chose to set memblock.current_limit to max_low_mapped. (that is under 4g).

but it broke big numa system as all boot mem is under 4G and system with lots of memory will need to
have big chunk for vmemmap .

Before that patch, we need to add another API to make sure those swiotlb under 4G.

Thanks

Yinghai

#5 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

In reply to Yinghai Lu

Hi,

Quoted 9 lines · Yinghai Lu

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Cc Russell King <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>

This seems to have been merged by Linus tonight as ad6492b80f, and it had fallout on ARM systems (boot failures with no console output on all but 5 of my machine/config combos).

Seems like it didn't have a chance to sit in -next, which is somewhat understandable given that it's considered a bugfix and it indeed fixed the bug it was meant to.

i'm out of time to debug this tonight (I noticed the failures as I was heading to bed and figured I'd at least bisect them), so I wouldn't mind seeing a revert of the ARM side change of ad6492b80f until it's been sorted out so we keep bisectabilty intact for the rest of the kernel.

Thanks,

-Olof

#6 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 21 lines · Olof Johansson

Hi,

Quoted 9 lines · Yinghai Lu

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Cc Russell King <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>

This seems to have been merged by Linus tonight as ad6492b80f, and it had fallout on ARM systems (boot failures with no console output on all but 5 of my machine/config combos).

Seems like it didn't have a chance to sit in -next, which is somewhat understandable given that it's considered a bugfix and it indeed fixed the bug it was meant to.

Quoted 5 lines

i'm out of time to debug this tonight (I noticed the failures as I was heading to bed and figured I'd at least bisect them), so I wouldn't mind seeing a revert of the ARM side change of ad6492b80f until it's been sorted out so we keep bisectabilty intact for the rest of the kernel.

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

Kevin

author
#7 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 28 lines · Kevin Hilman
Quoted 21 lines · Olof Johansson

Hi,

Quoted 9 lines · Yinghai Lu

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Cc Russell King <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>

This seems to have been merged by Linus tonight as ad6492b80f, and it had fallout on ARM systems (boot failures with no console output on all but 5 of my machine/config combos).

Seems like it didn't have a chance to sit in -next, which is somewhat understandable given that it's considered a bugfix and it indeed fixed the bug it was meant to.

Quoted 5 lines

i'm out of time to debug this tonight (I noticed the failures as I was heading to bed and figured I'd at least bisect them), so I wouldn't mind seeing a revert of the ARM side change of ad6492b80f until it's been sorted out so we keep bisectabilty intact for the rest of the kernel.

Quoted 3 lines

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

please try attached patch.

include/linux/bootmem.h +3 -0
Index: linux-2.6/include/linux/bootmem.h
===================================================================
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
                                                    NUMA_NO_NODE);
 }
 
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif
---
 include/linux/bootmem.h |    3 +++
 1 file changed, 3 insertions(+)
include/linux/bootmem.h +3 -0
Index: linux-2.6/include/linux/bootmem.h
===================================================================
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
 						    NUMA_NO_NODE);
 }
 
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif
#8 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 6 lines · Yinghai Lu
Quoted 3 lines · Kevin Hilman

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

please try attached patch.

Maybe I'm missing something, but there is no ARCH_LOW_ADDRESS_LIMIT defined in the ARM header files, so I don't see how adding that additional include changes anything.

Signature · 3 lines
FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up.  Estimation
in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
Estimate before purchase was "up to 13.2Mbit".
#9 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 6 lines · Russell King - ARM Linux
Quoted 5 lines · Yinghai Lu
Quoted 3 lines · Kevin Hilman

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

Quoted 6 lines
Quoted 1 line

please try attached patch.

Maybe I'm missing something, but there is no ARCH_LOW_ADDRESS_LIMIT defined in the ARM header files, so I don't see how adding that additional include changes anything.

Our emails crossed... You are right. It won't help.

Regards,
Santosh

author
#10 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

In reply to Russell King - ARM Linux
Quoted 11 lines · Russell King - ARM Linux
Quoted 6 lines · Yinghai Lu
Quoted 3 lines · Kevin Hilman

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

please try attached patch.

Maybe I'm missing something, but there is no ARCH_LOW_ADDRESS_LIMIT defined in the ARM header files, so I don't see how adding that additional include changes anything.

there is one for arm64 and s390.

arch/arm64/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT PHYS_MAS
arch/s390/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT 0x7fffff

#11 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 10 lines · Yinghai Lu
Quoted 8 lines · Russell King - ARM Linux
Quoted 6 lines · Yinghai Lu
Quoted 3 lines · Kevin Hilman

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

please try attached patch.

Quoted 8 lines
Quoted 3 lines

Maybe I'm missing something, but there is no ARCH_LOW_ADDRESS_LIMIT defined in the ARM header files, so I don't see how adding that additional include changes anything.

there is one for arm64 and s390.

arch/arm64/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT PHYS_MAS
arch/s390/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT 0x7fffff

Forgive me being difficult, but how exactly does your patch come anywhere close to sortting out the reported regression by Kevin and Olof, and now myself which is on ARM?

Your patch adds asm/processor.h to a header file, to allow architectures to override ARCH_LOW_ADDRESS_LIMIT, but there is /no/ override for ARM, so your patch has _zero_ effect there - we still end up with this being 0xffffffff, and we still end up with the thing being unable to boot.

Maybe you could describe what this ARCH_LOW_ADDRESS_LIMIT is, and what it should be set to - I'm less than clear on how to set this given that we have a multitude of different physical memory layouts - where memory can start at almost any physical address.

Signature · 3 lines
FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up.  Estimation
in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
Estimate before purchase was "up to 13.2Mbit".
author
#12 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 29 lines · Russell King - ARM Linux
Quoted 10 lines · Yinghai Lu
Quoted 8 lines · Russell King - ARM Linux
Quoted 6 lines · Yinghai Lu
Quoted 3 lines · Kevin Hilman

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

please try attached patch.

Quoted 8 lines
Quoted 3 lines

Maybe I'm missing something, but there is no ARCH_LOW_ADDRESS_LIMIT defined in the ARM header files, so I don't see how adding that additional include changes anything.

there is one for arm64 and s390.

arch/arm64/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT PHYS_MAS
arch/s390/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT 0x7fffff

Forgive me being difficult, but how exactly does your patch come anywhere close to sortting out the reported regression by Kevin and Olof, and now myself which is on ARM?

Your patch adds asm/processor.h to a header file, to allow architectures to override ARCH_LOW_ADDRESS_LIMIT, but there is /no/ override for ARM, so your patch has _zero_ effect there - we still end up with this being 0xffffffff, and we still end up with the thing being unable to boot.

Quoted 4 lines

Maybe you could describe what this ARCH_LOW_ADDRESS_LIMIT is, and what it should be set to - I'm less than clear on how to set this given that we have a multitude of different physical memory layouts - where memory can start at almost any physical address.

well, I just want to restore the alloc_bootmem_low by following patch.

commit 9233d2be108f573caa21eb450411bf8fa68cadbb

Author: Santosh Shilimkar <[email protected]>
Date:   Tue Jan 21 15:50:47 2014 -0800

arch/arm/kernel/: use memblock apis for early memory allocations

Switch to memblock interfaces for early memory allocator instead of bootmem allocator. No functional change in beahvior than what it is in current code from bootmem users points of view.

Archs already converted to NO_BOOTMEM now directly use memblock interfaces instead of bootmem wrappers build on top of memblock. And the archs which still uses bootmem, these new apis just fallback to exiting bootmem APIs.

...
@@ -717,7 +717,7 @@ static void __init request_standard_resources(const struct machine_desc *mdesc)

        kernel_data.end     = virt_to_phys(_end - 1);

        for_each_memblock(memory, region) {
-               res = alloc_bootmem_low(sizeof(*res));
+               res = memblock_virt_alloc(sizeof(*res), 0);
                res->name  = "System RAM";

but looks like memblock_virt_alloc_low is not the same as alloc_bootmem_low yet.

Let's me check further.

Thanks

Yinghai

author
#13 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 2 lines · Yinghai Lu

but looks like memblock_virt_alloc_low is not the same as alloc_bootmem_low yet.

The memblock_virt_alloc missed limit checking.

Please check attached patch.

Thanks

Yinghai

Subject: [PATCH] memblock: Add limit checking to memblock_virt_alloc

In original bootmem wrapper for memblock, we have limit checking.

Add it to memblock_virt_alloc, to address arm and x86 booting crash.

---
 mm/memblock.c |    3 +++
 1 file changed, 3 insertions(+)
mm/memblock.c +3 -0
Index: linux-2.6/mm/memblock.c
===================================================================
@@ -1077,6 +1077,9 @@ static void * __init memblock_virt_alloc
 	if (!align)
 		align = SMP_CACHE_BYTES;
 
+	if (max_addr > memblock.current_limit)
+		max_addr = memblock.current_limit;
+
 again:
 	alloc = memblock_find_in_range_node(size, align, min_addr, max_addr,
 					    nid);
#14 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

In reply to Yinghai Lu
Quoted 39 lines · Yinghai Lu
Quoted 28 lines · Kevin Hilman
Quoted 28 lines · Olof Johansson
Quoted 26 lines
Quoted 26 lines

Hi,

Quoted 9 lines · Yinghai Lu
Quoted 9 lines

The new memblock_virt APIs are used to replaced old bootmem API.

We need to allocate page below 4G for swiotlb.

That should fix regression on Andrew's system that is using swiotlb.

Cc Russell King <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>

This seems to have been merged by Linus tonight as ad6492b80f, and it had fallout on ARM systems (boot failures with no console output on all but 5 of my machine/config combos).

Seems like it didn't have a chance to sit in -next, which is somewhat understandable given that it's considered a bugfix and it indeed fixed the bug it was meant to.

i'm out of time to debug this tonight (I noticed the failures as I was heading to bed and figured I'd at least bisect them), so I wouldn't mind seeing a revert of the ARM side change of ad6492b80f until it's been sorted out so we keep bisectabilty intact for the rest of the kernel.

Quoted 3 lines
Quoted 3 lines

Like Olof, I noticed multiple boot failures on various ARM boards. I've confirmed that reverting the arch/arm part of this patch makes them all happily booting again.

please try attached patch.

Index: linux-2.6/include/linux/bootmem.h
===================================================================

include/linux/bootmem.h static inline void * __init memblock_vir · line 187

Quoted 9 lines
include/linux/bootmem.h +3 -0
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
                                                    NUMA_NO_NODE);
 }
 
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif

This won't help mostly since the ARM 32 arch don't set ARCH_LOW_ADDRESS_LIMIT. Sorry i couldn't respond to the thread earlier because of travel and don't have access to my board to try out the patches.
The issue is mostly because on ARM the allocations needs to be limited lowmem and that has been ensured using BOOTMEM_ALLOC_ACCESSIBLE in bootmem case and MEMBLOCK_ALLOC_ACCESSIBLE in memblock case.

Even though ARM change done to use alloc_low() is correct, it has undesired side effect of allocating memory from highmem because of non-use of ARCH_LOW_ADDRESS_LIMIT. Setting ARCH_LOW_ADDRESS_LIMIT to MEMBLOCK_ALLOC_ACCESSIBLE in ARM case also doesn't seem right.

Unless you have better idea, backing out the arch/arm change from the subject patch seems to be most safe.

Regards,
Ssantosh

#15 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 9 lines · Santosh Shilimkar
Quoted 8 lines · Yinghai Lu

Index: linux-2.6/include/linux/bootmem.h
===================================================================

include/linux/bootmem.h +0 -0
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
                                                    NUMA_NO_NODE);
 }
 
Quoted 10 lines

include/linux/bootmem.h static inline void * __init memblock_vir · line 187

Quoted 6 lines
include/linux/bootmem.h +3 -0
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif

This won't help mostly since the ARM 32 arch don't set ARCH_LOW_ADDRESS_LIMIT. Sorry i couldn't respond to the thread earlier because of travel and don't have access to my board to try out the patches.

Let's think about this for a moment, shall we...

What does memblock_alloc_virt*() return? It returns a virtual address.

How is that virtual address obtained? ptr = phys_to_virt(alloc);

What is the valid address range for passing into phys_to_virt() ? Only lowmem addresses.

Hence, having ARCH_LOW_ADDRESS_LIMIT set to 4GB-1 by default seems to be completely rediculous - and presumably this also fails on x86_32 if it returns memory up at 4GB.

So... yes, I think reverting the arch/arm part of this patch is the right solution, whether the rest of it should be reverted is something I can't comment on.

Signature · 3 lines
FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up.  Estimation
in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
Estimate before purchase was "up to 13.2Mbit".
#16 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

+ Gryagorii,

Quoted 30 lines · Russell King - ARM Linux
Quoted 9 lines · Santosh Shilimkar
Quoted 8 lines · Yinghai Lu

Index: linux-2.6/include/linux/bootmem.h
===================================================================

include/linux/bootmem.h +0 -0
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
                                                    NUMA_NO_NODE);
 }
 
Quoted 10 lines

include/linux/bootmem.h static inline void * __init memblock_vir · line 187

Quoted 6 lines
include/linux/bootmem.h +3 -0
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif

This won't help mostly since the ARM 32 arch don't set ARCH_LOW_ADDRESS_LIMIT. Sorry i couldn't respond to the thread earlier because of travel and don't have access to my board to try out the patches.

Let's think about this for a moment, shall we...

What does memblock_alloc_virt*() return? It returns a virtual address.

How is that virtual address obtained? ptr = phys_to_virt(alloc);

What is the valid address range for passing into phys_to_virt() ? Only lowmem addresses.

Quoted 8 lines

Hence, having ARCH_LOW_ADDRESS_LIMIT set to 4GB-1 by default seems to be completely rediculous - and presumably this also fails on x86_32 if it returns memory up at 4GB.

So... yes, I think reverting the arch/arm part of this patch is the right solution, whether the rest of it should be reverted is something I can't comment on.

Grygorri mentioned an alternate to update the memblock_find_in_range_node() so that it takes into account the limit.

Regards,
Santosh

#17 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 36 lines · Santosh Shilimkar

+ Gryagorii,

Quoted 34 lines · Russell King - ARM Linux
Quoted 9 lines · Santosh Shilimkar
Quoted 8 lines · Yinghai Lu

Index: linux-2.6/include/linux/bootmem.h
===================================================================

include/linux/bootmem.h +0 -0
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
                                                    NUMA_NO_NODE);
 }
 
Quoted 10 lines

include/linux/bootmem.h static inline void * __init memblock_vir · line 187

Quoted 6 lines
include/linux/bootmem.h +3 -0
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif

This won't help mostly since the ARM 32 arch don't set ARCH_LOW_ADDRESS_LIMIT. Sorry i couldn't respond to the thread earlier because of travel and don't have access to my board to try out the patches.

Let's think about this for a moment, shall we...

What does memblock_alloc_virt*() return? It returns a virtual address.

How is that virtual address obtained? ptr = phys_to_virt(alloc);

What is the valid address range for passing into phys_to_virt() ? Only lowmem addresses.

Hence, having ARCH_LOW_ADDRESS_LIMIT set to 4GB-1 by default seems to be completely rediculous - and presumably this also fails on x86_32 if it returns memory up at 4GB.

Quoted 6 lines
Quoted 4 lines

So... yes, I think reverting the arch/arm part of this patch is the right solution, whether the rest of it should be reverted is something I can't comment on.

Grygorri mentioned an alternate to update the memblock_find_in_range_node() so that it takes into account the limit.

This patch breaks also Xen and 32-bit guests (see
http://lists.xen.org/archives/html/xen-devel/2014-01/msg02476.html)

Reverting it fixes it.

Quoted 3 lines

Regards,
Santosh

#18 lore raw

RE: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Hi all,

Sorry, for the invalid mail & patch format - have no way to send it properly now.

Suppose there is another way to fix this issue (more generic)
- Correct memblock_virt_allocX() API to limit allocations below memblock.current_limit (patch attached).

Then the code behavior will become more similar to _alloc_memory_core_early.

Not tested.

Best regards,
- grygorii

________________________________________
From: Konrad Rzeszutek Wilk [[email protected]]
Sent: Tuesday, January 28, 2014 8:56 PM
To: Shilimkar, Santosh
Cc: Russell King - ARM Linux; Yinghai Lu; Kevin Hilman; Olof Johansson; Linus Torvalds; Andrew Morton; Ingo Molnar; H. Peter Anvin; Dave Hansen; [email protected]; Strashko, Grygorii; [email protected]
Subject: Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 36 lines · Santosh Shilimkar

+ Gryagorii,

Quoted 34 lines · Russell King - ARM Linux
Quoted 9 lines · Santosh Shilimkar
Quoted 8 lines · Yinghai Lu

Index: linux-2.6/include/linux/bootmem.h
===================================================================

include/linux/bootmem.h +0 -0
@@ -179,6 +179,9 @@ static inline void * __init memblock_vir
                                                    NUMA_NO_NODE);
 }
 
Quoted 10 lines

include/linux/bootmem.h static inline void * __init memblock_vir · line 187

Quoted 6 lines
include/linux/bootmem.h +3 -0
+/* Take arch's ARCH_LOW_ADDRESS_LIMIT at first*/
+#include <asm/processor.h>
+
 #ifndef ARCH_LOW_ADDRESS_LIMIT
 #define ARCH_LOW_ADDRESS_LIMIT  0xffffffffUL
 #endif

This won't help mostly since the ARM 32 arch don't set ARCH_LOW_ADDRESS_LIMIT. Sorry i couldn't respond to the thread earlier because of travel and don't have access to my board to try out the patches.

Let's think about this for a moment, shall we...

What does memblock_alloc_virt*() return? It returns a virtual address.

How is that virtual address obtained? ptr = phys_to_virt(alloc);

What is the valid address range for passing into phys_to_virt() ? Only lowmem addresses.

Hence, having ARCH_LOW_ADDRESS_LIMIT set to 4GB-1 by default seems to be completely rediculous - and presumably this also fails on x86_32 if it returns memory up at 4GB.

Quoted 6 lines
Quoted 4 lines

So... yes, I think reverting the arch/arm part of this patch is the right solution, whether the rest of it should be reverted is something I can't comment on.

Grygorri mentioned an alternate to update the memblock_find_in_range_node() so that it takes into account the limit.

This patch breaks also Xen and 32-bit guests (see
http://lists.xen.org/archives/html/xen-devel/2014-01/msg02476.html)

Reverting it fixes it.

Quoted 3 lines

Regards,
Santosh

From ee31afb9c5c0e78819ce624e3a930d31b97527cd Mon Sep 17 00:00:00 2001
From: grygoriis <[email protected]>
Date: Tue, 28 Jan 2014 21:59:30 +0200
Subject: [PATCH] mm/memblock: fix upper boundary of allocating region

Correct memblock_virt_allocX() API to limit allocations below memblock.current_limit.

---
 mm/memblock.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
mm/memblock.c +3 -0
index 87d21a6..e93d669 100644
@@ -1077,6 +1077,9 @@ static void * __init memblock_virt_alloc_internal(
 	if (!align)
 		align = SMP_CACHE_BYTES;
 
+        if (max_addr > memblock.current_limit)
+                max_addr = memblock.current_limit;
+
 again:
 	alloc = memblock_find_in_range_node(size, align, min_addr, max_addr,
 					    nid);
Signature · 1 line
1.7.4.1
#19 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

In reply to Russell King - ARM Linux
Quoted 4 lines · Russell King - ARM Linux

Hence, having ARCH_LOW_ADDRESS_LIMIT set to 4GB-1 by default seems to be completely rediculous - and presumably this also fails on x86_32 if it returns memory up at 4GB.

Agreed. That looks broken even on x86-32. The low address limit is not even *close* to 4GB in general on 32-bit, since you not only have the TASK_SIZE, you have the kmap and the vmalloc area. On x86-32, ARCH_LOW_ADDRESS_LIMIT should be MAXMEM, which iirc is somewhere around 890MB or so. Not 4G.

Linus

author
#20 lore raw

Re: [PATCH 1/3] memblock, nobootmem: Add memblock_virt_alloc_low()

Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Russell King - ARM Linux

Hence, having ARCH_LOW_ADDRESS_LIMIT set to 4GB-1 by default seems to be completely rediculous - and presumably this also fails on x86_32 if it returns memory up at 4GB.

Quoted 6 lines

Agreed. That looks broken even on x86-32. The low address limit is not even *close* to 4GB in general on 32-bit, since you not only have the TASK_SIZE, you have the kmap and the vmalloc area. On x86-32, ARCH_LOW_ADDRESS_LIMIT should be MAXMEM, which iirc is somewhere around 890MB or so. Not 4G.

yeah, Please check the patch that one minute ago.

Subject: [PATCH] memblock: Add limit checking to memblock_virt_alloc

In original bootmem wrapper for memblock, we have limit checking.

Add it to memblock_virt_alloc, to address arm and x86 booting crash.

---
 mm/memblock.c |    3 +++
 1 file changed, 3 insertions(+)
mm/memblock.c +3 -0
Index: linux-2.6/mm/memblock.c
===================================================================
@@ -1077,6 +1077,9 @@ static void * __init memblock_virt_alloc
     if (!align)
         align = SMP_CACHE_BYTES;
 
+    if (max_addr > memblock.current_limit)
+        max_addr = memblock.current_limit;
+
 again:
     alloc = memblock_find_in_range_node(size, align, min_addr, max_addr,
                         nid);

Thanks

Yinghai

author
#21 lore raw

PATCH 2/3 x86: Revert wrong memblock current limit setting.

In reply to Yinghai Lu

Dave reported big numa system booting is broken.

It turns out

| commit 5b6e529521d35e1bcaa0fe43456d1bbb335cae5d
| Author: Santosh Shilimkar <[email protected]>
| Date:   Tue Jan 21 15:50:03 2014 -0800
|
|    x86: memblock: set current limit to max low memory address

set limit to low wrongly.

max_low_pfn_mapped is different from max_pfn_mapped. max_low_pfn_mapped is always under 4G.

That will memblock_alloc_nid all go under 4G.

Revert that offending patch.

---
 arch/x86/include/asm/page_types.h |    4 ++--
 arch/x86/kernel/setup.c           |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/page_types.h +2 -2
Index: linux-2.6/arch/x86/include/asm/page_types.h
===================================================================
@@ -51,9 +51,9 @@ extern int devmem_is_allowed(unsigned lo
 extern unsigned long max_low_pfn_mapped;
 extern unsigned long max_pfn_mapped;
 
-static inline phys_addr_t get_max_low_mapped(void)
+static inline phys_addr_t get_max_mapped(void)
 {
-	return (phys_addr_t)max_low_pfn_mapped << PAGE_SHIFT;
+	return (phys_addr_t)max_pfn_mapped << PAGE_SHIFT;
 }
 
 bool pfn_range_is_mapped(unsigned long start_pfn, unsigned long end_pfn);
arch/x86/kernel/setup.c +1 -1
Index: linux-2.6/arch/x86/kernel/setup.c
===================================================================
@@ -1173,7 +1173,7 @@ void __init setup_arch(char **cmdline_p)
 
 	setup_real_mode();
 
-	memblock_set_current_limit(get_max_low_mapped());
+	memblock_set_current_limit(get_max_mapped());
 	dma_contiguous_reserve(0);
 
 	/*
author
#22 lore raw

PATCH 3/3 memblock: Don't align size silent in memblock_virt_alloc()

In reply to Yinghai Lu

In original __alloc_memory_core_early() for bootmem wrapper, we do not align size silently.

We should not do that, as later free with old size will leave some range not freed.

It's obvious that code is copied from memblock_base_nid(), and that code is wrong for the same reason.

Also remove that in memblock_alloc_base.

---
 mm/memblock.c |    6 ------
 1 file changed, 6 deletions(-)
mm/memblock.c +0 -6
Index: linux-2.6/mm/memblock.c
===================================================================
@@ -981,9 +981,6 @@ static phys_addr_t __init memblock_alloc
 	if (!align)
 		align = SMP_CACHE_BYTES;
 
-	/* align @size to avoid excessive fragmentation on reserved array */
-	size = round_up(size, align);
-
 	found = memblock_find_in_range_node(size, align, 0, max_addr, nid);
 	if (found && !memblock_reserve(found, size))
 		return found;
@@ -1077,9 +1074,6 @@ static void * __init memblock_virt_alloc
 	if (!align)
 		align = SMP_CACHE_BYTES;
 
-	/* align @size to avoid excessive fragmentation on reserved array */
-	size = round_up(size, align);
-
 again:
 	alloc = memblock_find_in_range_node(size, align, min_addr, max_addr,
 					    nid);
Acked-by on the v1 series
#23 lore raw
In reply to Yinghai Lu

Yinghai,

Quoted 5 lines · Yinghai Lu

Big numa system boot get broken while switch API from bootmem to memblock_virt.

Revert the offending patch, and also address swiotlb regression.

Thanks a lot for fixes and help to narrow down and fix these regressions. For all the patches in the series,

maintainer
#24 lore raw
Quoted 6 lines · On Fri, 24 Jan 2014 14:33:19 -0500 Santosh Shilimkar…

Yinghai,

Quoted 3 lines · Yinghai Lu

Big numa system boot get broken while switch API from bootmem to memblock_virt.

Quoted 6 lines
Quoted 2 lines

Revert the offending patch, and also address swiotlb regression.

Thanks a lot for fixes and help to narrow down and fix these regressions. For all the patches in the series,

That patchset continues to boot happily on my swiotlb test box.