Skip to contentSkip to conversation

arm64: mm: move dma_contiguous_reserve() to be after paging_init()

PATCH In review Barry Song 6 participants 9 messages Sep 16, 2020 – Sep 17, 2020

Subsystems (3)
lore
Share & tools
lore

Barry Song

author
#0 lore raw

PATCH 1/1 arm64: mm: move dma_contiguous_reserve() to be after paging_init()

Recent CMA change "cma: make number of CMA areas dynamic, remove CONFIG_CMA_AREAS" breaks the boot of arm64 kernel in linux-next. Knic is like:

Unable to handle kernel paging request at virtual address ffff0000438fff70

Mem abort info:
ESR = 0x96000044
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000044
CM = 0, WnR = 1
swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000041f61000 [ffff0000438fff70] pgd=0000000000000000, p4d=0000000000000000

Internal error: Oops: 96000044 [#1] PREEMPT SMP
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Not tainted 5.9.0-rc3-00020-ge1bce3d64c48 #2
Hardware name: linux,dummy-virt (DT)
pstate: 20000085 (nzCv daIf -PAN -UAO BTYPE=--)
pc : __memset+0x148/0x188
lr : memblock_alloc_try_nid+0xbc/0xd4

sp : ffff800011ab3d10
x29: ffff800011ab3d10 x28: 0000000041710018
x27: 0000000040000000 x26: ffff8000115d1000
x25: 0000000000000000 x24: ffff800011300428
x23: ffff800011d1bd60 x22: 0000000000000000
x21: 00000000ffffffff x20: ffff0000438fff70
x19: 0000000000000090 x18: 0000000000000010
x17: 0000000000001400 x16: 0000000000001c00
x15: ffff800011ac3530 x14: ffff800011ac3530
x13: fffffdfffe600000 x12: ffff800011ab3e44
x11: 0000000000000004 x10: 0000000000000018 x9 : 0000000000000000 x8 : ffff0000438fff70 x7 : 0000000000000000 x6 : 000000000000003f x5 : 0000000000000040 x4 : 0000000000000010 x3 : 0000000000000080 x2 : 0000000000000080 x1 : 0000000000000000 x0 : ffff0000438fff70

Call trace:
 __memset+0x148/0x188
 cma_init_reserved_mem+0x94/0x154
 cma_declare_contiguous_nid+0x240/0x2bc
 dma_contiguous_reserve_area+0x48/0x78
 dma_contiguous_reserve+0x78/0x88
 arm64_memblock_init+0x424/0x45c
 setup_arch+0x270/0x5f0
 start_kernel+0x84/0x4dc
Code: f101007f fa45a068 54fffc0b aa0303e2 (a9001d07)
random: get_random_bytes called from print_oops_end_marker+0x2c/0x68 with crng_init=0
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

The virtual address returned from memblock_alloc() is not ready till paging_init() is done.

Cc Roman Gushchin <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Will Deacon <[email protected]>
Cc Stephen Rothwell <[email protected]>

---
-v1: to fix the knic during boot after applying Mike's patch:
https://lore.kernel.org/linux-mm/[email protected]/

 arch/arm64/mm/init.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
arch/arm64/mm/init.c +7 -2
index f1c75957ff3c..8dd61d07fff5 100644
@@ -402,8 +402,6 @@ void __init arm64_memblock_init(void)
 	reserve_elfcorehdr();
 
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
-
-	dma_contiguous_reserve(arm64_dma32_phys_limit);
 }
 
 void __init bootmem_init(void)
@@ -415,6 +413,13 @@ void __init bootmem_init(void)
 
 	early_memtest(min << PAGE_SHIFT, max << PAGE_SHIFT);
 
+	/*
+	 * CMA is using memblock_alloc(), the virtual address returned
+	 * from memblock_alloc() isn't ready till paging_init().
+	 * So this has to happen after paging_init()
+	 */
+	dma_contiguous_reserve(arm64_dma32_phys_limit);
+
 	max_pfn = max_low_pfn = max;
 	min_low_pfn = min;
 
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/1 arm64: mm: move dma_contiguous_reserve() to be after paging_init()

Quoted 56 lines

Recent CMA change "cma: make number of CMA areas dynamic, remove CONFIG_CMA_AREAS" breaks the boot of arm64 kernel in linux-next. Knic is like:

Unable to handle kernel paging request at virtual address ffff0000438fff70

Mem abort info:
ESR = 0x96000044
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000044
CM = 0, WnR = 1
swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000041f61000 [ffff0000438fff70] pgd=0000000000000000, p4d=0000000000000000

Internal error: Oops: 96000044 [#1] PREEMPT SMP
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Not tainted 5.9.0-rc3-00020-ge1bce3d64c48 #2
Hardware name: linux,dummy-virt (DT)
pstate: 20000085 (nzCv daIf -PAN -UAO BTYPE=--)
pc : __memset+0x148/0x188
lr : memblock_alloc_try_nid+0xbc/0xd4

sp : ffff800011ab3d10
x29: ffff800011ab3d10 x28: 0000000041710018
x27: 0000000040000000 x26: ffff8000115d1000
x25: 0000000000000000 x24: ffff800011300428
x23: ffff800011d1bd60 x22: 0000000000000000
x21: 00000000ffffffff x20: ffff0000438fff70
x19: 0000000000000090 x18: 0000000000000010
x17: 0000000000001400 x16: 0000000000001c00
x15: ffff800011ac3530 x14: ffff800011ac3530
x13: fffffdfffe600000 x12: ffff800011ab3e44
x11: 0000000000000004 x10: 0000000000000018 x9 : 0000000000000000 x8 : ffff0000438fff70 x7 : 0000000000000000 x6 : 000000000000003f x5 : 0000000000000040 x4 : 0000000000000010 x3 : 0000000000000080 x2 : 0000000000000080 x1 : 0000000000000000 x0 : ffff0000438fff70

Call trace:
 __memset+0x148/0x188
 cma_init_reserved_mem+0x94/0x154
 cma_declare_contiguous_nid+0x240/0x2bc
 dma_contiguous_reserve_area+0x48/0x78
 dma_contiguous_reserve+0x78/0x88
 arm64_memblock_init+0x424/0x45c
 setup_arch+0x270/0x5f0
 start_kernel+0x84/0x4dc
Code: f101007f fa45a068 54fffc0b aa0303e2 (a9001d07)
random: get_random_bytes called from print_oops_end_marker+0x2c/0x68 with crng_init=0
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

The virtual address returned from memblock_alloc() is not ready till paging_init() is done.

Quoted 6 lines
Cc Roman Gushchin <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Will Deacon <[email protected]>
Cc Stephen Rothwell <[email protected]>

Cheers,
Anders

#2 lore raw
In reply to Barry Song

It looks like that change referenced may also break arm32 boots with today's next?

The following allows me to boot, but I have no idea if it's incorrect or not.

arch/arm/mm/init.c +3 -3
index 45f9d5ec2360..7118b98c1f5f 100644
@@ -226,9 +226,6 @@ void __init arm_memblock_init(const struct machine_desc *mdesc)
        early_init_fdt_reserve_self();
        early_init_fdt_scan_reserved_mem();
 
-       /* reserve memory for DMA contiguous allocations */
-       dma_contiguous_reserve(arm_dma_limit);
-
        arm_memblock_steal_permitted = false;
        memblock_dump_all();
 }
@@ -248,6 +245,9 @@ void __init bootmem_init(void)
         */
        sparse_init();
 
+       /* reserve memory for DMA contiguous allocations */
+       dma_contiguous_reserve(arm_dma_limit);
+
        /*
         * Now free the memory - free_area_init needs
         * the sparse mem_map arrays initialized by sparse_init()
#3 lore raw
Quoted 4 lines · Nick Desaulniers

It looks like that change referenced may also break arm32 boots with today's next?

The following allows me to boot, but I have no idea if it's incorrect or not.

Thanks Nick,

The referenced commit was pulled from Andrew's tree and subsequent versions of next.

I'm looking into other architectures as this is dependent on where in arch specific boot process first cma call is made. Hopefully, there is a some way to do this without potentially touching a bunch of arch code.

Signature · 1 line
Mike Kravetz
#4 lore raw
Quoted 20 lines

-----Original Message-----
From: Mike Kravetz [mailto:[email protected]]
Sent: Thursday, September 17, 2020 12:27 PM
To: Nick Desaulniers <[email protected]>; Song Bao Hua (Barry Song) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Linuxarm <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: arm64: mm: move dma_contiguous_reserve() to be after paging_init()

Quoted 4 lines · Nick Desaulniers

It looks like that change referenced may also break arm32 boots with today's next?

The following allows me to boot, but I have no idea if it's incorrect or not.

Quoted 8 lines

Thanks Nick,

The referenced commit was pulled from Andrew's tree and subsequent versions of next.

I'm looking into other architectures as this is dependent on where in arch specific boot process first cma call is made. Hopefully, there is a some way to do this without potentially touching a bunch of arch code.

The default cma is probably not the only who has been broken. I think the patch to enable the dynamic number of cma areas has also broken fdt-based CMA. For example, early_init_fdt_scan_reserved_mem() is called before paging_init() in arm64.

Considering many arch are calling early_init_fdt_scan_reserved_mem(), it seems using memblock_alloc() for cma_area will require lots of test on different platforms.
1. arc
2.arm
3.arm64
4.csky
5.h8300
6.mips
7.nsd32
8.nios32
9.openrise
10.powerpc
11.riscv
12.sh
13.xtensa

Quoted 2 lines

--
Mike Kravetz

Thanks
Barry

maintainer
#5 lore raw
In reply to Mike Kravetz

Hi Mike,

Quoted 10 lines · Mike Kravetz
Quoted 4 lines · Nick Desaulniers

It looks like that change referenced may also break arm32 boots with today's next?

The following allows me to boot, but I have no idea if it's incorrect or not.

Thanks Nick,

The referenced commit was pulled from Andrew's tree and subsequent versions of next.

By "pulled" do you mean removed? (potential terminology clash with git pull is confusing me here!).

Quoted 3 lines

I'm looking into other architectures as this is dependent on where in arch specific boot process first cma call is made. Hopefully, there is a some way to do this without potentially touching a bunch of arch code.

Ok, for now I won't queue this arm64 patch, then.

Cheers,

Will

#6 lore raw

Hi Will,

Quoted 3 lines · On Thu, 17 Sep 2020 10:02:35 +0100 Will Deacon <[email protected]> wrote:

By "pulled" do you mean removed? (potential terminology clash with git pull is confusing me here!).

"removed" It will not be in today's linux-next.

Signature · 2 lines
Cheers,
Stephen Rothwell
maintainer
#7 lore raw
Quoted 6 lines · Stephen Rothwell
Quoted 3 lines · On Thu, 17 Sep 2020 10:02:35 +0100 Will Deacon <[email protected]> wrote:

By "pulled" do you mean removed? (potential terminology clash with git pull is confusing me here!).

"removed" It will not be in today's linux-next.

Perfect, thanks Stephen.

Will

#8 lore raw
In reply to Nick Desaulniers
Quoted 15 lines

-----Original Message-----
From: ndesaulniers via sendgmr [mailto:[email protected]] On Behalf Of Nick Desaulniers
Sent: Thursday, September 17, 2020 12:20 PM
To: Song Bao Hua (Barry Song) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Linuxarm <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: arm64: mm: move dma_contiguous_reserve() to be after paging_init()

Quoted 5 lines

It looks like that change referenced may also break arm32 boots with today's next?

The following allows me to boot, but I have no idea if it's incorrect or not.

This is probably incorrect on arm32 as dma_contiguous_early_fixup() is done after dma_contiguous_remap() in paging_init(). That means dma_contiguous_remap() gets an empty list.
For arm64, there isn't early_fixup() and remap().

Quoted 16 lines
arch/arm/mm/init.c +0 -3
index 45f9d5ec2360..7118b98c1f5f 100644
@@ -226,9 +226,6 @@ void __init arm_memblock_init(const struct
 machine_desc *mdesc)
        early_init_fdt_reserve_self();
        early_init_fdt_scan_reserved_mem();
 
-       /* reserve memory for DMA contiguous allocations */
-       dma_contiguous_reserve(arm_dma_limit);
-
        arm_memblock_steal_permitted = false;
        memblock_dump_all();

}

arch/arm/mm/init.c void __init bootmem_init(void) · line 253

Quoted 9 lines
arch/arm/mm/init.c +3 -0
@@ -248,6 +245,9 @@ void __init bootmem_init(void)
         */
        sparse_init();
 
+       /* reserve memory for DMA contiguous allocations */
+       dma_contiguous_reserve(arm_dma_limit);
+
        /*
         * Now free the memory - free_area_init needs
         * the sparse mem_map arrays initialized by sparse_init()

Thanks
Barry