Skip to contentSkip to conversation

Bad virt_to_phys since commit 54c7a8916a887f35

DISCUSSION Mark Rutland 5 participants 13 messages May 16, 2019 – May 23, 2019 VFS

lore
Share & tools
lore
#0 lore raw

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

Thanks,
Mark.

[    5.251023][    T1] ------------[ cut here ]------------
[    5.252465][    T1] virt_to_phys used for non-linear address: (____ptrval____) (0x0)
[    5.254388][    T1] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x88/0xb8
[    5.256473][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.1.0-11058-g83f3ef3 #4
[    5.258513][    T1] Hardware name: linux,dummy-virt (DT)
[    5.259923][    T1] pstate: 80400005 (Nzcv daif +PAN -UAO)
[    5.261375][    T1] pc : __virt_to_phys+0x88/0xb8
[    5.262623][    T1] lr : __virt_to_phys+0x88/0xb8
[    5.263879][    T1] sp : ffff80000be4fc60
[    5.264941][    T1] x29: ffff80000be4fc60 x28: 0000000040000000 
[    5.266522][    T1] x27: ffff200015445000 x26: 0000000000000000 
[    5.268112][    T1] x25: 0000000000000000 x24: ffff2000163e0000 
[    5.269691][    T1] x23: ffff2000163e0440 x22: ffff2000163e0000 
[    5.271270][    T1] x21: ffff2000163e0400 x20: 0000000000000000 
[    5.272860][    T1] x19: 0000000000000000 x18: ffff200016aa0f80 
[    5.274430][    T1] x17: ffff2000153a0000 x16: 00000000f2000000 
[    5.276018][    T1] x15: 1fffe40002d5560d x14: 1ffff00007716109 
[    5.277596][    T1] x13: ffff200016e17000 x12: ffff040002a83fd9 
[    5.279179][    T1] x11: 1fffe40002a83fd8 x10: ffff040002a83fd8 
[    5.280765][    T1] x9 : 1fffe40002a83fd8 x8 : dfff200000000000 
[    5.282343][    T1] x7 : ffff040002a83fd9 x6 : ffff20001541fec0 
[    5.283929][    T1] x5 : ffff80003b8b0040 x4 : 0000000000000000 
[    5.285509][    T1] x3 : ffff2000102c6504 x2 : ffff1000017c9f54 
[    5.287091][    T1] x1 : 15eab2dadba38000 x0 : 0000000000000000 
[    5.288678][    T1] Call trace:
[    5.289532][    T1]  __virt_to_phys+0x88/0xb8
[    5.290701][    T1]  free_initrd_mem+0x3c/0x50
[    5.291894][    T1]  populate_rootfs+0x2f4/0x358
[    5.293123][    T1]  do_one_initcall+0x568/0xb94
[    5.294349][    T1]  kernel_init_freeable+0xd44/0xe2c
[    5.295695][    T1]  kernel_init+0x14/0x1c0
[    5.296814][    T1]  ret_from_fork+0x10/0x1c
[    5.297947][    T1] irq event stamp: 288672
[    5.299069][    T1] hardirqs last  enabled at (288671): [<ffff2000102c4cac>] console_unlock+0x89c/0xe50
[    5.301521][    T1] hardirqs last disabled at (288672): [<ffff2000100823e0>] do_debug_exception+0x268/0x3e0
[    5.304061][    T1] softirqs last  enabled at (288668): [<ffff2000100835e0>] __do_softirq+0xa38/0xf48
[    5.306457][    T1] softirqs last disabled at (288653): [<ffff2000101ac994>] irq_exit+0x2a4/0x318
[    5.308777][    T1] ---[ end trace 3cf83e3c184a4d3e ]---
#1 lore raw
Quoted 5 lines · Mark Rutland

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

Ugh, I dropped a paragarph here.

Since that commit, I'm seeing a boot-time splat on arm64 when using CONFIG_DEBUG_VIRTUAL. I'm running an arm64 syzkaller instance, and this kills the VM, preventing further testing, which is unfortunate.

Mark.

Quoted 49 lines

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

Thanks,
Mark.

[    5.251023][    T1] ------------[ cut here ]------------
[    5.252465][    T1] virt_to_phys used for non-linear address: (____ptrval____) (0x0)
[    5.254388][    T1] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x88/0xb8
[    5.256473][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.1.0-11058-g83f3ef3 #4
[    5.258513][    T1] Hardware name: linux,dummy-virt (DT)
[    5.259923][    T1] pstate: 80400005 (Nzcv daif +PAN -UAO)
[    5.261375][    T1] pc : __virt_to_phys+0x88/0xb8
[    5.262623][    T1] lr : __virt_to_phys+0x88/0xb8
[    5.263879][    T1] sp : ffff80000be4fc60
[    5.264941][    T1] x29: ffff80000be4fc60 x28: 0000000040000000 
[    5.266522][    T1] x27: ffff200015445000 x26: 0000000000000000 
[    5.268112][    T1] x25: 0000000000000000 x24: ffff2000163e0000 
[    5.269691][    T1] x23: ffff2000163e0440 x22: ffff2000163e0000 
[    5.271270][    T1] x21: ffff2000163e0400 x20: 0000000000000000 
[    5.272860][    T1] x19: 0000000000000000 x18: ffff200016aa0f80 
[    5.274430][    T1] x17: ffff2000153a0000 x16: 00000000f2000000 
[    5.276018][    T1] x15: 1fffe40002d5560d x14: 1ffff00007716109 
[    5.277596][    T1] x13: ffff200016e17000 x12: ffff040002a83fd9 
[    5.279179][    T1] x11: 1fffe40002a83fd8 x10: ffff040002a83fd8 
[    5.280765][    T1] x9 : 1fffe40002a83fd8 x8 : dfff200000000000 
[    5.282343][    T1] x7 : ffff040002a83fd9 x6 : ffff20001541fec0 
[    5.283929][    T1] x5 : ffff80003b8b0040 x4 : 0000000000000000 
[    5.285509][    T1] x3 : ffff2000102c6504 x2 : ffff1000017c9f54 
[    5.287091][    T1] x1 : 15eab2dadba38000 x0 : 0000000000000000 
[    5.288678][    T1] Call trace:
[    5.289532][    T1]  __virt_to_phys+0x88/0xb8
[    5.290701][    T1]  free_initrd_mem+0x3c/0x50
[    5.291894][    T1]  populate_rootfs+0x2f4/0x358
[    5.293123][    T1]  do_one_initcall+0x568/0xb94
[    5.294349][    T1]  kernel_init_freeable+0xd44/0xe2c
[    5.295695][    T1]  kernel_init+0x14/0x1c0
[    5.296814][    T1]  ret_from_fork+0x10/0x1c
[    5.297947][    T1] irq event stamp: 288672
[    5.299069][    T1] hardirqs last  enabled at (288671): [<ffff2000102c4cac>] console_unlock+0x89c/0xe50
[    5.301521][    T1] hardirqs last disabled at (288672): [<ffff2000100823e0>] do_debug_exception+0x268/0x3e0
[    5.304061][    T1] softirqs last  enabled at (288668): [<ffff2000100835e0>] __do_softirq+0xa38/0xf48
[    5.306457][    T1] softirqs last disabled at (288653): [<ffff2000101ac994>] irq_exit+0x2a4/0x318
[    5.308777][    T1] ---[ end trace 3cf83e3c184a4d3e ]---
#2 lore raw
Quoted 7 lines · Mark Rutland
Quoted 5 lines · Mark Rutland

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

Quoted 15 lines

Ugh, I dropped a paragarph here.

Since that commit, I'm seeing a boot-time splat on arm64 when using CONFIG_DEBUG_VIRTUAL. I'm running an arm64 syzkaller instance, and this kills the VM, preventing further testing, which is unfortunate.

Mark.

Quoted 7 lines

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

Your analysis looks right to me. In my review I'd managed to spot the change in behaviour when CONFIG_INITRAMFS_FORCE is set (the initrd is freed), but I'd overlooked what happens if initrd_start == 0 (the non-existent initrd is attempted to be freed).

I suspect the following is sufficient to fix the problem:

----8<-----

init/initramfs.c +1 -1
index 435a428c2af1..178130fd61c2 100644
@@ -669,7 +669,7 @@ static int __init populate_rootfs(void)
 	 * If the initrd region is overlapped with crashkernel reserved region,
 	 * free only memory that is not part of crashkernel region.
 	 */
-	if (!do_retain_initrd && !kexec_free_initrd())
+	if (!do_retain_initrd && initrd_start && !kexec_free_initrd())
 		free_initrd_mem(initrd_start, initrd_end);
 	initrd_start = 0;
 	initrd_end = 0;
author Tested-by
#3 lore raw
Quoted 37 lines · Steven Price
Quoted 7 lines · Mark Rutland
Quoted 5 lines · Mark Rutland

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

Quoted 15 lines

Ugh, I dropped a paragarph here.

Since that commit, I'm seeing a boot-time splat on arm64 when using CONFIG_DEBUG_VIRTUAL. I'm running an arm64 syzkaller instance, and this kills the VM, preventing further testing, which is unfortunate.

Mark.

Quoted 7 lines

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

Your analysis looks right to me. In my review I'd managed to spot the change in behaviour when CONFIG_INITRAMFS_FORCE is set (the initrd is freed), but I'd overlooked what happens if initrd_start == 0 (the non-existent initrd is attempted to be freed).

I suspect the following is sufficient to fix the problem:

----8<-----

init/initramfs.c +0 -0
index 435a428c2af1..178130fd61c2 100644

init/initramfs.c static int __init populate_rootfs(void) · line 675

Quoted 8 lines
init/initramfs.c +1 -1
@@ -669,7 +669,7 @@ static int __init populate_rootfs(void)
 	 * If the initrd region is overlapped with crashkernel reserved region,
 	 * free only memory that is not part of crashkernel region.
 	 */
-	if (!do_retain_initrd && !kexec_free_initrd())
+	if (!do_retain_initrd && initrd_start && !kexec_free_initrd())
 		free_initrd_mem(initrd_start, initrd_end);
 	initrd_start = 0;
 	initrd_end = 0;

That works for me. If you spin this as a real patch:

As I mentioned, initrd_start has not been initialized at all, so I suspect we should also update its declaration in init/do_mounts_initrd.c such that it is guaranteed to be initialized to zero. We get away with that today, but that won't necessarily hold with LTO and so on...

Thanks,
Mark.

#4 lore raw
Quoted 49 lines · Mark Rutland
Quoted 37 lines · Steven Price
Quoted 7 lines · Mark Rutland
Quoted 5 lines · Mark Rutland

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

Quoted 15 lines

Ugh, I dropped a paragarph here.

Since that commit, I'm seeing a boot-time splat on arm64 when using CONFIG_DEBUG_VIRTUAL. I'm running an arm64 syzkaller instance, and this kills the VM, preventing further testing, which is unfortunate.

Mark.

Quoted 7 lines

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

Your analysis looks right to me. In my review I'd managed to spot the change in behaviour when CONFIG_INITRAMFS_FORCE is set (the initrd is freed), but I'd overlooked what happens if initrd_start == 0 (the non-existent initrd is attempted to be freed).

I suspect the following is sufficient to fix the problem:

----8<-----

init/initramfs.c +0 -0
index 435a428c2af1..178130fd61c2 100644

init/initramfs.c static int __init populate_rootfs(void) · line 675

Quoted 8 lines
init/initramfs.c +1 -1
@@ -669,7 +669,7 @@ static int __init populate_rootfs(void)
 	 * If the initrd region is overlapped with crashkernel reserved region,
 	 * free only memory that is not part of crashkernel region.
 	 */
-	if (!do_retain_initrd && !kexec_free_initrd())
+	if (!do_retain_initrd && initrd_start && !kexec_free_initrd())
 		free_initrd_mem(initrd_start, initrd_end);
 	initrd_start = 0;
 	initrd_end = 0;

That works for me. If you spin this as a real patch:

Quoted 6 lines

As I mentioned, initrd_start has not been initialized at all, so I suspect we should also update its declaration in init/do_mounts_initrd.c such that it is guaranteed to be initialized to zero. We get away with that today, but that won't necessarily hold with LTO and so on...

Well it's a global variable, so the C standard says it should be initialised to 0...

I'll spin a real patch and add your Tested-by

Steve

#5 lore raw
Quoted 23 lines · Steven Price
Quoted 22 lines · Mark Rutland
Quoted 8 lines · Steven Price

I suspect the following is sufficient to fix the problem:

----8<-----

init/initramfs.c +0 -0
index 435a428c2af1..178130fd61c2 100644

init/initramfs.c static int __init populate_rootfs(void) · line 675

Quoted 8 lines
init/initramfs.c +1 -1
@@ -669,7 +669,7 @@ static int __init populate_rootfs(void)
 	 * If the initrd region is overlapped with crashkernel reserved region,
 	 * free only memory that is not part of crashkernel region.
 	 */
-	if (!do_retain_initrd && !kexec_free_initrd())
+	if (!do_retain_initrd && initrd_start && !kexec_free_initrd())
 		free_initrd_mem(initrd_start, initrd_end);
 	initrd_start = 0;
 	initrd_end = 0;

That works for me. If you spin this as a real patch:

Quoted 7 lines
Quoted 4 lines

As I mentioned, initrd_start has not been initialized at all, so I suspect we should also update its declaration in init/do_mounts_initrd.c such that it is guaranteed to be initialized to zero. We get away with that today, but that won't necessarily hold with LTO and so on...

Well it's a global variable, so the C standard says it should be initialised to 0...

For some reason I was under the impression that wasn't guaranteed, but I see that it is. Sorry for the noise.

Quoted 1 line

I'll spin a real patch and add your Tested-by

Great; thanks!

Mark.

#7 lore raw
Quoted 6 lines · Will Deacon

Hi Steven,

Quoted 1 line · Steven Price

I'll spin a real patch and add your Tested-by

Did you send this out? I can't spot it in my inbox.

https://lore.kernel.org/lkml/[email protected]

And Andrew already took it to the -mm tree.

Quoted 4 lines

Cheers,

Will

Signature · 2 lines
Sincerely yours,
Mike.
#8 lore raw
Quoted 11 lines · Mike Rapoport
Quoted 6 lines · Will Deacon

Hi Steven,

Quoted 1 line · Steven Price

I'll spin a real patch and add your Tested-by

Did you send this out? I can't spot it in my inbox.

https://lore.kernel.org/lkml/[email protected]

And Andrew already took it to the -mm tree.

And it's made its way to Linus' tree:
commit 5d59aa8f9ce972b472201aed86e904bb75879ff0

Steve

#9 lore raw
Quoted 7 lines · Steven Price
Quoted 6 lines · Mike Rapoport
Quoted 5 lines · Will Deacon

Hi Steven,

Quoted 1 line · Steven Price

I'll spin a real patch and add your Tested-by

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Did you send this out? I can't spot it in my inbox.

https://lore.kernel.org/lkml/[email protected]

And Andrew already took it to the -mm tree.

And it's made its way to Linus' tree:
commit 5d59aa8f9ce972b472201aed86e904bb75879ff0

Perfect, thanks. Sorry for the noise.

Will

#10 lore raw
In reply to Mark Rutland
Quoted 7 lines · Mark Rutland
Quoted 5 lines · Mark Rutland

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

Quoted 15 lines

Ugh, I dropped a paragarph here.

Since that commit, I'm seeing a boot-time splat on arm64 when using CONFIG_DEBUG_VIRTUAL. I'm running an arm64 syzkaller instance, and this kills the VM, preventing further testing, which is unfortunate.

Mark.

Quoted 7 lines

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

If I remember correctly, initrd_start would be 0 unless explicitly set by the arch setup code, so something like this could work:

init/initramfs.c +3 -0
index 435a428c2af1..05fe60437796 100644
@@ -529,6 +529,9 @@ extern unsigned long __initramfs_size;
 
 void __weak free_initrd_mem(unsigned long start, unsigned long end)
 {
+       if (!start)
+               return;
+
        free_reserved_area((void *)start, (void *)end, POISON_FREE_INITMEM,
                        "initrd");
 }
Quoted 43 lines
Quoted 42 lines

Thanks,
Mark.

[    5.251023][    T1] ------------[ cut here ]------------
[    5.252465][    T1] virt_to_phys used for non-linear address: (____ptrval____) (0x0)
[    5.254388][    T1] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x88/0xb8
[    5.256473][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.1.0-11058-g83f3ef3 #4
[    5.258513][    T1] Hardware name: linux,dummy-virt (DT)
[    5.259923][    T1] pstate: 80400005 (Nzcv daif +PAN -UAO)
[    5.261375][    T1] pc : __virt_to_phys+0x88/0xb8
[    5.262623][    T1] lr : __virt_to_phys+0x88/0xb8
[    5.263879][    T1] sp : ffff80000be4fc60
[    5.264941][    T1] x29: ffff80000be4fc60 x28: 0000000040000000 
[    5.266522][    T1] x27: ffff200015445000 x26: 0000000000000000 
[    5.268112][    T1] x25: 0000000000000000 x24: ffff2000163e0000 
[    5.269691][    T1] x23: ffff2000163e0440 x22: ffff2000163e0000 
[    5.271270][    T1] x21: ffff2000163e0400 x20: 0000000000000000 
[    5.272860][    T1] x19: 0000000000000000 x18: ffff200016aa0f80 
[    5.274430][    T1] x17: ffff2000153a0000 x16: 00000000f2000000 
[    5.276018][    T1] x15: 1fffe40002d5560d x14: 1ffff00007716109 
[    5.277596][    T1] x13: ffff200016e17000 x12: ffff040002a83fd9 
[    5.279179][    T1] x11: 1fffe40002a83fd8 x10: ffff040002a83fd8 
[    5.280765][    T1] x9 : 1fffe40002a83fd8 x8 : dfff200000000000 
[    5.282343][    T1] x7 : ffff040002a83fd9 x6 : ffff20001541fec0 
[    5.283929][    T1] x5 : ffff80003b8b0040 x4 : 0000000000000000 
[    5.285509][    T1] x3 : ffff2000102c6504 x2 : ffff1000017c9f54 
[    5.287091][    T1] x1 : 15eab2dadba38000 x0 : 0000000000000000 
[    5.288678][    T1] Call trace:
[    5.289532][    T1]  __virt_to_phys+0x88/0xb8
[    5.290701][    T1]  free_initrd_mem+0x3c/0x50
[    5.291894][    T1]  populate_rootfs+0x2f4/0x358
[    5.293123][    T1]  do_one_initcall+0x568/0xb94
[    5.294349][    T1]  kernel_init_freeable+0xd44/0xe2c
[    5.295695][    T1]  kernel_init+0x14/0x1c0
[    5.296814][    T1]  ret_from_fork+0x10/0x1c
[    5.297947][    T1] irq event stamp: 288672
[    5.299069][    T1] hardirqs last  enabled at (288671): [<ffff2000102c4cac>] console_unlock+0x89c/0xe50
[    5.301521][    T1] hardirqs last disabled at (288672): [<ffff2000100823e0>] do_debug_exception+0x268/0x3e0
[    5.304061][    T1] softirqs last  enabled at (288668): [<ffff2000100835e0>] __do_softirq+0xa38/0xf48
[    5.306457][    T1] softirqs last disabled at (288653): [<ffff2000101ac994>] irq_exit+0x2a4/0x318
[    5.308777][    T1] ---[ end trace 3cf83e3c184a4d3e ]---
Signature · 2 lines
Sincerely yours,
Mike.
#11 lore raw
Quoted 32 lines · Mike Rapoport
Quoted 7 lines · Mark Rutland
Quoted 5 lines · Mark Rutland

Hi,

Since commit:

54c7a8916a887f35 ("initramfs: free initrd memory if opening /initrd.image fails")

Quoted 15 lines

Ugh, I dropped a paragarph here.

Since that commit, I'm seeing a boot-time splat on arm64 when using CONFIG_DEBUG_VIRTUAL. I'm running an arm64 syzkaller instance, and this kills the VM, preventing further testing, which is unfortunate.

Mark.

Quoted 7 lines

IIUC prior to that commit, we'd only attempt to free an intird if we had one, whereas now we do so unconditionally. AFAICT, in this case initrd_start has not been initialized (I'm not using an initrd or initramfs on my system), so we end up trying virt_to_phys() on a bogus VA in free_initrd_mem().

Any ideas on the right way to fix this?

If I remember correctly, initrd_start would be 0 unless explicitly set by the arch setup code, so something like this could work:

init/initramfs.c +0 -0
index 435a428c2af1..05fe60437796 100644

init/initramfs.c extern unsigned long __initramfs_size; · line 537

Quoted 9 lines
init/initramfs.c +3 -0
@@ -529,6 +529,9 @@ extern unsigned long __initramfs_size;
 
 void __weak free_initrd_mem(unsigned long start, unsigned long end)
 {
+       if (!start)
+               return;
+
        free_reserved_area((void *)start, (void *)end, POISON_FREE_INITMEM,
                        "initrd");
 }

I think this should work, given Steven's patch checks the same thing.

I don't have a preference as to which patch should be taken, so I'll leave that to Christoph.

Thanks,
Mark.

#12 lore raw
Quoted 9 lines · Mark Rutland

init/initramfs.c extern unsigned long __initramfs_size; · line 537

Quoted 8 lines

void __weak free_initrd_mem(unsigned long start, unsigned long end) {

init/initramfs.c +3 -0
+       if (!start)
+               return;
+
        free_reserved_area((void *)start, (void *)end, POISON_FREE_INITMEM,
                        "initrd");
 }
Quoted 4 lines

I think this should work, given Steven's patch checks the same thing.

I don't have a preference as to which patch should be taken, so I'll leave that to Christoph.

We still have plenty of architectures not using the generic free_initrd_mem, so checking it in the caller gives us better coverage.