lockdep warning while booting POWER9 PowerNV
https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at,
if (WARN_ON_ONCE(static_obj(key)))
because
key = 0xc0000000019ad118&_stext = 0xc000000000000000&_end = 0xc0000000049d0000
i.e., it will cause static_obj() returns 1.
[ 23.738200][ T13] ahci 0004:03:00.0: enabling device (0541 -> 0543) [ 23.748692][ T5] tg3.c:v3.137 (May 11, 2014)
[ 23.748731][ T5] tg3 0005:01:00.0: enabling device (0140 -> 0142) [ 23.751478][ T13] ahci 0004:03:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[ 23.751517][ T13] ahci 0004:03:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
[ 23.791077][ T13] scsi host0: ahci
[ 23.802752][ T13] ------------[ cut here ]------------
[ 23.802786][ T13] WARNING: CPU: 0 PID: 13 at kernel/locking/lockdep.c:1120 lockdep_register_key+0x68/0x200
[ 23.802814][ T13] Modules linked in: mdio tg3(+) ahci(+) libahci libphy firmware_class libata dm_mirror dm_region_hash dm_log dm_mod
[ 23.802884][ T13] CPU: 0 PID: 13 Comm: kworker/0:1 Not tainted 5.3.0-rc6- next-20190830 #4
[ 23.802930][ T13] Workqueue: events work_for_cpu_fn
[ 23.802962][ T13] NIP: c00000000019eed8 LR: c000000000129400 CTR:c0000000008a46dc
[ 23.802988][ T13] REGS: c00000002db2f130 TRAP: 0700 Not tainted (5.3.0- rc6-next-20190830)
[ 23.803032][ T13] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48800a89 XER: 20040000
[ 23.803096][ T13] CFAR: c00000000019eeac IRQMASK: 0
[ 23.803096][ T13] GPR00: c000000000129400 c00000002db2f3c0 c000000002bb9400 c0000000049d0000
[ 23.803096][ T13] GPR04: 0000000000000000 c000000002d84718 0000000000000000 ffff0a01ffffff10
[ 23.803096][ T13] GPR08: c000000000b8a16b c000000000a515cc c000000000b8a16a 0000000000000044
[ 23.803096][ T13] GPR12: 0000000088800a89 c0000000049d0000 c00800000eea3290 0000000000000001
[ 23.803096][ T13] GPR16: c000001bca584500 c000000007244954 c000001bca584508 0000000000000020
[ 23.803096][ T13] GPR20: c000000001aea180 0000000000000001 c00800000eee7560 c000001bca584500
[ 23.803096][ T13] GPR24: c000001bca584448 000000000002000a c000000001aea148 c000000000b8a161
[ 23.803096][ T13] GPR28: c000000001aea020 c000000001aea118 c000000001aea000 c00000002db2f440
[ 23.803407][ T13] NIP [c00000000019eed8] lockdep_register_key+0x68/0x200 [ 23.803432][ T13] LR [c000000000129400] wq_init_lockdep+0x40/0xc0 [ 23.803461][ T13] Call Trace:
[ 23.803491][ T13] [c00000002db2f3c0] [c000000000b8a16a] trunc_msg+0x385f9/0x4c30f (unreliable)
[ 23.803531][ T13] [c00000002db2f440] [c000000000129400] wq_init_lockdep+0x40/0xc0
[ 23.803577][ T13] [c00000002db2f4c0] [c000000000128d90] alloc_workqueue+0x1e0/0x620
[ 23.803623][ T13] [c00000002db2f5c0] [c0000000006ca048] scsi_host_alloc+0x3d8/0x490
[ 23.803701][ T13] [c00000002db2f660] [c00800000ee73198] ata_scsi_add_hosts+0xd0/0x220 [libata]
[ 23.803771][ T13] [c00000002db2f6f0] [c00800000ee6d440] ata_host_register+0x178/0x400 [libata]
[ 23.803840][ T13] [c00000002db2f7d0] [c00800000ee6d8f4] ata_host_activate+0x17c/0x210 [libata]
[ 23.803883][ T13] [c00000002db2f880] [c00800000edf4c3c] ahci_host_activate+0x84/0x250 [libahci]
[ 23.803944][ T13] [c00000002db2f940] [c00800000f050c7c] ahci_init_one+0xc74/0xdc0 [ahci]
[ 23.803992][ T13] [c00000002db2faa0] [c00000000062b188] local_pci_probe+0x78/0x100
[ 23.804048][ T13] [c00000002db2fb30] [c00000000012c0b0] work_for_cpu_fn+0x40/0x70
[ 23.804087][ T13] [c00000002db2fb60] [c000000000130328] process_one_work+0x388/0x750
[ 23.804135][ T13] [c00000002db2fc60] [c00000000012fe30] process_scheduled_works+0x50/0x90
[ 23.804180][ T13] [c00000002db2fca0] [c000000000130c50]
worker_thread+0x3d0/0x570 [ 23.804216][ T13] [c00000002db2fda0] [c000000000139a08] kthread+0x1b8/0x1e0
[ 23.804253][ T13] [c00000002db2fe20] [c00000000000b660] ret_from_kernel_thread+0x5c/0x7c
[ 23.804298][ T13] Instruction dump:
[ 23.804326][ T13] fbc10070 4194002c 7fa3eb78 481864a5 60000000 70630001 41810018 7fa3eb78
[ 23.804373][ T13] 48075121 60000000 70630001 40810020 <0fe00000> ebc10070 eba10068 38210080
[ 23.804419][ T13] ---[ end trace 9ceca43eedad2a2b ]---
[ 23.810924][ T5] tg3 0005:01:00.0 eth0: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address 08:94:ef:80:81:c0
[ 23.810967][ T5] tg3 0005:01:00.0 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[ 23.811027][ T5] tg3 0005:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[ 23.811074][ T5] tg3 0005:01:00.0 eth0: dma_rwctrl[00000000] dma_mask[64- bit]
[ 23.811284][ T13] scsi host1: ahci
Quoted 7 lines · Qian Cai
https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in
lockdep_register_key()at,if (WARN_ON_ONCE(static_obj(key)))
Quoted 7 lines
because
key =
0xc0000000019ad118&_stext=0xc000000000000000&_end=0xc0000000049d0000i.e., it will cause
static_obj()returns 1.
(back from a trip)
Hi Qian,
Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end? Since I am not familiar with POWER9 nor have access to such a system, can you propose a patch?
Thanks,
Bart.
Michael Ellerman
reviewerQuoted 14 lines · Bart Van Assche
Quoted 13 lines · Qian Cai
https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in
lockdep_register_key()at,if (WARN_ON_ONCE(static_obj(key)))
because
key =
0xc0000000019ad118&_stext=0xc000000000000000&_end=0xc0000000049d0000
Quoted 8 lines
Quoted 1 line
i.e., it will cause
static_obj()returns 1.(back from a trip)
Hi Qian,
Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between
&_stextand&_end?
I thought that was true on all arches due to initmem, but seems not.
I guess we have the same problem as s390 and we need to define arch_is_kernel_initmem_freed().
Qian, can you try this:
arch/powerpc/include/asm/sections.h +14 -0
@@ -5,8 +5,22 @@#include <linux/elf.h> #include <linux/uaccess.h> + +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed +#include <asm-generic/sections.h> +extern bool init_mem_is_free; + +static inline int arch_is_kernel_initmem_freed (unsigned long addr) +{ +if (!init_mem_is_free) +return 0 ; + +return addr >= (unsigned long )__init_begin && + addr < (unsigned long )__init_end; +} +extern char __head_end[];#ifdef __powerpc64__
cheers
Quoted 22 lines · Michael Ellerman
Quoted 21 lines · Bart Van Assche
Quoted 8 lines · Qian Cai
https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate
a warning inlockdep_register_key()at,if (WARN_ON_ONCE(static_obj(key)))
Quoted 7 lines
because
key =
0xc0000000019ad118&_stext=0xc000000000000000&_end=0xc0000000049d0000i.e., it will cause
static_obj()returns 1.(back from a trip)
Hi Qian,
Quoted 7 lines
Quoted 2 lines
Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between
&_stextand&_end?I thought that was true on all arches due to initmem, but seems not.
I guess we have the same problem as s390 and we need to define
arch_is_kernel_initmem_freed().
Actually, it is in the .bss section. The commit 2d4f567103ff ("KVM: PPC:
Introduce kvm_tmp framework") adds kvm_tmp[] into the .bss section and then free the rest of unused spaces back to the page allocator.
kernel_init
kvm_guest_init
kvm_free_tmp
free_reserved_area
free_unref_page
free_unref_page_prepare
Later, alloc_workqueue() happens to allocate some pages from there, and triggers the warning. Not sure what the best way to solve this.
Quoted 53 lines · Michael Ellerman
Quoted 14 lines · Bart Van Assche
Quoted 13 lines · Qian Cai
https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in
lockdep_register_key()at,if (WARN_ON_ONCE(static_obj(key)))
because
key =
0xc0000000019ad118&_stext=0xc000000000000000&_end=0xc0000000049d0000Quoted 8 lines
Quoted 1 line
i.e., it will cause
static_obj()returns 1.(back from a trip)
Hi Qian,
Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between
&_stextand&_end?I thought that was true on all arches due to initmem, but seems not.
I guess we have the same problem as s390 and we need to define
arch_is_kernel_initmem_freed().Qian, can you try this:
arch/powerpc/include/asm/sections.h +7
-0@@ -5,8 +5,22 @@#include <linux/elf.h> #include <linux/uaccess.h> + +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed +#include <asm-generic/sections.h> +extern bool init_mem_is_free; + +static inline int arch_is_kernel_initmem_freed (unsigned long addr) +{
arch/powerpc/include/asm/sections.h line 26
Quoted 10 lines
arch/powerpc/include/asm/sections.h +7
-0+if (!init_mem_is_free) +return 0 ; + +return addr >= (unsigned long )__init_begin && + addr < (unsigned long )__init_end; +} +extern char __head_end[];#ifdef __powerpc64__
Michael, this fix is also needed as it starts to trigger another one of those where the allocated memory is from initmem.
[ 31.326825] key = c0000000019049a0 [ 31.326862] stext = c000000000000000, end = c0000000070e0000 [ 31.326907] init_start = c000000000c70000, init_end = c0000000020f0000
[ 31.325021] WARNING: CPU: 0 PID: 5 at kernel/locking/lockdep.c:1121 lockdep_register_key+0xb4/0x340
[ 31.325061] Modules linked in: tg3(+) ahci(+) libahci libata mdio libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod
[ 31.325128] CPU: 0 PID: 5 Comm: kworker/0:0 Not tainted 5.4.0-rc8-next-20191120+ #4
[ 31.325190] Workqueue: events work_for_cpu_fn
[ 31.325215] NIP: c0000000001a23a4 LR: c00000000075eccc CTR: 0000000000000000
[ 31.325257] REGS: c00000002e72f4c0 TRAP: 0700 Not tainted (5.4.0-rc8-next-20191120+)
[ 31.325320] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48000c20 XER: 20040000
[ 31.325392] CFAR: c0000000001a233c IRQMASK: 0
GPR00: c00000000075eccc c00000002e72f750 c000000002cff500 c0000000070df500
GPR04: c0000014beb01990 c0000000019042b8 0000000000000000 0000000000000000
GPR08: 0000000000000000 0000000000000000 c000000000425e28 c00c000004761020
GPR12: 0000000000000000 c0000000070e0000 c00000002e5214f8 c000001ffca018c8
GPR16: c000001ffca018e4 c000001ffca01c80 c000001ffca018d0 c00000002e6e3e48
GPR20: c000000002cbf500 c00000002e520080 c000001ffca05408 c00000002e6e3e00
GPR24: c0000000007d36d0 0000000000000005 0000000000000005 c000000001904000
GPR28: c0000000070e0000 c000000000000000 c0000000019049a0 c00000002e72f7f0
[ 31.325765] NIP [c0000000001a23a4] lockdep_register_key+0xb4/0x340
[ 31.325809] LR [c00000000075eccc] alloc_netdev_mqs+0x15c/0x500
[ 31.325848] Call Trace:
[ 31.325886] [c00000002e72f750] [0000000000000005] 0x5 (unreliable)
[ 31.325930] [c00000002e72f7f0] [c00000000075eccc] alloc_netdev_mqs+0x15c/0x500
[ 31.325984] [c00000002e72f8d0] [c0000000007d37f0] alloc_etherdev_mqs+0x60/0x90
[ 31.326047] [c00000002e72f910] [c00800000f150110] tg3_init_one+0x108/0x1d00 [tg3]
[ 31.326098] [c00000002e72fac0] [c000000000633b48] local_pci_probe+0x78/0x100
[ 31.326143] [c00000002e72fb50] [c000000000134b60] work_for_cpu_fn+0x40/0x70
[ 31.326190] [c00000002e72fb80] [c00000000013927c] process_one_work+0x3ac/0x710
[ 31.326221] [c00000002e72fc70] [c000000000138d90] process_scheduled_works+0x60/0xa0
[ 31.326274] [c00000002e72fcb0] [c000000000139ba4] worker_thread+0x344/0x4a0
[ 31.326317] [c00000002e72fda0] [c000000000142f68] kthread+0x1b8/0x1e0
[ 31.326363] [c00000002e72fe20] [c00000000000b748] ret_from_kernel_thread+0x5c/0x74
[ 31.326412] Instruction dump:
[ 31.326448] 28230000 418200a0 7fc3f378 48191fd9 60000000 70630001 41810018 7fc3f378
[ 31.326510] 4807fe25 60000000 70630001 40810060 <0fe00000> 3c62fffc 8883fa2f 70840001
[ 31.326573] irq event stamp: 806
[ 31.326617] hardirqs last enabled at (805): [<c0000000008daa4c>] _raw_write_unlock_irqrestore+0x5c/0xc0
[ 31.326666] hardirqs last disabled at (806): [<c000000000008fbc>] program_check_common+0x21c/0x230
[ 31.326710] softirqs last enabled at (0): [<c0000000000fa098>] copy_process+0x688/0x1850
[ 31.326752] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 31.326791] ---[ end trace c9674d7f7d278f30 ]---Hi Michael,
Quoted 45 lines
Quoted 11 lines
Quoted 11 lines
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in
lockdep_register_key()at,if (WARN_ON_ONCE(static_obj(key)))
because
key =
0xc0000000019ad118&_stext=0xc000000000000000&_end=0xc0000000049d0000Quoted 8 lines
Quoted 1 line
i.e., it will cause
static_obj()returns 1.(back from a trip)
Hi Qian,
Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between
&_stextand&_end?I thought that was true on all arches due to initmem, but seems not.
I guess we have the same problem as s390 and we need to define
arch_is_kernel_initmem_freed().Qian, can you try this:
arch/powerpc/include/asm/sections.h +7
-0@@ -5,8 +5,22 @@#include <linux/elf.h> #include <linux/uaccess.h> + +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed +#include <asm-generic/sections.h> +extern bool init_mem_is_free; + +static inline int arch_is_kernel_initmem_freed (unsigned long addr) +{
arch/powerpc/include/asm/sections.h line 26
Quoted 10 lines
arch/powerpc/include/asm/sections.h +7
-0+if (!init_mem_is_free) +return 0 ; + +return addr >= (unsigned long )__init_begin && + addr < (unsigned long )__init_end; +} +extern char __head_end[];#ifdef __powerpc64__
This also fixes the following syzkaller bug:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
https://github.com/linuxppc/issues/issues/284
Would you like me to do up a nice commit message for it?
Regards,
Daniel
Quoted 2 lines
cheers
Quoted 58 lines · Daniel Axtens
Hi Michael,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH)
would generate
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
a warning in
lockdep_register_key()at,if (WARN_ON_ONCE(static_obj(key)))
because
key =
0xc0000000019ad118&_stext=0xc000000000000000&_end=0xc0000000049d0000Quoted 7 lines
Quoted 7 lines
Quoted 1 line
i.e., it will cause
static_obj()returns 1.(back from a trip)
Hi Qian,
Does this mean that on POWER9 it can happen that a dynamically
allocated
Review
Quoted 10 lines
Quoted 1 line
object has an address that falls between
&_stextand&_end?I thought that was true on all arches due to initmem, but seems not.
I guess we have the same problem as s390 and we need to define
arch_is_kernel_initmem_freed().Qian, can you try this:
(quoted diff) +0
-0
b/arch/powerpc/include/asm/sections.hQuoted 25 lines
index
4a1664a8658d..616b1b7b7e52100644arch/powerpc/include/asm/sections.h +14
-0@@ -5,8 +5,22 @@#include <linux/elf.h> #include <linux/uaccess.h> + +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed +#include <asm-generic/sections.h> +extern bool init_mem_is_free; + +static inline int arch_is_kernel_initmem_freed (unsigned long addr) +{ +if (!init_mem_is_free) +return 0 ; + +return addr >= (unsigned long )__init_begin && + addr < (unsigned long )__init_end; +} +extern char __head_end[];
Quoted 8 lines
arch/powerpc/include/asm/sections.h line 26
Quoted 2 lines
arch/powerpc/include/asm/sections.h +0
-0#ifdef __powerpc64__This also fixes the following syzkaller bug:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
https://github.com/linuxppc/issues/issues/284Would you like me to do up a nice commit message for it?
That'd be great, thanks.
cheers
powerpc: define arch_is_kernel_initmem_freed() for lockdep
Under certain circumstances, we hit a warning in lockdep_register_key:
if (WARN_ON_ONCE(static_obj(key)))
return;This occurs when the key falls into initmem that has since been freed and can now be reused. This has been observed on boot, and under memory pressure.
Define arch_is_kernel_initmem_freed(), which allows lockdep to correctly identify this memory as dynamic.
This fixes a bug picked up by the powerpc64 syzkaller instance where we hit the WARN via alloc_netdev_mqs.
<mpe signoff here>
---
The ppc64 syzkaller link is probably not stable enough to go into the git history forever, but fwiw:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
--- arch/powerpc/include/asm/sections.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
arch/powerpc/include/asm/sections.h +14 -0
@@ -5,8 +5,22 @@#include <linux/elf.h> #include <linux/uaccess.h> + +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed +#include <asm-generic/sections.h> +extern bool init_mem_is_free; + +static inline int arch_is_kernel_initmem_freed (unsigned long addr) +{ +if (!init_mem_is_free) +return 0 ; + +return addr >= (unsigned long )__init_begin && + addr < (unsigned long )__init_end; +} +extern char __head_end[];#ifdef __powerpc64__