Skip to contentSkip to conversation

lockdep warning while booting POWER9 PowerNV

DISCUSSION Qian Cai 5 participants 8 messages Aug 30, 2019 – Nov 25, 2019 PowerPC

lore
Share & tools
lore

Qian Cai

author
#0 lore raw

https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

i.e., it will cause static_obj() returns 1.

[   23.738200][   T13] ahci 0004:03:00.0: enabling device (0541 -> 0543) [   23.748692][    T5] tg3.c:v3.137 (May 11, 2014)
[   23.748731][    T5] tg3 0005:01:00.0: enabling device (0140 -> 0142) [   23.751478][   T13] ahci 0004:03:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[   23.751517][   T13] ahci 0004:03:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
[   23.791077][   T13] scsi host0: ahci
[   23.802752][   T13] ------------[ cut here ]------------
[   23.802786][   T13] WARNING: CPU: 0 PID: 13 at kernel/locking/lockdep.c:1120 lockdep_register_key+0x68/0x200
[   23.802814][   T13] Modules linked in: mdio tg3(+) ahci(+) libahci libphy firmware_class libata dm_mirror dm_region_hash dm_log dm_mod
[   23.802884][   T13] CPU: 0 PID: 13 Comm: kworker/0:1 Not tainted 5.3.0-rc6- next-20190830 #4
[   23.802930][   T13] Workqueue: events work_for_cpu_fn
[   23.802962][   T13] NIP:  c00000000019eed8 LR: c000000000129400 CTR:
c0000000008a46dc
[   23.802988][   T13] REGS: c00000002db2f130 TRAP: 0700   Not tainted  (5.3.0- rc6-next-20190830)
[   23.803032][   T13] MSR:  900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 48800a89  XER: 20040000
[   23.803096][   T13] CFAR: c00000000019eeac IRQMASK: 0
[   23.803096][   T13] GPR00: c000000000129400 c00000002db2f3c0 c000000002bb9400 c0000000049d0000
[   23.803096][   T13] GPR04: 0000000000000000 c000000002d84718 0000000000000000 ffff0a01ffffff10
[   23.803096][   T13] GPR08: c000000000b8a16b c000000000a515cc c000000000b8a16a 0000000000000044
[   23.803096][   T13] GPR12: 0000000088800a89 c0000000049d0000 c00800000eea3290 0000000000000001
[   23.803096][   T13] GPR16: c000001bca584500 c000000007244954 c000001bca584508 0000000000000020
[   23.803096][   T13] GPR20: c000000001aea180 0000000000000001 c00800000eee7560 c000001bca584500
[   23.803096][   T13] GPR24: c000001bca584448 000000000002000a c000000001aea148 c000000000b8a161
[   23.803096][   T13] GPR28: c000000001aea020 c000000001aea118 c000000001aea000 c00000002db2f440
[   23.803407][   T13] NIP [c00000000019eed8] lockdep_register_key+0x68/0x200 [   23.803432][   T13] LR [c000000000129400] wq_init_lockdep+0x40/0xc0 [   23.803461][   T13] Call Trace:
[   23.803491][   T13] [c00000002db2f3c0] [c000000000b8a16a] trunc_msg+0x385f9/0x4c30f (unreliable)
[   23.803531][   T13] [c00000002db2f440] [c000000000129400] wq_init_lockdep+0x40/0xc0
[   23.803577][   T13] [c00000002db2f4c0] [c000000000128d90] alloc_workqueue+0x1e0/0x620
[   23.803623][   T13] [c00000002db2f5c0] [c0000000006ca048] scsi_host_alloc+0x3d8/0x490
[   23.803701][   T13] [c00000002db2f660] [c00800000ee73198] ata_scsi_add_hosts+0xd0/0x220 [libata]
[   23.803771][   T13] [c00000002db2f6f0] [c00800000ee6d440] ata_host_register+0x178/0x400 [libata]
[   23.803840][   T13] [c00000002db2f7d0] [c00800000ee6d8f4] ata_host_activate+0x17c/0x210 [libata]
[   23.803883][   T13] [c00000002db2f880] [c00800000edf4c3c] ahci_host_activate+0x84/0x250 [libahci]
[   23.803944][   T13] [c00000002db2f940] [c00800000f050c7c] ahci_init_one+0xc74/0xdc0 [ahci]
[   23.803992][   T13] [c00000002db2faa0] [c00000000062b188] local_pci_probe+0x78/0x100
[   23.804048][   T13] [c00000002db2fb30] [c00000000012c0b0] work_for_cpu_fn+0x40/0x70
[   23.804087][   T13] [c00000002db2fb60] [c000000000130328] process_one_work+0x388/0x750
[   23.804135][   T13] [c00000002db2fc60] [c00000000012fe30] process_scheduled_works+0x50/0x90
[   23.804180][   T13] [c00000002db2fca0] [c000000000130c50]

worker_thread+0x3d0/0x570
[   23.804216][   T13] [c00000002db2fda0] [c000000000139a08] kthread+0x1b8/0x1e0

[   23.804253][   T13] [c00000002db2fe20] [c00000000000b660] ret_from_kernel_thread+0x5c/0x7c
[   23.804298][   T13] Instruction dump:
[   23.804326][   T13] fbc10070 4194002c 7fa3eb78 481864a5 60000000 70630001 41810018 7fa3eb78
[   23.804373][   T13] 48075121 60000000 70630001 40810020 <0fe00000> ebc10070 eba10068 38210080
[   23.804419][   T13] ---[ end trace 9ceca43eedad2a2b ]---
[   23.810924][    T5] tg3 0005:01:00.0 eth0: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address 08:94:ef:80:81:c0
[   23.810967][    T5] tg3 0005:01:00.0 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   23.811027][    T5] tg3 0005:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[   23.811074][    T5] tg3 0005:01:00.0 eth0: dma_rwctrl[00000000] dma_mask[64- bit]
[   23.811284][   T13] scsi host1: ahci

#1 lore raw
Quoted 7 lines · Qian Cai

https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

Quoted 7 lines

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

i.e., it will cause static_obj() returns 1.

(back from a trip)

Hi Qian,

Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end? Since I am not familiar with POWER9 nor have access to such a system, can you propose a patch?

Thanks,

Bart.

#2 lore raw
Quoted 14 lines · Bart Van Assche
Quoted 13 lines · Qian Cai

https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

Quoted 8 lines
Quoted 1 line

i.e., it will cause static_obj() returns 1.

(back from a trip)

Hi Qian,

Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end?

I thought that was true on all arches due to initmem, but seems not.

I guess we have the same problem as s390 and we need to define arch_is_kernel_initmem_freed().

Qian, can you try this:

arch/powerpc/include/asm/sections.h +14 -0
index 4a1664a8658d..616b1b7b7e52 100644
@@ -5,8 +5,22 @@
 
 #include <linux/elf.h>
 #include <linux/uaccess.h>
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
 #include <asm-generic/sections.h>
 
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{
+	if (!init_mem_is_free)
+		return 0;
+
+	return addr >= (unsigned long)__init_begin &&
+		addr < (unsigned long)__init_end;
+}
+
 extern char __head_end[];
 
 #ifdef __powerpc64__

cheers

author
#3 lore raw
Quoted 22 lines · Michael Ellerman
Quoted 21 lines · Bart Van Assche
Quoted 8 lines · Qian Cai

https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate
a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

Quoted 7 lines

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

i.e., it will cause static_obj() returns 1.

(back from a trip)

Hi Qian,

Quoted 7 lines
Quoted 2 lines

Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end?

I thought that was true on all arches due to initmem, but seems not.

I guess we have the same problem as s390 and we need to define arch_is_kernel_initmem_freed().

Actually, it is in the .bss section. The commit 2d4f567103ff ("KVM: PPC:
Introduce kvm_tmp framework") adds kvm_tmp[] into the .bss section and then free the rest of unused spaces back to the page allocator.

kernel_init
  kvm_guest_init
    kvm_free_tmp
      free_reserved_area
        free_unref_page
          free_unref_page_prepare

Later, alloc_workqueue() happens to allocate some pages from there, and triggers the warning. Not sure what the best way to solve this.

author
#4 lore raw
In reply to Michael Ellerman
Quoted 53 lines · Michael Ellerman
Quoted 14 lines · Bart Van Assche
Quoted 13 lines · Qian Cai

https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

Quoted 8 lines
Quoted 1 line

i.e., it will cause static_obj() returns 1.

(back from a trip)

Hi Qian,

Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end?

I thought that was true on all arches due to initmem, but seems not.

I guess we have the same problem as s390 and we need to define arch_is_kernel_initmem_freed().

Qian, can you try this:

arch/powerpc/include/asm/sections.h +7 -0
index 4a1664a8658d..616b1b7b7e52 100644
@@ -5,8 +5,22 @@
 
 #include <linux/elf.h>
 #include <linux/uaccess.h>
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
 #include <asm-generic/sections.h>
 
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{

arch/powerpc/include/asm/sections.h line 26

Quoted 10 lines
arch/powerpc/include/asm/sections.h +7 -0
+	if (!init_mem_is_free)
+		return 0;
+
+	return addr >= (unsigned long)__init_begin &&
+		addr < (unsigned long)__init_end;
+}
+
 extern char __head_end[];
 
 #ifdef __powerpc64__

Michael, this fix is also needed as it starts to trigger another one of those where the allocated memory is from initmem.

[   31.326825] key = c0000000019049a0
[   31.326862] stext = c000000000000000, end = c0000000070e0000
[   31.326907] init_start = c000000000c70000, init_end = c0000000020f0000
[   31.325021] WARNING: CPU: 0 PID: 5 at kernel/locking/lockdep.c:1121 lockdep_register_key+0xb4/0x340
[   31.325061] Modules linked in: tg3(+) ahci(+) libahci libata mdio libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod
[   31.325128] CPU: 0 PID: 5 Comm: kworker/0:0 Not tainted 5.4.0-rc8-next-20191120+ #4
[   31.325190] Workqueue: events work_for_cpu_fn
[   31.325215] NIP:  c0000000001a23a4 LR: c00000000075eccc CTR: 0000000000000000
[   31.325257] REGS: c00000002e72f4c0 TRAP: 0700   Not tainted  (5.4.0-rc8-next-20191120+)
[   31.325320] MSR:  900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 48000c20  XER: 20040000
[   31.325392] CFAR: c0000000001a233c IRQMASK: 0 
               GPR00: c00000000075eccc c00000002e72f750 c000000002cff500 c0000000070df500 
               GPR04: c0000014beb01990 c0000000019042b8 0000000000000000 0000000000000000 
               GPR08: 0000000000000000 0000000000000000 c000000000425e28 c00c000004761020 
               GPR12: 0000000000000000 c0000000070e0000 c00000002e5214f8 c000001ffca018c8 
               GPR16: c000001ffca018e4 c000001ffca01c80 c000001ffca018d0 c00000002e6e3e48 
               GPR20: c000000002cbf500 c00000002e520080 c000001ffca05408 c00000002e6e3e00 
               GPR24: c0000000007d36d0 0000000000000005 0000000000000005 c000000001904000 
               GPR28: c0000000070e0000 c000000000000000 c0000000019049a0 c00000002e72f7f0 
[   31.325765] NIP [c0000000001a23a4] lockdep_register_key+0xb4/0x340
[   31.325809] LR [c00000000075eccc] alloc_netdev_mqs+0x15c/0x500
[   31.325848] Call Trace:
[   31.325886] [c00000002e72f750] [0000000000000005] 0x5 (unreliable)
[   31.325930] [c00000002e72f7f0] [c00000000075eccc] alloc_netdev_mqs+0x15c/0x500
[   31.325984] [c00000002e72f8d0] [c0000000007d37f0] alloc_etherdev_mqs+0x60/0x90
[   31.326047] [c00000002e72f910] [c00800000f150110] tg3_init_one+0x108/0x1d00 [tg3]
[   31.326098] [c00000002e72fac0] [c000000000633b48] local_pci_probe+0x78/0x100
[   31.326143] [c00000002e72fb50] [c000000000134b60] work_for_cpu_fn+0x40/0x70
[   31.326190] [c00000002e72fb80] [c00000000013927c] process_one_work+0x3ac/0x710
[   31.326221] [c00000002e72fc70] [c000000000138d90] process_scheduled_works+0x60/0xa0
[   31.326274] [c00000002e72fcb0] [c000000000139ba4] worker_thread+0x344/0x4a0
[   31.326317] [c00000002e72fda0] [c000000000142f68] kthread+0x1b8/0x1e0
[   31.326363] [c00000002e72fe20] [c00000000000b748] ret_from_kernel_thread+0x5c/0x74
[   31.326412] Instruction dump:
[   31.326448] 28230000 418200a0 7fc3f378 48191fd9 60000000 70630001 41810018 7fc3f378 
[   31.326510] 4807fe25 60000000 70630001 40810060 <0fe00000> 3c62fffc 8883fa2f 70840001 
[   31.326573] irq event stamp: 806
[   31.326617] hardirqs last  enabled at (805): [<c0000000008daa4c>] _raw_write_unlock_irqrestore+0x5c/0xc0
[   31.326666] hardirqs last disabled at (806): [<c000000000008fbc>] program_check_common+0x21c/0x230
[   31.326710] softirqs last  enabled at (0): [<c0000000000fa098>] copy_process+0x688/0x1850
[   31.326752] softirqs last disabled at (0): [<0000000000000000>] 0x0
[   31.326791] ---[ end trace c9674d7f7d278f30 ]---
#5 lore raw
In reply to Michael Ellerman

Hi Michael,

Quoted 45 lines
Quoted 11 lines
Quoted 11 lines

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

Quoted 8 lines
Quoted 1 line

i.e., it will cause static_obj() returns 1.

(back from a trip)

Hi Qian,

Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end?

I thought that was true on all arches due to initmem, but seems not.

I guess we have the same problem as s390 and we need to define arch_is_kernel_initmem_freed().

Qian, can you try this:

arch/powerpc/include/asm/sections.h +7 -0
index 4a1664a8658d..616b1b7b7e52 100644
@@ -5,8 +5,22 @@
 
 #include <linux/elf.h>
 #include <linux/uaccess.h>
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
 #include <asm-generic/sections.h>
 
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{

arch/powerpc/include/asm/sections.h line 26

Quoted 10 lines
arch/powerpc/include/asm/sections.h +7 -0
+	if (!init_mem_is_free)
+		return 0;
+
+	return addr >= (unsigned long)__init_begin &&
+		addr < (unsigned long)__init_end;
+}
+
 extern char __head_end[];
 
 #ifdef __powerpc64__

This also fixes the following syzkaller bug:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
https://github.com/linuxppc/issues/issues/284

Would you like me to do up a nice commit message for it?

Regards,
Daniel

Quoted 2 lines

cheers

#6 lore raw
Quoted 58 lines · Daniel Axtens

Hi Michael,

Quoted 1 line
Quoted 1 line
Quoted 1 line

Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH)

would generate

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

a warning in lockdep_register_key() at,

if (WARN_ON_ONCE(static_obj(key)))

because

key = 0xc0000000019ad118
&_stext = 0xc000000000000000
&_end = 0xc0000000049d0000

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

i.e., it will cause static_obj() returns 1.

(back from a trip)

Hi Qian,

Does this mean that on POWER9 it can happen that a dynamically

allocated

Review

Quoted 10 lines
Quoted 1 line

object has an address that falls between &_stext and &_end?

I thought that was true on all arches due to initmem, but seems not.

I guess we have the same problem as s390 and we need to define arch_is_kernel_initmem_freed().

Qian, can you try this:

(quoted diff) +0 -0

b/arch/powerpc/include/asm/sections.h

Quoted 25 lines

index 4a1664a8658d..616b1b7b7e52 100644

arch/powerpc/include/asm/sections.h +14 -0
@@ -5,8 +5,22 @@
 
 #include <linux/elf.h>
 #include <linux/uaccess.h>
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
 #include <asm-generic/sections.h>
 
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{
+	if (!init_mem_is_free)
+		return 0;
+
+	return addr >= (unsigned long)__init_begin &&
+		addr < (unsigned long)__init_end;
+}
+
 extern char __head_end[];
 
Quoted 8 lines

arch/powerpc/include/asm/sections.h line 26

Quoted 2 lines
arch/powerpc/include/asm/sections.h +0 -0
 #ifdef __powerpc64__

This also fixes the following syzkaller bug:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
https://github.com/linuxppc/issues/issues/284

Would you like me to do up a nice commit message for it?

That'd be great, thanks.

cheers

#7 lore raw

powerpc: define arch_is_kernel_initmem_freed() for lockdep

Under certain circumstances, we hit a warning in lockdep_register_key:

        if (WARN_ON_ONCE(static_obj(key)))
                return;

This occurs when the key falls into initmem that has since been freed and can now be reused. This has been observed on boot, and under memory pressure.

Define arch_is_kernel_initmem_freed(), which allows lockdep to correctly identify this memory as dynamic.

This fixes a bug picked up by the powerpc64 syzkaller instance where we hit the WARN via alloc_netdev_mqs.

<mpe signoff here>

---

The ppc64 syzkaller link is probably not stable enough to go into the git history forever, but fwiw:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b

---
 arch/powerpc/include/asm/sections.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/powerpc/include/asm/sections.h +14 -0
index 5a9b6eb651b6..d19871763ed4 100644
@@ -5,8 +5,22 @@
 
 #include <linux/elf.h>
 #include <linux/uaccess.h>
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
 #include <asm-generic/sections.h>
 
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{
+	if (!init_mem_is_free)
+		return 0;
+
+	return addr >= (unsigned long)__init_begin &&
+		addr < (unsigned long)__init_end;
+}
+
 extern char __head_end[];
 
 #ifdef __powerpc64__