Skip to contentSkip to conversation

d_lookup: Unable to handle kernel paging request

DISCUSSION Vicente Bergas 4 participants 19 messages May 22, 2019 – Jun 29, 2019 VFS

lore
Share & tools
lore
#0 lore raw

Hi,
since a recent update the kernel is reporting d_lookup errors. They appear randomly and after each error the affected file or directory is no longer accessible.
The kernel is built with GCC 9.1.0 on ARM64.
Four traces from different workloads follow.

This trace is from v5.1-12511-g72cf0b07418a while untaring into a tmpfs filesystem:

Unable to handle kernel paging request at virtual address 0000880001000018

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp = 000000007ccc6c7d [0000880001000018] pgd=0000000000000000

Internal error: Oops: 96000004 [#1] SMP
Process tar (pid: 1673, stack limit = 0x0000000083be9793)
CPU: 5 PID: 1673 Comm: tar Not tainted 5.1.0 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup+0x58/0x198
lr : d_lookup+0x38/0x68
sp : ffff0000126e3ba0
x29: ffff0000126e3ba0 x28: ffff0000126e3d68 x27: 0000000000000000 x26: ffff80008201d300 x25: 0000000000000001 x24: ffffffffffffffff x23: 00000000ce986489 x22: 0000000000000000 x21: 0000000000000001 x20: ffff0000126e3d68 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: a4d0a4a8a4fea3d0 x9 : 2f062c662d62dfa7 x8 : f2025989e6593ef3 x7 : b24a95208032f7e2 x6 : 0000000000000001 x5 : 0000000000000000 x4 : ffff0000126e3d68 x3 : ffff000010828a68 x2 : ffff000010828000 x1 : ffff8000f3000000 x0 : 00000000000674c3 Call trace:
__d_lookup+0x58/0x198
d_lookup+0x38/0x68
path_openat+0x4a8/0xfb8
do_filp_open+0x60/0xd8
do_sys_open+0x144/0x1f8
__arm64_sys_openat+0x20/0x28
el0_svc_handler+0x68/0xd8
el0_svc+0x8/0xc
Code: 92800018 a9025bf5 d2800016 52800035 (b9401a62) ---[ end trace 8d5c8dc953aa6402 ]---

This trace is from v5.2.0-rc1:

Unable to handle kernel paging request at virtual address 0000880001000018

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp = 000000004850c69c [0000880001000018] pgd=0000000000000000

Internal error: Oops: 96000004 [#1] SMP
Process read_sensors (pid: 926, stack limit = 0x00000000aaf00007)
CPU: 0 PID: 926 Comm: read_sensors Not tainted 5.2.0-rc1 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup+0x58/0x198
lr : d_lookup+0x38/0x68
sp : ffff000011ee3c60
x29: ffff000011ee3c60 x28: ffff000011ee3d98 x27: 0000000000000000 x26: ffff8000f28083c0 x25: 0000000000000276 x24: ffffffffffffffff x23: 00000000ce97b3cf x22: 0000000000000000 x21: 0000000000000001 x20: ffff000011ee3d98 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000002 x16: 0000000000000001 x15: ffff8000e4b3a8c8 x14: ffffffffffffffff x13: ffff000011ee3db8 x12: ffff000011ee3dad x11: 0000000000000000 x10: ffff000011ee3d20 x9 : 00000000ffffffd8 x8 : 000000000000039e x7 : 0000000000000000 x6 : 0000000000000002 x5 : 61c8864680b583eb x4 : 42bed11fefc04553 x3 : ffff000010828a68 x2 : ffff000010828000 x1 : ffff8000f3000000 x0 : 00000000000674bd Call trace:
__d_lookup+0x58/0x198
d_lookup+0x38/0x68
d_hash_and_lookup+0x50/0x68
proc_flush_task+0x98/0x198
release_task+0x60/0x4b8
do_exit+0x680/0xa68
__arm64_sys_exit+0x14/0x18
el0_svc_handler+0x68/0xd8
el0_svc+0x8/0xc
Code: 92800018 a9025bf5 d2800016 52800035 (b9401a62) ---[ end trace c9b8ee5d6aa547ae ]---

This trace is from v5.2.0-rc1 while executing 'git pull -r' from f2fs. It got repeated several times:

Unable to handle kernel paging request at virtual address 0000000000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000092bdb9cd [0000000000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#2] SMP
Process git (pid: 2996, stack limit = 0x000000004b733f9b)
CPU: 5 PID: 2996 Comm: git Tainted: G      D           5.2.0-rc1 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8
sp : ffff000013f83aa0
x29: ffff000013f83aa0 x28: 00000000ce9798e9 x27: ffffffffffffffff x26: 0000000000000015 x25: ffff800009268043 x24: ffff000013f83b6c x23: 0000000000000005 x22: 00000015ce9798e9 x21: ffff800039a7a780 x20: 0000000000000000 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: d0d0d0b3b3fea4a3 x9 : f2862b1e24d6cb78 x8 : fac1836b95d6b53a x7 : c1462108f502da45 x6 : 0847a816d22e0a31 x5 : ffff800009268043 x4 : ffff8000f3000000 x3 : ffff000013f83c88 x2 : ffff000013f83b6c x1 : 00000000000674bc x0 : ffff800039a7a780 Call trace:
__d_lookup_rcu+0x68/0x198
lookup_fast+0x44/0x2e8
walk_component+0x34/0x2e0
path_lookupat.isra.0+0x5c/0x1e0
filename_lookup+0x78/0xf0
user_path_at_empty+0x44/0x58
vfs_statx+0x70/0xd0
__se_sys_newfstatat+0x20/0x40
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x68/0xd8
el0_svc+0x8/0xc
Code: 9280001b 14000003 f9400273 b4000793 (b85fc265) ---[ end trace c9b8ee5d6aa547af ]---

This trace is from v5.2.0-rc1 while executing 'rm -rf' the directory affected from the previous trace:

Unable to handle kernel paging request at virtual address 0000000001000018

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp = 00000000649981ae [0000000001000018] pgd=0000000000000000

Internal error: Oops: 96000004 [#10] SMP
Process rm (pid: 6401, stack limit = 0x00000000e524cae1)
CPU: 5 PID: 6401 Comm: rm Tainted: G      D           5.2.0-rc1 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup+0x58/0x198
lr : d_lookup+0x38/0x68
sp : ffff000016993d00
x29: ffff000016993d00 x28: ffff000016993e70 x27: 0000000000000000 x26: ffff800039a7a780 x25: 0000000056000000 x24: ffffffffffffffff x23: 00000000ce9798e9 x22: 0000000000000000 x21: 0000000000000001 x20: ffff000016993e70 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: d0d0d0b3b3fea4a3 x9 : f2862b1e24d6cb78 x8 : fac1836b95d6b53a x7 : c1462108f502da45 x6 : 00000000ffffffff x5 : 0000000000000000 x4 : ffff8000eb31d500 x3 : ffff000010828a68 x2 : ffff000010828000 x1 : ffff8000f3000000 x0 : 00000000000674bc Call trace:
__d_lookup+0x58/0x198
d_lookup+0x38/0x68
lookup_dcache+0x20/0x80
__lookup_hash+0x20/0xc8
do_unlinkat+0x10c/0x278
__arm64_sys_unlinkat+0x34/0x60
el0_svc_handler+0x68/0xd8
el0_svc+0x8/0xc
Code: 92800018 a9025bf5 d2800016 52800035 (b9401a62) ---[ end trace c9b8ee5d6aa547b7 ]---

Regards,
Vicenç.

maintainer
#1 lore raw
Quoted 6 lines · Vicente Bergas

Hi,
since a recent update the kernel is reporting d_lookup errors. They appear randomly and after each error the affected file or directory is no longer accessible.
The kernel is built with GCC 9.1.0 on ARM64.
Four traces from different workloads follow.

Interesting... bisection would be useful.

Quoted 6 lines

This trace is from v5.1-12511-g72cf0b07418a while untaring into a tmpfs filesystem:

Unable to handle kernel paging request at virtual address 0000880001000018

user pgtable: 4k pages, 48-bit VAs, pgdp = 000000007ccc6c7d [0000880001000018] pgd=0000000000000000

Attempt to dereference 0x0000880001000018, which is not mapped at all?

Quoted 1 line
pc : __d_lookup+0x58/0x198

... and so would objdump of the function in question.

Quoted 2 lines

This trace is from v5.2.0-rc1:

Unable to handle kernel paging request at virtual address 0000880001000018

[apparently identical oops, modulo the call chain to d_lookup(); since that's almost certainly buggered data structures encountered during the hash lookup, exact callchain doesn't matter all that much; procfs is the filesystem involved]

Quoted 8 lines

This trace is from v5.2.0-rc1 while executing 'git pull -r' from f2fs. It got repeated several times:

Unable to handle kernel paging request at virtual address 0000000000fffffc
user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000092bdb9cd
[0000000000fffffc] pgd=0000000000000000
pc : __d_lookup_rcu+0x68/0x198
Quoted 4 lines

This trace is from v5.2.0-rc1 while executing 'rm -rf' the directory affected from the previous trace:

Unable to handle kernel paging request at virtual address 0000000001000018

... and addresses involved are

0000880001000018
0000000000fffffc
0000000001000018

AFAICS, the only registers with the value in the vicinity of those addresses had been (in all cases so far) x19 - 0000880001000000 in the first two traces, 0000000001000000 in the last two...

I'd really like to see the disassembly of the functions involved (as well as .config in question).

#2 lore raw

Hi Al,

Quoted 9 lines · Al Viro
Quoted 6 lines · Vicente Bergas

Hi,
since a recent update the kernel is reporting d_lookup errors. They appear randomly and after each error the affected file or directory is no longer accessible.
The kernel is built with GCC 9.1.0 on ARM64.
Four traces from different workloads follow.

Interesting... bisection would be useful.

Agreed, but that would be difficult because of the randomness. I have been running days with no issues with a known-bad kernel. The issue could also be related to the upgrade to GCC 9.

Quoted 7 lines
Quoted 6 lines

This trace is from v5.1-12511-g72cf0b07418a while untaring into a tmpfs filesystem:

Unable to handle kernel paging request at virtual address 0000880001000018

user pgtable: 4k pages, 48-bit VAs, pgdp = 000000007ccc6c7d [0000880001000018] pgd=0000000000000000

Quoted 5 lines

Attempt to dereference 0x0000880001000018, which is not mapped at all?

Quoted 1 line
pc : __d_lookup+0x58/0x198

... and so would objdump of the function in question.

Here is the dump from another build of the exact
same version (the build is reproducible).

objdump -x -S fs/dcache.o

...
0000000000002d00 <__d_lookup_rcu>:
2d00: a9b97bfd stp x29, x30, [sp, #-112]!
2d04: aa0103e3 mov x3, x1
2d08: 90000004 adrp x4, 0 <find_submount>

			2d08: R_AARCH64_ADR_PREL_PG_HI21	.data..read_mostly
   2d0c:	910003fd 	mov	x29, sp
   2d10:	a90153f3 	stp	x19, x20, [sp, #16]
   2d14:	91000081 	add	x1, x4, #0x0
			2d14: R_AARCH64_ADD_ABS_LO12_NC	.data..read_mostly
   2d18:	a9025bf5 	stp	x21, x22, [sp, #32]
   2d1c:	a9046bf9 	stp	x25, x26, [sp, #64]
   2d20:	a9406476 	ldp	x22, x25, [x3]
   2d24:	b9400821 	ldr	w1, [x1, #8]
   2d28:	f9400084 	ldr	x4, [x4]
			2d28: R_AARCH64_LDST64_ABS_LO12_NC	.data..read_mostly

2d2c: 1ac126c1 lsr w1, w22, w1
2d30: f8617893 ldr x19, [x4, x1, lsl #3]
2d34: f27ffa73 ands x19, x19, #0xfffffffffffffffe
2d38: 54000920 b.eq 2e5c <__d_lookup_rcu+0x15c> // b.none
2d3c: aa0003f5 mov x21, x0
2d40: d360feda lsr x26, x22, #32
2d44: a90363f7 stp x23, x24, [sp, #48]
2d48: aa0203f8 mov x24, x2
2d4c: d3608ad7 ubfx x23, x22, #32, #3
2d50: a90573fb stp x27, x28, [sp, #80]
2d54: 2a1603fc mov w28, w22
2d58: 9280001b mov x27, #0xffffffffffffffff // #-1
2d5c: 14000003 b 2d68 <__d_lookup_rcu+0x68>
2d60: f9400273 ldr x19, [x19]
2d64: b4000793 cbz x19, 2e54 <__d_lookup_rcu+0x154>
2d68: b85fc265 ldur w5, [x19, #-4]
2d6c: d50339bf dmb ishld
2d70: f9400a64 ldr x4, [x19, #16]
2d74: d1002260 sub x0, x19, #0x8
2d78: eb0402bf cmp x21, x4
2d7c: 54ffff21 b.ne 2d60 <__d_lookup_rcu+0x60> // b.any
2d80: 121f78b4 and w20, w5, #0xfffffffe
2d84: aa0003e9 mov x9, x0
2d88: f9400664 ldr x4, [x19, #8]
2d8c: b4fffea4 cbz x4, 2d60 <__d_lookup_rcu+0x60>
2d90: b94002a4 ldr w4, [x21]
2d94: 37080404 tbnz w4, #1, 2e14 <__d_lookup_rcu+0x114>
2d98: f9401000 ldr x0, [x0, #32]
2d9c: eb16001f cmp x0, x22
2da0: 54fffe01 b.ne 2d60 <__d_lookup_rcu+0x60> // b.any
2da4: f9401265 ldr x5, [x19, #32]
2da8: 2a1a03e6 mov w6, w26
2dac: cb050328 sub x8, x25, x5
2db0: 14000006 b 2dc8 <__d_lookup_rcu+0xc8>
2db4: 910020a5 add x5, x5, #0x8
2db8: eb07001f cmp x0, x7
2dbc: 54fffd21 b.ne 2d60 <__d_lookup_rcu+0x60> // b.any
2dc0: 710020c6 subs w6, w6, #0x8
2dc4: 54000160 b.eq 2df0 <__d_lookup_rcu+0xf0> // b.none
2dc8: 8b0800a4 add x4, x5, x8
2dcc: 6b1700df cmp w6, w23
2dd0: f9400087 ldr x7, [x4]
2dd4: f94000a0 ldr x0, [x5]
2dd8: 54fffee1 b.ne 2db4 <__d_lookup_rcu+0xb4> // b.any
2ddc: 531d72e1 lsl w1, w23, #3
2de0: ca070000 eor x0, x0, x7
2de4: 9ac12361 lsl x1, x27, x1
2de8: ea21001f bics xzr, x0, x1
2dec: 54fffba1 b.ne 2d60 <__d_lookup_rcu+0x60> // b.any
2df0: b9000314 str w20, [x24]
2df4: aa0903e0 mov x0, x9
2df8: a94153f3 ldp x19, x20, [sp, #16]
2dfc: a9425bf5 ldp x21, x22, [sp, #32]
2e00: a94363f7 ldp x23, x24, [sp, #48]
2e04: a9446bf9 ldp x25, x26, [sp, #64]
2e08: a94573fb ldp x27, x28, [sp, #80]
2e0c: a8c77bfd ldp x29, x30, [sp], #112
2e10: d65f03c0 ret
2e14: b9402001 ldr w1, [x0, #32]
2e18: 6b01039f cmp w28, w1
2e1c: 54fffa21 b.ne 2d60 <__d_lookup_rcu+0x60> // b.any
2e20: b9402401 ldr w1, [x0, #36]
2e24: f9401402 ldr x2, [x0, #40]
2e28: d50339bf dmb ishld
2e2c: b85fc264 ldur w4, [x19, #-4]
2e30: 6b04029f cmp w20, w4
2e34: 54000221 b.ne 2e78 <__d_lookup_rcu+0x178> // b.any
2e38: f94032a4 ldr x4, [x21, #96]
2e3c: a90627e3 stp x3, x9, [sp, #96]
2e40: f9400c84 ldr x4, [x4, #24]
2e44: d63f0080 blr x4
2e48: a94627e3 ldp x3, x9, [sp, #96]
2e4c: 34fffd20 cbz w0, 2df0 <__d_lookup_rcu+0xf0>
2e50: 17ffffc4 b 2d60 <__d_lookup_rcu+0x60>
2e54: a94363f7 ldp x23, x24, [sp, #48]
2e58: a94573fb ldp x27, x28, [sp, #80]
2e5c: d2800009 mov x9, #0x0 // #0
2e60: aa0903e0 mov x0, x9
2e64: a94153f3 ldp x19, x20, [sp, #16]
2e68: a9425bf5 ldp x21, x22, [sp, #32]
2e6c: a9446bf9 ldp x25, x26, [sp, #64]
2e70: a8c77bfd ldp x29, x30, [sp], #112
2e74: d65f03c0 ret
2e78: d503203f yield
2e7c: b85fc265 ldur w5, [x19, #-4]
2e80: d50339bf dmb ishld
2e84: f9400c01 ldr x1, [x0, #24]
2e88: 121f78b4 and w20, w5, #0xfffffffe
2e8c: eb15003f cmp x1, x21
2e90: 54fff681 b.ne 2d60 <__d_lookup_rcu+0x60> // b.any
2e94: 17ffffbd b 2d88 <__d_lookup_rcu+0x88>

0000000000002e98 <__d_lookup>:
2e98: a9b97bfd stp x29, x30, [sp, #-112]!
2e9c: 90000002 adrp x2, 0 <find_submount>

			2e9c: R_AARCH64_ADR_PREL_PG_HI21	.data..read_mostly
   2ea0:	91000043 	add	x3, x2, #0x0
			2ea0: R_AARCH64_ADD_ABS_LO12_NC	.data..read_mostly
   2ea4:	910003fd 	mov	x29, sp
   2ea8:	a90573fb 	stp	x27, x28, [sp, #80]
   2eac:	aa0103fc 	mov	x28, x1
   2eb0:	a90153f3 	stp	x19, x20, [sp, #16]
   2eb4:	a90363f7 	stp	x23, x24, [sp, #48]
   2eb8:	a9046bf9 	stp	x25, x26, [sp, #64]
   2ebc:	aa0003fa 	mov	x26, x0
   2ec0:	b9400397 	ldr	w23, [x28]
   2ec4:	b9400860 	ldr	w0, [x3, #8]
   2ec8:	f9400041 	ldr	x1, [x2]
			2ec8: R_AARCH64_LDST64_ABS_LO12_NC	.data..read_mostly
   2ecc:	1ac026e0 	lsr	w0, w23, w0
   2ed0:	f8607833 	ldr	x19, [x1, x0, lsl #3]
   2ed4:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2ed8:	54000320 	b.eq	2f3c <__d_lookup+0xa4>  // b.none
   2edc:	5280001b 	mov	w27, #0x0                   	// #0
   2ee0:	92800018 	mov	x24, #0xffffffffffffffff    	// #-1
   2ee4:	a9025bf5 	stp	x21, x22, [sp, #32]
   2ee8:	d2800016 	mov	x22, #0x0                   	// #0
   2eec:	52800035 	mov	w21, #0x1                   	// #1
   2ef0:	b9401a62 	ldr	w2, [x19, #24]
   2ef4:	d1002274 	sub	x20, x19, #0x8
   2ef8:	6b17005f 	cmp	w2, w23
   2efc:	540001a1 	b.ne	2f30 <__d_lookup+0x98>  // b.any
   2f00:	91014279 	add	x25, x19, #0x50
   2f04:	f9800331 	prfm	pstl1strm, [x25]
   2f08:	885fff21 	ldaxr	w1, [x25]
   2f0c:	4a160020 	eor	w0, w1, w22
   2f10:	35000060 	cbnz	w0, 2f1c <__d_lookup+0x84>
   2f14:	88007f35 	stxr	w0, w21, [x25]
   2f18:	35ffff80 	cbnz	w0, 2f08 <__d_lookup+0x70>
   2f1c:	35000521 	cbnz	w1, 2fc0 <__d_lookup+0x128>
   2f20:	f9400e82 	ldr	x2, [x20, #24]
   2f24:	eb1a005f 	cmp	x2, x26
   2f28:	540001a0 	b.eq	2f5c <__d_lookup+0xc4>  // b.none
   2f2c:	089fff3b 	stlrb	w27, [x25]
   2f30:	f9400273 	ldr	x19, [x19]
   2f34:	b5fffdf3 	cbnz	x19, 2ef0 <__d_lookup+0x58>
   2f38:	a9425bf5 	ldp	x21, x22, [sp, #32]
   2f3c:	d2800008 	mov	x8, #0x0                   	// #0
   2f40:	aa0803e0 	mov	x0, x8
   2f44:	a94153f3 	ldp	x19, x20, [sp, #16]
   2f48:	a94363f7 	ldp	x23, x24, [sp, #48]
   2f4c:	a9446bf9 	ldp	x25, x26, [sp, #64]
   2f50:	a94573fb 	ldp	x27, x28, [sp, #80]
   2f54:	a8c77bfd 	ldp	x29, x30, [sp], #112
   2f58:	d65f03c0 	ret
   2f5c:	f9400660 	ldr	x0, [x19, #8]
   2f60:	b4fffe60 	cbz	x0, 2f2c <__d_lookup+0x94>
   2f64:	b9400340 	ldr	w0, [x26]
   2f68:	aa1403e8 	mov	x8, x20
   2f6c:	b9402681 	ldr	w1, [x20, #36]
   2f70:	370802e0 	tbnz	w0, #1, 2fcc <__d_lookup+0x134>
   2f74:	b9400784 	ldr	w4, [x28, #4]
   2f78:	6b04003f 	cmp	w1, w4
   2f7c:	54fffd81 	b.ne	2f2c <__d_lookup+0x94>  // b.any
   2f80:	f9400787 	ldr	x7, [x28, #8]
   2f84:	12000881 	and	w1, w4, #0x7
   2f88:	f9401265 	ldr	x5, [x19, #32]
   2f8c:	cb0500e7 	sub	x7, x7, x5
   2f90:	14000003 	b	2f9c <__d_lookup+0x104>
   2f94:	71002084 	subs	w4, w4, #0x8
   2f98:	54000300 	b.eq	2ff8 <__d_lookup+0x160>  // b.none
   2f9c:	8b0700a2 	add	x2, x5, x7
   2fa0:	6b04003f 	cmp	w1, w4
   2fa4:	f9400046 	ldr	x6, [x2]
   2fa8:	f94000a0 	ldr	x0, [x5]
   2fac:	54000340 	b.eq	3014 <__d_lookup+0x17c>  // b.none
   2fb0:	910020a5 	add	x5, x5, #0x8
   2fb4:	eb06001f 	cmp	x0, x6
   2fb8:	54fffee0 	b.eq	2f94 <__d_lookup+0xfc>  // b.none
   2fbc:	17ffffdc 	b	2f2c <__d_lookup+0x94>
   2fc0:	aa1903e0 	mov	x0, x25
   2fc4:	94000000 	bl	0 <queued_spin_lock_slowpath>
			2fc4: R_AARCH64_CALL26	queued_spin_lock_slowpath

2fc8: 17ffffd6 b 2f20 <__d_lookup+0x88>
2fcc: f9403340 ldr x0, [x26, #96]
2fd0: aa1c03e3 mov x3, x28
2fd4: f9401682 ldr x2, [x20, #40]
2fd8: f90037f4 str x20, [sp, #104]
2fdc: f9400c04 ldr x4, [x0, #24]
2fe0: aa1403e0 mov x0, x20
2fe4: d63f0080 blr x4
2fe8: 7100001f cmp w0, #0x0
2fec: 1a9f17e0 cset w0, eq // eq = none
2ff0: f94037e8 ldr x8, [sp, #104]
2ff4: 34fff9c0 cbz w0, 2f2c <__d_lookup+0x94>
2ff8: b9405e80 ldr w0, [x20, #92]
2ffc: 52800001 mov w1, #0x0 // #0
3000: 11000400 add w0, w0, #0x1
3004: b9005e80 str w0, [x20, #92]
3008: 089fff21 stlrb w1, [x25]
300c: a9425bf5 ldp x21, x22, [sp, #32]
3010: 17ffffcc b 2f40 <__d_lookup+0xa8>
3014: 531d7021 lsl w1, w1, #3
3018: ca060000 eor x0, x0, x6
301c: 9ac12301 lsl x1, x24, x1
3020: ea21001f bics xzr, x0, x1
3024: 1a9f17e0 cset w0, eq // eq = none
3028: 34fff820 cbz w0, 2f2c <__d_lookup+0x94>
302c: 17fffff3 b 2ff8 <__d_lookup+0x160>

0000000000003030 <d_lookup>:
3030: a9bd7bfd stp x29, x30, [sp, #-48]!
3034: 910003fd mov x29, sp
3038: a90153f3 stp x19, x20, [sp, #16]
303c: 90000013 adrp x19, 0 <find_submount>

			303c: R_AARCH64_ADR_PREL_PG_HI21	.data..cacheline_aligned
   3040:	aa0103f4 	mov	x20, x1
   3044:	91000273 	add	x19, x19, #0x0
			3044: R_AARCH64_ADD_ABS_LO12_NC	.data..cacheline_aligned
   3048:	a9025bf5 	stp	x21, x22, [sp, #32]
   304c:	aa0003f5 	mov	x21, x0
   3050:	b9400276 	ldr	w22, [x19]
   3054:	370001d6 	tbnz	w22, #0, 308c <d_lookup+0x5c>
   3058:	d50339bf 	dmb	ishld
   305c:	aa1403e1 	mov	x1, x20
   3060:	aa1503e0 	mov	x0, x21
   3064:	94000000 	bl	2e98 <__d_lookup>
			3064: R_AARCH64_CALL26	__d_lookup

3068: b50000a0 cbnz x0, 307c <d_lookup+0x4c>
306c: d50339bf dmb ishld
3070: b9400261 ldr w1, [x19]
3074: 6b16003f cmp w1, w22
3078: 54fffec1 b.ne 3050 <d_lookup+0x20> // b.any
307c: a94153f3 ldp x19, x20, [sp, #16]
3080: a9425bf5 ldp x21, x22, [sp, #32]
3084: a8c37bfd ldp x29, x30, [sp], #48
3088: d65f03c0 ret
308c: d503203f yield
3090: 17fffff0 b 3050 <d_lookup+0x20>
3094: d503201f nop
...

Quoted 25 lines
Quoted 2 lines

This trace is from v5.2.0-rc1:

Unable to handle kernel paging request at virtual address 0000880001000018

[apparently identical oops, modulo the call chain to d_lookup(); since that's almost certainly buggered data structures encountered during the hash lookup, exact callchain doesn't matter all that much; procfs is the filesystem involved]

Quoted 8 lines

This trace is from v5.2.0-rc1 while executing 'git pull -r' from f2fs. It got repeated several times:

Unable to handle kernel paging request at virtual address 0000000000fffffc
user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000092bdb9cd
[0000000000fffffc] pgd=0000000000000000
pc : __d_lookup_rcu+0x68/0x198
Quoted 4 lines

This trace is from v5.2.0-rc1 while executing 'rm -rf' the directory affected from the previous trace:

Unable to handle kernel paging request at virtual address 0000000001000018

... and addresses involved are

0000880001000018
0000000000fffffc
0000000001000018

Quoted 6 lines

AFAICS, the only registers with the value in the vicinity of those addresses had been (in all cases so far) x19 - 0000880001000000 in the first two traces, 0000000001000000 in the last two...

I'd really like to see the disassembly of the functions involved (as well as .config in question).

Here is the .config: https://paste.debian.net/1082689

Regards,
Vicenç.

maintainer
#3 lore raw
Quoted 6 lines · Vicente Bergas
   2d30:	f8617893 	ldr	x19, [x4, x1, lsl #3]
   2d34:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2d38:	54000920 	b.eq	2e5c <__d_lookup_rcu+0x15c>  // b.none
   2d3c:	aa0003f5 	mov	x21, x0
   2d40:	d360feda 	lsr	x26, x22, #32
   2d44:	a90363f7 	stp	x23, x24, [sp, #48]
Quoted 8 lines
   2d48:	aa0203f8 	mov	x24, x2
   2d4c:	d3608ad7 	ubfx	x23, x22, #32, #3
   2d50:	a90573fb 	stp	x27, x28, [sp, #80]
   2d54:	2a1603fc 	mov	w28, w22
   2d58:	9280001b 	mov	x27, #0xffffffffffffffff    	// #-1
   2d5c:	14000003 	b	2d68 <__d_lookup_rcu+0x68>
   2d60:	f9400273 	ldr	x19, [x19]
   2d64:	b4000793 	cbz	x19, 2e54 <__d_lookup_rcu+0x154>

OK, that looks like
hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash)
in there, x19 being 'node'.

Quoted 2 lines
   2d68:	b85fc265 	ldur	w5, [x19, #-4]
   2d6c:	d50339bf 	dmb	ishld

... and that's seq = raw_seqcount_begin(&dentry->d_seq), with ->d_seq being 4 bytes before ->d_hash. So that one has stepped into 0x1000000 (i.e. 1<<24) in hlist forward pointer (or head - either is possible).

Quoted 10 lines

0000000000002e98 <__d_lookup>:

   2ed0:	f8607833 	ldr	x19, [x1, x0, lsl #3]
   2ed4:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2ed8:	54000320 	b.eq	2f3c <__d_lookup+0xa4>  // b.none
   2edc:	5280001b 	mov	w27, #0x0                   	// #0
   2ee0:	92800018 	mov	x24, #0xffffffffffffffff    	// #-1
   2ee4:	a9025bf5 	stp	x21, x22, [sp, #32]
   2ee8:	d2800016 	mov	x22, #0x0                   	// #0
   2eec:	52800035 	mov	w21, #0x1                   	// #1

That's

        hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
Quoted 1 line
   2ef0:	b9401a62 	ldr	w2, [x19, #24]

... and fetching dentry->d_name.hash for subsequent

                if (dentry->d_name.hash != hash)
                        continue;
Quoted 3 lines
   2ef4:	d1002274 	sub	x20, x19, #0x8
   2ef8:	6b17005f 	cmp	w2, w23
   2efc:	540001a1 	b.ne	2f30 <__d_lookup+0x98>  // b.any

IOW, here we have also run into bogus hlist forward pointer or head - same 0x1000000 in one case and 0x0000880001000000 in two others.

Have you tried to see if KASAN catches anything on those loads? Use-after-free, for example... Another thing to try: slap

	WARN_ON(entry->d_flags & DCACHE_NORCU);

in __d_rehash() and see if it triggers.

#4 lore raw
Quoted 9 lines · Al Viro

On Wed, May 22, 2019 at 05:44:30PM +0200, Vicente Bergas wrote:
...
IOW, here we have also run into bogus hlist forward pointer or head - same 0x1000000 in one case and 0x0000880001000000 in two others.

Have you tried to see if KASAN catches anything on those loads? Use-after-free, for example... Another thing to try: slap

	WARN_ON(entry->d_flags & DCACHE_NORCU);

in __d_rehash() and see if it triggers.

Hi,
i have been running 3 days with KASAN enabled and also with diff a/fs/dcache.c b/fs/dcache.c
@@ -2395,3 +2395,4 @@ static void __d_rehash(struct dentry *entry)

	struct hlist_bl_head *b = d_hash(entry->d_name.hash);
+	WARN_ON(entry->d_flags & DCACHE_NORCU);
	hlist_bl_lock(b);

but the issue has not appeared again.
Next week i will try -rc2 without KASAN and with WARN_ON and see if it triggers.

Regards,
Vicenç.

#5 lore raw
In reply to Al Viro
Quoted 8 lines · Al Viro

...
IOW, here we have also run into bogus hlist forward pointer or head - same 0x1000000 in one case and 0x0000880001000000 in two others.

Have you tried to see if KASAN catches anything on those loads? Use-after-free, for example... Another thing to try: slap

	WARN_ON(entry->d_flags & DCACHE_NORCU);

in __d_rehash() and see if it triggers.

Hi Al,
after 5 days with v5.2-rc1 + KASAN + WARN_ON could not reproduce the issue. Neither the first day running v5.3-rc2 + WARN_ON. But today 6 times. So, there is no KASAN and also the WARN_ON, being there, did not trigger. The first trace hapenned while untaring a big file into tmpfs. The other five while "git pull -r" severeal repos on f2fs.

Regards,
Vicenç.

Unable to handle kernel paging request at virtual address 0000000001000018

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=00000000aeab4000 [0000000001000018] pgd=0000000000000000

Internal error: Oops: 96000004 [#1] SMP
CPU: 4 PID: 1172 Comm: tar Not tainted 5.2.0-rc2 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup+0x58/0x198
lr : d_lookup+0x38/0x68
sp : ffff000012663b90
x29: ffff000012663b90 x28: ffff000012663d58 x27: 0000000000000000 x26: ffff8000ae7cc900 x25: 0000000000000001 x24: ffffffffffffffff x23: 00000000ce9c8f81 x22: 0000000000000000 x21: 0000000000000001 x20: ffff000012663d58 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: b4fea3d0a3a4b4fe x9 : d237122a91454b69 x8 : a0591ae4450bed6a x7 : 5845a2c80f79d4e7 x6 : 0000000000000004 x5 : 0000000000000000 x4 : ffff000012663d58 x3 : ffff000010828a68 x2 : ffff000010828000 x1 : ffff8000f3000000 x0 : 00000000000674e4 Call trace:
__d_lookup+0x58/0x198
d_lookup+0x38/0x68
path_openat+0x4a8/0xfb8
do_filp_open+0x60/0xd8
do_sys_open+0x144/0x1f8
__arm64_sys_openat+0x20/0x28
el0_svc_handler+0x74/0x140
el0_svc+0x8/0xc
Code: 92800018 a9025bf5 d2800016 52800035 (b9401a62) ---[ end trace 7fc40d1e6d2ed53e ]---
Unable to handle kernel paging request at virtual address 0000000000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000007af3e000 [0000000000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#1] SMP
CPU: 4 PID: 2124 Comm: git Not tainted 5.2.0-rc2 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8
sp : ffff0000130b3b60
x29: ffff0000130b3b60 x28: 00000000ce99d070 x27: ffffffffffffffff x26: 0000000000000026 x25: ffff8000ecec6030 x24: ffff0000130b3c2c x23: 0000000000000006 x22: 00000026ce99d070 x21: ffff8000811f3d80 x20: 0000000000020000 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: e4d0b2e6e2b4b6e9 x9 : 5096e90463dfacb0 x8 : 2b4f8961c30ebc93 x7 : aec349fb204a7256 x6 : 4fd9025392b5761a x5 : 02ff010101030100 x4 : ffff8000f3000000 x3 : ffff0000130b3d58 x2 : ffff0000130b3c2c x1 : 00000000000674ce x0 : ffff8000811f3d80 Call trace:
__d_lookup_rcu+0x68/0x198
lookup_fast+0x44/0x2e8
path_openat+0x19c/0xfb8
do_filp_open+0x60/0xd8
do_sys_open+0x144/0x1f8
__arm64_sys_openat+0x20/0x28
el0_svc_handler+0x74/0x140
el0_svc+0x8/0xc
Code: 9280001b 14000003 f9400273 b4000793 (b85fc265) ---[ end trace 6bd1b3b7588a78fe ]---
Unable to handle kernel paging request at virtual address 0000880000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=00000000867ac000 [0000880000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#2] SMP
CPU: 4 PID: 2183 Comm: git Tainted: G      D           5.2.0-rc2 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8
sp : ffff00001325ba90
x29: ffff00001325ba90 x28: 00000000ce99f075 x27: ffffffffffffffff x26: 0000000000000007 x25: ffff8000ecec402a x24: ffff00001325bb5c x23: 0000000000000007 x22: 00000007ce99f075 x21: ffff80007a810c00 x20: 0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: d0bbbcbfa6b2b9bc x9 : 0000000000000000 x8 : ffff80007a810c00 x7 : 6cad9ff29d8de19c x6 : ff94ec6f0ce3656c x5 : ffff8000ecec402a x4 : ffff8000f3000000 x3 : ffff00001325bc78 x2 : ffff00001325bb5c x1 : 00000000000674cf x0 : ffff80007a810c00 Call trace:
__d_lookup_rcu+0x68/0x198
lookup_fast+0x44/0x2e8
walk_component+0x34/0x2e0
path_lookupat.isra.0+0x5c/0x1e0
filename_lookup+0x78/0xf0
user_path_at_empty+0x44/0x58
vfs_statx+0x70/0xd0
__se_sys_newfstatat+0x20/0x40
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x74/0x140
el0_svc+0x8/0xc
Code: 9280001b 14000003 f9400273 b4000793 (b85fc265) ---[ end trace 6bd1b3b7588a78ff ]---
Unable to handle kernel paging request at virtual address 0000880000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=00000000867ac000 [0000880000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#3] SMP
CPU: 4 PID: 2180 Comm: git Tainted: G      D           5.2.0-rc2 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8
sp : ffff000012a3ba90
x29: ffff000012a3ba90 x28: 00000000ce99f075 x27: ffffffffffffffff x26: 0000000000000007 x25: ffff8000ecec702a x24: ffff000012a3bb5c x23: 0000000000000007 x22: 00000007ce99f075 x21: ffff80007a810c00 x20: 0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: d0bbbcbfa6b2b9bc x9 : 0000000000000000 x8 : ffff80007a810c00 x7 : 6cad9ff29d8de19c x6 : ff94ec6f0ce3656c x5 : ffff8000ecec702a x4 : ffff8000f3000000 x3 : ffff000012a3bc78 x2 : ffff000012a3bb5c x1 : 00000000000674cf x0 : ffff80007a810c00 Call trace:
__d_lookup_rcu+0x68/0x198
lookup_fast+0x44/0x2e8
walk_component+0x34/0x2e0
path_lookupat.isra.0+0x5c/0x1e0
filename_lookup+0x78/0xf0
user_path_at_empty+0x44/0x58
vfs_statx+0x70/0xd0
__se_sys_newfstatat+0x20/0x40
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x74/0x140
el0_svc+0x8/0xc
Code: 9280001b 14000003 f9400273 b4000793 (b85fc265) ---[ end trace 6bd1b3b7588a7900 ]---
Unable to handle kernel paging request at virtual address 0000880000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=0000000073f2f000 [0000880000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#4] SMP
CPU: 4 PID: 2210 Comm: git Tainted: G      D           5.2.0-rc2 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8
sp : ffff0000132bba90
x29: ffff0000132bba90 x28: 00000000ce99e1a6 x27: ffffffffffffffff x26: 000000000000000c x25: ffff8000f21dd036 x24: ffff0000132bbb5c x23: 0000000000000004 x22: 0000000cce99e1a6 x21: ffff800074dd8d80 x20: 0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: d0d0d0d0b8fea4b3 x9 : 40bcd8645005512e x8 : c433ade89ebd10f9 x7 : c6b69091eeb194d2 x6 : 848f758ca69635b4 x5 : ffff8000f21dd036 x4 : ffff8000f3000000 x3 : ffff0000132bbc78 x2 : ffff0000132bbb5c x1 : 00000000000674cf x0 : ffff800074dd8d80 Call trace:
__d_lookup_rcu+0x68/0x198
lookup_fast+0x44/0x2e8
walk_component+0x34/0x2e0
path_lookupat.isra.0+0x5c/0x1e0
filename_lookup+0x78/0xf0
user_path_at_empty+0x44/0x58
vfs_statx+0x70/0xd0
__se_sys_newfstatat+0x20/0x40
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x74/0x140
el0_svc+0x8/0xc
Code: 9280001b 14000003 f9400273 b4000793 (b85fc265) ---[ end trace 6bd1b3b7588a7901 ]---
Unable to handle kernel paging request at virtual address 0000880000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=0000000073f2f000 [0000880000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#5] SMP
CPU: 5 PID: 2200 Comm: git Tainted: G      D           5.2.0-rc2 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 00000005 (nzcv daif -PAN -UAO)
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8
sp : ffff000013263a90
x29: ffff000013263a90 x28: 00000000ce99e1a6 x27: ffffffffffffffff x26: 000000000000000c x25: ffff8000f0a6f036 x24: ffff000013263b5c x23: 0000000000000004 x22: 0000000cce99e1a6 x21: ffff800074dd8d80 x20: 0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: fefefefefefefeff x10: d0d0d0d0b8fea4b3 x9 : 40bcd8645005512e x8 : c433ade89ebd10f9 x7 : c6b69091eeb194d2 x6 : 848f758ca69635b4 x5 : ffff8000f0a6f036 x4 : ffff8000f3000000 x3 : ffff000013263c78 x2 : ffff000013263b5c x1 : 00000000000674cf x0 : ffff800074dd8d80 Call trace:
__d_lookup_rcu+0x68/0x198
lookup_fast+0x44/0x2e8
walk_component+0x34/0x2e0
path_lookupat.isra.0+0x5c/0x1e0
filename_lookup+0x78/0xf0
user_path_at_empty+0x44/0x58
vfs_statx+0x70/0xd0
__se_sys_newfstatat+0x20/0x40
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x74/0x140
el0_svc+0x8/0xc
Code: 9280001b 14000003 f9400273 b4000793 (b85fc265) ---[ end trace 6bd1b3b7588a7902 ]---

00000000000001c8 <__d_rehash>:

	return dentry_hashtable + (hash >> d_hash_shift);
    1c8:	90000001 	adrp	x1, 0 <find_submount>
			1c8: R_AARCH64_ADR_PREL_PG_HI21	.data..read_mostly
    1cc:	91000022 	add	x2, x1, #0x0
			1cc: R_AARCH64_ADD_ABS_LO12_NC	.data..read_mostly

static void __d_rehash(struct dentry *entry)

{
	struct hlist_bl_head *b = d_hash(entry->d_name.hash);

	WARN_ON(entry->d_flags & DCACHE_NORCU);
    1d0:	b9400003 	ldr	w3, [x0]
	return dentry_hashtable + (hash >> d_hash_shift);
    1d4:	f9400025 	ldr	x5, [x1]
			1d4: R_AARCH64_LDST64_ABS_LO12_NC	.data..read_mostly
    1d8:	b9400841 	ldr	w1, [x2, #8]
    1dc:	b9402002 	ldr	w2, [x0, #32]
    1e0:	1ac12442 	lsr	w2, w2, w1
    1e4:	8b020ca1 	add	x1, x5, x2, lsl #3
	WARN_ON(entry->d_flags & DCACHE_NORCU);
    1e8:	37f00343 	tbnz	w3, #30, 250 <__d_rehash+0x88>
	__READ_ONCE_SIZE;
    1ec:	f9400023 	ldr	x3, [x1]
	if (READ_ONCE(*p) & mask)
    1f0:	37000283 	tbnz	w3, #0, 240 <__d_rehash+0x78>
    1f4:	f9800031 	prfm	pstl1strm, [x1]
    1f8:	c85ffc23 	ldaxr	x3, [x1]
    1fc:	b2400064 	orr	x4, x3, #0x1
    200:	c8067c24 	stxr	w6, x4, [x1]
    204:	35ffffa6 	cbnz	w6, 1f8 <__d_rehash+0x30>
	while (unlikely(test_and_set_bit_lock(bitnum, addr))) {
    208:	370001c3 	tbnz	w3, #0, 240 <__d_rehash+0x78>
		((unsigned long)h->first & ~LIST_BL_LOCKMASK);
    20c:	f86278a3 	ldr	x3, [x5, x2, lsl #3]
	hlist_bl_lock(b);
	hlist_bl_add_head_rcu(&entry->d_hash, b);
    210:	91002004 	add	x4, x0, #0x8
    214:	927ff863 	and	x3, x3, #0xfffffffffffffffe
	struct hlist_bl_node *first;

	/* don't need hlist_bl_first_rcu because we're under lock */
	first = hlist_bl_first(h);

	n->next = first;
    218:	f9000403 	str	x3, [x0, #8]
	if (first)
    21c:	b4000043 	cbz	x3, 224 <__d_rehash+0x5c>
		first->pprev = &n->next;
    220:	f9000464 	str	x4, [x3, #8]
	rcu_assign_pointer(h->first,
    224:	b2400084 	orr	x4, x4, #0x1
	n->pprev = &h->first;
    228:	f9000801 	str	x1, [x0, #16]
	rcu_assign_pointer(h->first,
    22c:	c89ffc24 	stlr	x4, [x1]
    230:	f86278a0 	ldr	x0, [x5, x2, lsl #3]
	old &= ~BIT_MASK(nr);
    234:	927ff800 	and	x0, x0, #0xfffffffffffffffe
    238:	c89ffc20 	stlr	x0, [x1]
	hlist_bl_unlock(b);
}
    23c:	d65f03c0 	ret
    240:	d503203f 	yield
    244:	f9400023 	ldr	x3, [x1]
		} while (test_bit(bitnum, addr));
    248:	3707ffc3 	tbnz	w3, #0, 240 <__d_rehash+0x78>
    24c:	17ffffe8 	b	1ec <__d_rehash+0x24>
	WARN_ON(entry->d_flags & DCACHE_NORCU);
    250:	d4210000 	brk	#0x800
	preempt_disable();
    254:	17ffffe6 	b	1ec <__d_rehash+0x24>

...

0000000000002d10 <__d_lookup_rcu>:
{
2d10: a9b97bfd stp x29, x30, [sp, #-112]!
2d14: aa0103e3 mov x3, x1

	return dentry_hashtable + (hash >> d_hash_shift);
   2d18:	90000004 	adrp	x4, 0 <find_submount>
			2d18: R_AARCH64_ADR_PREL_PG_HI21	.data..read_mostly
{
   2d1c:	910003fd 	mov	x29, sp
   2d20:	a90153f3 	stp	x19, x20, [sp, #16]
	return dentry_hashtable + (hash >> d_hash_shift);
   2d24:	91000081 	add	x1, x4, #0x0
			2d24: R_AARCH64_ADD_ABS_LO12_NC	.data..read_mostly
{
   2d28:	a9025bf5 	stp	x21, x22, [sp, #32]
   2d2c:	a9046bf9 	stp	x25, x26, [sp, #64]
	const unsigned char *str = name->name;
   2d30:	a9406476 	ldp	x22, x25, [x3]
	return dentry_hashtable + (hash >> d_hash_shift);
   2d34:	b9400821 	ldr	w1, [x1, #8]
   2d38:	f9400084 	ldr	x4, [x4]
			2d38: R_AARCH64_LDST64_ABS_LO12_NC	.data..read_mostly
   2d3c:	1ac126c1 	lsr	w1, w22, w1
	__READ_ONCE_SIZE;
   2d40:	f8617893 	ldr	x19, [x4, x1, lsl #3]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2d44:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2d48:	54000920 	b.eq	2e6c <__d_lookup_rcu+0x15c>  // b.none
   2d4c:	aa0003f5 	mov	x21, x0
			if (dentry_cmp(dentry, str, hashlen_len(hashlen)) != 0)
   2d50:	d360feda 	lsr	x26, x22, #32
   2d54:	a90363f7 	stp	x23, x24, [sp, #48]
   2d58:	aa0203f8 	mov	x24, x2
   2d5c:	d3608ad7 	ubfx	x23, x22, #32, #3
   2d60:	a90573fb 	stp	x27, x28, [sp, #80]
   2d64:	2a1603fc 	mov	w28, w22
	mask = bytemask_from_count(tcount);
   2d68:	9280001b 	mov	x27, #0xffffffffffffffff    	// #-1
   2d6c:	14000003 	b	2d78 <__d_lookup_rcu+0x68>
   2d70:	f9400273 	ldr	x19, [x19]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2d74:	b4000793 	cbz	x19, 2e64 <__d_lookup_rcu+0x154>
   2d78:	b85fc265 	ldur	w5, [x19, #-4]
	smp_rmb();
   2d7c:	d50339bf 	dmb	ishld
		if (dentry->d_parent != parent)
   2d80:	f9400a64 	ldr	x4, [x19, #16]
   2d84:	d1002260 	sub	x0, x19, #0x8
   2d88:	eb0402bf 	cmp	x21, x4
   2d8c:	54ffff21 	b.ne	2d70 <__d_lookup_rcu+0x60>  // b.any
	return ret & ~1;
   2d90:	121f78b4 	and	w20, w5, #0xfffffffe
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2d94:	aa0003e9 	mov	x9, x0
		if (d_unhashed(dentry))
   2d98:	f9400664 	ldr	x4, [x19, #8]
   2d9c:	b4fffea4 	cbz	x4, 2d70 <__d_lookup_rcu+0x60>
		if (unlikely(parent->d_flags & DCACHE_OP_COMPARE)) {
   2da0:	b94002a4 	ldr	w4, [x21]
   2da4:	37080404 	tbnz	w4, #1, 2e24 <__d_lookup_rcu+0x114>
			if (dentry->d_name.hash_len != hashlen)
   2da8:	f9401000 	ldr	x0, [x0, #32]
   2dac:	eb16001f 	cmp	x0, x22
   2db0:	54fffe01 	b.ne	2d70 <__d_lookup_rcu+0x60>  // b.any
   2db4:	f9401265 	ldr	x5, [x19, #32]
	const unsigned char *cs = READ_ONCE(dentry->d_name.name);
   2db8:	2a1a03e6 	mov	w6, w26
   2dbc:	cb050328 	sub	x8, x25, x5
   2dc0:	14000006 	b	2dd8 <__d_lookup_rcu+0xc8>
		cs += sizeof(unsigned long);
   2dc4:	910020a5 	add	x5, x5, #0x8
		if (unlikely(a != b))
   2dc8:	eb07001f 	cmp	x0, x7
   2dcc:	54fffd21 	b.ne	2d70 <__d_lookup_rcu+0x60>  // b.any
		if (!tcount)
   2dd0:	710020c6 	subs	w6, w6, #0x8
   2dd4:	54000160 	b.eq	2e00 <__d_lookup_rcu+0xf0>  // b.none
		cs += sizeof(unsigned long);

2dd8: 8b0800a4 add x4, x5, x8
if (tcount < sizeof(unsigned long))
2ddc: 6b1700df cmp w6, w23
static inline unsigned long load_unaligned_zeropad(const void *addr)

{
	unsigned long ret, offset;

	/* Load word from unaligned pointer addr */
	asm(

2de0: f9400087 ldr x7, [x4]

static __no_kasan_or_inline
unsigned long read_word_at_a_time(const void *addr)

{
	kasan_check_read(addr, 1);
	return *(unsigned long *)addr;
   2de4:	f94000a0 	ldr	x0, [x5]
   2de8:	54fffee1 	b.ne	2dc4 <__d_lookup_rcu+0xb4>  // b.any
	mask = bytemask_from_count(tcount);
   2dec:	531d72e1 	lsl	w1, w23, #3
	return unlikely(!!((a ^ b) & mask));
   2df0:	ca070000 	eor	x0, x0, x7
	mask = bytemask_from_count(tcount);
   2df4:	9ac12361 	lsl	x1, x27, x1
			if (dentry_cmp(dentry, str, hashlen_len(hashlen)) != 0)
   2df8:	ea21001f 	bics	xzr, x0, x1
   2dfc:	54fffba1 	b.ne	2d70 <__d_lookup_rcu+0x60>  // b.any
		*seqp = seq;

2e00: b9000314 str w20, [x24]
}
2e04: aa0903e0 mov x0, x9
2e08: a94153f3 ldp x19, x20, [sp, #16]
2e0c: a9425bf5 ldp x21, x22, [sp, #32]

		return dentry;
   2e10:	a94363f7 	ldp	x23, x24, [sp, #48]
}
   2e14:	a9446bf9 	ldp	x25, x26, [sp, #64]
		return dentry;

2e18: a94573fb ldp x27, x28, [sp, #80]
}
2e1c: a8c77bfd ldp x29, x30, [sp], #112
2e20: d65f03c0 ret

			if (dentry->d_name.hash != hashlen_hash(hashlen))
   2e24:	b9402001 	ldr	w1, [x0, #32]
   2e28:	6b01039f 	cmp	w28, w1
   2e2c:	54fffa21 	b.ne	2d70 <__d_lookup_rcu+0x60>  // b.any
			tlen = dentry->d_name.len;
   2e30:	b9402401 	ldr	w1, [x0, #36]
			tname = dentry->d_name.name;
   2e34:	f9401402 	ldr	x2, [x0, #40]
	smp_rmb();
   2e38:	d50339bf 	dmb	ishld
	return unlikely(s->sequence != start);
   2e3c:	b85fc264 	ldur	w4, [x19, #-4]
			if (read_seqcount_retry(&dentry->d_seq, seq)) {
   2e40:	6b04029f 	cmp	w20, w4
   2e44:	54000221 	b.ne	2e88 <__d_lookup_rcu+0x178>  // b.any
			if (parent->d_op->d_compare(dentry,
   2e48:	f94032a4 	ldr	x4, [x21, #96]
   2e4c:	a90627e3 	stp	x3, x9, [sp, #96]
   2e50:	f9400c84 	ldr	x4, [x4, #24]
   2e54:	d63f0080 	blr	x4
   2e58:	a94627e3 	ldp	x3, x9, [sp, #96]
   2e5c:	34fffd20 	cbz	w0, 2e00 <__d_lookup_rcu+0xf0>
   2e60:	17ffffc4 	b	2d70 <__d_lookup_rcu+0x60>
   2e64:	a94363f7 	ldp	x23, x24, [sp, #48]
   2e68:	a94573fb 	ldp	x27, x28, [sp, #80]
	return NULL;

2e6c: d2800009 mov x9, #0x0 // #0
}
2e70: aa0903e0 mov x0, x9
2e74: a94153f3 ldp x19, x20, [sp, #16]
2e78: a9425bf5 ldp x21, x22, [sp, #32]
2e7c: a9446bf9 ldp x25, x26, [sp, #64]
2e80: a8c77bfd ldp x29, x30, [sp], #112
2e84: d65f03c0 ret
2e88: d503203f yield

	__READ_ONCE_SIZE;
   2e8c:	b85fc265 	ldur	w5, [x19, #-4]
	smp_rmb();
   2e90:	d50339bf 	dmb	ishld
		if (dentry->d_parent != parent)
   2e94:	f9400c01 	ldr	x1, [x0, #24]
	return ret & ~1;

2e98: 121f78b4 and w20, w5, #0xfffffffe
2e9c: eb15003f cmp x1, x21
2ea0: 54fff681 b.ne 2d70 <__d_lookup_rcu+0x60> // b.any
2ea4: 17ffffbd b 2d98 <__d_lookup_rcu+0x88>

0000000000002ea8 <__d_lookup>:
{
2ea8: a9b97bfd stp x29, x30, [sp, #-112]!

	return dentry_hashtable + (hash >> d_hash_shift);
   2eac:	90000002 	adrp	x2, 0 <find_submount>
			2eac: R_AARCH64_ADR_PREL_PG_HI21	.data..read_mostly
   2eb0:	91000043 	add	x3, x2, #0x0
			2eb0: R_AARCH64_ADD_ABS_LO12_NC	.data..read_mostly

{
2eb4: 910003fd mov x29, sp
2eb8: a90573fb stp x27, x28, [sp, #80]
2ebc: aa0103fc mov x28, x1
2ec0: a90153f3 stp x19, x20, [sp, #16]
2ec4: a90363f7 stp x23, x24, [sp, #48]
2ec8: a9046bf9 stp x25, x26, [sp, #64]
2ecc: aa0003fa mov x26, x0

	unsigned int hash = name->hash;
   2ed0:	b9400397 	ldr	w23, [x28]
	return dentry_hashtable + (hash >> d_hash_shift);
   2ed4:	b9400860 	ldr	w0, [x3, #8]
   2ed8:	f9400041 	ldr	x1, [x2]
			2ed8: R_AARCH64_LDST64_ABS_LO12_NC	.data..read_mostly
   2edc:	1ac026e0 	lsr	w0, w23, w0
   2ee0:	f8607833 	ldr	x19, [x1, x0, lsl #3]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2ee4:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2ee8:	54000320 	b.eq	2f4c <__d_lookup+0xa4>  // b.none
	smp_store_release(&lock->locked, 0);
   2eec:	5280001b 	mov	w27, #0x0                   	// #0
	mask = bytemask_from_count(tcount);
   2ef0:	92800018 	mov	x24, #0xffffffffffffffff    	// #-1
   2ef4:	a9025bf5 	stp	x21, x22, [sp, #32]
   2ef8:	d2800016 	mov	x22, #0x0                   	// #0
   2efc:	52800035 	mov	w21, #0x1                   	// #1
		if (dentry->d_name.hash != hash)
   2f00:	b9401a62 	ldr	w2, [x19, #24]
   2f04:	d1002274 	sub	x20, x19, #0x8
   2f08:	6b17005f 	cmp	w2, w23
   2f0c:	540001a1 	b.ne	2f40 <__d_lookup+0x98>  // b.any
   2f10:	91014279 	add	x25, x19, #0x50
   2f14:	f9800331 	prfm	pstl1strm, [x25]
   2f18:	885fff21 	ldaxr	w1, [x25]
   2f1c:	4a160020 	eor	w0, w1, w22
   2f20:	35000060 	cbnz	w0, 2f2c <__d_lookup+0x84>
   2f24:	88007f35 	stxr	w0, w21, [x25]
   2f28:	35ffff80 	cbnz	w0, 2f18 <__d_lookup+0x70>
   2f2c:	35000521 	cbnz	w1, 2fd0 <__d_lookup+0x128>
		if (dentry->d_parent != parent)
   2f30:	f9400e82 	ldr	x2, [x20, #24]
   2f34:	eb1a005f 	cmp	x2, x26
   2f38:	540001a0 	b.eq	2f6c <__d_lookup+0xc4>  // b.none
   2f3c:	089fff3b 	stlrb	w27, [x25]
   2f40:	f9400273 	ldr	x19, [x19]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2f44:	b5fffdf3 	cbnz	x19, 2f00 <__d_lookup+0x58>
   2f48:	a9425bf5 	ldp	x21, x22, [sp, #32]
	struct dentry *found = NULL;

2f4c: d2800008 mov x8, #0x0 // #0
}
2f50: aa0803e0 mov x0, x8
2f54: a94153f3 ldp x19, x20, [sp, #16]
2f58: a94363f7 ldp x23, x24, [sp, #48]
2f5c: a9446bf9 ldp x25, x26, [sp, #64]
2f60: a94573fb ldp x27, x28, [sp, #80]
2f64: a8c77bfd ldp x29, x30, [sp], #112
2f68: d65f03c0 ret
if (d_unhashed(dentry))
2f6c: f9400660 ldr x0, [x19, #8]
2f70: b4fffe60 cbz x0, 2f3c <__d_lookup+0x94>

	if (likely(!(parent->d_flags & DCACHE_OP_COMPARE))) {
   2f74:	b9400340 	ldr	w0, [x26]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2f78:	aa1403e8 	mov	x8, x20
	if (likely(!(parent->d_flags & DCACHE_OP_COMPARE))) {
   2f7c:	b9402681 	ldr	w1, [x20, #36]
   2f80:	370802e0 	tbnz	w0, #1, 2fdc <__d_lookup+0x134>
		if (dentry->d_name.len != name->len)
   2f84:	b9400784 	ldr	w4, [x28, #4]
   2f88:	6b04003f 	cmp	w1, w4
   2f8c:	54fffd81 	b.ne	2f3c <__d_lookup+0x94>  // b.any
		return dentry_cmp(dentry, name->name, name->len) == 0;

2f90: f9400787 ldr x7, [x28, #8]
static inline int dentry_string_cmp(const unsigned char *cs, const unsigned char *ct, unsigned tcount)
2f94: 12000881 and w1, w4, #0x7
2f98: f9401265 ldr x5, [x19, #32]
2f9c: cb0500e7 sub x7, x7, x5
2fa0: 14000003 b 2fac <__d_lookup+0x104>
if (!tcount)
2fa4: 71002084 subs w4, w4, #0x8
2fa8: 54000300 b.eq 3008 <__d_lookup+0x160> // b.none

		cs += sizeof(unsigned long);
   2fac:	8b0700a2 	add	x2, x5, x7
		if (tcount < sizeof(unsigned long))
   2fb0:	6b04003f 	cmp	w1, w4
   2fb4:	f9400046 	ldr	x6, [x2]
	return *(unsigned long *)addr;
   2fb8:	f94000a0 	ldr	x0, [x5]
   2fbc:	54000340 	b.eq	3024 <__d_lookup+0x17c>  // b.none
		cs += sizeof(unsigned long);
   2fc0:	910020a5 	add	x5, x5, #0x8
		if (unlikely(a != b))
   2fc4:	eb06001f 	cmp	x0, x6
   2fc8:	54fffee0 	b.eq	2fa4 <__d_lookup+0xfc>  // b.none
   2fcc:	17ffffdc 	b	2f3c <__d_lookup+0x94>
	queued_spin_lock_slowpath(lock, val);
   2fd0:	aa1903e0 	mov	x0, x25
   2fd4:	94000000 	bl	0 <queued_spin_lock_slowpath>
			2fd4: R_AARCH64_CALL26	queued_spin_lock_slowpath
   2fd8:	17ffffd6 	b	2f30 <__d_lookup+0x88>
	return parent->d_op->d_compare(dentry,
   2fdc:	f9403340 	ldr	x0, [x26, #96]
   2fe0:	aa1c03e3 	mov	x3, x28
   2fe4:	f9401682 	ldr	x2, [x20, #40]
   2fe8:	f90037f4 	str	x20, [sp, #104]
   2fec:	f9400c04 	ldr	x4, [x0, #24]
   2ff0:	aa1403e0 	mov	x0, x20
   2ff4:	d63f0080 	blr	x4
				       name) == 0;
   2ff8:	7100001f 	cmp	w0, #0x0
   2ffc:	1a9f17e0 	cset	w0, eq  // eq = none
   3000:	f94037e8 	ldr	x8, [sp, #104]
		if (!d_same_name(dentry, parent, name))
   3004:	34fff9c0 	cbz	w0, 2f3c <__d_lookup+0x94>
		dentry->d_lockref.count++;
   3008:	b9405e80 	ldr	w0, [x20, #92]
	smp_store_release(&lock->locked, 0);

300c: 52800001 mov w1, #0x0 // #0
3010: 11000400 add w0, w0, #0x1
3014: b9005e80 str w0, [x20, #92]
3018: 089fff21 stlrb w1, [x25]
}
301c: a9425bf5 ldp x21, x22, [sp, #32]
3020: 17ffffcc b 2f50 <__d_lookup+0xa8>

	mask = bytemask_from_count(tcount);
   3024:	531d7021 	lsl	w1, w1, #3
	return unlikely(!!((a ^ b) & mask));
   3028:	ca060000 	eor	x0, x0, x6
	mask = bytemask_from_count(tcount);

302c: 9ac12301 lsl x1, x24, x1
3030: ea21001f bics xzr, x0, x1
3034: 1a9f17e0 cset w0, eq // eq = none
if (!d_same_name(dentry, parent, name))
3038: 34fff820 cbz w0, 2f3c <__d_lookup+0x94>
303c: 17fffff3 b 3008 <__d_lookup+0x160>

0000000000003040 <d_lookup>:
{
3040: a9bd7bfd stp x29, x30, [sp, #-48]!
3044: 910003fd mov x29, sp
3048: a90153f3 stp x19, x20, [sp, #16]
304c: 90000013 adrp x19, 0 <find_submount>

			304c: R_AARCH64_ADR_PREL_PG_HI21	.data..cacheline_aligned
   3050:	aa0103f4 	mov	x20, x1
   3054:	91000273 	add	x19, x19, #0x0
			3054: R_AARCH64_ADD_ABS_LO12_NC	.data..cacheline_aligned
   3058:	a9025bf5 	stp	x21, x22, [sp, #32]
   305c:	aa0003f5 	mov	x21, x0
	__READ_ONCE_SIZE;
   3060:	b9400276 	ldr	w22, [x19]
	if (unlikely(ret & 1)) {
   3064:	370001d6 	tbnz	w22, #0, 309c <d_lookup+0x5c>
	smp_rmb();
   3068:	d50339bf 	dmb	ishld
		dentry = __d_lookup(parent, name);
   306c:	aa1403e1 	mov	x1, x20
   3070:	aa1503e0 	mov	x0, x21
   3074:	94000000 	bl	2ea8 <__d_lookup>
			3074: R_AARCH64_CALL26	__d_lookup
		if (dentry)
   3078:	b50000a0 	cbnz	x0, 308c <d_lookup+0x4c>
	smp_rmb();
   307c:	d50339bf 	dmb	ishld
	} while (read_seqretry(&rename_lock, seq));

3080: b9400261 ldr w1, [x19]
3084: 6b16003f cmp w1, w22
3088: 54fffec1 b.ne 3060 <d_lookup+0x20> // b.any
}
308c: a94153f3 ldp x19, x20, [sp, #16]
3090: a9425bf5 ldp x21, x22, [sp, #32]
3094: a8c37bfd ldp x29, x30, [sp], #48
3098: d65f03c0 ret
309c: d503203f yield
30a0: 17fffff0 b 3060 <d_lookup+0x20>
30a4: d503201f nop

maintainer
#6 lore raw
Quoted 17 lines · Vicente Bergas
Quoted 8 lines · Al Viro

...
IOW, here we have also run into bogus hlist forward pointer or head - same 0x1000000 in one case and 0x0000880001000000 in two others.

Have you tried to see if KASAN catches anything on those loads? Use-after-free, for example... Another thing to try: slap

	WARN_ON(entry->d_flags & DCACHE_NORCU);

in __d_rehash() and see if it triggers.

Hi Al,
after 5 days with v5.2-rc1 + KASAN + WARN_ON could not reproduce the issue. Neither the first day running v5.3-rc2 + WARN_ON. But today 6 times. So, there is no KASAN and also the WARN_ON, being there, did not trigger. The first trace hapenned while untaring a big file into tmpfs. The other five while "git pull -r" severeal repos on f2fs.

Quoted 3 lines

Regards,
Vicenç.

__d_lookup() running into &dentry->d_hash == 0x01000000 at some point in hash chain and trying to look at ->d_name.hash:

Quoted 13 lines
pc : __d_lookup+0x58/0x198
lr : d_lookup+0x38/0x68

sp : ffff000012663b90
x29: ffff000012663b90 x28: ffff000012663d58 x27: 0000000000000000 x26:
ffff8000ae7cc900 x25: 0000000000000001 x24: ffffffffffffffff x23:
00000000ce9c8f81 x22: 0000000000000000 x21: 0000000000000001 x20:
ffff000012663d58 x19: 0000000001000000 x18: 0000000000000000 x17:
0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14:
0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11:
fefefefefefefeff x10: b4fea3d0a3a4b4fe x9 : d237122a91454b69 x8 :
a0591ae4450bed6a x7 : 5845a2c80f79d4e7 x6 : 0000000000000004 x5 :
0000000000000000 x4 : ffff000012663d58 x3 : ffff000010828a68 x2 :
ffff000010828000 x1 : ffff8000f3000000 x0 : 00000000000674e4 Call trace:

__d_lookup_rcu() running into &dentry->d_hash == 0x01000000 at some point in hash chain and trying to look at ->d_seq:

Quoted 13 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff0000130b3b60
x29: ffff0000130b3b60 x28: 00000000ce99d070 x27: ffffffffffffffff x26:
0000000000000026 x25: ffff8000ecec6030 x24: ffff0000130b3c2c x23:
0000000000000006 x22: 00000026ce99d070 x21: ffff8000811f3d80 x20:
0000000000020000 x19: 0000000001000000 x18: 0000000000000000 x17:
0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14:
0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11:
fefefefefefefeff x10: e4d0b2e6e2b4b6e9 x9 : 5096e90463dfacb0 x8 :
2b4f8961c30ebc93 x7 : aec349fb204a7256 x6 : 4fd9025392b5761a x5 :
02ff010101030100 x4 : ffff8000f3000000 x3 : ffff0000130b3d58 x2 :
ffff0000130b3c2c x1 : 00000000000674ce x0 : ffff8000811f3d80 Call trace:

__d_lookup_rcu() running into &dentry->d_hash == 0x0000880001000000 at some point in hash chain and trying to look at ->d_seq:

Quoted 13 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff00001325ba90
x29: ffff00001325ba90 x28: 00000000ce99f075 x27: ffffffffffffffff x26:
0000000000000007 x25: ffff8000ecec402a x24: ffff00001325bb5c x23:
0000000000000007 x22: 00000007ce99f075 x21: ffff80007a810c00 x20:
0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17:
0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14:
0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11:
fefefefefefefeff x10: d0bbbcbfa6b2b9bc x9 : 0000000000000000 x8 :
ffff80007a810c00 x7 : 6cad9ff29d8de19c x6 : ff94ec6f0ce3656c x5 :
ffff8000ecec402a x4 : ffff8000f3000000 x3 : ffff00001325bc78 x2 :
ffff00001325bb5c x1 : 00000000000674cf x0 : ffff80007a810c00 Call trace:

ditto

Quoted 13 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff000012a3ba90
x29: ffff000012a3ba90 x28: 00000000ce99f075 x27: ffffffffffffffff x26:
0000000000000007 x25: ffff8000ecec702a x24: ffff000012a3bb5c x23:
0000000000000007 x22: 00000007ce99f075 x21: ffff80007a810c00 x20:
0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17:
0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14:
0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11:
fefefefefefefeff x10: d0bbbcbfa6b2b9bc x9 : 0000000000000000 x8 :
ffff80007a810c00 x7 : 6cad9ff29d8de19c x6 : ff94ec6f0ce3656c x5 :
ffff8000ecec702a x4 : ffff8000f3000000 x3 : ffff000012a3bc78 x2 :
ffff000012a3bb5c x1 : 00000000000674cf x0 : ffff80007a810c00 Call trace:

ditto

Quoted 13 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff0000132bba90
x29: ffff0000132bba90 x28: 00000000ce99e1a6 x27: ffffffffffffffff x26:
000000000000000c x25: ffff8000f21dd036 x24: ffff0000132bbb5c x23:
0000000000000004 x22: 0000000cce99e1a6 x21: ffff800074dd8d80 x20:
0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17:
0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14:
0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11:
fefefefefefefeff x10: d0d0d0d0b8fea4b3 x9 : 40bcd8645005512e x8 :
c433ade89ebd10f9 x7 : c6b69091eeb194d2 x6 : 848f758ca69635b4 x5 :
ffff8000f21dd036 x4 : ffff8000f3000000 x3 : ffff0000132bbc78 x2 :
ffff0000132bbb5c x1 : 00000000000674cf x0 : ffff800074dd8d80 Call trace:

... and ditto:

Quoted 13 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff000013263a90
x29: ffff000013263a90 x28: 00000000ce99e1a6 x27: ffffffffffffffff x26:
000000000000000c x25: ffff8000f0a6f036 x24: ffff000013263b5c x23:
0000000000000004 x22: 0000000cce99e1a6 x21: ffff800074dd8d80 x20:
0000000000000000 x19: 0000880001000000 x18: 0000000000000000 x17:
0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14:
0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11:
fefefefefefefeff x10: d0d0d0d0b8fea4b3 x9 : 40bcd8645005512e x8 :
c433ade89ebd10f9 x7 : c6b69091eeb194d2 x6 : 848f758ca69635b4 x5 :
ffff8000f0a6f036 x4 : ffff8000f3000000 x3 : ffff000013263c78 x2 :
ffff000013263b5c x1 : 00000000000674cf x0 : ffff800074dd8d80 Call trace:

All of those run under rcu_read_lock() and no dentry with DCACHE_NORCU has ever been inserted into a hash chain, so it doesn't look like a plain use-after-free. Could you try something like the following to see a bit more about where it comes from?

So far it looks like something is buggering a forward reference in hash chain in a fairly specific way - the values seen had been 00000000010000000 and
00008800010000000. Does that smell like anything from arm64-specific data structures (PTE, etc.)?

Alternatively, we might've gone off rails a step (or more) before, with the previous iteration going through bogus, but at least mapped address - the one that has never been a dentry in the first place.

fs/dcache.c +26 -2
index c435398f2c81..cb555edb5b55 100644
@@ -2114,6 +2114,22 @@ static inline bool d_same_name(const struct dentry *dentry,
 				       name) == 0;
 }
 
+static void dump(struct dentry *dentry)
+{
+	int i;
+	if (!dentry) {
+		printk(KERN_ERR "list fucked in head");
+		return;
+	}
+	printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = %x, count = %d",
+			dentry, dentry->d_hash.next, dentry->d_flags,
+			dentry->d_lockref.count
+			);
+	for (i = 0; i < sizeof(struct dentry); i++)
+		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',
+			((unsigned char *)dentry)[i]);
+}
+
 /**
  * __d_lookup_rcu - search for a dentry (racy, store-free)
  * @parent: parent dentry
@@ -2151,7 +2167,7 @@ struct dentry *__d_lookup_rcu(const struct dentry *parent,
 	const unsigned char *str = name->name;
 	struct hlist_bl_head *b = d_hash(hashlen_hash(hashlen));
 	struct hlist_bl_node *node;
-	struct dentry *dentry;
+	struct dentry *dentry, *last = NULL;
 
 	/*
 	 * Note: There is significant duplication with __d_lookup_rcu which is
@@ -2176,6 +2192,10 @@ struct dentry *__d_lookup_rcu(const struct dentry *parent,
 	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
 		unsigned seq;
 
+		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
+			dump(last);
+		last = dentry;
+
 seqretry:
 		/*
 		 * The dentry sequence count protects us from concurrent
@@ -2274,7 +2294,7 @@ struct dentry *__d_lookup(const struct dentry *parent, const struct qstr *name)
 	struct hlist_bl_head *b = d_hash(hash);
 	struct hlist_bl_node *node;
 	struct dentry *found = NULL;
-	struct dentry *dentry;
+	struct dentry *dentry, *last = NULL;
 
 	/*
 	 * Note: There is significant duplication with __d_lookup_rcu which is
@@ -2300,6 +2320,10 @@ struct dentry *__d_lookup(const struct dentry *parent, const struct qstr *name)
 	
 	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
 
+		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
+			dump(last);
+		last = dentry;
+
 		if (dentry->d_name.hash != hash)
 			continue;
 
maintainer
#7 lore raw
Quoted 5 lines · Al Viro

So far it looks like something is buggering a forward reference in hash chain in a fairly specific way - the values seen had been 00000000010000000 and
00008800010000000. Does that smell like anything from arm64-specific data structures (PTE, etc.)?

make that 0000000001000000 and 0000880001000000 resp. Tests in the patch are correct, just mistyped it here...

Quoted 3 lines

Alternatively, we might've gone off rails a step (or more) before, with the previous iteration going through bogus, but at least mapped address - the one that has never been a dentry in the first place.

#8 lore raw
In reply to Al Viro
Quoted 133 lines · Al Viro
Quoted 1 line · Vicente Bergas

On Wednesday, May 22, 2019 6:29:46 PM CEST, Al Viro wrote: ...

__d_lookup() running into &dentry->d_hash == 0x01000000 at some point in hash chain and trying to look at ->d_name.hash:

Quoted 5 lines
pc : __d_lookup+0x58/0x198
lr : d_lookup+0x38/0x68

sp : ffff000012663b90
x29: ffff000012663b90 x28: ffff000012663d58 x27: 0000000000000000 x26:
ffff8000ae7cc900 x25: 0000000000000001 x24: ffffffffffffffff x23: ...

__d_lookup_rcu() running into &dentry->d_hash == 0x01000000 at some point in hash chain and trying to look at ->d_seq:

Quoted 5 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff0000130b3b60
x29: ffff0000130b3b60 x28: 00000000ce99d070 x27: ffffffffffffffff x26:
0000000000000026 x25: ffff8000ecec6030 x24: ffff0000130b3c2c x23: ...

__d_lookup_rcu() running into &dentry->d_hash == 0x0000880001000000 at some point in hash chain and trying to look at ->d_seq:

Quoted 5 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff00001325ba90
x29: ffff00001325ba90 x28: 00000000ce99f075 x27: ffffffffffffffff x26:
0000000000000007 x25: ffff8000ecec402a x24: ffff00001325bb5c x23: ...

ditto

Quoted 5 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff000012a3ba90
x29: ffff000012a3ba90 x28: 00000000ce99f075 x27: ffffffffffffffff x26:
0000000000000007 x25: ffff8000ecec702a x24: ffff000012a3bb5c x23: ...

ditto

Quoted 5 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff0000132bba90
x29: ffff0000132bba90 x28: 00000000ce99e1a6 x27: ffffffffffffffff x26:
000000000000000c x25: ffff8000f21dd036 x24: ffff0000132bbb5c x23: ...

... and ditto:

Quoted 5 lines
pc : __d_lookup_rcu+0x68/0x198
lr : lookup_fast+0x44/0x2e8

sp : ffff000013263a90
x29: ffff000013263a90 x28: 00000000ce99e1a6 x27: ffffffffffffffff x26:
000000000000000c x25: ffff8000f0a6f036 x24: ffff000013263b5c x23: ...

All of those run under rcu_read_lock() and no dentry with DCACHE_NORCU has ever been inserted into a hash chain, so it doesn't look like a plain use-after-free. Could you try something like the following to see a bit more about where it comes from?

So far it looks like something is buggering a forward reference in hash chain in a fairly specific way - the values seen had been 00000000010000000 and
00008800010000000. Does that smell like anything from arm64-specific data structures (PTE, etc.)?

Alternatively, we might've gone off rails a step (or more) before, with the previous iteration going through bogus, but at least mapped address - the one that has never been a dentry in the first place.

fs/dcache.c +16 -0
index c435398f2c81..cb555edb5b55 100644
@@ -2114,6 +2114,22 @@ static inline bool d_same_name(const 
 struct dentry *dentry,
 				       name) == 0;
 }
 
+static void dump(struct dentry *dentry)
+{
+	int i;
+	if (!dentry) {
+		printk(KERN_ERR "list fucked in head");
+		return;
+	}
+	printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = 
 %x, count = %d",
+			dentry, dentry->d_hash.next, dentry->d_flags,
+			dentry->d_lockref.count
+			);
+	for (i = 0; i < sizeof(struct dentry); i++)
+		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',
+			((unsigned char *)dentry)[i]);
+}
+
 /**

* __d_lookup_rcu - search for a dentry (racy, store-free)
* @parent: parent dentry

fs/dcache.c +1 -1
@@ -2151,7 +2167,7 @@ struct dentry *__d_lookup_rcu(const 
 struct dentry *parent,
 	const unsigned char *str = name->name;
 	struct hlist_bl_head *b = d_hash(hashlen_hash(hashlen));
 	struct hlist_bl_node *node;
-	struct dentry *dentry;
+	struct dentry *dentry, *last = NULL;
 
 	/*
fs/dcache.c +4 -0
 	 * Note: There is significant duplication with __d_lookup_rcu which is
@@ -2176,6 +2192,10 @@ struct dentry *__d_lookup_rcu(const 
 struct dentry *parent,
 	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
 		unsigned seq;
 
+		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
+			dump(last);
+		last = dentry;
+
 seqretry:
 		/*
fs/dcache.c +1 -1
 		 * The dentry sequence count protects us from concurrent
@@ -2274,7 +2294,7 @@ struct dentry *__d_lookup(const struct 
 dentry *parent, const struct qstr *name)
 	struct hlist_bl_head *b = d_hash(hash);
 	struct hlist_bl_node *node;
 	struct dentry *found = NULL;
-	struct dentry *dentry;
+	struct dentry *dentry, *last = NULL;
 
 	/*
fs/dcache.c +0 -0
 	 * Note: There is significant duplication with __d_lookup_rcu which is

fs/dcache.c struct dentry *__d_lookup(const struct · line 2329

Quoted 10 lines
fs/dcache.c +4 -0
@@ -2300,6 +2320,10 @@ struct dentry *__d_lookup(const struct 
 dentry *parent, const struct qstr *name)
 
 	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
 
+		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
+			dump(last);
+		last = dentry;
+
 		if (dentry->d_name.hash != hash)
 			continue;

Hi Al,
i have been running the distro-provided kernel the last few weeks and had no issues at all.
https://archlinuxarm.org/packages/aarch64/linux-aarch64
It is from the v5.1 branch and is compiled with gcc 8.3.

IIRC, i also tested
https://archlinuxarm.org/packages/aarch64/linux-aarch64-rc
v5.2-rc1 and v5.2-rc2 (which at that time where compiled with gcc 8.2) with no issues.

This week tested v5.2-rc4 and v5.2-rc5 from archlinuxarm but there are regressions unrelated to d_lookup.

At this point i was convinced it was a gcc 9.1 issue and had nothing to do with the kernel, but anyways i gave your patch a try. The tested kernel is v5.2-rc5-224-gbed3c0d84e7e and
it has been compiled with gcc 8.3.
The sentinel you put there has triggered!
So, it is not a gcc 9.1 issue.

In any case, i have no idea if those addresses are arm64-specific in any way.

Regards,
Vicenç.

list fucked in head

Unable to handle kernel paging request at virtual address 0000000000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000005c989000 [0000000000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#1] SMP
CPU: 4 PID: 2427 Comm: git Not tainted 5.2.0-rc5 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : __d_lookup_rcu+0x90/0x1e8
lr : __d_lookup_rcu+0x84/0x1e8
sp : ffff000013413a90
x29: ffff000013413a90 x28: ffff000013413b5c x27: 0000000000000002 x26: ffff000013413c78 x25: 0000001ac1084259 x24: 0000000000fffff8 x23: 0000000001000000 x22: ffff8000586ed9c0 x21: ffff8000586ed9c0 x20: 0000000000fffff8 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: 0000000000000000 x10: ffffffffffffffff x9 : 00000000c1084259 x8 : ffff800054f31032 x7 : 000000000000001a x6 : ffff00001090454b x5 : 0000000000000013 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 00000000ffffffff x1 : 00008000e6f46000 x0 : 0000000000000013 Call trace:
__d_lookup_rcu+0x90/0x1e8
lookup_fast+0x44/0x300
walk_component+0x34/0x2e0
path_lookupat.isra.13+0x5c/0x1e0
filename_lookup.part.19+0x6c/0xe8
user_path_at_empty+0x4c/0x60
vfs_statx+0x78/0xd8
__se_sys_newfstatat+0x24/0x48
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x94/0x138
el0_svc+0x8/0xc
Code: 94000753 294c1fe9 9280000a f94037e8 (b85fc263) ---[ end trace 93a444e9b6bc67e8 ]---
list fucked in head
Unable to handle kernel paging request at virtual address 0000000000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000005c989000 [0000000000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#2] SMP
CPU: 5 PID: 2424 Comm: git Tainted: G      D           5.2.0-rc5 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : __d_lookup_rcu+0x90/0x1e8
lr : __d_lookup_rcu+0x84/0x1e8
sp : ffff0000133fba90
x29: ffff0000133fba90 x28: ffff0000133fbb5c x27: 0000000000000002 x26: ffff0000133fbc78 x25: 0000001ac1084259 x24: 0000000000fffff8 x23: 0000000001000000 x22: ffff8000586ed9c0 x21: ffff8000586ed9c0 x20: 0000000000fffff8 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: 0000000000000000 x10: ffffffffffffffff x9 : 00000000c1084259 x8 : ffff800079d7b032 x7 : 000000000000001a x6 : ffff00001090454b x5 : 0000000000000013 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 00000000ffffffff x1 : 00008000e6f5a000 x0 : 0000000000000013 Call trace:
__d_lookup_rcu+0x90/0x1e8
lookup_fast+0x44/0x300
walk_component+0x34/0x2e0
path_lookupat.isra.13+0x5c/0x1e0
filename_lookup.part.19+0x6c/0xe8
user_path_at_empty+0x4c/0x60
vfs_statx+0x78/0xd8
__se_sys_newfstatat+0x24/0x48
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x94/0x138
el0_svc+0x8/0xc
Code: 94000753 294c1fe9 9280000a f94037e8 (b85fc263) ---[ end trace 93a444e9b6bc67e9 ]---
list fucked in head
Unable to handle kernel paging request at virtual address 0000880000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000003ba5d000 [0000880000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#3] SMP
CPU: 2 PID: 2659 Comm: git Tainted: G      D           5.2.0-rc5 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : __d_lookup_rcu+0x90/0x1e8
lr : __d_lookup_rcu+0x84/0x1e8
sp : ffff0000135cba90
x29: ffff0000135cba90 x28: ffff0000135cbb5c x27: 0000000000000000 x26: ffff0000135cbc78 x25: 00000010cb63a9bb x24: 0000880000fffff8 x23: 0000000001000000 x22: ffff80003be53180 x21: ffff80003be53180 x20: 0000880000fffff8 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: 0000000000000000 x10: ffffffffffffffff x9 : 00000000cb63a9bb x8 : ffff8000f094102e x7 : 0000000000000010 x6 : ffff00001090454b x5 : 0000000000000013 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 00000000ffffffff x1 : 00008000e6f1e000 x0 : 0000000000000013 Call trace:
__d_lookup_rcu+0x90/0x1e8
lookup_fast+0x44/0x300
walk_component+0x34/0x2e0
path_lookupat.isra.13+0x5c/0x1e0
filename_lookup.part.19+0x6c/0xe8
user_path_at_empty+0x4c/0x60
vfs_statx+0x78/0xd8
__se_sys_newfstatat+0x24/0x48
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x94/0x138
el0_svc+0x8/0xc
Code: 94000753 294c1fe9 9280000a f94037e8 (b85fc263) ---[ end trace 93a444e9b6bc67ea ]---
list fucked in head
Unable to handle kernel paging request at virtual address 0000880000fffffc

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000003ba5d000 [0000880000fffffc] pgd=0000000000000000

Internal error: Oops: 96000004 [#4] SMP
CPU: 4 PID: 2658 Comm: git Tainted: G      D           5.2.0-rc5 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : __d_lookup_rcu+0x90/0x1e8
lr : __d_lookup_rcu+0x84/0x1e8
sp : ffff00001363ba90
x29: ffff00001363ba90 x28: ffff00001363bb5c x27: 0000000000000000 x26: ffff00001363bc78 x25: 00000010cb63a9bb x24: 0000880000fffff8 x23: 0000000001000000 x22: ffff80003be53180 x21: ffff80003be53180 x20: 0000880000fffff8 x19: 0000880001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: 0000000000000000 x10: ffffffffffffffff x9 : 00000000cb63a9bb x8 : ffff80004b94d02e x7 : 0000000000000010 x6 : ffff00001090454b x5 : 0000000000000013 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 00000000ffffffff x1 : 00008000e6f46000 x0 : 0000000000000013 Call trace:
__d_lookup_rcu+0x90/0x1e8
lookup_fast+0x44/0x300
walk_component+0x34/0x2e0
path_lookupat.isra.13+0x5c/0x1e0
filename_lookup.part.19+0x6c/0xe8
user_path_at_empty+0x4c/0x60
vfs_statx+0x78/0xd8
__se_sys_newfstatat+0x24/0x48
__arm64_sys_newfstatat+0x18/0x20
el0_svc_handler+0x94/0x138
el0_svc+0x8/0xc
Code: 94000753 294c1fe9 9280000a f94037e8 (b85fc263) ---[ end trace 93a444e9b6bc67eb ]---
list fucked in head
Unable to handle kernel paging request at virtual address 0000000001000018

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000002091a000 [0000000001000018] pgd=0000000000000000

Internal error: Oops: 96000004 [#5] SMP
CPU: 4 PID: 3205 Comm: update_all_gits Tainted: G      D           5.2.0-rc5 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : __d_lookup+0x88/0x1d8
lr : __d_lookup+0x7c/0x1d8
sp : ffff000013dabaa0
x29: ffff000013dabaa0 x28: ffff000013dabbd8 x27: ffff00001076f0f8 x26: ffff8000f2808780 x25: 0000000000fffff8 x24: 0000000001000000 x23: 00000000cb639d51 x22: 0000000000000000 x21: 0000000000000001 x20: 0000000000fffff8 x19: 0000000001000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: ffffffffffffffff x14: ffff000010898508 x13: ffff000013dabbf8 x12: ffff000013dabbed x11: 0000000000000001 x10: ffff000013daba60 x9 : ffff000013daba60 x8 : ffff000013daba60 x7 : ffff000013daba60 x6 : ffffffffffffffff x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 00000000ffffffff x1 : 00008000e6f46000 x0 : 0000000000000013 Call trace:
__d_lookup+0x88/0x1d8
d_lookup+0x34/0x68
d_hash_and_lookup+0x50/0x68
proc_flush_task+0x9c/0x198
release_task.part.3+0x68/0x4b8
wait_consider_task+0x91c/0x9b0
do_wait+0x120/0x1e0
kernel_wait4+0x7c/0x140
__se_sys_wait4+0x68/0xa8
__arm64_sys_wait4+0x18/0x20
el0_svc_handler+0x94/0x138
el0_svc+0x8/0xc
Code: 940006db b9406fe5 92800006 d503201f (b9402282) ---[ end trace 93a444e9b6bc67ec ]---
list fucked in head
Unable to handle kernel paging request at virtual address 0000880101000018

Mem abort info:
ESR = 0x96000004
Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000009bddd000 [0000880101000018] pgd=0000000000000000

Internal error: Oops: 96000004 [#6] SMP
CPU: 5 PID: 3978 Comm: tar Tainted: G      D           5.2.0-rc5 #1
Hardware name: Sapphire-RK3399 Board (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : __d_lookup+0x88/0x1d8
lr : __d_lookup+0x7c/0x1d8
sp : ffff000014dc3b90
x29: ffff000014dc3b90 x28: ffff000014dc3d58 x27: ffff000014dc3d48 x26: ffff8000a77becc0 x25: 0000880100fffff8 x24: 0000000001000000 x23: 00000000c1086fd8 x22: 0000000000000000 x21: 0000000000000001 x20: 0000880100fffff8 x19: 0000880101000000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 x11: 0000000000000000 x10: ffff000014dc3b50 x9 : ffff000014dc3b50 x8 : ffff000014dc3b50 x7 : ffff000014dc3b50 x6 : ffffffffffffffff x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 00000000ffffffff x1 : 00008000e6f5a000 x0 : 0000000000000013 Call trace:
__d_lookup+0x88/0x1d8
d_lookup+0x34/0x68
path_openat+0x528/0xfd0
do_filp_open+0x60/0xc0
do_sys_open+0x164/0x200
__arm64_sys_openat+0x20/0x28
el0_svc_handler+0x94/0x138
el0_svc+0x8/0xc
Code: 940006db b9406fe5 92800006 d503201f (b9402282) ---[ end trace 93a444e9b6bc67ed ]---

0000000000002d10 <__d_lookup_rcu>:
{
2d10: a9b97bfd stp x29, x30, [sp, #-112]!

	return dentry_hashtable + (hash >> d_hash_shift);

2d14: 90000003 adrp x3, 0 <d_shrink_del>
2d18: 91000065 add x5, x3, #0x0
{
2d1c: 910003fd mov x29, sp
2d20: a90153f3 stp x19, x20, [sp, #16]
2d24: a90363f7 stp x23, x24, [sp, #48]
2d28: a9046bf9 stp x25, x26, [sp, #64]

	const unsigned char *str = name->name;
   2d2c:	a9402039 	ldp	x25, x8, [x1]
	return dentry_hashtable + (hash >> d_hash_shift);
   2d30:	f9400064 	ldr	x4, [x3]
   2d34:	b94008a3 	ldr	w3, [x5, #8]
   2d38:	1ac32723 	lsr	w3, w25, w3
	__READ_ONCE_SIZE;
   2d3c:	f8637893 	ldr	x19, [x4, x3, lsl #3]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2d40:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2d44:	54000420 	b.eq	2dc8 <__d_lookup_rcu+0xb8>  // b.none
			if (dentry_cmp(dentry, str, hashlen_len(hashlen)) != 0)
   2d48:	d360ff27 	lsr	x7, x25, #32
   2d4c:	2a1903e9 	mov	w9, w25
   2d50:	aa0103fa 	mov	x26, x1
   2d54:	a90573fb 	stp	x27, x28, [sp, #80]
   2d58:	aa0203fc 	mov	x28, x2
   2d5c:	120008fb 	and	w27, w7, #0x7
		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
   2d60:	52a02017 	mov	w23, #0x1000000             	// #16777216
	mask = bytemask_from_count(tcount);
   2d64:	9280000a 	mov	x10, #0xffffffffffffffff    	// #-1
   2d68:	a9025bf5 	stp	x21, x22, [sp, #32]
   2d6c:	aa0003f5 	mov	x21, x0
	struct dentry *dentry, *last = NULL;
   2d70:	d2800000 	mov	x0, #0x0                   	// #0
   2d74:	d503201f 	nop
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2d78:	d1002274 	sub	x20, x19, #0x8
		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
   2d7c:	6b17027f 	cmp	w19, w23
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2d80:	aa1403f8 	mov	x24, x20
		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
   2d84:	540000e1 	b.ne	2da0 <__d_lookup_rcu+0x90>  // b.any
   2d88:	290c1fe9 	stp	w9, w7, [sp, #96]
   2d8c:	f90037e8 	str	x8, [sp, #104]
			dump(last);
   2d90:	94000000 	bl	0 <d_shrink_del>
   2d94:	294c1fe9 	ldp	w9, w7, [sp, #96]
   2d98:	9280000a 	mov	x10, #0xffffffffffffffff    	// #-1
   2d9c:	f94037e8 	ldr	x8, [sp, #104]
   2da0:	b85fc263 	ldur	w3, [x19, #-4]
	smp_rmb();
   2da4:	d50339bf 	dmb	ishld
		if (dentry->d_parent != parent)
   2da8:	f9400e80 	ldr	x0, [x20, #24]
   2dac:	eb15001f 	cmp	x0, x21
   2db0:	540001a0 	b.eq	2de4 <__d_lookup_rcu+0xd4>  // b.none
   2db4:	f9400273 	ldr	x19, [x19]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2db8:	aa1403e0 	mov	x0, x20
   2dbc:	b5fffdf3 	cbnz	x19, 2d78 <__d_lookup_rcu+0x68>
   2dc0:	a9425bf5 	ldp	x21, x22, [sp, #32]
   2dc4:	a94573fb 	ldp	x27, x28, [sp, #80]
	return NULL;

2dc8: d2800018 mov x24, #0x0 // #0
}
2dcc: aa1803e0 mov x0, x24
2dd0: a94153f3 ldp x19, x20, [sp, #16]
2dd4: a94363f7 ldp x23, x24, [sp, #48]
2dd8: a9446bf9 ldp x25, x26, [sp, #64]
2ddc: a8c77bfd ldp x29, x30, [sp], #112
2de0: d65f03c0 ret
if (d_unhashed(dentry))
2de4: f9400660 ldr x0, [x19, #8]
2de8: b4fffe60 cbz x0, 2db4 <__d_lookup_rcu+0xa4>

		if (unlikely(parent->d_flags & DCACHE_OP_COMPARE)) {
   2dec:	b94002a0 	ldr	w0, [x21]
	return ret & ~1;
   2df0:	121f7876 	and	w22, w3, #0xfffffffe
   2df4:	370804a0 	tbnz	w0, #1, 2e88 <__d_lookup_rcu+0x178>
			if (dentry->d_name.hash_len != hashlen)
   2df8:	f9401280 	ldr	x0, [x20, #32]
   2dfc:	eb19001f 	cmp	x0, x25
   2e00:	54fffda1 	b.ne	2db4 <__d_lookup_rcu+0xa4>  // b.any
   2e04:	f9401264 	ldr	x4, [x19, #32]
	const unsigned char *cs = READ_ONCE(dentry->d_name.name);
   2e08:	2a0703e5 	mov	w5, w7
   2e0c:	cb040101 	sub	x1, x8, x4
   2e10:	14000006 	b	2e28 <__d_lookup_rcu+0x118>
		cs += sizeof(unsigned long);
   2e14:	91002084 	add	x4, x4, #0x8
		if (unlikely(a != b))
   2e18:	eb06001f 	cmp	x0, x6
   2e1c:	54fffcc1 	b.ne	2db4 <__d_lookup_rcu+0xa4>  // b.any
		if (!tcount)
   2e20:	710020a5 	subs	w5, w5, #0x8
   2e24:	54000160 	b.eq	2e50 <__d_lookup_rcu+0x140>  // b.none
		cs += sizeof(unsigned long);

2e28: 8b010083 add x3, x4, x1
if (tcount < sizeof(unsigned long))
2e2c: 6b1b00bf cmp w5, w27
static inline unsigned long load_unaligned_zeropad(const void *addr)

{
	unsigned long ret, offset;

	/* Load word from unaligned pointer addr */
	asm(

2e30: f9400066 ldr x6, [x3]

static __no_kasan_or_inline
unsigned long read_word_at_a_time(const void *addr)

{
	kasan_check_read(addr, 1);
	return *(unsigned long *)addr;
   2e34:	f9400080 	ldr	x0, [x4]
   2e38:	54fffee1 	b.ne	2e14 <__d_lookup_rcu+0x104>  // b.any
	mask = bytemask_from_count(tcount);
   2e3c:	531d7361 	lsl	w1, w27, #3
	return unlikely(!!((a ^ b) & mask));
   2e40:	ca060000 	eor	x0, x0, x6
	mask = bytemask_from_count(tcount);
   2e44:	9ac12141 	lsl	x1, x10, x1
			if (dentry_cmp(dentry, str, hashlen_len(hashlen)) != 0)
   2e48:	ea21001f 	bics	xzr, x0, x1
   2e4c:	54fffb41 	b.ne	2db4 <__d_lookup_rcu+0xa4>  // b.any
		*seqp = seq;

2e50: b9000396 str w22, [x28]
}
2e54: aa1803e0 mov x0, x24
2e58: a94153f3 ldp x19, x20, [sp, #16]

		return dentry;

2e5c: a9425bf5 ldp x21, x22, [sp, #32]
}
2e60: a94363f7 ldp x23, x24, [sp, #48]
2e64: a9446bf9 ldp x25, x26, [sp, #64]

		return dentry;

2e68: a94573fb ldp x27, x28, [sp, #80]
}
2e6c: a8c77bfd ldp x29, x30, [sp], #112
2e70: d65f03c0 ret
if (d_unhashed(dentry))
2e74: f9400660 ldr x0, [x19, #8]
2e78: 121f7876 and w22, w3, #0xfffffffe
2e7c: b4fff9c0 cbz x0, 2db4 <__d_lookup_rcu+0xa4>

		if (unlikely(parent->d_flags & DCACHE_OP_COMPARE)) {
   2e80:	b94002a0 	ldr	w0, [x21]
   2e84:	360ffba0 	tbz	w0, #1, 2df8 <__d_lookup_rcu+0xe8>
			if (dentry->d_name.hash != hashlen_hash(hashlen))
   2e88:	b9402280 	ldr	w0, [x20, #32]
   2e8c:	6b00013f 	cmp	w9, w0
   2e90:	54fff921 	b.ne	2db4 <__d_lookup_rcu+0xa4>  // b.any
			tlen = dentry->d_name.len;
   2e94:	b9402681 	ldr	w1, [x20, #36]
			tname = dentry->d_name.name;
   2e98:	f9401682 	ldr	x2, [x20, #40]
	smp_rmb();
   2e9c:	d50339bf 	dmb	ishld
	return unlikely(s->sequence != start);
   2ea0:	b85fc260 	ldur	w0, [x19, #-4]
			if (read_seqcount_retry(&dentry->d_seq, seq)) {
   2ea4:	6b0002df 	cmp	w22, w0
   2ea8:	54000100 	b.eq	2ec8 <__d_lookup_rcu+0x1b8>  // b.none
   2eac:	d503203f 	yield
	__READ_ONCE_SIZE;
   2eb0:	b85fc263 	ldur	w3, [x19, #-4]
	smp_rmb();
   2eb4:	d50339bf 	dmb	ishld
		if (dentry->d_parent != parent)
   2eb8:	f9400e80 	ldr	x0, [x20, #24]
   2ebc:	eb15001f 	cmp	x0, x21
   2ec0:	54fff7a1 	b.ne	2db4 <__d_lookup_rcu+0xa4>  // b.any
   2ec4:	17ffffec 	b	2e74 <__d_lookup_rcu+0x164>
			if (parent->d_op->d_compare(dentry,

2ec8: f94032a4 ldr x4, [x21, #96]
2ecc: aa1a03e3 mov x3, x26
2ed0: aa1403e0 mov x0, x20
2ed4: 290c1fe9 stp w9, w7, [sp, #96]
2ed8: f90037e8 str x8, [sp, #104]
2edc: f9400c84 ldr x4, [x4, #24]
2ee0: d63f0080 blr x4
2ee4: 9280000a mov x10, #0xffffffffffffffff // #-1
2ee8: 294c1fe9 ldp w9, w7, [sp, #96]
2eec: f94037e8 ldr x8, [sp, #104]
2ef0: 34fffb00 cbz w0, 2e50 <__d_lookup_rcu+0x140>
2ef4: 17ffffb0 b 2db4 <__d_lookup_rcu+0xa4>

0000000000002ef8 <__d_lookup>:
{
2ef8: a9b97bfd stp x29, x30, [sp, #-112]!

	return dentry_hashtable + (hash >> d_hash_shift);

2efc: 90000002 adrp x2, 0 <d_shrink_del>
2f00: 91000044 add x4, x2, #0x0
{
2f04: 910003fd mov x29, sp
2f08: a90153f3 stp x19, x20, [sp, #16]
2f0c: a90363f7 stp x23, x24, [sp, #48]
2f10: a9046bf9 stp x25, x26, [sp, #64]

	return dentry_hashtable + (hash >> d_hash_shift);
   2f14:	f9400043 	ldr	x3, [x2]
	unsigned int hash = name->hash;
   2f18:	b9400037 	ldr	w23, [x1]
	return dentry_hashtable + (hash >> d_hash_shift);
   2f1c:	b9400882 	ldr	w2, [x4, #8]
   2f20:	1ac226e2 	lsr	w2, w23, w2
   2f24:	f8627873 	ldr	x19, [x3, x2, lsl #3]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2f28:	f27ffa73 	ands	x19, x19, #0xfffffffffffffffe
   2f2c:	54000520 	b.eq	2fd0 <__d_lookup+0xd8>  // b.none
   2f30:	aa0003fa 	mov	x26, x0
   2f34:	a90573fb 	stp	x27, x28, [sp, #80]
   2f38:	aa0103fc 	mov	x28, x1
   2f3c:	d2800002 	mov	x2, #0x0                   	// #0
		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
   2f40:	52a02018 	mov	w24, #0x1000000             	// #16777216
	smp_store_release(&lock->locked, 0);
   2f44:	52800005 	mov	w5, #0x0                   	// #0
	mask = bytemask_from_count(tcount);
   2f48:	92800006 	mov	x6, #0xffffffffffffffff    	// #-1
   2f4c:	a9025bf5 	stp	x21, x22, [sp, #32]
   2f50:	d2800016 	mov	x22, #0x0                   	// #0
   2f54:	52800035 	mov	w21, #0x1                   	// #1
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2f58:	d1002274 	sub	x20, x19, #0x8
		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
   2f5c:	6b18027f 	cmp	w19, w24
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2f60:	aa1403f9 	mov	x25, x20
		if (unlikely((u32)(unsigned long)&dentry->d_hash == 0x01000000))
   2f64:	540000e1 	b.ne	2f80 <__d_lookup+0x88>  // b.any
			dump(last);
   2f68:	aa0203e0 	mov	x0, x2
   2f6c:	b9006fe5 	str	w5, [sp, #108]
   2f70:	94000000 	bl	0 <d_shrink_del>
   2f74:	b9406fe5 	ldr	w5, [sp, #108]
   2f78:	92800006 	mov	x6, #0xffffffffffffffff    	// #-1
   2f7c:	d503201f 	nop
		if (dentry->d_name.hash != hash)
   2f80:	b9402282 	ldr	w2, [x20, #32]
   2f84:	6b17005f 	cmp	w2, w23
   2f88:	540001a1 	b.ne	2fbc <__d_lookup+0xc4>  // b.any
   2f8c:	9101427b 	add	x27, x19, #0x50
   2f90:	f9800371 	prfm	pstl1strm, [x27]
   2f94:	885fff61 	ldaxr	w1, [x27]
   2f98:	4a160020 	eor	w0, w1, w22
   2f9c:	35000060 	cbnz	w0, 2fa8 <__d_lookup+0xb0>
   2fa0:	88007f75 	stxr	w0, w21, [x27]
   2fa4:	35ffff80 	cbnz	w0, 2f94 <__d_lookup+0x9c>
   2fa8:	35000521 	cbnz	w1, 304c <__d_lookup+0x154>
		if (dentry->d_parent != parent)
   2fac:	f9400e80 	ldr	x0, [x20, #24]
   2fb0:	eb1a001f 	cmp	x0, x26
   2fb4:	540001c0 	b.eq	2fec <__d_lookup+0xf4>  // b.none
   2fb8:	089fff65 	stlrb	w5, [x27]
   2fbc:	f9400273 	ldr	x19, [x19]
	hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
   2fc0:	aa1403e2 	mov	x2, x20
   2fc4:	b5fffcb3 	cbnz	x19, 2f58 <__d_lookup+0x60>
   2fc8:	a9425bf5 	ldp	x21, x22, [sp, #32]
   2fcc:	a94573fb 	ldp	x27, x28, [sp, #80]
	struct dentry *found = NULL;

2fd0: d2800019 mov x25, #0x0 // #0
}
2fd4: aa1903e0 mov x0, x25
2fd8: a94153f3 ldp x19, x20, [sp, #16]
2fdc: a94363f7 ldp x23, x24, [sp, #48]
2fe0: a9446bf9 ldp x25, x26, [sp, #64]
2fe4: a8c77bfd ldp x29, x30, [sp], #112
2fe8: d65f03c0 ret
if (d_unhashed(dentry))
2fec: f9400660 ldr x0, [x19, #8]
2ff0: b4fffe40 cbz x0, 2fb8 <__d_lookup+0xc0>

	if (likely(!(parent->d_flags & DCACHE_OP_COMPARE))) {
   2ff4:	b9400340 	ldr	w0, [x26]
   2ff8:	b9402681 	ldr	w1, [x20, #36]
   2ffc:	37080340 	tbnz	w0, #1, 3064 <__d_lookup+0x16c>
		if (dentry->d_name.len != name->len)
   3000:	b9400783 	ldr	w3, [x28, #4]
   3004:	6b01007f 	cmp	w3, w1
   3008:	54fffd81 	b.ne	2fb8 <__d_lookup+0xc0>  // b.any
		return dentry_cmp(dentry, name->name, name->len) == 0;

300c: f9400787 ldr x7, [x28, #8]
static inline int dentry_string_cmp(const unsigned char *cs, const unsigned char *ct, unsigned tcount)
3010: 12000868 and w8, w3, #0x7
3014: f9401264 ldr x4, [x19, #32]
3018: cb0400e7 sub x7, x7, x4
301c: 14000003 b 3028 <__d_lookup+0x130>
if (!tcount)
3020: 71002063 subs w3, w3, #0x8
3024: 54000380 b.eq 3094 <__d_lookup+0x19c> // b.none

		cs += sizeof(unsigned long);
   3028:	8b070082 	add	x2, x4, x7
		if (tcount < sizeof(unsigned long))
   302c:	6b08007f 	cmp	w3, w8
   3030:	f9400041 	ldr	x1, [x2]
	return *(unsigned long *)addr;
   3034:	f9400080 	ldr	x0, [x4]
   3038:	540003e0 	b.eq	30b4 <__d_lookup+0x1bc>  // b.none
		cs += sizeof(unsigned long);
   303c:	91002084 	add	x4, x4, #0x8
		if (unlikely(a != b))
   3040:	eb01001f 	cmp	x0, x1
   3044:	54fffee0 	b.eq	3020 <__d_lookup+0x128>  // b.none
   3048:	17ffffdc 	b	2fb8 <__d_lookup+0xc0>
	queued_spin_lock_slowpath(lock, val);
   304c:	aa1b03e0 	mov	x0, x27
   3050:	b9006fe5 	str	w5, [sp, #108]
   3054:	94000000 	bl	0 <queued_spin_lock_slowpath>
   3058:	b9406fe5 	ldr	w5, [sp, #108]
   305c:	92800006 	mov	x6, #0xffffffffffffffff    	// #-1
   3060:	17ffffd3 	b	2fac <__d_lookup+0xb4>
	return parent->d_op->d_compare(dentry,
   3064:	f9403340 	ldr	x0, [x26, #96]
   3068:	aa1c03e3 	mov	x3, x28
   306c:	f9401682 	ldr	x2, [x20, #40]
   3070:	b9006fe5 	str	w5, [sp, #108]
   3074:	f9400c04 	ldr	x4, [x0, #24]
   3078:	aa1403e0 	mov	x0, x20
   307c:	d63f0080 	blr	x4
				       name) == 0;
   3080:	7100001f 	cmp	w0, #0x0
   3084:	1a9f17e0 	cset	w0, eq  // eq = none
   3088:	92800006 	mov	x6, #0xffffffffffffffff    	// #-1
   308c:	b9406fe5 	ldr	w5, [sp, #108]
		if (!d_same_name(dentry, parent, name))
   3090:	34fff940 	cbz	w0, 2fb8 <__d_lookup+0xc0>
		dentry->d_lockref.count++;
   3094:	b9405e80 	ldr	w0, [x20, #92]
	smp_store_release(&lock->locked, 0);
   3098:	52800001 	mov	w1, #0x0                   	// #0
   309c:	11000400 	add	w0, w0, #0x1
   30a0:	b9005e80 	str	w0, [x20, #92]
   30a4:	089fff61 	stlrb	w1, [x27]
	preempt_enable();
   30a8:	a9425bf5 	ldp	x21, x22, [sp, #32]
   30ac:	a94573fb 	ldp	x27, x28, [sp, #80]
   30b0:	17ffffc9 	b	2fd4 <__d_lookup+0xdc>
	mask = bytemask_from_count(tcount);
   30b4:	531d7062 	lsl	w2, w3, #3
	return unlikely(!!((a ^ b) & mask));
   30b8:	ca010000 	eor	x0, x0, x1
	mask = bytemask_from_count(tcount);

30bc: 9ac220c2 lsl x2, x6, x2
30c0: ea22001f bics xzr, x0, x2
30c4: 1a9f17e0 cset w0, eq // eq = none
if (!d_same_name(dentry, parent, name))
30c8: 34fff780 cbz w0, 2fb8 <__d_lookup+0xc0>
30cc: 17fffff2 b 3094 <__d_lookup+0x19c>

00000000000030d0 <d_lookup>:
{
30d0: a9bd7bfd stp x29, x30, [sp, #-48]!
30d4: 910003fd mov x29, sp
30d8: a90153f3 stp x19, x20, [sp, #16]
30dc: 90000014 adrp x20, 0 <d_shrink_del>
30e0: 91000294 add x20, x20, #0x0
30e4: a9025bf5 stp x21, x22, [sp, #32]
30e8: aa0003f6 mov x22, x0
30ec: aa0103f5 mov x21, x1
30f0: 1400000a b 3118 <d_lookup+0x48>

	smp_rmb();
   30f4:	d50339bf 	dmb	ishld
		dentry = __d_lookup(parent, name);
   30f8:	aa1503e1 	mov	x1, x21
   30fc:	aa1603e0 	mov	x0, x22
   3100:	94000000 	bl	2ef8 <__d_lookup>
		if (dentry)
   3104:	b5000120 	cbnz	x0, 3128 <d_lookup+0x58>
	smp_rmb();
   3108:	d50339bf 	dmb	ishld
	} while (read_seqretry(&rename_lock, seq));
   310c:	b9400281 	ldr	w1, [x20]
   3110:	6b13003f 	cmp	w1, w19
   3114:	540000a0 	b.eq	3128 <d_lookup+0x58>  // b.none
	__READ_ONCE_SIZE;
   3118:	b9400293 	ldr	w19, [x20]
	if (unlikely(ret & 1)) {

311c: 3607fed3 tbz w19, #0, 30f4 <d_lookup+0x24>
3120: d503203f yield
3124: 17fffffd b 3118 <d_lookup+0x48>
}
3128: a94153f3 ldp x19, x20, [sp, #16]
312c: a9425bf5 ldp x21, x22, [sp, #32]
3130: a8c37bfd ldp x29, x30, [sp], #48
3134: d65f03c0 ret

0000000000003138 <d_hash_and_lookup>:
{
3138: a9be7bfd stp x29, x30, [sp, #-32]!
313c: 910003fd mov x29, sp
3140: a90153f3 stp x19, x20, [sp, #16]
3144: aa0103f3 mov x19, x1
3148: aa0003f4 mov x20, x0

	name->hash = full_name_hash(dir, name->name, name->len);
   314c:	b9400422 	ldr	w2, [x1, #4]
   3150:	f9400421 	ldr	x1, [x1, #8]
   3154:	94000000 	bl	0 <full_name_hash>
   3158:	b9000260 	str	w0, [x19]
	if (dir->d_flags & DCACHE_OP_HASH) {
   315c:	b9400280 	ldr	w0, [x20]
   3160:	360000e0 	tbz	w0, #0, 317c <d_hash_and_lookup+0x44>
		int err = dir->d_op->d_hash(dir, name);
   3164:	f9403282 	ldr	x2, [x20, #96]
   3168:	aa1303e1 	mov	x1, x19
   316c:	aa1403e0 	mov	x0, x20
   3170:	f9400842 	ldr	x2, [x2, #16]
   3174:	d63f0040 	blr	x2
		if (unlikely(err < 0))
   3178:	37f800e0 	tbnz	w0, #31, 3194 <d_hash_and_lookup+0x5c>
	return d_lookup(dir, name);

317c: aa1303e1 mov x1, x19
3180: aa1403e0 mov x0, x20
3184: 94000000 bl 30d0 <d_lookup>
}
3188: a94153f3 ldp x19, x20, [sp, #16]
318c: a8c27bfd ldp x29, x30, [sp], #32
3190: d65f03c0 ret

			return ERR_PTR(err);

3194: 93407c00 sxtw x0, w0
3198: 17fffffc b 3188 <d_hash_and_lookup+0x50>
319c: d503201f nop

Disassembly of section .text.unlikely:

0000000000000000 <dump>:
{
0: a9bc7bfd stp x29, x30, [sp, #-64]!
4: 910003fd mov x29, sp
8: a90153f3 stp x19, x20, [sp, #16]
c: a9025bf5 stp x21, x22, [sp, #32]
10: f9001bf7 str x23, [sp, #48]

	if (!dentry) {
 14:	b50000a0 	cbnz	x0, 28 <dump+0x28>
		printk(KERN_ERR "list fucked in head");
 18:	90000000 	adrp	x0, 0 <dump>
 1c:	91000000 	add	x0, x0, #0x0
 20:	94000000 	bl	0 <printk>
		return;

24: 14000016 b 7c <dump+0x7c>
printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = %x, count =
%d",
28: aa0003f3 mov x19, x0

		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',

2c: 90000015 adrp x21, 0 <dump>
printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = %x, count =
%d",
30: 90000000 adrp x0, 0 <dump>
34: aa1303e1 mov x1, x19
38: 91000000 add x0, x0, #0x0
3c: d2800014 mov x20, #0x0 // #0
40: b9400263 ldr w3, [x19]

		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',

44: 52800157 mov w23, #0xa // #10
printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = %x, count =
%d",
48: b9405e64 ldr w4, [x19, #92]

		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',

4c: 52800416 mov w22, #0x20 // #32
printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = %x, count =
%d",
50: f9400662 ldr x2, [x19, #8]

		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',

54: 910002b5 add x21, x21, #0x0
printk(KERN_ERR "fucked dentry[%p]: d_hash.next = %p, flags = %x, count =
%d",
58: 94000000 bl 0 <printk>

		printk(KERN_CONT "%c%02x", i & 31 ? ' ' : '\n',
 5c:	38746a62 	ldrb	w2, [x19, x20]
 60:	f240129f 	tst	x20, #0x1f
 64:	1a9602e1 	csel	w1, w23, w22, eq  // eq = none
 68:	91000694 	add	x20, x20, #0x1
 6c:	aa1503e0 	mov	x0, x21
 70:	94000000 	bl	0 <printk>
	for (i = 0; i < sizeof(struct dentry); i++)

74: f103029f cmp x20, #0xc0
78: 54ffff21 b.ne 5c <dump+0x5c> // b.any
}
7c: a94153f3 ldp x19, x20, [sp, #16]
80: a9425bf5 ldp x21, x22, [sp, #32]
84: f9401bf7 ldr x23, [sp, #48]
88: a8c47bfd ldp x29, x30, [sp], #64
8c: d65f03c0 ret

maintainer
#9 lore raw

[arm64 maintainers Cc'd; I'm not adding a Cc to moderated list, sorry]

Quoted 21 lines · Vicente Bergas

Hi Al,
i have been running the distro-provided kernel the last few weeks and had no issues at all.
https://archlinuxarm.org/packages/aarch64/linux-aarch64
It is from the v5.1 branch and is compiled with gcc 8.3.

IIRC, i also tested
https://archlinuxarm.org/packages/aarch64/linux-aarch64-rc
v5.2-rc1 and v5.2-rc2 (which at that time where compiled with gcc 8.2) with no issues.

This week tested v5.2-rc4 and v5.2-rc5 from archlinuxarm but there are regressions unrelated to d_lookup.

At this point i was convinced it was a gcc 9.1 issue and had nothing to do with the kernel, but anyways i gave your patch a try. The tested kernel is v5.2-rc5-224-gbed3c0d84e7e and
it has been compiled with gcc 8.3.
The sentinel you put there has triggered!
So, it is not a gcc 9.1 issue.

Quoted 2 lines

In any case, i have no idea if those addresses are arm64-specific in any way.

Cute... So *all* of those are in dentry_hashtable itself. IOW, we have these two values (1<<24 and (1<<24)|(0x88L<<40)) cropping up in dentry_hashtable[...].first on that config.

That, at least, removes the possibility of corrupted forward pointer in the middle of a chain, with several pointers traversed before we run into something unmapped - the crap is in the very beginning.

I don't get it. The only things modifying these pointers should be:

static void ___d_drop(struct dentry *dentry)

{
        struct hlist_bl_head *b;
        /*
         * Hashed dentries are normally on the dentry hashtable,
         * with the exception of those newly allocated by
         * d_obtain_root, which are always IS_ROOT:
         */
        if (unlikely(IS_ROOT(dentry)))
                b = &dentry->d_sb->s_roots;
        else   
                b = d_hash(dentry->d_name.hash);

        hlist_bl_lock(b);
        __hlist_bl_del(&dentry->d_hash);
        hlist_bl_unlock(b);
}

and

static void __d_rehash(struct dentry *entry)

{
        struct hlist_bl_head *b = d_hash(entry->d_name.hash);

        hlist_bl_lock(b);
        hlist_bl_add_head_rcu(&entry->d_hash, b);
        hlist_bl_unlock(b);
}

The latter sets that pointer to (unsigned long)&entry->d_hash | LIST_BL_LOCKMASK), having dereferenced entry->d_hash prior to that. It can't be the source of those values, or we would've oopsed right there.

The former... __hlist_bl_del() does

        /* pprev may be `first`, so be careful not to lose the lock bit */
        WRITE_ONCE(*pprev,
                   (struct hlist_bl_node *)
                        ((unsigned long)next |
                         ((unsigned long)*pprev & LIST_BL_LOCKMASK)));
        if (next)
                next->pprev = pprev;

so to end up with that garbage in the list head we'd have to had next the same bogus pointer (modulo bit 0, possibly). And since it's non-NULL, we would've immediately oopsed on trying to set next->pprev.

There shouldn't be any pointers to hashtable elements other than ->d_hash.pprev of various dentries. And ->d_hash is not a part of anon unions in struct dentry, so it can't be mistaken access through the aliasing member.

Of course, there's always a possibility of something stomping on random places in memory and shitting those values all over, with the hashtable being the hottest place on the loads where it happens... Hell knows...

What's your config, BTW? SMP and DEBUG_SPINLOCK, specifically...

#10 lore raw
Quoted 88 lines · Al Viro

[arm64 maintainers Cc'd; I'm not adding a Cc to moderated list, sorry]

Quoted 21 lines · Vicente Bergas

Hi Al,
i have been running the distro-provided kernel the last few weeks and had no issues at all.
https://archlinuxarm.org/packages/aarch64/linux-aarch64
It is from the v5.1 branch and is compiled with gcc 8.3.

IIRC, i also tested
https://archlinuxarm.org/packages/aarch64/linux-aarch64-rc
v5.2-rc1 and v5.2-rc2 (which at that time where compiled with gcc 8.2) with no issues.

This week tested v5.2-rc4 and v5.2-rc5 from archlinuxarm but there are regressions unrelated to d_lookup.

At this point i was convinced it was a gcc 9.1 issue and had nothing to do with the kernel, but anyways i gave your patch a try. The tested kernel is v5.2-rc5-224-gbed3c0d84e7e and
it has been compiled with gcc 8.3.
The sentinel you put there has triggered!
So, it is not a gcc 9.1 issue.

Quoted 2 lines

In any case, i have no idea if those addresses are arm64-specific in any way.

Cute... So *all* of those are in dentry_hashtable itself. IOW, we have these two values (1<<24 and (1<<24)|(0x88L<<40)) cropping up in dentry_hashtable[...].first on that config.

That, at least, removes the possibility of corrupted forward pointer in the middle of a chain, with several pointers traversed before we run into something unmapped - the crap is in the very beginning.

I don't get it. The only things modifying these pointers should be:

static void ___d_drop(struct dentry *dentry)

{
        struct hlist_bl_head *b;
        /*
         * Hashed dentries are normally on the dentry hashtable,
         * with the exception of those newly allocated by
         * d_obtain_root, which are always IS_ROOT:
         */
        if (unlikely(IS_ROOT(dentry)))
                b = &dentry->d_sb->s_roots;
        else   
                b = d_hash(dentry->d_name.hash);

        hlist_bl_lock(b);
        __hlist_bl_del(&dentry->d_hash);
        hlist_bl_unlock(b);
}

and

static void __d_rehash(struct dentry *entry)

{
        struct hlist_bl_head *b = d_hash(entry->d_name.hash);

        hlist_bl_lock(b);
        hlist_bl_add_head_rcu(&entry->d_hash, b);
        hlist_bl_unlock(b);
}

The latter sets that pointer to (unsigned long)&entry->d_hash | LIST_BL_LOCKMASK), having dereferenced entry->d_hash prior to that. It can't be the source of those values, or we would've oopsed right there.

The former... __hlist_bl_del() does

        /* pprev may be `first`, so be careful not to lose the lock bit */
        WRITE_ONCE(*pprev,
                   (struct hlist_bl_node *)
                        ((unsigned long)next |
                         ((unsigned long)*pprev & LIST_BL_LOCKMASK)));
        if (next)
                next->pprev = pprev;

so to end up with that garbage in the list head we'd have to had next the same bogus pointer (modulo bit 0, possibly). And since it's non-NULL, we would've immediately oopsed on trying to set next->pprev.

There shouldn't be any pointers to hashtable elements other than ->d_hash.pprev of various dentries. And ->d_hash is not a part of anon unions in struct dentry, so it can't be mistaken access through the aliasing member.

Quoted 5 lines

Of course, there's always a possibility of something stomping on random places in memory and shitting those values all over, with the hashtable being the hottest place on the loads where it happens... Hell knows...

What's your config, BTW? SMP and DEBUG_SPINLOCK, specifically...

Hi Al,
here it is:
https://paste.debian.net/1088517

Regards,
Vicenç.

#11 lore raw
Quoted 2 lines · Vicente Bergas

here it is:
https://paste.debian.net/1088517

No modules and OPTIMIZE_INLINING=n, so this isn't either of my first thoughts. Hmm. I guess I should try to reproduce the issue locally.

Will

maintainer
#12 lore raw
In reply to Vicente Bergas
Quoted 5 lines · Vicente Bergas
Quoted 1 line

What's your config, BTW? SMP and DEBUG_SPINLOCK, specifically...

Hi Al,
here it is:
https://paste.debian.net/1088517

Aha... So LIST_BL_LOCKMASK is 1 there (same as on distro builds)...

Hell knows - how about
static inline void hlist_bl_lock(struct hlist_bl_head *b)

{
	BUG_ON(((u32)READ_ONCE(*b)&~LIST_BL_LOCKMASK) == 0x01000000);
        bit_spin_lock(0, (unsigned long *)b);
}

and

static inline void hlist_bl_unlock(struct hlist_bl_head *b)

{
        __bit_spin_unlock(0, (unsigned long *)b);
	BUG_ON(((u32)READ_ONCE(*b)&~LIST_BL_LOCKMASK) == 0x01000000);
}

to see if we can narrow down where that happens?

#13 lore raw

Hi Al,
i think have a hint of what is going on.
With the last kernel built with your sentinels at hlist_bl_*lock it is very easy to reproduce the issue.
In fact it is so unstable that i had to connect a serial port in order to save the kernel trace.
Unfortunately all the traces are at different addresses and your sentinel did not trigger.

Now i am writing this email from that same buggy kernel, which is v5.2-rc5-224-gbed3c0d84e7e.

The difference is that I changed the bootloader.
Before was booting 5.1.12 and kexec into this one.
Now booting from u-boot into this one.
I will continue booting with u-boot for some time to be sure it is stable and confirm this is the cause.

In case it is, who is the most probable offender?
the kernel before kexec or the kernel after?

The original report was sent to you because you appeared as the maintainer of fs/dcache.c, which appeared on the trace. Should this be redirected somewhere else now?

Regards,
Vicenç.

Quoted 26 lines · Al Viro
Quoted 5 lines · Vicente Bergas
Quoted 1 line

What's your config, BTW? SMP and DEBUG_SPINLOCK, specifically...

Hi Al,
here it is:
https://paste.debian.net/1088517

Aha... So LIST_BL_LOCKMASK is 1 there (same as on distro builds)...

Hell knows - how about
static inline void hlist_bl_lock(struct hlist_bl_head *b)

{
	BUG_ON(((u32)READ_ONCE(*b)&~LIST_BL_LOCKMASK) == 0x01000000);
        bit_spin_lock(0, (unsigned long *)b);
}

and

static inline void hlist_bl_unlock(struct hlist_bl_head *b)

{
        __bit_spin_unlock(0, (unsigned long *)b);
	BUG_ON(((u32)READ_ONCE(*b)&~LIST_BL_LOCKMASK) == 0x01000000);
}

to see if we can narrow down where that happens?

#14 lore raw
Quoted 12 lines · Vicente Bergas

Hi Al,
i think have a hint of what is going on.
With the last kernel built with your sentinels at hlist_bl_*lock it is very easy to reproduce the issue.
In fact it is so unstable that i had to connect a serial port in order to save the kernel trace.
Unfortunately all the traces are at different addresses and your sentinel did not trigger.

Now i am writing this email from that same buggy kernel, which is v5.2-rc5-224-gbed3c0d84e7e.

Quoted 8 lines

The difference is that I changed the bootloader.
Before was booting 5.1.12 and kexec into this one.
Now booting from u-boot into this one.
I will continue booting with u-boot for some time to be sure it is stable and confirm this is the cause.

In case it is, who is the most probable offender?
the kernel before kexec or the kernel after?

Has kexec ever worked reliably on this board? If you used to kexec successfully, then we can try to hunt down the regression using memtest. If you kexec into a problematic kernel with CONFIG_MEMTEST=y and pass "memtest=17" on the command-line, it will hopefully reveal any active memory corruption.

My first thought is that there is ongoing DMA which corrupts the dentry hash. The rk3399 SoC also has an IOMMU, which could contribute to the fun if it's not shutdown correctly (i.e. if it enters bypass mode).

Quoted 3 lines

The original report was sent to you because you appeared as the maintainer of fs/dcache.c, which appeared on the trace. Should this be redirected somewhere else now?

[email protected]

Probably worth adding Heiko Stuebner <[email protected]> to cc.

Will

#15 lore raw

[+Marc]

Hi again, Vicente,

Quoted 32 lines · Will Deacon
Quoted 12 lines · Vicente Bergas

Hi Al,
i think have a hint of what is going on.
With the last kernel built with your sentinels at hlist_bl_*lock it is very easy to reproduce the issue.
In fact it is so unstable that i had to connect a serial port in order to save the kernel trace.
Unfortunately all the traces are at different addresses and your sentinel did not trigger.

Now i am writing this email from that same buggy kernel, which is v5.2-rc5-224-gbed3c0d84e7e.

Quoted 8 lines

The difference is that I changed the bootloader.
Before was booting 5.1.12 and kexec into this one.
Now booting from u-boot into this one.
I will continue booting with u-boot for some time to be sure it is stable and confirm this is the cause.

In case it is, who is the most probable offender?
the kernel before kexec or the kernel after?

Has kexec ever worked reliably on this board? If you used to kexec successfully, then we can try to hunt down the regression using memtest. If you kexec into a problematic kernel with CONFIG_MEMTEST=y and pass "memtest=17" on the command-line, it will hopefully reveal any active memory corruption.

My first thought is that there is ongoing DMA which corrupts the dentry hash. The rk3399 SoC also has an IOMMU, which could contribute to the fun if it's not shutdown correctly (i.e. if it enters bypass mode).

Quoted 7 lines
Quoted 3 lines

The original report was sent to you because you appeared as the maintainer of fs/dcache.c, which appeared on the trace. Should this be redirected somewhere else now?

[email protected]

Probably worth adding Heiko Stuebner <[email protected]> to cc.

Before you rush over to LAKML, please could you provide your full dmesg output from the kernel that was crashing (i.e. the dmesg you see in the kexec'd kernel)? We've got a theory that the issue may be related to the interrupt controller, and the dmesg output should help to establish whether that is plausible or not.

Thanks,

Will

#16 lore raw
Quoted 13 lines · Will Deacon

[+Marc]

Hi again, Vicente,

Quoted 1 line · Will Deacon

On Sat, Jun 22, 2019 at 08:02:19PM +0200, Vicente Bergas wrote: ...

Before you rush over to LAKML, please could you provide your full dmesg output from the kernel that was crashing (i.e. the dmesg you see in the kexec'd kernel)? We've got a theory that the issue may be related to the interrupt controller, and the dmesg output should help to establish whether that is plausible or not.

Quoted 3 lines

Thanks,

Will

Hi Will,
the memtest test is still pending...

Regarding interrupts, the kernel before kexec has this parameter:
irqchip.gicv3_nolpi=1
Thanks to Marc:
https://freenode.irclog.whitequark.org/linux-rockchip/2018-11-20#23524255

The kernel messages are here:
https://paste.debian.net/1089170/
https://paste.debian.net/1089171/

Regards,
Vicenç.

#17 lore raw
Quoted 9 lines · Vicente Bergas
Quoted 5 lines · Will Deacon

[+Marc]

Hi again, Vicente,

On Mon, Jun 24, 2019 at 12:47:41PM +0100, Will Deacon wrote: ...

Hi Will,
the memtest test is still pending...

Hi Will,
i've just ran that memtest and no issues have been found. See below. I've also noticed that this is running very early, hence the 'earlycon'. Because of this i wondered if it was run with interrupts disabled, and indeed this is the case, am i wrong?
If the kernel before kexec is corrupting memory for the currently running kernel, which entry point does it have, besides interrupts? Or can the corruption come through DMA from a peripheral?

Quoted 1 line

Has kexec ever worked reliably on this board?

Yes, more or less. I have experienced some issues that could be caused because of this, like sometimes on-screen flickering or failing to boot once every 20 or 30 tries. But recently this d_lookup issue appeared and it is a show-stopper.

This way of booting is used on both the sapphire board and on kevin. The bootloader is https://gitlab.com/vicencb/kevinboot (not up to date) and a similar one for the sapphire board.

Regards,
Vicenç.

Booting Linux on physical CPU 0x0000000000 [0x410fd034]
Linux version 5.2.0-rc6 (local@host) (gcc version 8.3.0 (GCC)) #1 SMP @0
Machine model: Sapphire-RK3399 Board
earlycon: uart0 at MMIO32 0x00000000ff1a0000 (options '1500000n8')
printk: bootconsole [uart0] enabled
early_memtest: # of tests: 17
0x0000000000200000 - 0x0000000000280000 pattern 4c494e5558726c7a
0x0000000000b62000 - 0x0000000000b65000 pattern 4c494e5558726c7a
0x0000000000b7665f - 0x00000000f77e8a58 pattern 4c494e5558726c7a
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 4c494e5558726c7a
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 4c494e5558726c7a
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 4c494e5558726c7a
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 4c494e5558726c7a
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 4c494e5558726c7a
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 4c494e5558726c7a
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 4c494e5558726c7a
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 4c494e5558726c7a
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 4c494e5558726c7a
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 4c494e5558726c7a
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 4c494e5558726c7a
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 4c494e5558726c7a
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 4c494e5558726c7a
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 4c494e5558726c7a
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 4c494e5558726c7a
0x0000000000200000 - 0x0000000000280000 pattern eeeeeeeeeeeeeeee
0x0000000000b62000 - 0x0000000000b65000 pattern eeeeeeeeeeeeeeee
0x0000000000b7665f - 0x00000000f77e8a58 pattern eeeeeeeeeeeeeeee
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern eeeeeeeeeeeeeeee
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern eeeeeeeeeeeeeeee
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern eeeeeeeeeeeeeeee
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern eeeeeeeeeeeeeeee
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern eeeeeeeeeeeeeeee
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern eeeeeeeeeeeeeeee
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern eeeeeeeeeeeeeeee
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern eeeeeeeeeeeeeeee
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern eeeeeeeeeeeeeeee
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern eeeeeeeeeeeeeeee
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern eeeeeeeeeeeeeeee
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern eeeeeeeeeeeeeeee
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern eeeeeeeeeeeeeeee
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern eeeeeeeeeeeeeeee
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern eeeeeeeeeeeeeeee
0x0000000000200000 - 0x0000000000280000 pattern dddddddddddddddd
0x0000000000b62000 - 0x0000000000b65000 pattern dddddddddddddddd
0x0000000000b7665f - 0x00000000f77e8a58 pattern dddddddddddddddd
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern dddddddddddddddd
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern dddddddddddddddd
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern dddddddddddddddd
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern dddddddddddddddd
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern dddddddddddddddd
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern dddddddddddddddd
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern dddddddddddddddd
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern dddddddddddddddd
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern dddddddddddddddd
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern dddddddddddddddd
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern dddddddddddddddd
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern dddddddddddddddd
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern dddddddddddddddd
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern dddddddddddddddd
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern dddddddddddddddd
0x0000000000200000 - 0x0000000000280000 pattern bbbbbbbbbbbbbbbb
0x0000000000b62000 - 0x0000000000b65000 pattern bbbbbbbbbbbbbbbb
0x0000000000b7665f - 0x00000000f77e8a58 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern bbbbbbbbbbbbbbbb
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern bbbbbbbbbbbbbbbb
0x0000000000200000 - 0x0000000000280000 pattern 7777777777777777
0x0000000000b62000 - 0x0000000000b65000 pattern 7777777777777777
0x0000000000b7665f - 0x00000000f77e8a58 pattern 7777777777777777
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 7777777777777777
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 7777777777777777
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 7777777777777777
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 7777777777777777
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 7777777777777777
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 7777777777777777
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 7777777777777777
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 7777777777777777
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 7777777777777777
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 7777777777777777
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 7777777777777777
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 7777777777777777
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 7777777777777777
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 7777777777777777
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 7777777777777777
0x0000000000200000 - 0x0000000000280000 pattern cccccccccccccccc
0x0000000000b62000 - 0x0000000000b65000 pattern cccccccccccccccc
0x0000000000b7665f - 0x00000000f77e8a58 pattern cccccccccccccccc
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern cccccccccccccccc
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern cccccccccccccccc
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern cccccccccccccccc
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern cccccccccccccccc
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern cccccccccccccccc
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern cccccccccccccccc
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern cccccccccccccccc
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern cccccccccccccccc
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern cccccccccccccccc
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern cccccccccccccccc
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern cccccccccccccccc
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern cccccccccccccccc
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern cccccccccccccccc
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern cccccccccccccccc
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern cccccccccccccccc
0x0000000000200000 - 0x0000000000280000 pattern 9999999999999999
0x0000000000b62000 - 0x0000000000b65000 pattern 9999999999999999
0x0000000000b7665f - 0x00000000f77e8a58 pattern 9999999999999999
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 9999999999999999
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 9999999999999999
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 9999999999999999
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 9999999999999999
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 9999999999999999
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 9999999999999999
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 9999999999999999
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 9999999999999999
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 9999999999999999
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 9999999999999999
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 9999999999999999
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 9999999999999999
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 9999999999999999
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 9999999999999999
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 9999999999999999
0x0000000000200000 - 0x0000000000280000 pattern 6666666666666666
0x0000000000b62000 - 0x0000000000b65000 pattern 6666666666666666
0x0000000000b7665f - 0x00000000f77e8a58 pattern 6666666666666666
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 6666666666666666
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 6666666666666666
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 6666666666666666
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 6666666666666666
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 6666666666666666
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 6666666666666666
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 6666666666666666
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 6666666666666666
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 6666666666666666
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 6666666666666666
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 6666666666666666
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 6666666666666666
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 6666666666666666
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 6666666666666666
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 6666666666666666
0x0000000000200000 - 0x0000000000280000 pattern 3333333333333333
0x0000000000b62000 - 0x0000000000b65000 pattern 3333333333333333
0x0000000000b7665f - 0x00000000f77e8a58 pattern 3333333333333333
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 3333333333333333
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 3333333333333333
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 3333333333333333
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 3333333333333333
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 3333333333333333
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 3333333333333333
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 3333333333333333
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 3333333333333333
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 3333333333333333
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 3333333333333333
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 3333333333333333
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 3333333333333333
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 3333333333333333
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 3333333333333333
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 3333333333333333
0x0000000000200000 - 0x0000000000280000 pattern 8888888888888888
0x0000000000b62000 - 0x0000000000b65000 pattern 8888888888888888
0x0000000000b7665f - 0x00000000f77e8a58 pattern 8888888888888888
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 8888888888888888
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 8888888888888888
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 8888888888888888
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 8888888888888888
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 8888888888888888
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 8888888888888888
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 8888888888888888
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 8888888888888888
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 8888888888888888
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 8888888888888888
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 8888888888888888
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 8888888888888888
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 8888888888888888
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 8888888888888888
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 8888888888888888
0x0000000000200000 - 0x0000000000280000 pattern 4444444444444444
0x0000000000b62000 - 0x0000000000b65000 pattern 4444444444444444
0x0000000000b7665f - 0x00000000f77e8a58 pattern 4444444444444444
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 4444444444444444
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 4444444444444444
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 4444444444444444
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 4444444444444444
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 4444444444444444
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 4444444444444444
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 4444444444444444
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 4444444444444444
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 4444444444444444
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 4444444444444444
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 4444444444444444
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 4444444444444444
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 4444444444444444
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 4444444444444444
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 4444444444444444
0x0000000000200000 - 0x0000000000280000 pattern 2222222222222222
0x0000000000b62000 - 0x0000000000b65000 pattern 2222222222222222
0x0000000000b7665f - 0x00000000f77e8a58 pattern 2222222222222222
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 2222222222222222
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 2222222222222222
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 2222222222222222
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 2222222222222222
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 2222222222222222
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 2222222222222222
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 2222222222222222
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 2222222222222222
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 2222222222222222
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 2222222222222222
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 2222222222222222
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 2222222222222222
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 2222222222222222
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 2222222222222222
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 2222222222222222
0x0000000000200000 - 0x0000000000280000 pattern 1111111111111111
0x0000000000b62000 - 0x0000000000b65000 pattern 1111111111111111
0x0000000000b7665f - 0x00000000f77e8a58 pattern 1111111111111111
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 1111111111111111
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 1111111111111111
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 1111111111111111
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 1111111111111111
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 1111111111111111
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 1111111111111111
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 1111111111111111
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 1111111111111111
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 1111111111111111
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 1111111111111111
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 1111111111111111
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 1111111111111111
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 1111111111111111
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 1111111111111111
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 1111111111111111
0x0000000000200000 - 0x0000000000280000 pattern aaaaaaaaaaaaaaaa
0x0000000000b62000 - 0x0000000000b65000 pattern aaaaaaaaaaaaaaaa
0x0000000000b7665f - 0x00000000f77e8a58 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern aaaaaaaaaaaaaaaa
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern aaaaaaaaaaaaaaaa
0x0000000000200000 - 0x0000000000280000 pattern 5555555555555555
0x0000000000b62000 - 0x0000000000b65000 pattern 5555555555555555
0x0000000000b7665f - 0x00000000f77e8a58 pattern 5555555555555555
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 5555555555555555
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 5555555555555555
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 5555555555555555
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 5555555555555555
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 5555555555555555
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 5555555555555555
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 5555555555555555
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 5555555555555555
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 5555555555555555
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 5555555555555555
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 5555555555555555
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 5555555555555555
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 5555555555555555
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 5555555555555555
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 5555555555555555
0x0000000000200000 - 0x0000000000280000 pattern ffffffffffffffff
0x0000000000b62000 - 0x0000000000b65000 pattern ffffffffffffffff
0x0000000000b7665f - 0x00000000f77e8a58 pattern ffffffffffffffff
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern ffffffffffffffff
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern ffffffffffffffff
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern ffffffffffffffff
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern ffffffffffffffff
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern ffffffffffffffff
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern ffffffffffffffff
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern ffffffffffffffff
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern ffffffffffffffff
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern ffffffffffffffff
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern ffffffffffffffff
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern ffffffffffffffff
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern ffffffffffffffff
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern ffffffffffffffff
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern ffffffffffffffff
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern ffffffffffffffff
0x0000000000200000 - 0x0000000000280000 pattern 0000000000000000
0x0000000000b62000 - 0x0000000000b65000 pattern 0000000000000000
0x0000000000b7665f - 0x00000000f77e8a58 pattern 0000000000000000
0x00000000f77e8a87 - 0x00000000f77e8a88 pattern 0000000000000000
0x00000000f77e8ab7 - 0x00000000f77e8ab8 pattern 0000000000000000
0x00000000f77e8ae7 - 0x00000000f77e8ae8 pattern 0000000000000000
0x00000000f77e8b17 - 0x00000000f77e8b18 pattern 0000000000000000
0x00000000f77e8b47 - 0x00000000f77e8b48 pattern 0000000000000000
0x00000000f77e8b74 - 0x00000000f77e8b78 pattern 0000000000000000
0x00000000f77e8ba4 - 0x00000000f77e8ba8 pattern 0000000000000000
0x00000000f77e8bd4 - 0x00000000f77e8bd8 pattern 0000000000000000
0x00000000f77e8c04 - 0x00000000f77e8c08 pattern 0000000000000000
0x00000000f77e8c34 - 0x00000000f77e8c38 pattern 0000000000000000
0x00000000f77e8c64 - 0x00000000f77e8c68 pattern 0000000000000000
0x00000000f77e8c94 - 0x00000000f77e8c98 pattern 0000000000000000
0x00000000f77e8cc4 - 0x00000000f77e8cc8 pattern 0000000000000000
0x00000000f77e8cf4 - 0x00000000f77e8cf8 pattern 0000000000000000
0x00000000f77e8d29 - 0x00000000f77e8d30 pattern 0000000000000000
On node 0 totalpages: 1015296
DMA32 zone: 15864 pages used for memmap
DMA32 zone: 0 pages reserved
DMA32 zone: 1015296 pages, LIFO batch:63
psci: probing for conduit method from DT.
psci: PSCIv1.1 detected in firmware.
psci: Using standard PSCI v0.2 function IDs
psci: MIGRATE_INFO_TYPE not supported.
psci: SMC Calling Convention v1.1
percpu: Embedded 20 pages/cpu s50200 r0 d31720 u81920
pcpu-alloc: s50200 r0 d31720 u81920 alloc=20*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 Detected VIPT I-cache on CPU0
CPU features: detected: GIC system register CPU interface
Speculative Store Bypass Disable mitigation not required
Built 1 zonelists, mobility grouping on. Total pages: 999432
Kernel command line: rw root=/dev/sda2 rootwait console=tty0 console=ttyS2,1500000n8 earlycon memtest=17
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Memory: 3971436K/4061184K available (6078K kernel code, 430K rwdata, 1460K rodata, 640K init, 407K bss, 89748K reserved, 0K cma-reserved)
random: get_random_u64 called from cache_random_seq_create+0x50/0x118 with crng_init=0
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
rcu: Hierarchical RCU implementation.
rcu: RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=6.
rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=6
NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GICv3: GIC: Using split EOI/Deactivate mode
GICv3: Distributor has no Range Selector support
GICv3: no VLPI support, no direct LPI support
GICv3: CPU0: found redistributor 0 region 0:0x00000000fef00000
ITS [mem 0xfee20000-0xfee3ffff]
ITS@0x00000000fee20000: allocated 65536 Devices @f2400000 (flat, esz 8, psz 64K, shr 0)
ITS: using cache flushing for cmd queue
GICv3: using LPI property table @0x0000000000240000
GIC: using cache flushing for LPI property table
GICv3: CPU0: using allocated LPI pending table @0x0000000000250000
GICv3: GIC: PPI partition interrupt-partition-0[0] { /cpus/cpu@0[0] /cpus/cpu@1[1] /cpus/cpu@2[2] /cpus/cpu@3[3] }
GICv3: GIC: PPI partition interrupt-partition-1[1] { /cpus/cpu@100[4] /cpus/cpu@101[5] }
rockchip_mmc_get_phase: invalid clk rate
rockchip_mmc_get_phase: invalid clk rate
rockchip_mmc_get_phase: invalid clk rate
rockchip_mmc_get_phase: invalid clk rate
arch_timer: cp15 timer(s) running at 24.00MHz (phys).
clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
Console: colour dummy device 240x67
printk: console [tty0] enabled
Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=96000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
*** VALIDATE proc ***
*** VALIDATE cgroup1 ***
*** VALIDATE cgroup2 ***
ASID allocator initialised with 32768 entries
rcu: Hierarchical SRCU implementation.
Platform MSI: interrupt-controller@fee20000 domain created
smp: Bringing up secondary CPUs ...
Detected VIPT I-cache on CPU1
GICv3: CPU1: found redistributor 1 region 0:0x00000000fef20000
GICv3: CPU1: using allocated LPI pending table @0x0000000000260000
CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
Detected VIPT I-cache on CPU2
GICv3: CPU2: found redistributor 2 region 0:0x00000000fef40000
GICv3: CPU2: using allocated LPI pending table @0x0000000000270000
CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
Detected VIPT I-cache on CPU3
GICv3: CPU3: found redistributor 3 region 0:0x00000000fef60000
GICv3: CPU3: using allocated LPI pending table @0x00000000f2480000
CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
CPU features: detected: EL2 vector hardening
Detected PIPT I-cache on CPU4
GICv3: CPU4: found redistributor 100 region 0:0x00000000fef80000
GICv3: CPU4: using allocated LPI pending table @0x00000000f2490000
CPU4: Booted secondary processor 0x0000000100 [0x410fd082]
Detected PIPT I-cache on CPU5
GICv3: CPU5: found redistributor 101 region 0:0x00000000fefa0000
GICv3: CPU5: using allocated LPI pending table @0x00000000f24a0000
CPU5: Booted secondary processor 0x0000000101 [0x410fd082]
smp: Brought up 1 node, 6 CPUs
SMP: Total of 6 processors activated.
CPU features: detected: 32-bit EL0 Support
CPU features: detected: CRC32 instructions
CPU: All CPU(s) started at EL2
alternatives: patching kernel code
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
futex hash table entries: 2048 (order: 5, 131072 bytes)
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
DMA: preallocated 256 KiB pool for atomic allocations

Quoted 11 lines

Regarding interrupts, the kernel before kexec has this parameter:
irqchip.gicv3_nolpi=1
Thanks to Marc:
https://freenode.irclog.whitequark.org/linux-rockchip/2018-11-20#23524255

The kernel messages are here:
https://paste.debian.net/1089170/
https://paste.debian.net/1089171/

Regards,
Vicenç.

#18 lore raw
In reply to Al Viro

Hi all,

Quoted 2 lines · Al Viro

[arm64 maintainers Cc'd; I'm not adding a Cc to moderated list, sorry]

Thanks for adding us.

Quoted 23 lines · Vicente Bergas
Quoted 22 lines

Hi Al,
i have been running the distro-provided kernel the last few weeks and had no issues at all.
https://archlinuxarm.org/packages/aarch64/linux-aarch64
It is from the v5.1 branch and is compiled with gcc 8.3.

IIRC, i also tested
https://archlinuxarm.org/packages/aarch64/linux-aarch64-rc
v5.2-rc1 and v5.2-rc2 (which at that time where compiled with gcc 8.2) with no issues.

This week tested v5.2-rc4 and v5.2-rc5 from archlinuxarm but there are regressions unrelated to d_lookup.

At this point i was convinced it was a gcc 9.1 issue and had nothing to do with the kernel, but anyways i gave your patch a try. The tested kernel is v5.2-rc5-224-gbed3c0d84e7e and
it has been compiled with gcc 8.3.
The sentinel you put there has triggered!
So, it is not a gcc 9.1 issue.

Quoted 6 lines
Quoted 2 lines

In any case, i have no idea if those addresses are arm64-specific in any way.

Cute... So *all* of those are in dentry_hashtable itself. IOW, we have these two values (1<<24 and (1<<24)|(0x88L<<40)) cropping up in dentry_hashtable[...].first on that config.

Unfortunately, those values don't jump out at me as something particularly meaningful on arm64. Bloody weird though.

Quoted 9 lines

There shouldn't be any pointers to hashtable elements other than ->d_hash.pprev of various dentries. And ->d_hash is not a part of anon unions in struct dentry, so it can't be mistaken access through the aliasing member.

Of course, there's always a possibility of something stomping on random places in memory and shitting those values all over, with the hashtable being the hottest place on the loads where it happens... Hell knows...

What's your config, BTW? SMP and DEBUG_SPINLOCK, specifically...

I'd also be interesting in seeing the .config (the pastebin link earlier in the thread appears to have expired). Two areas where we've had issues recently are (1) module relocations and (2) CONFIG_OPTIMIZE_INLINING. However, this is the first report I've seen of the sort of crash you're reporting.

Will