device_cgroup: remove branch hint after code refactor
PATCH 1/1 device_cgroup: remove branch hint after code refactor
commit 4ef4ac360101 ("device_cgroup: avoid access to ->i_rdev in the common case in devcgroup_inode_permission()") reordered the checks in devcgroup_inode_permission() to check the inode mode before checking i_rdev, for better cache behavior.
However, the likely() annotation on the i_rdev check was not updated to reflect the new code flow. Originally, when i_rdev was checked first, likely(!inode->i_rdev) made sense because most inodes were(?) regular files/directories, thus i_rdev == 0.
After the reorder, by the time we reach the i_rdev check, we have already confirmed the inode IS a block or character device. Block and character special files are precisely defined by having a device number (i_rdev), so !inode->i_rdev is now the rare edge case, not the common case.
Branch profiling confirmed this is 100% mispredicted:
correct incorrect % Function File Line
------- --------- - -------- ---- ----
0 2631904 100 devcgroup_inode_permission device_cgroup.h 24Remove likely() to avoid giving the wrong hint to the CPU.
--- include/linux/device_cgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/device_cgroup.h +1 -1
@@ -21,7 +21,7 @@ static inline int devcgroup_inode_permission(struct inode *inode, int mask)if (likely (!S_ISBLK (inode->i_mode) && !S_ISCHR (inode->i_mode)))return 0 ; -if (likely (!inode->i_rdev)) +if (!inode->i_rdev)return 0 ;if (S_ISBLK (inode->i_mode))
---
base-commit: f0b9d8eb98dfee8d00419aa07543bdc2c1a44fb1
change-id: 20260107-likely_device-20dae82d502d
Best regards,
Signature · 1 line
Breno Leitao <[email protected]>
Quoted 36 lines · Breno Leitao
commit
4ef4ac360101("device_cgroup: avoid access to ->i_rdev in the common case indevcgroup_inode_permission()") reordered the checks indevcgroup_inode_permission()to check the inode mode before checkingi_rdev, for better cache behavior.However, the
likely()annotation on thei_rdevcheck was not updated to reflect the new code flow. Originally, wheni_rdevwas checked first,likely(!inode->i_rdev)made sense because most inodes were(?) regular files/directories, thusi_rdev== 0.After the reorder, by the time we reach the
i_rdevcheck, we have already confirmed the inode IS a block or character device. Block and character special files are precisely defined by having a device number (i_rdev), so !inode->i_rdev is now the rare edge case, not the common case.Branch profiling confirmed this is 100% mispredicted:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 0 2631904 100 devcgroup_inode_permission device_cgroup.h 24Remove
likely()to avoid giving the wrong hint to the CPU.Fixes 4ef4ac360101 ("device_cgroup: avoid access to ->i_rdev in the common case in devcgroup_inode_permission()")Signed-off-by Breno Leitao <[email protected]>--- include/linux/device_cgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/device_cgroup.h +0
-0
include/linux/device_cgroup.h static inline int devcgroup_inode_permission(struct inode *inode, int mask) · line 26
Quoted 7 lines
include/linux/device_cgroup.h +1
-1@@ -21,7 +21,7 @@ static inline int devcgroup_inode_permission(struct inode *inode, int mask)if (likely (!S_ISBLK (inode->i_mode) && !S_ISCHR (inode->i_mode)))return 0 ; -if (likely (!inode->i_rdev)) +if (!inode->i_rdev)return 0 ;
The branch was left there because I could not be bothered to analyze whether it can be straight up eleminated with the new checks in place.
A quick look at init_special_inode suggests it is an invariant rdev is there in this case.
So for the time being I would replace likely with WARN_ON_ONCE . Might be even a good candidate for the pending release.
Hello Mateusz,
Quoted 17 lines · Mateusz Guzik
include/linux/device_cgroup.h static inline int devcgroup_inode_permission(struct inode *inode, int mask) · line 26
Quoted 12 lines · Breno Leitao
include/linux/device_cgroup.h +1
-1@@ -21,7 +21,7 @@ static inline int devcgroup_inode_permission(struct inode *inode, int mask)if (likely (!S_ISBLK (inode->i_mode) && !S_ISCHR (inode->i_mode)))return 0 ; -if (likely (!inode->i_rdev)) +if (!inode->i_rdev)return 0 ;The branch was left there because I could not be bothered to analyze whether it can be straight up eleminated with the new checks in place.
Quoted 5 lines
A quick look at
init_special_inodesuggests it is an invariant rdev is there in this case.So for the time being I would replace likely with
WARN_ON_ONCE. Might be even a good candidate for the pending release.
Oh, in fact that was my first try, but, when I tested it, I found that it warns in some cases.
[ 126.951410] WARNING: ./include/linux/device_cgroup.h:24 at inode_permission+0x181/0x190, CPU#4: networkd)/1212 [ 126.971659] Modules linked in: intel_uncore_frequency(E) intel_uncore_frequency_common(E) skx_edac(E) skx_edac_common(E) nfit(E) libnvdimm(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) iTCO_vendor_support(E) irqbypass(E) acpi_cpufreq(E) i2c_i801(E) i2c_smbus(E) wmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) evdev(E) button(E) bpf_preload(E) sch_fq_codel(E) xhci_pci(E) xhci_hcd(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) loop(E) efivarfs(E) autofs4(E) [ 127.097485] Tainted: [S]=CPU_OUT_OF_SPEC, [E]=UNSIGNED_MODULE [ 127.109038] Hardware name: Quanta Delta Lake MP 29F0EMA01D0/Delta Lake-Class1, BIOS F0E_3A21 06/27/2024 [ 127.127889] RIP: 0010:inode_permission (./include/linux/device_cgroup.h:24 fs/namei.c:600) [ 127.137542] Code: 14 81 e2 ff ff 0f 00 31 ff 3d 00 60 00 00 41 8d 0c 48 40 0f 95 c7 ff c7 e8 5c 68 35 00 85 c0 0f 85 6d ff ff ff e9 29 ff ff ff <0f> 0b e9 22 ff ff ff 0f 1f 84 00 00 00 00 00 41 56 53 49 89 f6 48 All code ======== 0: 14 81 adc $0x81,%al 2: e2 ff loop 0x3 4: ff 0f decl (%rdi) 6: 00 31 add %dh,(%rcx) 8: ff (bad) 9: 3d 00 60 00 00 cmp $0x6000,%eax e: 41 8d 0c 48 lea (%r8,%rcx,2),%ecx 12: 40 0f 95 c7 setne %dil 16: ff c7 inc %edi 18: e8 5c 68 35 00 call 0x356879 1d: 85 c0 test %eax,%eax 1f: 0f 85 6d ff ff ff jne 0xffffffffffffff92 25: e9 29 ff ff ff jmp 0xffffffffffffff53 2a:* 0f 0b ud2 <-- trapping instruction 2c: e9 22 ff ff ff jmp 0xffffffffffffff53 31: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 38: 00 39: 41 56 push %r14 3b: 53 push %rbx 3c: 49 89 f6 mov %rsi,%r14 3f: 48 rex.W
Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: e9 22 ff ff ff jmp 0xffffffffffffff29 7: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) e: 00 f: 41 56 push %r14 11: 53 push %rbx 12: 49 89 f6 mov %rsi,%r14 15: 48 rex.W [ 127.175161] RSP: 0018:ffffc9000438fe60 EFLAGS: 00010246 [ 127.185667] RAX: 0000000000002000 RBX: 0000000000000010 RCX: 0000000000006000 [ 127.199992] RDX: 0000000000000000 RSI: ffff88842a9b5418 RDI: ffffffff83e31178 [ 127.214320] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 127.228667] R10: 0000000000000000 R11: ffff88842a9b54b0 R12: ffffffff83e31178 [ 127.243017] R13: ffff88842a9b5418 R14: ffff88842a9b5418 R15: ffff88842a9b5498 [ 127.257360] FS: 00007f86e4bc3900(0000) GS:ffff8890b23fa000(0000) knlGS:0000000000000000 [ 127.273624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 127.285172] CR2: 000055990443e068 CR3: 00000004183b2001 CR4: 00000000007726f0 [ 127.299500] PKRU: 55555554 [ 127.304961] Call Trace: [ 127.309885] <TASK> [ 127.314121] do_faccessat (fs/open.c:508) [ 127.321488] ? entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:131) [ 127.331973] __x64_sys_access (fs/open.c:549 fs/open.c:547 fs/open.c:547) [ 127.339677] do_syscall_64 (arch/x86/entry/syscall_64.c:?) [ 127.347038] ? rcu_is_watching (./include/linux/context_tracking.h:128 kernel/rcu/tree.c:751) [ 127.354775] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:131) [ 127.364914] RIP: 0033:0x7f86e42ff21b
Lines against commit f0b9d8eb98df ("Merge tag 'nfsd-6.19-3' ofgit://git.kernel.org/pub/scm/linux/kernel/git/cel/linux"), plus something like:
include/linux/device_cgroup.h +1 -1
@@ -21,7 +21,7 @@ static inline int devcgroup_inode_permission(struct inode *inode, int mask)if (likely (!S_ISBLK (inode->i_mode) && !S_ISCHR (inode->i_mode)))return 0 ; -if (likely (!inode->i_rdev)) +if (WARN_ON_ONCE (!inode->i_rdev))return 0 ;if (S_ISBLK (inode->i_mode))
Mateusz Guzik
Reviewed-byQuoted 21 lines · Breno Leitao
Hello Mateusz,
Quoted 17 lines · Mateusz Guzik
include/linux/device_cgroup.h static inline int devcgroup_inode_permission(struct inode *inode, int mask) · line 26
Quoted 12 lines · Breno Leitao
include/linux/device_cgroup.h +1
-1@@ -21,7 +21,7 @@ static inline int devcgroup_inode_permission(struct inode *inode, int mask)if (likely (!S_ISBLK (inode->i_mode) && !S_ISCHR (inode->i_mode)))return 0 ; -if (likely (!inode->i_rdev)) +if (!inode->i_rdev)return 0 ;The branch was left there because I could not be bothered to analyze whether it can be straight up eleminated with the new checks in place.
Quoted 9 lines
Quoted 5 lines
A quick look at
init_special_inodesuggests it is an invariant rdev is there in this case.So for the time being I would replace likely with
WARN_ON_ONCE. Might be even a good candidate for the pending release.Oh, in fact that was my first try, but, when I tested it, I found that it warns in some cases.
heh, bummer
in that case:
thanks
Quoted 80 lines
[ 126.951410] WARNING: ./include/linux/device_cgroup.h:24 at inode_permission+0x181/0x190, CPU#4: networkd)/1212 [ 126.971659] Modules linked in: intel_uncore_frequency(E) intel_uncore_frequency_common(E) skx_edac(E) skx_edac_common(E) nfit(E) libnvdimm(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) iTCO_vendor_support(E) irqbypass(E) acpi_cpufreq(E) i2c_i801(E) i2c_smbus(E) wmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) evdev(E) button(E) bpf_preload(E) sch_fq_codel(E) xhci_pci(E) xhci_hcd(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) loop(E) efivarfs(E) autofs4(E) [ 127.097485] Tainted: [S]=CPU_OUT_OF_SPEC, [E]=UNSIGNED_MODULE [ 127.109038] Hardware name: Quanta Delta Lake MP 29F0EMA01D0/Delta Lake-Class1, BIOS F0E_3A21 06/27/2024 [ 127.127889] RIP: 0010:inode_permission (./include/linux/device_cgroup.h:24 fs/namei.c:600) [ 127.137542] Code: 14 81 e2 ff ff 0f 00 31 ff 3d 00 60 00 00 41 8d 0c 48 40 0f 95 c7 ff c7 e8 5c 68 35 00 85 c0 0f 85 6d ff ff ff e9 29 ff ff ff <0f> 0b e9 22 ff ff ff 0f 1f 84 00 00 00 00 00 41 56 53 49 89 f6 48 All code ======== 0: 14 81 adc $0x81,%al 2: e2 ff loop 0x3 4: ff 0f decl (%rdi) 6: 00 31 add %dh,(%rcx) 8: ff (bad) 9: 3d 00 60 00 00 cmp $0x6000,%eax e: 41 8d 0c 48 lea (%r8,%rcx,2),%ecx 12: 40 0f 95 c7 setne %dil 16: ff c7 inc %edi 18: e8 5c 68 35 00 call 0x356879 1d: 85 c0 test %eax,%eax 1f: 0f 85 6d ff ff ff jne 0xffffffffffffff92 25: e9 29 ff ff ff jmp 0xffffffffffffff53 2a:* 0f 0b ud2 <-- trapping instruction 2c: e9 22 ff ff ff jmp 0xffffffffffffff53 31: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 38: 00 39: 41 56 push %r14 3b: 53 push %rbx 3c: 49 89 f6 mov %rsi,%r14 3f: 48 rex.WCode starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: e9 22 ff ff ff jmp 0xffffffffffffff29 7: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) e: 00 f: 41 56 push %r14 11: 53 push %rbx 12: 49 89 f6 mov %rsi,%r14 15: 48 rex.W [ 127.175161] RSP: 0018:ffffc9000438fe60 EFLAGS: 00010246 [ 127.185667] RAX: 0000000000002000 RBX: 0000000000000010 RCX: 0000000000006000 [ 127.199992] RDX: 0000000000000000 RSI: ffff88842a9b5418 RDI: ffffffff83e31178 [ 127.214320] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 127.228667] R10: 0000000000000000 R11: ffff88842a9b54b0 R12: ffffffff83e31178 [ 127.243017] R13: ffff88842a9b5418 R14: ffff88842a9b5418 R15: ffff88842a9b5498 [ 127.257360] FS: 00007f86e4bc3900(0000) GS:ffff8890b23fa000(0000) knlGS:0000000000000000 [ 127.273624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 127.285172] CR2: 000055990443e068 CR3: 00000004183b2001 CR4: 00000000007726f0 [ 127.299500] PKRU: 55555554 [ 127.304961] Call Trace: [ 127.309885] <TASK> [ 127.314121] do_faccessat (fs/open.c:508) [ 127.321488] ? entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:131) [ 127.331973] __x64_sys_access (fs/open.c:549 fs/open.c:547 fs/open.c:547) [ 127.339677] do_syscall_64 (arch/x86/entry/syscall_64.c:?) [ 127.347038] ? rcu_is_watching (./include/linux/context_tracking.h:128 kernel/rcu/tree.c:751) [ 127.354775] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:131) [ 127.364914] RIP: 0033:0x7f86e42ff21bLines against commit
f0b9d8eb98df("Merge tag 'nfsd-6.19-3' ofgit://git.kernel.org/pub/scm/linux/kernel/git/cel/linux"), plus something like:include/linux/device_cgroup.h +1
-1@@ -21,7 +21,7 @@ static inline int devcgroup_inode_permission(struct inode *inode, int mask)if (likely (!S_ISBLK (inode->i_mode) && !S_ISCHR (inode->i_mode)))return 0 ; -if (likely (!inode->i_rdev)) +if (WARN_ON_ONCE (!inode->i_rdev))return 0 ;if (S_ISBLK (inode->i_mode))
Quoted 11 lines · Breno Leitao
commit
4ef4ac360101("device_cgroup: avoid access to ->i_rdev in the common case indevcgroup_inode_permission()") reordered the checks indevcgroup_inode_permission()to check the inode mode before checkingi_rdev, for better cache behavior.However, the
likely()annotation on thei_rdevcheck was not updated to reflect the new code flow. Originally, wheni_rdevwas checked first,likely(!inode->i_rdev)made sense because most inodes were(?) regular files/directories, thusi_rdev== 0.[...]
Applied to the vfs-7.0.misc branch of the vfs/vfs.git tree. Patches in the vfs-7.0.misc branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.0.misc
[1/1] device_cgroup: remove branch hint after code refactor
https://git.kernel.org/vfs/vfs/c/2a255acce2e5