Support `likely`, `unlikely` and `cold_path`
Miguel Ojeda
maintainercover Support `likely`, `unlikely` and `cold_path`
Rust 1.95.0 stabilized the cold_path feature, which means that it is a good time to add the support for this pervasive feature in C.
The functions should work as expected for all Rust >= 1.84.0, which includes our future bumped Rust minimum version. For older versions, they are a no-op.
Miguel Ojeda (2):
rust: prelude: use the "kernel vertical" imports style
rust: std_vendor: add {likely,unlikely,cold_path}()
init/Kconfig | 6 ++ rust/kernel/lib.rs | 9 +++ rust/kernel/prelude.rs | 119 ++++++++++++++++++++++++------- rust/kernel/std_vendor.rs | 144 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 252 insertions(+), 26 deletions(-)
--
2.53.0
Miguel Ojeda
maintainerPATCH 1/2 rust: prelude: use the "kernel vertical" imports style
Format the Rust prelude to use the "kernel vertical" imports style [1].
No functional changes intended.
--- rust/kernel/prelude.rs | 114 +++++++++++++++++++++++++++++++---------- 1 file changed, 88 insertions(+), 26 deletions(-)
rust/kernel/prelude.rs +88 -26
@@ -13,43 +13,105 @@#[doc(no_inline)] pub use core::{ - mem::{align_of, align_of_val, size_of, size_of_val}, - pin::Pin , + mem::{ + align_of, + align_of_val, + size_of, + size_of_val,// + }, + pin::Pin ,// }; -pub use ::ffi::{ - c_char, c_int, c_long, c_longlong, c_schar, c_short, c_uchar, c_uint, c_ulong, c_ulonglong, - c_ushort, c_void,CStr , +pub use ffi::{ + c_char, + c_int, + c_long, + c_longlong, + c_schar, + c_short, + c_uchar, + c_uint, + c_ulong, + c_ulonglong, + c_ushort, + c_void, +CStr ,// }; -pub use crate ::alloc::{flags::*,Box ,KBox ,KVBox ,KVVec ,KVec ,VBox ,VVec ,Vec }; -#[doc(no_inline)] -pub use macros::{export, fmt, kunit_tests, module, vtable}; +pub use macros::{ + export, + fmt, + kunit_tests, + module, + vtable,// +}; -pub use pin_init::{init, pin_data, pin_init, pinned_drop,InPlaceWrite ,Init ,PinInit ,Zeroable }; +pub use pin_init::{ + init, + pin_data, + pin_init, + pinned_drop, +InPlaceWrite , +Init , +PinInit , +Zeroable ,// +}; -pub use super ::{build_assert, build_error}; +pub use super ::{ + alloc::{ + flags::*, +Box , +KBox , +KVBox , +KVVec , +KVec , +VBox , +VVec , +Vec ,// + }, + build_assert, + build_error, + current, + dev_alert, + dev_crit, + dev_dbg, + dev_emerg, + dev_err, + dev_info, + dev_notice, + dev_warn, + error::{ + code::*, +Error , +Result ,// + }, + init::{ +InPlaceInit ,// + }, + pr_alert, + pr_crit, + pr_debug, + pr_emerg, + pr_err, + pr_info, + pr_notice, + pr_warn, + static_assert, + str::{ +CStrExt as _,// + }, + try_init, + try_pin_init, + uaccess::{ +UserPtr ,// + }, +ThisModule ,// +};// `super::std_vendor` is hidden, which makes the macro inline for some reason. #[doc(no_inline)] pub use super ::dbg; -pub use super ::{dev_alert, dev_crit, dev_dbg, dev_emerg, dev_err, dev_info, dev_notice, dev_warn}; -pub use super ::{pr_alert, pr_crit, pr_debug, pr_emerg, pr_err, pr_info, pr_notice, pr_warn}; - -pub use super ::{try_init, try_pin_init}; - -pub use super ::static_assert; - -pub use super ::error::{code::*,Error ,Result }; - -pub use super ::{str::CStrExt as _,ThisModule }; - -pub use super ::init::InPlaceInit ; - -pub use super ::current; - -pub use super ::uaccess::UserPtr ;#[cfg(not( CONFIG_RUSTC_HAS_SLICE_AS_FLATTENED ))] pub use super ::slice::AsFlattened ;
Signature · 1 line
2.53.0
Re: [PATCH 1/2] rust: prelude: use the "kernel vertical" imports style
Quoted 7 lines · Miguel Ojeda
Format the Rust prelude to use the "kernel vertical" imports style [1].
No functional changes intended.
Signed-off-by Miguel Ojeda <[email protected]>
Applied (this one only) to rust-next -- thanks!
[ Rebased. - Miguel ]
Cheers,
Miguel
Miguel Ojeda
maintainerPATCH 2/2 rust: std_vendor: add `{likely,unlikely,cold_path}()`
In reply to Miguel Ojedacold_path is becoming stable [1] in the upcoming Rust 1.95.0 (expected 2026-04-16).
cold_path() can be used directly, but it also allows us to provide likely() and unlikely(), based on cold_path(), similar to the C side ones.
For instance, given:
fn f1(a: i32) -> i32 {
if a < 0 {
return 123;
}
42
}
fn f2(a: i32) -> i32 {
if likely(a < 0) {
return 124;
}
42
}
fn f3(a: i32) -> i32 {
if unlikely(a < 0) {
return 125;
}
42
}LLVM emits the same code it would for similar C functions:
f1:
test %edi,%edi
mov $0x7b,%ecx
mov $0x2a,%eax
cmovs %ecx,%eax
ret
f2:
mov $0x7c,%eax
test %edi,%edi
/-- jns <f2+0xa>
| ret
\-> mov $0x2a,%eax
ret
f3:
test %edi,%edi
/-- js <f3+0xa>
| mov $0x2a,%eax
| ret
\-> mov $0x7d,%eax
retThe feature itself, feature(cold_path), was added in Rust 1.86.0 [2]. For context, Rust 1.85.0 is likely going to be our next minimum supported version.
Previously, a PR in Rust 1.84.0 [3] fixed a number of issues with the likely() and unlikely() intrinsics (by implementing them on top of the new cold_path() intrinsic).
Thus add support for cold_path() by applying several approaches:
- For Rust >= 1.86.0, use directly core's cold_path().
- For Rust >= 1.84.0, we could choose to provide a no-op, but given Rust 1.85.0 will likely be our next minimum, do some effort to support the feature by vendoring core's implementation based on the intrinsic.
- For older versions, provide a no-op implementation since it is simpler (there was no cold_path() intrinsic), since the other intrinsics ({,un}likely()) were fixed later and since we will bump the minimum soon anyway.
And, for all versions, simply provide likely() and unlikely() based on cold_path(), by vendoring core's version (saving a layer using the intrinsics implementation).
In the future, if likely() and unlikely() become stable, we may want to use them directly as well.
Now, in the C side, the likely() and unlikely() macros come from compiler.h, which means it is pretty much available everywhere directly. Thus just add these to the prelude (instead of e.g. re-exporting them in the root or in a new hint module).
This will also mean less churn when we can remove the cold_path() version from std_vendor (and potentially the other two too).
I tested that Rust 1.84.1 and 1.93.0 both generate the code above, and that Rust 1.83.0 and 1.78.0 do not, as expected.
--- init/Kconfig | 6 ++ rust/kernel/lib.rs | 9 +++ rust/kernel/prelude.rs | 5 ++ rust/kernel/std_vendor.rs | 144 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 164 insertions(+)
init/Kconfig +6 -0
@@ -169,6 +169,12 @@ config RUSTC_HAS_FILE_WITH_NUL config RUSTC_HAS_FILE_AS_C_STR def_bool RUSTC_VERSION >= 109100 +config RUSTC_HAS_COLD_PATH_INTRINSIC + def_bool RUSTC_VERSION >= 108400 + +config RUSTC_HAS_COLD_PATH + def_bool RUSTC_VERSION >= 108600 + config PAHOLE_VERSION int default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE))
rust/kernel/lib.rs +9 -0
@@ -37,6 +37,15 @@#![feature(const_ptr_write)] #![feature(const_refs_to_cell)] // +// `feature(cold_path)` is stable since Rust 1.95.0 and available since Rust 1.86.0. For Rust 1.84 +// and 1.85, use the intrinsic. For older versions, use a no-op. +#![cfg_attr( CONFIG_RUSTC_HAS_COLD_PATH , feature(cold_path))] +#![cfg_attr( +all(not( CONFIG_RUSTC_HAS_COLD_PATH ), CONFIG_RUSTC_HAS_COLD_PATH_INTRINSIC ), +allow(internal_features), +feature(core_intrinsics) +)] +// // Expected to become stable. #![feature(arbitrary_self_types)] //
rust/kernel/prelude.rs +5 -0
@@ -98,6 +98,11 @@ pr_notice, pr_warn, static_assert, + std_vendor::{ + cold_path, + likely, + unlikely, // + }, str::{ CStrExt as _, // },
rust/kernel/std_vendor.rs +144 -0
147 lines not on this page · raw message
Signature · 1 line
2.53.0
Gary Guo
reviewerRe: [PATCH 2/2] rust: std_vendor: add `{likely,unlikely,cold_path}()`
Quoted 66 lines · Miguel Ojeda
cold_pathis becoming stable [1] in the upcoming Rust 1.95.0 (expected 2026-04-16).
cold_path()can be used directly, but it also allows us to providelikely()andunlikely(), based oncold_path(), similar to the C side ones.For instance, given:
fn f1(a: i32) -> i32 { if a < 0 { return 123; } 42 } fn f2(a: i32) -> i32 { if likely(a < 0) { return 124; } 42 } fn f3(a: i32) -> i32 { if unlikely(a < 0) { return 125; } 42 }LLVM emits the same code it would for similar C functions:
f1: test %edi,%edi mov $0x7b,%ecx mov $0x2a,%eax cmovs %ecx,%eax ret f2: mov $0x7c,%eax test %edi,%edi /-- jns <f2+0xa> | ret \-> mov $0x2a,%eax ret f3: test %edi,%edi /-- js <f3+0xa> | mov $0x2a,%eax | ret \-> mov $0x7d,%eax retThe feature itself,
feature(cold_path), was added in Rust 1.86.0 [2]. For context, Rust 1.85.0 is likely going to be our next minimum supported version.Previously, a PR in Rust 1.84.0 [3] fixed a number of issues with the
likely()andunlikely()intrinsics (by implementing them on top of the newcold_path()intrinsic).
Quoted 7 lines
Thus add support for
cold_path()by applying several approaches:- For Rust >= 1.86.0,
usedirectlycore'scold_path().- For Rust >= 1.84.0, we could choose to provide a no-op, but given Rust 1.85.0 will likely be our next minimum, do some effort to support the feature by vendoring
core's implementation based on the intrinsic.
Instead of using the intrinsics, we could use this instead:
#[inline(always)]
#[cold]
pub const fn cold_path() {}
This works all the way back to very old Rust versions, and the codegen seems identical, too.
Best,
Gary
Quoted 28 lines
- For older versions, provide a no-op implementation since it is simpler (there was no
cold_path()intrinsic), since the other intrinsics ({,un}likely()) were fixed later and since we will bump the minimum soon anyway.And, for all versions, simply provide
likely()andunlikely()based oncold_path(), by vendoringcore's version (saving a layer using the intrinsics implementation).In the future, if
likely()andunlikely()become stable, we may want to use them directly as well.Now, in the C side, the
likely()andunlikely()macros come fromcompiler.h, which means it is pretty much available everywhere directly. Thus just add these to the prelude (instead of e.g. re-exporting them in the root or in a newhintmodule).This will also mean less churn when we can remove the
cold_path()version fromstd_vendor(and potentially the other two too).I tested that Rust 1.84.1 and 1.93.0 both generate the code above, and that Rust 1.83.0 and 1.78.0 do not, as expected.
Andreas Hindborg
reviewerRe: [PATCH 2/2] rust: std_vendor: add `{likely,unlikely,cold_path}()`
Quoted 75 lines · Gary Guo
Quoted 66 lines · Miguel Ojeda
cold_pathis becoming stable [1] in the upcoming Rust 1.95.0 (expected 2026-04-16).
cold_path()can be used directly, but it also allows us to providelikely()andunlikely(), based oncold_path(), similar to the C side ones.For instance, given:
fn f1(a: i32) -> i32 { if a < 0 { return 123; } 42 } fn f2(a: i32) -> i32 { if likely(a < 0) { return 124; } 42 } fn f3(a: i32) -> i32 { if unlikely(a < 0) { return 125; } 42 }LLVM emits the same code it would for similar C functions:
f1: test %edi,%edi mov $0x7b,%ecx mov $0x2a,%eax cmovs %ecx,%eax ret f2: mov $0x7c,%eax test %edi,%edi /-- jns <f2+0xa> | ret \-> mov $0x2a,%eax ret f3: test %edi,%edi /-- js <f3+0xa> | mov $0x2a,%eax | ret \-> mov $0x7d,%eax retThe feature itself,
feature(cold_path), was added in Rust 1.86.0 [2]. For context, Rust 1.85.0 is likely going to be our next minimum supported version.Previously, a PR in Rust 1.84.0 [3] fixed a number of issues with the
likely()andunlikely()intrinsics (by implementing them on top of the newcold_path()intrinsic).Quoted 7 lines
Thus add support for
cold_path()by applying several approaches:- For Rust >= 1.86.0,
usedirectlycore'scold_path().- For Rust >= 1.84.0, we could choose to provide a no-op, but given Rust 1.85.0 will likely be our next minimum, do some effort to support the feature by vendoring
core's implementation based on the intrinsic.
Quoted 8 lines
Instead of using the intrinsics, we could use this instead:
#[inline(always)]
#[cold]
pub const fncold_path(){}This works all the way back to very old Rust versions, and the codegen seems identical, too.
Allowing feature(core_intrinsics) is allowing a lot more than just cold_path, like volatile_copy_memory. I don't think it's a good idea to allow all these in.
Best regards,
Andreas Hindborg
Re: [PATCH 2/2] rust: std_vendor: add `{likely,unlikely,cold_path}()`
Quoted 4 lines · Andreas Hindborg
Allowing
feature(core_intrinsics)is allowing a lot more than justcold_path, likevolatile_copy_memory. I don't think it's a good idea to allow all these in.
It doesn't really allow "all these in". It is just for two fixed versions, thus other versions would complain if you try to use them.
And, in any case, just allowing a feature doesn't imply we are OK with everyone in the kernel crate can happily use everything in the feature.
Cheers,
Miguel
Andreas Hindborg
reviewerRe: [PATCH 2/2] rust: std_vendor: add `{likely,unlikely,cold_path}()`
Quoted 8 lines · Miguel Ojeda
Quoted 4 lines · Andreas Hindborg
Allowing
feature(core_intrinsics)is allowing a lot more than justcold_path, likevolatile_copy_memory. I don't think it's a good idea to allow all these in.It doesn't really allow "all these in". It is just for two fixed versions, thus other versions would complain if you try to use them.
True.
Quoted 3 lines
And, in any case, just allowing a feature doesn't imply we are OK with everyone in the
kernelcrate can happily use everything in the feature.
I know, but the feature gate is a very effective first frontier. But as you say, with the allow being limited to a few versions, it is not going to matter much in this regard.
Best regards,
Andreas Hindborg
Re: [PATCH 2/2] rust: std_vendor: add `{likely,unlikely,cold_path}()`
In reply to Gary GuoQuoted 9 lines · Gary Guo
Instead of using the intrinsics, we could use this instead:
#[inline(always)]
#[cold]
pub const fncold_path(){}This works all the way back to very old Rust versions, and the codegen seems identical, too.
I was trying to mimic the standard library to be as conservative as possible, i.e. in principle there could be differences with the attribute (perhaps in the future).
But if you think #[cold] will work just as well, then yeah, let's do that.
It would just be for a single version after the bump anyway.
Cheers,
Miguel