rust: add `CacheAligned` for easy cache line alignment of values
Andreas Hindborg
reviewerPATCH 1/1 rust: add `CacheAligned` for easy cache line alignment of values
CacheAligned allows to easily align values to a 64 byte boundary.
An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.
---
--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +59 -0
@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. +#[repr(align( 64 ))] +#[pin_data] +pub struct CacheAligned <T : ?Sized > { +#[pin] + value:T , +} + +impl <T >CacheAligned <T > { +/// Creates an initializer for `CacheAligned<T>` form an initalizer for `T` +pub fn new (t:impl PinInit <T >) ->impl PinInit <CacheAligned <T >> { +pin_init! (CacheAligned { + value <- t + }) + } + +/// Creates a fallible initializer for `CacheAligned<T>` form a fallible +/// initalizer for `T` +pub fn try_new ( + t:impl PinInit <T ,crate ::error::Error >, + ) ->impl PinInit <CacheAligned <T >,crate ::error::Error > { +try_pin_init! (CacheAligned { + value <- t + }?crate ::error::Error ) + } + +/// Get a pointer to the contained value without creating a reference. +/// +/// # Safety +/// +/// - `ptr` must be dereferenceable. +pub const unsafe fn raw_get (ptr: *mut Self ) -> *mut T { +// SAFETY: by function safety requirements `ptr` is valid for read +unsafe { &raw mut ((*ptr).value) } + } +} + +impl <T : ?Sized > core::ops::Deref for CacheAligned <T > { +type Target =T ; + +fn deref (&self) -> &T { + &self.value + } +} + +impl <T : ?Sized > core::ops::DerefMut for CacheAligned <T > { +fn deref_mut (&mut self) -> &mut T { + &mut self.value + } +}
rust/kernel/lib.rs +2 -0
@@ -75,6 +75,7 @@pub mod bug;#[doc(hidden)] pub mod build_assert; +mod cache_aligned;pub mod clk;#[cfg( CONFIG_CONFIGFS_FS )] pub mod configfs;
@@ -156,6 +157,7 @@#[doc(hidden)] pub use bindings; +pub use cache_aligned::CacheAligned ;pub use macros;pub use uapi;
---
base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
change-id: 20260128-cache-aligned-c4c0acf870ff
Best regards,
Signature · 1 line
Andreas Hindborg <[email protected]>
Alexandre Courbot
reviewerQuoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0
rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line.
nit: s/alings/aligns
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))]
While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?
Gary Guo
reviewerQuoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. nit: s/alings/aligns
Quoted 4 lines
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))] While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use
L1_CACHE_BYTEShere?
Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.
On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
Best,
Gary
Quoted 7 lines · Gary Guo
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
The docs need to also avoid mentioning 64 themselves; otherwise, someone may use this and rely on it being 64, i.e. not just cache aligned, but actually 64.
If Andreas really wants a fixed 64 one, then perhaps we want several types like Aligned64 etc.
Cheers,
Miguel
Andreas Hindborg
reviewerQuoted 9 lines · Miguel Ojeda
Quoted 7 lines · Gary Guo
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
Quoted 6 lines
The docs need to also avoid mentioning 64 themselves; otherwise, someone may use this and rely on it being 64, i.e. not just cache aligned, but actually 64.
If Andreas really wants a fixed 64 one, then perhaps we want several types like
Aligned64etc.
I was considering all the options that are mentioned here, and I decided to go with least effort and hear you all out. I agree that Aligned64 is better than CacheAligned when the alignment is fixed and the type is available on all architectures.
How about we gate the module on architectures that use 64 byte cache line? Then we can add a proc macro to generate later if we need to, or we can gate in another module implementation.
Regarding generating the code, how would a proc macro for this work? Do we have environment variable access in proc macros?
Best regards,
Andreas Hindborg
Alice Ryhl
reviewerQuoted 8 lines · Gary Guo
Quoted 3 lines · Alexandre Courbot
On Wed Jan 28, 2026 at 11:05 PM JST, Andreas Hindborg wrote:
While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and useL1_CACHE_BYTEShere?On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
Are you sure? From Tokio:
Quoted 9 lines
Starting from Intel's Sandy Bridge, spatial prefetcher is now pulling pairs of 64-byte cache lines at a time, so we have to align to 128 bytes rather than 64.
Sources:
- https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
- https://github.com/facebook/folly/blob/1b5288e6eea6df074758f877c849b6e73bbb9fbb/folly/lang/Align.h#L107ARM's big.LITTLE architecture has asymmetric cores and "big" cores have 128-byte cache line size.
Quoted 7 lines
Sources:
- https://www.mono-project.com/news/2016/09/12/arm64-icache/powerpc64 has 128-byte cache line size.
https://github.com/tokio-rs/tokio/blob/master/tokio/src/util/cacheline.rs#L85
Gary Guo
reviewerQuoted 6 lines · Alice Ryhl
Quoted 5 lines · Gary Guo
Quoted 3 lines · Alexandre Courbot
On Wed Jan 28, 2026 at 11:05 PM JST, Andreas Hindborg wrote:
While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and useL1_CACHE_BYTEShere?
Quoted 8 lines
Quoted 3 lines
On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
Are you sure? From Tokio:
Quoted 2 lines
Starting from Intel's Sandy Bridge, spatial prefetcher is now pulling pairs of 64-byte cache lines at a time, so we have to align to 128 bytes rather than 64.
A cache line is still 64B, even if a prefetcher might pull in multiple cache lines.
The hardware prefetcher usually only engage when a sequential access pattern is discovered. So if you're doing array access with increasing index, it would engage and pull in next cache line; however if you are performing random access (e.g. following a link list), it would not engage, as otherwise you're effectively having half the number of cache lines available in your L1 cache.
If the software need to fight the hardware prefetcher in general (where there's no regular seqential access pattern) by spreading things further apart in memory, it means that the hardware prefetcher has failed its task and is a bad design :)
Quoted 9 lines
Quoted 9 lines
Sources:
- https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
- https://github.com/facebook/folly/blob/1b5288e6eea6df074758f877c849b6e73bbb9fbb/folly/lang/Align.h#L107ARM's big.LITTLE architecture has asymmetric cores and "big" cores have 128-byte cache line size.
Sources:
- https://www.mono-project.com/news/2016/09/12/arm64-icache/
arch/arm64/include/asm/cache.h defines L1_CACHE_BYTES as 64.
Quoted 5 lines
Quoted 5 lines
powerpc64 has 128-byte cache line size.
There's no PPC support today in kernel Rust today.
Best,
Gary
Andreas Hindborg
reviewerQuoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. nit: s/alings/aligns
Quoted 4 lines
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))] While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use
L1_CACHE_BYTEShere?
Quoted 6 lines
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have
#define L1_CACHE_SHIFT (CONFIG_X86_L1_CACHE_SHIFT)
and for arm64
#define L1_CACHE_SHIFT (6)
and so on.
One thing we could do is run the C preprocessor on a small snippet to expand the L1_CACHE_SHIFT symbol at some point before invoking rustc. Then we can pass the value to rustc via environment variable when building the macros crate. This is similar to how we pass RUST_MODFILE to rustc, sans the cpp invocation.
Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like CONFIG_L1_CACHE_SHIFT.
The latter options is probably the better one, what do you all think?
Best regards,
Andreas Hindborg
Gary Guo
reviewerQuoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. nit: s/alings/aligns
Quoted 4 lines
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))] While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use
L1_CACHE_BYTEShere?Quoted 6 lines
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have
#define L1_CACHE_SHIFT (CONFIG_X86_L1_CACHE_SHIFT)and for arm64
#define L1_CACHE_SHIFT (6)and so on.
One thing we could do is run the C preprocessor on a small snippet to expand the
L1_CACHE_SHIFTsymbol at some point before invokingrustc. Then we can pass the value torustcvia environment variable when building themacroscrate. This is similar to how we passRUST_MODFILEtorustc, sans the cpp invocation.
Quoted 7 lines
Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like
CONFIG_L1_CACHE_SHIFT.The latter options is probably the better one, what do you all think?
Best regards,
Andreas Hindborg
You can implement this with generics alone using associative types.
mod sealed {
pub trait Sealed {
type Repr;
}
}trait Alignment: sealed::Sealed {}
#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) where
Self: Alignment;
impl<const N: usize> Align<N>
where
Self: Alignment,
{
#[inline]
pub fn new() -> Self {
Align([])
}
}macro_rules! impl_align {
() => {};
($a:literal $($rest:literal)*) => {
const _: () = {
#[repr(align($a))]
struct Repr;
impl sealed::Sealed for Align<$a> {
type Repr = Repr;
}
impl Alignment for Align<$a> {}
};
impl_align!($($rest)*);
}
}impl_align!(32 64 128 256);
Best,
Gary
Andreas Hindborg
reviewerQuoted 123 lines · Gary Guo
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. nit: s/alings/aligns
Quoted 4 lines
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))] While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use
L1_CACHE_BYTEShere?Quoted 6 lines
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have
#define L1_CACHE_SHIFT (CONFIG_X86_L1_CACHE_SHIFT)and for arm64
#define L1_CACHE_SHIFT (6)and so on.
One thing we could do is run the C preprocessor on a small snippet to expand the
L1_CACHE_SHIFTsymbol at some point before invokingrustc. Then we can pass the value torustcvia environment variable when building themacroscrate. This is similar to how we passRUST_MODFILEtorustc, sans the cpp invocation.Quoted 7 lines
Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like
CONFIG_L1_CACHE_SHIFT.The latter options is probably the better one, what do you all think?
Best regards,
Andreas HindborgYou can implement this with generics alone using associative types.
mod sealed { pub trait Sealed { type Repr; } }trait Alignment:
sealed::Sealed{}#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) whereSelf: Alignment;impl<const N: usize> Align<N>
where
Self: Alignment,{ #[inline] pub fn new() -> Self { Align([]) } }macro_rules! impl_align { () => {}; ($a:literal $($rest:literal)*) => { const _: () = { #[repr(align($a))] struct Repr; impl sealed::Sealed for Align<$a> { type Repr = Repr; } impl Alignment for Align<$a> {} }; impl_align!($($rest)*); } }
Quoted 1 line
impl_align!(32 64 128 256);
Right, that is great for avoiding code duplication in case we want to have Align<16>, Align<32>, etc.
My discussion was about having just CacheAligned and having the alignment of that type automatically be the 1 << L1_CACHE_SHIFT.
Best regards,
Andreas Hindborg
Gary Guo
reviewerQuoted 125 lines · Andreas Hindborg
Quoted 123 lines · Gary Guo
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. nit: s/alings/aligns
Quoted 4 lines
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))] While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use
L1_CACHE_BYTEShere?Quoted 6 lines
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have
#define L1_CACHE_SHIFT (CONFIG_X86_L1_CACHE_SHIFT)and for arm64
#define L1_CACHE_SHIFT (6)and so on.
One thing we could do is run the C preprocessor on a small snippet to expand the
L1_CACHE_SHIFTsymbol at some point before invokingrustc. Then we can pass the value torustcvia environment variable when building themacroscrate. This is similar to how we passRUST_MODFILEtorustc, sans the cpp invocation.Quoted 7 lines
Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like
CONFIG_L1_CACHE_SHIFT.The latter options is probably the better one, what do you all think?
Best regards,
Andreas HindborgYou can implement this with generics alone using associative types.
mod sealed { pub trait Sealed { type Repr; } }trait Alignment:
sealed::Sealed{}#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) whereSelf: Alignment;impl<const N: usize> Align<N>
where
Self: Alignment,{ #[inline] pub fn new() -> Self { Align([]) } }macro_rules! impl_align { () => {}; ($a:literal $($rest:literal)*) => { const _: () = { #[repr(align($a))] struct Repr; impl sealed::Sealed for Align<$a> { type Repr = Repr; } impl Alignment for Align<$a> {} }; impl_align!($($rest)*); } }
Quoted 7 lines
Quoted 1 line
impl_align!(32 64 128 256);
Right, that is great for avoiding code duplication in case we want to have
Align<16>,Align<32>, etc.My discussion was about having just
CacheAlignedand having the alignment of that type automatically be the1 << L1_CACHE_SHIFT.
#[repr(transparent)]
struct CacheAligned(Align<{ 1 << L1_CACHE_SHIFT }>);
Best,
Gary
Andreas Hindborg
reviewerQuoted 128 lines · Gary Guo
Quoted 127 lines · Andreas Hindborg
Quoted 124 lines · Gary Guo
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +0
-0rust/kernel/cache_aligned.rs line 10
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. nit: s/alings/aligns
Quoted 4 lines
rust/kernel/cache_aligned.rs line 11
Quoted 1 line
rust/kernel/cache_aligned.rs new file +1
-0+#[repr(align( 64 ))] While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use
L1_CACHE_BYTEShere?Quoted 6 lines
Unfortunately
repr(align())does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.On all archs that we do support today, I think the value is always 64 However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.
I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have
#define L1_CACHE_SHIFT (CONFIG_X86_L1_CACHE_SHIFT)and for arm64
#define L1_CACHE_SHIFT (6)and so on.
One thing we could do is run the C preprocessor on a small snippet to expand the
L1_CACHE_SHIFTsymbol at some point before invokingrustc. Then we can pass the value torustcvia environment variable when building themacroscrate. This is similar to how we passRUST_MODFILEtorustc, sans the cpp invocation.Quoted 7 lines
Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like
CONFIG_L1_CACHE_SHIFT.The latter options is probably the better one, what do you all think?
Best regards,
Andreas HindborgYou can implement this with generics alone using associative types.
mod sealed { pub trait Sealed { type Repr; } }trait Alignment:
sealed::Sealed{}#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) whereSelf: Alignment;impl<const N: usize> Align<N>
where
Self: Alignment,{ #[inline] pub fn new() -> Self { Align([]) } }macro_rules! impl_align { () => {}; ($a:literal $($rest:literal)*) => { const _: () = { #[repr(align($a))] struct Repr; impl sealed::Sealed for Align<$a> { type Repr = Repr; } impl Alignment for Align<$a> {} }; impl_align!($($rest)*); } }impl_align!(32 64 128 256);
Quoted 8 lines
Quoted 5 lines
Right, that is great for avoiding code duplication in case we want to have
Align<16>,Align<32>, etc.My discussion was about having just
CacheAlignedand having the alignment of that type automatically be the1 << L1_CACHE_SHIFT.#[repr(transparent)]
structCacheAligned(Align<{ 1 << L1_CACHE_SHIFT }>);
Oh, I see. I somehow got the idea that this was gated behind generic_const_exprs. But apparently that is only if a generic parameter is part of the expression 🤦
Best regards,
Andreas Hindborg
Quoted 4 lines · Andreas Hindborg
Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>
Something strange is going on with the tags.
Quoted 1 line
+// SPDX-License-Identifier: GPL-2.0
Please give a title to the module even if it is not public.
rust/kernel/cache_aligned.rs line 10
Quoted 8 lines
rust/kernel/cache_aligned.rs new file +8
-0+use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line.
Typo.
More importantly, please add some docs and examples -- the commit message has more documentation than the code... :)
Also, it would be nice to show a user in a second patch.
Quoted 1 line
+#[repr(align(64))]
Even if 64 bytes is common, wouldn't this depend on the system?
rust/kernel/cache_aligned.rs line 27
Quoted 2 lines
rust/kernel/cache_aligned.rs new file +2
-0+/// Creates a fallible initializer for `CacheAligned<T>` form a fallible +/// initalizer for `T`
Two typos and missing period at the end.
Also just using [CacheAligned] would probably be simpler.
Quoted 1 line
+ // SAFETY: by function safety requirements
ptris valid for read
// SAFETY: By the the safety requirement, `ptr` is valid for read.
Quoted 1 line
+ try_pin_init!( CacheAligned {
This and another one is formatted differently than we usually do.
Quoted 1 line
+pub use
cache_aligned::CacheAligned;
It seems you want this short, in which case it should perhaps go in that case, but I think it is best to leave an extra level otherwise and let users import it.
Thanks!
Cheers,
Miguel
Quoted 3 lines · Miguel Ojeda
It seems you want this short, in which case it should perhaps go in that case,
s/in that/in the prelude in that/
Cheers,
Miguel
Andreas Hindborg
reviewerQuoted 39 lines · Miguel Ojeda
Quoted 4 lines · Andreas Hindborg
Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>Something strange is going on with the tags.
Quoted 1 line
+// SPDX-License-Identifier: GPL-2.0
Please give a title to the module even if it is not public.
rust/kernel/cache_aligned.rs line 10
Quoted 8 lines
rust/kernel/cache_aligned.rs new file +8
-0+use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. Typo.
More importantly, please add some docs and examples -- the commit message has more documentation than the code... :)
Also, it would be nice to show a user in a second patch.
Quoted 1 line
+#[repr(align(64))]
Even if 64 bytes is common, wouldn't this depend on the system?
rust/kernel/cache_aligned.rs line 27
Quoted 2 lines
rust/kernel/cache_aligned.rs new file +2
-0+/// Creates a fallible initializer for `CacheAligned<T>` form a fallible +/// initalizer for `T` Two typos and missing period at the end.
Also just using [
CacheAligned] would probably be simpler.
Quoted 7 lines
Quoted 1 line
+ // SAFETY: by function safety requirements
ptris valid for read// SAFETY: By the the safety requirement, `ptr` is valid for read.Quoted 1 line
+ try_pin_init!( CacheAligned {
This and another one is formatted differently than we usually do.
Thanks for the comments, I agree with all.
Quoted 6 lines
Quoted 2 lines
+pub use
cache_aligned::CacheAligned;It seems you want this short, in which case it should perhaps go in that case, but I think it is best to leave an extra level otherwise and let users import it.
I think I wrote this before kernel::prelude. I would put it in the prelude today instead of here, what do you think about that?
Best regards,
Andreas Hindborg
Quoted 3 lines · Andreas Hindborg
I think I wrote this before
kernel::prelude. I would put it in the prelude today instead of here, what do you think about that?
I guess it depends on how much we expect it to be used.
i.e. so far we put things in the prelude if they are both general ("core") enough and frequently used.
Cheers,
Miguel
Gary Guo
reviewerQuoted 8 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.
Do you mean CacheAligned<SpinLock<T>>?
Quoted 4 lines
Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>
Double SOB
Quoted 35 lines
--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +24
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. +#[repr(align( 64 ))] +#[pin_data] +pub struct CacheAligned <T : ?Sized > { +#[pin] + value:T , +} + +impl <T >CacheAligned <T > { +/// Creates an initializer for `CacheAligned<T>` form an initalizer for `T` +pub fn new (t:impl PinInit <T >) ->impl PinInit <CacheAligned <T >> { +pin_init! (CacheAligned { + value <- t + }) + }
rust/kernel/cache_aligned.rs line 34
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0+ +/// Creates a fallible initializer for `CacheAligned<T>` form a fallible +/// initalizer for `T` +pub fn try_new ( + t:impl PinInit <T ,crate ::error::Error >, + ) ->impl PinInit <CacheAligned <T >,crate ::error::Error > { +try_pin_init! (CacheAligned { + value <- t + }?crate ::error::Error ) + }
You don't need two methods. You can have a single new method that's generic over error type.
rust/kernel/cache_aligned.rs line 44
Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10
-0+ +/// Get a pointer to the contained value without creating a reference. +/// +/// # Safety +/// +/// - `ptr` must be dereferenceable. +pub const unsafe fn raw_get (ptr: *mut Self ) -> *mut T { +// SAFETY: by function safety requirements `ptr` is valid for read +unsafe { &raw mut ((*ptr).value) } + }
Have you had a case where you need this? Most wrapper types shouldn't need this.
Quoted 28 lines
rust/kernel/cache_aligned.rs new file +15
-0+} + +impl <T : ?Sized > core::ops::Deref for CacheAligned <T > { +type Target =T ; + +fn deref (&self) -> &T { + &self.value + } +} + +impl <T : ?Sized > core::ops::DerefMut for CacheAligned <T > { +fn deref_mut (&mut self) -> &mut T { + &mut self.value + } +}rust/kernel/lib.rs +1
-0@@ -75,6 +75,7 @@pub mod bug;#[doc(hidden)] pub mod build_assert; +mod cache_aligned;pub mod clk;#[cfg( CONFIG_CONFIGFS_FS )] pub mod configfs;
rust/kernel/lib.rs line 160
Quoted 4 lines
rust/kernel/lib.rs +1
-0@@ -156,6 +157,7 @@#[doc(hidden)] pub use bindings; +pub use cache_aligned::CacheAligned ;
Let's not expose this from top-level of kernel crate.
I have been thinking about a good namespace for these auxillary types. For my own project I would chuck them to crate::utils, but that won't be a very descriptive name.
I wonder for this and other types that tweak the memory layout, we could have a kernel::layout which contains utilities for precisely controlling the layout?
Best,
Gary
Quoted 9 lines
rust/kernel/lib.rs +0
-1pub use macros;pub use uapi; ---base-commit:
63804fed149a6750ffd28610c5c1c98cce6bd377
change-id: 20260128-cache-aligned-c4c0acf870ffBest regards,
Andreas Hindborg
reviewerQuoted 10 lines · Gary Guo
Quoted 8 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.
Quoted 8 lines
Do you mean
CacheAligned<SpinLock<T>>?Quoted 4 lines
Signed-off-by Andreas Hindborg <[email protected]>---
Signed-off-by Andreas Hindborg <[email protected]>Double SOB
b4 did this! It only has the @samsung SOB in my tree.
Quoted 41 lines
Quoted 41 lines
--- rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 2 ++ 2 files changed, 61 insertions(+)rust/kernel/cache_aligned.rs new file +29
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::try_pin_init; +use pin_init::{ + pin_data, + pin_init, +PinInit ,// +}; + +/// Wrapper type that alings content to a 64 byte cache line. +#[repr(align( 64 ))] +#[pin_data] +pub struct CacheAligned <T : ?Sized > { +#[pin] + value:T , +} + +impl <T >CacheAligned <T > { +/// Creates an initializer for `CacheAligned<T>` form an initalizer for `T` +pub fn new (t:impl PinInit <T >) ->impl PinInit <CacheAligned <T >> { +pin_init! (CacheAligned { + value <- t + }) + } + +/// Creates a fallible initializer for `CacheAligned<T>` form a fallible +/// initalizer for `T` +pub fn try_new ( + t: impl PinInit<T, crate::error::Error>,
Quoted 8 lines
rust/kernel/cache_aligned.rs line 34
Quoted 5 lines
rust/kernel/cache_aligned.rs new file +5
-0+ ) -> impl PinInit<CacheAligned<T>, crate::error::Error> { +try_pin_init! (CacheAligned { + value <- t + }?crate ::error::Error ) + }You don't need two methods. You can have a single
newmethod that's generic over error type.
Ok, cool.
Quoted 12 lines
rust/kernel/cache_aligned.rs line 44
Quoted 11 lines
rust/kernel/cache_aligned.rs new file +10
-0+ +/// Get a pointer to the contained value without creating a reference. +/// +/// # Safety +/// +/// - `ptr` must be dereferenceable. +pub const unsafe fn raw_get (ptr: *mut Self ) -> *mut T { +// SAFETY: by function safety requirements `ptr` is valid for read +unsafe { &raw mut ((*ptr).value) } + }
Quoted 1 line
Have you had a case where you need this? Most wrapper types shouldn't need this.
I did, but I do not currently. I'll drop it.
Quoted 30 lines
Quoted 30 lines
rust/kernel/cache_aligned.rs new file +15
-0+} + +impl <T : ?Sized > core::ops::Deref for CacheAligned <T > { +type Target =T ; + +fn deref (&self) -> &T { + &self.value + } +} + +impl <T : ?Sized > core::ops::DerefMut for CacheAligned <T > { +fn deref_mut (&mut self) -> &mut T { + &mut self.value + } +}rust/kernel/lib.rs +1
-0@@ -75,6 +75,7 @@pub mod bug;#[doc(hidden)] pub mod build_assert; +mod cache_aligned;pub mod clk;#[cfg( CONFIG_CONFIGFS_FS )] pub mod configfs;@@ -156,6 +157,7 @@
Quoted 5 lines
rust/kernel/lib.rs line 160
Quoted 3 lines
rust/kernel/lib.rs +1
-0#[doc(hidden)] pub use bindings; +pub use cache_aligned::CacheAligned ;Let's not expose this from top-level of kernel crate.
Right. As I told Miguel, I would put this in the prelude.
Quoted 7 lines
I have been thinking about a good namespace for these auxillary types. For my own project I would chuck them to
crate::utils, but that won't be a very descriptive name.I wonder for this and other types that tweak the memory layout, we could have a
kernel::layoutwhich contains utilities for precisely controlling the layout?
How about kernel::mem since it is memory related?
Best regards,
Andreas Hindborg
Andreas Hindborg
reviewerQuoted 12 lines · Gary Guo
Quoted 8 lines · Andreas Hindborg
CacheAlignedallows to easily align values to a 64 byte boundary.An example use case is the kernel
struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this forSpinLockis that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value inCacheAlignedto get aSpinLock<CacheAligned<T>>solves this problem.Do you mean
CacheAligned<SpinLock<T>>?
I missed this in my original reply.
I mean SpinLock<CacheAligned<T>> as I wrote. The SpinLock will be aligned to some alignment, possibly the start of a cache line. Taking the lock involves getting exclusive ownership of the cache line and updating the lock state. When the lock is contended, the CPU core that takes the lock will loose exclusive ownership of the cache line immediately after taking the lock, as other CPU cores are trying to take the lock.
The logical thing to do after taking a lock is looking at the value protected by the lock. As the winning CPU tries to do this, it has to obtain ownership of the cache line again, assuming the T lives on the same cache line as the lock state. If we align the T to a cache line, then that line will not be contended. Only the cache line holding the lock state will be contended. And then observing the T does not incur the overhead obtaining a highly contended cache line.
Best regards,
Andreas Hindborg