Skip to contentSkip to conversation

rust: add `CacheAligned` for easy cache line alignment of values

PATCH In review Andreas Hindborg 5 participants 19 messages Jan 28 – May 19 Rust

lore
Share & tools
lore

Andreas Hindborg

author reviewer
#0 lore raw

PATCH 1/1 rust: add `CacheAligned` for easy cache line alignment of values

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +59 -0
new file mode 100644
index 0000000000000..9c33b8613c077
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.
+#[repr(align(64))]
+#[pin_data]
+pub struct CacheAligned<T: ?Sized> {
+    #[pin]
+    value: T,
+}
+
+impl<T> CacheAligned<T> {
+    /// Creates an initializer for `CacheAligned<T>` form an initalizer for `T`
+    pub fn new(t: impl PinInit<T>) -> impl PinInit<CacheAligned<T>> {
+        pin_init!( CacheAligned {
+            value <- t
+        })
+    }
+
+    /// Creates a fallible initializer for `CacheAligned<T>` form a fallible
+    /// initalizer for `T`
+    pub fn try_new(
+        t: impl PinInit<T, crate::error::Error>,
+    ) -> impl PinInit<CacheAligned<T>, crate::error::Error> {
+        try_pin_init!( CacheAligned {
+            value <- t
+        }? crate::error::Error )
+    }
+
+    /// Get a pointer to the contained value without creating a reference.
+    ///
+    /// # Safety
+    ///
+    /// - `ptr` must be dereferenceable.
+    pub const unsafe fn raw_get(ptr: *mut Self) -> *mut T {
+        // SAFETY: by function safety requirements `ptr` is valid for read
+        unsafe { &raw mut ((*ptr).value) }
+    }
+}
+
+impl<T: ?Sized> core::ops::Deref for CacheAligned<T> {
+    type Target = T;
+
+    fn deref(&self) -> &T {
+        &self.value
+    }
+}
+
+impl<T: ?Sized> core::ops::DerefMut for CacheAligned<T> {
+    fn deref_mut(&mut self) -> &mut T {
+        &mut self.value
+    }
+}
rust/kernel/lib.rs +2 -0
index f812cf1200428..af6d48b078428 100644
@@ -75,6 +75,7 @@
 pub mod bug;
 #[doc(hidden)]
 pub mod build_assert;
+mod cache_aligned;
 pub mod clk;
 #[cfg(CONFIG_CONFIGFS_FS)]
 pub mod configfs;
@@ -156,6 +157,7 @@
 
 #[doc(hidden)]
 pub use bindings;
+pub use cache_aligned::CacheAligned;
 pub use macros;
 pub use uapi;
 

---
base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
change-id: 20260128-cache-aligned-c4c0acf870ff

Best regards,

Signature · 1 line
Andreas Hindborg <[email protected]>
#1 lore raw
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

reviewer
#2 lore raw
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

Quoted 4 lines

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

Best,
Gary

#3 lore raw
Quoted 7 lines · Gary Guo

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

The docs need to also avoid mentioning 64 themselves; otherwise, someone may use this and rely on it being 64, i.e. not just cache aligned, but actually 64.

If Andreas really wants a fixed 64 one, then perhaps we want several types like Aligned64 etc.

Cheers,
Miguel

author reviewer
#4 lore raw
Quoted 9 lines · Miguel Ojeda
Quoted 7 lines · Gary Guo

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

Quoted 6 lines

The docs need to also avoid mentioning 64 themselves; otherwise, someone may use this and rely on it being 64, i.e. not just cache aligned, but actually 64.

If Andreas really wants a fixed 64 one, then perhaps we want several types like Aligned64 etc.

I was considering all the options that are mentioned here, and I decided to go with least effort and hear you all out. I agree that Aligned64 is better than CacheAligned when the alignment is fixed and the type is available on all architectures.

How about we gate the module on architectures that use 64 byte cache line? Then we can add a proc macro to generate later if we need to, or we can gate in another module implementation.

Regarding generating the code, how would a proc macro for this work? Do we have environment variable access in proc macros?

Best regards,
Andreas Hindborg

reviewer
#5 lore raw
In reply to Gary Guo
Quoted 8 lines · Gary Guo
Quoted 3 lines · Alexandre Courbot

On Wed Jan 28, 2026 at 11:05 PM JST, Andreas Hindborg wrote:
While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

Are you sure? From Tokio:

Quoted 9 lines

Starting from Intel's Sandy Bridge, spatial prefetcher is now pulling pairs of 64-byte cache lines at a time, so we have to align to 128 bytes rather than 64.

Sources:
- https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
- https://github.com/facebook/folly/blob/1b5288e6eea6df074758f877c849b6e73bbb9fbb/folly/lang/Align.h#L107

ARM's big.LITTLE architecture has asymmetric cores and "big" cores have 128-byte cache line size.

Quoted 7 lines

Sources:
- https://www.mono-project.com/news/2016/09/12/arm64-icache/

powerpc64 has 128-byte cache line size.

Sources:
- https://github.com/golang/go/blob/3dd58676054223962cd915bb0934d1f9f489d4d2/src/internal/cpu/cpu_ppc64x.go#L9

https://github.com/tokio-rs/tokio/blob/master/tokio/src/util/cacheline.rs#L85

reviewer
#6 lore raw
Quoted 6 lines · Alice Ryhl
Quoted 5 lines · Gary Guo
Quoted 3 lines · Alexandre Courbot

On Wed Jan 28, 2026 at 11:05 PM JST, Andreas Hindborg wrote:
While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Quoted 8 lines
Quoted 3 lines

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

Are you sure? From Tokio:

Quoted 2 lines

Starting from Intel's Sandy Bridge, spatial prefetcher is now pulling pairs of 64-byte cache lines at a time, so we have to align to 128 bytes rather than 64.

A cache line is still 64B, even if a prefetcher might pull in multiple cache lines.

The hardware prefetcher usually only engage when a sequential access pattern is discovered. So if you're doing array access with increasing index, it would engage and pull in next cache line; however if you are performing random access (e.g. following a link list), it would not engage, as otherwise you're effectively having half the number of cache lines available in your L1 cache.

If the software need to fight the hardware prefetcher in general (where there's no regular seqential access pattern) by spreading things further apart in memory, it means that the hardware prefetcher has failed its task and is a bad design :)

Quoted 9 lines
Quoted 9 lines

Sources:
- https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
- https://github.com/facebook/folly/blob/1b5288e6eea6df074758f877c849b6e73bbb9fbb/folly/lang/Align.h#L107

ARM's big.LITTLE architecture has asymmetric cores and "big" cores have 128-byte cache line size.

Sources:
- https://www.mono-project.com/news/2016/09/12/arm64-icache/

arch/arm64/include/asm/cache.h defines L1_CACHE_BYTES as 64.

Quoted 5 lines
Quoted 5 lines

powerpc64 has 128-byte cache line size.

Sources:
- https://github.com/golang/go/blob/3dd58676054223962cd915bb0934d1f9f489d4d2/src/internal/cpu/cpu_ppc64x.go#L9

There's no PPC support today in kernel Rust today.

Best,
Gary

Quoted 2 lines

https://github.com/tokio-rs/tokio/blob/master/tokio/src/util/cacheline.rs#L85

author reviewer
#7 lore raw
In reply to Gary Guo
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

Quoted 4 lines

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Quoted 6 lines

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have

  #define L1_CACHE_SHIFT	(CONFIG_X86_L1_CACHE_SHIFT)

and for arm64

  #define L1_CACHE_SHIFT		(6)

and so on.

One thing we could do is run the C preprocessor on a small snippet to expand the L1_CACHE_SHIFT symbol at some point before invoking rustc. Then we can pass the value to rustc via environment variable when building the macros crate. This is similar to how we pass RUST_MODFILE to rustc, sans the cpp invocation.

Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like CONFIG_L1_CACHE_SHIFT.

The latter options is probably the better one, what do you all think?

Best regards,
Andreas Hindborg

reviewer
#8 lore raw
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

Quoted 4 lines

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Quoted 6 lines

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have

  #define L1_CACHE_SHIFT	(CONFIG_X86_L1_CACHE_SHIFT)

and for arm64

  #define L1_CACHE_SHIFT		(6)

and so on.

One thing we could do is run the C preprocessor on a small snippet to expand the L1_CACHE_SHIFT symbol at some point before invoking rustc. Then we can pass the value to rustc via environment variable when building the macros crate. This is similar to how we pass RUST_MODFILE to rustc, sans the cpp invocation.

Quoted 7 lines

Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like CONFIG_L1_CACHE_SHIFT.

The latter options is probably the better one, what do you all think?

Best regards,
Andreas Hindborg

You can implement this with generics alone using associative types.

mod sealed {
    pub trait Sealed {
        type Repr;
    }
}

trait Alignment: sealed::Sealed {}

#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) where

    Self: Alignment;

impl<const N: usize> Align<N>
where
Self: Alignment,

{
    #[inline]
    pub fn new() -> Self {
        Align([])
    }
}
macro_rules! impl_align {
    () => {};
    ($a:literal $($rest:literal)*) => {
        const _: () = {
            #[repr(align($a))]
            struct Repr;

            impl sealed::Sealed for Align<$a> {
                type Repr = Repr;
            }
            
            impl Alignment for Align<$a> {}
        };
        
        impl_align!($($rest)*);
    }
}

impl_align!(32 64 128 256);

Best,
Gary

author reviewer
#9 lore raw
Quoted 123 lines · Gary Guo
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

Quoted 4 lines

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Quoted 6 lines

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have

  #define L1_CACHE_SHIFT	(CONFIG_X86_L1_CACHE_SHIFT)

and for arm64

  #define L1_CACHE_SHIFT		(6)

and so on.

One thing we could do is run the C preprocessor on a small snippet to expand the L1_CACHE_SHIFT symbol at some point before invoking rustc. Then we can pass the value to rustc via environment variable when building the macros crate. This is similar to how we pass RUST_MODFILE to rustc, sans the cpp invocation.

Quoted 7 lines

Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like CONFIG_L1_CACHE_SHIFT.

The latter options is probably the better one, what do you all think?

Best regards,
Andreas Hindborg

You can implement this with generics alone using associative types.

mod sealed {
    pub trait Sealed {
        type Repr;
    }
}

trait Alignment: sealed::Sealed {}

#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) where

    Self: Alignment;

impl<const N: usize> Align<N>
where
Self: Alignment,

{
    #[inline]
    pub fn new() -> Self {
        Align([])
    }
}
macro_rules! impl_align {
    () => {};
    ($a:literal $($rest:literal)*) => {
        const _: () = {
            #[repr(align($a))]
            struct Repr;

            impl sealed::Sealed for Align<$a> {
                type Repr = Repr;
            }
            
            impl Alignment for Align<$a> {}
        };
        
        impl_align!($($rest)*);
    }
}
Quoted 1 line

impl_align!(32 64 128 256);

Right, that is great for avoiding code duplication in case we want to have Align<16>, Align<32>, etc.

My discussion was about having just CacheAligned and having the alignment of that type automatically be the 1 << L1_CACHE_SHIFT.

Best regards,
Andreas Hindborg

reviewer
#10 lore raw
Quoted 125 lines · Andreas Hindborg
Quoted 123 lines · Gary Guo
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

Quoted 4 lines

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Quoted 6 lines

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64. However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have

  #define L1_CACHE_SHIFT	(CONFIG_X86_L1_CACHE_SHIFT)

and for arm64

  #define L1_CACHE_SHIFT		(6)

and so on.

One thing we could do is run the C preprocessor on a small snippet to expand the L1_CACHE_SHIFT symbol at some point before invoking rustc. Then we can pass the value to rustc via environment variable when building the macros crate. This is similar to how we pass RUST_MODFILE to rustc, sans the cpp invocation.

Quoted 7 lines

Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like CONFIG_L1_CACHE_SHIFT.

The latter options is probably the better one, what do you all think?

Best regards,
Andreas Hindborg

You can implement this with generics alone using associative types.

mod sealed {
    pub trait Sealed {
        type Repr;
    }
}

trait Alignment: sealed::Sealed {}

#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) where

    Self: Alignment;

impl<const N: usize> Align<N>
where
Self: Alignment,

{
    #[inline]
    pub fn new() -> Self {
        Align([])
    }
}
macro_rules! impl_align {
    () => {};
    ($a:literal $($rest:literal)*) => {
        const _: () = {
            #[repr(align($a))]
            struct Repr;

            impl sealed::Sealed for Align<$a> {
                type Repr = Repr;
            }
            
            impl Alignment for Align<$a> {}
        };
        
        impl_align!($($rest)*);
    }
}
Quoted 7 lines
Quoted 1 line

impl_align!(32 64 128 256);

Right, that is great for avoiding code duplication in case we want to have Align<16>, Align<32>, etc.

My discussion was about having just CacheAligned and having the alignment of that type automatically be the 1 << L1_CACHE_SHIFT.

#[repr(transparent)]
struct CacheAligned(Align<{ 1 << L1_CACHE_SHIFT }>);

Best,
Gary

author reviewer
#11 lore raw
Quoted 128 lines · Gary Guo
Quoted 127 lines · Andreas Hindborg
Quoted 124 lines · Gary Guo
Quoted 71 lines · Andreas Hindborg
Quoted 43 lines · Gary Guo
Quoted 37 lines · Alexandre Courbot
Quoted 23 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

---

---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +0 -0
new file mode 100644
index 0000000000000..9c33b8613c077

rust/kernel/cache_aligned.rs line 10

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

nit: s/alings/aligns

Quoted 4 lines

rust/kernel/cache_aligned.rs line 11

Quoted 1 line
rust/kernel/cache_aligned.rs new file +1 -0
+#[repr(align(64))]

While 64 bytes is the most common cache line size, AFAIK this is not a universal value? Can we expose and use L1_CACHE_BYTES here?

Quoted 6 lines

Unfortunately repr(align()) does not accept expression or macro invocations. It's still possible with code-generation, but it'll be more tricky.

On all archs that we do support today, I think the value is always 64 However it'd worth putting a FIXME or TODO (or assertion, maybe?) in case new archs gets addded where this isn't true.

I was looking into how to implement this properly. Apparently, we don't have a config item that specifies the L1 cache line size for all architectures. Each architectures defines the cache line size as C define in a header. For x86 we have

  #define L1_CACHE_SHIFT	(CONFIG_X86_L1_CACHE_SHIFT)

and for arm64

  #define L1_CACHE_SHIFT		(6)

and so on.

One thing we could do is run the C preprocessor on a small snippet to expand the L1_CACHE_SHIFT symbol at some point before invoking rustc. Then we can pass the value to rustc via environment variable when building the macros crate. This is similar to how we pass RUST_MODFILE to rustc, sans the cpp invocation.

Quoted 7 lines

Otherwise we have to convince all architectures that support Rust to emit a config that we can rely on, like CONFIG_L1_CACHE_SHIFT.

The latter options is probably the better one, what do you all think?

Best regards,
Andreas Hindborg

You can implement this with generics alone using associative types.

mod sealed {
    pub trait Sealed {
        type Repr;
    }
}

trait Alignment: sealed::Sealed {}

#[repr(transparent)]
struct Align<const N: usize>([<Self as sealed::Sealed>::Repr; 0]) where

    Self: Alignment;

impl<const N: usize> Align<N>
where
Self: Alignment,

{
    #[inline]
    pub fn new() -> Self {
        Align([])
    }
}
macro_rules! impl_align {
    () => {};
    ($a:literal $($rest:literal)*) => {
        const _: () = {
            #[repr(align($a))]
            struct Repr;

            impl sealed::Sealed for Align<$a> {
                type Repr = Repr;
            }

            impl Alignment for Align<$a> {}
        };

        impl_align!($($rest)*);
    }
}

impl_align!(32 64 128 256);

Quoted 8 lines
Quoted 5 lines

Right, that is great for avoiding code duplication in case we want to have Align<16>, Align<32>, etc.

My discussion was about having just CacheAligned and having the alignment of that type automatically be the 1 << L1_CACHE_SHIFT.

#[repr(transparent)]
struct CacheAligned(Align<{ 1 << L1_CACHE_SHIFT }>);

Oh, I see. I somehow got the idea that this was gated behind generic_const_exprs. But apparently that is only if a generic parameter is part of the expression 🤦

Best regards,
Andreas Hindborg

#12 lore raw
In reply to Andreas Hindborg
Quoted 4 lines · Andreas Hindborg

---

Something strange is going on with the tags.

Quoted 1 line

+// SPDX-License-Identifier: GPL-2.0

Please give a title to the module even if it is not public.

rust/kernel/cache_aligned.rs line 10

Quoted 8 lines
rust/kernel/cache_aligned.rs new file +8 -0
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

Typo.

More importantly, please add some docs and examples -- the commit message has more documentation than the code... :)

Also, it would be nice to show a user in a second patch.

Quoted 1 line

+#[repr(align(64))]

Even if 64 bytes is common, wouldn't this depend on the system?

rust/kernel/cache_aligned.rs line 27

Quoted 2 lines
rust/kernel/cache_aligned.rs new file +2 -0
+    /// Creates a fallible initializer for `CacheAligned<T>` form a fallible
+    /// initalizer for `T`

Two typos and missing period at the end.

Also just using [CacheAligned] would probably be simpler.

Quoted 1 line

+ // SAFETY: by function safety requirements ptr is valid for read

    // SAFETY: By the the safety requirement, `ptr` is valid for read.
Quoted 1 line

+ try_pin_init!( CacheAligned {

This and another one is formatted differently than we usually do.

Quoted 1 line

+pub use cache_aligned::CacheAligned;

It seems you want this short, in which case it should perhaps go in that case, but I think it is best to leave an extra level otherwise and let users import it.

Thanks!

Cheers,
Miguel

#13 lore raw
Quoted 3 lines · Miguel Ojeda

It seems you want this short, in which case it should perhaps go in that case,

s/in that/in the prelude in that/

Cheers,
Miguel

author reviewer
#14 lore raw
In reply to Miguel Ojeda
Quoted 39 lines · Miguel Ojeda
Quoted 4 lines · Andreas Hindborg

---

Something strange is going on with the tags.

Quoted 1 line

+// SPDX-License-Identifier: GPL-2.0

Please give a title to the module even if it is not public.

rust/kernel/cache_aligned.rs line 10

Quoted 8 lines
rust/kernel/cache_aligned.rs new file +8 -0
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.

Typo.

More importantly, please add some docs and examples -- the commit message has more documentation than the code... :)

Also, it would be nice to show a user in a second patch.

Quoted 1 line

+#[repr(align(64))]

Even if 64 bytes is common, wouldn't this depend on the system?

rust/kernel/cache_aligned.rs line 27

Quoted 2 lines
rust/kernel/cache_aligned.rs new file +2 -0
+    /// Creates a fallible initializer for `CacheAligned<T>` form a fallible
+    /// initalizer for `T`

Two typos and missing period at the end.

Also just using [CacheAligned] would probably be simpler.

Quoted 7 lines
Quoted 1 line

+ // SAFETY: by function safety requirements ptr is valid for read

    // SAFETY: By the the safety requirement, `ptr` is valid for read.
Quoted 1 line

+ try_pin_init!( CacheAligned {

This and another one is formatted differently than we usually do.

Thanks for the comments, I agree with all.

Quoted 6 lines
Quoted 2 lines

+pub use cache_aligned::CacheAligned;

It seems you want this short, in which case it should perhaps go in that case, but I think it is best to leave an extra level otherwise and let users import it.

I think I wrote this before kernel::prelude. I would put it in the prelude today instead of here, what do you think about that?

Best regards,
Andreas Hindborg

#15 lore raw
Quoted 3 lines · Andreas Hindborg

I think I wrote this before kernel::prelude. I would put it in the prelude today instead of here, what do you think about that?

I guess it depends on how much we expect it to be used.

i.e. so far we put things in the prelude if they are both general ("core") enough and frequently used.

Cheers,
Miguel

reviewer
#16 lore raw
In reply to Andreas Hindborg
Quoted 8 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

Do you mean CacheAligned<SpinLock<T>>?

Quoted 4 lines

---

Double SOB

Quoted 35 lines
---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +24 -0
new file mode 100644
index 0000000000000..9c33b8613c077
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.
+#[repr(align(64))]
+#[pin_data]
+pub struct CacheAligned<T: ?Sized> {
+    #[pin]
+    value: T,
+}
+
+impl<T> CacheAligned<T> {
+    /// Creates an initializer for `CacheAligned<T>` form an initalizer for `T`
+    pub fn new(t: impl PinInit<T>) -> impl PinInit<CacheAligned<T>> {
+        pin_init!( CacheAligned {
+            value <- t
+        })
+    }

rust/kernel/cache_aligned.rs line 34

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
+
+    /// Creates a fallible initializer for `CacheAligned<T>` form a fallible
+    /// initalizer for `T`
+    pub fn try_new(
+        t: impl PinInit<T, crate::error::Error>,
+    ) -> impl PinInit<CacheAligned<T>, crate::error::Error> {
+        try_pin_init!( CacheAligned {
+            value <- t
+        }? crate::error::Error )
+    }

You don't need two methods. You can have a single new method that's generic over error type.

rust/kernel/cache_aligned.rs line 44

Quoted 10 lines
rust/kernel/cache_aligned.rs new file +10 -0
+
+    /// Get a pointer to the contained value without creating a reference.
+    ///
+    /// # Safety
+    ///
+    /// - `ptr` must be dereferenceable.
+    pub const unsafe fn raw_get(ptr: *mut Self) -> *mut T {
+        // SAFETY: by function safety requirements `ptr` is valid for read
+        unsafe { &raw mut ((*ptr).value) }
+    }

Have you had a case where you need this? Most wrapper types shouldn't need this.

Quoted 28 lines
rust/kernel/cache_aligned.rs new file +15 -0
+}
+
+impl<T: ?Sized> core::ops::Deref for CacheAligned<T> {
+    type Target = T;
+
+    fn deref(&self) -> &T {
+        &self.value
+    }
+}
+
+impl<T: ?Sized> core::ops::DerefMut for CacheAligned<T> {
+    fn deref_mut(&mut self) -> &mut T {
+        &mut self.value
+    }
+}
rust/kernel/lib.rs +1 -0
index f812cf1200428..af6d48b078428 100644
@@ -75,6 +75,7 @@
 pub mod bug;
 #[doc(hidden)]
 pub mod build_assert;
+mod cache_aligned;
 pub mod clk;
 #[cfg(CONFIG_CONFIGFS_FS)]
 pub mod configfs;

rust/kernel/lib.rs line 160

Quoted 4 lines
rust/kernel/lib.rs +1 -0
@@ -156,6 +157,7 @@
 
 #[doc(hidden)]
 pub use bindings;
+pub use cache_aligned::CacheAligned;

Let's not expose this from top-level of kernel crate.

I have been thinking about a good namespace for these auxillary types. For my own project I would chuck them to crate::utils, but that won't be a very descriptive name.

I wonder for this and other types that tweak the memory layout, we could have a kernel::layout which contains utilities for precisely controlling the layout?

Best,
Gary

Quoted 9 lines
rust/kernel/lib.rs +0 -1
 pub use macros;
 pub use uapi;
 
 
---

base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
change-id: 20260128-cache-aligned-c4c0acf870ff

Best regards,

author reviewer
#17 lore raw
Quoted 10 lines · Gary Guo
Quoted 8 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

Quoted 8 lines

Do you mean CacheAligned<SpinLock<T>>?

Quoted 4 lines

---

Double SOB

b4 did this! It only has the @samsung SOB in my tree.

Quoted 41 lines
Quoted 41 lines
---
 rust/kernel/cache_aligned.rs | 59 ++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  2 ++
 2 files changed, 61 insertions(+)
rust/kernel/cache_aligned.rs new file +29 -0
new file mode 100644
index 0000000000000..9c33b8613c077
@@ -0,0 +1,59 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::try_pin_init;
+use pin_init::{
+    pin_data,
+    pin_init,
+    PinInit, //
+};
+
+/// Wrapper type that alings content to a 64 byte cache line.
+#[repr(align(64))]
+#[pin_data]
+pub struct CacheAligned<T: ?Sized> {
+    #[pin]
+    value: T,
+}
+
+impl<T> CacheAligned<T> {
+    /// Creates an initializer for `CacheAligned<T>` form an initalizer for `T`
+    pub fn new(t: impl PinInit<T>) -> impl PinInit<CacheAligned<T>> {
+        pin_init!( CacheAligned {
+            value <- t
+        })
+    }
+
+    /// Creates a fallible initializer for `CacheAligned<T>` form a fallible
+    /// initalizer for `T`
+    pub fn try_new(
+        t: impl PinInit<T, crate::error::Error>,
Quoted 8 lines

rust/kernel/cache_aligned.rs line 34

Quoted 5 lines
rust/kernel/cache_aligned.rs new file +5 -0
+    ) -> impl PinInit<CacheAligned<T>, crate::error::Error> {
+        try_pin_init!( CacheAligned {
+            value <- t
+        }? crate::error::Error )
+    }

You don't need two methods. You can have a single new method that's generic over error type.

Ok, cool.

Quoted 12 lines

rust/kernel/cache_aligned.rs line 44

Quoted 11 lines
rust/kernel/cache_aligned.rs new file +10 -0
+
+    /// Get a pointer to the contained value without creating a reference.
+    ///
+    /// # Safety
+    ///
+    /// - `ptr` must be dereferenceable.
+    pub const unsafe fn raw_get(ptr: *mut Self) -> *mut T {
+        // SAFETY: by function safety requirements `ptr` is valid for read
+        unsafe { &raw mut ((*ptr).value) }
+    }
Quoted 1 line

Have you had a case where you need this? Most wrapper types shouldn't need this.

I did, but I do not currently. I'll drop it.

Quoted 30 lines
Quoted 30 lines
rust/kernel/cache_aligned.rs new file +15 -0
+}
+
+impl<T: ?Sized> core::ops::Deref for CacheAligned<T> {
+    type Target = T;
+
+    fn deref(&self) -> &T {
+        &self.value
+    }
+}
+
+impl<T: ?Sized> core::ops::DerefMut for CacheAligned<T> {
+    fn deref_mut(&mut self) -> &mut T {
+        &mut self.value
+    }
+}
rust/kernel/lib.rs +1 -0
index f812cf1200428..af6d48b078428 100644
@@ -75,6 +75,7 @@
 pub mod bug;
 #[doc(hidden)]
 pub mod build_assert;
+mod cache_aligned;
 pub mod clk;
 #[cfg(CONFIG_CONFIGFS_FS)]
 pub mod configfs;
@@ -156,6 +157,7 @@
 
Quoted 5 lines

rust/kernel/lib.rs line 160

Quoted 3 lines
rust/kernel/lib.rs +1 -0
 #[doc(hidden)]
 pub use bindings;
+pub use cache_aligned::CacheAligned;

Let's not expose this from top-level of kernel crate.

Right. As I told Miguel, I would put this in the prelude.

Quoted 7 lines

I have been thinking about a good namespace for these auxillary types. For my own project I would chuck them to crate::utils, but that won't be a very descriptive name.

I wonder for this and other types that tweak the memory layout, we could have a kernel::layout which contains utilities for precisely controlling the layout?

How about kernel::mem since it is memory related?

Best regards,
Andreas Hindborg

author reviewer
#18 lore raw
In reply to Gary Guo
Quoted 12 lines · Gary Guo
Quoted 8 lines · Andreas Hindborg

CacheAligned allows to easily align values to a 64 byte boundary.

An example use case is the kernel struct spinlock. This struct is 4 bytes on x86 when lockdep is not enabled. The structure is not padded to fit a cache line. The effect of this for SpinLock is that the lock variable and the value protected by the lock might share a cache line, depending on the alignment requirements of the protected value. Wrapping the value in CacheAligned to get a SpinLock<CacheAligned<T>> solves this problem.

Do you mean CacheAligned<SpinLock<T>>?

I missed this in my original reply.

I mean SpinLock<CacheAligned<T>> as I wrote. The SpinLock will be aligned to some alignment, possibly the start of a cache line. Taking the lock involves getting exclusive ownership of the cache line and updating the lock state. When the lock is contended, the CPU core that takes the lock will loose exclusive ownership of the cache line immediately after taking the lock, as other CPU cores are trying to take the lock.

The logical thing to do after taking a lock is looking at the value protected by the lock. As the winning CPU tries to do this, it has to obtain ownership of the cache line again, assuming the T lives on the same cache line as the lock state. If we align the T to a cache line, then that line will not be contended. Only the cache line holding the lock state will be contended. And then observing the T does not incur the overhead obtaining a highly contended cache line.

Best regards,
Andreas Hindborg