Align atomic storage
Subsystems (5)
v3 cover Align atomic storage
This series adds the __aligned attribute to atomic_t and atomic64_t definitions in include/asm-generic.
It also adds Kconfig options to enable a new runtime warning to help reveal misaligned atomic accesses on platforms which don't trap that.
Some people might assume scalars are aligned to 4-byte boundaries, while others might assume natural alignment. Best not to encourage such assumptions in the documentation.
Moreover, being that locks are performance sensitive, and being that atomic operations tend to involve further assumptions, there seems to be room for improvement here.
Pertinent to this discussion are the section "Memory Efficiency" in Documentation/RCU/Design/Requirements/Requirements.rst and the section "GUARANTEES" in Documentation/memory-barriers.txt
---
Changed since v2:
- Specify natural alignment for atomic64_t.
- CONFIG_DEBUG_ATOMIC checks for natural alignment again.
- New patch to add weakened alignment check.
- New patch for explicit alignment in BFP header.
---
Finn Thain (4):
documentation: Discourage alignment assumptions
bpf: Explicitly align bpf_res_spin_lock
atomic: Specify alignment for atomic_t and atomic64_t
atomic: Add option for weaker alignment check
Peter Zijlstra (1):
atomic: Add alignment check to instrumented atomic operations
.../core-api/unaligned-memory-access.rst | 7 ------- include/asm-generic/atomic64.h | 2 +- include/asm-generic/rqspinlock.h | 2 +- include/linux/instrumented.h | 12 ++++++++++++ include/linux/types.h | 2 +- kernel/bpf/rqspinlock.c | 1 - lib/Kconfig.debug | 19 +++++++++++++++++++ 7 files changed, 34 insertions(+), 11 deletions(-)
Signature · 1 line
2.49.1
RFC v3 1/5 documentation: Discourage alignment assumptions
Discourage assumptions that simply don't hold for all Linux ABIs. Exceptions to the natural alignment rule for scalar types include long long on i386 and sh.
--- Documentation/core-api/unaligned-memory-access.rst | 7 ------- 1 file changed, 7 deletions(-)
Documentation/core-api/unaligned-memory-access.rst +0 -7
@@ -40,9 +40,6 @@ The rule mentioned above forms what we refer to as natural alignment: When accessing N bytes of memory, the base memory address must be evenly divisible by N, i.e. addr % N == 0. -When writing code, assume the target architecture has natural alignment -requirements. - In reality, only a few architectures require natural alignment on all sizes of memory access. However, we must consider ALL supported architectures; writing code that satisfies natural alignment requirements is the easiest way
@@ -103,10 +100,6 @@ Therefore, for standard structure types you can always rely on the compiler to pad structures so that accesses to fields are suitably aligned (assuming you do not cast the field to a type of different length). -Similarly, you can also rely on the compiler to align variables and function -parameters to a naturally aligned scheme, based on the size of the type of -the variable. - At this point, it should be clear that accessing a single byte (u8 or char) will never cause an unaligned access, because all memory addresses are evenly divisible by one.
Signature · 1 line
2.49.1
Re: [RFC v3 1/5] documentation: Discourage alignment assumptions
Quoted 12 lines · On Wed, 08 Oct 2025 09:19:20 +1100 Finn Thain <[email protected]> wrote:
Discourage assumptions that simply don't hold for all Linux ABIs. Exceptions to the natural alignment rule for scalar types include long long on i386 and sh.
--- Documentation/core-api/unaligned-memory-access.rst | 7 ------- 1 file changed, 7 deletions(-)Documentation/core-api/unaligned-memory-access.rst +0
-0
Documentation/core-api/unaligned-memory-access.rst The rule mentioned above forms what we refer to as natural alignment: · line 44
Quoted 5 lines
Documentation/core-api/unaligned-memory-access.rst +0
-2@@ -40,9 +40,6 @@ The rule mentioned above forms what we refer to as natural alignment: When accessing N bytes of memory, the base memory address must be evenly divisible by N, i.e. addr % N == 0. -When writing code, assume the target architecture has natural alignment -requirements.
I think I'd be more explicit, perhaps:
Note that not all architectures align 64bit items on 8 byte boundaries or even 32bit items on 4 byte boundaries.
David
Quoted 15 lines
Documentation/core-api/unaligned-memory-access.rst +0
-5- In reality, only a few architectures require natural alignment on all sizes of memory access. However, we must consider ALL supported architectures; writing code that satisfies natural alignment requirements is the easiest way@@ -103,10 +100,6 @@ Therefore, for standard structure types you can always rely on the compiler to pad structures so that accesses to fields are suitably aligned (assuming you do not cast the field to a type of different length). -Similarly, you can also rely on the compiler to align variables and function -parameters to a naturally aligned scheme, based on the size of the type of -the variable. - At this point, it should be clear that accessing a single byte (u8 or char) will never cause an unaligned access, because all memory addresses are evenly divisible by one.
Re: [RFC v3 1/5] documentation: Discourage alignment assumptions
Quoted 18 lines · David Laight
Quoted 15 lines · On Wed, 08 Oct 2025 09:19:20 +1100 Finn Thain <[email protected]> wrote:
Discourage assumptions that simply don't hold for all Linux ABIs. Exceptions to the natural alignment rule for scalar types include long long on i386 and sh.
--- Documentation/core-api/unaligned-memory-access.rst | 7 ------- 1 file changed, 7 deletions(-)Documentation/core-api/unaligned-memory-access.rst +0
-0@@ -40,9 +40,6 @@ The rule mentioned above forms what we refer to as natural alignment: When accessing N bytes of memory, the base memory address must be evenly divisible by N, i.e. addr % N == 0.
Quoted 7 lines
Documentation/core-api/unaligned-memory-access.rst The rule mentioned above forms what we refer to as natural alignment: · line 44
Quoted 2 lines
Documentation/core-api/unaligned-memory-access.rst +0
-2-When writing code, assume the target architecture has natural alignment -requirements.I think I'd be more explicit, perhaps:
Note that not all architectures align 64bit items on 8 byte boundaries or even 32bit items on 4 byte boundaries.
That's what the next para is alluding to...
Quoted 4 lines
Quoted 4 lines
In reality, only a few architectures require natural alignment on all sizes of memory access. However, we must consider ALL supported architectures; writing code that satisfies natural alignment requirements is the easiest way to achieve full portability.
How about this?
"In reality, only a few architectures require natural alignment for all sizes of memory access. That is, not all architectures need 64-bit values to be aligned on 8-byte boundaries and 32-bit values on 4-byte boundaries. However, when writing code intended to achieve full portability, we must consider all supported architectures."
Quoted 13 lines
Quoted 11 lines
Documentation/core-api/unaligned-memory-access.rst +0
-4@@ -103,10 +100,6 @@ Therefore, for standard structure types you can always rely on the compiler to pad structures so that accesses to fields are suitably aligned (assuming you do not cast the field to a type of different length). -Similarly, you can also rely on the compiler to align variables and function -parameters to a naturally aligned scheme, based on the size of the type of -the variable. - At this point, it should be clear that accessing a single byte (u8 or char) will never cause an unaligned access, because all memory addresses are evenly divisible by one.
Re: [RFC v3 1/5] documentation: Discourage alignment assumptions
Quoted 35 lines · On Wed, 15 Oct 2025 18:40:39 +1100 (AEDT) Finn Thain <[email protected]>…
Quoted 18 lines · David Laight
Quoted 15 lines · On Wed, 08 Oct 2025 09:19:20 +1100 Finn Thain <[email protected]> wrote:
Discourage assumptions that simply don't hold for all Linux ABIs. Exceptions to the natural alignment rule for scalar types include long long on i386 and sh.
--- Documentation/core-api/unaligned-memory-access.rst | 7 ------- 1 file changed, 7 deletions(-)Documentation/core-api/unaligned-memory-access.rst +0
-0@@ -40,9 +40,6 @@ The rule mentioned above forms what we refer to as natural alignment: When accessing N bytes of memory, the base memory address must be evenly divisible by N, i.e. addr % N == 0.Quoted 7 lines
Documentation/core-api/unaligned-memory-access.rst The rule mentioned above forms what we refer to as natural alignment: · line 44
Quoted 2 lines
Documentation/core-api/unaligned-memory-access.rst +0
-2-When writing code, assume the target architecture has natural alignment -requirements.I think I'd be more explicit, perhaps:
Note that not all architectures align 64bit items on 8 byte boundaries or even 32bit items on 4 byte boundaries.That's what the next para is alluding to...
Quoted 4 lines
Quoted 4 lines
In reality, only a few architectures require natural alignment on all sizes of memory access. However, we must consider ALL supported architectures; writing code that satisfies natural alignment requirements is the easiest way to achieve full portability.
Quoted 7 lines
How about this?
"In reality, only a few architectures require natural alignment for all sizes of memory access. That is, not all architectures need 64-bit values to be aligned on 8-byte boundaries and 32-bit values on 4-byte boundaries. However, when writing code intended to achieve full portability, we must consider all supported architectures."
There are several separate alignments:
- The alignment the cpu needs, for most x86 instructions this is 1 byte [1]. Many RISC cpu require 'word' alignment (for some definition of 'word'). A problematic case is data that crosses page boundaries.
- The alignment the compiler uses for structure members; returned by _Alignof(). m68k only 16bit aligns 32bit values.
- The 'preferred' alignment returned by __alignof__().
32bit x86 returns 8 for 64bit types even though the ABI only 4-byte aligns them.
- The 'natural' alignment based on the size of the item.
I'd guess that 'complex double' (if supported) may only be 8 byte aligned.
What normally matters is the ABI alignment for structure members. If you mark anything 'packed' the compiler will generate shifts and masks (etc) to get working code.
Taking the address of an item in a packed structure generates a warning for very good reason.
[1] I've fallen foul of gcc deciding to 'vectorise' a loop and then having it crash because the buffer address was misaligned.
Nasty because the code worked in initial testing and I expected the loop (32bit adds of a buffer) to work fine even when misaligned.
David
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Documentation/core-api/unaligned-memory-access.rst +0
-4@@ -103,10 +100,6 @@ Therefore, for standard structure types you can always rely on the compiler to pad structures so that accesses to fields are suitably aligned (assuming you do not cast the field to a type of different length). -Similarly, you can also rely on the compiler to align variables and function -parameters to a naturally aligned scheme, based on the size of the type of -the variable. - At this point, it should be clear that accessing a single byte (u8 or char) will never cause an unaligned access, because all memory addresses are evenly divisible by one.
Re: [RFC v3 1/5] documentation: Discourage alignment assumptions
Quoted 12 lines · David Laight
There are several separate alignments:
- The alignment the cpu needs, for most x86 instructions this is 1 byte [1]. Many RISC cpu require 'word' alignment (for some definition of 'word'). A problematic case is data that crosses page boundaries.
- The alignment the compiler uses for structure members; returned by_Alignof(). m68k only 16bit aligns 32bit values.
- The 'preferred' alignment returned by__alignof__().
32bit x86 returns 8 for 64bit types even though the ABI only 4-byte aligns them.
- The 'natural' alignment based on the size of the item.
I'd guess that 'complex double' (if supported) may only be 8 byte aligned.
Those distinctions could be useful in a discussion about memory efficiency. But this document is concerned with avoiding a performance penalty -- it's entirely unconcerned with over-alignment and memory waste. Hence, "aligned" is used as shorthand for "naturally aligned".
The ambiguity in this document (and my proposed change) stems from using the word architecture to cover ABI, platform, CPU, ISA etc. I can improve upon that.
Quoted 6 lines
What normally matters is the ABI alignment for structure members. If you mark anything 'packed' the compiler will generate shifts and masks (etc) to get working code.
Taking the address of an item in a packed structure generates a warning for very good reason.
I believe the problem with 'packed' is already covered in this document.
Quoted 5 lines
[1] I've fallen foul of gcc deciding to 'vectorise' a loop and then having it crash because the buffer address was misaligned.
Nasty because the code worked in initial testing and I expected the loop (32bit adds of a buffer) to work fine even when misaligned.
I think that pitfall is already discussed also, along with a remedy.
There is also this,
... for standard structure types you can always rely on the compiler to pad structures so that accesses to fields are suitably aligned (assuming you do not cast the field to a type of different length).
So it seems to be fairly comprehensive but I may be missing something (?)
Arnd Bergmann
maintainerRe: [RFC v3 1/5] documentation: Discourage alignment assumptions
In reply to Finn ThainQuoted 4 lines · Finn Thain
Discourage assumptions that simply don't hold for all Linux ABIs. Exceptions to the natural alignment rule for scalar types include long long on i386 and sh.
---
I think both of the paragraphs you remove are still correct and I would not remove them:
Documentation/core-api/unaligned-memory-access.rst The rule mentioned above forms what we refer to as natural alignment: · line 45
Quoted 12 lines
Documentation/core-api/unaligned-memory-access.rst | 7 ------- 1 file changed, 7 deletions(-)Documentation/core-api/unaligned-memory-access.rst +0
-3-When writing code, assume the target architecture has natural alignment -requirements. -
It is clearly important to not intentionally misalign variables because that breaks on hardware that requires aligned data.
Assuming natural alignment is the safe choice here, but you could change 'architecture' to 'hardware' here if you think that is otherwise ambiguous.
Documentation/core-api/unaligned-memory-access.rst Therefore, for standard structure types you can always rely on the compiler · line 108
Quoted 3 lines
Documentation/core-api/unaligned-memory-access.rst +0
-3-Similarly, you can also rely on the compiler to align variables and function -parameters to a naturally aligned scheme, based on the size of the type of -the variable.
This also seems to refer to something else: even on m68k and i386, scalar stack and .data variables have natural alignment even though the ABI does not require that.
It's probably a good idea to list the specific exceptions to the struct layout rules in the previous paragraph, e.g.
[
Fortunately, the compiler understands the alignment constraints, so in the above case it would insert 2 bytes of padding in between field1 and field2. Therefore, for standard structure types you can always rely on the compiler
-to pad structures so that accesses to fields are suitably aligned (assuming -you do not cast the field to a type of different length).
+to pad structures so that accesses to fields are suitably aligned for +the CPU hardware.
+On all 64-bit architectures, this means that all scalar struct members +are naturally aligned. However, some 32-bit ABIs including i386 +only align 64-bit members on 32-bit offsets, and m68k uses at most +16-bit alignment.
]
Arnd
Re: [RFC v3 1/5] documentation: Discourage alignment assumptions
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines · Finn Thain
Discourage assumptions that simply don't hold for all Linux ABIs. Exceptions to the natural alignment rule for scalar types include long long on i386 and sh.
---I think both of the paragraphs you remove are still correct and I would not remove them:
Yes -- correct in some obscure sense, but misleading at face value. Hence this patch.
Quoted 16 lines
Documentation/core-api/unaligned-memory-access.rst The rule mentioned above forms what we refer to as natural alignment: · line 45
Quoted 12 lines
Documentation/core-api/unaligned-memory-access.rst | 7 ------- 1 file changed, 7 deletions(-)Documentation/core-api/unaligned-memory-access.rst +0
-3-When writing code, assume the target architecture has natural alignment -requirements. -It is clearly important to not intentionally misalign variables because that breaks on hardware that requires aligned data.
Quoted 4 lines
Assuming natural alignment is the safe choice here, but you could change 'architecture' to 'hardware' here if you think that is otherwise ambiguous.
Do you know of any hardware that has "natural alignment requirements" in the completely unqualified sense in which that the claim is made? i.e. considering all of its native vector and scalar types.
That's a rhetorical question. I'm not trying to provide an exhaustive and up-to-date list of platform quirks. With this patch, I'm merely trying to discourage faulty assumptions.
Quoted 8 lines
Documentation/core-api/unaligned-memory-access.rst Therefore, for standard structure types you can always rely on the compiler · line 108
Quoted 3 lines
Documentation/core-api/unaligned-memory-access.rst +0
-3-Similarly, you can also rely on the compiler to align variables and function -parameters to a naturally aligned scheme, based on the size of the type of -the variable.This also seems to refer to something else: even on m68k and i386, scalar stack and .data variables have natural alignment even though the ABI does not require that.
Then it is doubly misleading.
There is value in explaining what the compiler can and cannot be relied upon to deliver, but I think the myfunc() example already serves that purpose. Don't you agree?
Review
Quoted 12 lines
It's probably a good idea to list the specific exceptions to the struct layout rules in the previous paragraph, e.g.
[
Fortunately, the compiler understands the alignment constraints, so in the above case it would insert 2 bytes of padding in between field1 and field2. Therefore, for standard structure types you can always rely on the compiler(quoted diff) +3
-2-to pad structures so that accesses to fields are suitably aligned (assuming -you do not cast the field to a type of different length). +to pad structures so that accesses to fields are suitably aligned for +the CPU hardware. +On all 64-bit architectures,
The poor reader: "I wonder what '64-bit architecture' means in this context..."
Review
Quoted 4 lines
this means that all scalar struct members
(quoted diff) +3
-0+are naturally aligned. However, some 32-bit ABIs including i386 +only align 64-bit members on 32-bit offsets, and m68k uses at most +16-bit alignment. ]
"... oh, okay, so anything that naturally aligns a 64-bit word is a "64-bit architecture". I get it. Oh, hang-on, that doesn't make sense... Why would any 32-bit architecture be expected to naturally align 64-bit members? What is he talking about? CONFIG_64BIT maybe??"
Moreover, you have digressed into a discussion of the ABI. The aim of this document is to avoid the performance penalty for unaligned accesses.
Whereas, ABI traits would seem to be relevant to a discussion about memory efficiency, like I said to David.
RFC v3 2/5 bpf: Explicitly align bpf_res_spin_lock
In reply to Finn ThainAlign bpf_res_spin_lock to avoid a BUILD_BUG_ON() when the alignment changes, as it will do on m68k when, in a subsequent patch, the minimum alignment of the atomic_t member of struct rqspinlock gets increased. Drop the BUILD_BUG_ON() as it is now redundant.
--- include/asm-generic/rqspinlock.h | 2 +- kernel/bpf/rqspinlock.c | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
include/asm-generic/rqspinlock.h +1 -1
@@ -28,7 +28,7 @@ struct rqspinlock {*/ struct bpf_res_spin_lock {u32 val; -}; +}__aligned(__alignof__(struct rqspinlock)) ;struct qspinlock ;#ifdef CONFIG_QUEUED_SPINLOCKS
kernel/bpf/rqspinlock.c +0 -1
@@ -671,7 +671,6 @@ __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock)int ret;BUILD_BUG_ON (sizeof (rqspinlock_t) !=sizeof (struct bpf_res_spin_lock )); -BUILD_BUG_ON (__alignof__(rqspinlock_t ) != __alignof__(struct bpf_res_spin_lock ));preempt_disable (); ret =res_spin_lock ((rqspinlock_t *)lock);
Signature · 1 line
2.49.1
Re: [RFC v3 2/5] bpf: Explicitly align bpf_res_spin_lock
Quoted 25 lines · Finn Thain
Align
bpf_res_spin_lockto avoid aBUILD_BUG_ON()when the alignment changes, as it will do on m68k when, in a subsequent patch, the minimum alignment of theatomic_tmember of struct rqspinlock gets increased. Drop theBUILD_BUG_ON()as it is now redundant.Cc Martin KaFai Lau <[email protected]>Cc Eduard Zingerman <[email protected]>Cc Song Liu <[email protected]>Cc Yonghong Song <[email protected]>Cc John Fastabend <[email protected]>Cc KP Singh <[email protected]>Cc Stanislav Fomichev <[email protected]>Cc Hao Luo <[email protected]>Cc Jiri Olsa <[email protected]>--- include/asm-generic/rqspinlock.h | 2 +- kernel/bpf/rqspinlock.c | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)include/asm-generic/rqspinlock.h +0
-0
include/asm-generic/rqspinlock.h struct rqspinlock { · line 31
Quoted 5 lines
include/asm-generic/rqspinlock.h +1
-1@@ -28,7 +28,7 @@ struct rqspinlock {*/ struct bpf_res_spin_lock {u32 val; -}; +}__aligned(__alignof__(struct rqspinlock)) ;
I don't follow.
In the next patch you do:
typedef struct {
- int counter;
+ int __aligned(sizeof(int)) counter;
} atomic_t;
so it was 4 and still 4 ?
Are you saying 'int' on m68k is not 4 byte aligned by default, so you have to force 4 byte align?
kernel/bpf/rqspinlock.c __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock) · line 674
Quoted 11 lines
include/asm-generic/rqspinlock.h +0
-0struct qspinlock ;#ifdef CONFIG_QUEUED_SPINLOCKS kernel/bpf/rqspinlock.c +0
-1@@ -671,7 +671,6 @@ __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock)int ret;BUILD_BUG_ON (sizeof (rqspinlock_t) !=sizeof (struct bpf_res_spin_lock )); -BUILD_BUG_ON (__alignof__(rqspinlock_t ) != __alignof__(struct bpf_res_spin_lock ));
Why delete it? Didn't you make them equal in the above hunk?
Re: [RFC v3 2/5] bpf: Explicitly align bpf_res_spin_lock
Quoted 35 lines · Alexei Starovoitov
Quoted 25 lines · Finn Thain
Align
bpf_res_spin_lockto avoid aBUILD_BUG_ON()when the alignment changes, as it will do on m68k when, in a subsequent patch, the minimum alignment of theatomic_tmember of struct rqspinlock gets increased. Drop theBUILD_BUG_ON()as it is now redundant.Cc Martin KaFai Lau <[email protected]>Cc Eduard Zingerman <[email protected]>Cc Song Liu <[email protected]>Cc Yonghong Song <[email protected]>Cc John Fastabend <[email protected]>Cc KP Singh <[email protected]>Cc Stanislav Fomichev <[email protected]>Cc Hao Luo <[email protected]>Cc Jiri Olsa <[email protected]>--- include/asm-generic/rqspinlock.h | 2 +- kernel/bpf/rqspinlock.c | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)include/asm-generic/rqspinlock.h +0
-0include/asm-generic/rqspinlock.h struct rqspinlock { · line 31
Quoted 5 lines
include/asm-generic/rqspinlock.h +1
-1@@ -28,7 +28,7 @@ struct rqspinlock {*/ struct bpf_res_spin_lock {u32 val; -}; +}__aligned(__alignof__(struct rqspinlock)) ;I don't follow.
In the next patch you do:
Quoted 9 lines
typedef struct {
- int counter;
+ int __aligned(sizeof(int)) counter;
}atomic_t;so it was 4 and still 4 ?
Are you saying 'int' on m68k is not 4 byte aligned by default, so you have to force 4 byte align?
Right. __alignof(int) == 2 on m68k.
Quoted 9 lines
Quoted 9 lines
include/asm-generic/rqspinlock.h +0
-0struct qspinlock ;#ifdef CONFIG_QUEUED_SPINLOCKS kernel/bpf/rqspinlock.c +0
-0@@ -671,7 +671,6 @@ __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock)int ret;
Quoted 5 lines
kernel/bpf/rqspinlock.c __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock) · line 674
Quoted 2 lines
kernel/bpf/rqspinlock.c +0
-1BUILD_BUG_ON (sizeof (rqspinlock_t) !=sizeof (struct bpf_res_spin_lock )); -BUILD_BUG_ON (__alignof__(rqspinlock_t ) != __alignof__(struct bpf_res_spin_lock ));Why delete it? Didn't you make them equal in the above hunk?
I deleted it because it's tautological. I think "do not repeat yourself" applies here.
Peter Zijlstra
maintainerRe: [RFC v3 2/5] bpf: Explicitly align bpf_res_spin_lock
In reply to Alexei StarovoitovQuoted 2 lines · Alexei Starovoitov
Are you saying 'int' on m68k is not 4 byte aligned by default, so you have to force 4 byte align?
This; m68k has u16 alignment, just to keep life interesting I suppose :-)
Re: [RFC v3 2/5] bpf: Explicitly align bpf_res_spin_lock
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Alexei Starovoitov
Are you saying 'int' on m68k is not 4 byte aligned by default, so you have to force 4 byte align?
This; m68k has u16 alignment, just to keep life interesting I suppose :-)
It's not "interesting". It adds burden to the rest of the kernel for this architectural quirk.
Linus put the foot down for big-endian on arm64 and riscv. We should do the same here.
x86 uses -mcmodel=kernel for 64-bit and -mregparm=3 for 32-bit. m68k can do the same.
They can adjust the compiler to make 'int' 4 byte aligned under some compiler flag. The kernel is built standalone, so it doesn't have to conform to native calling convention or anything else.
Arnd Bergmann
maintainerRe: [RFC v3 2/5] bpf: Explicitly align bpf_res_spin_lock
Quoted 10 lines · Alexei Starovoitov
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Alexei Starovoitov
Are you saying 'int' on m68k is not 4 byte aligned by default, so you have to force 4 byte align?
This; m68k has u16 alignment, just to keep life interesting I suppose :-)
Quoted 9 lines
It's not "interesting". It adds burden to the rest of the kernel for this architectural quirk.
Linus put the foot down for big-endian on arm64 and riscv. We should do the same here.
x86 uses -mcmodel=kernel for 64-bit and -mregparm=3 for 32-bit. m68k can do the same.
They can adjust the compiler to make 'int' 4 byte aligned under some compiler flag. The kernel is built standalone, so it doesn't have to conform to native calling convention or anything else.
I agree that building the kernel with -malign-int makes a lot of sense here, there is even a project to rebuild the entire user space with the same flag.
However, changing either the kernel or userspace to build with -malign-int also has its cost, since for ABI compatibility reasons any include/uapi/*/*.h header that defines a structure with a misaligned word needs a custom annotation in order to still define the layout to be the same as before, and the annotations do complicate the common headers.
See
https://lore.kernel.org/all/[email protected]/
for a list of structures that likely need to be annotated, and the thread around it for more of the nasty details that make this nontrivial.
Arnd
Alexei Starovoitov
Acked-byRe: [RFC v3 2/5] bpf: Explicitly align bpf_res_spin_lock
Quoted 33 lines · Arnd Bergmann
Quoted 10 lines · Alexei Starovoitov
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Alexei Starovoitov
Are you saying 'int' on m68k is not 4 byte aligned by default, so you have to force 4 byte align?
This; m68k has u16 alignment, just to keep life interesting I suppose :-)
Quoted 9 lines
It's not "interesting". It adds burden to the rest of the kernel for this architectural quirk.
Linus put the foot down for big-endian on arm64 and riscv. We should do the same here.
x86 uses -mcmodel=kernel for 64-bit and -mregparm=3 for 32-bit. m68k can do the same.
They can adjust the compiler to make 'int' 4 byte aligned under some compiler flag. The kernel is built standalone, so it doesn't have to conform to native calling convention or anything else.I agree that building the kernel with -malign-int makes a lot of sense here, there is even a project to rebuild the entire user space with the same flag.
However, changing either the kernel or userspace to build with -malign-int also has its cost, since for ABI compatibility reasons any
include/uapi/*/*.hheader that defines a structure with a misaligned word needs a custom annotation in order to still define the layout to be the same as before, and the annotations do complicate the common headers.
Quoted 5 lines
See
https://lore.kernel.org/all/[email protected]/
for a list of structures that likely need to be annotated, and the thread around it for more of the nasty details that make this nontrivial.
I see. So this is a lesser evil.
for the patch then.
RFC v3 3/5 atomic: Specify alignment for atomic_t and atomic64_t
In reply to Finn ThainSome recent commits incorrectly assumed 4-byte alignment of locks. That assumption fails on Linux/m68k (and, interestingly, would have failed on Linux/cris also). Specify the minimum alignment of atomic variables for fewer surprises and (hopefully) better performance.
On an m68k system with 14 MB of RAM, this patch reduces the available memory by a couple of percent. On a 64 MB system, the cost is under 1% but still significant. I don't know whether there is sufficient performance gain to justify the memory cost; it still has to be measured.
---
Changed since v2:
- Specify natural alignment for atomic64_t.
Changed since v1:
- atomic64_t now gets an __aligned attribute too.
- The 'Fixes' tag has been dropped because Lance sent a different fix for commit e711faaafbe5 ("hung_task: replace blocker_mutex with encoded blocker") that's suitable for -stable.
--- include/asm-generic/atomic64.h | 2 +- include/linux/types.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/asm-generic/atomic64.h +1 -1
@@ -10,7 +10,7 @@#include <linux/types.h> typedef struct { -s64 counter; +s64 __aligned(sizeof(s64)) counter; }atomic64_t ;#define ATOMIC64_INIT (i) { (i) }
include/linux/types.h +1 -1
@@ -179,7 +179,7 @@ typedef phys_addr_t resource_size_t;typedef unsigned long irq_hw_number_t ;typedef struct { -int counter; +int __aligned(sizeof(int)) counter; }atomic_t ;#define ATOMIC_INIT (i) { (i) }
Signature · 1 line
2.49.1
RFC v3 4/5 atomic: Add alignment check to instrumented atomic operations
In reply to Finn ThainFrom: Peter Zijlstra <[email protected]>
Add a Kconfig option for debug builds which logs a warning when an instrumented atomic operation takes place that's misaligned. Some platforms don't trap for this.
---
Changed since v2:
- Always check for natural alignment.
---
To make this useful on those architectures that don't naturally align scalars (x86-32, m68k and sh come to mind), please also use "[PATCH] atomic: skip alignment check for try_cmpxchg() old arg"
https://lore.kernel.org/all/[email protected]/
--- include/linux/instrumented.h | 4 ++++ lib/Kconfig.debug | 10 ++++++++++ 2 files changed, 14 insertions(+)
include/linux/instrumented.h +4 -0
@@ -7,6 +7,7 @@#ifndef _LINUX_INSTRUMENTED_H#define _LINUX_INSTRUMENTED_H +#include <linux/bug.h> #include <linux/compiler.h> #include <linux/kasan-checks.h> #include <linux/kcsan-checks.h>
@@ -67,6 +68,7 @@ static __always_inline void instrument_atomic_read(const volatile void *v, size_ {kasan_check_read (v, size);kcsan_check_atomic_read (v, size); +WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ATOMIC ) && ((unsigned long )v & (size -1 ))); }/**
@@ -81,6 +83,7 @@ static __always_inline void instrument_atomic_write(const volatile void *v, size {kasan_check_write (v, size);kcsan_check_atomic_write (v, size); +WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ATOMIC ) && ((unsigned long )v & (size -1 ))); }/**
@@ -95,6 +98,7 @@ static __always_inline void instrument_atomic_read_write(const volatile void *v, {kasan_check_write (v, size);kcsan_check_atomic_read_write (v, size); +WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ATOMIC ) && ((unsigned long )v & (size -1 ))); }/**
lib/Kconfig.debug +10 -0
@@ -1363,6 +1363,16 @@ config DEBUG_PREEMPT depending on workload as it triggers debugging routines for each this_cpu operation. It should only be used for debugging purposes. +config DEBUG_ATOMIC + bool "Debug atomic variables" + depends on DEBUG_KERNEL + help + If you say Y here then the kernel will add a runtime alignment check + to atomic accesses. Useful for architectures that do not have trap on + mis-aligned access. + + This option has potentially significant overhead. + menu "Lock Debugging (spinlocks, mutexes, etc...)" config LOCK_DEBUGGING_SUPPORT
Signature · 1 line
2.49.1
RFC v3 5/5 atomic: Add option for weaker alignment check
In reply to Finn ThainAdd a new Kconfig symbol to make CONFIG_DEBUG_ATOMIC more useful on those architectures which do not align dynamic allocations to 8-byte boundaries. Without this, CONFIG_DEBUG_ATOMIC produces excessive WARN splats.
--- include/linux/instrumented.h | 14 +++++++++++--- lib/Kconfig.debug | 9 +++++++++ 2 files changed, 20 insertions(+), 3 deletions(-)
include/linux/instrumented.h +11 -3
@@ -8,11 +8,13 @@#define _LINUX_INSTRUMENTED_H#include <linux/bug.h> +#include <linux/cache.h> #include <linux/compiler.h> #include <linux/kasan-checks.h> #include <linux/kcsan-checks.h> #include <linux/kmsan-checks.h> #include <linux/types.h> +#include <vdso/align.h> /** * instrument_read - instrument regular read access
@@ -68,7 +70,9 @@ static __always_inline void instrument_atomic_read(const volatile void *v, size_ {kasan_check_read (v, size);kcsan_check_atomic_read (v, size); -WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ATOMIC ) && ((unsigned long )v & (size -1 ))); +if (IS_ENABLED (CONFIG_DEBUG_ATOMIC )) +WARN_ON_ONCE (v !=PTR_ALIGN (v,IS_ENABLED (CONFIG_DEBUG_ATOMIC_LARGEST_ALIGN ) ? + __LARGEST_ALIGN : size)); }/**
@@ -83,7 +87,9 @@ static __always_inline void instrument_atomic_write(const volatile void *v, size {kasan_check_write (v, size);kcsan_check_atomic_write (v, size); -WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ATOMIC ) && ((unsigned long )v & (size -1 ))); +if (IS_ENABLED (CONFIG_DEBUG_ATOMIC )) +WARN_ON_ONCE (v !=PTR_ALIGN (v,IS_ENABLED (CONFIG_DEBUG_ATOMIC_LARGEST_ALIGN ) ? + __LARGEST_ALIGN : size)); }/**
@@ -98,7 +104,9 @@ static __always_inline void instrument_atomic_read_write(const volatile void *v, {kasan_check_write (v, size);kcsan_check_atomic_read_write (v, size); -WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ATOMIC ) && ((unsigned long )v & (size -1 ))); +if (IS_ENABLED (CONFIG_DEBUG_ATOMIC )) +WARN_ON_ONCE (v !=PTR_ALIGN (v,IS_ENABLED (CONFIG_DEBUG_ATOMIC_LARGEST_ALIGN ) ? + __LARGEST_ALIGN : size)); }/**
lib/Kconfig.debug +9 -0
@@ -1373,6 +1373,15 @@ config DEBUG_ATOMIC This option has potentially significant overhead. +config DEBUG_ATOMIC_LARGEST_ALIGN + bool "Check alignment only up to __LARGEST_ALIGN" + depends on DEBUG_ATOMIC + help + Say Y here if you require natural alignment of atomic accesses + only up to long word boundaries. That is, char, short, int and long + will be checked for natural alignment but anything larger checked + only for alignment with a long word boundary. + menu "Lock Debugging (spinlocks, mutexes, etc...)" config LOCK_DEBUGGING_SUPPORT
Signature · 1 line
2.49.1