mm/slab: ensure all metadata in slab object are word-aligned
Subsystems (4)
PATCH v2 1/1 mm/slab: ensure all metadata in slab object are word-aligned
When the SLAB_STORE_USER debug flag is used, any metadata placed after the original kmalloc request size (orig_size) is not properly aligned on 64-bit architectures because its type is unsigned int. When both KASAN and SLAB_STORE_USER are enabled, kasan_alloc_meta is misaligned.
Note that 64-bit architectures without HAVE_EFFICIENT_UNALIGNED_ACCESS are assumed to require 64-bit accesses to be 64-bit aligned. See HAVE_64BIT_ALIGNED_ACCESS and commit adab66b71abf ("Revert:
"ring-buffer: Remove HAVE_64BIT_ALIGNED_ACCESS"") for more details.
Because not all architectures support unaligned memory accesses, ensure that all metadata (track, orig_size, kasan_{alloc,free}_meta) in a slab object are word-aligned. struct track, kasan_{alloc,free}_meta are aligned by adding __aligned(__alignof__(unsigned long)).
For orig_size, use ALIGN(sizeof(unsigned int), sizeof(unsigned long)) to make clear that its size remains unsigned int but it must be aligned to a word boundary. On 64-bit architectures, this reserves 8 bytes for orig_size, which is acceptable since kmalloc's original request size tracking is intended for debugging rather than production use.
---
v1 -> v2:
- Added Andrey's Acked-by.
- Added references to HAVE_64BIT_ALIGNED_ACCESS and the commit that resurrected it.
- Used __alignof__() instead of sizeof(), as suggested by Pedro (off-list).
Note: either __alignof__ or sizeof() produces the exactly same mm/slub.o files, so there's no functional difference.
Thanks!
mm/kasan/kasan.h | 4 ++-- mm/slub.c | 16 +++++++++++----- 2 files changed, 13 insertions(+), 7 deletions(-)
mm/kasan/kasan.h +2 -2
@@ -265,7 +265,7 @@ struct kasan_alloc_meta {struct kasan_track alloc_track;/* Free track is stored in kasan_free_meta. */ depot_stack_handle_t aux_stack[2 ]; -}; +}__aligned(__alignof__(unsigned long)) ;struct qlist_node {struct qlist_node *next;
@@ -289,7 +289,7 @@ struct qlist_node {struct kasan_free_meta {struct qlist_node quarantine_link;struct kasan_track free_track; -}; +}__aligned(__alignof__(unsigned long)) ;#endif /* CONFIG_KASAN_GENERIC */
mm/slub.c +11 -5
@@ -344,7 +344,7 @@ struct track {int cpu;/* Was running on cpu */ int pid;/* Pid context */ unsigned long when;/* When did the operation occur */ -}; +}__aligned(__alignof__(unsigned long)) ;enum track_item {TRACK_ALLOC ,TRACK_FREE };
@@ -1196,7 +1196,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p) off +=2 *sizeof (struct track );if (slub_debug_orig_size (s)) - off +=sizeof (unsigned int ); + off +=ALIGN (sizeof (unsigned int ), __alignof__(unsigned long )); off +=kasan_metadata_size (s, false);
@@ -1392,7 +1392,8 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p) off +=2 *sizeof (struct track );if (s->flags &SLAB_KMALLOC ) - off +=sizeof (unsigned int ); + off +=ALIGN (sizeof (unsigned int ), + __alignof__(unsigned long )); } off +=kasan_metadata_size (s, false);
@@ -7820,9 +7821,14 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)*/ size +=2 *sizeof (struct track ); -/* Save the original kmalloc request size */ +/* +* Save the original kmalloc request size. +* Although the request size is an unsigned int, +* make sure that is aligned to word boundary. +*/ if (flags &SLAB_KMALLOC ) - size += sizeof(unsigned int); + size +=ALIGN (sizeof (unsigned int ), + __alignof__(unsigned long)); }#endif
Signature · 1 line
2.43.0
Quoted 10 lines · Harry Yoo
When the
SLAB_STORE_USERdebug flag is used, any metadata placed after the original kmalloc request size (orig_size) is not properly aligned on 64-bit architectures because its type is unsigned int. When both KASAN andSLAB_STORE_USERare enabled,kasan_alloc_metais misaligned.Note that 64-bit architectures without
HAVE_EFFICIENT_UNALIGNED_ACCESSare assumed to require 64-bit accesses to be 64-bit aligned. SeeHAVE_64BIT_ALIGNED_ACCESSand commitadab66b71abf("Revert:
"ring-buffer: RemoveHAVE_64BIT_ALIGNED_ACCESS"") for more details.
Quoted 6 lines
Because not all architectures support unaligned memory accesses, ensure that all metadata (track,
orig_size, kasan_{alloc,free}_meta) in a slab object are word-aligned. struct track, kasan_{alloc,free}_meta are aligned by adding__aligned(__alignof__(unsigned long)).For
orig_size, useALIGN(sizeof(unsigned int), sizeof(unsigned long))to
^ Uh, here I intended to say:
__aligneof__(unsigned long))
Quoted 24 lines
make clear that its size remains unsigned int but it must be aligned to a word boundary. On 64-bit architectures, this reserves 8 bytes for
orig_size, which is acceptable since kmalloc's original request size tracking is intended for debugging rather than production use.Fixes 6edf2576a6cc ("mm/slub: enable debugging memory wasting of kmalloc")Acked-by Andrey Konovalov <[email protected]>Signed-off-by Harry Yoo <[email protected]>---
v1 -> v2:
- Added Andrey's Acked-by.
- Added references toHAVE_64BIT_ALIGNED_ACCESSand the commit that resurrected it.
- Used__alignof__()instead ofsizeof(), as suggested by Pedro (off-list).
Note: either__alignof__orsizeof()produces the exactly samemm/slub.ofiles, so there's no functional difference.Thanks!
mm/kasan/kasan.h | 4 ++-- mm/slub.c | 16 +++++++++++----- 2 files changed, 13 insertions(+), 7 deletions(-)
Signature · 2 lines
Cheers, Harry / Hyeonggon
Andrey Ryabinin
maintainerQuoted 5 lines · Harry Yoo
When the
SLAB_STORE_USERdebug flag is used, any metadata placed after the original kmalloc request size (orig_size) is not properly aligned on 64-bit architectures because its type is unsigned int. When both KASAN andSLAB_STORE_USERare enabled,kasan_alloc_metais misaligned.
kasan_alloc_meta is properly aligned. It consists of 4 32-bit words, so the proper alignment is 32bit regardless of architecture bitness.
kasan_free_meta however requires 'unsigned long' alignment and could be misaligned if placed at 32-bit boundary on 64-bit arch
Quoted 10 lines
Note that 64-bit architectures without
HAVE_EFFICIENT_UNALIGNED_ACCESSare assumed to require 64-bit accesses to be 64-bit aligned. SeeHAVE_64BIT_ALIGNED_ACCESSand commitadab66b71abf("Revert:
"ring-buffer: RemoveHAVE_64BIT_ALIGNED_ACCESS"") for more details.Because not all architectures support unaligned memory accesses, ensure that all metadata (track,
orig_size, kasan_{alloc,free}_meta) in a slab object are word-aligned. struct track, kasan_{alloc,free}_meta are aligned by adding__aligned(__alignof__(unsigned long)).
__aligned() attribute ensures nothing. It tells compiler what alignment to expect and affects compiler controlled placement of struct in memory (e.g. stack/.bss/.data) But it can't enforce placement in dynamic memory.
Also for struct kasan_free_meta, struct track alignof(unsigned long) already dictated by C standard, so adding this __aligned() have zero effect.
And there is no reason to increase alignment requirement for kasan_alloc_meta struct.
Quoted 5 lines
For
orig_size, useALIGN(sizeof(unsigned int), sizeof(unsigned long))to make clear that its size remains unsigned int but it must be aligned to a word boundary. On 64-bit architectures, this reserves 8 bytes fororig_size, which is acceptable since kmalloc's original request size tracking is intended for debugging rather than production use.
I would suggest to use 'unsigned long' for orig_size. It changes nothing for 32-bit, and it shouldn't increase memory usage for 64-bit since we currently wasting it anyway to align next object to ARCH_KMALLOC_MINALIGN.
Quoted 11 lines · Andrey Ryabinin
Quoted 5 lines · Harry Yoo
When the
SLAB_STORE_USERdebug flag is used, any metadata placed after the original kmalloc request size (orig_size) is not properly aligned on 64-bit architectures because its type is unsigned int. When both KASAN andSLAB_STORE_USERare enabled,kasan_alloc_metais misaligned.
kasan_alloc_metais properly aligned. It consists of 4 32-bit words, so the proper alignment is 32bit regardless of architecture bitness.
Right.
Quoted 2 lines
kasan_free_metahowever requires 'unsigned long' alignment and could be misaligned if placed at 32-bit boundary on 64-bit arch
Right.
Quoted 11 lines
Quoted 10 lines
Note that 64-bit architectures without
HAVE_EFFICIENT_UNALIGNED_ACCESSare assumed to require 64-bit accesses to be 64-bit aligned. SeeHAVE_64BIT_ALIGNED_ACCESSand commitadab66b71abf("Revert:
"ring-buffer: RemoveHAVE_64BIT_ALIGNED_ACCESS"") for more details.Because not all architectures support unaligned memory accesses, ensure that all metadata (track,
orig_size, kasan_{alloc,free}_meta) in a slab object are word-aligned. struct track, kasan_{alloc,free}_meta are aligned by adding__aligned(__alignof__(unsigned long)).
Quoted 3 lines
__aligned()attribute ensures nothing. It tells compiler what alignment to expect and affects compiler controlled placement of struct in memory (e.g. stack/.bss/.data) But it can't enforce placement in dynamic memory.
Right.
Quoted 2 lines
Also for struct
kasan_free_meta, struct track alignof(unsigned long) already dictated by C standard, so adding this__aligned()have zero effect.
Right.
Quoted 1 line
And there is no reason to increase alignment requirement for
kasan_alloc_metastruct.
Right.
Quoted 9 lines
Quoted 5 lines
For
orig_size, useALIGN(sizeof(unsigned int), sizeof(unsigned long))to make clear that its size remains unsigned int but it must be aligned to a word boundary. On 64-bit architectures, this reserves 8 bytes fororig_size, which is acceptable since kmalloc's original request size tracking is intended for debugging rather than production use.I would suggest to use 'unsigned long' for
orig_size. It changes nothing for 32-bit, and it shouldn't increase memory usage for 64-bit since we currently wasting it anyway to align next object toARCH_KMALLOC_MINALIGN.
Sounds fair! Patch soon. Thanks.
Signature · 2 lines
Cheers, Harry / Hyeonggon