Separate struct slab from struct page
Subsystems (15)
v2 cover Separate struct slab from struct page
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
The plan: as my SLUB PREEMPT_RT series in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/
Changes from v1/RFC:
https://lore.kernel.org/all/[email protected]/
- Added virt_to_folio() and folio_address() in the new Patch 1.
- Addressed feedback from Andrey Konovalov and Matthew Wilcox (Thanks!)
- Added Tested-by: Marco Elver for the KFENCE parts (Thanks!)
Previous version from Matthew Wilcox:
https://lore.kernel.org/all/20211004134650.4031813-1-willy-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org/
LWN coverage of the above:
https://lwn.net/Articles/871982/
This is originally an offshoot of the folio work by Matthew. One of the more complex parts of the struct page definition are the parts used by the slab allocators. It would be good for the MM in general if struct slab were its own data type, and it also helps to prevent tail pages from slipping in anywhere. As Matthew requested in his proof of concept series, I have taken over the development of this series, so it's a mix of patches from him (often modified by me) and my own.
One big difference is the use of coccinelle to perform the relatively trivial parts of the conversions automatically and at once, instead of a larger number of smaller incremental reviewable steps. Thanks to Julia Lawall and Luis Chamberlain for all their help!
Another notable difference is (based also on review feedback) I don't represent with a struct slab the large kmalloc allocations which are not really a slab, but use page allocator directly. When going from an object address to a struct slab, the code tests first folio slab flag, and only if it's set it converts to struct slab. This makes the struct slab type stronger.
Finally, although Matthew's version didn't use any of the folio work, the initial support has been merged meanwhile so my version builds on top of it where appropriate. This eliminates some of the redundant compound_head() being performed e.g. when testing the slab flag.
To sum up, after this series, struct page fields used by slab allocators are moved from struct page to a new struct slab, that uses the same physical storage. The availability of the fields is further distinguished by the selected slab allocator implementation. The advantages include:
- Similar to folios, if the slab is of order > 0, struct slab always is guaranteed to be the head page. Additionally it's guaranteed to be an actual slab page, not a large kmalloc. This removes uncertainty and potential for bugs.
- It's not possible to accidentally use fields of the slab implementation that's not configured.
- Other subsystems cannot use slab's fields in struct page anymore (some existing non-slab usages had to be adjusted in this series), so slab implementations have more freedom in rearranging them in the struct slab.
Matthew Wilcox (Oracle) (16):
mm: Split slab into its own type
mm: Add account_slab() and unaccount_slab()
mm: Convert virt_to_cache() to use struct slab
mm: Convert __ksize() to struct slab
mm: Use struct slab in kmem_obj_info()
mm: Convert check_heap_object() to use struct slab mm/slub: Convert detached_freelist to use a struct slab mm/slub: Convert kfree() to use a struct slab mm/slub: Convert print_page_info() to print_slab_info() mm/slub: Convert pfmemalloc_match() to take a struct slab mm/slob: Convert SLOB to use struct slab mm/kasan: Convert to struct folio and struct slab
zsmalloc: Stop using slab fields in struct page
bootmem: Use page->index instead of page->freelist
iommu: Use put_pages_list
mm: Remove slab from struct page
Vlastimil Babka (17):
mm: add virt_to_folio() and folio_address()
mm/slab: Dissolve slab_map_pages() in its caller
mm/slub: Make object_err() static
mm/slub: Convert __slab_lock() and __slab_unlock() to struct slab mm/slub: Convert alloc_slab_page() to return a struct slab mm/slub: Convert __free_slab() to use struct slab mm/slub: Convert most struct page to struct slab by spatch mm/slub: Finish struct page to struct slab conversion mm/slab: Convert kmem_getpages() and kmem_freepages() to struct slab mm/slab: Convert most struct page to struct slab by spatch mm/slab: Finish struct page to struct slab conversion
mm: Convert struct page to struct slab in functions used by other
subsystems
mm/memcg: Convert slab objcgs from struct page to struct slab mm/kfence: Convert kfence_guarded_alloc() to struct slab mm/sl*b: Differentiate struct slab fields by sl*b implementations mm/slub: Simplify struct slab slabs field definition mm/slub: Define struct slab fields for CONFIG_SLUB_CPU_PARTIAL only
when enabled
arch/x86/mm/init_64.c | 2 +- drivers/iommu/amd/io_pgtable.c | 59 +- drivers/iommu/dma-iommu.c | 11 +- drivers/iommu/intel/iommu.c | 89 +-- include/linux/bootmem_info.h | 2 +- include/linux/iommu.h | 3 +- include/linux/kasan.h | 9 +- include/linux/memcontrol.h | 48 -- include/linux/mm.h | 12 + include/linux/mm_types.h | 38 +- include/linux/page-flags.h | 37 - include/linux/slab.h | 8 - include/linux/slab_def.h | 16 +- include/linux/slub_def.h | 29 +- mm/bootmem_info.c | 7 +- mm/kasan/common.c | 27 +- mm/kasan/generic.c | 8 +- mm/kasan/kasan.h | 1 + mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 13 +- mm/kasan/report_tags.c | 10 +- mm/kfence/core.c | 17 +- mm/kfence/kfence_test.c | 6 +- mm/memcontrol.c | 43 +- mm/slab.c | 455 ++++++------- mm/slab.h | 322 ++++++++- mm/slab_common.c | 8 +- mm/slob.c | 46 +- mm/slub.c | 1164 ++++++++++++++++---------------- mm/sparse.c | 2 +- mm/usercopy.c | 13 +- mm/zsmalloc.c | 18 +- 32 files changed, 1317 insertions(+), 1208 deletions(-)
Signature · 1 line
2.33.1
PATCH v2 1/33 mm: add virt_to_folio() and folio_address()
These two wrappers around their respective struct page variants will be useful in the following patches.
--- include/linux/mm.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)
include/linux/mm.h +12 -0
@@ -863,6 +863,13 @@ static inline struct page *virt_to_head_page(const void *x)return compound_head (page); } +static inline struct folio *virt_to_folio (const void *x) +{ +struct page *page =virt_to_page (x); + +return page_folio (page); +} +void __put_page (struct page *page);void put_pages_list (struct list_head *pages);
@@ -1753,6 +1760,11 @@ void page_address_init(void);#define page_address_init () do { } while(0)#endif +static inline void *folio_address (const struct folio *folio) +{ +return page_address (&folio->page); +} +extern void *page_rmapping (struct page *page);extern struct anon_vma *page_anon_vma (struct page *page);extern pgoff_t __page_file_index (struct page *page);
Signature · 1 line
2.33.1
Re: [PATCH v2 01/33] mm: add virt_to_folio() and folio_address()
Quoted 4 lines · Vlastimil Babka
These two wrappers around their respective struct page variants will be useful in the following patches.
Signed-off-by Vlastimil Babka <[email protected]>
Slight reviewer preference: introduce helpers when they're first used, as it's a little easier to understand the application without having to chase new folio_address() -> new slab_address() -> first user of slab_address() through the patchset.
Re: [PATCH v2 01/33] mm: add virt_to_folio() and folio_address()
Quoted 7 lines · Johannes Weiner
Quoted 4 lines · Vlastimil Babka
These two wrappers around their respective struct page variants will be useful in the following patches.
Signed-off-by Vlastimil Babka <[email protected]>Acked-by Johannes Weiner <[email protected]>
Thanks!
Quoted 4 lines
Slight reviewer preference: introduce helpers when they're first used, as it's a little easier to understand the application without having to chase new
folio_address()-> newslab_address()-> first user ofslab_address()through the patchset.
Yeah would normally do that but with the current folio work going on there was a chance the helpers would be useful to somebody else too and cherry-picking a patch that adds them without user is simpler. And indeed Matthew just asked about virt_to_folio() few days ago :)
PATCH v2 2/33 mm/slab: Dissolve slab_map_pages() in its caller
In reply to Vlastimil BabkaThe function no longer does what its name and comment suggests, and just sets two struct page fields, which can be done directly in its sole caller.
--- mm/slab.c | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-)
mm/slab.c +2 -13
@@ -2546,18 +2546,6 @@ static void slab_put_obj(struct kmem_cache *cachep,set_free_obj (page, page->active, objnr); } -/* -* Map pages beginning at addr to the given cache and slab. This is required -* for the slab allocator to be able to lookup the cache and slab of a -* virtual address for kfree, ksize, and slab debugging. -*/ -static void slab_map_pages (struct kmem_cache *cache,struct page *page, -void *freelist) -{ - page->slab_cache = cache; - page->freelist = freelist; -} -/* * Grow (by 1) the number of slabs within a cache. This is called by * kmem_cache_alloc() when there are no active objs left in a cache.
@@ -2621,7 +2609,8 @@ static struct page *cache_grow_begin(struct kmem_cache *cachep,if (OFF_SLAB (cachep) && !freelist) goto opps1; -slab_map_pages (cachep, page, freelist); + page->slab_cache = cachep; + page->freelist = freelist;cache_init_objs (cachep, page);
Signature · 1 line
2.33.1
PATCH v2 3/33 mm/slub: Make object_err() static
In reply to Vlastimil BabkaThere are no callers outside of mm/slub.c anymore.
Move freelist_corrupted() that calls object_err() to avoid a need for forward declaration.
--- include/linux/slub_def.h | 3 --- mm/slub.c | 30 +++++++++++++++--------------- 2 files changed, 15 insertions(+), 18 deletions(-)
include/linux/slub_def.h +0 -3
@@ -156,9 +156,6 @@ static inline void sysfs_slab_release(struct kmem_cache *s) }#endif -void object_err (struct kmem_cache *s,struct page *page, -u8 *object,char *reason); -void *fixup_red_left (struct kmem_cache *s,void *p); static inline void *nearest_obj(struct kmem_cache *cache, struct page *page,
mm/slub.c +15 -15
@@ -822,20 +822,6 @@ static void slab_fix(struct kmem_cache *s, char *fmt, ...)va_end (args); } -static bool freelist_corrupted (struct kmem_cache *s,struct page *page, -void **freelist,void *nextfree) -{ -if ((s->flags &SLAB_CONSISTENCY_CHECKS ) && - !check_valid_pointer (s, page, nextfree) && freelist) { -object_err (s, page, *freelist,"Freechain corrupt" ); - *freelist =NULL ; -slab_fix (s,"Isolate corrupted freechain" ); -return true; - } - -return false; -} -static void print_trailer (struct kmem_cache *s,struct page *page,u8 *p) {unsigned int off;/* Offset of last byte */
@@ -875,7 +861,7 @@ static void print_trailer(struct kmem_cache *s, struct page *page, u8 *p)dump_stack (); } -void object_err (struct kmem_cache *s,struct page *page, +static void object_err (struct kmem_cache *s,struct page *page,u8 *object,char *reason) {if (slab_add_kunit_errors ())
@@ -886,6 +872,20 @@ void object_err(struct kmem_cache *s, struct page *page,add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } +static bool freelist_corrupted (struct kmem_cache *s,struct page *page, +void **freelist,void *nextfree) +{ +if ((s->flags &SLAB_CONSISTENCY_CHECKS ) && + !check_valid_pointer (s, page, nextfree) && freelist) { +object_err (s, page, *freelist,"Freechain corrupt" ); + *freelist =NULL ; +slab_fix (s,"Isolate corrupted freechain" ); +return true; + } + +return false; +} + static __printf(3, 4) void slab_err(struct kmem_cache *s, struct page *page, const char *fmt, ...) {
Signature · 1 line
2.33.1
PATCH v2 4/33 mm: Split slab into its own type
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Make struct slab independent of struct page. It still uses the underlying memory in struct page for storing slab-specific data, but slab and slub can now be weaned off using struct page directly. Some of the wrapper functions (slab_address() and slab_order()) still need to cast to struct folio, but this is a significant disentanglement.
[ [email protected]: Rebase on folios, use folio instead of page where possible.
Do not duplicate flags field in struct slab, instead make the related accessors go through slab_folio(). For testing pfmemalloc use the folio_*_active flag accessors directly so the PageSlabPfmemalloc wrappers can be removed later.
Make folio_slab() expect only folio_test_slab() == true folios and virt_to_slab() return NULL when folio_test_slab() == false.
Move struct slab to mm/slab.h.
Don't represent with struct slab pages that are not true slab pages, but just a compound page obtained directly rom page allocator (with large kmalloc() for SLUB and SLOB). ]
--- include/linux/mm_types.h | 10 +-- mm/slab.h | 167 +++++++++++++++++++++++++++++++++++++++ mm/slub.c | 8 +- 3 files changed, 176 insertions(+), 9 deletions(-)
include/linux/mm_types.h +5 -5
@@ -56,11 +56,11 @@ struct mem_cgroup;* in each subpage, but you may need to restore some of their values * afterwards. * -* SLUB uses cmpxchg_double() to atomically update its freelist and -* counters. That requires that freelist & counters be adjacent and -* double-word aligned. We align all struct pages to double-word -* boundaries, and ensure that 'freelist' is aligned within the -* struct. +* SLUB uses cmpxchg_double() to atomically update its freelist and counters. +* That requires that freelist & counters in struct slab be adjacent and +* double-word aligned. Because struct slab currently just reinterprets the +* bits of struct page, we align all struct pages to double-word boundaries, +* and ensure that 'freelist' is aligned within struct slab. */ #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE #define _struct_page_alignment __aligned(2 * sizeof(unsigned long))
mm/slab.h +167 -0
173 lines not on this page · raw message
mm/slub.c +4 -4
@@ -3787,7 +3787,7 @@ static unsigned int slub_min_objects;* requested a higher minimum order then we start with that one instead of * the smallest order which will fit the object. */ -static inline unsigned int slab_order (unsigned int size, +static inline unsigned int calc_slab_order (unsigned int size,unsigned int min_objects,unsigned int max_order,unsigned int fract_leftover) {
@@ -3851,7 +3851,7 @@ static inline int calculate_order(unsigned int size) fraction =16 ;while (fraction >=4 ) { - order =slab_order (size, min_objects, + order =calc_slab_order (size, min_objects, slub_max_order, fraction);if (order <= slub_max_order)return order;
@@ -3864,14 +3864,14 @@ static inline int calculate_order(unsigned int size)* We were unable to place multiple objects in a slab. Now * lets see if we can place a single object there. */ - order =slab_order (size,1 , slub_max_order,1 ); + order =calc_slab_order (size,1 , slub_max_order,1 );if (order <= slub_max_order)return order;/* * Doh this slab cannot be placed using slub_max_order. */ - order =slab_order (size,1 ,MAX_ORDER ,1 ); + order =calc_slab_order (size,1 ,MAX_ORDER ,1 );if (order <MAX_ORDER )return order;return -ENOSYS ;
Signature · 1 line
2.33.1
Re: [PATCH v2 04/33] mm: Split slab into its own type
Quoted 19 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
Make struct slab independent of struct page. It still uses the underlying memory in struct page for storing slab-specific data, but slab and slub can now be weaned off using struct page directly. Some of the wrapper functions (
slab_address()andslab_order()) still need to cast to struct folio, but this is a significant disentanglement.[ [email protected]: Rebase on folios, use folio instead of page where possible.
Do not duplicate flags field in struct slab, instead make the related accessors go through
slab_folio(). For testing pfmemalloc use the folio_*_active flag accessors directly so the PageSlabPfmemalloc wrappers can be removed later.Make
folio_slab()expect onlyfolio_test_slab()== true folios andvirt_to_slab()return NULL whenfolio_test_slab()== false.
Quoted 8 lines
Move struct slab to
mm/slab.h.Don't represent with struct slab pages that are not true slab pages, but just a compound page obtained directly rom page allocator (with large
kmalloc()for SLUB and SLOB). ]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
PATCH v2 5/33 mm: Add account_slab() and unaccount_slab()
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
These functions take struct slab instead of struct page and replace (un)account_slab_page(). For now their callers just convert page to slab.
[ [email protected]: replace existing functions instead of calling them ]
--- mm/slab.c | 4 ++-- mm/slab.h | 17 ++++++++--------- mm/slub.c | 4 ++-- 3 files changed, 12 insertions(+), 13 deletions(-)
mm/slab.c +2 -2
@@ -1380,7 +1380,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,return NULL ; } -account_slab_page (page, cachep->gfporder, cachep, flags); +account_slab (page_slab (page), cachep->gfporder, cachep, flags);__SetPageSlab (page);/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (page))
@@ -1405,7 +1405,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page)if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order; -unaccount_slab_page (page, order, cachep); +unaccount_slab (page_slab (page), order, cachep);__free_pages (page, order); }
mm/slab.h +8 -9
@@ -583,24 +583,23 @@ static inline struct kmem_cache *virt_to_cache(const void *obj)return page->slab_cache; } -static __always_inline void account_slab_page(struct page *page, int order, -struct kmem_cache *s, -gfp_t gfp) +static __always_inline void account_slab(struct slab *slab, int order, +struct kmem_cache *s,gfp_t gfp) {if (memcg_kmem_enabled () && (s->flags &SLAB_ACCOUNT )) -memcg_alloc_page_obj_cgroups (page, s, gfp, true); +memcg_alloc_page_obj_cgroups (slab_page (slab), s, gfp, true); -mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), +mod_node_page_state (slab_pgdat (slab),cache_vmstat_idx (s),PAGE_SIZE << order); } -static __always_inline void unaccount_slab_page(struct page *page, int order, -struct kmem_cache *s) +static __always_inline void unaccount_slab(struct slab *slab, int order, +struct kmem_cache *s) {if (memcg_kmem_enabled ()) -memcg_free_page_obj_cgroups (page); +memcg_free_page_obj_cgroups (slab_page (slab)); -mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), +mod_node_page_state (slab_pgdat (slab),cache_vmstat_idx (s), -(PAGE_SIZE << order)); }
mm/slub.c +2 -2
@@ -1943,7 +1943,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) page->objects =oo_objects (oo); -account_slab_page (page,oo_order (oo), s, flags); +account_slab (page_slab (page),oo_order (oo), s, flags); page->slab_cache = s;__SetPageSlab (page);
@@ -2014,7 +2014,7 @@ static void __free_slab(struct kmem_cache *s, struct page *page) page->slab_cache =NULL ;if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; -unaccount_slab_page (page, order, s); +unaccount_slab (page_slab (page), order, s);__free_pages (page, order); }
Signature · 1 line
2.33.1
Re: [PATCH v2 05/33] mm: Add account_slab() and unaccount_slab()
Quoted 9 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
These functions take struct slab instead of struct page and replace (un)account_slab_page(). For now their callers just convert page to slab.
[ [email protected]: replace existing functions instead of calling them ]
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
PATCH v2 6/33 mm: Convert virt_to_cache() to use struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
This function is entirely self-contained, so can be converted from page to slab.
--- mm/slab.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/slab.h +4 -4
@@ -574,13 +574,13 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s,static inline struct kmem_cache *virt_to_cache (const void *obj) { -struct page *page; +struct slab *slab; - page =virt_to_head_page (obj); -if (WARN_ONCE (!PageSlab (page),"%s: Object is not a Slab page!\n" , + slab =virt_to_slab (obj); +if (WARN_ONCE (!slab,"%s: Object is not a Slab page!\n" , __func__))return NULL ; -return page->slab_cache; +return slab->slab_cache; } static __always_inline void account_slab(struct slab *slab, int order,
Signature · 1 line
2.33.1
Re: [PATCH v2 06/33] mm: Convert virt_to_cache() to use struct slab
Quoted 7 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
This function is entirely self-contained, so can be converted from page to slab.
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
PATCH v2 7/33 mm: Convert __ksize() to struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
In SLUB, use folios, and struct slab to access slab_cache field. In SLOB, use folios to properly resolve pointers beyond PAGE_SIZE offset of the object.
[ [email protected]: use folios, and only convert folio_test_slab() == true folios
to struct slab ]
--- mm/slob.c | 8 ++++---- mm/slub.c | 12 +++++------- 2 files changed, 9 insertions(+), 11 deletions(-)
mm/slob.c +4 -4
@@ -570,7 +570,7 @@ EXPORT_SYMBOL(kfree);/* can't use ksize for kmem_cache_alloc memory, only kmalloc */ size_t __ksize (const void *block) { -struct page *sp; +struct folio *folio;int align;unsigned int *m;
@@ -578,9 +578,9 @@ size_t __ksize(const void *block)if (unlikely (block ==ZERO_SIZE_PTR ))return 0 ; - sp =virt_to_page (block); -if (unlikely (!PageSlab (sp))) -return page_size (sp); + folio =virt_to_folio (block); +if (unlikely (!folio_test_slab (folio))) +return folio_size (folio); align =max_t (size_t,ARCH_KMALLOC_MINALIGN ,ARCH_SLAB_MINALIGN ); m = (unsigned int *)(block - align);
mm/slub.c +5 -7
@@ -4527,19 +4527,17 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page,size_t __ksize (const void *object) { -struct page *page; +struct folio *folio;if (unlikely (object ==ZERO_SIZE_PTR ))return 0 ; - page =virt_to_head_page (object); + folio =virt_to_folio (object); -if (unlikely (!PageSlab (page))) { -WARN_ON (!PageCompound (page)); -return page_size (page); - } +if (unlikely (!folio_test_slab (folio))) +return folio_size (folio); -return slab_ksize (page->slab_cache); +return slab_ksize (folio_slab (folio)->slab_cache); }EXPORT_SYMBOL (__ksize);
Signature · 1 line
2.33.1
Re: [PATCH v2 07/33] mm: Convert __ksize() to struct slab
Quoted 11 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
In SLUB, use folios, and struct slab to access
slab_cachefield. In SLOB, use folios to properly resolve pointers beyondPAGE_SIZEoffset of the object.[ [email protected]: use folios, and only convert
folio_test_slab()== true folios
to struct slab ]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
Awesome.
PATCH v2 8/33 mm: Use struct slab in kmem_obj_info()
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
All three implementations of slab support kmem_obj_info() which reports details of an object allocated from the slab allocator. By using the slab type instead of the page type, we make it obvious that this can only be called for slabs.
--- mm/slab.c | 12 ++++++------ mm/slab.h | 4 ++-- mm/slab_common.c | 8 ++++---- mm/slob.c | 4 ++-- mm/slub.c | 12 ++++++------ 5 files changed, 20 insertions(+), 20 deletions(-)
mm/slab.c +6 -6
@@ -3646,21 +3646,21 @@ EXPORT_SYMBOL(__kmalloc_node_track_caller);#endif /* CONFIG_NUMA */ #ifdef CONFIG_PRINTK -void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct page *page) +void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct slab *slab) {struct kmem_cache *cachep;unsigned int objnr;void *objp; kpp->kp_ptr = object; - kpp->kp_page = page; - cachep = page->slab_cache; + kpp->kp_slab = slab; + cachep = slab->slab_cache; kpp->kp_slab_cache = cachep; objp = object -obj_offset (cachep); kpp->kp_data_offset =obj_offset (cachep); - page =virt_to_head_page (objp); - objnr =obj_to_index (cachep, page, objp); - objp =index_to_obj (cachep, page, objnr); + slab =virt_to_slab (objp); + objnr =obj_to_index (cachep,slab_page (slab), objp); + objp =index_to_obj (cachep,slab_page (slab), objnr); kpp->kp_objp = objp;if (DEBUG && cachep->flags &SLAB_STORE_USER ) kpp->kp_ret = *dbg_userword (cachep, objp);
mm/slab.h +2 -2
@@ -801,7 +801,7 @@ static inline void debugfs_slab_release(struct kmem_cache *s) { }#define KS_ADDRS_COUNT 16struct kmem_obj_info {void *kp_ptr; -struct page *kp_page; +struct slab *kp_slab;void *kp_objp;unsigned long kp_data_offset;struct kmem_cache *kp_slab_cache;
@@ -809,7 +809,7 @@ struct kmem_obj_info {void *kp_stack[KS_ADDRS_COUNT ];void *kp_free_stack[KS_ADDRS_COUNT ]; }; -void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct page *page); +void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct slab *slab);#endif #endif /* MM_SLAB_H */
mm/slab_common.c +4 -4
@@ -579,18 +579,18 @@ void kmem_dump_obj(void *object) {char *cp =IS_ENABLED (CONFIG_MMU ) ?"" :"/vmalloc" ;int i; -struct page *page; +struct slab *slab;unsigned long ptroffset;struct kmem_obj_info kp = { };if (WARN_ON_ONCE (!virt_addr_valid (object)))return ; - page =virt_to_head_page (object); -if (WARN_ON_ONCE (!PageSlab (page))) { + slab =virt_to_slab (object); +if (WARN_ON_ONCE (!slab)) {pr_cont (" non-slab memory.\n" );return ; } -kmem_obj_info (&kp, object, page); +kmem_obj_info (&kp, object, slab);if (kp.kp_slab_cache)pr_cont (" slab%s %s" , cp, kp.kp_slab_cache->name);else
mm/slob.c +2 -2
@@ -462,10 +462,10 @@ static void slob_free(void *block, int size) }#ifdef CONFIG_PRINTK -void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct page *page) +void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct slab *slab) { kpp->kp_ptr = object; - kpp->kp_page = page; + kpp->kp_slab = slab; }#endif
mm/slub.c +6 -6
@@ -4322,31 +4322,31 @@ int __kmem_cache_shutdown(struct kmem_cache *s) }#ifdef CONFIG_PRINTK -void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct page *page) +void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct slab *slab) {void *base;int __maybe_unused i;unsigned int objnr;void *objp;void *objp0; -struct kmem_cache *s = page->slab_cache; +struct kmem_cache *s = slab->slab_cache;struct track __maybe_unused *trackp; kpp->kp_ptr = object; - kpp->kp_page = page; + kpp->kp_slab = slab; kpp->kp_slab_cache = s; - base =page_address (page); + base =slab_address (slab); objp0 =kasan_reset_tag (object);#ifdef CONFIG_SLUB_DEBUG objp =restore_red_left (s, objp0);#else objp = objp0;#endif - objnr =obj_to_index (s, page, objp); + objnr =obj_to_index (s,slab_page (slab), objp); kpp->kp_data_offset = (unsigned long )((char *)objp0 - (char *)objp); objp = base + s->size * objnr; kpp->kp_objp = objp; -if (WARN_ON_ONCE (objp < base || objp >= base + page->objects * s->size || (objp - base) % s->size) || +if (WARN_ON_ONCE (objp < base || objp >= base + slab->objects * s->size || (objp - base) % s->size) || !(s->flags &SLAB_STORE_USER ))return ;#ifdef CONFIG_SLUB_DEBUG
Signature · 1 line
2.33.1
PATCH v2 9/33 mm: Convert check_heap_object() to use struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Ensure that we're not seeing a tail page inside __check_heap_object() by converting to a slab instead of a page. Take the opportunity to mark the slab as const since we're not modifying it. Also move the declaration of __check_heap_object() to mm/slab.h so it's not available to the wider kernel.
[ [email protected]: in check_heap_object() only convert to struct slab for
actual PageSlab pages; use folio as intermediate step instead of page ]
--- include/linux/slab.h | 8 -------- mm/slab.c | 14 +++++++------- mm/slab.h | 9 +++++++++ mm/slub.c | 10 +++++----- mm/usercopy.c | 13 +++++++------ 5 files changed, 28 insertions(+), 26 deletions(-)
include/linux/slab.h +0 -8
@@ -189,14 +189,6 @@ bool kmem_valid_obj(void *object);void kmem_dump_obj (void *object);#endif -#ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR -void __check_heap_object (const void *ptr,unsigned long n,struct page *page, -bool to_user); -#else -static inline void __check_heap_object (const void *ptr,unsigned long n, -struct page *page,bool to_user) { } -#endif -/* * Some archs want to perform DMA into kmalloc caches and need a guaranteed * alignment larger than the alignment of a 64-bit integer.
mm/slab.c +7 -7
@@ -372,8 +372,8 @@ static void **dbg_userword(struct kmem_cache *cachep, void *objp)static int slab_max_order =SLAB_MAX_ORDER_LO ;static bool slab_max_order_set__initdata ; -static inline void *index_to_obj (struct kmem_cache *cache,struct page *page, -unsigned int idx) +static inline void *index_to_obj (struct kmem_cache *cache, +const struct page *page,unsigned int idx) {return page->s_mem + cache->size * idx; }
@@ -4166,8 +4166,8 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer,* Returns NULL if check passes, otherwise const char * to name of cache * to indicate an error. */ -void __check_heap_object (const void *ptr,unsigned long n,struct page *page, -bool to_user) +void __check_heap_object (const void *ptr,unsigned long n, +const struct slab *slab,bool to_user) {struct kmem_cache *cachep;unsigned int objnr;
@@ -4176,15 +4176,15 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, ptr =kasan_reset_tag (ptr);/* Find and validate object. */ - cachep = page->slab_cache; - objnr =obj_to_index (cachep, page, (void *)ptr); + cachep = slab->slab_cache; + objnr =obj_to_index (cachep,slab_page (slab), (void *)ptr);BUG_ON (objnr >= cachep->num);/* Find offset within object. */ if (is_kfence_address (ptr)) offset = ptr -kfence_object_start (ptr);else - offset = ptr -index_to_obj (cachep, page, objnr) -obj_offset (cachep); + offset = ptr -index_to_obj (cachep,slab_page (slab), objnr) -obj_offset (cachep);/* Allow address range falling entirely within usercopy region. */ if (offset >= cachep->useroffset &&
mm/slab.h +9 -0
@@ -812,4 +812,13 @@ struct kmem_obj_info {void kmem_obj_info (struct kmem_obj_info *kpp,void *object,struct slab *slab);#endif +#ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR +void __check_heap_object (const void *ptr,unsigned long n, +const struct slab *slab,bool to_user); +#else +static inline +void __check_heap_object (const void *ptr,unsigned long n, +const struct slab *slab,bool to_user) { } +#endif +#endif /* MM_SLAB_H */
mm/slub.c +5 -5
@@ -4484,8 +4484,8 @@ EXPORT_SYMBOL(__kmalloc_node);* Returns NULL if check passes, otherwise const char * to name of cache * to indicate an error. */ -void __check_heap_object (const void *ptr,unsigned long n,struct page *page, -bool to_user) +void __check_heap_object (const void *ptr,unsigned long n, +const struct slab *slab,bool to_user) {struct kmem_cache *s;unsigned int offset;
@@ -4494,10 +4494,10 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, ptr =kasan_reset_tag (ptr);/* Find object and usable object size. */ - s = page->slab_cache; + s = slab->slab_cache;/* Reject impossible pointers. */ -if (ptr <page_address (page)) +if (ptr <slab_address (slab))usercopy_abort ("SLUB object not in SLUB page?!" ,NULL , to_user,0 , n);
@@ -4505,7 +4505,7 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page,if (is_kfence) offset = ptr -kfence_object_start (ptr);else - offset = (ptr -page_address (page)) % s->size; + offset = (ptr -slab_address (slab)) % s->size;/* Adjust for redzone and reject if within the redzone. */ if (!is_kfence &&kmem_cache_debug_flags (s,SLAB_RED_ZONE )) {
mm/usercopy.c +7 -6
@@ -20,6 +20,7 @@#include <linux/atomic.h> #include <linux/jump_label.h> #include <asm/sections.h> +#include "slab.h" /* * Checks if a given pointer and length is contained by the current
@@ -223,7 +224,7 @@ static inline void check_page_span(const void *ptr, unsigned long n,static inline void check_heap_object (const void *ptr,unsigned long n,bool to_user) { -struct page *page; +struct folio *folio;if (!virt_addr_valid (ptr))return ;
@@ -231,16 +232,16 @@ static inline void check_heap_object(const void *ptr, unsigned long n,/* * When CONFIG_HIGHMEM=y, kmap_to_page() will give either the * highmem page or fallback to virt_to_page(). The following -* is effectively a highmem-aware virt_to_head_page(). +* is effectively a highmem-aware virt_to_slab(). */ - page =compound_head (kmap_to_page ((void *)ptr)); + folio =page_folio (kmap_to_page ((void *)ptr)); -if (PageSlab (page)) { +if (folio_test_slab (folio)) {/* Check slab allocator for flags and size. */ -__check_heap_object (ptr, n, page, to_user); +__check_heap_object (ptr, n,folio_slab (folio), to_user); }else {/* Verify object does not incorrectly span multiple pages. */ -check_page_span (ptr, n, page, to_user); +check_page_span (ptr, n,folio_page (folio,0 ), to_user); } }
Signature · 1 line
2.33.1
PATCH v2 10/33 mm/slub: Convert detached_freelist to use a struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
This gives us a little bit of extra typesafety as we know that nobody called virt_to_page() instead of virt_to_head_page().
[ [email protected]: Use folio as intermediate step when filtering out
large kmalloc pages ]
--- mm/slub.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)
mm/slub.c +17 -14
@@ -3532,7 +3532,7 @@ void kmem_cache_free(struct kmem_cache *s, void *x)EXPORT_SYMBOL (kmem_cache_free);struct detached_freelist { -struct page *page; +struct slab *slab;void *tail;void *freelist;int cnt;
@@ -3554,8 +3554,8 @@ static inline void free_nonslab_page(struct page *page, void *object)/* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same -* page. It builds a detached freelist directly within the given -* page/objects. This can happen without any need for +* slab. It builds a detached freelist directly within the given +* slab/objects. This can happen without any need for * synchronization, because the objects are owned by running process. * The freelist is build up as a single linked list in the objects. * The idea is, that this detached freelist can then be bulk
@@ -3570,10 +3570,11 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,size_t first_skipped_index =0 ;int lookahead =3 ;void *object; -struct page *page; +struct folio *folio; +struct slab *slab;/* Always re-init detached_freelist */ - df->page =NULL ; + df->slab =NULL ; do { object = p[--size];
@@ -3583,17 +3584,19 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,if (!object)return 0 ; - page =virt_to_head_page (object); + folio =virt_to_folio (object);if (!s) {/* Handle kalloc'ed objects */ -if (unlikely (!PageSlab (page))) { -free_nonslab_page (page, object); +if (unlikely (!folio_test_slab (folio))) { +free_nonslab_page (folio_page (folio,0 ), object); p[size] =NULL ;/* mark object processed */ return size; }/* Derive kmem_cache from object */ - df->s = page->slab_cache; + slab =folio_slab (folio); + df->s = slab->slab_cache; }else { + slab =folio_slab (folio); df->s =cache_from_obj (s, object);/* Support for memcg */ }
@@ -3605,7 +3608,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, }/* Start new detached freelist */ - df->page = page; + df->slab = slab;set_freepointer (df->s, object,NULL ); df->tail = object; df->freelist = object;
@@ -3617,8 +3620,8 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,if (!object)continue ;/* Skip processed objects */ -/* df->page is always set at this point */ -if (df->page ==virt_to_head_page (object)) { +/* df->slab is always set at this point */ +if (df->slab ==virt_to_slab (object)) {/* Opportunity build freelist */ set_freepointer (df->s, object, df->freelist); df->freelist = object;
@@ -3650,10 +3653,10 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)struct detached_freelist df; size =build_detached_freelist (s, size, p, &df); -if (!df.page) +if (!df.slab)continue ; -slab_free (df.s, df.page, df.freelist, df.tail, df.cnt, _RET_IP_); +slab_free (df.s,slab_page (df.slab), df.freelist, df.tail, df.cnt, _RET_IP_); }while (likely (size)); }EXPORT_SYMBOL (kmem_cache_free_bulk);
Signature · 1 line
2.33.1
PATCH v2 11/33 mm/slub: Convert kfree() to use a struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Convert kfree(), kmem_cache_free() and ___cache_free() to resolve object addresses to struct slab, using folio as intermediate step where needed. Keep passing the result as struct page for now in preparation for mass conversion of internal functions.
[ [email protected]: Use folio as intermediate step when checking for
large kmalloc pages ]
--- mm/slub.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
mm/slub.c +9 -7
@@ -3517,7 +3517,7 @@ static __always_inline void slab_free(struct kmem_cache *s, struct page *page,#ifdef CONFIG_KASAN_GENERIC void ___cache_free (struct kmem_cache *cache,void *x,unsigned long addr) { -do_slab_free (cache,virt_to_head_page (x), x,NULL ,1 , addr); +do_slab_free (cache,slab_page (virt_to_slab (x)), x,NULL ,1 , addr); }#endif
@@ -3527,7 +3527,7 @@ void kmem_cache_free(struct kmem_cache *s, void *x)if (!s)return ;trace_kmem_cache_free (_RET_IP_, x, s->name); -slab_free (s,virt_to_head_page (x), x,NULL ,1 , _RET_IP_); +slab_free (s,slab_page (virt_to_slab (x)), x,NULL ,1 , _RET_IP_); }EXPORT_SYMBOL (kmem_cache_free);
@@ -4546,7 +4546,8 @@ EXPORT_SYMBOL(__ksize);void kfree (const void *x) { -struct page *page; +struct folio *folio; +struct slab *slab;void *object = (void *)x;trace_kfree (_RET_IP_, x);
@@ -4554,12 +4555,13 @@ void kfree(const void *x)if (unlikely (ZERO_OR_NULL_PTR (x)))return ; - page =virt_to_head_page (x); -if (unlikely (!PageSlab (page))) { -free_nonslab_page (page, object); + folio =virt_to_folio (x); +if (unlikely (!folio_test_slab (folio))) { +free_nonslab_page (folio_page (folio,0 ), object);return ; } -slab_free (page->slab_cache, page, object,NULL ,1 , _RET_IP_); + slab =folio_slab (folio); +slab_free (slab->slab_cache,slab_page (slab), object,NULL ,1 , _RET_IP_); }EXPORT_SYMBOL (kfree);
Signature · 1 line
2.33.1
PATCH v2 12/33 mm/slub: Convert __slab_lock() and __slab_unlock() to struct slab
In reply to Vlastimil BabkaThese functions operate on the PG_locked page flag, but make them accept struct slab to encapsulate this implementation detail.
--- mm/slub.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
mm/slub.c +11 -7
@@ -440,14 +440,18 @@ slub_set_cpu_partial(struct kmem_cache *s, unsigned int nr_objects)/* * Per slab locking using the pagelock */ -static __always_inline void __slab_lock(struct page *page) +static __always_inline void __slab_lock(struct slab *slab) { +struct page *page =slab_page (slab); +VM_BUG_ON_PAGE (PageTail (page), page);bit_spin_lock (PG_locked, &page->flags); } -static __always_inline void __slab_unlock(struct page *page) +static __always_inline void __slab_unlock(struct slab *slab) { +struct page *page =slab_page (slab); +VM_BUG_ON_PAGE (PageTail (page), page);__bit_spin_unlock (PG_locked, &page->flags); }
@@ -456,12 +460,12 @@ static __always_inline void slab_lock(struct page *page, unsigned long *flags) {if (IS_ENABLED (CONFIG_PREEMPT_RT ))local_irq_save (*flags); -__slab_lock (page); +__slab_lock (page_slab (page)); } static __always_inline void slab_unlock(struct page *page, unsigned long *flags) { -__slab_unlock (page); +__slab_unlock (page_slab (page));if (IS_ENABLED (CONFIG_PREEMPT_RT ))local_irq_restore (*flags); }
@@ -530,16 +534,16 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page,unsigned long flags;local_irq_save (flags); -__slab_lock (page); +__slab_lock (page_slab (page));if (page->freelist == freelist_old && page->counters == counters_old) { page->freelist = freelist_new; page->counters = counters_new; -__slab_unlock (page); +__slab_unlock (page_slab (page));local_irq_restore (flags);return true; } -__slab_unlock (page); +__slab_unlock (page_slab (page));local_irq_restore (flags); }
Signature · 1 line
2.33.1
PATCH v2 13/33 mm/slub: Convert print_page_info() to print_slab_info()
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Improve the type safety and prepare for further conversion. For flags access, convert to folio internally.
[ [email protected]: access flags via folio_flags() ]
--- mm/slub.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
mm/slub.c +7 -6
@@ -788,12 +788,13 @@ void print_tracking(struct kmem_cache *s, void *object)print_track ("Freed" ,get_track (s, object,TRACK_FREE ), pr_time); } -static void print_page_info (struct page *page) +static void print_slab_info (const struct slab *slab) { -pr_err ("Slab 0x%p objects=%u used=%u fp=0x%p flags=%pGp\n" , - page, page->objects, page->inuse, page->freelist, - &page->flags); +struct folio *folio = (struct folio *)slab_folio (slab); +pr_err ("Slab 0x%p objects=%u used=%u fp=0x%p flags=%pGp\n" , + slab, slab->objects, slab->inuse, slab->freelist, +folio_flags (folio,0 )); }static void slab_bug (struct kmem_cache *s,char *fmt, ...)
@@ -833,7 +834,7 @@ static void print_trailer(struct kmem_cache *s, struct page *page, u8 *p)print_tracking (s, p); -print_page_info (page); +print_slab_info (page_slab (page));pr_err ("Object 0x%p @offset=%tu fp=0x%p\n\n" , p, p - addr,get_freepointer (s, p));
@@ -903,7 +904,7 @@ static __printf(3, 4) void slab_err(struct kmem_cache *s, struct page *page,vsnprintf (buf,sizeof (buf), fmt, args);va_end (args);slab_bug (s,"%s" , buf); -print_page_info (page); +print_slab_info (page_slab (page));dump_stack ();add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); }
Signature · 1 line
2.33.1
PATCH v2 14/33 mm/slub: Convert alloc_slab_page() to return a struct slab
In reply to Vlastimil BabkaPreparatory, callers convert back to struct page for now.
Also move setting page flags to alloc_slab_page() where we still operate on a struct page. This means the page->slab_cache pointer is now set later than the PageSlab flag, which could theoretically confuse some pfn walker assuming PageSlab means there would be a valid cache pointer. But as the code had no barriers and used __set_bit() anyway, it could have happened already, so there shouldn't be such a walker.
--- mm/slub.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
mm/slub.c +16 -10
@@ -1788,18 +1788,27 @@ static void *setup_object(struct kmem_cache *s, struct page *page,/* * Slab allocation and freeing */ -static inline struct page *alloc_slab_page (struct kmem_cache *s, +static inline struct slab *alloc_slab_page (struct kmem_cache *s,gfp_t flags,int node,struct kmem_cache_order_objects oo) { -struct page *page; +struct folio *folio; +struct slab *slab;unsigned int order =oo_order (oo);if (node ==NUMA_NO_NODE ) - page =alloc_pages (flags, order); + folio = (struct folio *)alloc_pages (flags, order);else - page =__alloc_pages_node (node, flags, order); + folio = (struct folio *)__alloc_pages_node (node, flags, order); -return page; +if (!folio) +return NULL ; + + slab =folio_slab (folio); +__folio_set_slab (folio); +if (page_is_pfmemalloc (folio_page (folio,0 ))) +slab_set_pfmemalloc (slab); + +return slab; }#ifdef CONFIG_SLAB_FREELIST_RANDOM
@@ -1932,7 +1941,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)if ((alloc_gfp & __GFP_DIRECT_RECLAIM) &&oo_order (oo) >oo_order (s->min)) alloc_gfp = (alloc_gfp | __GFP_NOMEMALLOC) & ~(__GFP_RECLAIM|__GFP_NOFAIL); - page =alloc_slab_page (s, alloc_gfp, node, oo); + page =slab_page (alloc_slab_page (s, alloc_gfp, node, oo));if (unlikely (!page)) { oo = s->min; alloc_gfp = flags;
@@ -1940,7 +1949,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)* Allocation may have failed due to fragmentation. * Try a lower order alloc if possible */ - page =alloc_slab_page (s, alloc_gfp, node, oo); + page =slab_page (alloc_slab_page (s, alloc_gfp, node, oo));if (unlikely (!page)) goto out;stat (s,ORDER_FALLBACK );
@@ -1951,9 +1960,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)account_slab (page_slab (page),oo_order (oo), s, flags); page->slab_cache = s; -__SetPageSlab (page); -if (page_is_pfmemalloc (page)) -SetPageSlabPfmemalloc (page);kasan_poison_slab (page);
Signature · 1 line
2.33.1
PATCH v2 15/33 mm/slub: Convert __free_slab() to use struct slab
In reply to Vlastimil Babka__free_slab() is on the boundary of distinguishing struct slab and struct page so start with struct slab but convert to folio for working with flags and folio_page() to call functions that require struct page.
--- mm/slub.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
mm/slub.c +13 -13
@@ -2005,35 +2005,35 @@ static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node) flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node); } -static void __free_slab (struct kmem_cache *s,struct page *page) +static void __free_slab (struct kmem_cache *s,struct slab *slab) { -int order =compound_order (page); +struct folio *folio =slab_folio (slab); +int order =folio_order (folio);int pages =1 << order;if (kmem_cache_debug_flags (s,SLAB_CONSISTENCY_CHECKS )) {void *p; -slab_pad_check (s, page); -for_each_object (p, s,page_address (page), - page->objects) -check_object (s, page, p,SLUB_RED_INACTIVE ); +slab_pad_check (s,folio_page (folio,0 )); +for_each_object (p, s,slab_address (slab), slab->objects) +check_object (s,folio_page (folio,0 ), p,SLUB_RED_INACTIVE ); } -__ClearPageSlabPfmemalloc (page); -__ClearPageSlab (page); +__slab_clear_pfmemalloc (slab); +__folio_clear_slab (folio);/* In union with page->mapping where page allocator expects NULL */ - page->slab_cache =NULL ; + slab->slab_cache =NULL ;if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; -unaccount_slab (page_slab (page), order, s); -__free_pages (page, order); +unaccount_slab (slab, order, s); +__free_pages (folio_page (folio,0 ), order); }static void rcu_free_slab (struct rcu_head *h) { struct page *page = container_of(h, struct page, rcu_head); -__free_slab (page->slab_cache, page); +__free_slab (page->slab_cache,page_slab (page)); }static void free_slab (struct kmem_cache *s,struct page *page)
@@ -2041,7 +2041,7 @@ static void free_slab(struct kmem_cache *s, struct page *page)if (unlikely (s->flags &SLAB_TYPESAFE_BY_RCU )) {call_rcu (&page->rcu_head, rcu_free_slab); }else -__free_slab (s, page); +__free_slab (s,page_slab (page)); }static void discard_slab (struct kmem_cache *s,struct page *page)
Signature · 1 line
2.33.1
PATCH v2 16/33 mm/slub: Convert pfmemalloc_match() to take a struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Preparatory for mass conversion. Use the new slab_test_pfmemalloc() helper. As it doesn't do VM_BUG_ON(!PageSlab()) we no longer need the pfmemalloc_match_unsafe() variant.
--- mm/slub.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)
mm/slub.c +6 -19
@@ -2129,7 +2129,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain);static inline void put_cpu_partial (struct kmem_cache *s,struct page *page,int drain) { }#endif -static inline bool pfmemalloc_match (struct page *page,gfp_t gfpflags); +static inline bool pfmemalloc_match (struct slab *slab,gfp_t gfpflags);/* * Try to allocate a partial slab from a specific node.
@@ -2155,7 +2155,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n,list_for_each_entry_safe (page, page2, &n->partial, slab_list) {void *t; -if (!pfmemalloc_match (page, gfpflags)) +if (!pfmemalloc_match (page_slab (page), gfpflags))continue ; t =acquire_slab (s, n, page, object ==NULL );
@@ -2833,22 +2833,9 @@ slab_out_of_memory(struct kmem_cache *s, gfp_t gfpflags, int nid)#endif } -static inline bool pfmemalloc_match (struct page *page,gfp_t gfpflags) +static inline bool pfmemalloc_match (struct slab *slab,gfp_t gfpflags) { -if (unlikely (PageSlabPfmemalloc (page))) -return gfp_pfmemalloc_allowed (gfpflags); - -return true; -} - -/* -* A variant of pfmemalloc_match() that tests page flags without asserting -* PageSlab. Intended for opportunistic checks before taking a lock and -* rechecking that nobody else freed the page under us. -*/ -static inline bool pfmemalloc_match_unsafe (struct page *page,gfp_t gfpflags) -{ -if (unlikely (__PageSlabPfmemalloc (page))) +if (unlikely (slab_test_pfmemalloc (slab)))return gfp_pfmemalloc_allowed (gfpflags);return true;
@@ -2950,7 +2937,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,* PFMEMALLOC but right now, we are losing the pfmemalloc * information when the page leaves the per-cpu allocator */ -if (unlikely (!pfmemalloc_match_unsafe (page, gfpflags))) +if (unlikely (!pfmemalloc_match (page_slab (page), gfpflags))) goto deactivate_slab;/* must check again c->page in case we got preempted and it changed */
@@ -3062,7 +3049,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, } } -if (unlikely (!pfmemalloc_match (page, gfpflags))) +if (unlikely (!pfmemalloc_match (page_slab (page), gfpflags)))/* * For !pfmemalloc_match() case we don't load freelist so that * we don't make further mismatched allocations easier.
Signature · 1 line
2.33.1
PATCH v2 17/33 mm/slub: Convert most struct page to struct slab by spatch
In reply to Vlastimil BabkaThe majority of conversion from struct page to struct slab in SLUB internals can be delegated to a coccinelle semantic patch. This includes renaming of variables with 'page' in name to 'slab', and similar.
Big thanks to Julia Lawall and Luis Chamberlain for help with coccinelle.
// Options: --include-headers --no-includes --smpl-spacing include/linux/slub_def.h mm/slub.c // Note: needs coccinelle 1.1.1 to avoid breaking whitespace, and ocaml for the // embedded script script
// build list of functions to exclude from applying the next rule @initialize:ocaml@
@@
let ok_function p =
not (List.mem (List.hd p).current_element ["nearest_obj";"obj_to_index";"objs_per_slab_page";"__slab_lock";"__slab_unlock";"free_nonslab_page";"kmalloc_large_node"])
// convert the type from struct page to struct page in all functions except the // list from previous rule
// this also affects struct kmem_cache_cpu, but that's ok
@@
position p : script:ocaml() { ok_function p };
@@
- struct page@p
+ struct slab
// in struct kmem_cache_cpu, change the name from page to slab // the type was already converted by the previous rule @@
@@
struct kmem_cache_cpu {
...
-struct slab *page;
+struct slab *slab;
...
}
// there are many places that use c->page which is now c->slab after the // previous rule
@@
struct kmem_cache_cpu *c;
@@
-c->page
+c->slab
@@
@@
struct kmem_cache {
...
- unsigned int cpu_partial_pages;
+ unsigned int cpu_partial_slabs;
...
}
@@
struct kmem_cache *s;
@@
- s->cpu_partial_pages
+ s->cpu_partial_slabs
@@
@@
static void
- setup_page_debug(
+ setup_slab_debug(
...)
{...}
@@
@@
- setup_page_debug(
+ setup_slab_debug(
...);
// for all functions (with exceptions), change any "struct slab *page" // parameter to "struct slab *slab" in the signature, and generally all // occurences of "page" to "slab" in the body - with some special cases.
@@
identifier fn !~ "free_nonslab_page|obj_to_index|objs_per_slab_page|nearest_obj"; @@
fn(...,
- struct slab *page
+ struct slab *slab
,...)
{
<...
- page
+ slab
...>
}
// similar to previous but the param is called partial_page @@
identifier fn;
@@
fn(...,
- struct slab *partial_page
+ struct slab *partial_slab
,...)
{
<...
- partial_page
+ partial_slab
...>
}
// similar to previous but for functions that take pointer to struct page ptr @@
identifier fn;
@@
fn(...,
- struct slab **ret_page
+ struct slab **ret_slab
,...)
{
<...
- ret_page
+ ret_slab
...>
}
// functions converted by previous rules that were temporarily called using // slab_page(E) so we want to remove the wrapper now that they accept struct // slab ptr directly
@@
identifier fn =~ "slab_free|do_slab_free";
expression E;
@@
fn(...,
- slab_page(E)
+ E
,...)
// similar to previous but for another pattern
@@
identifier fn =~ "slab_pad_check|check_object";
@@
fn(...,
- folio_page(folio, 0)
+ slab
,...)
// functions that were returning struct page ptr and now will return struct // slab ptr, including slab_page() wrapper removal
@@
identifier fn =~ "allocate_slab|new_slab";
expression E;
@@
static
-struct slab *
+struct slab *
fn(...)
{
<...
- slab_page(E)
+ E
...>
}
// rename any former struct page * declarations
@@
@@
struct slab *
(
- page
+ slab
|
- partial_page
+ partial_slab
|
- oldpage
+ oldslab
)
;
// this has to be separate from previous rule as page and page2 appear at the // same line
@@
@@
struct slab *
-page2
+slab2
;
// similar but with initial assignment
@@
expression E;
@@
struct slab *
(
- page
+ slab
|
- flush_page
+ flush_slab
|
- discard_page
+ slab_to_discard
|
- page_to_unfreeze
+ slab_to_unfreeze
)
= E;
// convert most of struct page to struct slab usage inside functions (with
// exceptions), including specific variable renames
@@
identifier fn !~ "nearest_obj|obj_to_index|objs_per_slab_page|__slab_(un)*lock|__free_slab|free_nonslab_page|kmalloc_large_node"; expression E;
@@
fn(...)
{
<...
(
- int pages;
+ int slabs;
|
- int pages = E;
+ int slabs = E;
|
- page
+ slab
|
- flush_page
+ flush_slab
|
- partial_page
+ partial_slab
|
- oldpage->pages
+ oldslab->slabs
|
- oldpage
+ oldslab
|
- unsigned int nr_pages;
+ unsigned int nr_slabs;
|
- nr_pages
+ nr_slabs
|
- unsigned int partial_pages = E;
+ unsigned int partial_slabs = E;
|
- partial_pages
+ partial_slabs
)
...>
}
// this has to be split out from the previous rule so that lines containing
// multiple matching changes will be fully converted
@@
identifier fn !~ "nearest_obj|obj_to_index|objs_per_slab_page|__slab_(un)*lock|__free_slab|free_nonslab_page|kmalloc_large_node"; @@
fn(...)
{
<...
(
- slab->pages
+ slab->slabs
|
- pages
+ slabs
|
- page2
+ slab2
|
- discard_page
+ slab_to_discard
|
- page_to_unfreeze
+ slab_to_unfreeze
)
...>
}
// after we simply changed all occurences of page to slab, some usages need
// adjustment for slab-specific functions, or use slab_page() wrapper
@@
identifier fn !~ "nearest_obj|obj_to_index|objs_per_slab_page|__slab_(un)*lock|__free_slab|free_nonslab_page|kmalloc_large_node"; @@
fn(...)
{
<...
(
- page_slab(slab)
+ slab
|
- kasan_poison_slab(slab)
+ kasan_poison_slab(slab_page(slab))
|
- page_address(slab)
+ slab_address(slab)
|
- page_size(slab)
+ slab_size(slab)
|
- PageSlab(slab)
+ folio_test_slab(slab_folio(slab))
|
- page_to_nid(slab)
+ slab_nid(slab)
|
- compound_order(slab)
+ slab_order(slab)
)
...>
}
--- include/linux/slub_def.h | 6 +- mm/slub.c | 872 +++++++++++++++++++-------------------- 2 files changed, 439 insertions(+), 439 deletions(-)
include/linux/slub_def.h +3 -3
@@ -48,9 +48,9 @@ enum stat_item {struct kmem_cache_cpu {void **freelist;/* Pointer to next available object */ unsigned long tid;/* Globally unique transaction id */ -struct page *page;/* The slab from which we are allocating */ +struct slab *slab;/* The slab from which we are allocating */ #ifdef CONFIG_SLUB_CPU_PARTIAL -struct page *partial;/* Partially allocated frozen slabs */ +struct slab *partial;/* Partially allocated frozen slabs */ #endif local_lock_t lock;/* Protects the fields above */ #ifdef CONFIG_SLUB_STATS
@@ -100,7 +100,7 @@ struct kmem_cache {/* Number of per cpu partial objects to keep around */ unsigned int cpu_partial;/* Number of per cpu partial pages to keep around */ -unsigned int cpu_partial_pages; +unsigned int cpu_partial_slabs;#endif struct kmem_cache_order_objects oo;
mm/slub.c +436 -436
2,193 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH v2 18/33 mm/slub: Finish struct page to struct slab conversion
In reply to Vlastimil BabkaUpdate comments mentioning pages to mention slabs where appropriate. Also some goto labels.
--- include/linux/slub_def.h | 2 +- mm/slub.c | 105 +++++++++++++++++++-------------------- 2 files changed, 53 insertions(+), 54 deletions(-)
include/linux/slub_def.h +1 -1
@@ -99,7 +99,7 @@ struct kmem_cache {#ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; -/* Number of per cpu partial pages to keep around */ +/* Number of per cpu partial slabs to keep around */ unsigned int cpu_partial_slabs;#endif struct kmem_cache_order_objects oo;
mm/slub.c +52 -53
315 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH v2 19/33 mm/slab: Convert kmem_getpages() and kmem_freepages() to struct slab
In reply to Vlastimil BabkaThese functions sit at the boundary to page allocator. Also use folio internally to avoid extra compound_head() when dealing with page flags.
--- mm/slab.c | 52 ++++++++++++++++++++++++++++------------------------ 1 file changed, 28 insertions(+), 24 deletions(-)
mm/slab.c +28 -24
@@ -1367,57 +1367,61 @@ slab_out_of_memory(struct kmem_cache *cachep, gfp_t gfpflags, int nodeid)* did not request dmaable memory, we might get it, but that * would be relatively rare and ignorable. */ -static struct page *kmem_getpages (struct kmem_cache *cachep,gfp_t flags, +static struct slab *kmem_getpages (struct kmem_cache *cachep,gfp_t flags,int nodeid) { -struct page *page; +struct folio *folio; +struct slab *slab; flags |= cachep->allocflags; - page =__alloc_pages_node (nodeid, flags, cachep->gfporder); -if (!page) { + folio = (struct folio *)__alloc_pages_node (nodeid, flags, cachep->gfporder); +if (!folio) {slab_out_of_memory (cachep, flags, nodeid);return NULL ; } -account_slab (page_slab (page), cachep->gfporder, cachep, flags); -__SetPageSlab (page); + slab =folio_slab (folio); + +account_slab (slab, cachep->gfporder, cachep, flags); +__folio_set_slab (folio);/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ -if (sk_memalloc_socks () &&page_is_pfmemalloc (page)) -SetPageSlabPfmemalloc (page); +if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 ))) +slab_set_pfmemalloc (slab); -return page; +return slab; }/* * Interface to system's page release. */ -static void kmem_freepages (struct kmem_cache *cachep,struct page *page) +static void kmem_freepages (struct kmem_cache *cachep,struct slab *slab) {int order = cachep->gfporder; +struct folio *folio =slab_folio (slab); -BUG_ON (!PageSlab (page)); -__ClearPageSlabPfmemalloc (page); -__ClearPageSlab (page); -page_mapcount_reset (page); +BUG_ON (!folio_test_slab (folio)); +__slab_clear_pfmemalloc (slab); +__folio_clear_slab (folio); +page_mapcount_reset (folio_page (folio,0 ));/* In union with page->mapping where page allocator expects NULL */ - page->slab_cache =NULL ; + slab->slab_cache =NULL ;if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order; -unaccount_slab (page_slab (page), order, cachep); -__free_pages (page, order); +unaccount_slab (slab, order, cachep); +__free_pages (folio_page (folio,0 ), order); }static void kmem_rcu_free (struct rcu_head *head) {struct kmem_cache *cachep; -struct page *page; +struct slab *slab; - page = container_of(head, struct page, rcu_head); - cachep = page->slab_cache; + slab = container_of(head, struct slab, rcu_head); + cachep = slab->slab_cache; -kmem_freepages (cachep, page); +kmem_freepages (cachep, slab); }#if DEBUG
@@ -1624,7 +1628,7 @@ static void slab_destroy(struct kmem_cache *cachep, struct page *page)if (unlikely (cachep->flags &SLAB_TYPESAFE_BY_RCU ))call_rcu (&page->rcu_head, kmem_rcu_free);else -kmem_freepages (cachep, page); +kmem_freepages (cachep,page_slab (page));/* * From now on, we don't use freelist
@@ -2578,7 +2582,7 @@ static struct page *cache_grow_begin(struct kmem_cache *cachep,* Get mem for the objs. Attempt to allocate a physical page from * 'nodeid'. */ - page =kmem_getpages (cachep, local_flags, nodeid); + page =slab_page (kmem_getpages (cachep, local_flags, nodeid));if (!page) goto failed;
@@ -2620,7 +2624,7 @@ static struct page *cache_grow_begin(struct kmem_cache *cachep,return page; opps1: -kmem_freepages (cachep, page); +kmem_freepages (cachep,page_slab (page)); failed:if (gfpflags_allow_blocking (local_flags))local_irq_disable ();
Signature · 1 line
2.33.1
PATCH v2 20/33 mm/slab: Convert most struct page to struct slab by spatch
In reply to Vlastimil BabkaThe majority of conversion from struct page to struct slab in SLAB internals can be delegated to a coccinelle semantic patch. This includes renaming of variables with 'page' in name to 'slab', and similar.
Big thanks to Julia Lawall and Luis Chamberlain for help with coccinelle.
// Options: --include-headers --no-includes --smpl-spacing mm/slab.c // Note: needs coccinelle 1.1.1 to avoid breaking whitespace, and ocaml for the // embedded script script
// build list of functions for applying the next rule @initialize:ocaml@
@@
let ok_function p =
not (List.mem (List.hd p).current_element ["kmem_getpages";"kmem_freepages"])
// convert the type in selected functions
@@
position p : script:ocaml() { ok_function p };
@@
- struct page@p
+ struct slab
@@
@@
-PageSlabPfmemalloc(page)
+slab_test_pfmemalloc(slab)
@@
@@
-ClearPageSlabPfmemalloc(page)
+slab_clear_pfmemalloc(slab)
@@
@@
obj_to_index(
...,
- page
+ slab_page(slab)
,...)
// for all functions, change any "struct slab *page" parameter to "struct slab // *slab" in the signature, and generally all occurences of "page" to "slab" in // the body - with some special cases.
@@
identifier fn;
expression E;
@@
fn(...,
- struct slab *page
+ struct slab *slab
,...)
{
<...
(
- int page_node;
+ int slab_node;
|
- page_node
+ slab_node
|
- page_slab(page)
+ slab
|
- page_address(page)
+ slab_address(slab)
|
- page_size(page)
+ slab_size(slab)
|
- page_to_nid(page)
+ slab_nid(slab)
|
- virt_to_head_page(E)
+ virt_to_slab(E)
|
- page
+ slab
)
...>
}
// rename a function parameter
@@
identifier fn;
expression E;
@@
fn(...,
- int page_node
+ int slab_node
,...)
{
<...
- page_node
+ slab_node
...>
}
// functions converted by previous rules that were temporarily called using // slab_page(E) so we want to remove the wrapper now that they accept struct // slab ptr directly
@@
identifier fn =~ "index_to_obj";
expression E;
@@
fn(...,
- slab_page(E)
+ E
,...)
// functions that were returning struct page ptr and now will return struct // slab ptr, including slab_page() wrapper removal
@@
identifier fn =~ "cache_grow_begin|get_valid_first_slab|get_first_slab"; expression E;
@@
fn(...)
{
<...
- slab_page(E)
+ E
...>
}
// rename any former struct page * declarations
@@
@@
struct slab *
-page
+slab
;
// all functions (with exceptions) with a local "struct slab *page" variable // that will be renamed to "struct slab *slab"
@@
identifier fn !~ "kmem_getpages|kmem_freepages";
expression E;
@@
fn(...)
{
<...
(
- page_slab(page)
+ slab
|
- page_to_nid(page)
+ slab_nid(slab)
|
- kasan_poison_slab(page)
+ kasan_poison_slab(slab_page(slab))
|
- page_address(page)
+ slab_address(slab)
|
- page_size(page)
+ slab_size(slab)
|
- page->pages
+ slab->slabs
|
- page = virt_to_head_page(E)
+ slab = virt_to_slab(E)
|
- virt_to_head_page(E)
+ virt_to_slab(E)
|
- page
+ slab
)
...>
}
--- mm/slab.c | 360 +++++++++++++++++++++++++++--------------------------- 1 file changed, 180 insertions(+), 180 deletions(-)
mm/slab.c +180 -180
@@ -218,7 +218,7 @@ static void cache_reap(struct work_struct *unused);static inline void fixup_objfreelist_debug (struct kmem_cache *cachep,void **list);static inline void fixup_slab_list (struct kmem_cache *cachep, -struct kmem_cache_node *n,struct page *page, +struct kmem_cache_node *n,struct slab *slab,void **list);static int slab_early_init =1 ;
@@ -373,9 +373,9 @@ static int slab_max_order = SLAB_MAX_ORDER_LO;static bool slab_max_order_set__initdata ;static inline void *index_to_obj (struct kmem_cache *cache, -const struct page *page,unsigned int idx) +const struct slab *slab,unsigned int idx) { -return page->s_mem + cache->size * idx; +return slab->s_mem + cache->size * idx; }#define BOOT_CPUCACHE_ENTRIES 1
@@ -550,17 +550,17 @@ static struct array_cache *alloc_arraycache(int node, int entries, } static noinline void cache_free_pfmemalloc(struct kmem_cache *cachep, -struct page *page,void *objp) +struct slab *slab,void *objp) {struct kmem_cache_node *n; -int page_node; +int slab_node;LIST_HEAD (list); - page_node =page_to_nid (page); - n =get_node (cachep, page_node); + slab_node =slab_nid (slab); + n =get_node (cachep, slab_node);spin_lock (&n->list_lock); -free_block (cachep, &objp,1 , page_node, &list); +free_block (cachep, &objp,1 , slab_node, &list);spin_unlock (&n->list_lock);slabs_destroy (cachep, &list);
@@ -761,7 +761,7 @@ static void drain_alien_cache(struct kmem_cache *cachep, }static int __cache_free_alien (struct kmem_cache *cachep,void *objp, -int node,int page_node) +int node,int slab_node) {struct kmem_cache_node *n;struct alien_cache *alien =NULL ;
@@ -770,21 +770,21 @@ static int __cache_free_alien(struct kmem_cache *cachep, void *objp, n =get_node (cachep, node);STATS_INC_NODEFREES (cachep); -if (n->alien && n->alien[page_node]) { - alien = n->alien[page_node]; +if (n->alien && n->alien[slab_node]) { + alien = n->alien[slab_node]; ac = &alien->ac;spin_lock (&alien->lock);if (unlikely (ac->avail == ac->limit)) {STATS_INC_ACOVERFLOW (cachep); -__drain_alien_cache (cachep, ac, page_node, &list); +__drain_alien_cache (cachep, ac, slab_node, &list); }__free_one (ac, objp);spin_unlock (&alien->lock);slabs_destroy (cachep, &list); }else { - n =get_node (cachep, page_node); + n =get_node (cachep, slab_node);spin_lock (&n->list_lock); -free_block (cachep, &objp,1 , page_node, &list); +free_block (cachep, &objp,1 , slab_node, &list);spin_unlock (&n->list_lock);slabs_destroy (cachep, &list); }
@@ -1557,18 +1557,18 @@ static void check_poison_obj(struct kmem_cache *cachep, void *objp)/* Print some data about the neighboring objects, if they * exist: */ -struct page *page =virt_to_head_page (objp); +struct slab *slab =virt_to_slab (objp);unsigned int objnr; - objnr =obj_to_index (cachep, page, objp); + objnr =obj_to_index (cachep,slab_page (slab), objp);if (objnr) { - objp =index_to_obj (cachep, page, objnr -1 ); + objp =index_to_obj (cachep, slab, objnr -1 ); realobj = (char *)objp +obj_offset (cachep);pr_err ("Prev obj: start=%px, len=%d\n" , realobj, size);print_objinfo (cachep, objp,2 ); }if (objnr +1 < cachep->num) { - objp =index_to_obj (cachep, page, objnr +1 ); + objp =index_to_obj (cachep, slab, objnr +1 ); realobj = (char *)objp +obj_offset (cachep);pr_err ("Next obj: start=%px, len=%d\n" , realobj, size);print_objinfo (cachep, objp,2 );
@@ -1579,17 +1579,17 @@ static void check_poison_obj(struct kmem_cache *cachep, void *objp)#if DEBUG static void slab_destroy_debugcheck (struct kmem_cache *cachep, -struct page *page) +struct slab *slab) {int i;if (OBJFREELIST_SLAB (cachep) && cachep->flags &SLAB_POISON ) { -poison_obj (cachep, page->freelist -obj_offset (cachep), +poison_obj (cachep, slab->freelist -obj_offset (cachep),POISON_FREE ); }for (i =0 ; i < cachep->num; i++) { -void *objp =index_to_obj (cachep, page, i); +void *objp =index_to_obj (cachep, slab, i);if (cachep->flags &SLAB_POISON ) {check_poison_obj (cachep, objp);
@@ -1605,7 +1605,7 @@ static void slab_destroy_debugcheck(struct kmem_cache *cachep, }#else static void slab_destroy_debugcheck (struct kmem_cache *cachep, -struct page *page) +struct slab *slab) { }#endif
@@ -1619,16 +1619,16 @@ static void slab_destroy_debugcheck(struct kmem_cache *cachep,* Before calling the slab page must have been unlinked from the cache. The * kmem_cache_node ->list_lock is not held/needed. */ -static void slab_destroy (struct kmem_cache *cachep,struct page *page) +static void slab_destroy (struct kmem_cache *cachep,struct slab *slab) {void *freelist; - freelist = page->freelist; -slab_destroy_debugcheck (cachep, page); + freelist = slab->freelist; +slab_destroy_debugcheck (cachep, slab);if (unlikely (cachep->flags &SLAB_TYPESAFE_BY_RCU )) -call_rcu (&page->rcu_head, kmem_rcu_free); +call_rcu (&slab->rcu_head, kmem_rcu_free);else -kmem_freepages (cachep,page_slab (page)); +kmem_freepages (cachep, slab);/* * From now on, we don't use freelist
@@ -1644,11 +1644,11 @@ static void slab_destroy(struct kmem_cache *cachep, struct page *page)*/ static void slabs_destroy (struct kmem_cache *cachep,struct list_head *list) { -struct page *page, *n; +struct slab *slab, *n; -list_for_each_entry_safe (page, n, list, slab_list) { -list_del (&page->slab_list); -slab_destroy (cachep, page); +list_for_each_entry_safe (slab, n, list, slab_list) { +list_del (&slab->slab_list); +slab_destroy (cachep, slab); } }
@@ -2198,7 +2198,7 @@ static int drain_freelist(struct kmem_cache *cache, {struct list_head *p;int nr_freed; -struct page *page; +struct slab *slab; nr_freed =0 ;while (nr_freed < tofree && !list_empty (&n->slabs_free)) {
@@ -2210,8 +2210,8 @@ static int drain_freelist(struct kmem_cache *cache, goto out; } - page = list_entry(p, struct page, slab_list); -list_del (&page->slab_list); + slab = list_entry(p, struct slab, slab_list); +list_del (&slab->slab_list); n->free_slabs--; n->total_slabs--;/*
@@ -2220,7 +2220,7 @@ static int drain_freelist(struct kmem_cache *cache,*/ n->free_objects -= cache->num;spin_unlock_irq (&n->list_lock); -slab_destroy (cache, page); +slab_destroy (cache, slab); nr_freed++; } out:
@@ -2295,14 +2295,14 @@ void __kmem_cache_release(struct kmem_cache *cachep)* which are all initialized during kmem_cache_init(). */ static void *alloc_slabmgmt (struct kmem_cache *cachep, -struct page *page,int colour_off, +struct slab *slab,int colour_off,gfp_t local_flags,int nodeid) {void *freelist; -void *addr =page_address (page); +void *addr =slab_address (slab); - page->s_mem = addr + colour_off; - page->active =0 ; + slab->s_mem = addr + colour_off; + slab->active =0 ;if (OBJFREELIST_SLAB (cachep)) freelist =NULL ;
@@ -2319,24 +2319,24 @@ static void *alloc_slabmgmt(struct kmem_cache *cachep,return freelist; } -static inline freelist_idx_t get_free_obj (struct page *page,unsigned int idx) +static inline freelist_idx_t get_free_obj (struct slab *slab,unsigned int idx) { -return ((freelist_idx_t *)page->freelist)[idx]; +return ((freelist_idx_t *) slab->freelist)[idx]; } -static inline void set_free_obj (struct page *page, +static inline void set_free_obj (struct slab *slab,unsigned int idx,freelist_idx_t val) { - ((freelist_idx_t *)(page->freelist))[idx] = val; + ((freelist_idx_t *)(slab->freelist))[idx] = val; } -static void cache_init_objs_debug (struct kmem_cache *cachep,struct page *page) +static void cache_init_objs_debug (struct kmem_cache *cachep,struct slab *slab) {#if DEBUG int i;for (i =0 ; i < cachep->num; i++) { -void *objp =index_to_obj (cachep, page, i); +void *objp =index_to_obj (cachep, slab, i);if (cachep->flags &SLAB_STORE_USER ) *dbg_userword (cachep, objp) =NULL ;
@@ -2420,17 +2420,17 @@ static freelist_idx_t next_random_slot(union freelist_init_state *state) }/* Swap two freelist entries */ -static void swap_free_obj (struct page *page,unsigned int a,unsigned int b) +static void swap_free_obj (struct slab *slab,unsigned int a,unsigned int b) { -swap (((freelist_idx_t *)page->freelist)[a], - ((freelist_idx_t *)page->freelist)[b]); +swap (((freelist_idx_t *) slab->freelist)[a], + ((freelist_idx_t *) slab->freelist)[b]); }/* * Shuffle the freelist initialization state based on pre-computed lists. * return true if the list was successfully shuffled, false otherwise. */ -static bool shuffle_freelist (struct kmem_cache *cachep,struct page *page) +static bool shuffle_freelist (struct kmem_cache *cachep,struct slab *slab) {unsigned int objfreelist =0 , i, rand, count = cachep->num;union freelist_init_state state;
@@ -2447,7 +2447,7 @@ static bool shuffle_freelist(struct kmem_cache *cachep, struct page *page) objfreelist = count -1 ; else objfreelist =next_random_slot (&state); - page->freelist =index_to_obj (cachep, page, objfreelist) + + slab->freelist =index_to_obj (cachep, slab, objfreelist) +obj_offset (cachep); count--; }
@@ -2458,51 +2458,51 @@ static bool shuffle_freelist(struct kmem_cache *cachep, struct page *page)*/ if (!precomputed) {for (i =0 ; i < count; i++) -set_free_obj (page, i, i); +set_free_obj (slab, i, i);/* Fisher-Yates shuffle */ for (i = count -1 ; i >0 ; i--) { rand =prandom_u32_state (&state.rnd_state); rand %= (i +1 ); -swap_free_obj (page, i, rand); +swap_free_obj (slab, i, rand); } }else {for (i =0 ; i < count; i++) -set_free_obj (page, i,next_random_slot (&state)); +set_free_obj (slab, i,next_random_slot (&state)); }if (OBJFREELIST_SLAB (cachep)) -set_free_obj (page, cachep->num -1 , objfreelist); +set_free_obj (slab, cachep->num -1 , objfreelist);return true; }#else static inline bool shuffle_freelist (struct kmem_cache *cachep, -struct page *page) +struct slab *slab) {return false; }#endif /* CONFIG_SLAB_FREELIST_RANDOM */ static void cache_init_objs (struct kmem_cache *cachep, -struct page *page) +struct slab *slab) {int i;void *objp;bool shuffled; -cache_init_objs_debug (cachep, page); +cache_init_objs_debug (cachep, slab);/* Try to randomize the freelist if enabled */ - shuffled =shuffle_freelist (cachep, page); + shuffled =shuffle_freelist (cachep, slab);if (!shuffled &&OBJFREELIST_SLAB (cachep)) { - page->freelist =index_to_obj (cachep, page, cachep->num -1 ) + + slab->freelist =index_to_obj (cachep, slab, cachep->num -1 ) +obj_offset (cachep); }for (i =0 ; i < cachep->num; i++) { - objp =index_to_obj (cachep, page, i); + objp =index_to_obj (cachep, slab, i); objp =kasan_init_slab_obj (cachep, objp);/* constructor could break poison info */
@@ -2513,48 +2513,48 @@ static void cache_init_objs(struct kmem_cache *cachep, }if (!shuffled) -set_free_obj (page, i, i); +set_free_obj (slab, i, i); } } -static void *slab_get_obj (struct kmem_cache *cachep,struct page *page) +static void *slab_get_obj (struct kmem_cache *cachep,struct slab *slab) {void *objp; - objp =index_to_obj (cachep, page,get_free_obj (page, page->active)); - page->active++; + objp =index_to_obj (cachep, slab,get_free_obj (slab, slab->active)); + slab->active++;return objp; }static void slab_put_obj (struct kmem_cache *cachep, -struct page *page,void *objp) +struct slab *slab,void *objp) { -unsigned int objnr =obj_to_index (cachep, page, objp); +unsigned int objnr =obj_to_index (cachep,slab_page (slab), objp);#if DEBUG unsigned int i;/* Verify double free bug */ -for (i = page->active; i < cachep->num; i++) { -if (get_free_obj (page, i) == objnr) { +for (i = slab->active; i < cachep->num; i++) { +if (get_free_obj (slab, i) == objnr) {pr_err ("slab: double free detected in cache '%s', objp %px\n" , cachep->name, objp);BUG (); } }#endif - page->active--; -if (!page->freelist) - page->freelist = objp +obj_offset (cachep); + slab->active--; +if (!slab->freelist) + slab->freelist = objp +obj_offset (cachep); -set_free_obj (page, page->active, objnr); +set_free_obj (slab, slab->active, objnr); }/* * Grow (by 1) the number of slabs within a cache. This is called by * kmem_cache_alloc() when there are no active objs left in a cache. */ -static struct page *cache_grow_begin (struct kmem_cache *cachep, +static struct slab *cache_grow_begin (struct kmem_cache *cachep,gfp_t flags,int nodeid) {void *freelist;
@@ -2562,7 +2562,7 @@ static struct page *cache_grow_begin(struct kmem_cache *cachep,gfp_t local_flags;int page_node;struct kmem_cache_node *n; -struct page *page; +struct slab *slab;/* * Be lazy and only check for valid flags here, keeping it out of the
@@ -2582,11 +2582,11 @@ static struct page *cache_grow_begin(struct kmem_cache *cachep,* Get mem for the objs. Attempt to allocate a physical page from * 'nodeid'. */ - page =slab_page (kmem_getpages (cachep, local_flags, nodeid)); -if (!page) + slab =kmem_getpages (cachep, local_flags, nodeid); +if (!slab) goto failed; - page_node =page_to_nid (page); + page_node =slab_nid (slab); n =get_node (cachep, page_node);/* Get colour for the slab, and cal the next value. */
@@ -2605,55 +2605,55 @@ static struct page *cache_grow_begin(struct kmem_cache *cachep,* page_address() in the latter returns a non-tagged pointer, * as it should be for slab pages. */ -kasan_poison_slab (page); +kasan_poison_slab (slab_page (slab));/* Get slab management. */ - freelist =alloc_slabmgmt (cachep, page, offset, + freelist =alloc_slabmgmt (cachep, slab, offset, local_flags & ~GFP_CONSTRAINT_MASK , page_node);if (OFF_SLAB (cachep) && !freelist) goto opps1; - page->slab_cache = cachep; - page->freelist = freelist; + slab->slab_cache = cachep; + slab->freelist = freelist; -cache_init_objs (cachep, page); +cache_init_objs (cachep, slab);if (gfpflags_allow_blocking (local_flags))local_irq_disable (); -return page; +return slab; opps1: -kmem_freepages (cachep,page_slab (page)); +kmem_freepages (cachep, slab); failed:if (gfpflags_allow_blocking (local_flags))local_irq_disable ();return NULL ; } -static void cache_grow_end (struct kmem_cache *cachep,struct page *page) +static void cache_grow_end (struct kmem_cache *cachep,struct slab *slab) {struct kmem_cache_node *n;void *list =NULL ;check_irq_off (); -if (!page) +if (!slab)return ; -INIT_LIST_HEAD (&page->slab_list); - n =get_node (cachep,page_to_nid (page)); +INIT_LIST_HEAD (&slab->slab_list); + n =get_node (cachep,slab_nid (slab));spin_lock (&n->list_lock); n->total_slabs++; -if (!page->active) { -list_add_tail (&page->slab_list, &n->slabs_free); +if (!slab->active) { +list_add_tail (&slab->slab_list, &n->slabs_free); n->free_slabs++; }else -fixup_slab_list (cachep, n, page, &list); +fixup_slab_list (cachep, n, slab, &list);STATS_INC_GROWN (cachep); - n->free_objects += cachep->num - page->active; + n->free_objects += cachep->num - slab->active;spin_unlock (&n->list_lock);fixup_objfreelist_debug (cachep, &list);
@@ -2701,13 +2701,13 @@ static void *cache_free_debugcheck(struct kmem_cache *cachep, void *objp, unsigned long caller) {unsigned int objnr; -struct page *page; +struct slab *slab;BUG_ON (virt_to_cache (objp) != cachep); objp -=obj_offset (cachep);kfree_debugcheck (objp); - page =virt_to_head_page (objp); + slab =virt_to_slab (objp);if (cachep->flags &SLAB_RED_ZONE ) {verify_redzone_free (cachep, objp);
@@ -2717,10 +2717,10 @@ static void *cache_free_debugcheck(struct kmem_cache *cachep, void *objp,if (cachep->flags &SLAB_STORE_USER ) *dbg_userword (cachep, objp) = (void *)caller; - objnr =obj_to_index (cachep, page, objp); + objnr =obj_to_index (cachep,slab_page (slab), objp);BUG_ON (objnr >= cachep->num); -BUG_ON (objp !=index_to_obj (cachep, page, objnr)); +BUG_ON (objp !=index_to_obj (cachep, slab, objnr));if (cachep->flags &SLAB_POISON ) {poison_obj (cachep, objp,POISON_FREE );
@@ -2750,97 +2750,97 @@ static inline void fixup_objfreelist_debug(struct kmem_cache *cachep, }static inline void fixup_slab_list (struct kmem_cache *cachep, -struct kmem_cache_node *n,struct page *page, +struct kmem_cache_node *n,struct slab *slab,void **list) {/* move slabp to correct slabp list: */ -list_del (&page->slab_list); -if (page->active == cachep->num) { -list_add (&page->slab_list, &n->slabs_full); +list_del (&slab->slab_list); +if (slab->active == cachep->num) { +list_add (&slab->slab_list, &n->slabs_full);if (OBJFREELIST_SLAB (cachep)) {#if DEBUG /* Poisoning will be done without holding the lock */ if (cachep->flags &SLAB_POISON ) { -void **objp = page->freelist; +void **objp = slab->freelist; *objp = *list; *list = objp; }#endif - page->freelist =NULL ; + slab->freelist =NULL ; } }else -list_add (&page->slab_list, &n->slabs_partial); +list_add (&slab->slab_list, &n->slabs_partial); }/* Try to find non-pfmemalloc slab if needed */ -static noinline struct page *get_valid_first_slab(struct kmem_cache_node *n, -struct page *page,bool pfmemalloc) +static noinline struct slab *get_valid_first_slab(struct kmem_cache_node *n, +struct slab *slab,bool pfmemalloc) { -if (!page) +if (!slab)return NULL ;if (pfmemalloc) -return page; +return slab; -if (!PageSlabPfmemalloc (page)) -return page; +if (!slab_test_pfmemalloc (slab)) +return slab;/* No need to keep pfmemalloc slab if we have enough free objects */ if (n->free_objects > n->free_limit) { -ClearPageSlabPfmemalloc (page); -return page; +slab_clear_pfmemalloc (slab); +return slab; }/* Move pfmemalloc slab to the end of list to speed up next search */ -list_del (&page->slab_list); -if (!page->active) { -list_add_tail (&page->slab_list, &n->slabs_free); +list_del (&slab->slab_list); +if (!slab->active) { +list_add_tail (&slab->slab_list, &n->slabs_free); n->free_slabs++; }else -list_add_tail (&page->slab_list, &n->slabs_partial); +list_add_tail (&slab->slab_list, &n->slabs_partial); -list_for_each_entry (page, &n->slabs_partial, slab_list) { -if (!PageSlabPfmemalloc (page)) -return page; +list_for_each_entry (slab, &n->slabs_partial, slab_list) { +if (!slab_test_pfmemalloc (slab)) +return slab; } n->free_touched =1 ; -list_for_each_entry (page, &n->slabs_free, slab_list) { -if (!PageSlabPfmemalloc (page)) { +list_for_each_entry (slab, &n->slabs_free, slab_list) { +if (!slab_test_pfmemalloc (slab)) { n->free_slabs--; -return page; +return slab; } }return NULL ; } -static struct page *get_first_slab (struct kmem_cache_node *n,bool pfmemalloc) +static struct slab *get_first_slab (struct kmem_cache_node *n,bool pfmemalloc) { -struct page *page; +struct slab *slab;assert_spin_locked (&n->list_lock); - page = list_first_entry_or_null(&n->slabs_partial, struct page, + slab = list_first_entry_or_null(&n->slabs_partial, struct slab, slab_list); -if (!page) { +if (!slab) { n->free_touched =1 ; - page = list_first_entry_or_null(&n->slabs_free, struct page, + slab = list_first_entry_or_null(&n->slabs_free, struct slab, slab_list); -if (page) +if (slab) n->free_slabs--; }if (sk_memalloc_socks ()) - page =get_valid_first_slab (n, page, pfmemalloc); + slab =get_valid_first_slab (n, slab, pfmemalloc); -return page; +return slab; } static noinline void *cache_alloc_pfmemalloc(struct kmem_cache *cachep,struct kmem_cache_node *n,gfp_t flags) { -struct page *page; +struct slab *slab;void *obj;void *list =NULL ;
@@ -2848,16 +2848,16 @@ static noinline void *cache_alloc_pfmemalloc(struct kmem_cache *cachep,return NULL ;spin_lock (&n->list_lock); - page =get_first_slab (n, true); -if (!page) { + slab =get_first_slab (n, true); +if (!slab) {spin_unlock (&n->list_lock);return NULL ; } - obj =slab_get_obj (cachep, page); + obj =slab_get_obj (cachep, slab); n->free_objects--; -fixup_slab_list (cachep, n, page, &list); +fixup_slab_list (cachep, n, slab, &list);spin_unlock (&n->list_lock);fixup_objfreelist_debug (cachep, &list);
@@ -2870,20 +2870,20 @@ static noinline void *cache_alloc_pfmemalloc(struct kmem_cache *cachep,* or cache_grow_end() for new slab */ static __always_inline int alloc_block(struct kmem_cache *cachep, -struct array_cache *ac,struct page *page,int batchcount) +struct array_cache *ac,struct slab *slab,int batchcount) {/* * There must be at least one object available for * allocation. */ -BUG_ON (page->active >= cachep->num); +BUG_ON (slab->active >= cachep->num); -while (page->active < cachep->num && batchcount--) { +while (slab->active < cachep->num && batchcount--) {STATS_INC_ALLOCED (cachep);STATS_INC_ACTIVE (cachep);STATS_SET_HIGH (cachep); - ac->entry[ac->avail++] =slab_get_obj (cachep, page); + ac->entry[ac->avail++] =slab_get_obj (cachep, slab); }return batchcount;
@@ -2896,7 +2896,7 @@ static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)struct array_cache *ac, *shared;int node;void *list =NULL ; -struct page *page; +struct slab *slab;check_irq_off (); node =numa_mem_id ();
@@ -2929,14 +2929,14 @@ static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)while (batchcount >0 ) {/* Get slab alloc is to come from. */ - page =get_first_slab (n, false); -if (!page) + slab =get_first_slab (n, false); +if (!slab) goto must_grow;check_spinlock_acquired (cachep); - batchcount =alloc_block (cachep, ac, page, batchcount); -fixup_slab_list (cachep, n, page, &list); + batchcount =alloc_block (cachep, ac, slab, batchcount); +fixup_slab_list (cachep, n, slab, &list); } must_grow:
@@ -2955,16 +2955,16 @@ static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)return obj; } - page =cache_grow_begin (cachep,gfp_exact_node (flags), node); + slab =cache_grow_begin (cachep,gfp_exact_node (flags), node);/* * cache_grow_begin() can reenable interrupts, * then ac could change. */ ac =cpu_cache_get (cachep); -if (!ac->avail && page) -alloc_block (cachep, ac, page, batchcount); -cache_grow_end (cachep, page); +if (!ac->avail && slab) +alloc_block (cachep, ac, slab, batchcount); +cache_grow_end (cachep, slab);if (!ac->avail)return NULL ;
@@ -3094,7 +3094,7 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)struct zone *zone;enum zone_type highest_zoneidx =gfp_zone (flags);void *obj =NULL ; -struct page *page; +struct slab *slab;int nid;unsigned int cpuset_mems_cookie;
@@ -3130,10 +3130,10 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)* We may trigger various forms of reclaim on the allowed * set and go into memory reserves if necessary. */ - page =cache_grow_begin (cache, flags,numa_mem_id ()); -cache_grow_end (cache, page); -if (page) { - nid =page_to_nid (page); + slab =cache_grow_begin (cache, flags,numa_mem_id ()); +cache_grow_end (cache, slab); +if (slab) { + nid =slab_nid (slab); obj =____cache_alloc_node (cache,gfp_exact_node (flags), nid);
@@ -3157,7 +3157,7 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)static void *____cache_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid) { -struct page *page; +struct slab *slab;struct kmem_cache_node *n;void *obj =NULL ;void *list =NULL ;
@@ -3168,8 +3168,8 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags,check_irq_off ();spin_lock (&n->list_lock); - page =get_first_slab (n, false); -if (!page) + slab =get_first_slab (n, false); +if (!slab) goto must_grow;check_spinlock_acquired_node (cachep, nodeid);
@@ -3178,12 +3178,12 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags,STATS_INC_ACTIVE (cachep);STATS_SET_HIGH (cachep); -BUG_ON (page->active == cachep->num); +BUG_ON (slab->active == cachep->num); - obj =slab_get_obj (cachep, page); + obj =slab_get_obj (cachep, slab); n->free_objects--; -fixup_slab_list (cachep, n, page, &list); +fixup_slab_list (cachep, n, slab, &list);spin_unlock (&n->list_lock);fixup_objfreelist_debug (cachep, &list);
@@ -3191,12 +3191,12 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, must_grow:spin_unlock (&n->list_lock); - page =cache_grow_begin (cachep,gfp_exact_node (flags), nodeid); -if (page) { + slab =cache_grow_begin (cachep,gfp_exact_node (flags), nodeid); +if (slab) {/* This slab isn't counted yet so don't update free_objects */ - obj =slab_get_obj (cachep, page); + obj =slab_get_obj (cachep, slab); } -cache_grow_end (cachep, page); +cache_grow_end (cachep, slab);return obj ? obj :fallback_alloc (cachep, flags); }
@@ -3326,40 +3326,40 @@ static void free_block(struct kmem_cache *cachep, void **objpp, {int i;struct kmem_cache_node *n =get_node (cachep, node); -struct page *page; +struct slab *slab; n->free_objects += nr_objects;for (i =0 ; i < nr_objects; i++) {void *objp; -struct page *page; +struct slab *slab; objp = objpp[i]; - page =virt_to_head_page (objp); -list_del (&page->slab_list); + slab =virt_to_slab (objp); +list_del (&slab->slab_list);check_spinlock_acquired_node (cachep, node); -slab_put_obj (cachep, page, objp); +slab_put_obj (cachep, slab, objp);STATS_DEC_ACTIVE (cachep);/* fixup slab chains */ -if (page->active ==0 ) { -list_add (&page->slab_list, &n->slabs_free); +if (slab->active ==0 ) { +list_add (&slab->slab_list, &n->slabs_free); n->free_slabs++; }else {/* Unconditionally move a slab to the end of the * partial list on free - maximum time for the * other objects to be freed, too. */ -list_add_tail (&page->slab_list, &n->slabs_partial); +list_add_tail (&slab->slab_list, &n->slabs_partial); } }while (n->free_objects > n->free_limit && !list_empty (&n->slabs_free)) { n->free_objects -= cachep->num; - page = list_last_entry(&n->slabs_free, struct page, slab_list); -list_move (&page->slab_list, list); + slab = list_last_entry(&n->slabs_free, struct slab, slab_list); +list_move (&slab->slab_list, list); n->free_slabs--; n->total_slabs--; }
@@ -3395,10 +3395,10 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)#if STATS {int i =0 ; -struct page *page; +struct slab *slab; -list_for_each_entry (page, &n->slabs_free, slab_list) { -BUG_ON (page->active); +list_for_each_entry (slab, &n->slabs_free, slab_list) { +BUG_ON (slab->active); i++; }
@@ -3474,10 +3474,10 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, }if (sk_memalloc_socks ()) { -struct page *page =virt_to_head_page (objp); +struct slab *slab =virt_to_slab (objp); -if (unlikely (PageSlabPfmemalloc (page))) { -cache_free_pfmemalloc (cachep, page, objp); +if (unlikely (slab_test_pfmemalloc (slab))) { +cache_free_pfmemalloc (cachep, slab, objp);return ; } }
@@ -3664,7 +3664,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) kpp->kp_data_offset =obj_offset (cachep); slab =virt_to_slab (objp); objnr =obj_to_index (cachep,slab_page (slab), objp); - objp =index_to_obj (cachep,slab_page (slab), objnr); + objp =index_to_obj (cachep, slab, objnr); kpp->kp_objp = objp;if (DEBUG && cachep->flags &SLAB_STORE_USER ) kpp->kp_ret = *dbg_userword (cachep, objp);
@@ -4188,7 +4188,7 @@ void __check_heap_object(const void *ptr, unsigned long n,if (is_kfence_address (ptr)) offset = ptr -kfence_object_start (ptr);else - offset = ptr -index_to_obj (cachep,slab_page (slab), objnr) -obj_offset (cachep); + offset = ptr -index_to_obj (cachep, slab, objnr) -obj_offset (cachep);/* Allow address range falling entirely within usercopy region. */ if (offset >= cachep->useroffset &&
Signature · 1 line
2.33.1
PATCH v2 21/33 mm/slab: Finish struct page to struct slab conversion
In reply to Vlastimil BabkaChange cache_free_alien() to use slab_nid(virt_to_slab()). Otherwise just update of comments and some remaining variable names.
--- mm/slab.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
mm/slab.c +10 -10
@@ -793,16 +793,16 @@ static int __cache_free_alien(struct kmem_cache *cachep, void *objp,static inline int cache_free_alien (struct kmem_cache *cachep,void *objp) { -int page_node =page_to_nid (virt_to_page (objp)); +int slab_node =slab_nid (virt_to_slab (objp));int node =numa_mem_id ();/* * Make sure we are not freeing a object from another node to the array * cache on this cpu. */ -if (likely (node == page_node)) +if (likely (node == slab_node))return 0 ; -return __cache_free_alien (cachep, objp, node, page_node); +return __cache_free_alien (cachep, objp, node, slab_node); }/*
@@ -1613,10 +1613,10 @@ static void slab_destroy_debugcheck(struct kmem_cache *cachep,/** * slab_destroy - destroy and release all objects in a slab * @cachep: cache pointer being destroyed -* @page: page pointer being destroyed +* @slab: slab being destroyed * -* Destroy all the objs in a slab page, and release the mem back to the system. -* Before calling the slab page must have been unlinked from the cache. The +* Destroy all the objs in a slab, and release the mem back to the system. +* Before calling the slab must have been unlinked from the cache. The * kmem_cache_node ->list_lock is not held/needed. */ static void slab_destroy (struct kmem_cache *cachep,struct slab *slab)
@@ -2560,7 +2560,7 @@ static struct slab *cache_grow_begin(struct kmem_cache *cachep,void *freelist;size_t offset;gfp_t local_flags; -int page_node; +int slab_node;struct kmem_cache_node *n;struct slab *slab;
@@ -2586,8 +2586,8 @@ static struct slab *cache_grow_begin(struct kmem_cache *cachep,if (!slab) goto failed; - page_node =slab_nid (slab); - n =get_node (cachep, page_node); + slab_node =slab_nid (slab); + n =get_node (cachep, slab_node);/* Get colour for the slab, and cal the next value. */ n->colour_next++;
@@ -2609,7 +2609,7 @@ static struct slab *cache_grow_begin(struct kmem_cache *cachep,/* Get slab management. */ freelist =alloc_slabmgmt (cachep, slab, offset, - local_flags & ~GFP_CONSTRAINT_MASK , page_node); + local_flags & ~GFP_CONSTRAINT_MASK , slab_node);if (OFF_SLAB (cachep) && !freelist) goto opps1;
Signature · 1 line
2.33.1
PATCH v2 22/33 mm: Convert struct page to struct slab in functions used by other subsystems
In reply to Vlastimil BabkaKASAN, KFENCE and memcg interact with SLAB or SLUB internals through functions nearest_obj(), obj_to_index() and objs_per_slab() that use struct page as parameter. This patch converts it to struct slab including all callers, through a coccinelle semantic patch.
// Options: --include-headers --no-includes --smpl-spacing include/linux/slab_def.h include/linux/slub_def.h mm/slab.h mm/kasan/*.c mm/kfence/kfence_test.c mm/memcontrol.c mm/slab.c mm/slub.c // Note: needs coccinelle 1.1.1 to avoid breaking whitespace
@@
@@
-objs_per_slab_page(
+objs_per_slab(
...
)
{ ... }
@@
@@
-objs_per_slab_page(
+objs_per_slab(
...
)
@@
identifier fn =~ "obj_to_index|objs_per_slab";
@@
fn(...,
- const struct page *page
+ const struct slab *slab
,...)
{
<...
(
- page_address(page)
+ slab_address(slab)
|
- page
+ slab
)
...>
}
@@
identifier fn =~ "nearest_obj";
@@
fn(...,
- struct page *page
+ const struct slab *slab
,...)
{
<...
(
- page_address(page)
+ slab_address(slab)
|
- page
+ slab
)
...>
}
@@
identifier fn =~ "nearest_obj|obj_to_index|objs_per_slab"; expression E;
@@
fn(...,
(
- slab_page(E)
+ E
|
- virt_to_page(E)
+ virt_to_slab(E)
|
- virt_to_head_page(E)
+ virt_to_slab(E)
|
- page
+ page_slab(page)
)
,...)
--- include/linux/slab_def.h | 16 ++++++++-------- include/linux/slub_def.h | 18 +++++++++--------- mm/kasan/common.c | 4 ++-- mm/kasan/generic.c | 2 +- mm/kasan/report.c | 2 +- mm/kasan/report_tags.c | 2 +- mm/kfence/kfence_test.c | 4 ++-- mm/memcontrol.c | 4 ++-- mm/slab.c | 10 +++++----- mm/slab.h | 4 ++-- mm/slub.c | 2 +- 11 files changed, 34 insertions(+), 34 deletions(-)
include/linux/slab_def.h +8 -8
@@ -87,11 +87,11 @@ struct kmem_cache {struct kmem_cache_node *node[MAX_NUMNODES ]; }; -static inline void *nearest_obj (struct kmem_cache *cache,struct page *page, +static inline void *nearest_obj (struct kmem_cache *cache,const struct slab *slab,void *x) { -void *object = x - (x - page->s_mem) % cache->size; -void *last_object = page->s_mem + (cache->num -1 ) * cache->size; +void *object = x - (x - slab->s_mem) % cache->size; +void *last_object = slab->s_mem + (cache->num -1 ) * cache->size;if (unlikely (object > last_object))return last_object;
@@ -106,16 +106,16 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page,* reciprocal_divide(offset, cache->reciprocal_buffer_size) */ static inline unsigned int obj_to_index (const struct kmem_cache *cache, -const struct page *page,void *obj) +const struct slab *slab,void *obj) { -u32 offset = (obj - page->s_mem); +u32 offset = (obj - slab->s_mem);return reciprocal_divide (offset, cache->reciprocal_buffer_size); } -static inline int objs_per_slab_page (const struct kmem_cache *cache, -const struct page *page) +static inline int objs_per_slab (const struct kmem_cache *cache, +const struct slab *slab) { -if (is_kfence_address (page_address (page))) +if (is_kfence_address (slab_address (slab)))return 1 ;return cache->num; }
include/linux/slub_def.h +9 -9
@@ -158,11 +158,11 @@ static inline void sysfs_slab_release(struct kmem_cache *s)void *fixup_red_left (struct kmem_cache *s,void *p); -static inline void *nearest_obj (struct kmem_cache *cache,struct page *page, +static inline void *nearest_obj (struct kmem_cache *cache,const struct slab *slab,void *x) { -void *object = x - (x -page_address (page)) % cache->size; -void *last_object =page_address (page) + - (page->objects -1 ) * cache->size; +void *object = x - (x -slab_address (slab)) % cache->size; +void *last_object =slab_address (slab) + + (slab->objects -1 ) * cache->size;void *result = (unlikely (object > last_object)) ? last_object : object; result =fixup_red_left (cache, result);
@@ -178,16 +178,16 @@ static inline unsigned int __obj_to_index(const struct kmem_cache *cache, }static inline unsigned int obj_to_index (const struct kmem_cache *cache, -const struct page *page,void *obj) +const struct slab *slab,void *obj) {if (is_kfence_address (obj))return 0 ; -return __obj_to_index (cache,page_address (page), obj); +return __obj_to_index (cache,slab_address (slab), obj); } -static inline int objs_per_slab_page (const struct kmem_cache *cache, -const struct page *page) +static inline int objs_per_slab (const struct kmem_cache *cache, +const struct slab *slab) { -return page->objects; +return slab->objects; }#endif /* _LINUX_SLUB_DEF_H */
mm/kasan/common.c +2 -2
@@ -298,7 +298,7 @@ static inline u8 assign_tag(struct kmem_cache *cache,/* For caches that either have a constructor or SLAB_TYPESAFE_BY_RCU: */ #ifdef CONFIG_SLAB /* For SLAB assign tags based on the object index in the freelist. */ -return (u8 )obj_to_index (cache,virt_to_head_page (object), (void *)object); +return (u8 )obj_to_index (cache,virt_to_slab (object), (void *)object);#else /* * For SLUB assign a random tag during slab creation, otherwise reuse
@@ -341,7 +341,7 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object,if (is_kfence_address (object))return false; -if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != +if (unlikely (nearest_obj (cache,virt_to_slab (object), object) != object)) {kasan_report_invalid_free (tagged_object, ip);return true;
mm/kasan/generic.c +1 -1
@@ -339,7 +339,7 @@ static void __kasan_record_aux_stack(void *addr, bool can_alloc)return ; cache = page->slab_cache; - object =nearest_obj (cache, page, addr); + object =nearest_obj (cache,page_slab (page), addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (!alloc_meta)return ;
mm/kasan/report.c +1 -1
@@ -249,7 +249,7 @@ static void print_address_description(void *addr, u8 tag)if (page &&PageSlab (page)) {struct kmem_cache *cache = page->slab_cache; -void *object =nearest_obj (cache, page, addr); +void *object =nearest_obj (cache,page_slab (page), addr);describe_object (cache, object, addr, tag); }
mm/kasan/report_tags.c +1 -1
@@ -23,7 +23,7 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) page =kasan_addr_to_page (addr);if (page &&PageSlab (page)) { cache = page->slab_cache; - object =nearest_obj (cache, page, (void *)addr); + object =nearest_obj (cache,page_slab (page), (void *)addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (alloc_meta) {
mm/kfence/kfence_test.c +2 -2
@@ -291,8 +291,8 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat* even for KFENCE objects; these are required so that * memcg accounting works correctly. */ -KUNIT_EXPECT_EQ (test,obj_to_index (s, page, alloc),0U ); -KUNIT_EXPECT_EQ (test,objs_per_slab_page (s, page),1 ); +KUNIT_EXPECT_EQ (test,obj_to_index (s,page_slab (page), alloc),0U ); +KUNIT_EXPECT_EQ (test,objs_per_slab (s,page_slab (page)),1 );if (policy ==ALLOCATE_ANY )return alloc;
mm/memcontrol.c +2 -2
@@ -2819,7 +2819,7 @@ static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg)int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page) { -unsigned int objects =objs_per_slab_page (s, page); +unsigned int objects =objs_per_slab (s,page_slab (page));unsigned long memcg_data;void *vec;
@@ -2881,7 +2881,7 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p)struct obj_cgroup *objcg;unsigned int off; - off =obj_to_index (page->slab_cache, page, p); + off =obj_to_index (page->slab_cache,page_slab (page), p); objcg =page_objcgs (page)[off];if (objcg)return obj_cgroup_memcg (objcg);
mm/slab.c +5 -5
@@ -1560,7 +1560,7 @@ static void check_poison_obj(struct kmem_cache *cachep, void *objp)struct slab *slab =virt_to_slab (objp);unsigned int objnr; - objnr =obj_to_index (cachep,slab_page (slab), objp); + objnr =obj_to_index (cachep, slab, objp);if (objnr) { objp =index_to_obj (cachep, slab, objnr -1 ); realobj = (char *)objp +obj_offset (cachep);
@@ -2530,7 +2530,7 @@ static void *slab_get_obj(struct kmem_cache *cachep, struct slab *slab)static void slab_put_obj (struct kmem_cache *cachep,struct slab *slab,void *objp) { -unsigned int objnr =obj_to_index (cachep,slab_page (slab), objp); +unsigned int objnr =obj_to_index (cachep, slab, objp);#if DEBUG unsigned int i;
@@ -2717,7 +2717,7 @@ static void *cache_free_debugcheck(struct kmem_cache *cachep, void *objp,if (cachep->flags &SLAB_STORE_USER ) *dbg_userword (cachep, objp) = (void *)caller; - objnr =obj_to_index (cachep,slab_page (slab), objp); + objnr =obj_to_index (cachep, slab, objp);BUG_ON (objnr >= cachep->num);BUG_ON (objp !=index_to_obj (cachep, slab, objnr));
@@ -3663,7 +3663,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) objp = object -obj_offset (cachep); kpp->kp_data_offset =obj_offset (cachep); slab =virt_to_slab (objp); - objnr =obj_to_index (cachep,slab_page (slab), objp); + objnr =obj_to_index (cachep, slab, objp); objp =index_to_obj (cachep, slab, objnr); kpp->kp_objp = objp;if (DEBUG && cachep->flags &SLAB_STORE_USER )
@@ -4181,7 +4181,7 @@ void __check_heap_object(const void *ptr, unsigned long n,/* Find and validate object. */ cachep = slab->slab_cache; - objnr =obj_to_index (cachep,slab_page (slab), (void *)ptr); + objnr =obj_to_index (cachep, slab, (void *)ptr);BUG_ON (objnr >= cachep->num);/* Find offset within object. */
mm/slab.h +2 -2
@@ -483,7 +483,7 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s,continue ; } - off =obj_to_index (s, page, p[i]); + off =obj_to_index (s,page_slab (page), p[i]);obj_cgroup_get (objcg);page_objcgs (page)[off] = objcg; mod_objcg_state(objcg, page_pgdat(page),
@@ -522,7 +522,7 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s_orig,else s = s_orig; - off =obj_to_index (s, page, p[i]); + off =obj_to_index (s,page_slab (page), p[i]); objcg = objcgs[off];if (!objcg)continue ;
mm/slub.c +1 -1
@@ -4342,7 +4342,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)#else objp = objp0;#endif - objnr =obj_to_index (s,slab_page (slab), objp); + objnr =obj_to_index (s, slab, objp); kpp->kp_data_offset = (unsigned long )((char *)objp0 - (char *)objp); objp = base + s->size * objnr; kpp->kp_objp = objp;
Signature · 1 line
2.33.1
Re: [PATCH v2 22/33] mm: Convert struct page to struct slab in functions used by other subsystems
Quoted 380 lines · Vlastimil Babka
KASAN, KFENCE and memcg interact with SLAB or SLUB internals through functions
nearest_obj(),obj_to_index()andobjs_per_slab()that use struct page as parameter. This patch converts it to struct slab including all callers, through a coccinelle semantic patch.// Options: --include-headers --no-includes --smpl-spacing
include/linux/slab_def.hinclude/linux/slub_def.hmm/slab.hmm/kasan/*.cmm/kfence/kfence_test.cmm/memcontrol.cmm/slab.cmm/slub.c// Note: needs coccinelle 1.1.1 to avoid breaking whitespace@@
@@(quoted diff) +1
-1-objs_per_slab_page( +objs_per_slab( ) { ... }@@
@@(quoted diff) +1
-1-objs_per_slab_page( +objs_per_slab( )@@
identifier fn =~ "obj_to_index|objs_per_slab";
@@fn(...,
(quoted diff) +1
-1- const struct page *page + const struct slab *slab ,...) {<...
(
-page_address(page)
+slab_address(slab)
|
- page
+ slab
)
...>
}@@
identifier fn =~ "nearest_obj";
@@fn(...,
(quoted diff) +1
-1- struct page *page + const struct slab *slab ,...) {<...
(
-page_address(page)
+slab_address(slab)
|
- page
+ slab
)
...>
}@@
identifier fn =~ "nearest_obj|obj_to_index|objs_per_slab"; expression E;
@@fn(...,
(
-slab_page(E)
+ E
|
-virt_to_page(E)
+virt_to_slab(E)
|
-virt_to_head_page(E)
+virt_to_slab(E)
|
- page
+page_slab(page)
)
,...)Signed-off-by Vlastimil Babka <[email protected]>Cc Julia Lawall <[email protected]>Cc Luis Chamberlain <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Alexander Potapenko <[email protected]>Cc Andrey Konovalov <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Marco Elver <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc <kasan-dev-/[email protected]>Cc <[email protected]>--- include/linux/slab_def.h | 16 ++++++++-------- include/linux/slub_def.h | 18 +++++++++--------- mm/kasan/common.c | 4 ++-- mm/kasan/generic.c | 2 +- mm/kasan/report.c | 2 +- mm/kasan/report_tags.c | 2 +- mm/kfence/kfence_test.c | 4 ++-- mm/memcontrol.c | 4 ++-- mm/slab.c | 10 +++++----- mm/slab.h | 4 ++-- mm/slub.c | 2 +- 11 files changed, 34 insertions(+), 34 deletions(-)include/linux/slab_def.h +8
-8@@ -87,11 +87,11 @@ struct kmem_cache {struct kmem_cache_node *node[MAX_NUMNODES ]; }; -static inline void *nearest_obj (struct kmem_cache *cache,struct page *page, +static inline void *nearest_obj (struct kmem_cache *cache,const struct slab *slab,void *x) { -void *object = x - (x - page->s_mem) % cache->size; -void *last_object = page->s_mem + (cache->num -1 ) * cache->size; +void *object = x - (x - slab->s_mem) % cache->size; +void *last_object = slab->s_mem + (cache->num -1 ) * cache->size;if (unlikely (object > last_object))return last_object;@@ -106,16 +106,16 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page,* reciprocal_divide(offset, cache->reciprocal_buffer_size) */ static inline unsigned int obj_to_index (const struct kmem_cache *cache, -const struct page *page,void *obj) +const struct slab *slab,void *obj) { -u32 offset = (obj - page->s_mem); +u32 offset = (obj - slab->s_mem);return reciprocal_divide (offset, cache->reciprocal_buffer_size); } -static inline int objs_per_slab_page (const struct kmem_cache *cache, -const struct page *page) +static inline int objs_per_slab (const struct kmem_cache *cache, +const struct slab *slab) { -if (is_kfence_address (page_address (page))) +if (is_kfence_address (slab_address (slab)))return 1 ;return cache->num; }include/linux/slub_def.h +9
-9@@ -158,11 +158,11 @@ static inline void sysfs_slab_release(struct kmem_cache *s)void *fixup_red_left (struct kmem_cache *s,void *p); -static inline void *nearest_obj (struct kmem_cache *cache,struct page *page, +static inline void *nearest_obj (struct kmem_cache *cache,const struct slab *slab,void *x) { -void *object = x - (x -page_address (page)) % cache->size; -void *last_object =page_address (page) + - (page->objects -1 ) * cache->size; +void *object = x - (x -slab_address (slab)) % cache->size; +void *last_object =slab_address (slab) + + (slab->objects -1 ) * cache->size;void *result = (unlikely (object > last_object)) ? last_object : object; result =fixup_red_left (cache, result);@@ -178,16 +178,16 @@ static inline unsigned int __obj_to_index(const struct kmem_cache *cache, }static inline unsigned int obj_to_index (const struct kmem_cache *cache, -const struct page *page,void *obj) +const struct slab *slab,void *obj) {if (is_kfence_address (obj))return 0 ; -return __obj_to_index (cache,page_address (page), obj); +return __obj_to_index (cache,slab_address (slab), obj); } -static inline int objs_per_slab_page (const struct kmem_cache *cache, -const struct page *page) +static inline int objs_per_slab (const struct kmem_cache *cache, +const struct slab *slab) { -return page->objects; +return slab->objects; }#endif /* _LINUX_SLUB_DEF_H */ mm/kasan/common.c +2
-2@@ -298,7 +298,7 @@ static inline u8 assign_tag(struct kmem_cache *cache,/* For caches that either have a constructor or SLAB_TYPESAFE_BY_RCU: */ #ifdef CONFIG_SLAB /* For SLAB assign tags based on the object index in the freelist. */ -return (u8 )obj_to_index (cache,virt_to_head_page (object), (void *)object); +return (u8 )obj_to_index (cache,virt_to_slab (object), (void *)object);#else /* * For SLUB assign a random tag during slab creation, otherwise reuse @@ -341,7 +341,7 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object,if (is_kfence_address (object))return false; -if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != +if (unlikely (nearest_obj (cache,virt_to_slab (object), object) != object)) {kasan_report_invalid_free (tagged_object, ip);return true;mm/kasan/generic.c +1
-1@@ -339,7 +339,7 @@ static void __kasan_record_aux_stack(void *addr, bool can_alloc)return ; cache = page->slab_cache; - object =nearest_obj (cache, page, addr); + object =nearest_obj (cache,page_slab (page), addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (!alloc_meta)return ;mm/kasan/report.c +1
-1@@ -249,7 +249,7 @@ static void print_address_description(void *addr, u8 tag)if (page &&PageSlab (page)) {struct kmem_cache *cache = page->slab_cache; -void *object =nearest_obj (cache, page, addr); +void *object =nearest_obj (cache,page_slab (page), addr);describe_object (cache, object, addr, tag); }mm/kasan/report_tags.c +1
-1@@ -23,7 +23,7 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) page =kasan_addr_to_page (addr);if (page &&PageSlab (page)) { cache = page->slab_cache; - object =nearest_obj (cache, page, (void *)addr); + object =nearest_obj (cache,page_slab (page), (void *)addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (alloc_meta) {mm/kfence/kfence_test.c +2
-2@@ -291,8 +291,8 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat* even for KFENCE objects; these are required so that * memcg accounting works correctly. */ -KUNIT_EXPECT_EQ (test,obj_to_index (s, page, alloc),0U ); -KUNIT_EXPECT_EQ (test,objs_per_slab_page (s, page),1 ); +KUNIT_EXPECT_EQ (test,obj_to_index (s,page_slab (page), alloc),0U ); +KUNIT_EXPECT_EQ (test,objs_per_slab (s,page_slab (page)),1 );if (policy ==ALLOCATE_ANY )return alloc;mm/memcontrol.c +2
-2@@ -2819,7 +2819,7 @@ static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg)int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page) { -unsigned int objects =objs_per_slab_page (s, page); +unsigned int objects =objs_per_slab (s,page_slab (page));unsigned long memcg_data;void *vec;@@ -2881,7 +2881,7 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p)struct obj_cgroup *objcg;unsigned int off; - off =obj_to_index (page->slab_cache, page, p); + off =obj_to_index (page->slab_cache,page_slab (page), p); objcg =page_objcgs (page)[off];if (objcg)return obj_cgroup_memcg (objcg);mm/slab.c +5
-5@@ -1560,7 +1560,7 @@ static void check_poison_obj(struct kmem_cache *cachep, void *objp)struct slab *slab =virt_to_slab (objp);unsigned int objnr; - objnr =obj_to_index (cachep,slab_page (slab), objp); + objnr =obj_to_index (cachep, slab, objp);if (objnr) { objp =index_to_obj (cachep, slab, objnr -1 ); realobj = (char *)objp +obj_offset (cachep);@@ -2530,7 +2530,7 @@ static void *slab_get_obj(struct kmem_cache *cachep, struct slab *slab)static void slab_put_obj (struct kmem_cache *cachep,struct slab *slab,void *objp) { -unsigned int objnr =obj_to_index (cachep,slab_page (slab), objp); +unsigned int objnr =obj_to_index (cachep, slab, objp);#if DEBUG unsigned int i;@@ -2717,7 +2717,7 @@ static void *cache_free_debugcheck(struct kmem_cache *cachep, void *objp,if (cachep->flags &SLAB_STORE_USER ) *dbg_userword (cachep, objp) = (void *)caller; - objnr =obj_to_index (cachep,slab_page (slab), objp); + objnr =obj_to_index (cachep, slab, objp);BUG_ON (objnr >= cachep->num);BUG_ON (objp !=index_to_obj (cachep, slab, objnr));@@ -3663,7 +3663,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) objp = object -obj_offset (cachep); kpp->kp_data_offset =obj_offset (cachep); slab =virt_to_slab (objp); - objnr =obj_to_index (cachep,slab_page (slab), objp); + objnr =obj_to_index (cachep, slab, objp); objp =index_to_obj (cachep, slab, objnr); kpp->kp_objp = objp;if (DEBUG && cachep->flags &SLAB_STORE_USER )@@ -4181,7 +4181,7 @@ void __check_heap_object(const void *ptr, unsigned long n,/* Find and validate object. */ cachep = slab->slab_cache; - objnr =obj_to_index (cachep,slab_page (slab), (void *)ptr); + objnr =obj_to_index (cachep, slab, (void *)ptr);BUG_ON (objnr >= cachep->num);/* Find offset within object. */ mm/slab.h +2
-2@@ -483,7 +483,7 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s,continue ; } - off =obj_to_index (s, page, p[i]); + off =obj_to_index (s,page_slab (page), p[i]);obj_cgroup_get (objcg);page_objcgs (page)[off] = objcg; mod_objcg_state(objcg, page_pgdat(page),@@ -522,7 +522,7 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s_orig,else s = s_orig; - off =obj_to_index (s, page, p[i]); + off =obj_to_index (s,page_slab (page), p[i]); objcg = objcgs[off];if (!objcg)continue ;mm/slub.c +1
-2@@ -4342,7 +4342,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)#else objp = objp0;#endif - objnr =obj_to_index (s,slab_page (slab), objp); + objnr =obj_to_index (s, slab, objp); kpp->kp_data_offset = (unsigned long )((char *)objp0 - (char *)objp); objp = base + s->size * objnr; kpp->kp_objp = objp; --
Quoted 2 lines
2.33.1
Thanks!
Re: [PATCH v2 22/33] mm: Convert struct page to struct slab in functions used by other subsystems
In reply to Vlastimil BabkaQuoted 86 lines · Vlastimil Babka
KASAN, KFENCE and memcg interact with SLAB or SLUB internals through functions
nearest_obj(),obj_to_index()andobjs_per_slab()that use struct page as parameter. This patch converts it to struct slab including all callers, through a coccinelle semantic patch.// Options: --include-headers --no-includes --smpl-spacing
include/linux/slab_def.hinclude/linux/slub_def.hmm/slab.hmm/kasan/*.cmm/kfence/kfence_test.cmm/memcontrol.cmm/slab.cmm/slub.c// Note: needs coccinelle 1.1.1 to avoid breaking whitespace@@
@@(quoted diff) +1
-1-objs_per_slab_page( +objs_per_slab( ) { ... }@@
@@(quoted diff) +1
-1-objs_per_slab_page( +objs_per_slab( )@@
identifier fn =~ "obj_to_index|objs_per_slab";
@@fn(...,
(quoted diff) +1
-1- const struct page *page + const struct slab *slab ,...) {<...
(
-page_address(page)
+slab_address(slab)
|
- page
+ slab
)
...>
}@@
identifier fn =~ "nearest_obj";
@@fn(...,
(quoted diff) +1
-1- struct page *page + const struct slab *slab ,...) {<...
(
-page_address(page)
+slab_address(slab)
|
- page
+ slab
)
...>
}@@
identifier fn =~ "nearest_obj|obj_to_index|objs_per_slab"; expression E;
@@fn(...,
(
-slab_page(E)
+ E
|
-virt_to_page(E)
+virt_to_slab(E)
|
-virt_to_head_page(E)
+virt_to_slab(E)
|
- page
+page_slab(page)
)
,...)
Quoted 13 lines
Signed-off-by Vlastimil Babka <[email protected]>Cc Julia Lawall <[email protected]>Cc Luis Chamberlain <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Alexander Potapenko <[email protected]>Cc Andrey Konovalov <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Marco Elver <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc <kasan-dev-/[email protected]>Cc <[email protected]>
LGTM.
PATCH v2 23/33 mm/memcg: Convert slab objcgs from struct page to struct slab
In reply to Vlastimil Babkapage->memcg_data is used with MEMCG_DATA_OBJCGS flag only for slab pages so convert all the related infrastructure to struct slab.
To avoid include cycles, move the inline definitions of slab_objcgs() and slab_objcgs_check() from memcontrol.h to mm/slab.h.
This is not just mechanistic changing of types and names. Now in mem_cgroup_from_obj() we use PageSlab flag to decide if we interpret the page as slab, instead of relying on MEMCG_DATA_OBJCGS bit checked in page_objcgs_check() (now slab_objcgs_check()). Similarly in memcg_slab_free_hook() where we can encounter kmalloc_large() pages (here the PageSlab flag check is implied by virt_to_slab()).
--- include/linux/memcontrol.h | 48 ------------------ mm/memcontrol.c | 43 +++++++++------- mm/slab.h | 101 ++++++++++++++++++++++++++++--------- 3 files changed, 103 insertions(+), 89 deletions(-)
include/linux/memcontrol.h +0 -48
@@ -536,45 +536,6 @@ static inline bool folio_memcg_kmem(struct folio *folio) return folio->memcg_data & MEMCG_DATA_KMEM; } -/* -* page_objcgs - get the object cgroups vector associated with a page -* @page: a pointer to the page struct -* -* Returns a pointer to the object cgroups vector associated with the page, -* or NULL. This function assumes that the page is known to have an -* associated object cgroups vector. It's not safe to call this function -* against pages, which might have an associated memory cgroup: e.g. -* kernel stack pages. -*/ -static inline struct obj_cgroup **page_objcgs (struct page *page) -{ -unsigned long memcg_data =READ_ONCE (page->memcg_data); - -VM_BUG_ON_PAGE (memcg_data && !(memcg_data &MEMCG_DATA_OBJCGS ), page); -VM_BUG_ON_PAGE (memcg_data &MEMCG_DATA_KMEM , page); - -return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); -} - -/* -* page_objcgs_check - get the object cgroups vector associated with a page -* @page: a pointer to the page struct -* -* Returns a pointer to the object cgroups vector associated with the page, -* or NULL. This function is safe to use if the page can be directly associated -* with a memory cgroup. -*/ -static inline struct obj_cgroup **page_objcgs_check (struct page *page) -{ -unsigned long memcg_data =READ_ONCE (page->memcg_data); - -if (!memcg_data || !(memcg_data &MEMCG_DATA_OBJCGS )) -return NULL ; - -VM_BUG_ON_PAGE (memcg_data &MEMCG_DATA_KMEM , page); - -return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); -}#else static inline bool folio_memcg_kmem (struct folio *folio)
@@ -582,15 +543,6 @@ static inline bool folio_memcg_kmem(struct folio *folio) return false; } -static inline struct obj_cgroup **page_objcgs (struct page *page) -{ -return NULL ; -} - -static inline struct obj_cgroup **page_objcgs_check (struct page *page) -{ -return NULL ; -}#endif static inline bool PageMemcgKmem (struct page *page)
mm/memcontrol.c +25 -18
87 lines not on this page · raw message
mm/slab.h +78 -23
183 lines not on this page · raw message
Signature · 1 line
2.33.1
Johannes Weiner
maintainerRe: [PATCH v2 23/33] mm/memcg: Convert slab objcgs from struct page to struct slab
Quoted 6 lines · Vlastimil Babka
page->memcg_datais used withMEMCG_DATA_OBJCGSflag only for slab pages so convert all the related infrastructure to struct slab.To avoid include cycles, move the inline definitions of
slab_objcgs()andslab_objcgs_check()frommemcontrol.htomm/slab.h.
Quoted 6 lines
This is not just mechanistic changing of types and names. Now in
mem_cgroup_from_obj()we use PageSlab flag to decide if we interpret the page as slab, instead of relying onMEMCG_DATA_OBJCGSbit checked inpage_objcgs_check()(nowslab_objcgs_check()). Similarly inmemcg_slab_free_hook()where we can encounterkmalloc_large()pages (here the PageSlab flag check is implied byvirt_to_slab()).
Yup, this is great.
Quoted 19 lines
mm/memcontrol.c +2
-3@@ -2865,24 +2865,31 @@ int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s,*/ struct mem_cgroup *mem_cgroup_from_obj (void *p) { -struct page *page; +struct folio *folio;if (mem_cgroup_disabled ())return NULL ; - page =virt_to_head_page (p); + folio =virt_to_folio (p);/* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in * the page->obj_cgroups. */ -if (page_objcgs_check (page)) {
mm/memcontrol.c int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, · line 2887
Quoted 10 lines
mm/memcontrol.c +5
-2+if (folio_test_slab (folio)) { +struct obj_cgroup **objcgs;struct obj_cgroup *objcg; +struct slab *slab;unsigned int off; - off =obj_to_index (page->slab_cache,page_slab (page), p); - objcg =page_objcgs (page)[off]; + slab =folio_slab (folio); + objcgs =slab_objcgs_check (slab);
AFAICS the change to the _check() variant was accidental.
folio_test_slab() makes sure it's a slab page, so the legit options for memcg_data are NULL or |MEMCG_DATA_OBJCGS; using slab_objcgs() here would include the proper asserts, like page_objcgs() used to.
Re: [PATCH v2 23/33] mm/memcg: Convert slab objcgs from struct page to struct slab
Quoted 48 lines · Johannes Weiner
Quoted 6 lines · Vlastimil Babka
page->memcg_datais used withMEMCG_DATA_OBJCGSflag only for slab pages so convert all the related infrastructure to struct slab.To avoid include cycles, move the inline definitions of
slab_objcgs()andslab_objcgs_check()frommemcontrol.htomm/slab.h.Quoted 6 lines
This is not just mechanistic changing of types and names. Now in
mem_cgroup_from_obj()we use PageSlab flag to decide if we interpret the page as slab, instead of relying onMEMCG_DATA_OBJCGSbit checked inpage_objcgs_check()(nowslab_objcgs_check()). Similarly inmemcg_slab_free_hook()where we can encounterkmalloc_large()pages (here the PageSlab flag check is implied byvirt_to_slab()).Yup, this is great.
Quoted 19 lines
mm/memcontrol.c +2
-3@@ -2865,24 +2865,31 @@ int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s,*/ struct mem_cgroup *mem_cgroup_from_obj (void *p) { -struct page *page; +struct folio *folio;if (mem_cgroup_disabled ())return NULL ; - page =virt_to_head_page (p); + folio =virt_to_folio (p);/* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in * the page->obj_cgroups. */ -if (page_objcgs_check (page)) {mm/memcontrol.c int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, · line 2887
Quoted 10 lines
mm/memcontrol.c +5
-2+if (folio_test_slab (folio)) { +struct obj_cgroup **objcgs;struct obj_cgroup *objcg; +struct slab *slab;unsigned int off; - off =obj_to_index (page->slab_cache,page_slab (page), p); - objcg =page_objcgs (page)[off]; + slab =folio_slab (folio); + objcgs =slab_objcgs_check (slab);AFAICS the change to the
_check()variant was accidental.
Quoted 3 lines
folio_test_slab()makes sure it's a slab page, so the legit options formemcg_dataare NULL or |MEMCG_DATA_OBJCGS; usingslab_objcgs()here would include the proper asserts, likepage_objcgs()used to.
Well spotted. On closer look, it's actually the same in memcg_slab_free_hook() where I also added a folio_test_slab() as part of virt_to_slab(). In fact it means page_objcgs_check() can be just deleted instead of replaced by slab_objcgs_check(). Thanks!
PATCH v2 24/33 mm/slob: Convert SLOB to use struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Use struct slab throughout the slob allocator.
[ [email protected]: don't introduce wrappers for PageSlobFree in mm/slab.h just
for the single callers being wrappers in mm/slob.c ]
--- mm/slob.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
mm/slob.c +17 -17
@@ -105,21 +105,21 @@ static LIST_HEAD(free_slob_large);/* * slob_page_free: true for pages on free_slob_pages list. */ -static inline int slob_page_free (struct page *sp) +static inline int slob_page_free (struct slab *slab) { -return PageSlobFree (sp); +return PageSlobFree (slab_page (slab)); } -static void set_slob_page_free (struct page *sp,struct list_head *list) +static void set_slob_page_free (struct slab *slab,struct list_head *list) { -list_add (&sp->slab_list, list); -__SetPageSlobFree (sp); +list_add (&slab->slab_list, list); +__SetPageSlobFree (slab_page (slab)); } -static inline void clear_slob_page_free (struct page *sp) +static inline void clear_slob_page_free (struct slab *slab) { -list_del (&sp->slab_list); -__ClearPageSlobFree (sp); +list_del (&slab->slab_list); +__ClearPageSlobFree (slab_page (slab)); }#define SLOB_UNIT sizeof(slob_t)
@@ -234,7 +234,7 @@ static void slob_free_pages(void *b, int order)* freelist, in this case @page_removed_from_list will be set to * true (set to false otherwise). */ -static void *slob_page_alloc (struct page *sp,size_t size,int align, +static void *slob_page_alloc (struct slab *sp,size_t size,int align,int align_offset,bool *page_removed_from_list) {slob_t *prev, *cur, *aligned =NULL ;
@@ -301,7 +301,7 @@ static void *slob_page_alloc(struct page *sp, size_t size, int align,static void *slob_alloc (size_t size,gfp_t gfp,int align,int node,int align_offset) { -struct page *sp; +struct slab *sp;struct list_head *slob_list;slob_t *b =NULL ;unsigned long flags;
@@ -323,7 +323,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,* If there's a node specification, search for a partial * page with a matching node id in the freelist. */ -if (node !=NUMA_NO_NODE &&page_to_nid (sp) != node) +if (node !=NUMA_NO_NODE &&slab_nid (sp) != node)continue ;#endif /* Enough room on this page? */
@@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; - sp =virt_to_page (b); -__SetPageSlab (sp); + sp =virt_to_slab (b); +__SetPageSlab (slab_page (sp));spin_lock_irqsave (&slob_lock, flags); sp->units =SLOB_UNITS (PAGE_SIZE );
@@ -381,7 +381,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,*/ static void slob_free (void *block,int size) { -struct page *sp; +struct slab *sp;slob_t *prev, *next, *b = (slob_t *)block;slobidx_t units;unsigned long flags;
@@ -391,7 +391,7 @@ static void slob_free(void *block, int size)return ;BUG_ON (!size); - sp =virt_to_page (block); + sp =virt_to_slab (block); units =SLOB_UNITS (size);spin_lock_irqsave (&slob_lock, flags);
@@ -401,8 +401,8 @@ static void slob_free(void *block, int size)if (slob_page_free (sp))clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags); -__ClearPageSlab (sp); -page_mapcount_reset (sp); +__ClearPageSlab (slab_page (sp)); +page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }
Signature · 1 line
2.33.1
Re: [PATCH v2 24/33] mm/slob: Convert SLOB to use struct slab
Quoted 17 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
Use struct slab throughout the slob allocator.
[ [email protected]: don't introduce wrappers for PageSlobFree in
mm/slab.hjust
for the single callers being wrappers inmm/slob.c]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>--- mm/slob.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)mm/slob.c +0
-0
...
mm/slob.c static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, · line 362
Quoted 9 lines
mm/slob.c +2
-2/* Enough room on this page? */ @@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; - sp =virt_to_page (b); -__SetPageSlab (sp); + sp =virt_to_slab (b); +__SetPageSlab (slab_page (sp));
Hello Vlastimil.
I've tested this patch on my machine and it causes NULL pointer dereference. that's because virt_to_slab returns NULL if folio_test_slab is false. and __SetPageSlab is called with sp = NULL.
diff below fixed bug.
mm/slob.c +0 -0
@@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int a
lign, int node,
b = slob_new_pages(gfp & ~__GFP_ZERO, 0, node);
if (!b)
return NULL;
+ __SetPageSlab(virt_to_page(b));
sp = virt_to_slab(b);- __SetPageSlab(slab_page(sp));
spin_lock_irqsave(&slob_lock, flags);
sp->units = SLOB_UNITS(PAGE_SIZE); Thanks,
Hyeonggon.
Quoted 36 lines
mm/slob.c +4
-4spin_lock_irqsave (&slob_lock, flags); sp->units =SLOB_UNITS (PAGE_SIZE );@@ -381,7 +381,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,*/ static void slob_free (void *block,int size) { -struct page *sp; +struct slab *sp;slob_t *prev, *next, *b = (slob_t *)block;slobidx_t units;unsigned long flags;@@ -391,7 +391,7 @@ static void slob_free(void *block, int size)return ;BUG_ON (!size); - sp =virt_to_page (block); + sp =virt_to_slab (block); units =SLOB_UNITS (size);spin_lock_irqsave (&slob_lock, flags);@@ -401,8 +401,8 @@ static void slob_free(void *block, int size)if (slob_page_free (sp))clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags); -__ClearPageSlab (sp); -page_mapcount_reset (sp); +__ClearPageSlab (slab_page (sp)); +page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Signature · 1 line
2.33.1
Re: [PATCH v2 24/33] mm/slob: Convert SLOB to use struct slab
Quoted 33 lines · Hyeonggon Yoo
Quoted 17 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
Use struct slab throughout the slob allocator.
[ [email protected]: don't introduce wrappers for PageSlobFree in
mm/slab.hjust
for the single callers being wrappers inmm/slob.c]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>--- mm/slob.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)mm/slob.c +0
-0...
mm/slob.c static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, · line 362
Quoted 9 lines
mm/slob.c +2
-2/* Enough room on this page? */ @@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; - sp =virt_to_page (b); -__SetPageSlab (sp); + sp =virt_to_slab (b); +__SetPageSlab (slab_page (sp));Hello Vlastimil.
Quoted 3 lines
I've tested this patch on my machine and it causes NULL pointer dereference. that's because
virt_to_slabreturns NULL iffolio_test_slabis false. and__SetPageSlabis called with sp = NULL.
Oops, thanks for catching this.
Quoted 1 line
diff below fixed bug.
Changed it to use folio and will push an updated branch slab-struct_slab-v3r2
Thanks!
Quoted 55 lines
mm/slob.c +1
-1@@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int a lign, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; +__SetPageSlab (virt_to_page (b)); sp =virt_to_slab (b); -__SetPageSlab (slab_page (sp));spin_lock_irqsave (&slob_lock, flags);sp->units = SLOB_UNITS(PAGE_SIZE);Thanks,
Hyeonggon.Quoted 36 lines
mm/slob.c +4
-4spin_lock_irqsave (&slob_lock, flags); sp->units =SLOB_UNITS (PAGE_SIZE );@@ -381,7 +381,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,*/ static void slob_free (void *block,int size) { -struct page *sp; +struct slab *sp;slob_t *prev, *next, *b = (slob_t *)block;slobidx_t units;unsigned long flags;@@ -391,7 +391,7 @@ static void slob_free(void *block, int size)return ;BUG_ON (!size); - sp =virt_to_page (block); + sp =virt_to_slab (block); units =SLOB_UNITS (size);spin_lock_irqsave (&slob_lock, flags);@@ -401,8 +401,8 @@ static void slob_free(void *block, int size)if (slob_page_free (sp))clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags); -__ClearPageSlab (sp); -page_mapcount_reset (sp); +__ClearPageSlab (slab_page (sp)); +page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Signature · 1 line
2.33.1
Re: [PATCH v2 24/33] mm/slob: Convert SLOB to use struct slab
Quoted 38 lines · Vlastimil Babka
Quoted 33 lines · Hyeonggon Yoo
Quoted 17 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
Use struct slab throughout the slob allocator.
[ [email protected]: don't introduce wrappers for PageSlobFree in
mm/slab.hjust
for the single callers being wrappers inmm/slob.c]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>--- mm/slob.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)mm/slob.c +0
-0...
mm/slob.c static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, · line 362
Quoted 9 lines
mm/slob.c +2
-2/* Enough room on this page? */ @@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; - sp =virt_to_page (b); -__SetPageSlab (sp); + sp =virt_to_slab (b); +__SetPageSlab (slab_page (sp));Hello Vlastimil.
Quoted 3 lines
I've tested this patch on my machine and it causes NULL pointer dereference. that's because
virt_to_slabreturns NULL iffolio_test_slabis false. and__SetPageSlabis called with sp = NULL.
Quoted 8 lines
Oops, thanks for catching this.
Quoted 1 line
diff below fixed bug.
Changed it to use folio and will push an updated branch slab-struct_slab-v3r2
Thanks!
Tested again on slab-struct_slab-v3r2 and everyting works fine! And the code overall looks good to me.
And below is not about this patch,
but what about something like this?
Because it's not necessary that page->_mapcount and slab->units have same offset, so we can remove (somewhat confusing) page_mapcount_reset function call.
I want to remove calling page_mapcount_reset also in SLAB, but I have no idea. (maybe we can do that if we remove SLAB colouring but that's going too far)
mm/slab.h +2 -2
@@ -50,8 +50,8 @@ struct slab {struct list_head slab_list;void * __unused_1;void *freelist;/* first free block */ -void * __unused_2; -int units; +long units; +unsigned int __unused_2;#else #error "Unexpected slab allocator configured"
mm/slob.c +0 -1
@@ -404,7 +404,6 @@ static void slob_free(void *block, int size)clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags);__ClearPageSlab (slab_page (sp)); -page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }
Quoted 57 lines
Quoted 56 lines
mm/slob.c +1
-1@@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int a lign, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; +__SetPageSlab (virt_to_page (b)); sp =virt_to_slab (b); -__SetPageSlab (slab_page (sp));spin_lock_irqsave (&slob_lock, flags);sp->units = SLOB_UNITS(PAGE_SIZE);Thanks,
Hyeonggon.Quoted 36 lines
mm/slob.c +4
-4spin_lock_irqsave (&slob_lock, flags); sp->units =SLOB_UNITS (PAGE_SIZE );@@ -381,7 +381,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,*/ static void slob_free (void *block,int size) { -struct page *sp; +struct slab *sp;slob_t *prev, *next, *b = (slob_t *)block;slobidx_t units;unsigned long flags;@@ -391,7 +391,7 @@ static void slob_free(void *block, int size)return ;BUG_ON (!size); - sp =virt_to_page (block); + sp =virt_to_slab (block); units =SLOB_UNITS (size);spin_lock_irqsave (&slob_lock, flags);@@ -401,8 +401,8 @@ static void slob_free(void *block, int size)if (slob_page_free (sp))clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags); -__ClearPageSlab (sp); -page_mapcount_reset (sp); +__ClearPageSlab (slab_page (sp)); +page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Signature · 1 line
2.33.1
Re: [PATCH v2 24/33] mm/slob: Convert SLOB to use struct slab
Quoted 46 lines · Hyeonggon Yoo
Quoted 45 lines · Vlastimil Babka
Quoted 33 lines · Hyeonggon Yoo
Quoted 17 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
Use struct slab throughout the slob allocator.
[ [email protected]: don't introduce wrappers for PageSlobFree in
mm/slab.hjust
for the single callers being wrappers inmm/slob.c]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>--- mm/slob.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)mm/slob.c +0
-0...
mm/slob.c static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, · line 362
Quoted 9 lines
mm/slob.c +2
-2/* Enough room on this page? */ @@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; - sp =virt_to_page (b); -__SetPageSlab (sp); + sp =virt_to_slab (b); +__SetPageSlab (slab_page (sp));Hello Vlastimil.
Quoted 3 lines
I've tested this patch on my machine and it causes NULL pointer dereference. that's because
virt_to_slabreturns NULL iffolio_test_slabis false. and__SetPageSlabis called with sp = NULL.Oops, thanks for catching this.
Quoted 1 line
diff below fixed bug.
Changed it to use folio and will push an updated branch slab-struct_slab-v3r2
Quoted 7 lines
Quoted 1 line
Thanks!
Tested again on slab-struct_slab-v3r2 and everyting works fine! And the code overall looks good to me.
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>
Thanks!
Quoted 6 lines
And below is not about this patch,
but what about something like this?Because it's not necessary that
page->_mapcountandslab->unitshave same offset, so we can remove (somewhat confusing)page_mapcount_resetfunction call.
Yeah this could be done as new patch on top of 21/33. I wouldn't squash it there so that it remains just a split without functional change.
Quoted 89 lines
I want to remove calling
page_mapcount_resetalso in SLAB, but I have no idea. (maybe we can do that if we remove SLAB colouring but that's going too far)mm/slab.h +2
-2@@ -50,8 +50,8 @@ struct slab {struct list_head slab_list;void * __unused_1;void *freelist;/* first free block */ -void * __unused_2; -int units; +long units; +unsigned int __unused_2;#else #error "Unexpected slab allocator configured"mm/slob.c +0
-1@@ -404,7 +404,6 @@ static void slob_free(void *block, int size)clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags);__ClearPageSlab (slab_page (sp)); -page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Quoted 58 lines
Quoted 57 lines
mm/slob.c +1
-1@@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int a lign, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; +__SetPageSlab (virt_to_page (b)); sp =virt_to_slab (b); -__SetPageSlab (slab_page (sp));spin_lock_irqsave (&slob_lock, flags);sp->units = SLOB_UNITS(PAGE_SIZE);Thanks,
Hyeonggon.Quoted 36 lines
mm/slob.c +4
-4spin_lock_irqsave (&slob_lock, flags); sp->units =SLOB_UNITS (PAGE_SIZE );@@ -381,7 +381,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,*/ static void slob_free (void *block,int size) { -struct page *sp; +struct slab *sp;slob_t *prev, *next, *b = (slob_t *)block;slobidx_t units;unsigned long flags;@@ -391,7 +391,7 @@ static void slob_free(void *block, int size)return ;BUG_ON (!size); - sp =virt_to_page (block); + sp =virt_to_slab (block); units =SLOB_UNITS (size);spin_lock_irqsave (&slob_lock, flags);@@ -401,8 +401,8 @@ static void slob_free(void *block, int size)if (slob_page_free (sp))clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags); -__ClearPageSlab (sp); -page_mapcount_reset (sp); +__ClearPageSlab (slab_page (sp)); +page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Signature · 1 line
2.33.1
Re: [PATCH v2 24/33] mm/slob: Convert SLOB to use struct slab
Quoted 49 lines · Vlastimil Babka
Quoted 48 lines · Hyeonggon Yoo
Quoted 46 lines · Vlastimil Babka
Quoted 33 lines · Hyeonggon Yoo
Quoted 17 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
Use struct slab throughout the slob allocator.
[ [email protected]: don't introduce wrappers for PageSlobFree in
mm/slab.hjust
for the single callers being wrappers inmm/slob.c]Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>--- mm/slob.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)mm/slob.c +0
-0...
mm/slob.c static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, · line 362
Quoted 9 lines
mm/slob.c +2
-2/* Enough room on this page? */ @@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; - sp =virt_to_page (b); -__SetPageSlab (sp); + sp =virt_to_slab (b); +__SetPageSlab (slab_page (sp));Hello Vlastimil.
Quoted 3 lines
I've tested this patch on my machine and it causes NULL pointer dereference. that's because
virt_to_slabreturns NULL iffolio_test_slabis false. and__SetPageSlabis called with sp = NULL.Oops, thanks for catching this.
Quoted 1 line
diff below fixed bug.
Changed it to use folio and will push an updated branch slab-struct_slab-v3r2
Thanks!
Quoted 8 lines
Quoted 5 lines
Tested again on slab-struct_slab-v3r2 and everyting works fine! And the code overall looks good to me.
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Thanks!
You're welcome! :)
Quoted 10 lines
Quoted 6 lines
And below is not about this patch,
but what about something like this?Because it's not necessary that
page->_mapcountandslab->unitshave same offset, so we can remove (somewhat confusing)page_mapcount_resetfunction call.Yeah this could be done as new patch on top of 21/33. I wouldn't squash it there so that it remains just a split without functional change.
Yeah, It is better to be done as separate patch.
I'll send it as a patch.
Thanks,
Hyeonggon
Quoted 90 lines
Quoted 89 lines
I want to remove calling
page_mapcount_resetalso in SLAB, but I have no idea. (maybe we can do that if we remove SLAB colouring but that's going too far)mm/slab.h +2
-2@@ -50,8 +50,8 @@ struct slab {struct list_head slab_list;void * __unused_1;void *freelist;/* first free block */ -void * __unused_2; -int units; +long units; +unsigned int __unused_2;#else #error "Unexpected slab allocator configured"mm/slob.c +0
-1@@ -404,7 +404,6 @@ static void slob_free(void *block, int size)clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags);__ClearPageSlab (slab_page (sp)); -page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Quoted 58 lines
Quoted 57 lines
mm/slob.c +1
-1@@ -358,8 +358,8 @@ static void *slob_alloc(size_t size, gfp_t gfp, int a lign, int node, b =slob_new_pages (gfp & ~__GFP_ZERO,0 , node);if (!b)return NULL ; +__SetPageSlab (virt_to_page (b)); sp =virt_to_slab (b); -__SetPageSlab (slab_page (sp));spin_lock_irqsave (&slob_lock, flags);sp->units = SLOB_UNITS(PAGE_SIZE);Thanks,
Hyeonggon.Quoted 36 lines
mm/slob.c +4
-4spin_lock_irqsave (&slob_lock, flags); sp->units =SLOB_UNITS (PAGE_SIZE );@@ -381,7 +381,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,*/ static void slob_free (void *block,int size) { -struct page *sp; +struct slab *sp;slob_t *prev, *next, *b = (slob_t *)block;slobidx_t units;unsigned long flags;@@ -391,7 +391,7 @@ static void slob_free(void *block, int size)return ;BUG_ON (!size); - sp =virt_to_page (block); + sp =virt_to_slab (block); units =SLOB_UNITS (size);spin_lock_irqsave (&slob_lock, flags);@@ -401,8 +401,8 @@ static void slob_free(void *block, int size)if (slob_page_free (sp))clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags); -__ClearPageSlab (sp); -page_mapcount_reset (sp); +__ClearPageSlab (slab_page (sp)); +page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }Signature · 1 line
2.33.1
PATCH v2 25/33 mm/kasan: Convert to struct folio and struct slab
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
KASAN accesses some slab related struct page fields so we need to convert it to struct slab. Some places are a bit simplified thanks to kasan_addr_to_slab() encapsulating the PageSlab flag check through virt_to_slab().
When resolving object address to either a real slab or a large kmalloc, use struct folio as the intermediate type for testing the slab flag to avoid unnecessary implicit compound_head().
[ [email protected]: use struct folio, adjust to differences in previous patches ]
--- include/linux/kasan.h | 9 +++++---- mm/kasan/common.c | 23 +++++++++++++---------- mm/kasan/generic.c | 8 ++++---- mm/kasan/kasan.h | 1 + mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 13 +++++++++++-- mm/kasan/report_tags.c | 10 +++++----- mm/slab.c | 2 +- mm/slub.c | 2 +- 9 files changed, 42 insertions(+), 28 deletions(-)
include/linux/kasan.h +5 -4
@@ -9,6 +9,7 @@struct kmem_cache ;struct page ; +struct slab ;struct vm_struct ;struct task_struct ;
@@ -193,11 +194,11 @@ static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache)return 0 ; } -void __kasan_poison_slab (struct page *page); -static __always_inline void kasan_poison_slab(struct page *page) +void __kasan_poison_slab (struct slab *slab); +static __always_inline void kasan_poison_slab(struct slab *slab) {if (kasan_enabled ()) -__kasan_poison_slab (page); +__kasan_poison_slab (slab); }void __kasan_unpoison_object_data (struct kmem_cache *cache,void *object);
@@ -322,7 +323,7 @@ static inline void kasan_cache_create(struct kmem_cache *cache, slab_flags_t *flags) {}static inline void kasan_cache_create_kmalloc (struct kmem_cache *cache) {}static inline size_t kasan_metadata_size (struct kmem_cache *cache) {return 0 ; } -static inline void kasan_poison_slab (struct page *page) {} +static inline void kasan_poison_slab (struct slab *slab) {}static inline void kasan_unpoison_object_data (struct kmem_cache *cache,void *object) {} static inline void kasan_poison_object_data(struct kmem_cache *cache,
mm/kasan/common.c +13 -10
@@ -247,8 +247,9 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache, }#endif -void __kasan_poison_slab (struct page *page) +void __kasan_poison_slab (struct slab *slab) { +struct page *page =slab_page (slab);unsigned long i;for (i =0 ; i <compound_nr (page); i++)
@@ -401,9 +402,9 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)void __kasan_slab_free_mempool (void *ptr,unsigned long ip) { -struct page *page; +struct folio *folio; - page =virt_to_head_page (ptr); + folio =virt_to_folio (ptr);/* * Even though this function is only called for kmem_cache_alloc and
@@ -411,12 +412,14 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip)* !PageSlab() when the size provided to kmalloc is larger than * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ -if (unlikely (!PageSlab (page))) { +if (unlikely (!folio_test_slab (folio))) {if (____kasan_kfree_large (ptr, ip))return ; -kasan_poison (ptr,page_size (page),KASAN_FREE_PAGE , false); +kasan_poison (ptr,folio_size (folio),KASAN_FREE_PAGE , false); }else { -____kasan_slab_free (page->slab_cache, ptr, ip, false, false); +struct slab *slab =folio_slab (folio); + +____kasan_slab_free (slab->slab_cache, ptr, ip, false, false); } }
@@ -560,7 +563,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size,void *__must_check __kasan_krealloc (const void *object,size_t size,gfp_t flags) { -struct page *page; +struct slab *slab;if (unlikely (object ==ZERO_SIZE_PTR ))return (void *)object;
@@ -572,13 +575,13 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag*/ kasan_unpoison (object, size, false); - page =virt_to_head_page (object); + slab =virt_to_slab (object);/* Piggy-back on kmalloc() instrumentation to poison the redzone. */ -if (unlikely (!PageSlab (page))) +if (unlikely (!slab))return __kasan_kmalloc_large (object, size, flags);else -return ____kasan_kmalloc (page->slab_cache, object, size, flags); +return ____kasan_kmalloc (slab->slab_cache, object, size, flags); }bool __kasan_check_byte (const void *address,unsigned long ip)
mm/kasan/generic.c +4 -4
@@ -330,16 +330,16 @@ DEFINE_ASAN_SET_SHADOW(f8);static void __kasan_record_aux_stack (void *addr,bool can_alloc) { -struct page *page =kasan_addr_to_page (addr); +struct slab *slab =kasan_addr_to_slab (addr);struct kmem_cache *cache;struct kasan_alloc_meta *alloc_meta;void *object; -if (is_kfence_address (addr) || !(page &&PageSlab (page))) +if (is_kfence_address (addr) || !slab)return ; - cache = page->slab_cache; - object =nearest_obj (cache,page_slab (page), addr); + cache = slab->slab_cache; + object =nearest_obj (cache, slab, addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (!alloc_meta)return ;
mm/kasan/kasan.h +1 -0
@@ -265,6 +265,7 @@ bool kasan_report(unsigned long addr, size_t size,void kasan_report_invalid_free (void *object,unsigned long ip);struct page *kasan_addr_to_page (const void *addr); +struct slab *kasan_addr_to_slab (const void *addr);depot_stack_handle_t kasan_save_stack (gfp_t flags,bool can_alloc);void kasan_set_track (struct kasan_track *track,gfp_t flags);
mm/kasan/quarantine.c +1 -1
@@ -117,7 +117,7 @@ static unsigned long quarantine_batch_size;static struct kmem_cache *qlink_to_cache (struct qlist_node *qlink) { -return virt_to_head_page (qlink)->slab_cache; +return virt_to_slab (qlink)->slab_cache; }static void *qlink_to_object (struct qlist_node *qlink,struct kmem_cache *cache)
mm/kasan/report.c +11 -2
@@ -150,6 +150,14 @@ struct page *kasan_addr_to_page(const void *addr)return NULL ; } +struct slab *kasan_addr_to_slab (const void *addr) +{ +if ((addr >= (void *)PAGE_OFFSET ) && + (addr < high_memory)) +return virt_to_slab (addr); +return NULL ; +} +static void describe_object_addr (struct kmem_cache *cache,void *object,const void *addr) {
@@ -248,8 +256,9 @@ static void print_address_description(void *addr, u8 tag)pr_err ("\n" );if (page &&PageSlab (page)) { -struct kmem_cache *cache = page->slab_cache; -void *object =nearest_obj (cache,page_slab (page), addr); +struct slab *slab =page_slab (page); +struct kmem_cache *cache = slab->slab_cache; +void *object =nearest_obj (cache, slab, addr);describe_object (cache, object, addr, tag); }
mm/kasan/report_tags.c +5 -5
@@ -12,7 +12,7 @@ const char *kasan_get_bug_type(struct kasan_access_info *info)#ifdef CONFIG_KASAN_TAGS_IDENTIFY struct kasan_alloc_meta *alloc_meta;struct kmem_cache *cache; -struct page *page; +struct slab *slab;const void *addr;void *object;u8 tag;
@@ -20,10 +20,10 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) tag =get_tag (info->access_addr); addr =kasan_reset_tag (info->access_addr); - page =kasan_addr_to_page (addr); -if (page &&PageSlab (page)) { - cache = page->slab_cache; - object =nearest_obj (cache,page_slab (page), (void *)addr); + slab =kasan_addr_to_slab (addr); +if (slab) { + cache = slab->slab_cache; + object =nearest_obj (cache, slab, (void *)addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (alloc_meta) {
mm/slab.c +1 -1
@@ -2605,7 +2605,7 @@ static struct slab *cache_grow_begin(struct kmem_cache *cachep,* page_address() in the latter returns a non-tagged pointer, * as it should be for slab pages. */ -kasan_poison_slab (slab_page (slab)); +kasan_poison_slab (slab);/* Get slab management. */ freelist = alloc_slabmgmt(cachep, slab, offset,
mm/slub.c +1 -1
@@ -1961,7 +1961,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) slab->slab_cache = s; -kasan_poison_slab (slab_page (slab)); +kasan_poison_slab (slab); start =slab_address (slab);
Signature · 1 line
2.33.1
Re: [PATCH v2 25/33] mm/kasan: Convert to struct folio and struct slab
Quoted 274 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
KASAN accesses some slab related struct page fields so we need to convert it to struct slab. Some places are a bit simplified thanks to
kasan_addr_to_slab()encapsulating the PageSlab flag check throughvirt_to_slab().
When resolving object address to either a real slab or a large kmalloc, use struct folio as the intermediate type for testing the slab flag to avoid unnecessary implicitcompound_head().[ [email protected]: use struct folio, adjust to differences in previous patches ]
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Alexander Potapenko <[email protected]>Cc Andrey Konovalov <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc <[email protected]>--- include/linux/kasan.h | 9 +++++---- mm/kasan/common.c | 23 +++++++++++++---------- mm/kasan/generic.c | 8 ++++---- mm/kasan/kasan.h | 1 + mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 13 +++++++++++-- mm/kasan/report_tags.c | 10 +++++----- mm/slab.c | 2 +- mm/slub.c | 2 +- 9 files changed, 42 insertions(+), 28 deletions(-)include/linux/kasan.h +5
-4@@ -9,6 +9,7 @@struct kmem_cache ;struct page ; +struct slab ;struct vm_struct ;struct task_struct ;@@ -193,11 +194,11 @@ static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache)return 0 ; } -void __kasan_poison_slab (struct page *page); -static __always_inline void kasan_poison_slab(struct page *page) +void __kasan_poison_slab (struct slab *slab); +static __always_inline void kasan_poison_slab(struct slab *slab) {if (kasan_enabled ()) -__kasan_poison_slab (page); +__kasan_poison_slab (slab); }void __kasan_unpoison_object_data (struct kmem_cache *cache,void *object);@@ -322,7 +323,7 @@ static inline void kasan_cache_create(struct kmem_cache *cache, slab_flags_t *flags) {}static inline void kasan_cache_create_kmalloc (struct kmem_cache *cache) {}static inline size_t kasan_metadata_size (struct kmem_cache *cache) {return 0 ; } -static inline void kasan_poison_slab (struct page *page) {} +static inline void kasan_poison_slab (struct slab *slab) {}static inline void kasan_unpoison_object_data (struct kmem_cache *cache,void *object) {} static inline void kasan_poison_object_data(struct kmem_cache *cache,mm/kasan/common.c +13
-10@@ -247,8 +247,9 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache, }#endif -void __kasan_poison_slab (struct page *page) +void __kasan_poison_slab (struct slab *slab) { +struct page *page =slab_page (slab);unsigned long i;for (i =0 ; i <compound_nr (page); i++)@@ -401,9 +402,9 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)void __kasan_slab_free_mempool (void *ptr,unsigned long ip) { -struct page *page; +struct folio *folio; - page =virt_to_head_page (ptr); + folio =virt_to_folio (ptr);/* * Even though this function is only called for kmem_cache_alloc and @@ -411,12 +412,14 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip)* !PageSlab() when the size provided to kmalloc is larger than * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ -if (unlikely (!PageSlab (page))) { +if (unlikely (!folio_test_slab (folio))) {if (____kasan_kfree_large (ptr, ip))return ; -kasan_poison (ptr,page_size (page),KASAN_FREE_PAGE , false); +kasan_poison (ptr,folio_size (folio),KASAN_FREE_PAGE , false); }else { -____kasan_slab_free (page->slab_cache, ptr, ip, false, false); +struct slab *slab =folio_slab (folio); + +____kasan_slab_free (slab->slab_cache, ptr, ip, false, false); } }@@ -560,7 +563,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size,void *__must_check __kasan_krealloc (const void *object,size_t size,gfp_t flags) { -struct page *page; +struct slab *slab;if (unlikely (object ==ZERO_SIZE_PTR ))return (void *)object;@@ -572,13 +575,13 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag*/ kasan_unpoison (object, size, false); - page =virt_to_head_page (object); + slab =virt_to_slab (object);/* Piggy-back on kmalloc() instrumentation to poison the redzone. */ -if (unlikely (!PageSlab (page))) +if (unlikely (!slab))return __kasan_kmalloc_large (object, size, flags);else -return ____kasan_kmalloc (page->slab_cache, object, size, flags); +return ____kasan_kmalloc (slab->slab_cache, object, size, flags); }bool __kasan_check_byte (const void *address,unsigned long ip)mm/kasan/generic.c +4
-4@@ -330,16 +330,16 @@ DEFINE_ASAN_SET_SHADOW(f8);static void __kasan_record_aux_stack (void *addr,bool can_alloc) { -struct page *page =kasan_addr_to_page (addr); +struct slab *slab =kasan_addr_to_slab (addr);struct kmem_cache *cache;struct kasan_alloc_meta *alloc_meta;void *object; -if (is_kfence_address (addr) || !(page &&PageSlab (page))) +if (is_kfence_address (addr) || !slab)return ; - cache = page->slab_cache; - object =nearest_obj (cache,page_slab (page), addr); + cache = slab->slab_cache; + object =nearest_obj (cache, slab, addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (!alloc_meta)return ;mm/kasan/kasan.h +1
-0@@ -265,6 +265,7 @@ bool kasan_report(unsigned long addr, size_t size,void kasan_report_invalid_free (void *object,unsigned long ip);struct page *kasan_addr_to_page (const void *addr); +struct slab *kasan_addr_to_slab (const void *addr);depot_stack_handle_t kasan_save_stack (gfp_t flags,bool can_alloc);void kasan_set_track (struct kasan_track *track,gfp_t flags);mm/kasan/quarantine.c +1
-1@@ -117,7 +117,7 @@ static unsigned long quarantine_batch_size;static struct kmem_cache *qlink_to_cache (struct qlist_node *qlink) { -return virt_to_head_page (qlink)->slab_cache; +return virt_to_slab (qlink)->slab_cache; }static void *qlink_to_object (struct qlist_node *qlink,struct kmem_cache *cache)mm/kasan/report.c +11
-2@@ -150,6 +150,14 @@ struct page *kasan_addr_to_page(const void *addr)return NULL ; } +struct slab *kasan_addr_to_slab (const void *addr) +{ +if ((addr >= (void *)PAGE_OFFSET ) && + (addr < high_memory)) +return virt_to_slab (addr); +return NULL ; +} +static void describe_object_addr (struct kmem_cache *cache,void *object,const void *addr) {@@ -248,8 +256,9 @@ static void print_address_description(void *addr, u8 tag)pr_err ("\n" );if (page &&PageSlab (page)) { -struct kmem_cache *cache = page->slab_cache; -void *object =nearest_obj (cache,page_slab (page), addr); +struct slab *slab =page_slab (page); +struct kmem_cache *cache = slab->slab_cache; +void *object =nearest_obj (cache, slab, addr);describe_object (cache, object, addr, tag); }mm/kasan/report_tags.c +5
-5@@ -12,7 +12,7 @@ const char *kasan_get_bug_type(struct kasan_access_info *info)#ifdef CONFIG_KASAN_TAGS_IDENTIFY struct kasan_alloc_meta *alloc_meta;struct kmem_cache *cache; -struct page *page; +struct slab *slab;const void *addr;void *object;u8 tag;@@ -20,10 +20,10 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) tag =get_tag (info->access_addr); addr =kasan_reset_tag (info->access_addr); - page =kasan_addr_to_page (addr); -if (page &&PageSlab (page)) { - cache = page->slab_cache; - object =nearest_obj (cache,page_slab (page), (void *)addr); + slab =kasan_addr_to_slab (addr); +if (slab) { + cache = slab->slab_cache; + object =nearest_obj (cache, slab, (void *)addr); alloc_meta =kasan_get_alloc_meta (cache, object);if (alloc_meta) {mm/slab.c +1
-1@@ -2605,7 +2605,7 @@ static struct slab *cache_grow_begin(struct kmem_cache *cachep,* page_address() in the latter returns a non-tagged pointer, * as it should be for slab pages. */ -kasan_poison_slab (slab_page (slab)); +kasan_poison_slab (slab);/* Get slab management. */ freelist = alloc_slabmgmt(cachep, slab, offset,mm/slub.c +1
-2@@ -1961,7 +1961,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) slab->slab_cache = s; -kasan_poison_slab (slab_page (slab)); +kasan_poison_slab (slab); start =slab_address (slab); --
Quoted 2 lines
2.33.1
Great job with the overall struct page refactoring!
Thanks!
PATCH v2 26/33 mm/kfence: Convert kfence_guarded_alloc() to struct slab
In reply to Vlastimil BabkaThe function sets some fields that are being moved from struct page to struct slab so it needs to be converted.
--- mm/kfence/core.c | 12 ++++++------ mm/kfence/kfence_test.c | 6 +++--- 2 files changed, 9 insertions(+), 9 deletions(-)
mm/kfence/core.c +6 -6
@@ -360,7 +360,7 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g {struct kfence_metadata *meta =NULL ;unsigned long flags; -struct page *page; +struct slab *slab;void *addr;/* Try to obtain a free object. */
@@ -424,13 +424,13 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t galloc_covered_add (alloc_stack_hash,1 ); -/* Set required struct page fields. */ - page =virt_to_page (meta->addr); - page->slab_cache = cache; +/* Set required slab fields. */ + slab =virt_to_slab ((void *)meta->addr); + slab->slab_cache = cache;if (IS_ENABLED (CONFIG_SLUB )) - page->objects =1 ; + slab->objects =1 ;if (IS_ENABLED (CONFIG_SLAB )) - page->s_mem = addr; + slab->s_mem = addr;/* Memory initialization. */ for_each_canary (meta, set_canary_byte);
mm/kfence/kfence_test.c +3 -3
@@ -282,7 +282,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat alloc =kmalloc (size, gfp);if (is_kfence_address (alloc)) { -struct page *page =virt_to_head_page (alloc); +struct slab *slab =virt_to_slab (alloc);struct kmem_cache *s = test_cache ?: kmalloc_caches[kmalloc_type (GFP_KERNEL )][__kmalloc_index (size, false)];
@@ -291,8 +291,8 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat* even for KFENCE objects; these are required so that * memcg accounting works correctly. */ -KUNIT_EXPECT_EQ (test,obj_to_index (s,page_slab (page), alloc),0U ); -KUNIT_EXPECT_EQ (test,objs_per_slab (s,page_slab (page)),1 ); +KUNIT_EXPECT_EQ (test,obj_to_index (s, slab, alloc),0U ); +KUNIT_EXPECT_EQ (test,objs_per_slab (s, slab),1 );if (policy ==ALLOCATE_ANY )return alloc;
Signature · 1 line
2.33.1
PATCH v2 27/33 zsmalloc: Stop using slab fields in struct page
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
The ->freelist and ->units members of struct page are for the use of slab only. I'm not particularly familiar with zsmalloc, so generate the same code by using page->index to store 'page' (page->index and page->freelist are at the same offset in struct page). This should be cleaned up properly at some point by somebody who is familiar with zsmalloc.
--- mm/zsmalloc.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
mm/zsmalloc.c +9 -9
@@ -17,10 +17,10 @@* * Usage of struct page fields: * page->private: points to zspage -* page->freelist(index): links together all component pages of a zspage +* page->index: links together all component pages of a zspage * For the huge page, this is always 0, so we use this field * to store handle. -* page->units: first object offset in a subpage of zspage +* page->page_type: first object offset in a subpage of zspage * * Usage of struct page flags: * PG_private: identifies the first component page
@@ -489,12 +489,12 @@ static inline struct page *get_first_page(struct zspage *zspage)static inline int get_first_obj_offset (struct page *page) { -return page->units; +return page->page_type; }static inline void set_first_obj_offset (struct page *page,int offset) { - page->units = offset; + page->page_type = offset; }static inline unsigned int get_freeobj (struct zspage *zspage)
@@ -827,7 +827,7 @@ static struct page *get_next_page(struct page *page)if (unlikely (PageHugeObject (page)))return NULL ; -return page->freelist; +return (struct page *)page->index; }/**
@@ -901,7 +901,7 @@ static void reset_page(struct page *page)set_page_private (page,0 );page_mapcount_reset (page);ClearPageHugeObject (page); - page->freelist =NULL ; + page->index =0 ; }static int trylock_zspage (struct zspage *zspage)
@@ -1027,7 +1027,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,/* * Allocate individual pages and link them together as: -* 1. all pages are linked together using page->freelist +* 1. all pages are linked together using page->index * 2. each sub-page point to zspage using page->private * * we set PG_private to identify the first page (i.e. no other sub-page
@@ -1036,7 +1036,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,for (i =0 ; i < nr_pages; i++) { page = pages[i];set_page_private (page, (unsigned long )zspage); - page->freelist =NULL ; + page->index =0 ;if (i ==0 ) { zspage->first_page = page;SetPagePrivate (page);
@@ -1044,7 +1044,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage, class->pages_per_zspage == 1))SetPageHugeObject (page); }else { - prev_page->freelist = page; + prev_page->index = (unsigned long )page; } prev_page = page; }
Signature · 1 line
2.33.1
Re: [PATCH v2 27/33] zsmalloc: Stop using slab fields in struct page
Quoted 10 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
The ->freelist and ->units members of struct page are for the use of slab only. I'm not particularly familiar with zsmalloc, so generate the same code by using
page->indexto store 'page' (page->indexandpage->freelistare at the same offset in struct page). This should be cleaned up properly at some point by somebody who is familiar with zsmalloc.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
Re: [PATCH v2 27/33] zsmalloc: Stop using slab fields in struct page
In reply to Vlastimil BabkaQuoted 8 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
The ->freelist and ->units members of struct page are for the use of slab only. I'm not particularly familiar with zsmalloc, so generate the same code by using
page->indexto store 'page' (page->indexandpage->freelistare at the same offset in struct page). This should be cleaned up properly at some point by somebody who is familiar with zsmalloc.
Quoted 5 lines
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>Cc Minchan Kim <[email protected]>Cc Nitin Gupta <[email protected]>Cc Sergey Senozhatsky <[email protected]>
PATCH v2 28/33 bootmem: Use page->index instead of page->freelist
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
page->freelist is for the use of slab. Using page->index is the same set of bits as page->freelist, and by using an integer instead of a pointer, we can avoid casts.
--- arch/x86/mm/init_64.c | 2 +- include/linux/bootmem_info.h | 2 +- mm/bootmem_info.c | 7 +++---- mm/sparse.c | 2 +- 4 files changed, 6 insertions(+), 7 deletions(-)
arch/x86/mm/init_64.c +1 -1
@@ -981,7 +981,7 @@ static void __meminit free_pagetable(struct page *page, int order)if (PageReserved (page)) {__ClearPageReserved (page); - magic = (unsigned long )page->freelist; + magic = page->index;if (magic ==SECTION_INFO || magic ==MIX_SECTION_INFO ) {while (nr_pages--)put_page_bootmem (page++);
include/linux/bootmem_info.h +1 -1
@@ -30,7 +30,7 @@ void put_page_bootmem(struct page *page);*/ static inline void free_bootmem_page (struct page *page) { -unsigned long magic = (unsigned long )page->freelist; +unsigned long magic = page->index;/* * The reserve_bootmem_region sets the reserved flag on bootmem
mm/bootmem_info.c +3 -4
@@ -15,7 +15,7 @@void get_page_bootmem (unsigned long info,struct page *page,unsigned long type) { - page->freelist = (void *)type; + page->index = type;SetPagePrivate (page);set_page_private (page, info);page_ref_inc (page);
@@ -23,14 +23,13 @@ void get_page_bootmem(unsigned long info, struct page *page, unsigned long type)void put_page_bootmem (struct page *page) { -unsigned long type; +unsigned long type = page->index; - type = (unsigned long ) page->freelist;BUG_ON (type <MEMORY_HOTPLUG_MIN_BOOTMEM_TYPE || type >MEMORY_HOTPLUG_MAX_BOOTMEM_TYPE );if (page_ref_dec_return (page) ==1 ) { - page->freelist =NULL ; + page->index =0 ;ClearPagePrivate (page);set_page_private (page,0 );INIT_LIST_HEAD (&page->lru);
mm/sparse.c +1 -1
@@ -722,7 +722,7 @@ static void free_map_bootmem(struct page *memmap) >> PAGE_SHIFT;for (i =0 ; i < nr_pages; i++, page++) { - magic = (unsigned long ) page->freelist; + magic = page->index;BUG_ON (magic ==NODE_INFO );
Signature · 1 line
2.33.1
Re: [PATCH v2 28/33] bootmem: Use page->index instead of page->freelist
Quoted 6 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
page->freelistis for the use of slab. Usingpage->indexis the same set of bits aspage->freelist, and by using an integer instead of a pointer, we can avoid casts.
Quoted 10 lines
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>Cc Dave Hansen <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc <[email protected]>Cc "H. Peter Anvin" <[email protected]>
PATCH v2 29/33 iommu: Use put_pages_list
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
page->freelist is for the use of slab. We already have the ability to free a list of pages in the core mm, but it requires the use of a list_head and for the pages to be chained together through page->lru. Switch the iommu code over to using put_pages_list().
--- drivers/iommu/amd/io_pgtable.c | 59 +++++++++------------- drivers/iommu/dma-iommu.c | 11 +---- drivers/iommu/intel/iommu.c | 89 ++++++++++++---------------------- include/linux/iommu.h | 3 +- 4 files changed, 57 insertions(+), 105 deletions(-)
drivers/iommu/amd/io_pgtable.c +22 -37
@@ -74,27 +74,15 @@ static u64 *first_pte_l7(u64 *pte, unsigned long *page_size,* ****************************************************************************/ -static void free_page_list (struct page *freelist) -{ -while (freelist !=NULL ) { -unsigned long p = (unsigned long )page_address (freelist); - - freelist = freelist->freelist; -free_page (p); - } -} - -static struct page *free_pt_page (unsigned long pt,struct page *freelist) +static void free_pt_page (unsigned long pt,struct list_head *list) {struct page *p =virt_to_page ((void *)pt); - p->freelist = freelist; - -return p; +list_add (&p->lru, list); }#define DEFINE_FREE_PT_FN (LVL ,FN ) \ -static struct page *free_pt_##LVL (unsigned long __pt, struct page *freelist) \ +static void free_pt_##LVL (unsigned long __pt, struct list_head *list) \ { \ unsigned long p; \ u64 *pt; \
@@ -113,10 +101,10 @@ static struct page *free_pt_##LVL (unsigned long __pt, struct page *freelist) \continue ; \ \ p = (unsigned long )IOMMU_PTE_PAGE (pt[i]); \ - freelist =FN (p, freelist); \ +FN (p, list); \ } \ \ -return free_pt_page ((unsigned long )pt, freelist); \ +free_pt_page ((unsigned long )pt, list); \ }DEFINE_FREE_PT_FN (l2, free_pt_page)
@@ -125,36 +113,33 @@ DEFINE_FREE_PT_FN(l4, free_pt_l3) DEFINE_FREE_PT_FN(l5, free_pt_l4) DEFINE_FREE_PT_FN(l6, free_pt_l5) -static struct page *free_sub_pt (unsigned long root,int mode, -struct page *freelist) +static void free_sub_pt (unsigned long root,int mode,struct list_head *list) {switch (mode) {case PAGE_MODE_NONE :case PAGE_MODE_7_LEVEL :break ;case PAGE_MODE_1_LEVEL : - freelist =free_pt_page (root, freelist); +free_pt_page (root, list);break ;case PAGE_MODE_2_LEVEL : - freelist =free_pt_l2 (root, freelist); +free_pt_l2 (root, list);break ;case PAGE_MODE_3_LEVEL : - freelist =free_pt_l3 (root, freelist); +free_pt_l3 (root, list);break ;case PAGE_MODE_4_LEVEL : - freelist =free_pt_l4 (root, freelist); +free_pt_l4 (root, list);break ;case PAGE_MODE_5_LEVEL : - freelist =free_pt_l5 (root, freelist); +free_pt_l5 (root, list);break ;case PAGE_MODE_6_LEVEL : - freelist =free_pt_l6 (root, freelist); +free_pt_l6 (root, list);break ;default :BUG (); } - -return freelist; } void amd_iommu_domain_set_pgtable(struct protection_domain *domain,
@@ -362,7 +347,7 @@ static u64 *fetch_pte(struct amd_io_pgtable *pgtable,return pte; } -static struct page *free_clear_pte (u64 *pte,u64 pteval,struct page *freelist) +static void free_clear_pte (u64 *pte,u64 pteval,struct list_head *list) {unsigned long pt;int mode;
@@ -373,12 +358,12 @@ static struct page *free_clear_pte(u64 *pte, u64 pteval, struct page *freelist) }if (!IOMMU_PTE_PRESENT (pteval)) -return freelist; +return ; pt = (unsigned long )IOMMU_PTE_PAGE (pteval); mode =IOMMU_PTE_MODE (pteval); -return free_sub_pt (pt, mode, freelist); +free_sub_pt (pt, mode, list); }/*
@@ -392,7 +377,7 @@ static int iommu_v1_map_page(struct io_pgtable_ops *ops, unsigned long iova, phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct protection_domain *dom = io_pgtable_ops_to_domain(ops); -struct page *freelist =NULL ; +LIST_HEAD (freelist);bool updated = false;u64 __pte, *pte;int ret, i, count;
@@ -412,9 +397,9 @@ static int iommu_v1_map_page(struct io_pgtable_ops *ops, unsigned long iova, goto out;for (i =0 ; i < count; ++i) - freelist =free_clear_pte (&pte[i], pte[i], freelist); +free_clear_pte (&pte[i], pte[i], &freelist); -if (freelist !=NULL ) +if (!list_empty (&freelist)) updated = true;if (count >1 ) {
@@ -449,7 +434,7 @@ static int iommu_v1_map_page(struct io_pgtable_ops *ops, unsigned long iova, }/* Everything flushed out, free pages now */ -free_page_list (freelist); +put_pages_list (&freelist);return ret; }
@@ -511,7 +496,7 @@ static void v1_free_pgtable(struct io_pgtable *iop) { struct amd_io_pgtable *pgtable = container_of(iop, struct amd_io_pgtable, iop);struct protection_domain *dom; -struct page *freelist =NULL ; +LIST_HEAD (freelist);unsigned long root;if (pgtable->mode ==PAGE_MODE_NONE )
@@ -530,9 +515,9 @@ static void v1_free_pgtable(struct io_pgtable *iop) pgtable->mode > PAGE_MODE_6_LEVEL); root = (unsigned long )pgtable->root; - freelist =free_sub_pt (root, pgtable->mode, freelist); +free_sub_pt (root, pgtable->mode, &freelist); -free_page_list (freelist); +put_pages_list (&freelist); }static struct io_pgtable *v1_alloc_pgtable (struct io_pgtable_cfg *cfg,void *cookie)
drivers/iommu/dma-iommu.c +2 -9
@@ -66,14 +66,7 @@ early_param("iommu.forcedac", iommu_dma_forcedac_setup);static void iommu_dma_entry_dtor (unsigned long data) { -struct page *freelist = (struct page *)data; - -while (freelist) { -unsigned long p = (unsigned long )page_address (freelist); - - freelist = freelist->freelist; -free_page (p); - } +put_pages_list ((struct list_head *)data); }static inline size_t cookie_msi_granule (struct iommu_dma_cookie *cookie)
@@ -479,7 +472,7 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie,else if (gather && gather->queued)queue_iova (iovad,iova_pfn (iovad, iova), size >>iova_shift (iovad), - (unsigned long )gather->freelist); + (unsigned long )&gather->freelist);else free_iova_fast (iovad,iova_pfn (iovad, iova), size >>iova_shift (iovad));
drivers/iommu/intel/iommu.c +31 -58
177 lines not on this page · raw message
include/linux/iommu.h +2 -1
@@ -186,7 +186,7 @@ struct iommu_iotlb_gather {unsigned long start;unsigned long end;size_t pgsize; -struct page *freelist; +struct list_head freelist;bool queued; };
@@ -399,6 +399,7 @@ static inline void iommu_iotlb_gather_init(struct iommu_iotlb_gather *gather) { *gather = (struct iommu_iotlb_gather ) { .start =ULONG_MAX , + .freelist =LIST_HEAD_INIT (gather->freelist), }; }
Signature · 1 line
2.33.1
Re: [PATCH v2 29/33] iommu: Use put_pages_list
Quoted 6 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
page->freelistis for the use of slab. We already have the ability to free a list of pages in the core mm, but it requires the use of alist_headand for the pages to be chained together throughpage->lru. Switch the iommu code over to usingput_pages_list().
FYI, this is going to have conflicts with
https://lore.kernel.org/lkml/[email protected]/
I'm not sure what the appropriate resolution is going to be here; maybe hold back part of this patch series to the following merge window to give the iommu people time to merge their own patches?
Robin Murphy
maintainerRe: [PATCH v2 29/33] iommu: Use put_pages_list
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
page->freelistis for the use of slab. We already have the ability to free a list of pages in the core mm, but it requires the use of alist_headand for the pages to be chained together throughpage->lru. Switch the iommu code over to usingput_pages_list().
Quoted 6 lines
FYI, this is going to have conflicts with
https://lore.kernel.org/lkml/[email protected]/I'm not sure what the appropriate resolution is going to be here; maybe hold back part of this patch series to the following merge window to give the iommu people time to merge their own patches?
More than that, this version is subtly but catastrophically broken - we can't simply pass &gather->freelist through the current IOVA entry_dtor machinery, since gather is a stack variable from a few call frames up so the actual list head will be long gone by the time iommu_dma_entry_dtor() tries to dereference it. It took until I was elbow-deep in refactoring the RFC to notice that one :)
Thanks,
Robin.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v2 30/33 mm: Remove slab from struct page
In reply to Vlastimil BabkaFrom: "Matthew Wilcox (Oracle)" <[email protected]>
All members of struct slab can now be removed from struct page. This shrinks the definition of struct page by 30 LOC, making it easier to understand.
--- include/linux/mm_types.h | 28 ---------------------------- include/linux/page-flags.h | 37 ------------------------------------- mm/slab.h | 4 ---- 3 files changed, 69 deletions(-)
include/linux/mm_types.h +0 -28
@@ -118,31 +118,6 @@ struct page {atomic_long_t pp_frag_count; }; }; -struct {/* slab, slob and slub */ -union { -struct list_head slab_list; -struct {/* Partial pages */ -struct page *next; -#ifdef CONFIG_64BIT -int pages;/* Nr of pages left */ -#else -short int pages; -#endif - }; - }; -struct kmem_cache *slab_cache;/* not slob */ -/* Double-word boundary */ -void *freelist;/* first free object */ -union { -void *s_mem;/* slab: first object */ -unsigned long counters;/* SLUB */ -struct {/* SLUB */ -unsigned inuse:16 ; -unsigned objects:15 ; -unsigned frozen:1 ; - }; - }; - };struct {/* Tail pages of compound page */ unsigned long compound_head;/* Bit zero is set */
@@ -206,9 +181,6 @@ struct page {* which are currently stored here. */ unsigned int page_type; - -unsigned int active;/* SLAB */ -int units;/* SLOB */ };/* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */
include/linux/page-flags.h +0 -37
@@ -909,43 +909,6 @@ extern bool is_free_buddy_page(struct page *page);__PAGEFLAG (Isolated, isolated,PF_ANY ); -/* -* If network-based swap is enabled, sl*b must keep track of whether pages -* were allocated from pfmemalloc reserves. -*/ -static inline int PageSlabPfmemalloc (struct page *page) -{ -VM_BUG_ON_PAGE (!PageSlab (page), page); -return PageActive (page); -} - -/* -* A version of PageSlabPfmemalloc() for opportunistic checks where the page -* might have been freed under us and not be a PageSlab anymore. -*/ -static inline int __PageSlabPfmemalloc (struct page *page) -{ -return PageActive (page); -} - -static inline void SetPageSlabPfmemalloc (struct page *page) -{ -VM_BUG_ON_PAGE (!PageSlab (page), page); -SetPageActive (page); -} - -static inline void __ClearPageSlabPfmemalloc (struct page *page) -{ -VM_BUG_ON_PAGE (!PageSlab (page), page); -__ClearPageActive (page); -} - -static inline void ClearPageSlabPfmemalloc (struct page *page) -{ -VM_BUG_ON_PAGE (!PageSlab (page), page); -ClearPageActive (page); -} -#ifdef CONFIG_MMU #define __PG_MLOCKED (1UL << PG_mlocked)#else
mm/slab.h +0 -4
@@ -47,11 +47,7 @@ struct slab { static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl))SLAB_MATCH (flags, __page_flags);SLAB_MATCH (compound_head, slab_list);/* Ensure bit 0 is clear */ -SLAB_MATCH (slab_list, slab_list);SLAB_MATCH (rcu_head, rcu_head); -SLAB_MATCH (slab_cache, slab_cache); -SLAB_MATCH (s_mem, s_mem); -SLAB_MATCH (active, active);SLAB_MATCH (_refcount, __page_refcount);#ifdef CONFIG_MEMCG SLAB_MATCH (memcg_data, memcg_data);
Signature · 1 line
2.33.1
Re: [PATCH v2 30/33] mm: Remove slab from struct page
Quoted 8 lines · Vlastimil Babka
From: "Matthew Wilcox (Oracle)" <[email protected]>
All members of struct slab can now be removed from struct page. This shrinks the definition of struct page by 30 LOC, making it easier to understand.
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
PATCH v2 31/33 mm/sl*b: Differentiate struct slab fields by sl*b implementations
In reply to Vlastimil BabkaWith a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between __page_flags and __page_refcount placeholders now depends on the chosen CONFIG_SL*B. Some fields exist in all implementations (slab_list) but can be part of a union in some, so it's simpler to repeat them than complicate the definition with ifdefs even more.
The patch doesn't change physical offsets of the fields, although it could be done later - for example it's now clear that tighter packing in SLOB could be possible.
This should also prevent accidental use of fields that don't exist in given implementation. Before this patch virt_to_cache() and and cache_from_obj() was visible for SLOB (albeit not used), although it relies on the slab_cache field that isn't set by SLOB. With this patch it's now a compile error, so these functions are now hidden behind #ifndef CONFIG_SLOB.
--- mm/kfence/core.c | 9 +++++---- mm/slab.h | 46 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 41 insertions(+), 14 deletions(-)
mm/kfence/core.c +5 -4
@@ -427,10 +427,11 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g/* Set required slab fields. */ slab =virt_to_slab ((void *)meta->addr); slab->slab_cache = cache; -if (IS_ENABLED (CONFIG_SLUB )) - slab->objects =1 ; -if (IS_ENABLED (CONFIG_SLAB )) - slab->s_mem = addr; +#if defined(CONFIG_SLUB ) + slab->objects =1 ; +#elif defined (CONFIG_SLAB ) + slab->s_mem = addr; +#endif /* Memory initialization. */ for_each_canary (meta, set_canary_byte);
mm/slab.h +36 -10
89 lines not on this page · raw message
Signature · 1 line
2.33.1
Re: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
Quoted 4 lines · Vlastimil Babka
With a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between
__page_flagsand__page_refcountplaceholders now depends on the chosen CONFIG_SL*B.
When I read this patch series first, I thought struct slab is allocated separately from struct page.
But after reading it again, It uses same allocated space of struct page.
So, the code should care about fields that page allocator cares when freeing page. (->mapping, ->refcount, ->flags, ...)
And, we can change offset of fields between page->flags and page->refcount, If we care about the value of page->mapping before freeing it.
Did I get it right?
Quoted 3 lines
Some fields exist in all implementations (
slab_list)
but can be part of a union in some, so it's simpler to repeat them than complicate the definition with ifdefs even more.
Before this patch I always ran preprocessor in my brain. now it's MUCH easier to understand than before!
Quoted 5 lines
The patch doesn't change physical offsets of the fields, although it could be done later - for example it's now clear that tighter packing in SLOB could be possible.
Is there a benefit if we pack SLOB's struct slab tighter?
...
mm/slab.h struct slab { · line 73
Quoted 8 lines
#ifdef
CONFIG_MEMCGmm/slab.h +1
-0unsigned long memcg_data;@@ -47,7 +69,9 @@ struct slab { static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl))SLAB_MATCH (flags, __page_flags);SLAB_MATCH (compound_head, slab_list);/* Ensure bit 0 is clear */ +#ifndef CONFIG_SLOB SLAB_MATCH (rcu_head, rcu_head);
Because SLUB and SLAB sets slab->slab_cache = NULL (to set page->mapping = NULL), What about adding this?:
SLAB_MATCH(mapping, slab_cache);
there was SLAB_MATCH(slab_cache, slab_cache) but removed.
mm/slab.h struct slab { · line 77
Quoted 4 lines
mm/slab.h +1
-0+#endif SLAB_MATCH (_refcount, __page_refcount);#ifdef CONFIG_MEMCG SLAB_MATCH (memcg_data, memcg_data);
I couldn't find any functional problem on this patch. but it seems there's some style issues.
Below is what checkpatch.pl complains.
it's better to fix them!
WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line) #7:
With a struct slab definition separate from struct page, we can go further and
WARNING: Possible repeated word: 'and'
#19:
implementation. Before this patch virt_to_cache() and and cache_from_obj() was
WARNING: space prohibited between function name and open parenthesis '(' #49: FILE: mm/kfence/core.c:432:
+#elif defined (CONFIG_SLAB)
ERROR: "foo * bar" should be "foo *bar"
#73: FILE: mm/slab.h:20:
+void * s_mem;/* first object */
ERROR: "foo * bar" should be "foo *bar"
#111: FILE: mm/slab.h:53:
+void * __unused_1;
ERROR: "foo * bar" should be "foo *bar"
#113: FILE: mm/slab.h:55:
+void * __unused_2;
---
Thanks,
Hyeonggon.
Re: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
Quoted 10 lines · Hyeonggon Yoo
Quoted 4 lines · Vlastimil Babka
With a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between
__page_flagsand__page_refcountplaceholders now depends on the chosen CONFIG_SL*B.When I read this patch series first, I thought struct slab is allocated separately from struct page.
But after reading it again, It uses same allocated space of struct page.
Yes. Allocating it elsewhere is something that can be discussed later. It's not a simple clear win - more memory used, more overhead, complicated code...
Quoted 7 lines
So, the code should care about fields that page allocator cares when freeing page. (->mapping, ->refcount, ->flags, ...)
And, we can change offset of fields between
page->flagsandpage->refcount, If we care about the value ofpage->mappingbefore freeing it.Did I get it right?
Yeah. Also whatever aliases with compound_head must not have bit zero set as that means a tail page.
Quoted 8 lines
Quoted 3 lines
Some fields exist in all implementations (
slab_list)
but can be part of a union in some, so it's simpler to repeat them than complicate the definition with ifdefs even more.Before this patch I always ran preprocessor in my brain. now it's MUCH easier to understand than before!
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
The patch doesn't change physical offsets of the fields, although it could be done later - for example it's now clear that tighter packing in SLOB could be possible.
Is there a benefit if we pack SLOB's struct slab tighter?
I don't see any immediate benefit, except avoiding the page->mapping alias as you suggested.
Quoted 12 lines
...
mm/slab.h struct slab { · line 73
Quoted 8 lines
#ifdef
CONFIG_MEMCGmm/slab.h +1
-0unsigned long memcg_data;@@ -47,7 +69,9 @@ struct slab { static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl))SLAB_MATCH (flags, __page_flags);SLAB_MATCH (compound_head, slab_list);/* Ensure bit 0 is clear */ +#ifndef CONFIG_SLOB SLAB_MATCH (rcu_head, rcu_head);Because SLUB and SLAB sets
slab->slab_cache= NULL (to setpage->mapping= NULL),
Hm, now that you mention it, maybe it would be better to do a "folio->mapping = NULL" instead as we now have a more clearer view where we operate on struct slab, and where we transition between that and a plain folio. This is IMHO part of preparing the folio for freeing, not a struct slab cleanup as struct slab doesn't need this cleanup.
Quoted 5 lines
What about adding this?:
SLAB_MATCH(mapping, slab_cache);there was
SLAB_MATCH(slab_cache, slab_cache)but removed.
With the change suggested above, it wouldn't be needed as a safety check anymore.
Quoted 10 lines
mm/slab.h struct slab { · line 77
Quoted 4 lines
mm/slab.h +1
-0+#endif SLAB_MATCH (_refcount, __page_refcount);#ifdef CONFIG_MEMCG SLAB_MATCH (memcg_data, memcg_data);I couldn't find any functional problem on this patch. but it seems there's some style issues.
Below is what checkpatch.pl complains.
it's better to fix them!
Not all checkpatch suggestions are correct and have to be followed, but I'll check what I missed. Thanks.
Quoted 27 lines
WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line) #7:
With a struct slab definition separate from struct page, we can go further andWARNING: Possible repeated word: 'and'
#19:
implementation. Before this patchvirt_to_cache()and andcache_from_obj()wasWARNING: space prohibited between function name and open parenthesis '(' #49: FILE:
mm/kfence/core.c:432:
+#elif defined (CONFIG_SLAB)ERROR: "foo * bar" should be "foo *bar"
#73: FILE:mm/slab.h:20:
+void * s_mem;/* first object */ERROR: "foo * bar" should be "foo *bar"
#111: FILE:mm/slab.h:53:
+void *__unused_1;ERROR: "foo * bar" should be "foo *bar"
#113: FILE:mm/slab.h:55:
+void *__unused_2;---
Thanks,
Hyeonggon.
Hyeonggon Yoo
Reviewed-byRe: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
Quoted 7 lines · Vlastimil Babka
Quoted 6 lines · Hyeonggon Yoo
Quoted 4 lines · Vlastimil Babka
With a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between
__page_flagsand__page_refcountplaceholders now depends on the chosen CONFIG_SL*B.
Quoted 8 lines
Quoted 4 lines
When I read this patch series first, I thought struct slab is allocated separately from struct page.
But after reading it again, It uses same allocated space of struct page.
Yes. Allocating it elsewhere is something that can be discussed later. It's not a simple clear win - more memory used, more overhead, complicated code...
Right. That is a something that can be discussed,
But I don't think there will be much win.
Quoted 11 lines
Quoted 7 lines
So, the code should care about fields that page allocator cares when freeing page. (->mapping, ->refcount, ->flags, ...)
And, we can change offset of fields between
page->flagsandpage->refcount, If we care about the value ofpage->mappingbefore freeing it.Did I get it right?
Yeah. Also whatever aliases with
compound_headmust not have bit zero set as that means a tail page.
Oh I was missing that. Thank you.
Hmm then in struct slab, page->compound_head and slab->list_head (or slab->rcu_head) has same offset. And list_head / rcu_head both store pointers.
then it has a alignment requirement. (address saved in list_head/rcu_head should be multiple of 2)
Anyway, it was required long time before this patch, so it is not a problem for this patch.
Quoted 31 lines
Quoted 8 lines
Quoted 3 lines
Some fields exist in all implementations (
slab_list)
but can be part of a union in some, so it's simpler to repeat them than complicate the definition with ifdefs even more.Before this patch I always ran preprocessor in my brain. now it's MUCH easier to understand than before!
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
The patch doesn't change physical offsets of the fields, although it could be done later - for example it's now clear that tighter packing in SLOB could be possible.
Is there a benefit if we pack SLOB's struct slab tighter?
I don't see any immediate benefit, except avoiding the
page->mappingalias as you suggested.Quoted 12 lines
...
mm/slab.h struct slab { · line 73
Quoted 8 lines
#ifdef
CONFIG_MEMCGmm/slab.h +1
-0unsigned long memcg_data;@@ -47,7 +69,9 @@ struct slab { static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl))SLAB_MATCH (flags, __page_flags);SLAB_MATCH (compound_head, slab_list);/* Ensure bit 0 is clear */ +#ifndef CONFIG_SLOB SLAB_MATCH (rcu_head, rcu_head);Because SLUB and SLAB sets
slab->slab_cache= NULL (to setpage->mapping= NULL),
Quoted 4 lines
Hm, now that you mention it, maybe it would be better to do a "
folio->mapping= NULL" instead as we now have a more clearer view where we operate on struct slab, and where we transition between that and a plain folio.
Oh, folio->mapping = NULL seems more intuitive.
And we can reorder fields of struct slab more flexibly with folio->mapping = NULL because we have no reason to make page->mapping and slab->slab_cache to have same offset.
So it should be done in separate patch for SLUB/SLAB. Do you mind If I send a patch for this after some testing?
Quoted 2 lines
This is IMHO part of preparing the folio for freeing, not a struct slab cleanup as struct slab doesn't need this cleanup.
I agree that. it's needed for folio, not for struct slab.
Quoted 9 lines
Quoted 5 lines
What about adding this?:
SLAB_MATCH(mapping, slab_cache);there was
SLAB_MATCH(slab_cache, slab_cache)but removed.With the change suggested above, it wouldn't be needed as a safety check anymore.
Okay.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
mm/slab.h +1
-0+#endif SLAB_MATCH (_refcount, __page_refcount);#ifdef CONFIG_MEMCG SLAB_MATCH (memcg_data, memcg_data);
Quoted 9 lines
Quoted 5 lines
I couldn't find any functional problem on this patch. but it seems there's some style issues.
Below is what checkpatch.pl complains.
it's better to fix them!Not all checkpatch suggestions are correct and have to be followed, but I'll check what I missed. Thanks.
You're welcome.
They are just a typo in changelog and white space warnings.
So now, exept few style issues, I can't find any problem in this patch. And this patch gives us much better view of struct slab.
Thanks,
Hyeonggon.
Quoted 28 lines
Quoted 27 lines
WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line) #7:
With a struct slab definition separate from struct page, we can go further andWARNING: Possible repeated word: 'and'
#19:
implementation. Before this patchvirt_to_cache()and andcache_from_obj()wasWARNING: space prohibited between function name and open parenthesis '(' #49: FILE:
mm/kfence/core.c:432:
+#elif defined (CONFIG_SLAB)ERROR: "foo * bar" should be "foo *bar"
#73: FILE:mm/slab.h:20:
+void * s_mem;/* first object */ERROR: "foo * bar" should be "foo *bar"
#111: FILE:mm/slab.h:53:
+void *__unused_1;ERROR: "foo * bar" should be "foo *bar"
#113: FILE:mm/slab.h:55:
+void *__unused_2;---
Thanks,
Hyeonggon.
Re: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
Quoted 11 lines · Hyeonggon Yoo
Quoted 10 lines · Vlastimil Babka
Quoted 9 lines · Hyeonggon Yoo
Quoted 4 lines · Vlastimil Babka
With a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between
__page_flagsand__page_refcountplaceholders now depends on the chosen CONFIG_SL*B.When I read this patch series first, I thought struct slab is allocated separately from struct page.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
But after reading it again, It uses same allocated space of struct page.
Yes. Allocating it elsewhere is something that can be discussed later. It's not a simple clear win - more memory used, more overhead, complicated code...
Right. That is a something that can be discussed,
But I don't think there will be much win.
Oh no, there's a substantial win. If we can reduce struct page to a single pointer, that shrinks it from 64 bytes/4k to 8 bytes/4k. Set against that, you have to allocate the struct folio / struct slab / ... but then it's one _per allocation_ rather than one per page. So for an order-2 allocation, it takes 32 bytes + 64 bytes (= 96 bytes) rather than 4*64 = 256 bytes. It's an even bigger win for larger allocations, and it lets us grow the memory descriptors independently of each other.
But it's also a substantial amount of work, so don't expect us to get there any time soon. Everything currently using struct page needs to be converted to use another type, and that's just the pre-requisite step.
Some more thoughts on this here:
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 6 lines
Quoted 3 lines
Yeah. Also whatever aliases with
compound_headmust not have bit zero set as that means a tail page.Oh I was missing that. Thank you.
Quoted 8 lines
Hmm then in struct slab,
page->compound_headandslab->list_head(orslab->rcu_head) has same offset. Andlist_head/rcu_headboth store pointers.then it has a alignment requirement. (address saved in list_head/rcu_head should be multiple of 2)
Anyway, it was required long time before this patch, so it is not a problem for this patch.
Yes, that's why there's an assert that the list_heads all line up. This requirement will go away if we do get separately allocated memory descriptors (because that bottom bit is no longer PageTail).
Re: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
Quoted 21 lines · Matthew Wilcox
Quoted 11 lines · Hyeonggon Yoo
Quoted 10 lines · Vlastimil Babka
Quoted 9 lines · Hyeonggon Yoo
Quoted 4 lines · Vlastimil Babka
With a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between
__page_flagsand__page_refcountplaceholders now depends on the chosen CONFIG_SL*B.When I read this patch series first, I thought struct slab is allocated separately from struct page.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
But after reading it again, It uses same allocated space of struct page.
Yes. Allocating it elsewhere is something that can be discussed later. It's not a simple clear win - more memory used, more overhead, complicated code...
Right. That is a something that can be discussed,
But I don't think there will be much win.
Quoted 8 lines
Oh no, there's a substantial win. If we can reduce struct page to a single pointer, that shrinks it from 64 bytes/4k to 8 bytes/4k. Set against that, you have to allocate the struct folio / struct slab / ... but then it's one
_perallocation_rather than one per page. So for an order-2 allocation, it takes 32 bytes + 64 bytes (= 96 bytes) rather than 4*64 = 256 bytes. It's an even bigger win for larger allocations, and it lets us grow the memory descriptors independently of each other.
Oh I thought there won't be much win because I thought it was just allocating additional memory for struct slab and still allocating memory for struct page as we do now.
It will be more efficient if we can allocate descriptor of slab/page/...etc per *allocation*, which may have order > 1. And currently we're duplicating memory descriptor (struct page) even on high order allocation.
Even if we do not allocate high order page at all, it's still efficient if we can reduce struct page into double word. And we can allocate something like struct slab only when we need it.
One challenge here is that we should allocate the descriptors dynamically... I'm going to read the link you sent.
Quoted 4 lines
Everything currently using struct page needs to
be converted to use another type, and that's just the pre-requisite step.
Oh, you're planning to separate *everything* from
struct page, not only struct slab!
So your intention of this patch series is preparing for physical separation. It's fascinating...
Quoted 2 lines
But it's also a substantial amount of work, so don't expect us to get there any time soon.
Yeah, that will require much work. But I'll wait for your good work. It's so interesting.
Quoted 3 lines
Some more thoughts on this here:
https://lore.kernel.org/linux-mm/[email protected]/
Thank you for the link.
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Yeah. Also whatever aliases with
compound_headmust not have bit zero set as that means a tail page.Oh I was missing that. Thank you.
Hmm then in struct slab,
page->compound_headandslab->list_head(orslab->rcu_head) has same offset. Andlist_head/rcu_headboth store pointers.then it has a alignment requirement. (address saved in list_head/rcu_head should be multiple of 2)
Quoted 6 lines
Quoted 2 lines
Anyway, it was required long time before this patch, so it is not a problem for this patch.
Yes, that's why there's an assert that the
list_headsall line up. This requirement will go away if we do get separately allocated memory descriptors (because that bottom bit is no longer PageTail).
Yeah, we don't need to care that if we separately allocate memory for struct slab.
Thanks,
Hyeonggon.
Re: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
In reply to Vlastimil BabkaQuoted 7 lines · Vlastimil Babka
Quoted 1 line
Because SLUB and SLAB sets
slab->slab_cache= NULL (to setpage->mapping= NULL),Hm, now that you mention it, maybe it would be better to do a "
folio->mapping= NULL" instead as we now have a more clearer view where we operate on struct slab, and where we transition between that and a plain folio. This is IMHO part of preparing the folio for freeing, not a struct slab cleanup as struct slab doesn't need this cleanup.
Yes, I did that as part of "mm/slub: Convert slab freeing to struct slab" in my original series:
- __ClearPageSlabPfmemalloc(page);
+ __slab_clear_pfmemalloc(slab);
__ClearPageSlab(page);
- /* In union with page->mapping where page allocator expects NULL */
- page->slab_cache = NULL;
+ page->mapping = NULL;Re: [PATCH v2 31/33] mm/sl*b: Differentiate struct slab fields by sl*b implementations
Quoted 12 lines · Matthew Wilcox
Quoted 7 lines · Vlastimil Babka
Quoted 1 line
Because SLUB and SLAB sets
slab->slab_cache= NULL (to setpage->mapping= NULL),Hm, now that you mention it, maybe it would be better to do a "
folio->mapping= NULL" instead as we now have a more clearer view where we operate on struct slab, and where we transition between that and a plain folio. This is IMHO part of preparing the folio for freeing, not a struct slab cleanup as struct slab doesn't need this cleanup.Yes, I did that as part of "mm/slub: Convert slab freeing to struct slab" in my original series:
Review
Quoted 6 lines
(quoted diff) +2
-3- __ClearPageSlabPfmemalloc(page); + __slab_clear_pfmemalloc(slab); __ClearPageSlab(page); - /* In union with page->mapping where page allocator expects NULL */ - page->slab_cache = NULL; + page->mapping = NULL;
Didn't know your original patch series did it already. Vlastimil, would you update that patch?
PATCH 1/1 mm/slob: Remove unnecessary page_mapcount_reset() function call
In reply to Vlastimil BabkaAfter commit f1ac9059ca34 ("mm/sl*b: Differentiate struct slab fields by sl*b implementations"), we can reorder fields of struct slab depending on slab allocator.
For now, page_mapcount_reset() is called because page->_mapcount and slab->units have same offset. But this is not necessary for struct slab. Use unused field for units instead.
--- mm/slab.h | 4 ++-- mm/slob.c | 1 - 2 files changed, 2 insertions(+), 3 deletions(-)
mm/slab.h +2 -2
@@ -50,8 +50,8 @@ struct slab {struct list_head slab_list;void * __unused_1;void *freelist;/* first free block */ -void * __unused_2; -int units; +long units; +unsigned int __unused_2;#else #error "Unexpected slab allocator configured"
mm/slob.c +0 -1
@@ -404,7 +404,6 @@ static void slob_free(void *block, int size)clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags);__ClearPageSlab (slab_page (sp)); -page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }
Signature · 1 line
2.30.2
Re: [PATCH] mm/slob: Remove unnecessary page_mapcount_reset() function call
Quoted 9 lines · Hyeonggon Yoo
After commit
f1ac9059ca34("mm/sl*b: Differentiate struct slab fields by sl*b implementations"), we can reorder fields of struct slab depending on slab allocator.For now,
page_mapcount_reset()is called becausepage->_mapcountandslab->unitshave same offset. But this is not necessary for struct slab. Use unused field for units instead.Signed-off-by Hyeonggon Yoo <[email protected]>
Will add to the series, thanks!
Quoted 32 lines
--- mm/slab.h | 4 ++-- mm/slob.c | 1 - 2 files changed, 2 insertions(+), 3 deletions(-)mm/slab.h +2
-2@@ -50,8 +50,8 @@ struct slab {struct list_head slab_list;void * __unused_1;void *freelist;/* first free block */ -void * __unused_2; -int units; +long units; +unsigned int __unused_2;#else #error "Unexpected slab allocator configured"mm/slob.c +0
-1@@ -404,7 +404,6 @@ static void slob_free(void *block, int size)clear_slob_page_free (sp);spin_unlock_irqrestore (&slob_lock, flags);__ClearPageSlab (slab_page (sp)); -page_mapcount_reset (slab_page (sp));slob_free_pages (b,0 );return ; }
PATCH v2 32/33 mm/slub: Simplify struct slab slabs field definition
In reply to Vlastimil BabkaBefore commit b47291ef02b0 ("mm, slub: change percpu partial accounting from objects to pages") we had to fit two integer fields into a native word size, so we used short int on 32-bit and int on 64-bit via #ifdef. After that commit there is only one integer field, so we can simply define it as int everywhere.
--- mm/slab.h | 4 ---- 1 file changed, 4 deletions(-)
mm/slab.h +0 -4
@@ -27,11 +27,7 @@ struct slab {struct rcu_head rcu_head;struct {struct slab *next; -#ifdef CONFIG_64BIT int slabs;/* Nr of slabs left */ -#else -short int slabs; -#endif }; };struct kmem_cache *slab_cache;
Signature · 1 line
2.33.1
Re: [PATCH v2 32/33] mm/slub: Simplify struct slab slabs field definition
Quoted 6 lines · Vlastimil Babka
Before commit
b47291ef02b0("mm, slub: change percpu partial accounting from objects to pages") we had to fit two integer fields into a native word size, so we used short int on 32-bit and int on 64-bit via #ifdef. After that commit there is only one integer field, so we can simply define it as int everywhere.Signed-off-by Vlastimil Babka <[email protected]>
PATCH v2 33/33 mm/slub: Define struct slab fields for CONFIG_SLUB_CPU_PARTIAL only when enabled
In reply to Vlastimil BabkaThe fields 'next' and 'slabs' are only used when CONFIG_SLUB_CPU_PARTIAL is enabled. We can put their definition to #ifdef to prevent accidental use when disabled.
Currenlty show_slab_objects() and slabs_cpu_partial_show() contain code accessing the slabs field that's effectively dead with CONFIG_SLUB_CPU_PARTIAL=n through the wrappers slub_percpu_partial() and slub_percpu_partial_read_once(), but to prevent a compile error, we need to hide all this code behind #ifdef.
--- mm/slab.h | 2 ++ mm/slub.c | 8 ++++++-- 2 files changed, 8 insertions(+), 2 deletions(-)
mm/slab.h +2 -0
@@ -25,10 +25,12 @@ struct slab {union {struct list_head slab_list;struct rcu_head rcu_head; +#ifdef CONFIG_SLUB_CPU_PARTIAL struct {struct slab *next;int slabs;/* Nr of slabs left */ }; +#endif };struct kmem_cache *slab_cache;/* Double-word boundary */
mm/slub.c +6 -2
@@ -5256,6 +5256,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s, total += x; nodes[node] += x; +#ifdef CONFIG_SLUB_CPU_PARTIAL slab =slub_percpu_partial_read_once (c);if (slab) { node =slab_nid (slab);
@@ -5268,6 +5269,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s, total += x; nodes[node] += x; } +#endif } }
@@ -5467,9 +5469,10 @@ static ssize_t slabs_cpu_partial_show(struct kmem_cache *s, char *buf) {int objects =0 ;int slabs =0 ; -int cpu; +int cpu__maybe_unused ;int len =0 ; +#ifdef CONFIG_SLUB_CPU_PARTIAL for_each_online_cpu (cpu) {struct slab *slab;
@@ -5478,12 +5481,13 @@ static ssize_t slabs_cpu_partial_show(struct kmem_cache *s, char *buf)if (slab) slabs += slab->slabs; } +#endif /* Approximate half-full slabs, see slub_set_cpu_partial() */ objects = (slabs *oo_objects (s->oo)) /2 ; len +=sysfs_emit_at (buf, len,"%d(%d)" , objects, slabs); -#ifdef CONFIG_SMP +#if defined(CONFIG_SLUB_CPU_PARTIAL ) && defined(CONFIG_SMP )for_each_online_cpu (cpu) {struct slab *slab;
Signature · 1 line
2.33.1
slab tree for next
In reply to Vlastimil BabkaQuoted 11 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2The plan: as my SLUB
PREEMPT_RTseries in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.
Quoted 4 lines
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/
Hi Stephen,
Please include a new tree in linux-next:
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git slab-next
i.e.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-next
Which now is identical slab-struct_slab-v2r2 branch [1]
When I tried to merge this to next-20211201, there were minor conflicts with two patches from motm:
zsmalloc-move-huge-compressed-obj-from-page-to-zspage.patch mm-memcg-relocate-mod_objcg_mlstate-get_obj_stock-and-put_obj_stock.patch
Both appear to be just a change in context.
Thanks,
Vlastimil
[1] https://lore.kernel.org/all/[email protected]/
Quoted 124 lines
Changes from v1/RFC:
https://lore.kernel.org/all/[email protected]/
- Addedvirt_to_folio()andfolio_address()in the new Patch 1.
- Addressed feedback from Andrey Konovalov and Matthew Wilcox (Thanks!)
- Added Tested-by: Marco Elver for the KFENCE parts (Thanks!)Previous version from Matthew Wilcox:
https://lore.kernel.org/all/[email protected]/LWN coverage of the above:
https://lwn.net/Articles/871982/This is originally an offshoot of the folio work by Matthew. One of the more complex parts of the struct page definition are the parts used by the slab allocators. It would be good for the MM in general if struct slab were its own data type, and it also helps to prevent tail pages from slipping in anywhere. As Matthew requested in his proof of concept series, I have taken over the development of this series, so it's a mix of patches from him (often modified by me) and my own.
One big difference is the use of coccinelle to perform the relatively trivial parts of the conversions automatically and at once, instead of a larger number of smaller incremental reviewable steps. Thanks to Julia Lawall and Luis Chamberlain for all their help!
Another notable difference is (based also on review feedback) I don't represent with a struct slab the large kmalloc allocations which are not really a slab, but use page allocator directly. When going from an object address to a struct slab, the code tests first folio slab flag, and only if it's set it converts to struct slab. This makes the struct slab type stronger.
Finally, although Matthew's version didn't use any of the folio work, the initial support has been merged meanwhile so my version builds on top of it where appropriate. This eliminates some of the redundant
compound_head()being performed e.g. when testing the slab flag.To sum up, after this series, struct page fields used by slab allocators are moved from struct page to a new struct slab, that uses the same physical storage. The availability of the fields is further distinguished by the selected slab allocator implementation. The advantages include:
- Similar to folios, if the slab is of order > 0, struct slab always is guaranteed to be the head page. Additionally it's guaranteed to be an actual slab page, not a large kmalloc. This removes uncertainty and potential for bugs.
- It's not possible to accidentally use fields of the slab implementation that's not configured.
- Other subsystems cannot use slab's fields in struct page anymore (some existing non-slab usages had to be adjusted in this series), so slab implementations have more freedom in rearranging them in the struct slab.Matthew Wilcox (Oracle) (16):
mm: Split slab into its own type
mm: Addaccount_slab()andunaccount_slab()
mm: Convertvirt_to_cache()to use struct slab
mm: Convert__ksize()to struct slab
mm: Use struct slab inkmem_obj_info()
mm: Convertcheck_heap_object()to use struct slab mm/slub: Convertdetached_freelistto use a struct slab mm/slub: Convertkfree()to use a struct slab mm/slub: Convertprint_page_info()toprint_slab_info()mm/slub: Convertpfmemalloc_match()to take a struct slab mm/slob: Convert SLOB to use struct slab mm/kasan: Convert to struct folio and struct slab
zsmalloc: Stop using slab fields in struct page
bootmem: Usepage->indexinstead ofpage->freelist
iommu: Useput_pages_list
mm: Remove slab from struct pageVlastimil Babka (17):
mm: addvirt_to_folio()andfolio_address()
mm/slab: Dissolveslab_map_pages()in its caller
mm/slub: Makeobject_err()static
mm/slub: Convert__slab_lock()and__slab_unlock()to struct slab mm/slub: Convertalloc_slab_page()to return a struct slab mm/slub: Convert__free_slab()to use struct slab mm/slub: Convert most struct page to struct slab by spatch mm/slub: Finish struct page to struct slab conversion mm/slab: Convertkmem_getpages()andkmem_freepages()to struct slab mm/slab: Convert most struct page to struct slab by spatch mm/slab: Finish struct page to struct slab conversion
mm: Convert struct page to struct slab in functions used by other
subsystems
mm/memcg: Convert slab objcgs from struct page to struct slab mm/kfence: Convertkfence_guarded_alloc()to struct slabmm/sl*b: Differentiate struct slab fields by sl*b implementations mm/slub: Simplify struct slab slabs field definition mm/slub: Define struct slab fields forCONFIG_SLUB_CPU_PARTIALonly
when enabledarch/x86/mm/init_64.c | 2 +- drivers/iommu/amd/io_pgtable.c | 59 +- drivers/iommu/dma-iommu.c | 11 +- drivers/iommu/intel/iommu.c | 89 +-- include/linux/bootmem_info.h | 2 +- include/linux/iommu.h | 3 +- include/linux/kasan.h | 9 +- include/linux/memcontrol.h | 48 -- include/linux/mm.h | 12 + include/linux/mm_types.h | 38 +- include/linux/page-flags.h | 37 - include/linux/slab.h | 8 - include/linux/slab_def.h | 16 +- include/linux/slub_def.h | 29 +- mm/bootmem_info.c | 7 +- mm/kasan/common.c | 27 +- mm/kasan/generic.c | 8 +- mm/kasan/kasan.h | 1 + mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 13 +- mm/kasan/report_tags.c | 10 +- mm/kfence/core.c | 17 +- mm/kfence/kfence_test.c | 6 +- mm/memcontrol.c | 43 +- mm/slab.c | 455 ++++++------- mm/slab.h | 322 ++++++++- mm/slab_common.c | 8 +- mm/slob.c | 46 +- mm/slub.c | 1164 ++++++++++++++++---------------- mm/sparse.c | 2 +- mm/usercopy.c | 13 +- mm/zsmalloc.c | 18 +- 32 files changed, 1317 insertions(+), 1208 deletions(-)
Re: slab tree for next
Quoted 12 lines · Vlastimil Babka
Quoted 11 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2The plan: as my SLUB
PREEMPT_RTseries in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.
Quoted 8 lines
Quoted 4 lines
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/Hi Stephen,
Please include a new tree in linux-next:
OK, not yet, please:
https://lore.kernel.org/all/[email protected]/
I will reorder the patches so that changes needed in other subsystems and the dependent actual removal of slab fields from struct page will be ordered last, and only ask for the first, mm/ contained part to be in -next. The patches for other subsystems can then be reviewed and picked by their maintainers independently, while slab implementations are already using struct slab.
Quoted 144 lines
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.gitslab-nexti.e.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-nextWhich now is identical slab-struct_slab-v2r2 branch [1]
When I tried to merge this to next-20211201, there were minor conflicts with two patches from motm:
zsmalloc-move-huge-compressed-obj-from-page-to-zspage.patch mm-memcg-relocate-mod_objcg_mlstate-get_obj_stock-and-put_obj_stock.patchBoth appear to be just a change in context.
Thanks,
Vlastimil[1] https://lore.kernel.org/all/[email protected]/
Quoted 124 lines
Changes from v1/RFC:
https://lore.kernel.org/all/[email protected]/
- Addedvirt_to_folio()andfolio_address()in the new Patch 1.
- Addressed feedback from Andrey Konovalov and Matthew Wilcox (Thanks!)
- Added Tested-by: Marco Elver for the KFENCE parts (Thanks!)Previous version from Matthew Wilcox:
https://lore.kernel.org/all/[email protected]/LWN coverage of the above:
https://lwn.net/Articles/871982/This is originally an offshoot of the folio work by Matthew. One of the more complex parts of the struct page definition are the parts used by the slab allocators. It would be good for the MM in general if struct slab were its own data type, and it also helps to prevent tail pages from slipping in anywhere. As Matthew requested in his proof of concept series, I have taken over the development of this series, so it's a mix of patches from him (often modified by me) and my own.
One big difference is the use of coccinelle to perform the relatively trivial parts of the conversions automatically and at once, instead of a larger number of smaller incremental reviewable steps. Thanks to Julia Lawall and Luis Chamberlain for all their help!
Another notable difference is (based also on review feedback) I don't represent with a struct slab the large kmalloc allocations which are not really a slab, but use page allocator directly. When going from an object address to a struct slab, the code tests first folio slab flag, and only if it's set it converts to struct slab. This makes the struct slab type stronger.
Finally, although Matthew's version didn't use any of the folio work, the initial support has been merged meanwhile so my version builds on top of it where appropriate. This eliminates some of the redundant
compound_head()being performed e.g. when testing the slab flag.To sum up, after this series, struct page fields used by slab allocators are moved from struct page to a new struct slab, that uses the same physical storage. The availability of the fields is further distinguished by the selected slab allocator implementation. The advantages include:
- Similar to folios, if the slab is of order > 0, struct slab always is guaranteed to be the head page. Additionally it's guaranteed to be an actual slab page, not a large kmalloc. This removes uncertainty and potential for bugs.
- It's not possible to accidentally use fields of the slab implementation that's not configured.
- Other subsystems cannot use slab's fields in struct page anymore (some existing non-slab usages had to be adjusted in this series), so slab implementations have more freedom in rearranging them in the struct slab.Matthew Wilcox (Oracle) (16):
mm: Split slab into its own type
mm: Addaccount_slab()andunaccount_slab()
mm: Convertvirt_to_cache()to use struct slab
mm: Convert__ksize()to struct slab
mm: Use struct slab inkmem_obj_info()
mm: Convertcheck_heap_object()to use struct slab mm/slub: Convertdetached_freelistto use a struct slab mm/slub: Convertkfree()to use a struct slab mm/slub: Convertprint_page_info()toprint_slab_info()mm/slub: Convertpfmemalloc_match()to take a struct slab mm/slob: Convert SLOB to use struct slab mm/kasan: Convert to struct folio and struct slab
zsmalloc: Stop using slab fields in struct page
bootmem: Usepage->indexinstead ofpage->freelist
iommu: Useput_pages_list
mm: Remove slab from struct pageVlastimil Babka (17):
mm: addvirt_to_folio()andfolio_address()
mm/slab: Dissolveslab_map_pages()in its caller
mm/slub: Makeobject_err()static
mm/slub: Convert__slab_lock()and__slab_unlock()to struct slab mm/slub: Convertalloc_slab_page()to return a struct slab mm/slub: Convert__free_slab()to use struct slab mm/slub: Convert most struct page to struct slab by spatch mm/slub: Finish struct page to struct slab conversion mm/slab: Convertkmem_getpages()andkmem_freepages()to struct slab mm/slab: Convert most struct page to struct slab by spatch mm/slab: Finish struct page to struct slab conversion
mm: Convert struct page to struct slab in functions used by other
subsystems
mm/memcg: Convert slab objcgs from struct page to struct slab mm/kfence: Convertkfence_guarded_alloc()to struct slabmm/sl*b: Differentiate struct slab fields by sl*b implementations mm/slub: Simplify struct slab slabs field definition mm/slub: Define struct slab fields forCONFIG_SLUB_CPU_PARTIALonly
when enabledarch/x86/mm/init_64.c | 2 +- drivers/iommu/amd/io_pgtable.c | 59 +- drivers/iommu/dma-iommu.c | 11 +- drivers/iommu/intel/iommu.c | 89 +-- include/linux/bootmem_info.h | 2 +- include/linux/iommu.h | 3 +- include/linux/kasan.h | 9 +- include/linux/memcontrol.h | 48 -- include/linux/mm.h | 12 + include/linux/mm_types.h | 38 +- include/linux/page-flags.h | 37 - include/linux/slab.h | 8 - include/linux/slab_def.h | 16 +- include/linux/slub_def.h | 29 +- mm/bootmem_info.c | 7 +- mm/kasan/common.c | 27 +- mm/kasan/generic.c | 8 +- mm/kasan/kasan.h | 1 + mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 13 +- mm/kasan/report_tags.c | 10 +- mm/kfence/core.c | 17 +- mm/kfence/kfence_test.c | 6 +- mm/memcontrol.c | 43 +- mm/slab.c | 455 ++++++------- mm/slab.h | 322 ++++++++- mm/slab_common.c | 8 +- mm/slob.c | 46 +- mm/slub.c | 1164 ++++++++++++++++---------------- mm/sparse.c | 2 +- mm/usercopy.c | 13 +- mm/zsmalloc.c | 18 +- 32 files changed, 1317 insertions(+), 1208 deletions(-)
Re: slab tree for next
Quoted 25 lines · Vlastimil Babka
Quoted 12 lines · Vlastimil Babka
Quoted 11 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2The plan: as my SLUB
PREEMPT_RTseries in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.Quoted 8 lines
Quoted 4 lines
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/Hi Stephen,
Please include a new tree in linux-next:
OK, not yet, please:
https://lore.kernel.org/all/[email protected]/
Quoted 6 lines
I will reorder the patches so that changes needed in other subsystems and the dependent actual removal of slab fields from struct page will be ordered last, and only ask for the first,
mm/contained part to be in -next. The patches for other subsystems can then be reviewed and picked by their maintainers independently, while slab implementations are already using struct slab.
OK, I've did that and removed the iommu commit, and the dependent removal of slab-specific struct page fields, until that is resolved. slab implementatiosn is nevertheless converted to struct slab, and zsmalloc and bootmem stop using the slab's struct page fields, so the struct page cleanup can be finished any time later when iommu is dealt with.
Quoted 2 lines
Quoted 2 lines
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.gitslab-next
So the 'slab-next' branch is now ready for linux-next, now identical to
branch slab-struct_slab-v3r1, head d395d823b3ae.
Quoted 144 lines
Quoted 142 lines
i.e.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-nextWhich now is identical slab-struct_slab-v2r2 branch [1]
When I tried to merge this to next-20211201, there were minor conflicts with two patches from motm:
zsmalloc-move-huge-compressed-obj-from-page-to-zspage.patch mm-memcg-relocate-mod_objcg_mlstate-get_obj_stock-and-put_obj_stock.patchBoth appear to be just a change in context.
Thanks,
Vlastimil[1] https://lore.kernel.org/all/[email protected]/
Quoted 124 lines
Changes from v1/RFC:
https://lore.kernel.org/all/[email protected]/
- Addedvirt_to_folio()andfolio_address()in the new Patch 1.
- Addressed feedback from Andrey Konovalov and Matthew Wilcox (Thanks!)
- Added Tested-by: Marco Elver for the KFENCE parts (Thanks!)Previous version from Matthew Wilcox:
https://lore.kernel.org/all/[email protected]/LWN coverage of the above:
https://lwn.net/Articles/871982/This is originally an offshoot of the folio work by Matthew. One of the more complex parts of the struct page definition are the parts used by the slab allocators. It would be good for the MM in general if struct slab were its own data type, and it also helps to prevent tail pages from slipping in anywhere. As Matthew requested in his proof of concept series, I have taken over the development of this series, so it's a mix of patches from him (often modified by me) and my own.
One big difference is the use of coccinelle to perform the relatively trivial parts of the conversions automatically and at once, instead of a larger number of smaller incremental reviewable steps. Thanks to Julia Lawall and Luis Chamberlain for all their help!
Another notable difference is (based also on review feedback) I don't represent with a struct slab the large kmalloc allocations which are not really a slab, but use page allocator directly. When going from an object address to a struct slab, the code tests first folio slab flag, and only if it's set it converts to struct slab. This makes the struct slab type stronger.
Finally, although Matthew's version didn't use any of the folio work, the initial support has been merged meanwhile so my version builds on top of it where appropriate. This eliminates some of the redundant
compound_head()being performed e.g. when testing the slab flag.To sum up, after this series, struct page fields used by slab allocators are moved from struct page to a new struct slab, that uses the same physical storage. The availability of the fields is further distinguished by the selected slab allocator implementation. The advantages include:
- Similar to folios, if the slab is of order > 0, struct slab always is guaranteed to be the head page. Additionally it's guaranteed to be an actual slab page, not a large kmalloc. This removes uncertainty and potential for bugs.
- It's not possible to accidentally use fields of the slab implementation that's not configured.
- Other subsystems cannot use slab's fields in struct page anymore (some existing non-slab usages had to be adjusted in this series), so slab implementations have more freedom in rearranging them in the struct slab.Matthew Wilcox (Oracle) (16):
mm: Split slab into its own type
mm: Addaccount_slab()andunaccount_slab()
mm: Convertvirt_to_cache()to use struct slab
mm: Convert__ksize()to struct slab
mm: Use struct slab inkmem_obj_info()
mm: Convertcheck_heap_object()to use struct slab mm/slub: Convertdetached_freelistto use a struct slab mm/slub: Convertkfree()to use a struct slab mm/slub: Convertprint_page_info()toprint_slab_info()mm/slub: Convertpfmemalloc_match()to take a struct slab mm/slob: Convert SLOB to use struct slab mm/kasan: Convert to struct folio and struct slab
zsmalloc: Stop using slab fields in struct page
bootmem: Usepage->indexinstead ofpage->freelist
iommu: Useput_pages_list
mm: Remove slab from struct pageVlastimil Babka (17):
mm: addvirt_to_folio()andfolio_address()
mm/slab: Dissolveslab_map_pages()in its caller
mm/slub: Makeobject_err()static
mm/slub: Convert__slab_lock()and__slab_unlock()to struct slab mm/slub: Convertalloc_slab_page()to return a struct slab mm/slub: Convert__free_slab()to use struct slab mm/slub: Convert most struct page to struct slab by spatch mm/slub: Finish struct page to struct slab conversion mm/slab: Convertkmem_getpages()andkmem_freepages()to struct slab mm/slab: Convert most struct page to struct slab by spatch mm/slab: Finish struct page to struct slab conversion
mm: Convert struct page to struct slab in functions used by other
subsystems
mm/memcg: Convert slab objcgs from struct page to struct slab mm/kfence: Convertkfence_guarded_alloc()to struct slabmm/sl*b: Differentiate struct slab fields by sl*b implementations mm/slub: Simplify struct slab slabs field definition mm/slub: Define struct slab fields forCONFIG_SLUB_CPU_PARTIALonly
when enabledarch/x86/mm/init_64.c | 2 +- drivers/iommu/amd/io_pgtable.c | 59 +- drivers/iommu/dma-iommu.c | 11 +- drivers/iommu/intel/iommu.c | 89 +-- include/linux/bootmem_info.h | 2 +- include/linux/iommu.h | 3 +- include/linux/kasan.h | 9 +- include/linux/memcontrol.h | 48 -- include/linux/mm.h | 12 + include/linux/mm_types.h | 38 +- include/linux/page-flags.h | 37 - include/linux/slab.h | 8 - include/linux/slab_def.h | 16 +- include/linux/slub_def.h | 29 +- mm/bootmem_info.c | 7 +- mm/kasan/common.c | 27 +- mm/kasan/generic.c | 8 +- mm/kasan/kasan.h | 1 + mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 13 +- mm/kasan/report_tags.c | 10 +- mm/kfence/core.c | 17 +- mm/kfence/kfence_test.c | 6 +- mm/memcontrol.c | 43 +- mm/slab.c | 455 ++++++------- mm/slab.h | 322 ++++++++- mm/slab_common.c | 8 +- mm/slob.c | 46 +- mm/slub.c | 1164 ++++++++++++++++---------------- mm/sparse.c | 2 +- mm/usercopy.c | 13 +- mm/zsmalloc.c | 18 +- 32 files changed, 1317 insertions(+), 1208 deletions(-)
Re: slab tree for next
Hi Vlastimil,
Quoted 41 lines · On Thu, 2 Dec 2021 17:36:45 +0100 Vlastimil Babka <[email protected]> wrote:
Quoted 25 lines · Vlastimil Babka
Quoted 12 lines · Vlastimil Babka
Quoted 11 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2The plan: as my SLUB
PREEMPT_RTseries in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.Quoted 8 lines
Quoted 4 lines
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/Hi Stephen,
Please include a new tree in linux-next:
OK, not yet, please:
https://lore.kernel.org/all/[email protected]/Quoted 6 lines
I will reorder the patches so that changes needed in other subsystems and the dependent actual removal of slab fields from struct page will be ordered last, and only ask for the first,
mm/contained part to be in -next. The patches for other subsystems can then be reviewed and picked by their maintainers independently, while slab implementations are already using struct slab.OK, I've did that and removed the iommu commit, and the dependent removal of slab-specific struct page fields, until that is resolved. slab implementatiosn is nevertheless converted to struct slab, and zsmalloc and bootmem stop using the slab's struct page fields, so the struct page cleanup can be finished any time later when iommu is dealt with.
Quoted 1 line
Quoted 72 lines
Quoted 1 line
Quoted 1 line
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.gitslab-nextSo the 'slab-next' branch is now ready for linux-next, now identical to
branch slab-struct_slab-v3r1, headd395d823b3ae.Quoted 67 lines
Quoted 67 lines
i.e.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-nextWhich now is identical slab-struct_slab-v2r2 branch [1]
When I tried to merge this to next-20211201, there were minor conflicts with two patches from motm:
zsmalloc-move-huge-compressed-obj-from-page-to-zspage.patch mm-memcg-relocate-mod_objcg_mlstate-get_obj_stock-and-put_obj_stock.patchBoth appear to be just a change in context.
Thanks,
Vlastimil[1] https://lore.kernel.org/all/[email protected]/
Quoted 50 lines
Changes from v1/RFC:
https://lore.kernel.org/all/[email protected]/
- Addedvirt_to_folio()andfolio_address()in the new Patch 1.
- Addressed feedback from Andrey Konovalov and Matthew Wilcox (Thanks!)
- Added Tested-by: Marco Elver for the KFENCE parts (Thanks!)Previous version from Matthew Wilcox:
https://lore.kernel.org/all/[email protected]/LWN coverage of the above:
https://lwn.net/Articles/871982/This is originally an offshoot of the folio work by Matthew. One of the more complex parts of the struct page definition are the parts used by the slab allocators. It would be good for the MM in general if struct slab were its own data type, and it also helps to prevent tail pages from slipping in anywhere. As Matthew requested in his proof of concept series, I have taken over the development of this series, so it's a mix of patches from him (often modified by me) and my own.
One big difference is the use of coccinelle to perform the relatively trivial parts of the conversions automatically and at once, instead of a larger number of smaller incremental reviewable steps. Thanks to Julia Lawall and Luis Chamberlain for all their help!
Another notable difference is (based also on review feedback) I don't represent with a struct slab the large kmalloc allocations which are not really a slab, but use page allocator directly. When going from an object address to a struct slab, the code tests first folio slab flag, and only if it's set it converts to struct slab. This makes the struct slab type stronger.
Finally, although Matthew's version didn't use any of the folio work, the initial support has been merged meanwhile so my version builds on top of it where appropriate. This eliminates some of the redundant
compound_head()being performed e.g. when testing the slab flag.To sum up, after this series, struct page fields used by slab allocators are moved from struct page to a new struct slab, that uses the same physical storage. The availability of the fields is further distinguished by the selected slab allocator implementation. The advantages include:
- Similar to folios, if the slab is of order > 0, struct slab always is guaranteed to be the head page. Additionally it's guaranteed to be an actual slab page, not a large kmalloc. This removes uncertainty and potential for bugs.
- It's not possible to accidentally use fields of the slab implementation that's not configured.
- Other subsystems cannot use slab's fields in struct page anymore (some existing non-slab usages had to be adjusted in this series), so slab implementations have more freedom in rearranging them in the struct slab.
Added from today. It would be good to see some Reviewed-by/Tested-by tags in these commits ...
Thanks for adding your subsystem tree as a participant of linux-next. As you may know, this is not a judgement of your code. The purpose of linux-next is for integration testing and to lower the impact of conflicts between subsystems in the next merge window.
You will need to ensure that the patches/commits in your tree/series have been:
* submitted under GPL v2 (or later) and include the Contributor's Signed-off-by,
* posted to the relevant mailing list,
* reviewed by you (or another maintainer of your subsystem tree),
* successfully unit tested, and
* destined for the current or next Linux merge window.
Basically, this should be just what you would send to Linus (or ask him to fetch). It is allowed to be rebased if you deem it necessary.
Signature · 3 lines
Cheers, Stephen Rothwell [email protected]
Re: slab tree for next
In reply to Vlastimil BabkaQuoted 17 lines · Vlastimil Babka
Quoted 11 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2The plan: as my SLUB
PREEMPT_RTseries in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.Quoted 4 lines
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/
Quoted 8 lines
Hi Stephen,
Please include a new tree in linux-next:
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.gitslab-nexti.e.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-next
Hi Stephen,
can you please replace the slab for-next above with the new tree I made to more clearly separate slab maintenance from my personal branches:
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git branch for-next
Thanks,
Vlastimil
Re: slab tree for next
Hi Vlastimil,
Quoted 5 lines · On Tue, 4 Jan 2022 01:21:03 +0100 Vlastimil Babka <[email protected]> wrote:
can you please replace the slab for-next above with the new tree I made to more clearly separate slab maintenance from my personal branches:
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.gitbranch for-next
I have done that today.
Signature · 2 lines
Cheers, Stephen Rothwell
Re: slab tree for next
In reply to Vlastimil BabkaQuoted 27 lines · Vlastimil Babka
Quoted 17 lines · Vlastimil Babka
Quoted 11 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2The plan: as my SLUB
PREEMPT_RTseries in 5.15, I would prefer to go again with the git pull request way of eventually merging this, as it's also not a small series. I will thus reply to this mail with asking to include my branch in linux-next.Quoted 4 lines
As stated in the v1/RFC cover letter, I wouldn't mind to then continue with maintaining a git tree for all slab patches in general. It was apparently already done that way before, by Pekka:
https://lore.kernel.org/linux-mm/alpine.DEB.2.00.1107221108190.2996@tiger/Quoted 8 lines
Hi Stephen,
Please include a new tree in linux-next:
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.gitslab-nexti.e.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-next
Quoted 6 lines
Hi Stephen,
can you please replace the slab for-next above with the new tree I made to more clearly separate slab maintenance from my personal branches:
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.gitbranch for-next
Hi Stephen,
can you please start using a branch named "slab/for-next" instead of "for-next" as I've been consolidating the naming scheme locally.
Feel free to do that switch at your own convenience, i.e. after the merge window. I don't plan to push anything there until rc1 and will keep both old and new name available and in sync until you confirm the switch.
Thanks,
Vlastimil
Quoted 2 lines
Thanks,
Vlastimil
Re: slab tree for next
Hi Vlastimil,
Quoted 7 lines · On Tue, 29 Aug 2023 11:55:12 +0200 Vlastimil Babka <[email protected]> wrote:
can you please start using a branch named "slab/for-next" instead of "for-next" as I've been consolidating the naming scheme locally.
Feel free to do that switch at your own convenience, i.e. after the merge window. I don't plan to push anything there until rc1 and will keep both old and new name available and in sync until you confirm the switch.
Done.
Signature · 2 lines
Cheers, Stephen Rothwell
Re: slab tree for next
Quoted 4 lines · Stephen Rothwell
Hi Vlastimil,
Quoted 1 line · On Tue, 29 Aug 2023 11:55:12 +0200 Vlastimil Babka <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
can you please start using a branch named "slab/for-next" instead of "for-next" as I've been consolidating the naming scheme locally.
Feel free to do that switch at your own convenience, i.e. after the merge window. I don't plan to push anything there until rc1 and will keep both old and new name available and in sync until you confirm the switch.
Done.
Hi Stephen,
following the current discussions I would like to have a separate branch added for next's pending-fixes, which is "slab/for-next-fixes" [1]
Thanks in advance,
Vlastimil
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-next-fixes
Re: slab tree for next
Hi Vlastimil,
Quoted 6 lines · On Tue, 5 Nov 2024 17:33:04 +0100 Vlastimil Babka <[email protected]> wrote:
following the current discussions I would like to have a separate branch added for next's pending-fixes, which is "slab/for-next-fixes" [1]
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-next-fixes
Added from today.
Thanks for adding your subsystem tree as a participant of linux-next. As you may know, this is not a judgement of your code. The purpose of linux-next is for integration testing and to lower the impact of conflicts between subsystems in the next merge window.
You will need to ensure that the patches/commits in your tree/series have been:
* submitted under GPL v2 (or later) and include the Contributor's Signed-off-by,
* posted to the relevant mailing list,
* reviewed by you (or another maintainer of your subsystem tree),
* successfully unit tested, and
* destined for the current or next Linux merge window.
Basically, this should be just what you would send to Linus (or ask him to fetch). It is allowed to be rebased if you deem it necessary.
Signature · 3 lines
Cheers, Stephen Rothwell [email protected]
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
I have pushed a v3, but not going to resent immediately to avoid unnecessary spamming, the differences is just that some patches are removed and other reordered, so the current v2 posting should be still sufficient for on-list review:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v3r1
patch 29/33 iommu: Use put_pages_list
- removed as this version is broken and Robin Murphy has meanwhile incorporated it partially to his series:
https://lore.kernel.org/lkml/[email protected]/
patch 30/33 mm: Remove slab from struct page
- removed and postponed for later as this can be only be applied after the iommu use of page.freelist is resolved
patch 27/33 zsmalloc: Stop using slab fields in struct page
patch 28/33 bootmem: Use page->index instead of page->freelist
- moved towards the end of series, to further separate the part that adjusts non-slab users of slab fields towards removing those fields from struct page.
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
1: 10b656f9eb1e = 1: 10b656f9eb1e mm: add virt_to_folio() and folio_address()
2: 5e6ad846acf1 = 2: 5e6ad846acf1 mm/slab: Dissolve slab_map_pages() in its caller
3: 48d4e9407aa0 = 3: 48d4e9407aa0 mm/slub: Make object_err() static
4: fe1e19081321 = 4: fe1e19081321 mm: Split slab into its own type
5: af7fd46fbb9b = 5: af7fd46fbb9b mm: Add account_slab() and unaccount_slab()
6: 7ed088d601d9 = 6: 7ed088d601d9 mm: Convert virt_to_cache() to use struct slab
7: 1d41188b9401 = 7: 1d41188b9401 mm: Convert __ksize() to struct slab
8: 5d9d1231461f ! 8: 8fd22e0b086e mm: Use struct slab in kmem_obj_info()
@@ Commit message
slab type instead of the page type, we make it obvious that this can
only be called for slabs.
+ [ [email protected]: also convert the related kmem_valid_obj() to folios ]
+
Signed-off-by: Matthew Wilcox (Oracle) <[email protected]>
Signed-off-by: Vlastimil Babka <[email protected]>
@@ mm/slab.h: struct kmem_obj_info {
#endif /* MM_SLAB_H */
## mm/slab_common.c ##
+@@ mm/slab_common.c: bool slab_is_available(void)
+ */
+ bool kmem_valid_obj(void *object)
+ {
+- struct page *page;
++ struct folio *folio;
+
+ /* Some arches consider ZERO_SIZE_PTR to be a valid address. */
+ if (object < (void *)PAGE_SIZE || !virt_addr_valid(object))
+ return false;
+- page = virt_to_head_page(object);
+- return PageSlab(page);
++ folio = virt_to_folio(object);
++ return folio_test_slab(folio);
+ }
+ EXPORT_SYMBOL_GPL(kmem_valid_obj);
+
@@ mm/slab_common.c: void kmem_dump_obj(void *object)
{
char *cp = IS_ENABLED(CONFIG_MMU) ? "" : "/vmalloc";
@@ mm/slub.c: int __kmem_cache_shutdown(struct kmem_cache *s)
objp = base + s->size * objnr;
kpp->kp_objp = objp;
- if (WARN_ON_ONCE(objp < base || objp >= base + page->objects * s->size || (objp - base) % s->size) ||
-+ if (WARN_ON_ONCE(objp < base || objp >= base + slab->objects * s->size || (objp - base) % s->size) ||
++ if (WARN_ON_ONCE(objp < base || objp >= base + slab->objects * s->size
++ || (objp - base) % s->size) ||
!(s->flags & SLAB_STORE_USER))
return;
#ifdef CONFIG_SLUB_DEBUG
9: 3aef771be335 ! 9: c97e73c3b6c2 mm: Convert check_heap_object() to use struct slab
@@ mm/slab.h: struct kmem_obj_info {
+#else
+static inline
+void __check_heap_object(const void *ptr, unsigned long n,
-+ const struct slab *slab, bool to_user) { }
++ const struct slab *slab, bool to_user)
++{
++}
+#endif
+
#endif /* MM_SLAB_H */
10: 2253e45e6bef = 10: da05e0f7179c mm/slub: Convert detached_freelist to use a struct slab
11: f28202bc27ba = 11: 383887e77104 mm/slub: Convert kfree() to use a struct slab
12: 31b58b1e914f = 12: c46be093c637 mm/slub: Convert __slab_lock() and __slab_unlock() to struct slab
13: 636406a3ad59 = 13: 49dbbf917052 mm/slub: Convert print_page_info() to print_slab_info()
14: 3b49efda3b6f = 14: 4bb0c932156a mm/slub: Convert alloc_slab_page() to return a struct slab
15: 61a195526d3b ! 15: 4b9761b5cfab mm/slub: Convert __free_slab() to use struct slab
@@ mm/slub.c: static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int n
- __ClearPageSlabPfmemalloc(page);
- __ClearPageSlab(page);
+- /* In union with page->mapping where page allocator expects NULL */
+- page->slab_cache = NULL;
+ __slab_clear_pfmemalloc(slab);
+ __folio_clear_slab(folio);
- /* In union with page->mapping where page allocator expects NULL */
-- page->slab_cache = NULL;
-+ slab->slab_cache = NULL;
++ folio->mapping = NULL;
if (current->reclaim_state)
current->reclaim_state->reclaimed_slab += pages;
- unaccount_slab(page_slab(page), order, s);
16: 987c7ed31580 = 16: f384ec918065 mm/slub: Convert pfmemalloc_match() to take a struct slab
17: cc742564237e ! 17: 06738ade4e17 mm/slub: Convert most struct page to struct slab by spatch@@ Commit message
// Options: --include-headers --no-includes --smpl-spacing include/linux/slub_def.h mm/slub.c // Note: needs coccinelle 1.1.1 to avoid breaking whitespace, and ocaml for the
- // embedded script script
+ // embedded script
// build list of functions to exclude from applying the next rule @initialize:ocaml@
18: b45acac9aace = 18: 1a4f69a4cced mm/slub: Finish struct page to struct slab conversion
19: 76c3eeb39684 ! 19: 1d62d706e884 mm/slab: Convert kmem_getpages() and kmem_freepages() to struct slab
@@ mm/slab.c: slab_out_of_memory(struct kmem_cache *cachep, gfp_t gfpflags, int nod
- __ClearPageSlabPfmemalloc(page);
- __ClearPageSlab(page);
- page_mapcount_reset(page);
+- /* In union with page->mapping where page allocator expects NULL */
+- page->slab_cache = NULL;
+ BUG_ON(!folio_test_slab(folio));
+ __slab_clear_pfmemalloc(slab);
+ __folio_clear_slab(folio);
+ page_mapcount_reset(folio_page(folio, 0));
- /* In union with page->mapping where page allocator expects NULL */
-- page->slab_cache = NULL;
-+ slab->slab_cache = NULL;
++ folio->mapping = NULL;
if (current->reclaim_state)
current->reclaim_state->reclaimed_slab += 1 << order;20: ed6144dbebce ! 20: fd4c3aabacd3 mm/slab: Convert most struct page to struct slab by spatch
@@ Commit message
// Options: --include-headers --no-includes --smpl-spacing mm/slab.c // Note: needs coccinelle 1.1.1 to avoid breaking whitespace, and ocaml for the
- // embedded script script
+ // embedded script
// build list of functions for applying the next rule @initialize:ocaml@
21: 17fb81e601e6 = 21: b59720b2edba mm/slab: Finish struct page to struct slab conversion 22: 4e8d1faebc24 ! 22: 65ced071c3e7 mm: Convert struct page to struct slab in functions used by other subsystems
@@ Commit message
,...)
+ Reviewed-by: Andrey Konovalov <[email protected]>
Cc: Julia Lawall <[email protected]>
Cc: Luis Chamberlain <[email protected]>
Cc: Andrey Ryabinin <[email protected]>
23: eefa12e18a92 = 23: c9c8dee01e5d mm/memcg: Convert slab objcgs from struct page to struct slab
24: fa5ba4107ce2 ! 24: def731137335 mm/slob: Convert SLOB to use struct slab
@@ Metadata
Author: Matthew Wilcox (Oracle) <[email protected]>
## Commit message ##
- mm/slob: Convert SLOB to use struct slab
+ mm/slob: Convert SLOB to use struct slab and struct folio
- Use struct slab throughout the slob allocator.
+ Use struct slab throughout the slob allocator. Where non-slab page can appear
+ use struct folio instead of struct page.
[ [email protected]: don't introduce wrappers for PageSlobFree in mm/slab.h just
for the single callers being wrappers in mm/slob.c ]
+ [ Hyeonggon Yoo <[email protected]>: fix NULL pointer deference ]
+
Signed-off-by: Matthew Wilcox (Oracle) <[email protected]>
Signed-off-by: Vlastimil Babka <[email protected]>
## mm/slob.c ##
+@@
+ * If kmalloc is asked for objects of PAGE_SIZE or larger, it calls
+ * alloc_pages() directly, allocating compound pages so the page order
+ * does not have to be separately tracked.
+- * These objects are detected in kfree() because PageSlab()
++ * These objects are detected in kfree() because folio_test_slab()
+ * is false for them.
+ *
+ * SLAB is emulated on top of SLOB by simply calling constructors and
@@ mm/slob.c: static LIST_HEAD(free_slob_large);
/*
* slob_page_free: true for pages on free_slob_pages list.
@@ mm/slob.c: static void *slob_page_alloc(struct page *sp, size_t size, int align,
int align_offset)
{
- struct page *sp;
++ struct folio *folio;
+ struct slab *sp;
struct list_head *slob_list;
slob_t *b = NULL;
@@ mm/slob.c: static void *slob_alloc(size_t size, gfp_t gfp, int align, int node,
return NULL;
- sp = virt_to_page(b);
- __SetPageSlab(sp);
-+ sp = virt_to_slab(b);
-+ __SetPageSlab(slab_page(sp));
++ folio = virt_to_folio(b);
++ __folio_set_slab(folio);
++ sp = folio_slab(folio);
spin_lock_irqsave(&slob_lock, flags);
sp->units = SLOB_UNITS(PAGE_SIZE);
@@ mm/slob.c: static void slob_free(void *block, int size)
spin_unlock_irqrestore(&slob_lock, flags);
- __ClearPageSlab(sp);
- page_mapcount_reset(sp);
-+ __ClearPageSlab(slab_page(sp));
++ __folio_clear_slab(slab_folio(sp));
+ page_mapcount_reset(slab_page(sp));
slob_free_pages(b, 0);
return;
}
+@@ mm/slob.c: EXPORT_SYMBOL(__kmalloc_node_track_caller);
+
+ void kfree(const void *block)
+ {
+- struct page *sp;
++ struct folio *sp;
+
+ trace_kfree(_RET_IP_, block);
+
+@@ mm/slob.c: void kfree(const void *block)
+ return;
+ kmemleak_free(block);
+
+- sp = virt_to_page(block);
+- if (PageSlab(sp)) {
++ sp = virt_to_folio(block);
++ if (folio_test_slab(sp)) {
+ int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
+ unsigned int *m = (unsigned int *)(block - align);
+ slob_free(m, *m + align);
+ } else {
+- unsigned int order = compound_order(sp);
+- mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B,
++ unsigned int order = folio_order(sp);
++
++ mod_node_page_state(folio_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B,
+ -(PAGE_SIZE << order));
+- __free_pages(sp, order);
++ __free_pages(folio_page(sp, 0), order);
+
+ }
+ }25: aa4f573a4c96 ! 25: 466b9fb1f6e5 mm/kasan: Convert to struct folio and struct slab
@@ Commit message
+ Reviewed-by: Andrey Konovalov <[email protected]>
Cc: Andrey Ryabinin <[email protected]>
Cc: Alexander Potapenko <[email protected]>
Cc: Andrey Konovalov <[email protected]>
26: 67b7966d2fb6 = 26: b8159ae8e5cd mm/kfence: Convert kfence_guarded_alloc() to struct slab
31: d64dfe49c1e7 ! 27: 4525180926f9 mm/sl*b: Differentiate struct slab fields by sl*b implementations
@@ Commit message
possible.
This should also prevent accidental use of fields that don't exist in given
- implementation. Before this patch virt_to_cache() and and cache_from_obj() was
- visible for SLOB (albeit not used), although it relies on the slab_cache field
+ implementation. Before this patch virt_to_cache() and cache_from_obj() were
+ visible for SLOB (albeit not used), although they rely on the slab_cache field
that isn't set by SLOB. With this patch it's now a compile error, so these
functions are now hidden behind #ifndef CONFIG_SLOB.
@@ mm/kfence/core.c: static void *kfence_guarded_alloc(struct kmem_cache *cache, si
- slab->s_mem = addr;
+#if defined(CONFIG_SLUB)
+ slab->objects = 1;
-+#elif defined (CONFIG_SLAB)
++#elif defined(CONFIG_SLAB)
+ slab->s_mem = addr;
+#endif
@@ mm/slab.h
+
+#if defined(CONFIG_SLAB)
+
-+ union {
-+ struct list_head slab_list;
+ union {
+ struct list_head slab_list;
+- struct { /* Partial pages */
+ struct rcu_head rcu_head;
+ };
+ struct kmem_cache *slab_cache;
+ void *freelist; /* array of free object indexes */
-+ void * s_mem; /* first object */
++ void *s_mem; /* first object */
+ unsigned int active;
+
+#elif defined(CONFIG_SLUB)
+
- union {
- struct list_head slab_list;
-- struct { /* Partial pages */
++ union {
++ struct list_head slab_list;
+ struct rcu_head rcu_head;
+ struct {
struct slab *next;
@@ mm/slab.h: struct slab {
+#elif defined(CONFIG_SLOB)
+
+ struct list_head slab_list;
-+ void * __unused_1;
++ void *__unused_1;
+ void *freelist; /* first free block */
-+ void * __unused_2;
++ void *__unused_2;
+ int units;
+
+#else
@@ mm/slab.h: struct slab {
#ifdef CONFIG_MEMCG
unsigned long memcg_data;
@@ mm/slab.h: struct slab {
- static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl))
SLAB_MATCH(flags, __page_flags);
SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */
+ SLAB_MATCH(slab_list, slab_list);
+#ifndef CONFIG_SLOB
SLAB_MATCH(rcu_head, rcu_head);
+ SLAB_MATCH(slab_cache, slab_cache);
++#endif
++#ifdef CONFIG_SLAB
+ SLAB_MATCH(s_mem, s_mem);
+ SLAB_MATCH(active, active);
+#endif
SLAB_MATCH(_refcount, __page_refcount);
#ifdef CONFIG_MEMCG
32: 0abf87bae67e = 28: 94b78948d53f mm/slub: Simplify struct slab slabs field definition
33: 813c304f18e4 = 29: f5261e6375f0 mm/slub: Define struct slab fields for CONFIG_SLUB_CPU_PARTIAL only when enabled
27: ebce4b5b5ced ! 30: 1414e8c87de6 zsmalloc: Stop using slab fields in struct page@@ Commit message
- Cc: Minchan Kim <[email protected]>
+ Acked-by: Minchan Kim <[email protected]>
Cc: Nitin Gupta <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
28: f124425ae7de = 31: 8a3cda6b38eb bootmem: Use page->index instead of page->freelist 29: 82da48c73b2e < -: ------------ iommu: Use put_pages_list 30: 181e16dfefbb < -: ------------ mm: Remove slab from struct page -: ------------ > 32: 91e069ba116b mm/slob: Remove unnecessary page_mapcount_reset() function call
A message that is not in the archive (sent off-list or not yet received): 4c3dfdfa-2e19-a9a7-7945-3d75bc87ca05-AlSwsSmVLrQ@public.gmane.org
Quoted 10 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hello Vlastimil, Thank you for nice work.
I'm going to review and test new version soon in free time.
Btw, I gave you some review and test tags and seems to be missing in new series. Did I do review/test process wrongly? It's first time to review patches so please let me know if I did it wrongly.
Quoted 8 lines · Hyeonggon Yoo
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
Quoted 6 lines
Quoted 3 lines
Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hello Vlastimil, Thank you for nice work.
I'm going to review and test new version soon in free time.
Thanks!
Quoted 3 lines
Btw, I gave you some review and test tags and seems to be missing in new series. Did I do review/test process wrongly? It's first time to review patches so please let me know if I did it wrongly.
You did right, sorry! I didn't include them as those were for patches that I was additionally changing after your review/test and the decision what is substantial change enough to need a new test/review is often fuzzy. So if you can recheck the new versions it would be great and then I will pick that up, thanks!
Quoted 3 lines
--
Thank you.
Hyeonggon.
A message that is not in the archive (sent off-list or not yet received): 87584294-b1bc-aabe-d86a-1a8b93a7f4d4-AlSwsSmVLrQ@public.gmane.org
Quoted 9 lines · Vlastimil Babka
Quoted 8 lines · Hyeonggon Yoo
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hello Vlastimil, Thank you for nice work.
I'm going to review and test new version soon in free time.Thanks!
You're welcome!
Quoted 7 lines
Quoted 3 lines
Btw, I gave you some review and test tags and seems to be missing in new series. Did I do review/test process wrongly? It's first time to review patches so please let me know if I did it wrongly.
You did right, sorry! I didn't include them as those were for patches that I was additionally changing after your review/test and the decision what is substantial change enough to need a new test/review is often fuzzy.
Ah, Okay. review/test becomes invalid after some changing. that's okay. I was just unfamiliar with the process. Thank you!
Quoted 2 lines
So if you can recheck the new versions it would be great and then I will pick that up, thanks!
Okay. I'll new versions.
Quoted 5 lines
Quoted 4 lines
--
Thank you.
Hyeonggon.
Roman Gushchin
Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-byQuoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hi Vlastimil!
I've started to review this patchset (btw, a really nice work, I like the resulting code way more). Because I'm looking at v3 and I don't have the whole v2 in my mailbox, here is what I've now:
* mm: add virt_to_folio() and folio_address()
* mm/slab: Dissolve slab_map_pages() in its caller
* mm/slub: Make object_err() static
* mm: Split slab into its own type
1) Shouldn't SLAB_MATCH() macro use struct folio instead of struct page for the comparison?
2) page_slab() is used only in kasan and only in one place, so maybe it's better to not introduce it as a generic helper?
Other than that
* mm: Add account_slab() and unaccount_slab()
1) maybe change the title to convert/replace instead of add?
2) maybe move later changes to memcg_alloc_page_obj_cgroups() to this patch?
* mm: Convert virt_to_cache() to use struct slab
* mm: Convert __ksize() to struct slab
It looks like certain parts of __ksize() can be merged between slab, slub and slob?
* mm: Use struct slab in kmem_obj_info()
I'll try to finish reviewing the patchset until the end of the week.
Thanks!
Roman
Roman Gushchin
Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-by Reviewed-byQuoted 40 lines · Roman Gushchin
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hi Vlastimil!
I've started to review this patchset (btw, a really nice work, I like the resulting code way more). Because I'm looking at v3 and I don't have the whole v2 in my mailbox, here is what I've now:
* mm: add
virt_to_folio()andfolio_address()Reviewed-by Roman Gushchin <[email protected]>* mm/slab: Dissolve
slab_map_pages()in its callerReviewed-by Roman Gushchin <[email protected]>* mm/slub: Make
object_err()staticReviewed-by Roman Gushchin <[email protected]>* mm: Split slab into its own type
1) Shouldn'tSLAB_MATCH()macro use struct folio instead of struct page for the comparison?
2)page_slab()is used only in kasan and only in one place, so maybe it's better to not introduce it as a generic helper?
Other than thatReviewed-by Roman Gushchin <[email protected]>* mm: Add
account_slab()andunaccount_slab()
1) maybe change the title to convert/replace instead of add?
2) maybe move later changes tomemcg_alloc_page_obj_cgroups()to this patch?Reviewed-by Roman Gushchin <[email protected]>* mm: Convert
virt_to_cache()to use struct slab
Quoted 8 lines
Reviewed-by Roman Gushchin <[email protected]>* mm: Convert
__ksize()to struct slab
It looks like certain parts of__ksize()can be merged between slab, slub and slob?Reviewed-by Roman Gushchin <[email protected]>* mm: Use struct slab in
kmem_obj_info()Reviewed-by Roman Gushchin <[email protected]>
Part 2:
* mm: Convert check_heap_object() to use struct slab
* mm/slub: Convert detached_freelist to use a struct slab
How about to convert free_nonslab_page() to free_nonslab_folio()? And maybe rename it to something like free_large_kmalloc()?
If I'm not missing something, large kmallocs is the only way how we can end up there with a !slab folio/page.
* mm/slub: Convert kfree() to use a struct slab
* mm/slub: Convert __slab_lock() and __slab_unlock() to struct slab
* mm/slub: Convert print_page_info() to print_slab_info()
Do we really need to explicitly convert slab_folio()'s result to (struct folio *)?
* mm/slub: Convert alloc_slab_page() to return a struct slab
* mm/slub: Convert __free_slab() to use struct slab
* mm/slub: Convert pfmemalloc_match() to take a struct slab Cool! Removing pfmemalloc_unsafe() is really nice.
* mm/slub: Convert most struct page to struct slab by spatch
* mm/slub: Finish struct page to struct slab conversion
* mm/slab: Convert kmem_getpages() and kmem_freepages() to struct slab
* mm/slab: Convert most struct page to struct slab by spatch
Another patch with the same title? Rebase error?
* mm/slab: Finish struct page to struct slab conversion
And this one too?
Thanks!
Roman
Quoted 9 lines · Roman Gushchin
Quoted 8 lines · Roman Gushchin
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hi Vlastimil!
I've started to review this patchset (btw, a really nice work, I like the resulting code way more). Because I'm looking at v3 and I don't have
Thanks a lot, Roman!
...
Quoted 8 lines
* mm/slab: Convert most struct page to struct slab by spatch
Another patch with the same title? Rebase error?
* mm/slab: Finish struct page to struct slab conversion
And this one too?
No, these are for mm/slab.c, the previous were for mm/slub.c :)
Quoted 4 lines
Thanks!
Roman
Quoted 10 lines · Roman Gushchin
Part 2:
* mm: Convert
check_heap_object()to use struct slabReviewed-by Roman Gushchin <[email protected]>* mm/slub: Convert
detached_freelistto use a struct slab
How about to convertfree_nonslab_page()tofree_nonslab_folio()? And maybe rename it to something likefree_large_kmalloc()?
If I'm not missing something, large kmallocs is the only way how we can end up there with a !slab folio/page.
Good point, thanks! But did at as part of the following patch, where it fits logically better.
Quoted 2 lines
* mm/slub: Convert
kfree()to use a struct slabReviewed-by Roman Gushchin <[email protected]>
Didn't add your tag because of the addition of free_large_kmalloc() change.
Quoted 5 lines
* mm/slub: Convert
__slab_lock()and__slab_unlock()to struct slabReviewed-by Roman Gushchin <[email protected]>* mm/slub: Convert
print_page_info()toprint_slab_info()
Do we really need to explicitly convert slab_folio()'s result to (struct folio *)?
Unfortunately yes, as long as folio_flags() don't take const struct folio *, which will need some yak shaving.
Quoted 12 lines
Reviewed-by Roman Gushchin <[email protected]>* mm/slub: Convert
alloc_slab_page()to return a struct slabReviewed-by Roman Gushchin <[email protected]>* mm/slub: Convert
__free_slab()to use struct slabReviewed-by Roman Gushchin <[email protected]>* mm/slub: Convert
pfmemalloc_match()to take a struct slab Cool! Removingpfmemalloc_unsafe()is really nice.Reviewed-by Roman Gushchin <[email protected]>
Quoted 8 lines
* mm/slub: Convert most struct page to struct slab by spatch
Reviewed-by Roman Gushchin <[email protected]>* mm/slub: Finish struct page to struct slab conversion
Reviewed-by Roman Gushchin <[email protected]>* mm/slab: Convert
kmem_getpages()andkmem_freepages()to struct slabReviewed-by Roman Gushchin <[email protected]>
Thanks again!
Quoted 12 lines
* mm/slab: Convert most struct page to struct slab by spatch
Another patch with the same title? Rebase error?
* mm/slab: Finish struct page to struct slab conversion
And this one too?
Thanks!
Roman
Quoted 5 lines · Vlastimil Babka
Quoted 2 lines
* mm/slub: Convert
print_page_info()toprint_slab_info()
Do we really need to explicitly convert slab_folio()'s result to (struct folio *)?Unfortunately yes, as long as
folio_flags()don't take const struct folio *, which will need some yak shaving.
In case anyone's interested ...
folio_flags calls VM_BUG_ON_PGFLAGS() which would need its second argument to be const.
That means dump_page() needs to take a const struct page, which means __dump_page() needs its argument to be const.
That calls ...
is_migrate_cma_page()page_mapping()page_mapcount()page_ref_count()page_to_pgoff()page_to_pfn()hpage_pincount_available()head_compound_mapcount()head_compound_pincount()compound_order()PageKsm()PageAnon()PageCompound()
... and at that point, I ran out of motivation to track down some parts of this tarbaby that could be fixed. I did do:
mm: constify page_count and page_ref_count
mm: constify get_pfnblock_flags_mask and get_pfnblock_migratetype
mm: make compound_head const-preserving
mm/page_owner: constify dump_page_owner
so some of those are already done. But a lot of them just need to be done at the same time. For example, page_mapping() calls folio_mapping() which calls folio_test_slab() which calls folio_flags(), so dump_page() and page_mapping() need to be done at the same time.
One bit that could be broken off easily (I think ...) is PageTransTail() PageTail(), PageCompound(), PageHuge(), page_to_pgoff() and page_to_index(). One wrinkle is needed a temporary TESTPAGEFLAGS_FALSE_CONST. But I haven't tried it yet.
Quoted 8 lines · Roman Gushchin
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2
Quoted 8 lines
Quoted 2 lines
Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hi Vlastimil!
I've started to review this patchset (btw, a really nice work, I like the resulting code way more). Because I'm looking at v3 and I don't have the whole v2 in my mailbox, here is what I've now:
Thanks a lot, Roman!
Quoted 4 lines
* mm: add
virt_to_folio()andfolio_address()Reviewed-by Roman Gushchin <[email protected]>* mm/slab: Dissolve
slab_map_pages()in its caller
Quoted 8 lines
Reviewed-by Roman Gushchin <[email protected]>* mm/slub: Make
object_err()staticReviewed-by Roman Gushchin <[email protected]>* mm: Split slab into its own type
1) Shouldn'tSLAB_MATCH()macro use struct folio instead of struct page for the comparison?
Folio doesn't have define most of the fields, and matching some to page and others to folio seems like unnecessary complication. Maybe as part of the final struct page cleanup when the slab fields are gone from struct page, the rest could all be in folio - I'll check once we get there.
Quoted 2 lines
2)
page_slab()is used only in kasan and only in one place, so maybe it's better to not introduce it as a generic helper?
Yeah that's the case after the series, but as part of the incremental steps, page_slab() gets used in many places. I'll consider removing it on top though.
Quoted 5 lines
Other than that
Reviewed-by Roman Gushchin <[email protected]>* mm: Add
account_slab()andunaccount_slab()
1) maybe change the title to convert/replace instead of add?
Done.
Quoted 1 line
2) maybe move later changes to
memcg_alloc_page_obj_cgroups()to this patch?
Maybe possible, but that would distort the series more than I'd like to at this rc6 time.
Quoted 18 lines
Reviewed-by Roman Gushchin <[email protected]>* mm: Convert
virt_to_cache()to use struct slabReviewed-by Roman Gushchin <[email protected]>* mm: Convert
__ksize()to struct slab
It looks like certain parts of__ksize()can be merged between slab, slub and slob?Reviewed-by Roman Gushchin <[email protected]>* mm: Use struct slab in
kmem_obj_info()Reviewed-by Roman Gushchin <[email protected]>I'll try to finish reviewing the patchset until the end of the week.
Thanks!
Roman
Hyeonggon Yoo
Reviewed-by Tested-by Reviewed-by Tested-by Reviewed-by Tested-by Reviewed-by Tested-by Reviewed-by Tested-byQuoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Reviewing the whole patch series takes longer than I thought. I'll try to review and test rest of patches when I have time.
I added Tested-by if kernel builds okay and kselftests does not break the kernel on my machine. (with CONFIG_SLAB/SLUB/SLOB depending on the patch), Let me know me if you know better way to test a patch.
# mm/slub: Define struct slab fields for CONFIG_SLUB_CPU_PARTIAL only when enabled
Comment:
Works on both SLUB_CPU_PARTIAL and !SLUB_CPU_PARTIAL. btw, do we need slabs_cpu_partial attribute when we don't use cpu partials? (!SLUB_CPU_PARTIAL)
# mm/slub: Simplify struct slab slabs field definition Comment:
This is how struct page looks on the top of v3r3 branch:
struct page {
[...]
struct { /* slab, slob and slub */
union {
struct list_head slab_list;
struct { /* Partial pages */
struct page *next;
#ifdef CONFIG_64BIT
int pages; /* Nr of pages left */
#else
short int pages;
#endif
};
};[...]
It's not consistent with struct slab.
I think this is because "mm: Remove slab from struct page" was dropped. Would you update some of patches?
# mm/sl*b: Differentiate struct slab fields by sl*b implementations
Works SL[AUO]B on my machine and makes code much better.
# mm/slob: Convert SLOB to use struct slab and struct folio
It still works fine on SLOB.
# mm/slab: Convert kmem_getpages() and kmem_freepages() to struct slab
# mm/slub: Convert __free_slab() to use struct slab
Thanks,
Hyeonggon.
Quoted 335 lines
1: 10b656f9eb1e = 1: 10b656f9eb1e mm: add virt_to_folio() and folio_address() 2: 5e6ad846acf1 = 2: 5e6ad846acf1 mm/slab: Dissolve slab_map_pages() in its caller 3: 48d4e9407aa0 = 3: 48d4e9407aa0 mm/slub: Make object_err() static 4: fe1e19081321 = 4: fe1e19081321 mm: Split slab into its own type 5: af7fd46fbb9b = 5: af7fd46fbb9b mm: Add account_slab() and unaccount_slab() 6: 7ed088d601d9 = 6: 7ed088d601d9 mm: Convert virt_to_cache() to use struct slab 7: 1d41188b9401 = 7: 1d41188b9401 mm: Convert __ksize() to struct slab 8: 5d9d1231461f ! 8: 8fd22e0b086e mm: Use struct slab in kmem_obj_info() @@ Commit message slab type instead of the page type, we make it obvious that this can only be called for slabs. + [ [email protected]: also convert the related kmem_valid_obj() to folios ] + Signed-off-by: Matthew Wilcox (Oracle) <[email protected]> Signed-off-by: Vlastimil Babka <[email protected]> @@ mm/slab.h: struct kmem_obj_info { #endif /* MM_SLAB_H */ ## mm/slab_common.c ## +@@ mm/slab_common.c: bool slab_is_available(void) + */ + bool kmem_valid_obj(void *object) + { +- struct page *page; ++ struct folio *folio; + + /* Some arches consider ZERO_SIZE_PTR to be a valid address. */ + if (object < (void *)PAGE_SIZE || !virt_addr_valid(object)) + return false; +- page = virt_to_head_page(object); +- return PageSlab(page); ++ folio = virt_to_folio(object); ++ return folio_test_slab(folio); + } + EXPORT_SYMBOL_GPL(kmem_valid_obj); + @@ mm/slab_common.c: void kmem_dump_obj(void *object) { char *cp = IS_ENABLED(CONFIG_MMU) ? "" : "/vmalloc"; @@ mm/slub.c: int __kmem_cache_shutdown(struct kmem_cache *s) objp = base + s->size * objnr; kpp->kp_objp = objp; - if (WARN_ON_ONCE(objp < base || objp >= base + page->objects * s->size || (objp - base) % s->size) || -+ if (WARN_ON_ONCE(objp < base || objp >= base + slab->objects * s->size || (objp - base) % s->size) || ++ if (WARN_ON_ONCE(objp < base || objp >= base + slab->objects * s->size ++ || (objp - base) % s->size) || !(s->flags & SLAB_STORE_USER)) return; #ifdef CONFIG_SLUB_DEBUG 9: 3aef771be335 ! 9: c97e73c3b6c2 mm: Convert check_heap_object() to use struct slab @@ mm/slab.h: struct kmem_obj_info { +#else +static inline +void __check_heap_object(const void *ptr, unsigned long n, -+ const struct slab *slab, bool to_user) { } ++ const struct slab *slab, bool to_user) ++{ ++} +#endif + #endif /* MM_SLAB_H */ 10: 2253e45e6bef = 10: da05e0f7179c mm/slub: Convert detached_freelist to use a struct slab 11: f28202bc27ba = 11: 383887e77104 mm/slub: Convert kfree() to use a struct slab 12: 31b58b1e914f = 12: c46be093c637 mm/slub: Convert __slab_lock() and __slab_unlock() to struct slab 13: 636406a3ad59 = 13: 49dbbf917052 mm/slub: Convert print_page_info() to print_slab_info() 14: 3b49efda3b6f = 14: 4bb0c932156a mm/slub: Convert alloc_slab_page() to return a struct slab 15: 61a195526d3b ! 15: 4b9761b5cfab mm/slub: Convert __free_slab() to use struct slab @@ mm/slub.c: static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int n - __ClearPageSlabPfmemalloc(page); - __ClearPageSlab(page); +- /* In union with page->mapping where page allocator expects NULL */ +- page->slab_cache = NULL; + __slab_clear_pfmemalloc(slab); + __folio_clear_slab(folio); - /* In union with page->mapping where page allocator expects NULL */ -- page->slab_cache = NULL; -+ slab->slab_cache = NULL; ++ folio->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - unaccount_slab(page_slab(page), order, s); 16: 987c7ed31580 = 16: f384ec918065 mm/slub: Convert pfmemalloc_match() to take a struct slab 17: cc742564237e ! 17: 06738ade4e17 mm/slub: Convert most struct page to struct slab by spatch@@ Commit message
// Options: --include-headers --no-includes --smpl-spacing
include/linux/slub_def.hmm/slub.c// Note: needs coccinelle 1.1.1 to avoid breaking whitespace, and ocaml for the
- // embedded script script
+ // embedded script// build list of functions to exclude from applying the next rule @initialize:ocaml@
18: b45acac9aace = 18: 1a4f69a4cced mm/slub: Finish struct page to struct slab conversion 19: 76c3eeb39684 ! 19: 1d62d706e884 mm/slab: Convert kmem_getpages() and kmem_freepages() to struct slab @@ mm/slab.c: slab_out_of_memory(struct kmem_cache *cachep, gfp_t gfpflags, int nod - __ClearPageSlabPfmemalloc(page); - __ClearPageSlab(page); - page_mapcount_reset(page); +- /* In union with page->mapping where page allocator expects NULL */ +- page->slab_cache = NULL; + BUG_ON(!folio_test_slab(folio)); + __slab_clear_pfmemalloc(slab); + __folio_clear_slab(folio); + page_mapcount_reset(folio_page(folio, 0)); - /* In union with page->mapping where page allocator expects NULL */ -- page->slab_cache = NULL; -+ slab->slab_cache = NULL; ++ folio->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += 1 << order;20:
ed6144dbebce! 20:fd4c3aabacd3mm/slab: Convert most struct page to struct slab by spatch
@@ Commit message// Options: --include-headers --no-includes --smpl-spacing
mm/slab.c// Note: needs coccinelle 1.1.1 to avoid breaking whitespace, and ocaml for the
- // embedded script script
+ // embedded script// build list of functions for applying the next rule @initialize:ocaml@
21: 17fb81e601e6 = 21: b59720b2edba mm/slab: Finish struct page to struct slab conversion 22: 4e8d1faebc24 ! 22: 65ced071c3e7 mm: Convert struct page to struct slab in functions used by other subsystems@@ Commit message
,...)Signed-off-by Vlastimil Babka <[email protected]>+ Reviewed-by: Andrey Konovalov <[email protected]>
Cc: Julia Lawall <[email protected]>
Cc: Luis Chamberlain <[email protected]>
Cc: Andrey Ryabinin <[email protected]>23: eefa12e18a92 = 23: c9c8dee01e5d mm/memcg: Convert slab objcgs from struct page to struct slab 24: fa5ba4107ce2 ! 24: def731137335 mm/slob: Convert SLOB to use struct slab @@ Metadata Author: Matthew Wilcox (Oracle) <[email protected]> ## Commit message ## - mm/slob: Convert SLOB to use struct slab + mm/slob: Convert SLOB to use struct slab and struct folio - Use struct slab throughout the slob allocator. + Use struct slab throughout the slob allocator. Where non-slab page can appear + use struct folio instead of struct page. [ [email protected]: don't introduce wrappers for PageSlobFree in mm/slab.h just for the single callers being wrappers in mm/slob.c ] + [ Hyeonggon Yoo <[email protected]>: fix NULL pointer deference ] + Signed-off-by: Matthew Wilcox (Oracle) <[email protected]> Signed-off-by: Vlastimil Babka <[email protected]> ## mm/slob.c ## +@@ + * If kmalloc is asked for objects of PAGE_SIZE or larger, it calls + * alloc_pages() directly, allocating compound pages so the page order + * does not have to be separately tracked. +- * These objects are detected in kfree() because PageSlab() ++ * These objects are detected in kfree() because folio_test_slab() + * is false for them. + * + * SLAB is emulated on top of SLOB by simply calling constructors and @@ mm/slob.c: static LIST_HEAD(free_slob_large); /* * slob_page_free: true for pages on free_slob_pages list. @@ mm/slob.c: static void *slob_page_alloc(struct page *sp, size_t size, int align, int align_offset) { - struct page *sp; ++ struct folio *folio; + struct slab *sp; struct list_head *slob_list; slob_t *b = NULL; @@ mm/slob.c: static void *slob_alloc(size_t size, gfp_t gfp, int align, int node, return NULL; - sp = virt_to_page(b); - __SetPageSlab(sp); -+ sp = virt_to_slab(b); -+ __SetPageSlab(slab_page(sp)); ++ folio = virt_to_folio(b); ++ __folio_set_slab(folio); ++ sp = folio_slab(folio); spin_lock_irqsave(&slob_lock, flags); sp->units = SLOB_UNITS(PAGE_SIZE); @@ mm/slob.c: static void slob_free(void *block, int size) spin_unlock_irqrestore(&slob_lock, flags); - __ClearPageSlab(sp); - page_mapcount_reset(sp); -+ __ClearPageSlab(slab_page(sp)); ++ __folio_clear_slab(slab_folio(sp)); + page_mapcount_reset(slab_page(sp)); slob_free_pages(b, 0); return; } +@@ mm/slob.c: EXPORT_SYMBOL(__kmalloc_node_track_caller); + + void kfree(const void *block) + { +- struct page *sp; ++ struct folio *sp; + + trace_kfree(_RET_IP_, block); + +@@ mm/slob.c: void kfree(const void *block) + return; + kmemleak_free(block); + +- sp = virt_to_page(block); +- if (PageSlab(sp)) { ++ sp = virt_to_folio(block); ++ if (folio_test_slab(sp)) { + int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN); + unsigned int *m = (unsigned int *)(block - align); + slob_free(m, *m + align); + } else { +- unsigned int order = compound_order(sp); +- mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, ++ unsigned int order = folio_order(sp); ++ ++ mod_node_page_state(folio_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); +- __free_pages(sp, order); ++ __free_pages(folio_page(sp, 0), order); + + } + }25:
aa4f573a4c96! 25:466b9fb1f6e5mm/kasan: Convert to struct folio and struct slab
@@ Commit messageSigned-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>+ Reviewed-by: Andrey Konovalov <[email protected]>
Cc: Andrey Ryabinin <[email protected]>
Cc: Alexander Potapenko <[email protected]>
Cc: Andrey Konovalov <[email protected]>26: 67b7966d2fb6 = 26: b8159ae8e5cd mm/kfence: Convert kfence_guarded_alloc() to struct slab 31: d64dfe49c1e7 ! 27: 4525180926f9 mm/sl*b: Differentiate struct slab fields by sl*b implementations @@ Commit message possible. This should also prevent accidental use of fields that don't exist in given - implementation. Before this patch virt_to_cache() and and cache_from_obj() was - visible for SLOB (albeit not used), although it relies on the slab_cache field + implementation. Before this patch virt_to_cache() and cache_from_obj() were + visible for SLOB (albeit not used), although they rely on the slab_cache field that isn't set by SLOB. With this patch it's now a compile error, so these functions are now hidden behind #ifndef CONFIG_SLOB. @@ mm/kfence/core.c: static void *kfence_guarded_alloc(struct kmem_cache *cache, si - slab->s_mem = addr; +#if defined(CONFIG_SLUB) + slab->objects = 1; -+#elif defined (CONFIG_SLAB) ++#elif defined(CONFIG_SLAB) + slab->s_mem = addr; +#endif @@ mm/slab.h + +#if defined(CONFIG_SLAB) + -+ union { -+ struct list_head slab_list; + union { + struct list_head slab_list; +- struct { /* Partial pages */ + struct rcu_head rcu_head; + }; + struct kmem_cache *slab_cache; + void *freelist; /* array of free object indexes */ -+ void * s_mem; /* first object */ ++ void *s_mem; /* first object */ + unsigned int active; + +#elif defined(CONFIG_SLUB) + - union { - struct list_head slab_list; -- struct { /* Partial pages */ ++ union { ++ struct list_head slab_list; + struct rcu_head rcu_head; + struct { struct slab *next; @@ mm/slab.h: struct slab { +#elif defined(CONFIG_SLOB) + + struct list_head slab_list; -+ void * __unused_1; ++ void *__unused_1; + void *freelist; /* first free block */ -+ void * __unused_2; ++ void *__unused_2; + int units; + +#else @@ mm/slab.h: struct slab { #ifdef CONFIG_MEMCG unsigned long memcg_data; @@ mm/slab.h: struct slab { - static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl)) SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ + SLAB_MATCH(slab_list, slab_list); +#ifndef CONFIG_SLOB SLAB_MATCH(rcu_head, rcu_head); + SLAB_MATCH(slab_cache, slab_cache); ++#endif ++#ifdef CONFIG_SLAB + SLAB_MATCH(s_mem, s_mem); + SLAB_MATCH(active, active); +#endif SLAB_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG 32: 0abf87bae67e = 28: 94b78948d53f mm/slub: Simplify struct slab slabs field definition 33: 813c304f18e4 = 29: f5261e6375f0 mm/slub: Define struct slab fields for CONFIG_SLUB_CPU_PARTIAL only when enabled 27: ebce4b5b5ced ! 30: 1414e8c87de6 zsmalloc: Stop using slab fields in struct page@@ Commit message
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>- Cc: Minchan Kim <[email protected]>
+ Acked-by: Minchan Kim <[email protected]>
Cc: Nitin Gupta <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>28: f124425ae7de = 31: 8a3cda6b38eb bootmem: Use page->index instead of page->freelist 29: 82da48c73b2e < -: ------------ iommu: Use put_pages_list 30: 181e16dfefbb < -: ------------ mm: Remove slab from struct page -: ------------ > 32: 91e069ba116b mm/slob: Remove unnecessary page_mapcount_reset() function call
Quoted 11 lines · Hyeonggon Yoo
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Quoted 6 lines
Reviewing the whole patch series takes longer than I thought. I'll try to review and test rest of patches when I have time.
I added Tested-by if kernel builds okay and kselftests does not break the kernel on my machine. (with CONFIG_SLAB/SLUB/SLOB depending on the patch),
Thanks!
Quoted 1 line
Let me know me if you know better way to test a patch.
Testing on your machine is just fine.
Quoted 9 lines
# mm/slub: Define struct slab fields for
CONFIG_SLUB_CPU_PARTIALonly when enabledReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Comment:
Works on bothSLUB_CPU_PARTIALand !SLUB_CPU_PARTIAL. btw, do we needslabs_cpu_partialattribute when we don't use cpu partials? (!SLUB_CPU_PARTIAL)
The sysfs attribute? Yeah we should be consistent to userspace expecting to read it (even with zeroes), regardless of config.
Quoted 18 lines
# mm/slub: Simplify struct slab slabs field definition Comment:
This is how struct page looks on the top of v3r3 branch:
struct page {
[...]struct { /* slab, slob and slub */ union { struct list_head slab_list; struct { /* Partial pages */ struct page *next; #ifdef CONFIG_64BIT int pages; /* Nr of pages left */ #else short int pages; #endif }; };
Quoted 3 lines
[...]
It's not consistent with struct slab.
Hm right. But as we don't actually use the struct page version anymore, and it's not one of the fields checked by SLAB_MATCH(), we can ignore this.
Quoted 1 line
I think this is because "mm: Remove slab from struct page" was dropped.
That was just postponed until iommu changes are in. Matthew mentioned those might be merged too, so that final cleanup will happen too and take care of the discrepancy above, so no need for extra churn to address it speficially.
Quoted 16 lines
Would you update some of patches?
#
mm/sl*b: Differentiate struct slab fields by sl*b implementationsReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Works SL[AUO]B on my machine and makes code much better.
# mm/slob: Convert SLOB to use struct slab and struct folio
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>It still works fine on SLOB.
# mm/slab: Convert
kmem_getpages()andkmem_freepages()to struct slabReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>
Quoted 6 lines
# mm/slub: Convert
__free_slab()to use struct slabReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Thanks,
Hyeonggon.
Thanks again,
Vlastimil
Robin Murphy
maintainerQuoted 60 lines · Vlastimil Babka
Quoted 11 lines · Hyeonggon Yoo
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Quoted 6 lines
Reviewing the whole patch series takes longer than I thought. I'll try to review and test rest of patches when I have time.
I added Tested-by if kernel builds okay and kselftests does not break the kernel on my machine. (with CONFIG_SLAB/SLUB/SLOB depending on the patch),
Thanks!
Quoted 1 line
Let me know me if you know better way to test a patch.
Testing on your machine is just fine.
Quoted 9 lines
# mm/slub: Define struct slab fields for
CONFIG_SLUB_CPU_PARTIALonly when enabledReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Comment:
Works on bothSLUB_CPU_PARTIALand !SLUB_CPU_PARTIAL. btw, do we needslabs_cpu_partialattribute when we don't use cpu partials? (!SLUB_CPU_PARTIAL)The sysfs attribute? Yeah we should be consistent to userspace expecting to read it (even with zeroes), regardless of config.
Quoted 18 lines
# mm/slub: Simplify struct slab slabs field definition Comment:
This is how struct page looks on the top of v3r3 branch:
struct page {
[...]struct { /* slab, slob and slub */ union { struct list_head slab_list; struct { /* Partial pages */ struct page *next; #ifdef CONFIG_64BIT int pages; /* Nr of pages left */ #else short int pages; #endif }; };Quoted 3 lines
[...]
It's not consistent with struct slab.
Quoted 8 lines
Hm right. But as we don't actually use the struct page version anymore, and it's not one of the fields checked by
SLAB_MATCH(), we can ignore this.Quoted 1 line
I think this is because "mm: Remove slab from struct page" was dropped.
That was just postponed until iommu changes are in. Matthew mentioned those might be merged too, so that final cleanup will happen too and take care of the discrepancy above, so no need for extra churn to address it speficially.
FYI the IOMMU changes are now queued in linux-next, so if all goes well you might be able to sneak that final patch in too.
Robin.
Quoted 30 lines
Quoted 17 lines
Would you update some of patches?
#
mm/sl*b: Differentiate struct slab fields by sl*b implementationsReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Works SL[AUO]B on my machine and makes code much better.
# mm/slob: Convert SLOB to use struct slab and struct folio
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>It still works fine on SLOB.
# mm/slab: Convert
kmem_getpages()andkmem_freepages()to struct slabReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Quoted 6 lines
# mm/slub: Convert
__free_slab()to use struct slabReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Thanks,
Hyeonggon.Thanks again,
Vlastimil
_______________________________________________
iommu mailing list iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 12 lines · Vlastimil Babka
Quoted 11 lines · Hyeonggon Yoo
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Quoted 9 lines
Quoted 6 lines
Reviewing the whole patch series takes longer than I thought. I'll try to review and test rest of patches when I have time.
I added Tested-by if kernel builds okay and kselftests does not break the kernel on my machine. (with CONFIG_SLAB/SLUB/SLOB depending on the patch),
Thanks!
:)
Quoted 4 lines
Quoted 1 line
Let me know me if you know better way to test a patch.
Testing on your machine is just fine.
Good!
Quoted 5 lines
Quoted 5 lines
# mm/slub: Define struct slab fields for
CONFIG_SLUB_CPU_PARTIALonly when enabledReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>
Quoted 8 lines
Quoted 4 lines
Comment:
Works on bothSLUB_CPU_PARTIALand !SLUB_CPU_PARTIAL. btw, do we needslabs_cpu_partialattribute when we don't use cpu partials? (!SLUB_CPU_PARTIAL)The sysfs attribute? Yeah we should be consistent to userspace expecting to read it (even with zeroes), regardless of config.
I thought entirely disabling the attribute is simpler, But okay If it should be exposed even if it's always zero.
Quoted 20 lines
Quoted 20 lines
# mm/slub: Simplify struct slab slabs field definition Comment:
This is how struct page looks on the top of v3r3 branch:
struct page {
[...]struct { /* slab, slob and slub */ union { struct list_head slab_list; struct { /* Partial pages */ struct page *next; #ifdef CONFIG_64BIT int pages; /* Nr of pages left */ #else short int pages; #endif }; };[...]
Quoted 5 lines
Quoted 1 line
It's not consistent with struct slab.
Hm right. But as we don't actually use the struct page version anymore, and it's not one of the fields checked by
SLAB_MATCH(), we can ignore this.
Yeah this is not a big problem. just mentioned this because
it looked weird and I didn't know when the patch "mm: Remove slab from struct page" will come back.
Quoted 6 lines
Quoted 1 line
I think this is because "mm: Remove slab from struct page" was dropped.
That was just postponed until iommu changes are in. Matthew mentioned those might be merged too, so that final cleanup will happen too and take care of the discrepancy above, so no need for extra churn to address it speficially.
Okay it seems no extra work needed until the iommu changes are in!
BTW, in the patch (that I sent) ("mm/slob: Remove unnecessary page_mapcount_reset() function call"), it refers commit 4525180926f9 ("mm/sl*b: Differentiate struct slab fields by sl*b implementations"). But the commit hash 4525180926f9 changed after the
tree has been changed.
It will be nice to write a script to handle situations like this.
Quoted 20 lines
Quoted 20 lines
Would you update some of patches?
#
mm/sl*b: Differentiate struct slab fields by sl*b implementationsReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>Works SL[AUO]B on my machine and makes code much better.
# mm/slob: Convert SLOB to use struct slab and struct folio
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>It still works fine on SLOB.
# mm/slab: Convert
kmem_getpages()andkmem_freepages()to struct slabReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]># mm/slub: Convert
__free_slab()to use struct slabReviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>
Quoted 5 lines
Quoted 2 lines
Thanks,
Hyeonggon.Thanks again,
Vlastimil
Have a nice day, thanks!
Hyeonggon.
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Hi, I've pushed another update branch slab-struct_slab-v4r1, and also to -next. I've shortened git commit log lines to make checkpatch happier, so no range-diff as it would be too long. I believe it would be useless spam to post the whole series now, shortly before xmas, so I will do it at rc8 time, to hopefully collect remaining reviews. But if anyone wants a mailed version, I can do that.
Changes in v4:
- rebase to 5.16-rc6 to avoid a conflict with mainline
- collect acks/reviews/tested-by from Johannes, Roman, Hyeonggon Yoo - thanks!
- in patch "mm/slub: Convert detached_freelist to use a struct slab" renamed free_nonslab_page() to free_large_kmalloc() and use folio there, as suggested by Roman
- in "mm/memcg: Convert slab objcgs from struct page to struct slab" change one caller of slab_objcgs_check() to slab_objcgs() as suggested by Johannes, realize the other caller should be also changed, and remove slab_objcgs_check() completely.
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Quoted 6 lines
Hi, I've pushed another update branch slab-struct_slab-v4r1, and also to -next. I've shortened git commit log lines to make checkpatch happier, so no range-diff as it would be too long. I believe it would be useless spam to post the whole series now, shortly before xmas, so I will do it at rc8 time, to hopefully collect remaining reviews. But if anyone wants a mailed version, I can do that.
Hello Vlastimil, Merry Christmas!
This is part 2 of reviewing/testing patches.
# mm/kasan: Convert to struct folio and struct slab
I'm not familiar with kasan yet but kasan runs well on my machine and kasan's bug report functionality too works fine.
# mm: Convert struct page to struct slab in functions used by other subsystems I'm not familiar with kasan, but to ask:
Does ____kasan_slab_free detect invalid free if someone frees
an object that is not allocated from slab?
@@ -341,7 +341,7 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
- if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) !=
+ if (unlikely(nearest_obj(cache, virt_to_slab(object), object) !=
object)) {
kasan_report_invalid_free(tagged_object, ip);
return true;I'm asking this because virt_to_slab() will return NULL if folio_test_slab() returns false. That will cause NULL pointer dereference in nearest_obj. I don't think this change is intended.
This makes me think some of virt_to_head_page() -> virt_to_slab() conversion need to be reviewed with caution.
# mm/slab: Finish struct page to struct slab conversion
# mm/slab: Convert most struct page to struct slab by spatch
I'll come back with part 3 :)
Enjoy your Christmas!
Hyeonggon
Quoted 11 lines
Changes in v4:
- rebase to 5.16-rc6 to avoid a conflict with mainline
- collect acks/reviews/tested-by from Johannes, Roman, Hyeonggon Yoo - thanks!
- in patch "mm/slub: Convertdetached_freelistto use a struct slab" renamedfree_nonslab_page()tofree_large_kmalloc()and use folio there, as suggested by Roman
- in "mm/memcg: Convert slab objcgs from struct page to struct slab" change one caller ofslab_objcgs_check()toslab_objcgs()as suggested by Johannes, realize the other caller should be also changed, and removeslab_objcgs_check()completely.
Quoted 13 lines · Hyeonggon Yoo
# mm: Convert struct page to struct slab in functions used by other subsystems I'm not familiar with kasan, but to ask:
Does____kasan_slab_freedetect invalid free if someone frees
an object that is not allocated from slab?(quoted diff) +1
-1@@ -341,7 +341,7 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object, - if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) != + if (unlikely(nearest_obj(cache, virt_to_slab(object), object) != object)) { kasan_report_invalid_free(tagged_object, ip); return true; I'm asking this because virt_to_slab() will return NULL if folio_test_slab()
Quoted 2 lines
returns false. That will cause NULL pointer dereference in
nearest_obj. I don't think this change is intended.
You need to track down how this could happen. As far as I can tell, it's always called when we know the object is part of a slab. That's where the cachep pointer is deduced from.
Quoted 13 lines · Matthew Wilcox
Quoted 12 lines · Hyeonggon Yoo
# mm: Convert struct page to struct slab in functions used by other subsystems I'm not familiar with kasan, but to ask:
Does____kasan_slab_freedetect invalid free if someone frees
an object that is not allocated from slab?(quoted diff) +1
-1@@ -341,7 +341,7 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object, - if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) != + if (unlikely(nearest_obj(cache, virt_to_slab(object), object) != object)) { kasan_report_invalid_free(tagged_object, ip); return true;
Quoted 7 lines
Quoted 3 lines
(quoted diff) +0
-0I'm asking this because virt_to_slab() will return NULL if folio_test_slab()returns false. That will cause NULL pointer dereference in
nearest_obj. I don't think this change is intended.You need to track down how this could happen. As far as I can tell, it's always called when we know the object is part of a slab. That's where the cachep pointer is deduced from.
Thank you Matthew, you are right. I read the code too narrowly. when we call kasan hooks, we know that the object is allocated from the slab cache. (through cache_from_obj)
I'll review that patch again in part 3!
Thanks,
Hyeonggon
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Quoted 7 lines
Hi, I've pushed another update branch slab-struct_slab-v4r1, and also to -next. I've shortened git commit log lines to make checkpatch happier, so no range-diff as it would be too long. I believe it would be useless spam to post the whole series now, shortly before xmas, so I will do it at rc8 time, to hopefully collect remaining reviews. But if anyone wants a mailed version, I can do that.
Hello Matthew and Vlastimil.
it's part 3 of review.
# mm: Convert struct page to struct slab in functions used by other subsystems
# mm/slub: Convert most struct page to struct slab by spatch
with a question below.
-static int check_slab(struct kmem_cache *s, struct page *page) +static int check_slab(struct kmem_cache *s, struct slab *slab)
{
int maxobj;- if (!PageSlab(page)) {
- slab_err(s, page, "Not a valid slab page");
+ if (!folio_test_slab(slab_folio(slab))) {
+ slab_err(s, slab, "Not a valid slab page");
return 0;
}Can't we guarantee that struct slab * always points to a slab?
for struct page * it can be !PageSlab(page) because struct page * can be other than slab. but struct slab * can only be slab unlike struct page. code will be simpler if we guarantee that struct slab * always points to a slab (or NULL).
# mm/slub: Convert pfmemalloc_match() to take a struct slab It's confusing to me because the original pfmemalloc_match() is removed and pfmemalloc_match_unsafe() was renamed to pfmemalloc_match() and converted to use slab_test_pfmemalloc() helper.
But I agree with the resulting code. so:
# mm/slub: Convert alloc_slab_page() to return a struct slab
# mm/slub: Convert print_page_info() to print_slab_info()
I hope to review rest of patches in a week.
Thanks,
Hyeonggon
Quoted 11 lines
Changes in v4:
- rebase to 5.16-rc6 to avoid a conflict with mainline
- collect acks/reviews/tested-by from Johannes, Roman, Hyeonggon Yoo - thanks!
- in patch "mm/slub: Convertdetached_freelistto use a struct slab" renamedfree_nonslab_page()tofree_large_kmalloc()and use folio there, as suggested by Roman
- in "mm/memcg: Convert slab objcgs from struct page to struct slab" change one caller ofslab_objcgs_check()toslab_objcgs()as suggested by Johannes, realize the other caller should be also changed, and removeslab_objcgs_check()completely.
Quoted 44 lines · Hyeonggon Yoo
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Vlastimil Babka
Quoted 5 lines · Vlastimil Babka
Folks from non-slab subsystems are Cc'd only to patches affecting them, and this cover letter.
Series also available in git, based on 5.16-rc3:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-struct_slab-v2r2Pushed a new branch slab-struct-slab-v3r3 with accumulated fixes and small tweaks and a new patch from Hyeonggon Yoo on top. To avoid too much spam, here's a range diff:
Quoted 7 lines
Hi, I've pushed another update branch slab-struct_slab-v4r1, and also to -next. I've shortened git commit log lines to make checkpatch happier, so no range-diff as it would be too long. I believe it would be useless spam to post the whole series now, shortly before xmas, so I will do it at rc8 time, to hopefully collect remaining reviews. But if anyone wants a mailed version, I can do that.
Hello Matthew and Vlastimil.
it's part 3 of review.# mm: Convert struct page to struct slab in functions used by other subsystems
Reviewed-by Hyeonggon Yoo <[email protected]># mm/slub: Convert most struct page to struct slab by spatch
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]>with a question below.
(quoted diff) +3
-3-static int check_slab(struct kmem_cache *s, struct page *page) +static int check_slab(struct kmem_cache *s, struct slab *slab) { int maxobj; - if (!PageSlab(page)) { - slab_err(s, page, "Not a valid slab page"); + if (!folio_test_slab(slab_folio(slab))) { + slab_err(s, slab, "Not a valid slab page"); return 0; }
Quoted 1 line
Can't we guarantee that struct slab * always points to a slab?
Normally, yes.
Quoted 4 lines
for struct page * it can be !PageSlab(page) because struct page * can be other than slab. but struct slab * can only be slab unlike struct page. code will be simpler if we guarantee that struct slab * always points to a slab (or NULL).
That's what the code does indeed. But check_slab() is called as part of various consistency checks, so there we on purpose question all assumptions in order to find a bug (e.g. memory corruption) - such as a page that's still on the list of slabs while it was already freed and reused and thus e.g. lacks the slab page flag.
But it's nice how using struct slab makes such a check immediately stand out as suspicious, right?
Quoted 10 lines
# mm/slub: Convert
pfmemalloc_match()to take a struct slab It's confusing to me because the originalpfmemalloc_match()is removed andpfmemalloc_match_unsafe()was renamed topfmemalloc_match()and converted to useslab_test_pfmemalloc()helper.But I agree with the resulting code. so:
Reviewed-by Hyeonggon Yoo <[email protected]># mm/slub: Convert
alloc_slab_page()to return a struct slab
Quoted 8 lines
Reviewed-by Hyeonggon Yoo <[email protected]>Tested-by Hyeonggon Yoo <[email protected]># mm/slub: Convert
print_page_info()toprint_slab_info()Reviewed-by Hyeonggon Yoo <[email protected]>I hope to review rest of patches in a week.
Thanks for your reviews/tests!