mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool

PATCH Superseded [email protected] 4 participants 28 messages Jun 28, 2024 – Jul 2, 2024

Subsystems (2)

Memory management Zsmalloc compressed slab memory allocator

lore

Share & tools

lore

[email protected]

author

2024-06-28 03:06 UTC #0 lore raw

cover mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool

From: Alex Shi <[email protected]>

According to Metthew's plan, the page descriptor will be replace by a 8 bytes mem_desc on destination purpose.
https://lore.kernel.org/lkml/[email protected]/

Here is a implement on zsmalloc to replace page descriptor by 'zpdesc', which is still overlay on struct page now. but it's a step move forward above destination.

To name the struct zpdesc instead of zsdesc, since there are still 3 zpools under zswap: zbud, z3fold, zsmalloc for now(z3fold maybe removed soon), and we could easyly extend it to other zswap.zpool in needs.

For all zswap.zpools, they are all using single page since often used under memory pressure. So the conversion via folio series helper is better than page's for compound_head check saving.

For now, all zpools are using some page struct members, like page.flags for PG_private/PG_locked. and list_head lru, page.mapping for page migration.

This patachset could save 123Kbyetes zsmalloc.o size.

Thanks
Alex

Alex Shi (8):
mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool mm/zsmalloc: use zpdesc in trylock_zspage/lock_zspage mm/zsmalloc: convert create_page_chain() and its users to use zpdesc mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it mm/zsmalloc: convert SetZsPageMovable and remove unused funcs mm/zsmalloc: introduce __zpdesc_clear_movable mm/zsmalloc: introduce __zpdesc_clear_zsmalloc mm/zsmalloc: introduce __zpdesc_set_zsmalloc()

Hyeonggon Yoo (12):
mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc mm/zsmalloc: add and use pfn/zpdesc seeking funcs mm/zsmalloc: convert obj_malloc() to use zpdesc mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc mm/zsmalloc: convert init_zspage() to use zpdesc mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc mm/zsmalloc: add zpdesc_is_isolated/zpdesc_zone helper for
zs_page_migrate
mm/zsmalloc: convert __free_zspage() to use zdsesc mm/zsmalloc: convert location_to_obj() to take zpdesc mm/zsmalloc: convert migrate_zspage() to use zpdesc mm/zsmalloc: convert get_zspage() to take zpdesc mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc

 mm/zpdesc.h   | 134 +++++++++++++++
 mm/zsmalloc.c | 454 +++++++++++++++++++++++++++-----------------------
 2 files changed, 384 insertions(+), 204 deletions(-)
 create mode 100644 mm/zpdesc.h

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:06 UTC #1 lore raw

PATCH 1/20 mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool

From: Alex Shi <[email protected]>

The 1st patch introduces new memory decriptor zpdesc and rename zspage.first_page to zspage.first_zpdesc, no functional change.

Originally-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/zsmalloc.c | 19 ++++++++---------
 2 files changed, 66 insertions(+), 9 deletions(-)
 create mode 100644 mm/zpdesc.h

mm/zpdesc.h new file +56 -0

new file mode 100644
index 000000000000..a1ab5ebaa936
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* zpdesc.h: zswap.zpool memory descriptor
+ *
+ * Written by Alex Shi <[email protected]>
+ *	      Hyeonggon Yoo <[email protected]>
+ */
+#ifndef __MM_ZPDESC_H__
+#define __MM_ZPDESC_H__
+
+/*
+ * struct zpdesc -	Memory descriptor for zpool memory, now is for zsmalloc
+ * @flags:		Page flags, PG_private: identifies the first component page
+ * @lru:		Indirected used by page migration
+ * @next:		Next zpdesc in a zspage in zsmalloc zpool
+ * @handle:		For huge zspage in zsmalloc zpool
+ * @zspage:		Pointer to zspage in zsmalloc
+ *
+ * This struct overlays struct page for now. Do not modify without a good
+ * understanding of the issues.
+ */
+struct zpdesc {
+	unsigned long flags;
+	struct list_head lru;
+	unsigned long _zp_pad_1;
+	union {
+		/* Next zpdescs in a zspage in zsmalloc zpool */
+		struct zpdesc *next;
+		/* For huge zspage in zsmalloc zpool */
+		unsigned long handle;
+	};
+	struct zspage *zspage;
+};
+#define ZPDESC_MATCH(pg, zp) \
+	static_assert(offsetof(struct page, pg) == offsetof(struct zpdesc, zp))
+
+ZPDESC_MATCH(flags, flags);
+ZPDESC_MATCH(lru, lru);
+ZPDESC_MATCH(index, next);
+ZPDESC_MATCH(index, handle);
+ZPDESC_MATCH(private, zspage);
+#undef ZPDESC_MATCH
+static_assert(sizeof(struct zpdesc) <= sizeof(struct page));
+
+#define zpdesc_page(zp)			(_Generic((zp),			\
+	const struct zpdesc *:		(const struct page *)(zp),	\
+	struct zpdesc *:		(struct page *)(zp)))
+
+#define zpdesc_folio(zp)		(_Generic((zp),			\
+	const struct zpdesc *:		(const struct folio *)(zp),	\
+	struct zpdesc *:		(struct folio *)(zp)))
+
+#define page_zpdesc(p)			(_Generic((p),			\
+	const struct page *:		(const struct zpdesc *)(p),	\
+	struct page *:			(struct zpdesc *)(p)))
+
+#endif

mm/zsmalloc.c +10 -9

index fec1a39e5bbe..67bb80b7413a 100644
@@ -13,17 +13,17 @@
 
 /*
  * Following is how we use various fields and flags of underlying
- * struct page(s) to form a zspage.
+ * struct zpdesc(page) to form a zspage.
  *
- * Usage of struct page fields:
- *	page->private: points to zspage
- *	page->index: links together all component pages of a zspage
+ * Usage of struct zpdesc fields:
+ *	zpdesc->zspage: points to zspage
+ *	zpdesc->next: links together all component pages of a zspage
  *		For the huge page, this is always 0, so we use this field
  *		to store handle.
  *	page->page_type: PG_zsmalloc, lower 16 bit locate the first object
  *		offset in a subpage of a zspage
  *
- * Usage of struct page flags:
+ * Usage of struct zpdesc(page) flags:
  *	PG_private: identifies the first component page
  *	PG_owner_priv_1: identifies the huge component page
  *

@@ -64,6 +64,7 @@
 #include <linux/pagemap.h>
 #include <linux/fs.h>
 #include <linux/local_lock.h>
+#include "zpdesc.h"
 
 #define ZSPAGE_MAGIC	0x58

@@ -253,7 +254,7 @@ struct zspage {
 	};
 	unsigned int inuse;
 	unsigned int freeobj;
-	struct page *first_page;
+	struct zpdesc *first_zpdesc;
 	struct list_head list; /* fullness list */
 	struct zs_pool *pool;
 	rwlock_t lock;

@@ -448,7 +449,7 @@ static inline void mod_zspage_inuse(struct zspage *zspage, int val)
 
 static inline struct page *get_first_page(struct zspage *zspage)
 {
-	struct page *first_page = zspage->first_page;
+	struct page *first_page = zpdesc_page(zspage->first_zpdesc);
 
 	VM_BUG_ON_PAGE(!is_first_page(first_page), first_page);
 	return first_page;

@@ -948,7 +949,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
 		set_page_private(page, (unsigned long)zspage);
 		page->index = 0;
 		if (i == 0) {
-			zspage->first_page = page;
+			zspage->first_zpdesc = page_zpdesc(page);
 			SetPagePrivate(page);
 			if (unlikely(class->objs_per_zspage == 1 &&
 					class->pages_per_zspage == 1))

@@ -1325,7 +1326,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 		link->handle = handle;
 	else
 		/* record handle to page->index */
-		zspage->first_page->index = handle;
+		zspage->first_zpdesc->handle = handle;
 
 	kunmap_atomic(vaddr);
 	mod_zspage_inuse(zspage, 1);

Signature · 1 line

2.43.0

Hyeonggon Yoo

2024-06-30 13:45 UTC #2 lore raw

Quoted 6 lines · On Fri, Jun 28, 2024 at 12:06 PM <[email protected]> wrote:

From: Alex Shi <[email protected]>
The 1st patch introduces new memory decriptor zpdesc and rename zspage.first_page to zspage.first_zpdesc, no functional change.

Quoted 7 lines

Originally-by Hyeonggon Yoo <[email protected]>
Signed-off-by Alex Shi <[email protected]>
---
 mm/zpdesc.h   | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/zsmalloc.c | 19 ++++++++---------
 2 files changed, 66 insertions(+), 9 deletions(-)
 create mode 100644 mm/zpdesc.h

Hi Alex, thanks for your effort in pushing this forward!

Quoted 9 lines

mm/zpdesc.h new file +3 -0

new file mode 100644
index 000000000000..a1ab5ebaa936
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* zpdesc.h: zswap.zpool memory descriptor
+ *

mm/zpdesc.h line 13

Quoted 10 lines

mm/zpdesc.h new file +10 -0

+ * Written by Alex Shi <[email protected]>
+ *           Hyeonggon Yoo <[email protected]>
+ */
+#ifndef __MM_ZPDESC_H__
+#define __MM_ZPDESC_H__
+
+/*
+ * struct zpdesc -     Memory descriptor for zpool memory, now is for zsmalloc
+ * @flags:             Page flags, PG_private: identifies the first component page
+ * @lru:               Indirected used by page migration

maybe Indirected -> Indirectly?

mm/zpdesc.h line 24

Quoted 11 lines

mm/zpdesc.h new file +11 -0

+ * @next:              Next zpdesc in a zspage in zsmalloc zpool
+ * @handle:            For huge zspage in zsmalloc zpool
+ * @zspage:            Pointer to zspage in zsmalloc
+ *
+ * This struct overlays struct page for now. Do not modify without a good
+ * understanding of the issues.
+ */
+struct zpdesc {
+       unsigned long flags;
+       struct list_head lru;
+       unsigned long _zp_pad_1;

for understanding, I think it'd be better to replace _zp_pad_1 with movable ops, because mops reuses this 'mapping' field.

mm/zpdesc.h line 31

Quoted 7 lines

mm/zpdesc.h new file +7 -0

+       union {
+               /* Next zpdescs in a zspage in zsmalloc zpool */
+               struct zpdesc *next;
+               /* For huge zspage in zsmalloc zpool */
+               unsigned long handle;
+       };
+       struct zspage *zspage;

There was a discussion with Yosry on including memcg_data on zpdesc even if it's not used at the moment.

Maybe you can look at:
https://lore.kernel.org/linux-mm/CAB=+i9Quz9iP2-Lq=oQfKVVnzPDtOaKMm=hUPbnRg5hRxH+qaA@mail.gmail.com/

Quoted 16 lines

mm/zsmalloc.c +3 -4

index fec1a39e5bbe..67bb80b7413a 100644
@@ -13,17 +13,17 @@
 
 /*
  * Following is how we use various fields and flags of underlying
- * struct page(s) to form a zspage.
+ * struct zpdesc(page) to form a zspage.
  *
- * Usage of struct page fields:
- *     page->private: points to zspage
- *     page->index: links together all component pages of a zspage
+ * Usage of struct zpdesc fields:
+ *     zpdesc->zspage: points to zspage

mm/zsmalloc.c line 28

Quoted 10 lines

mm/zsmalloc.c +2 -1

+ *     zpdesc->next: links together all component pages of a zspage
  *             For the huge page, this is always 0, so we use this field
  *             to store handle.
  *     page->page_type: PG_zsmalloc, lower 16 bit locate the first object
  *             offset in a subpage of a zspage
  *
- * Usage of struct page flags:
+ * Usage of struct zpdesc(page) flags:
  *     PG_private: identifies the first component page
  *     PG_owner_priv_1: identifies the huge component page

the comment for PG_owner_priv_1 can safely be removed as it's not used after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page to zspage")

Quoted 10 lines

mm/zsmalloc.c +1 -1

@@ -948,7 +949,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
                set_page_private(page, (unsigned long)zspage);
                page->index = 0;
                if (i == 0) {
-                       zspage->first_page = page;
+                       zspage->first_zpdesc = page_zpdesc(page);
                        SetPagePrivate(page);
                        if (unlikely(class->objs_per_zspage == 1 &&
                                        class->pages_per_zspage == 1))

mm/zsmalloc.c static unsigned long obj_malloc(struct zs_pool *pool, · line 1329

Quoted 5 lines

mm/zsmalloc.c +1 -1

@@ -1325,7 +1326,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
                link->handle = handle;
        else
                /* record handle to page->index */
-               zspage->first_page->index = handle;
+               zspage->first_zpdesc->handle = handle;

FYI this line seems to conflict with
bcc6116e39f512 ("mm/zsmalloc: move record_obj() into obj_malloc()") on mm-unstable.

Best,
Hyeonggon

Alex Shi

2024-07-01 03:03 UTC #3 lore raw

Quoted 30 lines · Hyeonggon Yoo

Quoted 6 lines · On Fri, Jun 28, 2024 at 12:06 PM <[email protected]> wrote:
From: Alex Shi <[email protected]>
The 1st patch introduces new memory decriptor zpdesc and rename zspage.first_page to zspage.first_zpdesc, no functional change.
Quoted 7 lines
Originally-by Hyeonggon Yoo <[email protected]>
Signed-off-by Alex Shi <[email protected]>
---
 mm/zpdesc.h   | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/zsmalloc.c | 19 ++++++++---------
 2 files changed, 66 insertions(+), 9 deletions(-)
 create mode 100644 mm/zpdesc.h
Hi Alex, thanks for your effort in pushing this forward!
Quoted 13 lines
mm/zpdesc.h new file +7 -0
new file mode 100644
index 000000000000..a1ab5ebaa936
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* zpdesc.h: zswap.zpool memory descriptor
+ *
+ * Written by Alex Shi <[email protected]>
+ *           Hyeonggon Yoo <[email protected]>
+ */
+#ifndef __MM_ZPDESC_H__

Quoted 8 lines

mm/zpdesc.h line 13

Quoted 6 lines

mm/zpdesc.h new file +6 -0

+#define __MM_ZPDESC_H__
+
+/*
+ * struct zpdesc -     Memory descriptor for zpool memory, now is for zsmalloc
+ * @flags:             Page flags, PG_private: identifies the first component page
+ * @lru:               Indirected used by page migration

maybe Indirected -> Indirectly?

Hi Yoo,

Thanks for comments! Yes Indirectly is better. I will update it in next version.

Quoted 13 lines

mm/zpdesc.h line 24

Quoted 12 lines

mm/zpdesc.h new file +11 -0

+ * @next:              Next zpdesc in a zspage in zsmalloc zpool
+ * @handle:            For huge zspage in zsmalloc zpool
+ * @zspage:            Pointer to zspage in zsmalloc
+ *
+ * This struct overlays struct page for now. Do not modify without a good
+ * understanding of the issues.
+ */
+struct zpdesc {
+       unsigned long flags;
+       struct list_head lru;
+       unsigned long _zp_pad_1;

Quoted 2 lines

for understanding, I think it'd be better to replace _zp_pad_1 with movable ops, because mops reuses this 'mapping' field.

Right, 'mops' looks a bit more clear.

Quoted 9 lines

mm/zpdesc.h line 31

Quoted 8 lines

mm/zpdesc.h new file +7 -0

+       union {
+               /* Next zpdescs in a zspage in zsmalloc zpool */
+               struct zpdesc *next;
+               /* For huge zspage in zsmalloc zpool */
+               unsigned long handle;
+       };
+       struct zspage *zspage;

Quoted 5 lines

There was a discussion with Yosry on including memcg_data on zpdesc even if it's not used at the moment.
Maybe you can look at:
https://lore.kernel.org/linux-mm/CAB=+i9Quz9iP2-Lq=oQfKVVnzPDtOaKMm=hUPbnRg5hRxH+qaA@mail.gmail.com/

Thanks for notice.
The memcg_data isn't used for zpdesc. And I have a bit confusion since Yosry said: "I think to drop memcg_data we need to enlighten the code that ...". So we actually don't need to have this unused member, is this right, Yosry?

Quoted 23 lines

mm/zsmalloc.c +4 -4

index fec1a39e5bbe..67bb80b7413a 100644
@@ -13,17 +13,17 @@
 
 /*
  * Following is how we use various fields and flags of underlying
- * struct page(s) to form a zspage.
+ * struct zpdesc(page) to form a zspage.
  *
- * Usage of struct page fields:
- *     page->private: points to zspage
- *     page->index: links together all component pages of a zspage
+ * Usage of struct zpdesc fields:
+ *     zpdesc->zspage: points to zspage
+ *     zpdesc->next: links together all component pages of a zspage
  *             For the huge page, this is always 0, so we use this field
  *             to store handle.
  *     page->page_type: PG_zsmalloc, lower 16 bit locate the first object
  *             offset in a subpage of a zspage
  *

Quoted 8 lines

mm/zsmalloc.c line 28
Quoted 4 lines
mm/zsmalloc.c +1 -1
- * Usage of struct page flags:
+ * Usage of struct zpdesc(page) flags:
  *     PG_private: identifies the first component page
  *     PG_owner_priv_1: identifies the huge component page
the comment for PG_owner_priv_1 can safely be removed as it's not used after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page to zspage")

Right, thanks for info!

Quoted 12 lines

mm/zsmalloc.c +1 -1

@@ -948,7 +949,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
                set_page_private(page, (unsigned long)zspage);
                page->index = 0;
                if (i == 0) {
-                       zspage->first_page = page;
+                       zspage->first_zpdesc = page_zpdesc(page);
                        SetPagePrivate(page);
                        if (unlikely(class->objs_per_zspage == 1 &&
                                        class->pages_per_zspage == 1))

@@ -1325,7 +1326,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
                link->handle = handle;

Quoted 8 lines

mm/zsmalloc.c static unsigned long obj_malloc(struct zs_pool *pool, · line 1329
Quoted 4 lines
mm/zsmalloc.c +1 -1
        else
                /* record handle to page->index */
-               zspage->first_page->index = handle;
+               zspage->first_zpdesc->handle = handle;
FYI this line seems to conflict with
bcc6116e39f512 ("mm/zsmalloc: move record_obj() into obj_malloc()") on mm-unstable.

yes, a new commit made this conflict. will update this in next version.

Thanks
Alex

Quoted 3 lines

Best,
Hyeonggon

Alex Shi

2024-07-01 03:40 UTC #4 lore raw

Ops,

Sorry for missing Yosry and Nhat. How stupid I am!
Here is the first patch on lore: https://lore.kernel.org/all/[email protected]/

Sorry for this!
Alex

Quoted 144 lines · Alex Shi

Quoted 30 lines · Hyeonggon Yoo
Quoted 6 lines · On Fri, Jun 28, 2024 at 12:06 PM <[email protected]> wrote:
From: Alex Shi <[email protected]>
The 1st patch introduces new memory decriptor zpdesc and rename zspage.first_page to zspage.first_zpdesc, no functional change.
Quoted 7 lines
Originally-by Hyeonggon Yoo <[email protected]>
Signed-off-by Alex Shi <[email protected]>
---
 mm/zpdesc.h   | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/zsmalloc.c | 19 ++++++++---------
 2 files changed, 66 insertions(+), 9 deletions(-)
 create mode 100644 mm/zpdesc.h
Hi Alex, thanks for your effort in pushing this forward!
Quoted 13 lines
mm/zpdesc.h new file +7 -0
new file mode 100644
index 000000000000..a1ab5ebaa936
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* zpdesc.h: zswap.zpool memory descriptor
+ *
+ * Written by Alex Shi <[email protected]>
+ *           Hyeonggon Yoo <[email protected]>
+ */
+#ifndef __MM_ZPDESC_H__
Quoted 8 lines
mm/zpdesc.h line 13
Quoted 6 lines
mm/zpdesc.h new file +6 -0
+#define __MM_ZPDESC_H__
+
+/*
+ * struct zpdesc -     Memory descriptor for zpool memory, now is for zsmalloc
+ * @flags:             Page flags, PG_private: identifies the first component page
+ * @lru:               Indirected used by page migration
maybe Indirected -> Indirectly?
Hi Yoo,
Thanks for comments! Yes Indirectly is better. I will update it in next version.
Quoted 13 lines
mm/zpdesc.h line 24
Quoted 12 lines
mm/zpdesc.h new file +11 -0
+ * @next:              Next zpdesc in a zspage in zsmalloc zpool
+ * @handle:            For huge zspage in zsmalloc zpool
+ * @zspage:            Pointer to zspage in zsmalloc
+ *
+ * This struct overlays struct page for now. Do not modify without a good
+ * understanding of the issues.
+ */
+struct zpdesc {
+       unsigned long flags;
+       struct list_head lru;
+       unsigned long _zp_pad_1;
Quoted 2 lines
for understanding, I think it'd be better to replace _zp_pad_1 with movable ops, because mops reuses this 'mapping' field.
Right, 'mops' looks a bit more clear.
Quoted 9 lines
mm/zpdesc.h line 31
Quoted 8 lines
mm/zpdesc.h new file +7 -0
+       union {
+               /* Next zpdescs in a zspage in zsmalloc zpool */
+               struct zpdesc *next;
+               /* For huge zspage in zsmalloc zpool */
+               unsigned long handle;
+       };
+       struct zspage *zspage;
Quoted 5 lines
There was a discussion with Yosry on including memcg_data on zpdesc even if it's not used at the moment.
Maybe you can look at:
https://lore.kernel.org/linux-mm/CAB=+i9Quz9iP2-Lq=oQfKVVnzPDtOaKMm=hUPbnRg5hRxH+qaA@mail.gmail.com/
Thanks for notice.
The memcg_data isn't used for zpdesc. And I have a bit confusion since Yosry said: "I think to drop memcg_data we need to enlighten the code that ...". So we actually don't need to have this unused member, is this right, Yosry?
Quoted 23 lines
Quoted 23 lines
mm/zsmalloc.c +4 -4
index fec1a39e5bbe..67bb80b7413a 100644
@@ -13,17 +13,17 @@
 
 /*
  * Following is how we use various fields and flags of underlying
- * struct page(s) to form a zspage.
+ * struct zpdesc(page) to form a zspage.
  *
- * Usage of struct page fields:
- *     page->private: points to zspage
- *     page->index: links together all component pages of a zspage
+ * Usage of struct zpdesc fields:
+ *     zpdesc->zspage: points to zspage
+ *     zpdesc->next: links together all component pages of a zspage
  *             For the huge page, this is always 0, so we use this field
  *             to store handle.
  *     page->page_type: PG_zsmalloc, lower 16 bit locate the first object
  *             offset in a subpage of a zspage
  *
Quoted 8 lines
mm/zsmalloc.c line 28
Quoted 4 lines
mm/zsmalloc.c +1 -1
- * Usage of struct page flags:
+ * Usage of struct zpdesc(page) flags:
  *     PG_private: identifies the first component page
  *     PG_owner_priv_1: identifies the huge component page
the comment for PG_owner_priv_1 can safely be removed as it's not used after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page to zspage")
Right, thanks for info!
Quoted 12 lines
Quoted 12 lines
mm/zsmalloc.c +1 -1
@@ -948,7 +949,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
                set_page_private(page, (unsigned long)zspage);
                page->index = 0;
                if (i == 0) {
-                       zspage->first_page = page;
+                       zspage->first_zpdesc = page_zpdesc(page);
                        SetPagePrivate(page);
                        if (unlikely(class->objs_per_zspage == 1 &&
                                        class->pages_per_zspage == 1))
@@ -1325,7 +1326,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
                link->handle = handle;
Quoted 8 lines
mm/zsmalloc.c static unsigned long obj_malloc(struct zs_pool *pool, · line 1329
Quoted 4 lines
mm/zsmalloc.c +1 -1
        else
                /* record handle to page->index */
-               zspage->first_page->index = handle;
+               zspage->first_zpdesc->handle = handle;
FYI this line seems to conflict with
bcc6116e39f512 ("mm/zsmalloc: move record_obj() into obj_malloc()") on mm-unstable.
yes, a new commit made this conflict. will update this in next version.
Thanks
Alex
Quoted 3 lines
Best,
Hyeonggon

Yosry Ahmed

2024-07-01 13:42 UTC #5 lore raw

In reply to Alex Shi

[..]

Quoted 8 lines

Quoted 7 lines

mm/zpdesc.h new file +7 -0

+       union {
+               /* Next zpdescs in a zspage in zsmalloc zpool */
+               struct zpdesc *next;
+               /* For huge zspage in zsmalloc zpool */
+               unsigned long handle;
+       };
+       struct zspage *zspage;

Quoted 8 lines

Quoted 5 lines
There was a discussion with Yosry on including memcg_data on zpdesc even if it's not used at the moment.
Maybe you can look at:
https://lore.kernel.org/linux-mm/CAB=+i9Quz9iP2-Lq=oQfKVVnzPDtOaKMm=hUPbnRg5hRxH+qaA@mail.gmail.com/
Thanks for notice.
The memcg_data isn't used for zpdesc. And I have a bit confusion since Yosry said: "I think to drop memcg_data we need to enlighten the code that ...". So we actually don't need to have this unused member, is this right, Yosry?

I think we need to have it, at least for now. When the pages are freed, put_page() -> folio_put() -> __folio_put() will call mem_cgroup_uncharge(). The latter will call folio_memcg() (which reads folio->memcg_data) to figure out if uncharging needs to be done.

There are also other similar code paths that will check folio->memcg_data. It is currently expected to be present for all folios. So until we have custom code paths per-folio type for allocation/freeing/etc, we need to keep folio->memcg_data present and properly initialized.

Alex Shi

2024-07-02 02:39 UTC #6 lore raw

Quoted 23 lines · Yosry Ahmed

[..]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
mm/zpdesc.h new file +7 -0
+       union {
+               /* Next zpdescs in a zspage in zsmalloc zpool */
+               struct zpdesc *next;
+               /* For huge zspage in zsmalloc zpool */
+               unsigned long handle;
+       };
+       struct zspage *zspage;
Quoted 8 lines
Quoted 5 lines
There was a discussion with Yosry on including memcg_data on zpdesc even if it's not used at the moment.
Maybe you can look at:
https://lore.kernel.org/linux-mm/CAB=+i9Quz9iP2-Lq=oQfKVVnzPDtOaKMm=hUPbnRg5hRxH+qaA@mail.gmail.com/
Thanks for notice.
The memcg_data isn't used for zpdesc. And I have a bit confusion since Yosry said: "I think to drop memcg_data we need to enlighten the code that ...". So we actually don't need to have this unused member, is this right, Yosry?
I think we need to have it, at least for now. When the pages are freed, put_page() -> folio_put() -> __folio_put() will call mem_cgroup_uncharge(). The latter will call folio_memcg() (which reads folio->memcg_data) to figure out if uncharging needs to be done.

Quoted 5 lines

There are also other similar code paths that will check folio->memcg_data. It is currently expected to be present for all folios. So until we have custom code paths per-folio type for allocation/freeing/etc, we need to keep folio->memcg_data present and properly initialized.

Hi Yosry,

Many thanks for detailed comments and requirements!
I'm going to add this member in next version, maybe tomorrow, unless there are more comments for this patchset. :)

Thank a lot!
Alex

[email protected]

author

2024-06-28 03:06 UTC #7 lore raw

PATCH 2/20 mm/zsmalloc: use zpdesc in trylock_zspage/lock_zspage

In reply to [email protected]

From: Alex Shi <[email protected]>

To use zpdesc in trylock_zspage/lock_zspage funcs, we add couple of helpers:
zpdesc_lock/zpdesc_unlock/zpdesc_trylock/zpdesc_wait_locked and zpdesc_get/zpdesc_put for this purpose.

Here we use the folio series func in guts for 2 reasons, one zswap.zpool only get single page, and use folio could save some compound_head checking; two, folio_put could bypass devmap checking that we don't need.

Originally-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   | 30 ++++++++++++++++++++++++
 mm/zsmalloc.c | 64 ++++++++++++++++++++++++++++++++++-----------------
 2 files changed, 73 insertions(+), 21 deletions(-)

mm/zpdesc.h +30 -0

index a1ab5ebaa936..fd95277843d5 100644
@@ -53,4 +53,34 @@ static_assert(sizeof(struct zpdesc) <= sizeof(struct page));
 	const struct page *:		(const struct zpdesc *)(p),	\
 	struct page *:			(struct zpdesc *)(p)))
 
+static inline void zpdesc_lock(struct zpdesc *zpdesc)
+{
+	folio_lock(zpdesc_folio(zpdesc));
+}
+
+static inline bool zpdesc_trylock(struct zpdesc *zpdesc)
+{
+	return folio_trylock(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_unlock(struct zpdesc *zpdesc)
+{
+	folio_unlock(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_wait_locked(struct zpdesc *zpdesc)
+{
+	folio_wait_locked(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_get(struct zpdesc *zpdesc)
+{
+	folio_get(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_put(struct zpdesc *zpdesc)
+{
+	folio_put(zpdesc_folio(zpdesc));
+}
+
 #endif

mm/zsmalloc.c +43 ~~-21~~

119 lines not on this page · raw message

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #8 lore raw

PATCH 3/20 mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

These two functions take pointer to an array of struct page. Introduce zpdesc_kmap_atomic() and make __zs_{map,unmap}_object() take pointer to an array of zpdesc instead of page.

Add silly type casting when calling them. Casting will be removed late.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

mm/zsmalloc.c +13 -8

index 9835121109d1..cedd3dfb9124 100644
@@ -245,6 +245,11 @@ struct zs_pool {
 	atomic_t compaction_in_progress;
 };
 
+static inline void *zpdesc_kmap_atomic(struct zpdesc *zpdesc)
+{
+	return kmap_atomic(zpdesc_page(zpdesc));
+}
+
 struct zspage {
 	struct {
 		unsigned int huge:HUGE_BITS;

@@ -1063,7 +1068,7 @@ static inline void __zs_cpu_down(struct mapping_area *area)
 }
 
 static void *__zs_map_object(struct mapping_area *area,
-			struct page *pages[2], int off, int size)
+			struct zpdesc *zpdescs[2], int off, int size)
 {
 	int sizes[2];
 	void *addr;

@@ -1080,10 +1085,10 @@ static void *__zs_map_object(struct mapping_area *area,
 	sizes[1] = size - sizes[0];
 
 	/* copy object to per-cpu buffer */
-	addr = kmap_atomic(pages[0]);
+	addr = zpdesc_kmap_atomic(zpdescs[0]);
 	memcpy(buf, addr + off, sizes[0]);
 	kunmap_atomic(addr);
-	addr = kmap_atomic(pages[1]);
+	addr = zpdesc_kmap_atomic(zpdescs[1]);
 	memcpy(buf + sizes[0], addr, sizes[1]);
 	kunmap_atomic(addr);
 out:

@@ -1091,7 +1096,7 @@ static void *__zs_map_object(struct mapping_area *area,
 }
 
 static void __zs_unmap_object(struct mapping_area *area,
-			struct page *pages[2], int off, int size)
+			struct zpdesc *zpdescs[2], int off, int size)
 {
 	int sizes[2];
 	void *addr;

@@ -1110,10 +1115,10 @@ static void __zs_unmap_object(struct mapping_area *area,
 	sizes[1] = size - sizes[0];
 
 	/* copy per-cpu buffer to object */
-	addr = kmap_atomic(pages[0]);
+	addr = zpdesc_kmap_atomic(zpdescs[0]);
 	memcpy(addr + off, buf, sizes[0]);
 	kunmap_atomic(addr);
-	addr = kmap_atomic(pages[1]);
+	addr = zpdesc_kmap_atomic(zpdescs[1]);
 	memcpy(addr, buf + sizes[0], sizes[1]);
 	kunmap_atomic(addr);

@@ -1254,7 +1259,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 	pages[1] = get_next_page(page);
 	BUG_ON(!pages[1]);
 
-	ret = __zs_map_object(area, pages, off, class->size);
+	ret = __zs_map_object(area, (struct zpdesc **)pages, off, class->size);
 out:
 	if (likely(!ZsHugePage(zspage)))
 		ret += ZS_HANDLE_SIZE;

@@ -1289,7 +1294,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 		pages[1] = get_next_page(page);
 		BUG_ON(!pages[1]);
 
-		__zs_unmap_object(area, pages, off, class->size);
+		__zs_unmap_object(area, (struct zpdesc **)pages, off, class->size);
 	}
 	local_unlock(&zs_map_area.lock);

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #9 lore raw

PATCH 4/20 mm/zsmalloc: add and use pfn/zpdesc seeking funcs

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Add pfn_zpdesc conversion, convert obj_to_location() to take zpdesc and also convert its users to use zpdesc.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   |  9 +++++++
 mm/zsmalloc.c | 75 ++++++++++++++++++++++++++-------------------------
 2 files changed, 47 insertions(+), 37 deletions(-)

mm/zpdesc.h +9 -0

index fd95277843d5..3c0ebdc78ff8 100644
@@ -83,4 +83,13 @@ static inline void zpdesc_put(struct zpdesc *zpdesc)
 	folio_put(zpdesc_folio(zpdesc));
 }
 
+static inline unsigned long zpdesc_pfn(struct zpdesc *zpdesc)
+{
+	return page_to_pfn(zpdesc_page(zpdesc));
+}
+
+static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)
+{
+	return page_zpdesc(pfn_to_page(pfn));
+}
 #endif

mm/zsmalloc.c +38 ~~-37~~

189 lines not on this page · raw message

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #10 lore raw

PATCH 5/20 mm/zsmalloc: convert obj_malloc() to use zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Use get_first_zpdesc/get_next_zpdesc to replace get_first_page/get_next_page. no functional change.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

mm/zsmalloc.c +8 -8

index efb1d58b3c36..137b36515acf 100644
@@ -1324,12 +1324,12 @@ EXPORT_SYMBOL_GPL(zs_huge_class_size);
 static unsigned long obj_malloc(struct zs_pool *pool,
 				struct zspage *zspage, unsigned long handle)
 {
-	int i, nr_page, offset;
+	int i, nr_zpdesc, offset;
 	unsigned long obj;
 	struct link_free *link;
 	struct size_class *class;
 
-	struct page *m_page;
+	struct zpdesc *m_zpdesc;
 	unsigned long m_offset;
 	void *vaddr;

@@ -1338,14 +1338,14 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	obj = get_freeobj(zspage);
 
 	offset = obj * class->size;
-	nr_page = offset >> PAGE_SHIFT;
+	nr_zpdesc = offset >> PAGE_SHIFT;
 	m_offset = offset_in_page(offset);
-	m_page = get_first_page(zspage);
+	m_zpdesc = get_first_zpdesc(zspage);
 
-	for (i = 0; i < nr_page; i++)
-		m_page = get_next_page(m_page);
+	for (i = 0; i < nr_zpdesc; i++)
+		m_zpdesc = get_next_zpdesc(m_zpdesc);
 
-	vaddr = kmap_atomic(m_page);
+	vaddr = zpdesc_kmap_atomic(m_zpdesc);
 	link = (struct link_free *)vaddr + m_offset / sizeof(*link);
 	set_freeobj(zspage, link->next >> OBJ_TAG_BITS);
 	if (likely(!ZsHugePage(zspage)))

@@ -1358,7 +1358,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	kunmap_atomic(vaddr);
 	mod_zspage_inuse(zspage, 1);
 
-	obj = location_to_obj(m_page, obj);
+	obj = location_to_obj(zpdesc_page(m_zpdesc), obj);
 
 	return obj;
 }

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #11 lore raw

PATCH 6/20 mm/zsmalloc: convert create_page_chain() and its users to use zpdesc

In reply to [email protected]

From: Alex Shi <[email protected]>

Introduce a few helper functions for conversion to convert create_page_chain() to use zpdesc, then use zpdesc in replace_sub_page() too.

Originally-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   |   6 +++
 mm/zsmalloc.c | 115 +++++++++++++++++++++++++++++++++-----------------
 2 files changed, 82 insertions(+), 39 deletions(-)

mm/zpdesc.h +6 -0

index 3c0ebdc78ff8..f998d65c59d6 100644
@@ -92,4 +92,10 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)
 {
 	return page_zpdesc(pfn_to_page(pfn));
 }
+
+static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
+					const struct movable_operations *mops)
+{
+	__SetPageMovable(zpdesc_page(zpdesc), mops);
+}
 #endif

mm/zsmalloc.c +76 ~~-39~~

182 lines not on this page · raw message

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #12 lore raw

PATCH 7/20 mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Convert obj_allocated(), and related helpers to take zpdesc. Also make its callers to cast (struct page *) to (struct zpdesc *) when calling them. The users will be converted gradually as there are many.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

mm/zsmalloc.c +10 ~~-10~~

index ee890d513f6f..6efaa6279687 100644
@@ -847,15 +847,15 @@ static unsigned long handle_to_obj(unsigned long handle)
 	return *(unsigned long *)handle;
 }
 
-static inline bool obj_allocated(struct page *page, void *obj,
+static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
 				 unsigned long *phandle)
 {
 	unsigned long handle;
-	struct zspage *zspage = get_zspage(page);
+	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
 
 	if (unlikely(ZsHugePage(zspage))) {
-		VM_BUG_ON_PAGE(!is_first_page(page), page);
-		handle = page->index;
+		VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc));
+		handle = zpdesc->handle;
 	} else
 		handle = *(unsigned long *)obj;

@@ -1607,18 +1607,18 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
  * return handle.
  */
 static unsigned long find_alloced_obj(struct size_class *class,
-				      struct page *page, int *obj_idx)
+				      struct zpdesc *zpdesc, int *obj_idx)
 {
 	unsigned int offset;
 	int index = *obj_idx;
 	unsigned long handle = 0;
-	void *addr = kmap_atomic(page);
+	void *addr = zpdesc_kmap_atomic(zpdesc);
 
-	offset = get_first_obj_offset(page);
+	offset = get_first_obj_offset(zpdesc_page(zpdesc));
 	offset += class->size * index;
 
 	while (offset < PAGE_SIZE) {
-		if (obj_allocated(page, addr + offset, &handle))
+		if (obj_allocated(zpdesc, addr + offset, &handle))
 			break;
 
 		offset += class->size;

@@ -1642,7 +1642,7 @@ static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage,
 	struct size_class *class = pool->size_class[src_zspage->class];
 
 	while (1) {
-		handle = find_alloced_obj(class, s_page, &obj_idx);
+		handle = find_alloced_obj(class, page_zpdesc(s_page), &obj_idx);
 		if (!handle) {
 			s_page = get_next_page(s_page);
 			if (!s_page)

@@ -1876,7 +1876,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 
 	for (addr = s_addr + offset; addr < s_addr + PAGE_SIZE;
 					addr += class->size) {
-		if (obj_allocated(page, addr, &handle)) {
+		if (obj_allocated(page_zpdesc(page), addr, &handle)) {
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #13 lore raw

PATCH 8/20 mm/zsmalloc: convert init_zspage() to use zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Replace get_first/next_page func series and kmap_atomic to new helper, no functional change.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

mm/zsmalloc.c +8 -8

index 6efaa6279687..e90e3a638068 100644
@@ -950,16 +950,16 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 {
 	unsigned int freeobj = 1;
 	unsigned long off = 0;
-	struct page *page = get_first_page(zspage);
+	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
 
-	while (page) {
-		struct page *next_page;
+	while (zpdesc) {
+		struct zpdesc *next_zpdesc;
 		struct link_free *link;
 		void *vaddr;
 
-		set_first_obj_offset(page, off);
+		set_first_obj_offset(zpdesc_page(zpdesc), off);
 
-		vaddr = kmap_atomic(page);
+		vaddr = zpdesc_kmap_atomic(zpdesc);
 		link = (struct link_free *)vaddr + off / sizeof(*link);
 
 		while ((off += class->size) < PAGE_SIZE) {

@@ -972,8 +972,8 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 		 * page, which must point to the first object on the next
 		 * page (if present)
 		 */
-		next_page = get_next_page(page);
-		if (next_page) {
+		next_zpdesc = get_next_zpdesc(zpdesc);
+		if (next_zpdesc) {
 			link->next = freeobj++ << OBJ_TAG_BITS;
 		} else {
 			/*

@@ -983,7 +983,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 			link->next = -1UL << OBJ_TAG_BITS;
 		}
 		kunmap_atomic(vaddr);
-		page = next_page;
+		zpdesc = next_zpdesc;
 		off %= PAGE_SIZE;
 	}

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #14 lore raw

PATCH 9/20 mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Rename obj_to_page() to obj_to_zpdesc() and also convert it and its user zs_free() to use zpdesc.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

mm/zsmalloc.c +5 -5

index e90e3a638068..3ef8e13fb9e6 100644
@@ -822,9 +822,9 @@ static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc,
 	*obj_idx = (obj & OBJ_INDEX_MASK);
 }
 
-static void obj_to_page(unsigned long obj, struct page **page)
+static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
 {
-	*page = pfn_to_page(obj >> OBJ_INDEX_BITS);
+	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
 }
 
 /**

@@ -1500,7 +1500,7 @@ static void obj_free(int class_size, unsigned long obj)
 void zs_free(struct zs_pool *pool, unsigned long handle)
 {
 	struct zspage *zspage;
-	struct page *f_page;
+	struct zpdesc *f_zpdesc;
 	unsigned long obj;
 	struct size_class *class;
 	int fullness;

@@ -1514,8 +1514,8 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	 */
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
-	obj_to_page(obj, &f_page);
-	zspage = get_zspage(f_page);
+	obj_to_zpdesc(obj, &f_zpdesc);
+	zspage = get_zspage(zpdesc_page(f_zpdesc));
 	class = zspage_class(pool, zspage);
 	spin_lock(&class->lock);
 	read_unlock(&pool->migrate_lock);

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #15 lore raw

PATCH 10/20 mm/zsmalloc: add zpdesc_is_isolated/zpdesc_zone helper for zs_page_migrate

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

To convert page to zpdesc in zs_page_migrate(), we added zpdesc_is_isolated/zpdesc_zone helpers. No functional change.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   | 11 +++++++++++
 mm/zsmalloc.c | 30 ++++++++++++++++--------------
 2 files changed, 27 insertions(+), 14 deletions(-)

mm/zpdesc.h +11 -0

index f998d65c59d6..2e35b8220e05 100644
@@ -98,4 +98,15 @@ static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
 {
 	__SetPageMovable(zpdesc_page(zpdesc), mops);
 }
+
+static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc)
+{
+	return PageIsolated(zpdesc_page(zpdesc));
+}
+
+static inline struct zone *zpdesc_zone(struct zpdesc *zpdesc)
+{
+	return page_zone(zpdesc_page(zpdesc));
+}
+
 #endif

mm/zsmalloc.c +16 ~~-14~~

index 3ef8e13fb9e6..251a6e63e58a 100644
@@ -1835,19 +1835,21 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	struct size_class *class;
 	struct zspage *zspage;
 	struct zpdesc *dummy;
+	struct zpdesc *newzpdesc = page_zpdesc(newpage);
+	struct zpdesc *zpdesc = page_zpdesc(page);
 	void *s_addr, *d_addr, *addr;
 	unsigned int offset;
 	unsigned long handle;
 	unsigned long old_obj, new_obj;
 	unsigned int obj_idx;
 
-	VM_BUG_ON_PAGE(!PageIsolated(page), page);
+	VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc));
 
 	/* We're committed, tell the world that this is a Zsmalloc page. */
-	__SetPageZsmalloc(newpage);
+	__SetPageZsmalloc(zpdesc_page(newzpdesc));
 
 	/* The page is locked, so this pointer must remain valid */
-	zspage = get_zspage(page);
+	zspage = get_zspage(zpdesc_page(zpdesc));
 	pool = zspage->pool;
 
 	/*

@@ -1864,30 +1866,30 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	/* the migrate_write_lock protects zpage access via zs_map_object */
 	migrate_write_lock(zspage);
 
-	offset = get_first_obj_offset(page);
-	s_addr = kmap_atomic(page);
+	offset = get_first_obj_offset(zpdesc_page(zpdesc));
+	s_addr = zpdesc_kmap_atomic(zpdesc);
 
 	/*
 	 * Here, any user cannot access all objects in the zspage so let's move.
 	 */
-	d_addr = kmap_atomic(newpage);
+	d_addr = zpdesc_kmap_atomic(newzpdesc);
 	copy_page(d_addr, s_addr);
 	kunmap_atomic(d_addr);
 
 	for (addr = s_addr + offset; addr < s_addr + PAGE_SIZE;
 					addr += class->size) {
-		if (obj_allocated(page_zpdesc(page), addr, &handle)) {
+		if (obj_allocated(zpdesc, addr, &handle)) {
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-			new_obj = (unsigned long)location_to_obj(newpage,
+			new_obj = (unsigned long)location_to_obj(zpdesc_page(newzpdesc),
 								obj_idx);
 			record_obj(handle, new_obj);
 		}
 	}
 	kunmap_atomic(s_addr);
 
-	replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page));
+	replace_sub_page(class, zspage, newzpdesc, zpdesc);
 	/*
 	 * Since we complete the data copy and set up new zspage structure,
 	 * it's okay to release migration_lock.

@@ -1896,14 +1898,14 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	spin_unlock(&class->lock);
 	migrate_write_unlock(zspage);
 
-	get_page(newpage);
-	if (page_zone(newpage) != page_zone(page)) {
-		dec_zone_page_state(page, NR_ZSPAGES);
-		inc_zone_page_state(newpage, NR_ZSPAGES);
+	zpdesc_get(newzpdesc);
+	if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) {
+		zpdesc_dec_zone_page_state(zpdesc);
+		zpdesc_inc_zone_page_state(newzpdesc);
 	}
 
 	reset_page(page);
-	put_page(page);
+	zpdesc_put(zpdesc);
 
 	return MIGRATEPAGE_SUCCESS;
 }

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #16 lore raw

PATCH 11/20 mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it

In reply to [email protected]

From: Alex Shi <[email protected]>

zpdesc.zspage matches with page.private, zpdesc.next matches with page.index. They will be reset in reset_page() wich is called prior to free base pages of a zspage.
Use zpdesc to replace page struct and rename it to reset_zpdesc(), few page helper still left since they are used too widely.

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

mm/zsmalloc.c +7 -5

index 251a6e63e58a..216f8e6950ef 100644
@@ -867,12 +867,14 @@ static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
 	return true;
 }
 
-static void reset_page(struct page *page)
+static void reset_zpdesc(struct zpdesc *zpdesc)
 {
+	struct page *page = zpdesc_page(zpdesc);
+
 	__ClearPageMovable(page);
 	ClearPagePrivate(page);
-	set_page_private(page, 0);
-	page->index = 0;
+	zpdesc->zspage = NULL;
+	zpdesc->next = NULL;
 	reset_first_obj_offset(page);
 	__ClearPageZsmalloc(page);
 }

@@ -912,7 +914,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 	do {
 		VM_BUG_ON_PAGE(!PageLocked(page), page);
 		next = get_next_page(page);
-		reset_page(page);
+		reset_zpdesc(page_zpdesc(page));
 		unlock_page(page);
 		dec_zone_page_state(page, NR_ZSPAGES);
 		put_page(page);

@@ -1904,7 +1906,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 		zpdesc_inc_zone_page_state(newzpdesc);
 	}
 
-	reset_page(page);
+	reset_zpdesc(zpdesc);
 	zpdesc_put(zpdesc);
 
 	return MIGRATEPAGE_SUCCESS;

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #17 lore raw

PATCH 12/20 mm/zsmalloc: convert __free_zspage() to use zdsesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Introduce zpdesc_is_locked() and convert __free_zspage() to use zpdesc.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   |  4 ++++
 mm/zsmalloc.c | 20 ++++++++++----------
 2 files changed, 14 insertions(+), 10 deletions(-)

mm/zpdesc.h +4 -0

index 2e35b8220e05..186ff9711ffb 100644
@@ -109,4 +109,8 @@ static inline struct zone *zpdesc_zone(struct zpdesc *zpdesc)
 	return page_zone(zpdesc_page(zpdesc));
 }
 
+static inline bool zpdesc_is_locked(struct zpdesc *zpdesc)
+{
+	return PageLocked(zpdesc_page(zpdesc));
+}
 #endif

mm/zsmalloc.c +10 ~~-10~~

index 216f8e6950ef..bf5a1b63bb17 100644
@@ -903,23 +903,23 @@ static int trylock_zspage(struct zspage *zspage)
 static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 				struct zspage *zspage)
 {
-	struct page *page, *next;
+	struct zpdesc *zpdesc, *next;
 
 	assert_spin_locked(&class->lock);
 
 	VM_BUG_ON(get_zspage_inuse(zspage));
 	VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0);
 
-	next = page = get_first_page(zspage);
+	next = zpdesc = get_first_zpdesc(zspage);
 	do {
-		VM_BUG_ON_PAGE(!PageLocked(page), page);
-		next = get_next_page(page);
-		reset_zpdesc(page_zpdesc(page));
-		unlock_page(page);
-		dec_zone_page_state(page, NR_ZSPAGES);
-		put_page(page);
-		page = next;
-	} while (page != NULL);
+		VM_BUG_ON_PAGE(!zpdesc_is_locked(zpdesc), zpdesc_page(zpdesc));
+		next = get_next_zpdesc(zpdesc);
+		reset_zpdesc(zpdesc);
+		zpdesc_unlock(zpdesc);
+		zpdesc_dec_zone_page_state(zpdesc);
+		zpdesc_put(zpdesc);
+		zpdesc = next;
+	} while (zpdesc != NULL);
 
 	cache_free_zspage(pool, zspage);

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #18 lore raw

PATCH 13/20 mm/zsmalloc: convert location_to_obj() to take zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

As all users of location_to_obj() now use zpdesc, convert location_to_obj() to take zpdesc.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

mm/zsmalloc.c +6 -7

index bf5a1b63bb17..e8af01768ae0 100644
@@ -828,15 +828,15 @@ static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
 }
 
 /**
- * location_to_obj - get obj value encoded from (<page>, <obj_idx>)
- * @page: page object resides in zspage
+ * location_to_obj - get obj value encoded from (<zpdesc>, <obj_idx>)
+ * @zpdesc: zpdesc object resides in zspage
  * @obj_idx: object index
  */
-static unsigned long location_to_obj(struct page *page, unsigned int obj_idx)
+static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx)
 {
 	unsigned long obj;
 
-	obj = page_to_pfn(page) << OBJ_INDEX_BITS;
+	obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS;
 	obj |= obj_idx & OBJ_INDEX_MASK;
 
 	return obj;

@@ -1395,7 +1395,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	kunmap_atomic(vaddr);
 	mod_zspage_inuse(zspage, 1);
 
-	obj = location_to_obj(zpdesc_page(m_zpdesc), obj);
+	obj = location_to_obj(m_zpdesc, obj);
 
 	return obj;
 }

@@ -1884,8 +1884,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-			new_obj = (unsigned long)location_to_obj(zpdesc_page(newzpdesc),
-								obj_idx);
+			new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx);
 			record_obj(handle, new_obj);
 		}
 	}

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #19 lore raw

PATCH 14/20 mm/zsmalloc: convert migrate_zspage() to use zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Use get_first_zpdesc/get_next_zpdesc to replace get_first/next_page. No functional change.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

mm/zsmalloc.c +4 -4

index e8af01768ae0..a10af49d8d08 100644
@@ -1640,14 +1640,14 @@ static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage,
 	unsigned long used_obj, free_obj;
 	unsigned long handle;
 	int obj_idx = 0;
-	struct page *s_page = get_first_page(src_zspage);
+	struct zpdesc *s_zpdesc = get_first_zpdesc(src_zspage);
 	struct size_class *class = pool->size_class[src_zspage->class];
 
 	while (1) {
-		handle = find_alloced_obj(class, page_zpdesc(s_page), &obj_idx);
+		handle = find_alloced_obj(class, s_zpdesc, &obj_idx);
 		if (!handle) {
-			s_page = get_next_page(s_page);
-			if (!s_page)
+			s_zpdesc = get_next_zpdesc(s_zpdesc);
+			if (!s_zpdesc)
 				break;
 			obj_idx = 0;
 			continue;

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #20 lore raw

PATCH 15/20 mm/zsmalloc: convert get_zspage() to take zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Now that all users except get_next_page() (which will be removed in later patch) use zpdesc, convert get_zspage() to take zpdesc instead of page.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

mm/zsmalloc.c +10 ~~-10~~

index a10af49d8d08..c136124adb10 100644
@@ -781,9 +781,9 @@ static int fix_fullness_group(struct size_class *class, struct zspage *zspage)
 	return newfg;
 }
 
-static struct zspage *get_zspage(struct page *page)
+static struct zspage *get_zspage(struct zpdesc *zpdesc)
 {
-	struct zspage *zspage = (struct zspage *)page_private(page);
+	struct zspage *zspage = zpdesc->zspage;
 
 	BUG_ON(zspage->magic != ZSPAGE_MAGIC);
 	return zspage;

@@ -791,7 +791,7 @@ static struct zspage *get_zspage(struct page *page)
 
 static struct page *get_next_page(struct page *page)
 {
-	struct zspage *zspage = get_zspage(page);
+	struct zspage *zspage = get_zspage(page_zpdesc(page));
 
 	if (unlikely(ZsHugePage(zspage)))
 		return NULL;

@@ -801,7 +801,7 @@ static struct page *get_next_page(struct page *page)
 
 static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 {
-	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
+	struct zspage *zspage = get_zspage(zpdesc);
 
 	if (unlikely(ZsHugePage(zspage)))
 		return NULL;

@@ -851,7 +851,7 @@ static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
 				 unsigned long *phandle)
 {
 	unsigned long handle;
-	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
+	struct zspage *zspage = get_zspage(zpdesc);
 
 	if (unlikely(ZsHugePage(zspage))) {
 		VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc));

@@ -1267,7 +1267,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
 	obj_to_location(obj, &zpdesc, &obj_idx);
-	zspage = get_zspage(zpdesc_page(zpdesc));
+	zspage = get_zspage(zpdesc);
 
 	/*
 	 * migration cannot move any zpages in this zspage. Here, class->lock

@@ -1317,7 +1317,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 
 	obj = handle_to_obj(handle);
 	obj_to_location(obj, &zpdesc, &obj_idx);
-	zspage = get_zspage(zpdesc_page(zpdesc));
+	zspage = get_zspage(zpdesc);
 	class = zspage_class(pool, zspage);
 	off = offset_in_page(class->size * obj_idx);

@@ -1483,7 +1483,7 @@ static void obj_free(int class_size, unsigned long obj)
 
 	obj_to_location(obj, &f_zpdesc, &f_objidx);
 	f_offset = offset_in_page(class_size * f_objidx);
-	zspage = get_zspage(zpdesc_page(f_zpdesc));
+	zspage = get_zspage(f_zpdesc);
 
 	vaddr = zpdesc_kmap_atomic(f_zpdesc);
 	link = (struct link_free *)(vaddr + f_offset);

@@ -1517,7 +1517,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
 	obj_to_zpdesc(obj, &f_zpdesc);
-	zspage = get_zspage(zpdesc_page(f_zpdesc));
+	zspage = get_zspage(f_zpdesc);
 	class = zspage_class(pool, zspage);
 	spin_lock(&class->lock);
 	read_unlock(&pool->migrate_lock);

@@ -1851,7 +1851,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	__SetPageZsmalloc(zpdesc_page(newzpdesc));
 
 	/* The page is locked, so this pointer must remain valid */
-	zspage = get_zspage(zpdesc_page(zpdesc));
+	zspage = get_zspage(zpdesc);
 	pool = zspage->pool;
 
 	/*

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #21 lore raw

PATCH 16/20 mm/zsmalloc: convert SetZsPageMovable and remove unused funcs

In reply to [email protected]

From: Alex Shi <[email protected]>

Convert SetZsPageMovable() to use zpdesc, and then remove unused funcs:get_next_page/get_first_page/is_first_page.

Originally-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zsmalloc.c | 33 +++++----------------------------
 1 file changed, 5 insertions(+), 28 deletions(-)

mm/zsmalloc.c +5 ~~-28~~

index c136124adb10..951292dbcae4 100644
@@ -470,11 +470,6 @@ static DEFINE_PER_CPU(struct mapping_area, zs_map_area) = {
 	.lock	= INIT_LOCAL_LOCK(lock),
 };
 
-static __maybe_unused int is_first_page(struct page *page)
-{
-	return PagePrivate(page);
-}
-
 static int is_first_zpdesc(struct zpdesc *zpdesc)
 {
 	return PagePrivate(zpdesc_page(zpdesc));

@@ -491,14 +486,6 @@ static inline void mod_zspage_inuse(struct zspage *zspage, int val)
 	zspage->inuse += val;
 }
 
-static inline struct page *get_first_page(struct zspage *zspage)
-{
-	struct page *first_page = zpdesc_page(zspage->first_zpdesc);
-
-	VM_BUG_ON_PAGE(!is_first_page(first_page), first_page);
-	return first_page;
-}
-
 static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
 {
 	struct zpdesc *first_zpdesc = zspage->first_zpdesc;

@@ -789,16 +776,6 @@ static struct zspage *get_zspage(struct zpdesc *zpdesc)
 	return zspage;
 }
 
-static struct page *get_next_page(struct page *page)
-{
-	struct zspage *zspage = get_zspage(page_zpdesc(page));
-
-	if (unlikely(ZsHugePage(zspage)))
-		return NULL;
-
-	return (struct page *)page->index;
-}
-
 static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 {
 	struct zspage *zspage = get_zspage(zpdesc);

@@ -1974,13 +1951,13 @@ static void init_deferred_free(struct zs_pool *pool)
 
 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage)
 {
-	struct page *page = get_first_page(zspage);
+	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
 
 	do {
-		WARN_ON(!trylock_page(page));
-		__SetPageMovable(page, &zsmalloc_mops);
-		unlock_page(page);
-	} while ((page = get_next_page(page)) != NULL);
+		WARN_ON(!zpdesc_trylock(zpdesc));
+		__zpdesc_set_movable(zpdesc, &zsmalloc_mops);
+		zpdesc_unlock(zpdesc);
+	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
 }
 #else
 static inline void zs_flush_migration(struct zs_pool *pool) { }

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #22 lore raw

PATCH 17/20 mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc

In reply to [email protected]

From: Hyeonggon Yoo <[email protected]>

Now that all users of get/set_first_obj_offset() are converted to use zpdesc, convert them to take zpdesc.

Signed-off-by Hyeonggon Yoo <[email protected]>

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   |  3 +++
 mm/zsmalloc.c | 32 ++++++++++++++++----------------
 2 files changed, 19 insertions(+), 16 deletions(-)

mm/zpdesc.h +3 -0

index 186ff9711ffb..17941774c46e 100644
@@ -14,6 +14,7 @@
  * @next:		Next zpdesc in a zspage in zsmalloc zpool
  * @handle:		For huge zspage in zsmalloc zpool
  * @zspage:		Pointer to zspage in zsmalloc
+ * @first_obj_offset:	First object offset in zsmalloc zpool
  *
  * This struct overlays struct page for now. Do not modify without a good
  * understanding of the issues.

@@ -29,6 +30,7 @@ struct zpdesc {
 		unsigned long handle;
 	};
 	struct zspage *zspage;
+	unsigned int first_obj_offset;
 };
 #define ZPDESC_MATCH(pg, zp) \
 	static_assert(offsetof(struct page, pg) == offsetof(struct zpdesc, zp))

@@ -38,6 +40,7 @@ ZPDESC_MATCH(lru, lru);
 ZPDESC_MATCH(index, next);
 ZPDESC_MATCH(index, handle);
 ZPDESC_MATCH(private, zspage);
+ZPDESC_MATCH(page_type, first_obj_offset);
 #undef ZPDESC_MATCH
 static_assert(sizeof(struct zpdesc) <= sizeof(struct page));

mm/zsmalloc.c +16 ~~-16~~

index 951292dbcae4..3ce49f0372cc 100644
@@ -496,26 +496,26 @@ static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
 
 #define FIRST_OBJ_PAGE_TYPE_MASK	0xffff
 
-static inline void reset_first_obj_offset(struct page *page)
+static inline void reset_first_obj_offset(struct zpdesc *zpdesc)
 {
-	VM_WARN_ON_ONCE(!PageZsmalloc(page));
-	page->page_type |= FIRST_OBJ_PAGE_TYPE_MASK;
+	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
+	zpdesc->first_obj_offset |= FIRST_OBJ_PAGE_TYPE_MASK;
 }
 
-static inline unsigned int get_first_obj_offset(struct page *page)
+static inline unsigned int get_first_obj_offset(struct zpdesc *zpdesc)
 {
-	VM_WARN_ON_ONCE(!PageZsmalloc(page));
-	return page->page_type & FIRST_OBJ_PAGE_TYPE_MASK;
+	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
+	return zpdesc->first_obj_offset & FIRST_OBJ_PAGE_TYPE_MASK;
 }
 
-static inline void set_first_obj_offset(struct page *page, unsigned int offset)
+static inline void set_first_obj_offset(struct zpdesc *zpdesc, unsigned int offset)
 {
 	/* With 16 bit available, we can support offsets into 64 KiB pages. */
 	BUILD_BUG_ON(PAGE_SIZE > SZ_64K);
-	VM_WARN_ON_ONCE(!PageZsmalloc(page));
+	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
 	VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK);
-	page->page_type &= ~FIRST_OBJ_PAGE_TYPE_MASK;
-	page->page_type |= offset & FIRST_OBJ_PAGE_TYPE_MASK;
+	zpdesc->first_obj_offset &= ~FIRST_OBJ_PAGE_TYPE_MASK;
+	zpdesc->first_obj_offset |= offset & FIRST_OBJ_PAGE_TYPE_MASK;
 }
 
 static inline unsigned int get_freeobj(struct zspage *zspage)

@@ -852,7 +852,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc)
 	ClearPagePrivate(page);
 	zpdesc->zspage = NULL;
 	zpdesc->next = NULL;
-	reset_first_obj_offset(page);
+	reset_first_obj_offset(zpdesc);
 	__ClearPageZsmalloc(page);
 }

@@ -936,7 +936,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 		struct link_free *link;
 		void *vaddr;
 
-		set_first_obj_offset(zpdesc_page(zpdesc), off);
+		set_first_obj_offset(zpdesc, off);
 
 		vaddr = zpdesc_kmap_atomic(zpdesc);
 		link = (struct link_free *)vaddr + off / sizeof(*link);

@@ -1593,7 +1593,7 @@ static unsigned long find_alloced_obj(struct size_class *class,
 	unsigned long handle = 0;
 	void *addr = zpdesc_kmap_atomic(zpdesc);
 
-	offset = get_first_obj_offset(zpdesc_page(zpdesc));
+	offset = get_first_obj_offset(zpdesc);
 	offset += class->size * index;
 
 	while (offset < PAGE_SIZE) {

@@ -1789,8 +1789,8 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage,
 	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
 
 	create_page_chain(class, zspage, zpdescs);
-	first_obj_offset = get_first_obj_offset(zpdesc_page(oldzpdesc));
-	set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset);
+	first_obj_offset = get_first_obj_offset(oldzpdesc);
+	set_first_obj_offset(newzpdesc, first_obj_offset);
 	if (unlikely(ZsHugePage(zspage)))
 		newzpdesc->handle = oldzpdesc->handle;
 	__zpdesc_set_movable(newzpdesc, &zsmalloc_mops);

@@ -1845,7 +1845,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	/* the migrate_write_lock protects zpage access via zs_map_object */
 	migrate_write_lock(zspage);
 
-	offset = get_first_obj_offset(zpdesc_page(zpdesc));
+	offset = get_first_obj_offset(zpdesc);
 	s_addr = zpdesc_kmap_atomic(zpdesc);
 
 	/*

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #23 lore raw

PATCH 18/20 mm/zsmalloc: introduce __zpdesc_clear_movable

In reply to [email protected]

From: Alex Shi <[email protected]>

Add a helper __zpdesc_clear_movable() for __ClearPageMovable(), and use it in callers to make code clear.

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   | 5 +++++
 mm/zsmalloc.c | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

mm/zpdesc.h +5 -0

index 17941774c46e..c6a44fe04f97 100644
@@ -102,6 +102,11 @@ static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
 	__SetPageMovable(zpdesc_page(zpdesc), mops);
 }
 
+static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc)
+{
+	__ClearPageMovable(zpdesc_page(zpdesc));
+}
+
 static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc)
 {
 	return PageIsolated(zpdesc_page(zpdesc));

mm/zsmalloc.c +1 -1

index 3ce49f0372cc..abecc8a5c566 100644
@@ -848,7 +848,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc)
 {
 	struct page *page = zpdesc_page(zpdesc);
 
-	__ClearPageMovable(page);
+	__zpdesc_clear_movable(zpdesc);
 	ClearPagePrivate(page);
 	zpdesc->zspage = NULL;
 	zpdesc->next = NULL;

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:07 UTC #24 lore raw

PATCH 19/20 mm/zsmalloc: introduce __zpdesc_clear_zsmalloc

In reply to [email protected]

From: Alex Shi <[email protected]>

Add a helper __zpdesc_clear_zsmalloc() for __ClearPageZsmalloc(), and use it in callers to make code clear.

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   | 5 +++++
 mm/zsmalloc.c | 4 ++--
 2 files changed, 7 insertions(+), 2 deletions(-)

mm/zpdesc.h +5 -0

index c6a44fe04f97..e346b39aba59 100644
@@ -107,6 +107,11 @@ static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc)
 	__ClearPageMovable(zpdesc_page(zpdesc));
 }
 
+static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc)
+{
+	__ClearPageZsmalloc(zpdesc_page(zpdesc));
+}
+
 static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc)
 {
 	return PageIsolated(zpdesc_page(zpdesc));

mm/zsmalloc.c +2 -2

index abecc8a5c566..eaf8dad04f2c 100644
@@ -853,7 +853,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc)
 	zpdesc->zspage = NULL;
 	zpdesc->next = NULL;
 	reset_first_obj_offset(zpdesc);
-	__ClearPageZsmalloc(page);
+	__zpdesc_clear_zsmalloc(zpdesc);
 }
 
 static int trylock_zspage(struct zspage *zspage)

@@ -1026,7 +1026,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 		if (!zpdesc) {
 			while (--i >= 0) {
 				zpdesc_dec_zone_page_state(zpdescs[i]);
-				__ClearPageZsmalloc(zpdesc_page(zpdescs[i]));
+				__zpdesc_clear_zsmalloc(zpdescs[i]);
 				free_zpdesc(zpdescs[i]);
 			}
 			cache_free_zspage(pool, zspage);

Signature · 1 line

2.43.0

[email protected]

author

2024-06-28 03:08 UTC #25 lore raw

PATCH 20/20 mm/zsmalloc: introduce __zpdesc_set_zsmalloc()

In reply to [email protected]

From: Alex Shi <[email protected]>

Add a helper __zpdesc_set_zsmalloc() for __SetPageZsmalloc(), and use it in callers to make code clear.

Signed-off-by Alex Shi <[email protected]>

---
 mm/zpdesc.h   | 5 +++++
 mm/zsmalloc.c | 4 ++--
 2 files changed, 7 insertions(+), 2 deletions(-)

mm/zpdesc.h +5 -0

index e346b39aba59..ef36fc3de75d 100644
@@ -107,6 +107,11 @@ static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc)
 	__ClearPageMovable(zpdesc_page(zpdesc));
 }
 
+static inline void __zpdesc_set_zsmalloc(struct zpdesc *zpdesc)
+{
+	__SetPageZsmalloc(zpdesc_page(zpdesc));
+}
+
 static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc)
 {
 	__ClearPageZsmalloc(zpdesc_page(zpdesc));

mm/zsmalloc.c +2 -2

index eaf8dad04f2c..e2a506f53957 100644
@@ -1032,7 +1032,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 			cache_free_zspage(pool, zspage);
 			return NULL;
 		}
-		__SetPageZsmalloc(zpdesc_page(zpdesc));
+		__zpdesc_set_zsmalloc(zpdesc);
 
 		zpdesc_inc_zone_page_state(zpdesc);
 		zpdescs[i] = zpdesc;

@@ -1825,7 +1825,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc));
 
 	/* We're committed, tell the world that this is a Zsmalloc page. */
-	__SetPageZsmalloc(zpdesc_page(newzpdesc));
+	__zpdesc_set_zsmalloc(newzpdesc);
 
 	/* The page is locked, so this pointer must remain valid */
 	zspage = get_zspage(zpdesc);

Signature · 1 line

2.43.0

Alex Shi

2024-06-28 07:26 UTC #26 lore raw

In reply to [email protected]

Quoted 14 lines · On 6/28/24 11:11 AM, [email protected] wrote:

From: Alex Shi <[email protected]>
According to Metthew's plan, the page descriptor will be replace by a 8 bytes mem_desc on destination purpose.
https://lore.kernel.org/lkml/[email protected]/
Here is a implement on zsmalloc to replace page descriptor by 'zpdesc', which is still overlay on struct page now. but it's a step move forward above destination.
To name the struct zpdesc instead of zsdesc, since there are still 3 zpools under zswap: zbud, z3fold, zsmalloc for now(z3fold maybe removed soon), and we could easyly extend it to other zswap.zpool in needs.

Quoted 8 lines

For all zswap.zpools, they are all using single page since often used under memory pressure. So the conversion via folio series helper is better than page's for compound_head check saving.
For now, all zpools are using some page struct members, like page.flags for PG_private/PG_locked. and list_head lru, page.mapping for page migration.
This patachset could save 123Kbyetes zsmalloc.o size.

This patchset is based on next/mm-unstable.

Thanks
Alex

Quoted 34 lines

Thanks
Alex
Alex Shi (8):
mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool mm/zsmalloc: use zpdesc in trylock_zspage/lock_zspage mm/zsmalloc: convert create_page_chain() and its users to use zpdesc mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it mm/zsmalloc: convert SetZsPageMovable and remove unused funcs mm/zsmalloc: introduce __zpdesc_clear_movable mm/zsmalloc: introduce __zpdesc_clear_zsmalloc mm/zsmalloc: introduce __zpdesc_set_zsmalloc()
Hyeonggon Yoo (12):
mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc mm/zsmalloc: add and use pfn/zpdesc seeking funcs mm/zsmalloc: convert obj_malloc() to use zpdesc mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc mm/zsmalloc: convert init_zspage() to use zpdesc mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc mm/zsmalloc: add zpdesc_is_isolated/zpdesc_zone helper for
zs_page_migrate
mm/zsmalloc: convert __free_zspage() to use zdsesc mm/zsmalloc: convert location_to_obj() to take zpdesc mm/zsmalloc: convert migrate_zspage() to use zpdesc mm/zsmalloc: convert get_zspage() to take zpdesc mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc
 mm/zpdesc.h   | 134 +++++++++++++++
 mm/zsmalloc.c | 454 +++++++++++++++++++++++++++-----------------------
 2 files changed, 384 insertions(+), 204 deletions(-)
 create mode 100644 mm/zpdesc.h

Alex Shi

2024-06-28 07:30 UTC #27 lore raw

Quoted 20 lines · Alex Shi

Quoted 18 lines · On 6/28/24 11:11 AM, [email protected] wrote:
From: Alex Shi <[email protected]>
According to Metthew's plan, the page descriptor will be replace by a 8 bytes mem_desc on destination purpose.
https://lore.kernel.org/lkml/[email protected]/
Here is a implement on zsmalloc to replace page descriptor by 'zpdesc', which is still overlay on struct page now. but it's a step move forward above destination.
To name the struct zpdesc instead of zsdesc, since there are still 3 zpools under zswap: zbud, z3fold, zsmalloc for now(z3fold maybe removed soon), and we could easyly extend it to other zswap.zpool in needs.
For all zswap.zpools, they are all using single page since often used under memory pressure. So the conversion via folio series helper is better than page's for compound_head check saving.

Quoted 6 lines

Quoted 4 lines
For now, all zpools are using some page struct members, like page.flags for PG_private/PG_locked. and list_head lru, page.mapping for page migration.
This patachset could save 123Kbyetes zsmalloc.o size.
This patchset is based on next/mm-unstable.

Sorry, just double checked, this patchset based on git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm mm-unstable branch. "4b17ce353e02 mm: optimization on page allocation when CMA enabled"

Thanks
Alex

Quoted 35 lines

Quoted 35 lines
Thanks
Alex
Alex Shi (8):
mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool mm/zsmalloc: use zpdesc in trylock_zspage/lock_zspage mm/zsmalloc: convert create_page_chain() and its users to use zpdesc mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it mm/zsmalloc: convert SetZsPageMovable and remove unused funcs mm/zsmalloc: introduce __zpdesc_clear_movable mm/zsmalloc: introduce __zpdesc_clear_zsmalloc mm/zsmalloc: introduce __zpdesc_set_zsmalloc()
Hyeonggon Yoo (12):
mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc mm/zsmalloc: add and use pfn/zpdesc seeking funcs mm/zsmalloc: convert obj_malloc() to use zpdesc mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc mm/zsmalloc: convert init_zspage() to use zpdesc mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc mm/zsmalloc: add zpdesc_is_isolated/zpdesc_zone helper for
zs_page_migrate
mm/zsmalloc: convert __free_zspage() to use zdsesc mm/zsmalloc: convert location_to_obj() to take zpdesc mm/zsmalloc: convert migrate_zspage() to use zpdesc mm/zsmalloc: convert get_zspage() to take zpdesc mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc
 mm/zpdesc.h   | 134 +++++++++++++++
 mm/zsmalloc.c | 454 +++++++++++++++++++++++++++-----------------------
 2 files changed, 384 insertions(+), 204 deletions(-)
 create mode 100644 mm/zpdesc.h