Skip to contentSkip to conversation

mm/zswap & crypto/compress: remove a couple of memcpy

PATCH Superseded Barry Song 2 participants 7 messages Feb 17, 2024 – Feb 19, 2024

Subsystems (5)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Barry Song

author
#0 lore raw

v3 cover mm/zswap & crypto/compress: remove a couple of memcpy

From: Barry Song <[email protected]>

The patchset removes a couple of memcpy in zswap and crypto to improve zswap's performance.

Thanks for Chengming Zhou's test and perf data.
Quote from Chengming,
I just tested these three patches on my server, found improvement in the kernel build testcase on a tmpfs with zswap (lz4 + zsmalloc) enabled.

         mm-stable 501a06fe8e4c  patched
 real    1m38.028s               1m32.317s
 user    19m11.482s              18m39.439s
 sys     19m26.445s              17m5.646s

-v3:
* collect Acked-by of Yosry, Reviewed-by of Chengming
* enhance commit message with respect to Yosry's comment

Hi Andrew, Herbert,

As zswap is the direct use of this patchset and zswap benefits from this series, It is probably better for this patchset to go through Andrew's mm tree than Herbert's crypto tree if there is no objection from Herbert.

Barry Song (3):
crypto: introduce acomp_is_sleepable to expose if comp drivers might
sleep
mm/zswap: remove the memcpy if acomp is not sleepable
crypto: scompress: remove memcpy if sg_nents is 1

 crypto/acompress.c         |  8 ++++++++
 crypto/scompress.c         | 36 +++++++++++++++++++++++++++++-------
 include/crypto/acompress.h |  9 +++++++++
 mm/zswap.c                 |  6 ++++--
 4 files changed, 50 insertions(+), 9 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v3 1/3 crypto: introduce acomp_is_sleepable to expose if comp drivers might sleep

From: Barry Song <[email protected]>

Almost all CPU-based compressors/decompressors are actually synchronous though they support acomp APIs. While some hardware has hardware-based accelerators to offload CPU's work such as hisilicon and intel/qat/, their drivers are working in async mode.
Letting acomp's users know exactly if the acomp is really async will help users know if the compression and decompression procedure can sleep.
Generally speaking, async and sleepable are semantically similar but not equal. But for compress drivers, they are actually equal at least due to the below facts.
Firstly, scompress drivers - crypto/deflate.c, lz4.c, zstd.c, lzo.c etc have no sleep. Secondly, zRAM has been using these scompress drivers for years in atomic contexts, and never worried those drivers going to sleep.

---
 crypto/acompress.c         | 8 ++++++++
 include/crypto/acompress.h | 9 +++++++++
 2 files changed, 17 insertions(+)
crypto/acompress.c +8 -0
index 1c682810a484..fa15df394a4c 100644
@@ -152,6 +152,14 @@ struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type,
 }
 EXPORT_SYMBOL_GPL(crypto_alloc_acomp_node);
 
+bool acomp_is_sleepable(struct crypto_acomp *acomp)
+{
+	struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+
+	return tfm->__crt_alg->cra_type == &crypto_acomp_type;
+}
+EXPORT_SYMBOL_GPL(acomp_is_sleepable);
+
 struct acomp_req *acomp_request_alloc(struct crypto_acomp *acomp)
 {
 	struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
include/crypto/acompress.h +9 -0
index 574cffc90730..88ca33532313 100644
@@ -204,6 +204,15 @@ struct acomp_req *acomp_request_alloc(struct crypto_acomp *tfm);
  */
 void acomp_request_free(struct acomp_req *req);
 
+/**
+ * acomp_is_sleepable() -- check if an acomp is sleepable
+ *
+ * @tfm:	ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ *
+ * Return:	true if the acomp is sleepable, otherwise, false
+ */
+bool acomp_is_sleepable(struct crypto_acomp *tfm);
+
 /**
  * acomp_request_set_callback() -- Sets an asynchronous callback
  *
Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [PATCH v3 1/3] crypto: introduce acomp_is_sleepable to expose if comp drivers might sleep

On Sat, Feb 17, 2024 at 05:51:00PM +1300, Barry Song wrote:
.

Quoted 5 lines
crypto/acompress.c +0 -0
index 1c682810a484..fa15df394a4c 100644

crypto/acompress.c struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type, · line 161

Quoted 10 lines
crypto/acompress.c +7 -0
@@ -152,6 +152,14 @@ struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type,
 }
 EXPORT_SYMBOL_GPL(crypto_alloc_acomp_node);
 
+bool acomp_is_sleepable(struct crypto_acomp *acomp)
+{
+	struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+
+	return tfm->__crt_alg->cra_type == &crypto_acomp_type;
+}
+EXPORT_SYMBOL_GPL(acomp_is_sleepable);

Just because something is of acomp_type it doesn't mean that it's async. You should be testing the algorithm flags.

So introduce a helper crypto_acomp_get_flags (see the similar helper crypto_skcipher_get_flags) and test it against CRYPTO_ALG_ASYNC.

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
author
#3 lore raw

Re: [PATCH v3 1/3] crypto: introduce acomp_is_sleepable to expose if comp drivers might sleep

Quoted 11 lines · Herbert Xu

On Sat, Feb 17, 2024 at 05:51:00PM +1300, Barry Song wrote:
.

Quoted 8 lines
crypto/acompress.c +0 -0
index 1c682810a484..fa15df394a4c 100644
@@ -152,6 +152,14 @@ struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type,
 }
 EXPORT_SYMBOL_GPL(crypto_alloc_acomp_node);
 
Quoted 10 lines

crypto/acompress.c struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type, · line 161

Quoted 7 lines
crypto/acompress.c +7 -0
+bool acomp_is_sleepable(struct crypto_acomp *acomp)
+{
+     struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+
+     return tfm->__crt_alg->cra_type == &crypto_acomp_type;
+}
+EXPORT_SYMBOL_GPL(acomp_is_sleepable);

Just because something is of acomp_type it doesn't mean that it's async. You should be testing the algorithm flags.

I guess I got your point, drivers using acomp framework might actually be SYNC if they don't set CRYPTO_ALG_ASYNC.

Quoted 3 lines

So introduce a helper crypto_acomp_get_flags (see the similar helper crypto_skcipher_get_flags) and test it against CRYPTO_ALG_ASYNC.

On the other hand, some drivers which are actually ASYNC, are lacking CRYPTO_ALG_ASYNC for example:

(quoted diff) +0 -0

b/drivers/crypto/hisilicon/zip/zip_crypto.c
index c650c741a18d..94e2d66b04b6 100644

drivers/crypto/hisilicon/zip/zip_crypto.c +1 -0
@@ -591,6 +591,7 @@ static struct acomp_alg hisi_zip_acomp_deflate = {
        .base                   = {
                .cra_name               = "deflate",
                .cra_driver_name        = "hisi-deflate-acomp",
+               .cra_flags              = CRYPTO_ALG_ASYNC,
                .cra_module             = THIS_MODULE,
                .cra_priority           = HZIP_ALG_PRIORITY,
                .cra_ctxsize            = sizeof(struct hisi_zip_ctx),
(quoted diff) +0 -0

b/drivers/crypto/intel/iaa/iaa_crypto_main.c
index dfd3baf0a8d8..91adf9d76a2e 100644

drivers/crypto/intel/iaa/iaa_crypto_main.c +1 -0
@@ -1916,6 +1916,7 @@ static struct acomp_alg iaa_acomp_fixed_deflate = {
        .base                   = {
                .cra_name               = "deflate",
                .cra_driver_name        = "deflate-iaa",
+               .cra_flags              = CRYPTO_ALG_ASYNC,
                .cra_ctxsize            = sizeof(struct iaa_compression_ctx),
                .cra_module             = THIS_MODULE,
                .cra_priority           = IAA_ALG_PRIORITY,
include/crypto/acompress.h +0 -0
index 574cffc90730..5b10bd075a07 100644
@@ -160,6 +160,11 @@ static inline void acomp_request_set_tfm(struct

acomp_req *req,

        req->base.tfm = crypto_acomp_tfm(tfm);

}

+static inline u32 crypto_acomp_get_flags(struct crypto_acomp *tfm) +{
+ return crypto_tfm_get_flags(crypto_acomp_tfm(tfm));
+}
+
static inline struct crypto_acomp *crypto_acomp_reqtfm(struct acomp_req *req)

 {
        return __crypto_acomp_tfm(req->base.tfm);

Herbert, Is the above code what you would prefer?

Quoted 6 lines

Cheers,
--
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Thanks
Barry

author
#4 lore raw

Re: [PATCH v3 1/3] crypto: introduce acomp_is_sleepable to expose if comp drivers might sleep

Quoted 65 lines · Barry Song
Quoted 11 lines · Herbert Xu

On Sat, Feb 17, 2024 at 05:51:00PM +1300, Barry Song wrote:
.

Quoted 8 lines
crypto/acompress.c +0 -0
index 1c682810a484..fa15df394a4c 100644
@@ -152,6 +152,14 @@ struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type,
 }
 EXPORT_SYMBOL_GPL(crypto_alloc_acomp_node);
 
Quoted 10 lines

crypto/acompress.c struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type, · line 161

Quoted 7 lines
crypto/acompress.c +7 -0
+bool acomp_is_sleepable(struct crypto_acomp *acomp)
+{
+     struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+
+     return tfm->__crt_alg->cra_type == &crypto_acomp_type;
+}
+EXPORT_SYMBOL_GPL(acomp_is_sleepable);

Just because something is of acomp_type it doesn't mean that it's async. You should be testing the algorithm flags.

I guess I got your point, drivers using acomp framework might actually be SYNC if they don't set CRYPTO_ALG_ASYNC.

Quoted 3 lines

So introduce a helper crypto_acomp_get_flags (see the similar helper crypto_skcipher_get_flags) and test it against CRYPTO_ALG_ASYNC.

On the other hand, some drivers which are actually ASYNC, are lacking CRYPTO_ALG_ASYNC for example:

drivers/crypto/hisilicon/zip/zip_crypto.c +1 -0
index c650c741a18d..94e2d66b04b6 100644
@@ -591,6 +591,7 @@ static struct acomp_alg hisi_zip_acomp_deflate = {
        .base                   = {
                .cra_name               = "deflate",
                .cra_driver_name        = "hisi-deflate-acomp",
+               .cra_flags              = CRYPTO_ALG_ASYNC,
                .cra_module             = THIS_MODULE,
                .cra_priority           = HZIP_ALG_PRIORITY,
                .cra_ctxsize            = sizeof(struct hisi_zip_ctx),
drivers/crypto/intel/iaa/iaa_crypto_main.c +1 -0
index dfd3baf0a8d8..91adf9d76a2e 100644
@@ -1916,6 +1916,7 @@ static struct acomp_alg iaa_acomp_fixed_deflate = {
        .base                   = {
                .cra_name               = "deflate",
                .cra_driver_name        = "deflate-iaa",
+               .cra_flags              = CRYPTO_ALG_ASYNC,
                .cra_ctxsize            = sizeof(struct iaa_compression_ctx),
                .cra_module             = THIS_MODULE,
                .cra_priority           = IAA_ALG_PRIORITY,
include/crypto/acompress.h +0 -0
index 574cffc90730..5b10bd075a07 100644

include/crypto/acompress.h static inline void acomp_request_set_tfm(struct · line 167

Quoted 8 lines
include/crypto/acompress.h +4 -0
@@ -160,6 +160,11 @@ static inline void acomp_request_set_tfm(struct
 acomp_req *req,
        req->base.tfm = crypto_acomp_tfm(tfm);
 }
 
+static inline u32 crypto_acomp_get_flags(struct crypto_acomp *tfm)
+{
+       return crypto_tfm_get_flags(crypto_acomp_tfm(tfm));
+}

This seems to be wrong. we are expecting cra_flags not crt_flags. should be the below?

include/crypto/acompress.h +0 -0
index 574cffc90730..07bd8f6bc79a 100644
@@ -160,6 +160,11 @@ static inline void acomp_request_set_tfm(struct

acomp_req *req,

        req->base.tfm = crypto_acomp_tfm(tfm);

}

+static inline u32 crypto_acomp_get_alg_flags(struct crypto_acomp *tfm) +{
+ return crypto_tfm_alg_flags(crypto_acomp_tfm(tfm));
+}
+
static inline struct crypto_acomp *crypto_acomp_reqtfm(struct acomp_req *req)

 {
        return __crypto_acomp_tfm(req->base.tfm);
include/linux/crypto.h +0 -0
index b164da5e129e..811bfaf8b6f8 100644
@@ -467,6 +467,11 @@ static inline unsigned int

crypto_tfm_alg_blocksize(struct crypto_tfm *tfm)

        return tfm->__crt_alg->cra_blocksize;

}

+static inline unsigned int crypto_tfm_alg_flags(struct crypto_tfm *tfm) +{
+ return tfm->__crt_alg->cra_flags;
+}
+
static inline unsigned int crypto_tfm_alg_alignmask(struct crypto_tfm *tfm)

 {
        return tfm->__crt_alg->cra_alignmask;
Quoted 6 lines
include/crypto/acompress.h +1 -0
+
 static inline struct crypto_acomp *crypto_acomp_reqtfm(struct acomp_req *req)
 {
        return __crypto_acomp_tfm(req->base.tfm);
Quoted 9 lines

Herbert, Is the above code what you would prefer?

Quoted 6 lines

Cheers,
--
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Thanks
Barry

author
#5 lore raw

PATCH v3 2/3 mm/zswap: remove the memcpy if acomp is not sleepable

In reply to Barry Song

From: Barry Song <[email protected]>

Most compressors are actually CPU-based and won't sleep during compression and decompression. We should remove the redundant memcpy for them.

---
 mm/zswap.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
mm/zswap.c +4 -2
index 350dd2fc8159..6319d2281020 100644
@@ -168,6 +168,7 @@ struct crypto_acomp_ctx {
 	struct crypto_wait wait;
 	u8 *buffer;
 	struct mutex mutex;
+	bool is_sleepable;
 };
 
 /*
@@ -716,6 +717,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 		goto acomp_fail;
 	}
 	acomp_ctx->acomp = acomp;
+	acomp_ctx->is_sleepable = acomp_is_sleepable(acomp);
 
 	req = acomp_request_alloc(acomp_ctx->acomp);
 	if (!req) {
@@ -1368,7 +1370,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page)
 	mutex_lock(&acomp_ctx->mutex);
 
 	src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO);
-	if (!zpool_can_sleep_mapped(zpool)) {
+	if (acomp_ctx->is_sleepable && !zpool_can_sleep_mapped(zpool)) {
 		memcpy(acomp_ctx->buffer, src, entry->length);
 		src = acomp_ctx->buffer;
 		zpool_unmap_handle(zpool, entry->handle);
@@ -1382,7 +1384,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page)
 	BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE);
 	mutex_unlock(&acomp_ctx->mutex);
 
-	if (zpool_can_sleep_mapped(zpool))
+	if (!acomp_ctx->is_sleepable || zpool_can_sleep_mapped(zpool))
 		zpool_unmap_handle(zpool, entry->handle);
 }
 
Signature · 1 line
2.34.1
author
#6 lore raw

PATCH v3 3/3 crypto: scompress: remove memcpy if sg_nents is 1

In reply to Barry Song

From: Barry Song <[email protected]>

while sg_nents is 1 which is always true for the current kernel as the only user - zswap is the case, we should remove two big memcpy.

---
 crypto/scompress.c | 36 +++++++++++++++++++++++++++++-------
 1 file changed, 29 insertions(+), 7 deletions(-)
crypto/scompress.c +29 -7
index b108a30a7600..50a487eac792 100644
@@ -117,6 +117,7 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
 	struct crypto_scomp *scomp = *tfm_ctx;
 	void **ctx = acomp_request_ctx(req);
 	struct scomp_scratch *scratch;
+	void *src, *dst;
 	unsigned int dlen;
 	int ret;
 
@@ -134,13 +135,25 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
 	scratch = raw_cpu_ptr(&scomp_scratch);
 	spin_lock(&scratch->lock);
 
-	scatterwalk_map_and_copy(scratch->src, req->src, 0, req->slen, 0);
+	if (sg_nents(req->src) == 1) {
+		src = kmap_local_page(sg_page(req->src)) + req->src->offset;
+	} else {
+		scatterwalk_map_and_copy(scratch->src, req->src, 0,
+					 req->slen, 0);
+		src = scratch->src;
+	}
+
+	if (req->dst && sg_nents(req->dst) == 1)
+		dst = kmap_local_page(sg_page(req->dst)) + req->dst->offset;
+	else
+		dst = scratch->dst;
+
 	if (dir)
-		ret = crypto_scomp_compress(scomp, scratch->src, req->slen,
-					    scratch->dst, &req->dlen, *ctx);
+		ret = crypto_scomp_compress(scomp, src, req->slen,
+					    dst, &req->dlen, *ctx);
 	else
-		ret = crypto_scomp_decompress(scomp, scratch->src, req->slen,
-					      scratch->dst, &req->dlen, *ctx);
+		ret = crypto_scomp_decompress(scomp, src, req->slen,
+					      dst, &req->dlen, *ctx);
 	if (!ret) {
 		if (!req->dst) {
 			req->dst = sgl_alloc(req->dlen, GFP_ATOMIC, NULL);
@@ -152,10 +165,19 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
 			ret = -ENOSPC;
 			goto out;
 		}
-		scatterwalk_map_and_copy(scratch->dst, req->dst, 0, req->dlen,
-					 1);
+		if (dst == scratch->dst) {
+			scatterwalk_map_and_copy(scratch->dst, req->dst, 0,
+						 req->dlen, 1);
+		} else {
+			flush_dcache_page(sg_page(req->dst));
+		}
 	}
 out:
+	if (src != scratch->src)
+		kunmap_local(src);
+	if (dst != scratch->dst)
+		kunmap_local(dst);
+
 	spin_unlock(&scratch->lock);
 	return ret;
 }
Signature · 1 line
2.34.1