by_n compression and decompression with Intel IAA

With the introduction of the 'canned' compression algorithm [1], we see better latencies than the 'dynamic' Deflate, and a better compression ratio than 'fixed' Deflate.

When using IAA hardware accelerated 'canned' compression, we are able to take advantage of the IAA architecture and initiate independent parallel operations for both compress and decompress. In support of mTHP and large folio swap in/out, we have developed an algorithm based on 'canned' compression, called 'canned-by_n' that takes advantage of the IAA hardware that has multiple compression and decompression engines which creates parallelism for each single compress and/or decompress operation thus greatly reducing latency.

When using the 'canned-by_n' algorithm, the user provides an input buffer, an output buffer, and a parameter N. The 'canned-by_n' crypto algorithm compresses (or decompresses) a single input buffer into a single output buffer. This is done in such a way that the compress and decompress operations can be parallelized into up to N parallel operations from a single input buffer into a single output buffer.

Usage
=====

With the introduction of the 'canned-by_n' algorithm, the user would simply do the following to initiate an operation:

struct crypto_acomp *tfm;
struct acomp_req *req;
tfm = crypto_alloc_acomp("deflate-iaa-canned-by_n", 0, 0);

....

// Ignored by non 'by_n' algorithms

req->by_n = N;

err = crypto_wait_req(crypto_acomp_compress(req), &wait);

In the above example, the only new initialization for an acomp_req would be to specify the by_n number N, where N is a power of 2 and 1 <= N <= 64 (64 is the current limit but this can be changed to a greater value based on the hardware capability).

Performance
===========

'Canned-by_n' compression shows promising performance improvements when applied to recent patches pertaining to multi-sized THPs in mm-unstable (7cca940d) -- swapping out the large folios and storing them in zram as outlined in [2] and swapping them back in as large folios [3]. Our results with a simple madvise-based benchmark swapping out/in folios comprised of data folios collected from SPEC benchmarks shows an over 16x improvement in compression latency and close to 10x in decompression latency over lzo-rle on 64KB mTHPs. This translates to a greater than 10x improvement in zram write latency and 7x improvement in zram read latency. The achieved compression ratio, at 2.8 is better than that of lzo-rle. These are achieved with 'canned-by_n' compression by_n setting of 8. See table below for additional data.

With larger values of N, the latency of compression and decompression drops, due to more parallelism. Concurrently, the overheads also increase with larger N values, and start to dominate the cost after a point. Compression ratio also drops with the increased splitting with larger values of N.

Performance comparison for each 64KB folio with zram on Sapphire Rapids, whose core frequency is fixed at 2500MHz, is shown below:

+------------+-------------+---------+-------------+----------+----------+
|            | Compression | Decomp  | Compression | zram     | zram     |
| Algorithm  | latency     | latency | ratio       | write    | read     |
+------------+-------------+---------+-------------+----------+----------+
|            |       Median (ns)     |             |      Median (ns)    |
+------------+-------------+---------+-------------+----------+----------+
|            |             |         |             |          |          |
| IAA by_1   | 34,493      | 20,038  | 2.93        | 40,130   | 24,478   |
| IAA by_2   | 18,830      | 11,888  | 2.93        | 24,149   | 15,536   |
| IAA by_4   | 11,364      |  8,146  | 2.90        | 16,735   | 11,469   |
| IAA by_8   |  8,344      |  6,342  | 2.77        | 13,527   |  9,177   |
| IAA by_16  |  8,837      |  6,549  | 2.33        | 15,309   |  9,547   |
| IAA by_32  | 11,153      |  9,641  | 2.19        | 16,457   | 14,086   |
| IAA by_64  | 18,272      | 16,696  | 1.96        | 24,294   | 20,048   |
|            |             |         |             |          |          |
| lz4        | 139,190     | 33,687  | 2.40        | 144,940  | 37,312   |
|            |             |         |             |          |          |
| lzo-rle    | 138,235     | 61,055  | 2.52        | 143,666  | 64,321   |
|            |             |         |             |          |          |
| zstd       | 251,820     | 90,878  | 3.40        | 256,384  | 94,328   |
+------------+-------------+---------+-------------+----------+----------+

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/

Andre Glover (3):
crypto: Add pre_alloc and post_free callbacks for acomp algorithms
crypto: add by_n attribute to acomp_req
crypto: Add deflate-canned-byN algorithm to IAA

 crypto/acompress.c                         |  13 +
 drivers/crypto/intel/iaa/iaa_crypto.h      |   9 +
 drivers/crypto/intel/iaa/iaa_crypto_main.c | 402 ++++++++++++++++++++-
 include/crypto/acompress.h                 |   4 +
 include/crypto/internal/acompress.h        |   6 +
 5 files changed, 421 insertions(+), 13 deletions(-)

2.27.0

Add callbacks to acomp crypto algorithms that facilitate the allocation and subsequent freeing of resources required by an acomp_req before and after a series of compress and/or decompress operations.

---
 crypto/acompress.c                  | 13 +++++++++++++
 include/crypto/acompress.h          |  2 ++
 include/crypto/internal/acompress.h |  6 ++++++
 3 files changed, 21 insertions(+)

index 6fdf0ff9f3c0..873918be75cc 100644
@@ -71,9 +71,13 @@ static int crypto_acomp_init_tfm(struct crypto_tfm *tfm)
 
 	acomp->compress = alg->compress;
 	acomp->decompress = alg->decompress;
+
 	acomp->dst_free = alg->dst_free;
 	acomp->reqsize = alg->reqsize;
 
+	acomp->pre_alloc = alg->pre_alloc;
+	acomp->post_free = alg->post_free;
+
 	if (alg->exit)
 		acomp->base.exit = crypto_acomp_exit_tfm;
 

@@ -129,6 +133,12 @@ struct acomp_req *acomp_request_alloc(struct crypto_acomp *acomp)
 	struct acomp_req *req;
 
 	req = __acomp_request_alloc(acomp);
+
+	if (req && (acomp->pre_alloc && acomp->pre_alloc(req))) {
+		__acomp_request_free(req);
+		return NULL;
+	}
+
 	if (req && (tfm->__crt_alg->cra_type != &crypto_acomp_type))
 		return crypto_acomp_scomp_alloc_ctx(req);
 

@@ -144,6 +154,9 @@ void acomp_request_free(struct acomp_req *req)
 	if (tfm->__crt_alg->cra_type != &crypto_acomp_type)
 		crypto_acomp_scomp_free_ctx(req);
 
+	if (acomp->post_free)
+		acomp->post_free(req);
+
 	if (req->flags & CRYPTO_ACOMP_ALLOC_OUTPUT) {
 		acomp->dst_free(req->dst);
 		req->dst = NULL;

index 54937b615239..2b73cef2f430 100644
@@ -51,6 +51,8 @@ struct acomp_req {
 struct crypto_acomp {
 	int (*compress)(struct acomp_req *req);
 	int (*decompress)(struct acomp_req *req);
+	int (*pre_alloc)(struct acomp_req *req);
+	void (*post_free)(struct acomp_req *req);
 	void (*dst_free)(struct scatterlist *dst);
 	unsigned int reqsize;
 	struct crypto_tfm base;

index d00392d1988e..081e1cf5235f 100644
@@ -29,6 +29,10 @@
  * @exit:	Deinitialize the cryptographic transformation object. This is a
  *		counterpart to @init, used to remove various changes set in
  *		@init.
+ * @pre_alloc:	Function that performs any pre-allocation and setup that an
+ *		algorithm may require on a per req basis.
+ * @post_free:	Function that performs any post freeing that an algorithm
+ *		may require on a per req basis.
  *
  * @reqsize:	Context size for (de)compression requests
  * @base:	Common crypto API algorithm data structure

@@ -40,6 +44,8 @@ struct acomp_alg {
 	void (*dst_free)(struct scatterlist *dst);
 	int (*init)(struct crypto_acomp *tfm);
 	void (*exit)(struct crypto_acomp *tfm);
+	int (*pre_alloc)(struct acomp_req *req);
+	void (*post_free)(struct acomp_req *req);
 
 	unsigned int reqsize;
 

2.27.0

Add the 'by_n' attribute to the acomp_req. The 'by_n' attribute can be used a directive by acomp crypto algorithms for splitting compress and decompress operations into "n" separate jobs.

---
 include/crypto/acompress.h | 2 ++
 1 file changed, 2 insertions(+)

index 2b73cef2f430..c687729e1966 100644
@@ -25,6 +25,7 @@
  * @slen:	Size of the input buffer
  * @dlen:	Size of the output buffer and number of bytes produced
  * @flags:	Internal flags
+ * @by_n:	by_n setting used by acomp alg
  * @__ctx:	Start of private context data
  */
 struct acomp_req {

@@ -34,6 +35,7 @@ struct acomp_req {
 	unsigned int slen;
 	unsigned int dlen;
 	u32 flags;
+	u32 by_n;
 	void *__ctx[] CRYPTO_MINALIGN_ATTR;
 };
 

2.27.0

Hi Andre,

I am definitely in favor of the patchset idea. However, I'm not convinced that a separate by_n API is necessary. Couldn’t this functionality be handled automatically within your driver? For instance, if a large folio is detected, could it automatically apply the by_n concept?

Am I overlooking something that makes exposing the API necessary in this case?

Thanks
Barry

Hi Barry,

The 'deflate-iaa-canned' compression algorithm is fully compatible with the deflate standard. Andre's patchset introduces 'canned-by_n' as a new compression algorithm, which is not a deflate stream since it has a different header (for the by_n chunks). The same 'canned-by_n' algorithm along with the value of the acomp_req ‘by_n’ attribute would be used to compress and decompress a given input buffer.
Furthermore, with a tunable 'by_n' , the user can experiment with different values of by_n for different mTHP sizes to understand trade-offs in performance vs. compression ratio.

Thanks
Bala

When using IAA hardware accelerated 'canned' compression, we are able to take advantage of the IAA architecture and initiate independent parallel operations for both compress and decompress. In support of mTHP and large folio swap in/out, we have developed an algorithm based on 'canned' compression, called 'canned-by_n' that takes advantage of the IAA hardware that has multiple compression and decompression engines which creates parallelism for each single compress and/or decompress operation thus greatly reducing latency.

---
 drivers/crypto/intel/iaa/iaa_crypto.h      |   9 +
 drivers/crypto/intel/iaa/iaa_crypto_main.c | 402 ++++++++++++++++++++-
 2 files changed, 398 insertions(+), 13 deletions(-)

index 70abcbffd930..dd2c3056fd5d 100644
@@ -49,6 +49,9 @@
 #define DECOMP_INT_STATE_FMT1_LEN	1088
 #define DECOMP_INT_STATE_FMT2_LEN	1248
 
+#define HDR_ENTRY_SIZE			sizeof(u32)
+#define MAX_BYN_DESC			64
+
 /* Representation of IAA workqueue */
 struct iaa_wq {
 	struct list_head	list;

@@ -183,6 +186,12 @@ struct iaa_compression_ctx {
 	bool		verify_compress;
 	bool		async_mode;
 	bool		use_irq;
+	bool		by_n;
+};
+
+struct iaa_private_data {
+	struct idxd_desc		*desc[MAX_BYN_DESC];
+	int				num_comp_descs;
 };
 
 extern struct list_head iaa_devices;

2.27.0