Skip to content

RFC crypto: consolidate and clean up compression APIs

Ard Biesheuvel linux-block linux-crypto linux-mtd linuxppc-dev lkml netdev oe-kbuild-all 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest21 patches 23 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/21crypto: scomp - Revert "add support for deflate rfc1950 (zlib)"crypto/deflate.c crypto/testmgr.c crypto/testmgr.h+18 -1267 replies
2/21crypto: qat - Drop support for allocating destination buffersdrivers/crypto/intel/qat/qat_common/qat_bl.c drivers/crypto/intel/qat/qat_common/qat_bl.h drivers/crypto/intel/qat/qat_common/qat_comp_algs.c +1 more+1 -2600 replies
3/21crypto: acompress - Drop destination scatterlist allocation featurecrypto/acompress.c crypto/scompress.c crypto/testmgr.c +1 more+4 -610 replies
4/21net: ipcomp: Migrate to acomp API from deprecated comp APIinclude/crypto/acompress.h include/net/ipcomp.h net/xfrm/xfrm_algo.c +1 more+79 -441 reply
5/21ubifs: Pass worst-case buffer size to compression routinesfs/ubifs/journal.c+2 -05 replies
6/21ubifs: Avoid allocating buffer space unnecessarilyfs/ubifs/journal.c+8 -80 replies
7/21ubifs: Migrate to acomp compression APIfs/ubifs/compress.c fs/ubifs/file.c fs/ubifs/journal.c +1 more+90 -511 reply
8/21zram: Migrate to acomp compression APIdrivers/block/zram/zcomp.c drivers/block/zram/zcomp.h drivers/block/zram/zram_drv.c+57 -291 reply
9/21crypto: nx - Migrate to scomp APIdrivers/crypto/nx/nx-842.c drivers/crypto/nx/nx-842.h drivers/crypto/nx/nx-common-powernv.c +1 more+57 -530 replies
10/21crypto: 842 - drop obsolete 'comp' implementationcrypto/842.c+1 -620 replies
11/21crypto: deflate - drop obsolete 'comp' implementationcrypto/deflate.c+1 -570 replies
12/21crypto: lz4 - drop obsolete 'comp' implementationcrypto/lz4.c+1 -600 replies
13/21crypto: lz4hc - drop obsolete 'comp' implementationcrypto/lz4hc.c+1 -620 replies
14/21crypto: lzo-rle - drop obsolete 'comp' implementationcrypto/lzo-rle.c+1 -590 replies
15/21crypto: lzo - drop obsolete 'comp' implementationcrypto/lzo.c+1 -590 replies
16/21crypto: zstd - drop obsolete 'comp' implementationcrypto/zstd.c+1 -550 replies
17/21crypto: cavium/zip - drop obsolete 'comp' implementationdrivers/crypto/cavium/zip/zip_crypto.c drivers/crypto/cavium/zip/zip_crypto.h drivers/crypto/cavium/zip/zip_main.c+1 -990 replies
18/21crypto: compress_null - drop obsolete 'comp' implementationcrypto/crypto_null.c crypto/testmgr.c+5 -290 replies
19/21crypto: remove obsolete 'comp' compression APIDocumentation/crypto/architecture.rst crypto/Makefile crypto/api.c +6 more+12 -2441 reply
20/21crypto: deflate - implement acomp API directlycrypto/deflate.c include/crypto/scatterwalk.h+113 -2042 replies
21/21crypto: scompress - Drop the use of per-cpu scratch bufferscrypto/scompress.c include/crypto/internal/scompress.h+76 -850 replies

Reviews trailers per patch

1 of 21 patches has a Reviewed-by or Acked-by.

PatchZhihao Cheng
1/21
2/21
3/21
4/21
5/21R-b R-b
6/21
7/21
8/21
9/21
10/21
11/21
12/21
13/21
14/21
15/21
16/21
17/21
18/21
19/21
20/21
21/21

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 00/21] crypto: consolidate and clean up compression APIs

This series is presented as an RFC, because I haven't quite convinced myself that the acomp API really needs both scatterlists and request objects to encapsulate the in- and output buffers, and perhaps there are more drastic simplifications that we might consider.

However, the current situation with comp, scomp and acomp APIs is definitely something that needs cleaning up, and so I implemented this series under the working assumption that we will keep the current acomp semantics wrt scatterlists and request objects.

Patch #1 drops zlib-deflate support in software, along with the test cases we have for it. This has no users and should have never been added.

Patch #2 removes the support for on-the-fly allocation of destination buffers and scatterlists from the Intel QAT driver. This is never used, and not even implemented by all drivers (the HiSilicon ZIP driver does not support it). The diffstat of this patch makes a good case why the caller should be in charge of allocating the memory, not the driver.

Patch #3 removes this on-the-fly allocation from the core acomp API.

Patch #4 does a minimal conversion of IPcomp to the acomp API.

Patch #5 and #6 are independent UBIFS fixes for things I ran into while working on patch #7.

Patch #7 converts UBIFS to the acomp API.

Patch #8 converts the zram block driver to the acomp API.

Patches #9 to #19 remove the existing 'comp' API implementations as well as the core plumbing, now that all clients of the API have been converted. (Note that pstore stopped using the 'comp' API as well, but these changes are already queued elsewhere)

Patch #20 converts the generic deflate compression driver to the acomp API, so that it can natively operate on discontiguous buffers, rather than requiring scratch buffers. This is the only IPcomp compression algorithm we actually implement in software in the kernel, and this conversion could help IPcomp if we decide to convert it further, and remove the code that 'linearizes' SKBs in order to present them to the compression API as a contiguous range.

Patch #21 converts the acomp-to-scomp adaptation layer so it no longer requires per-CPU scratch buffers. This takes advantage of the fact that all existing users of the acomp API pass contiguous memory regions, and so scratch buffers are only needed in exceptional cases, and can be allocated and deallocated on the fly. This removes the need for preallocated per-CPU scratch buffers that can easily add up to tens of megabytes on modern systems with high core counts and SMT.

These changes have been build tested and only lightly runtime tested. In particular, I haven't performed any thorough testing on the acomp conversions of IPcomp, UBIFS and ZRAM. Any hints on which respective methods and test cases to use here are highly appreciated.

Cc Herbert Xu <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Kees Cook <[email protected]>
Cc Haren Myneni <[email protected]>
Cc Nick Terrell <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Sergey Senozhatsky <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Giovanni Cabiddu <[email protected]>
Cc Richard Weinberger <[email protected]>
Cc David Ahern <[email protected]>
Cc Eric Dumazet <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Paolo Abeni <[email protected]>
Cc Steffen Klassert <[email protected]>

Ard Biesheuvel (21):
crypto: scomp - Revert "add support for deflate rfc1950 (zlib)"
crypto: qat - Drop support for allocating destination buffers
crypto: acompress - Drop destination scatterlist allocation feature
net: ipcomp: Migrate to acomp API from deprecated comp API
ubifs: Pass worst-case buffer size to compression routines
ubifs: Avoid allocating buffer space unnecessarily
ubifs: Migrate to acomp compression API
zram: Migrate to acomp compression API
crypto: nx - Migrate to scomp API
crypto: 842 - drop obsolete 'comp' implementation
crypto: deflate - drop obsolete 'comp' implementation
crypto: lz4 - drop obsolete 'comp' implementation
crypto: lz4hc - drop obsolete 'comp' implementation
crypto: lzo-rle - drop obsolete 'comp' implementation
crypto: lzo - drop obsolete 'comp' implementation
crypto: zstd - drop obsolete 'comp' implementation
crypto: cavium/zip - drop obsolete 'comp' implementation
crypto: compress_null - drop obsolete 'comp' implementation
crypto: remove obsolete 'comp' compression API
crypto: deflate - implement acomp API directly
crypto: scompress - Drop the use of per-cpu scratch buffers

 Documentation/crypto/architecture.rst               |   2 -
 crypto/842.c                                        |  63 +---
 crypto/Makefile                                     |   2 +-
 crypto/acompress.c                                  |   6 -
 crypto/api.c                                        |   4 -
 crypto/compress.c                                   |  32 --
 crypto/crypto_null.c                                |  31 +-
 crypto/crypto_user_base.c                           |  16 -
 crypto/crypto_user_stat.c                           |   4 -
 crypto/deflate.c                                    | 386 ++++++--------------
 crypto/lz4.c                                        |  61 +---
 crypto/lz4hc.c                                      |  63 +---
 crypto/lzo-rle.c                                    |  60 +--
 crypto/lzo.c                                        |  60 +--
 crypto/proc.c                                       |   3 -
 crypto/scompress.c                                  | 169 ++++-----
 crypto/testmgr.c                                    | 184 +---------
 crypto/testmgr.h                                    |  75 ----
 crypto/zstd.c                                       |  56 +--
 drivers/block/zram/zcomp.c                          |  67 +++-
 drivers/block/zram/zcomp.h                          |   7 +-
 drivers/block/zram/zram_drv.c                       |  12 +-
 drivers/crypto/cavium/zip/zip_crypto.c              |  40 --
 drivers/crypto/cavium/zip/zip_crypto.h              |  10 -
 drivers/crypto/cavium/zip/zip_main.c                |  50 +--
 drivers/crypto/intel/qat/qat_common/qat_bl.c        | 159 --------
 drivers/crypto/intel/qat/qat_common/qat_bl.h        |   6 -
 drivers/crypto/intel/qat/qat_common/qat_comp_algs.c |  86 +----
 drivers/crypto/intel/qat/qat_common/qat_comp_req.h  |  10 -
 drivers/crypto/nx/nx-842.c                          |  34 +-
 drivers/crypto/nx/nx-842.h                          |  14 +-
 drivers/crypto/nx/nx-common-powernv.c               |  30 +-
 drivers/crypto/nx/nx-common-pseries.c               |  32 +-
 fs/ubifs/compress.c                                 |  61 +++-
 fs/ubifs/file.c                                     |  46 +--
 fs/ubifs/journal.c                                  |  33 +-
 fs/ubifs/ubifs.h                                    |  15 +-
 include/crypto/acompress.h                          |  21 +-
 include/crypto/internal/scompress.h                 |   2 -
 include/crypto/scatterwalk.h                        |   2 +-
 include/linux/crypto.h                              |  49 +--
 include/net/ipcomp.h                                |   4 +-
 net/xfrm/xfrm_algo.c                                |   7 +-
 net/xfrm/xfrm_ipcomp.c                              | 107 ++++--
 44 files changed, 502 insertions(+), 1679 deletions(-)
 delete mode 100644 crypto/compress.c
Signature · 1 line
2.39.2