Skip to contentSkip to conversation

Inline Encryption Support

PATCH Applied Satya Tangirala 6 participants 27 messages May 14, 2020 – Jun 3, 2020

Subsystems (8)
Documentation VFS SCSI Block layer +4 more subsystems
lore
Share & tools
lore
#0 lore raw

v13 cover Inline Encryption Support

This patch series adds support for Inline Encryption to the block layer, UFS, fscrypt, f2fs and ext4. It has been rebased onto linux-block/for-next.

Note that the patches in this series for the block layer (i.e. patches 1, 2, 3, 4 and 5) can be applied independently of the subsequent patches in this series.

Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device. Inline Encryption hardware has become increasingly common, and we want to support it in the kernel.

Inline Encryption hardware implementations often function around the concept of a limited number of "keyslots", which can hold an encryption context each. The storage device can be directed to en/decrypt any particular request with the encryption context stored in any particular keyslot.

Patch 1 documents the whole series.

Patch 2 introduces a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that blk-crypto (introduced in Patch 3), will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.

Patch 3 adds the block layer changes for inline encryption support. It introduces struct bio_crypt_ctx, and a ptr to one in struct bio, which allows struct bio to represent an encryption context that can be passed down the storage stack from the filesystem layer to the storage driver.

Patch 4 precludes inline encryption support in a device whenever it supports blk-integrity, because there is currently no known hardware that supports both features, and it is not completely straightfoward to support both of them properly, and doing it improperly might result in leaks of information about the plaintext.

Patch 5 introduces blk-crypto-fallback - a kernel crypto API fallback for blk-crypto to use when inline encryption hardware isn't present. This allows filesystems to specify encryption contexts for bios without having to worry about whether the underlying hardware has inline encryption support, and allows for testing without real hardware inline encryption support. This fallback is separately configurable from blk-crypto, and can be disabled if desired while keeping inline encryption support. It may also be possible to remove file content en/decryption from fscrypt and simply use blk-crypto-fallback in a future patch. For more details on blk-crypto and the fallback, refer to Documentation/block/inline-encryption.rst.

Patches 6-8 add support for inline encryption into the UFS driver according to the JEDEC UFS HCI v2.1 specification. Inline encryption support for other drivers (like eMMC) may be added in the same way - the device driver should set up a Keyslot Manager in the device's request_queue (refer to the UFS crypto additions in ufshcd-crypto.c and ufshcd.c for an example).

Patch 9 adds the SB_INLINECRYPT mount flag to the fs layer, which filesystems must set to indicate that they want to use blk-crypto for en/decryption of file contents.

Patch 10 adds support to fscrypt - to use inline encryption with fscrypt, the filesystem must be mounted with '-o inlinecrypt' - when this option is specified, the contents of any AES-256-XTS encrypted file will be encrypted using blk-crypto.

Patches 11 and 12 add support to f2fs and ext4 respectively, so that we have a complete stack that can make use of inline encryption.

The patches were tested running kvm-xfstests, by specifying the introduced "inlinecrypt" mount option, so that en/decryption happens with the blk-crypto fallback. The patches were also tested on a Pixel 4 with UFS hardware that has support for inline encryption.

There have been a few patch sets addressing Inline Encryption Support in the past. Briefly, this patch set differs from those as follows:

1) "crypto: qce: ice: Add support for Inline Crypto Engine" is specific to certain hardware, while our patch set's Inline Encryption support for UFS is implemented according to the JEDEC UFS specification.

2) "scsi: ufs: UFS Host Controller crypto changes" registers inline encryption support as a kernel crypto algorithm. Our patch views inline encryption as being fundamentally different from a generic crypto provider (in that inline encryption is tied to a device), and so does not use the kernel crypto API to represent inline encryption hardware.

3) "scsi: ufs: add real time/inline crypto support to UFS HCD" requires the device mapper to work - our patch does not.

Changes v12 => v13:
- Updated docs
- Minor cleanups
- rebased onto linux-block/for-next

Changes v11 => v12:
- Inlined some fscrypt functions
- Minor cleanups and improved comments

Changes v10 => v11:
- We now allocate a new bio_crypt_ctx for each request instead of pulling and reusing the one in the bio inserted into the request. The bio_crypt_ctx of a bio is freed after the bio is ended.
- Make each blk_ksm_keyslot store a pointer to the blk_crypto_key instead of a copy of the blk_crypto_key, so that each blk_crypto_key will have its own keyslot. We also won't need to compute the siphash for a blk_crypto_key anymore.
- Minor cleanups

Changes v9 => v10:
- Incorporate Eric's fix for allowing en/decryption to happen as usual via fscrypt in the case that hardware doesn't support the desired crypto configuration, but blk-crypto-fallback is disabled. (Introduce struct blk_crypto_config and blk_crypto_config_supported for fscrypt to call, to check that either blk-crypto-fallback is enabled or the device supports the crypto configuration).
- Update docs
- Lots of cleanups

Changes v8 => v9:
- Don't open code bio_has_crypt_ctx into callers of blk-crypto functions.
- Lots of cleanups

Changes v7 => v8:
- Pass a struct blk_ksm_keyslot * around instead of slot numbers which simplifies some functions and passes around arguments with better types
- Make bios with no encryption context avoid making calls into blk-crypto by checking for the presence of bi_crypt_context before making the call
- Make blk-integrity preclude inline encryption support at probe time
- Many many cleanups

Changes v6 => v7:
- Keyslot management is now done on a per-request basis rather than a per-bio basis.
- Storage drivers can now specify the maximum number of bytes they can accept for the data unit number (DUN) for each crypto algorithm, and upper layers can specify the minimum number of bytes of DUN they want with the blk_crypto_key they send with the bio - a driver is only considered to support a blk_crypto_key if the driver supports at least as many DUN bytes as the upper layer wants. This is necessary because storage drivers may not support as many bytes as the algorithm specification dictates (for e.g. UFS only supports 8 byte DUNs for AES-256-XTS, even though the algorithm specification says DUNs are 16 bytes long).
- Introduce SB_INLINECRYPT to keep track of whether inline encryption is enabled for a filesystem (instead of using an fscrypt_operation).
- Expose keyslot manager declaration and embed it within ufs_hba to clean up code.
- Make blk-crypto preclude blk-integrity.
- Some bug fixes
- Introduce UFSHCD_QUIRK_BROKEN_CRYPTO for UFS drivers that don't support inline encryption (yet)

Changes v5 => v6:
- Blk-crypto's kernel crypto API fallback is no longer restricted to 8-byte DUNs. It's also now separately configurable from blk-crypto, and can be disabled entirely, while still allowing the kernel to use inline encryption hardware. Further, struct bio_crypt_ctx takes up less space, and no longer contains the information needed by the crypto API fallback - the fallback allocates the required memory when necessary.
- Blk-crypto now supports all file content encryption modes supported by fscrypt.
- Fixed bio merging logic in blk-merge.c
- Fscrypt now supports inline encryption with the direct key policy, since blk-crypto now has support for larger DUNs.
- Keyslot manager now uses a hashtable to lookup which keyslot contains any particular key (thanks Eric!)
- Fscrypt support for inline encryption now handles filesystems with multiple underlying block devices (thanks Eric!)
- Numerous cleanups

Changes v4 => v5:
- The fscrypt patch has been separated into 2. The first adds support for the IV_INO_LBLK_64 policy (which was called INLINE_CRYPT_OPTIMIZED in past versions of this series). This policy is now purely an on disk format, and doesn't dictate whether blk-crypto is used for file content encryption or not. Instead, this is now decided based on the "inlinecrypt" mount option.
- Inline crypto key eviction is now handled by blk-crypto instead of fscrypt.
- More refactoring.

Changes v3 => v4:
- Fixed the issue with allocating crypto_skcipher in blk_crypto_keyslot_program.
- bio_crypto_alloc_ctx is now mempool backed.
- In f2fs, a bio's bi_crypt_context is now set up when the bio is allocated, rather than just before the bio is submitted - this fixes bugs in certain cases, like when an encrypted block is being moved without decryption.
- Lots of refactoring and cleanup of blk-crypto - thanks Eric!

Changes v2 => v3:
- Overhauled keyslot manager's get keyslot logic and optimized LRU.
- Block crypto en/decryption fallback now supports data unit sizes that divide the bvec length (instead of requiring each bvec's length to be the same as the data unit size).
- fscrypt master key is now keyed additionally by super_block and ci_ctfm != NULL.
- all references of "hw encryption" are replaced by inline encryption.
- address various other review comments from Eric.

Changes v1 => v2:
- Block layer and UFS changes are split into 3 patches each.
- We now only have a ptr to a struct bio_crypt_ctx in struct bio, instead of the struct itself.
- struct bio_crypt_ctx no longer has flags.
- blk-crypto now correctly handles the case when it fails to init (because of insufficient memory), but kernel continues to boot.
- ufshcd-crypto now works on big endian cpus.
- Many cleanups.

Eric Biggers (1):
ext4: add inline encryption support

Satya Tangirala (11):
Documentation: Document the blk-crypto framework
block: Keyslot Manager for Inline Encryption
block: Inline encryption support for blk-mq
block: Make blk-integrity preclude hardware inline encryption
block: blk-crypto-fallback for Inline Encryption
scsi: ufs: UFS driver v2.1 spec crypto additions
scsi: ufs: UFS crypto API
scsi: ufs: Add inline encryption support to UFS
fs: introduce SB_INLINECRYPT
fscrypt: add inline encryption support
f2fs: add inline encryption support

 Documentation/admin-guide/ext4.rst        |   6 +
 Documentation/block/index.rst             |   1 +
 Documentation/block/inline-encryption.rst | 263 +++++++++
 Documentation/filesystems/f2fs.rst        |   7 +-
 block/Kconfig                             |  17 +
 block/Makefile                            |   2 +
 block/bio-integrity.c                     |   3 +
 block/bio.c                               |   6 +
 block/blk-core.c                          |  27 +-
 block/blk-crypto-fallback.c               | 657 ++++++++++++++++++++++
 block/blk-crypto-internal.h               | 201 +++++++
 block/blk-crypto.c                        | 404 +++++++++++++
 block/blk-integrity.c                     |   7 +
 block/blk-map.c                           |   1 +
 block/blk-merge.c                         |  11 +
 block/blk-mq.c                            |  13 +
 block/blk.h                               |   2 +
 block/bounce.c                            |   2 +
 block/keyslot-manager.c                   | 397 +++++++++++++
 drivers/md/dm.c                           |   3 +
 drivers/scsi/ufs/Kconfig                  |   9 +
 drivers/scsi/ufs/Makefile                 |   1 +
 drivers/scsi/ufs/ufshcd-crypto.c          | 226 ++++++++
 drivers/scsi/ufs/ufshcd-crypto.h          |  60 ++
 drivers/scsi/ufs/ufshcd.c                 |  46 +-
 drivers/scsi/ufs/ufshcd.h                 |  24 +
 drivers/scsi/ufs/ufshci.h                 |  67 ++-
 fs/buffer.c                               |   7 +-
 fs/crypto/Kconfig                         |   6 +
 fs/crypto/Makefile                        |   1 +
 fs/crypto/bio.c                           |  50 ++
 fs/crypto/crypto.c                        |   2 +-
 fs/crypto/fname.c                         |   4 +-
 fs/crypto/fscrypt_private.h               | 120 +++-
 fs/crypto/inline_crypt.c                  | 339 +++++++++++
 fs/crypto/keyring.c                       |   4 +-
 fs/crypto/keysetup.c                      |  92 ++-
 fs/crypto/keysetup_v1.c                   |  16 +-
 fs/ext4/inode.c                           |   4 +-
 fs/ext4/page-io.c                         |   6 +-
 fs/ext4/readpage.c                        |  11 +-
 fs/ext4/super.c                           |   9 +
 fs/f2fs/compress.c                        |   2 +-
 fs/f2fs/data.c                            |  68 ++-
 fs/f2fs/super.c                           |  32 ++
 fs/proc_namespace.c                       |   1 +
 include/linux/blk-crypto.h                | 123 ++++
 include/linux/blk_types.h                 |   6 +
 include/linux/blkdev.h                    |  41 ++
 include/linux/fs.h                        |   1 +
 include/linux/fscrypt.h                   |  82 +++
 include/linux/keyslot-manager.h           | 106 ++++
 52 files changed, 3504 insertions(+), 92 deletions(-)
 create mode 100644 Documentation/block/inline-encryption.rst
 create mode 100644 block/blk-crypto-fallback.c
 create mode 100644 block/blk-crypto-internal.h
 create mode 100644 block/blk-crypto.c
 create mode 100644 block/keyslot-manager.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
 create mode 100644 fs/crypto/inline_crypt.c
 create mode 100644 include/linux/blk-crypto.h
 create mode 100644 include/linux/keyslot-manager.h
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#1 lore raw

PATCH v13 1/12 Documentation: Document the blk-crypto framework

The blk-crypto framework adds support for inline encryption. There are numerous changes throughout the storage stack. This patch documents the main design choices in the block layer, the API presented to users of the block layer (like fscrypt or layered devices) and the API presented to drivers for adding support for inline encryption.

---
 Documentation/block/index.rst             |   1 +
 Documentation/block/inline-encryption.rst | 263 ++++++++++++++++++++++
 2 files changed, 264 insertions(+)
 create mode 100644 Documentation/block/inline-encryption.rst
Documentation/block/index.rst +1 -0
index 3fa7a52fafa46..026addfc69bc9 100644
@@ -14,6 +14,7 @@ Block
    cmdline-partition
    data-integrity
    deadline-iosched
+   inline-encryption
    ioprio
    kyber-iosched
    null_blk
Documentation/block/inline-encryption.rst new file +263 -0

263 lines not on this page · raw message

Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#2 lore raw

PATCH v13 2/12 block: Keyslot Manager for Inline Encryption

In reply to Satya Tangirala

Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device.

Inline Encryption hardware implementations often function around the concept of "keyslots". These implementations often have a limited number of "keyslots", each of which can hold a key (we say that a key can be "programmed" into a keyslot). Requests made to the storage device may have a keyslot and a data unit number associated with them, and the inline encryption hardware will en/decrypt the data in the requests using the key programmed into that associated keyslot and the data unit number specified with the request.

As keyslots are limited, and programming keys may be expensive in many implementations, and multiple requests may use exactly the same encryption contexts, we introduce a Keyslot Manager to efficiently manage keyslots.

We also introduce a blk_crypto_key, which will represent the key that's programmed into keyslots managed by keyslot managers. The keyslot manager also functions as the interface that upper layers will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.

---
 block/Kconfig                   |   7 +
 block/Makefile                  |   1 +
 block/keyslot-manager.c         | 378 ++++++++++++++++++++++++++++++++
 include/linux/blk-crypto.h      |  52 +++++
 include/linux/blkdev.h          |   6 +
 include/linux/keyslot-manager.h | 106 +++++++++
 6 files changed, 550 insertions(+)
 create mode 100644 block/keyslot-manager.c
 create mode 100644 include/linux/blk-crypto.h
 create mode 100644 include/linux/keyslot-manager.h
block/Kconfig +7 -0
index 41cb34b0fcd1c..f8870c316a038 100644
@@ -186,6 +186,13 @@ config BLK_SED_OPAL
 	Enabling this option enables users to setup/unlock/lock
 	Locking ranges for SED devices using the Opal protocol.
 
+config BLK_INLINE_ENCRYPTION
+	bool "Enable inline encryption support in block layer"
+	help
+	  Build the blk-crypto subsystem. Enabling this lets the
+	  block layer handle encryption, so users can take
+	  advantage of inline encryption hardware if present.
+
 menu "Partition Types"
 
 source "block/partitions/Kconfig"
block/Makefile +1 -0
index 206b96e9387f4..fc963e4676b0a 100644
@@ -36,3 +36,4 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o
block/keyslot-manager.c new file +378 -0

378 lines not on this page · raw message

include/linux/blk-crypto.h new file +52 -0

52 lines not on this page · raw message

include/linux/blkdev.h +6 -0
index 5360696d85ff7..74e3989b3074f 100644
@@ -44,6 +44,7 @@ struct pr_ops;
 struct rq_qos;
 struct blk_queue_stats;
 struct blk_stat_callback;
+struct blk_keyslot_manager;
 
 #define BLKDEV_MIN_RQ	4
 #define BLKDEV_MAX_RQ	128	/* Default maximum */
@@ -469,6 +470,11 @@ struct request_queue {
 	unsigned int		dma_pad_mask;
 	unsigned int		dma_alignment;
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	/* Inline crypto capabilities */
+	struct blk_keyslot_manager *ksm;
+#endif
+
 	unsigned int		rq_timeout;
 	int			poll_nsec;
 
include/linux/keyslot-manager.h new file +106 -0

106 lines not on this page · raw message

Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#3 lore raw

PATCH v13 3/12 block: Inline encryption support for blk-mq

In reply to Satya Tangirala

We must have some way of letting a storage device driver know what encryption context it should use for en/decrypting a request. However, it's the upper layers (like the filesystem/fscrypt) that know about and manages encryption contexts. As such, when the upper layer submits a bio to the block layer, and this bio eventually reaches a device driver with support for inline encryption, the device driver will need to have been told the encryption context for that bio.

We want to communicate the encryption context from the upper layer to the storage device along with the bio, when the bio is submitted to the block layer. To do this, we add a struct bio_crypt_ctx to struct bio, which can represent an encryption context (note that we can't use the bi_private field in struct bio to do this because that field does not function to pass information across layers in the storage stack). We also introduce various functions to manipulate the bio_crypt_ctx and make the bio/request merging logic aware of the bio_crypt_ctx.

We also make changes to blk-mq to make it handle bios with encryption contexts. blk-mq can merge many bios into the same request. These bios need to have contiguous data unit numbers (the necessary changes to blk-merge are also made to ensure this) - as such, it suffices to keep the data unit number of just the first bio, since that's all a storage driver needs to infer the data unit number to use for each data block in each bio in a request. blk-mq keeps track of the encryption context to be used for all the bios in a request with the request's rq_crypt_ctx. When the first bio is added to an empty request, blk-mq will program the encryption context of that bio into the request_queue's keyslot manager, and store the returned keyslot in the request's rq_crypt_ctx. All the functions to operate on encryption contexts are in blk-crypto.c.

Upper layers only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-mq/blk-crypto handles that. Blk-crypto also makes it possible for request-based layered devices like dm-rq to make use of inline encryption hardware by cloning the rq_crypt_ctx and programming a keyslot in the new request_queue when necessary.

Note that any user of the block layer can submit bios with an encryption context, such as filesystems, device-mapper targets, etc.

---
 block/Makefile              |   2 +-
 block/bio.c                 |   6 +
 block/blk-core.c            |  27 ++-
 block/blk-crypto-internal.h | 166 ++++++++++++++++
 block/blk-crypto.c          | 376 ++++++++++++++++++++++++++++++++++++
 block/blk-map.c             |   1 +
 block/blk-merge.c           |  11 ++
 block/blk-mq.c              |  13 ++
 block/blk.h                 |   2 +
 block/bounce.c              |   2 +
 drivers/md/dm.c             |   3 +
 include/linux/blk-crypto.h  |  71 +++++++
 include/linux/blk_types.h   |   6 +
 include/linux/blkdev.h      |   5 +
 14 files changed, 684 insertions(+), 7 deletions(-)
 create mode 100644 block/blk-crypto-internal.h
 create mode 100644 block/blk-crypto.c
block/Makefile +1 -1
index fc963e4676b0a..3ee88d3e807d2 100644
@@ -36,4 +36,4 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o blk-crypto.o
block/bio.c +6 -0
index e4c46e2bd5ba5..1594804fe8bcf 100644
@@ -18,6 +18,7 @@
 #include <linux/blk-cgroup.h>
 #include <linux/highmem.h>
 #include <linux/sched/sysctl.h>
+#include <linux/blk-crypto.h>
 
 #include <trace/events/block.h>
 #include "blk.h"
@@ -237,6 +238,8 @@ void bio_uninit(struct bio *bio)
 
 	if (bio_integrity(bio))
 		bio_integrity_free(bio);
+
+	bio_crypt_free_ctx(bio);
 }
 EXPORT_SYMBOL(bio_uninit);
 
@@ -708,6 +711,8 @@ struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)
 
 	__bio_clone_fast(b, bio);
 
+	bio_crypt_clone(b, bio, gfp_mask);
+
 	if (bio_integrity(bio)) {
 		int ret;
 
@@ -1172,6 +1177,7 @@ void bio_advance(struct bio *bio, unsigned bytes)
 	if (bio_integrity(bio))
 		bio_integrity_advance(bio, bytes);
 
+	bio_crypt_advance(bio, bytes);
 	bio_advance_iter(bio, &bio->bi_iter, bytes);
 }
 EXPORT_SYMBOL(bio_advance);
block/blk-core.c +21 -6
index cf5b2163edfef..0461c24acc670 100644
@@ -38,6 +38,7 @@
 #include <linux/debugfs.h>
 #include <linux/bpf.h>
 #include <linux/psi.h>
+#include <linux/blk-crypto.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/block.h>
@@ -120,6 +121,7 @@ void blk_rq_init(struct request_queue *q, struct request *rq)
 	rq->start_time_ns = ktime_get_ns();
 	rq->part = NULL;
 	refcount_set(&rq->ref, 1);
+	blk_crypto_rq_set_defaults(rq);
 }
 EXPORT_SYMBOL(blk_rq_init);
 
@@ -651,6 +653,8 @@ bool bio_attempt_back_merge(struct request *req, struct bio *bio,
 	req->biotail = bio;
 	req->__data_len += bio->bi_iter.bi_size;
 
+	bio_crypt_free_ctx(bio);
+
 	blk_account_io_start(req, false);
 	return true;
 }
@@ -675,6 +679,8 @@ bool bio_attempt_front_merge(struct request *req, struct bio *bio,
 	req->__sector = bio->bi_iter.bi_sector;
 	req->__data_len += bio->bi_iter.bi_size;
 
+	bio_crypt_do_front_merge(req, bio);
+
 	blk_account_io_start(req, false);
 	return true;
 }
@@ -1124,10 +1130,12 @@ blk_qc_t generic_make_request(struct bio *bio)
 			/* Create a fresh bio_list for all subordinate requests */
 			bio_list_on_stack[1] = bio_list_on_stack[0];
 			bio_list_init(&bio_list_on_stack[0]);
-			if (q->make_request_fn)
-				ret = q->make_request_fn(q, bio);
-			else
-				ret = blk_mq_make_request(q, bio);
+			if (blk_crypto_bio_prep(&bio)) {
+				if (q->make_request_fn)
+					ret = q->make_request_fn(q, bio);
+				else
+					ret = blk_mq_make_request(q, bio);
+			}
 
 			blk_queue_exit(q);
 
@@ -1166,7 +1174,7 @@ EXPORT_SYMBOL(generic_make_request);
 blk_qc_t direct_make_request(struct bio *bio)
 {
 	struct request_queue *q = bio->bi_disk->queue;
-	blk_qc_t ret;
+	blk_qc_t ret = BLK_QC_T_NONE;
 
 	if (WARN_ON_ONCE(q->make_request_fn)) {
 		bio_io_error(bio);
@@ -1176,7 +1184,8 @@ blk_qc_t direct_make_request(struct bio *bio)
 		return BLK_QC_T_NONE;
 	if (unlikely(bio_queue_enter(bio)))
 		return BLK_QC_T_NONE;
-	ret = blk_mq_make_request(q, bio);
+	if (blk_crypto_bio_prep(&bio))
+		ret = blk_mq_make_request(q, bio);
 	blk_queue_exit(q);
 	return ret;
 }
@@ -1308,6 +1317,9 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *
 	    should_fail_request(&rq->rq_disk->part0, blk_rq_bytes(rq)))
 		return BLK_STS_IOERR;
 
+	if (blk_crypto_insert_cloned_request(rq))
+		return BLK_STS_IOERR;
+
 	if (blk_queue_io_stat(q))
 		blk_account_io_start(rq, true);
 
@@ -1672,6 +1684,9 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src,
 	rq->nr_phys_segments = rq_src->nr_phys_segments;
 	rq->ioprio = rq_src->ioprio;
 
+	if (rq->bio)
+		blk_crypto_rq_bio_prep(rq, rq->bio, gfp_mask);
+
 	return 0;
 
 free_and_out:
block/blk-crypto-internal.h new file +166 -0

166 lines not on this page · raw message

block/blk-crypto.c new file +376 -0

376 lines not on this page · raw message

block/blk-map.c +1 -0
index e3e4ac48db459..6e804892d5ec6 100644
@@ -550,6 +550,7 @@ int blk_rq_append_bio(struct request *rq, struct bio **bio)
 		rq->biotail->bi_next = *bio;
 		rq->biotail = *bio;
 		rq->__data_len += (*bio)->bi_iter.bi_size;
+		bio_crypt_free_ctx(*bio);
 	}
 
 	return 0;
block/blk-merge.c +11 -0
index a04e991b5ded9..1a90363982146 100644
@@ -562,6 +562,8 @@ int ll_back_merge_fn(struct request *req, struct bio *bio, unsigned int nr_segs)
 	if (blk_integrity_rq(req) &&
 	    integrity_req_gap_back_merge(req, bio))
 		return 0;
+	if (!bio_crypt_ctx_back_mergeable(req, bio))
+		return 0;
 	if (blk_rq_sectors(req) + bio_sectors(bio) >
 	    blk_rq_get_max_sectors(req, blk_rq_pos(req))) {
 		req_set_nomerge(req->q, req);
@@ -578,6 +580,8 @@ int ll_front_merge_fn(struct request *req, struct bio *bio, unsigned int nr_segs
 	if (blk_integrity_rq(req) &&
 	    integrity_req_gap_front_merge(req, bio))
 		return 0;
+	if (!bio_crypt_ctx_front_mergeable(req, bio))
+		return 0;
 	if (blk_rq_sectors(req) + bio_sectors(bio) >
 	    blk_rq_get_max_sectors(req, bio->bi_iter.bi_sector)) {
 		req_set_nomerge(req->q, req);
@@ -627,6 +631,9 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req,
 	if (blk_integrity_merge_rq(q, req, next) == false)
 		return 0;
 
+	if (!bio_crypt_ctx_merge_rq(req, next))
+		return 0;
+
 	/* Merge is OK... */
 	req->nr_phys_segments = total_phys_segments;
 	return 1;
@@ -851,6 +858,10 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
 	if (blk_integrity_merge_bio(rq->q, rq, bio) == false)
 		return false;
 
+	/* Only merge if the crypt contexts are compatible */
+	if (!bio_crypt_rq_ctx_compatible(rq, bio))
+		return false;
+
 	/* must be using the same buffer */
 	if (req_op(rq) == REQ_OP_WRITE_SAME &&
 	    !blk_write_same_mergeable(rq->bio, bio))
block/blk-mq.c +13 -0
index 9ee695bdf8739..4f8adef7fd0d9 100644
@@ -26,6 +26,7 @@
 #include <linux/delay.h>
 #include <linux/crash_dump.h>
 #include <linux/prefetch.h>
+#include <linux/blk-crypto.h>
 
 #include <trace/events/block.h>
 
@@ -317,6 +318,7 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
 #if defined(CONFIG_BLK_DEV_INTEGRITY)
 	rq->nr_integrity_segments = 0;
 #endif
+	blk_crypto_rq_set_defaults(rq);
 	/* tag was already set */
 	WRITE_ONCE(rq->deadline, 0);
 
@@ -473,6 +475,7 @@ static void __blk_mq_free_request(struct request *rq)
 	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
 	const int sched_tag = rq->internal_tag;
 
+	blk_crypto_free_request(rq);
 	blk_pm_mark_last_busy(rq);
 	rq->mq_hctx = NULL;
 	if (rq->tag != -1)
@@ -1820,6 +1823,7 @@ static void blk_mq_bio_to_request(struct request *rq, struct bio *bio,
 	rq->__sector = bio->bi_iter.bi_sector;
 	rq->write_hint = bio->bi_write_hint;
 	blk_rq_bio_prep(rq, bio, nr_segs);
+	blk_crypto_rq_bio_prep(rq, bio, GFP_NOIO);
 
 	blk_account_io_start(rq, true);
 }
@@ -2021,6 +2025,7 @@ blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 	struct request *same_queue_rq = NULL;
 	unsigned int nr_segs;
 	blk_qc_t cookie;
+	blk_status_t ret;
 
 	blk_queue_bounce(q, &bio);
 	__blk_queue_split(q, &bio, &nr_segs);
@@ -2054,6 +2059,14 @@ blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 
 	blk_mq_bio_to_request(rq, bio, nr_segs);
 
+	ret = blk_crypto_init_request(rq);
+	if (ret != BLK_STS_OK) {
+		bio->bi_status = ret;
+		bio_endio(bio);
+		blk_mq_free_request(rq);
+		return BLK_QC_T_NONE;
+	}
+
 	plug = blk_mq_plug(q, bio);
 	if (unlikely(is_flush_fua)) {
 		/* Bypass scheduler for flush requests */
block/blk.h +2 -0
index e5cd350ca3798..fc00537026a04 100644
@@ -5,7 +5,9 @@
 #include <linux/idr.h>
 #include <linux/blk-mq.h>
 #include <linux/part_stat.h>
+#include <linux/blk-crypto.h>
 #include <xen/xen.h>
+#include "blk-crypto-internal.h"
 #include "blk-mq.h"
 #include "blk-mq-sched.h"
 
block/bounce.c +2 -0
index f8ed677a1bf7e..c3aaed0701246 100644
@@ -267,6 +267,8 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,
 		break;
 	}
 
+	bio_crypt_clone(bio, bio_src, gfp_mask);
+
 	if (bio_integrity(bio_src)) {
 		int ret;
 
drivers/md/dm.c +3 -0
index 0eb93da44ea2a..8921cd79422c6 100644
@@ -26,6 +26,7 @@
 #include <linux/pr.h>
 #include <linux/refcount.h>
 #include <linux/part_stat.h>
+#include <linux/blk-crypto.h>
 
 #define DM_MSG_PREFIX "core"
 
@@ -1334,6 +1335,8 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
 
 	__bio_clone_fast(clone, bio);
 
+	bio_crypt_clone(clone, bio, GFP_NOIO);
+
 	if (bio_integrity(bio)) {
 		int r;
 
include/linux/blk-crypto.h +71 -0

81 lines not on this page · raw message

include/linux/blk_types.h +6 -0
index b90dca1fa4304..9322261fb3166 100644
@@ -18,6 +18,7 @@ struct block_device;
 struct io_context;
 struct cgroup_subsys_state;
 typedef void (bio_end_io_t) (struct bio *);
+struct bio_crypt_ctx;
 
 /*
  * Block error status values.  See block/blk-core:blk_errors for the details.
@@ -185,6 +186,11 @@ struct bio {
 	u64			bi_iocost_cost;
 #endif
 #endif
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	struct bio_crypt_ctx	*bi_crypt_context;
+#endif
+
 	union {
 #if defined(CONFIG_BLK_DEV_INTEGRITY)
 		struct bio_integrity_payload *bi_integrity; /* data integrity */
include/linux/blkdev.h +5 -0
index 74e3989b3074f..de38318a7acf0 100644
@@ -223,6 +223,11 @@ struct request {
 	unsigned short nr_integrity_segments;
 #endif
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	struct bio_crypt_ctx *crypt_ctx;
+	struct blk_ksm_keyslot *crypt_keyslot;
+#endif
+
 	unsigned short write_hint;
 	unsigned short ioprio;
 
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#4 lore raw

PATCH v13 4/12 block: Make blk-integrity preclude hardware inline encryption

In reply to Satya Tangirala

Whenever a device supports blk-integrity, make the kernel pretend that the device doesn't support inline encryption (essentially by setting the keyslot manager in the request queue to NULL).

There's no hardware currently that supports both integrity and inline encryption. However, it seems possible that there will be such hardware in the near future (like the NVMe key per I/O support that might support both inline encryption and PI).

But properly integrating both features is not trivial, and without real hardware that implements both, it is difficult to tell if it will be done correctly by the majority of hardware that support both. So it seems best not to support both features together right now, and to decide what to do at probe time.

---
 block/bio-integrity.c   |  3 +++
 block/blk-integrity.c   |  7 +++++++
 block/keyslot-manager.c | 19 +++++++++++++++++++
 include/linux/blkdev.h  | 30 ++++++++++++++++++++++++++++++
 4 files changed, 59 insertions(+)
block/bio-integrity.c +3 -0
index bf62c25cde8f4..3579ac0f6ec1f 100644
@@ -42,6 +42,9 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio,
 	struct bio_set *bs = bio->bi_pool;
 	unsigned inline_vecs;
 
+	if (WARN_ON_ONCE(bio_has_crypt_ctx(bio)))
+		return ERR_PTR(-EOPNOTSUPP);
+
 	if (!bs || !mempool_initialized(&bs->bio_integrity_pool)) {
 		bip = kmalloc(struct_size(bip, bip_inline_vecs, nr_vecs), gfp_mask);
 		inline_vecs = nr_vecs;
block/blk-integrity.c +7 -0
index ff1070edbb400..c03705cbb9c9f 100644
@@ -409,6 +409,13 @@ void blk_integrity_register(struct gendisk *disk, struct blk_integrity *template
 	bi->tag_size = template->tag_size;
 
 	disk->queue->backing_dev_info->capabilities |= BDI_CAP_STABLE_WRITES;
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	if (disk->queue->ksm) {
+		pr_warn("blk-integrity: Integrity and hardware inline encryption are not supported together. Disabling hardware inline encryption.\n");
+		blk_ksm_unregister(disk->queue);
+	}
+#endif
 }
 EXPORT_SYMBOL(blk_integrity_register);
 
block/keyslot-manager.c +19 -0
index fcd3fd469d7c1..c2ef41b3147ba 100644
@@ -25,6 +25,9 @@
  * Upper layers will call blk_ksm_get_slot_for_key() to program a
  * key into some slot in the inline encryption hardware.
  */
+
+#define pr_fmt(fmt) "blk-crypto: " fmt
+
 #include <linux/keyslot-manager.h>
 #include <linux/atomic.h>
 #include <linux/mutex.h>
@@ -376,3 +379,19 @@ void blk_ksm_destroy(struct blk_keyslot_manager *ksm)
 	memzero_explicit(ksm, sizeof(*ksm));
 }
 EXPORT_SYMBOL_GPL(blk_ksm_destroy);
+
+bool blk_ksm_register(struct blk_keyslot_manager *ksm, struct request_queue *q)
+{
+	if (blk_integrity_queue_supports_integrity(q)) {
+		pr_warn("Integrity and hardware inline encryption are not supported together. Disabling hardware inline encryption.\n");
+		return false;
+	}
+	q->ksm = ksm;
+	return true;
+}
+EXPORT_SYMBOL_GPL(blk_ksm_register);
+
+void blk_ksm_unregister(struct request_queue *q)
+{
+	q->ksm = NULL;
+}
include/linux/blkdev.h +30 -0
index de38318a7acf0..d8f85fe696752 100644
@@ -1582,6 +1582,12 @@ struct blk_integrity *bdev_get_integrity(struct block_device *bdev)
 	return blk_get_integrity(bdev->bd_disk);
 }
 
+static inline bool
+blk_integrity_queue_supports_integrity(struct request_queue *q)
+{
+	return q->integrity.profile;
+}
+
 static inline bool blk_integrity_rq(struct request *rq)
 {
 	return rq->cmd_flags & REQ_INTEGRITY;
@@ -1662,6 +1668,11 @@ static inline struct blk_integrity *blk_get_integrity(struct gendisk *disk)
 {
 	return NULL;
 }
+static inline bool
+blk_integrity_queue_supports_integrity(struct request_queue *q)
+{
+	return false;
+}
 static inline int blk_integrity_compare(struct gendisk *a, struct gendisk *b)
 {
 	return 0;
@@ -1713,6 +1724,25 @@ static inline struct bio_vec *rq_integrity_vec(struct request *rq)
 
 #endif /* CONFIG_BLK_DEV_INTEGRITY */
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+
+bool blk_ksm_register(struct blk_keyslot_manager *ksm, struct request_queue *q);
+
+void blk_ksm_unregister(struct request_queue *q);
+
+#else /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+static inline bool blk_ksm_register(struct blk_keyslot_manager *ksm,
+				    struct request_queue *q)
+{
+	return true;
+}
+
+static inline void blk_ksm_unregister(struct request_queue *q) { }
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+
 struct block_device_operations {
 	int (*open) (struct block_device *, fmode_t);
 	void (*release) (struct gendisk *, fmode_t);
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#5 lore raw

PATCH v13 5/12 block: blk-crypto-fallback for Inline Encryption

In reply to Satya Tangirala

Blk-crypto delegates crypto operations to inline encryption hardware when available. The separately configurable blk-crypto-fallback contains a software fallback to the kernel crypto API - when enabled, blk-crypto will use this fallback for en/decryption when inline encryption hardware is not available.

This lets upper layers not have to worry about whether or not the underlying device has support for inline encryption before deciding to specify an encryption context for a bio. It also allows for testing without actual inline encryption hardware - in particular, it makes it possible to test the inline encryption code in ext4 and f2fs simply by running xfstests with the inlinecrypt mount option, which in turn allows for things like the regular upstream regression testing of ext4 to cover the inline encryption code paths.

For more details, refer to Documentation/block/inline-encryption.rst.

---
 block/Kconfig               |  10 +
 block/Makefile              |   1 +
 block/blk-crypto-fallback.c | 657 ++++++++++++++++++++++++++++++++++++
 block/blk-crypto-internal.h |  35 ++
 block/blk-crypto.c          |  68 ++--
 include/linux/blk-crypto.h  |   2 +-
 6 files changed, 752 insertions(+), 21 deletions(-)
 create mode 100644 block/blk-crypto-fallback.c
block/Kconfig +10 -0
index f8870c316a038..9382a4acefc39 100644
@@ -193,6 +193,16 @@ config BLK_INLINE_ENCRYPTION
 	  block layer handle encryption, so users can take
 	  advantage of inline encryption hardware if present.
 
+config BLK_INLINE_ENCRYPTION_FALLBACK
+	bool "Enable crypto API fallback for blk-crypto"
+	depends on BLK_INLINE_ENCRYPTION
+	select CRYPTO
+	select CRYPTO_SKCIPHER
+	help
+	  Enabling this lets the block layer handle inline encryption
+	  by falling back to the kernel crypto API when inline
+	  encryption hardware is not present.
+
 menu "Partition Types"
 
 source "block/partitions/Kconfig"
block/Makefile +1 -0
index 3ee88d3e807d2..78719169fb2af 100644
@@ -37,3 +37,4 @@ obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
 obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o blk-crypto.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK)	+= blk-crypto-fallback.o
block/blk-crypto-fallback.c new file +657 -0

657 lines not on this page · raw message

block/blk-crypto-internal.h +35 -0
index 796f757fe8e92..d2b0f565d83cb 100644
@@ -11,10 +11,13 @@
 
 /* Represents a crypto mode supported by blk-crypto  */
 struct blk_crypto_mode {
+	const char *cipher_str; /* crypto API name (for fallback case) */
 	unsigned int keysize; /* key size in bytes */
 	unsigned int ivsize; /* iv size in bytes */
 };
 
+extern const struct blk_crypto_mode blk_crypto_modes[];
+
 #ifdef CONFIG_BLK_INLINE_ENCRYPTION
 
 void bio_crypt_dun_increment(u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE],
@@ -163,4 +166,36 @@ static inline blk_status_t blk_crypto_insert_cloned_request(struct request *rq)
 	return BLK_STS_OK;
 }
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK
+
+int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num);
+
+bool blk_crypto_fallback_bio_prep(struct bio **bio_ptr);
+
+int blk_crypto_fallback_evict_key(const struct blk_crypto_key *key);
+
+#else /* CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK */
+
+static inline int
+blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num)
+{
+	pr_warn_once("crypto API fallback is disabled\n");
+	return -ENOPKG;
+}
+
+static inline bool blk_crypto_fallback_bio_prep(struct bio **bio_ptr)
+{
+	pr_warn_once("crypto API fallback disabled; failing request.\n");
+	(*bio_ptr)->bi_status = BLK_STS_NOTSUPP;
+	return false;
+}
+
+static inline int
+blk_crypto_fallback_evict_key(const struct blk_crypto_key *key)
+{
+	return 0;
+}
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK */
+
 #endif /* __LINUX_BLK_CRYPTO_INTERNAL_H */
block/blk-crypto.c +48 -20

130 lines not on this page · raw message

include/linux/blk-crypto.h +1 -1
index 76095b07dd902..e82342907f2b1 100644
@@ -61,7 +61,7 @@ struct blk_crypto_key {
  *
  * A bio_crypt_ctx specifies that the contents of the bio will be encrypted (for
  * write requests) or decrypted (for read requests) inline by the storage device
- * or controller.
+ * or controller, or by the crypto API fallback.
  */
 struct bio_crypt_ctx {
 	const struct blk_crypto_key	*bc_key;
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#6 lore raw

PATCH v13 6/12 scsi: ufs: UFS driver v2.1 spec crypto additions

In reply to Satya Tangirala

Add the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI specification in preparation to add support for inline encryption to UFS.

---
 drivers/scsi/ufs/ufshcd.c |  2 ++
 drivers/scsi/ufs/ufshcd.h |  6 ++++
 drivers/scsi/ufs/ufshci.h | 67 +++++++++++++++++++++++++++++++++++++--
 3 files changed, 73 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +2 -0
index 698e8d20b4bac..2435c600cb2d9 100644
@@ -4767,6 +4767,8 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	case OCS_MISMATCH_RESP_UPIU_SIZE:
 	case OCS_PEER_COMM_FAILURE:
 	case OCS_FATAL_ERROR:
+	case OCS_INVALID_CRYPTO_CONFIG:
+	case OCS_GENERAL_CRYPTO_ERROR:
 	default:
 		result |= DID_ERROR << 16;
 		dev_err(hba->dev,
drivers/scsi/ufs/ufshcd.h +6 -0
index 6ffc08ad85f63..835b9a844aa21 100644
@@ -555,6 +555,12 @@ enum ufshcd_caps {
 	 * for userspace to control the power management.
 	 */
 	UFSHCD_CAP_RPM_AUTOSUSPEND			= 1 << 6,
+
+	/*
+	 * This capability allows the host controller driver to use the
+	 * inline crypto engine, if it is present
+	 */
+	UFSHCD_CAP_CRYPTO				= 1 << 7,
 };
 
 /**
drivers/scsi/ufs/ufshci.h +65 -2

106 lines not on this page · raw message

Signature · 1 line
2.26.2.645.ge9eca65c58-goog
Reviewed-by on v13 6/12
#7 lore raw

Re: [PATCH v13 06/12] scsi: ufs: UFS driver v2.1 spec crypto additions

Hi Satya,

Quoted 149 lines · Satya Tangirala

Add the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI specification in preparation to add support for inline encryption to UFS.

---
 drivers/scsi/ufs/ufshcd.c |  2 ++
 drivers/scsi/ufs/ufshcd.h |  6 ++++
 drivers/scsi/ufs/ufshci.h | 67 +++++++++++++++++++++++++++++++++++++--
 3 files changed, 73 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +2 -0
index 698e8d20b4bac..2435c600cb2d9 100644
@@ -4767,6 +4767,8 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	case OCS_MISMATCH_RESP_UPIU_SIZE:
 	case OCS_PEER_COMM_FAILURE:
 	case OCS_FATAL_ERROR:
+	case OCS_INVALID_CRYPTO_CONFIG:
+	case OCS_GENERAL_CRYPTO_ERROR:
 	default:
 		result |= DID_ERROR << 16;
 		dev_err(hba->dev,
drivers/scsi/ufs/ufshcd.h +6 -0
index 6ffc08ad85f63..835b9a844aa21 100644
@@ -555,6 +555,12 @@ enum ufshcd_caps {
 	 * for userspace to control the power management.
 	 */
 	UFSHCD_CAP_RPM_AUTOSUSPEND			= 1 << 6,
+
+	/*
+	 * This capability allows the host controller driver to use the
+	 * inline crypto engine, if it is present
+	 */
+	UFSHCD_CAP_CRYPTO				= 1 << 7,
 };
 
 /**
drivers/scsi/ufs/ufshci.h +62 -2
index c2961d37cc1cf..c0651fe6dbbc6 100644
@@ -90,6 +90,7 @@ enum {
 	MASK_64_ADDRESSING_SUPPORT		= 0x01000000,
 	MASK_OUT_OF_ORDER_DATA_DELIVERY_SUPPORT	= 0x02000000,
 	MASK_UIC_DME_TEST_MODE_SUPPORT		= 0x04000000,
+	MASK_CRYPTO_SUPPORT			= 0x10000000,
 };
 
 #define UFS_MASK(mask, offset)		((mask) << (offset))
@@ -143,6 +144,7 @@ enum {
 #define DEVICE_FATAL_ERROR			0x800
 #define CONTROLLER_FATAL_ERROR			0x10000
 #define SYSTEM_BUS_FATAL_ERROR			0x20000
+#define CRYPTO_ENGINE_FATAL_ERROR		0x40000
 
 #define UFSHCD_UIC_HIBERN8_MASK	(UIC_HIBERNATE_ENTER |\
 				UIC_HIBERNATE_EXIT)
@@ -155,11 +157,13 @@ enum {
 #define UFSHCD_ERROR_MASK	(UIC_ERROR |\
 				DEVICE_FATAL_ERROR |\
 				CONTROLLER_FATAL_ERROR |\
-				SYSTEM_BUS_FATAL_ERROR)
+				SYSTEM_BUS_FATAL_ERROR |\
+				CRYPTO_ENGINE_FATAL_ERROR)
 
 #define INT_FATAL_ERRORS	(DEVICE_FATAL_ERROR |\
 				CONTROLLER_FATAL_ERROR |\
-				SYSTEM_BUS_FATAL_ERROR)
+				SYSTEM_BUS_FATAL_ERROR |\
+				CRYPTO_ENGINE_FATAL_ERROR)
 
 /* HCS - Host Controller Status 30h */
 #define DEVICE_PRESENT				0x1
@@ -318,6 +322,61 @@ enum {
 	INTERRUPT_MASK_ALL_VER_21	= 0x71FFF,
 };
 
+/* CCAP - Crypto Capability 100h */
+union ufs_crypto_capabilities {
+	__le32 reg_val;
+	struct {
+		u8 num_crypto_cap;
+		u8 config_count;
+		u8 reserved;
+		u8 config_array_ptr;
+	};
+};
+
+enum ufs_crypto_key_size {
+	UFS_CRYPTO_KEY_SIZE_INVALID	= 0x0,
+	UFS_CRYPTO_KEY_SIZE_128		= 0x1,
+	UFS_CRYPTO_KEY_SIZE_192		= 0x2,
+	UFS_CRYPTO_KEY_SIZE_256		= 0x3,
+	UFS_CRYPTO_KEY_SIZE_512		= 0x4,
+};
+
+enum ufs_crypto_alg {
+	UFS_CRYPTO_ALG_AES_XTS			= 0x0,
+	UFS_CRYPTO_ALG_BITLOCKER_AES_CBC	= 0x1,
+	UFS_CRYPTO_ALG_AES_ECB			= 0x2,
+	UFS_CRYPTO_ALG_ESSIV_AES_CBC		= 0x3,
+};
+
+/* x-CRYPTOCAP - Crypto Capability X */
+union ufs_crypto_cap_entry {
+	__le32 reg_val;
+	struct {
+		u8 algorithm_id;
+		u8 sdus_mask; /* Supported data unit size mask */
+		u8 key_size;
+		u8 reserved;
+	};
+};
+
+#define UFS_CRYPTO_CONFIGURATION_ENABLE (1 << 7)
+#define UFS_CRYPTO_KEY_MAX_SIZE 64
+/* x-CRYPTOCFG - Crypto Configuration X */
+union ufs_crypto_cfg_entry {
+	__le32 reg_val[32];
+	struct {
+		u8 crypto_key[UFS_CRYPTO_KEY_MAX_SIZE];
+		u8 data_unit_size;
+		u8 crypto_cap_idx;
+		u8 reserved_1;
+		u8 config_enable;
+		u8 reserved_multi_host;
+		u8 reserved_2;
+		u8 vsb[2];
+		u8 reserved_3[56];
+	};
+};
+
 /*
  * Request Descriptor Definitions
  */
@@ -339,6 +398,7 @@ enum {
 	UTP_NATIVE_UFS_COMMAND		= 0x10000000,
 	UTP_DEVICE_MANAGEMENT_FUNCTION	= 0x20000000,
 	UTP_REQ_DESC_INT_CMD		= 0x01000000,
+	UTP_REQ_DESC_CRYPTO_ENABLE_CMD	= 0x00800000,
 };
 
 /* UTP Transfer Request Data Direction (DD) */

drivers/scsi/ufs/ufshci.h enum { · line 423

Quoted 6 lines
drivers/scsi/ufs/ufshci.h +3 -0
@@ -358,6 +418,9 @@ enum {
 	OCS_PEER_COMM_FAILURE		= 0x5,
 	OCS_ABORTED			= 0x6,
 	OCS_FATAL_ERROR			= 0x7,
+	OCS_DEVICE_FATAL_ERROR		= 0x8,
+	OCS_INVALID_CRYPTO_CONFIG	= 0x9,
+	OCS_GENERAL_CRYPTO_ERROR	= 0xA,

A nit is that OCS_DEVICE_FATAL_ERROR definition is added here but not added in ufshcd_transfer_rsp_status().

This will not have any side effect because OCS_DEVICE_FATAL_ERROR was already handled by "default" label in ufshcd_transfer_rsp_status().

drivers/scsi/ufs/ufshci.h enum { · line 426

Quoted 3 lines
drivers/scsi/ufs/ufshci.h +0 -0
 	OCS_INVALID_COMMAND_STATUS	= 0x0F,
 	MASK_OCS			= 0x0F,
 };

Otherwise looks good to me.

Thanks.

#8 lore raw

PATCH v13 7/12 scsi: ufs: UFS crypto API

In reply to Satya Tangirala

Introduce functions to manipulate UFS inline encryption hardware in line with the JEDEC UFSHCI v2.1 specification and to work with the block keyslot manager.

The UFS crypto API will assume by default that a vendor driver doesn't support UFS crypto, even if the hardware advertises the capability, because a lot of hardware requires some special handling that's not specified in the aforementioned JEDEC spec. Each vendor driver must explicity set hba->caps |= UFSHCD_CAP_CRYPTO before ufshcd_hba_init_crypto is called to opt-in to UFS crypto support.

---
 drivers/scsi/ufs/Kconfig         |   9 ++
 drivers/scsi/ufs/Makefile        |   1 +
 drivers/scsi/ufs/ufshcd-crypto.c | 226 +++++++++++++++++++++++++++++++
 drivers/scsi/ufs/ufshcd-crypto.h |  42 ++++++
 drivers/scsi/ufs/ufshcd.h        |  12 ++
 5 files changed, 290 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
drivers/scsi/ufs/Kconfig +9 -0
index e2005aeddc2db..5ed3f209f8810 100644
@@ -160,3 +160,12 @@ config SCSI_UFS_BSG
 
 	  Select this if you need a bsg device node for your UFS controller.
 	  If unsure, say N.
+
+config SCSI_UFS_CRYPTO
+	bool "UFS Crypto Engine Support"
+	depends on SCSI_UFSHCD && BLK_INLINE_ENCRYPTION
+	help
+	  Enable Crypto Engine Support in UFS.
+	  Enabling this makes it possible for the kernel to use the crypto
+	  capabilities of the UFS device (if present) to perform crypto
+	  operations on data being transferred to/from the device.
drivers/scsi/ufs/Makefile +1 -0
index 94c6c5d7334b6..197e178f44bce 100644
@@ -7,6 +7,7 @@ obj-$(CONFIG_SCSI_UFS_QCOM) += ufs-qcom.o
 obj-$(CONFIG_SCSI_UFSHCD) += ufshcd-core.o
 ufshcd-core-y				+= ufshcd.o ufs-sysfs.o
 ufshcd-core-$(CONFIG_SCSI_UFS_BSG)	+= ufs_bsg.o
+ufshcd-core-$(CONFIG_SCSI_UFS_CRYPTO) += ufshcd-crypto.o
 obj-$(CONFIG_SCSI_UFSHCD_PCI) += ufshcd-pci.o
 obj-$(CONFIG_SCSI_UFSHCD_PLATFORM) += ufshcd-pltfrm.o
 obj-$(CONFIG_SCSI_UFS_HISI) += ufs-hisi.o
drivers/scsi/ufs/ufshcd-crypto.c new file +226 -0

226 lines not on this page · raw message

drivers/scsi/ufs/ufshcd-crypto.h new file +42 -0

42 lines not on this page · raw message

drivers/scsi/ufs/ufshcd.h +12 -0
index 835b9a844aa21..c935ebb16cba3 100644
@@ -57,6 +57,7 @@
 #include <linux/regulator/consumer.h>
 #include <linux/bitfield.h>
 #include <linux/devfreq.h>
+#include <linux/keyslot-manager.h>
 #include "unipro.h"
 
 #include <asm/irq.h>
@@ -614,6 +615,10 @@ enum ufshcd_caps {
  * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
  *  device is known or not.
  * @scsi_block_reqs_cnt: reference counting for scsi block requests
+ * @crypto_capabilities: Content of crypto capabilities register (0x100)
+ * @crypto_cap_array: Array of crypto capabilities
+ * @crypto_cfg_register: Start of the crypto cfg array
+ * @ksm: the keyslot manager tied to this hba
  */
 struct ufs_hba {
 	void __iomem *mmio_base;
@@ -733,6 +738,13 @@ struct ufs_hba {
 
 	struct device		bsg_dev;
 	struct request_queue	*bsg_queue;
+
+#ifdef CONFIG_SCSI_UFS_CRYPTO
+	union ufs_crypto_capabilities crypto_capabilities;
+	union ufs_crypto_cap_entry *crypto_cap_array;
+	u32 crypto_cfg_register;
+	struct blk_keyslot_manager ksm;
+#endif
 };
 
 /* Returns true if clocks can be gated. Otherwise false */
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
Reviewed-by on v13 7/12
#9 lore raw

Re: [PATCH v13 07/12] scsi: ufs: UFS crypto API

Hi Satya,

Quoted 8 lines · Satya Tangirala

Introduce functions to manipulate UFS inline encryption hardware in line with the JEDEC UFSHCI v2.1 specification and to work with the block keyslot manager.

The UFS crypto API will assume by default that a vendor driver doesn't support UFS crypto, even if the hardware advertises the capability, because a lot of hardware requires some special handling that's not specified in the aforementioned JEDEC spec. Each vendor driver must explicity set

explicitly

Quoted 5 lines

hba->caps |= UFSHCD_CAP_CRYPTO before ufshcd_hba_init_crypto is called to opt-in to UFS crypto support.

#10 lore raw

PATCH v13 8/12 scsi: ufs: Add inline encryption support to UFS

In reply to Satya Tangirala

Wire up ufshcd.c with the UFS Crypto API, the block layer inline encryption additions and the keyslot manager.

---
 drivers/scsi/ufs/ufshcd-crypto.h | 18 +++++++++++++
 drivers/scsi/ufs/ufshcd.c        | 44 ++++++++++++++++++++++++++++----
 drivers/scsi/ufs/ufshcd.h        |  6 +++++
 3 files changed, 63 insertions(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd-crypto.h +18 -0
index 22677619de595..9578edb63e7b4 100644
@@ -10,6 +10,20 @@
 #include "ufshcd.h"
 #include "ufshci.h"
 
+static inline void ufshcd_prepare_lrbp_crypto(struct ufs_hba *hba,
+					      struct scsi_cmnd *cmd,
+					      struct ufshcd_lrb *lrbp)
+{
+	struct request *rq = cmd->request;
+
+	if (rq->crypt_keyslot) {
+		lrbp->crypto_key_slot = blk_ksm_get_slot_idx(rq->crypt_keyslot);
+		lrbp->data_unit_num = rq->crypt_ctx->bc_dun[0];
+	} else {
+		lrbp->crypto_key_slot = -1;
+	}
+}
+
 bool ufshcd_crypto_enable(struct ufs_hba *hba);
 
 int ufshcd_hba_init_crypto(struct ufs_hba *hba);
@@ -21,6 +35,10 @@ void ufshcd_crypto_destroy_keyslot_manager(struct ufs_hba *hba);
 
 #else /* CONFIG_SCSI_UFS_CRYPTO */
 
+static inline void ufshcd_prepare_lrbp_crypto(struct ufs_hba *hba,
+					      struct scsi_cmnd *cmd,
+					      struct ufshcd_lrb *lrbp) { }
+
 static inline bool ufshcd_crypto_enable(struct ufs_hba *hba)
 {
 	return false;
drivers/scsi/ufs/ufshcd.c +39 -5

111 lines not on this page · raw message

drivers/scsi/ufs/ufshcd.h +6 -0
index c935ebb16cba3..bc2f945d3f5f5 100644
@@ -183,6 +183,8 @@ struct ufs_pm_lvl_states {
  * @intr_cmd: Interrupt command (doesn't participate in interrupt aggregation)
  * @issue_time_stamp: time stamp for debug purposes
  * @compl_time_stamp: time stamp for statistics
+ * @crypto_key_slot: the key slot to use for inline crypto (-1 if none)
+ * @data_unit_num: the data unit number for the first block for inline crypto
  * @req_abort_skip: skip request abort task flag
  */
 struct ufshcd_lrb {
@@ -207,6 +209,10 @@ struct ufshcd_lrb {
 	bool intr_cmd;
 	ktime_t issue_time_stamp;
 	ktime_t compl_time_stamp;
+#ifdef CONFIG_SCSI_UFS_CRYPTO
+	int crypto_key_slot;
+	u64 data_unit_num;
+#endif
 
 	bool req_abort_skip;
 };
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
Reviewed-by on v13 8/12
#12 lore raw

Re: [PATCH v13 08/12] scsi: ufs: Add inline encryption support to UFS

In reply to Satya Tangirala

Hi Satya,

Quoted 4 lines · Satya Tangirala

Wire up ufshcd.c with the UFS Crypto API, the block layer inline encryption additions and the keyslot manager.

Thanks Satya and Eric so much to make inline encryption upstreamed.

I will provide essential MediaTek vops patch to adopt this framework soon.

Thanks,
Stanley Chu

#13 lore raw

PATCH v13 9/12 fs: introduce SB_INLINECRYPT

In reply to Satya Tangirala

Introduce SB_INLINECRYPT, which is set by filesystems that wish to use blk-crypto for file content en/decryption. This flag maps to the '-o inlinecrypt' mount option which multiple filesystems will implement, and code in fs/crypto/ needs to be able to check for this mount option in a filesystem-independent way.

---
 fs/proc_namespace.c | 1 +
 include/linux/fs.h  | 1 +
 2 files changed, 2 insertions(+)
fs/proc_namespace.c +1 -0
index 273ee82d8aa97..8bf195d3bda69 100644
@@ -49,6 +49,7 @@ static int show_sb_opts(struct seq_file *m, struct super_block *sb)
 		{ SB_DIRSYNC, ",dirsync" },
 		{ SB_MANDLOCK, ",mand" },
 		{ SB_LAZYTIME, ",lazytime" },
+		{ SB_INLINECRYPT, ",inlinecrypt" },
 		{ 0, NULL }
 	};
 	const struct proc_fs_info *fs_infop;
include/linux/fs.h +1 -0
index d3ebb49189df2..da6551c8d96fb 100644
@@ -1376,6 +1376,7 @@ extern int send_sigurg(struct fown_struct *fown);
 #define SB_NODIRATIME	2048	/* Do not update directory access times */
 #define SB_SILENT	32768
 #define SB_POSIXACL	(1<<16)	/* VFS does not apply the umask */
+#define SB_INLINECRYPT	(1<<17)	/* Use blk-crypto for encrypted files */
 #define SB_KERNMOUNT	(1<<22) /* this is a kern_mount call */
 #define SB_I_VERSION	(1<<23) /* Update inode I_version field */
 #define SB_LAZYTIME	(1<<25) /* Update the on-disk [acm]times lazily */
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#14 lore raw

PATCH v13 10/12 fscrypt: add inline encryption support

In reply to Satya Tangirala

Add support for inline encryption to fs/crypto/. With "inline encryption", the block layer handles the decryption/encryption as part of the bio, instead of the filesystem doing the crypto itself via Linux's crypto API. This model is needed in order to take advantage of the inline encryption hardware present on most modern mobile SoCs.

To use inline encryption, the filesystem needs to be mounted with '-o inlinecrypt'. The contents of any encrypted files will then be encrypted using blk-crypto, instead of using the traditional filesystem-layer crypto. Fscrypt still provides the key and IV to use, and the actual ciphertext on-disk is still the same; therefore it's testable using the existing fscrypt ciphertext verification tests.

Note that since blk-crypto has a fallack to Linux's crypto API, and also supports all the encryption modes currently supported by fscrypt, this feature is usable and testable even without actual inline encryption hardware.

Per-filesystem changes will be needed to set encryption contexts when submitting bios and to implement the 'inlinecrypt' mount option. This patch just adds the common code.

---
 fs/crypto/Kconfig           |   6 +
 fs/crypto/Makefile          |   1 +
 fs/crypto/bio.c             |  50 ++++++
 fs/crypto/crypto.c          |   2 +-
 fs/crypto/fname.c           |   4 +-
 fs/crypto/fscrypt_private.h | 120 +++++++++++--
 fs/crypto/inline_crypt.c    | 339 ++++++++++++++++++++++++++++++++++++
 fs/crypto/keyring.c         |   4 +-
 fs/crypto/keysetup.c        |  92 ++++++----
 fs/crypto/keysetup_v1.c     |  16 +-
 include/linux/fscrypt.h     |  82 +++++++++
 11 files changed, 659 insertions(+), 57 deletions(-)
 create mode 100644 fs/crypto/inline_crypt.c
fs/crypto/Kconfig +6 -0
index 8046d7c7a3e9c..f1f11a6228ebf 100644
@@ -24,3 +24,9 @@ config FS_ENCRYPTION_ALGS
 	select CRYPTO_SHA256
 	select CRYPTO_SHA512
 	select CRYPTO_XTS
+
+config FS_ENCRYPTION_INLINE_CRYPT
+	bool "Enable fscrypt to use inline crypto"
+	depends on FS_ENCRYPTION && BLK_INLINE_ENCRYPTION
+	help
+	  Enable fscrypt to use inline encryption hardware if available.
fs/crypto/Makefile +1 -0
index 232e2bb5a337b..652c7180ec6de 100644
@@ -11,3 +11,4 @@ fscrypto-y := crypto.o \
 	      policy.o
 
 fscrypto-$(CONFIG_BLOCK) += bio.o
+fscrypto-$(CONFIG_FS_ENCRYPTION_INLINE_CRYPT) += inline_crypt.o
fs/crypto/bio.c +50 -0

62 lines not on this page · raw message

fs/crypto/crypto.c +1 -1
index 1ecaac7ee3cb8..263bc676c73dd 100644
@@ -95,7 +95,7 @@ int fscrypt_crypt_block(const struct inode *inode, fscrypt_direction_t rw,
 	DECLARE_CRYPTO_WAIT(wait);
 	struct scatterlist dst, src;
 	struct fscrypt_info *ci = inode->i_crypt_info;
-	struct crypto_skcipher *tfm = ci->ci_ctfm;
+	struct crypto_skcipher *tfm = ci->ci_key.tfm;
 	int res = 0;
 
 	if (WARN_ON_ONCE(len <= 0))
fs/crypto/fname.c +2 -2
index 4c212442a8f7f..0fca2d7a56453 100644
@@ -117,7 +117,7 @@ int fscrypt_fname_encrypt(const struct inode *inode, const struct qstr *iname,
 	struct skcipher_request *req = NULL;
 	DECLARE_CRYPTO_WAIT(wait);
 	const struct fscrypt_info *ci = inode->i_crypt_info;
-	struct crypto_skcipher *tfm = ci->ci_ctfm;
+	struct crypto_skcipher *tfm = ci->ci_key.tfm;
 	union fscrypt_iv iv;
 	struct scatterlist sg;
 	int res;
@@ -170,7 +170,7 @@ static int fname_decrypt(const struct inode *inode,
 	DECLARE_CRYPTO_WAIT(wait);
 	struct scatterlist src_sg, dst_sg;
 	const struct fscrypt_info *ci = inode->i_crypt_info;
-	struct crypto_skcipher *tfm = ci->ci_ctfm;
+	struct crypto_skcipher *tfm = ci->ci_key.tfm;
 	union fscrypt_iv iv;
 	int res;
 
fs/crypto/fscrypt_private.h +107 -13

177 lines not on this page · raw message

fs/crypto/inline_crypt.c new file +339 -0

339 lines not on this page · raw message

fs/crypto/keyring.c +2 -2
index ab41b25d4fa1b..d8ab33f631ba2 100644
@@ -44,8 +44,8 @@ static void free_master_key(struct fscrypt_master_key *mk)
 	wipe_master_key_secret(&mk->mk_secret);
 
 	for (i = 0; i <= __FSCRYPT_MODE_MAX; i++) {
-		crypto_free_skcipher(mk->mk_direct_tfms[i]);
-		crypto_free_skcipher(mk->mk_iv_ino_lblk_64_tfms[i]);
+		fscrypt_destroy_prepared_key(&mk->mk_direct_keys[i]);
+		fscrypt_destroy_prepared_key(&mk->mk_iv_ino_lblk_64_keys[i]);
 	}
 
 	key_put(mk->mk_users);
fs/crypto/keysetup.c +62 -30

177 lines not on this page · raw message

fs/crypto/keysetup_v1.c +7 -9
index 801b48c0cd7f3..59c520b200cb0 100644
@@ -146,7 +146,7 @@ struct fscrypt_direct_key {
 	struct hlist_node		dk_node;
 	refcount_t			dk_refcount;
 	const struct fscrypt_mode	*dk_mode;
-	struct crypto_skcipher		*dk_ctfm;
+	struct fscrypt_prepared_key	dk_key;
 	u8				dk_descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE];
 	u8				dk_raw[FSCRYPT_MAX_KEY_SIZE];
 };
@@ -154,7 +154,7 @@ struct fscrypt_direct_key {
 static void free_direct_key(struct fscrypt_direct_key *dk)
 {
 	if (dk) {
-		crypto_free_skcipher(dk->dk_ctfm);
+		fscrypt_destroy_prepared_key(&dk->dk_key);
 		kzfree(dk);
 	}
 }
@@ -199,6 +199,8 @@ find_or_insert_direct_key(struct fscrypt_direct_key *to_insert,
 			continue;
 		if (ci->ci_mode != dk->dk_mode)
 			continue;
+		if (!fscrypt_is_key_prepared(&dk->dk_key, ci))
+			continue;
 		if (crypto_memneq(raw_key, dk->dk_raw, ci->ci_mode->keysize))
 			continue;
 		/* using existing tfm with same (descriptor, mode, raw_key) */
@@ -231,13 +233,9 @@ fscrypt_get_direct_key(const struct fscrypt_info *ci, const u8 *raw_key)
 		return ERR_PTR(-ENOMEM);
 	refcount_set(&dk->dk_refcount, 1);
 	dk->dk_mode = ci->ci_mode;
-	dk->dk_ctfm = fscrypt_allocate_skcipher(ci->ci_mode, raw_key,
-						ci->ci_inode);
-	if (IS_ERR(dk->dk_ctfm)) {
-		err = PTR_ERR(dk->dk_ctfm);
-		dk->dk_ctfm = NULL;
+	err = fscrypt_prepare_key(&dk->dk_key, raw_key, ci);
+	if (err)
 		goto err_free_dk;
-	}
 	memcpy(dk->dk_descriptor, ci->ci_policy.v1.master_key_descriptor,
 	       FSCRYPT_KEY_DESCRIPTOR_SIZE);
 	memcpy(dk->dk_raw, raw_key, ci->ci_mode->keysize);
@@ -259,7 +257,7 @@ static int setup_v1_file_key_direct(struct fscrypt_info *ci,
 	if (IS_ERR(dk))
 		return PTR_ERR(dk);
 	ci->ci_direct_key = dk;
-	ci->ci_ctfm = dk->dk_ctfm;
+	ci->ci_key = dk->dk_key;
 	return 0;
 }
 
include/linux/fscrypt.h +82 -0

94 lines not on this page · raw message

Signature · 1 line
2.26.2.645.ge9eca65c58-goog
maintainer
#15 lore raw

Re: [PATCH v13 10/12] fscrypt: add inline encryption support

A few minor things to clean up when you resend this after v5.8-rc1:

(You'll have to resolve the conflicts with
https://lore.kernel.org/r/[email protected]
too, but it shouldn't be too hard. Note that I made setup_per_mode_enc_key() use a mutex, like this patch does.)

Quoted 6 lines · Satya Tangirala

Add support for inline encryption to fs/crypto/. With "inline encryption", the block layer handles the decryption/encryption as part of the bio, instead of the filesystem doing the crypto itself via Linux's crypto API. This model is needed in order to take advantage of the inline encryption hardware present on most modern mobile SoCs.

Quoted 8 lines

To use inline encryption, the filesystem needs to be mounted with '-o inlinecrypt'. The contents of any encrypted files will then be encrypted using blk-crypto, instead of using the traditional filesystem-layer crypto. Fscrypt still provides the key and IV to use, and the actual ciphertext on-disk is still the same; therefore it's testable using the existing fscrypt ciphertext verification tests.

Note that since blk-crypto has a fallack to Linux's crypto API, and

"fallack" => "fallback"

fs/crypto/fscrypt_private.h struct fscrypt_symlink_data { · line 194

Quoted 7 lines

struct fscrypt_info {

fs/crypto/fscrypt_private.h +2 -2
-	/* The actual crypto transform used for encryption and decryption */
-	struct crypto_skcipher *ci_ctfm;
+	/* The key in a form prepared for actual encryption/decryption */
+	struct fscrypt_prepared_key	ci_key;

It would be clearer to call this field 'ci_enc_key' instead of 'ci_key'. Since there are several types of fscrypt keys, including the recently added ci_dirhash_key, I've been trying to clarify what type of key is meant when it's ambiguous. E.g. see https://git.kernel.org/torvalds/c/f592efe735a29c76

Quoted 1 line

/* True if the key should be freed when this fscrypt_info is freed */

If taking the above suggestion, this would need "the key" => "ci_enc_key"

fs/crypto/fscrypt_private.h struct fscrypt_info { · line 232

Quoted 5 lines
fs/crypto/fscrypt_private.h +1 -1
 	/*
 	 * If non-NULL, then encryption is done using the master key directly
-	 * and ci_ctfm will equal ci_direct_key->dk_ctfm.
+	 * and ci_key will equal ci_direct_key->dk_key.
 	 */

If taking the above suggestion, this would need "ci_key" => "ci_enc_key"

fs/crypto/fscrypt_private.h extern int fscrypt_hkdf_expand(const struct fscrypt_hkdf *hkdf, u8 context, · line 329

Quoted 3 lines
fs/crypto/fscrypt_private.h +3 -0
+/* inline_crypt.c */
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+extern void fscrypt_select_encryption_impl(struct fscrypt_info *ci);

I'm now trying to consistently not use 'extern' on function declarations. So, can you remove it from the new declarations here and include/linux/fscrypt.h?

Quoted 2 lines

+/**
+ * fscrypt_set_bio_crypt_ctx - prepare a file contents bio for inline encryption

I'm also now trying to consistently include the parentheses in the function names in kerneldoc comments. So:

* fscrypt_set_bio_crypt_ctx() - prepare a file data bio for inline crypto

* fscrypt_set_bio_crypt_ctx_bh() - prepare a file data bio for inline crypto

(similarly for the other new kerneldoc comments)

Make sure to also run

scripts/kernel-doc -v -none fs/crypto/*.{c,h} include/linux/fscrypt.h

to check for new kerneldoc warnings. In fscrypt.git#master I've gotten rid of all the existing ones.

Thanks!

- Eric

maintainer
#16 lore raw

Re: [PATCH v13 10/12] fscrypt: add inline encryption support

In reply to Satya Tangirala

One more thing:

Quoted 23 lines · Satya Tangirala
fs/crypto/inline_crypt.c new file +23 -0
+/* Enable inline encryption for this file if supported. */
+void fscrypt_select_encryption_impl(struct fscrypt_info *ci)
+{
+	const struct inode *inode = ci->ci_inode;
+	struct super_block *sb = inode->i_sb;
+	struct blk_crypto_config crypto_cfg;
+	int num_devs;
+	struct request_queue **devs;
+	int i;
+
+	/* The file must need contents encryption, not filenames encryption */
+	if (!fscrypt_needs_contents_encryption(inode))
+		return;
+
+	/* The crypto mode must be valid */
+	if (ci->ci_mode->blk_crypto_mode == BLK_ENCRYPTION_MODE_INVALID)
+		return;
+
+	/* The filesystem must be mounted with -o inlinecrypt */
+	if (!(sb->s_flags & SB_INLINECRYPT))
+		return;
+
+	/*

fs/crypto/inline_crypt.c line 87

Quoted 10 lines
fs/crypto/inline_crypt.c new file +10 -0
+	 * blk-crypto must support the crypto configuration we'll use for the
+	 * inode on all devices in the sb
+	 */
+	crypto_cfg.crypto_mode = ci->ci_mode->blk_crypto_mode;
+	crypto_cfg.data_unit_size = sb->s_blocksize;
+	crypto_cfg.dun_bytes = fscrypt_get_dun_bytes(ci);
+	num_devs = fscrypt_get_num_devices(sb);
+	devs = kmalloc_array(num_devs, sizeof(*devs), GFP_NOFS);
+	if (!devs)
+		return;

This function needs to return an error code, so that if this memory allocation fails, the error is not ignored.

- Eric

#17 lore raw

PATCH v13 11/12 f2fs: add inline encryption support

In reply to Satya Tangirala

Wire up f2fs to support inline encryption via the helper functions which fs/crypto/ now provides. This includes:

- Adding a mount option 'inlinecrypt' which enables inline encryption on encrypted files where it can be used.

- Setting the bio_crypt_ctx on bios that will be submitted to an inline-encrypted file.

- Not adding logically discontiguous data to bios that will be submitted to an inline-encrypted file.

- Not doing filesystem-layer crypto on inline-encrypted files.

---
 Documentation/filesystems/f2fs.rst |  7 ++-
 fs/f2fs/compress.c                 |  2 +-
 fs/f2fs/data.c                     | 68 +++++++++++++++++++++++++-----
 fs/f2fs/super.c                    | 32 ++++++++++++++
 4 files changed, 97 insertions(+), 12 deletions(-)
Documentation/filesystems/f2fs.rst +6 -1
index 87d794bc75a47..e0e0353f8a498 100644
@@ -254,7 +254,12 @@ compress_extension=%s  Support adding specified extension, so that f2fs can enab
                        on compression extension list and enable compression on
                        these file by default rather than to enable it via ioctl.
                        For other files, we can still enable compression via ioctl.
-====================== ============================================================
+inlinecrypt
+                       Encrypt/decrypt the contents of encrypted files using the
+                       blk-crypto framework rather than filesystem-layer encryption.
+                       This allows the use of inline encryption hardware. The on-disk
+                       format is unaffected. For more details, see
+                       Documentation/block/inline-encryption.rst.
 
 Debugfs Entries
 ===============
fs/f2fs/compress.c +1 -1
index df7b2d15eacde..a19c093711a68 100644
@@ -975,7 +975,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 		.submitted = false,
 		.io_type = io_type,
 		.io_wbc = wbc,
-		.encrypted = f2fs_encrypted_file(cc->inode),
+		.encrypted = fscrypt_inode_uses_fs_layer_crypto(cc->inode),
 	};
 	struct dnode_of_data dn;
 	struct node_info ni;
fs/f2fs/data.c +58 -10

142 lines not on this page · raw message

fs/f2fs/super.c +32 -0
index f2dfc21c6abb0..8ccda50f28888 100644
@@ -138,6 +138,7 @@ enum {
 	Opt_alloc,
 	Opt_fsync,
 	Opt_test_dummy_encryption,
+	Opt_inlinecrypt,
 	Opt_checkpoint_disable,
 	Opt_checkpoint_disable_cap,
 	Opt_checkpoint_disable_cap_perc,
@@ -203,6 +204,7 @@ static match_table_t f2fs_tokens = {
 	{Opt_alloc, "alloc_mode=%s"},
 	{Opt_fsync, "fsync_mode=%s"},
 	{Opt_test_dummy_encryption, "test_dummy_encryption"},
+	{Opt_inlinecrypt, "inlinecrypt"},
 	{Opt_checkpoint_disable, "checkpoint=disable"},
 	{Opt_checkpoint_disable_cap, "checkpoint=disable:%u"},
 	{Opt_checkpoint_disable_cap_perc, "checkpoint=disable:%u%%"},
@@ -788,6 +790,13 @@ static int parse_options(struct super_block *sb, char *options)
 			f2fs_info(sbi, "Test dummy encryption mode enabled");
 #else
 			f2fs_info(sbi, "Test dummy encryption mount option ignored");
+#endif
+			break;
+		case Opt_inlinecrypt:
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+			sb->s_flags |= SB_INLINECRYPT;
+#else
+			f2fs_info(sbi, "inline encryption not supported");
 #endif
 			break;
 		case Opt_checkpoint_disable_cap_perc:
@@ -1583,6 +1592,8 @@ static void default_options(struct f2fs_sb_info *sbi)
 	F2FS_OPTION(sbi).compress_ext_cnt = 0;
 	F2FS_OPTION(sbi).bggc_mode = BGGC_MODE_ON;
 
+	sbi->sb->s_flags &= ~SB_INLINECRYPT;
+
 	set_opt(sbi, INLINE_XATTR);
 	set_opt(sbi, INLINE_DATA);
 	set_opt(sbi, INLINE_DENTRY);
@@ -2427,6 +2438,25 @@ static void f2fs_get_ino_and_lblk_bits(struct super_block *sb,
 	*lblk_bits_ret = 8 * sizeof(block_t);
 }
 
+static int f2fs_get_num_devices(struct super_block *sb)
+{
+	struct f2fs_sb_info *sbi = F2FS_SB(sb);
+
+	if (f2fs_is_multi_device(sbi))
+		return sbi->s_ndevs;
+	return 1;
+}
+
+static void f2fs_get_devices(struct super_block *sb,
+			     struct request_queue **devs)
+{
+	struct f2fs_sb_info *sbi = F2FS_SB(sb);
+	int i;
+
+	for (i = 0; i < sbi->s_ndevs; i++)
+		devs[i] = bdev_get_queue(FDEV(i).bdev);
+}
+
 static const struct fscrypt_operations f2fs_cryptops = {
 	.key_prefix		= "f2fs:",
 	.get_context		= f2fs_get_context,
@@ -2436,6 +2466,8 @@ static const struct fscrypt_operations f2fs_cryptops = {
 	.max_namelen		= F2FS_NAME_LEN,
 	.has_stable_inodes	= f2fs_has_stable_inodes,
 	.get_ino_and_lblk_bits	= f2fs_get_ino_and_lblk_bits,
+	.get_num_devices	= f2fs_get_num_devices,
+	.get_devices		= f2fs_get_devices,
 };
 #endif
 
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
#18 lore raw

PATCH v13 12/12 ext4: add inline encryption support

In reply to Satya Tangirala

From: Eric Biggers <[email protected]>

Wire up ext4 to support inline encryption via the helper functions which fs/crypto/ now provides. This includes:

- Adding a mount option 'inlinecrypt' which enables inline encryption on encrypted files where it can be used.

- Setting the bio_crypt_ctx on bios that will be submitted to an inline-encrypted file.

Note: submit_bh_wbc() in fs/buffer.c also needed to be patched for this part, since ext4 sometimes uses ll_rw_block() on file data.

- Not adding logically discontiguous data to bios that will be submitted to an inline-encrypted file.

- Not doing filesystem-layer crypto on inline-encrypted files.

---
 Documentation/admin-guide/ext4.rst |  6 ++++++
 fs/buffer.c                        |  7 ++++---
 fs/ext4/inode.c                    |  4 ++--
 fs/ext4/page-io.c                  |  6 ++++--
 fs/ext4/readpage.c                 | 11 ++++++++---
 fs/ext4/super.c                    |  9 +++++++++
 6 files changed, 33 insertions(+), 10 deletions(-)
Documentation/admin-guide/ext4.rst +6 -0
index 9443fcef18760..ed997e3766781 100644
@@ -395,6 +395,12 @@ When mounting an ext4 filesystem, the following option are accepted:
         Documentation/filesystems/dax.txt.  Note that this option is
         incompatible with data=journal.
 
+  inlinecrypt
+        Encrypt/decrypt the contents of encrypted files using the blk-crypto
+        framework rather than filesystem-layer encryption. This allows the use
+        of inline encryption hardware. The on-disk format is unaffected. For
+        more details, see Documentation/block/inline-encryption.rst.
+
 Data Mode
 =========
 There are 3 different data modes:
fs/buffer.c +4 -3
index a60f60396cfa0..33827a55b5952 100644
@@ -328,9 +328,8 @@ static void decrypt_bh(struct work_struct *work)
 static void end_buffer_async_read_io(struct buffer_head *bh, int uptodate)
 {
 	/* Decrypt if needed */
-	if (uptodate && IS_ENABLED(CONFIG_FS_ENCRYPTION) &&
-	    IS_ENCRYPTED(bh->b_page->mapping->host) &&
-	    S_ISREG(bh->b_page->mapping->host->i_mode)) {
+	if (uptodate &&
+	    fscrypt_inode_uses_fs_layer_crypto(bh->b_page->mapping->host)) {
 		struct decrypt_bh_ctx *ctx = kmalloc(sizeof(*ctx), GFP_ATOMIC);
 
 		if (ctx) {
@@ -3047,6 +3046,8 @@ static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh,
 	 */
 	bio = bio_alloc(GFP_NOIO, 1);
 
+	fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO);
+
 	bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
 	bio_set_dev(bio, bh->b_bdev);
 	bio->bi_write_hint = write_hint;
fs/ext4/inode.c +2 -2
index 2a4aae6acdcb9..ac20b65766ece 100644
@@ -1088,7 +1088,7 @@ static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len,
 	}
 	if (unlikely(err)) {
 		page_zero_new_buffers(page, from, to);
-	} else if (IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode)) {
+	} else if (fscrypt_inode_uses_fs_layer_crypto(inode)) {
 		for (i = 0; i < nr_wait; i++) {
 			int err2;
 
@@ -3738,7 +3738,7 @@ static int __ext4_block_zero_page_range(handle_t *handle,
 		/* Uhhuh. Read error. Complain and punt. */
 		if (!buffer_uptodate(bh))
 			goto unlock;
-		if (S_ISREG(inode->i_mode) && IS_ENCRYPTED(inode)) {
+		if (fscrypt_inode_uses_fs_layer_crypto(inode)) {
 			/* We expect the key to be set. */
 			BUG_ON(!fscrypt_has_encryption_key(inode));
 			err = fscrypt_decrypt_pagecache_blocks(page, blocksize,
fs/ext4/page-io.c +4 -2
index de6fe969f7737..defd2e10dfd10 100644
@@ -402,6 +402,7 @@ static void io_submit_init_bio(struct ext4_io_submit *io,
 	 * __GFP_DIRECT_RECLAIM is set, see comments for bio_alloc_bioset().
 	 */
 	bio = bio_alloc(GFP_NOIO, BIO_MAX_PAGES);
+	fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO);
 	bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
 	bio_set_dev(bio, bh->b_bdev);
 	bio->bi_end_io = ext4_end_bio;
@@ -418,7 +419,8 @@ static void io_submit_add_bh(struct ext4_io_submit *io,
 {
 	int ret;
 
-	if (io->io_bio && bh->b_blocknr != io->io_next_block) {
+	if (io->io_bio && (bh->b_blocknr != io->io_next_block ||
+			   !fscrypt_mergeable_bio_bh(io->io_bio, bh))) {
 submit_and_retry:
 		ext4_io_submit(io);
 	}
@@ -506,7 +508,7 @@ int ext4_bio_write_page(struct ext4_io_submit *io,
 	 * (e.g. holes) to be unnecessarily encrypted, but this is rare and
 	 * can't happen in the common case of blocksize == PAGE_SIZE.
 	 */
-	if (IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode) && nr_to_submit) {
+	if (fscrypt_inode_uses_fs_layer_crypto(inode) && nr_to_submit) {
 		gfp_t gfp_flags = GFP_NOFS;
 		unsigned int enc_bytes = round_up(len, i_blocksize(inode));
 
fs/ext4/readpage.c +8 -3
index c1769afbf7995..68eac0aeffad3 100644
@@ -195,7 +195,7 @@ static void ext4_set_bio_post_read_ctx(struct bio *bio,
 {
 	unsigned int post_read_steps = 0;
 
-	if (IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode))
+	if (fscrypt_inode_uses_fs_layer_crypto(inode))
 		post_read_steps |= 1 << STEP_DECRYPT;
 
 	if (ext4_need_verity(inode, first_idx))
@@ -232,6 +232,7 @@ int ext4_mpage_readpages(struct address_space *mapping,
 	const unsigned blkbits = inode->i_blkbits;
 	const unsigned blocks_per_page = PAGE_SIZE >> blkbits;
 	const unsigned blocksize = 1 << blkbits;
+	sector_t next_block;
 	sector_t block_in_file;
 	sector_t last_block;
 	sector_t last_block_in_file;
@@ -264,7 +265,8 @@ int ext4_mpage_readpages(struct address_space *mapping,
 		if (page_has_buffers(page))
 			goto confused;
 
-		block_in_file = (sector_t)page->index << (PAGE_SHIFT - blkbits);
+		block_in_file = next_block =
+			(sector_t)page->index << (PAGE_SHIFT - blkbits);
 		last_block = block_in_file + nr_pages * blocks_per_page;
 		last_block_in_file = (ext4_readpage_limit(inode) +
 				      blocksize - 1) >> blkbits;
@@ -364,7 +366,8 @@ int ext4_mpage_readpages(struct address_space *mapping,
 		 * This page will go to BIO.  Do we need to send this
 		 * BIO off first?
 		 */
-		if (bio && (last_block_in_bio != blocks[0] - 1)) {
+		if (bio && (last_block_in_bio != blocks[0] - 1 ||
+			    !fscrypt_mergeable_bio(bio, inode, next_block))) {
 		submit_and_realloc:
 			submit_bio(bio);
 			bio = NULL;
@@ -376,6 +379,8 @@ int ext4_mpage_readpages(struct address_space *mapping,
 			 */
 			bio = bio_alloc(GFP_KERNEL,
 				min_t(int, nr_pages, BIO_MAX_PAGES));
+			fscrypt_set_bio_crypt_ctx(bio, inode, next_block,
+						  GFP_KERNEL);
 			ext4_set_bio_post_read_ctx(bio, inode, page->index);
 			bio_set_dev(bio, bdev);
 			bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9);
fs/ext4/super.c +9 -0
index bf5fcb477f667..fb4a293cac0c3 100644
@@ -1509,6 +1509,7 @@ enum {
 	Opt_journal_path, Opt_journal_checksum, Opt_journal_async_commit,
 	Opt_abort, Opt_data_journal, Opt_data_ordered, Opt_data_writeback,
 	Opt_data_err_abort, Opt_data_err_ignore, Opt_test_dummy_encryption,
+	Opt_inlinecrypt,
 	Opt_usrjquota, Opt_grpjquota, Opt_offusrjquota, Opt_offgrpjquota,
 	Opt_jqfmt_vfsold, Opt_jqfmt_vfsv0, Opt_jqfmt_vfsv1, Opt_quota,
 	Opt_noquota, Opt_barrier, Opt_nobarrier, Opt_err,
@@ -1606,6 +1607,7 @@ static const match_table_t tokens = {
 	{Opt_noinit_itable, "noinit_itable"},
 	{Opt_max_dir_size_kb, "max_dir_size_kb=%u"},
 	{Opt_test_dummy_encryption, "test_dummy_encryption"},
+	{Opt_inlinecrypt, "inlinecrypt"},
 	{Opt_nombcache, "nombcache"},
 	{Opt_nombcache, "no_mbcache"},	/* for backward compatibility */
 	{Opt_removed, "check=none"},	/* mount option from ext2/3 */
@@ -1893,6 +1895,13 @@ static int handle_mount_opt(struct super_block *sb, char *opt, int token,
 	case Opt_nolazytime:
 		sb->s_flags &= ~SB_LAZYTIME;
 		return 1;
+	case Opt_inlinecrypt:
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+		sb->s_flags |= SB_INLINECRYPT;
+#else
+		ext4_msg(sb, KERN_ERR, "inline encryption not supported");
+#endif
+		return 1;
 	}
 
 	for (m = ext4_mount_opts; m->token != Opt_err; m++)
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
maintainer
#19 lore raw
In reply to Satya Tangirala
Quoted 6 lines · Satya Tangirala

This patch series adds support for Inline Encryption to the block layer, UFS, fscrypt, f2fs and ext4. It has been rebased onto linux-block/for-next.

Note that the patches in this series for the block layer (i.e. patches 1, 2, 3, 4 and 5) can be applied independently of the subsequent patches in this series.

Thanks, the (small) changes from v12 look good. As usual, I made this available in git at:

Repo: https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git
Tag: inline-encryption-v13

Jens, do you have any feedback on this patchset? Is there any chance at taking the block layer part (patches 1-5) for 5.8? That part needs to go upstream first, since patches 6-12 depend on them. Then patches 6-12 can go upstream via the SCSI and fscrypt trees in the following release.

- Eric

maintainer Applied
#20 lore raw
Quoted 11 lines · Eric Biggers
Quoted 6 lines · Satya Tangirala

This patch series adds support for Inline Encryption to the block layer, UFS, fscrypt, f2fs and ext4. It has been rebased onto linux-block/for-next.

Note that the patches in this series for the block layer (i.e. patches 1, 2, 3, 4 and 5) can be applied independently of the subsequent patches in this series.

Thanks, the (small) changes from v12 look good. As usual, I made this available in git at:

Quoted 7 lines

Repo: https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git
Tag: inline-encryption-v13

Jens, do you have any feedback on this patchset? Is there any chance at taking the block layer part (patches 1-5) for 5.8? That part needs to go upstream first, since patches 6-12 depend on them. Then patches 6-12 can go upstream via the SCSI and fscrypt trees in the following release.

I have applied 1-5 for 5.8. Small tweak needed in patch 3 due to a header inclusion, but clean apart from that.

Signature · 1 line
Jens Axboe
#21 lore raw
Quoted 2 lines · Jens Axboe

I have applied 1-5 for 5.8. Small tweak needed in patch 3 due to a header inclusion, but clean apart from that.

I looked at this a bit more as it clashed with my outstanding q_usage_counter optimization, and I think we should move the blk_crypto_bio_prep call into blk-mq, similar to what we do about the integrity_prep call. Comments?

---
From b7a78be7de0f39ef972d6a2f97a3982a422bf3ab Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Fri, 15 May 2020 09:32:40 +0200
Subject: block: move blk_crypto_bio_prep into blk_mq_make_request

Currently blk_crypto_bio_prep is called for every block driver, including stacking drivers, which is probably not the right thing to do. Instead move it to blk_mq_make_request, similar to how we handle integrity data. If we ever grow a low-level make_request based driver that wants encryption it will have to call blk_crypto_bio_prep manually, but I really hope we don't grow more non-stacking make_request drivers to start with.

This also means we only need to do the crypto preparation after splitting and bouncing the bio, which means we don't bother allocating the fallback context for a bio that might only be a dummy and gets split or bounced later.

---
 block/blk-core.c | 13 +++++--------
 block/blk-mq.c   |  2 ++
 2 files changed, 7 insertions(+), 8 deletions(-)
block/blk-core.c +5 -8
index 1e97f99735232..ac59afaa26960 100644
@@ -1131,12 +1131,10 @@ blk_qc_t generic_make_request(struct bio *bio)
 			/* Create a fresh bio_list for all subordinate requests */
 			bio_list_on_stack[1] = bio_list_on_stack[0];
 			bio_list_init(&bio_list_on_stack[0]);
-			if (blk_crypto_bio_prep(&bio)) {
-				if (q->make_request_fn)
-					ret = q->make_request_fn(q, bio);
-				else
-					ret = blk_mq_make_request(q, bio);
-			}
+			if (q->make_request_fn)
+				ret = q->make_request_fn(q, bio);
+			else
+				ret = blk_mq_make_request(q, bio);
 
 			blk_queue_exit(q);
 
@@ -1185,8 +1183,7 @@ blk_qc_t direct_make_request(struct bio *bio)
 		return BLK_QC_T_NONE;
 	if (unlikely(bio_queue_enter(bio)))
 		return BLK_QC_T_NONE;
-	if (blk_crypto_bio_prep(&bio))
-		ret = blk_mq_make_request(q, bio);
+	ret = blk_mq_make_request(q, bio);
 	blk_queue_exit(q);
 	return ret;
 }
block/blk-mq.c +2 -0
index d2962863e629f..0b5a0fa0d124b 100644
@@ -2033,6 +2033,8 @@ blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 	blk_queue_bounce(q, &bio);
 	__blk_queue_split(q, &bio, &nr_segs);
 
+	if (!blk_crypto_bio_prep(&bio))
+		return BLK_QC_T_NONE;
 	if (!bio_integrity_prep(bio))
 		return BLK_QC_T_NONE;
 
Signature · 1 line
2.26.2
#22 lore raw
Quoted 15 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe

I have applied 1-5 for 5.8. Small tweak needed in patch 3 due to a header inclusion, but clean apart from that.

I looked at this a bit more as it clashed with my outstanding q_usage_counter optimization, and I think we should move the blk_crypto_bio_prep call into blk-mq, similar to what we do about the integrity_prep call. Comments?

---
From b7a78be7de0f39ef972d6a2f97a3982a422bf3ab Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Fri, 15 May 2020 09:32:40 +0200
Subject: block: move blk_crypto_bio_prep into blk_mq_make_request

Quoted 6 lines

Currently blk_crypto_bio_prep is called for every block driver, including stacking drivers, which is probably not the right thing to do. Instead move it to blk_mq_make_request, similar to how we handle integrity data. If we ever grow a low-level make_request based driver that wants encryption it will have to call blk_crypto_bio_prep manually, but I really hope we don't grow more non-stacking make_request drivers to start with.

One of the nice things about the current design is that regardless of what request queue an FS sends an encrypted bio to, blk-crypto will be able to handle the encryption (whether by using hardware inline encryption, or using the blk-crypto-fallback). The FS itself does not need to worry about what the request queue is.

But if we move blk_crypto_bio_prep into blk_mq_make_request, the FS loses this ability to not care about the underlying request queue - it can no longer send a bio with an encryption context to queue such that q->make_request_fn != blk_mq_make_request_fn. To restore that ability, we'll need to add calls to blk_crypto_bio_prep to every possible make_request_fn (although yes, if we do decide to add the call to blk_crypto_bio_prep in multiple places, I think it'll be fine to only add it to the non-stacking make_request_fns).

Also, I tried to look through the patch with the q_usage_counter optimization - is it this one?

[PATCH 4/4] block: allow blk_mq_make_request to consume the q_usage_counter reference

Quoted 59 lines

This also means we only need to do the crypto preparation after splitting and bouncing the bio, which means we don't bother allocating the fallback context for a bio that might only be a dummy and gets split or bounced later.

---
 block/blk-core.c | 13 +++++--------
 block/blk-mq.c   |  2 ++
 2 files changed, 7 insertions(+), 8 deletions(-)
block/blk-core.c +5 -8
index 1e97f99735232..ac59afaa26960 100644
@@ -1131,12 +1131,10 @@ blk_qc_t generic_make_request(struct bio *bio)
 			/* Create a fresh bio_list for all subordinate requests */
 			bio_list_on_stack[1] = bio_list_on_stack[0];
 			bio_list_init(&bio_list_on_stack[0]);
-			if (blk_crypto_bio_prep(&bio)) {
-				if (q->make_request_fn)
-					ret = q->make_request_fn(q, bio);
-				else
-					ret = blk_mq_make_request(q, bio);
-			}
+			if (q->make_request_fn)
+				ret = q->make_request_fn(q, bio);
+			else
+				ret = blk_mq_make_request(q, bio);
 
 			blk_queue_exit(q);
 
@@ -1185,8 +1183,7 @@ blk_qc_t direct_make_request(struct bio *bio)
 		return BLK_QC_T_NONE;
 	if (unlikely(bio_queue_enter(bio)))
 		return BLK_QC_T_NONE;
-	if (blk_crypto_bio_prep(&bio))
-		ret = blk_mq_make_request(q, bio);
+	ret = blk_mq_make_request(q, bio);
 	blk_queue_exit(q);
 	return ret;
 }
block/blk-mq.c +2 -0
index d2962863e629f..0b5a0fa0d124b 100644
@@ -2033,6 +2033,8 @@ blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 	blk_queue_bounce(q, &bio);
 	__blk_queue_split(q, &bio, &nr_segs);
 
+	if (!blk_crypto_bio_prep(&bio))
+		return BLK_QC_T_NONE;
 	if (!bio_integrity_prep(bio))
 		return BLK_QC_T_NONE;
 
Signature · 1 line
2.26.2
#23 lore raw
Quoted 5 lines · Satya Tangirala

One of the nice things about the current design is that regardless of what request queue an FS sends an encrypted bio to, blk-crypto will be able to handle the encryption (whether by using hardware inline encryption, or using the blk-crypto-fallback). The FS itself does not need to worry about what the request queue is.

True. Which just makes me despise that design with the pointless fallback even more..

maintainer
#24 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Satya Tangirala

One of the nice things about the current design is that regardless of what request queue an FS sends an encrypted bio to, blk-crypto will be able to handle the encryption (whether by using hardware inline encryption, or using the blk-crypto-fallback). The FS itself does not need to worry about what the request queue is.

True. Which just makes me despise that design with the pointless fallback even more..

The fallback is actually really useful. First, for testing: it allows all the filesystem code that uses inline crypto to be tested using gce-xfstests and kvm-xfstests, so that it's covered by the usual ext4 and f2fs regression testing and it's much easier to develop patches for. It also allowed us to enable the inlinecrypt mount option in Cuttlefish, which is the virtual Android device used to test the Android common kernels. So, it gets the kernel test platform as similar to a real Android device as possible.

Ideally we'd implement virtualized inline encryption as you suggested. But these platforms use a mix of VMM's (QEMU, GCE, and crosvm) and storage types (virtio-blk, virtio-scsi, and maybe others; none of these even have an inline encryption standard defined yet). So it's not currently feasible.

Second, it creates a clean design where users can just use blk-crypto, and not have to implement a second encryption implementation. For example, I'd eventually like to switch fscrypt over to just use blk-crypto. That would remove the duplicate code that you're concerned about. It would also make it much easier to implement direct I/O support in fscrypt which is something that people have been requesting for a long time.

The reason the fscrypt conversion isn't yet part of the patchset is just that I consider it super important that we don't cause any regressions in fscrypt and that it doesn't use inline encryption hardware by default. So it's not quite time to switch over for real yet, especially while the current patches are still pending upstream. But I think it will come eventually, especially if we see that most Linux distros are enabling CONFIG_BLK_INLINE_ENCRYPTION anyway. The inlinecrypt mount option will thten start controlling whether blk-crypto is allowed to to use real hardware or not, not whether blk-crypto is used or not.

Also, in the coming months we're planning to implement filesystem metadata encryption that is properly integrated with the fscrypt key derivation so that file contents don't have to be encrypted twice (as would be the case with dm-crypt + fscrypt). That's going to involve adding lots of encryption hooks to code in ext4, f2fs, and places like fs/buffer.c. blk-crypto-fallback is super helpful for this, since it will allow us to simply call fscrypt_set_bio_crypt_ctx() everywhere, and not have to both do that *and* implement a second case where we do all the crypto work scheduling, bounce page allocation, crypto API calls, etc. at the filesystem level.

- Eric

#25 lore raw
Quoted 8 lines · Eric Biggers

The fallback is actually really useful. First, for testing: it allows all the filesystem code that uses inline crypto to be tested using gce-xfstests and kvm-xfstests, so that it's covered by the usual ext4 and f2fs regression testing and it's much easier to develop patches for. It also allowed us to enable the inlinecrypt mount option in Cuttlefish, which is the virtual Android device used to test the Android common kernels. So, it gets the kernel test platform as similar to a real Android device as possible.

Quoted 4 lines

Ideally we'd implement virtualized inline encryption as you suggested. But these platforms use a mix of VMM's (QEMU, GCE, and crosvm) and storage types (virtio-blk, virtio-scsi, and maybe others; none of these even have an inline encryption standard defined yet). So it's not currently feasible.

Not that you don't need to implement it in the hypervisor. You can also trivially wire up for things like null_blk.

Quoted 2 lines

Second, it creates a clean design where users can just use blk-crypto, and not have to implement a second encryption implementation.

And I very much disagree about that being a clean implementation. It is fine if the user doesn't care, but you should catch this before hitting the block stack and do the encryption there without hardware blk-crypt support.

#26 lore raw
In reply to Eric Biggers

Eric,

Quoted 2 lines

Then patches 6-12 can go upstream via the SCSI and fscrypt trees in the following release.

I'd like our UFS folks to review the UFS patches. But otherwise no objection from me.

Signature · 7 lines
Martin K. Petersen	Oracle Linux Engineering


_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel