Skip to contentSkip to conversation

Inline Encryption Support

PATCH Superseded Satya Tangirala 4 participants 14 messages Jul 10, 2019 – Aug 2, 2019

Subsystems (7)
Documentation VFS SCSI Block layer +3 more subsystems

v3 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v3 cover Inline Encryption Support

This patch series adds support for Inline Encryption to the block layer, UFS, fscrypt and f2fs.

Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device. Inline Encryption hardware has become increasingly common, and we want to support it in the kernel.

Inline Encryption hardware implementations often function around the concept of a limited number of "keyslots", which can hold an encryption context each. The storage device can be directed to en/decrypt any particular request with the encryption context stored in any particular keyslot.

Patch 1 introduces a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that blk-crypto (introduced in Path 3), will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.

Patch 2 introduces struct bio_crypt_ctx, and a ptr to one in struct bio, which allows struct bio to represent an encryption context that can be passed down the storage stack from the filesystem layer to the storage driver.

Patch 3 introduces blk-crypto. Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware. Given that blk-crypto works as a software fallback, we are considering removing file content en/decryption from fscrypt and simply using blk-crypto in a future patch. For more details on blk-crypto, refer to Documentation/block/blk-crypto.txt.

Patches 4-6 add support for inline encryption into the UFS driver according to the JEDEC UFS HCI v2.1 specification. Inline encryption support for other drivers (like eMMC) may be added in the same way - the device driver should set up a Keyslot Manager in the device's request_queue (refer to the UFS crypto additions in ufshcd-crypto.c and ufshcd.c for an example).

Patches 7 and 8 add support to fscrypt and f2fs, so that we have a complete stack that can make use of inline encryption.

There have been a few patch sets addressing Inline Encryption Support in the past. Briefly, this patch set differs from those as follows:

1) "crypto: qce: ice: Add support for Inline Crypto Engine" is specific to certain hardware, while our patch set's Inline Encryption support for UFS is implemented according to the JEDEC UFS specification.

2) "scsi: ufs: UFS Host Controller crypto changes" registers inline encryption support as a kernel crypto algorithm. Our patch views inline encryption as being fundamentally different from a generic crypto provider (in that inline encryption is tied to a device), and so does not use the kernel crypto API to represent inline encryption hardware.

3) "scsi: ufs: add real time/inline crypto support to UFS HCD" requires the device mapper to work - our patch does not.

Changes v2 => v3:
- Overhauled keyslot manager's get keyslot logic and optimized LRU.
- Block crypto en/decryption fallback now supports data unit sizes that divide each of the bio's segment's lengths (instead of requiring each segment's length to be the same as the data unit size).
- fscrypt master key is now keyed additionally by super_block and ci_ctfm != NULL.
- all references of "hw encryption" are replaced by inline encryption.
- address various other review comments from Eric.

Changes v1 => v2:
- Block layer and UFS changes are split into 3 patches each.
- We now only have a ptr to a struct bio_crypt_ctx in struct bio, instead of the struct itself.
- struct bio_crypt_ctx no longer has flags.
- blk-crypto now correctly handles the case when it fails to init (because of insufficient memory), but kernel continues to boot.
- ufshcd-crypto now works on big endian cpus.
- Many cleanups.

Satya Tangirala (8):
block: Keyslot Manager for Inline Encryption
block: Add encryption context to struct bio
block: blk-crypto for Inline Encryption
scsi: ufs: UFS driver v2.1 spec crypto additions
scsi: ufs: UFS crypto API
scsi: ufs: Add inline encryption support to UFS
fscrypt: wire up fscrypt to use blk-crypto
f2fs: Wire up f2fs to use inline encryption via fscrypt

 Documentation/block/inline-encryption.txt | 185 +++++++
 block/Kconfig                             |   8 +
 block/Makefile                            |   2 +
 block/bio-crypt-ctx.c                     | 122 +++++
 block/bio.c                               |  16 +-
 block/blk-core.c                          |  11 +-
 block/blk-crypto.c                        | 585 ++++++++++++++++++++++
 block/blk-merge.c                         |  34 +-
 block/bounce.c                            |   9 +-
 block/keyslot-manager.c                   | 314 ++++++++++++
 drivers/md/dm.c                           |  15 +-
 drivers/scsi/ufs/Kconfig                  |  10 +
 drivers/scsi/ufs/Makefile                 |   1 +
 drivers/scsi/ufs/ufshcd-crypto.c          | 435 ++++++++++++++++
 drivers/scsi/ufs/ufshcd-crypto.h          |  86 ++++
 drivers/scsi/ufs/ufshcd.c                 |  84 +++-
 drivers/scsi/ufs/ufshcd.h                 |  29 ++
 drivers/scsi/ufs/ufshci.h                 |  67 ++-
 fs/crypto/Kconfig                         |   6 +
 fs/crypto/bio.c                           | 138 ++++-
 fs/crypto/crypto.c                        |   4 +
 fs/crypto/fscrypt_private.h               |  11 +
 fs/crypto/keyinfo.c                       |  94 +++-
 fs/crypto/policy.c                        |  10 +
 fs/f2fs/data.c                            |  83 ++-
 fs/f2fs/super.c                           |  13 +-
 include/linux/bio.h                       | 208 ++++++++
 include/linux/blk-crypto.h                |  40 ++
 include/linux/blk_types.h                 |   7 +
 include/linux/blkdev.h                    |   6 +
 include/linux/fscrypt.h                   |  62 +++
 include/linux/keyslot-manager.h           |  75 +++
 include/uapi/linux/fs.h                   |   3 +-
 33 files changed, 2697 insertions(+), 76 deletions(-)
 create mode 100644 Documentation/block/inline-encryption.txt
 create mode 100644 block/bio-crypt-ctx.c
 create mode 100644 block/blk-crypto.c
 create mode 100644 block/keyslot-manager.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
 create mode 100644 include/linux/blk-crypto.h
 create mode 100644 include/linux/keyslot-manager.h
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#1 lore raw

PATCH v3 1/8 block: Keyslot Manager for Inline Encryption

Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device.

Inline Encryption hardware implementations often function around the concept of "keyslots". These implementations often have a limited number of "keyslots", each of which can hold an encryption context (we say that an encryption context can be "programmed" into a keyslot). Requests made to the storage device may have a keyslot associated with them, and the inline encryption hardware will en/decrypt the data in the requests using the encryption context programmed into that associated keyslot. As keyslots are limited, and programming keys may be expensive in many implementations, and multiple requests may use exactly the same encryption contexts, we introduce a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that upper layers will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.

---
 block/Kconfig                   |   8 +
 block/Makefile                  |   1 +
 block/keyslot-manager.c         | 314 ++++++++++++++++++++++++++++++++
 include/linux/bio.h             |  11 ++
 include/linux/blkdev.h          |   6 +
 include/linux/keyslot-manager.h |  75 ++++++++
 6 files changed, 415 insertions(+)
 create mode 100644 block/keyslot-manager.c
 create mode 100644 include/linux/keyslot-manager.h
block/Kconfig +8 -0
index 2466dcc3ef1d..fef7c60f6d8e 100644
@@ -164,6 +164,14 @@ config BLK_SED_OPAL
 	Enabling this option enables users to setup/unlock/lock
 	Locking ranges for SED devices using the Opal protocol.
 
+config BLK_INLINE_ENCRYPTION
+	bool "Enable inline encryption support in block layer"
+	help
+	  Build the blk-crypto subsystem.
+	  Enabling this lets the block layer handle encryption,
+	  so users can take advantage of inline encryption
+	  hardware if present.
+
 menu "Partition Types"
 
 source "block/partitions/Kconfig"
block/Makefile +1 -0
index eee1b4ceecf9..a72abd61b220 100644
@@ -35,3 +35,4 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o
block/keyslot-manager.c new file +314 -0

314 lines not on this page · raw message

include/linux/bio.h +11 -0
index f87abaa898f0..ef9c6e2e92bc 100644
@@ -561,6 +561,17 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
 }
 #endif
 
+enum blk_crypt_mode_num {
+	BLK_ENCRYPTION_MODE_AES_256_XTS	= 0,
+	/*
+	 * TODO: Support these too
+	 * BLK_ENCRYPTION_MODE_AES_256_CTS	= 1,
+	 * BLK_ENCRYPTION_MODE_AES_128_CBC	= 2,
+	 * BLK_ENCRYPTION_MODE_AES_128_CTS	= 3,
+	 * BLK_ENCRYPTION_MODE_ADIANTUM		= 4,
+	 */
+};
+
 /*
  * BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
  *
include/linux/blkdev.h +6 -0
index 592669bcc536..3bb4e7e1bd78 100644
@@ -43,6 +43,7 @@ struct pr_ops;
 struct rq_qos;
 struct blk_queue_stats;
 struct blk_stat_callback;
+struct keyslot_manager;
 
 #define BLKDEV_MIN_RQ	4
 #define BLKDEV_MAX_RQ	128	/* Default maximum */
@@ -473,6 +474,11 @@ struct request_queue {
 	unsigned int		dma_pad_mask;
 	unsigned int		dma_alignment;
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	/* Inline crypto capabilities */
+	struct keyslot_manager *ksm;
+#endif
+
 	unsigned int		rq_timeout;
 	int			poll_nsec;
 
include/linux/keyslot-manager.h new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#2 lore raw

PATCH v3 2/8 block: Add encryption context to struct bio

In reply to Satya Tangirala

We must have some way of letting a storage device driver know what encryption context it should use for en/decrypting a request. However, it's the filesystem/fscrypt that knows about and manages encryption contexts. As such, when the filesystem layer submits a bio to the block layer, and this bio eventually reaches a device driver with support for inline encryption, the device driver will need to have been told the encryption context for that bio.

We want to communicate the encryption context from the filesystem layer to the storage device along with the bio, when the bio is submitted to the block layer. To do this, we add a struct bio_crypt_ctx to struct bio, which can represent an encryption context (note that we can't use the bi_private field in struct bio to do this because that field does not function to pass information across layers in the storage stack). We also introduce various functions to manipulate the bio_crypt_ctx and make the bio/request merging logic aware of the bio_crypt_ctx.

---
 block/Makefile            |   2 +-
 block/bio-crypt-ctx.c     | 117 +++++++++++++++++++++++
 block/bio.c               |  11 ++-
 block/blk-merge.c         |  34 ++++++-
 block/bounce.c            |   9 +-
 drivers/md/dm.c           |  15 ++-
 include/linux/bio.h       | 190 ++++++++++++++++++++++++++++++++++++++
 include/linux/blk_types.h |   7 ++
 8 files changed, 371 insertions(+), 14 deletions(-)
 create mode 100644 block/bio-crypt-ctx.c
block/Makefile +1 -1
index a72abd61b220..4147ffa63631 100644
@@ -35,4 +35,4 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o bio-crypt-ctx.o
block/bio-crypt-ctx.c new file +117 -0

117 lines not on this page · raw message

block/bio.c +9 -2
index ce797d73bb43..2a272cda6dfa 100644
@@ -240,6 +240,7 @@ static void bio_free(struct bio *bio)
 	struct bio_set *bs = bio->bi_pool;
 	void *p;
 
+	bio_crypt_free_ctx(bio);
 	bio_uninit(bio);
 
 	if (bs) {
@@ -612,6 +613,7 @@ EXPORT_SYMBOL(__bio_clone_fast);
 struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)
 {
 	struct bio *b;
+	int ret;
 
 	b = bio_alloc_bioset(gfp_mask, 0, bs);
 	if (!b)
@@ -619,9 +621,13 @@ struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)
 
 	__bio_clone_fast(b, bio);
 
-	if (bio_integrity(bio)) {
-		int ret;
+	ret = bio_crypt_clone(b, bio, gfp_mask);
+	if (ret < 0) {
+		bio_put(b);
+		return NULL;
+	}
 
+	if (bio_integrity(bio)) {
 		ret = bio_integrity_clone(b, bio, gfp_mask);
 
 		if (ret < 0) {
@@ -1022,6 +1028,7 @@ void bio_advance(struct bio *bio, unsigned bytes)
 	if (bio_integrity(bio))
 		bio_integrity_advance(bio, bytes);
 
+	bio_crypt_advance(bio, bytes);
 	bio_advance_iter(bio, &bio->bi_iter, bytes);
 }
 EXPORT_SYMBOL(bio_advance);
block/blk-merge.c +30 -4
index 17713d7d98d5..f416e7f38270 100644
@@ -531,6 +531,9 @@ static inline int ll_new_hw_segment(struct request_queue *q,
 	if (blk_integrity_merge_bio(q, req, bio) == false)
 		goto no_merge;
 
+	if (WARN_ON(!bio_crypt_ctx_compatible(bio, req->bio)))
+		goto no_merge;
+
 	/*
 	 * This will form the start of a new hw segment.  Bump both
 	 * counters.
@@ -696,8 +699,13 @@ static enum elv_merge blk_try_req_merge(struct request *req,
 {
 	if (blk_discard_mergable(req))
 		return ELEVATOR_DISCARD_MERGE;
-	else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next))
+	else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next)) {
+		if (!bio_crypt_ctx_back_mergeable(
+			req->bio, blk_rq_sectors(req), next->bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
 		return ELEVATOR_BACK_MERGE;
+	}
 
 	return ELEVATOR_NO_MERGE;
 }
@@ -733,6 +741,9 @@ static struct request *attempt_merge(struct request_queue *q,
 	if (req->ioprio != next->ioprio)
 		return NULL;
 
+	if (!bio_crypt_ctx_compatible(req->bio, next->bio))
+		return NULL;
+
 	/*
 	 * If we are allowed to merge, then append bio list
 	 * from next to rq and release next. merge_requests_fn
@@ -865,16 +876,31 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
 	if (rq->ioprio != bio_prio(bio))
 		return false;
 
+	/* Only merge if the crypt contexts are compatible */
+	if (!bio_crypt_ctx_compatible(bio, rq->bio))
+		return false;
+
 	return true;
 }
 
 enum elv_merge blk_try_merge(struct request *rq, struct bio *bio)
 {
-	if (blk_discard_mergable(rq))
+	if (blk_discard_mergable(rq)) {
 		return ELEVATOR_DISCARD_MERGE;
-	else if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector)
+	} else if (blk_rq_pos(rq) + blk_rq_sectors(rq) ==
+		   bio->bi_iter.bi_sector) {
+		if (!bio_crypt_ctx_back_mergeable(rq->bio,
+						  blk_rq_sectors(rq), bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
 		return ELEVATOR_BACK_MERGE;
-	else if (blk_rq_pos(rq) - bio_sectors(bio) == bio->bi_iter.bi_sector)
+	} else if (blk_rq_pos(rq) - bio_sectors(bio) ==
+		   bio->bi_iter.bi_sector) {
+		if (!bio_crypt_ctx_back_mergeable(bio,
+						  bio_sectors(bio), rq->bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
 		return ELEVATOR_FRONT_MERGE;
+	}
 	return ELEVATOR_NO_MERGE;
 }
block/bounce.c +7 -2
index f8ed677a1bf7..1307238d3951 100644
@@ -220,6 +220,7 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,
 	struct bvec_iter iter;
 	struct bio_vec bv;
 	struct bio *bio;
+	int ret;
 
 	/*
 	 * Pre immutable biovecs, __bio_clone() used to just do a memcpy from
@@ -267,9 +268,13 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,
 		break;
 	}
 
-	if (bio_integrity(bio_src)) {
-		int ret;
+	ret = bio_crypt_clone(bio, bio_src, gfp_mask);
+	if (ret < 0) {
+		bio_put(bio);
+		return NULL;
+	}
 
+	if (bio_integrity(bio_src)) {
 		ret = bio_integrity_clone(bio, bio_src, gfp_mask);
 		if (ret < 0) {
 			bio_put(bio);
drivers/md/dm.c +10 -5
index 5475081dcbd6..22517d03aa39 100644
@@ -1324,12 +1324,15 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
 		     sector_t sector, unsigned len)
 {
 	struct bio *clone = &tio->clone;
+	int ret;
 
 	__bio_clone_fast(clone, bio);
 
-	if (bio_integrity(bio)) {
-		int r;
+	ret = bio_crypt_clone(clone, bio, GFP_NOIO);
+	if (ret < 0)
+		return ret;
 
+	if (bio_integrity(bio)) {
 		if (unlikely(!dm_target_has_integrity(tio->ti->type) &&
 			     !dm_target_passes_integrity(tio->ti->type))) {
 			DMWARN("%s: the target %s doesn't support integrity data.",
@@ -1338,9 +1341,11 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
 			return -EIO;
 		}
 
-		r = bio_integrity_clone(clone, bio, GFP_NOIO);
-		if (r < 0)
-			return r;
+		ret = bio_integrity_clone(clone, bio, GFP_NOIO);
+		if (ret < 0) {
+			bio_crypt_free_ctx(clone);
+			return ret;
+		}
 	}
 
 	bio_advance(clone, to_bytes(sector - clone->bi_iter.bi_sector));
include/linux/bio.h +190 -0

196 lines not on this page · raw message

include/linux/blk_types.h +7 -0
index 95202f80676c..0b794fe3530a 100644
@@ -137,6 +137,8 @@ static inline void bio_issue_init(struct bio_issue *issue,
 			((u64)size << BIO_ISSUE_SIZE_SHIFT));
 }
 
+struct bio_crypt_ctx;
+
 /*
  * main unit of I/O for the block layer and lower layers (ie drivers and
  * stacking drivers)
@@ -175,6 +177,11 @@ struct bio {
 	struct blkcg_gq		*bi_blkg;
 	struct bio_issue	bi_issue;
 #endif
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	struct bio_crypt_ctx	*bi_crypt_context;
+#endif
+
 	union {
 #if defined(CONFIG_BLK_DEV_INTEGRITY)
 		struct bio_integrity_payload *bi_integrity; /* data integrity */
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
maintainer
#3 lore raw

Re: [PATCH 2/8] block: Add encryption context to struct bio

Quoted 8 lines · Satya Tangirala

We must have some way of letting a storage device driver know what encryption context it should use for en/decrypting a request. However, it's the filesystem/fscrypt that knows about and manages encryption contexts. As such, when the filesystem layer submits a bio to the block layer, and this bio eventually reaches a device driver with support for inline encryption, the device driver will need to have been told the encryption context for that bio.

Quoted 8 lines

We want to communicate the encryption context from the filesystem layer to the storage device along with the bio, when the bio is submitted to the block layer. To do this, we add a struct bio_crypt_ctx to struct bio, which can represent an encryption context (note that we can't use the bi_private field in struct bio to do this because that field does not function to pass information across layers in the storage stack). We also introduce various functions to manipulate the bio_crypt_ctx and make the bio/request merging logic aware of the bio_crypt_ctx.

A few minor comments below. Don't see something totally horrible with your approach.

Quoted 10 lines
block/bio-crypt-ctx.c new file +4 -0
new file mode 100644
index 000000000000..8b884ef32007
@@ -0,0 +1,117 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2019 Google LLC
+ */

block/bio-crypt-ctx.c line 14

Quoted 10 lines
block/bio-crypt-ctx.c new file +10 -0
+
+#include <linux/bio.h>
+#include <linux/blkdev.h>
+#include <linux/slab.h>
+#include <linux/keyslot-manager.h>
+
+struct bio_crypt_ctx *bio_crypt_alloc_ctx(gfp_t gfp_mask)
+{
+	return kzalloc(sizeof(struct bio_crypt_ctx), gfp_mask);
+}

I think you'll want this to be mempool backed.

block/bio-crypt-ctx.c line 27

Quoted 13 lines
block/bio-crypt-ctx.c new file +13 -0
+EXPORT_SYMBOL(bio_crypt_alloc_ctx);
+
+void bio_crypt_free_ctx(struct bio *bio)
+{
+	kzfree(bio->bi_crypt_context);
+	bio->bi_crypt_context = NULL;
+}
+EXPORT_SYMBOL(bio_crypt_free_ctx);
+
+int bio_crypt_clone(struct bio *dst, struct bio *src, gfp_t gfp_mask)
+{
+	if (!bio_is_encrypted(src))
+		return 0;

Was going to suggest dumping this helper, but that won't work for the case where inline encryption isn't enabled. How about renaming it so it is easier to grok what it tests, bio_has_crypt_ctx() or something like that.

block/bio-crypt-ctx.c line 31

Quoted 3 lines
block/bio-crypt-ctx.c new file +3 -0
+	dst->bi_crypt_context = bio_crypt_alloc_ctx(gfp_mask);
+	if (!dst->bi_crypt_context)
+		return -ENOMEM;

That's why you need the mempool...

block/blk-merge.c static inline int ll_new_hw_segment(struct request_queue *q, · line 535

Quoted 10 lines
block/blk-merge.c +2 -0
index 17713d7d98d5..f416e7f38270 100644
@@ -531,6 +531,9 @@ static inline int ll_new_hw_segment(struct request_queue *q,
  	if (blk_integrity_merge_bio(q, req, bio) == false)
  		goto no_merge;
 
+	if (WARN_ON(!bio_crypt_ctx_compatible(bio, req->bio)))
+		goto no_merge;

This is really a debug check, as that shouldn't happen unless you have a bug in your merge helpers. I think we can do one of two things here:

1) Rely on this check only for merging, similarly to what the integrity code does. This means the WARN() goes away and (most of) the other merge checks can go away.

2) Keep it, but change it to a WARN_ON_ONCE().

block/blk-merge.c static enum elv_merge blk_try_req_merge(struct request *req, · line 708

Quoted 12 lines
block/blk-merge.c +6 -1
@@ -696,8 +699,13 @@ static enum elv_merge blk_try_req_merge(struct request *req,
  {
  	if (blk_discard_mergable(req))
  		return ELEVATOR_DISCARD_MERGE;
-	else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next))
+	else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next)) {
+		if (!bio_crypt_ctx_back_mergeable(
+			req->bio, blk_rq_sectors(req), next->bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
  		return ELEVATOR_BACK_MERGE;
+	}

Weird line breaks aside, see above comment. More in this file.

Quoted 21 lines
include/linux/bio.h +13 -0
index ef9c6e2e92bc..4e664d6441d5 100644
@@ -572,6 +572,196 @@ enum blk_crypt_mode_num {
  	 */
  };
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+struct bio_crypt_ctx {
+	int keyslot;
+	u8 *raw_key;
+	enum blk_crypt_mode_num crypt_mode;
+	u64 data_unit_num;
+	unsigned int data_unit_size_bits;
+
+	/*
+	 * The keyslot manager where the key has been programmed
+	 * with keyslot.
+	 */
+	struct keyslot_manager *processing_ksm;

include/linux/bio.h enum blk_crypt_mode_num { · line 597

Quoted 10 lines
include/linux/bio.h +10 -0
+
+	/*
+	 * Copy of the bvec_iter when this bio was submitted.
+	 * We only want to en/decrypt the part of the bio
+	 * as described by the bvec_iter upon submission because
+	 * bio might be split before being resubmitted
+	 */
+	struct bvec_iter crypt_iter;
+	u64 sw_data_unit_num;
+};

[snip]

Let's move that to a separate file, with the other crypt specific bits. Just include it from bio.h.

include/linux/bio.h enum blk_crypt_mode_num { · line 685

Quoted 11 lines
include/linux/bio.h +11 -0
+static inline u64 bio_crypt_data_unit_num(struct bio *bio)
+{
+	WARN_ON(!bio_is_encrypted(bio));
+	return bio->bi_crypt_context->data_unit_num;
+}
+
+static inline u64 bio_crypt_sw_data_unit_num(struct bio *bio)
+{
+	WARN_ON(!bio_is_encrypted(bio));
+	return bio->bi_crypt_context->sw_data_unit_num;
+}

These WARN()'s are a bit weird.

include/linux/bio.h enum blk_crypt_mode_num { · line 743

Quoted 5 lines
include/linux/bio.h +5 -0
+static inline u64 bio_crypt_data_unit_num(struct bio *bio)
+{
+	WARN_ON(1);
+	return 0;
+}

And this one definitely needs to go.

include/linux/blk_types.h static inline void bio_issue_init(struct bio_issue *issue, · line 140

Quoted 9 lines
include/linux/blk_types.h +1 -0
index 95202f80676c..0b794fe3530a 100644
@@ -137,6 +137,8 @@ static inline void bio_issue_init(struct bio_issue *issue,
  			((u64)size << BIO_ISSUE_SIZE_SHIFT));
  }
 
+struct bio_crypt_ctx;

Place this with the other forward declarations.

Signature · 8 lines
Jens Axboe



_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
#4 lore raw

PATCH v3 3/8 block: blk-crypto for Inline Encryption

In reply to Satya Tangirala

We introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.

Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to Documentation/block/blk-crypto.txt.

Known issues:
1) We're allocating crypto_skcipher in blk_crypto_keyslot_program, which uses GFP_KERNEL to allocate memory, but this function is on the write path for IO - we need to add support for specifying a different flags to the crypto API.

---
 Documentation/block/inline-encryption.txt | 185 +++++++
 block/Makefile                            |   3 +-
 block/bio-crypt-ctx.c                     |   7 +-
 block/bio.c                               |   5 +
 block/blk-core.c                          |  11 +-
 block/blk-crypto.c                        | 585 ++++++++++++++++++++++
 include/linux/bio.h                       |   7 +
 include/linux/blk-crypto.h                |  40 ++
 8 files changed, 840 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/block/inline-encryption.txt
 create mode 100644 block/blk-crypto.c
 create mode 100644 include/linux/blk-crypto.h
Documentation/block/inline-encryption.txt new file +185 -0
new file mode 100644
index 000000000000..96a7983a117d
@@ -0,0 +1,185 @@
+BLK-CRYPTO and KEYSLOT MANAGER
+===========================
+
+CONTENTS
+1. Objective
+2. Constraints and notes
+3. Design
+4. Blk-crypto
+ 4-1 What does blk-crypto do on bio submission
+5. Layered Devices
+6. Future optimizations for layered devices
+
+1. Objective
+============
+
+We want to support inline encryption (IE) in the kernel.
+To allow for testing, we also want a software fallback when actual
+IE hardware is absent. We also want IE to work with layered devices
+like dm and loopback (i.e. we want to be able to use the IE hardware
+of the underlying devices if present, or else fall back to software
+en/decryption).
+
+
+2. Constraints and notes
+========================
+
+1) IE hardware have a limited number of “keyslots” that can be programmed
+with an encryption context (key, algorithm, data unit size, etc.) at any time.
+One can specify a keyslot in a data request made to the device, and the
+device will en/decrypt the data using the encryption context programmed into
+that specified keyslot. When possible, we want to make multiple requests with
+the same encryption context share the same keyslot.
+
+2) We need a way for filesystems to specify an encryption context to use for
+en/decrypting a struct bio, and a device driver (like UFS) needs to be able
+to use that encryption context when it processes the bio.
+
+3) We need a way for device drivers to expose their capabilities in a unified
+way to the upper layers.
+
+
+3. Design
+=========
+
+We add a struct bio_crypt_context to struct bio that can represent an
+encryption context, because we need to able to pass this encryption context
+from the FS layer to the device driver to act upon.
+
+While IE hardware works on the notion of keyslots, the FS layer has no
+knowledge of keyslots - it simply wants to specify an encryption context to
+use while en/decrypting a bio.
+
+We introduce a keyslot manager (KSM) that handles the translation from
+encryption contexts specified by the FS to keyslots on the IE hardware.
+This KSM also serves as the way IE hardware can expose their capabilities to
+upper layers. The generic mode of operation is: each device driver that wants
+to support IE will construct a KSM and set it up in its struct request_queue.
+Upper layers that want to use IE on this device can then use this KSM in
+the device’s struct request_queue to translate an encryption context into
+a keyslot. The presence of the KSM in the request queue shall be used to mean
+that the device supports IE.
+
+On the device driver end of the interface, the device driver needs to tell the
+KSM how to actually manipulate the IE hardware in the device to do things like
+programming the crypto key into the IE hardware into a particular keyslot. All
+this is achieved through the struct keyslot_mgmt_ll_ops that the device driver
+passes to the KSM when creating it.
+
+It uses refcounts to track which keyslots are idle (either they have no
+encryption context programmed, or there are no in flight struct bios
+referencing that keyslot). When a new encryption context needs a keyslot, it
+tries to find a keyslot that has already been programmed with the same
+encryption context, and if there is no such keyslot, it evicts the least
+recently used idle keyslot and programs the new encryption context into that
+one. If no idle keyslots are available, then the caller will sleep until there
+is at least one.
+
+
+4. Blk-crypto
+=============
+
+The above is sufficient for simple cases, but does not work if there is a
+need for a software fallback, or if we are want to use IE with layered devices.
+To these ends, we introduce blk-crypto. Blk-crypto allows us to present a
+unified view of encryption to the FS (so FS only needs to specify an
+encryption context and not worry about keyslots at all), and blk-crypto can
+decide whether to delegate the en/decryption to IE hardware or to software
+(i.e. to the kernel crypto API). Blk-crypto maintains an internal KSM that
+serves as the software fallback to the kernel crypto API.
+
+Blk-crypto needs to ensure that the encryption context is programmed into the
+"correct" keyslot manager for IE. If a bio is submitted to a layered device
+that eventually passes the bio down to a device that really does support IE, we
+want the encryption context to be programmed into a keyslot for the KSM of the
+device with IE support. However, blk-crypto does not know a priori whether a
+particular device is the final device in the layering structure for a bio or
+not. So in the case that a particular device does not support IE, since it is
+possibly the final destination device for the bio, if the bio requires
+encryption (i.e. the bio is doing a write operation), blk-crypto must fallback
+to software *before* sending the bio to the device.
+
+Blk-crypto ensures that
+1) The bio’s encryption context is programmed into a keyslot in the KSM of the
+request queue that the bio is being submitted to (or the software fallback KSM
+if the request queue doesn’t have a KSM), and that the processing_ksm in the
+bi_crypt_context is set to this KSM
+
+2) That the bio has its own individual reference to the keyslot in this KSM.
+Once the bio passes through blk-crypto, its encryption context is programmed
+in some KSM. The “its own individual reference to the keyslot” ensures that
+keyslots can be released by each bio independently of other bios while ensuring
+that the bio has a valid reference to the keyslot when, for e.g., the software
+fallback KSM in blk-crypto performs crypto for on the device’s behalf. The
+individual references are ensured by increasing the refcount for the keyslot in
+the processing_ksm when a bio with a programmed encryption context is cloned.
+
+
+4-1. What blk-crypto does on bio submission
+-------------------------------------------
+
+Case 1: blk-crypto is given a bio with only an encryption context that hasn’t
+been programmed into any keyslot in any KSM (for e.g. a bio from the FS). In
+this case, blk-crypto will program the encryption context into the KSM of the
+request queue the bio is being submitted to (and if this KSM does not exist,
+then it will program it into blk-crypto’s internal KSM for software fallback).
+The KSM that this encryption context was programmed into is stored as the
+processing_ksm in the bio’s bi_crypt_context.
+
+Case 2: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in the *software fallback KSM*. In this case,
+blk-crypto does nothing; it treats the bio as not having specified an
+encryption context. Note that we cannot do what we will do in Case 3 here
+because we would have already encrypted the bio in software by this point.
+
+Case 3: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in some KSM (that is *not* the software fallback
+KSM). In this case, blk-crypto first releases that keyslot from that KSM and
+then treats the bio as in Case 1.
+
+This way, when a device driver is processing a bio, it can be sure that
+the bio’s encryption context has been programmed into some KSM (either the
+device driver’s request queue’s KSM, or blk-crypto’s software fallback KSM).
+It then simply needs to check if the bio’s processing_ksm is the device’s
+request queue’s KSM. If so, then it should proceed with IE. If not, it should
+simply do nothing with respect to crypto, because some other KSM (perhaps the
+blk-crypto software fallback KSM) is handling the en/decryption.
+
+Blk-crypto will release the keyslot that is being held by the bio (and also
+decrypt it if the bio is using the software fallback KSM) once
+bio_remaining_done returns true for the bio.
+
+
+5. Layered Devices
+==================
+
+Layered devices that wish to support IE need to create their own keyslot
+manager for their request queue, and expose whatever functionality they choose.
+When a layered device wants to pass a bio to another layer (either by
+resubmitting the same bio, or by submitting a clone), it doesn’t need to do
+anything special because the bio (or the clone) will once again pass through
+blk-crypto, which will work as described in Case 3. If a layered device wants
+for some reason to do the IO by itself instead of passing it on to a child
+device, but it also chose to expose IE capabilities by setting up a KSM in its
+request queue, it is then responsible for en/decrypting the data itself. In
+such cases, the device can choose to call the blk-crypto function
+blk_crypto_fallback_to_software (TODO: Not yet implemented), which will
+cause the en/decryption to be done via software fallback.
+
+
+6. Future Optimizations for layered devices
+===========================================
+
+Creating a keyslot manager for the layered device uses up memory for each
+keyslot, and in general, a layered device (like dm-linear) merely passes the
+request on to a “child” device, so the keyslots in the layered device itself
+might be completely unused. We can instead define a new type of KSM; the
+“passthrough KSM”, that layered devices can use to let blk-crypto know that
+this layered device *will* pass the bio to some child device (and hence
+through blk-crypto again, at which point blk-crypto can program the encryption
+context, instead of programming it into the layered device’s KSM). Again, if
+the device “lies” and decides to do the IO itself instead of passing it on to
+a child device, it is responsible for doing the en/decryption (and can choose
+to call blk_crypto_fallback_to_software). Another use case for the
+"passthrough KSM" is for IE devices that want to manage their own keyslots/do
+not have a limited number of keyslots.
block/Makefile +2 -1
index 4147ffa63631..1ba7de84dbaf 100644
@@ -35,4 +35,5 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o bio-crypt-ctx.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o bio-crypt-ctx.o \
+					   blk-crypto.o
block/bio-crypt-ctx.c +6 -1
index 8b884ef32007..91015645c4ea 100644
@@ -23,7 +23,12 @@ EXPORT_SYMBOL(bio_crypt_free_ctx);
 
 int bio_crypt_clone(struct bio *dst, struct bio *src, gfp_t gfp_mask)
 {
-	if (!bio_is_encrypted(src))
+	/*
+	 * If a bio is swhandled, then it will be decrypted when bio_endio
+	 * is called. As we only want the data to be decrypted once, copies
+	 * of the bio must not have have a crypt context.
+	 */
+	if (!bio_is_encrypted(src) || bio_crypt_swhandled(src))
 		return 0;
 
 	dst->bi_crypt_context = bio_crypt_alloc_ctx(gfp_mask);
block/bio.c +5 -0
index 2a272cda6dfa..390777a76468 100644
@@ -16,6 +16,7 @@
 #include <linux/workqueue.h>
 #include <linux/cgroup.h>
 #include <linux/blk-cgroup.h>
+#include <linux/blk-crypto.h>
 
 #include <trace/events/block.h>
 #include "blk.h"
@@ -1832,6 +1833,10 @@ void bio_endio(struct bio *bio)
 again:
 	if (!bio_remaining_done(bio))
 		return;
+
+	if (!blk_crypto_endio(bio))
+		return;
+
 	if (!bio_integrity_endio(bio))
 		return;
 
block/blk-core.c +10 -1
index 8340f69670d8..aec4da8c27fa 100644
@@ -36,6 +36,7 @@
 #include <linux/blk-cgroup.h>
 #include <linux/debugfs.h>
 #include <linux/bpf.h>
+#include <linux/blk-crypto.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/block.h>
@@ -1018,7 +1019,9 @@ blk_qc_t generic_make_request(struct bio *bio)
 			/* Create a fresh bio_list for all subordinate requests */
 			bio_list_on_stack[1] = bio_list_on_stack[0];
 			bio_list_init(&bio_list_on_stack[0]);
-			ret = q->make_request_fn(q, bio);
+
+			if (!blk_crypto_submit_bio(&bio))
+				ret = q->make_request_fn(q, bio);
 
 			blk_queue_exit(q);
 
@@ -1071,6 +1074,9 @@ blk_qc_t direct_make_request(struct bio *bio)
 	if (!generic_make_request_checks(bio))
 		return BLK_QC_T_NONE;
 
+	if (blk_crypto_submit_bio(&bio))
+		return BLK_QC_T_NONE;
+
 	if (unlikely(blk_queue_enter(q, nowait ? BLK_MQ_REQ_NOWAIT : 0))) {
 		if (nowait && !blk_queue_dying(q))
 			bio->bi_status = BLK_STS_AGAIN;
@@ -1750,5 +1756,8 @@ int __init blk_dev_init(void)
 	blk_debugfs_root = debugfs_create_dir("block", NULL);
 #endif
 
+	if (blk_crypto_init() < 0)
+		panic("Failed to init blk-crypto\n");
+
 	return 0;
 }
block/blk-crypto.c new file +585 -0

585 lines not on this page · raw message

include/linux/bio.h +7 -0
index 4e664d6441d5..66ad1508911f 100644
@@ -612,6 +612,8 @@ static inline void bio_crypt_advance(struct bio *bio, unsigned int bytes)
 	}
 }
 
+extern bool bio_crypt_swhandled(struct bio *bio);
+
 static inline bool bio_crypt_has_keyslot(struct bio *bio)
 {
 	return bio->bi_crypt_context->keyslot >= 0;
@@ -717,6 +719,11 @@ static inline void bio_crypt_set_ctx(struct bio *bio,
 				     u64 dun,
 				     unsigned int dun_bits) { }
 
+static inline bool bio_crypt_swhandled(struct bio *bio)
+{
+	return false;
+}
+
 static inline bool bio_crypt_has_keyslot(struct bio *bio)
 {
 	return false;
include/linux/blk-crypto.h new file +40 -0
new file mode 100644
index 000000000000..cbb5bea6dcdb
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#ifndef __LINUX_BLK_CRYPTO_H
+#define __LINUX_BLK_CRYPTO_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+
+struct bio;
+
+int blk_crypto_init(void);
+
+int blk_crypto_submit_bio(struct bio **bio_ptr);
+
+bool blk_crypto_endio(struct bio *bio);
+
+#else /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+static inline int blk_crypto_init(void)
+{
+	return 0;
+}
+
+static inline int blk_crypto_submit_bio(struct bio **bio)
+{
+	return 0;
+}
+
+static inline bool blk_crypto_endio(struct bio *bio)
+{
+	return true;
+}
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+#endif /* __LINUX_BLK_CRYPTO_H */
Signature · 8 lines
2.22.0.410.gd8fdbe21b5-goog



_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
reviewer
#5 lore raw

Re: [PATCH 3/8] block: blk-crypto for Inline Encryption

Hi,

Documentation nits, typos. questions...

Quoted 41 lines · Satya Tangirala
Documentation/block/inline-encryption.txt new file +35 -0
new file mode 100644
index 000000000000..96a7983a117d
@@ -0,0 +1,185 @@
+BLK-CRYPTO and KEYSLOT MANAGER
+===========================
+
+CONTENTS
+1. Objective
+2. Constraints and notes
+3. Design
+4. Blk-crypto
+ 4-1 What does blk-crypto do on bio submission
+5. Layered Devices
+6. Future optimizations for layered devices
+
+1. Objective
+============
+
+We want to support inline encryption (IE) in the kernel.
+To allow for testing, we also want a software fallback when actual
+IE hardware is absent. We also want IE to work with layered devices
+like dm and loopback (i.e. we want to be able to use the IE hardware
+of the underlying devices if present, or else fall back to software
+en/decryption).
+
+
+2. Constraints and notes
+========================
+
+1) IE hardware have a limited number of “keyslots” that can be programmed
+with an encryption context (key, algorithm, data unit size, etc.) at any time.
+One can specify a keyslot in a data request made to the device, and the
+device will en/decrypt the data using the encryption context programmed into
+that specified keyslot. When possible, we want to make multiple requests with
+the same encryption context share the same keyslot.
+
+2) We need a way for filesystems to specify an encryption context to use for
+en/decrypting a struct bio, and a device driver (like UFS) needs to be able

Documentation/block/inline-encryption.txt line 45

Quoted 10 lines
Documentation/block/inline-encryption.txt new file +10 -0
+to use that encryption context when it processes the bio.
+
+3) We need a way for device drivers to expose their capabilities in a unified
+way to the upper layers.
+
+
+3. Design
+=========
+
+We add a struct bio_crypt_context to struct bio that can represent an

is this bi_crypt_context ??

Documentation/block/inline-encryption.txt line 46

Quoted 1 line
Documentation/block/inline-encryption.txt new file +1 -0
+encryption context, because we need to able to pass this encryption context

to be able

Quoted 14 lines
Documentation/block/inline-encryption.txt new file +14 -0
+from the FS layer to the device driver to act upon.
+
+While IE hardware works on the notion of keyslots, the FS layer has no
+knowledge of keyslots - it simply wants to specify an encryption context to
+use while en/decrypting a bio.
+
+We introduce a keyslot manager (KSM) that handles the translation from
+encryption contexts specified by the FS to keyslots on the IE hardware.
+This KSM also serves as the way IE hardware can expose their capabilities to
+upper layers. The generic mode of operation is: each device driver that wants
+to support IE will construct a KSM and set it up in its struct request_queue.
+Upper layers that want to use IE on this device can then use this KSM in
+the device’s struct request_queue to translate an encryption context into
+a keyslot. The presence of the KSM in the request queue shall be used to mean

Documentation/block/inline-encryption.txt line 70

Quoted 10 lines
Documentation/block/inline-encryption.txt new file +10 -0
+that the device supports IE.
+
+On the device driver end of the interface, the device driver needs to tell the
+KSM how to actually manipulate the IE hardware in the device to do things like
+programming the crypto key into the IE hardware into a particular keyslot. All
+this is achieved through the struct keyslot_mgmt_ll_ops that the device driver
+passes to the KSM when creating it.
+
+It uses refcounts to track which keyslots are idle (either they have no
+encryption context programmed, or there are no in flight struct bios

in-flight

Quoted 52 lines
Documentation/block/inline-encryption.txt new file +52 -0
+referencing that keyslot). When a new encryption context needs a keyslot, it
+tries to find a keyslot that has already been programmed with the same
+encryption context, and if there is no such keyslot, it evicts the least
+recently used idle keyslot and programs the new encryption context into that
+one. If no idle keyslots are available, then the caller will sleep until there
+is at least one.
+
+
+4. Blk-crypto
+=============
+
+The above is sufficient for simple cases, but does not work if there is a
+need for a software fallback, or if we are want to use IE with layered devices.
+To these ends, we introduce blk-crypto. Blk-crypto allows us to present a
+unified view of encryption to the FS (so FS only needs to specify an
+encryption context and not worry about keyslots at all), and blk-crypto can
+decide whether to delegate the en/decryption to IE hardware or to software
+(i.e. to the kernel crypto API). Blk-crypto maintains an internal KSM that
+serves as the software fallback to the kernel crypto API.
+
+Blk-crypto needs to ensure that the encryption context is programmed into the
+"correct" keyslot manager for IE. If a bio is submitted to a layered device
+that eventually passes the bio down to a device that really does support IE, we
+want the encryption context to be programmed into a keyslot for the KSM of the
+device with IE support. However, blk-crypto does not know a priori whether a
+particular device is the final device in the layering structure for a bio or
+not. So in the case that a particular device does not support IE, since it is
+possibly the final destination device for the bio, if the bio requires
+encryption (i.e. the bio is doing a write operation), blk-crypto must fallback
+to software *before* sending the bio to the device.
+
+Blk-crypto ensures that
+1) The bio’s encryption context is programmed into a keyslot in the KSM of the
+request queue that the bio is being submitted to (or the software fallback KSM
+if the request queue doesn’t have a KSM), and that the processing_ksm in the
+bi_crypt_context is set to this KSM
+
+2) That the bio has its own individual reference to the keyslot in this KSM.
+Once the bio passes through blk-crypto, its encryption context is programmed
+in some KSM. The “its own individual reference to the keyslot” ensures that
+keyslots can be released by each bio independently of other bios while ensuring
+that the bio has a valid reference to the keyslot when, for e.g., the software
+fallback KSM in blk-crypto performs crypto for on the device’s behalf. The
+individual references are ensured by increasing the refcount for the keyslot in
+the processing_ksm when a bio with a programmed encryption context is cloned.
+
+
+4-1. What blk-crypto does on bio submission
+-------------------------------------------
+
+Case 1: blk-crypto is given a bio with only an encryption context that hasn’t
+been programmed into any keyslot in any KSM (for e.g. a bio from the FS). In

Documentation/block/inline-encryption.txt line 132

Quoted 10 lines
Documentation/block/inline-encryption.txt new file +10 -0
+this case, blk-crypto will program the encryption context into the KSM of the
+request queue the bio is being submitted to (and if this KSM does not exist,
+then it will program it into blk-crypto’s internal KSM for software fallback).
+The KSM that this encryption context was programmed into is stored as the
+processing_ksm in the bio’s bi_crypt_context.
+
+Case 2: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in the *software fallback KSM*. In this case,
+blk-crypto does nothing; it treats the bio as not having specified an
+encryption context. Note that we cannot do what we will do in Case 3 here

Note that we cannot do here what we will do in Case 3

Quoted 53 lines
Documentation/block/inline-encryption.txt new file +53 -0
+because we would have already encrypted the bio in software by this point.
+
+Case 3: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in some KSM (that is *not* the software fallback
+KSM). In this case, blk-crypto first releases that keyslot from that KSM and
+then treats the bio as in Case 1.
+
+This way, when a device driver is processing a bio, it can be sure that
+the bio’s encryption context has been programmed into some KSM (either the
+device driver’s request queue’s KSM, or blk-crypto’s software fallback KSM).
+It then simply needs to check if the bio’s processing_ksm is the device’s
+request queue’s KSM. If so, then it should proceed with IE. If not, it should
+simply do nothing with respect to crypto, because some other KSM (perhaps the
+blk-crypto software fallback KSM) is handling the en/decryption.
+
+Blk-crypto will release the keyslot that is being held by the bio (and also
+decrypt it if the bio is using the software fallback KSM) once
+bio_remaining_done returns true for the bio.
+
+
+5. Layered Devices
+==================
+
+Layered devices that wish to support IE need to create their own keyslot
+manager for their request queue, and expose whatever functionality they choose.
+When a layered device wants to pass a bio to another layer (either by
+resubmitting the same bio, or by submitting a clone), it doesn’t need to do
+anything special because the bio (or the clone) will once again pass through
+blk-crypto, which will work as described in Case 3. If a layered device wants
+for some reason to do the IO by itself instead of passing it on to a child
+device, but it also chose to expose IE capabilities by setting up a KSM in its
+request queue, it is then responsible for en/decrypting the data itself. In
+such cases, the device can choose to call the blk-crypto function
+blk_crypto_fallback_to_software (TODO: Not yet implemented), which will
+cause the en/decryption to be done via software fallback.
+
+
+6. Future Optimizations for layered devices
+===========================================
+
+Creating a keyslot manager for the layered device uses up memory for each
+keyslot, and in general, a layered device (like dm-linear) merely passes the
+request on to a “child” device, so the keyslots in the layered device itself
+might be completely unused. We can instead define a new type of KSM; the
+“passthrough KSM”, that layered devices can use to let blk-crypto know that
+this layered device *will* pass the bio to some child device (and hence
+through blk-crypto again, at which point blk-crypto can program the encryption
+context, instead of programming it into the layered device’s KSM). Again, if
+the device “lies” and decides to do the IO itself instead of passing it on to
+a child device, it is responsible for doing the en/decryption (and can choose
+to call blk_crypto_fallback_to_software). Another use case for the
+"passthrough KSM" is for IE devices that want to manage their own keyslots/do
+not have a limited number of keyslots.
Signature · 7 lines
~Randy


_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
maintainer
#6 lore raw

Re: [PATCH 3/8] block: blk-crypto for Inline Encryption

In reply to Satya Tangirala

Hi Satya, here's yet another round of comments. I'd really like for people who are experts in the block layer to review this too, though. The integration into the block layer needs more review.

Quoted 10 lines · Satya Tangirala

We introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.

Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to Documentation/block/blk-crypto.txt.

It's not necessarily a "software fallback", since the kernel crypto API supports hardware crypto accelerators, just not *inline* hardware crypto accelerators. To avoid this inevitable confusion, can you please call it the "crypto API fallback"? This applies to the whole patch series, including both the code and documentation.

Quoted 6 lines

Known issues:
1) We're allocating crypto_skcipher in blk_crypto_keyslot_program, which uses GFP_KERNEL to allocate memory, but this function is on the write path for IO - we need to add support for specifying a different flags to the crypto API.

You could use memalloc_noio_save() and memalloc_noio_restore(), rather than gfp_flags.

block/blk-crypto.c line 4

Quoted 10 lines
block/blk-crypto.c new file +4 -0
new file mode 100644
index 000000000000..f41fb7819ae9
@@ -0,0 +1,585 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2019 Google LLC
+ */

Perhaps include a reference to the documentation file?

Also, how about adding:

	#define pr_fmt(fmt) "blk-crypto: " fmt

... so that all log messages in this file automatically get an appropriate prefix.

block/blk-crypto.c line 11

Quoted 7 lines
block/blk-crypto.c new file +7 -0
+#include <linux/blk-crypto.h>
+#include <linux/keyslot-manager.h>
+#include <linux/mempool.h>
+#include <linux/blk-cgroup.h>
+#include <crypto/skcipher.h>
+#include <crypto/algapi.h>
+#include <linux/module.h>

This code is using the crypto API, but as-is CONFIG_BLK_INLINE_ENCRYPTION can be enabled without also enabling the crypto API, which will cause a build error. The Kconfig option needs:

select CRYPTO
select CRYPTO_BLKCIPHER

block/blk-crypto.c line 16

Quoted 5 lines
block/blk-crypto.c new file +5 -0
+
+struct blk_crypt_mode {
+	const char *cipher_str;
+	size_t keysize;
+};

In general, I think more comments in this file would be really helpful.

For example, can you document the fields of blk_crypt_mode? E.g.

struct blk_crypt_mode {
const char *cipher_str; /* crypto API name (for fallback case) */
size_t keysize; /* key size in bytes */
};

block/blk-crypto.c line 27

Quoted 11 lines
block/blk-crypto.c new file +11 -0
+
+static const struct blk_crypt_mode blk_crypt_modes[] = {
+	[BLK_ENCRYPTION_MODE_AES_256_XTS] = {
+		.cipher_str = "xts(aes)",
+		.keysize = 64,
+	},
+};
+
+#define BLK_CRYPTO_MAX_KEY_SIZE 64
+/* TODO: Do we want to make this user configurable somehow? */
+static int blk_crypto_num_keyslots = 100;

This TODO is stale, since this patch already makes num_keyslots a kernel command line parameter.

Quoted 6 lines
block/blk-crypto.c new file +6 -0
+
+static struct blk_crypto_keyslot {
+	struct crypto_skcipher *tfm;
+	enum blk_crypt_mode_num crypt_mode;
+	u8 key[BLK_CRYPTO_MAX_KEY_SIZE];
+} *blk_crypto_keyslots;

block/blk-crypto.c line 43

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+
+struct work_mem {
+	struct work_struct crypto_work;
+	struct bio *bio;
+};
+
+static struct keyslot_manager *blk_crypto_ksm;
+static struct workqueue_struct *blk_crypto_wq;
+static mempool_t *blk_crypto_page_pool;
+static struct kmem_cache *blk_crypto_work_mem_cache;

It would be helpful to add a comment somewhere that makes it clear that all this stuff is for the crypto API fallback, which is not used if the disk natively supports inline encryption.

I'm concerned that people will read this code and draw the wrong conclusion about where the encryption is actually being done in their particular case.

block/blk-crypto.c line 62

Quoted 2 lines
block/blk-crypto.c new file +2 -0
+
+/* TODO: handle modes that need essiv */

There's a pending patchset that makes ESSIV accessible via crypto_skcipher. So ESSIV won't need special handling. I suggest just deleting this TODO comment for now.

block/blk-crypto.c line 74

Quoted 12 lines
block/blk-crypto.c new file +12 -0
+static int blk_crypto_keyslot_program(void *priv, const u8 *key,
+				      enum blk_crypt_mode_num crypt_mode,
+				      unsigned int data_unit_size,
+				      unsigned int slot)
+{
+	struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot];
+	struct crypto_skcipher *tfm = slotp->tfm;
+	const struct blk_crypt_mode *mode = &blk_crypt_modes[crypt_mode];
+	size_t keysize = mode->keysize;
+	int err;
+
+	if (crypt_mode != slotp->crypt_mode || !tfm) {

Nit: reversing this to '!tfm || crypt_mode != slotp->crypt_mode' would be more logical because as the code is written, unused slots have tfm == NULL but an undefined crypt_mode. It doesn't make sense to check the undefined thing first, then the defined thing.

block/blk-crypto.c line 77

Quoted 3 lines
block/blk-crypto.c new file +3 -0
+		evict_keyslot(slot);
+		tfm = crypto_alloc_skcipher(
+			mode->cipher_str, 0, 0);

Nit: the parameters to crypto_alloc_skcipher() fit in one line.

block/blk-crypto.c line 82

Quoted 5 lines
block/blk-crypto.c new file +5 -0
+		if (IS_ERR(tfm))
+			return PTR_ERR(tfm);
+
+		crypto_skcipher_set_flags(tfm,
+					  CRYPTO_TFM_REQ_FORBID_WEAK_KEYS);

Nit: the parameters to crypto_skcipher_set_flags() fit in one line.

Quoted 4 lines
block/blk-crypto.c new file +4 -0
+static int blk_crypto_keyslot_find(void *priv,
+				   const u8 *key,
+				   enum blk_crypt_mode_num crypt_mode,
+				   unsigned int data_unit_size_bytes)

block/blk-crypto.c line 121

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+{
+	int slot;
+	const size_t keysize = blk_crypt_modes[crypt_mode].keysize;
+
+	/* TODO: hashmap? */
+	for (slot = 0; slot < blk_crypto_num_keyslots; slot++) {
+		if (blk_crypto_keyslots[slot].crypt_mode == crypt_mode &&
+		    !crypto_memneq(blk_crypto_keyslots[slot].key, key, keysize))
+			return slot;
+	}

There should be no TODO comments in the code, so please do something about this one. Note that we musn't leak other keys via timing information, so a naive hashmap is not an appropriate solution. So perhaps this comment should just be deleted, or updated to explain the timing attack problem and how it's solved.

Quoted 4 lines
block/blk-crypto.c new file +4 -0
+static bool blk_crypt_mode_supported(void *priv,
+				     enum blk_crypt_mode_num crypt_mode,
+				     unsigned int data_unit_size)
+{

block/blk-crypto.c line 139

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	/* All blk_crypt_modes are required to have a software fallback. */
+	return true;
+}
+
+static const struct keyslot_mgmt_ll_ops blk_crypto_ksm_ll_ops = {
+	.keyslot_program	= blk_crypto_keyslot_program,
+	.keyslot_evict		= blk_crypto_keyslot_evict,
+	.keyslot_find		= blk_crypto_keyslot_find,
+	.crypt_mode_supported	= blk_crypt_mode_supported,
+};

There's a missing "o" in blk_crypt_mode_supported().

block/blk-crypto.c line 149

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+
+static void blk_crypto_encrypt_endio(struct bio *enc_bio)
+{
+	struct bio *src_bio = enc_bio->bi_private;
+	struct bio_vec *enc_bvec, *enc_bvec_end;
+
+	enc_bvec = enc_bio->bi_io_vec;
+	enc_bvec_end = enc_bvec + enc_bio->bi_vcnt;
+	for (; enc_bvec != enc_bvec_end; enc_bvec++)
+		mempool_free(enc_bvec->bv_page, blk_crypto_page_pool);

Nit: this could just be a regular loop over 'i':

	for (i = 0; i < enc_bio->bi_vcnt; i++)
		mempool_free(enc_bio->bi_io_vec[i].bv_page,
			     blk_crypto_page_pool);

It's one line shorter and slightly easier to read.

Quoted 18 lines
block/blk-crypto.c new file +18 -0
+static struct bio *blk_crypto_clone_bio(struct bio *bio_src)
+{
+	struct bvec_iter iter;
+	struct bio_vec bv;
+	struct bio *bio;
+
+	bio = bio_alloc_bioset(GFP_NOIO, bio_segments(bio_src), NULL);
+	if (!bio)
+		return NULL;
+	bio->bi_disk		= bio_src->bi_disk;
+	bio->bi_opf		= bio_src->bi_opf;
+	bio->bi_ioprio		= bio_src->bi_ioprio;
+	bio->bi_write_hint	= bio_src->bi_write_hint;
+	bio->bi_iter.bi_sector	= bio_src->bi_iter.bi_sector;
+	bio->bi_iter.bi_size	= bio_src->bi_iter.bi_size;
+
+	bio_for_each_segment(bv, bio_src, iter)
+		bio->bi_io_vec[bio->bi_vcnt++] = bv;

block/blk-crypto.c line 184

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+
+	if (bio_integrity(bio_src)) {
+		int ret;
+
+		ret = bio_integrity_clone(bio, bio_src, GFP_NOIO);
+		if (ret < 0) {
+			bio_put(bio);
+			return NULL;
+		}
+	}

Nit: the code under bio_integrity() can be written more concisely:

	if (bio_integrity(bio_src) &&
	    bio_integrity_clone(bio, bio_src, GFP_NOIO) < 0) {
		bio_put(bio);
		return NULL;
	}
Quoted 10 lines
block/blk-crypto.c new file +10 -0
+static int blk_crypto_encrypt_bio(struct bio **bio_ptr)
+{
+	struct bio *src_bio = *bio_ptr;
+	int slot;
+	struct skcipher_request *ciph_req = NULL;
+	DECLARE_CRYPTO_WAIT(wait);
+	struct bio_vec bv;
+	struct bvec_iter iter;
+	int err = 0;
+	u64 curr_dun;

block/blk-crypto.c line 211

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	union {
+		__le64 dun;
+		u8 bytes[16];
+	} iv;
+	struct scatterlist src, dst;
+	struct bio *enc_bio;
+	struct bio_vec *enc_bvec;
+	int i, j;
+	unsigned int num_sectors;
+	int data_unit_size;

As I mentioned on v2, this function is difficult to understand because it's so long. I gave a suggestion to make it slightly shorter, which was not taken. It would be helpful if you responded to suggestions that you're not accepting, since otherwise it's unclear whether the suggestion was rejected, or was missed or forgotten.

Anyway, a new suggestion: it also seems the bio splitting can be nicely made its own function, which would help. E.g.:

static int blk_crypto_split_bio_if_needed(struct bio **bio_ptr)

{
       struct bio *bio = *bio_ptr;
       unsigned int i = 0;
       unsigned int num_sectors = 0;
       struct bio_vec bv;
       struct bvec_iter iter;

       bio_for_each_segment(bv, bio, iter) {
               num_sectors += bv.bv_len >> SECTOR_SHIFT;
               if (++i == BIO_MAX_PAGES)
                       break;
       }
       if (num_sectors < bio_sectors(bio)) {
               struct bio *split_bio;

               split_bio = bio_split(bio, num_sectors, GFP_NOIO, NULL);
               if (!split_bio) {
                       bio->bi_status = BLK_STS_RESOURCE;
                       return -ENOMEM;
               }
               bio_chain(split_bio, bio);
               generic_make_request(bio);
               *bio_ptr = split_bio;
       }
       return 0;
}
Quoted 4 lines
block/blk-crypto.c new file +4 -0
+
+	/* Split the bio if it's too big for single page bvec */
+	i = 0;
+	num_sectors = 0;

block/blk-crypto.c line 225

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	data_unit_size = 1 << src_bio->bi_crypt_context->data_unit_size_bits;
+	bio_for_each_segment(bv, src_bio, iter) {
+		num_sectors += bv.bv_len >> 9;
+		if (bv.bv_len % data_unit_size != 0) {
+			src_bio->bi_status = BLK_STS_IOERR;
+			return -EIO;
+		}
+		if (++i == BIO_MAX_PAGES)
+			break;
+	}

1.) The alignment check should be done on both bv_len and bv_offset.

2.) To make the behavior consistent, I also think it should be done
regardless of whether the crypto API fallback is being used.

3.) As coded, this alignment check is broken because it's only being
done on the first BIO_MAX_PAGES pages of the bio.

4.) '%' or '/' by a non-constant generally should be avoided in kernel
code, because they can be very slow, and they aren't supported on 64-bit variables on some architectures. data_unit_size is always a power of 2, so use IS_ALIGNED() instead.

Instead, I suggest adding the following and calling it from early in blk_crypto_submit_bio():

/* All I/O segments must be data unit aligned */ static int bio_crypt_check_alignment(struct bio *bio)

{
        int data_unit_size = 1 << bio->bi_crypt_context->data_unit_size_bits;
        struct bvec_iter iter;
        struct bio_vec bv;

        bio_for_each_segment(bv, bio, iter) {
                if (!IS_ALIGNED(bv.bv_len | bv.bv_offset, data_unit_size))
                        return -EIO;
        }
        return 0;
}
Quoted 31 lines
block/blk-crypto.c new file +31 -0
+	if (num_sectors < bio_sectors(src_bio)) {
+		struct bio *split_bio;
+
+		split_bio = bio_split(src_bio, num_sectors, GFP_NOIO, NULL);
+		if (!split_bio) {
+			src_bio->bi_status = BLK_STS_RESOURCE;
+			return -ENOMEM;
+		}
+		bio_chain(split_bio, src_bio);
+		generic_make_request(src_bio);
+		*bio_ptr = split_bio;
+		src_bio = *bio_ptr;
+	}
+
+	enc_bio = blk_crypto_clone_bio(src_bio);
+	if (!enc_bio) {
+		src_bio->bi_status = BLK_STS_RESOURCE;
+		return -ENOMEM;
+	}
+
+	err = bio_crypt_ctx_acquire_keyslot(src_bio, blk_crypto_ksm);
+	if (err) {
+		src_bio->bi_status = BLK_STS_IOERR;
+		bio_put(enc_bio);
+		return err;
+	}
+	slot = bio_crypt_get_keyslot(src_bio);
+
+	ciph_req = skcipher_request_alloc(blk_crypto_keyslots[slot].tfm,
+					  GFP_NOIO);
+	if (!ciph_req) {

block/blk-crypto.c line 266

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+		src_bio->bi_status = BLK_STS_RESOURCE;
+		err = -ENOMEM;
+		bio_put(enc_bio);
+		goto out_release_keyslot;
+	}
+
+	skcipher_request_set_callback(ciph_req,
+				      CRYPTO_TFM_REQ_MAY_BACKLOG |
+				      CRYPTO_TFM_REQ_MAY_SLEEP,
+				      crypto_req_done, &wait);

To help people follow this code, a few brief comments describing the high-level tasks being accomplished here would be helpful. E.g.:

/*
* Use the crypto API fallback keyslot manager to get a crypto_skcipher
* for the algorithm and key, then allocate an skcipher_request for it. */

Then below:

	/* Encrypt each page in the bio */

and

		/* Encrypt each data unit in the page */

block/blk-crypto.c line 273

Quoted 7 lines
block/blk-crypto.c new file +7 -0
+
+	curr_dun = bio_crypt_data_unit_num(src_bio);
+	sg_init_table(&src, 1);
+	sg_init_table(&dst, 1);
+	for (i = 0, enc_bvec = enc_bio->bi_io_vec; i < enc_bio->bi_vcnt;
+	     enc_bvec++, i++) {
+		struct page *page = enc_bvec->bv_page;

Nit: to make this slightly easier to read, you could just use a regular loop over 'i' and assign enc_bvec in the loop:

	for (i = 0; i < enc_bio->bi_vcnt; i++) {
		struct bio_vec *enc_bvec = &enc_bio->bi_io_vec[i];

block/blk-crypto.c line 275

Quoted 2 lines
block/blk-crypto.c new file +2 -0
+		struct page *ciphertext_page =
+			mempool_alloc(blk_crypto_page_pool, GFP_NOIO);

Please rename 'page' => 'plaintext_page' to make it very clear which is the plaintext and which is the ciphertext.

Quoted 13 lines
block/blk-crypto.c new file +13 -0
+
+		enc_bvec->bv_page = ciphertext_page;
+
+		if (!ciphertext_page)
+			goto no_mem_for_ciph_page;
+
+		for (j = 0; j < enc_bvec->bv_len / data_unit_size; j++) {
+			memset(&iv, 0, sizeof(iv));
+			iv.dun = cpu_to_le64(curr_dun);
+
+			sg_set_page(&src, page, data_unit_size,
+				    enc_bvec->bv_offset + j * data_unit_size);
+			sg_set_page(&dst, ciphertext_page, data_unit_size,

block/blk-crypto.c line 298

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+				    enc_bvec->bv_offset + j * data_unit_size);
+
+			skcipher_request_set_crypt(ciph_req, &src, &dst,
+						   data_unit_size, iv.bytes);
+			err = crypto_wait_req(crypto_skcipher_encrypt(ciph_req),
+					      &wait);
+			if (err)
+				goto no_mem_for_ciph_page;
+			curr_dun++;
+		}

This could be optimized slightly by only calling skcipher_request_set_crypt() once per bio (since the scatterlists and IV buffer are always the same), and by only doing the two sg_set_page() calls once per page and instead updating the scatterlists after encrypting each data unit:

	src.offset += data_unit_size;
	dst.offset += data_unit_size;

And the loop can be:

	for (j = 0; j < enc_bvec->bv_len; j += data_unit_size) {

... so that no divisions or multiplications are needed.

block/blk-crypto.c line 309

Quoted 11 lines
block/blk-crypto.c new file +11 -0
+		continue;
+no_mem_for_ciph_page:
+		err = -ENOMEM;
+		for (j = i - 1; j >= 0; j--) {
+			mempool_free(enc_bio->bi_io_vec[j].bv_page,
+				     blk_crypto_page_pool);
+		}
+		src_bio->bi_status = BLK_STS_RESOURCE;
+		bio_put(enc_bio);
+		goto out_release_cipher;
+	}

If crypto_skcipher_encrypt() fails, a bounce page is leaked. Also, the failure is not necessarily caused by "no memory".

To fix the former, we can increment 'i' immediately after the page has been allocated. To fix the latter, we can set bi_status at the place the error happened.

To make the error handling easier to read and less error prone, how about it also moving it to the end of the function, including the bio_put(enc_bio):

        enc_bio = NULL;
        err = 0;
        goto out_free_ciph_req;

out_free_bounce_pages:

        while (i > 0)
                mempool_free(enc_bio->bi_io_vec[--i].bv_page,
                             blk_crypto_page_pool);
out_free_ciph_req:
        skcipher_request_free(ciph_req);
out_release_keyslot:
        bio_crypt_ctx_release_keyslot(src_bio);
out_put_enc_bio:
        if (enc_bio)
                bio_put(enc_bio);
        return err;
Quoted 6 lines
block/blk-crypto.c new file +6 -0
+
+	enc_bio->bi_private = src_bio;
+	enc_bio->bi_end_io = blk_crypto_encrypt_endio;
+
+	*bio_ptr = enc_bio;
+out_release_cipher:

block/blk-crypto.c line 325

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	skcipher_request_free(ciph_req);
+out_release_keyslot:
+	bio_crypt_ctx_release_keyslot(src_bio);
+	return err;
+}
+
+/*
+ * TODO: assumption right now is:
+ * each segment in bio has length divisible by the data_unit_size
+ */

Why is this a TODO? What's wrong with just failing bio's that aren't properly aligned?

Quoted 11 lines
block/blk-crypto.c new file +11 -0
+static void blk_crypto_decrypt_bio(struct work_struct *w)
+{
+	struct work_mem *work_mem =
+		container_of(w, struct work_mem, crypto_work);
+	struct bio *bio = work_mem->bio;
+	int slot;
+	struct skcipher_request *ciph_req;
+	DECLARE_CRYPTO_WAIT(wait);
+	struct bio_vec bv;
+	struct bvec_iter iter;
+	u64 curr_dun;

block/blk-crypto.c line 346

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	union {
+		__le64 dun;
+		u8 bytes[16];
+	} iv;
+	struct scatterlist sg;
+	int err;
+	int data_unit_size = 1 << bio->bi_crypt_context->data_unit_size_bits;
+	int i;
+
+	kmem_cache_free(blk_crypto_work_mem_cache, work_mem);

Nit: freeing the work_mem is cleanup work that always must be done, similar to calling bio_endio(). So I suggest moving this to the end of this function, after bio_endio(). Otherwise it would be easy for a future patch to accidentally introduce a use-after-free of work_mem.

Quoted 38 lines
block/blk-crypto.c new file +38 -0
+
+	err = bio_crypt_ctx_acquire_keyslot(bio, blk_crypto_ksm);
+	if (err) {
+		bio->bi_status = BLK_STS_RESOURCE;
+		goto out_no_keyslot;
+	}
+
+	slot = bio_crypt_get_keyslot(bio);
+	ciph_req = skcipher_request_alloc(blk_crypto_keyslots[slot].tfm,
+					  GFP_NOIO);
+	if (!ciph_req) {
+		bio->bi_status = BLK_STS_RESOURCE;
+		goto out;
+	}
+
+	skcipher_request_set_callback(ciph_req,
+				      CRYPTO_TFM_REQ_MAY_BACKLOG |
+				      CRYPTO_TFM_REQ_MAY_SLEEP,
+				      crypto_req_done, &wait);
+
+	curr_dun = bio_crypt_sw_data_unit_num(bio);
+	sg_init_table(&sg, 1);
+	__bio_for_each_segment(bv, bio, iter,
+			       bio->bi_crypt_context->crypt_iter) {
+		if (bv.bv_len % data_unit_size != 0) {
+			bio->bi_status = BLK_STS_IOERR;
+			err = -EIO;
+			goto out;
+		}
+		for (i = 0; i < bv.bv_len / data_unit_size; i++) {
+			struct page *page = bv.bv_page;
+
+			memset(&iv, 0, sizeof(iv));
+			iv.dun = cpu_to_le64(curr_dun);
+
+			sg_set_page(&sg, page, data_unit_size,
+				    bv.bv_offset + i * data_unit_size);
+			skcipher_request_set_crypt(ciph_req, &sg, &sg,

block/blk-crypto.c line 394

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+						   data_unit_size, iv.bytes);
+			err = crypto_wait_req(crypto_skcipher_decrypt(ciph_req),
+					      &wait);
+			if (err) {
+				bio->bi_status = BLK_STS_IOERR;
+				goto out;
+			}
+			curr_dun++;
+		}
+	}

Most of the comments I made for the encryption case apply here too.

Quoted 7 lines
block/blk-crypto.c new file +7 -0
+
+out:
+	skcipher_request_free(ciph_req);
+	bio_crypt_ctx_release_keyslot(bio);
+out_no_keyslot:
+	bio_endio(bio);
+}

block/blk-crypto.c line 411

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+
+static void blk_crypto_queue_decrypt_bio(struct bio *bio)
+{
+	struct work_mem *work_mem =
+		kmem_cache_zalloc(blk_crypto_work_mem_cache, GFP_ATOMIC);
+
+	if (!work_mem) {
+		bio->bi_status = BLK_STS_RESOURCE;
+		return bio_endio(bio);
+	}

It's not conventional to chain 'void' return values like this.

Try to be as boring as possible. Just write:

		bio_endio(bio);
		return;
Quoted 10 lines
block/blk-crypto.c new file +10 -0
+
+	INIT_WORK(&work_mem->crypto_work, blk_crypto_decrypt_bio);
+	work_mem->bio = bio;
+	queue_work(blk_crypto_wq, &work_mem->crypto_work);
+}
+
+/**
+ * blk_crypto_submit_bio - handle submitting bio for inline encryption
+ *
+ * @bio_ptr: pointer to original bio pointer

block/blk-crypto.c line 431

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+ *
+ * If the bio doesn't have inline encryption enabled or the submitter already
+ * specified a keyslot for the target device, do nothing.  Else, a raw key must
+ * have been provided, so acquire a device keyslot for it if supported.  Else,
+ * use the software crypto fallback.
+ *
+ * When the software fallback is used for encryption, blk-crypto may choose to
+ * split the bio into 2 - one that will continue to be processed and the other
+ * that will be resubmitted via generic_make_request. *bio_ptr will be updated
+ * to the first bio (the one that will continue to be processed).

The second paragraph is misleading because it doesn't explain that *bio_ptr is actually updated to point to the bounce bio.

Quoted 34 lines
block/blk-crypto.c new file +34 -0
+ *
+ * Return: 0 if bio submission should continue; nonzero if bio_endio() was
+ *        already called so bio submission should abort.
+ */
+int blk_crypto_submit_bio(struct bio **bio_ptr)
+{
+	struct bio *bio = *bio_ptr;
+	struct request_queue *q;
+	int err;
+	struct bio_crypt_ctx *crypt_ctx;
+
+	if (!bio_is_encrypted(bio) || !bio_has_data(bio))
+		return 0;
+
+	/*
+	 * When a read bio is marked for sw decryption, its bi_iter is saved
+	 * so that when we decrypt the bio later, we know what part of it was
+	 * marked for sw decryption (when the bio is passed down after
+	 * blk_crypto_submit bio, it may be split or advanced so we cannot rely
+	 * on the bi_iter while decrypting in blk_crypto_endio)
+	 */
+	if (bio_crypt_swhandled(bio))
+		return 0;
+
+	crypt_ctx = bio->bi_crypt_context;
+	q = bio->bi_disk->queue;
+
+	if (bio_crypt_has_keyslot(bio)) {
+		/* Key already programmed into device? */
+		if (q->ksm == crypt_ctx->processing_ksm)
+			return 0;
+
+		/* Nope, release the existing keyslot. */
+		bio_crypt_ctx_release_keyslot(bio);

block/blk-crypto.c line 475

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	}
+
+	/* Get device keyslot if supported */
+	if (q->ksm) {
+		err = bio_crypt_ctx_acquire_keyslot(bio, q->ksm);
+		if (!err) {
+			pr_warn_once("blk-crypto: failed to acquire keyslot for %s (err=%d).  Falling back to software crypto.\n",
+				     bio->bi_disk->disk_name, err);
+			return 0;
+		}

This warning message should go in the err != 0 case, not err == 0.

Quoted 38 lines
block/blk-crypto.c new file +38 -0
+	}
+
+	/* Fallback to software crypto */
+	if (bio_data_dir(bio) == WRITE) {
+		/* Encrypt the data now */
+		err = blk_crypto_encrypt_bio(bio_ptr);
+		if (err)
+			goto out_encrypt_err;
+	} else {
+		/* Mark bio as swhandled */
+		bio->bi_crypt_context->processing_ksm = blk_crypto_ksm;
+		bio->bi_crypt_context->crypt_iter = bio->bi_iter;
+		bio->bi_crypt_context->sw_data_unit_num =
+				bio->bi_crypt_context->data_unit_num;
+	}
+	return 0;
+out_encrypt_err:
+	bio_endio(bio);
+	return err;
+}
+
+/**
+ * blk_crypto_endio - clean up bio w.r.t inline encryption during bio_endio
+ *
+ * @bio - the bio to clean up
+ *
+ * If blk_crypto_submit_bio decided to fallback to software crypto for this
+ * bio, we queue the bio for decryption into a workqueue and return false,
+ * and call bio_endio(bio) at a later time (after the bio has been decrypted).
+ *
+ * If the bio is not to be decrypted in software, this function releases the
+ * reference to the keyslot that blk_crypto_submit_bio got.
+ *
+ * Return: true if bio_endio should continue; false otherwise (bio_endio will
+ * be called again when bio has been decrypted).
+ */
+bool blk_crypto_endio(struct bio *bio)
+{

block/blk-crypto.c line 523

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	if (bio_crypt_swhandled(bio)) {
+		/*
+		 * The only bios that are swhandled when they reach here
+		 * are those with bio_data_dir(bio) == READ, since WRITE
+		 * bios that are encrypted by software fallback are handled
+		 * by blk_crypto_encrypt_endio.
+		 */
+		blk_crypto_queue_decrypt_bio(bio);
+		return false;
+	}

This should only do the decryption if there was no I/O error.

block/blk-crypto.c line 528

Quoted 5 lines
block/blk-crypto.c new file +5 -0
+
+	if (bio_is_encrypted(bio) && bio_crypt_has_keyslot(bio))
+		bio_crypt_ctx_release_keyslot(bio);
+
+	return true;

Also it seems that blk_crypto_endio() should mirror blk_crypto_submit_bio() and start with:

	if (!bio_is_encrypted(bio) || !bio_has_data(bio))
		return true;

Otherwise the bio_has_data() case might be handled inconsistently. And it's good to check bio_is_encrypted() first, so that the unencrypted case is as fast and easy to understand as possible.

Also, blk_crypto_endio() always releases the keyslot, even if it was provided by the submitter of the bio. Shouldn't it only release the keyslot if it was acquired by blk_crypto_submit_bio()?

Quoted 40 lines
block/blk-crypto.c new file +40 -0
+}
+
+int __init blk_crypto_init(void)
+{
+	blk_crypto_ksm = keyslot_manager_create(blk_crypto_num_keyslots,
+						&blk_crypto_ksm_ll_ops,
+						NULL);
+	if (!blk_crypto_ksm)
+		goto out_ksm;
+
+	blk_crypto_wq = alloc_workqueue("blk_crypto_wq",
+					WQ_UNBOUND | WQ_HIGHPRI |
+					WQ_MEM_RECLAIM,
+					num_online_cpus());
+	if (!blk_crypto_wq)
+		goto out_wq;
+
+	blk_crypto_keyslots = kcalloc(blk_crypto_num_keyslots,
+				      sizeof(*blk_crypto_keyslots),
+				      GFP_KERNEL);
+	if (!blk_crypto_keyslots)
+		goto out_blk_crypto_keyslots;
+
+	blk_crypto_page_pool =
+		mempool_create_page_pool(num_prealloc_bounce_pg, 0);
+	if (!blk_crypto_page_pool)
+		goto out_bounce_pool;
+
+	blk_crypto_work_mem_cache = KMEM_CACHE(work_mem, SLAB_RECLAIM_ACCOUNT);
+	if (!blk_crypto_work_mem_cache)
+		goto out_work_mem_cache;
+
+	return 0;
+
+out_work_mem_cache:
+	mempool_destroy(blk_crypto_page_pool);
+	blk_crypto_page_pool = NULL;
+out_bounce_pool:
+	kzfree(blk_crypto_keyslots);
+	blk_crypto_keyslots = NULL;

block/blk-crypto.c line 578

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+out_blk_crypto_keyslots:
+	destroy_workqueue(blk_crypto_wq);
+	blk_crypto_wq = NULL;
+out_wq:
+	keyslot_manager_destroy(blk_crypto_ksm);
+	blk_crypto_ksm = NULL;
+out_ksm:
+	pr_warn("No memory for blk-crypto software fallback.");
+	return -ENOMEM;
+}

Nit: I find it easier to read error labels when they describe what is done at them, rather than what was done just before the goto. I.e. use out:, out_destroy_ksm:, out_destroy_workqueue:, etc.

block/blk-crypto.c line 585

Quoted 7 lines
block/blk-crypto.c new file +7 -0
+
+module_param(blk_crypto_num_keyslots, int, 0);
+MODULE_PARM_DESC(blk_crypto_num_keyslots,
+		 "Number of keyslots for software fallback in blk-crypto.");
+module_param(num_prealloc_bounce_pg, uint, 0);
+MODULE_PARM_DESC(num_prealloc_bounce_pg,
+	"Number of preallocated bounce pages for blk-crypto to use during software fallback encryption");

I believe it's more common for the module_param() statements to go at the top of the file, next to the actual variables.

Also, the module parameter names are inconsistent. The first parameter will be "blk_crypto.blk_crypto_num_keyslots" on command line and second parameter will be "blk_crypto.num_prealloc_bounce_pg" on command line.

It should be "blk_crypto.num_keyslots" for the first. No need to say "blk_crypto" twice.

module_param_named() can be used to still prefix the C variable names with "blk_crypto_".

Quoted 9 lines
include/linux/blk-crypto.h new file +3 -0
new file mode 100644
index 000000000000..cbb5bea6dcdb
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC

include/linux/blk-crypto.h line 13

Quoted 10 lines
include/linux/blk-crypto.h new file +10 -0
+ */
+
+#ifndef __LINUX_BLK_CRYPTO_H
+#define __LINUX_BLK_CRYPTO_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+
+struct bio;

This forward declaration should go outside the ifdef, since 'struct bio' is referred to in both cases.

Quoted 8 lines
include/linux/blk-crypto.h new file +8 -0
+
+int blk_crypto_init(void);
+
+int blk_crypto_submit_bio(struct bio **bio_ptr);
+
+bool blk_crypto_endio(struct bio *bio);
+
+#else /* CONFIG_BLK_INLINE_ENCRYPTION */

include/linux/blk-crypto.h line 31

Quoted 10 lines
include/linux/blk-crypto.h new file +10 -0
+
+static inline int blk_crypto_init(void)
+{
+	return 0;
+}
+
+static inline int blk_crypto_submit_bio(struct bio **bio)
+{
+	return 0;
+}

To be consistent with the other declaration, the parameter name here should be 'bio_ptr', not 'bio'.

Quoted 12 lines
include/linux/blk-crypto.h new file +9 -0
+
+static inline bool blk_crypto_endio(struct bio *bio)
+{
+	return true;
+}
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+#endif /* __LINUX_BLK_CRYPTO_H */
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog

- Eric

maintainer
#7 lore raw

Re: [PATCH 3/8] block: blk-crypto for Inline Encryption

In reply to Satya Tangirala
Quoted 11 lines · Satya Tangirala

We introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.

Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to Documentation/block/blk-crypto.txt.

Quoted 5 lines

Known issues:
1) We're allocating crypto_skcipher in blk_crypto_keyslot_program, which uses GFP_KERNEL to allocate memory, but this function is on the write path for IO - we need to add support for specifying a different flags to the crypto API.

That's a must-fix before merging, btw.

block/blk-core.c blk_qc_t generic_make_request(struct bio *bio) · line 1026

Quoted 10 lines
block/blk-core.c +3 -1
@@ -1018,7 +1019,9 @@ blk_qc_t generic_make_request(struct bio *bio)
  			/* Create a fresh bio_list for all subordinate requests */
  			bio_list_on_stack[1] = bio_list_on_stack[0];
  			bio_list_init(&bio_list_on_stack[0]);
-			ret = q->make_request_fn(q, bio);
+
+			if (!blk_crypto_submit_bio(&bio))
+				ret = q->make_request_fn(q, bio);
 
  			blk_queue_exit(q);

Why isn't this just stacking the ->make_request_fn() instead? Then we could get this out of the hot path.

Signature · 8 lines
Jens Axboe



_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
#8 lore raw

PATCH v3 4/8 scsi: ufs: UFS driver v2.1 spec crypto additions

In reply to Satya Tangirala

Add the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI specification in preparation to add support for inline encryption to UFS.

---
 drivers/scsi/ufs/ufshcd.c |  2 ++
 drivers/scsi/ufs/ufshcd.h |  5 +++
 drivers/scsi/ufs/ufshci.h | 67 +++++++++++++++++++++++++++++++++++++--
 3 files changed, 72 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +2 -0
index 3fe3029617a8..36413f97074d 100644
@@ -4724,6 +4724,8 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	case OCS_MISMATCH_RESP_UPIU_SIZE:
 	case OCS_PEER_COMM_FAILURE:
 	case OCS_FATAL_ERROR:
+	case OCS_INVALID_CRYPTO_CONFIG:
+	case OCS_GENERAL_CRYPTO_ERROR:
 	default:
 		result |= DID_ERROR << 16;
 		dev_err(hba->dev,
drivers/scsi/ufs/ufshcd.h +5 -0
index ecfa898b9ccc..d3b6a6b57a37 100644
@@ -692,6 +692,11 @@ struct ufs_hba {
 	 * the performance of ongoing read/write operations.
 	 */
 #define UFSHCD_CAP_KEEP_AUTO_BKOPS_ENABLED_EXCEPT_SUSPEND (1 << 5)
+	/*
+	 * This capability allows the host controller driver to use the
+	 * inline crypto engine, if it is present
+	 */
+#define UFSHCD_CAP_CRYPTO (1 << 6)
 
 	struct devfreq *devfreq;
 	struct ufs_clk_scaling clk_scaling;
drivers/scsi/ufs/ufshci.h +65 -2

106 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#9 lore raw

PATCH v3 5/8 scsi: ufs: UFS crypto API

In reply to Satya Tangirala

Introduce functions to manipulate UFS inline encryption hardware in line with the JEDEC UFSHCI v2.1 specification and to work with the block keyslot manager.

---
 drivers/scsi/ufs/Kconfig         |  10 +
 drivers/scsi/ufs/Makefile        |   1 +
 drivers/scsi/ufs/ufshcd-crypto.c | 435 +++++++++++++++++++++++++++++++
 drivers/scsi/ufs/ufshcd-crypto.h |  86 ++++++
 drivers/scsi/ufs/ufshcd.h        |  18 ++
 5 files changed, 550 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
drivers/scsi/ufs/Kconfig +10 -0
index 0b845ab7c3bf..861aabfe791b 100644
@@ -150,3 +150,13 @@ config SCSI_UFS_BSG
 
 	  Select this if you need a bsg device node for your UFS controller.
 	  If unsure, say N.
+
+config SCSI_UFS_CRYPTO
+	bool "UFS Crypto Engine Support"
+	depends on SCSI_UFSHCD && BLK_INLINE_ENCRYPTION
+	help
+	  Enable Crypto Engine Support in UFS.
+	  Enabling this makes it possible for the kernel to use the crypto
+	  capabilities of the UFS device (if present) to perform crypto
+	  operations on data being transferred to/from the device.
+
drivers/scsi/ufs/Makefile +1 -0
index 2a9097939bcb..094c39989a37 100644
@@ -11,3 +11,4 @@ obj-$(CONFIG_SCSI_UFSHCD_PCI) += ufshcd-pci.o
 obj-$(CONFIG_SCSI_UFSHCD_PLATFORM) += ufshcd-pltfrm.o
 obj-$(CONFIG_SCSI_UFS_HISI) += ufs-hisi.o
 obj-$(CONFIG_SCSI_UFS_MEDIATEK) += ufs-mediatek.o
+ufshcd-core-$(CONFIG_SCSI_UFS_CRYPTO) += ufshcd-crypto.o
drivers/scsi/ufs/ufshcd-crypto.c new file +435 -0

435 lines not on this page · raw message

drivers/scsi/ufs/ufshcd-crypto.h new file +86 -0

86 lines not on this page · raw message

drivers/scsi/ufs/ufshcd.h +18 -0
index d3b6a6b57a37..ef62b3ba03fd 100644
@@ -501,6 +501,13 @@ struct ufs_stats {
  * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
  *  device is known or not.
  * @scsi_block_reqs_cnt: reference counting for scsi block requests
+ * @crypto_capabilities: Content of crypto capabilities register (0x100)
+ * @crypto_cap_array: Array of crypto capabilities
+ * @crypto_cfg_register: Start of the crypto cfg array
+ * @crypto_cfgs: Array of crypto configurations (i.e. config for each slot)
+ * @ksm: the keyslot manager tied to this hba
+ * @ksm_lock: lock to protect initialization and refcount of ksm
+ * @ksm_num_refs: refcount for ksm
  */
 struct ufs_hba {
 	void __iomem *mmio_base;
@@ -711,6 +718,17 @@ struct ufs_hba {
 
 	struct device		bsg_dev;
 	struct request_queue	*bsg_queue;
+
+#ifdef CONFIG_SCSI_UFS_CRYPTO
+	/* crypto */
+	union ufs_crypto_capabilities crypto_capabilities;
+	union ufs_crypto_cap_entry *crypto_cap_array;
+	u32 crypto_cfg_register;
+	union ufs_crypto_cfg_entry *crypto_cfgs;
+	struct keyslot_manager *ksm;
+	struct mutex ksm_lock;
+	unsigned int ksm_num_refs;
+#endif /* CONFIG_SCSI_UFS_CRYPTO */
 };
 
 /* Returns true if clocks can be gated. Otherwise false */
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#10 lore raw

PATCH v3 6/8 scsi: ufs: Add inline encryption support to UFS

In reply to Satya Tangirala

Wire up ufshcd.c with the UFS Crypto API, the block layer inline encryption additions and the keyslot manager.

---
 drivers/scsi/ufs/ufshcd.c | 82 ++++++++++++++++++++++++++++++++++++---
 drivers/scsi/ufs/ufshcd.h |  6 +++
 2 files changed, 83 insertions(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +77 -5
index 36413f97074d..492e9db6b50f 100644
@@ -47,6 +47,7 @@
 #include "unipro.h"
 #include "ufs-sysfs.h"
 #include "ufs_bsg.h"
+#include "ufshcd-crypto.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/ufs.h>
@@ -855,7 +856,14 @@ static void ufshcd_enable_run_stop_reg(struct ufs_hba *hba)
  */
 static inline void ufshcd_hba_start(struct ufs_hba *hba)
 {
-	ufshcd_writel(hba, CONTROLLER_ENABLE, REG_CONTROLLER_ENABLE);
+	u32 val = CONTROLLER_ENABLE;
+
+	if (ufshcd_hba_is_crypto_supported(hba)) {
+		ufshcd_crypto_enable(hba);
+		val |= CRYPTO_GENERAL_ENABLE;
+	}
+
+	ufshcd_writel(hba, val, REG_CONTROLLER_ENABLE);
 }
 
 /**
@@ -2209,9 +2217,21 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,
 		dword_0 |= UTP_REQ_DESC_INT_CMD;
 
 	/* Transfer request descriptor header fields */
+	if (lrbp->crypto_enable) {
+		dword_0 |= UTP_REQ_DESC_CRYPTO_ENABLE_CMD;
+		dword_0 |= lrbp->crypto_key_slot;
+		req_desc->header.dword_1 =
+			cpu_to_le32((u32)lrbp->data_unit_num);
+		req_desc->header.dword_3 =
+			cpu_to_le32((u32)(lrbp->data_unit_num >> 32));
+	} else {
+		/* dword_1 and dword_3 are reserved, hence they are set to 0 */
+		req_desc->header.dword_1 = 0;
+		req_desc->header.dword_3 = 0;
+	}
+
 	req_desc->header.dword_0 = cpu_to_le32(dword_0);
-	/* dword_1 is reserved, hence it is set to 0 */
-	req_desc->header.dword_1 = 0;
+
 	/*
 	 * assigning invalid value for command status. Controller
 	 * updates OCS on command completion, with the command
@@ -2219,8 +2239,6 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,
 	 */
 	req_desc->header.dword_2 =
 		cpu_to_le32(OCS_INVALID_COMMAND_STATUS);
-	/* dword_3 is reserved, hence it is set to 0 */
-	req_desc->header.dword_3 = 0;
 
 	req_desc->prd_table_length = 0;
 }
@@ -2380,6 +2398,37 @@ static inline u16 ufshcd_upiu_wlun_to_scsi_wlun(u8 upiu_wlun_id)
 	return (upiu_wlun_id & ~UFS_UPIU_WLUN_ID) | SCSI_W_LUN_BASE;
 }
 
+static inline int ufshcd_prepare_lrbp_crypto(struct ufs_hba *hba,
+					     struct scsi_cmnd *cmd,
+					     struct ufshcd_lrb *lrbp)
+{
+	int key_slot;
+
+	if (!cmd->request->bio ||
+	    !bio_crypt_should_process(cmd->request->bio, cmd->request->q)) {
+		lrbp->crypto_enable = false;
+		return 0;
+	}
+
+	if (WARN_ON(!ufshcd_is_crypto_enabled(hba))) {
+		/*
+		 * Upper layer asked us to do inline encryption
+		 * but that isn't enabled, so we fail this request.
+		 */
+		return -EINVAL;
+	}
+	key_slot = bio_crypt_get_keyslot(cmd->request->bio);
+	if (!ufshcd_keyslot_valid(hba, key_slot))
+		return -EINVAL;
+
+	lrbp->crypto_enable = true;
+	lrbp->crypto_key_slot = key_slot;
+	lrbp->data_unit_num = bio_crypt_data_unit_num(cmd->request->bio);
+
+	return 0;
+}
+
+
 /**
  * ufshcd_queuecommand - main entry point for SCSI requests
  * @host: SCSI host pointer
@@ -2467,6 +2516,13 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	lrbp->task_tag = tag;
 	lrbp->lun = ufshcd_scsi_to_upiu_lun(cmd->device->lun);
 	lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed(hba) ? true : false;
+
+	err = ufshcd_prepare_lrbp_crypto(hba, cmd, lrbp);
+	if (err) {
+		lrbp->cmd = NULL;
+		clear_bit_unlock(tag, &hba->lrb_in_use);
+		goto out;
+	}
 	lrbp->req_abort_skip = false;
 
 	ufshcd_comp_scsi_upiu(hba, lrbp);
@@ -2500,6 +2556,7 @@ static int ufshcd_compose_dev_cmd(struct ufs_hba *hba,
 	lrbp->task_tag = tag;
 	lrbp->lun = 0; /* device management cmd is not specific to any LUN */
 	lrbp->intr_cmd = true; /* No interrupt aggregation */
+	lrbp->crypto_enable = false; /* No crypto operations */
 	hba->dev_cmd.type = cmd_type;
 
 	return ufshcd_comp_devman_upiu(hba, lrbp);
@@ -4192,6 +4249,8 @@ static inline void ufshcd_hba_stop(struct ufs_hba *hba, bool can_sleep)
 {
 	int err;
 
+	ufshcd_crypto_disable(hba);
+
 	ufshcd_writel(hba, CONTROLLER_DISABLE,  REG_CONTROLLER_ENABLE);
 	err = ufshcd_wait_for_register(hba, REG_CONTROLLER_ENABLE,
 					CONTROLLER_ENABLE, CONTROLLER_DISABLE,
@@ -4585,10 +4644,13 @@ static int ufshcd_change_queue_depth(struct scsi_device *sdev, int depth)
 static int ufshcd_slave_configure(struct scsi_device *sdev)
 {
 	struct request_queue *q = sdev->request_queue;
+	struct ufs_hba *hba = shost_priv(sdev->host);
 
 	blk_queue_update_dma_pad(q, PRDT_DATA_BYTE_COUNT_PAD - 1);
 	blk_queue_max_segment_size(q, PRDT_DATA_BYTE_COUNT_MAX);
 
+	ufshcd_crypto_setup_rq_keyslot_manager(hba, q);
+
 	return 0;
 }
 
@@ -4599,6 +4661,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)
 static void ufshcd_slave_destroy(struct scsi_device *sdev)
 {
 	struct ufs_hba *hba;
+	struct request_queue *q = sdev->request_queue;
 
 	hba = shost_priv(sdev->host);
 	/* Drop the reference as it won't be needed anymore */
@@ -4609,6 +4672,8 @@ static void ufshcd_slave_destroy(struct scsi_device *sdev)
 		hba->sdev_ufs_device = NULL;
 		spin_unlock_irqrestore(hba->host->host_lock, flags);
 	}
+
+	ufshcd_crypto_destroy_rq_keyslot_manager(hba, q);
 }
 
 /**
@@ -8293,6 +8358,13 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 		goto exit_gating;
 	}
 
+	/* Init crypto */
+	err = ufshcd_hba_init_crypto(hba);
+	if (err) {
+		dev_err(hba->dev, "crypto setup failed\n");
+		goto out_remove_scsi_host;
+	}
+
 	/* Host controller enable */
 	err = ufshcd_hba_enable(hba);
 	if (err) {
drivers/scsi/ufs/ufshcd.h +6 -0
index ef62b3ba03fd..ab9c1191c206 100644
@@ -167,6 +167,9 @@ struct ufs_pm_lvl_states {
  * @intr_cmd: Interrupt command (doesn't participate in interrupt aggregation)
  * @issue_time_stamp: time stamp for debug purposes
  * @compl_time_stamp: time stamp for statistics
+ * @crypto_enable: whether or not the request needs inline crypto operations
+ * @crypto_key_slot: the key slot to use for inline crypto
+ * @data_unit_num: the data unit number for the first block for inline crypto
  * @req_abort_skip: skip request abort task flag
  */
 struct ufshcd_lrb {
@@ -191,6 +194,9 @@ struct ufshcd_lrb {
 	bool intr_cmd;
 	ktime_t issue_time_stamp;
 	ktime_t compl_time_stamp;
+	bool crypto_enable;
+	u8 crypto_key_slot;
+	u64 data_unit_num;
 
 	bool req_abort_skip;
 };
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#11 lore raw

PATCH v3 7/8 fscrypt: wire up fscrypt to use blk-crypto

In reply to Satya Tangirala

Introduce fscrypt_set_bio_crypt_ctx for filesystems to call to set up encryption contexts in bios, and fscrypt_evict_crypt_key to evict the encryption context associated with an inode.

Inline encryption is controlled by a policy flag in the fscrypt_info in the inode, and filesystems may check if an inode should use inline encryption by calling fscrypt_inode_is_inline_crypted. Files can be marked as inline encrypted from userspace by appropriately modifying the flags (OR-ing FS_POLICY_FLAGS_INLINE_ENCRYPTION to it) in the fscrypt_policy passed to fscrypt_ioctl_set_policy.

To test inline encryption with the fscrypt dummy context, add ctx.flags |= FS_POLICY_FLAGS_INLINE_ENCRYPTION
when setting up the dummy context in fs/crypto/keyinfo.c.

Note that blk-crypto will fall back to software en/decryption in the absence of inline crypto hardware, so setting up the ctx.flags in the dummy context without inline crypto hardware serves as a test for the software fallback in blk-crypto.

---
 fs/crypto/Kconfig           |   6 ++
 fs/crypto/bio.c             | 138 +++++++++++++++++++++++++++++++-----
 fs/crypto/crypto.c          |   4 ++
 fs/crypto/fscrypt_private.h |  11 +++
 fs/crypto/keyinfo.c         |  94 +++++++++++++++++-------
 fs/crypto/policy.c          |  10 +++
 include/linux/fscrypt.h     |  62 ++++++++++++++++
 include/uapi/linux/fs.h     |   3 +-
 8 files changed, 285 insertions(+), 43 deletions(-)
fs/crypto/Kconfig +6 -0
index 24ed99e2eca0..738368394a32 100644
@@ -15,3 +15,9 @@ config FS_ENCRYPTION
 	  efficient since it avoids caching the encrypted and
 	  decrypted pages in the page cache.  Currently Ext4,
 	  F2FS and UBIFS make use of this feature.
+
+config FS_ENCRYPTION_INLINE_CRYPT
+	bool "Enable fscrypt to use inline crypto"
+	depends on FS_ENCRYPTION && BLK_INLINE_ENCRYPTION
+	help
+	  Enables fscrypt to use inline crypto hardware if available.
fs/crypto/bio.c +120 -18

186 lines not on this page · raw message

fs/crypto/crypto.c +4 -0
index 335a362ee446..58a01889fac7 100644
@@ -302,6 +302,10 @@ int fscrypt_decrypt_page(const struct inode *inode, struct page *page,
 	if (!(inode->i_sb->s_cop->flags & FS_CFLG_OWN_PAGES))
 		BUG_ON(!PageLocked(page));
 
+	/* If we have HW encryption, then this page is already decrypted */
+	if (fscrypt_inode_is_inline_crypted(inode))
+		return 0;
+
 	return fscrypt_do_page_crypto(inode, FS_DECRYPT, lblk_num, page, page,
 				      len, offs, GFP_NOFS);
 }
fs/crypto/fscrypt_private.h +11 -0
index 7da276159593..67a8ca058996 100644
@@ -49,6 +49,17 @@ struct fscrypt_symlink_data {
 	char encrypted_path[1];
 } __packed;
 
+/* Master key referenced by FS_POLICY_FLAG_DIRECT_KEY policy */
+struct fscrypt_master_key {
+	struct hlist_node mk_node;
+	refcount_t mk_refcount;
+	const struct fscrypt_mode *mk_mode;
+	struct crypto_skcipher *mk_ctfm;
+	u8 mk_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+	u8 mk_raw[FS_MAX_KEY_SIZE];
+	struct super_block *mk_sb;
+};
+
 /*
  * fscrypt_info - the "encryption key" for an inode
  *
fs/crypto/keyinfo.c +70 -24

187 lines not on this page · raw message

fs/crypto/policy.c +10 -0
index d536889ac31b..7bcf5cc0d20d 100644
@@ -36,6 +36,7 @@ static int create_encryption_context_from_policy(struct inode *inode,
 	struct fscrypt_context ctx;
 
 	ctx.format = FS_ENCRYPTION_CONTEXT_FORMAT_V1;
+
 	memcpy(ctx.master_key_descriptor, policy->master_key_descriptor,
 					FS_KEY_DESCRIPTOR_SIZE);
 
@@ -46,8 +47,17 @@ static int create_encryption_context_from_policy(struct inode *inode,
 	if (policy->flags & ~FS_POLICY_FLAGS_VALID)
 		return -EINVAL;
 
+	/*
+	 * TODO: expose inline encryption via some toggleable knob
+	 * instead of as a policy?
+	 */
+	if (!inode->i_sb->s_cop->inline_crypt_supp &&
+	    (policy->flags & FS_POLICY_FLAGS_INLINE_CRYPT))
+		return -EINVAL;
+
 	ctx.contents_encryption_mode = policy->contents_encryption_mode;
 	ctx.filenames_encryption_mode = policy->filenames_encryption_mode;
+
 	ctx.flags = policy->flags;
 	BUILD_BUG_ON(sizeof(ctx.nonce) != FS_KEY_DERIVATION_NONCE_SIZE);
 	get_random_bytes(ctx.nonce, FS_KEY_DERIVATION_NONCE_SIZE);
include/linux/fscrypt.h +62 -0

92 lines not on this page · raw message

include/uapi/linux/fs.h +2 -1
index 59c71fa8c553..ec5a9913886c 100644
@@ -224,7 +224,8 @@ struct fsxattr {
 #define FS_POLICY_FLAGS_PAD_32		0x03
 #define FS_POLICY_FLAGS_PAD_MASK	0x03
 #define FS_POLICY_FLAG_DIRECT_KEY	0x04	/* use master key directly */
-#define FS_POLICY_FLAGS_VALID		0x07
+#define FS_POLICY_FLAGS_INLINE_CRYPT	0x08
+#define FS_POLICY_FLAGS_VALID		0x0F
 
 /* Encryption algorithms */
 #define FS_ENCRYPTION_MODE_INVALID		0
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
maintainer
#12 lore raw

Re: [PATCH 7/8] fscrypt: wire up fscrypt to use blk-crypto

Hi Satya,

Quoted 15 lines · Satya Tangirala

Introduce fscrypt_set_bio_crypt_ctx for filesystems to call to set up encryption contexts in bios, and fscrypt_evict_crypt_key to evict the encryption context associated with an inode.

Inline encryption is controlled by a policy flag in the fscrypt_info in the inode, and filesystems may check if an inode should use inline encryption by calling fscrypt_inode_is_inline_crypted. Files can be marked as inline encrypted from userspace by appropriately modifying the flags (OR-ing FS_POLICY_FLAGS_INLINE_ENCRYPTION to it) in the fscrypt_policy passed to fscrypt_ioctl_set_policy.

To test inline encryption with the fscrypt dummy context, add ctx.flags |= FS_POLICY_FLAGS_INLINE_ENCRYPTION
when setting up the dummy context in fs/crypto/keyinfo.c.

Quoted 6 lines

Note that blk-crypto will fall back to software en/decryption in the absence of inline crypto hardware, so setting up the ctx.flags in the dummy context without inline crypto hardware serves as a test for the software fallback in blk-crypto.

Thanks for the new patches. I implemented a ciphertext verification test for this new encryption policy flag, using the framework for ciphertext verification tests I added to xfstests a few months ago. You can get it from here:
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/xfstests-dev.git/log/?h=inline-encryption
If you build a kvm-xfstests test appliance from that branch, the test can be run with 'kvm-xfstests -c f2fs generic/700'.

Or better: run 'kvm-xfstests -c ext4,f2fs -g encrypt' to run all fscrypt tests on ext4 and f2fs, including that one. I recommend adding that to the testing you do, if you haven't already. Note that this is separate from running xfstests with the "fscrypt dummy context" as you mention in the commit message; see the new documentation at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/fscrypt.rst#n653

I found a bug. The test passes when CONFIG_FS_ENCRYPTION_INLINE_CRYPT=y, but fails when CONFIG_FS_ENCRYPTION_INLINE_CRYPT=n, rather than being skipped. This is because the kernel incorrectly ignores the policy flag in the latter case and produces the wrong ciphertext, rather than rejecting it in FS_IOC_SET_ENCRYPTION_POLICY. So that needs to be fixed.

Quoted 5 lines
fs/crypto/crypto.c +0 -0
index 335a362ee446..58a01889fac7 100644

fs/crypto/crypto.c int fscrypt_decrypt_page(const struct inode *inode, struct page *page, · line 311

Quoted 10 lines
fs/crypto/crypto.c +4 -0
@@ -302,6 +302,10 @@ int fscrypt_decrypt_page(const struct inode *inode, struct page *page,
 	if (!(inode->i_sb->s_cop->flags & FS_CFLG_OWN_PAGES))
 		BUG_ON(!PageLocked(page));
 
+	/* If we have HW encryption, then this page is already decrypted */
+	if (fscrypt_inode_is_inline_crypted(inode))
+		return 0;
+
 	return fscrypt_do_page_crypto(inode, FS_DECRYPT, lblk_num, page, page,
 				      len, offs, GFP_NOFS);
 }

As I mentioned on v2, the purpose of this function is to decrypt the page. The filesystem also has to allocate a decryption context and schedule a workqueue item specifically to call this -- only for it to be a no-op in this new case this patch adds. I think the more logical approach would be for the filesystem to not call this at all if the inode is using inline encryption instead.

fs/crypto/crypto.c

Quoted 12 lines
fs/crypto/crypto.c +12 -0
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+bool fscrypt_inode_is_inline_crypted(const struct inode *inode)
+{
+	struct fscrypt_info *ci;
+
+	if (!inode)
+		return false;
+	ci = inode->i_crypt_info;
+
+	return ci && flags_inline_crypted(ci->ci_flags, inode);
+}
+EXPORT_SYMBOL(fscrypt_inode_is_inline_crypted);

What does it mean for the inode to be NULL here?

This also returns false whenever the inode's encryption key hasn't been set up, even though the inode may use an encryption policy with the inline encryption optimized flag set. Why? Also, because ->i_crypt_info is set locklessly, if it's being dereferenced conditionally it needs to be loaded with READ_ONCE().

So I'm confused about exactly what this is trying to do, and the lack of a kerneldoc comment doesn't help :-(

But AFAICS, this is only called if the encryption key is available. So I think the following would be better:

bool fscrypt_inode_is_inline_crypted(const struct inode *inode)

{
	return flags_inline_crypted(inode->i_crypt_info->ci_flags, inode);
}

... or if it's also expected to handle unencrypted inodes,

bool fscrypt_inode_is_inline_crypted(const struct inode *inode)

{
	return IS_ENCRYPTED(inode) &&
	       flags_inline_crypted(inode->i_crypt_info->ci_flags, inode);
}

The problem with trying to be "safe" and check for NULL ->i_crypt_info is that the fallback behavior this patch implements is to silently do the encryption incorrectly. I don't think that's a good idea. If someone is incorrectly calling this on an inode that hasn't had its key loaded yet, I'd much rather be notified of the bug immediately and fix it.

fs/crypto/crypto.c

Quoted 7 lines
fs/crypto/crypto.c +7 -0
+
+bool fscrypt_needs_fs_layer_crypto(const struct inode *inode)
+{
+	return IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode) &&
+	       !fscrypt_inode_is_inline_crypted(inode);
+}
+EXPORT_SYMBOL(fscrypt_needs_fs_layer_crypto);

Can you please add a kerneldoc comment for all new functions in fs/crypto/ that are exported to filesystems?

fs/crypto/crypto.c

Quoted 8 lines
fs/crypto/crypto.c +8 -0
+	/*
+	 * TODO: expose inline encryption via some toggleable knob
+	 * instead of as a policy?
+	 */
+	if (!inode->i_sb->s_cop->inline_crypt_supp &&
+	    (policy->flags & FS_POLICY_FLAGS_INLINE_CRYPT))
+		return -EINVAL;
+

This TODO doesn't make sense; the policy flag is fine.

I think the name of the flag is still confusing things. It's not enabling inline encryption per se (that's an implementation detail), but rather an on-disk format that's better suited for inline encryption.

How about renaming the flag to FS_POLICY_FLAG_INLINECRYPT_OPTIMIZED, like I suggested on v2?

- Eric

#13 lore raw

PATCH v3 8/8 f2fs: Wire up f2fs to use inline encryption via fscrypt

In reply to Satya Tangirala
---
 fs/f2fs/data.c  | 83 +++++++++++++++++++++++++++++++++++++++++++++----
 fs/f2fs/super.c | 13 ++++----
 2 files changed, 84 insertions(+), 12 deletions(-)
fs/f2fs/data.c +77 -6
index eda4181d2092..6fa947725fc2 100644
@@ -142,6 +142,8 @@ static bool f2fs_bio_post_read_required(struct bio *bio)
 
 static void f2fs_read_end_io(struct bio *bio)
 {
+	fscrypt_unset_bio_crypt_ctx(bio);
+
 	if (time_to_inject(F2FS_P_SB(bio_first_page_all(bio)),
 						FAULT_READ_IO)) {
 		f2fs_show_injection_info(FAULT_READ_IO);
@@ -165,6 +167,8 @@ static void f2fs_write_end_io(struct bio *bio)
 	struct bio_vec *bvec;
 	struct bvec_iter_all iter_all;
 
+	fscrypt_unset_bio_crypt_ctx(bio);
+
 	if (time_to_inject(sbi, FAULT_WRITE_IO)) {
 		f2fs_show_injection_info(FAULT_WRITE_IO);
 		bio->bi_status = BLK_STS_IOERR;
@@ -282,9 +286,18 @@ static struct bio *__bio_alloc(struct f2fs_sb_info *sbi, block_t blk_addr,
 	return bio;
 }
 
+static inline u64 inline_crypt_dun(struct inode *inode, pgoff_t offset)
+{
+	return (((u64)inode->i_ino) << 32) | lower_32_bits(offset);
+}
+
 static inline void __submit_bio(struct f2fs_sb_info *sbi,
 				struct bio *bio, enum page_type type)
 {
+	struct page *page;
+	struct inode *inode;
+	int err = 0;
+
 	if (!is_read_io(bio_op(bio))) {
 		unsigned int start;
 
@@ -326,7 +339,22 @@ static inline void __submit_bio(struct f2fs_sb_info *sbi,
 		trace_f2fs_submit_read_bio(sbi->sb, type, bio);
 	else
 		trace_f2fs_submit_write_bio(sbi->sb, type, bio);
-	submit_bio(bio);
+
+	if (bio_has_data(bio)) {
+		page = bio_page(bio);
+		if (page && page->mapping && page->mapping->host) {
+			inode = page->mapping->host;
+			err = fscrypt_set_bio_crypt_ctx(inode, bio,
+						inline_crypt_dun(inode,
+								 page->index));
+		}
+	}
+	if (err) {
+		bio->bi_status = BLK_STS_IOERR;
+		bio_endio(bio);
+	} else {
+		submit_bio(bio);
+	}
 }
 
 static void __submit_merged_bio(struct f2fs_bio_info *io)
@@ -487,6 +515,9 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)
 	enum page_type btype = PAGE_TYPE_OF_BIO(fio->type);
 	struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp;
 	struct page *bio_page;
+	struct inode *fio_inode, *bio_inode;
+	struct page *first_page;
+	u64 next_dun = 0;
 
 	f2fs_bug_on(sbi, is_read_io(fio->op));
 
@@ -513,10 +544,28 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)
 
 	inc_page_count(sbi, WB_DATA_TYPE(bio_page));
 
+	fio_inode = fio->page->mapping->host;
+	bio_inode = NULL;
+	first_page = NULL;
+	next_dun = 0;
+	if (io->bio && bio_page(io->bio)->mapping) {
+		first_page = bio_page(io->bio);
+		bio_inode = first_page->mapping->host;
+		if (fscrypt_inode_is_inline_crypted(bio_inode)) {
+			next_dun = inline_crypt_dun(bio_inode,
+						    first_page->index) +
+				   (io->bio->bi_iter.bi_size >> PAGE_SHIFT);
+		}
+	}
 	if (io->bio && (io->last_block_in_bio != fio->new_blkaddr - 1 ||
 	    (io->fio.op != fio->op || io->fio.op_flags != fio->op_flags) ||
-			!__same_bdev(sbi, fio->new_blkaddr, io->bio)))
+			!__same_bdev(sbi, fio->new_blkaddr, io->bio) ||
+			!fscrypt_inode_crypt_mergeable(bio_inode, fio_inode) ||
+			(fscrypt_inode_is_inline_crypted(bio_inode) &&
+			 next_dun != inline_crypt_dun(fio_inode,
+						  fio->page->index))))
 		__submit_merged_bio(io);
+
 alloc_new:
 	if (io->bio == NULL) {
 		if ((fio->type == DATA || fio->type == NODE) &&
@@ -568,8 +617,9 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr,
 	bio->bi_end_io = f2fs_read_end_io;
 	bio_set_op_attrs(bio, REQ_OP_READ, op_flag);
 
-	if (f2fs_encrypted_file(inode))
+	if (fscrypt_needs_fs_layer_crypto(inode))
 		post_read_steps |= 1 << STEP_DECRYPT;
+
 	if (post_read_steps) {
 		ctx = mempool_alloc(bio_post_read_ctx_pool, GFP_NOFS);
 		if (!ctx) {
@@ -1519,6 +1569,7 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 					struct f2fs_map_blocks *map,
 					struct bio **bio_ret,
 					sector_t *last_block_in_bio,
+					u64 *next_dun,
 					bool is_readahead)
 {
 	struct bio *bio = *bio_ret;
@@ -1592,6 +1643,13 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 		__submit_bio(F2FS_I_SB(inode), bio, DATA);
 		bio = NULL;
 	}
+
+	if (bio && fscrypt_inode_is_inline_crypted(inode) &&
+	    *next_dun != inline_crypt_dun(inode, page->index)) {
+		__submit_bio(F2FS_I_SB(inode), bio, DATA);
+		bio = NULL;
+	}
+
 	if (bio == NULL) {
 		bio = f2fs_grab_read_bio(inode, block_nr, nr_pages,
 				is_readahead ? REQ_RAHEAD : 0);
@@ -1611,6 +1669,9 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 	if (bio_add_page(bio, page, blocksize, 0) < blocksize)
 		goto submit_and_realloc;
 
+	if (fscrypt_inode_is_inline_crypted(inode))
+		*next_dun = inline_crypt_dun(inode, page->index) + 1;
+
 	inc_page_count(F2FS_I_SB(inode), F2FS_RD_DATA);
 	ClearPageError(page);
 	*last_block_in_bio = block_nr;
@@ -1644,6 +1705,7 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
 	struct inode *inode = mapping->host;
 	struct f2fs_map_blocks map;
 	int ret = 0;
+	u64 next_dun = 0;
 
 	map.m_pblk = 0;
 	map.m_lblk = 0;
@@ -1667,7 +1729,8 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
 		}
 
 		ret = f2fs_read_single_page(inode, page, nr_pages, &map, &bio,
-					&last_block_in_bio, is_readahead);
+					&last_block_in_bio, &next_dun,
+					is_readahead);
 		if (ret) {
 			SetPageError(page);
 			zero_user_segment(page, 0, PAGE_SIZE);
@@ -1720,7 +1783,7 @@ static int encrypt_one_page(struct f2fs_io_info *fio)
 	struct page *mpage;
 	gfp_t gfp_flags = GFP_NOFS;
 
-	if (!f2fs_encrypted_file(inode))
+	if (!fscrypt_needs_fs_layer_crypto(inode))
 		return 0;
 
 	/* wait for GCed page writeback via META_MAPPING */
@@ -1899,7 +1962,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 			f2fs_unlock_op(fio->sbi);
 		err = f2fs_inplace_write_data(fio);
 		if (err) {
-			if (f2fs_encrypted_file(inode))
+			if (fscrypt_needs_fs_layer_crypto(inode))
 				fscrypt_pullback_bio_page(&fio->encrypted_page,
 									true);
 			if (PageWriteback(page))
@@ -2617,6 +2680,8 @@ static void f2fs_dio_end_io(struct bio *bio)
 {
 	struct f2fs_private_dio *dio = bio->bi_private;
 
+	fscrypt_unset_bio_crypt_ctx(bio);
+
 	dec_page_count(F2FS_I_SB(dio->inode),
 			dio->write ? F2FS_DIO_WRITE : F2FS_DIO_READ);
 
@@ -2633,12 +2698,18 @@ static void f2fs_dio_submit_bio(struct bio *bio, struct inode *inode,
 {
 	struct f2fs_private_dio *dio;
 	bool write = (bio_op(bio) == REQ_OP_WRITE);
+	u64 data_unit_num = inline_crypt_dun(inode, file_offset >> PAGE_SHIFT);
 
 	dio = f2fs_kzalloc(F2FS_I_SB(inode),
 			sizeof(struct f2fs_private_dio), GFP_NOFS);
 	if (!dio)
 		goto out;
 
+	if (fscrypt_set_bio_crypt_ctx(inode, bio, data_unit_num) != 0) {
+		kvfree(dio);
+		goto out;
+	}
+
 	dio->inode = inode;
 	dio->orig_end_io = bio->bi_end_io;
 	dio->orig_private = bio->bi_private;
fs/f2fs/super.c +7 -6
index 6b959bbb336a..2a7b3bae98aa 100644
@@ -2223,12 +2223,13 @@ static bool f2fs_dummy_context(struct inode *inode)
 }
 
 static const struct fscrypt_operations f2fs_cryptops = {
-	.key_prefix	= "f2fs:",
-	.get_context	= f2fs_get_context,
-	.set_context	= f2fs_set_context,
-	.dummy_context	= f2fs_dummy_context,
-	.empty_dir	= f2fs_empty_dir,
-	.max_namelen	= F2FS_NAME_LEN,
+	.key_prefix		= "f2fs:",
+	.get_context		= f2fs_get_context,
+	.set_context		= f2fs_set_context,
+	.dummy_context		= f2fs_dummy_context,
+	.empty_dir		= f2fs_empty_dir,
+	.max_namelen		= F2FS_NAME_LEN,
+	.inline_crypt_supp	= true,
 };
 #endif
 
Signature · 8 lines
2.22.0.410.gd8fdbe21b5-goog



_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel