Skip to contentSkip to conversation

Inline Encryption Support

PATCH Superseded Satya Tangirala 4 participants 16 messages Aug 21, 2019 – Sep 24, 2019

Subsystems (7)
Documentation VFS SCSI Block layer +3 more subsystems

v4 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v4 cover Inline Encryption Support

This patch series adds support for Inline Encryption to the block layer, UFS, fscrypt and f2fs.

Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device. Inline Encryption hardware has become increasingly common, and we want to support it in the kernel.

Inline Encryption hardware implementations often function around the concept of a limited number of "keyslots", which can hold an encryption context each. The storage device can be directed to en/decrypt any particular request with the encryption context stored in any particular keyslot.

Patch 1 introduces a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that blk-crypto (introduced in Path 3), will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.

Patch 2 introduces struct bio_crypt_ctx, and a ptr to one in struct bio, which allows struct bio to represent an encryption context that can be passed down the storage stack from the filesystem layer to the storage driver.

Patch 3 introduces blk-crypto. Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware. Given that blk-crypto works as a software fallback, we are considering removing file content en/decryption from fscrypt and simply using blk-crypto in a future patch. For more details on blk-crypto, refer to Documentation/block/blk-crypto.txt.

Patches 4-6 add support for inline encryption into the UFS driver according to the JEDEC UFS HCI v2.1 specification. Inline encryption support for other drivers (like eMMC) may be added in the same way - the device driver should set up a Keyslot Manager in the device's request_queue (refer to the UFS crypto additions in ufshcd-crypto.c and ufshcd.c for an example).

Patches 7 and 8 add support to fscrypt and f2fs, so that we have a complete stack that can make use of inline encryption.

There have been a few patch sets addressing Inline Encryption Support in the past. Briefly, this patch set differs from those as follows:

1) "crypto: qce: ice: Add support for Inline Crypto Engine" is specific to certain hardware, while our patch set's Inline Encryption support for UFS is implemented according to the JEDEC UFS specification.

2) "scsi: ufs: UFS Host Controller crypto changes" registers inline encryption support as a kernel crypto algorithm. Our patch views inline encryption as being fundamentally different from a generic crypto provider (in that inline encryption is tied to a device), and so does not use the kernel crypto API to represent inline encryption hardware.

3) "scsi: ufs: add real time/inline crypto support to UFS HCD" requires the device mapper to work - our patch does not.

Changes v3 => v4:
- Fixed the issue with allocating crypto_skcipher in blk_crypto_keyslot_program.
- bio_crypto_alloc_ctx is now mempool backed.
- In f2fs, a bio's bi_crypt_context is now set up when the bio is allocated, rather than just before the bio is submitted - this fixes bugs in certain cases, like when an encrypted block is being moved without decryption.
- Lots of refactoring and cleanup of blk-crypto - thanks Eric!

Changes v2 => v3:
- Overhauled keyslot manager's get keyslot logic and optimized LRU.
- Block crypto en/decryption fallback now supports data unit sizes that divide the bvec length (instead of requiring each bvec's length to be the same as the data unit size).
- fscrypt master key is now keyed additionally by super_block and ci_ctfm != NULL.
- all references of "hw encryption" are replaced by inline encryption.
- address various other review comments from Eric.

Changes v1 => v2:
- Block layer and UFS changes are split into 3 patches each.
- We now only have a ptr to a struct bio_crypt_ctx in struct bio, instead of the struct itself.
- struct bio_crypt_ctx no longer has flags.
- blk-crypto now correctly handles the case when it fails to init (because of insufficient memory), but kernel continues to boot.
- ufshcd-crypto now works on big endian cpus.
- Many cleanups.

Satya Tangirala (8):
block: Keyslot Manager for Inline Encryption
block: Add encryption context to struct bio
block: blk-crypto for Inline Encryption
scsi: ufs: UFS driver v2.1 spec crypto additions
scsi: ufs: UFS crypto API
scsi: ufs: Add inline encryption support to UFS
fscrypt: wire up fscrypt to use blk-crypto
f2fs: Wire up f2fs to use inline encryption via fscrypt

 Documentation/block/inline-encryption.txt | 186 ++++++
 block/Kconfig                             |  10 +
 block/Makefile                            |   2 +
 block/bio-crypt-ctx.c                     | 142 +++++
 block/bio.c                               |  23 +-
 block/blk-core.c                          |  14 +-
 block/blk-crypto.c                        | 737 ++++++++++++++++++++++
 block/blk-merge.c                         |  35 +-
 block/bounce.c                            |  15 +-
 block/keyslot-manager.c                   | 351 +++++++++++
 drivers/md/dm.c                           |  15 +-
 drivers/scsi/ufs/Kconfig                  |  10 +
 drivers/scsi/ufs/Makefile                 |   1 +
 drivers/scsi/ufs/ufshcd-crypto.c          | 429 +++++++++++++
 drivers/scsi/ufs/ufshcd-crypto.h          |  86 +++
 drivers/scsi/ufs/ufshcd.c                 |  85 ++-
 drivers/scsi/ufs/ufshcd.h                 |  29 +
 drivers/scsi/ufs/ufshci.h                 |  67 +-
 fs/crypto/Kconfig                         |   6 +
 fs/crypto/bio.c                           | 137 +++-
 fs/crypto/fscrypt_private.h               |  23 +
 fs/crypto/keyinfo.c                       | 107 +++-
 fs/crypto/policy.c                        |   6 +
 fs/f2fs/data.c                            | 127 +++-
 fs/f2fs/super.c                           |  15 +-
 include/linux/bio-crypt-ctx.h             | 233 +++++++
 include/linux/bio.h                       |   1 +
 include/linux/blk-crypto.h                |  47 ++
 include/linux/blk_types.h                 |   6 +
 include/linux/blkdev.h                    |   6 +
 include/linux/fscrypt.h                   |  72 +++
 include/linux/keyslot-manager.h           |  94 +++
 include/uapi/linux/fs.h                   |   3 +-
 33 files changed, 3034 insertions(+), 86 deletions(-)
 create mode 100644 Documentation/block/inline-encryption.txt
 create mode 100644 block/bio-crypt-ctx.c
 create mode 100644 block/blk-crypto.c
 create mode 100644 block/keyslot-manager.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
 create mode 100644 include/linux/bio-crypt-ctx.h
 create mode 100644 include/linux/blk-crypto.h
 create mode 100644 include/linux/keyslot-manager.h
Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
#1 lore raw

PATCH v4 1/8 block: Keyslot Manager for Inline Encryption

Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device.

Inline Encryption hardware implementations often function around the concept of "keyslots". These implementations often have a limited number of "keyslots", each of which can hold an encryption context (we say that an encryption context can be "programmed" into a keyslot). Requests made to the storage device may have a keyslot associated with them, and the inline encryption hardware will en/decrypt the data in the requests using the encryption context programmed into that associated keyslot. As keyslots are limited, and programming keys may be expensive in many implementations, and multiple requests may use exactly the same encryption contexts, we introduce a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that upper layers will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.

---
 block/Kconfig                   |   8 +
 block/Makefile                  |   1 +
 block/keyslot-manager.c         | 351 ++++++++++++++++++++++++++++++++
 include/linux/bio.h             |  12 ++
 include/linux/blkdev.h          |   6 +
 include/linux/keyslot-manager.h |  94 +++++++++
 6 files changed, 472 insertions(+)
 create mode 100644 block/keyslot-manager.c
 create mode 100644 include/linux/keyslot-manager.h
block/Kconfig +8 -0
index 8b5f8e560eb4..1469efdd385b 100644
@@ -164,6 +164,14 @@ config BLK_SED_OPAL
 	Enabling this option enables users to setup/unlock/lock
 	Locking ranges for SED devices using the Opal protocol.
 
+config BLK_INLINE_ENCRYPTION
+	bool "Enable inline encryption support in block layer"
+	help
+	  Build the blk-crypto subsystem.
+	  Enabling this lets the block layer handle encryption,
+	  so users can take advantage of inline encryption
+	  hardware if present.
+
 menu "Partition Types"
 
 source "block/partitions/Kconfig"
block/Makefile +1 -0
index eee1b4ceecf9..a72abd61b220 100644
@@ -35,3 +35,4 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o
block/keyslot-manager.c new file +351 -0

351 lines not on this page · raw message

include/linux/bio.h +12 -0
index 3cdb84cdc488..6c9228dd3156 100644
@@ -564,6 +564,18 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
 }
 #endif
 
+enum blk_crypto_mode_num {
+	BLK_ENCRYPTION_MODE_INVALID	= -1,
+	BLK_ENCRYPTION_MODE_AES_256_XTS	= 0,
+	/*
+	 * TODO: Support these too
+	 * BLK_ENCRYPTION_MODE_AES_256_CTS	= 1,
+	 * BLK_ENCRYPTION_MODE_AES_128_CBC	= 2,
+	 * BLK_ENCRYPTION_MODE_AES_128_CTS	= 3,
+	 * BLK_ENCRYPTION_MODE_ADIANTUM		= 4,
+	 */
+};
+
 /*
  * BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
  *
include/linux/blkdev.h +6 -0
index 1ef375dafb1c..afcca4c1ff64 100644
@@ -43,6 +43,7 @@ struct pr_ops;
 struct rq_qos;
 struct blk_queue_stats;
 struct blk_stat_callback;
+struct keyslot_manager;
 
 #define BLKDEV_MIN_RQ	4
 #define BLKDEV_MAX_RQ	128	/* Default maximum */
@@ -478,6 +479,11 @@ struct request_queue {
 	unsigned int		dma_pad_mask;
 	unsigned int		dma_alignment;
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	/* Inline crypto capabilities */
+	struct keyslot_manager *ksm;
+#endif
+
 	unsigned int		rq_timeout;
 	int			poll_nsec;
 
include/linux/keyslot-manager.h new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
maintainer
#2 lore raw

Re: [PATCH v4 1/8] block: Keyslot Manager for Inline Encryption

Hi Satya, a few more minor comments:

Quoted 10 lines · Satya Tangirala

+/**
+ * keyslot_manager_create() - Create a keyslot manager
+ * @num_slots: The number of key slots to manage.
+ * @ksm_ll_ops: The struct keyslot_mgmt_ll_ops for the device that this keyslot
+ * manager will use to perform operations like programming and
+ * evicting keys.
+ * @ll_priv_data: Private data passed as is to the functions in ksm_ll_ops.
+ *
+ * Allocate memory for and initialize a keyslot manager. Called by for e.g.
+ * storage drivers to set up a keyslot manager in their request_queue.

The word "for" in "Called by for e.g." shouldn't be there.

Quoted 28 lines

+ *
+ * Context: May sleep
+ * Return: Pointer to constructed keyslot manager or NULL on error.
+ */

block/keyslot-manager.c new file +24 -0
+struct keyslot_manager *keyslot_manager_create(unsigned int num_slots,
+				const struct keyslot_mgmt_ll_ops *ksm_ll_ops,
+				void *ll_priv_data)
+{
+	struct keyslot_manager *ksm;
+	int slot;
+
+	if (num_slots == 0)
+		return NULL;
+
+	/* Check that all ops are specified */
+	if (ksm_ll_ops->keyslot_program == NULL ||
+	    ksm_ll_ops->keyslot_evict == NULL ||
+	    ksm_ll_ops->crypto_mode_supported == NULL ||
+	    ksm_ll_ops->keyslot_find == NULL)
+		return NULL;
+
+	ksm = kvzalloc(struct_size(ksm, slots, num_slots), GFP_KERNEL);
+	if (!ksm)
+		return NULL;
+
+	ksm->num_slots = num_slots;
+	atomic_set(&ksm->num_idle_slots, num_slots);
+	ksm->ksm_ll_ops = *ksm_ll_ops;

block/keyslot-manager.c line 106

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+	ksm->ll_priv_data = ll_priv_data;
+
+	init_rwsem(&ksm->lock);
+
+	init_waitqueue_head(&ksm->idle_slots_wait_queue);
+	INIT_LIST_HEAD(&ksm->idle_slots);
+
+	for (slot = 0; slot < num_slots; slot++)
+		list_add(&ksm->slots[slot].idle_slot_node, &ksm->idle_slots);
+

Since the keyslots are allocated from the head of the list, it might be more logical to use list_add_tail() here, so that the keyslots get allocated in the order 0, 1, 2, ... rather than in the order num_slots-1, num_slots-2, ...

It doesn't really matter, but the ascending order seems more natural.

Quoted 12 lines
block/keyslot-manager.c new file +12 -0
+static int find_and_grab_keyslot(struct keyslot_manager *ksm, const u8 *key,
+				 enum blk_crypto_mode_num crypto_mode,
+				 unsigned int data_unit_size)
+{
+	int slot;
+	unsigned long flags;
+
+	slot = ksm->ksm_ll_ops.keyslot_find(ksm->ll_priv_data, key,
+					    crypto_mode, data_unit_size);
+	if (slot < 0)
+		return slot;
+	if (WARN_ON(slot >= ksm->num_slots))

block/keyslot-manager.c line 134

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+		return -EINVAL;
+	if (atomic_inc_return(&ksm->slots[slot].slot_refs) == 1) {
+		/* Took first reference to this slot; remove it from LRU list */
+		spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+		list_del(&ksm->slots[slot].idle_slot_node);
+		spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+		atomic_dec(&ksm->num_idle_slots);
+	}
+	return slot;
+}

The code to remove a slot from the LRU list is duplicated both in find_and_grab_keyslot() and in keyslot_manager_get_slot_for_key(). How about making them both call a helper function:

static void remove_slot_from_lru_list(struct keyslot_manager *ksm, int slot)

{
	unsigned long flags;

	spin_lock_irqsave(&ksm->idle_slots_lock, flags);
	list_del(&ksm->slots[slot].idle_slot_node);
	spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);

	atomic_dec(&ksm->num_idle_slots);
}
Quoted 23 lines
block/keyslot-manager.c new file +23 -0
+
+/**
+ * keyslot_manager_get_slot_for_key() - Program a key into a keyslot.
+ * @ksm: The keyslot manager to program the key into.
+ * @key: Pointer to the bytes of the key to program. Must be the correct length
+ *      for the chosen @crypto_mode; see blk_crypto_modes in blk-crypto.c.
+ * @crypto_mode: Identifier for the encryption algorithm to use.
+ * @data_unit_size: The data unit size to use for en/decryption.
+ *
+ * Get a keyslot that's been programmed with the specified key, crypto_mode, and
+ * data_unit_size.  If one already exists, return it with incremented refcount.
+ * Otherwise, wait for a keyslot to become idle and program it.
+ *
+ * Context: Process context. Takes and releases ksm->lock.
+ * Return: The keyslot on success, else a -errno value.
+ */
+int keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm,
+				     const u8 *key,
+				     enum blk_crypto_mode_num crypto_mode,
+				     unsigned int data_unit_size)
+{
+	int slot;
+	int err;

block/keyslot-manager.c line 167

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+	struct keyslot *idle_slot;
+	unsigned long flags;
+
+	down_read(&ksm->lock);
+	slot = find_and_grab_keyslot(ksm, key, crypto_mode, data_unit_size);
+	up_read(&ksm->lock);
+	if (slot != -ENOKEY)
+		return slot;
+
+	while (true) {

Nit: in kernel code it's more common to use 'for (;;)'

Quoted 26 lines
block/keyslot-manager.c new file +26 -0
+		down_write(&ksm->lock);
+		slot = find_and_grab_keyslot(ksm, key, crypto_mode,
+					     data_unit_size);
+		if (slot != -ENOKEY) {
+			up_write(&ksm->lock);
+			return slot;
+		}
+
+		/*
+		 * If we're here, that means there wasn't a slot that was
+		 * already programmed with the key. So try to program it.
+		 */
+		if (atomic_read(&ksm->num_idle_slots) > 0)
+			break;
+
+		up_write(&ksm->lock);
+		wait_event(ksm->idle_slots_wait_queue,
+			(atomic_read(&ksm->num_idle_slots) > 0));
+	}
+
+	idle_slot = list_first_entry(&ksm->idle_slots, struct keyslot,
+					     idle_slot_node);
+	slot = idle_slot - ksm->slots;
+
+	err = ksm->ksm_ll_ops.keyslot_program(ksm->ll_priv_data, key,
+					      crypto_mode,

block/keyslot-manager.c line 203

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+					      data_unit_size,
+					      slot);
+
+	if (err) {
+		wake_up(&ksm->idle_slots_wait_queue);
+		up_write(&ksm->lock);
+		return err;
+	}
+
+	atomic_inc(&ksm->slots[slot].slot_refs);

This could be optimized to:

	atomic_set(&ksm->slots[slot].slot_refs, 1);

... since no one else can be incrementing it from 0 here.

Quoted 16 lines
block/keyslot-manager.c new file +16 -0
+/**
+ * keyslot_manager_put_slot() - Release a reference to a slot
+ * @ksm: The keyslot manager to release the reference from.
+ * @slot: The slot to release the reference from.
+ *
+ * Context: Any context.
+ */
+void keyslot_manager_put_slot(struct keyslot_manager *ksm, unsigned int slot)
+{
+	unsigned long flags;
+
+	if (WARN_ON(slot >= ksm->num_slots))
+		return;
+
+	spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+	if (atomic_dec_and_test(&ksm->slots[slot].slot_refs)) {

block/keyslot-manager.c line 261

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+		list_add_tail(&ksm->slots[slot].idle_slot_node,
+			      &ksm->idle_slots);
+		spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+		atomic_inc(&ksm->num_idle_slots);
+		wake_up(&ksm->idle_slots_wait_queue);
+	} else {
+		spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+	}
+}
+EXPORT_SYMBOL(keyslot_manager_put_slot);

This should use atomic_dec_and_lock_irqsave(), since the spinlock only needs to be taken if the reference count went to 0.

Separately: list_add_tail() is returning the keyslot to the end of the list. Therefore, the allocator will cycle through all the keyslots before allocating this one again. I wonder if it should return it to the beginning of the list instead (list_add())? Again it shouldn't matter, but it might be more logical. Or is it intentional that it cycles through the keyslots?

block/keyslot-manager.c line 351

Quoted 7 lines
block/keyslot-manager.c new file +7 -0
+void keyslot_manager_destroy(struct keyslot_manager *ksm)
+{
+	if (!ksm)
+		return;
+	kvfree(ksm);
+}
+EXPORT_SYMBOL(keyslot_manager_destroy);

No need to check for NULL before calling kvfree().

include/linux/bio.h static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) · line 577

Quoted 11 lines
include/linux/bio.h +11 -0
+enum blk_crypto_mode_num {
+	BLK_ENCRYPTION_MODE_INVALID	= -1,
+	BLK_ENCRYPTION_MODE_AES_256_XTS	= 0,
+	/*
+	 * TODO: Support these too
+	 * BLK_ENCRYPTION_MODE_AES_256_CTS	= 1,
+	 * BLK_ENCRYPTION_MODE_AES_128_CBC	= 2,
+	 * BLK_ENCRYPTION_MODE_AES_128_CTS	= 3,
+	 * BLK_ENCRYPTION_MODE_ADIANTUM		= 4,
+	 */
+};

I assume the list in this TODO comment came from available_modes[] in fs/crypto/keyinfo.c. But it doesn't make sense for CTS to be here because it's used for filenames encryption only, not file contents encryption. CTS isn't something that really makes sense for disk encryption. Also, CBC in fs/crypto/ is really CBC-ESSIV, not plain CBC; if we added it to blk-crypto we'd probably want to put ESSIV in the name.

I suggest just removing this comment for now, to avoid these considerations.

Quoted 15 lines
include/linux/keyslot-manager.h new file +9 -0
new file mode 100644
index 000000000000..284f53973271
@@ -0,0 +1,94 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#include <linux/bio.h>
+
+#ifndef __LINUX_KEYSLOT_MANAGER_H
+#define __LINUX_KEYSLOT_MANAGER_H

include/linux/keyslot-manager.h line 19

Quoted 10 lines
include/linux/keyslot-manager.h new file +10 -0
+
+/**
+ * struct keyslot_mgmt_ll_ops - functions to manage keyslots in hardware
+ * @keyslot_program:	Program the specified key and algorithm into the
+ *			specified slot in the inline encryption hardware.
+ * @keyslot_evict:	Evict key from the specified keyslot in the hardware.
+ *			The key, crypto_mode and data_unit_size are also passed
+ *			down so that for e.g. dm layers can evict keys from
+ *			the devices that they map over.
+ *			Returns 0 on success, -errno otherwise.

The word "for" in "so that for e.g. dm layers" shouldn't be there.

- Eric

maintainer
#3 lore raw

Re: [PATCH v4 1/8] block: Keyslot Manager for Inline Encryption

In reply to Satya Tangirala

Hi Satya, a few more minor comments:

Quoted 10 lines · Satya Tangirala

+/**
+ * keyslot_manager_create() - Create a keyslot manager
+ * @num_slots: The number of key slots to manage.
+ * @ksm_ll_ops: The struct keyslot_mgmt_ll_ops for the device that this keyslot
+ * manager will use to perform operations like programming and
+ * evicting keys.
+ * @ll_priv_data: Private data passed as is to the functions in ksm_ll_ops.
+ *
+ * Allocate memory for and initialize a keyslot manager. Called by for e.g.
+ * storage drivers to set up a keyslot manager in their request_queue.

The word "for" in "Called by for e.g." shouldn't be there.

Quoted 28 lines

+ *
+ * Context: May sleep
+ * Return: Pointer to constructed keyslot manager or NULL on error.
+ */

block/keyslot-manager.c new file +24 -0
+struct keyslot_manager *keyslot_manager_create(unsigned int num_slots,
+				const struct keyslot_mgmt_ll_ops *ksm_ll_ops,
+				void *ll_priv_data)
+{
+	struct keyslot_manager *ksm;
+	int slot;
+
+	if (num_slots == 0)
+		return NULL;
+
+	/* Check that all ops are specified */
+	if (ksm_ll_ops->keyslot_program == NULL ||
+	    ksm_ll_ops->keyslot_evict == NULL ||
+	    ksm_ll_ops->crypto_mode_supported == NULL ||
+	    ksm_ll_ops->keyslot_find == NULL)
+		return NULL;
+
+	ksm = kvzalloc(struct_size(ksm, slots, num_slots), GFP_KERNEL);
+	if (!ksm)
+		return NULL;
+
+	ksm->num_slots = num_slots;
+	atomic_set(&ksm->num_idle_slots, num_slots);
+	ksm->ksm_ll_ops = *ksm_ll_ops;

block/keyslot-manager.c line 106

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+	ksm->ll_priv_data = ll_priv_data;
+
+	init_rwsem(&ksm->lock);
+
+	init_waitqueue_head(&ksm->idle_slots_wait_queue);
+	INIT_LIST_HEAD(&ksm->idle_slots);
+
+	for (slot = 0; slot < num_slots; slot++)
+		list_add(&ksm->slots[slot].idle_slot_node, &ksm->idle_slots);
+

Since the keyslots are allocated from the head of the list, it might be more logical to use list_add_tail() here, so that the keyslots get allocated in the order 0, 1, 2, ... rather than in the order num_slots-1, num_slots-2, ...

It doesn't really matter, but the ascending order seems more natural.

Quoted 12 lines
block/keyslot-manager.c new file +12 -0
+static int find_and_grab_keyslot(struct keyslot_manager *ksm, const u8 *key,
+				 enum blk_crypto_mode_num crypto_mode,
+				 unsigned int data_unit_size)
+{
+	int slot;
+	unsigned long flags;
+
+	slot = ksm->ksm_ll_ops.keyslot_find(ksm->ll_priv_data, key,
+					    crypto_mode, data_unit_size);
+	if (slot < 0)
+		return slot;
+	if (WARN_ON(slot >= ksm->num_slots))

block/keyslot-manager.c line 134

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+		return -EINVAL;
+	if (atomic_inc_return(&ksm->slots[slot].slot_refs) == 1) {
+		/* Took first reference to this slot; remove it from LRU list */
+		spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+		list_del(&ksm->slots[slot].idle_slot_node);
+		spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+		atomic_dec(&ksm->num_idle_slots);
+	}
+	return slot;
+}

The code to remove a slot from the LRU list is duplicated both in find_and_grab_keyslot() and in keyslot_manager_get_slot_for_key(). How about making them both call a helper function:

static void remove_slot_from_lru_list(struct keyslot_manager *ksm, int slot)

{
	unsigned long flags;

	spin_lock_irqsave(&ksm->idle_slots_lock, flags);
	list_del(&ksm->slots[slot].idle_slot_node);
	spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);

	atomic_dec(&ksm->num_idle_slots);
}
Quoted 23 lines
block/keyslot-manager.c new file +23 -0
+
+/**
+ * keyslot_manager_get_slot_for_key() - Program a key into a keyslot.
+ * @ksm: The keyslot manager to program the key into.
+ * @key: Pointer to the bytes of the key to program. Must be the correct length
+ *      for the chosen @crypto_mode; see blk_crypto_modes in blk-crypto.c.
+ * @crypto_mode: Identifier for the encryption algorithm to use.
+ * @data_unit_size: The data unit size to use for en/decryption.
+ *
+ * Get a keyslot that's been programmed with the specified key, crypto_mode, and
+ * data_unit_size.  If one already exists, return it with incremented refcount.
+ * Otherwise, wait for a keyslot to become idle and program it.
+ *
+ * Context: Process context. Takes and releases ksm->lock.
+ * Return: The keyslot on success, else a -errno value.
+ */
+int keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm,
+				     const u8 *key,
+				     enum blk_crypto_mode_num crypto_mode,
+				     unsigned int data_unit_size)
+{
+	int slot;
+	int err;

block/keyslot-manager.c line 167

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+	struct keyslot *idle_slot;
+	unsigned long flags;
+
+	down_read(&ksm->lock);
+	slot = find_and_grab_keyslot(ksm, key, crypto_mode, data_unit_size);
+	up_read(&ksm->lock);
+	if (slot != -ENOKEY)
+		return slot;
+
+	while (true) {

Nit: in kernel code it's more common to use 'for (;;)'

Quoted 26 lines
block/keyslot-manager.c new file +26 -0
+		down_write(&ksm->lock);
+		slot = find_and_grab_keyslot(ksm, key, crypto_mode,
+					     data_unit_size);
+		if (slot != -ENOKEY) {
+			up_write(&ksm->lock);
+			return slot;
+		}
+
+		/*
+		 * If we're here, that means there wasn't a slot that was
+		 * already programmed with the key. So try to program it.
+		 */
+		if (atomic_read(&ksm->num_idle_slots) > 0)
+			break;
+
+		up_write(&ksm->lock);
+		wait_event(ksm->idle_slots_wait_queue,
+			(atomic_read(&ksm->num_idle_slots) > 0));
+	}
+
+	idle_slot = list_first_entry(&ksm->idle_slots, struct keyslot,
+					     idle_slot_node);
+	slot = idle_slot - ksm->slots;
+
+	err = ksm->ksm_ll_ops.keyslot_program(ksm->ll_priv_data, key,
+					      crypto_mode,

block/keyslot-manager.c line 203

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+					      data_unit_size,
+					      slot);
+
+	if (err) {
+		wake_up(&ksm->idle_slots_wait_queue);
+		up_write(&ksm->lock);
+		return err;
+	}
+
+	atomic_inc(&ksm->slots[slot].slot_refs);

This could be optimized to:

	atomic_set(&ksm->slots[slot].slot_refs, 1);

... since no one else can be incrementing it from 0 here.

Quoted 16 lines
block/keyslot-manager.c new file +16 -0
+/**
+ * keyslot_manager_put_slot() - Release a reference to a slot
+ * @ksm: The keyslot manager to release the reference from.
+ * @slot: The slot to release the reference from.
+ *
+ * Context: Any context.
+ */
+void keyslot_manager_put_slot(struct keyslot_manager *ksm, unsigned int slot)
+{
+	unsigned long flags;
+
+	if (WARN_ON(slot >= ksm->num_slots))
+		return;
+
+	spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+	if (atomic_dec_and_test(&ksm->slots[slot].slot_refs)) {

block/keyslot-manager.c line 261

Quoted 10 lines
block/keyslot-manager.c new file +10 -0
+		list_add_tail(&ksm->slots[slot].idle_slot_node,
+			      &ksm->idle_slots);
+		spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+		atomic_inc(&ksm->num_idle_slots);
+		wake_up(&ksm->idle_slots_wait_queue);
+	} else {
+		spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+	}
+}
+EXPORT_SYMBOL(keyslot_manager_put_slot);

This should use atomic_dec_and_lock_irqsave(), since the spinlock only needs to be taken if the reference count went to 0.

Separately: list_add_tail() is returning the keyslot to the end of the list. Therefore, the allocator will cycle through all the keyslots before allocating this one again. I wonder if it should return it to the beginning of the list instead (list_add())? Again it shouldn't matter, but it might be more logical. Or is it intentional that it cycles through the keyslots?

block/keyslot-manager.c line 351

Quoted 7 lines
block/keyslot-manager.c new file +7 -0
+void keyslot_manager_destroy(struct keyslot_manager *ksm)
+{
+	if (!ksm)
+		return;
+	kvfree(ksm);
+}
+EXPORT_SYMBOL(keyslot_manager_destroy);

No need to check for NULL before calling kvfree().

include/linux/bio.h static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) · line 577

Quoted 11 lines
include/linux/bio.h +11 -0
+enum blk_crypto_mode_num {
+	BLK_ENCRYPTION_MODE_INVALID	= -1,
+	BLK_ENCRYPTION_MODE_AES_256_XTS	= 0,
+	/*
+	 * TODO: Support these too
+	 * BLK_ENCRYPTION_MODE_AES_256_CTS	= 1,
+	 * BLK_ENCRYPTION_MODE_AES_128_CBC	= 2,
+	 * BLK_ENCRYPTION_MODE_AES_128_CTS	= 3,
+	 * BLK_ENCRYPTION_MODE_ADIANTUM		= 4,
+	 */
+};

I assume the list in this TODO comment came from available_modes[] in fs/crypto/keyinfo.c. But it doesn't make sense for CTS to be here because it's used for filenames encryption only, not file contents encryption. CTS isn't something that really makes sense for disk encryption. Also, CBC in fs/crypto/ is really CBC-ESSIV, not plain CBC; if we added it to blk-crypto we'd probably want to put ESSIV in the name.

I suggest just removing this comment for now, to avoid these considerations.

Quoted 15 lines
include/linux/keyslot-manager.h new file +9 -0
new file mode 100644
index 000000000000..284f53973271
@@ -0,0 +1,94 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#include <linux/bio.h>
+
+#ifndef __LINUX_KEYSLOT_MANAGER_H
+#define __LINUX_KEYSLOT_MANAGER_H

include/linux/keyslot-manager.h line 19

Quoted 10 lines
include/linux/keyslot-manager.h new file +10 -0
+
+/**
+ * struct keyslot_mgmt_ll_ops - functions to manage keyslots in hardware
+ * @keyslot_program:	Program the specified key and algorithm into the
+ *			specified slot in the inline encryption hardware.
+ * @keyslot_evict:	Evict key from the specified keyslot in the hardware.
+ *			The key, crypto_mode and data_unit_size are also passed
+ *			down so that for e.g. dm layers can evict keys from
+ *			the devices that they map over.
+ *			Returns 0 on success, -errno otherwise.

The word "for" in "so that for e.g. dm layers" shouldn't be there.

- Eric

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#4 lore raw

PATCH v4 2/8 block: Add encryption context to struct bio

In reply to Satya Tangirala

We must have some way of letting a storage device driver know what encryption context it should use for en/decrypting a request. However, it's the filesystem/fscrypt that knows about and manages encryption contexts. As such, when the filesystem layer submits a bio to the block layer, and this bio eventually reaches a device driver with support for inline encryption, the device driver will need to have been told the encryption context for that bio.

We want to communicate the encryption context from the filesystem layer to the storage device along with the bio, when the bio is submitted to the block layer. To do this, we add a struct bio_crypt_ctx to struct bio, which can represent an encryption context (note that we can't use the bi_private field in struct bio to do this because that field does not function to pass information across layers in the storage stack). We also introduce various functions to manipulate the bio_crypt_ctx and make the bio/request merging logic aware of the bio_crypt_ctx.

---
 block/Makefile                |   2 +-
 block/bio-crypt-ctx.c         | 137 +++++++++++++++++++++
 block/bio.c                   |  18 +--
 block/blk-core.c              |   3 +
 block/blk-merge.c             |  35 +++++-
 block/bounce.c                |  15 +--
 drivers/md/dm.c               |  15 ++-
 include/linux/bio-crypt-ctx.h | 226 ++++++++++++++++++++++++++++++++++
 include/linux/bio.h           |  13 +-
 include/linux/blk_types.h     |   6 +
 10 files changed, 433 insertions(+), 37 deletions(-)
 create mode 100644 block/bio-crypt-ctx.c
 create mode 100644 include/linux/bio-crypt-ctx.h
block/Makefile +1 -1
index a72abd61b220..4147ffa63631 100644
@@ -35,4 +35,4 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o bio-crypt-ctx.o
block/bio-crypt-ctx.c new file +137 -0

137 lines not on this page · raw message

block/bio.c +10 -8
index 299a0e7651ec..ada9850c90dc 100644
@@ -241,6 +241,7 @@ static void bio_free(struct bio *bio)
 	struct bio_set *bs = bio->bi_pool;
 	void *p;
 
+	bio_crypt_free_ctx(bio);
 	bio_uninit(bio);
 
 	if (bs) {
@@ -612,15 +613,15 @@ struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)
 
 	__bio_clone_fast(b, bio);
 
-	if (bio_integrity(bio)) {
-		int ret;
-
-		ret = bio_integrity_clone(b, bio, gfp_mask);
+	if (bio_crypt_clone(b, bio, gfp_mask) < 0) {
+		bio_put(b);
+		return NULL;
+	}
 
-		if (ret < 0) {
-			bio_put(b);
-			return NULL;
-		}
+	if (bio_integrity(bio) &&
+	    bio_integrity_clone(b, bio, gfp_mask) < 0) {
+		bio_put(b);
+		return NULL;
 	}
 
 	return b;
@@ -1007,6 +1008,7 @@ void bio_advance(struct bio *bio, unsigned bytes)
 	if (bio_integrity(bio))
 		bio_integrity_advance(bio, bytes);
 
+	bio_crypt_advance(bio, bytes);
 	bio_advance_iter(bio, &bio->bi_iter, bytes);
 }
 EXPORT_SYMBOL(bio_advance);
block/blk-core.c +3 -0
index d0cc6e14d2f0..35027e80e27d 100644
@@ -1769,5 +1769,8 @@ int __init blk_dev_init(void)
 	blk_debugfs_root = debugfs_create_dir("block", NULL);
 #endif
 
+	if (bio_crypt_ctx_init() < 0)
+		panic("Failed to allocate mem for bio crypt ctxs\n");
+
 	return 0;
 }
block/blk-merge.c +31 -4
index 57f7990b342d..ebfb26b536d2 100644
@@ -504,6 +504,9 @@ static inline int ll_new_hw_segment(struct request *req, struct bio *bio,
 	if (blk_integrity_merge_bio(req->q, req, bio) == false)
 		goto no_merge;
 
+	if (WARN_ON_ONCE(!bio_crypt_ctx_compatible(bio, req->bio)))
+		goto no_merge;
+
 	/*
 	 * This will form the start of a new hw segment.  Bump both
 	 * counters.
@@ -658,8 +661,14 @@ static enum elv_merge blk_try_req_merge(struct request *req,
 {
 	if (blk_discard_mergable(req))
 		return ELEVATOR_DISCARD_MERGE;
-	else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next))
+	else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next)) {
+		if (!bio_crypt_ctx_back_mergeable(req->bio,
+						  blk_rq_sectors(req),
+						  next->bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
 		return ELEVATOR_BACK_MERGE;
+	}
 
 	return ELEVATOR_NO_MERGE;
 }
@@ -695,6 +704,9 @@ static struct request *attempt_merge(struct request_queue *q,
 	if (req->ioprio != next->ioprio)
 		return NULL;
 
+	if (!bio_crypt_ctx_compatible(req->bio, next->bio))
+		return NULL;
+
 	/*
 	 * If we are allowed to merge, then append bio list
 	 * from next to rq and release next. merge_requests_fn
@@ -827,16 +839,31 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
 	if (rq->ioprio != bio_prio(bio))
 		return false;
 
+	/* Only merge if the crypt contexts are compatible */
+	if (!bio_crypt_ctx_compatible(bio, rq->bio))
+		return false;
+
 	return true;
 }
 
 enum elv_merge blk_try_merge(struct request *rq, struct bio *bio)
 {
-	if (blk_discard_mergable(rq))
+	if (blk_discard_mergable(rq)) {
 		return ELEVATOR_DISCARD_MERGE;
-	else if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector)
+	} else if (blk_rq_pos(rq) + blk_rq_sectors(rq) ==
+		   bio->bi_iter.bi_sector) {
+		if (!bio_crypt_ctx_back_mergeable(rq->bio,
+						  blk_rq_sectors(rq), bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
 		return ELEVATOR_BACK_MERGE;
-	else if (blk_rq_pos(rq) - bio_sectors(bio) == bio->bi_iter.bi_sector)
+	} else if (blk_rq_pos(rq) - bio_sectors(bio) ==
+		   bio->bi_iter.bi_sector) {
+		if (!bio_crypt_ctx_back_mergeable(bio,
+						  bio_sectors(bio), rq->bio)) {
+			return ELEVATOR_NO_MERGE;
+		}
 		return ELEVATOR_FRONT_MERGE;
+	}
 	return ELEVATOR_NO_MERGE;
 }
block/bounce.c +8 -7
index f8ed677a1bf7..6f9a2359b22a 100644
@@ -267,14 +267,15 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,
 		break;
 	}
 
-	if (bio_integrity(bio_src)) {
-		int ret;
+	if (bio_crypt_clone(bio, bio_src, gfp_mask) < 0) {
+		bio_put(bio);
+		return NULL;
+	}
 
-		ret = bio_integrity_clone(bio, bio_src, gfp_mask);
-		if (ret < 0) {
-			bio_put(bio);
-			return NULL;
-		}
+	if (bio_integrity(bio_src) &&
+	    bio_integrity_clone(bio, bio_src, gfp_mask) < 0) {
+		bio_put(bio);
+		return NULL;
 	}
 
 	bio_clone_blkg_association(bio, bio_src);
drivers/md/dm.c +10 -5
index d0beef033e2f..ce378c0b9ebd 100644
@@ -1322,12 +1322,15 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
 		     sector_t sector, unsigned len)
 {
 	struct bio *clone = &tio->clone;
+	int ret;
 
 	__bio_clone_fast(clone, bio);
 
-	if (bio_integrity(bio)) {
-		int r;
+	ret = bio_crypt_clone(clone, bio, GFP_NOIO);
+	if (ret < 0)
+		return ret;
 
+	if (bio_integrity(bio)) {
 		if (unlikely(!dm_target_has_integrity(tio->ti->type) &&
 			     !dm_target_passes_integrity(tio->ti->type))) {
 			DMWARN("%s: the target %s doesn't support integrity data.",
@@ -1336,9 +1339,11 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
 			return -EIO;
 		}
 
-		r = bio_integrity_clone(clone, bio, GFP_NOIO);
-		if (r < 0)
-			return r;
+		ret = bio_integrity_clone(clone, bio, GFP_NOIO);
+		if (ret < 0) {
+			bio_crypt_free_ctx(clone);
+			return ret;
+		}
 	}
 
 	bio_advance(clone, to_bytes(sector - clone->bi_iter.bi_sector));
include/linux/bio-crypt-ctx.h new file +226 -0

226 lines not on this page · raw message

include/linux/bio.h +1 -12
index 6c9228dd3156..9dddcd2af3e9 100644
@@ -12,6 +12,7 @@
 #ifdef CONFIG_BLOCK
 /* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */
 #include <linux/blk_types.h>
+#include <linux/bio-crypt-ctx.h>
 
 #define BIO_DEBUG
 
@@ -564,18 +565,6 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
 }
 #endif
 
-enum blk_crypto_mode_num {
-	BLK_ENCRYPTION_MODE_INVALID	= -1,
-	BLK_ENCRYPTION_MODE_AES_256_XTS	= 0,
-	/*
-	 * TODO: Support these too
-	 * BLK_ENCRYPTION_MODE_AES_256_CTS	= 1,
-	 * BLK_ENCRYPTION_MODE_AES_128_CBC	= 2,
-	 * BLK_ENCRYPTION_MODE_AES_128_CTS	= 3,
-	 * BLK_ENCRYPTION_MODE_ADIANTUM		= 4,
-	 */
-};
-
 /*
  * BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
  *
include/linux/blk_types.h +6 -0
index feff3fe4467e..e59ab0186441 100644
@@ -18,6 +18,7 @@ struct block_device;
 struct io_context;
 struct cgroup_subsys_state;
 typedef void (bio_end_io_t) (struct bio *);
+struct bio_crypt_ctx;
 
 /*
  * Block error status values.  See block/blk-core:blk_errors for the details.
@@ -170,6 +171,11 @@ struct bio {
 	struct blkcg_gq		*bi_blkg;
 	struct bio_issue	bi_issue;
 #endif
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	struct bio_crypt_ctx	*bi_crypt_context;
+#endif
+
 	union {
 #if defined(CONFIG_BLK_DEV_INTEGRITY)
 		struct bio_integrity_payload *bi_integrity; /* data integrity */
Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
#5 lore raw

Re: [PATCH v4 2/8] block: Add encryption context to struct bio

Hi Satya,

block/blk-merge.c bool blk_rq_merge_ok(struct request *rq, struct bio *bio) · line 845

Quoted 8 lines · Satya Tangirala
block/blk-merge.c +4 -0
@@ -827,16 +839,31 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
 	if (rq->ioprio != bio_prio(bio))
 		return false;
 
+	/* Only merge if the crypt contexts are compatible */
+	if (!bio_crypt_ctx_compatible(bio, rq->bio))
+		return false;
+

Since bio_crypt_ctx_compatible() lacks of consideration of inode, I am not sure if here may lead to incorrect merge decision, especially for f2fs which does not allow merging different files.

Thanks,
Stanley

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#6 lore raw

PATCH v4 3/8 block: blk-crypto for Inline Encryption

In reply to Satya Tangirala

We introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.

Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to Documentation/block/blk-crypto.txt.

---
 Documentation/block/inline-encryption.txt | 186 ++++++
 block/Kconfig                             |   2 +
 block/Makefile                            |   3 +-
 block/bio-crypt-ctx.c                     |   7 +-
 block/bio.c                               |   5 +
 block/blk-core.c                          |  11 +-
 block/blk-crypto.c                        | 737 ++++++++++++++++++++++
 include/linux/bio-crypt-ctx.h             |   7 +
 include/linux/blk-crypto.h                |  47 ++
 9 files changed, 1002 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/block/inline-encryption.txt
 create mode 100644 block/blk-crypto.c
 create mode 100644 include/linux/blk-crypto.h
Documentation/block/inline-encryption.txt new file +186 -0
new file mode 100644
index 000000000000..925611a5ea65
@@ -0,0 +1,186 @@
+BLK-CRYPTO and KEYSLOT MANAGER
+===========================
+
+CONTENTS
+1. Objective
+2. Constraints and notes
+3. Design
+4. Blk-crypto
+ 4-1 What does blk-crypto do on bio submission
+5. Layered Devices
+6. Future optimizations for layered devices
+
+1. Objective
+============
+
+We want to support inline encryption (IE) in the kernel.
+To allow for testing, we also want a crypto API fallback when actual
+IE hardware is absent. We also want IE to work with layered devices
+like dm and loopback (i.e. we want to be able to use the IE hardware
+of the underlying devices if present, or else fall back to crypto API
+en/decryption).
+
+
+2. Constraints and notes
+========================
+
+1) IE hardware have a limited number of “keyslots” that can be programmed
+with an encryption context (key, algorithm, data unit size, etc.) at any time.
+One can specify a keyslot in a data request made to the device, and the
+device will en/decrypt the data using the encryption context programmed into
+that specified keyslot. When possible, we want to make multiple requests with
+the same encryption context share the same keyslot.
+
+2) We need a way for filesystems to specify an encryption context to use for
+en/decrypting a struct bio, and a device driver (like UFS) needs to be able
+to use that encryption context when it processes the bio.
+
+3) We need a way for device drivers to expose their capabilities in a unified
+way to the upper layers.
+
+
+3. Design
+=========
+
+We add a struct bio_crypt_ctx to struct bio that can represent an
+encryption context, because we need to be able to pass this encryption
+context from the FS layer to the device driver to act upon.
+
+While IE hardware works on the notion of keyslots, the FS layer has no
+knowledge of keyslots - it simply wants to specify an encryption context to
+use while en/decrypting a bio.
+
+We introduce a keyslot manager (KSM) that handles the translation from
+encryption contexts specified by the FS to keyslots on the IE hardware.
+This KSM also serves as the way IE hardware can expose their capabilities to
+upper layers. The generic mode of operation is: each device driver that wants
+to support IE will construct a KSM and set it up in its struct request_queue.
+Upper layers that want to use IE on this device can then use this KSM in
+the device’s struct request_queue to translate an encryption context into
+a keyslot. The presence of the KSM in the request queue shall be used to mean
+that the device supports IE.
+
+On the device driver end of the interface, the device driver needs to tell the
+KSM how to actually manipulate the IE hardware in the device to do things like
+programming the crypto key into the IE hardware into a particular keyslot. All
+this is achieved through the struct keyslot_mgmt_ll_ops that the device driver
+passes to the KSM when creating it.
+
+It uses refcounts to track which keyslots are idle (either they have no
+encryption context programmed, or there are no in-flight struct bios
+referencing that keyslot). When a new encryption context needs a keyslot, it
+tries to find a keyslot that has already been programmed with the same
+encryption context, and if there is no such keyslot, it evicts the least
+recently used idle keyslot and programs the new encryption context into that
+one. If no idle keyslots are available, then the caller will sleep until there
+is at least one.
+
+
+4. Blk-crypto
+=============
+
+The above is sufficient for simple cases, but does not work if there is a
+need for a crypto API fallback, or if we are want to use IE with layered
+devices. To these ends, we introduce blk-crypto. Blk-crypto allows us to
+present a unified view of encryption to the FS (so FS only needs to specify
+an encryption context and not worry about keyslots at all), and blk-crypto
+can decide whether to delegate the en/decryption to IE hardware or to the
+crypto API. Blk-crypto maintains an internal KSM that serves as the crypto
+API fallback.
+
+Blk-crypto needs to ensure that the encryption context is programmed into the
+"correct" keyslot manager for IE. If a bio is submitted to a layered device
+that eventually passes the bio down to a device that really does support IE, we
+want the encryption context to be programmed into a keyslot for the KSM of the
+device with IE support. However, blk-crypto does not know a priori whether a
+particular device is the final device in the layering structure for a bio or
+not. So in the case that a particular device does not support IE, since it is
+possibly the final destination device for the bio, if the bio requires
+encryption (i.e. the bio is doing a write operation), blk-crypto must fallback
+to the crypto API *before* sending the bio to the device.
+
+Blk-crypto ensures that
+1) The bio’s encryption context is programmed into a keyslot in the KSM of the
+request queue that the bio is being submitted to (or the crypto API fallback KSM
+if the request queue doesn’t have a KSM), and that the processing_ksm in the
+bi_crypt_context is set to this KSM
+
+2) That the bio has its own individual reference to the keyslot in this KSM.
+Once the bio passes through blk-crypto, its encryption context is programmed
+in some KSM. The “its own individual reference to the keyslot” ensures that
+keyslots can be released by each bio independently of other bios while ensuring
+that the bio has a valid reference to the keyslot when, for e.g., the crypto API
+fallback KSM in blk-crypto performs crypto on the device’s behalf. The individual
+references are ensured by increasing the refcount for the keyslot in the
+processing_ksm when a bio with a programmed encryption context is cloned.
+
+
+4-1. What blk-crypto does on bio submission
+-------------------------------------------
+
+Case 1: blk-crypto is given a bio with only an encryption context that hasn’t
+been programmed into any keyslot in any KSM (for e.g. a bio from the FS). In
+this case, blk-crypto will program the encryption context into the KSM of the
+request queue the bio is being submitted to (and if this KSM does not exist,
+then it will program it into blk-crypto’s internal KSM for crypto API fallback).
+The KSM that this encryption context was programmed into is stored as the
+processing_ksm in the bio’s bi_crypt_context.
+
+Case 2: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in the *crypto API fallback KSM*. In this case,
+blk-crypto does nothing; it treats the bio as not having specified an
+encryption context. Note that we cannot do here what we will do in Case 3
+because we would have already encrypted the bio via the crypto API by this
+point.
+
+Case 3: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in some KSM (that is *not* the crypto API fallback
+KSM). In this case, blk-crypto first releases that keyslot from that KSM and
+then treats the bio as in Case 1.
+
+This way, when a device driver is processing a bio, it can be sure that
+the bio’s encryption context has been programmed into some KSM (either the
+device driver’s request queue’s KSM, or blk-crypto’s crypto API fallback KSM).
+It then simply needs to check if the bio’s processing_ksm is the device’s
+request queue’s KSM. If so, then it should proceed with IE. If not, it should
+simply do nothing with respect to crypto, because some other KSM (perhaps the
+blk-crypto crypto API fallback KSM) is handling the en/decryption.
+
+Blk-crypto will release the keyslot that is being held by the bio (and also
+decrypt it if the bio is using the crypto API fallback KSM) once
+bio_remaining_done returns true for the bio.
+
+
+5. Layered Devices
+==================
+
+Layered devices that wish to support IE need to create their own keyslot
+manager for their request queue, and expose whatever functionality they choose.
+When a layered device wants to pass a bio to another layer (either by
+resubmitting the same bio, or by submitting a clone), it doesn’t need to do
+anything special because the bio (or the clone) will once again pass through
+blk-crypto, which will work as described in Case 3. If a layered device wants
+for some reason to do the IO by itself instead of passing it on to a child
+device, but it also chose to expose IE capabilities by setting up a KSM in its
+request queue, it is then responsible for en/decrypting the data itself. In
+such cases, the device can choose to call the blk-crypto function
+blk_crypto_fallback_to_kernel_crypto_api (TODO: Not yet implemented), which will
+cause the en/decryption to be done via the crypto API fallback.
+
+
+6. Future Optimizations for layered devices
+===========================================
+
+Creating a keyslot manager for the layered device uses up memory for each
+keyslot, and in general, a layered device (like dm-linear) merely passes the
+request on to a “child” device, so the keyslots in the layered device itself
+might be completely unused. We can instead define a new type of KSM; the
+“passthrough KSM”, that layered devices can use to let blk-crypto know that
+this layered device *will* pass the bio to some child device (and hence
+through blk-crypto again, at which point blk-crypto can program the encryption
+context, instead of programming it into the layered device’s KSM). Again, if
+the device “lies” and decides to do the IO itself instead of passing it on to
+a child device, it is responsible for doing the en/decryption (and can choose
+to call blk_crypto_fallback_to_kernel_crypto_api). Another use case for the
+"passthrough KSM" is for IE devices that want to manage their own keyslots/do
+not have a limited number of keyslots.
block/Kconfig +2 -0
index 1469efdd385b..4f7e593d0a6d 100644
@@ -166,6 +166,8 @@ config BLK_SED_OPAL
 
 config BLK_INLINE_ENCRYPTION
 	bool "Enable inline encryption support in block layer"
+	select CRYPTO
+	select CRYPTO_BLKCIPHER
 	help
 	  Build the blk-crypto subsystem.
 	  Enabling this lets the block layer handle encryption,
block/Makefile +2 -1
index 4147ffa63631..1ba7de84dbaf 100644
@@ -35,4 +35,5 @@ obj-$(CONFIG_BLK_DEBUG_FS)	+= blk-mq-debugfs.o
 obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
 obj-$(CONFIG_BLK_SED_OPAL)	+= sed-opal.o
 obj-$(CONFIG_BLK_PM)		+= blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o bio-crypt-ctx.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= keyslot-manager.o bio-crypt-ctx.o \
+					   blk-crypto.o
block/bio-crypt-ctx.c +6 -1
index aa3571f72ee7..6a2b061865c6 100644
@@ -43,7 +43,12 @@ EXPORT_SYMBOL(bio_crypt_free_ctx);
 
 int bio_crypt_clone(struct bio *dst, struct bio *src, gfp_t gfp_mask)
 {
-	if (!bio_has_crypt_ctx(src))
+	/*
+	 * If a bio is swhandled, then it will be decrypted when bio_endio
+	 * is called. As we only want the data to be decrypted once, copies
+	 * of the bio must not have have a crypt context.
+	 */
+	if (!bio_has_crypt_ctx(src) || bio_crypt_swhandled(src))
 		return 0;
 
 	dst->bi_crypt_context = bio_crypt_alloc_ctx(gfp_mask);
block/bio.c +5 -0
index ada9850c90dc..e2537e5588ac 100644
@@ -17,6 +17,7 @@
 #include <linux/cgroup.h>
 #include <linux/blk-cgroup.h>
 #include <linux/highmem.h>
+#include <linux/blk-crypto.h>
 
 #include <trace/events/block.h>
 #include "blk.h"
@@ -1800,6 +1801,10 @@ void bio_endio(struct bio *bio)
 again:
 	if (!bio_remaining_done(bio))
 		return;
+
+	if (!blk_crypto_endio(bio))
+		return;
+
 	if (!bio_integrity_endio(bio))
 		return;
 
block/blk-core.c +10 -1
index 35027e80e27d..f699ecd9ca2e 100644
@@ -36,6 +36,7 @@
 #include <linux/blk-cgroup.h>
 #include <linux/debugfs.h>
 #include <linux/bpf.h>
+#include <linux/blk-crypto.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/block.h>
@@ -1049,7 +1050,9 @@ blk_qc_t generic_make_request(struct bio *bio)
 			/* Create a fresh bio_list for all subordinate requests */
 			bio_list_on_stack[1] = bio_list_on_stack[0];
 			bio_list_init(&bio_list_on_stack[0]);
-			ret = q->make_request_fn(q, bio);
+
+			if (!blk_crypto_submit_bio(&bio))
+				ret = q->make_request_fn(q, bio);
 
 			blk_queue_exit(q);
 
@@ -1102,6 +1105,9 @@ blk_qc_t direct_make_request(struct bio *bio)
 	if (!generic_make_request_checks(bio))
 		return BLK_QC_T_NONE;
 
+	if (blk_crypto_submit_bio(&bio))
+		return BLK_QC_T_NONE;
+
 	if (unlikely(blk_queue_enter(q, nowait ? BLK_MQ_REQ_NOWAIT : 0))) {
 		if (nowait && !blk_queue_dying(q))
 			bio->bi_status = BLK_STS_AGAIN;
@@ -1772,5 +1778,8 @@ int __init blk_dev_init(void)
 	if (bio_crypt_ctx_init() < 0)
 		panic("Failed to allocate mem for bio crypt ctxs\n");
 
+	if (blk_crypto_init() < 0)
+		panic("Failed to init blk-crypto\n");
+
 	return 0;
 }
block/blk-crypto.c new file +737 -0

737 lines not on this page · raw message

include/linux/bio-crypt-ctx.h +7 -0
index ebe456289338..b9e0515143a4 100644
@@ -60,6 +60,8 @@ static inline void bio_crypt_advance(struct bio *bio, unsigned int bytes)
 	}
 }
 
+extern bool bio_crypt_swhandled(struct bio *bio);
+
 static inline bool bio_crypt_has_keyslot(struct bio *bio)
 {
 	return bio->bi_crypt_context->keyslot >= 0;
@@ -177,6 +179,11 @@ static inline void bio_crypt_set_ctx(struct bio *bio,
 				     unsigned int dun_bits,
 				     gfp_t gfp_mask) { }
 
+static inline bool bio_crypt_swhandled(struct bio *bio)
+{
+	return false;
+}
+
 static inline void bio_set_data_unit_num(struct bio *bio, u64 dun) { }
 
 static inline bool bio_crypt_has_keyslot(struct bio *bio)
include/linux/blk-crypto.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
maintainer
#7 lore raw

Re: [PATCH v4 3/8] block: blk-crypto for Inline Encryption

Quoted 10 lines · On Wed, 21 Aug 2019 00:57:09 -0700 Satya Tangirala <[email protected]> wrote:

We introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.

Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to Documentation/block/blk-crypto.txt.

So that file doesn't seem to exist; did you mean inline-encryption.txt here?

Documentation/block/inline-encryption.txt

Quoted 21 lines
---
 Documentation/block/inline-encryption.txt | 186 ++++++
 block/Kconfig                             |   2 +
 block/Makefile                            |   3 +-
 block/bio-crypt-ctx.c                     |   7 +-
 block/bio.c                               |   5 +
 block/blk-core.c                          |  11 +-
 block/blk-crypto.c                        | 737 ++++++++++++++++++++++
 include/linux/bio-crypt-ctx.h             |   7 +
 include/linux/blk-crypto.h                |  47 ++
 9 files changed, 1002 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/block/inline-encryption.txt
 create mode 100644 block/blk-crypto.c
 create mode 100644 include/linux/blk-crypto.h
Documentation/block/inline-encryption.txt new file +0 -0
new file mode 100644
index 000000000000..925611a5ea65

So we've been doing our best to get rid of .txt files in the documentation tree. I'd really be a lot happier if this were an RST file instead. The good news is that it's already 99% RST, so little would have to change.

See the info in Documentation/doc-guide for details.

Documentation/block/inline-encryption.txt line 11

Quoted 12 lines
Documentation/block/inline-encryption.txt new file +11 -0
@@ -0,0 +1,186 @@
+BLK-CRYPTO and KEYSLOT MANAGER
+===========================
+
+CONTENTS
+1. Objective
+2. Constraints and notes
+3. Design
+4. Blk-crypto
+ 4-1 What does blk-crypto do on bio submission
+5. Layered Devices
+6. Future optimizations for layered devices

RST would generate this TOC for you, so you can take it out.

Quoted 5 lines
Documentation/block/inline-encryption.txt new file +5 -0
+1. Objective
+============
+
+We want to support inline encryption (IE) in the kernel.
+To allow for testing, we also want a crypto API fallback when actual

Documentation/block/inline-encryption.txt line 27

Quoted 10 lines
Documentation/block/inline-encryption.txt new file +10 -0
+IE hardware is absent. We also want IE to work with layered devices
+like dm and loopback (i.e. we want to be able to use the IE hardware
+of the underlying devices if present, or else fall back to crypto API
+en/decryption).
+
+
+2. Constraints and notes
+========================
+
+1) IE hardware have a limited number of “keyslots” that can be programmed

Some people get irate when they encounter non-ASCII characters in the docs; that includes "smart quotes".

Also, s/have/has/

Quoted 17 lines
Documentation/block/inline-encryption.txt new file +17 -0
+with an encryption context (key, algorithm, data unit size, etc.) at any time.
+One can specify a keyslot in a data request made to the device, and the
+device will en/decrypt the data using the encryption context programmed into
+that specified keyslot. When possible, we want to make multiple requests with
+the same encryption context share the same keyslot.
+
+2) We need a way for filesystems to specify an encryption context to use for
+en/decrypting a struct bio, and a device driver (like UFS) needs to be able
+to use that encryption context when it processes the bio.
+
+3) We need a way for device drivers to expose their capabilities in a unified
+way to the upper layers.
+
+
+3. Design
+=========
+

Documentation/block/inline-encryption.txt line 54

Quoted 10 lines
Documentation/block/inline-encryption.txt new file +10 -0
+We add a struct bio_crypt_ctx to struct bio that can represent an
+encryption context, because we need to be able to pass this encryption
+context from the FS layer to the device driver to act upon.
+
+While IE hardware works on the notion of keyslots, the FS layer has no
+knowledge of keyslots - it simply wants to specify an encryption context to
+use while en/decrypting a bio.
+
+We introduce a keyslot manager (KSM) that handles the translation from
+encryption contexts specified by the FS to keyslots on the IE hardware.

So...if this were RST, you could have directives to pull in the nice kerneldoc comments you've already put into the source.

I'll stop here...presumably I've made my point by now :)

Thanks for documenting this subsystem!

jon

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

maintainer
#8 lore raw

Re: [PATCH v4 3/8] block: blk-crypto for Inline Encryption

In reply to Satya Tangirala
Quoted 8 lines · Satya Tangirala
block/blk-crypto.c new file +2 -0
new file mode 100644
index 000000000000..c8f06264a0f5
@@ -0,0 +1,737 @@
+// SPDX-License-Identifier: GPL-2.0
+/*

block/blk-crypto.c line 12

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+ * Copyright 2019 Google LLC
+ */
+
+/*
+ * Refer to Documentation/block/inline-encryption.txt for detailed explanation.
+ */
+
+#ifdef pr_fmt
+#undef pr_fmt
+#endif

This is the beginning of the file, so the

#ifdef pr_fmt
#undef pr_fmt
#endif

is unnecessary.

block/blk-crypto.c line 55

Quoted 6 lines
block/blk-crypto.c new file +6 -0
+static struct blk_crypto_keyslot {
+	struct crypto_skcipher *tfm;
+	enum blk_crypto_mode_num crypto_mode;
+	u8 key[BLK_CRYPTO_MAX_KEY_SIZE];
+	struct crypto_skcipher *tfms[ARRAY_SIZE(blk_crypto_modes)];
+} *blk_crypto_keyslots;

It would be helpful if there was a comment somewhere explaining what's going on with the crypto tfms now, like:

/*
* Allocating a crypto tfm during I/O can deadlock, so we have to preallocate
* all a mode's tfms when that mode starts being used. Since each mode may need
* all the keyslots at some point, each mode needs its own tfm for each keyslot;
* thus, a keyslot may contain tfms for multiple modes. However, to match the
* behavior of real inline encryption hardware (which only supports a single
* encryption context per keyslot), we only allow one tfm per keyslot to be used
* at a time. Unused tfms have their keys cleared.
*/

Otherwise it's not at all obvious what's going on.

Quoted 22 lines
block/blk-crypto.c new file +22 -0
+
+static struct mutex tfms_lock[ARRAY_SIZE(blk_crypto_modes)];
+static bool tfms_inited[ARRAY_SIZE(blk_crypto_modes)];
+
+struct work_mem {
+	struct work_struct crypto_work;
+	struct bio *bio;
+};
+
+/* The following few vars are only used during the crypto API fallback */
+static struct keyslot_manager *blk_crypto_ksm;
+static struct workqueue_struct *blk_crypto_wq;
+static mempool_t *blk_crypto_page_pool;
+static struct kmem_cache *blk_crypto_work_mem_cache;
+
+bool bio_crypt_swhandled(struct bio *bio)
+{
+	return bio_has_crypt_ctx(bio) &&
+	       bio->bi_crypt_context->processing_ksm == blk_crypto_ksm;
+}
+
+static const u8 zeroes[BLK_CRYPTO_MAX_KEY_SIZE];

block/blk-crypto.c line 87

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+static void evict_keyslot(unsigned int slot)
+{
+	struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot];
+	enum blk_crypto_mode_num crypto_mode = slotp->crypto_mode;
+
+	/* Clear the key in the skcipher */
+	crypto_skcipher_setkey(slotp->tfms[crypto_mode], zeroes,
+			       blk_crypto_modes[crypto_mode].keysize);
+	memzero_explicit(slotp->key, BLK_CRYPTO_MAX_KEY_SIZE);
+}

Unfortunately setting the all-zeroes key won't work, because the all-zeroes key fails the "weak key" check for XTS, as its two halves are the same.

Presumably this wasn't noticed during testing because the return value of crypto_skcipher_setkey() is ignored. So I suggest adding a WARN_ON():

	err = crypto_skcipher_setkey(slotp->tfms[crypto_mode], blank_key,
				     blk_crypto_modes[crypto_mode].keysize);
	WARN_ON(err);

Then for the actual fix, maybe set a random key instead of an all-zeroes one?

Quoted 5 lines
block/blk-crypto.c new file +5 -0
+
+static int blk_crypto_keyslot_program(void *priv, const u8 *key,
+				      enum blk_crypto_mode_num crypto_mode,
+				      unsigned int data_unit_size,
+				      unsigned int slot)

block/blk-crypto.c line 102

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+{
+	struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot];
+	const struct blk_crypto_mode *mode = &blk_crypto_modes[crypto_mode];
+	size_t keysize = mode->keysize;
+	int err;
+
+	if (crypto_mode != slotp->crypto_mode) {
+		evict_keyslot(slot);
+		slotp->crypto_mode = crypto_mode;
+	}

Currently the crypto_mode of every blk_crypto_keyslot starts out as AES_256_XTS (0). So if the user starts by choosing some other mode, this will immediately call evict_keyslot() and crash dereferencing a NULL pointer.

To fix this, how about initializing all the modes to BLK_ENCRYPTION_MODE_INVALID?

Then here the code would need to be:

	if (crypto_mode != slotp->crypto_mode &&
	    slotp->crypto_mode != BLK_ENCRYPTION_MODE_INVALID)
		evict_keyslot(slot);

And evict_keyslot() should invalidate the crypto_mode:

static void evict_keyslot(unsigned int slot)

{
	...

	slotp->crypto_mode = BLK_ENCRYPTION_MODE_INVALID;
}

block/blk-crypto.c line 125

Quoted 9 lines
block/blk-crypto.c new file +9 -0
+
+static int blk_crypto_keyslot_evict(void *priv, const u8 *key,
+				    enum blk_crypto_mode_num crypto_mode,
+				    unsigned int data_unit_size,
+				    unsigned int slot)
+{
+	evict_keyslot(slot);
+	return 0;
+}

It might be useful to have a WARN_ON() here if the keyslot isn't in use (i.e., if slotp->crypto_mode == BLK_ENCRYPTION_MODE_INVALID).

Quoted 13 lines
block/blk-crypto.c new file +13 -0
+int blk_crypto_submit_bio(struct bio **bio_ptr)
+{
+	struct bio *bio = *bio_ptr;
+	struct request_queue *q;
+	int err;
+	struct bio_crypt_ctx *crypt_ctx;
+
+	if (!bio_has_crypt_ctx(bio) || !bio_has_data(bio))
+		return 0;
+
+	/*
+	 * When a read bio is marked for sw decryption, its bi_iter is saved
+	 * so that when we decrypt the bio later, we know what part of it was

block/blk-crypto.c line 528

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+	 * marked for sw decryption (when the bio is passed down after
+	 * blk_crypto_submit bio, it may be split or advanced so we cannot rely
+	 * on the bi_iter while decrypting in blk_crypto_endio)
+	 */
+	if (bio_crypt_swhandled(bio))
+		return 0;
+
+	err = bio_crypt_check_alignment(bio);
+	if (err)
+		goto out;

Need to set ->bi_status if bio_crypt_check_alignment() fails.

Quoted 13 lines
block/blk-crypto.c new file +13 -0
+bool blk_crypto_endio(struct bio *bio)
+{
+	if (!bio_has_crypt_ctx(bio))
+		return true;
+
+	if (bio_crypt_swhandled(bio)) {
+		/*
+		 * The only bios that are swhandled when they reach here
+		 * are those with bio_data_dir(bio) == READ, since WRITE
+		 * bios that are encrypted by the crypto API fallback are
+		 * handled by blk_crypto_encrypt_endio.
+		 */
+

block/blk-crypto.c line 614

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+		/* If there was an IO error, don't decrypt. */
+		if (bio->bi_status)
+			return true;
+
+		blk_crypto_queue_decrypt_bio(bio);
+		return false;
+	}
+
+	if (bio_has_crypt_ctx(bio) && bio_crypt_has_keyslot(bio))
+		bio_crypt_ctx_release_keyslot(bio);

No need to check bio_has_crypt_ctx(bio) here, as it was already checked above.

Quoted 5 lines
block/blk-crypto.c new file +5 -0
+int blk_crypto_mode_alloc_ciphers(enum blk_crypto_mode_num mode_num)
+{
+	struct blk_crypto_keyslot *slotp;
+	int err = 0;
+	int i;

block/blk-crypto.c line 643

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+
+	/* Fast path */
+	if (likely(READ_ONCE(tfms_inited[mode_num]))) {
+		/*
+		 * Ensure that updates to blk_crypto_keyslots[i].tfms[mode_num]
+		 * for each i are visible before we try to access them.
+		 */
+		smp_rmb();
+		return 0;
+	}

I think we want smp_load_acquire() here.

	/* pairs with smp_store_release() below */
	if (smp_load_acquire(&tfms_inited[mode_num]))
		return 0;
Quoted 17 lines
block/blk-crypto.c new file +17 -0
+
+	mutex_lock(&tfms_lock[mode_num]);
+	if (likely(tfms_inited[mode_num]))
+		goto out;
+
+	for (i = 0; i < blk_crypto_num_keyslots; i++) {
+		slotp = &blk_crypto_keyslots[i];
+		slotp->tfms[mode_num] = crypto_alloc_skcipher(
+					blk_crypto_modes[mode_num].cipher_str,
+					0, 0);
+		if (IS_ERR(slotp->tfms[mode_num])) {
+			err = PTR_ERR(slotp->tfms[mode_num]);
+			slotp->tfms[mode_num] = NULL;
+			goto out_free_tfms;
+		}
+
+		crypto_skcipher_set_flags(slotp->tfms[mode_num],

block/blk-crypto.c line 670

Quoted 10 lines
block/blk-crypto.c new file +10 -0
+					  CRYPTO_TFM_REQ_FORBID_WEAK_KEYS);
+	}
+
+	/*
+	 * Ensure that updates to blk_crypto_keyslots[i].tfms[mode_num]
+	 * for each i are visible before we set tfms_inited[mode_num].
+	 */
+	smp_wmb();
+	WRITE_ONCE(tfms_inited[mode_num], true);
+	goto out;

... and smp_store_release() here.

	/* pairs with smp_load_acquire() above */
	smp_store_release(&tfms_inited[mode_num], true);
	goto out;

- Eric

#9 lore raw

PATCH v4 4/8 scsi: ufs: UFS driver v2.1 spec crypto additions

In reply to Satya Tangirala

Add the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI specification in preparation to add support for inline encryption to UFS.

---
 drivers/scsi/ufs/ufshcd.c |  2 ++
 drivers/scsi/ufs/ufshcd.h |  5 +++
 drivers/scsi/ufs/ufshci.h | 67 +++++++++++++++++++++++++++++++++++++--
 3 files changed, 72 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +2 -0
index e274053109d0..212f7653c4c5 100644
@@ -4722,6 +4722,8 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	case OCS_MISMATCH_RESP_UPIU_SIZE:
 	case OCS_PEER_COMM_FAILURE:
 	case OCS_FATAL_ERROR:
+	case OCS_INVALID_CRYPTO_CONFIG:
+	case OCS_GENERAL_CRYPTO_ERROR:
 	default:
 		result |= DID_ERROR << 16;
 		dev_err(hba->dev,
drivers/scsi/ufs/ufshcd.h +5 -0
index 994d73d03207..10b5cd26a020 100644
@@ -692,6 +692,11 @@ struct ufs_hba {
 	 * the performance of ongoing read/write operations.
 	 */
 #define UFSHCD_CAP_KEEP_AUTO_BKOPS_ENABLED_EXCEPT_SUSPEND (1 << 5)
+	/*
+	 * This capability allows the host controller driver to use the
+	 * inline crypto engine, if it is present
+	 */
+#define UFSHCD_CAP_CRYPTO (1 << 6)
 
 	struct devfreq *devfreq;
 	struct ufs_clk_scaling clk_scaling;
drivers/scsi/ufs/ufshci.h +65 -2

106 lines not on this page · raw message

Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
#10 lore raw

PATCH v4 5/8 scsi: ufs: UFS crypto API

In reply to Satya Tangirala

Introduce functions to manipulate UFS inline encryption hardware in line with the JEDEC UFSHCI v2.1 specification and to work with the block keyslot manager.

---
 drivers/scsi/ufs/Kconfig         |  10 +
 drivers/scsi/ufs/Makefile        |   1 +
 drivers/scsi/ufs/ufshcd-crypto.c | 429 +++++++++++++++++++++++++++++++
 drivers/scsi/ufs/ufshcd-crypto.h |  86 +++++++
 drivers/scsi/ufs/ufshcd.h        |  18 ++
 5 files changed, 544 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
 create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
drivers/scsi/ufs/Kconfig +10 -0
index 0b845ab7c3bf..861aabfe791b 100644
@@ -150,3 +150,13 @@ config SCSI_UFS_BSG
 
 	  Select this if you need a bsg device node for your UFS controller.
 	  If unsure, say N.
+
+config SCSI_UFS_CRYPTO
+	bool "UFS Crypto Engine Support"
+	depends on SCSI_UFSHCD && BLK_INLINE_ENCRYPTION
+	help
+	  Enable Crypto Engine Support in UFS.
+	  Enabling this makes it possible for the kernel to use the crypto
+	  capabilities of the UFS device (if present) to perform crypto
+	  operations on data being transferred to/from the device.
+
drivers/scsi/ufs/Makefile +1 -0
index 2a9097939bcb..094c39989a37 100644
@@ -11,3 +11,4 @@ obj-$(CONFIG_SCSI_UFSHCD_PCI) += ufshcd-pci.o
 obj-$(CONFIG_SCSI_UFSHCD_PLATFORM) += ufshcd-pltfrm.o
 obj-$(CONFIG_SCSI_UFS_HISI) += ufs-hisi.o
 obj-$(CONFIG_SCSI_UFS_MEDIATEK) += ufs-mediatek.o
+ufshcd-core-$(CONFIG_SCSI_UFS_CRYPTO) += ufshcd-crypto.o
drivers/scsi/ufs/ufshcd-crypto.c new file +429 -0

429 lines not on this page · raw message

drivers/scsi/ufs/ufshcd-crypto.h new file +86 -0

86 lines not on this page · raw message

drivers/scsi/ufs/ufshcd.h +18 -0
index 10b5cd26a020..34e9849f00f0 100644
@@ -501,6 +501,13 @@ struct ufs_stats {
  * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
  *  device is known or not.
  * @scsi_block_reqs_cnt: reference counting for scsi block requests
+ * @crypto_capabilities: Content of crypto capabilities register (0x100)
+ * @crypto_cap_array: Array of crypto capabilities
+ * @crypto_cfg_register: Start of the crypto cfg array
+ * @crypto_cfgs: Array of crypto configurations (i.e. config for each slot)
+ * @ksm: the keyslot manager tied to this hba
+ * @ksm_lock: lock to protect initialization and refcount of ksm
+ * @ksm_num_refs: refcount for ksm
  */
 struct ufs_hba {
 	void __iomem *mmio_base;
@@ -711,6 +718,17 @@ struct ufs_hba {
 
 	struct device		bsg_dev;
 	struct request_queue	*bsg_queue;
+
+#ifdef CONFIG_SCSI_UFS_CRYPTO
+	/* crypto */
+	union ufs_crypto_capabilities crypto_capabilities;
+	union ufs_crypto_cap_entry *crypto_cap_array;
+	u32 crypto_cfg_register;
+	union ufs_crypto_cfg_entry *crypto_cfgs;
+	struct keyslot_manager *ksm;
+	struct mutex ksm_lock;
+	unsigned int ksm_num_refs;
+#endif /* CONFIG_SCSI_UFS_CRYPTO */
 };
 
 /* Returns true if clocks can be gated. Otherwise false */
Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
maintainer
#11 lore raw

Re: [PATCH v4 5/8] scsi: ufs: UFS crypto API

Quoted 20 lines · Satya Tangirala
drivers/scsi/ufs/ufshcd-crypto.c new file +20 -0
+static int ufshcd_crypto_cap_find(void *hba_p,
+			   enum blk_crypto_mode_num crypto_mode,
+			   unsigned int data_unit_size)
+{
+	struct ufs_hba *hba = hba_p;
+	enum ufs_crypto_alg ufs_alg;
+	u8 data_unit_mask;
+	int cap_idx;
+	enum ufs_crypto_key_size ufs_key_size;
+	union ufs_crypto_cap_entry *ccap_array = hba->crypto_cap_array;
+
+	if (!ufshcd_hba_is_crypto_supported(hba))
+		return -EINVAL;
+
+	switch (crypto_mode) {
+	case BLK_ENCRYPTION_MODE_AES_256_XTS:
+		ufs_alg = UFS_CRYPTO_ALG_AES_XTS;
+		ufs_key_size = UFS_CRYPTO_KEY_SIZE_256;
+		break;
+	/*

drivers/scsi/ufs/ufshcd-crypto.c line 65

Quoted 10 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +10 -0
+	 * case BLK_CRYPTO_ALG_BITLOCKER_AES_CBC:
+	 *	ufs_alg = UFS_CRYPTO_ALG_BITLOCKER_AES_CBC;
+	 *	break;
+	 * case BLK_CRYPTO_ALG_AES_ECB:
+	 *	ufs_alg = UFS_CRYPTO_ALG_AES_ECB;
+	 *	break;
+	 * case BLK_CRYPTO_ALG_ESSIV_AES_CBC:
+	 *	ufs_alg = UFS_CRYPTO_ALG_ESSIV_AES_CBC;
+	 *	break;
+	 */

Perhaps just delete this comment... the constants are already outdated.

drivers/scsi/ufs/ufshcd-crypto.c line 345

Quoted 5 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +5 -0
+	hba->crypto_cfgs =
+		devm_kcalloc(hba->dev,
+			     hba->crypto_capabilities.config_count + 1,
+			     sizeof(hba->crypto_cfgs[0]),
+			     GFP_KERNEL);

Can use NUM_KEYSLOTS(hba) here, to avoid hardcoding the awkward '+ 1' again.

drivers/scsi/ufs/ufshcd-crypto.c line 396

Quoted 13 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +13 -0
+void ufshcd_crypto_setup_rq_keyslot_manager(struct ufs_hba *hba,
+					    struct request_queue *q)
+{
+	if (!ufshcd_hba_is_crypto_supported(hba))
+		return;
+
+	if (q) {
+		mutex_lock(&hba->ksm_lock);
+		if (!hba->ksm) {
+			hba->ksm = keyslot_manager_create(
+				hba->crypto_capabilities.config_count + 1,
+				&ufshcd_ksm_ops, hba);
+			hba->ksm_num_refs = 0;

Same here.

Quoted 22 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +22 -0
+		}
+		hba->ksm_num_refs++;
+		mutex_unlock(&hba->ksm_lock);
+		q->ksm = hba->ksm;
+	}
+	/*
+	 * If we fail we make it look like
+	 * crypto is not supported, which will avoid issues
+	 * with reset
+	 */
+	if (!q || !q->ksm) {
+		ufshcd_crypto_disable(hba);
+		hba->crypto_capabilities.reg_val = 0;
+		devm_kfree(hba->dev, hba->crypto_cap_array);
+		devm_kfree(hba->dev, hba->crypto_cfgs);
+	}
+}
+
+void ufshcd_crypto_destroy_rq_keyslot_manager(struct ufs_hba *hba,
+					      struct request_queue *q)
+{
+	if (q && q->ksm) {

drivers/scsi/ufs/ufshcd-crypto.c line 428

Quoted 10 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +10 -0
+		q->ksm = NULL;
+		mutex_lock(&hba->ksm_lock);
+		hba->ksm_num_refs--;
+		if (hba->ksm_num_refs == 0) {
+			keyslot_manager_destroy(hba->ksm);
+			hba->ksm = NULL;
+		}
+		mutex_unlock(&hba->ksm_lock);
+	}
+}

Why is the keyslot_manager reference counted? Doesn't it live as long as the individual devices do? So, can't we just create the keyslot manager when the ufs_hba is created, and destroy it when the ufs_hba is destroyed? Then for each device we'd just set 'q->ksm = hba->ksm;', with no refcounting needed.

- Eric

#12 lore raw

PATCH v4 6/8 scsi: ufs: Add inline encryption support to UFS

In reply to Satya Tangirala

Wire up ufshcd.c with the UFS Crypto API, the block layer inline encryption additions and the keyslot manager.

---
 drivers/scsi/ufs/ufshcd.c | 83 ++++++++++++++++++++++++++++++++++++---
 drivers/scsi/ufs/ufshcd.h |  6 +++
 2 files changed, 84 insertions(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +78 -5
index 212f7653c4c5..dcb841c6be7e 100644
@@ -47,6 +47,7 @@
 #include "unipro.h"
 #include "ufs-sysfs.h"
 #include "ufs_bsg.h"
+#include "ufshcd-crypto.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/ufs.h>
@@ -855,7 +856,14 @@ static void ufshcd_enable_run_stop_reg(struct ufs_hba *hba)
  */
 static inline void ufshcd_hba_start(struct ufs_hba *hba)
 {
-	ufshcd_writel(hba, CONTROLLER_ENABLE, REG_CONTROLLER_ENABLE);
+	u32 val = CONTROLLER_ENABLE;
+
+	if (ufshcd_hba_is_crypto_supported(hba)) {
+		ufshcd_crypto_enable(hba);
+		val |= CRYPTO_GENERAL_ENABLE;
+	}
+
+	ufshcd_writel(hba, val, REG_CONTROLLER_ENABLE);
 }
 
 /**
@@ -2209,9 +2217,21 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,
 		dword_0 |= UTP_REQ_DESC_INT_CMD;
 
 	/* Transfer request descriptor header fields */
+	if (lrbp->crypto_enable) {
+		dword_0 |= UTP_REQ_DESC_CRYPTO_ENABLE_CMD;
+		dword_0 |= lrbp->crypto_key_slot;
+		req_desc->header.dword_1 =
+			cpu_to_le32((u32)lrbp->data_unit_num);
+		req_desc->header.dword_3 =
+			cpu_to_le32((u32)(lrbp->data_unit_num >> 32));
+	} else {
+		/* dword_1 and dword_3 are reserved, hence they are set to 0 */
+		req_desc->header.dword_1 = 0;
+		req_desc->header.dword_3 = 0;
+	}
+
 	req_desc->header.dword_0 = cpu_to_le32(dword_0);
-	/* dword_1 is reserved, hence it is set to 0 */
-	req_desc->header.dword_1 = 0;
+
 	/*
 	 * assigning invalid value for command status. Controller
 	 * updates OCS on command completion, with the command
@@ -2219,8 +2239,6 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,
 	 */
 	req_desc->header.dword_2 =
 		cpu_to_le32(OCS_INVALID_COMMAND_STATUS);
-	/* dword_3 is reserved, hence it is set to 0 */
-	req_desc->header.dword_3 = 0;
 
 	req_desc->prd_table_length = 0;
 }
@@ -2380,6 +2398,37 @@ static inline u16 ufshcd_upiu_wlun_to_scsi_wlun(u8 upiu_wlun_id)
 	return (upiu_wlun_id & ~UFS_UPIU_WLUN_ID) | SCSI_W_LUN_BASE;
 }
 
+static inline int ufshcd_prepare_lrbp_crypto(struct ufs_hba *hba,
+					     struct scsi_cmnd *cmd,
+					     struct ufshcd_lrb *lrbp)
+{
+	int key_slot;
+
+	if (!cmd->request->bio ||
+	    !bio_crypt_should_process(cmd->request->bio, cmd->request->q)) {
+		lrbp->crypto_enable = false;
+		return 0;
+	}
+
+	if (WARN_ON(!ufshcd_is_crypto_enabled(hba))) {
+		/*
+		 * Upper layer asked us to do inline encryption
+		 * but that isn't enabled, so we fail this request.
+		 */
+		return -EINVAL;
+	}
+	key_slot = bio_crypt_get_keyslot(cmd->request->bio);
+	if (!ufshcd_keyslot_valid(hba, key_slot))
+		return -EINVAL;
+
+	lrbp->crypto_enable = true;
+	lrbp->crypto_key_slot = key_slot;
+	lrbp->data_unit_num = bio_crypt_data_unit_num(cmd->request->bio);
+
+	return 0;
+}
+
+
 /**
  * ufshcd_queuecommand - main entry point for SCSI requests
  * @host: SCSI host pointer
@@ -2467,6 +2516,13 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	lrbp->task_tag = tag;
 	lrbp->lun = ufshcd_scsi_to_upiu_lun(cmd->device->lun);
 	lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed(hba) ? true : false;
+
+	err = ufshcd_prepare_lrbp_crypto(hba, cmd, lrbp);
+	if (err) {
+		lrbp->cmd = NULL;
+		clear_bit_unlock(tag, &hba->lrb_in_use);
+		goto out;
+	}
 	lrbp->req_abort_skip = false;
 
 	ufshcd_comp_scsi_upiu(hba, lrbp);
@@ -2500,6 +2556,7 @@ static int ufshcd_compose_dev_cmd(struct ufs_hba *hba,
 	lrbp->task_tag = tag;
 	lrbp->lun = 0; /* device management cmd is not specific to any LUN */
 	lrbp->intr_cmd = true; /* No interrupt aggregation */
+	lrbp->crypto_enable = false; /* No crypto operations */
 	hba->dev_cmd.type = cmd_type;
 
 	return ufshcd_comp_devman_upiu(hba, lrbp);
@@ -4192,6 +4249,8 @@ static inline void ufshcd_hba_stop(struct ufs_hba *hba, bool can_sleep)
 {
 	int err;
 
+	ufshcd_crypto_disable(hba);
+
 	ufshcd_writel(hba, CONTROLLER_DISABLE,  REG_CONTROLLER_ENABLE);
 	err = ufshcd_wait_for_register(hba, REG_CONTROLLER_ENABLE,
 					CONTROLLER_ENABLE, CONTROLLER_DISABLE,
@@ -4585,8 +4644,12 @@ static int ufshcd_change_queue_depth(struct scsi_device *sdev, int depth)
 static int ufshcd_slave_configure(struct scsi_device *sdev)
 {
 	struct request_queue *q = sdev->request_queue;
+	struct ufs_hba *hba = shost_priv(sdev->host);
 
 	blk_queue_update_dma_pad(q, PRDT_DATA_BYTE_COUNT_PAD - 1);
+
+	ufshcd_crypto_setup_rq_keyslot_manager(hba, q);
+
 	return 0;
 }
 
@@ -4597,6 +4660,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)
 static void ufshcd_slave_destroy(struct scsi_device *sdev)
 {
 	struct ufs_hba *hba;
+	struct request_queue *q = sdev->request_queue;
 
 	hba = shost_priv(sdev->host);
 	/* Drop the reference as it won't be needed anymore */
@@ -4607,6 +4671,8 @@ static void ufshcd_slave_destroy(struct scsi_device *sdev)
 		hba->sdev_ufs_device = NULL;
 		spin_unlock_irqrestore(hba->host->host_lock, flags);
 	}
+
+	ufshcd_crypto_destroy_rq_keyslot_manager(hba, q);
 }
 
 /**
@@ -8323,6 +8389,13 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 		goto exit_gating;
 	}
 
+	/* Init crypto */
+	err = ufshcd_hba_init_crypto(hba);
+	if (err) {
+		dev_err(hba->dev, "crypto setup failed\n");
+		goto out_remove_scsi_host;
+	}
+
 	/* Host controller enable */
 	err = ufshcd_hba_enable(hba);
 	if (err) {
drivers/scsi/ufs/ufshcd.h +6 -0
index 34e9849f00f0..c3e1f409e63d 100644
@@ -167,6 +167,9 @@ struct ufs_pm_lvl_states {
  * @intr_cmd: Interrupt command (doesn't participate in interrupt aggregation)
  * @issue_time_stamp: time stamp for debug purposes
  * @compl_time_stamp: time stamp for statistics
+ * @crypto_enable: whether or not the request needs inline crypto operations
+ * @crypto_key_slot: the key slot to use for inline crypto
+ * @data_unit_num: the data unit number for the first block for inline crypto
  * @req_abort_skip: skip request abort task flag
  */
 struct ufshcd_lrb {
@@ -191,6 +194,9 @@ struct ufshcd_lrb {
 	bool intr_cmd;
 	ktime_t issue_time_stamp;
 	ktime_t compl_time_stamp;
+	bool crypto_enable;
+	u8 crypto_key_slot;
+	u64 data_unit_num;
 
 	bool req_abort_skip;
 };
Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
#13 lore raw

PATCH v4 7/8 fscrypt: wire up fscrypt to use blk-crypto

In reply to Satya Tangirala

Introduce fscrypt_set_bio_crypt_ctx for filesystems to call to set up encryption contexts in bios, and fscrypt_evict_crypt_key to evict the encryption context associated with an inode.

Inline encryption is controlled by a policy flag in the fscrypt_info in the inode, and filesystems may check if an inode should use inline encryption by calling fscrypt_inode_is_inline_crypted. Files can be marked as inline encrypted from userspace by appropriately modifying the flags (OR-ing FS_POLICY_FLAGS_INLINE_ENCRYPTION to it) in the fscrypt_policy passed to fscrypt_ioctl_set_policy.

To test inline encryption with the fscrypt dummy context, add ctx.flags |= FS_POLICY_FLAGS_INLINE_ENCRYPTION
when setting up the dummy context in fs/crypto/keyinfo.c.

Note that blk-crypto will fall back to software en/decryption in the absence of inline crypto hardware, so setting up the ctx.flags in the dummy context without inline crypto hardware serves as a test for the software fallback in blk-crypto.

---
 fs/crypto/Kconfig           |   6 ++
 fs/crypto/bio.c             | 137 ++++++++++++++++++++++++++++++++----
 fs/crypto/fscrypt_private.h |  23 ++++++
 fs/crypto/keyinfo.c         | 107 +++++++++++++++++++++-------
 fs/crypto/policy.c          |   6 ++
 include/linux/fscrypt.h     |  72 +++++++++++++++++++
 include/uapi/linux/fs.h     |   3 +-
 7 files changed, 316 insertions(+), 38 deletions(-)
fs/crypto/Kconfig +6 -0
index 5fdf24877c17..8191e0ff5014 100644
@@ -14,3 +14,9 @@ config FS_ENCRYPTION
 	  efficient since it avoids caching the encrypted and
 	  decrypted pages in the page cache.  Currently Ext4,
 	  F2FS and UBIFS make use of this feature.
+
+config FS_ENCRYPTION_INLINE_CRYPT
+	bool "Enable fscrypt to use inline crypto"
+	depends on FS_ENCRYPTION && BLK_INLINE_ENCRYPTION
+	help
+	  Enables fscrypt to use inline crypto hardware if available.
fs/crypto/bio.c +125 -12

164 lines not on this page · raw message

fs/crypto/fscrypt_private.h +23 -0
index 8978eec9d766..3079405a2b12 100644
@@ -14,6 +14,7 @@
 
 #include <linux/fscrypt.h>
 #include <crypto/hash.h>
+#include <linux/blk-crypto.h>
 
 /* Encryption parameters */
 #define FS_KEY_DERIVATION_NONCE_SIZE	16
@@ -49,6 +50,17 @@ struct fscrypt_symlink_data {
 	char encrypted_path[1];
 } __packed;
 
+/* Master key referenced by FS_POLICY_FLAG_DIRECT_KEY policy */
+struct fscrypt_master_key {
+	struct hlist_node mk_node;
+	refcount_t mk_refcount;
+	const struct fscrypt_mode *mk_mode;
+	struct crypto_skcipher *mk_ctfm;
+	u8 mk_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+	u8 mk_raw[FS_MAX_KEY_SIZE];
+	struct super_block *mk_sb;
+};
+
 /*
  * fscrypt_info - the "encryption key" for an inode
  *
@@ -113,6 +125,17 @@ static inline bool fscrypt_valid_enc_modes(u32 contents_mode,
 	return false;
 }
 
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+extern enum blk_crypto_mode_num
+get_blk_crypto_mode_for_fscryptalg(u8 fscrypt_alg);
+#else
+static inline enum blk_crypto_mode_num
+get_blk_crypto_mode_for_fscryptalg(u8 fscrypt_alg)
+{
+	return BLK_ENCRYPTION_MODE_INVALID;
+}
+#endif
+
 /* crypto.c */
 extern struct kmem_cache *fscrypt_info_cachep;
 extern int fscrypt_initialize(unsigned int cop_flags);
fs/crypto/keyinfo.c +82 -25

216 lines not on this page · raw message

fs/crypto/policy.c +6 -0
index 4941fe8471ce..573407a2cda3 100644
@@ -36,6 +36,7 @@ static int create_encryption_context_from_policy(struct inode *inode,
 	struct fscrypt_context ctx;
 
 	ctx.format = FS_ENCRYPTION_CONTEXT_FORMAT_V1;
+
 	memcpy(ctx.master_key_descriptor, policy->master_key_descriptor,
 					FS_KEY_DESCRIPTOR_SIZE);
 
@@ -46,8 +47,13 @@ static int create_encryption_context_from_policy(struct inode *inode,
 	if (policy->flags & ~FS_POLICY_FLAGS_VALID)
 		return -EINVAL;
 
+	if (!inode->i_sb->s_cop->inline_crypt_supp &&
+	    (policy->flags & FS_POLICY_FLAGS_INLINE_CRYPT_OPTIMIZED))
+		return -EINVAL;
+
 	ctx.contents_encryption_mode = policy->contents_encryption_mode;
 	ctx.filenames_encryption_mode = policy->filenames_encryption_mode;
+
 	ctx.flags = policy->flags;
 	BUILD_BUG_ON(sizeof(ctx.nonce) != FS_KEY_DERIVATION_NONCE_SIZE);
 	get_random_bytes(ctx.nonce, FS_KEY_DERIVATION_NONCE_SIZE);
include/linux/fscrypt.h +72 -0

102 lines not on this page · raw message

include/uapi/linux/fs.h +2 -1
index 59c71fa8c553..dea16d0f9d2e 100644
@@ -224,7 +224,8 @@ struct fsxattr {
 #define FS_POLICY_FLAGS_PAD_32		0x03
 #define FS_POLICY_FLAGS_PAD_MASK	0x03
 #define FS_POLICY_FLAG_DIRECT_KEY	0x04	/* use master key directly */
-#define FS_POLICY_FLAGS_VALID		0x07
+#define FS_POLICY_FLAGS_INLINE_CRYPT_OPTIMIZED	0x08
+#define FS_POLICY_FLAGS_VALID		0x0F
 
 /* Encryption algorithms */
 #define FS_ENCRYPTION_MODE_INVALID		0
Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog
maintainer
#14 lore raw

Re: [PATCH v4 7/8] fscrypt: wire up fscrypt to use blk-crypto

Quoted 11 lines · Satya Tangirala

Introduce fscrypt_set_bio_crypt_ctx for filesystems to call to set up encryption contexts in bios, and fscrypt_evict_crypt_key to evict the encryption context associated with an inode.

Inline encryption is controlled by a policy flag in the fscrypt_info in the inode, and filesystems may check if an inode should use inline encryption by calling fscrypt_inode_is_inline_crypted. Files can be marked as inline encrypted from userspace by appropriately modifying the flags (OR-ing FS_POLICY_FLAGS_INLINE_ENCRYPTION to it) in the fscrypt_policy passed to fscrypt_ioctl_set_policy.

Quoted 3 lines

To test inline encryption with the fscrypt dummy context, add ctx.flags |= FS_POLICY_FLAGS_INLINE_ENCRYPTION
when setting up the dummy context in fs/crypto/keyinfo.c.

INLINE_ENCRYPTION => INLINE_CRYPT_OPTIMIZED. (Code was updated, but the commit message was not.)

Quoted 4 lines
fs/crypto/bio.c +0 -0
index 82da2510721f..d3c3f63ec109 100644

[...]

fs/crypto/bio.c int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · line 145

Quoted 10 lines
fs/crypto/bio.c +10 -0
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+enum blk_crypto_mode_num
+get_blk_crypto_mode_for_fscryptalg(u8 fscrypt_alg)
+{
+	switch (fscrypt_alg) {
+	case FS_ENCRYPTION_MODE_AES_256_XTS:
+		return BLK_ENCRYPTION_MODE_AES_256_XTS;
+	default: return BLK_ENCRYPTION_MODE_INVALID;
+	}
+}

This function isn't static, so it needs the "fscrypt_" prefix. How about: fscrypt_mode_to_block_mode(u8 fscrypt_mode);

Quoted 19 lines
fs/crypto/bio.c +19 -0
+int fscrypt_set_bio_crypt_ctx(struct bio *bio,
+			      const struct inode *inode,
+			      u64 data_unit_num,
+			      gfp_t gfp_mask)
+{
+	struct fscrypt_info *ci = inode->i_crypt_info;
+	int err;
+	enum blk_crypto_mode_num blk_crypto_mode;
+
+
+	/* If inode is not inline encrypted, nothing to do. */
+	if (!fscrypt_inode_is_inline_crypted(inode))
+		return 0;
+
+	blk_crypto_mode = get_blk_crypto_mode_for_fscryptalg(ci->ci_data_mode);
+	if (blk_crypto_mode == BLK_ENCRYPTION_MODE_INVALID)
+		return -EINVAL;
+
+	err = bio_crypt_set_ctx(bio, ci->ci_master_key->mk_raw,

fs/crypto/bio.c int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · line 175

Quoted 10 lines
fs/crypto/bio.c +10 -0
+				blk_crypto_mode,
+				data_unit_num,
+				inode->i_blkbits,
+				gfp_mask);
+	if (err)
+		return err;
+
+	return 0;
+}
+EXPORT_SYMBOL(fscrypt_set_bio_crypt_ctx);

The end can shortened to: return bio_crypt_set_ctx(...)

Quoted 13 lines
fs/crypto/bio.c +13 -0
+int fscrypt_evict_crypt_key(struct inode *inode)
+{
+	struct request_queue *q;
+	struct fscrypt_info *ci;
+
+	if (!inode)
+		return 0;
+
+	q = inode->i_sb->s_bdev->bd_queue;
+	ci = inode->i_crypt_info;
+
+	if (!q || !q->ksm || !ci ||
+	    !fscrypt_inode_is_inline_crypted(inode)) {

fs/crypto/bio.c int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · line 205

Quoted 10 lines
fs/crypto/bio.c +10 -0
+		return 0;
+	}
+
+	return keyslot_manager_evict_key(q->ksm,
+					 ci->ci_master_key->mk_raw,
+					 get_blk_crypto_mode_for_fscryptalg(
+						ci->ci_data_mode),
+					 1 << inode->i_blkbits);
+}
+EXPORT_SYMBOL(fscrypt_evict_crypt_key);

This is really evicting a master key that may be shared by many inodes... So it doesn't make sense to pass a specific inode here. Shouldn't it pass the struct fscrypt_master_key itself?

Also, this function needs kerneldoc.

Quoted 16 lines
fs/crypto/bio.c +16 -0
+
+bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+				   const struct inode *inode_2)
+{
+	struct fscrypt_info *ci_1, *ci_2;
+	bool enc_1 = !inode_1 || fscrypt_inode_is_inline_crypted(inode_1);
+	bool enc_2 = !inode_2 || fscrypt_inode_is_inline_crypted(inode_2);
+
+	if (enc_1 != enc_2)
+		return false;
+
+	if (!enc_1)
+		return true;
+
+	if (inode_1 == inode_2)
+		return true;

fs/crypto/bio.c int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · line 231

Quoted 10 lines
fs/crypto/bio.c +10 -0
+
+	ci_1 = inode_1->i_crypt_info;
+	ci_2 = inode_2->i_crypt_info;
+
+	return ci_1->ci_data_mode == ci_2->ci_data_mode &&
+	       crypto_memneq(ci_1->ci_master_key->mk_raw,
+			     ci_2->ci_master_key->mk_raw,
+			     ci_1->ci_master_key->mk_mode->keysize) == 0;
+}
+EXPORT_SYMBOL(fscrypt_inode_crypt_mergeable);

Needs kerneldoc.

Quoted 4 lines
fs/crypto/keyinfo.c +0 -0
index 207ebed918c1..989cf12217df 100644

[...]

Quoted 7 lines
fs/crypto/keyinfo.c +5 -1
-	if (cmpxchg_release(&inode->i_crypt_info, NULL, crypt_info) == NULL)
+	if (cmpxchg_release(&inode->i_crypt_info, NULL, crypt_info) == NULL) {
 		crypt_info = NULL;
+		if (!flags_inline_crypted(ctx.flags, inode))
+			goto out;
+		blk_crypto_mode = get_blk_crypto_mode_for_fscryptalg(
+			inode->i_crypt_info->ci_mode - available_modes);

fs/crypto/keyinfo.c int fscrypt_get_encryption_info(struct inode *inode) · line 614

Quoted 10 lines
fs/crypto/keyinfo.c +10 -0
+
+		if (keyslot_manager_rq_crypto_mode_supported(
+						inode->i_sb->s_bdev->bd_queue,
+						blk_crypto_mode,
+						(1 << inode->i_blkbits))) {
+			goto out;
+		}
+
+		blk_crypto_mode_alloc_ciphers(blk_crypto_mode);
+	}

As soon as ->i_crypt_info is set by the cmpxchg_release(), another thread can start I/O to the file. So it's too late to call blk_crypto_mode_alloc_ciphers() here; it needs to happen before the cmpxchg_release().

Also, shouldn't keyslot_manager_rq_crypto_mode_supported() and blk_crypto_mode_alloc_ciphers() be combined into a single function like blk_crypto_start_using_mode()? The fact that it may have to pre-allocate the crypto transforms is an implementation detail, not something that users of the block layer should care about, it seems...

Quoted 23 lines
include/linux/fscrypt.h +8 -0
index bd8f207a2fb6..6db1c7c5009d 100644
@@ -61,6 +61,7 @@ struct fscrypt_operations {
 	bool (*dummy_context)(struct inode *);
 	bool (*empty_dir)(struct inode *);
 	unsigned int max_namelen;
+	bool inline_crypt_supp;
 };
 
 /* Decryption work */
@@ -141,6 +142,23 @@ extern int fscrypt_inherit_context(struct inode *, struct inode *,
 extern int fscrypt_get_encryption_info(struct inode *);
 extern void fscrypt_put_encryption_info(struct inode *);
 extern void fscrypt_free_inode(struct inode *);
+extern bool fscrypt_needs_fs_layer_crypto(const struct inode *inode);
+
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+extern bool fscrypt_inode_is_inline_crypted(const struct inode *inode);
+extern bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+					  const struct inode *inode_2);
+#else

include/linux/fscrypt.h extern int fscrypt_inherit_context(struct inode *, struct inode *, · line 161

Quoted 10 lines
include/linux/fscrypt.h +10 -0
+static inline bool fscrypt_inode_is_inline_crypted(const struct inode *inode)
+{
+	return false;
+}
+static inline bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+						 const struct inode *inode_2)
+{
+	return true;
+}
+#endif /* CONFIG_FS_ENCRYPTION_INLINE_CRYPT */

Please try to put all the declarations in the right place. E.g. fscrypt_inode_crypt_mergeable() is in the /* keyinfo.c */ part of this header, but it's actually defined in fs/crypto/bio.c.

Quoted 20 lines
include/linux/fscrypt.h +13 -0
 /* fname.c */
 extern int fscrypt_setup_filename(struct inode *, const struct qstr *,
@@ -237,6 +255,29 @@ extern void fscrypt_enqueue_decrypt_bio(struct fscrypt_ctx *ctx,
 					struct bio *bio);
 extern int fscrypt_zeroout_range(const struct inode *, pgoff_t, sector_t,
 				 unsigned int);
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+extern int fscrypt_set_bio_crypt_ctx(struct bio *bio,
+				     const struct inode *inode,
+				     u64 data_unit_num,
+				     gfp_t gfp_mask);
+extern void fscrypt_unset_bio_crypt_ctx(struct bio *bio);
+extern int fscrypt_evict_crypt_key(struct inode *inode);
+#else
+static inline int fscrypt_set_bio_crypt_ctx(struct bio *bio,
+					    const struct inode *inode,
+					    u64 data_unit_num,
+					    gfp_t gfp_mask)
+{

include/linux/fscrypt.h extern void fscrypt_enqueue_decrypt_bio(struct fscrypt_ctx *ctx, · line 280

Quoted 10 lines
include/linux/fscrypt.h +10 -0
+	return 0;
+}
+
+static inline void fscrypt_unset_bio_crypt_ctx(struct bio *bio) { }
+
+static inline int fscrypt_evict_crypt_key(struct inode *inode)
+{
+	return 0;
+}
+#endif

fscrypt_evict_crypt_key() is only called by fs/crypto/ itself, so why is it being exported to filesystems?

include/uapi/linux/fs.h struct fsxattr { · line 227

Quoted 10 lines
include/uapi/linux/fs.h +1 -1
index 59c71fa8c553..dea16d0f9d2e 100644
@@ -224,7 +224,8 @@ struct fsxattr {
 #define FS_POLICY_FLAGS_PAD_32		0x03
 #define FS_POLICY_FLAGS_PAD_MASK	0x03
 #define FS_POLICY_FLAG_DIRECT_KEY	0x04	/* use master key directly */
-#define FS_POLICY_FLAGS_VALID		0x07
+#define FS_POLICY_FLAGS_INLINE_CRYPT_OPTIMIZED	0x08

Should be "FLAG" instead of "FLAGS" since it's a single flag, not a mask or multi-bit field. See FS_POLICY_FLAG_DIRECT_KEY.

- Eric

#15 lore raw

PATCH v4 8/8 f2fs: Wire up f2fs to use inline encryption via fscrypt

In reply to Satya Tangirala
---
 fs/f2fs/data.c  | 127 ++++++++++++++++++++++++++++++++++++++++++++----
 fs/f2fs/super.c |  15 +++---
 2 files changed, 126 insertions(+), 16 deletions(-)
fs/f2fs/data.c +117 -10
index abbf14e9bd72..a7294edce173 100644
@@ -143,6 +143,8 @@ static bool f2fs_bio_post_read_required(struct bio *bio)
 
 static void f2fs_read_end_io(struct bio *bio)
 {
+	fscrypt_unset_bio_crypt_ctx(bio);
+
 	if (time_to_inject(F2FS_P_SB(bio_first_page_all(bio)),
 						FAULT_READ_IO)) {
 		f2fs_show_injection_info(FAULT_READ_IO);
@@ -166,6 +168,8 @@ static void f2fs_write_end_io(struct bio *bio)
 	struct bio_vec *bvec;
 	struct bvec_iter_all iter_all;
 
+	fscrypt_unset_bio_crypt_ctx(bio);
+
 	if (time_to_inject(sbi, FAULT_WRITE_IO)) {
 		f2fs_show_injection_info(FAULT_WRITE_IO);
 		bio->bi_status = BLK_STS_IOERR;
@@ -283,6 +287,53 @@ static struct bio *__bio_alloc(struct f2fs_sb_info *sbi, block_t blk_addr,
 	return bio;
 }
 
+static int f2fs_init_bio_crypt_ctx(struct bio *bio, struct inode *inode,
+				   struct f2fs_io_info *fio, bool no_fail)
+{
+	gfp_t gfp_mask = GFP_NOIO;
+	/*
+	 * This function should be called with NULL inode iff the pages
+	 * being added to the bio are to be handled without inline
+	 * en/decryption.
+	 */
+	if (!inode)
+		return 0;
+
+	/*
+	 * If the fio has an encrypted page, that means we want to read/write
+	 * it without inline encryption (for e.g. when moving blocks)
+	 */
+	if (fio && fio->encrypted_page)
+		return 0;
+
+	if (no_fail)
+		gfp_mask |= __GFP_NOFAIL;
+
+	return fscrypt_set_bio_crypt_ctx(bio, inode, 0, gfp_mask);
+}
+
+static inline u64 inline_crypt_dun(struct inode *inode, pgoff_t offset)
+{
+	return (((u64)inode->i_ino) << 32) | lower_32_bits(offset);
+}
+
+bool f2fs_page_crypt_back_mergeable(const struct bio *bio,
+				    const struct page *page)
+{
+	struct inode *bio_inode = bio && bio_page(bio)->mapping ?
+				    bio_page(bio)->mapping->host : NULL;
+	struct inode *page_inode = page && page->mapping ? page->mapping->host
+							 : NULL;
+
+	if (!fscrypt_inode_crypt_mergeable(page_inode, bio_inode))
+		return false;
+	if (!bio_inode || !fscrypt_inode_is_inline_crypted(bio_inode))
+		return true;
+	return inline_crypt_dun(bio_inode, bio_page(bio)->index) +
+	       (bio->bi_iter.bi_size >> PAGE_SHIFT) ==
+	       inline_crypt_dun(page_inode, page->index);
+}
+
 static inline void __submit_bio(struct f2fs_sb_info *sbi,
 				struct bio *bio, enum page_type type)
 {
@@ -327,6 +378,14 @@ static inline void __submit_bio(struct f2fs_sb_info *sbi,
 		trace_f2fs_submit_read_bio(sbi->sb, type, bio);
 	else
 		trace_f2fs_submit_write_bio(sbi->sb, type, bio);
+
+	if (bio_has_data(bio) && bio_has_crypt_ctx(bio)) {
+		struct page *page = bio_page(bio);
+
+		bio_set_data_unit_num(bio, inline_crypt_dun(page->mapping->host,
+							    page->index));
+	}
+
 	submit_bio(bio);
 }
 
@@ -451,6 +510,7 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio)
 	struct bio *bio;
 	struct page *page = fio->encrypted_page ?
 			fio->encrypted_page : fio->page;
+	int err;
 
 	if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr,
 			fio->is_por ? META_POR : (__is_meta_io(fio) ?
@@ -464,9 +524,15 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio)
 	bio = __bio_alloc(fio->sbi, fio->new_blkaddr, fio->io_wbc,
 				1, is_read_io(fio->op), fio->type, fio->temp);
 
-	if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) {
+	err = f2fs_init_bio_crypt_ctx(bio, fio->page->mapping->host, fio,
+				      false);
+
+	if (!err && bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE)
+		err = -EFAULT;
+
+	if (err) {
 		bio_put(bio);
-		return -EFAULT;
+		return err;
 	}
 
 	if (fio->io_wbc && !is_read_io(fio->op))
@@ -486,6 +552,7 @@ int f2fs_merge_page_bio(struct f2fs_io_info *fio)
 	struct bio *bio = *fio->bio;
 	struct page *page = fio->encrypted_page ?
 			fio->encrypted_page : fio->page;
+	int err;
 
 	if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr,
 			__is_meta_io(fio) ? META_GENERIC : DATA_GENERIC))
@@ -495,7 +562,8 @@ int f2fs_merge_page_bio(struct f2fs_io_info *fio)
 	f2fs_trace_ios(fio, 0);
 
 	if (bio && (*fio->last_block + 1 != fio->new_blkaddr ||
-			!__same_bdev(fio->sbi, fio->new_blkaddr, bio))) {
+			!__same_bdev(fio->sbi, fio->new_blkaddr, bio) ||
+			!f2fs_page_crypt_back_mergeable(bio, page))) {
 		__submit_bio(fio->sbi, bio, fio->type);
 		bio = NULL;
 	}
@@ -503,6 +571,12 @@ int f2fs_merge_page_bio(struct f2fs_io_info *fio)
 	if (!bio) {
 		bio = __bio_alloc(fio->sbi, fio->new_blkaddr, fio->io_wbc,
 				BIO_MAX_PAGES, false, fio->type, fio->temp);
+		err = f2fs_init_bio_crypt_ctx(bio, fio->page->mapping->host,
+					      fio, false);
+		if (err) {
+			bio_put(bio);
+			return err;
+		}
 		bio_set_op_attrs(bio, fio->op, fio->op_flags);
 	}
 
@@ -570,8 +644,10 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)
 
 	if (io->bio && (io->last_block_in_bio != fio->new_blkaddr - 1 ||
 	    (io->fio.op != fio->op || io->fio.op_flags != fio->op_flags) ||
-			!__same_bdev(sbi, fio->new_blkaddr, io->bio)))
+			!__same_bdev(sbi, fio->new_blkaddr, io->bio) ||
+			!f2fs_page_crypt_back_mergeable(io->bio, bio_page)))
 		__submit_merged_bio(io);
+
 alloc_new:
 	if (io->bio == NULL) {
 		if ((fio->type == DATA || fio->type == NODE) &&
@@ -583,6 +659,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)
 		io->bio = __bio_alloc(sbi, fio->new_blkaddr, fio->io_wbc,
 						BIO_MAX_PAGES, false,
 						fio->type, fio->temp);
+		f2fs_init_bio_crypt_ctx(io->bio, fio->page->mapping->host,
+					fio, true);
 		io->fio = *fio;
 	}
 
@@ -615,16 +693,24 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr,
 	struct bio *bio;
 	struct bio_post_read_ctx *ctx;
 	unsigned int post_read_steps = 0;
+	int err;
 
 	bio = f2fs_bio_alloc(sbi, min_t(int, nr_pages, BIO_MAX_PAGES), false);
 	if (!bio)
 		return ERR_PTR(-ENOMEM);
+	err = f2fs_init_bio_crypt_ctx(bio, inode, NULL, false);
+	if (err) {
+		bio_put(bio);
+		return ERR_PTR(err);
+	}
+
 	f2fs_target_device(sbi, blkaddr, bio);
 	bio->bi_end_io = f2fs_read_end_io;
 	bio_set_op_attrs(bio, REQ_OP_READ, op_flag);
 
-	if (f2fs_encrypted_file(inode))
+	if (fscrypt_needs_fs_layer_crypto(inode))
 		post_read_steps |= 1 << STEP_DECRYPT;
+
 	if (post_read_steps) {
 		ctx = mempool_alloc(bio_post_read_ctx_pool, GFP_NOFS);
 		if (!ctx) {
@@ -1574,6 +1660,7 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 					struct f2fs_map_blocks *map,
 					struct bio **bio_ret,
 					sector_t *last_block_in_bio,
+					u64 *next_dun,
 					bool is_readahead)
 {
 	struct bio *bio = *bio_ret;
@@ -1648,6 +1735,13 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 		__submit_bio(F2FS_I_SB(inode), bio, DATA);
 		bio = NULL;
 	}
+
+	if (bio && fscrypt_inode_is_inline_crypted(inode) &&
+	    *next_dun != inline_crypt_dun(inode, page->index)) {
+		__submit_bio(F2FS_I_SB(inode), bio, DATA);
+		bio = NULL;
+	}
+
 	if (bio == NULL) {
 		bio = f2fs_grab_read_bio(inode, block_nr, nr_pages,
 				is_readahead ? REQ_RAHEAD : 0);
@@ -1667,6 +1761,9 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 	if (bio_add_page(bio, page, blocksize, 0) < blocksize)
 		goto submit_and_realloc;
 
+	if (fscrypt_inode_is_inline_crypted(inode))
+		*next_dun = inline_crypt_dun(inode, page->index) + 1;
+
 	inc_page_count(F2FS_I_SB(inode), F2FS_RD_DATA);
 	ClearPageError(page);
 	*last_block_in_bio = block_nr;
@@ -1700,6 +1797,7 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
 	struct inode *inode = mapping->host;
 	struct f2fs_map_blocks map;
 	int ret = 0;
+	u64 next_dun = 0;
 
 	map.m_pblk = 0;
 	map.m_lblk = 0;
@@ -1723,7 +1821,8 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
 		}
 
 		ret = f2fs_read_single_page(inode, page, nr_pages, &map, &bio,
-					&last_block_in_bio, is_readahead);
+					&last_block_in_bio, &next_dun,
+					is_readahead);
 		if (ret) {
 			SetPageError(page);
 			zero_user_segment(page, 0, PAGE_SIZE);
@@ -1777,12 +1876,12 @@ static int encrypt_one_page(struct f2fs_io_info *fio)
 	struct page *mpage;
 	gfp_t gfp_flags = GFP_NOFS;
 
-	if (!f2fs_encrypted_file(inode))
-		return 0;
-
 	/* wait for GCed page writeback via META_MAPPING */
 	f2fs_wait_on_block_writeback(inode, fio->old_blkaddr);
 
+	if (!fscrypt_needs_fs_layer_crypto(inode))
+		return 0;
+
 retry_encrypt:
 	fio->encrypted_page = fscrypt_encrypt_pagecache_blocks(fio->page,
 							       PAGE_SIZE, 0,
@@ -1957,7 +2056,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 			f2fs_unlock_op(fio->sbi);
 		err = f2fs_inplace_write_data(fio);
 		if (err) {
-			if (f2fs_encrypted_file(inode))
+			if (fscrypt_needs_fs_layer_crypto(inode))
 				fscrypt_finalize_bounce_page(&fio->encrypted_page);
 			if (PageWriteback(page))
 				end_page_writeback(page);
@@ -2692,6 +2791,8 @@ static void f2fs_dio_end_io(struct bio *bio)
 {
 	struct f2fs_private_dio *dio = bio->bi_private;
 
+	fscrypt_unset_bio_crypt_ctx(bio);
+
 	dec_page_count(F2FS_I_SB(dio->inode),
 			dio->write ? F2FS_DIO_WRITE : F2FS_DIO_READ);
 
@@ -2708,12 +2809,18 @@ static void f2fs_dio_submit_bio(struct bio *bio, struct inode *inode,
 {
 	struct f2fs_private_dio *dio;
 	bool write = (bio_op(bio) == REQ_OP_WRITE);
+	u64 data_unit_num = inline_crypt_dun(inode, file_offset >> PAGE_SHIFT);
 
 	dio = f2fs_kzalloc(F2FS_I_SB(inode),
 			sizeof(struct f2fs_private_dio), GFP_NOFS);
 	if (!dio)
 		goto out;
 
+	if (fscrypt_set_bio_crypt_ctx(bio, inode, data_unit_num, GFP_NOIO)) {
+		kvfree(dio);
+		goto out;
+	}
+
 	dio->inode = inode;
 	dio->orig_end_io = bio->bi_end_io;
 	dio->orig_private = bio->bi_private;
fs/f2fs/super.c +9 -6
index 78a1b873e48a..196145cb42f9 100644
@@ -2223,12 +2223,15 @@ static bool f2fs_dummy_context(struct inode *inode)
 }
 
 static const struct fscrypt_operations f2fs_cryptops = {
-	.key_prefix	= "f2fs:",
-	.get_context	= f2fs_get_context,
-	.set_context	= f2fs_set_context,
-	.dummy_context	= f2fs_dummy_context,
-	.empty_dir	= f2fs_empty_dir,
-	.max_namelen	= F2FS_NAME_LEN,
+	.key_prefix		= "f2fs:",
+	.get_context		= f2fs_get_context,
+	.set_context		= f2fs_set_context,
+	.dummy_context		= f2fs_dummy_context,
+	.empty_dir		= f2fs_empty_dir,
+	.max_namelen		= F2FS_NAME_LEN,
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+	.inline_crypt_supp	= true,
+#endif
 };
 #endif
 
Signature · 1 line
2.23.0.rc1.153.gdeed80330f-goog