Inline Encryption Support
Subsystems (7)
v2 cover Inline Encryption Support
This patch series adds support for Inline Encryption to the block layer, UFS, fscrypt and f2fs.
Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device. Inline Encryption hardware has become increasingly common, and we want to support it in the kernel.
Inline Encryption hardware implementations often function around the concept of a limited number of "keyslots", which can hold an encryption context each. The storage device can be directed to en/decrypt any particular request with the encryption context stored in any particular keyslot.
Patch 1 introduces a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that blk-crypto (introduced in Path 3), will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.
Patch 2 introduces struct bio_crypt_ctx, and a ptr to one in struct bio, which allows struct bio to represent an encryption context that can be passed down the storage stack from the filesystem layer to the storage driver.
Patch 3 introduces blk-crypto. Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware. Given that blk-crypto works as a software fallback, we are considering removing file content en/decryption from fscrypt and simply using blk-crypto in a future patch. For more details on blk-crypto, refer to Documentation/block/blk-crypto.txt.
Patches 4-6 add support for inline encryption into the UFS driver according to the JEDEC UFS HCI v2.1 specification. Inline encryption support for other drivers (like eMMC) may be added in the same way - the device driver should set up a Keyslot Manager in the device's request_queue (refer to the UFS crypto additions in ufshcd-crypto.c and ufshcd.c for an example).
Patches 7 and 8 add support to fscrypt and f2fs, so that we have a complete stack that can make use of inline encryption.
There have been a few patch sets addressing Inline Encryption Support in the past. Briefly, this patch set differs from those as follows:
1) "crypto: qce: ice: Add support for Inline Crypto Engine" is specific to certain hardware, while our patch set's Inline Encryption support for UFS is implemented according to the JEDEC UFS specification.
2) "scsi: ufs: UFS Host Controller crypto changes" registers inline encryption support as a kernel crypto algorithm. Our patch views inline encryption as being fundamentally different from a generic crypto provider (in that inline encryption is tied to a device), and so does not use the kernel crypto API to represent inline encryption hardware.
3) "scsi: ufs: add real time/inline crypto support to UFS HCD" requires the device mapper to work - our patch does not.
Changes v1 => v2:
- Block layer and UFS changes are split into 3 patches each.
- We now only have a ptr to a struct bio_crypt_ctx in struct bio, instead of the struct itself.
- struct bio_crypt_ctx no longer has flags.
- blk-crypto now correctly handles the case when it fails to init (because of insufficient memory), but kernel continues to boot.
- ufshcd-crypto now works on big endian cpus.
- Many cleanups.
Satya Tangirala (8):
block: Keyslot Manager for Inline Encryption
block: Add encryption context to struct bio
block: blk-crypto for Inline Encryption
scsi: ufs: UFS driver v2.1 spec crypto additions
scsi: ufs: UFS crypto API
scsi: ufs: Add inline encryption support to UFS
fscrypt: wire up fscrypt to use blk-crypto
f2fs: Wire up f2fs to use inline encryption via fscrypt
Documentation/block/blk-crypto.txt | 185 ++++++++++ block/Kconfig | 8 + block/Makefile | 2 + block/bio.c | 17 +- block/blk-core.c | 11 +- block/blk-crypt-ctx.c | 90 +++++ block/blk-crypto.c | 557 +++++++++++++++++++++++++++++ block/blk-merge.c | 34 +- block/bounce.c | 9 +- block/keyslot-manager.c | 315 ++++++++++++++++ drivers/md/dm.c | 15 +- drivers/scsi/ufs/Kconfig | 10 + drivers/scsi/ufs/Makefile | 1 + drivers/scsi/ufs/ufshcd-crypto.c | 438 +++++++++++++++++++++++ drivers/scsi/ufs/ufshcd-crypto.h | 69 ++++ drivers/scsi/ufs/ufshcd.c | 84 ++++- drivers/scsi/ufs/ufshcd.h | 23 ++ drivers/scsi/ufs/ufshci.h | 67 +++- fs/crypto/Kconfig | 7 + fs/crypto/bio.c | 159 ++++++-- fs/crypto/crypto.c | 9 + fs/crypto/fscrypt_private.h | 10 + fs/crypto/keyinfo.c | 69 ++-- fs/crypto/policy.c | 10 + fs/f2fs/data.c | 77 +++- fs/f2fs/super.c | 1 + include/linux/bio.h | 180 ++++++++++ include/linux/blk-crypto.h | 40 +++ include/linux/blk_types.h | 39 ++ include/linux/blkdev.h | 9 + include/linux/fscrypt.h | 64 ++++ include/linux/keyslot-manager.h | 116 ++++++ include/uapi/linux/fs.h | 12 +- 33 files changed, 2668 insertions(+), 69 deletions(-) create mode 100644 Documentation/block/blk-crypto.txt create mode 100644 block/blk-crypt-ctx.c create mode 100644 block/blk-crypto.c create mode 100644 block/keyslot-manager.c create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h create mode 100644 include/linux/blk-crypto.h create mode 100644 include/linux/keyslot-manager.h
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
RFC PATCH v2 1/8 block: Keyslot Manager for Inline Encryption
Inline Encryption hardware allows software to specify an encryption context (an encryption key, crypto algorithm, data unit num, data unit size, etc.) along with a data transfer request to a storage device, and the inline encryption hardware will use that context to en/decrypt the data. The inline encryption hardware is part of the storage device, and it conceptually sits on the data path between system memory and the storage device.
Inline Encryption hardware implementations often function around the concept of "keyslots". These implementations often have a limited number of "keyslots", each of which can hold an encryption context (we say that an encryption context can be "programmed" into a keyslot). Requests made to the storage device may have a keyslot associated with them, and the inline encryption hardware will en/decrypt the data in the requests using the encryption context programmed into that associated keyslot. As keyslots are limited, and programming keys may be expensive in many implementations, and multiple requests may use exactly the same encryption contexts, we introduce a Keyslot Manager to efficiently manage keyslots. The keyslot manager also functions as the interface that upper layers will use to program keys into inline encryption hardware. For more information on the Keyslot Manager, refer to documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.
Known issues:
1) Keyslot Manager has a performance bug where the same encryption context may be programmed into multiple keyslots at the same time in certain situations when all keyslots are being used.
--- block/keyslot-manager.c | 315 ++++++++++++++++++++++++++++++++ include/linux/blk_types.h | 11 ++ include/linux/blkdev.h | 9 + include/linux/keyslot-manager.h | 116 ++++++++++++ 4 files changed, 451 insertions(+) create mode 100644 block/keyslot-manager.c create mode 100644 include/linux/keyslot-manager.h
block/keyslot-manager.c new file +315 -0
@@ -0,0 +1,315 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* keyslot-manager.c +* +* Copyright 2019 Google LLC +*/ + +/** +* DOC: The Keyslot Manager +* +* Many devices with inline encryption support have a limited number of "slots" +* into which encryption contexts may be programmed, and requests can be tagged +* with a slot number to specify the key to use for en/decryption. +* +* As the number of slots are limited, and programming keys is expensive on +* many inline encryption hardware, we don't want to program the same key into +* multiple slots - if multiple requests are using the same key, we want to +* program just one slot with that key and use that slot for all requests. +* +* The keyslot manager manages these keyslots appropriately, and also acts as +* an abstraction between the inline encryption hardware and the upper layers. +* +* Lower layer devices will set up a keyslot manager in their request queue +* and tell it how to perform device specific operations like programming/ +* evicting keys from keyslots. +* +* Upper layers will call keyslot_manager_get_slot_for_key() to program a +* key into some slot in the inline encryption hardware. +*/ +#include <linux/slab.h> +#include <linux/keyslot-manager.h> +#include <linux/atomic.h> +#include <linux/mutex.h> +#include <linux/sched.h> +#include <linux/wait.h> + +struct keyslot_manager { +unsigned int num_slots; +atomic_t num_idle_slots; +struct keyslot_mgmt_ll_ops ksm_ll_ops; +void *ll_priv_data; +struct mutex lock; +wait_queue_head_t wait_queue; +u64 seq_num; +u64 *last_used_seq_nums; +atomic_t slot_refs[]; +}; + +/** +* keyslot_manager_create() - Create a keyslot manager +* @num_slots: The number of key slots to manage. +* @ksm_ll_ops: The struct keyslot_mgmt_ll_ops for the device that this keyslot +* manager will use to perform operations like programming and +* evicting keys. +* @ll_priv_data: Private data passed as is to the functions in ksm_ll_ops. +* +* Allocate memory for and initialize a keyslot manager. Called by for e.g. +* storage drivers to set up a keyslot manager in their request_queue. +* +* Context: May sleep +* Return: Pointer to constructed keyslot manager or NULL on error. +*/ +struct keyslot_manager *keyslot_manager_create (unsigned int num_slots, +const struct keyslot_mgmt_ll_ops *ksm_ll_ops, +void *ll_priv_data) +{ +struct keyslot_manager *ksm; + +if (num_slots ==0 ) +return NULL ; + +/* Check that all ops are specified */ +if (ksm_ll_ops->keyslot_program ==NULL || + ksm_ll_ops->keyslot_evict ==NULL || + ksm_ll_ops->crypt_mode_supported ==NULL || + ksm_ll_ops->keyslot_find ==NULL ) +return NULL ; + + ksm =kzalloc (struct_size (ksm, slot_refs, num_slots),GFP_KERNEL ); +if (!ksm) +return NULL ; + + ksm->num_slots = num_slots; +atomic_set (&ksm->num_idle_slots, num_slots); + ksm->ksm_ll_ops = *ksm_ll_ops; + ksm->ll_priv_data = ll_priv_data; + +mutex_init (&ksm->lock); +init_waitqueue_head (&ksm->wait_queue); + + ksm->last_used_seq_nums =kcalloc (num_slots,sizeof (u64),GFP_KERNEL ); +if (!ksm->last_used_seq_nums) { +kzfree (ksm); + ksm =NULL ; + } + +return ksm; +} +EXPORT_SYMBOL (keyslot_manager_create); + +/** +* keyslot_manager_get_slot_for_key() - Program a key into a keyslot. +* @ksm: The keyslot manager to program the key into. +* @key: Pointer to the bytes of the key to program. Must be the correct length +* for the chosen @crypt_mode; see blk_crypt_modes in blk-crypto.c. +* @crypt_mode: Identifier for the encryption algorithm to use. +* @data_unit_size: The data unit size to use for en/decryption. +* +* Get a keyslot that's been programmed with the specified key, crypt_mode, and +* data_unit_size. If one already exists, return it with incremented refcount. +* Otherwise, wait for a keyslot to become idle and program it. +* +* Context: Process context. Takes and releases ksm->lock. +* Return: The keyslot on success, else a -errno value. +*/ +int keyslot_manager_get_slot_for_key (struct keyslot_manager *ksm, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{ +int slot; +int err; +int c; +int lru_idle_slot; +u64 min_seq_num; + +mutex_lock (&ksm->lock); + slot = ksm->ksm_ll_ops.keyslot_find (ksm->ll_priv_data, key, + crypt_mode, + data_unit_size); + +if (slot <0 && slot != -ENOKEY ) { +mutex_unlock (&ksm->lock); +return slot; + } + +if (WARN_ON (slot >= (int )ksm->num_slots)) { +mutex_unlock (&ksm->lock); +return -EINVAL ; + } + +/* Try to use the returned slot */ +if (slot != -ENOKEY ) { +/* +* NOTE: We may fail to get a slot if the number of refs +* overflows UINT_MAX. I don't think we care enough about +* that possibility to make the refcounts u64, considering +* the only way for that to happen is for at least UINT_MAX +* requests to be in flight at the same time. +*/ +if ((unsigned int )atomic_read (&ksm->slot_refs[slot]) == +UINT_MAX ) { +mutex_unlock (&ksm->lock); +return -EBUSY ; + } + +if (atomic_fetch_inc (&ksm->slot_refs[slot]) ==0 ) +atomic_dec (&ksm->num_idle_slots); + + ksm->last_used_seq_nums[slot] = ++ksm->seq_num; + +mutex_unlock (&ksm->lock); +return slot; + } + +/* +* If we're here, that means there wasn't a slot that +* was already programmed with the key +*/ + +/* Wait till there is a free slot available */ +while (atomic_read (&ksm->num_idle_slots) ==0 ) { +mutex_unlock (&ksm->lock); +wait_event (ksm->wait_queue, + (atomic_read (&ksm->num_idle_slots) >0 )); +mutex_lock (&ksm->lock); + } + +/* Todo: fix linear scan? */ +/* Find least recently used idle slot (i.e. slot with minimum number) */ + lru_idle_slot =-1 ; + min_seq_num =0 ; +for (c =0 ; c < ksm->num_slots; c++) { +if (atomic_read (&ksm->slot_refs[c]) !=0 ) +continue ; + +if (lru_idle_slot ==-1 || + ksm->last_used_seq_nums[c] < min_seq_num) { + lru_idle_slot = c; + min_seq_num = ksm->last_used_seq_nums[c]; + } + } + +if (WARN_ON (lru_idle_slot ==-1 )) { +mutex_unlock (&ksm->lock); +return -EBUSY ; + } + +atomic_dec (&ksm->num_idle_slots); +atomic_inc (&ksm->slot_refs[lru_idle_slot]); + err = ksm->ksm_ll_ops.keyslot_program (ksm->ll_priv_data, key, + crypt_mode, + data_unit_size, + lru_idle_slot); + +if (err) { +atomic_dec (&ksm->slot_refs[lru_idle_slot]); +atomic_inc (&ksm->num_idle_slots); +wake_up (&ksm->wait_queue); +mutex_unlock (&ksm->lock); +return err; + } + + ksm->seq_num++; + ksm->last_used_seq_nums[lru_idle_slot] = ksm->seq_num; + +mutex_unlock (&ksm->lock); +return lru_idle_slot; +} +EXPORT_SYMBOL (keyslot_manager_get_slot_for_key); + +/** +* keyslot_manager_get_slot() - Increment the refcount on the specified slot. +* @ksm - The keyslot manager that we want to modify. +* @slot - The slot to increment the refcount of. +* +* This function assumes that there is already an active reference to that slot +* and simply increments the refcount. This is useful when cloning a bio that +* already has a reference to a keyslot, and we want the cloned bio to also have +* its own reference. +* +* Context: Any context. +*/ +void keyslot_manager_get_slot (struct keyslot_manager *ksm,unsigned int slot) +{ +if (WARN_ON (slot >= ksm->num_slots)) +return ; + +WARN_ON (atomic_inc_return (&ksm->slot_refs[slot]) <2 ); +} +EXPORT_SYMBOL (keyslot_manager_get_slot); + +/** +* keyslot_manager_put_slot() - Release a reference to a slot +* @ksm: The keyslot manager to release the reference from. +* @slot: The slot to release the reference from. +* +* Context: Any context. +*/ +void keyslot_manager_put_slot (struct keyslot_manager *ksm,unsigned int slot) +{ +if (WARN_ON (slot >= ksm->num_slots)) +return ; + +if (atomic_dec_and_test (&ksm->slot_refs[slot])) { +atomic_inc (&ksm->num_idle_slots); +wake_up (&ksm->wait_queue); + } +} +EXPORT_SYMBOL (keyslot_manager_put_slot); + +/** +* keyslot_manager_evict_key() - Evict a key from the lower layer device. +* @ksm - The keyslot manager to evict from +* @key - The key to evict +* @crypt_mode - The crypto algorithm the key was programmed with. +* @data_unit_size - The data_unit_size the key was programmed with. +* +* Finds the slot that the specified key, crypt_mode, data_unit_size combo +* was programmed into, and evicts that slot from the lower layer device if +* the refcount on the slot is 0. Returns -EBUSY if the refcount is not 0, and +* -errno on error. +* +* Context: Process context. Takes and releases ksm->lock. +*/ +int keyslot_manager_evict_key (struct keyslot_manager *ksm, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{ +int slot; +int err =0 ; + +mutex_lock (&ksm->lock); + slot = ksm->ksm_ll_ops.keyslot_find (ksm->ll_priv_data, key, + crypt_mode, + data_unit_size); + +if (slot <0 ) { +mutex_unlock (&ksm->lock); +return slot; + } + +if (atomic_read (&ksm->slot_refs[slot]) ==0 ) { + err = ksm->ksm_ll_ops.keyslot_evict (ksm->ll_priv_data, key, + crypt_mode, + data_unit_size, + slot); + }else { + err = -EBUSY ; + } + +mutex_unlock (&ksm->lock); +return err; +} +EXPORT_SYMBOL (keyslot_manager_evict_key); + +void keyslot_manager_destroy (struct keyslot_manager *ksm) +{ +if (!ksm) +return ; +kzfree (ksm->last_used_seq_nums); +kzfree (ksm); +} +EXPORT_SYMBOL (keyslot_manager_destroy);
include/linux/blk_types.h +11 -0
@@ -137,6 +137,17 @@ static inline void bio_issue_init(struct bio_issue *issue, ((u64)size << BIO_ISSUE_SIZE_SHIFT)); } +enum blk_crypt_mode_num { +BLK_ENCRYPTION_MODE_AES_256_XTS =0 , +/* +* TODO: Support these too +* BLK_ENCRYPTION_MODE_AES_256_CTS = 1, +* BLK_ENCRYPTION_MODE_AES_128_CBC = 2, +* BLK_ENCRYPTION_MODE_AES_128_CTS = 3, +* BLK_ENCRYPTION_MODE_ADIANTUM = 4, +*/ +}; +/* * main unit of I/O for the block layer and lower layers (ie drivers and * stacking drivers)
include/linux/blkdev.h +9 -0
@@ -385,6 +385,10 @@ static inline int blkdev_reset_zones_ioctl(struct block_device *bdev,#endif /* CONFIG_BLK_DEV_ZONED */ +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +struct keyslot_manager ; +#endif +struct request_queue {/* * Together with queue_head for cacheline sharing
@@ -473,6 +477,11 @@ struct request_queue {unsigned int dma_pad_mask;unsigned int dma_alignment; +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +/* Inline crypto capabilities */ +struct keyslot_manager *ksm; +#endif +unsigned int rq_timeout;int poll_nsec;
include/linux/keyslot-manager.h new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Eric Biggers
maintainerRe: [RFC PATCH v2 1/8] block: Keyslot Manager for Inline Encryption
Hi Satya,
Quoted 5 lines · Satya Tangirala
Known issues:
1) Keyslot Manager has a performance bug where the same encryption context may be programmed into multiple keyslots at the same time in certain situations when all keyslots are being used.
This is also a correctness bug, since keyslot_manager_evict_key() only evicts one copy of the key. It can be fixed by looking for the key again after waiting for an idle slot.
Review
Quoted 12 lines
(quoted diff) +12
-0+ +struct keyslot_manager { + unsigned int num_slots; + atomic_t num_idle_slots; + struct keyslot_mgmt_ll_ops ksm_ll_ops; + void *ll_priv_data; + struct mutex lock; + wait_queue_head_t wait_queue; + u64 seq_num; + u64 *last_used_seq_nums; + atomic_t slot_refs[]; +};
slot_refs[] and last_used_seq_nums[] both contain per-keyslot data. It would be cleaner to combine them into a single 'slots' array of 'struct keyslot'. That would also make it much easier to add or change per-keyslot data in the future.
Quoted 17 lines
/**
*keyslot_manager_create()- Create a keyslot manager
* @num_slots: The number of key slots to manage.
* @ksm_ll_ops: The structkeyslot_mgmt_ll_opsfor the device that this keyslot
* manager will use to perform operations like programming and
* evicting keys.
* @ll_priv_data: Private data passed as is to the functions inksm_ll_ops. *
* Allocate memory for and initialize a keyslot manager. Called by for e.g.
* storage drivers to set up a keyslot manager in theirrequest_queue. *
* Context: May sleep
* Return: Pointer to constructed keyslot manager or NULL on error. */
structkeyslot_manager*keyslot_manager_create(unsigned intnum_slots,
const structkeyslot_mgmt_ll_ops*ksm_ll_ops, void*ll_priv_data)
Quoted 16 lines
{
struct keyslot_manager *ksm;
if (num_slots == 0)
return NULL;
/* Check that all ops are specified */
if (ksm_ll_ops->keyslot_program == NULL ||
ksm_ll_ops->keyslot_evict == NULL ||
ksm_ll_ops->crypt_mode_supported == NULL ||
ksm_ll_ops->keyslot_find == NULL)
return NULL;
ksm = kzalloc(struct_size(ksm, slot_refs, num_slots), GFP_KERNEL);
if (!ksm)
return NULL;This should probably be kvzalloc(), just in case the number of keyslots is too large to fit comfortably into kmalloc memory. We don't need physically contiguous memory here.
Quoted 25 lines
(quoted diff) +25
-0+/** + * keyslot_manager_get_slot_for_key() - Program a key into a keyslot. + * @ksm: The keyslot manager to program the key into. + * @key: Pointer to the bytes of the key to program. Must be the correct length + * for the chosen @crypt_mode; see blk_crypt_modes in blk-crypto.c. + * @crypt_mode: Identifier for the encryption algorithm to use. + * @data_unit_size: The data unit size to use for en/decryption. + * + * Get a keyslot that's been programmed with the specified key, crypt_mode, and + * data_unit_size. If one already exists, return it with incremented refcount. + * Otherwise, wait for a keyslot to become idle and program it. + * + * Context: Process context. Takes and releases ksm->lock. + * Return: The keyslot on success, else a -errno value. + */ +int keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm, + const u8 *key, + enum blk_crypt_mode_num crypt_mode, + unsigned int data_unit_size) +{ + int slot; + int err; + int c; + int lru_idle_slot; + u64 min_seq_num;
Review
Quoted 10 lines
(quoted diff) +10
-0+ + mutex_lock(&ksm->lock); + slot = ksm->ksm_ll_ops.keyslot_find(ksm->ll_priv_data, key, + crypt_mode, + data_unit_size); + + if (slot < 0 && slot != -ENOKEY) { + mutex_unlock(&ksm->lock); + return slot; + }
This is the fast path: taking a reference to an existing key slot. There could be many processes issuing I/O concurrently, so I'm worried that the per-device mutex here will be a bottleneck. How about using a rw_semaphore instead? ->keyslot_find() would be called with (at least) the read lock, while ->keyslot_program() and ->keyslot_evict() would be called with the write lock.
Quoted 4 lines
(quoted diff) +4
-0+ /* Todo: fix linear scan? */ + /* Find least recently used idle slot (i.e. slot with minimum number) */ + lru_idle_slot = -1; + min_seq_num = 0;
Review
Quoted 10 lines
(quoted diff) +10
-0+ for (c = 0; c < ksm->num_slots; c++) { + if (atomic_read(&ksm->slot_refs[c]) != 0) + continue; + + if (lru_idle_slot == -1 || + ksm->last_used_seq_nums[c] < min_seq_num) { + lru_idle_slot = c; + min_seq_num = ksm->last_used_seq_nums[c]; + } + }
How about using a real LRU list instead? I.e., a linked list containing all keyslots with refs == 0 in order of last use. Then you could just grab the head of the list here, which would be much more efficient than iterating through every keyslot as the code does now.
The current LRU implementation is also broken since it orders the entries by when they were last removed from the LRU list (i.e. last *started* to be used), not by when they were last added to the LRU list (i.e. last used).
To better show what I mean, here's an incremental patch (compile-tested only!) that implements these changes:
block/keyslot-manager.c +100 -117
@@ -34,16 +34,26 @@#include <linux/sched.h> #include <linux/wait.h> +struct keyslot { +atomic_t refs;/* Number of users of this keyslot */ +struct list_head lru;/* Link in idle_slots LRU list (if refs == 0) */ +}; +struct keyslot_manager {unsigned int num_slots; -atomic_t num_idle_slots;struct keyslot_mgmt_ll_ops ksm_ll_ops;void *ll_priv_data; -struct mutex lock; -wait_queue_head_t wait_queue; -u64 seq_num; -u64 *last_used_seq_nums; -atomic_t slot_refs[]; + +/* Protects programming and evicting keys from the device */ +struct rw_semaphore lock; + +/* List of slots with refs == 0, with least recently used at front */ +struct list_head idle_slots; +spinlock_t idle_slots_lock; +wait_queue_head_t idle_slots_wait_queue; + +/* Per-keyslot data */ +struct keyslot slots[]; };/**
@@ -65,6 +75,7 @@ struct keyslot_manager *keyslot_manager_create(unsigned int num_slots, void *ll_priv_data) {struct keyslot_manager *ksm; +int slot;if (num_slots ==0 )return NULL ;
@@ -76,28 +87,47 @@ struct keyslot_manager *keyslot_manager_create(unsigned int num_slots, ksm_ll_ops->keyslot_find == NULL)return NULL ; - ksm =kzalloc (struct_size (ksm, slot_refs, num_slots),GFP_KERNEL ); + ksm =kvzalloc (struct_size (ksm, slots, num_slots),GFP_KERNEL );if (!ksm)return NULL ; ksm->num_slots = num_slots; -atomic_set (&ksm->num_idle_slots, num_slots); ksm->ksm_ll_ops = *ksm_ll_ops; ksm->ll_priv_data = ll_priv_data; -mutex_init (&ksm->lock); -init_waitqueue_head (&ksm->wait_queue); +init_rwsem (&ksm->lock); - ksm->last_used_seq_nums =kcalloc (num_slots,sizeof (u64),GFP_KERNEL ); -if (!ksm->last_used_seq_nums) { -kzfree (ksm); - ksm =NULL ; - } +INIT_LIST_HEAD (&ksm->idle_slots); +spin_lock_init (&ksm->idle_slots_lock); +init_waitqueue_head (&ksm->idle_slots_wait_queue); +for (slot =0 ; slot < num_slots; slot++) +list_add_tail (&ksm->slots[slot].lru, &ksm->idle_slots);return ksm; }EXPORT_SYMBOL (keyslot_manager_create); +static int find_and_grab_keyslot (struct keyslot_manager *ksm,const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{ +int slot; + + slot = ksm->ksm_ll_ops.keyslot_find (ksm->ll_priv_data, key, + crypt_mode, data_unit_size); +if (slot <0 ) +return slot; +if (WARN_ON (slot >= ksm->num_slots)) +return -EINVAL ; +if (atomic_inc_return (&ksm->slots[slot].refs) ==1 ) { +/* Took first reference to this slot; remove it from LRU list */ +spin_lock (&ksm->idle_slots_lock); +list_del (&ksm->slots[slot].lru); +spin_unlock (&ksm->idle_slots_lock); + } +return slot; +} +/** * keyslot_manager_get_slot_for_key() - Program a key into a keyslot. * @ksm: The keyslot manager to program the key into.
@@ -110,7 +140,7 @@ EXPORT_SYMBOL(keyslot_manager_create);* data_unit_size. If one already exists, return it with incremented refcount. * Otherwise, wait for a keyslot to become idle and program it. * -* Context: Process context. Takes and releases ksm->lock. +* Context: Process context. * Return: The keyslot on success, else a -errno value. */ int keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm,
@@ -119,103 +149,60 @@ int keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm, unsigned int data_unit_size) {int slot; +struct keyslot *slotp;int err; -int c; -int lru_idle_slot; -u64 min_seq_num; - -mutex_lock (&ksm->lock); - slot = ksm->ksm_ll_ops.keyslot_find (ksm->ll_priv_data, key, - crypt_mode, - data_unit_size); - -if (slot <0 && slot != -ENOKEY ) { -mutex_unlock (&ksm->lock); -return slot; - } -if (WARN_ON (slot >= (int )ksm->num_slots)) { -mutex_unlock (&ksm->lock); -return -EINVAL ; - } - -/* Try to use the returned slot */ -if (slot != -ENOKEY ) { -/* -* NOTE: We may fail to get a slot if the number of refs -* overflows UINT_MAX. I don't think we care enough about -* that possibility to make the refcounts u64, considering -* the only way for that to happen is for at least UINT_MAX -* requests to be in flight at the same time. -*/ -if ((unsigned int )atomic_read (&ksm->slot_refs[slot]) == -UINT_MAX ) { -mutex_unlock (&ksm->lock); -return -EBUSY ; - } - -if (atomic_fetch_inc (&ksm->slot_refs[slot]) ==0 ) -atomic_dec (&ksm->num_idle_slots); - - ksm->last_used_seq_nums[slot] = ++ksm->seq_num; - -mutex_unlock (&ksm->lock); +/* +* Fast path: take reference to existing keyslot, if there is one. +* For this we only need the read lock. +*/ +down_read (&ksm->lock); + slot =find_and_grab_keyslot (ksm, key, crypt_mode, data_unit_size); +up_read (&ksm->lock); +if (slot != -ENOKEY )return slot; - }/* -* If we're here, that means there wasn't a slot that -* was already programmed with the key +* Slow path: wait for a slot to become idle, *or* for someone else to +* have programmed the key while we dropped the lock. */ +for (;;) { +down_write (&ksm->lock); + slot =find_and_grab_keyslot (ksm, key, crypt_mode, + data_unit_size); +if (slot != -ENOKEY ) { +up_write (&ksm->lock); +return slot; + } -/* Wait till there is a free slot available */ -while (atomic_read (&ksm->num_idle_slots) ==0 ) { -mutex_unlock (&ksm->lock); -wait_event (ksm->wait_queue, - (atomic_read (&ksm->num_idle_slots) >0 )); -mutex_lock (&ksm->lock); - } +spin_lock (&ksm->idle_slots_lock); +if (!list_empty (&ksm->idle_slots)) +break ; +spin_unlock (&ksm->idle_slots_lock); -/* Todo: fix linear scan? */ -/* Find least recently used idle slot (i.e. slot with minimum number) */ - lru_idle_slot =-1 ; - min_seq_num =0 ; -for (c =0 ; c < ksm->num_slots; c++) { -if (atomic_read (&ksm->slot_refs[c]) !=0 ) -continue ; - -if (lru_idle_slot ==-1 || - ksm->last_used_seq_nums[c] < min_seq_num) { - lru_idle_slot = c; - min_seq_num = ksm->last_used_seq_nums[c]; - } +up_write (&ksm->lock); +wait_event (ksm->idle_slots_wait_queue, + !list_empty (&ksm->idle_slots)); } -if (WARN_ON (lru_idle_slot ==-1 )) { -mutex_unlock (&ksm->lock); -return -EBUSY ; - } +/* Remove least recently used idle slot from LRU list. */ + slotp = list_first_entry(&ksm->idle_slots, struct keyslot, lru); +list_del (&slotp->lru); +atomic_set (&slotp->refs,1 ); +spin_unlock (&ksm->idle_slots_lock); + slot = slotp - ksm->slots; -atomic_dec (&ksm->num_idle_slots); -atomic_inc (&ksm->slot_refs[lru_idle_slot]); +/* Program the key into it. */ err = ksm->ksm_ll_ops.keyslot_program (ksm->ll_priv_data, key, - crypt_mode, - data_unit_size, - lru_idle_slot); + crypt_mode, data_unit_size, slot); +up_write (&ksm->lock);if (err) { -atomic_dec (&ksm->slot_refs[lru_idle_slot]); -atomic_inc (&ksm->num_idle_slots); -wake_up (&ksm->wait_queue); -mutex_unlock (&ksm->lock); +/* Oops, programming the key failed. Return slot to LRU list */ +keyslot_manager_put_slot (ksm, slot);return err; } - - ksm->seq_num++; - ksm->last_used_seq_nums[lru_idle_slot] = ksm->seq_num; - -mutex_unlock (&ksm->lock); -return lru_idle_slot; +return slot; }EXPORT_SYMBOL (keyslot_manager_get_slot_for_key);
@@ -236,7 +223,7 @@ void keyslot_manager_get_slot(struct keyslot_manager *ksm, unsigned int slot)if (WARN_ON (slot >= ksm->num_slots))return ; -WARN_ON (atomic_inc_return (&ksm->slot_refs[slot]) <2 ); +WARN_ON (atomic_inc_return (&ksm->slots[slot].refs) <2 ); }EXPORT_SYMBOL (keyslot_manager_get_slot);
@@ -252,9 +239,12 @@ void keyslot_manager_put_slot(struct keyslot_manager *ksm, unsigned int slot)if (WARN_ON (slot >= ksm->num_slots))return ; -if (atomic_dec_and_test (&ksm->slot_refs[slot])) { -atomic_inc (&ksm->num_idle_slots); -wake_up (&ksm->wait_queue); +if (atomic_dec_and_lock (&ksm->slots[slot].refs, + &ksm->idle_slots_lock)) { +/* Dropped last reference to this slot; add it to LRU list */ +list_add_tail (&ksm->slots[slot].lru, &ksm->idle_slots); +spin_unlock (&ksm->idle_slots_lock); +wake_up (&ksm->idle_slots_wait_queue); } }EXPORT_SYMBOL (keyslot_manager_put_slot);
@@ -271,7 +261,7 @@ EXPORT_SYMBOL(keyslot_manager_put_slot);* the refcount on the slot is 0. Returns -EBUSY if the refcount is not 0, and * -errno on error. * -* Context: Process context. Takes and releases ksm->lock. +* Context: Process context. */ int keyslot_manager_evict_key(struct keyslot_manager *ksm, const u8 *key,
@@ -279,37 +269,30 @@ int keyslot_manager_evict_key(struct keyslot_manager *ksm, unsigned int data_unit_size) {int slot; -int err =0 ; +int err; -mutex_lock (&ksm->lock); +down_write (&ksm->lock); slot = ksm->ksm_ll_ops.keyslot_find (ksm->ll_priv_data, key, - crypt_mode, - data_unit_size); - + crypt_mode, data_unit_size);if (slot <0 ) { -mutex_unlock (&ksm->lock); +up_write (&ksm->lock);return slot; } -if (atomic_read (&ksm->slot_refs[slot]) ==0 ) { +if (atomic_read (&ksm->slots[slot].refs) ==0 ) { err = ksm->ksm_ll_ops.keyslot_evict (ksm->ll_priv_data, key, - crypt_mode, - data_unit_size, + crypt_mode, data_unit_size, slot); }else { err = -EBUSY ; } - -mutex_unlock (&ksm->lock); +up_write (&ksm->lock);return err; }EXPORT_SYMBOL (keyslot_manager_evict_key);void keyslot_manager_destroy (struct keyslot_manager *ksm) { -if (!ksm) -return ; -kzfree (ksm->last_used_seq_nums); -kzfree (ksm); +kvfree (ksm); }EXPORT_SYMBOL (keyslot_manager_destroy);
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerRe: [RFC PATCH v2 1/8] block: Keyslot Manager for Inline Encryption
In reply to Satya Tangiralainclude/linux/blkdev.h static inline int blkdev_reset_zones_ioctl(struct block_device *bdev, · line 391
Quoted 12 lines · Satya Tangirala
include/linux/blkdev.h +4
-0@@ -385,6 +385,10 @@ static inline int blkdev_reset_zones_ioctl(struct block_device *bdev,#endif /* CONFIG_BLK_DEV_ZONED */ +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +struct keyslot_manager ; +#endif +
This should be placed with the other forward declarations at the beginning of the file. It also doesn't need to be behind an #ifdef. See e.g. struct blkcg_gq which is another conditional field in struct request_queue.
Quoted 5 lines
include/linux/keyslot-manager.h new file +0
-0
[...]
Quoted 18 lines
include/linux/keyslot-manager.h new file +18
-0+#ifdef CONFIG_BLK_INLINE_ENCRYPTION +struct keyslot_manager ; + +extern struct keyslot_manager *keyslot_manager_create (unsigned int num_slots, +const struct keyslot_mgmt_ll_ops *ksm_ops, +void *ll_priv_data); + +extern int +keyslot_manager_get_slot_for_key (struct keyslot_manager *ksm, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size); + +extern void keyslot_manager_get_slot (struct keyslot_manager *ksm, +unsigned int slot); + +extern void keyslot_manager_put_slot (struct keyslot_manager *ksm, +unsigned int slot);
include/linux/keyslot-manager.h line 75
Quoted 10 lines
include/linux/keyslot-manager.h new file +10
-0+ +extern int keyslot_manager_evict_key (struct keyslot_manager *ksm, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size); + +extern void keyslot_manager_destroy (struct keyslot_manager *ksm); + +#else /* CONFIG_BLK_INLINE_ENCRYPTION */ +struct keyslot_manager {};
This is actually a struct definition, not a declaration. This doesn't make sense, since the CONFIG_BLK_INLINE_ENCRYPTION case only needs a forward declaration here. Both cases should just use a forward declaration.
Quoted 29 lines
include/linux/keyslot-manager.h new file +29
-0+ +static inline struct keyslot_manager * +keyslot_manager_create (unsigned int num_slots, +const struct keyslot_mgmt_ll_ops *ksm_ops, +void *ll_priv_data) +{ +return NULL ; +} + +static inline int +keyslot_manager_get_slot_for_key (struct keyslot_manager *ksm, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{ +return -EOPNOTSUPP ; +} + +static inline void keyslot_manager_get_slot (struct keyslot_manager *ksm, +unsigned int slot) { } + +static inline int keyslot_manager_put_slot (struct keyslot_manager *ksm, +unsigned int slot) +{ +return -EOPNOTSUPP ; +} + +static inline int keyslot_manager_evict_key(struct keyslot_manager *ksm, + const u8 *key,
include/linux/keyslot-manager.h line 114
Quoted 10 lines
include/linux/keyslot-manager.h new file +10
-0+enum blk_crypt_mode_num crypt_mode, + unsigned int data_unit_size) +{ + return -EOPNOTSUPP; +} + +static inline void keyslot_manager_destroy (struct keyslot_manager *ksm) +{ } + +#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
However, it seems we don't actually need these stub functions, since the keyslot_manager_ functions are only called from .c files that are only compiled when CONFIG_BLK_INLINE_ENCRYPTION, except for the call to keyslot_manager_evict_key() in fscrypt_evict_crypt_key(). But it would make more sense to stub out fscrypt_evict_crypt_key() instead.
So I suggest removing the keyslot_manager_* stubs for now.
- Eric
RFC PATCH v2 2/8 block: Add encryption context to struct bio
In reply to Satya TangiralaWe must have some way of letting a storage device driver know what encryption context it should use for en/decrypting a request. However, it's the filesystem/fscrypt that knows about and manages encryption contexts. As such, when the filesystem layer submits a bio to the block layer, and this bio eventually reaches a device driver with support for inline encryption, the device driver will need to have been told the encryption context for that bio.
We want to communicate the encryption context from the filesystem layer to the storage device along with the bio, when the bio is submitted to the block layer. To do this, we add a struct bio_crypt_ctx to struct bio, which can represent an encryption context (note that we can't use the bi_private field in struct bio to do this because that field does not function to pass information across layers in the storage stack). We also introduce various functions to manipulate the bio_crypt_ctx and make the bio/request merging logic aware of the bio_crypt_ctx.
--- block/bio.c | 12 ++- block/blk-crypt-ctx.c | 90 +++++++++++++++++++ block/blk-merge.c | 34 ++++++- block/bounce.c | 9 +- drivers/md/dm.c | 15 ++-- include/linux/bio.h | 180 ++++++++++++++++++++++++++++++++++++++ include/linux/blk_types.h | 28 ++++++ 7 files changed, 355 insertions(+), 13 deletions(-) create mode 100644 block/blk-crypt-ctx.c
block/bio.c +10 -2
@@ -16,6 +16,7 @@#include <linux/workqueue.h> #include <linux/cgroup.h> #include <linux/blk-cgroup.h> +#include <linux/keyslot-manager.h> #include <trace/events/block.h> #include "blk.h"
@@ -240,6 +241,7 @@ static void bio_free(struct bio *bio)struct bio_set *bs = bio->bi_pool;void *p; +bio_crypt_free_ctx (bio);bio_uninit (bio);if (bs) {
@@ -612,6 +614,7 @@ EXPORT_SYMBOL(__bio_clone_fast);struct bio *bio_clone_fast (struct bio *bio,gfp_t gfp_mask,struct bio_set *bs) {struct bio *b; +int ret; b =bio_alloc_bioset (gfp_mask,0 , bs);if (!b)
@@ -619,9 +622,13 @@ struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)__bio_clone_fast (b, bio); -if (bio_integrity (bio)) { -int ret; + ret =bio_clone_crypt_context (b, bio, gfp_mask); +if (ret <0 ) { +bio_put (b); +return NULL ; + } +if (bio_integrity (bio)) { ret =bio_integrity_clone (b, bio, gfp_mask);if (ret <0 ) {
@@ -1019,6 +1026,7 @@ void bio_advance(struct bio *bio, unsigned bytes)bio_integrity_advance (bio, bytes);bio_advance_iter (bio, &bio->bi_iter, bytes); +bio_crypt_advance (bio, bytes); }EXPORT_SYMBOL (bio_advance);
block/blk-crypt-ctx.c new file +90 -0
90 lines not on this page · raw message
block/blk-merge.c +30 -4
@@ -531,6 +531,9 @@ static inline int ll_new_hw_segment(struct request_queue *q,if (blk_integrity_merge_bio (q, req, bio) == false) goto no_merge; +if (WARN_ON (!bio_crypt_ctx_compatible (bio, req->bio))) + goto no_merge; +/* * This will form the start of a new hw segment. Bump both * counters.
@@ -696,8 +699,13 @@ static enum elv_merge blk_try_req_merge(struct request *req, {if (blk_discard_mergable (req))return ELEVATOR_DISCARD_MERGE ; -else if (blk_rq_pos (req) +blk_rq_sectors (req) ==blk_rq_pos (next)) +else if (blk_rq_pos (req) +blk_rq_sectors (req) ==blk_rq_pos (next)) { +if (!bio_crypt_ctx_back_mergeable ( + req->bio,blk_rq_sectors (req), next->bio)) { +return ELEVATOR_NO_MERGE ; + }return ELEVATOR_BACK_MERGE ; + }return ELEVATOR_NO_MERGE ; }
@@ -733,6 +741,9 @@ static struct request *attempt_merge(struct request_queue *q,if (req->ioprio != next->ioprio)return NULL ; +if (!bio_crypt_ctx_compatible (req->bio, next->bio)) +return NULL ; +/* * If we are allowed to merge, then append bio list * from next to rq and release next. merge_requests_fn
@@ -865,16 +876,31 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)if (rq->ioprio !=bio_prio (bio))return false; +/* Only merge if the crypt contexts are compatible */ +if (!bio_crypt_ctx_compatible (bio, rq->bio)) +return false; +return true; }enum elv_merge blk_try_merge (struct request *rq,struct bio *bio) { -if (blk_discard_mergable (rq)) +if (blk_discard_mergable (rq)) {return ELEVATOR_DISCARD_MERGE ; -else if (blk_rq_pos (rq) +blk_rq_sectors (rq) == bio->bi_iter.bi_sector) + }else if (blk_rq_pos (rq) +blk_rq_sectors (rq) == + bio->bi_iter.bi_sector) { +if (!bio_crypt_ctx_back_mergeable (rq->bio, +blk_rq_sectors (rq), bio)) { +return ELEVATOR_NO_MERGE ; + }return ELEVATOR_BACK_MERGE ; -else if (blk_rq_pos (rq) -bio_sectors (bio) == bio->bi_iter.bi_sector) + }else if (blk_rq_pos (rq) -bio_sectors (bio) == + bio->bi_iter.bi_sector) { +if (!bio_crypt_ctx_back_mergeable (bio, +bio_sectors (bio), rq->bio)) { +return ELEVATOR_NO_MERGE ; + }return ELEVATOR_FRONT_MERGE ; + }return ELEVATOR_NO_MERGE ; }
block/bounce.c +7 -2
@@ -220,6 +220,7 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,struct bvec_iter iter;struct bio_vec bv;struct bio *bio; +int ret;/* * Pre immutable biovecs, __bio_clone() used to just do a memcpy from
@@ -267,9 +268,13 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,break ; } -if (bio_integrity (bio_src)) { -int ret; + ret =bio_clone_crypt_context (bio, bio_src, gfp_mask); +if (ret <0 ) { +bio_put (bio); +return NULL ; + } +if (bio_integrity (bio_src)) { ret =bio_integrity_clone (bio, bio_src, gfp_mask);if (ret <0 ) {bio_put (bio);
drivers/md/dm.c +10 -5
@@ -1324,12 +1324,15 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio, sector_t sector, unsigned len) {struct bio *clone = &tio->clone; +int ret;__bio_clone_fast (clone, bio); -if (bio_integrity (bio)) { -int r; + ret =bio_clone_crypt_context (clone, bio,GFP_NOIO ); +if (ret <0 ) +return ret; +if (bio_integrity (bio)) {if (unlikely (!dm_target_has_integrity (tio->ti->type) && !dm_target_passes_integrity (tio->ti->type))) { DMWARN("%s: the target %s doesn't support integrity data.",
@@ -1338,9 +1341,11 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,return -EIO ; } - r =bio_integrity_clone (clone, bio,GFP_NOIO ); -if (r <0 ) -return r; + ret =bio_integrity_clone (clone, bio,GFP_NOIO ); +if (ret <0 ) { +bio_crypt_free_ctx (clone); +return ret; + } }bio_advance (clone,to_bytes (sector - clone->bi_iter.bi_sector));
include/linux/bio.h +180 -0
186 lines not on this page · raw message
include/linux/blk_types.h +28 -0
@@ -148,6 +148,29 @@ enum blk_crypt_mode_num {*/ }; +struct bio_crypt_ctx { +int keyslot; +u8 *raw_key; +enum blk_crypt_mode_num crypt_mode; +u64 data_unit_num; +unsigned int data_unit_size_bits; + +/* +* The keyslot manager where the key has been programmed +* with keyslot. +*/ +struct keyslot_manager *processing_ksm; + +/* +* Copy of the bvec_iter when this bio was submitted. +* We only want to en/decrypt the part of the bio +* as described by the bvec_iter upon submission because +* bio might be split before being resubmitted +*/ +struct bvec_iter crypt_iter; +u64 sw_data_unit_num; +}; +/* * main unit of I/O for the block layer and lower layers (ie drivers and * stacking drivers)
@@ -186,6 +209,11 @@ struct bio {struct blkcg_gq *bi_blkg;struct bio_issue bi_issue;#endif + +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +struct bio_crypt_ctx *bi_crypt_context; +#endif +union {#if defined(CONFIG_BLK_DEV_INTEGRITY )struct bio_integrity_payload *bi_integrity;/* data integrity */
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Eric Biggers
maintainerRe: [RFC PATCH v2 2/8] block: Add encryption context to struct bio
Hi Satya,
Quoted 34 lines · Satya Tangirala
We must have some way of letting a storage device driver know what encryption context it should use for en/decrypting a request. However, it's the filesystem/fscrypt that knows about and manages encryption contexts. As such, when the filesystem layer submits a bio to the block layer, and this bio eventually reaches a device driver with support for inline encryption, the device driver will need to have been told the encryption context for that bio.
We want to communicate the encryption context from the filesystem layer to the storage device along with the bio, when the bio is submitted to the block layer. To do this, we add a struct
bio_crypt_ctxto struct bio, which can represent an encryption context (note that we can't use thebi_privatefield in struct bio to do this because that field does not function to pass information across layers in the storage stack). We also introduce various functions to manipulate thebio_crypt_ctxand make the bio/request merging logic aware of thebio_crypt_ctx.Signed-off-by Satya Tangirala <[email protected]>--- block/bio.c | 12 ++- block/blk-crypt-ctx.c | 90 +++++++++++++++++++ block/blk-merge.c | 34 ++++++- block/bounce.c | 9 +- drivers/md/dm.c | 15 ++-- include/linux/bio.h | 180 ++++++++++++++++++++++++++++++++++++++ include/linux/blk_types.h | 28 ++++++ 7 files changed, 355 insertions(+), 13 deletions(-) create mode 100644 block/blk-crypt-ctx.cblock/bio.c +0
-0
block/bio.c line 19
Quoted 4 lines
block/bio.c +1
-0@@ -16,6 +16,7 @@#include <linux/workqueue.h> #include <linux/cgroup.h> #include <linux/blk-cgroup.h> +#include <linux/keyslot-manager.h>
No need to include keyslot-manager.h here.
block/bio.c void bio_advance(struct bio *bio, unsigned bytes) · line 1031
Quoted 7 lines
block/bio.c +1
-0@@ -1019,6 +1026,7 @@ void bio_advance(struct bio *bio, unsigned bytes)bio_integrity_advance (bio, bytes);bio_advance_iter (bio, &bio->bi_iter, bytes); +bio_crypt_advance (bio, bytes); }EXPORT_SYMBOL (bio_advance);
It would be more logical to do bio_crypt_advance() before bio_advance_iter(), so that the special features (encryption and integrity) are grouped together.
block/blk-crypt-ctx.c
Quoted 6 lines
block/blk-crypt-ctx.c new file +0
-0
It would be more logical for this file to be named "bio-crypt-ctx.c", as that would match 'struct bio_crypt_ctx' and help distinguish it from "blk-crypto".
Quoted 5 lines
block/blk-crypt-ctx.c new file +4
-0@@ -0,0 +1,90 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright 2019 Google LLC +*/
block/blk-crypt-ctx.c line 14
Quoted 10 lines
block/blk-crypt-ctx.c new file +10
-0+ +#include <linux/bio.h> +#include <linux/blkdev.h> +#include <linux/slab.h> +#include <linux/keyslot-manager.h> + +struct bio_crypt_ctx *bio_crypt_alloc_ctx (gfp_t gfp_mask) +{ +return kzalloc (sizeof (struct bio_crypt_ctx ), gfp_mask); +}
This needs EXPORT_SYMBOL(), since it's called by bio_crypt_set_ctx() which is an inline function that will be called by places that submit bios.
block/blk-crypt-ctx.c line 23
Quoted 9 lines
block/blk-crypt-ctx.c new file +9
-0+ +void bio_crypt_free_ctx (struct bio *bio) +{ +kzfree (bio->bi_crypt_context); + bio->bi_crypt_context =NULL ; +} + +int bio_clone_crypt_context (struct bio *dst,struct bio *src,gfp_t gfp_mask) +{
How about naming this function bio_crypt_clone(), for consistency with bio_integrity_clone()?
block/blk-crypt-ctx.c line 25
Quoted 2 lines
block/blk-crypt-ctx.c new file +2
-0+if (!bio_is_encrypted (src) ||bio_crypt_swhandled (src)) +return 0 ;
Why isn't cloning needed when bio_crypt_swhandled(src)?
Quoted 5 lines
block/blk-crypt-ctx.c new file +5
-0+ + dst->bi_crypt_context =bio_crypt_alloc_ctx (gfp_mask); +if (!dst->bi_crypt_context) +return -ENOMEM ; +
block/blk-crypt-ctx.c line 40
Quoted 10 lines
block/blk-crypt-ctx.c new file +10
-0+ *dst->bi_crypt_context = *src->bi_crypt_context; + +if (!bio_crypt_has_keyslot (src)) +return 0 ; + +keyslot_manager_get_slot (src->bi_crypt_context->processing_ksm, + src->bi_crypt_context->keyslot); + +return 0 ; +}
Nit: a conditional get would be cleaner than an early return here.
if (bio_crypt_has_keyslot(src)) keyslot_manager_get_slot(src->bi_crypt_context->processing_ksm, src->bi_crypt_context->keyslot);
Also, this function needs EXPORT_SYMBOL(), since it's called by drivers/md/dm.c, which can be a loadable module.
Quoted 8 lines
block/blk-crypt-ctx.c new file +8
-0+/* +* Checks that two bio crypt contexts are compatible - i.e. that +* they are mergeable except for data_unit_num continuity. +*/ +bool bio_crypt_ctx_compatible (struct bio *b_1,struct bio *b_2) +{ +struct bio_crypt_ctx *bc1 = b_1->bi_crypt_context; +struct bio_crypt_ctx *bc2 = b_2->bi_crypt_context;
block/blk-crypt-ctx.c line 69
Quoted 10 lines
block/blk-crypt-ctx.c new file +10
-0+ +if (bio_is_encrypted (b_1) !=bio_is_encrypted (b_2)) +return false; + +if (!bio_is_encrypted (b_1)) +return true; + +return bc1->keyslot != bc2->keyslot && + bc1->data_unit_size_bits == bc2->data_unit_size_bits; +}
It needs to be 'bc1->keyslot == bc2->keyslot'.
Quoted 11 lines
block/blk-crypt-ctx.c new file +11
-0+ +/* +* Checks that two bio crypt contexts are compatible, and also +* that their data_unit_nums are continuous (and can hence be merged) +*/ +bool bio_crypt_ctx_back_mergeable (struct bio *b_1, +unsigned int b1_sectors, +struct bio *b_2) +{ +struct bio_crypt_ctx *bc1 = b_1->bi_crypt_context; +struct bio_crypt_ctx *bc2 = b_2->bi_crypt_context;
block/blk-crypt-ctx.c line 90
Quoted 10 lines
block/blk-crypt-ctx.c new file +10
-0+ +if (!bio_crypt_ctx_compatible (b_1, b_2)) +return false; + +return !bio_is_encrypted (b_1) || + (bc1->data_unit_num + + (b1_sectors >> (bc1->data_unit_size_bits -9 )) == + bc2->data_unit_num); +} +
Unnecessary blank line at end of file.
Quoted 6 lines
include/linux/bio.h +0
-0@@ -561,6 +561,186 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) }
include/linux/bio.h static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) · line 571
Quoted 10 lines
include/linux/bio.h +8
-0#endif +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +extern int bio_clone_crypt_context (struct bio *dst,struct bio *src, +gfp_t gfp_mask); + +static inline bool bio_is_encrypted (struct bio *bio) +{ +return bio && bio->bi_crypt_context; +}
Is the 'bio != NULL' check actually needed? Most bio helper functions don't check for NULL, as it's not a meaningful case.
include/linux/bio.h static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) · line 588
Quoted 7 lines
include/linux/bio.h +7
-0+ +static inline bool bio_crypt_has_keyslot (struct bio *bio) +{ +return bio_is_encrypted (bio) && + bio->bi_crypt_context->keyslot >=0 ; +} +
I think the bio_is_encrypted() check here should be dropped, since all callers check it beforehand anyway. It doesn't really make sense for someone to call functions that are meant to access fields of the bio_crypt_ctx, before verifying that there actually is a bio_crypt_ctx. Other bio_crypt_* functions don't check for NULL, so it seems inconsistent that this one does.
include/linux/bio.h static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) · line 619
Quoted 5 lines
include/linux/bio.h +5
-0+ +static inline int bio_crypt_get_slot (struct bio *bio) +{ +return bio->bi_crypt_context->keyslot; +}
For consistency this should be named *_get_keyslot(), not *_get_slot().
Quoted 18 lines
include/linux/bio.h +18
-0+ +static inline void bio_crypt_set_keyslot (struct bio *bio, +unsigned int keyslot, +struct keyslot_manager *ksm) +{ + bio->bi_crypt_context->keyslot = keyslot; + bio->bi_crypt_context->processing_ksm = ksm; + + bio->bi_crypt_context->crypt_iter = bio->bi_iter; + bio->bi_crypt_context->sw_data_unit_num = + bio->bi_crypt_context->data_unit_num; +} + +static inline void bio_crypt_unset_keyslot (struct bio *bio) +{ + bio->bi_crypt_context->processing_ksm =NULL ; + bio->bi_crypt_context->keyslot =-1 ; +}
include/linux/bio.h static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags) · line 647
Quoted 10 lines
include/linux/bio.h +10
-0+ +static inline u8 *bio_crypt_raw_key (struct bio *bio) +{ +return bio->bi_crypt_context->raw_key; +} + +static inline enum blk_crypt_mode_num bio_crypt_mode (struct bio *bio) +{ +return bio->bi_crypt_context->crypt_mode; +}
bio_crypt_unset_keyslot(), bio_crypt_raw_key(), and bio_crypt_mode() are only used in blk-crypto.c. Is there any reason for block users or drivers to need to call them? If not, these fields should really just be accessed directly in blk-crypto.c. It's not needed to provide these functions in bio.h where they are available to everyone.
Quoted 21 lines
include/linux/blk_types.h +13
-0@@ -148,6 +148,29 @@ enum blk_crypt_mode_num {*/ }; +struct bio_crypt_ctx { +int keyslot; +u8 *raw_key; +enum blk_crypt_mode_num crypt_mode; +u64 data_unit_num; +unsigned int data_unit_size_bits; + +/* +* The keyslot manager where the key has been programmed +* with keyslot. +*/ +struct keyslot_manager *processing_ksm; +
include/linux/blk_types.h enum blk_crypt_mode_num { · line 173
Quoted 10 lines
include/linux/blk_types.h +10
-0+/* +* Copy of the bvec_iter when this bio was submitted. +* We only want to en/decrypt the part of the bio +* as described by the bvec_iter upon submission because +* bio might be split before being resubmitted +*/ +struct bvec_iter crypt_iter; +u64 sw_data_unit_num; +}; +
How about making this struct definition conditional on CONFIG_BLK_INLINE_ENCRYPTION? When !CONFIG_BLK_INLINE_ENCRYPTION, no code is compiled that dereferences any pointer to this struct.
For consistency with bio_integrity_payload and to avoid an extra #ifdef, I think this should also be moved to bio.h.
blk_crypt_mode_num can be moved to bio.h too, but it will need to be unconditional since it's used as a parameter to bio_crypt_set_ctx().
Quoted 18 lines
include/linux/blk_types.h +5
-0/* * main unit of I/O for the block layer and lower layers (ie drivers and * stacking drivers) @@ -186,6 +209,11 @@ struct bio {struct blkcg_gq *bi_blkg;struct bio_issue bi_issue;#endif + +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +struct bio_crypt_ctx *bi_crypt_context; +#endif +union {#if defined(CONFIG_BLK_DEV_INTEGRITY )struct bio_integrity_payload *bi_integrity;/* data integrity */ Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Is it actually meaningful to use the blk_integrity feature in combination with inline encryption? How might this be tested? If the features actually conflict anyway, bi_crypt_context and bi_integrity could share the same union.
- Eric
RFC PATCH v2 3/8 block: blk-crypto for Inline Encryption
In reply to Satya TangiralaWe introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with the encryption key, algorithm and data_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.
Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to Documentation/block/blk-crypto.txt.
Known issues:
1) We're allocating crypto_skcipher in blk_crypto_keyslot_program, which uses GFP_KERNEL to allocate memory, but this function is on the write path for IO - we need to add support for specifying a different flags to the crypto API.
--- Documentation/block/blk-crypto.txt | 185 ++++++++++ block/Kconfig | 8 + block/Makefile | 2 + block/bio.c | 5 + block/blk-core.c | 11 +- block/blk-crypto.c | 558 +++++++++++++++++++++++++++++ include/linux/blk-crypto.h | 40 +++ 7 files changed, 808 insertions(+), 1 deletion(-) create mode 100644 Documentation/block/blk-crypto.txt create mode 100644 block/blk-crypto.c create mode 100644 include/linux/blk-crypto.h
Documentation/block/blk-crypto.txt new file +185 -0
@@ -0,0 +1,185 @@ +BLK-CRYPTO and KEYSLOT MANAGER +=========================== + +CONTENTS +1. Objective +2. Constraints and notes +3. Design +4. Blk-crypto + 4-1 What does blk-crypto do on bio submission +5. Layered Devices +6. Future optimizations for layered devices + +1. Objective +============ + +We want to support inline encryption (IE) in the kernel. +To allow for testing, we also want a software fallback when actual +IE hardware is absent. We also want IE to work with layered devices +like dm and loopback (i.e. we want to be able to use the IE hardware +of the underlying devices if present, or else fall back to software +en/decryption). + + +2. Constraints and notes +======================== + +1) IE hardware have a limited number of “keyslots” that can be programmed +with an encryption context (key, algorithm, data unit size, etc.) at any time. +One can specify a keyslot in a data request made to the device, and the +device will en/decrypt the data using the encryption context programmed into +that specified keyslot. When possible, we want to make multiple requests with +the same encryption context share the same keyslot. + +2) We need a way for filesystems to specify an encryption context to use for +en/decrypting a struct bio, and a device driver (like UFS) needs to be able +to use that encryption context when it processes the bio. + +3) We need a way for device drivers to expose their capabilities in a unified +way to the upper layers. + + +3. Design +========= + +We add a struct bio_crypt_context to struct bio that can represent an +encryption context, because we need to able to pass this encryption context +from the FS layer to the device driver to act upon. + +While IE hardware works on the notion of keyslots, the FS layer has no +knowledge of keyslots - it simply wants to specify an encryption context to +use while en/decrypting a bio. + +We introduce a keyslot manager (KSM) that handles the translation from +encryption contexts specified by the FS to keyslots on the IE hardware. +This KSM also serves as the way IE hardware can expose their capabilities to +upper layers. The generic mode of operation is: each device driver that wants +to support IE will construct a KSM and set it up in its struct request_queue. +Upper layers that want to use IE on this device can then use this KSM in +the device’s struct request_queue to translate an encryption context into +a keyslot. The presence of the KSM in the request queue shall be used to mean +that the device supports IE. + +On the device driver end of the interface, the device driver needs to tell the +KSM how to actually manipulate the IE hardware in the device to do things like +programming the crypto key into the IE hardware into a particular keyslot. All +this is achieved through the struct keyslot_mgmt_ll_ops that the device driver +passes to the KSM when creating it. + +It uses refcounts to track which keyslots are idle (either they have no +encryption context programmed, or there are no in flight struct bios +referencing that keyslot). When a new encryption context needs a keyslot, it +tries to find a keyslot that has already been programmed with the same +encryption context, and if there is no such keyslot, it evicts the least +recently used idle keyslot and programs the new encryption context into that +one. If no idle keyslots are available, then the caller will sleep until there +is at least one. + + +4. Blk-crypto +============= + +The above is sufficient for simple cases, but does not work if there is a +need for a software fallback, or if we are want to use IE with layered devices. +To these ends, we introduce blk-crypto. Blk-crypto allows us to present a +unified view of encryption to the FS (so FS only needs to specify an +encryption context and not worry about keyslots at all), and blk-crypto can +decide whether to delegate the en/decryption to IE hardware or to software +(i.e. to the kernel crypto API). Blk-crypto maintains an internal KSM that +serves as the software fallback to the kernel crypto API. + +Blk-crypto needs to ensure that the encryption context is programmed into the +"correct" keyslot manager for IE. If a bio is submitted to a layered device +that eventually passes the bio down to a device that really does support IE, we +want the encryption context to be programmed into a keyslot for the KSM of the +device with IE support. However, blk-crypto does not know a priori whether a +particular device is the final device in the layering structure for a bio or +not. So in the case that a particular device does not support IE, since it is +possibly the final destination device for the bio, if the bio requires +encryption (i.e. the bio is doing a write operation), blk-crypto must fallback +to software *before* sending the bio to the device. + +Blk-crypto ensures that +1) The bio’s encryption context is programmed into a keyslot in the KSM of the +request queue that the bio is being submitted to (or the software fallback KSM +if the request queue doesn’t have a KSM), and that the processing_ksm in the +bi_crypt_context is set to this KSM + +2) That the bio has its own individual reference to the keyslot in this KSM. +Once the bio passes through blk-crypto, its encryption context is programmed +in some KSM. The “its own individual reference to the keyslot” ensures that +keyslots can be released by each bio independently of other bios while ensuring +that the bio has a valid reference to the keyslot when, for e.g., the software +fallback KSM in blk-crypto performs crypto for on the device’s behalf. The +individual references are ensured by increasing the refcount for the keyslot in +the processing_ksm when a bio with a programmed encryption context is cloned. + + +4-1. What blk-crypto does on bio submission +------------------------------------------- + +Case 1: blk-crypto is given a bio with only an encryption context that hasn’t +been programmed into any keyslot in any KSM (for e.g. a bio from the FS). In +this case, blk-crypto will program the encryption context into the KSM of the +request queue the bio is being submitted to (and if this KSM does not exist, +then it will program it into blk-crypto’s internal KSM for software fallback). +The KSM that this encryption context was programmed into is stored as the +processing_ksm in the bio’s bi_crypt_context. + +Case 2: blk-crypto is given a bio whose encryption context has already been +programmed into a keyslot in the *software fallback KSM*. In this case, +blk-crypto does nothing; it treats the bio as not having specified an +encryption context. Note that we cannot do what we will do in Case 3 here +because we would have already encrypted the bio in software by this point. + +Case 3: blk-crypto is given a bio whose encryption context has already been +programmed into a keyslot in some KSM (that is *not* the software fallback +KSM). In this case, blk-crypto first releases that keyslot from that KSM and +then treats the bio as in Case 1. + +This way, when a device driver is processing a bio, it can be sure that +the bio’s encryption context has been programmed into some KSM (either the +device driver’s request queue’s KSM, or blk-crypto’s software fallback KSM). +It then simply needs to check if the bio’s processing_ksm is the device’s +request queue’s KSM. If so, then it should proceed with IE. If not, it should +simply do nothing with respect to crypto, because some other KSM (perhaps the +blk-crypto software fallback KSM) is handling the en/decryption. + +Blk-crypto will release the keyslot that is being held by the bio (and also +decrypt it if the bio is using the software fallback KSM) once +bio_remaining_done returns true for the bio. + + +5. Layered Devices +================== + +Layered devices that wish to support IE need to create their own keyslot +manager for their request queue, and expose whatever functionality they choose. +When a layered device wants to pass a bio to another layer (either by +resubmitting the same bio, or by submitting a clone), it doesn’t need to do +anything special because the bio (or the clone) will once again pass through +blk-crypto, which will work as described in Case 3. If a layered device wants +for some reason to do the IO by itself instead of passing it on to a child +device, but it also chose to expose IE capabilities by setting up a KSM in its +request queue, it is then responsible for en/decrypting the data itself. In +such cases, the device can choose to call the blk-crypto function +blk_crypto_fallback_to_software (TODO: Not yet implemented), which will +cause the en/decryption to be done via software fallback. + + +6. Future Optimizations for layered devices +=========================================== + +Creating a keyslot manager for the layered device uses up memory for each +keyslot, and in general, a layered device (like dm-linear) merely passes the +request on to a “child” device, so the keyslots in the layered device itself +might be completely unused. We can instead define a new type of KSM; the +“passthrough KSM”, that layered devices can use to let blk-crypto know that +this layered device *will* pass the bio to some child device (and hence +through blk-crypto again, at which point blk-crypto can program the encryption +context, instead of programming it into the layered device’s KSM). Again, if +the device “lies” and decides to do the IO itself instead of passing it on to +a child device, it is responsible for doing the en/decryption (and can choose +to call blk_crypto_fallback_to_software). Another use case for the +"passthrough KSM" is for IE devices that want to manage their own keyslots/do +not have a limited number of keyslots.
block/Kconfig +8 -0
@@ -163,6 +163,14 @@ config BLK_SED_OPAL Enabling this option enables users to setup/unlock/lock Locking ranges for SED devices using the Opal protocol. +config BLK_INLINE_ENCRYPTION + bool "Enable inline encryption support in block layer" + help + Build the blk-crypto subsystem. + Enabling this lets the block layer handle encryption, + so users can take advantage of inline encryption + hardware if present. + menu "Partition Types" source "block/partitions/Kconfig"
block/Makefile +2 -0
@@ -35,3 +35,5 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o obj-$(CONFIG_BLK_PM) += blk-pm.o +obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypt-ctx.o blk-crypto.o \ + keyslot-manager.o
block/bio.c +5 -0
@@ -17,6 +17,7 @@#include <linux/cgroup.h> #include <linux/blk-cgroup.h> #include <linux/keyslot-manager.h> +#include <linux/blk-crypto.h> #include <trace/events/block.h> #include "blk.h"
@@ -1829,6 +1830,10 @@ void bio_endio(struct bio *bio) again:if (!bio_remaining_done (bio))return ; + +if (!blk_crypto_endio (bio)) +return ; +if (!bio_integrity_endio (bio))return ;
block/blk-core.c +10 -1
@@ -36,6 +36,7 @@#include <linux/blk-cgroup.h> #include <linux/debugfs.h> #include <linux/bpf.h> +#include <linux/blk-crypto.h> #define CREATE_TRACE_POINTS #include <trace/events/block.h>
@@ -1005,7 +1006,9 @@ blk_qc_t generic_make_request(struct bio *bio)/* Create a fresh bio_list for all subordinate requests */ bio_list_on_stack[1 ] = bio_list_on_stack[0 ];bio_list_init (&bio_list_on_stack[0 ]); - ret = q->make_request_fn (q, bio); + +if (!blk_crypto_submit_bio (&bio)) + ret = q->make_request_fn (q, bio);blk_queue_exit (q);
@@ -1058,6 +1061,9 @@ blk_qc_t direct_make_request(struct bio *bio)if (!generic_make_request_checks (bio))return BLK_QC_T_NONE ; +if (blk_crypto_submit_bio (&bio)) +return BLK_QC_T_NONE ; +if (unlikely (blk_queue_enter (q, nowait ?BLK_MQ_REQ_NOWAIT :0 ))) {if (nowait && !blk_queue_dying (q)) bio->bi_status =BLK_STS_AGAIN ;
@@ -1737,5 +1743,8 @@ int __init blk_dev_init(void) blk_debugfs_root =debugfs_create_dir ("block" ,NULL );#endif +if (blk_crypto_init () <0 ) +panic ("Failed to init blk-crypto\n" ); +return 0 ; }
block/blk-crypto.c new file +558 -0
558 lines not on this page · raw message
include/linux/blk-crypto.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright 2019 Google LLC +*/ + +#ifndef __LINUX_BLK_CRYPTO_H +#define __LINUX_BLK_CRYPTO_H + +#include <linux/types.h> + +#ifdef CONFIG_BLK_INLINE_ENCRYPTION + +struct bio ; + +int blk_crypto_init (void ); + +int blk_crypto_submit_bio (struct bio **bio_ptr); + +bool blk_crypto_endio (struct bio *bio); + +#else /* CONFIG_BLK_INLINE_ENCRYPTION */ + +static inline int blk_crypto_init (void ) +{ +return 0 ; +} + +static inline int blk_crypto_submit_bio (struct bio **bio) +{ +return 0 ; +} + +static inline bool blk_crypto_endio (struct bio *bio) +{ +return true; +} + +#endif /* CONFIG_BLK_INLINE_ENCRYPTION */ + +#endif /* __LINUX_BLK_CRYPTO_H */
Signature · 8 lines
2.22.0.rc1.311.g5d7573a151-goog _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerRe: [RFC PATCH v2 3/8] block: blk-crypto for Inline Encryption
Quoted 37 lines · Satya Tangirala
We introduce blk-crypto, which manages programming keyslots for struct bios. With blk-crypto, filesystems only need to call
bio_crypt_set_ctxwith the encryption key, algorithm anddata_unit_num; they don't have to worry about getting a keyslot for each encryption context, as blk-crypto handles that. Blk-crypto also makes it possible for layered devices like device mapper to make use of inline encryption hardware.Blk-crypto delegates crypto operations to inline encryption hardware when available, and also contains a software fallback to the kernel crypto API. For more details, refer to
Documentation/block/blk-crypto.txt.Known issues:
1) We're allocatingcrypto_skcipherinblk_crypto_keyslot_program, which usesGFP_KERNELto allocate memory, but this function is on the write path for IO - we need to add support for specifying a different flags to the crypto API.Signed-off-by Satya Tangirala <[email protected]>--- Documentation/block/blk-crypto.txt | 185 ++++++++++ block/Kconfig | 8 + block/Makefile | 2 + block/bio.c | 5 + block/blk-core.c | 11 +- block/blk-crypto.c | 558 +++++++++++++++++++++++++++++ include/linux/blk-crypto.h | 40 +++ 7 files changed, 808 insertions(+), 1 deletion(-) create mode 100644 Documentation/block/blk-crypto.txt create mode 100644 block/blk-crypto.c create mode 100644 include/linux/blk-crypto.hDocumentation/block/blk-crypto.txt new file +0
-0
Documentation/block/blk-crypto.txt line 2
Quoted 2 lines
Documentation/block/blk-crypto.txt new file +2
-0@@ -0,0 +1,185 @@ +BLK-CRYPTO and KEYSLOT MANAGER +===========================
How about renaming this documentation file to inline-encryption.txt and making sure it covers the inline encryption feature as a whole? "blk-crypto" is just part of it.
block/Makefile obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o · line 39
Quoted 10 lines
block/Makefile +2
-0@@ -35,3 +35,5 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o obj-$(CONFIG_BLK_PM) += blk-pm.o +obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypt-ctx.o blk-crypto.o \ + keyslot-manager.o
Two of these .c files were added by earlier patches, but they're not compiled until now. The usual practice is to make the code actually compiled after each patch, e.g. by introducing the kconfig option first. Otherwise there can be build errors that don't show up until suddenly all the code is enabled at once.
Quoted 14 lines
block/blk-crypto.c new file +8
-0@@ -0,0 +1,558 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright 2019 Google LLC +*/ +#include <linux/blk-crypto.h> +#include <linux/keyslot-manager.h> +#include <linux/mempool.h> +#include <linux/blk-cgroup.h>
block/blk-crypto.c line 18
Quoted 10 lines
block/blk-crypto.c new file +10
-0+#include <crypto/skcipher.h> +#include <crypto/algapi.h> + +struct blk_crypt_mode { +const char *friendly_name; +const char *cipher_str; +size_t keysize; +size_t ivsize; +bool needs_essiv; +};
'friendly_name', 'ivsize', and 'needs_essiv' are unused. So they should be removed until they're actually needed.
block/blk-crypto.c line 28
Quoted 10 lines
block/blk-crypto.c new file +10
-0+ +static const struct blk_crypt_mode blk_crypt_modes[] = { + [BLK_ENCRYPTION_MODE_AES_256_XTS ] = { + .friendly_name ="AES-256-XTS" , + .cipher_str ="xts(aes)" , + .keysize =64 , + .ivsize =16 , + }, +/* TODO: the rest of the algs that fscrypt supports */ +};
It's arguably a layering violation to mention fscrypt specifically here. There will eventually be other users of this too.
block/blk-crypto.c line 32
Quoted 2 lines
block/blk-crypto.c new file +2
-0+/* TODO: Do we want to make this user configurable somehow? */ +#define BLK_CRYPTO_NUM_KEYSLOTS 100
This should be a kernel command line parameter.
block/blk-crypto.c line 50
Quoted 2 lines
block/blk-crypto.c new file +2
-0+ +static unsigned int num_prealloc_bounce_pg =32 ;
This should be a kernel command line parameter too.
block/blk-crypto.c line 56
Quoted 6 lines
block/blk-crypto.c new file +6
-0+ +bool bio_crypt_swhandled (struct bio *bio) +{ +return bio_crypt_has_keyslot (bio) && + bio->bi_crypt_context->processing_ksm == blk_crypto_ksm; +}
processing_ksm is NULL when there isn't a keyslot, so calling bio_crypt_has_keyslot() isn't necessary here.
Quoted 43 lines
block/blk-crypto.c new file +43
-0+ +/* TODO: handle modes that need essiv */ +static int blk_crypto_keyslot_program (void *priv,const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size, +unsigned int slot) +{ +struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot]; +struct crypto_skcipher *tfm = slotp->tfm; +const struct blk_crypt_mode *mode = &blk_crypt_modes[crypt_mode]; +size_t keysize = mode->keysize; +int err; + +if (crypt_mode != slotp->crypt_mode || !tfm) { +crypto_free_skcipher (slotp->tfm); + slotp->tfm =NULL ; +memset (slotp->key,0 ,BLK_CRYPTO_MAX_KEY_SIZE ); + tfm =crypto_alloc_skcipher ( + mode->cipher_str,0 ,0 ); +if (IS_ERR (tfm)) +return PTR_ERR (tfm); + +crypto_skcipher_set_flags (tfm, +CRYPTO_TFM_REQ_FORBID_WEAK_KEYS ); + slotp->crypt_mode = crypt_mode; + slotp->tfm = tfm; + } + + + err =crypto_skcipher_setkey (tfm, key, keysize); + +if (err) { +crypto_free_skcipher (tfm); + slotp->tfm =NULL ; +return err; + } + +memcpy (slotp->key, key, keysize); + +return 0 ; +} + +static int blk_crypto_keyslot_evict(void *priv, const u8 *key,
block/blk-crypto.c line 109
Quoted 10 lines
block/blk-crypto.c new file +10
-0+enum blk_crypt_mode_num crypt_mode, + unsigned int data_unit_size, + unsigned int slot) +{ + crypto_free_skcipher(blk_crypto_keyslots[slot].tfm); + blk_crypto_keyslots[slot].tfm =NULL ; +memset (blk_crypto_keyslots[slot].key,0 ,BLK_CRYPTO_MAX_KEY_SIZE ); + +return 0 ; +}
If the call to crypto_skcipher_setkey() fails, then the ->tfm is set to NULL as if the keyslot were free, but the raw key isn't wiped. The state should be kept
consistent: the raw key of a free keyslot should always be zeroed.
The easiest way to handle this would be to add a helper function:
static void evict_keyslot(unsigned int slot)
{
struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot];
crypto_free_skcipher(slotp->tfm);
slotp->tfm = NULL;
memzero_explicit(slotp->key, BLK_CRYPTO_MAX_KEY_SIZE);
}Then call this from the two places in blk_crypto_keyslot_program(), and from blk_crypto_keyslot_evict().
(It doesn't really need to be memzero_explicit() instead of memset() here, but it's good to make the intent of "this is wiping a crypto key" clear.)
Quoted 6 lines
block/blk-crypto.c new file +6
-0+ +static int blk_crypto_keyslot_find (void *priv, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size_bytes) +{
block/blk-crypto.c line 125
Quoted 10 lines
block/blk-crypto.c new file +10
-0+int slot; +const size_t keysize = blk_crypt_modes[crypt_mode].keysize; + +/* TODO: hashmap? */ +for (slot =0 ; slot <BLK_CRYPTO_NUM_KEYSLOTS ; slot++) { +if (blk_crypto_keyslots[slot].crypt_mode == crypt_mode && + !crypto_memneq (blk_crypto_keyslots[slot].key, key, + keysize)) { +return slot; + }
Nit: can drop the braces here and fit the crypto_memneq() parameters on one line.
block/blk-crypto.c line 137
Quoted 7 lines
block/blk-crypto.c new file +7
-0+static bool blk_crypt_mode_supported (void *priv, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{ +// Of course, blk-crypto supports all blk_crypt_modes. +return true; +}
This actually isn't obvious, since there could be modes that are only supported by particular hardware drivers. It would be more helpful if the comment was:
/* All blk_crypt_modes are required to have a software fallback. */
Quoted 10 lines
block/blk-crypto.c new file +10
-0+static void blk_crypto_put_keyslot (struct bio *bio) +{ +struct bio_crypt_ctx *crypt_ctx = bio->bi_crypt_context; + +keyslot_manager_put_slot (crypt_ctx->processing_ksm, crypt_ctx->keyslot); +bio_crypt_unset_keyslot (bio); +} + +static int blk_crypto_get_keyslot (struct bio *bio, +struct keyslot_manager *ksm)
block/blk-crypto.c line 165
Quoted 10 lines
block/blk-crypto.c new file +10
-0+{ +int slot; +enum blk_crypt_mode_num crypt_mode =bio_crypt_mode (bio); + +if (!ksm) +return -ENOMEM ; + + slot =keyslot_manager_get_slot_for_key (ksm, +bio_crypt_raw_key (bio), + crypt_mode,PAGE_SIZE );
Needs to be '1 << crypt_ctx->data_unit_size_bits', not PAGE_SIZE.
block/blk-crypto.c line 171
Quoted 6 lines
block/blk-crypto.c new file +6
-0+if (slot <0 ) +return slot; + +bio_crypt_set_keyslot (bio, slot, ksm); +return 0 ; +}
Since blk_crypto_{get,put}_keyslot() support any keyslot manager, naming them blk_crypto is a bit confusing, since it suggests they might only be relevant to the software fallback (blk_crypto_keyslots). Maybe they should be renamed to bio_crypt_{acquire,release}_keyslot() and moved to bio-crypt-ctx.c?
Quoted 12 lines
block/blk-crypto.c new file +12
-0+static int blk_crypto_encrypt_bio (struct bio **bio_ptr) +{ +struct bio *src_bio = *bio_ptr; +int slot; +struct skcipher_request *ciph_req =NULL ; +DECLARE_CRYPTO_WAIT (wait); +struct bio_vec bv; +struct bvec_iter iter; +int err =0 ; +u64 curr_dun; +union { +__le64 dun;
block/blk-crypto.c line 245
Quoted 10 lines
block/blk-crypto.c new file +10
-0+u8 bytes[16 ]; + } iv; +struct scatterlist src, dst; +struct bio *enc_bio; +struct bio_vec *enc_bvec; +int i, j; +unsigned int num_sectors; + +if (!blk_crypto_keyslots) +return -ENOMEM ;
Why the NULL check for blk_crypto_keyslots? The kernel already panics if blk_crypto_init() fails.
Quoted 13 lines
block/blk-crypto.c new file +13
-0+ +/* Split the bio if it's too big for single page bvec */ + i =0 ; + num_sectors =0 ; +bio_for_each_segment (bv, src_bio, iter) { + num_sectors += bv.bv_len >>9 ; +if (++i ==BIO_MAX_PAGES ) +break ; + } +if (num_sectors <bio_sectors (src_bio)) { +struct bio *split_bio; + + split_bio =bio_split (src_bio, num_sectors,GFP_NOIO ,NULL );
block/blk-crypto.c line 268
Quoted 10 lines
block/blk-crypto.c new file +10
-0+if (!split_bio) { + src_bio->bi_status =BLK_STS_RESOURCE ; +return -ENOMEM ; + } +bio_chain (split_bio, src_bio); +generic_make_request (src_bio); + *bio_ptr = split_bio; + } + + src_bio = *bio_ptr;
This line can be moved into the previous 'if' block.
Quoted 18 lines
block/blk-crypto.c new file +18
-0+ + enc_bio =blk_crypto_clone_bio (src_bio); +if (!enc_bio) { + src_bio->bi_status =BLK_STS_RESOURCE ; +return -ENOMEM ; + } + + err =blk_crypto_get_keyslot (src_bio, blk_crypto_ksm); +if (err) { + src_bio->bi_status =BLK_STS_IOERR ; +bio_put (enc_bio); +return err; + } + slot =bio_crypt_get_slot (src_bio); + + ciph_req =skcipher_request_alloc (blk_crypto_keyslots[slot].tfm, +GFP_NOIO ); +if (!ciph_req) {
block/blk-crypto.c line 296
Quoted 10 lines
block/blk-crypto.c new file +10
-0+ src_bio->bi_status =BLK_STS_RESOURCE ; + err = -ENOMEM ; +bio_put (enc_bio); + goto out_release_keyslot; + } + +skcipher_request_set_callback (ciph_req, +CRYPTO_TFM_REQ_MAY_BACKLOG | +CRYPTO_TFM_REQ_MAY_SLEEP , + crypto_req_done, &wait);
This function and blk_crypto_decrypt_bio() are getting long. To help a tiny bit, maybe add a helper function blk_crypto_alloc_skcipher_request(bio) and call it from both places?
block/blk-crypto.c line 305
Quoted 9 lines
block/blk-crypto.c new file +9
-0+ + curr_dun =bio_crypt_sw_data_unit_num (src_bio); +sg_init_table (&src,1 ); +sg_init_table (&dst,1 ); +for (i =0 , enc_bvec = enc_bio->bi_io_vec; i < enc_bio->bi_vcnt; + enc_bvec++, i++) { +struct page *page = enc_bvec->bv_page; +struct page *ciphertext_page = +mempool_alloc (blk_crypto_page_pool,GFP_NOFS );
GFP_NOIO, not GFP_NOFS.
Quoted 17 lines
block/blk-crypto.c new file +17
-0+ + enc_bvec->bv_page = ciphertext_page; + +if (!ciphertext_page) + goto no_mem_for_ciph_page; + +memset (&iv,0 ,sizeof (iv)); + iv.dun =cpu_to_le64 (curr_dun); + +sg_set_page (&src, page, enc_bvec->bv_len, enc_bvec->bv_offset); +sg_set_page (&dst, ciphertext_page, enc_bvec->bv_len, + enc_bvec->bv_offset); + +skcipher_request_set_crypt (ciph_req, &src, &dst, + enc_bvec->bv_len, iv.bytes); + err =crypto_wait_req (crypto_skcipher_encrypt (ciph_req), &wait); +if (err)
block/blk-crypto.c line 332
Quoted 10 lines
block/blk-crypto.c new file +10
-0+ goto no_mem_for_ciph_page; + + curr_dun++; +continue ; +no_mem_for_ciph_page: + err = -ENOMEM ; +for (j = i -1 ; j >=0 ; j--) { +mempool_free (enc_bio->bi_io_vec->bv_page, + blk_crypto_page_pool); + }
The error path needs to free bi_io_vec[j], not bi_io_vec.
block/blk-crypto.c line 351
Quoted 4 lines
block/blk-crypto.c new file +4
-0+/* +* TODO: assumption right now is: +* each segment in bio has length == the data_unit_size +*/
This needs to be fixed, or else blk-crypto needs to reject using unsupported data unit sizes. But it seems it can be supported pretty easily by just looping through each data unit in each bio segment. To get some ideas you could look at my patches queued in fscrypt.git that handle encrypting/decrypting filesystem blocks smaller than PAGE_SIZE, e.g.
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git/commit/?id=53bc1d854c64c20d967dab15b111baca02a6d99e
Quoted 57 lines
block/blk-crypto.c new file +57
-0+static void blk_crypto_decrypt_bio (struct work_struct *w) +{ +struct work_mem *work_mem = + container_of(w, struct work_mem, crypto_work); +struct bio *bio = work_mem->bio; +int slot =bio_crypt_get_slot (bio); +struct skcipher_request *ciph_req; +DECLARE_CRYPTO_WAIT (wait); +struct bio_vec bv; +struct bvec_iter iter; +u64 curr_dun; +union { +__le64 dun; +u8 bytes[16 ]; + } iv; +struct scatterlist sg; + + curr_dun =bio_crypt_sw_data_unit_num (bio); + +kmem_cache_free (blk_crypto_work_mem_cache, work_mem); + ciph_req =skcipher_request_alloc (blk_crypto_keyslots[slot].tfm, +GFP_NOFS ); +if (!ciph_req) { + bio->bi_status =BLK_STS_RESOURCE ; + goto out; + } + +skcipher_request_set_callback (ciph_req, +CRYPTO_TFM_REQ_MAY_BACKLOG | +CRYPTO_TFM_REQ_MAY_SLEEP , + crypto_req_done, &wait); + +sg_init_table (&sg,1 ); +__bio_for_each_segment (bv, bio, iter, + bio->bi_crypt_context->crypt_iter) { +struct page *page = bv.bv_page; +int err; + +memset (&iv,0 ,sizeof (iv)); + iv.dun =cpu_to_le64 (curr_dun); + +sg_set_page (&sg, page, bv.bv_len, bv.bv_offset); +skcipher_request_set_crypt (ciph_req, &sg, &sg, + bv.bv_len, iv.bytes); + err =crypto_wait_req (crypto_skcipher_decrypt (ciph_req), &wait); +if (err) { + bio->bi_status =BLK_STS_IOERR ; + goto out; + } + curr_dun++; + } + +out: +skcipher_request_free (ciph_req); +blk_crypto_put_keyslot (bio); +bio_endio (bio); +}
block/blk-crypto.c line 418
Quoted 10 lines
block/blk-crypto.c new file +10
-0+ +static void blk_crypto_queue_decrypt_bio (struct bio *bio) +{ +struct work_mem *work_mem = +kmem_cache_zalloc (blk_crypto_work_mem_cache,GFP_ATOMIC ); + +if (!work_mem) { + bio->bi_status =BLK_STS_RESOURCE ; +return bio_endio (bio); + }
The keyslot needs to be released if allocating the work_mem fails.
However, I'm wondering: for software fallback decryption, why is the keyslot allocated before the bio is submitted, rather than in the workqueue work after the bio completes? The actual decryption is already sleepable, so why not just allocate the keyslot then too? It would also make it more similar to the software fallback encryption, which doesn't hold the keyslot during I/O.
Quoted 6 lines
block/blk-crypto.c new file +6
-0+ +INIT_WORK (&work_mem->crypto_work, blk_crypto_decrypt_bio); + work_mem->bio = bio; +queue_work (blk_crypto_wq, &work_mem->crypto_work); +} +
block/blk-crypto.c line 434
Quoted 10 lines
block/blk-crypto.c new file +10
-0+/* +* Ensures that: +* 1) The bio’s encryption context is programmed into a keyslot in the +* keyslot manager (KSM) of the request queue that the bio is being submitted +* to (or the software fallback KSM if the request queue doesn’t have a KSM), +* and that the processing_ksm in the bi_crypt_context of this bio is set to +* this KSM. +* +* 2) That the bio has a reference to this keyslot in this KSM. +*/
Make this into a proper kerneldoc comment that has a one-line function overview and documents the return value? For example:
/**
* blk_crypto_submit_bio - handle submitting bio for inline encryption *
* @bio_ptr: pointer to original bio pointer
*
* If the bio doesn't have inline encryption enabled or the submitter already
* specified a keyslot for the target device, do nothing. Else, a raw key must
* have been provided, so acquire a device keyslot for it if supported. Else,
* use the software crypto fallback.
*
* [Something about the software crypto fallback and how it may update
* *bio_ptr.]
*
* Return: 0 if bio submission should continue; nonzero if bio_endio() was
* already called so bio submission should abort.
*/
block/blk-crypto.c line 440
Quoted 6 lines
block/blk-crypto.c new file +6
-0+int blk_crypto_submit_bio (struct bio **bio_ptr) +{ +struct bio *bio = *bio_ptr; +struct request_queue *q; +int err; +enum blk_crypt_mode_num crypt_mode;
The 'crypt_mode' variable is never used.
block/blk-crypto.c line 447
Quoted 7 lines
block/blk-crypto.c new file +7
-0+struct bio_crypt_ctx *crypt_ctx; + +if (!bio_has_data (bio)) +return 0 ; + +if (!bio_is_encrypted (bio) ||bio_crypt_swhandled (bio)) +return 0 ;
Why is bio_crypt_swhandled() checked here?
Also consider reordering these checks to:
if (!bio_is_encrypted(bio) || !bio_has_data(bio)) return 0; /* comment */ if (bio_crypt_swhandled(bio)) return 0;
!bio_is_encrypted() is the most common case, so for efficiency should be checked first. !bio_is_encrypted() and !bio_has_data() are also easy to understand and kind of go together, while bio_crypt_swhandled() seems different; it's harder to understand and might need a comment.
Quoted 9 lines
block/blk-crypto.c new file +9
-0+ + crypt_ctx = bio->bi_crypt_context; + q = bio->bi_disk->queue; + crypt_mode =bio_crypt_mode (bio); + +if (bio_crypt_has_keyslot (bio)) { +/* Key already programmed into device? */ +if (q->ksm == crypt_ctx->processing_ksm) +return 0 ;
block/blk-crypto.c line 466
Quoted 10 lines
block/blk-crypto.c new file +10
-0+ +/* Nope, release the existing keyslot. */ +blk_crypto_put_keyslot (bio); + } + +/* Get device keyslot if supported */ +if (q->ksm) { + err =blk_crypto_get_keyslot (bio, q->ksm); +if (!err) +return 0 ;
Perhaps there should be a warning message here, since it may be unexpected for the software fallback encryption to be used, and it may perform poorly. E.g.
pr_warn_once("blk-crypto: failed to acquire keyslot for %s (err=%d). Falling back to software crypto.\n",
bio->bi_disk->disk_name, err);Quoted 48 lines
block/blk-crypto.c new file +48
-0+ } + +/* Fallback to software crypto */ +if (bio_data_dir (bio) ==WRITE ) { +/* Encrypt the data now */ + err =blk_crypto_encrypt_bio (bio_ptr); +if (err) + goto out_encrypt_err; + }else { + err =blk_crypto_get_keyslot (bio, blk_crypto_ksm); +if (err) + goto out_err; + } +return 0 ; +out_err: + bio->bi_status =BLK_STS_IOERR ; +out_encrypt_err: +bio_endio (bio); +return err; +} + +/* +* If the bio is not en/decrypted in software, this function releases the +* reference to the keyslot that blk_crypto_submit_bio got. +* If blk_crypto_submit_bio decided to fallback to software crypto for this +* bio, then if the bio is doing a write, we free the allocated bounce pages, +* and if the bio is doing a read, we queue the bio for decryption into a +* workqueue and return -EAGAIN. After the bio has been decrypted, we release +* the keyslot before we call bio_endio(bio). +*/ +bool blk_crypto_endio (struct bio *bio) +{ +if (!bio_crypt_has_keyslot (bio)) +return true; + +if (!bio_crypt_swhandled (bio)) { +blk_crypto_put_keyslot (bio); +return true; + } + +/* bio_data_dir(bio) == READ. So decrypt bio */ +blk_crypto_queue_decrypt_bio (bio); +return false; +} + +int __init blk_crypto_init (void ) +{ + blk_crypto_ksm = keyslot_manager_create(BLK_CRYPTO_NUM_KEYSLOTS,
block/blk-crypto.c line 524
Quoted 10 lines
block/blk-crypto.c new file +10
-0+ &blk_crypto_ksm_ll_ops, + NULL); +if (!blk_crypto_ksm) + goto out_ksm; + + blk_crypto_wq =alloc_workqueue ("blk_crypto_wq" , +WQ_UNBOUND |WQ_HIGHPRI , +num_online_cpus ()); +if (!blk_crypto_wq) + goto out_wq;
WQ_MEM_RECLAIM might be needed here.
- Eric
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
RFC PATCH v2 4/8 scsi: ufs: UFS driver v2.1 spec crypto additions
In reply to Satya TangiralaAdd the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI specification in preparation to add support for inline encryption to UFS.
--- drivers/scsi/ufs/ufshcd.h | 5 +++ drivers/scsi/ufs/ufshci.h | 67 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 70 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.h +5 -0
@@ -692,6 +692,11 @@ struct ufs_hba {* the performance of ongoing read/write operations. */ #define UFSHCD_CAP_KEEP_AUTO_BKOPS_ENABLED_EXCEPT_SUSPEND (1 << 5) +/* +* This capability allows the host controller driver to use the +* inline crypto engine, if it is present +*/ +#define UFSHCD_CAP_CRYPTO (1 << 6)struct devfreq *devfreq;struct ufs_clk_scaling clk_scaling;
drivers/scsi/ufs/ufshci.h +65 -2
106 lines not on this page · raw message
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
RFC PATCH v2 5/8 scsi: ufs: UFS crypto API
In reply to Satya TangiralaIntroduce functions to manipulate UFS inline encryption hardware in line with the JEDEC UFSHCI v2.1 specification and to work with the block keyslot manager.
--- drivers/scsi/ufs/Kconfig | 10 + drivers/scsi/ufs/Makefile | 1 + drivers/scsi/ufs/ufshcd-crypto.c | 438 +++++++++++++++++++++++++++++++ drivers/scsi/ufs/ufshcd-crypto.h | 69 +++++ 4 files changed, 518 insertions(+) create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
drivers/scsi/ufs/Kconfig +10 -0
@@ -150,3 +150,13 @@ config SCSI_UFS_BSG Select this if you need a bsg device node for your UFS controller. If unsure, say N. + +config SCSI_UFS_CRYPTO + bool "UFS Crypto Engine Support" + depends on SCSI_UFSHCD && BLK_INLINE_ENCRYPTION + help + Enable Crypto Engine Support in UFS. + Enabling this makes it possible for the kernel to use the crypto + capabilities of the UFS device (if present) to perform crypto + operations on data being transferred to/from the device. +
drivers/scsi/ufs/Makefile +1 -0
@@ -11,3 +11,4 @@ obj-$(CONFIG_SCSI_UFSHCD_PCI) += ufshcd-pci.o obj-$(CONFIG_SCSI_UFSHCD_PLATFORM) += ufshcd-pltfrm.o obj-$(CONFIG_SCSI_UFS_HISI) += ufs-hisi.o obj-$(CONFIG_SCSI_UFS_MEDIATEK) += ufs-mediatek.o +ufshcd-core-$(CONFIG_SCSI_UFS_CRYPTO) += ufshcd-crypto.o
drivers/scsi/ufs/ufshcd-crypto.c new file +438 -0
438 lines not on this page · raw message
drivers/scsi/ufs/ufshcd-crypto.h new file +69 -0
69 lines not on this page · raw message
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Eric Biggers
maintainerRe: [RFC PATCH v2 5/8] scsi: ufs: UFS crypto API
Hi Satya,
Quoted 5 lines · Satya Tangirala
Introduce functions to manipulate UFS inline encryption hardware in line with the JEDEC UFSHCI v2.1 specification and to work with the block keyslot manager.
Signed-off-by Satya Tangirala <[email protected]>
Quoted 9 lines
--- drivers/scsi/ufs/Kconfig | 10 + drivers/scsi/ufs/Makefile | 1 + drivers/scsi/ufs/ufshcd-crypto.c | 438 +++++++++++++++++++++++++++++++ drivers/scsi/ufs/ufshcd-crypto.h | 69 +++++ 4 files changed, 518 insertions(+) create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
There is a build error after this patch because it adds code that uses the crypto fields in struct ufs_hba, but those aren't added until the next patch.
It needs to be possible to compile a working kernel after each patch. Otherwise it breaks bisection.
So, perhaps add the fields in this patch instead.
Quoted 27 lines
+++
b/drivers/scsi/ufs/ufshcd-crypto.cdrivers/scsi/ufs/ufshcd-crypto.c new file +25
-0@@ -0,0 +1,438 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright 2019 Google LLC +*/ + +#include <crypto/algapi.h> + +#include "ufshcd.h" +#include "ufshcd-crypto.h" + +bool ufshcd_hba_is_crypto_supported (struct ufs_hba *hba) +{ +return hba->crypto_capabilities.reg_val !=0 ; +} + +bool ufshcd_is_crypto_enabled (struct ufs_hba *hba) +{ +return hba->caps &UFSHCD_CAP_CRYPTO ; +} + +static bool ufshcd_cap_idx_valid (struct ufs_hba *hba,unsigned int cap_idx) +{ +return cap_idx < hba->crypto_capabilities.num_crypto_cap; +} +
drivers/scsi/ufs/ufshcd-crypto.c line 35
Quoted 10 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +10
-0+#define NUM_KEYSLOTS (hba) (hba->crypto_capabilities.config_count + 1) + +bool ufshcd_keyslot_valid (struct ufs_hba *hba,unsigned int slot) +{ +/* +* The actual number of configurations supported is (CFGC+1), so slot +* numbers range from 0 to config_count inclusive. +*/ +return slot <NUM_KEYSLOTS (hba); +}
Since ufshcd_hba_is_crypto_supported(), ufshcd_is_crypto_enabled(), and ufshcd_keyslot_valid() are one-liners, don't access any private structures, and are used outside this file including on the command submission path, how about making them inline functions in ufshcd-crypto.h?
drivers/scsi/ufs/ufshcd-crypto.c line 60
Quoted 5 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +5
-0+ +static int ufshcd_crypto_alg_find (void *hba_p, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{
Now that the concept of "crypto alg IDs" is gone, rename this to ufshcd_crypto_cap_find() and rename the crypto_alg_id variables to cap_idx.
This would make it consistent with using cap_idx elsewhere in the code and avoid confusion with ufs_crypto_cap_entry::algorithm_id.
Quoted 16 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +16
-0+ +static int ufshcd_crypto_keyslot_program (void *hba_p,const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size, +unsigned int slot) +{ +struct ufs_hba *hba = hba_p; +int err =0 ; +u8 data_unit_mask; +union ufs_crypto_cfg_entry cfg; +union ufs_crypto_cfg_entry *cfg_arr = hba->crypto_cfgs; +int crypto_alg_id; + + crypto_alg_id =ufshcd_crypto_alg_find (hba_p, crypt_mode, + data_unit_size); +
drivers/scsi/ufs/ufshcd-crypto.c line 204
Quoted 10 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +10
-0+if (!ufshcd_is_crypto_enabled (hba) || + !ufshcd_keyslot_valid (hba, slot) || + !ufshcd_cap_idx_valid (hba, crypto_alg_id)) +return -EINVAL ; + + data_unit_mask =get_data_unit_size_mask (data_unit_size); + +if (!(data_unit_mask & + hba->crypto_cap_array[crypto_alg_id].sdus_mask)) +return -EINVAL ;
Nit: the 'if' expression with data_unit_mask fits on one line.
Quoted 15 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +15
-0+static int ufshcd_crypto_keyslot_find (void *hba_p, +const u8 *key, +enum blk_crypt_mode_num crypt_mode, +unsigned int data_unit_size) +{ +struct ufs_hba *hba = hba_p; +int err =0 ; +int slot; +u8 data_unit_mask; +union ufs_crypto_cfg_entry cfg; +union ufs_crypto_cfg_entry *cfg_arr = hba->crypto_cfgs; +int crypto_alg_id; + + crypto_alg_id =ufshcd_crypto_alg_find (hba_p, crypt_mode, + data_unit_size);
drivers/scsi/ufs/ufshcd-crypto.c line 248
Quoted 10 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +10
-0+ +if (!ufshcd_is_crypto_enabled (hba) || + !ufshcd_cap_idx_valid (hba, crypto_alg_id)) +return -EINVAL ; + + data_unit_mask =get_data_unit_size_mask (data_unit_size); + +if (!(data_unit_mask & + hba->crypto_cap_array[crypto_alg_id].sdus_mask)) +return -EINVAL ;
Same here.
drivers/scsi/ufs/ufshcd-crypto.c line 267
Quoted 11 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +11
-0+for (slot =0 ; slot <NUM_KEYSLOTS (hba); slot++) { +if ((cfg_arr[slot].config_enable & +UFS_CRYPTO_CONFIGURATION_ENABLE ) && + data_unit_mask == cfg_arr[slot].data_unit_size && + crypto_alg_id == cfg_arr[slot].crypto_cap_idx && +crypto_memneq (&cfg.crypto_key, cfg_arr[slot].crypto_key, +UFS_CRYPTO_KEY_MAX_SIZE ) ==0 ) { +memzero_explicit (&cfg,sizeof (cfg)); +return slot; + } + }
Nit: as I've mentioned before, I think !crypto_memneq() is easier to read than 'crypto_memneq() == 0'.
Quoted 9 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +9
-0+ hba->crypto_cap_array = +devm_kcalloc (hba->dev, + hba->crypto_capabilities.num_crypto_cap, +sizeof (hba->crypto_cap_array[0 ]), +GFP_KERNEL ); +if (!hba->crypto_cap_array) { + err = -ENOMEM ; + goto out; + }
drivers/scsi/ufs/ufshcd-crypto.c line 378
Quoted 10 lines
drivers/scsi/ufs/ufshcd-crypto.c new file +10
-0+ + hba->crypto_cfgs = +devm_kcalloc (hba->dev, + hba->crypto_capabilities.config_count +1 , +sizeof (union ufs_crypto_cfg_entry ), +GFP_KERNEL ); +if (!hba->crypto_cfgs) { + err = -ENOMEM ; + goto out_cfg_mem; + }
Nit: use 'sizeof(hba->crypto_cfgs[0])' rather than 'sizeof(union ufs_crypto_cfg_entry)', for consistency with the other array allocation.
Thanks,
- Eric
RFC PATCH v2 6/8 scsi: ufs: Add inline encryption support to UFS
In reply to Satya TangiralaWire up ufshcd.c with the UFS Crypto API, the block layer inline encryption additions and the keyslot manager.
--- drivers/scsi/ufs/ufshcd.c | 84 ++++++++++++++++++++++++++++++++++++--- drivers/scsi/ufs/ufshcd.h | 18 +++++++++ 2 files changed, 97 insertions(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +79 -5
@@ -47,6 +47,7 @@#include "unipro.h" #include "ufs-sysfs.h" #include "ufs_bsg.h" +#include "ufshcd-crypto.h" #define CREATE_TRACE_POINTS #include <trace/events/ufs.h>
@@ -855,7 +856,14 @@ static void ufshcd_enable_run_stop_reg(struct ufs_hba *hba)*/ static inline void ufshcd_hba_start (struct ufs_hba *hba) { -ufshcd_writel (hba,CONTROLLER_ENABLE ,REG_CONTROLLER_ENABLE ); +u32 val =CONTROLLER_ENABLE ; + +if (ufshcd_hba_is_crypto_supported (hba)) { +ufshcd_crypto_enable (hba); + val |=CRYPTO_GENERAL_ENABLE ; + } + +ufshcd_writel (hba, val,REG_CONTROLLER_ENABLE ); }/**
@@ -2208,9 +2216,21 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp, dword_0 |=UTP_REQ_DESC_INT_CMD ;/* Transfer request descriptor header fields */ +if (lrbp->crypto_enable) { + dword_0 |=UTP_REQ_DESC_CRYPTO_ENABLE_CMD ; + dword_0 |= lrbp->crypto_key_slot; + req_desc->header.dword_1 = +cpu_to_le32 ((u32 )lrbp->data_unit_num); + req_desc->header.dword_3 = +cpu_to_le32 ((u32)(lrbp->data_unit_num >>32 )); + }else { +/* dword_1 and dword_3 are reserved, hence they are set to 0 */ + req_desc->header.dword_1 =0 ; + req_desc->header.dword_3 =0 ; + } + req_desc->header.dword_0 =cpu_to_le32 (dword_0); -/* dword_1 is reserved, hence it is set to 0 */ - req_desc->header.dword_1 =0 ; +/* * assigning invalid value for command status. Controller * updates OCS on command completion, with the command
@@ -2218,8 +2238,6 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,*/ req_desc->header.dword_2 =cpu_to_le32 (OCS_INVALID_COMMAND_STATUS ); -/* dword_3 is reserved, hence it is set to 0 */ - req_desc->header.dword_3 =0 ; req_desc->prd_table_length =0 ; }
@@ -2379,6 +2397,37 @@ static inline u16 ufshcd_upiu_wlun_to_scsi_wlun(u8 upiu_wlun_id)return (upiu_wlun_id & ~UFS_UPIU_WLUN_ID ) |SCSI_W_LUN_BASE ; } +static inline int ufshcd_prepare_lrbp_crypto (struct ufs_hba *hba, +struct scsi_cmnd *cmd, +struct ufshcd_lrb *lrbp) +{ +int key_slot; + +if (!bio_crypt_should_process (cmd->request->bio, + cmd->request->q)) { + lrbp->crypto_enable = false; +return 0 ; + } + +if (WARN_ON (!ufshcd_is_crypto_enabled (hba))) { +/* +* Upper layer asked us to do inline encryption +* but that isn't enabled, so we fail this request. +*/ +return -EINVAL ; + } + key_slot =bio_crypt_get_slot (cmd->request->bio); +if (!ufshcd_keyslot_valid (hba, key_slot)) +return -EINVAL ; + + lrbp->crypto_enable = true; + lrbp->crypto_key_slot = key_slot; + lrbp->data_unit_num =bio_crypt_data_unit_num (cmd->request->bio); + +return 0 ; +} + +/** * ufshcd_queuecommand - main entry point for SCSI requests * @host: SCSI host pointer
@@ -2466,6 +2515,13 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) lrbp->task_tag = tag; lrbp->lun =ufshcd_scsi_to_upiu_lun (cmd->device->lun); lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed (hba) ? true : false; + + err =ufshcd_prepare_lrbp_crypto (hba, cmd, lrbp); +if (err) { + lrbp->cmd =NULL ; +clear_bit_unlock (tag, &hba->lrb_in_use); + goto out; + } lrbp->req_abort_skip = false;ufshcd_comp_scsi_upiu (hba, lrbp);
@@ -2499,6 +2555,7 @@ static int ufshcd_compose_dev_cmd(struct ufs_hba *hba, lrbp->task_tag = tag; lrbp->lun =0 ;/* device management cmd is not specific to any LUN */ lrbp->intr_cmd = true;/* No interrupt aggregation */ + lrbp->crypto_enable = false;/* No crypto operations */ hba->dev_cmd.type = cmd_type;return ufshcd_comp_devman_upiu (hba, lrbp);
@@ -4191,6 +4248,8 @@ static inline void ufshcd_hba_stop(struct ufs_hba *hba, bool can_sleep) {int err; +ufshcd_crypto_disable (hba); +ufshcd_writel (hba,CONTROLLER_DISABLE ,REG_CONTROLLER_ENABLE ); err = ufshcd_wait_for_register(hba, REG_CONTROLLER_ENABLE, CONTROLLER_ENABLE, CONTROLLER_DISABLE,
@@ -4584,10 +4643,13 @@ static int ufshcd_change_queue_depth(struct scsi_device *sdev, int depth)static int ufshcd_slave_configure (struct scsi_device *sdev) {struct request_queue *q = sdev->request_queue; +struct ufs_hba *hba =shost_priv (sdev->host);blk_queue_update_dma_pad (q,PRDT_DATA_BYTE_COUNT_PAD -1 );blk_queue_max_segment_size (q,PRDT_DATA_BYTE_COUNT_MAX ); +ufshcd_crypto_setup_rq_keyslot_manager (hba, q); +return 0 ; }
@@ -4598,6 +4660,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)static void ufshcd_slave_destroy (struct scsi_device *sdev) {struct ufs_hba *hba; +struct request_queue *q = sdev->request_queue; hba =shost_priv (sdev->host);/* Drop the reference as it won't be needed anymore */
@@ -4608,6 +4671,8 @@ static void ufshcd_slave_destroy(struct scsi_device *sdev) hba->sdev_ufs_device =NULL ;spin_unlock_irqrestore (hba->host->host_lock, flags); } + +ufshcd_crypto_destroy_rq_keyslot_manager (q); }/**
@@ -4723,6 +4788,8 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)case OCS_MISMATCH_RESP_UPIU_SIZE :case OCS_PEER_COMM_FAILURE :case OCS_FATAL_ERROR : +case OCS_INVALID_CRYPTO_CONFIG : +case OCS_GENERAL_CRYPTO_ERROR :default : result |=DID_ERROR <<16 ; dev_err(hba->dev,
@@ -8290,6 +8357,13 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq) goto exit_gating; } +/* Init crypto */ + err =ufshcd_hba_init_crypto (hba); +if (err) { +dev_err (hba->dev,"crypto setup failed\n" ); + goto out_remove_scsi_host; + } +/* Host controller enable */ err =ufshcd_hba_enable (hba);if (err) {
drivers/scsi/ufs/ufshcd.h +18 -0
@@ -167,6 +167,9 @@ struct ufs_pm_lvl_states {* @intr_cmd: Interrupt command (doesn't participate in interrupt aggregation) * @issue_time_stamp: time stamp for debug purposes * @compl_time_stamp: time stamp for statistics +* @crypto_enable: whether or not the request needs inline crypto operations +* @crypto_key_slot: the key slot to use for inline crypto +* @data_unit_num: the data unit number for the first block for inline crypto * @req_abort_skip: skip request abort task flag */ struct ufshcd_lrb {
@@ -191,6 +194,9 @@ struct ufshcd_lrb {bool intr_cmd;ktime_t issue_time_stamp;ktime_t compl_time_stamp; +bool crypto_enable; +u8 crypto_key_slot; +u64 data_unit_num;bool req_abort_skip; };
@@ -501,6 +507,10 @@ struct ufs_stats {* @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for * device is known or not. * @scsi_block_reqs_cnt: reference counting for scsi block requests +* @crypto_capabilities: Content of crypto capabilities register (0x100) +* @crypto_cap_array: Array of crypto capabilities +* @crypto_cfg_register: Start of the crypto cfg array +* @crypto_cfgs: Array of crypto configurations (i.e. config for each slot) */ struct ufs_hba {void __iomem *mmio_base;
@@ -711,6 +721,14 @@ struct ufs_hba {struct device bsg_dev;struct request_queue *bsg_queue; + +#ifdef CONFIG_SCSI_UFS_CRYPTO +/* crypto */ +union ufs_crypto_capabilities crypto_capabilities; +union ufs_crypto_cap_entry *crypto_cap_array; +u32 crypto_cfg_register; +union ufs_crypto_cfg_entry *crypto_cfgs; +#endif /* CONFIG_SCSI_UFS_CRYPTO */ };/* Returns true if clocks can be gated. Otherwise false */
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Eric Biggers
maintainerRe: [RFC PATCH v2 6/8] scsi: ufs: Add inline encryption support to UFS
drivers/scsi/ufs/ufshcd.c static inline u16 ufshcd_upiu_wlun_to_scsi_wlun(u8 upiu_wlun_id) · line 2410
Quoted 11 lines · Satya Tangirala
drivers/scsi/ufs/ufshcd.c +11
-0+static inline int ufshcd_prepare_lrbp_crypto (struct ufs_hba *hba, +struct scsi_cmnd *cmd, +struct ufshcd_lrb *lrbp) +{ +int key_slot; + +if (!bio_crypt_should_process (cmd->request->bio, + cmd->request->q)) { + lrbp->crypto_enable = false; +return 0 ; + }
Nit: this 'if' expression fits on one line.
Quoted 22 lines
static int
ufshcd_slave_configure(struct scsi_device *sdev){drivers/scsi/ufs/ufshcd.c +4
-0struct request_queue *q = sdev->request_queue; +struct ufs_hba *hba =shost_priv (sdev->host);blk_queue_update_dma_pad (q,PRDT_DATA_BYTE_COUNT_PAD -1 );blk_queue_max_segment_size (q,PRDT_DATA_BYTE_COUNT_MAX ); +ufshcd_crypto_setup_rq_keyslot_manager (hba, q); +return 0 ; }@@ -4598,6 +4660,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)static void ufshcd_slave_destroy (struct scsi_device *sdev) {struct ufs_hba *hba; +struct request_queue *q = sdev->request_queue; hba =shost_priv (sdev->host);/* Drop the reference as it won't be needed anymore */
drivers/scsi/ufs/ufshcd.c static void ufshcd_slave_destroy(struct scsi_device *sdev) · line 4676
Quoted 6 lines
drivers/scsi/ufs/ufshcd.c +2
-0@@ -4608,6 +4671,8 @@ static void ufshcd_slave_destroy(struct scsi_device *sdev) hba->sdev_ufs_device =NULL ;spin_unlock_irqrestore (hba->host->host_lock, flags); } + +ufshcd_crypto_destroy_rq_keyslot_manager (q); }
Each scsi_device is still getting its own keyslot manager. As discussed before, this is wrong because the keyslots are per-host controller, not per-device.
So the keyslot manager needs to be a property of the ufs_hba instead, and each device's request_queue needs to reference that same keyslot manager.
Quoted 15 lines
drivers/scsi/ufs/ufshcd.h +3
-0@@ -167,6 +167,9 @@ struct ufs_pm_lvl_states {* @intr_cmd: Interrupt command (doesn't participate in interrupt aggregation) * @issue_time_stamp: time stamp for debug purposes * @compl_time_stamp: time stamp for statistics +* @crypto_enable: whether or not the request needs inline crypto operations +* @crypto_key_slot: the key slot to use for inline crypto +* @data_unit_num: the data unit number for the first block for inline crypto * @req_abort_skip: skip request abort task flag */ struct ufshcd_lrb {
drivers/scsi/ufs/ufshcd.h struct ufshcd_lrb { · line 199
Quoted 6 lines
drivers/scsi/ufs/ufshcd.h +3
-0@@ -191,6 +194,9 @@ struct ufshcd_lrb {bool intr_cmd;ktime_t issue_time_stamp;ktime_t compl_time_stamp; +bool crypto_enable; +u8 crypto_key_slot; +u64 data_unit_num;
Maybe these fields should be conditional on CONFIG_SCSI_UFS_CRYPTO too?
Quoted 32 lines
drivers/scsi/ufs/ufshcd.h +12
-0bool req_abort_skip; };@@ -501,6 +507,10 @@ struct ufs_stats {* @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for * device is known or not. * @scsi_block_reqs_cnt: reference counting for scsi block requests +* @crypto_capabilities: Content of crypto capabilities register (0x100) +* @crypto_cap_array: Array of crypto capabilities +* @crypto_cfg_register: Start of the crypto cfg array +* @crypto_cfgs: Array of crypto configurations (i.e. config for each slot) */ struct ufs_hba {void __iomem *mmio_base;@@ -711,6 +721,14 @@ struct ufs_hba {struct device bsg_dev;struct request_queue *bsg_queue; + +#ifdef CONFIG_SCSI_UFS_CRYPTO +/* crypto */ +union ufs_crypto_capabilities crypto_capabilities; +union ufs_crypto_cap_entry *crypto_cap_array; +u32 crypto_cfg_register; +union ufs_crypto_cfg_entry *crypto_cfgs; +#endif /* CONFIG_SCSI_UFS_CRYPTO */ };/* Returns true if clocks can be gated. Otherwise false */ Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
- Eric
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
RFC PATCH v2 7/8 fscrypt: wire up fscrypt to use blk-crypto
In reply to Satya TangiralaIntroduce fscrypt_set_bio_crypt_ctx for filesystems to call to set up encryption contexts in bios, and fscrypt_evict_crypt_key to evict the encryption context associated with an inode.
Inline encryption is controlled by a policy flag in the fscrypt_info in the inode, and filesystems may check if an inode should use inline encryption by calling fscrypt_inode_is_hw_encrypted. Files can be marked as inline encrypted from userspace by appropriately modifying the flags (OR-ing FS_POLICY_FLAGS_HW_ENCRYPTION to it) in the fscrypt_policy passed to fscrypt_ioctl_set_policy.
To test inline encryption with the fscrypt dummy context, add ctx.flags |= FS_POLICY_FLAGS_HW_ENCRYPTION
when setting up the dummy context in fs/crypto/keyinfo.c.
Note that blk-crypto will fall back to software en/decryption in the absence of inline crypto hardware, so setting up the ctx.flags in the dummy context without inline crypto hardware serves as a test for the software fallback in blk-crypto.
--- block/blk-crypto.c | 1 - fs/crypto/Kconfig | 7 ++ fs/crypto/bio.c | 159 +++++++++++++++++++++++++++++++----- fs/crypto/crypto.c | 9 ++ fs/crypto/fscrypt_private.h | 10 +++ fs/crypto/keyinfo.c | 69 +++++++++++----- fs/crypto/policy.c | 10 +++ include/linux/fscrypt.h | 64 +++++++++++++++ include/uapi/linux/fs.h | 12 ++- 9 files changed, 296 insertions(+), 45 deletions(-)
block/blk-crypto.c +0 -1
@@ -82,7 +82,6 @@ static int blk_crypto_keyslot_program(void *priv, const u8 *key, slotp->tfm = tfm; } - err =crypto_skcipher_setkey (tfm, key, keysize);if (err) {
fs/crypto/Kconfig +7 -0
@@ -15,3 +15,10 @@ config FS_ENCRYPTION efficient since it avoids caching the encrypted and decrypted pages in the page cache. Currently Ext4, F2FS and UBIFS make use of this feature. + +config FS_ENCRYPTION_HW_CRYPT + tristate "Enable fscrypt to use inline crypto" + default n + depends on FS_ENCRYPTION && BLK_INLINE_ENCRYPTION + help + Enables fscrypt to use inline crypto hardware if available.
fs/crypto/bio.c +137 -22
227 lines not on this page · raw message
fs/crypto/crypto.c +9 -0
@@ -240,6 +240,11 @@ struct page *fscrypt_encrypt_page(const struct inode *inode,BUG_ON (len %FS_CRYPTO_BLOCK_SIZE !=0 ); +/* If HW encryption, then pretend we did in place encryption */ +if (fscrypt_inode_is_hw_encrypted (inode)) { +return ciphertext_page; + } +if (inode->i_sb->s_cop->flags &FS_CFLG_OWN_PAGES ) {/* with inplace-encryption we just encrypt the page */ err = fscrypt_do_page_crypto(inode, FS_ENCRYPT, lblk_num, page,
@@ -302,6 +307,10 @@ int fscrypt_decrypt_page(const struct inode *inode, struct page *page,if (!(inode->i_sb->s_cop->flags &FS_CFLG_OWN_PAGES ))BUG_ON (!PageLocked (page)); +/* If we have HW encryption, then this page is already decrypted */ +if (fscrypt_inode_is_hw_encrypted (inode)) +return 0 ; +return fscrypt_do_page_crypto (inode,FS_DECRYPT , lblk_num, page, page, len, offs,GFP_NOFS ); }
fs/crypto/fscrypt_private.h +10 -0
@@ -49,6 +49,16 @@ struct fscrypt_symlink_data {char encrypted_path[1 ]; } __packed; +/* Master key referenced by FS_POLICY_FLAG_DIRECT_KEY policy */ +struct fscrypt_master_key { +struct hlist_node mk_node; +refcount_t mk_refcount; +const struct fscrypt_mode *mk_mode; +struct crypto_skcipher *mk_ctfm; +u8 mk_descriptor[FS_KEY_DESCRIPTOR_SIZE ]; +u8 mk_raw[FS_MAX_KEY_SIZE ]; +}; +/* * fscrypt_info - the "encryption key" for an inode *
fs/crypto/keyinfo.c +48 -21
137 lines not on this page · raw message
fs/crypto/policy.c +10 -0
@@ -36,6 +36,7 @@ static int create_encryption_context_from_policy(struct inode *inode,struct fscrypt_context ctx; ctx.format =FS_ENCRYPTION_CONTEXT_FORMAT_V1 ; +memcpy (ctx.master_key_descriptor, policy->master_key_descriptor,FS_KEY_DESCRIPTOR_SIZE );
@@ -46,8 +47,17 @@ static int create_encryption_context_from_policy(struct inode *inode,if (policy->flags & ~FS_POLICY_FLAGS_VALID )return -EINVAL ; +/* +* TODO: expose HW encryption via some toggleable knob +* instead of as a policy? +*/ +if (!inode->i_sb->s_cop->hw_crypt_supp && + (policy->flags &FS_POLICY_FLAGS_HW_ENCRYPTION )) +return -EINVAL ; + ctx.contents_encryption_mode = policy->contents_encryption_mode; ctx.filenames_encryption_mode = policy->filenames_encryption_mode; + ctx.flags = policy->flags;BUILD_BUG_ON (sizeof (ctx.nonce) !=FS_KEY_DERIVATION_NONCE_SIZE );get_random_bytes (ctx.nonce,FS_KEY_DERIVATION_NONCE_SIZE );
include/linux/fscrypt.h +64 -0
94 lines not on this page · raw message
include/uapi/linux/fs.h +11 -1
@@ -224,7 +224,17 @@ struct fsxattr {#define FS_POLICY_FLAGS_PAD_32 0x03#define FS_POLICY_FLAGS_PAD_MASK 0x03#define FS_POLICY_FLAG_DIRECT_KEY 0x04/* use master key directly */ -#define FS_POLICY_FLAGS_VALID 0x07 +#define FS_POLICY_FLAGS_VALID_BASE 0x07 + +#if IS_ENABLED (CONFIG_FS_ENCRYPTION_HW_CRYPT ) +#define FS_POLICY_FLAGS_HW_ENCRYPTION 0x08 +#else +#define FS_POLICY_FLAGS_HW_ENCRYPTION 0x00 +#endif + + +#define FS_POLICY_FLAGS_VALID (FS_POLICY_FLAGS_VALID_BASE | \ + FS_POLICY_FLAGS_HW_ENCRYPTION)/* Encryption algorithms */ #define FS_ENCRYPTION_MODE_INVALID 0
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Eric Biggers
maintainerRe: [RFC PATCH v2 7/8] fscrypt: wire up fscrypt to use blk-crypto
Hi Satya,
Quoted 15 lines · Satya Tangirala
Introduce
fscrypt_set_bio_crypt_ctxfor filesystems to call to set up encryption contexts in bios, andfscrypt_evict_crypt_keyto evict the encryption context associated with an inode.Inline encryption is controlled by a policy flag in the
fscrypt_infoin the inode, and filesystems may check if an inode should use inline encryption by callingfscrypt_inode_is_hw_encrypted. Files can be marked as inline encrypted from userspace by appropriately modifying the flags (OR-ingFS_POLICY_FLAGS_HW_ENCRYPTIONto it) in thefscrypt_policypassed tofscrypt_ioctl_set_policy.To test inline encryption with the fscrypt dummy context, add ctx.flags |=
FS_POLICY_FLAGS_HW_ENCRYPTION
when setting up the dummy context infs/crypto/keyinfo.c.
Quoted 6 lines
Note that blk-crypto will fall back to software en/decryption in the absence of inline crypto hardware, so setting up the ctx.flags in the dummy context without inline crypto hardware serves as a test for the software fallback in blk-crypto.
Signed-off-by Satya Tangirala <[email protected]>
I think a fscrypt_policy flag is the right approach, but the "HW_ENCRYPTION" name is really confusing and wrong.
What it really enables is a cryptosystem and on-disk format change where, for the purpose of working better with inline encryption, file contents are encrypted with the master key directly (or for v2 encryption policies it will be a per-mode derived key as it really should be, once we can actually get the v2 encryption policy support reviewed and merged), and the inode numbers are added to the IVs. As we know, when ext4 support is added, this will also preclude the filesystem from being resized.
It's not necessarily "hardware encryption". You're implementing it using the block layer encryption, but that can fallback to the crypto API in blk-crypto.
Moreover fscrypt already supports hardware encryption via the crypto API, just not *inline* hardware encryption.
So calling it "hardware encryption" is wrong.
I think a much better name would be something like FS_POLICY_FLAG_DIRECT_CONTENTS or FS_POLICY_FLAG_INLINECRYPT_OPTIMIZED.
Similarly for everywhere else in this patch that references "hardware encryption" -- usually it should be "inline encryption".
block/blk-crypto.c static int blk_crypto_keyslot_program(void *priv, const u8 *key, · line 87
Quoted 12 lines
block/blk-crypto.c +0
-1@@ -82,7 +82,6 @@ static int blk_crypto_keyslot_program(void *priv, const u8 *key, slotp->tfm = tfm; } - err =crypto_skcipher_setkey (tfm, key, keysize);if (err) {
This should be folded into an earlier patch.
Quoted 5 lines
fs/crypto/Kconfig +0
-0
fs/crypto/Kconfig config FS_ENCRYPTION · line 24
Quoted 10 lines
fs/crypto/Kconfig +7
-0@@ -15,3 +15,10 @@ config FS_ENCRYPTION efficient since it avoids caching the encrypted and decrypted pages in the page cache. Currently Ext4, F2FS and UBIFS make use of this feature. + +config FS_ENCRYPTION_HW_CRYPT + tristate "Enable fscrypt to use inline crypto" + default n + depends on FS_ENCRYPTION && BLK_INLINE_ENCRYPTION + help + Enables fscrypt to use inline crypto hardware if available.
"Enable fscrypt to use inline crypto" isn't a loadable module, so it needs to be a bool, not a tristate.
That also means use '#ifdef CONFIG_...', not '#if IS_ENABLED(CONFIG...)'.
Also no need for 'default n', since 'n' is already the default.
Quoted 62 lines
fs/crypto/bio.c +19
-7@@ -24,17 +24,24 @@#include <linux/module.h> #include <linux/bio.h> #include <linux/namei.h> +#include <linux/keyslot-manager.h> +#include <linux/blkdev.h> +#include <crypto/algapi.h> #include "fscrypt_private.h" -static void __fscrypt_decrypt_bio (struct bio *bio,bool done) +static void __fscrypt_decrypt_bio (struct bio *bio,bool done,bool decrypt) {struct bio_vec *bv;struct bvec_iter_all iter_all;bio_for_each_segment_all (bv, bio, iter_all) {struct page *page = bv->bv_page; -int ret =fscrypt_decrypt_page (page->mapping->host, page, -PAGE_SIZE ,0 , page->index); +int ret =0 ; + +if (decrypt) { + ret =fscrypt_decrypt_page (page->mapping->host, page, +PAGE_SIZE ,0 , page->index); + }if (ret)SetPageError (page);@@ -47,7 +54,7 @@ static void __fscrypt_decrypt_bio(struct bio *bio, bool done)void fscrypt_decrypt_bio (struct bio *bio) { -__fscrypt_decrypt_bio (bio, false); +__fscrypt_decrypt_bio (bio, false, true); }EXPORT_SYMBOL (fscrypt_decrypt_bio);@@ -57,16 +64,27 @@ static void completion_pages(struct work_struct *work) container_of(work, struct fscrypt_ctx, r.work);struct bio *bio = ctx->r.bio; -__fscrypt_decrypt_bio (bio, true); +__fscrypt_decrypt_bio (bio, true, true); +fscrypt_release_ctx (ctx); +bio_put (bio); +} + +static void decrypt_bio_hwcrypt (struct fscrypt_ctx *ctx,struct bio *bio) +{ +__fscrypt_decrypt_bio (bio, true, false);fscrypt_release_ctx (ctx);bio_put (bio); }void fscrypt_enqueue_decrypt_bio (struct fscrypt_ctx *ctx,struct bio *bio) { -INIT_WORK (&ctx->r.work, completion_pages); - ctx->r.bio = bio;
fs/crypto/bio.c static void completion_pages(struct work_struct *work) · line 89
Quoted 10 lines
fs/crypto/bio.c +7
-1-fscrypt_enqueue_decrypt_work (&ctx->r.work); +if (bio_is_encrypted (bio)) { +decrypt_bio_hwcrypt (ctx, bio); + }else { +INIT_WORK (&ctx->r.work, completion_pages); + ctx->r.bio = bio; +fscrypt_enqueue_decrypt_work (&ctx->r.work); + } }EXPORT_SYMBOL (fscrypt_enqueue_decrypt_bio);
I don't think we should be repurposing the normal fscrypt decryption path like this. In the case of inline decryption, the decryption *already happened*, so there's no need for the filesystem to call into fs/crypto/ after the I/O completes. It's also inefficient since there is a fscrypt_ctx being allocated for every bio, but it's useless for inline decryption since the only purpose of fscrypt_ctx is to hold the bio decryption workqueue item.
Instead I think we should just make the filesystems' ->readpages() correctly check whether post-read decryption is needed. I.e. instead of checking...
IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode)
add a helper function...
fscrypt_needs_fs_layer_crypto()
(there may be a better name for it)
that returns true if IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode) && FS_POLICY_FLAG_INLINECRYPT_OPTIMIZED (or whatever we decide to call it) is unset in the inode's fscrypt policy.
Likewise for encryption: only call fscrypt_encrypt_page() when fscrypt_needs_fs_layer_crypto().
Quoted 5 lines
fs/crypto/bio.c +5
-0+int fscrypt_set_bio_crypt_ctx (const struct inode *inode, +struct bio *bio,u64 data_unit_num) +{ +struct fscrypt_info *ci = inode->i_crypt_info; +
fs/crypto/bio.c int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · line 210
Quoted 10 lines
fs/crypto/bio.c +10
-0+/* If inode is not hw encrypted, nothing to do. */ +if (!fscrypt_inode_is_hw_encrypted (inode)) +return 0 ; + +return bio_crypt_set_ctx (bio, ci->ci_master_key->mk_raw, +get_blk_crypto_alg_for_fscryptalg (ci->ci_data_mode), + data_unit_num, +PAGE_SHIFT ); +} +EXPORT_SYMBOL (fscrypt_set_bio_crypt_ctx);
To be ready for ext4 encryption with block_size < PAGE_SIZE this needs to pass inode->i_blkbits for the dun_bits, not inode->i_blkbits.
Quoted 37 lines
fs/crypto/keyinfo.c +18
-0@@ -25,6 +25,21 @@ static struct crypto_shash *essiv_hash_tfm; static DEFINE_HASHTABLE(fscrypt_master_keys, 6); /* 6 bits = 64 buckets */static DEFINE_SPINLOCK (fscrypt_master_keys_lock); +#if IS_ENABLED (CONFIG_FS_ENCRYPTION_HW_CRYPT ) +static inline bool __flags_hw_encrypted (u8 flags, +const struct inode *inode) +{ +return inode && (flags &FS_POLICY_FLAGS_HW_ENCRYPTION ) && +S_ISREG (inode->i_mode); +} +#else +static inline bool __flags_hw_encrypted (u8 flags, +const struct inode *inode) +{ +return false; +} +#endif /* CONFIG_FS_ENCRYPTION_HW_CRYPT */ +/* * Key derivation function. This generates the derived key by encrypting the * master key with AES-128-ECB using the inode's nonce as the AES key. @@ -220,6 +235,9 @@ static int find_and_derive_key(const struct inode *inode,memcpy (derived_key, payload->raw, mode->keysize); err =0 ; } + }else if (__flags_hw_encrypted (ctx->flags, inode)) { +memcpy (derived_key, payload->raw, mode->keysize); + err =0 ; }else { err =derive_key_aes (payload->raw, ctx, derived_key, mode->keysize);
fs/crypto/keyinfo.c allocate_skcipher_for_mode(struct fscrypt_mode *mode, const u8 *raw_key, · line 288
Quoted 2 lines
fs/crypto/keyinfo.c +0
-0@@ -269,16 +287,6 @@ allocate_skcipher_for_mode(struct fscrypt_mode *mode, const u8 *raw_key,return ERR_PTR (err); }
As I mentioned, to follow crypto best practices this really should use a per-mode key and not the master key directly... We can do that pretty easily after the v2 encryption policy support is merged.
Quoted 48 lines
fs/crypto/keyinfo.c +13
-8-static void put_master_key (struct fscrypt_master_key *mk) +static void put_master_key (struct fscrypt_master_key *mk, +struct inode *inode) {if (!refcount_dec_and_lock (&mk->mk_refcount, &fscrypt_master_keys_lock))return ;hash_del (&mk->mk_node);spin_unlock (&fscrypt_master_keys_lock); +fscrypt_evict_crypt_key (inode);free_master_key (mk); }@@ -360,11 +370,13 @@ fscrypt_get_master_key(const struct fscrypt_info *ci, struct fscrypt_mode *mode,return ERR_PTR (-ENOMEM );refcount_set (&mk->mk_refcount,1 ); mk->mk_mode = mode; - mk->mk_ctfm =allocate_skcipher_for_mode (mode, raw_key, inode); -if (IS_ERR (mk->mk_ctfm)) { - err =PTR_ERR (mk->mk_ctfm); - mk->mk_ctfm =NULL ; - goto err_free_mk; +if (!__flags_hw_encrypted (ci->ci_flags, inode)) { + mk->mk_ctfm =allocate_skcipher_for_mode (mode, raw_key, inode); +if (IS_ERR (mk->mk_ctfm)) { + err =PTR_ERR (mk->mk_ctfm); + mk->mk_ctfm =NULL ; + goto err_free_mk; + } }memcpy (mk->mk_descriptor, ci->ci_master_key_descriptor,FS_KEY_DESCRIPTOR_SIZE );@@ -456,7 +468,8 @@ static int setup_crypto_transform(struct fscrypt_info *ci,struct crypto_skcipher *ctfm;int err; -if (ci->ci_flags &FS_POLICY_FLAG_DIRECT_KEY ) { +if ((ci->ci_flags &FS_POLICY_FLAG_DIRECT_KEY ) || +__flags_hw_encrypted (ci->ci_flags, inode)) { mk =fscrypt_get_master_key (ci, mode, raw_key, inode);if (IS_ERR (mk))return PTR_ERR (mk);@@ -486,13 +499,13 @@ static int setup_crypto_transform(struct fscrypt_info *ci,return 0 ; } -static void put_crypt_info (struct fscrypt_info *ci) +static void put_crypt_info (struct fscrypt_info *ci,struct inode *inode)
fs/crypto/keyinfo.c static int setup_crypto_transform(struct fscrypt_info *ci, · line 511
Quoted 10 lines
fs/crypto/keyinfo.c +1
-1{if (!ci)return ;if (ci->ci_master_key) { -put_master_key (ci->ci_master_key); +put_master_key (ci->ci_master_key, inode); }else {crypto_free_skcipher (ci->ci_ctfm);crypto_free_cipher (ci->ci_essiv_tfm);
Using the struct fscrypt_master_key in this way is wrong because they are identified by (ci_mode, ci_descriptor, and ci_raw_key). That means:
- The same fscrypt_master_key might be used by both inodes with and without the FS_POLICY_FLAGS_HW_ENCRYPTION (or whatever it's renamed to) flag set. This will cause a NULL dereference on ->ci_ctfm if the fscrypt_master_key was initially created for a policy with FS_POLICY_FLAGS_HW_ENCRYPTION, and then later used by a policy without FS_POLICY_FLAGS_HW_ENCRYPTION.
- The same fscrypt_master_key can be used by inodes on multiple filesystems. This patch only makes the key be evicted from the keyslots on the last device to be used.
We can fix this by extending the identifier for fscrypt_master_key to (ci_mode, ci_descriptor, ci_raw_key, super_block, ci_ctfm != NULL). So you'd get separate fscrypt_master_key's for different filesystems, and for policies with and without FS_POLICY_FLAGS_HW_ENCRYPTION.
Of course, you are screwed if you use the same master key for inline encryption on multiple filesystems anyway, since IVs will be reused. What we really should be doing is use HKDF to derive the inline encryption key from (master_key, contents_encryption_mode, filesystem_uuid). Which again, depends on v2 policy support, which hopefully I can convince people should be merged so we don't have to keep piling on these cryptographically questionable hacks :-)
Quoted 12 lines
include/uapi/linux/fs.h +4
-1@@ -224,7 +224,17 @@ struct fsxattr {#define FS_POLICY_FLAGS_PAD_32 0x03#define FS_POLICY_FLAGS_PAD_MASK 0x03#define FS_POLICY_FLAG_DIRECT_KEY 0x04/* use master key directly */ -#define FS_POLICY_FLAGS_VALID 0x07 +#define FS_POLICY_FLAGS_VALID_BASE 0x07 + +#if IS_ENABLED (CONFIG_FS_ENCRYPTION_HW_CRYPT ) +#define FS_POLICY_FLAGS_HW_ENCRYPTION 0x08
include/uapi/linux/fs.h struct fsxattr { · line 240
Quoted 10 lines
include/uapi/linux/fs.h +7
-0+#else +#define FS_POLICY_FLAGS_HW_ENCRYPTION 0x00 +#endif + + +#define FS_POLICY_FLAGS_VALID (FS_POLICY_FLAGS_VALID_BASE | \ + FS_POLICY_FLAGS_HW_ENCRYPTION)/* Encryption algorithms */ #define FS_ENCRYPTION_MODE_INVALID 0
Checking the kernel config is meaningless in UAPI headers. Everyone gets the same <linux/fs.h> header, and they can build programs with it and run them on any arbitrary kernel. So flag needs to be unconditional.
Thanks,
- Eric
Re: [RFC PATCH v2 7/8] fscrypt: wire up fscrypt to use blk-crypto
[reduced CC list, since I don't think this is interesting outside ext4]
Quoted 8 lines · Eric Biggers
What it really enables is a cryptosystem and on-disk format change where, for the purpose of working better with inline encryption, file contents are encrypted with the master key directly (or for v2 encryption policies it will be a per-mode derived key as it really should be, once we can actually get the v2 encryption policy support reviewed and merged), and the inode numbers are added to the IVs. As we know, when ext4 support is added, this will also preclude the filesystem from being resized.
Just as an aside, I thought that the inode number would *not* be added to the IV, exactly so that ext4 filesystem resize would work?
I guess it shouldn't *strictly* preventing filesystem resizing, only the case of shrinking the filesystem and having to relocate encrypted inodes. Expanding the filesystem shouldn't have that problem at all, nor should shrinking if there isn't a need to relocate the encrypted inodes. Moving encrypted blocks should be OK, since the logical block numbers (and hence derived block IV) would stay the same.
Something like https://patchwork.ozlabs.org/patch/960766/ "Add block_high_watermark sysfs tunable" would allow pre-migrating encrypted files in userspace via data copy (read/decrypt+write/encrypt) before doing the resize, if necessary, so that files do not use inode numbers that will be cut off the end of the filesystem.
Cheers, Andreas
Eric Biggers
maintainerRe: [RFC PATCH v2 7/8] fscrypt: wire up fscrypt to use blk-crypto
[+Satya and Paul]
Quoted 12 lines · Andreas Dilger
[reduced CC list, since I don't think this is interesting outside ext4]
Quoted 8 lines · Eric Biggers
What it really enables is a cryptosystem and on-disk format change where, for the purpose of working better with inline encryption, file contents are encrypted with the master key directly (or for v2 encryption policies it will be a per-mode derived key as it really should be, once we can actually get the v2 encryption policy support reviewed and merged), and the inode numbers are added to the IVs. As we know, when ext4 support is added, this will also preclude the filesystem from being resized.
Quoted 9 lines
Just as an aside, I thought that the inode number would *not* be added to the IV, exactly so that ext4 filesystem resize would work?
I guess it shouldn't *strictly* preventing filesystem resizing, only the case of shrinking the filesystem and having to relocate encrypted inodes. Expanding the filesystem shouldn't have that problem at all, nor should shrinking if there isn't a need to relocate the encrypted inodes. Moving encrypted blocks should be OK, since the logical block numbers (and hence derived block IV) would stay the same.
Yes, this is all correct. The limitation on ext4 filesystem shrinking will be a tradeoff to get inline encryption support to work efficiently. Satya hasn't implemented ext4 support yet, but as part of it I think we'll need to add an ext4 superblock feature flag that forbids filesystem shrinking.
So unless we find a better way, people who need ext4 filesystem shrinking will need to use the existing ext4 encryption format instead, which isn't optimized for inline encryption.
This isn't an issue for Android (the motivating use case for this) since the user data partition on Android devices is never shrunk.
Quoted 4 lines
Something like https://patchwork.ozlabs.org/patch/960766/ "Add
block_high_watermarksysfs tunable" would allow pre-migrating encrypted files in userspace via data copy (read/decrypt+write/encrypt) before doing the resize, if necessary, so that files do not use inode numbers that will be cut off the end of the filesystem.
If I understand that patch correctly, it only implements a high watermark for data blocks, not inode numbers? We need to ensure that an inode number is never changed without also decrypting + encrypting the data.
- Eric
RFC PATCH v2 8/8 f2fs: Wire up f2fs to use inline encryption via fscrypt
In reply to Satya Tangirala--- fs/f2fs/data.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++--- fs/f2fs/super.c | 1 + 2 files changed, 74 insertions(+), 4 deletions(-)
fs/f2fs/data.c +73 -4
@@ -142,6 +142,8 @@ static bool f2fs_bio_post_read_required(struct bio *bio)static void f2fs_read_end_io (struct bio *bio) { +fscrypt_unset_bio_crypt_ctx (bio); +if (time_to_inject (F2FS_P_SB (bio_first_page_all (bio)),FAULT_READ_IO )) {f2fs_show_injection_info (FAULT_READ_IO );
@@ -165,6 +167,8 @@ static void f2fs_write_end_io(struct bio *bio)struct bio_vec *bvec;struct bvec_iter_all iter_all; +fscrypt_unset_bio_crypt_ctx (bio); +if (time_to_inject (sbi,FAULT_WRITE_IO )) {f2fs_show_injection_info (FAULT_WRITE_IO ); bio->bi_status =BLK_STS_IOERR ;
@@ -282,9 +286,18 @@ static struct bio *__bio_alloc(struct f2fs_sb_info *sbi, block_t blk_addr,return bio; } +static inline u64 hw_crypt_dun (struct inode *inode,pgoff_t offset) +{ +return (((u64 )inode->i_ino) <<32 ) |lower_32_bits (offset); +} +static inline void __submit_bio (struct f2fs_sb_info *sbi,struct bio *bio,enum page_type type) { +struct page *page; +struct inode *inode; +int err =0 ; +if (!is_read_io (bio_op (bio))) {unsigned int start;
@@ -326,7 +339,22 @@ static inline void __submit_bio(struct f2fs_sb_info *sbi,trace_f2fs_submit_read_bio (sbi->sb, type, bio); else trace_f2fs_submit_write_bio(sbi->sb, type, bio); -submit_bio (bio); + +if (bio_has_data (bio)) { + page =bio_page (bio); +if (page && page->mapping && page->mapping->host) { + inode = page->mapping->host; + err =fscrypt_set_bio_crypt_ctx (inode, bio, +hw_crypt_dun (inode, + page->index)); + } + } +if (err) { + bio->bi_status =BLK_STS_IOERR ; +bio_endio (bio); + }else { +submit_bio (bio); + } }static void __submit_merged_bio (struct f2fs_bio_info *io)
@@ -487,6 +515,9 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)enum page_type btype =PAGE_TYPE_OF_BIO (fio->type);struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp;struct page *bio_page; +struct inode *fio_inode, *bio_inode; +struct page *first_page; +u64 next_dun =0 ;f2fs_bug_on (sbi,is_read_io (fio->op));
@@ -513,10 +544,27 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)inc_page_count (sbi,WB_DATA_TYPE (bio_page)); + fio_inode = fio->page->mapping->host; + bio_inode =NULL ; + first_page =NULL ; + next_dun =0 ; +if (io->bio &&bio_page (io->bio)->mapping) { + first_page =bio_page (io->bio); + bio_inode = first_page->mapping->host; +if (fscrypt_inode_is_hw_encrypted (bio_inode)) { + next_dun =hw_crypt_dun (bio_inode, first_page->index) + + (io->bio->bi_iter.bi_size >>PAGE_SHIFT ); + } + }if (io->bio && (io->last_block_in_bio != fio->new_blkaddr -1 || (io->fio.op != fio->op || io->fio.op_flags != fio->op_flags) || - !__same_bdev (sbi, fio->new_blkaddr, io->bio))) + !__same_bdev (sbi, fio->new_blkaddr, io->bio) || + !fscrypt_inode_crypt_mergeable (bio_inode, fio_inode) || + (fscrypt_inode_is_hw_encrypted (bio_inode) && + next_dun !=hw_crypt_dun (fio_inode, + fio->page->index))))__submit_merged_bio (io); + alloc_new:if (io->bio ==NULL ) { if ((fio->type == DATA || fio->type == NODE) &&
@@ -568,7 +616,7 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr, bio->bi_end_io = f2fs_read_end_io;bio_set_op_attrs (bio,REQ_OP_READ , op_flag); -if (f2fs_encrypted_file (inode)) +if (f2fs_encrypted_file (inode) && !fscrypt_inode_is_hw_encrypted (inode)) post_read_steps |=1 <<STEP_DECRYPT ;if (post_read_steps) { ctx =mempool_alloc (bio_post_read_ctx_pool,GFP_NOFS );
@@ -1519,6 +1567,7 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,struct f2fs_map_blocks *map, struct bio **bio_ret, sector_t *last_block_in_bio, + u64 *next_dun, bool is_readahead) { struct bio *bio = *bio_ret;
@@ -1592,6 +1641,13 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,__submit_bio (F2FS_I_SB (inode), bio,DATA ); bio =NULL ; } + +if (bio &&fscrypt_inode_is_hw_encrypted (inode) && + *next_dun !=hw_crypt_dun (inode, page->index)) { +__submit_bio (F2FS_I_SB (inode), bio,DATA ); + bio =NULL ; + } +if (bio ==NULL ) { bio =f2fs_grab_read_bio (inode, block_nr, nr_pages, is_readahead ?REQ_RAHEAD :0 );
@@ -1611,6 +1667,9 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,if (bio_add_page (bio, page, blocksize,0 ) < blocksize) goto submit_and_realloc; +if (fscrypt_inode_is_hw_encrypted (inode)) + *next_dun =hw_crypt_dun (inode, page->index) +1 ; +inc_page_count (F2FS_I_SB (inode),F2FS_RD_DATA );ClearPageError (page); *last_block_in_bio = block_nr;
@@ -1644,6 +1703,7 @@ static int f2fs_mpage_readpages(struct address_space *mapping,struct inode *inode = mapping->host;struct f2fs_map_blocks map;int ret =0 ; +u64 next_dun =0 ; map.m_pblk =0 ; map.m_lblk =0 ;
@@ -1667,7 +1727,8 @@ static int f2fs_mpage_readpages(struct address_space *mapping, } ret =f2fs_read_single_page (inode, page, nr_pages, &map, &bio, - &last_block_in_bio, is_readahead); + &last_block_in_bio, &next_dun, + is_readahead);if (ret) {SetPageError (page);zero_user_segment (page,0 ,PAGE_SIZE );
@@ -2617,6 +2678,8 @@ static void f2fs_dio_end_io(struct bio *bio) {struct f2fs_private_dio *dio = bio->bi_private; +fscrypt_unset_bio_crypt_ctx (bio); +dec_page_count (F2FS_I_SB (dio->inode), dio->write ?F2FS_DIO_WRITE :F2FS_DIO_READ );
@@ -2633,12 +2696,18 @@ static void f2fs_dio_submit_bio(struct bio *bio, struct inode *inode, {struct f2fs_private_dio *dio;bool write = (bio_op (bio) ==REQ_OP_WRITE ); +u64 data_unit_num =hw_crypt_dun (inode, file_offset >>PAGE_SHIFT ); dio =f2fs_kzalloc (F2FS_I_SB (inode),sizeof (struct f2fs_private_dio ),GFP_NOFS );if (!dio) goto out; +if (fscrypt_set_bio_crypt_ctx (inode, bio, data_unit_num) !=0 ) { +kvfree (dio); + goto out; + } + dio->inode = inode; dio->orig_end_io = bio->bi_end_io; dio->orig_private = bio->bi_private;
fs/f2fs/super.c +1 -0
@@ -2229,6 +2229,7 @@ static const struct fscrypt_operations f2fs_cryptops = { .dummy_context = f2fs_dummy_context, .empty_dir = f2fs_empty_dir, .max_namelen = F2FS_NAME_LEN, + .hw_crypt_supp = true, };#endif
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog