Add RDMA inline crypto support
Subsystems (10)
cover Add RDMA inline crypto support
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
How to configure fscrypt to enable plaintext keys:
# mkfs.ext4 -O encrypt /dev/nvme0n1
# mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
# head -c 64 /dev/urandom > /tmp/master_key
# fscryptctl add_key /mnt/crypto/ < /tmp/master_key
# mkdir /mnt/crypto/test1
# fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
“fscryptctl add_key” command.
# echo foo > /mnt/crypto/test1/fooNotes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
- 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
There is no special configuration to support crypto at nvme modules.
Thanks
Israel Rukshin (13):
net/mlx5: Introduce crypto IFC bits and structures net/mlx5: Introduce crypto capabilities macro
RDMA: Split kernel-only create QP flags from uverbs create QP flags RDMA/core: Add cryptographic device capabilities RDMA/core: Add DEK management API
RDMA/core: Introduce MR type for crypto operations RDMA/core: Add support for creating crypto enabled QPs RDMA/mlx5: Add cryptographic device capabilities RDMA/mlx5: Add DEK management API
RDMA/mlx5: Add AES-XTS crypto support
nvme: Introduce a local variable
nvme: Add crypto profile at nvme controller
nvme-rdma: Add inline encryption support
drivers/infiniband/core/device.c | 3 + drivers/infiniband/core/mr_pool.c | 2 + drivers/infiniband/core/uverbs_std_types_qp.c | 12 +- drivers/infiniband/core/verbs.c | 91 ++++++- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 2 +- drivers/infiniband/hw/mlx4/mlx4_ib.h | 4 +- drivers/infiniband/hw/mlx4/qp.c | 4 +- drivers/infiniband/hw/mlx5/Makefile | 1 + drivers/infiniband/hw/mlx5/crypto.c | 115 +++++++++ drivers/infiniband/hw/mlx5/crypto.h | 46 ++++ drivers/infiniband/hw/mlx5/main.c | 5 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 5 +- drivers/infiniband/hw/mlx5/mr.c | 33 +++ drivers/infiniband/hw/mlx5/qp.c | 15 +- drivers/infiniband/hw/mlx5/wr.c | 164 +++++++++++- drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 + .../net/ethernet/mellanox/mlx5/core/main.c | 1 + drivers/nvme/host/core.c | 10 +- drivers/nvme/host/nvme.h | 4 + drivers/nvme/host/rdma.c | 236 +++++++++++++++++- include/linux/mlx5/device.h | 4 + include/linux/mlx5/mlx5_ifc.h | 36 ++- include/rdma/crypto.h | 118 +++++++++ include/rdma/ib_verbs.h | 46 +++- include/trace/events/rdma_core.h | 33 +++ 26 files changed, 954 insertions(+), 44 deletions(-) create mode 100644 drivers/infiniband/hw/mlx5/crypto.c create mode 100644 drivers/infiniband/hw/mlx5/crypto.h create mode 100644 include/rdma/crypto.h
Signature · 1 line
2.39.0
PATCH 1/13 net/mlx5: Introduce crypto IFC bits and structures
From: Israel Rukshin <[email protected]>
Add crypto related IFC structs, layouts and enumerations.
--- include/linux/mlx5/mlx5_ifc.h | 36 ++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +33 -3
@@ -1331,6 +1331,29 @@ struct mlx5_ifc_macsec_cap_bits {u8 reserved_at_40[0x7c0 ]; }; +enum { +MLX5_CRYPTO_WRAPPED_IMPORT_METHOD_CAP_AES_XTS =0x4 , +}; + +struct mlx5_ifc_crypto_cap_bits { +u8 wrapped_crypto_operational[0x1 ]; +u8 reserved_at_1[0x17 ]; +u8 wrapped_import_method[0x8 ]; + +u8 reserved_at_20[0xb ]; +u8 log_max_num_deks[0x5 ]; +u8 reserved_at_30[0x3 ]; +u8 log_max_num_import_keks[0x5 ]; +u8 reserved_at_38[0x3 ]; +u8 log_max_num_creds[0x5 ]; + +u8 failed_selftests[0x10 ]; +u8 num_nv_import_keks[0x8 ]; +u8 num_nv_credentials[0x8 ]; + +u8 reserved_at_60[0x7a0 ]; +}; +enum {MLX5_WQ_TYPE_LINKED_LIST =0x0 ,MLX5_WQ_TYPE_CYCLIC =0x1 ,
@@ -1758,7 +1781,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 reserved_at_3e8[0x2 ];u8 vhca_state[0x1 ];u8 log_max_vlan_list[0x5 ]; -u8 reserved_at_3f0[0x3 ]; +u8 reserved_at_3f0[0x1 ]; +u8 aes_xts_single_block_le_tweak[0x1 ]; +u8 aes_xts_multi_block_be_tweak[0x1 ];u8 log_max_current_mc_list[0x5 ];u8 reserved_at_3f8[0x3 ];u8 log_max_current_uc_list[0x5 ];
@@ -1774,7 +1799,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 ats[0x1 ];u8 reserved_at_462[0x1 ];u8 log_max_uctx[0x5 ]; -u8 reserved_at_468[0x2 ]; +u8 aes_xts_multi_block_le_tweak[0x1 ]; +u8 crypto[0x1 ];u8 ipsec_offload[0x1 ];u8 log_max_umem[0x5 ];u8 max_num_eqs[0x10 ];
@@ -3377,6 +3403,7 @@ union mlx5_ifc_hca_cap_union_bits {struct mlx5_ifc_virtio_emulation_cap_bits virtio_emulation_cap;struct mlx5_ifc_shampo_cap_bits shampo_cap;struct mlx5_ifc_macsec_cap_bits macsec_cap; +struct mlx5_ifc_crypto_cap_bits crypto_cap;u8 reserved_at_0[0x8000 ]; };
@@ -3995,7 +4022,9 @@ struct mlx5_ifc_mkc_bits {u8 reserved_at_1d9[0x1 ];u8 log_page_size[0x5 ]; -u8 reserved_at_1e0[0x20 ]; +u8 reserved_at_1e0[0x3 ]; +u8 crypto_en[0x2 ]; +u8 reserved_at_1e5[0x1b ]; };struct mlx5_ifc_pkey_bits {
@@ -11978,6 +12007,7 @@ enum {enum {MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_TLS =0x1 ,MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_IPSEC =0x2 , +MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_AES_XTS =0x3 ,MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_MACSEC =0x4 , };
Signature · 1 line
2.39.0
PATCH 3/13 RDMA: Split kernel-only create QP flags from uverbs create QP flags
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
The current code limits the create_flags to being the same bitmap that will be copied to user space. Extend the create QP flags to 64 bits and move all the flags that are only used internally in the kernel to the upper 32 bits. This cleanly split out the uverbs flags from the kernel flags to avoid confusion in the flags bitmap. Also add some short comments describing which each of the kernel flags is connected to.
--- drivers/infiniband/core/uverbs_std_types_qp.c | 12 +++++----- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 2 +- drivers/infiniband/hw/mlx4/mlx4_ib.h | 4 ++-- drivers/infiniband/hw/mlx4/qp.c | 4 ++-- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/qp.c | 9 ++++---- drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c | 2 +- include/rdma/ib_verbs.h | 22 ++++++++++++++----- 8 files changed, 35 insertions(+), 22 deletions(-)
drivers/infiniband/core/uverbs_std_types_qp.c +7 -5
@@ -97,6 +97,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(struct ib_uobject *xrcd_uobj =NULL ;struct ib_device *device;u64 user_handle; +u32 create_flags;int ret; ret = uverbs_copy_from_or_zero(&cap, attrs,
@@ -191,7 +192,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(return -EINVAL ; } - ret = uverbs_get_flags32(&attr.create_flags, attrs, + ret = uverbs_get_flags32(&create_flags, attrs, UVERBS_ATTR_CREATE_QP_FLAGS, IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK | IB_UVERBS_QP_CREATE_SCATTER_FCS |
@@ -201,7 +202,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(if (ret)return ret; - ret =check_creation_flags (attr.qp_type, attr.create_flags); + ret =check_creation_flags (attr.qp_type, create_flags);if (ret)return ret;
@@ -211,7 +212,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)( UVERBS_ATTR_CREATE_QP_SOURCE_QPN);if (ret)return ret; - attr.create_flags |=IB_QP_CREATE_SOURCE_QPN ; + create_flags |=IB_QP_CREATE_SOURCE_QPN ; } srq = uverbs_attr_get_obj(attrs,
@@ -234,16 +235,17 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)( attr.send_cq = send_cq; attr.recv_cq = recv_cq; attr.xrcd = xrcd; -if (attr.create_flags &IB_UVERBS_QP_CREATE_SQ_SIG_ALL ) { +if (create_flags &IB_UVERBS_QP_CREATE_SQ_SIG_ALL ) {/* This creation bit is uverbs one, need to mask before * calling drivers. It was added to prevent an extra user attr * only for that when using ioctl. */ - attr.create_flags &= ~IB_UVERBS_QP_CREATE_SQ_SIG_ALL ; + create_flags &= ~IB_UVERBS_QP_CREATE_SQ_SIG_ALL ; attr.sq_sig_type =IB_SIGNAL_ALL_WR ; }else { attr.sq_sig_type =IB_SIGNAL_REQ_WR ; } + attr.create_flags = create_flags;set_caps (&attr, &cap, true);mutex_init (&obj->mcast_lock);
drivers/infiniband/hw/bnxt_re/ib_verbs.c +1 -1
@@ -1273,7 +1273,7 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd, qplqp->dpi = &rdev->dpi_privileged;/* Doorbell page */ if (init_attr->create_flags) {ibdev_dbg (&rdev->ibdev, -"QP create flags 0x%x not supported" , +"QP create flags 0x%llx not supported" , init_attr->create_flags);return -EOPNOTSUPP ; }
drivers/infiniband/hw/mlx4/mlx4_ib.h +2 -2
@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, }; struct mlx4_ib_gid_entry {
@@ -342,7 +342,7 @@ struct mlx4_ib_qp {int buf_size;struct mutex mutex;u16 xrcdn; -u32 flags; +u64 flags;u8 port;u8 alt_port;u8 atomic_rd_en;
drivers/infiniband/hw/mlx4/qp.c +2 -2
@@ -287,7 +287,7 @@ static void mlx4_ib_wq_event(struct mlx4_qp *qp, enum mlx4_event type) type, qp->qpn); } -static int send_wqe_overhead (enum mlx4_ib_qp_type type,u32 flags) +static int send_wqe_overhead (enum mlx4_ib_qp_type type,u64 flags) {/* * UD WQEs must have a datagram segment.
@@ -1514,7 +1514,7 @@ static int _mlx4_ib_create_qp(struct ib_pd *pd, struct mlx4_ib_qp *qp, struct ib_udata *udata) {int err; -int sup_u_create_flags =MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK ; +u64 sup_u_create_flags =MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK ;u16 xrcdn =0 ;if (init_attr->rwq_ind_tbl)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -1
@@ -500,7 +500,7 @@ struct mlx5_ib_qp {*/ struct mutex mutex;/* cached variant of create_flags from struct ib_qp_init_attr */ -u32 flags; +u64 flags;u32 port;u8 state;int max_inline_data;
drivers/infiniband/hw/mlx5/qp.c +5 -4
@@ -2938,7 +2938,7 @@ static int process_vendor_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,return (flags) ? -EINVAL :0 ; } -static void process_create_flag (struct mlx5_ib_dev *dev,int *flags,int flag, +static void process_create_flag (struct mlx5_ib_dev *dev,u64 *flags,u64 flag,bool cond,struct mlx5_ib_qp *qp) {if (!(*flags & flag))
@@ -2958,7 +2958,8 @@ static void process_create_flag(struct mlx5_ib_dev *dev, int *flags, int flag, *flags &= ~MLX5_IB_QP_CREATE_WC_TEST ;return ; } -mlx5_ib_dbg (dev,"Verbs create QP flag 0x%X is not supported\n" , flag); +mlx5_ib_dbg (dev,"Verbs create QP flag 0x%llX is not supported\n" , + flag); } static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
@@ -2966,7 +2967,7 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, {enum ib_qp_type qp_type = qp->type;struct mlx5_core_dev *mdev = dev->mdev; -int create_flags = attr->create_flags; +u64 create_flags = attr->create_flags;bool cond;if (qp_type ==MLX5_IB_QPT_DCT )
@@ -3024,7 +3025,7 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, true, qp);if (create_flags) { -mlx5_ib_dbg (dev,"Create QP has unsupported flags 0x%X\n" , +mlx5_ib_dbg (dev,"Create QP has unsupported flags 0x%llX\n" , create_flags);return -EOPNOTSUPP ; }
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +1 -1
@@ -206,7 +206,7 @@ int pvrdma_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,if (init_attr->create_flags) {dev_warn (&dev->pdev->dev, -"invalid create queuepair flags %#x\n" , +"invalid create queuepair flags 0x%llx\n" , init_attr->create_flags);return -EOPNOTSUPP ; }
include/rdma/ib_verbs.h +16 -6
@@ -1140,16 +1140,15 @@ enum ib_qp_type { IB_QPT_RESERVED10, }; +/* +* bits 0, 5, 6 and 7 may be set by old kernels and should not be used. +*/ enum ib_qp_create_flags { -IB_QP_CREATE_IPOIB_UD_LSO =1 <<0 ,IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK =IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK ,IB_QP_CREATE_CROSS_CHANNEL =1 <<2 ,IB_QP_CREATE_MANAGED_SEND =1 <<3 ,IB_QP_CREATE_MANAGED_RECV =1 <<4 , -IB_QP_CREATE_NETIF_QP =1 <<5 , -IB_QP_CREATE_INTEGRITY_EN =1 <<6 , -IB_QP_CREATE_NETDEV_USE =1 <<7 ,IB_QP_CREATE_SCATTER_FCS =IB_UVERBS_QP_CREATE_SCATTER_FCS , IB_QP_CREATE_CVLAN_STRIPPING =
@@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING ,/* reserve bits 26-31 for low level drivers' internal use */ IB_QP_CREATE_RESERVED_START =1 <<26 , - IB_QP_CREATE_RESERVED_END = 1 << 31, +IB_QP_CREATE_RESERVED_END =1ULL <<31 , + +/* The below flags are used only by the kernel */ + +/* The created QP will be used for IPoIB UD LSO */ +IB_QP_CREATE_IPOIB_UD_LSO =1ULL <<32 , +/* Create a QP that supports flow-steering */ +IB_QP_CREATE_NETIF_QP =1ULL <<33 , +/* The created QP can carry out integrity handover operations */ +IB_QP_CREATE_INTEGRITY_EN =1ULL <<34 , +/* Create an accelerated UD QP */ + IB_QP_CREATE_NETDEV_USE = 1ULL << 35, };/*
@@ -1179,7 +1189,7 @@ struct ib_qp_init_attr {struct ib_qp_cap cap;enum ib_sig_type sq_sig_type;enum ib_qp_type qp_type; -u32 create_flags; +u64 create_flags;/* * Only needed for special QP types, or when using the RW API.
Signature · 1 line
2.39.0
PATCH 4/13 RDMA/core: Add cryptographic device capabilities
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Some advanced RDMA devices have HW engines with cryptographic capabilities. Those devices can encrypt/decrypt data when transmitting from memory domain to wire domain and when receiving data from wire domain to memory domain. Expose these capabilities via common RDMA device attributes. For now, add only AES-XTS cryptographic support.
--- include/rdma/crypto.h | 37 +++++++++++++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 2 ++ 2 files changed, 39 insertions(+) create mode 100644 include/rdma/crypto.h
include/rdma/crypto.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* +* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +*/ + +#ifndef _RDMA_CRYPTO_H_ +#define _RDMA_CRYPTO_H_ + +#include <linux/types.h> + +/** +* Encryption and decryption operations are done by attaching crypto properties +* to a memory region. Once done, every access to the memory via the crypto +* memory region will result in inline encryption or decryption of the data +* by the RDMA device. The crypto properties contain the Data Encryption Key +* (DEK) and the crypto standard that should be used and its attributes. +*/ + +/** +* Cryptographic engines in clear text mode capabilities. +* @IB_CRYPTO_ENGINES_CAP_AES_XTS: Support AES-XTS engine. +*/ +enum { +IB_CRYPTO_ENGINES_CAP_AES_XTS =1 <<0 , +}; + +/** +* struct ib_crypto_caps - Cryptographic capabilities +* @crypto_engines: From enum ib_crypto_engines_cap_bits. +* @max_num_deks: Maximum number of Data Encryption Keys. +*/ +struct ib_crypto_caps { +u32 crypto_engines; +u32 max_num_deks; +}; + +#endif /* _RDMA_CRYPTO_H_ */
include/rdma/ib_verbs.h +2 -0
@@ -40,6 +40,7 @@#include <rdma/rdma_counter.h> #include <rdma/restrack.h> #include <rdma/signature.h> +#include <rdma/crypto.h> #include <uapi/rdma/rdma_user_ioctl.h> #include <uapi/rdma/ib_user_ioctl_verbs.h>
@@ -450,6 +451,7 @@ struct ib_device_attr {u64 max_dm_size;/* Max entries for sgl for optimized performance per READ */ u32 max_sgl_rd; +struct ib_crypto_caps crypto_caps; };enum ib_mtu {
Signature · 1 line
2.39.0
PATCH 6/13 RDMA/core: Introduce MR type for crypto operations
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add crypto attributes for MKey. With those attributes, the device can encrypt/decrypt data when transmitting from memory domain to wire domain and when receiving data from wire domain to memory domain.
--- drivers/infiniband/core/device.c | 1 + drivers/infiniband/core/mr_pool.c | 2 ++ drivers/infiniband/core/verbs.c | 56 ++++++++++++++++++++++++++++++- include/rdma/crypto.h | 43 ++++++++++++++++++++++++ include/rdma/ib_verbs.h | 7 ++++ include/trace/events/rdma_core.h | 33 ++++++++++++++++++ 6 files changed, 141 insertions(+), 1 deletion(-)
drivers/infiniband/core/device.c +1 -0
@@ -2599,6 +2599,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, alloc_hw_device_stats);SET_DEVICE_OP (dev_ops, alloc_hw_port_stats);SET_DEVICE_OP (dev_ops, alloc_mr); +SET_DEVICE_OP (dev_ops, alloc_mr_crypto);SET_DEVICE_OP (dev_ops, alloc_mr_integrity);SET_DEVICE_OP (dev_ops, alloc_mw);SET_DEVICE_OP (dev_ops, alloc_pd);
drivers/infiniband/core/mr_pool.c +2 -0
@@ -44,6 +44,8 @@ int ib_mr_pool_init(struct ib_qp *qp, struct list_head *list, int nr,if (type ==IB_MR_TYPE_INTEGRITY ) mr =ib_alloc_mr_integrity (qp->pd, max_num_sg, max_num_meta_sg); +else if (type ==IB_MR_TYPE_CRYPTO ) + mr =ib_alloc_mr_crypto (qp->pd, max_num_sg);else mr =ib_alloc_mr (qp->pd, type, max_num_sg);if (IS_ERR (mr)) {
drivers/infiniband/core/verbs.c +55 -1
86 lines not on this page · raw message
include/rdma/crypto.h +43 -0
49 lines not on this page · raw message
include/rdma/ib_verbs.h +7 -0
@@ -876,6 +876,8 @@ __attribute_const__ int ib_rate_to_mbps(enum ib_rate rate);* without address translations (VA=PA) * @IB_MR_TYPE_INTEGRITY: memory region that is used for * data integrity operations +* @IB_MR_TYPE_CRYPTO: memory region that is used for cryptographic +* operations */ enum ib_mr_type {IB_MR_TYPE_MEM_REG ,
@@ -884,6 +886,7 @@ enum ib_mr_type {IB_MR_TYPE_USER ,IB_MR_TYPE_DMA ,IB_MR_TYPE_INTEGRITY , + IB_MR_TYPE_CRYPTO, }; enum ib_mr_status_check {
@@ -1854,6 +1857,7 @@ struct ib_mr {struct ib_dm *dm;struct ib_sig_attrs *sig_attrs;/* only for IB_MR_TYPE_INTEGRITY MRs */ +struct ib_crypto_attrs *crypto_attrs;/* only for IB_MR_TYPE_CRYPTO */ /* * Implementation details of the RDMA core, don't use in drivers: */
@@ -2512,6 +2516,7 @@ struct ib_device_ops {struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,u32 max_num_data_sg,u32 max_num_meta_sg); +struct ib_mr *(*alloc_mr_crypto)(struct ib_pd *pd,u32 max_num_sg);struct ib_dek *(*create_dek)(struct ib_pd *pd,struct ib_dek_attr *attr);void (*destroy_dek)(struct ib_dek *dek);
@@ -4295,6 +4300,8 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,u32 max_num_data_sg, u32 max_num_meta_sg); +struct ib_mr *ib_alloc_mr_crypto (struct ib_pd *pd,u32 max_num_sg); +/** * ib_update_fast_reg_key - updates the key portion of the fast_reg MR * R_Key and L_Key.
include/trace/events/rdma_core.h +33 -0
@@ -371,6 +371,39 @@ TRACE_EVENT(mr_integ_alloc, __entry->max_num_meta_sg, __entry->rc) ); +TRACE_EVENT (mr_crypto_alloc, +TP_PROTO ( + const struct ib_pd *pd, + u32 max_num_sg, + const struct ib_mr *mr + ), + +TP_ARGS (pd, max_num_sg, mr), + +TP_STRUCT__entry ( + __field(u32, pd_id) + __field(u32, mr_id) + __field(u32, max_num_sg) +__field (int, rc) + ), + +TP_fast_assign ( + __entry->pd_id = pd->res.id; + if (IS_ERR(mr)) { + __entry->mr_id = 0; + __entry->rc = PTR_ERR(mr); + } else { + __entry->mr_id = mr->res.id; + __entry->rc = 0; + } + __entry->max_num_sg = max_num_sg; + ), + +TP_printk ("pd.id=%u mr.id=%u max_num_sg=%u rc=%d" , + __entry->pd_id, __entry->mr_id, __entry->max_num_sg, + __entry->rc) +); + TRACE_EVENT(mr_dereg, TP_PROTO( const struct ib_mr *mr
Signature · 1 line
2.39.0
PATCH 7/13 RDMA/core: Add support for creating crypto enabled QPs
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add a list of crypto MRs and introduce a crypto WR type to post on those QPs.
--- drivers/infiniband/core/verbs.c | 3 +++ include/rdma/ib_verbs.h | 12 +++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-)
drivers/infiniband/core/verbs.c +3 -0
@@ -1223,6 +1223,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,spin_lock_init (&qp->mr_lock);INIT_LIST_HEAD (&qp->rdma_mrs);INIT_LIST_HEAD (&qp->sig_mrs); +INIT_LIST_HEAD (&qp->crypto_mrs); qp->send_cq = attr->send_cq; qp->recv_cq = attr->recv_cq;
@@ -1363,6 +1364,8 @@ struct ib_qp *ib_create_qp_kernel(struct ib_pd *pd, device->attrs.max_sge_rd);if (qp_init_attr->create_flags &IB_QP_CREATE_INTEGRITY_EN ) qp->integrity_en = true; +if (qp_init_attr->create_flags &IB_QP_CREATE_CRYPTO_EN ) + qp->crypto_en = true;return qp;
include/rdma/ib_verbs.h +11 -1
@@ -1175,6 +1175,8 @@ enum ib_qp_create_flags {IB_QP_CREATE_INTEGRITY_EN =1ULL <<34 ,/* Create an accelerated UD QP */ IB_QP_CREATE_NETDEV_USE =1ULL <<35 , +/* The created QP can carry out cryptographic handover operations */ + IB_QP_CREATE_CRYPTO_EN = 1ULL << 36, };/*
@@ -1352,6 +1354,12 @@ enum ib_wr_opcode {/* These are kernel only and can not be issued by userspace */ IB_WR_REG_MR =0x20 ,IB_WR_REG_MR_INTEGRITY , +/* +* It is used to assign crypto properties to a MKey. Use the MKey in +* any RDMA transaction (SEND/RECV/READ/WRITE) to encrypt/decrypt data +* on-the-fly. +*/ + IB_WR_REG_MR_CRYPTO,/* reserve values for low level drivers' internal use. * These values will not be used at all in the ib core layer.
@@ -1800,6 +1808,7 @@ struct ib_qp {int mrs_used;struct list_head rdma_mrs;struct list_head sig_mrs; +struct list_head crypto_mrs;struct ib_srq *srq;struct ib_xrcd *xrcd;/* XRC TGT QPs only */ struct list_head xrcd_list;
@@ -1822,7 +1831,8 @@ struct ib_qp {struct ib_qp_security *qp_sec;u32 port; -bool integrity_en; + u8 integrity_en:1; + u8 crypto_en:1;/* * Implementation details of the RDMA core, don't use in drivers: */
Signature · 1 line
2.39.0
PATCH 11/13 nvme: Introduce a local variable
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
The patch doesn't change any logic.
--- drivers/nvme/host/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +4 -3
@@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,sector_t capacity =nvme_lba_to_sect (ns,le64_to_cpu (id->nsze));unsigned short bs =1 << ns->lba_shift;u32 atomic_bs, phys_bs, io_opt =0 ; +struct nvme_ctrl *ctrl = ns->ctrl;/* * The block layer can't support LBA sizes larger than the page size
@@ -1892,7 +1893,7 @@ static void nvme_update_disk_info(struct gendisk *disk,if (id->nsfeat &NVME_NS_FEAT_ATOMICS && id->nawupf) atomic_bs = (1 +le16_to_cpu (id->nawupf)) * bs;else - atomic_bs = (1 + ns->ctrl->subsys->awupf) * bs; + atomic_bs = (1 + ctrl->subsys->awupf) * bs; }if (id->nsfeat &NVME_NS_FEAT_IO_OPT ) {
@@ -1922,7 +1923,7 @@ static void nvme_update_disk_info(struct gendisk *disk,if (IS_ENABLED (CONFIG_BLK_DEV_INTEGRITY ) && (ns->features &NVME_NS_METADATA_SUPPORTED ))nvme_init_integrity (disk, ns, - ns->ctrl->max_integrity_segments); + ctrl->max_integrity_segments);else if (!nvme_ns_has_pi (ns)) capacity =0 ; }
@@ -1931,7 +1932,7 @@ static void nvme_update_disk_info(struct gendisk *disk,nvme_config_discard (disk, ns);blk_queue_max_write_zeroes_sectors (disk->queue, - ns->ctrl->max_zeroes_sectors); + ctrl->max_zeroes_sectors); }static bool nvme_ns_is_readonly (struct nvme_ns *ns,struct nvme_ns_info *info)
Signature · 1 line
2.39.0
PATCH 12/13 nvme: Add crypto profile at nvme controller
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
The crypto profile will be filled by the transport drivers. This is a preparation patch for adding support of inline encryption at nvme-rdma driver.
--- drivers/nvme/host/core.c | 3 +++ drivers/nvme/host/nvme.h | 4 ++++ 2 files changed, 7 insertions(+)
drivers/nvme/host/core.c +3 -0
@@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk, capacity =0 ; } +if (ctrl->crypto_enable) +blk_crypto_register (&ctrl->crypto_profile, disk->queue); +set_capacity_and_notify (disk, capacity);nvme_config_discard (disk, ns);
drivers/nvme/host/nvme.h +4 -0
@@ -16,6 +16,7 @@#include <linux/rcupdate.h> #include <linux/wait.h> #include <linux/t10-pi.h> +#include <linux/blk-crypto-profile.h> #include <trace/events/block.h>
@@ -374,6 +375,9 @@ struct nvme_ctrl {enum nvme_ctrl_type cntrltype;enum nvme_dctype dctype; + +bool crypto_enable; +struct blk_crypto_profile crypto_profile; };enum nvme_iopolicy {
Signature · 1 line
2.39.0
PATCH 13/13 nvme-rdma: Add inline encryption support
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add support for inline encryption/decryption of the data at a similar way like integrity is used via a unique Mkey. The feature is enabled only when CONFIG_BLK_INLINE_ENCRYPTION is configured. There is no special configuration to enable crypto at nvme modules.
The code was tested with fscrypt and BF-3 HW, which had more than 50% CPU utilization improvement at 64K and bigger I/O sizes when comparing to the SW only solution at this case.
--- drivers/nvme/host/rdma.c | 236 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 232 insertions(+), 4 deletions(-)
drivers/nvme/host/rdma.c +232 -4
@@ -40,6 +40,10 @@#define NVME_RDMA_METADATA_SGL_SIZE \ (sizeof(struct scatterlist) * NVME_INLINE_METADATA_SG_CNT) +#define NVME_RDMA_NUM_CRYPTO_KEYSLOTS (32U) +/* Bitmask for 512B and 4K data unit sizes */ +#define NVME_RDMA_DATA_UNIT_SIZES (512U | 4096U) +struct nvme_rdma_device {struct ib_device *dev;struct ib_pd *pd;
@@ -75,6 +79,7 @@ struct nvme_rdma_request {struct nvme_rdma_sgl data_sgl;struct nvme_rdma_sgl *metadata_sgl;bool use_sig_mr; +bool use_crypto; };enum nvme_rdma_queue_flags {
@@ -97,6 +102,7 @@ struct nvme_rdma_queue {int cm_error;struct completion cm_done;bool pi_support; +bool crypto_support;int cq_size;struct mutex queue_lock; };
@@ -126,6 +132,8 @@ struct nvme_rdma_ctrl {struct nvme_ctrl ctrl;bool use_inline_data;u32 io_queues[HCTX_MAX_TYPES ]; + +struct ib_dek *dek[NVME_RDMA_NUM_CRYPTO_KEYSLOTS ]; };static inline struct nvme_rdma_ctrl *to_rdma_ctrl (struct nvme_ctrl *ctrl)
@@ -275,6 +283,8 @@ static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor) init_attr.recv_cq = queue->ib_cq;if (queue->pi_support) init_attr.create_flags |=IB_QP_CREATE_INTEGRITY_EN ; +if (queue->crypto_support) + init_attr.create_flags |=IB_QP_CREATE_CRYPTO_EN ; init_attr.qp_context = queue; ret =rdma_create_qp (queue->cm_id, dev->pd, &init_attr);
@@ -364,6 +374,77 @@ static int nvme_rdma_dev_get(struct nvme_rdma_device *dev)return kref_get_unless_zero (&dev->ref); } +static int nvme_rdma_crypto_keyslot_program (struct blk_crypto_profile *profile, +const struct blk_crypto_key *key, +unsigned int slot) +{ +struct nvme_ctrl *nctrl = + container_of(profile, struct nvme_ctrl, crypto_profile); +struct nvme_rdma_ctrl *ctrl =to_rdma_ctrl (nctrl); +struct ib_dek_attr dek_attr = { }; +int err =0 ; + +if (slot >NVME_RDMA_NUM_CRYPTO_KEYSLOTS ) { +dev_err (nctrl->device,"slot index reached the limit %d/%d" , + slot,NVME_RDMA_NUM_CRYPTO_KEYSLOTS ); +return -EOPNOTSUPP ; + } + +if (WARN_ON_ONCE (key->crypto_cfg.crypto_mode != +BLK_ENCRYPTION_MODE_AES_256_XTS )) +return -EOPNOTSUPP ; + +if (ctrl->dek[slot]) { +dev_dbg (nctrl->device,"slot %d is taken, free DEK ID %d\n" , + slot, ctrl->dek[slot]->id); +ib_destroy_dek (ctrl->dek[slot]); + } + + dek_attr.key_blob = key->raw; + dek_attr.key_blob_size = key->size; + dek_attr.key_type =IB_CRYPTO_KEY_TYPE_AES_XTS ; + ctrl->dek[slot] =ib_create_dek (ctrl->device->pd, &dek_attr); +if (IS_ERR (ctrl->dek[slot])) { + err =PTR_ERR (ctrl->dek[slot]); +dev_err (nctrl->device, +"failed to create DEK at slot %d, err %d\n" , slot, err); + ctrl->dek[slot] =NULL ; + }else { +dev_dbg (nctrl->device,"DEK ID %d was created at slot %d\n" , + ctrl->dek[slot]->id, slot); + } + +return err; +} + +static int nvme_rdma_crypto_keyslot_evict (struct blk_crypto_profile *profile, +const struct blk_crypto_key *key, +unsigned int slot) +{ +struct nvme_ctrl *nctrl = + container_of(profile, struct nvme_ctrl, crypto_profile); +struct nvme_rdma_ctrl *ctrl =to_rdma_ctrl (nctrl); + +if (slot >NVME_RDMA_NUM_CRYPTO_KEYSLOTS ) { +dev_err (nctrl->device,"slot index reached the limit %d/%d\n" , + slot,NVME_RDMA_NUM_CRYPTO_KEYSLOTS ); +return -EOPNOTSUPP ; + } + +if (!ctrl->dek[slot]) { +dev_err (nctrl->device,"slot %d is empty\n" , slot); +return -EINVAL ; + } + +dev_dbg (nctrl->device,"Destroy DEK ID %d slot %d\n" , + ctrl->dek[slot]->id, slot); + +ib_destroy_dek (ctrl->dek[slot]); + ctrl->dek[slot] =NULL ; + +return 0 ; +} +static struct nvme_rdma_device *nvme_rdma_find_get_device (struct rdma_cm_id *cm_id) {
@@ -430,6 +511,8 @@ static void nvme_rdma_destroy_queue_ib(struct nvme_rdma_queue *queue) dev = queue->device; ibdev = dev->dev; +if (queue->crypto_support) +ib_mr_pool_destroy (queue->qp, &queue->qp->crypto_mrs);if (queue->pi_support)ib_mr_pool_destroy (queue->qp, &queue->qp->sig_mrs);ib_mr_pool_destroy (queue->qp, &queue->qp->rdma_mrs);
@@ -553,10 +636,24 @@ static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue) } } +if (queue->crypto_support) { + ret =ib_mr_pool_init (queue->qp, &queue->qp->crypto_mrs, + queue->queue_size,IB_MR_TYPE_CRYPTO , + pages_per_mr,0 ); +if (ret) { +dev_err (queue->ctrl->ctrl.device, +"failed to initialize crypto MR pool sized %d for QID %d\n" , + queue->queue_size,nvme_rdma_queue_idx (queue)); + goto out_destroy_sig_mr_pool; + } + } +set_bit (NVME_RDMA_Q_TR_READY , &queue->flags);return 0 ; +out_destroy_sig_mr_pool: +ib_mr_pool_destroy (queue->qp, &queue->qp->sig_mrs); out_destroy_mr_pool:ib_mr_pool_destroy (queue->qp, &queue->qp->rdma_mrs); out_destroy_ring:
@@ -585,6 +682,9 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, queue->pi_support = true; else queue->pi_support = false; + + queue->crypto_support = idx && ctrl->ctrl.crypto_enable; +init_completion (&queue->cm_done);if (idx >0 )
@@ -805,15 +905,109 @@ static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl)static void nvme_rdma_destroy_admin_queue (struct nvme_rdma_ctrl *ctrl) { +unsigned int i; +if (ctrl->async_event_sqe.data) {cancel_work_sync (&ctrl->ctrl.async_event_work);nvme_rdma_free_qe (ctrl->device->dev, &ctrl->async_event_sqe,sizeof (struct nvme_command ),DMA_TO_DEVICE ); ctrl->async_event_sqe.data =NULL ; } + +for (i =0 ; i <NVME_RDMA_NUM_CRYPTO_KEYSLOTS ; i++) { +if (!ctrl->dek[i]) +continue ; +ib_destroy_dek (ctrl->dek[i]); + ctrl->dek[i] =NULL ; + } +nvme_rdma_free_queue (&ctrl->queues[0 ]); } +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +static const struct blk_crypto_ll_ops nvme_rdma_crypto_ops = { + .keyslot_program = nvme_rdma_crypto_keyslot_program, + .keyslot_evict = nvme_rdma_crypto_keyslot_evict, +}; + +static int nvme_rdma_crypto_profile_init (struct nvme_rdma_ctrl *ctrl,bool new) +{ +struct blk_crypto_profile *profile = &ctrl->ctrl.crypto_profile; +int ret; + +if (!new) { +blk_crypto_reprogram_all_keys (profile); +return 0 ; + } + + ret =devm_blk_crypto_profile_init (ctrl->ctrl.device, profile, +NVME_RDMA_NUM_CRYPTO_KEYSLOTS ); +if (ret) { +dev_err (ctrl->ctrl.device, +"devm_blk_crypto_profile_init failed err %d\n" , ret); +return ret; + } + + profile->ll_ops = nvme_rdma_crypto_ops; + profile->dev = ctrl->ctrl.device; + profile->max_dun_bytes_supported =IB_CRYPTO_XTS_TWEAK_MAX_SIZE ; + profile->modes_supported[BLK_ENCRYPTION_MODE_AES_256_XTS ] = +NVME_RDMA_DATA_UNIT_SIZES ; + +return 0 ; +} + +static void nvme_rdma_set_crypto_attrs (struct ib_crypto_attrs *crypto_attrs, +struct nvme_rdma_queue *queue,struct nvme_rdma_request *req) +{ +struct request *rq =blk_mq_rq_from_pdu (req); +u32 slot_idx =blk_crypto_keyslot_index (rq->crypt_keyslot); + +memset (crypto_attrs,0 ,sizeof (*crypto_attrs)); + + crypto_attrs->encrypt_domain =IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN ; + crypto_attrs->encrypt_standard =IB_CRYPTO_AES_XTS ; + crypto_attrs->data_unit_size = + rq->crypt_ctx->bc_key->crypto_cfg.data_unit_size; + crypto_attrs->dek = queue->ctrl->dek[slot_idx]->id; +memcpy (crypto_attrs->xts_init_tweak, rq->crypt_ctx->bc_dun, +sizeof (crypto_attrs->xts_init_tweak)); +} + +static int nvme_rdma_fill_crypto_caps (struct nvme_rdma_ctrl *ctrl,bool new) +{ +struct ib_crypto_caps *caps = &ctrl->device->dev->attrs.crypto_caps; + +if (caps->crypto_engines &IB_CRYPTO_ENGINES_CAP_AES_XTS ) { + ctrl->ctrl.crypto_enable = true; +return 0 ; + } + +if (!new && ctrl->ctrl.crypto_enable) { +dev_err (ctrl->ctrl.device,"Must support crypto!\n" ); +return -EOPNOTSUPP ; + } + ctrl->ctrl.crypto_enable = false; + +return 0 ; +} +#else +static int nvme_rdma_crypto_profile_init (struct nvme_rdma_ctrl *ctrl,bool new) +{ +return -EOPNOTSUPP ; +} + +static void nvme_rdma_set_crypto_attrs (struct ib_crypto_attrs *crypto_attrs, +struct nvme_rdma_queue *queue,struct nvme_rdma_request *req) +{ +} + +static int nvme_rdma_fill_crypto_caps (struct nvme_rdma_ctrl *ctrl,bool new) +{ +return 0 ; +} +#endif /* CONFIG_BLK_INLINE_ENCRYPTION */ +static int nvme_rdma_configure_admin_queue (struct nvme_rdma_ctrl *ctrl,bool new) {
@@ -835,6 +1029,10 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, ctrl->max_fr_pages =nvme_rdma_get_max_fr_pages (ctrl->device->dev, pi_capable); + error =nvme_rdma_fill_crypto_caps (ctrl, new); +if (error) + goto out_free_queue; +/* * Bind the async event SQE DMA mapping to the admin queue lifetime. * It's safe, since any chage in the underlying RDMA device will issue
@@ -870,6 +1068,12 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,else ctrl->ctrl.max_integrity_segments =0 ; +if (ctrl->ctrl.crypto_enable) { + error =nvme_rdma_crypto_profile_init (ctrl, new); +if (error) + goto out_quiesce_queue; + } +nvme_unquiesce_admin_queue (&ctrl->ctrl); error =nvme_init_ctrl_finish (&ctrl->ctrl, false);
@@ -1268,6 +1472,8 @@ static void nvme_rdma_unmap_data(struct nvme_rdma_queue *queue,if (req->use_sig_mr) pool = &queue->qp->sig_mrs; +else if (req->use_crypto) + pool = &queue->qp->crypto_mrs;if (req->mr) {ib_mr_pool_put (queue->qp, pool, req->mr);
@@ -1331,9 +1537,13 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue, int count) {struct nvme_keyed_sgl_desc *sg = &c->common.dptr.ksgl; +struct list_head *pool = &queue->qp->rdma_mrs;int nr; - req->mr =ib_mr_pool_get (queue->qp, &queue->qp->rdma_mrs); +if (req->use_crypto) + pool = &queue->qp->crypto_mrs; + + req->mr =ib_mr_pool_get (queue->qp, pool);if (WARN_ON_ONCE (!req->mr))return -EAGAIN ;
@@ -1344,18 +1554,24 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue, nr =ib_map_mr_sg (req->mr, req->data_sgl.sg_table.sgl, count,NULL ,SZ_4K );if (unlikely (nr < count)) { -ib_mr_pool_put (queue->qp, &queue->qp->rdma_mrs, req->mr); +ib_mr_pool_put (queue->qp, pool, req->mr); req->mr =NULL ;if (nr <0 )return nr;return -EINVAL ; } +if (req->use_crypto) +nvme_rdma_set_crypto_attrs (req->mr->crypto_attrs, queue, req); +ib_update_fast_reg_key (req->mr,ib_inc_rkey (req->mr->rkey)); req->reg_cqe.done = nvme_rdma_memreg_done;memset (&req->reg_wr,0 ,sizeof (req->reg_wr)); - req->reg_wr.wr.opcode =IB_WR_REG_MR ; +if (req->use_crypto) + req->reg_wr.wr.opcode =IB_WR_REG_MR_CRYPTO ; +else + req->reg_wr.wr.opcode =IB_WR_REG_MR ; req->reg_wr.wr.wr_cqe = &req->reg_cqe; req->reg_wr.wr.num_sge =0 ; req->reg_wr.mr = req->mr;
@@ -1571,7 +1787,7 @@ static int nvme_rdma_map_data(struct nvme_rdma_queue *queue, goto out; } - if (count <= dev->num_inline_segments) { + if (count <= dev->num_inline_segments && !req->use_crypto) { if (rq_data_dir(rq) == WRITE && nvme_rdma_queue_idx(queue) && queue->ctrl->use_inline_data && blk_rq_payload_bytes(rq) <=
@@ -2052,6 +2268,18 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,else req->use_sig_mr = false; +#ifdef CONFIG_BLK_INLINE_ENCRYPTION +if (rq->crypt_ctx) { +if (WARN_ON_ONCE (!queue->crypto_support)) { + err = -EIO ; + goto err; + } + req->use_crypto = true; + }else { + req->use_crypto = false; + } +#endif + err =nvme_rdma_map_data (queue, rq, c);if (unlikely (err <0 )) { dev_err(queue->ctrl->ctrl.device,
Signature · 1 line
2.39.0
PATCH 8/13 RDMA/mlx5: Add cryptographic device capabilities
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
The capabilities provide information on general cryptographic support, maximum number of DEKs and status for RDMA devices. Also, they include the supported cryptographic engines and their import method (wrapped or plaintext). Wrapped crypto operational flag indicates the import method mode that can be used. For now, add only AES-XTS cryptographic support.
--- drivers/infiniband/hw/mlx5/Makefile | 1 + drivers/infiniband/hw/mlx5/crypto.c | 31 ++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/crypto.h | 11 ++++++++++ drivers/infiniband/hw/mlx5/main.c | 5 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 ++ 5 files changed, 50 insertions(+) create mode 100644 drivers/infiniband/hw/mlx5/crypto.c create mode 100644 drivers/infiniband/hw/mlx5/crypto.h
drivers/infiniband/hw/mlx5/Makefile +1 -0
@@ -6,6 +6,7 @@ mlx5_ib-y := ah.o \ cong.o \ counters.o \ cq.o \ + crypto.o \ dm.o \ doorbell.o \ gsi.o \
drivers/infiniband/hw/mlx5/crypto.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */ + +#include "crypto.h" + +void mlx5r_crypto_caps_init (struct mlx5_ib_dev *dev) +{ +struct ib_crypto_caps *caps = &dev->crypto_caps; +struct mlx5_core_dev *mdev = dev->mdev; + +if (!(MLX5_CAP_GEN_64 (dev->mdev, general_obj_types) & +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY )) +return ; + +if (!MLX5_CAP_GEN (mdev, aes_xts_multi_block_le_tweak) && + !MLX5_CAP_GEN (mdev, aes_xts_multi_block_be_tweak)) +return ; + +if (MLX5_CAP_CRYPTO (mdev, wrapped_import_method) & +MLX5_CRYPTO_WRAPPED_IMPORT_METHOD_CAP_AES_XTS ) +return ; + +if (MLX5_CAP_CRYPTO (mdev, failed_selftests)) { +mlx5_ib_warn (dev,"crypto self-tests failed with error 0x%x\n" , +MLX5_CAP_CRYPTO (mdev, failed_selftests)); +return ; + } + + caps->crypto_engines |=IB_CRYPTO_ENGINES_CAP_AES_XTS ; + caps->max_num_deks =1 <<MLX5_CAP_CRYPTO (mdev, log_max_num_deks); +}
drivers/infiniband/hw/mlx5/crypto.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */ + +#ifndef _MLX5_IB_CRYPTO_H +#define _MLX5_IB_CRYPTO_H + +#include "mlx5_ib.h" + +void mlx5r_crypto_caps_init (struct mlx5_ib_dev *dev); + +#endif /* _MLX5_IB_CRYPTO_H */
drivers/infiniband/hw/mlx5/main.c +5 -0
@@ -39,6 +39,7 @@#include "srq.h" #include "qp.h" #include "wr.h" +#include "crypto.h" #include "restrack.h" #include "counters.h" #include "umr.h"
@@ -989,6 +990,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, props->max_ah =INT_MAX ; props->hca_core_clock =MLX5_CAP_GEN (mdev, device_frequency_khz); props->timestamp_mask =0x7FFFFFFFFFFFFFFFULL ; + props->crypto_caps = dev->crypto_caps;if (IS_ENABLED (CONFIG_INFINIBAND_ON_DEMAND_PAGING )) {if (dev->odp_caps.general_caps &IB_ODP_SUPPORT )
@@ -3826,6 +3828,9 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)if (MLX5_CAP_GEN (mdev, xrc))ib_set_device_ops (&dev->ib_dev, &mlx5_ib_dev_xrc_ops); +if (MLX5_CAP_GEN (mdev, crypto)) +mlx5r_crypto_caps_init (dev); +if (MLX5_CAP_DEV_MEM (mdev, memic) ||MLX5_CAP_GEN_64 (dev->mdev, general_obj_types) &MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM )
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -0
@@ -1100,6 +1100,8 @@ struct mlx5_ib_dev {struct mlx5_ib_delay_drop delay_drop;const struct mlx5_ib_profile *profile; +struct ib_crypto_caps crypto_caps; +struct mlx5_ib_lb_state lb;u8 umr_fence;struct list_head ib_dev_list;
Signature · 1 line
2.39.0
PATCH 2/13 net/mlx5: Introduce crypto capabilities macro
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add MLX5_CAP_CRYPTO() macro to the list of capabilities.
--- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 ++++++ drivers/net/ethernet/mellanox/mlx5/core/main.c | 1 + include/linux/mlx5/device.h | 4 ++++ 3 files changed, 11 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
@@ -147,6 +147,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)if (err)return err; +if (MLX5_CAP_GEN (dev, crypto)) { + err =mlx5_core_get_caps (dev,MLX5_CAP_CRYPTO ); +if (err) +return err; + } +if (MLX5_CAP_GEN (dev, port_selection_cap)) { err =mlx5_core_get_caps (dev,MLX5_CAP_PORT_SELECTION );if (err)
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -0
@@ -1555,6 +1555,7 @@ static const int types[] = {MLX5_CAP_DEV_SHAMPO ,MLX5_CAP_MACSEC ,MLX5_CAP_ADV_VIRTUALIZATION , + MLX5_CAP_CRYPTO, };static void mlx5_hca_caps_free (struct mlx5_core_dev *dev)
include/linux/mlx5/device.h +4 -0
@@ -1204,6 +1204,7 @@ enum mlx5_cap_type {MLX5_CAP_VDPA_EMULATION =0x13 ,MLX5_CAP_DEV_EVENT =0x14 ,MLX5_CAP_IPSEC , +MLX5_CAP_CRYPTO =0x1a ,MLX5_CAP_DEV_SHAMPO =0x1d ,MLX5_CAP_MACSEC =0x1f , MLX5_CAP_GENERAL_2 = 0x20,
@@ -1466,6 +1467,9 @@ enum mlx5_qcam_feature_groups {#define MLX5_CAP_MACSEC (mdev, cap)\ MLX5_GET(macsec_cap, (mdev)->caps.hca[MLX5_CAP_MACSEC]->cur, cap) +#define MLX5_CAP_CRYPTO (mdev, cap)\ + MLX5_GET(crypto_cap, (mdev)->caps.hca[MLX5_CAP_CRYPTO]->cur, cap) +enum {MLX5_CMD_STAT_OK =0x0 ,MLX5_CMD_STAT_INT_ERR =0x1 ,
Signature · 1 line
2.39.0
PATCH 10/13 RDMA/mlx5: Add AES-XTS crypto support
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add crypto attributes for MKey and QP. With this, the device can encrypt/decrypt data when transmitting data from memory to network and when receiving data from network to memory.
--- drivers/infiniband/hw/mlx5/crypto.c | 1 + drivers/infiniband/hw/mlx5/crypto.h | 27 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/mr.c | 33 ++++++ drivers/infiniband/hw/mlx5/qp.c | 6 + drivers/infiniband/hw/mlx5/wr.c | 164 +++++++++++++++++++++++++-- 6 files changed, 222 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/crypto.c +1 -0
@@ -82,6 +82,7 @@ static void mlx5r_destroy_dek(struct ib_dek *dek)static const struct ib_device_ops mlx5r_dev_crypto_ops = { .create_dek = mlx5r_create_dek, .destroy_dek = mlx5r_destroy_dek, + .alloc_mr_crypto = mlx5r_alloc_mr_crypto, };void mlx5r_crypto_caps_init (struct mlx5_ib_dev *dev)
drivers/infiniband/hw/mlx5/crypto.h +27 -0
@@ -6,6 +6,33 @@#include "mlx5_ib.h" +enum { +MLX5_CRYPTO_ENCRYPTED_WIRE =0x0 , +MLX5_CRYPTO_ENCRYPTED_MEM =0x1 , +}; + +enum { +MLX5_CRYPTO_AES_XTS =0x0 , +}; + +struct mlx5_bsf_crypto { +u8 bsf_size_type; +u8 encryption_order; +u8 rsvd0; +u8 encryption_standard; +__be32 raw_data_size; +u8 block_size_p; +u8 rsvd1[7 ]; +union { +__be32 be_xts_init_tweak[4 ]; +__le32 le_xts_init_tweak[4 ]; + }; +__be32 rsvd_dek_pointer; +u8 rsvd2[4 ]; +u8 keytag[8 ]; +u8 rsvd3[16 ]; +}; +/* * The standard AES-XTS key blob composed of two keys. * AES-128-XTS key blob composed of two 128-bit keys, which is 32 bytes and
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
@@ -1291,6 +1291,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,struct ib_mr *mlx5_ib_alloc_mr_integrity (struct ib_pd *pd,u32 max_num_sg,u32 max_num_meta_sg); +struct ib_mr *mlx5r_alloc_mr_crypto (struct ib_pd *pd,u32 max_num_sg);int mlx5_ib_map_mr_sg (struct ib_mr *ibmr,struct scatterlist *sg,int sg_nents,unsigned int *sg_offset); int mlx5_ib_map_mr_sg_pi(struct ib_mr *ibmr, struct scatterlist *data_sg,
drivers/infiniband/hw/mlx5/mr.c +33 -0
@@ -40,6 +40,7 @@#include <linux/dma-buf.h> #include <linux/dma-resv.h> #include <rdma/ib_umem_odp.h> +#include "crypto.h" #include "dm.h" #include "mlx5_ib.h" #include "umr.h"
@@ -1553,6 +1554,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,int add_size;int ret; +if (mr->ibmr.type ==IB_MR_TYPE_CRYPTO ) + size +=sizeof (struct mlx5_bsf_crypto ); add_size =max_t (int,MLX5_UMR_ALIGN -ARCH_KMALLOC_MINALIGN ,0 ); mr->descs_alloc =kzalloc (size + add_size,GFP_KERNEL );
@@ -1582,6 +1585,8 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr)int size = mr->max_descs * mr->desc_size;struct mlx5_ib_dev *dev =to_mdev (device); +if (mr->ibmr.type ==IB_MR_TYPE_CRYPTO ) + size +=sizeof (struct mlx5_bsf_crypto );dma_unmap_single (&dev->mdev->pdev->dev, mr->desc_map, size,DMA_TO_DEVICE );kfree (mr->descs_alloc);
@@ -1862,6 +1867,21 @@ static int mlx5_alloc_integrity_descs(struct ib_pd *pd, struct mlx5_ib_mr *mr,return err; } +static int mlx5_alloc_crypto_descs (struct ib_pd *pd,struct mlx5_ib_mr *mr, +int ndescs,u32 *in,int inlen) +{ +void *mkc; + + mkc =MLX5_ADDR_OF (create_mkey_in, in, memory_key_mkey_entry); +MLX5_SET (mkc, mkc, crypto_en,1 ); +/* Set bsf descriptors for mkey */ +MLX5_SET (mkc, mkc, bsf_en,1 ); +MLX5_SET (mkc, mkc, bsf_octword_size,MLX5_MKEY_BSF_OCTO_SIZE ); + +return _mlx5_alloc_mkey_descs (pd, mr, ndescs,sizeof (struct mlx5_klm ), +0 ,MLX5_MKC_ACCESS_MODE_KLMS , in, inlen); +} +static struct ib_mr *__mlx5_ib_alloc_mr (struct ib_pd *pd,enum ib_mr_type mr_type,u32 max_num_sg,u32 max_num_meta_sg)
@@ -1897,6 +1917,9 @@ static struct ib_mr *__mlx5_ib_alloc_mr(struct ib_pd *pd, err =mlx5_alloc_integrity_descs (pd, mr, max_num_sg, max_num_meta_sg, in, inlen);break ; +case IB_MR_TYPE_CRYPTO : + err =mlx5_alloc_crypto_descs (pd, mr, ndescs, in, inlen); +break ;default :mlx5_ib_warn (dev,"Invalid mr type %d\n" , mr_type); err = -EINVAL ;
@@ -1929,6 +1952,11 @@ struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd, max_num_meta_sg); } +struct ib_mr *mlx5r_alloc_mr_crypto (struct ib_pd *pd,u32 max_num_sg) +{ +return __mlx5_ib_alloc_mr (pd,IB_MR_TYPE_CRYPTO , max_num_sg,0 ); +} +int mlx5_ib_alloc_mw (struct ib_mw *ibmw,struct ib_udata *udata) {struct mlx5_ib_dev *dev =to_mdev (ibmw->device);
@@ -2368,6 +2396,11 @@ int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, mr->desc_size * mr->max_descs, DMA_TO_DEVICE); +if (ibmr->type ==IB_MR_TYPE_CRYPTO ) { +/* This is zero-based memory region */ + ibmr->iova =0 ; + } +return n; }
drivers/infiniband/hw/mlx5/qp.c +6 -0
@@ -40,6 +40,7 @@#include "ib_rep.h" #include "counters.h" #include "cmd.h" +#include "crypto.h" #include "umr.h" #include "qp.h" #include "wr.h"
@@ -554,6 +555,8 @@ static int calc_send_wqe(struct ib_qp_init_attr *attr) } size += attr->cap.max_send_sge *sizeof (struct mlx5_wqe_data_seg ); +if (attr->create_flags &IB_QP_CREATE_CRYPTO_EN ) + size +=sizeof (struct mlx5_bsf_crypto );if (attr->create_flags &IB_QP_CREATE_INTEGRITY_EN &&ALIGN (max_t (int, inl_size, size),MLX5_SEND_WQE_BB ) <MLX5_SIG_WQE_SIZE )return MLX5_SIG_WQE_SIZE ;
@@ -3024,6 +3027,9 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,process_create_flag (dev, &create_flags,MLX5_IB_QP_CREATE_SQPN_QP1 , true, qp); +process_create_flag (dev, &create_flags,IB_QP_CREATE_CRYPTO_EN , + !!dev->crypto_caps.crypto_engines, qp); +if (create_flags) {mlx5_ib_dbg (dev,"Create QP has unsupported flags 0x%llX\n" , create_flags);
drivers/infiniband/hw/mlx5/wr.c +154 -10
278 lines not on this page · raw message
Signature · 1 line
2.39.0
PATCH 9/13 RDMA/mlx5: Add DEK management API
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add an API to manage Data Encryption Keys (DEKs). The API allows creating and destroying a DEK. DEKs allow encryption and decryption of transmitted data and are used in MKeys for crypto operations.
--- drivers/infiniband/hw/mlx5/crypto.c | 83 +++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/crypto.h | 8 +++ 2 files changed, 91 insertions(+)
drivers/infiniband/hw/mlx5/crypto.c +83 -0
@@ -3,6 +3,87 @@#include "crypto.h" +static struct ib_dek *mlx5r_create_dek (struct ib_pd *pd, +struct ib_dek_attr *attr) +{ +u32 in[MLX5_ST_SZ_DW (create_encryption_key_in)] = {}; +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)]; +struct mlx5_ib_dev *dev =to_mdev (pd->device); +u32 key_blob_size = attr->key_blob_size; +void *ptr, *key_addr; +struct ib_dek *dek; +u8 key_size; +int err; + +if (attr->key_type !=IB_CRYPTO_KEY_TYPE_AES_XTS ) +return ERR_PTR (-EOPNOTSUPP ); + +switch (key_blob_size) { +case MLX5_IB_CRYPTO_AES_128_XTS_KEY_SIZE : + key_size =MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_128 ; +break ; +case MLX5_IB_CRYPTO_AES_256_XTS_KEY_SIZE : + key_size =MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_256 ; +break ; +default : +return ERR_PTR (-EOPNOTSUPP ); + } + + dek =kzalloc (sizeof (*dek),GFP_KERNEL ); +if (!dek) +return ERR_PTR (-ENOMEM ); + + ptr =MLX5_ADDR_OF (create_encryption_key_in, in, + general_obj_in_cmd_hdr); +MLX5_SET (general_obj_in_cmd_hdr, ptr, opcode, +MLX5_CMD_OP_CREATE_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, ptr, obj_type, +MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY ); + ptr =MLX5_ADDR_OF (create_encryption_key_in, in, encryption_key_object); +MLX5_SET (encryption_key_obj, ptr, key_size, key_size); +MLX5_SET (encryption_key_obj, ptr, key_type, +MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_AES_XTS ); +MLX5_SET (encryption_key_obj, ptr, pd,to_mpd (pd)->pdn); + key_addr =MLX5_ADDR_OF (encryption_key_obj, ptr, key); +memcpy (key_addr, attr->key_blob, key_blob_size); + + err =mlx5_cmd_exec (dev->mdev, in,sizeof (in), out,sizeof (out)); +/* avoid leaking key on the stack */ +memzero_explicit (in,sizeof (in)); +if (err) + goto err_free; + + dek->id =MLX5_GET (general_obj_out_cmd_hdr, out, obj_id); + dek->pd = pd; + +return dek; + +err_free: +kfree (dek); +return ERR_PTR (err); +} + +static void mlx5r_destroy_dek (struct ib_dek *dek) +{ +struct mlx5_ib_dev *dev =to_mdev (dek->pd->device); +u32 in[MLX5_ST_SZ_DW (general_obj_in_cmd_hdr)] = {}; +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)]; + +MLX5_SET (general_obj_in_cmd_hdr, in, opcode, +MLX5_CMD_OP_DESTROY_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, in, obj_type, +MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY ); +MLX5_SET (general_obj_in_cmd_hdr, in, obj_id, dek->id); + +mlx5_cmd_exec (dev->mdev, in,sizeof (in), out,sizeof (out)); +kfree (dek); +} + +static const struct ib_device_ops mlx5r_dev_crypto_ops = { + .create_dek = mlx5r_create_dek, + .destroy_dek = mlx5r_destroy_dek, +}; +void mlx5r_crypto_caps_init (struct mlx5_ib_dev *dev) {struct ib_crypto_caps *caps = &dev->crypto_caps;
@@ -28,4 +109,6 @@ void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev) caps->crypto_engines |=IB_CRYPTO_ENGINES_CAP_AES_XTS ; caps->max_num_deks =1 <<MLX5_CAP_CRYPTO (mdev, log_max_num_deks); + +ib_set_device_ops (&dev->ib_dev, &mlx5r_dev_crypto_ops); }
drivers/infiniband/hw/mlx5/crypto.h +8 -0
@@ -6,6 +6,14 @@#include "mlx5_ib.h" +/* +* The standard AES-XTS key blob composed of two keys. +* AES-128-XTS key blob composed of two 128-bit keys, which is 32 bytes and +* AES-256-XTS key blob composed of two 256-bit keys, which is 64 bytes. +*/ +#define MLX5_IB_CRYPTO_AES_128_XTS_KEY_SIZE 32 +#define MLX5_IB_CRYPTO_AES_256_XTS_KEY_SIZE 64 +void mlx5r_crypto_caps_init (struct mlx5_ib_dev *dev);#endif /* _MLX5_IB_CRYPTO_H */
Signature · 1 line
2.39.0
PATCH 5/13 RDMA/core: Add DEK management API
In reply to Leon RomanovskyFrom: Israel Rukshin <[email protected]>
Add an API to manage Data Encryption Keys (DEKs). The API allows creating and destroying a DEK. DEKs allow encryption and decryption of transmitted data and are used in MKeys for crypto operations. A crypto setter for the MKey configuration API will be added in the following commit.
--- drivers/infiniband/core/device.c | 2 ++ drivers/infiniband/core/verbs.c | 32 +++++++++++++++++++++++++++ include/rdma/crypto.h | 38 ++++++++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 3 +++ 4 files changed, 75 insertions(+)
drivers/infiniband/core/device.c +2 -0
@@ -2615,6 +2615,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, create_ah);SET_DEVICE_OP (dev_ops, create_counters);SET_DEVICE_OP (dev_ops, create_cq); +SET_DEVICE_OP (dev_ops, create_dek);SET_DEVICE_OP (dev_ops, create_flow);SET_DEVICE_OP (dev_ops, create_qp);SET_DEVICE_OP (dev_ops, create_rwq_ind_table);
@@ -2632,6 +2633,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, destroy_ah);SET_DEVICE_OP (dev_ops, destroy_counters);SET_DEVICE_OP (dev_ops, destroy_cq); +SET_DEVICE_OP (dev_ops, destroy_dek);SET_DEVICE_OP (dev_ops, destroy_flow);SET_DEVICE_OP (dev_ops, destroy_flow_action);SET_DEVICE_OP (dev_ops, destroy_qp);
drivers/infiniband/core/verbs.c +32 -0
@@ -2306,6 +2306,38 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd, }EXPORT_SYMBOL (ib_alloc_mr_integrity); +/** +* ib_create_dek - Create a DEK (Data Encryption Key) associated with the +* specific protection domain. +* @pd: The protection domain associated with the DEK. +* @attr: The attributes of the DEK. +* +* Return: Allocated DEK in case of success; IS_ERR() is true in case of an +* error, PTR_ERR() returns the error code. +*/ +struct ib_dek *ib_create_dek (struct ib_pd *pd,struct ib_dek_attr *attr) +{ +struct ib_device *device = pd->device; + +if (!device->ops.create_dek || !device->ops.destroy_dek) +return ERR_PTR (-EOPNOTSUPP ); + +return device->ops.create_dek (pd, attr); +} +EXPORT_SYMBOL (ib_create_dek); + +/** +* ib_destroy_dek - Destroys the specified DEK. +* @dek: The DEK to destroy. +*/ +void ib_destroy_dek (struct ib_dek *dek) +{ +struct ib_device *device = dek->pd->device; + + device->ops.destroy_dek (dek); +} +EXPORT_SYMBOL (ib_destroy_dek); +/* Multicast groups */ static bool is_valid_mcast_lid (struct ib_qp *qp,u16 lid)
include/rdma/crypto.h +38 -0
@@ -34,4 +34,42 @@ struct ib_crypto_caps { u32 max_num_deks; }; +/** +* enum ib_crypto_key_type - Cryptographic key types +* @IB_CRYPTO_KEY_TYPE_AES_XTS: Key of type AES-XTS, which can be used when +* IB_CRYPTO_AES_XTS is supported. +*/ +enum ib_crypto_key_type { +IB_CRYPTO_KEY_TYPE_AES_XTS , +}; + +/** +* struct ib_dek_attr - Parameters for DEK (Data Encryption Key) +* @key_blob: the key blob that will be used for encryption and decryption of +* transmitted data. Actual size and layout of this field depends on the +* provided key_type and key_blob_size. +* The layout of AES_XTS key is: key1_128b + key2_128b or key1_256b + +* key2_256b. +* @key_blob_size: size of the key blob in bytes. +* @key_type: specific cryptographic key type. +*/ +struct ib_dek_attr { +const void *key_blob; +u32 key_blob_size; +enum ib_crypto_key_type key_type; +}; + +/** +* struct ib_dek - Data Encryption Key +* @pd: The protection domain associated with the DEK. +* @id: DEK identifier. +*/ +struct ib_dek { +struct ib_pd *pd; +u32 id; +}; + +struct ib_dek *ib_create_dek (struct ib_pd *pd,struct ib_dek_attr *attr); +void ib_destroy_dek (struct ib_dek *dek); +#endif /* _RDMA_CRYPTO_H_ */
include/rdma/ib_verbs.h +3 -0
@@ -2512,6 +2512,9 @@ struct ib_device_ops {struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,u32 max_num_data_sg,u32 max_num_meta_sg); +struct ib_dek *(*create_dek)(struct ib_pd *pd, +struct ib_dek_attr *attr); +void (*destroy_dek)(struct ib_dek *dek); int (*advise_mr)(struct ib_pd *pd, enum ib_uverbs_advise_mr_advice advice, u32 flags, struct ib_sge *sg_list, u32 num_sge,
Signature · 1 line
2.39.0
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
In reply to Leon Romanovskydrivers/infiniband/hw/mlx4/mlx4_ib.h enum mlx4_ib_qp_flags { · line 188
Quoted 11 lines · Leon Romanovsky
drivers/infiniband/hw/mlx4/mlx4_ib.h +1
-1@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, };
These should be moved to a uapi if we are saying they are userspace available
But I'm not sure they are?
include/rdma/ib_verbs.h enum ib_qp_type { · line 1145
Quoted 11 lines
include/rdma/ib_verbs.h +3
-0@@ -1140,16 +1140,15 @@ enum ib_qp_type { IB_QPT_RESERVED10, }; +/* +* bits 0, 5, 6 and 7 may be set by old kernels and should not be used. +*/
This is backwards "bits 0 5 6 7 were understood by older kernels and should not be used"
Quoted 14 lines
include/rdma/ib_verbs.h +0
-4enum ib_qp_create_flags { -IB_QP_CREATE_IPOIB_UD_LSO =1 <<0 ,IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK =IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK ,IB_QP_CREATE_CROSS_CHANNEL =1 <<2 ,IB_QP_CREATE_MANAGED_SEND =1 <<3 ,IB_QP_CREATE_MANAGED_RECV =1 <<4 , -IB_QP_CREATE_NETIF_QP =1 <<5 , -IB_QP_CREATE_INTEGRITY_EN =1 <<6 , -IB_QP_CREATE_NETDEV_USE =1 <<7 ,IB_QP_CREATE_SCATTER_FCS =IB_UVERBS_QP_CREATE_SCATTER_FCS , IB_QP_CREATE_CVLAN_STRIPPING =
include/rdma/ib_verbs.h enum ib_qp_create_flags { · line 1161
Quoted 5 lines
include/rdma/ib_verbs.h +1
-1@@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING ,/* reserve bits 26-31 for low level drivers' internal use */ IB_QP_CREATE_RESERVED_START =1 <<26 , - IB_QP_CREATE_RESERVED_END = 1 << 31, + IB_QP_CREATE_RESERVED_END = 1ULL << 31,
And these should be shifted to the uapi header
Jason
Re: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme controller
In reply to Leon RomanovskyQuoted 27 lines
drivers/nvme/host/core.c +3
-0@@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk, capacity =0 ; } +if (ctrl->crypto_enable) +blk_crypto_register (&ctrl->crypto_profile, disk->queue); +set_capacity_and_notify (disk, capacity);nvme_config_discard (disk, ns);drivers/nvme/host/nvme.h +1
-0@@ -16,6 +16,7 @@#include <linux/rcupdate.h> #include <linux/wait.h> #include <linux/t10-pi.h> +#include <linux/blk-crypto-profile.h> #include <trace/events/block.h>
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 379
Quoted 5 lines
drivers/nvme/host/nvme.h +2
-0@@ -374,6 +375,9 @@ struct nvme_ctrl {enum nvme_ctrl_type cntrltype;enum nvme_dctype dctype; + +bool crypto_enable;
why not decalre crypto_profile a pointer, allocate that at init controller and NULL check against that pointer instead of using an extra variable crypto_enable ?
e.g. :-
if (ctrl->crypto_profile) blk_crypto_register(ctrl->crypto_profile, disk->queue);
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 380
Quoted 1 line
drivers/nvme/host/nvme.h +1
-0+struct blk_crypto_profile crypto_profile;
you are increasing the size of struct nvme_ctrl unconditionally, why not guard above with CONFIG_BLK_INLINE_ENCRYPTION ?
-ck
Re: [PATCH rdma-next 11/13] nvme: Introduce a local variable
In reply to Leon RomanovskyQuoted 15 lines · Leon Romanovsky
From: Israel Rukshin <[email protected]>
The patch doesn't change any logic.
Signed-off-by Israel Rukshin <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/nvme/host/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c static void nvme_update_disk_info(struct gendisk *disk, · line 1874
Quoted 5 lines
drivers/nvme/host/core.c +1
-0@@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,sector_t capacity =nvme_lba_to_sect (ns,le64_to_cpu (id->nsze));unsigned short bs =1 << ns->lba_shift;u32 atomic_bs, phys_bs, io_opt =0 ; +struct nvme_ctrl *ctrl = ns->ctrl;
I don't think this patch is needed, existing code is more readable and gives much clarity that we are accessing ctrl from namespace since we are in nvme_update_disk_info() which is namespace based.
-ck
Re: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme controller
In reply to Chaitanya KulkarniQuoted 28 lines · Chaitanya Kulkarni
Quoted 28 lines
drivers/nvme/host/core.c +3
-0@@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk, capacity =0 ; } +if (ctrl->crypto_enable) +blk_crypto_register (&ctrl->crypto_profile, disk->queue); +set_capacity_and_notify (disk, capacity);nvme_config_discard (disk, ns);drivers/nvme/host/nvme.h +1
-0@@ -16,6 +16,7 @@#include <linux/rcupdate.h> #include <linux/wait.h> #include <linux/t10-pi.h> +#include <linux/blk-crypto-profile.h> #include <trace/events/block.h> @@ -374,6 +375,9 @@ struct nvme_ctrl {
Quoted 8 lines
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 379
Quoted 4 lines
drivers/nvme/host/nvme.h +2
-0enum nvme_ctrl_type cntrltype;enum nvme_dctype dctype; + +bool crypto_enable;why not decalre
crypto_profilea pointer, allocate that at init controller and NULL check against that pointer instead of using an extra variablecrypto_enable?
The embedded struct allows us to rely on container_of() macro.
377 static int nvme_rdma_crypto_keyslot_program(struct blk_crypto_profile *profile,
378 const struct blk_crypto_key *key,
379 unsigned int slot)
380 {
381 struct nvme_ctrl *nctrl =
382 container_of(profile, struct nvme_ctrl, crypto_profile);
Quoted 10 lines
e.g. :-
if (ctrl->crypto_profile) blk_crypto_register(ctrl->crypto_profile, disk->queue);drivers/nvme/host/nvme.h struct nvme_ctrl { · line 380
Quoted 1 line
drivers/nvme/host/nvme.h +1
-0+struct blk_crypto_profile crypto_profile;you are increasing the size of struct
nvme_ctrlunconditionally, why not guard above withCONFIG_BLK_INLINE_ENCRYPTION?
We can do it.
Thanks
Quoted 3 lines
-ck
Re: [PATCH rdma-next 11/13] nvme: Introduce a local variable
In reply to Chaitanya KulkarniQuoted 11 lines · Chaitanya Kulkarni
Quoted 10 lines · Leon Romanovsky
From: Israel Rukshin <[email protected]>
The patch doesn't change any logic.
Signed-off-by Israel Rukshin <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/nvme/host/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 14 lines
drivers/nvme/host/core.c static void nvme_update_disk_info(struct gendisk *disk, · line 1874
Quoted 10 lines
drivers/nvme/host/core.c +1
-0@@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,sector_t capacity =nvme_lba_to_sect (ns,le64_to_cpu (id->nsze));unsigned short bs =1 << ns->lba_shift;u32 atomic_bs, phys_bs, io_opt =0 ; +struct nvme_ctrl *ctrl = ns->ctrl;I don't think this patch is needed, existing code is more readable and gives much clarity that we are accessing ctrl from namespace since we are in
nvme_update_disk_info()which is namespace based.
I don't have strong opinion here.
Thanks
Quoted 3 lines
-ck
Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
In reply to Jason GunthorpeQuoted 17 lines · Jason Gunthorpe
drivers/infiniband/hw/mlx4/mlx4_ib.h enum mlx4_ib_qp_flags { · line 188
Quoted 11 lines · Leon Romanovsky
drivers/infiniband/hw/mlx4/mlx4_ib.h +1
-1@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, };These should be moved to a uapi if we are saying they are userspace available
Quoted 1 line
But I'm not sure they are?
I don't think so.
Quoted 10 lines
Quoted 10 lines
include/rdma/ib_verbs.h +0
-0@@ -1140,16 +1140,15 @@ enum ib_qp_type { IB_QPT_RESERVED10, };
Quoted 6 lines
include/rdma/ib_verbs.h enum ib_qp_type { · line 1145
Quoted 3 lines
include/rdma/ib_verbs.h +3
-0+/* +* bits 0, 5, 6 and 7 may be set by old kernels and should not be used. +*/ This is backwards "bits 0 5 6 7 were understood by older kernels and should not be used"
will change
Quoted 14 lines
Quoted 14 lines
include/rdma/ib_verbs.h +0
-4enum ib_qp_create_flags { -IB_QP_CREATE_IPOIB_UD_LSO =1 <<0 ,IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK =IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK ,IB_QP_CREATE_CROSS_CHANNEL =1 <<2 ,IB_QP_CREATE_MANAGED_SEND =1 <<3 ,IB_QP_CREATE_MANAGED_RECV =1 <<4 , -IB_QP_CREATE_NETIF_QP =1 <<5 , -IB_QP_CREATE_INTEGRITY_EN =1 <<6 , -IB_QP_CREATE_NETDEV_USE =1 <<7 ,IB_QP_CREATE_SCATTER_FCS =IB_UVERBS_QP_CREATE_SCATTER_FCS , IB_QP_CREATE_CVLAN_STRIPPING =
Quoted 8 lines
include/rdma/ib_verbs.h enum ib_qp_create_flags { · line 1161
Quoted 6 lines
include/rdma/ib_verbs.h +1
-1@@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING ,/* reserve bits 26-31 for low level drivers' internal use */ IB_QP_CREATE_RESERVED_START =1 <<26 , - IB_QP_CREATE_RESERVED_END = 1 << 31, + IB_QP_CREATE_RESERVED_END = 1ULL << 31,And these should be shifted to the uapi header
No problem.
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Jason Gunthorpe
drivers/infiniband/hw/mlx4/mlx4_ib.h enum mlx4_ib_qp_flags { · line 188
Quoted 11 lines · Leon Romanovsky
drivers/infiniband/hw/mlx4/mlx4_ib.h +1
-1@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, };These should be moved to a uapi if we are saying they are userspace available
Quoted 3 lines
Quoted 1 line
But I'm not sure they are?
I don't think so.
Then they should be > 32 bits right?
Jason
Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
Quoted 19 lines · Jason Gunthorpe
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Jason Gunthorpe
drivers/infiniband/hw/mlx4/mlx4_ib.h enum mlx4_ib_qp_flags { · line 188
Quoted 11 lines · Leon Romanovsky
drivers/infiniband/hw/mlx4/mlx4_ib.h +1
-1@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, };These should be moved to a uapi if we are saying they are userspace available
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
But I'm not sure they are?
I don't think so.
Then they should be > 32 bits right?
Right now, they are in reserved range:
/* reserve bits 26-31 for low level drivers' internal use */
IB_QP_CREATE_RESERVED_START = 1 << 26,
IB_QP_CREATE_RESERVED_END = 1ULL << 31,
If we move them to kernel part, we will need to define reserved range there too. So we just "burn" extra bits just for mlx4, also I don't see any reason to promote mlx4 bits to be general ones.
Thanks
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
Quoted 26 lines · Leon Romanovsky
Quoted 19 lines · Jason Gunthorpe
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Jason Gunthorpe
drivers/infiniband/hw/mlx4/mlx4_ib.h enum mlx4_ib_qp_flags { · line 188
Quoted 11 lines · Leon Romanovsky
drivers/infiniband/hw/mlx4/mlx4_ib.h +1
-1@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, };These should be moved to a uapi if we are saying they are userspace available
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
But I'm not sure they are?
I don't think so.
Then they should be > 32 bits right?
Quoted 8 lines
Right now, they are in reserved range:
/* reserve bits 26-31 for low level drivers' internal use */
IB_QP_CREATE_RESERVED_START= 1 << 26,
IB_QP_CREATE_RESERVED_END= 1ULL << 31,If we move them to kernel part, we will need to define reserved range there too. So we just "burn" extra bits just for mlx4, also I don't see any reason to promote mlx4 bits to be general ones.
Is the reserved range kernel only? It would be nice to clarify that detail
If yes we should move it so that userspace cannot set it. Do we have a bug here already?
Jason
Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
Quoted 36 lines · Jason Gunthorpe
Quoted 26 lines · Leon Romanovsky
Quoted 19 lines · Jason Gunthorpe
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Jason Gunthorpe
drivers/infiniband/hw/mlx4/mlx4_ib.h enum mlx4_ib_qp_flags { · line 188
Quoted 11 lines · Leon Romanovsky
drivers/infiniband/hw/mlx4/mlx4_ib.h +1
-1@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */ MLX4_IB_ROCE_V2_GSI_QP =MLX4_IB_QP_CREATE_ROCE_V2_GSI ,MLX4_IB_SRIOV_TUNNEL_QP =1 <<30 , - MLX4_IB_SRIOV_SQP = 1 << 31, + MLX4_IB_SRIOV_SQP = 1ULL << 31, };These should be moved to a uapi if we are saying they are userspace available
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
But I'm not sure they are?
I don't think so.
Then they should be > 32 bits right?
Quoted 8 lines
Right now, they are in reserved range:
/* reserve bits 26-31 for low level drivers' internal use */
IB_QP_CREATE_RESERVED_START= 1 << 26,
IB_QP_CREATE_RESERVED_END= 1ULL << 31,If we move them to kernel part, we will need to define reserved range there too. So we just "burn" extra bits just for mlx4, also I don't see any reason to promote mlx4 bits to be general ones.
Quoted 5 lines
Is the reserved range kernel only? It would be nice to clarify that detail
If yes we should move it so that userspace cannot set it. Do we have a bug here already?
No, we always checked that users can't provide these bits and fail create QP.
It means that we can safely move that range too.
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 06/13] RDMA/core: Introduce MR type for crypto operations
In reply to Leon RomanovskyQuoted 30 lines · On Mon, 16 Jan 2023 15:05:53 +0200 Leon Romanovsky <[email protected]> wrote:
index
17642aa54437..b6a3d82b89ca100644include/trace/events/rdma_core.h +23
-0@@ -371,6 +371,39 @@ TRACE_EVENT(mr_integ_alloc, __entry->max_num_meta_sg, __entry->rc) ); +TRACE_EVENT(mr_crypto_alloc, + TP_PROTO( + const struct ib_pd *pd, + u32 max_num_sg, + const struct ib_mr *mr + ), + + TP_ARGS(pd, max_num_sg, mr), + + TP_STRUCT__entry( + __field(u32, pd_id) + __field(u32, mr_id) + __field(u32, max_num_sg) + __field(int, rc) + ), + + TP_fast_assign( + __entry->pd_id = pd->res.id; + if (IS_ERR(mr)) { + __entry->mr_id = 0; + __entry->rc = PTR_ERR(mr); + } else { + __entry->mr_id = mr->res.id;
include/trace/events/rdma_core.h TRACE_EVENT(mr_integ_alloc, · line 406
Quoted 10 lines
include/trace/events/rdma_core.h +10
-0+ __entry->rc =0 ; + } + __entry->max_num_sg = max_num_sg; + ), + +TP_printk ("pd.id=%u mr.id=%u max_num_sg=%u rc=%d" , + __entry->pd_id, __entry->mr_id, __entry->max_num_sg, + __entry->rc) +); +
-- Steve
Quoted 3 lines
include/trace/events/rdma_core.h +0
-0TRACE_EVENT(mr_dereg, TP_PROTO( const struct ib_mr *mr
Hi Leon,
Quoted 28 lines · Leon Romanovsky
Quoted 1 line
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
How to configure fscrypt to enable plaintext keys:
# mkfs.ext4 -O encrypt /dev/nvme0n1 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt # head -c 64 /dev/urandom > /tmp/master_key # fscryptctl add_key /mnt/crypto/ < /tmp/master_key # mkdir /mnt/crypto/test1 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1 ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the “fscryptctl add_key” command. # echo foo > /mnt/crypto/test1/fooNotes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
Quoted 3 lines
There is no special configuration to support crypto at nvme modules.
Thanks
Very interesting work! Can you Cc me on future versions?
I'm glad to see that this hardware allows all 16 IV bytes to be specified.
Does it also handle programming and evicting keys efficiently?
Also, just checking: have you tested that the ciphertext that this inline encryption hardware produces is correct? That's always super important to test. There are xfstests that test for it, e.g. generic/582. Another way to test it is to just manually test whether encrypted files that were created when the filesystem was mounted with '-o inlinecrypt' show the same contents when the filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
- Eric
Eric,
Quoted 16 lines
Quoted 9 lines
Notes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.There is no special configuration to support crypto at nvme modules.
Thanks
Very interesting work! Can you Cc me on future versions?
I'm glad to see that this hardware allows all 16 IV bytes to be specified.
Does it also handle programming and evicting keys efficiently?
Quoted 9 lines
Also, just checking: have you tested that the ciphertext that this inline encryption hardware produces is correct? That's always super important to test. There are xfstests that test for it, e.g. generic/582. Another way to test it is to just manually test whether encrypted files that were created when the filesystem was mounted with '-o inlinecrypt' show the same contents when the filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
- Eric
I'm wondering which are the xfstests that needs to run in order to establish the correctness/stability apart from generic/582 this work ?
-ck
Quoted 25 lines · Chaitanya Kulkarni
Eric,
Quoted 23 lines
Quoted 9 lines
Notes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.There is no special configuration to support crypto at nvme modules.
Thanks
Very interesting work! Can you Cc me on future versions?
I'm glad to see that this hardware allows all 16 IV bytes to be specified.
Does it also handle programming and evicting keys efficiently?
Also, just checking: have you tested that the ciphertext that this inline encryption hardware produces is correct? That's always super important to test. There are xfstests that test for it, e.g. generic/582. Another way to test it is to just manually test whether encrypted files that were created when the filesystem was mounted with '-o inlinecrypt' show the same contents when the filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
Quoted 7 lines
Quoted 2 lines
- Eric
I'm wondering which are the xfstests that needs to run in order to establish the correctness/stability apart from generic/582 this work ?
See https://docs.kernel.org/filesystems/fscrypt.html#tests.
- Eric
Christoph Hellwig
maintainerQuoted 10 lines · Leon Romanovsky
Quoted 1 line
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
One thing that needs to be solved before we can look into this is the interaction with protection information (or integrity data in Linux terms). Currently inline encryption and protection information are mutally incompatible.
Quoted 31 lines · Eric Biggers
Hi Leon,
Quoted 28 lines · Leon Romanovsky
Quoted 1 line
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
How to configure fscrypt to enable plaintext keys:
# mkfs.ext4 -O encrypt /dev/nvme0n1 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt # head -c 64 /dev/urandom > /tmp/master_key # fscryptctl add_key /mnt/crypto/ < /tmp/master_key # mkdir /mnt/crypto/test1 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1 ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the “fscryptctl add_key” command. # echo foo > /mnt/crypto/test1/fooNotes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
Quoted 5 lines
Quoted 3 lines
There is no special configuration to support crypto at nvme modules.
Thanks
Very interesting work! Can you Cc me on future versions?
Sure
Quoted 4 lines
I'm glad to see that this hardware allows all 16 IV bytes to be specified.
Does it also handle programming and evicting keys efficiently?
"efficiently" is a very subjective term. We are using FW command interface to program keys and this interface can do hundreds/thousands commands per-second.
Thanks
Hi Eric,
Quoted 38 lines · Eric Biggers
Hi Leon,
Quoted 28 lines · Leon Romanovsky
Quoted 1 line
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
How to configure fscrypt to enable plaintext keys:
# mkfs.ext4 -O encrypt /dev/nvme0n1 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt # head -c 64 /dev/urandom > /tmp/master_key # fscryptctl add_key /mnt/crypto/ < /tmp/master_key # mkdir /mnt/crypto/test1 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1 ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the “fscryptctl add_key” command. # echo foo > /mnt/crypto/test1/fooNotes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.Quoted 3 lines
There is no special configuration to support crypto at nvme modules.
Thanks
Very interesting work! Can you Cc me on future versions?
I'm glad to see that this hardware allows all 16 IV bytes to be specified.
Quoted 8 lines
Does it also handle programming and evicting keys efficiently?
Also, just checking: have you tested that the ciphertext that this inline encryption hardware produces is correct? That's always super important to test. There are xfstests that test for it, e.g. generic/582. Another way to test it is to just manually test whether encrypted files that were created when the filesystem was mounted with '-o inlinecrypt' show the same contents when the filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
sure, I ran the manual test of comparing the encrypted files content with and without the '-o inlinecrypt' at the mount command.
Quoted 2 lines
- Eric
- Israel
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Leon Romanovsky
Quoted 1 line
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
Quoted 8 lines
Quoted 4 lines
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
One thing that needs to be solved before we can look into this is the interaction with protection information (or integrity data in Linux terms). Currently inline encryption and protection information are mutally incompatible.
Well, for sure this should be solved. Not sure that it should be done before this series.
This patch set doesn't break/change existing behavior of PI and Encryption offloads so the above can be done incrementally. It's already big enough series and 2 subsystems are involved.
Today, if one would like to run both features using a capable device then the PI will be offloaded (no SW fallback) and the Encryption will be using SW fallback. There should be a serious instrumentation in the block layer to make both operations offloaded.
We'll start looking into it. Any suggestions and designs are welcomed.
We also prepared patches to extend the blk lavel encryption (in addition to fscrypt exist today).
Sagi Grimberg
maintainerQuoted 23 lines
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
How to configure fscrypt to enable plaintext keys:
# mkfs.ext4 -O encrypt /dev/nvme0n1 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt # head -c 64 /dev/urandom > /tmp/master_key # fscryptctl add_key /mnt/crypto/ < /tmp/master_key # mkdir /mnt/crypto/test1 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1 ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the “fscryptctl add_key” command. # echo foo > /mnt/crypto/test1/foo
Quoted 7 lines
Notes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.There is no special configuration to support crypto at nvme modules.
Hey, this looks sane to me in a very first glance.
Few high level questions:
- what happens with multipathing? when if not all devices are capable. SW fallback?
- Does the crypt stuff stay intact when bio is requeued?
I'm assuming you tested this with multipathing? This is not very useful if it is incompatible with it.
Hi Sagi,
Quoted 29 lines · Sagi Grimberg
Quoted 29 lines
From Israel,
The purpose of this patchset is to add support for inline encryption/decryption of the data at storage protocols like nvmf over RDMA (at a similar way like integrity is used via unique mkey).
This patchset adds support for plaintext keys. The patches were tested on BF-3 HW with fscrypt tool to test this feature, which showed reduce in CPU utilization when comparing at 64k or more IO size. The CPU utilization was improved by more than 50% comparing to the SW only solution at this case.
How to configure fscrypt to enable plaintext keys:
# mkfs.ext4 -O encrypt /dev/nvme0n1 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt # head -c 64 /dev/urandom > /tmp/master_key # fscryptctl add_key /mnt/crypto/ < /tmp/master_key # mkdir /mnt/crypto/test1 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1 ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the “fscryptctl add_key” command. # echo foo > /mnt/crypto/test1/fooNotes:
- At plaintext mode only, the user set a master key and the fscrypt driver derived from it the DEK and the key identifier.
-152c41b2ea39fa3d90ea06448456e7fbis the derived key identifier
- Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
Quoted 7 lines
Quoted 1 line
There is no special configuration to support crypto at nvme modules.
Hey, this looks sane to me in a very first glance.
Few high level questions:
- what happens with multipathing? when if not all devices are capable. SW fallback?
SW fallback happens every time the device doesn't support the specific crypto request (which include data-unit-size, mode and dun_bytes). So with multipathing, one path uses the HW crypto offload and the other one uses the SW fallback.
Quoted 1 line
- Does the crypt stuff stay intact when bio is requeued?
Yes, the crypto ctx is copied when cloning the bio.
Quoted 3 lines
I'm assuming you tested this with multipathing? This is not very useful if it is incompatible with it.
Yes, sure.
You can see the call to blk_crypto_reprogram_all_keys(), which is called when the controller was reconnected after port toggling.
- Israel
Christoph Hellwig
maintainerQuoted 3 lines · Max Gurtovoy
Today, if one would like to run both features using a capable device then the PI will be offloaded (no SW fallback) and the Encryption will be using SW fallback.
It's not just running both - it's offering both as currently registering these fatures is mutally incompatible.
Christoph Hellwig
maintainerQuoted 6 lines · Israel Rukshin
Quoted 2 lines
- what happens with multipathing? when if not all devices are capable. SW fallback?
SW fallback happens every time the device doesn't support the specific crypto request (which include data-unit-size, mode and
dun_bytes). So with multipathing, one path uses the HW crypto offload and the other one uses the SW fallback.
That's a big no-go. The blk-crypto-fallback code is just a toy example and not actually safe to use in prodution. Most importantly it just kmallocs a bio clone and pages for it without any mempool that guarantees forward progress.
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Max Gurtovoy
Today, if one would like to run both features using a capable device then the PI will be offloaded (no SW fallback) and the Encryption will be using SW fallback.
It's not just running both - it's offering both as currently registering these fatures is mutally incompatible.
For sure we would like to offer both, but as mentioned before, this will require blk layer instrumentation and ib_core instrumentation to some pipeline_mr or something like that. We are interested in doing this IO path but doing all in 1 series doesn't sounds possible.
First we would like to have at least the mutual exclusive solution since there are users that may want only one type of acceleration.
re-designing, both blk layer and ib_core for the PI + CRYPTO case in this series is not feasible IMO.
Sagi Grimberg
maintainerQuoted 8 lines
Quoted 6 lines
Quoted 4 lines
Today, if one would like to run both features using a capable device then the PI will be offloaded (no SW fallback) and the Encryption will be using SW fallback.
It's not just running both - it's offering both as currently registering these fatures is mutally incompatible.
For sure we would like to offer both, but as mentioned before, this will require blk layer instrumentation and
ib_coreinstrumentation to somepipeline_mror something like that.
Can you explain what is needed in the block layer that prevents queueing a request that has PI and crypto ctx?
As for the rdma portion, I don't know enough if this is something that is supported by the HW and lacks a SW interface, or inherently unsupported in HW...