Basic inline encryption support for ufs-exynos
Subsystems (5)
Eric Biggers
maintainerv3 cover Basic inline encryption support for ufs-exynos
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
Tested on gs101 (specifically Pixel 6) by running the 'encrypt' group of xfstests on a filesystem mounted with the 'inlinecrypt' mount option.
This patchset applies to v6.10-rc6, and it has no prerequisites that aren't already upstream.
Changed in v3:
- Made the FMP support depend on EXYNOS_UFS_OPT_UFSPR_SECURE, since the !EXYNOS_UFS_OPT_UFSPR_SECURE case has not yet been tested
- Replaced LOG2_DATA_UNIT_SIZE with ilog2(DATA_UNIT_SIZE)
- Added Reviewed-by tags
Changed in v2:
- Added DATA_UNIT_SIZE macro
- Changed a comment into kerneldoc
- Used ARM_SMCCC_CALL_VAL() to define SMC codes
- Used arm_smccc_smc() directly instead of via a wrapper function
Eric Biggers (6):
scsi: ufs: core: Add UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE
scsi: ufs: core: fold ufshcd_clear_keyslot() into its caller
scsi: ufs: core: Add UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE
scsi: ufs: core: Add fill_crypto_prdt variant op
scsi: ufs: core: Add UFSHCD_QUIRK_KEYS_IN_PRDT
scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)
drivers/ufs/core/ufshcd-crypto.c | 34 +++-- drivers/ufs/core/ufshcd-crypto.h | 36 +++++ drivers/ufs/core/ufshcd.c | 3 +- drivers/ufs/host/ufs-exynos.c | 240 ++++++++++++++++++++++++++++++- include/ufs/ufshcd.h | 28 ++++ 5 files changed, 320 insertions(+), 21 deletions(-)
base-commit: 22a40d14b572deb80c0648557f4bd502d7e83826
Signature · 1 line
2.45.2
Eric Biggers
maintainerPATCH v3 1/6 scsi: ufs: core: Add UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE
From: Eric Biggers <[email protected]>
Add UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE which lets UFS host drivers initialize the blk_crypto_profile themselves rather than have it be initialized by ufshcd-core according to the UFSHCI standard. This is needed to support inline encryption on the "Exynos" UFS controller which has a nonstandard interface.
--- drivers/ufs/core/ufshcd-crypto.c | 10 +++++++--- include/ufs/ufshcd.h | 9 +++++++++ 2 files changed, 16 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd-crypto.c +7 -3
@@ -157,10 +157,13 @@ int ufshcd_hba_init_crypto_capabilities(struct ufs_hba *hba) {int cap_idx;int err =0 ;enum blk_crypto_mode_num blk_mode_num; +if (hba->quirks &UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE ) +return 0 ; +/* * Don't use crypto if either the hardware doesn't advertise the * standard crypto capability bit *or* if the vendor specific driver * hasn't advertised that crypto is supported. */
@@ -226,13 +229,14 @@ void ufshcd_init_crypto(struct ufs_hba *hba)int slot;if (!(hba->caps &UFSHCD_CAP_CRYPTO ))return ; -/* Clear all keyslots - the number of keyslots is (CFGC + 1) */ -for (slot =0 ; slot < hba->crypto_capabilities.config_count +1 ; slot++) -ufshcd_clear_keyslot (hba, slot); +/* Clear all keyslots. */ +for (slot =0 ; slot < hba->crypto_profile.num_slots; slot++) + hba->crypto_profile.ll_ops.keyslot_evict (&hba->crypto_profile, +NULL , slot); }void ufshcd_crypto_register (struct ufs_hba *hba,struct request_queue *q) {if (hba->caps &UFSHCD_CAP_CRYPTO )
include/ufs/ufshcd.h +9 -0
@@ -641,10 +641,19 @@ enum ufshcd_quirks {/* * Some host does not implement SQ Run Time Command (SQRTC) register * thus need this quirk to skip related flow. */ UFSHCD_QUIRK_MCQ_BROKEN_RTC =1 <<21 , + +/* +* This quirk needs to be enabled if the host controller supports inline +* encryption but it needs to initialize the crypto capabilities in a +* nonstandard way and/or needs to override blk_crypto_ll_ops. If +* enabled, the standard code won't initialize the blk_crypto_profile; +* ufs_hba_variant_ops::init() must do it instead. +*/ + UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE = 1 << 22, };enum ufshcd_caps {/* Allow dynamic clk gating */ UFSHCD_CAP_CLK_GATING = 1 << 0,
Signature · 1 line
2.45.2
RE: [PATCH v3 1/6] scsi: ufs: core: Add UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE
Quoted 23 lines
-----Original Message-----
From: Eric Biggers <[email protected]>
Sent: Tuesday, July 9, 2024 5:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; Alim Akhtar <[email protected]>; Avri Altman <[email protected]>; Bart Van Assche <[email protected]>; Martin K . Petersen <[email protected]>; Peter Griffin <[email protected]>; André Draszik <[email protected]>; William McVicker <[email protected]>
Subject: [PATCH v3 1/6] scsi: ufs: core: AddUFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILEFrom: Eric Biggers <[email protected]>
Add
UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILEwhich lets UFS host drivers initialize theblk_crypto_profilethemselves rather than have it be initialized by ufshcd-core according to the UFSHCI standard. This is needed to support inline encryption on the "Exynos" UFS controller which has a nonstandard interface.
Quoted 4 lines
Reviewed-by Bart Van Assche <[email protected]>Reviewed-by Peter Griffin <[email protected]>Signed-off-by Eric Biggers <[email protected]>---
Quoted 4 lines
drivers/ufs/core/ufshcd-crypto.c | 10 +++++++--- include/ufs/ufshcd.h | 9 +++++++++ 2 files changed, 16 insertions(+), 3 deletions(-)
[Snip] = 1 << 0,
Quoted 2 lines
--
2.45.2
Eric Biggers
maintainerPATCH v3 2/6 scsi: ufs: core: fold ufshcd_clear_keyslot() into its caller
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Fold ufshcd_clear_keyslot() into its only remaining caller.
--- drivers/ufs/core/ufshcd-crypto.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
drivers/ufs/core/ufshcd-crypto.c +5 -11
@@ -93,31 +93,25 @@ static int ufshcd_crypto_keyslot_program(struct blk_crypto_profile *profile,memzero_explicit (&cfg,sizeof (cfg));return err; } -static int ufshcd_clear_keyslot (struct ufs_hba *hba,int slot) +static int ufshcd_crypto_keyslot_evict (struct blk_crypto_profile *profile, +const struct blk_crypto_key *key, +unsigned int slot) { +struct ufs_hba *hba = + container_of(profile, struct ufs_hba, crypto_profile);/* * Clear the crypto cfg on the device. Clearing CFGE * might not be sufficient, so just clear the entire cfg. */ union ufs_crypto_cfg_entry cfg = {};return ufshcd_program_key (hba, &cfg, slot); } -static int ufshcd_crypto_keyslot_evict (struct blk_crypto_profile *profile, -const struct blk_crypto_key *key, -unsigned int slot) -{ -struct ufs_hba *hba = - container_of(profile, struct ufs_hba, crypto_profile); - -return ufshcd_clear_keyslot (hba, slot); -} -bool ufshcd_crypto_enable (struct ufs_hba *hba) {if (!(hba->caps &UFSHCD_CAP_CRYPTO ))return false;
Signature · 1 line
2.45.2
RE: [PATCH v3 2/6] scsi: ufs: core: fold ufshcd_clear_keyslot() into its caller
Quoted 15 lines
-----Original Message-----
From: Eric Biggers <[email protected]>
Sent: Tuesday, July 9, 2024 5:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; Alim Akhtar <[email protected]>; Avri Altman <[email protected]>; Bart Van Assche <[email protected]>; Martin K . Petersen <[email protected]>; Peter Griffin <[email protected]>; André Draszik <[email protected]>; William McVicker <[email protected]>
Subject: [PATCH v3 2/6] scsi: ufs: core: foldufshcd_clear_keyslot()into its caller
Quoted 8 lines
From: Eric Biggers <[email protected]>
Fold
ufshcd_clear_keyslot()into its only remaining caller.Reviewed-by Bart Van Assche <[email protected]>Reviewed-by Peter Griffin <[email protected]>Signed-off-by Eric Biggers <[email protected]>---
Quoted 46 lines
drivers/ufs/core/ufshcd-crypto.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)drivers/ufs/core/ufshcd- renamed from drivers/ufs/core/ufshcd-crypto.c +0
-0
crypto.c
indexdebc925ae439..b4980fd91cee100644drivers/ufs/core/ufshcd-crypto.c +5
-12@@ -93,31 +93,25 @@ static int ufshcd_crypto_keyslot_program(struct blk_crypto_profile *profile,memzero_explicit (&cfg,sizeof (cfg));return err; } -static int ufshcd_clear_keyslot (struct ufs_hba *hba,int slot) +static int ufshcd_crypto_keyslot_evict (struct blk_crypto_profile *profile, +const struct blk_crypto_key *key, +unsigned int slot) { +struct ufs_hba *hba = + container_of(profile, struct ufs_hba, crypto_profile);/* * Clear the crypto cfg on the device. Clearing CFGE * might not be sufficient, so just clear the entire cfg. */ union ufs_crypto_cfg_entry cfg = {};return ufshcd_program_key (hba, &cfg, slot); } -static int ufshcd_crypto_keyslot_evict (struct blk_crypto_profile *profile, -const struct blk_crypto_key *key, -unsigned int slot) -{ -struct ufs_hba *hba = - container_of(profile, struct ufs_hba, crypto_profile); - -return ufshcd_clear_keyslot (hba, slot); -} -bool ufshcd_crypto_enable (struct ufs_hba *hba) {if (!(hba->caps &UFSHCD_CAP_CRYPTO ))return false; --2.45.2
Eric Biggers
maintainerPATCH v3 3/6 scsi: ufs: core: Add UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE which tells the UFS core to not use the crypto enable bit defined by the UFS specification. This is needed to support inline encryption on the "Exynos" UFS controller.
--- drivers/ufs/core/ufshcd-crypto.c | 8 ++++++++ include/ufs/ufshcd.h | 7 +++++++ 2 files changed, 15 insertions(+)
drivers/ufs/core/ufshcd-crypto.c +8 -0
@@ -108,17 +108,25 @@ static int ufshcd_crypto_keyslot_evict(struct blk_crypto_profile *profile,union ufs_crypto_cfg_entry cfg = {};return ufshcd_program_key (hba, &cfg, slot); } +/* +* Reprogram the keyslots if needed, and return true if CRYPTO_GENERAL_ENABLE +* should be used in the host controller initialization sequence. +*/ bool ufshcd_crypto_enable (struct ufs_hba *hba) {if (!(hba->caps &UFSHCD_CAP_CRYPTO ))return false;/* Reset might clear all keys, so reprogram all the keys. */ blk_crypto_reprogram_all_keys (&hba->crypto_profile); + +if (hba->quirks &UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE ) +return false; +return true; }static const struct blk_crypto_ll_ops ufshcd_crypto_ops = { .keyslot_program = ufshcd_crypto_keyslot_program,
include/ufs/ufshcd.h +7 -0
@@ -650,10 +650,17 @@ enum ufshcd_quirks {* nonstandard way and/or needs to override blk_crypto_ll_ops. If * enabled, the standard code won't initialize the blk_crypto_profile; * ufs_hba_variant_ops::init() must do it instead. */ UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE =1 <<22 , + +/* +* This quirk needs to be enabled if the host controller supports inline +* encryption but does not support the CRYPTO_GENERAL_ENABLE bit, i.e. +* host controller initialization fails if that bit is set. +*/ + UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE = 1 << 23, };enum ufshcd_caps {/* Allow dynamic clk gating */ UFSHCD_CAP_CLK_GATING = 1 << 0,
Signature · 1 line
2.45.2
RE: [PATCH v3 3/6] scsi: ufs: core: Add UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE
Quoted 17 lines
-----Original Message-----
From: Eric Biggers <[email protected]>
Sent: Tuesday, July 9, 2024 5:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; Alim Akhtar <[email protected]>; Avri Altman <[email protected]>; Bart Van Assche <[email protected]>; Martin K . Petersen <[email protected]>; Peter Griffin <[email protected]>; André Draszik <[email protected]>; William McVicker <[email protected]>
Subject: [PATCH v3 3/6] scsi: ufs: core: AddUFSHCD_QUIRK_BROKEN_CRYPTO_ENABLEFrom: Eric Biggers <[email protected]>
Quoted 8 lines
Add
UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLEwhich tells the UFS core to not use the crypto enable bit defined by the UFS specification. This is needed to support inline encryption on the "Exynos" UFS controller.Reviewed-by Bart Van Assche <[email protected]>Reviewed-by Peter Griffin <[email protected]>Signed-off-by Eric Biggers <[email protected]>---
Quoted 60 lines
drivers/ufs/core/ufshcd-crypto.c | 8 ++++++++ include/ufs/ufshcd.h | 7 +++++++ 2 files changed, 15 insertions(+)drivers/ufs/core/ufshcd- renamed from drivers/ufs/core/ufshcd-crypto.c +0
-0
crypto.c
indexb4980fd91cee..a714dad82cd1100644drivers/ufs/core/ufshcd-crypto.c +9
-0@@ -108,17 +108,25 @@ static int ufshcd_crypto_keyslot_evict(structblk_crypto_profile *profile, union ufs_crypto_cfg_entry cfg = {};return ufshcd_program_key (hba, &cfg, slot); } +/* +* Reprogram the keyslots if needed, and return true if +CRYPTO_GENERAL_ENABLE +* should be used in the host controller initialization sequence. +*/ bool ufshcd_crypto_enable (struct ufs_hba *hba) {if (!(hba->caps &UFSHCD_CAP_CRYPTO ))return false;/* Reset might clear all keys, so reprogram all the keys. */ blk_crypto_reprogram_all_keys (&hba->crypto_profile); + +if (hba->quirks &UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE ) +return false; +return true; }static const struct blk_crypto_ll_ops ufshcd_crypto_ops = { .keyslot_program = ufshcd_crypto_keyslot_program,include/ufs/ufshcd.h index renamed from include/ufs/ufshcd.h +0
-0
b354a7eee478..4b7ad23a4420100644include/ufs/ufshcd.h +4
-0@@ -650,10 +650,17 @@ enum ufshcd_quirks {* nonstandard way and/or needs to override blk_crypto_ll_ops. If * enabled, the standard code won't initialize the blk_crypto_profile; * ufs_hba_variant_ops::init() must do it instead. */ UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE = 1 << 22, + +/* +* This quirk needs to be enabled if the host controller supports inline +* encryption but does not support the CRYPTO_GENERAL_ENABLE bit, i.e.
include/ufs/ufshcd.h +3
-1+* host controller initialization fails if that bit is set. +*/ + UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE = 1 << 23, };enum ufshcd_caps {/* Allow dynamic clk gating */ UFSHCD_CAP_CLK_GATING =1 <<0 , --2.45.2
Eric Biggers
maintainerPATCH v3 4/6 scsi: ufs: core: Add fill_crypto_prdt variant op
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add a variant op to allow host drivers to initialize nonstandard crypto-related fields in the PRDT. This is needed to support inline encryption on the "Exynos" UFS controller.
Note that this will be used together with the support for overriding the PRDT entry size that was already added by commit ada1e653a5ea ("scsi:
ufs: core: Allow UFS host drivers to override the sg entry size").
--- drivers/ufs/core/ufshcd-crypto.h | 19 +++++++++++++++++++ drivers/ufs/core/ufshcd.c | 2 +- include/ufs/ufshcd.h | 4 ++++ 3 files changed, 24 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd-crypto.h +19 -0
@@ -35,10 +35,23 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp, h->cci = lrbp->crypto_key_slot; h->dunl =cpu_to_le32 (lower_32_bits (lrbp->data_unit_num)); h->dunu =cpu_to_le32 (upper_32_bits (lrbp->data_unit_num)); } +static inline int ufshcd_crypto_fill_prdt (struct ufs_hba *hba, +struct ufshcd_lrb *lrbp) +{ +struct scsi_cmnd *cmd = lrbp->cmd; +const struct bio_crypt_ctx *crypt_ctx =scsi_cmd_to_rq (cmd)->crypt_ctx; + +if (crypt_ctx && hba->vops && hba->vops->fill_crypto_prdt) +return hba->vops->fill_crypto_prdt (hba, crypt_ctx, + lrbp->ucd_prdt_ptr, +scsi_sg_count (cmd)); +return 0 ; +} +bool ufshcd_crypto_enable (struct ufs_hba *hba);int ufshcd_hba_init_crypto_capabilities (struct ufs_hba *hba);void ufshcd_init_crypto (struct ufs_hba *hba);
@@ -52,10 +65,16 @@ static inline void ufshcd_prepare_lrbp_crypto(struct request *rq,static inline void ufshcd_prepare_req_desc_hdr_crypto (struct ufshcd_lrb *lrbp,struct request_desc_header *h) { } +static inline int ufshcd_crypto_fill_prdt (struct ufs_hba *hba, +struct ufshcd_lrb *lrbp) +{ +return 0 ; +} +static inline bool ufshcd_crypto_enable (struct ufs_hba *hba) {return false; }
drivers/ufs/core/ufshcd.c +1 -1
@@ -2634,11 +2634,11 @@ static int ufshcd_map_sg(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)if (sg_segments <0 )return sg_segments;ufshcd_sgl_to_prdt (hba, lrbp, sg_segments,scsi_sglist (cmd)); -return 0 ; +return ufshcd_crypto_fill_prdt (hba, lrbp); }/** * ufshcd_enable_intr - enable interrupts * @hba: per adapter instance
include/ufs/ufshcd.h +4 -0
@@ -319,10 +319,11 @@ struct ufs_pwr_mode_info {* @dbg_register_dump: used to dump controller debug information * @phy_initialization: used to initialize phys * @device_reset: called to issue a reset pulse on the UFS device * @config_scaling_param: called to configure clock scaling parameters * @program_key: program or evict an inline encryption key +* @fill_crypto_prdt: initialize crypto-related fields in the PRDT * @event_notify: called to notify important events * @reinit_notify: called to notify reinit of UFSHCD during max gear switch * @mcq_config_resource: called to configure MCQ platform resources * @get_hba_mac: called to get vendor specific mac value, mandatory for mcq mode * @op_runtime_config: called to config Operation and runtime regs Pointers
@@ -363,10 +364,13 @@ struct ufs_hba_variant_ops {void (*config_scaling_param)(struct ufs_hba *hba,struct devfreq_dev_profile *profile,struct devfreq_simple_ondemand_data *data);int (*program_key)(struct ufs_hba *hba,const union ufs_crypto_cfg_entry *cfg,int slot); +int (*fill_crypto_prdt)(struct ufs_hba *hba, +const struct bio_crypt_ctx *crypt_ctx, +void *prdt,unsigned int num_segments);void (*event_notify)(struct ufs_hba *hba,enum ufs_event_type evt,void *data);void (*reinit_notify)(struct ufs_hba *);int (*mcq_config_resource)(struct ufs_hba *hba);int (*get_hba_mac)(struct ufs_hba *hba);
Signature · 1 line
2.45.2
RE: [PATCH v3 4/6] scsi: ufs: core: Add fill_crypto_prdt variant op
Quoted 20 lines
-----Original Message-----
From: Eric Biggers <[email protected]>
Sent: Tuesday, July 9, 2024 5:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; Alim Akhtar <[email protected]>; Avri Altman <[email protected]>; Bart Van Assche <[email protected]>; Martin K . Petersen <[email protected]>; Peter Griffin <[email protected]>; André Draszik <[email protected]>; William McVicker <[email protected]>
Subject: [PATCH v3 4/6] scsi: ufs: core: Addfill_crypto_prdtvariant opFrom: Eric Biggers <[email protected]>
Add a variant op to allow host drivers to initialize nonstandard crypto-related fields in the PRDT. This is needed to support inline encryption on the "Exynos" UFS controller.
Quoted 8 lines
Note that this will be used together with the support for overriding the PRDT entry size that was already added by commit
ada1e653a5ea("scsi:
ufs: core: Allow UFS host drivers to override the sg entry size").Reviewed-by Bart Van Assche <[email protected]>Reviewed-by Peter Griffin <[email protected]>Signed-off-by Eric Biggers <[email protected]>---
Eric Biggers
maintainerPATCH v3 5/6 scsi: ufs: core: Add UFSHCD_QUIRK_KEYS_IN_PRDT
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Since the nonstandard inline encryption support on Exynos SoCs requires that raw cryptographic keys be copied into the PRDT, it is desirable to zeroize those keys after each request to keep them from being left in memory. Therefore, add a quirk bit that enables the zeroization.
We could instead do the zeroization unconditionally. However, using a quirk bit avoids adding the zeroization overhead to standard devices.
--- drivers/ufs/core/ufshcd-crypto.h | 17 +++++++++++++++++ drivers/ufs/core/ufshcd.c | 1 + include/ufs/ufshcd.h | 8 ++++++++ 3 files changed, 26 insertions(+)
drivers/ufs/core/ufshcd-crypto.h +17 -0
@@ -48,10 +48,24 @@ static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba, lrbp->ucd_prdt_ptr, scsi_sg_count(cmd));return 0 ; } +static inline void ufshcd_crypto_clear_prdt (struct ufs_hba *hba, +struct ufshcd_lrb *lrbp) +{ +if (!(hba->quirks &UFSHCD_QUIRK_KEYS_IN_PRDT )) +return ; + +if (!(scsi_cmd_to_rq (lrbp->cmd)->crypt_ctx)) +return ; + +/* Zeroize the PRDT because it can contain cryptographic keys. */ +memzero_explicit (lrbp->ucd_prdt_ptr, +ufshcd_sg_entry_size (hba) *scsi_sg_count (lrbp->cmd)); +} +bool ufshcd_crypto_enable (struct ufs_hba *hba);int ufshcd_hba_init_crypto_capabilities (struct ufs_hba *hba);void ufshcd_init_crypto (struct ufs_hba *hba);
@@ -71,10 +85,13 @@ static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) {return 0 ; } +static inline void ufshcd_crypto_clear_prdt (struct ufs_hba *hba, +struct ufshcd_lrb *lrbp) { } +static inline bool ufshcd_crypto_enable (struct ufs_hba *hba) {return false; }
drivers/ufs/core/ufshcd.c +1 -0
@@ -5472,10 +5472,11 @@ void ufshcd_release_scsi_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) {struct scsi_cmnd *cmd = lrbp->cmd;scsi_dma_unmap (cmd); +ufshcd_crypto_clear_prdt (hba, lrbp);ufshcd_release (hba);ufshcd_clk_scaling_update_busy (hba); }/**
include/ufs/ufshcd.h +8 -0
@@ -661,10 +661,18 @@ enum ufshcd_quirks {* This quirk needs to be enabled if the host controller supports inline * encryption but does not support the CRYPTO_GENERAL_ENABLE bit, i.e. * host controller initialization fails if that bit is set. */ UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE =1 <<23 , + +/* +* This quirk needs to be enabled if the host controller driver copies +* cryptographic keys into the PRDT in order to send them to hardware, +* and therefore the PRDT should be zeroized after each request (as per +* the standard best practice for managing keys). +*/ + UFSHCD_QUIRK_KEYS_IN_PRDT = 1 << 24, };enum ufshcd_caps {/* Allow dynamic clk gating */ UFSHCD_CAP_CLK_GATING = 1 << 0,
Signature · 1 line
2.45.2
RE: [PATCH v3 5/6] scsi: ufs: core: Add UFSHCD_QUIRK_KEYS_IN_PRDT
Quoted 21 lines
-----Original Message-----
From: Eric Biggers <[email protected]>
Sent: Tuesday, July 9, 2024 5:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; Alim Akhtar <[email protected]>; Avri Altman <[email protected]>; Bart Van Assche <[email protected]>; Martin K . Petersen <[email protected]>; Peter Griffin <[email protected]>; André Draszik <[email protected]>; William McVicker <[email protected]>
Subject: [PATCH v3 5/6] scsi: ufs: core: AddUFSHCD_QUIRK_KEYS_IN_PRDTFrom: Eric Biggers <[email protected]>
Since the nonstandard inline encryption support on Exynos SoCs requires that raw cryptographic keys be copied into the PRDT, it is desirable to zeroize those keys after each request to keep them from being left in memory. Therefore, add a quirk bit that enables the zeroization.
Quoted 7 lines
We could instead do the zeroization unconditionally. However, using a quirk bit avoids adding the zeroization overhead to standard devices.
Reviewed-by Bart Van Assche <[email protected]>Reviewed-by Peter Griffin <[email protected]>Signed-off-by Eric Biggers <[email protected]>---
Eric Biggers
maintainerPATCH v3 6/6 scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
Tested on gs101 (specifically Pixel 6) by running the 'encrypt' group of xfstests on a filesystem mounted with the 'inlinecrypt' mount option.
--- drivers/ufs/host/ufs-exynos.c | 240 +++++++++++++++++++++++++++++++++- 1 file changed, 234 insertions(+), 6 deletions(-)
drivers/ufs/host/ufs-exynos.c +234 -6
@@ -6,10 +6,13 @@* Author: Seungwon Jeon <[email protected]> * Author: Alim Akhtar <[email protected]> * */ +#include <asm/unaligned.h> +#include <crypto/aes.h> +#include <linux/arm-smccc.h> #include <linux/clk.h> #include <linux/delay.h> #include <linux/module.h> #include <linux/of.h> #include <linux/of_address.h>
@@ -23,16 +26,17 @@#include <ufs/ufshci.h> #include <ufs/unipro.h> #include "ufs-exynos.h" +#define DATA_UNIT_SIZE 4096 +/* * Exynos's Vendor specific registers for UFSHCI */ #define HCI_TXPRDT_ENTRY_SIZE 0x00#define PRDT_PREFECT_EN BIT(31) -#define PRDT_SET_SIZE (x) ((x) & 0x1F)#define HCI_RXPRDT_ENTRY_SIZE 0x04#define HCI_1US_TO_CNT_VAL 0x0C#define CNT_VAL_1US_MASK 0x3FF#define HCI_UTRL_NEXUS_TYPE 0x40#define HCI_UTMRL_NEXUS_TYPE 0x44
@@ -1041,12 +1045,12 @@ static int exynos_ufs_post_link(struct ufs_hba *hba)exynos_ufs_establish_connt (ufs);exynos_ufs_fit_aggr_timeout (ufs);hci_writel (ufs,0xa ,HCI_DATA_REORDER ); -hci_writel (ufs,PRDT_SET_SIZE (12 ),HCI_TXPRDT_ENTRY_SIZE ); -hci_writel (ufs,PRDT_SET_SIZE (12 ),HCI_RXPRDT_ENTRY_SIZE ); +hci_writel (ufs,ilog2 (DATA_UNIT_SIZE ),HCI_TXPRDT_ENTRY_SIZE ); +hci_writel (ufs,ilog2 (DATA_UNIT_SIZE ),HCI_RXPRDT_ENTRY_SIZE );hci_writel (ufs, (1 << hba->nutrs) -1 ,HCI_UTRL_NEXUS_TYPE );hci_writel (ufs, (1 << hba->nutmrs) -1 ,HCI_UTMRL_NEXUS_TYPE );hci_writel (ufs,0xf ,HCI_AXIDMA_RWDATA_BURST_LEN );if (ufs->opts &EXYNOS_UFS_OPT_SKIP_CONNECTION_ESTAB )
@@ -1149,10 +1153,231 @@ static inline void exynos_ufs_priv_init(struct ufs_hba *hba, ufs->rx_sel_idx =0 ; hba->priv = (void *)ufs; hba->quirks = ufs->drv_data->quirks; } +#ifdef CONFIG_SCSI_UFS_CRYPTO + +/* +* Support for Flash Memory Protector (FMP), which is the inline encryption +* hardware on Exynos and Exynos-based SoCs. The interface to this hardware is +* not compatible with the standard UFS crypto. It requires that encryption be +* configured in the PRDT using a nonstandard extension. +*/ + +enum fmp_crypto_algo_mode { +FMP_BYPASS_MODE =0 , +FMP_ALGO_MODE_AES_CBC =1 , +FMP_ALGO_MODE_AES_XTS =2 , +}; +enum fmp_crypto_key_length { +FMP_KEYLEN_256BIT =1 , +}; + +/** +* struct fmp_sg_entry - nonstandard format of PRDT entries when FMP is enabled +* +* @base: The standard PRDT entry, but with nonstandard bitfields in the high +* bits of the 'size' field, i.e. the last 32-bit word. When these +* nonstandard bitfields are zero, the data segment won't be encrypted or +* decrypted. Otherwise they specify the algorithm and key length with +* which the data segment will be encrypted or decrypted. +* @file_iv: The initialization vector (IV) with all bytes reversed +* @file_enckey: The first half of the AES-XTS key with all bytes reserved +* @file_twkey: The second half of the AES-XTS key with all bytes reserved +* @disk_iv: Unused +* @reserved: Unused +*/ +struct fmp_sg_entry { +struct ufshcd_sg_entry base; +__be64 file_iv[2 ]; +__be64 file_enckey[4 ]; +__be64 file_twkey[4 ]; +__be64 disk_iv[2 ]; +__be64 reserved[2 ]; +}; + +#define SMC_CMD_FMP_SECURITY \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_SIP, 0x1810) +#define SMC_CMD_SMU \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_SIP, 0x1850) +#define SMC_CMD_FMP_SMU_RESUME \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_SIP, 0x1860) +#define SMU_EMBEDDED 0 +#define SMU_INIT 0 +#define CFG_DESCTYPE_3 3 + +static void exynos_ufs_fmp_init (struct ufs_hba *hba,struct exynos_ufs *ufs) +{ +struct blk_crypto_profile *profile = &hba->crypto_profile; +struct arm_smccc_res res; +int err; + +/* +* Check for the standard crypto support bit, since it's available even +* though the rest of the interface to FMP is nonstandard. +* +* This check should have the effect of preventing the driver from +* trying to use FMP on old Exynos SoCs that don't have FMP. +*/ +if (!(ufshcd_readl (hba,REG_CONTROLLER_CAPABILITIES ) & +MASK_CRYPTO_SUPPORT )) +return ; + +/* +* The below sequence of SMC calls to enable FMP can be found in the +* downstream driver source for gs101 and other Exynos-based SoCs. It +* is the only way to enable FMP that works on SoCs such as gs101 that +* don't make the FMP registers accessible to Linux. It probably works +* on other Exynos-based SoCs too, and might even still be the only way +* that works. But this hasn't been properly tested, and this code is +* mutually exclusive with exynos_ufs_config_smu(). So for now only +* enable FMP support on SoCs with EXYNOS_UFS_OPT_UFSPR_SECURE. +*/ +if (!(ufs->opts &EXYNOS_UFS_OPT_UFSPR_SECURE )) +return ; + +/* +* This call (which sets DESCTYPE to 0x3 in the FMPSECURITY0 register) +* is needed to make the hardware use the larger PRDT entry size. +*/ +BUILD_BUG_ON (sizeof (struct fmp_sg_entry ) !=128 ); +arm_smccc_smc (SMC_CMD_FMP_SECURITY ,0 ,SMU_EMBEDDED ,CFG_DESCTYPE_3 , +0 ,0 ,0 ,0 , &res); +if (res.a0) { +dev_warn (hba->dev, +"SMC_CMD_FMP_SECURITY failed on init: %ld. Disabling FMP support.\n" , + res.a0); +return ; + } +ufshcd_set_sg_entry_size (hba,sizeof (struct fmp_sg_entry )); + +/* +* This is needed to initialize FMP. Without it, errors occur when +* inline encryption is used. +*/ +arm_smccc_smc (SMC_CMD_SMU ,SMU_INIT ,SMU_EMBEDDED ,0 ,0 ,0 ,0 ,0 , &res); +if (res.a0) { +dev_err (hba->dev, +"SMC_CMD_SMU(SMU_INIT) failed: %ld. Disabling FMP support.\n" , + res.a0); +return ; + } + +/* Advertise crypto capabilities to the block layer. */ + err =devm_blk_crypto_profile_init (hba->dev, profile,0 ); +if (err) { +/* Only ENOMEM should be possible here. */ +dev_err (hba->dev,"Failed to initialize crypto profile: %d\n" , + err); +return ; + } + profile->max_dun_bytes_supported =AES_BLOCK_SIZE ; + profile->dev = hba->dev; + profile->modes_supported[BLK_ENCRYPTION_MODE_AES_256_XTS ] = +DATA_UNIT_SIZE ; + +/* Advertise crypto support to ufshcd-core. */ + hba->caps |=UFSHCD_CAP_CRYPTO ; + +/* Advertise crypto quirks to ufshcd-core. */ + hba->quirks |=UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE | +UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE | +UFSHCD_QUIRK_KEYS_IN_PRDT ; + +} + +static void exynos_ufs_fmp_resume (struct ufs_hba *hba) +{ +struct arm_smccc_res res; + +arm_smccc_smc (SMC_CMD_FMP_SECURITY ,0 ,SMU_EMBEDDED ,CFG_DESCTYPE_3 , +0 ,0 ,0 ,0 , &res); +if (res.a0) +dev_err (hba->dev, +"SMC_CMD_FMP_SECURITY failed on resume: %ld\n" , res.a0); + +arm_smccc_smc (SMC_CMD_FMP_SMU_RESUME ,0 ,SMU_EMBEDDED ,0 ,0 ,0 ,0 ,0 , + &res); +if (res.a0) +dev_err (hba->dev, +"SMC_CMD_FMP_SMU_RESUME failed: %ld\n" , res.a0); +} + +static inline __be64 fmp_key_word (const u8 *key,int j) +{ +return cpu_to_be64 (get_unaligned_le64 ( + key +AES_KEYSIZE_256 - (j +1 ) *sizeof (u64))); +} + +/* Fill the PRDT for a request according to the given encryption context. */ +static int exynos_ufs_fmp_fill_prdt (struct ufs_hba *hba, +const struct bio_crypt_ctx *crypt_ctx, +void *prdt,unsigned int num_segments) +{ +struct fmp_sg_entry *fmp_prdt = prdt; +const u8 *enckey = crypt_ctx->bc_key->raw; +const u8 *twkey = enckey +AES_KEYSIZE_256 ; +u64 dun_lo = crypt_ctx->bc_dun[0 ]; +u64 dun_hi = crypt_ctx->bc_dun[1 ]; +unsigned int i; + +/* If FMP wasn't enabled, we shouldn't get any encrypted requests. */ +if (WARN_ON_ONCE (!(hba->caps &UFSHCD_CAP_CRYPTO ))) +return -EIO ; + +/* Configure FMP on each segment of the request. */ +for (i =0 ; i < num_segments; i++) { +struct fmp_sg_entry *prd = &fmp_prdt[i]; +int j; + +/* Each segment must be exactly one data unit. */ +if (prd->base.size !=cpu_to_le32 (DATA_UNIT_SIZE -1 )) { +dev_err (hba->dev, +"data segment is misaligned for FMP\n" ); +return -EIO ; + } + +/* Set the algorithm and key length. */ + prd->base.size |=cpu_to_le32 ((FMP_ALGO_MODE_AES_XTS <<28 ) | + (FMP_KEYLEN_256BIT <<26 )); + +/* Set the IV. */ + prd->file_iv[0 ] =cpu_to_be64 (dun_hi); + prd->file_iv[1 ] =cpu_to_be64 (dun_lo); + +/* Set the key. */ +for (j =0 ; j <AES_KEYSIZE_256 /sizeof (u64); j++) { + prd->file_enckey[j] =fmp_key_word (enckey, j); + prd->file_twkey[j] =fmp_key_word (twkey, j); + } + +/* Increment the data unit number. */ + dun_lo++; +if (dun_lo ==0 ) + dun_hi++; + } +return 0 ; +} + +#else /* CONFIG_SCSI_UFS_CRYPTO */ + +static void exynos_ufs_fmp_init (struct ufs_hba *hba,struct exynos_ufs *ufs) +{ +} + +static void exynos_ufs_fmp_resume (struct ufs_hba *hba) +{ +} + +#define exynos_ufs_fmp_fill_prdt NULL + +#endif /* !CONFIG_SCSI_UFS_CRYPTO */ +static int exynos_ufs_init (struct ufs_hba *hba) {struct device *dev = hba->dev;struct platform_device *pdev =to_platform_device (dev);struct exynos_ufs *ufs;
@@ -1196,10 +1421,12 @@ static int exynos_ufs_init(struct ufs_hba *hba) goto out; }exynos_ufs_priv_init (hba, ufs); +exynos_ufs_fmp_init (hba, ufs); +if (ufs->drv_data->drv_init) { ret = ufs->drv_data->drv_init (dev, ufs);if (ret) {dev_err (dev,"failed to init drv-data\n" ); goto out;
@@ -1211,11 +1438,11 @@ static int exynos_ufs_init(struct ufs_hba *hba) goto out;exynos_ufs_specify_phy_time_attr (ufs);if (!(ufs->opts &EXYNOS_UFS_OPT_UFSPR_SECURE ))exynos_ufs_config_smu (ufs); - hba->host->dma_alignment =SZ_4K -1 ; + hba->host->dma_alignment =DATA_UNIT_SIZE -1 ;return 0 ; out: hba->priv =NULL ;return ret;
@@ -1330,11 +1557,11 @@ static int exynos_ufs_hce_enable_notify(struct ufs_hba *hba,* The maximum segment size must be set after scsi_host_alloc() * has been called and before LUN scanning starts * (ufshcd_async_scan()). Note: this callback may also be called * from other functions than ufshcd_init(). */ - hba->host->max_segment_size =SZ_4K ; + hba->host->max_segment_size =DATA_UNIT_SIZE ;if (ufs->drv_data->pre_hce_enable) { ret = ufs->drv_data->pre_hce_enable (ufs);if (ret)return ret;
@@ -1430,11 +1657,11 @@ static int exynos_ufs_resume(struct ufs_hba *hba, enum ufs_pm_op pm_op)if (!ufshcd_is_link_active (hba))phy_power_on (ufs->phy);exynos_ufs_config_smu (ufs); - +exynos_ufs_fmp_resume (hba);return 0 ; }static int exynosauto_ufs_vh_link_startup_notify (struct ufs_hba *hba,enum ufs_notify_change_status status)
@@ -1696,10 +1923,11 @@ static const struct ufs_hba_variant_ops ufs_hba_exynos_ops = { .setup_xfer_req = exynos_ufs_specify_nexus_t_xfer_req, .setup_task_mgmt = exynos_ufs_specify_nexus_t_tm_req, .hibern8_notify = exynos_ufs_hibern8_notify, .suspend = exynos_ufs_suspend, .resume = exynos_ufs_resume, + .fill_crypto_prdt = exynos_ufs_fmp_fill_prdt, };static struct ufs_hba_variant_ops ufs_hba_exynosauto_vh_ops = { .name ="exynosauto_ufs_vh" , .init = exynosauto_ufs_vh_init,
Signature · 1 line
2.45.2
Re: [PATCH v3 6/6] scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)
Hi Eric,
Quoted 12 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
Quoted 5 lines
Tested on gs101 (specifically Pixel 6) by running the 'encrypt' group of xfstests on a filesystem mounted with the 'inlinecrypt' mount option.
Signed-off-by Eric Biggers <[email protected]>---
and
Tested by running the encrypt group of xfstests on my Pixel 6, using the Yocto development env described here
https://git.codelinaro.org/linaro/googlelt/pixelscripts
Notes on testing, in addition to above README.
1. Enabled following additional kernel configs gs101_config.fragment CONFIG_FS_ENCRYPTION=y CONFIG_FS_ENCRYPTION_INLINE_CRYPT=y CONFIG_SCSI_UFS_CRYPTO=yCONFIG_BLK_INLINE_ENCRYPTION=y CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK=y CONFIG_CRYPTO_HCTR2=y
2. Add meta-security layer to bblayers.conf and relevant packages to local.conf BBLAYERS += "/yocto-builds/yocto/meta-security" IMAGE_INSTALL:append = " xfstests ecryptfs-utils fscryptctl keyutils cryptmount "
3. Rebuild/reflash Yocto rootfs
bitbake virtual/kernel core-image-full-cmdline
fastboot flash userdata core-image-full-cmdline-google-gs.rootfs.ext4
4. On the device ran the following
mkfs.ext4 -O encrypt /dev/sda26
mkfs.ext4 -O encrypt /dev/sda20
mkdir -p /mnt/scratchdev
mkdir -p /mnt/testdev
mount /dev/sda20 -o inlinecrypt /mnt/testdev
mount /dev/sda26 -o inlinecrypt /mnt/scratchdev
export TEST_DEV=/dev/sda20
export TEST_DIR=/mnt/testdev
export SCRATCH_DEV=/dev/sda26
export SCRATCH_MNT=/mnt/scratchdev
cd /usr/xfstests
check -g encrypt
All 28 tests passed
<snip>
Ran: ext4/024 generic/395 generic/396 generic/397 generic/398 generic/399 generic/419 generic/421 generic/429 generic/435 generic/440 generic/548 generic/549 generic/550 generic/576 generic/580 gener9
Not run: generic/399 generic/550 generic/576 generic/584 generic/613 Passed all 28 tests
kind regards,
Peter
[..]
Eric Biggers
maintainerRe: [PATCH v3 6/6] scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)
Quoted 76 lines · Peter Griffin
Hi Eric,
Quoted 12 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
Quoted 5 lines
Tested on gs101 (specifically Pixel 6) by running the 'encrypt' group of xfstests on a filesystem mounted with the 'inlinecrypt' mount option.
Signed-off-by Eric Biggers <[email protected]>---
Reviewed-by Peter Griffin <[email protected]>and
Tested-by Peter Griffin <[email protected]>Tested by running the encrypt group of xfstests on my Pixel 6, using the Yocto development env described here
https://git.codelinaro.org/linaro/googlelt/pixelscriptsNotes on testing, in addition to above README.
1. Enabled following additional kernel configs gs101_config.fragment
CONFIG_FS_ENCRYPTION=yCONFIG_FS_ENCRYPTION_INLINE_CRYPT=yCONFIG_SCSI_UFS_CRYPTO=yCONFIG_BLK_INLINE_ENCRYPTION=yCONFIG_BLK_INLINE_ENCRYPTION_FALLBACK=yCONFIG_CRYPTO_HCTR2=y2. Add meta-security layer to bblayers.conf and relevant packages to local.conf BBLAYERS += "/yocto-builds/yocto/meta-security" IMAGE_INSTALL:append = " xfstests ecryptfs-utils fscryptctl keyutils cryptmount "
3. Rebuild/reflash Yocto rootfs
bitbake virtual/kernel core-image-full-cmdline
fastboot flash userdata core-image-full-cmdline-google-gs.rootfs.ext44. On the device ran the following
mkfs.ext4 -O encrypt
/dev/sda26
mkfs.ext4 -O encrypt/dev/sda20
mkdir -p/mnt/scratchdev
mkdir -p/mnt/testdev
mount/dev/sda20-o inlinecrypt/mnt/testdev
mount/dev/sda26-o inlinecrypt/mnt/scratchdev
export TEST_DEV=/dev/sda20
export TEST_DIR=/mnt/testdev
export SCRATCH_DEV=/dev/sda26
export SCRATCH_MNT=/mnt/scratchdev
cd/usr/xfstests
check -g encryptAll 28 tests passed
<snip>
Ran: ext4/024 generic/395 generic/396 generic/397 generic/398 generic/399 generic/419 generic/421 generic/429 generic/435 generic/440 generic/548 generic/549 generic/550 generic/576 generic/580 gener9
Not run: generic/399 generic/550 generic/576 generic/584 generic/613 Passed all 28 tests
Quoted 2 lines
kind regards,
Thanks! This is similar to what I did. But, to get the inlinecrypt mount option to be used during the tests it's necessary to do the following:
export EXT_MOUNT_OPTIONS="-o inlinecrypt"
The following message will appear in the kernel log:
fscrypt: AES-256-XTS using blk-crypto (native)
- Eric
Peter Griffin
maintainerRe: [PATCH v3 6/6] scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)
Hi Eric,
Quoted 78 lines · Eric Biggers
Quoted 76 lines · Peter Griffin
Hi Eric,
Quoted 12 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
Quoted 5 lines
Tested on gs101 (specifically Pixel 6) by running the 'encrypt' group of xfstests on a filesystem mounted with the 'inlinecrypt' mount option.
Signed-off-by Eric Biggers <[email protected]>---
Reviewed-by Peter Griffin <[email protected]>and
Tested-by Peter Griffin <[email protected]>Tested by running the encrypt group of xfstests on my Pixel 6, using the Yocto development env described here
https://git.codelinaro.org/linaro/googlelt/pixelscriptsNotes on testing, in addition to above README.
1. Enabled following additional kernel configs gs101_config.fragment
CONFIG_FS_ENCRYPTION=yCONFIG_FS_ENCRYPTION_INLINE_CRYPT=yCONFIG_SCSI_UFS_CRYPTO=yCONFIG_BLK_INLINE_ENCRYPTION=yCONFIG_BLK_INLINE_ENCRYPTION_FALLBACK=yCONFIG_CRYPTO_HCTR2=y2. Add meta-security layer to bblayers.conf and relevant packages to local.conf BBLAYERS += "/yocto-builds/yocto/meta-security" IMAGE_INSTALL:append = " xfstests ecryptfs-utils fscryptctl keyutils cryptmount "
3. Rebuild/reflash Yocto rootfs
bitbake virtual/kernel core-image-full-cmdline
fastboot flash userdata core-image-full-cmdline-google-gs.rootfs.ext44. On the device ran the following
mkfs.ext4 -O encrypt
/dev/sda26
mkfs.ext4 -O encrypt/dev/sda20
mkdir -p/mnt/scratchdev
mkdir -p/mnt/testdev
mount/dev/sda20-o inlinecrypt/mnt/testdev
mount/dev/sda26-o inlinecrypt/mnt/scratchdev
export TEST_DEV=/dev/sda20
export TEST_DIR=/mnt/testdev
export SCRATCH_DEV=/dev/sda26
export SCRATCH_MNT=/mnt/scratchdev
cd/usr/xfstests
check -g encryptAll 28 tests passed
<snip>
Ran: ext4/024 generic/395 generic/396 generic/397 generic/398 generic/399 generic/419 generic/421 generic/429 generic/435 generic/440 generic/548 generic/549 generic/550 generic/576 generic/580 gener9
Not run: generic/399 generic/550 generic/576 generic/584 generic/613 Passed all 28 tests
Quoted 8 lines
Quoted 2 lines
kind regards,
Thanks! This is similar to what I did. But, to get the inlinecrypt mount option to be used during the tests it's necessary to do the following:
export EXT_MOUNT_OPTIONS="-o inlinecrypt"
OK great, thanks Eric! I will update my notes to include that. That was actually one reason to include all the test instructions in the email to check I was running this correctly :)
Quoted 3 lines
The following message will appear in the kernel log:
fscrypt: AES-256-XTS using blk-crypto (native)
I just ran the tests again setting EXT_MOUNT_OPTIONS and I see
root@google-gs:/usr/xfstests# dmesg | grep "fscrypt: AES-256-XTS" [ 1319.539742] fscrypt: AES-256-XTS using blk-crypto (native)
I also added in fsverity-utils and xz which are required by a couple of the skipped tests.
Thanks,
Peter
RE: [PATCH v3 6/6] scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)
In reply to Eric BiggersQuoted 73 lines
-----Original Message-----
From: Eric Biggers <[email protected]>
Sent: Tuesday, July 9, 2024 5:24 AM
To: [email protected]
Cc: [email protected]; [email protected]; Alim Akhtar <[email protected]>; Avri Altman <[email protected]>; Bart Van Assche <[email protected]>; Martin K . Petersen <[email protected]>; Peter Griffin <[email protected]>; André Draszik <[email protected]>; William McVicker <[email protected]>
Subject: [PATCH v3 6/6] scsi: ufs: exynos: Add support for Flash Memory Protector (FMP)From: Eric Biggers <[email protected]>
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
Tested on gs101 (specifically Pixel 6) by running the 'encrypt' group of xfstests on a filesystem mounted with the 'inlinecrypt' mount option.
Signed-off-by Eric Biggers <[email protected]>--- drivers/ufs/host/ufs-exynos.c | 240 +++++++++++++++++++++++++++++++++- 1 file changed, 234 insertions(+), 6 deletions(-)drivers/ufs/host/ufs-exynos.c +5
-1@@ -6,10 +6,13 @@* Author: Seungwon Jeon <[email protected]> * Author: Alim Akhtar <[email protected]> * */ +#include <asm/unaligned.h> +#include <crypto/aes.h> +#include <linux/arm-smccc.h> #include <linux/clk.h> #include <linux/delay.h> #include <linux/module.h> #include <linux/of.h> #include <linux/of_address.h> @@ -23,16 +26,17 @@#include <ufs/ufshci.h> #include <ufs/unipro.h> #include "ufs-exynos.h" +#define DATA_UNIT_SIZE 4096 +/* * Exynos's Vendor specific registers for UFSHCI */ #define HCI_TXPRDT_ENTRY_SIZE 0x00#define PRDT_PREFECT_EN BIT(31) -#define PRDT_SET_SIZE (x) ((x) & 0x1F)#define HCI_RXPRDT_ENTRY_SIZE 0x04#define HCI_1US_TO_CNT_VAL 0x0C#define CNT_VAL_1US_MASK 0x3FF#define HCI_UTRL_NEXUS_TYPE 0x40#define HCI_UTMRL_NEXUS_TYPE 0x44
drivers/ufs/host/ufs-exynos.c static int exynos_ufs_post_link(struct ufs_hba · line 1052
Quoted 10 lines
drivers/ufs/host/ufs-exynos.c +2
-2@@ -1041,12 +1045,12 @@ static int exynos_ufs_post_link(struct ufs_hba *hba)exynos_ufs_establish_connt (ufs);exynos_ufs_fit_aggr_timeout (ufs);hci_writel (ufs,0xa ,HCI_DATA_REORDER ); -hci_writel (ufs,PRDT_SET_SIZE (12 ),HCI_TXPRDT_ENTRY_SIZE ); -hci_writel (ufs,PRDT_SET_SIZE (12 ),HCI_RXPRDT_ENTRY_SIZE ); +hci_writel (ufs,ilog2 (DATA_UNIT_SIZE ),HCI_TXPRDT_ENTRY_SIZE ); +hci_writel (ufs,ilog2 (DATA_UNIT_SIZE ),HCI_RXPRDT_ENTRY_SIZE );
These changes can be added (as separate patch) irrespective of FMP support. Will leave upto you though.
LGTM,
Quoted 273 lines
drivers/ufs/host/ufs-exynos.c +0
-0hci_writel (ufs, (1 << hba->nutrs) -1 ,HCI_UTRL_NEXUS_TYPE );hci_writel (ufs, (1 << hba->nutmrs) -1 ,HCI_UTMRL_NEXUS_TYPE );hci_writel (ufs,0xf ,HCI_AXIDMA_RWDATA_BURST_LEN );drivers/ufs/host/ufs-exynos.c +26
-0if (ufs->opts &EXYNOS_UFS_OPT_SKIP_CONNECTION_ESTAB )@@ -1149,10 +1153,231 @@ static inline void exynos_ufs_priv_init(struct ufs_hba *hba, ufs->rx_sel_idx =0 ; hba->priv = (void *)ufs; hba->quirks = ufs->drv_data->quirks; } +#ifdef CONFIG_SCSI_UFS_CRYPTO + +/* +* Support for Flash Memory Protector (FMP), which is the inline +encryption +* hardware on Exynos and Exynos-based SoCs. The interface to this +hardware is +* not compatible with the standard UFS crypto. It requires that +encryption be +* configured in the PRDT using a nonstandard extension. +*/ + +enum fmp_crypto_algo_mode { +FMP_BYPASS_MODE =0 , +FMP_ALGO_MODE_AES_CBC =1 , +FMP_ALGO_MODE_AES_XTS =2 , +}; +enum fmp_crypto_key_length { +FMP_KEYLEN_256BIT =1 , +}; + +/** +* struct fmp_sg_entry - nonstandard format of PRDT entries when FMP is +enabled +* +* @base: The standard PRDT entry, but with nonstandard bitfields in the high
+ * bits of the 'size' field, i.e. the last 32-bit word. When these
+ * nonstandard bitfields are zero, the data segment won't be encrypted
or
+ * decrypted. Otherwise they specify the algorithm and key length with
+ * which the data segment will be encrypted or decrypted.
+ * @file_iv: The initialization vector (IV) with all bytes reversed
+ * @file_enckey: The first half of the AES-XTS key with all bytesdrivers/ufs/host/ufs-exynos.c +17
-0+reserved +* @file_twkey: The second half of the AES-XTS key with all bytes +reserved +* @disk_iv: Unused +* @reserved: Unused +*/ +struct fmp_sg_entry { +struct ufshcd_sg_entry base; +__be64 file_iv[2 ]; +__be64 file_enckey[4 ]; +__be64 file_twkey[4 ]; +__be64 disk_iv[2 ]; +__be64 reserved[2 ]; +}; + +#define SMC_CMD_FMP_SECURITY \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,
ARM_SMCCC_SMC_64, \drivers/ufs/host/ufs-exynos.c +3
-0+ ARM_SMCCC_OWNER_SIP, 0x1810) +#define SMC_CMD_SMU \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,
ARM_SMCCC_SMC_64, \drivers/ufs/host/ufs-exynos.c +3
-0+ ARM_SMCCC_OWNER_SIP, 0x1850) +#define SMC_CMD_FMP_SMU_RESUME \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,
ARM_SMCCC_SMC_64, \drivers/ufs/host/ufs-exynos.c +24
-0+ ARM_SMCCC_OWNER_SIP, 0x1860) +#define SMU_EMBEDDED 0 +#define SMU_INIT 0 +#define CFG_DESCTYPE_3 3 + +static void exynos_ufs_fmp_init (struct ufs_hba *hba,struct exynos_ufs +*ufs) { +struct blk_crypto_profile *profile = &hba->crypto_profile; +struct arm_smccc_res res; +int err; + +/* +* Check for the standard crypto support bit, since it's available even +* though the rest of the interface to FMP is nonstandard. +* +* This check should have the effect of preventing the driver from +* trying to use FMP on old Exynos SoCs that don't have FMP. +*/ +if (!(ufshcd_readl (hba,REG_CONTROLLER_CAPABILITIES ) & +MASK_CRYPTO_SUPPORT )) +return ; + +/* +* The below sequence of SMC calls to enable FMP can be found in the
drivers/ufs/host/ufs-exynos.c +1
-0+* downstream driver source for gs101 and other Exynos-based SoCs. It
drivers/ufs/host/ufs-exynos.c +1
-0+* is the only way to enable FMP that works on SoCs such as gs101 that
drivers/ufs/host/ufs-exynos.c +1
-0+* don't make the FMP registers accessible to Linux. It probably works
drivers/ufs/host/ufs-exynos.c +1
-0+* on other Exynos-based SoCs too, and might even still be the only way
drivers/ufs/host/ufs-exynos.c +2
-0+* that works. But this hasn't been properly tested, and this code is +* mutually exclusive with exynos_ufs_config_smu(). So for now only
drivers/ufs/host/ufs-exynos.c +1
-0+* enable FMP support on SoCs with
EXYNOS_UFS_OPT_UFSPR_SECURE.drivers/ufs/host/ufs-exynos.c +6
-0+*/ +if (!(ufs->opts &EXYNOS_UFS_OPT_UFSPR_SECURE )) +return ; + +/* +* This call (which sets DESCTYPE to 0x3 in the FMPSECURITY0 register)
drivers/ufs/host/ufs-exynos.c +4
-0+* is needed to make the hardware use the larger PRDT entry size. +*/ +BUILD_BUG_ON (sizeof (struct fmp_sg_entry ) !=128 ); + arm_smccc_smc(SMC_CMD_FMP_SECURITY, 0, SMU_EMBEDDED,
CFG_DESCTYPE_3,drivers/ufs/host/ufs-exynos.c +4
-0+ 0, 0, 0, 0, &res); +if (res.a0) { + dev_warn(hba->dev, + "SMC_CMD_FMP_SECURITY failed on init: %ld.Disabling FMP support.\n",
drivers/ufs/host/ufs-exynos.c +10
-0+ res.a0); +return ; + } +ufshcd_set_sg_entry_size (hba,sizeof (struct fmp_sg_entry )); + +/* +* This is needed to initialize FMP. Without it, errors occur when +* inline encryption is used. +*/ + arm_smccc_smc(SMC_CMD_SMU, SMU_INIT, SMU_EMBEDDED, 0,0, 0, 0, 0, &res);
drivers/ufs/host/ufs-exynos.c +3
-0+if (res.a0) { + dev_err(hba->dev, + "SMC_CMD_SMU(SMU_INIT) failed: %ld. DisablingFMP support.\n",
drivers/ufs/host/ufs-exynos.c +15
-0+ res.a0); +return ; + } + +/* Advertise crypto capabilities to the block layer. */ + err =devm_blk_crypto_profile_init (hba->dev, profile,0 ); +if (err) { +/* Only ENOMEM should be possible here. */ +dev_err (hba->dev,"Failed to initialize crypto profile: %d\n" , + err); +return ; + } + profile->max_dun_bytes_supported =AES_BLOCK_SIZE ; + profile->dev = hba->dev; + profile-Quoted 1 line
modes_supported[BLK_ENCRYPTION_MODE_AES_256_XTS] =
drivers/ufs/host/ufs-exynos.c +16
-0+DATA_UNIT_SIZE ; + +/* Advertise crypto support to ufshcd-core. */ + hba->caps |=UFSHCD_CAP_CRYPTO ; + +/* Advertise crypto quirks to ufshcd-core. */ + hba->quirks |=UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE | +UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE | +UFSHCD_QUIRK_KEYS_IN_PRDT ; + +} + +static void exynos_ufs_fmp_resume (struct ufs_hba *hba) { +struct arm_smccc_res res; + + arm_smccc_smc(SMC_CMD_FMP_SECURITY, 0, SMU_EMBEDDED,
CFG_DESCTYPE_3,drivers/ufs/host/ufs-exynos.c +4
-0+ 0, 0, 0, 0, &res); +if (res.a0) + dev_err(hba->dev, + "SMC_CMD_FMP_SECURITY failed on resume:%ld\n", res.a0);
drivers/ufs/host/ufs-exynos.c +2
-0+ + arm_smccc_smc(SMC_CMD_FMP_SMU_RESUME, 0,
SMU_EMBEDDED, 0, 0, 0, 0, 0,drivers/ufs/host/ufs-exynos.c +4
-0+ &res); +if (res.a0) + dev_err(hba->dev, + "SMC_CMD_FMP_SMU_RESUME failed: %ld\n",res.a0); }
drivers/ufs/host/ufs-exynos.c +17
-0+ +static inline __be64 fmp_key_word (const u8 *key,int j) { +return cpu_to_be64 (get_unaligned_le64 ( + key +AES_KEYSIZE_256 - (j +1 ) *sizeof (u64))); } + +/* Fill the PRDT for a request according to the given encryption +context. */ static int exynos_ufs_fmp_fill_prdt (struct ufs_hba *hba, +const struct bio_crypt_ctx *crypt_ctx, +void *prdt,unsigned int num_segments) { +struct fmp_sg_entry *fmp_prdt = prdt; +const u8 *enckey = crypt_ctx->bc_key->raw; +const u8 *twkey = enckey +AES_KEYSIZE_256 ; +u64 dun_lo = crypt_ctx->bc_dun[0 ]; +u64 dun_hi = crypt_ctx->bc_dun[1 ]; +unsigned int i; + +/* If FMP wasn't enabled, we shouldn't get any encrypted requests. */
drivers/ufs/host/ufs-exynos.c +17
-0+if (WARN_ON_ONCE (!(hba->caps &UFSHCD_CAP_CRYPTO ))) +return -EIO ; + +/* Configure FMP on each segment of the request. */ +for (i =0 ; i < num_segments; i++) { +struct fmp_sg_entry *prd = &fmp_prdt[i]; +int j; + +/* Each segment must be exactly one data unit. */ +if (prd->base.size !=cpu_to_le32 (DATA_UNIT_SIZE -1 )) { +dev_err (hba->dev, +"data segment is misaligned for FMP\n" ); +return -EIO ; + } + +/* Set the algorithm and key length. */ + prd->base.size |=cpu_to_le32((FMP_ALGO_MODE_AES_XTS << 28) |
drivers/ufs/host/ufs-exynos.c +33
-0+ (FMP_KEYLEN_256BIT << 26)); + +/* Set the IV. */ + prd->file_iv[0 ] =cpu_to_be64 (dun_hi); + prd->file_iv[1 ] =cpu_to_be64 (dun_lo); + +/* Set the key. */ +for (j =0 ; j <AES_KEYSIZE_256 /sizeof (u64); j++) { + prd->file_enckey[j] =fmp_key_word (enckey, j); + prd->file_twkey[j] =fmp_key_word (twkey, j); + } + +/* Increment the data unit number. */ + dun_lo++; +if (dun_lo ==0 ) + dun_hi++; + } +return 0 ; +} + +#else /* CONFIG_SCSI_UFS_CRYPTO */ + +static void exynos_ufs_fmp_init (struct ufs_hba *hba,struct exynos_ufs +*ufs) { } + +static void exynos_ufs_fmp_resume (struct ufs_hba *hba) { } + +#define exynos_ufs_fmp_fill_prdt NULL + +#endif /* !CONFIG_SCSI_UFS_CRYPTO */ +static int exynos_ufs_init (struct ufs_hba *hba) {struct device *dev = hba->dev;struct platform_device *pdev =to_platform_device (dev);struct exynos_ufs *ufs;@@ -1196,10 +1421,12 @@ static int exynos_ufs_init(struct ufs_hba *hba) goto out; }exynos_ufs_priv_init (hba, ufs); +exynos_ufs_fmp_init (hba, ufs); +if (ufs->drv_data->drv_init) { ret = ufs->drv_data->drv_init (dev, ufs);if (ret) {dev_err (dev,"failed to init drv-data\n" ); goto out;
drivers/ufs/host/ufs-exynos.c static int exynos_ufs_init(struct ufs_hba *hba) · line 1443
Quoted 7 lines
drivers/ufs/host/ufs-exynos.c +1
-1@@ -1211,11 +1438,11 @@ static int exynos_ufs_init(struct ufs_hba *hba) goto out;exynos_ufs_specify_phy_time_attr (ufs);if (!(ufs->opts &EXYNOS_UFS_OPT_UFSPR_SECURE ))exynos_ufs_config_smu (ufs); - hba->host->dma_alignment =SZ_4K -1 ; + hba->host->dma_alignment =DATA_UNIT_SIZE -1 ;
Same comment as above.
Quoted 12 lines
drivers/ufs/host/ufs-exynos.c +0
-0return 0 ; out: hba->priv =NULL ;return ret;@@ -1330,11 +1557,11 @@ static int exynos_ufs_hce_enable_notify(structufs_hba *hba, * The maximum segment size must be set after scsi_host_alloc() * has been called and before LUN scanning starts * (ufshcd_async_scan()). Note: this callback may also becalled
drivers/ufs/host/ufs-exynos.c static int exynos_ufs_hce_enable_notify(struct ufs_hba *hba, · line 1562
Quoted 4 lines
drivers/ufs/host/ufs-exynos.c +1
-1* from other functions than ufshcd_init(). */ - hba->host->max_segment_size =SZ_4K ; + hba->host->max_segment_size =DATA_UNIT_SIZE ;
And here.
Quoted 37 lines
drivers/ufs/host/ufs-exynos.c +1
-1if (ufs->drv_data->pre_hce_enable) { ret = ufs->drv_data->pre_hce_enable (ufs);if (ret)return ret;@@ -1430,11 +1657,11 @@ static int exynos_ufs_resume(struct ufs_hba *hba, enum ufs_pm_op pm_op)if (!ufshcd_is_link_active (hba))phy_power_on (ufs->phy);exynos_ufs_config_smu (ufs); - +exynos_ufs_fmp_resume (hba);return 0 ; } static int exynosauto_ufs_vh_link_startup_notify(struct ufs_hba *hba,enum
ufs_notify_change_statusstatus) @@ -1696,10 +1923,11 @@ static const structufs_hba_variant_opsufs_hba_exynos_ops= {
.setup_xfer_req =exynos_ufs_specify_nexus_t_xfer_req,
.setup_task_mgmt =exynos_ufs_specify_nexus_t_tm_req,
.hibern8_notify =exynos_ufs_hibern8_notify,drivers/ufs/host/ufs-exynos.c +1
-1.suspend = exynos_ufs_suspend, .resume = exynos_ufs_resume, + .fill_crypto_prdt = exynos_ufs_fmp_fill_prdt, };static struct ufs_hba_variant_ops ufs_hba_exynosauto_vh_ops = { .name ="exynosauto_ufs_vh" , .init = exynosauto_ufs_vh_init, --2.45.2
Martin K. Petersen
AppliedEric,
Quoted 2 lines
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Applied to 6.11/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin K. Petersen
AppliedQuoted 10 lines · Eric Biggers
Add support for Flash Memory Protector (FMP), which is the inline encryption hardware on Exynos and Exynos-based SoCs.
Specifically, add support for the "traditional FMP mode" that works on many Exynos-based SoCs including gs101. This is the mode that uses "software keys" and is compatible with the upstream kernel's existing inline encryption framework in the block and filesystem layers. I plan to add support for the wrapped key support on gs101 at a later time.
[...]
Applied to 6.11/scsi-queue, thanks!
[1/6] scsi: ufs: core: Add UFSHCD_QUIRK_CUSTOM_CRYPTO_PROFILE
https://git.kernel.org/mkp/scsi/c/c2a90eee29f4 [2/6] scsi: ufs: core: fold ufshcd_clear_keyslot() into its caller https://git.kernel.org/mkp/scsi/c/ec99818afb03 [3/6] scsi: ufs: core: Add UFSHCD_QUIRK_BROKEN_CRYPTO_ENABLE https://git.kernel.org/mkp/scsi/c/e95881e0081a [4/6] scsi: ufs: core: Add fill_crypto_prdt variant op https://git.kernel.org/mkp/scsi/c/8ecea3da1567 [5/6] scsi: ufs: core: Add UFSHCD_QUIRK_KEYS_IN_PRDT https://git.kernel.org/mkp/scsi/c/4c45dba50a37 [6/6] scsi: ufs: exynos: Add support for Flash Memory Protector (FMP) https://git.kernel.org/mkp/scsi/c/c96499fcb403
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering