dma-mapping: Fixes for memory encryption
Subsystems (3)
v3 cover dma-mapping: Fixes for memory encryption
Introduction
============
This is the third version of the fixes for direct-dma dealing with memory encryption and restricted-dma.
Changes in v3:
- Instead of extending the logic by using is_swiotlb_for_alloc(), follow Jason’s suggestion and propagate the state of the memory allocated.
- Remove checks out of dma_set_*() based on Jason suggestion
- Remove documentation for now until we are close to the final proposal and add it later if needed.
Background
==========
At the moment the following hypervisor guests will need to deal with memory encryption:
- pKVM (ARM): Documentation/virt/kvm/arm/hypercalls.rst
- ARM CCA: Documentation/arch/arm64/arm-cca.rst
- Intel TDX: Documentation/arch/x86/tdx.rst
- AMD SEV: Documentation/arch/x86/amd-memory-encryption.rst
- PPC SVM: Documentation/arch/powerpc/ultravisor.rst
- Hyper-V: Documentation/virt/hyperv/coco.rst
AFAICT, all (confidential) guests running under those have the memory encrypted by default and guests will then explicitly share the memory back if needed.
The main use cases for decrypting(sharing) memory are:
- Sharing memory back to the host through SWIOTLB (for virtio...)
- Hypervisor specific communication (ex: snp_msg, GHCB, VMBUS...)
- Shared/emulated resources: VGARAM (x86-SEV), GIC ITS tables (arm64)
While encrypting memory is typically used for reverting the set_memory_decrypted() either in error handling or in freeing shared resources back to the kernel.
Design
======
This series focuses mainly on dma-direct interaction with memory encryption which is the complicated case.
At the moment memory encryption and dma-direct interacts in 2 ways:
1) force_dma_direct(): if true, memory will be decrypted by default on allocation.
2) Restricted DMA: where memory is pre-decrypted and managed by SWIOTLB.
With a third possible usage on the way [1] where the DMA-API allows an attr for decrypted memory.
Instead of open coding many checks with is_swiotlb_for_alloc() and force_dma_unencrypted().
Make __dma_direct_alloc_pages() return the state of allocated memory encapsulated on the new internal type dma_page.
Then based on the memory state, dma-direct can identify what to do based on the cases:
- Memory needs to be decrypted but is not: dma-direct will decrypt the memory and use the proper phys address conversions and page table prot.
- Memory is already decrypted: dma-direct will not decrypt the memory but it will use the proper phys address conversions and page table prot.
The free part is more tricky as we already lose the information about allocation, so we have to check with each allocator separately, so swiotlb_is_decrypted() is added for SWIOTLB which is only allocator that can return decrypted memory.
Testing
=======
I was able to test this only under pKVM (arm64) as I have no access to other systems.
Future work
===========
Two other things I am also looking at which are related to restricted DMA pools, so they should be a different series.
1) Private pools: Currently all restricted DMA pools are decrypted (shared) by default. Having private pools would be useful for device assignment when bouncing is needed (as for non-coherent devices)
2) Optimizations for memory sharing. In some cases, allocations from restricted dma-pools are page aligned. For CoCo cases, that means that it will be cheaper to share memory in-place instead of bouncing.
Both of these add new semantics which need to be done carefully to avoid regressions, and might be a good candidate for a topic in the next LPC.
Patches
=======
- 1 Extend swiotlb
- 2-4 Refactoring
- 5 Fixes
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
Mostafa Saleh (5):
swiotlb: Return state of memory from swiotlb_alloc()
dma-mapping: Move encryption in __dma_direct_free_pages()
dma-mapping: Decrypt memory on remap
dma-mapping: Encapsulate memory state during allocation
dma-mapping: Fix memory decryption issues
include/linux/swiotlb.h | 25 +++++++- kernel/dma/direct.c | 134 +++++++++++++++++++++++++++------------- kernel/dma/swiotlb.c | 23 ++++++- 3 files changed, 135 insertions(+), 47 deletions(-)
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
RFC PATCH v3 1/5 swiotlb: Return state of memory from swiotlb_alloc()
Make swiotlb_alloc() return the state of the allocated memory, at the moment all the pools are decrypted but that would change soon. In the next patches dma-direct will use the returned state to determine whether to decrypt the memory and use the proper memory decryption/encryption related functions.
Also, add swiotlb_is_decrypted(), that will be used before calling swiotlb_free() to check whether the memory needs to be encrypted by the caller.
--- include/linux/swiotlb.h | 25 +++++++++++++++++++++++-- kernel/dma/direct.c | 2 +- kernel/dma/swiotlb.c | 23 ++++++++++++++++++++++- 3 files changed, 46 insertions(+), 4 deletions(-)
include/linux/swiotlb.h +23 -2
@@ -63,6 +63,7 @@ extern void __init swiotlb_update_mem_attributes(void);* @area_nslabs: Number of slots in each area. * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. +* @decrypted: Whether the pool was decrypted or left in default state. * @node: Member of the IO TLB memory pool list. * @rcu: RCU head for swiotlb_dyn_free(). * @transient: %true if transient memory pool.
@@ -77,6 +78,7 @@ struct io_tlb_pool {unsigned int area_nslabs;struct io_tlb_area *areas;struct io_tlb_slot *slots; +bool decrypted;#ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node;struct rcu_head rcu;
@@ -281,16 +283,31 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,extern void swiotlb_print_info (void ); +/* +* This contains the state of pages returned by swiotlb_alloc() +* A page can either be: +* SWIOTLB_PAGE_DEFAULT: The page was not decrypted by the pool. +* SWIOTLB_PAGE_DECRYPTED: The page was decrypted by the pool. +*/ +enum swiotlb_page_state { +SWIOTLB_PAGE_DEFAULT , +SWIOTLB_PAGE_DECRYPTED , +}; +#ifdef CONFIG_DMA_RESTRICTED_POOL -struct page *swiotlb_alloc (struct device *dev,size_t size); +struct page *swiotlb_alloc (struct device *dev,size_t size, +enum swiotlb_page_state *state);bool swiotlb_free (struct device *dev,struct page *page,size_t size); +bool swiotlb_is_decrypted (struct device *dev,struct page *page,size_t size); +static inline bool is_swiotlb_for_alloc (struct device *dev) {return dev->dma_io_tlb_mem->for_alloc; }#else -static inline struct page *swiotlb_alloc (struct device *dev,size_t size) +static inline struct page *swiotlb_alloc (struct device *dev,size_t size, +enum swiotlb_page_state *state) {return NULL ; }
@@ -299,6 +316,10 @@ static inline bool swiotlb_free(struct device *dev, struct page *page, {return false; } +static inline bool swiotlb_is_decrypted (struct device *dev,struct page *page,size_t size) +{ +return false; +}static inline bool is_swiotlb_for_alloc (struct device *dev) {return false;
kernel/dma/direct.c +1 -1
@@ -106,7 +106,7 @@ static void __dma_direct_free_pages(struct device *dev, struct page *page,static struct page *dma_direct_alloc_swiotlb (struct device *dev,size_t size) { -struct page *page =swiotlb_alloc (dev, size); +struct page *page =swiotlb_alloc (dev, size,NULL );if (page && !dma_coherent_ok (dev,page_to_phys (page), size)) {swiotlb_free (dev, page, size);
kernel/dma/swiotlb.c +22 -1
@@ -1763,7 +1763,8 @@ static inline void swiotlb_create_debugfs_files(struct io_tlb_mem *mem,#ifdef CONFIG_DMA_RESTRICTED_POOL -struct page *swiotlb_alloc (struct device *dev,size_t size) +struct page *swiotlb_alloc (struct device *dev,size_t size, +enum swiotlb_page_state *state) {struct io_tlb_mem *mem = dev->dma_io_tlb_mem;struct io_tlb_pool *pool;
@@ -1787,6 +1788,8 @@ struct page *swiotlb_alloc(struct device *dev, size_t size)return NULL ; } +if (state) + *state = pool->decrypted ?SWIOTLB_PAGE_DECRYPTED :SWIOTLB_PAGE_DEFAULT ;return pfn_to_page (PFN_DOWN (tlb_addr)); }
@@ -1804,6 +1807,18 @@ bool swiotlb_free(struct device *dev, struct page *page, size_t size)return true; } +bool swiotlb_is_decrypted (struct device *dev,struct page *page,size_t size) +{ +phys_addr_t tlb_addr =page_to_phys (page); +struct io_tlb_pool *pool; + + pool =swiotlb_find_pool (dev, tlb_addr); +if (!pool) +return false; + +return pool->decrypted; +} +static int rmem_swiotlb_device_init (struct reserved_mem *rmem,struct device *dev) {
@@ -1844,6 +1859,12 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem,return -ENOMEM ; } +/* +* At the moment all restricted dma pools are always decrypted, +* although that should change soon with CCA solutions introducing +* device passthrough. +*/ + pool->decrypted = true;set_memory_decrypted ((unsigned long )phys_to_virt (rmem->base), rmem->size >>PAGE_SHIFT ); swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs,
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Re: [RFC PATCH v3 1/5] swiotlb: Return state of memory from swiotlb_alloc()
Quoted 38 lines · Mostafa Saleh
Make
swiotlb_alloc()return the state of the allocated memory, at the moment all the pools are decrypted but that would change soon. In the next patches dma-direct will use the returned state to determine whether to decrypt the memory and use the proper memory decryption/encryption related functions.Also, add
swiotlb_is_decrypted(), that will be used before callingswiotlb_free()to check whether the memory needs to be encrypted by the caller.Signed-off-by Mostafa Saleh <[email protected]>--- include/linux/swiotlb.h | 25 +++++++++++++++++++++++-- kernel/dma/direct.c | 2 +- kernel/dma/swiotlb.c | 23 ++++++++++++++++++++++- 3 files changed, 46 insertions(+), 4 deletions(-)include/linux/swiotlb.h +2
-0@@ -63,6 +63,7 @@ extern void __init swiotlb_update_mem_attributes(void);* @area_nslabs: Number of slots in each area. * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. +* @decrypted: Whether the pool was decrypted or left in default state. * @node: Member of the IO TLB memory pool list. * @rcu: RCU head for swiotlb_dyn_free(). * @transient: %true if transient memory pool. @@ -77,6 +78,7 @@ struct io_tlb_pool {unsigned int area_nslabs;struct io_tlb_area *areas;struct io_tlb_slot *slots; +bool decrypted;#ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node;struct rcu_head rcu;
include/linux/swiotlb.h static inline void swiotlb_sync_single_for_cpu(struct device *dev, · line 283
Quoted 1 line
include/linux/swiotlb.h +0
-0@@ -281,16 +283,31 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,
Should this be a property of struct io_tlb_mem ?
-aneesh
Re: [RFC PATCH v3 1/5] swiotlb: Return state of memory from swiotlb_alloc()
Quoted 35 lines · Aneesh Kumar K.V
Quoted 33 lines · Mostafa Saleh
Make
swiotlb_alloc()return the state of the allocated memory, at the moment all the pools are decrypted but that would change soon. In the next patches dma-direct will use the returned state to determine whether to decrypt the memory and use the proper memory decryption/encryption related functions.Also, add
swiotlb_is_decrypted(), that will be used before callingswiotlb_free()to check whether the memory needs to be encrypted by the caller.Signed-off-by Mostafa Saleh <[email protected]>--- include/linux/swiotlb.h | 25 +++++++++++++++++++++++-- kernel/dma/direct.c | 2 +- kernel/dma/swiotlb.c | 23 ++++++++++++++++++++++- 3 files changed, 46 insertions(+), 4 deletions(-)include/linux/swiotlb.h +1
-0@@ -63,6 +63,7 @@ extern void __init swiotlb_update_mem_attributes(void);* @area_nslabs: Number of slots in each area. * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. +* @decrypted: Whether the pool was decrypted or left in default state. * @node: Member of the IO TLB memory pool list. * @rcu: RCU head for swiotlb_dyn_free(). * @transient: %true if transient memory pool. @@ -77,6 +78,7 @@ struct io_tlb_pool {unsigned int area_nslabs;struct io_tlb_area *areas;struct io_tlb_slot *slots;
Quoted 8 lines
include/linux/swiotlb.h static inline void swiotlb_sync_single_for_cpu(struct device *dev, · line 283
Quoted 6 lines
include/linux/swiotlb.h +1
-0+bool decrypted;#ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node;struct rcu_head rcu;@@ -281,16 +283,31 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,Should this be a property of struct
io_tlb_mem?
I envisioned that this would be mainly used by restricted-dma so in that case it doesn’t seem to matter.
But generally, I guess it would depend on the discovery mechanism of this memory property and I can imagine a memory allocator that has multiple pools with different attributes. So propably it's better to be per pool.
Thanks,
Mostafa
Quoted 2 lines
-aneesh
Re: [RFC PATCH v3 1/5] swiotlb: Return state of memory from swiotlb_alloc()
Quoted 45 lines · Mostafa Saleh
Quoted 35 lines · Aneesh Kumar K.V
Quoted 33 lines · Mostafa Saleh
Make
swiotlb_alloc()return the state of the allocated memory, at the moment all the pools are decrypted but that would change soon. In the next patches dma-direct will use the returned state to determine whether to decrypt the memory and use the proper memory decryption/encryption related functions.Also, add
swiotlb_is_decrypted(), that will be used before callingswiotlb_free()to check whether the memory needs to be encrypted by the caller.Signed-off-by Mostafa Saleh <[email protected]>--- include/linux/swiotlb.h | 25 +++++++++++++++++++++++-- kernel/dma/direct.c | 2 +- kernel/dma/swiotlb.c | 23 ++++++++++++++++++++++- 3 files changed, 46 insertions(+), 4 deletions(-)include/linux/swiotlb.h +1
-0@@ -63,6 +63,7 @@ extern void __init swiotlb_update_mem_attributes(void);* @area_nslabs: Number of slots in each area. * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. +* @decrypted: Whether the pool was decrypted or left in default state. * @node: Member of the IO TLB memory pool list. * @rcu: RCU head for swiotlb_dyn_free(). * @transient: %true if transient memory pool. @@ -77,6 +78,7 @@ struct io_tlb_pool {unsigned int area_nslabs;struct io_tlb_area *areas;struct io_tlb_slot *slots;Quoted 8 lines
include/linux/swiotlb.h static inline void swiotlb_sync_single_for_cpu(struct device *dev, · line 283
Quoted 6 lines
include/linux/swiotlb.h +1
-0+bool decrypted;#ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node;struct rcu_head rcu;@@ -281,16 +283,31 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,Should this be a property of struct
io_tlb_mem?
Quoted 7 lines
I envisioned that this would be mainly used by restricted-dma so in that case it doesn’t seem to matter.
But generally, I guess it would depend on the discovery mechanism of this memory property and I can imagine a memory allocator that has multiple pools with different attributes. So propably it's better to be per pool.
If we are going to make this generic, we may not be able to look at only restricted-dma. Instead, we will have to consider other SWIOTLB code paths as well, such as swiotlb_map(), swiotlb_alloc_pool() etc. For example, in swiotlb_dyn_alloc(), how do we determine whether to use decrypted or encrypted memory? I am thinking we want to use struct io_tlb_mem.decrypted field to determine that
struct io_tlb_mem *mem = container_of(work, struct io_tlb_mem, dyn_alloc); pool = swiotlb_alloc_pool(NULL, IO_TLB_MIN_SLABS, default_nslabs, default_nareas, mem->phys_limit, mem->decrypted, GFP_KERNEL);
Also, for things like swiotlb_tbl_map_single(), we might want to …
struct io_tlb_mem *mem = dev->dma_io_tlb_mem; /* if phys addr attribute is encrypted but the device is forcing an encrypted dma addr */ if (!(attrs & DMA_ATTR_CC_DECRYPTED) && force_dma_unencrypted(dev)) require_decrypted = true; if (require_decrypted != mem->decrypted) return (phys_addr_t)DMA_MAPPING_ERROR;
-aneesh
Re: [RFC PATCH v3 1/5] swiotlb: Return state of memory from swiotlb_alloc()
Quoted 72 lines · Aneesh Kumar K.V
Quoted 45 lines · Mostafa Saleh
Quoted 35 lines · Aneesh Kumar K.V
Quoted 33 lines · Mostafa Saleh
Make
swiotlb_alloc()return the state of the allocated memory, at the moment all the pools are decrypted but that would change soon. In the next patches dma-direct will use the returned state to determine whether to decrypt the memory and use the proper memory decryption/encryption related functions.Also, add
swiotlb_is_decrypted(), that will be used before callingswiotlb_free()to check whether the memory needs to be encrypted by the caller.Signed-off-by Mostafa Saleh <[email protected]>--- include/linux/swiotlb.h | 25 +++++++++++++++++++++++-- kernel/dma/direct.c | 2 +- kernel/dma/swiotlb.c | 23 ++++++++++++++++++++++- 3 files changed, 46 insertions(+), 4 deletions(-)include/linux/swiotlb.h +1
-0@@ -63,6 +63,7 @@ extern void __init swiotlb_update_mem_attributes(void);* @area_nslabs: Number of slots in each area. * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. +* @decrypted: Whether the pool was decrypted or left in default state. * @node: Member of the IO TLB memory pool list. * @rcu: RCU head for swiotlb_dyn_free(). * @transient: %true if transient memory pool. @@ -77,6 +78,7 @@ struct io_tlb_pool {unsigned int area_nslabs;struct io_tlb_area *areas;struct io_tlb_slot *slots;Quoted 8 lines
include/linux/swiotlb.h static inline void swiotlb_sync_single_for_cpu(struct device *dev, · line 283
Quoted 6 lines
include/linux/swiotlb.h +1
-0+bool decrypted;#ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node;struct rcu_head rcu;@@ -281,16 +283,31 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,Should this be a property of struct
io_tlb_mem?Quoted 7 lines
I envisioned that this would be mainly used by restricted-dma so in that case it doesn’t seem to matter.
But generally, I guess it would depend on the discovery mechanism of this memory property and I can imagine a memory allocator that has multiple pools with different attributes. So propably it's better to be per pool.If we are going to make this generic, we may not be able to look at only restricted-dma. Instead, we will have to consider other SWIOTLB code paths as well, such as
swiotlb_map(),swiotlb_alloc_pool()etc. For example, inswiotlb_dyn_alloc(), how do we determine whether to use decrypted or encrypted memory? I am thinking we want to use struct io_tlb_mem.decrypted field to determine thatstruct io_tlb_mem *mem = container_of(work, struct io_tlb_mem, dyn_alloc); pool = swiotlb_alloc_pool(NULL, IO_TLB_MIN_SLABS, default_nslabs, default_nareas, mem->phys_limit, mem->decrypted, GFP_KERNEL);Also, for things like
swiotlb_tbl_map_single(), we might want to …
Quoted 8 lines
struct io_tlb_mem *mem = dev->dma_io_tlb_mem; /* if phys addr attribute is encrypted but the device is forcing an encrypted dma addr */ if (!(attrs & DMA_ATTR_CC_DECRYPTED) && force_dma_unencrypted(dev)) require_decrypted = true; if (require_decrypted != mem->decrypted) return (phys_addr_t)DMA_MAPPING_ERROR;
I guess that depends on the discovery mechanism, I was thinking restricted-dma pools can represent that from firmware as device tree binding.
For SWIOTLB, I am not really sure how that works, as they are shared by all devices on the system. I was thinking it remains decrypted as it is typically used by simple devices that doesn't deal with encrypted memory and devices that require special pools (whether decrypted or encrypted) they would use restricted-dma.
Thanks,
Mostafa
Quoted 2 lines
-aneesh
RFC PATCH v3 2/5 dma-mapping: Move encryption in __dma_direct_free_pages()
In reply to Mostafa SalehIn the next patches, we will need to avoid encrypting memory allocated from SWIOTLB, so instead of calling dma_set_encrypted() before __dma_direct_free_pages(), call it inside, conditional on the memory state passed to the function.
--- kernel/dma/direct.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
kernel/dma/direct.c +11 -11
@@ -97,8 +97,11 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) }static void __dma_direct_free_pages (struct device *dev,struct page *page, -size_t size) +size_t size,bool encrypt) { +if (encrypt &&dma_set_encrypted (dev,page_address (page), size)) +return ; +if (swiotlb_free (dev, page, size))return ;dma_free_contiguous (dev, page, size);
@@ -203,7 +206,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { -bool remap = false, set_uncached = false; +bool remap = false, set_uncached = false, encrypt = false;struct page *page;void *ret;
@@ -298,10 +301,9 @@ void *dma_direct_alloc(struct device *dev, size_t size,return ret; out_encrypt_pages: -if (dma_set_encrypted (dev,page_address (page), size)) -return NULL ; + encrypt = true; out_free_pages: -__dma_direct_free_pages (dev, page, size); +__dma_direct_free_pages (dev, page, size, encrypt);return NULL ; out_leak_pages:return NULL ;
@@ -311,6 +313,7 @@ void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { unsigned int page_order = get_order(size); +bool encrypt = false;if ((attrs &DMA_ATTR_NO_KERNEL_MAPPING ) && !force_dma_unencrypted (dev) && !is_swiotlb_for_alloc (dev)) {
@@ -343,11 +346,10 @@ void dma_direct_free(struct device *dev, size_t size, }else {if (IS_ENABLED (CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED ))arch_dma_clear_uncached (cpu_addr, size); -if (dma_set_encrypted (dev, cpu_addr, size)) -return ; + encrypt = true; } -__dma_direct_free_pages (dev,dma_direct_to_page (dev, dma_addr), size); +__dma_direct_free_pages (dev,dma_direct_to_page (dev, dma_addr), size, encrypt); } struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
@@ -384,9 +386,7 @@ void dma_direct_free_pages(struct device *dev, size_t size, dma_free_from_pool(dev, vaddr, size))return ; -if (dma_set_encrypted (dev, vaddr, size)) -return ; -__dma_direct_free_pages (dev, page, size); +__dma_direct_free_pages (dev, page, size, true); } #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Re: [RFC PATCH v3 2/5] dma-mapping: Move encryption in __dma_direct_free_pages()
Quoted 15 lines · Mostafa Saleh
In the next patches, we will need to avoid encrypting memory allocated from SWIOTLB, so instead of calling
dma_set_encrypted()before__dma_direct_free_pages(), call it inside, conditional on the memory state passed to the function.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) · line 101
Quoted 6 lines
kernel/dma/direct.c +1
-1@@ -97,8 +97,11 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) }static void __dma_direct_free_pages (struct device *dev,struct page *page, -size_t size) +size_t size,bool encrypt) {
This feels like it would be nicer if it could be the swiotlb_page_state instead of a bool, maybe the enum needs a different name.
Jason
Re: [RFC PATCH v3 2/5] dma-mapping: Move encryption in __dma_direct_free_pages()
Quoted 18 lines · Jason Gunthorpe
Quoted 17 lines · Mostafa Saleh
In the next patches, we will need to avoid encrypting memory allocated from SWIOTLB, so instead of calling
dma_set_encrypted()before__dma_direct_free_pages(), call it inside, conditional on the memory state passed to the function.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)kernel/dma/direct.c +0
-0@@ -97,8 +97,11 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) }
Quoted 8 lines
kernel/dma/direct.c static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) · line 101
Quoted 4 lines
kernel/dma/direct.c +1
-1static void __dma_direct_free_pages (struct device *dev,struct page *page, -size_t size) +size_t size,bool encrypt) {This feels like it would be nicer if it could be the
swiotlb_page_stateinstead of a bool, maybe the enum needs a different name.
I am not sure I get this, at this point the swiotlb stuff is not introduced yet, I try to refactor the free path to make is easier to modify next, the bool encrypt here is typically set to false from the error path.
I can replace that with a wrapper doing the encryption and calling the free function.
Thanks,
Mostafa
Quoted 2 lines
Jason
Re: [RFC PATCH v3 2/5] dma-mapping: Move encryption in __dma_direct_free_pages()
Quoted 28 lines · Mostafa Saleh
Quoted 18 lines · Jason Gunthorpe
Quoted 17 lines · Mostafa Saleh
In the next patches, we will need to avoid encrypting memory allocated from SWIOTLB, so instead of calling
dma_set_encrypted()before__dma_direct_free_pages(), call it inside, conditional on the memory state passed to the function.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)kernel/dma/direct.c +0
-0@@ -97,8 +97,11 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) }Quoted 8 lines
kernel/dma/direct.c static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) · line 101
Quoted 4 lines
kernel/dma/direct.c +1
-1static void __dma_direct_free_pages (struct device *dev,struct page *page, -size_t size) +size_t size,bool encrypt) {This feels like it would be nicer if it could be the
swiotlb_page_stateinstead of a bool, maybe the enum needs a different name.
Quoted 2 lines
I am not sure I get this, at this point the swiotlb stuff is not introduced yet,
Just that using bool as a function argument for a flag is often frowned on, and you already have an enum encoding the same flag state.
Jason
Re: [RFC PATCH v3 2/5] dma-mapping: Move encryption in __dma_direct_free_pages()
Quoted 29 lines · Jason Gunthorpe
Quoted 28 lines · Mostafa Saleh
Quoted 26 lines · Jason Gunthorpe
Quoted 15 lines · Mostafa Saleh
In the next patches, we will need to avoid encrypting memory allocated from SWIOTLB, so instead of calling
dma_set_encrypted()before__dma_direct_free_pages(), call it inside, conditional on the memory state passed to the function.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) · line 101
Quoted 6 lines
kernel/dma/direct.c +1
-1@@ -97,8 +97,11 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) }static void __dma_direct_free_pages (struct device *dev,struct page *page, -size_t size) +size_t size,bool encrypt) {This feels like it would be nicer if it could be the
swiotlb_page_stateinstead of a bool, maybe the enum needs a different name.
Quoted 6 lines
Quoted 2 lines
I am not sure I get this, at this point the swiotlb stuff is not introduced yet,
Just that using bool as a function argument for a flag is often frowned on, and you already have an enum encoding the same flag state.
I see, the enum is not introduced at this point, but I can change that.
Thanks,
Mostafa
Quoted 2 lines
Jason
RFC PATCH v3 3/5 dma-mapping: Decrypt memory on remap
In reply to Mostafa SalehIn case memory needs to be remapped on systems with force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.
The memory still needs to be decrypted in that case.
With memory decryption, don't allow highmem allocations, but that shouldn't be a problem on such modern systems.
Also, move force_dma_unencrypted() outside of dma_set_* to make it clear to be able to use more generic logic to decided memory state.
--- kernel/dma/direct.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-)
kernel/dma/direct.c +14 -17
@@ -79,8 +79,6 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) -return 0 ;return set_memory_decrypted ((unsigned long )vaddr,PFN_UP (size)); }
@@ -88,8 +86,6 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) {int ret; -if (!force_dma_unencrypted (dev)) -return 0 ; ret =set_memory_encrypted ((unsigned long )vaddr,PFN_UP (size));if (ret)pr_warn_ratelimited ("leaking DMA memory that can't be re-encrypted\n" );
@@ -206,7 +202,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { -bool remap = false, set_uncached = false, encrypt = false; +bool remap = false, set_uncached = false, decrypt =force_dma_unencrypted (dev);struct page *page;void *ret;
@@ -215,7 +211,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, gfp |= __GFP_NOWARN;if ((attrs &DMA_ATTR_NO_KERNEL_MAPPING ) && - !force_dma_unencrypted (dev) && !is_swiotlb_for_alloc (dev)) + !decrypt && !is_swiotlb_for_alloc (dev))return dma_direct_alloc_no_mapping (dev, size, dma_handle, gfp);if (!dev_is_dma_coherent (dev)) {
@@ -249,12 +245,15 @@ void *dma_direct_alloc(struct device *dev, size_t size,* Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. */ -if ((remap ||force_dma_unencrypted (dev)) && +if ((remap || decrypt) &&dma_direct_use_pool (dev, gfp))return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp); -/* we always manually zero the memory once we are done */ - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); +/* +* we always manually zero the memory once we are done, and only allow +* high mem if pages doesn't need decryption. +*/ + page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, !decrypt);if (!page)return NULL ;
@@ -268,10 +267,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (decrypt &&dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages;if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs); -if (force_dma_unencrypted (dev)) +if (decrypt) prot =pgprot_decrypted (prot);/* remove any dirty cache lines on the kernel alias */
@@ -281,11 +282,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =dma_common_contiguous_remap (page, size, prot,__builtin_return_address (0 ));if (!ret) - goto out_free_pages; + goto out_encrypt_pages; }else { ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) - goto out_leak_pages; }memset (ret,0 , size);
@@ -301,9 +300,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,return ret; out_encrypt_pages: - encrypt = true; -out_free_pages: -__dma_direct_free_pages (dev, page, size, encrypt); +__dma_direct_free_pages (dev, page, size, decrypt);return NULL ; out_leak_pages:return NULL ;
@@ -366,7 +363,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,return NULL ; ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) +if (force_dma_unencrypted (dev) &&dma_set_decrypted (dev, ret, size)) goto out_leak_pages;memset (ret,0 , size); *dma_handle =phys_to_dma_direct (dev,page_to_phys (page));
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Re: [RFC PATCH v3 3/5] dma-mapping: Decrypt memory on remap
Quoted 7 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
For ARM CCA, we cannot mark a vmap address as decrypted. I don’t expect non-coherent DMA devices to be used in an ARM CCA configuration, but we may need a way to document this in the code.
Quoted 123 lines
With memory decryption, don't allow highmem allocations, but that shouldn't be a problem on such modern systems.
Also, move
force_dma_unencrypted()outside of dma_set_* to make it clear to be able to use more generic logic to decided memory state.Reported-by Catalin Marinas <[email protected]>Fixes f3c962226dbe ("dma-direct: clean up the remapping checks in dma_direct_alloc")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-)kernel/dma/direct.c +14
-17@@ -79,8 +79,6 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) -return 0 ;return set_memory_decrypted ((unsigned long )vaddr,PFN_UP (size)); }@@ -88,8 +86,6 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) {int ret; -if (!force_dma_unencrypted (dev)) -return 0 ; ret =set_memory_encrypted ((unsigned long )vaddr,PFN_UP (size));if (ret)pr_warn_ratelimited ("leaking DMA memory that can't be re-encrypted\n" );@@ -206,7 +202,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { -bool remap = false, set_uncached = false, encrypt = false; +bool remap = false, set_uncached = false, decrypt =force_dma_unencrypted (dev);struct page *page;void *ret;@@ -215,7 +211,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, gfp |= __GFP_NOWARN;if ((attrs &DMA_ATTR_NO_KERNEL_MAPPING ) && - !force_dma_unencrypted (dev) && !is_swiotlb_for_alloc (dev)) + !decrypt && !is_swiotlb_for_alloc (dev))return dma_direct_alloc_no_mapping (dev, size, dma_handle, gfp);if (!dev_is_dma_coherent (dev)) {@@ -249,12 +245,15 @@ void *dma_direct_alloc(struct device *dev, size_t size,* Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. */ -if ((remap ||force_dma_unencrypted (dev)) && +if ((remap || decrypt) &&dma_direct_use_pool (dev, gfp))return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp); -/* we always manually zero the memory once we are done */ - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); +/* +* we always manually zero the memory once we are done, and only allow +* high mem if pages doesn't need decryption. +*/ + page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, !decrypt);if (!page)return NULL ;@@ -268,10 +267,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (decrypt &&dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages;if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs); -if (force_dma_unencrypted (dev)) +if (decrypt) prot =pgprot_decrypted (prot);/* remove any dirty cache lines on the kernel alias */ @@ -281,11 +282,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =dma_common_contiguous_remap (page, size, prot,__builtin_return_address (0 ));if (!ret) - goto out_free_pages; + goto out_encrypt_pages; }else { ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) - goto out_leak_pages; }memset (ret,0 , size);@@ -301,9 +300,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,return ret; out_encrypt_pages: - encrypt = true; -out_free_pages: -__dma_direct_free_pages (dev, page, size, encrypt); +__dma_direct_free_pages (dev, page, size, decrypt);return NULL ; out_leak_pages:return NULL ;@@ -366,7 +363,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,return NULL ; ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) +if (force_dma_unencrypted (dev) &&dma_set_decrypted (dev, ret, size)) goto out_leak_pages;memset (ret,0 , size); *dma_handle =phys_to_dma_direct (dev,page_to_phys (page));Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Re: [RFC PATCH v3 3/5] dma-mapping: Decrypt memory on remap
Quoted 11 lines · Aneesh Kumar K.V
Quoted 7 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
For ARM CCA, we cannot mark a vmap address as decrypted.
Why not? pgprot_decrypted is passed to vmap, why can't it work?
Jason
Re: [RFC PATCH v3 3/5] dma-mapping: Decrypt memory on remap
Quoted 8 lines · Jason Gunthorpe
Quoted 7 lines · Aneesh Kumar K.V
Quoted 5 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The memory still needs to be decrypted in that case.
For ARM CCA, we cannot mark a vmap address as decrypted.
Why not?
pgprot_decryptedis passed to vmap, why can't it work?
I might have confused you in my previous reply. What I meant is that, if we do not have a linear map, we currently cannot change the page attributes. We are avoiding that by requesting a non-HighMem address. What I am suggesting here is that we should document this, or perhaps handle it as a separate patch as done in [1] by explicitly stating the challenges
[1] https://lore.kernel.org/all/[email protected]
Quoted 1 line
Jason
Re: [RFC PATCH v3 3/5] dma-mapping: Decrypt memory on remap
Quoted 15 lines · Aneesh Kumar K.V
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Aneesh Kumar K.V
Quoted 7 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
For ARM CCA, we cannot mark a vmap address as decrypted.
Quoted 7 lines
Quoted 2 lines
Why not?
pgprot_decryptedis passed to vmap, why can't it work?I might have confused you in my previous reply. What I meant is that, if we do not have a linear map, we currently cannot change the page attributes.
I'm not sure that is really a CCA issue..
We have an API design problem with set_memory_decrypted(), it needs three flavours to accommodate all the different users
kva void *
cpu memory struct page *
physical mmio phys_addr_t
Currently these are all being muddled up with quite some creative mis-use of APIs. The DMA API should just call the page * version and not invoke page address.
I guess this became extra complex because Intel needs to track down the KVAs and wipe them out.
Quoted 3 lines
What I am suggesting here is that we should document this, or perhaps handle it as a separate patch as done in [1] by explicitly stating the challenges
It should be centralized.. If we had a struct page argument version it could do the page highmem check and fail with an appropriate comment.
This shouldn't be sprinkled all over.
I suspect it could all be made to work even with highmem without alot of trouble.
Jason
Re: [RFC PATCH v3 3/5] dma-mapping: Decrypt memory on remap
In reply to Aneesh Kumar K.VQuoted 11 lines · Aneesh Kumar K.V
Quoted 6 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
For ARM CCA, we cannot mark a vmap address as decrypted. I don’t expect non-coherent DMA devices to be used in an ARM CCA configuration, but we may need a way to document this in the code.
I think you clarified what you meant later. The vmap here is just fine, we pass pgprot_decrypted(). The key is the dma_set_decrypted() on the page address before being remapped.
On arm64 we probably don't care but if we want to enforce it for all architectures, dma_direct_alloc() could return NULL if force_dma_unencrypted() && remap (maybe together with a WARN_ON_ONCE). However, the proposed patch is small enough to cover this case as well.
Signature · 1 line
Catalin
Re: [RFC PATCH v3 3/5] dma-mapping: Decrypt memory on remap
Quoted 13 lines · Catalin Marinas
Quoted 11 lines · Aneesh Kumar K.V
Quoted 6 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
For ARM CCA, we cannot mark a vmap address as decrypted. I don’t expect non-coherent DMA devices to be used in an ARM CCA configuration, but we may need a way to document this in the code.
Quoted 9 lines
I think you clarified what you meant later. The vmap here is just fine, we pass
pgprot_decrypted(). The key is thedma_set_decrypted()on the page address before being remapped.On arm64 we probably don't care but if we want to enforce it for all architectures,
dma_direct_alloc()could return NULL ifforce_dma_unencrypted()&& remap (maybe together with aWARN_ON_ONCE). However, the proposed patch is small enough to cover this case as well.
I have posted a new series clarifying the commit message and adding further changes here https://lore.kernel.org/all/[email protected]
-aneesh
RFC PATCH v3 4/5 dma-mapping: Encapsulate memory state during allocation
In reply to Mostafa SalehIntroduce a new dma-direct internal type dma_page which is "struct page" and a bit indicate whether the memory has been decrypted or not.
This is useful to pass such information encapsulated through allocation functions, which is currently set from swiotlb_alloc().
No functional changes.
--- kernel/dma/direct.c | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 12 deletions(-)
kernel/dma/direct.c +46 -12
@@ -16,6 +16,33 @@#include <linux/pci-p2pdma.h> #include "direct.h" +/* +* Represent DMA allocation and 1 bit flag for it's state +*/ +struct dma_page { +unsigned long val; +}; + +#define DMA_PAGE_DECRYPTED_FLAG BIT(0) + +#define DMA_PAGE_NULL ((struct dma_page){ .val = 0 }) + +static inline struct dma_page page_to_dma_page (struct page *page,bool decrypted) +{ +struct dma_page dma_page; + + dma_page.val = (unsigned long )page; +if (decrypted) + dma_page.val |=DMA_PAGE_DECRYPTED_FLAG ; + +return dma_page; +} + +static inline struct page *dma_page_to_page (struct dma_page dma_page) +{ +return (struct page *)(dma_page.val & ~DMA_PAGE_DECRYPTED_FLAG ); +} +/* * Most architectures use ZONE_DMA for the first 16 Megabytes, but some use * it for entirely different regions. In that case the arch code needs to
@@ -103,20 +130,21 @@ static void __dma_direct_free_pages(struct device *dev, struct page *page,dma_free_contiguous (dev, page, size); } -static struct page *dma_direct_alloc_swiotlb (struct device *dev,size_t size) +static struct dma_page dma_direct_alloc_swiotlb (struct device *dev,size_t size) { -struct page *page =swiotlb_alloc (dev, size,NULL ); +enum swiotlb_page_state state; +struct page *page =swiotlb_alloc (dev, size, &state);if (page && !dma_coherent_ok (dev,page_to_phys (page), size)) {swiotlb_free (dev, page, size); -return NULL ; +return DMA_PAGE_NULL ; } -return page; +return page_to_dma_page (page, state ==SWIOTLB_PAGE_DECRYPTED ); } -static struct page *__dma_direct_alloc_pages (struct device *dev,size_t size, -gfp_t gfp,bool allow_highmem) +static struct dma_page __dma_direct_alloc_pages (struct device *dev,size_t size, +gfp_t gfp,bool allow_highmem) {int node =dev_to_node (dev);struct page *page;
@@ -132,7 +160,7 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size,if (page) {if (dma_coherent_ok (dev,page_to_phys (page), size) && (allow_highmem || !PageHighMem (page))) -return page; +return page_to_dma_page (page, false);dma_free_contiguous (dev, page, size); }
@@ -148,10 +176,10 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size,else if (IS_ENABLED (CONFIG_ZONE_DMA ) && !(gfp &GFP_DMA )) gfp = (gfp & ~GFP_DMA32 ) |GFP_DMA ;else -return NULL ; +return DMA_PAGE_NULL ; } -return page; +return page_to_dma_page (page, false); }/*
@@ -184,9 +212,11 @@ static void *dma_direct_alloc_from_pool(struct device *dev, size_t size,static void *dma_direct_alloc_no_mapping (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp) { +struct dma_page dma_page;struct page *page; - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + dma_page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =dma_page_to_page (dma_page);if (!page)return NULL ;
@@ -203,6 +233,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap = false, set_uncached = false, decrypt = force_dma_unencrypted(dev); +struct dma_page dma_page;struct page *page;void *ret;
@@ -253,7 +284,8 @@ void *dma_direct_alloc(struct device *dev, size_t size,* we always manually zero the memory once we are done, and only allow * high mem if pages doesn't need decryption. */ - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, !decrypt); + dma_page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, !decrypt); + page =dma_page_to_page (dma_page);if (!page)return NULL ;
@@ -352,13 +384,15 @@ void dma_direct_free(struct device *dev, size_t size,struct page *dma_direct_alloc_pages (struct device *dev,size_t size,dma_addr_t *dma_handle,enum dma_data_direction dir,gfp_t gfp) { +struct dma_page dma_page;struct page *page;void *ret;if (force_dma_unencrypted (dev) &&dma_direct_use_pool (dev, gfp))return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp); - page =__dma_direct_alloc_pages (dev, size, gfp, false); + dma_page =__dma_direct_alloc_pages (dev, size, gfp, false); + page =dma_page_to_page (dma_page);if (!page)return NULL ;
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Re: [RFC PATCH v3 4/5] dma-mapping: Encapsulate memory state during allocation
Quoted 18 lines · Mostafa Saleh
Introduce a new dma-direct internal type
dma_pagewhich is "struct page" and a bit indicate whether the memory has been decrypted or not.
This is useful to pass such information encapsulated through allocation functions, which is currently set fromswiotlb_alloc().No functional changes.
Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 12 deletions(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c line 21
Quoted 6 lines
kernel/dma/direct.c +3
-0@@ -16,6 +16,33 @@#include <linux/pci-p2pdma.h> #include "direct.h" +/* +* Represent DMA allocation and 1 bit flag for it's state +*/
I'd explain this wrappers a pointer and uses the low PAGE_SHIFT bits for flags..
kernel/dma/direct.c line 23
Quoted 2 lines
kernel/dma/direct.c +2
-0+struct dma_page { +unsigned long val;
unintptr_t ?
Quoted 9 lines
kernel/dma/direct.c +2
-2@@ -103,20 +130,21 @@ static void __dma_direct_free_pages(struct device *dev, struct page *page,dma_free_contiguous (dev, page, size); } -static struct page *dma_direct_alloc_swiotlb (struct device *dev,size_t size) +static struct dma_page dma_direct_alloc_swiotlb (struct device *dev,size_t size) { -struct page *page =swiotlb_alloc (dev, size,NULL ); +enum swiotlb_page_state state;
kernel/dma/direct.c static void __dma_direct_free_pages(struct device *dev, struct page *page, · line 143
Quoted 10 lines
kernel/dma/direct.c +3
-2+struct page *page =swiotlb_alloc (dev, size, &state);if (page && !dma_coherent_ok (dev,page_to_phys (page), size)) {swiotlb_free (dev, page, size); -return NULL ; +return DMA_PAGE_NULL ; } -return page; +return page_to_dma_page (page, state ==SWIOTLB_PAGE_DECRYPTED );
Should the struct dma_page have been introduced earlier instead of the swiotlb_page_state ? Seems a bit odd to have both
If these are actually internally allocated struct pages, could you use the struct page memory itself to record the decrypted state? That would require more significant changes to the allocator calls.
kernel/dma/direct.c static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, · line 221
Quoted 12 lines
kernel/dma/direct.c +3
-1@@ -184,9 +212,11 @@ static void *dma_direct_alloc_from_pool(struct device *dev, size_t size,static void *dma_direct_alloc_no_mapping (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp) { +struct dma_page dma_page;struct page *page; - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + dma_page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =dma_page_to_page (dma_page);if (!page)return NULL ;
I would expect to see more usage of the dma_page here..
Like I don't think this is really right:
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
Does page_to_phys(page) really work on decrypted memory? On CCA it will return the protected alias which doesn't seem like something useful?
static inline dma_addr_t phys_to_dma_direct(struct device *dev,
phys_addr_t phys)
{
if (force_dma_unencrypted(dev))
return phys_to_dma_unencrypted(dev, phys);
return phys_to_dma(dev, phys);Above is all nonsense now that you have a direct indication of the address is decrypted memory or not, it should also be used right here directly.
if (is_dma_page_decrypted(dma_page))
*dma_handle = phys_to_dma_unencrypted(..)
else
*dma_handle = phys_to_dma(..);
The later patch just makes it worse by adding even more confusing flags to phys_to_dma_direct().
I think it should work out that everyone already knows what memory type they are working with before they call down to phys_to_dma_direct() - the calls to force_dma_unecrypted() here are just hacks because it previously did not.
Anyhow, I think this series is alot better than the previous one. If you work a little harder to make it so there is only one force_dma_unecrypted() per high level DMA API call that would be perfect.
Jason
Re: [RFC PATCH v3 4/5] dma-mapping: Encapsulate memory state during allocation
Quoted 108 lines · Jason Gunthorpe
Quoted 18 lines · Mostafa Saleh
Introduce a new dma-direct internal type
dma_pagewhich is "struct page" and a bit indicate whether the memory has been decrypted or not.
This is useful to pass such information encapsulated through allocation functions, which is currently set fromswiotlb_alloc().No functional changes.
Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 12 deletions(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c line 21
Quoted 6 lines
kernel/dma/direct.c +3
-0@@ -16,6 +16,33 @@#include <linux/pci-p2pdma.h> #include "direct.h" +/* +* Represent DMA allocation and 1 bit flag for it's state +*/ I'd explain this wrappers a pointer and uses the low
PAGE_SHIFTbits for flags..kernel/dma/direct.c line 23
Quoted 2 lines
kernel/dma/direct.c +2
-0+struct dma_page { +unsigned long val;
unintptr_t?Quoted 9 lines
kernel/dma/direct.c +2
-2@@ -103,20 +130,21 @@ static void __dma_direct_free_pages(struct device *dev, struct page *page,dma_free_contiguous (dev, page, size); } -static struct page *dma_direct_alloc_swiotlb (struct device *dev,size_t size) +static struct dma_page dma_direct_alloc_swiotlb (struct device *dev,size_t size) { -struct page *page =swiotlb_alloc (dev, size,NULL ); +enum swiotlb_page_state state;kernel/dma/direct.c static void __dma_direct_free_pages(struct device *dev, struct page *page, · line 143
Quoted 10 lines
kernel/dma/direct.c +3
-2+struct page *page =swiotlb_alloc (dev, size, &state);if (page && !dma_coherent_ok (dev,page_to_phys (page), size)) {swiotlb_free (dev, page, size); -return NULL ; +return DMA_PAGE_NULL ; } -return page; +return page_to_dma_page (page, state ==SWIOTLB_PAGE_DECRYPTED );Should the struct
dma_pagehave been introduced earlier instead of theswiotlb_page_state? Seems a bit odd to have bothIf these are actually internally allocated struct pages, could you use the struct page memory itself to record the decrypted state? That would require more significant changes to the allocator calls.
kernel/dma/direct.c static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, · line 221
Quoted 12 lines
kernel/dma/direct.c +3
-1@@ -184,9 +212,11 @@ static void *dma_direct_alloc_from_pool(struct device *dev, size_t size,static void *dma_direct_alloc_no_mapping (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp) { +struct dma_page dma_page;struct page *page; - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + dma_page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =dma_page_to_page (dma_page);if (!page)return NULL ;I would expect to see more usage of the
dma_pagehere..Like I don't think this is really right:
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));Does
page_to_phys(page)really work on decrypted memory? On CCA it will return the protected alias which doesn't seem like something useful?static inline
dma_addr_tphys_to_dma_direct(struct device *dev,
phys_addr_tphys){ if (force_dma_unencrypted(dev)) return phys_to_dma_unencrypted(dev, phys); return phys_to_dma(dev, phys);Above is all nonsense now that you have a direct indication of the address is decrypted memory or not, it should also be used right here directly.
if (
is_dma_page_decrypted(dma_page))
*dma_handle=phys_to_dma_unencrypted(..)
else
*dma_handle=phys_to_dma(..);The later patch just makes it worse by adding even more confusing flags to
phys_to_dma_direct().I think it should work out that everyone already knows what memory type they are working with before they call down to
phys_to_dma_direct()- the calls toforce_dma_unecrypted()here are just hacks because it previously did not.
Quoted 4 lines
Anyhow, I think this series is alot better than the previous one. If you work a little harder to make it so there is only one
force_dma_unecrypted()per high level DMA API call that would be perfect.
I tried the following:
1. Applied force_dma_unencrypted() at the high-level DMA APIs.
2. Marked the atomic pools as decrypted.
3. Marked swiotlb io_tlb_mem as decrypted.
4. Avoided adding dma_page; instead, we can cover all functions by passing arguments.
This is a fairly large change, and I was wondering whether this is actually the direction we want to take.
If this looks good, I can start working on splitting them as smaller patches.
drivers/iommu/dma-iommu.c +1 -1
@@ -1650,7 +1650,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,if (IS_ENABLED (CONFIG_DMA_DIRECT_REMAP ) && !gfpflags_allow_blocking (gfp) && !coherent) page =dma_alloc_from_pool (dev,PAGE_ALIGN (size), &cpu_addr, - gfp,NULL ); + gfp, attrs &DMA_ATTR_CC_DECRYPTED ,NULL );else cpu_addr =iommu_dma_alloc_pages (dev, size, &page, gfp, attrs);if (!cpu_addr)
include/linux/dma-direct.h +10 -2
@@ -90,14 +90,22 @@ static inline dma_addr_t phys_to_dma_unencrypted(struct device *dev, {return dma_addr_unencrypted (__phys_to_dma (dev, paddr)); } + +static inline dma_addr_t phys_to_dma_encrypted (struct device *dev, +phys_addr_t paddr) +{ +return dma_addr_encrypted (__phys_to_dma (dev, paddr)); +}/* * If memory encryption is supported, phys_to_dma will set the memory encryption * bit in the DMA address, and dma_to_phys will clear it. * phys_to_dma_unencrypted is for use on special unencrypted memory like swiotlb * buffers. */ -static inline dma_addr_t phys_to_dma (struct device *dev,phys_addr_t paddr) +static inline dma_addr_t phys_to_dma (struct device *dev,phys_addr_t paddr,bool unencrypted) { +if (unencrypted) +return dma_addr_unencrypted (__phys_to_dma (dev, paddr));return dma_addr_encrypted (__phys_to_dma (dev, paddr)); }
@@ -132,7 +140,7 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size,if (addr ==DMA_MAPPING_ERROR )return false;if (is_ram && !IS_ENABLED (CONFIG_ARCH_DMA_ADDR_T_64BIT ) && -min (addr, end) <phys_to_dma (dev,PFN_PHYS (min_low_pfn))) +min (addr, end) <__phys_to_dma (dev,PFN_PHYS (min_low_pfn)))return false;return end <=min_not_zero (*dev->dma_mask, dev->bus_dma_limit);
include/linux/dma-map-ops.h +1 -1
@@ -217,7 +217,7 @@ void *dma_common_pages_remap(struct page **pages, size_t size, pgprot_t prot,void dma_common_free_remap (void *cpu_addr,size_t size);struct page *dma_alloc_from_pool (struct device *dev,size_t size, -void **cpu_addr,gfp_t flags, +void **cpu_addr,gfp_t flags,bool require_decrypted,bool (*phys_addr_ok)(struct device *,phys_addr_t ,size_t ));bool dma_free_from_pool (struct device *dev,void *start,size_t size);
include/linux/dma-mapping.h +8 -0
@@ -93,6 +93,14 @@*/ #define DMA_ATTR_REQUIRE_COHERENT (1UL << 12) +/* +* DMA_ATTR_CC_DECRYPTED: Indicates memory that has been explicitly decrypted +* (shared) for confidential computing guests. The caller must have +* called set_memory_decrypted(). A struct page is required. +*/ +#define DMA_ATTR_CC_DECRYPTED (1UL << 13) + +/* * A dma_addr_t can hold any valid DMA or bus address for the platform. It can * be given to a device to use as a DMA source or target. It is specific to a
include/linux/swiotlb.h +3 -2
@@ -111,6 +111,7 @@ struct io_tlb_mem {struct dentry *debugfs;bool force_bounce;bool for_alloc; +bool decrypted;#ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow;u64 phys_limit;
@@ -282,7 +283,7 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,extern void swiotlb_print_info (void );#ifdef CONFIG_DMA_RESTRICTED_POOL -struct page *swiotlb_alloc (struct device *dev,size_t size); +struct page *swiotlb_alloc (struct device *dev,size_t size,bool require_decrypted);bool swiotlb_free (struct device *dev,struct page *page,size_t size);static inline bool is_swiotlb_for_alloc (struct device *dev)
@@ -290,7 +291,7 @@ static inline bool is_swiotlb_for_alloc(struct device *dev) return dev->dma_io_tlb_mem->for_alloc; }#else -static inline struct page *swiotlb_alloc (struct device *dev,size_t size) +static inline struct page *swiotlb_alloc (struct device *dev,size_t size,bool require_decrypted) {return NULL ; }
kernel/dma/coherent.c +1 -1
@@ -31,7 +31,7 @@ static inline dma_addr_t dma_get_device_base(struct device *dev, struct dma_coherent_mem * mem) {if (mem->use_dev_dma_pfn_offset) -return phys_to_dma (dev,PFN_PHYS (mem->pfn_base)); +return __phys_to_dma (dev,PFN_PHYS (mem->pfn_base));return mem->device_base; }
kernel/dma/direct.c +95 -37
294 lines not on this page · raw message
kernel/dma/direct.h +8 -1
@@ -83,6 +83,9 @@ static inline dma_addr_t dma_direct_map_phys(struct device *dev, {dma_addr_t dma_addr; +if (force_dma_unencrypted (dev)) + attrs |=DMA_ATTR_CC_DECRYPTED ; +if (is_swiotlb_force_bounce (dev)) {if (attrs & (DMA_ATTR_MMIO |DMA_ATTR_REQUIRE_COHERENT ))return DMA_MAPPING_ERROR ;
@@ -94,8 +97,12 @@ static inline dma_addr_t dma_direct_map_phys(struct device *dev, dma_addr = phys;if (unlikely (!dma_capable (dev, dma_addr, size, false))) goto err_overflow; + }else if (attrs &DMA_ATTR_CC_DECRYPTED ) { + dma_addr =phys_to_dma_unencrypted (dev, phys); +if (unlikely (!dma_capable (dev, dma_addr, size, false))) + goto err_overflow; }else { - dma_addr =phys_to_dma (dev, phys); + dma_addr =phys_to_dma_encrypted (dev, phys);if (unlikely (!dma_capable (dev, dma_addr, size, true)) ||dma_kmalloc_needs_bounce (dev, size, dir)) { if (is_swiotlb_active(dev) &&
kernel/dma/pool.c +98 -56
253 lines not on this page · raw message
kernel/dma/swiotlb.c +49 -19
185 lines not on this page · raw message
Re: [RFC PATCH v3 4/5] dma-mapping: Encapsulate memory state during allocation
In reply to Jason GunthorpeQuoted 29 lines · Jason Gunthorpe
Quoted 18 lines · Mostafa Saleh
Introduce a new dma-direct internal type
dma_pagewhich is "struct page" and a bit indicate whether the memory has been decrypted or not.
This is useful to pass such information encapsulated through allocation functions, which is currently set fromswiotlb_alloc().No functional changes.
Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 12 deletions(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c line 21
Quoted 6 lines
kernel/dma/direct.c +3
-0@@ -16,6 +16,33 @@#include <linux/pci-p2pdma.h> #include "direct.h" +/* +* Represent DMA allocation and 1 bit flag for it's state +*/ I'd explain this wrappers a pointer and uses the low
PAGE_SHIFTbits for flags..
Quoted 4 lines
kernel/dma/direct.c line 23
Quoted 2 lines
kernel/dma/direct.c +2
-0+struct dma_page { +unsigned long val;
unintptr_t?
I thought about that, but I don’t see unintptr_t anywhere in the kernel, it seems similar cases use “unsigned long” as in xarray.h
Quoted 18 lines
Quoted 18 lines
kernel/dma/direct.c +4
-3@@ -103,20 +130,21 @@ static void __dma_direct_free_pages(struct device *dev, struct page *page,dma_free_contiguous (dev, page, size); } -static struct page *dma_direct_alloc_swiotlb (struct device *dev,size_t size) +static struct dma_page dma_direct_alloc_swiotlb (struct device *dev,size_t size) { -struct page *page =swiotlb_alloc (dev, size,NULL ); +enum swiotlb_page_state state; +struct page *page =swiotlb_alloc (dev, size, &state);if (page && !dma_coherent_ok (dev,page_to_phys (page), size)) {swiotlb_free (dev, page, size); -return NULL ; +return DMA_PAGE_NULL ; }
Quoted 5 lines
kernel/dma/direct.c static void __dma_direct_free_pages(struct device *dev, struct page *page, · line 143
Quoted 2 lines
kernel/dma/direct.c +1
-1-return page; +return page_to_dma_page (page, state ==SWIOTLB_PAGE_DECRYPTED );Should the struct
dma_pagehave been introduced earlier instead of theswiotlb_page_state? Seems a bit odd to have both
It can be introduced earlier, but It looked cleaner to decouple swiotlb from direct-dma and keep the dma_page type completely internal.
Quoted 20 lines
If these are actually internally allocated struct pages, could you use the struct page memory itself to record the decrypted state? That would require more significant changes to the allocator calls.
kernel/dma/direct.c static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, · line 221
Quoted 12 lines
kernel/dma/direct.c +3
-1@@ -184,9 +212,11 @@ static void *dma_direct_alloc_from_pool(struct device *dev, size_t size,static void *dma_direct_alloc_no_mapping (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp) { +struct dma_page dma_page;struct page *page; - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + dma_page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =dma_page_to_page (dma_page);if (!page)return NULL ;I would expect to see more usage of the
dma_pagehere..
Quoted 8 lines
Like I don't think this is really right:
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));Does
page_to_phys(page)really work on decrypted memory? On CCA it will return the protected alias which doesn't seem like something useful?
Not sure, but that’s not related to this patch, that’s already the status quo, I can look more into it.
Quoted 24 lines
static inline
dma_addr_tphys_to_dma_direct(struct device *dev,
phys_addr_tphys){ if (force_dma_unencrypted(dev)) return phys_to_dma_unencrypted(dev, phys); return phys_to_dma(dev, phys);Above is all nonsense now that you have a direct indication of the address is decrypted memory or not, it should also be used right here directly.
if (
is_dma_page_decrypted(dma_page))
*dma_handle=phys_to_dma_unencrypted(..)
else
*dma_handle=phys_to_dma(..);The later patch just makes it worse by adding even more confusing flags to
phys_to_dma_direct().I think it should work out that everyone already knows what memory type they are working with before they call down to
phys_to_dma_direct()- the calls toforce_dma_unecrypted()here are just hacks because it previously did not.
Quoted 4 lines
Anyhow, I think this series is alot better than the previous one. If you work a little harder to make it so there is only one
force_dma_unecrypted()per high level DMA API call that would be perfect.
I see, that makes a lot of sense.
Thanks,
Mostafa
Quoted 2 lines
Jason
Re: [RFC PATCH v3 4/5] dma-mapping: Encapsulate memory state during allocation
Quoted 30 lines · Mostafa Saleh
Quoted 29 lines · Jason Gunthorpe
Quoted 18 lines · Mostafa Saleh
Introduce a new dma-direct internal type
dma_pagewhich is "struct page" and a bit indicate whether the memory has been decrypted or not.
This is useful to pass such information encapsulated through allocation functions, which is currently set fromswiotlb_alloc().No functional changes.
Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 12 deletions(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c line 21
Quoted 6 lines
kernel/dma/direct.c +3
-0@@ -16,6 +16,33 @@#include <linux/pci-p2pdma.h> #include "direct.h" +/* +* Represent DMA allocation and 1 bit flag for it's state +*/ I'd explain this wrappers a pointer and uses the low
PAGE_SHIFTbits for flags..
Quoted 7 lines
Quoted 4 lines
kernel/dma/direct.c line 23
Quoted 2 lines
kernel/dma/direct.c +2
-0+struct dma_page { +unsigned long val;
unintptr_t?I thought about that, but I don’t see
unintptr_tanywhere in the kernel, it seems similar cases use “unsigned long” as inxarray.h
Jason meant uintptr_t. However, we have a similar pattern with struct encoded_page as an incomplete type and encode_page() adds the bits directly to the pointer. I'd use something similar as there's precedent already.
Signature · 1 line
Catalin
Re: [RFC PATCH v3 4/5] dma-mapping: Encapsulate memory state during allocation
Quoted 36 lines · Catalin Marinas
Quoted 35 lines · Mostafa Saleh
Quoted 33 lines · Jason Gunthorpe
Quoted 18 lines · Mostafa Saleh
Introduce a new dma-direct internal type
dma_pagewhich is "struct page" and a bit indicate whether the memory has been decrypted or not.
This is useful to pass such information encapsulated through allocation functions, which is currently set fromswiotlb_alloc().No functional changes.
Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 12 deletions(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c line 21
Quoted 6 lines
kernel/dma/direct.c +3
-0@@ -16,6 +16,33 @@#include <linux/pci-p2pdma.h> #include "direct.h" +/* +* Represent DMA allocation and 1 bit flag for it's state +*/ I'd explain this wrappers a pointer and uses the low
PAGE_SHIFTbits for flags..kernel/dma/direct.c line 23
Quoted 2 lines
kernel/dma/direct.c +2
-0+struct dma_page { +unsigned long val;
unintptr_t?
Quoted 7 lines
Quoted 2 lines
I thought about that, but I don’t see
unintptr_tanywhere in the kernel, it seems similar cases use “unsigned long” as inxarray.hJason meant
uintptr_t. However, we have a similar pattern with structencoded_pageas an incomplete type andencode_page()adds the bits directly to the pointer. I'd use something similar as there's precedent already.
I see, makes sense, I will do that.
Thanks,
Mostafa
Quoted 3 lines
Signature · 2 lines
-- Catalin
RFC PATCH v3 5/5 dma-mapping: Fix memory decryption issues
In reply to Mostafa SalehFix 2 existing issues:
1) In case a device have a restricted DMA pool, memory will be decrypted (which is now returned in the state from swiotlb_alloc().
Later the main function will attempt to decrypt the memory if force_dma_unencrypted() is true.
Which results in the memory being decrypted twice. Change that to only encrypt/decrypt memory that is not already decrypted as indicated in the new dma_page struct.
2) Using phys_to_dma_unencrypted() is not enlighted about already decrypted memory and will use the wrong functions for that.
--- kernel/dma/direct.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
kernel/dma/direct.c +28 -13
@@ -43,6 +43,11 @@ static inline struct page *dma_page_to_page(struct dma_page dma_page)return (struct page *)(dma_page.val & ~DMA_PAGE_DECRYPTED_FLAG ); } +static inline bool is_dma_page_decrypted (struct dma_page dma_page) +{ +return dma_page.val &DMA_PAGE_DECRYPTED_FLAG ; +} +/* * Most architectures use ZONE_DMA for the first 16 Megabytes, but some use * it for entirely different regions. In that case the arch code needs to
@@ -51,9 +56,9 @@ static inline struct page *dma_page_to_page(struct dma_page dma_page)u64 zone_dma_limit__ro_after_init =DMA_BIT_MASK (24 );static inline dma_addr_t phys_to_dma_direct (struct device *dev, -phys_addr_t phys) +phys_addr_t phys,bool already_decrypted) { -if (force_dma_unencrypted (dev)) +if (already_decrypted ||force_dma_unencrypted (dev))return phys_to_dma_unencrypted (dev, phys);return phys_to_dma (dev, phys); }
@@ -67,7 +72,7 @@ static inline struct page *dma_direct_to_page(struct device *dev,u64 dma_direct_get_required_mask (struct device *dev) {phys_addr_t phys = (phys_addr_t)(max_pfn -1 ) <<PAGE_SHIFT ; -u64 max_dma =phys_to_dma_direct (dev, phys); +u64 max_dma =phys_to_dma_direct (dev, phys, false);return (1ULL << (fls64 (max_dma) -1 )) *2 -1 ; }
@@ -96,7 +101,7 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)bool dma_coherent_ok (struct device *dev,phys_addr_t phys,size_t size) { -dma_addr_t dma_addr =phys_to_dma_direct (dev, phys); +dma_addr_t dma_addr =phys_to_dma_direct (dev, phys, false);if (dma_addr ==DMA_MAPPING_ERROR )return false;
@@ -122,11 +127,14 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size)static void __dma_direct_free_pages (struct device *dev,struct page *page,size_t size,bool encrypt) { -if (encrypt &&dma_set_encrypted (dev,page_address (page), size)) +bool keep_encrypted =swiotlb_is_decrypted (dev, page, size); + +if (!keep_encrypted && encrypt &&dma_set_encrypted (dev,page_address (page), size))return ;if (swiotlb_free (dev, page, size))return ; +dma_free_contiguous (dev, page, size); }
@@ -205,7 +213,7 @@ static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, page =dma_alloc_from_pool (dev, size, &ret, gfp, dma_coherent_ok);if (!page)return NULL ; - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), false);return ret; }
@@ -225,7 +233,8 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,arch_dma_prep_coherent (page, size);/* return the page pointer as the opaque cookie */ - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), +is_dma_page_decrypted (dma_page));return page; }
@@ -234,6 +243,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, {bool remap = false, set_uncached = false, decrypt =force_dma_unencrypted (dev);struct dma_page dma_page; +bool already_decrypted;struct page *page;void *ret;
@@ -289,6 +299,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,if (!page)return NULL ; + already_decrypted =is_dma_page_decrypted (dma_page);/* * dma_alloc_contiguous can return highmem pages depending on a * combination the cma= arguments and per-arch setup. These need to be
@@ -299,12 +310,13 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } -if (decrypt &&dma_set_decrypted (dev,page_address (page), size)) +if (!already_decrypted && decrypt && +dma_set_decrypted (dev,page_address (page), size)) goto out_leak_pages;if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs); -if (decrypt) +if (decrypt || already_decrypted) prot =pgprot_decrypted (prot);/* remove any dirty cache lines on the kernel alias */
@@ -328,11 +340,11 @@ void *dma_direct_alloc(struct device *dev, size_t size, goto out_encrypt_pages; } - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), already_decrypted);return ret; out_encrypt_pages: -__dma_direct_free_pages (dev, page, size, decrypt); +__dma_direct_free_pages (dev, page, size, decrypt && !already_decrypted);return NULL ; out_leak_pages:return NULL ;
@@ -385,6 +397,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp) { struct dma_page dma_page; +bool already_decrypted;struct page *page;void *ret;
@@ -396,11 +409,13 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,if (!page)return NULL ; + already_decrypted =is_dma_page_decrypted (dma_page); ret =page_address (page); -if (force_dma_unencrypted (dev) &&dma_set_decrypted (dev, ret, size)) +if (!already_decrypted &&force_dma_unencrypted (dev) && +dma_set_decrypted (dev, ret, size)) goto out_leak_pages;memset (ret,0 , size); - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), already_decrypted);return page; out_leak_pages:return NULL ;
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Re: [RFC PATCH v3 5/5] dma-mapping: Fix memory decryption issues
Quoted 11 lines · Mostafa Saleh
Fix 2 existing issues:
1) In case a device have a restricted DMA pool, memory will be decrypted (which is now returned in the state fromswiotlb_alloc().Later the main function will attempt to decrypt the memory if
force_dma_unencrypted()is true.Which results in the memory being decrypted twice. Change that to only encrypt/decrypt memory that is not already decrypted as indicated in the new
dma_pagestruct.
As discussed in the v2 email thread, we should address this by deciding whether, for now, we treat all allocations from swiotlb pools as unencrypted
https://lore.kernel.org/all/[email protected]
Quoted 4 lines
2) Using
phys_to_dma_unencrypted()is not enlighted about already decrypted memory and will use the wrong functions for that.
Can you split this into a separate patch? I’m finding it difficult to understand what the issue is here. Adding the unencrypted flag multiple times to an address is not a problem in itself. Even so, I still do not follow when we would end up doing that.
At the interface level:phys_to_dma should always return an encrypted address. phys_to_dma_unencrypted should always return an unencrypted address. phys_to_dma_direct should depend on the device state.
Quoted 3 lines
Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>
-aneesh
Re: [RFC PATCH v3 5/5] dma-mapping: Fix memory decryption issues
Quoted 7 lines · Aneesh Kumar K.V
Quoted 2 lines
2) Using
phys_to_dma_unencrypted()is not enlighted about already decrypted memory and will use the wrong functions for that.Can you split this into a separate patch? I’m finding it difficult to understand what the issue is here. Adding the unencrypted flag multiple times to an address is not a problem in itself. Even so, I still do not follow when we would end up doing that.
I think my comments show how to address it right..
Quoted 1 line
phys_to_dma_directshould depend on the device state.
No, it depends on what state the CPU address is, which in some flows would have depended on the device state, but by the time you get to generating a dma_addr_t it should be based 100% on the current state of the phys_addr and nothing else.
Assuming that a T=0 device must be presented unencrypted memory is an easy hack but it doesn't work when we get to T=1 devices that can handle both encryped and decrypted memory. Then we need to track it explicitly.
The only places we we should check the device state for T=0 is at the very top when we decide if we force it to swiotlb and inside swiotlb when we decide if the allocation should be decrypted. Everything else should flow from tracking the phy's state, and be tied into the new DMA ATTR UNENCRYPTED.
Jason
Re: [RFC PATCH v3 5/5] dma-mapping: Fix memory decryption issues
Quoted 23 lines · Jason Gunthorpe
Quoted 7 lines · Aneesh Kumar K.V
Quoted 2 lines
2) Using
phys_to_dma_unencrypted()is not enlighted about already decrypted memory and will use the wrong functions for that.Can you split this into a separate patch? I’m finding it difficult to understand what the issue is here. Adding the unencrypted flag multiple times to an address is not a problem in itself. Even so, I still do not follow when we would end up doing that.
I think my comments show how to address it right..
Quoted 1 line
phys_to_dma_directshould depend on the device state.No, it depends on what state the CPU address is, which in some flows would have depended on the device state, but by the time you get to generating a
dma_addr_tit should be based 100% on the current state of thephys_addrand nothing else.Assuming that a T=0 device must be presented unencrypted memory is an easy hack but it doesn't work when we get to T=1 devices that can handle both encryped and decrypted memory. Then we need to track it explicitly.
Quoted 6 lines
The only places we we should check the device state for T=0 is at the very top when we decide if we force it to swiotlb and inside swiotlb when we decide if the allocation should be decrypted. Everything else should flow from tracking the phy's state, and be tied into the new DMA ATTR UNENCRYPTED.
For things like
#define dma_map_single(d, a, s, r) dma_map_single_attrs(d, a, s, r, 0)
Where do you suggest DMA_ATTR_CC_DECRYPTED be set?
Right now i have it around
static inline dma_addr_t dma_direct_map_phys(struct device *dev,
phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs)
{
dma_addr_t dma_addr;
if (force_dma_unencrypted(dev))
attrs |= DMA_ATTR_CC_DECRYPTED;I am wondering whether we should do earlier. But we have only audited dma-direct for memory encryption.
-aneesh
Re: [RFC PATCH v3 5/5] dma-mapping: Fix memory decryption issues
Quoted 32 lines · Aneesh Kumar K.V
Quoted 23 lines · Jason Gunthorpe
Quoted 7 lines · Aneesh Kumar K.V
Quoted 2 lines
2) Using
phys_to_dma_unencrypted()is not enlighted about already decrypted memory and will use the wrong functions for that.Can you split this into a separate patch? I’m finding it difficult to understand what the issue is here. Adding the unencrypted flag multiple times to an address is not a problem in itself. Even so, I still do not follow when we would end up doing that.
I think my comments show how to address it right..
Quoted 1 line
phys_to_dma_directshould depend on the device state.No, it depends on what state the CPU address is, which in some flows would have depended on the device state, but by the time you get to generating a
dma_addr_tit should be based 100% on the current state of thephys_addrand nothing else.Assuming that a T=0 device must be presented unencrypted memory is an easy hack but it doesn't work when we get to T=1 devices that can handle both encryped and decrypted memory. Then we need to track it explicitly.
Quoted 6 lines
The only places we we should check the device state for T=0 is at the very top when we decide if we force it to swiotlb and inside swiotlb when we decide if the allocation should be decrypted. Everything else should flow from tracking the phy's state, and be tied into the new DMA ATTR UNENCRYPTED.
Quoted 5 lines
For things like
#define
dma_map_single(d, a, s, r)dma_map_single_attrs(d, a, s, r, 0)Where do you suggest
DMA_ATTR_CC_DECRYPTEDbe set?
dma_map_single() assumes that a is encrypted.
If the caller passes an a that it decrypted then it must pass DMA_ATTR_CC_DECRYPTED.
It is NOT directly derived from force_dma_unencrypted().
If attr says encrypted and force_dma_unencrypted(), then we have to do swiotlb, we get a new address and we track the decrypted state of the new address along with it.
Lower levels always receive an address and a 'is decrypted' flag to make their decisions.
The place we check force_dma_unencrypted() is while branching to swiotlb.
swiotlb might re-use DMA_ATTR_CC_DECRYPTED, or it might use the dma_page idea, but logically the address and a matching flag flow together through the call chains.
Jason
Re: [RFC PATCH v3 5/5] dma-mapping: Fix memory decryption issues
In reply to Mostafa SalehQuoted 38 lines · Mostafa Saleh
Fix 2 existing issues:
1) In case a device have a restricted DMA pool, memory will be decrypted (which is now returned in the state fromswiotlb_alloc().Later the main function will attempt to decrypt the memory if
force_dma_unencrypted()is true.Which results in the memory being decrypted twice. Change that to only encrypt/decrypt memory that is not already decrypted as indicated in the new
dma_pagestruct.2) Using
phys_to_dma_unencrypted()is not enlighted about already decrypted memory and will use the wrong functions for that.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)kernel/dma/direct.c +5
-0@@ -43,6 +43,11 @@ static inline struct page *dma_page_to_page(struct dma_page dma_page)return (struct page *)(dma_page.val & ~DMA_PAGE_DECRYPTED_FLAG ); } +static inline bool is_dma_page_decrypted (struct dma_page dma_page) +{ +return dma_page.val &DMA_PAGE_DECRYPTED_FLAG ; +} +/* * Most architectures use ZONE_DMA for the first 16 Megabytes, but some use * it for entirely different regions. In that case the arch code needs to @@ -51,9 +56,9 @@ static inline struct page *dma_page_to_page(struct dma_page dma_page)u64 zone_dma_limit__ro_after_init =DMA_BIT_MASK (24 );
kernel/dma/direct.c static inline struct page *dma_page_to_page(struct dma_page dma_page) · line 64
Quoted 10 lines
kernel/dma/direct.c +2
-2static inline dma_addr_t phys_to_dma_direct (struct device *dev, -phys_addr_t phys) +phys_addr_t phys,bool already_decrypted) { -if (force_dma_unencrypted (dev)) +if (already_decrypted ||force_dma_unencrypted (dev))return phys_to_dma_unencrypted (dev, phys);return phys_to_dma (dev, phys);
Can you explain what is covered by the if () above? Do we expect to return an unencrypted DMA address even for devices for which force_dma_unencrypted(dev) does not return true?
Quoted 55 lines
}
kernel/dma/direct.c +9
-5@@ -67,7 +72,7 @@ static inline struct page *dma_direct_to_page(struct device *dev,u64 dma_direct_get_required_mask (struct device *dev) {phys_addr_t phys = (phys_addr_t)(max_pfn -1 ) <<PAGE_SHIFT ; -u64 max_dma =phys_to_dma_direct (dev, phys); +u64 max_dma =phys_to_dma_direct (dev, phys, false);return (1ULL << (fls64 (max_dma) -1 )) *2 -1 ; }@@ -96,7 +101,7 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)bool dma_coherent_ok (struct device *dev,phys_addr_t phys,size_t size) { -dma_addr_t dma_addr =phys_to_dma_direct (dev, phys); +dma_addr_t dma_addr =phys_to_dma_direct (dev, phys, false);if (dma_addr ==DMA_MAPPING_ERROR )return false;@@ -122,11 +127,14 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size)static void __dma_direct_free_pages (struct device *dev,struct page *page,size_t size,bool encrypt) { -if (encrypt &&dma_set_encrypted (dev,page_address (page), size)) +bool keep_encrypted =swiotlb_is_decrypted (dev, page, size); + +if (!keep_encrypted && encrypt &&dma_set_encrypted (dev,page_address (page), size))return ;if (swiotlb_free (dev, page, size))return ; +dma_free_contiguous (dev, page, size); }@@ -205,7 +213,7 @@ static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, page =dma_alloc_from_pool (dev, size, &ret, gfp, dma_coherent_ok);if (!page)return NULL ; - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), false);return ret; }@@ -225,7 +233,8 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,arch_dma_prep_coherent (page, size);/* return the page pointer as the opaque cookie */ - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), +is_dma_page_decrypted (dma_page));return page; }
kernel/dma/direct.c void *dma_direct_alloc(struct device *dev, size_t size, · line 249
Quoted 7 lines
kernel/dma/direct.c +1
-0@@ -234,6 +243,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, {bool remap = false, set_uncached = false, decrypt =force_dma_unencrypted (dev);struct dma_page dma_page; +bool already_decrypted;struct page *page;void *ret;
I am wondering wether we can avoid the already_decrypted logic if you pull dma_direct_alloc_swiotlb() out to the caller.
https://lore.kernel.org/all/[email protected]
Quoted 65 lines
kernel/dma/direct.c +11
-6@@ -289,6 +299,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,if (!page)return NULL ; + already_decrypted =is_dma_page_decrypted (dma_page);/* * dma_alloc_contiguous can return highmem pages depending on a * combination the cma= arguments and per-arch setup. These need to be @@ -299,12 +310,13 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } -if (decrypt &&dma_set_decrypted (dev,page_address (page), size)) +if (!already_decrypted && decrypt && +dma_set_decrypted (dev,page_address (page), size)) goto out_leak_pages;if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs); -if (decrypt) +if (decrypt || already_decrypted) prot =pgprot_decrypted (prot);/* remove any dirty cache lines on the kernel alias */ @@ -328,11 +340,11 @@ void *dma_direct_alloc(struct device *dev, size_t size, goto out_encrypt_pages; } - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), already_decrypted);return ret; out_encrypt_pages: -__dma_direct_free_pages (dev, page, size, decrypt); +__dma_direct_free_pages (dev, page, size, decrypt && !already_decrypted);return NULL ; out_leak_pages:return NULL ;@@ -385,6 +397,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp) { struct dma_page dma_page; +bool already_decrypted;struct page *page;void *ret;@@ -396,11 +409,13 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,if (!page)return NULL ; + already_decrypted =is_dma_page_decrypted (dma_page); ret =page_address (page); -if (force_dma_unencrypted (dev) &&dma_set_decrypted (dev, ret, size)) +if (!already_decrypted &&force_dma_unencrypted (dev) && +dma_set_decrypted (dev, ret, size)) goto out_leak_pages;memset (ret,0 , size); - *dma_handle =phys_to_dma_direct (dev,page_to_phys (page)); + *dma_handle =phys_to_dma_direct (dev,page_to_phys (page), already_decrypted);return page; out_leak_pages:return NULL ;Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Quoted 5 lines · Mostafa Saleh
Introduction
============
This is the third version of the fixes for direct-dma dealing with memory encryption and restricted-dma.
Quoted 7 lines
Changes in v3:
- Instead of extending the logic by usingis_swiotlb_for_alloc(), follow Jason’s suggestion and propagate the state of the memory allocated.
- Remove checks out of dma_set_*() based on Jason suggestion
- Remove documentation for now until we are close to the final proposal and add it later if needed.
There are a number of Sashiko remarks that look plausible that should be investigated:
https://sashiko.dev/#/patchset/20260408194750.2280873-1-smostafa%40google.com
Quoted 10 lines
Design
======
This series focuses mainly on dma-direct interaction with memory encryption which is the complicated case.
At the moment memory encryption and dma-direct interacts in 2 ways:
1)force_dma_direct(): if true, memory will be decrypted by default on allocation.
2) Restricted DMA: where memory is pre-decrypted and managed by SWIOTLB.
Quoted 2 lines
With a third possible usage on the way [1] where the DMA-API allows an attr for decrypted memory.
This [1] was merged now
Jason
Quoted 14 lines · Jason Gunthorpe
Quoted 5 lines · Mostafa Saleh
Introduction
============
This is the third version of the fixes for direct-dma dealing with memory encryption and restricted-dma.Quoted 7 lines
Changes in v3:
- Instead of extending the logic by usingis_swiotlb_for_alloc(), follow Jason’s suggestion and propagate the state of the memory allocated.
- Remove checks out of dma_set_*() based on Jason suggestion
- Remove documentation for now until we are close to the final proposal and add it later if needed.
Quoted 3 lines
There are a number of Sashiko remarks that look plausible that should be investigated:
https://sashiko.dev/#/patchset/20260408194750.2280873-1-smostafa%40google.com
I think the remap and NULL points are valid, I will address them.
The case of dma_coherent_ok() is more tricky, it is not a regression, but I think it’s still a theoretical problem for some CCA solutions where encrypted/decrypted memory have different DMA aliases. It’s not easy to fix it without having some helper to check the memory state as force_dma_unencrypted and swiotlb_is_decrypted() which kind of defeat the purpose of returning the memory state from swiotlb_alloc() ://
Quoted 11 lines
Quoted 11 lines
Design
======
This series focuses mainly on dma-direct interaction with memory encryption which is the complicated case.
At the moment memory encryption and dma-direct interacts in 2 ways:
1)force_dma_direct(): if true, memory will be decrypted by default on allocation.
2) Restricted DMA: where memory is pre-decrypted and managed by SWIOTLB.
Quoted 4 lines
Quoted 2 lines
With a third possible usage on the way [1] where the DMA-API allows an attr for decrypted memory.
This [1] was merged now
I see, I will rebase on top of it and send v4.
Thanks,
Mostafa
Quoted 2 lines
Jason