dma-mapping: Fixes for memory encryption
Subsystems (2)
v2 cover dma-mapping: Fixes for memory encryption
Introduction
============
This is the second version of the fixes for direct-dma dealing with memory encryption and restricted-dma.
In this version, one more fix is included and I go a step further and attempt to clean up the code to consolidate it and make it less error prone.
Especially with more users coming, such as using decrypted dma-bufs [1] which also interacts with dma-direct.
Lastly I added a new documentation for memory encryption based on my conclusion (Documentation/core-api/dma-direct-memory-encryption.rst)
Background
==========
At the moment the following hypervisor guests will need to deal with memory encryption:
- pKVM (ARM): Documentation/virt/kvm/arm/hypercalls.rst
- ARM CCA: Documentation/arch/arm64/arm-cca.rst
- Intel TDX: Documentation/arch/x86/tdx.rst
- AMD SEV: Documentation/arch/x86/amd-memory-encryption.rst
- PPC SVM: Documentation/arch/powerpc/ultravisor.rst
- Hyper-V: Documentation/virt/hyperv/coco.rst
AFAICT, all (confidential) guests running under those have the memory encrypted by default and guests will then explicitly share the memory back if needed.
The main use cases for decrypting(sharing) memory are:
- Sharing memory back to the host through SWIOTLB (for virtio...)
- Hypervisor specific communication (ex: snp_msg, GHCB, VMBUS...)
- Shared/emulated resources: VGARAM (x86-SEV), GIC ITS tables (arm64)
While encrypting memory is typically used for reverting the set_memory_decrypted() either in error handling or in freeing shared resources back to the kernel.
Design
======
This series focuses mainly on dma-direct interaction with memory encryption which is the complicated case.
At the moment memory encryption and dma-direct interacts in 2 ways:
1) force_dma_direct(): if true, memory will be decrypted by default on allocation.
2) Restricted DMA: where memory is pre-decrypted and managed by SWIOTLB.
With a third possible usage on the way [1] where the DMA-API allows an attr for decrypted memory.
Instead of open coding many checks with is_swiotlb_for_alloc() and force_dma_unencrypted() which doesn’t have the same exact semantics, add some helpers to abstract that logic into the code:
* dma_external_decryption(dev): Returns true if the pages are decrypted but managed externally.
* dma_owns_decryption(dev): Returns true if the pages need to be explicitly decrypted and managed by the dma-direct layer (as the architecture forces unencrypted DMA).
* is_dma_decrypted(dev): Returns true if the memory being used is in a decrypted state, regardless of who manages it.
Testing
=======
I was able to test this only under pKVM (arm64) as I have no access to other systems.
Future work
===========
Two other things I am also looking at which are related to restricted DMA pools, so they should be a different series.
1) Private pools: Currently all restricted DMA pools are decrypted (shared) by default. Having private pools would be useful for device assignment when bouncing is needed (as for non-coherent devices)
2) Optimizations for memory sharing. In some cases, allocations from restricted dma-pools are page aligned. For CoCo cases, that means that it will be cheaper to share memory in-place instead of bouncing.
Both of these add new semantics which need to be done carefully to avoid regressions, and might be a good candidate for a topic in the next LPC.
Patches
=======
- 1-3 Fixes
- 4 Refactoring
- 5 Documentation
v1: https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
Mostafa Saleh (5):
dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
dma-mapping: Decrypt memory on remap
dma-mapping: Refactor memory encryption usage
dma-mapping: Add doc for memory encryption
.../core-api/dma-direct-memory-encryption.rst | 77 +++++++++++++++++++ kernel/dma/direct.c | 51 ++++++++---- 2 files changed, 114 insertions(+), 14 deletions(-) create mode 100644 Documentation/core-api/dma-direct-memory-encryption.rst
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
RFC PATCH v2 1/5 dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
In case a device have a restricted DMA pool, it will be decrypted by default.
However, in the path of dma_direct_alloc() memory can be allocated from this pool using, __dma_direct_alloc_pages() => dma_direct_alloc_swiotlb()
After that from the same function, it will attempt to decrypt it using dma_set_decrypted() if force_dma_unencrypted().
Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- CCA: Clear a bit in the page table and call realm IPA_STATE_SET.
- TDX: Issue a hypercall.
- pKVM: Which doesn't implement force_dma_unencrypted() at the moment, uses a share hypercall.
Change that to only encrypt/decrypt memory that are not allocated from the restricted dma pools.
--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/dma/direct.c +2 -2
@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;return set_memory_decrypted ((unsigned long )vaddr,PFN_UP (size)); }
@@ -88,7 +88,7 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) {int ret; -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ; ret =set_memory_encrypted ((unsigned long )vaddr,PFN_UP (size));if (ret)
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 33 lines · Mostafa Saleh
In case a device have a restricted DMA pool, it will be decrypted by default.
However, in the path of
dma_direct_alloc()memory can be allocated from this pool using,__dma_direct_alloc_pages()=>dma_direct_alloc_swiotlb()After that from the same function, it will attempt to decrypt it using
dma_set_decrypted()ifforce_dma_unencrypted().Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- CCA: Clear a bit in the page table and call realmIPA_STATE_SET.
- TDX: Issue a hypercall.
- pKVM: Which doesn't implementforce_dma_unencrypted()at the moment, uses a share hypercall.Change that to only encrypt/decrypt memory that are not allocated from the restricted dma pools.
Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) · line 83
Quoted 6 lines
kernel/dma/direct.c +1
-1@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;
This seems really obtuse, I would expect the decryption state of the memory to be known by the caller. If dma_direct_alloc_swiotlb() can return decrypted or encrypted memory it needs to return a flag saying that. It shouldn't be deduced by checking dev flags in random places like this.
Double decryption is certainly a bug, I do not expect that to work.
Jason
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 35 lines · Jason Gunthorpe
Quoted 34 lines · Mostafa Saleh
In case a device have a restricted DMA pool, it will be decrypted by default.
However, in the path of
dma_direct_alloc()memory can be allocated from this pool using,__dma_direct_alloc_pages()=>dma_direct_alloc_swiotlb()After that from the same function, it will attempt to decrypt it using
dma_set_decrypted()ifforce_dma_unencrypted().Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- CCA: Clear a bit in the page table and call realmIPA_STATE_SET.
- TDX: Issue a hypercall.
- pKVM: Which doesn't implementforce_dma_unencrypted()at the moment, uses a share hypercall.Change that to only encrypt/decrypt memory that are not allocated from the restricted dma pools.
Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/dma/direct.c +0
-0@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
Quoted 11 lines
kernel/dma/direct.c bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) · line 83
Quoted 5 lines
kernel/dma/direct.c +1
-1static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;This seems really obtuse, I would expect the decryption state of the memory to be known by the caller. If
dma_direct_alloc_swiotlb()can return decrypted or encrypted memory it needs to return a flag saying that. It shouldn't be deduced by checking dev flags in random places like this.
At the moment restricted dma is always decrypted, also it’s per device so we don’t have to check this per allocation.
I can change the signature for __dma_direct_alloc_pages() to make it return an extra flag but that feels more complicated as it changes dma_direct_alloc_swiotlb() , swiotlb_alloc() with its callers.
I can investigate this approach further.
Thanks,
Mostafa
Quoted 4 lines
Double decryption is certainly a bug, I do not expect that to work.
Jason
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 48 lines · Mostafa Saleh
Quoted 35 lines · Jason Gunthorpe
Quoted 34 lines · Mostafa Saleh
In case a device have a restricted DMA pool, it will be decrypted by default.
However, in the path of
dma_direct_alloc()memory can be allocated from this pool using,__dma_direct_alloc_pages()=>dma_direct_alloc_swiotlb()After that from the same function, it will attempt to decrypt it using
dma_set_decrypted()ifforce_dma_unencrypted().Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- CCA: Clear a bit in the page table and call realmIPA_STATE_SET.
- TDX: Issue a hypercall.
- pKVM: Which doesn't implementforce_dma_unencrypted()at the moment, uses a share hypercall.Change that to only encrypt/decrypt memory that are not allocated from the restricted dma pools.
Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/dma/direct.c +0
-0@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)Quoted 11 lines
kernel/dma/direct.c bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) · line 83
Quoted 5 lines
kernel/dma/direct.c +1
-1static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;This seems really obtuse, I would expect the decryption state of the memory to be known by the caller. If
dma_direct_alloc_swiotlb()can return decrypted or encrypted memory it needs to return a flag saying that. It shouldn't be deduced by checking dev flags in random places like this.
Quoted 2 lines
At the moment restricted dma is always decrypted, also it’s per device so we don’t have to check this per allocation.
Doesn't the initial state depend on platform ? For CCA, the Realm must
decide how it wants to use a given region, which for the restricted DMA pool, it can be made decrypted. Could the VM OS decide to make this decrypted at boot ?
Suzuki
Quoted 13 lines
I can change the signature for
__dma_direct_alloc_pages()to make it return an extra flag but that feels more complicated as it changesdma_direct_alloc_swiotlb(),swiotlb_alloc()with its callers.I can investigate this approach further.
Thanks,
MostafaQuoted 4 lines
Double decryption is certainly a bug, I do not expect that to work.
Jason
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 48 lines · Suzuki K Poulose
Quoted 47 lines · Mostafa Saleh
Quoted 45 lines · Jason Gunthorpe
Quoted 33 lines · Mostafa Saleh
In case a device have a restricted DMA pool, it will be decrypted by default.
However, in the path of
dma_direct_alloc()memory can be allocated from this pool using,__dma_direct_alloc_pages()=>dma_direct_alloc_swiotlb()After that from the same function, it will attempt to decrypt it using
dma_set_decrypted()ifforce_dma_unencrypted().Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- CCA: Clear a bit in the page table and call realmIPA_STATE_SET.
- TDX: Issue a hypercall.
- pKVM: Which doesn't implementforce_dma_unencrypted()at the moment, uses a share hypercall.Change that to only encrypt/decrypt memory that are not allocated from the restricted dma pools.
Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) · line 82
Quoted 5 lines
kernel/dma/direct.c +1
-1@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;This seems really obtuse, I would expect the decryption state of the memory to be known by the caller. If
dma_direct_alloc_swiotlb()can return decrypted or encrypted memory it needs to return a flag saying that. It shouldn't be deduced by checking dev flags in random places like this.
Quoted 8 lines
Quoted 2 lines
At the moment restricted dma is always decrypted, also it’s per device so we don’t have to check this per allocation.
Doesn't the initial state depend on platform ? For CCA, the Realm must decide how it wants to use a given region, which for the restricted DMA pool, it can be made decrypted. Could the VM OS decide to make this decrypted at boot ?
At the moment no [1], the pool is decrypted unconditionally. As mentioned in the cover letter under "Future work", I believe giving the OS the ability to have undecrypted pools is important for confidential DMA.
Initially, I thought that can be a per device property (so the platform will keep the memory encrypted for physical devices and decrypt it for emulated ones).
But, that might cause runtime issues as a pool can be shared between multiple devices. So, I beleive it's better to have this as per-pool property(from device-tree for ex) and then the platform can do any validation of assumptions it needs in the runtime.
This is a bit hairy, as I mentioned it would be a good topic to discuss in the next LPC.
Thanks,
Mostafa
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/dma/swiotlb.c#n1847
Quoted 17 lines
Suzuki
Quoted 13 lines
I can change the signature for
__dma_direct_alloc_pages()to make it return an extra flag but that feels more complicated as it changesdma_direct_alloc_swiotlb(),swiotlb_alloc()with its callers.I can investigate this approach further.
Thanks,
MostafaQuoted 4 lines
Double decryption is certainly a bug, I do not expect that to work.
Jason
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
In reply to Mostafa SalehQuoted 48 lines · Mostafa Saleh
Quoted 35 lines · Jason Gunthorpe
Quoted 34 lines · Mostafa Saleh
In case a device have a restricted DMA pool, it will be decrypted by default.
However, in the path of
dma_direct_alloc()memory can be allocated from this pool using,__dma_direct_alloc_pages()=>dma_direct_alloc_swiotlb()After that from the same function, it will attempt to decrypt it using
dma_set_decrypted()ifforce_dma_unencrypted().Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- CCA: Clear a bit in the page table and call realmIPA_STATE_SET.
- TDX: Issue a hypercall.
- pKVM: Which doesn't implementforce_dma_unencrypted()at the moment, uses a share hypercall.Change that to only encrypt/decrypt memory that are not allocated from the restricted dma pools.
Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/dma/direct.c +0
-0@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)Quoted 11 lines
kernel/dma/direct.c bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) · line 83
Quoted 5 lines
kernel/dma/direct.c +1
-1static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;This seems really obtuse, I would expect the decryption state of the memory to be known by the caller. If
dma_direct_alloc_swiotlb()can return decrypted or encrypted memory it needs to return a flag saying that. It shouldn't be deduced by checking dev flags in random places like this.
Quoted 8 lines
At the moment restricted dma is always decrypted, also it’s per device so we don’t have to check this per allocation.
I can change the signature for__dma_direct_alloc_pages()to make it return an extra flag but that feels more complicated as it changesdma_direct_alloc_swiotlb(),swiotlb_alloc()with its callers.I can investigate this approach further.
I know there is a v3, but I’m commenting here to check whether we really need to do this now. In my opinion, we can keep this much simpler by treating all swiotlb allocations as decrypted allocations. We also need this to simplify the IPA change alignment updates I posted
https://lore.kernel.org/all/[email protected]
-aneesh
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 3 lines · Aneesh Kumar K.V
I know there is a v3, but I’m commenting here to check whether we really need to do this now. In my opinion, we can keep this much simpler by treating all swiotlb allocations as decrypted allocations.
That's not going to work when we get to supporting T=1 devices that may still want SWIOTLB for encrypted memory.
Jason
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Aneesh Kumar K.V
I know there is a v3, but I’m commenting here to check whether we really need to do this now. In my opinion, we can keep this much simpler by treating all swiotlb allocations as decrypted allocations.
That's not going to work when we get to supporting T=1 devices that may still want SWIOTLB for encrypted memory.
Earlier discussions indicated that a T=1 device would not require a bounce buffer. On ARM, we require such devices to support 64-bit DMA so they can operate with an unprotected IPA.
-aneesh
Re: [RFC PATCH v2 1/5] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 8 lines · Aneesh Kumar K.V
Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines · Aneesh Kumar K.V
I know there is a v3, but I’m commenting here to check whether we really need to do this now. In my opinion, we can keep this much simpler by treating all swiotlb allocations as decrypted allocations.
Quoted 7 lines
Quoted 3 lines
That's not going to work when we get to supporting T=1 devices that may still want SWIOTLB for encrypted memory.
Earlier discussions indicated that a T=1 device would not require a bounce buffer. On ARM, we require such devices to support 64-bit DMA so they can operate with an unprotected IPA.
That was somewhat different. To have a sane system a T=1 device should be able to access unprotected memory and there is no *requirement* that bounce buffering is used.
That is very different from saying that SWIOTLB in the kernel is completely broken and unworking on T=1 devices. We should not do that, it should work correctly and we should have a sensible architecture here. It is just something that would never/rarely be triggered.
Jason
RFC PATCH v2 2/5 dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
In reply to Mostafa SalehAs restricted dma pools are always decrypted, in swiotlb.c it uses phys_to_dma_unencrypted() for address conversion.
However, in DMA-direct, calls to phys_to_dma_direct() with force_dma_unencrypted() returning false, will fallback to phys_to_dma() which is inconsistent for memory allocated from restricted dma pools.
--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/dma/direct.c +1 -1
@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);return phys_to_dma (dev, phys); }
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 19 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 30
Quoted 6 lines
kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);
Same remark, I think the force_dma_unencrypted() was a hack to make up for a flag here. In these lower layers we need to annotate if phys is encrypted/decrypted and take the proper action universially.
The force_dma_unencrypted() should only be done way up the call chain where we decide to get a phys that is decrypted. Once we have a decrypted phys it should be carried with an annotation throughout all the other places.
Then this is more like:
if (flags & FLAG_DECRYPTED)
return phys_to_dma_unencrypted(dev, phys);
Jason
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 31 lines · Jason Gunthorpe
Quoted 19 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 30
Quoted 6 lines
kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);Same remark, I think the
force_dma_unencrypted()was a hack to make up for a flag here. In these lower layers we need to annotate if phys is encrypted/decrypted and take the proper action universially.
Quoted 4 lines
The
force_dma_unencrypted()should only be done way up the call chain where we decide to get a phys that is decrypted. Once we have a decrypted phys it should be carried with an annotation throughout all the other places.
Can you please clarify what you mean by annotation in this context? As I believe any tracking in the vmemmap is a big NO.
As replied to the first patch I can attempt to implement this approach (by passing a flag around) and see how intrusive it would be.
Thanks,
Mostafa
Quoted 10 lines
Then this is more like:
if (flags &
FLAG_DECRYPTED)
returnphys_to_dma_unencrypted(dev, phys);Jason
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 7 lines · Mostafa Saleh
Quoted 4 lines
The
force_dma_unencrypted()should only be done way up the call chain where we decide to get a phys that is decrypted. Once we have a decrypted phys it should be carried with an annotation throughout all the other places.Can you please clarify what you mean by annotation in this context? As I believe any tracking in the vmemmap is a big NO.
It would have to be a flag pass along the phys, or phys & flag in a struct or some kind of approach like that.
Quoted 2 lines
As replied to the first patch I can attempt to implement this approach (by passing a flag around) and see how intrusive it would be.
I'm less concerned about intrusive and more about making this understandable.
When we reach a function with a phys it should know what that phys is, not call a bunch of random helpers to hopefully correctly guess what it is, that's unmaintainable spaghetti even if it is fewer changes.
Jason
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
In reply to Mostafa SalehQuoted 19 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 31
Quoted 7 lines
kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);
So what kind of device is this? Is it a trusted device that needs to use swiotlb in unencrypted form?(is that a valid use case?) Can we add additional comment explaining the type of device for which we are allocating the DMA buffer?
Quoted 4 lines
kernel/dma/direct.c +0
-0return phys_to_dma (dev, phys);}
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
-aneesh
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 25 lines · Aneesh Kumar K.V
Quoted 23 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev))
Quoted 8 lines
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);
Quoted 3 lines
kernel/dma/direct.c +1
-0+if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);So what kind of device is this? Is it a trusted device that needs to use swiotlb in unencrypted form?(is that a valid use case?) Can we add additional comment explaining the type of device for which we are allocating the DMA buffer?
That’s used for devices that use restricted-dma pools which are currently always decrypted, typically virtio devices that are emulated by the untrusted host.
Thanks,
Mostafa
Quoted 7 lines
Quoted 5 lines
kernel/dma/direct.c +0
-0return phys_to_dma (dev, phys);}
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog-aneesh
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 35 lines · Mostafa Saleh
Quoted 25 lines · Aneesh Kumar K.V
Quoted 23 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev))Quoted 8 lines
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);
Quoted 3 lines
kernel/dma/direct.c +1
-0+if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);So what kind of device is this? Is it a trusted device that needs to use swiotlb in unencrypted form?(is that a valid use case?) Can we add additional comment explaining the type of device for which we are allocating the DMA buffer?
Quoted 4 lines
That’s used for devices that use restricted-dma pools which are currently always decrypted, typically virtio devices that are emulated by the untrusted host.
Why would those devices not force unencrypted DMA? Sorry, I may not be following all the details. The pool itself is decrypted by default, but the device does not force unencrypted DMA?”
-aneesh
Re: [RFC PATCH v2 2/5] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 37 lines · Aneesh Kumar K.V
Quoted 35 lines · Mostafa Saleh
Quoted 33 lines · Aneesh Kumar K.V
Quoted 19 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Fixes f4111e39a52a ("swiotlb: Add restricted DMA alloc/free support")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-0kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 31
Quoted 7 lines
kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);So what kind of device is this? Is it a trusted device that needs to use swiotlb in unencrypted form?(is that a valid use case?) Can we add additional comment explaining the type of device for which we are allocating the DMA buffer?
Quoted 9 lines
Quoted 4 lines
That’s used for devices that use restricted-dma pools which are currently always decrypted, typically virtio devices that are emulated by the untrusted host.
Why would those devices not force unencrypted DMA? Sorry, I may not be following all the details. The pool itself is decrypted by default, but the device does not force unencrypted DMA?”
That’s the case at the moment for pKVM guests; that doesn’t use force_dma_unencrypted(), and just rely on swiotlb.
When I was trying to enable that, I ended up with the double encryption issue fixed in this series.
I believe that swiotlb and force_dma_unencrypted() are completely independent and the code shouldn’t make any assumptions about those, ideally force_dma_unencrypted() is a property of the device in a protected guest and swiotlb is per pool.
Thanks,
Mostafa
Quoted 1 line
-aneesh
RFC PATCH v2 3/5 dma-mapping: Decrypt memory on remap
In reply to Mostafa SalehIn case memory needs to be remapped on systems with force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.
The memory still needs to be decrypted in that case.
With memory decryption, don't allow highmem allocations, but that shouldn't be a problem on such modern systems.
--- kernel/dma/direct.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
kernel/dma/direct.c +11 -5
@@ -203,6 +203,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { +bool allow_highmem = !force_dma_unencrypted (dev);bool remap = false, set_uncached = false;struct page *page;void *ret;
@@ -251,7 +252,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp);/* we always manually zero the memory once we are done */ - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, allow_highmem);if (!page)return NULL ;
@@ -265,6 +266,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages; +if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs);
@@ -278,11 +282,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =dma_common_contiguous_remap (page, size, prot,__builtin_return_address (0 ));if (!ret) - goto out_free_pages; + goto out_encrypt_pages; }else { ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) - goto out_leak_pages; }memset (ret,0 , size);
@@ -300,7 +302,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, out_encrypt_pages:if (dma_set_encrypted (dev,page_address (page), size))return NULL ; -out_free_pages:__dma_direct_free_pages (dev, page, size);return NULL ; out_leak_pages:
@@ -339,7 +340,12 @@ void dma_direct_free(struct device *dev, size_t size,return ;if (is_vmalloc_addr (cpu_addr)) { +void *vaddr =page_address (dma_direct_to_page (dev, dma_addr)); +vunmap (cpu_addr); + +if (dma_set_encrypted (dev, vaddr, size)) +return ; }else {if (IS_ENABLED (CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED ))arch_dma_clear_uncached (cpu_addr, size);
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
Re: [RFC PATCH v2 3/5] dma-mapping: Decrypt memory on remap
Quoted 10 lines · Mostafa Saleh
kernel/dma/direct.c +3
-0@@ -265,6 +266,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages; +if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs);
Quoted 2 lines
if (force_dma_unencrypted(dev))
prot = pgprot_decrypted(prot);It seems confusing, why do we unconditionally call something called dma_set_decrypted() and then conditionally call pgprot_decrypted()?
So, I think the same remark, lets not sprinkle these tests all over the place and risk them becoming inconsistent. It should be much more direct, like:
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO,
allow_highmem, &flags);
if (!dev_can_dma_from_encrypted(dev) && !(flags & FLAG_DECRYPTED)) {
dma_set_decrypted(dev, page_address(page));
flags = FLAG_DECRYPTED;
}
if (flags & FLAG_DECRYPTED)
prot = pgprot_decrypted(prot);And so on.
The one place we should see a force_dma_unencrypted() is directly before setting the flag.
Jason
Re: [RFC PATCH v2 3/5] dma-mapping: Decrypt memory on remap
Quoted 12 lines · Jason Gunthorpe
Quoted 10 lines · Mostafa Saleh
kernel/dma/direct.c +3
-0@@ -265,6 +266,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages; +if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs);
Quoted 5 lines
Quoted 2 lines
if (force_dma_unencrypted(dev)) prot = pgprot_decrypted(prot);It seems confusing, why do we unconditionally call something called
dma_set_decrypted()and then conditionally callpgprot_decrypted()?
dma_set_decrypted() will call force_dma_unencrypted() so that check is consistent.
Quoted 4 lines
So, I think the same remark, lets not sprinkle these tests all over the place and risk them becoming inconsistent. It should be much more direct, like:
I agree we shouldn’t be sprinkling all these random calls all over the code, that’s why I was trying to consolidate the logic in the next patch.
Quoted 12 lines
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO,
allow_highmem, &flags);
if (!dev_can_dma_from_encrypted(dev) && !(flags & FLAG_DECRYPTED)) {
dma_set_decrypted(dev, page_address(page));
flags = FLAG_DECRYPTED;
}
if (flags & FLAG_DECRYPTED)
prot = pgprot_decrypted(prot);Quoted 4 lines
And so on.
The one place we should see a
force_dma_unencrypted()is directly before setting the flag.
I will look more into this, but my main worry would be phys_to_dma_direct() and it's callers as I am not sure if is possible to preserve the alloction origin in all contexts.
Thanks,
Mostafa
Quoted 2 lines
Jason
Re: [RFC PATCH v2 3/5] dma-mapping: Decrypt memory on remap
Quoted 6 lines · Mostafa Saleh
Quoted 2 lines
The one place we should see a
force_dma_unencrypted()is directly before setting the flag.I will look more into this, but my main worry would be
phys_to_dma_direct()and it's callers as I am not sure if is possible to preserve the alloction origin in all contexts.
I'd rather have a very small number of places that recover the phys & flag through some convoluted logic than have it sprinkled all over the place in ever layer..
The helpers were not helpful, they were much more confusing.
Jason
Re: [RFC PATCH v2 3/5] dma-mapping: Decrypt memory on remap
In reply to Mostafa SalehQuoted 70 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
With memory decryption, don't allow highmem allocations, but that shouldn't be a problem on such modern systems.
Reported-by Catalin Marinas <[email protected]>Fixes f3c962226dbe ("dma-direct: clean up the remapping checks in dma_direct_alloc")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)kernel/dma/direct.c +6
-5@@ -203,6 +203,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { +bool allow_highmem = !force_dma_unencrypted (dev);bool remap = false, set_uncached = false;struct page *page;void *ret;@@ -251,7 +252,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp);/* we always manually zero the memory once we are done */ - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, allow_highmem);if (!page)return NULL ;@@ -265,6 +266,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages; +if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs);@@ -278,11 +282,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =dma_common_contiguous_remap (page, size, prot,__builtin_return_address (0 ));if (!ret) - goto out_free_pages; + goto out_encrypt_pages; }else { ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) - goto out_leak_pages; }memset (ret,0 , size);@@ -300,7 +302,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, out_encrypt_pages:if (dma_set_encrypted (dev,page_address (page), size))return NULL ; -out_free_pages:__dma_direct_free_pages (dev, page, size);return NULL ; out_leak_pages:
kernel/dma/direct.c void dma_direct_free(struct device *dev, size_t size, · line 348
Quoted 9 lines
kernel/dma/direct.c +5
-0@@ -339,7 +340,12 @@ void dma_direct_free(struct device *dev, size_t size,return ;if (is_vmalloc_addr (cpu_addr)) { +void *vaddr =page_address (dma_direct_to_page (dev, dma_addr)); +vunmap (cpu_addr); + +if (dma_set_encrypted (dev, vaddr, size)) +return ;
Right now, a remap is required under two conditions:
1. HighMem — I assume we are avoiding this for devices that require memory decryption.
2. The device is not DMA-coherent.
Can we assume that condition (2) will also not be supported alongside memory encryption/decryption? That would allow us to simplify all of this. We would then only need to carry the patch that disables HighMem for devices requiring unencrypted DMA buffers.
I did post a patch along similar lines some time back. There is also the challenge of presenting a vmap address as decrypted on ARM.
https://lore.kernel.org/all/[email protected]
Quoted 5 lines
kernel/dma/direct.c +0
-0}else {if (IS_ENABLED (CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED ))arch_dma_clear_uncached (cpu_addr, size);Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
Re: [RFC PATCH v2 3/5] dma-mapping: Decrypt memory on remap
Quoted 88 lines · Aneesh Kumar K.V
Quoted 70 lines · Mostafa Saleh
In case memory needs to be remapped on systems with
force_dma_unencrypted(), where this memory is not allocated from a restricted-dma pool, this was currently ignored, while only setting the decrypted pgprot in the remapped alias.The memory still needs to be decrypted in that case.
With memory decryption, don't allow highmem allocations, but that shouldn't be a problem on such modern systems.
Reported-by Catalin Marinas <[email protected]>Fixes f3c962226dbe ("dma-direct: clean up the remapping checks in dma_direct_alloc")Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)kernel/dma/direct.c +6
-5@@ -203,6 +203,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { +bool allow_highmem = !force_dma_unencrypted (dev);bool remap = false, set_uncached = false;struct page *page;void *ret;@@ -251,7 +252,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp);/* we always manually zero the memory once we are done */ - page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, true); + page =__dma_direct_alloc_pages (dev, size, gfp & ~__GFP_ZERO, allow_highmem);if (!page)return NULL ;@@ -265,6 +266,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached = false; } +if (dma_set_decrypted (dev,page_address (page), size)) + goto out_leak_pages; +if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs);@@ -278,11 +282,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =dma_common_contiguous_remap (page, size, prot,__builtin_return_address (0 ));if (!ret) - goto out_free_pages; + goto out_encrypt_pages; }else { ret =page_address (page); -if (dma_set_decrypted (dev, ret, size)) - goto out_leak_pages; }memset (ret,0 , size);@@ -300,7 +302,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, out_encrypt_pages:if (dma_set_encrypted (dev,page_address (page), size))return NULL ; -out_free_pages:__dma_direct_free_pages (dev, page, size);return NULL ; out_leak_pages:kernel/dma/direct.c void dma_direct_free(struct device *dev, size_t size, · line 348
Quoted 9 lines
kernel/dma/direct.c +5
-0@@ -339,7 +340,12 @@ void dma_direct_free(struct device *dev, size_t size,return ;if (is_vmalloc_addr (cpu_addr)) { +void *vaddr =page_address (dma_direct_to_page (dev, dma_addr)); +vunmap (cpu_addr); + +if (dma_set_encrypted (dev, vaddr, size)) +return ;Right now, a remap is required under two conditions:
1. HighMem — I assume we are avoiding this for devices that require memory decryption.
2. The device is not DMA-coherent.
Quoted 8 lines
Can we assume that condition (2) will also not be supported alongside memory encryption/decryption? That would allow us to simplify all of this. We would then only need to carry the patch that disables HighMem for devices requiring unencrypted DMA buffers.
I did post a patch along similar lines some time back. There is also the challenge of presenting a vmap address as decrypted on ARM.
https://lore.kernel.org/all/[email protected]
In v3, I prevent highmem allocations for devices that need decryption, I believe that modern systems with CCA won’t have a problem with that.
With pKVM, it’s possible to have non-coherent devices, but they always have an IOMMU, so direct-dma is only used for virtualized devices which are coherent. So, this case is not important.
This is a theoretical bug, I am not sure if any systems are impacted by, it but I included since Catalin mentioned it on v1 as I am already including other fixes. But I am happy to drop it or just add an assertion that it never happens.
Thanks,
Mostafa
Quoted 7 lines
Quoted 7 lines
kernel/dma/direct.c +0
-0}else {if (IS_ENABLED (CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED ))arch_dma_clear_uncached (cpu_addr, size);Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
RFC PATCH v2 4/5 dma-mapping: Refactor memory encryption usage
In reply to Mostafa SalehAt the moment dma-direct deals with memory encryption in 2 cases
- Pre-decrypted restricted dma-pools
- Arch code through force_dma_unencrypted()
In the first case, the memory is owned by the pool and the decryption is not managed by the dma-direct.
However, it should be aware of it to use the appropriate phys_to_dma* and page table prot.
For the second case, it’s the job of the dma-direct to manage the decryption of the allocated memory.
As there have been bugs in this code due to wrong or missing checks and there are more use cases coming for memory decryption, we need more robust checks in the code to abstract the core logic, so introduce some local helpers:
- dma_external_decryption(): For pages decrypted but managed externally
- dma_owns_decryption(): For pages need to be decrypted and managed by dma-direct
- is_dma_decrypted(): To check if memory is decrypted
Note that this patch is not a no-op as there are some subtle changes which are actually theoretical bug fixes in dma_direct_mmap() and dma_direct_alloc() where the wrong prot might be used for remap.
--- kernel/dma/direct.c | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-)
kernel/dma/direct.c +27 -10
@@ -23,10 +23,27 @@*/ u64 zone_dma_limit__ro_after_init =DMA_BIT_MASK (24 ); +/* Memory is decrypted and managed externally. */ +static inline bool dma_external_decryption (struct device *dev) +{ +return is_swiotlb_for_alloc (dev); +} + +/* Memory needs to be decrypted by the dma-direct layer. */ +static inline bool dma_owns_decryption (struct device *dev) +{ +return force_dma_unencrypted (dev) && !dma_external_decryption (dev); +} + +static inline bool is_dma_decrypted (struct device *dev) +{ +return force_dma_unencrypted (dev) ||dma_external_decryption (dev); +} +static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev)) +if (is_dma_decrypted (dev))return phys_to_dma_unencrypted (dev, phys);return phys_to_dma (dev, phys); }
@@ -79,7 +96,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev)) +if (!dma_owns_decryption (dev))return 0 ;return set_memory_decrypted ((unsigned long )vaddr,PFN_UP (size)); }
@@ -88,7 +105,7 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) {int ret; -if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev)) +if (!dma_owns_decryption (dev))return 0 ; ret =set_memory_encrypted ((unsigned long )vaddr,PFN_UP (size));if (ret)
@@ -203,7 +220,7 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,void *dma_direct_alloc (struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) { -bool allow_highmem = !force_dma_unencrypted (dev); +bool allow_highmem = !dma_owns_decryption (dev);bool remap = false, set_uncached = false;struct page *page;void *ret;
@@ -213,7 +230,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, gfp |= __GFP_NOWARN;if ((attrs &DMA_ATTR_NO_KERNEL_MAPPING ) && - !force_dma_unencrypted (dev) && !is_swiotlb_for_alloc (dev)) + !is_dma_decrypted (dev))return dma_direct_alloc_no_mapping (dev, size, dma_handle, gfp);if (!dev_is_dma_coherent (dev)) {
@@ -247,7 +264,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,* Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. */ -if ((remap ||force_dma_unencrypted (dev)) && +if ((remap ||dma_owns_decryption (dev)) &&dma_direct_use_pool (dev, gfp))return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp);
@@ -272,7 +289,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,if (remap) {pgprot_t prot =dma_pgprot (dev,PAGE_KERNEL , attrs); -if (force_dma_unencrypted (dev)) +if (is_dma_decrypted (dev)) prot =pgprot_decrypted (prot);/* remove any dirty cache lines on the kernel alias */
@@ -314,7 +331,7 @@ void dma_direct_free(struct device *dev, size_t size,unsigned int page_order =get_order (size);if ((attrs &DMA_ATTR_NO_KERNEL_MAPPING ) && - !force_dma_unencrypted (dev) && !is_swiotlb_for_alloc (dev)) { + !is_dma_decrypted (dev)) {/* cpu_addr is a struct page cookie, not a kernel address */ dma_free_contiguous (dev, cpu_addr, size);return ;
@@ -362,7 +379,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,struct page *page;void *ret; -if (force_dma_unencrypted (dev) &&dma_direct_use_pool (dev, gfp)) +if (dma_owns_decryption (dev) &&dma_direct_use_pool (dev, gfp))return dma_direct_alloc_from_pool (dev, size, dma_handle, gfp); page =__dma_direct_alloc_pages (dev, size, gfp, false);
@@ -530,7 +547,7 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma,int ret = -ENXIO ; vma->vm_page_prot =dma_pgprot (dev, vma->vm_page_prot, attrs); -if (force_dma_unencrypted (dev)) +if (is_dma_decrypted (dev)) vma->vm_page_prot =pgprot_decrypted (vma->vm_page_prot);if (dma_mmap_from_dev_coherent (dev, vma, cpu_addr, size, &ret))
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog
Re: [RFC PATCH v2 4/5] dma-mapping: Refactor memory encryption usage
Quoted 13 lines · Mostafa Saleh
At the moment dma-direct deals with memory encryption in 2 cases
- Pre-decrypted restricted dma-pools
- Arch code throughforce_dma_unencrypted()In the first case, the memory is owned by the pool and the decryption is not managed by the dma-direct.
However, it should be aware of it to use the appropriate phys_to_dma* and page table prot.
For the second case, it’s the job of the dma-direct to manage the decryption of the allocated memory.
Quoted 8 lines
As there have been bugs in this code due to wrong or missing checks and there are more use cases coming for memory decryption, we need more robust checks in the code to abstract the core logic, so introduce some local helpers:
-dma_external_decryption(): For pages decrypted but managed externally
-dma_owns_decryption(): For pages need to be decrypted and managed by dma-direct
-is_dma_decrypted(): To check if memory is decrypted
I can't even make sense of what this is trying to explain, talking about page management along with 'dev' is nonsense. The management of pages is intrinsic to the API, it doesn't change.
I think start with adding a direct flags annotation and then come back to figure out if we need some kind of helpers.
I would expect any helper taking in dev to only be answering two dev questions:
'dev can dma from encrypted(dev)'
'dev can dma from unencrytped(dev)'
At each of the points in the API flow the phys under consideration is known to be encrypted or decrypted, and those two helpers tell everything needed.
So I'd expect the restricted flow to look more like
- phys comes in to be dma mapped, it is encrypted
- 'dev can dma from encrypted(dev)' fails so we go to swiotlb
- swiotlb allocates from a restricted pool, and learns through flags that the new phys is decrypted
- decrypted phys flows through the rest of the machinery.
We never check 'dev can dma from encrypted(dev)' a second time.
Jason
RFC PATCH v2 5/5 dma-mapping: Add doc for memory encryption
In reply to Mostafa SalehAdd a document for memory encryption usage with dma-direct.
--- .../core-api/dma-direct-memory-encryption.rst | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 Documentation/core-api/dma-direct-memory-encryption.rst
Documentation/core-api/dma-direct-memory-encryption.rst new file +77 -0
@@ -0,0 +1,77 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================================ +DMA Direct and Memory Encryption Integration +============================================ + +Introduction +------------ +Modern platforms introduce memory encryption features (e.g., AMD SEV, Intel TDX, +ARM CCA, and pKVM) typically for CoCo when running protected virtual machines. + +These guests typically boot with their memory encrypted by default. + +In some cases this memory needs to be accessed by the untrusted host or the +VMM which then requires this memory to be decrypted. One typical case is +dealing with emulated device (e.g., virtio) which are handled by direct-dma +code as these devices are not behind an IOMMU. + +That means, the memory used by these devices must be decrypted before accessed +by the untrusted host. + +It must be clarified that encrypted/decrypted may not always be +cryptographic; in a broader sense, a decrypted page means that it is +accessible or "shared" with the untrusted host. + +Ownership +--------- +The direct-dma layer deals with memory encryption in two distinct scenarios: + +1. **Externally Managed Decryption (e.g., Restricted DMA Pools)** + In some setups (like a device restricted to a specific SWIOTLB pool, i.e., + `DMA_RESTRICTED_POOL`), an entire region of memory is pre-decrypted during + boot or pool initialization. The memory is owned by the pool, and the + transitions (encryption/decryption) are **not** managed by direct-dma on + a per-allocation basis. + See Documentation/core-api/swiotlb.rst + +2. **DMA Direct Managed Decryption (e.g., `force_dma_unencrypted()`)** + For standard coherent DMA allocations the direct-dma layer is explicitly + responsible for managing the decryption. It must decrypt the pages upon + allocation and re-encrypt them upon freeing. + +To cleanly separate these concerns, the core logic is abstracted via three +internal helpers: + +* ``dma_external_decryption(dev)``: Returns true if the pages are decrypted but + managed externally. For example, if the device allocates from a restricted + DMA pool. +* ``dma_owns_decryption(dev)``: Returns true if the pages need to be explicitly + decrypted and managed by the direct-dma layer (i.e., the architecture forces + unencrypted DMA, and it's not handled by an external pool). +* ``is_dma_decrypted(dev)``: Returns true if the memory being used is in a + decrypted state, regardless of who manages it. + +Addressing and Page Protections +------------------------------- +When memory is decrypted (whether externally or by direct-dma), the layer must +adjust physical-to-DMA address conversions and page protections: + +* **DMA Address Conversion:** + Decrypted memory often requires a specific bit to be cleared or set in the DMA + address (e.g., stripping the encryption bit). If ``is_dma_decrypted(dev)`` is + true, the conversion uses ``phys_to_dma_unencrypted()`` instead of the standard + ``phys_to_dma()``. + +* **Page Protections (Remap and Mmap):** + When remapping decrypted pages into the kernel virtual address space (vmalloc) + or mapping them to user space via ``mmap()``, the page protection attributes + must reflect the decrypted state. If ``is_dma_decrypted(dev)`` is true, the + layer applies ``pgprot_decrypted(prot)`` to ensure the CPU accesses the memory + with the correct encryption attributes. + +Notes +----- +In many cases when memory encryption/decryption fails the page will be leaked, +that's was added for TDX, where ``set_memory_encrypted()`` or +``set_memory_decrypted()``may fail while the page remains shared.
Signature · 1 line
2.53.0.1185.g05d4b7b318-goog