dma-mapping: DMA_RESTRICTED_POOL and encryption
cover dma-mapping: DMA_RESTRICTED_POOL and encryption
I have been looking into DMA code with DMA_RESTRICTED_POOL and how it interacts with the memory encryption API, mainly in the context of protected KVM (pKVM) on Arm64.
While trying to extend force_dma_unencrypted() to be pKVM aware I noticed some inconsistencies in direct-dma code which looks as bugs. I am not sure if there are any architectures affected by this at the moment as some of the logic of memory encryption is forwarded to the hypervisor as hypercalls ore realm calls.
I have wrote some fixes from my simplistic understanding.
However, Future looking, I feel like we would need to have a more solid API for memory encryption and decryption, that can be used consistently from both SWIOTLB(so we can also not decrypt per-device pools by default), DMA-direct and other subsystems.
That would be useful in cases (at least for pKVM) where a device would need to have a private encrypted pool, (if it needs to bounce memory for any reason with leaking information by decrypting the data). I am not sure how other CCA solutions deals with in Linux, I am assuming they won't to need to bounce at all?
I can send another series for this which adds a property to SWIOTLB buffers to be decrypted by default if that makes sense.
Mostafa Saleh (2):
dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
kernel/dma/direct.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
RFC PATCH 1/2 dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
In case a device have a restricted DMA pool, it will be decrypted. However, in the path of dma_direct_alloc() memory can be allocated from this pool using, __dma_direct_alloc_pages() => dma_direct_alloc_swiotlb()
After that from the same function, it will attempt to decrypt it using dma_set_decrypted() if force_dma_unencrypted().
Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- Clear a bit in the page table and call realm IPA_STATE_SET
- TDX: Seems to issue a hypercall also.
- pKVM: Which doesn't implement force_dma_unencrypted() at the moment, uses a share hypercall which is definitely not Idempotent.
This patch will only encrypt/decrypt memory that are not allocated form the restricted dma pools.
--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/dma/direct.c +2 -2
@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;return set_memory_decrypted ((unsigned long )vaddr,PFN_UP (size)); }
@@ -88,7 +88,7 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) {int ret; -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ; ret =set_memory_encrypted ((unsigned long )vaddr,PFN_UP (size));if (ret)
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
Re: [RFC PATCH 1/2] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 39 lines · Mostafa Saleh
In case a device have a restricted DMA pool, it will be decrypted. However, in the path of
dma_direct_alloc()memory can be allocated from this pool using,__dma_direct_alloc_pages()=>dma_direct_alloc_swiotlb()After that from the same function, it will attempt to decrypt it using
dma_set_decrypted()ifforce_dma_unencrypted().Which results in the memory being decrypted twice.
It's not clear how the does realm world/hypervisors deal with that, for example:
- Clear a bit in the page table and call realmIPA_STATE_SET
- TDX: Seems to issue a hypercall also.
- pKVM: Which doesn't implementforce_dma_unencrypted()at the moment, uses a share hypercall which is definitely not Idempotent.This patch will only encrypt/decrypt memory that are not allocated form the restricted dma pools.
Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/dma/direct.c +1
-1@@ -79,7 +79,7 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)static int dma_set_decrypted (struct device *dev,void *vaddr,size_t size) { -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ;return set_memory_decrypted ((unsigned long )vaddr,PFN_UP (size)); }
kernel/dma/direct.c static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) · line 94
Quoted 8 lines
kernel/dma/direct.c +1
-1@@ -88,7 +88,7 @@ static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) {int ret; -if (!force_dma_unencrypted (dev)) +if (!force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return 0 ; ret =set_memory_encrypted ((unsigned long )vaddr,PFN_UP (size));if (ret)
I think that's functionally correct for rmem buffers. Normally I'd have moved the is_swiotlb_for_alloc() condition in the caller but even dma_direct_alloc() doesn't know where the buffer came from, it's hidden in __dma_direct_alloc_pages().
However, it's unclear to me whether we can get encrypted pages when is_swiotlb_for_alloc() == false, remap == true and force_dma_unencrypted() == true in dma_direct_alloc(). dma_set_decrypted() is only called on the !remap path.
Signature · 1 line
Catalin
Re: [RFC PATCH 1/2] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 4 lines · Catalin Marinas
However, it's unclear to me whether we can get encrypted pages when
is_swiotlb_for_alloc()== false, remap == true andforce_dma_unencrypted()== true indma_direct_alloc().dma_set_decrypted()is only called on the !remap path.
Ah, I can see Anneesh trying to address this here:
https://lore.kernel.org/r/[email protected]
Signature · 1 line
Catalin
Re: [RFC PATCH 1/2] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 9 lines · Catalin Marinas
Quoted 4 lines · Catalin Marinas
However, it's unclear to me whether we can get encrypted pages when
is_swiotlb_for_alloc()== false, remap == true andforce_dma_unencrypted()== true indma_direct_alloc().dma_set_decrypted()is only called on the !remap path.Ah, I can see Anneesh trying to address this here:
I see, thanks for pointing that out, the case Aneesh is fixing is the missing decryption in the remap case. However, it’s not clear to me how we can get there for CCA, I left a comment on his patch.
I can inline the is_swiotlb_for_alloc() checks outside, but I believe adding this in the lowest level is better as indeed the memory is decrypted and we don’t have to open code the check in other places are dma_direct_alloc_pages()
Thanks,
Mostafa
Quoted 3 lines
Signature · 2 lines
-- Catalin
Re: [RFC PATCH 1/2] dma-mapping: Avoid double decrypting with DMA_RESTRICTED_POOL
Quoted 11 lines · Mostafa Saleh
Quoted 9 lines · Catalin Marinas
Quoted 4 lines · Catalin Marinas
However, it's unclear to me whether we can get encrypted pages when
is_swiotlb_for_alloc()== false, remap == true andforce_dma_unencrypted()== true indma_direct_alloc().dma_set_decrypted()is only called on the !remap path.Ah, I can see Anneesh trying to address this here:
Quoted 9 lines
I see, thanks for pointing that out, the case Aneesh is fixing is the missing decryption in the remap case. However, it’s not clear to me how we can get there for CCA, I left a comment on his patch.
I can inline the
is_swiotlb_for_alloc()checks outside, but I believe adding this in the lowest level is better as indeed the memory is decrypted and we don’t have to open code the check in other places aredma_direct_alloc_pages()
There are a few related changes that I have posted. However, I am wondering whether it would be simpler to treat the swiotlb pool as always decrypted. In that case, even when allocating from swiotlb we would not need to toggle between decrypt/encrypt.
Another reason to treat swiotlb as special is the alignment requirement when toggling between decrypted and encrypted states.
The patch implementing this approach is here
https://lore.kernel.org/all/[email protected]
With respect to remapping, there are two conditions that can currently trigger a remap: when the device is non-coherent, or when we receive a HighMem allocation. Neither of these conditions applies to CCA. We could potentially enforce the HighMem case by using the following hunk in the patch:
+
+ if (force_dma_unencrypted(dev))
+ /*
+ * Unencrypted/shared DMA requires a linear-mapped buffer
+ * address to look up the PFN and set architecture-required PFN
+ * attributes. This is not possible with HighMem. Avoid HighMem
+ * allocation.
+ */
+ allow_highmem = false;
+
/* we always manually zero the memory once we are done */ - page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); + page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, allow_highmem); if (!page) return NULL;
https://lore.kernel.org/all/[email protected]
I haven't got much feedback on that patch yet.
-aneesh
RFC PATCH 2/2 dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
In reply to Mostafa SalehAs restricted dma pools are always decrypted, in swiotlb.c it uses phys_to_dma_unencrypted() for address conversion.
However, in DMA-direct, calls to phys_to_dma_direct() with force_dma_unencrypted() returning false, will fallback to phys_to_dma() which is inconsistent for memory allocated from restricted dma pools.
--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/dma/direct.c +1 -1
@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);return phys_to_dma (dev, phys); }
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
Re: [RFC PATCH 2/2] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 18 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 32
Quoted 8 lines
kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);return phys_to_dma (dev, phys); }
I couldn't fully get my head around the DMA API but I think all the pools and bounce buffers are decrypted and protected guests (or realms for Arm CCA) should always return true for force_dma_unencrypted(). If that's the case, the above change wouldn't be necessary. I can see that arm64 only does this for CCA and not pKVM guests.
Device assignment is another story that requires reworking those DMA pools to support encrypted buffers.
Signature · 1 line
Catalin
Re: [RFC PATCH 2/2] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
Quoted 25 lines · Catalin Marinas
Quoted 24 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);
Quoted 8 lines
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 32
Quoted 2 lines
kernel/dma/direct.c +0
-0return phys_to_dma (dev, phys); }I couldn't fully get my head around the DMA API but I think all the pools and bounce buffers are decrypted and protected guests (or realms for Arm CCA) should always return true for
force_dma_unencrypted(). If that's the case, the above change wouldn't be necessary. I can see that arm64 only does this for CCA and not pKVM guests.
That is correct. Why would the force_dma_unencrypted() return false for a device ? As far as I can see, all CC guests are treating all devices as "untrusted" for now (and there is a series available that is adding support for "trusted devices [0]).
Quoted 4 lines
Device assignment is another story that requires reworking those DMA pools to support encrypted buffers.
[0] https://lkml.kernel.org/r/[email protected]
Suzuki
Re: [RFC PATCH 2/2] dma-mapping: Use the correct phys_to_dma() for DMA_RESTRICTED_POOL
In reply to Catalin MarinasQuoted 25 lines · Catalin Marinas
Quoted 24 lines · Mostafa Saleh
As restricted dma pools are always decrypted, in
swiotlb.cit usesphys_to_dma_unencrypted()for address conversion.However, in DMA-direct, calls to
phys_to_dma_direct()withforce_dma_unencrypted()returning false, will fallback tophys_to_dma()which is inconsistent for memory allocated from restricted dma pools.Signed-off-by Mostafa Saleh <[email protected]>--- kernel/dma/direct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/dma/direct.c +1
-1@@ -26,7 +26,7 @@ u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys) { -if (force_dma_unencrypted (dev)) +if (force_dma_unencrypted (dev) ||is_swiotlb_for_alloc (dev))return phys_to_dma_unencrypted (dev, phys);
Quoted 9 lines
kernel/dma/direct.c u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); · line 32
Quoted 2 lines
kernel/dma/direct.c +0
-0return phys_to_dma (dev, phys); }I couldn't fully get my head around the DMA API but I think all the pools and bounce buffers are decrypted and protected guests (or realms for Arm CCA) should always return true for
force_dma_unencrypted(). If that's the case, the above change wouldn't be necessary. I can see that arm64 only does this for CCA and not pKVM guests.
Yes, that’s the problem, pKVM relies on SWIOTLB to use decrypted buffers and not force_dma_unencrypted() in DMA-direct. So, at the moment pKVM guests actually call:
- phys_to_dma_unencrypted(): From swiotlb code
- phys_to_dma(): From Direct-DMA code
Which is in-consistent, but only works as the pKVM memory encryption/ decryption is in-place, so there is no address conversion.
I was looking into setting force_dma_unencrypted() to true for pKVM, which then resulted in the bug of double-decryption I am trying to solve with patch-1.
I think the main problem is that SWIOTLB(restricted DMA) decrypts stuff unconditionally, so we have to treat is_swiotlb_for_alloc() the same way as force_dma_unencrypted().
That is what these 2 patches do, otherwise we teach SWIOTLB code about force_dma_unencrypted().
Thanks,
Mostafa
Quoted 7 lines
Device assignment is another story that requires reworking those DMA pools to support encrypted buffers.
Signature · 1 line
Catalin