Enable IOMMU support for pseries Secure VMs
v4 cover Enable IOMMU support for pseries Secure VMs
This patch series enables IOMMU support for pseries Secure VMs.
Tested using QEMU command line option:
"-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4,
iommu_platform=on,disable-modern=off,disable-legacy=on"
and
"-device virtio-blk-pci,scsi=off,bus=pci.0,
addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, iommu_platform=on,disable-modern=off,disable-legacy=on"
changelog:
v4: Corrected the Subject, to include the keyword 'PATCH'.
No other changes.
v3: Better description of 2/2 patch.
Suggested by David Gibson.
v2: added comments describing the changes.
Suggested by Alexey and Michael Ellermen.
Ram Pai (2):
powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor. powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell.
arch/powerpc/platforms/pseries/iommu.c | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v4 1/2 powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
H_PUT_TCE_INDIRECT hcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.
--- arch/powerpc/platforms/pseries/iommu.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
arch/powerpc/platforms/pseries/iommu.c +20 -3
@@ -37,6 +37,7 @@#include <asm/mmzone.h> #include <asm/plpar_wrappers.h> #include <asm/svm.h> +#include <asm/ultravisor.h> #include "pseries.h"
@@ -179,6 +180,23 @@ static int tce_build_pSeriesLP(struct iommu_table *tbl, long tcenum, static DEFINE_PER_CPU(__be64 *, tce_page); +/* +* Allocate a tce page. If secure VM, share the page with the hypervisor. +* +* NOTE: the TCE page is shared with the hypervisor explicitly and remains +* shared for the lifetime of the kernel. It is implicitly unshared at kernel +* shutdown through a UV_UNSHARE_ALL_PAGES ucall. +*/ +static __be64 *alloc_tce_page (void ) +{ +__be64 *tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + +if (tcep &&is_secure_guest ()) +uv_share_page (PHYS_PFN (__pa (tcep)),1 ); + +return tcep; +} + static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, long npages, unsigned long uaddr, enum dma_data_direction direction,
@@ -206,8 +224,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,* from iommu_alloc{,_sg}() */ if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); -/* If allocation fails, fall back to the loop implementation */ + tcep =alloc_tce_page ();if (!tcep) {local_irq_restore (flags); return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,
@@ -405,7 +422,7 @@ static int tce_setrange_multi_pSeriesLP(unsigned long start_pfn, tcep =__this_cpu_read (tce_page);if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + tcep =alloc_tce_page ();if (!tcep) {local_irq_enable ();return -ENOMEM ;
Signature · 1 line
1.8.3.1
PATCH v4 2/2 powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell.
Commit edea902c1c1e ("powerpc/pseries/iommu: Don't use dma_iommu_ops on
secure guests")
disabled dma_iommu_ops path, for secure VMs. The rationale for disabling the dma_iommu_ops path, was to use the dma_direct path, since it had inbuilt support for bounce-buffering through SWIOTLB.
However dma_iommu_ops is functionally much richer. Depending on the capabilities of the platform, it can handle direct DMA; with or without bounce buffering, and it can handle indirect DMA. Hence its better to leverage the richer functionality supported by dma_iommu_ops.
Renable dma_iommu_ops path for pseries Secure VMs.
--- arch/powerpc/platforms/pseries/iommu.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
arch/powerpc/platforms/pseries/iommu.c +1 -10
@@ -36,7 +36,6 @@#include <asm/udbg.h> #include <asm/mmzone.h> #include <asm/plpar_wrappers.h> -#include <asm/svm.h> #include <asm/ultravisor.h> #include "pseries.h"
@@ -1337,15 +1336,7 @@ void iommu_init_early_pSeries(void)of_reconfig_notifier_register (&iommu_reconfig_nb);register_memory_notifier (&iommu_mem_nb); -/* -* Secure guest memory is inacessible to devices so regular DMA isn't -* possible. -* -* In that case keep devices' dma_map_ops as NULL so that the generic -* DMA code path will use SWIOTLB to bounce buffers for DMA. -*/ -if (!is_secure_guest ()) -set_pci_dma_ops (&dma_iommu_ops); +set_pci_dma_ops (&dma_iommu_ops); }static int __init disable_multitce (char *str)
Signature · 1 line
1.8.3.1
Re: [PATCH v4 2/2] powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell.
Quoted 10 lines · Ram Pai
Commit
edea902c1c1e("powerpc/pseries/iommu: Don't usedma_iommu_opson
secure guests")
disableddma_iommu_opspath, for secure VMs. The rationale for disabling thedma_iommu_opspath, was to use thedma_directpath, since it had inbuilt support for bounce-buffering through SWIOTLB.However
dma_iommu_opsis functionally much richer. Depending on the capabilities of the platform, it can handle direct DMA; with or without bounce buffering, and it can handle indirect DMA. Hence its better to leverage the richer functionality supported bydma_iommu_ops.
What exactly do we leverage after applying this patch? afaict things are going to work in exact same old way with this applied, no? Thanks,
Quoted 38 lines
Renable
dma_iommu_opspath for pseries Secure VMs.Signed-off-by Ram Pai <[email protected]>--- arch/powerpc/platforms/pseries/iommu.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)arch/powerpc/platforms/pseries/iommu.c +1
-10@@ -36,7 +36,6 @@#include <asm/udbg.h> #include <asm/mmzone.h> #include <asm/plpar_wrappers.h> -#include <asm/svm.h> #include <asm/ultravisor.h> #include "pseries.h" @@ -1337,15 +1336,7 @@ void iommu_init_early_pSeries(void)of_reconfig_notifier_register (&iommu_reconfig_nb);register_memory_notifier (&iommu_mem_nb); -/* -* Secure guest memory is inacessible to devices so regular DMA isn't -* possible. -* -* In that case keep devices' dma_map_ops as NULL so that the generic -* DMA code path will use SWIOTLB to bounce buffers for DMA. -*/ -if (!is_secure_guest ()) -set_pci_dma_ops (&dma_iommu_ops); +set_pci_dma_ops (&dma_iommu_ops); }static int __init disable_multitce (char *str)
Signature · 1 line
Alexey
RE: [PATCH v4 2/2] powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell.
Quoted 9 lines · Alexey Kardashevskiy
Quoted 6 lines · Ram Pai
Commit
edea902c1c1e("powerpc/pseries/iommu: Don't usedma_iommu_opson
secure guests")
disableddma_iommu_opspath, for secure VMs. The rationale for disabling thedma_iommu_opspath, was to use thedma_directpath, since it had inbuilt support for bounce-buffering through SWIOTLB.
Quoted 7 lines
Quoted 4 lines
However
dma_iommu_opsis functionally much richer. Depending on the capabilities of the platform, it can handle direct DMA; with or without bounce buffering, and it can handle indirect DMA. Hence its better to leverage the richer functionality supported bydma_iommu_ops.What exactly do we leverage after applying this patch? afaict things are going to work in exact same old way with this applied, no? Thanks,
Yes. You got it right. Another way of saying this is, it reverts the changes.
RP
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
In reply to Ram PaiQuoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there. A safer option would be not to use the hcall-multi-tce hyperrtas option (which translates FW_FEATURE_MULTITCE in the guest).
Also what is this for anyway? If I understand things right, you cannot map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,
Quoted 62 lines
Signed-off-by Ram Pai <[email protected]>--- arch/powerpc/platforms/pseries/iommu.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)arch/powerpc/platforms/pseries/iommu.c +20
-3@@ -37,6 +37,7 @@#include <asm/mmzone.h> #include <asm/plpar_wrappers.h> #include <asm/svm.h> +#include <asm/ultravisor.h> #include "pseries.h" @@ -179,6 +180,23 @@ static int tce_build_pSeriesLP(struct iommu_table *tbl, long tcenum, static DEFINE_PER_CPU(__be64 *, tce_page); +/* +* Allocate a tce page. If secure VM, share the page with the hypervisor. +* +* NOTE: the TCE page is shared with the hypervisor explicitly and remains +* shared for the lifetime of the kernel. It is implicitly unshared at kernel +* shutdown through a UV_UNSHARE_ALL_PAGES ucall. +*/ +static __be64 *alloc_tce_page (void ) +{ +__be64 *tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + +if (tcep &&is_secure_guest ()) +uv_share_page (PHYS_PFN (__pa (tcep)),1 ); + +return tcep; +} + static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, long npages, unsigned long uaddr, enum dma_data_direction direction,@@ -206,8 +224,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,* from iommu_alloc{,_sg}() */ if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); -/* If allocation fails, fall back to the loop implementation */ + tcep =alloc_tce_page ();if (!tcep) {local_irq_restore (flags); return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,@@ -405,7 +422,7 @@ static int tce_setrange_multi_pSeriesLP(unsigned long start_pfn, tcep =__this_cpu_read (tce_page);if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + tcep =alloc_tce_page ();if (!tcep) {local_irq_enable ();return -ENOMEM ;
Signature · 1 line
Alexey
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 11 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.
Quoted 6 lines
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled. This happens even on a normal VM when IOMMU is enabled.
Quoted 2 lines
Also what is this for anyway?
This is for sending indirect-TCE entries to the hypervisor. The hypervisor must be able to read those TCE entries, so that it can use those entires to populate the TCE table with the correct mappings.
Quoted 5 lines
if I understand things right, you cannot
map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,
Any random guest memory can be shared by the guest.
Maybe you are confusing this with the SWIOTLB bounce buffers used by PCI devices, to transfer data to the hypervisor?
Quoted 67 lines
Quoted 64 lines
Signed-off-by Ram Pai <[email protected]>--- arch/powerpc/platforms/pseries/iommu.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)arch/powerpc/platforms/pseries/iommu.c +20
-3@@ -37,6 +37,7 @@#include <asm/mmzone.h> #include <asm/plpar_wrappers.h> #include <asm/svm.h> +#include <asm/ultravisor.h> #include "pseries.h" @@ -179,6 +180,23 @@ static int tce_build_pSeriesLP(struct iommu_table *tbl, long tcenum, static DEFINE_PER_CPU(__be64 *, tce_page); +/* +* Allocate a tce page. If secure VM, share the page with the hypervisor. +* +* NOTE: the TCE page is shared with the hypervisor explicitly and remains +* shared for the lifetime of the kernel. It is implicitly unshared at kernel +* shutdown through a UV_UNSHARE_ALL_PAGES ucall. +*/ +static __be64 *alloc_tce_page (void ) +{ +__be64 *tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + +if (tcep &&is_secure_guest ()) +uv_share_page (PHYS_PFN (__pa (tcep)),1 ); + +return tcep; +} + static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, long npages, unsigned long uaddr, enum dma_data_direction direction,@@ -206,8 +224,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,* from iommu_alloc{,_sg}() */ if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); -/* If allocation fails, fall back to the loop implementation */ + tcep =alloc_tce_page ();if (!tcep) {local_irq_restore (flags); return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,@@ -405,7 +422,7 @@ static int tce_setrange_multi_pSeriesLP(unsigned long start_pfn, tcep =__this_cpu_read (tce_page);if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + tcep =alloc_tce_page ();if (!tcep) {local_irq_enable ();return -ENOMEM ;Signature · 1 line
Alexey
Signature · 1 line
Ram Pai
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 20 lines · Ram Pai
Quoted 11 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.Quoted 6 lines
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Quoted 11 lines
This happens even
on a normal VM when IOMMU is enabled.Quoted 2 lines
Also what is this for anyway?
This is for sending indirect-TCE entries to the hypervisor. The hypervisor must be able to read those TCE entries, so that it can use those entires to populate the TCE table with the correct mappings.
Quoted 7 lines
Quoted 5 lines
if I understand things right, you cannot
map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,Any random guest memory can be shared by the guest.
Yes but we do not want this to be this random. I thought the whole idea of swiotlb was to restrict the amount of shared memory to bare minimum, what do I miss?
Quoted 2 lines
Maybe you are confusing this with the SWIOTLB bounce buffers used by PCI devices, to transfer data to the hypervisor?
Is not this for pci+swiotlb? The cover letter suggests it is for
virtio-scsi-_pci_ with iommu_platform=on which makes it a normal pci
device just like emulated XHCI. Thanks,
Quoted 69 lines
Quoted 68 lines
Quoted 65 lines
Signed-off-by Ram Pai <[email protected]>--- arch/powerpc/platforms/pseries/iommu.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)arch/powerpc/platforms/pseries/iommu.c +20
-3@@ -37,6 +37,7 @@#include <asm/mmzone.h> #include <asm/plpar_wrappers.h> #include <asm/svm.h> +#include <asm/ultravisor.h> #include "pseries.h" @@ -179,6 +180,23 @@ static int tce_build_pSeriesLP(struct iommu_table *tbl, long tcenum, static DEFINE_PER_CPU(__be64 *, tce_page); +/* +* Allocate a tce page. If secure VM, share the page with the hypervisor. +* +* NOTE: the TCE page is shared with the hypervisor explicitly and remains +* shared for the lifetime of the kernel. It is implicitly unshared at kernel +* shutdown through a UV_UNSHARE_ALL_PAGES ucall. +*/ +static __be64 *alloc_tce_page (void ) +{ +__be64 *tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + +if (tcep &&is_secure_guest ()) +uv_share_page (PHYS_PFN (__pa (tcep)),1 ); + +return tcep; +} + static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, long npages, unsigned long uaddr, enum dma_data_direction direction,@@ -206,8 +224,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,* from iommu_alloc{,_sg}() */ if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); -/* If allocation fails, fall back to the loop implementation */ + tcep =alloc_tce_page ();if (!tcep) {local_irq_restore (flags); return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,@@ -405,7 +422,7 @@ static int tce_setrange_multi_pSeriesLP(unsigned long start_pfn, tcep =__this_cpu_read (tce_page);if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); + tcep =alloc_tce_page ();if (!tcep) {local_irq_enable ();return -ENOMEM ;Signature · 1 line
Alexey
Signature · 1 line
Alexey
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).
Quoted 4 lines
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Quoted 18 lines
Quoted 18 lines
This happens even
on a normal VM when IOMMU is enabled.Quoted 2 lines
Also what is this for anyway?
This is for sending indirect-TCE entries to the hypervisor. The hypervisor must be able to read those TCE entries, so that it can use those entires to populate the TCE table with the correct mappings.
Quoted 5 lines
if I understand things right, you cannot
map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,
Quoted 3 lines
Quoted 1 line
Any random guest memory can be shared by the guest.
Yes but we do not want this to be this random.
It is not sharing some random page. It is sharing a page that is ear-marked for communicating TCE entries. Yes the address of the page can be random, depending on where the allocator decides to allocate it. The purpose of the page is not random.
That page is used for one specific purpose; to communicate the TCE entries to the hypervisor.
Quoted 3 lines
I thought the whole idea
of swiotlb was to restrict the amount of shared memory to bare minimum, what do I miss?
I think, you are making a incorrect connection between this patch and SWIOTLB. This patch has nothing to do with SWIOTLB.
Quoted 5 lines
Quoted 3 lines
Maybe you are confusing this with the SWIOTLB bounce buffers used by PCI devices, to transfer data to the hypervisor?
Is not this for pci+swiotlb?
No. This patch is NOT for PCI+SWIOTLB. The SWIOTLB pages are a different set of pages allocated and earmarked for bounce buffering.
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Quoted 3 lines
The cover letter suggests it is for
virtio-scsi-_pci_ with iommu_platform=on which makes it a
normal pci device just like emulated XHCI. Thanks,
Well, I guess, the cover letter is probably confusing. There are two patches, which togather enable virtio on secure guests, in the presence of IOMMU.
The second patch enables virtio in the presence of a IOMMU, to use DMA_ops+SWIOTLB infrastructure, to correctly navigate the I/O to virtio devices.
However that by itself wont work if the TCE entires are not correctly setup in the TCE tables. The first patch; i.e this patch, helps accomplish that.
Hope this clears up the confusion.
RP
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 24 lines · Ram Pai
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
This happens even
on a normal VM when IOMMU is enabled.Quoted 2 lines
Also what is this for anyway?
This is for sending indirect-TCE entries to the hypervisor. The hypervisor must be able to read those TCE entries, so that it can use those entires to populate the TCE table with the correct mappings.
Quoted 5 lines
if I understand things right, you cannot
map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Any random guest memory can be shared by the guest.
Yes but we do not want this to be this random.
It is not sharing some random page. It is sharing a page that is ear-marked for communicating TCE entries. Yes the address of the page can be random, depending on where the allocator decides to allocate it. The purpose of the page is not random.
I was talking about the location.
Quoted 9 lines
That page is used for one specific purpose; to communicate the TCE entries to the hypervisor.
Quoted 3 lines
I thought the whole idea
of swiotlb was to restrict the amount of shared memory to bare minimum, what do I miss?I think, you are making a incorrect connection between this patch and SWIOTLB. This patch has nothing to do with SWIOTLB.
I can see this and this is the confusing part.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Maybe you are confusing this with the SWIOTLB bounce buffers used by PCI devices, to transfer data to the hypervisor?
Quoted 8 lines
Quoted 1 line
Is not this for pci+swiotlb?
No. This patch is NOT for PCI+SWIOTLB. The SWIOTLB pages are a different set of pages allocated and earmarked for bounce buffering.
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Then the hypervisor should be able to access the guest pages mapped for DMA and these pages should be made unsecure for this to work. Where/when does this happen?
Quoted 11 lines
Quoted 3 lines
The cover letter suggests it is for
virtio-scsi-_pci_ with iommu_platform=on which makes it a
normal pci device just like emulated XHCI. Thanks,Well, I guess, the cover letter is probably confusing. There are two patches, which togather enable virtio on secure guests, in the presence of IOMMU.
The second patch enables virtio in the presence of a IOMMU, to use DMA_ops+SWIOTLB infrastructure, to correctly navigate the I/O to virtio devices.
The second patch does nothing in relation to the problem being solved.
Quoted 4 lines
However that by itself wont work if the TCE entires are not correctly setup in the TCE tables. The first patch; i.e this patch, helps accomplish that.
Quoted 1 line
Hope this clears up the confusion.
Signature · 1 line
Alexey
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 30 lines · Alexey Kardashevskiy
Quoted 27 lines · Ram Pai
Quoted 26 lines · Alexey Kardashevskiy
Quoted 22 lines · Ram Pai
Quoted 11 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.Quoted 6 lines
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Quoted 56 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
This happens even
on a normal VM when IOMMU is enabled.Quoted 2 lines
Also what is this for anyway?
This is for sending indirect-TCE entries to the hypervisor. The hypervisor must be able to read those TCE entries, so that it can use those entires to populate the TCE table with the correct mappings.
Quoted 5 lines
if I understand things right, you cannot
map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Any random guest memory can be shared by the guest.
Yes but we do not want this to be this random.
It is not sharing some random page. It is sharing a page that is ear-marked for communicating TCE entries. Yes the address of the page can be random, depending on where the allocator decides to allocate it. The purpose of the page is not random.
I was talking about the location.
Quoted 9 lines
That page is used for one specific purpose; to communicate the TCE entries to the hypervisor.
Quoted 3 lines
I thought the whole idea
of swiotlb was to restrict the amount of shared memory to bare minimum, what do I miss?I think, you are making a incorrect connection between this patch and SWIOTLB. This patch has nothing to do with SWIOTLB.
I can see this and this is the confusing part.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
Maybe you are confusing this with the SWIOTLB bounce buffers used by PCI devices, to transfer data to the hypervisor?
Is not this for pci+swiotlb?
No. This patch is NOT for PCI+SWIOTLB. The SWIOTLB pages are a different set of pages allocated and earmarked for bounce buffering.
Quoted 6 lines
Quoted 2 lines
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Then the hypervisor should be able to access the guest pages mapped for DMA and these pages should be made unsecure for this to work. Where/when does this happen?
This happens in the SWIOTLB code. The code to do that is already upstream.
The sharing of the pages containing the SWIOTLB bounce buffers is done in init_svm() which calls swiotlb_update_mem_attributes() which calls set_memory_decrypted(). In the case of pseries, set_memory_decrypted() calls uv_share_page().
The code that bounces the contents of a I/O buffer through the SWIOTLB buffers, is in swiotlb_bounce().
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
The cover letter suggests it is for
virtio-scsi-_pci_ with iommu_platform=on which makes it a
normal pci device just like emulated XHCI. Thanks,Well, I guess, the cover letter is probably confusing. There are two patches, which togather enable virtio on secure guests, in the presence of IOMMU.
Quoted 5 lines
Quoted 3 lines
The second patch enables virtio in the presence of a IOMMU, to use DMA_ops+SWIOTLB infrastructure, to correctly navigate the I/O to virtio devices.
The second patch does nothing in relation to the problem being solved.
The second patch registers dma_iommu_ops with the PCI-system. Doing so enables I/O to take the dma_iommu_ops path, which internally leads it through the SWIOTLB path. Without that, the I/O fails to reach its destination.
Quoted 13 lines
Quoted 6 lines
However that by itself wont work if the TCE entires are not correctly setup in the TCE tables. The first patch; i.e this patch, helps accomplish that.
Quoted 1 line
Hope this clears up the confusion.
Signature · 1 line
Alexey
Signature · 1 line
Ram Pai
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 33 lines · Ram Pai
Quoted 32 lines · Alexey Kardashevskiy
Quoted 29 lines · Ram Pai
Quoted 27 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 11 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.Quoted 6 lines
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Via H_PUT_TCE which updates 1 entry at once (sets or clears). hcall-multi-tce enables H_PUT_TCE_INDIRECT (512 entries at once) and H_STUFF_TCE (clearing, up to 4bln at once? many), these are simply an optimization.
Quoted 64 lines
Quoted 57 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
This happens even
on a normal VM when IOMMU is enabled.Quoted 2 lines
Also what is this for anyway?
This is for sending indirect-TCE entries to the hypervisor. The hypervisor must be able to read those TCE entries, so that it can use those entires to populate the TCE table with the correct mappings.
Quoted 5 lines
if I understand things right, you cannot
map any random guest memory, you should only be mapping that 64MB-ish bounce buffer array but 1) I do not see that happening (I may have missed it) 2) it should be done once and it takes a little time for whatever memory size we allow for bounce buffers anyway. Thanks,Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Any random guest memory can be shared by the guest.
Yes but we do not want this to be this random.
It is not sharing some random page. It is sharing a page that is ear-marked for communicating TCE entries. Yes the address of the page can be random, depending on where the allocator decides to allocate it. The purpose of the page is not random.
I was talking about the location.
Quoted 9 lines
That page is used for one specific purpose; to communicate the TCE entries to the hypervisor.
Quoted 3 lines
I thought the whole idea
of swiotlb was to restrict the amount of shared memory to bare minimum, what do I miss?I think, you are making a incorrect connection between this patch and SWIOTLB. This patch has nothing to do with SWIOTLB.
I can see this and this is the confusing part.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
Maybe you are confusing this with the SWIOTLB bounce buffers used by PCI devices, to transfer data to the hypervisor?
Is not this for pci+swiotlb?
No. This patch is NOT for PCI+SWIOTLB. The SWIOTLB pages are a different set of pages allocated and earmarked for bounce buffering.
Quoted 6 lines
Quoted 2 lines
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Then the hypervisor should be able to access the guest pages mapped for DMA and these pages should be made unsecure for this to work. Where/when does this happen?
Quoted 7 lines
This happens in the SWIOTLB code. The code to do that is already upstream.
The sharing of the pages containing the SWIOTLB bounce buffers is done in
init_svm()which callsswiotlb_update_mem_attributes()which callsset_memory_decrypted(). In the case of pseries,set_memory_decrypted()callsuv_share_page().
This does not seem enough as when you enforce iommu_platform=on, QEMU starts accessing virtio buffers via IOMMU so bounce buffers have to be mapped explicitly, via H_PUT_TCE&co, where does this happen?
Quoted 18 lines
The code that bounces the contents of a I/O buffer through the SWIOTLB buffers, is in
swiotlb_bounce().Quoted 14 lines
Quoted 13 lines
Quoted 5 lines
The cover letter suggests it is for
virtio-scsi-_pci_ with iommu_platform=on which makes it a
normal pci device just like emulated XHCI. Thanks,Well, I guess, the cover letter is probably confusing. There are two patches, which togather enable virtio on secure guests, in the presence of IOMMU.
The second patch enables virtio in the presence of a IOMMU, to use DMA_ops+SWIOTLB infrastructure, to correctly navigate the I/O to virtio devices.
Quoted 6 lines
Quoted 1 line
The second patch does nothing in relation to the problem being solved.
The second patch registers
dma_iommu_opswith the PCI-system. Doing so enables I/O to take thedma_iommu_opspath, which internally leads it through the SWIOTLB path. Without that, the I/O fails to reach its destination.
This is not what the commit log says. What DMA ops was used before 2/2? I thought it was NULL which should have turned into direct which then would switch to swiotlb but since recent DMA reworks it is even harder to tell what happens with DMA setup. Thanks,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
However that by itself wont work if the TCE entires are not correctly setup in the TCE tables. The first patch; i.e this patch, helps accomplish that.
Quoted 1 line
Hope this clears up the confusion.
Signature · 1 line
Alexey
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 42 lines · Alexey Kardashevskiy
Quoted 39 lines · Ram Pai
Quoted 37 lines · Alexey Kardashevskiy
Quoted 24 lines · Ram Pai
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
Quoted 7 lines
Quoted 2 lines
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Via
H_PUT_TCEwhich updates 1 entry at once (sets or clears). hcall-multi-tce enablesH_PUT_TCE_INDIRECT(512 entries at once) andH_STUFF_TCE(clearing, up to 4bln at once? many), these are simply an optimization.
Do you still think, secure-VM should use H_PUT_TCE and not H_PUT_TCE_INDIRECT? And normal VM should use H_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs. In fact, we could make use of as much optimization as possible.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Is not this for pci+swiotlb?
..snip..
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Then the hypervisor should be able to access the guest pages mapped for DMA and these pages should be made unsecure for this to work. Where/when does this happen?
Quoted 7 lines
This happens in the SWIOTLB code. The code to do that is already upstream.
The sharing of the pages containing the SWIOTLB bounce buffers is done in
init_svm()which callsswiotlb_update_mem_attributes()which callsset_memory_decrypted(). In the case of pseries,set_memory_decrypted()callsuv_share_page().
Quoted 4 lines
This does not seem enough as when you enforce iommu_platform=on, QEMU starts accessing virtio buffers via IOMMU so bounce buffers have to be mapped explicitly, via H_PUT_TCE&co, where does this happen?
I think, it happens at boot time. Every page of the guest memory is TCE mapped, if iommu is enabled. SWIOTLB pages get implicitly TCE-mapped as part of that operation.
RP
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 46 lines · Ram Pai
Quoted 45 lines · Alexey Kardashevskiy
Quoted 41 lines · Ram Pai
Quoted 30 lines · Alexey Kardashevskiy
Quoted 27 lines · Ram Pai
Quoted 26 lines · Alexey Kardashevskiy
Quoted 22 lines · Ram Pai
Quoted 11 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.Quoted 6 lines
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Quoted 8 lines
Quoted 4 lines
Via
H_PUT_TCEwhich updates 1 entry at once (sets or clears). hcall-multi-tce enablesH_PUT_TCE_INDIRECT(512 entries at once) andH_STUFF_TCE(clearing, up to 4bln at once? many), these are simply an optimization.Do you still think, secure-VM should use
H_PUT_TCEand notH_PUT_TCE_INDIRECT? And normal VM should useH_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs.
Reducing the amount of insecure memory at random location.
Quoted 22 lines
In fact, we could make use of as much optimization as possible.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Is not this for pci+swiotlb?
..snip..
Quoted 16 lines
Quoted 14 lines
Quoted 6 lines
Quoted 2 lines
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Then the hypervisor should be able to access the guest pages mapped for DMA and these pages should be made unsecure for this to work. Where/when does this happen?
This happens in the SWIOTLB code. The code to do that is already upstream.
The sharing of the pages containing the SWIOTLB bounce buffers is done in
init_svm()which callsswiotlb_update_mem_attributes()which callsset_memory_decrypted(). In the case of pseries,set_memory_decrypted()callsuv_share_page().
Quoted 8 lines
Quoted 4 lines
This does not seem enough as when you enforce iommu_platform=on, QEMU starts accessing virtio buffers via IOMMU so bounce buffers have to be mapped explicitly, via H_PUT_TCE&co, where does this happen?
I think, it happens at boot time. Every page of the guest memory is TCE mapped, if iommu is enabled. SWIOTLB pages get implicitly TCE-mapped as part of that operation.
Ah I see. This works via the huge dma window. Ok, makes sense now.
It just seems like a waste that we could map swiotlb 1:1 via the always existing small DMA window but instead we rely on a huge window to map these small buffers. This way we are wasting the entire 32bit window and most of the huge window. We may fix it in the future (not right now) but for now I would still avoid unsecuring additional memory. Thanks,
Signature · 1 line
Alexey
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 54 lines · Alexey Kardashevskiy
Quoted 51 lines · Ram Pai
Quoted 49 lines · Alexey Kardashevskiy
Quoted 33 lines · Ram Pai
Quoted 32 lines · Alexey Kardashevskiy
Quoted 29 lines · Ram Pai
Quoted 27 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 11 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.Quoted 6 lines
This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Via
H_PUT_TCEwhich updates 1 entry at once (sets or clears). hcall-multi-tce enablesH_PUT_TCE_INDIRECT(512 entries at once) andH_STUFF_TCE(clearing, up to 4bln at once? many), these are simply an optimization.
Quoted 6 lines
Quoted 3 lines
Do you still think, secure-VM should use
H_PUT_TCEand notH_PUT_TCE_INDIRECT? And normal VM should useH_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs.Reducing the amount of insecure memory at random location.
The other approach we could use for that - which would still allow H_PUT_TCE_INDIRECT, would be to allocate the TCE buffer page from the same pool that we use for the bounce buffers. I assume there must already be some sort of allocator for that?
Quoted 41 lines
Quoted 22 lines
In fact, we could make use of as much optimization as possible.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Is not this for pci+swiotlb?
..snip..
Quoted 16 lines
Quoted 14 lines
Quoted 6 lines
Quoted 2 lines
This patch is purely to help the hypervisor setup the TCE table, in the presence of a IOMMU.
Then the hypervisor should be able to access the guest pages mapped for DMA and these pages should be made unsecure for this to work. Where/when does this happen?
This happens in the SWIOTLB code. The code to do that is already upstream.
The sharing of the pages containing the SWIOTLB bounce buffers is done in
init_svm()which callsswiotlb_update_mem_attributes()which callsset_memory_decrypted(). In the case of pseries,set_memory_decrypted()callsuv_share_page().Quoted 8 lines
Quoted 4 lines
This does not seem enough as when you enforce iommu_platform=on, QEMU starts accessing virtio buffers via IOMMU so bounce buffers have to be mapped explicitly, via H_PUT_TCE&co, where does this happen?
I think, it happens at boot time. Every page of the guest memory is TCE mapped, if iommu is enabled. SWIOTLB pages get implicitly TCE-mapped as part of that operation.
Ah I see. This works via the huge dma window. Ok, makes sense now.
It just seems like a waste that we could map swiotlb 1:1 via the always existing small DMA window but instead we rely on a huge window to map these small buffers. This way we are wasting the entire 32bit window and most of the huge window. We may fix it in the future (not right now) but for now I would still avoid unsecuring additional memory. Thanks,
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 60 lines · David Gibson
Quoted 59 lines · Alexey Kardashevskiy
Quoted 54 lines · Ram Pai
Quoted 42 lines · Alexey Kardashevskiy
Quoted 39 lines · Ram Pai
Quoted 37 lines · Alexey Kardashevskiy
Quoted 24 lines · Ram Pai
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
Quoted 7 lines
Quoted 2 lines
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Via
H_PUT_TCEwhich updates 1 entry at once (sets or clears). hcall-multi-tce enablesH_PUT_TCE_INDIRECT(512 entries at once) andH_STUFF_TCE(clearing, up to 4bln at once? many), these are simply an optimization.Do you still think, secure-VM should use
H_PUT_TCEand notH_PUT_TCE_INDIRECT? And normal VM should useH_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs.
Quoted 6 lines
Quoted 1 line
Reducing the amount of insecure memory at random location.
The other approach we could use for that - which would still allow
H_PUT_TCE_INDIRECT, would be to allocate the TCE buffer page from the same pool that we use for the bounce buffers. I assume there must already be some sort of allocator for that?
The allocator for swiotlb is buried deep in the swiotlb code. It is not exposed to the outside-swiotlb world. Will have to do major surgery to expose it.
I was thinking, maybe we share the page, finish the INDIRECT_TCE call, and unshare the page. This will address Alexey's concern of having shared pages at random location, and will also give me my performance optimization. Alexey: ok?
RP
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 68 lines · Ram Pai
Quoted 60 lines · David Gibson
Quoted 59 lines · Alexey Kardashevskiy
Quoted 54 lines · Ram Pai
Quoted 42 lines · Alexey Kardashevskiy
Quoted 39 lines · Ram Pai
Quoted 37 lines · Alexey Kardashevskiy
Quoted 24 lines · Ram Pai
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECThcall uses a page filled with TCE entries, as one of its parameters. One page is dedicated per cpu, for the lifetime of the kernel for this purpose. On secure VMs, contents of this page, when accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor needs to know the unencrypted entries, to update the TCE table accordingly. There is nothing secret or sensitive about these entries. Hence share the page with the hypervisor.This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
Quoted 7 lines
Quoted 2 lines
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Via
H_PUT_TCEwhich updates 1 entry at once (sets or clears). hcall-multi-tce enablesH_PUT_TCE_INDIRECT(512 entries at once) andH_STUFF_TCE(clearing, up to 4bln at once? many), these are simply an optimization.Do you still think, secure-VM should use
H_PUT_TCEand notH_PUT_TCE_INDIRECT? And normal VM should useH_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs.Quoted 6 lines
Quoted 1 line
Reducing the amount of insecure memory at random location.
The other approach we could use for that - which would still allow
H_PUT_TCE_INDIRECT, would be to allocate the TCE buffer page from the same pool that we use for the bounce buffers. I assume there must already be some sort of allocator for that?
Quoted 8 lines
The allocator for swiotlb is buried deep in the swiotlb code. It is not exposed to the outside-swiotlb world. Will have to do major surgery to expose it.
I was thinking, maybe we share the page, finish the
INDIRECT_TCEcall, and unshare the page. This will address Alexey's concern of having shared pages at random location, and will also give me my performance optimization. Alexey: ok?
I really do not see the point. I really think we should to 1:1 mapping of swtiotlb buffers using the default 32bit window using H_PUT_TCE and this should be more than enough, I do not think the amount of code will be dramatically different compared to unsecuring and securing a page or using one of swtiotlb pages for this purpose. Thanks,
Signature · 1 line
Alexey
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.y
Quoted 81 lines · Alexey Kardashevskiy
Quoted 68 lines · Ram Pai
Quoted 60 lines · David Gibson
Quoted 59 lines · Alexey Kardashevskiy
Quoted 54 lines · Ram Pai
Quoted 42 lines · Alexey Kardashevskiy
Quoted 39 lines · Ram Pai
Quoted 37 lines · Alexey Kardashevskiy
Quoted 24 lines · Ram Pai
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 7 lines · Ram Pai
H_PUT_TCE_INDIRECT hcall uses a page filled with TCE entries, as one of
its parameters. One page is dedicated per cpu, for the lifetime of the
kernel for this purpose. On secure VMs, contents of this page, when
accessed by the hypervisor, retrieves encrypted TCE entries. Hypervisor
needs to know the unencrypted entries, to update the TCE table
accordingly. There is nothing secret or sensitive about these entries.
Hence share the page with the hypervisor.This unsecures a page in the guest in a random place which creates an additional attack surface which is hard to exploit indeed but nevertheless it is there.
A safer option would be not to use the
hcall-multi-tce hyperrtas option (which translatesFW_FEATURE_MULTITCEin the guest).Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hmm... How do we not use it? AFAICT hcall-multi-tce option gets invoked automatically when IOMMU option is enabled.
It is advertised by QEMU but the guest does not have to use it.
Are you suggesting that even normal-guest, not use hcall-multi-tce? or just secure-guest?
Just secure.
Quoted 7 lines
Quoted 2 lines
hmm.. how are the TCE entries communicated to the hypervisor, if hcall-multi-tce is disabled?
Via
H_PUT_TCEwhich updates 1 entry at once (sets or clears). hcall-multi-tce enablesH_PUT_TCE_INDIRECT(512 entries at once) andH_STUFF_TCE(clearing, up to 4bln at once? many), these are simply an optimization.Do you still think, secure-VM should use
H_PUT_TCEand notH_PUT_TCE_INDIRECT? And normal VM should useH_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs.Quoted 6 lines
Quoted 1 line
Reducing the amount of insecure memory at random location.
The other approach we could use for that - which would still allow
H_PUT_TCE_INDIRECT, would be to allocate the TCE buffer page from the same pool that we use for the bounce buffers. I assume there must already be some sort of allocator for that?Quoted 8 lines
The allocator for swiotlb is buried deep in the swiotlb code. It is not exposed to the outside-swiotlb world. Will have to do major surgery to expose it.
I was thinking, maybe we share the page, finish the
INDIRECT_TCEcall, and unshare the page. This will address Alexey's concern of having shared pages at random location, and will also give me my performance optimization. Alexey: ok?
Quoted 5 lines
I really do not see the point. I really think we should to 1:1 mapping of swtiotlb buffers using the default 32bit window using
H_PUT_TCEand this should be more than enough, I do not think the amount of code will be dramatically different compared to unsecuring and securing a page or using one of swtiotlb pages for this purpose. Thanks,
Ok. I will address your major concern -- "do not create new shared pages at random location" in my next version of the patch. Using the 32bit DMA window just to map the SWIOTLB buffers, will be some effort. Hope we can stage it that way.
RP
RE: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor
In reply to Alexey KardashevskiyQuoted 3 lines · Alexey Kardashevskiy
On 05/12/2019 07:42, Ram Pai wrote:
.snip...
Quoted 22 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Do you still think, secure-VM should use
H_PUT_TCEand notH_PUT_TCE_INDIRECT? And normal VM should useH_PUT_TCE_INDIRECT? Is there any advantage of special casing it for secure-VMs.Reducing the amount of insecure memory at random location.
The other approach we could use for that - which would still allow
H_PUT_TCE_INDIRECT, would be to allocate the TCE buffer page from the same pool that we use for the bounce buffers. I assume there must already be some sort of allocator for that?Quoted 8 lines
The allocator for swiotlb is buried deep in the swiotlb code. It is not exposed to the outside-swiotlb world. Will have to do major surgery to expose it.
I was thinking, maybe we share the page, finish the
INDIRECT_TCEcall, and unshare the page. This will address Alexey's concern of having shared pages at random location, and will also give me my performance optimization. Alexey: ok?
Quoted 5 lines
I really do not see the point. I really think we should to 1:1 mapping of swtiotlb buffers using the default 32bit window using
H_PUT_TCEand this should be more than enough, I do not think the amount of code will be dramatically different compared to unsecuring and securing a page or using one of swtiotlb pages for this purpose. Thanks,
Ok. there are three different issues to be addressed here.
(a) How to map the TCE entries in the TCE table? Should we use H_PUT_TCE, or H_PUT_INDIRECT_TCE?
(b) How much of the guest memory must be mapped in the TCE table? Should it be the entire guest memory? or the memory used by the SWIOTLB?
(c) What mapping window must to be used? Is it the 64bit ddw? or the default 32 bit ddw?
Regardless of how we resolve issue (b) and (c), we still need to resolve (a). The main concern you have about solution (a) is that, random pages are permanently shared, something that can be exploited and can cause security issues. I tend to agree, this is possible, though I am not sure how. But yes we need to address this concern, since security is paramount to Secure Virtual Machines.
The way to resolve (a) is --
(i) grab a page from the SWIOTLB pool and use H_PUT_INDIRECT_TCE OR
(ii) simply use H_PUT_TCE.
OR
(iii) share the page prior to H_PUT_INDIRECT_TCE, and
unshare the page once done.
Solution (i) has layering violation; as Christoph alluded to in his previous reply. The swiotlb buffers are meant for I/O and DMA related actitivy. We will be abusing these swiotlb pages to communicate TCE entries to the hypervisor. Secondly IOMMU code has no idea where its pages are sourced from and should not know either. I am uncomfortable going this route. There is some upstream discussion about having a seperate pool of shared pages on secure VM, https://lkml.org/lkml/2019/11/14/381.
That solution; when ready, may be exploitable here.
I have coded solution (ii) and it works. But boot path slows down significantly. Huge amount H_PUT_TCE hcalls. Very hurtful.
I strongly think, solution (iii) is the right way to go. I have coded it, it works and bootpath is much faster. However I am not sure if you have a concern with this solution.
In any case, I am sending my next version of the patch based on solution (iii).
Once this is addressed, I will address (b) and (c).
RP
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
In reply to Ram PaiQuoted 8 lines · Ram Pai
Quoted 4 lines
The other approach we could use for that - which would still allow
H_PUT_TCE_INDIRECT, would be to allocate the TCE buffer page from the same pool that we use for the bounce buffers. I assume there must already be some sort of allocator for that?The allocator for swiotlb is buried deep in the swiotlb code. It is not exposed to the outside-swiotlb world. Will have to do major surgery to expose it.
I don't think it would require all that much changes, but I'd really hate the layering of calling into it directly. Do we have a struct device associated with the iommu that doesn't get iommu translations themselves? If we do a dma_alloc_coherent on that you'll get the memory pool for free.
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
In reply to Ram Paiarch/powerpc/platforms/pseries/iommu.c static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, · line 230
Quoted 10 lines · Ram Pai
arch/powerpc/platforms/pseries/iommu.c +1
-2@@ -206,8 +224,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,* from iommu_alloc{,_sg}() */ if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); -/* If allocation fails, fall back to the loop implementation */ + tcep =alloc_tce_page ();if (!tcep) {local_irq_restore (flags); return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,
The comment about failing allocation was removed, but I see no chage of behaviour here.
Can you please explain what/where it changes?
Best regards,
Leonardo
Re: [PATCH v4 1/2] powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor.
Quoted 15 lines · Leonardo Bras
arch/powerpc/platforms/pseries/iommu.c static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, · line 230
Quoted 10 lines · Ram Pai
arch/powerpc/platforms/pseries/iommu.c +1
-2@@ -206,8 +224,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,* from iommu_alloc{,_sg}() */ if (!tcep) { - tcep = (__be64 *)__get_free_page (GFP_ATOMIC ); -/* If allocation fails, fall back to the loop implementation */ + tcep =alloc_tce_page ();if (!tcep) {local_irq_restore (flags); return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,The comment about failing allocation was removed, but I see no chage of behaviour here.
Quoted 1 line
Can you please explain what/where it changes?
You observed it right. The comment should stay put. Will have it fixed in my next version.
Thanks,
RP