Clear shared pages after private-to-shared conversion
Subsystems (5)
v2 cover Clear shared pages after private-to-shared conversion
set_memory_decrypted() doesn't (currently) guarantee to preserve or zero memory, but both the GICv3 ITS driver and the system_cc_shared dma-buf heap currently allocate memory with __GFP_ZERO followed by calling set_memory_decrypted(). On an Arm CCA system with MEC this can cause ciphertext to be visible to the guest rather than the expected zeros.
Patches 1 and 3 fix this by zeroing after the set_memory_decrypted() call.
Patches 2 and 4 fix other related bugs that Sashiko found. Patch 2 fixes the issue that set_memory_decrypted() can be a sleeping call, so moves the allocation out of an atomic context.
Patch 4 deals with the situation where set_memory_decrypted() fails and the rollback path could attempt to re-encrypt memory which was never decrypted.
I've sorted the patches by area, but there's no (semantic) dependency between them.
Changes in v2:
* Switched to use BIT(n) rather than 1 << n in the GICv3 change.
* Added Jason's R-b.
* Patches 2 and 4 are new.
v1: https://lore.kernel.org/r/[email protected]
Steven Price (4):
irqchip/gic-v3-its: Zero shared pages after conversion irqchip/gic-v3-its: Allocate VPE tables from sleepable context
dma-buf: heaps: Zero system shared heap pages after conversion
dma-buf: heaps: Fix shared system heap allocation rollback
drivers/dma-buf/heaps/system_heap.c | 24 +++++++++++++----- drivers/irqchip/irq-gic-v3-its.c | 39 ++++++++++++++++------------- 2 files changed, 40 insertions(+), 23 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/4 irqchip/gic-v3-its: Zero shared pages after conversion
its_alloc_pages_node() passes __GFP_ZERO to the page allocator before calling set_memory_decrypted(). This assumes that converting a page from private to shared preserves its contents.
For Arm CCA with MEC (Memory Encryption Contexts) the key used to access the page will change, and so by default the visible data will change. The host could ensure that it zeros the page, but rather than relying on the host's behaviour it's best if the guest simply zeros after the decryption rather than before. Specifically in this case the ITS tables are required to be zeroed.
Mask out __GFP_ZERO from the allocation request, and do the zeroing as a separate step after decryption.
--- drivers/irqchip/irq-gic-v3-its.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +7 -2
@@ -213,16 +213,18 @@ static gfp_t gfp_flags_quirk;static struct page *its_alloc_pages_node (int node,gfp_t gfp,unsigned int order) { +bool want_zero = gfp & __GFP_ZERO;struct page *page;int ret =0 ; - page =alloc_pages_node (node, gfp | gfp_flags_quirk, order); + page =alloc_pages_node (node, (gfp & ~__GFP_ZERO) | gfp_flags_quirk, + order);if (!page)return NULL ; ret =set_memory_decrypted ((unsigned long )page_address (page), -1 << order); +BIT (order));/* * If set_memory_decrypted() fails then we don't know what state the * page is in, so we can't free it. Instead we leak it.
@@ -231,6 +233,9 @@ static struct page *its_alloc_pages_node(int node, gfp_t gfp,if (ret)return NULL ; +if (want_zero) +clear_pages (page_address (page),BIT (order)); +return page; }
Signature · 1 line
2.43.0
Re: [PATCH v2 1/4] irqchip/gic-v3-its: Zero shared pages after conversion
Quoted 5 lines · Steven Price
drivers/irqchip/irq-gic-v3-its.c +0
-0
drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 221
Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +3
-1@@ -213,16 +213,18 @@ static gfp_t gfp_flags_quirk;static struct page *its_alloc_pages_node (int node,gfp_t gfp,unsigned int order) { +bool want_zero = gfp & __GFP_ZERO;struct page *page;int ret =0 ; - page =alloc_pages_node (node, gfp | gfp_flags_quirk, order); + page =alloc_pages_node (node, (gfp & ~__GFP_ZERO) | gfp_flags_quirk, + order);
I don't think pKVM does any scrubbing on set_memory_decrypted(), so it potentially exposes confidential guest data before it reaches clear_pages() below.
Quoted 22 lines
drivers/irqchip/irq-gic-v3-its.c +4
-1if (!page)return NULL ; ret =set_memory_decrypted ((unsigned long )page_address (page), -1 << order); +BIT (order));/* * If set_memory_decrypted() fails then we don't know what state the * page is in, so we can't free it. Instead we leak it. @@ -231,6 +233,9 @@ static struct page *its_alloc_pages_node(int node, gfp_t gfp,if (ret)return NULL ; +if (want_zero) +clear_pages (page_address (page),BIT (order)); +return page; }Signature · 1 line
2.43.0
Signature · 1 line
Catalin
Re: [PATCH v2 1/4] irqchip/gic-v3-its: Zero shared pages after conversion
Quoted 13 lines · Catalin Marinas
Quoted 12 lines · Steven Price
drivers/irqchip/irq-gic-v3-its.c +1
-0@@ -213,16 +213,18 @@ static gfp_t gfp_flags_quirk;static struct page *its_alloc_pages_node (int node,gfp_t gfp,unsigned int order) { +bool want_zero = gfp & __GFP_ZERO;struct page *page;int ret =0 ;
Quoted 7 lines
drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 221
Quoted 3 lines
drivers/irqchip/irq-gic-v3-its.c +2
-1- page =alloc_pages_node (node, gfp | gfp_flags_quirk, order); + page =alloc_pages_node (node, (gfp & ~__GFP_ZERO) | gfp_flags_quirk, + order);I don't think pKVM does any scrubbing on
set_memory_decrypted(), so it potentially exposes confidential guest data before it reachesclear_pages()below.
IMHO that has got to be handled in the arch code implementing set memory decrypted.
Further pointing that maybe we should have an alloc decrypted so we can at least try to minimize the number of times we write to this memory. :(
Jason
Re: [PATCH v2 1/4] irqchip/gic-v3-its: Zero shared pages after conversion
Quoted 22 lines · Jason Gunthorpe
Quoted 13 lines · Catalin Marinas
Quoted 12 lines · Steven Price
drivers/irqchip/irq-gic-v3-its.c +1
-0@@ -213,16 +213,18 @@ static gfp_t gfp_flags_quirk;static struct page *its_alloc_pages_node (int node,gfp_t gfp,unsigned int order) { +bool want_zero = gfp & __GFP_ZERO;struct page *page;int ret =0 ;Quoted 7 lines
drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 221
Quoted 3 lines
drivers/irqchip/irq-gic-v3-its.c +2
-1- page =alloc_pages_node (node, gfp | gfp_flags_quirk, order); + page =alloc_pages_node (node, (gfp & ~__GFP_ZERO) | gfp_flags_quirk, + order);I don't think pKVM does any scrubbing on
set_memory_decrypted(), so it potentially exposes confidential guest data before it reachesclear_pages()below.
Quoted 2 lines
IMHO that has got to be handled in the arch code implementing set memory decrypted.
I guess that's a better separation. It probably needs to clear the MTE tags as well, it's not great to leak them (though not as bad as leaking data).
OTOH, pKVM would no longer need the clear_pages() afterwards since there's no encryption key changed. Not too bad, it's not a hot path.
Quoted 3 lines
Further pointing that maybe we should have an alloc decrypted so we can at least try to minimize the number of times we write to this memory. :(
This would be better.
Signature · 1 line
Catalin
PATCH v2 2/4 irqchip/gic-v3-its: Allocate VPE tables from sleepable context
In reply to Steven PriceThe VPE L1 table is allocated from the CPU-starting hotplug state, where interrupts are disabled. Although the page allocation uses GFP_ATOMIC, its_alloc_pages() subsequently calls set_memory_decrypted(), which can sleep while splitting the arm64 linear map.
Move the allocation to the existing CPU-online callback, which runs in sleepable context, and use GFP_KERNEL for both allocations performed there. Register the callback even without EFI, since it is now also responsible for VPE table allocation.
--- drivers/irqchip/irq-gic-v3-its.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +15 -15
@@ -2932,7 +2932,7 @@ static int allocate_vpe_l1_table(void)if (val &GICR_VPROPBASER_4_1_VALID ) goto out; -gic_data_rdist ()->vpe_table_mask =kzalloc_obj (cpumask_t,GFP_ATOMIC ); +gic_data_rdist ()->vpe_table_mask =kzalloc_obj (cpumask_t,GFP_KERNEL );if (!gic_data_rdist ()->vpe_table_mask)return -ENOMEM ;
@@ -2999,7 +2999,7 @@ static int allocate_vpe_l1_table(void)pr_debug ("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n" , np, npg, psz, epp, esz); - page =its_alloc_pages (GFP_ATOMIC | __GFP_ZERO,get_order (np *PAGE_SIZE )); + page =its_alloc_pages (GFP_KERNEL | __GFP_ZERO,get_order (np *PAGE_SIZE ));if (!page)return -ENOMEM ;
@@ -3268,16 +3268,6 @@ static void its_cpu_init_lpis(void) val =its_clear_vpend_valid (vlpi_base,0 ,0 ); } -if (allocate_vpe_l1_table ()) { -/* -* If the allocation has failed, we're in massive trouble. -* Disable direct injection, and pray that no VM was -* already running... -*/ - gic_rdists->has_rvpeid = false; - gic_rdists->has_vlpis = false; - } -/* Make sure the GIC has seen the above */ dsb (sy);gic_data_rdist ()->flags |=RD_LOCAL_LPI_ENABLED ;
@@ -5452,6 +5442,19 @@ static int its_cpu_memreserve_lpi(unsigned int cpu)if (gic_data_rdist ()->flags &RD_LOCAL_MEMRESERVE_DONE )return 0 ; +if (allocate_vpe_l1_table ()) { +/* +* If the allocation has failed, we're in massive trouble. +* Disable direct injection, and pray that no VM was +* already running... +*/ + gic_rdists->has_rvpeid = false; + gic_rdists->has_vlpis = false; + } + +if (!efi_enabled (EFI_CONFIG_TABLES )) + goto out; + pend_page =gic_data_rdist ()->pend_page;if (WARN_ON (!pend_page)) { ret = -ENOMEM ;
@@ -5793,9 +5796,6 @@ int __init its_lpi_memreserve_init(void) {int state; -if (!efi_enabled (EFI_CONFIG_TABLES )) -return 0 ; -if (list_empty (&its_nodes))return 0 ;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v2 2/4] irqchip/gic-v3-its: Allocate VPE tables from sleepable context
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Moving VPE L1 table allocation to a late CPU hotplug state exposes uninitialized GICv4.1…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Moving VPE L1 table allocation to a late CPU hotplug state exposes uninitialized GICv4.1 redistributor state to concurrent VPE mappings. --
--- Patch [2]: [PATCH v2 2/4] irqchip/gic-v3-its: Allocate VPE tables from sleepable context --- commit 7ed963cf27a623f2c21ad61ef0162e3eddac65ae
Author: Steven Price <[email protected]>
irqchip/gic-v3-its: Allocate VPE tables from sleepable context
This patch moves the VPE L1 table allocation from the early CPU-starting hotplug state to the later CPU-online callback. This permits the use of GFP_KERNEL allocations, as its_alloc_pages() can sleep. The CPU-online callback is now registered unconditionally to handle this setup.
Quoted 4 lines
drivers/irqchip/irq-gic-v3-its.c +0
-0
[ ... ]
Quoted 17 lines
drivers/irqchip/irq-gic-v3-its.c +0
-10@@ -3268,16 +3268,6 @@ static void its_cpu_init_lpis(void) val =its_clear_vpend_valid (vlpi_base,0 ,0 ); } -if (allocate_vpe_l1_table ()) { -/* -* If the allocation has failed, we're in massive trouble. -* Disable direct injection, and pray that no VM was -* already running... -*/ - gic_rdists->has_rvpeid = false; - gic_rdists->has_vlpis = false; - } -/* Make sure the GIC has seen the above */ dsb (sy);gic_data_rdist ()->flags |=RD_LOCAL_LPI_ENABLED ;
[ ... ]
Quoted 4 lines
drivers/irqchip/irq-gic-v3-its.c +0
-0@@ -5454,6 +5444,19 @@ static int its_cpu_memreserve_lpi(unsigned int cpu)if (gic_data_rdist ()->flags &RD_LOCAL_MEMRESERVE_DONE )return 0 ;
drivers/irqchip/irq-gic-v3-its.c static int its_cpu_memreserve_lpi(unsigned int cpu) · line 5456
Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +10
-0+if (allocate_vpe_l1_table ()) { +/* +* If the allocation has failed, we're in massive trouble. +* Disable direct injection, and pray that no VM was +* already running... +*/ + gic_rdists->has_rvpeid = false; + gic_rdists->has_vlpis = false; + } +
[Severity: High]
Does moving this allocation to the CPUHP_AP_ONLINE_DYN callback expose uninitialized GICv4.1 redistributor state to concurrent VPE mappings?
The CPUHP_AP_ONLINE_DYN state runs for its_cpu_memreserve_lpi() in the hotplug thread asynchronously after the CPU has already been marked online in cpu_online_mask.
Could a concurrent process changing IRQ affinities (like irqbalance) observe the CPU in cpu_online_mask and route a virtual interrupt to it via its_vpe_set_affinity() before the hotplug thread has executed this table allocation?
If a VMAPP command executes before allocate_vpe_l1_table() programs the GICR_VPROPBASER, it appears the hardware could use stale or uninitialized physical addresses, leading to unpredictable behavior or dropped interrupts.
Quoted 3 lines
drivers/irqchip/irq-gic-v3-its.c +0
-0if (!efi_enabled (EFI_CONFIG_TABLES )) goto out;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH v2 2/4] irqchip/gic-v3-its: Allocate VPE tables from sleepable context
In reply to Steven PriceQuoted 10 lines · Steven Price
The VPE L1 table is allocated from the CPU-starting hotplug state, where interrupts are disabled. Although the page allocation uses
GFP_ATOMIC,its_alloc_pages()subsequently callsset_memory_decrypted(), which can sleep while splitting the arm64 linear map.Move the allocation to the existing CPU-online callback, which runs in sleepable context, and use
GFP_KERNELfor both allocations performed there. Register the callback even without EFI, since it is now also responsible for VPE table allocation.
Ok, Sashiko pointed out this is bunk:
Quoted 7 lines
Does moving this allocation to the
CPUHP_AP_ONLINE_DYNcallback expose uninitialized GICv4.1 redistributor state to concurrent VPE mappings?The
CPUHP_AP_ONLINE_DYNstate runs forits_cpu_memreserve_lpi()in the hotplug thread asynchronously after the CPU has already been marked online incpu_online_mask.
Quoted 8 lines
Could a concurrent process changing IRQ affinities (like irqbalance) observe the CPU in
cpu_online_maskand route a virtual interrupt to it viaits_vpe_set_affinity()before the hotplug thread has executed this table allocation?If a VMAPP command executes before
allocate_vpe_l1_table()programs theGICR_VPROPBASER, it appears the hardware could use stale or uninitialized physical addresses, leading to unpredictable behavior or dropped interrupts.
Please ignore this patch. I thought this was a bit too simple to fix :( I guess some sort of preallocation might be the solution.
Thanks,
Steve
Quoted 75 lines
Fixes b08e2f42e86b ("irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor")Signed-off-by Steven Price <[email protected]>--- drivers/irqchip/irq-gic-v3-its.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)drivers/irqchip/irq-gic-v3-its.c +15
-15@@ -2932,7 +2932,7 @@ static int allocate_vpe_l1_table(void)if (val &GICR_VPROPBASER_4_1_VALID ) goto out; -gic_data_rdist ()->vpe_table_mask =kzalloc_obj (cpumask_t,GFP_ATOMIC ); +gic_data_rdist ()->vpe_table_mask =kzalloc_obj (cpumask_t,GFP_KERNEL );if (!gic_data_rdist ()->vpe_table_mask)return -ENOMEM ;@@ -2999,7 +2999,7 @@ static int allocate_vpe_l1_table(void)pr_debug ("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n" , np, npg, psz, epp, esz); - page =its_alloc_pages (GFP_ATOMIC | __GFP_ZERO,get_order (np *PAGE_SIZE )); + page =its_alloc_pages (GFP_KERNEL | __GFP_ZERO,get_order (np *PAGE_SIZE ));if (!page)return -ENOMEM ;@@ -3268,16 +3268,6 @@ static void its_cpu_init_lpis(void) val =its_clear_vpend_valid (vlpi_base,0 ,0 ); } -if (allocate_vpe_l1_table ()) { -/* -* If the allocation has failed, we're in massive trouble. -* Disable direct injection, and pray that no VM was -* already running... -*/ - gic_rdists->has_rvpeid = false; - gic_rdists->has_vlpis = false; - } -/* Make sure the GIC has seen the above */ dsb (sy);gic_data_rdist ()->flags |=RD_LOCAL_LPI_ENABLED ;@@ -5452,6 +5442,19 @@ static int its_cpu_memreserve_lpi(unsigned int cpu)if (gic_data_rdist ()->flags &RD_LOCAL_MEMRESERVE_DONE )return 0 ; +if (allocate_vpe_l1_table ()) { +/* +* If the allocation has failed, we're in massive trouble. +* Disable direct injection, and pray that no VM was +* already running... +*/ + gic_rdists->has_rvpeid = false; + gic_rdists->has_vlpis = false; + } + +if (!efi_enabled (EFI_CONFIG_TABLES )) + goto out; + pend_page =gic_data_rdist ()->pend_page;if (WARN_ON (!pend_page)) { ret = -ENOMEM ;@@ -5793,9 +5796,6 @@ int __init its_lpi_memreserve_init(void) {int state; -if (!efi_enabled (EFI_CONFIG_TABLES )) -return 0 ; -if (list_empty (&its_nodes))return 0 ;
PATCH v2 3/4 dma-buf: heaps: Zero system shared heap pages after conversion
In reply to Steven PriceThe system_cc_shared heap allocates pages with __GFP_ZERO before converting them from private to shared with set_memory_decrypted(). This assumes that the conversion preserves the contents of the pages.
For Arm CCA with MEC (Memory Encryption Contexts) the key used to access the page will change, and so by default the visible data will change. The host could ensure that it zeros the page after decryption, but rather than relying on the host's behaviour it's best if the guest simply zeros after the decryption rather than before.
For CC shared buffers, defer zeroing until each page has been converted successfully. For other buffers keep the existing behaviour.
--- drivers/dma-buf/heaps/system_heap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/dma-buf/heaps/system_heap.c +11 -3
@@ -376,7 +376,8 @@ static const struct dma_buf_ops system_heap_buf_ops = { };static struct page *alloc_largest_available (unsigned long size, -unsigned int max_order) +unsigned int max_order, +bool defer_zero) {struct page *page;int i;
@@ -388,6 +389,9 @@ static struct page *alloc_largest_available(unsigned long size,if (max_order < orders[i])continue ; flags = order_flags[i]; +/* Decryption can change the contents, so clear it afterwards. */ +if (defer_zero) + flags &= ~__GFP_ZERO;if (mem_accounting) flags |= __GFP_ACCOUNT; page =alloc_pages (flags, orders[i]);
@@ -438,7 +442,8 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, goto free_buffer; } - page =alloc_largest_available (size_remaining, max_order); + page =alloc_largest_available (size_remaining, max_order, +cc_shared_buffer (buffer));if (!page) goto free_buffer;
@@ -461,9 +466,12 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,if (cc_shared_buffer (buffer)) {for_each_sgtable_sg (table, sg, i) { - ret =system_heap_set_page_decrypted (sg_page (sg)); + page =sg_page (sg); + ret =system_heap_set_page_decrypted (page);if (ret) goto free_pages; + +clear_pages (page_address (page),1 <<compound_order (page)); } }
Signature · 1 line
2.43.0
PATCH v2 4/4 dma-buf: heaps: Fix shared system heap allocation rollback
In reply to Steven PriceIf converting one of the allocated pages to shared memory fails, the cleanup path attempts to convert every page back to private memory. Pages after the failed page have not been converted yet, so attempting to convert them back can fail and cause otherwise reusable memory to be leaked.
Count the pages converted successfully and only convert those and the failed allocation back during cleanup. If converting the failed allocation back succeeds it can be freed safely; otherwise it is leaked because its state is unknown. Allocations that were not converted can be freed directly.
--- drivers/dma-buf/heaps/system_heap.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/dma-buf/heaps/system_heap.c +7 -3
@@ -418,6 +418,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,struct scatterlist *sg;struct list_head pages;struct page *page, *tmp_page; +int nr_decrypted =0 ;int i, ret = -ENOMEM ; buffer =kzalloc_obj (*buffer);
@@ -472,6 +473,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, goto free_pages;clear_pages (page_address (page),1 <<compound_order (page)); + nr_decrypted++; } }
@@ -496,9 +498,11 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,* Intentionally leak pages that cannot be re-encrypted * to prevent shared memory from being reused. */ -if (cc_shared_buffer (buffer) && -system_heap_set_page_encrypted (p)) -continue ; +if (cc_shared_buffer (buffer)) { +if (i <= nr_decrypted && +system_heap_set_page_encrypted (p)) +continue ; + }__free_pages (p,compound_order (p)); }sg_free_table (table);
Signature · 1 line
2.43.0
Re: [PATCH v2 4/4] dma-buf: heaps: Fix shared system heap allocation rollback
Quoted 12 lines · Steven Price
If converting one of the allocated pages to shared memory fails, the cleanup path attempts to convert every page back to private memory. Pages after the failed page have not been converted yet, so attempting to convert them back can fail and cause otherwise reusable memory to be leaked.
Count the pages converted successfully and only convert those and the failed allocation back during cleanup. If converting the failed allocation back succeeds it can be freed safely; otherwise it is leaked because its state is unknown. Allocations that were not converted can be freed directly.
Quoted 5 lines
Fixes 78b30c50a7ac ("dma-buf: heaps: system: add system_cc_shared heap for explicitly shared memory")Signed-off-by Steven Price <[email protected]>--- drivers/dma-buf/heaps/system_heap.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
Jason