KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
v9 cover KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
s390 needs this series of three patches in order to enable a non-blocking path for irqfd injection on s390 via kvm_arch_set_irq_inatomic(). Before these changes, kvm_arch_set_irq_inatomic() would just return -EWOULDBLOCK and place all interrupts on the global work queue, which must subsequently be processed by a different thread. This series of patches implements an s390 version of inatomic and is relevant to virtio-blk and virtio-net and was tested against virtio-pci and virtio-ccw.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
s390 doesn't support a PREEMPT_RT kernel and this patch doesn't either. Given this fact, we are not using raw_spin_lock instead we are using regular spin_lock.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_no_inject. And
counters have been added to analyze map/unmap of the adapter_indicator pages in non-Secure Execution environments and to track fencing of Fast Inject in Secure Execution environments. In order to take advantage of this kernel series with virtio-pci, a QEMU that includes the 's390x/pci: set kvm_msi_via_irqfd_allowed' fix is needed. Additionally, the guest xml needs a thread pool and threads explicitly assigned per disk device using the common way of defining threads for disks.
Patch 1 enables map/unmap of adapter indicator pages but for Secure Execution environments it avoids the long term mapping.
v8->v9: Release mutex in register_io_adapter() in exit conditions v8->v9: Rename masked, coalesced, suppressed to io_390_inatomic_no_inject
Douglas Freimuth (3):
KVM: s390: Add map/unmap ioctl and clean mappings post-guest
KVM: s390: Enable adapter_indicators_set to use mapped pages
KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
arch/s390/include/asm/kvm_host.h | 11 +- arch/s390/kvm/intercept.c | 5 +- arch/s390/kvm/interrupt.c | 558 ++++++++++++++++++++++++------- arch/s390/kvm/kvm-s390.c | 30 +- arch/s390/kvm/kvm-s390.h | 5 +- 5 files changed, 489 insertions(+), 120 deletions(-)
Signature · 1 line
2.54.0
PATCH v9 1/3 KVM: s390: Add map/unmap ioctl and clean mappings post-guest
s390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed. pin_user_pages_remote is used for both the ioctl as well as the pin-on-demand logic in adapter_indicators_set().
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
--- arch/s390/include/asm/kvm_host.h | 5 + arch/s390/kvm/interrupt.c | 226 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 3 + arch/s390/kvm/kvm-s390.h | 2 + 4 files changed, 195 insertions(+), 41 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -0
@@ -448,6 +448,8 @@ struct kvm_vcpu_arch {struct kvm_vm_stat {struct kvm_vm_stat_generic generic;u64 inject_io; +u64 io_390_adapter_map; +u64 io_390_adapter_unmap;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
@@ -479,6 +481,9 @@ struct s390_io_adapter {bool masked;bool swap;bool suppressible; +spinlock_t maps_lock; +struct list_head maps; +unsigned int nr_maps; };#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)
arch/s390/kvm/interrupt.c +185 -41
287 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +3 -0
@@ -68,6 +68,8 @@const struct kvm_stats_desc kvm_vm_stats_desc[] = {KVM_GENERIC_VM_STATS (),STATS_DESC_COUNTER (VM , inject_io), +STATS_DESC_COUNTER (VM , io_390_adapter_map), +STATS_DESC_COUNTER (VM , io_390_adapter_unmap),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done),STATS_DESC_COUNTER (VM , inject_service_signal),
@@ -2513,6 +2515,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)if (kvm_s390_pv_is_protected (kvm))break ; +kvm_s390_unmap_all_adapters (kvm);mmap_write_lock (kvm->mm);/* * Disable creation of new THPs. Existing THPs can stay, they
arch/s390/kvm/kvm-s390.h +2 -0
@@ -560,6 +560,8 @@ void kvm_s390_gisa_disable(struct kvm *kvm);void kvm_s390_gisa_enable (struct kvm *kvm);int __init kvm_s390_gib_init (u8 nisc);void kvm_s390_gib_destroy (void ); +void kvm_s390_unmap_all_adapters (struct kvm *kvm); +/* implemented in guestdbg.c */ void kvm_s390_backup_guest_per_regs (struct kvm_vcpu *vcpu);
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v9 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] `copy_from_user` is called while holding `dev->kvm->lock`, allowing a malicious guest to stall…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] copy_from_user is called while holding dev->kvm->lock, allowing a malicious guest to stall the VM indefinitely via userfaultfd. --
commit 5fe548324e0631c69e2b3712b40114f91835f256
Author: Douglas Freimuth <[email protected]>
KVM: s390: Add map/unmap ioctl and clean mappings post-guest
This commit adds map/unmap ioctls for the s390 adapter set indicator pages, allowing access when interrupts are disabled. It utilizes pin_user_pages_remote for the mapping and includes cleanup logic during guest removal.
Quoted 11 lines
arch/s390/kvm/interrupt.c +2
-0@@ -2411,24 +2411,34 @@ static int register_io_adapter(struct kvm_device *dev, {struct s390_io_adapter *adapter;struct kvm_s390_io_adapter adapter_info; +int rc =0 ; +mutex_lock (&dev->kvm->lock); if (copy_from_user(&adapter_info,
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2421
Quoted 10 lines
arch/s390/kvm/interrupt.c +4
-6- (void __user *)attr->addr, sizeof(adapter_info))) -return -EFAULT ; - -if (adapter_info.id >=MAX_S390_IO_ADAPTERS ) -return -EINVAL ; - + (void __user *)attr->addr, sizeof(adapter_info))) { + rc = -EFAULT ; + goto out; + }
[Severity: Medium]
Does moving this copy_from_user() inside the dev->kvm->lock critical section allow a malicious userspace process to stall the VM?
If a userspace process provides an attr->addr backed by userfaultfd, it could indefinitely delay the page fault, forcing the kernel to hold the global kvm->lock. This blocks any other operations on this VM that require kvm->lock, causing a denial of service for the VM.
Could the copy_from_user() be performed into a local stack variable before acquiring the lock?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Janosch Frank
maintainerRe: [PATCH v9 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
In reply to Douglas FreimuthQuoted 53 lines · Douglas Freimuth
s390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
pin_user_pages_remoteis used for both the ioctl as well as the pin-on-demand logic inadapter_indicators_set().Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Reviewed-by Matthew Rosato <[email protected]>Signed-off-by Douglas Freimuth <[email protected]>--- arch/s390/include/asm/kvm_host.h | 5 + arch/s390/kvm/interrupt.c | 226 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 3 + arch/s390/kvm/kvm-s390.h | 2 + 4 files changed, 195 insertions(+), 41 deletions(-)arch/s390/include/asm/kvm_host.h +5
-0@@ -448,6 +448,8 @@ struct kvm_vcpu_arch {struct kvm_vm_stat {struct kvm_vm_stat_generic generic;u64 inject_io; +u64 io_390_adapter_map; +u64 io_390_adapter_unmap;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;@@ -479,6 +481,9 @@ struct s390_io_adapter {bool masked;bool swap;bool suppressible; +spinlock_t maps_lock; +struct list_head maps; +unsigned int nr_maps; };#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)arch/s390/kvm/interrupt.c +0
-0
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2414
Quoted 4 lines
arch/s390/kvm/interrupt.c +1
-0@@ -2411,24 +2411,34 @@ static int register_io_adapter(struct kvm_device *dev, {struct s390_io_adapter *adapter;struct kvm_s390_io_adapter adapter_info; +int rc =0 ;
Why do you even set it to 0 if you return 0 on the success case instead of return rc?
Quoted 15 lines
arch/s390/kvm/interrupt.c +7
-6+mutex_lock (&dev->kvm->lock);if (copy_from_user (&adapter_info, - (void __user *)attr->addr,sizeof (adapter_info))) -return -EFAULT ; - -if (adapter_info.id >=MAX_S390_IO_ADAPTERS ) -return -EINVAL ; - + (void __user *)attr->addr,sizeof (adapter_info))) { + rc = -EFAULT ; + goto out; + } +if (adapter_info.id >=MAX_S390_IO_ADAPTERS ) { + rc = -EINVAL ;
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2430
Quoted 10 lines
arch/s390/kvm/interrupt.c +4
-3+ goto out; + } adapter_info.id =array_index_nospec (adapter_info.id,MAX_S390_IO_ADAPTERS ); -if (dev->kvm->arch.adapters[adapter_info.id] !=NULL ) -return -EINVAL ; - +if (dev->kvm->arch.adapters[adapter_info.id] !=NULL ) { + rc = -EINVAL ;
You could move the rc assignment outside of the braces and have one less assignment. But if you like it more this way then keep it, I'm still able to read it :)
Quoted 17 lines
arch/s390/kvm/interrupt.c +9
-2+ goto out; + } adapter =kzalloc_obj (*adapter,GFP_KERNEL_ACCOUNT ); -if (!adapter) -return -ENOMEM ; +if (!adapter) { + rc = -ENOMEM ; + goto out; + } +INIT_LIST_HEAD (&adapter->maps); +spin_lock_init (&adapter->maps_lock); + adapter->nr_maps =0 ; adapter->id = adapter_info.id; adapter->isc = adapter_info.isc; adapter->maskable = adapter_info.maskable;
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2447
Quoted 1 line
arch/s390/kvm/interrupt.c +0
-0@@ -2437,8 +2447,13 @@ static int register_io_adapter(struct kvm_device *dev, adapter->suppressible = (adapter_info.flags) &
Why the braces?
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2456
Quoted 9 lines
arch/s390/kvm/interrupt.c +5
-0KVM_S390_ADAPTER_SUPPRESSIBLE ; dev->kvm->arch.adapters[adapter->id] = adapter; +mutex_unlock (&dev->kvm->lock);return 0 ; + +out: +mutex_unlock (&dev->kvm->lock); +return rc;
If you set rc to 0 you can actually have one return for all cases.
Quoted 7 lines
arch/s390/kvm/interrupt.c +0
-0}int kvm_s390_mask_adapter (struct kvm *kvm,unsigned int id,bool masked)@@ -2453,12 +2468,151 @@ int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked)return ret; }
Re: [PATCH v9 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Quoted 52 lines · Janosch Frank
Quoted 51 lines · Douglas Freimuth
s390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
pin_user_pages_remoteis used for both the ioctl as well as the pin-on-demand logic inadapter_indicators_set().Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Reviewed-by Matthew Rosato <[email protected]>Signed-off-by Douglas Freimuth <[email protected]>---
arch/s390/include/asm/kvm_host.h| 5 + arch/s390/kvm/interrupt.c | 226 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 3 + arch/s390/kvm/kvm-s390.h | 2 +4 files changed, 195 insertions(+), 41 deletions(-)arch/s390/include/asm/kvm_host.h +5
-0@@ -448,6 +448,8 @@ struct kvm_vcpu_arch { struct kvm_vm_stat { struct kvm_vm_stat_generic generic; u64 inject_io; + u64 io_390_adapter_map; + u64 io_390_adapter_unmap; u64 inject_float_mchk; u64 inject_pfault_done; u64 inject_service_signal;@@ -479,6 +481,9 @@ struct s390_io_adapter { bool masked; bool swap; bool suppressible; + spinlock_t maps_lock; + struct list_head maps; + unsigned int nr_maps; }; #define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)arch/s390/kvm/interrupt.c +0
-0
Quoted 8 lines
arch/s390/kvm/interrupt.c static int register_io_adapter(struct · line 2415
Quoted 6 lines
arch/s390/kvm/interrupt.c +1
-0@@ -2411,24 +2411,34 @@ static int register_io_adapter(struct kvm_device *dev, { struct s390_io_adapter *adapter; struct kvm_s390_io_adapter adapter_info; + int rc = 0;Why do you even set it to 0 if you return 0 on the success case instead of return rc?
Janosch, thanks for your input. Agreed, I intend to use rc.>
Quoted 17 lines
Quoted 17 lines
arch/s390/kvm/interrupt.c +9
-6+ mutex_lock(&dev->kvm->lock); if (copy_from_user(&adapter_info, - (void __user *)attr->addr, sizeof(adapter_info))) - return -EFAULT; - - if (adapter_info.id >= MAX_S390_IO_ADAPTERS) - return -EINVAL; - + (void __user *)attr->addr, sizeof(adapter_info))) { + rc = -EFAULT; + goto out; + } + if (adapter_info.id >= MAX_S390_IO_ADAPTERS) { + rc = -EINVAL; + goto out; + } adapter_info.id = array_index_nospec(adapter_info.id,
Quoted 8 lines
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2430
Quoted 6 lines
arch/s390/kvm/interrupt.c +2
-3MAX_S390_IO_ADAPTERS); - if (dev->kvm->arch.adapters[adapter_info.id] != NULL) - return -EINVAL; - + if (dev->kvm->arch.adapters[adapter_info.id] != NULL) { + rc = -EINVAL;You could move the rc assignment outside of the braces and have one less assignment. But if you like it more this way then keep it, I'm still able to read it :)
I will leave the assignments inside the braces which will rarely, if ever, execute. And leave rc assigned as 0.
Quoted 13 lines
Quoted 13 lines
arch/s390/kvm/interrupt.c +9
-2+ goto out; + } adapter = kzalloc_obj(*adapter, GFP_KERNEL_ACCOUNT); - if (!adapter) - return -ENOMEM; + if (!adapter) { + rc = -ENOMEM; + goto out; + } + INIT_LIST_HEAD(&adapter->maps); + spin_lock_init(&adapter->maps_lock); + adapter->nr_maps = 0;
Quoted 8 lines
Quoted 6 lines
arch/s390/kvm/interrupt.c +0
-0adapter->id = adapter_info.id; adapter->isc = adapter_info.isc; adapter->maskable = adapter_info.maskable;@@ -2437,8 +2447,13 @@ static int register_io_adapter(struct kvm_device *dev,
adapter->suppressible= (adapter_info.flags) &Why the braces?
That was existing code and not my change. But as a courtesy I will remove the braces to set it free.
Quoted 11 lines
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2456
Quoted 9 lines
KVM_S390_ADAPTER_SUPPRESSIBLE; dev->kvm->arch.adapters[adapter->id] = adapter; +mutex_unlock(&dev->kvm->lock);
return 0;arch/s390/kvm/interrupt.c +4
-0+ +out: + mutex_unlock(&dev->kvm->lock); + return rc;If you set rc to 0 you can actually have one return for all cases.
Agreed, I will remove two lines before the label and let it fall through for one return for all cases.
Quoted 7 lines
Quoted 7 lines
}
intkvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked)arch/s390/kvm/interrupt.c +0
-0@@ -2453,12 +2468,151 @@ int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) return ret; }
PATCH v9 2/3 KVM: s390: Enable adapter_indicators_set to use mapped pages
In reply to Douglas FreimuthThe s390 adapter_indicators_set function can now be optimized to use long-term mapped pages when available so that work can be processed on a fast path when interrupts are disabled.
If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.
--- arch/s390/kvm/interrupt.c | 87 ++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 24 deletions(-)
arch/s390/kvm/interrupt.c +63 -24
@@ -2895,41 +2895,80 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } +static struct s390_map_info *get_map_info (struct s390_io_adapter *adapter, +u64 addr) +{ +struct s390_map_info *map; + +if (!adapter) +return NULL ; + +list_for_each_entry (map, &adapter->maps, list) { +if (map->addr == addr) +return map; + } +return NULL ; +} +static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {unsigned long bit;int summary_set, idx; -struct page *ind_page, *summary_page; +struct s390_map_info *ind_info, *summary_info;void *map; +struct page *ind_page, *summary_page; +unsigned long flags; - ind_page =pin_map_page (kvm, adapter_int->ind_addr,0 ); -if (!ind_page) -return -1 ; - summary_page =pin_map_page (kvm, adapter_int->summary_addr,0 ); -if (!summary_page) { + ind_page =NULL ; + +spin_lock_irqsave (&adapter->maps_lock, flags); + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + ind_page =pin_map_page (kvm, adapter_int->ind_addr,0 ); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page); +srcu_read_unlock (&kvm->srcu, idx);unpin_user_page (ind_page); -return -1 ; + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); + } +spin_lock_irqsave (&adapter->maps_lock, flags); + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + summary_page =pin_map_page (kvm, adapter_int->summary_addr,0 ); +if (WARN_ON_ONCE (!summary_page)) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (summary_page); + bit =get_ind_bit (adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap); + summary_set =test_and_set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (summary_page); +srcu_read_unlock (&kvm->srcu, idx); +unpin_user_page (summary_page); + }else { + map =page_address (summary_info->page); + bit =get_ind_bit (summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set =test_and_set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); } - idx =srcu_read_lock (&kvm->srcu); - map =page_address (ind_page); - bit =get_ind_bit (adapter_int->ind_addr, - adapter_int->ind_offset, adapter->swap); -set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (ind_page); - map =page_address (summary_page); - bit =get_ind_bit (adapter_int->summary_addr, - adapter_int->summary_offset, adapter->swap); - summary_set =test_and_set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (summary_page); -srcu_read_unlock (&kvm->srcu, idx); - -unpin_user_page (ind_page); -unpin_user_page (summary_page);return summary_set ?0 :1 ; }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v9 2/3] KVM: s390: Enable adapter_indicators_set to use mapped pages
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Missing dirty page tracking in fast path breaks live migration. - [High] Unprivileged userspace…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Missing dirty page tracking in fast path breaks live migration.
- [High] Unprivileged userspace can trigger a host denial of service via WARN_ON_ONCE.
- [Medium] Atomicity of indicator and summary updates is broken if the summary page fails to pin. --
commit 30338b81c5f9c19e09c8c31062f7a9c7f3b4ecb4
Author: Douglas Freimuth <[email protected]>
KVM: s390: Enable adapter_indicators_set to use mapped pages
This patch optimizes the s390 adapter_indicators_set function to use long-term mapped pages when available. The fast path operates when interrupts are disabled, while unmapped pages fall back to the slower local mapping path.
Quoted 4 lines
arch/s390/kvm/interrupt.c +0
-0
[ ... ]
Quoted 5 lines
arch/s390/kvm/interrupt.c +0
-0@@ -2895,41 +2895,80 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {
[ ... ]
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+if (!ind_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + ind_page =pin_map_page (kvm, adapter_int->ind_addr,0 ); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map);
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2947
Quoted 10 lines
arch/s390/kvm/interrupt.c +9
-0+mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page); +srcu_read_unlock (&kvm->srcu, idx);unpin_user_page (ind_page); + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); + }
[Severity: High]
Could the omission of mark_page_dirty() and set_page_dirty_lock() in this fast path cause a regression during live migration?
Without notifying KVM's dirty page tracking mechanism, the destination VM might receive stale indicator pages, resulting in lost interrupts and guest device driver hangs.
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2954
Quoted 7 lines
arch/s390/kvm/interrupt.c +7
-0+spin_lock_irqsave (&adapter->maps_lock, flags); + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + summary_page =pin_map_page (kvm, adapter_int->summary_addr,0 ); +if (WARN_ON_ONCE (!summary_page)) +return -1 ;
[Severity: High]
Is it safe to use a WARN_ON_ONCE() here? Since summary_addr is a userspace virtual address provided by the VMM, an unprivileged host process could intentionally munmap() the memory region to trigger this warning. This creates a potential denial of service vector if panic_on_warn is enabled.
[Severity: Medium]
What happens to guest memory consistency if pin_map_page() fails here?
In the original code, both the indicator and summary pages were pinned and validated before any bits were modified. Now, the indicator page is modified in the preceding block before the summary page is validated.
If pin_map_page() fails and returns -1, the indicator bit remains permanently set in guest memory without the corresponding summary bit being updated, which breaks the update atomicity.
Quoted 6 lines
arch/s390/kvm/interrupt.c +6
-0+ idx =srcu_read_lock (&kvm->srcu); + map =page_address (summary_page); + bit =get_ind_bit (adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap); + summary_set =test_and_set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT );
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2970
Quoted 10 lines
arch/s390/kvm/interrupt.c +9
-0+set_page_dirty_lock (summary_page); +srcu_read_unlock (&kvm->srcu, idx); +unpin_user_page (summary_page); + }else { + map =page_address (summary_info->page); + bit =get_ind_bit (summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set =test_and_set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); }
[Severity: High]
Similar to the indicator fast path above, shouldn't this summary page fast path also update KVM's dirty page tracking? Modifying guest memory without calling mark_page_dirty() could corrupt migration state.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v9 3/3 KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas Freimuths390 needs a fast path for irq injection, and along those lines we introduce kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_no_inject. The no inject counter captures adapter masked, coalesced and suppressed interrupts.
--- arch/s390/include/asm/kvm_host.h | 6 +- arch/s390/kvm/intercept.c | 5 +- arch/s390/kvm/interrupt.c | 265 ++++++++++++++++++++++++------- arch/s390/kvm/kvm-s390.c | 27 +++- arch/s390/kvm/kvm-s390.h | 3 +- 5 files changed, 241 insertions(+), 65 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -1
@@ -359,7 +359,7 @@ struct kvm_s390_float_interrupt {struct kvm_s390_mchk_info mchk;struct kvm_s390_ext_info srv_signal;int last_sleep_cpu; -struct mutex ais_lock; +spinlock_t ais_lock;u8 simm;u8 nimm; };
@@ -450,6 +450,10 @@ struct kvm_vm_stat {u64 inject_io;u64 io_390_adapter_map;u64 io_390_adapter_unmap; +u64 io_390_inatomic; +u64 io_flic_inject_airq; +u64 io_set_adapter_int; +u64 io_390_inatomic_no_inject;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
arch/s390/kvm/intercept.c +3 -2
@@ -517,8 +517,9 @@ static int handle_pv_spx(struct kvm_vcpu *vcpu)static int handle_pv_sclp (struct kvm_vcpu *vcpu) {struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; +unsigned long flags; -spin_lock (&fi->lock); +spin_lock_irqsave (&fi->lock, flags);/* * 2 cases: * a: an sccb answering interrupt was already pending or in flight.
@@ -534,7 +535,7 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu) fi->srv_signal.ext_params |=0x43000 ;set_bit (IRQ_PEND_EXT_SERVICE , &fi->pending_irqs);clear_bit (IRQ_PEND_EXT_SERVICE , &fi->masked_irqs); -spin_unlock (&fi->lock); +spin_unlock_irqrestore (&fi->lock, flags);return 0 ; }
arch/s390/kvm/interrupt.c +208 -57
589 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +23 -4
@@ -70,6 +70,10 @@ const struct kvm_stats_desc kvm_vm_stats_desc[] = {STATS_DESC_COUNTER (VM , inject_io),STATS_DESC_COUNTER (VM , io_390_adapter_map),STATS_DESC_COUNTER (VM , io_390_adapter_unmap), +STATS_DESC_COUNTER (VM , io_390_inatomic), +STATS_DESC_COUNTER (VM , io_flic_inject_airq), +STATS_DESC_COUNTER (VM , io_set_adapter_int), +STATS_DESC_COUNTER (VM , io_390_inatomic_no_inject),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done), STATS_DESC_COUNTER(VM, inject_service_signal),
@@ -2851,6 +2855,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)void __user *argp = (void __user *)arg;struct kvm_device_attr attr;int r; +struct kvm_s390_interrupt_info *inti;switch (ioctl) {case KVM_S390_INTERRUPT : {
@@ -2859,7 +2864,12 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) r = -EFAULT ;if (copy_from_user (&s390int, argp,sizeof (s390int)))break ; - r =kvm_s390_inject_vm (kvm, &s390int); + inti =kzalloc_obj (*inti,GFP_KERNEL_ACCOUNT ); +if (!inti) +return -ENOMEM ; + r =kvm_s390_inject_vm (kvm, &s390int, inti); +if (r) +kfree (inti);break ; }case KVM_CREATE_IRQCHIP : {
@@ -3257,7 +3267,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)mutex_unlock (&kvm->lock); } -mutex_init (&kvm->arch.float_int.ais_lock); +spin_lock_init (&kvm->arch.float_int.ais_lock);spin_lock_init (&kvm->arch.float_int.lock);for (i =0 ; i <FIRQ_LIST_COUNT ; i++)INIT_LIST_HEAD (&kvm->arch.float_int.lists[i]);
@@ -4379,19 +4389,28 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo }static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token) +unsigned long token) {struct kvm_s390_interrupt inti;struct kvm_s390_irq irq; +struct kvm_s390_interrupt_info *inti_mem =NULL ; +int ret =0 ;if (start_token) { irq.u.ext.ext_params2 = token; irq.type =KVM_S390_INT_PFAULT_INIT ;WARN_ON_ONCE (kvm_s390_inject_vcpu (vcpu, &irq)); }else { + inti_mem =kzalloc_obj (*inti_mem,GFP_KERNEL_ACCOUNT ); +if (WARN_ON_ONCE (!inti_mem)) +return ; + inti.type =KVM_S390_INT_PFAULT_DONE ; inti.parm64 = token; -WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti)); + ret =kvm_s390_inject_vm (vcpu->kvm, &inti, inti_mem); +if (ret) +kfree (inti_mem); +WARN_ON_ONCE (ret); } }
arch/s390/kvm/kvm-s390.h +2 -1
@@ -376,7 +376,8 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu);void kvm_s390_clear_local_irqs (struct kvm_vcpu *vcpu);void kvm_s390_clear_float_irqs (struct kvm *kvm);int __must_check kvm_s390_inject_vm (struct kvm *kvm, -struct kvm_s390_interrupt *s390int); +struct kvm_s390_interrupt *s390int, +struct kvm_s390_interrupt_info *inti);int __must_check kvm_s390_inject_vcpu (struct kvm_vcpu *vcpu,struct kvm_s390_irq *irq); static inline int kvm_s390_inject_prog_irq(struct kvm_vcpu *vcpu,
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v9 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [High] The rollback path in adapter_indicators_set_fast early-exits if ind_info is missing, failing to clear the summary bit and permanently coalescing future interrupts.
- [High] Adversary-reachable memory allocation failure triggers WARN_ON_ONCE.
- [High] Use of GFP_ATOMIC in a preemption-disabled context (kvm_arch_set_irq_inatomic) includes __GFP_KSWAPD_RECLAIM, which risks lock recursion and deadlocks.
Pre-existing issues:
- [High] Uninitialized stack variable inti.parm is passed to trace_kvm_s390_inject_vm, leaking kernel stack memory to the trace buffer. --
commit db19d981b8f8dcf62bd14e1ad3e06ee6ac4f4929
Author: Douglas Freimuth <[email protected]>
KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
This commit introduces a fast path for irq injection (kvm_arch_set_irq_inatomic) instead of placing all interrupts on the global work queue. It adds map/unmap functions for adapter indicators, switches from mutexes and semaphores to spinlocks for shared resources, and implements memory allocations on the fast path using GFP_ATOMIC.
Quoted 4 lines
arch/s390/kvm/interrupt.c +0
-0
[ ... ]
Quoted 10 lines
arch/s390/kvm/interrupt.c +6
-0@@ -2972,6 +3001,44 @@ static int adapter_indicators_set(struct kvm *kvm,return summary_set ?0 :1 ; } +static int adapter_indicators_set_fast (struct kvm *kvm, +struct s390_io_adapter *adapter, +struct kvm_s390_adapter_int *adapter_int, +int setbit) +{ +unsigned long bit;
arch/s390/kvm/interrupt.c static int adapter_indicators_set(struct kvm *kvm, · line 3019
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+int summary_set; +struct s390_map_info *ind_info, *summary_info; +void *map; + +spin_lock (&adapter->maps_lock); + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) { +spin_unlock (&adapter->maps_lock); +return -EWOULDBLOCK ; + }
[Severity: High]
When an allocation or injection fails in kvm_arch_set_irq_inatomic(), it calls adapter_indicators_set_fast() with setbit=0 to roll back the summary bit. If the indicator page was unmapped concurrently, get_map_info() returns NULL and we early-return -EWOULDBLOCK.
Does this early return skip clearing the summary bit? If the summary bit remains stuck, could this cause all future interrupts for any indicator sharing that summary page to be silently dropped?
Quoted 2 lines
arch/s390/kvm/interrupt.c +2
-0+ map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap);
[ ... ]
arch/s390/kvm/interrupt.c int __init kvm_s390_gib_init(u8 nisc) · line 3752
Quoted 9 lines
arch/s390/kvm/interrupt.c +9
-0+ ret =adapter_indicators_set_fast (kvm, adapter, &e->adapter, setbit); +if (ret <0 ) +return -EWOULDBLOCK ; +if (!ret || adapter->masked) { + kvm->stat.io_390_inatomic_no_inject++; +return 0 ; + } + + inti =kzalloc_obj (*inti,GFP_ATOMIC );
[Severity: High]
Does this GFP_ATOMIC allocation risk lock recursion?
kvm_arch_set_irq_inatomic() is invoked as a fast-path for irqfd injection, executing from the eventfd wait queue callback with the waitqueue spinlock held (preemption disabled).
GFP_ATOMIC inherently includes __GFP_KSWAPD_RECLAIM, which triggers wakeup_kswapd() and re-enters the scheduler machinery via try_to_wake_up().
Should we strip the __GFP_KSWAPD_RECLAIM bit or use a bare flag like __GFP_NOWARN instead?
Quoted 5 lines
arch/s390/kvm/interrupt.c +5
-0+if (!inti) { + setbit =0 ; +adapter_indicators_set_fast (kvm, adapter, &e->adapter, setbit); +return -EWOULDBLOCK ; + }
[ ... ]
Quoted 4 lines
arch/s390/kvm/kvm-s390.c +0
-0
[ ... ]
arch/s390/kvm/kvm-s390.c int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo · line 4394
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +1
-1@@ -4379,19 +4389,28 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo }static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token) +unsigned long token) {struct kvm_s390_interrupt inti;
[Severity: High]
This isn't a bug introduced by this patch, but the stack variable inti is declared without initialization. While inti.type and inti.parm64 are subsequently populated, inti.parm remains uninitialized.
When inti is passed into kvm_s390_inject_vm(), trace_kvm_s390_inject_vm() unconditionally logs s390int->parm. Could this leak 32 bits of uninitialized kernel stack memory to userspace via the trace ring buffer?
arch/s390/kvm/kvm-s390.c int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo · line 4406
Quoted 12 lines
arch/s390/kvm/kvm-s390.c +5
-0struct kvm_s390_irq irq; +struct kvm_s390_interrupt_info *inti_mem =NULL ; +int ret =0 ;if (start_token) { irq.u.ext.ext_params2 = token; irq.type =KVM_S390_INT_PFAULT_INIT ;WARN_ON_ONCE (kvm_s390_inject_vcpu (vcpu, &irq)); }else { + inti_mem =kzalloc_obj (*inti_mem,GFP_KERNEL_ACCOUNT ); +if (WARN_ON_ONCE (!inti_mem)) +return ;
[Severity: High]
Is it safe to use WARN_ON_ONCE() for this memory allocation failure?
Since this path is reachable by a guest triggering async page faults, a malicious guest could spray memory to induce host memory pressure, hit this allocation failure, and trigger a host panic if panic_on_warn is set.
Could we handle this failure gracefully or return an error without the WARN?
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +5
-1+ inti.type =KVM_S390_INT_PFAULT_DONE ; inti.parm64 = token; -WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti)); + ret =kvm_s390_inject_vm (vcpu->kvm, &inti, inti_mem); +if (ret) +kfree (inti_mem); +WARN_ON_ONCE (ret); } }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH v9 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas FreimuthQuoted 20 lines · Douglas Freimuth
s390 needs a fast path for irq injection, and along those lines we introduce
kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the
adapter_indicatorspage needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizesGFP_KERNEL_ACCOUNTbut we had to implement the fast path withGFP_ATOMICallocation. Each of these enhancements were required to prevent blocking on the fast inject path.Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 6 lines
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including
io_390_inatomic,io_flic_inject_airq,io_set_adapter_intandio_390_inatomic_no_inject. The no inject counter captures adapter masked, coalesced and suppressed interrupts.Signed-off-by Douglas Freimuth <[email protected]>
Re: [PATCH v9 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
Quoted 21 lines · Matthew Rosato
Quoted 20 lines · Douglas Freimuth
s390 needs a fast path for irq injection, and along those lines we introduce
kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the
adapter_indicatorspage needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizesGFP_KERNEL_ACCOUNTbut we had to implement the fast path withGFP_ATOMICallocation. Each of these enhancements were required to prevent blocking on the fast inject path.Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 8 lines
Quoted 6 lines
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including
io_390_inatomic,io_flic_inject_airq,io_set_adapter_intandio_390_inatomic_no_inject. The no inject counter captures adapter masked, coalesced and suppressed interrupts.Signed-off-by Douglas Freimuth <[email protected]>Reviewed-by Matthew Rosato <[email protected]>
Matt, thank you for all of your inputs across the iterations.>