KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
v8 cover KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
s390 needs this series of three patches in order to enable a non-blocking path for irqfd injection on s390 via kvm_arch_set_irq_inatomic(). Before these changes, kvm_arch_set_irq_inatomic() would just return -EWOULDBLOCK and place all interrupts on the global work queue, which must subsequently be processed by a different thread. This series of patches implements an s390 version of inatomic and is relevant to virtio-blk and virtio-net and was tested against virtio-pci and virtio-ccw.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
s390 doesn't support a PREEMPT_RT kernel and this patch doesn't either. Given this fact, we are not using raw_spin_lock instead we are using regular spin_lock.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_adapter_masked_or_coalesced. And counters have been added to analyze map/unmap of the adapter_indicator pages in non-Secure Execution environments and to track fencing of Fast Inject in Secure Execution environments. In order to take advantage of this kernel series with virtio-pci, a QEMU that includes the 's390x/pci: set kvm_msi_via_irqfd_allowed' fix is needed. Additionally, the guest xml needs a thread pool and threads explicitly assigned per disk device using the common way of defining threads for disks.
Patch 1 enables map/unmap of adapter indicator pages but for Secure Execution environments it avoids the long term mapping.
v7->v8: Move acquisition of kvm->lock to top of register_io_adapter() v7->v8: If !summary_page WARN_ON_ONCE if set indicator but not summary v7->v8: Remove kfree(init) from kvm_s390_inject_vm() default case v7->v8: return 0 if !level in kvm_arch_set_irq_inatomic() v7->v8: Use irqsave/irqrestore for spin_lock/unlock on fi->lock v7->v8: Modify statistic name io_390_inatomic_adapter_masked_or_coalesced
Douglas Freimuth (3):
KVM: s390: Add map/unmap ioctl and clean mappings post-guest
KVM: s390: Enable adapter_indicators_set to use mapped pages
KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
arch/s390/include/asm/kvm_host.h | 11 +- arch/s390/kvm/intercept.c | 5 +- arch/s390/kvm/interrupt.c | 525 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 30 +- arch/s390/kvm/kvm-s390.h | 5 +- 5 files changed, 467 insertions(+), 109 deletions(-)
Signature · 1 line
2.54.0
PATCH v8 1/3 KVM: s390: Add map/unmap ioctl and clean mappings post-guest
s390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed. pin_user_pages_remote is used for both the ioctl as well as the pin-on-demand logic in adapter_indicators_set().
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
--- arch/s390/include/asm/kvm_host.h | 5 + arch/s390/kvm/interrupt.c | 194 ++++++++++++++++++++++++++----- arch/s390/kvm/kvm-s390.c | 3 + arch/s390/kvm/kvm-s390.h | 2 + 4 files changed, 174 insertions(+), 30 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -0
@@ -448,6 +448,8 @@ struct kvm_vcpu_arch {struct kvm_vm_stat {struct kvm_vm_stat_generic generic;u64 inject_io; +u64 io_390_adapter_map; +u64 io_390_adapter_unmap;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
@@ -479,6 +481,9 @@ struct s390_io_adapter {bool masked;bool swap;bool suppressible; +spinlock_t maps_lock; +struct list_head maps; +unsigned int nr_maps; };#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)
arch/s390/kvm/interrupt.c +164 -30
252 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +3 -0
@@ -68,6 +68,8 @@const struct kvm_stats_desc kvm_vm_stats_desc[] = {KVM_GENERIC_VM_STATS (),STATS_DESC_COUNTER (VM , inject_io), +STATS_DESC_COUNTER (VM , io_390_adapter_map), +STATS_DESC_COUNTER (VM , io_390_adapter_unmap),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done),STATS_DESC_COUNTER (VM , inject_service_signal),
@@ -2513,6 +2515,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)if (kvm_s390_pv_is_protected (kvm))break ; +kvm_s390_unmap_all_adapters (kvm);mmap_write_lock (kvm->mm);/* * Disable creation of new THPs. Existing THPs can stay, they
arch/s390/kvm/kvm-s390.h +2 -0
@@ -560,6 +560,8 @@ void kvm_s390_gisa_disable(struct kvm *kvm);void kvm_s390_gisa_enable (struct kvm *kvm);int __init kvm_s390_gib_init (u8 nisc);void kvm_s390_gib_destroy (void ); +void kvm_s390_unmap_all_adapters (struct kvm *kvm); +/* implemented in guestdbg.c */ void kvm_s390_backup_guest_per_regs (struct kvm_vcpu *vcpu);
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v8 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The register_io_adapter function acquires dev->kvm->lock but fails to release it on several early…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The register_io_adapter function acquires dev->kvm->lock but fails to release it on several early error return paths, causing a permanent deadlock. --
commit 846f99f02dcb9dca2543059b14bd01a7e4a020cf
Author: Douglas Freimuth <[email protected]>
KVM: s390: Add map/unmap ioctl and clean mappings post-guest
This patch introduces map and unmap ioctls for s390 adapter set indicator pages to allow access when interrupts are disabled. It adds logic to clean up mappings upon guest removal, and fences the ioctls in Secure Execution environments to prevent longterm pinning.
Quoted 3 lines
arch/s390/kvm/interrupt.c +0
-0
[ ... ]
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2418
Quoted 8 lines
arch/s390/kvm/interrupt.c +1
-0@@ -2412,6 +2412,7 @@ static int register_io_adapter(struct kvm_device *dev,struct s390_io_adapter *adapter;struct kvm_s390_io_adapter adapter_info; +mutex_lock (&dev->kvm->lock);if (copy_from_user (&adapter_info, (void __user *)attr->addr,sizeof (adapter_info)))return -EFAULT ;
[Severity: High]
Does this code leave dev->kvm->lock held on the early error return paths?
If copy_from_user() fails here, or if the subsequent adapter_info.id validation checks fail, the function returns an error directly without calling mutex_unlock(&dev->kvm->lock). This would leave the mutex permanently locked, potentially causing the VM to deadlock on any subsequent operations that require this lock.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v8 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
In reply to Douglas FreimuthQuoted 10 lines · Douglas Freimuth
s390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
pin_user_pages_remoteis used for both the ioctl as well as the pin-on-demand logic inadapter_indicators_set().Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 7 lines
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Signed-off-by Douglas Freimuth <[email protected]>
I gave you a reviewed-by tag on v7 with the caveat that you move the kvm lock earlier in register_io_adapter (which you did) so that review tag should have been included with this patch. Please include it in next version or it will get lost.
arch/s390/kvm/interrupt.c static int register_io_adapter(struct kvm_device *dev, · line 2418
Quoted 8 lines
arch/s390/kvm/interrupt.c +1
-0@@ -2412,6 +2412,7 @@ static int register_io_adapter(struct kvm_device *dev,struct s390_io_adapter *adapter;struct kvm_s390_io_adapter adapter_info; +mutex_lock (&dev->kvm->lock);if (copy_from_user (&adapter_info, (void __user *)attr->addr,sizeof (adapter_info)))return -EFAULT ;
That said...
Please also handle the sashiko comment for this patch, you moved the kvm lock earlier like I requested but that means now all of the nonzero return paths have to drop it before returning, the above is just one example.
Sounds like a case for either using guard(mutex) [1] or something like
int rc = 0;
if (bad_things_happen) {
rc = -WHATEVER;
goto out;
}
[...]
out:
mutex_unlock(&dev->kvm->lock);
return rc;
PATCH v8 2/3 KVM: s390: Enable adapter_indicators_set to use mapped pages
In reply to Douglas FreimuthThe s390 adapter_indicators_set function can now be optimized to use long-term mapped pages when available so that work can be processed on a fast path when interrupts are disabled.
If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.
--- arch/s390/kvm/interrupt.c | 87 ++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 24 deletions(-)
arch/s390/kvm/interrupt.c +63 -24
@@ -2885,41 +2885,80 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } +static struct s390_map_info *get_map_info (struct s390_io_adapter *adapter, +u64 addr) +{ +struct s390_map_info *map; + +if (!adapter) +return NULL ; + +list_for_each_entry (map, &adapter->maps, list) { +if (map->addr == addr) +return map; + } +return NULL ; +} +static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {unsigned long bit;int summary_set, idx; -struct page *ind_page, *summary_page; +struct s390_map_info *ind_info, *summary_info;void *map; +struct page *ind_page, *summary_page; +unsigned long flags; - ind_page =pin_map_page (kvm, adapter_int->ind_addr,0 ); -if (!ind_page) -return -1 ; - summary_page =pin_map_page (kvm, adapter_int->summary_addr,0 ); -if (!summary_page) { + ind_page =NULL ; + +spin_lock_irqsave (&adapter->maps_lock, flags); + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + ind_page =pin_map_page (kvm, adapter_int->ind_addr,0 ); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page); +srcu_read_unlock (&kvm->srcu, idx);unpin_user_page (ind_page); -return -1 ; + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); + } +spin_lock_irqsave (&adapter->maps_lock, flags); + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + summary_page =pin_map_page (kvm, adapter_int->summary_addr,0 ); +if (WARN_ON_ONCE (!summary_page)) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (summary_page); + bit =get_ind_bit (adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap); + summary_set =test_and_set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (summary_page); +srcu_read_unlock (&kvm->srcu, idx); +unpin_user_page (summary_page); + }else { + map =page_address (summary_info->page); + bit =get_ind_bit (summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set =test_and_set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); } - idx =srcu_read_lock (&kvm->srcu); - map =page_address (ind_page); - bit =get_ind_bit (adapter_int->ind_addr, - adapter_int->ind_offset, adapter->swap); -set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (ind_page); - map =page_address (summary_page); - bit =get_ind_bit (adapter_int->summary_addr, - adapter_int->summary_offset, adapter->swap); - summary_set =test_and_set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (summary_page); -srcu_read_unlock (&kvm->srcu, idx); - -unpin_user_page (ind_page); -unpin_user_page (summary_page);return summary_set ?0 :1 ; }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v8 2/3] KVM: s390: Enable adapter_indicators_set to use mapped pages
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] A userspace-triggerable WARN_ON_ONCE can crash the host kernel. - [High] Loss of atomicity leads…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] A userspace-triggerable WARN_ON_ONCE can crash the host kernel.
- [High] Loss of atomicity leads to partially updated, inconsistent guest adapter indicators on mapping failure.
- [High] The fast path for long-term mapped pages skips KVM dirty page tracking, breaking live migration. --
commit d5dc6d5087f20c38643d53d19e20548f19464244
Author: Douglas Freimuth <[email protected]>
KVM: s390: Enable adapter_indicators_set to use mapped pages
This commit optimizes the s390 adapter_indicators_set function to use long-term mapped pages when available. If adapter indicator pages are not mapped, local mapping is done on a slow path.
Quoted 4 lines
arch/s390/kvm/interrupt.c +0
-0
[ ... ]
Quoted 5 lines
arch/s390/kvm/interrupt.c +0
-0@@ -2885,41 +2885,80 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {
[ ... ]
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2935
Quoted 4 lines
arch/s390/kvm/interrupt.c +4
-0+ }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map);
[Severity: High]
Does this fast path need to track dirty pages for live migration?
The slow path calls mark_page_dirty() and set_page_dirty_lock() after setting the bit, but this mapped-page fast path modifies guest memory directly via set_bit() without updating KVM's dirty logging bitmap.
Could this leave the destination host with stale indicator states after a live migration?
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2942
Quoted 7 lines
arch/s390/kvm/interrupt.c +7
-0+spin_unlock_irqrestore (&adapter->maps_lock, flags); + } +spin_lock_irqsave (&adapter->maps_lock, flags); + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + summary_page =pin_map_page (kvm, adapter_int->summary_addr,0 );
[Severity: High]
Could this lead to a partially updated adapter state if the summary page fails to map?
The original code atomically pinned both the indicator and summary pages before making any modifications. Here, if pin_map_page() fails for the summary page, the function aborts and returns -1, but the indicator bit was already set in the preceding block and is not rolled back.
Can this result in the indicator being set without a corresponding summary bit, potentially causing lost events and I/O hangs?
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2943
Quoted 1 line
arch/s390/kvm/interrupt.c +1
-0+if (WARN_ON_ONCE (!summary_page))
[Severity: High]
Can a malicious userspace process trigger this WARN_ON_ONCE?
Since the summary_addr is provided by userspace (e.g., via QEMU setting up routing entries), an unprivileged process might be able to intentionally unmap the memory to force pin_map_page() to fail.
If panic_on_warn is enabled in a production environment, could this action crash the host kernel?
Quoted 5 lines
arch/s390/kvm/interrupt.c +5
-0+return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (summary_page); + bit =get_ind_bit (adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap);
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2958
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+ summary_set =test_and_set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (summary_page); +srcu_read_unlock (&kvm->srcu, idx); +unpin_user_page (summary_page); + }else { + map =page_address (summary_info->page); + bit =get_ind_bit (summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set =test_and_set_bit (bit, map);
[Severity: High]
Similarly, is dirty page tracking missing here for the summary page fast path, bypassing KVM's dirty logging bitmap?
Quoted 22 lines
arch/s390/kvm/interrupt.c +1
-17+spin_unlock_irqrestore (&adapter->maps_lock, flags); } - idx =srcu_read_lock (&kvm->srcu); - map =page_address (ind_page); - bit =get_ind_bit (adapter_int->ind_addr, - adapter_int->ind_offset, adapter->swap); -set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (ind_page); - map =page_address (summary_page); - bit =get_ind_bit (adapter_int->summary_addr, - adapter_int->summary_offset, adapter->swap); - summary_set =test_and_set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (summary_page); -srcu_read_unlock (&kvm->srcu, idx); - -unpin_user_page (ind_page); -unpin_user_page (summary_page);return summary_set ?0 :1 ; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH v8 2/3] KVM: s390: Enable adapter_indicators_set to use mapped pages
In reply to Douglas FreimuthQuoted 9 lines · Douglas Freimuth
The s390
adapter_indicators_setfunction can now be optimized to use long-term mapped pages when available so that work can be processed on a fast path when interrupts are disabled.
If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.Signed-off-by Douglas Freimuth <[email protected]>
Review tag was given in v7, please include for v9.
PATCH v8 3/3 KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas Freimuths390 needs a fast path for irq injection, and along those lines we introduce kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_adapter_masked_or_coalesced.
--- arch/s390/include/asm/kvm_host.h | 6 +- arch/s390/kvm/intercept.c | 5 +- arch/s390/kvm/interrupt.c | 264 ++++++++++++++++++++++++------- arch/s390/kvm/kvm-s390.c | 27 +++- arch/s390/kvm/kvm-s390.h | 3 +- 5 files changed, 240 insertions(+), 65 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -1
@@ -359,7 +359,7 @@ struct kvm_s390_float_interrupt {struct kvm_s390_mchk_info mchk;struct kvm_s390_ext_info srv_signal;int last_sleep_cpu; -struct mutex ais_lock; +spinlock_t ais_lock;u8 simm;u8 nimm; };
@@ -450,6 +450,10 @@ struct kvm_vm_stat {u64 inject_io;u64 io_390_adapter_map;u64 io_390_adapter_unmap; +u64 io_390_inatomic; +u64 io_flic_inject_airq; +u64 io_set_adapter_int; +u64 io_390_inatomic_adapter_masked_or_coalesced;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
arch/s390/kvm/intercept.c +3 -2
@@ -517,8 +517,9 @@ static int handle_pv_spx(struct kvm_vcpu *vcpu)static int handle_pv_sclp (struct kvm_vcpu *vcpu) {struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; +unsigned long flags; -spin_lock (&fi->lock); +spin_lock_irqsave (&fi->lock, flags);/* * 2 cases: * a: an sccb answering interrupt was already pending or in flight.
@@ -534,7 +535,7 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu) fi->srv_signal.ext_params |=0x43000 ;set_bit (IRQ_PEND_EXT_SERVICE , &fi->pending_irqs);clear_bit (IRQ_PEND_EXT_SERVICE , &fi->masked_irqs); -spin_unlock (&fi->lock); +spin_unlock_irqrestore (&fi->lock, flags);return 0 ; }
arch/s390/kvm/interrupt.c +207 -57
588 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +23 -4
@@ -70,6 +70,10 @@ const struct kvm_stats_desc kvm_vm_stats_desc[] = {STATS_DESC_COUNTER (VM , inject_io),STATS_DESC_COUNTER (VM , io_390_adapter_map),STATS_DESC_COUNTER (VM , io_390_adapter_unmap), +STATS_DESC_COUNTER (VM , io_390_inatomic), +STATS_DESC_COUNTER (VM , io_flic_inject_airq), +STATS_DESC_COUNTER (VM , io_set_adapter_int), +STATS_DESC_COUNTER (VM , io_390_inatomic_adapter_masked_or_coalesced),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done), STATS_DESC_COUNTER(VM, inject_service_signal),
@@ -2851,6 +2855,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)void __user *argp = (void __user *)arg;struct kvm_device_attr attr;int r; +struct kvm_s390_interrupt_info *inti;switch (ioctl) {case KVM_S390_INTERRUPT : {
@@ -2859,7 +2864,12 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) r = -EFAULT ;if (copy_from_user (&s390int, argp,sizeof (s390int)))break ; - r =kvm_s390_inject_vm (kvm, &s390int); + inti =kzalloc_obj (*inti,GFP_KERNEL_ACCOUNT ); +if (!inti) +return -ENOMEM ; + r =kvm_s390_inject_vm (kvm, &s390int, inti); +if (r) +kfree (inti);break ; }case KVM_CREATE_IRQCHIP : {
@@ -3257,7 +3267,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)mutex_unlock (&kvm->lock); } -mutex_init (&kvm->arch.float_int.ais_lock); +spin_lock_init (&kvm->arch.float_int.ais_lock);spin_lock_init (&kvm->arch.float_int.lock);for (i =0 ; i <FIRQ_LIST_COUNT ; i++)INIT_LIST_HEAD (&kvm->arch.float_int.lists[i]);
@@ -4379,19 +4389,28 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo }static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token) +unsigned long token) {struct kvm_s390_interrupt inti;struct kvm_s390_irq irq; +struct kvm_s390_interrupt_info *inti_mem =NULL ; +int ret =0 ;if (start_token) { irq.u.ext.ext_params2 = token; irq.type =KVM_S390_INT_PFAULT_INIT ;WARN_ON_ONCE (kvm_s390_inject_vcpu (vcpu, &irq)); }else { + inti_mem =kzalloc_obj (*inti_mem,GFP_KERNEL_ACCOUNT ); +if (WARN_ON_ONCE (!inti_mem)) +return ; + inti.type =KVM_S390_INT_PFAULT_DONE ; inti.parm64 = token; -WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti)); + ret =kvm_s390_inject_vm (vcpu->kvm, &inti, inti_mem); +if (ret) +kfree (inti_mem); +WARN_ON_ONCE (ret); } }
arch/s390/kvm/kvm-s390.h +2 -1
@@ -376,7 +376,8 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu);void kvm_s390_clear_local_irqs (struct kvm_vcpu *vcpu);void kvm_s390_clear_float_irqs (struct kvm *kvm);int __must_check kvm_s390_inject_vm (struct kvm *kvm, -struct kvm_s390_interrupt *s390int); +struct kvm_s390_interrupt *s390int, +struct kvm_s390_interrupt_info *inti);int __must_check kvm_s390_inject_vcpu (struct kvm_vcpu *vcpu,struct kvm_s390_irq *irq); static inline int kvm_s390_inject_prog_irq(struct kvm_vcpu *vcpu,
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v8 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Allocation in atomic context drops memcg accounting and potentially triggers kswapd lock…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Allocation in atomic context drops memcg accounting and potentially triggers kswapd lock recursion. --
commit 0463a419ba36eefec4ffef625bfb65cfcb7dbbd7
Author: Douglas Freimuth <[email protected]>
KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
This commit introduces a fast path for IRQ injection in s390 KVM via kvm_arch_set_irq_inatomic. It transitions shared resources to spinlocks and changes allocations to atomic to prevent blocking, as the path runs with interrupts disabled.
Quoted 4 lines
arch/s390/kvm/interrupt.c +0
-0
[ ... ]
Quoted 8 lines
arch/s390/kvm/interrupt.c +7
-0@@ -3632,3 +3700,85 @@ int __init kvm_s390_gib_init(u8 nisc) +/* +* kvm_arch_set_irq_inatomic: fast-path for irqfd injection +*/ +int kvm_arch_set_irq_inatomic (struct kvm_kernel_irq_routing_entry *e, +struct kvm *kvm,int irq_source_id,int level, +bool line_status) +{
[ ... ]
arch/s390/kvm/interrupt.c int __init kvm_s390_gib_init(u8 nisc) · line 3742
Quoted 6 lines
arch/s390/kvm/interrupt.c +6
-0+if (!ret || adapter->masked) { + kvm->stat.io_390_inatomic_adapter_masked_or_coalesced++; +return 0 ; + } + + inti =kzalloc_obj (*inti,GFP_ATOMIC );
[Severity: Medium]
Does this allocation drop the memcg accounting that is present on the slow path (GFP_KERNEL_ACCOUNT)? This might allow unprivileged guests to allocate kernel memory without being charged, bypassing resource limits.
Also, since this fast path runs with interrupts disabled, could the use of GFP_ATOMIC cause lock recursion? GFP_ATOMIC includes __GFP_KSWAPD_RECLAIM, which triggers wakeup_kswapd() and can acquire scheduler-internal or waitqueue locks. Would it be safer to use
(GFP_ATOMIC | __GFP_ACCOUNT) & ~__GFP_KSWAPD_RECLAIM here?
Quoted 5 lines
arch/s390/kvm/interrupt.c +5
-0+if (!inti) { + setbit =0 ; +adapter_indicators_set_fast (kvm, adapter, &e->adapter, setbit); +return -EWOULDBLOCK ; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH v8 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas FreimuthQuoted 16 lines · Douglas Freimuth
s390 needs a fast path for irq injection, and along those lines we introduce
kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the
adapter_indicatorspage needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizesGFP_KERNEL_ACCOUNTbut we had to implement the fast path withGFP_ATOMICallocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Quoted 7 lines
Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including
io_390_inatomic,io_flic_inject_airq,io_set_adapter_intandio_390_inatomic_adapter_masked_or_coalesced.
That's a mouthful. And it also doesn't include suppressed interrupts.
How about something like _no_inject to imply that interrupt was delivered via the adapter interrupt bits without needing to inject the irq.
[...]
Quoted 19 lines
arch/s390/kvm/interrupt.c +9
-3@@ -2719,25 +2728,41 @@ static int kvm_s390_inject_airq(struct kvm *kvm, .parm = 0, .parm64 = isc_to_int_word(adapter->isc), }; +struct kvm_s390_interrupt_info *inti; +unsigned long flags; +int ret =0 ; -if (!test_kvm_facility (kvm,72 ) || !adapter->suppressible) -return kvm_s390_inject_vm (kvm, &s390int); + inti =kzalloc_obj (*inti,GFP_KERNEL_ACCOUNT ); +if (!inti) +return -ENOMEM ; -mutex_lock (&fi->ais_lock); +if (!test_kvm_facility (kvm,72 ) || !adapter->suppressible) { + ret =kvm_s390_inject_vm (kvm, &s390int, inti); +if (ret)
arch/s390/kvm/interrupt.c static int kvm_s390_inject_airq(struct kvm *kvm, · line 2751
Quoted 10 lines
arch/s390/kvm/interrupt.c +7
-1+kfree (inti); +return ret; + } + +spin_lock_irqsave (&fi->ais_lock, flags);if (fi->nimm &AIS_MODE_MASK (adapter->isc)) {trace_kvm_s390_airq_suppressed (adapter->id, adapter->isc); - goto out; +spin_unlock_irqrestore (&fi->ais_lock, flags); +kfree (inti);
I suggested adding this to the same counter as masked and coalesced, and I still think you should: so add a _no_inject++ here
Otherwise, LGTM -- please send a v9 with this and the other small things I mentioned on the other 2 patches.