KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
v2 cover KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
S390 needs this series of three patches in order to enable a non-blocking path for irqfd injection on s390 via kvm_arch_set_irq_inatomic(). Before these changes, kvm_arch_set_irq_inatomic() would just return -EWOULDBLOCK and place all interrupts on the global work queue, which must subsequently be processed by a different thread. This series of patches implements an s390 version of inatomic and is relevant to virtio-blk and virtio-net and was tested against virtio-pci and virtio-ccw.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_adapter_masked. And counters have been added to analyze map/unmap of the adapter_indicator
pages in non-Secure Execution environments and to track fencing of Fast Inject in Secure Execution environments. In order to take advantage of this kernel series with virtio-pci, a QEMU that includes the 's390x/pci: set kvm_msi_via_irqfd_allowed' fix is needed. Additionally, the guest xml needs a thread pool and threads explicitly assigned per disk device using the common way of defining threads for disks.
Patch 1 enables map/unmap of adapter indicator pages but for Secure Execution environments it avoids the long term mapping.
v1->v2: Added kvm_s390_unmap_all_adapters_pv() for case KVM_PV_ENABLE v1->v2: Changed semaphore to spin_lock in Patch 1 and Patch 2 v1->v2: Fixed a build break in Patch 2v1->v2: Added KVM & S390 tags to sub files per Sean v1->v2: Rewrote commit messages to more clearly describe each patch
Douglas Freimuth (3):
Add map/unmap ioctl and clean mappings post-guest Enable adapter_indicators_set to use mapped pages Introducing kvm_arch_set_irq_inatomic fast inject
arch/s390/include/asm/kvm_host.h | 11 +- arch/s390/kvm/interrupt.c | 366 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 44 +++- arch/s390/kvm/kvm-s390.h | 3 +- 4 files changed, 356 insertions(+), 68 deletions(-)
Signature · 1 line
2.52.0
PATCH v2 1/3 KVM: s390: Add map/unmap ioctl and clean mappings post-guest
S390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
--- arch/s390/include/asm/kvm_host.h | 5 ++ arch/s390/kvm/interrupt.c | 145 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 20 +++++ 3 files changed, 144 insertions(+), 26 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -0
@@ -448,6 +448,8 @@ struct kvm_vcpu_arch {struct kvm_vm_stat {struct kvm_vm_stat_generic generic;u64 inject_io; +u64 io_390_adapter_map; +u64 io_390_adapter_unmap;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
@@ -479,6 +481,9 @@ struct s390_io_adapter {bool masked;bool swap;bool suppressible; +spinlock_t maps_lock; +struct list_head maps; +unsigned int nr_maps; };#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)
arch/s390/kvm/interrupt.c +119 -26
182 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +20 -0
@@ -68,6 +68,8 @@const struct kvm_stats_desc kvm_vm_stats_desc[] = {KVM_GENERIC_VM_STATS (),STATS_DESC_COUNTER (VM , inject_io), +STATS_DESC_COUNTER (VM , io_390_adapter_map), +STATS_DESC_COUNTER (VM , io_390_adapter_unmap),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done),STATS_DESC_COUNTER (VM , inject_service_signal),
@@ -2491,6 +2493,23 @@ static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,return r; } +static void kvm_s390_unmap_all_adapters_pv (struct kvm *kvm) +{ +int i; +struct s390_map_info *map, *tmp; + +for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) { +if (!kvm->arch.adapters[i]) +continue ; +list_for_each_entry_safe (map, tmp, + &kvm->arch.adapters[i]->maps, list) { +list_del (&map->list); +put_page (map->page); +kfree (map); + } + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {const bool need_lock = (cmd->cmd !=KVM_PV_ASYNC_CLEANUP_PERFORM );
@@ -2503,6 +2522,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)switch (cmd->cmd) {case KVM_PV_ENABLE : { +kvm_s390_unmap_all_adapters_pv (kvm); r = -EINVAL ;if (kvm_s390_pv_is_protected (kvm))break ;
Signature · 1 line
2.52.0
Christian Borntraeger
maintainerRe: [PATCH v2 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Am 26.03.26 um 02:42 schrieb Douglas Freimuth:
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2484
Quoted 11 lines
arch/s390/kvm/interrupt.c +11
-0+static int kvm_s390_adapter_map (struct kvm *kvm,unsigned int id,__u64 addr) +{ +struct s390_io_adapter *adapter =get_io_adapter (kvm, id); +struct s390_map_info *map; +unsigned long flags; +int ret; + +if (!adapter || !addr) +return -EINVAL ; + + map =kzalloc_obj (*map,GFP_KERNEL );
GFP_KERNEL_ACCOUNT certainly makes sense. Depending on other feedback can be added when applying.
Re: [PATCH v2 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Quoted 18 lines · Christian Borntraeger
Am 26.03.26 um 02:42 schrieb Douglas Freimuth:
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2484
Quoted 12 lines
+static int
kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr)arch/s390/kvm/interrupt.c +10
-0+{ + struct s390_io_adapter *adapter = get_io_adapter(kvm, id); + struct s390_map_info *map; + unsigned long flags; + int ret; + + if (!adapter || !addr) + return -EINVAL; + + map = kzalloc_obj(*map, GFP_KERNEL);
GFP_KERNEL_ACCOUNTcertainly makes sense. Depending on other feedback can be added when applying.
I will make that change.
Janosch Frank
maintainerRe: [PATCH v2 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
In reply to Douglas FreimuthQuoted 8 lines · Douglas Freimuth
S390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 8 lines
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Signed-off-by Douglas Freimuth <[email protected]>---
Looks good, two nits below.
[...]
Quoted 14 lines
if (ret > 0) ret = 0; break;arch/s390/kvm/interrupt.c +3
-6-/* -* The following operations are no longer needed and therefore no-ops. -* The gpa to hva translation is done when an IRQ route is set up. The -* set_irq code uses get_user_pages_remote() to do the actual write. -*/ case KVM_S390_IO_ADAPTER_MAP :case KVM_S390_IO_ADAPTER_UNMAP : - ret =0 ; +/* If in Secure Execution mode do not long term pin. */ +mutex_lock (&dev->kvm->lock); +if (kvm_s390_pv_is_protected (dev->kvm)) {
arch/s390/kvm/interrupt.c static int modify_io_adapter(struct kvm_device *dev, · line 2590
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+mutex_unlock (&dev->kvm->lock); +return 0 ; + } +mutex_unlock (&dev->kvm->lock); + idx =srcu_read_lock (&dev->kvm->srcu); + host_addr =gpa_to_hva (dev->kvm, req.addr); +if (kvm_is_error_hva (host_addr)) { +srcu_read_unlock (&dev->kvm->srcu, idx); +return -EFAULT ; + }
Alignment issue
[...]
arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · line 2525
Quoted 8 lines
static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) { const bool need_lock = (cmd->cmd != KVM_PV_ASYNC_CLEANUP_PERFORM);arch/s390/kvm/kvm-s390.c +1
-0@@ -2503,6 +2522,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)switch (cmd->cmd) {case KVM_PV_ENABLE : { +kvm_s390_unmap_all_adapters_pv (kvm);
Shouldn't this be located after the check that's below?
Quoted 3 lines
arch/s390/kvm/kvm-s390.c +0
-0r = -EINVAL ;if (kvm_s390_pv_is_protected (kvm))break ;
Re: [PATCH v2 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Quoted 48 lines · Janosch Frank
Quoted 8 lines · Douglas Freimuth
S390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 8 lines
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Signed-off-by Douglas Freimuth <[email protected]>---
Looks good, two nits below.
[...]
Quoted 18 lines
if (ret > 0)
ret = 0;
break;
- /*
- * The following operations are no longer needed and therefore no-ops.
- * The gpa to hva translation is done when an IRQ route is set up. The
- *set_irqcode usesget_user_pages_remote()to do the actual write. - */
caseKVM_S390_IO_ADAPTER_MAP:
caseKVM_S390_IO_ADAPTER_UNMAP:
- ret = 0;
+ /* If in Secure Execution mode do not long term pin. */ +mutex_lock(&dev->kvm->lock);
+ if (kvm_s390_pv_is_protected(dev->kvm)) {
+mutex_unlock(&dev->kvm->lock);
+ return 0;Quoted 8 lines
+ }
+mutex_unlock(&dev->kvm->lock);
+ idx =srcu_read_lock(&dev->kvm->srcu); +host_addr=gpa_to_hva(dev->kvm, req.addr); + if (kvm_is_error_hva(host_addr)) { +srcu_read_unlock(&dev->kvm->srcu, idx); + return-EFAULT;
+ }
Quoted 4 lines
Alignment issue
[...]
I will correct that. (Surprised checkpatch strict didn't catch it.)
Quoted 11 lines
arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, · line 2525
Quoted 8 lines
static int
kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd){
const boolneed_lock= (cmd->cmd!=KVM_PV_ASYNC_CLEANUP_PERFORM);arch/s390/kvm/kvm-s390.c +1
-0@@ -2503,6 +2522,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) switch (cmd->cmd) { case KVM_PV_ENABLE: { + kvm_s390_unmap_all_adapters_pv(kvm);Shouldn't this be located after the check that's below?
While all of my tests do not observe it, not sure if there is a state change where anything is left mapped while in pv. There should not be so I can move it after the check as you mention.
Quoted 4 lines
Quoted 3 lines
arch/s390/kvm/kvm-s390.c +0
-0r = -EINVAL; if (kvm_s390_pv_is_protected(kvm)) break;
Re: [PATCH v2 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
In reply to Douglas FreimuthQuoted 8 lines · Douglas Freimuth
S390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 7 lines
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Signed-off-by Douglas Freimuth <[email protected]>
FYI, in addition to my own review I looked thru what Sashiko found to see if anything was valid:
https://sashiko.dev/#/patchset/20260326014247.2085-1-freimuth%40linux.ibm.com
Quoted 80 lines
--- arch/s390/include/asm/kvm_host.h | 5 ++ arch/s390/kvm/interrupt.c | 145 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 20 +++++ 3 files changed, 144 insertions(+), 26 deletions(-)arch/s390/include/asm/kvm_host.h +5
-0@@ -448,6 +448,8 @@ struct kvm_vcpu_arch {struct kvm_vm_stat {struct kvm_vm_stat_generic generic;u64 inject_io; +u64 io_390_adapter_map; +u64 io_390_adapter_unmap;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;@@ -479,6 +481,9 @@ struct s390_io_adapter {bool masked;bool swap;bool suppressible; +spinlock_t maps_lock; +struct list_head maps; +unsigned int nr_maps; };#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)arch/s390/kvm/interrupt.c +36
-0@@ -2426,6 +2426,9 @@ static int register_io_adapter(struct kvm_device *dev,if (!adapter)return -ENOMEM ; +INIT_LIST_HEAD (&adapter->maps); +spin_lock_init (&adapter->maps_lock); + adapter->nr_maps =0 ; adapter->id = adapter_info.id; adapter->isc = adapter_info.isc; adapter->maskable = adapter_info.maskable;@@ -2450,12 +2453,104 @@ int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked)return ret; } +static struct page *get_map_page (struct kvm *kvm,u64 uaddr) +{ +struct mm_struct *mm = kvm->mm; +struct page *page =NULL ; +int locked =1 ; + +if (mmget_not_zero (mm)) { +mmap_read_lock (mm); +get_user_pages_remote (mm, uaddr,1 ,FOLL_WRITE , + &page, &locked); +if (locked) +mmap_read_unlock (mm); +mmput (mm); + } + +return page; +} + +static int kvm_s390_adapter_map (struct kvm *kvm,unsigned int id,__u64 addr) +{ +struct s390_io_adapter *adapter =get_io_adapter (kvm, id); +struct s390_map_info *map; +unsigned long flags; +int ret; + +if (!adapter || !addr) +return -EINVAL ; + + map =kzalloc_obj (*map,GFP_KERNEL ); +if (!map) +return -ENOMEM ; + + map->page =get_map_page (kvm, addr);
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2498
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+if (!map->page) { + ret = -EINVAL ; + goto out; + } + +INIT_LIST_HEAD (&map->list); + map->guest_addr = addr; + map->addr = addr; +spin_lock_irqsave (&adapter->maps_lock, flags); +if (adapter->nr_maps++ <MAX_S390_ADAPTER_MAPS ) {
Sashiko output correctly points out that here we keep incrementing nr_maps even once we hit MAX_S390_ADAPTER_MAPS for subsequent calls to this ioctl.
What it missed is that, besides an overflow you might also cause an underflow during unmap due to nr_maps exceeding the actual number of entries in the list.
You should check the current value FIRST and only increment nr_maps if we add to the list.
Quoted 19 lines
arch/s390/kvm/interrupt.c +19
-0+list_add_tail (&map->list, &adapter->maps); + ret =0 ; + }else { +put_page (map->page); + ret = -EINVAL ; + } +spin_unlock_irqrestore (&adapter->maps_lock, flags); +out: +if (ret) +kfree (map); +return ret; +} + +static int kvm_s390_adapter_unmap (struct kvm *kvm,unsigned int id,__u64 addr) +{ +struct s390_io_adapter *adapter =get_io_adapter (kvm, id); +struct s390_map_info *map, *tmp; +unsigned long flags; +int found =0 ;
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2527
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+ +if (!adapter || !addr) +return -EINVAL ; + +spin_lock_irqsave (&adapter->maps_lock, flags); +list_for_each_entry_safe (map, tmp, &adapter->maps, list) { +if (map->guest_addr == addr) { + found =1 ; + adapter->nr_maps--; +list_del (&map->list);
Before we unpin this via put_page you need to mark_page_dirty() and set_page_dirty_lock().
Because we are not doing this inline like the 'slow path' it is possible that we end up marking a page dirty that was mapped but never actually had bits indicated. That's better than not dirtying pages that did change.
BTW this should also be done as part of kvm_s390_unmap_all_adapters_pv.
Quoted 48 lines
arch/s390/kvm/interrupt.c +23
-6+put_page (map->page); +kfree (map); +break ; + } + } +spin_unlock_irqrestore (&adapter->maps_lock, flags); + +return found ?0 : -ENOENT ; +} +void kvm_s390_destroy_adapters (struct kvm *kvm) {int i; +struct s390_map_info *map, *tmp; -for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) +for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) { +if (!kvm->arch.adapters[i]) +continue ; +list_for_each_entry_safe (map, tmp, + &kvm->arch.adapters[i]->maps, list) { +list_del (&map->list); +put_page (map->page); +kfree (map); + }kfree (kvm->arch.adapters[i]); + } } static int modify_io_adapter(struct kvm_device *dev,@@ -2463,7 +2558,8 @@ static int modify_io_adapter(struct kvm_device *dev, {struct kvm_s390_io_adapter_req req;struct s390_io_adapter *adapter; -int ret; +__u64 host_addr; +int ret, idx;if (copy_from_user (&req, (void __user *)attr->addr,sizeof (req)))return -EFAULT ;@@ -2477,14 +2573,29 @@ static int modify_io_adapter(struct kvm_device *dev,if (ret >0 ) ret =0 ;break ; -/* -* The following operations are no longer needed and therefore no-ops. -* The gpa to hva translation is done when an IRQ route is set up. The -* set_irq code uses get_user_pages_remote() to do the actual write.
arch/s390/kvm/interrupt.c static int modify_io_adapter(struct kvm_device *dev, · line 2583
Quoted 10 lines
arch/s390/kvm/interrupt.c +6
-2-*/ case KVM_S390_IO_ADAPTER_MAP :case KVM_S390_IO_ADAPTER_UNMAP : - ret =0 ; +/* If in Secure Execution mode do not long term pin. */ +mutex_lock (&dev->kvm->lock); +if (kvm_s390_pv_is_protected (dev->kvm)) { +mutex_unlock (&dev->kvm->lock); +return 0 ; + }
I believe sashiko has a point here as well.
You rely on the fact that once the VM goes into protected mode all previously-pinned mappings will be released. While the list itself is protected by a spinlock the Secure Exeuction mode indicator is not and can change once you drop the kvm lock here.
So what could happen here is you drop the kvm lock and proceed to create this last mapping but a separate thread of execution acquires the KVM lock and calls kvm_s390_unmap_all_adapters_pv, releasing all old adapter mappings. And then finally the first thread of execution gets around to performing the contents of kvm_s390_adapter_map() and now you are left with a long-term mapping for a VM in SE mode.
To resolve I believe you need to hold the kvm lock over the actual mapping operation / addition to the list to ensure that the mode change does not happen until after that point -- then your code in kvm_s390_unmap_all_adapters_pv will be guaranteed to pick up that last adapter mapping that just got added.
You don't strictly have to do that for unmap (since we also do a wholesale unmap when going into SE mode), but it's probably just safer to do so in order to avoid unexpected error returns on the ioctl.
Quoted 57 lines
arch/s390/kvm/interrupt.c +15
-18+mutex_unlock (&dev->kvm->lock); + idx =srcu_read_lock (&dev->kvm->srcu); + host_addr =gpa_to_hva (dev->kvm, req.addr); +if (kvm_is_error_hva (host_addr)) { +srcu_read_unlock (&dev->kvm->srcu, idx); +return -EFAULT ; + } +srcu_read_unlock (&dev->kvm->srcu, idx); +if (req.type ==KVM_S390_IO_ADAPTER_MAP ) { + dev->kvm->stat.io_390_adapter_map++; + ret =kvm_s390_adapter_map (dev->kvm, req.id, host_addr); + }else { + dev->kvm->stat.io_390_adapter_unmap++; + ret =kvm_s390_adapter_unmap (dev->kvm, req.id, host_addr); + }break ;default : ret = -EINVAL ;@@ -2730,24 +2841,6 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } -static struct page *get_map_page (struct kvm *kvm,u64 uaddr) -{ -struct mm_struct *mm = kvm->mm; -struct page *page =NULL ; -int locked =1 ; - -if (mmget_not_zero (mm)) { -mmap_read_lock (mm); -get_user_pages_remote (mm, uaddr,1 ,FOLL_WRITE , - &page, &locked); -if (locked) -mmap_read_unlock (mm); -mmput (mm); - } - -return page; -} -static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int)arch/s390/kvm/kvm-s390.c +2
-0@@ -68,6 +68,8 @@const struct kvm_stats_desc kvm_vm_stats_desc[] = {KVM_GENERIC_VM_STATS (),STATS_DESC_COUNTER (VM , inject_io), +STATS_DESC_COUNTER (VM , io_390_adapter_map), +STATS_DESC_COUNTER (VM , io_390_adapter_unmap),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done),STATS_DESC_COUNTER (VM , inject_service_signal),
arch/s390/kvm/kvm-s390.c static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd, · line 2500
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +5
-0@@ -2491,6 +2493,23 @@ static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,return r; } +static void kvm_s390_unmap_all_adapters_pv (struct kvm *kvm) +{ +int i; +struct s390_map_info *map, *tmp; +
Sashiko also correctly points out that you are not acquiring the adapter->maps_lock to protect the list here. Please add.
arch/s390/kvm/kvm-s390.c static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd, · line 2506
Quoted 6 lines
arch/s390/kvm/kvm-s390.c +6
-0+for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) { +if (!kvm->arch.adapters[i]) +continue ; +list_for_each_entry_safe (map, tmp, + &kvm->arch.adapters[i]->maps, list) { +list_del (&map->list);
As mentioned above, mark_page_dirty() and set_page_dirty_lock() before put.
Quoted 17 lines
arch/s390/kvm/kvm-s390.c +7
-0+put_page (map->page); +kfree (map); + } + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {const bool need_lock = (cmd->cmd !=KVM_PV_ASYNC_CLEANUP_PERFORM );@@ -2503,6 +2522,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)switch (cmd->cmd) {case KVM_PV_ENABLE : { +kvm_s390_unmap_all_adapters_pv (kvm); r = -EINVAL ;if (kvm_s390_pv_is_protected (kvm))break ;
PATCH v2 2/3 KVM: s390: Enable adapter_indicators_set to use mapped pages
In reply to Douglas FreimuthThe S390 adapter_indicators_set function needs to be able to use mapped pages so that worked can be processed,on a fast path when interrupts are disabled. If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.
--- arch/s390/kvm/interrupt.c | 87 ++++++++++++++++++++++++++++----------- 1 file changed, 62 insertions(+), 25 deletions(-)
arch/s390/kvm/interrupt.c +62 -25
@@ -2841,41 +2841,75 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } +static struct s390_map_info *get_map_info (struct s390_io_adapter *adapter, +u64 addr) +{ +struct s390_map_info *map; + +if (!adapter) +return NULL ; + +list_for_each_entry (map, &adapter->maps, list) { +if (map->guest_addr == addr) +return map; + } +return NULL ; +} +static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {unsigned long bit;int summary_set, idx; -struct page *ind_page, *summary_page; +struct s390_map_info *ind_info, *summary_info;void *map; +struct page *ind_page, *summary_page; - ind_page =get_map_page (kvm, adapter_int->ind_addr); -if (!ind_page) -return -1 ; - summary_page =get_map_page (kvm, adapter_int->summary_addr); -if (!summary_page) { -put_page (ind_page); -return -1 ; + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) { + ind_page =get_map_page (kvm, adapter_int->ind_addr); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page); +srcu_read_unlock (&kvm->srcu, idx); + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); + } + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { + summary_page =get_map_page (kvm, adapter_int->summary_addr); +if (!summary_page) { +put_page (ind_page); +return -1 ; + } + idx =srcu_read_lock (&kvm->srcu); + map =page_address (summary_page); + bit =get_ind_bit (adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap); + summary_set =test_and_set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (summary_page); +srcu_read_unlock (&kvm->srcu, idx); + }else { + map =page_address (summary_info->page); + bit =get_ind_bit (summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set =test_and_set_bit (bit, map); } - idx =srcu_read_lock (&kvm->srcu); - map =page_address (ind_page); - bit =get_ind_bit (adapter_int->ind_addr, - adapter_int->ind_offset, adapter->swap); -set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (ind_page); - map =page_address (summary_page); - bit =get_ind_bit (adapter_int->summary_addr, - adapter_int->summary_offset, adapter->swap); - summary_set =test_and_set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (summary_page); -srcu_read_unlock (&kvm->srcu, idx); - -put_page (ind_page); -put_page (summary_page); +if (!ind_info) +put_page (ind_page); +if (!summary_info) +put_page (summary_page);return summary_set ?0 :1 ; }
@@ -2890,6 +2924,7 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, {int ret;struct s390_io_adapter *adapter; +unsigned long flags;/* We're only interested in the 0->1 transition. */ if (!level)
@@ -2897,7 +2932,9 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, adapter =get_io_adapter (kvm, e->adapter.adapter_id);if (!adapter)return -1 ; +spin_lock_irqsave (&adapter->maps_lock, flags); ret =adapter_indicators_set (kvm, adapter, &e->adapter); +spin_unlock_irqrestore (&adapter->maps_lock, flags);if ((ret >0 ) && !adapter->masked) { ret =kvm_s390_inject_airq (kvm, adapter);if (ret ==0 )
Signature · 1 line
2.52.0
Re: [PATCH v2 2/3] KVM: s390: Enable adapter_indicators_set to use mapped pages
Quoted 9 lines · Douglas Freimuth
The S390
adapter_indicators_setfunction needs to be able to use mapped pages so that worked can be processed,on a fast path when interrupts are disabled. If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.Signed-off-by Douglas Freimuth <[email protected]>---
[...]
Quoted 20 lines
static int adapter_indicators_set(struct kvm *kvm,
arch/s390/kvm/interrupt.c +4
-8struct s390_io_adapter *adapter, struct kvm_s390_adapter_int *adapter_int) { unsigned long bit;int summary_set, idx; -struct page *ind_page, *summary_page; +struct s390_map_info *ind_info, *summary_info;void *map; +struct page *ind_page, *summary_page; - ind_page =get_map_page (kvm, adapter_int->ind_addr); -if (!ind_page) -return -1 ; - summary_page =get_map_page (kvm, adapter_int->summary_addr); -if (!summary_page) { -put_page (ind_page); -return -1 ; + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) {
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2880
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+ ind_page =get_map_page (kvm, adapter_int->ind_addr); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page);
Sashiko's comments about this area of code should already be addressed by my comments in v1 wrt to marking the page dirty before unpinning.
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2891
Quoted 11 lines
arch/s390/kvm/interrupt.c +11
-0+srcu_read_unlock (&kvm->srcu, idx); + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); + } + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { + summary_page =get_map_page (kvm, adapter_int->summary_addr); +if (!summary_page) { +put_page (ind_page);
Sashiko has a point here. It is possible that you may have taken the else path above where ind_info != NULL and never used ind_page.
Initialize ind_page to a NULL value first and check for a NULL value here before attempting a put_page.
Or I think you can also check (!ind_info) and only put the page in this case like you do later in this function.
[...]
Quoted 10 lines
arch/s390/kvm/interrupt.c +1
-0@@ -2890,6 +2924,7 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, {int ret;struct s390_io_adapter *adapter; +unsigned long flags;/* We're only interested in the 0->1 transition. */ if (!level)
arch/s390/kvm/interrupt.c static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, · line 2937
Quoted 6 lines
arch/s390/kvm/interrupt.c +2
-0@@ -2897,7 +2932,9 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, adapter =get_io_adapter (kvm, e->adapter.adapter_id);if (!adapter)return -1 ; +spin_lock_irqsave (&adapter->maps_lock, flags); ret =adapter_indicators_set (kvm, adapter, &e->adapter); +spin_unlock_irqrestore (&adapter->maps_lock, flags);
Another point for Sashiko -- This is almost certainly a fallout from my recommendation to drop the semaphore from patch 1 and go straight to a spinlock vs adding semaphore in patch 1 and converting in patch 3.
You can't put a spinlock here as you may lose control inside adapter_indicators_set. You resolve the incompatibility in the next patch by moving the spinlock inside the call and dropping it before you could lose control -- but you have to already account for that in this patch.
Quoted 3 lines
arch/s390/kvm/interrupt.c +0
-0if ((ret >0 ) && !adapter->masked) { ret =kvm_s390_inject_airq (kvm, adapter);if (ret ==0 )
PATCH v2 3/3 KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas FreimuthS390 needs a fast path for irq injection, and along those lines we introduce kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_adapter_masked.
--- arch/s390/include/asm/kvm_host.h | 6 +- arch/s390/kvm/interrupt.c | 154 +++++++++++++++++++++++++++---- arch/s390/kvm/kvm-s390.c | 24 ++++- arch/s390/kvm/kvm-s390.h | 3 +- 4 files changed, 160 insertions(+), 27 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -1
@@ -359,7 +359,7 @@ struct kvm_s390_float_interrupt {struct kvm_s390_mchk_info mchk;struct kvm_s390_ext_info srv_signal;int last_sleep_cpu; -struct mutex ais_lock; +spinlock_t ais_lock;u8 simm;u8 nimm; };
@@ -450,6 +450,10 @@ struct kvm_vm_stat {u64 inject_io;u64 io_390_adapter_map;u64 io_390_adapter_unmap; +u64 io_390_inatomic; +u64 io_flic_inject_airq; +u64 io_set_adapter_int; +u64 io_390_inatomic_adapter_masked;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
arch/s390/kvm/interrupt.c +134 -20
288 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +19 -5
@@ -70,6 +70,10 @@ const struct kvm_stats_desc kvm_vm_stats_desc[] = {STATS_DESC_COUNTER (VM , inject_io),STATS_DESC_COUNTER (VM , io_390_adapter_map),STATS_DESC_COUNTER (VM , io_390_adapter_unmap), +STATS_DESC_COUNTER (VM , io_390_inatomic), +STATS_DESC_COUNTER (VM , io_flic_inject_airq), +STATS_DESC_COUNTER (VM , io_set_adapter_int), +STATS_DESC_COUNTER (VM , io_390_inatomic_adapter_masked),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done), STATS_DESC_COUNTER(VM, inject_service_signal),
@@ -2862,6 +2866,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)void __user *argp = (void __user *)arg;struct kvm_device_attr attr;int r; +struct kvm_s390_interrupt_info *inti;switch (ioctl) {case KVM_S390_INTERRUPT : {
@@ -2870,7 +2875,10 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) r = -EFAULT ;if (copy_from_user (&s390int, argp,sizeof (s390int)))break ; - r =kvm_s390_inject_vm (kvm, &s390int); + inti =kzalloc_obj (*inti,GFP_KERNEL_ACCOUNT ); +if (!inti) +return -ENOMEM ; + r =kvm_s390_inject_vm (kvm, &s390int, inti);break ; }case KVM_CREATE_IRQCHIP : {
@@ -3268,7 +3276,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)mutex_unlock (&kvm->lock); } -mutex_init (&kvm->arch.float_int.ais_lock); +spin_lock_init (&kvm->arch.float_int.ais_lock);spin_lock_init (&kvm->arch.float_int.lock);for (i =0 ; i <FIRQ_LIST_COUNT ; i++)INIT_LIST_HEAD (&kvm->arch.float_int.lists[i]);
@@ -4389,11 +4397,16 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_cloreturn 1 ; } -static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token) +static int __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, +unsigned long token) {struct kvm_s390_interrupt inti;struct kvm_s390_irq irq; +struct kvm_s390_interrupt_info *inti_mem; + + inti_mem =kzalloc_obj (*inti_mem,GFP_KERNEL_ACCOUNT ); +if (!inti_mem) +return -ENOMEM ;if (start_token) { irq.u.ext.ext_params2 = token;
@@ -4402,8 +4415,9 @@ static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, }else { inti.type =KVM_S390_INT_PFAULT_DONE ; inti.parm64 = token; -WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti)); +WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti, inti_mem)); } +return true; } bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
arch/s390/kvm/kvm-s390.h +2 -1
@@ -373,7 +373,8 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu);void kvm_s390_clear_local_irqs (struct kvm_vcpu *vcpu);void kvm_s390_clear_float_irqs (struct kvm *kvm);int __must_check kvm_s390_inject_vm (struct kvm *kvm, -struct kvm_s390_interrupt *s390int); +struct kvm_s390_interrupt *s390int, +struct kvm_s390_interrupt_info *inti);int __must_check kvm_s390_inject_vcpu (struct kvm_vcpu *vcpu,struct kvm_s390_irq *irq); static inline int kvm_s390_inject_prog_irq(struct kvm_vcpu *vcpu,
Signature · 1 line
2.52.0
Re: [PATCH v2 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
arch/s390/kvm/interrupt.c static int __inject_vm(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) · line 1966
Quoted 6 lines · Douglas Freimuth
arch/s390/kvm/interrupt.c +1
-1@@ -1963,15 +1963,10 @@ static int __inject_vm(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) }int kvm_s390_inject_vm (struct kvm *kvm, -struct kvm_s390_interrupt *s390int) +struct kvm_s390_interrupt *s390int,struct kvm_s390_interrupt_info *inti)
Sashiko makes a point here, sort of.
I believe it's wrong in the sense that you still left the code in this function that frees inti on error paths, effectively freeing it under the exact same circumstances as before this patch when it was allocated in kvm_s390_inject_vm.
But now that you have moved allocation of inti to be the responsibility of the caller of kvm_s390_inject_vm it generally would imply that the caller should ALSO be responsible for freeing that memory. But now both the passing case and the failing case have given up control of that memory and assume it's taken care of.
I think that's worth a comment explaining it.
[...]
Quoted 20 lines
arch/s390/kvm/interrupt.c +20
-0+/* +* kvm_arch_set_irq_inatomic: fast-path for irqfd injection +*/ +int kvm_arch_set_irq_inatomic (struct kvm_kernel_irq_routing_entry *e, +struct kvm *kvm,int irq_source_id,int level, +bool line_status) +{ +int ret; +struct s390_io_adapter *adapter; +struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; +struct kvm_s390_interrupt_info *inti; +struct kvm_s390_interrupt s390int = { + .type =KVM_S390_INT_IO (1 ,0 ,0 ,0 ), + .parm =0 , + }; + + kvm->stat.io_390_inatomic++; + +/* We're only interested in the 0->1 transition. */ +if (!level)
arch/s390/kvm/interrupt.c int __init kvm_s390_gib_init(u8 nisc) · line 3665
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+return -EWOULDBLOCK ; +if (e->type !=KVM_IRQ_ROUTING_S390_ADAPTER ) +return -EWOULDBLOCK ; + + adapter =get_io_adapter (kvm, e->adapter.adapter_id); +if (!adapter) +return -EWOULDBLOCK ; + + s390int.parm64 =isc_to_int_word (adapter->isc); + ret =adapter_indicators_set_fast (kvm, adapter, &e->adapter);
Sashiko asks an interesting question around this point. I think it's a bit more broad than it states, however:
What happens if we -EWOULDBLOCK for any reason past this point and adapter_indicators_set_fast above here has set the summary bit.
If we then fall back to the 'slow path' and the code in adapter_indicators_set(), we could possibly see the summary bit was already set (by adapter_indicators_set_fast() above) and assume that we don't need to inject anything because the thread of executing that set the summary bit ALSO did the inject. But in this particular case we actually never did.
Do we need to unset the summary bit for these subsequent -EWOULDBLOCK cases so that the subsequent fallback path will know it must inject? But we should only undo this if adapter_indicators_set_fast() actually set the summary bit in the first place.
The indicator bits themselves that might have been set by adapter_indicators_set_fast() should be fine to leave on, injection decision is based on the setting of the summary indicator.
Quoted 23 lines
arch/s390/kvm/interrupt.c +23
-0+if (ret <0 ) +return -EWOULDBLOCK ; +if (!ret || adapter->masked) { + kvm->stat.io_390_inatomic_adapter_masked++; +return 0 ; + } + + inti =kzalloc_obj (*inti,GFP_ATOMIC ); +if (!inti) +return -EWOULDBLOCK ; + +if (!test_kvm_facility (kvm,72 ) || !adapter->suppressible) { + ret =kvm_s390_inject_vm (kvm, &s390int, inti); +if (ret ==0 ) +return ret; +else +return -EWOULDBLOCK ; + } + +spin_lock (&fi->ais_lock); +if (fi->nimm &AIS_MODE_MASK (adapter->isc)) { +trace_kvm_s390_airq_suppressed (adapter->id, adapter->isc); +kfree (inti);
arch/s390/kvm/interrupt.c int __init kvm_s390_gib_init(u8 nisc) · line 3698
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+ goto out; + } + + ret =kvm_s390_inject_vm (kvm, &s390int, inti); +if (!ret && (fi->simm &AIS_MODE_MASK (adapter->isc))) { + fi->nimm |=AIS_MODE_MASK (adapter->isc); +trace_kvm_s390_modify_ais_mode (adapter->isc, +KVM_S390_AIS_MODE_SINGLE ,2 ); + } + goto out;
Goto is unncessary here.
Also, you don't check returns from kvm_s390_inject_vm on this path? If any errors we still return 0 to caller (compare this to the call above where any errors would instead yield -EWOULDBLOCK) Can you please double check this?
[...]
arch/s390/kvm/kvm-s390.c int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo · line 4401
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +2
-2-static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token) +static int __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, +unsigned long token)
You updated this to return a value but AFAICT you never update the callers of this function to actually handle the return value.