KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
v3 cover KVM: s390: Introducing kvm_arch_set_irq_inatomic Fast Inject
S390 needs this series of three patches in order to enable a non-blocking path for irqfd injection on s390 via kvm_arch_set_irq_inatomic(). Before these changes, kvm_arch_set_irq_inatomic() would just return -EWOULDBLOCK and place all interrupts on the global work queue, which must subsequently be processed by a different thread. This series of patches implements an s390 version of inatomic and is relevant to virtio-blk and virtio-net and was tested against virtio-pci and virtio-ccw.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_adapter_masked. And counters have been added to analyze map/unmap of the adapter_indicator
pages in non-Secure Execution environments and to track fencing of Fast Inject in Secure Execution environments. In order to take advantage of this kernel series with virtio-pci, a QEMU that includes the 's390x/pci: set kvm_msi_via_irqfd_allowed' fix is needed. Additionally, the guest xml needs a thread pool and threads explicitly assigned per disk device using the common way of defining threads for disks.
Patch 1 enables map/unmap of adapter indicator pages but for Secure Execution environments it avoids the long term mapping.
v2->v3: GFP_KERNEL to GFP_KERNEL_ACCOUNT in one instance of allocation. v2->v3: Fix alignment error.v2->v3: Increment nr_maps after new map added to listv2->v3: kvm_s390_adapter_unmap do mark_page_dirty and set_page_dirty_lock. v2->v3: In unmap_all_adapters_pv do mark_page_dirty, set_page_dirty_lock. v2->v3: Move kvm_s390_unmap_all_adapters_pv() to after check if in pv. v2->v3: Move mutex_unlock after map/unmap in modify_io_adapter. v2->v3: Add spin_lock to get maps->lock in kvm_s390_unmap_all_adapters_pv. v2->v3: Only put_page(ind_page) if !ind_info which allocates ind_page. v2->v3: Move the spin_lock inside of the adapter_indicators_set in patch 2. v2->v3: On last conditional in kvm_arch_set_irq_inatomic, add else clause. v2->v3: Clear ind/summ bits if inject fails upon return to inatomic.
Douglas Freimuth (3):
Add map/unmap ioctl and clean mappings post-guest Enable adapter_indicators_set to use mapped pages Introducing kvm_arch_set_irq_inatomic fast inject
arch/s390/include/asm/kvm_host.h | 11 +- arch/s390/kvm/interrupt.c | 392 ++++++++++++++++++++++++++----- arch/s390/kvm/kvm-s390.c | 51 +++- arch/s390/kvm/kvm-s390.h | 3 +- 4 files changed, 387 insertions(+), 70 deletions(-)
Signature · 1 line
2.52.0
PATCH v3 1/3 KVM: s390: Add map/unmap ioctl and clean mappings post-guest
S390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
--- arch/s390/include/asm/kvm_host.h | 5 + arch/s390/kvm/interrupt.c | 151 +++++++++++++++++++++++++------ arch/s390/kvm/kvm-s390.c | 27 ++++++ 3 files changed, 157 insertions(+), 26 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -0
@@ -448,6 +448,8 @@ struct kvm_vcpu_arch {struct kvm_vm_stat {struct kvm_vm_stat_generic generic;u64 inject_io; +u64 io_390_adapter_map; +u64 io_390_adapter_unmap;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
@@ -479,6 +481,9 @@ struct s390_io_adapter {bool masked;bool swap;bool suppressible; +spinlock_t maps_lock; +struct list_head maps; +unsigned int nr_maps; };#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8)
arch/s390/kvm/interrupt.c +125 -26
188 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +27 -0
@@ -68,6 +68,8 @@const struct kvm_stats_desc kvm_vm_stats_desc[] = {KVM_GENERIC_VM_STATS (),STATS_DESC_COUNTER (VM , inject_io), +STATS_DESC_COUNTER (VM , io_390_adapter_map), +STATS_DESC_COUNTER (VM , io_390_adapter_unmap),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done),STATS_DESC_COUNTER (VM , inject_service_signal),
@@ -2491,6 +2493,30 @@ static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,return r; } +static void kvm_s390_unmap_all_adapters_pv (struct kvm *kvm) +{ +unsigned long flags; +struct s390_map_info *map, *tmp; +int i, idx; + +for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) { +if (!kvm->arch.adapters[i]) +continue ; +list_for_each_entry_safe (map, tmp, + &kvm->arch.adapters[i]->maps, list) { +spin_lock_irqsave (&kvm->arch.adapters[i]->maps_lock, flags); +list_del (&map->list); +spin_unlock_irqrestore (&kvm->arch.adapters[i]->maps_lock, flags); + idx =srcu_read_lock (&kvm->srcu); +mark_page_dirty (kvm, map->addr >>PAGE_SHIFT ); +set_page_dirty_lock (map->page); +srcu_read_unlock (&kvm->srcu, idx); +put_page (map->page); +kfree (map); + } + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {const bool need_lock = (cmd->cmd !=KVM_PV_ASYNC_CLEANUP_PERFORM );
@@ -2507,6 +2533,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)if (kvm_s390_pv_is_protected (kvm))break ; +kvm_s390_unmap_all_adapters_pv (kvm);mmap_write_lock (kvm->mm);/* * Disable creation of new THPs. Existing THPs can stay, they
Signature · 1 line
2.52.0
Re: [PATCH v3 1/3] KVM: s390: Add map/unmap ioctl and clean mappings post-guest
Quoted 15 lines · Douglas Freimuth
S390 needs map/unmap ioctls, which map the adapter set indicator pages, so the pages can be accessed when interrupts are disabled. The mappings are cleaned up when the guest is removed.
Map/Unmap ioctls are fenced in order to avoid the longterm pinning in Secure Execution environments. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters to count map/unmap functions for adapter indicator pages are added. The counters can be used to analyze map/unmap functions in non-Secure Execution environments and similarly can be used to analyze Secure Execution environments where the counters will not be incremented as the adapter indicator pages are not mapped.
Signed-off-by Douglas Freimuth <[email protected]>
[...]
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2525
Quoted 13 lines
arch/s390/kvm/interrupt.c +13
-0+static int kvm_s390_adapter_unmap (struct kvm *kvm,unsigned int id,__u64 addr) +{ +struct s390_io_adapter *adapter =get_io_adapter (kvm, id); +struct s390_map_info *map, *tmp; +unsigned long flags; +int found =0 , idx; + +if (!adapter || !addr) +return -EINVAL ; + +list_for_each_entry_safe (map, tmp, &adapter->maps, list) { +if (map->guest_addr == addr) { +spin_lock_irqsave (&adapter->maps_lock, flags);
This lock needs to be acquired before the list_for_each_entry_safe call, that way it protects the list from changing until after you delete the specified entry.
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2531
Quoted 6 lines
arch/s390/kvm/interrupt.c +6
-0+ found =1 ; + adapter->nr_maps--; +list_del (&map->list); +spin_unlock_irqrestore (&adapter->maps_lock, flags); + idx =srcu_read_lock (&kvm->srcu); +mark_page_dirty (kvm, map->addr >>PAGE_SHIFT );
This isn't the right address to mark dirty. Per
898885477e0f KVM: s390: Use guest address to mark guest page dirty
You need to keep track of the gaddr and use that to mark the page dirty.
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+set_page_dirty_lock (map->page); +srcu_read_unlock (&kvm->srcu, idx); +put_page (map->page); +kfree (map); +break ; + } + } + +return found ?0 : -ENOENT ; +}
arch/s390/kvm/interrupt.c int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) · line 2550
Quoted 10 lines
arch/s390/kvm/interrupt.c +5
-1+void kvm_s390_destroy_adapters (struct kvm *kvm) {int i; +struct s390_map_info *map, *tmp; -for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) +for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) { +if (!kvm->arch.adapters[i]) +continue ;
Should also be holding the maps_lock over this until the list is emptied (e.g. right until the kfree(kvm->arch.adapters[i] call)
Quoted 33 lines
arch/s390/kvm/interrupt.c +9
-7+list_for_each_entry_safe (map, tmp, + &kvm->arch.adapters[i]->maps, list) { +list_del (&map->list); +put_page (map->page); +kfree (map); + }kfree (kvm->arch.adapters[i]); + } } static int modify_io_adapter(struct kvm_device *dev,@@ -2463,7 +2563,8 @@ static int modify_io_adapter(struct kvm_device *dev, {struct kvm_s390_io_adapter_req req;struct s390_io_adapter *adapter; -int ret; +__u64 host_addr; +int ret, idx;if (copy_from_user (&req, (void __user *)attr->addr,sizeof (req)))return -EFAULT ;@@ -2477,14 +2578,30 @@ static int modify_io_adapter(struct kvm_device *dev,if (ret >0 ) ret =0 ;break ; -/* -* The following operations are no longer needed and therefore no-ops. -* The gpa to hva translation is done when an IRQ route is set up. The -* set_irq code uses get_user_pages_remote() to do the actual write. -*/ case KVM_S390_IO_ADAPTER_MAP :case KVM_S390_IO_ADAPTER_UNMAP : - ret =0 ;
arch/s390/kvm/interrupt.c static int modify_io_adapter(struct kvm_device *dev, · line 2592
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+/* If in Secure Execution mode do not long term pin. */ +mutex_lock (&dev->kvm->lock); +if (kvm_s390_pv_is_protected (dev->kvm)) { +mutex_unlock (&dev->kvm->lock); +return 0 ; + } + idx =srcu_read_lock (&dev->kvm->srcu); + host_addr =gpa_to_hva (dev->kvm, req.addr); +if (kvm_is_error_hva (host_addr)) { +srcu_read_unlock (&dev->kvm->srcu, idx);
dev->kvm->lock also needs to be dropped here.
arch/s390/kvm/interrupt.c static int modify_io_adapter(struct kvm_device *dev, · line 2599
Quoted 7 lines
arch/s390/kvm/interrupt.c +7
-0+return -EFAULT ; + } +srcu_read_unlock (&dev->kvm->srcu, idx); +if (req.type ==KVM_S390_IO_ADAPTER_MAP ) { + dev->kvm->stat.io_390_adapter_map++; + ret =kvm_s390_adapter_map (dev->kvm, req.id, host_addr); +mutex_unlock (&dev->kvm->lock);
This unlock...
arch/s390/kvm/interrupt.c static int modify_io_adapter(struct kvm_device *dev, · line 2603
Quoted 4 lines
arch/s390/kvm/interrupt.c +4
-0+ }else { + dev->kvm->stat.io_390_adapter_unmap++; + ret =kvm_s390_adapter_unmap (dev->kvm, req.id, host_addr); +mutex_unlock (&dev->kvm->lock);
and this unlock...
Review static int modify_io_adapter(struct kvm_device *dev, · line 2604
Quoted 1 line
(quoted diff) +1
-0+ }
Could be combined and moved here
Quoted 47 lines
arch/s390/kvm/interrupt.c +0
-18break ;default : ret = -EINVAL ;@@ -2730,24 +2847,6 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } -static struct page *get_map_page (struct kvm *kvm,u64 uaddr) -{ -struct mm_struct *mm = kvm->mm; -struct page *page =NULL ; -int locked =1 ; - -if (mmget_not_zero (mm)) { -mmap_read_lock (mm); -get_user_pages_remote (mm, uaddr,1 ,FOLL_WRITE , - &page, &locked); -if (locked) -mmap_read_unlock (mm); -mmput (mm); - } - -return page; -} -static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int)arch/s390/kvm/kvm-s390.c +4
-0@@ -68,6 +68,8 @@const struct kvm_stats_desc kvm_vm_stats_desc[] = {KVM_GENERIC_VM_STATS (),STATS_DESC_COUNTER (VM , inject_io), +STATS_DESC_COUNTER (VM , io_390_adapter_map), +STATS_DESC_COUNTER (VM , io_390_adapter_unmap),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done),STATS_DESC_COUNTER (VM , inject_service_signal),@@ -2491,6 +2493,30 @@ static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,return r; } +static void kvm_s390_unmap_all_adapters_pv (struct kvm *kvm) +{
arch/s390/kvm/kvm-s390.c static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd, · line 2507
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+unsigned long flags; +struct s390_map_info *map, *tmp; +int i, idx; + +for (i =0 ; i <MAX_S390_IO_ADAPTERS ; i++) { +if (!kvm->arch.adapters[i]) +continue ; +list_for_each_entry_safe (map, tmp, + &kvm->arch.adapters[i]->maps, list) { +spin_lock_irqsave (&kvm->arch.adapters[i]->maps_lock, flags);
Same comment as kvm_s390_adapter_unmap, need to aquire this before the list_for_each_entry_safe call.
Actually this one will need to be a bit more creative since you need to drop the spinlock before each call to set_page_dirty_lock(), but you'll need to re-acquire it again each time you go back to the list. That makes list_for_each_entry_safe a bad choice.
Maybe using list_first_entry_or_null() each time you re-acquire the spinlock, until you get a null (meaning the list is empty)?
arch/s390/kvm/kvm-s390.c static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd, · line 2508
Quoted 1 line
arch/s390/kvm/kvm-s390.c +1
-0+list_del (&map->list);
You need to be decrementing nr_maps here before droppping the lock; unlike kvm_s390_destroy_adapters we are not freeing the structure and if we leave SE mode we could get more mappings later so the nr_maps value has to be kept up-to-date.
arch/s390/kvm/kvm-s390.c static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd, · line 2511
Quoted 3 lines
arch/s390/kvm/kvm-s390.c +3
-0+spin_unlock_irqrestore (&kvm->arch.adapters[i]->maps_lock, flags); + idx =srcu_read_lock (&kvm->srcu); +mark_page_dirty (kvm, map->addr >>PAGE_SHIFT );
Same comment as above, need to use the gaddr
Quoted 19 lines
arch/s390/kvm/kvm-s390.c +9
-0+set_page_dirty_lock (map->page); +srcu_read_unlock (&kvm->srcu, idx); +put_page (map->page); +kfree (map); + } + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {const bool need_lock = (cmd->cmd !=KVM_PV_ASYNC_CLEANUP_PERFORM );@@ -2507,6 +2533,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)if (kvm_s390_pv_is_protected (kvm))break ; +kvm_s390_unmap_all_adapters_pv (kvm);mmap_write_lock (kvm->mm);/* * Disable creation of new THPs. Existing THPs can stay, they
PATCH v3 2/3 KVM: s390: Enable adapter_indicators_set to use mapped pages
In reply to Douglas FreimuthThe S390 adapter_indicators_set function needs to be able to use mapped pages so that worked can be processed,on a fast path when interrupts are disabled. If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.
--- arch/s390/kvm/interrupt.c | 91 ++++++++++++++++++++++++++++----------- 1 file changed, 66 insertions(+), 25 deletions(-)
arch/s390/kvm/interrupt.c +66 -25
@@ -2847,41 +2847,82 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } +static struct s390_map_info *get_map_info (struct s390_io_adapter *adapter, +u64 addr) +{ +struct s390_map_info *map; + +if (!adapter) +return NULL ; + +list_for_each_entry (map, &adapter->maps, list) { +if (map->guest_addr == addr) +return map; + } +return NULL ; +} +static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {unsigned long bit;int summary_set, idx; -struct page *ind_page, *summary_page; +struct s390_map_info *ind_info, *summary_info;void *map; +struct page *ind_page, *summary_page; +unsigned long flags; - ind_page =get_map_page (kvm, adapter_int->ind_addr); -if (!ind_page) -return -1 ; - summary_page =get_map_page (kvm, adapter_int->summary_addr); -if (!summary_page) { -put_page (ind_page); -return -1 ; +spin_lock_irqsave (&adapter->maps_lock, flags); + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + ind_page =get_map_page (kvm, adapter_int->ind_addr); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page); +srcu_read_unlock (&kvm->srcu, idx); + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); + } +spin_lock_irqsave (&adapter->maps_lock, flags); + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + summary_page =get_map_page (kvm, adapter_int->summary_addr); +if (!summary_page && !ind_info) { +put_page (ind_page); +return -1 ; + } + idx =srcu_read_lock (&kvm->srcu); + map =page_address (summary_page); + bit =get_ind_bit (adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap); + summary_set =test_and_set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (summary_page); +srcu_read_unlock (&kvm->srcu, idx); + }else { + map =page_address (summary_info->page); + bit =get_ind_bit (summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set =test_and_set_bit (bit, map); +spin_unlock_irqrestore (&adapter->maps_lock, flags); } - idx =srcu_read_lock (&kvm->srcu); - map =page_address (ind_page); - bit =get_ind_bit (adapter_int->ind_addr, - adapter_int->ind_offset, adapter->swap); -set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (ind_page); - map =page_address (summary_page); - bit =get_ind_bit (adapter_int->summary_addr, - adapter_int->summary_offset, adapter->swap); - summary_set =test_and_set_bit (bit, map); -mark_page_dirty (kvm, adapter_int->summary_gaddr >>PAGE_SHIFT ); -set_page_dirty_lock (summary_page); -srcu_read_unlock (&kvm->srcu, idx); - -put_page (ind_page); -put_page (summary_page); +if (!ind_info) +put_page (ind_page); +if (!summary_info) +put_page (summary_page);return summary_set ?0 :1 ; }
Signature · 1 line
2.52.0
Re: [PATCH v3 2/3] KVM: s390: Enable adapter_indicators_set to use mapped pages
Quoted 73 lines · Douglas Freimuth
The S390
adapter_indicators_setfunction needs to be able to use mapped pages so that worked can be processed,on a fast path when interrupts are disabled. If adapter indicator pages are not mapped then local mapping is done on a slow path as it is prior to this patch. For example, Secure Execution environments will take the local mapping path as it does prior to this patch.Signed-off-by Douglas Freimuth <[email protected]>--- arch/s390/kvm/interrupt.c | 91 ++++++++++++++++++++++++++++----------- 1 file changed, 66 insertions(+), 25 deletions(-)arch/s390/kvm/interrupt.c +37
-8@@ -2847,41 +2847,82 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap)return swap ? (bit ^ (BITS_PER_LONG -1 )) : bit; } +static struct s390_map_info *get_map_info (struct s390_io_adapter *adapter, +u64 addr) +{ +struct s390_map_info *map; + +if (!adapter) +return NULL ; + +list_for_each_entry (map, &adapter->maps, list) { +if (map->guest_addr == addr) +return map; + } +return NULL ; +} +static int adapter_indicators_set (struct kvm *kvm,struct s390_io_adapter *adapter,struct kvm_s390_adapter_int *adapter_int) {unsigned long bit;int summary_set, idx; -struct page *ind_page, *summary_page; +struct s390_map_info *ind_info, *summary_info;void *map; +struct page *ind_page, *summary_page; +unsigned long flags; - ind_page =get_map_page (kvm, adapter_int->ind_addr); -if (!ind_page) -return -1 ; - summary_page =get_map_page (kvm, adapter_int->summary_addr); -if (!summary_page) { -put_page (ind_page); -return -1 ; +spin_lock_irqsave (&adapter->maps_lock, flags); + ind_info =get_map_info (adapter, adapter_int->ind_addr); +if (!ind_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + ind_page =get_map_page (kvm, adapter_int->ind_addr); +if (!ind_page) +return -1 ; + idx =srcu_read_lock (&kvm->srcu); + map =page_address (ind_page); + bit =get_ind_bit (adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); +set_bit (bit, map); +mark_page_dirty (kvm, adapter_int->ind_gaddr >>PAGE_SHIFT ); +set_page_dirty_lock (ind_page); +srcu_read_unlock (&kvm->srcu, idx); + }else { + map =page_address (ind_info->page); + bit =get_ind_bit (ind_info->addr, adapter_int->ind_offset, adapter->swap); +set_bit (bit, map);
arch/s390/kvm/interrupt.c static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · line 2904
Quoted 10 lines
arch/s390/kvm/interrupt.c +10
-0+spin_unlock_irqrestore (&adapter->maps_lock, flags); + } +spin_lock_irqsave (&adapter->maps_lock, flags); + summary_info =get_map_info (adapter, adapter_int->summary_addr); +if (!summary_info) { +spin_unlock_irqrestore (&adapter->maps_lock, flags); + summary_page =get_map_page (kvm, adapter_int->summary_addr); +if (!summary_page && !ind_info) { +put_page (ind_page); +return -1 ;
1) Sashiko mentions that this now allows for a path where we already set the indicator bits above but bail out early because we couldn't find the summary page. The old code validated that it could find the indicator page AND the summary page before setting any bits.
I think re-implementing to achieve that model is _preferable_ but I think we are also OK with the current approach; in this unlikely event the indicator bits are set but we fail to set the summary bit, then I can envision at least 2 scenarios...
1a) an already-running interrupt handler in the guest happens to see the indicator bits on because the summary bit was already set and proceeds to handle it and clear the indicator bits. I suppose we might get an over-indication of those bits later if the host attempts a re-delivery.
1b) Same as above but the summary bit was off (or the interrupt handler never was given initiative to run in the first place) so the indicator bits are not noticed and nothing is handled. But if re-delivery is attempted then this code would re-indicate the same bits which were already on -- this would not prevent an attempt at indicating the summary bit again. Assuming that succeeds, it will result in an adapter interrupt delivered.
If we continually fail to map the summary page I guess it could be an awkward dance of indicating the bits but never being able to set the summary bit. Realistically, if we are in this situation (cannot map the summary page that was previously valid) it seems to me we've got a bigger issue. Would a WARN_ON_ONCE make sense?
2) If you keep this approach then there is another issue here that is definitely a valid concern -- if summary_page is NULL but info_info is non-NULL, you continue on and use a summary_page of 0 below which is wrong - I think you wanted to return -1 (but without a put_page) in this case e.g.:
if (!summary_page) {
if (!ind_page)
put_page(ind_page);
return -1;
}PATCH v3 3/3 KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas FreimuthS390 needs a fast path for irq injection, and along those lines we introduce kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.
The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the adapter_indicators page needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizes GFP_KERNEL_ACCOUNT but we had to implement the fast path with GFP_ATOMIC allocation. Each of these enhancements were required to prevent blocking on the fast inject path.
Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including io_390_inatomic, io_flic_inject_airq, io_set_adapter_int and io_390_inatomic_adapter_masked.
--- arch/s390/include/asm/kvm_host.h | 6 +- arch/s390/kvm/interrupt.c | 160 +++++++++++++++++++++++++++---- arch/s390/kvm/kvm-s390.c | 24 ++++- arch/s390/kvm/kvm-s390.h | 3 +- 4 files changed, 169 insertions(+), 24 deletions(-)
arch/s390/include/asm/kvm_host.h +5 -1
@@ -359,7 +359,7 @@ struct kvm_s390_float_interrupt {struct kvm_s390_mchk_info mchk;struct kvm_s390_ext_info srv_signal;int last_sleep_cpu; -struct mutex ais_lock; +spinlock_t ais_lock;u8 simm;u8 nimm; };
@@ -450,6 +450,10 @@ struct kvm_vm_stat {u64 inject_io;u64 io_390_adapter_map;u64 io_390_adapter_unmap; +u64 io_390_inatomic; +u64 io_flic_inject_airq; +u64 io_set_adapter_int; +u64 io_390_inatomic_adapter_masked;u64 inject_float_mchk;u64 inject_pfault_done;u64 inject_service_signal;
arch/s390/kvm/interrupt.c +143 -17
275 lines not on this page · raw message
arch/s390/kvm/kvm-s390.c +19 -5
@@ -70,6 +70,10 @@ const struct kvm_stats_desc kvm_vm_stats_desc[] = {STATS_DESC_COUNTER (VM , inject_io),STATS_DESC_COUNTER (VM , io_390_adapter_map),STATS_DESC_COUNTER (VM , io_390_adapter_unmap), +STATS_DESC_COUNTER (VM , io_390_inatomic), +STATS_DESC_COUNTER (VM , io_flic_inject_airq), +STATS_DESC_COUNTER (VM , io_set_adapter_int), +STATS_DESC_COUNTER (VM , io_390_inatomic_adapter_masked),STATS_DESC_COUNTER (VM , inject_float_mchk),STATS_DESC_COUNTER (VM , inject_pfault_done), STATS_DESC_COUNTER(VM, inject_service_signal),
@@ -2869,6 +2873,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)void __user *argp = (void __user *)arg;struct kvm_device_attr attr;int r; +struct kvm_s390_interrupt_info *inti;switch (ioctl) {case KVM_S390_INTERRUPT : {
@@ -2877,7 +2882,10 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) r = -EFAULT ;if (copy_from_user (&s390int, argp,sizeof (s390int)))break ; - r =kvm_s390_inject_vm (kvm, &s390int); + inti =kzalloc_obj (*inti,GFP_KERNEL_ACCOUNT ); +if (!inti) +return -ENOMEM ; + r =kvm_s390_inject_vm (kvm, &s390int, inti);break ; }case KVM_CREATE_IRQCHIP : {
@@ -3275,7 +3283,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)mutex_unlock (&kvm->lock); } -mutex_init (&kvm->arch.float_int.ais_lock); +spin_lock_init (&kvm->arch.float_int.ais_lock);spin_lock_init (&kvm->arch.float_int.lock);for (i =0 ; i <FIRQ_LIST_COUNT ; i++)INIT_LIST_HEAD (&kvm->arch.float_int.lists[i]);
@@ -4396,11 +4404,16 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_cloreturn 1 ; } -static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token) +static int __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, +unsigned long token) {struct kvm_s390_interrupt inti;struct kvm_s390_irq irq; +struct kvm_s390_interrupt_info *inti_mem; + + inti_mem =kzalloc_obj (*inti_mem,GFP_KERNEL_ACCOUNT ); +if (!inti_mem) +return -ENOMEM ;if (start_token) { irq.u.ext.ext_params2 = token;
@@ -4409,8 +4422,9 @@ static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, }else { inti.type =KVM_S390_INT_PFAULT_DONE ; inti.parm64 = token; -WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti)); +WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti, inti_mem)); } +return true; } bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
arch/s390/kvm/kvm-s390.h +2 -1
@@ -373,7 +373,8 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu);void kvm_s390_clear_local_irqs (struct kvm_vcpu *vcpu);void kvm_s390_clear_float_irqs (struct kvm *kvm);int __must_check kvm_s390_inject_vm (struct kvm *kvm, -struct kvm_s390_interrupt *s390int); +struct kvm_s390_interrupt *s390int, +struct kvm_s390_interrupt_info *inti);int __must_check kvm_s390_inject_vcpu (struct kvm_vcpu *vcpu,struct kvm_s390_irq *irq); static inline int kvm_s390_inject_prog_irq(struct kvm_vcpu *vcpu,
Signature · 1 line
2.52.0
Re: [PATCH v3 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
Quoted 20 lines · Douglas Freimuth
S390 needs a fast path for irq injection, and along those lines we introduce
kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the
adapter_indicatorspage needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizesGFP_KERNEL_ACCOUNTbut we had to implement the fast path withGFP_ATOMICallocation. Each of these enhancements were required to prevent blocking on the fast inject path.Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Quoted 5 lines
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including
io_390_inatomic,io_flic_inject_airq,io_set_adapter_intandio_390_inatomic_adapter_masked.Signed-off-by Douglas Freimuth <[email protected]>
Sashiko complains about PREEMPT_RT kernels and spinlocks being sleepable in this case which would break the whole point of kvm_arch_set_irq_inatomic.
I suspect actually the kvm_arch_set_irq_inatomic() call itself shouldn't be used in this case, or in other words it wouldn't be an issue with just this s390 implementation but rather all of arch implementations?
I did not try enabling it and running a test, but I did do some searching of the codebase and I can found at least 1 spinlock acquired somewhere along the inatomic path for the existing implementations...
longarch (pch_pic_set_irq)
arm64 (vgic_its_inject_cached_translation)
powerpc (icp_deliver_irq)
riscv (kvm_riscv_aia_aplic_inject)
For x86 I didn't find a spinlock -- maybe I didn't look hard enough! -- but I did find a path that uses RCU (kvm_irq_delivery_to_apic_fast) which AFAIU would also become preemptible under PREEMPT_RT.
So for this series it seems reasonable to me to proceed as-is, with an open question whether there should be a KVM-wide avoidance of kvm_arch_set_irq_inatomic() under PREEMPT_RT?
Sean Christopherson
reviewerRe: [PATCH v3 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
Quoted 25 lines · Matthew Rosato
Quoted 24 lines · Douglas Freimuth
S390 needs a fast path for irq injection, and along those lines we introduce
kvm_arch_set_irq_inatomic. Instead of placing all interrupts on the global work queue as it does today, this patch provides a fast path for irq injection.The inatomic fast path cannot lose control since it is running with interrupts disabled. This meant making the following changes that exist on the slow path today. First, the
adapter_indicatorspage needs to be mapped since it is accessed with interrupts disabled, so we added map/unmap functions. Second, access to shared resources between the fast and slow paths needed to be changed from mutex and semaphores to spin_lock's. Finally, the memory allocation on the slow path utilizesGFP_KERNEL_ACCOUNTbut we had to implement the fast path withGFP_ATOMICallocation. Each of these enhancements were required to prevent blocking on the fast inject path.Fencing of Fast Inject in Secure Execution environments is enabled in the patch series by not mapping adapter indicator pages. In Secure Execution environments the path of execution available before this patch is followed.
Statistical counters have been added to enable analysis of irq injection on the fast path and slow path including
io_390_inatomic,io_flic_inject_airq,io_set_adapter_intandio_390_inatomic_adapter_masked.
Quoted 5 lines
Quoted 1 line
Signed-off-by Douglas Freimuth <[email protected]>Sashiko complains about
PREEMPT_RTkernels and spinlocks being sleepable in this case which would break the whole point ofkvm_arch_set_irq_inatomic.
Just make it a raw spinlock so that it stays an actual spinlock.
Quoted 9 lines
I suspect actually the
kvm_arch_set_irq_inatomic()call itself shouldn't be used in this case, or in other words it wouldn't be an issue with just this s390 implementation but rather all of arch implementations?I did not try enabling it and running a test, but I did do some searching of the codebase and I can found at least 1 spinlock acquired somewhere along the inatomic path for the existing implementations...
longarch (
pch_pic_set_irq)
I doubt anyone runs PREEMPT_RT VMs on LoongArch at this point.
Quoted 1 line
arm64 (
vgic_its_inject_cached_translation)
Uses raw.
Quoted 1 line
powerpc (
icp_deliver_irq)
Presumably arch_spin_lock() is also a "raw" version? PPC KVM is barely maintained at this point, so I wouldn't worry much about it.
Quoted 1 line
riscv (
kvm_riscv_aia_aplic_inject)
Uses "raw".
Quoted 3 lines
For x86 I didn't find a spinlock -- maybe I didn't look hard enough! -- but I did find a path that uses RCU (
kvm_irq_delivery_to_apic_fast) which AFAIU would also become preemptible underPREEMPT_RT.
This isn't about becoming preemptible per se, it's about non-raw spinlocks becoming sleepable locks. RCU can be made preemptible, but rcu_read_lock() doesn't become sleepable.
Quoted 3 lines
So for this series it seems reasonable to me to proceed as-is, with an open question whether there should be a KVM-wide avoidance of
kvm_arch_set_irq_inatomic()underPREEMPT_RT?
s390 should use a raw spinlock, same as arm64 and RISC-V.
Re: [PATCH v3 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
Quoted 6 lines
Quoted 4 lines
Sashiko complains about
PREEMPT_RTkernels and spinlocks being sleepable in this case which would break the whole point ofkvm_arch_set_irq_inatomic.Just make it a raw spinlock so that it stays an actual spinlock.
[...]
Quoted 2 lines
s390 should use a raw spinlock, same as arm64 and RISC-V.
Ahh, I missed that subtlety.
Thanks for the explanation!
Re: [PATCH v3 3/3] KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
In reply to Douglas FreimuthQuoted 4 lines
arch/s390/kvm/kvm-s390.c +0
-2-static void __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, -unsigned long token)
arch/s390/kvm/kvm-s390.c int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo · line 4416
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +7
-0+static int __kvm_inject_pfault_token (struct kvm_vcpu *vcpu,bool start_token, +unsigned long token) {struct kvm_s390_interrupt inti;struct kvm_s390_irq irq; +struct kvm_s390_interrupt_info *inti_mem; + + inti_mem =kzalloc_obj (*inti_mem,GFP_KERNEL_ACCOUNT ); +if (!inti_mem) +return -ENOMEM ;
You change this function to possibly return this value but you do not change the callers of this routine to actually look at the new return value?
AFAICT there are 2 callers of this today in arch/s390/kvm/kvm-s390.c - I assume one or both need updating, otherwise why do we need this change?
arch/s390/kvm/kvm-s390.c static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, · line 4427
Quoted 11 lines
arch/s390/kvm/kvm-s390.c +2
-1if (start_token) { irq.u.ext.ext_params2 = token;@@ -4409,8 +4422,9 @@ static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, }else { inti.type =KVM_S390_INT_PFAULT_DONE ; inti.parm64 = token; -WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti)); +WARN_ON_ONCE (kvm_s390_inject_vm (vcpu->kvm, &inti, inti_mem)); } +return true;
Since return value is an integer, return 0?