KVM: Improve IRQ assignment for device passthrough
Subsystems (2)
cover KVM: Improve IRQ assignment for device passthrough
This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches.
Jan Kiszka (6):
KVM: Clear assigned guest IRQ on release
KVM: Switch assigned device IRQ forwarding to threaded handler
KVM: Refactor IRQ names of assigned devices
KVM: Save/restore state of assigned PCI device
KVM: Clean up kvm_vm_ioctl_assigned_device
KVM: Document device assigment API
Documentation/kvm/api.txt | 178 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 ++++++++++++-------------------- 3 files changed, 227 insertions(+), 89 deletions(-)
PATCH 1/6 KVM: Clear assigned guest IRQ on release
From: Jan Kiszka <[email protected]>
When we deassign a guest IRQ, clear the potentially asserted guest line. There might be no chance for the guest to do this, specifically if we switch from INTx to MSI mode.
--- virt/kvm/assigned-dev.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
virt/kvm/assigned-dev.c +3 -0
@@ -141,6 +141,9 @@ static void deassign_guest_irq(struct kvm *kvm,kvm_unregister_irq_ack_notifier (kvm, &assigned_dev->ack_notifier); assigned_dev->ack_notifier.gsi =-1 ; +kvm_set_irq (assigned_dev->kvm, assigned_dev->irq_source_id, + assigned_dev->guest_irq,0 ); +if (assigned_dev->irq_source_id !=-1 )kvm_free_irq_source_id (kvm, assigned_dev->irq_source_id); assigned_dev->irq_source_id =-1 ;
Signature · 1 line
1.7.1
PATCH 2/6 KVM: Switch assigned device IRQ forwarding to threaded handler
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
This improves the IRQ forwarding for assigned devices: By using the kernel's threaded IRQ scheme, we can get rid of the latency-prone work queue and simplify the code in the same run.
Moreover, we no longer have to hold assigned_dev_lock while raising the guest IRQ, which can be a lenghty operation as we may have to iterate over all VCPUs. The lock is now only used for synchronizing masking vs. unmasking of INTx-type IRQs, thus is renames to intx_lock.
--- include/linux/kvm_host.h | 12 +---- virt/kvm/assigned-dev.c | 107 +++++++++++++++------------------------------- 2 files changed, 36 insertions(+), 83 deletions(-)
include/linux/kvm_host.h +2 -10
@@ -470,16 +470,8 @@ struct kvm_irq_ack_notifier {void (*irq_acked)(struct kvm_irq_ack_notifier *kian); }; -#define KVM_ASSIGNED_MSIX_PENDING 0x1 -struct kvm_guest_msix_entry { -u32 vector; -u16 entry; -u16 flags; -}; -struct kvm_assigned_dev_kernel {struct kvm_irq_ack_notifier ack_notifier; -struct work_struct interrupt_work;struct list_head list;int assigned_dev_id;int host_segnr;
@@ -490,13 +482,13 @@ struct kvm_assigned_dev_kernel {bool host_irq_disabled;struct msix_entry *host_msix_entries;int guest_irq; -struct kvm_guest_msix_entry *guest_msix_entries; +struct msix_entry *guest_msix_entries;unsigned long irq_requested_type;int irq_source_id;int flags;struct pci_dev *dev;struct kvm *kvm; -spinlock_t assigned_dev_lock; +spinlock_t intx_lock; };struct kvm_irq_mask_notifier {
virt/kvm/assigned-dev.c +34 -73
194 lines not on this page · raw message
Signature · 1 line
1.7.1
PATCH 3/6 KVM: Refactor IRQ names of assigned devices
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
Cosmetic change, but it helps to correlate IRQs with PCI devices.
--- include/linux/kvm_host.h | 1 + virt/kvm/assigned-dev.c | 11 ++++++----- 2 files changed, 7 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +1 -0
@@ -489,6 +489,7 @@ struct kvm_assigned_dev_kernel {struct pci_dev *dev;struct kvm *kvm;spinlock_t intx_lock; +char irq_name[32 ]; };struct kvm_irq_mask_notifier {
virt/kvm/assigned-dev.c +6 -5
@@ -231,8 +231,7 @@ static int assigned_device_enable_host_intx(struct kvm *kvm,* are going to be long delays in accepting, acking, etc. */ if (request_threaded_irq (dev->host_irq,NULL , kvm_assigned_dev_thread, -IRQF_ONESHOT ,"kvm_assigned_intx_device" , - (void *)dev)) +IRQF_ONESHOT , dev->irq_name, (void *)dev))return -EIO ;return 0 ; }
@@ -251,7 +250,7 @@ static int assigned_device_enable_host_msi(struct kvm *kvm, dev->host_irq = dev->dev->irq;if (request_threaded_irq (dev->host_irq,NULL , kvm_assigned_dev_thread, -0 ,"kvm_assigned_msi_device" , (void *)dev)) { +0 , dev->irq_name, (void *)dev)) {pci_disable_msi (dev->dev);return -EIO ; }
@@ -278,8 +277,7 @@ static int assigned_device_enable_host_msix(struct kvm *kvm,for (i =0 ; i < dev->entries_nr; i++) { r =request_threaded_irq (dev->host_msix_entries[i].vector,NULL , kvm_assigned_dev_thread, -0 ,"kvm_assigned_msix_device" , - (void *)dev); +0 , dev->irq_name, (void *)dev);if (r) goto err; }
@@ -336,6 +334,9 @@ static int assign_host_irq(struct kvm *kvm,if (dev->irq_requested_type &KVM_DEV_IRQ_HOST_MASK )return r; +snprintf (dev->irq_name,sizeof (dev->irq_name),"kvm:%s" , +pci_name (dev->dev)); +switch (host_irq_type) {case KVM_DEV_IRQ_HOST_INTX : r =assigned_device_enable_host_intx (kvm, dev);
Signature · 1 line
1.7.1
PATCH 4/6 KVM: Save/restore state of assigned PCI device
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
The guest may change states that pci_reset_function does not touch. So we better save/restore the assigned device across guest usage.
--- virt/kvm/assigned-dev.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
virt/kvm/assigned-dev.c +4 -1
@@ -197,7 +197,8 @@ static void kvm_free_assigned_device(struct kvm *kvm, {kvm_free_assigned_irq (kvm, assigned_dev); -pci_reset_function (assigned_dev->dev); +__pci_reset_function (assigned_dev->dev); +pci_restore_state (assigned_dev->dev);pci_release_regions (assigned_dev->dev);pci_disable_device (assigned_dev->dev);
@@ -514,6 +515,7 @@ static int kvm_vm_ioctl_assign_device(struct kvm *kvm, }pci_reset_function (dev); +pci_save_state (dev); match->assigned_dev_id = assigned_dev->assigned_dev_id; match->host_segnr = assigned_dev->segnr;
@@ -544,6 +546,7 @@ out:mutex_unlock (&kvm->lock);return r; out_list_del: +pci_restore_state (dev);list_del (&match->list);pci_release_regions (dev); out_disable:
Signature · 1 line
1.7.1
PATCH 5/6 KVM: Clean up kvm_vm_ioctl_assigned_device
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
Any arch not supporting device assigment will also not build assigned-dev.c. So testing for KVM_CAP_DEVICE_DEASSIGNMENT is pointless. KVM_CAP_ASSIGN_DEV_IRQ is unconditinally set. Moreover, add a default case for dispatching the ioctl.
--- virt/kvm/assigned-dev.c | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-)
virt/kvm/assigned-dev.c +4 -5
@@ -674,7 +674,7 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl, unsigned long arg) {void __user *argp = (void __user *)arg; -int r = -ENOTTY ; +int r;switch (ioctl) {case KVM_ASSIGN_PCI_DEVICE : {
@@ -692,7 +692,6 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl, r = -EOPNOTSUPP ;break ; } -#ifdef KVM_CAP_ASSIGN_DEV_IRQ case KVM_ASSIGN_DEV_IRQ : {struct kvm_assigned_irq assigned_irq;
@@ -715,8 +714,6 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl, goto out;break ; } -#endif -#ifdef KVM_CAP_DEVICE_DEASSIGNMENT case KVM_DEASSIGN_PCI_DEVICE : {struct kvm_assigned_pci_dev assigned_dev;
@@ -728,7 +725,6 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl, goto out;break ; } -#endif #ifdef KVM_CAP_IRQ_ROUTING case KVM_SET_GSI_ROUTING : {struct kvm_irq_routing routing;
@@ -781,6 +777,9 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl, break; }#endif +default : + r = -ENOTTY ; +break ; } out:return r;
Signature · 1 line
1.7.1
PATCH 6/6 KVM: Document device assigment API
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
Adds API documentation for KVM_[DE]ASSIGN_PCI_DEVICE, KVM_[DE]ASSIGN_DEV_IRQ, KVM_SET_GSI_ROUTING, KVM_ASSIGN_SET_MSIX_NR, and KVM_ASSIGN_SET_MSIX_ENTRY.
--- Documentation/kvm/api.txt | 178 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 178 insertions(+), 0 deletions(-)
Documentation/kvm/api.txt +178 -0
@@ -1085,6 +1085,184 @@ of 4 instructions that make up a hypercall. If any additional field gets added to this structure later on, a bit for that additional piece of information will be set in the flags bitmap. +4.47 KVM_ASSIGN_PCI_DEVICE + +Capability: KVM_CAP_DEVICE_ASSIGNMENT +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_assigned_pci_dev (in) +Returns: 0 on success, -1 on error + +Assigns a host PCI device to the VM. + +struct kvm_assigned_pci_dev { + __u32 assigned_dev_id; + __u32 busnr; + __u32 devfn; + __u32 flags; + __u32 segnr; + union { + __u32 reserved[11]; + }; +}; + +The PCI device is specified by the triple segnr, busnr, and devfn. +Identification in succeeding service requests is done via assigned_dev_id. The +following flags are specified: + +/* Depends on KVM_CAP_IOMMU */ +#define KVM_DEV_ASSIGN_ENABLE_IOMMU (1 << 0) + +4.48 KVM_DEASSIGN_PCI_DEVICE + +Capability: KVM_CAP_DEVICE_DEASSIGNMENT +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_assigned_pci_dev (in) +Returns: 0 on success, -1 on error + +Ends PCI device assignment, releasing all associated resources. + +See KVM_CAP_DEVICE_ASSIGNMENT for the data structure. Only assigned_dev_id is +used in kvm_assigned_pci_dev to identify the device. + +4.49 KVM_ASSIGN_DEV_IRQ + +Capability: KVM_CAP_ASSIGN_DEV_IRQ +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_assigned_irq (in) +Returns: 0 on success, -1 on error + +Assigns an IRQ to a passed-through device. + +struct kvm_assigned_irq { + __u32 assigned_dev_id; + __u32 host_irq; + __u32 guest_irq; + __u32 flags; + union { + struct { + __u32 addr_lo; + __u32 addr_hi; + __u32 data; + } guest_msi; + __u32 reserved[12]; + }; +}; + +The following flags are defined: + +#define KVM_DEV_IRQ_HOST_INTX (1 << 0) +#define KVM_DEV_IRQ_HOST_MSI (1 << 1) +#define KVM_DEV_IRQ_HOST_MSIX (1 << 2) + +#define KVM_DEV_IRQ_GUEST_INTX (1 << 8) +#define KVM_DEV_IRQ_GUEST_MSI (1 << 9) +#define KVM_DEV_IRQ_GUEST_MSIX (1 << 10) + +It is not valid to specify multiple types per host or guest IRQ. However, the +IRQ type of host and guest can differ or can even be null. + +4.50 KVM_DEASSIGN_DEV_IRQ + +Capability: KVM_CAP_ASSIGN_DEV_IRQ +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_assigned_irq (in) +Returns: 0 on success, -1 on error + +Ends an IRQ assignment to a passed-through device. + +See KVM_ASSIGN_DEV_IRQ for the data structure. The target device is specified +by assigned_dev_id, flags must correspond to the IRQ type specified on +KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed. + +4.51 KVM_SET_GSI_ROUTING + +Capability: KVM_CAP_IRQ_ROUTING +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_irq_routing (in) +Returns: 0 on success, -1 on error + +Sets the GSI routing table entries, overwriting any previously set entries. + +struct kvm_irq_routing { + __u32 nr; + __u32 flags; + struct kvm_irq_routing_entry entries[0]; +}; + +No flags are specified so far, the corresponding field must be set to zero. + +struct kvm_irq_routing_entry { + __u32 gsi; + __u32 type; + __u32 flags; + __u32 pad; + union { + struct kvm_irq_routing_irqchip irqchip; + struct kvm_irq_routing_msi msi; + __u32 pad[8]; + } u; +}; + +/* gsi routing entry types */ +#define KVM_IRQ_ROUTING_IRQCHIP 1 +#define KVM_IRQ_ROUTING_MSI 2 + +No flags are specified so far, the corresponding field must be set to zero. + +struct kvm_irq_routing_irqchip { + __u32 irqchip; + __u32 pin; +}; + +struct kvm_irq_routing_msi { + __u32 address_lo; + __u32 address_hi; + __u32 data; + __u32 pad; +}; + +4.52 KVM_ASSIGN_SET_MSIX_NR + +Capability: KVM_CAP_DEVICE_MSIX +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_assigned_msix_nr (in) +Returns: 0 on success, -1 on error + +Set the number of MSI-X interrupts for an assigned device. This service can +only be called once in the lifetime of an assigned device. + +struct kvm_assigned_msix_nr { + __u32 assigned_dev_id; + __u16 entry_nr; + __u16 padding; +}; + +#define KVM_MAX_MSIX_PER_DEV 256 + +4.53 KVM_ASSIGN_SET_MSIX_ENTRY + +Capability: KVM_CAP_DEVICE_MSIX +Architectures: x86 ia64 +Type: vm ioctl +Parameters: struct kvm_assigned_msix_entry (in) +Returns: 0 on success, -1 on error + +Specifies the routing of an MSI-X assigned device interrupt to a GSI. Setting +the GSI vector to zero means disabling the interrupt. + +struct kvm_assigned_msix_entry { + __u32 assigned_dev_id; + __u32 gsi; + __u16 entry; /* The index of entry in the MSI-X table */ + __u16 padding[3]; +}; + 5. The kvm_run structure Application code obtains a pointer to the kvm_run structure by
Signature · 1 line
1.7.1
Quoted 12 lines · Jan Kiszka
This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches.
Jan Kiszka (6):
KVM: Clear assigned guest IRQ on release
KVM: Switch assigned device IRQ forwarding to threaded handler
KVM: Refactor IRQ names of assigned devices
KVM: Save/restore state of assigned PCI device
KVM: Clean upkvm_vm_ioctl_assigned_device
KVM: Document device assigment API
Quoted 5 lines
Documentation/kvm/api.txt | 178 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 ++++++++++++-------------------- 3 files changed, 227 insertions(+), 89 deletions(-)
Series looks good to me.
Quoted 4 lines · Jan Kiszka
This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches.
Looks good, especially the threaded irq - a real winner.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Jan Kiszka
This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches.
Jan Kiszka (6):
KVM: Clear assigned guest IRQ on release
KVM: Switch assigned device IRQ forwarding to threaded handler
KVM: Refactor IRQ names of assigned devices
KVM: Save/restore state of assigned PCI device
KVM: Clean upkvm_vm_ioctl_assigned_device
KVM: Document device assigment API
Quoted 4 lines
Documentation/kvm/api.txt | 178 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 ++++++++++++-------------------- 3 files changed, 227 insertions(+), 89 deletions(-)
Marcelo Tosatti
AppliedQuoted 12 lines · Jan Kiszka
This is the rebased light version of the previous series, i.e. without PCI-2.3-based IRQ masking or any SRCU conversion. PCI-2.3 support is under rework to explore options for automatic mode switches.
Jan Kiszka (6):
KVM: Clear assigned guest IRQ on release
KVM: Switch assigned device IRQ forwarding to threaded handler
KVM: Refactor IRQ names of assigned devices
KVM: Save/restore state of assigned PCI device
KVM: Clean upkvm_vm_ioctl_assigned_device
KVM: Document device assigment API
Quoted 4 lines
Documentation/kvm/api.txt | 178 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/kvm_host.h | 13 +--- virt/kvm/assigned-dev.c | 125 ++++++++++++-------------------- 3 files changed, 227 insertions(+), 89 deletions(-)
Applied, thanks.