Skip to contentSkip to conversation

KVM: Improve IRQ assignment for device passthrough

PATCH In review Jan Kiszka 6 participants 35 messages Nov 8, 2010 – Nov 26, 2010

Subsystems (4)
lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

v4 cover KVM: Improve IRQ assignment for device passthrough

Nine patches (yeah, it's getting more and more) to improve "classic" device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers.

As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid:

disable_irq_nosync:           ~600 cycles
pci_2_3_irq_check_and_mask:  ~6000 cycles (EHCI)
                            ~22000 cycles (AR9287, with peaks >100000)

Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well.

Changes in v4:
- Allow user space to push its INTx mask state to the kernel and respect this on IRQ masking and delivery
- Specify IRQF_ONESHOT for threaded IRQ of assigned device
- Switch IRQ subsystem locking from RCU to SRCU
- Fix for SRCU struct leakage
- Small cleanup for kvm_vm_ioctl_assigned_device
- Documentation for device assignment API (please check carefully if I got it right)

Changes in v3:
- Save/restore PCI device state across guest usage
- Make PCI-2.3-based IRQ sharing configurable by user space
- Fix potential nesting of pci_block_user_cfg_access
- Do not track PCI-level IRQ masking in software, user space may change it concurrently
- Optimize kvm_assigned_dev_ack_irq for the case another IRQ is already pending
- Cleanups according to review comments

Changes in v2:
- Reworked IRQ forwarding path to use threaded IRQs (direct signalling from IRQ context does not work out of the box and may be too lengthy)
- Refactored host IRQ naming of assigned devices (cosmetic change)
- Avoid unmask on ack when the next IRQ is pending, rather reassert the guest line (PCI-2.3 patch)
- Refactored PCI-2.3 patch (but still no control knob for shared mode - is that a must?)

Jan Kiszka (9):
KVM: Fix srcu struct leakage
KVM: Switch IRQ subsystem to SRCU
KVM: Clear assigned guest IRQ on release
KVM: Switch assigned device IRQ forwarding to threaded handler
KVM: Refactor IRQ names of assigned devices
KVM: Save/restore state of assigned PCI device
KVM: Clean up kvm_vm_ioctl_assigned_device
KVM: Document device assigment API
KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

 Documentation/kvm/api.txt |  203 ++++++++++++++++++++++++
 arch/x86/kvm/x86.c        |    1 +
 include/linux/kvm.h       |    6 +
 include/linux/kvm_host.h  |   16 +--
 virt/kvm/assigned-dev.c   |  383 ++++++++++++++++++++++++++++++++++----------
 virt/kvm/irq_comm.c       |   31 ++--
 virt/kvm/kvm_main.c       |   20 ++-
 7 files changed, 541 insertions(+), 119 deletions(-)
author
#1 lore raw

PATCH v4 1/9 KVM: Fix srcu struct leakage

Clean up the srcu struct on vm destruction and refactor its release on early errors.

---
 virt/kvm/kvm_main.c |   15 +++++++--------
 1 files changed, 7 insertions(+), 8 deletions(-)
virt/kvm/kvm_main.c +7 -8
index 4111a4b..6cfcde7 100644
@@ -401,23 +401,19 @@ static struct kvm *kvm_create_vm(void)
 	r = -ENOMEM;
 	kvm->memslots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 	if (!kvm->memslots)
-		goto out_err;
+		goto out_err_nosrcu;
 	if (init_srcu_struct(&kvm->srcu))
-		goto out_err;
+		goto out_err_nosrcu;
 	for (i = 0; i < KVM_NR_BUSES; i++) {
 		kvm->buses[i] = kzalloc(sizeof(struct kvm_io_bus),
 					GFP_KERNEL);
-		if (!kvm->buses[i]) {
-			cleanup_srcu_struct(&kvm->srcu);
+		if (!kvm->buses[i])
 			goto out_err;
-		}
 	}
 
 	r = kvm_init_mmu_notifier(kvm);
-	if (r) {
-		cleanup_srcu_struct(&kvm->srcu);
+	if (r)
 		goto out_err;
-	}
 
 	kvm->mm = current->mm;
 	atomic_inc(&kvm->mm->mm_count);
@@ -435,6 +431,8 @@ out:
 	return kvm;
 
 out_err:
+	cleanup_srcu_struct(&kvm->srcu);
+out_err_nosrcu:
 	hardware_disable_all();
 out_err_nodisable:
 	for (i = 0; i < KVM_NR_BUSES; i++)
@@ -513,6 +511,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 #else
 	kvm_arch_flush_shadow(kvm);
 #endif
+	cleanup_srcu_struct(&kvm->srcu);
 	kvm_arch_destroy_vm(kvm);
 	hardware_disable_all();
 	mmdrop(mm);
Signature · 1 line
1.7.1
Acked-by on v4 1/9
#2 lore raw

Re: [PATCH v4 1/9] KVM: Fix srcu struct leakage

Quoted 4 lines · Jan Kiszka

Clean up the srcu struct on vm destruction and refactor its release on early errors.

Yay, I suspected something's wrong with error handling in srcu.

This one actually has nothing to do with assignment, and it looks like it's needed for -stable and 2.6.37. Avi/Marcelo?

Quoted 55 lines
---
 virt/kvm/kvm_main.c |   15 +++++++--------
 1 files changed, 7 insertions(+), 8 deletions(-)
virt/kvm/kvm_main.c +7 -8
index 4111a4b..6cfcde7 100644
@@ -401,23 +401,19 @@ static struct kvm *kvm_create_vm(void)
 	r = -ENOMEM;
 	kvm->memslots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 	if (!kvm->memslots)
-		goto out_err;
+		goto out_err_nosrcu;
 	if (init_srcu_struct(&kvm->srcu))
-		goto out_err;
+		goto out_err_nosrcu;
 	for (i = 0; i < KVM_NR_BUSES; i++) {
 		kvm->buses[i] = kzalloc(sizeof(struct kvm_io_bus),
 					GFP_KERNEL);
-		if (!kvm->buses[i]) {
-			cleanup_srcu_struct(&kvm->srcu);
+		if (!kvm->buses[i])
 			goto out_err;
-		}
 	}
 
 	r = kvm_init_mmu_notifier(kvm);
-	if (r) {
-		cleanup_srcu_struct(&kvm->srcu);
+	if (r)
 		goto out_err;
-	}
 
 	kvm->mm = current->mm;
 	atomic_inc(&kvm->mm->mm_count);
@@ -435,6 +431,8 @@ out:
 	return kvm;
 
 out_err:
+	cleanup_srcu_struct(&kvm->srcu);
+out_err_nosrcu:
 	hardware_disable_all();
 out_err_nodisable:
 	for (i = 0; i < KVM_NR_BUSES; i++)
@@ -513,6 +511,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 #else
 	kvm_arch_flush_shadow(kvm);
 #endif
+	cleanup_srcu_struct(&kvm->srcu);
 	kvm_arch_destroy_vm(kvm);
 	hardware_disable_all();
 	mmdrop(mm);
Signature · 1 line
1.7.1
author
#3 lore raw

Re: [PATCH v4 1/9] KVM: Fix srcu struct leakage

Quoted 11 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka

Clean up the srcu struct on vm destruction and refactor its release on early errors.

Yay, I suspected something's wrong with error handling in srcu.

This one actually has nothing to do with assignment,

Yes, it is just mechanically needed (and I found it while hacking in more SRCU).

Jan

Quoted 58 lines

and it looks like it's needed for -stable and 2.6.37. Avi/Marcelo?

Quoted 55 lines
---
 virt/kvm/kvm_main.c |   15 +++++++--------
 1 files changed, 7 insertions(+), 8 deletions(-)
virt/kvm/kvm_main.c +7 -8
index 4111a4b..6cfcde7 100644
@@ -401,23 +401,19 @@ static struct kvm *kvm_create_vm(void)
 	r = -ENOMEM;
 	kvm->memslots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 	if (!kvm->memslots)
-		goto out_err;
+		goto out_err_nosrcu;
 	if (init_srcu_struct(&kvm->srcu))
-		goto out_err;
+		goto out_err_nosrcu;
 	for (i = 0; i < KVM_NR_BUSES; i++) {
 		kvm->buses[i] = kzalloc(sizeof(struct kvm_io_bus),
 					GFP_KERNEL);
-		if (!kvm->buses[i]) {
-			cleanup_srcu_struct(&kvm->srcu);
+		if (!kvm->buses[i])
 			goto out_err;
-		}
 	}
 
 	r = kvm_init_mmu_notifier(kvm);
-	if (r) {
-		cleanup_srcu_struct(&kvm->srcu);
+	if (r)
 		goto out_err;
-	}
 
 	kvm->mm = current->mm;
 	atomic_inc(&kvm->mm->mm_count);
@@ -435,6 +431,8 @@ out:
 	return kvm;
 
 out_err:
+	cleanup_srcu_struct(&kvm->srcu);
+out_err_nosrcu:
 	hardware_disable_all();
 out_err_nodisable:
 	for (i = 0; i < KVM_NR_BUSES; i++)
@@ -513,6 +511,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 #else
 	kvm_arch_flush_shadow(kvm);
 #endif
+	cleanup_srcu_struct(&kvm->srcu);
 	kvm_arch_destroy_vm(kvm);
 	hardware_disable_all();
 	mmdrop(mm);
Signature · 1 line
1.7.1
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
author
#4 lore raw

PATCH v4 2/9 KVM: Switch IRQ subsystem to SRCU

In reply to Jan Kiszka

Ack and mask notifiers typically call back into kvm_set_irq, thus may iterate over all VCPUs of a VM. Better keep this path preemptible to prevent that user-space can massivle influence scheduling latencies. Use sleepable RCU for the protection of irq_routing and the notfier lists.

---
 include/linux/kvm_host.h |    1 +
 virt/kvm/irq_comm.c      |   31 +++++++++++++++++--------------
 virt/kvm/kvm_main.c      |    5 +++++
 3 files changed, 23 insertions(+), 14 deletions(-)
include/linux/kvm_host.h +1 -0
index bcf71c7..83bf8e1 100644
@@ -238,6 +238,7 @@ struct kvm {
 #endif
 
 	struct mutex irq_lock;
+	struct srcu_struct irq_srcu;
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
 	struct kvm_irq_routing_table __rcu *irq_routing;
 	struct hlist_head mask_notifier_list;
virt/kvm/irq_comm.c +17 -14
index 8edca91..2331587 100644
@@ -150,6 +150,7 @@ int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level)
 	int ret = -1, i = 0;
 	struct kvm_irq_routing_table *irq_rt;
 	struct hlist_node *n;
+	int idx;
 
 	trace_kvm_set_irq(irq, level, irq_source_id);
 
@@ -157,12 +158,12 @@ int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level)
 	 * IOAPIC.  So set the bit in both. The guest will ignore
 	 * writes to the unused one.
 	 */
-	rcu_read_lock();
-	irq_rt = rcu_dereference(kvm->irq_routing);
+	idx = srcu_read_lock(&kvm->irq_srcu);
+	irq_rt = srcu_dereference(kvm->irq_routing, &kvm->irq_srcu);
 	if (irq < irq_rt->nr_rt_entries)
 		hlist_for_each_entry(e, n, &irq_rt->map[irq], link)
 			irq_set[i++] = *e;
-	rcu_read_unlock();
+	srcu_read_unlock(&kvm->irq_srcu, idx);
 
 	while(i--) {
 		int r;
@@ -180,18 +181,19 @@ void kvm_notify_acked_irq(struct kvm *kvm, unsigned irqchip, unsigned pin)
 {
 	struct kvm_irq_ack_notifier *kian;
 	struct hlist_node *n;
-	int gsi;
+	int gsi, idx;
 
 	trace_kvm_ack_irq(irqchip, pin);
 
-	rcu_read_lock();
-	gsi = rcu_dereference(kvm->irq_routing)->chip[irqchip][pin];
+	idx = srcu_read_lock(&kvm->irq_srcu);
+	gsi = srcu_dereference(kvm->irq_routing,
+			       &kvm->irq_srcu)->chip[irqchip][pin];
 	if (gsi != -1)
 		hlist_for_each_entry_rcu(kian, n, &kvm->irq_ack_notifier_list,
 					 link)
 			if (kian->gsi == gsi)
 				kian->irq_acked(kian);
-	rcu_read_unlock();
+	srcu_read_unlock(&kvm->irq_srcu, idx);
 }
 
 void kvm_register_irq_ack_notifier(struct kvm *kvm,
@@ -208,7 +210,7 @@ void kvm_unregister_irq_ack_notifier(struct kvm *kvm,
 	mutex_lock(&kvm->irq_lock);
 	hlist_del_init_rcu(&kian->link);
 	mutex_unlock(&kvm->irq_lock);
-	synchronize_rcu();
+	synchronize_srcu(&kvm->irq_srcu);
 }
 
 int kvm_request_irq_source_id(struct kvm *kvm)
@@ -276,7 +278,7 @@ void kvm_unregister_irq_mask_notifier(struct kvm *kvm, int irq,
 	mutex_lock(&kvm->irq_lock);
 	hlist_del_rcu(&kimn->link);
 	mutex_unlock(&kvm->irq_lock);
-	synchronize_rcu();
+	synchronize_srcu(&kvm->irq_srcu);
 }
 
 void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin,
@@ -284,15 +286,16 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin,
 {
 	struct kvm_irq_mask_notifier *kimn;
 	struct hlist_node *n;
-	int gsi;
+	int gsi, idx;
 
-	rcu_read_lock();
-	gsi = rcu_dereference(kvm->irq_routing)->chip[irqchip][pin];
+	idx = srcu_read_lock(&kvm->irq_srcu);
+	gsi = srcu_dereference(kvm->irq_routing,
+			       &kvm->irq_srcu)->chip[irqchip][pin];
 	if (gsi != -1)
 		hlist_for_each_entry_rcu(kimn, n, &kvm->mask_notifier_list, link)
 			if (kimn->irq == gsi)
 				kimn->func(kimn, mask);
-	rcu_read_unlock();
+	srcu_read_unlock(&kvm->irq_srcu, idx);
 }
 
 void kvm_free_irq_routing(struct kvm *kvm)
@@ -411,7 +414,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
 	old = kvm->irq_routing;
 	rcu_assign_pointer(kvm->irq_routing, new);
 	mutex_unlock(&kvm->irq_lock);
-	synchronize_rcu();
+	synchronize_srcu(&kvm->irq_srcu);
 
 	new = old;
 	r = 0;
virt/kvm/kvm_main.c +5 -0
index 6cfcde7..b0ac2c0 100644
@@ -404,6 +404,8 @@ static struct kvm *kvm_create_vm(void)
 		goto out_err_nosrcu;
 	if (init_srcu_struct(&kvm->srcu))
 		goto out_err_nosrcu;
+	if (init_srcu_struct(&kvm->irq_srcu))
+		goto out_err_noirqsrcu;
 	for (i = 0; i < KVM_NR_BUSES; i++) {
 		kvm->buses[i] = kzalloc(sizeof(struct kvm_io_bus),
 					GFP_KERNEL);
@@ -431,6 +433,8 @@ out:
 	return kvm;
 
 out_err:
+	cleanup_srcu_struct(&kvm->irq_srcu);
+out_err_noirqsrcu:
 	cleanup_srcu_struct(&kvm->srcu);
 out_err_nosrcu:
 	hardware_disable_all();
@@ -511,6 +515,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 #else
 	kvm_arch_flush_shadow(kvm);
 #endif
+	cleanup_srcu_struct(&kvm->irq_srcu);
 	cleanup_srcu_struct(&kvm->srcu);
 	kvm_arch_destroy_vm(kvm);
 	hardware_disable_all();
Signature · 1 line
1.7.1
#5 lore raw

Re: [PATCH v4 2/9] KVM: Switch IRQ subsystem to SRCU

Quoted 5 lines · Jan Kiszka

Ack and mask notifiers typically call back into kvm_set_irq, thus may iterate over all VCPUs of a VM. Better keep this path preemptible to prevent that user-space can massivle influence scheduling latencies. Use sleepable RCU for the protection of irq_routing and the notfier lists.

What about preemptible RCU, now in mainline?

/*
* Tree-preemptable RCU implementation for rcu_read_lock().
* Just increment ->rcu_read_lock_nesting, shared state will be updated
* if we block.
*/
void __rcu_read_lock(void)

{
    current->rcu_read_lock_nesting++;
    barrier();  /* needed if we ever invoke rcu_read_lock in rcutree.c */
}

EXPORT_SYMBOL_GPL(__rcu_read_lock);

(wow, quite clever in its simplicity)

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#6 lore raw

Re: [PATCH v4 2/9] KVM: Switch IRQ subsystem to SRCU

Quoted 8 lines · Avi Kivity
Quoted 5 lines · Jan Kiszka

Ack and mask notifiers typically call back into kvm_set_irq, thus may iterate over all VCPUs of a VM. Better keep this path preemptible to prevent that user-space can massivle influence scheduling latencies. Use sleepable RCU for the protection of irq_routing and the notfier lists.

What about preemptible RCU, now in mainline?

Granted, that would resolve the preemption issue but not allow to use sleepy locks inside this section (as done by the last patch in this series).

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
author
#7 lore raw

PATCH v4 3/9 KVM: Clear assigned guest IRQ on release

In reply to Jan Kiszka

When we deassign a guest IRQ, clear the potentially asserted guest line. There might be no chance for the guest to do this, specifically if we switch from INTx to MSI mode.

---
 virt/kvm/assigned-dev.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
virt/kvm/assigned-dev.c +3 -0
index 7c98928..ecc4419 100644
@@ -141,6 +141,9 @@ static void deassign_guest_irq(struct kvm *kvm,
 	kvm_unregister_irq_ack_notifier(kvm, &assigned_dev->ack_notifier);
 	assigned_dev->ack_notifier.gsi = -1;
 
+	kvm_set_irq(assigned_dev->kvm, assigned_dev->irq_source_id,
+		    assigned_dev->guest_irq, 0);
+
 	if (assigned_dev->irq_source_id != -1)
 		kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
 	assigned_dev->irq_source_id = -1;
Signature · 1 line
1.7.1
#8 lore raw

Re: [PATCH v4 3/9] KVM: Clear assigned guest IRQ on release

Quoted 14 lines · Jan Kiszka

When we deassign a guest IRQ, clear the potentially asserted guest line. There might be no chance for the guest to do this, specifically if we switch from INTx to MSI mode.

---
  virt/kvm/assigned-dev.c |    3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)
virt/kvm/assigned-dev.c +0 -0
index 7c98928..ecc4419 100644

virt/kvm/assigned-dev.c static void deassign_guest_irq(struct kvm *kvm, · line 149

Quoted 9 lines
virt/kvm/assigned-dev.c +3 -0
@@ -141,6 +141,9 @@ static void deassign_guest_irq(struct kvm *kvm,
  	kvm_unregister_irq_ack_notifier(kvm,&assigned_dev->ack_notifier);
  	assigned_dev->ack_notifier.gsi = -1;
 
+	kvm_set_irq(assigned_dev->kvm, assigned_dev->irq_source_id,
+		    assigned_dev->guest_irq, 0);
+
  	if (assigned_dev->irq_source_id != -1)
  		kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
  	assigned_dev->irq_source_id = -1;

I guess this can't hurt. Did you see it happen in practice?

Note: all this kvm_set_irq(..., [01]) is incorrect as it doesn't account for polarity. Currently the qemu-emulated chipset uses level high pci interrupts, but that's not a given from kvm's point of view.

I think vfio fixes this by only routing msi interrupts via the kernel, and routing level interrupts through userspace, which can adjust polarity. Alex/Michael?

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#9 lore raw

Re: [PATCH v4 3/9] KVM: Clear assigned guest IRQ on release

Quoted 18 lines · Avi Kivity
Quoted 17 lines · Jan Kiszka

When we deassign a guest IRQ, clear the potentially asserted guest line. There might be no chance for the guest to do this, specifically if we switch from INTx to MSI mode.

---
  virt/kvm/assigned-dev.c |    3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)
virt/kvm/assigned-dev.c +0 -0
index 7c98928..ecc4419 100644
@@ -141,6 +141,9 @@ static void deassign_guest_irq(struct kvm *kvm,
  	kvm_unregister_irq_ack_notifier(kvm,&assigned_dev->ack_notifier);
  	assigned_dev->ack_notifier.gsi = -1;
 
Quoted 8 lines

virt/kvm/assigned-dev.c static void deassign_guest_irq(struct kvm *kvm, · line 149

Quoted 6 lines
virt/kvm/assigned-dev.c +3 -0
+	kvm_set_irq(assigned_dev->kvm, assigned_dev->irq_source_id,
+		    assigned_dev->guest_irq, 0);
+
  	if (assigned_dev->irq_source_id != -1)
  		kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
  	assigned_dev->irq_source_id = -1;

I guess this can't hurt. Did you see it happen in practice?

Yes, with prefer_msi=off and an e1000e-driven NIC: The legacy IRQ stayed asserted when the guest actually enabled MSI.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#10 lore raw

Re: [PATCH v4 3/9] KVM: Clear assigned guest IRQ on release

In reply to Avi Kivity
Quoted 27 lines · Avi Kivity
Quoted 14 lines · Jan Kiszka

When we deassign a guest IRQ, clear the potentially asserted guest line. There might be no chance for the guest to do this, specifically if we switch from INTx to MSI mode.

---
  virt/kvm/assigned-dev.c |    3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)
virt/kvm/assigned-dev.c +0 -0
index 7c98928..ecc4419 100644

virt/kvm/assigned-dev.c static void deassign_guest_irq(struct kvm *kvm, · line 149

Quoted 9 lines
virt/kvm/assigned-dev.c +3 -0
@@ -141,6 +141,9 @@ static void deassign_guest_irq(struct kvm *kvm,
  	kvm_unregister_irq_ack_notifier(kvm,&assigned_dev->ack_notifier);
  	assigned_dev->ack_notifier.gsi = -1;
 
+	kvm_set_irq(assigned_dev->kvm, assigned_dev->irq_source_id,
+		    assigned_dev->guest_irq, 0);
+
  	if (assigned_dev->irq_source_id != -1)
  		kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
  	assigned_dev->irq_source_id = -1;

I guess this can't hurt. Did you see it happen in practice?

Quoted 7 lines

Note: all this kvm_set_irq(..., [01]) is incorrect as it doesn't account for polarity. Currently the qemu-emulated chipset uses level high pci interrupts, but that's not a given from kvm's point of view.

I think vfio fixes this by only routing msi interrupts via the kernel, and routing level interrupts through userspace, which can adjust polarity. Alex/Michael?

The latest patches will route legacy interrupts via irqfd if available too. We do have the issue that KVM pulses interrupts injected this way, but it seems to work nonetheless. I was thinking about proposing a level flag for set_irqfd to only set it, allowing the ack notifier to deassert it. Perhaps we also need a flag to toggle the polarity.

Alex

#11 lore raw

Re: [PATCH v4 3/9] KVM: Clear assigned guest IRQ on release

Quoted 5 lines · Alex Williamson
Quoted 5 lines

Note: all this kvm_set_irq(..., [01]) is incorrect as it doesn't account for polarity. Currently the qemu-emulated chipset uses level high pci interrupts, but that's not a given from kvm's point of view.

Quoted 7 lines
Quoted 3 lines

I think vfio fixes this by only routing msi interrupts via the kernel, and routing level interrupts through userspace, which can adjust polarity. Alex/Michael?

The latest patches will route legacy interrupts via irqfd if available too. We do have the issue that KVM pulses interrupts injected this way, but it seems to work nonetheless.

That doesn't sound too good.

Quoted 3 lines

I was thinking about proposing a
level flag for set_irqfd to only set it, allowing the ack notifier to deassert it. Perhaps we also need a flag to toggle the polarity.

Or maybe irqfd is not up to this task.

I'd rather stick with the ordinary ioctls for this until proven they're too slow. By proven, I mean a userspace irq forwarding implementation that doesn't suck like we it does now (can be just a prototype).

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#12 lore raw

Re: [PATCH v4 3/9] KVM: Clear assigned guest IRQ on release

Quoted 10 lines · Avi Kivity
Quoted 9 lines · Alex Williamson
Quoted 8 lines

Note: all this kvm_set_irq(..., [01]) is incorrect as it doesn't account for polarity. Currently the qemu-emulated chipset uses level high pci interrupts, but that's not a given from kvm's point of view.

I think vfio fixes this by only routing msi interrupts via the kernel, and routing level interrupts through userspace, which can adjust polarity. Alex/Michael?

Quoted 5 lines
Quoted 3 lines

The latest patches will route legacy interrupts via irqfd if available too. We do have the issue that KVM pulses interrupts injected this way, but it seems to work nonetheless.

That doesn't sound too good.

It's more than luck that it works: there is a backend device that keeps the interrupt asserted, if driver did not handle the interrupt but the OS did ack it, we will unmask the interrupt, get another one from the device
(or just notice status bit is set) and send another pulse.

Whether all this improves or hurts performance, or whether some guest might notice the difference, I don't know. So maybe pulsing the interrupt is actually a good thing? Anyone has any theories?

What I think is broken, is
1. interrupt sharing, as irqfds use the userspace id instead of a per-device id.
2. emulated devices as there's no one to reassert the interrupt.

Quoted 19 lines
Quoted 3 lines

I was thinking about proposing a
level flag for set_irqfd to only set it, allowing the ack notifier to deassert it. Perhaps we also need a flag to toggle the polarity.

Or maybe irqfd is not up to this task.

I'd rather stick with the ordinary ioctls for this until proven they're too slow. By proven, I mean a userspace irq forwarding implementation that doesn't suck like we it does now (can be just a prototype).

Signature · 7 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#13 lore raw

PATCH v4 4/9 KVM: Switch assigned device IRQ forwarding to threaded handler

In reply to Jan Kiszka

This improves the IRQ forwarding for assigned devices: By using the kernel's threaded IRQ scheme, we can get rid of the latency-prone work queue and simplify the code in the same run.

Moreover, we no longer have to hold assigned_dev_lock while raising the guest IRQ, which can be a lenghty operation as we may have to iterate over all VCPUs. The lock is now only used for synchronizing masking vs. unmasking of INTx-type IRQs, thus is renames to intx_lock.

---
 include/linux/kvm_host.h |   12 +----
 virt/kvm/assigned-dev.c  |  107 +++++++++++++++-------------------------------
 2 files changed, 36 insertions(+), 83 deletions(-)
include/linux/kvm_host.h +2 -10
index 83bf8e1..050674f 100644
@@ -458,16 +458,8 @@ struct kvm_irq_ack_notifier {
 	void (*irq_acked)(struct kvm_irq_ack_notifier *kian);
 };
 
-#define KVM_ASSIGNED_MSIX_PENDING		0x1
-struct kvm_guest_msix_entry {
-	u32 vector;
-	u16 entry;
-	u16 flags;
-};
-
 struct kvm_assigned_dev_kernel {
 	struct kvm_irq_ack_notifier ack_notifier;
-	struct work_struct interrupt_work;
 	struct list_head list;
 	int assigned_dev_id;
 	int host_segnr;
@@ -478,13 +470,13 @@ struct kvm_assigned_dev_kernel {
 	bool host_irq_disabled;
 	struct msix_entry *host_msix_entries;
 	int guest_irq;
-	struct kvm_guest_msix_entry *guest_msix_entries;
+	struct msix_entry *guest_msix_entries;
 	unsigned long irq_requested_type;
 	int irq_source_id;
 	int flags;
 	struct pci_dev *dev;
 	struct kvm *kvm;
-	spinlock_t assigned_dev_lock;
+	spinlock_t intx_lock;
 };
 
 struct kvm_irq_mask_notifier {
virt/kvm/assigned-dev.c +34 -73

194 lines not on this page · raw message

Signature · 1 line
1.7.1
#14 lore raw

Re: [PATCH v4 4/9] KVM: Switch assigned device IRQ forwarding to threaded handler

Quoted 8 lines · Jan Kiszka

This improves the IRQ forwarding for assigned devices: By using the kernel's threaded IRQ scheme, we can get rid of the latency-prone work queue and simplify the code in the same run.

Moreover, we no longer have to hold assigned_dev_lock while raising the guest IRQ, which can be a lenghty operation as we may have to iterate over all VCPUs. The lock is now only used for synchronizing masking vs. unmasking of INTx-type IRQs, thus is renames to intx_lock.

Nice stuff.

include/linux/kvm_host.h struct kvm_irq_ack_notifier { · line 467

Quoted 8 lines
include/linux/kvm_host.h +0 -7
-#define KVM_ASSIGNED_MSIX_PENDING		0x1
-struct kvm_guest_msix_entry {
-	u32 vector;
-	u16 entry;
-	u16 flags;
-};
-

Ok, so .flags was used to handle the delay between the interrupt handler and the work queue, which is now done automatically by threaded interrupts. Good.

Quoted 5 lines
virt/kvm/assigned-dev.c +0 -1
@@ -313,10 +276,10 @@ static int assigned_device_enable_host_msix(struct kvm *kvm,
  		return r;
 
  	for (i = 0; i<  dev->entries_nr; i++) {
-		r = request_irq(dev->host_msix_entries[i].vector,

virt/kvm/assigned-dev.c static int assigned_device_enable_host_msix(struct kvm *kvm, · line 285

Quoted 10 lines
virt/kvm/assigned-dev.c +4 -3
-				kvm_assigned_dev_intr, 0,
-				"kvm_assigned_msix_device",
-				(void *)dev);
+		r = request_threaded_irq(dev->host_msix_entries[i].vector,
+					 NULL, kvm_assigned_dev_thread,
+					 0, "kvm_assigned_msix_device",
+					 (void *)dev);
  		if (r)
  			goto err;
  	}

Should eventually be done from interrupt context. msix delivery only needs a routing table lookup and smp reschedule IPI, which can be done from interrupt context.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#15 lore raw

Re: [PATCH v4 4/9] KVM: Switch assigned device IRQ forwarding to threaded handler

Quoted 13 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka

This improves the IRQ forwarding for assigned devices: By using the kernel's threaded IRQ scheme, we can get rid of the latency-prone work queue and simplify the code in the same run.

Moreover, we no longer have to hold assigned_dev_lock while raising the guest IRQ, which can be a lenghty operation as we may have to iterate over all VCPUs. The lock is now only used for synchronizing masking vs. unmasking of INTx-type IRQs, thus is renames to intx_lock.

Nice stuff.

Quoted 1 line
Quoted 11 lines

include/linux/kvm_host.h struct kvm_irq_ack_notifier { · line 467

Quoted 7 lines
include/linux/kvm_host.h +0 -7
-#define KVM_ASSIGNED_MSIX_PENDING		0x1
-struct kvm_guest_msix_entry {
-	u32 vector;
-	u16 entry;
-	u16 flags;
-};
-

Ok, so .flags was used to handle the delay between the interrupt handler and the work queue, which is now done automatically by threaded interrupts. Good.

Yes, a nice consolidating side effect.

Quoted 13 lines
Quoted 13 lines
virt/kvm/assigned-dev.c +4 -4
@@ -313,10 +276,10 @@ static int assigned_device_enable_host_msix(struct kvm *kvm,
  		return r;
 
  	for (i = 0; i<  dev->entries_nr; i++) {
-		r = request_irq(dev->host_msix_entries[i].vector,
-				kvm_assigned_dev_intr, 0,
-				"kvm_assigned_msix_device",
-				(void *)dev);
+		r = request_threaded_irq(dev->host_msix_entries[i].vector,
+					 NULL, kvm_assigned_dev_thread,
+					 0, "kvm_assigned_msix_device",
+					 (void *)dev);
Quoted 9 lines

virt/kvm/assigned-dev.c static int assigned_device_enable_host_msix(struct kvm *kvm, · line 285

Quoted 3 lines
virt/kvm/assigned-dev.c +0 -0
  		if (r)
  			goto err;
  	}

Should eventually be done from interrupt context. msix delivery only needs a routing table lookup and smp reschedule IPI, which can be done from interrupt context.

Yes, definitely. But we would have to catch the case when the guest configures the MSI-X message to be a broadcast to all its trillion VCPUs (once we support trillions of them).

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
author
#16 lore raw

PATCH v4 5/9 KVM: Refactor IRQ names of assigned devices

In reply to Jan Kiszka

Cosmetic change, but it helps to correlate IRQs with PCI devices.

---
 include/linux/kvm_host.h |    1 +
 virt/kvm/assigned-dev.c  |   11 ++++++-----
 2 files changed, 7 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +1 -0
index 050674f..fe83eb0 100644
@@ -477,6 +477,7 @@ struct kvm_assigned_dev_kernel {
 	struct pci_dev *dev;
 	struct kvm *kvm;
 	spinlock_t intx_lock;
+	char irq_name[32];
 };
 
 struct kvm_irq_mask_notifier {
virt/kvm/assigned-dev.c +6 -5
index 1d77ce1..7623408 100644
@@ -231,8 +231,7 @@ static int assigned_device_enable_host_intx(struct kvm *kvm,
 	 * are going to be long delays in accepting, acking, etc.
 	 */
 	if (request_threaded_irq(dev->host_irq, NULL, kvm_assigned_dev_thread,
-				 IRQF_ONESHOT, "kvm_assigned_intx_device",
-				 (void *)dev))
+				 IRQF_ONESHOT, dev->irq_name, (void *)dev))
 		return -EIO;
 	return 0;
 }
@@ -251,7 +250,7 @@ static int assigned_device_enable_host_msi(struct kvm *kvm,
 
 	dev->host_irq = dev->dev->irq;
 	if (request_threaded_irq(dev->host_irq, NULL, kvm_assigned_dev_thread,
-				 0, "kvm_assigned_msi_device", (void *)dev)) {
+				 0, dev->irq_name, (void *)dev)) {
 		pci_disable_msi(dev->dev);
 		return -EIO;
 	}
@@ -278,8 +277,7 @@ static int assigned_device_enable_host_msix(struct kvm *kvm,
 	for (i = 0; i < dev->entries_nr; i++) {
 		r = request_threaded_irq(dev->host_msix_entries[i].vector,
 					 NULL, kvm_assigned_dev_thread,
-					 0, "kvm_assigned_msix_device",
-					 (void *)dev);
+					 0, dev->irq_name, (void *)dev);
 		if (r)
 			goto err;
 	}
@@ -336,6 +334,9 @@ static int assign_host_irq(struct kvm *kvm,
 	if (dev->irq_requested_type & KVM_DEV_IRQ_HOST_MASK)
 		return r;
 
+	snprintf(dev->irq_name, sizeof(dev->irq_name), "kvm:%s",
+		 pci_name(dev->dev));
+
 	switch (host_irq_type) {
 	case KVM_DEV_IRQ_HOST_INTX:
 		r = assigned_device_enable_host_intx(kvm, dev);
Signature · 1 line
1.7.1
author
#17 lore raw

PATCH v4 6/9 KVM: Save/restore state of assigned PCI device

In reply to Jan Kiszka

The guest may change states that pci_reset_function does not touch. So we better save/restore the assigned device across guest usage.

---
 virt/kvm/assigned-dev.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
virt/kvm/assigned-dev.c +4 -1
index 7623408..d389207 100644
@@ -197,7 +197,8 @@ static void kvm_free_assigned_device(struct kvm *kvm,
 {
 	kvm_free_assigned_irq(kvm, assigned_dev);
 
-	pci_reset_function(assigned_dev->dev);
+	__pci_reset_function(assigned_dev->dev);
+	pci_restore_state(assigned_dev->dev);
 
 	pci_release_regions(assigned_dev->dev);
 	pci_disable_device(assigned_dev->dev);
@@ -514,6 +515,7 @@ static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
 	}
 
 	pci_reset_function(dev);
+	pci_save_state(dev);
 
 	match->assigned_dev_id = assigned_dev->assigned_dev_id;
 	match->host_segnr = assigned_dev->segnr;
@@ -544,6 +546,7 @@ out:
 	mutex_unlock(&kvm->lock);
 	return r;
 out_list_del:
+	pci_restore_state(dev);
 	list_del(&match->list);
 	pci_release_regions(dev);
 out_disable:
Signature · 1 line
1.7.1
#18 lore raw

Re: [PATCH v4 6/9] KVM: Save/restore state of assigned PCI device

Quoted 2 lines · Jan Kiszka

The guest may change states that pci_reset_function does not touch. So we better save/restore the assigned device across guest usage.

Why do we care? Shouldn't the next user reset the state to its taste? Or are you worried about information leakage?

Quoted 9 lines
virt/kvm/assigned-dev.c +1 -0
@@ -514,6 +515,7 @@ static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
  	}
 
  	pci_reset_function(dev);
+	pci_save_state(dev);
 
  	match->assigned_dev_id = assigned_dev->assigned_dev_id;
  	match->host_segnr = assigned_dev->segnr;

virt/kvm/assigned-dev.c out: · line 552

Quoted 7 lines
virt/kvm/assigned-dev.c +1 -0
@@ -544,6 +546,7 @@ out:
  	mutex_unlock(&kvm->lock);
  	return r;
  out_list_del:
+	pci_restore_state(dev);
  	list_del(&match->list);
  	pci_release_regions(dev);
  out_disable:

This assumes no one else calls pci_save_state()/pci_restore_state() while the guest is running. Is this true? What about suspend/resume? (can this even work without guest assistance?)

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#19 lore raw

Re: [PATCH v4 6/9] KVM: Save/restore state of assigned PCI device

Quoted 5 lines · Avi Kivity
Quoted 2 lines · Jan Kiszka

The guest may change states that pci_reset_function does not touch. So we better save/restore the assigned device across guest usage.

Why do we care? Shouldn't the next user reset the state to its taste?

Maybe he should, but are we sure this actually happens? E.g. pci_reset_function preserves the config state, thus does not remove the traces of guest.

Quoted 1 line

Or are you worried about information leakage?

Anyone who's worried about security should not assign the same device to different users (host or guest) that should be isolated from each other. You never know in what state the firmware and/or internal memory of the device are.

Probably, anyone concerned about security should not give a device in untrusted hands at all...

Quoted 6 lines
Quoted 6 lines
virt/kvm/assigned-dev.c +1 -0
@@ -514,6 +515,7 @@ static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
  	}
 
  	pci_reset_function(dev);
+	pci_save_state(dev);
 
Quoted 12 lines

virt/kvm/assigned-dev.c out: · line 552

Quoted 10 lines
virt/kvm/assigned-dev.c +1 -0
  	match->assigned_dev_id = assigned_dev->assigned_dev_id;
  	match->host_segnr = assigned_dev->segnr;
@@ -544,6 +546,7 @@ out:
  	mutex_unlock(&kvm->lock);
  	return r;
  out_list_del:
+	pci_restore_state(dev);
  	list_del(&match->list);
  	pci_release_regions(dev);
  out_disable:

This assumes no one else calls pci_save_state()/pci_restore_state()

Correct.

Quoted 2 lines

while the guest is running. Is this true? What about suspend/resume? (can this even work without guest assistance?)

Well, so far suspend/resume does not work at all once assigned devices are in use. IIUC, the guest is not informed about the fact that power will be cut off, trashing the states of its assigned devices. We would have to signal this event to the guest and give it a chance to save the states.

Maybe we could generate an ACPI event that corresponds to pressing a standby button. Or we would have to signal PCI unplug events for all assigned devices. In any case, suspend/resume of the host then becomes dependent on guest proper reaction.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#20 lore raw

Re: [PATCH v4 6/9] KVM: Save/restore state of assigned PCI device

Quoted 10 lines · Jan Kiszka
Quoted 5 lines · Avi Kivity
Quoted 2 lines · Jan Kiszka

The guest may change states that pci_reset_function does not touch. So we better save/restore the assigned device across guest usage.

Why do we care? Shouldn't the next user reset the state to its taste?

Maybe he should, but are we sure this actually happens? E.g. pci_reset_function preserves the config state, thus does not remove the traces of guest.

Oh yes, I read the code but it didn't register. Of course this change is quite necessary.

(I understood you to mean that the PCI 2.3 reset doesn't reset everything, but that isn't what you said).

Quoted 9 lines
Quoted 1 line

Or are you worried about information leakage?

Anyone who's worried about security should not assign the same device to different users (host or guest) that should be isolated from each other. You never know in what state the firmware and/or internal memory of the device are.

Probably, anyone concerned about security should not give a device in untrusted hands at all...

Agreed, it's much more suitable to the embedded/fixed function case.

Quoted 4 lines
Quoted 2 lines

This assumes no one else calls pci_save_state()/pci_restore_state()

Correct.

Given the above, this is of secondary concern.

Quoted 9 lines
Quoted 2 lines

while the guest is running. Is this true? What about suspend/resume? (can this even work without guest assistance?)

Well, so far suspend/resume does not work at all once assigned devices are in use. IIUC, the guest is not informed about the fact that power will be cut off, trashing the states of its assigned devices. We would have to signal this event to the guest and give it a chance to save the states.

Quoted 4 lines

Maybe we could generate an ACPI event that corresponds to pressing a standby button. Or we would have to signal PCI unplug events for all assigned devices. In any case, suspend/resume of the host then becomes dependent on guest proper reaction.

Yeah.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#21 lore raw

Re: [PATCH v4 6/9] KVM: Save/restore state of assigned PCI device

Quoted 12 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 5 lines · Avi Kivity
Quoted 2 lines · Jan Kiszka

The guest may change states that pci_reset_function does not touch. So we better save/restore the assigned device across guest usage.

Why do we care? Shouldn't the next user reset the state to its taste?

Maybe he should, but are we sure this actually happens? E.g. pci_reset_function preserves the config state, thus does not remove the traces of guest.

Quoted 5 lines

Oh yes, I read the code but it didn't register. Of course this change is quite necessary.

(I understood you to mean that the PCI 2.3 reset doesn't reset everything, but that isn't what you said).

What the hardware makes out of the reset is even another story. No guarantees I bet (isn't function-level reset an optional thing anyway?).

At least I can report that I managed to kick my Intel 82577LM into limbo land by trying to load the wrong driver in a guest - host reset was required afterward to reclaim its functionality.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#22 lore raw

Re: [PATCH v4 6/9] KVM: Save/restore state of assigned PCI device

Quoted 9 lines · Jan Kiszka
Quoted 6 lines

Oh yes, I read the code but it didn't register. Of course this change is quite necessary.

(I understood you to mean that the PCI 2.3 reset doesn't reset everything, but that isn't what you said).

What the hardware makes out of the reset is even another story. No guarantees I bet (isn't function-level reset an optional thing anyway?).

It is.

Signature · 1 line
error compiling committee.c: too many arguments to function
#23 lore raw

Re: [PATCH v4 6/9] KVM: Save/restore state of assigned PCI device

Quoted 5 lines · Avi Kivity
Quoted 4 lines · Jan Kiszka
Quoted 4 lines

Oh yes, I read the code but it didn't register. Of course this change is quite necessary.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

(I understood you to mean that the PCI 2.3 reset doesn't reset everything, but that isn't what you said).

What the hardware makes out of the reset is even another story. No guarantees I bet (isn't function-level reset an optional thing anyway?).

It is.

optional for PF's; mandatory for VFs.

author
#24 lore raw

PATCH v4 7/9 KVM: Clean up kvm_vm_ioctl_assigned_device

In reply to Jan Kiszka

Any arch not supporting device assigment will also not build assigned-dev.c. So testing for KVM_CAP_DEVICE_DEASSIGNMENT is pointless. KVM_CAP_ASSIGN_DEV_IRQ is unconditinally set. Moreover, add a default case for dispatching the ioctl.

---
 virt/kvm/assigned-dev.c |    9 ++++-----
 1 files changed, 4 insertions(+), 5 deletions(-)
virt/kvm/assigned-dev.c +4 -5
index d389207..ae72ae6 100644
@@ -674,7 +674,7 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 				  unsigned long arg)
 {
 	void __user *argp = (void __user *)arg;
-	int r = -ENOTTY;
+	int r;
 
 	switch (ioctl) {
 	case KVM_ASSIGN_PCI_DEVICE: {
@@ -692,7 +692,6 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 		r = -EOPNOTSUPP;
 		break;
 	}
-#ifdef KVM_CAP_ASSIGN_DEV_IRQ
 	case KVM_ASSIGN_DEV_IRQ: {
 		struct kvm_assigned_irq assigned_irq;
 
@@ -715,8 +714,6 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 			goto out;
 		break;
 	}
-#endif
-#ifdef KVM_CAP_DEVICE_DEASSIGNMENT
 	case KVM_DEASSIGN_PCI_DEVICE: {
 		struct kvm_assigned_pci_dev assigned_dev;
 
@@ -728,7 +725,6 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 			goto out;
 		break;
 	}
-#endif
 #ifdef KVM_CAP_IRQ_ROUTING
 	case KVM_SET_GSI_ROUTING: {
 		struct kvm_irq_routing routing;
@@ -781,6 +777,9 @@ long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 		break;
 	}
 #endif
+	default:
+		r = -ENOTTY;
+		break;
 	}
 out:
 	return r;
Signature · 1 line
1.7.1
author
#25 lore raw

PATCH v4 8/9 KVM: Document device assigment API

In reply to Jan Kiszka

Adds API documentation for KVM_[DE]ASSIGN_PCI_DEVICE, KVM_[DE]ASSIGN_DEV_IRQ, KVM_SET_GSI_ROUTING, KVM_ASSIGN_SET_MSIX_NR, and KVM_ASSIGN_SET_MSIX_ENTRY.

---
 Documentation/kvm/api.txt |  178 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 178 insertions(+), 0 deletions(-)
Documentation/kvm/api.txt +178 -0
index b336266..e1a9297 100644
@@ -1085,6 +1085,184 @@ of 4 instructions that make up a hypercall.
 If any additional field gets added to this structure later on, a bit for that
 additional piece of information will be set in the flags bitmap.
 
+4.47 KVM_ASSIGN_PCI_DEVICE
+
+Capability: KVM_CAP_DEVICE_ASSIGNMENT
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Assigns a host PCI device to the VM.
+
+struct kvm_assigned_pci_dev {
+	__u32 assigned_dev_id;
+	__u32 busnr;
+	__u32 devfn;
+	__u32 flags;
+	__u32 segnr;
+	union {
+		__u32 reserved[11];
+	};
+};
+
+The PCI device is specified by the triple segnr, busnr, and devfn.
+Identification in succeeding service requests is done via assigned_dev_id. The
+following flags are specified:
+
+/* Depends on KVM_CAP_IOMMU */
+#define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
+
+4.48 KVM_DEASSIGN_PCI_DEVICE
+
+Capability: KVM_CAP_DEVICE_DEASSIGNMENT
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Ends PCI device assignment, releasing all associated resources.
+
+See KVM_CAP_DEVICE_ASSIGNMENT for the data structure. Only assigned_dev_id is
+used in kvm_assigned_pci_dev to identify the device.
+
+4.49 KVM_ASSIGN_DEV_IRQ
+
+Capability: KVM_CAP_ASSIGN_DEV_IRQ
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_assigned_irq (in)
+Returns: 0 on success, -1 on error
+
+Assigns an IRQ to a passed-through device.
+
+struct kvm_assigned_irq {
+	__u32 assigned_dev_id;
+	__u32 host_irq;
+	__u32 guest_irq;
+	__u32 flags;
+	union {
+		struct {
+			__u32 addr_lo;
+			__u32 addr_hi;
+			__u32 data;
+		} guest_msi;
+		__u32 reserved[12];
+	};
+};
+
+The following flags are defined:
+
+#define KVM_DEV_IRQ_HOST_INTX    (1 << 0)
+#define KVM_DEV_IRQ_HOST_MSI     (1 << 1)
+#define KVM_DEV_IRQ_HOST_MSIX    (1 << 2)
+
+#define KVM_DEV_IRQ_GUEST_INTX   (1 << 8)
+#define KVM_DEV_IRQ_GUEST_MSI    (1 << 9)
+#define KVM_DEV_IRQ_GUEST_MSIX   (1 << 10)
+
+It is not valid to specify multiple types per host or guest IRQ. However, the
+IRQ type of host and guest can differ or can even be null.
+
+4.50 KVM_DEASSIGN_DEV_IRQ
+
+Capability: KVM_CAP_ASSIGN_DEV_IRQ
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_assigned_irq (in)
+Returns: 0 on success, -1 on error
+
+Ends an IRQ assignment to a passed-through device.
+
+See KVM_ASSIGN_DEV_IRQ for the data structure. The target device is specified
+by assigned_dev_id, flags must correspond to the IRQ type specified on
+KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
+
+4.51 KVM_SET_GSI_ROUTING
+
+Capability: KVM_CAP_IRQ_ROUTING
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_irq_routing (in)
+Returns: 0 on success, -1 on error
+
+Sets the GSI routing table entries, overwriting any previously set entries.
+
+struct kvm_irq_routing {
+	__u32 nr;
+	__u32 flags;
+	struct kvm_irq_routing_entry entries[0];
+};
+
+No flags are specified so far, the corresponding field must be set to zero.
+
+struct kvm_irq_routing_entry {
+	__u32 gsi;
+	__u32 type;
+	__u32 flags;
+	__u32 pad;
+	union {
+		struct kvm_irq_routing_irqchip irqchip;
+		struct kvm_irq_routing_msi msi;
+		__u32 pad[8];
+	} u;
+};
+
+/* gsi routing entry types */
+#define KVM_IRQ_ROUTING_IRQCHIP 1
+#define KVM_IRQ_ROUTING_MSI 2
+
+No flags are specified so far, the corresponding field must be set to zero.
+
+struct kvm_irq_routing_irqchip {
+	__u32 irqchip;
+	__u32 pin;
+};
+
+struct kvm_irq_routing_msi {
+	__u32 address_lo;
+	__u32 address_hi;
+	__u32 data;
+	__u32 pad;
+};
+
+4.52 KVM_ASSIGN_SET_MSIX_NR
+
+Capability: KVM_CAP_DEVICE_MSIX
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_assigned_msix_nr (in)
+Returns: 0 on success, -1 on error
+
+Set the number of MSI-X interrupts for an assigned device. This service can
+only be called once in the lifetime of an assigned device.
+
+struct kvm_assigned_msix_nr {
+	__u32 assigned_dev_id;
+	__u16 entry_nr;
+	__u16 padding;
+};
+
+#define KVM_MAX_MSIX_PER_DEV		256
+
+4.53 KVM_ASSIGN_SET_MSIX_ENTRY
+
+Capability: KVM_CAP_DEVICE_MSIX
+Architectures: x86 ia64
+Type: vm ioctl
+Parameters: struct kvm_assigned_msix_entry (in)
+Returns: 0 on success, -1 on error
+
+Specifies the routing of an MSI-X assigned device interrupt to a GSI. Setting
+the GSI vector to zero means disabling the interrupt.
+
+struct kvm_assigned_msix_entry {
+	__u32 assigned_dev_id;
+	__u32 gsi;
+	__u16 entry; /* The index of entry in the MSI-X table */
+	__u16 padding[3];
+};
+
 5. The kvm_run structure
 
 Application code obtains a pointer to the kvm_run structure by
Signature · 1 line
1.7.1
author
#26 lore raw

PATCH v4 9/9 KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

In reply to Jan Kiszka

PCI 2.3 allows to generically disable IRQ sources at device level. This enables us to share IRQs of such devices between on the host side when passing them to a guest. This feature is optional, user space has to request it explicitly. Moreover, user space can inform us about its view of PCI_COMMAND_INTX_DISABLE so that we can avoid unmasking the interrupt and signaling it if the guest masked it via the PCI config space.

---
 Documentation/kvm/api.txt |   25 ++++
 arch/x86/kvm/x86.c        |    1 +
 include/linux/kvm.h       |    6 +
 include/linux/kvm_host.h  |    2 +
 virt/kvm/assigned-dev.c   |  288 +++++++++++++++++++++++++++++++++++++++++----
 5 files changed, 299 insertions(+), 23 deletions(-)
Documentation/kvm/api.txt +25 -0
index e1a9297..dbb126c 100644
@@ -1112,6 +1112,14 @@ following flags are specified:
 
 /* Depends on KVM_CAP_IOMMU */
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
+/* The following two depend on KVM_CAP_PCI_2_3 */
+#define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
+#define KVM_DEV_ASSIGN_MASK_INTX	(1 << 2)
+
+If KVM_DEV_ASSIGN_PCI_2_3 is set, the kernel will manage legacy INTx interrupts
+via the PCI-2.3-compliant device-level mask, thus enable IRQ sharing with other
+assigned devices or host devices. KVM_DEV_ASSIGN_MASK_INTX specifies the
+guest's view on the INTx mask, see KVM_ASSIGN_SET_INTX_MASK for details.
 
 4.48 KVM_DEASSIGN_PCI_DEVICE
 
@@ -1263,6 +1271,23 @@ struct kvm_assigned_msix_entry {
 	__u16 padding[3];
 };
 
+5.54 KVM_ASSIGN_SET_INTX_MASK
+
+Capability: KVM_CAP_PCI_2_3
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Informs the kernel about the guest's view on the INTx mask. As long as the
+guest masks the legacy INTx, the kernel will refrain from unmasking it at
+hardware level and will not assert the guest's IRQ line. User space is still
+responsible for applying this state to the assigned device's real config space.
+
+See KVM_ASSIGN_DEV_IRQ for the data structure. The target device is specified
+by assigned_dev_id. In the flags field, only KVM_DEV_ASSIGN_MASK_INTX is
+evaluated.
+
 5. The kvm_run structure
 
 Application code obtains a pointer to the kvm_run structure by
arch/x86/kvm/x86.c +1 -0
index 2044302..ed1b417 100644
@@ -1966,6 +1966,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
 	case KVM_CAP_XSAVE:
 	case KVM_CAP_ASYNC_PF:
+	case KVM_CAP_PCI_2_3:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:
include/linux/kvm.h +6 -0
index ea2dc1a..3cadb42 100644
@@ -541,6 +541,7 @@ struct kvm_ppc_pvinfo {
 #define KVM_CAP_PPC_GET_PVINFO 57
 #define KVM_CAP_PPC_IRQ_LEVEL 58
 #define KVM_CAP_ASYNC_PF 59
+#define KVM_CAP_PCI_2_3 60
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -677,6 +678,9 @@ struct kvm_clock_data {
 #define KVM_SET_PIT2              _IOW(KVMIO,  0xa0, struct kvm_pit_state2)
 /* Available with KVM_CAP_PPC_GET_PVINFO */
 #define KVM_PPC_GET_PVINFO	  _IOW(KVMIO,  0xa1, struct kvm_ppc_pvinfo)
+/* Available with KVM_CAP_PCI_2_3 */
+#define KVM_ASSIGN_SET_INTX_MASK  _IOW(KVMIO,  0xa2, \
+				       struct kvm_assigned_pci_dev)
 
 /*
  * ioctls for vcpu fds
@@ -742,6 +746,8 @@ struct kvm_clock_data {
 #define KVM_SET_XCRS		  _IOW(KVMIO,  0xa7, struct kvm_xcrs)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
+#define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
+#define KVM_DEV_ASSIGN_MASK_INTX	(1 << 2)
 
 struct kvm_assigned_pci_dev {
 	__u32 assigned_dev_id;
include/linux/kvm_host.h +2 -0
index fe83eb0..7f1627c 100644
@@ -468,6 +468,7 @@ struct kvm_assigned_dev_kernel {
 	unsigned int entries_nr;
 	int host_irq;
 	bool host_irq_disabled;
+	bool pci_2_3;
 	struct msix_entry *host_msix_entries;
 	int guest_irq;
 	struct msix_entry *guest_msix_entries;
@@ -477,6 +478,7 @@ struct kvm_assigned_dev_kernel {
 	struct pci_dev *dev;
 	struct kvm *kvm;
 	spinlock_t intx_lock;
+	struct mutex intx_mask_lock;
 	char irq_name[32];
 };
 
virt/kvm/assigned-dev.c +265 -23

394 lines not on this page · raw message

Signature · 1 line
1.7.1
#27 lore raw

Re: [PATCH v4 9/9] KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

Quoted 7 lines · Jan Kiszka

PCI 2.3 allows to generically disable IRQ sources at device level. This enables us to share IRQs of such devices between on the host side when passing them to a guest. This feature is optional, user space has to request it explicitly. Moreover, user space can inform us about its view of PCI_COMMAND_INTX_DISABLE so that we can avoid unmasking the interrupt and signaling it if the guest masked it via the PCI config space.

It's a pity this cannot be done transparently. We could detect multiple devices sharing the line, but what about PCI_COMMAND_INTX_DISABLE?

Perhaps we can hook the kernel's handler for this bit?

Quoted 15 lines
  /* Depends on KVM_CAP_IOMMU */
  #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1<<  0)
Documentation/kvm/api.txt +8 -0
+/* The following two depend on KVM_CAP_PCI_2_3 */
+#define KVM_DEV_ASSIGN_PCI_2_3		(1<<  1)
+#define KVM_DEV_ASSIGN_MASK_INTX	(1<<  2)
+
+If KVM_DEV_ASSIGN_PCI_2_3 is set, the kernel will manage legacy INTx interrupts
+via the PCI-2.3-compliant device-level mask, thus enable IRQ sharing with other
+assigned devices or host devices. KVM_DEV_ASSIGN_MASK_INTX specifies the
+guest's view on the INTx mask, see KVM_ASSIGN_SET_INTX_MASK for details.
 
  4.48 KVM_DEASSIGN_PCI_DEVICE

Documentation/kvm/api.txt struct kvm_assigned_msix_entry { · line 1274

Quoted 4 lines
Documentation/kvm/api.txt +1 -0
@@ -1263,6 +1271,23 @@ struct kvm_assigned_msix_entry {
  	__u16 padding[3];
  };
 
+5.54 KVM_ASSIGN_SET_INTX_MASK

4.54?

(54? wow.)

Documentation/kvm/api.txt struct kvm_assigned_msix_entry { · line 1285

Quoted 11 lines
Documentation/kvm/api.txt +11 -0
+
+Capability: KVM_CAP_PCI_2_3
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Informs the kernel about the guest's view on the INTx mask. As long as the
+guest masks the legacy INTx, the kernel will refrain from unmasking it at
+hardware level and will not assert the guest's IRQ line. User space is still
+responsible for applying this state to the assigned device's real config space.

What if userspace lies?

Quoted 19 lines
Documentation/kvm/api.txt +5 -0
+
+See KVM_ASSIGN_DEV_IRQ for the data structure. The target device is specified
+by assigned_dev_id. In the flags field, only KVM_DEV_ASSIGN_MASK_INTX is
+evaluated.
+
 
include/linux/kvm_host.h +1 -0
index fe83eb0..7f1627c 100644
@@ -468,6 +468,7 @@ struct kvm_assigned_dev_kernel {
  	unsigned int entries_nr;
  	int host_irq;
  	bool host_irq_disabled;
+	bool pci_2_3;
  	struct msix_entry *host_msix_entries;
  	int guest_irq;
  	struct msix_entry *guest_msix_entries;

include/linux/kvm_host.h struct kvm_assigned_dev_kernel { · line 483

Quoted 6 lines
include/linux/kvm_host.h +1 -0
@@ -477,6 +478,7 @@ struct kvm_assigned_dev_kernel {
  	struct pci_dev *dev;
  	struct kvm *kvm;
  	spinlock_t intx_lock;
+	struct mutex intx_mask_lock;
  	char irq_name[32];
  };

I saw no reason this can't be a spinlock, but perhaps I missed something. This would allow us to avoid srcu, which is slightly more expensive than rcu. Since pci 2.3 assigned devices are not a major use case, I'd like not to penalize the mainstream users for this.

This patch undoes some of the niceness of the previous patches, but I have no alternative to suggest.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#28 lore raw

Re: [PATCH v4 9/9] KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

Quoted 11 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka

PCI 2.3 allows to generically disable IRQ sources at device level. This enables us to share IRQs of such devices between on the host side when passing them to a guest. This feature is optional, user space has to request it explicitly. Moreover, user space can inform us about its view of PCI_COMMAND_INTX_DISABLE so that we can avoid unmasking the interrupt and signaling it if the guest masked it via the PCI config space.

It's a pity this cannot be done transparently. We could detect multiple devices sharing the line,

Even that is not possible. Assigned or host devices may be activated after we registered exclusively, pushing the breakage from VM start-up to a different operation.

Quoted 3 lines

but what about PCI_COMMAND_INTX_DISABLE?

Perhaps we can hook the kernel's handler for this bit?

Some IRQ registration notifier that would allow us to reregister our handler with IRQ sharing support? Maybe.

Quoted 15 lines
Quoted 15 lines

  /* Depends on KVM_CAP_IOMMU */
  #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1<<  0)
Documentation/kvm/api.txt +8 -0
+/* The following two depend on KVM_CAP_PCI_2_3 */
+#define KVM_DEV_ASSIGN_PCI_2_3		(1<<  1)
+#define KVM_DEV_ASSIGN_MASK_INTX	(1<<  2)
+
+If KVM_DEV_ASSIGN_PCI_2_3 is set, the kernel will manage legacy INTx interrupts
+via the PCI-2.3-compliant device-level mask, thus enable IRQ sharing with other
+assigned devices or host devices. KVM_DEV_ASSIGN_MASK_INTX specifies the
+guest's view on the INTx mask, see KVM_ASSIGN_SET_INTX_MASK for details.
 
  4.48 KVM_DEASSIGN_PCI_DEVICE
Quoted 7 lines

Documentation/kvm/api.txt struct kvm_assigned_msix_entry { · line 1274

Quoted 5 lines
Documentation/kvm/api.txt +1 -0
@@ -1263,6 +1271,23 @@ struct kvm_assigned_msix_entry {
  	__u16 padding[3];
  };
 
+5.54 KVM_ASSIGN_SET_INTX_MASK

4.54?

Of course.

Quoted 2 lines

(54? wow.)

And I don't think all IOCTLs are already documented (though the majority now).

Quoted 14 lines

Documentation/kvm/api.txt struct kvm_assigned_msix_entry { · line 1285

Quoted 12 lines
Documentation/kvm/api.txt +11 -0
+
+Capability: KVM_CAP_PCI_2_3
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Informs the kernel about the guest's view on the INTx mask. As long as the
+guest masks the legacy INTx, the kernel will refrain from unmasking it at
+hardware level and will not assert the guest's IRQ line. User space is still
+responsible for applying this state to the assigned device's real config space.

What if userspace lies?

User space problem. We will at worst receive one IRQ, mask it, and then user space need to react again.

Quoted 23 lines
Quoted 23 lines
Documentation/kvm/api.txt +5 -0
+
+See KVM_ASSIGN_DEV_IRQ for the data structure. The target device is specified
+by assigned_dev_id. In the flags field, only KVM_DEV_ASSIGN_MASK_INTX is
+evaluated.
+
 
include/linux/kvm_host.h +1 -0
index fe83eb0..7f1627c 100644
@@ -468,6 +468,7 @@ struct kvm_assigned_dev_kernel {
  	unsigned int entries_nr;
  	int host_irq;
  	bool host_irq_disabled;
+	bool pci_2_3;
  	struct msix_entry *host_msix_entries;
  	int guest_irq;
  	struct msix_entry *guest_msix_entries;
@@ -477,6 +478,7 @@ struct kvm_assigned_dev_kernel {
  	struct pci_dev *dev;
  	struct kvm *kvm;
  	spinlock_t intx_lock;
Quoted 8 lines

include/linux/kvm_host.h struct kvm_assigned_dev_kernel { · line 483

Quoted 3 lines
include/linux/kvm_host.h +1 -0
+	struct mutex intx_mask_lock;
  	char irq_name[32];
  };

I saw no reason this can't be a spinlock, but perhaps I missed something. This would allow us to avoid srcu, which is slightly more expensive than rcu. Since pci 2.3 assigned devices are not a major use case, I'd like not to penalize the mainstream users for this.

The lock has to be held across kvm_set_irq, which is the potentially expensive (O(n), n == number of VCPUs) operation.

Quoted 4 lines

This patch undoes some of the niceness of the previous patches, but I have no alternative to suggest.

Yes, it surely does not make things simpler. But much of the complexity is avoided during runtime when MSIs are used.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#29 lore raw

Re: [PATCH v4 9/9] KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

Quoted 10 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka

PCI 2.3 allows to generically disable IRQ sources at device level. This enables us to share IRQs of such devices between on the host side when passing them to a guest. This feature is optional, user space has to request it explicitly. Moreover, user space can inform us about its view of PCI_COMMAND_INTX_DISABLE so that we can avoid unmasking the interrupt and signaling it if the guest masked it via the PCI config space.

Quoted 6 lines
Quoted 2 lines

It's a pity this cannot be done transparently. We could detect multiple devices sharing the line,

Even that is not possible. Assigned or host devices may be activated after we registered exclusively, pushing the breakage from VM start-up to a different operation.

We could detect that and switch the interrupt mode. Or we could always to IRQF_SHARED and fake something in the immediate callback.

Quoted 6 lines
Quoted 3 lines

but what about PCI_COMMAND_INTX_DISABLE?

Perhaps we can hook the kernel's handler for this bit?

Some IRQ registration notifier that would allow us to reregister our handler with IRQ sharing support? Maybe.

Adding an internal API if preferable to an external one (it may be a pain to kvm-kmod users though).

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

+
+Capability: KVM_CAP_PCI_2_3
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Informs the kernel about the guest's view on the INTx mask. As long as the +guest masks the legacy INTx, the kernel will refrain from unmasking it at +hardware level and will not assert the guest's IRQ line. User space is still +responsible for applying this state to the assigned device's real config space.

Quoted 4 lines
Quoted 1 line

What if userspace lies?

User space problem. We will at worst receive one IRQ, mask it, and then user space need to react again.

Ok.

Quoted 8 lines
Quoted 5 lines

I saw no reason this can't be a spinlock, but perhaps I missed something. This would allow us to avoid srcu, which is slightly more expensive than rcu. Since pci 2.3 assigned devices are not a major use case, I'd like not to penalize the mainstream users for this.

The lock has to be held across kvm_set_irq, which is the potentially expensive (O(n), n == number of VCPUs) operation.

What we should probably do is have broadcast interrupts deferred to a thread. I agree it isn't pretty.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#30 lore raw

Re: [PATCH v4 9/9] KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

Quoted 23 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka

PCI 2.3 allows to generically disable IRQ sources at device level. This enables us to share IRQs of such devices between on the host side when passing them to a guest. This feature is optional, user space has to request it explicitly. Moreover, user space can inform us about its view of PCI_COMMAND_INTX_DISABLE so that we can avoid unmasking the interrupt and signaling it if the guest masked it via the PCI config space.

Quoted 6 lines
Quoted 2 lines

It's a pity this cannot be done transparently. We could detect multiple devices sharing the line,

Even that is not possible. Assigned or host devices may be activated after we registered exclusively, pushing the breakage from VM start-up to a different operation.

We could detect that and switch the interrupt mode. Or we could always to IRQF_SHARED and fake something in the immediate callback.

Quoted 2 lines
Quoted 2 lines

but what about PCI_COMMAND_INTX_DISABLE?

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Perhaps we can hook the kernel's handler for this bit?

Some IRQ registration notifier that would allow us to reregister our handler with IRQ sharing support? Maybe.

Adding an internal API if preferable to an external one (it may be a pain to kvm-kmod users though).

Primary concern should be a clean and robust API. From that POV, I would prefer an official hook with genirq maintainer blessing over fragile detection heuristics in kvm. Given that VSIO should benefit from any transparent pattern we develop here as well, it's probably worth to go that path - if it is really preferred over manual control like this patch proposes.

For kvm-kmod, we could simply enforce IRQ sharing measures unconditionally. Not optimal from the performance POV, but people concerned that much about performance should better use KVM over the corresponding kernel anyway.

Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

+
+Capability: KVM_CAP_PCI_2_3
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_assigned_pci_dev (in)
+Returns: 0 on success, -1 on error
+
+Informs the kernel about the guest's view on the INTx mask. As long as the +guest masks the legacy INTx, the kernel will refrain from unmasking it at +hardware level and will not assert the guest's IRQ line. User space is still +responsible for applying this state to the assigned device's real config space.

Quoted 4 lines
Quoted 1 line

What if userspace lies?

User space problem. We will at worst receive one IRQ, mask it, and then user space need to react again.

Ok.

Quoted 6 lines
Quoted 5 lines

I saw no reason this can't be a spinlock, but perhaps I missed something. This would allow us to avoid srcu, which is slightly more expensive than rcu. Since pci 2.3 assigned devices are not a major use case, I'd like not to penalize the mainstream users for this.

Quoted 5 lines
Quoted 2 lines

The lock has to be held across kvm_set_irq, which is the potentially expensive (O(n), n == number of VCPUs) operation.

What we should probably do is have broadcast interrupts deferred to a thread. I agree it isn't pretty.

Not sure we could defer it that easily (if at all). However, I think such improvements should be done on top if this already complex change.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#31 lore raw

Re: [PATCH v4 9/9] KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

Quoted 8 lines · Jan Kiszka
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

but what about PCI_COMMAND_INTX_DISABLE?

Perhaps we can hook the kernel's handler for this bit?

Some IRQ registration notifier that would allow us to reregister our handler with IRQ sharing support? Maybe.

Quoted 6 lines
Quoted 2 lines

Adding an internal API if preferable to an external one (it may be a pain to kvm-kmod users though).

Primary concern should be a clean and robust API. From that POV, I would prefer an official hook with genirq maintainer blessing over fragile detection heuristics in kvm.

That is what I meant (internal to the kernel, not to kvm, vs the external kvm/user interface).

Quoted 4 lines

Given that VSIO should benefit from any
transparent pattern we develop here as well, it's probably worth to go that path - if it is really preferred over manual control like this patch proposes.

Yes. The API is already sufficiently complicated that it is very hard for userspace to get right.

Quoted 4 lines

For kvm-kmod, we could simply enforce IRQ sharing measures unconditionally. Not optimal from the performance POV, but people concerned that much about performance should better use KVM over the corresponding kernel anyway.

Or backport the needed patches. kvm-kmod is now no longer needed for enterprise users with fixed distros, it's mostly useful for the embedded guys which have more flexibility.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

I saw no reason this can't be a spinlock, but perhaps I missed something. This would allow us to avoid srcu, which is slightly more expensive than rcu. Since pci 2.3 assigned devices are not a major use case, I'd like not to penalize the mainstream users for this.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The lock has to be held across kvm_set_irq, which is the potentially expensive (O(n), n == number of VCPUs) operation.

What we should probably do is have broadcast interrupts deferred to a thread. I agree it isn't pretty.

Not sure we could defer it that easily (if at all). However, I think such improvements should be done on top if this already complex change.

That means flipping locking to srcu and back to rcu. What I'm proposing is to keep broadcast within the spinlock for now, and move it later, provided we agree that it's doable.

Signature · 1 line
error compiling committee.c: too many arguments to function
#32 lore raw
In reply to Jan Kiszka
Quoted 12 lines · Jan Kiszka

Nine patches (yeah, it's getting more and more) to improve "classic" device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers.

As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid:

disable_irq_nosync:           ~600 cycles
pci_2_3_irq_check_and_mask:  ~6000 cycles (EHCI)
                            ~22000 cycles (AR9287, with peaks >100000)
Quoted 5 lines

Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well.

Looks fine to me. Michael, Alex?

Also needs a rebase.

author
#33 lore raw
Quoted 19 lines · Marcelo Tosatti
Quoted 12 lines · Jan Kiszka

Nine patches (yeah, it's getting more and more) to improve "classic" device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers.

As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid:

disable_irq_nosync:           ~600 cycles
pci_2_3_irq_check_and_mask:  ~6000 cycles (EHCI)
                            ~22000 cycles (AR9287, with peaks >100000)
Quoted 5 lines

Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well.

Quoted 3 lines

Looks fine to me. Michael, Alex?

Also needs a rebase.

I think Avi wanted to explore possibilities to avoid host_pci_2_3=on|off by switching between both modes on demand. I've some ideas, but still need to look into design details. Given that this would have impact on the ABI, I guess we better wait with merging 9/9.

Moreover, Avi preferred to skip the srcu conversion and run with a simply lock across kvm_set_irq (just like we do now).

But I could offer to rebase and repost the independent rest of the series tonight.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#34 lore raw
Quoted 20 lines · Jan Kiszka
Quoted 19 lines · Marcelo Tosatti
Quoted 17 lines · Jan Kiszka

Nine patches (yeah, it's getting more and more) to improve "classic" device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers.

As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid:

disable_irq_nosync: ~600 cycles
pci_2_3_irq_check_and_mask: ~6000 cycles (EHCI)

                             ~22000 cycles (AR9287, with peaks>100000)

Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well.

Quoted 8 lines
Quoted 3 lines

Looks fine to me. Michael, Alex?

Also needs a rebase.

I think Avi wanted to explore possibilities to avoid host_pci_2_3=on|off by switching between both modes on demand. I've some ideas, but still need to look into design details. Given that this would have impact on the ABI, I guess we better wait with merging 9/9.

Indeed.

Quoted 2 lines

Moreover, Avi preferred to skip the srcu conversion and run with a simply lock across kvm_set_irq (just like we do now).

Yes. It was predicated on agreeing that deferring broadcast/multicast IPIs to a workqueue is possible.

Signature · 1 line
error compiling committee.c: too many arguments to function