MSI-X mask support for assigned device

We need to query the entry later.

virt/kvm/irq_comm.c +10 -0

+int kvm_get_irq_routing_entry(struct kvm *kvm, int gsi,
+		struct kvm_kernel_irq_routing_entry *entry)
+{
+	int count = 0;
+	struct kvm_kernel_irq_routing_entry *ei = NULL;
+	struct kvm_irq_routing_table *irq_rt;
+	struct hlist_node *n;
+
+	rcu_read_lock();
+	irq_rt = rcu_dereference(kvm->irq_routing);

Quoted 12 lines · Sheng Yang

We need to query the entry later.

virt/kvm/irq_comm.c +10 -0

+int kvm_get_irq_routing_entry(struct kvm *kvm, int gsi,
+		struct kvm_kernel_irq_routing_entry *entry)
+{
+	int count = 0;
+	struct kvm_kernel_irq_routing_entry *ei = NULL;
+	struct kvm_irq_routing_table *irq_rt;
+	struct hlist_node *n;
+
+	rcu_read_lock();
+	irq_rt = rcu_dereference(kvm->irq_routing);

virt/kvm/irq_comm.c out: · line 443

Quoted 10 lines

virt/kvm/irq_comm.c +10 -0

+	if (gsi<  irq_rt->nr_rt_entries)
+		hlist_for_each_entry(ei, n,&irq_rt->map[gsi], link)
+			count++;
+	if (count == 1)
+		*entry = *ei;
+	rcu_read_unlock();
+
+	return (count != 1);
+}
+

Not good form to rely on ei being valid after the loop.

I guess this is only useful for msi? Need to document it.

*entry may be stale after rcu_read_unlock(). Is this a problem?

Quoted 24 lines · On Wednesday 17 November 2010 22:01:41 Avi Kivity wrote:

Quoted 22 lines · Sheng Yang

We need to query the entry later.

+int kvm_get_irq_routing_entry(struct kvm *kvm, int gsi,
+ struct kvm_kernel_irq_routing_entry *entry)
+{
+ int count = 0;
+ struct kvm_kernel_irq_routing_entry *ei = NULL;
+ struct kvm_irq_routing_table *irq_rt;
+ struct hlist_node *n;
+
+ rcu_read_lock();
+ irq_rt = rcu_dereference(kvm->irq_routing);
+ if (gsi< irq_rt->nr_rt_entries)
+ hlist_for_each_entry(ei, n,&irq_rt->map[gsi], link)
+ count++;
+ if (count == 1)
+ *entry = *ei;
+ rcu_read_unlock();
+
+ return (count != 1);
+}
+

Quoted 3 lines

Not good form to rely on ei being valid after the loop.

I guess this is only useful for msi? Need to document it.

May can be used for others later, it's somehow generic. Where should I document it?

Quoted 2 lines

*entry may be stale after rcu_read_unlock(). Is this a problem?

I suppose not. All MSI-X MMIO accessing would be executed without delay, so no re- order issue would happen. If the guest is reading and writing the field at the same time(from two cpus), it should got some kinds of sync method for itself - or it may not care what's the reading result(like the one after msix_mask_irq()).

Quoted 7 lines

Quoted 2 lines

*entry may be stale after rcu_read_unlock(). Is this a problem?

I suppose not. All MSI-X MMIO accessing would be executed without delay, so no re- order issue would happen. If the guest is reading and writing the field at the same time(from two cpus), it should got some kinds of sync method for itself - or it may not care what's the reading result(like the one after msix_mask_irq()).

I guess so. Michael/Alex?

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 8 lines · Avi Kivity

Quoted 7 lines

Quoted 2 lines

 *entry may be stale after rcu_read_unlock().  Is this a problem?

I suppose not. All MSI-X MMIO accessing would be executed without delay, so no re- order issue would happen. If the guest is reading and writing the field at the same time(from two cpus), it should got some kinds of sync method for itself - or it may not care what's the reading result(like the one after msix_mask_irq()).

Quoted 1 line

I guess so.  Michael/Alex?

This is kvm_get_irq_routing_entry which is used for table reads, correct?  Actually, the pci read *is* the sync method that guests use, they rely on reads to flush out all previous writes.

Quoted 9 lines · Michael S. Tsirkin

Quoted 8 lines · Avi Kivity

Quoted 7 lines

Quoted 2 lines

 *entry may be stale after rcu_read_unlock().  Is this a problem?

I suppose not. All MSI-X MMIO accessing would be executed without delay, so no re- order issue would happen. If the guest is reading and writing the field at the same time(from two cpus), it should got some kinds of sync method for itself - or it may not care what's the reading result(like the one after msix_mask_irq()).

Quoted 5 lines

Quoted 1 line

I guess so.  Michael/Alex?

This is kvm_get_irq_routing_entry which is used for table reads, correct?  Actually, the pci read *is* the sync method that guests use, they rely on reads to flush out all previous writes.

Michael, I think the *sync* you are talking about is not the one I meant. I was talking about two cpus case, one is reading and the other is writing, the order can't be determined if guest doesn't use lock or some other synchronize methods; and you're talking about to flush out all previous writes of the only one CPU...

Signature · 2 lines

regards,
Yang, Sheng

Quoted 16 lines · Sheng Yang

Quoted 14 lines · Michael S. Tsirkin

Quoted 9 lines · Avi Kivity

Quoted 7 lines

Quoted 2 lines

 *entry may be stale after rcu_read_unlock().  Is this a problem?

I suppose not. All MSI-X MMIO accessing would be executed without delay, so no re- order issue would happen. If the guest is reading and writing the field at the same time(from two cpus), it should got some kinds of sync method for itself - or it may not care what's the reading result(like the one after msix_mask_irq()).

I guess so.  Michael/Alex?

This is kvm_get_irq_routing_entry which is used for table reads, correct?  Actually, the pci read *is* the sync method that guests use, they rely on reads to flush out all previous writes.

Quoted 5 lines

Michael, I think the *sync* you are talking about is not the one I meant. I was talking about two cpus case, one is reading and the other is writing, the order can't be determined if guest doesn't use lock or some other synchronize methods; and you're talking about to flush out all previous writes of the only one CPU...

Yes, but you don't seem to flush out writes on a read, either.

This patch enable per-vector mask for assigned devices using MSI-X.

This patch provided two new APIs: one is for guest to specific device's MSI-X table address in MMIO, the other is for userspace to get information about mask bit.

All the mask bit operation are kept in kernel, in order to accelerate. Userspace shouldn't access the device MMIO directly for the information, instead it should uses provided API to do so.

Signed-off-by Sheng Yang <[email protected]>

---
 arch/x86/kvm/x86.c       |    1 +
 include/linux/kvm.h      |   32 +++++
 include/linux/kvm_host.h |    5 +
 virt/kvm/assigned-dev.c  |  318 +++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 355 insertions(+), 1 deletions(-)

arch/x86/kvm/x86.c +1 -0

index fc29223..37602e2 100644
@@ -1966,6 +1966,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
 	case KVM_CAP_XSAVE:
 	case KVM_CAP_ASYNC_PF:
+	case KVM_CAP_MSIX_MASK:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:

include/linux/kvm.h +6 -0

index ea2dc1a..b3e5ffe 100644
@@ -541,6 +541,9 @@ struct kvm_ppc_pvinfo {
 #define KVM_CAP_PPC_GET_PVINFO 57
 #define KVM_CAP_PPC_IRQ_LEVEL 58
 #define KVM_CAP_ASYNC_PF 59
+#ifdef __KVM_HAVE_MSIX
+#define KVM_CAP_MSIX_MASK 60
+#endif
 
 #ifdef KVM_CAP_IRQ_ROUTING
 

@@ -672,6 +675,9 @@ struct kvm_clock_data {
 #define KVM_XEN_HVM_CONFIG        _IOW(KVMIO,  0x7a, struct kvm_xen_hvm_config)
 #define KVM_SET_CLOCK             _IOW(KVMIO,  0x7b, struct kvm_clock_data)
 #define KVM_GET_CLOCK             _IOR(KVMIO,  0x7c, struct kvm_clock_data)
+/* Available with KVM_CAP_MSIX_MASK */
+#define KVM_GET_MSIX_ENTRY        _IOWR(KVMIO,  0x7d, struct kvm_msix_entry)
+#define KVM_UPDATE_MSIX_MMIO      _IOW(KVMIO,  0x7e, struct kvm_msix_mmio)
 /* Available with KVM_CAP_PIT_STATE2 */
 #define KVM_GET_PIT2              _IOR(KVMIO,  0x9f, struct kvm_pit_state2)
 #define KVM_SET_PIT2              _IOW(KVMIO,  0xa0, struct kvm_pit_state2)

@@ -795,4 +801,30 @@ struct kvm_assigned_msix_entry {
 	__u16 padding[3];

include/linux/kvm.h +18 -0

+	__u32 entry; /* The index of entry in the MSI-X table */
+	__u32 flags;
+	__u32 query_flags;
+	__u32 reserved[5];
+};
+
+#define KVM_MSIX_MMIO_FLAG_REGISTER	(1 << 0)
+#define KVM_MSIX_MMIO_FLAG_UNREGISTER	(1 << 1)
+
+struct kvm_msix_mmio {
+	__u32 id;
+	__u32 type;
+	__u64 base_addr;
+	__u32 max_entries_nr;
+	__u32 flags;
+	__u32 reserved[6];
+};
+
 #endif /* __LINUX_KVM_H */

include/linux/kvm_host.h +5 -0

index f09db87..57a437a 100644
@@ -501,6 +501,7 @@ struct kvm_guest_msix_entry {
 };
 
 #define KVM_ASSIGNED_ENABLED_IOMMU		(1 << 0)
+#define KVM_ASSIGNED_ENABLED_MSIX_MMIO		(1 << 1)
 struct kvm_assigned_dev_kernel {
 	struct kvm_irq_ack_notifier ack_notifier;
 	struct work_struct interrupt_work;

@@ -521,6 +522,10 @@ struct kvm_assigned_dev_kernel {
 	struct pci_dev *dev;
 	struct kvm *kvm;
 	spinlock_t assigned_dev_lock;
+	DECLARE_BITMAP(msix_mask_bitmap, KVM_MAX_MSIX_PER_DEV);
+	gpa_t msix_mmio_base;
+	struct kvm_io_device msix_mmio_dev;
+	int msix_max_entries_nr;
 };
 
 struct kvm_irq_mask_notifier {

virt/kvm/assigned-dev.c +131 -1

index 5c6b96d..76a1f12 100644
@@ -226,12 +226,27 @@ static void kvm_free_assigned_irq(struct kvm *kvm,
 	kvm_deassign_irq(kvm, assigned_dev, assigned_dev->irq_requested_type);
 }
 
+static void unregister_msix_mmio(struct kvm *kvm,
+				 struct kvm_assigned_dev_kernel *adev)
+{
+	if (adev->flags & KVM_ASSIGNED_ENABLED_MSIX_MMIO) {
+		mutex_lock(&kvm->slots_lock);
+		kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
+				&adev->msix_mmio_dev);
+		mutex_unlock(&kvm->slots_lock);
+		adev->flags &= ~KVM_ASSIGNED_ENABLED_MSIX_MMIO;
+	}
+}
+
 static void kvm_free_assigned_device(struct kvm *kvm,
 				     struct kvm_assigned_dev_kernel
 				     *assigned_dev)
 {
 	kvm_free_assigned_irq(kvm, assigned_dev);
 
+#ifdef __KVM_HAVE_MSIX
+	unregister_msix_mmio(kvm, assigned_dev);
+#endif
 	pci_reset_function(assigned_dev->dev);
 
 	pci_release_regions(assigned_dev->dev);

@@ -504,7 +519,7 @@ out:
 static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
 				      struct kvm_assigned_pci_dev *assigned_dev)
 {
-	int r = 0, idx;
+	int r = 0, idx, i;
 	struct kvm_assigned_dev_kernel *match;
 	struct pci_dev *dev;
 

@@ -564,6 +579,10 @@ static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
 
 	list_add(&match->list, &kvm->arch.assigned_dev_head);
 
+	/* The state after reset of MSI-X table is all masked */
+	for (i = 0; i < KVM_MAX_MSIX_PER_DEV; i++)
+		set_bit(i, match->msix_mask_bitmap);
+
 	if (assigned_dev->flags & KVM_ASSIGNED_ENABLED_IOMMU) {
 		if (!kvm->arch.iommu_domain) {
 			r = kvm_iommu_map_guest(kvm);

@@ -667,6 +686,43 @@ msix_nr_out:
 	return r;
 }
 
+static void update_msix_mask(struct kvm_assigned_dev_kernel *adev,
+			     int idx, bool new_mask_flag)
+{
+	int irq;
+	bool old_mask_flag, need_flush = false;
+
+	spin_lock_irq(&adev->assigned_dev_lock);
+
+	if (!adev->dev->msix_enabled ||
+	    !(adev->irq_requested_type & KVM_DEV_IRQ_HOST_MSIX))
+		goto out;
+
+	old_mask_flag = test_bit(adev->guest_msix_entries[idx].entry,
+			adev->msix_mask_bitmap);
+	if (old_mask_flag == new_mask_flag)
+		goto out;
+
+	irq = adev->host_msix_entries[idx].vector;
+	BUG_ON(irq == 0);
+
+	if (new_mask_flag) {
+		set_bit(adev->guest_msix_entries[idx].entry,
+				adev->msix_mask_bitmap);
+		disable_irq_nosync(irq);
+		need_flush = true;
+	} else {
+		clear_bit(adev->guest_msix_entries[idx].entry,
+				adev->msix_mask_bitmap);
+		enable_irq(irq);
+	}
+out:
+	spin_unlock_irq(&adev->assigned_dev_lock);
+
+	if (need_flush)
+		flush_work(&adev->interrupt_work);
+}
+
 static int kvm_vm_ioctl_set_msix_entry(struct kvm *kvm,
 				       struct kvm_assigned_msix_entry *entry)
 {

@@ -701,6 +757,235 @@ msix_entry_out:
 
 	return r;
 }
+
+static int kvm_vm_ioctl_get_msix_entry(struct kvm *kvm,
+				       struct kvm_msix_entry *entry)
+{
+	int r = 0;
+	struct kvm_assigned_dev_kernel *adev;
+
+	if (entry->type != KVM_MSIX_TYPE_ASSIGNED_DEV)
+		return -EINVAL;
+
+	if (!entry->query_flags)
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+
+	adev = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
+				      entry->id);
+
+	if (!adev) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if (entry->entry >= adev->msix_max_entries_nr) {
+		r = -ENOSPC;
+		goto out;
+	}
+
+	if (entry->query_flags & KVM_MSIX_FLAG_QUERY_MASKBIT) {
+		if (test_bit(entry->entry, adev->msix_mask_bitmap))
+			entry->flags |= KVM_MSIX_FLAG_MASKBIT;
+		else
+			entry->flags &= ~KVM_MSIX_FLAG_MASKBIT;
+	}
+
+out:
+	mutex_unlock(&kvm->lock);
+
+	return r;
+}
+
+static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel *adev,
+			      gpa_t addr, int len)
+{
+	gpa_t start, end;
+
+	BUG_ON(!(adev->flags & KVM_ASSIGNED_ENABLED_MSIX_MMIO));
+	start = adev->msix_mmio_base;
+	end = adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE *
+		adev->msix_max_entries_nr;
+	if (addr >= start && addr + len <= end)
+		return true;
+
+	return false;
+}
+
+static int msix_get_enabled_idx(struct kvm_assigned_dev_kernel *adev,
+			      gpa_t addr, int len)
+{
+	int i, index = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+
+	for (i = 0; i < adev->entries_nr; i++)
+		if (adev->guest_msix_entries[i].entry == index)
+			return i;
+
+	return -EINVAL;
+}
+
+static int msix_mmio_read(struct kvm_io_device *this, gpa_t addr, int len,
+			  void *val)
+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);

virt/kvm/assigned-dev.c +18 -0

+	if ((addr & 0x3) || len != 4)
+		goto out;
+
+	idx = msix_get_enabled_idx(adev, addr, len);
+	if (idx < 0) {
+		idx = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+		if ((addr % PCI_MSIX_ENTRY_SIZE) ==
+				PCI_MSIX_ENTRY_VECTOR_CTRL)
+			*(unsigned long *)val =
+				test_bit(idx, adev->msix_mask_bitmap) ?
+				PCI_MSIX_ENTRY_CTRL_MASKBIT : 0;
+		else
+			r = -EOPNOTSUPP;
+		goto out;
+	}
+
+	r = kvm_get_irq_routing_entry(adev->kvm,
+			adev->guest_msix_entries[idx].vector, &e);

Quoted 10 lines · Sheng Yang

This patch enable per-vector mask for assigned devices using MSI-X.

This patch provided two new APIs: one is for guest to specific device's MSI-X table address in MMIO, the other is for userspace to get information about mask bit.

All the mask bit operation are kept in kernel, in order to accelerate. Userspace shouldn't access the device MMIO directly for the information, instead it should uses provided API to do so.

Quoted 8 lines

Signed-off-by Sheng Yang <[email protected]>

---

 arch/x86/kvm/x86.c       |    1 +
 include/linux/kvm.h      |   32 +++++
 include/linux/kvm_host.h |    5 +
 virt/kvm/assigned-dev.c  |  318

+++++++++++++++++++++++++++++++++++++++++++++- 4 files changed,

Quoted 56 lines

insertions(+), 1 deletions(-)

arch/x86/kvm/x86.c +1 -0

index fc29223..37602e2 100644
@@ -1966,6 +1966,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
 	case KVM_CAP_XSAVE:
 
 	case KVM_CAP_ASYNC_PF:
+	case KVM_CAP_MSIX_MASK:
 		r = 1;

arch/x86/kvm/x86.c +0 -0

 		break;
 
 	case KVM_CAP_COALESCED_MMIO:

include/linux/kvm.h +3 -0

index ea2dc1a..b3e5ffe 100644
@@ -541,6 +541,9 @@ struct kvm_ppc_pvinfo {
 
 #define KVM_CAP_PPC_GET_PVINFO 57
 #define KVM_CAP_PPC_IRQ_LEVEL 58
 #define KVM_CAP_ASYNC_PF 59
 
+#ifdef __KVM_HAVE_MSIX
+#define KVM_CAP_MSIX_MASK 60
+#endif
 

#ifdef KVM_CAP_IRQ_ROUTING

include/linux/kvm.h +2 -0

@@ -672,6 +675,9 @@ struct kvm_clock_data {
 
 #define KVM_XEN_HVM_CONFIG        _IOW(KVMIO,  0x7a, struct
 kvm_xen_hvm_config) #define KVM_SET_CLOCK             _IOW(KVMIO, 
 0x7b, struct kvm_clock_data) #define KVM_GET_CLOCK            
 _IOR(KVMIO,  0x7c, struct kvm_clock_data)
 
+/* Available with KVM_CAP_MSIX_MASK */
+#define KVM_GET_MSIX_ENTRY        _IOWR(KVMIO,  0x7d, struct

kvm_msix_entry) +#define KVM_UPDATE_MSIX_MMIO _IOW(KVMIO, 0x7e,
struct kvm_msix_mmio)

/* Available with KVM_CAP_PIT_STATE2 */
#define KVM_GET_PIT2 _IOR(KVMIO, 0x9f, struct
kvm_pit_state2) #define KVM_SET_PIT2 _IOW(KVMIO, 0xa0,
struct kvm_pit_state2)

include/linux/kvm.h +0 -0

@@ -795,4 +801,30 @@ struct kvm_assigned_msix_entry {
 
 	__u16 padding[3];
 
 };

include/linux/kvm.h struct kvm_assigned_msix_entry { · line 811

Quoted 8 lines

include/linux/kvm.h +8 -0

+#define KVM_MSIX_TYPE_ASSIGNED_DEV	1
+
+#define KVM_MSIX_FLAG_MASKBIT		(1 << 0)
+#define KVM_MSIX_FLAG_QUERY_MASKBIT	(1 << 0)
+
+struct kvm_msix_entry {
+	__u32 id;
+	__u32 type;

Is type really necessary? Will it ever differ from KVM_MSIX_TYPE_ASSIGNED_DEV?

Quoted 251 lines

include/linux/kvm.h +18 -0

+	__u32 entry; /* The index of entry in the MSI-X table */
+	__u32 flags;
+	__u32 query_flags;
+	__u32 reserved[5];
+};
+
+#define KVM_MSIX_MMIO_FLAG_REGISTER	(1 << 0)
+#define KVM_MSIX_MMIO_FLAG_UNREGISTER	(1 << 1)
+
+struct kvm_msix_mmio {
+	__u32 id;
+	__u32 type;
+	__u64 base_addr;
+	__u32 max_entries_nr;
+	__u32 flags;
+	__u32 reserved[6];
+};
+

#endif /* __LINUX_KVM_H */

include/linux/kvm_host.h +1 -0

index f09db87..57a437a 100644
@@ -501,6 +501,7 @@ struct kvm_guest_msix_entry {
 
 };
 
 #define KVM_ASSIGNED_ENABLED_IOMMU		(1 << 0)
 
+#define KVM_ASSIGNED_ENABLED_MSIX_MMIO		(1 << 1)
 

struct kvm_assigned_dev_kernel {

include/linux/kvm_host.h +4 -0

 	struct kvm_irq_ack_notifier ack_notifier;
 	struct work_struct interrupt_work;

@@ -521,6 +522,10 @@ struct kvm_assigned_dev_kernel {
 
 	struct pci_dev *dev;
 	struct kvm *kvm;
 	spinlock_t assigned_dev_lock;
 
+	DECLARE_BITMAP(msix_mask_bitmap, KVM_MAX_MSIX_PER_DEV);
+	gpa_t msix_mmio_base;
+	struct kvm_io_device msix_mmio_dev;
+	int msix_max_entries_nr;
 

};

struct kvm_irq_mask_notifier {

virt/kvm/assigned-dev.c +16 -1

index 5c6b96d..76a1f12 100644
@@ -226,12 +226,27 @@ static void kvm_free_assigned_irq(struct kvm *kvm,
 
 	kvm_deassign_irq(kvm, assigned_dev, assigned_dev->irq_requested_type);
 
 }
 
+static void unregister_msix_mmio(struct kvm *kvm,
+				 struct kvm_assigned_dev_kernel *adev)
+{
+	if (adev->flags & KVM_ASSIGNED_ENABLED_MSIX_MMIO) {
+		mutex_lock(&kvm->slots_lock);
+		kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
+				&adev->msix_mmio_dev);
+		mutex_unlock(&kvm->slots_lock);
+		adev->flags &= ~KVM_ASSIGNED_ENABLED_MSIX_MMIO;
+	}
+}
+
 
 static void kvm_free_assigned_device(struct kvm *kvm,
 
 				     struct kvm_assigned_dev_kernel
 				     *assigned_dev)
 
 {
 
 	kvm_free_assigned_irq(kvm, assigned_dev);
 
+#ifdef __KVM_HAVE_MSIX
+	unregister_msix_mmio(kvm, assigned_dev);
+#endif
 
 	pci_reset_function(assigned_dev->dev);
 
 	pci_release_regions(assigned_dev->dev);

@@ -504,7 +519,7 @@ out:
 static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
 
 				      struct kvm_assigned_pci_dev *assigned_dev)
 
 {
 
-	int r = 0, idx;
+	int r = 0, idx, i;

virt/kvm/assigned-dev.c +4 -0

 	struct kvm_assigned_dev_kernel *match;
 	struct pci_dev *dev;

@@ -564,6 +579,10 @@ static int kvm_vm_ioctl_assign_device(struct kvm
 *kvm,
 
 	list_add(&match->list, &kvm->arch.assigned_dev_head);
 
+	/* The state after reset of MSI-X table is all masked */
+	for (i = 0; i < KVM_MAX_MSIX_PER_DEV; i++)
+		set_bit(i, match->msix_mask_bitmap);
+
 
 	if (assigned_dev->flags & KVM_ASSIGNED_ENABLED_IOMMU) {

virt/kvm/assigned-dev.c +37 -0

 		if (!kvm->arch.iommu_domain) {
 
 			r = kvm_iommu_map_guest(kvm);

@@ -667,6 +686,43 @@ msix_nr_out:
 	return r;
 
 }
 
+static void update_msix_mask(struct kvm_assigned_dev_kernel *adev,
+			     int idx, bool new_mask_flag)
+{
+	int irq;
+	bool old_mask_flag, need_flush = false;
+
+	spin_lock_irq(&adev->assigned_dev_lock);
+
+	if (!adev->dev->msix_enabled ||
+	    !(adev->irq_requested_type & KVM_DEV_IRQ_HOST_MSIX))
+		goto out;
+
+	old_mask_flag = test_bit(adev->guest_msix_entries[idx].entry,
+			adev->msix_mask_bitmap);
+	if (old_mask_flag == new_mask_flag)
+		goto out;
+
+	irq = adev->host_msix_entries[idx].vector;
+	BUG_ON(irq == 0);
+
+	if (new_mask_flag) {
+		set_bit(adev->guest_msix_entries[idx].entry,
+				adev->msix_mask_bitmap);
+		disable_irq_nosync(irq);
+		need_flush = true;
+	} else {
+		clear_bit(adev->guest_msix_entries[idx].entry,
+				adev->msix_mask_bitmap);
+		enable_irq(irq);
+	}
+out:
+	spin_unlock_irq(&adev->assigned_dev_lock);
+
+	if (need_flush)
+		flush_work(&adev->interrupt_work);
+}
+
 
 static int kvm_vm_ioctl_set_msix_entry(struct kvm *kvm,

virt/kvm/assigned-dev.c +73 -0

 				       struct kvm_assigned_msix_entry *entry)
 
 {

@@ -701,6 +757,235 @@ msix_entry_out:
 	return r;
 
 }
 
+
+static int kvm_vm_ioctl_get_msix_entry(struct kvm *kvm,
+				       struct kvm_msix_entry *entry)
+{
+	int r = 0;
+	struct kvm_assigned_dev_kernel *adev;
+
+	if (entry->type != KVM_MSIX_TYPE_ASSIGNED_DEV)
+		return -EINVAL;
+
+	if (!entry->query_flags)
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+
+	adev = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
+				      entry->id);
+
+	if (!adev) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if (entry->entry >= adev->msix_max_entries_nr) {
+		r = -ENOSPC;
+		goto out;
+	}
+
+	if (entry->query_flags & KVM_MSIX_FLAG_QUERY_MASKBIT) {
+		if (test_bit(entry->entry, adev->msix_mask_bitmap))
+			entry->flags |= KVM_MSIX_FLAG_MASKBIT;
+		else
+			entry->flags &= ~KVM_MSIX_FLAG_MASKBIT;
+	}
+
+out:
+	mutex_unlock(&kvm->lock);
+
+	return r;
+}
+
+static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel *adev,
+			      gpa_t addr, int len)
+{
+	gpa_t start, end;
+
+	BUG_ON(!(adev->flags & KVM_ASSIGNED_ENABLED_MSIX_MMIO));
+	start = adev->msix_mmio_base;
+	end = adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE *
+		adev->msix_max_entries_nr;
+	if (addr >= start && addr + len <= end)
+		return true;
+
+	return false;
+}
+
+static int msix_get_enabled_idx(struct kvm_assigned_dev_kernel *adev,
+			      gpa_t addr, int len)
+{
+	int i, index = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+
+	for (i = 0; i < adev->entries_nr; i++)
+		if (adev->guest_msix_entries[i].entry == index)
+			return i;
+
+	return -EINVAL;
+}
+
+static int msix_mmio_read(struct kvm_io_device *this, gpa_t addr, int
 len, +			  void *val)
+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);

virt/kvm/assigned-dev.c msix_entry_out: · line 844

Quoted 10 lines

virt/kvm/assigned-dev.c +10 -0

+	int idx, r = 0;
+	u32 entry[4];
+	struct kvm_kernel_irq_routing_entry e;
+
+	/* TODO: Get big-endian machine work */
+	mutex_lock(&adev->kvm->lock);
+	if (!msix_mmio_in_range(adev, addr, len)) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}

The unregister path does:

mutex_lock(kvm->lock)
kvm_io_bus_unregister_dev()
synchronize_srcu()

If an instance of msix_mmio_read/msix_mmio_write is waiting on kvm->lock, synchronize_srcu will never complete.

You should use a separate lock for the in range check (and have it mind that reads/writes can trigger after kvm_io_bus_register_dev, so all state accessible in the r/w handlers should be complete by that time).

Quoted 23 lines

virt/kvm/assigned-dev.c +22 -0

+	if ((addr & 0x3) || len != 4)
+		goto out;
+
+	idx = msix_get_enabled_idx(adev, addr, len);
+	if (idx < 0) {
+		idx = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+		if ((addr % PCI_MSIX_ENTRY_SIZE) ==
+				PCI_MSIX_ENTRY_VECTOR_CTRL)
+			*(unsigned long *)val =
+				test_bit(idx, adev->msix_mask_bitmap) ?
+				PCI_MSIX_ENTRY_CTRL_MASKBIT : 0;
+		else
+			r = -EOPNOTSUPP;
+		goto out;
+	}
+
+	r = kvm_get_irq_routing_entry(adev->kvm,
+			adev->guest_msix_entries[idx].vector, &e);
+	if (r || e.type != KVM_IRQ_ROUTING_MSI) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}

virt/kvm/assigned-dev.c msix_entry_out: · line 871

Quoted 6 lines

virt/kvm/assigned-dev.c +6 -0

+	entry[0] = e.msi.address_lo;
+	entry[1] = e.msi.address_hi;
+	entry[2] = e.msi.data;
+	entry[3] = test_bit(adev->guest_msix_entries[idx].entry,
+			adev->msix_mask_bitmap);
+	memcpy(val, &entry[addr % PCI_MSIX_ENTRY_SIZE / sizeof *entry], len);

Division by zero?

Quoted 4 lines

virt/kvm/assigned-dev.c msix_entry_out: · line 871

Quoted 2 lines

virt/kvm/assigned-dev.c +2 -0

+			adev->msix_mask_bitmap);
+	memcpy(val, &entry[addr % PCI_MSIX_ENTRY_SIZE / sizeof *entry], len);

Division by zero?

Not quite understand. You mean sizeof *entry or PCI_MSIX_ENTRY_SIZE? Both of them should be positive integer I think... Maybe I should use sizeof u32 here?

Quoted 11 lines

include/linux/kvm.h struct kvm_assigned_msix_entry { · line 811

Quoted 8 lines

include/linux/kvm.h +8 -0

+#define KVM_MSIX_TYPE_ASSIGNED_DEV	1
+
+#define KVM_MSIX_FLAG_MASKBIT		(1 << 0)
+#define KVM_MSIX_FLAG_QUERY_MASKBIT	(1 << 0)
+
+struct kvm_msix_entry {
+	__u32 id;
+	__u32 type;

Is type really necessary? Will it ever differ from KVM_MSIX_TYPE_ASSIGNED_DEV?

This is the suggestion from Michael. He want it to be reused by emulated/pv devices. So I add the type field here.

This patch enable per-vector mask for assigned devices using MSI-X.

This patch provided two new APIs: one is for guest to specific device's MSI-X table address in MMIO, the other is for userspace to get information about mask bit.

All the mask bit operation are kept in kernel, in order to accelerate. Userspace shouldn't access the device MMIO directly for the information, instead it should uses provided API to do so.

Signed-off-by Sheng Yang<[email protected]>

---
  arch/x86/kvm/x86.c       |    1 +
  include/linux/kvm.h      |   32 +++++
  include/linux/kvm_host.h |    5 +
  virt/kvm/assigned-dev.c  |  318 +++++++++++++++++++++++++++++++++++++++++++++-
  4 files changed, 355 insertions(+), 1 deletions(-)

virt/kvm/assigned-dev.c +48 -0

+static int msix_mmio_read(struct kvm_io_device *this, gpa_t addr, int len,
+			  void *val)
+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);
+	int idx, r = 0;
+	u32 entry[4];
+	struct kvm_kernel_irq_routing_entry e;
+
+	/* TODO: Get big-endian machine work */
+	mutex_lock(&adev->kvm->lock);
+	if (!msix_mmio_in_range(adev, addr, len)) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+	if ((addr&  0x3) || len != 4)
+		goto out;
+
+	idx = msix_get_enabled_idx(adev, addr, len);
+	if (idx<  0) {
+		idx = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+		if ((addr % PCI_MSIX_ENTRY_SIZE) ==
+				PCI_MSIX_ENTRY_VECTOR_CTRL)
+			*(unsigned long *)val =
+				test_bit(idx, adev->msix_mask_bitmap) ?
+				PCI_MSIX_ENTRY_CTRL_MASKBIT : 0;
+		else
+			r = -EOPNOTSUPP;
+		goto out;
+	}
+
+	r = kvm_get_irq_routing_entry(adev->kvm,
+			adev->guest_msix_entries[idx].vector,&e);
+	if (r || e.type != KVM_IRQ_ROUTING_MSI) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+	entry[0] = e.msi.address_lo;
+	entry[1] = e.msi.address_hi;
+	entry[2] = e.msi.data;
+	entry[3] = test_bit(adev->guest_msix_entries[idx].entry,
+			adev->msix_mask_bitmap);
+	memcpy(val,&entry[addr % PCI_MSIX_ENTRY_SIZE / sizeof *entry], len);
+
+out:
+	mutex_unlock(&adev->kvm->lock);
+	return r;

virt/kvm/assigned-dev.c +7 -0

+
+	idx = msix_get_enabled_idx(adev, addr, len);
+	if (idx<  0) {
+		idx = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+		if (((addr % PCI_MSIX_ENTRY_SIZE) ==
+				PCI_MSIX_ENTRY_VECTOR_CTRL)) {
+			if (new_val&  ~PCI_MSIX_ENTRY_CTRL_MASKBIT)

virt/kvm/assigned-dev.c +16 -0

+		goto out;
+	}
+	if (addr % PCI_MSIX_ENTRY_SIZE != PCI_MSIX_ENTRY_VECTOR_CTRL) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+	if (new_val&  ~PCI_MSIX_ENTRY_CTRL_MASKBIT)
+		goto out;
+	update_msix_mask(adev, idx, !!(new_val&  PCI_MSIX_ENTRY_CTRL_MASKBIT));
+out:
+	mutex_unlock(&adev->kvm->lock);
+
+	return r;
+}
+
+static int kvm_vm_ioctl_update_msix_mmio(struct kvm *kvm,

virt/kvm/assigned-dev.c +37 -0

+
+	mutex_lock(&kvm->lock);
+	adev = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
+				      msix_mmio->id);
+	if (!adev) {
+		r = -EINVAL;
+		goto out;
+	}
+	if (msix_mmio->base_addr == 0) {
+		r = -EINVAL;
+		goto out;
+	}
+	if (msix_mmio->max_entries_nr == 0 ||
+			msix_mmio->max_entries_nr>  KVM_MAX_MSIX_PER_DEV) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if ((msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_REGISTER)&&
+			(msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_UNREGISTER)) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if (msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_REGISTER) {
+		if (!(adev->flags&  KVM_ASSIGNED_ENABLED_MSIX_MMIO)) {
+			mutex_lock(&kvm->slots_lock);
+			kvm_iodevice_init(&adev->msix_mmio_dev,
+					&msix_mmio_ops);
+			r = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS,
+					&adev->msix_mmio_dev);
+			if (!r)
+				adev->flags |= KVM_ASSIGNED_ENABLED_MSIX_MMIO;
+			mutex_unlock(&kvm->slots_lock);
+		}
+		if (!r) {
+			adev->msix_mmio_base = msix_mmio->base_addr;

Quoted 10 lines · Sheng Yang

This patch enable per-vector mask for assigned devices using MSI-X.

This patch provided two new APIs: one is for guest to specific device's MSI-X table address in MMIO, the other is for userspace to get information about mask bit.

All the mask bit operation are kept in kernel, in order to accelerate. Userspace shouldn't access the device MMIO directly for the information, instead it should uses provided API to do so.

Quoted 8 lines

Signed-off-by Sheng Yang<[email protected]>

---

  arch/x86/kvm/x86.c       |    1 +
  include/linux/kvm.h      |   32 +++++
  include/linux/kvm_host.h |    5 +
  virt/kvm/assigned-dev.c  |  318

+++++++++++++++++++++++++++++++++++++++++++++- 4 files changed,

Quoted 1 line

insertions(+), 1 deletions(-)

Documentation?

Quoted 53 lines

+static int msix_mmio_read(struct kvm_io_device *this, gpa_t addr, int
len, + void *val)

virt/kvm/assigned-dev.c +49 -0

+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);
+	int idx, r = 0;
+	u32 entry[4];
+	struct kvm_kernel_irq_routing_entry e;
+
+	/* TODO: Get big-endian machine work */
+	mutex_lock(&adev->kvm->lock);
+	if (!msix_mmio_in_range(adev, addr, len)) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+	if ((addr&  0x3) || len != 4)
+		goto out;
+
+	idx = msix_get_enabled_idx(adev, addr, len);
+	if (idx<  0) {
+		idx = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+		if ((addr % PCI_MSIX_ENTRY_SIZE) ==
+				PCI_MSIX_ENTRY_VECTOR_CTRL)
+			*(unsigned long *)val =
+				test_bit(idx, adev->msix_mask_bitmap) ?
+				PCI_MSIX_ENTRY_CTRL_MASKBIT : 0;
+		else
+			r = -EOPNOTSUPP;
+		goto out;
+	}
+
+	r = kvm_get_irq_routing_entry(adev->kvm,
+			adev->guest_msix_entries[idx].vector,&e);
+	if (r || e.type != KVM_IRQ_ROUTING_MSI) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+	entry[0] = e.msi.address_lo;
+	entry[1] = e.msi.address_hi;
+	entry[2] = e.msi.data;
+	entry[3] = test_bit(adev->guest_msix_entries[idx].entry,
+			adev->msix_mask_bitmap);
+	memcpy(val,&entry[addr % PCI_MSIX_ENTRY_SIZE / sizeof *entry], len);
+
+out:
+	mutex_unlock(&adev->kvm->lock);
+	return r;
+}
+
+static int msix_mmio_write(struct kvm_io_device *this, gpa_t addr, int

len, + const void *val)

virt/kvm/assigned-dev.c msix_entry_out: · line 885

Quoted 6 lines

virt/kvm/assigned-dev.c +6 -0

+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);
+	int idx, r = 0;
+	unsigned long new_val = *(unsigned long *)val;

What if it's a 64-bit write on a 32-bit host?

Are we sure the trailing bytes of val are zero?

Review

Quoted 2 lines

(quoted diff) +2 -0

+
+	/* TODO: Get big-endian machine work */

BUILD_BUG_ON(something)

virt/kvm/assigned-dev.c msix_entry_out: · line 843

Quoted 6 lines

virt/kvm/assigned-dev.c +5 -0

+	mutex_lock(&adev->kvm->lock);
+	if (!msix_mmio_in_range(adev, addr, len)) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}

Why is this needed? Didn't the iodev check already do this?

Review

Quoted 3 lines

(quoted diff) +2 -0

+	if ((addr&  0x3) || len != 4)
+		goto out;

What if len == 8? I think mst said it was legal.

Quoted 14 lines

virt/kvm/assigned-dev.c +13 -0

+
+	idx = msix_get_enabled_idx(adev, addr, len);
+	if (idx<  0) {
+		idx = (addr - adev->msix_mmio_base) / PCI_MSIX_ENTRY_SIZE;
+		if (((addr % PCI_MSIX_ENTRY_SIZE) ==
+				PCI_MSIX_ENTRY_VECTOR_CTRL)) {
+			if (new_val&  ~PCI_MSIX_ENTRY_CTRL_MASKBIT)
+				goto out;
+			if (new_val&  PCI_MSIX_ENTRY_CTRL_MASKBIT)
+				set_bit(idx, adev->msix_mask_bitmap);
+			else
+				clear_bit(idx, adev->msix_mask_bitmap);
+			/* It's possible that we need re-enable MSI-X, so go

virt/kvm/assigned-dev.c msix_entry_out: · line 911

Quoted 4 lines

virt/kvm/assigned-dev.c +4 -0

+			 * back to userspace */
+		}
+		/* Userspace would handle other MMIO writing */
+		r = -EOPNOTSUPP;

That's not very good. We should do the entire thing in the kernel or in userspace. We can have a new EXIT_REASON to let userspace know an msix entry changed, and it should read it from the kernel.

Quoted 21 lines

virt/kvm/assigned-dev.c +20 -0

+		goto out;
+	}
+	if (addr % PCI_MSIX_ENTRY_SIZE != PCI_MSIX_ENTRY_VECTOR_CTRL) {
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+	if (new_val&  ~PCI_MSIX_ENTRY_CTRL_MASKBIT)
+		goto out;
+	update_msix_mask(adev, idx, !!(new_val&  PCI_MSIX_ENTRY_CTRL_MASKBIT));
+out:
+	mutex_unlock(&adev->kvm->lock);
+
+	return r;
+}
+
+static int kvm_vm_ioctl_update_msix_mmio(struct kvm *kvm,
+				struct kvm_msix_mmio *msix_mmio)
+{
+	int r = 0;
+	struct kvm_assigned_dev_kernel *adev;

virt/kvm/assigned-dev.c msix_entry_out: · line 942

Quoted 6 lines

virt/kvm/assigned-dev.c +6 -0

+
+	if (msix_mmio->type != KVM_MSIX_TYPE_ASSIGNED_DEV)
+		return -EINVAL;
+
+	if (!msix_mmio->flags)
+		return -EINVAL;

Need to check flags for undefined bits too.

Quoted 39 lines

virt/kvm/assigned-dev.c +38 -0

+
+	mutex_lock(&kvm->lock);
+	adev = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
+				      msix_mmio->id);
+	if (!adev) {
+		r = -EINVAL;
+		goto out;
+	}
+	if (msix_mmio->base_addr == 0) {
+		r = -EINVAL;
+		goto out;
+	}
+	if (msix_mmio->max_entries_nr == 0 ||
+			msix_mmio->max_entries_nr>  KVM_MAX_MSIX_PER_DEV) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if ((msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_REGISTER)&&
+			(msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_UNREGISTER)) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if (msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_REGISTER) {
+		if (!(adev->flags&  KVM_ASSIGNED_ENABLED_MSIX_MMIO)) {
+			mutex_lock(&kvm->slots_lock);
+			kvm_iodevice_init(&adev->msix_mmio_dev,
+					&msix_mmio_ops);
+			r = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS,
+					&adev->msix_mmio_dev);
+			if (!r)
+				adev->flags |= KVM_ASSIGNED_ENABLED_MSIX_MMIO;
+			mutex_unlock(&kvm->slots_lock);
+		}
+		if (!r) {
+			adev->msix_mmio_base = msix_mmio->base_addr;
+			adev->msix_max_entries_nr = msix_mmio->max_entries_nr;

virt/kvm/assigned-dev.c msix_entry_out:

Quoted 10 lines

virt/kvm/assigned-dev.c +8 -0

+		}
+	} else if (msix_mmio->flags&  KVM_MSIX_MMIO_FLAG_UNREGISTER)
+		unregister_msix_mmio(kvm, adev);
+out:
+	mutex_unlock(&kvm->lock);
+
+	return r;
+}
 
  #endif

Question: how do we do this with vfio? One idea is to have

ioctl(vmfd, KVM_IO_PIPE, { KVM_IO_MMIO /* or KVM_IO_PIO */, range,
pipefd1, pipefd2 })

and have kvm convert mmio or pio in that range to commands and responses sent over that pipe. We could have qemu threads, other userspace processes, or kernel components like vfio implement the other side of the channel.

Quoted 10 lines

virt/kvm/assigned-dev.c msix_entry_out: · line 885

Quoted 8 lines

+static int msix_mmio_write(struct kvm_io_device *this, gpa_t addr, int
len, + const void *val)

virt/kvm/assigned-dev.c +6 -0

+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);
+	int idx, r = 0;
+	unsigned long new_val = *(unsigned long *)val;

What if it's a 64-bit write on a 32-bit host?

In fact we haven't support QWORD(64bit) accessing now. The reason is we haven't seen any OS is using it in this way now, so I think we can leave it later.

Also seems QEmu doesn't got the way to handle 64bit MMIO.

Quoted 15 lines · Sheng Yang

Quoted 10 lines

virt/kvm/assigned-dev.c msix_entry_out: · line 885

Quoted 8 lines

+static int msix_mmio_write(struct kvm_io_device *this, gpa_t addr,
int len, + const void *val)

virt/kvm/assigned-dev.c +6 -0

+{
+	struct kvm_assigned_dev_kernel *adev =
+			container_of(this, struct kvm_assigned_dev_kernel,
+				     msix_mmio_dev);
+	int idx, r = 0;
+	unsigned long new_val = *(unsigned long *)val;

What if it's a 64-bit write on a 32-bit host?

In fact we haven't support QWORD(64bit) accessing now. The reason is we haven't seen any OS is using it in this way now, so I think we can leave it later.

Quoted 1 line

Also seems QEmu doesn't got the way to handle 64bit MMIO.

I think it does. I think it simply splits these to 32-bit transactions and handles as such. That seems to be spec-compilant. I wouldn't want us to regress.

Quoted 11 lines · On Wednesday 17 November 2010 21:58:00 Avi Kivity wrote:

Quoted 10 lines · Sheng Yang

This patch enable per-vector mask for assigned devices using MSI-X.

This patch provided two new APIs: one is for guest to specific device's MSI-X table address in MMIO, the other is for userspace to get information about mask bit.

All the mask bit operation are kept in kernel, in order to accelerate. Userspace shouldn't access the device MMIO directly for the information, instead it should uses provided API to do so.

Quoted 8 lines

Quoted 8 lines

Signed-off-by Sheng Yang<[email protected]>

---

   arch/x86/kvm/x86.c       |    1 +
   include/linux/kvm.h      |   32 +++++
   include/linux/kvm_host.h |    5 +
   virt/kvm/assigned-dev.c  |  318

+++++++++++++++++++++++++++++++++++++++++++++- 4 files changed,

355

Quoted 3 lines

Quoted 1 line

   insertions(+), 1 deletions(-)

Documentation?

For we are keeping changing the API for last several versions, I'd like to settle down the API first. Would bring back the document after API was agreed.

Quoted 1 line

What if it's a 64-bit write on a 32-bit host?

In fact we haven't support QWORD(64bit) accessing now. The reason is we haven't seen any OS is using it in this way now, so I think we can leave it later.

Also seems QEmu doesn't got the way to handle 64bit MMIO.

Quoted 4 lines

That's not very good. We should do the entire thing in the kernel or in userspace. We can have a new EXIT_REASON to let userspace know an msix entry changed, and it should read it from the kernel.

If you look it in this way:
1. Mask bit owned by kernel.
2. Routing owned by userspace.
3. Read the routing in kernel is an speed up for normal operation - because kernel can read from them.

So I think the logic here is clear to understand.

But if we can modify the routing in kernel, it would be raise some sync issues due to both kernel and userspace own routing. So maybe the better solution is move the routing to kernel.

Quoted 20 lines · Sheng Yang

Quoted 11 lines · On Wednesday 17 November 2010 21:58:00 Avi Kivity wrote:

Quoted 10 lines · Sheng Yang

This patch enable per-vector mask for assigned devices using MSI-X.

This patch provided two new APIs: one is for guest to specific device's MSI-X table address in MMIO, the other is for userspace to get information about mask bit.

All the mask bit operation are kept in kernel, in order to accelerate. Userspace shouldn't access the device MMIO directly for the information, instead it should uses provided API to do so.

Quoted 8 lines

Quoted 8 lines

Signed-off-by Sheng Yang<[email protected]>

---

   arch/x86/kvm/x86.c       |    1 +
   include/linux/kvm.h      |   32 +++++
   include/linux/kvm_host.h |    5 +
   virt/kvm/assigned-dev.c  |  318

+++++++++++++++++++++++++++++++++++++++++++++- 4 files changed,

Quoted 7 lines

355

Quoted 3 lines

Quoted 1 line

   insertions(+), 1 deletions(-)

Documentation?

For we are keeping changing the API for last several versions, I'd like to settle down the API first. Would bring back the document after API was agreed.

Maybe for APIs we should start with only the documentation patch, agree on that, and move on to the implementation.

Quoted 6 lines

Quoted 1 line

What if it's a 64-bit write on a 32-bit host?

In fact we haven't support QWORD(64bit) accessing now. The reason is we haven't seen any OS is using it in this way now, so I think we can leave it later.

Also seems QEmu doesn't got the way to handle 64bit MMIO.

There's a difference, if the API doesn't support it, we can't add it later without changing both kernel and userspace.

Quoted 5 lines

Quoted 4 lines

That's not very good. We should do the entire thing in the kernel or in userspace. We can have a new EXIT_REASON to let userspace know an msix entry changed, and it should read it from the kernel.

Quoted 7 lines

If you look it in this way:
1. Mask bit owned by kernel.
2. Routing owned by userspace.
3. Read the routing in kernel is an speed up for normal operation - because kernel can read from them.

So I think the logic here is clear to understand.

Still, it's complicated and the state is split across multiple components.

Quoted 3 lines

But if we can modify the routing in kernel, it would be raise some sync issues due to both kernel and userspace own routing. So maybe the better solution is move the routing to kernel.

That may work, but I don't think we can do this for vfio.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 27 lines · Sheng Yang

Quoted 14 lines · On Wednesday 17 November 2010 21:58:00 Avi Kivity wrote:

 On 11/15/2010 11:15 AM, Sheng Yang wrote:
 >  This patch enable per-vector mask for assigned devices using MSI-X. >
 >  This patch provided two new APIs: one is for guest to specific device's
 >  MSI-X table address in MMIO, the other is for userspace to get >  information about mask bit.
 >
 >  All the mask bit operation are kept in kernel, in order to accelerate.
 >  Userspace shouldn't access the device MMIO directly for the information,
 >  instead it should uses provided API to do so.
 >

Quoted 8 lines

 >  Signed-off-by: Sheng Yang<[email protected]>
 >  ---
 >
 >    arch/x86/kvm/x86.c       |    1 +
 >    include/linux/kvm.h      |   32 +++++
 >    include/linux/kvm_host.h |    5 +
 >    virt/kvm/assigned-dev.c  |  318
 >    +++++++++++++++++++++++++++++++++++++++++++++- 4 files changed,

355

Quoted 2 lines

 >    insertions(+), 1 deletions(-)

Quoted 5 lines

Quoted 1 line

 Documentation?

For we are keeping changing the API for last several versions, I'd like to settle
down the API first. Would bring back the document after API was agreed.

Maybe for APIs we should start with only the documentation patch, agree on that, and move on to the implementation.

Quoted 8 lines

Quoted 2 lines

 What if it's a 64-bit write on a 32-bit host?

In fact we haven't support QWORD(64bit) accessing now. The reason is we haven't
seen any OS is using it in this way now, so I think we can leave it later.

Also seems QEmu doesn't got the way to handle 64bit MMIO.

There's a difference, if the API doesn't support it, we can't add it later without changing both kernel and userspace.

Quoted 14 lines

Quoted 6 lines

 That's not very good.  We should do the entire thing in the kernel or in
 userspace.  We can have a new EXIT_REASON to let userspace know an msix entry changed, and it should read it from the kernel.

If you look it in this way:
1. Mask bit owned by kernel.
2. Routing owned by userspace.
3. Read the routing in kernel is an speed up for normal operation - because kernel
can read from them.

Quoted 1 line

So I think the logic here is clear to understand.

Still, it's complicated and the state is split across multiple components.

Quoted 6 lines

But if we can modify the routing in kernel, it would be raise some sync issues due
to both kernel and userspace own routing. So maybe the better solution is move the
routing to kernel.

That may work, but I don't think we can do this for vfio.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html