MSI-X enabling
v2 cover MSI-X enabling
Based on former patchset.
PATCH v2 1/3 KVM: Ioctls for init MSI-X entry
Introduce KVM_SET_MSIX_NR and KVM_SET_MSIX_ENTRY two ioctls.
This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
--- include/linux/kvm.h | 14 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 0 deletions(-)
include/linux/kvm.h +14 -0
@@ -474,6 +474,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
@@ -594,4 +596,16 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +}; + +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry; +__u16 pos; +}; +#endif
include/linux/kvm_host.h +3 -0
@@ -325,9 +325,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)
virt/kvm/kvm_main.c +96 -0
109 lines not on this page · raw message
Signature · 1 line
1.5.4.5
Re: [PATCH 1/3] KVM: Ioctls for init MSI-X entry
Quoted 34 lines · Sheng Yang
Introduce
KVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRYtwo ioctls.This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
Signed-off-by Sheng Yang <[email protected]>--- include/linux/kvm.h | 14 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 0 deletions(-)include/linux/kvm.h +3
-0@@ -474,6 +474,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds @@ -594,4 +596,16 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr {
include/linux/kvm.h struct kvm_assigned_irq { · line 609
Quoted 10 lines
include/linux/kvm.h +10
-0+__u32 assigned_dev_id; +__u16 entry_nr; +}; + +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry; +__u16 pos; +}
I'm guessing the intent here is "msi-x number 'pos' on the host assigned device will be injected to the guest as gsi 'gsi'"? What's the meaning of 'entry' and 'pos'?
Both structures need better padding and some documentation.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 1/3] KVM: Ioctls for init MSI-X entry
Quoted 37 lines · On Wednesday 11 February 2009 20:44:57 Avi Kivity wrote:
Quoted 36 lines · Sheng Yang
Introduce
KVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRYtwo ioctls.This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
Signed-off-by Sheng Yang <[email protected]>--- include/linux/kvm.h | 14 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 96++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 0 deletions(-)
include/linux/kvm.h +2
-0@@ -474,6 +474,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
include/linux/kvm.h +0
-0@@ -594,4 +596,16 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0)
Quoted 15 lines
include/linux/kvm.h struct kvm_assigned_irq {
Quoted 11 lines
include/linux/kvm.h +11
-0+struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +}; + +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry; +__u16 pos; +}I'm guessing the intent here is "msi-x number 'pos' on the host assigned device will be injected to the guest as gsi 'gsi'"? What's the meaning of 'entry' and 'pos'?
Sorry for the ambiguous. "entry" is the index in device MSI-X table(including empty entry); "pos" is without the empty entry, and should be less than the total valid entry numbers that table have. "pos" is mostly a assistant here, I think it can be estimated.
Quoted 2 lines
Both structures need better padding and some documentation.
Of course. Would update the patches.
Signature · 2 lines
regards Yang, Sheng
PATCH v2 1/3 KVM: Ioctls for init MSI-X entry
Introduce KVM_SET_MSIX_NR and KVM_SET_MSIX_ENTRY two ioctls.
This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
--- include/linux/kvm.h | 15 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+), 0 deletions(-)
include/linux/kvm.h +15 -0
@@ -475,6 +475,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
@@ -595,4 +597,17 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +__u16 padding; +}; + +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry;/* The index of entry in the MSI-X table */ +__u16 padding[3 ]; +}; +#endif
include/linux/kvm_host.h +3 -0
@@ -326,9 +326,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)
virt/kvm/kvm_main.c +101 -0
114 lines not on this page · raw message
Signature · 1 line
1.5.4.5
Re: [PATCH 1/3] KVM: Ioctls for init MSI-X entry
Quoted 11 lines · Sheng Yang
Introduce
KVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRYtwo ioctls.This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
Signed-off-by Sheng Yang <[email protected]>
virt/kvm/kvm_main.c static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) · line 1619
Quoted 5 lines
virt/kvm/kvm_main.c +4
-0+ adev->host_msix_entries =kzalloc (sizeof (struct msix_entry ) * + entry_nr->entry_nr, +GFP_KERNEL ); +if (!adev->host_msix_entries) {
Please verify its within a sane limit. Like the msr entry ioctl code. Also free host_msix_entries if guest_msix_entries allocation fails.
Otherwise seems OK.
PATCH v2 1/3 KVM: Ioctls for init MSI-X entry
Introduce KVM_SET_MSIX_NR and KVM_SET_MSIX_ENTRY two ioctls.
This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
--- include/linux/kvm.h | 16 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 0 deletions(-)
include/linux/kvm.h +16 -0
@@ -475,6 +475,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
@@ -595,4 +597,18 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +__u16 padding; +}; + +#define KVM_MAX_MSIX_PER_DEV 512 +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry;/* The index of entry in the MSI-X table */ +__u16 padding[3 ]; +}; +#endif
include/linux/kvm_host.h +3 -0
@@ -326,9 +326,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)
virt/kvm/kvm_main.c +103 -0
116 lines not on this page · raw message
Signature · 1 line
1.5.4.5
PATCH v2 1/3 KVM: Ioctls for init MSI-X entry
(oops... fix a typo)
Introduce KVM_SET_MSIX_NR and KVM_SET_MSIX_ENTRY two ioctls.
This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
--- include/linux/kvm.h | 16 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 0 deletions(-)
include/linux/kvm.h +16 -0
@@ -475,6 +475,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
@@ -595,4 +597,18 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +__u16 padding; +}; + +#define KVM_MAX_MSIX_PER_DEV 512 +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry;/* The index of entry in the MSI-X table */ +__u16 padding[3 ]; +}; +#endif
include/linux/kvm_host.h +3 -0
@@ -326,9 +326,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)
virt/kvm/kvm_main.c +103 -0
116 lines not on this page · raw message
Signature · 1 line
1.5.4.5
Re: [PATCH 1/3] KVM: Ioctls for init MSI-X entry
Quoted 83 lines · Sheng Yang
(oops... fix a typo)
Introduce
KVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRYtwo ioctls.This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
Signed-off-by Sheng Yang <[email protected]>--- include/linux/kvm.h | 16 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 0 deletions(-)include/linux/kvm.h +16
-0@@ -475,6 +475,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds @@ -595,4 +597,18 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +__u16 padding; +}; + +#define KVM_MAX_MSIX_PER_DEV 512 +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry;/* The index of entry in the MSI-X table */ +__u16 padding[3 ]; +}; +#endif include/linux/kvm_host.h +3
-0@@ -326,9 +326,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)virt/kvm/kvm_main.c +8
-0@@ -1593,6 +1593,87 @@ static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset)return 0 ; } +static int kvm_vm_ioctl_set_msix_nr (struct kvm *kvm, +struct kvm_assigned_msix_nr *entry_nr) +{ +int r =0 ; +struct kvm_assigned_dev_kernel *adev; + +mutex_lock (&kvm->lock); +
virt/kvm/kvm_main.c static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) · line 1613
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ adev =kvm_find_assigned_dev (&kvm->arch.assigned_dev_head, + entry_nr->assigned_dev_id); +if (!adev) { + r = -EINVAL ; + goto msix_nr_out; + } + +if (adev->entries_nr ==0 ) { + adev->entries_nr = entry_nr->entry_nr; + if (adev->entries_nr == 0 ||
<= 0 ?
virt/kvm/kvm_main.c static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) · line 1614
Quoted 1 line
virt/kvm/kvm_main.c +1
-0+ adev->entries_nr >= KVM_MAX_MSIX_PER_DEV)
Otherwise looks OK, thanks.
Re: [PATCH 1/3] KVM: Ioctls for init MSI-X entry
Quoted 91 lines · On Tuesday 17 February 2009 07:23:23 Marcelo Tosatti wrote:
Quoted 90 lines · Sheng Yang
(oops... fix a typo)
Introduce
KVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRYtwo ioctls.This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
Signed-off-by Sheng Yang <[email protected]>--- include/linux/kvm.h | 16 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 103++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 0 deletions(-)
include/linux/kvm.h +2
-0@@ -475,6 +475,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
include/linux/kvm.h +14
-0@@ -595,4 +597,18 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +__u16 padding; +}; + +#define KVM_MAX_MSIX_PER_DEV 512 +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry;/* The index of entry in the MSI-X table */ +__u16 padding[3 ]; +}; +#endif include/linux/kvm_host.h +3
-0@@ -326,9 +326,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)virt/kvm/kvm_main.c +12
-0@@ -1593,6 +1593,87 @@ static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) return 0; } +static int kvm_vm_ioctl_set_msix_nr (struct kvm *kvm, +struct kvm_assigned_msix_nr *entry_nr) +{ +int r =0 ; +struct kvm_assigned_dev_kernel *adev; + +mutex_lock (&kvm->lock); + + adev =kvm_find_assigned_dev (&kvm->arch.assigned_dev_head, + entry_nr->assigned_dev_id); +if (!adev) { + r = -EINVAL ;
Quoted 8 lines
virt/kvm/kvm_main.c static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) · line 1613
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+ goto msix_nr_out; + } + +if (adev->entries_nr ==0 ) { + adev->entries_nr = entry_nr->entry_nr; + if (adev->entries_nr == 0 ||<= 0 ?
Oh, yeah, that's right. :)
I would change the type of adev->entries_nr to unsigned int.
Signature · 2 lines
regards Yang, Sheng
Quoted 4 lines
virt/kvm/kvm_main.c static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) · line 1614
Quoted 2 lines
virt/kvm/kvm_main.c +1
-0+ adev->entries_nr >= KVM_MAX_MSIX_PER_DEV)Otherwise looks OK, thanks.
PATCH v2 1/3 KVM: Ioctls for init MSI-X entry
Introduce KVM_SET_MSIX_NR and KVM_SET_MSIX_ENTRY two ioctls.
This two ioctls are used by userspace to specific guest device MSI-X entry number and correlate MSI-X entry with GSI during the initialization stage.
MSI-X should be well initialzed before enabling.
Don't support change MSI-X entry number for now.
--- include/linux/kvm.h | 16 +++++++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 0 deletions(-)
include/linux/kvm.h +16 -0
@@ -475,6 +475,8 @@ struct kvm_irq_routing {#define KVM_ASSIGN_IRQ _IOR(KVMIO, 0x70, \ struct kvm_assigned_irq)#define KVM_REINJECT_CONTROL _IO(KVMIO, 0x71) +#define KVM_SET_MSIX_NR _IOR(KVMIO, 0x72, struct kvm_assigned_msix_nr) +#define KVM_SET_MSIX_ENTRY _IOR(KVMIO, 0x73, struct kvm_assigned_msix_entry)/* * ioctls for vcpu fds
@@ -595,4 +597,18 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +struct kvm_assigned_msix_nr { +__u32 assigned_dev_id; +__u16 entry_nr; +__u16 padding; +}; + +#define KVM_MAX_MSIX_PER_DEV 512 +struct kvm_assigned_msix_entry { +__u32 assigned_dev_id; +__u32 gsi; +__u16 entry;/* The index of entry in the MSI-X table */ +__u16 padding[3 ]; +}; +#endif
include/linux/kvm_host.h +3 -0
@@ -326,9 +326,12 @@ struct kvm_assigned_dev_kernel {int assigned_dev_id;int host_busnr;int host_devfn; +unsigned int entries_nr;int host_irq;bool host_irq_disabled; +struct msix_entry *host_msix_entries;int guest_irq; +struct msix_entry *guest_msix_entries;#define KVM_ASSIGNED_DEV_GUEST_INTX (1 << 0)#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)
virt/kvm/kvm_main.c +103 -0
116 lines not on this page · raw message
Signature · 1 line
1.5.4.5
PATCH v2 2/3 KVM: Add gsi_msg_pending_bitmap for MSI-X
In reply to Sheng YangWe have to handle more than one interrupt with one handler for MSI-X. So we need a bitmap to track the triggered interrupts.
--- include/linux/kvm_host.h | 5 +- virt/kvm/kvm_main.c | 103 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 103 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +3 -2
@@ -144,6 +144,8 @@ struct kvm {#ifdef CONFIG_HAVE_KVM_IRQCHIP struct list_head irq_routing;/* of kvm_kernel_irq_routing_entry */ struct hlist_head mask_notifier_list; +#define KVM_MAX_IRQ_ROUTES 1024 +DECLARE_BITMAP (irq_routes_pending_bitmap,KVM_MAX_IRQ_ROUTES );#endif #ifdef KVM_ARCH_WANT_MMU_NOTIFIER
@@ -335,6 +337,7 @@ struct kvm_assigned_dev_kernel {#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)#define KVM_ASSIGNED_DEV_HOST_MSI (1 << 9) +#define KVM_ASSIGNED_DEV_MSIX ((1 << 2) | (1 << 10))unsigned long irq_requested_type;int irq_source_id;int flags;
@@ -502,8 +505,6 @@ static inline int mmu_notifier_retry(struct kvm_vcpu *vcpu, unsigned long mmu_se#ifdef CONFIG_HAVE_KVM_IRQCHIP -#define KVM_MAX_IRQ_ROUTES 1024 -int kvm_setup_default_irq_routing (struct kvm *kvm); int kvm_set_irq_routing(struct kvm *kvm, const struct kvm_irq_routing_entry *entries,
virt/kvm/kvm_main.c +100 -3
131 lines not on this page · raw message
Signature · 1 line
1.5.4.5
Re: [PATCH 2/3] KVM: Add gsi_msg_pending_bitmap for MSI-X
Hi Sheng,
Quoted 11 lines · Sheng Yang
We have to handle more than one interrupt with one handler for MSI-X. So we need a bitmap to track the triggered interrupts.
Signed-off-by Sheng Yang <[email protected]>--- include/linux/kvm_host.h | 5 +- virt/kvm/kvm_main.c | 103 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 103 insertions(+), 5 deletions(-)
include/linux/kvm_host.h struct kvm_assigned_dev_kernel { · line 340
Quoted 4 lines
include/linux/kvm_host.h +1
-0@@ -335,6 +337,7 @@ struct kvm_assigned_dev_kernel {#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)#define KVM_ASSIGNED_DEV_HOST_MSI (1 << 9) +#define KVM_ASSIGNED_DEV_MSIX ((1 << 2) | (1 << 10))
Can you explain the usage of the two bits?
Quoted 27 lines
virt/kvm/kvm_main.c +19
-0@@ -95,25 +95,113 @@ static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *hreturn NULL ; } +static int find_host_irq_from_gsi (struct kvm_assigned_dev_kernel *assigned_dev, +u32 gsi) +{ +int i, entry, irq; +struct msix_entry *host_msix_entries, *guest_msix_entries; + + host_msix_entries = assigned_dev->host_msix_entries; + guest_msix_entries = assigned_dev->guest_msix_entries; + + entry =-1 ; + irq =0 ; +for (i =0 ; i < assigned_dev->entries_nr; i++) +if (gsi == (guest_msix_entries + i)->vector) { + entry = (guest_msix_entries + i)->entry; +break ; + } +if (entry <0 ) { +printk (KERN_WARNING "Fail to find correlated MSI-X entry!\n" ); +return 0 ;
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 126
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ } +for (i =0 ; i < assigned_dev->entries_nr; i++) +if (entry == (host_msix_entries + i)->entry) { + irq = (host_msix_entries + i)->vector; +break ; + } +if (irq ==0 ) { +printk (KERN_WARNING "Fail to find correlated MSI-X irq!\n" ); +return 0 ; + }
Can drop the printk's (also from find_gsi_from_host_irq).
Quoted 23 lines
virt/kvm/kvm_main.c +11
-3struct kvm_assigned_dev_kernel *assigned_dev; +struct kvm *kvm; +u32 gsi; +int irq; assigned_dev = container_of(work, struct kvm_assigned_dev_kernel, interrupt_work); + kvm = assigned_dev->kvm;/* This is taken to safely inject irq inside the guest. When * the interrupt injection (or the ioapic code) uses a * finer-grained lock, update this */ -mutex_lock (&assigned_dev->kvm->lock); -kvm_set_irq (assigned_dev->kvm, assigned_dev->irq_source_id, - assigned_dev->guest_irq,1 ); +mutex_lock (&kvm->lock); +handle_irq: +if (assigned_dev->irq_requested_type &KVM_ASSIGNED_DEV_MSIX ) { + gsi =find_first_bit (kvm->irq_routes_pending_bitmap, +KVM_MAX_IRQ_ROUTES ); +BUG_ON (gsi >=KVM_MAX_IRQ_ROUTES ); +clear_bit (gsi, kvm->irq_routes_pending_bitmap);
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 196
Quoted 10 lines
virt/kvm/kvm_main.c +6
-0+ }else + gsi = assigned_dev->guest_irq; + +kvm_set_irq (assigned_dev->kvm, assigned_dev->irq_source_id, gsi,1 );if (assigned_dev->irq_requested_type &KVM_ASSIGNED_DEV_GUEST_MSI ) {enable_irq (assigned_dev->host_irq); assigned_dev->host_irq_disabled = false; + }else if (assigned_dev->irq_requested_type &KVM_ASSIGNED_DEV_MSIX ) { + irq =find_host_irq_from_gsi (assigned_dev, gsi);
Check the return value?
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 197
Quoted 1 line
virt/kvm/kvm_main.c +1
-0+enable_irq (irq);
Do you guarantee that particular irq you're enable_irq'ing is not bogus? Its has been passed from userspace after all.
In a later patch you can assign KVM_ASSIGNED_DEV_MSIX if the irqchip is not in-kernel in assigned_device_update_msix:
+ adev->irq_requested_type |= KVM_ASSIGNED_DEV_MSIX;
+
Just trying to harden the code against bogosity elsewhere.
Quoted 11 lines
virt/kvm/kvm_main.c +6
-0+ assigned_dev->host_irq_disabled = false; + gsi =find_first_bit (kvm->irq_routes_pending_bitmap, +KVM_MAX_IRQ_ROUTES ); +if (gsi <KVM_MAX_IRQ_ROUTES ) + goto handle_irq; } +mutex_unlock (&assigned_dev->kvm->lock); }
virt/kvm/kvm_main.c static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) · line 218
Quoted 10 lines
virt/kvm/kvm_main.c +7
-0@@ -121,6 +209,15 @@ static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) {struct kvm_assigned_dev_kernel *assigned_dev = (struct kvm_assigned_dev_kernel *) dev_id; +struct kvm *kvm = assigned_dev->kvm; + +if (assigned_dev->irq_requested_type ==KVM_ASSIGNED_DEV_MSIX ) { +u32 gsi; + gsi =find_gsi_from_host_irq (assigned_dev, irq); +if (gsi ==0 ) +return IRQ_HANDLED ;
So you chose GSI == 0 as invalid because of x86 assumptions? Or is there any other reason?
IRQ sharing in the host side is not supported correct?
Re: [PATCH 2/3] KVM: Add gsi_msg_pending_bitmap for MSI-X
Quoted 14 lines · On Friday 13 February 2009 03:51:18 Marcelo Tosatti wrote:
Hi Sheng,
Quoted 11 lines · Sheng Yang
We have to handle more than one interrupt with one handler for MSI-X. So we need a bitmap to track the triggered interrupts.
Signed-off-by Sheng Yang <[email protected]>--- include/linux/kvm_host.h | 5 +- virt/kvm/kvm_main.c | 103++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 103 insertions(+), 5 deletions(-)
Quoted 7 lines
include/linux/kvm_host.h struct kvm_assigned_dev_kernel { · line 340
Quoted 5 lines
include/linux/kvm_host.h +1
-0@@ -335,6 +337,7 @@ struct kvm_assigned_dev_kernel {#define KVM_ASSIGNED_DEV_GUEST_MSI (1 << 1)#define KVM_ASSIGNED_DEV_HOST_INTX (1 << 8)#define KVM_ASSIGNED_DEV_HOST_MSI (1 << 9) +#define KVM_ASSIGNED_DEV_MSIX ((1 << 2) | (1 << 10))Can you explain the usage of the two bits?
Um... Just to keep consistent with formers(one for guest and one for host), at cost of one bit.
Quoted 31 lines
Quoted 31 lines
virt/kvm/kvm_main.c +22
-0@@ -95,25 +95,113 @@ static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h return NULL; } +static int find_host_irq_from_gsi (struct kvm_assigned_dev_kernel *assigned_dev, + u32 gsi) +{ +int i, entry, irq; +struct msix_entry *host_msix_entries, *guest_msix_entries; + + host_msix_entries = assigned_dev->host_msix_entries; + guest_msix_entries = assigned_dev->guest_msix_entries; + + entry =-1 ; + irq =0 ; +for (i =0 ; i < assigned_dev->entries_nr; i++) +if (gsi == (guest_msix_entries + i)->vector) { + entry = (guest_msix_entries + i)->entry; +break ; + } +if (entry <0 ) { +printk (KERN_WARNING "Fail to find correlated MSI-X entry!\n" ); +return 0 ; + } +for (i =0 ; i < assigned_dev->entries_nr; i++) +if (entry == (host_msix_entries + i)->entry) { + irq = (host_msix_entries + i)->vector;
Quoted 8 lines
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 126
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+break ; + } +if (irq ==0 ) { +printk (KERN_WARNING "Fail to find correlated MSI-X irq!\n" ); +return 0 ; + }Can drop the printk's (also from
find_gsi_from_host_irq).
Not that confident. In fact, I often triggered this during debug... IIRC, userspace program shouldn't trigger this if kernel space works well. Maybe it can be changed to WARN_ON() or BUG_ON() later.
Quoted 29 lines
Quoted 29 lines
virt/kvm/kvm_main.c +15
-3struct kvm_assigned_dev_kernel *assigned_dev; +struct kvm *kvm; +u32 gsi; +int irq; assigned_dev = container_of(work, struct kvm_assigned_dev_kernel, interrupt_work); + kvm = assigned_dev->kvm;/* This is taken to safely inject irq inside the guest. When * the interrupt injection (or the ioapic code) uses a * finer-grained lock, update this */ -mutex_lock (&assigned_dev->kvm->lock); -kvm_set_irq (assigned_dev->kvm, assigned_dev->irq_source_id, - assigned_dev->guest_irq,1 ); +mutex_lock (&kvm->lock); +handle_irq: +if (assigned_dev->irq_requested_type &KVM_ASSIGNED_DEV_MSIX ) { + gsi =find_first_bit (kvm->irq_routes_pending_bitmap, +KVM_MAX_IRQ_ROUTES ); +BUG_ON (gsi >=KVM_MAX_IRQ_ROUTES ); +clear_bit (gsi, kvm->irq_routes_pending_bitmap); + }else + gsi = assigned_dev->guest_irq; + +kvm_set_irq (assigned_dev->kvm, assigned_dev->irq_source_id, gsi,1 );
Quoted 7 lines
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 196
Quoted 5 lines
virt/kvm/kvm_main.c +2
-0if (assigned_dev->irq_requested_type &KVM_ASSIGNED_DEV_GUEST_MSI ) {enable_irq (assigned_dev->host_irq); assigned_dev->host_irq_disabled = false; + }else if (assigned_dev->irq_requested_type &KVM_ASSIGNED_DEV_MSIX ) { + irq =find_host_irq_from_gsi (assigned_dev, gsi);Check the return value?
Yeah...
Quoted 5 lines
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 197
Quoted 2 lines
virt/kvm/kvm_main.c +1
-0+enable_irq (irq);Do you guarantee that particular irq you're enable_irq'ing is not bogus? Its has been passed from userspace after all.
It isn't passed from userspace. This one is filled by pci_enable_msix(), which should be OK.
Quoted 8 lines
In a later patch you can assign
KVM_ASSIGNED_DEV_MSIXif the irqchip is not in-kernel inassigned_device_update_msix:(quoted diff) +2
-0+ adev->irq_requested_type |= KVM_ASSIGNED_DEV_MSIX; +Just trying to harden the code against bogosity elsewhere.
Yeah, of course. :)
Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c +6
-0+ assigned_dev->host_irq_disabled = false; + gsi =find_first_bit (kvm->irq_routes_pending_bitmap, +KVM_MAX_IRQ_ROUTES ); +if (gsi <KVM_MAX_IRQ_ROUTES ) + goto handle_irq; } +mutex_unlock (&assigned_dev->kvm->lock); }
Quoted 14 lines
virt/kvm/kvm_main.c static irqreturn_t kvm_assigned_dev_intr(int irq, · line 218
Quoted 11 lines
virt/kvm/kvm_main.c +7
-0@@ -121,6 +209,15 @@ static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) {struct kvm_assigned_dev_kernel *assigned_dev = (struct kvm_assigned_dev_kernel *) dev_id; +struct kvm *kvm = assigned_dev->kvm; + +if (assigned_dev->irq_requested_type ==KVM_ASSIGNED_DEV_MSIX ) { +u32 gsi; + gsi =find_gsi_from_host_irq (assigned_dev, irq); +if (gsi ==0 ) +return IRQ_HANDLED ;So you chose GSI == 0 as invalid because of x86 assumptions? Or is there any other reason?
Yeah, it based on x86 and IA64 IRQ 0 can't be used by MSI-X. And only x86 support MSI-X now(and IA64 would follow later).
Quoted 2 lines
IRQ sharing in the host side is not supported correct?
Um? Yeah... And seems we won't support it forever...
Signature · 2 lines
regards Yang, Sheng
Re: [PATCH 2/3] KVM: Add gsi_msg_pending_bitmap for MSI-X
Quoted 6 lines · Sheng Yang
Quoted 3 lines
Quoted 1 line
+#define
KVM_ASSIGNED_DEV_MSIX((1 << 2) | (1 << 10))Can you explain the usage of the two bits?
Um... Just to keep consistent with formers(one for guest and one for host), at cost of one bit.
OK
Quoted 5 lines
Quoted 1 line
Can drop the printk's (also from
find_gsi_from_host_irq).Not that confident. In fact, I often triggered this during debug... IIRC, userspace program shouldn't trigger this if kernel space works well. Maybe it can be changed to
WARN_ON()orBUG_ON()later.
OK
Quoted 11 lines
Quoted 1 line
Check the return value?
Yeah...
Quoted 5 lines
virt/kvm/kvm_main.c static struct kvm_assigned_dev_kernel *kvm_find_assigned_dev(struct list_head *h · line 197
Quoted 2 lines
virt/kvm/kvm_main.c +1
-0+enable_irq (irq);Do you guarantee that particular irq you're enable_irq'ing is not bogus? Its has been passed from userspace after all.
It isn't passed from userspace. This one is filled by
pci_enable_msix(), which should be OK.
Alright.
Quoted 5 lines
Quoted 2 lines
So you chose GSI == 0 as invalid because of x86 assumptions? Or is there any other reason?
Yeah, it based on x86 and IA64 IRQ 0 can't be used by MSI-X. And only x86 support MSI-X now(and IA64 would follow later).
OK.
Quoted 4 lines
Quoted 2 lines
IRQ sharing in the host side is not supported correct?
Um? Yeah... And seems we won't support it forever...
OK.
Re: [PATCH 2/3] KVM: Add gsi_msg_pending_bitmap for MSI-X
In reply to Sheng YangQuoted 10 lines · Sheng Yang
Quoted 9 lines
virt/kvm/kvm_main.c static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) · line 218
Quoted 6 lines
virt/kvm/kvm_main.c +5
-0+if (assigned_dev->irq_requested_type ==KVM_ASSIGNED_DEV_MSIX ) { +u32 gsi; + gsi =find_gsi_from_host_irq (assigned_dev, irq); +if (gsi ==0 ) +return IRQ_HANDLED ;So you chose GSI == 0 as invalid because of x86 assumptions? Or is there any other reason?
Quoted 2 lines
Yeah, it based on x86 and IA64 IRQ 0 can't be used by MSI-X. And only x86 support MSI-X now(and IA64 would follow later).
gsi != irq... as used in kvm it is just a cookie and is never visible to the guest. I'd prefer an illegal value here.
Also, please repost the entire patchset so I can be sure I apply the latest; too many versions floating around.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [PATCH 2/3] KVM: Add gsi_msg_pending_bitmap for MSI-X
Quoted 13 lines · On Wednesday 18 February 2009 02:09:49 Avi Kivity wrote:
Quoted 11 lines · Sheng Yang
Quoted 8 lines
virt/kvm/kvm_main.c static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) · line 218
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+if (assigned_dev->irq_requested_type ==KVM_ASSIGNED_DEV_MSIX ) { +u32 gsi; + gsi =find_gsi_from_host_irq (assigned_dev, irq); +if (gsi ==0 ) +return IRQ_HANDLED ;So you chose GSI == 0 as invalid because of x86 assumptions? Or is there any other reason?
Yeah, it based on x86 and IA64 IRQ 0 can't be used by MSI-X. And only x86 support MSI-X now(and IA64 would follow later).
Quoted 6 lines
gsi != irq... as used in kvm it is just a cookie and is never visible to the guest. I'd prefer an illegal value here.
Also, please repost the entire patchset so I can be sure I apply the latest; too many versions floating around.
OK. And please don't forget there is a patchset named "Optimize and unify IOAPIC/MSI delivery" before this one. :)
Signature · 2 lines
regards Yang, Sheng
PATCH v2 3/3 KVM: Enable MSI-X for KVM assigned device
In reply to Sheng YangThis patch finally enable MSI-X.
What we need for MSI-X:
1. Intercept one page in MMIO region of device. So that we can get guest desired MSI-X table and set up the real one. Now this have been done by guest, and transfer to kernel using ioctl KVM_SET_MSIX_NR and KVM_SET_MSIX_ENTRY.
2. Information for incoming interrupt. Now one device can have more than one interrupt, and they are all handled by one workqueue structure. So we need to identify them. The previous patch enable gsi_msg_pending_bitmap get this done.
3. Mapping from host IRQ to guest gsi as well as guest gsi to real MSI/MSI-X message address/data. We used same entry number for the host and guest here, so that it's easy to find the correlated guest gsi.
What we lack for now:
1. The PCI spec said nothing can existed with MSI-X table in the same page of MMIO region, except pending bits. The patch ignore pending bits as the first step (so they are always 0 - no pending).
2. The PCI spec allowed to change MSI-X table dynamically. That means, the OS can enable MSI-X, then mask one MSI-X entry, modify it, and unmask it. The patch didn't support this, and Linux also don't work in this way.
3. The patch didn't implement MSI-X mask all and mask single entry. I would implement the former in driver/pci/msi.c later. And for single entry, userspace should have reposibility to handle it.
--- include/linux/kvm.h | 8 ++++ virt/kvm/kvm_main.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 109 insertions(+), 6 deletions(-)
include/linux/kvm.h +8 -0
@@ -439,6 +439,9 @@ struct kvm_irq_routing { };#endif +#if defined(CONFIG_X86 ) +#define KVM_CAP_DEVICE_MSIX 26 +#endif /* * ioctls for VM fds
@@ -596,6 +599,11 @@ struct kvm_assigned_irq {#define KVM_DEV_IRQ_ASSIGN_MSI_ACTION KVM_DEV_IRQ_ASSIGN_ENABLE_MSI#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSI (1 << 0) +#define KVM_DEV_IRQ_ASSIGN_MSIX_ACTION (KVM_DEV_IRQ_ASSIGN_ENABLE_MSIX |\ + KVM_DEV_IRQ_ASSIGN_MASK_MSIX) +#define KVM_DEV_IRQ_ASSIGN_ENABLE_MSIX (1 << 1) +#define KVM_DEV_IRQ_ASSIGN_MASK_MSIX (1 << 2) +struct kvm_assigned_msix_nr {__u32 assigned_dev_id;__u16 entry_nr;
virt/kvm/kvm_main.c +101 -6
132 lines not on this page · raw message
Signature · 1 line
1.5.4.5
Re: [PATCH 3/3] KVM: Enable MSI-X for KVM assigned device
Quoted 8 lines · Sheng Yang
This patch finally enable MSI-X.
What we need for MSI-X:
1. Intercept one page in MMIO region of device. So that we can get guest desired MSI-X table and set up the real one. Now this have been done by guest, and transfer to kernel using ioctlKVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRY.
Looking at the patch, you're doing that in userspace. Good.
Quoted 4 lines
2. The PCI spec allowed to change MSI-X table dynamically. That means, the OS can enable MSI-X, then mask one MSI-X entry, modify it, and unmask it. The patch didn't support this, and Linux also don't work in this way.
What about Windows?
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 3/3] KVM: Enable MSI-X for KVM assigned device
Quoted 9 lines · On Wednesday 11 February 2009 20:48:55 Avi Kivity wrote:
Quoted 7 lines · Sheng Yang
This patch finally enable MSI-X.
What we need for MSI-X:
1. Intercept one page in MMIO region of device. So that we can get guest desired MSI-X table and set up the real one. Now this have been done by guest, and transfer to kernel using ioctlKVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRY.
Quoted 7 lines
Looking at the patch, you're doing that in userspace. Good.
Quoted 3 lines
2. The PCI spec allowed to change MSI-X table dynamically. That means, the OS can enable MSI-X, then mask one MSI-X entry, modify it, and unmask it. The patch didn't support this, and Linux also don't work in this way.
What about Windows?
Not sure. Don't have Windows driver to test... And only Vista and later support MSI/MSI-X, not sure it would be so aggressive... We can add some debug if such condition happened.
Signature · 2 lines
regards Yang, Sheng
Re: [PATCH 3/3] KVM: Enable MSI-X for KVM assigned device
In reply to Sheng YangQuoted 15 lines · Sheng Yang
This patch finally enable MSI-X.
What we need for MSI-X:
1. Intercept one page in MMIO region of device. So that we can get guest desired MSI-X table and set up the real one. Now this have been done by guest, and transfer to kernel using ioctlKVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRY.2. Information for incoming interrupt. Now one device can have more than one interrupt, and they are all handled by one workqueue structure. So we need to identify them. The previous patch enable
gsi_msg_pending_bitmapget this done.3. Mapping from host IRQ to guest gsi as well as guest gsi to real MSI/MSI-X message address/data. We used same entry number for the host and guest here, so that it's easy to find the correlated guest gsi.
Quoted 8 lines
What we lack for now:
1. The PCI spec said nothing can existed with MSI-X table in the same page of MMIO region, except pending bits. The patch ignore pending bits as the first step (so they are always 0 - no pending).2. The PCI spec allowed to change MSI-X table dynamically. That means, the OS can enable MSI-X, then mask one MSI-X entry, modify it, and unmask it. The patch didn't support this, and Linux also don't work in this way.
Have you thought about supporting this with the current ioctl interface? Point is that once ioctl's are in and userspace code uses it, you can't change. So:
- If userspace calls kvm_vm_ioctl_set_msix_entry twice don't you leak memory (that is a bug actually unless I'm mistaken)? So in the future kvm_vm_ioctl_set_msix_entry will deallocate the currently allocated guest/host entries and replace with userspace supplied entries?
- interrupt context can read the table while kvm_vm_ioctl_set_msix_entry is modifying it. So you either need to forbid more than one kvm_vm_ioctl_set_msix_entry call in the lifetime of a guest (which you can later allow when you support MSI table change), or handle accesses from multiple contexes now. It seems forbidding is enough for the moment from what you said.
But in general looks good to me (not familiar with the internals of pci / msi).
Re: [PATCH 3/3] KVM: Enable MSI-X for KVM assigned device
Quoted 27 lines · On Friday 13 February 2009 04:40:05 Marcelo Tosatti wrote:
Quoted 17 lines · Sheng Yang
This patch finally enable MSI-X.
What we need for MSI-X:
1. Intercept one page in MMIO region of device. So that we can get guest desired MSI-X table and set up the real one. Now this have been done by guest, and transfer to kernel using ioctlKVM_SET_MSIX_NRandKVM_SET_MSIX_ENTRY.2. Information for incoming interrupt. Now one device can have more than one interrupt, and they are all handled by one workqueue structure. So we need to identify them. The previous patch enable
gsi_msg_pending_bitmapget this done.3. Mapping from host IRQ to guest gsi as well as guest gsi to real MSI/MSI-X message address/data. We used same entry number for the host and guest here, so that it's easy to find the correlated guest gsi.
Quoted 8 lines
What we lack for now:
1. The PCI spec said nothing can existed with MSI-X table in the same page of MMIO region, except pending bits. The patch ignore pending bits as the first step (so they are always 0 - no pending).2. The PCI spec allowed to change MSI-X table dynamically. That means, the OS can enable MSI-X, then mask one MSI-X entry, modify it, and unmask it. The patch didn't support this, and Linux also don't work in this way.
Quoted 9 lines
Have you thought about supporting this with the current ioctl interface? Point is that once ioctl's are in and userspace code uses it, you can't change. So:
- If userspace calls
kvm_vm_ioctl_set_msix_entrytwice don't you leak memory (that is a bug actually unless I'm mistaken)? So in the futurekvm_vm_ioctl_set_msix_entrywill deallocate the currently allocated guest/host entries and replace with userspace supplied entries?
The allocation only happen once, at the second time it would report error in current code. But allocate/deallocate is also acceptable for future.
Quoted 6 lines
- interrupt context can read the table while
kvm_vm_ioctl_set_msix_entryis modifying it. So you either need to forbid more than onekvm_vm_ioctl_set_msix_entrycall in the lifetime of a guest (which you can later allow when you support MSI table change), or handle accesses from multiple contexes now. It seems forbidding is enough for the moment from what you said.
Yeah.
But for the modifying the MSI-X table, the most critical problem is, current Linux didn't support it IIRC. So I have to disable MSI-X then enable it again with new table, and it would result in lost interrupt.
So seems the most reasonable method is to modify pci_enable_msix() and related function's action to support this...
Signature · 2 lines
regards Yang, Sheng
Quoted 3 lines
But in general looks good to me (not familiar with the internals of pci / msi).
Re: [PATCH 3/3] KVM: Enable MSI-X for KVM assigned device
Quoted 2 lines · Sheng Yang
The allocation only happen once, at the second time it would report error in current code. But allocate/deallocate is also acceptable for future.
Oops, it OK.
Quoted 7 lines
Quoted 6 lines
- interrupt context can read the table while
kvm_vm_ioctl_set_msix_entryis modifying it. So you either need to forbid more than onekvm_vm_ioctl_set_msix_entrycall in the lifetime of a guest (which you can later allow when you support MSI table change), or handle accesses from multiple contexes now. It seems forbidding is enough for the moment from what you said.
Quoted 8 lines
Yeah.
But for the modifying the MSI-X table, the most critical problem is, current Linux didn't support it IIRC. So I have to disable MSI-X then enable it again with new table, and it would result in lost interrupt.
So seems the most reasonable method is to modify
pci_enable_msix()and related function's action to support this...
Alright, then just reply "[PATCH 1/3] KVM: Ioctls for init MSI-X entry" with the minor comments reworked so Avi can apply.
Thanks.