Skip to contentSkip to conversation

Add VT-d Posted-Interrupts support

PATCH Superseded Feng Wu 2 participants 6 messages Feb 19, 2016 – Feb 23, 2016

v12 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Feng Wu

author
#0 lore raw

v12 cover Add VT-d Posted-Interrupts support

VT-d Posted-Interrupts is an enhancement to CPU side Posted-Interrupt. With VT-d Posted-Interrupts enabled, external interrupts from direct-assigned devices can be delivered to guests without VMM intervention when guest is running in non-root mode.

You can find the VT-d Posted-Interrtups Spec. in the following URL:
http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/vt-directed-io-spec.html

Feng Wu (2):
vmx: VT-d posted-interrupt core logic handling Add a command line parameter for VT-d posted-interrupts

 docs/misc/xen-command-line.markdown |   9 +-
 xen/arch/x86/hvm/vmx/vmcs.c         |   2 +
 xen/arch/x86/hvm/vmx/vmx.c          | 185 ++++++++++++++++++++++++++++++++++++
 xen/common/schedule.c               |   4 +
 xen/drivers/passthrough/iommu.c     |   3 +
 xen/drivers/passthrough/vtd/iommu.c |   7 ++
 xen/include/asm-arm/domain.h        |   2 +
 xen/include/asm-x86/hvm/hvm.h       |   5 +
 xen/include/asm-x86/hvm/vmx/vmcs.h  |  67 +++++++++++++
 xen/include/asm-x86/hvm/vmx/vmx.h   |   5 +
 10 files changed, 288 insertions(+), 1 deletion(-)
Signature · 1 line
2.1.0
author
#1 lore raw

PATCH v12 1/2 vmx: VT-d posted-interrupt core logic handling

This is the core logic handling for VT-d posted-interrupts. Basically it deals with how and when to update posted-interrupts during the following scenarios:
- vCPU is preempted
- vCPU is slept
- vCPU is blocked

When vCPU is preempted/slept, we update the posted-interrupts during scheduling by introducing two new architecutral scheduler hooks:
vmx_pi_switch_from() and vmx_pi_switch_to(). When vCPU is blocked, we introduce a new architectural hook: arch_vcpu_block() to update posted-interrupts descriptor.

Besides that, before VM-entry, we will make sure the 'NV' filed is set to 'posted_intr_vector' and the vCPU is not in any blocking lists, which is needed when vCPU is running in non-root mode. The reason we do this check is because we change the posted-interrupts descriptor in vcpu_block(), however, we don't change it back in vcpu_unblock() or when vcpu_block() directly returns due to event delivery (in fact, we don't need to do it in the two places, that is why we do it before VM-Entry).

When we handle the lazy context switch for the following two scenarios:
- Preempted by a tasklet, which uses in an idle context.
- the prev vcpu is in offline and no new available vcpus in run queue. We don't change the 'SN' bit in posted-interrupt descriptor, this may incur spurious PI notification events, but since PI notification event is only sent when 'ON' is clear, and once the PI notificatoin is sent, ON is set by hardware, hence no more notification events before 'ON' is clear. Besides that, spurious PI notification events are going to happen from time to time in Xen hypervisor, such as, when guests trap to Xen and PI notification event happens, there is nothing Xen actually needs to do about it, the interrupts will be delivered to guest atht the next time we do a VMENTRY.

---
v12:
- Move the ASSERT to the locked region in vmx_vcpu_block()
- Add barrier() before using the local variable in vmx_pi_do_resume()
- Split vmx_pi_hooks_reassign() to two functions:
* vmx_pi_hooks_assign()
* vmx_pi_hooks_deassign()
- Add more comments about how PI works during vCPU state transition
- coding style

v11:
- Add ASSERT() in vmx_vcpu_block()
- Add some comments in vmx_pi_switch_from()
- Remove some comments which should have been removed when the related code was removed during v9 -> v10
- Rename 'vmx_pi_state_to_normal' to 'vmx_pi_do_resume'
- Coding style
- Make arch_vcpu_block() a macro
- Make 'pi_wakeup_vector' static
- Move hook 'vcpu_block' to 'struct hvm_vcpu'
- Initial hook 'vcpu_block' when assigning the first pci device and zap it on removal of the last device
- Save pointer to the block list lock instead of the processor id in 'struct arch_vmx_struct'
- Implement the following functions as hooks, so we can elimilate lots of checkings and spinlocks in scheduling related code path, which is good for performance. vmx_pi_switch_from
vmx_pi_switch_to
vmx_pi_do_resume

v10:
- Check iommu_intpost first
- Remove pointless checking of has_hvm_container_vcpu(v)
- Rename 'vmx_pi_state_change' to 'vmx_pi_state_to_normal'
- Since vcpu_unblock() doesn't acquire 'pi_blocked_vcpu_lock', we don't need use another list to save the vCPUs with 'ON' set, just directly call vcpu_unblock(v).

v9:
- Remove arch_vcpu_block_cancel() and arch_vcpu_wake_prepare()
- Add vmx_pi_state_change() and call it before VM Entry

v8:
- Remove the lazy context switch handling for PI state transition
- Change PI state in vcpu_block() and do_poll() when the vCPU is going to be blocked

v7:
- Merge [PATCH v6 16/18] vmx: Add some scheduler hooks for VT-d posted interrupts and "[PATCH v6 14/18] vmx: posted-interrupt handling when vCPU is blocked" into this patch, so it is self-contained and more convenient
for code review.
- Make 'pi_blocked_vcpu' and 'pi_blocked_vcpu_lock' static
- Coding style
- Use per_cpu() instead of this_cpu() in pi_wakeup_interrupt()
- Move ack_APIC_irq() to the beginning of pi_wakeup_interrupt()
- Rename 'pi_ctxt_switch_from' to 'ctxt_switch_prepare'
- Rename 'pi_ctxt_switch_to' to 'ctxt_switch_cancel'
- Use 'has_hvm_container_vcpu' instead of 'is_hvm_vcpu'
- Use 'spin_lock' and 'spin_unlock' when the interrupt has been already disabled.
- Rename arch_vcpu_wake_prepare to vmx_vcpu_wake_prepare
- Define vmx_vcpu_wake_prepare in xen/arch/x86/hvm/hvm.c
- Call .pi_ctxt_switch_to() __context_switch() instead of directly calling vmx_post_ctx_switch_pi() in vmx_ctxt_switch_to()
- Make .pi_block_cpu unsigned int
- Use list_del() instead of list_del_init()
- Coding style

One remaining item in v7:
Jan has concern about calling vcpu_unblock() in vmx_pre_ctx_switch_pi(), need Dario or George's input about this.

v6:
- Add two static inline functions for pi context switch
- Fix typos

v5:
- Rename arch_vcpu_wake to arch_vcpu_wake_prepare
- Make arch_vcpu_wake_prepare() inline for ARM
- Merge the ARM dummy hook with together
- Changes to some code comments
- Leave 'pi_ctxt_switch_from' and 'pi_ctxt_switch_to' NULL if PI is disabled or the vCPU is not in HVM
- Coding style

v4:
- Newly added

Changlog for "vmx: posted-interrupt handling when vCPU is blocked" v6:
- Fix some typos
- Ack the interrupt right after the spin_unlock in pi_wakeup_interrupt()

v4:
- Use local variables in pi_wakeup_interrupt()
- Remove vcpu from the blocked list when pi_desc.on==1, this
- avoid kick vcpu multiple times.
- Remove tasklet

v3:
- This patch is generated by merging the following three patches in v2:
[RFC v2 09/15] Add a new per-vCPU tasklet to wakeup the blocked vCPU [RFC v2 10/15] vmx: Define two per-cpu variables
[RFC v2 11/15] vmx: Add a global wake-up vector for VT-d Posted-Interrupts
- rename 'vcpu_wakeup_tasklet' to 'pi_vcpu_wakeup_tasklet'
- Move the definition of 'pi_vcpu_wakeup_tasklet' to 'struct arch_vmx_struct'
- rename 'vcpu_wakeup_tasklet_handler' to 'pi_vcpu_wakeup_tasklet_handler'
- Make pi_wakeup_interrupt() static
- Rename 'blocked_vcpu_list' to 'pi_blocked_vcpu_list'
- move 'pi_blocked_vcpu_list' to 'struct arch_vmx_struct'
- Rename 'blocked_vcpu' to 'pi_blocked_vcpu'
- Rename 'blocked_vcpu_lock' to 'pi_blocked_vcpu_lock'

 xen/arch/x86/hvm/vmx/vmcs.c         |   2 +
 xen/arch/x86/hvm/vmx/vmx.c          | 185 ++++++++++++++++++++++++++++++++++++
 xen/common/schedule.c               |   4 +
 xen/drivers/passthrough/vtd/iommu.c |   7 ++
 xen/include/asm-arm/domain.h        |   2 +
 xen/include/asm-x86/hvm/hvm.h       |   5 +
 xen/include/asm-x86/hvm/vmx/vmcs.h  |  67 +++++++++++++
 xen/include/asm-x86/hvm/vmx/vmx.h   |   5 +
 8 files changed, 277 insertions(+)
xen/arch/x86/hvm/vmx/vmcs.c +2 -0
index edd4c8d..2e535de 100644
@@ -676,6 +676,8 @@ int vmx_cpu_up(void)
     if ( cpu_has_vmx_vpid )
         vpid_sync_all();
 
+    vmx_pi_per_cpu_init(cpu);
+
     return 0;
 }
 
xen/arch/x86/hvm/vmx/vmx.c +185 -0

229 lines not on this page · raw message

xen/common/schedule.c +4 -0
index d121896..2d87021 100644
@@ -802,6 +802,8 @@ void vcpu_block(void)
 
     set_bit(_VPF_blocked, &v->pause_flags);
 
+    arch_vcpu_block(v);
+
     /* Check for events /after/ blocking: avoids wakeup waiting race. */
     if ( local_events_need_delivery() )
     {
@@ -839,6 +841,8 @@ static long do_poll(struct sched_poll *sched_poll)
     v->poll_evtchn = -1;
     set_bit(v->vcpu_id, d->poll_mask);
 
+    arch_vcpu_block(v);
+
 #ifndef CONFIG_X86 /* set_bit() implies mb() on x86 */
     /* Check for events /after/ setting flags: avoids wakeup waiting race. */
     smp_mb();
xen/drivers/passthrough/vtd/iommu.c +7 -0
index ec31c6b..6b6e704 100644
@@ -2283,9 +2283,14 @@ static int reassign_device_ownership(
     if ( ret )
         return ret;
 
+    vmx_pi_hooks_assign(target);
+
     ret = domain_context_mapping(target, devfn, pdev);
     if ( ret )
+    {
+        vmx_pi_hooks_deassign(target);
         return ret;
+    }
 
     if ( devfn == pdev->devfn )
     {
@@ -2293,6 +2298,8 @@ static int reassign_device_ownership(
         pdev->domain = target;
     }
 
+    vmx_pi_hooks_deassign(source);
+
     return ret;
 }
 
xen/include/asm-arm/domain.h +2 -0
index aa7f283..37afa80 100644
@@ -310,6 +310,8 @@ static inline void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
     xfree(vgc);
 }
 
+static inline void arch_vcpu_block(struct vcpu *v) {}
+
 #endif /* __ASM_DOMAIN_H__ */
 
 /*
xen/include/asm-x86/hvm/hvm.h +5 -0
index b9d893d..4590456 100644
@@ -565,6 +565,11 @@ const char *hvm_efer_valid(const struct vcpu *v, uint64_t value,
                            signed int cr0_pg);
 unsigned long hvm_cr4_guest_reserved_bits(const struct vcpu *v, bool_t restore);
 
+#define arch_vcpu_block(v) ({                                               \
+    if ( (v)->domain->arch.hvm_domain.vmx.vcpu_block )                      \
+        (v)->domain->arch.hvm_domain.vmx.vcpu_block((v));                   \
+})
+
 #endif /* __ASM_X86_HVM_HVM_H__ */
 
 /*
xen/include/asm-x86/hvm/vmx/vmcs.h +67 -0

79 lines not on this page · raw message

xen/include/asm-x86/hvm/vmx/vmx.h +5 -0
index 1719965..90b5ca7 100644
@@ -563,6 +563,11 @@ int alloc_p2m_hap_data(struct p2m_domain *p2m);
 void free_p2m_hap_data(struct p2m_domain *p2m);
 void p2m_init_hap_data(struct p2m_domain *p2m);
 
+void vmx_pi_per_cpu_init(unsigned int cpu);
+
+void vmx_pi_hooks_assign(struct domain *target);
+void vmx_pi_hooks_deassign(struct domain *source);
+
 /* EPT violation qualifications definitions */
 #define _EPT_READ_VIOLATION         0
 #define EPT_READ_VIOLATION          (1UL<<_EPT_READ_VIOLATION)
Signature · 1 line
2.1.0
#2 lore raw

Re: [PATCH v12 1/2] vmx: VT-d posted-interrupt core logic handling

xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content); · line 113

Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 19.02.16 at 02:55, <[email protected]> wrote:
xen/arch/x86/hvm/vmx/vmx.c +12 -0
+static void vmx_vcpu_block(struct vcpu *v)
+{
+    unsigned long flags;
+    unsigned int dest;
+    spinlock_t *old_lock;
+    spinlock_t *pi_block_list_lock =
+                    &per_cpu(pi_blocked_vcpu_lock, v->processor);
+    struct pi_desc *pi_desc = &v->arch.hvm_vmx.pi_desc;
+
+    spin_lock_irqsave(pi_block_list_lock, flags);
+    old_lock = cmpxchg(&v->arch.hvm_vmx.pi_block_list_lock, NULL,
+                       &per_cpu(pi_blocked_vcpu_lock, v->processor));

Why don't you use the local variable here?

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +7 -0
+    /*
+     * 'v->arch.hvm_vmx.pi_block_list_lock' should be NULL before being assigned
+     * to a new value, since the vCPU is currently running and it cannot be on
+     * any blocking list.
+     */
+    ASSERT(old_lock == NULL);
+

xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content); · line 131

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +10 -0
+    list_add_tail(&v->arch.hvm_vmx.pi_blocked_vcpu_list,
+                  &per_cpu(pi_blocked_vcpu, v->processor));
+    spin_unlock_irqrestore(pi_block_list_lock, flags);
+
+    ASSERT(!pi_test_sn(pi_desc));
+
+    dest = cpu_physical_id(v->processor);
+
+    ASSERT(pi_desc->ndst ==
+           (x2apic_enabled ? dest : MASK_INSR(dest, PI_xAPIC_NDST_MASK)));

May I ask for consistency between this and ...

xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content); · line 152

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +7 -0
+static void vmx_pi_switch_to(struct vcpu *v)
+{
+    struct pi_desc *pi_desc = &v->arch.hvm_vmx.pi_desc;
+
+    write_atomic(&pi_desc->ndst, x2apic_enabled ?
+                 cpu_physical_id(v->processor) :
+                 MASK_INSR(cpu_physical_id(v->processor), PI_xAPIC_NDST_MASK));

... this? I.e. to either in both cases use a local variable "dest", or in both cases to avoid doing so?

Quoted 12 lines
xen/arch/x86/hvm/vmx/vmx.c +12 -0
+static void vmx_pi_do_resume(struct vcpu *v)
+{
+    unsigned long flags;
+    spinlock_t *pi_block_list_lock;
+    struct pi_desc *pi_desc = &v->arch.hvm_vmx.pi_desc;
+
+    ASSERT(!test_bit(_VPF_blocked, &v->pause_flags));
+
+    /*
+     * Set 'NV' field back to posted_intr_vector, so the
+     * Posted-Interrupts can be delivered to the vCPU when
+     * it is running in non-root mode.

xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content); · line 178

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +10 -0
+     */
+    write_atomic(&pi_desc->nv, posted_intr_vector);
+
+    /* The vCPU is not on any blocking list. */
+    pi_block_list_lock = v->arch.hvm_vmx.pi_block_list_lock;
+    if ( pi_block_list_lock == NULL )
+        return;
+
+    /* Prevent the compiler from eliminating the local variable.*/
+    barrier();

As said before - this is too late: This way you still don't guarantee pi_block_list_lock != NULL. The barrier needs to be after the read, and _before_ the first use. Also I think the right barrier here would be smp_rmb().

xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content); · line 187

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +8 -0
+    spin_lock_irqsave(pi_block_list_lock, flags);
+
+    /*
+     * v->arch.hvm_vmx.pi_block_list_lock == NULL here means the vCPU was
+     * removed from the blocking list while we are acquiring the lock.
+     */
+    if ( v->arch.hvm_vmx.pi_block_list_lock != NULL )
+    {

ASSERT(v->arch.hvm_vmx.pi_block_list_lock == pi_block_list_lock)?

Quoted 15 lines
xen/arch/x86/hvm/vmx/vmx.c +15 -0
+/* This function is called when pcidevs_lock is held */
+void vmx_pi_hooks_assign(struct domain *target)
+{
+    if ( !iommu_intpost )
+        return;
+
+    if ( has_hvm_container_domain(target) &&
+         !target->arch.hvm_domain.vmx.vcpu_block  )
+    {
+        target->arch.hvm_domain.vmx.vcpu_block = vmx_vcpu_block;
+        target->arch.hvm_domain.vmx.pi_switch_from = vmx_pi_switch_from;
+        target->arch.hvm_domain.vmx.pi_switch_to = vmx_pi_switch_to;
+        target->arch.hvm_domain.vmx.pi_do_resume = vmx_pi_do_resume;
+    }
+}

xen/arch/x86/hvm/vmx/vmx.c static int vmx_msr_write_intercept(unsigned int msr, uint64_t msr_content); · line 219

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +10 -0
+
+/* This function is called when pcidevs_lock is held */
+void vmx_pi_hooks_deassign(struct domain *source)
+{
+    if ( !iommu_intpost )
+        return;
+
+    if ( has_hvm_container_domain(source) &&
+         source->arch.hvm_domain.vmx.vcpu_block && !has_arch_pdevs(source) )
+    {

I think it would be nice for the conditions in both functions to match up. Since you can't add has_arch_pdevs(target) to the earlier, maybe it should be the caller of the latter to check !has_arch_pdevs(source)? And if you then made the caller(s) of vmx_pi_hooks_assign() do so too, the checks on ->arch.hvm_domain.vmx.vcpu_block could apparently become ASSERT()-s instead.

Also since neither function now takes two domain pointers anymore, please name the function parameters just "d". And finally please, in each function, combine the two if()-s.

xen/arch/x86/hvm/vmx/vmx.c static int vmx_vcpu_initialise(struct vcpu *v) · line 256

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +3 -0
@@ -112,6 +251,10 @@ static int vmx_vcpu_initialise(struct vcpu *v)
 
     spin_lock_init(&v->arch.hvm_vmx.vmcs_lock);
 
+    INIT_LIST_HEAD(&v->arch.hvm_vmx.pi_blocked_vcpu_list);
+
+    v->arch.hvm_vmx.pi_block_list_lock = NULL;

The latter is pointless, as struct vcpu starts out zero-initialized.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_ctxt_switch_from(struct vcpu *v) · line 887

Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -0
@@ -740,6 +883,8 @@ static void vmx_ctxt_switch_from(struct vcpu *v)
     vmx_save_guest_msrs(v);
     vmx_restore_host_msrs();
     vmx_save_dr(v);
+    if (v->domain->arch.hvm_domain.vmx.pi_switch_from)
+        v->domain->arch.hvm_domain.vmx.pi_switch_from(v);

Coding style.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_ctxt_switch_to(struct vcpu *v) · line 901

Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -0
@@ -752,6 +897,8 @@ static void vmx_ctxt_switch_to(struct vcpu *v)
 
     vmx_restore_guest_msrs(v);
     vmx_restore_dr(v);
+    if (v->domain->arch.hvm_domain.vmx.pi_switch_to)
+        v->domain->arch.hvm_domain.vmx.pi_switch_to(v);

Again. I'm pretty sure this has been pointed out before. And there are more of these.

xen/include/asm-x86/hvm/hvm.h const char *hvm_efer_valid(const struct vcpu *v, uint64_t value, · line 572

Quoted 11 lines
xen/include/asm-x86/hvm/hvm.h +4 -0
@@ -565,6 +565,11 @@ const char *hvm_efer_valid(const struct vcpu *v, uint64_t value,
                            signed int cr0_pg);
 unsigned long hvm_cr4_guest_reserved_bits(const struct vcpu *v, bool_t 
 restore);
 
+#define arch_vcpu_block(v) ({                                               \
+    if ( (v)->domain->arch.hvm_domain.vmx.vcpu_block )                      \
+        (v)->domain->arch.hvm_domain.vmx.vcpu_block((v));                   \
+})

I'd prefer if v got evaluated just once in this macro. Also note the redundant parentheses in the function argument.

xen/include/asm-x86/hvm/hvm.h struct arch_vmx_struct { · line 230

Quoted 13 lines
xen/include/asm-x86/hvm/hvm.h +8 -0
@@ -160,6 +219,14 @@ struct arch_vmx_struct {
     struct page_info     *vmwrite_bitmap;
 
     struct page_info     *pml_pg;
+
+    /*
+     * Before it is blocked, vCPU is added to the per-cpu list.
+     * VT-d engine can send wakeup notification event to the
+     * pCPU and wakeup the related vCPU.
+     */
+    struct list_head     pi_blocked_vcpu_list;
+    spinlock_t           *pi_block_list_lock;
 };

Didn't we settle on making this a struct with a "list" and a "lock" member? And along those lines the local per-CPU variables added to xen/arch/x86/hvm/vmx/vmx.c would also benefit from being put in a struct, making more obvious their close relationship.

Jan

author
#3 lore raw

Re: [PATCH v12 1/2] vmx: VT-d posted-interrupt core logic handling

Quoted 15 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, February 22, 2016 9:08 PM
To: Wu, Feng <[email protected]>
Cc: Andrew Cooper <[email protected]>; Dario Faggioli <[email protected]>; George Dunlap <[email protected]>; Tian, Kevin <[email protected]>; [email protected]; Keir Fraser <[email protected]>
Subject: Re: [PATCH v12 1/2] vmx: VT-d posted-interrupt core logic handling

Quoted 3 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +0 -0
@@ -160,6 +219,14 @@ struct arch_vmx_struct {
     struct page_info     *vmwrite_bitmap;
 
Quoted 13 lines

xen/include/asm-x86/hvm/vmx/vmcs.h struct arch_vmx_struct { · line 230

Quoted 10 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +8 -0
     struct page_info     *pml_pg;
+
+    /*
+     * Before it is blocked, vCPU is added to the per-cpu list.
+     * VT-d engine can send wakeup notification event to the
+     * pCPU and wakeup the related vCPU.
+     */
+    struct list_head     pi_blocked_vcpu_list;
+    spinlock_t           *pi_block_list_lock;
 };

Didn't we settle on making this a struct with a "list" and a "lock" member?

I don't think so. I will change like this in the next version.

Quoted 3 lines

And along those lines the local per-CPU variables added to xen/arch/x86/hvm/vmx/vmx.c would also benefit from being put in a struct, making more obvious their close relationship.

' pi_block_list_lock 'in this structure is a pointer to spinlock_t, but the local per-CPU lock is typeof spinlock_t instead of a pointer, so I cannot reuse the struct for the "list" and "lock" member. Do you mean I should create another structure for this two local per-CPU variables?

Thanks,
Feng

#4 lore raw

Re: [PATCH v12 1/2] vmx: VT-d posted-interrupt core logic handling

Quoted 36 lines
Quoted 16 lines
Quoted 1 line · On 23.02.16 at 05:59, <[email protected]> wrote:

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, February 22, 2016 9:08 PM
To: Wu, Feng <[email protected]>
Cc: Andrew Cooper <[email protected]>; Dario Faggioli <[email protected]>; George Dunlap <[email protected]>; Tian, Kevin <[email protected]>; [email protected]; Keir Fraser <[email protected]>
Subject: Re: [PATCH v12 1/2] vmx: VT-d posted-interrupt core logic handling

Quoted 3 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +0 -0
@@ -160,6 +219,14 @@ struct arch_vmx_struct {
     struct page_info     *vmwrite_bitmap;
 
Quoted 13 lines

xen/include/asm-x86/hvm/vmx/vmcs.h struct arch_vmx_struct { · line 230

Quoted 10 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +8 -0
     struct page_info     *pml_pg;
+
+    /*
+     * Before it is blocked, vCPU is added to the per-cpu list.
+     * VT-d engine can send wakeup notification event to the
+     * pCPU and wakeup the related vCPU.
+     */
+    struct list_head     pi_blocked_vcpu_list;
+    spinlock_t           *pi_block_list_lock;
 };

Didn't we settle on making this a struct with a "list" and a "lock" member?

I don't think so. I will change like this in the next version.

Quoted 3 lines

And along those lines the local per-CPU variables added to xen/arch/x86/hvm/vmx/vmx.c would also benefit from being put in a struct, making more obvious their close relationship.

Quoted 5 lines

' pi_block_list_lock 'in this structure is a pointer to spinlock_t, but the local per-CPU lock is typeof spinlock_t instead of a pointer, so I cannot reuse the struct for the "list" and "lock" member. Do you mean I should create another structure for this two local per-CPU variables?

Yes, that's why I said "a struct" instead of "the same struct".

Jan

author
#5 lore raw

PATCH v12 2/2 Add a command line parameter for VT-d posted-interrupts

In reply to Feng Wu

Enable VT-d Posted-Interrupts and add a command line parameter for it.

---
 docs/misc/xen-command-line.markdown | 9 ++++++++-
 xen/drivers/passthrough/iommu.c     | 3 +++
 2 files changed, 11 insertions(+), 1 deletion(-)
docs/misc/xen-command-line.markdown +8 -1
index 467dc8f..ea1d60d 100644
@@ -868,7 +868,7 @@ debug hypervisor only).
 > Default: `new` unless directed-EOI is supported
 
 ### iommu
-> `= List of [ <boolean> | force | required | intremap | qinval | snoop | sharept | dom0-passthrough | dom0-strict | amd-iommu-perdev-intremap | workaround_bios_bug | igfx | verbose | debug ]`
+> `= List of [ <boolean> | force | required | intremap | intpost | qinval | snoop | sharept | dom0-passthrough | dom0-strict | amd-iommu-perdev-intremap | workaround_bios_bug | igfx | verbose | debug ]`
 
 > Sub-options:
 
@@ -895,6 +895,13 @@ debug hypervisor only).
 >> Control the use of interrupt remapping (DMA remapping will always be enabled
 >> if IOMMU functionality is enabled).
 
+> `intpost`
+
+> Default: `false`
+
+>> Control the use of interrupt posting, which depends on the availability of
+>> interrupt remapping.
+
 > `qinval` (VT-d)
 
 > Default: `true`
xen/drivers/passthrough/iommu.c +3 -0
index 0b2abf4..50d74a5 100644
@@ -32,6 +32,7 @@ static void iommu_dump_p2m_table(unsigned char key);
  *   off|no|false|disable       Disable IOMMU (default)
  *   force|required             Don't boot unless IOMMU is enabled
  *   no-intremap                Disable interrupt remapping
+ *   no-intpost                 Disable VT-d Interrupt posting
  *   verbose                    Be more verbose
  *   debug                      Enable debugging messages and checks
  *   workaround_bios_bug        Workaround some bios issue to still enable
@@ -105,6 +106,8 @@ static void __init parse_iommu_param(char *s)
             iommu_qinval = val;
         else if ( !strcmp(s, "intremap") )
             iommu_intremap = val;
+        else if ( !strcmp(s, "intpost") )
+            iommu_intpost = val;
         else if ( !strcmp(s, "debug") )
         {
             iommu_debug = val;
Signature · 1 line
2.1.0