xen: fix/enhance temporary vcpu pinning
cover xen: fix/enhance temporary vcpu pinning
While trying to handle temporary vcpu pinnings in a sane way in my core scheduling series I stumbled over a bug and found a nice way to simplify the temporary pinning cases.
I'm sending the two patches independently from my core scheduling series as they should be considered even without core scheduling.
Juergen Gross (2):
xen/sched: fix locking in restore_vcpu_affinity()
xen: merge temporary vcpu pinning scenarios
xen/arch/x86/pv/traps.c | 20 +------------------- xen/arch/x86/traps.c | 8 ++------ xen/common/domain.c | 4 +--- xen/common/schedule.c | 40 ++++++++++++++++++++++++++++------------ xen/common/wait.c | 26 ++++++++------------------ xen/include/xen/sched.h | 8 +++++--- 6 files changed, 45 insertions(+), 61 deletions(-)
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/2 xen/sched: fix locking in restore_vcpu_affinity()
Commit 0763cd2687897b55e7 ("xen/sched: don't disable scheduler on cpus during suspend") removed a lock in restore_vcpu_affinity() which needs to stay: cpumask_scratch_cpu() must be protected by the scheduler lock. restore_vcpu_affinity() is being called by thaw_domains(), so with multiple domains in the system another domain might already be running and the scheduler might make use of cpumask_scratch_cpu() already.
--- xen/common/schedule.c | 5 +++++ 1 file changed, 5 insertions(+)
xen/common/schedule.c +5 -0
@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)* set v->processor of each of their vCPUs to something that will * make sense for the scheduler of the cpupool in which they are in. */ + lock =vcpu_schedule_lock_irq (v); +cpumask_and (cpumask_scratch_cpu (cpu), v->cpu_hard_affinity,cpupool_domain_cpumask (d));if (cpumask_empty (cpumask_scratch_cpu (cpu)) )
@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d) v->processor =cpumask_any (cpumask_scratch_cpu (cpu)); +spin_unlock_irq (lock); + +/* v->processor might have changed, so reacquire the lock. */ lock =vcpu_schedule_lock_irq (v); v->processor =sched_pick_cpu (vcpu_scheduler (v), v);spin_unlock_irq (lock);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()
Quoted 7 lines · Juergen Gross
Commit
0763cd2687897b55e7("xen/sched: don't disable scheduler on cpus
during suspend") removed a lock inrestore_vcpu_affinity()which needs
to stay:cpumask_scratch_cpu()must be protected by the scheduler lock.
And indeed I recall the thing looked weird, and bringing it up... but then I also managed to convince myself that it wasn't a problem, which apparently is. :-(
Sorry for having overlooked that.
Quoted 7 lines
restore_vcpu_affinity()is being called bythaw_domains(), so with multiple domains in the system another domain might already be running and the scheduler might make use ofcpumask_scratch_cpu()already.Signed-off-by Juergen Gross <[email protected]>
Thanks and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()
In reply to Juergen GrossQuoted 12 lines · Juergen Gross
xen/common/schedule.c +2
-0@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)* set v->processor of each of their vCPUs to something that will * make sense for the scheduler of the cpupool in which they are in. */ + lock =vcpu_schedule_lock_irq (v); +cpumask_and (cpumask_scratch_cpu (cpu), v->cpu_hard_affinity,cpupool_domain_cpumask (d));if (cpumask_empty (cpumask_scratch_cpu (cpu)) )
xen/common/schedule.c void restore_vcpu_affinity(struct domain *d) · line 738
Quoted 6 lines
xen/common/schedule.c +3
-0@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d) v->processor =cpumask_any (cpumask_scratch_cpu (cpu)); +spin_unlock_irq (lock); + +/* v->processor might have changed, so reacquire the lock. */
Whoever commits this (perhaps me, later today) will want to replace the hard tab here.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Juergen Gross
xen/common/schedule.c +2
-0@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)* set v->processor of each of their vCPUs to something that will * make sense for the scheduler of the cpupool in which they are in. */ + lock =vcpu_schedule_lock_irq (v); +cpumask_and (cpumask_scratch_cpu (cpu), v->cpu_hard_affinity,cpupool_domain_cpumask (d));if (cpumask_empty (cpumask_scratch_cpu (cpu)) )@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d)
Quoted 8 lines
xen/common/schedule.c void restore_vcpu_affinity(struct domain *d) · line 738
Quoted 5 lines
xen/common/schedule.c +3
-0v->processor =cpumask_any (cpumask_scratch_cpu (cpu)); +spin_unlock_irq (lock); + +/* v->processor might have changed, so reacquire the lock. */ Whoever commits this (perhaps me, later today) will want to replace the hard tab here.
Oh, I'm sorry for that!
We really want checkpatch in Xen!
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()
In reply to Jan BeulichQuoted 14 lines · Jan Beulich
Quoted 13 lines · Juergen Gross
xen/common/schedule.c +2
-0@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)* set v->processor of each of their vCPUs to something that will * make sense for the scheduler of the cpupool in which they are in. */ + lock =vcpu_schedule_lock_irq (v); +cpumask_and (cpumask_scratch_cpu (cpu), v->cpu_hard_affinity,cpupool_domain_cpumask (d));if (cpumask_empty (cpumask_scratch_cpu (cpu)) )@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d)
Quoted 7 lines
xen/common/schedule.c void restore_vcpu_affinity(struct domain *d) · line 738
Quoted 5 lines
xen/common/schedule.c +3
-0v->processor =cpumask_any (cpumask_scratch_cpu (cpu)); +spin_unlock_irq (lock); + +/* v->processor might have changed, so reacquire the lock. */ Whoever commits this (perhaps me, later today) will want to replace the hard tab here.
I've already committed this, and did fix up the tab.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 2/2 xen: merge temporary vcpu pinning scenarios
In reply to Juergen GrossToday there are three scenarios which are pinning vcpus temporarily to a single physical cpu:
- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling
Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.
So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).
Note that we don't need to call domain_update_node_affinity() as we are only pinning for a brief period of time.
--- xen/arch/x86/pv/traps.c | 20 +------------------- xen/arch/x86/traps.c | 8 ++------ xen/common/domain.c | 4 +--- xen/common/schedule.c | 35 +++++++++++++++++++++++------------ xen/common/wait.c | 26 ++++++++------------------ xen/include/xen/sched.h | 8 +++++--- 6 files changed, 40 insertions(+), 61 deletions(-)
xen/arch/x86/pv/traps.c +1 -19
@@ -151,25 +151,7 @@ static void nmi_mce_softirq(void)BUG_ON (st->vcpu ==NULL ); -/* -* Set the tmp value unconditionally, so that the check in the iret -* hypercall works. -*/ -cpumask_copy (st->vcpu->cpu_hard_affinity_tmp, - st->vcpu->cpu_hard_affinity); - -if ( (cpu != st->processor) || - (st->processor != st->vcpu->processor) ) - { - -/* -* We are on a different physical cpu. Make sure to wakeup the vcpu on -* the specified processor. -*/ -vcpu_set_hard_affinity (st->vcpu,cpumask_of (st->processor)); - -/* Affinity is restored in the iret hypercall. */ - } +vcpu_set_tmp_affinity (st->vcpu, st->processor,VCPU_AFFINITY_NMI );/* * Only used to defer wakeup of domain/vcpu to a safe (non-NMI/MCE)
xen/arch/x86/traps.c +2 -6
@@ -1601,12 +1601,8 @@ void async_exception_cleanup(struct vcpu *curr)return ;/* Restore affinity. */ -if ( !cpumask_empty (curr->cpu_hard_affinity_tmp) && - !cpumask_equal (curr->cpu_hard_affinity_tmp, curr->cpu_hard_affinity) ) - { -vcpu_set_hard_affinity (curr, curr->cpu_hard_affinity_tmp); -cpumask_clear (curr->cpu_hard_affinity_tmp); - } +if ( curr->affinity_broken &VCPU_AFFINITY_NMI ) +vcpu_set_tmp_affinity (curr,-1 ,VCPU_AFFINITY_NMI );if ( !(curr->async_exception_mask & (curr->async_exception_mask -1 )) ) trap =__scanbit (curr->async_exception_mask,VCPU_TRAP_NONE );
xen/common/domain.c +1 -3
@@ -133,7 +133,6 @@ static void vcpu_info_reset(struct vcpu *v)static void vcpu_destroy (struct vcpu *v) {free_cpumask_var (v->cpu_hard_affinity); -free_cpumask_var (v->cpu_hard_affinity_tmp);free_cpumask_var (v->cpu_hard_affinity_saved);free_cpumask_var (v->cpu_soft_affinity);
@@ -161,7 +160,6 @@ struct vcpu *vcpu_create(grant_table_init_vcpu (v);if ( !zalloc_cpumask_var (&v->cpu_hard_affinity) || - !zalloc_cpumask_var (&v->cpu_hard_affinity_tmp) || !zalloc_cpumask_var (&v->cpu_hard_affinity_saved) || !zalloc_cpumask_var (&v->cpu_soft_affinity) ) goto fail;
@@ -1269,7 +1267,7 @@ int vcpu_reset(struct vcpu *v) v->async_exception_mask =0 ;memset (v->async_exception_state,0 ,sizeof (v->async_exception_state));#endif -cpumask_clear (v->cpu_hard_affinity_tmp); + v->affinity_broken =0 ;clear_bit (_VPF_blocked, &v->pause_flags);clear_bit (_VPF_in_reset, &v->pause_flags);
xen/common/schedule.c +23 -12
@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)kill_timer (&d->watchdog_timer[i]); } -int vcpu_pin_override (struct vcpu *v,int cpu) +int vcpu_set_tmp_affinity (struct vcpu *v,int cpu,uint8_t reason) {spinlock_t *lock;int ret = -EINVAL ; +bool migrate; lock =vcpu_schedule_lock_irq (v);if ( cpu <0 ) { -if ( v->affinity_broken ) +if ( v->affinity_broken & reason ) { -sched_set_affinity (v, v->cpu_hard_affinity_saved,NULL ); - v->affinity_broken =0 ; ret =0 ; + v->affinity_broken &= ~reason; } +if ( !ret && !v->affinity_broken ) +sched_set_affinity (v, v->cpu_hard_affinity_saved,NULL ); }else if ( cpu < nr_cpu_ids ) { -if ( v->affinity_broken ) +if ( (v->affinity_broken & reason) || + (v->affinity_broken && v->processor != cpu) ) ret = -EBUSY ;else if (cpumask_test_cpu (cpu,VCPU2ONLINE (v)) ) { -cpumask_copy (v->cpu_hard_affinity_saved, v->cpu_hard_affinity); - v->affinity_broken =1 ; -sched_set_affinity (v,cpumask_of (cpu),NULL ); +if ( !v->affinity_broken ) + { +cpumask_copy (v->cpu_hard_affinity_saved, v->cpu_hard_affinity); +sched_set_affinity (v,cpumask_of (cpu),NULL ); + } + v->affinity_broken |= reason; ret =0 ; } } -if ( ret ==0 ) + migrate = !ret && !cpumask_test_cpu (v->processor, v->cpu_hard_affinity); +if ( migrate )vcpu_migrate_start (v);vcpu_schedule_unlock_irq (lock, v); -domain_update_node_affinity (v->domain); - -vcpu_migrate_finish (v); +if ( migrate ) +vcpu_migrate_finish (v);return ret; } +int vcpu_pin_override (struct vcpu *v,int cpu) +{ +return vcpu_set_tmp_affinity (v, cpu,VCPU_AFFINITY_OVERRIDE ); +} +typedef long ret_t ;#endif /* !COMPAT */
xen/common/wait.c +8 -18
@@ -34,8 +34,6 @@ struct waitqueue_vcpu {*/ void *esp;char *stack; -cpumask_t saved_affinity; -unsigned int wakeup_cpu;#endif };
@@ -131,9 +129,7 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)ASSERT (wqv->esp ==0 );/* Save current VCPU affinity; force wakeup on *this* CPU only. */ - wqv->wakeup_cpu =smp_processor_id (); -cpumask_copy (&wqv->saved_affinity, curr->cpu_hard_affinity); -if (vcpu_set_hard_affinity (curr,cpumask_of (wqv->wakeup_cpu)) ) +if (vcpu_set_tmp_affinity (curr,smp_processor_id (),VCPU_AFFINITY_WAIT ) ) {gdprintk (XENLOG_ERR ,"Unable to set vcpu affinity\n" );domain_crash (current->domain);
@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)static void __finish_wait (struct waitqueue_vcpu *wqv) { wqv->esp =NULL ; - (void )vcpu_set_hard_affinity (current, &wqv->saved_affinity); +vcpu_set_tmp_affinity (current,-1 ,VCPU_AFFINITY_WAIT ); }void check_wakeup_from_wait (void ) { -struct waitqueue_vcpu *wqv = current->waitqueue_vcpu; +struct vcpu *curr = current; +struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;ASSERT (list_empty (&wqv->list));if (likely (wqv->esp ==NULL ) )return ; -/* Check if we woke up on the wrong CPU. */ -if (unlikely (smp_processor_id () != wqv->wakeup_cpu) ) +/* Check if we are still pinned. */ +if (unlikely (!(curr->affinity_broken &VCPU_AFFINITY_WAIT )) ) { -/* Re-set VCPU affinity and re-enter the scheduler. */ -struct vcpu *curr = current; -cpumask_copy (&wqv->saved_affinity, curr->cpu_hard_affinity); -if (vcpu_set_hard_affinity (curr,cpumask_of (wqv->wakeup_cpu)) ) - { -gdprintk (XENLOG_ERR ,"Unable to set vcpu affinity\n" ); -domain_crash (current->domain); - } -wait ();/* takes us back into the scheduler */ +gdprintk (XENLOG_ERR ,"vcpu affinity lost\n" ); +domain_crash (current->domain); }/*
xen/include/xen/sched.h +5 -3
@@ -200,7 +200,10 @@ struct vcpu/* VCPU is paused following shutdown request (d->is_shutting_down)? */ bool paused_for_shutdown;/* VCPU need affinity restored */ -bool affinity_broken; +uint8_t affinity_broken; +#define VCPU_AFFINITY_OVERRIDE 0x01 +#define VCPU_AFFINITY_NMI 0x02 +#define VCPU_AFFINITY_WAIT 0x04/* A hypercall has been preempted. */ bool hcall_preempted;
@@ -246,8 +249,6 @@ struct vcpu/* Bitmask of CPUs on which this VCPU may run. */ cpumask_var_t cpu_hard_affinity;/* Used to change affinity temporarily. */ -cpumask_var_t cpu_hard_affinity_tmp; -/* Used to restore affinity across S3. */ cpumask_var_t cpu_hard_affinity_saved;/* Bitmask of CPUs on which this VCPU prefers to run. */
@@ -875,6 +876,7 @@ int cpu_disable_scheduler(unsigned int cpu);/* We need it in dom0_setup_vcpu */ void sched_set_affinity (struct vcpu *v,const cpumask_t *hard,const cpumask_t *soft); +int vcpu_set_tmp_affinity (struct vcpu *v,int cpu,uint8_t reason);int vcpu_set_hard_affinity (struct vcpu *v,const cpumask_t *affinity);int vcpu_set_soft_affinity (struct vcpu *v,const cpumask_t *affinity);void restore_vcpu_affinity (struct domain *d);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 49 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:
- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handlingEach of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while
vcpu_pin_override()is allowed to fail.So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).
Note that we don't need to call
domain_update_node_affinity()as we are only pinning for a brief period of time.Signed-off-by Juergen Gross <[email protected]>--- xen/arch/x86/pv/traps.c | 20 +------------------- xen/arch/x86/traps.c | 8 ++------ xen/common/domain.c | 4 +--- xen/common/schedule.c | 35 +++++++++++++++++++++++------------ xen/common/wait.c | 26 ++++++++------------------ xen/include/xen/sched.h | 8 +++++--- 6 files changed, 40 insertions(+), 61 deletions(-)xen/arch/x86/pv/traps.c +0
-10@@ -151,25 +151,7 @@ static void nmi_mce_softirq(void)BUG_ON (st->vcpu ==NULL ); -/* -* Set the tmp value unconditionally, so that the check in the iret -* hypercall works. -*/ -cpumask_copy (st->vcpu->cpu_hard_affinity_tmp, - st->vcpu->cpu_hard_affinity); - -if ( (cpu != st->processor) || - (st->processor != st->vcpu->processor) ) - {
xen/arch/x86/pv/traps.c static void nmi_mce_softirq(void) · line 154
Quoted 10 lines
xen/arch/x86/pv/traps.c +1
-9- -/* -* We are on a different physical cpu. Make sure to wakeup the vcpu on -* the specified processor. -*/ -vcpu_set_hard_affinity (st->vcpu,cpumask_of (st->processor)); - -/* Affinity is restored in the iret hypercall. */ - } +vcpu_set_tmp_affinity (st->vcpu, st->processor,VCPU_AFFINITY_NMI );
Please can we keep the comment explaining where the affinity is restored, which is a disguised explanation of why it is PV-only.
Quoted 58 lines
xen/common/schedule.c +16
-9@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)kill_timer (&d->watchdog_timer[i]); } -int vcpu_pin_override (struct vcpu *v,int cpu) +int vcpu_set_tmp_affinity (struct vcpu *v,int cpu,uint8_t reason) {spinlock_t *lock;int ret = -EINVAL ; +bool migrate; lock =vcpu_schedule_lock_irq (v);if ( cpu <0 ) { -if ( v->affinity_broken ) +if ( v->affinity_broken & reason ) { -sched_set_affinity (v, v->cpu_hard_affinity_saved,NULL ); - v->affinity_broken =0 ; ret =0 ; + v->affinity_broken &= ~reason; } +if ( !ret && !v->affinity_broken ) +sched_set_affinity (v, v->cpu_hard_affinity_saved,NULL ); }else if ( cpu < nr_cpu_ids ) { -if ( v->affinity_broken ) +if ( (v->affinity_broken & reason) || + (v->affinity_broken && v->processor != cpu) ) ret = -EBUSY ;else if (cpumask_test_cpu (cpu,VCPU2ONLINE (v)) ) { -cpumask_copy (v->cpu_hard_affinity_saved, v->cpu_hard_affinity); - v->affinity_broken =1 ; -sched_set_affinity (v,cpumask_of (cpu),NULL ); +if ( !v->affinity_broken ) + { +cpumask_copy (v->cpu_hard_affinity_saved, v->cpu_hard_affinity); +sched_set_affinity (v,cpumask_of (cpu),NULL ); + } + v->affinity_broken |= reason; ret =0 ; } } -if ( ret ==0 ) + migrate = !ret && !cpumask_test_cpu (v->processor, v->cpu_hard_affinity); +if ( migrate )vcpu_migrate_start (v);vcpu_schedule_unlock_irq (lock, v);
xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1156
Quoted 10 lines
xen/common/schedule.c +3
-3-domain_update_node_affinity (v->domain); - -vcpu_migrate_finish (v); +if ( migrate ) +vcpu_migrate_finish (v);return ret; } +int vcpu_pin_override (struct vcpu *v,int cpu)
There are exactly two callers of vcpu_pin_override(). I'd take the opportunity to make vcpu_set_tmp_affinity() the single API call for adjusting affinity.
Quoted 36 lines
xen/common/schedule.c +4
-0+{ +return vcpu_set_tmp_affinity (v, cpu,VCPU_AFFINITY_OVERRIDE ); +} +typedef long ret_t ;#endif /* !COMPAT */ xen/common/wait.c +5
-5@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)static void __finish_wait (struct waitqueue_vcpu *wqv) { wqv->esp =NULL ; - (void )vcpu_set_hard_affinity (current, &wqv->saved_affinity); +vcpu_set_tmp_affinity (current,-1 ,VCPU_AFFINITY_WAIT ); }void check_wakeup_from_wait (void ) { -struct waitqueue_vcpu *wqv = current->waitqueue_vcpu; +struct vcpu *curr = current; +struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;ASSERT (list_empty (&wqv->list));if (likely (wqv->esp ==NULL ) )return ; -/* Check if we woke up on the wrong CPU. */ -if (unlikely (smp_processor_id () != wqv->wakeup_cpu) ) +/* Check if we are still pinned. */ +if (unlikely (!(curr->affinity_broken &VCPU_AFFINITY_WAIT )) ) { -/* Re-set VCPU affinity and re-enter the scheduler. */
xen/common/wait.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 198
Quoted 10 lines
xen/common/wait.c +2
-8-struct vcpu *curr = current; -cpumask_copy (&wqv->saved_affinity, curr->cpu_hard_affinity); -if (vcpu_set_hard_affinity (curr,cpumask_of (wqv->wakeup_cpu)) ) - { -gdprintk (XENLOG_ERR ,"Unable to set vcpu affinity\n" ); -domain_crash (current->domain); - } -wait ();/* takes us back into the scheduler */ +gdprintk (XENLOG_ERR ,"vcpu affinity lost\n" ); +domain_crash (current->domain);
curr
Quoted 8 lines
xen/common/wait.c +0
-0}/* xen/include/xen/sched.h +0
-0
xen/include/xen/sched.h struct vcpu · line 205
Quoted 7 lines
xen/include/xen/sched.h +3
-1@@ -200,7 +200,10 @@ struct vcpu/* VCPU is paused following shutdown request (d->is_shutting_down)? */ bool paused_for_shutdown;/* VCPU need affinity restored */ -bool affinity_broken; +uint8_t affinity_broken; +#define VCPU_AFFINITY_OVERRIDE 0x01 +#define VCPU_AFFINITY_NMI 0x02
VCPU_AFFINITY_NMI_MCE ? It is used for more than just NMIs.
~Andrew
xen/include/xen/sched.h struct vcpu · line 209
Quoted 5 lines
xen/include/xen/sched.h +1
-0+#define VCPU_AFFINITY_WAIT 0x04/* A hypercall has been preempted. */ bool hcall_preempted;
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 55 lines · Andrew Cooper
Quoted 54 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:
- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handlingEach of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while
vcpu_pin_override()is allowed to fail.So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).
Note that we don't need to call
domain_update_node_affinity()as we are only pinning for a brief period of time.Signed-off-by Juergen Gross <[email protected]>--- xen/arch/x86/pv/traps.c | 20 +------------------- xen/arch/x86/traps.c | 8 ++------ xen/common/domain.c | 4 +--- xen/common/schedule.c | 35 +++++++++++++++++++++++------------ xen/common/wait.c | 26 ++++++++------------------ xen/include/xen/sched.h | 8 +++++--- 6 files changed, 40 insertions(+), 61 deletions(-)xen/arch/x86/pv/traps.c +0
-15@@ -151,25 +151,7 @@ static void nmi_mce_softirq(void)BUG_ON (st->vcpu ==NULL ); -/* -* Set the tmp value unconditionally, so that the check in the iret -* hypercall works. -*/ -cpumask_copy (st->vcpu->cpu_hard_affinity_tmp, - st->vcpu->cpu_hard_affinity); - -if ( (cpu != st->processor) || - (st->processor != st->vcpu->processor) ) - { - -/* -* We are on a different physical cpu. Make sure to wakeup the vcpu on -* the specified processor. -*/
Quoted 8 lines
xen/arch/x86/pv/traps.c static void nmi_mce_softirq(void) · line 154
Quoted 5 lines
xen/arch/x86/pv/traps.c +1
-4-vcpu_set_hard_affinity (st->vcpu,cpumask_of (st->processor)); - -/* Affinity is restored in the iret hypercall. */ - } +vcpu_set_tmp_affinity (st->vcpu, st->processor,VCPU_AFFINITY_NMI );Please can we keep the comment explaining where the affinity is restored, which is a disguised explanation of why it is PV-only.
Okay.
Quoted 65 lines
Quoted 65 lines
xen/common/schedule.c +18
-12@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)kill_timer (&d->watchdog_timer[i]); } -int vcpu_pin_override (struct vcpu *v,int cpu) +int vcpu_set_tmp_affinity (struct vcpu *v,int cpu,uint8_t reason) {spinlock_t *lock;int ret = -EINVAL ; +bool migrate; lock =vcpu_schedule_lock_irq (v);if ( cpu <0 ) { -if ( v->affinity_broken ) +if ( v->affinity_broken & reason ) { -sched_set_affinity (v, v->cpu_hard_affinity_saved,NULL ); - v->affinity_broken =0 ; ret =0 ; + v->affinity_broken &= ~reason; } +if ( !ret && !v->affinity_broken ) +sched_set_affinity (v, v->cpu_hard_affinity_saved,NULL ); }else if ( cpu < nr_cpu_ids ) { -if ( v->affinity_broken ) +if ( (v->affinity_broken & reason) || + (v->affinity_broken && v->processor != cpu) ) ret = -EBUSY ;else if (cpumask_test_cpu (cpu,VCPU2ONLINE (v)) ) { -cpumask_copy (v->cpu_hard_affinity_saved, v->cpu_hard_affinity); - v->affinity_broken =1 ; -sched_set_affinity (v,cpumask_of (cpu),NULL ); +if ( !v->affinity_broken ) + { +cpumask_copy (v->cpu_hard_affinity_saved, v->cpu_hard_affinity); +sched_set_affinity (v,cpumask_of (cpu),NULL ); + } + v->affinity_broken |= reason; ret =0 ; } } -if ( ret ==0 ) + migrate = !ret && !cpumask_test_cpu (v->processor, v->cpu_hard_affinity); +if ( migrate )vcpu_migrate_start (v);vcpu_schedule_unlock_irq (lock, v); -domain_update_node_affinity (v->domain); - -vcpu_migrate_finish (v); +if ( migrate ) +vcpu_migrate_finish (v);
Quoted 8 lines
xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1156
Quoted 4 lines
xen/common/schedule.c +1
-0return ret; } +int vcpu_pin_override (struct vcpu *v,int cpu)There are exactly two callers of vcpu_pin_override(). I'd take the opportunity to make
vcpu_set_tmp_affinity()the single API call for adjusting affinity.
Fine with me.
Quoted 41 lines
Quoted 41 lines
xen/common/schedule.c +4
-0+{ +return vcpu_set_tmp_affinity (v, cpu,VCPU_AFFINITY_OVERRIDE ); +} +typedef long ret_t ;#endif /* !COMPAT */ xen/common/wait.c +5
-9@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)static void __finish_wait (struct waitqueue_vcpu *wqv) { wqv->esp =NULL ; - (void )vcpu_set_hard_affinity (current, &wqv->saved_affinity); +vcpu_set_tmp_affinity (current,-1 ,VCPU_AFFINITY_WAIT ); }void check_wakeup_from_wait (void ) { -struct waitqueue_vcpu *wqv = current->waitqueue_vcpu; +struct vcpu *curr = current; +struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;ASSERT (list_empty (&wqv->list));if (likely (wqv->esp ==NULL ) )return ; -/* Check if we woke up on the wrong CPU. */ -if (unlikely (smp_processor_id () != wqv->wakeup_cpu) ) +/* Check if we are still pinned. */ +if (unlikely (!(curr->affinity_broken &VCPU_AFFINITY_WAIT )) ) { -/* Re-set VCPU affinity and re-enter the scheduler. */ -struct vcpu *curr = current; -cpumask_copy (&wqv->saved_affinity, curr->cpu_hard_affinity); -if (vcpu_set_hard_affinity (curr,cpumask_of (wqv->wakeup_cpu)) ) - {
Quoted 8 lines
xen/common/wait.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 198
Quoted 6 lines
xen/common/wait.c +2
-4-gdprintk (XENLOG_ERR ,"Unable to set vcpu affinity\n" ); -domain_crash (current->domain); - } -wait ();/* takes us back into the scheduler */ +gdprintk (XENLOG_ERR ,"vcpu affinity lost\n" ); +domain_crash (current->domain);curr
Okay.
Quoted 18 lines
Quoted 9 lines
xen/common/wait.c +0
-0}/* xen/include/xen/sched.h +0
-0xen/include/xen/sched.h struct vcpu · line 205
Quoted 7 lines
xen/include/xen/sched.h +3
-1@@ -200,7 +200,10 @@ struct vcpu/* VCPU is paused following shutdown request (d->is_shutting_down)? */ bool paused_for_shutdown;/* VCPU need affinity restored */ -bool affinity_broken; +uint8_t affinity_broken; +#define VCPU_AFFINITY_OVERRIDE 0x01 +#define VCPU_AFFINITY_NMI 0x02
VCPU_AFFINITY_NMI_MCE? It is used for more than just NMIs.
Okay.
BTW: The MCE case is never triggered (there is no caller of pv_raise_interrupt() with TRAP_machine_check). Is this code left in place on purpose or could it be removed?
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 21 lines · Juergen Gross
Quoted 11 lines · Andrew Cooper
Quoted 9 lines · Juergen Gross
}
/*xen/include/xen/sched.h +0
-0@@ -200,7 +200,10 @@ struct vcpu /* VCPU is paused following shutdown request(
d->is_shutting_down)? */Quoted 8 lines
xen/include/xen/sched.h struct vcpu · line 205
Quoted 6 lines
bool
paused_for_shutdown;
/* VCPU need affinity restored */
- boolaffinity_broken;
+ uint8_taffinity_broken;xen/include/xen/sched.h +2
-0+#define VCPU_AFFINITY_OVERRIDE 0x01 +#define VCPU_AFFINITY_NMI 0x02
VCPU_AFFINITY_NMI_MCE? It is used for more than just NMIs.
Quoted 5 lines
Okay.
BTW: The MCE case is never triggered (there is no caller of
pv_raise_interrupt()withTRAP_machine_check). Is this code left in place on purpose or could it be removed?
It come from the restore_all_guest path in assembly, via process_mce.
This code is horribly tangled and in severe need of rewriting in C.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 22 lines · Andrew Cooper
Quoted 21 lines · Juergen Gross
Quoted 19 lines · Andrew Cooper
Quoted 13 lines · Juergen Gross
}
/*xen/include/xen/sched.h +0
-0@@ -200,7 +200,10 @@ struct vcpu /* VCPU is paused following shutdown request(
d->is_shutting_down)? */
boolpaused_for_shutdown;
/* VCPU need affinity restored */
- boolaffinity_broken;
+ uint8_taffinity_broken;xen/include/xen/sched.h struct vcpu · line 205
Quoted 2 lines
xen/include/xen/sched.h +2
-0+#define VCPU_AFFINITY_OVERRIDE 0x01 +#define VCPU_AFFINITY_NMI 0x02
VCPU_AFFINITY_NMI_MCE? It is used for more than just NMIs.
Quoted 7 lines
Quoted 5 lines
Okay.
BTW: The MCE case is never triggered (there is no caller of
pv_raise_interrupt()withTRAP_machine_check). Is this code left in place on purpose or could it be removed?It come from the
restore_all_guestpath in assembly, viaprocess_mce.
Are you sure? I can see that it would call set_guest_machinecheck_trapbounce(), but I don't see how NMI_MCE_SOFTIRQ would be triggered on this path, which would be required for hitting nmi_mce_softirq().
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 29 lines · Juergen Gross
Quoted 28 lines · Andrew Cooper
Quoted 26 lines · Juergen Gross
Quoted 11 lines · Andrew Cooper
Quoted 9 lines · Juergen Gross
}
/*xen/include/xen/sched.h +0
-0@@ -200,7 +200,10 @@ struct vcpu /* VCPU is paused following shutdown request(
d->is_shutting_down)? */Quoted 8 lines
xen/include/xen/sched.h struct vcpu · line 205
Quoted 6 lines
bool
paused_for_shutdown;
/* VCPU need affinity restored */
- boolaffinity_broken;
+ uint8_taffinity_broken;xen/include/xen/sched.h +2
-0+#define VCPU_AFFINITY_OVERRIDE 0x01 +#define VCPU_AFFINITY_NMI 0x02
VCPU_AFFINITY_NMI_MCE? It is used for more than just NMIs.Okay.
BTW: The MCE case is never triggered (there is no caller of
pv_raise_interrupt()withTRAP_machine_check). Is this code left in place on purpose or could it be removed?
Quoted 6 lines
Quoted 1 line
It come from the
restore_all_guestpath in assembly, viaprocess_mce.Are you sure? I can see that it would call
set_guest_machinecheck_trapbounce(), but I don't see howNMI_MCE_SOFTIRQwould be triggered on this path, which would be required for hittingnmi_mce_softirq().
Looking at it, I'm not so sure...
There is certainly room for further clean-up here, for anyone with enough time to disentangle this rats nest.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
In reply to Juergen GrossQuoted 9 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:
- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handlingEach of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
And what guarantees that no two of them will be in use at the same time? You don't even mention ...
Quoted 3 lines
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while
vcpu_pin_override()is allowed to fail.
.. the NMI/#MC injection case here (despite the use of "the two" and "while" giving the impression). Or (looking at the actual code) did you mean "former" instead of "latter"? But if so - id that true? v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.
Independent of that introducing new failure cases for vcpu_pin_override() isn't really nice. Then again any two racing/conflicting pinning attempts can't result in anything good.
Nevertheless - nice idea, so a few minor comments on the code as well, in the hope that my point above can be addressed.
xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1109
Quoted 8 lines
xen/common/schedule.c +1
-1@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)kill_timer (&d->watchdog_timer[i]); } -int vcpu_pin_override (struct vcpu *v,int cpu) +int vcpu_set_tmp_affinity (struct vcpu *v,int cpu,uint8_t reason)
I'd find it pretty nice if at this occasion the type of "cpu" was changed to "unsigned int", using UINT_MAX or NR_CPUS instead of -1 for the "remove override" case.
I'd also prefer if you didn't use "tmp" as an infix here. Make it "temporary", "transient", or some such. Perhaps even omit "set", the more that the function may also clear it.
Quoted 26 lines
xen/common/schedule.c +5
-6@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)static void __finish_wait (struct waitqueue_vcpu *wqv) { wqv->esp =NULL ; - (void )vcpu_set_hard_affinity (current, &wqv->saved_affinity); +vcpu_set_tmp_affinity (current,-1 ,VCPU_AFFINITY_WAIT ); }void check_wakeup_from_wait (void ) { -struct waitqueue_vcpu *wqv = current->waitqueue_vcpu; +struct vcpu *curr = current; +struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;ASSERT (list_empty (&wqv->list));if (likely (wqv->esp ==NULL ) )return ; -/* Check if we woke up on the wrong CPU. */ -if (unlikely (smp_processor_id () != wqv->wakeup_cpu) ) +/* Check if we are still pinned. */ +if (unlikely (!(curr->affinity_broken &VCPU_AFFINITY_WAIT )) ) { -/* Re-set VCPU affinity and re-enter the scheduler. */ -struct vcpu *curr = current;
xen/common/schedule.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 199
Quoted 10 lines
xen/common/schedule.c +2
-7-cpumask_copy (&wqv->saved_affinity, curr->cpu_hard_affinity); -if (vcpu_set_hard_affinity (curr,cpumask_of (wqv->wakeup_cpu)) ) - { -gdprintk (XENLOG_ERR ,"Unable to set vcpu affinity\n" ); -domain_crash (current->domain); - } -wait ();/* takes us back into the scheduler */ +gdprintk (XENLOG_ERR ,"vcpu affinity lost\n" ); +domain_crash (current->domain); }
Please use curr in favor of current.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to
a single physical cpu:- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handling
Quoted 7 lines
Quoted 2 lines
Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
And what guarantees that no two of them will be in use at the same time?
You don't even mention ...
AFAIUI, the case of any two of the above being in use at the same time is handled. Basically, it is fine, as far as they (temporarily) set the affinity to the same CPU, which is said to always be the case, here...
Quoted 5 lines
Quoted 4 lines
The three cases can be combined as the two latter cases will only pin
a vcpu to the physical cpu it is already running on, whilevcpu_pin_override()is allowed to fail.
...for the first two. For vcpu_pin_override(), if it is called for pinning the vcpu to CPU X, with any of the other twos in effect, and having temporarily pinned the vcpu to CPU X already, then things are fine again. If vcpu_pin_override() wants to pin to CPU Y, it fails.
All that being said, I agree it'd be good to have a bit more comments (and I asked the same, although about a different path, in my own reply).
Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
In reply to Jan BeulichQuoted 14 lines · Jan Beulich
Quoted 9 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:
- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handlingEach of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
And what guarantees that no two of them will be in use at the same time? You don't even mention ...
Quoted 7 lines
Quoted 3 lines
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while
vcpu_pin_override()is allowed to fail... the NMI/#MC injection case here (despite the use of "the two" and "while" giving the impression). Or (looking at the actual code) did you mean "former" instead of "latter"? But if so - id that true?
Yes, I meant former.
Quoted 5 lines
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.
You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Quoted 3 lines
Independent of that introducing new failure cases for
vcpu_pin_override()isn't really nice. Then again any two racing/conflicting pinning attempts can't result in anything good.
Right.
Quoted 9 lines
Nevertheless - nice idea, so a few minor comments on the code as well, in the hope that my point above can be addressed.
Quoted 6 lines
xen/common/schedule.c +0
-0@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)kill_timer (&d->watchdog_timer[i]); }
Quoted 6 lines
xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1109
Quoted 2 lines
xen/common/schedule.c +1
-1-int vcpu_pin_override (struct vcpu *v,int cpu) +int vcpu_set_tmp_affinity (struct vcpu *v,int cpu,uint8_t reason)I'd find it pretty nice if at this occasion the type of "cpu" was changed to "unsigned int", using
UINT_MAXorNR_CPUSinstead of -1 for the "remove override" case.
Fine with me.
Quoted 3 lines
I'd also prefer if you didn't use "tmp" as an infix here. Make it "temporary", "transient", or some such. Perhaps even omit "set", the more that the function may also clear it.
Okay.
Quoted 31 lines
Quoted 31 lines
xen/common/schedule.c +5
-10@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)static void __finish_wait (struct waitqueue_vcpu *wqv) { wqv->esp =NULL ; - (void )vcpu_set_hard_affinity (current, &wqv->saved_affinity); +vcpu_set_tmp_affinity (current,-1 ,VCPU_AFFINITY_WAIT ); }void check_wakeup_from_wait (void ) { -struct waitqueue_vcpu *wqv = current->waitqueue_vcpu; +struct vcpu *curr = current; +struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;ASSERT (list_empty (&wqv->list));if (likely (wqv->esp ==NULL ) )return ; -/* Check if we woke up on the wrong CPU. */ -if (unlikely (smp_processor_id () != wqv->wakeup_cpu) ) +/* Check if we are still pinned. */ +if (unlikely (!(curr->affinity_broken &VCPU_AFFINITY_WAIT )) ) { -/* Re-set VCPU affinity and re-enter the scheduler. */ -struct vcpu *curr = current; -cpumask_copy (&wqv->saved_affinity, curr->cpu_hard_affinity); -if (vcpu_set_hard_affinity (curr,cpumask_of (wqv->wakeup_cpu)) ) - { -gdprintk (XENLOG_ERR ,"Unable to set vcpu affinity\n" );
Quoted 8 lines
xen/common/schedule.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 199
Quoted 6 lines
xen/common/schedule.c +2
-3-domain_crash (current->domain); - } -wait ();/* takes us back into the scheduler */ +gdprintk (XENLOG_ERR ,"vcpu affinity lost\n" ); +domain_crash (current->domain); }Please use curr in favor of current.
Yes.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
Quoted 3 lines
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Then it was never finished or got broken, I would guess. Especially for #MC the idea is/was for the domain to be able to access the MSRs in question (). Iirc, that is, and prior to vMCE appearing.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
Quoted 5 lines
Quoted 3 lines
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.
Ah, okay, so no need any longer to rename VCPU_AFFINITY_NMI. :-)
I'll add a patch removing the MCE cruft.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
In reply to Jan BeulichQuoted 17 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.
So just to be sure: you are fine for me removing the pinning for NMIs?
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 22 lines · Juergen Gross
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Juergen Gross
Quoted 5 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.So just to be sure: you are fine for me removing the pinning for NMIs?
No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).
What can go away if the #MC part of the logic.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 25 lines · Jan Beulich
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Jan Beulich
Quoted 21 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.
Quoted 3 lines
One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.
Then it was never finished or got broken, I would guess.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.So just to be sure: you are fine for me removing the pinning for NMIs?
No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).
I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?
Quoted 1 line
What can go away if the #MC part of the logic.
Okay.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 42 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 39 lines · Juergen Gross
Quoted 31 lines · Jan Beulich
Quoted 30 lines · Jan Beulich
Quoted 28 lines · Juergen Gross
Quoted 9 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq,
but can't the vCPU be moved elsewhere by the time the softirq handler
actually gains control? If that's not possible (and if it's not obvious
why, and as you can see it's not obvious to me), then I think a code
comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu
it is executing (or has last executed) at the time the NMI/MCE is being
queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the
cpu selection when the softirq handler is running.One question to understand the idea nehind all that: _why_ is the vcpu
pinned until it does an iret? I could understand if it would be pinned
to the cpu where the NMI/MCE was happening, but this is not the case.Quoted 5 lines
Quoted 1 line
Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.So just to be sure: you are fine for me removing the pinning for NMIs?
Quoted 8 lines
Quoted 3 lines
No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).
I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?
Its (allegedly) for when dom0 knows some system-specific way of getting extra information out of the platform, that happens to be core-specific.
There are rare cases where SMI's need to be executed on CPU0, and I wouldn't put it past hardware designers to have similar aspects for NMIs.
That said, as soon as the gaping security hole which is the default-readibility of all MSRs, I bet the utility of this pinning mechanism will be 0.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 52 lines · Andrew Cooper
Quoted 42 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 39 lines · Juergen Gross
Quoted 31 lines · Jan Beulich
Quoted 30 lines · Jan Beulich
Quoted 28 lines · Juergen Gross
Quoted 9 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq,
but can't the vCPU be moved elsewhere by the time the softirq handler
actually gains control? If that's not possible (and if it's not obvious
why, and as you can see it's not obvious to me), then I think a code
comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu
it is executing (or has last executed) at the time the NMI/MCE is being
queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the
cpu selection when the softirq handler is running.One question to understand the idea nehind all that: _why_ is the vcpu
pinned until it does an iret? I could understand if it would be pinned
to the cpu where the NMI/MCE was happening, but this is not the case.Quoted 5 lines
Quoted 1 line
Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.So just to be sure: you are fine for me removing the pinning for NMIs?
Quoted 8 lines
Quoted 3 lines
No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).
I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?
Quoted 5 lines
Its (allegedly) for when dom0 knows some system-specific way of getting extra information out of the platform, that happens to be core-specific.
There are rare cases where SMI's need to be executed on CPU0, and I wouldn't put it past hardware designers to have similar aspects for NMIs.
Understood. But today vcpu0 is _not_ bound to CPU0, but to any cpu it happened to run on.
Quoted 4 lines
That said, as soon as the gaping security hole which is the default-readibility of all MSRs, I bet the utility of this pinning mechanism will be 0.
And my reasoning is that this is the case today already, as there is no pinning to CPU0 done, at least not on purpose.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 68 lines · Juergen Gross
Quoted 56 lines · Andrew Cooper
Quoted 49 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 40 lines · Juergen Gross
Quoted 38 lines · Jan Beulich
Quoted 24 lines · Jan Beulich
Quoted 23 lines · Juergen Gross
Quoted 9 lines · Jan Beulich
v->processorgets latched intost->processorbefore raising the softirq,
but can't the vCPU be moved elsewhere by the time the softirq handler
actually gains control? If that's not possible (and if it's not obvious
why, and as you can see it's not obvious to me), then I think a code
comment wants to be added there.You are right, it might be possible for the vcpu to move around.
OTOH is it really important to run the target vcpu exactly on the cpu
it is executing (or has last executed) at the time the NMI/MCE is being
queued? This is in no way related to the cpu the MCE or NMI has been
happening on. It is just a random cpu, and so it would be if we'd do the
cpu selection when the softirq handler is running.Quoted 8 lines
Quoted 6 lines
One question to understand the idea nehind all that: _why_ is the vcpu
pinned until it does an iret? I could understand if it would be pinned
to the cpu where the NMI/MCE was happening, but this is not the case.Then it was never finished or got broken, I would guess.
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning
up other code. So there doesn't seem to be any such requirement anymore.Quoted 6 lines
Quoted 2 lines
So just to be sure: you are fine for me removing the pinning for NMIs?
No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).
Quoted 5 lines
I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always
to CPU0?Quoted 6 lines
Its (allegedly) for when dom0 knows some system-specific way of getting extra information out of the platform, that happens to be core-specific.
There are rare cases where SMI's need to be executed on CPU0, and I wouldn't put it past hardware designers to have similar aspects for NMIs.
Understood. But today vcpu0 is _not_ bound to CPU0, but to any cpu it happened to run on.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
That said, as soon as the gaping security hole which is the default-readibility of all MSRs, I bet the utility of this pinning mechanism will be 0.
And my reasoning is that this is the case today already, as there is no pinning to CPU0 done, at least not on purpose.
Based on this analysis, I'd be tempted to drop the pinning completely. It clearly isn't working in a rational way.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
In reply to Juergen GrossQuoted 8 lines · Juergen Gross
Quoted 7 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
Quoted 2 lines · Jan Beulich
Oh, no. The #MC side use has gone away in
3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.So just to be sure: you are fine for me removing the pinning for NMIs?
Quoted 8 lines
Quoted 3 lines
No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).
I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?
Hmm, I've been confused. To me it doesn't make sense to force affinity other than to CPU0 for an NMI. Hence I was mis-reading the original send_guest_trap() invocation (in particular its middle argument). Looking at commit 355b0469a8 ("x86: MCA support") I would guess the affinity change for NMI delivery was done by mistake, or at best just to not distinguish NMI from #MC.
As a result, considering that before that there was no affinity change at all, and considering further than changing to other than CPU0 doesn't really make sense, but we got away with not doing so for so many years, I think I agree that it can be ripped out altogether. Personally I'd prefer a fix to make it use CPU0, but I'm in no way going to insist.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
In reply to Juergen GrossQuoted 11 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to
a single physical cpu:- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handlingEach of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
Quoted 8 lines
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while
vcpu_pin_override()is allowed to fail.So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).
Right. And, after this patch, all the three cases above and suspend/resume, all use cpu_hard_affinity_saved.
Can you add a paragraph, either here in the changelog, on in a comment (e.g., at the top or inside of vcpu_set_tmp_affinity()), about why this is all fine?
Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios
Quoted 20 lines · Dario Faggioli
Quoted 11 lines · Juergen Gross
Today there are three scenarios which are pinning vcpus temporarily to
a single physical cpu:- NMI/MCE injection into PV domains
-wait_event()handling
-vcpu_pin_override()handlingEach of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.
Quoted 8 lines
The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while
vcpu_pin_override()is allowed to fail.So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).
Quoted 6 lines
Right. And, after this patch, all the three cases above and suspend/resume, all use
cpu_hard_affinity_saved.Can you add a paragraph, either here in the changelog, on in a comment (e.g., at the top or inside of
vcpu_set_tmp_affinity()), about why this is all fine?
Yes.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel