xen: add core scheduling support
RFC PATCH v2 33/45 xen/sched: add code to sync scheduling of all vcpus of a sched item
In reply to Juergen GrossWhen switching sched items synchronize all vcpus of the new item to be scheduled at the same time.
A variable sched_granularity is added which holds the number of vcpus per schedule item.
As tasklets require to schedule the idle item it is required to set the tasklet_work_scheduled parameter of do_schedule() to true if any cpu covered by the current schedule() call has any pending tasklet work.
For joining other vcpus of the schedule item we need to add a new softirq SCHED_SLAVE_SOFTIRQ in order to have a way to initiate a context switch without calling the generic schedule() function selecting the vcpu to switch to, as we already know which vcpu we want to run. This has the other advantage not to loose any other concurrent SCHEDULE_SOFTIRQ events.
---
RFC V2: move syncing after context_switch() to schedule.c
--- xen/arch/arm/domain.c | 2 +- xen/arch/x86/domain.c | 3 +- xen/common/schedule.c | 339 +++++++++++++++++++++++++++++++++------------ xen/common/softirq.c | 6 +- xen/include/xen/sched-if.h | 1 + xen/include/xen/sched.h | 16 ++- xen/include/xen/softirq.h | 1 + 7 files changed, 269 insertions(+), 99 deletions(-)
xen/arch/arm/domain.c +1 -1
@@ -311,7 +311,7 @@ static void schedule_tail(struct vcpu *prev)local_irq_enable (); -context_saved (prev); +sched_context_switched (prev, current);if ( prev != current )update_runstate_area (current);
xen/arch/x86/domain.c +1 -2
@@ -1714,7 +1714,6 @@ static void __context_switch(void)per_cpu (curr_vcpu, cpu) = n; } -void context_switch (struct vcpu *prev,struct vcpu *next) {unsigned int cpu =smp_processor_id ();
@@ -1792,7 +1791,7 @@ void context_switch(struct vcpu *prev, struct vcpu *next) } } -context_saved (prev); +sched_context_switched (prev, next);if ( prev != next ) {
xen/common/schedule.c +251 -88
403 lines not on this page · raw message
xen/common/softirq.c +3 -3
@@ -33,8 +33,8 @@ static void __do_softirq(unsigned long ignore_mask)for ( ; ; ) {/* -* Initialise @cpu on every iteration: SCHEDULE_SOFTIRQ may move -* us to another processor. +* Initialise @cpu on every iteration: SCHEDULE_SOFTIRQ or +* SCHED_SLAVE_SOFTIRQ may move us to another processor. */ cpu =smp_processor_id ();
@@ -55,7 +55,7 @@ void process_pending_softirqs(void) {ASSERT (!in_irq () &&local_irq_is_enabled ());/* Do not enter scheduler as it can preempt the calling context. */ -__do_softirq (1ul <<SCHEDULE_SOFTIRQ ); +__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ )); }void do_softirq (void )
xen/include/xen/sched-if.h +1 -0
@@ -41,6 +41,7 @@ struct sched_resource {struct timer s_timer;/* scheduling timer */ atomic_t urgent_count;/* how many urgent vcpus */ unsigned processor; +const cpumask_t *cpus;/* cpus covered by this struct */ };#define curr_on_cpu (c) (per_cpu(sched_res, c)->curr)
xen/include/xen/sched.h +11 -5
@@ -295,6 +295,12 @@ struct sched_item {/* Next item to run. */ struct sched_item *next_task;s_time_t next_time; + +/* Number of vcpus not yet joined for context switch. */ +unsigned int rendezvous_in_cnt; + +/* Number of vcpus not yet finished with context switch. */ +atomic_t rendezvous_out_cnt; };#define for_each_sched_item (d, e) \
@@ -698,10 +704,10 @@ void sync_local_execstate(void);/* * Called by the scheduler to switch to another VCPU. This function must -* call context_saved(@prev) when the local CPU is no longer running in -* @prev's context, and that context is saved to memory. Alternatively, if -* implementing lazy context switching, it suffices to ensure that invoking -* sync_vcpu_execstate() will switch and commit @prev's state. +* call sched_context_switched(@prev, @next) when the local CPU is no longer +* running in @prev's context, and that context is saved to memory. +* Alternatively, if implementing lazy context switching, it suffices to ensure +* that invoking sync_vcpu_execstate() will switch and commit @prev's state. */ void context_switch( struct vcpu *prev,
@@ -713,7 +719,7 @@ void context_switch(* saved to memory. Alternatively, if implementing lazy context switching, * ensure that invoking sync_vcpu_execstate() will switch and commit @prev. */ -void context_saved (struct vcpu *prev); +void sched_context_switched (struct vcpu *prev,struct vcpu *vnext);/* Called by the scheduler to continue running the current VCPU. */ void continue_running(
xen/include/xen/softirq.h +1 -0
@@ -4,6 +4,7 @@/* Low-latency softirqs come first in the following list. */ enum {TIMER_SOFTIRQ =0 , +SCHED_SLAVE_SOFTIRQ ,SCHEDULE_SOFTIRQ ,NEW_TLBFLUSH_CLOCK_PERIOD_SOFTIRQ ,RCU_SOFTIRQ ,
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 34/45 xen/sched: introduce item_runnable_state()
In reply to Juergen GrossToday the vcpu runstate of a new scheduled vcpu is always set to "running" even if at that time vcpu_runnable() is already returning false due to a race (e.g. with pausing the vcpu).
With core scheduling this can no longer work as not all vcpus of a schedule item have to be "running" when being scheduled. So the vcpu's new runstate has to be selected at the same time as the runnability of the related schedule item is probed.
For this purpose introduce a new helper item_runnable_state() which will save the new runstate of all tested vcpus in a new field of the vcpu struct.
---
RFC V2: new patch
--- xen/common/domain.c | 1 + xen/common/sched_arinc653.c | 2 +- xen/common/sched_credit.c | 49 ++++++++++++++++++++++++--------------------- xen/common/sched_credit2.c | 7 ++++--- xen/common/sched_null.c | 3 ++- xen/common/sched_rt.c | 8 +++++++- xen/common/schedule.c | 2 +- xen/include/xen/sched-if.h | 14 +++++++++++++ xen/include/xen/sched.h | 1 + 9 files changed, 57 insertions(+), 30 deletions(-)
xen/common/domain.c +1 -0
@@ -151,6 +151,7 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)if (is_idle_domain (d) ) { v->runstate.state = RUNSTATE_running; + v->new_state = RUNSTATE_running; }else {
xen/common/sched_arinc653.c +1 -1
@@ -557,7 +557,7 @@ a653sched_do_schedule(if ( !((new_task !=NULL ) && (AITEM (new_task) !=NULL ) &&AITEM (new_task)->awake - &&item_runnable (new_task)) ) + &&item_runnable_state (new_task)) ) new_task =IDLETASK (cpu);BUG_ON (new_task ==NULL );
xen/common/sched_credit.c +26 -23
66 lines not on this page · raw message
xen/common/sched_credit2.c +4 -3
@@ -3288,7 +3288,7 @@ runq_candidate(struct csched2_runqueue_data *rqd,* In fact, it may be the case that scurr is about to spin, and there's * no point forcing it to do so until rate limiting expires. */ -if ( !yield && prv->ratelimit_us &&item_runnable (scurr->item) && +if ( !yield && prv->ratelimit_us &&item_runnable_state (scurr->item) && (now - scurr->item->state_entry_time) <MICROSECS (prv->ratelimit_us) ) {if (unlikely (tb_init_done) )
@@ -3342,7 +3342,7 @@ runq_candidate(struct csched2_runqueue_data *rqd,* * Of course, we also default to idle also if scurr is not runnable. */ -if (item_runnable (scurr->item) && !soft_aff_preempt ) +if (item_runnable_state (scurr->item) && !soft_aff_preempt ) snext = scurr;else snext =csched2_item (sched_idle_item (cpu));
@@ -3402,7 +3402,8 @@ runq_candidate(struct csched2_runqueue_data *rqd,* some budget, then choose it. */ if ( (yield || svc->credit > snext->credit) && - (!has_cap (svc) ||item_grab_budget (svc)) ) + (!has_cap (svc) ||item_grab_budget (svc)) && +item_runnable_state (svc->item) ) snext = svc;/* In any case, if we got this far, break. */
xen/common/sched_null.c +2 -1
@@ -784,7 +784,8 @@ static void null_schedule(const struct scheduler *ops, struct sched_item *prev,spin_unlock (&prv->waitq_lock); } -if (unlikely (prev->next_task ==NULL || !item_runnable (prev->next_task)) ) +if (unlikely (prev->next_task ==NULL || + !item_runnable_state (prev->next_task)) ) prev->next_task =sched_idle_item (sched_cpu);NULL_ITEM_CHECK (prev->next_task);
xen/common/sched_rt.c +7 -1
@@ -1101,12 +1101,18 @@ rt_schedule(const struct scheduler *ops, struct sched_item *curritem,else { snext =runq_pick (ops,cpumask_of (sched_cpu)); +if ( snext ==NULL ) snext =rt_item (sched_idle_item (sched_cpu)); +else if ( !item_runnable_state (snext->item) ) + { +q_remove (snext); + snext =rt_item (sched_idle_item (sched_cpu)); + }/* if scurr has higher priority and budget, still pick scurr */ if ( !is_idle_item (curritem) && -item_runnable (curritem) && +item_runnable_state (curritem) && scurr->cur_budget >0 && (is_idle_item (snext->item) ||compare_item_priority (scurr, snext) >0 ) )
xen/common/schedule.c +1 -1
@@ -206,7 +206,7 @@ static inline void sched_item_runstate_change(struct sched_item *item,struct vcpu *v = item->vcpu;if ( running ) -vcpu_runstate_change (v, RUNSTATE_running, new_entry_time); +vcpu_runstate_change (v, v->new_state, new_entry_time);else vcpu_runstate_change(v, ((v->pause_flags & VPF_blocked) ? RUNSTATE_blocked :
xen/include/xen/sched-if.h +14 -0
@@ -60,6 +60,20 @@ static inline bool item_runnable(const struct sched_item *item)return vcpu_runnable (item->vcpu); } +static inline bool item_runnable_state (const struct sched_item *item) +{ +struct vcpu *v; +bool runnable; + + v = item->vcpu; + runnable =vcpu_runnable (v); + + v->new_state = runnable ? RUNSTATE_running + : (v->pause_flags & VPF_blocked) + ? RUNSTATE_blocked : RUNSTATE_offline; +return runnable; +} +static inline void sched_set_res (struct sched_item *item,struct sched_resource *res) {
xen/include/xen/sched.h +1 -0
@@ -174,6 +174,7 @@ struct vcpu XEN_GUEST_HANDLE(vcpu_runstate_info_compat_t) compat; } runstate_guest;/* guest address */ #endif +int new_state;/* Has the FPU been initialised? */ bool fpu_initialised;
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 35/45 xen/sched: add support for multiple vcpus per sched item where missing
In reply to Juergen GrossIn several places there is support for multiple vcpus per sched item missing. Add that missing support (with the exception of initial allocation) and missing helpers for that.
---
RFC V2: fix vcpu_runstate_helper()
--- xen/common/schedule.c | 26 ++++++++-------- xen/include/xen/sched-if.h | 74 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 73 insertions(+), 27 deletions(-)
xen/common/schedule.c +14 -12
@@ -180,8 +180,9 @@ static inline void vcpu_runstate_change(s_time_t delta;struct sched_item *item = v->sched_item; -ASSERT (v->runstate.state != new_state);ASSERT (spin_is_locked (per_cpu (sched_res, v->processor)->schedule_lock)); +if ( v->runstate.state == new_state ) +return ;vcpu_urgent_count_update (v);
@@ -203,15 +204,16 @@ static inline void vcpu_runstate_change(static inline void sched_item_runstate_change (struct sched_item *item,bool running,s_time_t new_entry_time) { -struct vcpu *v = item->vcpu; +struct vcpu *v; -if ( running ) -vcpu_runstate_change (v, v->new_state, new_entry_time); -else -vcpu_runstate_change (v, - ((v->pause_flags & VPF_blocked) ? RUNSTATE_blocked : - (vcpu_runnable (v) ? RUNSTATE_runnable : RUNSTATE_offline)), - new_entry_time); +for_each_sched_item_vcpu ( item, v ) +if ( running ) +vcpu_runstate_change (v, v->new_state, new_entry_time); +else +vcpu_runstate_change (v, + ((v->pause_flags & VPF_blocked) ? RUNSTATE_blocked : + (vcpu_runnable (v) ? RUNSTATE_runnable : RUNSTATE_offline)), + new_entry_time); }void vcpu_runstate_get (struct vcpu *v,struct vcpu_runstate_info *runstate)
@@ -1580,7 +1582,7 @@ static void sched_switch_items(struct sched_resource *sd, (next->vcpu->runstate.state == RUNSTATE_runnable) ? (now - next->state_entry_time) : 0, prev->next_time); -ASSERT (prev->vcpu->runstate.state == RUNSTATE_running); +ASSERT (item_running (prev));TRACE_4D (TRC_SCHED_SWITCH , prev->domain->domain_id, prev->item_id, next->domain->domain_id, next->item_id);
@@ -1588,7 +1590,7 @@ static void sched_switch_items(struct sched_resource *sd,sched_item_runstate_change (prev, false, now); prev->last_run_time = now; -ASSERT (next->vcpu->runstate.state != RUNSTATE_running); +ASSERT (!item_running (next));sched_item_runstate_change (next, true, now);/*
@@ -1703,7 +1705,7 @@ void sched_context_switched(struct vcpu *vprev, struct vcpu *vnext)while (atomic_read (&next->rendezvous_out_cnt) )cpu_relax (); } -else if ( vprev != vnext ) +else if ( vprev != vnext && sched_granularity ==1 )context_saved (vprev); }
xen/include/xen/sched-if.h +59 -15
125 lines not on this page · raw message
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 36/45 x86: make loading of GDT at context switch more modular
In reply to Juergen GrossIn preparation for core scheduling carve out the GDT related functionality (writing GDT related PTEs, loading default of full GDT) into sub-functions.
---
RFC V2: split off non-refactoring part
--- xen/arch/x86/domain.c | 57 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 22 deletions(-)
xen/arch/x86/domain.c +35 -22
@@ -1619,6 +1619,37 @@ static inline bool need_full_gdt(const struct domain *d)return is_pv_domain (d) && !is_idle_domain (d); } +static inline void write_full_gdt_ptes (seg_desc_t *gdt,struct vcpu *v) +{ +unsigned long mfn =virt_to_mfn (gdt); +l1_pgentry_t *pl1e =pv_gdt_ptes (v); +unsigned int i; + +for ( i =0 ; i <NR_RESERVED_GDT_PAGES ; i++ ) +l1e_write (pl1e +FIRST_RESERVED_GDT_PAGE + i, +l1e_from_pfn (mfn + i, __PAGE_HYPERVISOR_RW)); +} + +static inline void load_full_gdt (struct vcpu *v,unsigned int cpu) +{ +struct desc_ptr gdt_desc; + + gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; + gdt_desc.base =GDT_VIRT_START (v); + +lgdt (&gdt_desc); +} + +static inline void load_default_gdt (seg_desc_t *gdt,unsigned int cpu) +{ +struct desc_ptr gdt_desc; + + gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; + gdt_desc.base = (unsigned long )(gdt -FIRST_RESERVED_GDT_ENTRY ); + +lgdt (&gdt_desc); +} +static void __context_switch (void ) {struct cpu_user_regs *stack_regs =guest_cpu_user_regs ();
@@ -1627,7 +1658,6 @@ static void __context_switch(void)struct vcpu *n = current;struct domain *pd = p->domain, *nd = n->domain;seg_desc_t *gdt; -struct desc_ptr gdt_desc;ASSERT (p != n);ASSERT (!vcpu_cpu_dirty (n));
@@ -1669,25 +1699,13 @@ static void __context_switch(void) gdt = !is_pv_32bit_domain (nd) ?per_cpu (gdt_table, cpu) :per_cpu (compat_gdt_table, cpu); -if (need_full_gdt (nd) ) - { -unsigned long mfn =virt_to_mfn (gdt); -l1_pgentry_t *pl1e =pv_gdt_ptes (n); -unsigned int i; -for ( i =0 ; i <NR_RESERVED_GDT_PAGES ; i++ ) -l1e_write (pl1e +FIRST_RESERVED_GDT_PAGE + i, -l1e_from_pfn (mfn + i, __PAGE_HYPERVISOR_RW)); - } +if (need_full_gdt (nd) ) +write_full_gdt_ptes (gdt, n);if (need_full_gdt (pd) && ((p->vcpu_id != n->vcpu_id) || !need_full_gdt (nd)) ) - { - gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; - gdt_desc.base = (unsigned long )(gdt -FIRST_RESERVED_GDT_ENTRY ); - -lgdt (&gdt_desc); - } +load_default_gdt (gdt, cpu);write_ptbase (n);
@@ -1700,12 +1718,7 @@ static void __context_switch(void)if (need_full_gdt (nd) && ((p->vcpu_id != n->vcpu_id) || !need_full_gdt (pd)) ) - { - gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; - gdt_desc.base =GDT_VIRT_START (n); - -lgdt (&gdt_desc); - } +load_full_gdt (n, cpu);if ( pd != nd )cpumask_clear_cpu (cpu, pd->dirty_cpumask);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 36/45] x86: make loading of GDT at context switch more modular
Quoted 14 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +7
-0@@ -1619,6 +1619,37 @@ static inline bool need_full_gdt(const struct domain *d)return is_pv_domain (d) && !is_idle_domain (d); } +static inline void write_full_gdt_ptes (seg_desc_t *gdt,struct vcpu *v) +{ +unsigned long mfn =virt_to_mfn (gdt); +l1_pgentry_t *pl1e =pv_gdt_ptes (v); +unsigned int i; + +for ( i =0 ; i <NR_RESERVED_GDT_PAGES ; i++ )
xen/arch/x86/domain.c static inline bool need_full_gdt(const struct domain *d) · line 1638
Quoted 10 lines
xen/arch/x86/domain.c +10
-0+l1e_write (pl1e +FIRST_RESERVED_GDT_PAGE + i, +l1e_from_pfn (mfn + i, __PAGE_HYPERVISOR_RW)); +} + +static inline void load_full_gdt (struct vcpu *v,unsigned int cpu) +{ +struct desc_ptr gdt_desc; + + gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; + gdt_desc.base =GDT_VIRT_START (v);
Make this and ...
xen/arch/x86/domain.c static inline bool need_full_gdt(const struct domain *d) · line 1648
Quoted 9 lines
xen/arch/x86/domain.c +9
-0+lgdt (&gdt_desc); +} + +static inline void load_default_gdt (seg_desc_t *gdt,unsigned int cpu) +{ +struct desc_ptr gdt_desc; + + gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; + gdt_desc.base = (unsigned long )(gdt -FIRST_RESERVED_GDT_ENTRY );
... the variables' initializers? With this and with const suitably applied to all function parameters where this is possible
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH RFC V2 36/45] x86: make loading of GDT at context switch more modular
Quoted 39 lines · Jan Beulich
Quoted 14 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +7
-0@@ -1619,6 +1619,37 @@ static inline bool need_full_gdt(const struct domain *d)return is_pv_domain (d) && !is_idle_domain (d); } +static inline void write_full_gdt_ptes (seg_desc_t *gdt,struct vcpu *v) +{ +unsigned long mfn =virt_to_mfn (gdt); +l1_pgentry_t *pl1e =pv_gdt_ptes (v); +unsigned int i; + +for ( i =0 ; i <NR_RESERVED_GDT_PAGES ; i++ )xen/arch/x86/domain.c static inline bool need_full_gdt(const struct domain *d) · line 1638
Quoted 10 lines
xen/arch/x86/domain.c +10
-0+l1e_write (pl1e +FIRST_RESERVED_GDT_PAGE + i, +l1e_from_pfn (mfn + i, __PAGE_HYPERVISOR_RW)); +} + +static inline void load_full_gdt (struct vcpu *v,unsigned int cpu) +{ +struct desc_ptr gdt_desc; + + gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; + gdt_desc.base =GDT_VIRT_START (v);Make this and ...
xen/arch/x86/domain.c static inline bool need_full_gdt(const struct domain *d) · line 1648
Quoted 9 lines
xen/arch/x86/domain.c +9
-0+lgdt (&gdt_desc); +} + +static inline void load_default_gdt (seg_desc_t *gdt,unsigned int cpu) +{ +struct desc_ptr gdt_desc; + + gdt_desc.limit =LAST_RESERVED_GDT_BYTE ; + gdt_desc.base = (unsigned long )(gdt -FIRST_RESERVED_GDT_ENTRY );... the variables' initializers? With this and with const suitably applied to all function parameters where this is possible
Quoted 1 line
Acked-by Jan Beulich <[email protected]>
Thanks, will do.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 37/45 x86: optimize loading of GDT at context switch
In reply to Juergen GrossInstead of dynamically decide whether the previous vcpu was using full or default GDT just add a percpu variable for that purpose. This at once removes the need for testing vcpu_ids to differ twice.
Cache the need_full_gdt(nd) value in a local variable.
---
RFC V2: new patch (split from previous one)
--- xen/arch/x86/domain.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
xen/arch/x86/domain.c +13 -5
@@ -72,6 +72,8 @@ DEFINE_PER_CPU(struct vcpu *, curr_vcpu); +static DEFINE_PER_CPU(bool, full_gdt_loaded); +static void default_idle (void );void (*pm_idle) (void )__read_mostly = default_idle;void (*dead_idle) (void )__read_mostly = default_dead_idle;
@@ -1638,6 +1640,8 @@ static inline void load_full_gdt(struct vcpu *v, unsigned int cpu) gdt_desc.base =GDT_VIRT_START (v);lgdt (&gdt_desc); + +per_cpu (full_gdt_loaded, cpu) = true; }static inline void load_default_gdt (seg_desc_t *gdt,unsigned int cpu)
@@ -1648,6 +1652,8 @@ static inline void load_default_gdt(seg_desc_t *gdt, unsigned int cpu) gdt_desc.base = (unsigned long )(gdt -FIRST_RESERVED_GDT_ENTRY );lgdt (&gdt_desc); + +per_cpu (full_gdt_loaded, cpu) = false; }static void __context_switch (void )
@@ -1658,6 +1664,7 @@ static void __context_switch(void)struct vcpu *n = current;struct domain *pd = p->domain, *nd = n->domain;seg_desc_t *gdt; +bool need_full_gdt_n;ASSERT (p != n);ASSERT (!vcpu_cpu_dirty (n));
@@ -1700,11 +1707,13 @@ static void __context_switch(void) gdt = !is_pv_32bit_domain (nd) ?per_cpu (gdt_table, cpu) :per_cpu (compat_gdt_table, cpu); -if (need_full_gdt (nd) ) + need_full_gdt_n =need_full_gdt (nd); + +if ( need_full_gdt_n )write_full_gdt_ptes (gdt, n); -if (need_full_gdt (pd) && - ((p->vcpu_id != n->vcpu_id) || !need_full_gdt (nd)) ) +if (per_cpu (full_gdt_loaded, cpu) && + ((p->vcpu_id != n->vcpu_id) || !need_full_gdt_n) )load_default_gdt (gdt, cpu);write_ptbase (n);
@@ -1716,8 +1725,7 @@ static void __context_switch(void)svm_load_segs (0 ,0 ,0 ,0 ,0 ,0 ,0 );#endif -if (need_full_gdt (nd) && - ((p->vcpu_id != n->vcpu_id) || !need_full_gdt (pd)) ) +if ( need_full_gdt_n && !per_cpu (full_gdt_loaded, cpu) )load_full_gdt (n, cpu);if ( pd != nd )
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 37/45] x86: optimize loading of GDT at context switch
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
Instead of dynamically decide whether the previous vcpu was using full or default GDT just add a percpu variable for that purpose. This at once removes the need for testing
vcpu_idsto differ twice.Cache the
need_full_gdt(nd)value in a local variable.Signed-off-by Juergen Gross <[email protected]>
To be honest I'm not entirely convinced this is a good move. But since you've done the work, and since the larger source size is hopefully outweighed by slightly smaller binary size (per-CPU data accesses aren't entirely cheap either), I'm not going to object.
xen/arch/x86/domain.c static void __context_switch(void) · line 1667
Quoted 5 lines
xen/arch/x86/domain.c +1
-0@@ -1658,6 +1664,7 @@ static void __context_switch(void)struct vcpu *n = current;struct domain *pd = p->domain, *nd = n->domain;seg_desc_t *gdt; +bool need_full_gdt_n;
This variable is too long, or more precisely has too many underscores for my taste. Seeing that only a single invocation of need_full_gdt() remains, I don't think just "full_gdt" would be ambiguous in any way. At which point
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH RFC V2 37/45] x86: optimize loading of GDT at context switch
Quoted 15 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
Instead of dynamically decide whether the previous vcpu was using full or default GDT just add a percpu variable for that purpose. This at once removes the need for testing
vcpu_idsto differ twice.Cache the
need_full_gdt(nd)value in a local variable.Signed-off-by Juergen Gross <[email protected]>To be honest I'm not entirely convinced this is a good move. But since you've done the work, and since the larger source size is hopefully outweighed by slightly smaller binary size (per-CPU data accesses aren't entirely cheap either), I'm not going to object.
Quoted 9 lines
xen/arch/x86/domain.c static void __context_switch(void) · line 1667
Quoted 5 lines
xen/arch/x86/domain.c +1
-0@@ -1658,6 +1664,7 @@ static void __context_switch(void)struct vcpu *n = current;struct domain *pd = p->domain, *nd = n->domain;seg_desc_t *gdt; +bool need_full_gdt_n;This variable is too long, or more precisely has too many underscores for my taste. Seeing that only a single invocation of
need_full_gdt()remains, I don't think just "full_gdt" would be ambiguous in any way.
Fine with me.
Quoted 2 lines
At which point
Reviewed-by Jan Beulich <[email protected]>
Thanks,
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 38/45 xen/sched: modify cpupool_domain_cpumask() to be an item mask
In reply to Juergen Grosscpupool_domain_cpumask() is used by scheduling to select cpus or to iterate over cpus. In order to support scheduling items spanning multiple cpus let cpupool_domain_cpumask() return a cpumask with only one bit set per scheduling resource.
--- xen/common/cpupool.c | 30 +++++++++++++++++++++--------- xen/common/schedule.c | 5 +++-- xen/include/xen/sched-if.h | 5 ++++- 3 files changed, 28 insertions(+), 12 deletions(-)
xen/common/cpupool.c +21 -9
@@ -38,26 +38,35 @@ DEFINE_PER_CPU(struct cpupool *, cpupool); #define cpupool_dprintk(x...) ((void)0) +static void free_cpupool_struct (struct cpupool *c) +{ +if ( c ) + { +free_cpumask_var (c->res_valid); +free_cpumask_var (c->cpu_valid); + } +xfree (c); +} +static struct cpupool *alloc_cpupool_struct (void ) { struct cpupool *c = xzalloc(struct cpupool); -if ( !c || !zalloc_cpumask_var (&c->cpu_valid) ) +if ( !c ) +return NULL ; + +zalloc_cpumask_var (&c->cpu_valid); +zalloc_cpumask_var (&c->res_valid); + +if ( !c->cpu_valid || !c->res_valid ) { -xfree (c); +free_cpupool_struct (c); c =NULL ; }return c; } -static void free_cpupool_struct (struct cpupool *c) -{ -if ( c ) -free_cpumask_var (c->cpu_valid); -xfree (c); -} -/* * find a cpupool by it's id. to be called with cpupool lock held * if exact is not specified, the first cpupool with an id larger or equal to
@@ -271,6 +280,7 @@ static int cpupool_assign_cpu_locked(struct cpupool *c, unsigned int cpu) cpupool_cpu_moving =NULL ; }cpumask_set_cpu (cpu, c->cpu_valid); +cpumask_and (c->res_valid, c->cpu_valid, sched_res_mask);rcu_read_lock (&domlist_read_lock);for_each_domain_in_cpupool (d, c)
@@ -393,6 +403,7 @@ static int cpupool_unassign_cpu(struct cpupool *c, unsigned int cpu)atomic_inc (&c->refcnt); cpupool_cpu_moving = c;cpumask_clear_cpu (cpu, c->cpu_valid); +cpumask_and (c->res_valid, c->cpu_valid, sched_res_mask);spin_unlock (&cpupool_lock); work_cpu =smp_processor_id ();
@@ -509,6 +520,7 @@ static int cpupool_cpu_remove(unsigned int cpu)* allowed only for CPUs in pool0. */ cpumask_clear_cpu (cpu, cpupool0->cpu_valid); +cpumask_and (cpupool0->res_valid, cpupool0->cpu_valid, sched_res_mask); ret =0 ; }
xen/common/schedule.c +3 -2
@@ -57,6 +57,7 @@ integer_param("sched_ratelimit_us", sched_ratelimit_us);/* Number of vcpus per struct sched_item. */ static unsigned int sched_granularity =1 ; +const cpumask_t *sched_res_mask = &cpumask_all;/* Various timer handlers. */ static void s_timer_fn (void *unused);
@@ -347,9 +348,9 @@ static unsigned int sched_select_initial_cpu(struct vcpu *v)cpumask_clear (&cpus); for_each_node_mask ( node, d->node_affinity )cpumask_or (&cpus, &cpus, &node_to_cpumask (node)); -cpumask_and (&cpus, &cpus,cpupool_domain_cpumask (d)); +cpumask_and (&cpus, &cpus, d->cpupool->cpu_valid);if (cpumask_empty (&cpus) ) -cpumask_copy (&cpus,cpupool_domain_cpumask (d)); +cpumask_copy (&cpus, d->cpupool->cpu_valid);if ( v->vcpu_id ==0 )return cpumask_first (&cpus);
xen/include/xen/sched-if.h +4 -1
@@ -22,6 +22,8 @@ extern cpumask_t cpupool_free_cpus;#define SCHED_DEFAULT_RATELIMIT_US 1000extern int sched_ratelimit_us; +/* Scheduling resource mask. */ +extern const cpumask_t *sched_res_mask;/* * In order to allow a scheduler to remap the lock->cpu mapping,
@@ -527,6 +529,7 @@ struct cpupool {int cpupool_id;cpumask_var_t cpu_valid;/* all cpus assigned to pool */ +cpumask_var_t res_valid;/* all scheduling resources of pool */ struct cpupool *next;unsigned int n_dom;struct scheduler *sched;
@@ -543,7 +546,7 @@ static inline cpumask_t* cpupool_domain_cpumask(struct domain *d)* be interested in calling this for the idle domain. */ ASSERT (d->cpupool !=NULL ); -return d->cpupool->cpu_valid; +return d->cpupool->res_valid; }/*
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 39/45 xen/sched: support allocating multiple vcpus into one sched item
In reply to Juergen GrossWith a scheduling granularity greater than 1 multiple vcpus share the same struct sched_item. Support that.
Setting the initial processor must be done carefully: we can't use sched_set_res() as that relies on for_each_sched_item_vcpu() which in turn needs the vcpu already as a member of the domain's vcpu linked list, which isn't the case.
--- xen/common/schedule.c | 75 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 62 insertions(+), 13 deletions(-)
xen/common/schedule.c +62 -13
@@ -275,10 +275,25 @@ static void sched_spin_unlock_double(spinlock_t *lock1, spinlock_t *lock2,spin_unlock_irqrestore (lock1, flags); } -static void sched_free_item (struct sched_item *item) +static void sched_free_item (struct sched_item *item,struct vcpu *v) {struct sched_item *prev_item;struct domain *d = item->domain; +struct vcpu *vitem; +unsigned int cnt =0 ; + +/* Don't count to be released vcpu, might be not in vcpu list yet. */ + for_each_sched_item_vcpu ( item, vitem ) +if ( vitem != v ) + cnt++; + + v->sched_item =NULL ; + +if ( cnt ) +return ; + +if ( item->vcpu == v ) + item->vcpu = v->next_in_list;if ( d->sched_item_list == item ) d->sched_item_list = item->next_in_list;
@@ -294,8 +309,6 @@ static void sched_free_item(struct sched_item *item) } } - item->vcpu->sched_item =NULL ; -free_cpumask_var (item->cpu_hard_affinity);free_cpumask_var (item->cpu_hard_affinity_tmp);free_cpumask_var (item->cpu_hard_affinity_saved);
@@ -304,17 +317,36 @@ static void sched_free_item(struct sched_item *item)xfree (item); } +static void sched_item_add_vcpu (struct sched_item *item,struct vcpu *v) +{ + v->sched_item = item; +if ( !item->vcpu || item->vcpu->vcpu_id > v->vcpu_id ) + { + item->vcpu = v; + item->item_id = v->vcpu_id; + } +} +static struct sched_item *sched_alloc_item (struct vcpu *v) {struct sched_item *item, **prev_item;struct domain *d = v->domain; + for_each_sched_item ( d, item ) +if ( item->vcpu->vcpu_id / sched_granularity == + v->vcpu_id / sched_granularity ) +break ; + +if ( item ) + { +sched_item_add_vcpu (item, v); +return item; + } + if ( (item = xzalloc(struct sched_item)) == NULL )return NULL ; - v->sched_item = item; - item->vcpu = v; - item->item_id = v->vcpu_id; +sched_item_add_vcpu (item, v); item->domain = d; for ( prev_item = &d->sched_item_list; *prev_item;
@@ -335,7 +367,7 @@ static struct sched_item *sched_alloc_item(struct vcpu *v)return item; fail: -sched_free_item (item); +sched_free_item (item, v);return NULL ; }
@@ -377,8 +409,6 @@ int sched_init_vcpu(struct vcpu *v)else processor =sched_select_initial_cpu (v); -sched_set_res (item,per_cpu (sched_res, processor)); -/* Initialise the per-vcpu timers. */ init_timer (&v->periodic_timer, vcpu_periodic_timer_fn, v, v->processor);
@@ -387,10 +417,22 @@ int sched_init_vcpu(struct vcpu *v)init_timer (&v->poll_timer, poll_timer_fn, v, v->processor); +/* If this is not the first vcpu of the item we are done. */ +if ( item->priv !=NULL ) + { +/* We can rely on previous vcpu to exist. */ + v->processor =cpumask_next (d->vcpu[v->vcpu_id -1 ]->processor, + item->res->cpus); +return 0 ; + } + +/* The first vcpu of an item can be set via sched_set_res(). */ +sched_set_res (item,per_cpu (sched_res, processor)); + item->priv =sched_alloc_vdata (dom_scheduler (d), item, d->sched_priv);if ( item->priv ==NULL ) { -sched_free_item (item); +sched_free_item (item, v);return 1 ; }
@@ -544,9 +586,16 @@ void sched_destroy_vcpu(struct vcpu *v)kill_timer (&v->poll_timer);if (test_and_clear_bool (v->is_urgent) )atomic_dec (&per_cpu (sched_res, v->processor)->urgent_count); -sched_remove_item (vcpu_scheduler (v), item); -sched_free_vdata (vcpu_scheduler (v), item->priv); -sched_free_item (item); +/* +* Vcpus are being destroyed top-down. So being the first vcpu of an item +* is the same as being the only one. +*/ +if ( item->vcpu == v ) + { +sched_remove_item (vcpu_scheduler (v), item); +sched_free_vdata (vcpu_scheduler (v), item->priv); +sched_free_item (item, v); + } }int sched_init_domain (struct domain *d,int poolid)
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 40/45 xen/sched: add a scheduler_percpu_init() function
In reply to Juergen GrossFor support of core scheduling the scheduler cpu callback for CPU_STARTING has to be moved into a dedicated function called by start_secondary() as it needs to run before spin_debug_enable() then due to potentially calling xfree().
---
RFC V2: fix ARM build
--- xen/arch/arm/smpboot.c | 2 ++ xen/arch/x86/smpboot.c | 2 ++ xen/common/schedule.c | 19 ++++++++++++------- xen/include/xen/sched.h | 1 + 4 files changed, 17 insertions(+), 7 deletions(-)
xen/arch/arm/smpboot.c +2 -0
@@ -350,6 +350,8 @@ void start_secondary(unsigned long boot_phys_offset,setup_cpu_sibling_map (cpuid); +scheduler_percpu_init (cpuid); +/* Run local notifiers */ notify_cpu_starting (cpuid);/*
xen/arch/x86/smpboot.c +2 -0
@@ -383,6 +383,8 @@ void start_secondary(void *unused)set_cpu_sibling_map (cpu); +scheduler_percpu_init (cpu); +init_percpu_time ();setup_secondary_APIC_clock ();
xen/common/schedule.c +12 -7
@@ -2018,6 +2018,15 @@ static void cpu_schedule_down(unsigned int cpu)per_cpu (sched_res, cpu) =NULL ; } +void scheduler_percpu_init (unsigned int cpu) +{ +struct scheduler *sched =per_cpu (scheduler, cpu); +struct sched_resource *sd =per_cpu (sched_res, cpu); + +if ( system_state != SYS_STATE_resume ) +sched_init_pdata (sched, sd->sched_priv, cpu); +} +static int cpu_schedule_callback (struct notifier_block *nfb,unsigned long action,void *hcpu) {
@@ -2036,8 +2045,8 @@ static int cpu_schedule_callback(* data can avoid implementing alloc_pdata. init_pdata may, however, be * necessary/useful in this case too (e.g., it can contain the "register * the pCPU to the scheduler" part). alloc_pdata (if present) is called -* during CPU_UP_PREPARE. init_pdata (if present) is called during -* CPU_STARTING. +* during CPU_UP_PREPARE. init_pdata (if present) is called before +* CPU_STARTING in scheduler_percpu_init(). * * On the other hand, at teardown, we need to reverse what has been done * during initialization, and then free the per-pCPU specific data. This
@@ -2060,10 +2069,6 @@ static int cpu_schedule_callback(*/ switch ( action ) { -case CPU_STARTING : -if ( system_state != SYS_STATE_resume ) -sched_init_pdata (sched, sd->sched_priv, cpu); -break ;case CPU_UP_PREPARE :if ( system_state != SYS_STATE_resume ) rc =cpu_schedule_up (cpu);
@@ -2160,7 +2165,7 @@ void __init scheduler_init(void)this_cpu (sched_res)->curr = idle_vcpu[0 ]->sched_item;this_cpu (sched_res)->sched_priv =sched_alloc_pdata (&ops,0 );BUG_ON (IS_ERR (this_cpu (sched_res)->sched_priv)); -sched_init_pdata (&ops,this_cpu (sched_res)->sched_priv,0 ); +scheduler_percpu_init (0 ); }/*
xen/include/xen/sched.h +1 -0
@@ -678,6 +678,7 @@ void __domain_crash(struct domain *d);void noreturnasm_domain_crash_synchronous (unsigned long addr);void scheduler_init (void ); +void scheduler_percpu_init (unsigned int cpu);int sched_init_vcpu (struct vcpu *v);void sched_destroy_vcpu (struct vcpu *v);int sched_init_domain (struct domain *d,int poolid);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 41/45 xen/sched: add a percpu resource index
In reply to Juergen GrossAdd a percpu variable holding the index of the cpu in the current sched_resource structure. This index is used to get the correct vcpu of a sched_item on a specific cpu.
For now this index will be zero for all cpus, but with core scheduling it will be possible to have higher values, too.
---
RFC V2: new patch (carved out from RFC V1 patch 49)
--- xen/common/schedule.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
xen/common/schedule.c +9 -2
@@ -68,6 +68,7 @@ static void poll_timer_fn(void *data);/* This is global for now so that private implementations can reach it */ DEFINE_PER_CPU(struct scheduler *, scheduler); DEFINE_PER_CPU(struct sched_resource *, sched_res); +static DEFINE_PER_CPU(unsigned int, sched_res_idx);/* Scratch space for cpumasks. */ DEFINE_PER_CPU (cpumask_t, cpumask_scratch);
@@ -78,6 +79,12 @@ extern const struct scheduler *__start_schedulers_array[], *__end_schedulers_arrstatic struct scheduler __read_mostly ops; +static inline struct vcpu *sched_item2vcpu_cpu (struct sched_item *item, +unsigned int cpu) +{ +return item->domain->vcpu[item->item_id +per_cpu (sched_res_idx, cpu)]; +} +static inline struct scheduler *dom_scheduler (const struct domain *d) {if (likely (d->cpupool !=NULL ) )
@@ -1844,7 +1851,7 @@ static void sched_slave(void)pcpu_schedule_unlock_irq (lock, cpu); -sched_context_switch (vprev, next->vcpu, now); +sched_context_switch (vprev,sched_item2vcpu_cpu (next, cpu), now); }/*
@@ -1903,7 +1910,7 @@ static void schedule(void)pcpu_schedule_unlock_irq (lock, cpu); - vnext = next->vcpu; + vnext =sched_item2vcpu_cpu (next, cpu);sched_context_switch (vprev, vnext, now); }
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 42/45 xen/sched: add fall back to idle vcpu when scheduling item
In reply to Juergen GrossWhen scheduling an item with multiple vcpus there is no guarantee all vcpus are available (e.g. above maxvcpus or vcpu offline). Fall back to idle vcpu of the current cpu in that case. This requires to store the correct schedule_item pointer in the idle vcpu as long as it used as fallback vcpu.
In order to modify the runstates of the correct vcpus when switching schedule items merge sched_item_runstate_change() into sched_switch_items() and loop over the affected physical cpus instead of the item's vcpus. This in turn requires an access function to the current variable of other cpus.
Today context_saved() is called in case previous and next vcpus differ when doing a context switch. With an idle vcpu being capable to be a substitute for an offline vcpu this is problematic when switching to an idle scheduling item. An idle previous vcpu leaves us in doubt which schedule item was active previously, so save the previous item pointer in the per-schedule resource area and use its value being non-NULL as a hint whether context_saved() should be called.
---
RFC V2: new patch (Andrew Cooper)
--- xen/arch/x86/domain.c | 21 ++++++ xen/common/schedule.c | 153 +++++++++++++++++++++++++++--------------- xen/include/asm-arm/current.h | 1 + xen/include/asm-x86/current.h | 7 +- xen/include/asm-x86/smp.h | 3 + xen/include/xen/sched-if.h | 4 +- 6 files changed, 134 insertions(+), 55 deletions(-)
xen/arch/x86/domain.c +21 -0
@@ -154,6 +154,24 @@ static void idle_loop(void) } } +/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent. +*/ +static void guest_idle_loop (void ) +{ +unsigned int cpu =smp_processor_id (); + +for ( ; ; ) + { +if ( !softirq_pending (cpu) ) +default_idle (); +do_softirq (); + } +} +void startup_cpu_idle_loop (void ) {struct vcpu *v = current;
@@ -167,6 +185,9 @@ void startup_cpu_idle_loop(void)static void noreturncontinue_idle_domain (struct vcpu *v) { +if ( !is_idle_item (v->sched_item) ) +reset_stack_and_jump (guest_idle_loop); +reset_stack_and_jump (idle_loop); }
xen/common/schedule.c +100 -53
263 lines not on this page · raw message
xen/include/asm-arm/current.h +1 -0
@@ -18,6 +18,7 @@ DECLARE_PER_CPU(struct vcpu *, curr_vcpu);#define current (this_cpu(curr_vcpu))#define set_current (vcpu) do { current = (vcpu); } while (0) +#define get_cpu_current (cpu) (per_cpu(curr_vcpu, cpu))/* Per-VCPU state that lives at the top of the stack */ struct cpu_info {
xen/include/asm-x86/current.h +6 -1
@@ -76,6 +76,11 @@ struct cpu_info {/* get_stack_bottom() must be 16-byte aligned */ }; +static inline struct cpu_info *get_cpu_info_from_stack (unsigned long sp) +{ +return (struct cpu_info *)((sp | (STACK_SIZE -1 )) +1 ) -1 ; +} +static inline struct cpu_info *get_cpu_info (void ) {#ifdef __clang__
@@ -86,7 +91,7 @@ static inline struct cpu_info *get_cpu_info(void) registerunsigned long sp asm("rsp" );#endif -return (struct cpu_info *)((sp | (STACK_SIZE -1 )) +1 ) -1 ; +return get_cpu_info_from_stack (sp); }#define get_current () (get_cpu_info()->current_vcpu)
xen/include/asm-x86/smp.h +3 -0
@@ -76,6 +76,9 @@ void set_nr_sockets(void);/* Representing HT and core siblings in each socket. */ extern cpumask_t **socket_cpumask; +#define get_cpu_current (cpu) \ + (get_cpu_info_from_stack((unsigned long)stack_base[cpu])->current_vcpu) +#endif /* !__ASSEMBLY__ */ #endif
xen/include/xen/sched-if.h +3 -1
@@ -39,6 +39,8 @@ struct sched_resource {spinlock_t *schedule_lock, _lock;struct sched_item *curr;/* current task */ +struct sched_item *sched_item_idle; +struct sched_item *prev;/* previous task */ void *sched_priv;struct timer s_timer;/* scheduling timer */ atomic_t urgent_count;/* how many urgent vcpus */
@@ -152,7 +154,7 @@ static inline void sched_clear_pause_flags_atomic(struct sched_item *item,static inline struct sched_item *sched_idle_item (unsigned int cpu) { -return idle_vcpu[cpu]->sched_item; +return per_cpu (sched_res, cpu)->sched_item_idle; }static inline unsigned int sched_get_resource_cpu (unsigned int cpu)
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
xen/arch/x86/domain.c static void idle_loop(void) · line 161
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +5
-0@@ -154,6 +154,24 @@ static void idle_loop(void) } } +/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent.
I guess I'm missing a part of the description which explains all this:
What's wrong with doing scrubbing work, for example? Why is doing tasklet work not okay, but softirqs are? What is the deal with v->is_urgent, i.e. what justifies not entering a decent power saving mode here on Intel, but doing so on AMD?
xen/include/asm-x86/smp.h void set_nr_sockets(void); · line 80
Quoted 8 lines
xen/include/asm-x86/smp.h +2
-0@@ -76,6 +76,9 @@ void set_nr_sockets(void);/* Representing HT and core siblings in each socket. */ extern cpumask_t **socket_cpumask; +#define get_cpu_current (cpu) \ + (get_cpu_info_from_stack((unsigned long)stack_base[cpu])->current_vcpu)
Yet another, slightly different notion of "current". If "current" itself is not suitable (I can't immediately see why that would be, but I also didn't look at all the scheduler specific changes earlier in this series), why isn't per_cpu(curr_vcpu, cpu) either?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +0
-0@@ -154,6 +154,24 @@ static void idle_loop(void) } }
Quoted 11 lines
xen/arch/x86/domain.c static void idle_loop(void) · line 161
Quoted 5 lines
xen/arch/x86/domain.c +5
-0+/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent. I guess I'm missing a part of the description which explains all this:
What's wrong with doing scrubbing work, for example? Why is doing tasklet work not okay, but softirqs are? What is the deal withv->is_urgent, i.e. what justifies not entering a decent power saving mode here on Intel, but doing so on AMD?
One of the reasons for using core scheduling is to avoid running vcpus of different domains on the same core in order to minimize the chances for side channel attacks to data of other domains. Not allowing scrubbing or tasklets here is due to avoid accessing data of other domains.
As with core scheduling we can be sure the other thread is active (otherwise we would schedule the idle item) and hoping for saving power by using mwait is moot.
Quoted 7 lines
Quoted 7 lines
xen/include/asm-x86/smp.h +0
-0@@ -76,6 +76,9 @@ void set_nr_sockets(void);/* Representing HT and core siblings in each socket. */ extern cpumask_t **socket_cpumask;
Quoted 7 lines
xen/include/asm-x86/smp.h void set_nr_sockets(void); · line 80
Quoted 2 lines
xen/include/asm-x86/smp.h +2
-0+#define get_cpu_current (cpu) \ + (get_cpu_info_from_stack((unsigned long)stack_base[cpu])->current_vcpu)Yet another, slightly different notion of "current". If "current" itself is not suitable (I can't immediately see why that would be, but I also didn't look at all the scheduler specific changes earlier in this series), why isn't
per_cpu(curr_vcpu, cpu)either?
current is always the vcpu running on the current physical cpu. curr_vcpu is the vcpu which was the one running in guest mode last (this avoids the need to save/restore context in case a vcpu is blocked for a short time without another guest vcpu running on the physical cpu in between), so with current being idle the two can differ.
Here I need "current" from another physical cpu which is not easily available.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 16.05.19 at 15:51, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +0
-0@@ -154,6 +154,24 @@ static void idle_loop(void) } }Quoted 11 lines
xen/arch/x86/domain.c static void idle_loop(void) · line 161
Quoted 5 lines
xen/arch/x86/domain.c +5
-0+/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent. I guess I'm missing a part of the description which explains all this:
What's wrong with doing scrubbing work, for example? Why is doing tasklet work not okay, but softirqs are? What is the deal withv->is_urgent, i.e. what justifies not entering a decent power saving mode here on Intel, but doing so on AMD?
Quoted 5 lines
One of the reasons for using core scheduling is to avoid running vcpus of different domains on the same core in order to minimize the chances for side channel attacks to data of other domains. Not allowing scrubbing or tasklets here is due to avoid accessing data of other domains.
So how is running softirqs okay then? And how is scrubbing accessing other domains' data?
Quoted 3 lines
As with core scheduling we can be sure the other thread is active (otherwise we would schedule the idle item) and hoping for saving power by using mwait is moot.
Saving power may be indirect, by the CPU re-arranging resource assignment between threads when one goes idle. I have no idea whether they do this when entering C1, or only when entering deeper C states.
And anyway - I'm still none the wiser as to the v->is_urgent relationship.
Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
xen/include/asm-x86/smp.h +0
-0@@ -76,6 +76,9 @@ void set_nr_sockets(void);/* Representing HT and core siblings in each socket. */ extern cpumask_t **socket_cpumask;Quoted 7 lines
xen/include/asm-x86/smp.h void set_nr_sockets(void); · line 80
Quoted 2 lines
xen/include/asm-x86/smp.h +2
-0+#define get_cpu_current (cpu) \ + (get_cpu_info_from_stack((unsigned long)stack_base[cpu])->current_vcpu)Yet another, slightly different notion of "current". If "current" itself is not suitable (I can't immediately see why that would be, but I also didn't look at all the scheduler specific changes earlier in this series), why isn't
per_cpu(curr_vcpu, cpu)either?current is always the vcpu running on the current physical cpu.
curr_vcpuis the vcpu which was the one running in guest mode last (this avoids the need to save/restore context in case a vcpu is blocked for a short time without another guest vcpu running on the physical cpu in between), so with current being idle the two can differ.
Quoted 2 lines
Here I need "current" from another physical cpu which is not easily available.
Oh, right - I should have been able to spot this.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 43/45 xen/sched: make vcpu_wake() and vcpu_sleep() core scheduling aware
In reply to Juergen Grossvcpu_wake() and vcpu_sleep() need to be made core scheduling aware:
they might need to switch a single vcpu of an already scheduled item between running and not running.
Especially when vcpu_sleep() for a vcpu is being called by a vcpu of the same scheduling item special care must be taken in order to avoid a deadlock: the vcpu to be put asleep must be forced through a context switch without doing so for the calling vcpu. For this purpose add a vcpu flag handled in sched_slave() and in sched_wait_rendezvous_in() allowing a vcpu of the currently running item to switch state at a higher priority than a normal schedule event.
Use the same mechanism when waking up a vcpu of a currently active item.
While at it make vcpu_sleep_nosync_locked() static as it is used in schedule.c only.
---
RFC V2: add vcpu_sleep() handling and force_context_switch flag
--- xen/common/schedule.c | 144 ++++++++++++++++++++++++++++++++++++++++----- xen/include/xen/sched-if.h | 9 ++- xen/include/xen/sched.h | 2 + 3 files changed, 136 insertions(+), 19 deletions(-)
xen/common/schedule.c +128 -16
@@ -79,21 +79,21 @@ extern const struct scheduler *__start_schedulers_array[], *__end_schedulers_arrstatic struct scheduler __read_mostly ops; -static inline struct vcpu *sched_item2vcpu_cpu (struct sched_item *item, -unsigned int cpu) +static inline struct vcpu *item2vcpu_cpu (struct sched_item *item, +unsigned int cpu) {unsigned int idx = item->item_id +per_cpu (sched_res_idx, cpu);const struct domain *d = item->domain; -struct vcpu *v; -if ( idx < d->max_vcpus && d->vcpu[idx] ) - { - v = d->vcpu[idx]; -if ( v->new_state == RUNSTATE_running ) -return v; - } +return (idx < d->max_vcpus && d->vcpu[idx]) ? d->vcpu[idx] :NULL ; +} -return idle_vcpu[cpu]; +static inline struct vcpu *sched_item2vcpu_cpu (struct sched_item *item, +unsigned int cpu) +{ +struct vcpu *v =item2vcpu_cpu (item, cpu); + +return (v && v->new_state == RUNSTATE_running) ? v : idle_vcpu[cpu]; }static inline struct scheduler *dom_scheduler (const struct domain *d)
@@ -644,8 +644,10 @@ void sched_destroy_domain(struct domain *d) } } -void vcpu_sleep_nosync_locked (struct vcpu *v) +static void vcpu_sleep_nosync_locked (struct vcpu *v) { +struct sched_item *item = v->sched_item; +ASSERT (spin_is_locked (per_cpu (sched_res, v->processor)->schedule_lock));if (likely (!vcpu_runnable (v)) )
@@ -653,7 +655,14 @@ void vcpu_sleep_nosync_locked(struct vcpu *v)if ( v->runstate.state == RUNSTATE_runnable )vcpu_runstate_change (v, RUNSTATE_offline,NOW ()); -sched_sleep (vcpu_scheduler (v), v->sched_item); +if (likely (!item_runnable (item)) ) +sched_sleep (vcpu_scheduler (v), item); +else if (item_running (item) >1 && v->is_running && + !v->force_context_switch ) + { + v->force_context_switch = true; +cpu_raise_softirq (v->processor,SCHED_SLAVE_SOFTIRQ ); + } } }
@@ -685,16 +694,22 @@ void vcpu_wake(struct vcpu *v) {unsigned long flags;spinlock_t *lock; +struct sched_item *item = v->sched_item;TRACE_2D (TRC_SCHED_WAKE , v->domain->domain_id, v->vcpu_id); - lock =item_schedule_lock_irqsave (v->sched_item, &flags); + lock =item_schedule_lock_irqsave (item, &flags);if (likely (vcpu_runnable (v)) ) {if ( v->runstate.state >= RUNSTATE_blocked )vcpu_runstate_change (v, RUNSTATE_runnable,NOW ()); -sched_wake (vcpu_scheduler (v), v->sched_item); +sched_wake (vcpu_scheduler (v), item); +if ( item->is_running && !v->is_running && !v->force_context_switch ) + { + v->force_context_switch = true; +cpu_raise_softirq (v->processor,SCHED_SLAVE_SOFTIRQ ); + } }else if ( !(v->pause_flags & VPF_blocked) ) {
@@ -702,7 +717,7 @@ void vcpu_wake(struct vcpu *v)vcpu_runstate_change (v, RUNSTATE_offline,NOW ()); } -item_schedule_unlock_irqrestore (lock, flags, v->sched_item); +item_schedule_unlock_irqrestore (lock, flags, item); }void vcpu_unblock (struct vcpu *v)
@@ -1835,6 +1850,61 @@ static void sched_context_switch(struct vcpu *vprev, struct vcpu *vnext,context_switch (vprev, vnext); } +/* +* Force a context switch of a single vcpu of an item. +* Might be called either if a vcpu of an already running item is woken up +* or if a vcpu of a running item is put asleep with other vcpus of the same +* item still running. +*/ +static struct vcpu *sched_force_context_switch (struct vcpu *vprev, +struct vcpu *v, +int cpu,s_time_t now) +{ + v->force_context_switch = false; + +if (vcpu_runnable (v) == v->is_running ) +return NULL ; + +if (vcpu_runnable (v) ) + { +if (is_idle_vcpu (vprev) ) + { +vcpu_runstate_change (vprev, RUNSTATE_runnable, now); + vprev->sched_item =this_cpu (sched_res)->sched_item_idle; + } +vcpu_runstate_change (v, RUNSTATE_running, now); + } +else + { +/* Make sure not to switch last vcpu of an item away. */ +if (item_running (v->sched_item) ==1 ) +return NULL ; + +vcpu_runstate_change (v,vcpu_runstate_blocked (v), now); + v =sched_item2vcpu_cpu (vprev->sched_item, cpu); +if ( v != vprev ) + { +if (is_idle_vcpu (vprev) ) + { +vcpu_runstate_change (vprev, RUNSTATE_runnable, now); + vprev->sched_item =this_cpu (sched_res)->sched_item_idle; + } +else + { + v->sched_item = vprev->sched_item; +vcpu_runstate_change (v, RUNSTATE_running, now); + } + } + } + + v->is_running =1 ; + +/* Make sure not to loose another slave call. */ +raise_softirq (SCHED_SLAVE_SOFTIRQ ); + +return v; +} +/* * Rendezvous before taking a scheduling decision. * Called with schedule lock held, so all accesses to the rendezvous counter
@@ -1850,6 +1920,7 @@ static struct sched_item *sched_wait_rendezvous_in(struct sched_item *prev, s_time_t now) {struct sched_item *next; +struct vcpu *v;if ( !--prev->rendezvous_in_cnt ) {
@@ -1858,8 +1929,28 @@ static struct sched_item *sched_wait_rendezvous_in(struct sched_item *prev,return next; } + v =item2vcpu_cpu (prev, cpu);while ( prev->rendezvous_in_cnt ) { +if ( v && v->force_context_switch ) + { +struct vcpu *vprev = current; + + v =sched_force_context_switch (vprev, v, cpu, now); + +if ( v ) + { +/* We'll come back another time, so adjust rendezvous_in_cnt. */ + prev->rendezvous_in_cnt++; + +pcpu_schedule_unlock_irq (lock, cpu); + +sched_context_switch (vprev, v, false, now); + } + + v =item2vcpu_cpu (prev, cpu); + } +pcpu_schedule_unlock_irq (lock, cpu);cpu_relax ();pcpu_schedule_lock_irq (cpu);
@@ -1870,10 +1961,11 @@ static struct sched_item *sched_wait_rendezvous_in(struct sched_item *prev,static void sched_slave (void ) { -struct vcpu *vprev = current; +struct vcpu *v, *vprev = current;struct sched_item *prev = vprev->sched_item, *next;s_time_t now;spinlock_t *lock; +bool do_softirq = false;int cpu =smp_processor_id ();ASSERT_NOT_IN_ATOMIC ();
@@ -1882,9 +1974,29 @@ static void sched_slave(void) now =NOW (); + v =item2vcpu_cpu (prev, cpu); +if ( v && v->force_context_switch ) + { + v =sched_force_context_switch (vprev, v, cpu, now); + +if ( v ) + { +pcpu_schedule_unlock_irq (lock, cpu); + +sched_context_switch (vprev, v, false, now); + } + + do_softirq = true; + } +if ( !prev->rendezvous_in_cnt ) {pcpu_schedule_unlock_irq (lock, cpu); + +/* Check for failed forced context switch. */ +if ( do_softirq ) +raise_softirq (SCHEDULE_SOFTIRQ ); +return ; }
xen/include/xen/sched-if.h +6 -3
@@ -75,6 +75,11 @@ static inline bool item_runnable(const struct sched_item *item)return false; } +static inline int vcpu_runstate_blocked (struct vcpu *v) +{ +return (v->pause_flags & VPF_blocked) ? RUNSTATE_blocked : RUNSTATE_offline; +} +static inline bool item_runnable_state (const struct sched_item *item) {struct vcpu *v;
@@ -84,9 +89,7 @@ static inline bool item_runnable_state(const struct sched_item *item) { runnable =vcpu_runnable (v); - v->new_state = runnable ? RUNSTATE_running - : (v->pause_flags & VPF_blocked) - ? RUNSTATE_blocked : RUNSTATE_offline; + v->new_state = runnable ? RUNSTATE_running :vcpu_runstate_blocked (v);if ( runnable ) ret = true;
xen/include/xen/sched.h +2 -0
@@ -186,6 +186,8 @@ struct vcpubool is_running;/* VCPU should wake fast (do not deep sleep the CPU). */ bool is_urgent; +/* VCPU must context_switch without scheduling item. */ +bool force_context_switch;#ifdef VCPU_TRAP_LAST #define VCPU_TRAP_NONE 0
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 44/45 xen/sched: carve out freeing sched_item memory into dedicated function
In reply to Juergen GrossWe'll need a way to free a sched_item structure without side effects in a later patch.
---
RFC V2: new patch, carved out from RFC V1 patch 49
--- xen/common/schedule.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)
xen/common/schedule.c +20 -16
@@ -281,25 +281,10 @@ static void sched_spin_unlock_double(spinlock_t *lock1, spinlock_t *lock2,spin_unlock_irqrestore (lock1, flags); } -static void sched_free_item (struct sched_item *item,struct vcpu *v) +static void sched_free_item_mem (struct sched_item *item) {struct sched_item *prev_item;struct domain *d = item->domain; -struct vcpu *vitem; -unsigned int cnt =0 ; - -/* Don't count to be released vcpu, might be not in vcpu list yet. */ - for_each_sched_item_vcpu ( item, vitem ) -if ( vitem != v ) - cnt++; - - v->sched_item =NULL ; - -if ( cnt ) -return ; - -if ( item->vcpu == v ) - item->vcpu = v->next_in_list;if ( d->sched_item_list == item ) d->sched_item_list = item->next_in_list;
@@ -323,6 +308,25 @@ static void sched_free_item(struct sched_item *item, struct vcpu *v)xfree (item); } +static void sched_free_item (struct sched_item *item,struct vcpu *v) +{ +struct vcpu *vitem; +unsigned int cnt =0 ; + +/* Don't count to be released vcpu, might be not in vcpu list yet. */ + for_each_sched_item_vcpu ( item, vitem ) +if ( vitem != v ) + cnt++; + + v->sched_item =NULL ; + +if ( item->vcpu == v ) + item->vcpu = v->next_in_list; + +if ( !cnt ) +sched_free_item_mem (item); +} +static void sched_item_add_vcpu (struct sched_item *item,struct vcpu *v) { v->sched_item = item;
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 45/45 xen/sched: add scheduling granularity enum
In reply to Juergen GrossAdd a scheduling granularity enum ("thread", "core", "socket") for specification of the scheduling granularity. Initially it is set to "thread", this can be modified by the new boot parameter (x86 only) "sched_granularity".
According to the selected granularity sched_granularity is set after all cpus are online. The sched items of the idle vcpus and the sched resources of the physical cpus need to be combined in case sched_granularity > 1, this happens before the init_pdata hook of the active scheduler is being called.
A test is added for all sched resources holding the same number of cpus. For now panic if this is not the case.
---
RFC V2:
- fixed freeing of sched_res when merging cpus
- rename parameter to "sched-gran" (Jan Beulich)
- rename parameter option from "thread" to "cpu" (Jan Beulich)
--- xen/arch/x86/setup.c | 2 + xen/common/schedule.c | 155 +++++++++++++++++++++++++++++++++++++++++++-- xen/include/xen/sched-if.h | 4 +- xen/include/xen/sched.h | 1 + 4 files changed, 153 insertions(+), 9 deletions(-)
xen/arch/x86/setup.c +2 -0
@@ -1701,6 +1701,8 @@ void __init noreturn __start_xen(unsigned long mbi_p)printk (XENLOG_INFO "Parked %u CPUs\n" , num_parked);smp_cpus_done (); +scheduler_smp_init (); +do_initcalls ();if ( opt_watchdog )
xen/common/schedule.c +148 -7
213 lines not on this page · raw message
xen/include/xen/sched-if.h +2 -2
@@ -23,7 +23,7 @@ extern cpumask_t cpupool_free_cpus;extern int sched_ratelimit_us;/* Scheduling resource mask. */ -extern const cpumask_t *sched_res_mask; +extern cpumask_var_t sched_res_mask;/* * In order to allow a scheduler to remap the lock->cpu mapping,
@@ -45,7 +45,7 @@ struct sched_resource {struct timer s_timer;/* scheduling timer */ atomic_t urgent_count;/* how many urgent vcpus */ unsigned processor; -const cpumask_t *cpus;/* cpus covered by this struct */ +cpumask_var_t cpus;/* cpus covered by this struct */ };#define curr_on_cpu (c) (per_cpu(sched_res, c)->curr)
xen/include/xen/sched.h +1 -0
@@ -681,6 +681,7 @@ void noreturn asm_domain_crash_synchronous(unsigned long addr);void scheduler_init (void );void scheduler_percpu_init (unsigned int cpu); +void scheduler_smp_init (void );int sched_init_vcpu (struct vcpu *v);void sched_destroy_vcpu (struct vcpu *v);int sched_init_domain (struct domain *d,int poolid);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1706
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/setup.c +2
-0@@ -1701,6 +1701,8 @@ void __init noreturn __start_xen(unsigned long mbi_p)printk (XENLOG_INFO "Parked %u CPUs\n" , num_parked);smp_cpus_done (); +scheduler_smp_init (); +do_initcalls ();
This placement and the actual implementation of the function make me wonder: Why didn't you make this an initcall, thus taking care of Arm (at least in an abstract way) at the same time?
Quoted 20 lines
xen/arch/x86/setup.c +0
-0void scheduler_percpu_init (unsigned int cpu) {struct scheduler *sched = per_cpu(scheduler, cpu); struct sched_resource *sd = per_cpu(sched_res, cpu);xen/arch/x86/setup.c +16
-0+const cpumask_t *mask; +unsigned int master_cpu; +spinlock_t *lock; +struct sched_item *old_item, *master_item; + +if ( system_state == SYS_STATE_resume ) +return ; + +switch ( opt_sched_granularity ) + { +case SCHED_GRAN_cpu: + mask =cpumask_of (cpu); +break ; +case SCHED_GRAN_core: + mask =per_cpu (cpu_sibling_mask, cpu); +break ;
xen/arch/x86/setup.c
Quoted 10 lines
xen/arch/x86/setup.c +8
-1+case SCHED_GRAN_socket: + mask =per_cpu (cpu_core_mask, cpu); +break ; +default : +ASSERT_UNREACHABLE (); +return ; + } -if ( system_state != SYS_STATE_resume ) +if ( cpu ==0 ||cpumask_weight (mask) ==1 )
At least outside of x86 specific code I think we should avoid introducing (further?) assumptions that seeing CPU 0 on a CPU initialization path implies this being while booting the system. I wonder anyway whether the right side of the || doesn't render the left side redundant.
xen/arch/x86/setup.c
Quoted 6 lines
xen/arch/x86/setup.c +6
-0+static unsigned int __init sched_check_granularity (void ) +{ +unsigned int cpu; +unsigned int siblings, gran =0 ; + +for_each_online_cpu ( cpu )
You want to decide for one of two possible styles, but not a mixture of both:
for_each_online_cpu ( cpu )
or
for_each_online_cpu(cpu)
. Yet then I'm a little puzzled by its use here in the first place. Generally I think for_each_cpu() uses in __init functions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/setup.c +0
-0@@ -1701,6 +1701,8 @@ void __init noreturn __start_xen(unsigned long mbi_p)printk (XENLOG_INFO "Parked %u CPUs\n" , num_parked);smp_cpus_done ();
Quoted 7 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1706
Quoted 3 lines
xen/arch/x86/setup.c +2
-0+scheduler_smp_init (); +do_initcalls ();This placement and the actual implementation of the function make me wonder: Why didn't you make this an initcall, thus taking care of Arm (at least in an abstract way) at the same time?
Hmm, true. Will change.
Quoted 29 lines
Quoted 29 lines
xen/arch/x86/setup.c +0
-0void scheduler_percpu_init (unsigned int cpu) {struct scheduler *sched = per_cpu(scheduler, cpu); struct sched_resource *sd = per_cpu(sched_res, cpu);xen/arch/x86/setup.c +23
-0+const cpumask_t *mask; +unsigned int master_cpu; +spinlock_t *lock; +struct sched_item *old_item, *master_item; + +if ( system_state == SYS_STATE_resume ) +return ; + +switch ( opt_sched_granularity ) + { +case SCHED_GRAN_cpu: + mask =cpumask_of (cpu); +break ; +case SCHED_GRAN_core: + mask =per_cpu (cpu_sibling_mask, cpu); +break ; +case SCHED_GRAN_socket: + mask =per_cpu (cpu_core_mask, cpu); +break ; +default : +ASSERT_UNREACHABLE (); +return ; + }
Quoted 8 lines
xen/arch/x86/setup.c
Quoted 2 lines
xen/arch/x86/setup.c +1
-1-if ( system_state != SYS_STATE_resume ) +if ( cpu ==0 ||cpumask_weight (mask) ==1 )At least outside of x86 specific code I think we should avoid introducing (further?) assumptions that seeing CPU 0 on a CPU initialization path implies this being while booting the system. I wonder anyway whether the right side of the || doesn't render the left side redundant.
On the boot cpu this function is called before e.g. cpu_sibling_mask is initialized. I can have a try using:
if ( cpumask_weight(mask) <= 1 )
Quoted 11 lines
xen/arch/x86/setup.c
Quoted 7 lines
xen/arch/x86/setup.c +6
-0+static unsigned int __init sched_check_granularity (void ) +{ +unsigned int cpu; +unsigned int siblings, gran =0 ; + +for_each_online_cpu ( cpu )You want to decide for one of two possible styles, but not a mixture of both:
Quoted 5 lines
for_each_online_cpu( cpu )or
for_each_online_cpu(cpu)
Sorry, will correct.
Quoted 6 lines
. Yet then I'm a little puzzled by its use here in the first place. Generally I think
for_each_cpu()uses in__initfunctions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.
And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 40 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
Quoted 29 lines · Jan Beulich
Quoted 29 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
void
scheduler_percpu_init(unsigned int cpu){ struct scheduler *sched = per_cpu(scheduler, cpu); struct sched_resource *sd = per_cpu(sched_res, cpu);xen/common/schedule.c +23
-0+const cpumask_t *mask; +unsigned int master_cpu; +spinlock_t *lock; +struct sched_item *old_item, *master_item; + +if ( system_state == SYS_STATE_resume ) +return ; + +switch ( opt_sched_granularity ) + { +case SCHED_GRAN_cpu: + mask =cpumask_of (cpu); +break ; +case SCHED_GRAN_core: + mask =per_cpu (cpu_sibling_mask, cpu); +break ; +case SCHED_GRAN_socket: + mask =per_cpu (cpu_core_mask, cpu); +break ; +default : +ASSERT_UNREACHABLE (); +return ; + }Quoted 8 lines
xen/common/schedule.c static int cpu_schedule_up(unsigned int cpu) · line 2261
Quoted 2 lines
xen/common/schedule.c +1
-1-if ( system_state != SYS_STATE_resume ) +if ( cpu ==0 ||cpumask_weight (mask) ==1 )At least outside of x86 specific code I think we should avoid introducing (further?) assumptions that seeing CPU 0 on a CPU initialization path implies this being while booting the system. I wonder anyway whether the right side of the || doesn't render the left side redundant.
Quoted 4 lines
On the boot cpu this function is called before e.g.
cpu_sibling_maskis initialized. I can have a try using:if (
cpumask_weight(mask)<= 1 )
Or re-order things such that it gets set in time?
Quoted 25 lines
Quoted 10 lines
xen/common/schedule.c static struct notifier_block cpu_schedule_nfb = { · line 2374
Quoted 6 lines
xen/common/schedule.c +6
-0+static unsigned int __init sched_check_granularity (void ) +{ +unsigned int cpu; +unsigned int siblings, gran =0 ; + +for_each_online_cpu ( cpu )You want to decide for one of two possible styles, but not a mixture of both:
Quoted 5 lines
for_each_online_cpu( cpu )or
for_each_online_cpu(cpu)Sorry, will correct.
Quoted 6 lines
. Yet then I'm a little puzzled by its use here in the first place. Generally I think
for_each_cpu()uses in__initfunctions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.
Quoted 6 lines
And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
I have started to address cpu on/offlining now.
There are multiple design decisions to take.
1. Interaction between sched-gran and smt boot parameters
2. Interaction between sched-gran and xen-hptool smt switching
3. Interaction between sched-gran and single cpu on/offlining
Right now any guest won't see a difference regarding sched-gran selection. This means we don't have to think about potential migration restrictions. This might change in future when we want to enable the guest to e.g. use core scheduling themselves in order to mitigate against side channel attacks within the guest.
The most simple solution would be (and I'd like to send out V1 of my series with that implemented):
sched-gran=core and sched-gran=socket don't allow dynamical switching of smt via xen-hptool.
With sched-gran=core or sched-gran=socket offlining a single cpu results in moving the complete core or socket to cpupool_free_cpus and then offlining from there. Only complete cores/sockets can be moved to any cpupool. When onlining a cpu it is added to cpupool_free_cpus and if the core/socket is completely online it will automatically be added to Pool-0 (as today any single onlined cpu).
The next steps (for future patches) could be:
- per-cpupool smt settings (static at cpupool creation, moving a domain between cpupools with different smt settings not supported)
- support moving domains between cpupools with different smt settings (a guest started with smt=0 would only ever use 1 thread per core)
- support per-cpupool scheduling granularity
Thoughts?
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 16:36, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
I have started to address cpu on/offlining now.
There are multiple design decisions to take.
1. Interaction between sched-gran and smt boot parameters
2. Interaction between sched-gran and xen-hptool smt switching
3. Interaction between sched-gran and single cpu on/offliningRight now any guest won't see a difference regarding sched-gran selection. This means we don't have to think about potential migration restrictions. This might change in future when we want to enable the guest to e.g. use core scheduling themselves in order to mitigate against side channel attacks within the guest.
The most simple solution would be (and I'd like to send out V1 of my series with that implemented):
sched-gran=core and sched-gran=socket don't allow dynamical switching of smt via xen-hptool.
Quoted 6 lines
With sched-gran=core or sched-gran=socket offlining a single cpu results in moving the complete core or socket to
cpupool_free_cpusand then offlining from there. Only complete cores/sockets can be moved to any cpupool. When onlining a cpu it is added tocpupool_free_cpusand if the core/socket is completely online it will automatically be added to Pool-0 (as today any single onlined cpu).
Well, this is in line with what was discussed on the call yesterday, so I think it's an acceptable initial state to end up in. Albeit, just for completeness, I'm not convinced there's no use for "smt-{dis,en}able" anymore with core-aware scheduling implemented just in Xen - it may still be considered useful as long as we don't expose proper topology to guests, for them to be able to do something similar.
Quoted 7 lines
The next steps (for future patches) could be:
- per-cpupool smt settings (static at cpupool creation, moving a domain between cpupools with different smt settings not supported)
- support moving domains between cpupools with different smt settings (a guest started with smt=0 would only ever use 1 thread per core)
Yes, in its most general terms: Such movement may be wasteful, but should be possible to be carried out safely in all cases.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 1 line · Juergen Gross
Add support for core- and socket-scheduling in the Xen hypervisor.
The series is available under:
github.com/jgross1/xen.git sched-rfc-v2
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 40 lines · Jan Beulich
Quoted 40 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
Quoted 37 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
void
scheduler_percpu_init(unsigned int cpu){ struct scheduler *sched = per_cpu(scheduler, cpu); struct sched_resource *sd = per_cpu(sched_res, cpu);(quoted diff) +16
-0+ const cpumask_t *mask; + unsigned int master_cpu; + spinlock_t *lock; + struct sched_item *old_item, *master_item; + + if ( system_state == SYS_STATE_resume ) + return; + + switch ( opt_sched_granularity ) + { + case SCHED_GRAN_cpu: + mask = cpumask_of(cpu); + break; + case SCHED_GRAN_core: + mask = per_cpu(cpu_sibling_mask, cpu); + break;Review
Quoted 10 lines
(quoted diff) +8
-1+ case SCHED_GRAN_socket: + mask = per_cpu(cpu_core_mask, cpu); + break; + default: + ASSERT_UNREACHABLE(); + return; + } - if ( system_state != SYS_STATE_resume ) + if ( cpu == 0 || cpumask_weight(mask) == 1 )At least outside of x86 specific code I think we should avoid introducing (further?) assumptions that seeing CPU 0 on a CPU initialization path implies this being while booting the system. I wonder anyway whether the right side of the || doesn't render the left side redundant.
Quoted 6 lines
Quoted 4 lines
On the boot cpu this function is called before e.g.
cpu_sibling_maskis initialized. I can have a try using:if (
cpumask_weight(mask)<= 1 )Or re-order things such that it gets set in time?
That might be difficult.
I've ended up with:
if ( !mask || cpumask_weight(mask) == 1 )
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
Review
Quoted 7 lines
(quoted diff) +6
-0+static unsigned int __init sched_check_granularity(void) +{ + unsigned int cpu; + unsigned int siblings, gran = 0; + + for_each_online_cpu( cpu )You want to decide for one of two possible styles, but not a mixture of both:
Quoted 5 lines
for_each_online_cpu( cpu )or
for_each_online_cpu(cpu)Sorry, will correct.
Quoted 6 lines
. Yet then I'm a little puzzled by its use here in the first place. Generally I think
for_each_cpu()uses in__initfunctions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
Especially with hotplug things are becoming more complicated: I'd like to have the final version fall back to smaller granularities in case e.g. the user has selected socket scheduling and two sockets have different numbers of cores. With hotplug such a situation might be discovered only with some domUs already running, so how should we react in that case? Doing panic() is no option, so either we reject onlining the additional socket, or we adapt by dynamically modifying the scheduling granularity. Without that being discussed I don't think it makes sense to put a lot effort into a solution which is going to be rejected in the end.
I'm fine with doing a proper implementation for the non-RFC variant with a generally accepted design.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 12:20, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
Quoted 5 lines · Jan Beulich
. Yet then I'm a little puzzled by its use here in the first place. Generally I think
for_each_cpu()uses in__initfunctions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
Quoted 10 lines
Especially with hotplug things are becoming more complicated: I'd like to have the final version fall back to smaller granularities in case e.g. the user has selected socket scheduling and two sockets have different numbers of cores. With hotplug such a situation might be discovered only with some domUs already running, so how should we react in that case? Doing
panic()is no option, so either we reject onlining the additional socket, or we adapt by dynamically modifying the scheduling granularity. Without that being discussed I don't think it makes sense to put a lot effort into a solution which is going to be rejected in the end.
Hmm, where's the symmetry requirement coming from? Socket scheduling should mean as many vCPU-s on one socket as there are cores * threads; similarly core scheduling (number of threads). Statically partitioning domains would seem an intermediate step at best only anyway, as that requires (on average) leaving more resources (cores/threads) idle than with a dynamic partitioning model.
As to your specific question how to react: Since bringing online a full new socket implies bringing online all its cores / threads one by one anyway, a "too small" socket in your scheme would simply result in the socket remaining unused until "enough" cores/threads have appeared. Similarly the socket would go out of use as soon as one of its cores/threads gets offlined. Obviously this ends up problematic for the last usable socket.
But with the static partitioning you describe I also can't really see how "xen-hptool smt-disable" is going to work.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 33 lines · Jan Beulich
Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 12:20, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
Quoted 5 lines · Jan Beulich
. Yet then I'm a little puzzled by its use here in the first place. Generally I think
for_each_cpu()uses in__initfunctions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
Quoted 10 lines
Especially with hotplug things are becoming more complicated: I'd like to have the final version fall back to smaller granularities in case e.g. the user has selected socket scheduling and two sockets have different numbers of cores. With hotplug such a situation might be discovered only with some domUs already running, so how should we react in that case? Doing
panic()is no option, so either we reject onlining the additional socket, or we adapt by dynamically modifying the scheduling granularity. Without that being discussed I don't think it makes sense to put a lot effort into a solution which is going to be rejected in the end.
Quoted 7 lines
Hmm, where's the symmetry requirement coming from? Socket scheduling should mean as many vCPU-s on one socket as there are cores * threads; similarly core scheduling (number of threads). Statically partitioning domains would seem an intermediate step at best only anyway, as that requires (on average) leaving more resources (cores/threads) idle than with a dynamic partitioning model.
And that is exactly the purpose of core/socket scheduling. How else would it be possible (in future) to pass through the topology below the scheduling granularity to the guest? And how should it be of any use for fighting security issues due to side channel attacks?
Quoted 6 lines
As to your specific question how to react: Since bringing online a full new socket implies bringing online all its cores / threads one by one anyway, a "too small" socket in your scheme would simply result in the socket remaining unused until "enough" cores/threads have appeared. Similarly the socket would go out of use as soon as one of its cores/threads gets offlined.
Yes, this is a possible way to do it. It should be spelled out, though.
Quoted 1 line
Obviously this ends up problematic for the last usable socket.
Yes, like today for the last cpu/thread.
Quoted 2 lines
But with the static partitioning you describe I also can't really see how "xen-hptool smt-disable" is going to work.
It won't work. It just makes no sense to use it with core scheduling active. In the best case it would render all cores but the last one unusable (the last one would be refused to be disabled) and all domains would then fight for that last core.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 43 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 14:23, <[email protected]> wrote:
Quoted 33 lines · Jan Beulich
Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 12:20, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
Quoted 5 lines · Jan Beulich
. Yet then I'm a little puzzled by its use here in the first place. Generally I think
for_each_cpu()uses in__initfunctions are problematic, as they then require further code elsewhere to deal with hot-onlining. A pre-SMP-initcall plus use of CPU notifiers is typically more appropriate.And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
Quoted 10 lines
Especially with hotplug things are becoming more complicated: I'd like to have the final version fall back to smaller granularities in case e.g. the user has selected socket scheduling and two sockets have different numbers of cores. With hotplug such a situation might be discovered only with some domUs already running, so how should we react in that case? Doing
panic()is no option, so either we reject onlining the additional socket, or we adapt by dynamically modifying the scheduling granularity. Without that being discussed I don't think it makes sense to put a lot effort into a solution which is going to be rejected in the end.Quoted 7 lines
Hmm, where's the symmetry requirement coming from? Socket scheduling should mean as many vCPU-s on one socket as there are cores * threads; similarly core scheduling (number of threads). Statically partitioning domains would seem an intermediate step at best only anyway, as that requires (on average) leaving more resources (cores/threads) idle than with a dynamic partitioning model.
Quoted 3 lines
And that is exactly the purpose of core/socket scheduling. How else would it be possible (in future) to pass through the topology below the scheduling granularity to the guest?
True. Albeit nevertheless an (at least) unfortunate limitation.
Quoted 2 lines
And how should it be of any
use for fighting security issues due to side channel attacks?
From Xen's pov all is still fine afaict. It's the lack of (correct) topology exposure (as per above) which would make guest side mitigation impossible.
Quoted 7 lines
Quoted 6 lines
As to your specific question how to react: Since bringing online a full new socket implies bringing online all its cores / threads one by one anyway, a "too small" socket in your scheme would simply result in the socket remaining unused until "enough" cores/threads have appeared. Similarly the socket would go out of use as soon as one of its cores/threads gets offlined.
Quoted 6 lines
Yes, this is a possible way to do it. It should be spelled out, though.
Quoted 1 line
Obviously this ends up problematic for the last usable socket.
Yes, like today for the last cpu/thread.
Well, only kind of. It's quite expected that the last thread can't be offlined. I'd call it rather unexpected that a random thread on the last socket can't be offlined just because each other socket also has a single offline thread: There might still be hundreds of online threads in this case, after all.
Quoted 5 lines
Quoted 2 lines
But with the static partitioning you describe I also can't really see how "xen-hptool smt-disable" is going to work.
It won't work. It just makes no sense to use it with core scheduling active.
Why not? Disabling HT may be for purposes other than mitigating vulnerabilities like L1TF. And the system is in a symmetric state at the beginning and end of the entire operation; it's merely intermediate state which doesn't fit the expectations you set forth.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 01/45] xen/sched: add inline wrappers for calling per-scheduler functions
Quoted 59 lines · Jan Beulich
Quoted 52 lines
Quoted 1 line
Quoted 1 line · On 09.05.19 at 14:03, <[email protected]> wrote:
Quoted 45 lines · Jan Beulich
Quoted 37 lines
Quoted 1 line
Quoted 1 line · On 09.05.19 at 12:56, <[email protected]> wrote:
Quoted 27 lines · George Dunlap
Quoted 26 lines · Juergen Gross
Quoted 24 lines · George Dunlap
Quoted 4 lines · Juergen Gross
Instead of using the
SCHED_OP()macro to call the different scheduler specific functions add inline wrappers for that purpose.Signed-off-by Juergen Gross <[email protected]>This seems like a great idea. One minor comment...
Review
Quoted 11 lines
(quoted diff) +11
-0+static inline int sched_init(struct scheduler *s) +{ + ASSERT(s->init); + return s->init(s); +} + +static inline void sched_deinit(struct scheduler *s) +{ + ASSERT(s->deinit); + s->deinit(s); +}I think these would better as BUG_ON()s. These aren't hot paths, and if we do somehow hit this situation in production, 1) it's safer to
BUG_ON()than dereferencing NULL, and 2) you'll get a more helpful error message.Quoted 7 lines
Quoted 3 lines
Only for those 2 instances above? Or would you like
BUG_ON()instead ofASSERT()in the other added inlines, too (maybe not forpick_cpu, but e.g. the ones in free_*) ?Why not for
pick_cpu()? It's the same basic logic -- in production, if it *is* NULL, then you'll either crash with a segfault, or start executing an exploit. Much better toBUG_ON().Quoted 7 lines
pick_cpuis called rather often, so maybe we should avoid additional tests.Hmm, thinking more about it: why don't we just drop those ASSERT/BUG_ON for mandatory functions and test them when doing the
global_init()loop over all schedulers. We could just reject schedulers with missing functions.Quoted 4 lines
This would imply pointers can't be zapped off the structures. IMO this would require, as minimal (language level) protection, that all instances of struct scheduler be const, which doesn't look doable without some further rework
Quoted 4 lines
They are const already.
The default scheduler's struct is copied to a non-const struct scheduler in
scheduler_init().Exactly, and then we have things like
Quoted 8 lines
static int
rt_init(struct scheduler *ops){ ... ops->sched_data = prv;I.e. it would be quite easy for a specific scheduler to zap one or more of its pointers.
So you suggest to ASSERT all pointers before dereferencing them? Why don't we have such ASSERTs in places where we use function vectors hooked to dynamic data (and I don't mean the single functions, but the pointers to the vector, e.g. domain->arch.ctxt_switch)?
Seriously, that would be a major programming bug and I don't think we need to catch that by debug code sprinkled around everywhere.
After my core scheduling series is finished I'd like to do a major scheduler cleanup series. One action item will be to have a single instance const scheduler_funcs structure for each scheduler and a per-cpupool scheduler_data pointer.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 01/45] xen/sched: add inline wrappers for calling per-scheduler functions
Quoted 67 lines
Quoted 1 line
Quoted 1 line · On 09.05.19 at 14:44, <[email protected]> wrote:
Quoted 65 lines · Jan Beulich
Quoted 52 lines
Quoted 1 line
Quoted 1 line · On 09.05.19 at 14:03, <[email protected]> wrote:
Quoted 45 lines · Jan Beulich
Quoted 37 lines
Quoted 1 line
Quoted 1 line · On 09.05.19 at 12:56, <[email protected]> wrote:
Quoted 27 lines · George Dunlap
Quoted 26 lines · Juergen Gross
Quoted 24 lines · George Dunlap
Quoted 4 lines · Juergen Gross
Instead of using the
SCHED_OP()macro to call the different scheduler specific functions add inline wrappers for that purpose.Signed-off-by Juergen Gross <[email protected]>This seems like a great idea. One minor comment...
Review
Quoted 11 lines
(quoted diff) +11
-0+static inline int sched_init(struct scheduler *s) +{ + ASSERT(s->init); + return s->init(s); +} + +static inline void sched_deinit(struct scheduler *s) +{ + ASSERT(s->deinit); + s->deinit(s); +}I think these would better as BUG_ON()s. These aren't hot paths, and if we do somehow hit this situation in production, 1) it's safer to
BUG_ON()than dereferencing NULL, and 2) you'll get a more helpful error message.Quoted 7 lines
Quoted 3 lines
Only for those 2 instances above? Or would you like
BUG_ON()instead ofASSERT()in the other added inlines, too (maybe not forpick_cpu, but e.g. the ones in free_*) ?Why not for
pick_cpu()? It's the same basic logic -- in production, if it *is* NULL, then you'll either crash with a segfault, or start executing an exploit. Much better toBUG_ON().Quoted 7 lines
pick_cpuis called rather often, so maybe we should avoid additional tests.Hmm, thinking more about it: why don't we just drop those ASSERT/BUG_ON for mandatory functions and test them when doing the
global_init()loop over all schedulers. We could just reject schedulers with missing functions.Quoted 4 lines
This would imply pointers can't be zapped off the structures. IMO this would require, as minimal (language level) protection, that all instances of struct scheduler be const, which doesn't look doable without some further rework
Quoted 4 lines
They are const already.
The default scheduler's struct is copied to a non-const struct scheduler in
scheduler_init().Exactly, and then we have things like
static int
rt_init(struct scheduler *ops){ ... ops->sched_data = prv;
Quoted 7 lines
Quoted 2 lines
I.e. it would be quite easy for a specific scheduler to zap one or more of its pointers.
So you suggest to ASSERT all pointers before dereferencing them? Why don't we have such ASSERTs in places where we use function vectors hooked to dynamic data (and I don't mean the single functions, but the pointers to the vector, e.g.
domain->arch.ctxt_switch)?
Where justified I'm certainly in favor of omitting such checks, but without the constification suggested I'm not convinced there is sufficient justification. But here it's the scheduler maintainer to judge anyway - I've merely voiced an opinion.
Quoted 2 lines
Seriously, that would be a major programming bug and I don't think we need to catch that by debug code sprinkled around everywhere.
In fact we've been discussing to gradually add such checks, in order to trade - as explained by George - privilege escalations for DoS-es.
Quoted 4 lines
After my core scheduling series is finished I'd like to do a major scheduler cleanup series. One action item will be to have a single instance const
scheduler_funcsstructure for each scheduler and a per-cpupoolscheduler_datapointer.
That's good to know, being exactly what I would have hoped things would be.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 42 lines · Jan Beulich
Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 16:36, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
I have started to address cpu on/offlining now.
There are multiple design decisions to take.
1. Interaction between sched-gran and smt boot parameters
2. Interaction between sched-gran and xen-hptool smt switching
3. Interaction between sched-gran and single cpu on/offliningRight now any guest won't see a difference regarding sched-gran selection. This means we don't have to think about potential migration restrictions. This might change in future when we want to enable the guest to e.g. use core scheduling themselves in order to mitigate against side channel attacks within the guest.
The most simple solution would be (and I'd like to send out V1 of my series with that implemented):
sched-gran=core and sched-gran=socket don't allow dynamical switching of smt via xen-hptool.
Quoted 6 lines
With sched-gran=core or sched-gran=socket offlining a single cpu results in moving the complete core or socket to
cpupool_free_cpusand then offlining from there. Only complete cores/sockets can be moved to any cpupool. When onlining a cpu it is added tocpupool_free_cpusand if the core/socket is completely online it will automatically be added to Pool-0 (as today any single onlined cpu).
Quoted 6 lines
Well, this is in line with what was discussed on the call yesterday, so I think it's an acceptable initial state to end up in. Albeit, just for completeness, I'm not convinced there's no use for "smt-{dis,en}able" anymore with core-aware scheduling implemented just in Xen - it may still be considered useful as long as we don't expose proper topology to guests, for them to be able to do something similar.
As the extra complexity for supporting that is significant I'd like to at least postpone it. And with the (later) introduction of per-cpupool smt on/off I guess this would be even less important.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
Quoted 25 lines · Juergen Gross
Quoted 14 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 16:36, <[email protected]> wrote:
With sched-gran=core or sched-gran=socket offlining a single cpu results
in moving the complete core or socket tocpupool_free_cpusand then
offlining from there. Only complete cores/sockets can be moved to any
cpupool. When onlining a cpu it is added tocpupool_free_cpusand if
the core/socket is completely online it will automatically be added to
Pool-0 (as today any single onlined cpu).Quoted 9 lines
Well, this is in line with what was discussed on the call yesterday, so
I think it's an acceptable initial state to end up in. Albeit, just for
completeness, I'm not convinced there's no use for "smt- {dis,en}able"
anymore with core-aware scheduling implemented just in Xen - it may still be considered useful as long as we don't expose proper topology to guests, for them to be able to do something similar.
Quoted 6 lines
As the extra complexity for supporting that is significant I'd like to
at least postpone it. And with the (later) introduction of per- cpupool
smt on/off I guess this would be even less important.
I agree.
Isn't it the case that (but note that I'm just thinking out loud here), if we make smt= and sched-gran= per-cpupool, the user gains the chance to use both, if he/she wants (e.g., for testing)?
If yes, is such a thing valuable enough that it'd it make sense to work on that, as a first thing, I mean?
We'd still forbid moving things from pools with different configuration, at least at the beginning, of course.
Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 45/45] xen/sched: add scheduling granularity enum
In reply to Juergen GrossQuoted 51 lines
Quoted 1 line
Quoted 1 line · On 10.05.19 at 11:00, <[email protected]> wrote:
Quoted 42 lines · Jan Beulich
Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 16:36, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 11:23, <[email protected]> wrote:
And that was mentioned in the cover letter: cpu hotplug is not yet handled (hence the RFC status of the series).
Quoted 8 lines
Quoted 3 lines
When cpu hotplug is being added it might be appropriate to switch the scheme as you suggested. Right now the current solution is much more simple.
I see (I did notice the cover letter remark, but managed to not honor it when writing the reply), but I'm unconvinced if incurring more code churn by not dealing with things the "dynamic" way right away is indeed the "more simple" (overall) solution.
I have started to address cpu on/offlining now.
There are multiple design decisions to take.
1. Interaction between sched-gran and smt boot parameters
2. Interaction between sched-gran and xen-hptool smt switching
3. Interaction between sched-gran and single cpu on/offliningRight now any guest won't see a difference regarding sched-gran selection. This means we don't have to think about potential migration restrictions. This might change in future when we want to enable the guest to e.g. use core scheduling themselves in order to mitigate against side channel attacks within the guest.
The most simple solution would be (and I'd like to send out V1 of my series with that implemented):
sched-gran=core and sched-gran=socket don't allow dynamical switching of smt via xen-hptool.
Quoted 6 lines
With sched-gran=core or sched-gran=socket offlining a single cpu results in moving the complete core or socket to
cpupool_free_cpusand then offlining from there. Only complete cores/sockets can be moved to any cpupool. When onlining a cpu it is added tocpupool_free_cpusand if the core/socket is completely online it will automatically be added to Pool-0 (as today any single onlined cpu).Quoted 6 lines
Well, this is in line with what was discussed on the call yesterday, so I think it's an acceptable initial state to end up in. Albeit, just for completeness, I'm not convinced there's no use for "smt-{dis,en}able" anymore with core-aware scheduling implemented just in Xen - it may still be considered useful as long as we don't expose proper topology to guests, for them to be able to do something similar.
Quoted 2 lines
As the extra complexity for supporting that is significant I'd like to at least postpone it.
Understood.
Quoted 2 lines
And with the (later) introduction of per-cpupool smt on/off I guess this would be even less important.
Likely, since pools themselves can be created and destroyed dynamically. At that point this would basically be a more fine-grained smt-{en,dis}able.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 16.05.19 at 15:51, <[email protected]> wrote:
Quoted 18 lines · Jan Beulich
xen/arch/x86/domain.c static void idle_loop(void) · line 161
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +5
-0@@ -154,6 +154,24 @@ static void idle_loop(void) } } +/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent. I guess I'm missing a part of the description which explains all this:
What's wrong with doing scrubbing work, for example? Why is doing tasklet work not okay, but softirqs are? What is the deal withv->is_urgent, i.e. what justifies not entering a decent power saving mode here on Intel, but doing so on AMD?
Quoted 8 lines
Quoted 5 lines
One of the reasons for using core scheduling is to avoid running vcpus of different domains on the same core in order to minimize the chances for side channel attacks to data of other domains. Not allowing scrubbing or tasklets here is due to avoid accessing data of other domains.
So how is running softirqs okay then? And how is scrubbing accessing other domains' data?
Right now I'm not sure whether it is a good idea to block any softirqs. We definitely need to process scheduling requests and I believe RCU and tasklets, too. The tlbflush one should be uncritical, so timers is the remaining one which might be questionable. This can be fine-tuned later IMO e.g. by defining a softirq mask of critical softirqs to block and eventually splitting up e.g. timer and tasklet softirqs into critical and uncritical ones.
Scrubbing will probably pull the cache lines of the dirty pages into the L1 cache of the cpu. For me this sounds problematic. In case we are fine to do scrubbing as there is no risk associated I'm fine to add it back in.
Quoted 8 lines
Quoted 3 lines
As with core scheduling we can be sure the other thread is active (otherwise we would schedule the idle item) and hoping for saving power by using mwait is moot.
Saving power may be indirect, by the CPU re-arranging resource assignment between threads when one goes idle. I have no idea whether they do this when entering C1, or only when entering deeper C states.
SDM Vol. 3 chapter 8.10.1 "HLT instruction":
"Here shared resources that were being used by the halted logical processor become available to active logical processors, allowing them to execute at greater efficiency."
Quoted 2 lines
And anyway - I'm still none the wiser as to the
v->is_urgentrelationship.
With v->is_urgent set today's idle loop will drop into default_idle(). I can remove this sentence in case it is just confusing.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
Quoted 32 lines
Quoted 1 line
Quoted 1 line · On 17.05.19 at 07:13, <[email protected]> wrote:
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 16.05.19 at 15:51, <[email protected]> wrote:
Quoted 18 lines · Jan Beulich
xen/arch/x86/domain.c static void idle_loop(void) · line 161
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +5
-0@@ -154,6 +154,24 @@ static void idle_loop(void) } } +/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent. I guess I'm missing a part of the description which explains all this:
What's wrong with doing scrubbing work, for example? Why is doing tasklet work not okay, but softirqs are? What is the deal withv->is_urgent, i.e. what justifies not entering a decent power saving mode here on Intel, but doing so on AMD?Quoted 8 lines
Quoted 5 lines
One of the reasons for using core scheduling is to avoid running vcpus of different domains on the same core in order to minimize the chances for side channel attacks to data of other domains. Not allowing scrubbing or tasklets here is due to avoid accessing data of other domains.
So how is running softirqs okay then? And how is scrubbing accessing other domains' data?
Quoted 7 lines
Right now I'm not sure whether it is a good idea to block any softirqs. We definitely need to process scheduling requests and I believe RCU and tasklets, too. The tlbflush one should be uncritical, so timers is the remaining one which might be questionable. This can be fine-tuned later IMO e.g. by defining a softirq mask of critical softirqs to block and eventually splitting up e.g. timer and tasklet softirqs into critical and uncritical ones.
Well, okay, but please add an abridged version of this to the patch description then.
Quoted 4 lines
Scrubbing will probably pull the cache lines of the dirty pages into the L1 cache of the cpu. For me this sounds problematic. In case we are fine to do scrubbing as there is no risk associated I'm fine to add it back in.
Well, of course there's going to be a brief period of time where a cache line will be present in CPU internal buffers (it's not just the cache after all, as we've learned with XSA-297). So I can certainly buy that when using core granularity you don't want to scrub on the other thread. But what about the socket granularity case? Scrubbing on fully idle cores should still be fine, I would think.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
As with core scheduling we can be sure the other thread is active (otherwise we would schedule the idle item) and hoping for saving power by using mwait is moot.
Saving power may be indirect, by the CPU re-arranging resource assignment between threads when one goes idle. I have no idea whether they do this when entering C1, or only when entering deeper C states.
Quoted 5 lines
SDM Vol. 3 chapter 8.10.1 "HLT instruction":
"Here shared resources that were being used by the halted logical processor become available to active logical processors, allowing them to execute at greater efficiency."
To be honest, this is to broad/generic a statement to fully trust it, judging from other areas of the SDM. And then, as per above, what about the socket granularity case? Putting entirely idle cores to sleep is surely worthwhile?
Quoted 5 lines
Quoted 2 lines
And anyway - I'm still none the wiser as to the
v->is_urgentrelationship.With
v->is_urgentset today's idle loop will drop intodefault_idle(). I can remove this sentence in case it is just confusing.
I'd prefer if the connection would become more obvious. One needs to go from ->is_urgent via ->urgent_count to sched_has_urgent_vcpu() to find where the described behavior really lives.
What's worse though: This won't work as intended on AMD at all. I don't think it's correct to fall back to default_idle() in this case. Instead sched_has_urgent_vcpu() returning true should amount to the same effect as max_cstate being set to 1. There's
(a) no reason not to use MWAIT on Intel CPUs in this case, if MWAIT can enter C1, and
(b) a strong need to use MWAIT on (at least) AMD Fam17, or else it won't be C1 that gets entered.
I'll see about making a patch in due course.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
Quoted 40 lines · Jan Beulich
Quoted 32 lines
Quoted 1 line
Quoted 1 line · On 17.05.19 at 07:13, <[email protected]> wrote:
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 16.05.19 at 15:51, <[email protected]> wrote:
Quoted 18 lines · Jan Beulich
xen/arch/x86/domain.c static void idle_loop(void) · line 161
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 06.05.19 at 08:56, <[email protected]> wrote:
xen/arch/x86/domain.c +5
-0@@ -154,6 +154,24 @@ static void idle_loop(void) } } +/* +* Idle loop for siblings of active schedule items. +* We don't do any standard idle work like tasklets, page scrubbing or +* livepatching. +* Use default_idle() in order to simulate v->is_urgent. I guess I'm missing a part of the description which explains all this:
What's wrong with doing scrubbing work, for example? Why is doing tasklet work not okay, but softirqs are? What is the deal withv->is_urgent, i.e. what justifies not entering a decent power saving mode here on Intel, but doing so on AMD?Quoted 8 lines
Quoted 5 lines
One of the reasons for using core scheduling is to avoid running vcpus of different domains on the same core in order to minimize the chances for side channel attacks to data of other domains. Not allowing scrubbing or tasklets here is due to avoid accessing data of other domains.
So how is running softirqs okay then? And how is scrubbing accessing other domains' data?
Quoted 7 lines
Right now I'm not sure whether it is a good idea to block any softirqs. We definitely need to process scheduling requests and I believe RCU and tasklets, too. The tlbflush one should be uncritical, so timers is the remaining one which might be questionable. This can be fine-tuned later IMO e.g. by defining a softirq mask of critical softirqs to block and eventually splitting up e.g. timer and tasklet softirqs into critical and uncritical ones.
Quoted 2 lines
Well, okay, but please add an abridged version of this to the patch description then.
Okay.
Quoted 6 lines
Quoted 5 lines
Scrubbing will probably pull the cache lines of the dirty pages into the L1 cache of the cpu. For me this sounds problematic. In case we are fine to do scrubbing as there is no risk associated I'm fine to add it back in.
Quoted 6 lines
Well, of course there's going to be a brief period of time where a cache line will be present in CPU internal buffers (it's not just the cache after all, as we've learned with XSA-297). So I can certainly buy that when using core granularity you don't want to scrub on the other thread. But what about the socket granularity case? Scrubbing on fully idle cores should still be fine, I would think.
I think this would depend on the reason for selecting socket scheduling. I'd at least would want to have a way to select that as I could think of e.g. L3-cache side channel attacks, too.
So maybe I could add a patch on top for adding a sub-option to the sched-gran parameter which will allow (or disallow?) scrubbing on idle cores or threads.
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 4 lines
As with core scheduling we can be sure the other thread is active (otherwise we would schedule the idle item) and hoping for saving power by using mwait is moot.
Saving power may be indirect, by the CPU re-arranging resource assignment between threads when one goes idle. I have no idea whether they do this when entering C1, or only when entering deeper C states.
SDM Vol. 3 chapter 8.10.1 "HLT instruction":
Quoted 8 lines
Quoted 3 lines
"Here shared resources that were being used by the halted logical processor become available to active logical processors, allowing them to execute at greater efficiency."
To be honest, this is to broad/generic a statement to fully trust it, judging from other areas of the SDM. And then, as per above, what about the socket granularity case? Putting entirely idle cores to sleep is surely worthwhile?
Yes, I assume it is. OTOH this might affect context switches badly as the reaction time for the coordinated switch will rise. Maybe a good reason for another sub-option?
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
And anyway - I'm still none the wiser as to the
v->is_urgentrelationship.With
v->is_urgentset today's idle loop will drop intodefault_idle(). I can remove this sentence in case it is just confusing.I'd prefer if the connection would become more obvious. One needs to go from ->is_urgent via ->urgent_count to
sched_has_urgent_vcpu()to find where the described behavior really lives.
Quoted 10 lines
What's worse though: This won't work as intended on AMD at all. I don't think it's correct to fall back to
default_idle()in this case. Insteadsched_has_urgent_vcpu()returning true should amount to the same effect asmax_cstatebeing set to 1. There's
(a) no reason not to use MWAIT on Intel CPUs in this case, if MWAIT can enter C1, and
(b) a strong need to use MWAIT on (at least) AMD Fam17, or else it won't be C1 that gets entered.
I'll see about making a patch in due course.
Thanks. Would you mind doing it in a way that the caller can specify max_cstate? This would remove the need to call sched_has_urgent_vcpu() deep down the idle handling and I could re-use it for my purpose.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH RFC V2 42/45] xen/sched: add fall back to idle vcpu when scheduling item
Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 17.05.19 at 09:48, <[email protected]> wrote:
Quoted 18 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 17.05.19 at 07:13, <[email protected]> wrote:
Quoted 9 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 16.05.19 at 15:51, <[email protected]> wrote:
As with core scheduling we can be sure the other thread is active (otherwise we would schedule the idle item) and hoping for saving power by using mwait is moot.
Saving power may be indirect, by the CPU re-arranging resource assignment between threads when one goes idle. I have no idea whether they do this when entering C1, or only when entering deeper C states.
SDM Vol. 3 chapter 8.10.1 "HLT instruction":
"Here shared resources that were being used by the halted logical processor become available to active logical processors, allowing them to execute at greater efficiency."
Quoted 8 lines
Quoted 4 lines
To be honest, this is to broad/generic a statement to fully trust it, judging from other areas of the SDM. And then, as per above, what about the socket granularity case? Putting entirely idle cores to sleep is surely worthwhile?
Yes, I assume it is. OTOH this might affect context switches badly as the reaction time for the coordinated switch will rise. Maybe a good reason for another sub-option?
While I agree that fine grained control is useful, I'm seeing an increasing risk of there going to be too many controls to actually be certain in the end that all possible combinations work correctly.
Quoted 22 lines
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
And anyway - I'm still none the wiser as to the
v->is_urgentrelationship.With
v->is_urgentset today's idle loop will drop intodefault_idle(). I can remove this sentence in case it is just confusing.I'd prefer if the connection would become more obvious. One needs to go from ->is_urgent via ->urgent_count to
sched_has_urgent_vcpu()to find where the described behavior really lives.Quoted 10 lines
What's worse though: This won't work as intended on AMD at all. I don't think it's correct to fall back to
default_idle()in this case. Insteadsched_has_urgent_vcpu()returning true should amount to the same effect asmax_cstatebeing set to 1. There's
(a) no reason not to use MWAIT on Intel CPUs in this case, if MWAIT can enter C1, and
(b) a strong need to use MWAIT on (at least) AMD Fam17, or else it won't be C1 that gets entered.
I'll see about making a patch in due course.
Quoted 3 lines
Thanks. Would you mind doing it in a way that the caller can specify
max_cstate? This would remove the need to callsched_has_urgent_vcpu()deep down the idle handling and I could re-use it for my purpose.
Hmm, to be honest I'm not fancying giving a parameter to default_idle(), pm_idle(), and friends. Conceptually it is not the business of the callers to control the C states to be used.
What about the opposite: You simply mark idle (v)CPUs in question as "urgent", thus achieving the intended effect as well.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel