Multi-thread per-cpu ksoftirqd
Subsystems (4)
cover Multi-thread per-cpu ksoftirqd
Another attempt to solve softirq deferring problems. There are at least two problems, AFAIK:
o deferring one softirq to ksoftirqd results in latencies for other (different type) softirqs by the reason of ksoftirqd_running() decision for deferring/servicing.
o The logic in __do_softirq() that checks if (pending) after 2ms of processing doesn't work on some machines during i.e. UDP storm.
So, what's done here in attempt to improve this is:
- added boot param to separate softirqs in deffer-groups
- per each softirq-group there is a ksoftirqd (per-cpu also)
The last two patches might be just a brain fart as I tried to improve the metric on which the decision to defer is based.
I measure the time spent to serve each softirq and account that time to ksoftirqd thread of that softirq-group. After that the decision to serve/defer a softirq is based on the comparison:
(current->vruntime < ksoftirqd->vruntime)
Ugh, time measures and updating ksoftirqd cpu time each tick might be costly.. And it looks like it doesn't work as expected: a new task is being started with normalized vruntime (min_vruntime), which is lower than ksoftirqd's. And time spent on servicing softirqs are still bigger than any running task.
Anyway, sending this as RFC, may be some one will like the approach (or suggests some other ideas).
Dmitry Safonov (6):
softirq: Add softirq_groups boot parameter
softirq: Introduce mask for __do_softirq()
softirq: Add reverse group-to-softirq map
softirq: Run per-group per-cpu ksoftirqd thread
softirq: Add time accounting per-softirq type softirq/sched: Account si cpu time to ksoftirqd(s)
Documentation/admin-guide/kernel-parameters.txt | 16 ++ include/linux/hardirq.h | 2 +- include/linux/interrupt.h | 26 +- include/linux/vtime.h | 10 +- init/Kconfig | 10 + kernel/sched/cputime.c | 60 +++- kernel/sched/fair.c | 38 +++ kernel/sched/sched.h | 20 ++ kernel/softirq.c | 362 ++++++++++++++++++++---- net/ipv4/tcp_output.c | 2 +- 10 files changed, 464 insertions(+), 82 deletions(-)
Signature · 1 line
2.13.6
RFC 1/6 softirq: Add softirq_groups boot parameter
ksoftirqd thread allows to defer softirqs if the system is under storm. While it prevents userspace from cpu-time starving, it increases latencies for other softirqs (that are not raised under storm).
As creation of one ksoftirqd thread per-each-softirq-per-cpu will be insane on a huge machines, separate softirqs by groups. It will allow to defer softirqs of one group and continue servicing from other. That means that under a storm of one group's softirqs, softirqs from the other group will be serviced as they come and will not have latency issues.
For each softirq group will be created a per-cpu kthread which will process deferred softirqs of the group.
The parameter will allow an admin define how many ksoftirqd threads will be created on each cpu and which softirqs have the same deferring group.
--- Documentation/admin-guide/kernel-parameters.txt | 16 +++++ include/linux/interrupt.h | 1 + kernel/softirq.c | 87 +++++++++++++++++++++++++ 3 files changed, 104 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +16 -0
@@ -3940,6 +3940,22 @@ Format: <integer> Default: -1 (no limit) + softirq_groups= + [KNL] The count and contents of softirq groups. + Format:[group1],[group2],[groupN] + where group is <softirq1>/<softirq2>/<softirqM> + E.g: softirq_groups=HI/TIMER/HRTIMER,NET_TX/NET_RX,BLOCK + + Defines how many ksoftirqd threads create *per-cpu*. + For each group one ksoftirqd thread is created. + The total number of threads created is + (NR_CPUS * NR_SOFTIRQ_GROUPS). + Admin can define one softirq in different softirq + groups. Softirqs those have no group defined will + be put in default softirq_group. If all softirqs + have been placed into groups default group is not + created. + softlockup_panic= [KNL] Should the soft-lockup detector generate panics. Format: <integer>
include/linux/interrupt.h +1 -0
@@ -486,6 +486,7 @@ extern const char * const softirq_to_name[NR_SOFTIRQS];struct softirq_action {void (*action)(struct softirq_action *); +u32 group_mask; }; asmlinkage void do_softirq(void);
kernel/softirq.c +87 -0
109 lines not on this page · raw message
Signature · 1 line
2.13.6
RFC 2/6 softirq: Introduce mask for __do_softirq()
In reply to Dmitry Safonov__do_softirq() serves all pending softirqs.
As we need to separate softirqs by different groups, we need to serve softirqs from one group and deffer softirqs from the other. Change __do_softirq() so it'll have a mask of softirqs it needs to serve instead of servicing all pending softirqs.
--- include/linux/interrupt.h | 8 ++++---- kernel/softirq.c | 27 ++++++++++++++------------- 2 files changed, 18 insertions(+), 17 deletions(-)
include/linux/interrupt.h +4 -4
@@ -490,14 +490,14 @@ struct softirq_action }; asmlinkage void do_softirq(void); -asmlinkage void __do_softirq(void); +asmlinkage void __do_softirq(__u32 mask);#ifdef __ARCH_HAS_DO_SOFTIRQ -void do_softirq_own_stack (void ); +void do_softirq_own_stack (__u32 mask);#else -static inline void do_softirq_own_stack (void ) +static inline void do_softirq_own_stack (__u32 mask) { -__do_softirq (); +__do_softirq (mask); }#endif
kernel/softirq.c +14 -13
@@ -240,7 +240,7 @@ static inline bool lockdep_softirq_start(void) { return false; }static inline void lockdep_softirq_end (bool in_hardirq) { }#endif -asmlinkage __visible void __softirq_entry __do_softirq(void) +asmlinkage __visible void __softirq_entry __do_softirq(__u32 mask) {unsigned long end = jiffies +MAX_SOFTIRQ_TIME ;unsigned long old_flags = current->flags;
@@ -265,7 +265,8 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) restart:/* Reset the pending bitmask before enabling irqs */ -set_softirq_pending (0 ); +set_softirq_pending (pending & ~mask); + pending &= mask;local_irq_enable ();
@@ -299,7 +300,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)local_irq_disable (); pending =local_softirq_pending (); -if (pending) { +if (pending & mask) {if (time_before (jiffies, end) && !need_resched () && --max_restart) goto restart;
@@ -316,18 +317,16 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) asmlinkage __visible void do_softirq(void) { -__u32 pending; +__u32 pending =local_softirq_pending ();unsigned long flags; -if (in_interrupt ()) +if (in_interrupt () || !pending)return ;local_irq_save (flags); - pending =local_softirq_pending (); - -if (pending && !ksoftirqd_running ()) -do_softirq_own_stack (); +if (!ksoftirqd_running ()) +do_softirq_own_stack (pending);local_irq_restore (flags); }
@@ -353,7 +352,9 @@ void irq_enter(void)static inline void invoke_softirq (void ) { -if (ksoftirqd_running ()) +__u32 pending =local_softirq_pending (); + +if (!pending || !ksoftirqd_running ())return ;if (!force_irqthreads) {
@@ -363,14 +364,14 @@ static inline void invoke_softirq(void)* it is the irq stack, because it should be near empty * at this stage. */ -__do_softirq (); +__do_softirq (pending);#else /* * Otherwise, irq_exit() is called on the task stack that can * be potentially deep already. So call softirq in its own stack * to prevent from any overrun. */ -do_softirq_own_stack (); +do_softirq_own_stack (pending);#endif }else {wakeup_softirqd ();
@@ -679,7 +680,7 @@ static void run_ksoftirqd(unsigned int cpu)* We can safely run softirq on inline stack, as we are not deep * in the task stack here. */ -__do_softirq (); +__do_softirq (~0 );local_irq_enable ();cond_resched_rcu_qs ();return ;
Signature · 1 line
2.13.6
RFC 3/6 softirq: Add reverse group-to-softirq map
In reply to Dmitry SafonovFor faster operation with pending mask:
pending &= group_to_softirq[group_nr];
--- kernel/softirq.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/softirq.c +18 -0
@@ -54,6 +54,7 @@ EXPORT_SYMBOL(irq_stat);#endif static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp; +static unsigned group_to_softirqs[sizeof(softirq_vec[0].group_mask)] __cacheline_aligned_in_smp;static unsigned __initdata nr_softirq_groups =0 ; DEFINE_PER_CPU(struct task_struct *, ksoftirqd);
@@ -650,11 +651,28 @@ static void __init setup_default_softirq_group(unsigned nr) } } +static void __init fill_group_to_softirq_maps (void ) +{ +unsigned i; + +for (i =0 ; i <NR_SOFTIRQS ; i++) { +u32 mask = softirq_vec[i].group_mask; +unsigned j, group =0 ; + +while ((j =ffs (mask))) { + group += j -1 ; + group_to_softirqs[group] |= (1 << i); + mask >>= j; + } + } +} +void __init softirq_init (void ) {int cpu;setup_default_softirq_group (nr_softirq_groups++); +fill_group_to_softirq_maps ();for_each_possible_cpu (cpu) { per_cpu(tasklet_vec, cpu).tail =
Signature · 1 line
2.13.6
RFC 4/6 softirq: Run per-group per-cpu ksoftirqd thread
In reply to Dmitry SafonovRunning one ksoftirqd per-cpu allows to defer processing softirqs under storm. But having only one ksoftirqd thread for that make it worse for other kinds of softirqs. As we check if (ksoftirqd_running()) and defer all softirqs till ksoftirqd time-slice it introduces latencies. While it's acceptable for softirqs under storm, the other softirqs are impeded.
For each softirq-group create ksoftirqd thread which will serve deferred softirqs of the group. Softirqs of other groups will be served as they come.
Without kernel parameter it'll work as it was previously: creating default softirq-group which includes all softirqs and running only one ksoftirqd thread per-cpu.
--- include/linux/interrupt.h | 16 +++- kernel/sched/cputime.c | 27 ++++--- kernel/softirq.c | 187 ++++++++++++++++++++++++++++++++++++---------- net/ipv4/tcp_output.c | 2 +- 4 files changed, 177 insertions(+), 55 deletions(-)
include/linux/interrupt.h +13 -3
@@ -508,11 +508,21 @@ extern void __raise_softirq_irqoff(unsigned int nr);extern void raise_softirq_irqoff (unsigned int nr);extern void raise_softirq (unsigned int nr); -DECLARE_PER_CPU(struct task_struct *, ksoftirqd); +extern struct task_struct *__percpu **ksoftirqd; +extern unsigned nr_softirq_groups; -static inline struct task_struct *this_cpu_ksoftirqd (void ) +extern bool servicing_softirq (unsigned nr); +static inline bool current_is_ksoftirqd (void ) { -return this_cpu_read (ksoftirqd); +unsigned i; + +if (!ksoftirqd) +return false; + +for (i =0 ; i < nr_softirq_groups; i++) +if (*this_cpu_ptr (ksoftirqd[i]) == current) +return true; +return false; }/* Tasklets --- multithreaded analogue of BHs.
kernel/sched/cputime.c +17 -10
@@ -46,6 +46,18 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,u64_stats_update_end (&irqtime->sync); } +static void irqtime_account_softirq (struct irqtime *irqtime,s64 delta) +{ +/* +* We do not account for softirq time from ksoftirqd here. +* We want to continue accounting softirq time to ksoftirqd thread +* in that case, so as not to confuse scheduler with a special task +* that do not consume any time, but still wants to run. +*/ +if (!current_is_ksoftirqd ()) +irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); +} +/* * Called before incrementing preempt_count on {soft,}irq_enter * and before decrementing preempt_count on {soft,}irq_exit.
@@ -63,16 +75,11 @@ void irqtime_account_irq(struct task_struct *curr) delta =sched_clock_cpu (cpu) - irqtime->irq_start_time; irqtime->irq_start_time += delta; -/* -* We do not account for softirq time from ksoftirqd here. -* We want to continue accounting softirq time to ksoftirqd thread -* in that case, so as not to confuse scheduler with a special task -* that do not consume any time, but still wants to run. -*/ -if (hardirq_count ()) +if (hardirq_count ()) {irqtime_account_delta (irqtime, delta,CPUTIME_IRQ ); -else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ()) -irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); + }else if (in_serving_softirq ()) { +irqtime_account_softirq (irqtime, delta); + } }EXPORT_SYMBOL_GPL (irqtime_account_irq);
@@ -375,7 +382,7 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick, cputime -= other; -if (this_cpu_ksoftirqd () == p) { +if (current_is_ksoftirqd ()) {/* * ksoftirqd time do not get accounted in cpu_softirq_time. * So, we have to handle it separately here.
kernel/softirq.c +146 -41
249 lines not on this page · raw message
net/ipv4/tcp_output.c +1 -1
@@ -919,7 +919,7 @@ void tcp_wfree(struct sk_buff *skb)* - chance for incoming ACK (processed by another cpu maybe) * to migrate this flow (skb->ooo_okay will be eventually set) */ -if (refcount_read (&sk->sk_wmem_alloc) >=SKB_TRUESIZE (1 ) &&this_cpu_ksoftirqd () == current) +if (refcount_read (&sk->sk_wmem_alloc) >=SKB_TRUESIZE (1 ) &&servicing_softirq (NET_TX_SOFTIRQ )) goto out;for (oval =READ_ONCE (sk->sk_tsq_flags);; oval = nval) {
Signature · 1 line
2.13.6
Re: [RFC 4/6] softirq: Run per-group per-cpu ksoftirqd thread
Quoted 19 lines · Dmitry Safonov
include/linux/interrupt.h +5
-3@@ -508,11 +508,21 @@ extern void __raise_softirq_irqoff(unsigned int nr);extern void raise_softirq_irqoff (unsigned int nr);extern void raise_softirq (unsigned int nr); -DECLARE_PER_CPU(struct task_struct *, ksoftirqd); +extern struct task_struct *__percpu **ksoftirqd; +extern unsigned nr_softirq_groups; -static inline struct task_struct *this_cpu_ksoftirqd (void ) +extern bool servicing_softirq (unsigned nr); +static inline bool current_is_ksoftirqd (void ) { -return this_cpu_read (ksoftirqd); +unsigned i;
include/linux/interrupt.h extern void __raise_softirq_irqoff(unsigned int nr); · line 527
Quoted 10 lines
include/linux/interrupt.h +8
-0+ +if (!ksoftirqd) +return false; + +for (i =0 ; i < nr_softirq_groups; i++) +if (*this_cpu_ptr (ksoftirqd[i]) == current) +return true; +return false; }
I haven't read all this, but in a quick drive-by this poked me in the eye. For RT tree fully threaded softirqs, I stole a ->flags bit to identify threads ala PF_KTHREAD (PF_KSOFTIRQD). In previous versions, I added a bit field to do the same, either is quicker than rummaging.
-Mike
Re: [RFC 4/6] softirq: Run per-group per-cpu ksoftirqd thread
Quoted 29 lines · Mike Galbraith
Quoted 28 lines · Dmitry Safonov
include/linux/interrupt.h +12
-3@@ -508,11 +508,21 @@ extern void __raise_softirq_irqoff(unsigned int nr);extern void raise_softirq_irqoff (unsigned int nr);extern void raise_softirq (unsigned int nr); -DECLARE_PER_CPU(struct task_struct *, ksoftirqd); +extern struct task_struct *__percpu **ksoftirqd; +extern unsigned nr_softirq_groups; -static inline struct task_struct *this_cpu_ksoftirqd (void ) +extern bool servicing_softirq (unsigned nr); +static inline bool current_is_ksoftirqd (void ) { -return this_cpu_read (ksoftirqd); +unsigned i; + +if (!ksoftirqd) +return false; + +for (i =0 ; i < nr_softirq_groups; i++) +if (*this_cpu_ptr (ksoftirqd[i]) == current) +return true;
Quoted 8 lines
include/linux/interrupt.h extern void __raise_softirq_irqoff(unsigned · line 528
Quoted 2 lines
include/linux/interrupt.h +1
-0+return false; }I haven't read all this, but in a quick drive-by this poked me in the eye. For RT tree fully threaded softirqs, I stole a ->flags bit to identify threads ala
PF_KTHREAD(PF_KSOFTIRQD). In previous versions,
I added a bit field to do the same, either is quicker than rummaging.
Yeah, thank you. It perfectly makes sense to use flags to identify ksoftirqd thread. How do you identify in RT one ksoftirqd thread from another? I mean, to find which softirq nr the thread is servicing?
Signature · 2 lines
Thanks,
DmitryRe: [RFC 4/6] softirq: Run per-group per-cpu ksoftirqd thread
Quoted 2 lines · Dmitry Safonov
How do you identify in RT one ksoftirqd thread from another? I mean, to find which softirq nr the thread is servicing?
static void do_raise_softirq_irqoff(unsigned int nr)
{
struct task_struct *tsk = __this_cpu_ksoftirqd(nr);
unsigned int mask = 1UL << nr;
trace_softirq_raise(nr);
or_softirq_pending(mask);
/*
* If we are not in a hard interrupt and inside a bh disabled
* region, we simply raise the flag on current. local_bh_enable()
* will make sure that the softirq is executed. Otherwise we
* delegate it to the proper softirqd thread for this softirq.
*/
if (!in_irq() && current->softirq_nestcnt) {
if (!(current->flags & PF_KSOFTIRQD) || current == tsk)
current->softirqs_raised |= mask;
else if (tsk) {
tsk->softirqs_raised |= mask;
wakeup_softirqd(nr);
}
} else if (tsk)
tsk->softirqs_raised |= mask;
}RFC 5/6 softirq: Add time accounting per-softirq type
In reply to Dmitry SafonovWarning: not a merge-ready in any sense
As discussed, softirqs will be deferred or processed right away according to how much time this type of softirq spent on CPU. This will improve e.g. handling of net-rx softirqs during packet storm and also give fair slice of cpu time for a userspace process to serve incoming packages.
The time-based decision will work better than check of re-raised softirq after processing previous one. Because the check might not work even under softirq storm if softirqs are raised too slowly (e.g. because of hw).
--- include/linux/hardirq.h | 2 +- include/linux/vtime.h | 10 +++++----- init/Kconfig | 10 ++++++++++ kernel/sched/cputime.c | 41 ++++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 1 + kernel/softirq.c | 16 ++++++++++++++-- 6 files changed, 65 insertions(+), 15 deletions(-)
include/linux/hardirq.h +1 -1
@@ -51,7 +51,7 @@ extern void irq_enter(void);#define __irq_exit () \ do { \ trace_hardirq_exit(); \ - account_irq_exit_time(current); \ + account_irq_exit_time(current, 0); \ preempt_count_sub(HARDIRQ_OFFSET); \ } while (0)
include/linux/vtime.h +5 -5
@@ -97,21 +97,21 @@ static inline void vtime_flush(struct task_struct *tsk) { }#ifdef CONFIG_IRQ_TIME_ACCOUNTING -extern void irqtime_account_irq (struct task_struct *tsk); +extern void irqtime_account_irq (struct task_struct *tsk,u64 *si_times);#else -static inline void irqtime_account_irq (struct task_struct *tsk) { } +static inline void irqtime_account_irq (struct task_struct *tsk,u64 *si_times) { }#endif static inline void account_irq_enter_time (struct task_struct *tsk) {vtime_account_irq_enter (tsk); -irqtime_account_irq (tsk); +irqtime_account_irq (tsk,0 ); } -static inline void account_irq_exit_time (struct task_struct *tsk) +static inline void account_irq_exit_time (struct task_struct *tsk,u64 *si_times) {vtime_account_irq_exit (tsk); -irqtime_account_irq (tsk); +irqtime_account_irq (tsk, si_times); }#endif /* _LINUX_KERNEL_VTIME_H */
init/Kconfig +10 -0
@@ -387,6 +387,16 @@ config IRQ_TIME_ACCOUNTING If in doubt, say N here. +config FAIR_SOFTIRQ_SCHEDULE + bool "Fair schedule softirqs on process context" + depends on IRQ_TIME_ACCOUNTING + default n + help + Account softirq CPU time per softirq-type. Process pending softirq + on current context only if it'll be fair for the task. + + If in doubt, say N here. + config BSD_PROCESS_ACCT bool "BSD Process Accounting" depends on MULTIUSER
kernel/sched/cputime.c +34 -7
78 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -2055,6 +2055,7 @@ struct irqtime {u64 total;u64 tick_delta;u64 irq_start_time; +u64 total_si[NR_SOFTIRQS ];struct u64_stats_sync sync; };
kernel/softirq.c +14 -2
@@ -22,6 +22,7 @@#include <linux/kthread.h> #include <linux/rcupdate.h> #include <linux/ftrace.h> +#include <linux/sched/clock.h> #include <linux/smp.h> #include <linux/smpboot.h> #include <linux/tick.h>
@@ -287,6 +288,14 @@ static inline bool lockdep_softirq_start(void) { return false; }static inline void lockdep_softirq_end (bool in_hardirq) { }#endif +static inline u64 time_softirq (u64 start) +{ +#ifdef CONFIG_FAIR_SOFTIRQ_SCHEDULE +return local_clock () - start; +#endif +return 0 ; +} + asmlinkage __visible void __softirq_entry __do_softirq(__u32 mask) {unsigned long end = jiffies +MAX_SOFTIRQ_TIME ;
@@ -296,6 +305,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(__u32 mask)bool in_hardirq;__u32 pending;int softirq_bit; +u64 si_times[NR_SOFTIRQS ] = {0 };/* * Mask out PF_MEMALLOC s current task context is borrowed for the
@@ -322,6 +332,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(__u32 mask)while ((softirq_bit =ffs (pending))) {unsigned int vec_nr;int prev_count; +u64 start_time =time_softirq (0 ); h += softirq_bit -1 ;
@@ -341,6 +352,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(__u32 mask) } h++; pending >>= softirq_bit; + si_times[vec_nr] +=time_softirq (start_time); }rcu_bh_qs ();
@@ -357,7 +369,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(__u32 mask) }lockdep_softirq_end (in_hardirq); -account_irq_exit_time (current); +account_irq_exit_time (current, si_times);__local_bh_enable (SOFTIRQ_OFFSET );WARN_ON_ONCE (in_interrupt ());current_restore_flags (old_flags,PF_MEMALLOC );
@@ -449,7 +461,7 @@ void irq_exit(void)#else lockdep_assert_irqs_disabled ();#endif -account_irq_exit_time (current); +account_irq_exit_time (current,0 );preempt_count_sub (HARDIRQ_OFFSET );if (!in_interrupt () &&local_softirq_pending ())invoke_softirq ();
Signature · 1 line
2.13.6
RFC 6/6 softirq/sched: Account si cpu time to ksoftirqd(s)
In reply to Dmitry SafonovWarning: non-merge-ready in any sense
Under CONFIG_FAIR_SOFTIRQ_SCHEDULE each sched tick will account cpu time spent on processing softirqs to ksoftirqd of the softirq's group. Update then ksoftirqd->se.sum_exec_runtime and recalculate ksoftirqd->se.vruntime.
Use CFS's vrutime to decide if softirq needs to be served or deferred. It's possible to tune this with ksoftirqd nice policy.
--- include/linux/interrupt.h | 1 + kernel/sched/fair.c | 38 ++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 19 +++++++++++++++++++ kernel/softirq.c | 45 +++++++++++++++++++++++++++++++++++++-------- 4 files changed, 95 insertions(+), 8 deletions(-)
include/linux/interrupt.h +1 -0
@@ -512,6 +512,7 @@ extern struct task_struct *__percpu **ksoftirqd;extern unsigned nr_softirq_groups;extern bool servicing_softirq (unsigned nr); +extern unsigned group_softirqs (unsigned nr);static inline bool current_is_ksoftirqd (void ) {unsigned i;
kernel/sched/fair.c +38 -0
@@ -813,6 +813,42 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq, int force) }#endif /* CONFIG_SMP */ +static void update_ksoftirqd (struct cfs_rq *cfs_rq) +{ +#ifdef CONFIG_FAIR_SOFTIRQ_SCHEDULE +int rq_cpu =cpu_of (rq_of (cfs_rq)); +u64 si_times[NR_SOFTIRQS ], delta[NR_SOFTIRQS ]; +unsigned i; + +if (unlikely (!ksoftirqd)) +return ; + +softirq_time_read (rq_cpu, si_times); + +for (i =0 ; i <NR_SOFTIRQS ; i++) { + delta[i] = si_times[i] - cfs_rq->prev_si_time[i]; + cfs_rq->prev_si_time[i] = si_times[i]; +if (unlikely ((s64 )delta[i] <0 )) + delta[i] =0 ; + } + +for (i =0 ; i < nr_softirq_groups; i++) { +unsigned j, softirq =0 , group_mask =group_softirqs (i); +struct task_struct *tsk = *this_cpu_ptr (ksoftirqd[i]); +u64 sum_delta =0 ; + +while ((j =ffs (group_mask))) { + softirq += j -1 ; + group_mask >>= j; + sum_delta += delta[softirq]; + } + + tsk->se.sum_exec_runtime += sum_delta; + tsk->se.vruntime +=calc_delta_fair (sum_delta, &tsk->se); + } +#endif +} +/* * Update the current task's runtime statistics. */
@@ -822,6 +858,8 @@ static void update_curr(struct cfs_rq *cfs_rq)u64 now =rq_clock_task (rq_of (cfs_rq));u64 delta_exec; +update_ksoftirqd (cfs_rq); +if (unlikely (!curr))return ;
kernel/sched/sched.h +19 -0
@@ -487,6 +487,10 @@ struct cfs_rq {struct list_head leaf_cfs_rq_list;struct task_group *tg;/* group that "owns" this runqueue */ +#ifdef CONFIG_FAIR_SOFTIRQ_SCHEDULE +u64 prev_si_time[NR_SOFTIRQS ]; +#endif +#ifdef CONFIG_CFS_BANDWIDTH int runtime_enabled;u64 runtime_expires;
@@ -2081,6 +2085,21 @@ static inline u64 irq_time_read(int cpu) }#endif /* CONFIG_IRQ_TIME_ACCOUNTING */ +static inline void softirq_time_read (int cpu,u64 si_times[NR_SOFTIRQS ]) +{ +#ifdef CONFIG_FAIR_SOFTIRQ_SCHEDULE +struct irqtime *irqtime = &per_cpu (cpu_irqtime, cpu); +unsigned int seq, i; + +for (i =0 ; i <NR_SOFTIRQS ; i++) { +do { + seq =__u64_stats_fetch_begin (&irqtime->sync); + si_times[i] = irqtime->total_si[i]; + }while (__u64_stats_fetch_retry (&irqtime->sync, seq)); + } +#endif +} +#ifdef CONFIG_CPU_FREQ DECLARE_PER_CPU(struct update_util_data *, cpufreq_update_util_data);
kernel/softirq.c +37 -8
78 lines not on this page · raw message
Signature · 1 line
2.13.6