irq: Reorder time handling against HARDIRQ_OFFSET on IRQ entry v3
cover irq: Reorder time handling against HARDIRQ_OFFSET on IRQ entry v3
Slightly different design this time. As per Peter and Thomas reviews and suggestions, use the following layout:
account_softirq_enter() -> irqtime_account_irq(curr, SOFTIRQ_OFFSET); account_softirq_exit() -> irqtime_account_irq(curr, 0); account_hardirq_enter() -> irqtime_account_irq(curr, HARDIRQ_OFFSET); account_hardirq_exit() -> irqtime_account_irq(curr, 0);
So now the preempt count offset is substracted in the cputime dispatch function and the resulting code is better consolidated.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
irq/core-v3
HEAD: 24a2d6c76759bd4496cbdcd365012c821a984eec
Thanks,
Frederic
---
Frederic Weisbecker (5):
sched/cputime: Remove symbol exports from IRQ time accounting s390/vtime: Use the generic IRQ entry accounting sched/vtime: Consolidate IRQ time accounting
irqtime: Move irqtime entry accounting after irq offset incrementation
irq: Call tick_irq_enter() inside HARDIRQ_OFFSET
arch/Kconfig | 7 +++++- arch/ia64/kernel/time.c | 20 ++++++++++++---- arch/powerpc/kernel/time.c | 56 ++++++++++++++++++++++++++++++------------- arch/s390/Kconfig | 1 + arch/s390/include/asm/vtime.h | 1 - arch/s390/kernel/vtime.c | 51 +++++++++++++++++++++++++-------------- include/linux/hardirq.h | 4 ++-- include/linux/vtime.h | 42 +++++++++++++++++++------------- kernel/sched/cputime.c | 36 +++++++++++++--------------- kernel/softirq.c | 20 +++++++--------- 10 files changed, 148 insertions(+), 90 deletions(-)
PATCH 1/5 sched/cputime: Remove symbol exports from IRQ time accounting
account_irq_enter_time() and account_irq_exit_time() are not called from modules. EXPORT_SYMBOL_GPL() can be safely removed from the IRQ cputime accounting functions called from there.
--- arch/s390/kernel/vtime.c | 10 +++++----- kernel/sched/cputime.c | 2 -- 2 files changed, 5 insertions(+), 7 deletions(-)
arch/s390/kernel/vtime.c +5 -5
@@ -226,7 +226,7 @@ void vtime_flush(struct task_struct *tsk)* Update process times based on virtual cpu times stored by entry.S * to the lowcore fields user_timer, system_timer & steal_clock. */ -void vtime_account_irq_enter (struct task_struct *tsk) +void vtime_account_kernel (struct task_struct *tsk) {u64 timer;
@@ -245,12 +245,12 @@ void vtime_account_irq_enter(struct task_struct *tsk)virt_timer_forward (timer); } -EXPORT_SYMBOL_GPL (vtime_account_irq_enter); - -void vtime_account_kernel (struct task_struct *tsk) -__attribute__((alias ("vtime_account_irq_enter" )));EXPORT_SYMBOL_GPL (vtime_account_kernel); +void vtime_account_irq_enter (struct task_struct *tsk) +__attribute__((alias ("vtime_account_kernel" ))); + +/* * Sorted add to a list. List is linear searched until first bigger * element is found.
kernel/sched/cputime.c +0 -2
@@ -71,7 +71,6 @@ void irqtime_account_irq(struct task_struct *curr)else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ())irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); } -EXPORT_SYMBOL_GPL (irqtime_account_irq);static u64 irqtime_tick_accounted (u64 maxtime) {
@@ -434,7 +433,6 @@ void vtime_account_irq_enter(struct task_struct *tsk) elsevtime_account_kernel (tsk ); } -EXPORT_SYMBOL_GPL (vtime_account_irq_enter);#endif /* __ARCH_HAS_VTIME_ACCOUNT */ void cputime_adjust(struct task_cputime *curr, struct prev_cputime *prev,
Signature · 1 line
2.25.1
PATCH 2/5 s390/vtime: Use the generic IRQ entry accounting
In reply to Frederic Weisbeckers390 has its own version of IRQ entry accounting because it doesn't account the idle time the same way the other architectures do. Only the actual idle sleep time is accounted as idle time, the rest of the idle task execution is accounted as system time.
Make the generic IRQ entry accounting aware of architectures that have their own way of accounting idle time and convert s390 to use it.
This prepares s390 to get involved in further consolidations of IRQ time accounting.
--- arch/Kconfig | 7 ++++++- arch/s390/Kconfig | 1 + arch/s390/include/asm/vtime.h | 1 - arch/s390/kernel/vtime.c | 4 ---- kernel/sched/cputime.c | 13 ++----------- 5 files changed, 9 insertions(+), 17 deletions(-)
arch/Kconfig +6 -1
@@ -627,6 +627,12 @@ config HAVE_TIF_NOHZ config HAVE_VIRT_CPU_ACCOUNTING bool +config HAVE_VIRT_CPU_ACCOUNTING_IDLE + bool + help + Architecture has its own way to account idle CPU time and therefore + doesn't implement vtime_account_idle(). + config ARCH_HAS_SCALED_CPUTIME bool
@@ -641,7 +647,6 @@ config HAVE_VIRT_CPU_ACCOUNTING_GEN some 32-bit arches may require multiple accesses, so proper locking is needed to protect against concurrent accesses. - config HAVE_IRQ_TIME_ACCOUNTING bool help
arch/s390/Kconfig +1 -0
@@ -181,6 +181,7 @@ config S390 select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING + select HAVE_VIRT_CPU_ACCOUNTING_IDLE select IOMMU_HELPER if PCI select IOMMU_SUPPORT if PCI select MODULES_USE_ELF_RELA
arch/s390/include/asm/vtime.h +0 -1
@@ -2,7 +2,6 @@#ifndef _S390_VTIME_H#define _S390_VTIME_H -#define __ARCH_HAS_VTIME_ACCOUNT#define __ARCH_HAS_VTIME_TASK_SWITCH#endif /* _S390_VTIME_H */
arch/s390/kernel/vtime.c +0 -4
@@ -247,10 +247,6 @@ void vtime_account_kernel(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (vtime_account_kernel); -void vtime_account_irq_enter (struct task_struct *tsk) -__attribute__((alias ("vtime_account_kernel" ))); - -/* * Sorted add to a list. List is linear searched until first bigger * element is found.
kernel/sched/cputime.c +2 -11
@@ -417,23 +417,14 @@ void vtime_task_switch(struct task_struct *prev) }# endif -/* -* Archs that account the whole time spent in the idle task -* (outside irq) as idle time can rely on this and just implement -* vtime_account_kernel() and vtime_account_idle(). Archs that -* have other meaning of the idle time (s390 only includes the -* time spent by the CPU when it's in low power mode) must override -* vtime_account(). -*/ -#ifndef __ARCH_HAS_VTIME_ACCOUNTvoid vtime_account_irq_enter (struct task_struct *tsk) { -if (!in_interrupt () &&is_idle_task (tsk)) +if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && + !in_interrupt () &&is_idle_task (tsk))vtime_account_idle (tsk);else vtime_account_kernel (tsk); } -#endif /* __ARCH_HAS_VTIME_ACCOUNT */ void cputime_adjust (struct task_cputime *curr,struct prev_cputime *prev,u64 *ut,u64 *st)
Signature · 1 line
2.25.1
PATCH 3/5 sched/vtime: Consolidate IRQ time accounting
In reply to Frederic WeisbeckerThe 3 architectures implementing CONFIG_VIRT_CPU_ACCOUNTING_NATIVE all have their own version of irq time accounting that dispatch the cputime to the appropriate index: hardirq, softirq, system, idle, guest... from an all-in-one function.
Instead of having these ad-hoc versions, move the cputime destination dispatch decision to the core code and leave only the actual per-index cputime accounting to the architecture.
--- arch/ia64/kernel/time.c | 20 +++++++++---- arch/powerpc/kernel/time.c | 58 +++++++++++++++++++++++++++----------- arch/s390/kernel/vtime.c | 51 ++++++++++++++++++++++----------- include/linux/vtime.h | 16 ++++------- kernel/sched/cputime.c | 13 ++++++--- 5 files changed, 106 insertions(+), 52 deletions(-)
arch/ia64/kernel/time.c +15 -5
@@ -138,12 +138,8 @@ void vtime_account_kernel(struct task_struct *tsk)struct thread_info *ti =task_thread_info (tsk);__u64 stime =vtime_delta (tsk); -if ((tsk->flags &PF_VCPU ) && !irq_count ()) +if (tsk->flags &PF_VCPU ) ti->gtime += stime; -else if (hardirq_count ()) - ti->hardirq_time += stime; -else if (in_serving_softirq ()) - ti->softirq_time += stime;else ti->stime += stime; }
@@ -156,6 +152,20 @@ void vtime_account_idle(struct task_struct *tsk) ti->idle_time +=vtime_delta (tsk); } +void vtime_account_softirq (struct task_struct *tsk) +{ +struct thread_info *ti =task_thread_info (tsk); + + ti->softirq_time +=vtime_delta (tsk); +} + +void vtime_account_hardirq (struct task_struct *tsk) +{ +struct thread_info *ti =task_thread_info (tsk); + + ti->hardirq_time +=vtime_delta (tsk); +} +#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ static irqreturn_t
arch/powerpc/kernel/time.c +41 -17
92 lines not on this page · raw message
arch/s390/kernel/vtime.c +35 -16
66 lines not on this page · raw message
include/linux/vtime.h +6 -10
@@ -83,16 +83,12 @@ static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { }#endif #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE -extern void vtime_account_irq_enter (struct task_struct *tsk); -static inline void vtime_account_irq_exit (struct task_struct *tsk) -{ -/* On hard|softirq exit we always account to hard|softirq cputime */ -vtime_account_kernel (tsk); -} +extern void vtime_account_irq (struct task_struct *tsk); +extern void vtime_account_softirq (struct task_struct *tsk); +extern void vtime_account_hardirq (struct task_struct *tsk);extern void vtime_flush (struct task_struct *tsk);#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ -static inline void vtime_account_irq_enter (struct task_struct *tsk) { } -static inline void vtime_account_irq_exit (struct task_struct *tsk) { } +static inline void vtime_account_irq (struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { }#endif
@@ -105,13 +101,13 @@ static inline void irqtime_account_irq(struct task_struct *tsk) { }static inline void account_irq_enter_time (struct task_struct *tsk) { -vtime_account_irq_enter (tsk); +vtime_account_irq (tsk);irqtime_account_irq (tsk); }static inline void account_irq_exit_time (struct task_struct *tsk) { -vtime_account_irq_exit (tsk); +vtime_account_irq (tsk);irqtime_account_irq (tsk); }
kernel/sched/cputime.c +9 -4
@@ -417,13 +417,18 @@ void vtime_task_switch(struct task_struct *prev) }# endif -void vtime_account_irq_enter (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk) { -if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && - !in_interrupt () &&is_idle_task (tsk)) +if (hardirq_count ()) { +vtime_account_hardirq (tsk); + }else if (in_serving_softirq ()) { +vtime_account_softirq (tsk); + }else if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && +is_idle_task (tsk)) {vtime_account_idle (tsk); -else + }else {vtime_account_kernel (tsk); + } } void cputime_adjust(struct task_cputime *curr, struct prev_cputime *prev,
Signature · 1 line
2.25.1
PATCH 4/5 irqtime: Move irqtime entry accounting after irq offset incrementation
In reply to Frederic WeisbeckerIRQ time entry is currently accounted before HARDIRQ_OFFSET or SOFTIRQ_OFFSET are incremented. This is convenient to decide to which index the cputime to account is dispatched.
Unfortunately it prevents tick_irq_enter() from being called under HARDIRQ_OFFSET because tick_irq_enter() has to be called before the IRQ entry accounting due to the necessary clock catch up. As a result we don't benefit from appropriate lockdep coverage on tick_irq_enter().
To prepare for fixing this, move the IRQ entry cputime accounting after the preempt offset is incremented. This requires the cputime dispatch code to handle the extra offset.
--- include/linux/hardirq.h | 4 ++-- include/linux/vtime.h | 34 ++++++++++++++++++++++++---------- kernel/sched/cputime.c | 18 +++++++++++------- kernel/softirq.c | 6 +++--- 4 files changed, 40 insertions(+), 22 deletions(-)
include/linux/hardirq.h +2 -2
@@ -32,9 +32,9 @@ static __always_inline void rcu_irq_enter_check_tick(void)*/ #define __irq_enter () \ do { \ - account_irq_enter_time(current); \ preempt_count_add(HARDIRQ_OFFSET); \ lockdep_hardirq_enter(); \ + account_hardirq_enter(current); \ } while (0)/*
@@ -62,8 +62,8 @@ void irq_enter_rcu(void);*/ #define __irq_exit () \ do { \ + account_hardirq_exit(current); \ lockdep_hardirq_exit(); \ - account_irq_exit_time(current); \ preempt_count_sub(HARDIRQ_OFFSET); \ } while (0)
include/linux/vtime.h +24 -10
@@ -83,32 +83,46 @@ static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { }#endif #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE -extern void vtime_account_irq (struct task_struct *tsk); +extern void vtime_account_irq (struct task_struct *tsk,unsigned int offset);extern void vtime_account_softirq (struct task_struct *tsk);extern void vtime_account_hardirq (struct task_struct *tsk);extern void vtime_flush (struct task_struct *tsk);#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ -static inline void vtime_account_irq (struct task_struct *tsk) { } +static inline void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { } +static inline void vtime_account_softirq (struct task_struct *tsk) { } +static inline void vtime_account_hardirq (struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { }#endif #ifdef CONFIG_IRQ_TIME_ACCOUNTING -extern void irqtime_account_irq (struct task_struct *tsk); +extern void irqtime_account_irq (struct task_struct *tsk,unsigned int offset);#else -static inline void irqtime_account_irq (struct task_struct *tsk) { } +static inline void irqtime_account_irq (struct task_struct *tsk,unsigned int offset) { }#endif -static inline void account_irq_enter_time (struct task_struct *tsk) +static inline void account_softirq_enter (struct task_struct *tsk) { -vtime_account_irq (tsk); -irqtime_account_irq (tsk); +vtime_account_irq (tsk,SOFTIRQ_OFFSET ); +irqtime_account_irq (tsk,SOFTIRQ_OFFSET ); } -static inline void account_irq_exit_time (struct task_struct *tsk) +static inline void account_softirq_exit (struct task_struct *tsk) { -vtime_account_irq (tsk); -irqtime_account_irq (tsk); +vtime_account_softirq (tsk); +irqtime_account_irq (tsk,0 ); +} + +static inline void account_hardirq_enter (struct task_struct *tsk) +{ +vtime_account_irq (tsk,HARDIRQ_OFFSET ); +irqtime_account_irq (tsk,HARDIRQ_OFFSET ); +} + +static inline void account_hardirq_exit (struct task_struct *tsk) +{ +vtime_account_hardirq (tsk); +irqtime_account_irq (tsk,0 ); }#endif /* _LINUX_KERNEL_VTIME_H */
kernel/sched/cputime.c +11 -7
@@ -44,12 +44,13 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta, }/* -* Called before incrementing preempt_count on {soft,}irq_enter +* Called after incrementing preempt_count on {soft,}irq_enter * and before decrementing preempt_count on {soft,}irq_exit. */ -void irqtime_account_irq (struct task_struct *curr) +void irqtime_account_irq (struct task_struct *curr,unsigned int offset) {struct irqtime *irqtime =this_cpu_ptr (&cpu_irqtime); +unsigned int pc;s64 delta;int cpu;
@@ -59,6 +60,7 @@ void irqtime_account_irq(struct task_struct *curr) cpu =smp_processor_id (); delta =sched_clock_cpu (cpu) - irqtime->irq_start_time; irqtime->irq_start_time += delta; + pc =preempt_count () - offset;/* * We do not account for softirq time from ksoftirqd here.
@@ -66,9 +68,9 @@ void irqtime_account_irq(struct task_struct *curr)* in that case, so as not to confuse scheduler with a special task * that do not consume any time, but still wants to run. */ -if (hardirq_count ()) +if (pc &HARDIRQ_MASK )irqtime_account_delta (irqtime, delta,CPUTIME_IRQ ); -else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ()) +else if ((pc &SOFTIRQ_OFFSET ) && curr !=this_cpu_ksoftirqd ())irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); }
@@ -417,11 +419,13 @@ void vtime_task_switch(struct task_struct *prev) }# endif -void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {vtime_account_hardirq (tsk); - }else if (in_serving_softirq ()) { + }else if (pc &SOFTIRQ_OFFSET ) {vtime_account_softirq (tsk); }else if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) &&is_idle_task (tsk)) {
kernel/softirq.c +3 -3
@@ -315,10 +315,10 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) current->flags &= ~PF_MEMALLOC ; pending =local_softirq_pending (); -account_irq_enter_time (current);__local_bh_disable_ip (_RET_IP_,SOFTIRQ_OFFSET ); in_hardirq =lockdep_softirq_start (); +account_softirq_enter (current); restart:/* Reset the pending bitmask before enabling irqs */
@@ -365,8 +365,8 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)wakeup_softirqd (); } +account_softirq_exit (current);lockdep_softirq_end (in_hardirq); -account_irq_exit_time (current);__local_bh_enable (SOFTIRQ_OFFSET );WARN_ON_ONCE (in_interrupt ());current_restore_flags (old_flags,PF_MEMALLOC );
@@ -418,7 +418,7 @@ static inline void __irq_exit_rcu(void)#else lockdep_assert_irqs_disabled ();#endif -account_irq_exit_time (current); +account_hardirq_exit (current);preempt_count_sub (HARDIRQ_OFFSET );if (!in_interrupt () &&local_softirq_pending ())invoke_softirq ();
Signature · 1 line
2.25.1
PATCH 5/5 irq: Call tick_irq_enter() inside HARDIRQ_OFFSET
In reply to Frederic WeisbeckerNow that account_hardirq_enter() is called after HARDIRQ_OFFSET has been incremented, there is nothing left that prevents us from also moving tick_irq_enter() after HARDIRQ_OFFSET is incremented.
The desired outcome is to remove the nasty hack that prevents softirqs from being raised through ksoftirqd instead of the hardirq bottom half. Also tick_irq_enter() then becomes appropriately covered by lockdep.
--- kernel/softirq.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
kernel/softirq.c +5 -9
@@ -377,16 +377,12 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)*/ void irq_enter_rcu (void ) { -if (is_idle_task (current) && !in_interrupt ()) { -/* -* Prevent raise_softirq from needlessly waking up ksoftirqd -* here, as softirq will be serviced on return from interrupt. -*/ -local_bh_disable (); +__irq_enter_raw (); + +if (is_idle_task (current) && (irq_count () ==HARDIRQ_OFFSET ))tick_irq_enter (); -_local_bh_enable (); - } -__irq_enter (); + +account_hardirq_enter (current); }/**
Signature · 1 line
2.25.1
Re: [PATCH 4/5] irqtime: Move irqtime entry accounting after irq offset incrementation
In reply to Frederic WeisbeckerQuoted 9 lines · Frederic Weisbecker
IRQ time entry is currently accounted before
HARDIRQ_OFFSETorSOFTIRQ_OFFSETare incremented. This is convenient to decide to which index the cputime to account is dispatched.Unfortunately it prevents
tick_irq_enter()from being called underHARDIRQ_OFFSETbecausetick_irq_enter()has to be called before the IRQ entry accounting due to the necessary clock catch up. As a result we don't benefit from appropriate lockdep coverage ontick_irq_enter().
Quoted 5 lines
To prepare for fixing this, move the IRQ entry cputime accounting after the preempt offset is incremented. This requires the cputime dispatch code to handle the extra offset.
Signed-off-by Frederic Weisbecker <[email protected]>
[tip: irq/core] sched/cputime: Remove symbol exports from IRQ time accounting
In reply to Frederic WeisbeckerThe following commit has been merged into the irq/core branch of tip:
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 7197688b2006357da75a014e0a76be89ca9c2d46 Gitweb: https://git.kernel.org/tip/7197688b2006357da75a014e0a76be89ca9c2d46 Author: Frederic Weisbecker <[email protected]> AuthorDate: Wed, 02 Dec 2020 12:57:28 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 02 Dec 2020 20:20:04 +01:00
sched/cputime: Remove symbol exports from IRQ time accounting
account_irq_enter_time() and account_irq_exit_time() are not called from modules. EXPORT_SYMBOL_GPL() can be safely removed from the IRQ cputime accounting functions called from there.
--- arch/s390/kernel/vtime.c | 10 +++++----- kernel/sched/cputime.c | 2 -- 2 files changed, 5 insertions(+), 7 deletions(-)
arch/s390/kernel/vtime.c +5 -5
@@ -226,7 +226,7 @@ void vtime_flush(struct task_struct *tsk)* Update process times based on virtual cpu times stored by entry.S * to the lowcore fields user_timer, system_timer & steal_clock. */ -void vtime_account_irq_enter (struct task_struct *tsk) +void vtime_account_kernel (struct task_struct *tsk) {u64 timer;
@@ -245,12 +245,12 @@ void vtime_account_irq_enter(struct task_struct *tsk)virt_timer_forward (timer); } -EXPORT_SYMBOL_GPL (vtime_account_irq_enter); - -void vtime_account_kernel (struct task_struct *tsk) -__attribute__((alias ("vtime_account_irq_enter" )));EXPORT_SYMBOL_GPL (vtime_account_kernel); +void vtime_account_irq_enter (struct task_struct *tsk) +__attribute__((alias ("vtime_account_kernel" ))); + +/* * Sorted add to a list. List is linear searched until first bigger * element is found.
kernel/sched/cputime.c +0 -2
@@ -71,7 +71,6 @@ void irqtime_account_irq(struct task_struct *curr)else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ())irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); } -EXPORT_SYMBOL_GPL (irqtime_account_irq);static u64 irqtime_tick_accounted (u64 maxtime) {
@@ -434,7 +433,6 @@ void vtime_account_irq_enter(struct task_struct *tsk) elsevtime_account_kernel (tsk ); } -EXPORT_SYMBOL_GPL (vtime_account_irq_enter);#endif /* __ARCH_HAS_VTIME_ACCOUNT */ void cputime_adjust(struct task_cputime *curr, struct prev_cputime *prev,
[tip: irq/core] irq: Call tick_irq_enter() inside HARDIRQ_OFFSET
In reply to Frederic WeisbeckerThe following commit has been merged into the irq/core branch of tip:
The following commit has been merged into the irq/core branch of tip:
Commit-ID: d14ce74f1fb376ccbbc0b05ded477ada51253729 Gitweb: https://git.kernel.org/tip/d14ce74f1fb376ccbbc0b05ded477ada51253729 Author: Frederic Weisbecker <[email protected]> AuthorDate: Wed, 02 Dec 2020 12:57:32 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 02 Dec 2020 20:20:05 +01:00
irq: Call tick_irq_enter() inside HARDIRQ_OFFSET
Now that account_hardirq_enter() is called after HARDIRQ_OFFSET has been incremented, there is nothing left that prevents us from also moving tick_irq_enter() after HARDIRQ_OFFSET is incremented.
The desired outcome is to remove the nasty hack that prevents softirqs from being raised through ksoftirqd instead of the hardirq bottom half. Also tick_irq_enter() then becomes appropriately covered by lockdep.
--- kernel/softirq.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
kernel/softirq.c +5 -9
@@ -377,16 +377,12 @@ restart:*/ void irq_enter_rcu (void ) { -if (is_idle_task (current) && !in_interrupt ()) { -/* -* Prevent raise_softirq from needlessly waking up ksoftirqd -* here, as softirq will be serviced on return from interrupt. -*/ -local_bh_disable (); +__irq_enter_raw (); + +if (is_idle_task (current) && (irq_count () ==HARDIRQ_OFFSET ))tick_irq_enter (); -_local_bh_enable (); - } -__irq_enter (); + +account_hardirq_enter (current); }/**
[tip: irq/core] s390/vtime: Use the generic IRQ entry accounting
In reply to Frederic WeisbeckerThe following commit has been merged into the irq/core branch of tip:
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 2b91ec9f551b56751cde48792f1c0a1130358844 Gitweb: https://git.kernel.org/tip/2b91ec9f551b56751cde48792f1c0a1130358844 Author: Frederic Weisbecker <[email protected]> AuthorDate: Wed, 02 Dec 2020 12:57:29 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 02 Dec 2020 20:20:04 +01:00
s390/vtime: Use the generic IRQ entry accounting
s390 has its own version of IRQ entry accounting because it doesn't account the idle time the same way the other architectures do. Only the actual idle sleep time is accounted as idle time, the rest of the idle task execution is accounted as system time.
Make the generic IRQ entry accounting aware of architectures that have their own way of accounting idle time and convert s390 to use it.
This prepares s390 to get involved in further consolidations of IRQ time accounting.
--- arch/Kconfig | 7 ++++++- arch/s390/Kconfig | 1 + arch/s390/include/asm/vtime.h | 1 - arch/s390/kernel/vtime.c | 4 ---- kernel/sched/cputime.c | 13 ++----------- 5 files changed, 9 insertions(+), 17 deletions(-)
arch/Kconfig +6 -1
@@ -627,6 +627,12 @@ config HAVE_TIF_NOHZ config HAVE_VIRT_CPU_ACCOUNTING bool +config HAVE_VIRT_CPU_ACCOUNTING_IDLE + bool + help + Architecture has its own way to account idle CPU time and therefore + doesn't implement vtime_account_idle(). + config ARCH_HAS_SCALED_CPUTIME bool
@@ -641,7 +647,6 @@ config HAVE_VIRT_CPU_ACCOUNTING_GEN some 32-bit arches may require multiple accesses, so proper locking is needed to protect against concurrent accesses. - config HAVE_IRQ_TIME_ACCOUNTING bool help
arch/s390/Kconfig +1 -0
@@ -181,6 +181,7 @@ config S390 select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING + select HAVE_VIRT_CPU_ACCOUNTING_IDLE select IOMMU_HELPER if PCI select IOMMU_SUPPORT if PCI select MODULES_USE_ELF_RELA
arch/s390/include/asm/vtime.h +0 -1
@@ -2,7 +2,6 @@#ifndef _S390_VTIME_H#define _S390_VTIME_H -#define __ARCH_HAS_VTIME_ACCOUNT#define __ARCH_HAS_VTIME_TASK_SWITCH#endif /* _S390_VTIME_H */
arch/s390/kernel/vtime.c +0 -4
@@ -247,10 +247,6 @@ void vtime_account_kernel(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (vtime_account_kernel); -void vtime_account_irq_enter (struct task_struct *tsk) -__attribute__((alias ("vtime_account_kernel" ))); - -/* * Sorted add to a list. List is linear searched until first bigger * element is found.
kernel/sched/cputime.c +2 -11
@@ -417,23 +417,14 @@ void vtime_task_switch(struct task_struct *prev) }# endif -/* -* Archs that account the whole time spent in the idle task -* (outside irq) as idle time can rely on this and just implement -* vtime_account_kernel() and vtime_account_idle(). Archs that -* have other meaning of the idle time (s390 only includes the -* time spent by the CPU when it's in low power mode) must override -* vtime_account(). -*/ -#ifndef __ARCH_HAS_VTIME_ACCOUNTvoid vtime_account_irq_enter (struct task_struct *tsk) { -if (!in_interrupt () &&is_idle_task (tsk)) +if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && + !in_interrupt () &&is_idle_task (tsk))vtime_account_idle (tsk);else vtime_account_kernel (tsk); } -#endif /* __ARCH_HAS_VTIME_ACCOUNT */ void cputime_adjust (struct task_cputime *curr,struct prev_cputime *prev,u64 *ut,u64 *st)
[tip: irq/core] sched/vtime: Consolidate IRQ time accounting
In reply to Frederic WeisbeckerThe following commit has been merged into the irq/core branch of tip:
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 8a6a5920d3286eb0eae9f36a4ec4fc9df511eccb Gitweb: https://git.kernel.org/tip/8a6a5920d3286eb0eae9f36a4ec4fc9df511eccb Author: Frederic Weisbecker <[email protected]> AuthorDate: Wed, 02 Dec 2020 12:57:30 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 02 Dec 2020 20:20:05 +01:00
sched/vtime: Consolidate IRQ time accounting
The 3 architectures implementing CONFIG_VIRT_CPU_ACCOUNTING_NATIVE all have their own version of irq time accounting that dispatch the cputime to the appropriate index: hardirq, softirq, system, idle, guest... from an all-in-one function.
Instead of having these ad-hoc versions, move the cputime destination dispatch decision to the core code and leave only the actual per-index cputime accounting to the architecture.
--- arch/ia64/kernel/time.c | 20 +++++++++---- arch/powerpc/kernel/time.c | 56 ++++++++++++++++++++++++++----------- arch/s390/kernel/vtime.c | 45 +++++++++++++++++++++--------- include/linux/vtime.h | 16 ++++------- kernel/sched/cputime.c | 13 ++++++--- 5 files changed, 102 insertions(+), 48 deletions(-)
arch/ia64/kernel/time.c +15 -5
@@ -138,12 +138,8 @@ void vtime_account_kernel(struct task_struct *tsk)struct thread_info *ti =task_thread_info (tsk);__u64 stime =vtime_delta (tsk); -if ((tsk->flags &PF_VCPU ) && !irq_count ()) +if (tsk->flags &PF_VCPU ) ti->gtime += stime; -else if (hardirq_count ()) - ti->hardirq_time += stime; -else if (in_serving_softirq ()) - ti->softirq_time += stime;else ti->stime += stime; }
@@ -156,6 +152,20 @@ void vtime_account_idle(struct task_struct *tsk) ti->idle_time +=vtime_delta (tsk); } +void vtime_account_softirq (struct task_struct *tsk) +{ +struct thread_info *ti =task_thread_info (tsk); + + ti->softirq_time +=vtime_delta (tsk); +} + +void vtime_account_hardirq (struct task_struct *tsk) +{ +struct thread_info *ti =task_thread_info (tsk); + + ti->hardirq_time +=vtime_delta (tsk); +} +#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ static irqreturn_t
arch/powerpc/kernel/time.c +40 -16
91 lines not on this page · raw message
arch/s390/kernel/vtime.c +32 -13
63 lines not on this page · raw message
include/linux/vtime.h +6 -10
@@ -83,16 +83,12 @@ static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { }#endif #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE -extern void vtime_account_irq_enter (struct task_struct *tsk); -static inline void vtime_account_irq_exit (struct task_struct *tsk) -{ -/* On hard|softirq exit we always account to hard|softirq cputime */ -vtime_account_kernel (tsk); -} +extern void vtime_account_irq (struct task_struct *tsk); +extern void vtime_account_softirq (struct task_struct *tsk); +extern void vtime_account_hardirq (struct task_struct *tsk);extern void vtime_flush (struct task_struct *tsk);#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ -static inline void vtime_account_irq_enter (struct task_struct *tsk) { } -static inline void vtime_account_irq_exit (struct task_struct *tsk) { } +static inline void vtime_account_irq (struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { }#endif
@@ -105,13 +101,13 @@ static inline void irqtime_account_irq(struct task_struct *tsk) { }static inline void account_irq_enter_time (struct task_struct *tsk) { -vtime_account_irq_enter (tsk); +vtime_account_irq (tsk);irqtime_account_irq (tsk); }static inline void account_irq_exit_time (struct task_struct *tsk) { -vtime_account_irq_exit (tsk); +vtime_account_irq (tsk);irqtime_account_irq (tsk); }
kernel/sched/cputime.c +9 -4
@@ -417,13 +417,18 @@ void vtime_task_switch(struct task_struct *prev) }# endif -void vtime_account_irq_enter (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk) { -if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && - !in_interrupt () &&is_idle_task (tsk)) +if (hardirq_count ()) { +vtime_account_hardirq (tsk); + }else if (in_serving_softirq ()) { +vtime_account_softirq (tsk); + }else if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && +is_idle_task (tsk)) {vtime_account_idle (tsk); -else + }else {vtime_account_kernel (tsk); + } } void cputime_adjust(struct task_cputime *curr, struct prev_cputime *prev,
[tip: irq/core] irqtime: Move irqtime entry accounting after irq offset incrementation
In reply to Frederic WeisbeckerThe following commit has been merged into the irq/core branch of tip:
The following commit has been merged into the irq/core branch of tip:
Commit-ID: d3759e7184f8f6187e62f8c4e7dcb1f6c47c075a Gitweb: https://git.kernel.org/tip/d3759e7184f8f6187e62f8c4e7dcb1f6c47c075a Author: Frederic Weisbecker <[email protected]> AuthorDate: Wed, 02 Dec 2020 12:57:31 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 02 Dec 2020 20:20:05 +01:00
irqtime: Move irqtime entry accounting after irq offset incrementation
IRQ time entry is currently accounted before HARDIRQ_OFFSET or SOFTIRQ_OFFSET are incremented. This is convenient to decide to which index the cputime to account is dispatched.
Unfortunately it prevents tick_irq_enter() from being called under HARDIRQ_OFFSET because tick_irq_enter() has to be called before the IRQ entry accounting due to the necessary clock catch up. As a result we don't benefit from appropriate lockdep coverage on tick_irq_enter().
To prepare for fixing this, move the IRQ entry cputime accounting after the preempt offset is incremented. This requires the cputime dispatch code to handle the extra offset.
--- include/linux/hardirq.h | 4 ++-- include/linux/vtime.h | 34 ++++++++++++++++++++++++---------- kernel/sched/cputime.c | 18 +++++++++++------- kernel/softirq.c | 6 +++--- 4 files changed, 40 insertions(+), 22 deletions(-)
include/linux/hardirq.h +2 -2
@@ -32,9 +32,9 @@ static __always_inline void rcu_irq_enter_check_tick(void)*/ #define __irq_enter () \ do { \ - account_irq_enter_time(current); \ preempt_count_add(HARDIRQ_OFFSET); \ lockdep_hardirq_enter(); \ + account_hardirq_enter(current); \ } while (0)/*
@@ -62,8 +62,8 @@ void irq_enter_rcu(void);*/ #define __irq_exit () \ do { \ + account_hardirq_exit(current); \ lockdep_hardirq_exit(); \ - account_irq_exit_time(current); \ preempt_count_sub(HARDIRQ_OFFSET); \ } while (0)
include/linux/vtime.h +24 -10
@@ -83,32 +83,46 @@ static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { }#endif #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE -extern void vtime_account_irq (struct task_struct *tsk); +extern void vtime_account_irq (struct task_struct *tsk,unsigned int offset);extern void vtime_account_softirq (struct task_struct *tsk);extern void vtime_account_hardirq (struct task_struct *tsk);extern void vtime_flush (struct task_struct *tsk);#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ -static inline void vtime_account_irq (struct task_struct *tsk) { } +static inline void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { } +static inline void vtime_account_softirq (struct task_struct *tsk) { } +static inline void vtime_account_hardirq (struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { }#endif #ifdef CONFIG_IRQ_TIME_ACCOUNTING -extern void irqtime_account_irq (struct task_struct *tsk); +extern void irqtime_account_irq (struct task_struct *tsk,unsigned int offset);#else -static inline void irqtime_account_irq (struct task_struct *tsk) { } +static inline void irqtime_account_irq (struct task_struct *tsk,unsigned int offset) { }#endif -static inline void account_irq_enter_time (struct task_struct *tsk) +static inline void account_softirq_enter (struct task_struct *tsk) { -vtime_account_irq (tsk); -irqtime_account_irq (tsk); +vtime_account_irq (tsk,SOFTIRQ_OFFSET ); +irqtime_account_irq (tsk,SOFTIRQ_OFFSET ); } -static inline void account_irq_exit_time (struct task_struct *tsk) +static inline void account_softirq_exit (struct task_struct *tsk) { -vtime_account_irq (tsk); -irqtime_account_irq (tsk); +vtime_account_softirq (tsk); +irqtime_account_irq (tsk,0 ); +} + +static inline void account_hardirq_enter (struct task_struct *tsk) +{ +vtime_account_irq (tsk,HARDIRQ_OFFSET ); +irqtime_account_irq (tsk,HARDIRQ_OFFSET ); +} + +static inline void account_hardirq_exit (struct task_struct *tsk) +{ +vtime_account_hardirq (tsk); +irqtime_account_irq (tsk,0 ); }#endif /* _LINUX_KERNEL_VTIME_H */
kernel/sched/cputime.c +11 -7
@@ -44,12 +44,13 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta, }/* -* Called before incrementing preempt_count on {soft,}irq_enter +* Called after incrementing preempt_count on {soft,}irq_enter * and before decrementing preempt_count on {soft,}irq_exit. */ -void irqtime_account_irq (struct task_struct *curr) +void irqtime_account_irq (struct task_struct *curr,unsigned int offset) {struct irqtime *irqtime =this_cpu_ptr (&cpu_irqtime); +unsigned int pc;s64 delta;int cpu;
@@ -59,6 +60,7 @@ void irqtime_account_irq(struct task_struct *curr) cpu =smp_processor_id (); delta =sched_clock_cpu (cpu) - irqtime->irq_start_time; irqtime->irq_start_time += delta; + pc =preempt_count () - offset;/* * We do not account for softirq time from ksoftirqd here.
@@ -66,9 +68,9 @@ void irqtime_account_irq(struct task_struct *curr)* in that case, so as not to confuse scheduler with a special task * that do not consume any time, but still wants to run. */ -if (hardirq_count ()) +if (pc &HARDIRQ_MASK )irqtime_account_delta (irqtime, delta,CPUTIME_IRQ ); -else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ()) +else if ((pc &SOFTIRQ_OFFSET ) && curr !=this_cpu_ksoftirqd ())irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); }
@@ -417,11 +419,13 @@ void vtime_task_switch(struct task_struct *prev) }# endif -void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {vtime_account_hardirq (tsk); - }else if (in_serving_softirq ()) { + }else if (pc &SOFTIRQ_OFFSET ) {vtime_account_softirq (tsk); }else if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) &&is_idle_task (tsk)) {
kernel/softirq.c +3 -3
@@ -315,10 +315,10 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) current->flags &= ~PF_MEMALLOC ; pending =local_softirq_pending (); -account_irq_enter_time (current);__local_bh_disable_ip (_RET_IP_,SOFTIRQ_OFFSET ); in_hardirq =lockdep_softirq_start (); +account_softirq_enter (current); restart:/* Reset the pending bitmask before enabling irqs */
@@ -365,8 +365,8 @@ restart:wakeup_softirqd (); } +account_softirq_exit (current);lockdep_softirq_end (in_hardirq); -account_irq_exit_time (current);__local_bh_enable (SOFTIRQ_OFFSET );WARN_ON_ONCE (in_interrupt ());current_restore_flags (old_flags,PF_MEMALLOC );
@@ -418,7 +418,7 @@ static inline void __irq_exit_rcu(void)#else lockdep_assert_irqs_disabled ();#endif -account_irq_exit_time (current); +account_hardirq_exit (current);preempt_count_sub (HARDIRQ_OFFSET );if (!in_interrupt () &&local_softirq_pending ())invoke_softirq ();
Re: [PATCH 1/5] sched/cputime: Remove symbol exports from IRQ time accounting
In reply to Frederic WeisbeckerQuoted 36 lines · Frederic Weisbecker
account_irq_enter_time()andaccount_irq_exit_time()are not called from modules.EXPORT_SYMBOL_GPL()can be safely removed from the IRQ cputime accounting functions called from there.Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Tony Luck <[email protected]>Cc Fenghua Yu <[email protected]>Cc Michael Ellerman <[email protected]>Cc Benjamin Herrenschmidt <[email protected]>Cc Paul Mackerras <[email protected]>Cc Heiko Carstens <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Christian Borntraeger <[email protected]>--- arch/s390/kernel/vtime.c | 10 +++++----- kernel/sched/cputime.c | 2 -- 2 files changed, 5 insertions(+), 7 deletions(-)arch/s390/kernel/vtime.c +1
-1@@ -226,7 +226,7 @@ void vtime_flush(struct task_struct *tsk)* Update process times based on virtual cpu times stored by entry.S * to the lowcore fields user_timer, system_timer & steal_clock. */ -void vtime_account_irq_enter (struct task_struct *tsk) +void vtime_account_kernel (struct task_struct *tsk) {u64 timer;@@ -245,12 +245,12 @@ void vtime_account_irq_enter(struct task_struct *tsk)virt_timer_forward (timer); }
arch/s390/kernel/vtime.c void vtime_account_irq_enter(struct task_struct *tsk) · line 253
Quoted 10 lines
arch/s390/kernel/vtime.c +4
-4-EXPORT_SYMBOL_GPL (vtime_account_irq_enter); - -void vtime_account_kernel (struct task_struct *tsk) -__attribute__((alias ("vtime_account_irq_enter" )));EXPORT_SYMBOL_GPL (vtime_account_kernel); +void vtime_account_irq_enter (struct task_struct *tsk) +__attribute__((alias ("vtime_account_kernel" ))); + +
One new line is enough I think. Apart from that this looks sane from an s390 perspective.
Re: [PATCH 2/5] s390/vtime: Use the generic IRQ entry accounting
In reply to Frederic WeisbeckerQuoted 11 lines · Frederic Weisbecker
s390 has its own version of IRQ entry accounting because it doesn't account the idle time the same way the other architectures do. Only the actual idle sleep time is accounted as idle time, the rest of the idle task execution is accounted as system time.
Make the generic IRQ entry accounting aware of architectures that have their own way of accounting idle time and convert s390 to use it.
This prepares s390 to get involved in further consolidations of IRQ time accounting.
Quoted 10 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Tony Luck <[email protected]>Cc Fenghua Yu <[email protected]>Cc Michael Ellerman <[email protected]>Cc Benjamin Herrenschmidt <[email protected]>Cc Paul Mackerras <[email protected]>Cc Heiko Carstens <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Christian Borntraeger <[email protected]>
As far as I can tel,l this patch should be a no-op for s390 function-wise.
Quoted 103 lines
--- arch/Kconfig | 7 ++++++- arch/s390/Kconfig | 1 + arch/s390/include/asm/vtime.h | 1 - arch/s390/kernel/vtime.c | 4 ---- kernel/sched/cputime.c | 13 ++----------- 5 files changed, 9 insertions(+), 17 deletions(-)arch/Kconfig +6
-1@@ -627,6 +627,12 @@ config HAVE_TIF_NOHZ config HAVE_VIRT_CPU_ACCOUNTING bool +config HAVE_VIRT_CPU_ACCOUNTING_IDLE + bool + help + Architecture has its own way to account idle CPU time and therefore + doesn't implement vtime_account_idle(). + config ARCH_HAS_SCALED_CPUTIME bool@@ -641,7 +647,6 @@ config HAVE_VIRT_CPU_ACCOUNTING_GEN some 32-bit arches may require multiple accesses, so proper locking is needed to protect against concurrent accesses. - config HAVE_IRQ_TIME_ACCOUNTING bool helparch/s390/Kconfig +1
-0@@ -181,6 +181,7 @@ config S390 select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING + select HAVE_VIRT_CPU_ACCOUNTING_IDLE select IOMMU_HELPER if PCI select IOMMU_SUPPORT if PCI select MODULES_USE_ELF_RELAarch/s390/include/asm/vtime.h +0
-1@@ -2,7 +2,6 @@#ifndef _S390_VTIME_H#define _S390_VTIME_H -#define __ARCH_HAS_VTIME_ACCOUNT#define __ARCH_HAS_VTIME_TASK_SWITCH#endif /* _S390_VTIME_H */ arch/s390/kernel/vtime.c +0
-4@@ -247,10 +247,6 @@ void vtime_account_kernel(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (vtime_account_kernel); -void vtime_account_irq_enter (struct task_struct *tsk) -__attribute__((alias ("vtime_account_kernel" ))); - -/* * Sorted add to a list. List is linear searched until first bigger * element is found. kernel/sched/cputime.c +2
-11@@ -417,23 +417,14 @@ void vtime_task_switch(struct task_struct *prev) }# endif -/* -* Archs that account the whole time spent in the idle task -* (outside irq) as idle time can rely on this and just implement -* vtime_account_kernel() and vtime_account_idle(). Archs that -* have other meaning of the idle time (s390 only includes the -* time spent by the CPU when it's in low power mode) must override -* vtime_account(). -*/ -#ifndef __ARCH_HAS_VTIME_ACCOUNTvoid vtime_account_irq_enter (struct task_struct *tsk) { -if (!in_interrupt () &&is_idle_task (tsk)) +if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) && + !in_interrupt () &&is_idle_task (tsk))vtime_account_idle (tsk);else vtime_account_kernel (tsk); } -#endif /* __ARCH_HAS_VTIME_ACCOUNT */ void cputime_adjust (struct task_cputime *curr,struct prev_cputime *prev,u64 *ut,u64 *st)
Re: [PATCH 4/5] irqtime: Move irqtime entry accounting after irq offset incrementation
In reply to Frederic WeisbeckerHi Frederic
Quoted 30 lines · Frederic Weisbecker
#endif /*
_LINUX_KERNEL_VTIME_H*/kernel/sched/cputime.c +4
-2@@ -44,12 +44,13 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta, }/* -* Called before incrementing preempt_count on {soft,}irq_enter +* Called after incrementing preempt_count on {soft,}irq_enter * and before decrementing preempt_count on {soft,}irq_exit. */ -void irqtime_account_irq (struct task_struct *curr) +void irqtime_account_irq (struct task_struct *curr,unsigned int offset) {struct irqtime *irqtime =this_cpu_ptr (&cpu_irqtime); +unsigned int pc;s64 delta;int cpu;@@ -59,6 +60,7 @@ void irqtime_account_irq(struct task_struct *curr) cpu =smp_processor_id (); delta =sched_clock_cpu (cpu) - irqtime->irq_start_time; irqtime->irq_start_time += delta; + pc =preempt_count () - offset;/* * We do not account for softirq time from ksoftirqd here.
kernel/sched/cputime.c void irqtime_account_irq(struct task_struct *curr) · line 73
Quoted 8 lines
kernel/sched/cputime.c +2
-2@@ -66,9 +68,9 @@ void irqtime_account_irq(struct task_struct *curr)* in that case, so as not to confuse scheduler with a special task * that do not consume any time, but still wants to run. */ -if (hardirq_count ()) +if (pc &HARDIRQ_MASK )irqtime_account_delta (irqtime, delta,CPUTIME_IRQ ); -else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ()) +else if ((pc &SOFTIRQ_OFFSET ) && curr !=this_cpu_ksoftirqd ())
Noob question. Why for SOFTIRQs we do sofirq_count() & *SOFTIRQ_OFFSET*? It seems we're in-softirq only if the count is odd numbered.
/me tries to dig more
Hmm could it be because the softirq count is actually 1 bit and the rest is for SOFTIRQ_DISABLE_OFFSET (BH disabled)?
IOW, 1 bit is for we're in softirq context, and the remaining 7 bits are to count BH disable nesting, right?
I guess this would make sense; we don't nest softirqs processing AFAIK. But I could be misreading the code too :-)
Quoted 4 lines
kernel/sched/cputime.c +0
-0irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); }
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 426
Quoted 10 lines
kernel/sched/cputime.c +4
-2@@ -417,11 +419,13 @@ void vtime_task_switch(struct task_struct *prev) }# endif -void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {
Shouldn't this be HARDIRQ_MASK like above?
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 431
Quoted 6 lines
kernel/sched/cputime.c +1
-1vtime_account_hardirq (tsk); - }else if (in_serving_softirq ()) { + }else if (pc &SOFTIRQ_OFFSET ) {vtime_account_softirq (tsk); }else if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) &&is_idle_task (tsk)) {
Thanks
--
Qais Yousef
Re: [PATCH 4/5] irqtime: Move irqtime entry accounting after irq offset incrementation
Quoted 16 lines · Qais Yousef
Hi Frederic
kernel/sched/cputime.c void irqtime_account_irq(struct task_struct *curr) · line 73
Quoted 9 lines · Frederic Weisbecker
kernel/sched/cputime.c +2
-2@@ -66,9 +68,9 @@ void irqtime_account_irq(struct task_struct *curr)* in that case, so as not to confuse scheduler with a special task * that do not consume any time, but still wants to run. */ -if (hardirq_count ()) +if (pc &HARDIRQ_MASK )irqtime_account_delta (irqtime, delta,CPUTIME_IRQ ); -else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ()) +else if ((pc &SOFTIRQ_OFFSET ) && curr !=this_cpu_ksoftirqd ())Noob question. Why for SOFTIRQs we do
sofirq_count()& *SOFTIRQ_OFFSET*? It seems we're in-softirq only if the count is odd numbered.
Quoted 4 lines
/me tries to dig more
Hmm could it be because the softirq count is actually 1 bit and the rest is for
SOFTIRQ_DISABLE_OFFSET(BH disabled)?
Exactly!
Quoted 6 lines
IOW, 1 bit is for we're in softirq context, and the remaining 7 bits are to count BH disable nesting, right?
I guess this would make sense; we don't nest softirqs processing AFAIK. But I could be misreading the code too :-)
You got it right!
This is commented in softirq.c somewhere:
/*
* preempt_count and SOFTIRQ_OFFSET usage:
* - preempt_count is changed by SOFTIRQ_OFFSET on entering or leaving
* softirq processing.
* - preempt_count is changed by SOFTIRQ_DISABLE_OFFSET (= 2 * SOFTIRQ_OFFSET)
* on local_bh_disable or local_bh_enable.
* This lets us distinguish between whether we are currently processing
* softirq and whether we just have bh disabled.
*/
But we should elaborate on the fact that, indeed, softirq processing can't nest, while softirq disablement can. I should try to send a patch and comment more thoroughly on the subtleties of preempt mask in preempt.h.
Quoted 8 lines
Quoted 8 lines
kernel/sched/cputime.c +0
-0irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); }@@ -417,11 +419,13 @@ void vtime_task_switch(struct task_struct *prev) }# endif
Quoted 9 lines
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 426
Quoted 7 lines
kernel/sched/cputime.c +4
-2-void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {Shouldn't this be
HARDIRQ_MASKlike above?
In the rare cases of nested hardirqs happening with broken drivers, Only the outer hardirq does matter. All the time spent in the inner hardirqs is included in the outer one.
Thanks.
Quoted 12 lines
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 431
Quoted 7 lines
kernel/sched/cputime.c +1
-1vtime_account_hardirq (tsk); - }else if (in_serving_softirq ()) { + }else if (pc &SOFTIRQ_OFFSET ) {vtime_account_softirq (tsk); }else if (!IS_ENABLED (CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE ) &&is_idle_task (tsk)) {Thanks
--
Qais Yousef
Re: [PATCH 4/5] irqtime: Move irqtime entry accounting after irq offset incrementation
Quoted 45 lines · Frederic Weisbecker
Quoted 16 lines · Qais Yousef
Hi Frederic
kernel/sched/cputime.c void irqtime_account_irq(struct task_struct *curr) · line 73
Quoted 9 lines · Frederic Weisbecker
kernel/sched/cputime.c +2
-2@@ -66,9 +68,9 @@ void irqtime_account_irq(struct task_struct *curr)* in that case, so as not to confuse scheduler with a special task * that do not consume any time, but still wants to run. */ -if (hardirq_count ()) +if (pc &HARDIRQ_MASK )irqtime_account_delta (irqtime, delta,CPUTIME_IRQ ); -else if (in_serving_softirq () && curr !=this_cpu_ksoftirqd ()) +else if ((pc &SOFTIRQ_OFFSET ) && curr !=this_cpu_ksoftirqd ())Noob question. Why for SOFTIRQs we do
sofirq_count()& *SOFTIRQ_OFFSET*? It seems we're in-softirq only if the count is odd numbered.Quoted 4 lines
/me tries to dig more
Hmm could it be because the softirq count is actually 1 bit and the rest is for
SOFTIRQ_DISABLE_OFFSET(BH disabled)?Exactly!
Quoted 6 lines
IOW, 1 bit is for we're in softirq context, and the remaining 7 bits are to count BH disable nesting, right?
I guess this would make sense; we don't nest softirqs processing AFAIK. But I could be misreading the code too :-)
You got it right!
This is commented in
softirq.csomewhere:/*
*preempt_countandSOFTIRQ_OFFSETusage:
* -preempt_countis changed bySOFTIRQ_OFFSETon entering or leaving
* softirq processing.
* -preempt_countis changed bySOFTIRQ_DISABLE_OFFSET(= 2 *SOFTIRQ_OFFSET)
* onlocal_bh_disableorlocal_bh_enable.
* This lets us distinguish between whether we are currently processing
* softirq and whether we just have bh disabled.
*/
Quoted 3 lines
But we should elaborate on the fact that, indeed, softirq processing can't nest, while softirq disablement can. I should try to send a patch and comment more thoroughly on the subtleties of preempt mask in
preempt.h.
Thanks for the info!
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
kernel/sched/cputime.c +0
-0irqtime_account_delta (irqtime, delta,CPUTIME_SOFTIRQ ); }@@ -417,11 +419,13 @@ void vtime_task_switch(struct task_struct *prev) }# endif Quoted 9 lines
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 426
Quoted 7 lines
kernel/sched/cputime.c +4
-2-void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {Shouldn't this be
HARDIRQ_MASKlike above?
Quoted 3 lines
In the rare cases of nested hardirqs happening with broken drivers, Only the outer hardirq does matter. All the time spent in the inner hardirqs is included in the outer one.
Ah I see. The original code was doing hardirq_count(), which apparently wasn't right either.
Shouldn't it be pc == HARDIRQ_OFFSET then? All odd nest counts will trigger this otherwise, and IIUC we want this to trigger once on first entry only.
Thanks
--
Qais Yousef
Re: [PATCH 4/5] irqtime: Move irqtime entry accounting after irq offset incrementation
Quoted 15 lines · Qais Yousef
Quoted 10 lines · Frederic Weisbecker
Quoted 9 lines
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 426
Quoted 7 lines
kernel/sched/cputime.c +4
-2-void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {Shouldn't this be
HARDIRQ_MASKlike above?Quoted 3 lines
In the rare cases of nested hardirqs happening with broken drivers, Only the outer hardirq does matter. All the time spent in the inner hardirqs is included in the outer one.
Quoted 5 lines
Ah I see. The original code was doing
hardirq_count(), which apparently wasn't right either.Shouldn't it be pc ==
HARDIRQ_OFFSETthen? All odd nest counts will trigger this otherwise, and IIUC we want this to trigger once on first entry only.
Right but we must also handle hardirqs interrupting either preempt disabled sections or softirq servicing/disabled section.
3 stacking hardirqs should be rare enough that we don't really care. In the worst case we are going to account the third IRQ seperately. Not a correctness issue, just a rare unoptimized case.
Quoted 5 lines
Thanks
--
Qais Yousef
Re: [PATCH 4/5] irqtime: Move irqtime entry accounting after irq offset incrementation
Quoted 22 lines · Frederic Weisbecker
Quoted 15 lines · Qais Yousef
Quoted 10 lines · Frederic Weisbecker
Quoted 9 lines
kernel/sched/cputime.c void vtime_task_switch(struct task_struct *prev) · line 426
Quoted 7 lines
kernel/sched/cputime.c +4
-2-void vtime_account_irq (struct task_struct *tsk) +void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { -if (hardirq_count ()) { +unsigned int pc =preempt_count () - offset; + +if (pc &HARDIRQ_OFFSET ) {Shouldn't this be
HARDIRQ_MASKlike above?Quoted 3 lines
In the rare cases of nested hardirqs happening with broken drivers, Only the outer hardirq does matter. All the time spent in the inner hardirqs is included in the outer one.
Quoted 5 lines
Ah I see. The original code was doing
hardirq_count(), which apparently wasn't right either.Shouldn't it be pc ==
HARDIRQ_OFFSETthen? All odd nest counts will trigger this otherwise, and IIUC we want this to trigger once on first entry only.
Quoted 6 lines
Right but we must also handle hardirqs interrupting either preempt disabled sections or softirq servicing/disabled section.
3 stacking hardirqs should be rare enough that we don't really care. In the worst case we are going to account the third IRQ seperately. Not a correctness issue, just a rare unoptimized case.
I admit I need to wrap my head around some more details to fully comprehend that, but that's my own confusion to clear out :-)
Thanks for your answer.
Cheers
--
Qais Yousef