remove put_cpu_no_resched()
Subsystems (2)
PATCH 1/1 remove put_cpu_no_resched()
Impact: remove a high latency source
put_cpu_no_resched() is an optimization of put_cpu() which unfortunately can cause high latencies.
The nfs iostats code uses put_cpu_no_resched() in a code sequence where a reschedule request caused by an interrupt between the get_cpu() and the put_cpu_no_resched() can delay the reschedule for at least HZ.
The other users of put_cpu_no_resched() optimize correctly in interrupt code, but there is no real harm in using the put_cpu() function which is an alias for preempt_enable(). The extra check of the preemmpt count is not as critical as the potential source of missing a reschedule.
Debugged in the preempt-rt tree and verified in mainline.
--- arch/ia64/kernel/perfmon.c | 2 +- arch/x86/kernel/tlb_32.c | 2 +- arch/x86/mach-voyager/voyager_smp.c | 2 +- fs/nfs/iostat.h | 4 ++-- include/linux/smp.h | 1 - 5 files changed, 5 insertions(+), 6 deletions(-)
arch/ia64/kernel/perfmon.c +1 -1
@@ -5595,7 +5595,7 @@ pfm_interrupt_handler(int irq, void *arg (*pfm_alt_intr_handler->handler)(irq, arg, regs); } -put_cpu_no_resched (); +put_cpu ();return IRQ_HANDLED ; }
arch/x86/kernel/tlb_32.c +1 -1
@@ -117,7 +117,7 @@ void smp_invalidate_interrupt(struct pt_cpu_clear (cpu, flush_cpumask);smp_mb__after_clear_bit (); out: -put_cpu_no_resched (); +put_cpu ();inc_irq_stat (irq_tlb_count); }
arch/x86/mach-voyager/voyager_smp.c +1 -1
@@ -940,7 +940,7 @@ static void smp_enable_irq_interrupt(voi vic_irq_enable_mask[cpu] =0 ;spin_unlock (&vic_irq_lock); -put_cpu_no_resched (); +put_cpu (); }/*
fs/nfs/iostat.h +2 -2
@@ -28,7 +28,7 @@ static inline void nfs_inc_server_stats( cpu =get_cpu (); iostats =per_cpu_ptr (server->io_stats, cpu); iostats->events[stat]++; -put_cpu_no_resched (); +put_cpu (); } static inline void nfs_inc_stats(const struct inode *inode,
@@ -47,7 +47,7 @@ static inline void nfs_add_server_stats( cpu =get_cpu (); iostats =per_cpu_ptr (server->io_stats, cpu); iostats->bytes[stat] += addend; -put_cpu_no_resched (); +put_cpu (); } static inline void nfs_add_stats(const struct inode *inode,
include/linux/smp.h +0 -1
@@ -174,7 +174,6 @@ static inline void init_call_single_data#define get_cpu () ({ preempt_disable(); smp_processor_id(); })#define put_cpu () preempt_enable() -#define put_cpu_no_resched () preempt_enable_no_resched()void smp_setup_processor_id (void );
Quoted 17 lines · Thomas Gleixner
plain text document attachment (remove-put-cpu-no-resched.patch)
Impact: remove a high latency source
put_cpu_no_resched()is an optimization ofput_cpu()which unfortunately can cause high latencies.The nfs iostats code uses
put_cpu_no_resched()in a code sequence where a reschedule request caused by an interrupt between theget_cpu()and theput_cpu_no_resched()can delay the reschedule for at least HZ.The other users of
put_cpu_no_resched()optimize correctly in interrupt code, but there is no real harm in using theput_cpu()function which is an alias forpreempt_enable(). The extra check of the preemmpt count is not as critical as the potential source of missing a reschedule.
Quoted 6 lines
Debugged in the preempt-rt tree and verified in mainline.
Signed-off-by Thomas Gleixner <[email protected]>Reviewed-by Ingo Molnar <[email protected]>Cc Tony Luck <[email protected]>Cc Trond Myklebust <[email protected]>
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com