Skip to contentSkip to conversation

preempt-timing-2.6.8-rc1

PATCH New William Lee Irwin III 42 participants 450 messages Jul 13, 2004 – Aug 28, 2004 2 messages not in the archive Page 2 of 2

Subsystems (12)
lore
Share & tools
lore
#296 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar
Quoted 1 line · Lee Revell
Quoted 1 line

Any comments on the unmap_vmas issue?

Quoted 2 lines · Ingo Molnar

wli indicated he's working on the pagetable zapping critical section issue - wli?

What I have is meant to remove clear_page_tables() by incrementally pruning at munmap() -time. As far as unmap_vmas() goes, it's not directly pertinent. It rather affects (and is meant to remove) clear_page_tables().

-- wli

#297 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar
Quoted 4 lines · On Thu, 19 Aug 2004 09:32:47 +0200 Ingo Molnar <[email protected]> wrote:

i've uploaded the -P4 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

Hi, this patch doesn't really apply cleanly. Do i need to worry about the "fuzz" and the offsets?

flo

patch -p1 --dry-run < /home/tapas/downloads/voluntary-preempt-2.6.8.1-P4 patching file arch/i386/kernel/entry.S
patching file arch/i386/kernel/i386_ksyms.c
patching file arch/i386/kernel/i8259.c
patching file arch/i386/kernel/io_apic.c
patching file arch/i386/kernel/irq.c
Hunk #12 succeeded at 656 with fuzz 2.
patching file arch/i386/kernel/apic.c
patching file arch/i386/kernel/smp.c
patching file arch/i386/kernel/traps.c
patching file arch/i386/mm/fault.c
patching file arch/i386/Kconfig
patching file arch/i386/lib/usercopy.c
patching file arch/i386/mach-visws/setup.c
patching file arch/i386/mach-visws/visws_apic.c
patching file arch/i386/mach-voyager/setup.c
patching file arch/i386/boot/compressed/misc.c
patching file arch/i386/mach-default/setup.c
patching file include/asm-ppc64/uaccess.h
patching file include/asm-m68k/uaccess.h
patching file include/asm-parisc/uaccess.h
patching file include/asm-mips/uaccess.h
patching file include/asm-ppc/uaccess.h
patching file include/asm-x86_64/uaccess.h
patching file include/asm-ia64/uaccess.h
patching file include/linux/irq.h
patching file include/linux/sched.h
patching file include/linux/blkdev.h
patching file include/linux/buffer_head.h
patching file include/linux/interrupt.h
patching file include/linux/kernel.h
patching file include/linux/kernel_stat.h
patching file include/linux/pagemap.h
patching file include/linux/preempt.h
patching file include/linux/sysctl.h
patching file include/linux/writeback.h
patching file include/asm-m68knommu/uaccess.h
patching file include/asm-um/uaccess.h
patching file include/asm-arm/uaccess.h
patching file include/asm-sparc/uaccess.h
patching file include/asm-alpha/uaccess.h
patching file include/asm-h8300/uaccess.h
patching file include/asm-i386/irq.h
patching file include/asm-i386/checksum.h
patching file include/asm-i386/hardirq.h
patching file include/asm-i386/hw_irq.h
patching file include/asm-i386/io_apic.h
patching file include/asm-i386/signal.h
patching file include/asm-i386/uaccess.h
patching file include/asm-i386/system.h
patching file include/asm-s390/uaccess.h
patching file include/asm-cris/uaccess.h
patching file include/asm-sparc64/uaccess.h
patching file include/asm-arm26/uaccess.h
patching file include/asm-v850/uaccess.h
patching file include/asm-sh/uaccess.h
patching file include/asm-sh64/uaccess.h
patching file drivers/char/tty_io.c
patching file drivers/char/random.c
Hunk #1 succeeded at 947 (offset 5 lines).
patching file drivers/block/ll_rw_blk.c
Hunk #3 succeeded at 2400 (offset 1 line).
Hunk #4 succeeded at 2448 (offset 1 line).
Hunk #5 succeeded at 3065 (offset 3 lines).
Hunk #6 succeeded at 3132 (offset 3 lines).
patching file drivers/input/keyboard/atkbd.c
patching file drivers/input/mouse/psmouse-base.c
patching file drivers/ide/ide-io.c
patching file drivers/md/dm-table.c
patching file drivers/md/linear.c
patching file drivers/md/multipath.c
patching file drivers/md/raid0.c
patching file drivers/md/raid1.c
patching file fs/ext3/ialloc.c
patching file fs/ext3/inode.c
patching file fs/ext3/namei.c
patching file fs/ext3/super.c
patching file fs/proc/proc_misc.c
patching file fs/proc/task_mmu.c
patching file fs/jbd/checkpoint.c
patching file fs/jbd/commit.c
patching file fs/jbd/revoke.c
patching file fs/buffer.c
patching file fs/dcache.c
patching file fs/exec.c
patching file fs/file_table.c
patching file fs/fs-writeback.c
patching file fs/inode.c
patching file fs/select.c
patching file net/ipv4/route.c
patching file net/ipv4/tcp_minisocks.c
patching file net/core/dev.c
patching file net/core/sock.c
patching file kernel/Makefile
patching file kernel/exit.c
patching file kernel/hardirq.c
patching file kernel/kthread.c
patching file kernel/printk.c
patching file kernel/profile.c
patching file kernel/rcupdate.c
patching file kernel/sched.c
patching file kernel/softirq.c
patching file kernel/sysctl.c
patching file kernel/timer.c
patching file kernel/latency.c
patching file kernel/kallsyms.c
patching file mm/filemap.c
patching file mm/memory.c
patching file mm/mempool.c
patching file mm/mmap.c
patching file mm/slab.c
patching file mm/vmscan.c
patching file mm/msync.c
patching file init/Kconfig
patching file init/main.c
patching file Makefile

#298 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 11 lines · Florian Schmidt
Quoted 4 lines · On Thu, 19 Aug 2004 09:32:47 +0200 Ingo Molnar <[email protected]> wrote:

i've uploaded the -P4 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

Hi, this patch doesn't really apply cleanly. Do i need to worry about the "fuzz" and the offsets?

Does not seem to hurt anything. Ingo probably has some differences in his tree, the 'fuzz' just means that patch was smart enough to figure out that they don't matter.

Lee

#299 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 8 lines · Lee Revell
Quoted 4 lines
Quoted 1 line

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

Hi, this patch doesn't really apply cleanly. Do i need to worry about the "fuzz" and the offsets?

Does not seem to hurt anything. Ingo probably has some differences in his tree, the 'fuzz' just means that patch was smart enough to figure out that they don't matter.

yeah, it shouldnt be an issue, until there's no reject. (I'll rebuild my tree because i thought i have a vanilla tree, but apparently not.)

Ingo

#300 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar
Quoted 9 lines · --- Ingo Molnar

i've uploaded the -P4 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

the main change is a more robust latency tracer - the previous one was not 100% correct for interrupts. People who have exprienced those weird ~1msec latencies in the idle task please re-check and re-post latency traces.

I receive the following errors on compile, after make has gotten through building all config options:

-- SNIP --

  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1

arch/i386/kernel/built-in.o(.text+0x398e): In function `do_nmi':
: undefined reference to __trace' arch/i386/kernel/built-in.o(.text+0x4101): In function do_IRQ':
: undefined reference to __trace' arch/i386/kernel/built-in.o(.text+0xe1e9): In function smp_apic_timer_interrupt':
: undefined reference to __trace' arch/i386/mm/built-in.o(.text+0x71b): In function do_page_fault':
: undefined reference to `__trace'
make: *** [.tmp_vmlinux1] Error 1

-P2 and -P3 gave me the same errors except for `smp_apic_timer_interrupt'.
I have tried my .config, default, allyes, and allno, and I get the same result regardless of config. Tree is fresh, patch applied with -Np1 <../[patchname]. I also get fuzz and offsets, but they don't seem to bear on the problem directly. I'm currently running -P1, which doesn't have this problem. gcc is 3.3.4.

With thanks for a smoother system feel,
Matthew Frost
artusemrys at sbcglobal dot net

#301 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar

When logging into my Fedora Core 2 box on the console (normal vga text console), I get the following latency trace:

(preempt=1 voluntary-preempt=3 elevator=cfq)

preemption latency trace v1.0.1
-------------------------------
latency: 3568 us, entries: 137 (137)

    -----------------
    | task: setfont/2209, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: voluntary_resched+0x31/0x60
=> ended at: voluntary_resched+0x31/0x60
=======>
0.000ms (+0.000ms): touch_preempt_timing (voluntary_resched) 0.000ms (+0.000ms): vgacon_font_set (con_font_set) 0.000ms (+0.000ms): vgacon_do_font_op (vgacon_font_set) 0.057ms (+0.056ms): smp_apic_timer_interrupt (vgacon_do_font_op) 0.057ms (+0.000ms): profile_hook (smp_apic_timer_interrupt) 0.057ms (+0.000ms): notifier_call_chain (profile_hook) 0.120ms (+0.063ms): do_IRQ (vgacon_do_font_op)
0.121ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 0.121ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 0.121ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.121ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 0.121ms (+0.000ms): mark_offset_pmtmr (timer_interrupt) 0.128ms (+0.006ms): do_timer (timer_interrupt)
0.128ms (+0.000ms): update_process_times (do_timer) 0.128ms (+0.000ms): update_one_process (update_process_times) 0.128ms (+0.000ms): run_local_timers (update_process_times) 0.128ms (+0.000ms): raise_softirq (update_process_times) 0.128ms (+0.000ms): scheduler_tick (update_process_times) 0.128ms (+0.000ms): sched_clock (scheduler_tick) 0.128ms (+0.000ms): rcu_check_callbacks (scheduler_tick) 0.128ms (+0.000ms): idle_cpu (rcu_check_callbacks) 0.129ms (+0.000ms): __tasklet_schedule (scheduler_tick) 0.129ms (+0.000ms): update_wall_time (do_timer) 0.129ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 0.129ms (+0.000ms): generic_note_interrupt (do_IRQ) 0.129ms (+0.000ms): end_edge_ioapic_irq (do_IRQ) 0.130ms (+0.000ms): do_softirq (do_IRQ)
0.130ms (+0.000ms): __do_softirq (do_softirq)
0.130ms (+0.000ms): wake_up_process (do_softirq) 0.130ms (+0.000ms): try_to_wake_up (wake_up_process) 0.130ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.130ms (+0.000ms): activate_task (try_to_wake_up) 0.130ms (+0.000ms): sched_clock (activate_task) 0.130ms (+0.000ms): recalc_task_prio (activate_task) 0.130ms (+0.000ms): effective_prio (recalc_task_prio) 0.130ms (+0.000ms): enqueue_task (activate_task) 1.058ms (+0.927ms): smp_apic_timer_interrupt (vgacon_do_font_op) 1.058ms (+0.000ms): profile_hook (smp_apic_timer_interrupt) 1.058ms (+0.000ms): notifier_call_chain (profile_hook) 1.058ms (+0.000ms): preempt_schedule (smp_apic_timer_interrupt) 1.058ms (+0.000ms): do_softirq (smp_apic_timer_interrupt) 1.058ms (+0.000ms): __do_softirq (do_softirq)
1.122ms (+0.063ms): do_IRQ (vgacon_do_font_op)
1.122ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 1.122ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 1.122ms (+0.000ms): preempt_schedule (do_IRQ)
1.122ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 1.122ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 1.123ms (+0.000ms): mark_offset_pmtmr (timer_interrupt) 1.129ms (+0.006ms): do_timer (timer_interrupt)
1.129ms (+0.000ms): update_process_times (do_timer) 1.129ms (+0.000ms): update_one_process (update_process_times) 1.129ms (+0.000ms): run_local_timers (update_process_times) 1.129ms (+0.000ms): raise_softirq (update_process_times) 1.129ms (+0.000ms): scheduler_tick (update_process_times) 1.129ms (+0.000ms): sched_clock (scheduler_tick) 1.129ms (+0.000ms): rcu_check_callbacks (scheduler_tick) 1.129ms (+0.000ms): idle_cpu (rcu_check_callbacks) 1.130ms (+0.000ms): update_wall_time (do_timer) 1.130ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 1.130ms (+0.000ms): generic_note_interrupt (do_IRQ) 1.130ms (+0.000ms): end_edge_ioapic_irq (do_IRQ) 1.130ms (+0.000ms): preempt_schedule (do_IRQ)
1.130ms (+0.000ms): do_softirq (do_IRQ)
1.130ms (+0.000ms): __do_softirq (do_softirq)
2.059ms (+0.928ms): smp_apic_timer_interrupt (vgacon_do_font_op) 2.059ms (+0.000ms): profile_hook (smp_apic_timer_interrupt) 2.059ms (+0.000ms): notifier_call_chain (profile_hook) 2.059ms (+0.000ms): preempt_schedule (smp_apic_timer_interrupt) 2.059ms (+0.000ms): do_softirq (smp_apic_timer_interrupt) 2.059ms (+0.000ms): __do_softirq (do_softirq)
2.123ms (+0.063ms): do_IRQ (vgacon_do_font_op)
2.123ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 2.123ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 2.123ms (+0.000ms): preempt_schedule (do_IRQ)
2.123ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 2.123ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 2.123ms (+0.000ms): mark_offset_pmtmr (timer_interrupt) 2.130ms (+0.006ms): do_timer (timer_interrupt)
2.130ms (+0.000ms): update_process_times (do_timer) 2.130ms (+0.000ms): update_one_process (update_process_times) 2.130ms (+0.000ms): run_local_timers (update_process_times) 2.130ms (+0.000ms): raise_softirq (update_process_times) 2.130ms (+0.000ms): scheduler_tick (update_process_times) 2.130ms (+0.000ms): sched_clock (scheduler_tick) 2.130ms (+0.000ms): rcu_check_callbacks (scheduler_tick) 2.131ms (+0.000ms): idle_cpu (rcu_check_callbacks) 2.131ms (+0.000ms): dequeue_task (scheduler_tick) 2.131ms (+0.000ms): effective_prio (scheduler_tick) 2.131ms (+0.000ms): task_timeslice (scheduler_tick) 2.131ms (+0.000ms): enqueue_task (scheduler_tick) 2.131ms (+0.000ms): update_wall_time (do_timer) 2.131ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 2.131ms (+0.000ms): generic_note_interrupt (do_IRQ) 2.132ms (+0.000ms): end_edge_ioapic_irq (do_IRQ) 2.132ms (+0.000ms): preempt_schedule (do_IRQ)
2.132ms (+0.000ms): do_softirq (do_IRQ)
2.132ms (+0.000ms): __do_softirq (do_softirq)
3.060ms (+0.927ms): smp_apic_timer_interrupt (vgacon_do_font_op) 3.060ms (+0.000ms): profile_hook (smp_apic_timer_interrupt) 3.060ms (+0.000ms): notifier_call_chain (profile_hook) 3.060ms (+0.000ms): preempt_schedule (smp_apic_timer_interrupt) 3.060ms (+0.000ms): do_softirq (smp_apic_timer_interrupt) 3.060ms (+0.000ms): __do_softirq (do_softirq)
3.124ms (+0.063ms): do_IRQ (vgacon_do_font_op)
3.124ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 3.124ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 3.124ms (+0.000ms): preempt_schedule (do_IRQ)
3.124ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 3.124ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 3.124ms (+0.000ms): mark_offset_pmtmr (timer_interrupt) 3.131ms (+0.006ms): do_timer (timer_interrupt)
3.131ms (+0.000ms): update_process_times (do_timer) 3.131ms (+0.000ms): update_one_process (update_process_times) 3.131ms (+0.000ms): run_local_timers (update_process_times) 3.131ms (+0.000ms): raise_softirq (update_process_times) 3.131ms (+0.000ms): scheduler_tick (update_process_times) 3.131ms (+0.000ms): sched_clock (scheduler_tick) 3.131ms (+0.000ms): rcu_check_callbacks (scheduler_tick) 3.131ms (+0.000ms): idle_cpu (rcu_check_callbacks) 3.131ms (+0.000ms): update_wall_time (do_timer) 3.131ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 3.132ms (+0.000ms): generic_note_interrupt (do_IRQ) 3.132ms (+0.000ms): end_edge_ioapic_irq (do_IRQ) 3.132ms (+0.000ms): preempt_schedule (do_IRQ)
3.132ms (+0.000ms): do_softirq (do_IRQ)
3.132ms (+0.000ms): __do_softirq (do_softirq)
3.485ms (+0.353ms): preempt_schedule (vgacon_do_font_op) 3.486ms (+0.000ms): vgacon_adjust_height (vgacon_font_set) 3.486ms (+0.000ms): release_console_sem (con_font_set) 3.486ms (+0.000ms): preempt_schedule (release_console_sem) 3.486ms (+0.000ms): kfree (con_font_set)
3.487ms (+0.000ms): copy_to_user (vt_ioctl)
3.487ms (+0.000ms): __might_sleep (copy_to_user) 3.487ms (+0.000ms): voluntary_resched (copy_to_user) 3.487ms (+0.000ms): __might_sleep (voluntary_resched) 3.487ms (+0.000ms): touch_preempt_timing (voluntary_resched)

$ cat /proc/interrupts
           CPU0
  0:    1286100    IO-APIC-edge  timer
  1:       4572    IO-APIC-edge  i8042
  8:          1    IO-APIC-edge  rtc
  9:         10   IO-APIC-level  acpi
 12:      86351    IO-APIC-edge  i8042
 14:      14089    IO-APIC-edge  ide0
 15:        477    IO-APIC-edge  ide1
 19:      13568   IO-APIC-level  eth0
 20:         94   IO-APIC-level  ehci_hcd, ohci_hcd
 21:          0   IO-APIC-level  ohci_hcd
 22:        593   IO-APIC-level  NVidia nForce3
NMI:          0
LOC:    1286057
ERR:          0
MIS:          0

Machine is an ACER Aspire 1510 notebook
(AMD64 processor running a 32-bit kernel)

#302 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

The following latency trace is generated each time the sound driver is opened by an application on my box.

# lspci -s 00:06.0
00:06.0 Multimedia audio controller: nVidia Corporation nForce3 Audio (rev a2)

Audio driver:
# lsmod|grep snd

snd_seq                51856  0
snd_intel8x0           30600  0
snd_ac97_codec         66820  1 snd_intel8x0
snd_pcm                88264  1 snd_intel8x0
snd_timer              22596  2 snd_seq,snd_pcm
snd_page_alloc          9288  2 snd_intel8x0,snd_pcm
snd_mpu401_uart         6656  1 snd_intel8x0
snd_rawmidi            21156  1 snd_mpu401_uart
snd_seq_device          7176  2 snd_seq,snd_rawmidi
snd                    48804  8 snd_seq,snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer,snd_mpu401_uart,snd_rawmidi,snd_seq_device
soundcore               7840  1 snd

preemption latency trace v1.0.1
-------------------------------
latency: 50752 us, entries: 267 (267)

    -----------------
    | task: artsd/2502, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: voluntary_resched+0x31/0x60
=> ended at: sys_ioctl+0xf4/0x2b0
=======>
0.000ms (+0.000ms): touch_preempt_timing (voluntary_resched) 0.000ms (+0.000ms): snd_power_wait (snd_pcm_prepare)
0.000ms (+0.000ms): snd_pcm_action_lock_irq (snd_pcm_prepare) 0.001ms (+0.000ms): snd_pcm_action_single (snd_pcm_action_lock_irq) 0.001ms (+0.000ms): snd_pcm_pre_prepare (snd_pcm_action_single) 0.001ms (+0.000ms): snd_pcm_do_prepare (snd_pcm_action_single) 0.001ms (+0.000ms): snd_intel8x0_pcm_prepare (snd_pcm_do_prepare) 0.002ms (+0.000ms): snd_intel8x0_setup_pcm_out (snd_intel8x0_pcm_prepare) 0.002ms (+0.000ms): igetdword (snd_intel8x0_setup_pcm_out) 0.002ms (+0.000ms): iputdword (snd_intel8x0_setup_pcm_out) 0.003ms (+0.000ms): __const_udelay (snd_intel8x0_setup_pcm_out) 0.003ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 0.003ms (+0.000ms): delay_pmtmr (__delay)
0.995ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 0.995ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 0.995ms (+0.000ms): delay_pmtmr (__delay)
1.986ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 1.986ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 1.986ms (+0.000ms): delay_pmtmr (__delay)
2.978ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 2.978ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 2.978ms (+0.000ms): delay_pmtmr (__delay)
3.969ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 3.969ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 3.970ms (+0.000ms): delay_pmtmr (__delay)
4.961ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 4.961ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 4.961ms (+0.000ms): delay_pmtmr (__delay)
5.953ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 5.953ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 5.953ms (+0.000ms): delay_pmtmr (__delay)
6.944ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 6.944ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 6.944ms (+0.000ms): delay_pmtmr (__delay)
7.936ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 7.936ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 7.936ms (+0.000ms): delay_pmtmr (__delay)
8.927ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 8.927ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 8.927ms (+0.000ms): delay_pmtmr (__delay)
9.919ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 9.919ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 9.919ms (+0.000ms): delay_pmtmr (__delay)
10.910ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 10.910ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 10.911ms (+0.000ms): delay_pmtmr (__delay)
11.902ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 11.902ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 11.902ms (+0.000ms): delay_pmtmr (__delay)
12.893ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 12.894ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 12.894ms (+0.000ms): delay_pmtmr (__delay)
13.885ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 13.885ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 13.885ms (+0.000ms): delay_pmtmr (__delay)
14.877ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 14.877ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 14.877ms (+0.000ms): delay_pmtmr (__delay)
15.868ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 15.868ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 15.868ms (+0.000ms): delay_pmtmr (__delay)
16.860ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 16.860ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 16.860ms (+0.000ms): delay_pmtmr (__delay)
17.851ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 17.851ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 17.851ms (+0.000ms): delay_pmtmr (__delay)
18.843ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 18.843ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 18.843ms (+0.000ms): delay_pmtmr (__delay)
19.834ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 19.835ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 19.835ms (+0.000ms): delay_pmtmr (__delay)
20.826ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 20.826ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 20.826ms (+0.000ms): delay_pmtmr (__delay)
21.818ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 21.818ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 21.818ms (+0.000ms): delay_pmtmr (__delay)
22.809ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 22.809ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 22.809ms (+0.000ms): delay_pmtmr (__delay)
23.801ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 23.801ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 23.801ms (+0.000ms): delay_pmtmr (__delay)
24.792ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 24.792ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 24.792ms (+0.000ms): delay_pmtmr (__delay)
25.784ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 25.784ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 25.784ms (+0.000ms): delay_pmtmr (__delay)
26.775ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 26.775ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 26.776ms (+0.000ms): delay_pmtmr (__delay)
27.767ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 27.767ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 27.767ms (+0.000ms): delay_pmtmr (__delay)
28.759ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 28.759ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 28.759ms (+0.000ms): delay_pmtmr (__delay)
29.750ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 29.750ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 29.750ms (+0.000ms): delay_pmtmr (__delay)
30.742ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 30.742ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 30.742ms (+0.000ms): delay_pmtmr (__delay)
31.733ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 31.733ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 31.733ms (+0.000ms): delay_pmtmr (__delay)
32.725ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 32.725ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 32.725ms (+0.000ms): delay_pmtmr (__delay)
33.716ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 33.716ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 33.717ms (+0.000ms): delay_pmtmr (__delay)
34.708ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 34.708ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 34.708ms (+0.000ms): delay_pmtmr (__delay)
35.700ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 35.700ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 35.700ms (+0.000ms): delay_pmtmr (__delay)
36.691ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 36.691ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 36.691ms (+0.000ms): delay_pmtmr (__delay)
37.683ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 37.683ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 37.683ms (+0.000ms): delay_pmtmr (__delay)
38.674ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 38.674ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 38.674ms (+0.000ms): delay_pmtmr (__delay)
39.666ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 39.666ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 39.666ms (+0.000ms): delay_pmtmr (__delay)
40.657ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 40.657ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 40.657ms (+0.000ms): delay_pmtmr (__delay)
41.649ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 41.649ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 41.649ms (+0.000ms): delay_pmtmr (__delay)
42.640ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 42.641ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 42.641ms (+0.000ms): delay_pmtmr (__delay)
43.632ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 43.632ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 43.632ms (+0.000ms): delay_pmtmr (__delay)
44.624ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 44.624ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 44.624ms (+0.000ms): delay_pmtmr (__delay)
45.615ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 45.615ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 45.615ms (+0.000ms): delay_pmtmr (__delay)
46.607ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 46.607ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 46.607ms (+0.000ms): delay_pmtmr (__delay)
47.598ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 47.598ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 47.598ms (+0.000ms): delay_pmtmr (__delay)
48.590ms (+0.991ms): __const_udelay (snd_intel8x0_setup_pcm_out) 48.590ms (+0.000ms): __delay (snd_intel8x0_setup_pcm_out) 48.590ms (+0.000ms): delay_pmtmr (__delay)
49.581ms (+0.991ms): iputdword (snd_intel8x0_setup_pcm_out) 49.582ms (+0.000ms): snd_intel8x0_setup_periods (snd_intel8x0_pcm_prepare) 49.582ms (+0.000ms): iputdword (snd_intel8x0_setup_periods) 49.585ms (+0.002ms): iputbyte (snd_intel8x0_setup_periods) 49.585ms (+0.000ms): iputbyte (snd_intel8x0_setup_periods) 49.586ms (+0.000ms): iputbyte (snd_intel8x0_setup_periods) 49.586ms (+0.000ms): snd_pcm_do_reset (snd_pcm_action_single) 49.586ms (+0.000ms): snd_pcm_lib_ioctl (snd_pcm_do_reset) 49.586ms (+0.000ms): snd_pcm_lib_ioctl_reset (snd_pcm_do_reset) 49.587ms (+0.000ms): snd_pcm_post_prepare (snd_pcm_action_single) 49.587ms (+0.000ms): smp_apic_timer_interrupt (snd_pcm_action_lock_irq) 49.587ms (+0.000ms): profile_hook (smp_apic_timer_interrupt) 49.588ms (+0.000ms): notifier_call_chain (profile_hook) 49.588ms (+0.000ms): do_IRQ (snd_pcm_action_lock_irq)
49.589ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ)
49.589ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 49.589ms (+0.000ms): wake_up_process (generic_redirect_hardirq) 49.589ms (+0.000ms): try_to_wake_up (wake_up_process)
49.589ms (+0.000ms): task_rq_lock (try_to_wake_up)
49.589ms (+0.000ms): activate_task (try_to_wake_up)
49.589ms (+0.000ms): sched_clock (activate_task)
49.590ms (+0.000ms): recalc_task_prio (activate_task)
49.590ms (+0.000ms): effective_prio (recalc_task_prio) 49.590ms (+0.000ms): enqueue_task (activate_task)
49.590ms (+0.000ms): preempt_schedule (try_to_wake_up) 49.590ms (+0.000ms): preempt_schedule (do_IRQ)
49.591ms (+0.000ms): do_IRQ (snd_pcm_action_lock_irq)
49.591ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ)
49.591ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 49.591ms (+0.000ms): preempt_schedule (do_IRQ)
49.591ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 49.591ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 49.592ms (+0.000ms): mark_offset_pmtmr (timer_interrupt) 49.598ms (+0.006ms): do_timer (timer_interrupt)
49.598ms (+0.000ms): update_process_times (do_timer)
49.598ms (+0.000ms): update_one_process (update_process_times) 49.598ms (+0.000ms): run_local_timers (update_process_times) 49.598ms (+0.000ms): raise_softirq (update_process_times) 49.598ms (+0.000ms): scheduler_tick (update_process_times) 49.598ms (+0.000ms): sched_clock (scheduler_tick)
49.599ms (+0.000ms): task_timeslice (scheduler_tick)
49.599ms (+0.000ms): update_wall_time (do_timer)
49.599ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.599ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.599ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.599ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.599ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.599ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.600ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.601ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.602ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 49.603ms (+0.000ms): generic_note_interrupt (do_IRQ)
49.604ms (+0.000ms): end_edge_ioapic_irq (do_IRQ)
49.604ms (+0.000ms): preempt_schedule (do_IRQ)
49.604ms (+0.000ms): do_softirq (do_IRQ)
49.604ms (+0.000ms): __do_softirq (do_softirq)
49.604ms (+0.000ms): wake_up_process (do_softirq)
49.604ms (+0.000ms): try_to_wake_up (wake_up_process)
49.604ms (+0.000ms): task_rq_lock (try_to_wake_up)
49.604ms (+0.000ms): activate_task (try_to_wake_up)
49.604ms (+0.000ms): sched_clock (activate_task)
49.604ms (+0.000ms): recalc_task_prio (activate_task)
49.605ms (+0.000ms): effective_prio (recalc_task_prio) 49.605ms (+0.000ms): enqueue_task (activate_task)
49.605ms (+0.000ms): preempt_schedule (snd_pcm_action_lock_irq) 49.605ms (+0.000ms): sub_preempt_count (sys_ioctl)

sound/pci/intel8x0.c contains:

                if (chip->device_type == DEVICE_NFORCE) {
                        /* reset to 2ch once to keep the 6 channel data in alignment,
                         * to start from Front Left always
                         */
                        iputdword(chip, ICHREG(GLOB_CNT), (cnt & 0xcfffff));
                        mdelay(50); /* grrr... */
                } else if (chip->device_type == DEVICE_INTEL_ICH4) {

So I guess it's that mdelay(50) which is generating these (each time the device is opened).

#303 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 3 lines · On Thu, 2004-08-19 at 16:37, [email protected] wrote:

The following latency trace is generated each time the sound driver is opened by an application on my box.

This is a pretty big trace. Please try to trim these, especially if a few lines repeat hundreds of times (common).

The comment seems to imply that the author didn't like the mdelay but it didn't work otherwise. What happens if you get rid of the mdelay?

Lee

#304 lore raw

PATCH 1/1 intel8x0 latency fix

In reply to [email protected]
Quoted 4 lines · [email protected] writes:
# lspci -s 00:06.0
00:06.0 Multimedia audio controller: nVidia Corporation nForce3 Audio (rev a2)

latency: 50752 us, entries: 267 (267)

Following patch fixes it for 2 channel devices (like my notebook). Although I'm not sure if this is all that useful... if there are workloads other than audio work that need low latency, then this might be useful.

Without this patch, my machine was generating the latency each time artsd (KDE) re-opened the audio device.

sound/pci/intel8x0.c +4 -2
@@ -1021,8 +1021,10 @@
 			/* reset to 2ch once to keep the 6 channel data in alignment,
 			 * to start from Front Left always
 			 */
-			iputdword(chip, ICHREG(GLOB_CNT), (cnt & 0xcfffff));
-			mdelay(50); /* grrr... */
+			if (chip->multi4 || chip->multi6) {
+				iputdword(chip, ICHREG(GLOB_CNT), (cnt & 0xcfffff));
+				mdelay(50); /* grrr... */
+			}
 		} else if (chip->device_type == DEVICE_INTEL_ICH4) {
 			if (sample_bits > 16)
 				cnt |= ICH_PCM_20BIT;
#305 lore raw

Re: [patch] intel8x0 latency fix

At Fri, 20 Aug 2004 18:19:12 +0200,

Quoted 6 lines · [email protected] wrote:
Quoted 3 lines · [email protected] writes:
# lspci -s 00:06.0
00:06.0 Multimedia audio controller: nVidia Corporation nForce3 Audio (rev a2)
Quoted 7 lines
Quoted 1 line

latency: 50752 us, entries: 267 (267)

Following patch fixes it for 2 channel devices (like my notebook). Although I'm not sure if this is all that useful... if there are workloads other than audio work that need low latency, then this might be useful.

In the recent version (linux-sound tree), we can use msleep() (that is more latency friendely) in that callback. The following patch was applied to the current version.

Takashi

sound/pci/intel8x0.c +4 -2
@@ -1024,8 +1024,10 @@
 			/* reset to 2ch once to keep the 6 channel data in alignment,
 			 * to start from Front Left always
 			 */
-			iputdword(chip, ICHREG(GLOB_CNT), (cnt & 0xcfffff));
-			mdelay(50); /* grrr... */
+			if (cnt & ICH_PCM_246_MASK) {
+				iputdword(chip, ICHREG(GLOB_CNT), cnt & ~ICH_PCM_246_MASK);
+				msleep(50);
+			}
 		} else if (chip->device_type == DEVICE_INTEL_ICH4) {
 			if (sample_bits > 16)
 				cnt |= ICH_PCM_20BIT;
#306 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

i've uploaded the -P4 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

I think I am seeing those weird ~1 ms latencies still (actually I am not sure I saw before now):

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/kswapd_latency_trace.txt

This was the result of doing `make -j12' with a large C++ program (I know, I wanted to see what would happen). This gradually slowed the machine to a crawl - for a while I could watch the /proc/latency_trace fiels and see the latency gradually climbing. Eventually tha machine ground almost to a halt, with lots of disk activity., and trying to switch from X to a console window resulted in a blank screen for 2-3 minutes. I did eventually get a text console and was able to kill all the cc1plus processes, and the machine went back to normal.

The link above is the highest latency recorded during this episode.

Lee

#307 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 6 lines · Lee Revell
Quoted 1 line

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

I think I am seeing those weird ~1 ms latencies still (actually I am not sure I saw before now):

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/kswapd_latency_trace.txt

this is a 9 msec latency in fact. I know what's going on - it's the get_swap_page() locking - i looked at it once already but there's no immediate silver arrow. As we allocate more and more swap entries the longer the scan gets.

Ingo

#308 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Lee Revell
Quoted 1 line

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P4

I think I am seeing those weird ~1 ms latencies still (actually I am not sure I saw before now):

Quoted 7 lines
Quoted 1 line

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/kswapd_latency_trace.txt

this is a 9 msec latency in fact. I know what's going on - it's the get_swap_page() locking - i looked at it once already but there's no immediate silver arrow. As we allocate more and more swap entries the longer the scan gets.

This is an extreme load situation, so I don't think it will be a real-world problem. I have not seen it under any normal workload.

What about this one:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/netif_receive_skb_latency_trace.txt

This appears during normal use.

Lee

#309 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 2 lines · Lee Revell

This is an extreme load situation, so I don't think it will be a real-world problem. I have not seen it under any normal workload.

well, 9 msecs is still not nice. I've been able to trigger larger than 10msec latencies too on a 2 GHz box.

Quoted 5 lines

What about this one:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/netif_receive_skb_latency_trace.txt

This appears during normal use.

hm, tcp_collapse() in net/ipv4/tcp_input.c. Could you try to just return from that function? Collapsing skbs of a given socket is not a necessary functionality (it is only a 'nice' thing to have in OOM situations) and it indeed can introduce quite high latencies.

Ingo

#310 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 8 lines · Ingo Molnar
Quoted 2 lines · Lee Revell

This is an extreme load situation, so I don't think it will be a real-world problem. I have not seen it under any normal workload.

well, 9 msecs is still not nice. I've been able to trigger larger than 10msec latencies too on a 2 GHz box.

If this is the case then should a make -j12 have ground the machine to a halt the way it did?

Quoted 5 lines

hm, tcp_collapse() in net/ipv4/tcp_input.c. Could you try to just return from that function? Collapsing skbs of a given socket is not a necessary functionality (it is only a 'nice' thing to have in OOM situations) and it indeed can introduce quite high latencies.

OK, will test this.

Lee

#311 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 5 lines · Lee Revell
Quoted 2 lines

well, 9 msecs is still not nice. I've been able to trigger larger than 10msec latencies too on a 2 GHz box.

If this is the case then should a make -j12 have ground the machine to a halt the way it did?

well, does it slow down that much with tracing and voluntary-preempt disabled too?

Ingo

#312 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

Quoted 11 lines · Ingo Molnar
Quoted 5 lines · Lee Revell
Quoted 2 lines

well, 9 msecs is still not nice. I've been able to trigger larger than 10msec latencies too on a 2 GHz box.

If this is the case then should a make -j12 have ground the machine to a halt the way it did?

well, does it slow down that much with tracing and voluntary-preempt disabled too?

Yup. turns out, this is enough to use all available RAM and swap space (1G total). I probably went OOM.

Good stress test though.

Lee

#313 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 2 lines · Lee Revell

This is an extreme load situation, so I don't think it will be a real-world problem. I have not seen it under any normal workload.

well, 9 msecs is still not nice. I've been able to trigger larger than 10msec latencies too on a 2 GHz box.

Here is another one:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/cache_reap_latency_trace.txt

Lee

#314 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P4

In reply to Ingo Molnar
Quoted 14 lines · Ingo Molnar
Quoted 2 lines · Lee Revell

This is an extreme load situation, so I don't think it will be a real-world problem. I have not seen it under any normal workload.

well, 9 msecs is still not nice. I've been able to trigger larger than 10msec latencies too on a 2 GHz box.

Quoted 5 lines

What about this one:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/netif_receive_skb_latency_trace.txt

This appears during normal use.

Here is another:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P4#/var/www/2.6.8.1-P4/rt_run_flush_latency_trace.txt

Lee

#315 lore raw

PATCH 1/1 voluntary-preempt-2.6.8.1-P5

In reply to Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

Changes since -P4:

- increase PERCPU_ENOUGH_ROOM to avoid percpu overflow on SMP. (Mark H Johnson)

- reduce ZAP_BLOCK_SIZE to 16 when vp != 0. This pushes the exit latency down to below 100 usecs on Lee Revell's box.

- added a preempt_count field to /proc/latency_trace. This makes it easier to spot IRQ contexts and generally it gives a nice overview of how the preemption depth changes. It should also help us debug those 900usec weirdnesses related to cpu_idle. (if they still occur)

- made the tcp packet-queue collapsing dependent on VOLUNTARY_PREEMPT.

- fixed 10-20 msec latencies triggered by 'netstat', which occur when there are lots of sockets on a box.

- rediffed against 2.6.8.1 for the patch to apply without fuzz

Ingo

#316 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 4 lines · Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

<snip>

I have been running the voluntary-preempt patches on one of my slower
(450) servers at home. The question is would latency traces from such a slow system be useful to you in this testing. About the most load that gets generated on this system, usually, is compiling the kernel or a very large app. that I do development on. What I tend to do to load the system is just run the stress-kernel suite and sometimes Andrew's amlat program to provide RT scheduling pressure.

If this would be useful, let me know as I have an interest in seeing latencies reduced as much as possible. I am building with the -P5 patch right now. If it would be more useful to try this on a faster system or to stess the system in a different manner, I could do that as well.

kr

#317 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 15 lines · K.R. Foley
Quoted 4 lines · Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

<snip>

I have been running the voluntary-preempt patches on one of my slower
(450) servers at home. The question is would latency traces from such a slow system be useful to you in this testing. About the most load that gets generated on this system, usually, is compiling the kernel or a very large app. that I do development on. What I tend to do to load the system is just run the stress-kernel suite and sometimes Andrew's amlat program to provide RT scheduling pressure.

Quoted 5 lines

If this would be useful, let me know as I have an interest in seeing latencies reduced as much as possible. I am building with the -P5 patch right now. If it would be more useful to try this on a faster system or to stess the system in a different manner, I could do that as well.

sure, all traces are interesting! On your system the latencies will be a bit larger than on others, but that's all - the maximum latency points should be the same (or similar) critical sections. So it's just as useful as the other traces.

Ingo

#318 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

Here is a latency trace generated by the NFS compile in the stress-kernel. It is ~1.7 ms which doesn't really surprise me for the NFS compile. I am going to take the NFS compile out of the test now and see what else falls out.

If it would be useful for me to provide additional information with this, please let me know.

http://www.cybsft.com/testresults/2.6.8.1-P5/latency_trace1.txt

kr

#319 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 4 lines · K.R. Foley

Here is a latency trace generated by the NFS compile in the stress-kernel. It is ~1.7 ms which doesn't really surprise me for the NFS compile. I am going to take the NFS compile out of the test now and see what else falls out.

NFS uses lock_kernel() for most of its locking. This means that the sharing and locking rules and assumptions are not clear (it's all within this opaque lock). I tried to figure out whether it would be safe to do a lock-break in nfs_cached_lookup() but it's quite hard ... Please bug the NFS-client maintainer(s) about this. To break this latency we'd have to do a lock-break in fs/nfs/dir.c:nfs_cached_lookup()'s main loop:

        for(;(page = find_get_page(dir->i_mapping, desc.page_index)); desc.page_index++) {

                res = -EIO;

you could try this at the end of the loop:

                if (res == 0)
                        goto out_found;
                if (res != -EAGAIN)
                        break;
+               cond_resched();
        }

but i cannot guarantee that this is correct ... It might seem to work now but break later under load, etc. It's the NFS maintainer's call.

Quoted 2 lines

If it would be useful for me to provide additional information with this, please let me know.

reports like this are perfect for most latencies. I tried to include everything that is needed normally in /proc/latency_trace itself.

Ingo

#320 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

In reply to K.R. Foley
Quoted 11 lines · K.R. Foley
Quoted 4 lines · Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

Here is a latency trace generated by the NFS compile in the stress-kernel. It is ~1.7 ms which doesn't really surprise me for the NFS compile. I am going to take the NFS compile out of the test now and see what else falls out.

Quoted 5 lines

If it would be useful for me to provide additional information with this, please let me know.

http://www.cybsft.com/testresults/2.6.8.1-P5/latency_trace1.txt

Ugh, XDR rears its ugly head...

Lee

#321 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

In reply to K.R. Foley
Quoted 11 lines · K. R. Foley
Quoted 4 lines · Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

Here is a latency trace generated by the NFS compile in the stress-kernel. It is ~1.7 ms which doesn't really surprise me for the NFS compile. I am going to take the NFS compile out of the test now and see what else falls out.

Quoted 4 lines

If it would be useful for me to provide additional information with this, please let me know.

http://www.cybsft.com/testresults/2.6.8.1-P5/latency_trace1.txt

You don't need to do a NFS compilation to trigger it. just to a "find" on a huge NFS-mounted tree and you will see it.

Another candidate for cond_resched() is the loop in nfs_readdir(). It seems safe to do so be cause it calls readdir_search_pagecache() which would call schedule().

Regards

Signature · 1 line
Linh Dang
#322 lore raw

PATCH 1/1 voluntary-preempt-2.6.8.1-P6

In reply to Ingo Molnar

i've uploaded the -P6 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P6

i'm releasing another patch today because the mystic 900 usec latency seems to be nailed finally, with the help of Mark H Johnson's traces:
there were two places that set ->preempt_count back to 0 without the tracer noticing it: preempt_schedule() and entry.S's return path.

other changes since -P5:

- generic kernel fix: do not ignore idle=poll on non-P4 CPUs. (Mark H Johnson)

- fix the __trace link bug reported by Martin Rumori. All combinations of CONFIG_PREEMPT_TIMING & CONFIG_LATENCY_TRACE should work now.

- make kernel_preemption=1 the default. Most people use this anyway.

Ingo

#324 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

i've uploaded the -P6 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P6

What is the number in the left column?

04000000 0.000ms (+0.000ms): preempt_schedule (unix_stream_sendmsg) 04000000 0.000ms (+0.000ms): schedule (preempt_schedule) 04000001 0.001ms (+0.000ms): sched_clock (schedule) 04000002 0.002ms (+0.001ms): dequeue_task (schedule)

Lee

#325 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

i've uploaded the -P6 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P6

Here's a 171 usec latency from ext3_free_blocks:

preemption latency trace v1.0.1
-------------------------------
latency: 171 us, entries: 2 (2)

    -----------------
    | task: evolution/863, uid:1000 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: ext3_free_blocks+0x1d0/0x4b0
=> ended at: ext3_free_blocks+0x229/0x4b0
=======>
00000001 0.000ms (+0.000ms): ext3_free_blocks (ext3_free_data) 00000001 0.167ms (+0.167ms): sub_preempt_count (ext3_free_blocks)

Also I have noticed a pattern with the XFree86 schedule() latencies, they all have a section like this:

04000002 0.003ms (+0.000ms): effective_prio (recalc_task_prio) 04000002 0.003ms (+0.000ms): enqueue_task (schedule) 00000002 0.006ms (+0.003ms): __switch_to (schedule) 00000002 0.088ms (+0.082ms): finish_task_switch (schedule) 00010002 0.090ms (+0.001ms): do_IRQ (finish_task_switch) 00010003 0.091ms (+0.000ms): mask_and_ack_8259A (do_IRQ)

04000002 0.002ms (+0.000ms): effective_prio (recalc_task_prio) 04000002 0.002ms (+0.000ms): enqueue_task (schedule) 00000002 0.005ms (+0.002ms): __switch_to (schedule) 00000002 0.067ms (+0.062ms): finish_task_switch (schedule) 00010002 0.068ms (+0.001ms): do_IRQ (finish_task_switch) 00010003 0.069ms (+0.000ms): mask_and_ack_8259A (do_IRQ)

I presume the 04000002 -> 00000002 is some interrupt being unmasked (or interrupts being globally enabled), then there's a 60-80 usec latency in schedule().

Lee

#326 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6

Quoted 9 lines · Lee Revell

Also I have noticed a pattern with the XFree86 schedule() latencies, they all have a section like this:

04000002 0.003ms (+0.000ms): effective_prio (recalc_task_prio) 04000002 0.003ms (+0.000ms): enqueue_task (schedule) 00000002 0.006ms (+0.003ms): __switch_to (schedule) 00000002 0.088ms (+0.082ms): finish_task_switch (schedule) 00010002 0.090ms (+0.001ms): do_IRQ (finish_task_switch)

Quoted 3 lines

I presume the 04000002 -> 00000002 is some interrupt being unmasked (or interrupts being globally enabled), then there's a 60-80 usec latency in schedule().

0x04000000 is PREEMPT_ACTIVE - which is just a bit we set to make sure we dont try to preempt recursively.

but the XFree86 latency is interesting indeed. It could be the effect of the now-enlarged ioperm() bitmap! 80 usecs is excessive.

Ingo

#327 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6

Quoted 2 lines · Ingo Molnar

but the XFree86 latency is interesting indeed. It could be the effect of the now-enlarged ioperm() bitmap! 80 usecs is excessive.

yeah, that's the reason. The quick fix is to reduce IO_BITMAP_BITS to 1024 in include/asm-i386/processor.h. I'm working on a real fix.

Ingo

#328 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6

In reply to Lee Revell
Quoted 5 lines · Lee Revell

Here's a 171 usec latency from ext3_free_blocks:

preemption latency trace v1.0.1
-------------------------------
latency: 171 us, entries: 2 (2)

Quoted 8 lines
    -----------------
    | task: evolution/863, uid:1000 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: ext3_free_blocks+0x1d0/0x4b0
=> ended at: ext3_free_blocks+0x229/0x4b0
=======>
00000001 0.000ms (+0.000ms): ext3_free_blocks (ext3_free_data) 00000001 0.167ms (+0.167ms): sub_preempt_count (ext3_free_blocks)

ok, i broke this lock in my tree, will show up in -P7.

Ingo

#329 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6

In reply to Ingo Molnar

preemption latency trace v1.0.1
-------------------------------
latency: 408 us, entries: 2273 (2273)

    -----------------
    | task: ksoftirqd/0/2, uid:0 nice:-10 policy:0 rt_prio:0
    -----------------

=> started at: netif_receive_skb+0x74/0x1f0
=> ended at: netif_receive_skb+0x171/0x1f0
=======>

This one is hard to trim to post here, so full trace is posted on:
http://users.telenet.be/kvogel/network.txt

.config at:
http://users.telenet.be/kvogel/config.txt

#331 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 11 lines · Ingo Molnar

i've uploaded the -P7 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P7

Changes since -P6:

- fixed the XFree86/X.org context-switch latency. (let me know if you see any weirdness like X not starting up while it did before.)

- halved the pagevec size, to reduce the radix gang-lookup costs.

Great, this is a significant improvement. Most of the worst case latencies (~150 usec) seem related to the TCP stack now, and a minor one (51 usec) in the ext3 journaling:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P7

Lee

#332 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 15 lines · Lee Revell
Quoted 11 lines · Ingo Molnar

i've uploaded the -P7 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P7

Changes since -P6:

- fixed the XFree86/X.org context-switch latency. (let me know if you see any weirdness like X not starting up while it did before.)

- halved the pagevec size, to reduce the radix gang-lookup costs.

Quoted 8 lines

Great, this is a significant improvement. Most of the worst case latencies (~150 usec) seem related to the TCP stack now, and a minor one (51 usec) in the ext3 journaling:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P7

Lee

I just posted a similar trace of ~4141 usec from P6 here:

http://www.cybsft.com/testresults/2.6.8.1-P6/latency-trace1.txt

This was part of a run from yesterday evening. After just rebooting, I too am seeing several of these. With the system not being hammered by the stress tests though, the max is only up to ~103 usec. I will be updating to P7 shortly.

kr

#333 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 4 lines · K.R. Foley

I just posted a similar trace of ~4141 usec from P6 here:

http://www.cybsft.com/testresults/2.6.8.1-P6/latency-trace1.txt

This looks wrong:

00000003 0.008ms (+0.001ms): dummy_socket_sock_rcv_skb (tcp_v4_rcv) 00000004 0.008ms (+0.000ms): tcp_v4_do_rcv (tcp_v4_rcv) 00000004 0.009ms (+0.000ms): tcp_rcv_established (tcp_v4_do_rcv) 00010004 3.998ms (+3.989ms): do_IRQ (tcp_rcv_established) 00010005 3.999ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 00010005 4.001ms (+0.002ms): generic_redirect_hardirq (do_IRQ) 00010004 4.002ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)

Probably a false positive, Ingo would know better. What kind of stress testing were you doing?

Lee

#334 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 18 lines · Lee Revell
Quoted 4 lines · K.R. Foley

I just posted a similar trace of ~4141 usec from P6 here:

http://www.cybsft.com/testresults/2.6.8.1-P6/latency-trace1.txt

This looks wrong:

00000003 0.008ms (+0.001ms): dummy_socket_sock_rcv_skb (tcp_v4_rcv) 00000004 0.008ms (+0.000ms): tcp_v4_do_rcv (tcp_v4_rcv) 00000004 0.009ms (+0.000ms): tcp_rcv_established (tcp_v4_do_rcv) 00010004 3.998ms (+3.989ms): do_IRQ (tcp_rcv_established) 00010005 3.999ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 00010005 4.001ms (+0.002ms): generic_redirect_hardirq (do_IRQ) 00010004 4.002ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)

Quoted 5 lines

Probably a false positive, Ingo would know better. What kind of stress testing were you doing?

Lee

This is while running the stress-kernel suite. I don't know about it being a false possitive, it very well may be. Looking back through the logs though I am not sure this is a valid latency anyway. This trace was from 06:37 this morning. About 19:54 last night I got an oops in kswapd and this morning around 10:30ish I had stuff getting killed right and left by oom. So, I am thinking that this probably is not a very reliable test.

kr

#335 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

In reply to Lee Revell
Quoted 16 lines · Lee Revell
Quoted 4 lines · K.R. Foley

I just posted a similar trace of ~4141 usec from P6 here:

http://www.cybsft.com/testresults/2.6.8.1-P6/latency-trace1.txt

This looks wrong:

00000003 0.008ms (+0.001ms): dummy_socket_sock_rcv_skb (tcp_v4_rcv) 00000004 0.008ms (+0.000ms): tcp_v4_do_rcv (tcp_v4_rcv) 00000004 0.009ms (+0.000ms): tcp_rcv_established (tcp_v4_do_rcv) 00010004 3.998ms (+3.989ms): do_IRQ (tcp_rcv_established) 00010005 3.999ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 00010005 4.001ms (+0.002ms): generic_redirect_hardirq (do_IRQ) 00010004 4.002ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)

Quoted 2 lines

Probably a false positive, Ingo would know better. What kind of stress testing were you doing?

indeed this looks suspect. Is this an SMP system?

Ingo

#336 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 20 lines · Ingo Molnar
Quoted 17 lines · Lee Revell
Quoted 4 lines · K.R. Foley

I just posted a similar trace of ~4141 usec from P6 here:

http://www.cybsft.com/testresults/2.6.8.1-P6/latency-trace1.txt

This looks wrong:

00000003 0.008ms (+0.001ms): dummy_socket_sock_rcv_skb (tcp_v4_rcv) 00000004 0.008ms (+0.000ms): tcp_v4_do_rcv (tcp_v4_rcv) 00000004 0.009ms (+0.000ms): tcp_rcv_established (tcp_v4_do_rcv) 00010004 3.998ms (+3.989ms): do_IRQ (tcp_rcv_established) 00010005 3.999ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 00010005 4.001ms (+0.002ms): generic_redirect_hardirq (do_IRQ) 00010004 4.002ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)

Quoted 8 lines
Quoted 2 lines

Probably a false positive, Ingo would know better. What kind of stress testing were you doing?

indeed this looks suspect. Is this an SMP system?

Ingo

Actually no. It is an SMP ready system, but with a single PII 450. As I responded to Lee's response, I am not sure that I completely trust the results of this trace anyway.

I would like to know why you guys think this may be a false positive. Is it just the extremely long latency? Or is there something else that makes it look suspect?

By the way I just posted two more traces, one that I caught last night and one from this morning:

This is another one similar to the last but much more reasonably latency:

http://www.cybsft.com/testresults/2.6.8.1-P7/2.6.8.1-P7-1.txt

And this one this morning appears to be from updatedb running, while the
tests were running. It's worth noting that this one appears to have
happened about the same time today that the other ~4100+ one happened yesterday. Also worth noting is that the system was probably swapping pretty good when this occurred.

http://www.cybsft.com/testresults/2.6.8.1-P7/2.6.8.1-P7-2.txt

kr

#338 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 7 lines · Lee Revell
Quoted 3 lines · Ingo Molnar

i've uploaded the -P7 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P7

Here's a pretty bad one, over 500 usecs in rt_garbage_collect:

Grr, apt-get upgrade (debian unstable) broke php on my web server. subsequent apt-get upgrades have not fixed it, and I have not had time to file a bug report. Traces can be found here:

http://krustophenia.net/2.6.8.1-P7

Lee

#339 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 3 lines · Lee Revell

to file a bug report. Traces can be found here:

http://krustophenia.net/2.6.8.1-P7

re the skb latencies: Mark H Johnson managed to reduce net-input latencies by decreasing /proc/sys/net/core/netdev_max_backlog to 8 - could you try this, does it help? (you could try an even more extreme setting like 4.)

Ingo

#340 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Lee Revell

to file a bug report. Traces can be found here:

http://krustophenia.net/2.6.8.1-P7

re the skb latencies: Mark H Johnson managed to reduce net-input latencies by decreasing /proc/sys/net/core/netdev_max_backlog to 8 - could you try this, does it help? (you could try an even more extreme setting like 4.)

Spoke too soon with my last report. I am actually still seeing latencies close to 200 usecs in netdev_process_backlog. Reducing netdev_max_backlog to 4, then 2 did not seem to help.

netdev_process_backlog = 4:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P8#/var/www/2.6.8.1-P8/trace9.txt

netdev_process_backlog = 2:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P8#/var/www/2.6.8.1-P8/trace12.txt

Lee

#341 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

In reply to Lee Revell
Quoted 9 lines · Lee Revell
Quoted 3 lines · Ingo Molnar

i've uploaded the -P7 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P7

Here's a pretty bad one, over 500 usecs in rt_garbage_collect:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P7#/var/www/2.6.8.1-P7/trace7.txt

Thanks to everyone who pointed out that PHP had broken on my web server. It's fixed, all the links in these threads should work now.

Lee

#342 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar

i've uploaded the -P7 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P7

Changes since -P6:

- fixed the XFree86/X.org context-switch latency. (let me know if you see any weirdness like X not starting up while it did before.)

- halved the pagevec size, to reduce the radix gang-lookup costs.

It does not compile on x86-64. I get this:

CC arch/x86_64/ia32/syscall32.o
kernel/hardirq.c: In function `recalculate_desc_flags':
kernel/hardirq.c:309: error: `SA_NODELAY' undeclared (first use in this function)
kernel/hardirq.c:309: error: (Each undeclared identifier is reported only once kernel/hardirq.c:309: error: for each function it appears in.) kernel/hardirq.c: In function `generic_setup_irq':
kernel/hardirq.c:339: error: `SA_NODELAY' undeclared (first use in this function)
kernel/hardirq.c: In function `threaded_read_proc':
kernel/hardirq.c:611: error: `SA_NODELAY' undeclared (first use in this function)
kernel/hardirq.c: In function `threaded_write_proc':
kernel/hardirq.c:629: error: `SA_NODELAY' undeclared (first use in this function)
make[1]: *** [kernel/hardirq.o] Error 1
make: *** [kernel] Error 2

The kernel config is attached.

Greetings,

Signature · 5 lines
Rafael J. Wysocki
----------------------------
For a successful technology, reality must take precedence over public 
relations, for nature cannot be fooled.
					-- Richard P. Feynman

#
# Automatically generated make config: don't edit
#
CONFIG_X86_64=y
CONFIG_64BIT=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_X86_CMPXCHG=y
CONFIG_EARLY_PRINTK=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_GENERIC_ISA_DMA=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y

#
# General setup
#
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HOTPLUG=y
# CONFIG_IKCONFIG is not set
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Processor type and features
#
CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_GOOD_APIC=y
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_MTRR=y
CONFIG_SMP=y
# CONFIG_PREEMPT is not set
# CONFIG_SCHED_SMT is not set
CONFIG_K8_NUMA=y
CONFIG_DISCONTIGMEM=y
CONFIG_NUMA=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_NR_CPUS=2
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_X86_MCE=y

#
# Power management options
#
CONFIG_PM=y
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support #
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
# CONFIG_ACPI_SLEEP is not set
CONFIG_ACPI_AC=y
# CONFIG_ACPI_BATTERY is not set CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
# CONFIG_CPU_FREQ_PROC_INTF is not set CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_GOV_POWERSAVE is not set # CONFIG_CPU_FREQ_GOV_USERSPACE is not set CONFIG_CPU_FREQ_TABLE=y

#
# CPUFreq processor drivers
#
CONFIG_X86_POWERNOW_K8=m

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y

#
# PCMCIA/CardBus support
#
# CONFIG_PCMCIA is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=y
CONFIG_COMPAT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_UID16=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_FW_LOADER is not set
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_PC_CML1=m
# CONFIG_PARPORT_SERIAL is not set
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_OTHER=y
CONFIG_PARPORT_1284=y

#
# Plug and Play support
#

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_LBD=y

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives #
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=m
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set CONFIG_BLK_DEV_IDESCSI=m
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_IDE_TASKFILE_IO is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_OFFBOARD=y
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
CONFIG_BLK_DEV_ADMA=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=y
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_IVB=y
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
# CONFIG_SCSI_MULTI_LUN is not set CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=y
# CONFIG_SCSI_FC_ATTRS is not set

#
# SCSI low-level drivers
#
CONFIG_BLK_DEV_3W_XXXX_RAID=y
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_MEGARAID is not set
CONFIG_SCSI_SATA=y
# CONFIG_SCSI_SATA_SVW is not set
# CONFIG_SCSI_ATA_PIIX is not set
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_SX4 is not set
CONFIG_SCSI_SATA_SIL=y
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_VIA is not set
# CONFIG_SCSI_SATA_VITESSE is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_SCSI_SYM53C8XX_IOMAPPED is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA6322 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
CONFIG_SCSI_DEBUG=m

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
# CONFIG_MD_LINEAR is not set
# CONFIG_MD_RAID0 is not set
CONFIG_MD_RAID1=y
CONFIG_MD_RAID5=y
# CONFIG_MD_RAID6 is not set
CONFIG_MD_MULTIPATH=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_IEEE1394=m

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_IEEE1394_OUI_DB is not set
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
CONFIG_IEEE1394_CONFIG_ROM_IP1394=y

#
# Device Drivers
#

#
# Texas Instruments PCILynx requires I2C
#
CONFIG_IEEE1394_OHCI1394=m

#
# Protocol Drivers
#
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_DV1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_CMP=m
CONFIG_IEEE1394_AMDTP=m

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=m
CONFIG_PACKET_MMAP=y
# CONFIG_NETLINK_DEV is not set
CONFIG_UNIX=y
CONFIG_NET_KEY=y
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=y
CONFIG_IP_NF_FTP=y
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_LOCAL=y
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# CONFIG_IP_NF_RAW is not set
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_REALM is not set
CONFIG_XFRM=y
CONFIG_XFRM_USER=m

#
# SCTP Configuration (EXPERIMENTAL)
#
CONFIG_IP_SCTP=m
CONFIG_SCTP_DBG_MSG=y
CONFIG_SCTP_DBG_OBJCNT=y
CONFIG_SCTP_HMAC_NONE=y
# CONFIG_SCTP_HMAC_SHA1 is not set
# CONFIG_SCTP_HMAC_MD5 is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_HW_FLOWCONTROL is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=m
# CONFIG_TYPHOON is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
CONFIG_AMD8111_ETH=m
# CONFIG_AMD8111E_NAPI is not set
# CONFIG_ADAPTEC_STARFIRE is not set
CONFIG_B44=m
# CONFIG_FORCEDETH is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=m
# CONFIG_E100_NAPI is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_VELOCITY is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
CONFIG_TIGON3=m

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPPOE=m
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
CONFIG_SHAPER=m
# CONFIG_NETCONSOLE is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1280
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1024
CONFIG_INPUT_JOYDEV=m
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input I/O drivers
#
CONFIG_GAMEPORT=m
CONFIG_SOUND_GAMEPORT=m
# CONFIG_GAMEPORT_NS558 is not set
# CONFIG_GAMEPORT_L4 is not set
# CONFIG_GAMEPORT_EMU10K1 is not set
# CONFIG_GAMEPORT_VORTEX is not set
# CONFIG_GAMEPORT_FM801 is not set
# CONFIG_GAMEPORT_CS461x is not set
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
CONFIG_SERIO_PCIPS2=m

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
# CONFIG_SERIAL_8250_MULTIPORT is not set
# CONFIG_SERIAL_8250_RSA is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set
# CONFIG_QIC02_TAPE is not set

#
# IPMI
#
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
# CONFIG_IPMI_PANIC_STRING is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
# CONFIG_IPMI_SI is not set
CONFIG_IPMI_WATCHDOG=m

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=y
CONFIG_NVRAM=m
CONFIG_RTC=y
CONFIG_DTLK=m
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set

#
# Ftape, the floppy tape device driver
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL_MCH is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_GAMMA is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m

#
# I2C support
#
# CONFIG_I2C is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
CONFIG_FB=y
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
# CONFIG_FB_HGA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON_OLD is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_FONT_6x11 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
# CONFIG_FONT_MINI_4x6 is not set
# CONFIG_FONT_SUN8x16 is not set
# CONFIG_FONT_SUN12x22 is not set

#
# Logo configuration
#
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_BIT32_EMUL=m
CONFIG_SND_RTCTIMER=m
CONFIG_SND_VERBOSE_PRINTK=y
CONFIG_SND_DEBUG=y
CONFIG_SND_DEBUG_MEMORY=y
CONFIG_SND_DEBUG_DETECT=y

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VX222 is not set

#
# ALSA USB devices
#
# CONFIG_SND_USB_AUDIO is not set

#
# Open Sound System
#
CONFIG_SOUND_PRIME=m
# CONFIG_SOUND_BT878 is not set
# CONFIG_SOUND_CMPCI is not set
# CONFIG_SOUND_EMU10K1 is not set
# CONFIG_SOUND_FUSION is not set
# CONFIG_SOUND_CS4281 is not set
# CONFIG_SOUND_ES1370 is not set
# CONFIG_SOUND_ES1371 is not set
# CONFIG_SOUND_ESSSOLO1 is not set
# CONFIG_SOUND_MAESTRO is not set
# CONFIG_SOUND_MAESTRO3 is not set
CONFIG_SOUND_ICH=m
# CONFIG_SOUND_SONICVIBES is not set
# CONFIG_SOUND_TRIDENT is not set
# CONFIG_SOUND_MSNDCLAS is not set
# CONFIG_SOUND_MSNDPIN is not set
# CONFIG_SOUND_VIA82CXXX is not set
# CONFIG_SOUND_OSS is not set
# CONFIG_SOUND_ALI5455 is not set
# CONFIG_SOUND_FORTE is not set
# CONFIG_SOUND_RME96XX is not set
# CONFIG_SOUND_AD1980 is not set

#
# USB support
#
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_BANDWIDTH=y
CONFIG_USB_DYNAMIC_MINORS=y

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_UHCI_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_AUDIO=m
# CONFIG_USB_BLUETOOTH_TTY is not set
# CONFIG_USB_MIDI is not set
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=m
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_RW_DETECT is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_HP8200e=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y

#
# USB Human Interface Devices (HID)
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y

#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
CONFIG_USB_HPUSBSCSI=m

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set

#
# Video4Linux support is needed for USB Multimedia device support #

#
# USB Network adaptors
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OMNINET is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_TIGL is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETSERVO is not set
CONFIG_USB_TEST=m

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_QUOTA=y
CONFIG_QFMT_V1=y
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=m
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_SMB_FS=m
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_SMB_NLS_REMOTE="cp852"
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_XATTR is not set
CONFIG_CIFS_POSIX=y
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-2"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
CONFIG_NLS_CODEPAGE_852=m
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=y
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=m
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y

#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SLAB=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_INIT_DEBUG is not set
# CONFIG_DEBUG_INFO is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_IOMMU_DEBUG is not set

#
# Security options
#
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_CAPABILITIES=y
CONFIG_SECURITY_ROOTPLUG=m
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
# CONFIG_SECURITY_SELINUX_DISABLE is not set
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_MLS=y

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_TEST=m

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y

#345 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 11 lines · Ingo Molnar

i've uploaded the -P8 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P8

Changes since -P8:

- fixes the DRI/DRM latency in radeon (and other drivers). The concept was investigated/tested by Dave Airlie.

- reduce netdev_max_backlog to 8 (Mark H Johnson)

Should this fix the 500+ usec latency I saw in rt_garbage_collect? This one took a while to occur (overnight).

Lee

#346 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 7 lines · Lee Revell
Quoted 6 lines · Ingo Molnar

i've uploaded the -P8 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P8

Changes since -P8:

Quoted 8 lines
Quoted 5 lines

- fixes the DRI/DRM latency in radeon (and other drivers). The concept was investigated/tested by Dave Airlie.

- reduce netdev_max_backlog to 8 (Mark H Johnson)

Should this fix the 500+ usec latency I saw in rt_garbage_collect? This one took a while to occur (overnight).

i dont think it will. Does the patch below help?

Ingo

ipv4/route.c renamed from ipv4/route.c.orig +1 -1
@@ -738,7 +738,7 @@ static int rt_garbage_collect(void)
 
 		if (atomic_read(&ipv4_dst_ops.entries) < ip_rt_max_size)
 			goto out;
-	} while (!in_softirq() && time_before_eq(jiffies, now));
+	} while (!in_softirq() && time_before_eq(jiffies, now) && !need_resched());
 
 	if (atomic_read(&ipv4_dst_ops.entries) < ip_rt_max_size)
 		goto out;
#347 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Lee Revell
Quoted 3 lines · Ingo Molnar

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P8

Should this fix the 500+ usec latency I saw in rt_garbage_collect? This one took a while to occur (overnight).

i dont think it will. Does the patch below help?

Probably. This one is pretty rare, it might take a day to recur, and it's pretty clear the patch will fix the problem, so I recommend adding it to P9.

Lee

#348 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

In reply to Ingo Molnar
Quoted 12 lines · Ingo Molnar
Quoted 2 lines · Lee Revell

Should this fix the 500+ usec latency I saw in rt_garbage_collect? This one took a while to occur (overnight).

i dont think it will. Does the patch below help?

Ingo

ipv4/route.c static int rt_garbage_collect(void) · line 744

Quoted 8 lines
ipv4/route.c renamed from ipv4/route.c.orig +1 -1
@@ -738,7 +738,7 @@ static int rt_garbage_collect(void)
 
 		if (atomic_read(&ipv4_dst_ops.entries) < ip_rt_max_size)
 			goto out;
-	} while (!in_softirq() && time_before_eq(jiffies, now));
+	} while (!in_softirq() && time_before_eq(jiffies, now) && !need_resched());
 
 	if (atomic_read(&ipv4_dst_ops.entries) < ip_rt_max_size)
 		goto out;

Nope, the above does not actually fix it. I got a 716 usec latency in rt_garbage_collect:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P9#/var/www/2.6.8.1-P9/trace12.txt

I believe this is associated with heavy route cache activity, I did not see this one again until I left a gnutella client running overnight.

Lee

#349 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar

i've uploaded the -P8 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P8

Changes since -P8:

- fixes the DRI/DRM latency in radeon (and other drivers). The concept was investigated/tested by Dave Airlie.

- reduce netdev_max_backlog to 8 (Mark H Johnson)

With -P8 I have not seen a latency over 100 usecs yet. The most common latencies are those involving zap_pte_range, journal_get_undo_access and netdev_process_backlog (even with netdev_max_backlog set to 4).

Traces:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P8

Lee

#350 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

In reply to Ingo Molnar
Quoted 2 lines · Ingo Molnar

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related latencies. I tested setting netdev_max_backlog to every power of two between 1 and 128, and regardless of this setting, I can produce a 450-600 usec latency with:

ping -s 65507 -f $DEFAULT_GATEWAY

Looks like skb_checksum is the problem. Here is one of the traces:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P8#/var/www/2.6.8.1-P8/trace14.txt

Lee

#351 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 13 lines · Lee Revell
Quoted 2 lines · Ingo Molnar

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related
latencies. I tested setting netdev_max_backlog to every power of two between 1 and 128, and regardless of this setting, I can produce a 450-600 usec latency with:

ping -s 65507 -f $DEFAULT_GATEWAY

Quoted 7 lines

Looks like skb_checksum is the problem. Here is one of the traces:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P8#/var/www/2.6.8.1-P8/trace14.txt

Lee

-

Setting netdev_max_backlog doesn't seem to have any effect for me either.

I get a similar latency when I try to reproduce your result above, ~612 usec. However, my trace is very different than yours:

http://www.cybsft.com/testresults/2.6.8.1-P8/latency_trace3.txt

In fact most any network activity, with the POSSIBLE exception of already open connections, seem to be able to trigger higher latencies. As you can see here:

http://www.cybsft.com/testresults/2.6.8.1-P8/

kr

#352 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 21 lines · K.R. Foley
Quoted 20 lines · Lee Revell
Quoted 2 lines · Ingo Molnar

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related latencies. I tested setting netdev_max_backlog to every power of two between 1 and 128, and regardless of this setting, I can produce a 450-600 usec latency with:

ping -s 65507 -f $DEFAULT_GATEWAY

Looks like skb_checksum is the problem. Here is one of the traces:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P8#/var/www/2.6.8.1-P8/trace14.txt

Lee

Quoted 9 lines
Quoted 1 line

-

Setting netdev_max_backlog doesn't seem to have any effect for me either.

I get a similar latency when I try to reproduce your result above, ~612 usec. However, my trace is very different than yours:

http://www.cybsft.com/testresults/2.6.8.1-P8/latency_trace3.txt

I have some that are similar to this. Looks like many of the differences are due to different network drivers (e100 vs via-rhine).

Quoted 6 lines

In fact most any network activity, with the POSSIBLE exception of already open connections, seem to be able to trigger higher latencies. As you can see here:

http://www.cybsft.com/testresults/2.6.8.1-P8/

Looks like except for the one above, all of these involve the extract_entropy issue. I am using a hack to disable the random driver to avoid these latencies.

There are a few other threads going on discussing a real solution to the extract_entropy issue. However, none of the solutions that has been posted yet fix the problem.

Lee

#354 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 11 lines · Ingo Molnar
Quoted 1 line · K.R. Foley

http://www.cybsft.com/testresults/2.6.8.1-P8/latency_trace3.txt

this trace shows a TX processing latency of 45 frames - please try 'ifconfig eth0 txqueuelen 8', does it help?

Ingo

There's no difference that jumps out at me. Here are a couple more with txqueuelen=8

http://www.cybsft.com/testresults/2.6.8.1-P8/latency_trace7.txt
http://www.cybsft.com/testresults/2.6.8.1-P8/latency_trace8.txt

kr

#355 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

In reply to Lee Revell
Quoted 4 lines · Lee Revell
Quoted 1 line

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related latencies. [...]

it has an effect on input queue length. Output queue lengths can be reduced via 'ifconfig eth0 txqueuelen 8'.

Ingo

#356 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 10 lines · Ingo Molnar
Quoted 4 lines · Lee Revell
Quoted 1 line

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related latencies. [...]

it has an effect on input queue length. Output queue lengths can be reduced via 'ifconfig eth0 txqueuelen 8'.

OK. With both of these set to 4, the largest latency I was able to generate by ping flooding was 253 usecs.

Is there any particular reason one of these parameters is set via a sysctl/proc entry, and one by ifconfig?

Lee

#357 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 8 lines · Lee Revell
Quoted 2 lines

it has an effect on input queue length. Output queue lengths can be reduced via 'ifconfig eth0 txqueuelen 8'.

OK. With both of these set to 4, the largest latency I was able to generate by ping flooding was 253 usecs.

Is there any particular reason one of these parameters is set via a sysctl/proc entry, and one by ifconfig?

well, the input packet backlog is global, the output one is local. No good reason otherwise.

Ingo

#358 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

In reply to Lee Revell
Quoted 10 lines · Lee Revell
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · Lee Revell
Quoted 1 line

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related latencies. [...]

Quoted 8 lines
Quoted 3 lines

it has an effect on input queue length. Output queue lengths can be reduced via 'ifconfig eth0 txqueuelen 8'.

OK. With both of these set to 4, the largest latency I was able to generate by ping flooding was 253 usecs.

Even with both of these set to 4, I still get similar results ~647 usec :(

<snip>

#359 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Quoted 16 lines · K.R. Foley
Quoted 15 lines · Lee Revell
Quoted 11 lines · Ingo Molnar
Quoted 4 lines · Lee Revell
Quoted 1 line

- reduce netdev_max_backlog to 8 (Mark H Johnson)

On my system this setting has absolutely no effect on the skb related latencies. [...]

it has an effect on input queue length. Output queue lengths can be reduced via 'ifconfig eth0 txqueuelen 8'.

Quoted 6 lines
Quoted 3 lines

OK. With both of these set to 4, the largest latency I was able to generate by ping flooding was 253 usecs.

Even with both of these set to 4, I still get similar results ~647 usec :(

I am able to generate a 1012 usec latency by flood pinging the broadcast address. These are pretty pathological cases, but if we are going for bounded latency it seems like they should be addressed.

Lee

#360 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P8

Lee Revell wrote:
<snip>

Quoted 7 lines

I am able to generate a 1012 usec latency by flood pinging the broadcast address. These are pretty pathological cases, but if we are going for bounded latency it seems like they should be addressed.

Lee

I agree, if that is possible. Whether it's possible in all cases is not something I am really qualified to answer.

kr

#361 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P7

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

i've uploaded the -P7 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P7

Here are a couple more traces in the ~200 - ~300 usec range:

http://www.cybsft.com/testresults/2.6.8.1-P7/2.6.8.1-P7-4.txt

http://www.cybsft.com/testresults/2.6.8.1-P7/2.6.8.1-P7-10.txt

Also, can someone tell me why I often see traces in the 4 ms range with
3 - 4 ms being reported by a single routine?

kr

#362 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6 -- False positive?!

In reply to Ingo Molnar

I'm not sure if the following is a false positive or not:

preemption latency trace v1.0.1
-------------------------------
latency: 3022 us, entries: 15 (15)

    -----------------
    | task: swapper/0, uid:0 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: do_IRQ+0x27/0x230
=> ended at: do_IRQ+0x1a3/0x230
=======>
00010000 0.000ms (+0.000ms): do_IRQ (common_interrupt) 00010000 0.000ms (+0.000ms): do_IRQ (acpi_processor_idle) 00010001 0.000ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 00010001 0.000ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 00010001 2.953ms (+2.952ms): wake_up_process (generic_redirect_hardirq) 00010001 2.953ms (+0.000ms): try_to_wake_up (wake_up_process) 00010001 2.953ms (+0.000ms): task_rq_lock (try_to_wake_up) 00010002 2.953ms (+0.000ms): activate_task (try_to_wake_up) 00010002 2.953ms (+0.000ms): sched_clock (activate_task) 00010002 2.953ms (+0.000ms): recalc_task_prio (activate_task) 00010002 2.953ms (+0.000ms): effective_prio (recalc_task_prio) 00010002 2.953ms (+0.000ms): enqueue_task (activate_task) 00010001 2.954ms (+0.000ms): preempt_schedule (try_to_wake_up) 00010000 2.954ms (+0.000ms): preempt_schedule (do_IRQ) 00000001 2.954ms (+0.000ms): sub_preempt_count (do_IRQ)

The following looks similar:

latency: 517 us, entries: 29 (29)

    -----------------
    | task: swapper/0, uid:0 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: do_IRQ+0x27/0x230
=> ended at: do_IRQ+0x1a3/0x230
=======>
00010000 0.000ms (+0.000ms): do_IRQ (common_interrupt) 00010000 0.000ms (+0.000ms): do_IRQ (acpi_processor_idle) 00010001 0.000ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 00010001 0.000ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 00010000 0.000ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 00010000 0.000ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 00010001 0.000ms (+0.000ms): mark_offset_tsc (timer_interrupt) 00010001 0.009ms (+0.008ms): do_timer (timer_interrupt) 00010001 0.503ms (+0.493ms): update_process_times (do_timer) 00010001 0.503ms (+0.000ms): update_one_process (update_process_times) 00010001 0.503ms (+0.000ms): run_local_timers (update_process_times) 00010001 0.503ms (+0.000ms): raise_softirq (update_process_times) 00010001 0.503ms (+0.000ms): scheduler_tick (update_process_times) 00010001 0.504ms (+0.000ms): sched_clock (scheduler_tick) 00010001 0.504ms (+0.000ms): update_wall_time (do_timer) 00010001 0.504ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 00010001 0.504ms (+0.000ms): generic_note_interrupt (do_IRQ) 00010001 0.504ms (+0.000ms): end_edge_ioapic_irq (do_IRQ) 00000001 0.504ms (+0.000ms): do_softirq (do_IRQ)
00000100 0.505ms (+0.000ms): __do_softirq (do_softirq) 00000100 0.505ms (+0.000ms): wake_up_process (do_softirq) 00000100 0.505ms (+0.000ms): try_to_wake_up (wake_up_process) 00000100 0.505ms (+0.000ms): task_rq_lock (try_to_wake_up) 00000101 0.505ms (+0.000ms): activate_task (try_to_wake_up) 00000101 0.505ms (+0.000ms): sched_clock (activate_task) 00000101 0.505ms (+0.000ms): recalc_task_prio (activate_task) 00000101 0.505ms (+0.000ms): effective_prio (recalc_task_prio) 00000101 0.505ms (+0.000ms): enqueue_task (activate_task) 00000001 0.506ms (+0.000ms): sub_preempt_count (do_IRQ)

#363 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P6 -- False positive?!

I think this may have been my own mistake. In between all the kernel compiles, I seem to have managed to do the following:

# CONFIG_X86_PM_TIMER is not set

Which was probably the cause. (was getting other side effects too, like keys stuck in repeat, 1-2 second freezes etc etc)

#364 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar

i've uploaded the -P5 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P5

- added a preempt_count field to /proc/latency_trace. This makes it easier to spot IRQ contexts and generally it gives a nice overview of how the preemption depth changes. It should also help us debug those 900usec weirdnesses related to cpu_idle. (if they still occur)

Ok, disregard my question about what this field was.

Quoted 2 lines

- made the tcp packet-queue collapsing dependent on VOLUNTARY_PREEMPT.

I am seeing ~180usec latencies related to netif_receive_skb:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace5.txt

Lee

#366 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 9 lines · Lee Revell
Quoted 3 lines · Ingo Molnar

- reduce ZAP_BLOCK_SIZE to 16 when vp != 0. This pushes the exit latency down to below 100 usecs on Lee Revell's box.

Almost:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace8.txt

i suspect if you turn off tracing (or disable it in the kernel completely) then you'd see below-100 usec latencies here. A trace entry costs ~0.3 usecs on your box, so this 50-entry trace has roughly 15 usecs of tracing overhead :-)

Ingo

#367 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Lee Revell
Quoted 3 lines · Ingo Molnar

- reduce ZAP_BLOCK_SIZE to 16 when vp != 0. This pushes the exit latency down to below 100 usecs on Lee Revell's box.

Quoted 9 lines
Quoted 3 lines

Almost:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace8.txt

i suspect if you turn off tracing (or disable it in the kernel completely) then you'd see below-100 usec latencies here. A trace entry costs ~0.3 usecs on your box, so this 50-entry trace has roughly 15 usecs of tracing overhead :-)

FWIW, I did see one of these over 200 usecs.

Lee

#368 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 9 lines · Lee Revell
Quoted 7 lines
Quoted 1 line

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace8.txt

i suspect if you turn off tracing (or disable it in the kernel completely) then you'd see below-100 usec latencies here. A trace entry costs ~0.3 usecs on your box, so this 50-entry trace has roughly 15 usecs of tracing overhead :-)

FWIW, I did see one of these over 200 usecs.

which trace is this?

Ingo

#369 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Lee Revell
Quoted 7 lines
Quoted 1 line

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace8.txt

i suspect if you turn off tracing (or disable it in the kernel completely) then you'd see below-100 usec latencies here. A trace entry costs ~0.3 usecs on your box, so this 50-entry trace has roughly 15 usecs of tracing overhead :-)

Quoted 4 lines
Quoted 1 line

FWIW, I did see one of these over 200 usecs.

which trace is this?

I don't think I was able to save this one. I will post it if I see it again.

Lee

#370 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Lee Revell
Quoted 7 lines
Quoted 1 line

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace8.txt

i suspect if you turn off tracing (or disable it in the kernel completely) then you'd see below-100 usec latencies here. A trace entry costs ~0.3 usecs on your box, so this 50-entry trace has roughly 15 usecs of tracing overhead :-)

Quoted 4 lines
Quoted 1 line

FWIW, I did see one of these over 200 usecs.

which trace is this?

Sorry, I was thinking of this one:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace10.txt

Lee

#372 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Lee Revell

Sorry, I was thinking of this one:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P6#/var/www/2.6.8.1-P6/trace10.txt

ok, reduced the zap block size to 8 pages a time. It should not be a big issue to have this at a low value because we dont retry anything and we do a full TLB flush only once. (subsequent flushes are caught by the tlb->need_flush optimization.)

This one is interesting. What is going on here?

preemption latency trace v1.0.1
-------------------------------
latency: 146 us, entries: 45 (45)

    -----------------
    | task: pdflush/33, uid:0 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: find_get_pages_tag+0x19/0x90
=> ended at: find_get_pages_tag+0x61/0x90
=======>
00000001 0.000ms (+0.000ms): find_get_pages_tag (pagevec_lookup_tag) 00000001 0.000ms (+0.000ms): radix_tree_gang_lookup_tag (find_get_pages_tag) 00000001 0.001ms (+0.000ms): __lookup_tag (radix_tree_gang_lookup_tag) 00000001 0.007ms (+0.005ms): __lookup_tag (radix_tree_gang_lookup_tag)

[ 20-30 more of these ]

00000001 0.119ms (+0.003ms): __lookup_tag (radix_tree_gang_lookup_tag) 00000001 0.125ms (+0.005ms): __lookup_tag (radix_tree_gang_lookup_tag) 00010001 0.132ms (+0.007ms): do_IRQ (find_get_pages_tag) 00010002 0.133ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 00010002 0.137ms (+0.004ms): generic_redirect_hardirq (do_IRQ) 00010002 0.138ms (+0.000ms): wake_up_process (generic_redirect_hardirq) 00010002 0.138ms (+0.000ms): try_to_wake_up (wake_up_process) 00010002 0.138ms (+0.000ms): task_rq_lock (try_to_wake_up) 00010003 0.139ms (+0.000ms): activate_task (try_to_wake_up) 00010003 0.139ms (+0.000ms): sched_clock (activate_task) 00010003 0.140ms (+0.000ms): recalc_task_prio (activate_task) 00010003 0.141ms (+0.000ms): effective_prio (recalc_task_prio) 00010003 0.141ms (+0.000ms): enqueue_task (activate_task) 00010002 0.141ms (+0.000ms): preempt_schedule (try_to_wake_up) 00010001 0.142ms (+0.000ms): preempt_schedule (do_IRQ) 00000001 0.143ms (+0.001ms): sub_preempt_count (find_get_pages_tag)

Lee

#373 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

Quoted 17 lines · Lee Revell

This one is interesting. What is going on here?

preemption latency trace v1.0.1
-------------------------------
latency: 146 us, entries: 45 (45)

    -----------------
    | task: pdflush/33, uid:0 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: find_get_pages_tag+0x19/0x90
=> ended at: find_get_pages_tag+0x61/0x90
=======>
00000001 0.000ms (+0.000ms): find_get_pages_tag (pagevec_lookup_tag) 00000001 0.000ms (+0.000ms): radix_tree_gang_lookup_tag (find_get_pages_tag) 00000001 0.001ms (+0.000ms): __lookup_tag (radix_tree_gang_lookup_tag) 00000001 0.007ms (+0.005ms): __lookup_tag (radix_tree_gang_lookup_tag)

Quoted 1 line

[ 20-30 more of these ]

this is the pagevec code, triggering the radix tree multi-entry-lookup code. Could you try to reduce PAGEVEC_SIZE from 16 to 8 (or 4) in include/linux/pagevec.h - does this reduce these particular latencies?

Ingo

#374 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P5

In reply to Ingo Molnar

I'm regularly seeing acpi_ec_read latencies. (e.g. when closing the lid of my notebook, when the processor cooler activates, ...)

I'm guessing there isn't much that can be done about this, right?!

preemption latency trace v1.0.1
-------------------------------
latency: 1151 us, entries: 94 (94)

    -----------------
    | task: kacpid/6, uid:0 nice:-10 policy:0 rt_prio:0
    -----------------

=> started at: acpi_ec_read+0x6b/0xf7
=> ended at: acpi_ec_read+0xc8/0xf7
=======>
00000001 0.000ms (+0.000ms): acpi_ec_read (acpi_ec_space_handler) 00000001 0.000ms (+0.000ms): acpi_hw_low_level_write (acpi_ec_read) 00000001 0.000ms (+0.000ms): acpi_os_write_port (acpi_hw_low_level_write) 00000001 0.001ms (+0.001ms): acpi_ec_wait (acpi_ec_read) 00000001 0.001ms (+0.000ms): acpi_hw_low_level_read (acpi_ec_wait) 00000001 0.001ms (+0.000ms): acpi_os_read_port (acpi_hw_low_level_read) 00000001 0.003ms (+0.001ms): __const_udelay (acpi_ec_wait) 00000001 0.003ms (+0.000ms): __delay (acpi_ec_wait)
00000001 0.003ms (+0.000ms): delay_tsc (__delay)
[...]
00000001 1.113ms (+0.001ms): smp_apic_timer_interrupt (acpi_ec_read) 00010001 1.113ms (+0.000ms): profile_hook (smp_apic_timer_interrupt) 00010002 1.113ms (+0.000ms): notifier_call_chain (profile_hook) 00000002 1.113ms (+0.000ms): do_softirq (smp_apic_timer_interrupt) 00000100 1.113ms (+0.000ms): __do_softirq (do_softirq) 00010001 1.113ms (+0.000ms): do_IRQ (acpi_ec_read)
00010002 1.113ms (+0.000ms): ack_edge_ioapic_irq (do_IRQ) 00010002 1.113ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 00010000 1.113ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 00010000 1.113ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 00010001 1.113ms (+0.000ms): mark_offset_tsc (timer_interrupt) 00010001 1.123ms (+0.009ms): do_timer (timer_interrupt) 00010001 1.123ms (+0.000ms): update_process_times (do_timer) 00010001 1.123ms (+0.000ms): update_one_process (update_process_times) 00010001 1.123ms (+0.000ms): run_local_timers (update_process_times) 00010001 1.123ms (+0.000ms): raise_softirq (update_process_times) 00010001 1.123ms (+0.000ms): scheduler_tick (update_process_times) 00010001 1.123ms (+0.000ms): sched_clock (scheduler_tick) 00010002 1.123ms (+0.000ms): task_timeslice (scheduler_tick) 00010001 1.123ms (+0.000ms): update_wall_time (do_timer) 00010001 1.123ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 00010002 1.123ms (+0.000ms): generic_note_interrupt (do_IRQ) 00010002 1.123ms (+0.000ms): end_edge_ioapic_irq (do_IRQ) 00000002 1.123ms (+0.000ms): do_softirq (do_IRQ)
00000100 1.123ms (+0.000ms): __do_softirq (do_softirq) 00000001 1.124ms (+0.000ms): sub_preempt_count (acpi_ec_read)

Full trace output available on:
http://users.telenet.be/kvogel/ec-latency.txt

Culprit is the following code in drivers/acpi/ec.c function acpi_ec_wait (edited for mail):

----

#define ACPI_EC_UDELAY          100     /* Poll @ 100us increments */
#define ACPI_EC_UDELAY_COUNT    1000    /* Wait 10ms max. during EC ops */

u32 i = ACPI_EC_UDELAY_COUNT;

do {
    acpi_hw_low_level_read(8, &acpi_ec_status, &ec->status_addr);
    if (acpi_ec_status & ACPI_EC_FLAG_OBF)
            return 0;
    udelay(ACPI_EC_UDELAY);
} while (--i>0);

----

NOTE: isn't the comment on the ACPI_EC_UDELAY_COUNT wrong?! Or is my arithmetic not what it used to be. (1000 loops of 100us = 100ms instead of 10ms, no?)

#375 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar

here's -P2:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2

Changes since -P1:

Quoted 7 lines

- trace interrupted kernel code (via hardirqs, NMIs and pagefaults)

- yet another shot at trying to fix the IO-APIC/USB issues.

- mcount speedups - tracing should be faster

Ingo

I think I stumbled across some bugs/false positives.
Those tests were run with acpi=off, so that this specific issue doesn't interfere. voluntary_preemption is set to 3 throughout.

The first problem was already reported by Lee Revell. Creating a new tab in gnome-terminal gives :

[...]
0.064ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range)
[... plenty of preempt_schedule (copy_page_range) skipped ...]
0.202ms (+0.000ms): preempt_schedule (copy_page_range) 0.202ms (+0.000ms): preempt_schedule (copy_page_range) 0.202ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

This is induced by kernel_preemption=0 and disappears with kernel_preemption=1. It seems to be a side-effect of the current design. Is there a way to avoid this ?

The second one still involves creating a new tab in gnome-terminal, but with kernel_preemption=1 :

preemption latency trace v1.0
-----------------------------
latency: 130 us, entries: 6 (6)
process: gnome-terminal/14381, uid: 1000
nice: 0, policy: 0, rt_priority: 0
=======>
0.000ms (+0.000ms): __vma_link_rb (copy_mm)
0.000ms (+0.000ms): rb_insert_color (copy_mm)
0.000ms (+0.000ms): __rb_rotate_left (rb_insert_color) 0.000ms (+0.000ms): copy_page_range (copy_mm)
0.000ms (+0.000ms): pte_alloc_map (copy_page_range) 0.127ms (+0.126ms): check_preempt_timing (touch_preempt_timing)

When entering check_preempt_timing, preempt_thresh was 0, and preempt_max_latency had been freshly reset to 100. It should have triggered this code :

		if (latency < preempt_max_latency)
			goto out;

but for some reason it didn't (or there is a problem in the tracing code, not showing events that would have increased 'latency').

Thomas

#376 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

Quoted 9 lines · Thomas Charbonnel

When entering check_preempt_timing, preempt_thresh was 0, and preempt_max_latency had been freshly reset to 100. It should have triggered this code :

		if (latency < preempt_max_latency)
			goto out;

but for some reason it didn't (or there is a problem in the tracing code, not showing events that would have increased 'latency').

there is one case where we could 'miss' a new latency: when /proc/latency_trace is accessed. For the duration of /proc/latency_trace access, the updating of the max latency is stopped:

        if (down_trylock(&max_mutex))
                goto out;

this is not really a practical problem and fixing it would be quite complex.

Ingo

#377 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar

here's -P2:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2

Changes since -P1:

Quoted 8 lines

- trace interrupted kernel code (via hardirqs, NMIs and pagefaults)

- yet another shot at trying to fix the IO-APIC/USB issues.

- mcount speedups - tracing should be faster

Ingo
-

The next problem I have relates to irq sharing.
On my laptop I can't avoid it :
10: 1070631 XT-PIC yenta, yenta, uhci_hcd, Intel
82801CA-ICH3, hdsp, eth0
If I set the sound card's interrupt to be non threaded, then I get a rather long non preemptible section :
http://www.undata.org/~thomas/irq_sharing.trace

As a side note, and this has already been reported here several times, the SA_INTERRUPT flag set notably by the sound card drivers handlers is not honored on current kernels if the device is not the first one to be registered. A simple fix would be to add SA_INTERRUPT handlers at the beginning instead of the end of the irq queue in setup_irq.

Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq contribute to the entropy, even those that have a predictable interrupt rate (e.g. sound cards), and/or for which the number of interrupts could outweight the number of interrupts of the original SA_SAMPLE_RANDOM driver.

Thomas

#378 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

At Wed, 18 Aug 2004 14:22:07 +0200,

Quoted 7 lines · Thomas Charbonnel

As a side note, and this has already been reported here several times, the SA_INTERRUPT flag set notably by the sound card drivers handlers is not honored on current kernels if the device is not the first one to be registered. A simple fix would be to add SA_INTERRUPT handlers at the beginning instead of the end of the irq queue in setup_irq.

Quoted 5 lines

Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq contribute to the entropy, even those that have a predictable interrupt rate (e.g. sound cards), and/or for which the number of interrupts could outweight the number of interrupts of the original SA_SAMPLE_RANDOM driver.

Hmm, something like this?

Takashi

arch/i386/kernel/irq.c +13 -6
@@ -219,15 +219,22 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
+	int status;
+	int ret, retval = 0;
+	struct irqaction *ac;
+
+	status = 0;
+	for (ac = action; ac; ac = ac->next)
+		status |= ac->flags;
+	if (!(status & SA_INTERRUPT))
 		local_irq_enable();
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)
#379 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

Quoted 41 lines · Takashi Iwai

At Wed, 18 Aug 2004 14:22:07 +0200,

Quoted 7 lines · Thomas Charbonnel

As a side note, and this has already been reported here several times, the SA_INTERRUPT flag set notably by the sound card drivers handlers is not honored on current kernels if the device is not the first one to be registered. A simple fix would be to add SA_INTERRUPT handlers at the beginning instead of the end of the irq queue in setup_irq.

Quoted 5 lines

Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq contribute to the entropy, even those that have a predictable interrupt rate (e.g. sound cards), and/or for which the number of interrupts could outweight the number of interrupts of the original SA_SAMPLE_RANDOM driver.

Hmm, something like this?

Takashi

arch/i386/kernel/irq.c +9 -4
@@ -219,15 +219,22 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
+	int status;
+	int ret, retval = 0;
+	struct irqaction *ac;
+
+	status = 0;
+	for (ac = action; ac; ac = ac->next)
+		status |= ac->flags;
+	if (!(status & SA_INTERRUPT))
 		local_irq_enable();
 
+	status = 0;

arch/i386/kernel/irq.c inline void synchronize_irq(unsigned int · line 240

Quoted 10 lines
arch/i386/kernel/irq.c +4 -2
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)

I was thinking more of something like this :

irq.c 2004-08-18 17:23:18.011059064 +0200 renamed from irq.c.orig 2004-08-16 14:26:34.000000000 +0200 +10 -5
@@ -955,11 +955,16 @@
                        return -EBUSY;
                }
 
-               /* add new interrupt at end of irq queue */
-               do {
-                       p = &old->next;
-                       old = *p;
-               } while (old);
+               if (new->flags & SA_INTERRUPT)
+                       /* add interrupt at the start of the queue */
+                       new->next = old;
+               else
+                       /* add new interrupt at end of irq queue */
+                       do {
+                               p = &old->next;
+                               old = *p;
+                       } while (old);
+
                shared = 1;
        }
 

Thomas

#380 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

At Wed, 18 Aug 2004 17:26:16 +0200,

Quoted 16 lines · Thomas Charbonnel

I was thinking more of something like this :

irq.c 2004-08-18 17:23:18.011059064 +0200 renamed from irq.c.orig 2004-08-16 14:26:34.000000000 +0200 +2 -5
@@ -955,11 +955,16 @@
                        return -EBUSY;
                }
 
-               /* add new interrupt at end of irq queue */
-               do {
-                       p = &old->next;
-                       old = *p;
-               } while (old);
+               if (new->flags & SA_INTERRUPT)
+                       /* add interrupt at the start of the queue */

irq.c 2004-08-18 17:23:18.011059064 +0200 line 969

Quoted 10 lines
irq.c 2004-08-18 17:23:18.011059064 +0200 renamed from irq.c.orig 2004-08-16 14:26:34.000000000 +0200 +8 -0
+                       new->next = old;
+               else
+                       /* add new interrupt at end of irq queue */
+                       do {
+                               p = &old->next;
+                               old = *p;
+                       } while (old);
+
                shared = 1;
        }

Ok, how about this one? Together with your patch, the irq is turned on for handlers without SA_INTERRUPT.

Takashi

arch/i386/kernel/irq.c +25 -12
@@ -219,15 +219,23 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
-		local_irq_enable();
+	int status;
+	int ret, retval = 0;
+	int irq_off = 1;
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		/* Assume that all SA_INTERRUPT handlers are at the head
+		 * of the irq queue
+		 */
+		if (irq_off && ! (action->flags & SA_INTERRUPT)) {
+			local_irq_enable();
+			irq_off = 0;
+		}
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)
@@ -955,11 +963,16 @@ int setup_irq(unsigned int irq, struct i
 			return -EBUSY;
 		}
 
-		/* add new interrupt at end of irq queue */
-		do {
-			p = &old->next;
-			old = *p;
-		} while (old);
+		if (new->flags & SA_INTERRUPT)
+			/* add interrupt at the start of the queue */
+			new->next = old;
+		else
+			/* add new interrupt at end of irq queue */
+			do {
+				p = &old->next;
+				old = *p;
+			} while (old);
+
 		shared = 1;
 	}
 
#381 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

Takashi Iwai wrote :

[...]

Quoted 51 lines

Ok, how about this one? Together with your patch, the irq is turned on for handlers without SA_INTERRUPT.

Takashi

arch/i386/kernel/irq.c +18 -12
@@ -219,15 +219,23 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
-		local_irq_enable();
+	int status;
+	int ret, retval = 0;
+	int irq_off = 1;
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		/* Assume that all SA_INTERRUPT handlers are at the head
+		 * of the irq queue
+		 */
+		if (irq_off && ! (action->flags & SA_INTERRUPT)) {
+			local_irq_enable();
+			irq_off = 0;
+		}
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)
@@ -955,11 +963,16 @@ int setup_irq(unsigned int irq, struct i
 			return -EBUSY;
 		}
 
-		/* add new interrupt at end of irq queue */
-		do {
-			p = &old->next;
-			old = *p;
-		} while (old);
+		if (new->flags & SA_INTERRUPT)
+			/* add interrupt at the start of the queue */
+			new->next = old;

arch/i386/kernel/irq.c int setup_irq(unsigned int irq, struct i · line 978

Quoted 10 lines
arch/i386/kernel/irq.c +7 -0
+		else
+			/* add new interrupt at end of irq queue */
+			do {
+				p = &old->next;
+				old = *p;
+			} while (old);
+
 		shared = 1;
 	}
 

You're right, of course, my mistake. It looks good now.

Thomas

#382 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P2

In reply to Thomas Charbonnel
Quoted 7 lines · Thomas Charbonnel

The next problem I have relates to irq sharing.
On my laptop I can't avoid it :
10: 1070631 XT-PIC yenta, yenta, uhci_hcd, Intel
82801CA-ICH3, hdsp, eth0
If I set the sound card's interrupt to be non threaded, then I get a rather long non preemptible section :
http://www.undata.org/~thomas/irq_sharing.trace

i'm not sure the IRQ sharing problem can be solved.

we could execute certain handlers immediately, and defer others to an IRQ thread. But when we defer an IRQ we must keep the IRQ masked - which prevents further interrupts (possibly from a high-prio non-threaded handler) to be executed. So we'd see similar (or in fact worse, due to the redirection cost) latencies than with the current 'all or nothing' approach.

now in theory we only have to keep the IRQ line masked for level triggered interrupts (most APIC interrupts are level-triggered). Edge-triggered interrupts (such as the XT-PIC ones you have) could be acked immediately. I'll try to do something later, but right now there are still some IRQ problems (USB issues, PS2 mouse/keyboard issues) so i'd not like to complicate the design just yet.

Ingo

#383 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

In reply to Ingo Molnar
Quoted 16 lines · Ingo Molnar
Quoted 11 lines · Thomas Charbonnel

Tested P1. The biggest offender on my system is still reiserfs' search_by_key. The trace is made of a very long loop of :
0.016ms (+0.000ms): reiserfs_in_journal (scan_bitmap_block) 0.016ms (+0.000ms): find_next_zero_bit (scan_bitmap_block) (...)
0.977ms (+0.000ms): reiserfs_in_journal (scan_bitmap_block) 0.977ms (+0.000ms): find_next_zero_bit (scan_bitmap_block) with interrupts showing up in the trace from time to time. Do you have plans to fix this, or should I switch to ext3 ?

Quoted 5 lines

i took a quick look and the reiserfs locking rules in that place do not seem to be easily fixable - this is the tree-lookup code which i suspect cannot be preempted. The reiser journalling code also makes use of the big kernel lock. I'd suggest reporting this to the reiserfs folks

the fix to reiserfs doing single threaded balancing/searching is called reiser4;-) It was not a trivial fix. It is however released.

Quoted 40 lines

and
(if it's not too much effort to migrate) use ext3 meanwhile.

Quoted 11 lines

but the weird thing is that the latency sum goes way above those 100us, the culprit being do_IRQ, regularly chewing up to 1ms !

0.011ms (+0.000ms): enable_8259A_irq (startup_8259A_irq) 0.954ms (+0.943ms): do_IRQ (common_interrupt)

weird. This has the looks of a preempt-timing bug (we get a timer IRQ every 1 msec) - but there should be no preempt-timing when we are in the idle task (swapper).

Quoted 3 lines

http://www.undata.org/~thomas/swapper.trace

i'll upload -P2 in a couple of minutes, it will trace the code that do_IRQ() interrupted too - that would be quite useful in your case.

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#384 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

Quoted 10 lines · Hans Reiser
Quoted 6 lines · Ingo Molnar

i took a quick look and the reiserfs locking rules in that place do not seem to be easily fixable - this is the tree-lookup code which i suspect cannot be preempted. The reiser journalling code also makes use of the big kernel lock. I'd suggest reporting this to the reiserfs folks

the fix to reiserfs doing single threaded balancing/searching is called reiser4;-) It was not a trivial fix. It is however released.

When will reiser4 will be in the stock kernel?

Lee

#385 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

Quoted 9 lines · Lee Revell

When will reiser4 will be in the stock kernel?

Lee

I hope for 3-6 weeks from now, but it is not my decision.....

We are going into -mm this week.

#386 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

In reply to Ingo Molnar
Quoted 11 lines · On Mon, 16 Aug 2004 06:05:15 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 2 lines · Lee Revell

Anyway, the change to sched.c fixes the mlockall bug, it works perfectly now. Thanks!

great! This fix also means that we've got one more lock-break in the ext3 journalling code and one more lock-break in dcache.c. I've released-P1 with the fix included:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P1

Hi,

i don't know if this was mentioned before, but i sometimes see traces like this where half the entries are "preempt_schedule (copy_page_range)". I just wanted to ask if this is normal and expected behaviour.

preemption latency trace v1.0
-----------------------------
latency: 162 us, entries: 361 (361)
process: qjackctl/828, uid: 1000
nice: 0, policy: 0, rt_priority: 0
=======>
0.000ms (+0.000ms): __vma_link_rb (copy_mm)
0.000ms (+0.000ms): rb_insert_color (copy_mm)
0.000ms (+0.000ms): __rb_rotate_left (rb_insert_color) 0.000ms (+0.000ms): copy_page_range (copy_mm)
0.000ms (+0.000ms): pte_alloc_map (copy_page_range) 0.053ms (+0.053ms): do_IRQ (common_interrupt)
0.054ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 0.056ms (+0.001ms): generic_redirect_hardirq (do_IRQ) 0.056ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.056ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 0.057ms (+0.000ms): mark_offset_tsc (timer_interrupt) 0.064ms (+0.006ms): do_timer (timer_interrupt)
0.064ms (+0.000ms): update_process_times (do_timer) 0.064ms (+0.000ms): update_one_process (update_process_times) 0.064ms (+0.000ms): run_local_timers (update_process_times) 0.064ms (+0.000ms): raise_softirq (update_process_times) 0.065ms (+0.000ms): scheduler_tick (update_process_times) 0.065ms (+0.000ms): sched_clock (scheduler_tick) 0.066ms (+0.001ms): task_timeslice (scheduler_tick) 0.066ms (+0.000ms): update_wall_time (do_timer) 0.066ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 0.068ms (+0.001ms): generic_note_interrupt (do_IRQ) 0.068ms (+0.000ms): end_8259A_irq (do_IRQ)
0.069ms (+0.000ms): enable_8259A_irq (do_IRQ)
0.070ms (+0.000ms): do_softirq (do_IRQ)
0.070ms (+0.000ms): __do_softirq (do_softirq)
0.070ms (+0.000ms): wake_up_process (do_softirq) 0.070ms (+0.000ms): try_to_wake_up (wake_up_process) 0.070ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.071ms (+0.000ms): activate_task (try_to_wake_up) 0.071ms (+0.000ms): sched_clock (activate_task) 0.071ms (+0.000ms): recalc_task_prio (activate_task) 0.071ms (+0.000ms): effective_prio (recalc_task_prio) 0.071ms (+0.000ms): enqueue_task (activate_task) 0.072ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.113ms (+0.000ms): preempt_schedule (copy_page_range) 0.113ms (+0.000ms): preempt_schedule (copy_page_range) 0.113ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.115ms (+0.000ms): preempt_schedule (copy_page_range) 0.115ms (+0.000ms): preempt_schedule (copy_page_range) 0.115ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.121ms (+0.000ms): preempt_schedule (copy_page_range) 0.121ms (+0.000ms): preempt_schedule (copy_page_range) 0.121ms (+0.000ms): preempt_schedule (copy_page_range) 0.122ms (+0.000ms): preempt_schedule (copy_page_range) 0.122ms (+0.000ms): preempt_schedule (copy_page_range) 0.122ms (+0.000ms): preempt_schedule (copy_page_range) 0.123ms (+0.000ms): preempt_schedule (copy_page_range) 0.123ms (+0.000ms): preempt_schedule (copy_page_range) 0.123ms (+0.000ms): preempt_schedule (copy_page_range) 0.123ms (+0.000ms): preempt_schedule (copy_page_range) 0.124ms (+0.000ms): preempt_schedule (copy_page_range) 0.124ms (+0.000ms): preempt_schedule (copy_page_range) 0.124ms (+0.000ms): preempt_schedule (copy_page_range) 0.124ms (+0.000ms): preempt_schedule (copy_page_range) 0.125ms (+0.000ms): preempt_schedule (copy_page_range) 0.125ms (+0.000ms): preempt_schedule (copy_page_range) 0.125ms (+0.000ms): preempt_schedule (copy_page_range) 0.125ms (+0.000ms): preempt_schedule (copy_page_range) 0.126ms (+0.000ms): preempt_schedule (copy_page_range) 0.126ms (+0.000ms): preempt_schedule (copy_page_range) 0.126ms (+0.000ms): preempt_schedule (copy_page_range) 0.126ms (+0.000ms): preempt_schedule (copy_page_range) 0.126ms (+0.000ms): preempt_schedule (copy_page_range) 0.127ms (+0.000ms): preempt_schedule (copy_page_range) 0.127ms (+0.000ms): preempt_schedule (copy_page_range) 0.127ms (+0.000ms): preempt_schedule (copy_page_range) 0.127ms (+0.000ms): preempt_schedule (copy_page_range) 0.128ms (+0.000ms): preempt_schedule (copy_page_range) 0.128ms (+0.000ms): preempt_schedule (copy_page_range) 0.128ms (+0.000ms): preempt_schedule (copy_page_range) 0.128ms (+0.000ms): preempt_schedule (copy_page_range) 0.129ms (+0.000ms): preempt_schedule (copy_page_range) 0.129ms (+0.000ms): preempt_schedule (copy_page_range) 0.130ms (+0.000ms): preempt_schedule (copy_page_range) 0.130ms (+0.000ms): preempt_schedule (copy_page_range) 0.130ms (+0.000ms): preempt_schedule (copy_page_range) 0.130ms (+0.000ms): preempt_schedule (copy_page_range) 0.130ms (+0.000ms): preempt_schedule (copy_page_range) 0.131ms (+0.000ms): preempt_schedule (copy_page_range) 0.131ms (+0.000ms): preempt_schedule (copy_page_range) 0.131ms (+0.000ms): preempt_schedule (copy_page_range) 0.132ms (+0.000ms): preempt_schedule (copy_page_range) 0.132ms (+0.000ms): preempt_schedule (copy_page_range) 0.132ms (+0.000ms): preempt_schedule (copy_page_range) 0.133ms (+0.000ms): preempt_schedule (copy_page_range) 0.133ms (+0.000ms): preempt_schedule (copy_page_range) 0.133ms (+0.000ms): preempt_schedule (copy_page_range) 0.133ms (+0.000ms): preempt_schedule (copy_page_range) 0.134ms (+0.000ms): preempt_schedule (copy_page_range) 0.134ms (+0.000ms): preempt_schedule (copy_page_range) 0.134ms (+0.000ms): preempt_schedule (copy_page_range) 0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.138ms (+0.000ms): preempt_schedule (copy_page_range) 0.138ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.140ms (+0.000ms): preempt_schedule (copy_page_range) 0.140ms (+0.000ms): preempt_schedule (copy_page_range) 0.140ms (+0.000ms): preempt_schedule (copy_page_range) 0.141ms (+0.000ms): preempt_schedule (copy_page_range) 0.141ms (+0.000ms): preempt_schedule (copy_page_range) 0.141ms (+0.000ms): preempt_schedule (copy_page_range) 0.141ms (+0.000ms): preempt_schedule (copy_page_range) 0.142ms (+0.000ms): preempt_schedule (copy_page_range) 0.142ms (+0.000ms): preempt_schedule (copy_page_range) 0.143ms (+0.000ms): preempt_schedule (copy_page_range) 0.143ms (+0.000ms): preempt_schedule (copy_page_range) 0.143ms (+0.000ms): preempt_schedule (copy_page_range) 0.144ms (+0.000ms): preempt_schedule (copy_page_range) 0.144ms (+0.000ms): preempt_schedule (copy_page_range) 0.144ms (+0.000ms): preempt_schedule (copy_page_range) 0.144ms (+0.000ms): preempt_schedule (copy_page_range) 0.144ms (+0.000ms): preempt_schedule (copy_page_range) 0.145ms (+0.000ms): preempt_schedule (copy_page_range) 0.145ms (+0.000ms): preempt_schedule (copy_page_range) 0.145ms (+0.000ms): preempt_schedule (copy_page_range) 0.146ms (+0.000ms): preempt_schedule (copy_page_range) 0.146ms (+0.000ms): preempt_schedule (copy_page_range) 0.147ms (+0.000ms): preempt_schedule (copy_page_range) 0.147ms (+0.000ms): preempt_schedule (copy_page_range) 0.147ms (+0.000ms): preempt_schedule (copy_page_range) 0.147ms (+0.000ms): preempt_schedule (copy_page_range) 0.148ms (+0.000ms): preempt_schedule (copy_page_range) 0.148ms (+0.000ms): preempt_schedule (copy_page_range) 0.148ms (+0.000ms): preempt_schedule (copy_page_range) 0.148ms (+0.000ms): preempt_schedule (copy_page_range) 0.149ms (+0.000ms): preempt_schedule (copy_page_range) 0.149ms (+0.000ms): preempt_schedule (copy_page_range) 0.149ms (+0.000ms): preempt_schedule (copy_page_range) 0.149ms (+0.000ms): preempt_schedule (copy_page_range) 0.149ms (+0.000ms): preempt_schedule (copy_page_range) 0.150ms (+0.000ms): preempt_schedule (copy_page_range) 0.150ms (+0.000ms): preempt_schedule (copy_page_range) 0.150ms (+0.000ms): preempt_schedule (copy_page_range) 0.151ms (+0.000ms): preempt_schedule (copy_page_range) 0.151ms (+0.000ms): preempt_schedule (copy_page_range) 0.151ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.158ms (+0.000ms): preempt_schedule (copy_page_range) 0.158ms (+0.000ms): preempt_schedule (copy_page_range) 0.159ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

preemption latency trace v1.0
-----------------------------
latency: 258 us, entries: 722 (722)
process: sylpheed-claws/796, uid: 1000
nice: 0, policy: 0, rt_priority: 0
=======>
0.000ms (+0.000ms): do_IRQ (common_interrupt)
0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
0.002ms (+0.001ms): generic_redirect_hardirq (do_IRQ) 0.002ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.002ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 0.002ms (+0.000ms): mark_offset_tsc (timer_interrupt) 0.009ms (+0.007ms): do_timer (timer_interrupt)
0.010ms (+0.000ms): update_process_times (do_timer) 0.010ms (+0.000ms): update_one_process (update_process_times) 0.010ms (+0.000ms): run_local_timers (update_process_times) 0.011ms (+0.000ms): raise_softirq (update_process_times) 0.011ms (+0.000ms): scheduler_tick (update_process_times) 0.011ms (+0.000ms): sched_clock (scheduler_tick)
0.012ms (+0.000ms): task_timeslice (scheduler_tick) 0.013ms (+0.000ms): update_wall_time (do_timer)
0.013ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 0.014ms (+0.000ms): generic_note_interrupt (do_IRQ) 0.014ms (+0.000ms): end_8259A_irq (do_IRQ)
0.015ms (+0.000ms): enable_8259A_irq (do_IRQ)
0.015ms (+0.000ms): do_softirq (do_IRQ)
0.015ms (+0.000ms): __do_softirq (do_softirq)
0.016ms (+0.000ms): wake_up_process (do_softirq)
0.016ms (+0.000ms): try_to_wake_up (wake_up_process) 0.016ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.016ms (+0.000ms): activate_task (try_to_wake_up) 0.016ms (+0.000ms): sched_clock (activate_task)
0.016ms (+0.000ms): recalc_task_prio (activate_task) 0.016ms (+0.000ms): effective_prio (recalc_task_prio) 0.017ms (+0.000ms): enqueue_task (activate_task)
0.017ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.017ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.023ms (+0.000ms): preempt_schedule (copy_page_range) 0.023ms (+0.000ms): preempt_schedule (copy_page_range) 0.023ms (+0.000ms): preempt_schedule (copy_page_range) 0.023ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.027ms (+0.000ms): preempt_schedule (copy_page_range) 0.027ms (+0.000ms): preempt_schedule (copy_page_range) 0.027ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.043ms (+0.000ms): preempt_schedule (copy_page_range) 0.043ms (+0.000ms): preempt_schedule (copy_page_range) 0.043ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.048ms (+0.000ms): preempt_schedule (copy_page_range) 0.048ms (+0.000ms): preempt_schedule (copy_page_range) 0.048ms (+0.000ms): preempt_schedule (copy_page_range) 0.048ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.053ms (+0.000ms): preempt_schedule (copy_page_range) 0.053ms (+0.000ms): preempt_schedule (copy_page_range) 0.053ms (+0.000ms): preempt_schedule (copy_page_range) 0.053ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.066ms (+0.000ms): preempt_schedule (copy_page_range) 0.066ms (+0.000ms): preempt_schedule (copy_page_range) 0.066ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.110ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.111ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.112ms (+0.000ms): preempt_schedule (copy_page_range) 0.113ms (+0.000ms): preempt_schedule (copy_page_range) 0.113ms (+0.000ms): preempt_schedule (copy_page_range) 0.113ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.114ms (+0.000ms): preempt_schedule (copy_page_range) 0.115ms (+0.000ms): preempt_schedule (copy_page_range) 0.115ms (+0.000ms): preempt_schedule (copy_page_range) 0.115ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.116ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.117ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.118ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.119ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.120ms (+0.000ms): preempt_schedule (copy_page_range) 0.121ms (+0.000ms): preempt_schedule (copy_page_range) 0.121ms (+0.000ms): preempt_schedule (copy_page_range) 0.121ms (+0.000ms): preempt_schedule (copy_page_range) 0.122ms (+0.001ms): do_IRQ (common_interrupt)
0.122ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
0.124ms (+0.001ms): preempt_schedule (do_IRQ)
0.124ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 0.124ms (+0.000ms): preempt_schedule (do_IRQ)
0.124ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.125ms (+0.000ms): snd_cs46xx_interrupt (generic_handle_IRQ_event) 0.127ms (+0.001ms): snd_pcm_period_elapsed (snd_cs46xx_interrupt) 0.128ms (+0.001ms): snd_cs46xx_playback_direct_pointer
(snd_pcm_period_elapsed)
0.131ms (+0.002ms): __wake_up (snd_pcm_period_elapsed) 0.131ms (+0.000ms): __wake_up_common (__wake_up)
0.131ms (+0.000ms): default_wake_function (__wake_up_common) 0.131ms (+0.000ms): try_to_wake_up (__wake_up_common) 0.131ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.131ms (+0.000ms): activate_task (try_to_wake_up) 0.131ms (+0.000ms): sched_clock (activate_task)
0.131ms (+0.000ms): recalc_task_prio (activate_task) 0.132ms (+0.000ms): effective_prio (recalc_task_prio) 0.132ms (+0.000ms): enqueue_task (activate_task)
0.132ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.132ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.132ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.133ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.133ms (+0.000ms): kill_fasync (snd_pcm_period_elapsed) 0.133ms (+0.000ms): generic_note_interrupt (do_IRQ) 0.134ms (+0.000ms): end_8259A_irq (do_IRQ)
0.134ms (+0.000ms): enable_8259A_irq (do_IRQ)
0.134ms (+0.000ms): preempt_schedule (do_IRQ)
0.134ms (+0.000ms): preempt_schedule (do_IRQ)
0.135ms (+0.000ms): do_softirq (do_IRQ)
0.135ms (+0.000ms): __do_softirq (do_softirq)
0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.135ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.136ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.137ms (+0.000ms): preempt_schedule (copy_page_range) 0.138ms (+0.000ms): preempt_schedule (copy_page_range) 0.138ms (+0.000ms): preempt_schedule (copy_page_range) 0.138ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.139ms (+0.000ms): preempt_schedule (copy_page_range) 0.140ms (+0.000ms): preempt_schedule (copy_page_range) 0.140ms (+0.000ms): preempt_schedule (copy_page_range) 0.140ms (+0.000ms): preempt_schedule (copy_page_range) 0.141ms (+0.000ms): do_IRQ (common_interrupt)
0.141ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
0.142ms (+0.001ms): preempt_schedule (do_IRQ)
0.143ms (+0.000ms): generic_redirect_hardirq (do_IRQ) 0.143ms (+0.000ms): preempt_schedule (do_IRQ)
0.143ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.143ms (+0.000ms): snd_cs46xx_interrupt (generic_handle_IRQ_event) 0.144ms (+0.001ms): snd_pcm_period_elapsed (snd_cs46xx_interrupt) 0.145ms (+0.001ms): snd_cs46xx_capture_indirect_pointer
(snd_pcm_period_elapsed)
0.146ms (+0.000ms): snd_cs46xx_capture_transfer
(snd_cs46xx_capture_indirect_pointer)
0.148ms (+0.002ms): __wake_up (snd_pcm_period_elapsed) 0.148ms (+0.000ms): __wake_up_common (__wake_up)
0.149ms (+0.000ms): default_wake_function (__wake_up_common) 0.149ms (+0.000ms): try_to_wake_up (__wake_up_common) 0.149ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.149ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.149ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.149ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.149ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.150ms (+0.000ms): kill_fasync (snd_pcm_period_elapsed) 0.150ms (+0.000ms): generic_note_interrupt (do_IRQ) 0.150ms (+0.000ms): end_8259A_irq (do_IRQ)
0.150ms (+0.000ms): enable_8259A_irq (do_IRQ)
0.151ms (+0.000ms): preempt_schedule (do_IRQ)
0.151ms (+0.000ms): preempt_schedule (do_IRQ)
0.151ms (+0.000ms): do_softirq (do_IRQ)
0.151ms (+0.000ms): __do_softirq (do_softirq)
0.151ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.152ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.153ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.154ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.155ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.156ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.157ms (+0.000ms): preempt_schedule (copy_page_range) 0.158ms (+0.000ms): preempt_schedule (copy_page_range) 0.158ms (+0.000ms): preempt_schedule (copy_page_range) 0.158ms (+0.000ms): preempt_schedule (copy_page_range) 0.158ms (+0.000ms): preempt_schedule (copy_page_range) 0.159ms (+0.000ms): preempt_schedule (copy_page_range) 0.159ms (+0.000ms): preempt_schedule (copy_page_range) 0.159ms (+0.000ms): preempt_schedule (copy_page_range) 0.159ms (+0.000ms): preempt_schedule (copy_page_range) 0.159ms (+0.000ms): preempt_schedule (copy_page_range) 0.160ms (+0.000ms): preempt_schedule (copy_page_range) 0.160ms (+0.000ms): preempt_schedule (copy_page_range) 0.160ms (+0.000ms): preempt_schedule (copy_page_range) 0.160ms (+0.000ms): preempt_schedule (copy_page_range) 0.160ms (+0.000ms): preempt_schedule (copy_page_range) 0.161ms (+0.000ms): preempt_schedule (copy_page_range) 0.161ms (+0.000ms): preempt_schedule (copy_page_range) 0.161ms (+0.000ms): preempt_schedule (copy_page_range) 0.161ms (+0.000ms): preempt_schedule (copy_page_range) 0.162ms (+0.000ms): preempt_schedule (copy_page_range) 0.162ms (+0.000ms): preempt_schedule (copy_page_range) 0.162ms (+0.000ms): preempt_schedule (copy_page_range) 0.162ms (+0.000ms): preempt_schedule (copy_page_range) 0.163ms (+0.000ms): preempt_schedule (copy_page_range) 0.163ms (+0.000ms): preempt_schedule (copy_page_range) 0.163ms (+0.000ms): preempt_schedule (copy_page_range) 0.163ms (+0.000ms): preempt_schedule (copy_page_range) 0.163ms (+0.000ms): preempt_schedule (copy_page_range) 0.164ms (+0.000ms): preempt_schedule (copy_page_range) 0.164ms (+0.000ms): preempt_schedule (copy_page_range) 0.164ms (+0.000ms): preempt_schedule (copy_page_range) 0.164ms (+0.000ms): preempt_schedule (copy_page_range) 0.165ms (+0.000ms): preempt_schedule (copy_page_range) 0.165ms (+0.000ms): preempt_schedule (copy_page_range) 0.165ms (+0.000ms): preempt_schedule (copy_page_range) 0.165ms (+0.000ms): preempt_schedule (copy_page_range) 0.166ms (+0.000ms): preempt_schedule (copy_page_range) 0.166ms (+0.000ms): preempt_schedule (copy_page_range) 0.166ms (+0.000ms): preempt_schedule (copy_page_range) 0.166ms (+0.000ms): preempt_schedule (copy_page_range) 0.167ms (+0.000ms): preempt_schedule (copy_page_range) 0.167ms (+0.000ms): preempt_schedule (copy_page_range) 0.167ms (+0.000ms): preempt_schedule (copy_page_range) 0.167ms (+0.000ms): preempt_schedule (copy_page_range) 0.167ms (+0.000ms): preempt_schedule (copy_page_range) 0.168ms (+0.000ms): preempt_schedule (copy_page_range) 0.168ms (+0.000ms): preempt_schedule (copy_page_range) 0.168ms (+0.000ms): preempt_schedule (copy_page_range) 0.168ms (+0.000ms): preempt_schedule (copy_page_range) 0.169ms (+0.000ms): preempt_schedule (copy_page_range) 0.169ms (+0.000ms): preempt_schedule (copy_page_range) 0.169ms (+0.000ms): preempt_schedule (copy_page_range) 0.169ms (+0.000ms): preempt_schedule (copy_page_range) 0.170ms (+0.000ms): preempt_schedule (copy_page_range) 0.170ms (+0.000ms): preempt_schedule (copy_page_range) 0.170ms (+0.000ms): preempt_schedule (copy_page_range) 0.170ms (+0.000ms): preempt_schedule (copy_page_range) 0.170ms (+0.000ms): preempt_schedule (copy_page_range) 0.170ms (+0.000ms): preempt_schedule (copy_page_range) 0.171ms (+0.000ms): preempt_schedule (copy_page_range) 0.171ms (+0.000ms): preempt_schedule (copy_page_range) 0.171ms (+0.000ms): preempt_schedule (copy_page_range) 0.171ms (+0.000ms): preempt_schedule (copy_page_range) 0.172ms (+0.000ms): preempt_schedule (copy_page_range) 0.172ms (+0.000ms): preempt_schedule (copy_page_range) 0.172ms (+0.000ms): preempt_schedule (copy_page_range) 0.172ms (+0.000ms): preempt_schedule (copy_page_range) 0.173ms (+0.000ms): preempt_schedule (copy_page_range) 0.173ms (+0.000ms): preempt_schedule (copy_page_range) 0.173ms (+0.000ms): preempt_schedule (copy_page_range) 0.173ms (+0.000ms): preempt_schedule (copy_page_range) 0.174ms (+0.000ms): preempt_schedule (copy_page_range) 0.174ms (+0.000ms): preempt_schedule (copy_page_range) 0.174ms (+0.000ms): preempt_schedule (copy_page_range) 0.174ms (+0.000ms): preempt_schedule (copy_page_range) 0.175ms (+0.000ms): preempt_schedule (copy_page_range) 0.175ms (+0.000ms): preempt_schedule (copy_page_range) 0.175ms (+0.000ms): preempt_schedule (copy_page_range) 0.175ms (+0.000ms): preempt_schedule (copy_page_range) 0.175ms (+0.000ms): preempt_schedule (copy_page_range) 0.176ms (+0.000ms): preempt_schedule (copy_page_range) 0.176ms (+0.000ms): preempt_schedule (copy_page_range) 0.176ms (+0.000ms): preempt_schedule (copy_page_range) 0.176ms (+0.000ms): preempt_schedule (copy_page_range) 0.177ms (+0.000ms): preempt_schedule (copy_page_range) 0.177ms (+0.000ms): preempt_schedule (copy_page_range) 0.177ms (+0.000ms): preempt_schedule (copy_page_range) 0.177ms (+0.000ms): preempt_schedule (copy_page_range) 0.177ms (+0.000ms): preempt_schedule (copy_page_range) 0.178ms (+0.000ms): preempt_schedule (copy_page_range) 0.178ms (+0.000ms): preempt_schedule (copy_page_range) 0.178ms (+0.000ms): preempt_schedule (copy_page_range) 0.179ms (+0.000ms): preempt_schedule (copy_page_range) 0.179ms (+0.000ms): preempt_schedule (copy_page_range) 0.179ms (+0.000ms): preempt_schedule (copy_page_range) 0.179ms (+0.000ms): preempt_schedule (copy_page_range) 0.180ms (+0.000ms): preempt_schedule (copy_page_range) 0.180ms (+0.000ms): preempt_schedule (copy_page_range) 0.180ms (+0.000ms): preempt_schedule (copy_page_range) 0.180ms (+0.000ms): preempt_schedule (copy_page_range) 0.181ms (+0.000ms): preempt_schedule (copy_page_range) 0.181ms (+0.000ms): preempt_schedule (copy_page_range) 0.181ms (+0.000ms): preempt_schedule (copy_page_range) 0.181ms (+0.000ms): preempt_schedule (copy_page_range) 0.181ms (+0.000ms): preempt_schedule (copy_page_range) 0.182ms (+0.000ms): preempt_schedule (copy_page_range) 0.182ms (+0.000ms): preempt_schedule (copy_page_range) 0.182ms (+0.000ms): preempt_schedule (copy_page_range) 0.182ms (+0.000ms): preempt_schedule (copy_page_range) 0.182ms (+0.000ms): preempt_schedule (copy_page_range) 0.182ms (+0.000ms): preempt_schedule (copy_page_range) 0.183ms (+0.000ms): preempt_schedule (copy_page_range) 0.183ms (+0.000ms): preempt_schedule (copy_page_range) 0.183ms (+0.000ms): preempt_schedule (copy_page_range) 0.184ms (+0.000ms): preempt_schedule (copy_page_range) 0.184ms (+0.000ms): preempt_schedule (copy_page_range) 0.184ms (+0.000ms): preempt_schedule (copy_page_range) 0.184ms (+0.000ms): preempt_schedule (copy_page_range) 0.185ms (+0.000ms): preempt_schedule (copy_page_range) 0.185ms (+0.000ms): preempt_schedule (copy_page_range) 0.185ms (+0.000ms): preempt_schedule (copy_page_range) 0.185ms (+0.000ms): preempt_schedule (copy_page_range) 0.186ms (+0.000ms): preempt_schedule (copy_page_range) 0.186ms (+0.000ms): preempt_schedule (copy_page_range) 0.186ms (+0.000ms): preempt_schedule (copy_page_range) 0.186ms (+0.000ms): preempt_schedule (copy_page_range) 0.186ms (+0.000ms): preempt_schedule (copy_page_range) 0.187ms (+0.000ms): preempt_schedule (copy_page_range) 0.187ms (+0.000ms): preempt_schedule (copy_page_range) 0.187ms (+0.000ms): preempt_schedule (copy_page_range) 0.187ms (+0.000ms): preempt_schedule (copy_page_range) 0.188ms (+0.000ms): preempt_schedule (copy_page_range) 0.188ms (+0.000ms): preempt_schedule (copy_page_range) 0.188ms (+0.000ms): preempt_schedule (copy_page_range) 0.188ms (+0.000ms): preempt_schedule (copy_page_range) 0.188ms (+0.000ms): preempt_schedule (copy_page_range) 0.189ms (+0.000ms): preempt_schedule (copy_page_range) 0.189ms (+0.000ms): preempt_schedule (copy_page_range) 0.189ms (+0.000ms): preempt_schedule (copy_page_range) 0.189ms (+0.000ms): preempt_schedule (copy_page_range) 0.189ms (+0.000ms): preempt_schedule (copy_page_range) 0.190ms (+0.000ms): preempt_schedule (copy_page_range) 0.190ms (+0.000ms): preempt_schedule (copy_page_range) 0.190ms (+0.000ms): preempt_schedule (copy_page_range) 0.190ms (+0.000ms): preempt_schedule (copy_page_range) 0.191ms (+0.000ms): preempt_schedule (copy_page_range) 0.191ms (+0.000ms): preempt_schedule (copy_page_range) 0.191ms (+0.000ms): preempt_schedule (copy_page_range) 0.191ms (+0.000ms): preempt_schedule (copy_page_range) 0.191ms (+0.000ms): preempt_schedule (copy_page_range) 0.192ms (+0.000ms): preempt_schedule (copy_page_range) 0.192ms (+0.000ms): preempt_schedule (copy_page_range) 0.192ms (+0.000ms): preempt_schedule (copy_page_range) 0.192ms (+0.000ms): preempt_schedule (copy_page_range) 0.193ms (+0.000ms): preempt_schedule (copy_page_range) 0.193ms (+0.000ms): preempt_schedule (copy_page_range) 0.193ms (+0.000ms): preempt_schedule (copy_page_range) 0.193ms (+0.000ms): preempt_schedule (copy_page_range) 0.193ms (+0.000ms): preempt_schedule (copy_page_range) 0.194ms (+0.000ms): preempt_schedule (copy_page_range) 0.194ms (+0.000ms): preempt_schedule (copy_page_range) 0.194ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

preemption latency trace v1.0
-----------------------------
latency: 212 us, entries: 402 (402)
process: sylpheed-claws/796, uid: 1000
nice: 0, policy: 0, rt_priority: 0
=======>
0.000ms (+0.000ms): do_IRQ (common_interrupt)
0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
0.001ms (+0.001ms): generic_redirect_hardirq (do_IRQ) 0.001ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.001ms (+0.000ms): snd_cs46xx_interrupt (generic_handle_IRQ_event) 0.003ms (+0.001ms): snd_pcm_period_elapsed (snd_cs46xx_interrupt) 0.004ms (+0.000ms): snd_cs46xx_capture_indirect_pointer
(snd_pcm_period_elapsed)
0.005ms (+0.001ms): snd_cs46xx_capture_transfer
(snd_cs46xx_capture_indirect_pointer)
0.007ms (+0.001ms): __wake_up (snd_pcm_period_elapsed) 0.007ms (+0.000ms): __wake_up_common (__wake_up)
0.007ms (+0.000ms): default_wake_function (__wake_up_common) 0.007ms (+0.000ms): try_to_wake_up (__wake_up_common) 0.007ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.007ms (+0.000ms): activate_task (try_to_wake_up) 0.007ms (+0.000ms): sched_clock (activate_task)
0.007ms (+0.000ms): recalc_task_prio (activate_task) 0.007ms (+0.000ms): effective_prio (recalc_task_prio) 0.008ms (+0.000ms): enqueue_task (activate_task)
0.008ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.008ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.008ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.008ms (+0.000ms): preempt_schedule (snd_pcm_period_elapsed) 0.008ms (+0.000ms): kill_fasync (snd_pcm_period_elapsed) 0.009ms (+0.000ms): generic_note_interrupt (do_IRQ) 0.009ms (+0.000ms): end_8259A_irq (do_IRQ)
0.009ms (+0.000ms): enable_8259A_irq (do_IRQ)
0.010ms (+0.000ms): preempt_schedule (do_IRQ)
0.010ms (+0.000ms): preempt_schedule (do_IRQ)
0.010ms (+0.000ms): preempt_schedule (copy_page_range) 0.011ms (+0.000ms): preempt_schedule (copy_page_range) 0.011ms (+0.000ms): preempt_schedule (copy_page_range) 0.011ms (+0.000ms): preempt_schedule (copy_page_range) 0.011ms (+0.000ms): preempt_schedule (copy_page_range) 0.012ms (+0.000ms): preempt_schedule (copy_page_range) 0.012ms (+0.000ms): preempt_schedule (copy_page_range) 0.012ms (+0.000ms): preempt_schedule (copy_page_range) 0.012ms (+0.000ms): preempt_schedule (copy_page_range) 0.012ms (+0.000ms): preempt_schedule (copy_page_range) 0.013ms (+0.000ms): preempt_schedule (copy_page_range) 0.013ms (+0.000ms): preempt_schedule (copy_page_range) 0.013ms (+0.000ms): preempt_schedule (copy_page_range) 0.013ms (+0.000ms): preempt_schedule (copy_page_range) 0.014ms (+0.000ms): preempt_schedule (copy_page_range) 0.014ms (+0.000ms): preempt_schedule (copy_page_range) 0.014ms (+0.000ms): preempt_schedule (copy_page_range) 0.014ms (+0.000ms): preempt_schedule (copy_page_range) 0.015ms (+0.000ms): preempt_schedule (copy_page_range) 0.015ms (+0.000ms): preempt_schedule (copy_page_range) 0.015ms (+0.000ms): preempt_schedule (copy_page_range) 0.016ms (+0.000ms): preempt_schedule (copy_page_range) 0.016ms (+0.000ms): preempt_schedule (copy_page_range) 0.016ms (+0.000ms): preempt_schedule (copy_page_range) 0.017ms (+0.000ms): preempt_schedule (copy_page_range) 0.017ms (+0.000ms): preempt_schedule (copy_page_range) 0.017ms (+0.000ms): preempt_schedule (copy_page_range) 0.017ms (+0.000ms): preempt_schedule (copy_page_range) 0.017ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.018ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.019ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.020ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.021ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.022ms (+0.000ms): preempt_schedule (copy_page_range) 0.023ms (+0.000ms): preempt_schedule (copy_page_range) 0.023ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.024ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.025ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.026ms (+0.000ms): preempt_schedule (copy_page_range) 0.027ms (+0.000ms): preempt_schedule (copy_page_range) 0.027ms (+0.000ms): preempt_schedule (copy_page_range) 0.027ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.028ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.029ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.030ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.031ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.032ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.033ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.034ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.035ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.036ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.037ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.038ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.039ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.040ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.041ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.042ms (+0.000ms): preempt_schedule (copy_page_range) 0.043ms (+0.000ms): preempt_schedule (copy_page_range) 0.043ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.044ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.045ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.046ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.047ms (+0.000ms): preempt_schedule (copy_page_range) 0.048ms (+0.000ms): preempt_schedule (copy_page_range) 0.048ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.049ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.050ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.051ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.052ms (+0.000ms): preempt_schedule (copy_page_range) 0.053ms (+0.000ms): preempt_schedule (copy_page_range) 0.053ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.054ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.055ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.056ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.057ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.058ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.059ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.060ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.061ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.062ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.063ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.064ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.065ms (+0.000ms): preempt_schedule (copy_page_range) 0.066ms (+0.000ms): preempt_schedule (copy_page_range) 0.066ms (+0.000ms): preempt_schedule (copy_page_range) 0.066ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.067ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.068ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.069ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.070ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.071ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.072ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.073ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.074ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.075ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.076ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.077ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.078ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.079ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.080ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.081ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.082ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.083ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.084ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.085ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.086ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.087ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.088ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.089ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.090ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.091ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.092ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.093ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.094ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.095ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.096ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.097ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.098ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.099ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.100ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.101ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.102ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.103ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.104ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.105ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.106ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.107ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.108ms (+0.000ms): preempt_schedule (copy_page_range) 0.109ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

Signature · 2 lines
Palimm Palimm!
http://affenbande.org/~tapas/
#387 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

Quoted 13 lines · Florian Schmidt
Quoted 10 lines · On Mon, 16 Aug 2004 06:05:15 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 2 lines · Lee Revell

Anyway, the change to sched.c fixes the mlockall bug, it works perfectly now. Thanks!

great! This fix also means that we've got one more lock-break in the ext3 journalling code and one more lock-break in dcache.c. I've released-P1 with the fix included:

Quoted 9 lines
Quoted 1 line

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P1

Hi,

i don't know if this was mentioned before, but i sometimes see traces like this where half the entries are "preempt_schedule (copy_page_range)". I just wanted to ask if this is normal and expected behaviour.

Yes, Ingo identified an issue with copy_page_range, I don't think it's fixed yet. See the voluntary-preempt-2.6.8.1-P0 thread.

Lee

#388 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

Quoted 7 lines · Lee Revell
Quoted 4 lines

i don't know if this was mentioned before, but i sometimes see traces like this where half the entries are "preempt_schedule (copy_page_range)". I just wanted to ask if this is normal and expected behaviour.

Yes, Ingo identified an issue with copy_page_range, I don't think it's fixed yet. See the voluntary-preempt-2.6.8.1-P0 thread.

right, it's not fixed yet. It's not a trivial critical section - we are holding two locks and are mapping two atomic kmaps.

Ingo

#389 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

Quoted 5 lines · Ingo Molnar
Quoted 2 lines

Yes, Ingo identified an issue with copy_page_range, I don't think it's fixed yet. See the voluntary-preempt-2.6.8.1-P0 thread.

right, it's not fixed yet. It's not a trivial critical section - we are holding two locks and are mapping two atomic kmaps.

fortunately it's easier than i thought - neither the source pmd nor the target pmd can go away so we can simply drop the locks, reschedule, and remap. Does the patch below (ontop of -P3) fix the copy_page_range() latencies you are seeing?

Ingo

mm/memory.c renamed from mm/memory.c.orig +9 -0
@@ -337,6 +337,15 @@ cont_copy_pte_range_noset:
 				}
 				src_pte++;
 				dst_pte++;
+				if (voluntary_need_resched()) {
+					pte_unmap_nested(src_pte);
+					pte_unmap(dst_pte);
+					spin_unlock(&src->page_table_lock);
+					cond_resched_lock(&dst->page_table_lock);
+					spin_lock(&src->page_table_lock);
+					dst_pte = pte_offset_map(dst_pmd, address);
+					src_pte = pte_offset_map_nested(src_pmd, address);
+				}
 			} while ((unsigned long)src_pte & PTE_TABLE_MASK);
 			pte_unmap_nested(src_pte-1);
 			pte_unmap(dst_pte-1);
#390 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
Quoted 2 lines

Yes, Ingo identified an issue with copy_page_range, I don't think it's fixed yet. See the voluntary-preempt-2.6.8.1-P0 thread.

Quoted 8 lines
Quoted 2 lines

right, it's not fixed yet. It's not a trivial critical section - we are holding two locks and are mapping two atomic kmaps.

fortunately it's easier than i thought - neither the source pmd nor the target pmd can go away so we can simply drop the locks, reschedule, and remap. Does the patch below (ontop of -P3) fix the copy_page_range() latencies you are seeing?

Yes, this fixes it. Now the most frequent issue is the xfree86 unmap_vmas latency:

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P3

Lee

#391 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P1

In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Ingo Molnar
Quoted 2 lines

Yes, Ingo identified an issue with copy_page_range, I don't think it's fixed yet. See the voluntary-preempt-2.6.8.1-P0 thread.

right, it's not fixed yet. It's not a trivial critical section - we are holding two locks and are mapping two atomic kmaps.

Quoted 7 lines

fortunately it's easier than i thought - neither the source pmd nor the target pmd can go away so we can simply drop the locks, reschedule, and remap. Does the patch below (ontop of -P3) fix the copy_page_range() latencies you are seeing?

Ingo

Ooops, sorry about my previous post, I had missed this one. It fixes the problem, thanks.

Thomas

#392 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

In reply to Ingo Molnar
Quoted 11 lines · Ingo Molnar
Quoted 5 lines · Lee Revell

I believe the constant-time behavior that I reported was an artifact of ALSA xrun debugging. Now it seems like the latency produced *does* correspond directly to the amount of memory being mlockall'ed. If ./mlockall-test 1500 triggers an xrun at all it's ~0.2ms. 3000 triggers a ~1ms xrun, and 10000 a ~3 ms xrun.

could you try more extreme values - e.g. does 100 MB cause a 30 msec xrun?

This causes a series of xruns, ranging from maybe 2 to 10ms.

Lee

#393 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

In reply to Lee Revell
Quoted 1 line · Lee Revell

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P0

nice. (What is the difference between the left-hand and the right-hand graphs - why is the right-hand side one 'wider'?)

Quoted 7 lines

The peaks on this graph should correspond directly to the length of the non-preemptible critical section reported by Ingo's latency tracer. I think the large peak around 580-600usecs is caused by the extract_entropy issue (which can be hit by regular processes and ksoftirqd), and the large peak around 80-100 by the XFree86 unmap_vmas issue, as the times match and these are by far the most common reported in latency_trace.

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

Ingo

#394 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 7 lines · Ingo Molnar
Quoted 1 line · Lee Revell

http://krustophenia.net/testresults.php?dataset=2.6.8.1-P0

nice. (What is the difference between the left-hand and the right-hand graphs - why is the right-hand side one 'wider'?)

The right hand graph is logarithmically scaled on the y axis (set logscale y in gnuplot). Some of the latencies are rare enough to not show up at all on the linear scale, like the peak in the 400s.

Quoted 8 lines
Quoted 7 lines

The peaks on this graph should correspond directly to the length of the non-preemptible critical section reported by Ingo's latency tracer. I think the large peak around 580-600usecs is caused by the extract_entropy issue (which can be hit by regular processes and ksoftirqd), and the large peak around 80-100 by the XFree86 unmap_vmas issue, as the times match and these are by far the most common reported in latency_trace.

Quoted 5 lines

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

Sure, will try this next.

Lee

#395 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

hm, could you disable the random driver in the .config rather? It seems that adding to the entropy pool (from hardirq context) alone is quite expensive too.

Ingo

#396 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 11 lines · Ingo Molnar
Quoted 4 lines · Ingo Molnar

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

hm, could you disable the random driver in the .config rather? It seems that adding to the entropy pool (from hardirq context) alone is quite expensive too.

Can this be disabled in the .config? I can't find an option for it.

Lee

#397 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 11 lines · Lee Revell
Quoted 9 lines
Quoted 4 lines

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

hm, could you disable the random driver in the .config rather? It seems that adding to the entropy pool (from hardirq context) alone is quite expensive too.

Can this be disabled in the .config? I can't find an option for it.

oh well, indeed it cannot be disabled. Then i'd suggest to return early from extract_entropy(), without doing anything. That is the function that seems to introduce the worst overhead.

Ingo

#398 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 12 lines · Ingo Molnar
Quoted 10 lines · Lee Revell
Quoted 9 lines
Quoted 4 lines

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

hm, could you disable the random driver in the .config rather? It seems that adding to the entropy pool (from hardirq context) alone is quite expensive too.

Quoted 6 lines
Quoted 1 line

Can this be disabled in the .config? I can't find an option for it.

oh well, indeed it cannot be disabled. Then i'd suggest to return early from extract_entropy(), without doing anything. That is the function that seems to introduce the worst overhead.

Yes, I really seem to remember this could be disabled in the past... maybe I am thinking of BSD/OS, or maybe this CONFIG_ option was removed.

This was caused by 'Actions -> Run -> rxvt':

preemption latency trace v1.0
-----------------------------
latency: 409 us, entries: 300 (300)
process: gnome-panel/7575, uid: 1000
nice: 0, policy: 0, rt_priority: 0
=======>
0.000ms (+0.000ms): __vma_link_rb (copy_mm)
0.000ms (+0.000ms): rb_insert_color (copy_mm)
0.000ms (+0.000ms): __rb_rotate_left (rb_insert_color) 0.000ms (+0.000ms): copy_page_range (copy_mm)
0.001ms (+0.000ms): pte_alloc_map (copy_page_range) 0.205ms (+0.204ms): do_IRQ (common_interrupt)
0.206ms (+0.000ms): mask_and_ack_8259A (do_IRQ) 0.209ms (+0.003ms): generic_redirect_hardirq (do_IRQ) 0.210ms (+0.000ms): generic_handle_IRQ_event (do_IRQ) 0.210ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event) 0.210ms (+0.000ms): mark_offset_tsc (timer_interrupt) 0.217ms (+0.006ms): do_timer (timer_interrupt)
0.217ms (+0.000ms): update_process_times (do_timer) 0.217ms (+0.000ms): update_one_process (update_process_times) 0.217ms (+0.000ms): run_local_timers (update_process_times) 0.218ms (+0.000ms): raise_softirq (update_process_times) 0.218ms (+0.000ms): scheduler_tick (update_process_times) 0.218ms (+0.000ms): sched_clock (scheduler_tick) 0.219ms (+0.001ms): task_timeslice (scheduler_tick) 0.220ms (+0.000ms): update_wall_time (do_timer) 0.220ms (+0.000ms): update_wall_time_one_tick (update_wall_time) 0.221ms (+0.000ms): generic_note_interrupt (do_IRQ) 0.221ms (+0.000ms): end_8259A_irq (do_IRQ)
0.222ms (+0.000ms): enable_8259A_irq (do_IRQ)
0.223ms (+0.001ms): do_softirq (do_IRQ)
0.223ms (+0.000ms): __do_softirq (do_softirq)
0.224ms (+0.000ms): wake_up_process (do_softirq) 0.224ms (+0.000ms): try_to_wake_up (wake_up_process) 0.224ms (+0.000ms): task_rq_lock (try_to_wake_up) 0.225ms (+0.000ms): activate_task (try_to_wake_up) 0.225ms (+0.000ms): sched_clock (activate_task) 0.225ms (+0.000ms): recalc_task_prio (activate_task) 0.226ms (+0.000ms): effective_prio (recalc_task_prio) 0.226ms (+0.000ms): enqueue_task (activate_task) 0.227ms (+0.000ms): preempt_schedule (try_to_wake_up) 0.228ms (+0.000ms): preempt_schedule (copy_page_range)

[...]

0.399ms (+0.000ms): preempt_schedule (copy_page_range) 0.400ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

Lee

#399 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 9 lines · Lee Revell

This was caused by 'Actions -> Run -> rxvt':

0.001ms (+0.000ms): pte_alloc_map (copy_page_range) 0.205ms (+0.204ms): do_IRQ (common_interrupt)

0.228ms (+0.000ms): preempt_schedule (copy_page_range)

0.399ms (+0.000ms): preempt_schedule (copy_page_range) 0.400ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

seems we need a lock-break in the innermost loop of copy_page_range(). That loop processes up to 1024 pages currently, before the lock-break in the outer loop happens. Large GUI processes are more likely to have full 4MB regions mapped & populated.

i suspect you could trigger a similarly bad latency by doing a fork() in mlockall-test.cc - the attached mlockall-test2.cc does this. Do you get bad latencies?

Ingo

// here is the code i used to test the mlockall caused xruns #include <sys/mman.h>
#include <iostream>
#include <sstream>
#include <unistd.h>

int main (int argc, char *argv[]) {

        if (argc < 2) {
                std::cout << "how many kbytes you want allocated and mlockall'ed?" << std::endl;
        }

        std::stringstream stream(argv[1]);
        int kbytes;
        stream >> kbytes;
        char *mem = new char[kbytes*1024];
        std::cout << "filling with 0's" << std::endl;
        for (int i = 0; i < kbytes*1024; ++i) {
                mem[i] = 0;
        }

        std::cout << "ok, you want " << kbytes << "kb of memory mlocked.  going for it.." << std::endl;
        int error = mlockall(MCL_CURRENT);
        if (error != 0) { std::cout << "mlock error" << std::endl; }
        else { std::cout << "mlock successfull" << std::endl;}
	fork();
}
#400 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 11 lines · Ingo Molnar

* Lee Revell <[email protected]> wrote:

seems we need a lock-break in the innermost loop of copy_page_range(). That loop processes up to 1024 pages currently, before the lock-break in the outer loop happens. Large GUI processes are more likely to have full 4MB regions mapped & populated.

i suspect you could trigger a similarly bad latency by doing a fork() in mlockall-test.cc - the attached mlockall-test2.cc does this. Do you get bad latencies?

Yes, mlockall-test2 with 10+MB of memory produces a 200us-3ms xrun about half the time.

Lee

#401 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Lee Revell
Quoted 5 lines
Quoted 4 lines

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

hm, could you disable the random driver in the .config rather? It seems that adding to the entropy pool (from hardirq context) alone is quite expensive too.

Can this be disabled in the .config? I can't find an option for it.

oh well, indeed it cannot be disabled.

Hmm, what happens if CONFIG_HW_RANDOM is set then? I would call it a bug if having a hardware RNG didn't disable the software /dev/random driver. This implies that the software RNG can be easily disabled.

Lee

#402 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

In reply to Ingo Molnar
Quoted 12 lines · Ingo Molnar
Quoted 10 lines · Lee Revell
Quoted 9 lines
Quoted 4 lines

just to check this theory, could you make __check_and_rekey() an empty function? This should still produce a working random driver, albeit at much reduced entropy. If these latencies have a relationship to the mlockall() issue then this change should have an effect.

hm, could you disable the random driver in the .config rather? It seems that adding to the entropy pool (from hardirq context) alone is quite expensive too.

Quoted 6 lines
Quoted 1 line

Can this be disabled in the .config? I can't find an option for it.

oh well, indeed it cannot be disabled. Then i'd suggest to return early from extract_entropy(), without doing anything. That is the function that seems to introduce the worst overhead.

OK, I tried the quick hack of just returning 0 before taking the spinlock in extract_entropy, but this broke /dev/random which prevented me from logging in. I guess we will have to properly fake it here.

Alas, it looks like the software RNG is not disabled when CONFIG_HW_RANDOM is set. I would call this a bug - imagine incurring the overhead of software 3D acceleration even with hardware 3D acceleration enabled.

Lee

#403 lore raw

Re: [patch] voluntary-preempt-2.6.8.1-P0

Quoted 8 lines · Lee Revell
Quoted 3 lines

oh well, indeed it cannot be disabled. Then i'd suggest to return early from extract_entropy(), without doing anything. That is the function that seems to introduce the worst overhead.

OK, I tried the quick hack of just returning 0 before taking the spinlock in extract_entropy, but this broke /dev/random which prevented me from logging in. I guess we will have to properly fake it here.

you should return nbytes, not zero.

Ingo

#404 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc4-O7

In reply to Ingo Molnar
Quoted 2 lines · Ingo Molnar

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc4-O7

Here is the trace for a long ide i/o completion from the hardirq thread (IRQ 15 in my case). These are getting very long, so I modified my latency-graphing PHP script to display any .txt files I put in that directory. All of the traces for a given kernel can be found like so:

http://krustophenia.net/testresults.php?dataset=2.6.8-rc4-bk3-O7

Lee

#405 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc4-O7

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

* Lee Revell <[email protected]> wrote:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc4-O7

Some more interesting results. I am not seeing much of a difference in performance, but the shape of the distribution changes quite a bit:

http://krustophenia.net/testresults.php?dataset=2.6.8-rc3-O5
http://krustophenia.net/testresults.php?dataset=2.6.8-rc4-bk3-O7

Lee

#406 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc4-O7

Quoted 8 lines · Lee Revell
Quoted 2 lines

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc4-O7

Some more interesting results. I am not seeing much of a difference in performance, but the shape of the distribution changes quite a bit:

http://krustophenia.net/testresults.php?dataset=2.6.8-rc3-O5
http://krustophenia.net/testresults.php?dataset=2.6.8-rc4-bk3-O7

nice graphs - i suspect the shape difference is due to the tracing overhead: it adds visible (and measurable) overhead to all kernel execution paths, but since the overwhelming majority of paths have very low latencies it doesnt degrade the max_latency results visibly. This is good empirical data - it means what we are after a few offenders who are more than a magnitude slower than the common paths. It also shows that most of the extra overhead is rather in isolated functions (e.g. the memcpy's), not in some big latency path involving many small functions.

i also suspect that if you magnified the 'sharp' nontracing graphs near zero they'd show a similar shape to the tracing graphs.

Ingo

#407 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc4-O7

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

ok, it seems the lock-break of the outer loop was not enough - the up to 1024 iterations in the inner loop can generate quite high latencies too.

In some of the traces, like this one:

http://krustophenia.net/testresults.php?dataset=2.6.8-rc4-bk3-O7#/var/www/2.6.8-rc4-bk3-O7/mount_reiserfs_latency_trace.txt

there are calls to voluntary_resched. How is this possible? Does it mean that we called voluntary_resched while holding a spinlock, where we needed to call voluntary_preempt_lock(&foo_lock), and thus failed to reschedule?

Lee

#408 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc4-O7

Quoted 8 lines · Lee Revell

In some of the traces, like this one:

http://krustophenia.net/testresults.php?dataset=2.6.8-rc4-bk3-O7#/var/www/2.6.8-rc4-bk3-O7/mount_reiserfs_latency_trace.txt

there are calls to voluntary_resched. How is this possible? Does it mean that we called voluntary_resched while holding a spinlock, where we needed to call voluntary_preempt_lock(&foo_lock), and thus failed to reschedule?

voluntary_resched() was probably called as part of the might_sleep_if() done in mm/slab.c. If you have CONFIG_DEBUG_SPINLOCK_SLEEP enabled then the kernel should have complained about 'Debug: sleeping function ...'.

but what i think happened here is that reiserfs still uses lock_kernel()/unlock_kernel() quite alot (eg. ext3 or xfs doesnt), which from a preemptability POV is just as much of a critical section as a spinlock, but processes can sleep (the scheduler auto-releases and auto-reacquires the big kernel lock).

Ingo

kernel/sched.c renamed from kernel/sched.c.orig +2 -2
@@ -3210,11 +3210,11 @@ __setup("voluntary-preempt=", voluntary_
 
 int __sched voluntary_resched(void)
 {
-	if (kernel_preemption || !voluntary_preemption)
-		return 0;
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 	__might_sleep(__FILE__, __LINE__);
 #endif
+	if (kernel_preemption || !voluntary_preemption)
+		return 0;
 	/*
 	 * The system_state check is somewhat ugly but we might be
 	 * called during early boot when we are not yet ready to reschedule.
#409 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

In reply to Ingo Molnar
Quoted 6 lines · On Fri, 13 Aug 2004 01:51:16 +0200 Ingo Molnar <[email protected]> wrote:

-O6 also adds another timing option besides preempt_thresh: if preempt_thresh is set to 0 then the tracer will automatically track the largest-previous latency. (i.e. the system does a search for the absolute maximum latency.) The /proc/sys/kernel/preempt_max_latency control can be used to reset this value to conduct a new search for a new workload, without having to reboot the system.

Hmm, should a new report only be generated when the newly measured latency is really > than all other ones? I get the feeling that >= seems to be enough. Here a dmesg extract:

(swapper/1): new 6 us maximum-latency critical section.
=> started at: <schedule+0x5c/0x5b0>
=> ended at: <finish_task_switch+0x32/0x90>

 [<c01064ae>] dump_stack+0x1e/0x20
 [<c0130c64>] check_preempt_timing+0x184/0x1e0
 [<c0130db4>] sub_preempt_count+0x44/0x50
 [<c0112c22>] finish_task_switch+0x32/0x90
 [<c0112c9e>] schedule_tail+0x1e/0x60
 [<c0105df6>] ret_from_fork+0x6/0x14
(swapper/1): new 8 us maximum-latency critical section.
 => started at: <voluntary_resched+0x1e/0x50>
 => ended at:   <voluntary_resched+0x1e/0x50>
 [<c01064ae>] dump_stack+0x1e/0x20
 [<c0130c64>] check_preempt_timing+0x184/0x1e0
 [<c0130cf8>] touch_preempt_timing+0x38/0x60
 [<c0292a9e>] voluntary_resched+0x1e/0x50
 [<c013a1b6>] kmem_cache_alloc+0x56/0x60
 [<c01160c0>] copy_process+0x9e0/0xbd0
 [<c0116302>] do_fork+0x52/0x1b3
 [<c0104135>] kernel_thread+0x85/0x90
 [<c012a907>] keventd_create_kthread+0x27/0x50
 [<c012a9a7>] kthread_create+0x77/0xd0
 [<c031dc9b>] cpu_callback+0x5b/0xc0
 [<c031dd22>] spawn_ksoftirqd+0x22/0x50
 [<c0100334>] init+0x34/0x170
 [<c01040a5>] kernel_thread_helper+0x5/0x10
(swapper/1): new 9 us maximum-latency critical section.
 => started at: <voluntary_resched+0x1e/0x50>
 => ended at:   <voluntary_resched+0x1e/0x50>
 [<c01064ae>] dump_stack+0x1e/0x20
 [<c0130c64>] check_preempt_timing+0x184/0x1e0
 [<c0130cf8>] touch_preempt_timing+0x38/0x60
 [<c0292a9e>] voluntary_resched+0x1e/0x50
 [<c013a1b6>] kmem_cache_alloc+0x56/0x60
 [<c0116009>] copy_process+0x929/0xbd0
 [<c0116302>] do_fork+0x52/0x1b3
 [<c0104135>] kernel_thread+0x85/0x90
 [<c012a907>] keventd_create_kthread+0x27/0x50
 [<c012a9a7>] kthread_create+0x77/0xd0
 [<c031dc9b>] cpu_callback+0x5b/0xc0
 [<c031dd22>] spawn_ksoftirqd+0x22/0x50
 [<c0100334>] init+0x34/0x170
 [<c01040a5>] kernel_thread_helper+0x5/0x10

Here are two reports with the same maximum-latency (31 us):

(swapper/1): new 31 us maximum-latency critical section.
=> started at: <voluntary_resched+0x1e/0x50>
=> ended at: <voluntary_resched+0x1e/0x50>

 [<c01064ae>] dump_stack+0x1e/0x20
 [<c0130c64>] check_preempt_timing+0x184/0x1e0
 [<c0130cf8>] touch_preempt_timing+0x38/0x60
 [<c0292a9e>] voluntary_resched+0x1e/0x50
 [<c013a3f0>] __kmalloc+0x80/0x90
 [<c0316279>] malloc+0x19/0x20
 [<c01024d4>] huft_build+0x2d4/0x590
 [<c0102ea1>] inflate_fixed+0xc1/0x1a0
 [<c010379a>] inflate+0x4a/0xb0
 [<c0103b89>] gunzip+0x2f9/0x530
 [<c0316e41>] unpack_to_rootfs+0x191/0x220
 [<c0316efb>] populate_rootfs+0x2b/0x130
 [<c010033e>] init+0x3e/0x170
 [<c01040a5>] kernel_thread_helper+0x5/0x10
(swapper/1): new 31 us maximum-latency critical section.
 => started at: <voluntary_resched+0x1e/0x50>
 => ended at:   <voluntary_resched+0x1e/0x50>
 [<c01064ae>] dump_stack+0x1e/0x20
 [<c0130c64>] check_preempt_timing+0x184/0x1e0
 [<c0130cf8>] touch_preempt_timing+0x38/0x60
 [<c0292a9e>] voluntary_resched+0x1e/0x50
 [<c013a1b6>] kmem_cache_alloc+0x56/0x60
 [<c015c8e9>] getname+0x29/0xc0
 [<c015f3fd>] sys_mkdir+0x1d/0xf0
 [<c03169af>] do_name+0x10f/0x1b0
 [<c0316bb1>] write_buffer+0x21/0x30
 [<c0316be4>] flush_buffer+0x24/0x60
 [<c0316c64>] flush_window+0x24/0x70
 [<c01037db>] inflate+0x8b/0xb0
 [<c0103b89>] gunzip+0x2f9/0x530
 [<c0316e41>] unpack_to_rootfs+0x191/0x220
 [<c0316efb>] populate_rootfs+0x2b/0x130
 [<c010033e>] init+0x3e/0x170
 [<c01040a5>] kernel_thread_helper+0x5/0x10

NET: Registered protocol family 16

Btw: i do have some regular ca. 300 us latencies.. Here are some traces (these happen with an average frequency of ca. 0.3hz):

Aug 13 12:40:49 mango kernel: (ksoftirqd/0/2): 307 us critical section
violates 250 us threshold.
Aug 13 12:40:49 mango kernel:  => started at: <___do_softirq+0x20/0x90>
Aug 13 12:40:49 mango kernel:  => ended at:  
<cond_resched_softirq+0x59/0x70>
Aug 13 12:40:49 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 13 12:40:49 mango kernel:  [<c0130c64>]
check_preempt_timing+0x184/0x1e0
Aug 13 12:40:49 mango kernel:  [<c0130cf8>]
touch_preempt_timing+0x38/0x60
Aug 13 12:40:49 mango kernel:  [<c0113df9>]
cond_resched_softirq+0x59/0x70
Aug 13 12:40:49 mango kernel:  [<c011faef>] run_timer_softirq+0xdf/0x1d0
Aug 13 12:40:49 mango kernel:  [<c011adbc>] ___do_softirq+0x7c/0x90
Aug 13 12:40:49 mango kernel:  [<c011ae19>] _do_softirq+0x9/0x10
Aug 13 12:40:49 mango kernel:  [<c011b1d4>] ksoftirqd+0x84/0xd0
Aug 13 12:40:49 mango kernel:  [<c012a8da>] kthread+0xaa/0xb0
Aug 13 12:40:49 mango kernel:  [<c01040a5>]
kernel_thread_helper+0x5/0x10
Aug 13 12:40:54 mango kernel: (ksoftirqd/0/2): 307 us critical section
violates 250 us threshold.
Aug 13 12:40:54 mango kernel:  => started at: <___do_softirq+0x20/0x90>
Aug 13 12:40:54 mango kernel:  => ended at:  
<cond_resched_softirq+0x59/0x70>
Aug 13 12:40:54 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 13 12:40:54 mango kernel:  [<c0130c64>]
check_preempt_timing+0x184/0x1e0
Aug 13 12:40:54 mango kernel:  [<c0130cf8>]
touch_preempt_timing+0x38/0x60
Aug 13 12:40:54 mango kernel:  [<c0113df9>]
cond_resched_softirq+0x59/0x70
Aug 13 12:40:54 mango kernel:  [<c011faef>] run_timer_softirq+0xdf/0x1d0
Aug 13 12:40:54 mango kernel:  [<c011adbc>] ___do_softirq+0x7c/0x90
Aug 13 12:40:54 mango kernel:  [<c011ae19>] _do_softirq+0x9/0x10
Aug 13 12:40:54 mango kernel:  [<c011b1d4>] ksoftirqd+0x84/0xd0
Aug 13 12:40:54 mango kernel:  [<c012a8da>] kthread+0xaa/0xb0
Aug 13 12:40:54 mango kernel:  [<c01040a5>]
kernel_thread_helper+0x5/0x10
Aug 13 12:40:59 mango kernel: (ksoftirqd/0/2): 308 us critical section
violates 250 us threshold.
Aug 13 12:40:59 mango kernel:  => started at: <___do_softirq+0x20/0x90>
Aug 13 12:40:59 mango kernel:  => ended at:  
<cond_resched_softirq+0x59/0x70>
Aug 13 12:40:59 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 13 12:40:59 mango kernel:  [<c0130c64>]
check_preempt_timing+0x184/0x1e0
Aug 13 12:40:59 mango kernel:  [<c0130cf8>]
touch_preempt_timing+0x38/0x60
Aug 13 12:40:59 mango kernel:  [<c0113df9>]
cond_resched_softirq+0x59/0x70
Aug 13 12:40:59 mango kernel:  [<c011faef>] run_timer_softirq+0xdf/0x1d0
Aug 13 12:40:59 mango kernel:  [<c011adbc>] ___do_softirq+0x7c/0x90
Aug 13 12:40:59 mango kernel:  [<c011ae19>] _do_softirq+0x9/0x10
Aug 13 12:40:59 mango kernel:  [<c011b1d4>] ksoftirqd+0x84/0xd0
Aug 13 12:40:59 mango kernel:  [<c012a8da>] kthread+0xaa/0xb0
Aug 13 12:40:59 mango kernel:  [<c01040a5>]
kernel_thread_helper+0x5/0x10
Aug 13 12:41:04 mango kernel: (ksoftirqd/0/2): 307 us critical section
violates 250 us threshold.
Aug 13 12:41:04 mango kernel:  => started at: <___do_softirq+0x20/0x90>
Aug 13 12:41:04 mango kernel:  => ended at:  
<cond_resched_softirq+0x59/0x70>
Aug 13 12:41:04 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 13 12:41:04 mango kernel:  [<c0130c64>]
check_preempt_timing+0x184/0x1e0
Aug 13 12:41:04 mango kernel:  [<c0130cf8>]
touch_preempt_timing+0x38/0x60
Aug 13 12:41:04 mango kernel:  [<c0113df9>]
cond_resched_softirq+0x59/0x70
Aug 13 12:41:04 mango kernel:  [<c011faef>] run_timer_softirq+0xdf/0x1d0
Aug 13 12:41:04 mango kernel:  [<c011adbc>] ___do_softirq+0x7c/0x90
Aug 13 12:41:04 mango kernel:  [<c011ae19>] _do_softirq+0x9/0x10
Aug 13 12:41:04 mango kernel:  [<c011b1d4>] ksoftirqd+0x84/0xd0
Aug 13 12:41:04 mango kernel:  [<c012a8da>] kthread+0xaa/0xb0
Aug 13 12:41:04 mango kernel:  [<c01040a5>]
kernel_thread_helper+0x5/0x10
Aug 13 12:41:09 mango kernel: (ksoftirqd/0/2): 307 us critical section
violates 250 us threshold.
Aug 13 12:41:09 mango kernel:  => started at: <___do_softirq+0x20/0x90>
Aug 13 12:41:09 mango kernel:  => ended at:  
<cond_resched_softirq+0x59/0x70>
Aug 13 12:41:09 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 13 12:41:09 mango kernel:  [<c0130c64>]
check_preempt_timing+0x184/0x1e0
Aug 13 12:41:09 mango kernel:  [<c0130cf8>]
touch_preempt_timing+0x38/0x60
Aug 13 12:41:09 mango kernel:  [<c0113df9>]
cond_resched_softirq+0x59/0x70
Aug 13 12:41:09 mango kernel:  [<c011faef>] run_timer_softirq+0xdf/0x1d0
Aug 13 12:41:09 mango kernel:  [<c011adbc>] ___do_softirq+0x7c/0x90
Aug 13 12:41:09 mango kernel:  [<c011ae19>] _do_softirq+0x9/0x10
Aug 13 12:41:09 mango kernel:  [<c011b1d4>] ksoftirqd+0x84/0xd0
Aug 13 12:41:09 mango kernel:  [<c012a8da>] kthread+0xaa/0xb0
Aug 13 12:41:09 mango kernel:  [<c01040a5>]

kernel_thread_helper+0x5/0x10

Some system info:

I do not habe preempt-tracing enabled, only voluntary preempt and preempt timing..

Linux mango.fruits.de 2.6.8-rc4 #1 Fri Aug 13 03:05:04 CEST 2004 i686 GNU/Linux
-----------
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 7
model name : AMD Duron(tm) Processor
stepping : 1
cpu MHz : 1195.497
cache size : 64 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips : 2359.29

-----------
/proc/interrupts:
CPU0

  0:    1832325          XT-PIC  timer
  1:       8892          XT-PIC  i8042
  2:          0          XT-PIC  cascade
  5:          0          XT-PIC  CS46XX
  8:          4          XT-PIC  rtc
 10:       2066          XT-PIC  eth0
 12:      39786          XT-PIC  i8042
 14:        900          XT-PIC  ide0
 15:      12719          XT-PIC  ide1
NMI:          0 
ERR:          0

-----------
/proc/irq/10/eth0/threaded:1
/proc/irq/12/i8042/threaded:1
/proc/irq/14/ide0/threaded:1
/proc/irq/15/ide1/threaded:1
/proc/irq/1/i8042/threaded:1
/proc/irq/5/CS46XX/threaded:0
/proc/irq/8/rtc/threaded:0
-----------
/sys/block/hda/queue/max_sectors_kb:16
/sys/block/hdd/queue/max_sectors_kb:16
-----------
voluntary_preemption
3
kernel_preemption
1
-----------
preempt_thresh
250

Signature · 2 lines
Palimm Palimm!
http://affenbande.org/~tapas/
#410 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 4 lines · Florian Schmidt

Hmm, should a new report only be generated when the newly measured latency is really > than all other ones? I get the feeling that >= seems to be enough. Here a dmesg extract:

Quoted 8 lines

(swapper/1): new 6 us maximum-latency critical section. (swapper/1): new 8 us maximum-latency critical section. (swapper/1): new 9 us maximum-latency critical section.

Here are two reports with the same maximum-latency (31 us):

(swapper/1): new 31 us maximum-latency critical section. (swapper/1): new 31 us maximum-latency critical section.

the latency tracer tracks latencies in cycle units - but they are displayed at microsecond accuracy - hence these 'equal' latencies.

to jump-start all those smaller latencies you can do this:

echo 100 > /proc/sys/kernel/preempt_max_latency echo 0 > /proc/sys/kernel/preempt_thresh

this way the maximum-searching only starts at 100 usecs.

Quoted 6 lines

Btw: i do have some regular ca. 300 us latencies.. Here are some traces (these happen with an average frequency of ca. 0.3hz):

(ksoftirqd/0/2): 307 us critical section violates 250 us threshold.
=> started at: <___do_softirq+0x20/0x90>
=> ended at: <cond_resched_softirq+0x59/0x70>

this is too opaque - could you try -O7, enable tracing and save a /proc/latency_trace instance of such a latency? It looks like some sort of softirq latency - perhaps one particular driver's timer fn causes it
- we'll be able to tell more from the trace.

Ingo

#411 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 8 lines · On Fri, 13 Aug 2004 12:54:06 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 3 lines

(ksoftirqd/0/2): 307 us critical section violates 250 us threshold.
=> started at: <___do_softirq+0x20/0x90>
=> ended at: <cond_resched_softirq+0x59/0x70>

this is too opaque - could you try -O7, enable tracing and save a /proc/latency_trace instance of such a latency? It looks like some sort of softirq latency - perhaps one particular driver's timer fn causes it- we'll be able to tell more from the trace.

Hi, this looks like one of them:

mango:~# cat /proc/latency_trace 
preemption latency trace v1.0
-----------------------------
 latency: 308 us, entries: 12 (12)
 process: ksoftirqd/0/2, uid: 0
 nice: -10, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): run_timer_softirq (___do_softirq)
 0.000ms (+0.000ms): sis900_timer (run_timer_softirq)
 0.001ms (+0.000ms): mdio_read (sis900_timer)
 0.002ms (+0.000ms): mdio_reset (mdio_read)
 0.071ms (+0.069ms): mdio_idle (mdio_read)
 0.151ms (+0.079ms): mdio_read (sis900_timer)
 0.151ms (+0.000ms): mdio_reset (mdio_read)
 0.220ms (+0.069ms): mdio_idle (mdio_read)
 0.300ms (+0.079ms): __mod_timer (sis900_timer)
 0.300ms (+0.000ms): internal_add_timer (__mod_timer)
 0.300ms (+0.000ms): cond_resched_softirq (run_timer_softirq)
 0.301ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

Flo

Signature · 2 lines
Palimm Palimm!
http://affenbande.org/~tapas/
#412 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 25 lines · Florian Schmidt
Quoted 8 lines · On Fri, 13 Aug 2004 12:54:06 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 3 lines

(ksoftirqd/0/2): 307 us critical section violates 250 us threshold.
=> started at: <___do_softirq+0x20/0x90>
=> ended at: <cond_resched_softirq+0x59/0x70>

this is too opaque - could you try -O7, enable tracing and save a /proc/latency_trace instance of such a latency? It looks like some sort of softirq latency - perhaps one particular driver's timer fn causes it- we'll be able to tell more from the trace.

Hi, this looks like one of them:

mango:~# cat /proc/latency_trace 
preemption latency trace v1.0
-----------------------------
 latency: 308 us, entries: 12 (12)
 process: ksoftirqd/0/2, uid: 0
 nice: -10, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): run_timer_softirq (___do_softirq)
 0.000ms (+0.000ms): sis900_timer (run_timer_softirq)
 0.001ms (+0.000ms): mdio_read (sis900_timer)
 0.002ms (+0.000ms): mdio_reset (mdio_read)
Quoted 8 lines
 0.071ms (+0.069ms): mdio_idle (mdio_read)
 0.151ms (+0.079ms): mdio_read (sis900_timer)
 0.151ms (+0.000ms): mdio_reset (mdio_read)
 0.220ms (+0.069ms): mdio_idle (mdio_read)
 0.300ms (+0.079ms): __mod_timer (sis900_timer)
 0.300ms (+0.000ms): internal_add_timer (__mod_timer)
 0.300ms (+0.000ms): cond_resched_softirq (run_timer_softirq)
 0.301ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

indeed this seems to be a driver related timer: driver/net/sis900.c's sis900_timer() function. This timer polls your network card for link status once every second. The mdio_read() function does alot of in/out instructions which are quite slow.

could you try the following and disable mdio_delay():

	#define mdio_delay() do { } while (0)

normally it should work just fine. Worst-case you'd get a non-working network driver. Does this change reduce the latency?

Ingo

A message that is not in the archive (sent off-list or not yet received): [email protected]

#414 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 6 lines · Florian Schmidt

This is something i wanted to ask anyways: In one of your first VP announcements you mentioned you wanted to eliminate all latencies > 1 ms. To me it seems, that that goal is pretty much reached [at least i don't see any longer ones except for at boot and shutdown]. So the question is: What is the lower limit for laterncies that you want to hear reports about?

well ... i'm interested in all latencies that are well above the typical average latencies in the system. E.g. when the average is around 20-30 usecs then reports of 200-300 usecs would be interesting.

there's no hard limit, really. Also, sometimes latencies that are 0.3 msec in the report could be 3 msec if triggered properly. So a seemingly lower than 1 msec latency can very well pinpoint a problem area.

Quoted 7 lines

WRT mlockall: i tried mlockall'ing 500 megs. This produced a new max latency of 299 us. the trace is rather long. This one is with jackd running and the one below this is w/o jackd running:

0.010ms (+0.000ms): free_page_and_swap_cache (clear_page_tables) 0.010ms (+0.000ms): __page_cache_release (clear_page_tables) 0.010ms (+0.000ms): free_hot_page (clear_page_tables)

hm, the reason for this one is that clear_page_tables() does all the freeing in a single uninterrupted critical section covered by mm->page_table_lock.

This function needs a lock-break i believe. Especially in the process-exit case (exit_mmap()) the lock seems unjustified - the current task is the sole owner of a never-to-be-used-again collection of pagetables.

Ingo

#415 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

i've uploaded the latest version of the voluntary-preempt patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc4-O6

<snip>

Quoted 3 lines

reports, suggestions welcome,

Ingo

I used O7.

I have tested this for a day now, and I have noticed problems:
1)
kernel syslog gets a record like this:

Quoted 12 lines

(default.hotplug/1470): 121 us critical section violates 100 us threshold.
=> started at: <kmap_high+0x2b/0x2d0>
=> ended at: <kmap_high+0x1a9/0x2d0>

  [<c0105a23>] dump_stack+0x23/0x30
  [<c0140d14>] check_preempt_timing+0x184/0x1e0
  [<c0140e84>] sub_preempt_count+0x54/0x5d
  [<c0152959>] kmap_high+0x1a9/0x2d0
  [<c017655a>] copy_strings+0xea/0x230
  [<c01766db>] copy_strings_kernel+0x3b/0x50
  [<c017840d>] do_execve+0x12d/0x1f0
  [<c0103284>] sys_execve+0x44/0x80
  [<c0104b95>] sysenter_past_esp+0x52/0x71

and the /proc/latency_trace gets:

Quoted 5 lines

preemption latency trace v1.0
-----------------------------
latency: 121 us, entries: 1032 (1032)
process: default.hotplug/1470, uid: 0
nice: -10, policy: 0, rt_priority: 0

Quoted 8 lines

=======>
0.000ms (+0.000ms): page_address (kmap_high) 0.000ms (+0.000ms): page_slot (page_address) 0.000ms (+0.000ms): flush_all_zero_pkmaps (kmap_high) 0.000ms (+0.000ms): set_page_address (flush_all_zero_pkmaps)
[...]
0.118ms (+0.000ms): page_slot (set_page_address) 0.118ms (+0.000ms): check_preempt_timing (sub_preempt_count)

Could the patch be adjusted to make the syslog and the /proc/latency_trace produce the same output?

2)
I suspect that there is a problem with reiserfs, but when I detect a momentary hang in the system(mouse stops moving), no latency_trace appears.

#416 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 18 lines · James Courtier-Dutton

I have tested this for a day now, and I have noticed problems:
1)
kernel syslog gets a record like this:

Quoted 1 line

(default.hotplug/1470): 121 us critical section violates 100 us

threshold.

and the /proc/latency_trace gets:

Quoted 10 lines

preemption latency trace v1.0
-----------------------------
latency: 121 us, entries: 1032 (1032)
process: default.hotplug/1470, uid: 0
nice: -10, policy: 0, rt_priority: 0
=======>
0.000ms (+0.000ms): page_address (kmap_high) 0.000ms (+0.000ms): page_slot (page_address) 0.000ms (+0.000ms): flush_all_zero_pkmaps (kmap_high) 0.000ms (+0.000ms): set_page_address (flush_all_zero_pkmaps)

Quoted 2 lines

Could the patch be adjusted to make the syslog and the /proc/latency_trace produce the same output?

We cannot include the full trace in the syslog - it's possibly thousands of lines long.

Quoted 3 lines

2)
I suspect that there is a problem with reiserfs, but when I detect a momentary hang in the system(mouse stops moving), no latency_trace appears.

well, the mouse could stop moving for a number of reasons. It's handled via the X server and if the X server is preempted (for whatever reason) then the mouse pointer isnt updated.

you could try to change the mouse IRQ to be non-threaded. Also, do you have kernel_preemption set to 1? It defaults to 0.

Ingo

#417 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

This is just for info, now that we have a nice latency testing tool, we might as well collect some useful traces that we can later work on.

Here is a trace showing a latency of 39034us. http://www.superbug.demon.co.uk/kernel/

The interesting parts of it are:
0.002ms (+0.000ms): __preempt_spin_lock (schedule) 0.523ms (+0.520ms): do_IRQ (common_interrupt)
...
0.532ms (+0.000ms): end_edge_ioapic_irq (do_IRQ)
0.763ms (+0.230ms): smp_apic_timer_interrupt (apic_timer_interrupt)
...
0.768ms (+0.000ms): preempt_schedule (try_to_wake_up) 1.523ms (+0.754ms): do_IRQ (common_interrupt)
...
1.533ms (+0.000ms): __do_softirq (do_softirq)
1.763ms (+0.229ms): smp_apic_timer_interrupt (apic_timer_interrupt)
...
1.765ms (+0.000ms): __do_softirq (do_softirq)
2.523ms (+0.757ms): do_IRQ (common_interrupt)
...
2.533ms (+0.000ms): do_softirq (do_IRQ)
2.533ms (+0.000ms): __do_softirq (do_softirq)
2.763ms (+0.230ms): smp_apic_timer_interrupt (apic_timer_interrupt)
...

This looks to me to be a bug somewhere. Either in the O7 patch, or in the kernel. Surely, do_IRQ should happen quickly, and not take 39ms.

#418 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 8 lines · James Courtier-Dutton

This is just for info, now that we have a nice latency testing tool, we might as well collect some useful traces that we can later work on.

Here is a trace showing a latency of 39034us.
http://www.superbug.demon.co.uk/kernel/

This looks to me to be a bug somewhere. Either in the O7 patch, or in the kernel. Surely, do_IRQ should happen quickly, and not take 39ms.

something's wrong indeed. Is this an x86 SMP system? If it's SMP then please apply the patch below (ontop of -O7), it fixes an SMP false positive bug in the latency timing code.

the process is looping somewhere. Here are the non-IRQ trace entries:

0.001ms (+0.000ms): __switch_to (schedule) 0.002ms (+0.000ms): finish_task_switch (schedule) 0.002ms (+0.000ms): __preempt_spin_lock (schedule)
... [lots of IRQs] ...
38.126ms (+0.362ms): preempt_schedule (schedule) 38.126ms (+0.000ms): sched_clock (schedule) 38.127ms (+0.000ms): find_next_bit (schedule) 38.127ms (+0.000ms): task_timeslice (schedule)

this shows that we are looping in __preempt_spin_lock() - most likely via schedule()'s reqacquire_kernel_lock() code.

i.e. another CPU is holding the big kernel lock and this CPU is looping. _but_ this CPU is fully preemptible so the trace produces this false positive.

Ingo

kernel/sched.c renamed from kernel/sched.c.orig2 +4 -0
@@ -4210,7 +4210,9 @@ void __sched __preempt_spin_lock(spinloc
 	do {
 		preempt_enable();
 		while (spin_is_locked(lock)) {
+			preempt_disable();
 			touch_preempt_timing();
+			preempt_enable();
 			cpu_relax();
 		}
 		preempt_disable();
@@ -4229,7 +4231,9 @@ void __sched __preempt_write_lock(rwlock
 	do {
 		preempt_enable();
 		while (rwlock_is_locked(lock)) {
+			preempt_disable();
 			touch_preempt_timing();
+			preempt_enable();
 			cpu_relax();
 		}
 		preempt_disable();
#419 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 28 lines · Ingo Molnar
Quoted 8 lines · James Courtier-Dutton

This is just for info, now that we have a nice latency testing tool, we might as well collect some useful traces that we can later work on.

Here is a trace showing a latency of 39034us. http://www.superbug.demon.co.uk/kernel/

This looks to me to be a bug somewhere. Either in the O7 patch, or in the kernel. Surely, do_IRQ should happen quickly, and not take 39ms.

something's wrong indeed. Is this an x86 SMP system? If it's SMP then please apply the patch below (ontop of -O7), it fixes an SMP false positive bug in the latency timing code.

the process is looping somewhere. Here are the non-IRQ trace entries:

0.001ms (+0.000ms): __switch_to (schedule) 0.002ms (+0.000ms): finish_task_switch (schedule) 0.002ms (+0.000ms): __preempt_spin_lock (schedule)
... [lots of IRQs] ...
38.126ms (+0.362ms): preempt_schedule (schedule) 38.126ms (+0.000ms): sched_clock (schedule) 38.127ms (+0.000ms): find_next_bit (schedule) 38.127ms (+0.000ms): task_timeslice (schedule)

Quoted 8 lines

this shows that we are looping in __preempt_spin_lock() - most likely via schedule()'s reqacquire_kernel_lock() code.

i.e. another CPU is holding the big kernel lock and this CPU is looping. _but_ this CPU is fully preemptible so the trace produces this false positive.

Ingo

I have not seen that particular problem any more, so I think the patch helped.

I have found a new problem though:

# cat latency_trace
preemption latency trace v1.0
-----------------------------
 latency: 1883455195 us, entries: 1 (1)
 process: ksoftirqd/1/5, uid: 0
 nice: -10, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): cond_resched_softirq (___do_softirq)

That looks bad to me. The user did not notice any latency, but 1883 seconds seems like a high latency to me!

James

#420 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

Quoted 11 lines · James Courtier-Dutton

I have found a new problem though:

# cat latency_trace
preemption latency trace v1.0
-----------------------------
 latency: 1883455195 us, entries: 1 (1)
 process: ksoftirqd/1/5, uid: 0
 nice: -10, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): cond_resched_softirq (___do_softirq)
Quoted 2 lines

That looks bad to me. The user did not notice any latency, but 1883 seconds seems like a high latency to me!

yeah, it doesnt look healthy. This could be some sort of tracing underflow - what is the 'grep MHz /proc/cpuinfo' value of your box?

Also, could you try the -P4 patch, it has a more robust timestamping mechanism.

Ingo

#421 lore raw

Re: [patch] Latency Tracer, voluntary-preempt-2.6.8-rc4-O6

In reply to James Courtier-Dutton
Quoted 2 lines · James Courtier-Dutton

Could the patch be adjusted to make the syslog and the /proc/latency_trace produce the same output?

to further unify the formats i've added the start/end entries to latency_trace in -P4. Saving the stacktrace would be too much trouble though.

Ingo

#422 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc3-O4

In reply to Ingo Molnar
Quoted 5 lines · On Mon, 9 Aug 2004 12:46:49 +0200 Ingo Molnar <[email protected]> wrote:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc3-O4

(it should apply to -rc2 cleanly too.)

-O4 fixes a two more latency sources:

Hi, i also wanted to report that i also still see these mysterious (at least to me) "(swapper/0)" critical section reports (i use O4 with the preempt-timing-on-2.6.8-rc3-O4.patch): These are basically the only xruns/critical section reports i see when using jackd at 2*64 and with some clients connected. I'm not sure if i see them when jackd is not running.. Will test later this evening.

Flo

Aug 10 16:05:50 mango kernel: (swapper/0): 2463us non-preemptible
critical section violated 1000 us preempt threshold starting at
do_IRQ+0x19/0x170 and ending at do_IRQ+0x126/0x170
Aug 10 16:05:50 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:05:50 mango kernel:  [<c0114735>] sub_preempt_count+0x45/0x60
Aug 10 16:05:50 mango kernel:  [<c0107936>] do_IRQ+0x126/0x170
Aug 10 16:05:50 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:05:50 mango kernel:  [<c0103eb9>] cpu_idle+0x39/0x50
Aug 10 16:05:50 mango kernel:  [<c03148c1>] start_kernel+0x1a1/0x1f0
Aug 10 16:05:50 mango kernel:  [<c010019f>] 0xc010019f
Aug 10 16:05:50 mango kernel: ALSA sound/core/pcm_lib.c:169: XRUN:
pcmC0D0p
Aug 10 16:05:50 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:05:50 mango kernel:  [<f08c2431>]
snd_pcm_period_elapsed+0x311/0x480 [snd_pcm]
Aug 10 16:05:50 mango kernel:  [<f089ebce>]
snd_cs46xx_interrupt+0x1be/0x1f0 [snd_cs46xx]
Aug 10 16:05:50 mango kernel:  [<c011b55b>]
generic_handle_IRQ_event+0x3b/0x70
Aug 10 16:05:50 mango kernel:  [<c01078c6>] do_IRQ+0xb6/0x170
Aug 10 16:05:50 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:07:11 mango kernel: ALSA sound/core/pcm_lib.c:169: XRUN:
pcmC0D0c
Aug 10 16:07:11 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:07:11 mango kernel:  [<f08c2431>]
snd_pcm_period_elapsed+0x311/0x480 [snd_pcm]
Aug 10 16:07:11 mango kernel:  [<f089ebce>]
snd_cs46xx_interrupt+0x1be/0x1f0 [snd_cs46xx]
Aug 10 16:07:11 mango kernel:  [<c011b55b>]
generic_handle_IRQ_event+0x3b/0x70
Aug 10 16:07:11 mango kernel:  [<c01078c6>] do_IRQ+0xb6/0x170
Aug 10 16:07:11 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:07:11 mango kernel:  [<c0103eb9>] cpu_idle+0x39/0x50
Aug 10 16:07:11 mango kernel:  [<c03148c1>] start_kernel+0x1a1/0x1f0
Aug 10 16:07:11 mango kernel:  [<c010019f>] 0xc010019f
Aug 10 16:07:11 mango kernel: (swapper/0): 2462us non-preemptible
critical section violated 1000 us preempt threshold starting at
do_IRQ+0x19/0x170 and ending at do_IRQ+0x126/0x170
Aug 10 16:07:11 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:07:11 mango kernel:  [<c0114735>] sub_preempt_count+0x45/0x60
Aug 10 16:07:11 mango kernel:  [<c0107936>] do_IRQ+0x126/0x170
Aug 10 16:07:11 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:07:11 mango kernel:  [<c0103eb9>] cpu_idle+0x39/0x50
Aug 10 16:07:11 mango kernel:  [<c03148c1>] start_kernel+0x1a1/0x1f0
Aug 10 16:07:11 mango kernel:  [<c010019f>] 0xc010019f
Aug 10 16:07:11 mango kernel: ALSA sound/core/pcm_lib.c:169: XRUN:
pcmC0D0p
Aug 10 16:07:11 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:07:11 mango kernel:  [<f08c2431>]
snd_pcm_period_elapsed+0x311/0x480 [snd_pcm]
Aug 10 16:07:11 mango kernel:  [<f089ebce>]
snd_cs46xx_interrupt+0x1be/0x1f0 [snd_cs46xx]
Aug 10 16:07:11 mango kernel:  [<c011b55b>]
generic_handle_IRQ_event+0x3b/0x70
Aug 10 16:07:11 mango kernel:  [<c01078c6>] do_IRQ+0xb6/0x170
Aug 10 16:07:11 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:09:52 mango kernel: ALSA sound/core/pcm_lib.c:169: XRUN:
pcmC0D0p
Aug 10 16:09:52 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:09:52 mango kernel:  [<f08c2431>]
snd_pcm_period_elapsed+0x311/0x480 [snd_pcm]
Aug 10 16:09:52 mango kernel:  [<f089ebce>]
snd_cs46xx_interrupt+0x1be/0x1f0 [snd_cs46xx]
Aug 10 16:09:52 mango kernel:  [<c011b55b>]
generic_handle_IRQ_event+0x3b/0x70
Aug 10 16:09:52 mango kernel:  [<c01078c6>] do_IRQ+0xb6/0x170
Aug 10 16:09:52 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:09:52 mango kernel:  [<c0103eb9>] cpu_idle+0x39/0x50
Aug 10 16:09:52 mango kernel:  [<c03148c1>] start_kernel+0x1a1/0x1f0
Aug 10 16:09:52 mango kernel:  [<c010019f>] 0xc010019f
Aug 10 16:09:52 mango kernel: (swapper/0): 2459us non-preemptible
critical section violated 1000 us preempt threshold starting at
do_IRQ+0x19/0x170 and ending at do_IRQ+0x126/0x170
Aug 10 16:09:52 mango kernel:  [<c01064ae>] dump_stack+0x1e/0x20
Aug 10 16:09:52 mango kernel:  [<c0114735>] sub_preempt_count+0x45/0x60
Aug 10 16:09:52 mango kernel:  [<c0107936>] do_IRQ+0x126/0x170
Aug 10 16:09:52 mango kernel:  [<c0106098>] common_interrupt+0x18/0x20
Aug 10 16:09:52 mango kernel:  [<c0103eb9>] cpu_idle+0x39/0x50
Aug 10 16:09:52 mango kernel:  [<c03148c1>] start_kernel+0x1a1/0x1f0
Aug 10 16:09:52 mango kernel:  [<c010019f>] 0xc010019f

System info:

Linux mango.fruits.de 2.6.8-rc3 #2 Tue Aug 10 12:46:08 CEST 2004 i686 GNU/Linux
-----------
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 7
model name : AMD Duron(tm) Processor
stepping : 1
cpu MHz : 1194.993
cache size : 64 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat
pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips : 2359.29

-----------
/proc/interrupts:
CPU0

  0:    8341883          XT-PIC  timer
  1:      11031          XT-PIC  i8042
  2:          0          XT-PIC  cascade
  5:   10546999          XT-PIC  CS46XX
  8:          4          XT-PIC  rtc
 10:      17097          XT-PIC  eth0
 12:     292978          XT-PIC  i8042
 14:       2675          XT-PIC  ide0
 15:      43327          XT-PIC  ide1
NMI:          0 
ERR:          0

-----------
/proc/irq/10/eth0/threaded:1
/proc/irq/12/i8042/threaded:1
/proc/irq/14/ide0/threaded:1
/proc/irq/15/ide1/threaded:1
/proc/irq/1/i8042/threaded:1
/proc/irq/5/CS46XX/threaded:0
/proc/irq/8/rtc/threaded:0
-----------
/sys/block/hda/queue/max_sectors_kb:16
/sys/block/hdd/queue/max_sectors_kb:16
-----------
voluntary_preemption
3
kernel_preemption
1
-----------
preempt_thresh
1000

Signature · 2 lines
Palimm Palimm!
http://affenbande.org/~tapas/
#423 lore raw

PATCH 1/1 voluntary-preempt-2.6.8-rc2-J4

In reply to Ingo Molnar

i've uploaded the -J4 patch:

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-J4

Changes since -J3:

- make block device max_sectors sysfs tunable. There's a new entry /sys/block/*/queue/max_sectors_kb which stores the current max request size in KB. You can write it to change the size.

Jens: i've attached a standalone patch against 2.6.8-rc2 below. Please apply if it looks good to you. (I've added extra locking to make sure max_sectors and readahead_pages gets updated at once, for the unlikely event of two CPUs updating max_sectors at once.) I've tested this on IDE using the UP kernel.

- include the unlock-tty changes suggested by akpm and tested by Lee Revell. I've also unlocked the tty_read path - only the release/open paths remain under the big kernel lock. Please report any tty related regressions.

Ingo

--
introduce max_sectors_kb tunable.

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c.orig +44 -0
@@ -3062,13 +3063,50 @@ queue_ra_store(struct request_queue *q, 
 	unsigned long ra_kb;
 	ssize_t ret = queue_var_store(&ra_kb, page, count);
 
+	spin_lock_irq(q->queue_lock);
 	if (ra_kb > (q->max_sectors >> 1))
 		ra_kb = (q->max_sectors >> 1);
 
 	q->backing_dev_info.ra_pages = ra_kb >> (PAGE_CACHE_SHIFT - 10);
+	spin_unlock_irq(q->queue_lock);
+
+	return ret;
+}
+
+static ssize_t queue_max_sectors_show(struct request_queue *q, char *page)
+{
+	int max_sectors_kb = q->max_sectors >> 1;
+
+	return queue_var_show(max_sectors_kb, (page));
+}
+
+static ssize_t
+queue_max_sectors_store(struct request_queue *q, const char *page, size_t count)
+{
+	unsigned long max_sectors_kb;
+	ssize_t ret = queue_var_store(&max_sectors_kb, page, count);
+	int ra_kb;
+
+	/*
+	 * Take the queue lock to update the readahead and max_sectors
+	 * values synchronously:
+	 */
+	spin_lock_irq(q->queue_lock);
+	/*
+	 * Trim readahead window as well, if necessary:
+	 */
+	ra_kb = q->backing_dev_info.ra_pages << (PAGE_CACHE_SHIFT - 10);
+	if (ra_kb > max_sectors_kb)
+		q->backing_dev_info.ra_pages =
+				max_sectors_kb >> (PAGE_CACHE_SHIFT - 10);
+
+	q->max_sectors = max_sectors_kb << 1;
+	spin_unlock_irq(q->queue_lock);
+
 	return ret;
 }
 
+
 static struct queue_sysfs_entry queue_requests_entry = {
 	.attr = {.name = "nr_requests", .mode = S_IRUGO | S_IWUSR },
 	.show = queue_requests_show,
@@ -3081,9 +3119,16 @@ static struct queue_sysfs_entry queue_ra
 	.store = queue_ra_store,
 };
 
+static struct queue_sysfs_entry queue_max_sectors_entry = {
+	.attr = {.name = "max_sectors_kb", .mode = S_IRUGO | S_IWUSR },
+	.show = queue_max_sectors_show,
+	.store = queue_max_sectors_store,
+};
+
 static struct attribute *default_attrs[] = {
 	&queue_requests_entry.attr,
 	&queue_ra_entry.attr,
+	&queue_max_sectors_entry.attr,
 	NULL,
 };
 
#424 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J4

Quoted 7 lines

Changes since -J3:

- make block device max_sectors sysfs tunable. There's a new entry /sys/block/*/queue/max_sectors_kb which stores the current max request size in KB. You can write it to change the size.

Quoted 5 lines

Jens: i've attached a standalone patch against 2.6.8-rc2 below. Please apply if it looks good to you. (I've added extra locking to make sure max_sectors and readahead_pages gets updated at once, for the unlikely event of two CPUs updating max_sectors at once.) I've tested this on IDE using the UP kernel.

i've refined the patch (new version attached below): drivers use blk_queue_max_sectors() to set the maximum # of sectors that the driver or hw can handle.

so i've introduced a new queue entry called max_hw_sectors, and the new /sys entry listens to this maximum and only updates max_sectors. This entry is also exported to /sys as a readonly entry. (so that users can see the maximum the driver supports.)

i've added another fix as well: do not allow users to set a max_sectors value lower than the pagecache page size.

these changes should make the new tunable pretty fool-proof.

Ingo

include/linux/blkdev.h renamed from include/linux/blkdev.h.orig +1 -0
@@ -338,6 +338,7 @@ struct request_queue
 	unsigned int		nr_congestion_off;
 
 	unsigned short		max_sectors;
+	unsigned short		max_hw_sectors;
 	unsigned short		max_phys_segments;
 	unsigned short		max_hw_segments;
 	unsigned short		hardsect_size;
drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c.orig +62 -1

91 lines not on this page · raw message

#425 lore raw

no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

Quoted 8 lines
Quoted 7 lines

Changes since -J3:

- make block device max_sectors sysfs tunable. There's a new entry /sys/block/*/queue/max_sectors_kb which stores the current max request size in KB. You can write it to change the size.

[...]

Quoted 8 lines

i've refined the patch (new version attached below): drivers use blk_queue_max_sectors() to set the maximum # of sectors that the driver or hw can handle.

so i've introduced a new queue entry called max_hw_sectors, and the new /sys entry listens to this maximum and only updates max_sectors. This entry is also exported to /sys as a readonly entry. (so that users can see the maximum the driver supports.)

Hi there.

I do not seem to have success with tuning the max_sectors_kb value.

After setting it to 32 (the hw max is 128), userland programs fail with I/O errors. Setting it back to 128 gets it back to working, sort of. The errors probably get bufferred somewhere.

During the "bad" setting (32), messages like this one show up in kernel log.

bio too big device hda3 (104 > 64)

I am using J7 voluntary_preemption patch (set at 2:1), cfq io scheduler, via82cxxx IDE driver. I will gladly provide further details.

Rudo.

#426 lore raw

Re: no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

Quoted 7 lines · Rudo Thomas

After setting it to 32 (the hw max is 128), userland programs fail with I/O errors. Setting it back to 128 gets it back to working, sort of. The errors probably get bufferred somewhere.

During the "bad" setting (32), messages like this one show up in kernel log.

bio too big device hda3 (104 > 64)

does the patch below, ontop of -J7, help?

Ingo

block/ll_rw_blk.c renamed from block/ll_rw_blk.c.orig +2 -2
@@ -2447,11 +2447,11 @@ end_io:
 			break;
 		}
 
-		if (unlikely(bio_sectors(bio) > q->max_sectors)) {
+		if (unlikely(bio_sectors(bio) > q->max_hw_sectors)) {
 			printk("bio too big device %s (%u > %u)\n", 
 				bdevname(bio->bi_bdev, b),
 				bio_sectors(bio),
-				q->max_sectors);
+				q->max_hw_sectors);
 			goto end_io;
 		}
 
#427 lore raw

Re: no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

Quoted 1 line · Ingo Molnar

does the patch below, ontop of -J7, help?

i've added this patch to -J7, so re-downloading the -J7 patch should give you the patch too.

Ingo

#428 lore raw

Re: no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

In reply to Ingo Molnar
Quoted 9 lines
Quoted 7 lines

After setting it to 32 (the hw max is 128), userland programs fail with I/O errors. Setting it back to 128 gets it back to working, sort of. The errors probably get bufferred somewhere.

During the "bad" setting (32), messages like this one show up in kernel log.

bio too big device hda3 (104 > 64)

does the patch below, ontop of -J7, help?

I tried it, but did not test tuning the value, as the patch has problems of its
own: with device-mapper.

bio too big device dm-0 (256 > 255)

Again, some files cannot be read on the device.

Bye for now.

Rudo.

#429 lore raw

Re: no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

Quoted 8 lines · Rudo Thomas
Quoted 1 line

does the patch below, ontop of -J7, help?

I tried it, but did not test tuning the value, as the patch has problems of its own: with device-mapper.

bio too big device dm-0 (256 > 255)

Again, some files cannot be read on the device.

ok, dm (and some other layered block drivers) set q->max_sectors directly instead of using blk_queue_max_sectors().

Ingo

#430 lore raw

Re: no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

Quoted 2 lines · Ingo Molnar

ok, dm (and some other layered block drivers) set q->max_sectors directly instead of using blk_queue_max_sectors().

does the patch below fix your DM problems?

Ingo

drivers/md/dm-table.c renamed from drivers/md/dm-table.c.orig +1 -1
@@ -825,7 +825,7 @@ void dm_table_set_restrictions(struct dm
 	 * Make sure we obey the optimistic sub devices
 	 * restrictions.
 	 */
-	q->max_sectors = t->limits.max_sectors;
+	blk_queue_max_sectors(q, t->limits.max_sectors);
 	q->max_phys_segments = t->limits.max_phys_segments;
 	q->max_hw_segments = t->limits.max_hw_segments;
 	q->hardsect_size = t->limits.hardsect_size;
#431 lore raw

Re: no luck with max_sectors_kb (Re: voluntary-preempt-2.6.8-rc2-J4)

Quoted 4 lines
Quoted 2 lines

ok, dm (and some other layered block drivers) set q->max_sectors directly instead of using blk_queue_max_sectors().

does the patch below fix your DM problems?

Indeed it does.

Rudo.

#432 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J4

In reply to Ingo Molnar
Quoted 88 lines · Ingo Molnar

i've uploaded the -J4 patch:

   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-J4

Changes since -J3:

- make block device max_sectors sysfs tunable. There's a new entry /sys/block/*/queue/max_sectors_kb which stores the current max request size in KB. You can write it to change the size.

Jens: i've attached a standalone patch against 2.6.8-rc2 below. Please apply if it looks good to you. (I've added extra locking to make sure max_sectors and readahead_pages gets updated at once, for the unlikely event of two CPUs updating max_sectors at once.) I've tested this on IDE using the UP kernel.

- include the unlock-tty changes suggested by akpm and tested by Lee Revell. I've also unlocked the tty_read path - only the release/open paths remain under the big kernel lock. Please report any tty related regressions.

Ingo

--
introduce max_sectors_kb tunable.

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c.orig +39 -0
@@ -3062,13 +3063,50 @@ queue_ra_store(struct request_queue *q, 
 	unsigned long ra_kb;
 	ssize_t ret = queue_var_store(&ra_kb, page, count);
 
+	spin_lock_irq(q->queue_lock);
 	if (ra_kb > (q->max_sectors >> 1))
 		ra_kb = (q->max_sectors >> 1);
 
 	q->backing_dev_info.ra_pages = ra_kb >> (PAGE_CACHE_SHIFT - 10);
+	spin_unlock_irq(q->queue_lock);
+
+	return ret;
+}
+
+static ssize_t queue_max_sectors_show(struct request_queue *q, char *page)
+{
+	int max_sectors_kb = q->max_sectors >> 1;
+
+	return queue_var_show(max_sectors_kb, (page));
+}
+
+static ssize_t
+queue_max_sectors_store(struct request_queue *q, const char *page, size_t count)
+{
+	unsigned long max_sectors_kb;
+	ssize_t ret = queue_var_store(&max_sectors_kb, page, count);
+	int ra_kb;
+
+	/*
+	 * Take the queue lock to update the readahead and max_sectors
+	 * values synchronously:
+	 */
+	spin_lock_irq(q->queue_lock);
+	/*
+	 * Trim readahead window as well, if necessary:
+	 */
+	ra_kb = q->backing_dev_info.ra_pages << (PAGE_CACHE_SHIFT - 10);
+	if (ra_kb > max_sectors_kb)
+		q->backing_dev_info.ra_pages =
+				max_sectors_kb >> (PAGE_CACHE_SHIFT - 10);
+
+	q->max_sectors = max_sectors_kb << 1;
+	spin_unlock_irq(q->queue_lock);
+
 	return ret;
 }
 
+
 static struct queue_sysfs_entry queue_requests_entry = {
 	.attr = {.name = "nr_requests", .mode = S_IRUGO | S_IWUSR },
 	.show = queue_requests_show,
@@ -3081,9 +3119,16 @@ static struct queue_sysfs_entry queue_ra
 	.store = queue_ra_store,
 };
 
+static struct queue_sysfs_entry queue_max_sectors_entry = {
+	.attr = {.name = "max_sectors_kb", .mode = S_IRUGO | S_IWUSR },

drivers/block/ll_rw_blk.c static struct queue_sysfs_entry queue_ra · line 3133

Quoted 10 lines
drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c.orig +5 -0
+	.show = queue_max_sectors_show,
+	.store = queue_max_sectors_store,
+};
+
 static struct attribute *default_attrs[] = {
 	&queue_requests_entry.attr,
 	&queue_ra_entry.attr,
+	&queue_max_sectors_entry.attr,
 	NULL,
 };

I don't like it. First of all, the implementation really should drain the queue first, then set max value before allowing people to queue more io. The queue lock doesn't help here, readers don't even attempt to serialize access to max_sectors.

Secondly, I don't like the concept of exposing this value. If you want to do something like this, we must split the value into two like proposed (and patched) some months ago into a hardware and user value.

I don't see why we can't just drop ata48 default value to 256kb instead. There's very little command over head on ide, I bet the majority of the change in performance when playing with 256kb vs 1024kb is not the command overhead itself, rather things like read-ahead that could be more intelligent.

Signature · 1 line
Jens Axboe
#433 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J4

[i've sent a second patch too since the first version.]

Quoted 4 lines · Jens Axboe

I don't like it. First of all, the implementation really should drain the queue first, then set max value before allowing people to queue more io. The queue lock doesn't help here, readers don't even attempt to serialize access to max_sectors.

why should the queue be drained? There might be a few leftover big requests, but these are not a problem.

Quoted 3 lines

Secondly, I don't like the concept of exposing this value. If you want to do something like this, we must split the value into two like proposed (and patched) some months ago into a hardware and user value.

yes, agreed - that's what the second patch does.

Quoted 5 lines

I don't see why we can't just drop ata48 default value to 256kb instead. There's very little command over head on ide, I bet the majority of the change in performance when playing with 256kb vs 1024kb is not the command overhead itself, rather things like read-ahead that could be more intelligent.

256kb isnt enough from a latency POV either - and if a user wants some extreme setting like 16KB per request why not allow it? Especially since these tunables cause zero runtime overhead.

Ingo

#435 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J4

In reply to Ingo Molnar
Quoted 5 lines · Ingo Molnar

[i've sent a second patch too since the first version.]

* Jens Axboe <[email protected]> wrote:

Quoted 7 lines
Quoted 4 lines

I don't like it. First of all, the implementation really should drain the queue first, then set max value before allowing people to queue more io. The queue lock doesn't help here, readers don't even attempt to serialize access to max_sectors.

why should the queue be drained? There might be a few leftover big requests, but these are not a problem.

The patch I commented on did not split sectors value into two, in that case it's a big problem. When split and expose only the one, it's not an issue.

Quoted 5 lines
Quoted 3 lines

Secondly, I don't like the concept of exposing this value. If you want to do something like this, we must split the value into two like proposed (and patched) some months ago into a hardware and user value.

yes, agreed - that's what the second patch does.

Great.

Quoted 9 lines
Quoted 5 lines

I don't see why we can't just drop ata48 default value to 256kb instead. There's very little command over head on ide, I bet the majority of the change in performance when playing with 256kb vs 1024kb is not the command overhead itself, rather things like read-ahead that could be more intelligent.

256kb isnt enough from a latency POV either - and if a user wants some extreme setting like 16KB per request why not allow it? Especially since these tunables cause zero runtime overhead.

Note that I've been travelling and didn't really track the thread closely before that, but it seems to me that most of the latency incurred is an artifact of long code paths hanging off bi_end_io. Maybe it would be better to try and fix some of that and get both good irq completion latencies, while still maintaining decent drive throughput.

That said, this second version looks better from an inclusion point of view. Better fix those manual q->max_sectors settings first.

Signature · 1 line
Jens Axboe
#436 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J4

Quoted 2 lines · Jens Axboe

That said, this second version looks better from an inclusion point of view. Better fix those manual q->max_sectors settings first.

ok - the third patch i sent is supposed to deal with all those cases as well.

Ingo

#437 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

In reply to Ingo Molnar
Quoted 2 lines · Ingo Molnar

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-J3

The bigger this thing gets, the more worried I get. Sometime this is going to need to be split up into individual fixes, and they need to be based upon an overall approach which we haven't yet settled on.

In particular your whole approach (with voluntary_need_resched()) doesn't work on SMP.

The approach I'm using is to unconditionally drop locks on every Nth pass around the loop to allow another CPU to grab the lock, do some work, drop the lock, then be preempted. eg:

@@ -773,6 +774,12 @@ int get_user_pages(struct task_struct *t
struct page *map = NULL; int lookup_write = write;

+ if ((++nr_pages & 63) == 0) {
+ spin_unlock(&mm->page_table_lock);
+ cpu_relax();
+ spin_lock(&mm->page_table_lock);
+ }
+
/*
* We don't follow pagetables for VM_IO regions - they

This is a bit nasty, because it assumes that the other CPU will be able to get in there and acquire the lock which is in a different cpu's cache. Long transfer latencies may defeat this.

For voluntary preempt the above will need a cond_resched added to it.

So we need to come up with a suitable approach to covering voluntary and involuntary preemption on both UP and SMP and set up the header file infrastructure to support it all. After that we need to redo all the points-of-high-latency whcih have thus far been discovered to use that infrastructure.

#438 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

Quoted 3 lines · Andrew Morton

The bigger this thing gets, the more worried I get. Sometime this is going to need to be split up into individual fixes, and they need to be based upon an overall approach which we haven't yet settled on.

i will do that splitup. Right now i'm simply mapping how widespread the problem is and what type of fixes we need. The situation isnt all that bad but we might need (an optional) mechanism to make softirqs synchronous. All of this stuff is nicely modular and i'll do a splitup post 2.6.8 (i dont think we want to disturb 2.6.8 with any of this).

Quoted 2 lines

In particular your whole approach (with voluntary_need_resched()) doesn't work on SMP.

(for the record, voluntary_need_resched() == need_resched() - i'm only keeping a distinction between the two to be able to track progress and regressions between the vanilla and modified kernels.)

need_resched() indeed doesnt do a lock-break for SMP purposes.

Quoted 5 lines

The approach I'm using is to unconditionally drop locks on every Nth pass around the loop to allow another CPU to grab the lock, do some work, drop the lock, then be preempted. eg:

Review int get_user_pages(struct task_struct *t · line 782

Quoted 9 lines
(quoted diff) +6 -0
@@ -773,6 +774,12 @@ int get_user_pages(struct task_struct *t
 			struct page *map = NULL;
 			int lookup_write = write;
 
+			if ((++nr_pages & 63) == 0) {
+				spin_unlock(&mm->page_table_lock);
+				cpu_relax();
+				spin_lock(&mm->page_table_lock);
+			}
+

guaranteeing latencies on SMP is hard, because the latency of a CPU might depend on the latency of a task on another CPU - which CPU isnt notified of the rescheduling request.

one alternative technique to yours would be to notify _all_ CPUs that a high-prio RT task is ready to run (via a broadcast need-resched). That way the UP latency-break techniques map to SMP in a 1:1 way.

non-RT tasks dont get this benefit, which is a difference to the UP situation, but i dont think it would be appropriate to use the UP behavior, due to the overhead of broadcasting.

a combination of the two techniques could be used too: a global 'break locks from now on' flag which gets set if a (RT?) task wants to reschedule. Normally this flag would be zero and the cacheline would be clean and shared between all CPUs, causing no overhead. Once a task wants to reschedule it would increase by 1 until that task has been scheduled. This would remove the ugliness factor too, your sample code would look:

Review int get_user_pages(struct task_struct *t

Quoted 6 lines
(quoted diff) +6 -0
+			if (need_lock_break()) {
+				spin_unlock(&mm->page_table_lock);
+				cpu_relax();
+				spin_lock(&mm->page_table_lock);
+			}
+

or a shortcut:

Review

Quoted 1 line
(quoted diff) +1 -0
+			cond_lock_break(&mm->page_table_lock);

there are two problems with the unconditional lock-break:

- i'm not sure cpu_relax() guarantees that another CPU can grab this lock. It all depends on whether the lock cacheline can bounce over to the other CPU faster than cpu_relax() finishes and this CPU re-acquires the lock.

- the latencies will still be higher than on UP, in a hard to predict way. Every time the codepath encounters a spinlock it has to take in order to exit for a reschedule, the latency gets re-added. Also, the now scheduled high-prio task will encounter the latency every time it acquires a spinlock - while on UP it has the CPU for itself with no interaction from other CPUs.

Ingo

#439 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

Quoted 9 lines · Ingo Molnar

...
guaranteeing latencies on SMP is hard, because the latency of a CPU might depend on the latency of a task on another CPU - which CPU isnt notified of the rescheduling request.

one alternative technique to yours would be to notify _all_ CPUs that a high-prio RT task is ready to run (via a broadcast need-resched). That way the UP latency-break techniques map to SMP in a 1:1 way.

I did exactly that in the 2.4 ll patch. It works, but doesn't seem very nice.

Quoted 10 lines

non-RT tasks dont get this benefit, which is a difference to the UP situation, but i dont think it would be appropriate to use the UP behavior, due to the overhead of broadcasting.

a combination of the two techniques could be used too: a global 'break locks from now on' flag which gets set if a (RT?) task wants to reschedule. Normally this flag would be zero and the cacheline would be clean and shared between all CPUs, causing no overhead. Once a task wants to reschedule it would increase by 1 until that task has been scheduled.

Something like that. The obscure livelock opportunities worry me.

Quoted 12 lines

This would remove the ugliness factor too, your sample code would look:

Review

Quoted 6 lines
(quoted diff) +6 -0
+			if (need_lock_break()) {
+				spin_unlock(&mm->page_table_lock);
+				cpu_relax();
+				spin_lock(&mm->page_table_lock);
+			}
+

or a shortcut:

Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+			cond_lock_break(&mm->page_table_lock);

The usual problem is, of course, long-running loops under locks. These loops usually have some sort of counter (or you add one). The below patch changes cond_resched_lock() so that it takes the counter, and the tuned upper-loop-count limit. It also does the right thing on UP. It needs enhancements for the situations where we already have a counter in the loop.

I dunno if it's sufficient, but I do think that we need to settle on some infrastructure of this sort before proceeding too far into this thing.

Quoted 6 lines

there are two problems with the unconditional lock-break:

- i'm not sure cpu_relax() guarantees that another CPU can grab this lock. It all depends on whether the lock cacheline can bounce over to the other CPU faster than cpu_relax() finishes and this CPU re-acquires the lock.

precisely.

include/linux/sched.h renamed from include/linux/sched.h~cond_resched_lock-smp-fix +22 -12
diff -puN include/linux/sched.h~cond_resched_lock-smp-fix include/linux/sched.h
@@ -1038,22 +1038,32 @@ static inline void cond_resched(void)
 }
 
 /*
- * cond_resched_lock() - if a reschedule is pending, drop the given lock,
- * call schedule, and on return reacquire the lock.
- *
  * This works OK both with and without CONFIG_PREEMPT.  We do strange low-level
  * operations here to prevent schedule() from being called twice (once via
  * spin_unlock(), once by hand).
  */
-static inline void cond_resched_lock(spinlock_t * lock)
-{
-	if (need_resched()) {
-		_raw_spin_unlock(lock);
-		preempt_enable_no_resched();
-		__cond_resched();
-		spin_lock(lock);
-	}
-}
+
+#ifdef CONFIG_SMP
+#define cond_resched_lock(lock, counter, limit)		\
+	do {						\
+		if (++(counter) >= limit) {		\
+			spin_unlock(lock);		\
+			cpu_relax();			\
+			spin_lock(lock);		\
+		}					\
+		(counter) = 0;				\
+	} while (0)
+#else
+#define cond_resched_lock(lock, counter, limit)		\
+	do {						\
+		if (need_resched()) {			\
+			_raw_spin_unlock(lock);		\
+			preempt_enable_no_resched();	\
+			__cond_resched();		\
+			spin_lock(lock);		\
+		}					\
+	} while (0)
+#endif
 
 /* Reevaluate whether the task has signals pending delivery.
    This is required every time the blocked sigset_t changes.
fs/reiserfs/journal.c renamed from fs/reiserfs/journal.c~cond_resched_lock-smp-fix +3 -2
diff -puN fs/reiserfs/journal.c~cond_resched_lock-smp-fix fs/reiserfs/journal.c
@@ -796,6 +796,7 @@ static int write_ordered_buffers(spinloc
     struct buffer_chunk chunk;
     struct list_head tmp;
     INIT_LIST_HEAD(&tmp);
+    int resched_counter = 0;
 
     chunk.nr = 0;
     spin_lock(lock);
@@ -827,7 +828,7 @@ static int write_ordered_buffers(spinloc
 	}
 loop_next:
 	put_bh(bh);
-	cond_resched_lock(lock);
+	cond_resched_lock(lock, resched_counter, 64);
     }
     if (chunk.nr) {
 	spin_unlock(lock);
@@ -848,7 +849,7 @@ loop_next:
 	if (!buffer_uptodate(bh))
 	    ret = -EIO;
 	put_bh(bh);
-	cond_resched_lock(lock);
+	cond_resched_lock(lock, resched_counter, 64);
     }
     spin_unlock(lock);
     return ret;
mm/shmem.c renamed from mm/shmem.c~cond_resched_lock-smp-fix +2 -1
diff -puN mm/shmem.c~cond_resched_lock-smp-fix mm/shmem.c
@@ -424,6 +424,7 @@ static void shmem_truncate(struct inode 
 	swp_entry_t *ptr;
 	int offset;
 	int freed;
+	int resched_counter = 0;
 
 	inode->i_ctime = inode->i_mtime = CURRENT_TIME;
 	idx = (inode->i_size + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
@@ -496,7 +497,7 @@ static void shmem_truncate(struct inode 
 				shmem_free_block(inode);
 			}
 			empty = subdir;
-			cond_resched_lock(&info->lock);
+			cond_resched_lock(&info->lock, resched_counter, 32);
 			dir = shmem_dir_map(subdir);
 		}
 		subdir = *dir;
mm/memory.c renamed from mm/memory.c~cond_resched_lock-smp-fix +7 -3
diff -puN mm/memory.c~cond_resched_lock-smp-fix mm/memory.c
@@ -224,6 +224,7 @@ int copy_page_range(struct mm_struct *ds
 	unsigned long address = vma->vm_start;
 	unsigned long end = vma->vm_end;
 	unsigned long cow;
+	int resched_counter = 0;
 
 	if (is_vm_hugetlb_page(vma))
 		return copy_hugetlb_page_range(dst, src, vma);
@@ -341,7 +342,8 @@ cont_copy_pte_range_noset:
 			pte_unmap_nested(src_pte-1);
 			pte_unmap(dst_pte-1);
 			spin_unlock(&src->page_table_lock);
-			cond_resched_lock(&dst->page_table_lock);
+			cond_resched_lock(&dst->page_table_lock,
+						resched_counter, 256);
 cont_copy_pmd_range:
 			src_pmd++;
 			dst_pmd++;
@@ -530,6 +532,7 @@ int unmap_vmas(struct mmu_gather **tlbp,
 	int tlb_start_valid = 0;
 	int ret = 0;
 	int atomic = details && details->atomic;
+	int resched_counter = 0;
 
 	for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) {
 		unsigned long start;
@@ -567,10 +570,11 @@ int unmap_vmas(struct mmu_gather **tlbp,
 			zap_bytes -= block;
 			if ((long)zap_bytes > 0)
 				continue;
-			if (!atomic && need_resched()) {
+			if (!atomic) {
 				int fullmm = tlb_is_full_mm(*tlbp);
 				tlb_finish_mmu(*tlbp, tlb_start, start);
-				cond_resched_lock(&mm->page_table_lock);
+				cond_resched_lock(&mm->page_table_lock,
+						resched_counter, 1);
 				*tlbp = tlb_gather_mmu(mm, fullmm);
 				tlb_start_valid = 0;
 			}
mm/rmap.c renamed from mm/rmap.c~cond_resched_lock-smp-fix +3 -2
diff -puN mm/rmap.c~cond_resched_lock-smp-fix mm/rmap.c
@@ -661,6 +661,7 @@ static inline int try_to_unmap_file(stru
 	unsigned long max_nl_cursor = 0;
 	unsigned long max_nl_size = 0;
 	unsigned int mapcount;
+	int resched_counter = 0;
 
 	if (!spin_trylock(&mapping->i_mmap_lock))
 		return ret;
@@ -699,7 +700,7 @@ static inline int try_to_unmap_file(stru
 	 */
 	mapcount = page->mapcount;
 	page_map_unlock(page);
-	cond_resched_lock(&mapping->i_mmap_lock);
+	cond_resched_lock(&mapping->i_mmap_lock, resched_counter, 1);
 
 	max_nl_size = (max_nl_size + CLUSTER_SIZE - 1) & CLUSTER_MASK;
 	if (max_nl_cursor == 0)
@@ -728,7 +729,7 @@ static inline int try_to_unmap_file(stru
 					(void *) max_nl_cursor;
 			ret = SWAP_AGAIN;
 		}
-		cond_resched_lock(&mapping->i_mmap_lock);
+		cond_resched_lock(&mapping->i_mmap_lock, resched_counter, 8);
 		max_nl_cursor += CLUSTER_SIZE;
 	} while (max_nl_cursor <= max_nl_size);
 

_

A message that is not in the archive (sent off-list or not yet received): [email protected]

#441 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"AM" == Andrew Morton <[email protected]> writes:

AM> +#ifdef CONFIG_SMP
AM> +#define cond_resched_lock(lock, counter, limit) \
AM> + do { \
AM> + if (++(counter) >= limit) { \
AM> + spin_unlock(lock); \
AM> + cpu_relax(); \
AM> + spin_lock(lock); \
AM> + } \
AM> + (counter) = 0; \
AM> + } while (0)
AM> +#else

I am wondering if you meant to reset the counter to zero inside of the if(){}, probably after reaquiring the lock...

#442 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar
Quoted 3 lines · Andrew Morton

The bigger this thing gets, the more worried I get. Sometime this is going to need to be split up into individual fixes, and they need to be based upon an overall approach which we haven't yet settled on.

Quoted 8 lines

i will do that splitup. Right now i'm simply mapping how widespread the problem is and what type of fixes we need. The situation isnt all that bad but we might need (an optional) mechanism to make softirqs synchronous. All of this stuff is nicely modular and i'll do a splitup post 2.6.8 (i dont think we want to disturb 2.6.8 with any of this).

From a user's perspective, and based on my own testing, I do not see the

patch having to get much bigger, the vast majority of the hot spots have been fixed, and the patch remains quite comprehensible even to someone with a sketchy knowledge of the kernel. There is only one issue I can think of that has not been addressed at all (the PS/2 Caps Lock issue). All that seems to remain are tweaks to individual fixes that are already in the patch.

Lee

#443 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

one alternative technique to yours would be to notify _all_ CPUs that a high-prio RT task is ready to run (via a broadcast need-resched). That way the UP latency-break techniques map to SMP in a 1:1 way.

Quoted 8 lines

non-RT tasks dont get this benefit, which is a difference to the UP situation, but i dont think it would be appropriate to use the UP behavior, due to the overhead of broadcasting.

a combination of the two techniques could be used too: a global 'break locks from now on' flag which gets set if a (RT?) task wants to reschedule. Normally this flag would be zero and the cacheline would be clean and shared between all CPUs, causing no overhead. Once a task

It might be possible to eliminate the need_resched flag. Here is an article that talk about a event-driven preemption model.

Quoting :

Over the long term, MontaVista is investigating whether preemption locks can be eliminated (or at least greatly reduced in number) by protecting all the short-duration critical regions with spinlocks that also disable interrupts on the local CPU, and the long-duration critical regions with mutex locks. ("Long duration" means much greater than the time taken by two context switches.) This will reduce the overhead of the preemptable kernel, since there will no longer be any need to test for preemption ("polling for preemption") at the end of a preemption-locked region (which could happen tens of thousands of times per second on an average system). Instead, preemption would happen automatically as part of the interrupt servicing that causes a higher-priority process to become runnable ("event-driven preemption"). Typically, this only happens a few times to a few tens of times per second with an average system workload, making the "event-driven preemption" model much more efficient than the "polling for preemption" model. This method also has an added efficiency in that the system will take advantage of the cache disruption caused by the interrupt (which is unavoidable) to continue with the preemption.

Reference:

http://www.linuxdevices.com/cgi-bin/printerfriendly.cgi?id=AT4185744181

Best regards,

Eric St-Laurent

#444 lore raw

Re: [patch] voluntary-preempt-2.6.8-rc2-J3

Quoted 10 lines · Eric St-Laurent

It might be possible to eliminate the need_resched flag. Here is an article that talk about a event-driven preemption model.

Quoting :

Over the long term, MontaVista is investigating whether preemption locks can be eliminated (or at least greatly reduced in number) by protecting all the short-duration critical regions with spinlocks that also disable interrupts on the local CPU, and the long-duration critical regions with mutex locks.

This is a pretty old article, from 2000, describing the preemption model they implemented on 2.4. I believe the above paragraph describes the advantages of their model vs. the approach taken by the old 2.4 low latency patches.

My understanding is that the preemptible 2.6 kernel already works this way.

Lee

#445 lore raw
In reply to William Lee Irwin III
Quoted 10 lines · William Lee Irwin III

This patch uses the preemption counter increments and decrements to time non-preemptible critical sections.

This is an instrumentation patch intended to help determine the causes of scheduling latency related to long non-preemptible critical sections.

Changes from 2.6.7-based patch:
(1) fix unmap_vmas() check correctly this time
(2) add touch_preempt_timing() to cond_resched_lock()
(3) depend on preempt until it's worked out wtf goes wrong without it

You preemption-block hold times will improve *enormously* if you move all softirq processing down to the daemon (and possibly raise the daemon to one of the higher SCHED_RR priorities, to compensate for softirq processing no longer happening at interrupt level).

Joe

#446 lore raw
Quoted 8 lines · William Lee Irwin III
Quoted 8 lines

This patch uses the preemption counter increments and decrements to time non-preemptible critical sections.
This is an instrumentation patch intended to help determine the causes of scheduling latency related to long non-preemptible critical sections. Changes from 2.6.7-based patch:
(1) fix unmap_vmas() check correctly this time
(2) add touch_preempt_timing() to cond_resched_lock()
(3) depend on preempt until it's worked out wtf goes wrong without it

Quoted 4 lines · Joe Korty

You preemption-block hold times will improve *enormously* if you move all softirq processing down to the daemon (and possibly raise the daemon to one of the higher SCHED_RR priorities, to compensate for softirq processing no longer happening at interrupt level).

Plausible. Got a patch?

-- wli

#447 lore raw
Quoted 14 lines · William Lee Irwin III
Quoted 8 lines · William Lee Irwin III
Quoted 8 lines

This patch uses the preemption counter increments and decrements to time non-preemptible critical sections.
This is an instrumentation patch intended to help determine the causes of scheduling latency related to long non-preemptible critical sections. Changes from 2.6.7-based patch:
(1) fix unmap_vmas() check correctly this time
(2) add touch_preempt_timing() to cond_resched_lock()
(3) depend on preempt until it's worked out wtf goes wrong without it

On Tue, Jul 13, 2004 at 10:36:00AM -0400, Joe Korty wrote:

Quoted 8 lines
Quoted 4 lines

You preemption-block hold times will improve *enormously* if you move all softirq processing down to the daemon (and possibly raise the daemon to one of the higher SCHED_RR priorities, to compensate for softirq processing no longer happening at interrupt level).

Plausible. Got a patch?

Make MAX_SOFTIRQ_RESTART 1?

I don't think you should make ksoftirq a realtime task, because that defeats the purpose of having it to prevent livelocking userspace doesn't it?

However, you may want to increase it from nice +19. Probably just to nice 0 would be an idea.

#448 lore raw
In reply to William Lee Irwin III
Quoted 17 lines · William Lee Irwin III
Quoted 10 lines · William Lee Irwin III
Quoted 10 lines

This patch uses the preemption counter increments and decrements to time non-preemptible critical sections.
This is an instrumentation patch intended to help determine the causes of scheduling latency related to long non-preemptible critical sections. Changes from 2.6.7-based patch:
(1) fix unmap_vmas() check correctly this time
(2) add touch_preempt_timing() to cond_resched_lock()
(3) depend on preempt until it's worked out wtf goes wrong without it

On Tue, Jul 13, 2004 at 10:36:00AM -0400, Joe Korty wrote:

Quoted 10 lines
Quoted 6 lines

You preemption-block hold times will improve *enormously* if you move all softirq processing down to the daemon (and possibly raise the daemon to one of the higher SCHED_RR priorities, to compensate for softirq processing no longer happening at interrupt level).

Plausible. Got a patch?

How about this? It moves softirqs to a thread but keeps the old spill thread.

This is a patch against 2.6.6 written primarily by Scott Wood.

Signature · 1 line
 Anyone who quotes me in their sig is an idiot. -- Rusty Russell's sig
include/asm-alpha/hardirq.h +11 -7
@@ -86,14 +86,18 @@
 #define in_atomic()	(preempt_count() != 0)
 #define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 # endif
-#define irq_exit()						\
-do {								\
-		preempt_count() -= IRQ_EXIT_OFFSET;		\
-		if (!in_interrupt() &&				\
-		    softirq_pending(smp_processor_id()))	\
-			do_softirq();				\
-		preempt_enable_no_resched();			\
+
+#ifndef CONFIG_SOFTIRQ_THREADS
+#define irq_exit()							\
+do {									\
+		preempt_count() -= IRQ_EXIT_OFFSET;			\
+		if (!in_interrupt() && softirq_pending(smp_processor_id())) \
+			do_softirq();					\
+		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-arm/hardirq.h +4 -0
@@ -81,6 +81,7 @@
 
 extern asmlinkage void __do_softirq(void);
 
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 	do {								\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -88,6 +89,9 @@
 			__do_softirq();					\
 		preempt_enable_no_resched();				\
 	} while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #define synchronize_irq(irq)	barrier()
 #else
include/asm-arm26/hardirq.h +5 -0
@@ -76,6 +76,8 @@
 #endif
 
 #ifndef CONFIG_SMP
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 	do {								\
 		preempt_count() -= HARDIRQ_OFFSET;			\
@@ -83,6 +85,9 @@
 			__asm__("bl%? __do_softirq": : : "lr");/* out of line */\
 		preempt_enable_no_resched();				\
 	} while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #define synchronize_irq(irq)	barrier()
 #else
include/asm-cris/hardirq.h +6 -0
@@ -84,6 +84,9 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -91,6 +94,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #define synchronize_irq(irq)	barrier()
 
include/asm-h8300/hardirq.h +5 -0
@@ -82,6 +82,8 @@
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
 
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -89,6 +91,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-i386/hardirq.h +5 -0
@@ -84,6 +84,8 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -91,6 +93,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-m68k/hardirq.h +5 -0
@@ -78,6 +78,8 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -85,6 +87,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #define synchronize_irq(irq)	barrier()
 
include/asm-m68knommu/hardirq.h +4 -0
@@ -86,6 +86,7 @@
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
 
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -93,6 +94,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-mips/hardirq.h +11 -6
@@ -87,13 +87,18 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
-#define irq_exit()                                                     \
-do {                                                                   \
-	preempt_count() -= IRQ_EXIT_OFFSET;                     \
-	if (!in_interrupt() && softirq_pending(smp_processor_id())) \
-		do_softirq();                                   \
-	preempt_enable_no_resched();                            \
+
+#ifndef CONFIG_SOFTIRQ_THREADS
+#define irq_exit()							\
+do {									\
+	preempt_count() -= IRQ_EXIT_OFFSET;				\
+	if (!in_interrupt() && softirq_pending(smp_processor_id()))	\
+		do_softirq();						\
+	preempt_enable_no_resched();					\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-parisc/hardirq.h +10 -6
@@ -96,13 +96,17 @@
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
 
-#define irq_exit()								\
-do {										\
-		preempt_count() -= IRQ_EXIT_OFFSET;				\
-		if (!in_interrupt() && softirq_pending(smp_processor_id()))	\
-			do_softirq();						\
-		preempt_enable_no_resched();					\
+#ifndef CONFIG_SOFTIRQ_THREADS
+#define irq_exit()							\
+do {									\
+		preempt_count() -= IRQ_EXIT_OFFSET;			\
+		if (!in_interrupt() && softirq_pending(smp_processor_id())) \
+			do_softirq();					\
+		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifdef CONFIG_SMP
   extern void synchronize_irq (unsigned int irq);
include/asm-ppc/hardirq.h +6 -1
@@ -88,13 +88,18 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
-do {									\
+do {                                                                    \
 	preempt_count() -= IRQ_EXIT_OFFSET;				\
 	if (!in_interrupt() && softirq_pending(smp_processor_id()))	\
 		do_softirq();						\
 	preempt_enable_no_resched();					\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-ppc64/hardirq.h +5 -0
@@ -87,6 +87,8 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -94,6 +96,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-sh/hardirq.h +5 -0
@@ -81,6 +81,8 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -88,6 +90,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-sparc/hardirq.h +11 -6
@@ -87,13 +87,18 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
-#define irq_exit()                                                      \
-do {                                                                    \
-                preempt_count() -= IRQ_EXIT_OFFSET;                     \
-                if (!in_interrupt() && softirq_pending(smp_processor_id())) \
-                        do_softirq();                                   \
-                preempt_enable_no_resched();                            \
+
+#ifndef CONFIG_SOFTIRQ_THREADS
+#define irq_exit()							\
+do {									\
+		preempt_count() -= IRQ_EXIT_OFFSET;			\
+		if (!in_interrupt() && softirq_pending(smp_processor_id())) \
+			do_softirq();					\
+		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-sparc64/hardirq.h +5 -0
@@ -86,6 +86,8 @@
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -93,6 +95,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-v850/hardirq.h +10 -6
@@ -80,13 +80,17 @@
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
 
-#define irq_exit()							      \
-do {									      \
-	preempt_count() -= IRQ_EXIT_OFFSET;				      \
-	if (!in_interrupt() && softirq_pending(smp_processor_id()))	      \
-		do_softirq();						      \
-	preempt_enable_no_resched();					      \
+#ifndef CONFIG_SOFTIRQ_THREADS
+#define irq_exit()							\
+do {									\
+	preempt_count() -= IRQ_EXIT_OFFSET;				\
+	if (!in_interrupt() && softirq_pending(smp_processor_id()))	\
+		do_softirq();						\
+	preempt_enable_no_resched();					\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/asm-x86_64/hardirq.h +5 -0
@@ -85,6 +85,8 @@
 # define in_atomic()   (preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
 #endif
+
+#ifndef CONFIG_SOFTIRQ_THREADS
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
@@ -92,6 +94,9 @@
 			do_softirq();					\
 		preempt_enable_no_resched();				\
 } while (0)
+#else
+#define irq_exit()              (preempt_count() -= HARDIRQ_OFFSET)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
include/linux/interrupt.h +23 -0
@@ -59,6 +59,8 @@
 #endif
 
 /* SoftIRQ primitives.  */
+#ifndef CONFIG_SOFTIRQ_THREADS
+
 #define local_bh_disable() \
 		do { preempt_count() += SOFTIRQ_OFFSET; barrier(); } while (0)
 #define __local_bh_enable() \
@@ -66,6 +68,27 @@
 
 extern void local_bh_enable(void);
 
+#else
+
+/* As far as I can tell, local_bh_disable() didn't stop ksoftirqd
+   from running before.  Since all softirqs now run from one of
+   the ksoftirqds, this shouldn't be necessary. */
+
+static inline void local_bh_disable(void)
+{
+}
+
+static inline void __local_bh_enable(void)
+{
+}
+
+static inline void local_bh_enable(void)
+{
+}
+
+#endif
+
+
 /* PLEASE, avoid to allocate new softirqs, if you need not _really_ high
    frequency threaded job scheduling. For almost all the purposes
    tasklets are more than enough. F.e. all serial device BHs et
include/linux/sched.h +8 -0
@@ -541,6 +541,11 @@
 #define PF_LESS_THROTTLE 0x00100000	/* Throttle me less: I clean memory */
 #define PF_SYNCWRITE	0x00200000	/* I am doing a sync write */
 
+/* Thread is an IRQ handler.  This is used to determine which softirq
+   thread to wake. */
+
+#define PF_IRQHANDLER 0x10000000
+
 #ifdef CONFIG_SMP
 extern int set_cpus_allowed(task_t *p, cpumask_t new_mask);
 #else
@@ -916,6 +921,9 @@
 
 extern void signal_wake_up(struct task_struct *t, int resume_stopped);
 
+asmlinkage long sys_sched_setscheduler(pid_t pid, int policy,
+                                       struct sched_param __user *param);
+
 /*
  * Wrappers for p->thread_info->cpu access. No-op on UP.
  */
init/Kconfig +12 -0
@@ -265,6 +265,18 @@
 
 	  If unsure, say N.
 
+config SOFTIRQ_THREADS
+	bool "Run all softirqs in threads"
+	depends on PREEMPT
+	help
+	  This option creates a second softirq thread per CPU, which
+	  runs at high real-time priority, to replace the softirqs
+	  which were previously run immediately.  This allows these
+	  softirqs to be prioritized, so as to avoid preempting
+	  very high priority real-time tasks.  This also allows
+	  certain spinlocks to be converted into sleeping mutexes,
+	  for futher reduction of scheduling latency.
+
 endmenu		# General setup
 
 
kernel/softirq.c +99 -4

171 lines not on this page · raw message

kernel/timer.c +5 -1
@@ -910,12 +910,16 @@
 void do_timer(struct pt_regs *regs)
 {
 	jiffies_64++;
+	update_times();
+
 #ifndef CONFIG_SMP
 	/* SMP process accounting uses the local APIC timer */
 
+	// Do this after update_times(), so that the softirq thread
+	// is not counted as running all the time.
+
 	update_process_times(user_mode(regs));
 #endif
-	update_times();
 }
 
 #if !defined(__alpha__) && !defined(__ia64__)
mm/slab.c +1 -1
@@ -2593,7 +2593,7 @@
 {
 	struct list_head *walk;
 
-#if DEBUG
+#if DEBUG && !defined(CONFIG_SOFTIRQ_THREADS)
 	BUG_ON(!in_interrupt());
 	BUG_ON(in_irq());
 #endif
net/ipv4/ipconfig.c +7 -2
@@ -1085,8 +1085,13 @@
 
 		jiff = jiffies + (d->next ? CONF_INTER_TIMEOUT : timeout);
 		while (time_before(jiffies, jiff) && !ic_got_reply) {
-			barrier();
-			cpu_relax();
+			// Wait queues are apparently too hard
+			// for the ipconfig people, but we
+			// need to drop the BKL here to allow
+			// preemption.
+			
+			set_current_state(TASK_INTERRUPTIBLE);
+			schedule_timeout(1);
 		}
 #ifdef IPCONFIG_DHCP
 		/* DHCP isn't done until we get a DHCPACK. */
#449 lore raw
In reply to William Lee Irwin III
Quoted 10 lines · William Lee Irwin III
Quoted 8 lines · William Lee Irwin III
Quoted 8 lines

This patch uses the preemption counter increments and decrements to time non-preemptible critical sections.
This is an instrumentation patch intended to help determine the causes of scheduling latency related to long non-preemptible critical sections. Changes from 2.6.7-based patch:
(1) fix unmap_vmas() check correctly this time
(2) add touch_preempt_timing() to cond_resched_lock()
(3) depend on preempt until it's worked out wtf goes wrong without it

Quoted 7 lines
Quoted 4 lines · Joe Korty

You preemption-block hold times will improve *enormously* if you move all softirq processing down to the daemon (and possibly raise the daemon to one of the higher SCHED_RR priorities, to compensate for softirq processing no longer happening at interrupt level).

Plausible. Got a patch?

I can scrape one up. It'll be a day or so.
Joe

#450 lore raw

PATCH 1/1 fix arbitrarily long preemption lockout times [was: re: preempt-timing-2.6.8-rc1]

In reply to William Lee Irwin III
Quoted 7 lines · William Lee Irwin III
Quoted 4 lines · Joe Korty

You preemption-block hold times will improve *enormously* if you move all softirq processing down to the daemon (and possibly raise the daemon to one of the higher SCHED_RR priorities, to compensate for softirq processing no longer happening at interrupt level).

Plausible. Got a patch?

Here it is. Against 2.6.7. Testing: compiles, boots, shutdown ok, ran some disk and networking traffic under it.

IIRC, my first attempt had problems booting some configurations, my guess is that some drivers try to raise softIRQs before the daemons came up. So this version uses the global 'system_running' to switch at runtime between the interrupt/daemon and daemon-only modes of operation.

This also has the advantage of switching back to the safer, interrupt/daemon mode during system shutdown and while crash dumps are in progress (for those who have applied that patch).

IMO, this patch would be better if it introduced a seperate, hi-priority softirq daemon, as La Monte H.P. Yarroll's patch does.

Regards,
Joe

arch/i386/kernel/irq.c +2 -2
diff -ura base/arch/i386/kernel/irq.c new/arch/i386/kernel/irq.c
@@ -1154,7 +1154,7 @@
 
 extern asmlinkage void __do_softirq(void);
 
-asmlinkage void do_softirq(void)
+asmlinkage void do_softirq_arch(void)
 {
 	unsigned long flags;
 	struct thread_info *curctx;
@@ -1189,5 +1189,5 @@
 	local_irq_restore(flags);
 }
 
-EXPORT_SYMBOL(do_softirq);
+EXPORT_SYMBOL(do_softirq_arch);
 #endif
arch/ppc64/kernel/irq.c +2 -2
diff -ura base/arch/ppc64/kernel/irq.c new/arch/ppc64/kernel/irq.c
@@ -1015,7 +1015,7 @@
 	}
 }
 
-void do_softirq(void)
+void do_softirq_arch(void)
 {
 	unsigned long flags;
 	struct thread_info *curtp, *irqtp;
@@ -1035,7 +1035,7 @@
 
 	local_irq_restore(flags);
 }
-EXPORT_SYMBOL(do_softirq);
+EXPORT_SYMBOL(do_softirq_arch);
 
 #endif /* CONFIG_IRQSTACKS */
 
include/linux/interrupt.h +1 -0
diff -ura base/include/linux/interrupt.h new/include/linux/interrupt.h
@@ -93,6 +93,7 @@
 };
 
 asmlinkage void do_softirq(void);
+asmlinkage void do_softirq_arch(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action*), void *data);
 extern void softirq_init(void);
 #define __raise_softirq_irqoff(nr) do { local_softirq_pending() |= 1UL << (nr); } while (0)
init/Kconfig +37 -0
diff -ura base/init/Kconfig new/init/Kconfig
@@ -218,6 +218,43 @@
 	  This option enables access to kernel configuration file and build
 	  information through /proc/config.gz.
 
+config SOFTIRQ_PREEMPT_BLOCK
+	bool "SoftIRQs to be affected by preemption blocks"
+	default y
+	help
+	  If N then SoftIRQ interrupt handlers are to be blocked by
+	  exactly the same conditions as hard IRQ handlers.  If Y then
+	  SoftIRQ handlers are also to be blocked by preemption blocks.
+
+	  The reasons why Y is useful are too long to elaborate here.
+	  Suffice it to say that SoftIRQ handlers unlike hardIRQ handlers
+	  have no upper bounds on their execution time and allowing
+	  them to interrupt spinlocked critical regions protected only
+	  by a preemption block means that no such region can have an
+	  upper bound on the lock hold time.
+
+	  Realtime systems should say Y here, all others may say N or
+	  Y.  If unsure, say N.
+
+config SOFTIRQ_PRI
+	int "SoftIRQ daemon priority"
+	default 0
+	help
+	  Select the scheduling policy and priority for the per-cpu
+	  SoftIRQ daemons.
+
+	  If >0 then the daemons will have a policy of SCHED_FIFO and
+	  their priorities will each be set to the given value.
+
+	  If ==0 then the daemons will run with a SCHED_OTHER policy
+	  unless SOFTIRQ_PREEMPT_BLOCK is set to Y.  In that case the
+	  daemons will run under SCHED_FIFO with a priority one less
+	  than the system maximum.
+
+	  Most users should say 0 unless there is a specific priority
+	  the softIRQ daemons must run at.  If unsure, say 0.
+
+
 
 menuconfig EMBEDDED
 	bool "Configure standard kernel features (for small systems)"
kernel/softirq.c +43 -3

80 lines not on this page · raw message

#451 lore raw
In reply to William Lee Irwin III

Hi,

More violations found.

This time during NFS umount/mount. Since (u)mounting is not always
the operation that is only done at boot-time (at least in my case), I thought I report.

umount /nfs/dir
16ms non-preemptible critical section violated 2 ms preempt threshold starting at truncate_inode_pages+0x9d/0x280 and ending at generic_shutdown_super+0xd0/0x190

[<c0153340>] generic_shutdown_super+0xd0/0x190
[<c0117d30>] dec_preempt_count+0x110/0x120
[<c0153340>] generic_shutdown_super+0xd0/0x190
[<c0117c70>] dec_preempt_count+0x50/0x120
[<c0153cf9>] kill_anon_super+0x9/0x40
[<f935884c>] nfs_kill_super+0xc/0x20 [nfs]
[<c015317c>] deactivate_super+0x6c/0x90
[<c016809b>] sys_umount+0x3b/0x90
[<c0164a85>] destroy_inode+0x35/0x40
[<c014e645>] __fput+0xb5/0x120
[<c0168107>] sys_oldumount+0x17/0x20
[<c0103ee1>] sysenter_past_esp+0x52/0x71

mount /nfs/dir
nfs warning: mount version older than kernel
15ms non-preemptible critical section violated 2 ms preempt threshold starting at sys_mount+0x78/0x110 and ending at schedule+0x237/0x480

[<c0279677>] schedule+0x237/0x480
[<c0117d30>] dec_preempt_count+0x110/0x120
[<c0279677>] schedule+0x237/0x480
[<c0127c21>] worker_thread+0x1d1/0x240
[<c0127c72>] worker_thread+0x222/0x240
[<c01166f8>] activate_task+0x68/0x80
[<c0219800>] fb_flashcursor+0x0/0x80
[<c0116d60>] default_wake_function+0x0/0x10
[<c0279677>] schedule+0x237/0x480
[<c0116d60>] default_wake_function+0x0/0x10
[<c0127a50>] worker_thread+0x0/0x240
[<c012b2c4>] kthread+0x94/0xa0
[<c012b230>] kthread+0x0/0xa0
[<c010227d>] kernel_thread_helper+0x5/0x18

umount /nfs/dir
19ms non-preemptible critical section violated 2 ms preempt threshold starting at generic_shutdown_super+0x69/0x190 and ending at generic_shutdown_super+0xd0/0x190

[<c0153340>] generic_shutdown_super+0xd0/0x190
[<c0117d30>] dec_preempt_count+0x110/0x120
[<c0153340>] generic_shutdown_super+0xd0/0x190
[<c0117c70>] dec_preempt_count+0x50/0x120
[<c0153cf9>] kill_anon_super+0x9/0x40
[<f935884c>] nfs_kill_super+0xc/0x20 [nfs]
[<c015317c>] deactivate_super+0x6c/0x90
[<c016809b>] sys_umount+0x3b/0x90
[<c0164a85>] destroy_inode+0x35/0x40
[<c014e645>] __fput+0xb5/0x120
[<c0168107>] sys_oldumount+0x17/0x20
[<c0103ee1>] sysenter_past_esp+0x52/0x71

Lenar