Scheduler fairness problem on 2.6 series (Attn: Nick Piggin and others)
A message that is not in the archive (sent off-list or not yet received): [email protected]
I tried the actual server with a stress test, and I do eventually get timeouts.
I tried with what seemed to be the best setup earlier:
pb modemax_ia_bonus set to 0
I tried several values for base_promotion_interval but the system eventually times out after a few hours (it's still better than it used to be, with a stock kernel, it timeouts in less than an hour).
Nicolas
Quoted 9 lines · [email protected] wrote:
I tried the actual server with a stress test, and I do eventually get timeouts.
I tried with what seemed to be the best setup earlier:
pb modemax_ia_bonusset to 0I tried several values for
base_promotion_intervalbut the system eventually times out after a few hours (it's still better than it used to be, with a stock kernel, it timeouts in less than an hour).
Could you try it in "pb" mode with both max_ia_bonus and max_tpt_bonus set to zero? That will disable all "priority" fiddling and tasks should just round robin at a priority determined solely by their "nice" value and since (according to your earlier mail) all the daemons have the same "nice" value they should just round robin with each other.
Thanks
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 5 lines · --- Peter Williams
Could you try it in "pb" mode with both
max_ia_bonusandmax_tpt_bonusset to zero? That will disable all "priority" fiddling and tasks should just round robin at a priority determined solely by their "nice" value and since (according to your earlier mail) all the daemons have the same "nice" value they should just round robin with each other.
Hi, I tried the latest V-5.0 patch over 2.6.8.1 in these conditions with the actual server subsystem, and I get components timeouts :(
I also ran the watchdog script on the box while running the test, and saw deltas of around 3 seconds every few hours:
Tue Aug 24 03:02:13 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 05:50:14 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 09:05:24 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 4
Tue Aug 24 09:06:20 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 4
Tue Aug 24 09:36:22 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 10:20:16 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 13:28:19 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Could I do something more useful than just displaying those deltas? Maybe I could dump the process list in some way, or enable some debugging code in the kernel to find out what is going on?
Thanks
Nicolas
=====
------------------------------------------------------------
video meliora proboque deteriora sequor
------------------------------------------------------------
Quoted 36 lines · [email protected] wrote:
Quoted 1 line · --- Peter Williams
Could you try it in "pb" mode with both
max_ia_bonusandmax_tpt_bonusset to zero? That will disable all "priority" fiddling and tasks should just round robin at a priority determined solely by their "nice" value and since (according to your earlier mail) all the daemons have the same "nice" value they should just round robin with each other.Hi, I tried the latest V-5.0 patch over 2.6.8.1 in these conditions with the actual server subsystem, and I get components timeouts :(
I also ran the watchdog script on the box while running the test, and saw deltas of around 3 seconds every few hours:Tue Aug 24 03:02:13 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 05:50:14 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 09:05:24 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 4
Tue Aug 24 09:06:20 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 4
Tue Aug 24 09:36:22 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Tue Aug 24 10:20:16 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Quoted 8 lines
Tue Aug 24 13:28:19 PDT 2004
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
delta = 3
Could I do something more useful than just displaying those deltas? Maybe I could dump the process list in some way, or enable some debugging code in the kernel to find out what is going on?
You could try Lee Revell's ([email protected]) latency measuring patches and also try applying Ingo Molnar's ([email protected]) voluntary-preempt patches.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 10 lines · Peter Williams
Quoted 4 lines · [email protected] wrote:
Could I do something more useful than just displaying those deltas? Maybe I could dump the process list in some way, or enable some debugging code in the kernel to find out what is going on?
You could try Lee Revell's ([email protected]) latency measuring patches and also try applying Ingo Molnar's ([email protected]) voluntary-preempt patches.
Most of the tools I am using are probably too specific to the audio subsystem to be of much use to you. Just use Ingo's voluntary preemption patch; if this is a scheduler/preemption problem, then it will definitely show up in the traces.
Lee
Quoted 5 lines · --- Peter Williams
You could try Lee Revell's ([email protected]) latency measuring patches and also try applying Ingo Molnar's ([email protected]) voluntary-preempt patches.
Peter
I tried 2.6.8.1 with voluntary-preempt-2.6.8.1-P9 and I am getting latency messages, they trigger at around the same time I get "delta = 3" messages.
I guess that there is no way to have the latency reporting work with the zaphod patch?
I hope those messages are giving a clue on what is going on on this box...
Are this kind of latencies normal (reminder this is an Athlon XP 1800+ ie 1.5MHz)?
00010006 0.144ms (+0.000ms): task_rq_lock (try_to_wake_up) 00010007 0.144ms (+0.000ms): activate_task (try_to_wake_up) 00010007 0.144ms (+0.000ms): sched_clock (activate_task) 00010007 0.144ms (+0.000ms): recalc_task_prio (activate_task) 00010007 0.144ms (+0.000ms): effective_prio (recalc_task_prio) 00010007 0.144ms (+0.000ms): enqueue_task (activate_task)
Anyway, here are the full messages:
Aug 25 19:10:01 localhost kernel: (ksoftirqd/0/3): new 163 us maximum-latency critical section. Aug 25 19:10:01 localhost kernel: => started at: <netif_receive_skb+0x81/0x1e0> Aug 25 19:10:01 localhost kernel: => ended at: <netif_receive_skb+0x167/0x1e0> Aug 25 19:10:01 localhost kernel: [<c013eec7>] check_preempt_timing+0x1c7/0x230 Aug 25 19:10:01 localhost kernel: [<c031ae57>] netif_receive_skb+0x167/0x1e0 Aug 25 19:10:01 localhost kernel: [<c031ae57>] netif_receive_skb+0x167/0x1e0 Aug 25 19:10:01 localhost kernel: [<c013f074>] sub_preempt_count+0x54/0x60 Aug 25 19:10:01 localhost kernel: [<c013f074>] sub_preempt_count+0x54/0x60 Aug 25 19:10:01 localhost kernel: [<c031ae57>] netif_receive_skb+0x167/0x1e0 Aug 25 19:10:01 localhost kernel: [<c0310008>] sock_release+0xf8/0x110 Aug 25 19:10:01 localhost kernel: [<c031af5a>] process_backlog+0x8a/0x150 Aug 25 19:10:01 localhost kernel: [<c031b0ae>] net_rx_action+0x8e/0x120 Aug 25 19:10:01 localhost kernel: [<c01248a2>] ___do_softirq+0xa2/0xb0 Aug 25 19:10:01 localhost kernel: [<c012492e>] _do_softirq+0xe/0x20 Aug 25 19:10:01 localhost kernel: [<c0124eb5>] ksoftirqd+0xa5/0x110 Aug 25 19:10:01 localhost kernel: [<c013723c>] kthread+0xbc/0xd0 Aug 25 19:10:01 localhost kernel: [<c0124e10>] ksoftirqd+0x0/0x110 Aug 25 19:10:01 localhost kernel: [<c0137180>] kthread+0x0/0xd0 Aug 25 19:10:01 localhost kernel: [<c01044b9>] kernel_thread_helper+0x5/0xc
preemption latency trace v1.0.2
-------------------------------
latency: 163 us, entries: 194 (194)
-----------------
| task: ksoftirqd/0/3, uid:0 nice:-10 policy:0 rt_prio:0
----------------- => started at: netif_receive_skb+0x81/0x1e0
=> ended at: netif_receive_skb+0x167/0x1e0
=======>
00000001 0.000ms (+0.000ms): netif_receive_skb (process_backlog) 00000001 0.001ms (+0.001ms): ip_rcv (netif_receive_skb)
00000001 0.005ms (+0.003ms): nf_hook_slow (ip_rcv)
00000002 0.006ms (+0.001ms): nf_iterate (nf_hook_slow)
00000002 0.007ms (+0.001ms): ip_conntrack_defrag (nf_iterate) 00000002 0.008ms (+0.000ms): ip_conntrack_in (nf_iterate) 00000002 0.008ms (+0.000ms): ip_ct_find_proto (ip_conntrack_in) 00000103 0.009ms (+0.000ms): __ip_ct_find_proto (ip_ct_find_proto) 00000102 0.009ms (+0.000ms): local_bh_enable (ip_ct_find_proto) 00000002 0.010ms (+0.000ms): get_tuple (ip_conntrack_in)
00000002 0.010ms (+0.000ms): tcp_pkt_to_tuple (get_tuple) 00000002 0.011ms (+0.000ms): skb_copy_bits (tcp_pkt_to_tuple) 00000002 0.013ms (+0.001ms): ip_conntrack_find_get (ip_conntrack_in) 00000103 0.013ms (+0.000ms): __ip_conntrack_find (ip_conntrack_find_get) 00000103 0.013ms (+0.000ms): hash_conntrack (__ip_conntrack_find) 00000102 0.015ms (+0.001ms): local_bh_enable (ip_conntrack_find_get) 00000002 0.015ms (+0.000ms): tcp_packet (ip_conntrack_in) 00000002 0.015ms (+0.000ms): skb_copy_bits (tcp_packet)
00000103 0.016ms (+0.000ms): get_conntrack_index (tcp_packet) 00000102 0.017ms (+0.000ms): local_bh_enable (tcp_packet) 00000002 0.017ms (+0.000ms): ip_ct_refresh (tcp_packet)
00000103 0.018ms (+0.000ms): del_timer (ip_ct_refresh)
00000103 0.019ms (+0.001ms): __mod_timer (ip_ct_refresh)
00000105 0.019ms (+0.000ms): internal_add_timer (__mod_timer) 00000102 0.019ms (+0.000ms): local_bh_enable (tcp_packet) 00000002 0.020ms (+0.000ms): ipt_route_hook (nf_iterate)
00000002 0.020ms (+0.000ms): ipt_do_table (ipt_route_hook) 00000102 0.024ms (+0.003ms): local_bh_enable (ipt_do_table) 00000002 0.024ms (+0.000ms): ip_nat_fn (nf_iterate)
00000002 0.024ms (+0.000ms): ip_conntrack_get (ip_nat_fn) 00000002 0.026ms (+0.001ms): do_bindings (ip_nat_fn)
00000102 0.027ms (+0.001ms): local_bh_enable (do_bindings) 00000002 0.029ms (+0.001ms): ip_rcv_finish (nf_hook_slow) 00000002 0.030ms (+0.001ms): ip_route_input (ip_rcv_finish) 00000002 0.030ms (+0.000ms): rt_hash_code (ip_route_input) 00000002 0.035ms (+0.004ms): ip_local_deliver (ip_rcv_finish) 00000002 0.036ms (+0.000ms): nf_hook_slow (ip_local_deliver) 00000003 0.036ms (+0.000ms): nf_iterate (nf_hook_slow)
00000003 0.036ms (+0.000ms): ipt_route_hook (nf_iterate)
00000003 0.036ms (+0.000ms): ipt_do_table (ipt_route_hook) 00000103 0.037ms (+0.001ms): local_bh_enable (ipt_do_table) 00000003 0.038ms (+0.000ms): ipt_hook (nf_iterate)
00000003 0.038ms (+0.000ms): ipt_do_table (ipt_hook)
00000103 0.039ms (+0.001ms): local_bh_enable (ipt_do_table) 00000003 0.040ms (+0.000ms): ip_confirm (nf_iterate)
00000003 0.040ms (+0.000ms): ip_local_deliver_finish (nf_hook_slow) 00000004 0.042ms (+0.002ms): tcp_v4_rcv (ip_local_deliver_finish) 00000004 0.043ms (+0.001ms): tcp_v4_checksum_init (tcp_v4_rcv) 00000004 0.045ms (+0.001ms): skb_checksum (tcp_v4_checksum_init) 00000005 0.053ms (+0.008ms): tcp_v4_do_rcv (tcp_v4_rcv)
00000005 0.054ms (+0.000ms): tcp_rcv_state_process (tcp_v4_do_rcv) 00000005 0.057ms (+0.002ms): tcp_ack (tcp_rcv_state_process) 00000005 0.058ms (+0.001ms): tcp_ack_update_window (tcp_ack) 00000005 0.061ms (+0.002ms): tcp_clean_rtx_queue (tcp_ack) 00000005 0.063ms (+0.002ms): __kfree_skb (tcp_clean_rtx_queue) 00000005 0.064ms (+0.000ms): kfree_skbmem (__kfree_skb)
00000005 0.064ms (+0.000ms): skb_release_data (kfree_skbmem) 00000005 0.066ms (+0.001ms): kfree (kfree_skbmem)
00000005 0.066ms (+0.000ms): kmem_cache_free (kfree_skbmem) 00000005 0.067ms (+0.000ms): tcp_ack_no_tstamp (tcp_clean_rtx_queue) 00000005 0.067ms (+0.000ms): tcp_rtt_estimator (tcp_ack_no_tstamp) 00000005 0.074ms (+0.006ms): tcp_reset_keepalive_timer (tcp_rcv_state_process) 00000005 0.075ms (+0.000ms): sk_reset_timer (tcp_reset_keepalive_timer) 00000005 0.075ms (+0.000ms): mod_timer (sk_reset_timer)
00000005 0.075ms (+0.000ms): __mod_timer (sk_reset_timer) 00000007 0.076ms (+0.000ms): internal_add_timer (__mod_timer) 00000005 0.077ms (+0.000ms): tcp_urg (tcp_rcv_state_process) 00000005 0.077ms (+0.000ms): tcp_data_queue (tcp_rcv_state_process) 00000005 0.080ms (+0.002ms): sk_stream_mem_schedule (tcp_data_queue) 00000005 0.081ms (+0.001ms): tcp_fin (tcp_data_queue)
00000005 0.083ms (+0.001ms): tcp_send_ack (tcp_fin)
00000005 0.083ms (+0.000ms): alloc_skb (tcp_send_ack)
00000005 0.083ms (+0.000ms): kmem_cache_alloc (alloc_skb) 00000005 0.083ms (+0.000ms): __kmalloc (alloc_skb)
00000005 0.084ms (+0.001ms): tcp_transmit_skb (tcp_send_ack) 00000005 0.087ms (+0.003ms): __tcp_select_window (tcp_transmit_skb) 00000005 0.090ms (+0.003ms): tcp_v4_send_check (tcp_transmit_skb) 00000005 0.092ms (+0.001ms): ip_queue_xmit (tcp_transmit_skb) 00000005 0.097ms (+0.005ms): nf_hook_slow (ip_queue_xmit) 00000006 0.097ms (+0.000ms): nf_iterate (nf_hook_slow)
00000006 0.098ms (+0.000ms): ip_conntrack_defrag (nf_iterate) 00000006 0.098ms (+0.000ms): ip_conntrack_local (nf_iterate) 00000006 0.098ms (+0.000ms): ip_conntrack_in (nf_iterate) 00000006 0.099ms (+0.000ms): ip_ct_find_proto (ip_conntrack_in) 00000107 0.099ms (+0.000ms): __ip_ct_find_proto (ip_ct_find_proto) 00000106 0.099ms (+0.000ms): local_bh_enable (ip_ct_find_proto) 00000006 0.099ms (+0.000ms): get_tuple (ip_conntrack_in)
00000006 0.100ms (+0.000ms): tcp_pkt_to_tuple (get_tuple) 00000006 0.100ms (+0.000ms): skb_copy_bits (tcp_pkt_to_tuple) 00000006 0.100ms (+0.000ms): ip_conntrack_find_get (ip_conntrack_in) 00000107 0.100ms (+0.000ms): __ip_conntrack_find (ip_conntrack_find_get) 00000107 0.100ms (+0.000ms): hash_conntrack (__ip_conntrack_find) 00000106 0.101ms (+0.000ms): local_bh_enable (ip_conntrack_find_get) 00000006 0.101ms (+0.000ms): tcp_packet (ip_conntrack_in) 00000006 0.101ms (+0.000ms): skb_copy_bits (tcp_packet)
00000107 0.101ms (+0.000ms): get_conntrack_index (tcp_packet) 00000106 0.101ms (+0.000ms): local_bh_enable (tcp_packet) 00000006 0.102ms (+0.000ms): ip_ct_refresh (tcp_packet)
00000107 0.102ms (+0.000ms): del_timer (ip_ct_refresh)
00000107 0.102ms (+0.000ms): __mod_timer (ip_ct_refresh)
00000109 0.102ms (+0.000ms): internal_add_timer (__mod_timer) 00000106 0.102ms (+0.000ms): local_bh_enable (tcp_packet) 00000006 0.103ms (+0.000ms): ipt_local_hook (nf_iterate)
00000006 0.103ms (+0.000ms): ipt_do_table (ipt_local_hook) 00000106 0.104ms (+0.001ms): local_bh_enable (ipt_do_table) 00000006 0.105ms (+0.000ms): ipt_local_out_hook (nf_iterate) 00000006 0.105ms (+0.000ms): ipt_do_table (ipt_local_out_hook) 00000106 0.107ms (+0.001ms): local_bh_enable (ipt_do_table) 00000006 0.107ms (+0.000ms): dst_output (nf_hook_slow)
00000006 0.108ms (+0.000ms): ip_output (dst_output)
00000006 0.108ms (+0.000ms): ip_finish_output (dst_output) 00000006 0.109ms (+0.000ms): nf_hook_slow (ip_finish_output) 00000007 0.109ms (+0.000ms): nf_iterate (nf_hook_slow)
00000007 0.109ms (+0.000ms): ipt_route_hook (nf_iterate)
00000007 0.109ms (+0.000ms): ipt_do_table (ipt_route_hook) 00000107 0.110ms (+0.000ms): local_bh_enable (ipt_do_table) 00000007 0.110ms (+0.000ms): ip_nat_out (nf_iterate)
00000007 0.111ms (+0.000ms): ip_nat_fn (nf_iterate)
00000007 0.111ms (+0.000ms): ip_conntrack_get (ip_nat_fn) 00000007 0.111ms (+0.000ms): do_bindings (ip_nat_fn)
00000107 0.112ms (+0.000ms): local_bh_enable (do_bindings) 00000007 0.112ms (+0.000ms): ip_refrag (nf_iterate)
00000007 0.112ms (+0.000ms): ip_confirm (ip_refrag)
00000007 0.112ms (+0.000ms): ip_finish_output2 (nf_hook_slow) 00000107 0.114ms (+0.001ms): local_bh_enable (ip_finish_output2) 00000007 0.115ms (+0.000ms): dev_queue_xmit (ip_finish_output2) 00000109 0.116ms (+0.001ms): pfifo_fast_enqueue (dev_queue_xmit) 00000109 0.117ms (+0.000ms): qdisc_restart (dev_queue_xmit) 00000109 0.118ms (+0.000ms): pfifo_fast_dequeue (qdisc_restart) 00000109 0.120ms (+0.001ms): speedo_start_xmit (qdisc_restart) 0000010a 0.122ms (+0.002ms): __const_udelay (speedo_start_xmit) 0000010a 0.123ms (+0.000ms): __delay (speedo_start_xmit)0000010a 0.123ms (+0.000ms): delay_tsc (__delay)
00000109 0.126ms (+0.002ms): qdisc_restart (dev_queue_xmit) 00000109 0.126ms (+0.000ms): pfifo_fast_dequeue (qdisc_restart) 00000108 0.126ms (+0.000ms): local_bh_enable (dev_queue_xmit) 00000005 0.128ms (+0.001ms): tcp_time_wait (tcp_fin)
00000005 0.129ms (+0.000ms): kmem_cache_alloc (tcp_time_wait) 00000005 0.130ms (+0.001ms): __tcp_tw_hashdance (tcp_time_wait) 00000005 0.134ms (+0.003ms): tcp_tw_schedule (tcp_time_wait) 00000005 0.136ms (+0.001ms): tcp_update_metrics (tcp_time_wait) 00010005 0.138ms (+0.001ms): do_IRQ (tcp_update_metrics)
00010006 0.138ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
00010006 0.143ms (+0.005ms): generic_redirect_hardirq (do_IRQ) 00010006 0.144ms (+0.000ms): wake_up_process (generic_redirect_hardirq) 00010006 0.144ms (+0.000ms): try_to_wake_up (wake_up_process) 00010006 0.144ms (+0.000ms): task_rq_lock (try_to_wake_up) 00010007 0.144ms (+0.000ms): activate_task (try_to_wake_up) 00010007 0.144ms (+0.000ms): sched_clock (activate_task)
00010007 0.144ms (+0.000ms): recalc_task_prio (activate_task) 00010007 0.144ms (+0.000ms): effective_prio (recalc_task_prio) 00010007 0.144ms (+0.000ms): enqueue_task (activate_task) 00000005 0.145ms (+0.000ms): smp_apic_timer_interrupt (tcp_update_metrics) 00010005 0.145ms (+0.000ms): update_process_times (smp_apic_timer_interrupt) 00010005 0.146ms (+0.000ms): update_one_process (update_process_times) 00010005 0.146ms (+0.000ms): run_local_timers (update_process_times) 00010005 0.147ms (+0.000ms): raise_softirq (update_process_times) 00010005 0.147ms (+0.000ms): scheduler_tick (update_process_times) 00010005 0.147ms (+0.000ms): sched_clock (scheduler_tick) 00010006 0.148ms (+0.000ms): __bitmap_weight (scheduler_tick) 00010006 0.148ms (+0.000ms): task_timeslice (scheduler_tick) 00010005 0.148ms (+0.000ms): rebalance_tick (scheduler_tick) 00000006 0.148ms (+0.000ms): do_softirq (smp_apic_timer_interrupt) 00000006 0.148ms (+0.000ms): __do_softirq (do_softirq)
00000005 0.150ms (+0.001ms): tcp_unhash (tcp_time_wait)
00000005 0.151ms (+0.001ms): tcp_put_port (tcp_time_wait) 00000105 0.152ms (+0.000ms): __tcp_put_port (tcp_put_port) 00000106 0.152ms (+0.000ms): tcp_bucket_destroy (__tcp_put_port) 00000105 0.153ms (+0.000ms): local_bh_enable (tcp_time_wait) 00000005 0.153ms (+0.000ms): do_softirq (local_bh_enable) 00000005 0.153ms (+0.000ms): __do_softirq (do_softirq)
00000005 0.153ms (+0.000ms): tcp_clear_xmit_timers (tcp_time_wait) 00000005 0.154ms (+0.000ms): sk_stop_timer (tcp_clear_xmit_timers) 00000005 0.154ms (+0.000ms): del_timer (sk_stop_timer)
00000005 0.155ms (+0.001ms): sk_stop_timer (tcp_clear_xmit_timers) 00000005 0.155ms (+0.000ms): sk_stop_timer (tcp_clear_xmit_timers) 00000005 0.155ms (+0.000ms): del_timer (sk_stop_timer)
00000005 0.156ms (+0.000ms): tcp_destroy_sock (tcp_fin)
00000005 0.157ms (+0.001ms): tcp_v4_destroy_sock (tcp_destroy_sock) 00000005 0.157ms (+0.000ms): tcp_clear_xmit_timers (tcp_v4_destroy_sock) 00000005 0.157ms (+0.000ms): sk_stop_timer (tcp_clear_xmit_timers) 00000005 0.157ms (+0.000ms): sk_stop_timer (tcp_clear_xmit_timers) 00000005 0.158ms (+0.000ms): sk_stop_timer (tcp_clear_xmit_timers) 00000005 0.159ms (+0.001ms): sk_stream_kill_queues (tcp_destroy_sock) 00000005 0.159ms (+0.000ms): __kfree_skb (sk_stream_kill_queues) 00000005 0.159ms (+0.000ms): sk_stream_rfree (__kfree_skb) 00000005 0.160ms (+0.000ms): kfree_skbmem (__kfree_skb)
00000005 0.160ms (+0.000ms): skb_release_data (kfree_skbmem) 00000005 0.160ms (+0.000ms): kfree (kfree_skbmem)
00000005 0.160ms (+0.000ms): kmem_cache_free (kfree_skbmem) 00000005 0.161ms (+0.001ms): __sk_stream_mem_reclaim (sk_stream_kill_queues) 00000001 0.164ms (+0.002ms): sub_preempt_count (netif_receive_skb) 00000001 0.164ms (+0.000ms): _mmx_memcpy (check_preempt_timing) 00000001 0.165ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
Quoted 4 lines · [email protected] wrote:
Quoted 2 lines · --- Peter Williams
You could try Lee Revell's ([email protected]) latency measuring patches and also try applying Ingo Molnar's ([email protected]) voluntary-preempt patches.
Quoted 8 lines
Quoted 1 line
Peter
I tried 2.6.8.1 with voluntary-preempt-2.6.8.1-P9 and I am getting latency messages, they trigger at around the same time I get "delta = 3" messages.
I guess that there is no way to have the latency reporting work with the zaphod patch?
I'll see what I can do and let you know.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 10 lines · Peter Williams
Quoted 8 lines · [email protected] wrote:
Quoted 3 lines · --- Peter Williams
You could try Lee Revell's ([email protected]) latency measuring patches and also try applying Ingo Molnar's ([email protected]) voluntary-preempt patches.
Peter
Quoted 8 lines
Quoted 5 lines
I tried 2.6.8.1 with voluntary-preempt-2.6.8.1-P9 and I am getting latency messages, they trigger at around the same time I get "delta = 3" messages.
I guess that there is no way to have the latency reporting work with the zaphod patch?
I'll see what I can do and let you know.
A (gzipped) combined ZAPHOD and P9 voluntary preempt patch for 2.6.8.1 is available at:
<http://prdownloads.sourceforge.net/cpuse/patch-2.6.8.1-zaphod-vp-v5.0.1.gz?download>
This patch has had minimal testing so use with care and please let me know if there are any problems.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 4 lines · --- Peter Williams
A (gzipped) combined ZAPHOD and P9 voluntary preempt patch for 2.6.8.1 is available at:
<http://prdownloads.sourceforge.net/cpuse/patch-2.6.8.1-zaphod-vp-v5.0.1.gz?download>
Quoted 4 lines
This patch has had minimal testing so use with care and please let me know if there are any problems.
I tried this patch, and I get a pretty high latency in "sub_preempt_count" 00000001 0.730ms (+0.730ms): sub_preempt_count (_mmx_memcpy)
I am not sure if that makes sense and what it means.
Nicolas
Here are the full messages:
Aug 27 18:42:11 localhost kernel: (events/0/4): new 730 us maximum-latency critical section. Aug 27 18:42:11 localhost kernel: => started at: <kernel_fpu_begin+0x21/0x60> Aug 27 18:42:11 localhost kernel: => ended at: <_mmx_memcpy+0x131/0x180> Aug 27 18:42:11 localhost kernel: [<c014106a>] check_preempt_timing+0x1aa/0x240 Aug 27 18:42:11 localhost kernel: [<c0225751>] _mmx_memcpy+0x131/0x180 Aug 27 18:42:11 localhost kernel: [<c0225751>] _mmx_memcpy+0x131/0x180 Aug 27 18:42:11 localhost kernel: [<c0141244>] sub_preempt_count+0x54/0x60 Aug 27 18:42:11 localhost kernel: [<c0141244>] sub_preempt_count+0x54/0x60 Aug 27 18:42:11 localhost kernel: [<c0225751>] _mmx_memcpy+0x131/0x180 Aug 27 18:42:11 localhost kernel: [<c02dd9fe>] vgacon_save_screen+0x7e/0x80 Aug 27 18:42:11 localhost kernel: [<c0267d32>] do_blank_screen+0x182/0x2b0 Aug 27 18:42:11 localhost kernel: [<c0122fa4>] acquire_console_sem+0x44/0x70 Aug 27 18:42:11 localhost kernel: [<c0266ab2>] console_callback+0x72/0xf0 Aug 27 18:42:11 localhost kernel: [<c0134dcb>] worker_thread+0x1eb/0x2d0 Aug 27 18:42:11 localhost kernel: [<c0266a40>] console_callback+0x0/0xf0 Aug 27 18:42:11 localhost kernel: [<c011c000>] default_wake_function+0x0/0x20 Aug 27 18:42:11 localhost kernel: [<c011c000>] default_wake_function+0x0/0x20 Aug 27 18:42:11 localhost kernel: [<c013963c>] kthread+0xbc/0xd0 Aug 27 18:42:11 localhost kernel: [<c0134be0>] worker_thread+0x0/0x2d0 Aug 27 18:42:11 localhost kernel: [<c0139580>] kthread+0x0/0xd0 Aug 27 18:42:11 localhost kernel: [<c0104389>] kernel_thread_helper+0x5/0xc
preemption latency trace v1.0.2
-------------------------------
latency: 730 us, entries: 4 (4)
-----------------
| task: events/0/4, uid:0 nice:-10 policy:0 rt_prio:0
----------------- => started at: kernel_fpu_begin+0x21/0x60
=> ended at: _mmx_memcpy+0x131/0x180
=======>
00000001 0.000ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy) 00000001 0.730ms (+0.730ms): sub_preempt_count (_mmx_memcpy) 00000001 0.730ms (+0.000ms): _mmx_memcpy (check_preempt_timing) 00000001 0.730ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
Quoted 52 lines · [email protected] wrote:
Quoted 3 lines · --- Peter Williams
A (gzipped) combined ZAPHOD and P9 voluntary preempt patch for 2.6.8.1 is available at:
<http://prdownloads.sourceforge.net/cpuse/patch-2.6.8.1-zaphod-vp-v5.0.1.gz?download>
Quoted 2 lines
This patch has had minimal testing so use with care and please let me know if there are any problems.
I tried this patch, and I get a pretty high latency in "
sub_preempt_count" 00000001 0.730ms (+0.730ms):sub_preempt_count(_mmx_memcpy)I am not sure if that makes sense and what it means.
Nicolas
Here are the full messages:
Aug 27 18:42:11 localhost kernel: (events/0/4): new 730 us maximum-latency critical section. Aug 27 18:42:11 localhost kernel: => started at: <kernel_fpu_begin+0x21/0x60> Aug 27 18:42:11 localhost kernel: => ended at: <_mmx_memcpy+0x131/0x180> Aug 27 18:42:11 localhost kernel: [<c014106a>] check_preempt_timing+0x1aa/0x240 Aug 27 18:42:11 localhost kernel: [<c0225751>] _mmx_memcpy+0x131/0x180 Aug 27 18:42:11 localhost kernel: [<c0225751>] _mmx_memcpy+0x131/0x180 Aug 27 18:42:11 localhost kernel: [<c0141244>] sub_preempt_count+0x54/0x60 Aug 27 18:42:11 localhost kernel: [<c0141244>] sub_preempt_count+0x54/0x60 Aug 27 18:42:11 localhost kernel: [<c0225751>] _mmx_memcpy+0x131/0x180 Aug 27 18:42:11 localhost kernel: [<c02dd9fe>] vgacon_save_screen+0x7e/0x80 Aug 27 18:42:11 localhost kernel: [<c0267d32>] do_blank_screen+0x182/0x2b0 Aug 27 18:42:11 localhost kernel: [<c0122fa4>] acquire_console_sem+0x44/0x70 Aug 27 18:42:11 localhost kernel: [<c0266ab2>] console_callback+0x72/0xf0 Aug 27 18:42:11 localhost kernel: [<c0134dcb>] worker_thread+0x1eb/0x2d0 Aug 27 18:42:11 localhost kernel: [<c0266a40>] console_callback+0x0/0xf0 Aug 27 18:42:11 localhost kernel: [<c011c000>] default_wake_function+0x0/0x20 Aug 27 18:42:11 localhost kernel: [<c011c000>] default_wake_function+0x0/0x20 Aug 27 18:42:11 localhost kernel: [<c013963c>] kthread+0xbc/0xd0 Aug 27 18:42:11 localhost kernel: [<c0134be0>] worker_thread+0x0/0x2d0 Aug 27 18:42:11 localhost kernel: [<c0139580>] kthread+0x0/0xd0 Aug 27 18:42:11 localhost kernel: [<c0104389>] kernel_thread_helper+0x5/0xcpreemption latency trace v1.0.2
-------------------------------
latency: 730 us, entries: 4 (4)----------------- | task: events/0/4, uid:0 nice:-10 policy:0 rt_prio:0 -----------------
Quoted 8 lines
=> started at:
kernel_fpu_begin+0x21/0x60
=> ended at:_mmx_memcpy+0x131/0x180
=======>
00000001 0.000ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy) 00000001 0.730ms (+0.730ms):sub_preempt_count(_mmx_memcpy) 00000001 0.730ms (+0.000ms):_mmx_memcpy(check_preempt_timing) 00000001 0.730ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy)
As far as I can see sub_preempt_count() is part of the latency measuring component of the voluntary preempt patch so, like you, I'm not sure whether this report makes any sense.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 13 lines · Peter Williams
Quoted 11 lines · [email protected] wrote:
--- Peter Williams <[email protected]> wrote:
-----------------=> started at:
kernel_fpu_begin+0x21/0x60
=> ended at:_mmx_memcpy+0x131/0x180
=======>
00000001 0.000ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy) 00000001 0.730ms (+0.730ms):sub_preempt_count(_mmx_memcpy) 00000001 0.730ms (+0.000ms):_mmx_memcpy(check_preempt_timing) 00000001 0.730ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy)
Quoted 3 lines
As far as I can see
sub_preempt_count()is part of the latency measuring component of the voluntary preempt patch so, like you, I'm not sure whether this report makes any sense.
Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
Lee
Quoted 14 lines · Lee Revell
Quoted 13 lines · Peter Williams
Quoted 11 lines · [email protected] wrote:
--- Peter Williams <[email protected]> wrote:
-----------------=> started at:
kernel_fpu_begin+0x21/0x60
=> ended at:_mmx_memcpy+0x131/0x180
=======>
00000001 0.000ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy) 00000001 0.730ms (+0.730ms):sub_preempt_count(_mmx_memcpy) 00000001 0.730ms (+0.000ms):_mmx_memcpy(check_preempt_timing) 00000001 0.730ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy)
Quoted 8 lines
Quoted 3 lines
As far as I can see
sub_preempt_count()is part of the latency measuring component of the voluntary preempt patch so, like you, I'm not sure whether this report makes any sense.Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
Hmm, after rereading the entire thread, I am not sure that voluntary preemption will help you here. Voluntary preemption (and preemption in general) deals with the situation in which you have a high priority task, often the highest priority task on the system, that spends most of its time sleeping on some resource, and this task needs to run as soon as possible once it becomes runnable. In that situation the scheduler doesn't have a very difficult decision, there is no question that it should run this task ASAP. How long 'ASAP' is depends on how long it takes whatever task was running when our high priority task became runnable to yield the processor. The scheduler has a very easy job here, there is only one right thing to do. Also the intervals involved are very small, usually less than 1ms, whereas you are talking about a variance of several seconds.
In the situation you describe, you have two tasks running at the same base priority, and the scheduler does not seem to be doing a good job balancing them. This is a different situation, and much more dependent on the scheduling policy.
Lee
Quoted 26 lines · Lee Revell
Quoted 16 lines · Lee Revell
Quoted 14 lines · Peter Williams
Quoted 11 lines · [email protected] wrote:
--- Peter Williams <[email protected]> wrote:
-----------------=> started at:
kernel_fpu_begin+0x21/0x60
=> ended at:_mmx_memcpy+0x131/0x180
=======>
00000001 0.000ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy) 00000001 0.730ms (+0.730ms):sub_preempt_count(_mmx_memcpy) 00000001 0.730ms (+0.000ms):_mmx_memcpy(check_preempt_timing) 00000001 0.730ms (+0.000ms):kernel_fpu_begin(_mmx_memcpy)Quoted 6 lines
Quoted 1 line
As far as I can see
sub_preempt_count()is part of the latency measuring component of the voluntary preempt patch so, like you, I'm not sure whether this report makes any sense.Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
Quoted 13 lines
Hmm, after rereading the entire thread, I am not sure that voluntary preemption will help you here. Voluntary preemption (and preemption in general) deals with the situation in which you have a high priority task, often the highest priority task on the system, that spends most of its time sleeping on some resource, and this task needs to run as soon as possible once it becomes runnable. In that situation the scheduler doesn't have a very difficult decision, there is no question that it should run this task ASAP. How long 'ASAP' is depends on how long it takes whatever task was running when our high priority task became runnable to yield the processor. The scheduler has a very easy job here, there is only one right thing to do. Also the intervals involved are very small, usually less than 1ms, whereas you are talking about a variance of several seconds.
My understanding is that it's a very occasional (every few hours) time out of several seconds not something with a variance of several seconds.
So it seems to fit the characteristics of the type of problem that you
are hunting i.e. every now and then going through a (rarely exercised) code path that hogs the CPU for too long. But, as you say, the time scales are radically different.
Quoted 5 lines
In the situation you describe, you have two tasks running at the same base priority, and the scheduler does not seem to be doing a good job balancing them. This is a different situation, and much more dependent on the scheduling policy.
The mode in which the scheduler was being used had all priority fiddling (except promotion) turned off so the tasks should have been just round robinning with each other. Also, the time outs are fairly rare (every few hours according to Nicolas's e-mail) and happen with several different schedulers (with ZAPHOD (the one being used by Nicolas) and Con's staircase schedulers having less problem than the vanilla scheduler) which is why I thought it might be something outside the scheduler. Perhaps it's something outside the kernel?
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 9 lines · --- Peter Williams
The mode in which the scheduler was being used had all priority fiddling (except promotion) turned off so the tasks should have been just round robinning with each other. Also, the time outs are fairly rare (every few hours according to Nicolas's e-mail) and happen with several different schedulers (with ZAPHOD (the one being used by Nicolas) and Con's staircase schedulers having less problem than the vanilla scheduler) which is why I thought it might be something outside the scheduler. Perhaps it's something outside the kernel?
I can add to this that this problem occured on a variety of systems, single CPU Pentium IIIs and 4s, Athlon, dual PIIIs ;
the one thing in common is that everything works fine on all those machines with 2.4, but breaks with 2.5 (or redhat 2.4 kernel with some backported code). When I do the tests, the only thing I switch is the kernel and reboot.
It's true that it could be something broken outside of the scheduling code (like the way IRQ events are handled maybe, or the way signals are delivered).
The one difference between the artificial test (from the original post) and the real life test I do now, is that the real test combines disks I/O, network I/O (TCP/IP and UDP) and several multithreaded processes.
Where things are kind of bad is that I am far from saturating the machine (the load average is less than 2), but still some processes get those annoying timeouts.
Nicolas
Quoted 10 lines · [email protected] wrote:
Quoted 2 lines · --- Peter Williams
The mode in which the scheduler was being used had all priority fiddling (except promotion) turned off so the tasks should have been just round robinning with each other. Also, the time outs are fairly rare (every few hours according to Nicolas's e-mail) and happen with several different schedulers (with ZAPHOD (the one being used by Nicolas) and Con's staircase schedulers having less problem than the vanilla scheduler) which is why I thought it might be something outside the scheduler. Perhaps it's something outside the kernel?
I can add to this that this problem occured on a variety of systems, single CPU Pentium IIIs and 4s, Athlon, dual PIIIs ;
the one thing in common is that everything works fine on all those machines with 2.4, but breaks with 2.5 (or redhat 2.4 kernel with some backported code).
I don't suppose you know what the backported code was? If you could provide a patch of the backport it might provide some clues.
Quoted 11 lines
When I do the tests, the only thing I switch is the kernel and reboot.
It's true that it could be something broken outside of the scheduling code (like the way IRQ events are handled maybe, or the way signals are delivered).
The one difference between the artificial test (from the original post) and the real life test I do now, is that the real test combines disks I/O, network I/O (TCP/IP and UDP) and several multithreaded processes.
Where things are kind of bad is that I am far from saturating the machine (the load average is less than 2), but still some processes get those annoying timeouts.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 4 lines · --- Lee Revell
Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
No, it's a single CPU Athlon 1800+, the kernel is compiled in with support for SMP system, but that should not have any impact.
Nicolas
Quoted 8 lines · On Sat, 2004-08-28 at 21:19, [email protected] wrote:
Quoted 4 lines · --- Lee Revell
Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
No, it's a single CPU Athlon 1800+, the kernel is compiled in with support for SMP system, but that should not have any impact.
I believe people were also having problems running SMP kernels with the VP patches on UP. Try the latest version. Q3 as of this writing.
Lee
Quoted 9 lines · Lee Revell
Quoted 7 lines · On Sat, 2004-08-28 at 21:19, [email protected] wrote:
Quoted 4 lines · --- Lee Revell
Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
Quoted 7 lines
Quoted 3 lines
No, it's a single CPU Athlon 1800+, the kernel is compiled in with support for SMP system, but that should not have any impact.
I believe people were also having problems running SMP kernels with the VP patches on UP. Try the latest version. Q3 as of this writing.
Nicolas,
I'll generate a combined patch and let you know when it's ready. In
the mean time, could you try increasing the "base_promotion_interval" to about twice the time slice size?
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 9 lines · --- Peter Williams
Quoted 4 lines
Nicolas,
I'll generate a combined patch and let you know when it's ready. In
the mean time, could you try increasing the "base_promotion_interval" to about twice the time slice size?A patch for the ZAPHOD compiler on top of the R5 voluntary preemption patches is available at:
Quoted 7 lines
<http://prdownloads.sourceforge.net/cpuse/patch-2.6.9-rc1-vp-R5-zaphod-v5.0.1?download>
Due to the fact that the R5 patch requires the bk12 patch to be applied to 2.6.9-rc1 before it is applied, generating a combined patch resulted in a very large patch (and lots of duplicated effort) so this patch is not a combined patch but is relative to a 2.6.9-rc1 kernel with bk12 and
voluntary preempt R5 patches already applied.
I have been running for several days with this patched kernel, with 0 for max_ia_bonus
and 0 for max_tpt_bonus
in "pb" mode
and there are no slow downs at all: my system is running very steadily now!
So it seems that we are getting somewhere!!
Nicolas
Quoted 8 lines · On Sat, 2004-08-28 at 21:19, [email protected] wrote:
Quoted 4 lines · --- Lee Revell
Is this an SMP machine? There were problems with that version of the voluntary preemption patches on SMP. The latest version, Q3, should fix these.
No, it's a single CPU Athlon 1800+, the kernel is compiled in with support for SMP system, but that should not have any impact.
It shouldn't, but it can. For example taking a spinlock just disables preemption with a UP kernel, but with an SMP kernel I believe you can actually end up spinning. You would have to have hit a locking bug or race condition for this to happen. Just to be certain, can you reproduce the problem with a UP kernel?
Lee