2.6.14-rt22 (and mainline): netstat -anop triggers excessive latencies
Subsystems (4)
Here is a 13ms+ trace caused by "netstat -anop". I noticed that the preempt count is 0 so there are no locks to drop, would adding a cond_resched() be acceptable? Or should this be considered more evidence of the need for softirq preemption?
preemption latency trace v1.1.5 on 2.6.14-rt22
--------------------------------------------------------------------
latency: 13824 us, #16533/16533, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)
-----------------
| task: softirq-timer/0-3 (uid:0 nice:0 policy:1 rt_prio:1)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
netstat-29157 0D.h2 0us : __trace_start_sched_wakeup (try_to_wake_up)
netstat-29157 0D.h2 1us : __trace_start_sched_wakeup <<...>-3> (62 0)
netstat-29157 0D.h. 2us : wake_up_process (wakeup_softirqd)
netstat-29157 0D.h. 3us : scheduler_tick (update_process_times)
netstat-29157 0D.h. 3us : sched_clock (scheduler_tick)
netstat-29157 0D.h1 5us : task_timeslice (scheduler_tick)
netstat-29157 0D.h. 6us : run_posix_cpu_timers (update_process_times)
netstat-29157 0D.h1 7us : note_interrupt (__do_IRQ)
netstat-29157 0D.h1 8us : end_8259A_irq (__do_IRQ)
netstat-29157 0D.h1 8us+: enable_8259A_irq (end_8259A_irq)
netstat-29157 0DnH. 10us : irq_exit (do_IRQ)
netstat-29157 0Dns. 11us < (608)
netstat-29157 0.ns. 13us : preempt_schedule (established_get_first)
netstat-29157 0.ns. 13us : cond_resched_softirq (established_get_first)
netstat-29157 0.ns. 14us : preempt_schedule (established_get_first)
netstat-29157 0.ns. 15us : cond_resched_softirq (established_get_first)
netstat-29157 0.ns. 16us : preempt_schedule (established_get_first)
netstat-29157 0.ns. 17us : cond_resched_softirq (established_get_first)
netstat-29157 0.ns. 18us : preempt_schedule (established_get_first) [ at about 1ms switches to established_get_next ]
netstat-29157 0.ns. 13765us : cond_resched_softirq (established_get_next) netstat-29157 0.ns. 13766us : preempt_schedule (established_get_next) netstat-29157 0.ns. 13767us : cond_resched_softirq (established_get_next) netstat-29157 0.ns. 13768us : preempt_schedule (established_get_next) netstat-29157 0.ns. 13769us : cond_resched_softirq (established_get_next) netstat-29157 0.ns. 13770us : tcp_seq_stop (seq_read) netstat-29157 0.ns. 13771us : local_bh_enable (tcp_seq_stop) netstat-29157 0.n.1 13772us : do_softirq (local_bh_enable) netstat-29157 0D.s. 13774us : __do_softirq (do_softirq) netstat-29157 0D.s. 13775us : ___do_softirq (__do_softirq) netstat-29157 0..s. 13776us : run_timer_softirq (___do_softirq) netstat-29157 0..s. 13778us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13779us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13779us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13780us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13781us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13782us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13783us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13784us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13785us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13786us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13787us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13788us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13789us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13790us : cond_resched_softirq (run_timer_softirq) netstat-29157 0..s. 13791us : ktimer_run_queues (run_timer_softirq) netstat-29157 0..s. 13792us : get_realtime_clock (ktimer_run_queues) netstat-29157 0..s. 13793us : read_tsc (get_realtime_clock) netstat-29157 0..s. 13795us : ktime_add_ns (get_realtime_clock) netstat-29157 0..s. 13796us : get_monotonic_clock (ktimer_run_queues) netstat-29157 0..s. 13797us : read_tsc (get_monotonic_clock) netstat-29157 0..s. 13798us+: ktime_add_ns (get_monotonic_clock) netstat-29157 0..s. 13802us : __wake_up (run_timer_softirq) netstat-29157 0D.s1 13802us : __wake_up_common (__wake_up) netstat-29157 0..s. 13804us : cond_resched_all (___do_softirq) netstat-29157 0..s. 13804us : cond_resched_softirq (cond_resched_all) netstat-29157 0.n.. 13805us : preempt_schedule (local_bh_enable) netstat-29157 0Dn.. 13806us : __schedule (preempt_schedule) netstat-29157 0Dn.. 13807us : profile_hit (__schedule) netstat-29157 0Dn.1 13808us+: sched_clock (__schedule)
<...>-3 0D..2 13814us+: __switch_to (__schedule) <...>-3 0D..2 13820us : __schedule <netstat-29157> (75 62) <...>-3 0...1 13821us : trace_stop_sched_switched (__schedule) <...>-3 0D..2 13821us+: trace_stop_sched_switched <<...>-3> (62 0) <...>-3 0D..2 13824us : trace_stop_sched_switched (__schedule)
Quoted 8 lines · Lee Revell
Here is a 13ms+ trace caused by "netstat -anop". I noticed that the preempt count is 0 so there are no locks to drop, would adding a
cond_resched()be acceptable? Or should this be considered more evidence of the need for softirq preemption?preemption latency trace v1.1.5 on 2.6.14-rt22
--------------------------------------------------------------------
latency: 13824 us, #16533/16533, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)
Quoted 16 lines
-----------------
| task: softirq-timer/0-3 (uid:0 nice:0 policy:1 rt_prio:1)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
netstat-29157 0D.h2 0us : __trace_start_sched_wakeup (try_to_wake_up)
netstat-29157 0D.h2 1us : __trace_start_sched_wakeup <<...>-3> (62 0)
netstat-29157 0D.h. 2us : wake_up_process (wakeup_softirqd)Hi Lee, I've seen the same thing here. I wrote up this patch as a case study to see if it can work and it does. Now this sacrifices memory for speed. I added a bitmask in the inet_hash to keep track of all the buckets that are used, and use the find_next_bit to search the list in established_get_first.
After running the netstat -anop I get the following:
preemption latency trace v1.1.5 on 2.6.14-rt22
--------------------------------------------------------------------
latency: 377 us, #225/225, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)
-----------------
| task: softirq-timer/0-3 (uid:0 nice:0 policy:1 rt_prio:1)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
<idle>-0 0D.h4 0us : __trace_start_sched_wakeup (try_to_wake_up)
<idle>-0 0D.h4 0us : __trace_start_sched_wakeup <<...>-3> (62 0)
<idle>-0 0Dnh3 0us : try_to_wake_up <<...>-3> (62 8c)
<idle>-0 0Dnh2 1us : preempt_schedule (try_to_wake_up)
<idle>-0 0Dnh2 1us : wake_up_process (wakeup_softirqd)[snip]
So it really does improve the latency here. Now is this worth the overhead? This might be useful in other places to.
Ingo,
What do you think?
-- Steve
include/net/inet_hashtables.h +32 -1
@@ -101,6 +101,7 @@* is for TIME_WAIT sockets only. */ struct inet_ehash_bucket *ehash; +unsigned long *ebitmask;/* Ok, let's try this, I give up, we do need a local binding * TCP hash as well as the others for fast bind/connect.
@@ -157,6 +158,13 @@return &hashinfo->ehash[hash & (hashinfo->ehash_size -1 )]; } +static inline unsigned int inet_ehash_index ( +struct inet_hashinfo *hashinfo, +unsigned int hash) +{ +return hash & (hashinfo->ehash_size -1 ); +} +extern struct inet_bind_bucket * inet_bind_bucket_create(kmem_cache_t *cachep, struct inet_bind_hashbucket *head,
@@ -229,11 +237,25 @@wake_up (&hashinfo->lhash_wait); } +static inline void __inet_hash_setbit (unsigned long *bitmask,unsigned int index) +{ +if (bitmask) +set_bit (index, bitmask); +} + +static inline void __inet_hash_clearbit (unsigned long *bitmask,unsigned int index) +{ +if (bitmask) +clear_bit (index, bitmask); +} +static inline void __inet_hash (struct inet_hashinfo *hashinfo,struct sock *sk,const int listen_possible) {struct hlist_head *list;rwlock_t *lock; +unsigned long *bitmask =NULL ; +unsigned int index =0 ;BUG_TRAP (sk_unhashed (sk));if (listen_possible && sk->sk_state ==TCP_LISTEN ) {
@@ -243,12 +265,15 @@ }else {struct inet_ehash_bucket *head; sk->sk_hash =inet_sk_ehashfn (sk); + index =inet_ehash_index (hashinfo, sk->sk_hash); head =inet_ehash_bucket (hashinfo, sk->sk_hash); list = &head->chain; lock = &head->lock; + bitmask = hashinfo->ebitmask;write_lock (lock); }__sk_add_node (sk, list); +__inet_hash_setbit (bitmask, index);sock_prot_inc_use (sk->sk_prot);write_unlock (lock);if (listen_possible && sk->sk_state ==TCP_LISTEN )
@@ -267,6 +292,8 @@static inline void inet_unhash (struct inet_hashinfo *hashinfo,struct sock *sk) {rwlock_t *lock; +unsigned long *bitmask =NULL ; +unsigned int index =0 ;if (sk_unhashed (sk)) goto out;
@@ -276,12 +303,16 @@inet_listen_wlock (hashinfo); lock = &hashinfo->lhash_lock; }else { + index =inet_ehash_index (hashinfo, sk->sk_hash); lock = &inet_ehash_bucket (hashinfo, sk->sk_hash)->lock; + bitmask = hashinfo->ebitmask;write_lock_bh (lock); } -if (__sk_del_node_init (sk)) +if (__sk_del_node_init (sk)) { +__inet_hash_clearbit (bitmask, index);sock_prot_dec_use (sk->sk_prot); + }write_unlock_bh (lock); out:if (sk->sk_state ==TCP_LISTEN )
mm/page_alloc.c +27 -0
@@ -2601,3 +2601,30 @@return table; } + +void *__init alloc_large_system_bitmask (char *bitmaskname, +unsigned long bits,int flags) +{ +unsigned long words = bits / (sizeof (unsigned long )*8 ); +unsigned long size = words *sizeof (unsigned long ); +unsigned long *bitmask =NULL ; + +if (flags &HASH_EARLY ) + bitmask =alloc_bootmem (size); +else if (hashdist) + bitmask =__vmalloc (size,GFP_ATOMIC ,PAGE_KERNEL ); +else { + bitmask =kmalloc (size,GFP_ATOMIC ); +if (!bitmask) { +unsigned long order; +for (order =0 ; ((1UL << order) <<PAGE_SHIFT ) < size; order++) + ; + bitmask = (void *)__get_free_pages (GFP_ATOMIC , order); + } + } + +if (!bitmask) +panic ("Failed to allocate %s bitmask\n" , bitmaskname); + +return bitmask; +}
net/ipv4/tcp.c +6 -0
@@ -2038,6 +2038,8 @@ }__setup ("thash_entries=" , set_thash_entries); +void *__init alloc_large_system_bitmask (char *bitmaskname, +unsigned long bits,int flags);void __init tcp_init (void ) {struct sk_buff *skb =NULL ;
@@ -2071,6 +2073,10 @@NULL , 0); tcp_hashinfo.ehash_size = (1 << tcp_hashinfo.ehash_size) >>1 ; + tcp_hashinfo.ebitmask = +alloc_large_system_bitmask ("TCP established" , + tcp_hashinfo.ehash_size, +HASH_HIGHMEM );for (i =0 ; i < (tcp_hashinfo.ehash_size <<1 ); i++) {rwlock_init (&tcp_hashinfo.ehash[i].lock);
net/ipv4/tcp_ipv4.c +6 -1
@@ -1590,7 +1590,12 @@struct tcp_iter_state * st = seq->private;void *rc =NULL ; -for (st->bucket =0 ; st->bucket < tcp_hashinfo.ehash_size; ++st->bucket) { +for (st->bucket =find_first_bit (tcp_hashinfo.ebitmask, + tcp_hashinfo.ehash_size); + st->bucket < tcp_hashinfo.ehash_size; + st->bucket =find_next_bit (tcp_hashinfo.ebitmask, + tcp_hashinfo.ehash_size, + st->bucket+1 )) {struct sock *sk;struct hlist_node *node;struct inet_timewait_sock *tw;
Quoted 2 lines · Steven Rostedt
So it really does improve the latency here. Now is this worth the overhead? This might be useful in other places to.
Thanks, I'll try this.
But, AFAICT CONFIG_PREEMPT_SOFTIRQS is a much smaller patch... ;-)
Lee
Quoted 55 lines · Steven Rostedt
Quoted 8 lines · Lee Revell
Here is a 13ms+ trace caused by "netstat -anop". I noticed that the preempt count is 0 so there are no locks to drop, would adding a
cond_resched()be acceptable? Or should this be considered more evidence of the need for softirq preemption?preemption latency trace v1.1.5 on 2.6.14-rt22
--------------------------------------------------------------------
latency: 13824 us, #16533/16533, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)Quoted 16 lines
----------------- | task: softirq-timer/0-3 (uid:0 nice:0 policy:1 rt_prio:1) ----------------- _------=> CPU# / _-----=> irqs-off | / _----=> need-resched || / _---=> hardirq/softirq ||| / _--=> preempt-depth |||| / ||||| delay cmd pid ||||| time | caller \ / ||||| \ | / netstat-29157 0D.h2 0us : __trace_start_sched_wakeup (try_to_wake_up) netstat-29157 0D.h2 1us : __trace_start_sched_wakeup <<...>-3> (62 0) netstat-29157 0D.h. 2us : wake_up_process (wakeup_softirqd)Hi Lee, I've seen the same thing here. I wrote up this patch as a case study to see if it can work and it does. Now this sacrifices memory for speed. I added a bitmask in the
inet_hashto keep track of all the buckets that are used, and use thefind_next_bitto search the list inestablished_get_first.After running the netstat -anop I get the following:
preemption latency trace v1.1.5 on 2.6.14-rt22
--------------------------------------------------------------------
latency: 377 us, #225/225, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)----------------- | task: softirq-timer/0-3 (uid:0 nice:0 policy:1 rt_prio:1) ----------------- _------=> CPU# / _-----=> irqs-off | / _----=> need-resched || / _---=> hardirq/softirq ||| / _--=> preempt-depth |||| / ||||| delay cmd pid ||||| time | caller \ / ||||| \ | / <idle>-0 0D.h4 0us : __trace_start_sched_wakeup (try_to_wake_up) <idle>-0 0D.h4 0us : __trace_start_sched_wakeup <<...>-3> (62 0) <idle>-0 0Dnh3 0us : try_to_wake_up <<...>-3> (62 8c) <idle>-0 0Dnh2 1us : preempt_schedule (try_to_wake_up) <idle>-0 0Dnh2 1us : wake_up_process (wakeup_softirqd)
Quoted 4 lines
[snip]
So it really does improve the latency here. Now is this worth the overhead? This might be useful in other places to.
Any chance you can regenerate the patch against 2.6.15-rc5-rt4?
Lee
Quoted 7 lines · Lee Revell
Quoted 4 lines
[snip]
So it really does improve the latency here. Now is this worth the overhead? This might be useful in other places to.
Any chance you can regenerate the patch against 2.6.15-rc5-rt4?
Sure, if I can find the damn thing. Too many kernels, and too many patch directories.
-- Steve
Quoted 4 lines · Steven Rostedt
Quoted 2 lines · Lee Revell
Quoted 2 lines
[snip]
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
So it really does improve the latency here. Now is this worth the overhead? This might be useful in other places to.
Any chance you can regenerate the patch against 2.6.15-rc5-rt4?
Sure, if I can find the damn thing. Too many kernels, and too many patch directories.
Never mind, I applied it by hand. I'll let you know how it works.
Lee
Quoted 8 lines · Lee Revell
Quoted 7 lines · Steven Rostedt
Quoted 5 lines · Lee Revell
Quoted 4 lines
[snip]
So it really does improve the latency here. Now is this worth the overhead? This might be useful in other places to.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Any chance you can regenerate the patch against 2.6.15-rc5-rt4?
Sure, if I can find the damn thing. Too many kernels, and too many patch directories.
Never mind, I applied it by hand. I'll let you know how it works.
OK, I did find it though, and it only had one rej. So you probably can easily do that change yourself.
Aw heck, here it is anyway. (look everybody, a patch pulled in with pine!). Complements of quilt.
-- Steve
include/net/inet_hashtables.h +32 -1
@@ -101,6 +101,7 @@* is for TIME_WAIT sockets only. */ struct inet_ehash_bucket *ehash; +unsigned long *ebitmask;/* Ok, let's try this, I give up, we do need a local binding * TCP hash as well as the others for fast bind/connect.
@@ -155,6 +156,13 @@return &hashinfo->ehash[hash & (hashinfo->ehash_size -1 )]; } +static inline unsigned int inet_ehash_index ( +struct inet_hashinfo *hashinfo, +unsigned int hash) +{ +return hash & (hashinfo->ehash_size -1 ); +} +extern struct inet_bind_bucket * inet_bind_bucket_create(kmem_cache_t *cachep, struct inet_bind_hashbucket *head,
@@ -227,11 +235,25 @@wake_up (&hashinfo->lhash_wait); } +static inline void __inet_hash_setbit (unsigned long *bitmask,unsigned int index) +{ +if (bitmask) +set_bit (index, bitmask); +} + +static inline void __inet_hash_clearbit (unsigned long *bitmask,unsigned int index) +{ +if (bitmask) +clear_bit (index, bitmask); +} +static inline void __inet_hash (struct inet_hashinfo *hashinfo,struct sock *sk,const int listen_possible) {struct hlist_head *list;rwlock_t *lock; +unsigned long *bitmask =NULL ; +unsigned int index =0 ;BUG_TRAP (sk_unhashed (sk));if (listen_possible && sk->sk_state ==TCP_LISTEN ) {
@@ -241,12 +263,15 @@ }else {struct inet_ehash_bucket *head; sk->sk_hash =inet_sk_ehashfn (sk); + index =inet_ehash_index (hashinfo, sk->sk_hash); head =inet_ehash_bucket (hashinfo, sk->sk_hash); list = &head->chain; lock = &head->lock; + bitmask = hashinfo->ebitmask;write_lock (lock); }__sk_add_node (sk, list); +__inet_hash_setbit (bitmask, index);sock_prot_inc_use (sk->sk_prot);write_unlock (lock);if (listen_possible && sk->sk_state ==TCP_LISTEN )
@@ -265,6 +290,8 @@static inline void inet_unhash (struct inet_hashinfo *hashinfo,struct sock *sk) {rwlock_t *lock; +unsigned long *bitmask =NULL ; +unsigned int index =0 ;if (sk_unhashed (sk)) goto out;
@@ -274,12 +301,16 @@inet_listen_wlock (hashinfo); lock = &hashinfo->lhash_lock; }else { + index =inet_ehash_index (hashinfo, sk->sk_hash); lock = &inet_ehash_bucket (hashinfo, sk->sk_hash)->lock; + bitmask = hashinfo->ebitmask;write_lock_bh (lock); } -if (__sk_del_node_init (sk)) +if (__sk_del_node_init (sk)) { +__inet_hash_clearbit (bitmask, index);sock_prot_dec_use (sk->sk_prot); + }write_unlock_bh (lock); out:if (sk->sk_state ==TCP_LISTEN )
mm/page_alloc.c +27 -0
@@ -2664,3 +2664,30 @@return table; } + +void *__init alloc_large_system_bitmask (char *bitmaskname, +unsigned long bits,int flags) +{ +unsigned long words = bits / (sizeof (unsigned long )*8 ); +unsigned long size = words *sizeof (unsigned long ); +unsigned long *bitmask =NULL ; + +if (flags &HASH_EARLY ) + bitmask =alloc_bootmem (size); +else if (hashdist) + bitmask =__vmalloc (size,GFP_ATOMIC ,PAGE_KERNEL ); +else { + bitmask =kmalloc (size,GFP_ATOMIC ); +if (!bitmask) { +unsigned long order; +for (order =0 ; ((1UL << order) <<PAGE_SHIFT ) < size; order++) + ; + bitmask = (void *)__get_free_pages (GFP_ATOMIC , order); + } + } + +if (!bitmask) +panic ("Failed to allocate %s bitmask\n" , bitmaskname); + +return bitmask; +}
net/ipv4/tcp.c +6 -0
@@ -2039,6 +2039,8 @@ }__setup ("thash_entries=" , set_thash_entries); +void *__init alloc_large_system_bitmask (char *bitmaskname, +unsigned long bits,int flags);void __init tcp_init (void ) {struct sk_buff *skb =NULL ;
@@ -2071,6 +2073,10 @@NULL , 0); tcp_hashinfo.ehash_size = (1 << tcp_hashinfo.ehash_size) >>1 ; + tcp_hashinfo.ebitmask = +alloc_large_system_bitmask ("TCP established" , + tcp_hashinfo.ehash_size, +HASH_HIGHMEM );for (i =0 ; i < (tcp_hashinfo.ehash_size <<1 ); i++) {rwlock_init (&tcp_hashinfo.ehash[i].lock);INIT_HLIST_HEAD (&tcp_hashinfo.ehash[i].chain);
net/ipv4/tcp_ipv4.c +6 -1
@@ -1581,7 +1581,12 @@struct tcp_iter_state * st = seq->private;void *rc =NULL ; -for (st->bucket =0 ; st->bucket < tcp_hashinfo.ehash_size; ++st->bucket) { +for (st->bucket =find_first_bit (tcp_hashinfo.ebitmask, + tcp_hashinfo.ehash_size); + st->bucket < tcp_hashinfo.ehash_size; + st->bucket =find_next_bit (tcp_hashinfo.ebitmask, + tcp_hashinfo.ehash_size, + st->bucket+1 )) {struct sock *sk;struct hlist_node *node;struct inet_timewait_sock *tw;
Quoted 5 lines · Steven Rostedt
OK, I did find it though, and it only had one rej. So you probably can easily do that change yourself.
Aw heck, here it is anyway. (look everybody, a patch pulled in with pine!). Complements of quilt.
Nope, does not help. We still do way too much work in softirq context.
preemption latency trace v1.1.5 on 2.6.15-rc5-rt4
--------------------------------------------------------------------
latency: 11184 us, #13458/13458, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)
-----------------
| task: softirq-timer/0-3 (uid:0 nice:0 policy:1 rt_prio:1)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
netstat-6169 0D.h2 0us : __trace_start_sched_wakeup (try_to_wake_up)
netstat-6169 0D.h2 1us : __trace_start_sched_wakeup <<...>-3> (62 0)
netstat-6169 0D.h. 2us : wake_up_process (wakeup_softirqd)
netstat-6169 0D.h. 3us : scheduler_tick (update_process_times)
netstat-6169 0D.h. 3us : sched_clock (scheduler_tick)
netstat-6169 0D.h1 5us : task_timeslice (scheduler_tick)
netstat-6169 0D.h. 6us : run_posix_cpu_timers (update_process_times)
netstat-6169 0D.h1 7us : note_interrupt (__do_IRQ)
netstat-6169 0D.h1 7us : end_8259A_irq (__do_IRQ)
netstat-6169 0D.h1 8us+: enable_8259A_irq (end_8259A_irq)
netstat-6169 0DnH1 10us : irq_exit (do_IRQ)
netstat-6169 0Dns1 11us < (608)
netstat-6169 0.ns. 12us : preempt_schedule (established_get_first)
netstat-6169 0.ns. 13us : find_next_bit (established_get_first)
netstat-6169 0.ns. 13us : cond_resched_softirq (established_get_first)
netstat-6169 0.ns. 14us : preempt_schedule (established_get_first)
netstat-6169 0.ns. 15us : find_next_bit (established_get_first)
netstat-6169 0.ns. 16us : cond_resched_softirq (established_get_first)
netstat-6169 0.ns. 17us : preempt_schedule (established_get_first)
netstat-6169 0.ns. 17us : find_next_bit (established_get_first)
netstat-6169 0.ns. 18us : cond_resched_softirq (established_get_first)
netstat-6169 0.ns. 19us : preempt_schedule (established_get_first)
netstat-6169 0.ns. 20us : find_next_bit (established_get_first)[...]
netstat-6169 0.ns. 11098us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11098us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11099us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11100us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11101us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11102us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11103us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11103us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11104us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11105us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11106us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11107us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11107us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11108us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11109us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11110us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11110us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11111us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11112us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11113us : preempt_schedule (established_get_next) netstat-6169 0.ns. 11114us : cond_resched_softirq (established_get_next) netstat-6169 0.ns. 11115us : tcp_seq_stop (seq_read) netstat-6169 0.ns. 11115us : local_bh_enable (tcp_seq_stop) netstat-6169 0.n.1 11116us : do_softirq (local_bh_enable) netstat-6169 0D.s. 11118us : __do_softirq (do_softirq) netstat-6169 0D.s. 11119us : ___do_softirq (__do_softirq) netstat-6169 0..s. 11120us : run_timer_softirq (___do_softirq) netstat-6169 0..s. 11122us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11123us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11124us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11125us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11126us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11127us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11128us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11129us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11129us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11130us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11131us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11132us : cond_resched_softirq (run_timer_softirq) netstat-6169 0..s. 11133us : hrtimer_run_queues (run_timer_softirq) netstat-6169 0..s. 11134us : get_realtime_clock (hrtimer_run_queues) netstat-6169 0..s. 11136us : read_tsc (get_realtime_clock) netstat-6169 0..s. 11137us : get_monotonic_clock (hrtimer_run_queues) netstat-6169 0..s. 11138us+: read_tsc (get_monotonic_clock) netstat-6169 0..s. 11142us : timeofday_periodic_hook (run_timer_softirq) netstat-6169 0D.s1 11143us+: read_tsc (timeofday_periodic_hook) netstat-6169 0D.s1 11146us+: ntp_advance (timeofday_periodic_hook) netstat-6169 0D.s1 11150us : get_next_clocksource (timeofday_periodic_hook) netstat-6169 0D.s1 11152us : tsc_update_callback (timeofday_periodic_hook) netstat-6169 0D.s1 11153us : ntp_get_ppm_adjustment (timeofday_periodic_hook) netstat-6169 0D.s1 11154us : update_legacy_time_values (timeofday_periodic_hook) netstat-6169 0D.s2 11156us : set_normalized_timespec (update_legacy_time_values) netstat-6169 0..s. 11157us : mod_timer (timeofday_periodic_hook) netstat-6169 0..s. 11158us : __mod_timer (mod_timer) netstat-6169 0..s. 11158us : lock_timer_base (__mod_timer) netstat-6169 0D.s1 11160us : internal_add_timer (__mod_timer) netstat-6169 0..s. 11161us : cond_resched_all (run_timer_softirq) netstat-6169 0..s. 11162us : cond_resched_softirq (cond_resched_all) netstat-6169 0..s. 11164us : __wake_up (run_timer_softirq) netstat-6169 0D.s1 11164us : __wake_up_common (__wake_up) netstat-6169 0..s. 11165us : cond_resched_all (___do_softirq) netstat-6169 0..s. 11166us : cond_resched_softirq (cond_resched_all) netstat-6169 0.n.. 11167us : preempt_schedule (local_bh_enable) netstat-6169 0Dn.. 11168us : __schedule (preempt_schedule) netstat-6169 0Dn.. 11169us : profile_hit (__schedule) netstat-6169 0Dn.1 11170us+: sched_clock (__schedule) <...>-3 0D..2 11176us+: __switch_to (__schedule) <...>-3 0D..2 11179us : __schedule <netstat-6169> (75 62) <...>-3 0...1 11180us : trace_stop_sched_switched (__schedule) <...>-3 0D..2 11181us+: trace_stop_sched_switched <<...>-3> (62 0) <...>-3 0D..2 11183us : trace_stop_sched_switched (__schedule)
Quoted 9 lines · Lee Revell
Quoted 5 lines · Steven Rostedt
OK, I did find it though, and it only had one rej. So you probably can easily do that change yourself.
Aw heck, here it is anyway. (look everybody, a patch pulled in with pine!). Complements of quilt.
Nope, does not help. We still do way too much work in softirq context.
Are you still doing the 'netstat -anop'? And what does it look like. On my test machine the max latency I get is 83 usecs.
-- Steve
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · Lee Revell
Quoted 3 lines · Steven Rostedt
OK, I did find it though, and it only had one rej. So you probably can easily do that change yourself.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Aw heck, here it is anyway. (look everybody, a patch pulled in with pine!). Complements of quilt.
Nope, does not help. We still do way too much work in softirq context.
Are you still doing the 'netstat -anop'? And what does it look like. On my test machine the max latency I get is 83 usecs.
Oh, try out 2.6.15-rc7-rt1, it has the patch as well. That's what I was testing.
-- Steve
Quoted 16 lines · Steven Rostedt
Quoted 13 lines · Steven Rostedt
Quoted 9 lines · Lee Revell
Quoted 5 lines · Steven Rostedt
OK, I did find it though, and it only had one rej. So you probably can easily do that change yourself.
Aw heck, here it is anyway. (look everybody, a patch pulled in with pine!). Complements of quilt.
Nope, does not help. We still do way too much work in softirq context.
Quoted 6 lines
Quoted 3 lines
Are you still doing the 'netstat -anop'? And what does it look like. On my test machine the max latency I get is 83 usecs.
Oh, try out 2.6.15-rc7-rt1, it has the patch as well. That's what I was testing.
OK, I'll try that. Meanwhile here's the output:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3733/portmap off (0.00/0/0) tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 4119/pdnsd off (0.00/0/0) tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 4175/sshd off (0.00/0/0) tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 4226/(squid) off (0.00/0/0) tcp 0 0 0.0.0.0:988 0.0.0.0:* LISTEN 4198/rpc.statd off (0.00/0/0) tcp 0 0 127.0.0.1:3128 127.0.0.1:46440 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 679 192.168.0.186:52783 216.239.37.99:80 ESTABLISHED4226/(squid) on (1.44/0/0) tcp 0 0 127.0.0.1:3128 127.0.0.1:46441 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 127.0.0.1:3128 127.0.0.1:46442 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 127.0.0.1:3128 127.0.0.1:46436 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 192.168.0.186:45773 216.239.37.147:80 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 127.0.0.1:46435 127.0.0.1:3128 ESTABLISHED7090/epiphany off (0.00/0/0) tcp 0 0 127.0.0.1:46436 127.0.0.1:3128 ESTABLISHED7090/epiphany off (0.00/0/0) tcp 0 0 127.0.0.1:46441 127.0.0.1:3128 ESTABLISHED7090/epiphany off (0.00/0/0) tcp 0 0 127.0.0.1:46440 127.0.0.1:3128 ESTABLISHED7090/epiphany off (0.00/0/0) tcp 0 0 127.0.0.1:46442 127.0.0.1:3128 ESTABLISHED7090/epiphany off (0.00/0/0) tcp 0 689 192.168.0.186:44819 216.239.37.104:80 ESTABLISHED4226/(squid) on (81.35/4/0) tcp 0 0 127.0.0.1:3128 127.0.0.1:46435 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 192.168.0.186:52674 216.239.37.147:80 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 192.168.0.186:40319 204.179.240.224:80 ESTABLISHED4226/(squid) off (0.00/0/0) tcp 0 0 192.168.0.186:37416 207.245.101.66:110 ESTABLISHED6602/evolution-2.4 off (0.00/0/0) udp 0 0 0.0.0.0:32770 0.0.0.0:* 4226/(squid) off (0.00/0/0) udp 0 0 127.0.0.1:53 0.0.0.0:* 4119/pdnsd off (0.00/0/0) udp 0 0 0.0.0.0:3130 0.0.0.0:* 4226/(squid) off (0.00/0/0) udp 0 0 0.0.0.0:68 0.0.0.0:* 3714/dhclient off (0.00/0/0) udp 0 0 0.0.0.0:982 0.0.0.0:* 4198/rpc.statd off (0.00/0/0) udp 0 0 0.0.0.0:985 0.0.0.0:* 4198/rpc.statd off (0.00/0/0) udp 0 0 0.0.0.0:111 0.0.0.0:* 3733/portmap off (0.00/0/0) udp 0 0 192.168.0.186:123 0.0.0.0:* 4213/ntpd off (0.00/0/0) udp 0 0 127.0.0.1:123 0.0.0.0:* 4213/ntpd off (0.00/0/0) udp 0 0 0.0.0.0:123 0.0.0.0:* 4213/ntpd off (0.00/0/0) raw 0 0 0.0.0.0:1 0.0.0.0:* 7 4119/pdnsd off (0.00/0/0) Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 2 [ ACC ] STREAM LISTENING 8199 4405/gnome-panel /tmp/orbit-rlrevell/linc-1135-0-75aee2989367b unix 2 [ ACC ] STREAM LISTENING 7798 4362/dbus-daemon @/tmp/dbus-kRDDuSrN1q unix 2 [ ACC ] STREAM LISTENING 7438 4186/xfs /tmp/.font-unix/fs7100 unix 2 [ ACC ] STREAM LISTENING 22817 6624/evolution-alar /tmp/orbit-rlrevell/linc-19e0-0-418a226b84ab1 unix 2 [ ACC ] STREAM LISTENING 8232 4410/nautilus /tmp/orbit-rlrevell/linc-113a-0-20dcab1dc9146 unix 2 [ ACC ] STREAM LISTENING 22837 6627/evolution-data /tmp/orbit-rlrevell/linc-19e3-0-418a226bed67c unix 2 [ ACC ] STREAM LISTENING 6824 4032/hald @/tmp/hald-local/dbus-UBJ3XWbS5r unix 2 [ ACC ] STREAM LISTENING 6716 3961/X /tmp/.X11-unix/X0 unix 2 [ ACC ] STREAM LISTENING 8263 4426/wnck-applet /tmp/orbit-rlrevell/linc-114a-0-126d407ed6dcb unix 2 [ ACC ] STREAM LISTENING 6804 4017/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ACC ] STREAM LISTENING 8278 4429/gnome-vfs-daem /tmp/orbit-rlrevell/linc-114d-0-5fd29980f3c38 unix 2 [ ACC ] STREAM LISTENING 7356 4119/pdnsd /var/cache/pdnsd/pdnsd.status unix 2 [ ACC ] STREAM LISTENING 6499 3914/gdm /tmp/.gdm_socket unix 2 [ ACC ] STREAM LISTENING 8038 4296/x-session-mana /tmp/.ICE-unix/4296 unix 2 [ ACC ] STREAM LISTENING 7790 4357/ssh-agent /tmp/ssh-xSlHgN4296/agent.4296 unix 2 [ ACC ] STREAM LISTENING 8047 4372/gnome-keyring- /tmp/keyring-CUdodH/socket unix 2 [ ACC ] STREAM LISTENING 7793 4358/ssh-agent /tmp/ssh-TlyMwW4296/agent.4296 unix 2 [ ACC ] STREAM LISTENING 8563 4449/clock-applet /tmp/orbit-rlrevell/linc-1161-0-5fbb14bfadfe1 unix 5 [ ] DGRAM 12665 5224/syslogd /dev/log unix 2 [ ACC ] STREAM LISTENING 8059 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 2 [ ACC ] STREAM LISTENING 8325 4435/mapping-daemon /tmp/mapping-rlrevell unix 2 [ ACC ] STREAM LISTENING 7817 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 2 [ ACC ] STREAM LISTENING 8081 4376/metacity /tmp/orbit-rlrevell/linc-1118-0-5420b519e2d9 unix 2 [ ACC ] STREAM LISTENING 7827 4296/x-session-mana /tmp/orbit-rlrevell/linc-10c8-0-1f3d77523943b unix 2 [ ACC ] STREAM LISTENING 8096 4378/gnome-settings /tmp/orbit-rlrevell/linc-111a-0-4c7b8685621 unix 2 [ ] DGRAM 3191 1704/udevd @udevd unix 2 [ ACC ] STREAM LISTENING 8107 4381/gam_server @/tmp/fam-rlrevell- unix 2 [ ] DGRAM 6825 4032/hald @/var/run/hal/hotplug_socket2 unix 2 [ ACC ] STREAM LISTENING 22718 6602/evolution-2.4 /tmp/orbit-rlrevell/linc-19ca-0-b33d517def1 unix 2 [ ACC ] STREAM LISTENING 24819 7090/epiphany /tmp/orbit-rlrevell/linc-1bb2-0-2751674193866 unix 2 [ ] DGRAM 25181 7118/su unix 3 [ ] STREAM CONNECTED 24826 7090/epiphany /tmp/orbit-rlrevell/linc-1bb2-0-2751674193866 unix 3 [ ] STREAM CONNECTED 24825 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 24824 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 24823 7090/epiphany unix 3 [ ] STREAM CONNECTED 24822 7090/epiphany /tmp/orbit-rlrevell/linc-1bb2-0-2751674193866 unix 3 [ ] STREAM CONNECTED 24821 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 24818 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 24817 7090/epiphany unix 3 [ ] STREAM CONNECTED 24816 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 24815 7090/epiphany unix 3 [ ] STREAM CONNECTED 24811 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 24810 7090/epiphany unix 3 [ ] STREAM CONNECTED 23157 6627/evolution-data /tmp/orbit-rlrevell/linc-19e3-0-418a226bed67c unix 3 [ ] STREAM CONNECTED 23156 6602/evolution-2.4 unix 3 [ ] STREAM CONNECTED 23153 6602/evolution-2.4 /tmp/orbit-rlrevell/linc-19ca-0-b33d517def1 unix 3 [ ] STREAM CONNECTED 23150 6627/evolution-data unix 3 [ ] STREAM CONNECTED 22855 6624/evolution-alar /tmp/orbit-rlrevell/linc-19e0-0-418a226b84ab1 unix 3 [ ] STREAM CONNECTED 22854 6627/evolution-data unix 3 [ ] STREAM CONNECTED 22853 6627/evolution-data /tmp/orbit-rlrevell/linc-19e3-0-418a226bed67c unix 3 [ ] STREAM CONNECTED 22852 6624/evolution-alar unix 3 [ ] STREAM CONNECTED 22851 6627/evolution-data /tmp/orbit-rlrevell/linc-19e3-0-418a226bed67c unix 3 [ ] STREAM CONNECTED 22849 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 22848 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 22847 6627/evolution-data unix 3 [ ] STREAM CONNECTED 22840 6627/evolution-data /tmp/orbit-rlrevell/linc-19e3-0-418a226bed67c unix 3 [ ] STREAM CONNECTED 22839 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 22836 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 22835 6627/evolution-data unix 3 [ ] STREAM CONNECTED 22824 6624/evolution-alar /tmp/orbit-rlrevell/linc-19e0-0-418a226b84ab1 unix 3 [ ] STREAM CONNECTED 22823 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 22822 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 22821 6624/evolution-alar unix 3 [ ] STREAM CONNECTED 22820 6624/evolution-alar /tmp/orbit-rlrevell/linc-19e0-0-418a226b84ab1 unix 3 [ ] STREAM CONNECTED 22819 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 22816 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 22815 6624/evolution-alar unix 3 [ ] STREAM CONNECTED 22814 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 22813 6624/evolution-alar unix 3 [ ] STREAM CONNECTED 22809 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 22808 6624/evolution-alar unix 3 [ ] STREAM CONNECTED 22725 6602/evolution-2.4 /tmp/orbit-rlrevell/linc-19ca-0-b33d517def1 unix 3 [ ] STREAM CONNECTED 22724 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 22723 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 22722 6602/evolution-2.4 unix 3 [ ] STREAM CONNECTED 22721 6602/evolution-2.4 /tmp/orbit-rlrevell/linc-19ca-0-b33d517def1 unix 3 [ ] STREAM CONNECTED 22720 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 22717 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 22716 6602/evolution-2.4 unix 3 [ ] STREAM CONNECTED 22715 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 22714 6602/evolution-2.4 unix 3 [ ] STREAM CONNECTED 22710 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 22709 6602/evolution-2.4 unix 2 [ ] DGRAM 18248 4003/klogd unix 2 [ ] DGRAM 12726 3714/dhclient unix 3 [ ] STREAM CONNECTED 9737 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 9736 4562/xterm unix 3 [ ] STREAM CONNECTED 9735 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 9734 4562/xterm unix 3 [ ] STREAM CONNECTED 8582 4405/gnome-panel /tmp/orbit-rlrevell/linc-1135-0-75aee2989367b unix 3 [ ] STREAM CONNECTED 8581 4449/clock-applet unix 3 [ ] STREAM CONNECTED 8580 4449/clock-applet /tmp/orbit-rlrevell/linc-1161-0-5fbb14bfadfe1 unix 3 [ ] STREAM CONNECTED 8579 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8576 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 8575 4447/xterm unix 3 [ ] STREAM CONNECTED 8570 4449/clock-applet /tmp/orbit-rlrevell/linc-1161-0-5fbb14bfadfe1 unix 3 [ ] STREAM CONNECTED 8569 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8568 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8567 4449/clock-applet unix 3 [ ] STREAM CONNECTED 8566 4449/clock-applet /tmp/orbit-rlrevell/linc-1161-0-5fbb14bfadfe1 unix 3 [ ] STREAM CONNECTED 8565 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8562 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8561 4449/clock-applet unix 3 [ ] STREAM CONNECTED 8557 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8556 4447/xterm unix 3 [ ] STREAM CONNECTED 8550 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8549 4449/clock-applet unix 3 [ ] STREAM CONNECTED 8331 4381/gam_server @/tmp/fam-rlrevell- unix 3 [ ] STREAM CONNECTED 8330 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8329 4435/mapping-daemon /tmp/mapping-rlrevell unix 3 [ ] STREAM CONNECTED 8321 4410/nautilus unix 3 [ ] STREAM CONNECTED 8320 4405/gnome-panel /tmp/orbit-rlrevell/linc-1135-0-75aee2989367b unix 3 [ ] STREAM CONNECTED 8319 4429/gnome-vfs-daem unix 3 [ ] STREAM CONNECTED 8318 4429/gnome-vfs-daem /tmp/orbit-rlrevell/linc-114d-0-5fd29980f3c38 unix 3 [ ] STREAM CONNECTED 8317 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8312 4381/gam_server @/tmp/fam-rlrevell- unix 3 [ ] STREAM CONNECTED 8311 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8309 4405/gnome-panel /tmp/orbit-rlrevell/linc-1135-0-75aee2989367b unix 3 [ ] STREAM CONNECTED 8308 4426/wnck-applet unix 3 [ ] STREAM CONNECTED 8307 4426/wnck-applet /tmp/orbit-rlrevell/linc-114a-0-126d407ed6dcb unix 3 [ ] STREAM CONNECTED 8306 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8299 4410/nautilus /tmp/orbit-rlrevell/linc-113a-0-20dcab1dc9146 unix 3 [ ] STREAM CONNECTED 8298 4429/gnome-vfs-daem unix 3 [ ] STREAM CONNECTED 8297 4429/gnome-vfs-daem /tmp/orbit-rlrevell/linc-114d-0-5fd29980f3c38 unix 3 [ ] STREAM CONNECTED 8296 4410/nautilus unix 3 [ ] STREAM CONNECTED 8293 4429/gnome-vfs-daem /tmp/orbit-rlrevell/linc-114d-0-5fd29980f3c38 unix 3 [ ] STREAM CONNECTED 8292 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8291 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8290 4429/gnome-vfs-daem unix 3 [ ] STREAM CONNECTED 8285 4381/gam_server @/tmp/fam-rlrevell- unix 3 [ ] STREAM CONNECTED 8284 4429/gnome-vfs-daem unix 3 [ ] STREAM CONNECTED 8283 4017/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 8282 4429/gnome-vfs-daem unix 3 [ ] STREAM CONNECTED 8281 4429/gnome-vfs-daem /tmp/orbit-rlrevell/linc-114d-0-5fd29980f3c38 unix 3 [ ] STREAM CONNECTED 8280 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8277 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8276 4429/gnome-vfs-daem unix 3 [ ] STREAM CONNECTED 8275 4426/wnck-applet /tmp/orbit-rlrevell/linc-114a-0-126d407ed6dcb unix 3 [ ] STREAM CONNECTED 8274 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8273 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8272 4426/wnck-applet unix 3 [ ] STREAM CONNECTED 8268 4426/wnck-applet /tmp/orbit-rlrevell/linc-114a-0-126d407ed6dcb unix 3 [ ] STREAM CONNECTED 8265 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8262 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8261 4426/wnck-applet unix 3 [ ] STREAM CONNECTED 8257 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8256 4426/wnck-applet unix 3 [ ] STREAM CONNECTED 8250 4410/nautilus /tmp/orbit-rlrevell/linc-113a-0-20dcab1dc9146 unix 3 [ ] STREAM CONNECTED 8249 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8248 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8247 4410/nautilus unix 3 [ ] STREAM CONNECTED 8242 4381/gam_server @/tmp/fam-rlrevell- unix 3 [ ] STREAM CONNECTED 8241 4410/nautilus unix 3 [ ] STREAM CONNECTED 8235 4410/nautilus /tmp/orbit-rlrevell/linc-113a-0-20dcab1dc9146 unix 3 [ ] STREAM CONNECTED 8234 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8231 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8228 4410/nautilus unix 3 [ ] STREAM CONNECTED 8227 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 8226 4410/nautilus unix 3 [ ] STREAM CONNECTED 8221 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8220 4410/nautilus unix 3 [ ] STREAM CONNECTED 8206 4405/gnome-panel /tmp/orbit-rlrevell/linc-1135-0-75aee2989367b unix 3 [ ] STREAM CONNECTED 8205 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8204 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8203 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8202 4405/gnome-panel /tmp/orbit-rlrevell/linc-1135-0-75aee2989367b unix 3 [ ] STREAM CONNECTED 8201 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8198 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8197 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8196 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 8195 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8191 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8190 4405/gnome-panel unix 3 [ ] STREAM CONNECTED 8170 4378/gnome-settings /tmp/orbit-rlrevell/linc-111a-0-4c7b8685621 unix 3 [ ] STREAM CONNECTED 8160 4296/x-session-mana unix 3 [ ] STREAM CONNECTED 8151 4378/gnome-settings /tmp/orbit-rlrevell/linc-111a-0-4c7b8685621 unix 3 [ ] STREAM CONNECTED 8150 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8149 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8148 4378/gnome-settings unix 3 [ ] STREAM CONNECTED 8141 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8140 4393/xscreensaver unix 3 [ ] STREAM CONNECTED 8109 4381/gam_server @/tmp/fam-rlrevell- unix 3 [ ] STREAM CONNECTED 8108 4378/gnome-settings unix 3 [ ] STREAM CONNECTED 8159 4296/x-session-mana /tmp/.ICE-unix/4296 unix 3 [ ] STREAM CONNECTED 8100 4376/metacity unix 3 [ ] STREAM CONNECTED 8099 4378/gnome-settings /tmp/orbit-rlrevell/linc-111a-0-4c7b8685621 unix 3 [ ] STREAM CONNECTED 8098 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8095 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8094 4378/gnome-settings unix 3 [ ] STREAM CONNECTED 8091 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8090 4376/metacity unix 3 [ ] STREAM CONNECTED 8088 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 8087 4378/gnome-settings unix 3 [ ] STREAM CONNECTED 8084 4376/metacity /tmp/orbit-rlrevell/linc-1118-0-5420b519e2d9 unix 3 [ ] STREAM CONNECTED 8083 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8080 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 8079 4376/metacity unix 3 [ ] STREAM CONNECTED 8065 4296/x-session-mana /tmp/orbit-rlrevell/linc-10c8-0-1f3d77523943b unix 3 [ ] STREAM CONNECTED 8064 4374/bonobo-activat unix 3 [ ] STREAM CONNECTED 8063 4374/bonobo-activat /tmp/orbit-rlrevell/linc-1116-0-3300a4bb6a4e4 unix 3 [ ] STREAM CONNECTED 8062 4296/x-session-mana unix 2 [ ] DGRAM 8025 4370/ntpd unix 3 [ ] STREAM CONNECTED 8017 4296/x-session-mana /tmp/orbit-rlrevell/linc-10c8-0-1f3d77523943b unix 3 [ ] STREAM CONNECTED 8016 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 8015 4364/gconfd-2 /tmp/orbit-rlrevell/linc-110c-0-2626af2ccf5c unix 3 [ ] STREAM CONNECTED 7826 4296/x-session-mana unix 2 [ ] DGRAM 7812 4364/gconfd-2 unix 3 [ ] STREAM CONNECTED 7806 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 7805 4296/x-session-mana unix 3 [ ] STREAM CONNECTED 7802 4362/dbus-daemon unix 3 [ ] STREAM CONNECTED 7801 4362/dbus-daemon unix 3 [ ] STREAM CONNECTED 7800 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 7799 4361/dbus-launch unix 2 [ ] DGRAM 7526 4226/(squid) unix 2 [ ] DGRAM 7523 4224/squid unix 2 [ ] DGRAM 7492 4213/ntpd unix 2 [ ] DGRAM 7462 4198/rpc.statd unix 3 [ ] STREAM CONNECTED 7240 4017/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 7239 4032/hald unix 3 [ ] STREAM CONNECTED 6808 4017/dbus-daemon unix 3 [ ] STREAM CONNECTED 6807 4017/dbus-daemon unix 8 [ ] STREAM CONNECTED 7314 3961/X /tmp/.X11-unix/X0 unix 3 [ ] STREAM CONNECTED 6730 3917/gdm