Skip to contentSkip to conversation

posible latency issues in seq_read

DISCUSSION Chris Friesen 3 participants 5 messages Jul 20, 2007 – Jul 23, 2007

lore
Share & tools
lore
#0 lore raw

We've run into an issue (on 2.6.10) where calling "lsof" triggers lost packets on our server. Preempt is disabled, and NAPI is enabled.

It appears that for some reason the networking softirq is not being handled in a timely fashion, which means that the rx ring buffer fills up and packets overflow.

It appears that the problem path is:

seq_read
tcp_seq_next
established_get_next
read_lock/read_unlock

The issue appears to be related to the amount of time that this syscall takes. While we're in the syscall we cannot run the softirqd thread, and so the rx buffer is not being cleaned.

The fact that there are kmalloc(GFP_KERNEL) calls in seq_read() seems to indicate that sleeping is safe, so would it be reasonable to call schedule() periodically (maybe based on elapsed time) to ensure that system latency is kept under control?

Thanks,

Chris

#1 lore raw
Quoted 4 lines · Chris Friesen

We've run into an issue (on 2.6.10) where calling "lsof" triggers lost packets on our server. Preempt is disabled, and NAPI is enabled.

Can you reproduce with a recent kernel? Lots of latency issues have been fixed since then.

Lee

#2 lore raw
Quoted 7 lines · Lee Revell
Quoted 2 lines · Chris Friesen

We've run into an issue (on 2.6.10) where calling "lsof" triggers lost packets on our server. Preempt is disabled, and NAPI is enabled.

Can you reproduce with a recent kernel? Lots of latency issues have been fixed since then.

Unfortunately I have to fix it on this version (the bug was found on shipped product), so if there was a difference I'd have to isolate the changes and backport them. Also, I can't run the software that triggers the problem on a newer kernel as it has dependencies on various patches that are not in mainline.

Basically what I'd like to know is whether calling schedule() in seq_read() is safe or whether it would break assumptions made by seq_file users.

Chris

#3 lore raw
Quoted 6 lines · Chris Friesen
Quoted 4 lines · Lee Revell
Quoted 2 lines · Chris Friesen

We've run into an issue (on 2.6.10) where calling "lsof" triggers lost packets on our server. Preempt is disabled, and NAPI is enabled.

Quoted 7 lines
Quoted 2 lines

Can you reproduce with a recent kernel? Lots of latency issues have been fixed since then.

Unfortunately I have to fix it on this version (the bug was found on shipped product), so if there was a difference I'd have to isolate the changes and backport them. Also, I can't run the software that triggers the problem on a newer kernel as it has dependencies on various patches that are not in mainline.

Basically what I'd like to know is whether calling schedule() in seq_read() is safe or whether it would break assumptions made by seq_file users.

It wont help much. seq_read() is fine in itself.

The problem is in established_get_next() and established_get_first() not allowing softirq processing, while scanning a possibly huge hash table, even if few sockets are hashed in.

As cond_resched_softirq() was added in linux-2.6.11, you probably *need* to check the diffs between linux-2.6.10 & linux-2.6.11

files :

include/linux/sched.h
net/core/sock.c (__release_sock() latency)
net/ipv4/tcp_ipv4.c (/proc/net/tcp latency)

#4 lore raw
Quoted 3 lines · Eric Dumazet

The problem is in established_get_next() and established_get_first() not allowing softirq processing, while scanning a possibly huge hash table, even if few sockets are hashed in.

As cond_resched_softirq() was added in linux-2.6.11, you probably *need* to check the diffs between linux-2.6.10 & linux-2.6.11

Thanks for the pointers to the likely culprits.

Chris