Skip to contentSkip to conversation

e1000 jumbo problems

E1000-devel [email protected] 3 participants 9 messages Jun 23, 2004 – Jul 1, 2004 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 3 lines · [email protected] wrote:

Another related issue, is that the driver uses 4KiB buffers for MTUs in the 1500 -> 2000 range which seems a bit silly. Any particular reason for that?

I changed the driver to use 2KiB buffers for frames in the 1518 -> 2048 range (BSEX=0, LPE=1). This breaks however
as packets are not dropped that are larger than the max specified? Instead they're scribbled into memory causing a lockup after a while.

I noticed in e1000_change_mtu() that adapter->hw.max_frame_size is only set after e1000_down();e1000_up(); Is that correct?

Are there any anwsers for the general questions I had even?

1. Is there a public dev tree available for the e1000 driver?
2. Are there programming docs for the various GigE chipsets?

thanks,
Pádraig.

#2 lore raw

TCP receiver's window calculation problem

Hi,

We have been running some iperf experiments over long-latency high-capacity networks for protocol testing. We noticed a strange receiver's window limitation of 3,147,776 bytes even when the iperf server was setup to request 32 MB of socket buffer (for which kernel grants twice that).

After doing printk with various window calculation functions at the receiver, we believe there may be a possible problem with tp->rcv_ssthresh calculation in __tcp_grow_window in tcp_input.c.

With input parameters of tcp memory of 64 MB, a jumbo MTU (9000 bytes) setting at the receiver, which gives a skb_true_size of 16660 bytes, and a standard MTU (1500 byte) at the sender side that yields a skb_len of 1448 bytes. tp->rcv_ssthresh gets stuck at 3,148,472 (see the code segment below). Because the tcp receiver's window needs to be in multiple of mss (/1448 then *1448) and window scaling (>>10 and then <<10), the sender sees a limit of 3,147,776 bytes.

I include an example code (stripped away the data structs and expanded whatever macros there are) that reproduces this problem. The function __tcp_grow_window itself may have problems for other combinations of input.

#include <stdio.h>
#include <stdlib.h>

typedef unsigned int __u32;

static int
__tcp_grow_window(__u32 rcv_ssthresh, __u32 tcp_full_space,

					__u32 skb_true_size, __u32 skb_len)
{
	int truesize = skb_true_size*3/8;
	int window = tcp_full_space*3/8;

	while ( rcv_ssthresh <= window ) {
		if ( truesize <= skb_len )
			return 2896;

		truesize >>= 1;
		window >>= 1;
	}
	return 0;
}

int main()
{

	__u32 iperf_mem = 64*1024*1024;
	__u32 skb_true_size = 16660;
	__u32 skb_len = 1448;
	__u32 rcv_ssthresh = 3148472;

	int i, incr;

	for (i=0; i<1000; ++i)
	{
		incr = __tcp_grow_window(rcv_ssthresh, iperf_mem, skb_true_size, skb_len);
		printf("i=%d incr=%d\n", i, incr);
	}
}

Cheng

Signature · 1 line
Lab # 626 395 8820
#3 lore raw

Re: TCP receiver's window calculation problem

I've run in to this problem, too. This code prevents advertising more rcvbuf space than you are likely to need. This is good for something like an X11 connection, but obviously very bad for the bulk transfer mixed MTU case. Apparently some drivers use multiple packet buffer sizes which helps, but at least e1000 and sk98lin do not. If you don't want to overrun the rcvbuf bounds, then the only other recourse is to coalesce packets, which works well but is pretty expensive. This will happen already if you take out the rcv_ssthresh bound.

I think the most desirable answer is to not have a hard per-connection memory bound, but this is problematic because of denial-of-service conerns.

-John

Quoted 73 lines · Cheng Jin

Hi,

We have been running some iperf experiments over long-latency high-capacity networks for protocol testing. We noticed a strange receiver's window limitation of 3,147,776 bytes even when the iperf server was setup to request 32 MB of socket buffer (for which kernel grants twice that).

After doing printk with various window calculation functions at the receiver, we believe there may be a possible problem with tp->rcv_ssthresh calculation in __tcp_grow_window in tcp_input.c.

With input parameters of tcp memory of 64 MB, a jumbo MTU (9000 bytes) setting at the receiver, which gives a skb_true_size of 16660 bytes, and a standard MTU (1500 byte) at the sender side that yields a skb_len of 1448 bytes. tp->rcv_ssthresh gets stuck at 3,148,472 (see the code segment below). Because the tcp receiver's window needs to be in multiple of mss (/1448 then *1448) and window scaling (>>10 and then <<10), the sender sees a limit of 3,147,776 bytes.

I include an example code (stripped away the data structs and expanded whatever macros there are) that reproduces this problem. The function __tcp_grow_window itself may have problems for other combinations of input.

#include <stdio.h>
#include <stdlib.h>

typedef unsigned int __u32;

static int
__tcp_grow_window(__u32 rcv_ssthresh, __u32 tcp_full_space,

					__u32 skb_true_size, __u32 skb_len)
{
	int truesize = skb_true_size*3/8;
	int window = tcp_full_space*3/8;

	while ( rcv_ssthresh <= window ) {
		if ( truesize <= skb_len )
			return 2896;

		truesize >>= 1;
		window >>= 1;
	}
	return 0;
}

int main()
{

	__u32 iperf_mem = 64*1024*1024;
	__u32 skb_true_size = 16660;
	__u32 skb_len = 1448;
	__u32 rcv_ssthresh = 3148472;

	int i, incr;

	for (i=0; i<1000; ++i)
	{
		incr = __tcp_grow_window(rcv_ssthresh, iperf_mem, skb_true_size, skb_len);
		printf("i=%d incr=%d\n", i, incr);
	}
}

Cheng

--
Lab # 626 395 8820

#4 lore raw

Re: TCP receiver's window calculation problem

Hi, John,

Thanks for confirming this problem.

Quoted 3 lines

I've run in to this problem, too. This code prevents advertising more rcvbuf space than you are likely to need. This is good for something like an X11 connection, but obviously very bad for the bulk transfer mixed MTU

I would think this is already taken care of at the sender by application limited cwnd so cwnd wouldn't increase beyond what is being actually used.

Quoted 3 lines

I think the most desirable answer is to not have a hard per-connection memory bound, but this is problematic because of denial-of-service conerns.

I think having a default limit on tcp memory is acceptable to prevent DoS, but when a user increases the memory limit by explicitly setting tcp_rmem, that should take effect. The code itself shouldnt pose any limit like it does now.

Actually, I am not clear what that window-calculation algorithm is. Is it recommended by some RFC?

Cheng

#5 lore raw

Re: TCP receiver's window calculation problem

Quoted 4 lines · Cheng Jin

I think having a default limit on tcp memory is acceptable to prevent DoS, but when a user increases the memory limit by explicitly setting tcp_rmem, that should take effect. The code itself shouldnt pose any limit like it does now.

The core of the problem is that you are describing a truesize of each skb at about 16k, but each of those only contains < 1500 bytes of payload. You are wasting 90% of your socket memory. Announcing a 3 MB window with a 30 MB socket buffer is the right thing to do, from a certain point of view. OTOH, it kills performance.

Quoted 2 lines

Actually, I am not clear what that window-calculation algorithm is. Is it recommended by some RFC?

No, it's not standard. I'm not sure who wrote this code.

-John

#6 lore raw

Re: TCP receiver's window calculation problem

John,

Quoted 10 lines
Quoted 4 lines

I think having a default limit on tcp memory is acceptable to prevent DoS, but when a user increases the memory limit by explicitly setting tcp_rmem, that should take effect. The code itself shouldnt pose any limit like it does now.

The core of the problem is that you are describing a truesize of each skb at about 16k, but each of those only contains < 1500 bytes of payload. You are wasting 90% of your socket memory. Announcing a 3 MB window with a 30 MB socket buffer is the right thing to do, from a certain point of view. OTOH, it kills performance.

The receiver is set to use a 9000 MTU, but the sender uses a 1500-byte MTU, which is not really a pathological case. It would have made more sense for the receiver to allocate skbs of the right size as incoming packets are received.

Is it due to effciency reasons that the skbs are just fixed in size according to the set MTU on the interface card? I suppose that the receiver has no real way of knowing the right MTU size at the sender.

Thanks,

Cheng

#7 lore raw

Re: TCP receiver's window calculation problem

Quoted 4 lines · Cheng Jin

The receiver is set to use a 9000 MTU, but the sender uses a 1500-byte MTU, which is not really a pathological case. It would have made more sense for the receiver to allocate skbs of the right size as incoming packets are received.

Some drivers are apparently optimized for this case. I have not confirmed this.

Quoted 3 lines

Is it due to effciency reasons that the skbs are just fixed in size according to the set MTU on the interface card? I suppose that the receiver has no real way of knowing the right MTU size at the sender.

I haven't looked too hard in to this. Any device driver people want to chime in?

-John

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 9 lines · Chris Leech
Quoted 3 lines
Quoted 3 lines

Another related issue, is that the driver uses 4KiB buffers for MTUs in the 1500 -> 2000 range which seems a bit silly. Any particular reason for that?

It is wasteful, but does anyone actually use an MTU in the range of 1501 - 2030? It seems silly to me to go with a non-standard frame size, but not go up to something that might give you a performance benefit (9k).

I'm seeing it with MPLS in some configs.
MPLS labels are just prepended onto ethernet frames giving frames up to 1546 bytes. Using 4KiB frames for this situation is wasteful of memory but more importantly for my application it has a noticeable impact on receive performance.

Quoted 6 lines
Quoted 4 lines

I changed the driver to use 2KiB buffers for frames in the 1518 -> 2048 range (BSEX=0, LPE=1). This breaks however
as packets are not dropped that are larger than the max specified? Instead they're scribbled into memory causing a lockup after a while.

Quoted 8 lines

That sounds right, if you actually got the RCTL register set correctly. In e1000_setup_rctl the adapter->rx_buffer_len is used to set that register, and it's currently written to only set LPE if the buffer size is bigger than 2k (thus, why 4k buffers are used even when the MTU is in the 1501 - 2030 range). To use 2k buffers for slightly large frames, you'd want some new flag in the adapter for LPE (or check netdev->mtu I guess) and do something like: rctl |= E1000_RCTL_SZ_2048 | E1000_RCTL_LPE

yep, that's what I did.

Quoted 4 lines

e1000 devices don't have a programmable MTU for receive filtering, they drop anything larger than 1518 unless LPE (long packet enable) is set. If LPE is set they accept anything that fits in the FIFO and has a valid FCS.

thanks for that. What I'm noticing now is the same thing happens with the official driver (5.2.52 or 5.2.30.1). I.E. set the MTU to 4000 for e.g., then send in frames larger than 4096 and they're accepted? Doing this for a while causes mem to get corrupted.

Quoted 4 lines

An MTU setting needs to be valid across your ethernet, why is the
e1000 receiving a frame larger than the MTU? (jabber should be rare) But, if the length of receive buffers matches what was set in RCTL, larger than expected valid frames will spill over to the next buffer
and be dropped in the driver without corrupting memory.

Are the buffers in contiguous mem? What happens for the last buffer?

Quoted 7 lines
Quoted 2 lines

I noticed in e1000_change_mtu() that adapter->hw.max_frame_size is only set after e1000_down();e1000_up(); Is that correct?

There might be a slight race there (I'll think about it some more),
but it's not something that would cause memory corruption. hw.max_frame_size is only used in a software workaround for 82543 based copper gigabit cards (vendor:device 8086:1004) when paired with
certain link partners.

fair enough.

Quoted 5 lines
Quoted 4 lines

Are there any anwsers for the general questions I had even?

1. Is there a public dev tree available for the e1000 driver?

Quoted 8 lines

No, the best source base to work from is what is in the 2.6 kernel tree (or Jeff's net-drivers tree). We keep that as up to date as possible, and it's always fairly close to our internal development sources.

Quoted 1 line

2. Are there programming docs for the various GigE chipsets?

Not publicly available at this time.

thanks a million,
Pádraig.

#10 lore raw
In reply to a message not in the archive

This patch is not for applying, just for discussion. comments below...

Quoted 26 lines · Chris Leech
Quoted 3 lines
Quoted 3 lines

Another related issue, is that the driver uses 4KiB buffers for MTUs in the 1500 -> 2000 range which seems a bit silly. Any particular reason for that?

It is wasteful, but does anyone actually use an MTU in the range of 1501 - 2030? It seems silly to me to go with a non-standard frame size, but not go up to something that might give you a performance benefit (9k).

Quoted 4 lines

I changed the driver to use 2KiB buffers for frames in the 1518 -> 2048 range (BSEX=0, LPE=1). This breaks however
as packets are not dropped that are larger than the max specified? Instead they're scribbled into memory causing a lockup after a while.

That sounds right, if you actually got the RCTL register set correctly. In e1000_setup_rctl the adapter->rx_buffer_len is used to set that register, and it's currently written to only set LPE if the buffer size is bigger than 2k (thus, why 4k buffers are used even when the MTU is in the 1501 - 2030 range). To use 2k buffers for slightly large frames, you'd want some new flag in the adapter for LPE (or check netdev->mtu I guess) and do something like: rctl |= E1000_RCTL_SZ_2048 | E1000_RCTL_LPE

Quoted 4 lines

e1000 devices don't have a programmable MTU for receive filtering, they drop anything larger than 1518 unless LPE (long packet enable) is set. If LPE is set they accept anything that fits in the FIFO and has a valid FCS.

More accurately e1000s accept anything (even greater than a FIFO). When a large packet is written into multiple FIFOs, only the last rx descriptor has the EOP (end of packet) flag set. The driver doesn't handle this at all currently and will drop the initial buffers (because they don't have the EOP set) which is fine, but it will accept the last buffer (part of the packet).

I've attached a patch that fixes this. Also the patch drops packets that fit within a buffer but are larger than MTU. So in summary the patch will stop packets > MTU being accepted by the driver.

Note also this patch changes to using 2KiB buffers (from 4KiB) for MTUs between 1500 and 2030, and also it enables large frame reception (LFE) always, but ingore these as they're just for debugging.

The patch makes my system completely stable now for MTUs <= 2500, However I can still get the system to freeze repeatedly by sending packets larger than this.

cheers,
Pádraig.

src/e1000.h +2 -0
diff -Naru e1000-5.2.52-k3/src/e1000.h e1000-pb/src/e1000.h
@@ -177,6 +177,8 @@
 	unsigned int next_to_use;
 	/* next descriptor to check for DD status bit */
 	unsigned int next_to_clean;
+	/* whether next buffer is partial packet */
+	unsigned int multi_buf_pkt;
 	/* array of buffer information structs */
 	struct e1000_buffer *buffer_info;
 };
src/e1000_hw.h +1 -0
diff -Naru e1000-5.2.52-k3/src/e1000_hw.h e1000-pb/src/e1000_hw.h
@@ -922,6 +922,7 @@
     uint64_t sec;
     uint64_t cexterr;
     uint64_t rlec;
+    uint64_t multibuf;
     uint64_t xonrxc;
     uint64_t xontxc;
     uint64_t xoffrxc;
src/e1000_main.c +55 -9

143 lines not on this page · raw message