Skip to contentSkip to conversation

receiver window questions

DISCUSSION Vlad Yasevich 3 participants 12 messages May 29, 2008 – May 30, 2008 SCTP protocol

lore
Share & tools
lore
#0 lore raw

Hi Michael

Quoted 21 lines · Michael Tuexen

Hi Vlad,

we are currently testing the receive behaviour of Linux (and other systems).

We are using Fedora 9, kernel 2.6.25-14.

The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.

The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.

When setting the receive buffer space (using the SO_RCVBUF socket option) to 10000 we oberserve the following:

- When sending messages of size 1000 bytes, the receiver SACK the first, announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The a_rwnd goes down to 0 and discards messages. Everything is fine.

Quoted 4 lines

- When sending messages of size 100 bytes, the receiver SACKs the first messages and reduces the a_rwnd accordingly. Then it looks like the receive buffer grows, because messages are accepted and the a_rwnd does not shrink. Is this intended?

No. The a_rwnd should go down to 0 as before.

Quoted 2 lines

We also figured out that after about 670947 messages of size 100 bytes the association is aborted. Is this intended?

This is also not intended. We'll take a look.

Thanks
-vlad

Quoted 7 lines

- Sending messages of size 10 bytes is similar to 100 byes.

Best regards
Michael

#1 lore raw

Hi Vlad,

Irene has done some more testing.

The ABORT was due to the fact that the testtool was not responding to HEARTBEATs. She fixed that.

She is still investigating, but the conjecture is that for small messages
the receiver accepts floor(2*s/(178+msgsize)) messages, where s is the receive space set by a socket option. Small means smaller than 180, but 178 could be the real number. Irene will check this tomorrow. In that case the a_rwnd goes down, and when the above limit is reached, a_rwnd is not decremented anymore, the messages are not put into the receive buffer anymore (checked if they are delivered to the application),
but the cumtsn is still incremented. So the sender thinks, the messages are
successfully transmitted, but the receiver just drops them.

Irene will provide definite test case to reproduce the problem tomorrow.

Best regards
Michael

Quoted 43 lines · Vlad Yasevich

Hi Michael

Quoted 15 lines · Michael Tuexen

Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using the SO_RCVBUF socket option)
to 10000 we oberserve the following:
- When sending messages of size 1000 bytes, the receiver SACK the first,
announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The a_rwnd goes down to 0 and discards

Quoted 8 lines

messages.
Everything is fine.
- When sending messages of size 100 bytes, the receiver SACKs the first
messages and reduces the a_rwnd accordingly. Then it looks like the receive buffer grows, because messages are accepted and the a_rwnd
does
not shrink. Is this intended?

No. The a_rwnd should go down to 0 as before.

Quoted 3 lines

We also figured out that after about 670947 messages of size 100
bytes
the association is aborted. Is this intended?

This is also not intended. We'll take a look.

Thanks
-vlad

Quoted 3 lines

- Sending messages of size 10 bytes is similar to 100 byes. Best regards
Michael

#2 lore raw
In reply to Vlad Yasevich
Quoted 29 lines · Vlad Yasevich

Hi Michael

Quoted 21 lines · Michael Tuexen

Hi Vlad,

we are currently testing the receive behaviour of Linux (and other systems).

We are using Fedora 9, kernel 2.6.25-14.

The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.

The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.

When setting the receive buffer space (using the SO_RCVBUF socket option) to 10000 we oberserve the following:

- When sending messages of size 1000 bytes, the receiver SACK the first, announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The a_rwnd goes down to 0 and discards messages. Everything is fine.

Quoted 4 lines

- When sending messages of size 100 bytes, the receiver SACKs the first messages and reduces the a_rwnd accordingly. Then it looks like the receive buffer grows, because messages are accepted and the a_rwnd does not shrink. Is this intended?

Quoted 7 lines

No. The a_rwnd should go down to 0 as before.

Quoted 2 lines

We also figured out that after about 670947 messages of size 100 bytes the association is aborted. Is this intended?

This is also not intended. We'll take a look.

That does sound odd, yes, a_rwnd should be reduced in response to each received frame. do you have tcpdumps of this test available?
Thanks
Neil

Signature · 6 lines
/***************************************************
 *Neil Horman
 *[email protected]
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/
#3 lore raw
In reply to Vlad Yasevich
Quoted 41 lines · Neil Horman
Quoted 40 lines · Vlad Yasevich

Hi Michael

Quoted 24 lines · Michael Tuexen

Hi Vlad,

we are currently testing the receive behaviour of Linux (and other systems).

We are using Fedora 9, kernel 2.6.25-14.

The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.

The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.

When setting the receive buffer space (using the SO_RCVBUF socket option)
to 10000 we oberserve the following:

- When sending messages of size 1000 bytes, the receiver SACK the first,
announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The a_rwnd goes down to 0 and discards messages.
Everything is fine.

Quoted 6 lines

- When sending messages of size 100 bytes, the receiver SACKs the first
messages and reduces the a_rwnd accordingly. Then it looks like the receive buffer grows, because messages are accepted and the a_rwnd does
not shrink. Is this intended?

No. The a_rwnd should go down to 0 as before.

Quoted 3 lines

We also figured out that after about 670947 messages of size 100 bytes
the association is aborted. Is this intended?

Quoted 7 lines
Quoted 2 lines

This is also not intended. We'll take a look.

That does sound odd, yes, a_rwnd should be reduced in response to each received
frame. do you have tcpdumps of this test available? Thanks
Neil

It looks like that you do not reduce the a_rwnd, because you discard the message.
The problem seems to be that you ack it by incrementing the cumtsn_ack...

Irene did the tests in the lab. We'll provide traces and a testprogram tomorrow.

Quoted 11 lines
Signature · 7 lines
-- 
/***************************************************
*Neil Horman
*[email protected]
*gpg keyid: 1024D / 0x92A74FA1
*http://pgp.mit.edu
***************************************************/
#4 lore raw
In reply to Vlad Yasevich
Quoted 31 lines · Neil Horman
Quoted 30 lines · Vlad Yasevich

Hi Michael

Quoted 21 lines · Michael Tuexen

Hi Vlad,

we are currently testing the receive behaviour of Linux (and other systems).

We are using Fedora 9, kernel 2.6.25-14.

The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.

The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.

When setting the receive buffer space (using the SO_RCVBUF socket option) to 10000 we oberserve the following:

- When sending messages of size 1000 bytes, the receiver SACK the first, announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The a_rwnd goes down to 0 and discards messages. Everything is fine.

Quoted 4 lines

- When sending messages of size 100 bytes, the receiver SACKs the first messages and reduces the a_rwnd accordingly. Then it looks like the receive buffer grows, because messages are accepted and the a_rwnd does not shrink. Is this intended?

No. The a_rwnd should go down to 0 as before.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

We also figured out that after about 670947 messages of size 100 bytes the association is aborted. Is this intended?

This is also not intended. We'll take a look.

That does sound odd, yes, a_rwnd should be reduced in response to each received frame. do you have tcpdumps of this test available?
Thanks
Neil

I think this is hitting a condition where the receiver buffer is exhausted prior to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue reporting the tsn as received and move the cum-tsn if appropriate.

We've been trying to figure out what the correct way to solve this condition is and so far haven't come up with a workable solution.

One approach is to reduce the a_rwnd to 1 when the receive buffer reaches some consumption threshold. This would allow 1 packet in flight to allow receive buffer to drain. Another piece of this puzzle is also SWS avoidance. We currently have somewhat poor algorithm that allows a max of 1 MTU in transit.

-vlad

#5 lore raw
In reply to Vlad Yasevich
Quoted 48 lines · Vlad Yasevich
Quoted 39 lines · Neil Horman
Quoted 38 lines · Vlad Yasevich

Hi Michael

Quoted 24 lines · Michael Tuexen

Hi Vlad,

we are currently testing the receive behaviour of Linux (and other systems).

We are using Fedora 9, kernel 2.6.25-14.

The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.

The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.

When setting the receive buffer space (using the SO_RCVBUF socket option)
to 10000 we oberserve the following:

- When sending messages of size 1000 bytes, the receiver SACK the first,
announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The a_rwnd goes down to 0 and discards messages.
Everything is fine.

Quoted 6 lines

- When sending messages of size 100 bytes, the receiver SACKs the first
messages and reduces the a_rwnd accordingly. Then it looks like the receive buffer grows, because messages are accepted and the a_rwnd does
not shrink. Is this intended?

No. The a_rwnd should go down to 0 as before.

Quoted 3 lines

We also figured out that after about 670947 messages of size 100 bytes
the association is aborted. Is this intended?

Quoted 7 lines
Quoted 2 lines

This is also not intended. We'll take a look.

That does sound odd, yes, a_rwnd should be reduced in response to each received
frame. do you have tcpdumps of this test available? Thanks
Neil

Quoted 6 lines

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

This would explain the behaviour completely.

Quoted 5 lines

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

We are looking at the behavior of different implementations to look at interoperability
issues. That is how we came to test this...

Quoted 8 lines

One approach is to reduce the a_rwnd to 1 when the receive buffer reaches some
consumption threshold. This would allow 1 packet in flight to allow receive buffer to drain. Another piece of this puzzle is also SWS avoidance. We currently have somewhat poor algorithm that allows a max of 1 MTU in transit.

BSD reduces the a_rwnd to 1 in case of small windows (less than 3000 bytes) for SWS avoidance,
which you have to do anyway. And it makes sure that it does not report a message as received when it
is discarded.

Quoted 4 lines

-vlad

#6 lore raw
In reply to Vlad Yasevich
Quoted 10 lines · Vlad Yasevich
Quoted 1 line · Neil Horman

On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

Quoted 4 lines

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

You're right it does sound like that. You know, I haven't visited that code since we rewrote the receive buffer management code to expand according to available memory with the sk_mem_schedule api. Do you think this could be as simple as removing this drop point?

Neil

#7 lore raw
In reply to Vlad Yasevich

Dear all,

for message size larger or equal to 176 byte, the behaviour seems to be OK, for smaller ones there is a problem.

Please find attached two traces. arwnd175.pcap shows the buggy behaviour, arwnd176.pcap shows the acceptable behaviour.

One the receiver side we just opened a listening 1-to-many style socket, waited until the sender (a testtool) sent all its data and then started to receive all messages. This is how we figured out that messages were lost.

Please let us know if you need more information.

Best regards
Michael

Quoted 23 lines · Neil Horman
Quoted 10 lines · Vlad Yasevich
Quoted 1 line · Neil Horman

On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

Quoted 4 lines

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

You're right it does sound like that. You know, I haven't visited that code since we rewrote the receive buffer management code to expand according to available memory with the sk_mem_schedule api. Do you think this could be as simple as removing this drop point?

Neil

#8 lore raw
In reply to Vlad Yasevich
Quoted 10 lines · Neil Horman
Quoted 9 lines · Vlad Yasevich
Quoted 1 line · Neil Horman

On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

Quoted 9 lines
Quoted 4 lines

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

You're right it does sound like that. You know, I haven't visited that code since we rewrote the receive buffer management code to expand according to available memory with the sk_mem_schedule api. Do you think this could be as simple as removing this drop point?

The problem is that if the user sets the buffer size, we don't auto-tune it. This is what happened in this case. However, even with auto-tunning, this still shows up with 1 byte data chunks.

So, 2 things need to happen:
1) when we drop the chunk due to allocation failure, we have to remove the tsn from the map. This one is easy

2) We need to properly detect SWS. I haven't looked fully at this, but it feels a bit more involved. Implementing something like what BSD has would also work, i.e. reducing a_rwnd 1 when receive buffer is about to be exhausted.

-vlad

#9 lore raw
In reply to Vlad Yasevich

Hi Vlad,

is there any description available how the auto-tuning works? But even with auto-tuning you still have the a_rwnd you report to the outside world (in SACKs) and an internal value which you use for tracking you buffer, right?

Best regards
Michael

Quoted 44 lines · Vlad Yasevich
Quoted 14 lines · Neil Horman
Quoted 9 lines · Vlad Yasevich
Quoted 1 line · Neil Horman

On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

Quoted 4 lines

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

Quoted 7 lines

You're right it does sound like that. You know, I haven't visited that code
since we rewrote the receive buffer management code to expand according to
available memory with the sk_mem_schedule api. Do you think this could be as
simple as removing this drop point?

The problem is that if the user sets the buffer size, we don't auto- tune it.
This is what happened in this case. However, even with auto- tunning, this
still shows up with 1 byte data chunks.

So, 2 things need to happen:
1) when we drop the chunk due to allocation failure, we have to
remove
the tsn from the map. This one is easy

2) We need to properly detect SWS. I haven't looked fully at
this, but
it feels a bit more involved. Implementing something
like what BSD
has would also work, i.e. reducing a_rwnd 1 when receive buffer
is
about to be exhausted.

-vlad

#10 lore raw
In reply to Vlad Yasevich
Quoted 3 lines · Michael Tuexen

Hi Vlad,

is there any description available how the auto-tuning works?

There is no formal writeup. In short it works the same as TCP.

Essentially, we start out with a set buffer space and calculated rwnd. Then as buffer space and window is consumed, we see if buffer space is consumed faster then rwnd. If we reach a point where default buffer space is consumed, but rwnd is not, we grow the buffer space slowly up to the next threshold, thus allowing all of rwnd to be consumed.

There is also a system-wide threshold that we look at when growing
the buffer. So that if the system is experiencing memory pressure
conditions, the growth and allocations will fail.

Quoted 3 lines

But even with auto-tuning you still have the a_rwnd you report to the outside world (in SACKs) and an internal value which you use for tracking you buffer, right?

Buffer space tracking is separate from rwnd and a_rwnd that we advertise.

-vlad

Quoted 4 lines

Best regards
Michael

#11 lore raw
In reply to Vlad Yasevich
Quoted 7 lines · Michael Tuexen

Hi Vlad,

is there any description available how the auto-tuning works? But even with auto-tuning you still have the a_rwnd you report to the outside world (in SACKs) and an internal value which you use for tracking you buffer, right?

As Vlad said, there is no formal description available, but if you want a functional overview, the sk_mem_schedule api is the place to look to understand how thresholds, and memory pressure detection work

Regards
Neil

Quoted 45 lines

Best regards
Michael

Quoted 40 lines · Vlad Yasevich
Quoted 14 lines · Neil Horman
Quoted 9 lines · Vlad Yasevich
Quoted 1 line · Neil Horman

On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

Quoted 4 lines

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

Quoted 7 lines

You're right it does sound like that. You know, I haven't visited that code
since we rewrote the receive buffer management code to expand according to
available memory with the sk_mem_schedule api. Do you think this could be as
simple as removing this drop point?

The problem is that if the user sets the buffer size, we don't auto-tune it.
This is what happened in this case. However, even with auto-tunning, this still shows up with 1 byte data chunks.

So, 2 things need to happen:
1) when we drop the chunk due to allocation failure, we have to remove the tsn from the map. This one is easy

2) We need to properly detect SWS. I haven't looked fully at this, but it feels a bit more involved. Implementing something like what
BSD
has would also work, i.e. reducing a_rwnd 1 when receive buffer is about to be exhausted.

-vlad

Signature · 4 lines
/****************************************************
 * Neil Horman <[email protected]>
 * Software Engineer, Red Hat
 ****************************************************/