receiver window questions
Hi Michael
Quoted 21 lines · Michael Tuexen
Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using the
SO_RCVBUFsocket option) to 10000 we oberserve the following:- When sending messages of size 1000 bytes, the receiver SACK the first, announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The
a_rwndgoes down to 0 and discards messages. Everything is fine.
Quoted 4 lines
- When sending messages of size 100 bytes, the receiver SACKs the first messages and reduces the
a_rwndaccordingly. Then it looks like the receive buffer grows, because messages are accepted and thea_rwnddoes not shrink. Is this intended?
No. The a_rwnd should go down to 0 as before.
Quoted 2 lines
We also figured out that after about 670947 messages of size 100 bytes the association is aborted. Is this intended?
This is also not intended. We'll take a look.
Thanks
-vlad
Quoted 7 lines
- Sending messages of size 10 bytes is similar to 100 byes.
Best regards
Michael
Hi Vlad,
Irene has done some more testing.
The ABORT was due to the fact that the testtool was not responding to HEARTBEATs. She fixed that.
She is still investigating, but the conjecture is that for small messages
the receiver accepts floor(2*s/(178+msgsize)) messages, where s is the receive space set by a socket option. Small means smaller than 180, but 178 could be the real number. Irene will check this tomorrow. In that case the a_rwnd goes down, and when the above limit is reached, a_rwnd is not decremented anymore, the messages are not put into the receive buffer anymore (checked if they are delivered to the application),
but the cumtsn is still incremented. So the sender thinks, the messages are
successfully transmitted, but the receiver just drops them.
Irene will provide definite test case to reproduce the problem tomorrow.
Best regards
Michael
Quoted 43 lines · Vlad Yasevich
Hi Michael
Quoted 15 lines · Michael Tuexen
Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using theSO_RCVBUFsocket option)
to 10000 we oberserve the following:
- When sending messages of size 1000 bytes, the receiver SACK the first,
announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. Thea_rwndgoes down to 0 and discardsQuoted 8 lines
messages.
Everything is fine.
- When sending messages of size 100 bytes, the receiver SACKs the first
messages and reduces thea_rwndaccordingly. Then it looks like the receive buffer grows, because messages are accepted and thea_rwnd
does
not shrink. Is this intended?No. The
a_rwndshould go down to 0 as before.Quoted 3 lines
We also figured out that after about 670947 messages of size 100
bytes
the association is aborted. Is this intended?This is also not intended. We'll take a look.
Thanks
-vladQuoted 3 lines
- Sending messages of size 10 bytes is similar to 100 byes. Best regards
Michael
Quoted 29 lines · Vlad Yasevich
Hi Michael
Quoted 21 lines · Michael Tuexen
Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using the
SO_RCVBUFsocket option) to 10000 we oberserve the following:- When sending messages of size 1000 bytes, the receiver SACK the first, announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The
a_rwndgoes down to 0 and discards messages. Everything is fine.Quoted 4 lines
- When sending messages of size 100 bytes, the receiver SACKs the first messages and reduces the
a_rwndaccordingly. Then it looks like the receive buffer grows, because messages are accepted and thea_rwnddoes not shrink. Is this intended?
Quoted 7 lines
No. The
a_rwndshould go down to 0 as before.Quoted 2 lines
We also figured out that after about 670947 messages of size 100 bytes the association is aborted. Is this intended?
This is also not intended. We'll take a look.
That does sound odd, yes, a_rwnd should be reduced in response to each received frame. do you have tcpdumps of this test available?
Thanks
Neil
Signature · 6 lines
/*************************************************** *Neil Horman *[email protected] *gpg keyid: 1024D / 0x92A74FA1 *http://pgp.mit.edu ***************************************************/
Quoted 41 lines · Neil Horman
Quoted 40 lines · Vlad Yasevich
Hi Michael
Quoted 24 lines · Michael Tuexen
Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using the
SO_RCVBUFsocket option)
to 10000 we oberserve the following:- When sending messages of size 1000 bytes, the receiver SACK the first,
announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. Thea_rwndgoes down to 0 and discards messages.
Everything is fine.Quoted 6 lines
- When sending messages of size 100 bytes, the receiver SACKs the first
messages and reduces thea_rwndaccordingly. Then it looks like the receive buffer grows, because messages are accepted and thea_rwnddoes
not shrink. Is this intended?No. The
a_rwndshould go down to 0 as before.Quoted 3 lines
We also figured out that after about 670947 messages of size 100 bytes
the association is aborted. Is this intended?
Quoted 7 lines
Quoted 2 lines
This is also not intended. We'll take a look.
That does sound odd, yes,
a_rwndshould be reduced in response to each received
frame. do you have tcpdumps of this test available? Thanks
Neil
It looks like that you do not reduce the a_rwnd, because you discard the message.
The problem seems to be that you ack it by incrementing the cumtsn_ack...
Irene did the tests in the lab. We'll provide traces and a testprogram tomorrow.
Quoted 11 lines
Signature · 7 lines
-- /*************************************************** *Neil Horman *[email protected] *gpg keyid: 1024D / 0x92A74FA1 *http://pgp.mit.edu ***************************************************/
Quoted 31 lines · Neil Horman
Quoted 30 lines · Vlad Yasevich
Hi Michael
Quoted 21 lines · Michael Tuexen
Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using the
SO_RCVBUFsocket option) to 10000 we oberserve the following:- When sending messages of size 1000 bytes, the receiver SACK the first, announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. The
a_rwndgoes down to 0 and discards messages. Everything is fine.Quoted 4 lines
- When sending messages of size 100 bytes, the receiver SACKs the first messages and reduces the
a_rwndaccordingly. Then it looks like the receive buffer grows, because messages are accepted and thea_rwnddoes not shrink. Is this intended?No. The
a_rwndshould go down to 0 as before.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
We also figured out that after about 670947 messages of size 100 bytes the association is aborted. Is this intended?
This is also not intended. We'll take a look.
That does sound odd, yes,
a_rwndshould be reduced in response to each received frame. do you have tcpdumps of this test available?
Thanks
Neil
I think this is hitting a condition where the receiver buffer is exhausted prior to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue reporting the tsn as received and move the cum-tsn if appropriate.
We've been trying to figure out what the correct way to solve this condition is and so far haven't come up with a workable solution.
One approach is to reduce the a_rwnd to 1 when the receive buffer reaches some consumption threshold. This would allow 1 packet in flight to allow receive buffer to drain. Another piece of this puzzle is also SWS avoidance. We currently have somewhat poor algorithm that allows a max of 1 MTU in transit.
-vlad
Quoted 48 lines · Vlad Yasevich
Quoted 39 lines · Neil Horman
Quoted 38 lines · Vlad Yasevich
Hi Michael
Quoted 24 lines · Michael Tuexen
Hi Vlad,
we are currently testing the receive behaviour of Linux (and other systems).
We are using Fedora 9, kernel 2.6.25-14.
The receiver application just opens a 1-to-many style socket and sleeps forever, not reading any messages.
The sender (on a different machine), sends a lot of messages of the same size, all on stream 0, ordered.
When setting the receive buffer space (using the
SO_RCVBUFsocket option)
to 10000 we oberserve the following:- When sending messages of size 1000 bytes, the receiver SACK the first,
announces 9000 bytes windows, SACKs the second announces 8000 bytes and so on. Look fine. Thea_rwndgoes down to 0 and discards messages.
Everything is fine.Quoted 6 lines
- When sending messages of size 100 bytes, the receiver SACKs the first
messages and reduces thea_rwndaccordingly. Then it looks like the receive buffer grows, because messages are accepted and thea_rwnddoes
not shrink. Is this intended?No. The
a_rwndshould go down to 0 as before.Quoted 3 lines
We also figured out that after about 670947 messages of size 100 bytes
the association is aborted. Is this intended?Quoted 7 lines
Quoted 2 lines
This is also not intended. We'll take a look.
That does sound odd, yes,
a_rwndshould be reduced in response to each received
frame. do you have tcpdumps of this test available? Thanks
Neil
Quoted 6 lines
I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.
This would explain the behaviour completely.
Quoted 5 lines
We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.
We are looking at the behavior of different implementations to look at interoperability
issues. That is how we came to test this...
Quoted 8 lines
One approach is to reduce the
a_rwndto 1 when the receive buffer reaches some
consumption threshold. This would allow 1 packet in flight to allow receive buffer to drain. Another piece of this puzzle is also SWS avoidance. We currently have somewhat poor algorithm that allows a max of 1 MTU in transit.
BSD reduces the a_rwnd to 1 in case of small windows (less than 3000 bytes) for SWS avoidance,
which you have to do anyway. And it makes sure that it does not report a message as received when it
is discarded.
Quoted 4 lines
-vlad
Quoted 10 lines · Vlad Yasevich
Quoted 1 line · Neil Horman
On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.
Quoted 4 lines
We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.
You're right it does sound like that. You know, I haven't visited that code since we rewrote the receive buffer management code to expand according to available memory with the sk_mem_schedule api. Do you think this could be as simple as removing this drop point?
Neil
Dear all,
for message size larger or equal to 176 byte, the behaviour seems to be OK, for smaller ones there is a problem.
Please find attached two traces. arwnd175.pcap shows the buggy behaviour, arwnd176.pcap shows the acceptable behaviour.
One the receiver side we just opened a listening 1-to-many style socket, waited until the sender (a testtool) sent all its data and then started to receive all messages. This is how we figured out that messages were lost.
Please let us know if you need more information.
Best regards
Michael
Quoted 23 lines · Neil Horman
Quoted 10 lines · Vlad Yasevich
Quoted 1 line · Neil Horman
On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.Quoted 4 lines
We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.You're right it does sound like that. You know, I haven't visited that code since we rewrote the receive buffer management code to expand according to available memory with the
sk_mem_scheduleapi. Do you think this could be as simple as removing this drop point?Neil
Quoted 10 lines · Neil Horman
Quoted 9 lines · Vlad Yasevich
Quoted 1 line · Neil Horman
On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.
Quoted 9 lines
Quoted 4 lines
We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.You're right it does sound like that. You know, I haven't visited that code since we rewrote the receive buffer management code to expand according to available memory with the
sk_mem_scheduleapi. Do you think this could be as simple as removing this drop point?
The problem is that if the user sets the buffer size, we don't auto-tune it. This is what happened in this case. However, even with auto-tunning, this still shows up with 1 byte data chunks.
So, 2 things need to happen:
1) when we drop the chunk due to allocation failure, we have to remove the tsn from the map. This one is easy
2) We need to properly detect SWS. I haven't looked fully at this, but it feels a bit more involved. Implementing something like what BSD has would also work, i.e. reducing a_rwnd 1 when receive buffer is about to be exhausted.
-vlad
Hi Vlad,
is there any description available how the auto-tuning works? But even with auto-tuning you still have the a_rwnd you report to the outside world (in SACKs) and an internal value which you use for tracking you buffer, right?
Best regards
Michael
Quoted 44 lines · Vlad Yasevich
Quoted 14 lines · Neil Horman
Quoted 9 lines · Vlad Yasevich
Quoted 1 line · Neil Horman
On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.Quoted 4 lines
We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.Quoted 7 lines
You're right it does sound like that. You know, I haven't visited that code
since we rewrote the receive buffer management code to expand according to
available memory with thesk_mem_scheduleapi. Do you think this could be as
simple as removing this drop point?The problem is that if the user sets the buffer size, we don't auto- tune it.
This is what happened in this case. However, even with auto- tunning, this
still shows up with 1 byte data chunks.So, 2 things need to happen:
1) when we drop the chunk due to allocation failure, we have to
remove
the tsn from the map. This one is easy2) We need to properly detect SWS. I haven't looked fully at
this, but
it feels a bit more involved. Implementing something
like what BSD
has would also work, i.e. reducinga_rwnd1 when receive buffer
is
about to be exhausted.-vlad
Quoted 3 lines · Michael Tuexen
Hi Vlad,
is there any description available how the auto-tuning works?
There is no formal writeup. In short it works the same as TCP.
Essentially, we start out with a set buffer space and calculated rwnd. Then as buffer space and window is consumed, we see if buffer space is consumed faster then rwnd. If we reach a point where default buffer space is consumed, but rwnd is not, we grow the buffer space slowly up to the next threshold, thus allowing all of rwnd to be consumed.
There is also a system-wide threshold that we look at when growing
the buffer. So that if the system is experiencing memory pressure
conditions, the growth and allocations will fail.
Quoted 3 lines
But even with auto-tuning you still have the
a_rwndyou report to the outside world (in SACKs) and an internal value which you use for tracking you buffer, right?
Buffer space tracking is separate from rwnd and a_rwnd that we advertise.
-vlad
Quoted 4 lines
Best regards
Michael
Quoted 7 lines · Michael Tuexen
Hi Vlad,
is there any description available how the auto-tuning works? But even with auto-tuning you still have the
a_rwndyou report to the outside world (in SACKs) and an internal value which you use for tracking you buffer, right?
As Vlad said, there is no formal description available, but if you want a functional overview, the sk_mem_schedule api is the place to look to understand how thresholds, and memory pressure detection work
Regards
Neil
Quoted 45 lines
Best regards
MichaelQuoted 40 lines · Vlad Yasevich
Quoted 14 lines · Neil Horman
Quoted 9 lines · Vlad Yasevich
Quoted 1 line · Neil Horman
On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd. We generally mark the TSN as received prior to attempting an internal allocation to carry the data. Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.Quoted 4 lines
We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.Quoted 7 lines
You're right it does sound like that. You know, I haven't visited that code
since we rewrote the receive buffer management code to expand according to
available memory with thesk_mem_scheduleapi. Do you think this could be as
simple as removing this drop point?The problem is that if the user sets the buffer size, we don't auto-tune it.
This is what happened in this case. However, even with auto-tunning, this still shows up with 1 byte data chunks.So, 2 things need to happen:
1) when we drop the chunk due to allocation failure, we have to remove the tsn from the map. This one is easy2) We need to properly detect SWS. I haven't looked fully at this, but it feels a bit more involved. Implementing something like what
BSD
has would also work, i.e. reducinga_rwnd1 when receive buffer is about to be exhausted.-vlad
Signature · 4 lines
/**************************************************** * Neil Horman <[email protected]> * Software Engineer, Red Hat ****************************************************/