Send fail notifications
Lets say between server and client i have an MTU of 4K. I send 16K with partial reliability setting.
If the data gets through the receiver sees (I think SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably defaults to MTU):
Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOR
OTOH, at sender if the send fails because PR timeout on some data chunk occurs i receive from the kernel not 1 but 4 notifications. Msg1, 4k, Notification(send failed, ssf_length=4k), EOR Msg2, 4k, Notification(send failed, ssf_length=4k), EOR Msg3, 4k, Notification(send failed, ssf_length=4k), EOR Msg4, 4k, Notification(send failed, ssf_length=4k), EOR .... some last message here to account for notification overhead ....
I was expecting only the last one to have an EOR or least the ssf_length to be telling me there's more than 4K. Stoopid question:
Is there some other field I should be looking at?
The only way i could tell the 5 notifications are from the same original message is when i provide a sinfo_context at sendmsg() time.
cheers,
jamal
A little more info:
- When i have MTU > 16K, no problem, 1 notification makes it through with an EOR and proper info
EOR.
- ssf_flags are always zero (thats one other place that may have carried such extra info)
- kernel 3.12.0 (with other non-sctp patches)
cheers,
jamal
Quoted 30 lines · Jamal Hadi Salim
Lets say between server and client i have an MTU of 4K. I send 16K with partial reliability setting.
If the data gets through the receiver sees (I thinkSCTP_PARTIAL_DELIVERY_POINTmay change this behavior but it probably defaults to MTU):Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOROTOH, at sender if the send fails because PR timeout on some data chunk occurs i receive from the kernel not 1 but 4 notifications. Msg1, 4k,
Notification(send failed, ssf_length=4k), EOR Msg2, 4k,Notification(send failed, ssf_length=4k), EOR Msg3, 4k,Notification(send failed, ssf_length=4k), EOR Msg4, 4k,Notification(send failed, ssf_length=4k), EOR .... some last message here to account for notification overhead ....I was expecting only the last one to have an EOR or least the
ssf_lengthto be telling me there's more than 4K. Stoopid question:
Is there some other field I should be looking at?The only way i could tell the 5 notifications are from the same original message is when i provide a
sinfo_contextatsendmsg()time.cheers,
jamal
Quoted 7 lines · Jamal Hadi Salim
A little more info:
- When i have MTU > 16K, no problem, 1 notification makes it through with an EOR and proper info
EOR.
-ssf_flagsare always zero (thats one other place that may have carried such extra info)
meaning that chunks was unsent.
Quoted 25 lines
- kernel 3.12.0 (with other non-sctp patches)
cheers,
jamalQuoted 19 lines · Jamal Hadi Salim
Lets say between server and client i have an MTU of 4K. I send 16K with partial reliability setting.
If the data gets through the receiver sees (I thinkSCTP_PARTIAL_DELIVERY_POINTmay change this behavior but it probably defaults to MTU):Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOROTOH, at sender if the send fails because PR timeout on some data chunk occurs i receive from the kernel not 1 but 4 notifications. Msg1, 4k,
Notification(send failed, ssf_length=4k), EOR Msg2, 4k,Notification(send failed, ssf_length=4k), EOR Msg3, 4k,Notification(send failed, ssf_length=4k), EOR Msg4, 4k,Notification(send failed, ssf_length=4k), EOR .... some last message here to account for notification overhead ....
You receive one notification per chunk. This is the way lksctp implemented failure notification. Whether this is the right or wrong way to do it may be up for debate.
As I said in the private thread that started this, current spec is referring to messages when it talks about SEND_FAILED[_EVENT]. It covers the case where the whole message has failed and when partial message has failed, but it misses the case where parts of message failed for different reasons (SENT vs. UNSENT). In particular the setting sinfo_flags in the sndrcvinfo structure aren't covered in this case.
So, even when talking about messages, it's feasible that you would receive multiple notifications for the same message. One may specify things like:
ssf_flags = SENT
sinfo.sinfo_flags = DATA_FIRST_FRAG
the other may have:
ssf_info = UNSENT
sinfo.sinfo_flags = DATA_LAST_FRAG
The EOR flags would be sent on each notification if was received by your application in full, since in this case EOR signals the end of a particular notification, the the end of the user data message we are reporting about.
Quoted 4 lines
Quoted 4 lines
I was expecting only the last one to have an EOR or least the
ssf_lengthto be telling me there's more than 4K. Stoopid question:
Is there some other field I should be looking at?
You could look at the sinfo_flags. These flags will mirror the flags from the chunk thus allowing you to kind-of reassemble the data. The hitch here is the following text in the spec:
ssf_info: This field includes the ancillary data (struct
sctp_sndrcvinfo) used to send the undelivered message
^^^^^^^^^^^^^^^^^^^^^^^
lksctp follows that by simply copying the user supplied data, thus missing things like chunk ssn, tsn and other fields that could be useful in identifying this part of the message.
Quoted 4 lines
Quoted 4 lines
The only way i could tell the 5 notifications are from the same original message is when i provide a
sinfo_contextatsendmsg()time.
That really is the only reliable way to tell if the parts came from the same message.
In fact, if you look at the description of sinfo_context in Section 5.3.2, you'll see that this is exactly what it's there for.
-vlad
-vlad
Quoted 6 lines · Vlad Yasevich
On 12/18/2013 09:06 AM, Jamal Hadi Salim wrote:
You receive one notification per chunk. This is the way lksctp implemented failure notification. Whether this is the right or wrong way to do it may be up for debate.
Ok, assuming the chunk size is corellated to MTU.
With large enough MTU i do receive the full notification + original message in one read.
Quoted 18 lines
As I said in the private thread that started this, current spec is referring to messages when it talks about SEND_FAILED[_EVENT]. It covers the case where the whole message has failed and when partial message has failed, but it misses the case where parts of message failed for different reasons (SENT vs. UNSENT). In particular the setting
sinfo_flagsin the sndrcvinfo structure aren't covered in this case.So, even when talking about messages, it's feasible that you would receive multiple notifications for the same message. One may specify things like:
ssf_flags= SENT
sinfo.sinfo_flags =DATA_FIRST_FRAGthe other may have:
ssf_info= UNSENT
sinfo.sinfo_flags =DATA_LAST_FRAG
Quoted 7 lines
The EOR flags would be sent on each notification if was received by your application in full, since in this case EOR signals the end of a particular notification, the the end of the user data message we are reporting about.
You could look at the
sinfo_flags. These flags will mirror the flags from the chunk thus allowing you to kind-of reassemble the data.
Aha, thanks.
Yes, I think these would good enough - painful but sufficient. I wasnt paying close attention to the sinfo flags earlier
I still expect to receive the full message (although i have complained about receiving the full message in the past ;->). I see the flag starting with "2" middle is "0" and termination is "1". Sample log for one of the messages that failed on my side is:
------handle_send_failed: Data(cntx 91267) 1024B (0x2) possibly lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) possibly lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) possibly lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) possibly lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) possibly lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) possibly lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 1024B (0x0) never lefthandle_send_failed: Data(cntx 91267) 892B (0x1) never left
-----
Quoted 8 lines
The hitch here is the following text in the spec:
ssf_info: This field includes the ancillary data (struct
sctp_sndrcvinfo) used to send the undelivered message
^^^^^^^^^^^^^^^^^^^^^^^
lksctp follows that by simply copying the user supplied data, thus missing things like chunk ssn, tsn and other fields that could be useful in identifying this part of the message.
I am probably fine with what you described above.
Quoted 5 lines
That really is the only reliable way to tell if the parts came from the same message.
In fact, if you look at the description ofsinfo_contextin Section 5.3.2, you'll see that this is exactly what it's there for.
I love that feature ;-> So for now i will have to use both. Note, our current use does not desire to retransmit such a message - if it didnt make it then it is obsolete. But we need to keep track which particular message didnt get through; and for that we only need the first chunk.
cheers,
jamal