Skip to contentSkip to conversation

Send fail notifications

DISCUSSION Jamal Hadi Salim 2 participants 4 messages Dec 18, 2013 – Dec 19, 2013 SCTP protocol

lore
Share & tools
lore
#0 lore raw

Lets say between server and client i have an MTU of 4K. I send 16K with partial reliability setting.
If the data gets through the receiver sees (I think SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably defaults to MTU):

Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOR

OTOH, at sender if the send fails because PR timeout on some data chunk occurs i receive from the kernel not 1 but 4 notifications. Msg1, 4k, Notification(send failed, ssf_length=4k), EOR Msg2, 4k, Notification(send failed, ssf_length=4k), EOR Msg3, 4k, Notification(send failed, ssf_length=4k), EOR Msg4, 4k, Notification(send failed, ssf_length=4k), EOR .... some last message here to account for notification overhead ....

I was expecting only the last one to have an EOR or least the ssf_length to be telling me there's more than 4K. Stoopid question:
Is there some other field I should be looking at?

The only way i could tell the 5 notifications are from the same original message is when i provide a sinfo_context at sendmsg() time.

cheers,
jamal

#1 lore raw

A little more info:
- When i have MTU > 16K, no problem, 1 notification makes it through with an EOR and proper info
EOR.
- ssf_flags are always zero (thats one other place that may have carried such extra info)
- kernel 3.12.0 (with other non-sctp patches)

cheers,
jamal

Quoted 30 lines · Jamal Hadi Salim

Lets say between server and client i have an MTU of 4K. I send 16K with partial reliability setting.
If the data gets through the receiver sees (I think SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably defaults to MTU):

Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOR

OTOH, at sender if the send fails because PR timeout on some data chunk occurs i receive from the kernel not 1 but 4 notifications. Msg1, 4k, Notification(send failed, ssf_length=4k), EOR Msg2, 4k, Notification(send failed, ssf_length=4k), EOR Msg3, 4k, Notification(send failed, ssf_length=4k), EOR Msg4, 4k, Notification(send failed, ssf_length=4k), EOR .... some last message here to account for notification overhead ....

I was expecting only the last one to have an EOR or least the ssf_length to be telling me there's more than 4K. Stoopid question:
Is there some other field I should be looking at?

The only way i could tell the 5 notifications are from the same original message is when i provide a sinfo_context at sendmsg() time.

cheers,
jamal

#2 lore raw
In reply to Jamal Hadi Salim
Quoted 7 lines · Jamal Hadi Salim

A little more info:
- When i have MTU > 16K, no problem, 1 notification makes it through with an EOR and proper info
EOR.
- ssf_flags are always zero (thats one other place that may have carried such extra info)

meaning that chunks was unsent.

Quoted 25 lines

- kernel 3.12.0 (with other non-sctp patches)

cheers,
jamal

Quoted 19 lines · Jamal Hadi Salim

Lets say between server and client i have an MTU of 4K. I send 16K with partial reliability setting.
If the data gets through the receiver sees (I think SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably defaults to MTU):

Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOR

OTOH, at sender if the send fails because PR timeout on some data chunk occurs i receive from the kernel not 1 but 4 notifications. Msg1, 4k, Notification(send failed, ssf_length=4k), EOR Msg2, 4k, Notification(send failed, ssf_length=4k), EOR Msg3, 4k, Notification(send failed, ssf_length=4k), EOR Msg4, 4k, Notification(send failed, ssf_length=4k), EOR .... some last message here to account for notification overhead ....

You receive one notification per chunk. This is the way lksctp implemented failure notification. Whether this is the right or wrong way to do it may be up for debate.

As I said in the private thread that started this, current spec is referring to messages when it talks about SEND_FAILED[_EVENT]. It covers the case where the whole message has failed and when partial message has failed, but it misses the case where parts of message failed for different reasons (SENT vs. UNSENT). In particular the setting sinfo_flags in the sndrcvinfo structure aren't covered in this case.

So, even when talking about messages, it's feasible that you would receive multiple notifications for the same message. One may specify things like:
ssf_flags = SENT
sinfo.sinfo_flags = DATA_FIRST_FRAG

the other may have:
ssf_info = UNSENT
sinfo.sinfo_flags = DATA_LAST_FRAG

The EOR flags would be sent on each notification if was received by your application in full, since in this case EOR signals the end of a particular notification, the the end of the user data message we are reporting about.

Quoted 4 lines
Quoted 4 lines

I was expecting only the last one to have an EOR or least the ssf_length to be telling me there's more than 4K. Stoopid question:
Is there some other field I should be looking at?

You could look at the sinfo_flags. These flags will mirror the flags from the chunk thus allowing you to kind-of reassemble the data. The hitch here is the following text in the spec:
ssf_info: This field includes the ancillary data (struct
sctp_sndrcvinfo) used to send the undelivered message
^^^^^^^^^^^^^^^^^^^^^^^
lksctp follows that by simply copying the user supplied data, thus missing things like chunk ssn, tsn and other fields that could be useful in identifying this part of the message.

Quoted 4 lines
Quoted 4 lines

The only way i could tell the 5 notifications are from the same original message is when i provide a sinfo_context at sendmsg() time.

That really is the only reliable way to tell if the parts came from the same message.
In fact, if you look at the description of sinfo_context in Section 5.3.2, you'll see that this is exactly what it's there for.

-vlad

-vlad

#3 lore raw
In reply to Jamal Hadi Salim
Quoted 6 lines · Vlad Yasevich

On 12/18/2013 09:06 AM, Jamal Hadi Salim wrote:

You receive one notification per chunk. This is the way lksctp implemented failure notification. Whether this is the right or wrong way to do it may be up for debate.

Ok, assuming the chunk size is corellated to MTU.
With large enough MTU i do receive the full notification + original message in one read.

Quoted 18 lines

As I said in the private thread that started this, current spec is referring to messages when it talks about SEND_FAILED[_EVENT]. It covers the case where the whole message has failed and when partial message has failed, but it misses the case where parts of message failed for different reasons (SENT vs. UNSENT). In particular the setting sinfo_flags in the sndrcvinfo structure aren't covered in this case.

So, even when talking about messages, it's feasible that you would receive multiple notifications for the same message. One may specify things like:
ssf_flags = SENT
sinfo.sinfo_flags = DATA_FIRST_FRAG

the other may have:
ssf_info = UNSENT
sinfo.sinfo_flags = DATA_LAST_FRAG

Quoted 7 lines

The EOR flags would be sent on each notification if was received by your application in full, since in this case EOR signals the end of a particular notification, the the end of the user data message we are reporting about.

You could look at the sinfo_flags. These flags will mirror the flags from the chunk thus allowing you to kind-of reassemble the data.

Aha, thanks.
Yes, I think these would good enough - painful but sufficient. I wasnt paying close attention to the sinfo flags earlier
I still expect to receive the full message (although i have complained about receiving the full message in the past ;->). I see the flag starting with "2" middle is "0" and termination is "1". Sample log for one of the messages that failed on my side is:
------
handle_send_failed: Data(cntx 91267) 1024B (0x2) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 892B (0x1) never left
-----

Quoted 8 lines

The hitch here is the following text in the spec:
ssf_info: This field includes the ancillary data (struct
sctp_sndrcvinfo) used to send the undelivered message
^^^^^^^^^^^^^^^^^^^^^^^
lksctp follows that by simply copying the user supplied data, thus missing things like chunk ssn, tsn and other fields that could be useful in identifying this part of the message.

I am probably fine with what you described above.

Quoted 5 lines

That really is the only reliable way to tell if the parts came from the same message.
In fact, if you look at the description of sinfo_context in Section 5.3.2, you'll see that this is exactly what it's there for.

I love that feature ;-> So for now i will have to use both. Note, our current use does not desire to retransmit such a message - if it didnt make it then it is obsolete. But we need to keep track which particular message didnt get through; and for that we only need the first chunk.

cheers,
jamal