Extending socket timestamping API for NTP
Subsystems (2)
I'd like to propose some changes and new options for the timestamping interface that I think would be useful for NTP implementations and maybe also other applications. Before I or someone else tries to implement them, do you think they would actually make sense and fit well in the current code?
1) new rx_filter for NTP
Some NICs can't timestamp all received packets and are currently unusable for NTP with HW timestamping. The new filter would allow NTP support in new NICs and adding support to existing NICs with firmware/driver updates. The filter would apply to IPv4 and IPv6 UDP packets received from or sent to the port number 123.
Should be the current drivers of HW that can timestamp all packets updated to fall back to HWTSTAMP_FILTER_ALL?
2) new SO_TIMESTAMPING option to receive from the error queue only user data as was passed to sendmsg() instead of Ethernet frames
Parsing Ethernet and IP headers (especially IPv6 options) is not fun and SOF_TIMESTAMPING_OPT_ID is not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.
3) target address in msg_name of messages from the error queue
With 2) and unconnected sockets, there needs to be a way to get the address to which the packet was sent. Is it ok to always fill msg_name, or does it need to be a new option?
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available. Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
5) new SO_TIMESTAMPING options to get transposed RX timestamps
PTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
The existing SOF_TIMESTAMPING_RX_HARDWARE flag could be aliased to SOF_TIMESTAMPING_RX_HARDWARE_PREAMBLE and the new flag could be SOF_TIMESTAMPING_RX_HARDWARE_TRAILER.
PTP has a similar problem with SW RX timestamps, which are closer to the trailer timestamps rather than preamble timestamps. A new SOF_TIMESTAMPING_RX_SOFTWARE_PREAMBLE flag could be added for PTP implementations to get transposed timestamps in order to improve accuracy.
6) new SO_TIMESTAMPING option to get PHC index with HW timestamps
With bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
I'm not sure what would be the best place to put it. I guess the second timespec in scm_timestamping could be reused for this, but that sounds like a gross hack. Do we need to define a new struct?
Thoughts?
Signature · 1 line
Miroslav Lichvar
Hi Miroslav,
Quoted 15 lines
-----Original Message-----
From: Miroslav Lichvar [mailto:[email protected]]
Sent: Tuesday, February 07, 2017 6:02 AM
To: [email protected]
Cc: Richard Cochran <[email protected]>; Jiri Benc <[email protected]>; Keller, Jacob E <[email protected]>; Denny Page <[email protected]>; Willem de Bruijn <[email protected]>
Subject: Extending socket timestamping API for NTPI'd like to propose some changes and new options for the timestamping interface that I think would be useful for NTP implementations and maybe also other applications. Before I or someone else tries to implement them, do you think they would actually make sense and fit well in the current code?
Quoted 7 lines
1) new
rx_filterfor NTPSome NICs can't timestamp all received packets and are currently unusable for NTP with HW timestamping. The new filter would allow NTP support in new NICs and adding support to existing NICs with
firmware/driverupdates. The filter would apply to IPv4 and IPv6 UDP packets received from or sent to the port number 123.
The main problem here is that most hardware that *can't* timestamp all packets is pretty limited to timestamping only PTP frames. It's possible with firmware upgrades this could be worked around, but I do not know if it would actually happen. Still, it can't really hurt too much to add a new filter, and those drivers which can support it already should be easy to implement.
Quoted 3 lines
Should be the current drivers of HW that can timestamp all packets updated to fall back to
HWTSTAMP_FILTER_ALL?
Generally, the drivers I am aware of that support timestamping all packets do so for any filter request, rather than actually limiting the timestamping.
Quoted 8 lines
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet framesParsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.
This would be useful for application writing.
Quoted 6 lines
3) target address in
msg_nameof messages from the error queueWith 2) and unconnected sockets, there needs to be a way to get the address to which the packet was sent. Is it ok to always fill
msg_name, or does it need to be a new option?
I'm not sure.
Quoted 9 lines
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available. Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
This seems useful, but not sure how best to implement it.
Quoted 4 lines
Thoughts?
--
Miroslav Lichvar
Quoted 9 lines
Quoted 7 lines
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet framesParsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.This would be useful for application writing.
What kind of user data are you suggesting? Just a user-defined ID passed as a cmsg? Allowing such metadata to override skb_shinfo(skb)->tskey sounds fine.
Quoted 8 lines
Quoted 5 lines
3) target address in
msg_nameof messages from the error queueWith 2) and unconnected sockets, there needs to be a way to get the address to which the packet was sent. Is it ok to always fill
msg_name, or does it need to be a new option?I'm not sure.
This would be an argument to just loop the original packet.
Quoted 11 lines
Quoted 8 lines
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available. Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
This seems useful,
Agreed, as long as it is optional so that it does not change the behavior for existing applications.
Quoted 1 line
but not sure how best to implement it.
It might be sufficient to just remove the second line in sw_tx_timestamp
static inline void sw_tx_timestamp(struct sk_buff *skb)
{
if (skb_shinfo(skb)->tx_flags & SKBTX_SW_TSTAMP &&
!(skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS))
skb_tstamp_tx(skb, NULL);
}Quoted 8 lines · Willem de Bruijn
Quoted 8 lines
Quoted 7 lines
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet framesParsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.
Quoted 5 lines
Quoted 1 line
This would be useful for application writing.
What kind of user data are you suggesting? Just a user-defined ID passed as a cmsg? Allowing such metadata to override skb_shinfo(skb)->tskey sounds fine.
Yes, exactly. Just a user-defined ID that overrides the skb_shinfo(skb)->tskey.
Thanks,
Soheil
Quoted 10 lines · Willem de Bruijn
Quoted 10 lines
Quoted 8 lines
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available. Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
Quoted 4 lines
Quoted 1 line
This seems useful,
Agreed, as long as it is optional so that it does not change the behavior for existing applications.
Do you think it is safe to assume that no application enabled both SW and HW TX timestamping? Do we need a new option for this?
Quoted 10 lines
Quoted 1 line
but not sure how best to implement it.
It might be sufficient to just remove the second line in
sw_tx_timestampstatic inline void
sw_tx_timestamp(struct sk_buff *skb){ if (skb_shinfo(skb)->tx_flags & SKBTX_SW_TSTAMP && !(skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS)) skb_tstamp_tx(skb, NULL); }
With this change I'm getting two error messages per transmission, but it looks like it may need some additional changes.
If the first error message is received after the HW timestamp was captured, it contains both timestamps as the HW timestamp is in the shared info of the skb. Is it possible it could contain a partially updated HW timestamp? I'm not sure how locking works here. Is scm_timestamping actually allowed to contain more than one timestamp? The timestamping.txt document says "Only one field is non-zero at any time.", but that wasn't true even before if both SW and HW RX timestamping was enabled.
If SO_TIMESTAMP{,NS} is enabled, ts[0] in the second error message will contain a bogus SW timestamp added by __sock_recv_timestamp() for a "Race occurred between timestamp enabling and packet receiving". Is there a guarantee applications will get a timestamp for all messages after enabling SO_TIMESTAMP? The original code is older than the git repo, so I'm not sure what was the reason for this. To me it would make more sense to not add any SCM_TIMESTAMP (and SW timestamp in SCM_TIMESTAMPING) when the the timestamp is missing. If that's not always acceptable, maybe it could be restricted to sockets that have HW timestamping enabled?
Some drivers don't call skb_tx_timestamp() when HW timestamp was requested. From a cursory look it is e1000e, xgbe, sxgbe, and stmmac. This should hopefully be an easy fix.
Thoughts?
Signature · 1 line
Miroslav Lichvar
Quoted 11 lines · Miroslav Lichvar
Quoted 10 lines · Willem de Bruijn
Quoted 10 lines
Quoted 8 lines
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available. Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
This seems useful,
Agreed, as long as it is optional so that it does not change the behavior for existing applications.
Do you think it is safe to assume that no application enabled both SW and HW TX timestamping?
We cannot rule out that a process set both flags.
Quoted 1 line
Do we need a new option for this?
Similar to OPT_TSONLY or OPT_ID, but to signal the intent of receiving both timestamps. Yes, agreed.
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
but not sure how best to implement it.
It might be sufficient to just remove the second line in
sw_tx_timestampstatic inline void
sw_tx_timestamp(struct sk_buff *skb){ if (skb_shinfo(skb)->tx_flags & SKBTX_SW_TSTAMP && !(skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS)) skb_tstamp_tx(skb, NULL); }
Quoted 5 lines
With this change I'm getting two error messages per transmission, but it looks like it may need some additional changes.
If the first error message is received after the HW timestamp was captured,
When does this happen? The first timestamp is generated from skb_tx_timestamp in the device driver's ndo_start_xmit before passing the packet to the NIC, the second when the device driver cleans the tx descriptor on completion.
Is this for drivers that do not have skb_tx_timestamp, as you mention below? Then the solution is to add that call.
Quoted 11 lines
it contains both timestamps as the HW timestamp is in the shared info of the skb. Is it possible it could contain a partially updated HW timestamp? I'm not sure how locking works here. Is
scm_timestampingactually allowed to contain more than one timestamp? The timestamping.txt document says "Only one field is non-zero at any time.", but that wasn't true even before if both SW and HW RX timestamping was enabled.If SO_TIMESTAMP{,NS} is enabled, ts[0] in the second error message will contain a bogus SW timestamp added by
__sock_recv_timestamp()for a "Race occurred between timestamp enabling and packet receiving". Is
Good point. That should not be set on transmit timestamps.
Quoted 7 lines
there a guarantee applications will get a timestamp for all messages after enabling
SO_TIMESTAMP? The original code is older than the git repo, so I'm not sure what was the reason for this. To me it would make more sense to not add anySCM_TIMESTAMP(and SW timestamp inSCM_TIMESTAMPING) when the the timestamp is missing. If that's not always acceptable, maybe it could be restricted to sockets that have HW timestamping enabled?
I would limit scope to tx timestamping and leave rx semantics as is.
Quoted 3 lines
Some drivers don't call
skb_tx_timestamp()when HW timestamp was requested. From a cursory look it is e1000e, xgbe, sxgbe, and stmmac. This should hopefully be an easy fix.
Indeed. that should be added, then.
Quoted 8 lines · Willem de Bruijn
Quoted 4 lines · Miroslav Lichvar
Quoted 1 line · Willem de Bruijn
Quoted 1 line
Quoted 1 line
4) allow sockets to use both SW and HW TX timestamping at the same time
Do we need a new option for this?
Similar to
OPT_TSONLYorOPT_ID, but to signal the intent of receiving both timestamps. Yes, agreed.
Ok. Thanks.
Quoted 10 lines
Quoted 5 lines
With this change I'm getting two error messages per transmission, but it looks like it may need some additional changes.
If the first error message is received after the HW timestamp was captured,
When does this happen? The first timestamp is generated from
skb_tx_timestampin the device driver'sndo_start_xmitbefore passing the packet to the NIC, the second when the device driver cleans the tx descriptor on completion.
As I understand it, it happens when the first skb (created by the skb_tx_timestamp() call) is received by the application after the driver called skb_tstamp_tx() with the HW timestamp. The SW timestamps are separate, but the HW timestamp is shared between clones. It probably doesn't happen with the TSONLY option as it allocates a new skb. When I print timestamps from scm_timestamping I see a mix of two cases:
TX 1488268812.193945472 0.000000000 1488286813.273760139 TX 0.000000000 0.000000000 1488286813.273760139 RX 1488268812.354356188 0.000000000 1488286813.434096389
TX 1488268816.364407934 0.000000000 0.000000000 TX 0.000000000 0.000000000 1488286817.444251014 RX 1488268816.525150589 0.000000000 1488286817.604749889
In the first case I assume the HW timestamp was saved before the first error message was received, so both error messages have the same HW timestamp, but only one has the SW timestamp. In the second case, the HW timestamp was saved later, so there is one message with SW timestamp and one message with HW timestamp.
Quoted 1 line
From the application point of view it would make sense if in the first
case there was only one error message containing both timestamps. I'm not sure how easy/safe it would be to drop the second skb. The other approach would be to not put HW timestamp in the first message when this "dual TX timestamping" option is enabled, so each error message has only one timestamp.
Quoted 2 lines
Is this for drivers that do not have
skb_tx_timestamp, as you mention below? Then the solution is to add that call.
FWIW, I saw that with the e1000e driver after I made the skb_tx_timestamp() call unconditional.
Quoted 7 lines
Quoted 7 lines
it contains both timestamps as the HW timestamp is in the shared info of the skb. Is it possible it could contain a partially updated HW timestamp? I'm not sure how locking works here. Is
scm_timestampingactually allowed to contain more than one timestamp? The timestamping.txt document says "Only one field is non-zero at any time.", but that wasn't true even before if both SW and HW RX timestamping was enabled.
Signature · 1 line
Miroslav Lichvar
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
With this change I'm getting two error messages per transmission, but it looks like it may need some additional changes.
If the first error message is received after the HW timestamp was captured,
When does this happen? The first timestamp is generated from
skb_tx_timestampin the device driver'sndo_start_xmitbefore passing the packet to the NIC, the second when the device driver cleans the tx descriptor on completion.
Quoted 4 lines
As I understand it, it happens when the first skb (created by the
skb_tx_timestamp()call) is received by the application after the driver calledskb_tstamp_tx()with the HW timestamp. The SW timestamps are separate, but the HW timestamp is shared between clones. It
Oh right, the conversion to struct scm_timestamping only happens on socket read in __sock_recv_timestamp.
Quoted 12 lines
probably doesn't happen with the TSONLY option as it allocates a new skb. When I print timestamps from
scm_timestampingI see a mix of two cases:TX 1488268812.193945472 0.000000000 1488286813.273760139 TX 0.000000000 0.000000000 1488286813.273760139 RX 1488268812.354356188 0.000000000 1488286813.434096389
TX 1488268816.364407934 0.000000000 0.000000000 TX 0.000000000 0.000000000 1488286817.444251014 RX 1488268816.525150589 0.000000000 1488286817.604749889
Quoted 8 lines
In the first case I assume the HW timestamp was saved before the first error message was received, so both error messages have the same HW timestamp, but only one has the SW timestamp. In the second case, the HW timestamp was saved later, so there is one message with SW timestamp and one message with HW timestamp.
From the application point of view it would make sense if in the first case there was only one error message containing both timestamps. I'm
Agreed. I just proposed something similar on the error queue for zerocopy notifications in http://patchwork.ozlabs.org/patch/731214/
Quoted 4 lines
not sure how easy/safe it would be to drop the second skb. The other approach would be to not put HW timestamp in the first message when this "dual TX timestamping" option is enabled, so each error message has only one timestamp.
If it's possible to avoid one skb_clone completely, then that is preferable over creating both and consuming one. If either approach becomes complex, then queuing two separate messages is fine. A process can recvmmsg(), after all. As long as the behavior is consistent.
[Resend without rich text]
Quoted 3 lines · Jacob E
The main problem here is that most hardware that *can't* timestamp all packets is pretty limited to timestamping only PTP frames.
Most, but not all. The TI DP83630 doesn’t support timestamping for all packets, but it does support either PTP or NTP:
===
2.3.2.3 NTP Packet Timestamp
The DP83630 may be programmed to timestamp NTP packets instead of PTP packets. This operation is enabled by setting the NTP_TS_EN control in the PTP_TXCFG0 register. When configured for NTP timestamps, the DP83630 will timestamp packets with the NTP UDP port number rather than the PTP port number (note that the device cannot be configured to timestamp both PTP and NTP packets). One-Step operation is not supported for NTP timestamps, so transmit timestamps cannot be inserted directly into outgoing NTP packets. Timestamp insertion is available for receive timestamps but must use a single, fixed location.
===
Right now, there is no API to signal to the driver that NTP timestamping is desired.
Even if the hardware does not directly support filtering, it can be implemented in the driver.
Denny
Quoted 1 line · Denny Page
Most, but not all. The TI DP83630 doesn’t support timestamping for all packets, but it does support either PTP or NTP:
That is the one and only device that explicitly supports NTP. This is a nice idea, of course, but it just did not take off among other products.
Thanks,
Richard
Quoted 7 lines · Richard Cochran
Quoted 1 line · Denny Page
Most, but not all. The TI DP83630 doesn’t support timestamping for all packets, but it does support either PTP or NTP:
That is the one and only device that explicitly supports NTP. This is a nice idea, of course, but it just did not take off among other products.
I have to say I haven’t gone looking for others, and will take your word regarding the DP83630 being the one and only. I only learned about the DP83630 because I have a couple of stratum 1 devices that use this phy and have been working with the vendor regarding integration of hardware timestamps for NTP.
Denny
Quoted 9 lines · Richard Cochran
Quoted 1 line · Denny Page
Most, but not all. The TI DP83630 doesn’t support timestamping for all packets, but it does support either PTP or NTP:
That is the one and only device that explicitly supports NTP. This is a nice idea, of course, but it just did not take off among other products.
With a commonly available NTP implementation supporting hardware timestamping, this might get a bit more traction with manufacturers.
Denny
Quoted 7 lines · Miroslav Lichvar
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet framesParsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.
This is going to be quite useful. However, I'm not sure if sending back the original packet would be a proper API. Instead, one option is to add a control message, so that applications can set the OPT_ID for the timestamp. Perhaps, something like from user's perspective:
cmsg->cmsg_level = SOL_SOCKET; cmsg->cmsg_type = SCM_TIMESTAMPING_OPT_ID; cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));
*((__u32 *) CMSG_DATA(cmsg)) = my_id;
Thanks,
Soheil
Quoted 14 lines · Soheil Hassas Yeganeh
Quoted 7 lines · Miroslav Lichvar
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet framesParsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.This is going to be quite useful. However, I'm not sure if sending back the original packet would be a proper API. Instead, one option is to add a control message, so that applications can set the
OPT_IDfor the timestamp. Perhaps, something like from user's perspective:
Quoted 4 lines
cmsg->cmsg_level = SOL_SOCKET; cmsg->cmsg_type = SCM_TIMESTAMPING_OPT_ID; cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));*((__u32 *)
CMSG_DATA(cmsg)) =my_id;
That could be very useful. The question is if 32 bits worth of user data would be good enough for all applications. In the case of the NTP server, I currently save 128 bits per client in order to support the interleaved mode. Half of that is the receive timestamp, which is compared to the receive timestamp from messages received from the error queue. Matching only lower 32 bits of the timestamp would probably still work fine. However, if NTP supported follow up messages like PTP, 32 bits would not be enough to create a valid message for the client without saving some additional state. Getting the original message would be very convenient here. NTP packets are normally very short, so I'm not sure how much benefit there would be in using the OPT_ID.
Signature · 1 line
Miroslav Lichvar
Quoted 8 lines · Miroslav Lichvar
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
Is this a requirement ? RFC 5905 does not seem to imply this and a search does not show any issues being reported.
Quoted 1 line
From RFC 5905
Reference Timestamp: Time when the system clock was last set or corrected, in NTP timestamp format.
Origin Timestamp (org): Time at the client when the request departed for the server, in NTP timestamp format.
Receive Timestamp (rec): Time at the server when the request arrived from the client, in NTP timestamp format.
Transmit Timestamp (xmt): Time at the server when the response left for the client, in NTP timestamp format.
Destination Timestamp (dst): Time at the client when the reply arrived from the server, in NTP timestamp format.
Note: The Destination Timestamp field is not included as a header field; it is determined upon arrival of the packet and made available in the packet buffer data structure.
If the NTP has access to the physical layer, then the timestamps are associated with the beginning of the symbol after the start of frame. Otherwise, implementations should attempt to associate the timestamp to the earliest accessible point in the frame.
Quoted 15 lines
The existing
SOF_TIMESTAMPING_RX_HARDWAREflag could be aliased toSOF_TIMESTAMPING_RX_HARDWARE_PREAMBLEand the new flag could beSOF_TIMESTAMPING_RX_HARDWARE_TRAILER.PTP has a similar problem with SW RX timestamps, which are closer to the trailer timestamps rather than preamble timestamps. A new
SOF_TIMESTAMPING_RX_SOFTWARE_PREAMBLEflag could be added for PTP implementations to get transposed timestamps in order to improve accuracy.6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestamps
Quoted 7 lines
With bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?
What is the use case for this. even if the delay though the PHY's how would that be compensated ?
RMS
Quoted 5 lines
Thoughts?
--
Miroslav Lichvar
Quoted 7 lines · sdncurious
Quoted 6 lines · Miroslav Lichvar
6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Quoted 6 lines
Quoted 3 lines
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?What is the use case for this. even if the delay though the PHY's how would that be compensated ?
The idea was that applications like NTP servers and clients wouldn't have to care about interfaces and how they map together with addresses to PHCs over time. Currently, I use the interface index from IP_PKTINFO to get the PHC, but that doesn't work with bridges and other virtual interfaces. Another possibility would be an option to modify the behavior of IP_PKTINFO to save the index of the real interface. I'm not sure how would that compare in difficulty to extending SCM_TIMESTAMPING with PHC index.
Signature · 1 line
Miroslav Lichvar
Dealing with individual interfaces does not make sense. This seems to be a case where Reciprocity property is violated and hence should be handled as such. This is different than when the two sides have single but different speed NIC's. In this case the NIC used and the speed can change with each packet. Although I am not sure if that is possible because the hash should always land the packet on the NIC of the bond.
7. Reciprocity Errors
The above analysis assumes that the delays on the outbound and inbound paths are the same; that is, the paths are reciprocal. This is assured if the ropagation delays are the same, the transmission rates are the same and the packet lengths are the same. In the NTP on-wire protocol all packets have the the same length. If we assume the transmission rates are the same, the only difference in path delays must be due to nonreciprocal transmission paths. This often occurs if one way is via landline and the other via satellite. It can also occur when the paths traverse tag-switched core networks.
RMS
Quoted 26 lines · Miroslav Lichvar
Quoted 7 lines · sdncurious
Quoted 6 lines · Miroslav Lichvar
6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Quoted 6 lines
Quoted 3 lines
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?What is the use case for this. even if the delay though the PHY's how would that be compensated ?
The idea was that applications like NTP servers and clients wouldn't have to care about interfaces and how they map together with addresses to PHCs over time. Currently, I use the interface index from
IP_PKTINFOto get the PHC, but that doesn't work with bridges and other virtual interfaces. Another possibility would be an option to modify the behavior ofIP_PKTINFOto save the index of the real interface. I'm not sure how would that compare in difficulty to extendingSCM_TIMESTAMPINGwith PHC index.--
Miroslav Lichvar
Quoted 15 lines · Miroslav Lichvar
Quoted 7 lines · sdncurious
Quoted 6 lines · Miroslav Lichvar
6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Quoted 6 lines
Quoted 3 lines
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?What is the use case for this. even if the delay though the PHY's how would that be compensated ?
Quoted 8 lines
The idea was that applications like NTP servers and clients wouldn't have to care about interfaces and how they map together with addresses to PHCs over time. Currently, I use the interface index from
IP_PKTINFOto get the PHC, but that doesn't work with bridges and other virtual interfaces. Another possibility would be an option to modify the behavior ofIP_PKTINFOto save the index of the real interface. I'm not sure how would that compare in difficulty to extendingSCM_TIMESTAMPINGwith PHC index.
Why not just return the digest that is in the message ?
Though I am not sure if the least 32 bits will result in too many collisions.
RMS
Quoted 3 lines
--
Miroslav Lichvar
A message that is not in the archive (sent off-list or not yet received): CAHoNx58u=Fze4e5V2Wb_LiBhka1Mzny3zOVNfvuzjnmQ4wBO=Q@mail.gmail.com
[Resend without rich text]
Quoted 4 lines · sdncurious
If the NTP has access to the physical layer, then the timestamps are
associated with the beginning of the symbol after the start of frame. Otherwise, implementations should attempt to associate the timestamp to the earliest accessible point in the frame.
The spec is unfortunately a bit ambiguous and probably should be clarified.
NTP is sensitive to transmission asymmetry. While using the SFD is appropriate for transmit timestamps, it is not appropriate for receive timestamps. A simple reason for this is port speed mismatch. Consider a 1Gb entity communicating with a 100Mb entity on a local switch: leaving aside internal switch delays, if SFD timestamping is used for both transmit and receive, then there is a baked in asymmetry of 6768ns between the forward and reverse paths; if SFD is used for transmit, and FCS end is used for receive, there is no asymmetry.
There is a good explanation of this written by David Mills (NTP's author) here: https://www.eecis.udel.edu/~mills/stamp.html#require
Denny
Quoted 11 lines · Miroslav Lichvar
1) new
rx_filterfor NTPSome NICs can't timestamp all received packets and are currently unusable for NTP with HW timestamping. The new filter would allow NTP support in new NICs and adding support to existing NICs with
firmware/driverupdates. The filter would apply to IPv4 and IPv6 UDP packets received from or sent to the port number 123.
I think this is a good idea. Even if the hardware doesn’t support it, the filtering could be done in the kernel. Save a huge number of context switches.
Quoted 9 lines
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available. Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
Highly agreed. The current interface pretty much forces a socket per physical interface, which should not be necessary.
Quoted 13 lines
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
The existing
SOF_TIMESTAMPING_RX_HARDWAREflag could be aliased toSOF_TIMESTAMPING_RX_HARDWARE_PREAMBLEand the new flag could beSOF_TIMESTAMPING_RX_HARDWARE_TRAILER.
Quoted 6 lines
PTP has a similar problem with SW RX timestamps, which are closer to the trailer timestamps rather than preamble timestamps. A new
SOF_TIMESTAMPING_RX_SOFTWARE_PREAMBLEflag could be added for PTP implementations to get transposed timestamps in order to improve accuracy.
Also highly agreed.
Denny
[Resend as plain text]
Quoted 23 lines · Miroslav Lichvar
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
The existing
SOF_TIMESTAMPING_RX_HARDWAREflag could be aliased toSOF_TIMESTAMPING_RX_HARDWARE_PREAMBLEand the new flag could beSOF_TIMESTAMPING_RX_HARDWARE_TRAILER.PTP has a similar problem with SW RX timestamps, which are closer to the trailer timestamps rather than preamble timestamps. A new
SOF_TIMESTAMPING_RX_SOFTWARE_PREAMBLEflag could be added for PTP implementations to get transposed timestamps in order to improve accuracy.6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestamps
Quoted 7 lines
With bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?
Miroslav, if #5 were implemented, would #6 still needed?
Denny
Quoted 10 lines · Denny Page
Quoted 10 lines · Miroslav Lichvar
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestamps6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Quoted 5 lines
Quoted 3 lines
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?Miroslav, if #5 were implemented, would #6 still needed?
Yes. With #5 we wouldn't have to know the link speed of the interface and guess the length of received packets (or use raw sockets), but we would still not know which clock has actually timestamped the packet.
Signature · 1 line
Miroslav Lichvar
Quoted 33 lines · Denny Page
[Resend as plain text]
Quoted 27 lines · Miroslav Lichvar
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
The existing
SOF_TIMESTAMPING_RX_HARDWAREflag could be aliased toSOF_TIMESTAMPING_RX_HARDWARE_PREAMBLEand the new flag could beSOF_TIMESTAMPING_RX_HARDWARE_TRAILER.PTP has a similar problem with SW RX timestamps, which are closer to the trailer timestamps rather than preamble timestamps. A new
SOF_TIMESTAMPING_RX_SOFTWARE_PREAMBLEflag could be added for PTP implementations to get transposed timestamps in order to improve accuracy.6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Quoted 8 lines
Quoted 3 lines
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?Miroslav, if #5 were implemented, would #6 still needed?
Denny
Miroslav, please ignore this. Of course you still need the index in order to get the PHC offset. My bad.
Denny
Quoted 1 line · Miroslav Lichvar
1) new
rx_filterfor NTP
This is an easy one. No objections here.
Quoted 2 lines
Should be the current drivers of HW that can timestamp all packets updated to fall back to
HWTSTAMP_FILTER_ALL?
Yes, and the phyter, the only driver that can support this directly, would need some work.
Quoted 7 lines
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet framesParsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.
This doesn't seem justified to me. From the application POV, it is easier to hash the transmitted frames than to parse loop backed packets.
Quoted 5 lines
3) target address in
msg_nameof messages from the error queueWith 2) and unconnected sockets, there needs to be a way to get the address to which the packet was sent. Is it ok to always fill
msg_name, or does it need to be a new option?
Again, a hash table cures this.
Quoted 6 lines
4) allow sockets to use both SW and HW TX timestamping at the same time
When using a socket which is not bound to a specific interface, it would be nice to get transmit SW timestamps when HW timestamps are missing. I suspect it's difficult to predict if a HW timestamp will be available.
Right.
Quoted 3 lines
Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
I like this idea better.
However, I doubt the utility of this. If you provide SW time stamps always and TX mostly, but not always, this forces the application to keep two sets of filtered data or two servos, one designed for SW and one for HW accuracy.
Quoted 8 lines
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
Impossible, because the link speed may change between the time when the MAC receives the data the kernel gets around to calculating the time stamp.
Quoted 5 lines
6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Again, this only makes writing the application harder, as it would be forced to sort packets by PHC index. It is much easier to open multiple sockets, each bound to one physical interface.
Quoted 3 lines
I'm not sure what would be the best place to put it. I guess the second timespec in
scm_timestampingcould be reused for this, but that sounds like a gross hack. Do we need to define a new struct?
Yes, I think so.
Thanks,
Richard
Quoted 4 lines · Richard Cochran
Quoted 3 lines · Miroslav Lichvar
2) new
SO_TIMESTAMPINGoption to receive from the error queue only user data as was passed tosendmsg()instead of Ethernet frames
Quoted 8 lines
Quoted 4 lines
Parsing Ethernet and IP headers (especially IPv6 options) is not fun and
SOF_TIMESTAMPING_OPT_IDis not always practical, e.g. in applications which process messages from the error queue asynchronously and don't bind/connect their sockets.This doesn't seem justified to me. From the application POV, it is easier to hash the transmitted frames than to parse loop backed packets.
At least in the case of the NTP implementation I'm working on that would not be easier. I'm not saving transmitted packets. I think that would be a waste of memory, complicating the code, and duplicating work that the kernel is already doing. A public NTP server can handle hundreds of thousands of requests per second, but not all of them may get a SW/HW transmit timestamp. How would I know which will actually get it and how long should I wait for it?
If the packet contains all data needed to process the TX timestamp, it's much easier for me to use data from the kernel queue. If the kernel drops it, it's not a problem. If the kernel loops it back, I have everything I need.
Quoted 7 lines
Quoted 5 lines
3) target address in
msg_nameof messages from the error queueWith 2) and unconnected sockets, there needs to be a way to get the address to which the packet was sent. Is it ok to always fill
msg_name, or does it need to be a new option?Again, a hash table cures this.
It does, but I'm not sure it's always the best option.
Quoted 10 lines
Quoted 3 lines
Maybe it would be acceptable to get from the error queue two messages per transmission if the interface supports both SW and HW timestamping?
I like this idea better.
However, I doubt the utility of this. If you provide SW time stamps always and TX mostly, but not always, this forces the application to keep two sets of filtered data or two servos, one designed for SW and one for HW accuracy.
I think that depends on how is the application designed. In my case each sample is using the best timestamps that were available (any combination of daemon/SW/HW timestamps is possible) and they are all mixed together. The NTP filtering algorithms then drop samples based on their delay, not the timestamping source. Samples using SW timestamps have larger delay than samples using HW timestamp, so they will be dropped unless HW timestamps are missing for long time. In my testing, and from what others have reported, this work well. An occasional missing HW timestamp is not a problem.
Quoted 9 lines
Quoted 8 lines
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
Quoted 3 lines
Impossible, because the link speed may change between the time when the MAC receives the data the kernel gets around to calculating the time stamp.
I think that would be an acceptable limitation. The application certainly couldn't do a better job than the kernel and it won't have to use raw sockets.
Quoted 9 lines
Quoted 5 lines
6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Again, this only makes writing the application harder, as it would be forced to sort packets by PHC index. It is much easier to open multiple sockets, each bound to one physical interface.
With multiple sockets I'd have to know which packet belongs to which socket and track routing changes. I'm not sure if that's even possible with bonding. One socket for everything seems much easier to me. I don't care about interfaces, I just need to know which clock timestamped the packet.
Signature · 1 line
Miroslav Lichvar
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
Quoted 8 lines
Quoted 3 lines
Impossible, because the link speed may change between the time when the MAC receives the data the kernel gets around to calculating the time stamp.
I think that would be an acceptable limitation. The application certainly couldn't do a better job than the kernel and it won't have to use raw sockets.
As you are using HW that supports NTP time stamping won't it by default time stamp the receiving packet correctly at the CRC ? Or if someone came out with such a HW than what ?
I am still at a loss as to why transpose is required in case of HW time stamping. If STF is used for both Tx and Rx time stamping the timing is absolutely correct. Any delay in the PHY is nothing more than usual kernel processing delay, which NTP should be able to deal with when trying to calculate the round trip time, unless there is an issue with the algorithm.
The application can even calculate the complete delay in kernel processing if we provide another time stamp when the packet is read by the application. That seems to provide more accuracy and seems like a better idea.
RMS.
Quoted 6 lines · sdncurious
As you are using HW that supports NTP time stamping won't it by default time stamp the receiving packet correctly at the CRC ? Or if someone came out with such a HW than what ?
As discussed in private email, all hardware operates at the end of the SFD, and in makes sense for the hardware to always do so regardless of what protocol is passing through.
Denny
Quoted 6 lines · sdncurious
I am still at a loss as to why transpose is required in case of HW time stamping. If STF is used for both Tx and Rx time stamping the timing is absolutely correct.
Perhaps this will help. The specific transposition is:
transposed_timestamp_ns = timestamp_ns + (frame_len_bits * 1000000000) / (interface_speed * 1000000)
The transposition is applied to received timestamps only.
Denny
Quoted 10 lines · Denny Page
Quoted 7 lines · sdncurious
I am still at a loss as to why transpose is required in case of HW time stamping. If STF is used for both Tx and Rx time stamping the timing is absolutely correct.
Quoted 5 lines
Perhaps this will help. The specific transposition is:
transposed_timestamp_ns=timestamp_ns+ (frame_len_bits* 1000000000) / (interface_speed* 1000000)The transposition is applied to received timestamps only.
Before anyone else asks, yes, I know this can be reduced. :)
Quoted 11 lines · Miroslav Lichvar
Quoted 10 lines · Richard Cochran
Quoted 8 lines · Miroslav Lichvar
5) new
SO_TIMESTAMPINGoptions to get transposed RX timestampsPTP uses preamble RX timestamps, but NTP works with trailer RX timestamps. This means NTP implementations currently need to transpose HW RX timestamps. The calculation requires the link speed and the length of the packet at layer 2. It seems this can be reliably done only using raw sockets. It would be very nice if the kernel could tranpose the timestamps automatically.
Quoted 7 lines
Quoted 3 lines
Impossible, because the link speed may change between the time when the MAC receives the data the kernel gets around to calculating the time stamp.
I think that would be an acceptable limitation. The application certainly couldn't do a better job than the kernel and it won't have to use raw sockets.
After becoming a bit more familiar with the code I don't think this is a good idea anymore :). I suspect there would be a noticeable performance impact if each timestamped packet could trigger reading of the current link speed. If the value had to be cached it would make more sense to do it in the application.
With no option to get transposed timestamps the point 6 can be scratched too.
A better approach might be a control message that would provide the original interface index together with the length of the packet, so the application could transpose the HW timestamp and map the HW interface to the PHC.
The two values could be saved in the skb_shared_info structure. Now my question is if they could be useful also for other things than timestamping and if it should be a new socket option which would work on any socket independently from timestamping, or if it should rather be a new flag for the SO_TIMESTAMPING option. If the latter, would it make sense to put them in the skb_shared_hwtstamps structure and modify all drivers to set the values when a HW timestamp is captured instead of adding more code to __netif_receive_skb_core() or similar?
What do you think?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
6) new
SO_TIMESTAMPINGoption to get PHC index with HW timestampsWith bridges, bonding and other things it's difficult to determine which PHC timestamped the packet. It would be very useful if the PHC index was provided with each HW timestamp.
Signature · 1 line
Miroslav Lichvar
[Resend as plain text for netdev]
Quoted 7 lines · Miroslav Lichvar
After becoming a bit more familiar with the code I don't think this is a good idea anymore :). I suspect there would be a noticeable performance impact if each timestamped packet could trigger reading of the current link speed. If the value had to be cached it would make more sense to do it in the application.
I am very surprised at this. The application caching approach requires the application retrieve the value via a system call. The system call overhead is huge in comparison to everything else. More importantly, the application cached value may be wrong. If the application takes a sample every 5 seconds, there are 5 seconds of timestamps that can be wildly wrong.
At the driver level, if the speed check is done on packet receive, retrieving the link speed is a single register read which is a small overhead compared with processing the timestamp. The alternative approach of caching still makes more sense in the driver rather than the application. The driver receives an interrupt when negotiation happens, and It’s trivial to cache the value at that point. And a cached value by the driver will always be correct. Implementing it in the driver also allows for hardware to provide the functionality where available. Yes, there is only one chip that provides this currently, but if there is sufficient demand others will appear. There is no way to take advantage of this functionality unless this is handled by the driver.
I think it makes a lot of sense to leave this to the driver developer.
Quoted 4 lines · Miroslav Lichvar
A better approach might be a control message that would provide the original interface index together with the length of the packet, so the application could transpose the HW timestamp and map the HW interface to the PHC.
This sounds better than trying to auto-magically transpose and correct for link speed.
BTW, isn't there already a control message for "original interface index"?
Quoted 3 lines
The two values could be saved in the
skb_shared_infostructure. Now my question is if they could be useful also for other things than timestamping
such as?
Quoted 6 lines
and if it should be a new socket option which would work on any socket independently from timestamping, or if it should rather be a new flag for the
SO_TIMESTAMPINGoption. If the latter, would it make sense to put them in theskb_shared_hwtstampsstructure and modify all drivers to set the values when a HW timestamp is captured instead of adding more code to__netif_receive_skb_core()or similar?
This information is solely for a highly specialized NTP application. No normal program would ever need this, AFAICT. So, if possible, getting the original frame length should be done in a way that doesn't affect users that don't need it.
Thanks,
Richard
Quoted 11 lines · Richard Cochran
Quoted 4 lines · Miroslav Lichvar
A better approach might be a control message that would provide the original interface index together with the length of the packet, so the application could transpose the HW timestamp and map the HW interface to the PHC.
This sounds better than trying to auto-magically transpose and correct for link speed.
BTW, isn't there already a control message for "original interface index"?
There is the PACKET_ORIGDEV option, but it works only with packet sockets, and it doesn't look to me like it could be easily turned into a SO_ORIGDEV option. If there was such an option and also a SO_ORIGLEN option, I think that would work nicely for me.
Quoted 5 lines
Quoted 3 lines
The two values could be saved in the
skb_shared_infostructure. Now my question is if they could be useful also for other things than timestampingsuch as?
I'm not sure. What people do with PACKET_ORIGDEV and would it make sense with other sockets? Googling "PACKET_ORIGDEV" shows implementations of some low-level protocols.
Quoted 11 lines
Quoted 6 lines
and if it should be a new socket option which would work on any socket independently from timestamping, or if it should rather be a new flag for the
SO_TIMESTAMPINGoption. If the latter, would it make sense to put them in theskb_shared_hwtstampsstructure and modify all drivers to set the values when a HW timestamp is captured instead of adding more code to__netif_receive_skb_core()or similar?This information is solely for a highly specialized NTP application. No normal program would ever need this, AFAICT. So, if possible, getting the original frame length should be done in a way that doesn't affect users that don't need it.
Ok. I'll put the two fields to skb_shared_hwtstamps, taking the place of the old syststamp field, and try to avoid adding any code to paths not specific to timestamping.
Thanks,
Signature · 1 line
Miroslav Lichvar
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Denny Page
Quoted 7 lines · Miroslav Lichvar
After becoming a bit more familiar with the code I don't think this is a good idea anymore :). I suspect there would be a noticeable performance impact if each timestamped packet could trigger reading of the current link speed. If the value had to be cached it would make more sense to do it in the application.
I am very surprised at this. The application caching approach requires the application retrieve the value via a system call. The system call overhead is huge in comparison to everything else. More importantly, the application cached value may be wrong. If the application takes a sample every 5 seconds, there are 5 seconds of timestamps that can be wildly wrong.
I'm just trying to be practical and minimize the performance impact and the amount of code that needs to be written, reviewed and maintained.
How common is to have link speed changing in normal operation on LAN?
There are other problems with changing link speed. It does not affect only the transposition.
- If the change happens during a measurement, anywhere on the path between the server and client, the measured offset will have an error due to the asymmetry in the network delay. An NTP measurement in the basic mode is short (it's just the rount-trip time), so it's not very likely to be hit by a link speed change, but in the interleaved mode the probability is exactly the opposite.
- Even if the measurement is not hit and the measured offset is accurate, the change in the measured delay may confuse the NTP client (e.g. temporarily disrupt its sample filtering).
- The TX/RX compensation values depend on the link speed. If the application doesn't reliably know link speed for each packet, the compensation cannot be reliable either.
It seems to me for best timekeeping with NTP it is necessary to make sure the link speed in the network is constant, even if the transposition was always accurate.
Signature · 1 line
Miroslav Lichvar
Quoted 12 lines · Miroslav Lichvar
Quoted 8 lines · Denny Page
Quoted 7 lines · Miroslav Lichvar
After becoming a bit more familiar with the code I don't think this is a good idea anymore :). I suspect there would be a noticeable performance impact if each timestamped packet could trigger reading of the current link speed. If the value had to be cached it would make more sense to do it in the application.
Quoted 7 lines
Quoted 1 line
I am very surprised at this. The application caching approach requires the application retrieve the value via a system call. The system call overhead is huge in comparison to everything else. More importantly, the application cached value may be wrong. If the application takes a sample every 5 seconds, there are 5 seconds of timestamps that can be wildly wrong.
I'm just trying to be practical and minimize the performance impact and the amount of code that needs to be written, reviewed and maintained.
How common is to have link speed changing in normal operation on LAN?
In my case, it’s currently every few minutes because I’m doing hw timestamp testing. :)
But this does speak to my point. If it’s cached by the application, the application has to check it regularly to minimize the possibility of bad timestamps. If the link speed doesn’t change, every call by the application is wasted overhead. If it’s cached by the driver, there is no waste, and the stamps are always correct.
I should have remembered this yesterday... I went and looked at my favorite driver, Intel's igb. Not only is the igb driver already caching link speed, it is also performing timestamp correction based on that link speed. It appears that all Intel drivers are caching link speed. I looked at a few other popular manufacturers, and it appears that caching link speed is common. The only one I quickly found that didn’t cache was realtek.
I believe that timestamp correction, whether it be speed based latency, header -> trailer, or whatever else might be needed later down the line, are properly done in the driver. It’s a lot for the application to try and figure out if it should or should not be doing corrections and what correction to apply. The driver knows.
Quoted 48 lines
-----Original Message-----
From: Denny Page [mailto:[email protected]]
Sent: Friday, March 24, 2017 10:18 AM
To: Miroslav Lichvar <[email protected]>
Cc: Richard Cochran <[email protected]>; [email protected]; Jiri Benc <[email protected]>; Keller, Jacob E <[email protected]>; Willem de Bruijn <[email protected]>
Subject: Re: Extending socket timestamping API for NTPQuoted 5 lines · Miroslav Lichvar
Quoted 2 lines · Denny Page
Quoted 2 lines · Miroslav Lichvar
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
After becoming a bit more familiar with the code I don't think this is a good idea anymore :). I suspect there would be a noticeable performance impact if each timestamped packet could trigger reading of the current link speed. If the value had to be cached it would make more sense to do it in the application.
I am very surprised at this. The application caching approach requires the
application retrieve the value via a system call. The system call overhead is huge in comparison to everything else. More importantly, the application cached value may be wrong. If the application takes a sample every 5 seconds, there are 5 seconds of timestamps that can be wildly wrong.
Quoted 6 lines
I'm just trying to be practical and minimize the performance impact and the amount of code that needs to be written, reviewed and maintained.
How common is to have link speed changing in normal operation on LAN?
In my case, it’s currently every few minutes because I’m doing hw timestamp testing. :)
But this does speak to my point. If it’s cached by the application, the application has to check it regularly to minimize the possibility of bad timestamps. If the link speed doesn’t change, every call by the application is wasted overhead. If it’s cached by the driver, there is no waste, and the stamps are always correct.
I should have remembered this yesterday... I went and looked at my favorite driver, Intel's igb. Not only is the igb driver already caching link speed, it is also performing timestamp correction based on that link speed. It appears that all Intel drivers are caching link speed. I looked at a few other popular manufacturers, and it appears that caching link speed is common. The only one I quickly found that didn’t cache was realtek.
Quoted 4 lines
I believe that timestamp correction, whether it be speed based latency, header -
Quoted 1 line
trailer, or whatever else might be needed later down the line, are properly
done in the driver. It’s a lot for the application to try and figure out if it should or should not be doing corrections and what correction to apply. The driver knows.
I also believe the right place for these corrections is in the driver.
Thanks,
Jake
Quoted 6 lines · Denny Page
Quoted 2 lines · Miroslav Lichvar
How common is to have link speed changing in normal operation on LAN?
In my case, it’s currently every few minutes because I’m doing hw timestamp testing. :)
But this does speak to my point. If it’s cached by the application, the application has to check it regularly to minimize the possibility of bad timestamps. If the link speed doesn’t change, every call by the application is wasted overhead. If it’s cached by the driver, there is no waste, and the stamps are always correct.
At least on the HW I'm testing, reading the link speed from user space doesn't take much. It's about 10-15x faster than reading the PHC for instance, which must be done periodically in any case.
Quoted 1 line
I should have remembered this yesterday... I went and looked at my favorite driver, Intel's igb. Not only is the igb driver already caching link speed, it is also performing timestamp correction based on that link speed.
Isn't the i210 the only NIC for which the correction is actually implemented? Will this ever be done for all HW with timestamping support, so that the applications wouldn't have to care about link speed?
Quoted 1 line
I believe that timestamp correction, whether it be speed based latency, header -> trailer, or whatever else might be needed later down the line, are properly done in the driver. It’s a lot for the application to try and figure out if it should or should not be doing corrections and what correction to apply. The driver knows.
I agree, but I'm not sure how feasible that is.
Signature · 1 line
Miroslav Lichvar
Quoted 5 lines · Miroslav Lichvar
Quoted 1 line · Denny Page
I should have remembered this yesterday... I went and looked at my favorite driver, Intel's igb. Not only is the igb driver already caching link speed, it is also performing timestamp correction based on that link speed.
Isn't the i210 the only NIC for which the correction is actually implemented?
Yes.
Quoted 3 lines
Will this ever be done for all HW with timestamping support, so that the applications wouldn't have to care about link speed?
No.
At the end of the day, the correction in the igb driver is useless and even harmful. Why? Because if the app cares about this level of accuracy, it is going to have to implement special logic anyhow, and having a special case for the igb is even more work for the app.
In addition, if you look into the igb data sheet, you will find a range of correction values, with little indication of how they measured the latency and what the ranges depend on. In my experiments, I have seen the igb consistently land on the extreme of one of the ranges (who knows why), but the driver corrects using the average, forcing me then to correct the remaining offset by hand.
Quoted 3 lines
Quoted 1 line
I believe that timestamp correction, whether it be speed based latency, header -> trailer, or whatever else might be needed later down the line, are properly done in the driver. It’s a lot for the application to try and figure out if it should or should not be doing corrections and what correction to apply. The driver knows.
I agree, but I'm not sure how feasible that is.
+1
Thanks,
Richard
Quoted 7 lines · Richard Cochran
At the end of the day, the correction in the igb driver is useless and even harmful. Why? Because if the app cares about this level of accuracy, it is going to have to implement special logic anyhow, and having a special case for the igb is even more work for the app.
If you are doing correction in the application, _every_ driver is a special case. The a driver making average (known) correction is no more special than any other.
Quoted 3 lines
In addition, if you look into the igb data sheet, you will find a range of correction values, with little indication of how they measured the latency and what the ranges depend on.
You are looking at the 2.2 datasheet I expect. The values for 10Mb and 1Gb have been removed from subsequent datasheets, however they have added a bit more detail as to how the values are measured and what the values.
Quoted 4 lines
In my
experiments, I have seen the igb consistently land on the extreme of one of the ranges (who knows why), but the driver corrects using the average, forcing me then to correct the remaining offset by hand.
I agree that the values in the igb driver are incorrect. They were middle of the range values from the old tables. At least for 100Mb, Intel seems to know that the original table was incorrect. I’ve done extensive measurements of the i210 and i211 at both 100Mb and 1Gb. The “external link partner” numbers Intel currently publishes for the 100Mb appear accurate. I’m still finalizing the values for 1Gb, but one thing I will note is that the values for master mode and slave mode are quite different. FWIW, master/slave mode correction is also something that can only be corrected in the driver :)
I am curious to know any data you developed in your experiments and how you did the measurements. Please email me directly if you are willing to share.
Thanks,
Denny
Quoted 6 lines · Denny Page
I agree that the values in the igb driver are incorrect. They were middle of the range values from the old tables. At least for 100Mb, Intel seems to know that the original table was incorrect. I’ve done extensive measurements of the i210 and i211 at both 100Mb and 1Gb. The “external link partner” numbers Intel currently publishes for the 100Mb appear accurate.
Well, after reading this, I am more convinced than ever that doing the correction in user space is the right way. If the one and only vendor who publishes numbers can't even get them straight, how on earth will we ever get the drivers right?
Quoted 4 lines
I’m still finalizing the values for 1Gb, but one thing I will note is that the values for master mode and slave mode are quite different. FWIW, master/slave mode correction is also something that can only be corrected in the driver :)
Actually, adding ethtool support for SyncE (and consequently Gigabit Ethernet slave/master status) is something we have discussed in the past. I would support expanding the interface to accommodate this...
Quoted 3 lines
I am curious to know any data you developed in your experiments and how you did the measurements. Please email me directly if you are willing to share.
I didn't do anything super methodical, and I didn't keep notes, but I had a phyter (whose delays were published by TI and independently confirmed in a ISPCS paper by Christian Riesch) and an i210 with a 100 MBit link and with a PPS between them. The phyter's numbers are correct to within a nanosecond, and I saw that the i210 was repeatedly landing at the published extreme of the range. I don't remember which extreme, and I didn't repeat more than a few times, however.
Thanks,
Richard
Quoted 12 lines · Richard Cochran
Quoted 6 lines · Denny Page
I agree that the values in the igb driver are incorrect. They were middle of the range values from the old tables. At least for 100Mb, Intel seems to know that the original table was incorrect. I’ve done extensive measurements of the i210 and i211 at both 100Mb and 1Gb. The “external link partner” numbers Intel currently publishes for the 100Mb appear accurate.
Quoted 4 lines
Well, after reading this, I am more convinced than ever that doing the correction in user space is the right way. If the one and only vendor who publishes numbers can't even get them straight, how on earth will we ever get the drivers right?
I think that on average, the Vendor’s numbers are likely to be more accurate than anyone else’s. The concept that independent software implementations are going to somehow obtain and maintain better numbers is too much of a stretch.
FWIW, My testing indicates that the 100Mb numbers that Intel currently publishes are quite accurate. I don’t believe that Intel did the driver corrections btw, if memory serves these values were lifted from the Mac.
Denny
Quoted 4 lines · Denny Page
I think that on average, the Vendor’s numbers are likely to be more accurate than anyone else’s. The concept that independent software implementations are going to somehow obtain and maintain better numbers is too much of a stretch.
But you just said that Intel's first published numbers were wrong. If the vendors would have published accurate information, then you would not have to have made your own measurements, and the drivers could simply use the correct values.
Sadly, this will never happen. The vendor's track record is 100% fail. The apps will always need to implement their own, truly correct values. Having "almost correct" values hard coded into the drivers only makes things worse.
Quoted 4 lines
FWIW, My testing indicates that the 100Mb numbers that Intel currently publishes are quite accurate. I don’t believe that Intel did the driver corrections btw, if memory serves these values were lifted from the Mac.
Huh? Mac? -ENOPARSE.
Thanks,
Richard
Quoted 14 lines · Richard Cochran
Quoted 4 lines · Denny Page
I think that on average, the Vendor’s numbers are likely to be more accurate than anyone else’s. The concept that independent software implementations are going to somehow obtain and maintain better numbers is too much of a stretch.
But you just said that Intel's first published numbers were wrong. If the vendors would have published accurate information, then you would not have to have made your own measurements, and the drivers could simply use the correct values.
Quoted 4 lines
Sadly, this will never happen. The vendor's track record is 100% fail. The apps will always need to implement their own, truly correct values. Having "almost correct" values hard coded into the drivers only makes things worse.
Yes, Intel’s original numbers were wrong. But that doesn’t mean that other’s people’s numbers are going to be particularly better. Even Intel’s original numbers were far better than most will be able to achieve.
But let’s bring this back to the driver. If someone conducts tests and believes that they have better numbers than currently used in the driver, let them come forward with their information and propose a kernel patch. No harm in that at all. And much easier than brining a patch for dozens of applications.
Denny
[Resend in plain text for vger]
Quoted 11 lines · Richard Cochran
Quoted 6 lines · Denny Page
I agree that the values in the igb driver are incorrect. They were middle of the range values from the old tables. At least for 100Mb, Intel seems to know that the original table was incorrect. I’ve done extensive measurements of the i210 and i211 at both 100Mb and 1Gb. The “external link partner” numbers Intel currently publishes for the 100Mb appear accurate.
Quoted 4 lines
Well, after reading this, I am more convinced than ever that doing the correction in user space is the right way. If the one and only vendor who publishes numbers can't even get them straight, how on earth will we ever get the drivers right?
I think that on average, the Vendor’s numbers are likely to be more accurate than anyone else’s. The concept that independent software implementations are going to somehow obtain and maintain better numbers is too much of a stretch.
FWIW, My testing indicates that the 100Mb numbers that Intel currently publishes are quite accurate. I don’t believe that Intel did the driver corrections btw, if memory serves these values were lifted from the Mac.
Denny
[Resend in plain text for vger]
Quoted 9 lines · Richard Cochran
I didn't do anything super methodical, and I didn't keep notes, but I had a phyter (whose delays were published by TI and independently confirmed in a ISPCS paper by Christian Riesch) and an i210 with a 100 MBit link and with a PPS between them. The phyter's numbers are correct to within a nanosecond, and I saw that the i210 was repeatedly landing at the published extreme of the range. I don't remember which extreme, and I didn't repeat more than a few times, however.
Do you still have the resulting correction values from this?
Thanks,
Denny
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Denny Page
Do you still have the resulting correction values from this?
No, I don't, but next time I drag out the phyter I will take another look and let you know.
Thanks,
Richard
Quoted 6 lines · Denny Page
I am very surprised at this. The application caching approach requires the application retrieve the value via a system call. The system call overhead is huge in comparison to everything else. More importantly, the application cached value may be wrong. If the application takes a sample every 5 seconds, there are 5 seconds of timestamps that can be wildly wrong.
You can add a netlink event that is sent on speed change. No need for polling then and the wrong timestamp window will be very tiny. (It can't be zero, even with per-packet data.)
ethtool needs to be converted to netlink, anyway.
Jiri