net: dsa: sja1105: Fix double delivery of TX timestamps to socket error queue
Subsystems (6)
Vladimir Oltean
maintainerPATCH 1/1 net: dsa: sja1105: Fix double delivery of TX timestamps to socket error queue
On the LS1021A-TSN board, it can be seen that under rare conditions, ptp4l gets unexpected extra data in the event socket error queue.
This is because the DSA master driver (gianfar) has TX timestamping logic along the lines of:
1. In gfar_start_xmit:
do_tstamp = (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) && priv->hwts_tx_en; (...) if (unlikely(do_tstamp)) skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 2. Later in gfar_clean_tx_ring: if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS)) (...) skb_tstamp_tx(skb, &shhwtstamps);
That is to say, between the first and second check, it drops priv->hwts_tx_en (it assumes that it is the only one who can set SKBTX_IN_PROGRESS, disregarding stacked net devices such as DSA switches or PTP-capable PHY drivers). Any such driver (like sja1105 in this case) that would set SKBTX_IN_PROGRESS would trip up this check and gianfar would deliver a garbage TX timestamp for this skb too, which can in turn have unpredictable and surprising effects to user space.
In fact gianfar is by far not the only driver which uses SKBTX_IN_PROGRESS to identify skb's that need special handling. The flag used to have a historical purpose and is now evaluated in the networking stack in a single place: in __skb_tstamp_tx, only on the software timestamping path (hwtstamps == NULL) which is not relevant for us.
So do the wise thing and drop the unneeded assignment. Even though this patch alone will not protect against all classes of Ethernet driver TX timestamping bugs, it will circumvent those related to the incorrect interpretation of this skb tx flag.
--- drivers/net/dsa/sja1105/sja1105_ptp.c | 2 -- 1 file changed, 2 deletions(-)
drivers/net/dsa/sja1105/sja1105_ptp.c +0 -2
@@ -668,8 +668,6 @@ void sja1105_ptp_txtstamp_skb(struct dsa_switch *ds, int slot,u64 ticks, ts;int rc; -skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; -mutex_lock (&ptp_data->lock); rc =sja1105_ptpclkval_read (priv, &ticks,NULL );
Signature · 1 line
2.17.1
Quoted 18 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Vladimir Oltean
Sent: Monday, December 16, 2019 2:34 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vladimir Oltean <[email protected]>
Subject: [PATCH net] net: dsa: sja1105: Fix double delivery of TX timestamps to socket error queueOn the LS1021A-TSN board, it can be seen that under rare conditions, ptp4l gets unexpected extra data in the event socket error queue.
This is because the DSA master driver (gianfar) has TX timestamping logic along the lines of:
1. In
gfar_start_xmit:
Quoted 9 lines
do_tstamp = (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) && priv->hwts_tx_en; (...) if (unlikely(do_tstamp)) skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 2. Later in gfar_clean_tx_ring: if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS)) (...) skb_tstamp_tx(skb, &shhwtstamps);
I'm not sure I fully understand the problem.
Quoted 15 lines
That is to say, between the first and second check, it drops
priv->hwts_tx_en(it assumes that it is the only one who can setSKBTX_IN_PROGRESS, disregarding stacked net devices such as DSA switches or PTP-capable PHY drivers). Any such driver (like sja1105 in this case) that would setSKBTX_IN_PROGRESSwould trip up this check and gianfar would deliver a garbage TX timestamp for this skb too, which can in turn have unpredictable and surprising effects to user space.In fact gianfar is by far not the only driver which uses
SKBTX_IN_PROGRESSto identify skb's that need special handling. The flag used to have a historical purpose and is now evaluated in the networking stack in a single place: in__skb_tstamp_tx, only on the software timestamping path (hwtstamps == NULL) which is not relevant for us.
Quoted 5 lines
So do the wise thing and drop the unneeded assignment. Even though this patch alone will not protect against all classes of Ethernet driver TX timestamping bugs, it will circumvent those related to the incorrect interpretation of this skb tx flag.
I thought the point of SKBTX_IN_PROGRESS was to inform the stack that a timestamp was pending. By not setting it, you no longer do this.
Maybe that has changed since the original implementation? Or am I misunderstanding this patch..?
You're removing the sja1105 assignment, not the one from the gianfar. Hmm
Ok, so the issue is that sja1105_ptp.c was incorrectly setting the flag.
Would it make more sense for gianfar to set SKBTX_IN_PROGRESS, but then use some other indicator internally, so that other callers who set it don't cause the gianfar driver to behave incorrectly? I believe we handle it in the Intel drivers that way by storing the skb. Then we don't check the SKBTX_IN_PROGRESS later.
Thanks,
Jake
Quoted 24 lines
Fixes 47ed985e97f5 ("net: dsa: sja1105: Add logic for TX timestamping")Suggested-by Richard Cochran <[email protected]>Signed-off-by Vladimir Oltean <[email protected]>--- drivers/net/dsa/sja1105/sja1105_ptp.c | 2 -- 1 file changed, 2 deletions(-)drivers/net/dsa/sja1105/sja1105_ptp.c +0
-2@@ -668,8 +668,6 @@ void sja1105_ptp_txtstamp_skb(struct dsa_switch *ds, int slot, u64 ticks, ts;int rc; -skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; -mutex_lock (&ptp_data->lock);rc =
sja1105_ptpclkval_read(priv, &ticks, NULL);
--
2.17.1
Vladimir Oltean
maintainerHi Jake,
Quoted 29 lines · Jacob E
Quoted 19 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Vladimir Oltean
Sent: Monday, December 16, 2019 2:34 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vladimir Oltean <[email protected]>
Subject: [PATCH net] net: dsa: sja1105: Fix double delivery of TX timestamps to socket error queueOn the LS1021A-TSN board, it can be seen that under rare conditions, ptp4l gets unexpected extra data in the event socket error queue.
This is because the DSA master driver (gianfar) has TX timestamping logic along the lines of:
1. In
gfar_start_xmit:Quoted 9 lines
do_tstamp = (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) && priv->hwts_tx_en; (...) if (unlikely(do_tstamp)) skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 2. Later in gfar_clean_tx_ring: if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS)) (...) skb_tstamp_tx(skb, &shhwtstamps);
Quoted 6 lines
I'm not sure I fully understand the problem.
I thought the point of
SKBTX_IN_PROGRESSwas to inform the stack that a timestamp was pending. By not setting it, you no longer do this.Maybe that has changed since the original implementation? Or am I misunderstanding this patch..?
I am not quite sure what the point of SKBTX_IN_PROGRESS is. If you search through the kernel you will find a single occurrence right now that is supposed to do something (I don't understand what exactly) when there is a concurrent sw and hw timestamp.
Quoted 5 lines
You're removing the sja1105 assignment, not the one from the gianfar. Hmm
Ok, so the issue is that
sja1105_ptp.cwas incorrectly setting the flag.Would it make more sense for gianfar to set
SKBTX_IN_PROGRESS, but then use some other indicator internally, so that other callers who set it don't cause the gianfar driver to behave incorrectly? I believe we handle it in the Intel drivers that way by storing the skb. Then we don't check theSKBTX_IN_PROGRESSlater.
Yes, the point is that it should check priv->hwts_tx_en && (skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS). That was my initial fix for the bug, but Richard argued that setting SKBTX_IN_PROGRESS in itself isn't needed.
There are many more drivers that are in principle broken with DSA PTP, since they don't even have the equivalent check for priv->hwts_tx_en.
Quoted 5 lines
Thanks,
Jake
Thanks,
-Vladimir
Quoted 16 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Vladimir Oltean
Sent: Tuesday, December 17, 2019 12:07 PM
To: Keller, Jacob E <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH net] net: dsa: sja1105: Fix double delivery of TX timestamps to socket error queueHi Jake,
Quoted 1 line
I thought the point of
SKBTX_IN_PROGRESSwas to inform the stack that atimestamp was pending. By not setting it, you no longer do this.
Quoted 1 line
Quoted 9 lines
Quoted 1 line
Maybe that has changed since the original implementation? Or am I
misunderstanding this patch..?
Quoted 1 line
I am not quite sure what the point of
SKBTX_IN_PROGRESSis. If you search through the kernel you will find a single occurrence right now that is supposed to do something (I don't understand what exactly) when there is a concurrent sw and hw timestamp.
My understanding was that setting it prevented the stack from generating a SW timestamp if the hardware timestamp was going to be provided. Basically, this is because we would otherwise report the timestamp twice to applications that expect only one timestamp.
There were some patches from Miroslav that enabled optionally allowed the reporting of both SW and HW timestamps at the same time.
Quoted 5 lines
Quoted 5 lines
You're removing the sja1105 assignment, not the one from the gianfar. Hmm
Ok, so the issue is that
sja1105_ptp.cwas incorrectly setting the flag.Would it make more sense for gianfar to set
SKBTX_IN_PROGRESS, but then
Quoted 9 lines
use some other indicator internally, so that other callers who set it don't cause the gianfar driver to behave incorrectly? I believe we handle it in the Intel drivers that way by storing the skb. Then we don't check the
SKBTX_IN_PROGRESSlater.Yes, the point is that it should check
priv->hwts_tx_en&& (skb_shinfo(skb)->tx_flags &SKBTX_IN_PROGRESS). That was my initial fix for the bug, but Richard argued that settingSKBTX_IN_PROGRESSin itself isn't needed.
I'd trust Richard on this one for sure.
Quoted 3 lines
There are many more drivers that are in principle broken with DSA PTP, since they don't even have the equivalent check for
priv->hwts_tx_en.
Right. I'm wondering what the correct fix would be so that we can fix the drivers and hopefully avoid introducing a similar issue in the future.
Thanks,
Jake
Quoted 8 lines
Quoted 5 lines
Thanks,
JakeThanks,
-Vladimir
Richard Cochran
maintainerQuoted 8 lines · Jacob E
Quoted 3 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Vladimir OlteanMy understanding was that setting it prevented the stack from generating a SW timestamp if the hardware timestamp was going to be provided. Basically, this is because we would otherwise report the timestamp twice to applications that expect only one timestamp.
Correct.
Quoted 2 lines
There were some patches from Miroslav that enabled optionally allowed the reporting of both SW and HW timestamps at the same time.
Not quite. If the user sets SOF_TIMESTAMPING_OPT_TX_SWHW, then there will be two packets delivered to the error queue, one SW and one HW.
Quoted 2 lines
Quoted 2 lines
There are many more drivers that are in principle broken with DSA PTP, since they don't even have the equivalent check for
priv->hwts_tx_en.
Please stop saying that. It is not true.
Quoted 3 lines
Right. I'm wondering what the correct fix would be so that we can fix the drivers and hopefully avoid introducing a similar issue in the future.
No fix is needed. MAC drivers must set SKBTX_IN_PROGRESS and call skb_tstamp_tx() to deliver the transmit time stamp. DSA drivers should call skb_complete_tx_timestamp() to deliver the transmit time stamp, and they should *not* set SKBTX_IN_PROGRESS.
Thanks,
Richard
Hi Richard,
Quoted 11 lines · Richard Cochran
Quoted 6 lines · Jacob E
Quoted 6 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Vladimir OlteanThere are many more drivers that are in principle broken with DSA PTP, since they don't even have the equivalent check for
priv->hwts_tx_en.Please stop saying that. It is not true.
Why isn't it true?
You mean to say that this code in cavium/liquidio/lio_main.c will ever work with a PTP-cable DSA switch or PHY?
static netdev_tx_t liquidio_xmit(struct sk_buff *skb, struct net_device *netdev) {
(...)
if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) {
skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
cmdsetup.s.timestamp = 1;
}Or this one in cavium/octeon/octeon_mgmt.c?
re.s.tstamp = ((skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) != 0);
Or this one in mscc/ocelot.c:
/* Check if timestamping is needed */
if (ocelot->ptp && shinfo->tx_flags & SKBTX_HW_TSTAMP) {
info.rew_op = ocelot_port->ptp_cmd;
if (ocelot_port->ptp_cmd == IFH_REW_OP_TWO_STEP_PTP)
info.rew_op |= (ocelot_port->ts_id % 4) << 3;
}Or this one in mellanox/mlx5/core/en_tx.c:
if (unlikely(skb_shinfo(skb)->tx_flags &
SKBTX_HW_TSTAMP)) {
struct skb_shared_hwtstamps hwts = {};
hwts.hwtstamp =
mlx5_timecounter_cyc2time(sq->clock,
get_cqe_ts(cqe));
skb_tstamp_tx(skb, &hwts);
}etc etc.
How will these drivers not transmit a second hw TX timestamp to the stack, if they don't check whether TX timestamping is enabled for their netdev?
Of course, at least that breakage is going to be much more binary and
obvious: PTP simply won't work at all for drivers stacked on top of them until they are fixed.
Quoted 5 lines
No fix is needed. MAC drivers must set
SKBTX_IN_PROGRESSand callskb_tstamp_tx()to deliver the transmit time stamp. DSA drivers should callskb_complete_tx_timestamp()to deliver the transmit time stamp, and they should *not* setSKBTX_IN_PROGRESS.
Who says so, and why? How would it be better than fixing gianfar in this case? How would it be better in avoiding compatibility with the drivers mentioned above?
Does the TI PHYTER driver count?
commit e2e2f51dd0254fa0002bcd1c5fda180348163f09
Author: Stefan Sørensen <[email protected]> Date: Mon Feb 3 15:36:35 2014 +0100
net:phy:dp83640: Declare that TX timestamping possible
Set the SKBTX_IN_PROGRESS bit in tx_flags dp83640_txtstamp when doing tx timestamps as per Documentation/networking/timestamping.txt.
drivers/net/phy/dp83640.c +1 -0
@@ -1281,6 +1281,7 @@ static void dp83640_txtstamp(struct phy_device *phydev, }/* fall through */ case HWTSTAMP_TX_ON : +skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ;skb_queue_tail (&dp83640->tx_queue, skb);schedule_work (&dp83640->ts_work);break ;
I think this rule is somewhat made-up and arbitrary.
Quoted 2 lines
Thanks,
Richard
Thanks,
-Vladimir
Richard Cochran
maintainerQuoted 3 lines · Vladimir Oltean
How will these drivers not transmit a second hw TX timestamp to the stack, if they don't check whether TX timestamping is enabled for their netdev?
Ah, so they are checking SKBTX_HW_TSTAMP on the socket without checking for HWTSTAMP first? Yeah, that won't work with DSA time stamping. But who cares? Most of the real world doesn't have gigabit DSA switches in front of MACs that provide 10 gigabit links or higher.
Quoted 3 lines
Of course, at least that breakage is going to be much more binary and
obvious: PTP simply won't work at all for drivers stacked on top of them until they are fixed.
Right, you can fix individual MAC drivers to work with DSA, one by one. You are free to try to get the maintainers to ack your fixes. (But, here is a friendly hint: don't start out by declaring their drivers "broken").
Quoted 1 line
Does the TI PHYTER driver count?
No. It really doesn't count. It won't work together with MAC time stamping, but this is a known limitation. That part is 100 mbit only, and there are very few implementations. Unfortunately the so_timestamping API did not foresee the possibility of simultaneous MAC and PHY time stamping. Too bad, that's life. I didn't invent the API, and I don't have to defend it, either.
Thanks,
Richard
Richard Cochran
maintainerQuoted 7 lines · Richard Cochran
Quoted 3 lines · Vladimir Oltean
How will these drivers not transmit a second hw TX timestamp to the stack, if they don't check whether TX timestamping is enabled for their netdev?
Ah, so they are checking
SKBTX_HW_TSTAMPon the socket without checking for HWTSTAMP first?
Thinking a bit more about this, MAC drivers should not attempt any time stamping unless that functionality has been enabled at the device level using HWTSTAMP. This is the user space API.
So, if you want to fix those drivers, you can submit patches with the above justification. That is a stronger argument than saying it breaks DSA drivers!
Thanks,
Richard
Vladimir Oltean
maintainerQuoted 14 lines · Richard Cochran
Quoted 7 lines · Richard Cochran
Quoted 3 lines · Vladimir Oltean
How will these drivers not transmit a second hw TX timestamp to the stack, if they don't check whether TX timestamping is enabled for their netdev?
Ah, so they are checking
SKBTX_HW_TSTAMPon the socket without checking for HWTSTAMP first?Thinking a bit more about this, MAC drivers should not attempt any time stamping unless that functionality has been enabled at the device level using HWTSTAMP. This is the user space API.
Quoted 6 lines
So, if you want to fix those drivers, you can submit patches with the above justification. That is a stronger argument than saying it breaks DSA drivers!
Thanks,
Richard
But in the case that I _do_ care about, it _is_ caused by DSA. The gianfar driver is not expecting anybody else to set SKBTX_IN_PROGRESS, which in itself is not illegal, whether or not you argue that it is needed or not.
For the rest of the drivers, sure, they commit even more flagrant API breaches, such as not checking previous calls of the HWTSTAMP ioctl. But the flagrant issues are at least easier to catch (aka PTP is never going to work), so I'm not as concerned about them, as long as user space (e.g. ptp4l) has the necessary checks in place to detect such error conditions.
Thanks.
-Vladimir
Richard Cochran
maintainerQuoted 4 lines · Vladimir Oltean
But in the case that I _do_ care about, it _is_ caused by DSA. The gianfar driver is not expecting anybody else to set
SKBTX_IN_PROGRESS, which in itself is not illegal, whether or not you argue that it is needed or not.
Ok, so I've reviewed the whole SKBTX_IN_PROGRESS thing again, and I'm starting to see it your way. (Or maybe not ;^)
The purpose of SKBTX_IN_PROGRESS is to prevent a SW transmit time stamp from being delivered (unless SOF_TIMESTAMPING_OPT_TX_SWHW is set on the socket).
So DSA drivers (and PHY drivers) should indeed set this flag.
However, most (or all?) MAC drivers do not expect anyone else to have set this flag.
In the case of DSA, the DSA driver has the chance to set the flag first, before the skb is passed to the MAC driver. (PHY drivers don't have first dibs, but let's concentrate on DSA for now.)
So, you could introduce a new rule that MAC drivers must check to see if the flag is already set, and if so leave it alone.
MAC drivers should in any case respect their current SIOCSHWTSTAMP configuration and not attempt time stamping when it is not enabled.
So I'm afraid that, as I've said before, you'll have to patch the MAC drivers one by. It will be lots of work.
But if and when you submit patches for the MAC drivers, please remember that DSA time stamping is the new kid on the block, and yelling that the MAC drivers are broken won't make you any friends.
Thanks,
Richard