bonding: add software tx timestamping support
Subsystems (3)
PATCH v4 1/1 bonding: add software tx timestamping support
Currently, bonding only obtain the timestamp (ts) information of the active slave, which is available only for modes 1, 5, and 6. For other modes, bonding only has software rx timestamping support.
However, some users who use modes such as LACP also want tx timestamp support. To address this issue, let's check the ts information of each slave. If all slaves support tx timestamping, we can enable tx timestamping support for the bond.
---
v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.
v3: remove dev_hold/dev_put. remove the '\' for line continuation.
v2: check each slave's ts info to make sure bond support sw tx
timestamping
--- drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
drivers/net/bonding/bond_main.c +32 -0
@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, struct ethtool_ts_info *info) {struct bonding *bond =netdev_priv (bond_dev); +struct ethtool_ts_info ts_info;const struct ethtool_ops *ops;struct net_device *real_dev; +bool sw_tx_support = false;struct phy_device *phydev; +struct list_head *iter; +struct slave *slave;int ret =0 ; +ASSERT_RTNL (); +rcu_read_lock (); real_dev =bond_option_active_slave_get_rcu (bond);dev_hold (real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info (real_dev, info); goto out; } + }else { +/* Check if all slaves support software tx timestamping */ +rcu_read_lock (); +bond_for_each_slave_rcu (bond, slave, iter) { + ret =-1 ; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + +if (phy_has_tsinfo (phydev)) + ret =phy_ts_info (phydev, &ts_info); +else if (ops->get_ts_info) + ret = ops->get_ts_info (slave->dev, &ts_info); + +if (!ret && (ts_info.so_timestamping &SOF_TIMESTAMPING_TX_SOFTWARE )) { + sw_tx_support = true; +continue ; + } + + sw_tx_support = false; +break ; + } +rcu_read_unlock (); } + ret =0 ; info->so_timestamping =SOF_TIMESTAMPING_RX_SOFTWARE |SOF_TIMESTAMPING_SOFTWARE ; +if (sw_tx_support) + info->so_timestamping |=SOF_TIMESTAMPING_TX_SOFTWARE ; + info->phc_index =-1 ; out:
Signature · 1 line
2.38.1
Quoted 11 lines · Hangbin Liu
Currently, bonding only obtain the timestamp (ts) information of the active slave, which is available only for modes 1, 5, and 6. For other modes, bonding only has software rx timestamping support.
However, some users who use modes such as LACP also want tx timestamp support. To address this issue, let's check the ts information of each slave. If all slaves support tx timestamping, we can enable tx timestamping support for the bond.
Suggested-by Miroslav Lichvar <[email protected]>Signed-off-by Hangbin Liu <[email protected]>
-J
Quoted 72 lines
---
v4: addASSERT_RTNLto make surebond_ethtool_get_ts_info()called via
RTNL. Only check_TX_SOFTWAREfor the slaves.
v3: remove dev_hold/dev_put. remove the '\' for line continuation.
v2: check each slave's ts info to make sure bond support sw tx
timestamping
---drivers/net/bonding/bond_main.c| 32 ++++++++++++++++++++++++++++++++1 file changed, 32 insertions(+)drivers/net/bonding/bond_main.c +32
-0@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, struct ethtool_ts_info *info) {struct bonding *bond =netdev_priv (bond_dev); +struct ethtool_ts_info ts_info;const struct ethtool_ops *ops;struct net_device *real_dev; +bool sw_tx_support = false;struct phy_device *phydev; +struct list_head *iter; +struct slave *slave;int ret =0 ; +ASSERT_RTNL (); +rcu_read_lock (); real_dev =bond_option_active_slave_get_rcu (bond);dev_hold (real_dev);@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info (real_dev, info); goto out; } + }else { +/* Check if all slaves support software tx timestamping */ +rcu_read_lock (); +bond_for_each_slave_rcu (bond, slave, iter) { + ret =-1 ; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + +if (phy_has_tsinfo (phydev)) + ret =phy_ts_info (phydev, &ts_info); +else if (ops->get_ts_info) + ret = ops->get_ts_info (slave->dev, &ts_info); + +if (!ret && (ts_info.so_timestamping &SOF_TIMESTAMPING_TX_SOFTWARE )) { + sw_tx_support = true; +continue ; + } + + sw_tx_support = false; +break ; + } +rcu_read_unlock (); } + ret =0 ; info->so_timestamping =SOF_TIMESTAMPING_RX_SOFTWARE |SOF_TIMESTAMPING_SOFTWARE ; +if (sw_tx_support) + info->so_timestamping |=SOF_TIMESTAMPING_TX_SOFTWARE ; + info->phc_index =-1 ;out:
Signature · 1 line
2.38.1
Jakub Kicinski
maintainerQuoted 15 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
v4: add
ASSERT_RTNLto make surebond_ethtool_get_ts_info()called via
RTNL. Only check_TX_SOFTWAREfor the slaves.drivers/net/bonding/bond_main.c +5
-0+ASSERT_RTNL (); +rcu_read_lock (); real_dev =bond_option_active_slave_get_rcu (bond);dev_hold (real_dev);@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info (real_dev, info); goto out; } + }else { +/* Check if all slaves support software tx timestamping */ +rcu_read_lock ();
drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5728
Quoted 10 lines
drivers/net/bonding/bond_main.c +9
-0+bond_for_each_slave_rcu (bond, slave, iter) { + ret =-1 ; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + +if (phy_has_tsinfo (phydev)) + ret =phy_ts_info (phydev, &ts_info); +else if (ops->get_ts_info) + ret = ops->get_ts_info (slave->dev, &ts_info);
My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use bond_for_each_slave() ?
As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback.
It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
v4: add
ASSERT_RTNLto make surebond_ethtool_get_ts_info()called via
RTNL. Only check_TX_SOFTWAREfor the slaves.drivers/net/bonding/bond_main.c +6
-0+ASSERT_RTNL (); +rcu_read_lock (); real_dev =bond_option_active_slave_get_rcu (bond);dev_hold (real_dev);@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info (real_dev, info); goto out; } + }else { +/* Check if all slaves support software tx timestamping */ +rcu_read_lock (); +bond_for_each_slave_rcu (bond, slave, iter) {
Quoted 13 lines
drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5727
Quoted 8 lines
drivers/net/bonding/bond_main.c +8
-0+ ret =-1 ; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + +if (phy_has_tsinfo (phydev)) + ret =phy_ts_info (phydev, &ts_info); +else if (ops->get_ts_info) + ret = ops->get_ts_info (slave->dev, &ts_info);My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use
bond_for_each_slave()?
Ah, I missed that nuance. And, yes, you're correct,bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL is held.
Quoted 4 lines
As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback.
Agreed.
Quoted 2 lines
It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.
Understood; I should have remembered this, as it's been an issue
arising from the "in the middle" aspect of bonding in the past.
-J
---
-Jay Vosburgh, [email protected]
Quoted 34 lines · Jay Vosburgh
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
v4: add
ASSERT_RTNLto make surebond_ethtool_get_ts_info()called via
RTNL. Only check_TX_SOFTWAREfor the slaves.drivers/net/bonding/bond_main.c +6
-0+ASSERT_RTNL (); +rcu_read_lock (); real_dev =bond_option_active_slave_get_rcu (bond);dev_hold (real_dev);@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info (real_dev, info); goto out; } + }else { +/* Check if all slaves support software tx timestamping */ +rcu_read_lock (); +bond_for_each_slave_rcu (bond, slave, iter) {Quoted 13 lines
drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5727
Quoted 8 lines
drivers/net/bonding/bond_main.c +8
-0+ ret =-1 ; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + +if (phy_has_tsinfo (phydev)) + ret =phy_ts_info (phydev, &ts_info); +else if (ops->get_ts_info) + ret = ops->get_ts_info (slave->dev, &ts_info);My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use
bond_for_each_slave()?
Quoted 3 lines
Ah, I missed that nuance. And, yes, you're correct,
bond_for_each_slave()works with RTNL and we don't need RCU here if RTNL is held.
Hi Jay, Jakub,
I remember why I use bond_for_each_slave_rcu() here now. In commit 9b80ccda233f ("bonding: fix missed rcu protection"), I added the rcu_read_lock() as syzbot reported[1] the following path doesn't hold rtnl lock.
- sock_setsockopt
- sock_set_timestamping
- sock_timestamping_bind_phc
- ethtool_get_phc_vclocks
- __ethtool_get_ts_info
- bond_ethtool_get_ts_info
[1] https://lore.kernel.org/netdev/20220513084819.zrg4ssnw667rhndt@skbuf/T/
Thanks
Hangbin
Jakub Kicinski
maintainerQuoted 10 lines · On Mon, 17 Apr 2023 08:17:39 +0800 Hangbin Liu wrote:
I remember why I use
bond_for_each_slave_rcu()here now. In commit9b80ccda233f("bonding: fix missed rcu protection"), I added thercu_read_lock()as syzbot reported[1] the following path doesn't hold rtnl lock.
-sock_setsockopt
-sock_set_timestamping
-sock_timestamping_bind_phc
-ethtool_get_phc_vclocks
-__ethtool_get_ts_info
-bond_ethtool_get_ts_info
Well spotted, okay :(
Could you respin with this info added to the commit message and an update to the kdoc in include/linux/ethtool.h for @get_ts_info that it may be called with RCU, or rtnl or reference on the device?
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:
v4: add
ASSERT_RTNLto make surebond_ethtool_get_ts_info()called via
RTNL. Only check_TX_SOFTWAREfor the slaves.drivers/net/bonding/bond_main.c +6
-0+ASSERT_RTNL (); +rcu_read_lock (); real_dev =bond_option_active_slave_get_rcu (bond);dev_hold (real_dev);@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info (real_dev, info); goto out; } + }else { +/* Check if all slaves support software tx timestamping */ +rcu_read_lock (); +bond_for_each_slave_rcu (bond, slave, iter) {
Quoted 13 lines
drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5727
Quoted 8 lines
drivers/net/bonding/bond_main.c +8
-0+ ret =-1 ; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + +if (phy_has_tsinfo (phydev)) + ret =phy_ts_info (phydev, &ts_info); +else if (ops->get_ts_info) + ret = ops->get_ts_info (slave->dev, &ts_info);My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use
bond_for_each_slave()?
Sorry, I misunderstood your comment in patchv3. I thought you agreed to use the same use case like rlb_next_rx_slave(). I will post a new fix version.
Quoted 8 lines
As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback.
It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.
Thanks for the explanation.
Regards
Hangbin