Skip to contentSkip to conversation

bonding: add software tx timestamping support

PATCH Superseded Hangbin Liu 3 participants 7 messages Apr 14, 2023 – Apr 17, 2023

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 bonding: add software tx timestamping support

Currently, bonding only obtain the timestamp (ts) information of the active slave, which is available only for modes 1, 5, and 6. For other modes, bonding only has software rx timestamping support.

However, some users who use modes such as LACP also want tx timestamp support. To address this issue, let's check the ts information of each slave. If all slaves support tx timestamping, we can enable tx timestamping support for the bond.

---
v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.
v3: remove dev_hold/dev_put. remove the '\' for line continuation.
v2: check each slave's ts info to make sure bond support sw tx
timestamping

---
 drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
drivers/net/bonding/bond_main.c +32 -0
index 00646aa315c3..9cf49b61f4b3 100644
@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 				    struct ethtool_ts_info *info)
 {
 	struct bonding *bond = netdev_priv(bond_dev);
+	struct ethtool_ts_info ts_info;
 	const struct ethtool_ops *ops;
 	struct net_device *real_dev;
+	bool sw_tx_support = false;
 	struct phy_device *phydev;
+	struct list_head *iter;
+	struct slave *slave;
 	int ret = 0;
 
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();
+		bond_for_each_slave_rcu(bond, slave, iter) {
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);
+
+			if (!ret && (ts_info.so_timestamping & SOF_TIMESTAMPING_TX_SOFTWARE)) {
+				sw_tx_support = true;
+				continue;
+			}
+
+			sw_tx_support = false;
+			break;
+		}
+		rcu_read_unlock();
 	}
 
+	ret = 0;
 	info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE |
 				SOF_TIMESTAMPING_SOFTWARE;
+	if (sw_tx_support)
+		info->so_timestamping |= SOF_TIMESTAMPING_TX_SOFTWARE;
+
 	info->phc_index = -1;
 
 out:
Signature · 1 line
2.38.1
Acked-by on v4 1/1
#1 lore raw
Quoted 11 lines · Hangbin Liu

Currently, bonding only obtain the timestamp (ts) information of the active slave, which is available only for modes 1, 5, and 6. For other modes, bonding only has software rx timestamping support.

However, some users who use modes such as LACP also want tx timestamp support. To address this issue, let's check the ts information of each slave. If all slaves support tx timestamping, we can enable tx timestamping support for the bond.

	-J
Quoted 72 lines

---
v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.
v3: remove dev_hold/dev_put. remove the '\' for line continuation.
v2: check each slave's ts info to make sure bond support sw tx
timestamping
---
drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++

1 file changed, 32 insertions(+)
drivers/net/bonding/bond_main.c +32 -0
index 00646aa315c3..9cf49b61f4b3 100644
@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 				    struct ethtool_ts_info *info)
 {
 	struct bonding *bond = netdev_priv(bond_dev);
+	struct ethtool_ts_info ts_info;
 	const struct ethtool_ops *ops;
 	struct net_device *real_dev;
+	bool sw_tx_support = false;
 	struct phy_device *phydev;
+	struct list_head *iter;
+	struct slave *slave;
 	int ret = 0;
 
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();
+		bond_for_each_slave_rcu(bond, slave, iter) {
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);
+
+			if (!ret && (ts_info.so_timestamping & SOF_TIMESTAMPING_TX_SOFTWARE)) {
+				sw_tx_support = true;
+				continue;
+			}
+
+			sw_tx_support = false;
+			break;
+		}
+		rcu_read_unlock();
 	}
 
+	ret = 0;
 	info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE |
 				SOF_TIMESTAMPING_SOFTWARE;
+	if (sw_tx_support)
+		info->so_timestamping |= SOF_TIMESTAMPING_TX_SOFTWARE;
+
 	info->phc_index = -1;
 

out:

Signature · 1 line
2.38.1
maintainer
#2 lore raw
In reply to Hangbin Liu
Quoted 15 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:

v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.

drivers/net/bonding/bond_main.c +5 -0
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();

drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5728

Quoted 10 lines
drivers/net/bonding/bond_main.c +9 -0
+		bond_for_each_slave_rcu(bond, slave, iter) {
 
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);

My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use bond_for_each_slave() ?

As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback.

It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.

#3 lore raw
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:

v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.

drivers/net/bonding/bond_main.c +6 -0
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();
+		bond_for_each_slave_rcu(bond, slave, iter) {
Quoted 13 lines

drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5727

Quoted 8 lines
drivers/net/bonding/bond_main.c +8 -0
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);

My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use bond_for_each_slave() ?

Ah, I missed that nuance. And, yes, you're correct,
bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL is held.

Quoted 4 lines

As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback.

Agreed.

Quoted 2 lines

It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.

Understood; I should have remembered this, as it's been an issue
arising from the "in the middle" aspect of bonding in the past.

	-J

---
-Jay Vosburgh, [email protected]

#4 lore raw
Quoted 34 lines · Jay Vosburgh
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:

v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.

drivers/net/bonding/bond_main.c +6 -0
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();
+		bond_for_each_slave_rcu(bond, slave, iter) {
Quoted 13 lines

drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5727

Quoted 8 lines
drivers/net/bonding/bond_main.c +8 -0
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);

My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use bond_for_each_slave() ?

Quoted 3 lines

Ah, I missed that nuance. And, yes, you're correct,
bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL is held.

Hi Jay, Jakub,

I remember why I use bond_for_each_slave_rcu() here now. In commit 9b80ccda233f ("bonding: fix missed rcu protection"), I added the rcu_read_lock() as syzbot reported[1] the following path doesn't hold rtnl lock.
- sock_setsockopt
- sock_set_timestamping
- sock_timestamping_bind_phc
- ethtool_get_phc_vclocks
- __ethtool_get_ts_info
- bond_ethtool_get_ts_info

[1] https://lore.kernel.org/netdev/20220513084819.zrg4ssnw667rhndt@skbuf/T/

Thanks
Hangbin

maintainer
#5 lore raw
Quoted 10 lines · On Mon, 17 Apr 2023 08:17:39 +0800 Hangbin Liu wrote:

I remember why I use bond_for_each_slave_rcu() here now. In commit 9b80ccda233f ("bonding: fix missed rcu protection"), I added the rcu_read_lock() as syzbot reported[1] the following path doesn't hold rtnl lock.
- sock_setsockopt
- sock_set_timestamping
- sock_timestamping_bind_phc
- ethtool_get_phc_vclocks
- __ethtool_get_ts_info
- bond_ethtool_get_ts_info

Well spotted, okay :(
Could you respin with this info added to the commit message and an update to the kdoc in include/linux/ethtool.h for @get_ts_info that it may be called with RCU, or rtnl or reference on the device?

#6 lore raw
In reply to Jakub Kicinski
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote:

v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via
RTNL. Only check _TX_SOFTWARE for the slaves.

drivers/net/bonding/bond_main.c +6 -0
+	ASSERT_RTNL();
+
 	rcu_read_lock();
 	real_dev = bond_option_active_slave_get_rcu(bond);
 	dev_hold(real_dev);
@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev,
 			ret = ops->get_ts_info(real_dev, info);
 			goto out;
 		}
+	} else {
+		/* Check if all slaves support software tx timestamping */
+		rcu_read_lock();
+		bond_for_each_slave_rcu(bond, slave, iter) {
Quoted 13 lines

drivers/net/bonding/bond_main.c static int bond_ethtool_get_ts_info(struct net_device *bond_dev, · line 5727

Quoted 8 lines
drivers/net/bonding/bond_main.c +8 -0
+			ret = -1;
+			ops = slave->dev->ethtool_ops;
+			phydev = slave->dev->phydev;
+
+			if (phy_has_tsinfo(phydev))
+				ret = phy_ts_info(phydev, &ts_info);
+			else if (ops->get_ts_info)
+				ret = ops->get_ts_info(slave->dev, &ts_info);

My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use bond_for_each_slave() ?

Sorry, I misunderstood your comment in patchv3. I thought you agreed to use the same use case like rlb_next_rx_slave(). I will post a new fix version.

Quoted 8 lines

As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback.

It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.

Thanks for the explanation.

Regards
Hangbin