Skip to contentSkip to conversation

Revert "net/ncsi: change from ndo_set_mac_address to dev_set_mac_address"

PATCH In review Jakub Kicinski 5 participants 13 messages Jan 8, 2025 – Jan 14, 2025 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH v2 1/1 Revert "net/ncsi: change from ndo_set_mac_address to dev_set_mac_address"

From: Potin Lai <[email protected]>

This reverts commit 790071347a0a1a89e618eedcd51c687ea783aeb3.

We are seeing kernel panic when enabling two NCSI interfaces at same time. It looks like mutex lock is being used in softirq caused the issue.

Kernel panic log:

8021q: adding VLAN 0 to HW filter on device eth0
ftgmac100 1e670000.ethernet eth0: NCSI: Handler for packet type 0x82 returned -19

  BUG: scheduling while atomic: systemd-network/697/0x00000100
  Modules linked in:
  8021q: adding VLAN 0 to HW filter on device eth1
  CPU: 0 PID: 697 Comm: systemd-network Tainted: G        W          6.6.62-8ea1fc6-dirty-cbd80d0-gcbd80d04d13c #1
  Hardware name: Generic DT based system
   unwind_backtrace from show_stack+0x18/0x1c
   show_stack from dump_stack_lvl+0x40/0x4c
   dump_stack_lvl from __schedule_bug+0x5c/0x70
   __schedule_bug from __schedule+0x884/0x968
   __schedule from schedule+0x58/0xa8
   schedule from schedule_preempt_disabled+0x14/0x18
   schedule_preempt_disabled from __mutex_lock.constprop.0+0x350/0x76c
   __mutex_lock.constprop.0 from ncsi_rsp_handler_oem_gma+0x104/0x1a0
   ncsi_rsp_handler_oem_gma from ncsi_rcv_rsp+0x120/0x2cc
   ncsi_rcv_rsp from __netif_receive_skb_one_core+0x60/0x84
   __netif_receive_skb_one_core from netif_receive_skb+0x38/0x148
   netif_receive_skb from ftgmac100_poll+0x358/0x444
   ftgmac100_poll from __napi_poll.constprop.0+0x34/0x1d0
   __napi_poll.constprop.0 from net_rx_action+0x350/0x43c
   net_rx_action from handle_softirqs+0x114/0x32c
   handle_softirqs from irq_exit+0x88/0xb8
   irq_exit from call_with_stack+0x18/0x20
   call_with_stack from __irq_usr+0x78/0xa0
  Exception stack(0xe075dfb0 to 0xe075dff8)
  dfa0:                                     00000000 00000000 00000000 00000020
  dfc0: 00000069 aefde3e0 00000000 00000000 00000000 00000000 00000000 aefde4e4
  dfe0: 01010101 aefddf20 a6b4331c a6b43618 600f0010 ffffffff
Fixes 790071347a0a ("net/ncsi: change from ndo_set_mac_address to dev_set_mac_address")

---
Looks like we're not making any progress on this one, so let's go with the revert for 6.13.

Original posting by Potin Lai:
https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.com

I added the Fixes tag and trimmed the kernel log.

---
 net/ncsi/ncsi-rsp.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
net/ncsi/ncsi-rsp.c +2 -3
index e28be33bdf2c..0cd7b916d3f8 100644
@@ -629,6 +629,7 @@ static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)
 {
 	struct ncsi_dev_priv *ndp = nr->ndp;
 	struct net_device *ndev = ndp->ndev.dev;
+	const struct net_device_ops *ops = ndev->netdev_ops;
 	struct ncsi_rsp_oem_pkt *rsp;
 	struct sockaddr saddr;
 	u32 mac_addr_off = 0;
@@ -655,9 +656,7 @@ static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)
 	/* Set the flag for GMA command which should only be called once */
 	ndp->gma_flag = 1;
 
-	rtnl_lock();
-	ret = dev_set_mac_address(ndev, &saddr, NULL);
-	rtnl_unlock();
+	ret = ops->ndo_set_mac_address(ndev, &saddr);
 	if (ret < 0)
 		netdev_warn(ndev, "NCSI: 'Writing mac address to device failed\n");
 
Signature · 1 line
2.47.1
reviewer
#1 lore raw

Hello,

Quoted 2 lines · Jakub Kicinski

Looks like we're not making any progress on this one, so let's go with the revert for 6.13.

But this does break userspace, the commit was there for a reason.

Potin Lai, have you tried deferring this to a work queue instead of reverting to the code which has always been wrong?

Quoted 2 lines

Original posting by Potin Lai:
https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.com

Unfortunately I wasn't on CC and missed that. Can we fix this proper way please, do we have few more days?

If I get it right dev_set_mac_address() just isn't meant to be ever called from a softirq (and NCSI here is special in getting a MAC address update from a "network" frame so that happens in net_rx_action() context). So postponing the actual processing of this reply looks like the way to go, right?

net/ncsi/ncsi-rsp.c static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)

Quoted 7 lines
net/ncsi/ncsi-rsp.c +1 -3
-	rtnl_lock();
-	ret = dev_set_mac_address(ndev, &saddr, NULL);
-	rtnl_unlock();
+	ret = ops->ndo_set_mac_address(ndev, &saddr);
 
 	if (ret < 0)
 		netdev_warn(ndev, "NCSI: 'Writing mac address to device failed\n");

I expect if this is scheduled on events/x with queue_work() then we'll get what we need.

Signature · 2 lines
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:[email protected]
author maintainer
#2 lore raw
Quoted 5 lines · On Wed, 8 Jan 2025 23:23:23 +0300 Paul Fertser wrote:
Quoted 2 lines

Original posting by Potin Lai:
https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.com

Unfortunately I wasn't on CC and missed that. Can we fix this proper way please, do we have few more days?

Sure thing, we need it in the tree early next week, so probably on the list by Friday so that folks have a chance to review.

Quoted 5 lines

If I get it right dev_set_mac_address() just isn't meant to be ever called from a softirq (and NCSI here is special in getting a MAC address update from a "network" frame so that happens in net_rx_action() context). So postponing the actual processing of this reply looks like the way to go, right?

I think so. Only trickiness I can think of is the ordering of the work vs admin setting the address from user space. Do venture a patch, I don't know enough about NC-SI.. :(

#3 lore raw
In reply to Paul Fertser
Quoted 10 lines · Paul Fertser

Hello,

Quoted 2 lines · Jakub Kicinski

Looks like we're not making any progress on this one, so let's go with the revert for 6.13.

But this does break userspace, the commit was there for a reason.

Potin Lai, have you tried deferring this to a work queue instead of reverting to the code which has always been wrong?

Jakub, thanks for letting know about revert. ndo_set_mac_address do not notify userspace about MAC change and as Paul stated it was always been wrong here.

And we talked about reverts in this thread -
https://lore.kernel.org/all/[email protected]/

Probably, incremental proper fix would be better here?

Common case is 1 NCSI interface for server, we tested on this one and works fine for us and that's the reason why we didn't catch that situation. Nowadays some new servers has more than one NCSI interface. Probably we missed that is softirq context which is obviously not a place for rtnl_lock/unlock. Is there any other solution about except delaying dev_set_mac_address in work queue? Or any suggestions about how to deal with that in a proper way?

Don't have any hw with >2 NCSI interface, probably need help with testing.

Potin, can you help with checking the fix?

Thanks.

author maintainer
#4 lore raw
Quoted 7 lines · On Thu, 09 Jan 2025 00:48:07 +0300 Ivan Mikhaylov wrote:

Common case is 1 NCSI interface for server, we tested on this one and works fine for us and that's the reason why we didn't catch that situation. Nowadays some new servers has more than one NCSI interface. Probably we missed that is softirq context which is obviously not a place for rtnl_lock/unlock. Is there any other solution about except delaying dev_set_mac_address in work queue? Or any suggestions about how to deal with that in a proper way?

Nothing obvious comes to mind, unfortunately.
But the workqueue shouldn't be too bad.

reviewer
#5 lore raw

PATCH 1/1 net/ncsi: fix locking in Get MAC Address handling

In reply to Jakub Kicinski

Obtaining RTNL lock in a response handler is not allowed since it runs in an atomic softirq context. Postpone setting the MAC address by adding a dedicated step to the configuration FSM.

Fixes 790071347a0a ("net/ncsi: change from ndo_set_mac_address to dev_set_mac_address")
Cc Potin Lai <[email protected]>
---
 net/ncsi/internal.h    |  2 ++
 net/ncsi/ncsi-manage.c | 16 ++++++++++++++--
 net/ncsi/ncsi-rsp.c    | 19 ++++++-------------
 3 files changed, 22 insertions(+), 15 deletions(-)
net/ncsi/internal.h +2 -0
index ef0f8f73826f..4e0842df5234 100644
@@ -289,6 +289,7 @@ enum {
 	ncsi_dev_state_config_sp	= 0x0301,
 	ncsi_dev_state_config_cis,
 	ncsi_dev_state_config_oem_gma,
+	ncsi_dev_state_config_apply_mac,
 	ncsi_dev_state_config_clear_vids,
 	ncsi_dev_state_config_svf,
 	ncsi_dev_state_config_ev,
@@ -322,6 +323,7 @@ struct ncsi_dev_priv {
 #define NCSI_DEV_RESHUFFLE	4
 #define NCSI_DEV_RESET		8            /* Reset state of NC          */
 	unsigned int        gma_flag;        /* OEM GMA flag               */
+	struct sockaddr     pending_mac;     /* MAC address received from GMA */
 	spinlock_t          lock;            /* Protect the NCSI device    */
 	unsigned int        package_probe_id;/* Current ID during probe    */
 	unsigned int        package_num;     /* Number of packages         */
net/ncsi/ncsi-manage.c +14 -2
index 5cf55bde366d..bf276eaf9330 100644
@@ -1038,7 +1038,7 @@ static void ncsi_configure_channel(struct ncsi_dev_priv *ndp)
 			  : ncsi_dev_state_config_clear_vids;
 		break;
 	case ncsi_dev_state_config_oem_gma:
-		nd->state = ncsi_dev_state_config_clear_vids;
+		nd->state = ncsi_dev_state_config_apply_mac;
 
 		nca.package = np->id;
 		nca.channel = nc->id;
@@ -1050,10 +1050,22 @@ static void ncsi_configure_channel(struct ncsi_dev_priv *ndp)
 			nca.type = NCSI_PKT_CMD_OEM;
 			ret = ncsi_gma_handler(&nca, nc->version.mf_id);
 		}
-		if (ret < 0)
+		if (ret < 0) {
+			nd->state = ncsi_dev_state_config_clear_vids;
 			schedule_work(&ndp->work);
+		}
 
 		break;
+	case ncsi_dev_state_config_apply_mac:
+		rtnl_lock();
+		ret = dev_set_mac_address(dev, &ndp->pending_mac, NULL);
+		rtnl_unlock();
+		if (ret < 0)
+			netdev_warn(dev, "NCSI: 'Writing MAC address to device failed\n");
+
+		nd->state = ncsi_dev_state_config_clear_vids;
+
+		fallthrough;
 	case ncsi_dev_state_config_clear_vids:
 	case ncsi_dev_state_config_svf:
 	case ncsi_dev_state_config_ev:
net/ncsi/ncsi-rsp.c +6 -13
index e28be33bdf2c..14bd66909ca4 100644
@@ -628,16 +628,14 @@ static int ncsi_rsp_handler_snfc(struct ncsi_request *nr)
 static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)
 {
 	struct ncsi_dev_priv *ndp = nr->ndp;
+	struct sockaddr *saddr = &ndp->pending_mac;
 	struct net_device *ndev = ndp->ndev.dev;
 	struct ncsi_rsp_oem_pkt *rsp;
-	struct sockaddr saddr;
 	u32 mac_addr_off = 0;
-	int ret = 0;
 
 	/* Get the response header */
 	rsp = (struct ncsi_rsp_oem_pkt *)skb_network_header(nr->rsp);
 
-	saddr.sa_family = ndev->type;
 	ndev->priv_flags |= IFF_LIVE_ADDR_CHANGE;
 	if (mfr_id == NCSI_OEM_MFR_BCM_ID)
 		mac_addr_off = BCM_MAC_ADDR_OFFSET;
@@ -646,22 +644,17 @@ static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)
 	else if (mfr_id == NCSI_OEM_MFR_INTEL_ID)
 		mac_addr_off = INTEL_MAC_ADDR_OFFSET;
 
-	memcpy(saddr.sa_data, &rsp->data[mac_addr_off], ETH_ALEN);
+	saddr->sa_family = ndev->type;
+	memcpy(saddr->sa_data, &rsp->data[mac_addr_off], ETH_ALEN);
 	if (mfr_id == NCSI_OEM_MFR_BCM_ID || mfr_id == NCSI_OEM_MFR_INTEL_ID)
-		eth_addr_inc((u8 *)saddr.sa_data);
-	if (!is_valid_ether_addr((const u8 *)saddr.sa_data))
+		eth_addr_inc((u8 *)saddr->sa_data);
+	if (!is_valid_ether_addr((const u8 *)saddr->sa_data))
 		return -ENXIO;
 
 	/* Set the flag for GMA command which should only be called once */
 	ndp->gma_flag = 1;
 
-	rtnl_lock();
-	ret = dev_set_mac_address(ndev, &saddr, NULL);
-	rtnl_unlock();
-	if (ret < 0)
-		netdev_warn(ndev, "NCSI: 'Writing mac address to device failed\n");
-
-	return ret;
+	return 0;
 }
 
 /* Response handler for Mellanox card */
Signature · 1 line
2.34.1
author maintainer
#6 lore raw

Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

Quoted 7 lines · On Thu, 9 Jan 2025 17:50:54 +0300 Paul Fertser wrote:

Obtaining RTNL lock in a response handler is not allowed since it runs in an atomic softirq context. Postpone setting the MAC address by adding a dedicated step to the configuration FSM.

Fixes 790071347a0a ("net/ncsi: change from ndo_set_mac_address to dev_set_mac_address")
Cc Potin Lai <[email protected]>

Neat!

Potin, please give this a test ASAP.

author maintainer
#8 lore raw

Re: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

In reply to a message not in the archive
Quoted 6 lines · On Fri, 10 Jan 2025 06:02:04 +0000 Potin Lai (賴柏廷) wrote:
Quoted 2 lines

Neat!
Potin, please give this a test ASAP.

Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.

Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..

#9 lore raw

Re: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

Quoted 5 lines · Jakub Kicinski
Quoted 3 lines · On Fri, 10 Jan 2025 06:02:04 +0000 Potin Lai (賴柏廷) wrote:
Quoted 2 lines

Neat!
Potin, please give this a test ASAP.

Quoted 7 lines
Quoted 3 lines

Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.

Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..

Hi Jakub & Paul,

I had a test yesterday, the patch is working and the kernel panic does not happen any more, but we notice sometimes the config_apply_mac state runs before the gma command is handled.

Cosmo helped me to find a potential state handling issue, and I submitted the v2 version.
Please kindly have a look at v2 version with the link below.
v2: https://lore.kernel.org/all/[email protected]/

Best Regards,
Potin

author maintainer
#10 lore raw

Re: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

Quoted 10 lines · On Sat, 11 Jan 2025 19:12:51 +0800 Potin Lai wrote:
Quoted 7 lines
Quoted 3 lines

Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.

Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..

Hi Jakub & Paul,

Quoted 8 lines

I had a test yesterday, the patch is working and the kernel panic does not happen any more, but we notice sometimes the config_apply_mac state runs before the gma command is handled.

Cosmo helped me to find a potential state handling issue, and I submitted the v2 version.
Please kindly have a look at v2 version with the link below.
v2: https://lore.kernel.org/all/[email protected]/

Is there any reason why you reposted Paul's patch? Patch 2 looks like a fix for a separate issue (but for the same use case), am I wrong?

Also one thing you have not done is to provide the Tested-by: tag on Paul's patch :)

Tested-by
#11 lore raw

Re: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

Quoted 16 lines · Jakub Kicinski
Quoted 14 lines · On Sat, 11 Jan 2025 19:12:51 +0800 Potin Lai wrote:
Quoted 7 lines
Quoted 3 lines

Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.

Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..

Hi Jakub & Paul,

I had a test yesterday, the patch is working and the kernel panic does not happen any more, but we notice sometimes the config_apply_mac state runs before the gma command is handled.

Quoted 8 lines
Quoted 4 lines

Cosmo helped me to find a potential state handling issue, and I submitted the v2 version.
Please kindly have a look at v2 version with the link below.
v2: https://lore.kernel.org/all/[email protected]/

Is there any reason why you reposted Paul's patch? Patch 2 looks like a fix for a separate issue (but for the same use case), am I wrong?

Sorry, I thought the second patch needs to be followed by the first patch. Yes, these 2 patches are fixing different issues, I will remove Paul's patch in the next version (v3).

Quoted 3 lines

Also one thing you have not done is to provide the Tested-by: tag on Paul's patch :)

author maintainer
#12 lore raw

Re: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

Quoted 4 lines · On Tue, 14 Jan 2025 09:56:13 +0800 Potin Lai wrote:
Quoted 2 lines

Also one thing you have not done is to provide the Tested-by: tag on Paul's patch :)

Thanks!

#13 lore raw

Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling

In reply to Paul Fertser
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 11 lines · On Thu, 9 Jan 2025 17:50:54 +0300 you wrote:

Obtaining RTNL lock in a response handler is not allowed since it runs in an atomic softirq context. Postpone setting the MAC address by adding a dedicated step to the configuration FSM.

Fixes 790071347a0a ("net/ncsi: change from ndo_set_mac_address to dev_set_mac_address")
Cc Potin Lai <[email protected]>

[...]

Here is the summary with links:
- net/ncsi: fix locking in Get MAC Address handling

    https://git.kernel.org/netdev/net/c/9e2bbab94b88

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html