Revert "net/ncsi: change from ndo_set_mac_address to dev_set_mac_address"
Subsystems (4)
Jakub Kicinski
maintainerPATCH v2 1/1 Revert "net/ncsi: change from ndo_set_mac_address to dev_set_mac_address"
From: Potin Lai <[email protected]>
This reverts commit 790071347a0a1a89e618eedcd51c687ea783aeb3.
We are seeing kernel panic when enabling two NCSI interfaces at same time. It looks like mutex lock is being used in softirq caused the issue.
Kernel panic log:
8021q: adding VLAN 0 to HW filter on device eth0
ftgmac100 1e670000.ethernet eth0: NCSI: Handler for packet type 0x82 returned -19
BUG: scheduling while atomic: systemd-network/697/0x00000100 Modules linked in: 8021q: adding VLAN 0 to HW filter on device eth1 CPU: 0 PID: 697 Comm: systemd-network Tainted: G W 6.6.62-8ea1fc6-dirty-cbd80d0-gcbd80d04d13c #1 Hardware name: Generic DT based system unwind_backtrace from show_stack+0x18/0x1c show_stack from dump_stack_lvl+0x40/0x4c dump_stack_lvl from __schedule_bug+0x5c/0x70 __schedule_bug from __schedule+0x884/0x968 __schedule from schedule+0x58/0xa8 schedule from schedule_preempt_disabled+0x14/0x18 schedule_preempt_disabled from __mutex_lock.constprop.0+0x350/0x76c __mutex_lock.constprop.0 from ncsi_rsp_handler_oem_gma+0x104/0x1a0 ncsi_rsp_handler_oem_gma from ncsi_rcv_rsp+0x120/0x2cc ncsi_rcv_rsp from __netif_receive_skb_one_core+0x60/0x84 __netif_receive_skb_one_core from netif_receive_skb+0x38/0x148 netif_receive_skb from ftgmac100_poll+0x358/0x444 ftgmac100_poll from __napi_poll.constprop.0+0x34/0x1d0 __napi_poll.constprop.0 from net_rx_action+0x350/0x43c net_rx_action from handle_softirqs+0x114/0x32c handle_softirqs from irq_exit+0x88/0xb8 irq_exit from call_with_stack+0x18/0x20 call_with_stack from __irq_usr+0x78/0xa0 Exception stack(0xe075dfb0 to 0xe075dff8) dfa0: 00000000 00000000 00000000 00000020 dfc0: 00000069 aefde3e0 00000000 00000000 00000000 00000000 00000000 aefde4e4 dfe0: 01010101 aefddf20 a6b4331c a6b43618 600f0010 ffffffff
---
Looks like we're not making any progress on this one, so let's go with the revert for 6.13.
Original posting by Potin Lai:
https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.com
I added the Fixes tag and trimmed the kernel log.
--- net/ncsi/ncsi-rsp.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
net/ncsi/ncsi-rsp.c +2 -3
@@ -629,6 +629,7 @@ static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id) {struct ncsi_dev_priv *ndp = nr->ndp;struct net_device *ndev = ndp->ndev.dev; +const struct net_device_ops *ops = ndev->netdev_ops;struct ncsi_rsp_oem_pkt *rsp;struct sockaddr saddr;u32 mac_addr_off =0 ;
@@ -655,9 +656,7 @@ static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)/* Set the flag for GMA command which should only be called once */ ndp->gma_flag =1 ; -rtnl_lock (); - ret =dev_set_mac_address (ndev, &saddr,NULL ); -rtnl_unlock (); + ret = ops->ndo_set_mac_address (ndev, &saddr);if (ret <0 )netdev_warn (ndev,"NCSI: 'Writing mac address to device failed\n" );
Signature · 1 line
2.47.1
Paul Fertser
reviewerHello,
Quoted 2 lines · Jakub Kicinski
Looks like we're not making any progress on this one, so let's go with the revert for 6.13.
But this does break userspace, the commit was there for a reason.
Potin Lai, have you tried deferring this to a work queue instead of reverting to the code which has always been wrong?
Quoted 2 lines
Original posting by Potin Lai:
https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.com
Unfortunately I wasn't on CC and missed that. Can we fix this proper way please, do we have few more days?
If I get it right dev_set_mac_address() just isn't meant to be ever called from a softirq (and NCSI here is special in getting a MAC address update from a "network" frame so that happens in net_rx_action() context). So postponing the actual processing of this reply looks like the way to go, right?
net/ncsi/ncsi-rsp.c static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)
Quoted 7 lines
net/ncsi/ncsi-rsp.c +1
-3-rtnl_lock (); - ret =dev_set_mac_address (ndev, &saddr,NULL ); -rtnl_unlock (); + ret = ops->ndo_set_mac_address (ndev, &saddr);if (ret <0 )netdev_warn (ndev,"NCSI: 'Writing mac address to device failed\n" );
I expect if this is scheduled on events/x with queue_work() then we'll get what we need.
Signature · 2 lines
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software! mailto:[email protected]
Jakub Kicinski
maintainerQuoted 5 lines · On Wed, 8 Jan 2025 23:23:23 +0300 Paul Fertser wrote:
Quoted 2 lines
Original posting by Potin Lai:
https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.comUnfortunately I wasn't on CC and missed that. Can we fix this proper way please, do we have few more days?
Sure thing, we need it in the tree early next week, so probably on the list by Friday so that folks have a chance to review.
Quoted 5 lines
If I get it right
dev_set_mac_address()just isn't meant to be ever called from a softirq (and NCSI here is special in getting a MAC address update from a "network" frame so that happens innet_rx_action()context). So postponing the actual processing of this reply looks like the way to go, right?
I think so. Only trickiness I can think of is the ordering of the work vs admin setting the address from user space. Do venture a patch, I don't know enough about NC-SI.. :(
Quoted 10 lines · Paul Fertser
Hello,
Quoted 2 lines · Jakub Kicinski
Looks like we're not making any progress on this one, so let's go with the revert for 6.13.
But this does break userspace, the commit was there for a reason.
Potin Lai, have you tried deferring this to a work queue instead of reverting to the code which has always been wrong?
Jakub, thanks for letting know about revert. ndo_set_mac_address do not notify userspace about MAC change and as Paul stated it was always been wrong here.
And we talked about reverts in this thread -
https://lore.kernel.org/all/[email protected]/
Probably, incremental proper fix would be better here?
Common case is 1 NCSI interface for server, we tested on this one and works fine for us and that's the reason why we didn't catch that situation. Nowadays some new servers has more than one NCSI interface. Probably we missed that is softirq context which is obviously not a place for rtnl_lock/unlock. Is there any other solution about except delaying dev_set_mac_address in work queue? Or any suggestions about how to deal with that in a proper way?
Don't have any hw with >2 NCSI interface, probably need help with testing.
Potin, can you help with checking the fix?
Thanks.
Jakub Kicinski
maintainerQuoted 7 lines · On Thu, 09 Jan 2025 00:48:07 +0300 Ivan Mikhaylov wrote:
Common case is 1 NCSI interface for server, we tested on this one and works fine for us and that's the reason why we didn't catch that situation. Nowadays some new servers has more than one NCSI interface. Probably we missed that is softirq context which is obviously not a place for rtnl_lock/unlock. Is there any other solution about except delaying
dev_set_mac_addressin work queue? Or any suggestions about how to deal with that in a proper way?
Nothing obvious comes to mind, unfortunately.
But the workqueue shouldn't be too bad.
Paul Fertser
reviewerPATCH 1/1 net/ncsi: fix locking in Get MAC Address handling
In reply to Jakub KicinskiObtaining RTNL lock in a response handler is not allowed since it runs in an atomic softirq context. Postpone setting the MAC address by adding a dedicated step to the configuration FSM.
--- net/ncsi/internal.h | 2 ++ net/ncsi/ncsi-manage.c | 16 ++++++++++++++-- net/ncsi/ncsi-rsp.c | 19 ++++++------------- 3 files changed, 22 insertions(+), 15 deletions(-)
net/ncsi/internal.h +2 -0
@@ -289,6 +289,7 @@ enum { ncsi_dev_state_config_sp =0x0301 , ncsi_dev_state_config_cis, ncsi_dev_state_config_oem_gma, + ncsi_dev_state_config_apply_mac, ncsi_dev_state_config_clear_vids, ncsi_dev_state_config_svf, ncsi_dev_state_config_ev,
@@ -322,6 +323,7 @@ struct ncsi_dev_priv {#define NCSI_DEV_RESHUFFLE 4#define NCSI_DEV_RESET 8/* Reset state of NC */ unsigned int gma_flag;/* OEM GMA flag */ +struct sockaddr pending_mac;/* MAC address received from GMA */ spinlock_t lock;/* Protect the NCSI device */ unsigned int package_probe_id;/* Current ID during probe */ unsigned int package_num;/* Number of packages */
net/ncsi/ncsi-manage.c +14 -2
@@ -1038,7 +1038,7 @@ static void ncsi_configure_channel(struct ncsi_dev_priv *ndp) : ncsi_dev_state_config_clear_vids;break ;case ncsi_dev_state_config_oem_gma: - nd->state = ncsi_dev_state_config_clear_vids; + nd->state = ncsi_dev_state_config_apply_mac; nca.package = np->id; nca.channel = nc->id;
@@ -1050,10 +1050,22 @@ static void ncsi_configure_channel(struct ncsi_dev_priv *ndp) nca.type =NCSI_PKT_CMD_OEM ; ret =ncsi_gma_handler (&nca, nc->version.mf_id); } -if (ret <0 ) +if (ret <0 ) { + nd->state = ncsi_dev_state_config_clear_vids;schedule_work (&ndp->work); + }break ; +case ncsi_dev_state_config_apply_mac: +rtnl_lock (); + ret =dev_set_mac_address (dev, &ndp->pending_mac,NULL ); +rtnl_unlock (); +if (ret <0 ) +netdev_warn (dev,"NCSI: 'Writing MAC address to device failed\n" ); + + nd->state = ncsi_dev_state_config_clear_vids; + + fallthrough;case ncsi_dev_state_config_clear_vids:case ncsi_dev_state_config_svf:case ncsi_dev_state_config_ev:
net/ncsi/ncsi-rsp.c +6 -13
@@ -628,16 +628,14 @@ static int ncsi_rsp_handler_snfc(struct ncsi_request *nr)static int ncsi_rsp_handler_oem_gma (struct ncsi_request *nr,int mfr_id) {struct ncsi_dev_priv *ndp = nr->ndp; +struct sockaddr *saddr = &ndp->pending_mac;struct net_device *ndev = ndp->ndev.dev;struct ncsi_rsp_oem_pkt *rsp; -struct sockaddr saddr;u32 mac_addr_off =0 ; -int ret =0 ;/* Get the response header */ rsp = (struct ncsi_rsp_oem_pkt *)skb_network_header (nr->rsp); - saddr.sa_family = ndev->type; ndev->priv_flags |=IFF_LIVE_ADDR_CHANGE ;if (mfr_id ==NCSI_OEM_MFR_BCM_ID ) mac_addr_off =BCM_MAC_ADDR_OFFSET ;
@@ -646,22 +644,17 @@ static int ncsi_rsp_handler_oem_gma(struct ncsi_request *nr, int mfr_id)else if (mfr_id ==NCSI_OEM_MFR_INTEL_ID ) mac_addr_off =INTEL_MAC_ADDR_OFFSET ; -memcpy (saddr.sa_data, &rsp->data[mac_addr_off],ETH_ALEN ); + saddr->sa_family = ndev->type; +memcpy (saddr->sa_data, &rsp->data[mac_addr_off],ETH_ALEN );if (mfr_id ==NCSI_OEM_MFR_BCM_ID || mfr_id ==NCSI_OEM_MFR_INTEL_ID ) -eth_addr_inc ((u8 *)saddr.sa_data); -if (!is_valid_ether_addr ((const u8 *)saddr.sa_data)) +eth_addr_inc ((u8 *)saddr->sa_data); +if (!is_valid_ether_addr ((const u8 *)saddr->sa_data))return -ENXIO ;/* Set the flag for GMA command which should only be called once */ ndp->gma_flag =1 ; -rtnl_lock (); - ret =dev_set_mac_address (ndev, &saddr,NULL ); -rtnl_unlock (); -if (ret <0 ) -netdev_warn (ndev,"NCSI: 'Writing mac address to device failed\n" ); - -return ret; +return 0 ; }/* Response handler for Mellanox card */
Signature · 1 line
2.34.1
Jakub Kicinski
maintainerRe: [PATCH] net/ncsi: fix locking in Get MAC Address handling
Quoted 7 lines · On Thu, 9 Jan 2025 17:50:54 +0300 Paul Fertser wrote:
Obtaining RTNL lock in a response handler is not allowed since it runs in an atomic softirq context. Postpone setting the MAC address by adding a dedicated step to the configuration FSM.
Fixes 790071347a0a ("net/ncsi: change from ndo_set_mac_address to dev_set_mac_address")Cc Potin Lai <[email protected]>
Neat!
Potin, please give this a test ASAP.
Jakub Kicinski
maintainerRe: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling
In reply to a message not in the archiveQuoted 6 lines · On Fri, 10 Jan 2025 06:02:04 +0000 Potin Lai (賴柏廷) wrote:
Quoted 2 lines
Neat!
Potin, please give this a test ASAP.Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.
Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..
Re: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling
Quoted 5 lines · Jakub Kicinski
Quoted 3 lines · On Fri, 10 Jan 2025 06:02:04 +0000 Potin Lai (賴柏廷) wrote:
Quoted 2 lines
Neat!
Potin, please give this a test ASAP.
Quoted 7 lines
Quoted 3 lines
Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..
Hi Jakub & Paul,
I had a test yesterday, the patch is working and the kernel panic does not happen any more, but we notice sometimes the config_apply_mac state runs before the gma command is handled.
Cosmo helped me to find a potential state handling issue, and I submitted the v2 version.
Please kindly have a look at v2 version with the link below.
v2: https://lore.kernel.org/all/[email protected]/
Best Regards,
Potin
Jakub Kicinski
maintainerRe: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling
Quoted 10 lines · On Sat, 11 Jan 2025 19:12:51 +0800 Potin Lai wrote:
Quoted 7 lines
Quoted 3 lines
Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..
Hi Jakub & Paul,
Quoted 8 lines
I had a test yesterday, the patch is working and the kernel panic does not happen any more, but we notice sometimes the
config_apply_macstate runs before the gma command is handled.Cosmo helped me to find a potential state handling issue, and I submitted the v2 version.
Please kindly have a look at v2 version with the link below.
v2: https://lore.kernel.org/all/[email protected]/
Is there any reason why you reposted Paul's patch? Patch 2 looks like a fix for a separate issue (but for the same use case), am I wrong?
Also one thing you have not done is to provide the Tested-by: tag on Paul's patch :)
Potin Lai
Tested-byRe: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling
Quoted 16 lines · Jakub Kicinski
Quoted 14 lines · On Sat, 11 Jan 2025 19:12:51 +0800 Potin Lai wrote:
Quoted 7 lines
Quoted 3 lines
Thanks for the new patch.
I am currently tied up with other tasks, but I’ll make sure to test it as soon as possible and share the results with you.Understood, would you be able to test it by January 13th? Depending on how long we need to wait we may be better off applying the patch already or waiting with committing..
Hi Jakub & Paul,
I had a test yesterday, the patch is working and the kernel panic does not happen any more, but we notice sometimes the
config_apply_macstate runs before the gma command is handled.
Quoted 8 lines
Quoted 4 lines
Cosmo helped me to find a potential state handling issue, and I submitted the v2 version.
Please kindly have a look at v2 version with the link below.
v2: https://lore.kernel.org/all/[email protected]/Is there any reason why you reposted Paul's patch? Patch 2 looks like a fix for a separate issue (but for the same use case), am I wrong?
Sorry, I thought the second patch needs to be followed by the first patch. Yes, these 2 patches are fixing different issues, I will remove Paul's patch in the next version (v3).
Quoted 3 lines
Also one thing you have not done is to provide the Tested-by: tag on Paul's patch :)
Jakub Kicinski
maintainerRe: 回覆: [External] Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling
Quoted 4 lines · On Tue, 14 Jan 2025 09:56:13 +0800 Potin Lai wrote:
Quoted 2 lines
Also one thing you have not done is to provide the Tested-by: tag on Paul's patch :)
Tested-by Potin Lai <[email protected]>
Thanks!
Re: [PATCH] net/ncsi: fix locking in Get MAC Address handling
In reply to Paul FertserThis patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 11 lines · On Thu, 9 Jan 2025 17:50:54 +0300 you wrote:
Obtaining RTNL lock in a response handler is not allowed since it runs in an atomic softirq context. Postpone setting the MAC address by adding a dedicated step to the configuration FSM.
Fixes 790071347a0a ("net/ncsi: change from ndo_set_mac_address to dev_set_mac_address")Cc Potin Lai <[email protected]>Link https://lore.kernel.org/20241129-potin-revert-ncsi-set-mac-addr-v1-1-94ea2cb596af@gmail.com open hereSigned-off-by Paul Fertser <[email protected]>[...]
Here is the summary with links:
- net/ncsi: fix locking in Get MAC Address handling
https://git.kernel.org/netdev/net/c/9e2bbab94b88
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html