mlx5 updates 2023-06-16
Subsystems (6)
From: Saeed Mahameed <[email protected]>
Hi Dave, Hi Jakub,
This series adds misc updates to mlx5 driver.
For more information please see tag log below.
Please pull and let me know if there is any problem.
Thanks,
Saeed.
The following changes since commit 5a6f6873606e03a0a95afe40ba5e84bb6e28a26f:
ip, ip6: Fix splice to raw and ping sockets (2023-06-16 11:45:16 -0700)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2023-06-16
for you to fetch changes up to 5f2cf757f9c56255470c23a2a4a5574a34edad4b:
net/mlx5: Remove unused ecpu field from struct mlx5_sf_table (2023-06-16 12:02:09 -0700)
----------------------------------------------------------------
mlx5-updates-2023-06-16
1) Added a new event handler to firmware sync reset, which is used to support firmware sync reset flow on smart NIC. Adding this new stage to the flow enables the firmware to ensure host PFs unload before ECPFs unload, to avoid race of PFs recovery.
2) Debugfs for mlx5 eswitch bridge offloads
3) Added two new counters for vport stats
4) Minor Fixups and cleanups for net-next branch
----------------------------------------------------------------
Daniel Jurgens (2):
net/mlx5: Fix the macro for accessing EC VF vports net/mlx5: DR, update query of HCA caps for EC VFs
Gal Pressman (1):
net/mlx5e: Remove mlx5e_dbg() and msglvl support
Jiri Pirko (1):
net/mlx5: Remove unused ecpu field from struct mlx5_sf_table
Juhee Kang (1):
net/mlx5: Add header file for events
Moshe Shemesh (4):
net/mlx5: Ack on sync_reset_request only if PF can do reset_now net/mlx5: Expose timeout for sync reset unload stage net/mlx5: Check DTOR entry value is not zero net/mlx5: Handle sync reset unload event
Or Har-Toov (2):
net/mlx5: Expose bits for local loopback counter net/mlx5e: Add local loopback counter to vport stats
Saeed Mahameed (1):
net/mlx5: E-Switch, remove redundant else statements
Vlad Buslov (3):
net/mlx5: Create eswitch debugfs root directory net/mlx5: Bridge, pass net device when linking vport to bridge net/mlx5: Bridge, expose FDB state via debugfs
.../ethernet/mellanox/mlx5/counters.rst | 10 ++ drivers/net/ethernet/mellanox/mlx5/core/Makefile | 3 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 10 -- .../ethernet/mellanox/mlx5/core/en/port_buffer.c | 44 +++--- .../ethernet/mellanox/mlx5/core/en/rep/bridge.c | 9 +- drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 8 +- drivers/net/ethernet/mellanox/mlx5/core/en_dcbnl.c | 26 ++-- .../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 18 +-- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 5 +- drivers/net/ethernet/mellanox/mlx5/core/en_stats.c | 25 +++- .../net/ethernet/mellanox/mlx5/core/esw/bridge.c | 39 ++--- .../net/ethernet/mellanox/mlx5/core/esw/bridge.h | 12 +- .../mellanox/mlx5/core/esw/bridge_debugfs.c | 89 ++++++++++++ .../ethernet/mellanox/mlx5/core/esw/bridge_priv.h | 6 + .../net/ethernet/mellanox/mlx5/core/esw/legacy.c | 6 +- drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 4 + drivers/net/ethernet/mellanox/mlx5/core/eswitch.h | 4 + drivers/net/ethernet/mellanox/mlx5/core/events.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c | 160 ++++++++++++++++++--- drivers/net/ethernet/mellanox/mlx5/core/health.c | 1 + drivers/net/ethernet/mellanox/mlx5/core/lag/mp.c | 2 +- .../net/ethernet/mellanox/mlx5/core/lag/mpesw.c | 2 +- .../net/ethernet/mellanox/mlx5/core/lib/events.h | 40 ++++++ drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 34 ----- drivers/net/ethernet/mellanox/mlx5/core/lib/tout.c | 7 +- drivers/net/ethernet/mellanox/mlx5/core/lib/tout.h | 1 + drivers/net/ethernet/mellanox/mlx5/core/main.c | 3 + .../net/ethernet/mellanox/mlx5/core/mlx5_core.h | 7 + .../net/ethernet/mellanox/mlx5/core/sf/devlink.c | 1 - .../ethernet/mellanox/mlx5/core/steering/dr_cmd.c | 4 +- drivers/net/ethernet/mellanox/mlx5/core/vport.c | 6 - include/linux/mlx5/device.h | 1 + include/linux/mlx5/mlx5_ifc.h | 13 +- 33 files changed, 435 insertions(+), 167 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/esw/bridge_debugfs.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/events.h
PATCH 1/15 net/mlx5: Ack on sync_reset_request only if PF can do reset_now
From: Moshe Shemesh <[email protected]>
Verify at reset_request stage that PF is capable to do reset_now. In case PF is not capable, notify the firmware that the sync reset can not happen and so firmware will abort the sync reset at early stage and will not send reset_now event to any PF.
--- .../ethernet/mellanox/mlx5/core/fw_reset.c | 57 ++++++++++++++----- 1 file changed, 44 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +44 -13
@@ -276,6 +276,44 @@ static void mlx5_fw_live_patch_event(struct work_struct *work)mlx5_core_err (dev,"Failed to reload FW tracer\n" ); } +static int mlx5_check_dev_ids (struct mlx5_core_dev *dev,u16 dev_id) +{ +struct pci_bus *bridge_bus = dev->pdev->bus; +struct pci_dev *sdev; +u16 sdev_id; +int err; + +/* Check that all functions under the pci bridge are PFs of +* this device otherwise fail this function. +*/ +list_for_each_entry (sdev, &bridge_bus->devices, bus_list) { + err =pci_read_config_word (sdev,PCI_DEVICE_ID , &sdev_id); +if (err) +return err; +if (sdev_id != dev_id) { +mlx5_core_warn (dev,"unrecognized dev_id (0x%x)\n" , sdev_id); +return -EPERM ; + } + } +return 0 ; +} + +static bool mlx5_is_reset_now_capable (struct mlx5_core_dev *dev) +{ +u16 dev_id; +int err; + +if (!MLX5_CAP_GEN (dev, fast_teardown)) { +mlx5_core_warn (dev,"fast teardown is not supported by firmware\n" ); +return -EOPNOTSUPP ; + } + + err =pci_read_config_word (dev->pdev,PCI_DEVICE_ID , &dev_id); +if (err) +return false; +return (!mlx5_check_dev_ids (dev, dev_id)); +} +static void mlx5_sync_reset_request_event (struct work_struct *work) { struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
@@ -283,7 +321,8 @@ static void mlx5_sync_reset_request_event(struct work_struct *work)struct mlx5_core_dev *dev = fw_reset->dev;int err; -if (test_bit (MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST , &fw_reset->reset_flags)) { +if (test_bit (MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST , &fw_reset->reset_flags) || + !mlx5_is_reset_now_capable (dev)) { err =mlx5_fw_reset_set_reset_sync_nack (dev);mlx5_core_warn (dev,"PCI Sync FW Update Reset Nack %s" , err ?"Failed" :"Sent" );
@@ -303,26 +342,18 @@ static int mlx5_pci_link_toggle(struct mlx5_core_dev *dev) {struct pci_bus *bridge_bus = dev->pdev->bus;struct pci_dev *bridge = bridge_bus->self; -u16 reg16, dev_id, sdev_id;unsigned long timeout;struct pci_dev *sdev; +u16 reg16, dev_id;int cap, err;u32 reg32; -/* Check that all functions under the pci bridge are PFs of -* this device otherwise fail this function. -*/ err =pci_read_config_word (dev->pdev,PCI_DEVICE_ID , &dev_id);if (err)return err; -list_for_each_entry (sdev, &bridge_bus->devices, bus_list) { - err =pci_read_config_word (sdev,PCI_DEVICE_ID , &sdev_id); -if (err) -return err; -if (sdev_id != dev_id) -return -EPERM ; - } - + err =mlx5_check_dev_ids (dev, dev_id); +if (err) +return err; cap =pci_find_capability (bridge,PCI_CAP_ID_EXP );if (!cap)return -EOPNOTSUPP ;
Signature · 1 line
2.40.1
Re: [net-next 01/15] net/mlx5: Ack on sync_reset_request only if PF can do reset_now
This series was applied to netdev/net-next.git (main) by Saeed Mahameed <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Saeed Mahameed <[email protected]>:
Quoted 8 lines · On Fri, 16 Jun 2023 13:10:59 -0700 you wrote:
From: Moshe Shemesh <[email protected]>
Verify at
reset_requeststage that PF is capable to doreset_now. In case PF is not capable, notify the firmware that the sync reset can not happen and so firmware will abort the sync reset at early stage and will not sendreset_nowevent to any PF.[...]
Here is the summary with links:
- [net-next,01/15] net/mlx5: Ack on sync_reset_request only if PF can do reset_now
https://git.kernel.org/netdev/net-next/c/92501fa6e421 - [net-next,02/15] net/mlx5: Expose timeout for sync reset unload stage https://git.kernel.org/netdev/net-next/c/8bb42ed4210e - [net-next,03/15] net/mlx5: Check DTOR entry value is not zero https://git.kernel.org/netdev/net-next/c/6f8551f8d9e4 - [net-next,04/15] net/mlx5: Handle sync reset unload event https://git.kernel.org/netdev/net-next/c/7a9770f1bfea - [net-next,05/15] net/mlx5: Create eswitch debugfs root directory https://git.kernel.org/netdev/net-next/c/f405787a0aba - [net-next,06/15] net/mlx5: Bridge, pass net device when linking vport to bridge https://git.kernel.org/netdev/net-next/c/ade19f0d6a3a - [net-next,07/15] net/mlx5: Bridge, expose FDB state via debugfs https://git.kernel.org/netdev/net-next/c/791eb78285e8 - [net-next,08/15] net/mlx5: E-Switch, remove redundant else statements https://git.kernel.org/netdev/net-next/c/8a955da230d3 - [net-next,09/15] net/mlx5e: Remove mlx5e_dbg() and msglvl support https://git.kernel.org/netdev/net-next/c/559f4c32ebff - [net-next,10/15] net/mlx5: Expose bits for local loopback counter https://git.kernel.org/netdev/net-next/c/0bd2e6fc78fd - [net-next,11/15] net/mlx5e: Add local loopback counter to vport stats https://git.kernel.org/netdev/net-next/c/c8013a1f714f - [net-next,12/15] net/mlx5: Fix the macro for accessing EC VF vports https://git.kernel.org/netdev/net-next/c/b3bd68925ebb - [net-next,13/15] net/mlx5: DR, update query of HCA caps for EC VFs https://git.kernel.org/netdev/net-next/c/8bbe544e0380 - [net-next,14/15] net/mlx5: Add header file for events https://git.kernel.org/netdev/net-next/c/2bd3b292955f - [net-next,15/15] net/mlx5: Remove unused ecpu field from struct mlx5_sf_table https://git.kernel.org/netdev/net-next/c/5f2cf757f9c5
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
PATCH 2/15 net/mlx5: Expose timeout for sync reset unload stage
In reply to Saeed MahameedFrom: Moshe Shemesh <[email protected]>
Expose new timoueout in Default Timeouts Register to be used on sync reset flow running on smart NIC. In this flow the driver should know how much time to wait from getting unload request till firmware will ask the PF to continue to next stage of the flow.
--- drivers/net/ethernet/mellanox/mlx5/core/lib/tout.c | 4 +++- drivers/net/ethernet/mellanox/mlx5/core/lib/tout.h | 1 + include/linux/mlx5/mlx5_ifc.h | 4 +++- 3 files changed, 7 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lib/tout.c +3 -1
@@ -24,7 +24,8 @@ static const u32 tout_def_sw_val[MAX_TIMEOUT_TYPES] = { [MLX5_TO_TEARDOWN_MS] = 3000, [MLX5_TO_FSM_REACTIVATE_MS] = 5000, [MLX5_TO_RECLAIM_PAGES_MS] = 5000, - [MLX5_TO_RECLAIM_VFS_PAGES_MS] = 120000 + [MLX5_TO_RECLAIM_VFS_PAGES_MS] = 120000, + [MLX5_TO_RESET_UNLOAD_MS] = 300000 };static void tout_set (struct mlx5_core_dev *dev,u64 val,enum mlx5_timeouts_types type)
@@ -146,6 +147,7 @@ static int tout_query_dtor(struct mlx5_core_dev *dev)MLX5_TIMEOUT_FILL (fsm_reactivate_to, out, dev,MLX5_TO_FSM_REACTIVATE_MS ,0 );MLX5_TIMEOUT_FILL (reclaim_pages_to, out, dev,MLX5_TO_RECLAIM_PAGES_MS ,0 );MLX5_TIMEOUT_FILL (reclaim_vfs_pages_to, out, dev,MLX5_TO_RECLAIM_VFS_PAGES_MS ,0 ); +MLX5_TIMEOUT_FILL (reset_unload_to, out, dev,MLX5_TO_RESET_UNLOAD_MS ,0 );return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/lib/tout.h +1 -0
@@ -26,6 +26,7 @@ enum mlx5_timeouts_types {MLX5_TO_FSM_REACTIVATE_MS ,MLX5_TO_RECLAIM_PAGES_MS ,MLX5_TO_RECLAIM_VFS_PAGES_MS , +MLX5_TO_RESET_UNLOAD_MS , MAX_TIMEOUT_TYPES };
include/linux/mlx5/mlx5_ifc.h +3 -1
@@ -3117,7 +3117,9 @@ struct mlx5_ifc_dtor_reg_bits {struct mlx5_ifc_default_timeout_bits reclaim_vfs_pages_to; -u8 reserved_at_1c0[0x40 ]; +struct mlx5_ifc_default_timeout_bits reset_unload_to; + +u8 reserved_at_1c0[0x20 ]; };enum {
Signature · 1 line
2.40.1
PATCH 3/15 net/mlx5: Check DTOR entry value is not zero
In reply to Saeed MahameedFrom: Moshe Shemesh <[email protected]>
The Default Timeout Register (DTOR) provides timeout values to driver for flows that are device dependent. Zero value for DTOR entry is not valid and should not be used. In case of reading zero value from DTOR, the driver should use the hard coded SW default value instead.
--- drivers/net/ethernet/mellanox/mlx5/core/lib/tout.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/lib/tout.c +2 -1
@@ -119,7 +119,8 @@ u64 _mlx5_tout_ms(struct mlx5_core_dev *dev, enum mlx5_timeouts_types type)#define MLX5_TIMEOUT_FILL (fld, reg_out, dev, to_type, to_extra) \ ({ \ u64 fw_to = MLX5_TIMEOUT_QUERY(fld, reg_out); \ - tout_set(dev, fw_to + (to_extra), to_type); \ + if (fw_to) \ + tout_set(dev, fw_to + (to_extra), to_type); \ fw_to; \ })
Signature · 1 line
2.40.1
PATCH 4/15 net/mlx5: Handle sync reset unload event
In reply to Saeed MahameedFrom: Moshe Shemesh <[email protected]>
Added a new event handler to firmware sync reset, which is used to support firmware sync reset flow on smart NIC. Adding this new stage to the flow enables the firmware to ensure host PFs unload before ECPFs unload, to avoid race of PFs recovery.
If firmware sends sync_reset_unload event to driver the driver should unload and close all HW resources of the function. Once the driver finishes unloading part, it can't get any more events from firmware as event queues are closed, so it polls the reset state field to know when to continue to next stage of the sync reset flow.
Added capability bit for supporting sync_reset_unload event.
--- .../ethernet/mellanox/mlx5/core/fw_reset.c | 103 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/main.c | 3 + include/linux/mlx5/device.h | 1 + include/linux/mlx5/mlx5_ifc.h | 3 +- 4 files changed, 104 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +98 -5
@@ -21,6 +21,7 @@ struct mlx5_fw_reset {struct workqueue_struct *wq;struct work_struct fw_live_patch_work;struct work_struct reset_request_work; +struct work_struct reset_unload_work;struct work_struct reset_reload_work;struct work_struct reset_now_work;struct work_struct reset_abort_work;
@@ -30,6 +31,26 @@ struct mlx5_fw_reset {int ret; }; +enum { +MLX5_FW_RST_STATE_IDLE =0 , +MLX5_FW_RST_STATE_TOGGLE_REQ =4 , +}; + +enum { +MLX5_RST_STATE_BIT_NUM =12 , +MLX5_RST_ACK_BIT_NUM =22 , +}; + +static u8 mlx5_get_fw_rst_state (struct mlx5_core_dev *dev) +{ +return (ioread32be (&dev->iseg->initializing) >>MLX5_RST_STATE_BIT_NUM ) &0xF ; +} + +static void mlx5_set_fw_rst_ack (struct mlx5_core_dev *dev) +{ +iowrite32be (BIT (MLX5_RST_ACK_BIT_NUM ), &dev->iseg->initializing); +} +static int mlx5_fw_reset_enable_remote_dev_reset_set (struct devlink *devlink,u32 id,struct devlink_param_gset_ctx *ctx) {
@@ -155,7 +176,7 @@ int mlx5_fw_reset_set_live_patch(struct mlx5_core_dev *dev)return mlx5_reg_mfrl_set (dev,MLX5_MFRL_REG_RESET_LEVEL0 ,0 ,0 , false); } -static void mlx5_fw_reset_complete_reload (struct mlx5_core_dev *dev) +static void mlx5_fw_reset_complete_reload (struct mlx5_core_dev *dev,bool unloaded) {struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
@@ -163,7 +184,8 @@ static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)if (test_bit (MLX5_FW_RESET_FLAGS_PENDING_COMP , &fw_reset->reset_flags)) {complete (&fw_reset->done); }else { -mlx5_unload_one (dev, false); +if (!unloaded) +mlx5_unload_one (dev, false);if (mlx5_health_wait_pci_up (dev))mlx5_core_err (dev,"reset reload flow aborted, PCI reads still not working\n" );else
@@ -204,7 +226,7 @@ static void mlx5_sync_reset_reload_work(struct work_struct *work)mlx5_sync_reset_clear_reset_requested (dev, false);mlx5_enter_error_state (dev, true); -mlx5_fw_reset_complete_reload (dev); +mlx5_fw_reset_complete_reload (dev, false); }#define MLX5_RESET_POLL_INTERVAL (HZ / 10)
@@ -458,7 +480,70 @@ static void mlx5_sync_reset_now_event(struct work_struct *work)mlx5_enter_error_state (dev, true); done: fw_reset->ret = err; -mlx5_fw_reset_complete_reload (dev); +mlx5_fw_reset_complete_reload (dev, false); +} + +static void mlx5_sync_reset_unload_event (struct work_struct *work) +{ +struct mlx5_fw_reset *fw_reset; +struct mlx5_core_dev *dev; +unsigned long timeout; +bool reset_action; +u8 rst_state; +int err; + + fw_reset = container_of(work, struct mlx5_fw_reset, reset_unload_work); + dev = fw_reset->dev; + +if (mlx5_sync_reset_clear_reset_requested (dev, false)) +return ; + +mlx5_core_warn (dev,"Sync Reset Unload. Function is forced down.\n" ); + + err =mlx5_cmd_fast_teardown_hca (dev); +if (err) +mlx5_core_warn (dev,"Fast teardown failed, unloading, err %d\n" , err); +else +mlx5_enter_error_state (dev, true); + +if (test_bit (MLX5_FW_RESET_FLAGS_PENDING_COMP , &fw_reset->reset_flags)) +mlx5_unload_one_devl_locked (dev, false); +else +mlx5_unload_one (dev, false); + +mlx5_set_fw_rst_ack (dev); +mlx5_core_warn (dev,"Sync Reset Unload done, device reset expected\n" ); + + reset_action = false; + timeout = jiffies +msecs_to_jiffies (mlx5_tout_ms (dev,RESET_UNLOAD )); +do { + rst_state =mlx5_get_fw_rst_state (dev); +if (rst_state ==MLX5_FW_RST_STATE_TOGGLE_REQ || + rst_state ==MLX5_FW_RST_STATE_IDLE ) { + reset_action = true; +break ; + } +msleep (20 ); + }while (!time_after (jiffies, timeout)); + +if (!reset_action) { +mlx5_core_err (dev,"Got timeout waiting for sync reset action, state = %u\n" , + rst_state); + fw_reset->ret = -ETIMEDOUT ; + goto done; + } + +mlx5_core_warn (dev,"Sync Reset, got reset action. rst_state = %u\n" , rst_state); +if (rst_state ==MLX5_FW_RST_STATE_TOGGLE_REQ ) { + err =mlx5_pci_link_toggle (dev); +if (err) { +mlx5_core_warn (dev,"mlx5_pci_link_toggle failed, err %d\n" , err); + fw_reset->ret = err; + } + } + +done: +mlx5_fw_reset_complete_reload (dev, true); }static void mlx5_sync_reset_abort_event (struct work_struct *work)
@@ -483,6 +568,9 @@ static void mlx5_sync_reset_events_handle(struct mlx5_fw_reset *fw_reset, structcase MLX5_SYNC_RST_STATE_RESET_REQUEST :queue_work (fw_reset->wq, &fw_reset->reset_request_work);break ; +case MLX5_SYNC_RST_STATE_RESET_UNLOAD : +queue_work (fw_reset->wq, &fw_reset->reset_unload_work); +break ;case MLX5_SYNC_RST_STATE_RESET_NOW :queue_work (fw_reset->wq, &fw_reset->reset_now_work);break ;
@@ -517,10 +605,13 @@ static int fw_reset_event_notifier(struct notifier_block *nb, unsigned long actiint mlx5_fw_reset_wait_reset_done (struct mlx5_core_dev *dev) {unsigned long pci_sync_update_timeout =mlx5_tout_ms (dev,PCI_SYNC_UPDATE ); -unsigned long timeout =msecs_to_jiffies (pci_sync_update_timeout);struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset; +unsigned long timeout;int err; +if (MLX5_CAP_GEN (dev, pci_sync_for_fw_update_with_driver_unload)) + pci_sync_update_timeout +=mlx5_tout_ms (dev,RESET_UNLOAD ); + timeout =msecs_to_jiffies (pci_sync_update_timeout);if (!wait_for_completion_timeout (&fw_reset->done, timeout)) {mlx5_core_warn (dev,"FW sync reset timeout after %lu seconds\n" , pci_sync_update_timeout /1000 );
@@ -557,6 +648,7 @@ void mlx5_drain_fw_reset(struct mlx5_core_dev *dev)set_bit (MLX5_FW_RESET_FLAGS_DROP_NEW_REQUESTS , &fw_reset->reset_flags);cancel_work_sync (&fw_reset->fw_live_patch_work);cancel_work_sync (&fw_reset->reset_request_work); +cancel_work_sync (&fw_reset->reset_unload_work);cancel_work_sync (&fw_reset->reset_reload_work);cancel_work_sync (&fw_reset->reset_now_work);cancel_work_sync (&fw_reset->reset_abort_work);
@@ -595,6 +687,7 @@ int mlx5_fw_reset_init(struct mlx5_core_dev *dev)INIT_WORK (&fw_reset->fw_live_patch_work, mlx5_fw_live_patch_event);INIT_WORK (&fw_reset->reset_request_work, mlx5_sync_reset_request_event); +INIT_WORK (&fw_reset->reset_unload_work, mlx5_sync_reset_unload_event);INIT_WORK (&fw_reset->reset_reload_work, mlx5_sync_reset_reload_work);INIT_WORK (&fw_reset->reset_now_work, mlx5_sync_reset_now_event);INIT_WORK (&fw_reset->reset_abort_work, mlx5_sync_reset_abort_event);
drivers/net/ethernet/mellanox/mlx5/core/main.c +3 -0
@@ -619,6 +619,9 @@ static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)if (MLX5_CAP_GEN_MAX (dev, pci_sync_for_fw_update_event))MLX5_SET (cmd_hca_cap, set_hca_cap, pci_sync_for_fw_update_event,1 ); +if (MLX5_CAP_GEN_MAX (dev, pci_sync_for_fw_update_with_driver_unload)) +MLX5_SET (cmd_hca_cap, set_hca_cap, + pci_sync_for_fw_update_with_driver_unload,1 );if (MLX5_CAP_GEN_MAX (dev, num_vhca_ports)) MLX5_SET(cmd_hca_cap,
include/linux/mlx5/device.h +1 -0
@@ -716,6 +716,7 @@ enum sync_rst_state_type {MLX5_SYNC_RST_STATE_RESET_REQUEST =0x0 ,MLX5_SYNC_RST_STATE_RESET_NOW =0x1 ,MLX5_SYNC_RST_STATE_RESET_ABORT =0x2 , + MLX5_SYNC_RST_STATE_RESET_UNLOAD = 0x3, }; struct mlx5_eqe_sync_fw_update {
include/linux/mlx5/mlx5_ifc.h +2 -1
@@ -1755,7 +1755,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 reserved_at_328[0x2 ];u8 relaxed_ordering_read[0x1 ];u8 log_max_pd[0x5 ]; -u8 reserved_at_330[0x7 ]; +u8 reserved_at_330[0x6 ]; +u8 pci_sync_for_fw_update_with_driver_unload[0x1 ];u8 vnic_env_cnt_steering_fail[0x1 ];u8 reserved_at_338[0x1 ];u8 q_counter_aggregation[0x1 ];
Signature · 1 line
2.40.1
PATCH 5/15 net/mlx5: Create eswitch debugfs root directory
In reply to Saeed MahameedFrom: Vlad Buslov <[email protected]>
Following patch in series uses the new directory for bridge FDB debugfs. The new directory is intended for all future eswitch-specific debugfs files.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 4 ++++ drivers/net/ethernet/mellanox/mlx5/core/eswitch.h | 3 +++ 2 files changed, 7 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +4 -0
@@ -31,6 +31,7 @@*/ #include <linux/etherdevice.h> +#include <linux/debugfs.h> #include <linux/mlx5/driver.h> #include <linux/mlx5/mlx5_ifc.h> #include <linux/mlx5/vport.h>
@@ -1765,6 +1766,7 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev) esw->manager_vport =mlx5_eswitch_manager_vport (dev); esw->first_host_vport =mlx5_eswitch_first_host_vport_num (dev); + esw->debugfs_root =debugfs_create_dir ("esw" ,mlx5_debugfs_get_dev_root (dev)); esw->work_queue =create_singlethread_workqueue ("mlx5_esw_wq" );if (!esw->work_queue) { err = -ENOMEM ;
@@ -1818,6 +1820,7 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev) abort:if (esw->work_queue)destroy_workqueue (esw->work_queue); +debugfs_remove_recursive (esw->debugfs_root);kfree (esw); unregister_param: devl_params_unregister(priv_to_devlink(dev), mlx5_eswitch_params,
@@ -1844,6 +1847,7 @@ void mlx5_eswitch_cleanup(struct mlx5_eswitch *esw)mutex_destroy (&esw->offloads.decap_tbl_lock);esw_offloads_cleanup (esw);mlx5_esw_vports_cleanup (esw); +debugfs_remove_recursive (esw->debugfs_root);kfree (esw);devl_params_unregister (priv_to_devlink (esw->dev), mlx5_eswitch_params,ARRAY_SIZE (mlx5_eswitch_params));
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +3 -0
@@ -304,6 +304,8 @@ enum { MLX5_ESW_FDB_CREATED = BIT(0), }; +struct dentry ; +struct mlx5_eswitch {struct mlx5_core_dev *dev;struct mlx5_nb nb;
@@ -312,6 +314,7 @@ struct mlx5_eswitch {struct hlist_head mc_table[MLX5_L2_ADDR_HASH_SIZE ];struct esw_mc_addr mc_promisc;/* end of legacy */ +struct dentry *debugfs_root;struct workqueue_struct *work_queue;struct xarray vports;u32 flags;
Signature · 1 line
2.40.1
PATCH 6/15 net/mlx5: Bridge, pass net device when linking vport to bridge
In reply to Saeed MahameedFrom: Vlad Buslov <[email protected]>
Following patch requires access to additional data in bridge net_device. Pass the whole structure down the stack instead of adding necessary fields as function arguments one-by-one.
--- .../mellanox/mlx5/core/en/rep/bridge.c | 9 +++-- .../ethernet/mellanox/mlx5/core/esw/bridge.c | 35 ++++++++++--------- .../ethernet/mellanox/mlx5/core/esw/bridge.h | 10 +++--- 3 files changed, 29 insertions(+), 25 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c +4 -5
@@ -136,7 +136,6 @@ static int mlx5_esw_bridge_port_changeupper(struct notifier_block *nb, void *ptrstruct mlx5_eswitch *esw = br_offloads->esw;u16 vport_num, esw_owner_vhca_id;struct netlink_ext_ack *extack; -int ifindex = upper->ifindex;int err =0 ;if (!netif_is_bridge_master (upper))
@@ -150,15 +149,15 @@ static int mlx5_esw_bridge_port_changeupper(struct notifier_block *nb, void *ptrif (mlx5_esw_bridge_is_local (dev, rep, esw)) err = info->linking ? -mlx5_esw_bridge_vport_link (ifindex, vport_num, esw_owner_vhca_id, +mlx5_esw_bridge_vport_link (upper, vport_num, esw_owner_vhca_id, br_offloads, extack) : -mlx5_esw_bridge_vport_unlink (ifindex, vport_num, esw_owner_vhca_id, +mlx5_esw_bridge_vport_unlink (upper, vport_num, esw_owner_vhca_id, br_offloads, extack);else if (mlx5_esw_bridge_dev_same_hw (rep, esw)) err = info->linking ? -mlx5_esw_bridge_vport_peer_link (ifindex, vport_num, esw_owner_vhca_id, +mlx5_esw_bridge_vport_peer_link (upper, vport_num, esw_owner_vhca_id, br_offloads, extack) : -mlx5_esw_bridge_vport_peer_unlink (ifindex, vport_num, esw_owner_vhca_id, +mlx5_esw_bridge_vport_peer_unlink (upper, vport_num, esw_owner_vhca_id, br_offloads, extack);return err;
drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c +19 -16
@@ -834,7 +834,7 @@ mlx5_esw_bridge_egress_miss_flow_create(struct mlx5_flow_table *egress_ft,return handle; } -static struct mlx5_esw_bridge *mlx5_esw_bridge_create (int ifindex, +static struct mlx5_esw_bridge *mlx5_esw_bridge_create (struct net_device *br_netdev,struct mlx5_esw_bridge_offloads *br_offloads) {struct mlx5_esw_bridge *bridge;
@@ -858,7 +858,7 @@ static struct mlx5_esw_bridge *mlx5_esw_bridge_create(int ifindex, goto err_mdb_ht;INIT_LIST_HEAD (&bridge->fdb_list); - bridge->ifindex = ifindex; + bridge->ifindex = br_netdev->ifindex; bridge->refcnt =1 ; bridge->ageing_time =clock_t_to_jiffies (BR_DEFAULT_AGEING_TIME ); bridge->vlan_proto =ETH_P_8021Q ;
@@ -898,14 +898,14 @@ static void mlx5_esw_bridge_put(struct mlx5_esw_bridge_offloads *br_offloads, }static struct mlx5_esw_bridge * -mlx5_esw_bridge_lookup (int ifindex,struct mlx5_esw_bridge_offloads *br_offloads) +mlx5_esw_bridge_lookup (struct net_device *br_netdev,struct mlx5_esw_bridge_offloads *br_offloads) {struct mlx5_esw_bridge *bridge;ASSERT_RTNL ();list_for_each_entry (bridge, &br_offloads->bridges, list) { -if (bridge->ifindex == ifindex) { +if (bridge->ifindex == br_netdev->ifindex) {mlx5_esw_bridge_get (bridge);return bridge; }
@@ -918,7 +918,7 @@ mlx5_esw_bridge_lookup(int ifindex, struct mlx5_esw_bridge_offloads *br_offloadsreturn ERR_PTR (err); } - bridge =mlx5_esw_bridge_create (ifindex, br_offloads); + bridge =mlx5_esw_bridge_create (br_netdev, br_offloads);if (IS_ERR (bridge) &&list_empty (&br_offloads->bridges))mlx5_esw_bridge_ingress_table_cleanup (br_offloads);return bridge;
@@ -1601,15 +1601,15 @@ static int mlx5_esw_bridge_vport_cleanup(struct mlx5_esw_bridge_offloads *br_offreturn 0 ; } -static int mlx5_esw_bridge_vport_link_with_flags (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, -u16 flags, +static int mlx5_esw_bridge_vport_link_with_flags (struct net_device *br_netdev,u16 vport_num, +u16 esw_owner_vhca_id,u16 flags,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack) {struct mlx5_esw_bridge *bridge;int err; - bridge =mlx5_esw_bridge_lookup (ifindex, br_offloads); + bridge =mlx5_esw_bridge_lookup (br_netdev, br_offloads);if (IS_ERR (bridge)) {NL_SET_ERR_MSG_MOD (extack,"Error checking for existing bridge with same ifindex" );return PTR_ERR (bridge);
@@ -1627,15 +1627,16 @@ static int mlx5_esw_bridge_vport_link_with_flags(int ifindex, u16 vport_num, u16return err; } -int mlx5_esw_bridge_vport_link (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_link (struct net_device *br_netdev,u16 vport_num,u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack) { -return mlx5_esw_bridge_vport_link_with_flags (ifindex, vport_num, esw_owner_vhca_id,0 , +return mlx5_esw_bridge_vport_link_with_flags (br_netdev, vport_num, esw_owner_vhca_id,0 , br_offloads, extack); } -int mlx5_esw_bridge_vport_unlink (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_unlink (struct net_device *br_netdev,u16 vport_num, +u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack) {
@@ -1647,7 +1648,7 @@ int mlx5_esw_bridge_vport_unlink(int ifindex, u16 vport_num, u16 esw_owner_vhca_NL_SET_ERR_MSG_MOD (extack,"Port is not attached to any bridge" );return -EINVAL ; } -if (port->bridge->ifindex != ifindex) { +if (port->bridge->ifindex != br_netdev->ifindex) {NL_SET_ERR_MSG_MOD (extack,"Port is attached to another bridge" );return -EINVAL ; }
@@ -1658,23 +1659,25 @@ int mlx5_esw_bridge_vport_unlink(int ifindex, u16 vport_num, u16 esw_owner_vhca_return err; } -int mlx5_esw_bridge_vport_peer_link (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_peer_link (struct net_device *br_netdev,u16 vport_num, +u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack) {if (!MLX5_CAP_ESW (br_offloads->esw->dev, merged_eswitch))return 0 ; -return mlx5_esw_bridge_vport_link_with_flags (ifindex, vport_num, esw_owner_vhca_id, +return mlx5_esw_bridge_vport_link_with_flags (br_netdev, vport_num, esw_owner_vhca_id,MLX5_ESW_BRIDGE_PORT_FLAG_PEER , br_offloads, extack); } -int mlx5_esw_bridge_vport_peer_unlink (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_peer_unlink (struct net_device *br_netdev,u16 vport_num, +u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack) { -return mlx5_esw_bridge_vport_unlink (ifindex, vport_num, esw_owner_vhca_id, br_offloads, +return mlx5_esw_bridge_vport_unlink (br_netdev, vport_num, esw_owner_vhca_id, br_offloads, extack); }
drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h +6 -4
@@ -43,16 +43,18 @@ struct mlx5_esw_bridge_offloads {struct mlx5_esw_bridge_offloads *mlx5_esw_bridge_init (struct mlx5_eswitch *esw);void mlx5_esw_bridge_cleanup (struct mlx5_eswitch *esw); -int mlx5_esw_bridge_vport_link (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_link (struct net_device *br_netdev,u16 vport_num,u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack); -int mlx5_esw_bridge_vport_unlink (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_unlink (struct net_device *br_netdev,u16 vport_num,u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack); -int mlx5_esw_bridge_vport_peer_link (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_peer_link (struct net_device *br_netdev,u16 vport_num, +u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack); -int mlx5_esw_bridge_vport_peer_unlink (int ifindex,u16 vport_num,u16 esw_owner_vhca_id, +int mlx5_esw_bridge_vport_peer_unlink (struct net_device *br_netdev,u16 vport_num, +u16 esw_owner_vhca_id,struct mlx5_esw_bridge_offloads *br_offloads,struct netlink_ext_ack *extack); void mlx5_esw_bridge_fdb_update_used(struct net_device *dev, u16 vport_num, u16 esw_owner_vhca_id,
Signature · 1 line
2.40.1
PATCH 7/15 net/mlx5: Bridge, expose FDB state via debugfs
In reply to Saeed MahameedFrom: Vlad Buslov <[email protected]>
For debugging purposes expose offloaded FDB state (flags, counters, etc.) via debugfs inside 'esw' root directory. Example debugfs file output:
$ cat mlx5/0000\:08\:00.0/esw/bridge/bridge1/fdb DEV MAC VLAN PACKETS BYTES LASTUSE FLAGS enp8s0f0_1 e4:0a:05:08:00:06 2 2 204 4295567112 0x0 enp8s0f0_0 e4:0a:05:08:00:03 2 3 278 4295567112 0x0
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 3 +- .../ethernet/mellanox/mlx5/core/esw/bridge.c | 4 + .../ethernet/mellanox/mlx5/core/esw/bridge.h | 2 + .../mellanox/mlx5/core/esw/bridge_debugfs.c | 89 +++++++++++++++++++ .../mellanox/mlx5/core/esw/bridge_priv.h | 6 ++ 5 files changed, 103 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/esw/bridge_debugfs.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -1
@@ -75,7 +75,8 @@ mlx5_core-$(CONFIG_MLX5_ESWITCH) += esw/acl/helper.o \ esw/acl/egress_lgcy.o esw/acl/egress_ofld.o \ esw/acl/ingress_lgcy.o esw/acl/ingress_ofld.o -mlx5_core-$(CONFIG_MLX5_BRIDGE) += esw/bridge.o esw/bridge_mcast.o en/rep/bridge.o +mlx5_core-$(CONFIG_MLX5_BRIDGE) += esw/bridge.o esw/bridge_mcast.o esw/bridge_debugfs.o \ + en/rep/bridge.o mlx5_core-$(CONFIG_THERMAL) += thermal.o mlx5_core-$(CONFIG_MLX5_MPFS) += lib/mpfs.o
drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c +4 -0
@@ -863,6 +863,7 @@ static struct mlx5_esw_bridge *mlx5_esw_bridge_create(struct net_device *br_netd bridge->ageing_time =clock_t_to_jiffies (BR_DEFAULT_AGEING_TIME ); bridge->vlan_proto =ETH_P_8021Q ;list_add (&bridge->list, &br_offloads->bridges); +mlx5_esw_bridge_debugfs_init (br_netdev, bridge);return bridge;
@@ -886,6 +887,7 @@ static void mlx5_esw_bridge_put(struct mlx5_esw_bridge_offloads *br_offloads,if (--bridge->refcnt)return ; +mlx5_esw_bridge_debugfs_cleanup (bridge);mlx5_esw_bridge_egress_table_cleanup (bridge);mlx5_esw_bridge_mcast_disable (bridge);list_del (&bridge->list);
@@ -1904,6 +1906,7 @@ struct mlx5_esw_bridge_offloads *mlx5_esw_bridge_init(struct mlx5_eswitch *esw)xa_init (&br_offloads->ports); br_offloads->esw = esw; esw->br_offloads = br_offloads; +mlx5_esw_bridge_debugfs_offloads_init (br_offloads);return br_offloads; }
@@ -1919,6 +1922,7 @@ void mlx5_esw_bridge_cleanup(struct mlx5_eswitch *esw)mlx5_esw_bridge_flush (br_offloads);WARN_ON (!xa_empty (&br_offloads->ports)); +mlx5_esw_bridge_debugfs_offloads_cleanup (br_offloads); esw->br_offloads =NULL ;kvfree (br_offloads);
drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h +2 -0
@@ -10,6 +10,7 @@#include <linux/xarray.h> #include "eswitch.h" +struct dentry ;struct mlx5_flow_table ;struct mlx5_flow_group ;
@@ -17,6 +18,7 @@ struct mlx5_esw_bridge_offloads {struct mlx5_eswitch *esw;struct list_head bridges;struct xarray ports; +struct dentry *debugfs_root;struct notifier_block netdev_nb;struct notifier_block nb_blk;
drivers/net/ethernet/mellanox/mlx5/core/esw/bridge_debugfs.c new file +89 -0
89 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/esw/bridge_priv.h +6 -0
@@ -199,6 +199,7 @@ struct mlx5_esw_bridge {int refcnt;struct list_head list;struct mlx5_esw_bridge_offloads *br_offloads; +struct dentry *debugfs_dir;struct list_head fdb_list;struct rhashtable fdb_ht;
@@ -241,4 +242,9 @@ void mlx5_esw_bridge_port_mdb_vlan_flush(struct mlx5_esw_bridge_port *port, struct mlx5_esw_bridge_vlan *vlan);void mlx5_esw_bridge_mdb_flush (struct mlx5_esw_bridge *bridge); +void mlx5_esw_bridge_debugfs_offloads_init (struct mlx5_esw_bridge_offloads *br_offloads); +void mlx5_esw_bridge_debugfs_offloads_cleanup (struct mlx5_esw_bridge_offloads *br_offloads); +void mlx5_esw_bridge_debugfs_init (struct net_device *br_netdev,struct mlx5_esw_bridge *bridge); +void mlx5_esw_bridge_debugfs_cleanup (struct mlx5_esw_bridge *bridge); +#endif /* _MLX5_ESW_BRIDGE_PRIVATE_ */
Signature · 1 line
2.40.1
Jakub Kicinski
maintainerRe: [net-next 07/15] net/mlx5: Bridge, expose FDB state via debugfs
Quoted 4 lines · On Fri, 16 Jun 2023 13:11:05 -0700 Saeed Mahameed wrote:
$ cat mlx5/0000\:08\:00.0/esw/bridge/bridge1/fdb DEV MAC VLAN PACKETS BYTES LASTUSE FLAGS enp8s0f0_1 e4:0a:05:08:00:06 2 2 204 4295567112 0x0 enp8s0f0_0 e4:0a:05:08:00:03 2 3 278 4295567112 0x0
The flags here are the only thing that's mlx5 specific? Why not add an API for dumping this kind of stats that other drivers can reuse?
The rest of the patches LGTM
Signature · 1 line
pw-bot: cr
Re: [net-next 07/15] net/mlx5: Bridge, expose FDB state via debugfs
Quoted 7 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 16 Jun 2023 13:11:05 -0700 Saeed Mahameed wrote:
$ cat mlx5/0000\:08\:00.0/esw/bridge/bridge1/fdb DEV MAC VLAN PACKETS BYTES LASTUSE FLAGS enp8s0f0_1 e4:0a:05:08:00:06 2 2 204 4295567112 0x0 enp8s0f0_0 e4:0a:05:08:00:03 2 3 278 4295567112 0x0The flags here are the only thing that's mlx5 specific?
Not exactly. This debugfs exposes the state of our bridge offload layer. For example, when VF representors from different eswitches are added to the same bridge every FDB entry on such bridge will have multiple underlying offloaded steering rules (one per eswitch instance connected to the bridge). User will observe the entries in all connected 'fdb' debugfs' (all except the 'main' entry will have flag MLX5_ESW_BRIDGE_FLAG_PEER set) and their corresponding counters will increment only on the eswitch instance that is actually processing the packets, which depends on the mode (when bonding device is added to the bridge in single FDB LAG mode all traffic appears on eswitch 0, without it the the traffic is on the eswitch of parent uplink of the VF). I understand that this is rather convoluted but this is exactly why we are going with debugfs.
Quoted 2 lines
Why not add an API for dumping this kind of stats that other drivers can reuse?
As explained in previous paragraph we would like to expose internal mlx5 bridge layer for debug purposes, not to design generic bridge FDB counter interface. Also, the debugging needs of our implementation may not correspond to other drivers because we don't have a 'hardware switch' on our NIC, so we do things like learning and ageing in software, and have to deal with multiple possible mode of operations (single FDB vs merged eswitch from previous example, etc.).
Quoted 2 lines
The rest of the patches LGTM
Jakub Kicinski
maintainerRe: [net-next 07/15] net/mlx5: Bridge, expose FDB state via debugfs
Quoted 26 lines · On Mon, 19 Jun 2023 11:37:30 +0300 Vlad Buslov wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 16 Jun 2023 13:11:05 -0700 Saeed Mahameed wrote:
$ cat mlx5/0000\:08\:00.0/esw/bridge/bridge1/fdb DEV MAC VLAN PACKETS BYTES LASTUSE FLAGS enp8s0f0_1 e4:0a:05:08:00:06 2 2 204 4295567112 0x0 enp8s0f0_0 e4:0a:05:08:00:03 2 3 278 4295567112 0x0The flags here are the only thing that's mlx5 specific?
Not exactly. This debugfs exposes the state of our bridge offload layer. For example, when VF representors from different eswitches are added to the same bridge every FDB entry on such bridge will have multiple underlying offloaded steering rules (one per eswitch instance connected to the bridge). User will observe the entries in all connected 'fdb' debugfs' (all except the 'main' entry will have flag
MLX5_ESW_BRIDGE_FLAG_PEERset) and their corresponding counters will increment only on the eswitch instance that is actually processing the packets, which depends on the mode (when bonding device is added to the bridge in single FDB LAG mode all traffic appears on eswitch 0, without it the the traffic is on the eswitch of parent uplink of the VF). I understand that this is rather convoluted but this is exactly why we are going with debugfs.Quoted 2 lines
Why not add an API for dumping this kind of stats that other drivers can reuse?
Quoted 7 lines
As explained in previous paragraph we would like to expose internal mlx5 bridge layer for debug purposes, not to design generic bridge FDB counter interface. Also, the debugging needs of our implementation may not correspond to other drivers because we don't have a 'hardware switch' on our NIC, so we do things like learning and ageing in software, and have to deal with multiple possible mode of operations (single FDB vs merged eswitch from previous example, etc.).
Looks like my pw-bot shenanigans backfired / crashed, patches didn't get marked as Changes Requested and Dave applied the series :S
I understand the motivation but the information is easy enough to understand to potentially tempt a user to start depending on it for production needs. Then another vendor may get asked to implement similar but not exactly the same set of stats etc. etc.
Do you have customer who will need this?
At the very least please follow up to make the files readable to only root. Normal users should never look at debugfs IMO.
Re: [net-next 07/15] net/mlx5: Bridge, expose FDB state via debugfs
Quoted 35 lines · Jakub Kicinski
Quoted 26 lines · On Mon, 19 Jun 2023 11:37:30 +0300 Vlad Buslov wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 16 Jun 2023 13:11:05 -0700 Saeed Mahameed wrote:
$ cat mlx5/0000\:08\:00.0/esw/bridge/bridge1/fdb DEV MAC VLAN PACKETS BYTES LASTUSE FLAGS enp8s0f0_1 e4:0a:05:08:00:06 2 2 204 4295567112 0x0 enp8s0f0_0 e4:0a:05:08:00:03 2 3 278 4295567112 0x0The flags here are the only thing that's mlx5 specific?
Not exactly. This debugfs exposes the state of our bridge offload layer. For example, when VF representors from different eswitches are added to the same bridge every FDB entry on such bridge will have multiple underlying offloaded steering rules (one per eswitch instance connected to the bridge). User will observe the entries in all connected 'fdb' debugfs' (all except the 'main' entry will have flag
MLX5_ESW_BRIDGE_FLAG_PEERset) and their corresponding counters will increment only on the eswitch instance that is actually processing the packets, which depends on the mode (when bonding device is added to the bridge in single FDB LAG mode all traffic appears on eswitch 0, without it the the traffic is on the eswitch of parent uplink of the VF). I understand that this is rather convoluted but this is exactly why we are going with debugfs.Quoted 2 lines
Why not add an API for dumping this kind of stats that other drivers can reuse?
Quoted 7 lines
As explained in previous paragraph we would like to expose internal mlx5 bridge layer for debug purposes, not to design generic bridge FDB counter interface. Also, the debugging needs of our implementation may not correspond to other drivers because we don't have a 'hardware switch' on our NIC, so we do things like learning and ageing in software, and have to deal with multiple possible mode of operations (single FDB vs merged eswitch from previous example, etc.).
Quoted 7 lines
Looks like my pw-bot shenanigans backfired / crashed, patches didn't get marked as Changes Requested and Dave applied the series :S
I understand the motivation but the information is easy enough to understand to potentially tempt a user to start depending on it for production needs. Then another vendor may get asked to implement similar but not exactly the same set of stats etc. etc.
That could happen (although consider that bridge offload functionality significantly predates mlx5 implementation and apparently no one really needed that until now), but such API would supplement, not replace the debugfs since we would like to have per-eswitch FDB state exposed together with our internal flags and everything as explained in my previous email.
Quoted 2 lines
Do you have customer who will need this?
Yes. But strictly for debugging (by human), not for building some proprietary weird user-space switch-controller application that would query this in normal mode of operation, if I understand your concern correctly.
Quoted 3 lines
At the very least please follow up to make the files readable to only root. Normal users should never look at debugfs IMO.
Hmm, all other debugfs' in mlx5 that I tend to use for switching-related functionality debugging seems to be 0444 (lag, steering, tc hairpin). Why would this one be any different?
Jakub Kicinski
maintainerRe: [net-next 07/15] net/mlx5: Bridge, expose FDB state via debugfs
Quoted 8 lines · On Mon, 19 Jun 2023 21:34:02 +0300 Vlad Buslov wrote:
Quoted 7 lines
Looks like my pw-bot shenanigans backfired / crashed, patches didn't get marked as Changes Requested and Dave applied the series :S
I understand the motivation but the information is easy enough to understand to potentially tempt a user to start depending on it for production needs. Then another vendor may get asked to implement similar but not exactly the same set of stats etc. etc.
Quoted 6 lines
That could happen (although consider that bridge offload functionality significantly predates mlx5 implementation and apparently no one really needed that until now), but such API would supplement, not replace the debugfs since we would like to have per-eswitch FDB state exposed together with our internal flags and everything as explained in my previous email.
Because crossing between eswitches incurs additional cost?
Quoted 7 lines
Quoted 1 line
Do you have customer who will need this?
Yes. But strictly for debugging (by human), not for building some proprietary weird user-space switch-controller application that would query this in normal mode of operation, if I understand your concern correctly.
Quoted 6 lines
Quoted 2 lines
At the very least please follow up to make the files readable to only root. Normal users should never look at debugfs IMO.
Hmm, all other debugfs' in mlx5 that I tend to use for switching-related functionality debugging seems to be 0444 (lag, steering, tc hairpin). Why would this one be any different?
Querying the stats seems generally useful, so I'd like to narrow down the access as much as possible. This way if the usage spreads we'll hear complaints and can go back to creating a more appropriate API.
Re: [net-next 07/15] net/mlx5: Bridge, expose FDB state via debugfs
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 19 Jun 2023 21:34:02 +0300 Vlad Buslov wrote:
Quoted 7 lines
Looks like my pw-bot shenanigans backfired / crashed, patches didn't get marked as Changes Requested and Dave applied the series :S
I understand the motivation but the information is easy enough to understand to potentially tempt a user to start depending on it for production needs. Then another vendor may get asked to implement similar but not exactly the same set of stats etc. etc.
Quoted 8 lines
Quoted 6 lines
That could happen (although consider that bridge offload functionality significantly predates mlx5 implementation and apparently no one really needed that until now), but such API would supplement, not replace the debugfs since we would like to have per-eswitch FDB state exposed together with our internal flags and everything as explained in my previous email.
Because crossing between eswitches incurs additional cost?
It is not about performance. I install multiple steering rules (one per eswitch), I would like to understand which one is processing the packets when something goes wrong (main or peer). User/field engineer complains that some FDB is (not) aged out according to the expectations, I would like them to dump the file several times while running traffic to see how the lastused and counters changed during that. Just the basic debugging stuff because, again, ConnectX doesn't implement 802.1D in hardware so all the FDB management is done purely in software and we need a way to expose the state.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Do you have customer who will need this?
Yes. But strictly for debugging (by human), not for building some proprietary weird user-space switch-controller application that would query this in normal mode of operation, if I understand your concern correctly.
Quoted 2 lines
At the very least please follow up to make the files readable to only root. Normal users should never look at debugfs IMO.
Quoted 7 lines
Quoted 3 lines
Hmm, all other debugfs' in mlx5 that I tend to use for switching-related functionality debugging seems to be 0444 (lag, steering, tc hairpin). Why would this one be any different?
Querying the stats seems generally useful, so I'd like to narrow down the access as much as possible. This way if the usage spreads we'll hear complaints and can go back to creating a more appropriate API.
Ack.
PATCH 8/15 net/mlx5: E-Switch, remove redundant else statements
In reply to Saeed MahameedFrom: Saeed Mahameed <[email protected]>
These else statement blocks are redundant since the if block already jumps to the function abort label.
--- drivers/net/ethernet/mellanox/mlx5/core/esw/legacy.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/legacy.c +2 -4
@@ -285,9 +285,8 @@ static int _mlx5_eswitch_set_vepa_locked(struct mlx5_eswitch *esw,if (IS_ERR (flow_rule)) { err =PTR_ERR (flow_rule); goto out; - }else { - esw->fdb_table.legacy.vepa_uplink_rule = flow_rule; } + esw->fdb_table.legacy.vepa_uplink_rule = flow_rule;/* Star rule to forward all traffic to uplink vport */ memset (&dest,0 ,sizeof (dest));
@@ -299,9 +298,8 @@ static int _mlx5_eswitch_set_vepa_locked(struct mlx5_eswitch *esw,if (IS_ERR (flow_rule)) { err =PTR_ERR (flow_rule); goto out; - }else { - esw->fdb_table.legacy.vepa_star_rule = flow_rule; } + esw->fdb_table.legacy.vepa_star_rule = flow_rule; out:kvfree (spec);
Signature · 1 line
2.40.1
PATCH 9/15 net/mlx5e: Remove mlx5e_dbg() and msglvl support
In reply to Saeed MahameedFrom: Gal Pressman <[email protected]>
The msglvl support was implemented using the mlx5e_dbg() macro which is rarely used in the driver, and is not very useful when you can just use dynamic debug instead.
Remove mlx5e_dbg() and convert its usages to netdev_dbg().
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 10 ----- .../mellanox/mlx5/core/en/port_buffer.c | 44 ++++++++++--------- .../net/ethernet/mellanox/mlx5/core/en_arfs.c | 8 ++-- .../ethernet/mellanox/mlx5/core/en_dcbnl.c | 26 +++++------ .../ethernet/mellanox/mlx5/core/en_ethtool.c | 18 ++------ .../net/ethernet/mellanox/mlx5/core/en_main.c | 5 +-- 6 files changed, 45 insertions(+), 66 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -10
@@ -165,15 +165,6 @@ struct page_pool;#define MLX5E_MAX_KLM_PER_WQE (mdev) \ MLX5E_KLM_ENTRIES_PER_WQE(MLX5_SEND_WQE_BB * mlx5e_get_max_sq_aligned_wqebbs(mdev)) -#define MLX5E_MSG_LEVEL NETIF_MSG_LINK - -#define mlx5e_dbg (mlevel, priv, format, ...) \ -do { \ - if (NETIF_MSG_##mlevel & (priv)->msglevel) \ - netdev_warn(priv->netdev, format, \ - ##__VA_ARGS__); \ -} while (0) -#define mlx5e_state_dereference (priv, p) \ rcu_dereference_protected((p), lockdep_is_held(&(priv)->state_lock))
@@ -880,7 +871,6 @@ struct mlx5e_priv {#endif /* priv data path fields - end */ -u32 msglevel;unsigned long state;struct mutex state_lock;/* Protects Interface state */ struct mlx5e_rq drop_rq;
drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.c +23 -21
99 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c +4 -4
@@ -570,10 +570,10 @@ static struct mlx5_flow_handle *arfs_add_rule(struct mlx5e_priv *priv,if (IS_ERR (rule)) { err =PTR_ERR (rule); priv->channel_stats[arfs_rule->rxq]->rq.arfs_err++; -mlx5e_dbg (HW , priv, -"%s: add rule(filter id=%d, rq idx=%d, ip proto=0x%x) failed,err=%d\n" , - __func__, arfs_rule->filter_id, arfs_rule->rxq, - tuple->ip_proto, err); +netdev_dbg (priv->netdev, +"%s: add rule(filter id=%d, rq idx=%d, ip proto=0x%x) failed,err=%d\n" , + __func__, arfs_rule->filter_id, arfs_rule->rxq, + tuple->ip_proto, err); } out:
drivers/net/ethernet/mellanox/mlx5/core/en_dcbnl.c +13 -13
@@ -275,10 +275,10 @@ static int mlx5e_dcbnl_ieee_setets_core(struct mlx5e_priv *priv, struct ieee_etsmemcpy (priv->dcbx.tc_tsa, ets->tc_tsa,sizeof (ets->tc_tsa));for (i =0 ; i <IEEE_8021QAZ_MAX_TCS ; i++) { -mlx5e_dbg (HW , priv,"%s: prio_%d <=> tc_%d\n" , - __func__, i, ets->prio_tc[i]); -mlx5e_dbg (HW , priv,"%s: tc_%d <=> tx_bw_%d%%, group_%d\n" , - __func__, i, tc_tx_bw[i], tc_group[i]); +netdev_dbg (priv->netdev,"%s: prio_%d <=> tc_%d\n" , + __func__, i, ets->prio_tc[i]); +netdev_dbg (priv->netdev,"%s: tc_%d <=> tx_bw_%d%%, group_%d\n" , + __func__, i, tc_tx_bw[i], tc_group[i]); }return err;
@@ -399,9 +399,9 @@ static int mlx5e_dcbnl_ieee_setpfc(struct net_device *dev, }if (!ret) { -mlx5e_dbg (HW , priv, -"%s: PFC per priority bit mask: 0x%x\n" , - __func__, pfc->pfc_en); +netdev_dbg (dev, +"%s: PFC per priority bit mask: 0x%x\n" , + __func__, pfc->pfc_en); }return ret; }
@@ -611,8 +611,8 @@ static int mlx5e_dcbnl_ieee_setmaxrate(struct net_device *netdev, }for (i =0 ; i <IEEE_8021QAZ_MAX_TCS ; i++) { -mlx5e_dbg (HW , priv,"%s: tc_%d <=> max_bw %d Gbps\n" , - __func__, i, max_bw_value[i]); +netdev_dbg (netdev,"%s: tc_%d <=> max_bw %d Gbps\n" , + __func__, i, max_bw_value[i]); }return mlx5_modify_port_ets_rate_limit (mdev, max_bw_value, max_bw_unit);
@@ -640,10 +640,10 @@ static u8 mlx5e_dcbnl_setall(struct net_device *netdev) ets.tc_rx_bw[i] = cee_cfg->pg_bw_pct[i]; ets.tc_tsa[i] =IEEE_8021QAZ_TSA_ETS ; ets.prio_tc[i] = cee_cfg->prio_to_pg_map[i]; -mlx5e_dbg (HW , priv, -"%s: Priority group %d: tx_bw %d, rx_bw %d, prio_tc %d\n" , - __func__, i, ets.tc_tx_bw[i], ets.tc_rx_bw[i], - ets.prio_tc[i]); +netdev_dbg (netdev, +"%s: Priority group %d: tx_bw %d, rx_bw %d, prio_tc %d\n" , + __func__, i, ets.tc_tx_bw[i], ets.tc_rx_bw[i], + ets.prio_tc[i]); } err =mlx5e_dbcnl_validate_ets (netdev, &ets, true);
drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +3 -15
@@ -1689,16 +1689,6 @@ static int mlx5e_set_fecparam(struct net_device *netdev,return 0 ; } -static u32 mlx5e_get_msglevel (struct net_device *dev) -{ -return ((struct mlx5e_priv *)netdev_priv (dev))->msglevel; -} - -static void mlx5e_set_msglevel (struct net_device *dev,u32 val) -{ - ((struct mlx5e_priv *)netdev_priv (dev))->msglevel = val; -} -static int mlx5e_set_phys_id (struct net_device *dev,enum ethtool_phys_id_state state) {
@@ -1952,9 +1942,9 @@ int mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool new_valif (err)return err; -mlx5e_dbg (DRV , priv,"MLX5E: RxCqeCmprss was turned %s\n" , -MLX5E_GET_PFLAG (&priv->channels.params, -MLX5E_PFLAG_RX_CQE_COMPRESS ) ?"ON" :"OFF" ); +netdev_dbg (priv->netdev,"MLX5E: RxCqeCmprss was turned %s\n" , +MLX5E_GET_PFLAG (&priv->channels.params, +MLX5E_PFLAG_RX_CQE_COMPRESS ) ?"ON" :"OFF" );return 0 ; }
@@ -2444,8 +2434,6 @@ const struct ethtool_ops mlx5e_ethtool_ops = { .get_priv_flags = mlx5e_get_priv_flags, .set_priv_flags = mlx5e_set_priv_flags, .self_test = mlx5e_self_test, - .get_msglevel = mlx5e_get_msglevel, - .set_msglevel = mlx5e_set_msglevel, .get_fec_stats = mlx5e_get_fec_stats, .get_fecparam = mlx5e_get_fecparam, .set_fecparam = mlx5e_set_fecparam,
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -3
@@ -2401,7 +2401,7 @@ static int mlx5e_channel_stats_alloc(struct mlx5e_priv *priv, int ix, int cpu)/* Asymmetric dynamic memory allocation. * Freed in mlx5e_priv_arrays_free, not on channel closure. */ -mlx5e_dbg (DRV , priv,"Creating channel stats %d\n" , ix); +netdev_dbg (priv->netdev,"Creating channel stats %d\n" , ix); priv->channel_stats[ix] =kvzalloc_node (sizeof (**priv->channel_stats),GFP_KERNEL ,cpu_to_node (cpu));if (!priv->channel_stats[ix])
@@ -2779,7 +2779,7 @@ int mlx5e_update_tx_netdev_queues(struct mlx5e_priv *priv)if (MLX5E_GET_PFLAG (&priv->channels.params,MLX5E_PFLAG_TX_PORT_TS )) num_txqs += ntc; -mlx5e_dbg (DRV , priv,"Setting num_txqs %d\n" , num_txqs); +netdev_dbg (priv->netdev,"Setting num_txqs %d\n" , num_txqs); err =netif_set_real_num_tx_queues (priv->netdev, num_txqs);if (err)netdev_warn (priv->netdev,"netif_set_real_num_tx_queues failed, %d\n" , err);
@@ -5585,7 +5585,6 @@ int mlx5e_priv_init(struct mlx5e_priv *priv,/* priv init */ priv->mdev = mdev; priv->netdev = netdev; - priv->msglevel =MLX5E_MSG_LEVEL ; priv->max_nch = nch; priv->max_opened_tc =1 ;
Signature · 1 line
2.40.1
PATCH 10/15 net/mlx5: Expose bits for local loopback counter
In reply to Saeed MahameedFrom: Or Har-Toov <[email protected]>
Add needed HW bits for querying local loopback counter and the HCA capability for it.
--- include/linux/mlx5/mlx5_ifc.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
include/linux/mlx5/mlx5_ifc.h +4 -2
@@ -1758,7 +1758,7 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 reserved_at_330[0x6 ];u8 pci_sync_for_fw_update_with_driver_unload[0x1 ];u8 vnic_env_cnt_steering_fail[0x1 ]; -u8 reserved_at_338[0x1 ]; +u8 vport_counter_local_loopback[0x1 ];u8 q_counter_aggregation[0x1 ];u8 q_counter_other_vport[0x1 ];u8 log_max_xrcd[0x5 ];
@@ -5190,7 +5190,9 @@ struct mlx5_ifc_query_vport_counter_out_bits {struct mlx5_ifc_traffic_counter_bits transmitted_eth_multicast; -u8 reserved_at_680[0xa00 ]; +struct mlx5_ifc_traffic_counter_bits local_loopback; + +u8 reserved_at_700[0x980 ]; };enum {
Signature · 1 line
2.40.1
PATCH 11/15 net/mlx5e: Add local loopback counter to vport stats
In reply to Saeed MahameedFrom: Or Har-Toov <[email protected]>
Add counter for number of unicast, multicast and broadcast packets/ octets that were loopback.
--- .../ethernet/mellanox/mlx5/counters.rst | 10 ++++++++ .../ethernet/mellanox/mlx5/core/en_stats.c | 23 ++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-)
Documentation/networking/device_drivers/ethernet/mellanox/mlx5/counters.rst +10 -0
@@ -797,6 +797,16 @@ Counters on the NIC port that is connected to a eSwitch. RoCE/UD/RC traffic) [#accel]_. - Acceleration + * - `vport_loopback_packets` + - Unicast, multicast and broadcast packets that were loop-back (received + and transmitted), IB/Eth [#accel]_. + - Acceleration + + * - `vport_loopback_bytes` + - Unicast, multicast and broadcast bytes that were loop-back (received + and transmitted), IB/Eth [#accel]_. + - Acceleration + * - `rx_steer_missed_packets` - Number of packets that was received by the NIC, however was discarded because it did not match any flow in the NIC flow table.
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +22 -1
@@ -748,11 +748,22 @@ static const struct counter_desc vport_stats_desc[] = { VPORT_COUNTER_OFF(transmitted_ib_multicast.octets) }, }; +static const struct counter_desc vport_loopback_stats_desc[] = { + {"vport_loopback_packets" , +VPORT_COUNTER_OFF (local_loopback.packets) }, + {"vport_loopback_bytes" , +VPORT_COUNTER_OFF (local_loopback.octets) }, +}; +#define NUM_VPORT_COUNTERS ARRAY_SIZE(vport_stats_desc) +#define NUM_VPORT_LOOPBACK_COUNTERS (dev) \ + (MLX5_CAP_GEN(dev, vport_counter_local_loopback) ? \ + ARRAY_SIZE(vport_loopback_stats_desc) : 0)static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS (vport) { -return NUM_VPORT_COUNTERS ; +return NUM_VPORT_COUNTERS + +NUM_VPORT_LOOPBACK_COUNTERS (priv->mdev); }static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS (vport)
@@ -761,6 +772,11 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(vport)for (i =0 ; i <NUM_VPORT_COUNTERS ; i++)strcpy (data + (idx++) *ETH_GSTRING_LEN , vport_stats_desc[i].format); + +for (i =0 ; i <NUM_VPORT_LOOPBACK_COUNTERS (priv->mdev); i++) +strcpy (data + (idx++) *ETH_GSTRING_LEN , + vport_loopback_stats_desc[i].format); +return idx; }
@@ -771,6 +787,11 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STATS(vport)for (i =0 ; i <NUM_VPORT_COUNTERS ; i++) data[idx++] =MLX5E_READ_CTR64_BE (priv->stats.vport.query_vport_out, vport_stats_desc, i); + +for (i =0 ; i <NUM_VPORT_LOOPBACK_COUNTERS (priv->mdev); i++) + data[idx++] =MLX5E_READ_CTR64_BE (priv->stats.vport.query_vport_out, + vport_loopback_stats_desc, i); +return idx; }
Signature · 1 line
2.40.1
PATCH 12/15 net/mlx5: Fix the macro for accessing EC VF vports
In reply to Saeed MahameedFrom: Daniel Jurgens <[email protected]>
The last value is not set correctly. This results in representors not being created for all EC VFs when the base value is higher than 0.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch.h | 1 + 1 file changed, 1 insertion(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +1 -0
@@ -668,6 +668,7 @@ void mlx5e_tc_clean_fdb_peer_flows(struct mlx5_eswitch *esw); index, \ vport, \ MLX5_CAP_GEN_2((esw->dev), ec_vf_vport_base), \ + MLX5_CAP_GEN_2((esw->dev), ec_vf_vport_base) +\ (last) - 1)struct mlx5_eswitch *mlx5_devlink_eswitch_get (struct devlink *devlink);
Signature · 1 line
2.40.1
PATCH 13/15 net/mlx5: DR, update query of HCA caps for EC VFs
In reply to Saeed MahameedFrom: Daniel Jurgens <[email protected]>
This change is needed to use EC VFs with metadata based steering.
There was an assumption that vport was equal to function ID. That's not the case for EC VF functions. Adjust to function ID and set the ec_vf_function bit accordingly.
--- drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h | 7 +++++++ drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c | 4 +++- drivers/net/ethernet/mellanox/mlx5/core/vport.c | 6 ------ 3 files changed, 10 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +7 -0
@@ -358,4 +358,11 @@ static inline bool mlx5_core_is_ec_vf_vport(const struct mlx5_core_dev *dev, u16 return (vport_num >= base_vport && vport_num < max_vport); } + +static inline int mlx5_vport_to_func_id (const struct mlx5_core_dev *dev,u16 vport,bool ec_vf_func) +{ +return ec_vf_func ? vport -mlx5_core_ec_vf_vport_base (dev) + : vport; +} +#endif /* __MLX5_CORE_H__ */
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c +3 -1
@@ -34,6 +34,7 @@ int mlx5dr_cmd_query_esw_vport_context(struct mlx5_core_dev *mdev,int mlx5dr_cmd_query_gvmi (struct mlx5_core_dev *mdev,bool other_vport,u16 vport_number,u16 *gvmi) { +bool ec_vf_func = other_vport ?mlx5_core_is_ec_vf_vport (mdev, vport_number) : false;u32 in[MLX5_ST_SZ_DW (query_hca_cap_in)] = {};int out_size;void *out;
@@ -46,7 +47,8 @@ int mlx5dr_cmd_query_gvmi(struct mlx5_core_dev *mdev, bool other_vport,MLX5_SET (query_hca_cap_in, in, opcode,MLX5_CMD_OP_QUERY_HCA_CAP );MLX5_SET (query_hca_cap_in, in, other_function, other_vport); -MLX5_SET (query_hca_cap_in, in, function_id, vport_number); +MLX5_SET (query_hca_cap_in, in, function_id,mlx5_vport_to_func_id (mdev, vport_number, ec_vf_func)); +MLX5_SET (query_hca_cap_in, in, ec_vf_function, ec_vf_func);MLX5_SET (query_hca_cap_in, in, op_mod,MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE <<1 |HCA_CAP_OPMOD_GET_CUR );
drivers/net/ethernet/mellanox/mlx5/core/vport.c +0 -6
@@ -1161,12 +1161,6 @@ u64 mlx5_query_nic_system_image_guid(struct mlx5_core_dev *mdev) }EXPORT_SYMBOL_GPL (mlx5_query_nic_system_image_guid); -static int mlx5_vport_to_func_id (const struct mlx5_core_dev *dev,u16 vport,bool ec_vf_func) -{ -return ec_vf_func ? vport -mlx5_core_ec_vf_vport_base (dev) - : vport; -} -int mlx5_vport_get_other_func_cap (struct mlx5_core_dev *dev,u16 vport,void *out,u16 opmod) {
Signature · 1 line
2.40.1
PATCH 14/15 net/mlx5: Add header file for events
In reply to Saeed MahameedFrom: Juhee Kang <[email protected]>
Separate the event API defined in the generic mlx5.h header into a dedicated header. And remove the TODO comment in commit 69c1280b1f3b ("net/mlx5: Device events, Use async events chain").
--- .../ethernet/mellanox/mlx5/core/en_stats.c | 2 +- .../net/ethernet/mellanox/mlx5/core/events.c | 2 +- .../net/ethernet/mellanox/mlx5/core/health.c | 1 + .../net/ethernet/mellanox/mlx5/core/lag/mp.c | 2 +- .../ethernet/mellanox/mlx5/core/lag/mpesw.c | 2 +- .../ethernet/mellanox/mlx5/core/lib/events.h | 40 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/lib/mlx5.h | 34 ---------------- 7 files changed, 45 insertions(+), 38 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/events.h
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +1 -1
@@ -30,7 +30,7 @@* SOFTWARE. */ -#include "lib/mlx5.h" +#include "lib/events.h" #include "en.h" #include "en_accel/ktls.h" #include "en_accel/en_accel.h"
drivers/net/ethernet/mellanox/mlx5/core/events.c +1 -1
@@ -5,7 +5,7 @@#include "mlx5_core.h" #include "lib/eq.h" -#include "lib/mlx5.h" +#include "lib/events.h" struct mlx5_event_nb {struct mlx5_nb nb;
drivers/net/ethernet/mellanox/mlx5/core/health.c +1 -0
@@ -39,6 +39,7 @@#include "mlx5_core.h" #include "lib/eq.h" #include "lib/mlx5.h" +#include "lib/events.h" #include "lib/pci_vsc.h" #include "lib/tout.h" #include "diag/fw_tracer.h"
drivers/net/ethernet/mellanox/mlx5/core/lag/mp.c +1 -1
@@ -7,7 +7,7 @@#include "lag/mp.h" #include "mlx5_core.h" #include "eswitch.h" -#include "lib/mlx5.h" +#include "lib/events.h" static bool __mlx5_lag_is_multipath (struct mlx5_lag *ldev) {
drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.c +1 -1
@@ -6,7 +6,7 @@#include "lag/lag.h" #include "eswitch.h" #include "esw/acl/ofld.h" -#include "lib/mlx5.h" +#include "lib/events.h" static void mlx5_mpesw_metadata_cleanup (struct mlx5_lag *ldev) {
drivers/net/ethernet/mellanox/mlx5/core/lib/events.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __LIB_EVENTS_H__ +#define __LIB_EVENTS_H__ + +#include "mlx5_core.h" + +#define PORT_MODULE_EVENT_MODULE_STATUS_MASK 0xF +#define PORT_MODULE_EVENT_ERROR_TYPE_MASK 0xF + +enum port_module_event_status_type { +MLX5_MODULE_STATUS_PLUGGED =0x1 , +MLX5_MODULE_STATUS_UNPLUGGED =0x2 , +MLX5_MODULE_STATUS_ERROR =0x3 , +MLX5_MODULE_STATUS_DISABLED =0x4 , +MLX5_MODULE_STATUS_NUM , +}; + +enum port_module_event_error_type { +MLX5_MODULE_EVENT_ERROR_POWER_BUDGET_EXCEEDED =0x0 , +MLX5_MODULE_EVENT_ERROR_LONG_RANGE_FOR_NON_MLNX =0x1 , +MLX5_MODULE_EVENT_ERROR_BUS_STUCK =0x2 , +MLX5_MODULE_EVENT_ERROR_NO_EEPROM_RETRY_TIMEOUT =0x3 , +MLX5_MODULE_EVENT_ERROR_ENFORCE_PART_NUMBER_LIST =0x4 , +MLX5_MODULE_EVENT_ERROR_UNKNOWN_IDENTIFIER =0x5 , +MLX5_MODULE_EVENT_ERROR_HIGH_TEMPERATURE =0x6 , +MLX5_MODULE_EVENT_ERROR_BAD_CABLE =0x7 , +MLX5_MODULE_EVENT_ERROR_PCIE_POWER_SLOT_EXCEEDED =0xc , +MLX5_MODULE_EVENT_ERROR_NUM , +}; + +struct mlx5_pme_stats { +u64 status_counters[MLX5_MODULE_STATUS_NUM ]; +u64 error_counters[MLX5_MODULE_EVENT_ERROR_NUM ]; +}; + +void mlx5_get_pme_stats (struct mlx5_core_dev *dev,struct mlx5_pme_stats *stats); +int mlx5_notifier_call_chain (struct mlx5_events *events,unsigned int event,void *data); +#endif
drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +0 -34
@@ -45,40 +45,6 @@ int mlx5_crdump_enable(struct mlx5_core_dev *dev);void mlx5_crdump_disable (struct mlx5_core_dev *dev);int mlx5_crdump_collect (struct mlx5_core_dev *dev,u32 *cr_data); -/* TODO move to lib/events.h */ - -#define PORT_MODULE_EVENT_MODULE_STATUS_MASK 0xF -#define PORT_MODULE_EVENT_ERROR_TYPE_MASK 0xF - -enum port_module_event_status_type { -MLX5_MODULE_STATUS_PLUGGED =0x1 , -MLX5_MODULE_STATUS_UNPLUGGED =0x2 , -MLX5_MODULE_STATUS_ERROR =0x3 , -MLX5_MODULE_STATUS_DISABLED =0x4 , -MLX5_MODULE_STATUS_NUM , -}; - -enum port_module_event_error_type { -MLX5_MODULE_EVENT_ERROR_POWER_BUDGET_EXCEEDED =0x0 , -MLX5_MODULE_EVENT_ERROR_LONG_RANGE_FOR_NON_MLNX =0x1 , -MLX5_MODULE_EVENT_ERROR_BUS_STUCK =0x2 , -MLX5_MODULE_EVENT_ERROR_NO_EEPROM_RETRY_TIMEOUT =0x3 , -MLX5_MODULE_EVENT_ERROR_ENFORCE_PART_NUMBER_LIST =0x4 , -MLX5_MODULE_EVENT_ERROR_UNKNOWN_IDENTIFIER =0x5 , -MLX5_MODULE_EVENT_ERROR_HIGH_TEMPERATURE =0x6 , -MLX5_MODULE_EVENT_ERROR_BAD_CABLE =0x7 , -MLX5_MODULE_EVENT_ERROR_PCIE_POWER_SLOT_EXCEEDED =0xc , -MLX5_MODULE_EVENT_ERROR_NUM , -}; - -struct mlx5_pme_stats { -u64 status_counters[MLX5_MODULE_STATUS_NUM ]; -u64 error_counters[MLX5_MODULE_EVENT_ERROR_NUM ]; -}; - -void mlx5_get_pme_stats (struct mlx5_core_dev *dev,struct mlx5_pme_stats *stats); -int mlx5_notifier_call_chain (struct mlx5_events *events,unsigned int event,void *data); -static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) {return devlink_net (priv_to_devlink (dev));
Signature · 1 line
2.40.1
PATCH 15/15 net/mlx5: Remove unused ecpu field from struct mlx5_sf_table
In reply to Saeed MahameedFrom: Jiri Pirko <[email protected]>
"ecpu" field in struct mlx5_sf_table is not used anywhere. Remove it.
--- drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c | 1 - 1 file changed, 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c +0 -1
@@ -28,7 +28,6 @@ struct mlx5_sf_table {struct mutex sf_state_lock;/* Serializes sf state among user cmds & vhca event handler. */ struct notifier_block esw_nb;struct notifier_block vhca_nb; - u8 ecpu: 1; }; static struct mlx5_sf *
Signature · 1 line
2.40.1