net/mlx5: Add switchdev mode support for Socket Direct single netdev, part 2/2
Subsystems (4)
Tariq Toukan
maintainerv3 cover net/mlx5: Add switchdev mode support for Socket Direct single netdev, part 2/2
Hi,
This is part 2. Find part 1 here:
https://lore.kernel.org/all/[email protected]/
This series enables Socket Direct single netdev to operate in switchdev mode with shared FDB. SD single netdev combines multiple PCI functions behind a single netdev interface. To support switchdev offloads, these functions must participate in virtual LAG (shared FDB).
Design
Rather than introducing a separate LAG instance for SD, this series integrates SD secondary devices into the existing LAG structure (priv.lag) created at probe time. Each lag_func entry carries a group_id field that identifies its SD group membership (0 means not part of any SD group). An xarray mark (XA_MARK_PORT) distinguishes physical port entries from SD secondaries, enabling a single unified iterator that filters by group:
- MLX5_LAG_FILTER_PORTS: iterate port-level entries only (existing behavior, used by bonding, FW LAG commands, v2p_map)
- MLX5_LAG_FILTER_ALL: iterate all devices including SD secondaries (used by MPESW shared FDB across all devices)
- specific group_id: iterate only devices in that SD group (used by per-group SD shared FDB operations)
Existing callers use mlx5_ldev_for_each() which maps to MLX5_LAG_FILTER_PORTS, preserving current behavior for non-SD configurations.
Lifecycle and ownership
The SD LAG lifecycle is tied to the SD group, not to bonding events:
1. At PCI probe, mlx5_lag_add_mdev() creates the LAG structure (priv.lag) for each LAG-capable PF. e.g.: SD primary devices
2. During mlx5_sd_init(), after the SD group is fully formed (primary and secondaries paired), sd_lag_init() registers the secondary devices into the primary's existing priv.lag by calling mlx5_ldev_add_mdev() with the SD group_id. The primary's lag_func also gets its group_id set. No separate LAG instance is created.
3. After all the devices in SD group transition to switchdev, mlx5_lag_shared_fdb_create() is invoked with the group_id to create a software-only shared FDB scoped to that SD group. This sets sd_fdb_active on all lag_func entries in the group. No FW LAG commands are issued since SD devices share the same physical port.
4. If MPESW (multi-port eswitch) is enabled on top of SD groups, the per-group SD shared FDB is torn down first, then MPESW shared FDB is created spanning all devices (ports + SD secondaries) using MLX5_LAG_FILTER_ALL. On MPESW disable, per-group SD shared FDB is restored.
5. On SD teardown (mlx5_sd_cleanup or device unbind), sd_lag_cleanup() removes secondaries from priv.lag and clears the primary's group_id. The LAG structure itself is not destroyed.
The sd_fdb_active flag is set on all lag_func entries in a group (not just the primary), so any device can detect the SD shared FDB state during lag_disable_change teardown without needing to look up peer entries.
SD shared FDB is a pure software construct -- unlike regular LAG modes (ROCE, SRIOV, MPESW), it does not issue FW create_lag/destroy_lag commands. The software vport LAG for SD is implemented via eswitch egress ACL bounce rules, managed by the IB layer through mlx5_eth_lag_init(). And the software LAG demux is implemented via steering rules that utilize new destination, VHCA_RX.
Patches
E-Switch preparation (patch 1):
- Skip uplink IB rep load for SD secondary devices
Devcom support (patches 2-3):
- Expose locked variant of send_event
- Add DEVCOM_CANT_FAIL for non-rollback events
SD core hardening (patches 4-6):
- Make primary/secondary role determination more robust
- Add L2 table silent mode query support
- Expand vport metadata for SD secondary devices
SD switchdev transition (patches 7-8):
- Support switchdev mode transition with shared FDB
- Notify SD on eswitch disable
LAG integration (patches 9-12):
- Store demux resources per master lag_func
- Disable both regular and SD LAG on lag_disable_change
- Introduce software vport LAG implementation
- Add MPESW over SD LAG support
Deferred init (patches 13-14):
- Tie rep load/unload to SD LAG state
- Defer vport metadata init until SD is ready
Enablement (patch 15):
- Enable SD over ECPF and allow switchdev transition
V3:
- Patch 4: Mark sd_devcom as ready only if next_secondary_idx + 1 == sd->host_buses.
- Patch 7: Set tx_root_silent explicitly in sd_cmd_set_secondary().
- Patch 13: Expand this patch to completely tie rep load/unload to SD LAG state.
- Patch 14: Take esw->state_lock when refreshing vports ACL.
V2:
https://lore.kernel.org/all/[email protected]/
V1:
https://lore.kernel.org/all/[email protected]/
Sashiko comments from both runs on the ML, and the internal run here:
- commit "net/mlx5: SD, make primary/secondary role determination more robust"
Quoted 3 lines
+
sd->primary= true;
+ err = mlx5_devcom_locked_send_event(devcom,SD_PRIMARY_SET,
+SD_PRIMARY_SET, dev);
sashiko.dev says:
What happens during failure rollback if a subsequent peer fails during registration?SD_PRIMARY_SET is passed as both the forward event and the rollback event. If the event handler fails for a peer, the devcom rollback loop invokes sd_handle_primary_set() again for the previously successful peers.
Since the handler receives the exact same arguments and logic, does it unconditionally repeat the assignment peer_sd->primary_dev = dev instead of clearing the state?
If sd_register() subsequently returns an error and the device unbinds and frees the dev structure, does this leave the successfully processed peers with dangling primary_dev pointers referencing the freed device?
[SD] on broadcast failure sd_register goes to err_devcom_unreg (sd.c:566) and never sets the comp ready. Every reader of primary_dev gates on mlx5_devcom_comp_is_ready() (sd.c:66), so the stale pointer is written and but never dereferenced. sd_register returns failure, so all is ok - no fix needed.
- commit "net/mlx5: SD, support switchdev mode transition with shared FDB"
Quoted 6 lines
Because
mlx5_sd_eswitch_mode_set()returns void, does ignoring this error
leave the secondary device in an inconsistent state?
If TX root reconfiguration fails, the execution aborts via goto unlock, butmlx5_devlink_eswitch_mode_set()will still report a successful transition to userspace.
[SD] This is by design-any SD switchdev related operations are best effort.
- commit "net/mlx5: LAG, store demux resources per master lag_func":
- this note was on sashiko.dev as well.
Quoted 18 lines
Can this lockless lookup lead to a use-after-free if the master device is
removed concurrently?mlx5_lag_dev_get_master_pf()internally usesmlx5_lag_pf(), which performs a
locklessxa_load()fromldev->pfs. If the master device is unbound or hot-removed concurrently,mlx5_ldev_remove_mdev()will remove the master'slag_funcfrom the XArray and immediately free it using a synchronous kfree(pf).
Since this path doesn't appear to holdldev->lockor an overarching lock that
protects the master device'slag_funclifecycle, could a race like this occur?
CPU 1 (Adding rule)master_pf=mlx5_lag_dev_get_master_pf(ldev, vport_dev);
CPU 2 (Master removal)mlx5_ldev_remove_mdev()
Quoted 8 lines
xa_erase(&ldev->pfs, idx); kfree(pf);CPU 1 (Resumes)
if (xa_load(&master_pf->lag_demux_rules, index)) Previously, these resources were stored directly in structmlx5_lag(ldev),
which is refcounted and remains alive as long as any device in the LAG (including the secondary device) still references it.
[SD] no, before ldev is removed, the IB device is destroyed, and with it the demux rules, group and table.
- commit "net/mlx5: LAG, introduce software vport LAG implementation"
- this note was on sashiko.dev as well
Quoted 7 lines
Does this code safely handle shared ACLs?
Theesw_slave_egress_create_resources()function correctly reuses an existingvport->egress.aclif one is present. However, this teardown function unconditionally destroys bothvport->egress.offloads.bounce_grpandvport->egress.aclwithout checking if thebounce_rulesxarray is empty. In contrast,esw_unset_master_egress_rule()handles this safely:
Quoted 8 lines
if (xa_empty(&vport->egress.offloads.bounce_rules)) { esw_acl_egress_ofld_cleanup(vport); xa_destroy(&vport->egress.offloads.bounce_rules);}
Could this unconditional destruction tear down an ACL that is still being actively used by other features, potentially leading to a use-after-free when other subsystems attempt to access their flow rules?
[SD] this is an ACL with a single rule, so it is ok.
Quoted 6 lines
(quoted diff) +6
-0+static u32 mlx5_lag_dev_group_id(struct mlx5_core_dev *dev) +{ + struct mlx5_lag *ldev = mlx5_lag_dev(dev); + struct lag_func *pf; + int i; +
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (!ldev) + return 0; + + mlx5_lag_for_each(i, 0, ldev, MLX5_LAG_FILTER_ALL) { + pf = mlx5_lag_pf(ldev, i); + if (pf->dev == dev) + return pf->sd_fdb_active ? pf->group_id : 0; + } + return 0; +}
sashiko.dev says:
In mlx5_lag_shared_fdb_destroy(), pf->sd_fdb_active is set to false for the target group before calling mlx5_lag_reload_ib_reps_from_locked() to unload the IB representors.
During the IB unload process, mlx5_eth_lag_cleanup() calls mlx5_cmd_destroy_vport_lag(), which queries mlx5_lag_dev_group_id(). Since sd_fdb_active was already cleared, will mlx5_lag_dev_group_id() evaluate to 0 here?
If it does, there might be a risk that mlx5_cmd_destroy_vport_lag() will pass 0 to mlx5_lag_destroy_vport_lag(), which treats 0 as MLX5_LAG_FILTER_ALL and unconditionally destroys the manager vports' egress ACLs for all independent SD groups on the device.
[SD] sd_fdb_active is cleared after, not before, the IB unload. mlx5_lag_remove_devices_filter() is removing the IB rep, and The mlx5_lag_reload_ib_reps_from_locked() call at the end of mlx5_lag_shared_fdb_destroy() is the re-add path, not an unload.
- commit "net/mlx5: LAG, add MPESW over SD LAG support"
Quoted 8 lines
Does this code introduce a use-after-free when iterating over
ldev->pfs? In the concurrent LAG teardown path, elements are removed from the XArray and synchronously freed via kfree(pf) (notkfree_rcu()), protected only byldev->lock. Sincexa_for_each()(used inmlx5_lag_pf_by_dev()) only holds the RCU read lock internally while retrieving the element and releases it before the loop body, calling this function locklessly could dereference freed memory when accessingPf->idx.
[SD] this API caller assume dev is a valid device not in teardown flow. If this break, the caller is broken as well.
Quoted 4 lines
Can this call create an invalid global FDB or perform destructive redundant operations?
Unlikemlx5_mpesw_teardown_sd_fdb()which filters PFs by checking if (!pf->sd_fdb_active), this loop iterates unconditionally.
Quoted 8 lines
For non-SD ports,
pf->group_idis 0, which instructsmlx5_lag_shared_fdb_create()to create a global shared FDB covering the entire LAG (MLX5_LAG_FILTER_ALL), resulting in a broken global LAG activation.
Additionally, for SD groups containing multiple PFs, the first call creates the FDB, but subsequent calls for other PFs in the same group will re-execute the creation process, improperly attempting to recreate the already active FDB.
[SD] no, the iterator is running only over the PORT devices and for each SD group there is only one such device, and when NIC is SD, all its PFs (devices) are SD.
Quoted 9 lines
(quoted diff) +8
-0+static void mlx5_mpesw_restore_sd_fdb(struct mlx5_lag *ldev) +{ + struct lag_func *pf; + int err, i; + + mlx5_ldev_for_each(i, 0, ldev) { + pf = mlx5_lag_pf(ldev, i); + err = mlx5_lag_shared_fdb_create(ldev, NULL, 0,
pf->group_id);
sashiko.dev says:
Does this accidentally activate global FW LAG for non-SD ports? This loop unconditionally calls mlx5_lag_shared_fdb_create() using pf->group_id. For standalone non-SD ports, pf->group_id is 0. Inside mlx5_lag_shared_fdb_create(), passing group_id = 0 sets the filter to MLX5_LAG_FILTER_ALL, which bypasses the SD-specific single-FDB path and incorrectly triggers a full global LAG activation via mlx5_activate_lag().
Should there be a check for pf->group_id != 0 or pf->sd_fdb_active here, similar to the check in mlx5_mpesw_teardown_sd_fdb()?
[SD] pf->sd_fdb_active is false here by definition-we are re-creating it. And this API is already gated by pf->group_id check. When MPESW is destroyed, all PFs have group ID (if this is SD NIC)
- commit "net/mlx5: SD, defer vport metadata init until SD is ready"
Review
Quoted 3 lines
(quoted diff) +3
-0+err_acl: + esw_offloads_metadata_uninit(esw); + return err;
sashiko.dev says:
If a vport update fails, this jumps to err_acl and calls esw_offloads_metadata_uninit().
Looking at esw_offloads_metadata_uninit(), it frees the default_metadata IDAs and zeros vport->default_metadata:
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:esw_offloads_vport_metadata_cleanup() {
...
mlx5_esw_match_metadata_free(esw, vport->default_metadata);
vport->default_metadata = 0;
}However, it does not reset vport->metadata or revert the hardware ACL rules for the vports that were already successfully updated. Will this leave the successfully updated vports with hardware rules actively matching against freed IDAs? If these IDAs are reallocated later, could multiple vports share the same metadata tag and cause packet misdirection?
[SD] no rep is loaded until SD shared fdb is created.
No reps -> no traffic -> the stale HW rules / freed IDAs can't misdirect anything.
- commit "Tie rep load/unload to SD LAG state",
was "net/mlx5: E-Switch, defer rep load while SD LAG is not active"
Quoted 9 lines
Does this rely on the InfiniBand representor probe to trigger the reload of the ethernet representors once the SD LAG becomes active? If the kernel is built without
CONFIG_MLX5_INFINIBANDor if the ib-rep fails to probe, the ib-rep driver is never registered. Consequently, themlx5_eswitch_reload_repswork is never queued.
Since there is no explicit call to reloadREP_ETHanywhere in the activation path, would the ethernet representors for the VF/SFs remain permanently unloaded? This seems like it would leave the vport representors completely non-functional for ethernet-only deployments.
[SD] This code isn't relevant if IB rep device doesn't probed.
- commit "net/mlx5: SD, defer vport metadata init until SD is ready"
Quoted 5 lines
Does this operation need to hold
pos->priv.eswitch->mode_lock? If this runs concurrently with a devlink command changing the peer device's mode back to legacy,esw_offloads_disable()could tear down the ACL tables and free offloads objects while this locklessly accesses and modifies metadata and ACLs. Could this lead to a Use-After-Free?
[SD] it won't. Peer E-switch and it vports are destroyed only after SD is cleanup. Switching to legacy don't destroy resources used in meta_date_init().
Shay Drory (15):
net/mlx5: E-Switch, skip uplink IB rep load for SD secondary devices net/mlx5: devcom, expose locked variant of send_event net/mlx5: devcom, add DEVCOM_CANT_FAIL for non-rollback events net/mlx5: SD, make primary/secondary role determination more robust net/mlx5: SD, add L2 table silent mode query support net/mlx5: SD, expend vport metadata for SD secondary devices net/mlx5: SD, support switchdev mode transition with shared FDB net/mlx5: E-Switch, notify SD on eswitch disable net/mlx5: LAG, store demux resources per master lag_func net/mlx5: LAG, disable both regular and SD LAG on lag_disable_change net/mlx5: LAG, introduce software vport LAG implementation net/mlx5: LAG, add MPESW over SD LAG support
net/mlx5: E-Switch, defer rep load while SD LAG is not active net/mlx5: SD, defer vport metadata init until SD is ready net/mlx5: SD, enable SD over ECPF and allow switchdev transition
.../net/ethernet/mellanox/mlx5/core/eswitch.c | 1 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 5 + .../mellanox/mlx5/core/eswitch_offloads.c | 275 +++++++++++- .../net/ethernet/mellanox/mlx5/core/fs_cmd.c | 21 + .../net/ethernet/mellanox/mlx5/core/fs_cmd.h | 2 + .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 179 ++++++-- .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 29 +- .../ethernet/mellanox/mlx5/core/lag/mpesw.c | 95 +++- .../ethernet/mellanox/mlx5/core/lag/mpesw.h | 4 + .../mellanox/mlx5/core/lag/shared_fdb.c | 74 +++- .../ethernet/mellanox/mlx5/core/lib/devcom.c | 36 +- .../ethernet/mellanox/mlx5/core/lib/devcom.h | 5 + .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 406 +++++++++++++++--- .../net/ethernet/mellanox/mlx5/core/lib/sd.h | 8 + 14 files changed, 1018 insertions(+), 122 deletions(-)
base-commit: 5855479abc796c3b5d7b2f2ca147d68fc56cae1f
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 1/15 net/mlx5: E-Switch, skip uplink IB rep load for SD secondary devices
From: Shay Drory <[email protected]>
SD secondary devices share the primary's uplink and do not have their own uplink representor. When reloading IB reps on secondary devices, skip the uplink and only load VF/SF vport IB reps.
--- .../mellanox/mlx5/core/eswitch_offloads.c | 25 ++++++++++++++++--- 1 file changed, 21 insertions(+), 4 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +21 -4
@@ -3643,11 +3643,19 @@ int mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)if (atomic_read (&rep->rep_data[REP_ETH ].state) !=REP_LOADED )return 0 ; - ret =__esw_offloads_load_rep (esw, rep,REP_IB ,NULL ); -if (ret) -return ret; +/* SD secondary devices share the primary's uplink and do not +* have their own uplink representor. Only load VF/SF vports. +*/ +if (mlx5_sd_is_primary (esw->dev)) { + ret =__esw_offloads_load_rep (esw, rep,REP_IB ,NULL ); +if (ret) +return ret; + }mlx5_esw_for_each_rep (esw, i, rep) { +if (!mlx5_sd_is_primary (esw->dev) && + rep->vport ==MLX5_VPORT_UPLINK ) +continue ;if (atomic_read (&rep->rep_data[REP_ETH ].state) ==REP_LOADED )__esw_offloads_load_rep (esw, rep,REP_IB ,NULL ); }
@@ -4586,14 +4594,23 @@ mlx5_eswitch_register_vport_reps_blocked(struct mlx5_eswitch *esw,static void mlx5_eswitch_reload_reps_blocked (struct mlx5_eswitch *esw) { +struct mlx5_eswitch_rep *uplink;struct mlx5_vport *vport; +bool newly_loaded;unsigned long i;if (esw->mode !=MLX5_ESWITCH_OFFLOADS )return ; -if (mlx5_esw_offloads_rep_load (esw,MLX5_VPORT_UPLINK )) + uplink =mlx5_eswitch_get_rep (esw,MLX5_VPORT_UPLINK ); +if (__esw_offloads_load_rep (esw, uplink,REP_ETH , &newly_loaded)) +return ; +if (mlx5_sd_is_primary (esw->dev) && +__esw_offloads_load_rep (esw, uplink,REP_IB ,NULL )) { +if (newly_loaded) +__esw_offloads_unload_rep (esw, uplink,REP_ETH );return ; + }mlx5_esw_for_each_vport (esw, i, vport) {if (!vport)
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 2/15 net/mlx5: devcom, expose locked variant of send_event
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Factor mlx5_devcom_send_event() into two functions:
- mlx5_devcom_locked_send_event(): performs the dispatch (and rollback) with comp->sem already held by the caller.
- mlx5_devcom_send_event(): unchanged wrapper that takes comp->sem, calls the locked variant, and releases it.
This lets callers bracket multiple event broadcasts under a single held write lock, eliminating the gap between consecutive dispatches where peer state could change.
Will be used by a downstream patch.
--- .../ethernet/mellanox/mlx5/core/lib/devcom.c | 29 ++++++++++++++----- .../ethernet/mellanox/mlx5/core/lib/devcom.h | 3 ++ 2 files changed, 25 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c +22 -7
@@ -287,9 +287,9 @@ int mlx5_devcom_comp_get_size(struct mlx5_devcom_comp_dev *devcom)return kref_read (&comp->ref); } -int mlx5_devcom_send_event (struct mlx5_devcom_comp_dev *devcom, -int event,int rollback_event, -void *event_data) +int mlx5_devcom_locked_send_event (struct mlx5_devcom_comp_dev *devcom, +int event,int rollback_event, +void *event_data) {struct mlx5_devcom_comp_dev *pos;struct mlx5_devcom_comp *comp;
@@ -299,8 +299,8 @@ int mlx5_devcom_send_event(struct mlx5_devcom_comp_dev *devcom,if (!devcom)return -ENODEV ; +lockdep_assert_held_write (&devcom->comp->sem); comp = devcom->comp; -down_write (&comp->sem);list_for_each_entry (pos, &comp->comp_dev_list_head, list) { data =rcu_dereference_protected (pos->data,lockdep_is_held (&comp->sem));
@@ -311,12 +311,11 @@ int mlx5_devcom_send_event(struct mlx5_devcom_comp_dev *devcom, } } -up_write (&comp->sem);return 0 ; rollback:if (list_entry_is_head (pos, &comp->comp_dev_list_head, list)) - goto out; +return err; pos =list_prev_entry (pos, list);list_for_each_entry_from_reverse (pos, &comp->comp_dev_list_head, list) { data =rcu_dereference_protected (pos->data,lockdep_is_held (&comp->sem));
@@ -324,7 +323,23 @@ int mlx5_devcom_send_event(struct mlx5_devcom_comp_dev *devcom,if (pos != devcom && data) comp->handler (rollback_event, data, event_data); } -out: +return err; +} + +int mlx5_devcom_send_event (struct mlx5_devcom_comp_dev *devcom, +int event,int rollback_event, +void *event_data) +{ +struct mlx5_devcom_comp *comp; +int err; + +if (!devcom) +return -ENODEV ; + + comp = devcom->comp; +down_write (&comp->sem); + err =mlx5_devcom_locked_send_event (devcom, event, rollback_event, + event_data);up_write (&comp->sem);return err; }
drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.h +3 -0
@@ -46,6 +46,9 @@ mlx5_devcom_register_component(struct mlx5_devcom_dev *devc, void *data);void mlx5_devcom_unregister_component (struct mlx5_devcom_comp_dev *devcom); +int mlx5_devcom_locked_send_event (struct mlx5_devcom_comp_dev *devcom, +int event,int rollback_event, +void *event_data);int mlx5_devcom_send_event (struct mlx5_devcom_comp_dev *devcom,int event,int rollback_event,void *event_data);
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 3/15 net/mlx5: devcom, add DEVCOM_CANT_FAIL for non-rollback events
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Some devcom events are not expected to fail. Rather than attempting a rollback that may not be meaningful, allow callers to pass DEVCOM_CANT_FAIL as the rollback_event to indicate that the event handler should not fail. If it does, emit a warning and stop propagating to further peers, but skip the rollback path.
--- drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c | 7 ++++++- drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.h | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c +6 -1
@@ -306,8 +306,13 @@ int mlx5_devcom_locked_send_event(struct mlx5_devcom_comp_dev *devcom,if (pos != devcom && data) { err = comp->handler (event, data, event_data); -if (err) +if (err && rollback_event !=DEVCOM_CANT_FAIL ) { goto rollback; + }else if (err && rollback_event ==DEVCOM_CANT_FAIL ) { +WARN_ONCE (1 ,"devcom component %d event %d failed: %d\n" , + comp->id, event, err); +return err; + } } }
drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.h +2 -0
@@ -46,6 +46,8 @@ mlx5_devcom_register_component(struct mlx5_devcom_dev *devc, void *data);void mlx5_devcom_unregister_component (struct mlx5_devcom_comp_dev *devcom); +#define DEVCOM_CANT_FAIL (INT_MAX) +int mlx5_devcom_locked_send_event (struct mlx5_devcom_comp_dev *devcom,int event,int rollback_event,void *event_data);
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 4/15 net/mlx5: SD, make primary/secondary role determination more robust
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Refactor SD group registration to use devcom event-driven role determination to ensure SD is marked as ready only after roles are fully assigned and the group state is consistent, making outside accessors, which will be added in downstream patches, safe to use without races.
The devcom events:
- SD_PRIMARY_SET event: each device compares bus numbers with peers to determine which should be primary
- SD_SECONDARIES_SET event: secondaries register themselves with the elected primary device
--- .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 138 +++++++++++++----- 1 file changed, 104 insertions(+), 34 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +104 -34
@@ -26,6 +26,8 @@ struct mlx5_sd {struct {/* primary */ struct mlx5_core_dev *secondaries[MLX5_SD_MAX_GROUP_SZ -1 ];struct mlx5_flow_table *tx_ft; +/* Next index for secondary registration */ +u8 next_secondary_idx; };struct {/* secondary */ struct mlx5_core_dev *primary_dev;
@@ -374,62 +376,128 @@ static void sd_lag_cleanup(struct mlx5_core_dev *dev)mutex_unlock (&ldev->lock); } +enum { +SD_PRIMARY_SET , +SD_SECONDARIES_SET , +}; + +static void sd_handle_primary_set (struct mlx5_core_dev *dev, +struct mlx5_core_dev *peer) +{ +struct mlx5_sd *peer_sd =mlx5_get_sd (peer); +struct mlx5_sd *sd =mlx5_get_sd (dev); +struct mlx5_core_dev *candidate; +struct mlx5_sd *candidate_sd; + +/* Peer is the device that being sent to all the other devices in the +* group. Hence, use peer to get the candidate device. +*/ + candidate = peer_sd->primary ? peer : peer_sd->primary_dev; + +if (dev->pdev->bus->number >= candidate->pdev->bus->number) +return ; + + candidate_sd =mlx5_get_sd (candidate); + + sd->primary = true; + candidate_sd->primary = false; + candidate_sd->primary_dev = dev; + peer_sd->primary = false; + peer_sd->primary_dev = dev; +} + +static void sd_handle_secondaries_set (struct mlx5_core_dev *dev, +struct mlx5_core_dev *peer) +{ +struct mlx5_sd *peer_sd =mlx5_get_sd (peer); +struct mlx5_sd *sd =mlx5_get_sd (dev); +u8 idx; + +/* Primary has nothing to register with itself. */ +if (sd->primary) +return ; + +/* dev is a secondary device, peer is the primary device. +* Secondary registers itself with the primary. +*/ + idx = peer_sd->next_secondary_idx++; + peer_sd->secondaries[idx] = dev; + sd->primary_dev = peer; +} + +static int mlx5_sd_devcom_event (int event,void *my_data,void *event_data) +{ +struct mlx5_core_dev *peer = event_data; +struct mlx5_core_dev *dev = my_data; + +switch (event) { +case SD_PRIMARY_SET : +sd_handle_primary_set (dev, peer); +break ; +case SD_SECONDARIES_SET : +sd_handle_secondaries_set (dev, peer); +break ; + } + +return 0 ; +} +static int sd_register (struct mlx5_core_dev *dev) { -struct mlx5_devcom_comp_dev *devcom, *pos;struct mlx5_devcom_match_attr attr = {}; -struct mlx5_core_dev *peer, *primary; -struct mlx5_sd *sd, *primary_sd; -int err, i; +struct mlx5_devcom_comp_dev *devcom; +struct mlx5_core_dev *primary; +struct mlx5_sd *primary_sd; +struct mlx5_sd *sd; +int err; sd =mlx5_get_sd (dev); attr.key.val = sd->group_id; attr.flags =MLX5_DEVCOM_MATCH_FLAGS_NS ; attr.net =mlx5_core_net (dev); - devcom =mlx5_devcom_register_component (dev->priv.devc,MLX5_DEVCOM_SD_GROUP , - &attr,NULL , dev); + devcom =mlx5_devcom_register_component (dev->priv.devc, +MLX5_DEVCOM_SD_GROUP , + &attr, mlx5_sd_devcom_event, + dev);if (!devcom)return -EINVAL ; sd->devcom = devcom; -if (mlx5_devcom_comp_get_size (devcom) != sd->host_buses) -return 0 ; -mlx5_devcom_comp_lock (devcom); -mlx5_devcom_comp_set_ready (devcom, true); -mlx5_devcom_comp_unlock (devcom); +if (mlx5_devcom_comp_get_size (devcom) != sd->host_buses || +mlx5_devcom_comp_is_ready (devcom)) + goto out; -if (!mlx5_devcom_for_each_peer_begin (devcom)) { - err = -ENODEV ; +/* Send SD_PRIMARY_SET event with this device. +* All peers will receive this event and compare to this device. +* The one with lowest bus number will be marked as primary. +*/ + sd->primary = true; + err =mlx5_devcom_locked_send_event (devcom,SD_PRIMARY_SET , +SD_PRIMARY_SET , dev); +if (err) goto err_devcom_unreg; - } - primary = dev; -mlx5_devcom_for_each_peer_entry (devcom, peer, pos) -if (peer->pdev->bus->number < primary->pdev->bus->number) - primary = peer; +/* Broadcast SD_SECONDARIES_SET. Each non-sender peer's handler runs; +* the primary's handler returns early so only secondaries register. +*/ + primary = sd->primary ? dev : sd->primary_dev; +if (!sd->primary) +sd_handle_secondaries_set (dev, primary); +mlx5_devcom_locked_send_event (devcom,SD_SECONDARIES_SET , +DEVCOM_CANT_FAIL , primary); primary_sd =mlx5_get_sd (primary); - primary_sd->primary = true; - i =0 ; -/* loop the secondaries */ -mlx5_devcom_for_each_peer_entry (primary_sd->devcom, peer, pos) { -struct mlx5_sd *peer_sd =mlx5_get_sd (peer); - - primary_sd->secondaries[i++] = peer; - peer_sd->primary = false; - peer_sd->primary_dev = primary; - } - -mlx5_devcom_for_each_peer_end (devcom); +if (primary_sd->next_secondary_idx +1 == sd->host_buses) +mlx5_devcom_comp_set_ready (devcom, true); +out: +mlx5_devcom_comp_unlock (devcom);return 0 ; err_devcom_unreg: -mlx5_devcom_comp_lock (sd->devcom); -mlx5_devcom_comp_set_ready (sd->devcom, false); -mlx5_devcom_comp_unlock (sd->devcom); -mlx5_devcom_unregister_component (sd->devcom); +mlx5_devcom_comp_unlock (devcom); +mlx5_devcom_unregister_component (devcom);return err; }
@@ -672,6 +740,7 @@ int mlx5_sd_init(struct mlx5_core_dev *dev) peer_sd->primary_dev =NULL ; } primary_sd->primary = false; + primary_sd->next_secondary_idx =0 ;mlx5_devcom_comp_set_ready (sd->devcom, false);mlx5_devcom_comp_unlock (sd->devcom);sd_unregister (dev);
@@ -719,6 +788,7 @@ void mlx5_sd_cleanup(struct mlx5_core_dev *dev) peer_sd->primary_dev =NULL ; } primary_sd->primary = false; + primary_sd->next_secondary_idx =0 ; out_ready_false:mlx5_devcom_comp_set_ready (sd->devcom, false); out_unlock:
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 5/15 net/mlx5: SD, add L2 table silent mode query support
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Add mlx5_fs_cmd_query_l2table_silent() to query the current silent mode state from firmware. This allows detecting if firmware has already put secondary devices into silent mode.
During SD group registration, query the silent mode of each device. If a device is already in silent mode (set by firmware), record this in the fw_silents_secondaries flag and use it to help determine the primary/secondary roles.
When fw_silents_secondaries is set, skip the driver-initiated silent mode set/unset operations since firmware manages this state. This handles configurations where firmware persistently silences secondary devices.
--- .../net/ethernet/mellanox/mlx5/core/fs_cmd.c | 21 ++++ .../net/ethernet/mellanox/mlx5/core/fs_cmd.h | 2 + .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 105 +++++++++++++++--- 3 files changed, 114 insertions(+), 14 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +21 -0
@@ -1217,3 +1217,24 @@ int mlx5_fs_cmd_set_tx_flow_table_root(struct mlx5_core_dev *dev, u32 ft_id, booreturn mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); } + +int mlx5_fs_cmd_query_l2table_silent (struct mlx5_core_dev *dev,u8 *silent_mode) +{ +u32 out[MLX5_ST_SZ_DW (query_l2_table_entry_out)] = {}; +u32 in[MLX5_ST_SZ_DW (query_l2_table_entry_in)] = {}; +int err; + +if (!MLX5_CAP_GEN (dev, silent_mode_query)) +return -EOPNOTSUPP ; + +MLX5_SET (query_l2_table_entry_in, in, opcode, +MLX5_CMD_OP_QUERY_L2_TABLE_ENTRY ); +MLX5_SET (query_l2_table_entry_in, in, silent_mode_query,1 ); + + err =mlx5_cmd_exec_inout (dev, query_l2_table_entry, in, out); +if (err) +return err; + + *silent_mode =MLX5_GET (query_l2_table_entry_out, out, silent_mode); +return 0 ; +}
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h +2 -0
@@ -124,6 +124,8 @@ const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void);int mlx5_fs_cmd_set_l2table_entry_silent (struct mlx5_core_dev *dev,u8 silent_mode);int mlx5_fs_cmd_set_tx_flow_table_root (struct mlx5_core_dev *dev,u32 ft_id,bool disconnect); +int mlx5_fs_cmd_query_l2table_silent (struct mlx5_core_dev *dev, +u8 *silent_mode);static inline bool mlx5_fs_cmd_is_fw_term_table (struct mlx5_flow_table *ft) {
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +91 -14
178 lines not on this page · raw message
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 6/15 net/mlx5: SD, expend vport metadata for SD secondary devices
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
In Socket Direct configurations the primary and secondary PFs share the same native_port_num. The eswitch vport metadata encodes pf_num in its upper bits to distinguish vports across PFs. Without SD-awareness, both PFs generate identical metadata, causing FDB rules to steer traffic to the wrong representor.
Add mlx5_sd_pf_num_get() which remaps the pf_num for SD devices. Use it so each PF in an SD group produces unique vport metadata.
--- .../mellanox/mlx5/core/eswitch_offloads.c | 6 +++--- .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 21 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/lib/sd.h | 1 + 3 files changed, 25 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3 -3
@@ -3472,12 +3472,12 @@ u32 mlx5_esw_match_metadata_alloc(struct mlx5_eswitch *esw)u32 vport_end_ida = (1 <<ESW_VPORT_BITS ) -1 ;/* Reserve 0xf for internal port offload */ u32 max_pf_num = (1 <<ESW_PFNUM_BITS ) -2 ; -u32 pf_num; +int pf_num;int id;/* Only 4 bits of pf_num */ - pf_num =mlx5_get_dev_index (esw->dev); -if (pf_num > max_pf_num) + pf_num =mlx5_sd_pf_num_get (esw->dev); +if (pf_num <0 || pf_num > max_pf_num)return 0 ;/* Metadata is 4 bits of PFNUM and 12 bits of unique id */
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +21 -0
@@ -85,6 +85,27 @@ bool mlx5_sd_is_primary(struct mlx5_core_dev *dev)return sd->primary; } +int mlx5_sd_pf_num_get (struct mlx5_core_dev *dev) +{ +struct mlx5_sd *sd =mlx5_get_sd (dev); +int pf_num =mlx5_get_dev_index (dev); +struct mlx5_core_dev *pos; +int i; + +if (!sd) +return pf_num; + +mlx5_devcom_comp_assert_locked (sd->devcom); +if (!mlx5_devcom_comp_is_ready (sd->devcom)) +return -ENODEV ; + +mlx5_sd_for_each_dev (i,mlx5_sd_get_primary (dev), pos) +if (pos == dev) +break ; + +return pf_num * sd->host_buses + i; +} +struct mlx5_core_dev *mlx5_sd_primary_get_peer (struct mlx5_core_dev *primary,int idx) {
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.h +1 -0
@@ -12,6 +12,7 @@ struct mlx5_sd;struct mlx5_core_dev *mlx5_sd_get_primary (struct mlx5_core_dev *dev);bool mlx5_sd_is_primary (struct mlx5_core_dev *dev); +int mlx5_sd_pf_num_get (struct mlx5_core_dev *dev);struct mlx5_core_dev *mlx5_sd_primary_get_peer (struct mlx5_core_dev *primary,int idx);int mlx5_sd_ch_ix_get_dev_ix (struct mlx5_core_dev *dev,int ch_ix);int mlx5_sd_ch_ix_get_vec_ix (struct mlx5_core_dev *dev,int ch_ix);
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 7/15 net/mlx5: SD, support switchdev mode transition with shared FDB
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
When the eswitch transitions, propagate the change to SD: secondaries get their TX flow table root reconfigured for the new mode, and when all group devices move to switchdev, the per-group shared FDB is activated.
Shared FDB activation is best-effort - failure does not block the eswitch transition; the next transition retries.
Note: the existing mlx5_get_sd() guard that blocks switchdev for SD devices is intentionally retained. It will be removed once all supporting patches are in place.
--- .../mellanox/mlx5/core/eswitch_offloads.c | 12 +- .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 138 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/lib/sd.h | 7 + 3 files changed, 154 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +11 -1
@@ -46,6 +46,7 @@#include "fs_core.h" #include "lib/mlx5.h" #include "lib/devcom.h" +#include "lib/sd.h" #include "lib/eq.h" #include "lib/fs_chains.h" #include "en_tc.h"
@@ -3164,6 +3165,9 @@ static void esw_unset_master_egress_rule(struct mlx5_core_dev *dev, vport =mlx5_eswitch_get_vport (dev->priv.eswitch, dev->priv.eswitch->manager_vport); +if (!vport->egress.acl) +return ; +esw_acl_egress_ofld_bounce_rule_destroy (vport,MLX5_CAP_GEN (slave_dev, vhca_id));if (xa_empty (&vport->egress.offloads.bounce_rules)) {
@@ -3182,6 +3186,9 @@ int mlx5_eswitch_offloads_single_fdb_add_one(struct mlx5_eswitch *master_esw,if (err)return err; +if (!mlx5_sd_is_primary (slave_esw->dev)) +return 0 ; + err =esw_set_master_egress_rule (master_esw->dev, slave_esw->dev, max_slaves);if (err)
@@ -3401,7 +3408,7 @@ void mlx5_esw_offloads_devcom_init(struct mlx5_eswitch *esw,return ;if ((MLX5_VPORT_MANAGER (esw->dev) ||mlx5_core_is_ecpf_esw_manager (esw->dev)) && - !mlx5_lag_is_supported (esw->dev)) + (!mlx5_lag_is_supported (esw->dev) && !mlx5_get_sd (esw->dev)))return ;xa_init (&esw->paired);
@@ -4306,6 +4313,9 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode,mlx5_esw_unlock (esw); enable_lag:mlx5_lag_enable_change (esw->dev); +/* Shared FDB activation is creating LAG which is changing reps. */ +if (!err) +mlx5_sd_eswitch_mode_set (esw->dev, mlx5_mode);return err; }
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +136 -2
176 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.h +7 -0
@@ -45,6 +45,13 @@ mlx5_sd_get_devcom(struct mlx5_core_dev *dev) }#endif +#ifdef CONFIG_MLX5_ESWITCH +void mlx5_sd_eswitch_mode_set (struct mlx5_core_dev *dev,u16 mlx5_mode); +#else +static inline void +mlx5_sd_eswitch_mode_set (struct mlx5_core_dev *dev,u16 mlx5_mode) {return ; } +#endif +#define mlx5_sd_for_each_dev_from_to (i, primary, ix_from, to, pos) \ for (i = ix_from; \ (pos = mlx5_sd_primary_get_peer(primary, i)) && pos != (to); i++)
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 8/15 net/mlx5: E-Switch, notify SD on eswitch disable
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
When eswitch is disabled, notify the SD layer so it can clean up SD-specific resources such as the TX flow table root configuration on secondary devices.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 1 + 1 file changed, 1 insertion(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +1 -0
@@ -2072,6 +2072,7 @@ void mlx5_eswitch_disable(struct mlx5_eswitch *esw)mlx5_esw_reps_unblock (esw); esw->mode =MLX5_ESWITCH_LEGACY ; +mlx5_sd_eswitch_mode_set (esw->dev,MLX5_ESWITCH_LEGACY );mlx5_lag_enable_change (esw->dev); }
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 9/15 net/mlx5: LAG, store demux resources per master lag_func
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
The lag demux resources (flow table, flow group, and rules xarray) are stored on the shared ldev. With Socket Direct, multiple SD groups each create their own demux FT/FG during their master's IB device initialization. Since they all write to the same ldev fields, the second group's init overwrites the first group's pointers, leaking the first group's FT/FG.
During teardown, the cleanup uses the overwritten pointers, destroying the wrong group's resources and leaving leaked flow tables in the LAG namespace. These leaked tables can interfere with subsequently created demux tables.
Move the demux resources from the shared ldev to per-master lag_func instances. Each master device now owns its own independent demux state. The rule_add and rule_del helpers look up the appropriate master's lag_func via the existing filter/group infrastructure.
--- .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 95 +++++++++++++------ .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 7 +- 2 files changed, 68 insertions(+), 34 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +64 -31
@@ -1590,7 +1590,7 @@ struct mlx5_devcom_comp_dev *mlx5_lag_get_devcom_comp(struct mlx5_lag *ldev)static int mlx5_lag_demux_ft_fg_init (struct mlx5_core_dev *dev,struct mlx5_flow_table_attr *ft_attr, -struct mlx5_lag *ldev) +struct lag_func *pf) {#ifdef CONFIG_MLX5_ESWITCH struct mlx5_flow_namespace *ns;
@@ -1601,20 +1601,20 @@ static int mlx5_lag_demux_ft_fg_init(struct mlx5_core_dev *dev,if (!ns)return 0 ; - ldev->lag_demux_ft =mlx5_create_flow_table (ns, ft_attr); -if (IS_ERR (ldev->lag_demux_ft)) -return PTR_ERR (ldev->lag_demux_ft); + pf->lag_demux_ft =mlx5_create_flow_table (ns, ft_attr); +if (IS_ERR (pf->lag_demux_ft)) +return PTR_ERR (pf->lag_demux_ft); fg =mlx5_esw_lag_demux_fg_create (dev->priv.eswitch, - ldev->lag_demux_ft); + pf->lag_demux_ft);if (IS_ERR (fg)) { err =PTR_ERR (fg); -mlx5_destroy_flow_table (ldev->lag_demux_ft); - ldev->lag_demux_ft =NULL ; +mlx5_destroy_flow_table (pf->lag_demux_ft); + pf->lag_demux_ft =NULL ;return err; } - ldev->lag_demux_fg = fg; + pf->lag_demux_fg = fg;return 0 ;#else return -EOPNOTSUPP ;
@@ -1623,7 +1623,7 @@ static int mlx5_lag_demux_ft_fg_init(struct mlx5_core_dev *dev,static int mlx5_lag_demux_fw_init (struct mlx5_core_dev *dev,struct mlx5_flow_table_attr *ft_attr, -struct mlx5_lag *ldev) +struct lag_func *pf) {struct mlx5_flow_namespace *ns;int err;
@@ -1632,12 +1632,12 @@ static int mlx5_lag_demux_fw_init(struct mlx5_core_dev *dev,if (!ns)return 0 ; - ldev->lag_demux_fg =NULL ; + pf->lag_demux_fg =NULL ; ft_attr->max_fte =1 ; - ldev->lag_demux_ft =mlx5_create_lag_demux_flow_table (ns, ft_attr); -if (IS_ERR (ldev->lag_demux_ft)) { - err =PTR_ERR (ldev->lag_demux_ft); - ldev->lag_demux_ft =NULL ; + pf->lag_demux_ft =mlx5_create_lag_demux_flow_table (ns, ft_attr); +if (IS_ERR (pf->lag_demux_ft)) { + err =PTR_ERR (pf->lag_demux_ft); + pf->lag_demux_ft =NULL ;return err; }
@@ -1648,6 +1648,7 @@ int mlx5_lag_demux_init(struct mlx5_core_dev *dev, struct mlx5_flow_table_attr *ft_attr) {struct mlx5_lag *ldev; +struct lag_func *pf;if (!ft_attr)return -EINVAL ;
@@ -1656,12 +1657,16 @@ int mlx5_lag_demux_init(struct mlx5_core_dev *dev,if (!ldev)return -ENODEV ; -xa_init (&ldev->lag_demux_rules); + pf =mlx5_lag_pf_by_dev (ldev, dev); +if (!pf) +return -ENODEV ; + +xa_init (&pf->lag_demux_rules);if (mlx5_get_sd (dev)) -return mlx5_lag_demux_ft_fg_init (dev, ft_attr, ldev); +return mlx5_lag_demux_ft_fg_init (dev, ft_attr, pf); -return mlx5_lag_demux_fw_init (dev, ft_attr, ldev); +return mlx5_lag_demux_fw_init (dev, ft_attr, pf); }EXPORT_SYMBOL (mlx5_lag_demux_init);
@@ -1670,40 +1675,63 @@ void mlx5_lag_demux_cleanup(struct mlx5_core_dev *dev)struct mlx5_flow_handle *rule;struct mlx5_lag *ldev;unsigned long vport_num; +struct lag_func *pf; ldev =mlx5_lag_dev (dev);if (!ldev)return ; -xa_for_each (&ldev->lag_demux_rules, vport_num, rule) + pf =mlx5_lag_pf_by_dev (ldev, dev); +if (!pf) +return ; + +xa_for_each (&pf->lag_demux_rules, vport_num, rule)mlx5_del_flow_rules (rule); -xa_destroy (&ldev->lag_demux_rules); +xa_destroy (&pf->lag_demux_rules); -if (ldev->lag_demux_fg) -mlx5_destroy_flow_group (ldev->lag_demux_fg); -if (ldev->lag_demux_ft) -mlx5_destroy_flow_table (ldev->lag_demux_ft); - ldev->lag_demux_fg =NULL ; - ldev->lag_demux_ft =NULL ; +if (pf->lag_demux_fg) +mlx5_destroy_flow_group (pf->lag_demux_fg); +if (pf->lag_demux_ft) +mlx5_destroy_flow_table (pf->lag_demux_ft); + pf->lag_demux_fg =NULL ; + pf->lag_demux_ft =NULL ; }EXPORT_SYMBOL (mlx5_lag_demux_cleanup); +static struct lag_func *mlx5_lag_dev_get_master_pf (struct mlx5_lag *ldev, +struct mlx5_core_dev *dev) +{ +u32 filter =mlx5_lag_get_filter (ldev, dev); +int idx; + + idx =mlx5_lag_get_dev_index_by_seq_filter (ldev,MLX5_LAG_P1 , filter); +if (idx <0 ) +return NULL ; + +return mlx5_lag_pf (ldev, idx); +} +int mlx5_lag_demux_rule_add (struct mlx5_core_dev *vport_dev,u16 vport_num,int index) {struct mlx5_flow_handle *rule; +struct lag_func *master;struct mlx5_lag *ldev;int err; ldev =mlx5_lag_dev (vport_dev); -if (!ldev || !ldev->lag_demux_fg) +if (!ldev)return 0 ; -if (xa_load (&ldev->lag_demux_rules, index)) + master =mlx5_lag_dev_get_master_pf (ldev, vport_dev); +if (!master || !master->lag_demux_fg) +return 0 ; + +if (xa_load (&master->lag_demux_rules, index))return 0 ; rule =mlx5_esw_lag_demux_rule_create (vport_dev->priv.eswitch, - vport_num, ldev->lag_demux_ft); + vport_num, master->lag_demux_ft);if (IS_ERR (rule)) { err =PTR_ERR (rule); mlx5_core_warn(vport_dev,
@@ -1712,7 +1740,7 @@ int mlx5_lag_demux_rule_add(struct mlx5_core_dev *vport_dev, u16 vport_num,return err; } - err =xa_err (xa_store (&ldev->lag_demux_rules, index, rule, + err =xa_err (xa_store (&master->lag_demux_rules, index, rule,GFP_KERNEL ));if (err) {mlx5_del_flow_rules (rule);
@@ -1728,13 +1756,18 @@ EXPORT_SYMBOL(mlx5_lag_demux_rule_add);void mlx5_lag_demux_rule_del (struct mlx5_core_dev *dev,int index) {struct mlx5_flow_handle *rule; +struct lag_func *master_pf;struct mlx5_lag *ldev; ldev =mlx5_lag_dev (dev); -if (!ldev || !ldev->lag_demux_fg) +if (!ldev) +return ; + + master_pf =mlx5_lag_dev_get_master_pf (ldev, dev); +if (!master_pf || !master_pf->lag_demux_fg)return ; - rule =xa_erase (&ldev->lag_demux_rules, index); + rule =xa_erase (&master_pf->lag_demux_rules, index);if (rule)mlx5_del_flow_rules (rule); }
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h +4 -3
@@ -59,6 +59,10 @@ struct lag_func {struct mlx5_nb port_change_nb;u32 group_id;/* SD group ID, 0 = not SD */ bool sd_fdb_active;/* set on all SD group members */ +/* Lag demux resources - only populated on master devices */ +struct mlx5_flow_table *lag_demux_ft; +struct mlx5_flow_group *lag_demux_fg; +struct xarray lag_demux_rules; };/* Used for collection of netdev event info. */
@@ -95,9 +99,6 @@ struct mlx5_lag {/* Protect lag fields/state changes */ struct mutex lock;struct lag_mpesw lag_mpesw; -struct mlx5_flow_table *lag_demux_ft; -struct mlx5_flow_group *lag_demux_fg; -struct xarray lag_demux_rules; }; static inline struct mlx5_lag *
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 10/15 net/mlx5: LAG, disable both regular and SD LAG on lag_disable_change
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Extend mlx5_lag_disable_change() to properly disable both regular LAG and SD LAG when requested. Each LAG type uses its own devcom component for locking.
Use mlx5_sd_get_devcom() helper to retrieve the SD devcom component, needed for proper locking when disabling SD LAG.
--- .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +27 -2
@@ -2494,13 +2494,22 @@ EXPORT_SYMBOL(mlx5_lag_is_shared_fdb);void mlx5_lag_disable_change (struct mlx5_core_dev *dev) { +struct mlx5_devcom_comp_dev *sd_devcom =mlx5_sd_get_devcom (dev); +struct mlx5_core_dev *primary = dev;struct mlx5_lag *ldev; +struct lag_func *pf; +int i; ldev =mlx5_lag_dev (dev);if (!ldev)return ; -mlx5_devcom_comp_lock (dev->priv.hca_devcom_comp); +if (sd_devcom) { +mlx5_devcom_comp_lock (sd_devcom); + primary =mlx5_sd_get_primary (dev) ?: dev; +mlx5_devcom_comp_unlock (sd_devcom); + } +mlx5_devcom_comp_lock (primary->priv.hca_devcom_comp);mutex_lock (&ldev->lock); ldev->mode_changes_in_progress++;
@@ -2512,7 +2521,23 @@ void mlx5_lag_disable_change(struct mlx5_core_dev *dev) }mutex_unlock (&ldev->lock); -mlx5_devcom_comp_unlock (dev->priv.hca_devcom_comp); +mlx5_devcom_comp_unlock (primary->priv.hca_devcom_comp); + +if (!sd_devcom) +return ; + +/* Teardown SD shared FDB for this device's group if active */ +mlx5_devcom_comp_lock (sd_devcom); +mutex_lock (&ldev->lock); +mlx5_lag_for_each (i,0 , ldev,MLX5_LAG_FILTER_ALL ) { + pf =mlx5_lag_pf (ldev, i); +if (pf->dev == dev && pf->sd_fdb_active) { +mlx5_lag_shared_fdb_destroy (ldev, pf->group_id); +break ; + } + } +mutex_unlock (&ldev->lock); +mlx5_devcom_comp_unlock (sd_devcom); }void mlx5_lag_enable_change (struct mlx5_core_dev *dev)
Signature · 1 line
2.44.0
Re: [PATCH net-next V3 10/15] net/mlx5: LAG, disable both regular and SD LAG on lag_disable_change
Quoted 56 lines · Tariq Toukan
From: Shay Drory <[email protected]>
Extend
mlx5_lag_disable_change()to properly disable both regular LAG and SD LAG when requested. Each LAG type uses its own devcom component for locking.Use
mlx5_sd_get_devcom()helper to retrieve the SD devcom component, needed for proper locking when disabling SD LAG.Signed-off-by Shay Drory <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Tariq Toukan <[email protected]>--- .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +17
-2@@ -2494,13 +2494,22 @@ EXPORT_SYMBOL(mlx5_lag_is_shared_fdb);void mlx5_lag_disable_change (struct mlx5_core_dev *dev) { +struct mlx5_devcom_comp_dev *sd_devcom =mlx5_sd_get_devcom (dev); +struct mlx5_core_dev *primary = dev;struct mlx5_lag *ldev; +struct lag_func *pf; +int i; ldev =mlx5_lag_dev (dev);if (!ldev)return ; -mlx5_devcom_comp_lock (dev->priv.hca_devcom_comp); +if (sd_devcom) { +mlx5_devcom_comp_lock (sd_devcom); + primary =mlx5_sd_get_primary (dev) ?: dev; +mlx5_devcom_comp_unlock (sd_devcom); + } +mlx5_devcom_comp_lock (primary->priv.hca_devcom_comp);mutex_lock (&ldev->lock); ldev->mode_changes_in_progress++;@@ -2512,7 +2521,23 @@ void mlx5_lag_disable_change(struct mlx5_core_dev *dev) }mutex_unlock (&ldev->lock); -mlx5_devcom_comp_unlock (dev->priv.hca_devcom_comp); +mlx5_devcom_comp_unlock (primary->priv.hca_devcom_comp); + +if (!sd_devcom) +return ; + +/* Teardown SD shared FDB for this device's group if active */ +mlx5_devcom_comp_lock (sd_devcom);
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c void mlx5_lag_disable_change(struct mlx5_core_dev *dev) · line 2540
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +10
-0+mutex_lock (&ldev->lock); +mlx5_lag_for_each (i,0 , ldev,MLX5_LAG_FILTER_ALL ) { + pf =mlx5_lag_pf (ldev, i); +if (pf->dev == dev && pf->sd_fdb_active) { +mlx5_lag_shared_fdb_destroy (ldev, pf->group_id); +break ; + } + } +mutex_unlock (&ldev->lock); +mlx5_devcom_comp_unlock (sd_devcom);
sashiko.dev says:
Does holding the sd_devcom lock while calling mlx5_lag_shared_fdb_destroy() introduce an AB-BA deadlock with auxiliary device probe?
This path acquires sd_devcom, and mlx5_lag_shared_fdb_destroy() eventually reaches mlx5_rescan_drivers_locked() calling device_del() on auxiliary devices, which attempts to acquire device_lock(&adev->dev). This gives us:sd_devcom -> device_lock()
However, during auxiliary device probe, the driver core holds device_lock(&adev->dev) before calling mlx5e_probe(). mlx5e_probe() then calls mlx5_sd_get_adev() which acquires sd_devcom, giving us the reverse:device_lock() -> sd_devcom
Could the teardown be performed without holding the sd_devcom lock here to prevent this deadlock?
[SD] No — the teardown's device_del runs on the IB aux devices, while the device_lock held during probe is the ETH aux device (mlx5e_probe); different struct devices, so no AB-BA
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +0
-0}void mlx5_lag_enable_change (struct mlx5_core_dev *dev)
Tariq Toukan
maintainerPATCH v3 11/15 net/mlx5: LAG, introduce software vport LAG implementation
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
SD LAG is a virtual LAG without hardware LAG support, so it cannot use the firmware vport LAG commands. Implement a software-based vport LAG using egress ACL bounce rules.
Add esw_set_slave_egress_rule() to create an egress ACL rule on the slave's manager vport that bounces traffic to the master's manager vport. This achieves the same traffic steering as hardware vport LAG.
Redirect mlx5_cmd_create_vport_lag() and mlx5_cmd_destroy_vport_lag() to the software implementation when operating in SD LAG mode. In addition, adjust lag_demux creation to check SD LAG mode as well.
--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 4 + .../mellanox/mlx5/core/eswitch_offloads.c | 142 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 49 +++++- .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 14 ++ .../mellanox/mlx5/core/lag/shared_fdb.c | 74 ++++++++- 5 files changed, 280 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +4 -0
@@ -950,6 +950,10 @@ void esw_vport_change_handle_locked(struct mlx5_vport *vport);bool mlx5_esw_offloads_controller_valid (const struct mlx5_eswitch *esw,u32 controller); +int mlx5_eswitch_offloads_vport_lag_add_one (struct mlx5_eswitch *master_esw, +struct mlx5_eswitch *slave_esw); +void mlx5_eswitch_offloads_vport_lag_del_one (struct mlx5_eswitch *master_esw, +struct mlx5_eswitch *slave_esw);int mlx5_eswitch_offloads_single_fdb_add_one (struct mlx5_eswitch *master_esw,struct mlx5_eswitch *slave_esw,int max_slaves); void mlx5_eswitch_offloads_single_fdb_del_one(struct mlx5_eswitch *master_esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +142 -0
154 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +48 -1
70 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h +14 -0
@@ -175,6 +175,8 @@ int mlx5_lag_shared_fdb_create(struct mlx5_lag *ldev,enum mlx5_lag_mode mode, u32 group_id);void mlx5_lag_shared_fdb_destroy (struct mlx5_lag *ldev,u32 group_id); +int mlx5_lag_create_vport_lag (struct mlx5_lag *ldev,u32 group_id); +int mlx5_lag_destroy_vport_lag (struct mlx5_lag *ldev,u32 group_id);int mlx5_lag_create_single_fdb (struct mlx5_lag *ldev);void mlx5_lag_destroy_single_fdb (struct mlx5_lag *ldev);bool mlx5_lag_shared_fdb_supported (struct mlx5_lag *ldev);
@@ -191,6 +193,18 @@ static inline int mlx5_lag_shared_fdb_create(struct mlx5_lag *ldev,static inline void mlx5_lag_shared_fdb_destroy (struct mlx5_lag *ldev,u32 group_id) {} +static inline int mlx5_lag_create_vport_lag (struct mlx5_lag *ldev, +u32 group_id) +{ +return -EOPNOTSUPP ; +} + +static inline int mlx5_lag_destroy_vport_lag (struct mlx5_lag *ldev, +u32 group_id) +{ +return -EOPNOTSUPP ; +} +static inline int mlx5_lag_create_single_fdb (struct mlx5_lag *ldev) {return -EOPNOTSUPP ;
drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c +72 -2
92 lines not on this page · raw message
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 12/15 net/mlx5: LAG, add MPESW over SD LAG support
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Enable MPESW LAG creation over SD LAG members, forming a composite LAG hierarchy. This allows bonding multiple SD groups together under a single MPESW configuration with shared FDB.
When enabling composite MPESW, the individual SD LAG shared FDB configurations are temporarily torn down and recreated when the composite LAG is disabled.
--- .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 6 ++ .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 8 ++ .../ethernet/mellanox/mlx5/core/lag/mpesw.c | 95 +++++++++++++++++-- .../ethernet/mellanox/mlx5/core/lag/mpesw.h | 4 + 4 files changed, 105 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +6 -0
@@ -2545,6 +2545,7 @@ void mlx5_lag_disable_change(struct mlx5_core_dev *dev)struct mlx5_core_dev *primary = dev;struct mlx5_lag *ldev;struct lag_func *pf; +bool mpesw;int i; ldev =mlx5_lag_dev (dev);
@@ -2557,6 +2558,9 @@ void mlx5_lag_disable_change(struct mlx5_core_dev *dev)mlx5_devcom_comp_unlock (sd_devcom); }mlx5_devcom_comp_lock (primary->priv.hca_devcom_comp); + mpesw = ldev->mode ==MLX5_LAG_MODE_MPESW ; +if (mpesw) +mlx5_mpesw_sd_devcoms_lock (ldev);mutex_lock (&ldev->lock); ldev->mode_changes_in_progress++;
@@ -2568,6 +2572,8 @@ void mlx5_lag_disable_change(struct mlx5_core_dev *dev) }mutex_unlock (&ldev->lock); +if (mpesw) +mlx5_mpesw_sd_devcoms_unlock (ldev);mlx5_devcom_comp_unlock (primary->priv.hca_devcom_comp);if (!sd_devcom)
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h +8 -0
@@ -157,6 +157,14 @@ __mlx5_lag_is_sd(struct mlx5_lag *ldev, struct mlx5_core_dev *dev)return pf && pf->group_id !=0 ; } +static inline bool +__mlx5_lag_dev_is_port (struct mlx5_lag *ldev,struct mlx5_core_dev *dev) +{ +struct lag_func *pf =mlx5_lag_pf_by_dev (ldev, dev); + +return pf &&xa_get_mark (&ldev->pfs, pf->idx,MLX5_LAG_XA_MARK_PORT ); +} +static inline bool __mlx5_lag_is_active (struct mlx5_lag *ldev) {
drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.c +87 -8
152 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.h +4 -0
@@ -33,8 +33,12 @@ void mlx5_lag_mpesw_disable(struct mlx5_core_dev *dev);int mlx5_lag_mpesw_enable (struct mlx5_core_dev *dev);#ifdef CONFIG_MLX5_ESWITCH void mlx5_lag_disable_mpesw (struct mlx5_lag *ldev); +void mlx5_mpesw_sd_devcoms_lock (struct mlx5_lag *ldev); +void mlx5_mpesw_sd_devcoms_unlock (struct mlx5_lag *ldev);#else static inline void mlx5_lag_disable_mpesw (struct mlx5_lag *ldev) {} +static inline void mlx5_mpesw_sd_devcoms_lock (struct mlx5_lag *ldev) {} +static inline void mlx5_mpesw_sd_devcoms_unlock (struct mlx5_lag *ldev) {}#endif /* CONFIG_MLX5_ESWITCH */ #ifdef CONFIG_MLX5_ESWITCH
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 13/15 net/mlx5: E-Switch, Tie rep load/unload to SD LAG state
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
On an SD device, vport representors are not functional until the SD group is combined and shared FDB is active. Skip the initial load and the reload paths in that window; reps are loaded as part of the SD LAG activation flow once it becomes active.
In addition, explicitly unload representors when SD LAG is destroyed.
--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 4 +++ .../mellanox/mlx5/core/eswitch_offloads.c | 26 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 26 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 1 + .../mellanox/mlx5/core/lag/shared_fdb.c | 1 + 5 files changed, 58 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +4 -0
@@ -959,6 +959,7 @@ int mlx5_eswitch_offloads_single_fdb_add_one(struct mlx5_eswitch *master_esw,void mlx5_eswitch_offloads_single_fdb_del_one (struct mlx5_eswitch *master_esw,struct mlx5_eswitch *slave_esw);int mlx5_eswitch_reload_ib_reps (struct mlx5_eswitch *esw); +void mlx5_eswitch_unload_reps (struct mlx5_eswitch *esw);bool mlx5_eswitch_is_peer (struct mlx5_eswitch *esw,struct mlx5_eswitch *peer_esw);
@@ -1063,6 +1064,9 @@ mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)return 0 ; } +static inline void +mlx5_eswitch_unload_reps (struct mlx5_eswitch *esw) {} +static inline bool mlx5_eswitch_block_encap (struct mlx5_core_dev *dev,bool from_fdb) {
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +26 -0
@@ -2863,6 +2863,10 @@ static int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num)int rep_type;int err; +if (vport_num !=MLX5_VPORT_UPLINK && +mlx5_get_sd (esw->dev) && !mlx5_lag_is_active (esw->dev)) +return 0 ; + rep =mlx5_eswitch_get_rep (esw, vport_num);for (rep_type =0 ; rep_type <NUM_REP_TYPES ; rep_type++) { err = __esw_offloads_load_rep(esw, rep, rep_type,
@@ -3779,6 +3783,21 @@ static void esw_destroy_offloads_acl_tables(struct mlx5_eswitch *esw)esw_vport_destroy_offloads_acl_tables (esw, vport); } +void mlx5_eswitch_unload_reps (struct mlx5_eswitch *esw) +{ +struct mlx5_eswitch_rep *rep; +unsigned long i; + +if (!esw || esw->mode !=MLX5_ESWITCH_OFFLOADS ) +return ; + +mlx5_esw_for_each_rep (esw, i, rep) { +if (rep->vport ==MLX5_VPORT_UPLINK ) +continue ; +mlx5_esw_offloads_rep_unload (esw, rep->vport); + } +} +int mlx5_eswitch_reload_ib_reps (struct mlx5_eswitch *esw) {struct mlx5_eswitch_rep *rep;
@@ -3805,6 +3824,10 @@ int mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)if (!mlx5_sd_is_primary (esw->dev) && rep->vport ==MLX5_VPORT_UPLINK )continue ; +if (rep->vport !=MLX5_VPORT_UPLINK && +mlx5_get_sd (esw->dev) && !mlx5_lag_is_active (esw->dev)) +continue ; +if (atomic_read (&rep->rep_data[REP_ETH ].state) ==REP_LOADED )__esw_offloads_load_rep (esw, rep,REP_IB ,NULL ); }
@@ -4764,6 +4787,9 @@ static void mlx5_eswitch_reload_reps_blocked(struct mlx5_eswitch *esw)return ; } +if (mlx5_get_sd (esw->dev) && !mlx5_lag_is_active (esw->dev)) +return ; +mlx5_esw_for_each_vport (esw, i, vport) {if (!vport)continue ;
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +26 -0
@@ -1312,6 +1312,32 @@ int mlx5_lag_reload_ib_reps_from_locked(struct mlx5_lag *ldev, u32 flags,return mlx5_lag_reload_ib_reps (ldev, flags, filter, cont_on_fail); } +static void mlx5_lag_unload_reps_unlocked (struct mlx5_lag *ldev,u32 filter) +{ +struct lag_func *pf; +int i; + +mlx5_lag_for_each (i,0 , ldev, filter) { +struct mlx5_eswitch *esw; + + pf =mlx5_lag_pf (ldev, i); + esw = pf->dev->priv.eswitch; +mlx5_esw_reps_block (esw); +mlx5_eswitch_unload_reps (esw); +mlx5_esw_reps_unblock (esw); + } +} + +void mlx5_lag_unload_reps_from_locked (struct mlx5_lag *ldev,u32 filter) +{ +/* Same lock dance as mlx5_lag_reload_ib_reps: drop ldev->lock around +* the per-eswitch reps_lock to keep the reps_lock -> ldev->lock order. +*/ +mlx5_lag_drop_lock_for_reps (ldev, filter); +mlx5_lag_unload_reps_unlocked (ldev, filter); +mlx5_lag_retake_lock_after_reps (ldev); +} +void mlx5_disable_lag (struct mlx5_lag *ldev) {bool shared_fdb =test_bit (MLX5_LAG_MODE_FLAG_SHARED_FDB , &ldev->mode_flags);
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h +1 -0
@@ -310,6 +310,7 @@ int mlx5_lag_num_devs(struct mlx5_lag *ldev);int mlx5_lag_num_netdevs (struct mlx5_lag *ldev);int mlx5_lag_reload_ib_reps_from_locked (struct mlx5_lag *ldev,u32 flags,u32 filter,bool cont_on_fail); +void mlx5_lag_unload_reps_from_locked (struct mlx5_lag *ldev,u32 filter);int mlx5_ldev_add_mdev (struct mlx5_lag *ldev,struct mlx5_core_dev *dev,u32 group_id);void mlx5_ldev_remove_mdev (struct mlx5_lag *ldev,struct mlx5_core_dev *dev);
drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c +1 -0
@@ -296,6 +296,7 @@ void mlx5_lag_shared_fdb_destroy(struct mlx5_lag *ldev, u32 group_id) pf->sd_fdb_active = false; }mlx5_lag_destroy_single_fdb_filter (ldev, group_id); +mlx5_lag_unload_reps_from_locked (ldev, filter); }mlx5_lag_add_devices_filter (ldev, filter);
Signature · 1 line
2.44.0
Re: [PATCH net-next V3 13/15] net/mlx5: E-Switch, Tie rep load/unload to SD LAG state
Quoted 47 lines · Tariq Toukan
From: Shay Drory <[email protected]>
On an SD device, vport representors are not functional until the SD group is combined and shared FDB is active. Skip the initial load and the reload paths in that window; reps are loaded as part of the SD LAG activation flow once it becomes active.
In addition, explicitly unload representors when SD LAG is destroyed.
Signed-off-by Shay Drory <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Tariq Toukan <[email protected]>--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 4 +++ .../mellanox/mlx5/core/eswitch_offloads.c | 26 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 26 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/lag/lag.h | 1 + .../mellanox/mlx5/core/lag/shared_fdb.c | 1 + 5 files changed, 58 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +4
-0@@ -959,6 +959,7 @@ int mlx5_eswitch_offloads_single_fdb_add_one(struct mlx5_eswitch *master_esw,void mlx5_eswitch_offloads_single_fdb_del_one (struct mlx5_eswitch *master_esw,struct mlx5_eswitch *slave_esw);int mlx5_eswitch_reload_ib_reps (struct mlx5_eswitch *esw); +void mlx5_eswitch_unload_reps (struct mlx5_eswitch *esw);bool mlx5_eswitch_is_peer (struct mlx5_eswitch *esw,struct mlx5_eswitch *peer_esw);@@ -1063,6 +1064,9 @@ mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)return 0 ; } +static inline void +mlx5_eswitch_unload_reps (struct mlx5_eswitch *esw) {} +static inline bool mlx5_eswitch_block_encap (struct mlx5_core_dev *dev,bool from_fdb) {drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0
-0
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c static int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num) · line 2868
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0@@ -2863,6 +2863,10 @@ static int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num)int rep_type;int err; +if (vport_num !=MLX5_VPORT_UPLINK && +mlx5_get_sd (esw->dev) && !mlx5_lag_is_active (esw->dev)) +return 0 ;
sashiko.dev says:
If we intentionally skip the initial load of the Ethernet representors (REP_ETH) here, will secondary devices permanently fail to load them later? Looking at mlx5_lag_shared_fdb_create(), the code triggers an asynchronous rescan exclusively for the primary device (dev0):
mlx5_lag_rescan_dev_locked(ldev, dev0, true);
err = mlx5_lag_reload_ib_reps_from_locked(ldev, 0, filter, false);Because secondary devices are not rescanned, their auxiliary eth-rep drivers are never re-probed, and their asynchronous reload is never queued. Since the synchronous reload depends on REP_ETH already being loaded, skipping it here seems to leave secondary devices permanently without network interfaces.
[SD] primary will invoke mlx5_esw_add_work() for secondary devices as well via mlx5_eswitch_register_vport_reps_nested().
Quoted 27 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +16
-0+ rep =mlx5_eswitch_get_rep (esw, vport_num);for (rep_type =0 ; rep_type <NUM_REP_TYPES ; rep_type++) { err = __esw_offloads_load_rep(esw, rep, rep_type,@@ -3779,6 +3783,21 @@ static void esw_destroy_offloads_acl_tables(struct mlx5_eswitch *esw)esw_vport_destroy_offloads_acl_tables (esw, vport); } +void mlx5_eswitch_unload_reps (struct mlx5_eswitch *esw) +{ +struct mlx5_eswitch_rep *rep; +unsigned long i; + +if (!esw || esw->mode !=MLX5_ESWITCH_OFFLOADS ) +return ; + +mlx5_esw_for_each_rep (esw, i, rep) { +if (rep->vport ==MLX5_VPORT_UPLINK ) +continue ; +mlx5_esw_offloads_rep_unload (esw, rep->vport); + } +} +int mlx5_eswitch_reload_ib_reps (struct mlx5_eswitch *esw) {struct mlx5_eswitch_rep *rep;
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw) · line 3830
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0@@ -3805,6 +3824,10 @@ int mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)if (!mlx5_sd_is_primary (esw->dev) && rep->vport ==MLX5_VPORT_UPLINK )continue ; +if (rep->vport !=MLX5_VPORT_UPLINK && +mlx5_get_sd (esw->dev) && !mlx5_lag_is_active (esw->dev)) +continue ; +
Is there a race condition here during SD LAG activation that bypasses the synchronous load of primary device representors?
In mlx5_lag_shared_fdb_create(), the unbind/rebind of auxiliary drivers for the primary device queues an asynchronous work item to load REP_ETH. Immediately following this, mlx5_lag_reload_ib_reps_from_locked() executes synchronously.
Because the asynchronous work hasn't run yet, REP_ETH is not loaded. Consequently, this synchronous loop will evaluate the REP_LOADED check as false and silently skip loading REP_IB.
[SD] The async reload loads both REP_ETH and REP_IB for VF/SF. The synchronous reload_ib_reps only re-adds IB for reps whose ETH is already loaded; skipping IB when ETH isn't up yet is not a loss — the async path loads both. No race.
Quoted 67 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0if (atomic_read (&rep->rep_data[REP_ETH ].state) ==REP_LOADED )__esw_offloads_load_rep (esw, rep,REP_IB ,NULL ); }@@ -4764,6 +4787,9 @@ static void mlx5_eswitch_reload_reps_blocked(struct mlx5_eswitch *esw)return ; } +if (mlx5_get_sd (esw->dev) && !mlx5_lag_is_active (esw->dev)) +return ; +mlx5_esw_for_each_vport (esw, i, vport) {if (!vport)continue ;drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +26
-0@@ -1312,6 +1312,32 @@ int mlx5_lag_reload_ib_reps_from_locked(struct mlx5_lag *ldev, u32 flags,return mlx5_lag_reload_ib_reps (ldev, flags, filter, cont_on_fail); } +static void mlx5_lag_unload_reps_unlocked (struct mlx5_lag *ldev,u32 filter) +{ +struct lag_func *pf; +int i; + +mlx5_lag_for_each (i,0 , ldev, filter) { +struct mlx5_eswitch *esw; + + pf =mlx5_lag_pf (ldev, i); + esw = pf->dev->priv.eswitch; +mlx5_esw_reps_block (esw); +mlx5_eswitch_unload_reps (esw); +mlx5_esw_reps_unblock (esw); + } +} + +void mlx5_lag_unload_reps_from_locked (struct mlx5_lag *ldev,u32 filter) +{ +/* Same lock dance as mlx5_lag_reload_ib_reps: drop ldev->lock around +* the per-eswitch reps_lock to keep the reps_lock -> ldev->lock order. +*/ +mlx5_lag_drop_lock_for_reps (ldev, filter); +mlx5_lag_unload_reps_unlocked (ldev, filter); +mlx5_lag_retake_lock_after_reps (ldev); +} +void mlx5_disable_lag (struct mlx5_lag *ldev) {bool shared_fdb =test_bit (MLX5_LAG_MODE_FLAG_SHARED_FDB , &ldev->mode_flags);drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h +1
-0@@ -310,6 +310,7 @@ int mlx5_lag_num_devs(struct mlx5_lag *ldev);int mlx5_lag_num_netdevs (struct mlx5_lag *ldev);int mlx5_lag_reload_ib_reps_from_locked (struct mlx5_lag *ldev,u32 flags,u32 filter,bool cont_on_fail); +void mlx5_lag_unload_reps_from_locked (struct mlx5_lag *ldev,u32 filter);int mlx5_ldev_add_mdev (struct mlx5_lag *ldev,struct mlx5_core_dev *dev,u32 group_id);void mlx5_ldev_remove_mdev (struct mlx5_lag *ldev,struct mlx5_core_dev *dev);drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c +0
-0
drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c void mlx5_lag_shared_fdb_destroy(struct mlx5_lag *ldev, u32 group_id) · line 299
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c +1
-0@@ -296,6 +296,7 @@ void mlx5_lag_shared_fdb_destroy(struct mlx5_lag *ldev, u32 group_id) pf->sd_fdb_active = false; }mlx5_lag_destroy_single_fdb_filter (ldev, group_id); +mlx5_lag_unload_reps_from_locked (ldev, filter);
sashiko.dev says:
Does explicitly unloading all representors here render the subsequent IB reload a dead code no-op?
Immediately after mlx5_lag_unload_reps_from_locked() forcefully unloads REP_ETH and other representors, this function calls mlx5_lag_reload_ib_reps_from_locked().
Because REP_ETH was just unloaded, the condition checking if the state is REP_LOADED inside mlx5_eswitch_reload_ib_reps() will evaluate to false, silently skipping all IB representors.
[SD] this is intended
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c +0
-0}mlx5_lag_add_devices_filter (ldev, filter);
Tariq Toukan
maintainerPATCH v3 14/15 net/mlx5: SD, defer vport metadata init until SD is ready
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Allow SD devices to transition to switchdev before the SD group is fully up. Metadata allocation requires the SD group to be ready, so defer it from esw_offloads_enable() until SD shared-FDB activation.
Add mlx5_esw_offloads_init_deferred_metadata() which allocates per-vport metadata and refreshes the ingress ACLs that were previously programmed with metadata=0. The helper is idempotent and can be called multiple times.
--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 1 + .../mellanox/mlx5/core/eswitch_offloads.c | 79 ++++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 16 ++++ 3 files changed, 93 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +1 -0
@@ -440,6 +440,7 @@ struct mlx5_eswitch {void esw_offloads_disable (struct mlx5_eswitch *esw);int esw_offloads_enable (struct mlx5_eswitch *esw); +int mlx5_esw_offloads_init_deferred_metadata (struct mlx5_eswitch *esw);void esw_offloads_cleanup (struct mlx5_eswitch *esw);int esw_offloads_init (struct mlx5_eswitch *esw);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +76 -3
97 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +16 -0
@@ -992,6 +992,7 @@ static bool mlx5_sd_all_paired(struct mlx5_core_dev *primary)static void mlx5_sd_activate_shared_fdb (struct mlx5_core_dev *primary) {struct mlx5_sd *sd =mlx5_get_sd (primary); +struct mlx5_core_dev *pos;struct mlx5_lag *ldev;struct lag_func *pf;int err;
@@ -1024,6 +1025,21 @@ static void mlx5_sd_activate_shared_fdb(struct mlx5_core_dev *primary) goto unlock; } +/* Initialize vport metadata for all group devices. This is deferred +* from esw_offloads_enable() because mlx5_sd_pf_num_get() requires +* the SD group to be ready. +*/ +mlx5_sd_for_each_dev (i, primary, pos) { +struct mlx5_eswitch *esw = pos->priv.eswitch; + + err =mlx5_esw_offloads_init_deferred_metadata (esw); +if (err) { +sd_warn (primary,"Failed to init metadata for %s: %d\n" , +dev_name (pos->device), err); + goto unlock; + } + } + err =mlx5_lag_shared_fdb_create (ldev,NULL ,0 , sd->group_id);if (err)sd_warn (primary,"Failed to create shared FDB: %d\n" , err);
Signature · 1 line
2.44.0
Re: [PATCH net-next V3 14/15] net/mlx5: SD, defer vport metadata init until SD is ready
Quoted 65 lines · Tariq Toukan
From: Shay Drory <[email protected]>
Allow SD devices to transition to switchdev before the SD group is fully up. Metadata allocation requires the SD group to be ready, so defer it from
esw_offloads_enable()until SD shared-FDB activation.Add
mlx5_esw_offloads_init_deferred_metadata()which allocates per-vport metadata and refreshes the ingress ACLs that were previously programmed with metadata=0. The helper is idempotent and can be called multiple times.Signed-off-by Shay Drory <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Tariq Toukan <[email protected]>--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 1 + .../mellanox/mlx5/core/eswitch_offloads.c | 79 ++++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/lib/sd.c | 16 ++++ 3 files changed, 93 insertions(+), 3 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +1
-0@@ -440,6 +440,7 @@ struct mlx5_eswitch {void esw_offloads_disable (struct mlx5_eswitch *esw);int esw_offloads_enable (struct mlx5_eswitch *esw); +int mlx5_esw_offloads_init_deferred_metadata (struct mlx5_eswitch *esw);void esw_offloads_cleanup (struct mlx5_eswitch *esw);int esw_offloads_init (struct mlx5_eswitch *esw);drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +18
-0@@ -3675,6 +3675,7 @@ static void esw_offloads_vport_metadata_cleanup(struct mlx5_eswitch *esw,WARN_ON (vport->metadata != vport->default_metadata);mlx5_esw_match_metadata_free (esw, vport->default_metadata); + vport->default_metadata =0 ; }static void esw_offloads_metadata_uninit (struct mlx5_eswitch *esw)@@ -3711,6 +3712,73 @@ static int esw_offloads_metadata_init(struct mlx5_eswitch *esw)return err; } +/* Deferred metadata init for SD devices: allocate vport metadata and +* refresh the ingress ACL for every vport whose ACL was created with +* metadata=0 in esw_create_offloads_acl_tables() / esw_vport_setup(). +* +* No Rep is loaded at this point ==> no Rep net-dev exists, so no need +* to take rtnl lock. +* +* Safe to call multiple times - subsequent calls are no-ops. +*/ +int mlx5_esw_offloads_init_deferred_metadata (struct mlx5_eswitch *esw) +{ +struct mlx5_vport *manager, *vport; +unsigned long i; +int err; + +if (!mlx5_eswitch_vport_match_metadata_enabled (esw)) +return 0 ;
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c static int esw_offloads_metadata_init(struct mlx5_eswitch *esw) · line 3741
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+ + manager =mlx5_eswitch_get_vport (esw, esw->manager_vport); +if (IS_ERR (manager)) +return PTR_ERR (manager); + +/* Sanity check: skip if metadata was already initialized */ +if (manager->default_metadata) +return 0 ; + + err =esw_offloads_metadata_init (esw);
sashiko.dev says:
Does unconditionally allocating new metadata here overwrite any dynamically assigned vport metadata?
[SD] No. no REPs are loaded until SD LAG is created, so user can't do any REPs bonding
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +8
-0+if (err) +return err; + +mutex_lock (&esw->state_lock); +/* Manager vport doesn't have a rep/netdev loaded but its ingress ACL +* was programmed with metadata=0 - refresh it explicitly. +*/ + err = mlx5_esw_acl_ingress_vport_metadata_update(esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c static int esw_offloads_metadata_init(struct mlx5_eswitch *esw) · line 3759
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+ esw->manager_vport, + 0); +if (err) + goto err_acl; + +/* UPLINK is never marked enabled but its ACL is programmed in +* esw_create_offloads_acl_tables(); refresh it explicitly. +*/ + err =mlx5_esw_acl_ingress_vport_metadata_update (esw,MLX5_VPORT_UPLINK , +0 );
sashiko.dev says:
Is the rtnl_lock required during these deferred ACL updates?
The code comment assumes no representor netdev exists at this point. However, for the primary SD device, representor netdevs were loaded and exposed to user-space much earlier when its switchdev transition completed.
[SD] same as above, no REPs net-dev exists at this point
Quoted 27 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +22
-0+if (err) + goto err_acl; + +mlx5_esw_for_each_vport (esw, i, vport) { +if (!vport || !vport->enabled) +continue ; + err =mlx5_esw_acl_ingress_vport_metadata_update (esw, + vport->vport, +0 ); +if (err) + goto err_acl; + } + +mutex_unlock (&esw->state_lock); +return 0 ; + +err_acl: +esw_offloads_metadata_uninit (esw); +mutex_unlock (&esw->state_lock); +return err; +} +int esw_vport_create_offloads_acl_tables (struct mlx5_eswitch *esw,struct mlx5_vport *vport)@@ -4072,9 +4140,14 @@ int esw_offloads_enable(struct mlx5_eswitch *esw)if (err)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int esw_offloads_enable(struct mlx5_eswitch *esw) · line 4147
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +5
-3goto err_roce; - err =esw_offloads_metadata_init (esw); -if (err) - goto err_metadata; +/* SD devices defer metadata init until SD is ready and +* mlx5_sd_pf_num_get() can return the correct pf_num. +*/ +if (!mlx5_get_sd (esw->dev)) { + err =esw_offloads_metadata_init (esw);
sashiko.dev says:
Does skipping metadata initialization here prematurely expose representors with uninitialized metadata?esw_offloads_enable() skips metadata initialization for SD devices, leaving vport->metadata = 0. Later in this function, representor netdevs are loaded and exposed to userspace, which allows userspace to add TC rules matching metadata=0 before the metadata is actually initialized.
[SD] same as above
Quoted 40 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0+if (err) + goto err_metadata; + } err =esw_set_passing_vport_metadata (esw, true);if (err)drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +16
-0@@ -992,6 +992,7 @@ static bool mlx5_sd_all_paired(struct mlx5_core_dev *primary)static void mlx5_sd_activate_shared_fdb (struct mlx5_core_dev *primary) {struct mlx5_sd *sd =mlx5_get_sd (primary); +struct mlx5_core_dev *pos;struct mlx5_lag *ldev;struct lag_func *pf;int err;@@ -1024,6 +1025,21 @@ static void mlx5_sd_activate_shared_fdb(struct mlx5_core_dev *primary) goto unlock; } +/* Initialize vport metadata for all group devices. This is deferred +* from esw_offloads_enable() because mlx5_sd_pf_num_get() requires +* the SD group to be ready. +*/ +mlx5_sd_for_each_dev (i, primary, pos) { +struct mlx5_eswitch *esw = pos->priv.eswitch; + + err =mlx5_esw_offloads_init_deferred_metadata (esw); +if (err) { +sd_warn (primary,"Failed to init metadata for %s: %d\n" , +dev_name (pos->device), err); + goto unlock; + } + } + err =mlx5_lag_shared_fdb_create (ldev,NULL ,0 , sd->group_id);if (err)sd_warn (primary,"Failed to create shared FDB: %d\n" , err);
Tariq Toukan
maintainerPATCH v3 15/15 net/mlx5: SD, enable SD over ECPF and allow switchdev transition
In reply to Tariq ToukanFrom: Shay Drory <[email protected]>
Remove the restriction blocking SD on embedded CPU PFs (ECPF), enabling SD functionality on BlueField DPUs. Remove the blocker preventing SD devices from transitioning to switchdev mode.
The infrastructure added in earlier patches properly handles this case.
--- .../net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 6 ------ drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c | 8 -------- 2 files changed, 14 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0 -6
@@ -4472,12 +4472,6 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode,if (esw_mode_from_devlink (mode, &mlx5_mode))return -EINVAL ; -if (mlx5_mode ==MLX5_ESWITCH_OFFLOADS &&mlx5_get_sd (esw->dev)) { -NL_SET_ERR_MSG_MOD (extack, -"Can't change E-Switch mode to switchdev when multi-PF netdev (Socket Direct) is configured." ); -return -EPERM ; - } -/* Avoid try_lock, active/inactive mode change is not restricted */ if (mlx5_devlink_switchdev_active_mode_change (esw, mode))return 0 ;
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +0 -8
@@ -222,10 +222,6 @@ bool mlx5_sd_is_supported(struct mlx5_core_dev *dev)if (!mlx5_core_is_pf (dev))return false; -/* Block on embedded CPU PFs */ -if (mlx5_core_is_ecpf (dev)) -return false; - err =mlx5_query_nic_vport_sd_group (dev, &sd_group);if (err || !sd_group)return false;
@@ -252,10 +248,6 @@ static int sd_init(struct mlx5_core_dev *dev)if (!mlx5_core_is_pf (dev))return 0 ; -/* Block on embedded CPU PFs */ -if (mlx5_core_is_ecpf (dev)) -return 0 ; - err =mlx5_query_nic_vport_sd_group (dev, &sd_group);if (err)return err;
Signature · 1 line
2.44.0
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 11 lines · On Fri, 12 Jun 2026 14:38:49 +0300 you wrote:
Hi,
This is part 2. Find part 1 here:
https://lore.kernel.org/all/[email protected]/This series enables Socket Direct single netdev to operate in switchdev mode with shared FDB. SD single netdev combines multiple PCI functions behind a single netdev interface. To support switchdev offloads, these functions must participate in virtual LAG (shared FDB).
[...]
Here is the summary with links:
- [net-next,V3,01/15] net/mlx5: E-Switch, skip uplink IB rep load for SD secondary devices
https://git.kernel.org/netdev/net-next/c/597baeb467d8 - [net-next,V3,02/15] net/mlx5: devcom, expose locked variant of send_event https://git.kernel.org/netdev/net-next/c/95e26588c84b - [net-next,V3,03/15] net/mlx5: devcom, add DEVCOM_CANT_FAIL for non-rollback events https://git.kernel.org/netdev/net-next/c/d5e77e4d3023 - [net-next,V3,04/15] net/mlx5: SD, make primary/secondary role determination more robust https://git.kernel.org/netdev/net-next/c/4b918a198389 - [net-next,V3,05/15] net/mlx5: SD, add L2 table silent mode query support https://git.kernel.org/netdev/net-next/c/13158554a302 - [net-next,V3,06/15] net/mlx5: SD, expend vport metadata for SD secondary devices https://git.kernel.org/netdev/net-next/c/a1bfe9f1da83 - [net-next,V3,07/15] net/mlx5: SD, support switchdev mode transition with shared FDB https://git.kernel.org/netdev/net-next/c/2a3fb8b2f450 - [net-next,V3,08/15] net/mlx5: E-Switch, notify SD on eswitch disable https://git.kernel.org/netdev/net-next/c/232de72bdea2 - [net-next,V3,09/15] net/mlx5: LAG, store demux resources per master lag_func https://git.kernel.org/netdev/net-next/c/eaaf1ff178a0 - [net-next,V3,10/15] net/mlx5: LAG, disable both regular and SD LAG on lag_disable_change https://git.kernel.org/netdev/net-next/c/ebd629e70045 - [net-next,V3,11/15] net/mlx5: LAG, introduce software vport LAG implementation https://git.kernel.org/netdev/net-next/c/98d56915eef5 - [net-next,V3,12/15] net/mlx5: LAG, add MPESW over SD LAG support https://git.kernel.org/netdev/net-next/c/de464720489c - [net-next,V3,13/15] net/mlx5: E-Switch, Tie rep load/unload to SD LAG state https://git.kernel.org/netdev/net-next/c/68c2dd59a6c7 - [net-next,V3,14/15] net/mlx5: SD, defer vport metadata init until SD is ready https://git.kernel.org/netdev/net-next/c/e3a02f3ecb13 - [net-next,V3,15/15] net/mlx5: SD, enable SD over ECPF and allow switchdev transition https://git.kernel.org/netdev/net-next/c/7bcfb19465fc
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html