pds_core: add switchdev and tc for vlan offload
Subsystems (3)
cover pds_core: add switchdev and tc for vlan offload
This is an RFC for adding to the pds_core driver some very simple support for VF representors and a tc command for offloading VF port vlans.
The problem to solve is how to request that a NIC do the push/pop of port vlans on a VF. The initial pds_core patchset[0] included this support through the legacy ip-link methods with a PF netdev that had no datapath, simply existing to enable commands such as
ip link set <pf> vf <vfid> vlan <vid>
This was soundly squashed with a request to create proper VF representors. The pds_core driver has since been reworked and merged without this feature.
This pair of patches is a first attempt at adding support for a simple VF representor and tc offload which I've been tinkering with off and on over the last few weeks. I will acknowledge that we have no proper filtering offload language in our firmware's adminq interface yet. This has been mentioned internally and is a "future project" with no actual schedule yet. Given that, I have worked here with what I have, using the existing vf_setattr function.
An alternative that later occured to me is to make this a "devlink port function" thing, similar to the existing port mac. This would have the benefit of using a familiar concept from and similar single command as the legacy method, would allow early port setup as with setting the mac and other port features, and would not need to create a lot of mostly empty netdevs for the VF representors. I don't know if this would then lead to adding "trust" and "spoofcheck" as well, but I'm not aware of any other solutions for them, either. This also might make more sense for devices that don't end up as user network interfaces, such as a virtio block device that runs over ethernet on the back end. I don't have RFC code for this idea, but thought I would toss it out for discussion - I didn't see any previous related discussion in a (rather quick) search.
I welcome your comments and suggestions.
Thanks,
sln
[0]: https://lore.kernel.org/netdev/[email protected]/
Shannon Nelson (2):
pds_core: netdev representors for each VF
pds_core: tc command handling for vlan push-pop
drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/core.h | 12 + drivers/net/ethernet/amd/pds_core/main.c | 28 +- drivers/net/ethernet/amd/pds_core/rep.c | 322 +++++++++++++++++++++ 4 files changed, 361 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/rep.c
Signature · 1 line
2.17.1
RFC PATCH 1/2 pds_core: netdev representors for each VF
When using pds_core VFs, we want to do some amount of control and configuration - specifically, we need the ability to configure port vlans. Here we add switchdev and create representor netdevs for the VFs.
The pds_core switchdev can be enabled before or after the VFs have been enabled. This means that they can be brought up specifically for configuring the VF vlan, then can be immediately turned back off. Disabling the switchdev will not affect VF vlan configuration.
Due to the design of this particular device, there is no packet switching to the PF: the only datapaths available are through the VFs. Therefore, neither the PF nor the VF representors have packet processing queues. For now, these representors are just enough to send a few tc configuration commands to the FW for setting port vlans on each VF.
--- drivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/core.h | 10 ++ drivers/net/ethernet/amd/pds_core/main.c | 28 ++++- drivers/net/ethernet/amd/pds_core/rep.c | 140 +++++++++++++++++++++ 4 files changed, 177 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/amd/pds_core/rep.c
drivers/net/ethernet/amd/pds_core/Makefile +1 -0
@@ -9,6 +9,7 @@ pds_core-y := main.o \ dev.o \ adminq.o \ core.o \ + rep.o \ fw.o pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
drivers/net/ethernet/amd/pds_core/core.h +10 -0
@@ -171,6 +171,7 @@ struct pdsc {unsigned int wdtimer_period;struct work_struct health_work;struct devlink_health_reporter *fw_reporter; +struct devlink_port dl_port;u32 fw_recoveries;struct pdsc_devinfo dev_info;
@@ -196,6 +197,9 @@ struct pdsc {struct pdsc_qcq notifyqcq;u64 last_eid;struct pdsc_viftype *viftype_status; + +struct net_device *netdev; +enum devlink_eswitch_mode eswitch_mode; };/** enum pds_core_dbell_bits - bitwise composition of dbell values.
@@ -309,4 +313,10 @@ irqreturn_t pdsc_adminq_isr(int irq, void *data);int pdsc_firmware_update (struct pdsc *pdsc,const struct firmware *fw,struct netlink_ext_ack *extack); + +int pdsc_add_rep (struct pdsc *vf,struct pdsc *pf); +void pdsc_del_rep (struct pdsc *vf); +int pdsc_dl_eswitch_mode_get (struct devlink *devlink,u16 *mode); +int pdsc_dl_eswitch_mode_set (struct devlink *devlink,u16 mode, +struct netlink_ext_ack *extack);#endif /* _PDSC_H_ */
drivers/net/ethernet/amd/pds_core/main.c +26 -2
@@ -185,12 +185,27 @@ static int pdsc_init_vf(struct pdsc *vf) pf->vfs[vf->vf_id].vf = vf; err =pdsc_auxbus_dev_add (vf, pf); -if (err) { +if (err) + goto err_dl_unreg; + +if (pf->eswitch_mode ==DEVLINK_ESWITCH_MODE_SWITCHDEV ) {devl_lock (dl); -devl_unregister (dl); + err =pdsc_add_rep (vf, pf);devl_unlock (dl); +if (err) + goto err_del_aux; } +return 0 ; + +err_del_aux: +pdsc_auxbus_dev_del (vf, pf); + pf->vfs[vf->vf_id].vf =NULL ; +err_dl_unreg: +devl_lock (dl); +devl_unregister (dl); +devl_unlock (dl); +return err; }
@@ -262,6 +277,8 @@ static int pdsc_init_pf(struct pdsc *pdsc) goto err_out_unlock_dl; } + pdsc->eswitch_mode =DEVLINK_ESWITCH_MODE_LEGACY ; + hr =devl_health_reporter_create (dl, &pdsc_fw_reporter_ops,0 , pdsc);if (IS_ERR (hr)) {dev_warn (pdsc->dev,"Failed to create fw reporter: %pe\n" , hr);
@@ -304,6 +321,8 @@ static int pdsc_init_pf(struct pdsc *pdsc)static const struct devlink_ops pdsc_dl_ops = { .info_get = pdsc_dl_info_get, .flash_update = pdsc_dl_flash_update, + .eswitch_mode_set = pdsc_dl_eswitch_mode_set, + .eswitch_mode_get = pdsc_dl_eswitch_mode_get, };static const struct devlink_ops pdsc_dl_vf_ops = {
@@ -406,6 +425,11 @@ static void pdsc_remove(struct pci_dev *pdev) pf =pdsc_get_pf_struct (pdsc->pdev);if (!IS_ERR (pf)) { +if (pf->eswitch_mode ==DEVLINK_ESWITCH_MODE_SWITCHDEV ) { +devl_lock (dl); +pdsc_del_rep (pdsc); +devl_unlock (dl); + }pdsc_auxbus_dev_del (pdsc, pf); pf->vfs[pdsc->vf_id].vf =NULL ; }
drivers/net/ethernet/amd/pds_core/rep.c new file +140 -0
140 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 2/2 pds_core: tc command handling for vlan push-pop
In reply to Shannon NelsonSet up handling of tc commands for adding VF vlan push-pop HW offload through the VF's representor. The FW doesn't currently have a proper set of adminq commands for any tc filtering, that's a future discussion. For now we have to get by with using the existing VF_SETATTR command that will add the push and pop in a single request.
Example commands for use:
# add a qdisc for context
tc qdisc add dev eth0 handle ffff: ingress # add a rule to wrap outgoing traffic with vlan id 124
tc filter add dev eth0 parent ffff: pref 11 protocol all u32 skip_sw \
match u32 0 0 flowid 1:1 action vlan push id 124 # add a rule to pop the tag from incoming traffic
# this is required normally, but redundant with current FW
tc filter add dev eth0 parent ffff: pref 1 protocol 802.1Q u32 skip_sw \
match u32 0 0 action vlan pop # remove rules
tc filter del dev eth0 parent ffff:--- drivers/net/ethernet/amd/pds_core/core.h | 2 + drivers/net/ethernet/amd/pds_core/rep.c | 182 +++++++++++++++++++++++ 2 files changed, 184 insertions(+)
drivers/net/ethernet/amd/pds_core/core.h +2 -0
@@ -37,6 +37,8 @@ struct pdsc_vf {struct pdsc *vf;u16 index;__le16 vif_types[PDS_DEV_TYPE_MAX ]; + +u16 vlanid; };struct pdsc_devinfo {
drivers/net/ethernet/amd/pds_core/rep.c +182 -0
203 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 32 lines · Shannon Nelson
This is an RFC for adding to the
pds_coredriver some very simple support for VF representors and a tc command for offloading VF port vlans.The problem to solve is how to request that a NIC do the push/pop of port vlans on a VF. The initial
pds_corepatchset[0] included this support through the legacy ip-link methods with a PF netdev that had no datapath, simply existing to enable commands such as
ip link set <pf> vf <vfid> vlan <vid>
This was soundly squashed with a request to create proper VF representors. Thepds_coredriver has since been reworked and merged without this feature.This pair of patches is a first attempt at adding support for a simple VF representor and tc offload which I've been tinkering with off and on over the last few weeks. I will acknowledge that we have no proper filtering offload language in our firmware's adminq interface yet. This has been mentioned internally and is a "future project" with no actual schedule yet. Given that, I have worked here with what I have, using the existing
vf_setattrfunction.An alternative that later occured to me is to make this a "devlink port function" thing, similar to the existing port mac. This would have the benefit of using a familiar concept from and similar single command as the legacy method, would allow early port setup as with setting the mac and other port features, and would not need to create a lot of mostly empty netdevs for the VF representors. I don't know if this would then lead to adding "trust" and "spoofcheck" as well, but I'm not aware of any other solutions for them, either. This also might make more sense for devices that don't end up as user network interfaces, such as a virtio block device that runs over ethernet on the back end. I don't have RFC code for this idea, but thought I would toss it out for discussion - I didn't see any previous related discussion in a (rather quick) search.
Quoted 1 line
I welcome your comments and suggestions.
Adding a tc rule doesn't seem to be the right interface to configure port-vlan for all packets sent/received on a port. tc would be appropriate if we want to do push/pop vlan on certain flows.
bridge vlan add dev $VF_REP vid 200 pvid untagged master
would be a better option. This may require adding vf reps to a bridge and enable vlan_filtering on the bridge. The driver needs to register for SWITCHDEV_PORT_OBJ_ADD events.
Extending devlink port function to support port-vlan similar to setting a mac address also looks like a good option without the requirement of having to add port reps to a bridge.
Quoted 17 lines
Thanks,
sln[0]: https://lore.kernel.org/netdev/[email protected]/
Shannon Nelson (2):
pds_core: netdev representors for each VF
pds_core: tc command handling for vlan push-popdrivers/net/ethernet/amd/pds_core/Makefile | 1 + drivers/net/ethernet/amd/pds_core/core.h | 12 + drivers/net/ethernet/amd/pds_core/main.c | 28 +- drivers/net/ethernet/amd/pds_core/rep.c | 322 +++++++++++++++++++++ 4 files changed, 361 insertions(+), 2 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/rep.c
Quoted 34 lines · Sridhar
Quoted 32 lines · Shannon Nelson
This is an RFC for adding to the
pds_coredriver some very simple support for VF representors and a tc command for offloading VF port vlans.The problem to solve is how to request that a NIC do the push/pop of port vlans on a VF. The initial
pds_corepatchset[0] included this support through the legacy ip-link methods with a PF netdev that had no datapath, simply existing to enable commands such as
ip link set <pf> vf <vfid> vlan <vid>
This was soundly squashed with a request to create proper VF representors. Thepds_coredriver has since been reworked and merged without this feature.This pair of patches is a first attempt at adding support for a simple VF representor and tc offload which I've been tinkering with off and on over the last few weeks. I will acknowledge that we have no proper filtering offload language in our firmware's adminq interface yet. This has been mentioned internally and is a "future project" with no actual schedule yet. Given that, I have worked here with what I have, using the existing
vf_setattrfunction.An alternative that later occured to me is to make this a "devlink port function" thing, similar to the existing port mac. This would have the benefit of using a familiar concept from and similar single command as the legacy method, would allow early port setup as with setting the mac and other port features, and would not need to create a lot of mostly empty netdevs for the VF representors. I don't know if this would then lead to adding "trust" and "spoofcheck" as well, but I'm not aware of any other solutions for them, either. This also might make more sense for devices that don't end up as user network interfaces, such as a virtio block device that runs over ethernet on the back end. I don't have RFC code for this idea, but thought I would toss it out for discussion - I didn't see any previous related discussion in a (rather quick) search.
Quoted 5 lines
Quoted 1 line
I welcome your comments and suggestions.
Adding a tc rule doesn't seem to be the right interface to configure port-vlan for all packets sent/received on a port. tc would be appropriate if we want to do push/pop vlan on certain flows.
This is something that had occurred to me. The tc commands seem to be aimed at a finer grain control than what I need for this case.
Quoted 5 lines
bridge vlan add dev $VF_REP vid 200 pvid untagged master
would be a better option. This may require adding vf reps to a bridge and enablevlan_filteringon the bridge. The driver needs to register forSWITCHDEV_PORT_OBJ_ADDevents.
I thought about this as well, but this also seems to be abusing the model when there are no data streams and no representor for the PF.
Quoted 4 lines
Extending devlink port function to support port-vlan similar to setting a mac address also looks like a good option without the requirement of having to add port reps to a bridge.
I'm hoping that others will chime in on this idea. Meanwhile, I'll see if I can work up an RFC for this thought later this week.
sln
Quoted 18 lines
Quoted 18 lines
Thanks,
sln[0]: https://lore.kernel.org/netdev/[email protected]/
Shannon Nelson (2):
pds_core: netdev representors for each VF
pds_core: tc command handling for vlan push-pop
drivers/net/ethernet/amd/pds_core/Makefile| 1 + drivers/net/ethernet/amd/pds_core/core.h | 12 + drivers/net/ethernet/amd/pds_core/main.c | 28 +- drivers/net/ethernet/amd/pds_core/rep.c | 322 +++++++++++++++++++++4 files changed, 361 insertions(+), 2 deletions(-)create mode 100644
drivers/net/ethernet/amd/pds_core/rep.c
Jakub Kicinski
maintainerQuoted 10 lines · On Thu, 27 Apr 2023 09:45:44 -0700 Shannon Nelson wrote:
This is an RFC for adding to the
pds_coredriver some very simple support for VF representors and a tc command for offloading VF port vlans.The problem to solve is how to request that a NIC do the push/pop of port vlans on a VF. The initial
pds_corepatchset[0] included this support through the legacy ip-link methods with a PF netdev that had no datapath, simply existing to enable commands such as
ip link set <pf> vf <vfid> vlan <vid>
This was soundly squashed with a request to create proper VF representors. Thepds_coredriver has since been reworked and merged without this feature.
Have you read the representors documentation? Passing traffic is crucial.
Quoted 8 lines
This pair of patches is a first attempt at adding support for a simple VF representor and tc offload which I've been tinkering with off and on over the last few weeks. I will acknowledge that we have no proper filtering offload language in our firmware's adminq interface yet. This has been mentioned internally and is a "future project" with no actual schedule yet. Given that, I have worked here with what I have, using the existing
vf_setattrfunction.
Quoted 12 lines
An alternative that later occured to me is to make this a "devlink port function" thing, similar to the existing port mac. This would have the benefit of using a familiar concept from and similar single command as the legacy method, would allow early port setup as with setting the mac and other port features, and would not need to create a lot of mostly empty netdevs for the VF representors. I don't know if this would then lead to adding "trust" and "spoofcheck" as well, but I'm not aware of any other solutions for them, either. This also might make more sense for devices that don't end up as user network interfaces, such as a virtio block device that runs over ethernet on the back end. I don't have RFC code for this idea, but thought I would toss it out for discussion - I didn't see any previous related discussion in a (rather quick) search.
No, no -- the problem is not rtnetlink vs devlink but the fact that the old API was inventing its own parallel way of configuring forwarding outside of normal/SW netdev concepts.
Quoted 13 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 27 Apr 2023 09:45:44 -0700 Shannon Nelson wrote:
This is an RFC for adding to the
pds_coredriver some very simple support for VF representors and a tc command for offloading VF port vlans.The problem to solve is how to request that a NIC do the push/pop of port vlans on a VF. The initial
pds_corepatchset[0] included this support through the legacy ip-link methods with a PF netdev that had no datapath, simply existing to enable commands such as
ip link set <pf> vf <vfid> vlan <vid>
This was soundly squashed with a request to create proper VF representors. Thepds_coredriver has since been reworked and merged without this feature.
Quoted 2 lines
Have you read the representors documentation? Passing traffic is crucial.
Given that there is no traffic to the host PF in this case, and I can't do anything more than wave my hands at vague promises for the future, perhaps tc and representors is the wrong path for now.
But there remains a need for some port related configuration. I can take a stab at adding this concept to "devlink port function" and see what discussion follows from there.
sln
Jakub Kicinski
maintainerQuoted 3 lines · On Wed, 3 May 2023 15:49:27 -0700 Shannon Nelson wrote:
Given that there is no traffic to the host PF in this case, and I can't do anything more than wave my hands at vague promises for the future, perhaps tc and representors is the wrong path for now.
If it's not a priority for AMD to have an upstream implementation that's fine. There's not point over-complicating the discussion.
Quoted 3 lines
But there remains a need for some port related configuration. I can take a stab at adding this concept to "devlink port function" and see what discussion follows from there.
You mean setting vlan encap via devlink?
I don't know why you'd do that. It will certainly aggravate me, and I doubt anyone will care/support you.
Quoted 9 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 3 May 2023 15:49:27 -0700 Shannon Nelson wrote:
Given that there is no traffic to the host PF in this case, and I can't do anything more than wave my hands at vague promises for the future, perhaps tc and representors is the wrong path for now.
If it's not a priority for AMD to have an upstream implementation that's fine. There's not point over-complicating the discussion.
Quoted 7 lines
Quoted 3 lines
But there remains a need for some port related configuration. I can take a stab at adding this concept to "devlink port function" and see what discussion follows from there.
You mean setting vlan encap via devlink?
I don't know why you'd do that. It will certainly aggravate me, and I doubt anyone will care/support you.
We're trying to solve what would seem to be a simple problem for our customer: how to do basic vlan encap/decap on all traffic going in and of a VF. With no host PF traffic, the legacy ip-link and the newer switchdev+tc solutions don't fit. As this is VF port setup, and devlink is meant for device setup, it would seem to fit in the devlink port function model similar to the setting of the port hw_addr. What am I missing that makes this an unacceptable answer?
I understand you don't like this devlink port function suggestion, but when I go back to negotiate with internal management, architects, etc, I can get a lot further with them if I have a technical explanation of why this is not acceptable. So, at the risk of further aggravating you, can I request a little more detail on why this is a bad idea?
Thanks,
sln
Jakub Kicinski
maintainerQuoted 8 lines · On Thu, 4 May 2023 13:51:13 -0700 Shannon Nelson wrote:
Quoted 3 lines · Jakub Kicinski
You mean setting vlan encap via devlink?
I don't know why you'd do that. It will certainly aggravate me, and I doubt anyone will care/support you.We're trying to solve what would seem to be a simple problem for our
customer: how to do basic vlan encap/decap on all traffic going in and of a VF.
Offload bridge.
If your customer doesn't want the bridge offload you can decide to ship an out of tree driver for them with whatever deprecated APIs you please.
Quoted 11 lines
With no host PF traffic, the legacy ip-link and the newer switchdev+tc solutions don't fit. As this is VF port setup, and devlink is meant for device setup, it would seem to fit in the devlink port function model similar to the setting of the port
hw_addr. What am I missing that makes this an unacceptable answer?I understand you don't like this devlink port function suggestion, but when I go back to negotiate with internal management, architects, etc, I can get a lot further with them if I have a technical explanation of why this is not acceptable. So, at the risk of further aggravating you, can I request a little more detail on why this is a bad idea?
The direction of the community was to use offload of standard networking concepts like switching, routing and flow matching for probably a decade now.