Skip to contentSkip to conversation

Offloading Priority Tables for queue classification

DISCUSSION Maxime Chevallier 2 participants 3 messages Apr 15, 2022 – Apr 25, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello everyone,

I'm starting this thread before any kind of series submission to discuss on the manner with which we could deal with Queue Classification in NICs, and more specifically how we could handle internal classification tables like DSCP, VLAN Prio and any other tables that we can populate in modern NICs to quickly assign a priority to both egress and ingress traffic, and use that priority to select a queue in which the packet will be enqueued/dequeued from.

The use-case we have in mind is to offload all of these classification steps into a switch , where traffic would be classified internally into queues, that we could configure for Frame-Preemption or any other QoS techniques. I know that Frame Preemption itself and the way to configure it is still under discussion, with debate on where to configure the queue preemptability (hence why some of you ended-up CC'd to that thread) :

https://lore.kernel.org/netdev/[email protected]/

There are already ways to do this classification, but from what I've gathered, it looks like it's scattered across several places :

- In TC, we can of course use TC flower for that. We can neatly decide which flows goes where, match on any of the fields that we can use to determine the priority of the packet. This however scales poorly when the underlying hardware uses tables dedicated only to matching specific fields, to assign each DSCP or VLAN a priority.

TC flower works well when we want to use a full-featured classifier, using a TCAM of some sort combined with complete classification rules. Using TC flower to configure such tables would mean entering one rule per entry in our tables, which could work for VLAN prio, but not that much for DSCP tables for example.

- TC skbedit with the priority offloading is exactly what we want to achieve, that is to emulate the skb->priority behaviour that we can configure with various ways, and map this priority to queues with mqpriofor example. tc-skbedit priority when offloaded handles that notion of "packet priority" that is used internally in a switch.

- TC mqprio and TC taprio can be used for the actual prio->queue mapping, even though there's the "traffic class" layer sitting in the middle.

- It looks like DCB could be a way to go to configure the DSCP/VLAN prio/any other QoS tables, since we can configure all of these tables with the "dcb app" command, which then calls hooks into the driver to configure offloading of these tables. Using DCB for this is perfect, since the traffic to prio assignment really is independant to the mqprio layer.

- Finally for the last part of the chain, we can setup the queues for PFC or Frame-Preemption, possibly using ethtool as suggested in the above-mentionned thread.

So in the end, my question would be : Is it OK to mix all of these together ? Using the dcb layer to configure the internal mapping of traffic -> priority, then using mqprio to configure the priority -> queue mapping, and then either TC again or ethtool do configure the behaviour of the queues themselves ? Or is there some other way that we've missed ?

Thanks in advance,

Best regards,

Maxime

#1 lore raw

Hi Maxime!

Quoted 56 lines · Maxime Chevallier

Hello everyone,

I'm starting this thread before any kind of series submission to discuss on the manner with which we could deal with Queue Classification in NICs, and more specifically how we could handle internal classification tables like DSCP, VLAN Prio and any other tables that we can populate in modern NICs to quickly assign a priority to both egress and ingress traffic, and use that priority to select a queue in which the packet will be enqueued/dequeued from.

The use-case we have in mind is to offload all of these classification steps into a switch , where traffic would be classified internally into queues, that we could configure for Frame-Preemption or any other QoS techniques. I know that Frame Preemption itself and the way to configure it is still under discussion, with debate on where to configure the queue preemptability (hence why some of you ended-up CC'd to that thread) :

https://lore.kernel.org/netdev/[email protected]/

There are already ways to do this classification, but from what I've gathered, it looks like it's scattered across several places :

- In TC, we can of course use TC flower for that. We can neatly decide which flows goes where, match on any of the fields that we can use to determine the priority of the packet. This however scales poorly when the underlying hardware uses tables dedicated only to matching specific fields, to assign each DSCP or VLAN a priority.

TC flower works well when we want to use a full-featured classifier, using a TCAM of some sort combined with complete classification rules. Using TC flower to configure such tables would mean entering one rule per entry in our tables, which could work for VLAN prio, but not that much for DSCP tables for example.

- TC skbedit with the priority offloading is exactly what we want to achieve, that is to emulate the skb->priority behaviour that we can configure with various ways, and map this priority to queues with mqpriofor example. tc-skbedit priority when offloaded handles that notion of "packet priority" that is used internally in a switch.

- TC mqprio and TC taprio can be used for the actual prio->queue mapping, even though there's the "traffic class" layer sitting in the middle.

- It looks like DCB could be a way to go to configure the DSCP/VLAN prio/any other QoS tables, since we can configure all of these tables with the "dcb app" command, which then calls hooks into the driver to configure offloading of these tables. Using DCB for this is perfect, since the traffic to prio assignment really is independant to the mqprio layer.

- Finally for the last part of the chain, we can setup the queues for PFC or Frame-Preemption, possibly using ethtool as suggested in the above-mentionned thread.

Quoted 6 lines

So in the end, my question would be : Is it OK to mix all of these together ? Using the dcb layer to configure the internal mapping of traffic -> priority, then using mqprio to configure the priority -> queue mapping, and then either TC again or ethtool do configure the behaviour of the queues themselves ? Or is there some other way that we've missed ?

I think it's ok to mix all of those together. At least the ocelot/felix DSA switches do support both advance QoS classification using tc-flower
+ skbedit priority action, and basic QoS classification (port-based or IP DSCP based) using the dcbnl application table. In short, at the end of the QoS classification process, a traffic class for the packet is selected. Then, the frame preemption would operate on the packet's traffic class.

Do you have any particular concerns?

#2 lore raw

Hi Vladimir !

Quoted 10 lines · On Fri, 15 Apr 2022 15:42:29 +0000 Vladimir Oltean <[email protected]>…

Hi Maxime!

Quoted 6 lines · Maxime Chevallier

So in the end, my question would be : Is it OK to mix all of these together ? Using the dcb layer to configure the internal mapping of traffic -> priority, then using mqprio to configure the priority -> queue mapping, and then either TC again or ethtool do configure the behaviour of the queues themselves ? Or is there some other way that we've missed ?

Quoted 8 lines

I think it's ok to mix all of those together. At least the ocelot/felix DSA switches do support both advance QoS classification using tc-flower
+ skbedit priority action, and basic QoS classification (port-based or IP DSCP based) using the dcbnl application table. In short, at the end of the QoS classification process, a traffic class for the packet is selected. Then, the frame preemption would operate on the packet's traffic class.

Thank you very much for that answer, it helps a lot. TBH when digging into classification, especially with TC, it's a bit overwhelming and it gets difficult pretty quickly to get what's the right approach.

Quoted 1 line

Do you have any particular concerns?

My concerns were mainly about not reinventing the wheel, but also making sure that I have the correct understanding on these classification steps. I'll make sure to CC you when I have a first series implementing that.

Best regards,

Maxime