Skip to contentSkip to conversation

tc H/W offload issue with vxlan tunnels [was: nfp: flower vxlan tunnel offload]

DISCUSSION Paolo Abeni 3 participants 8 messages Sep 27, 2017

Subsystems (2)
lore
Share & tools
lore

Paolo Abeni

author maintainer
#0 lore raw

Hi,

Moving to a separate theread, since I think this is more related to the flower core infrastructure than to the netrome patches.

Quoted 4 lines · Jiri Pirko

This kind of hooks are giving me nightmares. The code is screwed up as it is already. I'm currently working on conversion to callbacks. This part is handled in:
https://github.com/jpirko/linux_mlxsw/commits/jiri_devel_egdevcb

Thanks for the pointer.

I skimmed quickly on the code and indeed it cleans this area a lot. If I read it correctly the ('good') command:

tc filter add dev vxlan0 protocol ip parent ffff: flower enc_key_id 102
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

will generate a call to:

mlx5e_setup_tc(eth0, TC_SETUP_CLSFLOWER, &cls_flower) via:

fl_hw_replace_filter() ->
tc_setup_cb_call() ->

    tc_exts_setup_cb_egdev_call() ->
      tc_setup_cb_egdev_call() ->
        tcf_action_egdev_cb_call() ->
          mlx5e_rep_setup_tc_cb()

and the 'bad' command:

tc filter add dev eth0 protocol ip parent ffff: flower enc_key_id 102 \
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

will also call:

mlx5e_setup_tc(eth0, TC_SETUP_CLSFLOWER, &cls_flower) via:

fl_hw_replace_filter() ->
ndo_setup_tc()

So it looks like the H/W offload hook will still be called with the same arguments in both case, and 'bad' rule will still be pushed to the H/W as the driver itself has no way to distinct between the two scenarios.

[ Note: I referred to the mlx hook just for convenience, should be the same with any driver implementing the same APIs ]

Am I missing something?

Thanks,

Paolo

#1 lore raw
Quoted 11 lines · Wed, Sep 27, 2017 at 10:29:35AM CEST, [email protected] wrote:

Hi,

Moving to a separate theread, since I think this is more related to the flower core infrastructure than to the netrome patches.

Quoted 4 lines · Jiri Pirko

This kind of hooks are giving me nightmares. The code is screwed up as it is already. I'm currently working on conversion to callbacks. This part is handled in:
https://github.com/jpirko/linux_mlxsw/commits/jiri_devel_egdevcb

Quoted 7 lines

Thanks for the pointer.

I skimmed quickly on the code and indeed it cleans this area a lot. If I read it correctly the ('good') command:

tc filter add dev vxlan0 protocol ip parent ffff: flower enc_key_id 102
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

I suppose "action mirred redirect eth0". Then yes, it will generate the callpath you described below.

Quoted 17 lines

will generate a call to:

mlx5e_setup_tc(eth0, TC_SETUP_CLSFLOWER, &cls_flower) via:

fl_hw_replace_filter() ->
tc_setup_cb_call() ->
tc_exts_setup_cb_egdev_call() ->

     tc_setup_cb_egdev_call() ->
       tcf_action_egdev_cb_call() ->
         mlx5e_rep_setup_tc_cb()

and the 'bad' command:

tc filter add dev eth0 protocol ip parent ffff: flower enc_key_id 102 \
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

Quoted 6 lines

will also call:

mlx5e_setup_tc(eth0, TC_SETUP_CLSFLOWER, &cls_flower) via:

fl_hw_replace_filter() ->
ndo_setup_tc()

Sure. You are adding a rule to eth0, the call goes down to eth0 driver. I'm missing why is it a problem? Why the call should not go down to the eth0 driver?

Quoted 5 lines

So it looks like the H/W offload hook will still be called with the same arguments in both case, and 'bad' rule will still be pushed to the H/W as the driver itself has no way to distinct between the two scenarios.

Why "bad"?

Regarding the distinction, driver knows if user add a rule directly to the eth0, or if the eth0 is egress device in the action. Those are 2 separete driver entrypoints - of course, talking about code with my changes.

Quoted 9 lines

[ Note: I referred to the mlx hook just for convenience, should be the same with any driver implementing the same APIs ]

Am I missing something?

Thanks,

Paolo

author maintainer
#2 lore raw
Quoted 7 lines · Jiri Pirko
Quoted 4 lines · Wed, Sep 27, 2017 at 10:29:35AM CEST, [email protected] wrote:

So it looks like the H/W offload hook will still be called with the same arguments in both case, and 'bad' rule will still be pushed to the H/W as the driver itself has no way to distinct between the two scenarios.

Why "bad"?

Such rule is coped differently by the SW and the HW data path.

a rule like:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_hw \
action action mirred redirect eth0_vf_1

will match 0 packets, while:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_sw \
action action mirred redirect eth0_vf_1

[just flipped 'skip_sw' and 'skip_hw' ]
will match the vxlan-tunneled packets. I understand that one of the design goal for the h/w offload path is being consistent with the sw one, but that does not hold in the above scenario.

Quoted 4 lines

Regarding the distinction, driver knows if user add a rule directly to the eth0, or if the eth0 is egress device in the action. Those are 2 separete driver entrypoints - of course, talking about code with my changes.

ok, but than each driver should catch the scenario "rule with tunnel match over non tunnel device" and cope with them properly - never match it - why don't simply avoiding pushing such rules to the H/W ?

Cheers,

Paolo

#3 lore raw
Quoted 19 lines · Wed, Sep 27, 2017 at 11:46:58AM CEST, [email protected] wrote:
Quoted 7 lines · Jiri Pirko
Quoted 4 lines · Wed, Sep 27, 2017 at 10:29:35AM CEST, [email protected] wrote:

So it looks like the H/W offload hook will still be called with the same arguments in both case, and 'bad' rule will still be pushed to the H/W as the driver itself has no way to distinct between the two scenarios.

Why "bad"?

Such rule is coped differently by the SW and the HW data path.

a rule like:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_hw \
action action mirred redirect eth0_vf_1

will match 0 packets, while:

Quoted 8 lines

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_sw \
action action mirred redirect eth0_vf_1

[just flipped 'skip_sw' and 'skip_hw' ]
will match the vxlan-tunneled packets. I understand that one of the design goal for the h/w offload path is being consistent with the sw one, but that does not hold in the above scenario.

Sure, the consistency is important. Howcome "skip_hw" won't match and "skip_sw" will match? What's different?

Quoted 13 lines
Quoted 5 lines

Regarding the distinction, driver knows if user add a rule directly to the eth0, or if the eth0 is egress device in the action. Those are 2 separete driver entrypoints - of course, talking about code with my changes.

ok, but than each driver should catch the scenario "rule with tunnel match over non tunnel device" and cope with them properly - never match it - why don't simply avoiding pushing such rules to the H/W ?

Cheers,

Paolo

author maintainer
#4 lore raw
Quoted 24 lines · Jiri Pirko
Quoted 23 lines · Wed, Sep 27, 2017 at 11:46:58AM CEST, [email protected] wrote:
Quoted 7 lines · Jiri Pirko
Quoted 4 lines · Wed, Sep 27, 2017 at 10:29:35AM CEST, [email protected] wrote:

So it looks like the H/W offload hook will still be called with the same arguments in both case, and 'bad' rule will still be pushed to the H/W as the driver itself has no way to distinct between the two scenarios.

Why "bad"?

Such rule is coped differently by the SW and the HW data path.

a rule like:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_hw \
action action mirred redirect eth0_vf_1

will match 0 packets, while:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_sw \
action action mirred redirect eth0_vf_1

Quoted 7 lines
Quoted 4 lines

[just flipped 'skip_sw' and 'skip_hw' ]
will match the vxlan-tunneled packets. I understand that one of the design goal for the h/w offload path is being consistent with the sw one, but that does not hold in the above scenario.

Sure, the consistency is important. Howcome "skip_hw" won't match and "skip_sw" will match? What's different?

For the SW datapath, we need a metadata based/lwt tunnel to collect the tunnel information. eth0 is not a such device and does not provide the metadata. Any match on tunnel based field will fail - correctly.

When the HW datapath is used, the underlaying NIC is programmed exactly as done when we replace eth0 with vxlan0. The programmed flow matches vxlan encapsulated traffic.

Cheers,

Paolo

#5 lore raw
Quoted 30 lines · Wed, Sep 27, 2017 at 02:31:36PM CEST, [email protected] wrote:
Quoted 29 lines · Jiri Pirko
Quoted 27 lines · Wed, Sep 27, 2017 at 11:46:58AM CEST, [email protected] wrote:
Quoted 7 lines · Jiri Pirko
Quoted 4 lines · Wed, Sep 27, 2017 at 10:29:35AM CEST, [email protected] wrote:

So it looks like the H/W offload hook will still be called with the same arguments in both case, and 'bad' rule will still be pushed to the H/W as the driver itself has no way to distinct between the two scenarios.

Why "bad"?

Such rule is coped differently by the SW and the HW data path.

a rule like:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_hw \
action action mirred redirect eth0_vf_1

will match 0 packets, while:

tc filter add dev eth0 protocol ip parent ffff: flower \
enc_key_id 102 enc_dst_port 4789 src_ip 3.4.5.6 skip_sw \
action action mirred redirect eth0_vf_1

[just flipped 'skip_sw' and 'skip_hw' ]
will match the vxlan-tunneled packets. I understand that one of the design goal for the h/w offload path is being consistent with the sw one, but that does not hold in the above scenario.

Quoted 6 lines
Quoted 2 lines

Sure, the consistency is important. Howcome "skip_hw" won't match and "skip_sw" will match? What's different?

For the SW datapath, we need a metadata based/lwt tunnel to collect the tunnel information. eth0 is not a such device and does not provide the metadata. Any match on tunnel based field will fail - correctly.

So where do you attach the tc filter instead of eth0? vxlan0?

Quoted 4 lines

When the HW datapath is used, the underlaying NIC is programmed exactly as done when we replace eth0 with vxlan0. The programmed flow matches vxlan encapsulated traffic.

Ok, so we should unify the behaviour with sw path and don't allow such mathing in hw.

Quoted 4 lines

Cheers,

Paolo

#6 lore raw
Quoted 1 line · Jiri Pirko

So where do you attach the tc filter instead of eth0? vxlan0?

Yes, vxlan0. I'm pasting the example from earlier in this thread again:

This will match:

ip link add vxlan0 type vxlan dstport 4789 dev eth0 external ip link set dev vxlan0 up
tc qdisc add dev vxlan0 ingress
ethtool -K eth0 hw-tc-offload on
tc filter add dev vxlan0 protocol ip parent ffff: flower enc_key_id 102 \
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

while this must NOT match:

ip link add vxlan0 type vxlan dstport 4789 dev eth0 external ip link set dev vxlan0 up
tc qdisc add dev eth0 ingress
ethtool -K eth0 hw-tc-offload on
tc filter add dev eth0 protocol ip parent ffff: flower enc_key_id 102 \
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

Jiri

#7 lore raw
Quoted 14 lines · Wed, Sep 27, 2017 at 05:27:29PM CEST, [email protected] wrote:
Quoted 1 line · Jiri Pirko

So where do you attach the tc filter instead of eth0? vxlan0?

Yes, vxlan0. I'm pasting the example from earlier in this thread again:

This will match:

ip link add vxlan0 type vxlan dstport 4789 dev eth0 external ip link set dev vxlan0 up
tc qdisc add dev vxlan0 ingress
ethtool -K eth0 hw-tc-offload on
tc filter add dev vxlan0 protocol ip parent ffff: flower enc_key_id 102 \
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

Quoted 8 lines

while this must NOT match:

ip link add vxlan0 type vxlan dstport 4789 dev eth0 external ip link set dev vxlan0 up
tc qdisc add dev eth0 ingress
ethtool -K eth0 hw-tc-offload on
tc filter add dev eth0 protocol ip parent ffff: flower enc_key_id 102 \
enc_dst_port 4789 src_ip 3.4.5.6 skip_sw action [...]

Right. Driver of eth0 should get ndo_setup_tc calls for vxlan0. Similar thing will be needed for bonding/team. I'm taking this into considaration for my sharedblock patchset. Work in progress:
https://github.com/jpirko/linux_mlxsw/commits/jiri_devel_shblock
Basically the eth0 driver will register a callback function that would be called whenever filter is added/deleted on vxlan0