Extend XFRM core to allow full offload configuration

* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.

Quoted 7 lines · On Tue, 23 Aug 2022 16:31:57 +0300 Leon Romanovsky wrote:

* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.

Since the use case is somewhat in question, perhaps switch to RFC postings until the drivers side incl. tc forwarding is implemented?

Also the perf traces, I don't see them here.

Quoted 11 lines · Jakub Kicinski

Quoted 7 lines · On Tue, 23 Aug 2022 16:31:57 +0300 Leon Romanovsky wrote:

* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.

Since the use case is somewhat in question, perhaps switch to RFC postings until the drivers side incl. tc forwarding is implemented?

Proposed driver implementation works fine with eswitch representors. All our flow steering magic is performed on local table entry and it ensures that representors receives/sends "clean" traffic.

We are using the following configuration snippet to achieve that.

---------------------------------------------------------------------
#!/bin/bash
P0_OUTER_REMOTE_IP=192.168.50.2
P0_OUTER_LOCAL_IP=192.168.50.1
PF0=enp8s0f0
VF0_REP=enp8s0f0_0

set -v
# Configure IP and turn VF_REP on
ifconfig $PF0 $P0_OUTER_LOCAL_IP/24 up
ifconfig $VF0_REP up

# Clean all TC rules, start fresh
tc qdisc del dev enp8s0f0 ingress >/dev/null 2>&1 tc qdisc del dev enp8s0f0_0 ingress >/dev/null 2>&1

# Make sure steering mode is dmfs(FW) and eswitch encap is none
devlink dev param set pci/0000:08:00.0 name flow_steering_mode value dmfs cmode runtime devlink dev eswitch set pci/0000:08:00.0 mode legacy
devlink dev eswitch set pci/0000:08:00.0 encap none
devlink dev eswitch set pci/0000:08:00.0 mode switchdev

sleep 2

tc qdisc add dev enp8s0f0 ingress
tc qdisc add dev enp8s0f0_0 ingress

# Add TC rules
tc filter add dev $PF0 parent ffff: protocol 802.1q chain 0 flower vlan_id 10 vlan_ethtype 802.1q cvlan_id 5 action vlan pop action vlan pop action mirred egress redirect dev $VF0_REP tc filter add dev $VF0_REP parent ffff: protocol all chain 0 flower action vlan push protocol 802.1q id 5 action vlan push protocol 802.1q id 10 action mirred egress redirect dev $PF0 tc filter show dev $PF0 ingress
----------------------------------------------------------------------------------------------------

We also don't offload anything related to routing as we can't differentiate between local traffic.

Quoted 1 line

Also the perf traces, I don't see them here.

It is worth to separate it to standalone discussion with a title:
"why crypto is not fast enough?". I don't think that mixed discussions about full offload which Steffen said that he is interested and research about crypto bottlenecks will be productive. These discussions are orthogonal.

Quoted 12 lines · On Fri, 26 Aug 2022 09:26:57 +0300 Leon Romanovsky wrote:

Quoted 11 lines · Jakub Kicinski

Quoted 7 lines · On Tue, 23 Aug 2022 16:31:57 +0300 Leon Romanovsky wrote:

* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.

Since the use case is somewhat in question, perhaps switch to RFC postings until the drivers side incl. tc forwarding is implemented?

Quoted 2 lines

We also don't offload anything related to routing as we can't differentiate between local traffic.

Yeah, nah, that's not what I'm asking for.
I said forwarding, not sending traffic thru a different virtual interface. The TC rules must forward from or two the IPSec ifc.

That was the use case Jason mentioned.

Quoted 8 lines

Quoted 2 lines

Also the perf traces, I don't see them here.

It is worth to separate it to standalone discussion with a title:
"why crypto is not fast enough?". I don't think that mixed discussions about full offload which Steffen said that he is interested and research about crypto bottlenecks will be productive. These discussions are orthogonal.

What do you mean by crypto bottlenecks?

Please use more precise language. crypto here may mean "crypto only offload" or "crypto as done by CPU". I have no idea which one you mean.

We are very much interested in the former, the latter is indeed out of scope here.

Quoted 6 lines

# Add TC rules
tc filter add dev $PF0 parent ffff: protocol 802.1q chain 0 flower vlan_id 10 vlan_ethtype 802.1q cvlan_id 5 action vlan pop action vlan pop action mirred egress redirect dev $VF0_REP tc filter add dev $VF0_REP parent ffff: protocol all chain 0 flower action vlan push protocol 802.1q id 5 action vlan push protocol 802.1q id 10 action mirred egress redirect dev $PF0 tc filter show dev $PF0 ingress
----------------------------------------------------------------------------------------------------

Quoted 2 lines

We also don't offload anything related to routing as we can't differentiate between local traffic.

Yeah, nah, that's not what I'm asking for.
I said forwarding, not sending traffic thru a different virtual interface. The TC rules must forward from or two the IPSec ifc.

That was the use case Jason mentioned.

I was meaning rather generically handling the packets in the hypervisor side without involving the CPU.

We have customers deploying many different models for this in their hypervisor, including a significant deployment using a model like the above.

It achieves a kind of connectivity to a VM with 0 hypervisor CPU involvement with the vlan push/pop done in HW.

We other use-models, like the offloaded OVS switching model you are alluding to, that is Leon has as a followup.

Quoted 3 lines · On Tue, 30 Aug 2022 14:36:52 -0300 Jason Gunthorpe wrote:

I was meaning rather generically handling the packets in the hypervisor side without involving the CPU.

Quoted 6 lines

We have customers deploying many different models for this in their hypervisor, including a significant deployment using a model like the above.

It achieves a kind of connectivity to a VM with 0 hypervisor CPU involvement with the vlan push/pop done in HW.

I don't see how that necessitates the full IPsec offload.

Sorry, perhaps I assumed we were on the same page too hastily.

I was referring to a model where the TC rules redirect to a IPsec tunnel which is then routed to the uplink. All offloaded. Like we do today for VxLAN encap/decap.

Quoted 9 lines · Jakub Kicinski

Quoted 7 lines · On Tue, 23 Aug 2022 16:31:57 +0300 Leon Romanovsky wrote:

* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.

Quoted 2 lines

Since the use case is somewhat in question, perhaps switch to RFC postings until the drivers side incl. tc forwarding is implemented?

Quoted 2 lines

Also the perf traces, I don't see them here.

It is worth to separate it to standalone discussion with a title:
"why crypto is not fast enough?".

I don't think that mixed discussions
about full offload which Steffen said that he is interested and research about crypto bottlenecks will be productive. These discussions are orthogonal.

From: Leon Romanovsky <[email protected]>

Changelog:
v3:
* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.
v2: https://lore.kernel.org/all/[email protected]
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

The following series extends XFRM core code to handle a new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for the whole data path, so both policy and state should be offloaded.

IPsec full offload is an improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

Devices that implement IPsec full offload mode offload policies too. In the RX path, it causes the situation that HW can't effectively handle mixed SW and HW priorities unless users make sure that HW offloaded policies have higher priorities.

To make sure that users have a coherent picture, we require that HW offloaded policies have always (both RX and TX) higher priorities than SW ones.

To not over-engineer the code, HW policies are treated as SW ones and don't take into account netdev to allow reuse of the same priorities for different devices.

There are several deliberate limitations:
* No software fallback
* Fragments are dropped, both in RX and TX
* No sockets policies
* Only IPsec transport mode is implemented

Quoted 45 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog:
v3:
* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.
v2: https://lore.kernel.org/all/[email protected]
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

The following series extends XFRM core code to handle a new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for the whole data path, so both policy and state should be offloaded.

IPsec full offload is an improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

Devices that implement IPsec full offload mode offload policies too. In the RX path, it causes the situation that HW can't effectively handle mixed SW and HW priorities unless users make sure that HW offloaded policies have higher priorities.

To make sure that users have a coherent picture, we require that HW offloaded policies have always (both RX and TX) higher priorities than SW ones.

To not over-engineer the code, HW policies are treated as SW ones and don't take into account netdev to allow reuse of the same priorities for different devices.

Quoted 5 lines

There are several deliberate limitations:
* No software fallback
* Fragments are dropped, both in RX and TX
* No sockets policies
* Only IPsec transport mode is implemented

... and you still have not answered the fundamental questions:

As implemented, the software does not hold any state. I.e. there is no sync between hardware and software regarding stats, liftetime, lifebyte, packet counts and replay window. IKE rekeying and auditing is based on these, how should this be done?

How can tunnel mode work with this offload?

I want to see the full picture before I consider to apply this.

Quoted 45 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog:
v3:
* I didn't hear any suggestion what term to use instead of
"full offload", so left it as is. It is used in commit messages and documentation only and easy to rename.
* Added performance data and background info to cover letter
* Reused xfrm_output_resume() function to support multiple XFRM transformations
* Add PMTU check in addition to driver .xdo_dev_offload_ok validation
* Documentation is in progress, but not part of this series yet.
v2: https://lore.kernel.org/all/[email protected]
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

The following series extends XFRM core code to handle a new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for the whole data path, so both policy and state should be offloaded.

IPsec full offload is an improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

Devices that implement IPsec full offload mode offload policies too. In the RX path, it causes the situation that HW can't effectively handle mixed SW and HW priorities unless users make sure that HW offloaded policies have higher priorities.

To make sure that users have a coherent picture, we require that HW offloaded policies have always (both RX and TX) higher priorities than SW ones.

To not over-engineer the code, HW policies are treated as SW ones and don't take into account netdev to allow reuse of the same priorities for different devices.

Quoted 5 lines

There are several deliberate limitations:
* No software fallback
* Fragments are dropped, both in RX and TX
* No sockets policies
* Only IPsec transport mode is implemented

... and you still have not answered the fundamental questions:

As implemented, the software does not hold any state. I.e. there is no sync between hardware and software regarding stats, liftetime, lifebyte, packet counts and replay window. IKE rekeying and auditing is based on these, how should this be done?

How can tunnel mode work with this offload?

I want to see the full picture before I consider to apply this.