Extend XFRM core to allow full offload configuration

From: Leon Romanovsky <[email protected]>

Allow users to configure xfrm states with full offload mode. The full mode must be requested both for policy and state, and such requires us to do not implement fallback.

We explicitly return an error if requested full mode can't be configured.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

Quoted 6 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Allow users to configure xfrm states with full offload mode. The full mode must be requested both for policy and state, and such requires us to do not implement fallback.

Quoted 5 lines

We explicitly return an error if requested full mode can't be configured.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

This one needs to be ACKed by the driver maintainers.

Quoted 10 lines · Steffen Klassert

Quoted 9 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Allow users to configure xfrm states with full offload mode. The full mode must be requested both for policy and state, and such requires us to do not implement fallback.

We explicitly return an error if requested full mode can't be configured.

Quoted 4 lines

Quoted 2 lines

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

This one needs to be ACKed by the driver maintainers.

Why? Only crypto is supported in upstream kernel.

Quoted 14 lines · Leon Romanovsky

Quoted 13 lines · Steffen Klassert

Quoted 11 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Allow users to configure xfrm states with full offload mode. The full mode must be requested both for policy and state, and such requires us to do not implement fallback.

We explicitly return an error if requested full mode can't be configured.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

Quoted 3 lines

Quoted 1 line

This one needs to be ACKed by the driver maintainers.

Why? Only crypto is supported in upstream kernel.

Because it touches code hat is under their maintainance. They should at least be CCed on this patch.

From: Leon Romanovsky <[email protected]>

In IPsec full mode, the device is going to encrypt and encapsulate packets that are associated with offloaded policy. After successful policy lookup to indicate if packets should be offloaded or not, the stack forwards packets to the device to do the magic.

Signed-off-by Raed Salem <[email protected]>

Signed-off-by Huy Nguyen <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 net/xfrm/xfrm_device.c | 13 +++++++++++++
 net/xfrm/xfrm_output.c | 20 ++++++++++++++++++++
 2 files changed, 33 insertions(+)

net/xfrm/xfrm_device.c +0 -0

index 1cc482e9c87d..db5ebd36f68c 100644

net/xfrm/xfrm_output.c +8 -0

index 555ab35cd119..27a8dac9ca7d 100644
@@ -719,6 +719,26 @@ int xfrm_output(struct sock *sk, struct sk_buff *skb)
 		break;
 	}
 
+	if (x->xso.type == XFRM_DEV_OFFLOAD_FULL) {
+		struct dst_entry *dst = skb_dst_pop(skb);
+
+		if (!dst || !xfrm_dev_offload_ok(skb, x)) {
+			XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
+			kfree_skb(skb);
+			return -EHOSTUNREACH;
+		}

Quoted 24 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

In IPsec full mode, the device is going to encrypt and encapsulate packets that are associated with offloaded policy. After successful policy lookup to indicate if packets should be offloaded or not, the stack forwards packets to the device to do the magic.

Signed-off-by Raed Salem <[email protected]>

Signed-off-by Huy Nguyen <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 net/xfrm/xfrm_device.c | 13 +++++++++++++
 net/xfrm/xfrm_output.c | 20 ++++++++++++++++++++
 2 files changed, 33 insertions(+)

net/xfrm/xfrm_device.c +0 -0

index 1cc482e9c87d..db5ebd36f68c 100644
@@ -366,6 +376,9 @@ bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x)
 	struct xfrm_dst *xdst = (struct xfrm_dst *)dst;
 	struct net_device *dev = x->xso.dev;
 

net/xfrm/xfrm_device.c bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) · line 380

Quoted 2 lines

net/xfrm/xfrm_device.c +2 -0

+	if (x->xso.type == XFRM_DEV_OFFLOAD_FULL)
+		goto ok;

You skip the PMTU checks here. I've seen that you check the packet length against the device MTU in one of your mlx5 patches, but that does not help if the PMTU is below.

Quoted 23 lines

net/xfrm/xfrm_output.c +13 -0

index 555ab35cd119..27a8dac9ca7d 100644
@@ -719,6 +719,26 @@ int xfrm_output(struct sock *sk, struct sk_buff *skb)
 		break;
 	}
 
+	if (x->xso.type == XFRM_DEV_OFFLOAD_FULL) {
+		struct dst_entry *dst = skb_dst_pop(skb);
+
+		if (!dst || !xfrm_dev_offload_ok(skb, x)) {
+			XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
+			kfree_skb(skb);
+			return -EHOSTUNREACH;
+		}
+
+		skb_dst_set(skb, dst);
+		err = skb_dst(skb)->ops->local_out(net, skb->sk, skb);
+		if (unlikely(err != 1))
+			return err;

net/xfrm/xfrm_output.c int xfrm_output(struct sock *sk, struct sk_buff *skb) · line 739

Quoted 5 lines

net/xfrm/xfrm_output.c +5 -0

+
+		if (!skb_dst(skb)->xfrm)
+			return dst_output(net, skb->sk, skb);
+
+		return 0;

You leak skb here. Also, this skb needs another tfm because skb_dst(skb)->xfrm is set.

Quoted 25 lines · Steffen Klassert

Quoted 24 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

In IPsec full mode, the device is going to encrypt and encapsulate packets that are associated with offloaded policy. After successful policy lookup to indicate if packets should be offloaded or not, the stack forwards packets to the device to do the magic.

Signed-off-by Raed Salem <[email protected]>

Signed-off-by Huy Nguyen <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 net/xfrm/xfrm_device.c | 13 +++++++++++++
 net/xfrm/xfrm_output.c | 20 ++++++++++++++++++++
 2 files changed, 33 insertions(+)

net/xfrm/xfrm_device.c +0 -0

index 1cc482e9c87d..db5ebd36f68c 100644
@@ -366,6 +376,9 @@ bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x)
 	struct xfrm_dst *xdst = (struct xfrm_dst *)dst;
 	struct net_device *dev = x->xso.dev;
 

Quoted 6 lines

net/xfrm/xfrm_device.c bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) · line 380

Quoted 2 lines

net/xfrm/xfrm_device.c +2 -0

+	if (x->xso.type == XFRM_DEV_OFFLOAD_FULL)
+		goto ok;

You skip the PMTU checks here. I've seen that you check the packet length against the device MTU in one of your mlx5 patches, but that does not help if the PMTU is below.

If device supports transformation of the packet, this packet won't be counted as XFRM anymore. I'm not sure that we need to perform XFRM specific checks.

Quoted 33 lines · Leon Romanovsky

Quoted 31 lines · Steffen Klassert

Quoted 21 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

In IPsec full mode, the device is going to encrypt and encapsulate packets that are associated with offloaded policy. After successful policy lookup to indicate if packets should be offloaded or not, the stack forwards packets to the device to do the magic.

Signed-off-by Raed Salem <[email protected]>

Signed-off-by Huy Nguyen <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 net/xfrm/xfrm_device.c | 13 +++++++++++++
 net/xfrm/xfrm_output.c | 20 ++++++++++++++++++++
 2 files changed, 33 insertions(+)

net/xfrm/xfrm_device.c +0 -0

index 1cc482e9c87d..db5ebd36f68c 100644

net/xfrm/xfrm_device.c bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) · line 380

Quoted 5 lines

net/xfrm/xfrm_device.c +2 -0

@@ -366,6 +376,9 @@ bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x)
 	struct xfrm_dst *xdst = (struct xfrm_dst *)dst;
 	struct net_device *dev = x->xso.dev;
 
+	if (x->xso.type == XFRM_DEV_OFFLOAD_FULL)
+		goto ok;

You skip the PMTU checks here. I've seen that you check the packet length against the device MTU in one of your mlx5 patches, but that does not help if the PMTU is below.

Quoted 3 lines

If device supports transformation of the packet, this packet won't be counted as XFRM anymore. I'm not sure that we need to perform XFRM specific checks.

This is not xfrm specific, it makes sure that the packet you want to send fits into the PMTU.

From: Leon Romanovsky <[email protected]>

Traffic received by device with enabled IPsec full offload should be forwarded to the stack only after decryption, packet headers and trailers removed.

Such packets are expected to be seen as normal (non-XFRM) ones, while not-supported packets should be dropped by the HW.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

include/net/xfrm.h +2 -0

@@ -1125,6 +1148,15 @@ static inline int __xfrm_policy_check2(struct sock *sk, int dir,
 {
 	struct net *net = dev_net(skb->dev);
 	int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
+	struct xfrm_offload *xo = xfrm_offload(skb);
+	struct xfrm_state *x;

Quoted 26 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Traffic received by device with enabled IPsec full offload should be forwarded to the stack only after decryption, packet headers and trailers removed.

Such packets are expected to be seen as normal (non-XFRM) ones, while not-supported packets should be dropped by the HW.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

include/net/xfrm.h +9 -0

@@ -1125,6 +1148,15 @@ static inline int __xfrm_policy_check2(struct sock *sk, int dir,
 {
 	struct net *net = dev_net(skb->dev);
 	int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
+	struct xfrm_offload *xo = xfrm_offload(skb);
+	struct xfrm_state *x;
+
+	if (xo) {
+		x = xfrm_input_state(skb);
+		if (x->xso.type == XFRM_DEV_OFFLOAD_FULL)
+			return (xo->flags & CRYPTO_DONE) &&
+			       (xo->status & CRYPTO_SUCCESS);
+	}
 

include/net/xfrm.h static inline int __xfrm_policy_check2(struct sock *sk, int dir, · line 1162

Quoted 2 lines

include/net/xfrm.h +0 -0

 	if (sk && sk->sk_policy[XFRM_POLICY_IN])
 		return __xfrm_policy_check(sk, ndir, skb, family);

What happens here if there is a socket policy configured?

Quoted 27 lines · Steffen Klassert

Quoted 26 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Traffic received by device with enabled IPsec full offload should be forwarded to the stack only after decryption, packet headers and trailers removed.

Such packets are expected to be seen as normal (non-XFRM) ones, while not-supported packets should be dropped by the HW.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

include/net/xfrm.h +9 -0

@@ -1125,6 +1148,15 @@ static inline int __xfrm_policy_check2(struct sock *sk, int dir,
 {
 	struct net *net = dev_net(skb->dev);
 	int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
+	struct xfrm_offload *xo = xfrm_offload(skb);
+	struct xfrm_state *x;
+
+	if (xo) {
+		x = xfrm_input_state(skb);
+		if (x->xso.type == XFRM_DEV_OFFLOAD_FULL)
+			return (xo->flags & CRYPTO_DONE) &&
+			       (xo->status & CRYPTO_SUCCESS);
+	}
 

Quoted 4 lines

include/net/xfrm.h static inline int __xfrm_policy_check2(struct sock *sk, int dir, · line 1162

Quoted 2 lines

include/net/xfrm.h +0 -0

 	if (sk && sk->sk_policy[XFRM_POLICY_IN])
 		return __xfrm_policy_check(sk, ndir, skb, family);

What happens here if there is a socket policy configured?

No change, we don't support offload of socket policies.

Quoted 33 lines · Leon Romanovsky

Quoted 31 lines · Steffen Klassert

Quoted 18 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Traffic received by device with enabled IPsec full offload should be forwarded to the stack only after decryption, packet headers and trailers removed.

Such packets are expected to be seen as normal (non-XFRM) ones, while not-supported packets should be dropped by the HW.

Reviewed-by Raed Salem <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

include/net/xfrm.h +2 -0

@@ -1125,6 +1148,15 @@ static inline int __xfrm_policy_check2(struct sock *sk, int dir,
 {
 	struct net *net = dev_net(skb->dev);
 	int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
+	struct xfrm_offload *xo = xfrm_offload(skb);
+	struct xfrm_state *x;

include/net/xfrm.h static inline int __xfrm_policy_check2(struct sock *sk, int dir, · line 1162

Quoted 10 lines

include/net/xfrm.h +7 -0

+
+	if (xo) {
+		x = xfrm_input_state(skb);
+		if (x->xso.type == XFRM_DEV_OFFLOAD_FULL)
+			return (xo->flags & CRYPTO_DONE) &&
+			       (xo->status & CRYPTO_SUCCESS);
+	}
 
 	if (sk && sk->sk_policy[XFRM_POLICY_IN])
 		return __xfrm_policy_check(sk, ndir, skb, family);

What happens here if there is a socket policy configured?

Quoted 1 line

No change, we don't support offload of socket policies.

But the user can confugure it, so it should be enforced regardless if we had an offload before.

The following series extends XFRM core code to handle new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for whole data path, so both policy and state should be offloaded.

Quoted 5 lines · On Tue, 16 Aug 2022 11:59:21 +0300 Leon Romanovsky wrote:

The following series extends XFRM core code to handle new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for whole data path, so both policy and state should be offloaded.

This is making a precedent for full tunnel offload in netdev, right?

Could you indulge us with a more detailed description, motivation, performance results, where the behavior of offload may differ (if at all), what visibility users have, how SW and HW work together on the datapath? Documentation would be great.

Quoted 1 line · Jakub Kicinski

This is making a precedent for full tunnel offload in netdev, right?

Not really. SW IPsec supports two modes: tunnel and transport.

However HW and SW stack supports only offload of transport mode. This is the case for already merged IPsec crypto offload mode and the case for this full offload.

Quoted 4 lines

Could you indulge us with a more detailed description, motivation, performance results, where the behavior of offload may differ (if at all), what visibility users have, how SW and HW work together on the datapath? Documentation would be great.

IPsec full offload is actually improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

My main motivation is to perform IPsec on RoCE traffic and in our preliminary results, we are able to do IPsec full offload in line rate. The same goes for ETH traffic.

Regarding behavior differences - they are not expected.

We (Raed and me) tried very hard to make sure that IPsec full offload will behave exactly as SW path.

There are some limitations to reduce complexity, but they can be removed later if needs will arise. Right now, none of them are "real" limitations for various *swarn forks, which we extend as well.

Some of them:
1. Request to have reqid for policy and state. I use reqid for HW matching between policy and state.

2. Automatic separation between HW and SW priorities, because HW sees packet first.

3. Only main template is supported.
4. No fallback to SW if IPsec HW failed to handle packet. HW should drop such packet.

Visibility:
Users can see the mode through iproute2
https://lore.kernel.org/netdev/[email protected]/
and see statistics through ethtool.

Documentation will come as well. I assume that IPsec folks are familiar with this topic as it was discussed in IPsec coffee hour.

Quoted 8 lines · On Wed, 17 Aug 2022 08:22:02 +0300 Leon Romanovsky wrote:

Quoted 1 line · Jakub Kicinski

This is making a precedent for full tunnel offload in netdev, right?

Not really. SW IPsec supports two modes: tunnel and transport.

However HW and SW stack supports only offload of transport mode. This is the case for already merged IPsec crypto offload mode and the case for this full offload.

My point is on what you called "full offload" vs "crypto offload". The policy so far has always been that Linux networking stack should populate all the headers and instruct the device to do crypto, no header insertion. Obviously we do header insertion in switch/router offloads but that's different and stateless.

I believe the reasoning was to provide as much flexibility and control to the software as possible while retaining most of the performance gains.

You must provide a clear analysis (as in examination in data) and discussion (as in examination in writing) if you're intending to change the "let's keep packet formation in the SW" policy. What you got below is a good start but not sufficient.

Quoted 11 lines

Quoted 5 lines

Could you indulge us with a more detailed description, motivation, performance results, where the behavior of offload may differ (if at all), what visibility users have, how SW and HW work together on the datapath? Documentation would be great.

IPsec full offload is actually improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

Quoted 3 lines

My main motivation is to perform IPsec on RoCE traffic and in our preliminary results, we are able to do IPsec full offload in line rate. The same goes for ETH traffic.

If the motivation is RoCE I personally see no reason to provide the configuration of this functionality via netdev interfaces, but I'll obviously leave the final decision to Steffen.

Quoted 10 lines

Regarding behavior differences - they are not expected.

We (Raed and me) tried very hard to make sure that IPsec full offload will behave exactly as SW path.

There are some limitations to reduce complexity, but they can be removed later if needs will arise. Right now, none of them are "real" limitations for various *swarn forks, which we extend as well.

Quoted 3 lines

Some of them:
1. Request to have reqid for policy and state. I use reqid for HW matching between policy and state.

reqid?

Quoted 3 lines

2. Automatic separation between HW and SW priorities, because HW sees packet first.

More detail needed on that.

Quoted 4 lines

3. Only main template is supported.
4. No fallback to SW if IPsec HW failed to handle packet. HW should drop such packet.

Not great for debug.

Quoted 5 lines

Visibility:
Users can see the mode through iproute2
https://lore.kernel.org/netdev/[email protected]/
and see statistics through ethtool.

Custom vendor stats?

Quoted 3 lines

Documentation will come as well. I assume that IPsec folks are familiar with this topic as it was discussed in IPsec coffee hour.

Quoted 16 lines · Jakub Kicinski

Quoted 8 lines · On Wed, 17 Aug 2022 08:22:02 +0300 Leon Romanovsky wrote:

Quoted 1 line · Jakub Kicinski

This is making a precedent for full tunnel offload in netdev, right?

Not really. SW IPsec supports two modes: tunnel and transport.

However HW and SW stack supports only offload of transport mode. This is the case for already merged IPsec crypto offload mode and the case for this full offload.

My point is on what you called "full offload" vs "crypto offload". The policy so far has always been that Linux networking stack should populate all the headers and instruct the device to do crypto, no header insertion. Obviously we do header insertion in switch/router offloads but that's different and stateless.

Quoted 3 lines

I believe the reasoning was to provide as much flexibility and control to the software as possible while retaining most of the performance gains.

I honestly don't know the reasoning, but "performance gains" are very limited as long as IPsec stack involved with various policy/state lookups. These lookups are expensive in terms of CPU and they can't hold 400 Gb/s line rate.

https://docs.nvidia.com/networking/display/connectx7en/Introduction#Introduction-ConnectX-7400GbEAdapterCards

Quoted 5 lines

You must provide a clear analysis (as in examination in data) and discussion (as in examination in writing) if you're intending to change the "let's keep packet formation in the SW" policy. What you got below is a good start but not sufficient.

Quoted 5 lines

Quoted 3 lines

Some of them:
1. Request to have reqid for policy and state. I use reqid for HW matching between policy and state.

reqid?

Policy and state are matched based on their selectors (src/deet IP, direction ...), but they independent. The reqid is XFRM identification that this specific policy is connected to this specific state.
https://www.man7.org/linux/man-pages/man8/ip-xfrm.8.html
https://docs.kernel.org/networking/xfrm_device.html
ip x s add ....
reqid 0x07 ...
offload dev eth4 dir in

Quoted 17 lines · Leon Romanovsky

Quoted 16 lines · Jakub Kicinski

Quoted 8 lines · On Wed, 17 Aug 2022 08:22:02 +0300 Leon Romanovsky wrote:

Quoted 1 line · Jakub Kicinski

This is making a precedent for full tunnel offload in netdev, right?

Not really. SW IPsec supports two modes: tunnel and transport.

However HW and SW stack supports only offload of transport mode. This is the case for already merged IPsec crypto offload mode and the case for this full offload.

My point is on what you called "full offload" vs "crypto offload". The policy so far has always been that Linux networking stack should populate all the headers and instruct the device to do crypto, no header insertion. Obviously we do header insertion in switch/router offloads but that's different and stateless.

Quoted 8 lines

Quoted 3 lines

I believe the reasoning was to provide as much flexibility and control to the software as possible while retaining most of the performance gains.

I honestly don't know the reasoning, but "performance gains" are very limited as long as IPsec stack involved with various policy/state lookups. These lookups are expensive in terms of CPU and they can't hold 400 Gb/s line rate.

Can you provide some performance results that show the difference between crypto and full offload? In particular because on the TX path, the full policy and state offload is done twice (in software to find the offloading device and then in hardware to match policy to state).

Quoted 9 lines

https://docs.nvidia.com/networking/display/connectx7en/Introduction#Introduction-ConnectX-7400GbEAdapterCards

Quoted 5 lines

You must provide a clear analysis (as in examination in data) and discussion (as in examination in writing) if you're intending to change the "let's keep packet formation in the SW" policy. What you got below is a good start but not sufficient.

I'm still a bit unease about this approach. I fear that doing parts of statefull IPsec procesing in software and parts in hardware will lead to all sort of problems. E.g. with this implementation the software has no stats, liftetime, lifebyte and packet count information but is responsible to do the IKE communication.

We might be able to sort out all problems during the upstraming process, but I still have no clear picture how this should work in the end with all corener cases this creates.

Also the name full offload is a bit missleading, because the software still has to hold all offloaded states and policies. In a full offload, the stack would IMO just act as a stub layer between IKE and hardware.

Quoted 7 lines

Quoted 6 lines

Quoted 4 lines

Some of them:
1. Request to have reqid for policy and state. I use reqid for HW matching between policy and state.

reqid?

Quoted 8 lines

Policy and state are matched based on their selectors (src/deet IP, direction ...), but they independent. The reqid is XFRM identification that this specific policy is connected to this specific state.
https://www.man7.org/linux/man-pages/man8/ip-xfrm.8.html
https://docs.kernel.org/networking/xfrm_device.html
ip x s add ....
reqid 0x07 ...
offload dev eth4 dir in

Can you elaborate this a bit more? Does that matching happen in hardware? The reqid is not a unique identifyer to match between policy and state. You MUST match the selectors as defined in
https://www.rfc-editor.org/rfc/rfc4301

Quoted 4 lines

Quoted 4 lines

You must provide a clear analysis (as in examination in data) and discussion (as in examination in writing) if you're intending to change the "let's keep packet formation in the SW" policy. What you got below is a good start but not sufficient.

I'm still a bit unease about this approach. I fear that doing parts of statefull IPsec procesing in software and parts in hardware will lead to all sort of problems. E.g. with this implementation the software has no stats, liftetime, lifebyte and packet count information but is responsible to do the IKE communication.

We might be able to sort out all problems during the upstraming process, but I still have no clear picture how this should work in the end with all corener cases this creates.

Also the name full offload is a bit missleading, because the software still has to hold all offloaded states and policies. In a full offload, the stack would IMO just act as a stub layer between IKE and hardware.

Quoted 6 lines · Jakub Kicinski

My point is on what you called "full offload" vs "crypto offload". The policy so far has always been that Linux networking stack should populate all the headers and instruct the device to do crypto, no header insertion. Obviously we do header insertion in switch/router offloads but that's different and stateless.

Quoted 3 lines

I believe the reasoning was to provide as much flexibility and control to the software as possible while retaining most of the performance gains.

I honestly don't know the reasoning, but "performance gains" are very limited as long as IPsec stack involved with various policy/state

lookups. These lookups are expensive in terms of CPU and they can't hold 400 Gb/s line rate.

https://docs.nvidia.com/networking/display/connectx7en/Introduction#Introduction-ConnectX-7400GbEAdapterCards

Quoted 4 lines

You must provide a clear analysis (as in examination in data) and discussion (as in examination in writing) if you're intending to change the "let's keep packet formation in the SW" policy. What you got below is a good start but not sufficient.

Can you please point me to an example of such analysis, so I will know what is needed/expected?

Quoted 5 lines

Quoted 5 lines

IPsec full offload is actually improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

Quoted 7 lines

Quoted 3 lines

My main motivation is to perform IPsec on RoCE traffic and in our preliminary results, we are able to do IPsec full offload in line rate. The same goes for ETH traffic.

If the motivation is RoCE I personally see no reason to provide the configuration of this functionality via netdev interfaces, but I'll obviously leave the final decision to Steffen.

This is not limited to RoCE, our customers use this offload for ethernet traffic as well.

RoCE is a good example of traffic that performs all headers magic in HW, without SW involved.

IPsec clearly belongs to netdev and we don't want to duplicate netdev functionality in RDMA. Like I said above, this feature is needed for regular ETH traffic as well.

Right now, RoCE and iWARP devices are based on netdev and long-standing agreement ( >20 years ????) that all netdev configurations are done there they belong - in netdev.

Quoted 11 lines · On Thu, 18 Aug 2022 08:24:13 +0300 Leon Romanovsky wrote:

Quoted 9 lines · Jakub Kicinski

My point is on what you called "full offload" vs "crypto offload". The policy so far has always been that Linux networking stack should populate all the headers and instruct the device to do crypto, no header insertion. Obviously we do header insertion in switch/router offloads but that's different and stateless.

I believe the reasoning was to provide as much flexibility and control to the software as possible while retaining most of the performance gains.

Quoted 2 lines

I honestly don't know the reasoning, but "performance gains" are very limited as long as IPsec stack involved with various policy/state

Herm. So you didn't bother figuring out what the current problems are but unsurprisingly the solution is "buy our product and let us do it"?

Quoted 11 lines

lookups. These lookups are expensive in terms of CPU and they can't hold 400 Gb/s line rate.

https://docs.nvidia.com/networking/display/connectx7en/Introduction#Introduction-ConnectX-7400GbEAdapterCards

Quoted 4 lines

You must provide a clear analysis (as in examination in data) and discussion (as in examination in writing) if you're intending to change the "let's keep packet formation in the SW" policy. What you got below is a good start but not sufficient.

Quoted 2 lines

Can you please point me to an example of such analysis, so I will know what is needed/expected?

I can't, as I said twice now, we don't have any "full crypto" offloads AFAIK.

Quoted 20 lines

Quoted 6 lines

Quoted 6 lines

IPsec full offload is actually improved version of IPsec crypto mode, In full mode, HW is responsible to trim/add headers in addition to decrypt/encrypt. In this mode, the packet arrives to the stack as already decrypted and vice versa for TX (exits to HW as not-encrypted).

Quoted 7 lines

Quoted 3 lines

My main motivation is to perform IPsec on RoCE traffic and in our preliminary results, we are able to do IPsec full offload in line rate. The same goes for ETH traffic.

If the motivation is RoCE I personally see no reason to provide the configuration of this functionality via netdev interfaces, but I'll obviously leave the final decision to Steffen.

This is not limited to RoCE, our customers use this offload for ethernet traffic as well.

RoCE is a good example of traffic that performs all headers magic in HW, without SW involved.

Quoted 7 lines

IPsec clearly belongs to netdev and we don't want to duplicate netdev functionality in RDMA. Like I said above, this feature is needed for regular ETH traffic as well.

Right now, RoCE and iWARP devices are based on netdev and long-standing agreement ( >20 years ????) that all netdev configurations are done there they belong - in netdev.

Let me be very clear - as far as I'm concerned no part of the RDMA stack belongs in netdev. What's there is there, but do not try to use that argument to justify more stuff.

If someone from the community thinks that I should have interest in working on / helping proprietary protocol stacks please let me know, because right now I have none.

Quoted 7 lines

Let me be very clear - as far as I'm concerned no part of the RDMA stack belongs in netdev. What's there is there, but do not try to use that argument to justify more stuff.

If someone from the community thinks that I should have interest in working on / helping proprietary protocol stacks please let me know, because right now I have none.

No one is asking from you to work on proprietary protocols.

RoCE is IBTA standard protocol and iWARP is IETF one. They both fully documented and backed by multiple vendors (Intel, IBM, Mellanox, Cavium ...).

There is also interoperability lab https://www.iol.unh.edu/ that runs various tests. In addition to distro interoperability labs testing.

I invite you to take a look on Jason's presentation "Challenges of the RDMA subsystem", which he gave 3 years ago, about RDMA and challenges with netdev.
https://lpc.events/event/4/contributions/364/

Quoted 4 lines

I invite you to take a look on Jason's presentation "Challenges of the RDMA subsystem", which he gave 3 years ago, about RDMA and challenges with netdev.
https://lpc.events/event/4/contributions/364/

I appreciate the invite, but it's not high enough on my list of interest to spend time on.

Regardless, RDMA doesn't really intersect with this netdev work for XFRM beyond the usual ways that RDMA IP traffic can be captured by or run parallel to netdev.

A significant use case here is for switchdev modes where the switch will subject traffic from a switch port to ESP, not unlike it already does with vlan, vxlan, etc and other already fully offloaded switching transforms.

Quoted 4 lines · On Fri, 19 Aug 2022 13:01:22 -0300 Jason Gunthorpe wrote:

Regardless, RDMA doesn't really intersect with this netdev work for XFRM beyond the usual ways that RDMA IP traffic can be captured by or run parallel to netdev.

Quoted 4 lines

A significant use case here is for switchdev modes where the switch will subject traffic from a switch port to ESP, not unlike it already does with vlan, vxlan, etc and other already fully offloaded switching transforms.

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Quoted 10 lines · Jakub Kicinski

Quoted 8 lines · On Fri, 19 Aug 2022 13:01:22 -0300 Jason Gunthorpe wrote:

Regardless, RDMA doesn't really intersect with this netdev work for XFRM beyond the usual ways that RDMA IP traffic can be captured by or run parallel to netdev.

A significant use case here is for switchdev modes where the switch will subject traffic from a switch port to ESP, not unlike it already does with vlan, vxlan, etc and other already fully offloaded switching transforms.

Quoted 2 lines

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Yes, everything that can help to overcome the performance problems can help and I'm interested in this type of offload. But we need to make sure the API is usable by the whole community, so I don't want an API for some special case one of the NIC vendors is interested in.

Quoted 9 lines · Steffen Klassert

Quoted 2 lines · Jakub Kicinski

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Yes, everything that can help to overcome the performance problems can help and I'm interested in this type of offload. But we need to make sure the API is usable by the whole community, so I don't want an API for some special case one of the NIC vendors is interested in.

BTW, we have a performance data, I planned to send it as part of cover letter for v3, but it is worth to share it now.

================================================================================
Performance results:

TCP multi-stream, using iperf3 instance per-CPU.

 +----------------------+--------+--------+--------+--------+---------+---------+
 |                      | 1 CPU  | 2 CPUs | 4 CPUs | 8 CPUs | 16 CPUs | 32 CPUs |
 |                      +--------+--------+--------+--------+---------+---------+
 |                      |                   BW (Gbps)                           |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Baseline             | 27.9   | 59     | 93.1  | 92.8    | 93.7    | 94.4    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Software IPsec       | 6      | 11.9   | 23.3  | 45.9    | 83.8    | 91.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec crypto offload | 15     | 29.7   | 58.5  | 89.6    | 90.4    | 90.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec full offload   | 28     | 57     | 90.7  | 91      | 91.3    | 91.9    |
 +----------------------+--------+--------+-------+---------+---------+---------+

IPsec full offload mode behaves as baseline and reaches linerate with same amount of CPUs.

Setups details (similar for both sides):
* NIC: ConnectX6-DX dual port, 100 Gbps each. Single port used in the tests.
* CPU: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz

Quoted 32 lines · On Mon, 22 Aug 2022 11:54:42 +0300 Leon Romanovsky wrote:

Quoted 9 lines · Steffen Klassert

Quoted 2 lines · Jakub Kicinski

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Yes, everything that can help to overcome the performance problems can help and I'm interested in this type of offload. But we need to make sure the API is usable by the whole community, so I don't want an API for some special case one of the NIC vendors is interested in.

BTW, we have a performance data, I planned to send it as part of cover letter for v3, but it is worth to share it now.

================================================================================
Performance results:

TCP multi-stream, using iperf3 instance per-CPU.

 +----------------------+--------+--------+--------+--------+---------+---------+
 |                      | 1 CPU  | 2 CPUs | 4 CPUs | 8 CPUs | 16 CPUs | 32 CPUs |
 |                      +--------+--------+--------+--------+---------+---------+
 |                      |                   BW (Gbps)                           |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Baseline             | 27.9   | 59     | 93.1  | 92.8    | 93.7    | 94.4    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Software IPsec       | 6      | 11.9   | 23.3  | 45.9    | 83.8    | 91.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec crypto offload | 15     | 29.7   | 58.5  | 89.6    | 90.4    | 90.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec full offload   | 28     | 57     | 90.7  | 91      | 91.3    | 91.9    |
 +----------------------+--------+--------+-------+---------+---------+---------+

Quoted 3 lines

IPsec full offload mode behaves as baseline and reaches linerate with same amount of CPUs.

Quoted 4 lines

Setups details (similar for both sides):
* NIC: ConnectX6-DX dual port, 100 Gbps each. Single port used in the tests.
* CPU: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz

My questions about performance were more about where does the performance loss originate. Is it because of loss of GRO?

Maybe sharing perf traces could answer some of those questions?

Quoted 2 lines

My questions about performance were more about where does the performance loss originate. Is it because of loss of GRO?

Performance loss between full and baseline ? it's hardly measurable .. less than 3% in the worst case.

Quoted 5 lines · On Mon, 22 Aug 2022 14:27:16 -0700 Saeed Mahameed wrote:

Quoted 2 lines

My questions about performance were more about where does the performance loss originate. Is it because of loss of GRO?

Performance loss between full and baseline ? it's hardly measurable .. less than 3% in the worst case.

The loss for crypto only vs baseline is what I meant. Obviously if we offload everything the CPU perf may look great but we're giving up flexibility and ability to fix problems in SW.

Quoted 10 lines · Jakub Kicinski

Quoted 5 lines · On Mon, 22 Aug 2022 14:27:16 -0700 Saeed Mahameed wrote:

Quoted 2 lines

My questions about performance were more about where does the performance loss originate. Is it because of loss of GRO?

Performance loss between full and baseline ? it's hardly measurable .. less than 3% in the worst case.

The loss for crypto only vs baseline is what I meant. Obviously if we offload everything the CPU perf may look great but we're giving up flexibility and ability to fix problems in SW.

The main difference between baseline TCP and crypto offload is the policy/state lookup and ESP encapsulation/decapsulation.

We don't loose GRO on crypto offload. The ESP header/trailer is stripped away in the GRO layer so that the inner TCP packet gets GRO. But we loose hardware LRO, as the ESP headers are not stripped in HW.

It would be really good to see where the bottlenecks are with crypto offload (RX or TX).

Also, it would be good to see why the full offload performs better. Some perf traces would be helpfull.

When I thought about possible offloads, I came to three different types:

1. encapsulation offload:
- Kernel does policy enforcement
- NIC does encapsulation
- NIC manages anti replay window and lifetime of SAs
- NIC sends lifetime and anti replay notifications to the kernel
- The Kernel talks to the keymanager

2. statefull offload with fallback:
- NIC handles the full datapath, but kernel can take over (fragments

     etc.)

- Kernel and NIC hold the full SA and policy database
- Kernel and NIC must sync the state (lifetime, replay window etc.) of SAs and policies
- The Kernel talks to the keymanager

3. statefull offload:
- NIC handles the full datapath
- NIC talks directly with the keymanager
- Kernel xfrm acts as a stub layer to pass messages from userspace to the NIC.

The offload that is implemented here comes option 2 closest. The statefull handling is shared between host and NIC. Unfortunalely, this is the most complicated one.

If just encapsulation/decapsulation brings the performance we could also go with option 1. That would be still a stateless offload.

Option 3 is what I would consider as a full offload. Kernel acts as a stateless stub layer, NIC does statefull IPsec.

Quoted 33 lines · Jakub Kicinski

Quoted 32 lines · On Mon, 22 Aug 2022 11:54:42 +0300 Leon Romanovsky wrote:

Quoted 9 lines · Steffen Klassert

Quoted 2 lines · Jakub Kicinski

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Yes, everything that can help to overcome the performance problems can help and I'm interested in this type of offload. But we need to make sure the API is usable by the whole community, so I don't want an API for some special case one of the NIC vendors is interested in.

BTW, we have a performance data, I planned to send it as part of cover letter for v3, but it is worth to share it now.

================================================================================
Performance results:

TCP multi-stream, using iperf3 instance per-CPU.

 +----------------------+--------+--------+--------+--------+---------+---------+
 |                      | 1 CPU  | 2 CPUs | 4 CPUs | 8 CPUs | 16 CPUs | 32 CPUs |
 |                      +--------+--------+--------+--------+---------+---------+
 |                      |                   BW (Gbps)                           |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Baseline             | 27.9   | 59     | 93.1  | 92.8    | 93.7    | 94.4    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Software IPsec       | 6      | 11.9   | 23.3  | 45.9    | 83.8    | 91.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec crypto offload | 15     | 29.7   | 58.5  | 89.6    | 90.4    | 90.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec full offload   | 28     | 57     | 90.7  | 91      | 91.3    | 91.9    |
 +----------------------+--------+--------+-------+---------+---------+---------+

Quoted 3 lines

Quoted 3 lines

IPsec full offload mode behaves as baseline and reaches linerate with same amount of CPUs.

Just making sure: Baseline == "Clear text TCP" ?

Quoted 38 lines · Saeed Mahameed

Quoted 36 lines · Jakub Kicinski

Quoted 35 lines · On Mon, 22 Aug 2022 11:54:42 +0300 Leon Romanovsky wrote:

Quoted 9 lines · Steffen Klassert

Quoted 2 lines · Jakub Kicinski

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Yes, everything that can help to overcome the performance problems can help and I'm interested in this type of offload. But we need to make sure the API is usable by the whole community, so I don't want an API for some special case one of the NIC vendors is interested in.

BTW, we have a performance data, I planned to send it as part of cover letter for v3, but it is worth to share it now.

================================================================================
Performance results:

TCP multi-stream, using iperf3 instance per-CPU.

 +----------------------+--------+--------+--------+--------+---------+---------+
 |                      | 1 CPU  | 2 CPUs | 4 CPUs | 8 CPUs | 16 CPUs | 32 CPUs |
 |                      +--------+--------+--------+--------+---------+---------+
 |                      |                   BW (Gbps)                           |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Baseline             | 27.9   | 59     | 93.1  | 92.8    | 93.7    | 94.4    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Software IPsec       | 6      | 11.9   | 23.3  | 45.9    | 83.8    | 91.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec crypto offload | 15     | 29.7   | 58.5  | 89.6    | 90.4    | 90.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec full offload   | 28     | 57     | 90.7  | 91      | 91.3    | 91.9    |
 +----------------------+--------+--------+-------+---------+---------+---------+

IPsec full offload mode behaves as baseline and reaches linerate with same amount of CPUs.

Quoted 1 line

Just making sure: Baseline == "Clear text TCP" ?

Yes, baseline is plain TCP without any encryption.

We can get higher numbers with Tariq's improvements, but it was not important to achieve maximum as we are interested to see differences between various modes.

Quoted 35 lines · On Mon, 22 Aug 2022 11:54:42 +0300 Leon Romanovsky wrote:

Quoted 9 lines · Steffen Klassert

Quoted 2 lines · Jakub Kicinski

Yup, that's what I thought you'd say. Can't argue with that use case if Steffen is satisfied with the technical aspects.

Yes, everything that can help to overcome the performance problems can help and I'm interested in this type of offload. But we need to make sure the API is usable by the whole community, so I don't want an API for some special case one of the NIC vendors is interested in.

BTW, we have a performance data, I planned to send it as part of cover letter for v3, but it is worth to share it now.

================================================================================
Performance results:

TCP multi-stream, using iperf3 instance per-CPU.

 +----------------------+--------+--------+--------+--------+---------+---------+
 |                      | 1 CPU  | 2 CPUs | 4 CPUs | 8 CPUs | 16 CPUs | 32 CPUs |
 |                      +--------+--------+--------+--------+---------+---------+
 |                      |                   BW (Gbps)                           |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Baseline             | 27.9   | 59     | 93.1  | 92.8    | 93.7    | 94.4    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | Software IPsec       | 6      | 11.9   | 23.3  | 45.9    | 83.8    | 91.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec crypto offload | 15     | 29.7   | 58.5  | 89.6    | 90.4    | 90.8    |
 +----------------------+--------+--------+-------+---------+---------+---------+
 | IPsec full offload   | 28     | 57     | 90.7  | 91      | 91.3    | 91.9    |
 +----------------------+--------+--------+-------+---------+---------+---------+

IPsec full offload mode behaves as baseline and reaches linerate with same amount of CPUs.

Quoted 4 lines

Setups details (similar for both sides):
* NIC: ConnectX6-DX dual port, 100 Gbps each. Single port used in the tests.
* CPU: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz

My questions about performance were more about where does the performance loss originate. Is it because of loss of GRO? Maybe sharing perf traces could answer some of those questions?

From: Leon Romanovsky <[email protected]>

Changelog:
v2:
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

The following series extends XFRM core code to handle new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for whole data path, so both policy and state should be offloaded.

Hi Leon,

Quoted 13 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog:
v2:
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

Quoted 5 lines

The following series extends XFRM core code to handle new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for whole data path, so both policy and state should be offloaded.

some general comments about the pachset:

As implemented, the software does not hold any state. I.e. there is no sync between hardware and software regarding stats, liftetime, lifebyte, packet counts and replay window. IKE rekeying and auditing is based on these, how should this be done?

I have not seen anything that catches configurations that stack multiple tunnels with the outer offloaded.

Where do we make sure that policy offloading device is the same as the state offloading device?

Quoted 22 lines · Steffen Klassert

Hi Leon,

Quoted 13 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog:
v2:
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

Quoted 5 lines

The following series extends XFRM core code to handle new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for whole data path, so both policy and state should be offloaded.

Quoted 7 lines

some general comments about the pachset:

As implemented, the software does not hold any state. I.e. there is no sync between hardware and software regarding stats, liftetime, lifebyte, packet counts and replay window. IKE rekeying and auditing is based on these, how should this be done?

This is only rough idea as we only started to implement needed support in libreswan, but our plan is to configure IKE with highest possible priority

Quoted 7 lines

I have not seen anything that catches configurations that stack multiple tunnels with the outer offloaded.

Where do we make sure that policy offloading device is the same as the state offloading device?

It is configuration error and we don't check it. Should we?

Quoted 31 lines · Leon Romanovsky

Quoted 29 lines · Steffen Klassert

Hi Leon,

Quoted 13 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog:
v2:
* Rebased to latest 6.0-rc1
* Add an extra check in TX datapath patch to validate packets before forwarding to HW.
* Added policy cleanup logic in case of netdev down event
v1: https://lore.kernel.org/all/[email protected]
* Moved comment to be before if (...) in third patch.
v0: https://lore.kernel.org/all/[email protected]
-----------------------------------------------------------------------

Quoted 5 lines

The following series extends XFRM core code to handle new type of IPsec offload - full offload.

In this mode, the HW is going to be responsible for whole data path, so both policy and state should be offloaded.

some general comments about the pachset:

As implemented, the software does not hold any state. I.e. there is no sync between hardware and software regarding stats, liftetime, lifebyte, packet counts and replay window. IKE rekeying and auditing is based on these, how should this be done?

Quoted 3 lines

This is only rough idea as we only started to implement needed support in libreswan, but our plan is to configure IKE with highest possible priority

If it is only a rough idea, then mark it as RFC. I want to see the whole picture before we merge it. And btw. tunnel mode belongs to the whoule picture too.

Quoted 6 lines

Quoted 6 lines

I have not seen anything that catches configurations that stack multiple tunnels with the outer offloaded.

Quoted 4 lines

Quoted 2 lines

Where do we make sure that policy offloading device is the same as the state offloading device?

It is configuration error and we don't check it. Should we?

We should at least make sure to not send out packets untransformed in this case.