Skip to contentSkip to conversation

netfilter: flowtable: tear down flow entries with stale dst from GC

PATCH In review Pablo Neira Ayuso 2 participants 6 messages Jul 10 – Jul 13

Subsystems (2)
lore
Share & tools
lore

Pablo Neira Ayuso

author maintainer
#0 lore raw

PATCH v2 1/1 netfilter: flowtable: tear down flow entries with stale dst from GC

In case of route updates, tear down flow entries with stale dst to give them a chance to obtain a fresh route.

This is specifically useful for hardware offloaded entries, where the flowtable software dataplane sees no packet, where the existing check for stale dst entries does not help.

---
v2: - reuse nf_flow_dst_check(), move it to .h file
- use correct logic in nf_flow_dst_check() from GC step

This patch has been repurposed to the nf.git tree, because net-next.git is still missing a recent fix and I would like sashiko kicks it for review. So I am still leaning towards including this in nf-next.

 include/net/netfilter/nf_flow_table.h | 8 ++++++++
 net/netfilter/nf_flow_table_core.c    | 2 ++
 net/netfilter/nf_flow_table_ip.c      | 8 --------
 3 files changed, 10 insertions(+), 8 deletions(-)
include/net/netfilter/nf_flow_table.h +8 -0
index ce414118962f..a090ec3ffef2 100644
@@ -310,6 +310,14 @@ int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow)
 void flow_offload_refresh(struct nf_flowtable *flow_table,
 			  struct flow_offload *flow, bool force);
 
+static inline bool nf_flow_dst_check(struct flow_offload_tuple *tuple)
+{
+	if (!tuple->dst_cache)
+		return true;
+
+	return dst_check(tuple->dst_cache, tuple->dst_cookie);
+}
+
 struct flow_offload_tuple_rhash *flow_offload_lookup(struct nf_flowtable *flow_table,
 						     struct flow_offload_tuple *tuple);
 void nf_flow_table_gc_run(struct nf_flowtable *flow_table);
net/netfilter/nf_flow_table_core.c +2 -0
index 2a829b5e8240..fe655e5e3b06 100644
@@ -571,6 +571,8 @@ static void nf_flow_offload_gc_step(struct nf_flowtable *flow_table,
 
 	if (nf_flow_has_expired(flow) ||
 	    nf_ct_is_dying(flow->ct) ||
+	    !nf_flow_dst_check(&flow->tuplehash[FLOW_OFFLOAD_DIR_ORIGINAL].tuple) ||
+	    !nf_flow_dst_check(&flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].tuple) ||
 	    nf_flow_custom_gc(flow_table, flow)) {
 		flow_offload_teardown(flow);
 		teardown = true;
net/netfilter/nf_flow_table_ip.c +0 -8
index 44f4d74f2982..55f6a0dedb03 100644
@@ -297,14 +297,6 @@ static bool nf_flow_exceeds_mtu(const struct sk_buff *skb, unsigned int mtu)
 	return true;
 }
 
-static inline bool nf_flow_dst_check(struct flow_offload_tuple *tuple)
-{
-	if (!tuple->dst_cache)
-		return true;
-
-	return dst_check(tuple->dst_cache, tuple->dst_cookie);
-}
-
 static unsigned int nf_flow_xmit_xfrm(struct sk_buff *skb,
 				      const struct nf_hook_state *state,
 				      struct dst_entry *dst)
Signature · 1 line
2.47.3
#1 lore raw
Quoted 27 lines · Pablo Neira Ayuso

In case of route updates, tear down flow entries with stale dst to give them a chance to obtain a fresh route.

This is specifically useful for hardware offloaded entries, where the flowtable software dataplane sees no packet, where the existing check for stale dst entries does not help.

---
v2: - reuse nf_flow_dst_check(), move it to .h file
- use correct logic in nf_flow_dst_check() from GC step

This patch has been repurposed to the nf.git tree, because net-next.git is still missing a recent fix and I would like sashiko kicks it for review. So I am still leaning towards including this in nf-next.

 include/net/netfilter/nf_flow_table.h | 8 ++++++++
 net/netfilter/nf_flow_table_core.c    | 2 ++
 net/netfilter/nf_flow_table_ip.c      | 8 --------
 3 files changed, 10 insertions(+), 8 deletions(-)
include/net/netfilter/nf_flow_table.h +0 -0
index ce414118962f..a090ec3ffef2 100644

include/net/netfilter/nf_flow_table.h int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow) · line 319

Quoted 10 lines
include/net/netfilter/nf_flow_table.h +7 -0
@@ -310,6 +310,14 @@ int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow)
 void flow_offload_refresh(struct nf_flowtable *flow_table,
                          struct flow_offload *flow, bool force);
 
+static inline bool nf_flow_dst_check(struct flow_offload_tuple *tuple)
+{
+       if (!tuple->dst_cache)
+               return true;
+
+       return dst_check(tuple->dst_cache, tuple->dst_cookie);
+}

I am testing this patch and keep getting some splats. I am testing with a MTK7621 hw which
to my understanding, marks the tuple's xmit_type DIRECT (not neigh or XFRM).

In nft_dev_forward_path(), out.h_source is set and this overrides the dst_cache (same union)
this seems to be causing the splat when the dst_cache is dereferenced. (btw, not like his patch,
in the latest HEAD, nf_flow_dst_check() guards tuple->dst_**** by checks on xmit_type)

So, to support DIRECT types, we can:
1 - go back to my v1 patch (no dependency on dst_cache)
2 - same patch but use dst_check() only for NEIGH/XFRM
3 - or maybe we can have the dst_cache out of the union and available for all xmit_types.

I have some time to work on this, let me know if I can help.

Thanks.

author maintainer
#2 lore raw

Hi,

Quoted 43 lines · Ahmed Zaki
Quoted 27 lines · Pablo Neira Ayuso

In case of route updates, tear down flow entries with stale dst to give them a chance to obtain a fresh route.

This is specifically useful for hardware offloaded entries, where the flowtable software dataplane sees no packet, where the existing check for stale dst entries does not help.

---
v2: - reuse nf_flow_dst_check(), move it to .h file
- use correct logic in nf_flow_dst_check() from GC step

This patch has been repurposed to the nf.git tree, because net-next.git is still missing a recent fix and I would like sashiko kicks it for review. So I am still leaning towards including this in nf-next.

 include/net/netfilter/nf_flow_table.h | 8 ++++++++
 net/netfilter/nf_flow_table_core.c    | 2 ++
 net/netfilter/nf_flow_table_ip.c      | 8 --------
 3 files changed, 10 insertions(+), 8 deletions(-)
include/net/netfilter/nf_flow_table.h +0 -0
index ce414118962f..a090ec3ffef2 100644

include/net/netfilter/nf_flow_table.h int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow) · line 319

Quoted 10 lines
include/net/netfilter/nf_flow_table.h +7 -0
@@ -310,6 +310,14 @@ int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow)
 void flow_offload_refresh(struct nf_flowtable *flow_table,
                          struct flow_offload *flow, bool force);
 
+static inline bool nf_flow_dst_check(struct flow_offload_tuple *tuple)
+{
+       if (!tuple->dst_cache)
+               return true;
+
+       return dst_check(tuple->dst_cache, tuple->dst_cookie);
+}

I am testing this patch and keep getting some splats. I am testing with a MTK7621 hw which
to my understanding, marks the tuple's xmit_type DIRECT (not neigh or XFRM).

Quoted 6 lines

In nft_dev_forward_path(), out.h_source is set and this overrides the dst_cache (same union)
this seems to be causing the splat when the dst_cache is dereferenced. (btw, not like his patch,
in the latest HEAD, nf_flow_dst_check() guards tuple->dst_**** by checks on xmit_type)

We have move dst_cache out of the union quite recently so... (see below)

Quoted 7 lines

So, to support DIRECT types, we can:
1 - go back to my v1 patch (no dependency on dst_cache)
2 - same patch but use dst_check() only for NEIGH/XFRM
3 - or maybe we can have the dst_cache out of the union and available for all xmit_types.

I have some time to work on this, let me know if I can help.

... probably you are missing this series? They got merged quite recently, I am not sure what tree you are using as reference:

fa7395c02d95 netfilter: flowtable: support IPIP tunnel with direct xmit 6c5dcab95f4c netfilter: flowtable: IPIP tunnel hardware offload is not yet support c328b90c17fc netfilter: flowtable: use dst in this direction when pushing IPIP header

There is also this patch which is needed:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/

which has been included in the last PR this Friday.

#3 lore raw
Quoted 1 line · Pablo Neira Ayuso

.
.
.

Quoted 19 lines
Quoted 10 lines

I am testing this patch and keep getting some splats. I am testing with a MTK7621 hw which
to my understanding, marks the tuple's xmit_type DIRECT (not neigh or XFRM).

In nft_dev_forward_path(), out.h_source is set and this overrides the dst_cache (same union)
this seems to be causing the splat when the dst_cache is dereferenced. (btw, not like his patch,
in the latest HEAD, nf_flow_dst_check() guards tuple->dst_**** by checks on xmit_type)

We have move dst_cache out of the union quite recently so... (see below)

Quoted 6 lines

So, to support DIRECT types, we can:
1 - go back to my v1 patch (no dependency on dst_cache)
2 - same patch but use dst_check() only for NEIGH/XFRM
3 - or maybe we can have the dst_cache out of the union and available for all xmit_types.

Quoted 4 lines
Quoted 1 line

I have some time to work on this, let me know if I can help.

... probably you are missing this series? They got merged quite recently, I am not sure what tree you are using as reference:

yeah, I was looking at nf-next. These seem to be in nf only as of now.

Quoted 10 lines

fa7395c02d95 netfilter: flowtable: support IPIP tunnel with direct xmit 6c5dcab95f4c netfilter: flowtable: IPIP tunnel hardware offload is not yet support c328b90c17fc netfilter: flowtable: use dst in this direction when pushing IPIP header

There is also this patch which is needed:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/

which has been included in the last PR this Friday.

I have some limitations so I can only test with 6.6 or 6.12. I will try to trim down these patches to
"move dst_cache out of union and use the gc to check all entries" and will let you know if I see
any more problems.

The one thing I believe is still missing is to remove the if condition in flow_offload_fill_route()
when setting "flow_tuple->dst_cache/dst_cookie":

- if (route->tuple[!dir].in.num_tuns) {

We now need "flow_tuple->dst_cache/dst_cookie" for all DIRECT xmit even with no tunnels
(quick look on nft_dev_path_info() shows "num_tuns" set for "path->type == DEV_PATH_TUN" only).

Thank you.

author maintainer
#4 lore raw
Quoted 23 lines · Ahmed Zaki
Quoted 1 line · Pablo Neira Ayuso

.

Quoted 19 lines
Quoted 10 lines

I am testing this patch and keep getting some splats. I am testing with a MTK7621 hw which
to my understanding, marks the tuple's xmit_type DIRECT (not neigh or XFRM).

In nft_dev_forward_path(), out.h_source is set and this overrides the dst_cache (same union)
this seems to be causing the splat when the dst_cache is dereferenced. (btw, not like his patch,
in the latest HEAD, nf_flow_dst_check() guards tuple->dst_**** by checks on xmit_type)

We have move dst_cache out of the union quite recently so... (see below)

Quoted 6 lines

So, to support DIRECT types, we can:
1 - go back to my v1 patch (no dependency on dst_cache)
2 - same patch but use dst_check() only for NEIGH/XFRM
3 - or maybe we can have the dst_cache out of the union and available for all xmit_types.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I have some time to work on this, let me know if I can help.

... probably you are missing this series? They got merged quite recently, I am not sure what tree you are using as reference:

yeah, I was looking at nf-next. These seem to be in nf only as of now.

OK.

Quoted 8 lines
Quoted 8 lines

fa7395c02d95 netfilter: flowtable: support IPIP tunnel with direct xmit 6c5dcab95f4c netfilter: flowtable: IPIP tunnel hardware offload is not yet support c328b90c17fc netfilter: flowtable: use dst in this direction when pushing IPIP header

There is also this patch which is needed:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/

Quoted 7 lines
Quoted 1 line

which has been included in the last PR this Friday.

I have some limitations so I can only test with 6.6 or 6.12. I will try to trim down these patches to
"move dst_cache out of union and use the gc to check all entries" and will let you know if I see
any more problems.

You will have to backport the move of dst_cache out of the union.

Quoted 8 lines

The one thing I believe is still missing is to remove the if condition in flow_offload_fill_route()
when setting "flow_tuple->dst_cache/dst_cookie":

- if (route->tuple[!dir].in.num_tuns) {

We now need "flow_tuple->dst_cache/dst_cookie" for all DIRECT xmit even with no tunnels

How so? ->dst_cache is not available when sending packets to a port behind a master bridge device.

#5 lore raw
Quoted 4 lines · Pablo Neira Ayuso
Quoted 2 lines · Ahmed Zaki
Quoted 1 line · Pablo Neira Ayuso

.
.
.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

fa7395c02d95 netfilter: flowtable: support IPIP tunnel with direct xmit 6c5dcab95f4c netfilter: flowtable: IPIP tunnel hardware offload is not yet support c328b90c17fc netfilter: flowtable: use dst in this direction when pushing IPIP header

There is also this patch which is needed:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/

which has been included in the last PR this Friday.

Quoted 8 lines
Quoted 5 lines

I have some limitations so I can only test with 6.6 or 6.12. I will try to trim down these patches to
"move dst_cache out of union and use the gc to check all entries" and will let you know if I see
any more problems.

You will have to backport the move of dst_cache out of the union.

yes, and I tested for a couple of hours with no issues. GC was taking care of the
expired dst_cache.

Quoted 11 lines
Quoted 8 lines

The one thing I believe is still missing is to remove the if condition in flow_offload_fill_route()
when setting "flow_tuple->dst_cache/dst_cookie":

- if (route->tuple[!dir].in.num_tuns) {

We now need "flow_tuple->dst_cache/dst_cookie" for all DIRECT xmit even with no tunnels

How so? ->dst_cache is not available when sending packets to a port behind a master bridge device.

Sorry I must be missing something. I do not have a bridge in my setup. I have a Mediatek PPE that is offloading from one port to another and there are no bridges:

# nft list ruleset
table ip fw4 {
    flowtable f1 {
        hook ingress priority filter
        devices = { eth0, eth1 }
        flags offload
    }
chain forward {
    type filter hook forward priority filter; policy accept;
    tcp dport 5201 flow add @f1
    }
}

My previous comment is that in this case, AFAIK, the "xmit_type = DIRECT" and in order
to set dst_cache/cookie, the guard "if (route->tuple[!dir].in.num_tuns)" needs to be
taken away (which I did in my 6.6 patch, let me know if you want me to send this as ref).

Thank you.