Skip to contentSkip to conversation

netfilter: flowtable: tear down HW offloaded flows on FIB route changes

PATCH In review Ahmed Zaki 4 participants 5 messages Jul 8 – Jul 10

Subsystems (3)
lore
Share & tools
lore

Ahmed Zaki

author
#0 lore raw

PATCH 1/1 netfilter: flowtable: tear down HW offloaded flows on FIB route changes

Hardware-offloaded flows bypass the CPU and, unlike the software datapath, dst_check() does not invalidate them when a route changes. For ephemeral flows, this is usually not a problem as the flow expire on its own and the driver clears the entry in the HW. However, for persistent flows forwarded through the device, the HW is never informed that the route has expired.

For tables marked with NF_FLOWTABLE_HW_OFFLOAD, listen to the per-net FIB notifier chain and tear down the affected flows so they are re-evaluated by the SW forwarding path.

A lockless list is used to reduce the work items overhead in case of a route change storm allowing many FIB events to be processed by one work item.

Fixes c29f74e0df7a ("netfilter: nf_flow_table: hardware offload support")
Assisted-by Claude:claude-opus-4.8
---
 include/net/netfilter/nf_flow_table.h |   3 +
 net/netfilter/nf_flow_table_core.c    | 202 ++++++++++++++++++++++++++
 2 files changed, 205 insertions(+)
include/net/netfilter/nf_flow_table.h +3 -0
index 7b23b245a5a8..7c36255f5a4f 100644
@@ -84,6 +84,9 @@ struct nf_flowtable {
 	struct flow_block		flow_block;
 	struct rw_semaphore		flow_block_lock; /* Guards flow_block */
 	possible_net_t			net;
+	struct notifier_block		fib_nb;
+	struct work_struct		fib_work;
+	struct llist_head		fib_events;
 };
 
 static inline bool nf_flowtable_hw_offload(struct nf_flowtable *flowtable)
net/netfilter/nf_flow_table_core.c +202 -0

240 lines not on this page · raw message

Signature · 1 line
2.43.0
#1 lore raw
kernel test robot noticed the following build warnings:

Hi Ahmed,

kernel test robot noticed the following build warnings:

[auto build test WARNING on netfilter-nf/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Ahmed-Zaki/netfilter-flowtable-tear-down-HW-offloaded-flows-on-FIB-route-changes/20260709-050000
base:   https://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf.git main
patch link:    https://lore.kernel.org/r/20260708205404.911832-1-anzaki%40gmail.com
patch subject: [PATCH nf] netfilter: flowtable: tear down HW offloaded flows on FIB route changes
config: hexagon-allmodconfig (https://download.01.org/0day-ci/archive/20260709/[email protected]/config)
compiler: clang version 23.0.0git (https://github.com/llvm/llvm-project b3e6e6dabdc02153552a64fc74ff5c7532447eed)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260709/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> net/netfilter/nf_flow_table_core.c:833:3: warning: label followed by a declaration is a C23 extension [-Wc23-extensions]
     833 |                 struct fib_entry_notifier_info *fen;
         |                 ^
   net/netfilter/nf_flow_table_core.c:843:3: warning: label followed by a declaration is a C23 extension [-Wc23-extensions]
     843 |                 struct fib6_entry_notifier_info *fen6;
         |                 ^

2 warnings generated.

vim +833 net/netfilter/nf_flow_table_core.c

804
805 /* Called with rcu_read_lock() */
806 static int nf_flow_table_fib_event(struct notifier_block *nb,
807 unsigned long event, void *ptr)
808 {
809 struct nf_flowtable *flow_table =
810 container_of(nb, struct nf_flowtable, fib_nb);
811 struct fib_notifier_info *info = ptr;
812 struct nf_flow_fib_event *ev;
813
814 switch (event) {
815 case FIB_EVENT_ENTRY_REPLACE:
816 case FIB_EVENT_ENTRY_APPEND:
817 case FIB_EVENT_ENTRY_DEL:
818 break;
819 default:
820 return NOTIFY_DONE;
821 }
822
823 /* Skip events for an address family this table cannot hold. */
824 if (!nf_flowtable_fib_family_match(flow_table, info->family))
825 return NOTIFY_DONE;
826
827 ev = kzalloc(sizeof(*ev), GFP_ATOMIC);
828 if (!ev)
829 return NOTIFY_DONE;
830
831 switch (info->family) {
832 case NFPROTO_IPV4:

 > 833			struct fib_entry_notifier_info *fen;

834
835 fen = container_of(info, struct fib_entry_notifier_info, info);
836 ev->family = NFPROTO_IPV4;
837 ev->addr.ip4 = htonl(fen->dst);
838 ev->prefix_len = fen->dst_len;
839 break;
840

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#2 lore raw
In reply to Ahmed Zaki
Quoted 7 lines · Ahmed Zaki

Hardware-offloaded flows bypass the CPU and, unlike the software datapath, dst_check() does not invalidate them when a route changes. For ephemeral flows, this is usually not a problem as the flow expire on its own and the driver clears the entry in the HW. However, for persistent flows forwarded through the device, the HW is never informed that the route has expired.

Quoted 7 lines

For tables marked with NF_FLOWTABLE_HW_OFFLOAD, listen to the per-net FIB notifier chain and tear down the affected flows so they are re-evaluated by the SW forwarding path.

A lockless list is used to reduce the work items overhead in case of a route change storm allowing many FIB events to be processed by one work item.

This walks the hashtable anyway in case of fib event, maybe simply walk over the hashtable and call dst_check() to check if the cached dst is still current.

Quoted 1 line
Fixes c29f74e0df7a ("netfilter: nf_flow_table: hardware offload support")

No, this is an enhancement, not a fix. This must be targeted to nf-next.

Thanks.

author
#3 lore raw
Quoted 13 lines · Pablo Neira Ayuso
Quoted 11 lines · Ahmed Zaki

Hardware-offloaded flows bypass the CPU and, unlike the software datapath, dst_check() does not invalidate them when a route changes. For ephemeral flows, this is usually not a problem as the flow expire on its own and the driver clears the entry in the HW. However, for persistent flows forwarded through the device, the HW is never informed that the route has expired.

For tables marked with NF_FLOWTABLE_HW_OFFLOAD, listen to the per-net FIB notifier chain and tear down the affected flows so they are re-evaluated by the SW forwarding path.

Quoted 7 lines
Quoted 3 lines

A lockless list is used to reduce the work items overhead in case of a route change storm allowing many FIB events to be processed by one work item.

This walks the hashtable anyway in case of fib event, maybe simply walk over the hashtable and call dst_check() to check if the cached dst is still current.

Good idea. I will replace the tuple matches and the lockless list with a dst_check in the hashtable iter func: nf_flow_offload_fib_cb().

Quoted 4 lines
Quoted 1 line
Fixes c29f74e0df7a ("netfilter: nf_flow_table: hardware offload support")

No, this is an enhancement, not a fix. This must be targeted to nf-next.

Will tag v2 to nf-next.

Thanks.
Ahmed

syzbot Tested-by [email protected] on v1 1/1
#4 lore raw
In reply to Ahmed Zaki
syzbot ci has tested the following series

syzbot ci has tested the following series

[v1] netfilter: flowtable: tear down HW offloaded flows on FIB route changes
https://lore.kernel.org/all/[email protected]
* [PATCH nf] netfilter: flowtable: tear down HW offloaded flows on FIB route changes

and found the following issue:

general protection fault in net_generic

Full report is available here:
https://ci.syzbot.org/series/75cd5a68-8e8e-41ef-9da6-dea3554407a7

***

general protection fault in net_generic
tree:      nf
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/netfilter/nf.git
base:      60444706aa17616efc03190d099ac347e28b3d0a
arch:      amd64
compiler:  Debian clang version 22.1.6 (++20260514074242+fc4aad7b5db3-1~exp1~20260514074407.73), Debian LLD 22.1.6
config:    https://ci.syzbot.org/builds/b941607c-6459-4d25-8127-6c11f2a37ef0/config
syz repro: https://ci.syzbot.org/findings/93de0afd-01d4-4d54-9d3f-13094335f314/syz_repro
Oops: general protection fault, probably for non-canonical address 0xdffffc00000002c8: 0000 [#1] SMP KASAN PTI
KASAN: probably user-memory-access in range [0x0000000000001640-0x0000000000001647]
CPU: 1 UID: 0 PID: 5813 Comm: syz.2.19 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:net_generic+0xd4/0x240 include/net/netns/generic.h:46
Code: 8c be 4a 03 00 00 48 c7 c2 c0 83 e0 8c e8 14 1b 09 f8 49 bf 00 00 00 00 00 fc ff df 49 81 c6 40 16 00 00 4c 89 f0 48 c1 e8 03 <42> 80 3c 38 00 74 08 4c 89 f7 e8 0d a4 9a f8 4d 8b 26 e8 45 e8 1b
RSP: 0018:ffffc90003bde998 EFLAGS: 00010202
RAX: 00000000000002c8 RBX: ffffffff8999417e RCX: ffff8881699c5940
RDX: 0000000000000000 RSI: ffffffff8c2ab760 RDI: ffffffff8c2ab720
RBP: 0000000000000005 R08: ffffffff8999417e R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff8e959c20 R12: ffff8881b1679870
R13: dffffc0000000000 R14: 0000000000001640 R15: dffffc0000000000
FS:  00007ff6de31c6c0(0000) GS:ffff8882a921d000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6dd272780 CR3: 000000016bc8e000 CR4: 00000000000006f0
Call Trace:
 <TASK>
 fib_seq_sum+0x29/0x280 net/core/fib_notifier.c:43
 register_fib_notifier+0x49/0x360 net/core/fib_notifier.c:106
 nf_flow_table_init+0x34a/0x460 net/netfilter/nf_flow_table_core.c:888
 tcf_ct_flow_table_get+0x13b6/0x1f40 net/sched/act_ct.c:352
 tcf_ct_init+0x6bd/0xb10 net/sched/act_ct.c:1423
 tcf_action_init_1+0x4ba/0x740 net/sched/act_api.c:1433
 tcf_action_init+0x30e/0xb10 net/sched/act_api.c:1508
 tcf_action_add net/sched/act_api.c:2106 [inline]
 tc_ctl_action+0x43b/0xc70 net/sched/act_api.c:2163
 rtnetlink_rcv_msg+0x7b9/0xc00 net/core/rtnetlink.c:7085
 netlink_rcv_skb+0x226/0x4a0 net/netlink/af_netlink.c:2556
 netlink_unicast_kernel net/netlink/af_netlink.c:1319 [inline]
 netlink_unicast+0x7bb/0x940 net/netlink/af_netlink.c:1345
 netlink_sendmsg+0x813/0xb40 net/netlink/af_netlink.c:1900
 sock_sendmsg_nosec+0x13a/0x180 net/socket.c:775
 __sock_sendmsg net/socket.c:790 [inline]
 ____sys_sendmsg+0x54e/0x850 net/socket.c:2684
 ___sys_sendmsg+0x2a5/0x360 net/socket.c:2738
 __sys_sendmsg net/socket.c:2770 [inline]
 __do_sys_sendmsg net/socket.c:2775 [inline]
 __se_sys_sendmsg net/socket.c:2773 [inline]
 __x64_sys_sendmsg+0x1b1/0x290 net/socket.c:2773
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x174/0x580 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7ff6dd39ce59
Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ff6de31c028 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007ff6dd615fa0 RCX: 00007ff6dd39ce59
RDX: 0000000000000040 RSI: 0000200000000080 RDI: 0000000000000003
RBP: 00007ff6dd432e6f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ff6dd616038 R14: 00007ff6dd615fa0 R15: 00007ffd83f8d058
 </TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:net_generic+0xd4/0x240 include/net/netns/generic.h:46
Code: 8c be 4a 03 00 00 48 c7 c2 c0 83 e0 8c e8 14 1b 09 f8 49 bf 00 00 00 00 00 fc ff df 49 81 c6 40 16 00 00 4c 89 f0 48 c1 e8 03 <42> 80 3c 38 00 74 08 4c 89 f7 e8 0d a4 9a f8 4d 8b 26 e8 45 e8 1b
RSP: 0018:ffffc90003bde998 EFLAGS: 00010202
RAX: 00000000000002c8 RBX: ffffffff8999417e RCX: ffff8881699c5940
RDX: 0000000000000000 RSI: ffffffff8c2ab760 RDI: ffffffff8c2ab720
RBP: 0000000000000005 R08: ffffffff8999417e R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff8e959c20 R12: ffff8881b1679870
R13: dffffc0000000000 R14: 0000000000001640 R15: dffffc0000000000
FS:  00007ff6de31c6c0(0000) GS:ffff8882a921d000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6dd3ea540 CR3: 000000016bc8e000 CR4: 00000000000006f0
----------------
Code disassembly (best guess):
   0:	8c be 4a 03 00 00    	mov    %?,0x34a(%rsi)
   6:	48 c7 c2 c0 83 e0 8c 	mov    $0xffffffff8ce083c0,%rdx
   d:	e8 14 1b 09 f8       	call   0xf8091b26
  12:	49 bf 00 00 00 00 00 	movabs $0xdffffc0000000000,%r15
  19:	fc ff df
  1c:	49 81 c6 40 16 00 00 	add    $0x1640,%r14
  23:	4c 89 f0             	mov    %r14,%rax
  26:	48 c1 e8 03          	shr    $0x3,%rax
* 2a:	42 80 3c 38 00       	cmpb   $0x0,(%rax,%r15,1) <-- trapping instruction
  2f:	74 08                	je     0x39
  31:	4c 89 f7             	mov    %r14,%rdi
  34:	e8 0d a4 9a f8       	call   0xf89aa446
  39:	4d 8b 26             	mov    (%r14),%r12
  3c:	e8                   	.byte 0xe8
  3d:	45                   	rex.RB
  3e:	e8                   	.byte 0xe8
  3f:	1b                   	.byte 0x1b

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

To test a patch for this bug, please reply with #syz test (should be on a separate line).

The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.