netfilter: flowtable: tear down HW offloaded flows on FIB route changes
Subsystems (3)
PATCH 1/1 netfilter: flowtable: tear down HW offloaded flows on FIB route changes
Hardware-offloaded flows bypass the CPU and, unlike the software datapath, dst_check() does not invalidate them when a route changes. For ephemeral flows, this is usually not a problem as the flow expire on its own and the driver clears the entry in the HW. However, for persistent flows forwarded through the device, the HW is never informed that the route has expired.
For tables marked with NF_FLOWTABLE_HW_OFFLOAD, listen to the per-net FIB notifier chain and tear down the affected flows so they are re-evaluated by the SW forwarding path.
A lockless list is used to reduce the work items overhead in case of a route change storm allowing many FIB events to be processed by one work item.
--- include/net/netfilter/nf_flow_table.h | 3 + net/netfilter/nf_flow_table_core.c | 202 ++++++++++++++++++++++++++ 2 files changed, 205 insertions(+)
include/net/netfilter/nf_flow_table.h +3 -0
@@ -84,6 +84,9 @@ struct nf_flowtable {struct flow_block flow_block;struct rw_semaphore flow_block_lock;/* Guards flow_block */ possible_net_t net; +struct notifier_block fib_nb; +struct work_struct fib_work; +struct llist_head fib_events; };static inline bool nf_flowtable_hw_offload (struct nf_flowtable *flowtable)
net/netfilter/nf_flow_table_core.c +202 -0
240 lines not on this page · raw message
Signature · 1 line
2.43.0
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi Ahmed,
kernel test robot noticed the following build warnings:
[auto build test WARNING on netfilter-nf/main]
url: https://github.com/intel-lab-lkp/linux/commits/Ahmed-Zaki/netfilter-flowtable-tear-down-HW-offloaded-flows-on-FIB-route-changes/20260709-050000 base: https://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf.git main patch link: https://lore.kernel.org/r/20260708205404.911832-1-anzaki%40gmail.com patch subject: [PATCH nf] netfilter: flowtable: tear down HW offloaded flows on FIB route changes config: hexagon-allmodconfig (https://download.01.org/0day-ci/archive/20260709/[email protected]/config) compiler: clang version 23.0.0git (https://github.com/llvm/llvm-project b3e6e6dabdc02153552a64fc74ff5c7532447eed) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260709/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> net/netfilter/nf_flow_table_core.c:833:3: warning: label followed by a declaration is a C23 extension [-Wc23-extensions]
833 | struct fib_entry_notifier_info *fen;
| ^
net/netfilter/nf_flow_table_core.c:843:3: warning: label followed by a declaration is a C23 extension [-Wc23-extensions]
843 | struct fib6_entry_notifier_info *fen6;
| ^2 warnings generated.
vim +833 net/netfilter/nf_flow_table_core.c
804
805 /* Called with rcu_read_lock() */
806 static int nf_flow_table_fib_event(struct notifier_block *nb,
807 unsigned long event, void *ptr)
808 {
809 struct nf_flowtable *flow_table =
810 container_of(nb, struct nf_flowtable, fib_nb);
811 struct fib_notifier_info *info = ptr;
812 struct nf_flow_fib_event *ev;
813
814 switch (event) {
815 case FIB_EVENT_ENTRY_REPLACE:
816 case FIB_EVENT_ENTRY_APPEND:
817 case FIB_EVENT_ENTRY_DEL:
818 break;
819 default:
820 return NOTIFY_DONE;
821 }
822
823 /* Skip events for an address family this table cannot hold. */
824 if (!nf_flowtable_fib_family_match(flow_table, info->family))
825 return NOTIFY_DONE;
826
827 ev = kzalloc(sizeof(*ev), GFP_ATOMIC);
828 if (!ev)
829 return NOTIFY_DONE;
830
831 switch (info->family) {
832 case NFPROTO_IPV4:
> 833 struct fib_entry_notifier_info *fen;
834
835 fen = container_of(info, struct fib_entry_notifier_info, info);
836 ev->family = NFPROTO_IPV4;
837 ev->addr.ip4 = htonl(fen->dst);
838 ev->prefix_len = fen->dst_len;
839 break;
840
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Pablo Neira Ayuso
maintainerQuoted 7 lines · Ahmed Zaki
Hardware-offloaded flows bypass the CPU and, unlike the software datapath,
dst_check()does not invalidate them when a route changes. For ephemeral flows, this is usually not a problem as the flow expire on its own and the driver clears the entry in the HW. However, for persistent flows forwarded through the device, the HW is never informed that the route has expired.
Quoted 7 lines
For tables marked with
NF_FLOWTABLE_HW_OFFLOAD, listen to the per-net FIB notifier chain and tear down the affected flows so they are re-evaluated by the SW forwarding path.A lockless list is used to reduce the work items overhead in case of a route change storm allowing many FIB events to be processed by one work item.
This walks the hashtable anyway in case of fib event, maybe simply walk over the hashtable and call dst_check() to check if the cached dst is still current.
Quoted 1 line
Fixes c29f74e0df7a ("netfilter: nf_flow_table: hardware offload support")
No, this is an enhancement, not a fix. This must be targeted to nf-next.
Thanks.
Quoted 13 lines · Pablo Neira Ayuso
Quoted 11 lines · Ahmed Zaki
Hardware-offloaded flows bypass the CPU and, unlike the software datapath,
dst_check()does not invalidate them when a route changes. For ephemeral flows, this is usually not a problem as the flow expire on its own and the driver clears the entry in the HW. However, for persistent flows forwarded through the device, the HW is never informed that the route has expired.For tables marked with
NF_FLOWTABLE_HW_OFFLOAD, listen to the per-net FIB notifier chain and tear down the affected flows so they are re-evaluated by the SW forwarding path.
Quoted 7 lines
Quoted 3 lines
A lockless list is used to reduce the work items overhead in case of a route change storm allowing many FIB events to be processed by one work item.
This walks the hashtable anyway in case of fib event, maybe simply walk over the hashtable and call
dst_check()to check if the cached dst is still current.
Good idea. I will replace the tuple matches and the lockless list with a dst_check in the hashtable iter func: nf_flow_offload_fib_cb().
Quoted 4 lines
Quoted 1 line
Fixes c29f74e0df7a ("netfilter: nf_flow_table: hardware offload support")No, this is an enhancement, not a fix. This must be targeted to nf-next.
Will tag v2 to nf-next.
Thanks.
Ahmed
syzbot ci has tested the following series
syzbot ci has tested the following series
[v1] netfilter: flowtable: tear down HW offloaded flows on FIB route changes
https://lore.kernel.org/all/[email protected]
* [PATCH nf] netfilter: flowtable: tear down HW offloaded flows on FIB route changes
and found the following issue:
general protection fault in net_generic
Full report is available here:
https://ci.syzbot.org/series/75cd5a68-8e8e-41ef-9da6-dea3554407a7
***
general protection fault in net_generic
tree: nf URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/netfilter/nf.git base: 60444706aa17616efc03190d099ac347e28b3d0a arch: amd64 compiler: Debian clang version 22.1.6 (++20260514074242+fc4aad7b5db3-1~exp1~20260514074407.73), Debian LLD 22.1.6 config: https://ci.syzbot.org/builds/b941607c-6459-4d25-8127-6c11f2a37ef0/config syz repro: https://ci.syzbot.org/findings/93de0afd-01d4-4d54-9d3f-13094335f314/syz_repro
Oops: general protection fault, probably for non-canonical address 0xdffffc00000002c8: 0000 [#1] SMP KASAN PTI KASAN: probably user-memory-access in range [0x0000000000001640-0x0000000000001647] CPU: 1 UID: 0 PID: 5813 Comm: syz.2.19 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:net_generic+0xd4/0x240 include/net/netns/generic.h:46 Code: 8c be 4a 03 00 00 48 c7 c2 c0 83 e0 8c e8 14 1b 09 f8 49 bf 00 00 00 00 00 fc ff df 49 81 c6 40 16 00 00 4c 89 f0 48 c1 e8 03 <42> 80 3c 38 00 74 08 4c 89 f7 e8 0d a4 9a f8 4d 8b 26 e8 45 e8 1b RSP: 0018:ffffc90003bde998 EFLAGS: 00010202 RAX: 00000000000002c8 RBX: ffffffff8999417e RCX: ffff8881699c5940 RDX: 0000000000000000 RSI: ffffffff8c2ab760 RDI: ffffffff8c2ab720 RBP: 0000000000000005 R08: ffffffff8999417e R09: 0000000000000000 R10: 0000000000000000 R11: ffffffff8e959c20 R12: ffff8881b1679870 R13: dffffc0000000000 R14: 0000000000001640 R15: dffffc0000000000 FS: 00007ff6de31c6c0(0000) GS:ffff8882a921d000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff6dd272780 CR3: 000000016bc8e000 CR4: 00000000000006f0 Call Trace: <TASK> fib_seq_sum+0x29/0x280 net/core/fib_notifier.c:43 register_fib_notifier+0x49/0x360 net/core/fib_notifier.c:106 nf_flow_table_init+0x34a/0x460 net/netfilter/nf_flow_table_core.c:888 tcf_ct_flow_table_get+0x13b6/0x1f40 net/sched/act_ct.c:352 tcf_ct_init+0x6bd/0xb10 net/sched/act_ct.c:1423 tcf_action_init_1+0x4ba/0x740 net/sched/act_api.c:1433 tcf_action_init+0x30e/0xb10 net/sched/act_api.c:1508 tcf_action_add net/sched/act_api.c:2106 [inline] tc_ctl_action+0x43b/0xc70 net/sched/act_api.c:2163 rtnetlink_rcv_msg+0x7b9/0xc00 net/core/rtnetlink.c:7085 netlink_rcv_skb+0x226/0x4a0 net/netlink/af_netlink.c:2556 netlink_unicast_kernel net/netlink/af_netlink.c:1319 [inline] netlink_unicast+0x7bb/0x940 net/netlink/af_netlink.c:1345 netlink_sendmsg+0x813/0xb40 net/netlink/af_netlink.c:1900 sock_sendmsg_nosec+0x13a/0x180 net/socket.c:775 __sock_sendmsg net/socket.c:790 [inline] ____sys_sendmsg+0x54e/0x850 net/socket.c:2684 ___sys_sendmsg+0x2a5/0x360 net/socket.c:2738 __sys_sendmsg net/socket.c:2770 [inline] __do_sys_sendmsg net/socket.c:2775 [inline] __se_sys_sendmsg net/socket.c:2773 [inline] __x64_sys_sendmsg+0x1b1/0x290 net/socket.c:2773 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0x174/0x580 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7ff6dd39ce59 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ff6de31c028 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 00007ff6dd615fa0 RCX: 00007ff6dd39ce59 RDX: 0000000000000040 RSI: 0000200000000080 RDI: 0000000000000003 RBP: 00007ff6dd432e6f R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007ff6dd616038 R14: 00007ff6dd615fa0 R15: 00007ffd83f8d058 </TASK> Modules linked in: ---[ end trace 0000000000000000 ]--- RIP: 0010:net_generic+0xd4/0x240 include/net/netns/generic.h:46 Code: 8c be 4a 03 00 00 48 c7 c2 c0 83 e0 8c e8 14 1b 09 f8 49 bf 00 00 00 00 00 fc ff df 49 81 c6 40 16 00 00 4c 89 f0 48 c1 e8 03 <42> 80 3c 38 00 74 08 4c 89 f7 e8 0d a4 9a f8 4d 8b 26 e8 45 e8 1b RSP: 0018:ffffc90003bde998 EFLAGS: 00010202 RAX: 00000000000002c8 RBX: ffffffff8999417e RCX: ffff8881699c5940 RDX: 0000000000000000 RSI: ffffffff8c2ab760 RDI: ffffffff8c2ab720 RBP: 0000000000000005 R08: ffffffff8999417e R09: 0000000000000000 R10: 0000000000000000 R11: ffffffff8e959c20 R12: ffff8881b1679870 R13: dffffc0000000000 R14: 0000000000001640 R15: dffffc0000000000 FS: 00007ff6de31c6c0(0000) GS:ffff8882a921d000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff6dd3ea540 CR3: 000000016bc8e000 CR4: 00000000000006f0 ---------------- Code disassembly (best guess): 0: 8c be 4a 03 00 00 mov %?,0x34a(%rsi) 6: 48 c7 c2 c0 83 e0 8c mov $0xffffffff8ce083c0,%rdx d: e8 14 1b 09 f8 call 0xf8091b26 12: 49 bf 00 00 00 00 00 movabs $0xdffffc0000000000,%r15 19: fc ff df 1c: 49 81 c6 40 16 00 00 add $0x1640,%r14 23: 4c 89 f0 mov %r14,%rax 26: 48 c1 e8 03 shr $0x3,%rax * 2a: 42 80 3c 38 00 cmpb $0x0,(%rax,%r15,1) <-- trapping instruction 2f: 74 08 je 0x39 31: 4c 89 f7 mov %r14,%rdi 34: e8 0d a4 9a f8 call 0xf89aa446 39: 4d 8b 26 mov (%r14),%r12 3c: e8 .byte 0xe8 3d: 45 rex.RB 3e: e8 .byte 0xe8 3f: 1b .byte 0x1b
***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
To test a patch for this bug, please reply with #syz test (should be on a separate line).
The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.