Netfilter/IPVS updates for net-next
Subsystems (8)
Pablo Neira Ayuso
maintainercover Netfilter/IPVS updates for net-next
Hi,
The following patchset contains Netfilter/IPVS updates for net-next:
1) Incorrect error check in nft_expr_inner_parse(), from Dan Carpenter.
2) Add DATA_SENT state to SCTP connection tracking helper, from Sriram Yagnaraman.
3) Consolidate nf_confirm for ipv4 and ipv6, from Florian Westphal.
4) Add bitmask support for ipset, from Vishwanath Pai.
5) Handle icmpv6 redirects as RELATED, from Florian Westphal.
6) Add WARN_ON_ONCE() to impossible case in flowtable datapath, from Li Qiong.
7) A large batch of IPVS updates to replace timer-based estimators by kthreads to scale up wrt. CPUs and workload (millions of estimators).
Julian Anastasov says:
This patchset implements stats estimation in kthread context.
It replaces the code that runs on single CPU in timer context every 2 seconds and causing latency splats as shown in reports [1], [2], [3]. The solution targets setups with thousands of IPVS services, destinations and multi-CPU boxes.
Spread the estimation on multiple (configured) CPUs and multiple
time slots (timer ticks) by using multiple chains organized under RCU rules. When stats are not needed, it is recommended to use run_estimation=0 as already implemented before this change.
RCU Locking:
- As stats are now RCU-locked, tot_stats, svc and dest which hold estimator structures are now always freed from RCU callback. This ensures RCU grace period after the ip_vs_stop_estimator() call.
Kthread data:
- every kthread works over its own data structure and all such structures are attached to array. For now we limit kthreads depending on the number of CPUs.
- even while there can be a kthread structure, its task may not be running, eg. before first service is added or while the sysctl var is set to an empty cpulist or when run_estimation is set to 0 to disable the estimation.
- the allocated kthread context may grow from 1 to 50 allocated structures for timer ticks which saves memory for setups with small number of estimators
- a task and its structure may be released if all estimators are unlinked from its chains, leaving the slot in the array empty
- every kthread data structure allows limited number of estimators. Kthread 0 is also used to initially calculate the max number of estimators to allow in every chain considering a sub-100 microsecond cond_resched rate. This number can be from 1 to hundreds.
- kthread 0 has an additional job of optimizing the adding of estimators: they are first added in temp list (est_temp_list) and later kthread 0 distributes them to other kthreads. The optimization is based on the fact that newly added estimator should be estimated after 2 seconds, so we have the time to offload the adding to chain from controlling process to kthread 0.
- to add new estimators we use the last added kthread context (est_add_ktid). The new estimators are linked to the chains just before the estimated one, based on add_row. This ensures their estimation will start after 2 seconds. If estimators are added in bursts, common case if all services and dests are initially configured, we may spread the estimators to more chains and as result, reducing the initial delay below 2 seconds.
Many thanks to Jiri Wiesner for his valuable comments and for spending a lot of time reviewing and testing the changes on different platforms with 48-256 CPUs and 1-8 NUMA nodes under different cpufreq governors.
The new IPVS estimators do not use workqueue infrastructure because:
- The estimation can take long time when using multiple IPVS rules (eg. millions estimator structures) and especially when box has multiple CPUs due to the for_each_possible_cpu usage that expects packets from any CPU. With est_nice sysctl we have more control how to prioritize the estimation kthreads compared to other processes/kthreads that have latency requirements (such as servers). As a benefit, we can see these kthreads in top and decide if we will need some further control to limit their CPU usage (max number of structure to estimate per kthread).
- with kthreads we run code that is read-mostly, no write/lock operations to process the estimators in 2-second intervals.
- work items are one-shot: as estimators are processed every 2 seconds, they need to be re-added every time. This again loads the timers (add_timer) if we use delayed works, as there are no kthreads to do the timings.
[1] Report from Yunhong Jiang:
https://lore.kernel.org/netdev/[email protected]/ [2] https://marc.info/?l=linux-virtual-server&m=159679809118027&w=2 [3] Report from Dust: https://archive.linuxvirtualserver.org/html/lvs-devel/2020-12/msg00000.html
Please, pull these changes from:
git://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf-next.git
Thanks.
----------------------------------------------------------------
The following changes since commit 339e79dfb087075cbc27d3a902457574c4dac182:
Merge branch 'cleanup-ocelot_stats-exposure' (2022-11-22 15:36:46 +0100)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf-next.git HEAD
for you to fetch changes up to 144361c1949f227df9244302da02c258a363b674:
ipvs: run_estimation should control the kthread tasks (2022-12-10 22:44:43 +0100)
----------------------------------------------------------------
Dan Carpenter (1):
netfilter: nft_inner: fix IS_ERR() vs NULL check
Florian Westphal (2):
netfilter: conntrack: merge ipv4+ipv6 confirm functions
netfilter: conntrack: set icmpv6 redirects as RELATED
Julian Anastasov (6):
ipvs: add rcu protection to stats
ipvs: use common functions for stats allocation
ipvs: use u64_stats_t for the per-cpu counters
ipvs: use kthreads for stats estimation
ipvs: add est_cpulist and est_nice sysctl vars
ipvs: run_estimation should control the kthread tasks
Li Qiong (1):
netfilter: flowtable: add a 'default' case to flowtable datapath
Sriram Yagnaraman (1):
netfilter: conntrack: add sctp DATA_SENT state
Vishwanath Pai (1):
netfilter: ipset: Add support for new bitmask parameter
Documentation/networking/ipvs-sysctl.rst | 24 +- include/linux/netfilter/ipset/ip_set.h | 10 + include/net/ip_vs.h | 171 +++- include/net/netfilter/nf_conntrack_core.h | 3 +- include/uapi/linux/netfilter/ipset/ip_set.h | 2 + include/uapi/linux/netfilter/nf_conntrack_sctp.h | 1 + include/uapi/linux/netfilter/nfnetlink_cttimeout.h | 1 + net/bridge/netfilter/nf_conntrack_bridge.c | 32 +- net/netfilter/ipset/ip_set_hash_gen.h | 71 +- net/netfilter/ipset/ip_set_hash_ip.c | 19 +- net/netfilter/ipset/ip_set_hash_ipport.c | 24 +- net/netfilter/ipset/ip_set_hash_netnet.c | 26 +- net/netfilter/ipvs/ip_vs_core.c | 40 +- net/netfilter/ipvs/ip_vs_ctl.c | 448 +++++++++-- net/netfilter/ipvs/ip_vs_est.c | 882 +++++++++++++++++++-- net/netfilter/nf_conntrack_proto.c | 124 ++- net/netfilter/nf_conntrack_proto_icmpv6.c | 53 ++ net/netfilter/nf_conntrack_proto_sctp.c | 104 ++- net/netfilter/nf_conntrack_standalone.c | 8 + net/netfilter/nf_flow_table_ip.c | 8 + net/netfilter/nf_tables_api.c | 4 +- .../selftests/netfilter/conntrack_icmp_related.sh | 36 +- 22 files changed, 1731 insertions(+), 360 deletions(-)
Pablo Neira Ayuso
maintainerPATCH 1/12 netfilter: nft_inner: fix IS_ERR() vs NULL check
From: Dan Carpenter <[email protected]>
The __nft_expr_type_get() function returns NULL on error. It never returns error pointers.
--- net/netfilter/nf_tables_api.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/netfilter/nf_tables_api.c +2 -2
@@ -2873,8 +2873,8 @@ int nft_expr_inner_parse(const struct nft_ctx *ctx, const struct nlattr *nla,return -EINVAL ; type =__nft_expr_type_get (ctx->family, tb[NFTA_EXPR_NAME ]); -if (IS_ERR (type)) -return PTR_ERR (type); +if (!type) +return -ENOENT ;if (!type->inner_ops)return -EOPNOTSUPP ;
Signature · 1 line
2.30.2
Re: [PATCH net-next 01/12] netfilter: nft_inner: fix IS_ERR() vs NULL check
This series was applied to netdev/net-next.git (master) by Pablo Neira Ayuso <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by Pablo Neira Ayuso <[email protected]>:
Quoted 10 lines · On Sun, 11 Dec 2022 11:11:53 +0100 you wrote:
From: Dan Carpenter <[email protected]>
The
__nft_expr_type_get()function returns NULL on error. It never returns error pointers.Fixes 3a07327d10a0 ("netfilter: nft_inner: support for inner tunnel header matching")Signed-off-by Dan Carpenter <[email protected]>Signed-off-by Pablo Neira Ayuso <[email protected]>[...]
Here is the summary with links:
- [net-next,01/12] netfilter: nft_inner: fix IS_ERR() vs NULL check
https://git.kernel.org/netdev/net-next/c/98cbc40e4f7d - [net-next,02/12] netfilter: conntrack: add sctp DATA_SENT state https://git.kernel.org/netdev/net-next/c/bff3d0534804 - [net-next,03/12] netfilter: conntrack: merge ipv4+ipv6 confirm functions https://git.kernel.org/netdev/net-next/c/a70e483460d5 - [net-next,04/12] netfilter: ipset: Add support for new bitmask parameter https://git.kernel.org/netdev/net-next/c/e93745249505 - [net-next,05/12] netfilter: conntrack: set icmpv6 redirects as RELATED https://git.kernel.org/netdev/net-next/c/7d7cfb48d813 - [net-next,06/12] netfilter: flowtable: add a 'default' case to flowtable datapath https://git.kernel.org/netdev/net-next/c/895fa59647cd - [net-next,07/12] ipvs: add rcu protection to stats https://git.kernel.org/netdev/net-next/c/5df7d714d8cb - [net-next,08/12] ipvs: use common functions for stats allocation https://git.kernel.org/netdev/net-next/c/de39afb3d811 - [net-next,09/12] ipvs: use u64_stats_t for the per-cpu counters https://git.kernel.org/netdev/net-next/c/1dbd8d9a82e3 - [net-next,10/12] ipvs: use kthreads for stats estimation https://git.kernel.org/netdev/net-next/c/705dd3444081 - [net-next,11/12] ipvs: add est_cpulist and est_nice sysctl vars https://git.kernel.org/netdev/net-next/c/f0be83d54217 - [net-next,12/12] ipvs: run_estimation should control the kthread tasks https://git.kernel.org/netdev/net-next/c/144361c1949f
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Pablo Neira Ayuso
maintainerPATCH 2/12 netfilter: conntrack: add sctp DATA_SENT state
In reply to Pablo Neira AyusoFrom: Sriram Yagnaraman <[email protected]>
SCTP conntrack currently assumes that the SCTP endpoints will probe secondary paths using HEARTBEAT before sending traffic.
But, according to RFC 9260, SCTP endpoints can send any traffic on any of the confirmed paths after SCTP association is up. SCTP endpoints that sends INIT will confirm all peer addresses that upper layer configures, and the SCTP endpoint that receives COOKIE_ECHO will only confirm the address it sent the INIT_ACK to.
So, we can have a situation where the INIT sender can start to use secondary paths without the need to send HEARTBEAT. This patch allows DATA/SACK packets to create new connection tracking entry.
A new state has been added to indicate that a DATA/SACK chunk has been seen in the original direction - SCTP_CONNTRACK_DATA_SENT. State transitions mostly follows the HEARTBEAT_SENT, except on receiving HEARTBEAT/HEARTBEAT_ACK/DATA/SACK in the reply direction.
State transitions in original direction:
- DATA_SENT behaves similar to HEARTBEAT_SENT for all chunks, except that it remains in DATA_SENT on receving HEARTBEAT, HEARTBEAT_ACK/DATA/SACK chunks
State transitions in reply direction:
- DATA_SENT behaves similar to HEARTBEAT_SENT for all chunks, except that it moves to HEARTBEAT_ACKED on receiving HEARTBEAT/HEARTBEAT_ACK/DATA/SACK chunks
Note: This patch still doesn't solve the problem when the SCTP endpoint decides to use primary paths for association establishment but uses a secondary path for association shutdown. We still have to depend on timeout for connections to expire in such a case.
--- .../uapi/linux/netfilter/nf_conntrack_sctp.h | 1 + .../linux/netfilter/nfnetlink_cttimeout.h | 1 + net/netfilter/nf_conntrack_proto_sctp.c | 104 ++++++++++-------- net/netfilter/nf_conntrack_standalone.c | 8 ++ 4 files changed, 71 insertions(+), 43 deletions(-)
include/uapi/linux/netfilter/nf_conntrack_sctp.h +1 -0
@@ -16,6 +16,7 @@ enum sctp_conntrack {SCTP_CONNTRACK_SHUTDOWN_ACK_SENT ,SCTP_CONNTRACK_HEARTBEAT_SENT ,SCTP_CONNTRACK_HEARTBEAT_ACKED , +SCTP_CONNTRACK_DATA_SENT , SCTP_CONNTRACK_MAX };
include/uapi/linux/netfilter/nfnetlink_cttimeout.h +1 -0
@@ -95,6 +95,7 @@ enum ctattr_timeout_sctp {CTA_TIMEOUT_SCTP_SHUTDOWN_ACK_SENT ,CTA_TIMEOUT_SCTP_HEARTBEAT_SENT ,CTA_TIMEOUT_SCTP_HEARTBEAT_ACKED , +CTA_TIMEOUT_SCTP_DATA_SENT , __CTA_TIMEOUT_SCTP_MAX };#define CTA_TIMEOUT_SCTP_MAX (__CTA_TIMEOUT_SCTP_MAX - 1)
net/netfilter/nf_conntrack_proto_sctp.c +61 -43
168 lines not on this page · raw message
net/netfilter/nf_conntrack_standalone.c +8 -0
@@ -602,6 +602,7 @@ enum nf_ct_sysctl_index {NF_SYSCTL_CT_PROTO_TIMEOUT_SCTP_SHUTDOWN_ACK_SENT ,NF_SYSCTL_CT_PROTO_TIMEOUT_SCTP_HEARTBEAT_SENT ,NF_SYSCTL_CT_PROTO_TIMEOUT_SCTP_HEARTBEAT_ACKED , +NF_SYSCTL_CT_PROTO_TIMEOUT_SCTP_DATA_SENT , #endif #ifdef CONFIG_NF_CT_PROTO_DCCP NF_SYSCTL_CT_PROTO_TIMEOUT_DCCP_REQUEST,
@@ -892,6 +893,12 @@ static struct ctl_table nf_ct_sysctl_table[] = { .mode = 0644, .proc_handler = proc_dointvec_jiffies, }, + [NF_SYSCTL_CT_PROTO_TIMEOUT_SCTP_DATA_SENT] = { + .procname = "nf_conntrack_sctp_timeout_data_sent", + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_jiffies, + },#endif #ifdef CONFIG_NF_CT_PROTO_DCCP [NF_SYSCTL_CT_PROTO_TIMEOUT_DCCP_REQUEST] = {
@@ -1036,6 +1043,7 @@ static void nf_conntrack_standalone_init_sctp_sysctl(struct net *net,XASSIGN (SHUTDOWN_ACK_SENT , sn);XASSIGN (HEARTBEAT_SENT , sn);XASSIGN (HEARTBEAT_ACKED , sn); +XASSIGN (DATA_SENT , sn);#undef XASSIGN#endif }
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 3/12 netfilter: conntrack: merge ipv4+ipv6 confirm functions
In reply to Pablo Neira AyusoFrom: Florian Westphal <[email protected]>
No need to have distinct functions. After merge, ipv6 can avoid protooff computation if the connection neither needs sequence adjustment nor helper invocation -- this is the normal case.
--- include/net/netfilter/nf_conntrack_core.h | 3 +- net/bridge/netfilter/nf_conntrack_bridge.c | 32 +----- net/netfilter/nf_conntrack_proto.c | 124 +++++++++------------ 3 files changed, 57 insertions(+), 102 deletions(-)
include/net/netfilter/nf_conntrack_core.h +1 -2
@@ -71,8 +71,7 @@ static inline int nf_conntrack_confirm(struct sk_buff *skb)return ret; } -unsigned int nf_confirm (struct sk_buff *skb,unsigned int protoff, -struct nf_conn *ct,enum ip_conntrack_info ctinfo); +unsigned int nf_confirm (void *priv,struct sk_buff *skb,const struct nf_hook_state *state);void print_tuple (struct seq_file *s,const struct nf_conntrack_tuple *tuple,const struct nf_conntrack_l4proto *proto);
net/bridge/netfilter/nf_conntrack_bridge.c +1 -31
@@ -366,42 +366,12 @@ static int nf_ct_bridge_refrag_post(struct net *net, struct sock *sk,return br_dev_queue_push_xmit (net, sk, skb); } -static unsigned int nf_ct_bridge_confirm (struct sk_buff *skb) -{ -enum ip_conntrack_info ctinfo; -struct nf_conn *ct; -int protoff; - - ct =nf_ct_get (skb, &ctinfo); -if (!ct || ctinfo ==IP_CT_RELATED_REPLY ) -return nf_conntrack_confirm (skb); - -switch (skb->protocol) { -case htons (ETH_P_IP ): - protoff =skb_network_offset (skb) +ip_hdrlen (skb); -break ; -case htons (ETH_P_IPV6 ): { -unsigned char pnum =ipv6_hdr (skb)->nexthdr; -__be16 frag_off; - - protoff =ipv6_skip_exthdr (skb,sizeof (struct ipv6hdr ), &pnum, - &frag_off); -if (protoff <0 || (frag_off &htons (~0x7 )) !=0 ) -return nf_conntrack_confirm (skb); - } -break ; -default : -return NF_ACCEPT ; - } -return nf_confirm (skb, protoff, ct, ctinfo); -} -static unsigned int nf_ct_bridge_post (void *priv,struct sk_buff *skb,const struct nf_hook_state *state) {int ret; - ret =nf_ct_bridge_confirm (skb); + ret =nf_confirm (priv, skb, state);if (ret !=NF_ACCEPT )return ret;
net/netfilter/nf_conntrack_proto.c +55 -69
178 lines not on this page · raw message
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 4/12 netfilter: ipset: Add support for new bitmask parameter
In reply to Pablo Neira AyusoFrom: Vishwanath Pai <[email protected]>
Add a new parameter to complement the existing 'netmask' option. The main difference between netmask and bitmask is that bitmask takes any arbitrary ip address as input, it does not have to be a valid netmask.
The name of the new parameter is 'bitmask'. This lets us mask out arbitrary bits in the ip address, for example:
ipset create set1 hash:ip bitmask 255.128.255.0
ipset create set2 hash:ip,port family inet6 bitmask ffff::ff80
--- include/linux/netfilter/ipset/ip_set.h | 10 +++ include/uapi/linux/netfilter/ipset/ip_set.h | 2 + net/netfilter/ipset/ip_set_hash_gen.h | 71 ++++++++++++++++++--- net/netfilter/ipset/ip_set_hash_ip.c | 19 +++--- net/netfilter/ipset/ip_set_hash_ipport.c | 24 ++++++- net/netfilter/ipset/ip_set_hash_netnet.c | 26 ++++++-- 6 files changed, 126 insertions(+), 26 deletions(-)
include/linux/netfilter/ipset/ip_set.h +10 -0
@@ -515,6 +515,16 @@ ip_set_init_skbinfo(struct ip_set_skbinfo *skbinfo, *skbinfo = ext->skbinfo; } +static inline void +nf_inet_addr_mask_inplace (union nf_inet_addr *a1, +const union nf_inet_addr *mask) +{ + a1->all[0 ] &= mask->all[0 ]; + a1->all[1 ] &= mask->all[1 ]; + a1->all[2 ] &= mask->all[2 ]; + a1->all[3 ] &= mask->all[3 ]; +} +#define IP_SET_INIT_KEXT (skb, opt, set) \ { .bytes = (skb)->len, .packets = 1, .target = true,\ .timeout = ip_set_adt_opt_timeout(opt, set) }
include/uapi/linux/netfilter/ipset/ip_set.h +2 -0
@@ -85,6 +85,7 @@ enum {IPSET_ATTR_CADT_LINENO =IPSET_ATTR_LINENO ,/* 9 */ IPSET_ATTR_MARK ,/* 10 */ IPSET_ATTR_MARKMASK ,/* 11 */ +IPSET_ATTR_BITMASK ,/* 12 */ /* Reserve empty slots */ IPSET_ATTR_CADT_MAX = 16,/* Create-only specific attributes */
@@ -153,6 +154,7 @@ enum ipset_errno {IPSET_ERR_COMMENT ,IPSET_ERR_INVALID_MARKMASK ,IPSET_ERR_SKBINFO , +IPSET_ERR_BITMASK_NETMASK_EXCL ,/* Type specific error codes */ IPSET_ERR_TYPE_SPECIFIC = 4352,
net/netfilter/ipset/ip_set_hash_gen.h +62 -9
121 lines not on this page · raw message
net/netfilter/ipset/ip_set_hash_ip.c +8 -11
@@ -24,7 +24,8 @@/* 2 Comments support */ /* 3 Forceadd support */ /* 4 skbinfo support */ -#define IPSET_TYPE_REV_MAX 5/* bucketsize, initval support */ +/* 5 bucketsize, initval support */ +#define IPSET_TYPE_REV_MAX 6/* bitmask support */ MODULE_LICENSE ("GPL" );MODULE_AUTHOR ("Jozsef Kadlecsik <[email protected]>" );
@@ -34,6 +35,7 @@ MODULE_ALIAS("ip_set_hash:ip");/* Type specific function prefix */ #define HTYPE hash_ip#define IP_SET_HASH_WITH_NETMASK +#define IP_SET_HASH_WITH_BITMASK /* IPv4 variant */
@@ -86,7 +88,7 @@ hash_ip4_kadt(struct ip_set *set, const struct sk_buff *skb,__be32 ip;ip4addrptr (skb, opt->flags &IPSET_DIM_ONE_SRC , &ip); - ip &=ip_set_netmask (h->netmask); + ip &= h->bitmask.ip;if (ip ==0 )return -EINVAL ;
@@ -119,7 +121,7 @@ hash_ip4_uadt(struct ip_set *set, struct nlattr *tb[],if (ret)return ret; - ip &=ip_set_hostmask (h->netmask); + ip &=ntohl (h->bitmask.ip); e.ip =htonl (ip);if (e.ip ==0 )return -IPSET_ERR_HASH_ELEM ;
@@ -187,12 +189,6 @@ hash_ip6_data_equal(const struct hash_ip6_elem *ip1,return ipv6_addr_equal (&ip1->ip.in6, &ip2->ip.in6); } -static void -hash_ip6_netmask (union nf_inet_addr *ip,u8 prefix) -{ -ip6_netmask (ip, prefix); -} -static bool hash_ip6_data_list (struct sk_buff *skb,const struct hash_ip6_elem *e) {
@@ -229,7 +225,7 @@ hash_ip6_kadt(struct ip_set *set, const struct sk_buff *skb,struct ip_set_ext ext =IP_SET_INIT_KEXT (skb, opt, set);ip6addrptr (skb, opt->flags &IPSET_DIM_ONE_SRC , &e.ip.in6); -hash_ip6_netmask (&e.ip, h->netmask); +nf_inet_addr_mask_inplace (&e.ip, &h->bitmask);if (ipv6_addr_any (&e.ip.in6))return -EINVAL ;
@@ -268,7 +264,7 @@ hash_ip6_uadt(struct ip_set *set, struct nlattr *tb[],if (ret)return ret; -hash_ip6_netmask (&e.ip, h->netmask); +nf_inet_addr_mask_inplace (&e.ip, &h->bitmask);if (ipv6_addr_any (&e.ip.in6))return -IPSET_ERR_HASH_ELEM ;
@@ -295,6 +291,7 @@ static struct ip_set_type hash_ip_type __read_mostly = { [IPSET_ATTR_RESIZE] = { .type = NLA_U8 }, [IPSET_ATTR_TIMEOUT] = { .type = NLA_U32 }, [IPSET_ATTR_NETMASK] = { .type = NLA_U8 }, + [IPSET_ATTR_BITMASK] = { .type = NLA_NESTED }, [IPSET_ATTR_CADT_FLAGS ] = { .type =NLA_U32 }, }, .adt_policy = {
net/netfilter/ipset/ip_set_hash_ipport.c +23 -1
@@ -26,7 +26,8 @@/* 3 Comments support added */ /* 4 Forceadd support added */ /* 5 skbinfo support added */ -#define IPSET_TYPE_REV_MAX 6/* bucketsize, initval support added */ +/* 6 bucketsize, initval support added */ +#define IPSET_TYPE_REV_MAX 7/* bitmask support added */ MODULE_LICENSE ("GPL" );MODULE_AUTHOR ("Jozsef Kadlecsik <[email protected]>" );
@@ -35,6 +36,8 @@ MODULE_ALIAS("ip_set_hash:ip,port");/* Type specific function prefix */ #define HTYPE hash_ipport +#define IP_SET_HASH_WITH_NETMASK +#define IP_SET_HASH_WITH_BITMASK /* IPv4 variant */
@@ -92,12 +95,16 @@ hash_ipport4_kadt(struct ip_set *set, const struct sk_buff *skb,ipset_adtfn adtfn = set->variant->adt[adt];struct hash_ipport4_elem e = { .ip =0 };struct ip_set_ext ext =IP_SET_INIT_KEXT (skb, opt, set); +const struct MTYPE *h = set->data;if (!ip_set_get_ip4_port (skb, opt->flags &IPSET_DIM_TWO_SRC , &e.port, &e.proto))return -EINVAL ;ip4addrptr (skb, opt->flags &IPSET_DIM_ONE_SRC , &e.ip); + e.ip &= h->bitmask.ip; +if (e.ip ==0 ) +return -EINVAL ;return adtfn (set, &e, &ext, &opt->ext, opt->cmdflags); }
@@ -129,6 +136,10 @@ hash_ipport4_uadt(struct ip_set *set, struct nlattr *tb[],if (ret)return ret; + e.ip &= h->bitmask.ip; +if (e.ip ==0 ) +return -EINVAL ; + e.port =nla_get_be16 (tb[IPSET_ATTR_PORT ]);if (tb[IPSET_ATTR_PROTO ]) {
@@ -253,12 +264,17 @@ hash_ipport6_kadt(struct ip_set *set, const struct sk_buff *skb,ipset_adtfn adtfn = set->variant->adt[adt];struct hash_ipport6_elem e = { .ip = { .all = {0 } } };struct ip_set_ext ext =IP_SET_INIT_KEXT (skb, opt, set); +const struct MTYPE *h = set->data;if (!ip_set_get_ip6_port (skb, opt->flags &IPSET_DIM_TWO_SRC , &e.port, &e.proto))return -EINVAL ;ip6addrptr (skb, opt->flags &IPSET_DIM_ONE_SRC , &e.ip.in6); +nf_inet_addr_mask_inplace (&e.ip, &h->bitmask); +if (ipv6_addr_any (&e.ip.in6)) +return -EINVAL ; +return adtfn (set, &e, &ext, &opt->ext, opt->cmdflags); }
@@ -298,6 +314,10 @@ hash_ipport6_uadt(struct ip_set *set, struct nlattr *tb[],if (ret)return ret; +nf_inet_addr_mask_inplace (&e.ip, &h->bitmask); +if (ipv6_addr_any (&e.ip.in6)) +return -EINVAL ; + e.port =nla_get_be16 (tb[IPSET_ATTR_PORT ]);if (tb[IPSET_ATTR_PROTO ]) {
@@ -356,6 +376,8 @@ static struct ip_set_type hash_ipport_type __read_mostly = { [IPSET_ATTR_PROTO] = { .type = NLA_U8 }, [IPSET_ATTR_TIMEOUT] = { .type = NLA_U32 }, [IPSET_ATTR_CADT_FLAGS] = { .type = NLA_U32 }, + [IPSET_ATTR_NETMASK] = { .type = NLA_U8 }, + [IPSET_ATTR_BITMASK ] = { .type =NLA_NESTED }, }, .adt_policy = { [IPSET_ATTR_IP ] = { .type =NLA_NESTED },
net/netfilter/ipset/ip_set_hash_netnet.c +21 -5
@@ -23,7 +23,8 @@#define IPSET_TYPE_REV_MIN 0/* 1 Forceadd support added */ /* 2 skbinfo support added */ -#define IPSET_TYPE_REV_MAX 3/* bucketsize, initval support added */ +/* 3 bucketsize, initval support added */ +#define IPSET_TYPE_REV_MAX 4/* bitmask support added */ MODULE_LICENSE ("GPL" );MODULE_AUTHOR ("Oliver Smith <[email protected]>" );
@@ -33,6 +34,8 @@ MODULE_ALIAS("ip_set_hash:net,net");/* Type specific function prefix */ #define HTYPE hash_netnet#define IP_SET_HASH_WITH_NETS +#define IP_SET_HASH_WITH_NETMASK +#define IP_SET_HASH_WITH_BITMASK #define IPSET_NET_COUNT 2/* IPv4 variants */
@@ -153,8 +156,8 @@ hash_netnet4_kadt(struct ip_set *set, const struct sk_buff *skb,ip4addrptr (skb, opt->flags &IPSET_DIM_ONE_SRC , &e.ip[0 ]);ip4addrptr (skb, opt->flags &IPSET_DIM_TWO_SRC , &e.ip[1 ]); - e.ip[0 ] &=ip_set_netmask (e.cidr[0 ]); - e.ip[1 ] &=ip_set_netmask (e.cidr[1 ]); + e.ip[0 ] &= (ip_set_netmask (e.cidr[0 ]) & h->bitmask.ip); + e.ip[1 ] &= (ip_set_netmask (e.cidr[1 ]) & h->bitmask.ip);return adtfn (set, &e, &ext, &opt->ext, opt->cmdflags); }
@@ -213,8 +216,8 @@ hash_netnet4_uadt(struct ip_set *set, struct nlattr *tb[],if (adt ==IPSET_TEST || !(tb[IPSET_ATTR_IP_TO ] || tb[IPSET_ATTR_IP2_TO ])) { - e.ip[0 ] =htonl (ip &ip_set_hostmask (e.cidr[0 ])); - e.ip[1 ] =htonl (ip2_from &ip_set_hostmask (e.cidr[1 ])); + e.ip[0 ] =htonl (ip &ntohl (h->bitmask.ip) &ip_set_hostmask (e.cidr[0 ])); + e.ip[1 ] =htonl (ip2_from &ntohl (h->bitmask.ip) &ip_set_hostmask (e.cidr[1 ])); ret =adtfn (set, &e, &ext, &ext, flags);return ip_set_enomatch (ret, flags, adt, set) ? -ret :ip_set_eexist (ret, flags) ?0 : ret;
@@ -404,6 +407,11 @@ hash_netnet6_kadt(struct ip_set *set, const struct sk_buff *skb,ip6_netmask (&e.ip[0 ], e.cidr[0 ]);ip6_netmask (&e.ip[1 ], e.cidr[1 ]); +nf_inet_addr_mask_inplace (&e.ip[0 ], &h->bitmask); +nf_inet_addr_mask_inplace (&e.ip[1 ], &h->bitmask); +if (e.cidr[0 ] ==HOST_MASK &&ipv6_addr_any (&e.ip[0 ].in6)) +return -EINVAL ; +return adtfn (set, &e, &ext, &opt->ext, opt->cmdflags); }
@@ -414,6 +422,7 @@ hash_netnet6_uadt(struct ip_set *set, struct nlattr *tb[],ipset_adtfn adtfn = set->variant->adt[adt];struct hash_netnet6_elem e = { };struct ip_set_ext ext =IP_SET_INIT_UEXT (set); +const struct hash_netnet6 *h = set->data;int ret;if (tb[IPSET_ATTR_LINENO ])
@@ -453,6 +462,11 @@ hash_netnet6_uadt(struct ip_set *set, struct nlattr *tb[],ip6_netmask (&e.ip[0 ], e.cidr[0 ]);ip6_netmask (&e.ip[1 ], e.cidr[1 ]); +nf_inet_addr_mask_inplace (&e.ip[0 ], &h->bitmask); +nf_inet_addr_mask_inplace (&e.ip[1 ], &h->bitmask); +if (e.cidr[0 ] ==HOST_MASK &&ipv6_addr_any (&e.ip[0 ].in6)) +return -IPSET_ERR_HASH_ELEM ; +if (tb[IPSET_ATTR_CADT_FLAGS ]) {u32 cadt_flags =ip_set_get_h32 (tb[IPSET_ATTR_CADT_FLAGS ]);
@@ -484,6 +498,8 @@ static struct ip_set_type hash_netnet_type __read_mostly = { [IPSET_ATTR_RESIZE] = { .type = NLA_U8 }, [IPSET_ATTR_TIMEOUT] = { .type = NLA_U32 }, [IPSET_ATTR_CADT_FLAGS] = { .type = NLA_U32 }, + [IPSET_ATTR_NETMASK] = { .type = NLA_U8 }, + [IPSET_ATTR_BITMASK ] = { .type =NLA_NESTED }, }, .adt_policy = { [IPSET_ATTR_IP ] = { .type =NLA_NESTED },
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 5/12 netfilter: conntrack: set icmpv6 redirects as RELATED
In reply to Pablo Neira AyusoFrom: Florian Westphal <[email protected]>
icmp conntrack will set icmp redirects as RELATED, but icmpv6 will not do this.
For icmpv6, only icmp errors (code <= 128) are examined for RELATED state. ICMPV6 Redirects are part of neighbour discovery mechanism, those are handled by marking a selected subset (e.g. neighbour solicitations) as UNTRACKED, but not REDIRECT -- they will thus be flagged as INVALID.
Add minimal support for REDIRECTs. No parsing of neighbour options is added for simplicity, so this will only check that we have the embeeded original header (ND_OPT_REDIRECT_HDR), and then attempt to do a flow lookup for this tuple.
Also extend the existing test case to cover redirects.
--- net/netfilter/nf_conntrack_proto_icmpv6.c | 53 +++++++++++++++++++ .../netfilter/conntrack_icmp_related.sh | 36 ++++++++++++- 2 files changed, 87 insertions(+), 2 deletions(-)
net/netfilter/nf_conntrack_proto_icmpv6.c +53 -0
@@ -129,6 +129,56 @@ static void icmpv6_error_log(const struct sk_buff *skb,nf_l4proto_log_invalid (skb, state,IPPROTO_ICMPV6 ,"%s" , msg); } +static noinline_for_stack int +nf_conntrack_icmpv6_redirect (struct nf_conn *tmpl,struct sk_buff *skb, +unsigned int dataoff, +const struct nf_hook_state *state) +{ +u8 hl =ipv6_hdr (skb)->hop_limit; +union nf_inet_addr outer_daddr; +union { +struct nd_opt_hdr nd_opt; +struct rd_msg rd_msg; + } tmp; +const struct nd_opt_hdr *nd_opt; +const struct rd_msg *rd_msg; + + rd_msg =skb_header_pointer (skb, dataoff,sizeof (*rd_msg), &tmp.rd_msg); +if (!rd_msg) { +icmpv6_error_log (skb, state,"short redirect" ); +return -NF_ACCEPT ; + } + +if (rd_msg->icmph.icmp6_code !=0 ) +return NF_ACCEPT ; + +if (hl !=255 || !(ipv6_addr_type (&ipv6_hdr (skb)->saddr) &IPV6_ADDR_LINKLOCAL )) { +icmpv6_error_log (skb, state,"invalid saddr or hoplimit for redirect" ); +return -NF_ACCEPT ; + } + + dataoff +=sizeof (*rd_msg); + +/* warning: rd_msg no longer usable after this call */ + nd_opt =skb_header_pointer (skb, dataoff,sizeof (*nd_opt), &tmp.nd_opt); +if (!nd_opt || nd_opt->nd_opt_len ==0 ) { +icmpv6_error_log (skb, state,"redirect without options" ); +return -NF_ACCEPT ; + } + +/* We could call ndisc_parse_options(), but it would need +* skb_linearize() and a bit more work. +*/ +if (nd_opt->nd_opt_type !=ND_OPT_REDIRECT_HDR ) +return NF_ACCEPT ; + +memcpy (&outer_daddr.ip6, &ipv6_hdr (skb)->daddr, +sizeof (outer_daddr.ip6)); + dataoff +=8 ; +return nf_conntrack_inet_error (tmpl, skb, dataoff, state, +IPPROTO_ICMPV6 , &outer_daddr); +} + int nf_conntrack_icmpv6_error(struct nf_conn *tmpl, struct sk_buff *skb, unsigned int dataoff,
@@ -159,6 +209,9 @@ int nf_conntrack_icmpv6_error(struct nf_conn *tmpl,return NF_ACCEPT ; } +if (icmp6h->icmp6_type ==NDISC_REDIRECT ) +return nf_conntrack_icmpv6_redirect (tmpl, skb, dataoff, state); +/* is not error message ? */ if (icmp6h->icmp6_type >=128 )return NF_ACCEPT ;
tools/testing/selftests/netfilter/conntrack_icmp_related.sh +34 -2
@@ -35,6 +35,8 @@ cleanup() { for i in 1 2;do ip netns del nsrouter$i;done } +trap cleanup EXIT + ipv4() { echo -n 192.168.$1.2 }
@@ -146,11 +148,17 @@ ip netns exec nsclient1 nft -f - <<EOF table inet filter { counter unknown { } counter related { } + counter redir4 { } + counter redir6 { } chain input { type filter hook input priority 0; policy accept; - meta l4proto { icmp, icmpv6 } ct state established,untracked accept + icmp type "redirect" ct state "related" counter name "redir4" accept + icmpv6 type "nd-redirect" ct state "related" counter name "redir6" accept + + meta l4proto { icmp, icmpv6 } ct state established,untracked accept meta l4proto { icmp, icmpv6 } ct state "related" counter name "related" accept + counter name "unknown" drop } }
@@ -279,5 +287,29 @@ else echo "ERROR: icmp error RELATED state test has failed" fi -cleanup +# add 'bad' route, expect icmp REDIRECT to be generated +ip netns exec nsclient1 ip route add 192.168.1.42 via 192.168.1.1 +ip netns exec nsclient1 ip route add dead:1::42 via dead:1::1 + +ip netns exec "nsclient1" ping -q -c 2 192.168.1.42 > /dev/null + +expect="packets 1 bytes 112" +check_counter nsclient1 "redir4" "$expect" +if [ $? -ne 0 ];then + ret=1 +fi + +ip netns exec "nsclient1" ping -c 1 dead:1::42 > /dev/null +expect="packets 1 bytes 192" +check_counter nsclient1 "redir6" "$expect" +if [ $? -ne 0 ];then + ret=1 +fi + +if [ $ret -eq 0 ];then + echo "PASS: icmp redirects had RELATED state" +else + echo "ERROR: icmp redirect RELATED state test has failed" +fi + exit $ret
Signature · 1 line
2.30.2
Re: [PATCH net-next 05/12] netfilter: conntrack: set icmpv6 redirects as RELATED
Quoted 23 lines · Pablo Neira Ayuso
From: Florian Westphal <[email protected]>
icmp conntrack will set icmp redirects as RELATED, but icmpv6 will not do this.
For icmpv6, only icmp errors (code <= 128) are examined for RELATED state. ICMPV6 Redirects are part of neighbour discovery mechanism, those are handled by marking a selected subset (e.g. neighbour solicitations) as UNTRACKED, but not REDIRECT -- they will thus be flagged as INVALID.
Add minimal support for REDIRECTs. No parsing of neighbour options is added for simplicity, so this will only check that we have the embeeded original header (
ND_OPT_REDIRECT_HDR), and then attempt to do a flow lookup for this tuple.Also extend the existing test case to cover redirects.
Fixes 9fb9cbb1082d ("[NETFILTER]: Add nf_conntrack subsystem.")Reported-by Eric Garver <[email protected]>Signed-off-by Florian Westphal <[email protected]>Acked-by Eric Garver <[email protected]>Signed-off-by Pablo Neira Ayuso <[email protected]>
Quoted 4 lines
--- net/netfilter/nf_conntrack_proto_icmpv6.c | 53 +++++++++++++++++++ .../netfilter/conntrack_icmp_related.sh | 36 ++++++++++++- 2 files changed, 87 insertions(+), 2 deletions(-)
Hi, Florian.
The new ipv4 redirects test case doesn't seem to work, is there a problem with my testing steps?
# sh tools/testing/selftests/netfilter/conntrack_icmp_related.sh
PASS: icmp mtu error had RELATED state
ERROR: counter redir4 in nsclient1 has unexpected value (expected packets 1 bytes 112)
table inet filter {
counter redir4 {
packets 0 bytes 0
}
}
ERROR: icmp redirect RELATED state test has failed.The test is based on commit f6feea56f66d ("Merge tag 'mm-hotfixes-stable-2023-02-13-13-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm")
Signature · 1 line
Wang Hai
Re: [PATCH net-next 05/12] netfilter: conntrack: set icmpv6 redirects as RELATED
Quoted 43 lines · Wang Hai
Quoted 23 lines · Pablo Neira Ayuso
From: Florian Westphal <[email protected]>
icmp conntrack will set icmp redirects as RELATED, but icmpv6 will not do this.
For icmpv6, only icmp errors (code <= 128) are examined for RELATED state. ICMPV6 Redirects are part of neighbour discovery mechanism, those are handled by marking a selected subset (e.g. neighbour solicitations) as UNTRACKED, but not REDIRECT -- they will thus be flagged as INVALID.
Add minimal support for REDIRECTs. No parsing of neighbour options is added for simplicity, so this will only check that we have the embeeded original header (
ND_OPT_REDIRECT_HDR), and then attempt to do a flow lookup for this tuple.Also extend the existing test case to cover redirects.
Fixes 9fb9cbb1082d ("[NETFILTER]: Add nf_conntrack subsystem.")Reported-by Eric Garver <[email protected]>Signed-off-by Florian Westphal <[email protected]>Acked-by Eric Garver <[email protected]>Signed-off-by Pablo Neira Ayuso <[email protected]>Quoted 4 lines
---
net/netfilter/nf_conntrack_proto_icmpv6.c | 53 +++++++++++++++++++ .../netfilter/conntrack_icmp_related.sh | 36 ++++++++++++-2 files changed, 87 insertions(+), 2 deletions(-)Hi, Florian.
The new ipv4 redirects test case doesn't seem to work, is there a problem with my testing steps?
# sh tools/testing/selftests/netfilter/conntrack_icmp_related.sh PASS: icmp mtu error had RELATED state ERROR: counter redir4 in nsclient1 has unexpected value (expected packets 1 bytes 112) table inet filter { counter redir4 { packets 0 bytes 0 } } ERROR: icmp redirect RELATED state test has failed.
Quoted 2 lines
The test is based on commit
f6feea56f66d("Merge tag 'mm-hotfixes-stable-2023-02-13-13-50' ofgit://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm")
Hi, Florian.
I found the reason why it failed. This needs to be configured on the host with net.ipv4.conf.default.send_redirects=1.
Sorry to bother you.
Signature · 1 line
Wang Hai
Pablo Neira Ayuso
maintainerPATCH 6/12 netfilter: flowtable: add a 'default' case to flowtable datapath
In reply to Pablo Neira AyusoFrom: Li Qiong <[email protected]>
Add a 'default' case in case return a uninitialized value of ret, this should not ever happen since the follow transmit path types:
- FLOW_OFFLOAD_XMIT_UNSPEC
- FLOW_OFFLOAD_XMIT_TC
are never observed from this path. Add this check for safety reasons.
--- net/netfilter/nf_flow_table_ip.c | 8 ++++++++ 1 file changed, 8 insertions(+)
net/netfilter/nf_flow_table_ip.c +8 -0
@@ -421,6 +421,10 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,if (ret ==NF_DROP )flow_offload_teardown (flow);break ; +default : +WARN_ON_ONCE (1 ); + ret =NF_DROP ; +break ; }return ret;
@@ -682,6 +686,10 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,if (ret ==NF_DROP )flow_offload_teardown (flow);break ; +default : +WARN_ON_ONCE (1 ); + ret =NF_DROP ; +break ; }return ret;
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 7/12 ipvs: add rcu protection to stats
In reply to Pablo Neira AyusoFrom: Julian Anastasov <[email protected]>
In preparation to using RCU locking for the list with estimators, make sure the struct ip_vs_stats are released after RCU grace period by using RCU callbacks. This affects ipvs->tot_stats where we can not use RCU callbacks for ipvs, so we use allocated struct ip_vs_stats_rcu. For services and dests we force RCU callbacks for all cases.
--- include/net/ip_vs.h | 8 ++++- net/netfilter/ipvs/ip_vs_core.c | 10 ++++-- net/netfilter/ipvs/ip_vs_ctl.c | 64 ++++++++++++++++++++++----------- 3 files changed, 57 insertions(+), 25 deletions(-)
include/net/ip_vs.h +7 -1
@@ -405,6 +405,11 @@ struct ip_vs_stats {struct ip_vs_kstats kstats0;/* reset values */ }; +struct ip_vs_stats_rcu { +struct ip_vs_stats s; +struct rcu_head rcu_head; +}; +struct dst_entry ;struct iphdr ;struct ip_vs_conn ;
@@ -688,6 +693,7 @@ struct ip_vs_dest {union nf_inet_addr vaddr;/* virtual IP address */ __u32 vfwmark;/* firewall mark of service */ +struct rcu_head rcu_head;struct list_head t_list;/* in dest_trash */ unsigned int in_rs_table:1; /* we are in rs_table */ };
@@ -869,7 +875,7 @@ struct netns_ipvs {atomic_t conn_count;/* connection counter */ /* ip_vs_ctl */ -struct ip_vs_stats tot_stats;/* Statistics & est. */ +struct ip_vs_stats_rcu *tot_stats;/* Statistics & est. */ int num_services;/* no of virtual services */ int num_services6;/* IPv6 virtual services */
net/netfilter/ipvs/ip_vs_core.c +7 -3
@@ -143,7 +143,7 @@ ip_vs_in_stats(struct ip_vs_conn *cp, struct sk_buff *skb) s->cnt.inbytes += skb->len;u64_stats_update_end (&s->syncp); - s =this_cpu_ptr (ipvs->tot_stats.cpustats); + s =this_cpu_ptr (ipvs->tot_stats->s.cpustats);u64_stats_update_begin (&s->syncp); s->cnt.inpkts++; s->cnt.inbytes += skb->len;
@@ -179,7 +179,7 @@ ip_vs_out_stats(struct ip_vs_conn *cp, struct sk_buff *skb) s->cnt.outbytes += skb->len;u64_stats_update_end (&s->syncp); - s =this_cpu_ptr (ipvs->tot_stats.cpustats); + s =this_cpu_ptr (ipvs->tot_stats->s.cpustats);u64_stats_update_begin (&s->syncp); s->cnt.outpkts++; s->cnt.outbytes += skb->len;
@@ -208,7 +208,7 @@ ip_vs_conn_stats(struct ip_vs_conn *cp, struct ip_vs_service *svc) s->cnt.conns++;u64_stats_update_end (&s->syncp); - s =this_cpu_ptr (ipvs->tot_stats.cpustats); + s =this_cpu_ptr (ipvs->tot_stats->s.cpustats);u64_stats_update_begin (&s->syncp); s->cnt.conns++;u64_stats_update_end (&s->syncp);
@@ -2448,6 +2448,10 @@ static void __exit ip_vs_cleanup(void)ip_vs_conn_cleanup ();ip_vs_protocol_cleanup ();ip_vs_control_cleanup (); +/* common rcu_barrier() used by: +* - ip_vs_control_cleanup() +*/ +rcu_barrier ();pr_info ("ipvs unloaded.\n" ); }
net/netfilter/ipvs/ip_vs_ctl.c +43 -21
171 lines not on this page · raw message
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 8/12 ipvs: use common functions for stats allocation
In reply to Pablo Neira AyusoFrom: Julian Anastasov <[email protected]>
Move alloc_percpu/free_percpu logic in new functions
--- include/net/ip_vs.h | 5 ++ net/netfilter/ipvs/ip_vs_ctl.c | 96 +++++++++++++++++++--------------- 2 files changed, 60 insertions(+), 41 deletions(-)
include/net/ip_vs.h +5 -0
@@ -410,6 +410,11 @@ struct ip_vs_stats_rcu {struct rcu_head rcu_head; }; +int ip_vs_stats_init_alloc (struct ip_vs_stats *s); +struct ip_vs_stats *ip_vs_stats_alloc (void ); +void ip_vs_stats_release (struct ip_vs_stats *stats); +void ip_vs_stats_free (struct ip_vs_stats *stats); +struct dst_entry ;struct iphdr ;struct ip_vs_conn ;
net/netfilter/ipvs/ip_vs_ctl.c +55 -41
190 lines not on this page · raw message
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 9/12 ipvs: use u64_stats_t for the per-cpu counters
In reply to Pablo Neira AyusoFrom: Julian Anastasov <[email protected]>
Use the provided u64_stats_t type to avoid
load/store tearing.
--- include/net/ip_vs.h | 10 +++++----- net/netfilter/ipvs/ip_vs_core.c | 30 +++++++++++++++--------------- net/netfilter/ipvs/ip_vs_ctl.c | 10 +++++----- net/netfilter/ipvs/ip_vs_est.c | 20 ++++++++++---------- 4 files changed, 35 insertions(+), 35 deletions(-)
include/net/ip_vs.h +5 -5
@@ -351,11 +351,11 @@ struct ip_vs_seq {/* counters per cpu */ struct ip_vs_counters { -__u64 conns;/* connections scheduled */ -__u64 inpkts;/* incoming packets */ -__u64 outpkts;/* outgoing packets */ -__u64 inbytes;/* incoming bytes */ -__u64 outbytes;/* outgoing bytes */ +u64_stats_t conns;/* connections scheduled */ +u64_stats_t inpkts;/* incoming packets */ +u64_stats_t outpkts;/* outgoing packets */ +u64_stats_t inbytes;/* incoming bytes */ +u64_stats_t outbytes;/* outgoing bytes */ };/* Stats per cpu */ struct ip_vs_cpu_stats {
net/netfilter/ipvs/ip_vs_core.c +15 -15
@@ -132,21 +132,21 @@ ip_vs_in_stats(struct ip_vs_conn *cp, struct sk_buff *skb) s =this_cpu_ptr (dest->stats.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.inpkts++; - s->cnt.inbytes += skb->len; +u64_stats_inc (&s->cnt.inpkts); +u64_stats_add (&s->cnt.inbytes, skb->len);u64_stats_update_end (&s->syncp); svc =rcu_dereference (dest->svc); s =this_cpu_ptr (svc->stats.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.inpkts++; - s->cnt.inbytes += skb->len; +u64_stats_inc (&s->cnt.inpkts); +u64_stats_add (&s->cnt.inbytes, skb->len);u64_stats_update_end (&s->syncp); s =this_cpu_ptr (ipvs->tot_stats->s.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.inpkts++; - s->cnt.inbytes += skb->len; +u64_stats_inc (&s->cnt.inpkts); +u64_stats_add (&s->cnt.inbytes, skb->len);u64_stats_update_end (&s->syncp);local_bh_enable ();
@@ -168,21 +168,21 @@ ip_vs_out_stats(struct ip_vs_conn *cp, struct sk_buff *skb) s =this_cpu_ptr (dest->stats.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.outpkts++; - s->cnt.outbytes += skb->len; +u64_stats_inc (&s->cnt.outpkts); +u64_stats_add (&s->cnt.outbytes, skb->len);u64_stats_update_end (&s->syncp); svc =rcu_dereference (dest->svc); s =this_cpu_ptr (svc->stats.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.outpkts++; - s->cnt.outbytes += skb->len; +u64_stats_inc (&s->cnt.outpkts); +u64_stats_add (&s->cnt.outbytes, skb->len);u64_stats_update_end (&s->syncp); s =this_cpu_ptr (ipvs->tot_stats->s.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.outpkts++; - s->cnt.outbytes += skb->len; +u64_stats_inc (&s->cnt.outpkts); +u64_stats_add (&s->cnt.outbytes, skb->len);u64_stats_update_end (&s->syncp);local_bh_enable ();
@@ -200,17 +200,17 @@ ip_vs_conn_stats(struct ip_vs_conn *cp, struct ip_vs_service *svc) s =this_cpu_ptr (cp->dest->stats.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.conns++; +u64_stats_inc (&s->cnt.conns);u64_stats_update_end (&s->syncp); s =this_cpu_ptr (svc->stats.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.conns++; +u64_stats_inc (&s->cnt.conns);u64_stats_update_end (&s->syncp); s =this_cpu_ptr (ipvs->tot_stats->s.cpustats);u64_stats_update_begin (&s->syncp); - s->cnt.conns++; +u64_stats_inc (&s->cnt.conns);u64_stats_update_end (&s->syncp);local_bh_enable ();
net/netfilter/ipvs/ip_vs_ctl.c +5 -5
@@ -2335,11 +2335,11 @@ static int ip_vs_stats_percpu_show(struct seq_file *seq, void *v)do { start =u64_stats_fetch_begin (&u->syncp); - conns = u->cnt.conns; - inpkts = u->cnt.inpkts; - outpkts = u->cnt.outpkts; - inbytes = u->cnt.inbytes; - outbytes = u->cnt.outbytes; + conns =u64_stats_read (&u->cnt.conns); + inpkts =u64_stats_read (&u->cnt.inpkts); + outpkts =u64_stats_read (&u->cnt.outpkts); + inbytes =u64_stats_read (&u->cnt.inbytes); + outbytes =u64_stats_read (&u->cnt.outbytes); }while (u64_stats_fetch_retry (&u->syncp, start)); seq_printf(seq, "%3X %8LX %8LX %8LX %16LX %16LX\n",
net/netfilter/ipvs/ip_vs_est.c +10 -10
@@ -67,11 +67,11 @@ static void ip_vs_read_cpu_stats(struct ip_vs_kstats *sum,if (add) {do { start =u64_stats_fetch_begin (&s->syncp); - conns = s->cnt.conns; - inpkts = s->cnt.inpkts; - outpkts = s->cnt.outpkts; - inbytes = s->cnt.inbytes; - outbytes = s->cnt.outbytes; + conns =u64_stats_read (&s->cnt.conns); + inpkts =u64_stats_read (&s->cnt.inpkts); + outpkts =u64_stats_read (&s->cnt.outpkts); + inbytes =u64_stats_read (&s->cnt.inbytes); + outbytes =u64_stats_read (&s->cnt.outbytes); }while (u64_stats_fetch_retry (&s->syncp, start)); sum->conns += conns; sum->inpkts += inpkts;
@@ -82,11 +82,11 @@ static void ip_vs_read_cpu_stats(struct ip_vs_kstats *sum, add = true;do { start =u64_stats_fetch_begin (&s->syncp); - sum->conns = s->cnt.conns; - sum->inpkts = s->cnt.inpkts; - sum->outpkts = s->cnt.outpkts; - sum->inbytes = s->cnt.inbytes; - sum->outbytes = s->cnt.outbytes; + sum->conns =u64_stats_read (&s->cnt.conns); + sum->inpkts =u64_stats_read (&s->cnt.inpkts); + sum->outpkts =u64_stats_read (&s->cnt.outpkts); + sum->inbytes =u64_stats_read (&s->cnt.inbytes); + sum->outbytes =u64_stats_read (&s->cnt.outbytes); }while (u64_stats_fetch_retry (&s->syncp, start)); } }
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 10/12 ipvs: use kthreads for stats estimation
In reply to Pablo Neira AyusoFrom: Julian Anastasov <[email protected]>
Estimating all entries in single list in timer context by single CPU causes large latency with multiple IPVS rules as reported in [1], [2], [3].
Spread the estimator structures in multiple chains and use kthread(s) for the estimation. The chains are processed in multiple (50) timer ticks to ensure the 2-second interval between estimations with some accuracy. Every chain is processed under RCU lock.
Every kthread works over its own data structure and all such contexts are attached to array. The contexts can be preserved while the kthread tasks are stopped or restarted. When estimators are removed, unused kthread contexts are released and the slots in array are left empty.
First kthread determines parameters to use, eg. maximum number of estimators to process per kthread based on chain's length (chain_max), allowing sub-100us cond_resched rate and estimation taking up to 1/8 of the CPU capacity to avoid any problems if chain_max is not correctly calculated.
chain_max is calculated taking into account factors such as CPU speed and memory/cache speed where the cache_factor (4) is selected from real tests with current generation of CPU/NUMA configurations to correct the difference in CPU usage between cached (during calc phase) and non-cached (working) state of the estimated per-cpu data.
First kthread also plays the role of distributor of added estimators to all kthreads, keeping low the time to add estimators. The optimization is based on the fact that newly added estimator should be estimated after 2 seconds, so we have the time to offload the adding to chain from controlling process to kthread 0.
The allocated kthread context may grow from 1 to 50 allocated structures for timer ticks which saves memory for setups with small number of estimators.
We also add delayed work est_reload_work that will make sure the kthread tasks are properly started/stopped.
ip_vs_start_estimator() is changed to report errors which allows to safely store the estimators in allocated structures.
Many thanks to Jiri Wiesner for his valuable comments and for spending a lot of time reviewing and testing the changes on different platforms with 48-256 CPUs and 1-8 NUMA nodes under different cpufreq governors.
[1] Report from Yunhong Jiang:
https://lore.kernel.org/netdev/[email protected]/
[2]
https://marc.info/?l=linux-virtual-server&m=159679809118027&w=2
[3] Report from Dust:
https://archive.linuxvirtualserver.org/html/lvs-devel/2020-12/msg00000.html
--- include/net/ip_vs.h | 88 +++- net/netfilter/ipvs/ip_vs_ctl.c | 126 ++++- net/netfilter/ipvs/ip_vs_est.c | 876 ++++++++++++++++++++++++++++++--- 3 files changed, 990 insertions(+), 100 deletions(-)
include/net/ip_vs.h +83 -5
@@ -42,6 +42,8 @@ static inline struct netns_ipvs *net_ipvs(struct net* net)/* Connections' size value needed by ip_vs_ctl.c */ extern int ip_vs_conn_tab_size; +extern struct mutex __ip_vs_mutex; +struct ip_vs_iphdr {int hdr_flags;/* ipvs flags */ __u32 off;/* Where IP or IPv4 header starts */
@@ -365,7 +367,7 @@ struct ip_vs_cpu_stats {/* IPVS statistics objects */ struct ip_vs_estimator { -struct list_head list; +struct hlist_node list;u64 last_inbytes;u64 last_outbytes;
@@ -378,6 +380,10 @@ struct ip_vs_estimator {u64 outpps;u64 inbps;u64 outbps; + + s32 ktid:16, /* kthread ID, -1=temp list */ + ktrow:8, /* row/tick ID for kthread */ + ktcid:8; /* chain ID for kthread tick */ };/*
@@ -415,6 +421,66 @@ struct ip_vs_stats *ip_vs_stats_alloc(void);void ip_vs_stats_release (struct ip_vs_stats *stats);void ip_vs_stats_free (struct ip_vs_stats *stats); +/* Process estimators in multiple timer ticks (20/50/100, see ktrow) */ +#define IPVS_EST_NTICKS 50 +/* Estimation uses a 2-second period containing ticks (in jiffies) */ +#define IPVS_EST_TICK ((2 * HZ) / IPVS_EST_NTICKS) + +/* Limit of CPU load per kthread (8 for 12.5%), ratio of CPU capacity (1/C). +* Value of 4 and above ensures kthreads will take work without exceeding +* the CPU capacity under different circumstances. +*/ +#define IPVS_EST_LOAD_DIVISOR 8 + +/* Kthreads should not have work that exceeds the CPU load above 50% */ +#define IPVS_EST_CPU_KTHREADS (IPVS_EST_LOAD_DIVISOR / 2) + +/* Desired number of chains per timer tick (chain load factor in 100us units), +* 48=4.8ms of 40ms tick (12% CPU usage): +* 2 sec * 1000 ms in sec * 10 (100us in ms) / 8 (12.5%) / 50 +*/ +#define IPVS_EST_CHAIN_FACTOR \ + ALIGN_DOWN(2 * 1000 * 10 / IPVS_EST_LOAD_DIVISOR / IPVS_EST_NTICKS, 8) + +/* Compiled number of chains per tick +* The defines should match cond_resched_rcu +*/ +#if defined(CONFIG_DEBUG_ATOMIC_SLEEP ) || !defined(CONFIG_PREEMPT_RCU ) +#define IPVS_EST_TICK_CHAINS IPVS_EST_CHAIN_FACTOR +#else +#define IPVS_EST_TICK_CHAINS 1 +#endif + +#if IPVS_EST_NTICKS >127 +#error Too many timer ticks for ktrow +#endif + +/* Multiple chains processed in same tick */ +struct ip_vs_est_tick_data { +struct hlist_head chains[IPVS_EST_TICK_CHAINS ]; + DECLARE_BITMAP(present, IPVS_EST_TICK_CHAINS); + DECLARE_BITMAP(full, IPVS_EST_TICK_CHAINS); +int chain_len[IPVS_EST_TICK_CHAINS ]; +}; + +/* Context for estimation kthread */ +struct ip_vs_est_kt_data { +struct netns_ipvs *ipvs; +struct task_struct *task;/* task if running */ +struct ip_vs_est_tick_data __rcu *ticks[IPVS_EST_NTICKS ]; + DECLARE_BITMAP(avail, IPVS_EST_NTICKS); /* tick has space for ests */ +unsigned long est_timer;/* estimation timer (jiffies) */ +struct ip_vs_stats *calc_stats;/* Used for calculation */ +int tick_len[IPVS_EST_NTICKS ];/* est count */ +int id;/* ktid per netns */ +int chain_max;/* max ests per tick chain */ +int tick_max;/* max ests per tick */ +int est_count;/* attached ests to kthread */ +int est_max_count;/* max ests per kthread */ +int add_row;/* row for new ests */ +int est_row;/* estimated row */ +}; +struct dst_entry ;struct iphdr ;struct ip_vs_conn ;
@@ -953,9 +1019,17 @@ struct netns_ipvs {struct ctl_table_header *lblcr_ctl_header;struct ctl_table *lblcr_ctl_table;/* ip_vs_est */ -struct list_head est_list;/* estimator list */ -spinlock_t est_lock; -struct timer_list est_timer;/* Estimation timer */ +struct delayed_work est_reload_work;/* Reload kthread tasks */ +struct mutex est_mutex;/* protect kthread tasks */ +struct hlist_head est_temp_list;/* Ests during calc phase */ +struct ip_vs_est_kt_data **est_kt_arr;/* Array of kthread data ptrs */ +unsigned long est_max_threads;/* Hard limit of kthreads */ +int est_calc_phase;/* Calculation phase */ +int est_chain_max;/* Calculated chain_max */ +int est_kt_count;/* Allocated ptrs */ +int est_add_ktid;/* ktid where to add ests */ +atomic_t est_genid;/* kthreads reload genid */ +atomic_t est_genid_done;/* applied genid */ /* ip_vs_sync */ spinlock_t sync_lock;struct ipvs_master_sync_state *ms;
@@ -1486,10 +1560,14 @@ int stop_sync_thread(struct netns_ipvs *ipvs, int state);void ip_vs_sync_conn (struct netns_ipvs *ipvs,struct ip_vs_conn *cp,int pkts);/* IPVS rate estimator prototypes (from ip_vs_est.c) */ -void ip_vs_start_estimator (struct netns_ipvs *ipvs,struct ip_vs_stats *stats); +int ip_vs_start_estimator (struct netns_ipvs *ipvs,struct ip_vs_stats *stats);void ip_vs_stop_estimator (struct netns_ipvs *ipvs,struct ip_vs_stats *stats);void ip_vs_zero_estimator (struct ip_vs_stats *stats);void ip_vs_read_estimator (struct ip_vs_kstats *dst,struct ip_vs_stats *stats); +void ip_vs_est_reload_start (struct netns_ipvs *ipvs); +int ip_vs_est_kthread_start (struct netns_ipvs *ipvs, +struct ip_vs_est_kt_data *kd); +void ip_vs_est_kthread_stop (struct ip_vs_est_kt_data *kd);/* Various IPVS packet transmitters (from ip_vs_xmit.c) */ int ip_vs_null_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
net/netfilter/ipvs/ip_vs_ctl.c +99 -27
269 lines not on this page · raw message
net/netfilter/ipvs/ip_vs_est.c +808 -68
929 lines not on this page · raw message
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 11/12 ipvs: add est_cpulist and est_nice sysctl vars
In reply to Pablo Neira AyusoFrom: Julian Anastasov <[email protected]>
Allow the kthreads for stats to be configured for specific cpulist (isolation) and niceness (scheduling priority).
--- Documentation/networking/ipvs-sysctl.rst | 20 ++++ include/net/ip_vs.h | 58 +++++++++ net/netfilter/ipvs/ip_vs_ctl.c | 143 ++++++++++++++++++++++- net/netfilter/ipvs/ip_vs_est.c | 12 +- 4 files changed, 229 insertions(+), 4 deletions(-)
Documentation/networking/ipvs-sysctl.rst +20 -0
@@ -129,6 +129,26 @@ drop_packet - INTEGER threshold. When the mode 3 is set, the always mode drop rate is controlled by the /proc/sys/net/ipv4/vs/am_droprate. +est_cpulist - CPULIST + Allowed CPUs for estimation kthreads + + Syntax: standard cpulist format + empty list - stop kthread tasks and estimation + default - the system's housekeeping CPUs for kthreads + + Example: + "all": all possible CPUs + "0-N": all possible CPUs, N denotes last CPU number + "0,1-N:1/2": first and all CPUs with odd number + "": empty list + +est_nice - INTEGER + default 0 + Valid range: -20 (more favorable) .. 19 (less favorable) + + Niceness value to use for the estimation kthreads (scheduling + priority) + expire_nodest_conn - BOOLEAN - 0 - disabled (default) - not 0 - enabled
include/net/ip_vs.h +58 -0
94 lines not on this page · raw message
net/netfilter/ipvs/ip_vs_ctl.c +142 -1
179 lines not on this page · raw message
net/netfilter/ipvs/ip_vs_est.c +9 -3
@@ -57,6 +57,9 @@ - kthread contexts are created and attached to array - the kthread tasks are started when first service is added, before that the total stats are not estimated + - when configuration (cpulist/nice) is changed, the tasks are restarted + by work (est_reload_work) + - kthread tasks are stopped while the cpulist is empty - the kthread context holds lists with estimators (chains) which are processed every 2 seconds - as estimators can be added dynamically and in bursts, we try to spread
@@ -229,6 +232,7 @@ void ip_vs_est_reload_start(struct netns_ipvs *ipvs)/* Ignore reloads before first service is added */ if (!ipvs->enable)return ; +ip_vs_est_stopped_recalc (ipvs);/* Bump the kthread configuration genid */ atomic_inc (&ipvs->est_genid);queue_delayed_work (system_long_wq, &ipvs->est_reload_work,0 );
@@ -259,6 +263,9 @@ int ip_vs_est_kthread_start(struct netns_ipvs *ipvs, goto out; } +set_user_nice (kd->task,sysctl_est_nice (ipvs)); +set_cpus_allowed_ptr (kd->task,sysctl_est_cpulist (ipvs)); +pr_info ("starting estimator thread %d...\n" , kd->id);wake_up_process (kd->task);
@@ -334,7 +341,7 @@ static int ip_vs_est_add_kthread(struct netns_ipvs *ipvs) }/* Start kthread tasks only when services are present */ -if (ipvs->enable) { +if (ipvs->enable && !ip_vs_est_stopped (ipvs)) { ret =ip_vs_est_kthread_start (ipvs, kd);if (ret <0 ) goto out;
@@ -478,8 +485,7 @@ int ip_vs_start_estimator(struct netns_ipvs *ipvs, struct ip_vs_stats *stats)int ret;if (!ipvs->est_max_threads && ipvs->enable) - ipvs->est_max_threads =IPVS_EST_CPU_KTHREADS * -num_possible_cpus (); + ipvs->est_max_threads =ip_vs_est_max_threads (ipvs); est->ktid =-1 ; est->ktrow =IPVS_EST_NTICKS -1 ;/* Initial delay */
Signature · 1 line
2.30.2
Pablo Neira Ayuso
maintainerPATCH 12/12 ipvs: run_estimation should control the kthread tasks
In reply to Pablo Neira AyusoFrom: Julian Anastasov <[email protected]>
Change the run_estimation flag to start/stop the kthread tasks.
--- Documentation/networking/ipvs-sysctl.rst | 4 ++-- include/net/ip_vs.h | 6 +++-- net/netfilter/ipvs/ip_vs_ctl.c | 29 +++++++++++++++++++++++- net/netfilter/ipvs/ip_vs_est.c | 2 +- 4 files changed, 35 insertions(+), 6 deletions(-)
Documentation/networking/ipvs-sysctl.rst +2 -2
@@ -324,8 +324,8 @@ run_estimation - BOOLEAN 0 - disabled not 0 - enabled (default) - If disabled, the estimation will be stop, and you can't see - any update on speed estimation data. + If disabled, the estimation will be suspended and kthread tasks + stopped. You can always re-enable estimation by setting this value to 1. But be careful, the first estimation after re-enable is not
include/net/ip_vs.h +4 -2
@@ -1605,8 +1605,10 @@ void ip_vs_est_kthread_stop(struct ip_vs_est_kt_data *kd);static inline void ip_vs_est_stopped_recalc (struct netns_ipvs *ipvs) {#ifdef CONFIG_SYSCTL - ipvs->est_stopped = ipvs->est_cpulist_valid && -cpumask_empty (sysctl_est_cpulist (ipvs)); +/* Stop tasks while cpulist is empty or if disabled with flag */ + ipvs->est_stopped = !sysctl_run_estimation (ipvs) || + (ipvs->est_cpulist_valid && +cpumask_empty (sysctl_est_cpulist (ipvs)));#endif }
net/netfilter/ipvs/ip_vs_ctl.c +28 -1
@@ -2056,6 +2056,32 @@ static int ipvs_proc_est_nice(struct ctl_table *table, int write,return ret; } +static int ipvs_proc_run_estimation (struct ctl_table *table,int write, +void *buffer,size_t *lenp,loff_t *ppos) +{ +struct netns_ipvs *ipvs = table->extra2; +int *valp = table->data; +int val = *valp; +int ret; + +struct ctl_table tmp_table = { + .data = &val, + .maxlen =sizeof (int ), + .mode = table->mode, + }; + + ret =proc_dointvec (&tmp_table, write, buffer, lenp, ppos); +if (write && ret >=0 ) { +mutex_lock (&ipvs->est_mutex); +if (*valp != val) { + *valp = val; +ip_vs_est_reload_start (ipvs); + } +mutex_unlock (&ipvs->est_mutex); + } +return ret; +} +/* * IPVS sysctl table (under the /proc/sys/net/ipv4/vs/) * Do not change order or insert new entries without
@@ -2230,7 +2256,7 @@ static struct ctl_table vs_vars[] = { .procname = "run_estimation", .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = ipvs_proc_run_estimation, }, { .procname = "est_cpulist",
@@ -4323,6 +4349,7 @@ static int __net_init ip_vs_control_net_init_sysctl(struct netns_ipvs *ipvs) tbl[idx++].data = &ipvs->sysctl_schedule_icmp; tbl[idx++].data = &ipvs->sysctl_ignore_tunneled; ipvs->sysctl_run_estimation =1 ; + tbl[idx].extra2 = ipvs; tbl[idx++].data = &ipvs->sysctl_run_estimation; ipvs->est_cpulist_valid =0 ;
net/netfilter/ipvs/ip_vs_est.c +1 -1
@@ -212,7 +212,7 @@ static int ip_vs_estimation_kthread(void *data) kd->est_timer = now; } -if (sysctl_run_estimation (ipvs) && kd->tick_len[row]) +if (kd->tick_len[row])ip_vs_tick_estimation (kd, row); row++;
Signature · 1 line
2.30.2