ipv6: misc changes in output path
Subsystems (5)
cover ipv6: misc changes in output path
Small optimizations mostly in ip6_xmit() path.
TX performance increases by about 3 %.
Patches 5-7: add dst4_mtu() and dst6_mtu() to save space.
Last patch colocates inet6_cork in inet_cork_full.
This series reduces kernel size by 494 bytes on x86_64:
scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 4/2 grow/shrink: 9/23 up/down: 665/-1159 (-494)
Function old new deltaip6_finish_output_gso_slowpath_drop - 197 +197
ip6_xmit 1452 1595 +143 do_ipv6_getsockopt 2855 2950 +95 kzalloc_noprof - 55 +55 ip4ip6_err 918 955 +37 __icmp_send 1499 1532 +33 do_ip_getsockopt 2573 2605 +32 __ip6_append_data 4109 4137 +28 __pfx_kzalloc_noprof - 16 +16 __pfx_ip6_finish_output_gso_slowpath_drop - 16 +16 ipmr_prepare_xmit 1232 1238 +6 ip6_forward 1905 1909 +4 ip6_cork_release 108 111 +3 ipv6_push_nfrag_opts 489 486 -3 ipv6_push_frag_opts 90 87 -3 ip6_finish_output2 1446 1437 -9 ip6_tnl_xmit 2639 2627 -12 ip6_default_advmss 176 160 -16 __ip6_rt_update_pmtu 1087 1071 -16 tcp_v6_syn_recv_sock 1715 1696 -19 tcp_v4_syn_recv_sock 1107 1088 -19 __ip_make_skb 1339 1320 -19 ip_setup_cork 406 385 -21 ip6_setup_cork 732 710 -22 rawv6_push_pending_frames 581 556 -25 ip6_push_pending_frames 184 157 -27 udpv6_splice_eof 203 170 -33 ip6_flush_pending_frames 220 183 -37 ip6_append_data 349 312 -37 udp_v6_push_pending_frames 155 115 -40 sit_tunnel_xmit 1957 1914 -43 __pfx_dst_mtu 64 - -64 tcp_v4_mtu_reduced 289 220 -69 tcp_v6_mtu_reduced 209 139 -70 ip6_make_skb 574 484 -90 ip6_finish_output 827 697 -130 dst_mtu 160 - -160 fib6_nh_mtu_change 511 336 -175
Total: Before=22584400, After=22583906, chg -0.00%
Eric Dumazet (8):
ipv6: pass proto by value to ipv6_push_nfrag_opts() and
ipv6_push_frag_opts()
ipv6: add some unlikely()/likely() clauses in ip6_output.c
ipv6: use __skb_push() in ip6_xmit()
ipv6: use SKB_DROP_REASON_PKT_TOO_BIG in ip6_xmit()
inet: add dst4_mtu() and dst6_mtu() helpers
ipv6: use dst6_mtu() instead of dst_mtu()
ipv4: use dst4_mtu() instead of dst_mtu()
ipv6: colocate inet6_cork in inet_cork_full
include/linux/ipv6.h | 8 --- include/net/dst.h | 6 ++ include/net/inet_sock.h | 10 +++ include/net/ip6_route.h | 6 ++ include/net/ipv6.h | 16 ++--- net/ipv4/icmp.c | 2 +- net/ipv4/ip_output.c | 4 +- net/ipv4/ip_sockglue.c | 2 +- net/ipv4/ipmr.c | 2 +- net/ipv4/netfilter/nf_reject_ipv4.c | 2 +- net/ipv4/tcp_ipv4.c | 13 ++-- net/ipv6/exthdrs.c | 55 +++++++------- net/ipv6/ip6_gre.c | 2 +- net/ipv6/ip6_output.c | 108 ++++++++++++++-------------- net/ipv6/ip6_tunnel.c | 6 +- net/ipv6/ipv6_sockglue.c | 4 +- net/ipv6/raw.c | 2 +- net/ipv6/route.c | 10 +-- net/ipv6/sit.c | 2 +- net/ipv6/tcp_ipv6.c | 9 +-- 20 files changed, 144 insertions(+), 125 deletions(-)
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
PATCH 1/8 ipv6: pass proto by value to ipv6_push_nfrag_opts() and ipv6_push_frag_opts()
With CONFIG_STACKPROTECTOR_STRONG=y, it is better to avoid passing a pointer to an automatic variable.
Change these exported functions to return 'u8 proto' instead of void.
- ipv6_push_nfrag_opts()
- ipv6_push_frag_opts()
For instance, replace
ipv6_push_frag_opts(skb, opt, &proto); with: proto = ipv6_push_frag_opts(skb, opt, proto);
Note that even after this change, ip6_xmit() has to use a stack canary because of @first_hop variable.
Signed-off-by Eric Dumazet <[email protected]>
--- include/net/ipv6.h | 10 ++++---- net/ipv6/exthdrs.c | 55 +++++++++++++++++++++++-------------------- net/ipv6/ip6_output.c | 19 ++++++++------- net/ipv6/ip6_tunnel.c | 2 +- 4 files changed, 46 insertions(+), 40 deletions(-)
include/net/ipv6.h +5 -5
@@ -1151,11 +1151,11 @@ int ip6_local_out(struct net *net, struct sock *sk, struct sk_buff *skb);* Extension header (options) processing */ -void ipv6_push_nfrag_opts (struct sk_buff *skb,struct ipv6_txoptions *opt, -u8 *proto,struct in6_addr **daddr_p, -struct in6_addr *saddr); -void ipv6_push_frag_opts (struct sk_buff *skb,struct ipv6_txoptions *opt, -u8 *proto); +u8 ipv6_push_nfrag_opts (struct sk_buff *skb,struct ipv6_txoptions *opt, +u8 proto,struct in6_addr **daddr_p, +struct in6_addr *saddr); +u8 ipv6_push_frag_opts (struct sk_buff *skb,struct ipv6_txoptions *opt, +u8 proto);int ipv6_skip_exthdr (const struct sk_buff *,int start,u8 *nexthdrp,__be16 *frag_offp);
net/ipv6/exthdrs.c +29 -26
109 lines not on this page · raw message
net/ipv6/ip6_output.c +11 -8
@@ -305,11 +305,12 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6, seg_len += opt->opt_nflen + opt->opt_flen;if (opt->opt_flen) -ipv6_push_frag_opts (skb, opt, &proto); + proto =ipv6_push_frag_opts (skb, opt, proto);if (opt->opt_nflen) -ipv6_push_nfrag_opts (skb, opt, &proto, &first_hop, - &fl6->saddr); + proto =ipv6_push_nfrag_opts (skb, opt, proto, + &first_hop, + &fl6->saddr); }if (unlikely (seg_len >IPV6_MAXPLEN )) {
@@ -1940,11 +1941,13 @@ struct sk_buff *__ip6_make_skb(struct sock *sk,__skb_pull (skb,skb_network_header_len (skb)); final_dst = &fl6->daddr; -if (opt && opt->opt_flen) -ipv6_push_frag_opts (skb, opt, &proto); -if (opt && opt->opt_nflen) -ipv6_push_nfrag_opts (skb, opt, &proto, &final_dst, &fl6->saddr); - +if (opt) { +if (opt->opt_flen) + proto =ipv6_push_frag_opts (skb, opt, proto); +if (opt->opt_nflen) + proto =ipv6_push_nfrag_opts (skb, opt, proto, + &final_dst, &fl6->saddr); + }skb_push (skb,sizeof (struct ipv6hdr ));skb_reset_network_header (skb); hdr =ipv6_hdr (skb);
net/ipv6/ip6_tunnel.c +1 -1
@@ -1265,7 +1265,7 @@ int ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev, __u8 dsfield,if (encap_limit >=0 ) {init_tel_txopt (&opt, encap_limit); -ipv6_push_frag_opts (skb, &opt.ops, &proto); + proto =ipv6_push_frag_opts (skb, &opt.ops, proto); }skb_push (skb,sizeof (struct ipv6hdr ));
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 1/8] ipv6: pass proto by value to ipv6_push_nfrag_opts() and ipv6_push_frag_opts()
Quoted 11 lines · Eric Dumazet
With
CONFIG_STACKPROTECTOR_STRONG=y, it is better to avoid passing a pointer to an automatic variable.Change these exported functions to return 'u8 proto' instead of void.
-
ipv6_push_nfrag_opts()
-ipv6_push_frag_opts()For instance, replace
Quoted 8 lines
ipv6_push_frag_opts(skb, opt, &proto); with: proto = ipv6_push_frag_opts(skb, opt, proto);Note that even after this change,
ip6_xmit()has to use a stack canary because of @first_hop variable.Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 2/8 ipv6: add some unlikely()/likely() clauses in ip6_output.c
In reply to Eric Dumazet1) daddr is unlikely a multicast in ip6_finish_output2().
2) ip6_finish_output_gso_slowpath_drop() should not be called often.
3) ip6_fragment() should not be called often.
4) opt is unlikely to be set.
5) ip6_xmit() and ip6_forward() mostly sends not too big packets.
6) Most __ip6_make_skb() calls are for UDP packets, not ICMPV6 ones.
Signed-off-by Eric Dumazet <[email protected]>
--- net/ipv6/ip6_output.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
net/ipv6/ip6_output.c +12 -12
@@ -80,7 +80,7 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * hdr =ipv6_hdr (skb); daddr = &hdr->daddr; -if (ipv6_addr_is_multicast (daddr)) { +if (unlikely (ipv6_addr_is_multicast (daddr))) { if (!(dev->flags & IFF_LOOPBACK) && sk_mc_loop(sk) && ((mroute6_is_socket(net, skb) && !(IP6CB(skb)->flags & IP6SKB_FORWARDED)) ||
@@ -179,8 +179,8 @@ ip6_finish_output_gso_slowpath_drop(struct net *net, struct sock *sk,static int ip6_finish_output_gso (struct net *net,struct sock *sk,struct sk_buff *skb,unsigned int mtu) { -if (!(IP6CB (skb)->flags &IP6SKB_FAKEJUMBO ) && - !skb_gso_validate_network_len (skb, mtu)) +if (unlikely (!(IP6CB (skb)->flags &IP6SKB_FAKEJUMBO ) && + !skb_gso_validate_network_len (skb, mtu)))return ip6_finish_output_gso_slowpath_drop (net, sk, skb, mtu);return ip6_finish_output2 (net, sk, skb);
@@ -202,8 +202,8 @@ static int __ip6_finish_output(struct net *net, struct sock *sk, struct sk_buffif (skb_is_gso (skb))return ip6_finish_output_gso (net, sk, skb, mtu); -if (skb->len > mtu || - (IP6CB (skb)->frag_max_size && skb->len >IP6CB (skb)->frag_max_size)) +if (unlikely (skb->len > mtu || + (IP6CB (skb)->frag_max_size && skb->len >IP6CB (skb)->frag_max_size)))return ip6_fragment (net, sk, skb, ip6_finish_output2);return ip6_finish_output2 (net, sk, skb);
@@ -301,7 +301,7 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6, } } -if (opt) { +if (unlikely (opt)) { seg_len += opt->opt_nflen + opt->opt_flen;if (opt->opt_flen)
@@ -354,7 +354,7 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6, skb->mark = mark; mtu =dst_mtu (dst); -if ((skb->len <= mtu) || skb->ignore_df ||skb_is_gso (skb)) { +if (likely ((skb->len <= mtu) || skb->ignore_df ||skb_is_gso (skb))) {IP6_INC_STATS (net, idev,IPSTATS_MIB_OUTREQUESTS );/* if egress device is enslaved to an L3 master device pass the
@@ -654,7 +654,7 @@ int ip6_forward(struct sk_buff *skb)if (mtu <IPV6_MIN_MTU ) mtu =IPV6_MIN_MTU ; -if (ip6_pkt_too_big (skb, mtu)) { +if (unlikely (ip6_pkt_too_big (skb, mtu))) {/* Again, force OUTPUT device used as source address */ skb->dev = dev;icmpv6_send (skb,ICMPV6_PKT_TOOBIG ,0 , mtu);
@@ -1368,7 +1368,7 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork,/* * setup for corking */ -if (opt) { +if (unlikely (opt)) {if (WARN_ON (v6_cork->opt))return -EINVAL ;
@@ -1885,7 +1885,7 @@ static void ip6_cork_steal_dst(struct sk_buff *skb, struct inet_cork_full *cork)static void ip6_cork_release (struct inet_cork_full *cork,struct inet6_cork *v6_cork) { -if (v6_cork->opt) { +if (unlikely (v6_cork->opt)) {struct ipv6_txoptions *opt = v6_cork->opt;kfree (opt->dst0opt);
@@ -1941,7 +1941,7 @@ struct sk_buff *__ip6_make_skb(struct sock *sk,__skb_pull (skb,skb_network_header_len (skb)); final_dst = &fl6->daddr; -if (opt) { +if (unlikely (opt)) {if (opt->opt_flen) proto =ipv6_push_frag_opts (skb, opt, proto);if (opt->opt_nflen)
@@ -1969,7 +1969,7 @@ struct sk_buff *__ip6_make_skb(struct sock *sk,ip6_cork_steal_dst (skb, cork);IP6_INC_STATS (net, rt->rt6i_idev,IPSTATS_MIB_OUTREQUESTS ); -if (proto ==IPPROTO_ICMPV6 ) { +if (unlikely (proto ==IPPROTO_ICMPV6 )) {struct inet6_dev *idev =ip6_dst_idev (skb_dst (skb));u8 icmp6_type;
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 2/8] ipv6: add some unlikely()/likely() clauses in ip6_output.c
Quoted 7 lines · Eric Dumazet
1) daddr is unlikely a multicast in
ip6_finish_output2().2)
ip6_finish_output_gso_slowpath_drop()should not be called often.3)
ip6_fragment()should not be called often.
Quoted 8 lines
4) opt is unlikely to be set.
5)
ip6_xmit()andip6_forward()mostly sends not too big packets.6) Most
__ip6_make_skb()calls are for UDP packets, not ICMPV6 ones.Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 3/8 ipv6: use __skb_push() in ip6_xmit()
In reply to Eric Dumazetip6_xmit() makes sure there is enough headroom in the skb, it can uses __skb_push() instead of the out-of-line skb_push().
Signed-off-by Eric Dumazet <[email protected]>
--- net/ipv6/ip6_output.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/ipv6/ip6_output.c +2 -2
@@ -314,7 +314,7 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6, }if (unlikely (seg_len >IPV6_MAXPLEN )) { - hop_jumbo =skb_push (skb, hoplen); + hop_jumbo =__skb_push (skb, hoplen); hop_jumbo->nexthdr = proto; hop_jumbo->hdrlen =0 ;
@@ -327,7 +327,7 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6,IP6CB (skb)->flags |=IP6SKB_FAKEJUMBO ; } -skb_push (skb,sizeof (struct ipv6hdr )); +__skb_push (skb,sizeof (struct ipv6hdr ));skb_reset_network_header (skb); hdr =ipv6_hdr (skb);
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 3/8] ipv6: use __skb_push() in ip6_xmit()
Quoted 5 lines · Eric Dumazet
ip6_xmit()makes sure there is enough headroom in the skb, it can uses__skb_push()instead of the out-of-lineskb_push().Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 4/8 ipv6: use SKB_DROP_REASON_PKT_TOO_BIG in ip6_xmit()
In reply to Eric DumazetWhen a too big packet is dropped, use SKB_DROP_REASON_PKT_TOO_BIG.
Signed-off-by Eric Dumazet <[email protected]>
--- net/ipv6/ip6_output.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv6/ip6_output.c +1 -1
@@ -383,7 +383,7 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6,ipv6_local_error ((struct sock *)sk,EMSGSIZE , fl6, mtu);IP6_INC_STATS (net, idev,IPSTATS_MIB_FRAGFAILS ); -kfree_skb (skb); +kfree_skb_reason (skb,SKB_DROP_REASON_PKT_TOO_BIG ); unlock:rcu_read_unlock ();return ret;
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 4/8] ipv6: use SKB_DROP_REASON_PKT_TOO_BIG in ip6_xmit()
Quoted 4 lines · Eric Dumazet
When a too big packet is dropped, use
SKB_DROP_REASON_PKT_TOO_BIG.Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 5/8 inet: add dst4_mtu() and dst6_mtu() helpers
In reply to Eric DumazetWith CONFIG_MITIGATION_RETPOLINE=y dst_mtu() is a bit fat, because it is generic.
Indeed, clang does not always inline it.
Add dst4_mtu() and dst6_mtu() helpers for callers that expect either ipv4_mtu() or ip6_mtu() to be called.
These helpers are always inlined.
Signed-off-by Eric Dumazet <[email protected]>
--- include/net/dst.h | 6 ++++++ include/net/ip6_route.h | 6 ++++++ 2 files changed, 12 insertions(+)
include/net/dst.h +6 -0
@@ -219,6 +219,12 @@ static inline u32 dst_mtu(const struct dst_entry *dst)return INDIRECT_CALL_INET (dst->ops->mtu, ip6_mtu, ipv4_mtu, dst); } +/* Variant of dst_mtu() for IPv4 users. */ +static inline u32 dst4_mtu (const struct dst_entry *dst) +{ +return INDIRECT_CALL_1 (dst->ops->mtu, ipv4_mtu, dst); +} +/* RTT metrics are stored in milliseconds for user ABI, but used as jiffies */ static inline unsigned long dst_metric_rtt (const struct dst_entry *dst,int metric) {
include/net/ip6_route.h +6 -0
@@ -266,6 +266,12 @@ static inline bool ipv6_anycast_destination(const struct dst_entry *dst,int ip6_fragment (struct net *net,struct sock *sk,struct sk_buff *skb,int (*output)(struct net *,struct sock *,struct sk_buff *)); +/* Variant of dst_mtu() for IPv6 users */ +static inline u32 dst6_mtu (const struct dst_entry *dst) +{ +return INDIRECT_CALL_1 (dst->ops->mtu, ip6_mtu, dst); +} +static inline unsigned int ip6_skb_dst_mtu (const struct sk_buff *skb) { const struct ipv6_pinfo *np = skb->sk && !dev_recursion_level() ?
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 5/8] inet: add dst4_mtu() and dst6_mtu() helpers
Quoted 4 lines · Eric Dumazet
With
CONFIG_MITIGATION_RETPOLINE=ydst_mtu()is a bit fat, because it is generic.
Quoted 8 lines
Indeed, clang does not always inline it.
Add
dst4_mtu()anddst6_mtu()helpers for callers that expect eitheripv4_mtu()orip6_mtu()to be called.These helpers are always inlined.
Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 6/8 ipv6: use dst6_mtu() instead of dst_mtu()
In reply to Eric DumazetWhen we expect an IPv6 dst, use dst6_mtu() instead of dst_mtu() to save some code space.
Due to current dst6_mtu() implementation, only convert users in IPv6 stack.
Signed-off-by Eric Dumazet <[email protected]>
--- net/ipv6/ip6_gre.c | 2 +- net/ipv6/ip6_output.c | 6 +++--- net/ipv6/ip6_tunnel.c | 4 ++-- net/ipv6/ipv6_sockglue.c | 4 ++-- net/ipv6/route.c | 10 ++++++---- net/ipv6/tcp_ipv6.c | 9 +++++---- 6 files changed, 19 insertions(+), 16 deletions(-)
net/ipv6/ip6_gre.c +1 -1
@@ -1057,7 +1057,7 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb,/* TooBig packet may have updated dst->dev's mtu */ if (!t->parms.collect_md && dst) { mtu =READ_ONCE (dst_dev (dst)->mtu); -if (dst_mtu (dst) > mtu) +if (dst6_mtu (dst) > mtu) dst->ops->update_pmtu (dst,NULL , skb, mtu, false); } err = ip6_tnl_xmit(skb, dev, dsfield, &fl6, encap_limit, &mtu,
net/ipv6/ip6_output.c +3 -3
@@ -353,7 +353,7 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6, skb->priority = priority; skb->mark = mark; - mtu =dst_mtu (dst); + mtu =dst6_mtu (dst);if (likely ((skb->len <= mtu) || skb->ignore_df ||skb_is_gso (skb))) {IP6_INC_STATS (net, idev,IPSTATS_MIB_OUTREQUESTS );
@@ -1403,10 +1403,10 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, v6_cork->dontfrag = ipc6->dontfrag;if (rt->dst.flags &DST_XFRM_TUNNEL ) mtu =READ_ONCE (np->pmtudisc) >=IPV6_PMTUDISC_PROBE ? -READ_ONCE (rt->dst.dev->mtu) :dst_mtu (&rt->dst); +READ_ONCE (rt->dst.dev->mtu) :dst6_mtu (&rt->dst);else mtu =READ_ONCE (np->pmtudisc) >=IPV6_PMTUDISC_PROBE ? -READ_ONCE (rt->dst.dev->mtu) :dst_mtu (xfrm_dst_path (&rt->dst)); +READ_ONCE (rt->dst.dev->mtu) :dst6_mtu (xfrm_dst_path (&rt->dst)); frag_size =READ_ONCE (np->frag_size);if (frag_size && frag_size < mtu)
net/ipv6/ip6_tunnel.c +2 -2
@@ -638,7 +638,7 @@ ip4ip6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,/* change mtu on this route */ if (rel_type ==ICMP_DEST_UNREACH && rel_code ==ICMP_FRAG_NEEDED ) { -if (rel_info >dst_mtu (skb_dst (skb2))) +if (rel_info >dst6_mtu (skb_dst (skb2))) goto out;skb_dst_update_pmtu_no_confirm (skb2, rel_info);
@@ -1187,7 +1187,7 @@ int ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev, __u8 dsfield, t->parms.name); goto tx_err_dst_release; } - mtu =dst_mtu (dst) - eth_hlen - psh_hlen - t->tun_hlen; + mtu =dst6_mtu (dst) - eth_hlen - psh_hlen - t->tun_hlen;if (encap_limit >=0 ) { max_headroom +=8 ; mtu -=8 ;
net/ipv6/ipv6_sockglue.c +2 -2
@@ -1184,7 +1184,7 @@ int do_ipv6_getsockopt(struct sock *sk, int level, int optname,rcu_read_lock (); dst =__sk_dst_get (sk);if (dst) - val =dst_mtu (dst); + val =dst6_mtu (dst);rcu_read_unlock ();if (!val)return -ENOTCONN ;
@@ -1283,7 +1283,7 @@ int do_ipv6_getsockopt(struct sock *sk, int level, int optname,rcu_read_lock (); dst =__sk_dst_get (sk);if (dst) - mtuinfo.ip6m_mtu =dst_mtu (dst); + mtuinfo.ip6m_mtu =dst6_mtu (dst);rcu_read_unlock ();if (!mtuinfo.ip6m_mtu)return -ENOTCONN ;
net/ipv6/route.c +6 -4
@@ -2049,6 +2049,8 @@ static void rt6_update_exception_stamp_rt(struct rt6_info *rt)static bool rt6_mtu_change_route_allowed (struct inet6_dev *idev,struct rt6_info *rt,int mtu) { +u32 dmtu =dst6_mtu (&rt->dst); +/* If the new MTU is lower than the route PMTU, this new MTU will be the * lowest MTU in the path: always allow updating the route PMTU to * reflect PMTU decreases.
@@ -2059,10 +2061,10 @@ static bool rt6_mtu_change_route_allowed(struct inet6_dev *idev,* handle this. */ -if (dst_mtu (&rt->dst) >= mtu) +if (dmtu >= mtu)return true; -if (dst_mtu (&rt->dst) == idev->cnf.mtu6) +if (dmtu == idev->cnf.mtu6)return true;return false;
@@ -2932,7 +2934,7 @@ static void __ip6_rt_update_pmtu(struct dst_entry *dst, const struct sock *sk,if (mtu <IPV6_MIN_MTU )return ; -if (mtu >=dst_mtu (dst)) +if (mtu >=dst6_mtu (dst))return ;if (!rt6_cache_allowed_for_pmtu (rt6)) {
@@ -3248,7 +3250,7 @@ EXPORT_SYMBOL_GPL(ip6_sk_redirect);static unsigned int ip6_default_advmss (const struct dst_entry *dst) { -unsigned int mtu =dst_mtu (dst); +unsigned int mtu =dst6_mtu (dst);struct net *net; mtu -=sizeof (struct ipv6hdr ) +sizeof (struct tcphdr );
net/ipv6/tcp_ipv6.c +5 -4
@@ -351,7 +351,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr_unsized *uaddr,static void tcp_v6_mtu_reduced (struct sock *sk) {struct dst_entry *dst; -u32 mtu; +u32 mtu, dmtu;if ((1 << sk->sk_state) & (TCPF_LISTEN |TCPF_CLOSE ))return ;
@@ -368,8 +368,9 @@ static void tcp_v6_mtu_reduced(struct sock *sk)if (!dst)return ; -if (inet_csk (sk)->icsk_pmtu_cookie >dst_mtu (dst)) { -tcp_sync_mss (sk,dst_mtu (dst)); + dmtu =dst6_mtu (dst); +if (inet_csk (sk)->icsk_pmtu_cookie > dmtu) { +tcp_sync_mss (sk, dmtu);tcp_simple_retransmit (sk); } }
@@ -1467,7 +1468,7 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *tcp_ca_openreq_child (newsk, dst); -tcp_sync_mss (newsk,dst_mtu (dst)); +tcp_sync_mss (newsk,dst6_mtu (dst)); newtp->advmss =tcp_mss_clamp (tcp_sk (sk),dst_metric_advmss (dst));tcp_initialize_rcv_mss (newsk);
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 6/8] ipv6: use dst6_mtu() instead of dst_mtu()
Quoted 8 lines · Eric Dumazet
When we expect an IPv6 dst, use
dst6_mtu()instead ofdst_mtu()to save some code space.Due to current
dst6_mtu()implementation, only convert users in IPv6 stack.Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 7/8 ipv4: use dst4_mtu() instead of dst_mtu()
In reply to Eric DumazetWhen we expect an IPv4 dst, use dst4_mtu() instead of dst_mtu() to save some code space.
Signed-off-by Eric Dumazet <[email protected]>
--- net/ipv4/icmp.c | 2 +- net/ipv4/ip_output.c | 4 ++-- net/ipv4/ip_sockglue.c | 2 +- net/ipv4/ipmr.c | 2 +- net/ipv4/netfilter/nf_reject_ipv4.c | 2 +- net/ipv4/tcp_ipv4.c | 13 ++++++------- net/ipv6/sit.c | 2 +- 7 files changed, 13 insertions(+), 14 deletions(-)
net/ipv4/icmp.c +1 -1
@@ -945,7 +945,7 @@ void __icmp_send(struct sk_buff *skb_in, int type, int code, __be32 info,/* RFC says return as much as we can without exceeding 576 bytes. */ - room =dst_mtu (&rt->dst); + room =dst4_mtu (&rt->dst);if (room >576 ) room =576 ; room -=sizeof (struct iphdr ) + icmp_param->replyopts.opt.optlen;
net/ipv4/ip_output.c +2 -2
@@ -1300,7 +1300,7 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork,return -EFAULT ; cork->fragsize =ip_sk_use_pmtu (sk) ? -dst_mtu (&rt->dst) :READ_ONCE (rt->dst.dev->mtu); +dst4_mtu (&rt->dst) :READ_ONCE (rt->dst.dev->mtu);if (!inetdev_valid_mtu (cork->fragsize))return -ENETUNREACH ;
@@ -1439,7 +1439,7 @@ struct sk_buff *__ip_make_skb(struct sock *sk, pmtudisc =READ_ONCE (inet->pmtudisc);if (pmtudisc ==IP_PMTUDISC_DO || pmtudisc ==IP_PMTUDISC_PROBE || - (skb->len <=dst_mtu (&rt->dst) && + (skb->len <=dst4_mtu (&rt->dst) &&ip_dont_fragment (sk, &rt->dst))) df =htons (IP_DF );
net/ipv4/ip_sockglue.c +1 -1
@@ -1634,7 +1634,7 @@ int do_ip_getsockopt(struct sock *sk, int level, int optname, val =0 ; dst =sk_dst_get (sk);if (dst) { - val =dst_mtu (dst); + val =dst4_mtu (dst);dst_release (dst); }if (!val)
net/ipv4/ipmr.c +1 -1
@@ -1895,7 +1895,7 @@ static int ipmr_prepare_xmit(struct net *net, struct mr_table *mrt,return -1 ; } -if (skb->len+encap >dst_mtu (&rt->dst) && (ntohs (iph->frag_off) &IP_DF )) { +if (skb->len+encap >dst4_mtu (&rt->dst) && (ntohs (iph->frag_off) &IP_DF )) {/* Do not fragment multicasts. Alas, IPv4 does not * allow to send ICMP, so that packets will disappear * to blackhole.
net/ipv4/netfilter/nf_reject_ipv4.c +1 -1
@@ -303,7 +303,7 @@ void nf_send_reset(struct net *net, struct sock *sk, struct sk_buff *oldskb, goto free_nskb;/* "Never happens" */ -if (nskb->len >dst_mtu (skb_dst (nskb))) +if (nskb->len >dst4_mtu (skb_dst (nskb))) goto free_nskb;nf_ct_attach (nskb, oldskb);
net/ipv4/tcp_ipv4.c +6 -7
@@ -374,7 +374,7 @@ void tcp_v4_mtu_reduced(struct sock *sk) {struct inet_sock *inet =inet_sk (sk);struct dst_entry *dst; -u32 mtu; +u32 mtu, dmtu;if ((1 << sk->sk_state) & (TCPF_LISTEN |TCPF_CLOSE ))return ;
@@ -386,15 +386,14 @@ void tcp_v4_mtu_reduced(struct sock *sk)/* Something is about to be wrong... Remember soft error * for the case, if this connection will not able to recover. */ -if (mtu <dst_mtu (dst) &&ip_dont_fragment (sk, dst)) + dmtu =dst4_mtu (dst); +if (mtu < dmtu &&ip_dont_fragment (sk, dst))WRITE_ONCE (sk->sk_err_soft,EMSGSIZE ); - mtu =dst_mtu (dst); -if (inet->pmtudisc !=IP_PMTUDISC_DONT &&ip_sk_accept_pmtu (sk) && -inet_csk (sk)->icsk_pmtu_cookie > mtu) { -tcp_sync_mss (sk, mtu); +inet_csk (sk)->icsk_pmtu_cookie > dmtu) { +tcp_sync_mss (sk, dmtu);/* Resend the TCP packet because it's * clear that the old packet has been
@@ -1760,7 +1759,7 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,tcp_ca_openreq_child (newsk, dst); -tcp_sync_mss (newsk,dst_mtu (dst)); +tcp_sync_mss (newsk,dst4_mtu (dst)); newtp->advmss =tcp_mss_clamp (tcp_sk (sk),dst_metric_advmss (dst));tcp_initialize_rcv_mss (newsk);
net/ipv6/sit.c +1 -1
@@ -962,7 +962,7 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb, }if (df) { - mtu =dst_mtu (&rt->dst) - t_hlen; + mtu =dst4_mtu (&rt->dst) - t_hlen;if (mtu <IPV4_MIN_MTU ) {DEV_STATS_INC (dev, collisions);
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 7/8] ipv4: use dst4_mtu() instead of dst_mtu()
Quoted 5 lines · Eric Dumazet
When we expect an IPv4 dst, use
dst4_mtu()instead ofdst_mtu()to save some code space.Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
PATCH 8/8 ipv6: colocate inet6_cork in inet_cork_full
In reply to Eric DumazetAll inet6_cork users also use one inet_cork_full.
Reduce number of parameters and increase data locality.
This saves ~275 bytes of code on x86_64.
Signed-off-by Eric Dumazet <[email protected]>
--- include/linux/ipv6.h | 8 ------ include/net/inet_sock.h | 10 ++++++++ include/net/ipv6.h | 6 ++--- net/ipv6/ip6_output.c | 55 ++++++++++++++++++++--------------------- net/ipv6/raw.c | 2 +- 5 files changed, 40 insertions(+), 41 deletions(-)
include/linux/ipv6.h +0 -8
@@ -205,13 +205,6 @@ struct ipv6_mc_socklist;struct ipv6_ac_socklist ;struct ipv6_fl_socklist ; -struct inet6_cork { -struct ipv6_txoptions *opt; -u8 hop_limit; -u8 tclass; -u8 dontfrag:1 ; -}; -/* struct ipv6_pinfo - ipv6 private area */ struct ipv6_pinfo {/* Used in tx path (inet6_csk_route_socket(), ip6_xmit()) */
@@ -267,7 +260,6 @@ struct ipv6_pinfo {struct sk_buff *pktoptions;struct sk_buff *rxpmtu; -struct inet6_cork cork;struct ipv6_mc_socklist __rcu *ipv6_mc_list;struct ipv6_ac_socklist *ipv6_ac_list;
include/net/inet_sock.h +10 -0
@@ -159,6 +159,13 @@ static inline bool inet_sk_bound_dev_eq(const struct net *net,#endif } +struct inet6_cork { +struct ipv6_txoptions *opt; +u8 hop_limit; +u8 tclass; +u8 dontfrag:1 ; +}; +struct inet_cork {unsigned int flags;__be32 addr;
@@ -179,6 +186,9 @@ struct inet_cork {struct inet_cork_full {struct inet_cork base;struct flowi fl; +#if IS_ENABLED (CONFIG_IPV6 ) +struct inet6_cork base6; +#endif };struct ip_mc_socklist ;
include/net/ipv6.h +2 -4
@@ -1107,8 +1107,7 @@ void ip6_flush_pending_frames(struct sock *sk);int ip6_send_skb (struct sk_buff *skb);struct sk_buff *__ip6_make_skb (struct sock *sk,struct sk_buff_head *queue, -struct inet_cork_full *cork, -struct inet6_cork *v6_cork); +struct inet_cork_full *cork); struct sk_buff *ip6_make_skb(struct sock *sk, int getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb),
@@ -1119,8 +1118,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk,static inline struct sk_buff *ip6_finish_skb (struct sock *sk) { -return __ip6_make_skb (sk, &sk->sk_write_queue, &inet_sk (sk)->cork, - &inet6_sk (sk)->cork); +return __ip6_make_skb (sk, &sk->sk_write_queue, &inet_sk (sk)->cork); } int ip6_dst_lookup(struct net *net, struct sock *sk, struct dst_entry **dst,
net/ipv6/ip6_output.c +27 -28
175 lines not on this page · raw message
net/ipv6/raw.c +1 -1
@@ -529,7 +529,7 @@ static int rawv6_push_pending_frames(struct sock *sk, struct flowi6 *fl6, offset = rp->offset; total_len =inet_sk (sk)->cork.base.length; - opt =inet6_sk (sk)->cork.opt; + opt =inet_sk (sk)->cork.base6.opt; total_len -= opt ? opt->opt_flen :0 ;if (offset >= total_len -1 ) {
Signature · 1 line
2.53.0.rc1.225.gd81095ad13-goog
Re: [PATCH net-next 8/8] ipv6: colocate inet6_cork in inet_cork_full
Quoted 8 lines · Eric Dumazet
All
inet6_corkusers also use oneinet_cork_full.Reduce number of parameters and increase data locality.
This saves ~275 bytes of code on
x86_64.Signed-off-by Eric Dumazet <[email protected]>
Reviewed-by Kuniyuki Iwashima <[email protected]>
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 9 lines · On Fri, 30 Jan 2026 21:02:55 +0000 you wrote:
Small optimizations mostly in
ip6_xmit()path.TX performance increases by about 3 %.
Patches 5-7: add
dst4_mtu()anddst6_mtu()to save space.Last patch colocates
inet6_corkininet_cork_full.[...]
Here is the summary with links:
- [net-next,1/8] ipv6: pass proto by value to ipv6_push_nfrag_opts() and ipv6_push_frag_opts()
https://git.kernel.org/netdev/net-next/c/1bc46dd20966 - [net-next,2/8] ipv6: add some unlikely()/likely() clauses in ip6_output.c https://git.kernel.org/netdev/net-next/c/2855e4925416 - [net-next,3/8] ipv6: use __skb_push() in ip6_xmit() https://git.kernel.org/netdev/net-next/c/b5b1b676a30b - [net-next,4/8] ipv6: use SKB_DROP_REASON_PKT_TOO_BIG in ip6_xmit() https://git.kernel.org/netdev/net-next/c/94a150bf14af - [net-next,5/8] inet: add dst4_mtu() and dst6_mtu() helpers https://git.kernel.org/netdev/net-next/c/8776c4ef3ab2 - [net-next,6/8] ipv6: use dst6_mtu() instead of dst_mtu() https://git.kernel.org/netdev/net-next/c/b40f0130a23b - [net-next,7/8] ipv4: use dst4_mtu() instead of dst_mtu() https://git.kernel.org/netdev/net-next/c/fe8570186f10 - [net-next,8/8] ipv6: colocate inet6_cork in inet_cork_full https://git.kernel.org/netdev/net-next/c/b409a7f7176b
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html