UDP memory accounting (take 4)
Subsystems (6)
cover UDP memory accounting (take 4)
Hi,
I post a patch set of UDP memory accounting and limitation.
This patch set is for kernel 2.6.23-rc9. The differences from take 3 are
* fixing double accounting bug of ip_send_page()
* adding UDP memory limitation.
How to use UDP memory limitation:
This patch set add
/proc/sys/net/ipv4/udp_mem
as a tuning parameter.
When you give the number that is greater than 4096, UDP memory limitation will work. The number of pages for socket buffer is limited up to udp_mem[pages].
Currently this function drops the packet when it is sent or received and the number of pages for socket buffer is beyond the limit. It won't collect the buffer that is already allocated.
On the other hand, udp_mem is specified as 4096 or smaller, UDP memory limitaion will not work. The deafult number is 0.
Comment, review and test are welcome.
By the way, David pointed out that we should have the better solution such as memory reclaiming by callback from vmm. I seriously consider it. But As Herbert pointed out, it is very difficult to apply it to TCP because some of the buffer is already acked. I couldn't find the good solution that is applicable for TCP, UDP, route cache and so on.
Let me know, if you find the good way to solve this problem.
Thanks,
Satoshi Oshima
PATCH 1/4 UDP memory usage accounting (take 4): fix send buffer check
This patch introduces sndbuf size check before
memory allcation for send buffer.
net/ipv4/ip_output.c +5 -0
@@ -1004,6 +1004,11 @@ alloc_new_skb: frag = &skb_shinfo (skb)->frags[i]; } }else if (i <MAX_SKB_FRAGS ) { +if (atomic_read (&sk->sk_wmem_alloc) +PAGE_SIZE + >2 * sk->sk_sndbuf) { + err = -ENOBUFS ; + goto error; + }if (copy >PAGE_SIZE ) copy =PAGE_SIZE ; page =alloc_pages (sk->sk_allocation,0 );
PATCH 2/4 UDP memory usage accounting (take 4): accounting unit and variable
In reply to Satoshi OSHIMAThis patch introduces global variable for UDP memory accounting. The unit is page.
include/net/sock.h +7 -0
@@ -723,6 +723,13 @@ static inline int sk_stream_wmem_schedulsk_stream_mem_schedule (sk, size,0 ); } +#define SK_DATAGRAM_MEM_QUANTUM ((int)PAGE_SIZE) + +static inline int sk_datagram_pages (int amt) +{ +return DIV_ROUND_UP (amt,SK_DATAGRAM_MEM_QUANTUM ); +} +/* Used by processes to "lock" a socket state, so that * interrupts and bottom half handlers won't change it * from under us. It essentially blocks any incoming
include/net/udp.h +2 -0
@@ -65,6 +65,8 @@ extern rwlock_t udp_hash_lock;extern struct proto udp_prot; +extern atomic_t udp_memory_allocated; +struct sk_buff ;/*
net/ipv4/proc.c +2 -1
@@ -66,7 +66,8 @@ static int sockstat_seq_show(struct seq_fold_prot_inuse (&tcp_prot),atomic_read (&tcp_orphan_count), tcp_death_row.tw_count,atomic_read (&tcp_sockets_allocated), atomic_read(&tcp_memory_allocated)); -seq_printf (seq,"UDP: inuse %d\n" ,fold_prot_inuse (&udp_prot)); +seq_printf (seq,"UDP: inuse %d mem %d\n" ,fold_prot_inuse (&udp_prot), +atomic_read (&udp_memory_allocated));seq_printf (seq,"UDPLITE: inuse %d\n" ,fold_prot_inuse (&udplite_prot));seq_printf (seq,"RAW: inuse %d\n" ,fold_prot_inuse (&raw_prot)); seq_printf(seq, "FRAG: inuse %d memory %d\n", ip_frag_nqueues,
net/ipv4/udp.c +4 -0
@@ -113,6 +113,10 @@ DEFINE_SNMP_STAT(struct udp_mib, udp_stastruct hlist_head udp_hash[UDP_HTABLE_SIZE ];DEFINE_RWLOCK (udp_hash_lock); +atomic_t udp_memory_allocated; + +EXPORT_SYMBOL (udp_memory_allocated); +static int udp_port_rover;static inline int __udp_lib_lport_inuse (__u16 num,struct hlist_head udptable[])
Re: [RFC/PATCH 2/4] UDP memory usage accounting (take 4): accounting unit and variable
Hi.
net/ipv4/udp.c DEFINE_SNMP_STAT(struct udp_mib, udp_sta · line 119
Quoted 10 lines · Satoshi OSHIMA
net/ipv4/udp.c +4
-0@@ -113,6 +113,10 @@ DEFINE_SNMP_STAT(struct udp_mib, udp_stastruct hlist_head udp_hash[UDP_HTABLE_SIZE ];DEFINE_RWLOCK (udp_hash_lock); +atomic_t udp_memory_allocated; + +EXPORT_SYMBOL (udp_memory_allocated); +
Why do you export this variable?
It is not accessed from modules in your patchset.
Signature · 1 line
Evgeniy Polyakov
Re: [RFC/PATCH 2/4] UDP memory usage accounting (take 4): accounting unit and variable
Hi Evgeniy,
Thank you for your comment.
Quoted 9 lines
Hi.
Quoted 6 lines · Satoshi OSHIMA
net/ipv4/udp.c +0
-0@@ -113,6 +113,10 @@ DEFINE_SNMP_STAT(struct udp_mib, udp_stastruct hlist_head udp_hash[UDP_HTABLE_SIZE ];DEFINE_RWLOCK (udp_hash_lock);
Quoted 7 lines
net/ipv4/udp.c DEFINE_SNMP_STAT(struct udp_mib, udp_sta · line 119
Quoted 4 lines
net/ipv4/udp.c +4
-0+atomic_t udp_memory_allocated; + +EXPORT_SYMBOL (udp_memory_allocated); +Why do you export this variable?
It is not accessed from modules in your patchset.
Good point! I'll fix it.
Satoshi Oshima
PATCH 3/4 UDP memory usage accounting (take 4): memory usage accounting
In reply to Satoshi OSHIMAThis patch introduces memory usage accounting for UDP.
net/ipv4/ip_output.c +22 -3
@@ -743,6 +743,8 @@ static inline int ip_ufo_append_data(str/* specify the length of each IP datagram fragment*/ skb_shinfo (skb)->gso_size = mtu - fragheaderlen;skb_shinfo (skb)->gso_type =SKB_GSO_UDP ; +atomic_add (sk_datagram_pages (skb->truesize), + sk->sk_prot->memory_allocated);__skb_queue_tail (&sk->sk_write_queue, skb);return 0 ;
@@ -924,6 +926,9 @@ alloc_new_skb: }if (skb ==NULL ) goto error; +if (sk->sk_prot->memory_allocated) +atomic_add (sk_datagram_pages (skb->truesize), + sk->sk_prot->memory_allocated);/* * Fill in the control structures
@@ -1023,6 +1028,8 @@ alloc_new_skb: frag = &skb_shinfo (skb)->frags[i]; skb->truesize +=PAGE_SIZE ;atomic_add (PAGE_SIZE , &sk->sk_wmem_alloc); +if (sk->sk_prot->memory_allocated) +atomic_inc (sk->sk_prot->memory_allocated); }else { err = -EMSGSIZE ; goto error;
@@ -1123,7 +1130,9 @@ ssize_t ip_append_page(struct sock *sk,if (unlikely (!skb)) { err = -ENOBUFS ; goto error; - } + }else if (sk->sk_prot->memory_allocated) +atomic_add (sk_datagram_pages (skb->truesize), + sk->sk_prot->memory_allocated);/* * Fill in the control structures
@@ -1202,13 +1211,14 @@ int ip_push_pending_frames(struct sock *struct iphdr *iph;__be16 df =0 ;__u8 ttl; -int err =0 ; +int err =0 , send_page_size;if ((skb =__skb_dequeue (&sk->sk_write_queue)) ==NULL ) goto out; tail_skb = &(skb_shinfo (skb)->frag_list);/* move skb->data to ip header from ext header */ + send_page_size =sk_datagram_pages (skb->truesize);if (skb->data <skb_network_header (skb))__skb_pull (skb,skb_network_offset (skb));while ((tmp_skb =__skb_dequeue (&sk->sk_write_queue)) !=NULL ) {
@@ -1218,6 +1228,7 @@ int ip_push_pending_frames(struct sock * skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; + send_page_size +=sk_datagram_pages (tmp_skb->truesize);__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ;
@@ -1269,6 +1280,8 @@ int ip_push_pending_frames(struct sock */* Netfilter gets whole the not fragmented skb. */ err =NF_HOOK (PF_INET ,NF_IP_LOCAL_OUT , skb,NULL , skb->dst->dev, dst_output); +if (sk->sk_prot->memory_allocated) +atomic_sub (send_page_size, sk->sk_prot->memory_allocated);if (err) {if (err >0 ) err = inet->recverr ?net_xmit_errno (err) :0 ;
@@ -1298,9 +1311,15 @@ void ip_flush_pending_frames(struct sock {struct inet_sock *inet =inet_sk (sk);struct sk_buff *skb; +int num_flush_mem =0 ; -while ((skb =__skb_dequeue_tail (&sk->sk_write_queue)) !=NULL ) +while ((skb =__skb_dequeue_tail (&sk->sk_write_queue)) !=NULL ) { + num_flush_mem +=sk_datagram_pages (skb->truesize);kfree_skb (skb); + } + +if (sk->sk_prot->memory_allocated) +atomic_sub (num_flush_mem, sk->sk_prot->memory_allocated); inet->cork.flags &= ~IPCORK_OPT ;kfree (inet->cork.opt);
net/ipv4/udp.c +10 -0
@@ -887,6 +887,9 @@ try_again: err = ulen; out_free: +atomic_sub (sk_datagram_pages (skb->truesize), + sk->sk_prot->memory_allocated); +skb_free_datagram (sk, skb); out:return err;
@@ -894,6 +897,9 @@ out: csum_copy_err:UDP_INC_STATS_BH (UDP_MIB_INERRORS , is_udplite); +atomic_sub (sk_datagram_pages (skb->truesize), + sk->sk_prot->memory_allocated); +skb_kill_datagram (sk, skb, flags);if (noblock)
@@ -1019,6 +1025,9 @@ int udp_queue_rcv_skb(struct sock * sk, goto drop; } +atomic_add (sk_datagram_pages (skb->truesize), + sk->sk_prot->memory_allocated); +UDP_INC_STATS_BH (UDP_MIB_INDATAGRAMS , up->pcflag);return 0 ;
@@ -1443,6 +1452,7 @@ struct proto udp_prot = { .hash = udp_lib_hash, .unhash = udp_lib_unhash, .get_port = udp_v4_get_port, + .memory_allocated = &udp_memory_allocated, .obj_size =sizeof (struct udp_sock ), #ifdef CONFIG_COMPAT .compat_setsockopt = compat_udp_setsockopt,
net/ipv4/af_inet.c +29 -1
@@ -126,13 +126,41 @@ extern void ip_mc_drop_socket(struct socstatic struct list_head inetsw[SOCK_MAX ];static DEFINE_SPINLOCK (inetsw_lock); +/** +* __skb_queue_purge_and_sub_memory_allocated +* - empty a list and subtruct memory allocation counter +* @sk: sk +* @list: list to empty +* Delete all buffers on an &sk_buff list and subtruct the +* turesize of the sk_buff for memory accounting. Each buffer +* is removed from the list and one reference dropped. This +* function does not take the list lock and the caller must +* hold the relevant locks to use it. +*/ +void __skb_queue_purge_and_sub_memory_allocated (struct sock *sk, +struct sk_buff_head *list) +{ +struct sk_buff *skb; +int purged_skb_size =0 ; +while ((skb =__skb_dequeue (list)) !=NULL ) { + purged_skb_size +=sk_datagram_pages (skb->truesize); +kfree_skb (skb); + } +atomic_sub (purged_skb_size, sk->sk_prot->memory_allocated); +} +/* New destruction routine */ void inet_sock_destruct (struct sock *sk) {struct inet_sock *inet =inet_sk (sk); -__skb_queue_purge (&sk->sk_receive_queue); +if (sk->sk_prot->memory_allocated && sk->sk_type !=SOCK_STREAM ) +__skb_queue_purge_and_sub_memory_allocated (sk, + &sk->sk_receive_queue); +else +__skb_queue_purge (&sk->sk_receive_queue); +__skb_queue_purge (&sk->sk_error_queue);if (sk->sk_type ==SOCK_STREAM && sk->sk_state !=TCP_CLOSE ) {
PATCH 4/4 UDP memory usage accounting (take 4): memory limitation
In reply to Satoshi OSHIMAThis patch introduces memory limitation for UDP.
include/net/udp.h +3 -0
@@ -65,7 +65,10 @@ extern rwlock_t udp_hash_lock;extern struct proto udp_prot; +/* Used by memory accounting and capping */ +#define UDP_MIN_SKB_PAGES 4096extern atomic_t udp_memory_allocated; +extern int sysctl_udp_mem;struct sk_buff ;
net/ipv4/udp.c +13 -0
@@ -114,8 +114,10 @@ struct hlist_head udp_hash[UDP_HTABLE_SIDEFINE_RWLOCK (udp_hash_lock);atomic_t udp_memory_allocated; +int sysctl_udp_mem =0 ;EXPORT_SYMBOL (udp_memory_allocated); +EXPORT_SYMBOL (sysctl_udp_mem);static int udp_port_rover;
@@ -1018,6 +1020,16 @@ int udp_queue_rcv_skb(struct sock * sk, goto drop; } +if (sk->sk_prot->sysctl_mem[0 ] >UDP_MIN_SKB_PAGES ) { +if ((atomic_read (sk->sk_prot->memory_allocated) + +sk_datagram_pages (skb->truesize)) + >= sk->sk_prot->sysctl_mem[0 ]) { +UDP_INC_STATS_BH (UDP_MIB_RCVBUFERRORS , + up->pcflag); + goto drop; + } + } +if ((rc =sock_queue_rcv_skb (sk,skb)) <0 ) {/* Note that an ENOMEM error is charged twice */ if (rc == -ENOMEM )
@@ -1453,6 +1465,7 @@ struct proto udp_prot = { .unhash = udp_lib_unhash, .get_port = udp_v4_get_port, .memory_allocated = &udp_memory_allocated, + .sysctl_mem = &sysctl_udp_mem, .obj_size =sizeof (struct udp_sock ), #ifdef CONFIG_COMPAT .compat_setsockopt = compat_udp_setsockopt,
net/ipv4/sysctl_net_ipv4.c +9 -0
@@ -17,6 +17,7 @@#include <net/ip.h> #include <net/route.h> #include <net/tcp.h> +#include <net/udp.h> #include <net/cipso_ipv4.h> /* From af_inet.c */
@@ -599,6 +600,14 @@ ctl_table ipv4_table[] = { .proc_handler = &proc_dointvec }, { + .ctl_name = NET_UDP_MEM, + .procname = "udp_mem", + .data = &sysctl_udp_mem, + .maxlen = sizeof(sysctl_udp_mem), + .mode = 0644, + .proc_handler = &proc_dointvec + }, + { .ctl_name = NET_TCP_APP_WIN, .procname = "tcp_app_win", .data = &sysctl_tcp_app_win,
include/linux/sysctl.h +1 -0
@@ -441,6 +441,7 @@ enumNET_TCP_ALLOWED_CONG_CONTROL =123 ,NET_TCP_MAX_SSTHRESH =124 ,NET_TCP_FRTO_RESPONSE =125 , + NET_UDP_MEM=126, }; enum {
net/ipv4/ip_output.c +32 -0
@@ -75,6 +75,7 @@#include <net/icmp.h> #include <net/checksum.h> #include <net/inetpeer.h> +#include <net/udp.h> #include <linux/igmp.h> #include <linux/netfilter_ipv4.h> #include <linux/netfilter_bridge.h>
@@ -910,6 +911,17 @@ alloc_new_skb:if (datalen == length + fraggap) alloclen += rt->u.dst.trailer_len; +if (sk->sk_prot->sysctl_mem) +if (sk->sk_prot->sysctl_mem[0 ] >UDP_MIN_SKB_PAGES ) +if ((atomic_read (sk->sk_prot->memory_allocated) + +sk_datagram_pages ( +SKB_DATA_ALIGN (alloclen + hh_len +15 ) + +sizeof (struct sk_buff ))) + >= sk->sk_prot->sysctl_mem[0 ]) { + err = -ENOBUFS ; + goto error; + } +if (transhdrlen) { skb = sock_alloc_send_skb(sk, alloclen + hh_len + 15,
@@ -1009,6 +1021,15 @@ alloc_new_skb: frag = &skb_shinfo (skb)->frags[i]; } }else if (i <MAX_SKB_FRAGS ) { +if (sk->sk_prot->sysctl_mem) +if (sk->sk_prot->sysctl_mem[0 ] >UDP_MIN_SKB_PAGES ) +if ((atomic_read (sk->sk_prot->memory_allocated) + +sk_datagram_pages (PAGE_SIZE )) + >= sk->sk_prot->sysctl_mem[0 ]) { + err = -ENOBUFS ; + goto error; + } +if (atomic_read (&sk->sk_wmem_alloc) +PAGE_SIZE >2 * sk->sk_sndbuf) { err = -ENOBUFS ;
@@ -1126,6 +1147,17 @@ ssize_t ip_append_page(struct sock *sk, fraggap = skb_prev->len - maxfraglen; alloclen = fragheaderlen + hh_len + fraggap +15 ; + +if (sk->sk_prot->sysctl_mem) +if (sk->sk_prot->sysctl_mem[0 ] >UDP_MIN_SKB_PAGES ) +if ((atomic_read (sk->sk_prot->memory_allocated) + +sk_datagram_pages (alloclen + +sizeof (struct sk_buff ))) + >= sk->sk_prot->sysctl_mem[0 ]) { + err = -ENOBUFS ; + goto error; + } + skb =sock_wmalloc (sk, alloclen,1 , sk->sk_allocation);if (unlikely (!skb)) { err = -ENOBUFS ;
Re: [RFC/PATCH 4/4] UDP memory usage accounting (take 4): memory limitation
Quoted 23 lines · On Sat, 06 Oct 2007 00:02:47 +0900 Satoshi OSHIMA…
This patch introduces memory limitation for UDP.
Signed-off-by Satoshi Oshima <[email protected]>Signed-off-by Hideo Aoki <[email protected]>Index:
2.6.23-rc9-udp_limit/net/ipv4/sysctl_net_ipv4.c
===================================================================net/ipv4/sysctl_net_ipv4.c +2
-0@@ -17,6 +17,7 @@#include <net/ip.h> #include <net/route.h> #include <net/tcp.h> +#include <net/udp.h> #include <net/cipso_ipv4.h> /* From af_inet.c */ @@ -599,6 +600,14 @@ ctl_table ipv4_table[] = { .proc_handler = &proc_dointvec }, { + .ctl_name = NET_UDP_MEM,
net/ipv4/sysctl_net_ipv4.c ctl_table ipv4_table[] = { · line 613
Quoted 10 lines
net/ipv4/sysctl_net_ipv4.c +7
-0+ .procname = "udp_mem", + .data = &sysctl_udp_mem, + .maxlen = sizeof(sysctl_udp_mem), + .mode = 0644, + .proc_handler = &proc_dointvec + }, + { .ctl_name = NET_TCP_APP_WIN, .procname = "tcp_app_win", .data = &sysctl_tcp_app_win,
if you use &proc_dointvec_minmax, then you could inforce min/max values for udp_mem for the sysctl
Signature · 1 line
Stephen Hemminger <[email protected]>
Re: [RFC/PATCH 4/4] UDP memory usage accounting (take 4): memory limitation
Hi Stephen,
Thank you for your comment.
Quoted 7 lines
Quoted 7 lines
net/ipv4/sysctl_net_ipv4.c +6
-0{ + .ctl_name = NET_UDP_MEM, + .procname = "udp_mem", + .data = &sysctl_udp_mem, + .maxlen = sizeof(sysctl_udp_mem), + .mode =0644 , + .proc_handler = &proc_dointvec
Quoted 8 lines
net/ipv4/sysctl_net_ipv4.c ctl_table ipv4_table[] = { · line 613
Quoted 5 lines
net/ipv4/sysctl_net_ipv4.c +2
-0+ }, + { .ctl_name = NET_TCP_APP_WIN, .procname = "tcp_app_win", .data = &sysctl_tcp_app_win,if you use
&proc_dointvec_minmax, then you could inforce min/max values forudp_memfor the sysctl
udp_mem has two meanings:
* turn off this limitation function (currently udp_mem<=4096)
* limit udp memory (currently udp_mem>4096)
To realize this, udp_mem is evaluated whether udp_mem equals 4096 or smaller in UDP and IP layers.
If udp_mem has proc_dointvec_minmax or dedicated proc handler, turn off check must be done in UDP and IP layers. This means there is no reduction of the check in UDP and IP layers.
If you pointed out that minus value of udp_mem is strange, I agree. I'll fix it.
How about this?
min=4096 (and turn off limitation)
udp_mem>4096 (and turn on limitation)
Satoshi Oshima
Re: [RFC/PATCH 4/4] UDP memory usage accounting (take 4): memory limitation
Quoted 19 lines · On Thu, 11 Oct 2007 21:51:14 +0900 Satoshi OSHIMA…
Hi Stephen,
Thank you for your comment.
Quoted 15 lines
net/ipv4/sysctl_net_ipv4.c ctl_table ipv4_table[] = { · line 613
Quoted 12 lines
net/ipv4/sysctl_net_ipv4.c +8
-0{ + .ctl_name = NET_UDP_MEM, + .procname = "udp_mem", + .data = &sysctl_udp_mem, + .maxlen = sizeof(sysctl_udp_mem), + .mode = 0644, + .proc_handler = &proc_dointvec + }, + { .ctl_name = NET_TCP_APP_WIN, .procname = "tcp_app_win", .data = &sysctl_tcp_app_win,if you use
&proc_dointvec_minmax, then you could inforce min/max values forudp_memfor the sysctl
One other comment. Sysctl value indexes are deprecated at this point so all new values should use CTL_UNNUMBERED. Therefore unless NET_UDP_MEM already exists, please don't add it.
Signature · 1 line
Stephen Hemminger <[email protected]>
Re: [RFC/PATCH 4/4] UDP memory usage accounting (take 4): memory limitation
Hi Stephen,
Quoted 22 lines
Quoted 18 lines · On Thu, 11 Oct 2007 21:51:14 +0900 Satoshi OSHIMA…
Hi Stephen,
Thank you for your comment.
Quoted 14 lines
net/ipv4/sysctl_net_ipv4.c ctl_table ipv4_table[] = { · line 613
Quoted 12 lines
net/ipv4/sysctl_net_ipv4.c +8
-0{ + .ctl_name = NET_UDP_MEM, + .procname = "udp_mem", + .data = &sysctl_udp_mem, + .maxlen = sizeof(sysctl_udp_mem), + .mode = 0644, + .proc_handler = &proc_dointvec + }, + { .ctl_name = NET_TCP_APP_WIN, .procname = "tcp_app_win", .data = &sysctl_tcp_app_win,if you use
&proc_dointvec_minmax, then you could inforce min/max values forudp_memfor the sysctl
Quoted 3 lines
One other comment. Sysctl value indexes are deprecated at this point so all new values should use
CTL_UNNUMBERED. Therefore unlessNET_UDP_MEMalready exists, please don't add it.
Thank you for letting me know. I will fix it.
Satoshi Oshima