tcp: Fast/early SYN handling to mitigate SYN floods
Subsystems (3)
RFC PATCH 1/1 tcp: Fast/early SYN handling to mitigate SYN floods
Hi Eric,
I have been doing some TCP performance measurements with SYN flooding, and have found that, we don't handle this case well.
I have made a patch for fast/early SYN handling in tcp_v4_rcv() in net/ipv4/tcp_ipv4.c. This increases SYN performance from 130 kpps to 750 kpps (max of the generator), with idle CPU cycles.
Current locking:
During a SYN flood (against a single port) all CPUs are spinning on
the same spinlock, namely bh_lock_sock_nested(sk), in tcp_ipv4.c. The lock dates back to a commit by DaveM in May 1999, see historic commit[1]. It seem that TCP runs fully locked, per sock.
I need some help with locking, as the patch seems to work fine, with NO-PREEMPT, but with PREEMPT enabled I start to see warnings (in reqsk_queue_destroy) and oopses (in inet_csk_reqsk_queue_prune).
What am I missing?
[1] Historic commit: http://git.kernel.org/?p=linux/kernel/git/davem/netdev-vger-cvs.git;a=commitdiff;h=5744fad55cefbd6f079410500a507443d92d63ff
Signature · 25 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer [RFC PATCH] tcp: Fast/early SYN handling to mitigate SYN floods TCP SYN handling is on the slow path via tcp_v4_rcv(), and is performed while holding spinlock bh_lock_sock(). Real-life and testlab experiments show, that the kernel choks when reaching 130Kpps SYN floods (powerful Nehalem 16 cores). Measuring with perf reveals, that its caused by bh_lock_sock_nested() call in tcp_v4_rcv(). With this patch, the machine can handle 750Kpps (max of the SYN flood generator) with cycles to spare. Signed-off-by: Jesper Dangaard Brouer <[email protected]> --- net/ipv4/tcp_ipv4.c | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-)
net/ipv4/tcp_ipv4.c +16 -0
@@ -1718,6 +1718,22 @@ int tcp_v4_rcv(struct sk_buff *skb)if (!sk) goto no_tcp_socket; +/* Fast/early SYN handling, to mitigate SYN attacks */ +if (sk->sk_state ==TCP_LISTEN && th->syn && !th->ack && !th->fin) { +//bh_lock_sock_nested(sk); /* Don't think lock is needed */ +/* Handles syn cookie, normally called from +* tcp_rcv_state_process() */ +tcp_v4_conn_request (sk, skb); +//bh_unlock_sock(sk); + +/* Questions, do we (really) need to create a new sk, +* as in tcp_v4_hnd_req() ? +*/ +sock_put (sk); +kfree_skb (skb); +return 0 ; + } + process:if (sk->sk_state ==TCP_TIME_WAIT ) goto do_time_wait;
Hi Jesper
We are also working with this issue right now,
Quoted 15 lines · On Thursday 24 May 2012 15:01:07 Jesper Dangaard Brouer wrote:
Hi Eric,
I have been doing some TCP performance measurements with SYN flooding, and have found that, we don't handle this case well.
I have made a patch for fast/early SYN handling in
tcp_v4_rcv()innet/ipv4/tcp_ipv4.c. This increases SYN performance from 130 kpps to 750 kpps (max of the generator), with idle CPU cycles.Current locking:
During a SYN flood (against a single port) all CPUs are spinning on
the same spinlock, namelybh_lock_sock_nested(sk), intcp_ipv4.c. The lock dates back to a commit by DaveM in May 1999, see historic commit[1]. It seem that TCP runs fully locked, per sock.
Quoted 25 lines
I need some help with locking, as the patch seems to work fine, with NO-PREEMPT, but with PREEMPT enabled I start to see warnings (in
reqsk_queue_destroy) and oopses (ininet_csk_reqsk_queue_prune).What am I missing?
[1] Historic commit: http://git.kernel.org/?p=linux/kernel/git/davem/netdev-vger-cvs.git;a=commitdiff;h=5744fad55cefbd6f079410500a507443d92d63ff
Signature · 16 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer [RFC PATCH] tcp: Fast/early SYN handling to mitigate SYN floods TCP SYN handling is on the slow path via tcp_v4_rcv(), and is performed while holding spinlock bh_lock_sock(). Real-life and testlab experiments show, that the kernel choks when reaching 130Kpps SYN floods (powerful Nehalem 16 cores). Measuring with perf reveals, that its caused by bh_lock_sock_nested() call in tcp_v4_rcv().
I can confirm this too, and it doesn't scale with more cores
Quoted 3 lines
With this patch, the machine can handle 750Kpps (max of the SYN flood generator) with cycles to spare.
This looks great.
I'm also working with a solution that not trash conntack i.e. have conntrack working during a heavy SYN attack
Signature · 2 lines
Regards Hans Schillstrom
Quoted 3 lines · Hans Schillstrom
Hi Jesper
We are also working with this issue right now,
[..]
Quoted 10 lines
Quoted 9 lines
[RFC PATCH] tcp: Fast/early SYN handling to mitigate SYN floods
TCP SYN handling is on the slow path via
tcp_v4_rcv(), and is performed while holding spinlockbh_lock_sock().Real-life and testlab experiments show, that the kernel choks when reaching 130Kpps SYN floods (powerful Nehalem 16 cores). Measuring with perf reveals, that its caused by
bh_lock_sock_nested()call intcp_v4_rcv().
Quoted 7 lines
I can confirm this too, and it doesn't scale with more cores
Quoted 3 lines
With this patch, the machine can handle 750Kpps (max of the SYN flood generator) with cycles to spare.
This looks great.
Yes, its definitely shows that there is huge performance gain hidden here! But we still have to handle locking (which will affect perf).
Quoted 2 lines
I'm also working with a solution that not trash conntack i.e. have conntrack working during a heavy SYN attack
Sounds interesting, but that's a separate problem. In this case I have disabled conntracking (I even disabled flow-control and drop the syn-ack responses on the generator).
Signature · 5 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer
Hello,
Quoted 13 lines · Jesper Dangaard Brouer
I have been doing some TCP performance measurements with SYN flooding, and have found that, we don't handle this case well.
I have made a patch for fast/early SYN handling in
tcp_v4_rcv()innet/ipv4/tcp_ipv4.c. This increases SYN performance from 130 kpps to 750 kpps (max of the generator), with idle CPU cycles.Current locking:
During a SYN flood (against a single port) all CPUs are spinning on
the same spinlock, namelybh_lock_sock_nested(sk), intcp_ipv4.c. The lock dates back to a commit by DaveM in May 1999, see historic commit[1]. It seem that TCP runs fully locked, per sock.
Quoted 5 lines
I need some help with locking, as the patch seems to work fine, with NO-PREEMPT, but with PREEMPT enabled I start to see warnings (in
reqsk_queue_destroy) and oopses (ininet_csk_reqsk_queue_prune).What am I missing?
For each retransmission of a SYN you will add a request-sock to the syn_table, because you do not pass by tcp_v4_hnd_req(), which checks this by calling inet_csk_search_req().
And your warning in reqsk_queue_destroy is because the access to the the request_sock_queue is no more protected by a lock.
The request_sock_queue is a shared resource, which must be protect by a lock. As you allow "parallel" SYN-processing, the queue will get corrupted.
Cheers,
Christoph
Signature · 7 lines
Christoph Paasch PhD Student IP Networking Lab --- http://inl.info.ucl.ac.be MultiPath TCP in the Linux Kernel --- http://mptcp.info.ucl.ac.be Université Catholique de Louvain --
Quoted 26 lines · Christoph Paasch
Hello,
Quoted 13 lines · Jesper Dangaard Brouer
I have been doing some TCP performance measurements with SYN flooding, and have found that, we don't handle this case well.
I have made a patch for fast/early SYN handling in
tcp_v4_rcv()innet/ipv4/tcp_ipv4.c. This increases SYN performance from 130 kpps to 750 kpps (max of the generator), with idle CPU cycles.Current locking:
During a SYN flood (against a single port) all CPUs are spinning on
the same spinlock, namelybh_lock_sock_nested(sk), intcp_ipv4.c. The lock dates back to a commit by DaveM in May 1999, see historic commit[1]. It seem that TCP runs fully locked, per sock.Quoted 5 lines
I need some help with locking, as the patch seems to work fine, with NO-PREEMPT, but with PREEMPT enabled I start to see warnings (in
reqsk_queue_destroy) and oopses (ininet_csk_reqsk_queue_prune).What am I missing?
For each retransmission of a SYN you will add a request-sock to the
syn_table, because you do not pass bytcp_v4_hnd_req(), which checks this by callinginet_csk_search_req().
Quoted 7 lines
And your warning in
reqsk_queue_destroyis because the access to the therequest_sock_queueis no more protected by a lock.The
request_sock_queueis a shared resource, which must be protect by a lock. As you allow "parallel" SYN-processing, the queue will get corrupted.
Hi guys, that's a very interesting subject.
I began work on fully converting this stuff to RCU some weeks ago but got distracted by codel / fq_codel and other cool stuff (TCP coalescing and skb->frag_head)
I dont know if you remember the SO_REUSEPORT patch(s) posted by Tom Herbert in the past. The remaining issue was about adding/removing a new listener to a pool of listeners to same port, and hash function was changed so we could lost some connexions in SYN_RECV state at this stage.
So I was working having a shared table, and not anymore using a central spinlock, but an array of spinlock, as done elsewhere (ESTABLISHED/TIMEWAIT hash tables)
My work is probably a ~500 LOC target, allowing concurrent processing by all cpus of the host.
Jesper, my goals are probably different than yours, unless I misunderstood your intention.
I feel you want to have an emergency mode, when listener is overflowed to immediately send a SYNCOOKIE ?
Quoted 21 lines · Eric Dumazet
Quoted 20 lines · Christoph Paasch
Hello,
Quoted 17 lines · Jesper Dangaard Brouer
I have been doing some TCP performance measurements with SYN flooding, and have found that, we don't handle this case well.
I have made a patch for fast/early SYN handling in
tcp_v4_rcv()innet/ipv4/tcp_ipv4.c. This increases SYN performance from 130 kpps to 750 kpps (max of the generator), with idle CPU cycles.Current locking:
During a SYN flood (against a single port) all CPUs are spinning on
the same spinlock, namelybh_lock_sock_nested(sk), intcp_ipv4.c. The lock dates back to a commit by DaveM in May 1999, see historic commit[1]. It seem that TCP runs fully locked, per sock.I need some help with locking, as the patch seems to work fine, with NO-PREEMPT, but with PREEMPT enabled I start to see warnings (in
reqsk_queue_destroy) and oopses (ininet_csk_reqsk_queue_prune).
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
What am I missing?
For each retransmission of a SYN you will add a request-sock to the
syn_table, because you do not pass bytcp_v4_hnd_req(), which checks this by callinginet_csk_search_req().
Thanks that's good hint. I was suspecting that tcp_v4_hnd_req() was somehow needed (as noted in the comment in the patch)
Quoted 2 lines
Quoted 2 lines
And your warning in
reqsk_queue_destroyis because the access to the therequest_sock_queueis no more protected by a lock.
Yes, I was suspecting that.
Quoted 10 lines
Quoted 3 lines
The
request_sock_queueis a shared resource, which must be protect by a lock. As you allow "parallel" SYN-processing, the queue will get corrupted.Hi guys, that's a very interesting subject.
I began work on fully converting this stuff to RCU some weeks ago but got distracted by codel /
fq_codeland other cool stuff (TCP coalescing andskb->frag_head)
Quoted 5 lines
I dont know if you remember the
SO_REUSEPORTpatch(s) posted by Tom Herbert in the past. The remaining issue was about adding/removing a new listener to a pool of listeners to same port, and hash function was changed so we could lost some connexions inSYN_RECVstate at this stage.
Sorry, don't remember.
Quoted 6 lines
So I was working having a shared table, and not anymore using a central spinlock, but an array of spinlock, as done elsewhere (ESTABLISHED/TIMEWAIT hash tables)
My work is probably a ~500 LOC target, allowing concurrent processing by all cpus of the host.
Sounds really promising, especially coming from the network-ninja :-)
Quoted 5 lines
Jesper, my goals are probably different than yours, unless I misunderstood your intention.
I feel you want to have an emergency mode, when listener is overflowed to immediately send a SYNCOOKIE ?
Yes, this is more an emergency mode.
I was thinking of only handling the SYN cookie case in parallel. That should be easier locking wise, right.
I'm also considering writing a netfilter/iptables syn-cookie module, as this would allow people to use it in combination with IPset, to e.g create a whitelist feature of known-good-hosts (which have completed the TCP handshake). But it would be nicer if the base kernel was just fast enough to handle these SYN floods.
Signature · 5 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer
Quoted 1 line · Jesper Dangaard Brouer
Sorry, don't remember.
http://kerneltrap.org/mailarchive/linux-netdev/2010/4/19/6274993
Quoted 1 line
Sounds really promising, especially coming from the network-ninja :-)
;)
Quoted 11 lines
Yes, this is more an emergency mode.
I was thinking of only handling the SYN cookie case in parallel. That should be easier locking wise, right.
I'm also considering writing a netfilter/iptables syn-cookie module, as this would allow people to use it in combination with IPset, to e.g create a whitelist feature of known-good-hosts (which have completed the TCP handshake). But it would be nicer if the base kernel was just fast enough to handle these SYN floods.
Indeed, I believe I can make this happen eventually in a short term.