ipv6 issues after an DDoS for kernel 4.6.3
Subsystems (2)
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
The log gave just :
Jul 7 15:36:28 ms-magpie kernel: ------------[ cut here ]------------ Jul 7 15:36:28 ms-magpie kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:306 dev_watchdog+0x243/0x260 Jul 7 15:36:28 ms-magpie kernel: NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out Jul 7 15:36:28 ms-magpie kernel: Modules linked in: af_packet nf_log_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables i2c_i801 i2c_core tpm_tis tpm thermal processor atkbd button x86_pkg_temp_thermal Jul 7 15:36:28 ms-magpie kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.6.3-hardened #1 Jul 7 15:36:28 ms-magpie kernel: Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 9002 05/30/2014 Jul 7 15:36:28 ms-magpie kernel: 0000000000000000 ffff88041fa03db8 ffffffffbb3d655b 0000000000000007 Jul 7 15:36:28 ms-magpie kernel: ffff88041fa03e08 0000000000000000 ffff88041fa03df8 ffffffffbb07f7dd Jul 7 15:36:28 ms-magpie kernel: 000001321fa11640 0000000000000000 ffff88040d354080 0000000000000000 Jul 7 15:36:28 ms-magpie kernel: Call Trace: Jul 7 15:36:28 ms-magpie kernel: <IRQ> [<ffffffffbb3d655b>] dump_stack+0x4e/0x83 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb07f7dd>] __warn+0xcd/0x100 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb07f85a>] warn_slowpath_fmt+0x4a/0x70 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb59d633>] dev_watchdog+0x243/0x260 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb59d3f0>] ? dev_deactivate_queue+0x80/0x80 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb0db7b3>] call_timer_fn.isra.24+0x33/0xa0 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb59d3f0>] ? dev_deactivate_queue+0x80/0x80 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb0dba52>] run_timer_softirq+0x232/0x3c0 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb0eb188>] ? clockevents_program_event+0x98/0x160 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb08444d>] __do_softirq+0xfd/0x210 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb0846d0>] irq_exit+0x80/0xa0 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb03e9a4>] smp_apic_timer_interrupt+0x54/0x80 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb67805b>] apic_timer_interrupt+0x8b/0x90 Jul 7 15:36:28 ms-magpie kernel: <EOI> [<ffffffffbb53fa75>] ? cpuidle_enter_state+0x185/0x240 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb53fb82>] cpuidle_enter+0x12/0x30 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb0c0530>] cpu_startup_entry+0x1d0/0x220 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe13120>] ? early_idt_handler_array+0x120/0x120 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbb6701f5>] rest_init+0x6d/0x88 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe14c6c>] start_kernel+0x64c/0x692 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe13120>] ? early_idt_handler_array+0x120/0x120 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe7c7ff>] ? memblock_reserve+0x76/0x9c Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe136d7>] x86_64_start_reservations+0x53/0x75 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe136d7>] ? x86_64_start_reservations+0x53/0x75 Jul 7 15:36:28 ms-magpie kernel: [<ffffffffbbe1382d>] x86_64_start_kernel+0x134/0x16f Jul 7 15:36:28 ms-magpie kernel: ---[ end trace b779686b40691d67 ]--- Jul 7 15:36:28 ms-magpie kernel: r8169 0000:03:00.0 enp3s0: link up
I did not try to restart just the firewall or so.
WHat let me wonder were why just the IPv6 had a problem, whereas ipV4 worked smoothly.
Signature · 2 lines
Toralf PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7
Quoted 8 lines · Toralf Förster
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
IPv4 no longer has a routing cache, but IPv6 still has one.
Are you sure conntrack is needed at all ?
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Toralf Förster
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
Quoted 5 lines
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
IPv4 no longer has a routing cache, but IPv6 still has one.
Any pointers as to which part of the kernel to look for to implement one for IPv6 ?
Quoted 3 lines
Are you sure conntrack is needed at all ?
Quoted 11 lines · Loganaden Velvindron
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Toralf Förster
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
Quoted 8 lines
Quoted 5 lines
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
IPv4 no longer has a routing cache, but IPv6 still has one.
Any pointers as to which part of the kernel to look for to implement one for IPv6 ?
net/ipv6/route.c and net/ipv4/route.c ?
Quoted 1 line · Eric Dumazet
Are you sure conntrack is needed at all ?
Erm, I didn't mention conntrack - but yes, I do have in the firewall rules.
It is my understanding that conntrack is best practise, right ?
Signature · 2 lines
Toralf PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7
Quoted 6 lines · Toralf Förster
Quoted 1 line · Eric Dumazet
Are you sure conntrack is needed at all ?
Erm, I didn't mention conntrack - but yes, I do have in the firewall rules.
It is my understanding that conntrack is best practise, right ?
It depends what you want to protect ?
linux TCP stack should work quite well without conntrack.
If you are aware of any known defect, we should fix TCP stack instead of working around by adding a very expensive framework.
Quoted 2 lines · Eric Dumazet
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
Well, not too much I got from my ISP :
Quoted 10 lines · On 07 Jul 15:42, [email protected] wrote:
Direction IN
Internal 5.9.158.75
Threshold Packets 300.000 packets/s
Sum 100.982.000 packets/300s (336.606 packets/s), 100.769 flows/300s (335 flows/s), 6,093 GByte/300s (166 MBit/s) External 84.1.57.68, 824.000 packets/300s (2.746 packets/s), 823 flows/300s (2 flows/s), 0,046 GByte/300s (1 MBit/s) External 78.189.187.159, 748.000 packets/300s (2.493 packets/s), 748 flows/300s (2 flows/s), 0,042 GByte/300s (1 MBit/s) External 85.105.165.143, 725.000 packets/300s (2.416 packets/s), 725 flows/300s (2 flows/s), 0,041 GByte/300s (1 MBit/s) External 88.250.158.160, 715.000 packets/300s (2.383 packets/s), 715 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s) External 88.248.51.37, 714.000 packets/300s (2.380 packets/s), 714 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s) External 78.186.151.2, 708.000 packets/300s (2.360 packets/s), 708 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)
Signature · 2 lines
Toralf PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Toralf Förster
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
Quoted 4 lines
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
IPv4 no longer has a routing cache, but IPv6 still has one.
The difference is that routing exceptions are stored in "the" trie instead of hash tables in the fib nodes. IPv4 limits that by the size of the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty low.
Only redirects and mtu updates could potentially increase its size. Redirects are limited to the same L2 network, MTU updates must hit the socket to be acted upon appropriately. All limited to max_size, so I currently see a major problem in the routing code.
Unfortunately your report has not enough details to pinpoint a specific problem in the kernel
Bye,
Hannes
Quoted 21 lines · Hannes Frederic Sowa
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Toralf Förster
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
Quoted 4 lines
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
IPv4 no longer has a routing cache, but IPv6 still has one.
The difference is that routing exceptions are stored in "the" trie instead of hash tables in the fib nodes. IPv4 limits that by the size of the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty low.
Quoted 7 lines
Only redirects and mtu updates could potentially increase its size. Redirects are limited to the same L2 network, MTU updates must hit the socket to be acted upon appropriately. All limited to
max_size, so I currently see a major problem in the routing code.Unfortunately your report has not enough details to pinpoint a specific problem in the kernel
Well, the typical DDOS simply use SYN flood, right ? ;)
With IPv4, a server can typically absorb 10 Mpps SYN without major disruption on linux-4.6
With IPv6, kernel hits the route rwlock quite hard, at less than 2 Mpps
30.44% [kernel] [k] ip6_pol_route.isra.49
12.93% [kernel] [k] fib6_lookup
12.35% [kernel] [k] fib6_get_table
10.36% [kernel] [k] _raw_read_lock_bh
8.29% [kernel] [k] _raw_read_unlock_bh
2.02% [kernel] [k] dst_release
1.86% [kernel] [k] memcpy_erms I guess that switching to plain spinlock could help a bit, before major surgery.
Quoted 27 lines · Eric Dumazet
Quoted 26 lines · Hannes Frederic Sowa
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Toralf Förster
I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.
The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.
Quoted 4 lines
Hard to tell without knowing DDOS details, but IPv6 lacks some scalability improvements found in IPv4.
IPv4 no longer has a routing cache, but IPv6 still has one.
The difference is that routing exceptions are stored in "the" trie instead of hash tables in the fib nodes. IPv4 limits that by the size of the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty low.
Only redirects and mtu updates could potentially increase its size. Redirects are limited to the same L2 network, MTU updates must hit the socket to be acted upon appropriately. All limited to
max_size, so I currently see a major problem in the routing code.
Quoted 4 lines
Quoted 2 lines
Unfortunately your report has not enough details to pinpoint a specific problem in the kernel
Well, the typical DDOS simply use SYN flood, right ? ;)
Exactly, I can very well imagine that the stack becomes unresponsive during DDoS, but after the DDoS I don't see a reason why services should come up like in IPv4.
Quoted 13 lines
With IPv4, a server can typically absorb 10 Mpps SYN without major disruption on linux-4.6
With IPv6, kernel hits the route rwlock quite hard, at less than 2 Mpps
30.44% [kernel] [k] ip6_pol_route.isra.49 12.93% [kernel] [k] fib6_lookup 12.35% [kernel] [k] fib6_get_table 10.36% [kernel] [k] _raw_read_lock_bh 8.29% [kernel] [k] _raw_read_unlock_bh 2.02% [kernel] [k] dst_release 1.86% [kernel] [k] memcpy_erms
Quoted 2 lines
I guess that switching to plain spinlock could help a bit, before major surgery.
Hmm, interesting idea.
Quoted 3 lines · Hannes Frederic Sowa
Exactly, I can very well imagine that the stack becomes unresponsive during DDoS, but after the DDoS I don't see a reason why services should come up like in IPv4.
If the service uses different listeners, one for IPV4, one (or more) for IPv6, it is possible that an IPV6 flood leaves the IPV6 part in a sad state (this can also be an application bug, dealing with some backlog, like DNS requests)
Quoted 2 lines · Eric Dumazet
With IPv4, a server can typically absorb 10 Mpps SYN without major disruption on linux-4.6
Well, this particular server even survived >900 MBit/sec w/o any service disruption at IPv4 ([1]) but yesterday with a much more less attack the IPv6 issue was bothering me.
[1] https://www.zwiebeltoralf.de/torserver/ddos_sysstat_example.txt
Signature · 2 lines
Toralf PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7