Recommended hardware for iptables based firewall/router
Hi,
we recently bought an Uqbiquity EdgeRouter Pro but it seems the claims about 2 Mio. pps that it should be able to handle are not real-world numbers. We are running about 120mbit through this system and are already seeing the two risc cores struggling with high softirq load and packet drops.
So my question is what a good hardware base would look like for a linux based firewall using iptables/conntrack/ipset. Do offload features help or can't these be used because iptables needs to process the packets anyway? I assume multiqueuing would be nice too.
The idea is to be able to actually process 1gbit of traffic i.e. handle two gbit ports (WAN and LAN) at wire-speed.
Does anyone have any specific recommendations for NICs and maybe tips for other bottlenecks to look out for?
Regards,
Dennis
Quoted 14 lines · Dennis Jacobfeuerborn
Hi,
we recently bought an Uqbiquity EdgeRouter Pro but it seems the claims about 2 Mio. pps that it should be able to handle are not real-world numbers. We are running about 120mbit through this system and are already seeing the two risc cores struggling with high softirq load and packet drops.So my question is what a good hardware base would look like for a linux based firewall using iptables/conntrack/ipset. Do offload features help or can't these be used because iptables needs to process the packets anyway? I assume multiqueuing would be nice too.
The idea is to be able to actually process 1gbit of traffic i.e. handle two gbit ports (WAN and LAN) at wire-speed.
Quoted 2 lines
Does anyone have any specific recommendations for NICs and maybe tips for other bottlenecks to look out for?
I've been using a Lanner 7530 for some time now (the 7525 is the current 'replacement' for it); it runs the recently released Smoothwall 3.1 firewall*. Basically, a dual-core 1.6GHz Atom CPU with Intel NICs and 64MiB RAM can saturate four gigE links long term using 17-25W.
If you want more than netfilter (such as squid, snort, clamav, et al.), you'll want 1-2 GiB RAM and faster CPUs. And maybe more CPUs. If you want VPNs IPSEC and/or OpenVPN), you'll need at least faster CPUs.
Offload features usually preclude proper operation of netfilter.
GigE PCI NICs usually top out at 250-350Mb/s (limited by the PCI bus).
Intel NICs are generally the best. (I believe their lineage goes back to DEC, which might explain it.) RealTek's offerings of the last five years or so are also pretty good.
N
* Proper disclosure dictates that I state: I'm currently the lead dev. for Smoothwall Express (GNU/Linux/iptables). I mention it and its capabilities solely because I am very familiar with v3.1 and with testing/running it on a number of platforms--from a Y2K 600MHz Gateway PIII to an 8CPU/6GiB/virtio KVM on a Vishera 8350 with 16GiB RAM.
Quoted 27 lines · Neal Murphy
Quoted 14 lines · Dennis Jacobfeuerborn
Hi,
we recently bought an Uqbiquity EdgeRouter Pro but it seems the claims about 2 Mio. pps that it should be able to handle are not real-world numbers. We are running about 120mbit through this system and are already seeing the two risc cores struggling with high softirq load and packet drops.So my question is what a good hardware base would look like for a linux based firewall using iptables/conntrack/ipset. Do offload features help or can't these be used because iptables needs to process the packets anyway? I assume multiqueuing would be nice too.
The idea is to be able to actually process 1gbit of traffic i.e. handle two gbit ports (WAN and LAN) at wire-speed.Quoted 2 lines
Does anyone have any specific recommendations for NICs and maybe tips for other bottlenecks to look out for?
I've been using a Lanner 7530 for some time now (the 7525 is the current 'replacement' for it); it runs the recently released Smoothwall 3.1 firewall*. Basically, a dual-core 1.6GHz Atom CPU with Intel NICs and 64MiB RAM can saturate four gigE links long term using 17-25W.
If you want more than netfilter (such as squid, snort, clamav, et al.), you'll want 1-2 GiB RAM and faster CPUs. And maybe more CPUs. If you want VPNs IPSEC and/or OpenVPN), you'll need at least faster CPUs.
Quoted 7 lines
Offload features usually preclude proper operation of netfilter.
GigE PCI NICs usually top out at 250-350Mb/s (limited by the PCI bus).
Intel NICs are generally the best. (I believe their lineage goes back to DEC, which might explain it.) RealTek's offerings of the last five years or so are also pretty good.
After seeing the EdgeRouter not being able to handle the promised capacity even remotely I'm a bit suspicious of these "embedded" solutions. These processors only seem to be able to handle a decent amount of traffic in a pure routing best case scenario. I get the impression that as soon as you do add a bit of firewalling they need to fall back to slow execution paths and then the cpu's are seriously underpowered.
But then it might just be the EdgeRouter that is especially terrible. We immediately ran into problems when I found out that they had *reduced* the size of the connection tracking table to 16k entries from the default 64k. I cannot imagine why anyone would do this for a system that is designed to be a firewall.
Regards,
Dennis
A message that is not in the archive (sent off-list or not yet received): CAJygYd07-y0bDSr8THXWjNEW-e1rK5ZhGiE8aeJ_jXYJpFiL2A@mail.gmail.com
Dennis Jacobfeuerborn <[email protected]>
The EdgeRouter 's asic couldn't handle all use cases , Having some special rule will make it go to "offload" disabled mode. You should research if that's the problem.
As for Linux as a router, the key thing you want to test for is PPS, not BPS. Commodity hardware should be able to handle up to 1Mpps. Buy the best Xeon within your budget. Don't bother look at anything else. (if your project is serious and need to survive a ddos attack)
Cheers.
Quoted 12 lines · Yucong Sun
The EdgeRouter 's asic couldn't handle all use cases , Having some special rule will make it go to "offload" disabled mode.
You should research if that's the problem.
As for Linux as a router, the key thing you want to test for is PPS, not BPS. Commodity hardware should be able to handle up to 1Mpps. Buy the best Xeon within your budget. Don't bother look at anything else. (if your project is serious and need to survive a ddos attack)
Cheers
Quoted 5 lines · Yucong Sun
Dennis Jacobfeuerborn <[email protected]>
The EdgeRouter 's asic couldn't handle all use cases , Having some special rule will make it go to "offload" disabled mode. You should research if that's the problem.
Yes that seems to be the problem. Unfortunately the only things we use are vlan's and iptables+conntrack which I consider the be fairly standard features required for basic firewalling. I the system cannot handle traffic at a decent rate with these features than its hardware seems to be ill-spec'ed for its purpose.
Things got better when I was able to enable vlan offloading...until the cpu stalled and the system rebooted itself. Apparently the offloading is unstable.
None of this inspires confidence in a Product that is specifically advertised as a router/firewall that is sold with 8 Gbit ports and promises to handle 2 Mio+ pps.
Quoted 4 lines
As for Linux as a router, the key thing you want to test for is PPS, not BPS. Commodity hardware should be able to handle up to 1Mpps. Buy the best Xeon within your budget. Don't bother look at anything else. (if your project is serious and need to survive a ddos attack)
For now I have chosen a 2 quad-core cpu Xeon system I already have here and that has multiqueue capable Intel nics and have configured the appropriate irq affinity and XPS so each queue is handled by a dedicated core. I think this should provide relatively decent performance.
Regards,
Dennis