Kernel Blocking Firewall
One of the benefits of being the target of trolls and a variety of other malicious groups is that my servers have been subjected to non-stop denial of service attacks, hacking, email and spam floods, and just about every conceivable type of malicious activity someone with a computer is capable of perpetrating. I get thousands of spam emails a day with not less that 50 spambots every minute of every day lambasting my servers.
And I have to say, it's been a very rewarding experience from a technology perspective. Since I develop forensics software, this situation has never ceased to provide me with a wonderful testing environment for my forensics software. It got so bad, I had to write a firewall that blocks network traffic a lot like wikipedia does when dealing with malicious bots and this firewall is interactive with postfix and other apps.
One of the biggest problems of dealing with armies of spambots is even if you use greylisting or RBL blocking, it does not stop the connection flooding when they come back over and over and over and over.
iptables is just too cumbersome and memory comsumptive to work well and has a shitty app inteface so I wrote one with a kernel level database and
combined it with postfix. This firewall actually drops packets on the
floor by port, or in their entirety by IP address to deal with these jerks.
The code is a kernel module that will build an RBL database to disk and it will cache up to 500,000 IP addresses efficiently on a 1GB home personal computer. The more memory you have, the more IP addresses you can cache. It is configurable and possible to hold millions of them if you have 4GB of memory in the server.
You will need one of the forenfs patches to properly patch net/core/dev.c with the ff_filter hook. Included is postfix-2.4.5 which I have modified and integrated into the ff module. My mail server uses the standard rejection tests to determine whether or not to block an IP address FOREVER on port 25.
ftp://ftp.wolfmountaingroup.org/pub/forenfs/patches/ffs-2.6.27.8-el5-12-11-08.patch
ftp://ftp.wolfmountaingroup.org/pub/ff/postfix/postfix-2.4.5.tar.gz ftp://ftp.wolfmountaingroup.org/pub/ff/ff.tar.gz
It reduces the spam traffic from botnets by 98% on average since any system identified as a bot get perma-blocked. It's also useful if you have folks who like to perform DOS attacks or password guessing. The IOCTL interface can be quickly integrated into just about any app quickly, giving you the power to be a Wikipedia-style admin with your network traffic and either block by port or ban, and its a lot faster and more flexible than the iptables netfilter interface.
This software is designed specifically for active blocking with email programs and other servers.
Enjoy,
Jeff
Quoted 11 lines · On Thu, Jan 08, 2009 at 07:23:43PM -0700, [email protected] wrote:
iptables is just too cumbersome and memory comsumptive to work well and has a shitty app inteface so I wrote one with a kernel level database and
combined it with postfix. This firewall actually drops packets on the
floor by port, or in their entirety by IP address to deal with these jerks.The code is a kernel module that will build an RBL database to disk and it will cache up to 500,000 IP addresses efficiently on a 1GB home personal computer. The more memory you have, the more IP addresses you can cache. It is configurable and possible to hold millions of them if you have 4GB of memory in the server.
why didn't you use ipset for that ? It's designed exactly for this usage and is a lot easier to use than plain iptables for dynamic filtering.
Willy
Quoted 17 lines
Quoted 7 lines · On Thu, Jan 08, 2009 at 07:23:43PM -0700, [email protected] wrote:
iptables is just too cumbersome and memory comsumptive to work well and has a shitty app inteface so I wrote one with a kernel level database and
combined it with postfix. This firewall actually drops packets on the
floor by port, or in their entirety by IP address to deal with these jerks.Quoted 7 lines
The code is a kernel module that will build an RBL database to disk and it
will cache up to 500,000 IP addresses efficiently on a 1GB home personal computer. The more memory you have, the more IP addresses you can cache.
It is configurable and possible to hold millions of them if you have 4GB of memory in the server.
Quoted 4 lines
why didn't you use ipset for that ? It's designed exactly for this usage and is a lot easier to use than plain iptables for dynamic filtering.
Willy
No database to store the 500,000+ addresses you will harvest in about 2 months, that's why. The one I did uses an lru cached database that runs in the kernel, and not userspace, so you can filer real time, and manage the database real time.
Jeff
Quoted 2 lines
Quoted 20 lines · On Fri, Jan 09, 2009 at 12:36:06AM -0700, [email protected] wrote:
Quoted 20 lines
Quoted 7 lines · On Thu, Jan 08, 2009 at 07:23:43PM -0700, [email protected] wrote:
iptables is just too cumbersome and memory comsumptive to work well and has a shitty app inteface so I wrote one with a kernel level database and
combined it with postfix. This firewall actually drops packets on the
floor by port, or in their entirety by IP address to deal with these jerks.Quoted 7 lines
The code is a kernel module that will build an RBL database to disk and it
will cache up to 500,000 IP addresses efficiently on a 1GB home personal computer. The more memory you have, the more IP addresses you can cache.
It is configurable and possible to hold millions of them if you have 4GB of memory in the server.why didn't you use ipset for that ? It's designed exactly for this usage and is a lot easier to use than plain iptables for dynamic filtering.
Quoted 7 lines
Quoted 1 line
Willy
No database to store the 500,000+ addresses you will harvest in about 2 months, that's why. The one I did uses an lru cached database that runs in the kernel, and not userspace, so you can filer real time, and manage the database real time.
ipset runs in kernel too, you just add/remove entries from userspace without having to touch all other ones. It has no problem storing one million addresses and doing fast lookups on them.
I'm not dismissing your work, I just think it's a duplicate effort.
Also, since you're speaking about botnets, you should support automatic expiration of those addresses, because almost all those addresses are dynamic and will match a bot for a small amount of time, then match a normal non-infected user. One of the reasons you found 500k addresses might very well be because each bot appears one hundred times at different addresses.
Willy
... snip
Quoted 6 lines
ipset runs in kernel too, you just add/remove entries from userspace without having to touch all other ones. It has no problem storing one million addresses and doing fast lookups on them.
I'm not dismissing your work, I just think it's a duplicate effort.
Quoted 10 lines
Also, since you're speaking about botnets, you should support automatic expiration of those addresses, because almost all those addresses are dynamic and will match a bot for a small amount of time, then match a normal non-infected user. One of the reasons you found 500k addresses might very well be because each bot appears one hundred times at different addresses.
Willy
You should go and look at the code, 1) the window of addresses cached in memory is designed to act as an LRU windows for the addresses stored in the database to use less memory, so no, the in-memory only ip tables is primitive in comparison 2) the database can just keep growing ad growing
3) the code I posted also loads the database if the system reboots, so your applications remember all those botnet addresses 4) their is the ability to set a timer to expire and recycle the oldest addresses (while still remembering all of them).
Quoted 1 line
From my experience with dealing with these systems, and observation of how
RBL databases work, when an infected system gets blacklisted, it stays that way until the user goes to the websites and requests removal. I have found these zombie systems tend to stay that way, and no, by default you NEVER want to unblock them for at least 6 months.
Jeff
Quoted 4 lines · On Fri, Jan 09, 2009 at 01:15:06AM -0700, [email protected] wrote:
You should go and look at the code, 1) the window of addresses cached in memory is designed to act as an LRU windows for the addresses stored in the database to use less memory, so no, the in-memory only ip tables is primitive in comparison
I was not speaking about iptables but ipset which is an iptables extension.
Quoted 1 line
From my memories, you can have addresses stored as bitmaps, where one bit
equals one address. This would lead to less than 100 kB of RAM for your 500k addresses. But I agree that the concept of the LRU cache is interesting.
Quoted 5 lines
2) the database can just keep growing ad growing
3) the code I posted also loads the database if the system reboots, so your applications remember all those botnet addresses 4) their is the ability to set a timer to expire and recycle the oldest addresses (while still remembering all of them).
IIRC, this is also supported in ipset (I just have not looked at it for a while now).
Quoted 5 lines
From my experience with dealing with these systems, and observation of how RBL databases work, when an infected system gets blacklisted, it stays that way until the user goes to the websites and requests removal. I have found these zombie systems tend to stay that way, and no, by default you NEVER want to unblock them for at least 6 months.
This is stupid considering that most of them change their IP address every 24 hours, or at most every 7 days. This is just used to show that spam rate drops, hiding the fact that valid mails drop for similar reasons. For your own use, you might consider that you'll never receive mails from people hosted at the same ISP as the bots you block, but doing this on a large scale or for companies who do their business around e-mail is plain stupid.
I'm on a static IP, but a lot of people I know are not. It would be unfair to block them from posting to, say, LKML just because the week before, someone with their address had been sending spam. And no, it does not help getting the problem solved since the only users annoyed are not the ones with the faulty installation.
Willy
... snip
Quoted 17 lines
Quoted 7 lines
From my experience with dealing with these systems, and observation of how
RBL databases work, when an infected system gets blacklisted, it stays that way until the user goes to the websites and requests removal. I have
found these zombie systems tend to stay that way, and no, by default you NEVER want to unblock them for at least 6 months.This is stupid considering that most of them change their IP address every 24 hours, or at most every 7 days. This is just used to show that spam rate
drops, hiding the fact that valid mails drop for similar reasons. For your own use, you might consider that you'll never receive mails from people hosted at the same ISP as the bots you block, but doing this on a large scale or for companies who do their business around e-mail is plain stupid.
Quoted 9 lines
I'm on a static IP, but a lot of people I know are not. It would be unfair to block them from posting to, say, LKML just because the week before, someone with their address had been sending spam. And no, it does not help getting the problem solved since the only users annoyed are not the ones with the faulty installation.
Willy
Allowing a server on a dynamic IP range to act as a mail server is what is stupid. Most email blockers also block dynamic IP ranges since these systems are typically the ones most commonly infected, and the RBL databases. They just use a different technique, they base it on DNS maps of internet address ranges, and not behavior. It will only block the smtp port for these systems -- and mail servers originating from dynamic ranges should be blocked anyway -- and are by most commercial email gateway boxes.
Jeff
Quoted 2 lines · [email protected] wrote:
Allowing a server on a dynamic IP range to act as a mail server is what is stupid.
Sorry to be argue semantics, but I think it's SMTP clients that are the major cause of spam. Many, probably most, ISPs provide an SMTP relay for their subscribers, and blocking these dynamic addresses doesn't present a hard impediment for delivery of their mail to you. None the less, to say that people on a dynamic IP address should be *obliged* to go through a relay is offensive. (Not that you did say that, but I think it's what you had in mind.)
Quoted 5 lines
Quoted 3 lines · [email protected] wrote:
Allowing a server on a dynamic IP range to act as a mail server is what is
stupid.
Quoted 8 lines
Sorry to be argue semantics, but I think it's SMTP clients that are the major cause of spam. Many, probably most, ISPs provide an SMTP relay for their subscribers, and blocking these dynamic addresses doesn't present a hard impediment for delivery of their mail to you. None the less, to say that people on a dynamic IP address should be *obliged* to go through a relay is offensive. (Not that you did say that, but I think it's what you had in mind.)
That's good point, but not what I see in actual practice. These zombie systems don't seem to use the smtp gateway for their ranges, they send the emails direct. From my observations, most of these systems tend to stay zombies, since almost all of them are windows systems.
I have discovered that by using this method to block them, once they open a connection and I start dropping traffic while the connection is left open it "tarpits" the bots and causes a cascade failure in the botnet network for about 20 minutes until the systems start skipping the mail server doing the bocking.
Jeff
Quoted 2 lines · Willy Tarreau
why didn't you use ipset for that ? It's designed exactly for this usage and is a lot easier to use than plain iptables for dynamic filtering.
Any ideas when it will hit mainline? ipsets and PF_RING are the only ways to get two important jobs done: non-trivial firewalling on high-speed links, and packet capture in said links... and neither is in mainline AFAIK.
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Quoted 5 lines · Henrique de Moraes Holschuh
Quoted 2 lines · Willy Tarreau
why didn't you use ipset for that ? It's designed exactly for this usage and is a lot easier to use than plain iptables for dynamic filtering.
Any ideas when it will hit mainline?
no idea.
Quoted 3 lines
ipsets and
PF_RINGare the only ways
to get two important jobs done: non-trivial firewalling on high-speed links, and packet capture in said links...
yes, that's a bit right.
Quoted 1 line
and neither is in mainline AFAIK.
Well, you should ask Joszef for the former and Luca for the later :-)
Regards,
Willy