Forking inside netfilter queue
Hi,
I am building netfilter queue application, that implement simple HTTP filter.
It basically will capture the packet, extract the URL, and issue MySQL lookup to see if the address is permitted or blocked.
My only fear that address lookup will block the traffic until I get the lookup reply, so I am thinking of using preforking model to fork 10-20 processes to handle the same queue.
Has any one done multi prcoess handling of the queue, is it even doable?
Rayed
Quoted 4 lines · Rayed
I am building netfilter queue application, that implement simple HTTP filter.
Quoted 8 lines
It basically will capture the packet, extract the URL, and issue MySQL lookup to see if the address is permitted or blocked.
My only fear that address lookup will block the traffic until I get the lookup reply, so I am thinking of using preforking model to fork 10-20 processes to handle the same queue.
Has any one done multi prcoess handling of the queue, is it even doable?
See Apache, it uses preforking. But perhaps threading is much simpler and equally fast.
Jan
Thanks Jan for the reply.
I know how preforking works (almost), what I want to know if multi processes can access the same queue API without breaking anything.
Quoted 21 lines · Jan Engelhardt
Quoted 12 lines · Rayed
I am building netfilter queue application, that implement simple HTTP filter.
It basically will capture the packet, extract the URL, and issue MySQL lookup to see if the address is permitted or blocked.
My only fear that address lookup will block the traffic until I get the lookup reply, so I am thinking of using preforking model to fork 10-20 processes to handle the same queue.
Has any one done multi prcoess handling of the queue, is it even doable?
See Apache, it uses preforking. But perhaps threading is much simpler and equally fast.
Jan
Quoted 5 lines · Rayed
Thanks Jan for the reply.
I know how preforking works (almost), what I want to know if multi processes can access the same queue API without breaking anything.
Well, you should be aware that calling ipq_read() [or whatever it's called now] in a thread may return a packet that does not actually belong to the connection you created a certain thread for. Maybe you even need to pthread_mutex_* around calls to ipq_*, I am not sure if libipq (or whatever we use now) does that by itself.
Jan
Can't get --dport to work
Hi,
At one time I had this working. I could match on the ports and deny http traffic from certain interfaces.
Now I get an error message and I am not sure what is causing it. Can anyone help?
# iptables -A OUTPUT -i ath0 -p tcp --dport 80 -j ACCEPT iptables v1.3.6-20061129: Unknown arg `--dport' Try `iptables -h' or 'iptables --help' for more information.
Thanks,
Joel Lindsay, B.Eng
Project Engineer
RE: Can't get --dport to work
I don't know why it complains about the --dport option. However, I don't see how -i ath0 option is compatible with the OUTPUT chain.
Siva
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Joel Lindsay
Sent: Tuesday, May 01, 2007 11:27 AM
To: [email protected]
Subject: Can't get --dport to work
Hi,
At one time I had this working. I could match on the ports and deny http
traffic from certain interfaces.
Now I get an error message and I am not sure what is causing it. Can anyone
help?
# iptables -A OUTPUT -i ath0 -p tcp --dport 80 -j ACCEPT iptables v1.3.6-20061129: Unknown arg `--dport' Try `iptables -h' or 'iptables --help' for more information.
Thanks,
Joel Lindsay, B.Eng
Project Engineer
Unless there are a real lot of these addresses that are permitted, and they change rapidly, then I'd expect it would be a better design to get all the valid URIs from the MySQL query on init and store them in a linked list.
Also, people already do this type of thing with an HTTP proxy like squid before the HTTPd.
----- Original Message ----- From: "Rayed" <[email protected]>
To: <[email protected]>
Sent: Monday, April 30, 2007 2:18 PM
Subject: Forking inside netfilter queue
Quoted 13 lines
Hi,
I am building netfilter queue application, that implement simple HTTP filter.
It basically will capture the packet, extract the URL, and issue MySQL lookup to see if the address is permitted or blocked.
My only fear that address lookup will block the traffic until I get the lookup reply, so I am thinking of using preforking model to fork 10-20 processes to handle the same queue.
Has any one done multi prcoess handling of the queue, is it even doable?
Rayed
If you were to do this with the queue lib, then you'd ataualy have to either let the client establish the connection OR fake that the connection was established, before you get the HTTP request is sent to rule on.
Also keep in mind that the servers/client might want to do a Keep-alive or long lived HTTP session, with multiple HTTP requests. Keeping track of that, from an NFNetlink/Queue based interface would be hecktic.
Quoted 23 lines
Quoted 5 lines
Unless there are a real lot of these addresses that are permitted, and they change rapidly, then I'd expect it would be a better design to get all the valid URIs from the MySQL query on init and store them in a linked list.
I am afraid this not an option, because my final goal is to integrate my application with external vendor database using the vendor own API.
Quoted 2 lines
Also, people already do this type of thing with an HTTP proxy like squid before the HTTPd.
I've tested HTTP proxy before, but the overhead is very big, and I couldn't get the proxy to handle high loads.
Another problem with proxy servers is using the proxy own IP for outgoing request, unfortunately using the same IP for many requests trigger many IDSes to block the proxy IP.
What I am trying to do is a light weight HTTP filter, by searching for HTTP requests in packet traveling on the wire, with minimal TCP reassembly to find HTTP requests spanning more than one packet.
Quoted 5 lines
If you were to do this with the queue lib, then you'd ataualy have to either let the client establish the connection OR fake that the connection was established, before you get the HTTP request is sent to rule on.
Of course! I want my application to be totally transparent from the client. The client will use regular browser without any proxy, and I won't inspect TCP handshaking packets, only packets that looks like HTTP request.
Quoted 4 lines
Also keep in mind that the servers/client might want to do a Keep-alive or long lived HTTP session, with multiple HTTP requests. Keeping track of that, from an NFNetlink/Queue based interface would be hecktic.
According to my tests and observations, most HTTP requests reside in a single packet, for request that span more than one request, I'll start tracking the session when I get a packet that looks like the beginning of HTTP request, and stop when I get "Host" header.
For example:
3rd Packet: "GET /ver_long_uri\r\n"
It looks like the start of HTTP request, Start tracking.
4th Packet: "Host: ad-ware.domain\r\n\r\n"
I have URI+HOST, Stop tracking.
In this case I won't have to keep track of the whole TCP session only what I need to get URI and HOST.
This will allow me to process more packets with minimal session tracking.
Quoted 2 lines
Unless there are a real lot of these addresses that are permitted, and they change rapidly, then I'd expect it would be a better design to get all the valid URIs from the MySQL query on init and store them in a linked list.
I am afraid this not an option, because my final goal is to integrate my application with external vendor database using the vendor own API.
Quoted 1 line
Also, people already do this type of thing with an HTTP proxy like squid before the HTTPd.
I've tested HTTP proxy before, but the overhead is very big, and I couldn't get the proxy to handle high loads.
Another problem with proxy servers is using the proxy own IP for outgoing request, unfortunately using the same IP for many requests trigger many IDSes to block the proxy IP.
What I am trying to do is a light weight HTTP filter, by searching for HTTP requests in packet traveling on the wire, with minimal TCP reassembly to find HTTP requests spanning more than one packet.