Skip to contentSkip to conversation

How to fight with encrypted p2p

LARTC sAwAr 13 participants 18 messages Nov 12, 2007 – Dec 10, 2007

lore
Share & tools
lore

sAwAr

author
#0 lore raw

Hi

I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

Thanks in advance.

Pozdrawiam
Szymon Turkiewicz

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#1 lore raw
Quoted 8 lines · sAwAr

Hi

I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

Quoted 4 lines

Thanks in advance.

Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#2 lore raw
In reply to sAwAr

Some clients P2P clients are nice about there encryption and negotiate encryption ahead of time using plain communication. I.E. Limewire, Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to believe they track that kind of info.

David Bierce

Quoted 14 lines · Mohan Sundaram
Quoted 9 lines · sAwAr

Hi
I believe that whole question is in topic. Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Quoted 5 lines

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

author
#3 lore raw
In reply to sAwAr

Rtorrent which I use sometimes have ability to completely disable plain text communication :

man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable unencrypted handshakes), require_RC4 (also disable plaintext transmission after the initial encrypted handshake), enable_retry (if the initial outgoing connection fails, retry with encryption turned on if it was off or off if it was on), prefer_plain text (choose plaintext when peer offers a choice between plaintext transmission and RC4 encryption, otherwise RC4 will be used).

and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a matter of time and we will lose this "fight" very soon.

Quoted 33 lines

Some clients P2P clients are nice about there encryption and negotiate encryption ahead of time using plain communication. I.E. Limewire, Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to believe they track that kind of info.

David Bierce

Quoted 15 lines · Mohan Sundaram
Quoted 9 lines · sAwAr

Hi
I believe that whole question is in topic. Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Quoted 5 lines

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 4 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#4 lore raw
In reply to sAwAr

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines

Hi

I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

Thanks in advance.

As far as I am concerned there is no way to recognize encrypted p2p. Ive tested ipp2p, and layer7, and also Mikrotiks 3.0 p2p mark systems [ which in fact uses modified layer7 ], and none of them cant recognize this traffic simply because it is enctypted.

- --

[ ... and the Slackware for All... ]

gpg_key: http://szuwar.mayhem.net.pl/keys/szuwar_mayhem-public.gpg

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)

iD8DBQFHOZF96pLYJHrOtfoRAhxNAKCoOEqujQ8ibpeKKLcy4ODHXQmgAgCgh8Ur k+pLmHu2JzhZmM1mtxD9pL4=DKs2
-----END PGP SIGNATURE-----
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#5 lore raw
In reply to sAwAr
Quoted 4 lines · sAwAr

Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

Does this mean that we are down to handling traffic based on the sustained stream(s)? I.e. how long the streams have been active, how many packets per second, how many streams a given end point has, speed of traffic, average size of packets?

Encrypted or not, I believe all traffic can be somewhat recognized by its usage pattern(s). However there may be more false positives. We may end up recognizing what we know as good and putting the rest at a lower class of service.

Thought's / comments / objections / flame wars?

Grant. . . .
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#6 lore raw
In reply to sAwAr
Quoted 11 lines · Grant Taylor
Quoted 4 lines · sAwAr

Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

Does this mean that we are down to handling traffic based on the sustained stream(s)? I.e. how long the streams have been active, how many packets per second, how many streams a given end point has, speed of traffic, average size of packets?

Quoted 4 lines

Encrypted or not, I believe all traffic can be somewhat recognized by its usage pattern(s). However there may be more false positives. We may end up recognizing what we know as good and putting the rest at a lower class of service.

Well, you can surely try. But then again, I have been doing research (publication pending) in traffic-pattern-based detection of VoIP flows and peer-to-peer connections. While it usually is easy to find a pattern matching your particular traffic class very well, part of this research has been dedicated to automatically circumvent these systems. Why that? Simple. Application evolve to circumvent detection. If you can simulate that evolution in the lab, you can find out where your detection algorithms will fail. Of course, that enumeration of possible failure modes is non-exhaustive.

Bottom line: This is an arms race. Unless you do lots of research and testing, detection will always be trying to catch up. If detection manages to catch up, circumvention will advance, but you may have a small time window where you can enjoy the "win". However, winning becomes more and more expensive. Clients can expend considerable amount of CPU power to avoid detection. You don't have that luxury in filter systems unless you have one filter per client.

Regards,
Carl-Daniel
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#7 lore raw
In reply to sAwAr
Quoted 10 lines · Carl-Daniel Hailfinger

Well, you can surely try. But then again, I have been doing research (publication pending) in traffic-pattern-based detection of VoIP flows and peer-to-peer connections. While it usually is easy to find a pattern matching your particular traffic class very well, part of this research has been dedicated to automatically circumvent these systems. Why that? Simple. Application evolve to circumvent detection. If you can simulate that evolution in the lab, you can find out where your detection algorithms will fail. Of course, that enumeration of possible failure modes is non-exhaustive.

Quoted 7 lines

Bottom line: This is an arms race. Unless you do lots of research and testing, detection will always be trying to catch up. If detection manages to catch up, circumvention will advance, but you may have a small time window where you can enjoy the "win". However, winning becomes more and more expensive. Clients can expend considerable amount of CPU power to avoid detection. You don't have that luxury in filter systems unless you have one filter per client.

All very good points with regard to pattern based detecting P2P (and the likes) traffic. What do you think about recognizing the traffic you do want and treating all else as a second or third class citizen. Or is this just a form of net neutrality? Or really is this entire discussion such. Further does the net neutrality issue apply to companies (read:
non ISPs) wanting to filter their own internal traffic.

Additionally as an aside will you please provide more information on your pending publication? I'd likely be curious to read (what ever) when ever it is published. Thanks in advance.

Grant. . . .
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#8 lore raw
In reply to sAwAr

As you might have seen, these are words from ipp2p author:

"""

I have seen some pieces of code from ipoque which can detect encypted bittorrent and edonkey traffic. Unforunately, this code will not work with iptables, because it needs
more information about the flow history and the history of an ip address.

Right now, I do not have the time and the money to develop a filter like this, but
if you are interested in a developement in this direction, please contact me.

"""

I *think* that we need something like a "bittorrent helper" in the kernel to keep this extra information about the flow history and then an iptables plugin to match. What do you think? Maybe we could contact him to know what kind of information is it?

Quoted 60 lines · sawar

Rtorrent which I use sometimes have ability to completely disable plain text communication :

man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable unencrypted handshakes), require_RC4 (also disable plaintext transmission after the initial encrypted handshake), enable_retry (if the initial outgoing connection fails, retry with encryption turned on if it was off or off if it was on), prefer_plain text (choose plaintext when peer offers a choice between plaintext transmission and RC4 encryption, otherwise RC4 will be used).

and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a matter of time and we will lose this "fight" very soon.

Quoted 33 lines

Some clients P2P clients are nice about there encryption and negotiate encryption ahead of time using plain communication. I.E. Limewire, Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to believe they track that kind of info.

David Bierce

Quoted 15 lines · Mohan Sundaram
Quoted 9 lines · sAwAr

Hi
I believe that whole question is in topic. Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Quoted 5 lines

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 4 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Signature · 7 lines
Marco Casaroli
SapucaiNet Telecom
+55 35 34712377 ext. 5
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#9 lore raw
In reply to sAwAr

About ipp2p,

Right now, the battle against p2p is lost with l7 detection from ipp2p, l7 filter and others.

Why ?? It is a known fact that pattern matching does not work with full encrypted P2P handshakes based on DHT key exchange algorithms with byte padding. You have absolutely no byte pattern and no fixed packet lengths in the stream. So something like a flow history will fail or might have a very high false +ve rate.

The thing is that there are proprietary solutions which can detect fully encrypted p2p streams based on a heuristic approach. (AFAIK ipoque is selling a proprietary library for this which is integrated in some firewall vendors). I have not seen any open source development into this direction.

Klaus, (former) maintainer of ipp2p

Quoted 15 lines · Marco Aurelio

As you might have seen, these are words from ipp2p author:

"""

I have seen some pieces of code from ipoque which can detect encypted bittorrent and edonkey traffic. Unforunately, this code will not work with iptables, because it needs
more information about the flow history and the history of an ip address.

Right now, I do not have the time and the money to develop a filter like this, but
if you are interested in a developement in this direction, please contact me.

"""

Quoted 68 lines

I *think* that we need something like a "bittorrent helper" in the kernel to keep this extra information about the flow history and then an iptables plugin to match. What do you think? Maybe we could contact him to know what kind of information is it?

Quoted 58 lines · sawar

Rtorrent which I use sometimes have ability to completely disable plain text communication :

man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable unencrypted handshakes), require_RC4 (also disable plaintext transmission after the initial encrypted handshake), enable_retry (if the initial outgoing connection fails, retry with encryption turned on if it was off or off if it was on), prefer_plain text (choose plaintext when peer offers a choice between plaintext transmission and RC4 encryption, otherwise RC4 will be used).

and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a matter of time and we will lose this "fight" very soon.

Quoted 31 lines

Some clients P2P clients are nice about there encryption and negotiate encryption ahead of time using plain communication. I.E. Limewire, Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to believe they track that kind of info.

David Bierce

Quoted 14 lines · Mohan Sundaram
Quoted 9 lines · sAwAr

Hi
I believe that whole question is in topic. Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Quoted 5 lines

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 4 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#10 lore raw
In reply to sAwAr

Klauss,

Could you
Might be you can sponsor the development ...

Regards

Sébastien

Quoted 112 lines · Klaus

About ipp2p,

Right now, the battle against p2p is lost with l7 detection from ipp2p, l7 filter and others.

Why ?? It is a known fact that pattern matching does not work with full encrypted P2P handshakes based on DHT key exchange algorithms with byte padding. You have absolutely no byte pattern and no fixed packet lengths in the stream. So something like a flow history will fail or might have a very high false +ve rate.

The thing is that there are proprietary solutions which can detect fully encrypted p2p streams based on a heuristic approach. (AFAIK ipoque is selling a proprietary library for this which is integrated in some firewall vendors). I have not seen any open source development into this direction.

Klaus, (former) maintainer of ipp2p

Quoted 15 lines · Marco Aurelio

As you might have seen, these are words from ipp2p author:

"""

I have seen some pieces of code from ipoque which can detect encypted bittorrent and edonkey traffic. Unforunately, this code will not work with iptables, because it needs
more information about the flow history and the history of an ip address.

Right now, I do not have the time and the money to develop a filter like this, but
if you are interested in a developement in this direction, please contact me.

"""

Quoted 75 lines

I *think* that we need something like a "bittorrent helper" in the kernel to keep this extra information about the flow history and then an iptables plugin to match. What do you think? Maybe we could contact him to know what kind of information is it?

Quoted 65 lines · sawar

Rtorrent which I use sometimes have ability to completely disable plain text communication :

man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable unencrypted handshakes), require_RC4 (also disable plaintext transmission after the initial encrypted handshake), enable_retry (if the initial outgoing connection fails, retry with encryption turned on if it was off or off if it was on), prefer_plain text (choose plaintext when peer offers a choice between plaintext transmission and RC4 encryption, otherwise RC4 will be used).

and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a matter of time and we will lose this "fight" very soon.

Quoted 35 lines

Some clients P2P clients are nice about there encryption and negotiate encryption ahead of time using plain communication. I.E. Limewire, Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to believe they track that kind of info.

David Bierce

Quoted 16 lines · Mohan Sundaram
Quoted 10 lines · sAwAr

Hi
I believe that whole question is in topic. Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Quoted 6 lines

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 5 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 7 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#11 lore raw
In reply to sAwAr

Sorry ... I'm little bite tired ...
I mean that we might sponsor Klauss and L7 team to develop this ...

Regards

Quoted 142 lines · Sébastien CRAMATTE

Klauss,

Could you
Might be you can sponsor the development ...

Regards

Sébastien

Quoted 121 lines · Klaus

About ipp2p,

Right now, the battle against p2p is lost with l7 detection from ipp2p, l7 filter and others.

Why ?? It is a known fact that pattern matching does not work with full encrypted P2P handshakes based on DHT key exchange algorithms with byte padding. You have absolutely no byte pattern and no fixed packet lengths in the stream. So something like a flow history will fail or might have a very high false +ve rate.

The thing is that there are proprietary solutions which can detect fully encrypted p2p streams based on a heuristic approach. (AFAIK ipoque is selling a proprietary library for this which is integrated in some firewall vendors). I have not seen any open source development into this direction.

Klaus, (former) maintainer of ipp2p

Quoted 15 lines · Marco Aurelio

As you might have seen, these are words from ipp2p author:

"""

I have seen some pieces of code from ipoque which can detect encypted bittorrent and edonkey traffic. Unforunately, this code will not work with iptables, because it needs
more information about the flow history and the history of an ip address.

Right now, I do not have the time and the money to develop a filter like this, but
if you are interested in a developement in this direction, please contact me.

"""

Quoted 83 lines

I *think* that we need something like a "bittorrent helper" in the kernel to keep this extra information about the flow history and then an iptables plugin to match. What do you think? Maybe we could contact him to know what kind of information is it?

Quoted 72 lines · sawar

Rtorrent which I use sometimes have ability to completely disable plain text communication :

man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable unencrypted handshakes), require_RC4 (also disable plaintext transmission after the initial encrypted handshake), enable_retry (if the initial outgoing connection fails, retry with encryption turned on if it was off or off if it was on), prefer_plain text (choose plaintext when peer offers a choice between plaintext transmission and RC4 encryption, otherwise RC4 will be used).

and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a matter of time and we will lose this "fight" very soon.

Quoted 39 lines

Some clients P2P clients are nice about there encryption and negotiate encryption ahead of time using plain communication. I.E. Limewire, Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to believe they track that kind of info.

David Bierce

Quoted 18 lines · Mohan Sundaram
Quoted 11 lines · sAwAr

Hi
I believe that whole question is in topic. Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted AFAIK. Thus connections can be marked and related traffic shaped. If initiation is also encrypted, then I think we have a serious problem.

Quoted 7 lines

Mohan
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 6 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 8 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 7 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#12 lore raw
In reply to sAwAr
Quoted 4 lines

I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

One way to do this is to look for the style of traffic. For example, I look for lots of connections from one PC to port numbers above 1024. This will also incorrectly recognise some other traffic, but on the whole it works well for me.

The following are some examples using connlimit (now included in vanilla kernel) and ipset (see http://ipset.netfilter.org/)

# first look for style of traffic and log that client to an ipset
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
# then shape traffic above port 1024 for those detected clients
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60

Regards,

Andy Beverley

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#13 lore raw
In reply to sAwAr

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I believe "fighting" is the wrong approach. Badly shaping the wrong traffic is just as bad, if not worse IMO. An ISP in my neck of the woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a result of their P2P fight. Needless to say we no longer use them, and we encourage clients, friends, and colleagues not to as well. I don't use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with these services is not useful. Limiting your rules to specific ports is pretty useless. This has been done before, and it failed miserably.

For me, if P2P does not belong at all, for instance on a corporate network, then a default deny on the outbound works much better. We then
only allow specific connections on a case by case basis. For instances
where I am not able to block p2p, I define specific rules for high and low priority, and leave everything else in the default. If the end user wants to use the bulk of his or her bandwidth for P2P, so be it. Of course in this case bandwidth accounting is far more useful.

I would also encourage your users to use software that is or can be well behaved. Software that allows you set a proper TOS for instance. If possible work with the end users.

I have personally found that the best solutions are not tech solutions.
Having a well defined Acceptable Use Policy, plus a constructive
dialogue with my users has been far more effective than any shaping routine I/we could come up with.

Ask yourself, what is the problem you are really trying to solve.

Quoted 36 lines · Andrew Beverley
Quoted 4 lines

I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted? Modern p2p clients have this ability moreover some of them have this enabled by default. Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

One way to do this is to look for the style of traffic. For example, I look for lots of connections from one PC to port numbers above 1024. This will also incorrectly recognise some other traffic, but on the whole it works well for me.

The following are some examples using connlimit (now included in vanilla kernel) and ipset (see http://ipset.netfilter.org/)

# first look for style of traffic and log that client to an ipset
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
	-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
# then shape traffic above port 1024 for those detected clients
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
	-m set --set p2p dst -j MARK --set-mark 60

Regards,

Quoted 7 lines

Andy Beverley

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHU98owRXgH3rKGfMRAmszAKCIhVoUnfuWDIaWQqwE1WfuSz9sNwCgipFZ wqrptNaNg3HMFE79AvbQ+fI=gb3i
-----END PGP SIGNATURE-----
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#14 lore raw
In reply to sAwAr
Quoted 7 lines

I believe "fighting" is the wrong approach. Badly shaping the wrong traffic is just as bad, if not worse IMO. An ISP in my neck of the woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a result of their P2P fight. Needless to say we no longer use them, and we encourage clients, friends, and colleagues not to as well. I don't use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with these services is not useful.

Using the rules in the example previously given specifically steers well clear of these services.

Quoted 2 lines

Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Quoted 3 lines

For me, if P2P does not belong at all, for instance on a corporate network, then a default deny on the outbound works much better. We then only allow specific connections on a case by case basis.

I have seen this work very well on corporate networks, and would recommend this
approach where possible. Unfortunately though, on a normal home user network, there are so many different possibilities that this isn't very practical.

Quoted 5 lines

For instances
where I am not able to block p2p, I define specific rules for high and low priority, and leave everything else in the default. If the end user wants to use the bulk of his or her bandwidth for P2P, so be it. Of course in this case bandwidth accounting is far more useful.

Again, this depends on the circumstances. If you only have 2Mbit/s to share between 100 users then each user cannot have their own 'share' of the connection. Equally, people downloading in a responsible way are lumped into the
same category as p2p users, which is not fair. Bandwidth accounting is a possibility, and something I haven't investigated.

For those who want to fairly share bandwidth beween users, I would recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis, rather than per connection. This prevents the hundreds of p2p connections from drowning out single downloads.

Quoted 7 lines

I would also encourage your users to use software that is or can be well behaved. Software that allows you set a proper TOS for instance. If possible work with the end users.
I have personally found that the best solutions are not tech solutions. Having a well defined Acceptable Use Policy, plus a constructive dialogue with my users has been far more effective than any shaping routine I/we could come up with.

Agreed. However, in a situation where you have a lot of users coming and going,
it is not easy to educate the many hundreds of users.

I guess it all boils down to your own situation. Traffic shaping on a corporate
network or on a network where your users are static can be done using the above
techniques. However, sharing a small connection between hundreds of regularly changing users is difficult, and I have found the 'blunt' rules previously described to work very well with no complaints.

Regards,

Andy Beverley

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#15 lore raw
In reply to sAwAr

Hi,

We had similiar problem with p2p, used ipp2p and L7filter together before and worked well until clients( mostly clever ones) started getting around it with encryption. We have about 700 wireless clients hitting our network and our network was taking big knocks with guys using couple of gigs day on entry level packages.

Was going to use Ipoque, but was quite pricy for us, Only solutions for us to use a daily limit of eg 500MB, then they get slowed down to slower speeds, This worked like a charm

Out of interest we used freeradius / pptpd|pppd with some custom perl scripts and tc rules

Sew

Quoted 67 lines · Andrew Beverley
Quoted 7 lines

I believe "fighting" is the wrong approach. Badly shaping the wrong traffic is just as bad, if not worse IMO. An ISP in my neck of the woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a result of their P2P fight. Needless to say we no longer use them, and we encourage clients, friends, and colleagues not to as well. I don't use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with these services is not useful.

Using the rules in the example previously given specifically steers well clear of these services.

Quoted 2 lines

Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Quoted 3 lines

For me, if P2P does not belong at all, for instance on a corporate network, then a default deny on the outbound works much better. We then only allow specific connections on a case by case basis.

I have seen this work very well on corporate networks, and would recommend this
approach where possible. Unfortunately though, on a normal home user network, there are so many different possibilities that this isn't very practical.

Quoted 5 lines

For instances
where I am not able to block p2p, I define specific rules for high and low priority, and leave everything else in the default. If the end user wants to use the bulk of his or her bandwidth for P2P, so be it. Of course in this case bandwidth accounting is far more useful.

Again, this depends on the circumstances. If you only have 2Mbit/s to share between 100 users then each user cannot have their own 'share' of the connection. Equally, people downloading in a responsible way are lumped into the
same category as p2p users, which is not fair. Bandwidth accounting is a possibility, and something I haven't investigated.

For those who want to fairly share bandwidth beween users, I would recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis, rather than per connection. This prevents the hundreds of p2p connections from drowning out single downloads.

Quoted 7 lines

I would also encourage your users to use software that is or can be well behaved. Software that allows you set a proper TOS for instance. If possible work with the end users.
I have personally found that the best solutions are not tech solutions. Having a well defined Acceptable Use Policy, plus a constructive dialogue with my users has been far more effective than any shaping routine I/we could come up with.

Agreed. However, in a situation where you have a lot of users coming and going,
it is not easy to educate the many hundreds of users.

I guess it all boils down to your own situation. Traffic shaping on a corporate
network or on a network where your users are static can be done using the above
techniques. However, sharing a small connection between hundreds of regularly changing users is difficult, and I have found the 'blunt' rules previously described to work very well with no complaints.

Regards,

Quoted 8 lines

Andy Beverley

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#16 lore raw
In reply to sAwAr

Thanks for sharing.

Just one question, how are you implementing the daily limit?

Regards,

Mario Antonio

----- Original Message -----
From: "the sew" <[email protected]>
To: "Andrew Beverley" <[email protected]>
Cc: [email protected]
Sent: Monday, December 10, 2007 8:37:07 AM (GMT-0500) America/New_York
Subject: Re: [LARTC] How to fight with encrypted p2p

Hi,

We had similiar problem with p2p, used ipp2p and L7filter together before and worked well until clients( mostly clever ones) started getting around it with encryption. We have about 700 wireless clients hitting our network and our network was taking big knocks with guys using couple of gigs day on entry level packages.

Was going to use Ipoque, but was quite pricy for us, Only solutions for us to use a daily limit of eg 500MB, then they get slowed down to slower speeds, This worked like a charm

Out of interest we used freeradius / pptpd|pppd with some custom perl scripts and tc rules

Sew

Quoted 67 lines · Andrew Beverley
Quoted 7 lines

I believe "fighting" is the wrong approach. Badly shaping the wrong traffic is just as bad, if not worse IMO. An ISP in my neck of the woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a result of their P2P fight. Needless to say we no longer use them, and we encourage clients, friends, and colleagues not to as well. I don't use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with these services is not useful.

Using the rules in the example previously given specifically steers well clear of these services.

Quoted 2 lines

Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Quoted 3 lines

For me, if P2P does not belong at all, for instance on a corporate network, then a default deny on the outbound works much better. We then only allow specific connections on a case by case basis.

I have seen this work very well on corporate networks, and would recommend this
approach where possible. Unfortunately though, on a normal home user network, there are so many different possibilities that this isn't very practical.

Quoted 5 lines

For instances
where I am not able to block p2p, I define specific rules for high and low priority, and leave everything else in the default. If the end user wants to use the bulk of his or her bandwidth for P2P, so be it. Of course in this case bandwidth accounting is far more useful.

Again, this depends on the circumstances. If you only have 2Mbit/s to share between 100 users then each user cannot have their own 'share' of the connection. Equally, people downloading in a responsible way are lumped into the
same category as p2p users, which is not fair. Bandwidth accounting is a possibility, and something I haven't investigated.

For those who want to fairly share bandwidth beween users, I would recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis, rather than per connection. This prevents the hundreds of p2p connections from drowning out single downloads.

Quoted 7 lines

I would also encourage your users to use software that is or can be well behaved. Software that allows you set a proper TOS for instance. If possible work with the end users.
I have personally found that the best solutions are not tech solutions. Having a well defined Acceptable Use Policy, plus a constructive dialogue with my users has been far more effective than any shaping routine I/we could come up with.

Agreed. However, in a situation where you have a lot of users coming and going,
it is not easy to educate the many hundreds of users.

I guess it all boils down to your own situation. Traffic shaping on a corporate
network or on a network where your users are static can be done using the above
techniques. However, sharing a small connection between hundreds of regularly changing users is difficult, and I have found the 'blunt' rules previously described to work very well with no complaints.

Regards,

Quoted 8 lines

Andy Beverley

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#17 lore raw
In reply to sAwAr
Quoted 1 line · Mario Antonio Garcia

Thanks for sharing.

No problem

Quoted 2 lines

Just one question, how are you implementing the daily limit?

two ways of detecting them:

1st is the /etc/ppp/ip-up.local which executes the script to check usage against the radius db and shape them on authentication,

Once they logged in , I dont want to kick everone off every few hours to check usage, I have a "nice" cron job running every 3 hours, to check every single user against the db and if they reached their quotas ,they get shaped while being online., Radius stores all info about the nas in the db, so makes it quite strait forward.

Also nicely added is our reselling guys who maintain the clients get a report everytime of users who exceeded the limit and can be aware of which clients is the problem if they phone to complain, Also nice is that usually the infected pc's gets knocked off first to save alot of bandwidth

Sew

Quoted 106 lines

Regards,

Mario Antonio

----- Original Message -----
From: "the sew" <[email protected]>
To: "Andrew Beverley" <[email protected]>
Cc: [email protected]
Sent: Monday, December 10, 2007 8:37:07 AM (GMT-0500) America/New_York
Subject: Re: [LARTC] How to fight with encrypted p2p

Hi,

We had similiar problem with p2p, used ipp2p and L7filter together before and worked well until clients( mostly clever ones) started getting around it with encryption. We have about 700 wireless clients hitting our network and our network was taking big knocks with guys using couple of gigs day on entry level packages.

Was going to use Ipoque, but was quite pricy for us, Only solutions for us to use a daily limit of eg 500MB, then they get slowed down to slower speeds, This worked like a charm

Out of interest we used freeradius / pptpd|pppd with some custom perl scripts and tc rules

Sew

Quoted 67 lines · Andrew Beverley
Quoted 7 lines

I believe "fighting" is the wrong approach. Badly shaping the wrong traffic is just as bad, if not worse IMO. An ISP in my neck of the woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a result of their P2P fight. Needless to say we no longer use them, and we encourage clients, friends, and colleagues not to as well. I don't use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with these services is not useful.

Using the rules in the example previously given specifically steers well clear of these services.

Quoted 2 lines

Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Quoted 3 lines

For me, if P2P does not belong at all, for instance on a corporate network, then a default deny on the outbound works much better. We then only allow specific connections on a case by case basis.

I have seen this work very well on corporate networks, and would recommend this
approach where possible. Unfortunately though, on a normal home user network, there are so many different possibilities that this isn't very practical.

Quoted 5 lines

For instances
where I am not able to block p2p, I define specific rules for high and low priority, and leave everything else in the default. If the end user wants to use the bulk of his or her bandwidth for P2P, so be it. Of course in this case bandwidth accounting is far more useful.

Again, this depends on the circumstances. If you only have 2Mbit/s to share between 100 users then each user cannot have their own 'share' of the connection. Equally, people downloading in a responsible way are lumped into the
same category as p2p users, which is not fair. Bandwidth accounting is a possibility, and something I haven't investigated.

For those who want to fairly share bandwidth beween users, I would recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis, rather than per connection. This prevents the hundreds of p2p connections from drowning out single downloads.

Quoted 7 lines

I would also encourage your users to use software that is or can be well behaved. Software that allows you set a proper TOS for instance. If possible work with the end users.
I have personally found that the best solutions are not tech solutions. Having a well defined Acceptable Use Policy, plus a constructive dialogue with my users has been far more effective than any shaping routine I/we could come up with.

Agreed. However, in a situation where you have a lot of users coming and going,
it is not easy to educate the many hundreds of users.

I guess it all boils down to your own situation. Traffic shaping on a corporate
network or on a network where your users are static can be done using the above
techniques. However, sharing a small connection between hundreds of regularly changing users is difficult, and I have found the 'blunt' rules previously described to work very well with no complaints.

Regards,

Quoted 8 lines

Andy Beverley

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoted 9 lines

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc