Limit bandwidth per-user (uid/gid)
Dear all,
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
What I want to do is:
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.
I played a bit with iptables and tc, but the only way I found to do something like this is to manually set a different mark for each user and then use tc, but I'd prefer a solution where there's no need to set up any rule manually if a user is added or removed. Also, the --uid-owner option works only for outbound traffic.
Do you have any suggestion?
I think that you understood the problem, so even if a different approach comes to your mind please let me know.
Thank you,
PL
You might have a look at cgroups and net_cls to set classid for all the processes a user owns/starts and then make a tc filter that matches the classid and sends the traffic to a suitable class with the type of sharing/limiting you wish.
Joseph
Quoted 41 lines · Paride Legovini
Dear all,
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
What I want to do is:
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.I played a bit with iptables and tc, but the only way I found to do something like this is to manually set a different mark for each user and then use tc, but I'd prefer a solution where there's no need to set up any rule manually if a user is added or removed. Also, the --uid-owner option works only for outbound traffic.
Do you have any suggestion?
I think that you understood the problem, so even if a different approach comes to your mind please let me know.Thank you,
PL
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Dear Paride,
that's a really exciting environment.
Am 22.12.2013 18:10 schrieb Paride Legovini:
Quoted 2 lines
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
Assuming that the satellite link is expensive and has long roundtrip times, it is probably a wise idea not to throw away any data which has already been sent over the satellite link.
Quoted 9 lines
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
What I want to do is:
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
Shaping outbound traffic is mostly a standard procedure with lots of documentation available. Nowadays you have some more options than a few years ago, and quite a few of them are easy to set up. I'll conveniently ignore outbound shaping in my response and talk about the hard stuff instead.
You can't limit inbound traffic to Antactica unless you do it on the other end of the satellite link (Europe). Well, you can limit the inbound bandwidth by throwing away packets locally, but that's stupid for packets which have already come over an expensive satellite link. Only the remote side decides how much bandwidth to send over the satellite link. What has been sent by the remote side will go over the satellite link and you can't undo this on the local side. What you can do in Antarctica is:
1. Perform the transfer with bandwidth-aware tools (rsync has the bwlimit option, but that is too crude to be useful).
2. Rely on crude implicit+indirect throttling of inbound traffic by shaping the bandwidth passed to local applications, which will cause a sort of backpressure (changed Ack behaviour) once the kernel buffers for your connection are full. Some people also recommend to throttle/police outbound Acks, but with current TCP congestion control mechanisms this can have unintended side effects.
Quoted 8 lines
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.
Quoted 5 lines
I played a bit with iptables and tc, but the only way I found to do something like this is to manually set a different mark for each user and then use tc, but I'd prefer a solution where there's no need to set up any rule manually if a user is added or removed. Also, the --uid-owner option works only for outbound traffic.
AFAICS even if you use net_cls as suggested by Joseph Santaniello, you're limited to managing outbound traffic (net_cls may indeed a cleaner approach than uid-owner, I haven't tried that). With CONNMARK, you should be able to tie the pieces together. I did such a setup (CONNMARK, not net_cls) for inbound+outbound throttling quite a few years ago, but sadly the code has been lost somewhere in the depths of decommissioned harddisks.
Quoted 3 lines
Do you have any suggestion?
I think that you understood the problem, so even if a different approach comes to your mind please let me know.
Good luck and please report back on how you did it.
Regards,
Carl-Daniel
Signature · 1 line
http://www.hailfinger.org/
Dear Joseph,
thanks for your suggestion, using cgroups is probably a better option than using --uid-owner. Still I'll have to setup a rule for each user, but I'm coming to the conclusion that I can't avoid that.
Paride
Quoted 53 lines · Joseph Santaniello
You might have a look at cgroups and
net_clsto set classid for all the processes a user owns/starts and then make a tc filter that matches the classid and sends the traffic to a suitable class with the type of sharing/limiting you wish.Joseph
Quoted 33 lines · Paride Legovini
Dear all,
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
What I want to do is:
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.I played a bit with iptables and tc, but the only way I found to do something like this is to manually set a different mark for each user and then use tc, but I'd prefer a solution where there's no need to set up any rule manually if a user is added or removed. Also, the --uid-owner option works only for outbound traffic.
Do you have any suggestion?
I think that you understood the problem, so even if a different approach comes to your mind please let me know.Quoted 8 lines
Thank you,
PL
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Dear Carl-Daniel,
thanks for your reply, it contains a lot of wise suggestions.
Quoted 11 lines · Carl-Daniel Hailfinger
Dear Paride,
that's a really exciting environment.
Am 22.12.2013 18:10 schrieb Paride Legovini:
Quoted 2 lines
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
Assuming that the satellite link is expensive and has long roundtrip times, it is probably a wise idea not to throw away any data which has already been sent over the satellite link.
You're perfectly right here: the link is expensive and the round trip is 700ms at best, but reaches easily 3000ms or even 4000ms when the network is congested.
Quoted 47 lines
Quoted 9 lines
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
What I want to do is:
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
Shaping outbound traffic is mostly a standard procedure with lots of documentation available. Nowadays you have some more options than a few years ago, and quite a few of them are easy to set up. I'll conveniently ignore outbound shaping in my response and talk about the hard stuff instead.
You can't limit inbound traffic to Antactica unless you do it on the other end of the satellite link (Europe). Well, you can limit the inbound bandwidth by throwing away packets locally, but that's stupid for packets which have already come over an expensive satellite link. Only the remote side decides how much bandwidth to send over the satellite link. What has been sent by the remote side will go over the satellite link and you can't undo this on the local side. What you can do in Antarctica is:
1. Perform the transfer with bandwidth-aware tools (rsync has the bwlimit option, but that is too crude to be useful).
2. Rely on crude implicit+indirect throttling of inbound traffic by shaping the bandwidth passed to local applications, which will cause a sort of backpressure (changed Ack behaviour) once the kernel buffers for your connection are full. Some people also recommend to throttle/police outbound Acks, but with current TCP congestion control mechanisms this can have unintended side effects.Quoted 8 lines
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.Quoted 5 lines
I played a bit with iptables and tc, but the only way I found to do something like this is to manually set a different mark for each user and then use tc, but I'd prefer a solution where there's no need to set up any rule manually if a user is added or removed. Also, the --uid-owner option works only for outbound traffic.
Quoted 7 lines
AFAICS even if you use
net_clsas suggested by Joseph Santaniello, you're limited to managing outbound traffic (net_clsmay indeed a cleaner approach than uid-owner, I haven't tried that). With CONNMARK, you should be able to tie the pieces together. I did such a setup (CONNMARK, notnet_cls) for inbound+outbound throttling quite a few years ago, but sadly the code has been lost somewhere in the depths of decommissioned harddisks.
If I understand correctly, this corresponds to your second suggestion regarding the inbound traffic throttling. The difficult thing here is that I not only have to mark a connection, but I have to mark all the connection started by a certain user (cgroup) in the same (known) way, so I can use tc on them.
I see that for inbound traffic shaping Intermediate Functional Block (ifb) devices can also be used, but I don't think it will help me to do a per-user (cgroup) throttling or shaping for the incoming traffic. This seems a difficult task.
I'll think about it, but at first I think I'll concentrate on the outgoing traffic.
Sunny greetings,
Paride
Hi,
Quoted 5 lines · Paride Legovini
Quoted 4 lines · Carl-Daniel Hailfinger
Am 22.12.2013 18:10 schrieb Paride Legovini:
Quoted 2 lines
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
Quoted 7 lines
Quoted 3 lines
Assuming that the satellite link is expensive and has long roundtrip times, it is probably a wise idea not to throw away any data which has already been sent over the satellite link.
You're perfectly right here: the link is expensive and the round trip is 700ms at best, but reaches easily 3000ms or even 4000ms when the network is congested.
Those long RTTs will get even longer if you add shaping of your own. Most software is written with much shorter RTTs in mind, triggering retransmits after one to three seconds without acknowledgement or reply.
One example I have seen in packet traces were three DNS replies to the same request, spaced 1 second apart, but altogether delivered 30 seconds after the first DNS request. This resulted in two retries of the DNS request and a subsequent failure, while delivering three full DNS request and reply pairs.
This kind of behavior can cripple goodput of the satellite link. You might want to keep this in mind when designing traffic shaping.
Erik
Quoted 16 lines · Erik Auerswald
Hi,
Quoted 5 lines · Paride Legovini
Quoted 4 lines · Carl-Daniel Hailfinger
Am 22.12.2013 18:10 schrieb Paride Legovini:
Quoted 2 lines
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
Quoted 7 lines
Quoted 3 lines
Assuming that the satellite link is expensive and has long roundtrip times, it is probably a wise idea not to throw away any data which has already been sent over the satellite link.
You're perfectly right here: the link is expensive and the round trip is 700ms at best, but reaches easily 3000ms or even 4000ms when the network is congested.
Quoted 7 lines
Those long RTTs will get even longer if you add shaping of your own. Most software is written with much shorter RTTs in mind, triggering retransmits after one to three seconds without acknowledgement or reply.
This kind of behavior can cripple goodput of the satellite link. You might want to keep this in mind when designing traffic shaping.
Dear Erik,
I agree that this is a very important point. The fact is that I don't work in the ICT team here (and I'm happy I don't!). I'm in charge for some scientific projects here and I somehow took over the task of developing a system to send data to the civilized world, as I want it to be done as I say. This means that I'm already behind a firewall, a commercial product which configuration if unknown to me.
That's it!
Paride
Quoted 87 lines · Paride Legovini
Dear Carl-Daniel,
thanks for your reply, it contains a lot of wise suggestions.
Quoted 11 lines · Carl-Daniel Hailfinger
Dear Paride,
that's a really exciting environment.
Am 22.12.2013 18:10 schrieb Paride Legovini:
Quoted 2 lines
I'm working in an Antarctic research station where our connection to the Internet is a 512kbps satellite link.
Assuming that the satellite link is expensive and has long roundtrip times, it is probably a wise idea not to throw away any data which has already been sent over the satellite link.
You're perfectly right here: the link is expensive and the round trip is 700ms at best, but reaches easily 3000ms or even 4000ms when the network is congested.
Quoted 47 lines
Quoted 9 lines
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
What I want to do is:
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
Shaping outbound traffic is mostly a standard procedure with lots of documentation available. Nowadays you have some more options than a few years ago, and quite a few of them are easy to set up. I'll conveniently ignore outbound shaping in my response and talk about the hard stuff instead.
You can't limit inbound traffic to Antactica unless you do it on the other end of the satellite link (Europe). Well, you can limit the inbound bandwidth by throwing away packets locally, but that's stupid for packets which have already come over an expensive satellite link. Only the remote side decides how much bandwidth to send over the satellite link. What has been sent by the remote side will go over the satellite link and you can't undo this on the local side. What you can do in Antarctica is:
1. Perform the transfer with bandwidth-aware tools (rsync has the bwlimit option, but that is too crude to be useful).
2. Rely on crude implicit+indirect throttling of inbound traffic by shaping the bandwidth passed to local applications, which will cause a sort of backpressure (changed Ack behaviour) once the kernel buffers for your connection are full. Some people also recommend to throttle/police outbound Acks, but with current TCP congestion control mechanisms this can have unintended side effects.Quoted 8 lines
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.Quoted 5 lines
I played a bit with iptables and tc, but the only way I found to do something like this is to manually set a different mark for each user and then use tc, but I'd prefer a solution where there's no need to set up any rule manually if a user is added or removed. Also, the --uid-owner option works only for outbound traffic.
Quoted 7 lines
AFAICS even if you use
net_clsas suggested by Joseph Santaniello, you're limited to managing outbound traffic (net_clsmay indeed a cleaner approach than uid-owner, I haven't tried that). With CONNMARK, you should be able to tie the pieces together. I did such a setup (CONNMARK, notnet_cls) for inbound+outbound throttling quite a few years ago, but sadly the code has been lost somewhere in the depths of decommissioned harddisks.If I understand correctly, this corresponds to your second suggestion regarding the inbound traffic throttling. The difficult thing here is that I not only have to mark a connection, but I have to mark all the connection started by a certain user (cgroup) in the same (known) way, so I can use tc on them.
I see that for inbound traffic shaping Intermediate Functional Block (ifb) devices can also be used, but I don't think it will help me to do a per-user (cgroup) throttling or shaping for the incoming traffic. This seems a difficult task.
Quoted 7 lines
I'll think about it, but at first I think I'll concentrate on the outgoing traffic.
Sunny greetings,
Paride
The 700-4000 latency might very well improve with throttled outbound.
I admit to very cursory knowledge of this, but as I understand the way tc works with outbound queues is that the ack rate is adjusted so the other side essentially sees it as a slower interface, and no packets are intentionally dropped unless the buffers get full.
Going the cgroups way, you could either start the cronjobs in some wrapper that shoves the tasks into the cgroup you want, or something like cgred to automatically by uid/gid/process move tasks into cgroups as they are started.
Or maybe tunneled IP to a machine back in "The World" that acts as a gateway and does the trottling for you?
Joseph
I saw a couple things go by on this thread that were interesting, so let me try to answer them all in one go.
Quoted 4 lines
I want to set up a server where each research project has an account where they send data via sftp or rsync; this data is then transferred overnight to a server in Europe. My idea is to use a separate cronjob or daemon for each user that runs with the user's privileges.
If you have control of both ends, you can run lossless by enabling ecn on your shaping code (RED, PIE, Codel and FQ_codel support ECN), and enabling tcp to use ecn on both sides via sysctl.conf
Rather than mucking with a network cgroup, you could just depriorize rsync (port 883) into a background htb class, so it will use a minimal amount of bandwidth by default and scale up to full bandwidth if its available. Then add a fq system on top of that to get fairness between users.
Quoted 7 lines
1. Limit the total bandwidth that a group (GID) can generate. There should be separate limits for inbound and outbound traffic.
Quoted 4 lines
You can't limit inbound traffic to Antactica unless you do it on the other end of the satellite link (Europe). Well, you can limit the inbound bandwidth by throwing away packets locally, but that's stupid for packets which have already come over an expensive satellite link.
Dropping packets is the only way that a non-ecn-enabled TCP can slow down, so although it is a "waste", it is a necessary one. Certainly this is a good use case for ecn in support of the science.
Quoted 3 lines
Quoted 3 lines
Only the remote side decides how much bandwidth to send over the satellite link. What has been sent by the remote side will go over the satellite link and you can't undo this on the local side.
If you wanted total control over the satellite link you could set it up as a tunnel from point A to point B and control matters better on both sides that way. You'd then shape the tunnel on the remote side to drop packets appropriately.
(the ecn suggestion is only going to work on endpoints you control)
Quoted 7 lines
2. Limit the bandwidth per-user (UID), so if the GID is allowed to generate 384kbps of traffic, and 3 users are using the network, each user can at most benefit of 128kbps. If there's only one user he gets all the 384kbps.
Again there should be different limits for inbound and outbound traffic.
This should work regardless the number of connections the user makes.
As noted elsewhere in this thread, a "user" is not a network object per se'. It does sound like you want per-user fairness not per flow fairness in your design (fq_codel and sfq are per flow by default). I think that is indeed doable with a tc filter for per user network cgroups on egress nowadays but haven't tried it.
On ingress, stuck on ways to do it.
Anyway, cerowrt's swm-scripts work on openwrt and cerowrt, can also be made to work on mainline linux,
and this is a decent start at a shaper for mainline linuxes:
https://wiki.gentoo.org/wiki/Traffic_shaping
In your case I'd probably set the fq_codel target at 60ms or so, and interval pretty high as well (700ms RTTs? really?), and enable ecn in both directions
I recently wrote a rant on everything wrong with wondershaper.
http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
Signature · 1 line
Dave Täht
Quoted 2 lines
In your case I'd probably set the
fq_codeltarget at 60ms or so, and interval pretty high as well (700ms RTTs? really?), and enable ecn in both directions
Well, if their satellite link is via a geosynchronous satellite, it is something like 22,236 miles above the equator [1]. One RTT that way will be 4x that distance. However, since they are in Antarctica it will be a bit farther. The diameter of the Earth is something like 7918 miles (Google). So, if we ass-u-me they are actually at the pole, we can add 1/2 that (3959 miles) to the 22,236 miles and call that one leg of a right triangle with the right angle at the center of the Earth, the other leg then is 3959 miles from the center of the Earth to the pole, and we can use good old Pythagoras to get the length of the hypotenuse, which will be the distance from the pole to the satellite.
Distance^2 = 3959^2 + (22236 + 3959)^2
Distance^2 = 15673681 + 686178025
Distance^2 = 701851706
Distance = 26492 miles
4x that is just shy of 105969 miles, which when divided by 186,000 miles per second yields 570 milliseconds. Of course that assumes one is talking pole to pole which is unlikely, but it does get us a good chunk of the way to 700 milliseconds.
I think the initial posting said 512 kbit/s for the link speed, so if we further assume Ye Olde 64 Byte Ping that will be 64 + 20 + 14 bytes or 784 bits. At 512000 bits/s that would seem to be another 1.5 milliseconds for each time it gets transmitted or 6.125 milliseconds all told taking us to 576. That assumes 512 Kbit/s is after FEC and such and that this is all store-and-forward. And probably assumes other things about the transmission but I've no idea what they would be.
If it is actually something like Iridium then it gets more complicated - there will be some average number of satellite to satellite hops on the way to/from the Iridium base station.
And of course there is whatever the RTT's are from the satellite base station to the actual destination(s). I could see that easily adding another 100 milliseconds but that is just handwaving.
rick jones
[1] http://en.wikipedia.org/wiki/Geostationary_orbit
There are probably some papers out there on Iriduim performance one could find via a web search