New IMQ device implementation supporting device EOS
Hello All
Its first time i posting to this mail list :). I have done something (maybe) important. I write new IMQ device driver based from Martin Devera and Patrick McHardy implementation with device EOS support. My current implementation supporting only egress trafic shaping and kernel 2.4.25. For more details, source and examples have look at my page
http://hyperfighter.jinak.cz/qos
Ill apreticate your feedback :)
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello Roy
Quoted 4 lines · Roy
Hello,
imq became realy popular now ;)
Sure, same as whole GNU/Linux :)
Quoted 1 line
It would be good ir you wrote what is that EOS to save time for some
people
Quoted 4 lines
on searches.
EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation. Any of current IMQ implementations do not use it, so the calculation by qdisc may not be correct, whish causing filling physical device fifos without control. In devices with variable speed (WiFi, Modems) it causes overload because the physisal trafics is out of control. It happen only on busy boxes, like my routers where is about 70 ppl at once. In a simply test enviroment this issue will never happen.
Quoted 1 line
and it would be good if you posted source code directly (not inside of
diff
Quoted 5 lines
file)
for review purposes
The whole patch contain only few new files, all other are patches of the original kernel files. BTW Midnight Comander 4.6.0 contain very nice diff virtual filesystem, so you can browse .diff files like a tar archive.
Quoted 7 lines
also you can look at my imq version, because it is much easer to develop since is independent from kernel.
and does not need to patch it.
http://pupa.da.ru/imq/
I have play with with your version :) Having independet implementation is nice idea. I play with that idea too but it was more dificult to have stable solution in short time i need and with EOS support its quite imposible ...
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 1 line · Roy
this is qute intersting, and it happens on every computer not only on
bussy
Quoted 2 lines
ones
but in reality I dont see any need for that, because tcp itself should
take
Quoted 2 lines
care of this issue.
Right but you lost TCP interactivity and overloaded WiFi interface is painfull itself !!!
Quoted 3 lines
anyway this may solve some problems with crashes, because I suspect imq is unstable because of this problem.
Stability if any current implementations is issue by design. Simply something betwen netfilter and TCP/IP stack. I tried found it but my suceess was only more crashes.
Quoted 1 line
I looked at yor code and do not understand what it does, do you
implemented
Quoted 2 lines
that EOS for imq interface ifself?
this is not logical, since imq is not interface, it is placeholder to
attach
Quoted 7 lines
shapers.
Or you made that your imq takes packets from kernel ant transmits them by itself without returning to netfilter
withdev_queue_xmit(skb)
this way you are very limited in what you are able to do because you can hook only on the last hook.
There is skb->imq_flags whis getting marked by netfilter to IMQ_F_ENQUEUE (part of flag is target IMQ network interface) At "end of life in kernel" every packet coming into dev_queue_xmit. There also come our marked skb. When you have look at begin of the modified dev_queue_xmit
there is imq_flags decoder. When decoder hit our marked SKB its enqueued in to IMQ queue. Then we run qdisc dequeue (qdisc_run). qdisc_run function is a loop tat run qdisc restart until is queue empty, need to be schedulet (netif_schedule) or until is not set netif_queue_stopped atomic bit (bingo the trick is here). After dequeue in qdisc restart our skb passing hard_start_xmit function in IMQ driver. After some checkings is there stopped queue of the imq device (netif_queue_stopped), changet skb->imq_flags to IMQ_F_ENQUEUED and passed back to dev_queue_xmit. Now it will hit our real device, and at end of the qdisc_restart (after hard_start_xmit) is function that decode the skb->imq_flags and if is necesary wake_up the queue of our IMQ device (so we have perfect synchronization of the EOS). There is some protection to lock up the queue in imq_dev_xmit (imq.c). Also there are counted throtled skb by collision counter. I also use device watchdog to solve problem if skb stuck somewhere in physical device queue and netif_wake_queue is not called in right time( its only protection "for sure" because this should happen and if this happen without protection behind this, whole imq should lockup and kill the trafics forewer)
Quoted 3 lines
seems you may be unable shape ingress then. what basicaly voids all imq purpose.
Iam not sure. Every packet comin in in to router must come out !!! So we need only shape out trafics. Incoming trafics we cannot shape, only police. We cannot change speed and rate of incoming packets !!! have look at example includet in my IMQ. There i use the corret way to shape incoming and outgoung traffic. BTW: I have writen a TODO ....
Quoted 3 lines
unfortunately you made your driver even more invasive than old one, it need to patch almost all kernel network system.
Yes its also angle of view :) I was need to solve problem and i solve it. Everithing there is clear. IMHO Cannot be solved by more clear solution. ;)
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 1 line · Roy
Quoted 1 line
Quoted 1 line
seems you may be unable shape ingress then. what basicaly voids
all imq
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
purpose.
Iam not sure. Every packet comin in in to router must come out !!! So we need only shape out trafics. Incoming trafics we cannot shape, only police. We cannot change speed and rate of incoming packets !!! have look at example includet in my IMQ. There i use the corret way to shape incoming and outgoung traffic. BTW: I have writen a TODO ....
Quoted 5 lines
Here you are completely wrong what makes you think that you cant control incoming trafic?
some wrong publication on internet?
Almost everybody is doing this and you say not possible, doesnt your isp control how many packets you receive?
About what is trafics control ? And how can i control how much packet someone send for me ??? I can control how much can i send. Anything else is waste of time. EXCLUDE of application server where i want control traffic coming in to a specified port, eg sendmail etc. But combining router+traffic controler+application server in to one box is good idea ? No. Because of this its lower priority task for me. A quote from TODO:
* support for ingress queue ;)
Quoted 1 line
as I know main imq purpose was to control ingress.
Really ? Iam not sure. Have look in to first original IMQ device driver writen by Martin Devera:
http://luxik.cdi.cz/~devik/qos/imq.htm
Quoted 5 lines
Ok, enough about this,
I would like to know how stable is your driver, if it shapes localy generated trafic,
since it was the main problem for imq all time.
if it became stable I will implement your idea in my imq version too.
Its quite stable. I using it now on two busy routers in my production enviroment (avreage 20 NICs per router, average 70 users, 280 htb classes). Uptime is until kernel upgrade restart (week ago). localy generated trafic is only DNS and ssh for managment and statistics collection (rrd graphs from Stef Coene, www.docum.org, but the data collections is done by ssh) Setup scripts i usink like my example but with clasification of the intranet trafics (for this that much htb classes).
Thanks for the fedback.
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 9 lines · Jiri Fojtasek
Hello Roy
Quoted 3 lines · Roy
Hello,
imq became realy popular now ;)Quoted 1 line
Sure, same as whole GNU/Linux :)
Quoted 7 lines
Quoted 1 line
It would be good ir you wrote what is that EOS to save time for some
people
Quoted 3 lines
on searches.
Quoted 1 line
EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.
Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?
Any of current IMQ implementations do not
Quoted 28 lines
use it, so the calculation by qdisc may not be correct, whish causing filling physical device fifos without control. In devices with variable speed (WiFi, Modems) it causes overload because the physisal trafics is out of control. It happen only on busy boxes, like my routers where is about 70 ppl at once. In a simply test enviroment this issue will never happen.
Quoted 1 line
and it would be good if you posted source code directly (not inside of
diff
Quoted 4 lines
file)
for review purposesQuoted 1 line
The whole patch contain only few new files, all other are patches of the original kernel files. BTW Midnight Comander 4.6.0 contain very nice diff virtual filesystem, so you can browse .diff files like a tar archive.
Quoted 6 lines
also you can look at my imq version, because it is much easer to develop since is independent from kernel.
and does not need to patch it.
http://pupa.da.ru/imq/Quoted 1 line
I have play with with your version :) Having independet implementation is nice idea. I play with that idea too but it was more dificult to have stable solution in short time i need and with EOS support its quite imposible ...
Quoted 8 lines
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 13 lines · Jiri Fojtasek
Quoted 1 line · Roy
Quoted 1 line
Quoted 1 line
seems you may be unable shape ingress then. what basicaly voids
all imq
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
purpose.
Iam not sure. Every packet comin in in to router must come out !!! So we need only shape out trafics. Incoming trafics we cannot shape, only police. We cannot change speed and rate of incoming packets !!! have look at example includet in my IMQ. There i use the corret way to shape incoming and outgoung traffic. BTW: I have writen a TODO ....
Quoted 8 lines
Quoted 5 lines
Here you are completely wrong what makes you think that you cant control incoming trafic?
some wrong publication on internet?
Almost everybody is doing this and you say not possible, doesnt your isp control how many packets you receive?About what is trafics control ? And how can i control how much packet someone send for me ???
You can sort of control TCP from the wrong end of the bottleneck by dropping/queueing. It's not perfect but is alot better than nothing. To do better you could write an app that manipulates rwin and keeps account of the state of each connection so an estimate of what's unstoppably on the way can be used to "ease in" new connections. Better still get an ISP that will do it on the right end for you - some do.
Andy.
I can control how much can i send. Anything
Quoted 29 lines
else is waste of time. EXCLUDE of application server where i want control traffic coming in to a specified port, eg sendmail etc. But combining router+traffic controler+application server in to one box is good idea ? No. Because of this its lower priority task for me. A quote from TODO:
* support for ingress queue ;)Quoted 1 line
as I know main imq purpose was to control ingress.
Really ? Iam not sure. Have look in to first original IMQ device driver writen by Martin Devera:
http://luxik.cdi.cz/~devik/qos/imq.htmQuoted 5 lines
Ok, enough about this,
I would like to know how stable is your driver, if it shapes localy generated trafic,
since it was the main problem for imq all time.
if it became stable I will implement your idea in my imq version too.Its quite stable. I using it now on two busy routers in my production enviroment (avreage 20 NICs per router, average 70 users, 280 htb classes). Uptime is until kernel upgrade restart (week ago). localy generated trafic is only DNS and ssh for managment and statistics collection (rrd graphs from Stef Coene, www.docum.org, but the data collections is done by ssh) Setup scripts i usink like my example but with clasification of the intranet trafics (for this that much htb classes).
Thanks for the fedback.
Quoted 8 lines
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 16 lines · Andy Furniss
Quoted 14 lines · Jiri Fojtasek
Hello Roy
Quoted 3 lines · Roy
Hello,
imq became realy popular now ;)Quoted 1 line
Sure, same as whole GNU/Linux :)
Quoted 1 line
It would be good ir you wrote what is that EOS to save time for
some people
Quoted 3 lines
on searches.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.
Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?
Any of current IMQ implementations do not
It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 27 lines · Jiri Fojtasek
Quoted 24 lines · Andy Furniss
Quoted 9 lines · Jiri Fojtasek
Hello Roy
Quoted 3 lines · Roy
Hello,
imq became realy popular now ;)Quoted 1 line
Sure, same as whole GNU/Linux :)
Quoted 7 lines
Quoted 1 line
It would be good ir you wrote what is that EOS to save time for
some people
Quoted 3 lines
on searches.
Quoted 1 line
EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.
Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?
Quoted 6 lines
Quoted 1 line
Any of current IMQ implementations do not
It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc
I think I understand - in the case of imq -> ppp -> usb/pci -> radsl modem with big buffer -> phone line. imq will only let one packet wait in ppp queue, but there could still be a full buffer in the "real" device.
Andy.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 11 lines · Andy Furniss
Quoted 5 lines · Jiri Fojtasek
It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc
I think I understand - in the case of imq -> ppp -> usb/pci -> radsl modem with big buffer -> phone line. imq will only let one packet wait in ppp queue, but there could still be a full buffer in the "real" device.
I haven't followed the discussion, but this is the way it is supposed to work. As IMQ can queue packets for multiple devices it is not possible to couple the dequeue event with the "real" output device's dequeue event, simply because there may be multiple. Use non-work-conserving qdiscs in this case and make sure you don't exceed the rate of the individual output devices.
Regards
Patrick
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
----- Original Message -----
From: "Patrick McHardy" <[email protected]>
To: "Andy Furniss" <[email protected]>
Cc: "Jiri Fojtasek" <[email protected]>; "LARTC" <[email protected]>
Sent: Wednesday, March 31, 2004 3:46 AM
Subject: Re: [LARTC] New IMQ device implementation supporting device EOS
Quoted 25 lines
Quoted 4 lines · Andy Furniss
Quoted 3 lines · Jiri Fojtasek
It leave always only one packet in the device queue. It is
there until
Quoted 1 line
Quoted 1 line
is not dequeued (sent to the device driver) and then is
inserted
Quoted 4 lines
Quoted 1 line
another etc, etc
I think I understand - in the case of imq -> ppp ->
usb/pci -> radsl
Quoted 1 line
modem with big buffer -> phone line. imq will only let one
packet wait
Quoted 1 line
in ppp queue, but there could still be a full buffer in the
'"'real'"' device.
I haven't followed the discussion, but this is the way it is supposed to work. As IMQ can queue packets for multiple devices it is not possible to couple the dequeue event with the '"'real'"' output device's dequeue event, simply because there may be multiple. Use non-work-conserving qdiscs in this case and make sure you don't exceed the rate of the individual output devices.
Quoted 2 lines
Regards
Patrick
Even if I still not understand if all this is nesecary,
( I only suspect it can make imq stable for local trafic shaping what nome of us was able to do)
But ths idea is posible to implement (i am not sure if what he did realy works as supposed)
multiple devices do not harm anything because of the way how dequeuing works.
if device refuses to accept packet it should be requeued in same place later next packet will be dequeued which may be for other interface.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 34 lines · Andy Furniss
Quoted 32 lines · Jiri Fojtasek
Quoted 26 lines · Andy Furniss
Quoted 9 lines · Jiri Fojtasek
Hello Roy
Quoted 3 lines · Roy
Hello,
imq became realy popular now ;)Quoted 1 line
Sure, same as whole GNU/Linux :)
Quoted 7 lines
Quoted 1 line
It would be good ir you wrote what is that EOS to save time for
some people
Quoted 3 lines
on searches.
Quoted 1 line
EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.
Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?
Any of current IMQ implementations do not
Quoted 8 lines
Quoted 3 lines
It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc
I think I understand - in the case of imq -> ppp -> usb/pci -> radsl modem with big buffer -> phone line. imq will only let one packet wait in ppp queue, but there could still be a full buffer in the "real" device.
Your problem cannot be solved IMQ and also by standart qdisc atached to ppp, but you can place at you ppp qdisc like tbf that will manage maximal rate, so the hardware buffers will not be filled quick. With usage with my IMQ, IMQ queue will follow the rate of the tbf qdisc atached to the ppp device :) I have tested this thing and working great. But for this you will not need IMQ :D
Upper layer do not have anything with hardware buffer. There is not kernel interface to get devices state about his hardware buffer. Especially if you use PPP. Its software device. But device can indicate his busy state by netif_queue_stopped(dev). In time when netif_queue_stopped(dev) bit is set (hardware xmiting the packet), dequeueing is stopped and schedulet in to other time via softirq netif_schedule(dev). The same following my IMQ driver. After dequeueing stoping itself by netif_stop_queue() and sending the packet to real device queue. When the device accept this packet for delivery ( after dev->hard_start_xmit() ) queue of my IMQ driver is waked up by netif_wake_queue(). Version 1.1 of my driver have all comented in the source.
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 9 lines · Patrick McHardy
I haven't followed the discussion, but this is the way it is supposed to work. As IMQ can queue packets for multiple devices it is not possible to couple the dequeue event with the "real" output device's dequeue event, simply because there may be multiple. Use non-work-conserving qdiscs in this case and make sure you don't exceed the rate of the individual output devices.
Regards
Patrick
Not posible ? With trick i did its posible ! And working very well :)
Jiri
Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/