Skip to contentSkip to conversation

New IMQ device implementation supporting device EOS

LARTC Jiri Fojtasek 4 participants 12 messages Mar 29, 2004 – Mar 31, 2004

lore
Share & tools
lore
#0 lore raw

Hello All

Its first time i posting to this mail list :). I have done something (maybe) important. I write new IMQ device driver based from Martin Devera and Patrick McHardy implementation with device EOS support. My current implementation supporting only egress trafic shaping and kernel 2.4.25. For more details, source and examples have look at my page
http://hyperfighter.jinak.cz/qos

Ill apreticate your feedback :)

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

Hello Roy

Quoted 4 lines · Roy

Hello,
imq became realy popular now ;)

Sure, same as whole GNU/Linux :)

Quoted 1 line

It would be good ir you wrote what is that EOS to save time for some

people

Quoted 4 lines

on searches.

EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation. Any of current IMQ implementations do not use it, so the calculation by qdisc may not be correct, whish causing filling physical device fifos without control. In devices with variable speed (WiFi, Modems) it causes overload because the physisal trafics is out of control. It happen only on busy boxes, like my routers where is about 70 ppl at once. In a simply test enviroment this issue will never happen.

Quoted 1 line

and it would be good if you posted source code directly (not inside of

diff

Quoted 5 lines

file)
for review purposes

The whole patch contain only few new files, all other are patches of the original kernel files. BTW Midnight Comander 4.6.0 contain very nice diff virtual filesystem, so you can browse .diff files like a tar archive.

Quoted 7 lines

also you can look at my imq version, because it is much easer to develop since is independent from kernel.
and does not need to patch it.
http://pupa.da.ru/imq/

I have play with with your version :) Having independet implementation is nice idea. I play with that idea too but it was more dificult to have stable solution in short time i need and with EOS support its quite imposible ...

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Jiri Fojtasek
Quoted 1 line · Roy

this is qute intersting, and it happens on every computer not only on

bussy

Quoted 2 lines

ones
but in reality I dont see any need for that, because tcp itself should

take

Quoted 2 lines

care of this issue.

Right but you lost TCP interactivity and overloaded WiFi interface is painfull itself !!!

Quoted 3 lines

anyway this may solve some problems with crashes, because I suspect imq is unstable because of this problem.

Stability if any current implementations is issue by design. Simply something betwen netfilter and TCP/IP stack. I tried found it but my suceess was only more crashes.

Quoted 1 line

I looked at yor code and do not understand what it does, do you

implemented

Quoted 2 lines

that EOS for imq interface ifself?
this is not logical, since imq is not interface, it is placeholder to

attach

Quoted 7 lines

shapers.
Or you made that your imq takes packets from kernel ant transmits them by itself without returning to netfilter
with dev_queue_xmit(skb)
this way you are very limited in what you are able to do because you can hook only on the last hook.

There is skb->imq_flags whis getting marked by netfilter to IMQ_F_ENQUEUE (part of flag is target IMQ network interface) At "end of life in kernel" every packet coming into dev_queue_xmit. There also come our marked skb. When you have look at begin of the modified dev_queue_xmit
there is imq_flags decoder. When decoder hit our marked SKB its enqueued in to IMQ queue. Then we run qdisc dequeue (qdisc_run). qdisc_run function is a loop tat run qdisc restart until is queue empty, need to be schedulet (netif_schedule) or until is not set netif_queue_stopped atomic bit (bingo the trick is here). After dequeue in qdisc restart our skb passing hard_start_xmit function in IMQ driver. After some checkings is there stopped queue of the imq device (netif_queue_stopped), changet skb->imq_flags to IMQ_F_ENQUEUED and passed back to dev_queue_xmit. Now it will hit our real device, and at end of the qdisc_restart (after hard_start_xmit) is function that decode the skb->imq_flags and if is necesary wake_up the queue of our IMQ device (so we have perfect synchronization of the EOS). There is some protection to lock up the queue in imq_dev_xmit (imq.c). Also there are counted throtled skb by collision counter. I also use device watchdog to solve problem if skb stuck somewhere in physical device queue and netif_wake_queue is not called in right time( its only protection "for sure" because this should happen and if this happen without protection behind this, whole imq should lockup and kill the trafics forewer)

Quoted 3 lines

seems you may be unable shape ingress then. what basicaly voids all imq purpose.

Iam not sure. Every packet comin in in to router must come out !!! So we need only shape out trafics. Incoming trafics we cannot shape, only police. We cannot change speed and rate of incoming packets !!! have look at example includet in my IMQ. There i use the corret way to shape incoming and outgoung traffic. BTW: I have writen a TODO ....

Quoted 3 lines

unfortunately you made your driver even more invasive than old one, it need to patch almost all kernel network system.

Yes its also angle of view :) I was need to solve problem and i solve it. Everithing there is clear. IMHO Cannot be solved by more clear solution. ;)

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Jiri Fojtasek
Quoted 1 line · Roy
Quoted 1 line
Quoted 1 line

seems you may be unable shape ingress then. what basicaly voids

all imq

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

purpose.

Iam not sure. Every packet comin in in to router must come out !!! So we need only shape out trafics. Incoming trafics we cannot shape, only police. We cannot change speed and rate of incoming packets !!! have look at example includet in my IMQ. There i use the corret way to shape incoming and outgoung traffic. BTW: I have writen a TODO ....

Quoted 5 lines

Here you are completely wrong what makes you think that you cant control incoming trafic?
some wrong publication on internet?
Almost everybody is doing this and you say not possible, doesnt your isp control how many packets you receive?

About what is trafics control ? And how can i control how much packet someone send for me ??? I can control how much can i send. Anything else is waste of time. EXCLUDE of application server where i want control traffic coming in to a specified port, eg sendmail etc. But combining router+traffic controler+application server in to one box is good idea ? No. Because of this its lower priority task for me. A quote from TODO:
* support for ingress queue ;)

Quoted 1 line

as I know main imq purpose was to control ingress.

Really ? Iam not sure. Have look in to first original IMQ device driver writen by Martin Devera:
http://luxik.cdi.cz/~devik/qos/imq.htm

Quoted 5 lines

Ok, enough about this,
I would like to know how stable is your driver, if it shapes localy generated trafic,
since it was the main problem for imq all time.
if it became stable I will implement your idea in my imq version too.

Its quite stable. I using it now on two busy routers in my production enviroment (avreage 20 NICs per router, average 70 users, 280 htb classes). Uptime is until kernel upgrade restart (week ago). localy generated trafic is only DNS and ssh for managment and statistics collection (rrd graphs from Stef Coene, www.docum.org, but the data collections is done by ssh) Setup scripts i usink like my example but with clasification of the intranet trafics (for this that much htb classes).

Thanks for the fedback.

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Jiri Fojtasek
Quoted 9 lines · Jiri Fojtasek

Hello Roy

Quoted 3 lines · Roy

Hello,
imq became realy popular now ;)

Quoted 1 line

Sure, same as whole GNU/Linux :)

Quoted 7 lines
Quoted 1 line

It would be good ir you wrote what is that EOS to save time for some

people

Quoted 3 lines

on searches.

Quoted 1 line

EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.

Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?

Any of current IMQ implementations do not

Quoted 28 lines

use it, so the calculation by qdisc may not be correct, whish causing filling physical device fifos without control. In devices with variable speed (WiFi, Modems) it causes overload because the physisal trafics is out of control. It happen only on busy boxes, like my routers where is about 70 ppl at once. In a simply test enviroment this issue will never happen.

Quoted 1 line

and it would be good if you posted source code directly (not inside of

diff

Quoted 4 lines

file)
for review purposes

Quoted 1 line

The whole patch contain only few new files, all other are patches of the original kernel files. BTW Midnight Comander 4.6.0 contain very nice diff virtual filesystem, so you can browse .diff files like a tar archive.

Quoted 6 lines

also you can look at my imq version, because it is much easer to develop since is independent from kernel.
and does not need to patch it.
http://pupa.da.ru/imq/

Quoted 1 line

I have play with with your version :) Having independet implementation is nice idea. I play with that idea too but it was more dificult to have stable solution in short time i need and with EOS support its quite imposible ...

Quoted 8 lines

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Jiri Fojtasek
Quoted 13 lines · Jiri Fojtasek
Quoted 1 line · Roy
Quoted 1 line
Quoted 1 line

seems you may be unable shape ingress then. what basicaly voids

all imq

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

purpose.

Iam not sure. Every packet comin in in to router must come out !!! So we need only shape out trafics. Incoming trafics we cannot shape, only police. We cannot change speed and rate of incoming packets !!! have look at example includet in my IMQ. There i use the corret way to shape incoming and outgoung traffic. BTW: I have writen a TODO ....

Quoted 8 lines
Quoted 5 lines

Here you are completely wrong what makes you think that you cant control incoming trafic?
some wrong publication on internet?
Almost everybody is doing this and you say not possible, doesnt your isp control how many packets you receive?

About what is trafics control ? And how can i control how much packet someone send for me ???

You can sort of control TCP from the wrong end of the bottleneck by dropping/queueing. It's not perfect but is alot better than nothing. To do better you could write an app that manipulates rwin and keeps account of the state of each connection so an estimate of what's unstoppably on the way can be used to "ease in" new connections. Better still get an ISP that will do it on the right end for you - some do.

Andy.

I can control how much can i send. Anything

Quoted 29 lines

else is waste of time. EXCLUDE of application server where i want control traffic coming in to a specified port, eg sendmail etc. But combining router+traffic controler+application server in to one box is good idea ? No. Because of this its lower priority task for me. A quote from TODO:
* support for ingress queue ;)

Quoted 1 line

as I know main imq purpose was to control ingress.

Really ? Iam not sure. Have look in to first original IMQ device driver writen by Martin Devera:
http://luxik.cdi.cz/~devik/qos/imq.htm

Quoted 5 lines

Ok, enough about this,
I would like to know how stable is your driver, if it shapes localy generated trafic,
since it was the main problem for imq all time.
if it became stable I will implement your idea in my imq version too.

Its quite stable. I using it now on two busy routers in my production enviroment (avreage 20 NICs per router, average 70 users, 280 htb classes). Uptime is until kernel upgrade restart (week ago). localy generated trafic is only DNS and ssh for managment and statistics collection (rrd graphs from Stef Coene, www.docum.org, but the data collections is done by ssh) Setup scripts i usink like my example but with clasification of the intranet trafics (for this that much htb classes).

Thanks for the fedback.

Quoted 8 lines

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Jiri Fojtasek
Quoted 16 lines · Andy Furniss
Quoted 14 lines · Jiri Fojtasek

Hello Roy

Quoted 3 lines · Roy

Hello,
imq became realy popular now ;)

Quoted 1 line

Sure, same as whole GNU/Linux :)

Quoted 1 line

It would be good ir you wrote what is that EOS to save time for

some people

Quoted 3 lines

on searches.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.

Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?

Any of current IMQ implementations do not

It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Jiri Fojtasek
Quoted 27 lines · Jiri Fojtasek
Quoted 24 lines · Andy Furniss
Quoted 9 lines · Jiri Fojtasek

Hello Roy

Quoted 3 lines · Roy

Hello,
imq became realy popular now ;)

Quoted 1 line

Sure, same as whole GNU/Linux :)

Quoted 7 lines
Quoted 1 line

It would be good ir you wrote what is that EOS to save time for

some people

Quoted 3 lines

on searches.

Quoted 1 line

EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.

Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?

Quoted 6 lines
Quoted 1 line

Any of current IMQ implementations do not

It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc

I think I understand - in the case of imq -> ppp -> usb/pci -> radsl modem with big buffer -> phone line. imq will only let one packet wait in ppp queue, but there could still be a full buffer in the "real" device.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
In reply to Jiri Fojtasek
Quoted 11 lines · Andy Furniss
Quoted 5 lines · Jiri Fojtasek

It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc

I think I understand - in the case of imq -> ppp -> usb/pci -> radsl modem with big buffer -> phone line. imq will only let one packet wait in ppp queue, but there could still be a full buffer in the "real" device.

I haven't followed the discussion, but this is the way it is supposed to work. As IMQ can queue packets for multiple devices it is not possible to couple the dequeue event with the "real" output device's dequeue event, simply because there may be multiple. Use non-work-conserving qdiscs in this case and make sure you don't exceed the rate of the individual output devices.

Regards
Patrick
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Jiri Fojtasek

----- Original Message -----
From: "Patrick McHardy" <[email protected]>
To: "Andy Furniss" <[email protected]>
Cc: "Jiri Fojtasek" <[email protected]>; "LARTC" <[email protected]>
Sent: Wednesday, March 31, 2004 3:46 AM
Subject: Re: [LARTC] New IMQ device implementation supporting device EOS

Quoted 25 lines
Quoted 4 lines · Andy Furniss
Quoted 3 lines · Jiri Fojtasek

It leave always only one packet in the device queue. It is

there until

Quoted 1 line
Quoted 1 line

is not dequeued (sent to the device driver) and then is

inserted

Quoted 4 lines
Quoted 1 line

another etc, etc

I think I understand - in the case of imq -> ppp ->

usb/pci -> radsl

Quoted 1 line

modem with big buffer -> phone line. imq will only let one

packet wait

Quoted 1 line

in ppp queue, but there could still be a full buffer in the

'"'real'"' device.

I haven't followed the discussion, but this is the way it is supposed to work. As IMQ can queue packets for multiple devices it is not possible to couple the dequeue event with the '"'real'"' output device's dequeue event, simply because there may be multiple. Use non-work-conserving qdiscs in this case and make sure you don't exceed the rate of the individual output devices.

Quoted 2 lines

Regards
Patrick

Even if I still not understand if all this is nesecary,
( I only suspect it can make imq stable for local trafic shaping what nome of us was able to do)

But ths idea is posible to implement (i am not sure if what he did realy works as supposed)
multiple devices do not harm anything because of the way how dequeuing works.
if device refuses to accept packet it should be requeued in same place later next packet will be dequeued which may be for other interface.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#10 lore raw
In reply to Jiri Fojtasek
Quoted 34 lines · Andy Furniss
Quoted 32 lines · Jiri Fojtasek
Quoted 26 lines · Andy Furniss
Quoted 9 lines · Jiri Fojtasek

Hello Roy

Quoted 3 lines · Roy

Hello,
imq became realy popular now ;)

Quoted 1 line

Sure, same as whole GNU/Linux :)

Quoted 7 lines
Quoted 1 line

It would be good ir you wrote what is that EOS to save time for

some people

Quoted 3 lines

on searches.

Quoted 1 line

EOS - End of Send. Its time betwen two dequeues and packet transmits used by qdisc for calculation.

Is that a kernel transmit or a real "on the wire" transmit - ie can you keep the devices' buffer empty or just detect when it's full?

Any of current IMQ implementations do not

Quoted 8 lines
Quoted 3 lines

It leave always only one packet in the device queue. It is there until is not dequeued (sent to the device driver) and then is inserted another etc, etc

I think I understand - in the case of imq -> ppp -> usb/pci -> radsl modem with big buffer -> phone line. imq will only let one packet wait in ppp queue, but there could still be a full buffer in the "real" device.

Your problem cannot be solved IMQ and also by standart qdisc atached to ppp, but you can place at you ppp qdisc like tbf that will manage maximal rate, so the hardware buffers will not be filled quick. With usage with my IMQ, IMQ queue will follow the rate of the tbf qdisc atached to the ppp device :) I have tested this thing and working great. But for this you will not need IMQ :D

Upper layer do not have anything with hardware buffer. There is not kernel interface to get devices state about his hardware buffer. Especially if you use PPP. Its software device. But device can indicate his busy state by netif_queue_stopped(dev). In time when netif_queue_stopped(dev) bit is set (hardware xmiting the packet), dequeueing is stopped and schedulet in to other time via softirq netif_schedule(dev). The same following my IMQ driver. After dequeueing stoping itself by netif_stop_queue() and sending the packet to real device queue. When the device accept this packet for delivery ( after dev->hard_start_xmit() ) queue of my IMQ driver is waked up by netif_wake_queue(). Version 1.1 of my driver have all comented in the source.

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#11 lore raw
In reply to Jiri Fojtasek
Quoted 9 lines · Patrick McHardy

I haven't followed the discussion, but this is the way it is supposed to work. As IMQ can queue packets for multiple devices it is not possible to couple the dequeue event with the "real" output device's dequeue event, simply because there may be multiple. Use non-work-conserving qdiscs in this case and make sure you don't exceed the rate of the individual output devices.

Regards
Patrick

Not posible ? With trick i did its posible ! And working very well :)

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/