Packetlost when "tc qdisc del dev eth0 root"
Subsystems (2)
Hello all. Have packetlost when do "tc qdisc del dev eth0 root".
look:
slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
.
.....................................................................................................................................................................................................................................................................
.
--- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, ipg/ewma 0.650/2.197 ms
Its normal?
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new. Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.
Thanks for answers.
Slavon.
Quoted 9 lines · Badalian Vyacheslav
Hello all. Have packetlost when do "tc qdisc del dev eth0 root".
look:
slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data. .
.....................................................................................................................................................................................................................................................................
Quoted 6 lines
.
--- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, ipg/ewma 0.650/2.197 msIts normal?
Yes, packets in the old qdisc are lost.
Quoted 7 lines
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new. Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.
That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.
I understand. Thanks!
Quoted 28 lines
Quoted 12 lines · Badalian Vyacheslav
Hello all. Have packetlost when do "tc qdisc del dev eth0 root".
look:
slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
. ..................................................................................................................................................................................................................................................................... . --- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms
rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, ipg/ewma 0.650/2.197 msQuoted 1 line
Its normal?
Yes, packets in the old qdisc are lost.
Quoted 9 lines
Maybe if tc do changes - need create second queue (hash of rules or
how you named it?) and do changes at it. Then replace old queue
rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.
Quoted 4 lines
That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Patrick McHardy wrote, On 01/15/2008 05:05 PM:
Quoted 1 line
Badalian Vyacheslav wrote:
...
Quoted 13 lines
Yes, packets in the old qdisc are lost.
Quoted 9 lines
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.
Quoted 4 lines
That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.
But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?
Regards,
Jarek P.
Quoting Jarek Poplawski <[email protected]>:
Quoted 31 lines
Patrick McHardy wrote, On 01/15/2008 05:05 PM:
Quoted 1 line
Badalian Vyacheslav wrote:
...
Quoted 13 lines
Yes, packets in the old qdisc are lost.
Quoted 9 lines
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.Quoted 4 lines
That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.
But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?
Quoted 2 lines
Regards,
Jarek P.
Hmmm... i found way to fix this for me... but its not look good
Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root) --- Class 3
-------- Copy of all table (Last this qdisc be root)
2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2
All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (> 0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.
Thanks =)
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Good night! =)
Sorry... i was wrong...
I see that problem more serious....
Lets see to scheme
Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes
All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)... PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?
Its normal or code need to more accurate lock?
Thanks!
Quoted 57 lines
Quoting Jarek Poplawski <[email protected]>:
Quoted 31 lines
Patrick McHardy wrote, On 01/15/2008 05:05 PM:
Quoted 1 line
Badalian Vyacheslav wrote:
...
Quoted 13 lines
Yes, packets in the old qdisc are lost.
Quoted 9 lines
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.Quoted 4 lines
That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.
But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?
Quoted 2 lines
Regards,
Jarek P.Hmmm... i found way to fix this for me... but its not look good
Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root) --- Class 3
-------- Copy of all table (Last this qdisc be root)2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (>0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.
Quoted 4 lines
Thanks =)
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
On Wed, Jan 16, 2008 at 12:46:02AM +0300, [email protected] wrote:
...
Quoted 16 lines
Hmmm... i found way to fix this for me... but its not look good
Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root) --- Class 3
-------- Copy of all table (Last this qdisc be root)2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2
Quoted 4 lines
All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (>0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.
Right! I get your idea (not all details...), and this really shows there is needed something simpler for this.
Thanks,
Jarek P.
Quoted 1 line · On Wed, Jan 16, 2008 at 01:04:59AM +0300, [email protected] wrote:
Good night! =)
Good morning! ;)
Quoted 13 lines
Sorry... i was wrong...
I see that problem more serious....Lets see to scheme
Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes
Quoted 7 lines
All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)...
PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?Its normal or code need to more accurate lock?
I don't think it's normal. On the other hand I've never had such problems... Probably this all isn't optimized for such big operations, so maybe you should try to do this with some smaller chunks?
Jarek P.
Quoted 3 lines · On Wed, 2008-01-16 at 00:46 +0300, [email protected] wrote:
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (>
0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.
In contrast a "brand name" router will support 4 to 16 queues per (sub-)interface. Your large number of queues exceeds expectations.
What are you trying to do? You may be better off inventing a new qdisc to meet your need (eg, to do per-IP traffic shaping or, less complexly, a traffic shaping based on the value of mark which might offend DiffServ purists) or have, say, 1000 output rates based on a marking and use the ipset feature of Netfilter to set the mark. Using 1000 rates gives an error of 0.1% which is unlikely to be noticed by your customers given the larger effects of shaping on TCP performance but is beneath the level where you are noticing performance issues.
Quoted 21 lines · Jarek Poplawski
Patrick McHardy wrote, On 01/15/2008 05:05 PM:
Quoted 1 line
Badalian Vyacheslav wrote:
...
Quoted 10 lines
Yes, packets in the old qdisc are lost.
Quoted 7 lines
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new. Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.Quoted 4 lines
That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.
Quoted 6 lines
But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?
This would need support from the qdiscs to do it properly. Looks non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.
Quoted 14 lines · [email protected] wrote:
Good night! =)
Sorry... i was wrong...
I see that problem more serious....Lets see to scheme
Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes
Quoted 4 lines
All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)... PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?
Its normal or code need to more accurate lock?
htb class destruction can be quite expansive if one of the rb trees needs to be rebalanced. Doing that for 10000 classes would explain the delay.
On Wed, Jan 16, 2008 at 06:02:00AM +0100, Patrick McHardy wrote:
...
Quoted 2 lines
This would need support from the qdiscs to do it properly. Looks non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.
Yes. At first I've thought this would need quite a lot of work, but it seems, there could be probably used something very simple too, like e.g. a 'dummy' sched switcher, which after replacing as a root the old qdisc and knowing the pointer to the new one could simply call for dequeing the old one and the new one for everything else. Then, after completely dequeuing it would call destroy for the old qdisc and probably switch itself with the new one as a root. If this new one were created temporarily e.g. on a dummyX dev, and the switch qdisc added to dummyY (as a temporary holder) with ethX and dummyX as parameters, it seems this could be done without any API changes. (But, of course, something more sophisticated should be even better.)
Regards,
Jarek P.
Quoted 5 lines · Jarek Poplawski
On Wed, Jan 16, 2008 at 06:02:00AM +0100, Patrick McHardy wrote:
...Quoted 2 lines
This would need support from the qdiscs to do it properly. Looks non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.
Quoted 11 lines
Yes. At first I've thought this would need quite a lot of work, but it seems, there could be probably used something very simple too, like e.g. a 'dummy' sched switcher, which after replacing as a root the old qdisc and knowing the pointer to the new one could simply call for dequeing the old one and the new one for everything else. Then, after completely dequeuing it would call destroy for the old qdisc and probably switch itself with the new one as a root. If this new one were created temporarily e.g. on a dummyX dev, and the switch qdisc added to dummyY (as a temporary holder) with ethX and dummyX as parameters, it seems this could be done without any API changes. (But, of course, something more sophisticated should be even better.)
Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".
Badalian, what are you actually doing?
Quoted 11 lines
Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".
Badalian, what are you actually doing?
Sorry. Resend to all.
I simple recreate all rules. I change idea from do many add,change,delete because have many kernel panics on many kernels 2.6.x
First i have panics on "delete filter" operation... was fix it... great.. then have panics on "delete htb" operation... long time wait to fix it (maintainer of tc not have time to fix it i think).. but have 1-5 panics at day.... i think my clients hate me =) i rewrite script to simple recreate all rules... in this method i have small packetlost 1 time in hour (then recreate root qdisc), but not have panics... now i use new logic of scripts... =) maybe need go back to accurate logic of scripts, but i to fear kernel panics =)
P.S. Feature request:
tc class show [ dev STRING ] [ root | parent CLASSID ]
may add classid filter to show?
i need do like this/sbin/tc -s class show dev eth0 | grep -A2 "htb $HTB "
to get stats of class... but i every run real look all table.. its not good
Thanks that are you interesting history of my problems =) Slavon.
On Wed, Jan 16, 2008 at 09:05:47AM +0100, Patrick McHardy wrote:
...
Quoted 7 lines
Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".
Badalian, what are you actually doing?
I'm not sure Vyacheslav needs just this, but I've thought about the possibility to recreate the 'shadow' copy of currently used qdisc tree (with some updates of course) while it's running. So, the possibility of using all the same handles and classids, and even dev names if possible, and doing such a switch without any visible break.
Jarek P.
Jarek Poplawski пишет:
Quoted 12 lines · Patrick McHardy
...
Quoted 8 lines
Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".
Badalian, what are you actually doing?
Quoted 10 lines
I'm not sure Vyacheslav needs just this, but I've thought about the possibility to recreate the 'shadow' copy of currently used qdisc tree (with some updates of course) while it's running. So, the possibility of using all the same handles and classids, and even dev names if possible, and doing such a switch without any visible break.
Jarek P.
I also think that system must forward all packets what it get if it not dropped manual (by iptables or shaper).
Maybe someone need to test delete big TREE.. simple delete, not recreate... linux unavailable some time (if its realy big table its time may be 10-20 sec on 2xXeon).
I think need helper to do that operations more accurate. Now see situation that linux is PPTP server... its get 2000k connection... try delete qdisc on eth0 (incoming from wan to pptp clients)... i think many sessions will drop.
Thanks!
On Wed, Jan 16, 2008 at 11:35:35AM +0300, Badalian Vyacheslav wrote:
...
Quoted 11 lines
I simple recreate all rules. I change idea from do many add,change,delete because have many kernel panics on many kernels 2.6.x
First i have panics on "delete filter" operation... was fix it... great.. then have panics on "delete htb" operation... long time wait to fix it (maintainer of tc not have time to fix it i think).. but have 1-5 panics at day.... i think my clients hate me =) i rewrite script to simple recreate all rules... in this method i have small packetlost 1 time in hour (then recreate root qdisc), but not have panics... now i use new logic of scripts... =)
maybe need go back to accurate logic of scripts, but i to fear kernel panics =)
BTW, I don't know about others, but it seems this bugzilla #9632 waits for your testing, to find the reason of this bug... IMHO, if you can't try this now, it's better to close this or/and at least add some comment. And if there are any other bugs unfixed which are reproducible and you can test (or have tested) some patches, please resend them as new threads to the list. Alas, omitting the panics can't help in removing these bugs.
Jarek P.
Patrick McHardy wrote, On 01/16/2008 06:12 AM:
Quoted 21 lines
Quoted 20 lines · [email protected] wrote:
Good night! =)
Sorry... i was wrong...
I see that problem more serious....Lets see to scheme
Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classesAll traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)...
PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?
Quoted 6 lines
Quoted 1 line
Its normal or code need to more accurate lock?
htb class destruction can be quite expansive if one of the rb trees needs to be rebalanced. Doing that for 10000 classes would explain the delay.
I've just looked at this and it seems there should be no rbtrees operations during htb qdisc deletion (or I miss something).
Slavon, I see nearby that you teach oprofile...
Regards,
Jarek P.