Skip to contentSkip to conversation

Packetlost when "tc qdisc del dev eth0 root"

DISCUSSION Badalian Vyacheslav 4 participants 18 messages Jan 15, 2008 – Jan 19, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello all. Have packetlost when do "tc qdisc del dev eth0 root".

look:

slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
.
.....................................................................................................................................................................................................................................................................
.
--- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, ipg/ewma 0.650/2.197 ms

Its normal?
Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new. Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Thanks for answers.
Slavon.

#1 lore raw
Quoted 9 lines · Badalian Vyacheslav

Hello all. Have packetlost when do "tc qdisc del dev eth0 root".

look:

slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data. .
.....................................................................................................................................................................................................................................................................

Quoted 6 lines

.
--- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, ipg/ewma 0.650/2.197 ms

Its normal?

Yes, packets in the old qdisc are lost.

Quoted 7 lines

Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new. Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.

#2 lore raw

I understand. Thanks!

Quoted 28 lines
Quoted 12 lines · Badalian Vyacheslav

Hello all. Have packetlost when do "tc qdisc del dev eth0 root".

look:

slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
. ..................................................................................................................................................................................................................................................................... . --- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms
rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, ipg/ewma 0.650/2.197 ms

Quoted 1 line

Its normal?

Yes, packets in the old qdisc are lost.

Quoted 9 lines

Maybe if tc do changes - need create second queue (hash of rules or
how you named it?) and do changes at it. Then replace old queue
rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Quoted 4 lines

That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

#3 lore raw
In reply to Patrick McHardy

Patrick McHardy wrote, On 01/15/2008 05:05 PM:

Quoted 1 line

Badalian Vyacheslav wrote:

...

Quoted 13 lines

Yes, packets in the old qdisc are lost.

Quoted 9 lines

Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Quoted 4 lines

That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.

But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?

Regards,
Jarek P.

#4 lore raw

Quoting Jarek Poplawski <[email protected]>:

Quoted 31 lines

Patrick McHardy wrote, On 01/15/2008 05:05 PM:

Quoted 1 line

Badalian Vyacheslav wrote:

...

Quoted 13 lines

Yes, packets in the old qdisc are lost.

Quoted 9 lines

Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Quoted 4 lines

That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.

But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?

Quoted 2 lines

Regards,
Jarek P.

Hmmm... i found way to fix this for me... but its not look good

Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root) --- Class 3
-------- Copy of all table (Last this qdisc be root)

2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2

All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (> 0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.

Thanks =)

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

#5 lore raw

Good night! =)

Sorry... i was wrong...
I see that problem more serious....

Lets see to scheme

Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes

All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)... PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?

Its normal or code need to more accurate lock?

Thanks!

Quoted 57 lines

Quoting Jarek Poplawski <[email protected]>:

Quoted 31 lines

Patrick McHardy wrote, On 01/15/2008 05:05 PM:

Quoted 1 line

Badalian Vyacheslav wrote:

...

Quoted 13 lines

Yes, packets in the old qdisc are lost.

Quoted 9 lines

Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Quoted 4 lines

That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.

But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?

Quoted 2 lines

Regards,
Jarek P.

Hmmm... i found way to fix this for me... but its not look good

Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root) --- Class 3
-------- Copy of all table (Last this qdisc be root)

2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2

All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (> 0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.

Quoted 4 lines

Thanks =)

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

#6 lore raw
Quoted 1 line · On Wed, Jan 16, 2008 at 01:04:59AM +0300, [email protected] wrote:

Good night! =)

Good morning! ;)

Quoted 13 lines

Sorry... i was wrong...
I see that problem more serious....

Lets see to scheme

Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes

Quoted 7 lines

All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)...
PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?

Its normal or code need to more accurate lock?

I don't think it's normal. On the other hand I've never had such problems... Probably this all isn't optimized for such big operations, so maybe you should try to do this with some smaller chunks?

Jarek P.

#7 lore raw
In reply to [email protected]
Quoted 14 lines · [email protected] wrote:

Good night! =)

Sorry... i was wrong...
I see that problem more serious....

Lets see to scheme

Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes

Quoted 4 lines

All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)... PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?

Its normal or code need to more accurate lock?

htb class destruction can be quite expansive if one of the rb trees needs to be rebalanced. Doing that for 10000 classes would explain the delay.

#8 lore raw

Patrick McHardy wrote, On 01/16/2008 06:12 AM:

Quoted 21 lines
Quoted 20 lines · [email protected] wrote:

Good night! =)

Sorry... i was wrong...
I see that problem more serious....

Lets see to scheme

Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes

All traffic go to class 2... class 1 qdisc not have packets and if we delete it - packets not lost... in theory... lets try delete class 1 qdisc (all childrens delete too)...
PC freeze on 2-5 seconds... its not forward any traffic at this moment... its great tree lock?

Quoted 6 lines
Quoted 1 line

Its normal or code need to more accurate lock?

htb class destruction can be quite expansive if one of the rb trees needs to be rebalanced. Doing that for 10000 classes would explain the delay.

I've just looked at this and it seems there should be no rbtrees operations during htb qdisc deletion (or I miss something).

Slavon, I see nearby that you teach oprofile...

Regards,
Jarek P.

#9 lore raw
In reply to [email protected]

On Wed, Jan 16, 2008 at 12:46:02AM +0300, [email protected] wrote:
...

Quoted 16 lines

Hmmm... i found way to fix this for me... but its not look good

Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root) --- Class 3
-------- Copy of all table (Last this qdisc be root)

2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2

Quoted 4 lines

All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (> 0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.

Right! I get your idea (not all details...), and this really shows there is needed something simpler for this.

Thanks,
Jarek P.

#10 lore raw
In reply to [email protected]
Quoted 3 lines · On Wed, 2008-01-16 at 00:46 +0300, [email protected] wrote:

But i have above 45 k classes and qdiscs.... After some time i will need patch to up max qdisc and classes more then 65k (> 0xfffe) =))) Also i have very bad TC commands performance then i have more then 10k rules.

In contrast a "brand name" router will support 4 to 16 queues per (sub-)interface. Your large number of queues exceeds expectations.

What are you trying to do? You may be better off inventing a new qdisc to meet your need (eg, to do per-IP traffic shaping or, less complexly, a traffic shaping based on the value of mark which might offend DiffServ purists) or have, say, 1000 output rates based on a marking and use the ipset feature of Netfilter to set the mark. Using 1000 rates gives an error of 0.1% which is unlikely to be noticed by your customers given the larger effects of shaping on TCP performance but is beneath the level where you are noticing performance issues.

#11 lore raw
In reply to Jarek Poplawski
Quoted 21 lines · Jarek Poplawski

Patrick McHardy wrote, On 01/15/2008 05:05 PM:

Quoted 1 line

Badalian Vyacheslav wrote:

...

Quoted 10 lines

Yes, packets in the old qdisc are lost.

Quoted 7 lines

Maybe if tc do changes - need create second queue (hash of rules or how you named it?) and do changes at it. Then replace old queue rules by created new. Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Quoted 4 lines

That doesn't really work since qdiscs keep internal state that in large parts depends on the packets queued. Take the qlen as a simple example, the new qdisc doesn't know about the packets in the old one and will exceed the limit.

Quoted 6 lines

But, some similar alternative to killing packets 'to death' could be imagined, I suppose (in the future, of course!). So, e.g. doing the switch automatically after last packet has been dequeued (maybe even with some 'special' function/mode for this). After all even with accuracy lost, it could be less visible for clients than current way?

This would need support from the qdiscs to do it properly. Looks non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.

#12 lore raw

On Wed, Jan 16, 2008 at 06:02:00AM +0100, Patrick McHardy wrote:
...

Quoted 2 lines

This would need support from the qdiscs to do it properly. Looks non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.

Yes. At first I've thought this would need quite a lot of work, but it seems, there could be probably used something very simple too, like e.g. a 'dummy' sched switcher, which after replacing as a root the old qdisc and knowing the pointer to the new one could simply call for dequeing the old one and the new one for everything else. Then, after completely dequeuing it would call destroy for the old qdisc and probably switch itself with the new one as a root. If this new one were created temporarily e.g. on a dummyX dev, and the switch qdisc added to dummyY (as a temporary holder) with ethX and dummyX as parameters, it seems this could be done without any API changes. (But, of course, something more sophisticated should be even better.)

Regards,
Jarek P.

#13 lore raw
Quoted 5 lines · Jarek Poplawski

On Wed, Jan 16, 2008 at 06:02:00AM +0100, Patrick McHardy wrote:
...

Quoted 2 lines

This would need support from the qdiscs to do it properly. Looks non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.

Quoted 11 lines

Yes. At first I've thought this would need quite a lot of work, but it seems, there could be probably used something very simple too, like e.g. a 'dummy' sched switcher, which after replacing as a root the old qdisc and knowing the pointer to the new one could simply call for dequeing the old one and the new one for everything else. Then, after completely dequeuing it would call destroy for the old qdisc and probably switch itself with the new one as a root. If this new one were created temporarily e.g. on a dummyX dev, and the switch qdisc added to dummyY (as a temporary holder) with ethX and dummyX as parameters, it seems this could be done without any API changes. (But, of course, something more sophisticated should be even better.)

Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".

Badalian, what are you actually doing?

#14 lore raw
Quoted 11 lines

Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".

Badalian, what are you actually doing?

Sorry. Resend to all.

I simple recreate all rules. I change idea from do many add,change,delete because have many kernel panics on many kernels 2.6.x
First i have panics on "delete filter" operation... was fix it... great.. then have panics on "delete htb" operation... long time wait to fix it (maintainer of tc not have time to fix it i think).. but have 1-5 panics at day.... i think my clients hate me =) i rewrite script to simple recreate all rules... in this method i have small packetlost 1 time in hour (then recreate root qdisc), but not have panics... now i use new logic of scripts... =) maybe need go back to accurate logic of scripts, but i to fear kernel panics =)

P.S. Feature request:
tc class show [ dev STRING ] [ root | parent CLASSID ]
may add classid filter to show?
i need do like this
/sbin/tc -s class show dev eth0 | grep -A2 "htb $HTB "
to get stats of class... but i every run real look all table.. its not good

Thanks that are you interesting history of my problems =) Slavon.

#15 lore raw

On Wed, Jan 16, 2008 at 11:35:35AM +0300, Badalian Vyacheslav wrote:
...

Quoted 11 lines

I simple recreate all rules. I change idea from do many add,change,delete because have many kernel panics on many kernels 2.6.x
First i have panics on "delete filter" operation... was fix it... great.. then have panics on "delete htb" operation... long time wait to fix it (maintainer of tc not have time to fix it i think).. but have 1-5 panics at day.... i think my clients hate me =) i rewrite script to simple recreate all rules... in this method i have small packetlost 1 time in hour (then recreate root qdisc), but not have panics... now i use new logic of scripts... =)
maybe need go back to accurate logic of scripts, but i to fear kernel panics =)

BTW, I don't know about others, but it seems this bugzilla #9632 waits for your testing, to find the reason of this bug... IMHO, if you can't try this now, it's better to close this or/and at least add some comment. And if there are any other bugs unfixed which are reproducible and you can test (or have tested) some patches, please resend them as new threads to the list. Alas, omitting the panics can't help in removing these bugs.

Jarek P.

#16 lore raw
In reply to Patrick McHardy

On Wed, Jan 16, 2008 at 09:05:47AM +0100, Patrick McHardy wrote:
...

Quoted 7 lines

Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".

Badalian, what are you actually doing?

I'm not sure Vyacheslav needs just this, but I've thought about the possibility to recreate the 'shadow' copy of currently used qdisc tree (with some updates of course) while it's running. So, the possibility of using all the same handles and classids, and even dev names if possible, and doing such a switch without any visible break.

Jarek P.

#17 lore raw

Jarek Poplawski пишет:

Quoted 12 lines · Patrick McHardy

...

Quoted 8 lines

Yes, thats one possibility (without the dummy device though please). But I wonder what this would actually be useful for. I don't think replacing the root qdisc by a different type is a common scenario, for the same type you can simply use "tc qdisc change", "tc class change" and "tc class replace".

Badalian, what are you actually doing?

Quoted 10 lines

I'm not sure Vyacheslav needs just this, but I've thought about the possibility to recreate the 'shadow' copy of currently used qdisc tree (with some updates of course) while it's running. So, the possibility of using all the same handles and classids, and even dev names if possible, and doing such a switch without any visible break.

Jarek P.

I also think that system must forward all packets what it get if it not dropped manual (by iptables or shaper).
Maybe someone need to test delete big TREE.. simple delete, not recreate... linux unavailable some time (if its realy big table its time may be 10-20 sec on 2xXeon).
I think need helper to do that operations more accurate. Now see situation that linux is PPTP server... its get 2000k connection... try delete qdisc on eth0 (incoming from wan to pptp clients)... i think many sessions will drop.

Thanks!