deadlocks if use htb

Hello all!

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

Quoted 3 lines

Hello all!

Hello Slavon,

Quoted 3 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

Quoted 4 lines

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

Jarek P.

(some offsets are OK when patching 2.6.26)
---

 net/sched/sch_htb.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

net/sched/sch_htb.c +7 -1

index 30c999c..ff9e965 100644
@@ -162,6 +162,7 @@ struct htb_sched {
 
 	int rate2quantum;	/* quant = rate / rate2quantum */
 	psched_time_t now;	/* cached dequeue time */
+	psched_time_t next_watchdog;
 	struct qdisc_watchdog watchdog;
 
 	/* non shaped skbs; let them go directly thru */

@@ -920,7 +921,11 @@ static struct sk_buff *htb_dequeue(struct Qdisc *sch)
 		}
 	}
 	sch->qstats.overlimits++;
-	qdisc_watchdog_schedule(&q->watchdog, next_event);
+	if (q->next_watchdog < q->now || next_event <=
+	     q->next_watchdog - PSCHED_TICKS_PER_SEC / HZ) {
+		qdisc_watchdog_schedule(&q->watchdog, next_event);
+		q->next_watchdog = next_event;
+	}
 fin:
 	return skb;
 }

@@ -973,6 +978,7 @@ static void htb_reset(struct Qdisc *sch)
 		}
 	}
 	qdisc_watchdog_cancel(&q->watchdog);
+	q->next_watchdog = 0;
 	__skb_queue_purge(&q->direct_queue);
 	sch->q.qlen = 0;
 	memset(q->row, 0, sizeof(q->row));

Quoted 7 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

Jarek P.

Oh... sorry. I Wrong you understand... i patch 2.6.27 with this patch and will test it...

Quoted 14 lines

Quoted 8 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

Jarek P.

Quoted 2 lines · Badalian Vyacheslav

Oh... sorry. I Wrong you understand... i patch 2.6.27 with this patch and will test it...

No, you understood it right. But it seems 2.6.27 fix doesn't work for you. So, yes, try this patch with 2.6.26 or 2.6.27.

Quoted 15 lines

Quoted 14 lines

Quoted 8 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

Jarek P.

Quoted 9 lines · Badalian Vyacheslav

Quoted 4 lines

Oh... sorry. I Wrong you understand... i patch 2.6.27 with this patch and will test it...

No, you understood it right. But it seems 2.6.27 fix doesn't work for you. So, yes, try this patch with 2.6.26 or 2.6.27.

There is also a parameter you could try (without this patch):

modprobe sch_htb htb_hysteresis=1

Jarek P.

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

Quoted 8 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

Jarek P.

Sorry for long answer.

We have troubles with power in our server place. Now its gone and i will test again all this.

With patch + htb_hysteresis=0 and htb_hysteresis=1 without patch all PC work done 2 days and 18 hours. After this we have power crash.... =(

...

Quoted 8 lines

Sorry for long answer.

We have troubles with power in our server place. Now its gone and i will test again all this.

With patch + htb_hysteresis=0 and htb_hysteresis=1 without patch all PC work done 2 days and 18 hours. After this we have power crash.... =(

No need to hurry: you've written it's not everyday. Better try to make sure there is really a diffrence after any of these changes.

Thanks,
Jarek P.

Hello!

I get more information.

Statistics of PC:
1. 2.6.26.6 Dunamic Timer, HiResTimer, 1000HZ, htb_hysteresis=0 - crashed 1d 18h ago
2. 2.6.26.5 HZ300, NO Dunamic Timer, No HiResTimer, htb_hysteresis=0 - uptime 5d 17h (no crashes for now, but it crashed some time ago with htb_hysteresis=1)

3. 2.6.27, 1000HZ, NO Dunamic Timer, No HiResTimer, htb_hysteresis=0 + PATCH - uptime 5d 17h (no crashes for now, but it crashed some time ago without patch)

Also attach crash log of lash crash PC 1:

[10610.110729] BUG: NMI Watchdog detected LOCKUP on CPU1, ip c01fd939,
registers:
[10610.110729] Modules linked in: netconsole e1000e i2c_i801 i2c_core e1000
[10610.110729]
[10610.110729] Pid: 0, comm: swapper Not tainted (2.6.26.6-fw #1)
[10610.110729] EIP: 0060:[<c01fd939>] EFLAGS: 00000082 CPU: 1
[10610.110729] EIP is at rb_insert_color+0x19/0xc0

Quoted 3 lines

Hello!

Hi!

Quoted 9 lines

I get more information.

Statistics of PC:
1. 2.6.26.6 Dunamic Timer, HiResTimer, 1000HZ, htb_hysteresis=0 - crashed 1d 18h ago
2. 2.6.26.5 HZ300, NO Dunamic Timer, No HiResTimer, htb_hysteresis=0 - uptime 5d 17h (no crashes for now, but it crashed some time ago with htb_hysteresis=1)

So it looks like htb_hysteresis change could be not enough, and it's a pity because this could be easiest to "reverse" in the kernel.

Quoted 4 lines

3. 2.6.27, 1000HZ, NO Dunamic Timer, No HiResTimer, htb_hysteresis=0 + PATCH - uptime 5d 17h (no crashes for now, but it crashed some time ago without patch)

So I guess this patch seems to make some difference, but it also could be hard to convince people to fix it this way, because it makes scheduling less exact.

Alas I have still no idea of the real reason. IMHO this should be rather debugged by hrtimers/NMI people, but there were no respose last time I Cc-ed them - anyway I added linux-kernel to Cc again here.

I attach below a slightly modified version of the previous patch, which lets for more exact scheduling, but could be more vulnerable for this bug. Alas, even if it works, it still could be not the final solution of this problem.

Quoted 10 lines

Also attach crash log of lash crash PC 1:

[10610.110729] BUG: NMI Watchdog detected LOCKUP on CPU1, ip c01fd939,
registers:
[10610.110729] Modules linked in: netconsole e1000e i2c_i801 i2c_core e1000
[10610.110729]
[10610.110729] Pid: 0, comm: swapper Not tainted (2.6.26.6-fw #1)
[10610.110729] EIP: 0060:[<c01fd939>] EFLAGS: 00000082 CPU: 1
[10610.110729] EIP is at rb_insert_color+0x19/0xc0

Thanks,
Jarek P.

--- (testing patch #2)

 net/sched/sch_htb.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

net/sched/sch_htb.c +7 -1

index 30c999c..ff9e965 100644
@@ -162,6 +162,7 @@ struct htb_sched {
 
 	int rate2quantum;	/* quant = rate / rate2quantum */
 	psched_time_t now;	/* cached dequeue time */
+	psched_time_t next_watchdog;
 	struct qdisc_watchdog watchdog;
 
 	/* non shaped skbs; let them go directly thru */

@@ -920,7 +921,11 @@ static struct sk_buff *htb_dequeue(struct Qdisc *sch)
 		}
 	}
 	sch->qstats.overlimits++;
-	qdisc_watchdog_schedule(&q->watchdog, next_event);
+	if (q->next_watchdog < q->now || next_event <=
+	     q->next_watchdog - PSCHED_TICKS_PER_SEC / (10 * HZ)) {
+		qdisc_watchdog_schedule(&q->watchdog, next_event);
+		q->next_watchdog = next_event;
+	}
 fin:
 	return skb;
 }

@@ -973,6 +978,7 @@ static void htb_reset(struct Qdisc *sch)
 		}
 	}
 	qdisc_watchdog_cancel(&q->watchdog);
+	q->next_watchdog = 0;
 	__skb_queue_purge(&q->direct_queue);
 	sch->q.qlen = 0;
 	memset(q->row, 0, sizeof(q->row));

Hello again! Sorry for long away.

I was go away from this work for long time.

May we return to this bug?
Servers at last stable kernel 2.6.27.8
HZ=1000, HR=off, DynamicTicks=off, hysteresis=1
Sorry - no patched, update do not i. Do you have fresh patches or ideas for tests?

Also debug "Debug object operations" and "Debug timer objects" is on but i not see additional information at dump.

Thanks!

That dump:

[ 1500.932813] BUG: NMI Watchdog detected LOCKUP on CPU3, ip c0208ed9,
registers:
[ 1500.932813] Modules linked in: cls_u32 sch_sfq sch_htb netconsole
e1000e e1000 i2c_i801
[ 1500.932813]
[ 1500.932813] Pid: 0, comm: swapper Not tainted (2.6.27-gentoo-r5-fw #1)
[ 1500.932813] EIP: 0060:[<c0208ed9>] EFLAGS: 00000082 CPU: 3
[ 1500.932813] EIP is at rb_insert_color+0x29/0xf0

net/sched/sch_htb.c

Quoted 7 lines

--- (testing patch #2)

 net/sched/sch_htb.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

net/sched/sch_htb.c +0 -0

index 30c999c..ff9e965 100644

Quoted 1 line · Badalian Vyacheslav

Hello again! Sorry for long away.

Hi!

Quoted 7 lines

I was go away from this work for long time.

May we return to this bug?
Servers at last stable kernel 2.6.27.8
HZ=1000, HR=off, DynamicTicks=off, hysteresis=1
Sorry - no patched, update do not i. Do you have fresh patches or ideas for tests?

Not much, but I can have if you only are willing to test them... I attach below a patch which combines 2 patches I sent yesterday to netdev (PATCH 7/6 and 8/6) vs. 2.6.27.7 (named testing patch #3 here).

You can still try the testing patch #2 I sent previously (quoted below) with or without this new #3 patch.

Quoted 16 lines · Badalian Vyacheslav

Quoted 3 lines

Hello again! Sorry for long away.

Hi!

Quoted 8 lines

I was go away from this work for long time.

May we return to this bug?
Servers at last stable kernel 2.6.27.8
HZ=1000, HR=off, DynamicTicks=off, hysteresis=1
Sorry - no patched, update do not i. Do you have fresh patches or ideas for tests?

Quoted 8 lines

Not much, but I can have if you only are willing to test them... I attach below a patch which combines 2 patches I sent yesterday to netdev (PATCH 7/6 and 8/6) vs. 2.6.27.7 (named testing patch #3 here).

You can still try the testing patch #2 I sent previously (quoted below) with or without this new #3 patch.

Here is another idea worth checking (instead of patch #2).

Jarek P.

--- (testing patch #4)

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -0

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;

Quoted 16 lines · Badalian Vyacheslav

Quoted 3 lines

Hello again! Sorry for long away.

Hi!

Quoted 8 lines

I was go away from this work for long time.

May we return to this bug?
Servers at last stable kernel 2.6.27.8
HZ=1000, HR=off, DynamicTicks=off, hysteresis=1
Sorry - no patched, update do not i. Do you have fresh patches or ideas for tests?

Quoted 8 lines

Not much, but I can have if you only are willing to test them... I attach below a patch which combines 2 patches I sent yesterday to netdev (PATCH 7/6 and 8/6) vs. 2.6.27.7 (named testing patch #3 here).

You can still try the testing patch #2 I sent previously (quoted below) with or without this new #3 patch.

Here is another idea worth checking (instead of patch #2).

Jarek P.

--- (testing patch #4)

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -0

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;

Hello
result: Patch 2+3 = uptime 7 days without crashes. May i revert patches and try single new patch?

Quoted 47 lines · Jarek Poplawski

Quoted 16 lines · Badalian Vyacheslav

Quoted 3 lines

Hello again! Sorry for long away.

Hi!

Quoted 8 lines

I was go away from this work for long time.

May we return to this bug?
Servers at last stable kernel 2.6.27.8
HZ=1000, HR=off, DynamicTicks=off, hysteresis=1
Sorry - no patched, update do not i. Do you have fresh patches or ideas for tests?

Quoted 8 lines

Not much, but I can have if you only are willing to test them... I attach below a patch which combines 2 patches I sent yesterday to netdev (PATCH 7/6 and 8/6) vs. 2.6.27.7 (named testing patch #3 here).

You can still try the testing patch #2 I sent previously (quoted below) with or without this new #3 patch.

Here is another idea worth checking (instead of patch #2).

Jarek P.

--- (testing patch #4)

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -0

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;

Quoted 5 lines

Hello
result: Patch 2+3 = uptime 7 days without crashes. May i revert patches and try single new patch?

Here is my current opinion on this bug:

1) I'm almost sure it's not a htb, but hrtimers bug (some race),

2) the htb patches you've tested are not "the proper" way of fixing it; I see substantial changes in hrtimers code in the "-tip" tree (probably for 2.6.29), which, probably, you'll be advised by hrtimers maintainers to try, and I guess, it's not easy on a production system,

So, it's up to you:

1) since these patches work for you, you can stop with testing and wait with these patched kernels until 2.6.29 (I can propose this #2 patch as a temporary fix then),

2) for curiosity you could try this patch #4 alone on one box first (after reverting at least patch #2), but again: if it works, it could be only treated as a temporary hack, and alternative of #2.

Thanks,
Jarek P.

Quoted 52 lines

Quoted 52 lines · Jarek Poplawski

Quoted 21 lines · Badalian Vyacheslav

Quoted 5 lines

Hello again! Sorry for long away.

Hi!

Quoted 9 lines

I was go away from this work for long time.

May we return to this bug?
Servers at last stable kernel 2.6.27.8
HZ=1000, HR=off, DynamicTicks=off, hysteresis=1
Sorry - no patched, update do not i. Do you have fresh patches or ideas for tests?

Quoted 9 lines

Not much, but I can have if you only are willing to test them... I attach below a patch which combines 2 patches I sent yesterday to netdev (PATCH 7/6 and 8/6) vs. 2.6.27.7 (named testing patch #3 here).

You can still try the testing patch #2 I sent previously (quoted below) with or without this new #3 patch.

Here is another idea worth checking (instead of patch #2).

Jarek P.

--- (testing patch #4)

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -0

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;

Thanks for all Jarek!

Quoted 3 lines · Badalian Vyacheslav

Hello
result: Patch 2+3 = uptime 7 days without crashes. May i revert patches and try single new patch?

Here is my current opinion on this bug:

1) I'm almost sure it's not a htb, but hrtimers bug (some race),

2) the htb patches you've tested are not "the proper" way of fixing it; I see substantial changes in hrtimers code in the "-tip" tree (probably for 2.6.29), which, probably, you'll be advised by hrtimers maintainers to try, and I guess, it's not easy on a production system,

So, it's up to you:

1) since these patches work for you, you can stop with testing and wait with these patched kernels until 2.6.29 (I can propose this #2 patch as a temporary fix then),

2) for curiosity you could try this patch #4 alone on one box first (after reverting at least patch #2), but again: if it works, it could be only treated as a temporary hack, and alternative of #2.

Per Jarek's suggestion in bugzilla, I ran 2.6.28 plus Peter Zijlstra's "hrtimer: removing all ur callback modes" patches dated 2008-11-25, 2008-12-04 and 2008-12-08. Uptime was 2 days 22 hours before I hit what appears to be an unrelated bug related to the IPv6 FIB. (Separately reported with subject 'panic with 2.6.28 while doing "ip -6 route"'.)

On Wed, Jan 14, 2009 at 02:43:13AM +0000, Chris Caputo wrote:
...

Quoted 5 lines

Per Jarek's suggestion in bugzilla, I ran 2.6.28 plus Peter Zijlstra's "hrtimer: removing all ur callback modes" patches dated 2008-11-25, 2008-12-04 and 2008-12-08. Uptime was 2 days 22 hours before I hit what appears to be an unrelated bug related to the IPv6 FIB. (Separately reported with subject 'panic with 2.6.28 while doing "ip -6 route"'.)

Chris, this is a very good news, thanks for testing this!

Peter, great work! IMHO these patches should go to the stable (at least 2.6.28).

Thanks,
Jarek P.

I will try that patches too, when i got this message, after 3 minutes i got crash of my router :-) after working around 17 hours :-(
There is 3 of them by the way, 2 fixes also.

hrtimer: removing all ur callback modes
hrtimer: removing all ur callback modes, fix hotplug
hrtimer: removing all ur callback modes, fix

Quoted 4 lines · Denys Fedoryschenko

I will try that patches too, when i got this message, after 3 minutes i got crash of my router :-) after working around 17 hours :-( There is 3 of them by the way, 2 fixes also.

Quoted 3 lines

hrtimer: removing all ur callback modes
hrtimer: removing all ur callback modes, fix hotplug
hrtimer: removing all ur callback modes, fix

Yes, it's a very nasty bug which, BTW you reported the first AFAIK, but I hope it's really gone with these patches.

I will try that patches too, when i got this message, after 3 minutes i got crash of my router :-) after working around 17 hours :-(
There is 3 of them by the way, 2 fixes also.

hrtimer: removing all ur callback modes
hrtimer: removing all ur callback modes, fix hotplug
hrtimer: removing all ur callback modes, fix

Quoted 7 lines · Denys Fedoryschenko

I will try that patches too, when i got this message, after 3 minutes i got crash of my router :-) after working around 17 hours :-(
There is 3 of them by the way, 2 fixes also.

hrtimer: removing all ur callback modes
hrtimer: removing all ur callback modes, fix hotplug
hrtimer: removing all ur callback modes, fix

I'm afraid its a bit more than that:

ca109491f612aab5c8152207631c0444f63da97f
37810659ea7d9572c5ac284ade272f806ef8f788
a0a99b227da57f81319dd239bc4de811b0f530ec
b2e3c0adec918ea22b6c9d7c76193dd3aaba9bd4
8bdec955b0da2ffbd10eb9b200651dd1f9e366f2
d5fd43c4ae04523e1dcd7794f9c511b289851350
731a55ba0f17064f85903b7bf8e24849ec6cfa20
a6037b61c2f5fc99c57c15b26d7cfa58bbb34008
e3f1d883740b09e5116d4d4e30a6a6987264a83c
82c5b7b527ccc4b5d3cf832437e842f9d2920a79

and

6e5c172cf7ca1ab878cc6a6a4c1d52fef60f3ee0

Also, all this is rather invasive and large, so I'm not sure it will meet the -stable criteria.

I'm afraid its a bit more than that:

ca109491f612aab5c8152207631c0444f63da97f
37810659ea7d9572c5ac284ade272f806ef8f788
a0a99b227da57f81319dd239bc4de811b0f530ec
b2e3c0adec918ea22b6c9d7c76193dd3aaba9bd4
8bdec955b0da2ffbd10eb9b200651dd1f9e366f2
d5fd43c4ae04523e1dcd7794f9c511b289851350
731a55ba0f17064f85903b7bf8e24849ec6cfa20
a6037b61c2f5fc99c57c15b26d7cfa58bbb34008
e3f1d883740b09e5116d4d4e30a6a6987264a83c
82c5b7b527ccc4b5d3cf832437e842f9d2920a79

and

6e5c172cf7ca1ab878cc6a6a4c1d52fef60f3ee0

Also, all this is rather invasive and large, so I'm not sure it will meet the -stable criteria.

Quoted 17 lines

I'm afraid its a bit more than that:

ca109491f612aab5c8152207631c0444f63da97f
37810659ea7d9572c5ac284ade272f806ef8f788
a0a99b227da57f81319dd239bc4de811b0f530ec
b2e3c0adec918ea22b6c9d7c76193dd3aaba9bd4
8bdec955b0da2ffbd10eb9b200651dd1f9e366f2
d5fd43c4ae04523e1dcd7794f9c511b289851350
731a55ba0f17064f85903b7bf8e24849ec6cfa20
a6037b61c2f5fc99c57c15b26d7cfa58bbb34008
e3f1d883740b09e5116d4d4e30a6a6987264a83c
82c5b7b527ccc4b5d3cf832437e842f9d2920a79

and

6e5c172cf7ca1ab878cc6a6a4c1d52fef60f3ee0

Quoted 2 lines

Also, all this is rather invasive and large, so I'm not sure it will meet the -stable criteria.

Then i will try to run 2.6.29-rc1-git4. Kind of risky to run rc1 on loaded router, but i am hitting this bug with hrtimers hardly, so no choice for me seems. And i cannot disable hrtimers because of bad shaper precision. Thanks a lot for info and patches :-)

Quoted 17 lines · Denys Fedoryschenko

Quoted 17 lines

I'm afraid its a bit more than that:

ca109491f612aab5c8152207631c0444f63da97f
37810659ea7d9572c5ac284ade272f806ef8f788
a0a99b227da57f81319dd239bc4de811b0f530ec
b2e3c0adec918ea22b6c9d7c76193dd3aaba9bd4
8bdec955b0da2ffbd10eb9b200651dd1f9e366f2
d5fd43c4ae04523e1dcd7794f9c511b289851350
731a55ba0f17064f85903b7bf8e24849ec6cfa20
a6037b61c2f5fc99c57c15b26d7cfa58bbb34008
e3f1d883740b09e5116d4d4e30a6a6987264a83c
82c5b7b527ccc4b5d3cf832437e842f9d2920a79

and

6e5c172cf7ca1ab878cc6a6a4c1d52fef60f3ee0

Quoted 8 lines

Quoted 2 lines

Also, all this is rather invasive and large, so I'm not sure it will meet the -stable criteria.

Then i will try to run 2.6.29-rc1-git4. Kind of risky to run rc1 on loaded router, but i am hitting this bug with hrtimers hardly, so no choice for me seems. And i cannot disable hrtimers because of bad shaper precision. Thanks a lot for info and patches :-)

Denys, since these three patches worked for Chris, I guess it's safer to stay with 2.6.28 anyway (or my patches mentioned earlier in this thread). rc1 is really not stable...

Quoted 4 lines

Denys, since these three patches worked for Chris, I guess it's safer to stay with 2.6.28 anyway (or my patches mentioned earlier in this thread). rc1 is really not stable...

Yeah, but those three patches have some holes in them still, so I cannot recommend just patching in those three.

Quoted 27 lines · Jarek Poplawski

Quoted 25 lines · Denys Fedoryschenko

Quoted 13 lines

I'm afraid its a bit more than that:

ca109491f612aab5c8152207631c0444f63da97f
37810659ea7d9572c5ac284ade272f806ef8f788
a0a99b227da57f81319dd239bc4de811b0f530ec
b2e3c0adec918ea22b6c9d7c76193dd3aaba9bd4
8bdec955b0da2ffbd10eb9b200651dd1f9e366f2
d5fd43c4ae04523e1dcd7794f9c511b289851350
731a55ba0f17064f85903b7bf8e24849ec6cfa20
a6037b61c2f5fc99c57c15b26d7cfa58bbb34008
e3f1d883740b09e5116d4d4e30a6a6987264a83c
82c5b7b527ccc4b5d3cf832437e842f9d2920a79

Quoted 6 lines

and

6e5c172cf7ca1ab878cc6a6a4c1d52fef60f3ee0

Also, all this is rather invasive and large, so I'm not sure it will meet the -stable criteria.

Then i will try to run 2.6.29-rc1-git4. Kind of risky to run rc1 on loaded router, but i am hitting this bug with hrtimers hardly, so no choice for me seems. And i cannot disable hrtimers because of bad shaper precision. Thanks a lot for info and patches :-)

Quoted 3 lines

Denys, since these three patches worked for Chris, I guess it's safer to stay with 2.6.28 anyway (or my patches mentioned earlier in this thread). rc1 is really not stable...

Yeah, but those three patches have some holes in them still, so I cannot recommend just patching in those three.

OK, I hope Denys can apply more, but what about others? Without any patches the hole seems to be much bigger.

Quoted 2 lines · Jarek Poplawski

OK, I hope Denys can apply more, but what about others? Without any patches the hole seems to be much bigger.

OK, I read most of this thread on netdev, but didn't find a clear clue on the specific hrtimer insertion race.

Do you have any clear ideas or should I poke at the htb/hrtimer code a little?

Quoted 2 lines · Jarek Poplawski

OK, I hope Denys can apply more, but what about others? Without any patches the hole seems to be much bigger.

OK, I read most of this thread on netdev, but didn't find a clear clue on the specific hrtimer insertion race.

Do you have any clear ideas or should I poke at the htb/hrtimer code a little?

Quoted 7 lines · Peter Zijlstra

Quoted 2 lines · Jarek Poplawski

OK, I hope Denys can apply more, but what about others? Without any patches the hole seems to be much bigger.

OK, I read most of this thread on netdev, but didn't find a clear clue on the specific hrtimer insertion race.

There is something at the beginning of this thread, plus earlier threads mostly with Denys as sender, and "htb bug" in the subject.

Quoted 1 line

Quoted 3 lines

Do you have any clear ideas or should I poke at the htb/hrtimer code a little?

....And htb code is htb_dequeue(): qdisc_watchdog_schedule().

Quoted 13 lines · Jarek Poplawski

Quoted 7 lines · Peter Zijlstra

Quoted 2 lines · Jarek Poplawski

OK, I hope Denys can apply more, but what about others? Without any patches the hole seems to be much bigger.

OK, I read most of this thread on netdev, but didn't find a clear clue on the specific hrtimer insertion race.

There is something at the beginning of this thread, plus earlier threads mostly with Denys as sender, and "htb bug" in the subject.

Quoted 1 line

Quoted 5 lines

Quoted 3 lines

Do you have any clear ideas or should I poke at the htb/hrtimer code a little?

....And htb code is htb_dequeue(): qdisc_watchdog_schedule().

Right, found all that...

Can't spot anything obviously wrong though.. hrtimer_start*() does remove_hrtimer() which checks STATE_ENQUEUED, STATE_PENDING and pulls it off the relevant list before it continues the enqueue.

However a loop in enqueue_hrtimer() would suggest a corrupted RB-tree, but I cannot find an RB-op that doesn't hold base-lock.

Right, found all that...

Can't spot anything obviously wrong though.. hrtimer_start*() does remove_hrtimer() which checks STATE_ENQUEUED, STATE_PENDING and pulls it off the relevant list before it continues the enqueue.

However a loop in enqueue_hrtimer() would suggest a corrupted RB-tree, but I cannot find an RB-op that doesn't hold base-lock.

On Wed, Jan 14, 2009 at 03:28:03PM +0100, Peter Zijlstra wrote:
....

Quoted 9 lines

Right, found all that...

Can't spot anything obviously wrong though.. hrtimer_start*() does remove_hrtimer() which checks STATE_ENQUEUED, STATE_PENDING and pulls it off the relevant list before it continues the enqueue.

However a loop in enqueue_hrtimer() would suggest a corrupted RB-tree, but I cannot find an RB-op that doesn't hold base-lock.

I've revisited it yesterday, and if I don't miss something, there is possible a scenario similar to this:

cpu1: cpu2:

run_hrtimer_pending
spin_unlock
restart = fn(timer)

hrtimer_start
enqueue_hrtimer

hrtimer_start
remove_hrtimer
(the HRTIMER_STATE_CALLBACK is removed)

switch_hrtimer_base
spin_lock
(not this hrtimer's anymore) __remove_hrtimer
list_add_tail enqueue_hrtimer

On Thu, 2009-01-15 at 09:01 +0000, Jarek Poplawski wrote:

Quoted 5 lines

spin_lock
(not this hrtimer's anymore)
__remove_hrtimer
list_add_tail enqueue_hrtimer

(looking at .28 code)

run_hrtimer_pending() reads like:

while (pending timers) {
__remove_hrtimer(timer, HRTIMER_STATE_CALLBACK); spin_unlock(&cpu_base->lock);

  fn(timer);

spin_lock(&cpu_base->lock);
timer->state &= ~HRTIMER_STATE_CALLBACK; // _should_ result in HRTIMER_STATE_INACTIVE if (HRTIMER_RESTART)
re-queue

  else if (timer->state != INACTIVE) {
    // so another cpu re-queued this timer _while_ we were executing it.
    if (timer is first && !reprogramm) {
      __remove_hrtimer(timer, HRTIMER_STATE_PENDING);
      list_add_tail(timer, &cb_pending);
    }

}
}

So in the window where we drop the lock, one can, as you said, have another cpu requeue the timer, but the rb_entry and list_entry are free, so it should not cause the data corruption we're seeing.

Quoted 7 lines · Denys Fedoryschenko

I will try that patches too, when i got this message, after 3 minutes i got crash of my router :-) after working around 17 hours :-(
There is 3 of them by the way, 2 fixes also.

hrtimer: removing all ur callback modes
hrtimer: removing all ur callback modes, fix hotplug
hrtimer: removing all ur callback modes, fix

I'm afraid its a bit more than that:

ca109491f612aab5c8152207631c0444f63da97f
37810659ea7d9572c5ac284ade272f806ef8f788
a0a99b227da57f81319dd239bc4de811b0f530ec
b2e3c0adec918ea22b6c9d7c76193dd3aaba9bd4
8bdec955b0da2ffbd10eb9b200651dd1f9e366f2
d5fd43c4ae04523e1dcd7794f9c511b289851350
731a55ba0f17064f85903b7bf8e24849ec6cfa20
a6037b61c2f5fc99c57c15b26d7cfa58bbb34008
e3f1d883740b09e5116d4d4e30a6a6987264a83c
82c5b7b527ccc4b5d3cf832437e842f9d2920a79

and

6e5c172cf7ca1ab878cc6a6a4c1d52fef60f3ee0

Also, all this is rather invasive and large, so I'm not sure it will meet the -stable criteria.

That said, I would not recommend just the three for -stable unless they get a much wider amount of testing, on multiple platforms. I don't see that as likely to happen, plus Peter says they are incomplete, so maybe it is just best to recommend that 2.6.28 users getting crashes while using HTB try these specific patches at first, and then the rest of the patches if they do not work.

...

Quoted 7 lines

That said, I would not recommend just the three for -stable unless they get a much wider amount of testing, on multiple platforms. I don't see that as likely to happen, plus Peter says they are incomplete, so maybe it is just best to recommend that 2.6.28 users getting crashes while using HTB try these specific patches at first, and then the rest of the patches if they do not work.

The main problem is my patches, at least the tested ones, harm htb's exactness, and I doubt I could convince anybody to merege them, at least before your case. It was only reported by two users here (plus one more on private), and looked like something very rare. After your report it looks much more necessary.

If there is nothing better, I can recommend it, but IMHO the best candidate for this is the testing patch #4 from this thread, which alas wasn't even tested... So, Chris, if you could give it a try in the meantime (without any other patches)?

Thanks,
Jarek P.

(resend testing patch #4 - for 2.6.27 or 2.6.28)
---

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

Hello all.
I also can say this, maybe its help:
At old kernels my servers do 100% soft interupt if traffic more 600mbs. Without your patches at new kernel i get crash only at heavy network load PCs (more then 400mbs-500mbs). Servers that get 100-200 mbs not crashed long time.
I remember that i not test patch #4, because you sat what its only another way to temporary fix and mainline problem in hrtimer , but i try turn on HiRes and Dynamic Tics in kernel - its not help for me.

Slavon, I'm very glad of your testing and reporting, and I never expected you test anything on production, when something else works for you. On the other hand, what works for you could be unacceptable for other users, who don't have such crashes, and prefer better exactness. So it's hard to recommend such patches if there is not enough people interested in them.

Best regards,
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jarek, can you give me exact name of patch or link to it? I will test it on production.
And i am interested in searching, what is a problem.

...

Quoted 4 lines

Jarek, can you give me exact name of patch or link to it? I will test it on production.
And i am interested in searching, what is a problem.

If there is nothing better I could recommend the patch below for -stable (2.6.28), when it's tested.

Thanks,
Jarek P.
---

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

Jarek, i easy can test patch 4 without 2+3 if it needed at heavy production server. I use dynamic routing and if server crashed - traffic go to another pc after few seconds. I not test it because you say that its not needed if for me help 3+2. I apply it today and test at few servers.

On Thu, Jan 15, 2009 at 12:40:59PM +0300, Badalian Vyacheslav wrote:
...

Quoted 5 lines

Jarek, i easy can test patch 4 without 2+3 if it needed at heavy production server. I use dynamic routing and if server crashed - traffic go to another pc after few seconds. I not test it because you say that its not needed if for me help 3+2. I apply it today and test at few servers.

Yes, I thought it's so rare we can wait till 2.6.29, while you had this patched, and Denys seemed to not see it after changing his sever. But I have changed my mind after seeing Chris's report...

I'm not sure if Peter would recomend something better yet, so I guess you could wait with this testing for his opinion.

Thanks,
Jarek P.

I didn't notice a bug, just because all that time i was using it without hrtimers (because of nmi_watchdog).

I didn't notice a bug, just because all that time i was using it without hrtimers (because of nmi_watchdog).

Denys, maybe I forgot something, but I think you reported this bug with nmi_watchdog too...

Jarek P.

On Wed, Jan 14, 2009 at 06:02:04PM +0000, Chris Caputo wrote:
...

Quoted 6 lines

That said, I would not recommend just the three for -stable unless they get a much wider amount of testing, on multiple platforms. I don't see that as likely to happen, plus Peter says they are incomplete, so maybe it is just best to recommend that 2.6.28 users getting crashes while using HTB try these specific patches at first, and then the rest of the patches if they do not work.

The main problem is my patches, at least the tested ones, harm htb's exactness, and I doubt I could convince anybody to merege them, at least before your case. It was only reported by two users here (plus one more on private), and looked like something very rare. After your report it looks much more necessary.

If there is nothing better, I can recommend it, but IMHO the best candidate for this is the testing patch #4 from this thread, which alas wasn't even tested... So, Chris, if you could give it a try in the meantime (without any other patches)?

Thanks,
Jarek P.

(resend testing patch #4 - for 2.6.27 or 2.6.28)
---

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

Quoted 8 lines · Jarek Poplawski

On Wed, Jan 14, 2009 at 06:02:04PM +0000, Chris Caputo wrote:
...

Quoted 6 lines

That said, I would not recommend just the three for -stable unless they get a much wider amount of testing, on multiple platforms. I don't see that as likely to happen, plus Peter says they are incomplete, so maybe it is just best to recommend that 2.6.28 users getting crashes while using HTB try these specific patches at first, and then the rest of the patches if they do not work.

I hope you can forgive me, but since I have something that works on a production machine, I am not in a position in which I can test this.

Quoted 9 lines · Chris Caputo

Quoted 8 lines · Jarek Poplawski

On Wed, Jan 14, 2009 at 06:02:04PM +0000, Chris Caputo wrote:
...

Quoted 6 lines

That said, I would not recommend just the three for -stable unless they get a much wider amount of testing, on multiple platforms. I don't see that as likely to happen, plus Peter says they are incomplete, so maybe it is just best to recommend that 2.6.28 users getting crashes while using HTB try these specific patches at first, and then the rest of the patches if they do not work.

...

Quoted 2 lines

I hope you can forgive me, but since I have something that works on a production machine, I am not in a position in which I can test this.

So, IOW, you don't recommend Peter's patches but prefere to use them, and recommend my patches, but you are afraid of using them...

Quoted 14 lines · Jarek Poplawski

Quoted 9 lines · Chris Caputo

Quoted 8 lines · Jarek Poplawski

On Wed, Jan 14, 2009 at 06:02:04PM +0000, Chris Caputo wrote:
...

Quoted 6 lines

That said, I would not recommend just the three for -stable unless they get a much wider amount of testing, on multiple platforms. I don't see that as likely to happen, plus Peter says they are incomplete, so maybe it is just best to recommend that 2.6.28 users getting crashes while using HTB try these specific patches at first, and then the rest of the patches if they do not work.

...

Quoted 2 lines

I hope you can forgive me, but since I have something that works on a production machine, I am not in a position in which I can test this.

Quoted 2 lines

So, IOW, you don't recommend Peter's patches but prefere to use them, and recommend my patches, but you are afraid of using them...

Sorry, Chris! It looks like I misunderstood, and you recomend Peter's patches yet. (But I guess there are users relying on official kernel versions.)

I hope you can forgive me,

I didn't notice a bug, just because all that time i was using it without hrtimers (because of nmi_watchdog).

I wish I had a repro scenario in a non-production environment, so I could help out further with this. If I did, I would test without nmi_watchdog while trying just 2.6.28 and Jarek's #4 patch.

On Thu, Jan 15, 2009 at 12:00:50PM +0000, Chris Caputo wrote:
...

Quoted 3 lines

I wish I had a repro scenario in a non-production environment, so I could help out further with this. If I did, I would test without nmi_watchdog while trying just 2.6.28 and Jarek's #4 patch.

I think this watchdog doesn't matter too much yet. Probably it's more about hardware (smp, maybe kind of hw timer), and high traffic vs. htb rules.

Quoted 12 lines · Chris Caputo

...

Quoted 4 lines

I wish I had a repro scenario in a non-production environment, so I could help out further with this. If I did, I would test without nmi_watchdog while trying just 2.6.28 and Jarek's #4 patch.

I think this watchdog doesn't matter too much yet. Probably it's more about hardware (smp, maybe kind of hw timer), and high traffic vs. htb rules.

Per Thomas' comment in http://bugzilla.kernel.org/show_bug.cgi?id=10944
regarding broken hrtimers:

--
nmi_watchdog=1 disables the local apic timers and therefor highres/nohz

Try to use nmi_watchdog=2 instead.
--

Chris

(resend testing patch #4 - for 2.6.27 or 2.6.28)

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -1

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;
--

To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

From: Jarek Poplawski <[email protected]>
Date: Thu, 15 Jan 2009 06:53:22 +0000

Quoted 1 line

(resend testing patch #4 - for 2.6.27 or 2.6.28)

Jarek, if you deem that this is in fact what we should submit for -stable please give me a submission with a suitable commit message and signoffs, and I will queue it up for -stable.

Quoted 5 lines

From: Jarek Poplawski <[email protected]>
Date: Thu, 15 Jan 2009 06:53:22 +0000

Quoted 7 lines

Quoted 2 lines

(resend testing patch #4 - for 2.6.27 or 2.6.28)

Jarek, if you deem that this is in fact what we should submit for -stable please give me a submission with a suitable commit message and signoffs, and I will queue it up for -stable.

It looks like this should be needed, but I think it's better to wait 2 or 3 days for "Tested-by" from Denys and/or maybe Vyacheslav yet. (I hoped they would rather test some hrtimers patch, but it looks like Peter was busy.)

Thanks,
Jarek P.
-----------------> (needed only for -stables: 2.6.28 and older)

pkt_sched: sch_htb: Fix deadlock in hrtimers triggered by HTB

Most probably there is a (still unproven) race in hrtimers (before 2.6.29 kernels), which causes a corruption of hrtimers rbtree. This patch doesn't fix it, but should let HTB avoid triggering the bug.

Reported-by Denys Fedoryschenko <[email protected]>

Reported-by Badalian Vyacheslav <[email protected]>

Reported-by Chris Caputo <[email protected]>

Signed-off-by Jarek Poplawski <[email protected]>

---

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -0

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;

Tested 2 days (weekend is stress days for this bug, because its have many traffic) at 3 servers. Fly normal. For completed test need up to 7 days, but i think (imho) this patch not break any functional and may safely added to stable. 100% help for me patch # 2+3.

Quoted 48 lines · David Miller

Quoted 5 lines

From: Jarek Poplawski <[email protected]>
Date: Thu, 15 Jan 2009 06:53:22 +0000

Quoted 7 lines

Quoted 2 lines

(resend testing patch #4 - for 2.6.27 or 2.6.28)

Jarek, if you deem that this is in fact what we should submit for -stable please give me a submission with a suitable commit message and signoffs, and I will queue it up for -stable.

It looks like this should be needed, but I think it's better to wait 2 or 3 days for "Tested-by" from Denys and/or maybe Vyacheslav yet. (I hoped they would rather test some hrtimers patch, but it looks like Peter was busy.)

Thanks,
Jarek P.
-----------------> (needed only for -stables: 2.6.28 and older)

pkt_sched: sch_htb: Fix deadlock in hrtimers triggered by HTB

Most probably there is a (still unproven) race in hrtimers (before 2.6.29 kernels), which causes a corruption of hrtimers rbtree. This patch doesn't fix it, but should let HTB avoid triggering the bug.

Reported-by Denys Fedoryschenko <[email protected]>

Reported-by Badalian Vyacheslav <[email protected]>

Reported-by Chris Caputo <[email protected]>

Signed-off-by Jarek Poplawski <[email protected]>

---

diff -Nurp a2.6.27.7/net/sched/sch_htb.c b2.6.27.7/net/sched/sch_htb.c

net/sched/sch_htb.c +1 -0

@@ -924,6 +924,7 @@ static struct sk_buff *htb_dequeue(struc
 		}
 	}
 	sch->qstats.overlimits++;
+	qdisc_watchdog_cancel(&q->watchdog);
 	qdisc_watchdog_schedule(&q->watchdog, next_event);
 fin:
 	return skb;

Quoted 4 lines

Tested 2 days (weekend is stress days for this bug, because its have many traffic) at 3 servers. Fly normal. For completed test need up to 7 days, but i think (imho) this patch not break any functional and may safely added to stable. 100% help for me patch # 2+3.

Thank you very much, Slavon! (2+3 are of course better tested, but I hope this #4 is more appropriate for -stable.)

Quoted 1 line · Badalian Vyacheslav

Hello all!

Hello Slavon,

Quoted 3 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

Quoted 3 lines

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

Quoted 8 lines · Jarek Poplawski

Quoted 1 line · Badalian Vyacheslav

Hello all!

Hello Slavon,

Quoted 3 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

Quoted 7 lines

Quoted 3 lines

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

I doubt its hardware related, whats happening is that the hrtimer insertion gets into an endless loop because the rb tree (or node) apparently has a loop. I went through the qdiscs' use of hrtimers again, but can't spot any error there.

Denys, did your systems also have CONFIG_HIGH_RES_TIMERS=n?

Quoted 22 lines · Patrick McHardy

Quoted 8 lines · Jarek Poplawski

Quoted 1 line · Badalian Vyacheslav

Hello all!

Hello Slavon,

Quoted 3 lines

Please look to if you have time:
http://bugzilla.kernel.org/show_bug.cgi?id=11718

Quoted 7 lines

Quoted 3 lines

We have deadlocks at few PC one times in week.
I can test any patches to detect and fix problem.
Now i test 2.6.27-rc kernel at one PC.

A similar bug was reported by Denys Fedoryshchenko but it wasn't fully diagnosed. Anyway it looks like hardware dependent. The patch below can sometimes help. 2.6.27 may have this fixed too (some other way).

I doubt its hardware related, whats happening is that the hrtimer insertion gets into an endless loop because the rb tree (or node) apparently has a loop. I went through the qdiscs' use of hrtimers again, but can't spot any error there.

Quoted 1 line

Denys, did your systems also have CONFIG_HIGH_RES_TIMERS=n?

Badalian, please try enabling CONFIG_DEBUG_OBJECTS and post the results, if any.

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html