Skip to contentSkip to conversation

Lockups on 2.4.1

Linux-ia64 Jack Steiner 3 participants 5 messages Feb 22, 2001 – Feb 28, 2001

lore
Share & tools
lore
#0 lore raw
Quoted 16 lines
Quoted 7 lines

Anyway, I have ITPs connected to the IBM hardware and have noticed that when the lockup occurs, and we lose video, at least one of the CPUs is executing in flush_tlb_no_ptcg() or handle_IPI(), in the 'do' loop where TLB
entries are being purged. What I have observed is that the end address and the start address are in completely different regions. Usually, the start address

Quoted 8 lines

is in region register 1 (address of 0x2000XXXXXXXXXXXX) and the end address is in region register 3 (address of 0x6000XXXXXXXXXXXX). I don't know if this
is the same problem I am seeing on the Lion, but I plan to connect and ITP and
a serial console (although we haven't been able to get one to work yet on the
Lion with BIOS 71) to see if the symptoms are the same.

Quoted 5 lines

FWIW, we have seen EXACTLY the same hang running here on our system. The start/end addresses for the purge cross region boundaries.

We are running a 2.4.0 kernel.

I found a problem that was causing the lockup described above & I suspect this may responsible for some of the other hangs various folks have seen.

There is code in flush_tlb_no_ptcg() that resends the IPI if other cpus have not responded within a short time. If this code get invoked, then it is possible for flush_cpu_count to get corrupted. When that happens, a cpu can be executing in handle_IPI() while flush_start/flush_end are changing. A cpu can pick up a non-matching flush_start/flush_end. This leads to hangs or lost TLB flushes.

To verify that this could cause the hang, I changed the timeout in flush_tlb_no_ptcg() from 40000UL to 400UL. I hung before getting to multiuser mode with flush_start/flush_end in different regions.

Here is the patch I used. Note: this is against 2.4.0,

arch/ia64/kernel/smp.c +10 -0
@@ -321,6 +321,16 @@
 {
 	send_IPI_allbutself(IPI_FLUSH_TLB);
 }
+
+void
+smp_resend_flush_tlb(void)
+{
+	/*
+	 * Really need a null IPI but since this rarely should happen &
+	 * since this code will go away, lets not add one.
+	 */
+	send_IPI_allbutself(IPI_RESCHEDULE);
+}
 #endif	/* !CONFIG_ITANIUM_PTCG */
 
 /*
arch/ia64/mm/tlb.c +3 -3
@@ -59,6 +59,7 @@
 flush_tlb_no_ptcg (unsigned long start, unsigned long end, unsigned long nbits)
 {
 	extern void smp_send_flush_tlb (void);
+	extern void smp_resend_flush_tlb (void);
 	unsigned long saved_tpr = 0;
 	unsigned long flags;
 
@@ -101,9 +102,8 @@
 	{
 		unsigned long start = ia64_get_itc();
 		while (atomic_read(&flush_cpu_count) > 0) {
-			if ((ia64_get_itc() - start) > 40000UL) {
-				atomic_set(&flush_cpu_count, smp_num_cpus - 1);
-				smp_send_flush_tlb();
+			if ((ia64_get_itc() - start) > 400UL) {
+				smp_resend_flush_tlb();
 				start = ia64_get_itc();
 			}
 		}
Signature · 3 lines
Thanks

Jack Steiner    (651-683-5302)   (vnet 233-5302)      [email protected]
#1 lore raw

Jack,

Thanks for investigating the problem and the patch. The problem is happening because the timeout (40000UL) is not long enough. The processor is taking long time than this to complete the handle_IPI and processor doing the flush_tlb_no_ptcg is timing out and sending the IPI again. So, we should increase the timeout rather than decreasing the timeout to avoid extra reschedule IPIs.

Thanks,
Asit

Quoted 117 lines

-----Original Message-----
From: Jack Steiner [mailto:[email protected]]
Sent: Thursday, February 22, 2001 12:48 PM
To: [email protected]
Subject: Re: Re: Re: [Linux-ia64] Re: Lockups on 2.4.1

Quoted 1 line
Quoted 1 line

Anyway, I have ITPs connected to the IBM hardware and

have noticed that

Quoted 1 line
Quoted 1 line

when the lockup occurs, and we lose video, at least one

of the CPUs is

Quoted 1 line
Quoted 1 line

executing in flush_tlb_no_ptcg() or handle_IPI(), in the

'do' loop where

Quoted 2 lines
Quoted 2 lines

TLB
entries are being purged. What I have observed is that

the end address and

Quoted 1 line
Quoted 1 line

the start address are in completely different regions.

Usually, the start

Quoted 2 lines
Quoted 2 lines

address
is in region register 1 (address of 0x2000XXXXXXXXXXXX)

and the end address

Quoted 1 line
Quoted 1 line

is in region register 3 (address of 0x6000XXXXXXXXXXXX).

I don't know if

Quoted 2 lines
Quoted 2 lines

this
is the same problem I am seeing on the Lion, but I plan

to connect and ITP

Quoted 2 lines
Quoted 2 lines

and
a serial console (although we haven't been able to get

one to work yet on

Quoted 8 lines
Quoted 2 lines

the
Lion with BIOS 71) to see if the symptoms are the same.

FWIW, we have seen EXACTLY the same hang running here on our system. The start/end addresses for the purge cross region boundaries.

We are running a 2.4.0 kernel.

I found a problem that was causing the lockup described above & I suspect this
may responsible for some of the other hangs various folks have seen.

There is code in flush_tlb_no_ptcg() that resends the IPI if other cpus have not responded within a short time. If this code get invoked, then
it is possible for flush_cpu_count to get corrupted. When that happens, a cpu
can be executing in handle_IPI() while flush_start/flush_end are changing.
A cpu can pick up a non-matching flush_start/flush_end. This leads to hangs or
lost TLB flushes.

To verify that this could cause the hang, I changed the timeout in flush_tlb_no_ptcg() from 40000UL to 400UL. I hung before getting to multiuser mode
with flush_start/flush_end in different regions.

Here is the patch I used. Note: this is against 2.4.0,

arch/ia64/kernel/smp.c +10 -0
@@ -321,6 +321,16 @@
 {
 	send_IPI_allbutself(IPI_FLUSH_TLB);
 }
+
+void
+smp_resend_flush_tlb(void)
+{
+	/*
+	 * Really need a null IPI but since this rarely should happen &
+	 * since this code will go away, lets not add one.
+	 */
+	send_IPI_allbutself(IPI_RESCHEDULE);
+}
 #endif	/* !CONFIG_ITANIUM_PTCG */
 
 /*
arch/ia64/mm/tlb.c +3 -3
@@ -59,6 +59,7 @@
 flush_tlb_no_ptcg (unsigned long start, unsigned long end, 
 unsigned long nbits)
 {
 	extern void smp_send_flush_tlb (void);
+	extern void smp_resend_flush_tlb (void);
 	unsigned long saved_tpr = 0;
 	unsigned long flags;
@@ -101,9 +102,8 @@
 	{
 		unsigned long start = ia64_get_itc();
 		while (atomic_read(&flush_cpu_count) > 0) {
-			if ((ia64_get_itc() - start) > 40000UL) {
-				atomic_set(&flush_cpu_count, 
 smp_num_cpus - 1);
-				smp_send_flush_tlb();
+			if ((ia64_get_itc() - start) > 400UL) {
+				smp_resend_flush_tlb();
 				start = ia64_get_itc();
 			}

}

Signature · 9 lines
Thanks

Jack Steiner    (651-683-5302)   (vnet 233-5302)      [email protected]


_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64
#2 lore raw
In reply to Jack Steiner

OK, this makes sense: our systems have ptc.g enabled, which explains why we haven't seen this problem. I made the change of using smp_resend_flush_tlb() but also increased the timeout by a factor of 10.

Thanks,

	--david
Quoted 16 lines
Quoted 16 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line · Jack Steiner

Anyway, I have ITPs connected to the IBM hardware and have

noticed that > when the lockup occurs, and we lose video, at least one of the CPUs is > executing in flush_tlb_no_ptcg() or handle_IPI(), in the 'do' loop where > TLB > entries are being purged. What I have observed is that the end address and > the start address are in completely different regions. Usually, the start > address > is in region register 1 (address of 0x2000XXXXXXXXXXXX) and the end address > is in region register 3 (address of 0x6000XXXXXXXXXXXX). I don't know if > this > is the same problem I am seeing on the Lion, but I plan to connect and ITP > and > a serial console (although we haven't been able to get one to work yet on > the > Lion with BIOS 71) to see if the symptoms are the same.

Quoted 6 lines
Quoted 6 lines

FWIW, we have seen EXACTLY the same hang running here on our system. The start/end addresses for the purge cross region boundaries.

We are running a 2.4.0 kernel.

Jack> I found a problem that was causing the lockup described above Jack> & I suspect this may responsible for some of the other hangs Jack> various folks have seen.

Jack> There is code in flush_tlb_no_ptcg() that resends the IPI if Jack> other cpus have not responded within a short time. If this Jack> code get invoked, then it is possible for flush_cpu_count to Jack> get corrupted. When that happens, a cpu can be executing in Jack> handle_IPI() while flush_start/flush_end are changing. A cpu Jack> can pick up a non-matching flush_start/flush_end. This leads Jack> to hangs or lost TLB flushes.

Jack> To verify that this could cause the hang, I changed the Jack> timeout in flush_tlb_no_ptcg() from 40000UL to 400UL. I hung Jack> before getting to multiuser mode with flush_start/flush_end in Jack> different regions.

Jack> Here is the patch I used. Note: this is against 2.4.0,

Jack> --- linux-trillian/arch/ia64/kernel/smp.c Thu Feb 22 14:35:28 Jack> 2001 +++ linux/arch/ia64/kernel/smp.c Thu Feb 22 14:19:46 2001 Jack> @@ -321,6 +321,16 @@ { send_IPI_allbutself(IPI_FLUSH_TLB); } + Jack> +void +smp_resend_flush_tlb(void) +{ + /* + * Really need a Jack> null IPI but since this rarely should happen & + * since this Jack> code will go away, lets not add one. + */ +
Jack> send_IPI_allbutself(IPI_RESCHEDULE); +} #endif /* Jack> !CONFIG_ITANIUM_PTCG */

Jack> /* --- linux-trillian/arch/ia64/mm/tlb.c Thu Feb 22 14:35:28 Jack> 2001 +++ linux/arch/ia64/mm/tlb.c Thu Feb 22 14:19:50 2001 @@ Jack> -59,6 +59,7 @@ flush_tlb_no_ptcg (unsigned long start, Jack> unsigned long end, unsigned long nbits) { extern void Jack> smp_send_flush_tlb (void); + extern void smp_resend_flush_tlb Jack> (void); unsigned long saved_tpr = 0; unsigned long flags;

  Jack> @@ -101,9 +102,8 @@ { unsigned long start = ia64_get_itc();
  Jack> while (atomic_read(&flush_cpu_count) > 0) { - if
  Jack> ((ia64_get_itc() - start) > 40000UL) { -
  Jack> atomic_set(&flush_cpu_count, smp_num_cpus - 1); -
  Jack> smp_send_flush_tlb(); + if ((ia64_get_itc() - start) > 400UL)
  Jack> { + smp_resend_flush_tlb(); start = ia64_get_itc(); } }

Jack> -- Thanks

Jack> Jack Steiner (651-683-5302) (vnet 233-5302) [email protected]

Jack> _______________________________________________ Linux-IA64 Jack> mailing list [email protected]
Jack> http://lists.linuxia64.org/lists/listinfo/linux-ia64

#3 lore raw
In reply to Jack Steiner
Quoted 6 lines

Thanks for investigating the problem and the patch. The problem is happening because the timeout (40000UL) is not long enough. The processor is taking long time than this to complete the handle_IPI and processor doing the flush_tlb_no_ptcg is timing out and sending the IPI again. So, we should increase the timeout rather than decreasing the timeout to avoid extra reschedule IPIs.

I wondered about the timeout too. I agree that we do not want to go thru the resend_IPI code very often. I planned to add some stats to see how often the resend occurred & also whether any IPIs were really being dropped.

Is the code to resend IPIs going to remain OR is it just to workaround an earlier sighting (#117) that reported IPIs being dropped?? If the resend is permanent, then the "reschedule IPI" hack needs to be changed to something more like a null IPI. If the resend is temporary, then just increasing the timeout seems fine.

Signature · 3 lines
Thanks

Jack Steiner    (651-683-5302)   (vnet 233-5302)      [email protected]
#4 lore raw
In reply to Jack Steiner
Quoted 9 lines

Is the code to resend IPIs going to remain OR is it just to workaround
an earlier sighting (#117) that reported IPIs being dropped?? If the resend
is permanent, then the "reschedule IPI" hack needs to be changed to something
more like a null IPI. If the resend is temporary, then just increasing the timeout seems fine.

This is a workaround and is not needed for B3 and later steppings.

Thanks,
Asit

Thanks,
Asit