O2 RM7000 Issues

hinv

So I finally managed to get my hands on one of those super-rare RM7000 CPU's for the SGI O2, and, as expected, there are some problems.

For the most part, the system will boot into userland, but it seems userland isn't at all very happy. It seems bash is the unhappiest customer so far (or rather, the only userland program I've seen fail repeatedly). Running the Gentoo init scripts at startup, several scripts will terminate will a variety of messages, from Trace/breakpoint traps, to bus errors to illegal instructions. However, in the init scripts, they happen at specific points; usually when booting our network startup scripts (neth.eth0, net.lo), and usually on an exit() function in the script. Our emerge process, while python at heart, seems to fail sporadically on the bash sections (parsing the ebuild code) as well.

I've got a feeling this is likely a problem in the kernel more than it is a problem in the userland, but the question is how to go about determining which and where. The RM7K's are pretty rare, so I imagine there's probably a few undiscovered quirks in the code (notably the SC code in arch/mips/mm/sc-rm7k.c). Not to mention, we can't even use the 1MB tertiary cache these things have.

For reference, system info:

Quoted 1 line

hinv

System: IP32
Processor: 350 Mhz RM7000, with FPU
Primary I-cache size: 16 Kbytes Primary D-cache size: 16 Kbytes Secondary cache size: 256 Kbytes
Ternary cache size: 1024 Kbytes
Memory size: 512 Mbytes
Graphics: CRM, Rev C
Audio: A3 version 1
SCSI Disk: scsi(0)disk(2) SCSI Disk: scsi(0)disk(3)
SCSI CDROM: scsi(0)cdrom(4)

# cat /proc/cpuinfo
system type             : SGI O2
processor               : 0
cpu model               : RM7000 V3.3  FPU V2.0
BogoMIPS                : 350.20
byteorder               : big endian
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 48
extra interrupt vector  : no
hardware watchpoint     : no
ASEs implemented        :
VCED exceptions         : not available
VCEI exceptions         : not available

And errors (from various points in the execution and multiple reboots):

* Starting eth0
/sbin/runscript.sh: line 428: 2475 Illegal instruction ( function exit ()

* Starting lo
/sbin/runscript.sh: line 428: 1464 Illegal instruction ( function exit ()

* Starting eth0
/etc/init.d/net.eth0: line 248: 1650 Bus error ( u=0; module_load_minimum "${MODULES[i]}" || u=1; if [[ ${u} == 0 ]]; then

/sbin/runscript.sh: line 428: 2779 Bus error ( function exit ()
{

* Stopping syslog-ng
... [ ok ]
/lib/rcscripts/sh/rc-services.sh: line 444: 4093 Illegal instruction ( "/etc/init.d/${service}" stop )

/lib/rcscripts/sh/rc-services.sh: line 384: 1095 Trace/breakpoint trap "/etc/init.d/${service}" start

So if anyone's got some old rm7k patches sitting around they want tested, or spots where to look/debug options to turn on, let me know. I'll try switching back to an RM5200 and rebuild bash with -g and make sure gdb is installed, them change back to the RM7000 to try and capture some asm call or something that's causing these exit() failures in bash (which seem to be the primary symptom)

Cheers,

--Kumba

Ah, finally someone else got a hold of these things :-) Still got mine unused at all.
Not done much linux at al for the past few years, but always interested to see how things might work out for either R5K or RM7K O2's.

Tom

Quoted 101 lines · Kumba

So I finally managed to get my hands on one of those super-rare RM7000 CPU's for the SGI O2, and, as expected, there are some problems.

For the most part, the system will boot into userland, but it seems userland isn't at all very happy. It seems bash is the unhappiest customer so far (or rather, the only userland program I've seen fail repeatedly). Running the Gentoo init scripts at startup, several scripts will terminate will a variety of messages, from Trace/breakpoint traps, to bus errors to illegal instructions. However, in the init scripts, they happen at specific points; usually when booting our network startup scripts (neth.eth0, net.lo), and usually on an exit() function in the script. Our emerge process, while python at heart, seems to fail sporadically on the bash sections (parsing the ebuild code) as well.

I've got a feeling this is likely a problem in the kernel more than it is a problem in the userland, but the question is how to go about determining which and where. The RM7K's are pretty rare, so I imagine there's probably a few undiscovered quirks in the code (notably the SC code in arch/mips/mm/sc-rm7k.c). Not to mention, we can't even use the 1MB tertiary cache these things have.

For reference, system info:

Quoted 1 line

hinv

System: IP32
Processor: 350 Mhz RM7000, with FPU
Primary I-cache size: 16 Kbytes Primary D-cache size: 16 Kbytes Secondary cache size: 256 Kbytes
Ternary cache size: 1024 Kbytes
Memory size: 512 Mbytes
Graphics: CRM, Rev C
Audio: A3 version 1
SCSI Disk: scsi(0)disk(2) SCSI Disk: scsi(0)disk(3)
SCSI CDROM: scsi(0)cdrom(4)

# cat /proc/cpuinfo
system type             : SGI O2
processor               : 0
cpu model               : RM7000 V3.3  FPU V2.0
BogoMIPS                : 350.20
byteorder               : big endian
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 48
extra interrupt vector  : no
hardware watchpoint     : no
ASEs implemented        :
VCED exceptions         : not available
VCEI exceptions         : not available

And errors (from various points in the execution and multiple reboots):

* Starting eth0
/sbin/runscript.sh: line 428: 2475 Illegal instruction ( function
exit ()

* Starting lo
/sbin/runscript.sh: line 428: 1464 Illegal instruction ( function
exit ()

* Starting eth0
/etc/init.d/net.eth0: line 248: 1650 Bus error ( u=0;
module_load_minimum "${MODULES[i]}" || u=1; if [[ ${u} == 0 ]]; then

/sbin/runscript.sh: line 428: 2779 Bus error ( function
exit ()
{

* Stopping syslog-ng
... [ ok ]
/lib/rcscripts/sh/rc-services.sh: line 444: 4093 Illegal
instruction ( "/etc/init.d/${service}" stop )

/lib/rcscripts/sh/rc-services.sh: line 384: 1095 Trace/breakpoint trap "/etc/init.d/${service}" start

So if anyone's got some old rm7k patches sitting around they want tested, or spots where to look/debug options to turn on, let me know. I'll try switching back to an RM5200 and rebuild bash with -g and make sure gdb is installed, them change back to the RM7000 to try and capture some asm call or something that's causing these exit() failures in bash (which seem to be the primary symptom)

Cheers,

--Kumba

From: [email protected]
Date: Mon, 2 Jul 2007 09:08:43 -0400 (EDT)
To: Linux MIPS List <[email protected]>
Subject: Re: O2 RM7000 Issues
Content-Type: text/plain;charset=UTF-8

I have one of the 600 Mhz RM7000s, i.e. no tertiary cache since the module was originally a 300 Mhz RM5200. However, mine hasn't given any problems with Debian or Gentoo.

What kernel and target are you compiling for? I'm using 2.6.21.3 compiled for R5K. All my userspace is compiled for R5K too. I'll compile a new kernel for RM7000 and see if I have any issue then poke around to see what kernel code gets changed. I'm not 100% sure, but I didn't think it was that much so my gut reaction is this might be a gcc issue since the RM7000 isn't a common processor.

R5000, RM5200 and RM7000 are all MIPS IV processors so have the same instruction set. That leaves the usual suspects - pipeline hazards, cache problems and CPU bugs to research.

Big loud bell began ringing. The RM7000 fetches and decodes multiple instructions in one go. And just like the E9000 cores it does throw an exception if it doesn't like one of the opcodes even if that doesn't actually get executed. The kernel has a workaround for this PMC-Sierra peculiarity (I call it a bug) but it's only being activated for E9000 platforms.

Quoted 6 lines

Big loud bell began ringing. The RM7000 fetches and decodes multiple instructions in one go. And just like the E9000 cores it does throw an exception if it doesn't like one of the opcodes even if that doesn't actually get executed. The kernel has a workaround for this PMC-Sierra peculiarity (I call it a bug) but it's only being activated for E9000 platforms.

We have had a similar problems with shell on RM7000 based system. It seems, the reason listed above is only half of the problem, another is:
linux works incorrectly with RM7000 caches hierarchy. One visible effect
is errors in userspace on signal delivery trampolines.
Lets imagine we deliver a signal to application: we write signal trampoline instructions to stack, writeback (and invalidate) corresponding dcache line, invalidate corresponding icache line. Thats all, and we think that we can safely execute the trampoline, but this is wrong on RM7000! Our trampoline is now in scache, and everything seems to be ok, but after some number of load/stores corresponding scache line can be moved to dcache, replaced in scache by another data and not written to memory (this is a feature of RM7000 caches, its dcache is not a subset of scache, you can find a possible scenario of similar (but not the same) cache line transference in RM7000 manual (7.1.5 Orphaned Cache Lines)). After that it is possible that on signal trampoline execution

icache fetch old memory content instead of instruction written. If we want to execute instruction written by cpu, we must not only writeback corresponding dcache lines, but also writeback corresponding scache lines after it. The error is very sensitively to kernel/user code and data arrangement, it can be visible with one kernel configuration and irreproducible with another.
The problem affects not only signal trampoline flush to memory, but most cases of icache invalidation in kernel.

Quoted 23 lines · Sergey Rogozhkin

Quoted 6 lines

Big loud bell began ringing. The RM7000 fetches and decodes multiple instructions in one go. And just like the E9000 cores it does throw an exception if it doesn't like one of the opcodes even if that doesn't actually get executed. The kernel has a workaround for this PMC-Sierra peculiarity (I call it a bug) but it's only being activated for E9000 platforms.

We have had a similar problems with shell on RM7000 based system. It seems, the reason listed above is only half of the problem, another is: linux works incorrectly with RM7000 caches hierarchy. One visible effect is errors in userspace on signal delivery trampolines. Lets imagine we deliver a signal to application: we write signal trampoline instructions to stack, writeback (and invalidate) corresponding dcache line, invalidate corresponding icache line. Thats all, and we think that we can safely execute the trampoline, but this is wrong on RM7000! Our trampoline is now in scache, and everything seems to be ok, but after some number of load/stores corresponding scache line can be moved to dcache, replaced in scache by another data and not written to memory (this is a feature of RM7000 caches, its dcache is not a subset of scache, you can find a possible scenario of similar (but not the same) cache line transference in RM7000 manual (7.1.5 Orphaned Cache Lines)). After that it is possible that on signal trampoline

Quoted 8 lines

execution icache fetch old memory content instead of instruction written. If we want to execute instruction written by cpu, we must not only writeback corresponding dcache lines, but also writeback corresponding scache lines after it. The error is very sensitively to kernel/user code and data arrangement, it can be visible with one kernel configuration and irreproducible with another. The problem affects not only signal trampoline flush to memory, but most cases of icache invalidation in kernel.

Hmm... Makes sense. I guess I can cook up a patch based on that analysis.

I hungrily await said patch, as I believe this is a problem on RM9000 processors as well. I'm seeing "random" SIGILLs on user processes, particularly large complicated shell scripts like configure on an RM9k platform.

Quoted 5 lines · Andrew Sharp

I hungrily await said patch, as I believe this is a problem on RM9000 processors as well. I'm seeing "random" SIGILLs on user processes, particularly large complicated shell scripts like configure on an RM9k platform.

This was more or less exactly what I was seeing on an O2 RM7000 setup until the fix for errata #28 was put in (which should already be enabled for RM9000 systems).

Check include/asm-mips/war.h and make sure your machine is included in the list that define ICACHE_REFILLS_WORKAROUND_WAR. If not, add it and test; and fire off a patch. Should fix that issue (especially if bash is the only userland process dying while complex g++ compiles behave fine)

Quoted 11 lines · On Tue, 17 Jul 2007 10:01:24 -0400 Kumba <[email protected]> wrote:

Quoted 5 lines · Andrew Sharp

I hungrily await said patch, as I believe this is a problem on RM9000 processors as well. I'm seeing "random" SIGILLs on user processes, particularly large complicated shell scripts like configure on an RM9k platform.

This was more or less exactly what I was seeing on an O2 RM7000 setup until the fix for errata #28 was put in (which should already be enabled for RM9000 systems).

Quoted 5 lines

Check include/asm-mips/war.h and make sure your machine is included in the list that define ICACHE_REFILLS_WORKAROUND_WAR. If not, add it and test; and fire off a patch. Should fix that issue (especially if bash is the only userland process dying while complex g++ compiles behave fine)

Thanks, I had added this about a month ago, but the l-users were reporting that the problem persisted. Now that I've had a chance to examine it myself, it appears they were confused. There's a first time for everything.

I will be sending some patches to be sure, once I get all the bugs worked out. This architecture, a bifurcated RM9000x2 together with a marvell south bridge, is a searing pain I have to deal with daily.

Cheers,

a

Big loud bell began ringing. The RM7000 fetches and decodes multiple instructions in one go. And just like the E9000 cores it does throw an exception if it doesn't like one of the opcodes even if that doesn't actually get executed. The kernel has a workaround for this PMC-Sierra peculiarity (I call it a bug) but it's only being activated for E9000 platforms.

Are you really sure RM7000 has this bug? Workaround mentioned above breaks gcc signal frame unwinding mechanism: it search for sigcontext struct at fixed offset from signal trampoline.

Quoted 1 line

for E9000 platforms.

Are you really sure RM7000 has this bug? Workaround mentioned above breaks gcc signal frame unwinding mechanism: it search for sigcontext struct at fixed offset from signal trampoline.

And one another known RM7000 bug, maybe not taken into account by linux:
errata 38. r4k_wait is not suitable for RM7000 on some systems. I don't know if "O2" is affected.

Signed-off-by Ralf Baechle <[email protected]>

arch/mips/kernel/cpu-probe.c +24 -1

index f599e79..7ee0cb0 100644
@@ -75,6 +75,26 @@ static void r4k_wait_irqoff(void)
 	local_irq_enable();
 }
 
+/*
+ * The RM7000 variant has to handle erratum 38.  The workaround is to not
+ * have any pending stores when the WAIT instruction is executed.
+ */
+static void rm7k_wait_irqoff(void)
+{
+	local_irq_disable();
+	if (!need_resched())
+		__asm__(
+		"	.set	push		\n"
+		"	.set	mips3		\n"
+		"	.set	noat		\n"
+		"	mfc0	$1, $12		\n"
+		"	sync			\n"
+		"	mtc0	$1, $12		\n"
+		"	wait			\n"
+		"	.set	pop		\n");
+	local_irq_enable();
+}
+
 /* The Au1xxx wait is available only if using 32khz counter or
  * external timer source, but specifically not CP0 Counter. */
 int allow_au1k_wait;

@@ -132,7 +152,6 @@ static inline void check_wait(void)
 	case CPU_R4700:
 	case CPU_R5000:
 	case CPU_NEVADA:
-	case CPU_RM7000:
 	case CPU_4KC:
 	case CPU_4KEC:
 	case CPU_4KSC:

@@ -142,6 +161,10 @@ static inline void check_wait(void)
 		cpu_wait = r4k_wait;
 		break;
 
+	case CPU_RM7000:
+		cpu_wait = rm7k_wait_irqoff;
+		break;
+
 	case CPU_24K:
 	case CPU_34K:
 		cpu_wait = r4k_wait;

I am having a similar problem where complicated bash scripts on boot randomly throw SIGILL. I am running on a PMC MSP8510 platform - E9000 core. I have applied the patch to "war.h" mentioned in this thread and that did greatly reduce the number of occurences of this problem but has not fixed it. I was getting at least 2 illegal instructions every boot and now I can boot without any problems about 90% of the time.

Does the patch you mention below apply to the E9000 core as well?

Signed-off-by Ralf Baechle <[email protected]>

arch/mips/kernel/cpu-probe.c +10 -0

index f599e79..7ee0cb0 100644
@@ -75,6 +75,26 @@ static void r4k_wait_irqoff(void)
 	local_irq_enable();
 }
 
+/*
+ * The RM7000 variant has to handle erratum 38.  The workaround is to not
+ * have any pending stores when the WAIT instruction is executed.
+ */
+static void rm7k_wait_irqoff(void)
+{
+	local_irq_disable();
+	if (!need_resched())
+		__asm__(
+		"	.set	push		\n"

Quoted 24 lines · Ralf Baechle

Signed-off-by Ralf Baechle <[email protected]>

arch/mips/kernel/cpu-probe.c +14 -0

index f599e79..7ee0cb0 100644
@@ -75,6 +75,26 @@ static void r4k_wait_irqoff(void)
 	local_irq_enable();
 }
 
+/*
+ * The RM7000 variant has to handle erratum 38.  The workaround is to not
+ * have any pending stores when the WAIT instruction is executed.
+ */
+static void rm7k_wait_irqoff(void)
+{
+	local_irq_disable();
+	if (!need_resched())
+		__asm__(
+		"	.set	push		\n"
+		"	.set	mips3		\n"
+		"	.set	noat		\n"
+		"	mfc0	$1, $12		\n"
+		"	sync			\n"

arch/mips/kernel/cpu-probe.c static void r4k_wait_irqoff(void) · line 97

Quoted 6 lines

arch/mips/kernel/cpu-probe.c +6 -0

+		"	mtc0	$1, $12		\n"
+		"	wait			\n"
+		"	.set	pop		\n");
+	local_irq_enable();
+}
+

Technically, shouldn't that __asm__ be volatile?

Ah, finally someone else got a hold of these things :-) Still got mine unused at all.
Not done much linux at al for the past few years, but always interested to see how things might work out for either R5K or RM7K O2's.

Tom

Quoted 101 lines · Kumba

So I finally managed to get my hands on one of those super-rare RM7000 CPU's for the SGI O2, and, as expected, there are some problems.

For the most part, the system will boot into userland, but it seems userland isn't at all very happy. It seems bash is the unhappiest customer so far (or rather, the only userland program I've seen fail repeatedly). Running the Gentoo init scripts at startup, several scripts will terminate will a variety of messages, from Trace/breakpoint traps, to bus errors to illegal instructions. However, in the init scripts, they happen at specific points; usually when booting our network startup scripts (neth.eth0, net.lo), and usually on an exit() function in the script. Our emerge process, while python at heart, seems to fail sporadically on the bash sections (parsing the ebuild code) as well.

I've got a feeling this is likely a problem in the kernel more than it is a problem in the userland, but the question is how to go about determining which and where. The RM7K's are pretty rare, so I imagine there's probably a few undiscovered quirks in the code (notably the SC code in arch/mips/mm/sc-rm7k.c). Not to mention, we can't even use the 1MB tertiary cache these things have.

For reference, system info:

Quoted 1 line

hinv

System: IP32
Processor: 350 Mhz RM7000, with FPU
Primary I-cache size: 16 Kbytes Primary D-cache size: 16 Kbytes Secondary cache size: 256 Kbytes
Ternary cache size: 1024 Kbytes
Memory size: 512 Mbytes
Graphics: CRM, Rev C
Audio: A3 version 1
SCSI Disk: scsi(0)disk(2) SCSI Disk: scsi(0)disk(3)
SCSI CDROM: scsi(0)cdrom(4)

# cat /proc/cpuinfo
system type             : SGI O2
processor               : 0
cpu model               : RM7000 V3.3  FPU V2.0
BogoMIPS                : 350.20
byteorder               : big endian
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 48
extra interrupt vector  : no
hardware watchpoint     : no
ASEs implemented        :
VCED exceptions         : not available
VCEI exceptions         : not available

And errors (from various points in the execution and multiple reboots):

* Starting eth0
/sbin/runscript.sh: line 428: 2475 Illegal instruction ( function
exit ()

* Starting lo
/sbin/runscript.sh: line 428: 1464 Illegal instruction ( function
exit ()

* Starting eth0
/etc/init.d/net.eth0: line 248: 1650 Bus error ( u=0;
module_load_minimum "${MODULES[i]}" || u=1; if [[ ${u} == 0 ]]; then

/sbin/runscript.sh: line 428: 2779 Bus error ( function
exit ()
{

* Stopping syslog-ng
... [ ok ]
/lib/rcscripts/sh/rc-services.sh: line 444: 4093 Illegal
instruction ( "/etc/init.d/${service}" stop )

/lib/rcscripts/sh/rc-services.sh: line 384: 1095 Trace/breakpoint trap "/etc/init.d/${service}" start

So if anyone's got some old rm7k patches sitting around they want tested, or spots where to look/debug options to turn on, let me know. I'll try switching back to an RM5200 and rebuild bash with -g and make sure gdb is installed, them change back to the RM7000 to try and capture some asm call or something that's causing these exit() failures in bash (which seem to be the primary symptom)

Cheers,

--Kumba

I've got a feeling this is likely a problem in the kernel more than it is a problem in the userland, but the question is how to go about determining which and where. The RM7K's are pretty rare, so I imagine there's probably a few undiscovered quirks in the code (notably the SC code in arch/mips/mm/sc-rm7k.c).

I am having a similar problem where complicated bash scripts on boot randomly throw SIGILL. I am running on a PMC MSP8510 platform - E9000 core. I have applied the patch to "war.h" mentioned in this thread and that did greatly reduce the number of occurences of this problem but has not fixed it. I was getting at least 2 illegal instructions every boot and now I can boot without any problems about 90% of the time.

Does the patch you mention below apply to the E9000 core as well?

rm7k_wait_irqoff patch doesn't seem to improve the things for illegal instructions for E9k core.
But many customers have reported that they do get illegal instruction when ICACHE_REFILLS_WORKAROUND_WAR is not enabled.

I've just recently fixed this problem on my E9000 core which is a MSP85XX. I did some digging and found that the problem started to occur in 2.6.16 and is not there in 2.6.15. I looked into the deltas and found the specific change that broke me. The file is c-r4k.c.

In the function "local_r4k_flush_cache_sigtramp" there is a conditional:

if (!cpu_icache_snoops_remote_store && scache_size)

    protected_writeback_scache_line(addr & ~(sc_lsize - 1));

This additional "scache_size" has been added to this conditional. On my platform, "scache_size" is set to zero so the "protected_writeback_scache_line" is now not being called. I took out the "scache_size" from the conditional and now I boot without any illegal instructions.

As a side note, I also took out the workaround in "war.h". This workaround only hid the problem, it didn't fix it. Before I changed the conditional, I would crash on every boot without the workaround. The workaround reduced the crashes to maybe 1 in 3. Now, without the workaround, and with the change in the conditional, I haven't experienced any problems.

I'm sure this change was made for a reason in 2.6.16 so I'm not sure what the official fix needs to be but that solved my issues on my platform.

Let me know if there is anything anyone wants me to try on my platform to help come to an official fix for this problem.