Skip to contentSkip to conversation

Tester with IP27/IP30 needed

DISCUSSION Thomas Bogendoerfer 6 participants 30 messages Jan 15, 2008 – Feb 8, 2008 MIPS

lore
Share & tools
lore

Thomas Bogendoerfer

author maintainer
#0 lore raw

Hi,

we are facing a strange problem with lenny/sid chroots on IP28. The machine locks up after issuing a few ls/ps commands in a chroot bash. This only happens with a lenny/sid chroot, but not with etch. The major difference is probably the updare to glibc2.7. Since IP28 isn't really a nice R10k machine, it would be good, if someone with a working IP27/IP30 could try a lenny/sid chroot and tell us, if it's working/not working.

Thanks in advance.
Thomas.

Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]
#1 lore raw
Quoted 9 lines · Thomas Bogendoerfer

Hi,

we are facing a strange problem with lenny/sid chroots on IP28. The machine locks up after issuing a few ls/ps commands in a chroot bash. This only happens with a lenny/sid chroot, but not with etch. The major difference is probably the updare to glibc2.7. Since IP28 isn't really a nice R10k machine, it would be good, if someone with a working IP27/IP30 could try a lenny/sid chroot and tell us, if it's working/not working.

Simple testcase for me is:

/chroots/chroot-sid/lib/ld.so.1 --library-path /chroots/chroot-sid/lib /bin/bash

than the machine locks up hard ... This is with

Linux ip28 2.6.24-rc7-g0f154c48-dirty #38 Fri Jan 11 17:03:25 CET 2008 mips64 GNU/Linux

Flo

Signature · 3 lines
Florian Lohoff                  [email protected]             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
author maintainer
#2 lore raw
Quoted 1 line · Florian Lohoff

Simple testcase for me is:

now even simpler:

----------------------------------------------------------------------
void spin(void *a0)

{
	while (1) {
		asm volatile(
		"    .set mips3       \n"
		"    sync             \n"
		"1:  ll $5, 0($4)     \n"
		"    sc $3, 0($4)     \n"
		"    beqz $3, 1b      \n"
		"    .word 0x7c03e83b \n" /* rdhwr */
		"    lw $3, 0($4)     \n"
		"    nop              \n"
		);
	}
}

int main()

{
	int a;

	spin(&a);
}

----------------------------------------------------------------------

this kills my IP28 after a few seconds. If I drop rdhwr or sync the machine hasn't locked up after running for several minutes. Looks like we are hiting a strange condition.

This sort of code could be found in glibc 2.7 all over the place...

Thomas.

PS: Using rdhwr_noopt doesn't make a difference...

Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]
#3 lore raw
Quoted 7 lines · Thomas Bogendoerfer
Quoted 1 line · Florian Lohoff

Simple testcase for me is:

this kills my IP28 after a few seconds. If I drop rdhwr or sync the machine hasn't locked up after running for several minutes. Looks like we are hiting a strange condition.

Quoted 5 lines

This sort of code could be found in glibc 2.7 all over the place...

Thomas.

PS: Using rdhwr_noopt doesn't make a difference...

Kills my ip28 after 2 seconds ...

Flo

Signature · 3 lines
Florian Lohoff                  [email protected]             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
author maintainer
#4 lore raw
Quoted 8 lines · Florian Lohoff
Quoted 7 lines · Thomas Bogendoerfer
Quoted 1 line · Florian Lohoff

Simple testcase for me is:

this kills my IP28 after a few seconds. If I drop rdhwr or sync the machine hasn't locked up after running for several minutes. Looks like we are hiting a strange condition.

Quoted 7 lines
Quoted 5 lines

This sort of code could be found in glibc 2.7 all over the place...

Thomas.

PS: Using rdhwr_noopt doesn't make a difference...

Kills my ip28 after 2 seconds ...

I checked the ErrorEPC and it's pointing right after the sc. This is consistent to the dump Ralf did on his IP27.

Thomas.

Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]
#5 lore raw
In reply to Florian Lohoff
Quoted 11 lines · Florian Lohoff
Quoted 9 lines

this kills my IP28 after a few seconds. If I drop rdhwr or sync the machine hasn't locked up after running for several minutes. Looks like we are hiting a strange condition.

This sort of code could be found in glibc 2.7 all over the place...

Thomas.

PS: Using rdhwr_noopt doesn't make a difference...

Kills my ip28 after 2 seconds ...

Doesn't harm IP27. I even tried running two copies running in parallel.

Ralf

author maintainer
#6 lore raw
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · Florian Lohoff
Quoted 6 lines

this kills my IP28 after a few seconds. If I drop rdhwr or sync the machine hasn't locked up after running for several minutes. Looks like we are hiting a strange condition.

This sort of code could be found in glibc 2.7 all over the place...

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Thomas.

PS: Using rdhwr_noopt doesn't make a difference...

Kills my ip28 after 2 seconds ...

Doesn't harm IP27. I even tried running two copies running in parallel.

IP28 only locks up if spin() spans two I-cache lines. The lockup also happens if I use a different reserved instruction and skip it via SIGILL handler. As I don't have a working compiler/assembler for Irix I couldn't check, if this lockup also happens with Irix.

Thomas.

Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]
#7 lore raw
In reply to Thomas Bogendoerfer
Quoted 23 lines · Thomas Bogendoerfer

----------------------------------------------------------------------
void spin(void *a0)

{
	while (1) {
		asm volatile(
		"    .set mips3       \n"
		"    sync             \n"
		"1:  ll $5, 0($4)     \n"
		"    sc $3, 0($4)     \n"
		"    beqz $3, 1b      \n"
		"    .word 0x7c03e83b \n" /* rdhwr */
		"    lw $3, 0($4)     \n"
		"    nop              \n"
		);
	}
}

int main()

{
	int a;

	spin(&a);
}
Quoted 5 lines

----------------------------------------------------------------------

this kills my IP28 after a few seconds. If I drop rdhwr or sync the machine hasn't locked up after running for several minutes. Looks like we are hiting a strange condition.

SYNC on the R10000 will only graduate if the external signal SyncGblPerf is asserted. A simple system could simply always set it. I wonder if that has any affect. Logic analyzer time ...

Ralf

#8 lore raw
In reply to Thomas Bogendoerfer
Quoted 3 lines · Thomas Bogendoerfer
Quoted 1 line · Florian Lohoff

Simple testcase for me is:

[snip]

No effect on Octane R14000A, as far as lockups. Spikes the CPU usage in 'ps aux', but that's about it.

If I can get my plucky IP32 R10K to boot again soon, I may try it there for kicks and giggles. Maybe we're also seeing a side effect of the R10K's spec exec knocking the non-cache-coherent machines out?

Also, tried building the code with the R10K cache barrier on to see if anything else changes? Generally reserved for kernel stuff, but Peter once speculated userland might have a use for it.

--Kumba

Signature · 4 lines
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands
do them because they must, while the eyes of the great are elsewhere."  --Elrond
#9 lore raw
Quoted 2 lines · Kumba

No effect on Octane R14000A, as far as lockups. Spikes the CPU usage in 'ps aux', but that's about it.

So far it seems R12000 and R14000 are unaffected.

Quoted 7 lines

If I can get my plucky IP32 R10K to boot again soon, I may try it there for kicks and giggles. Maybe we're also seeing a side effect of the R10K's spec exec knocking the non-cache-coherent machines out?

Also, tried building the code with the R10K cache barrier on to see if anything else changes? Generally reserved for kernel stuff, but Peter once speculated userland might have a use for it.

It's a cache instruction so priviledged which means userspace can't execute it. It's also entirely unclear if a cache barrier instruction would make a difference at all.

Ralf

#10 lore raw
Quoted 4 lines · Ralf Baechle

It's a cache instruction so priviledged which means userspace can't execute it. It's also entirely unclear if a cache barrier instruction would make a difference at all.

The cache barrier has an interesting effect. I built three binaries: f, f2, and f3 (I'm cheap on the names):

f - cache barriers on load and stores (-mr10k-cache-barrier=2) f2 - cache barriers on loads only (-mr10k-cache-barrier=1) f3 - no cache barriers (flag omitted from gcc)

Running 'f' and 'f2' generates an "Illegal instruction" error, then drops back to the command line, while 'f3' hangs the box. This is an IP28 running on 2.6.23.9, using Thomas' patches backported to fit (plus Peter's Impact code and two sgiseeq patches from upstream).

This is similar to using a gentoo stage3 in a chroot environment that was built back in May of 2007, so I think this hang up pre-dates glibc-2.7 by some degree, as that chroot uses glibc-2.5. Chroot into this userland, and run our "env-update" script, and you'll hang the box.

FYI, CPU rev in this machine is R10000 v2.5. I think that's the same for all IP28 systems.

--Kumba

Signature · 3 lines
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond
author maintainer
#11 lore raw
Quoted 6 lines · Kumba

f - cache barriers on load and stores (-mr10k-cache-barrier=2) f2 - cache barriers on loads only (-mr10k-cache-barrier=1) f3 - no cache barriers (flag omitted from gcc)

Running 'f' and 'f2' generates an "Illegal instruction" error, then drops back to the command line, while 'f3' hangs the box. This is an IP28

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Quoted 2 lines

FYI, CPU rev in this machine is R10000 v2.5. I think that's the same for all IP28 systems.

Flo and mine also have rev 2.5 cpus.

Thomas.

Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]
#12 lore raw
Quoted 5 lines · Thomas Bogendoerfer

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Ah, didn't know the 'cache' instructions was kernel-mode only. Explains why it survived then :)

How does one enable the LLSC war workaround in glibc?

--Kumba

Signature · 3 lines
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond
#13 lore raw
Quoted 8 lines · Kumba
Quoted 5 lines · Thomas Bogendoerfer

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Quoted 4 lines

Ah, didn't know the 'cache' instructions was kernel-mode only. Explains why it survived then :)

How does one enable the LLSC war workaround in glibc?

By modifying the code ;-)

Ralf

#14 lore raw
Quoted 10 lines · Ralf Baechle
Quoted 8 lines · Kumba
Quoted 5 lines · Thomas Bogendoerfer

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Quoted 6 lines
Quoted 4 lines

Ah, didn't know the 'cache' instructions was kernel-mode only. Explains why it survived then :)

How does one enable the LLSC war workaround in glibc?

By modifying the code ;-)

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112

Flo

Signature · 3 lines
Florian Lohoff                  [email protected]             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
#15 lore raw
Quoted 9 lines · Florian Lohoff
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · Kumba
Quoted 5 lines · Thomas Bogendoerfer

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Ah, didn't know the 'cache' instructions was kernel-mode only. Explains why it survived then :)

How does one enable the LLSC war workaround in glibc?

By modifying the code ;-)

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112

Flo

Interesting. Is there a reason the kernel uses an #ifdef to choose between 'bezq' and 'bezql' that's not needed in glibc itself? Or does glibc itself lack a mechanism to detect CPU types to single out this specific change?

And any idea if uClibc will need similar mods?

Thanks!,

--Kumba

Signature · 3 lines
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond
#16 lore raw
Quoted 16 lines · Kumba
Quoted 15 lines · Florian Lohoff
Quoted 13 lines · Ralf Baechle
Quoted 10 lines · Kumba
Quoted 5 lines · Thomas Bogendoerfer

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Ah, didn't know the 'cache' instructions was kernel-mode only. Explains why it survived then :)

How does one enable the LLSC war workaround in glibc?

By modifying the code ;-)

Quoted 8 lines
Quoted 3 lines

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112

Flo

Interesting. Is there a reason the kernel uses an #ifdef to choose between 'bezq' and 'bezql' that's not needed in glibc itself? Or does glibc itself lack a mechanism to detect CPU types to single out this specific change?

glibc for mips has currently no such mechanism. Note that this change breaks MIPS I CPUs, so it is not generally applicable.

Quoted 1 line

And any idea if uClibc will need similar mods?

It needs a similiar change to support R10000 v2.5.

Thiemo

#17 lore raw
Quoted 4 lines · Thiemo Seufer

Kumba wrote:

glibc for mips has currently no such mechanism. Note that this change breaks MIPS I CPUs, so it is not generally applicable.

I'll have to ask one of our devs who knows autoconf really well. I figure that's probably a good place to catch something like this. Have configure check /proc/cpuinfo and look for "R10000", and if it finds it, mod CFLAGS to pass -DR10k_LLSC_WAR, and #ifdef on that in atomic.h.

Sound plausible?

Quoted 3 lines
Quoted 1 line

And any idea if uClibc will need similar mods?

It needs a similiar change to support R10000 v2.5.

Thought it would. I'll keep this in mind if we ever get that running again.

Cheers,

--Kumba

Signature · 3 lines
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond
#18 lore raw
Quoted 11 lines · Kumba
Quoted 4 lines

Kumba wrote:

glibc for mips has currently no such mechanism. Note that this change breaks MIPS I CPUs, so it is not generally applicable.

I'll have to ask one of our devs who knows autoconf really well. I figure that's probably a good place to catch something like this. Have configure check /proc/cpuinfo and look for "R10000", and if it finds it, mod CFLAGS to pass -DR10k_LLSC_WAR, and #ifdef on that in atomic.h.

Sound plausible?

No - the very same GLIBC does not work on mips1 machines and vice versa. Might by okay for gentoo but debian needs a run everywhere glibc which means some ld.so tricks like with the libc6-i686 to load a different glibc from my understanding.

Flo

Signature · 3 lines
Florian Lohoff                  [email protected]             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
#19 lore raw
Quoted 4 lines · Florian Lohoff

No - the very same GLIBC does not work on mips1 machines and vice versa. Might by okay for gentoo but debian needs a run everywhere glibc which means some ld.so tricks like with the libc6-i686 to load a different glibc from my understanding.

There is the long standing plan to generate a shared library on on the fly during kernel initialization and move atomic operations and performance relevant functions like memcpy to it. Thiemo's latest work on tlbex.c got us a tiny step closer to that.

Ralf

#20 lore raw
Quoted 11 lines · Ralf Baechle
Quoted 4 lines · Florian Lohoff

No - the very same GLIBC does not work on mips1 machines and vice versa. Might by okay for gentoo but debian needs a run everywhere glibc which means some ld.so tricks like with the libc6-i686 to load a different glibc from my understanding.

There is the long standing plan to generate a shared library on on the fly during kernel initialization and move atomic operations and performance relevant functions like memcpy to it. Thiemo's latest work on tlbex.c got us a tiny step closer to that.

You mean a single page in every processes address space or some /proc/sys/kernel/libatomic.so which would be a really cool hack?

Flo

Signature · 3 lines
Florian Lohoff                  [email protected]             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
#21 lore raw
Quoted 8 lines · Florian Lohoff
Quoted 7 lines · Ralf Baechle
Quoted 4 lines · Florian Lohoff

No - the very same GLIBC does not work on mips1 machines and vice versa. Might by okay for gentoo but debian needs a run everywhere glibc which means some ld.so tricks like with the libc6-i686 to load a different glibc from my understanding.

Quoted 7 lines
Quoted 4 lines

There is the long standing plan to generate a shared library on on the fly during kernel initialization and move atomic operations and performance relevant functions like memcpy to it. Thiemo's latest work on tlbex.c got us a tiny step closer to that.

You mean a single page in every processes address space or some /proc/sys/kernel/libatomic.so which would be a really cool hack?

We probably want to call it librandom-stuff.so. :-)

Thiemo

#22 lore raw
In reply to Florian Lohoff
Quoted 2 lines · Florian Lohoff

You mean a single page in every processes address space or some /proc/sys/kernel/libatomic.so which would be a really cool hack?

The way it's being done on x86 doesn't work for MIPS unless we use supervisor mode but supervisor mode is not implemented on all CPUs and also of interest for virtualization.

Ralf

#23 lore raw
In reply to Florian Lohoff
Quoted 4 lines · Florian Lohoff

No - the very same GLIBC does not work on mips1 machines and vice versa. Might by okay for gentoo but debian needs a run everywhere glibc which means some ld.so tricks like with the libc6-i686 to load a different glibc from my understanding.

While I could test this easily on gentoo, I was thinking of it more as an upstream fix. I suppose one of those configure switches could be included to skip the check as well, with the default being on. Figured I'd see what you guys thought, since it does seem to be a bug that should to be addressed somehow rather than patched forever in one of the distros.

--Kumba

Signature · 3 lines
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond
#24 lore raw
In reply to Kumba
Quoted 14 lines · Kumba
Quoted 14 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines · Thomas Bogendoerfer

no suprise here. As Ralf already noted cache barrier is a restricted instruction, it will always cause a illegal instruction when used in user space. Nevertheless it looks like all IP28 are affected by the simple exploit. Flo built glibc 2.7 with LLSC war workaround and this avoids triggering the hang.

Ah, didn't know the 'cache' instructions was kernel-mode only. Explains why it survived then :)

How does one enable the LLSC war workaround in glibc?

By modifying the code ;-)

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112

Quoted 7 lines
Quoted 1 line

Flo

Interesting. Is there a reason the kernel uses an #ifdef to choose between 'bezq' and 'bezql' that's not needed in glibc itself? Or does glibc itself lack a mechanism to detect CPU types to single out this specific change?

And any idea if uClibc will need similar mods?

The kernel has rather detailed knowledge about which workarounds are required for what platform and is optimized based on this knowledge.

Userspace is different. The basic promise is that userspace will run on any platform above certain minimum specs. That is something like MIPS II code is expected to run find on MIPS III or MIPS32 r1 or MIPS64 r2 hardware for example. This promise includes even workarounds as far as practicable and occasionally requires doing things that are somewhat suboptimal for performance or coding style. But it keeps things deterministic for users.

Ralf

#25 lore raw
In reply to Kumba
Quoted 10 lines · Kumba

Date: Tue, 22 Jan 2008 10:20:06 -0500
From: Kumba <[email protected]>
To: Thomas Bogendoerfer <[email protected]>
Cc: Florian Lohoff <[email protected]>, [email protected],
[email protected]
Subject: Re: Tester with IP27/IP30 needed

...
else changes? Generally reserved for kernel stuff, but Peter once speculated userland might have a use for it.

Oh, not userland. I must have used some misunderstandable formulation somewhere... :-(

Quoted 1 line

...

kind regards

peter

#26 lore raw
In reply to Thomas Bogendoerfer
Quoted 7 lines · Thomas Bogendoerfer

we are facing a strange problem with lenny/sid chroots on IP28. The machine locks up after issuing a few ls/ps commands in a chroot bash. This only happens with a lenny/sid chroot, but not with etch. The major difference is probably the updare to glibc2.7. Since IP28 isn't really a nice R10k machine, it would be good, if someone with a working IP27/IP30 could try a lenny/sid chroot and tell us, if it's working/not working.

Which CPU revision do you hit these problems on?

Ralf

#27 lore raw
Quoted 9 lines · Ralf Baechle
Quoted 7 lines

we are facing a strange problem with lenny/sid chroots on IP28. The machine locks up after issuing a few ls/ps commands in a chroot bash. This only happens with a lenny/sid chroot, but not with etch. The major difference is probably the updare to glibc2.7. Since IP28 isn't really a nice R10k machine, it would be good, if someone with a working IP27/IP30 could try a lenny/sid chroot and tell us, if it's working/not working.

Which CPU revision do you hit these problems on?

On IRC Thomas said it's rev 2.5.

R10000 upto version 2.6 has a broken store conditional so needs R10000_LLSC_WAR enabled. The sympthom is that SC succeed even though it should have failed so for example two multiple competing CPUs can take a spinlock. There is an erratum for this one.

Another bug is when a rdhwr $29 opcode is encountered in a branch delay slot. This will result in the CPU stopping execution of instructions but an NMI can recover it. For emulation performance reasons gcc no longer places rdhwr $29 in delay slots, so this one is no longer encountered in C code but still could be in assembler code. This one isn't covered by any errata.

There seem to be more funnies but to the best I can say they were never officially documented in errata either.

Ralf

#28 lore raw
Quoted 24 lines · Ralf Baechle
Quoted 9 lines
Quoted 7 lines

we are facing a strange problem with lenny/sid chroots on IP28. The machine locks up after issuing a few ls/ps commands in a chroot bash. This only happens with a lenny/sid chroot, but not with etch. The major difference is probably the updare to glibc2.7. Since IP28 isn't really a nice R10k machine, it would be good, if someone with a working IP27/IP30 could try a lenny/sid chroot and tell us, if it's working/not working.

Which CPU revision do you hit these problems on?

On IRC Thomas said it's rev 2.5.

R10000 upto version 2.6 has a broken store conditional so needs R10000_LLSC_WAR enabled. The sympthom is that SC succeed even though it should have failed so for example two multiple competing CPUs can take a spinlock. There is an erratum for this one.

Another bug is when a rdhwr $29 opcode is encountered in a branch delay slot. This will result in the CPU stopping execution of instructions but an NMI can recover it. For emulation performance reasons gcc no longer places rdhwr $29 in delay slots, so this one is no longer encountered in C code but still could be in assembler code. This one isn't covered by any errata.

Quoted 2 lines

There seem to be more funnies but to the best I can say they were never officially documented in errata either.

So I tested the rootfs provided by Florian and more or less as expected it immediately took out the 2 CPU R10000 v2.7 Origin I was testing on. Seems like only one CPU stopped, as the machine was pinging and reacts to NMIs. So could well be the effect I observed ages ago when trying to convert to glibc 2.4.

Ralf

#29 lore raw

On Tue, Jan 15, 2008 at 06:18:12PM +0000, Ralf Baechle wrote:

So I sent an NMI to the IP27 and used the POD to extract as much information as I could. Below the disassembly of the code. The addresses are looking a little odd because I had to disassembly at the XKPHYS address even though the code was actually executing in userspace.

1B 000: 0xa80000007bd5f008:       8c658b98        lw      a1,0x8b98(v1)
1B 000: 0xa80000007bd5f00c:       24060001        li      a2,0x1
1B 000: 0xa80000007bd5f010:       34a50001        ori     a1,a1,0x1
1B 000: 0xa80000007bd5f014:       00003821        move    a3,zero
1B 000: 0xa80000007bd5f018:       2402108e        li      v0,0x108e
1B 000: 0xa80000007bd5f01c:       0000000c        syscall
1B 000: 0xa80000007bd5f020:       1000ffd3        b       0xa80000007bd5ef70
1B 000: 0xa80000007bd5f024:       00000000        nop
1B 000: 0xa80000007bd5f028:       3c1c0010        lui     gp,0x10
1B 000: 0xa80000007bd5f02c:       279ce938        addiu   gp,gp,0xffffffe938
1B 000: 0xa80000007bd5f030:       0399e021        addu    gp,gp,t9
1B 000: 0xa80000007bd5f034:       27bdffd8        addiu   sp,sp,0xffffffffd8
1B 000: 0xa80000007bd5f038:       afbf0020        sw      ra,0x20(sp)
1B 000: 0xa80000007bd5f03c:       afb1001c        sw      s1,0x1c(sp)
1B 000: 0xa80000007bd5f040:       afb00018        sw      s0,0x18(sp)
1B 000: 0xa80000007bd5f044:       afbc0010        sw      gp,0x10(sp)
1B 000: 0xa80000007bd5f048:       7c03e83b        op1f    v1,zero,0xfffffffff

1B 000: fffe83b
1B 000: 0xa80000007bd5f04c: 8f848018 lw a0,0x8018(gp)

EPC is pointing to this lw so the subsequent instruction from the op1f which is rdhwr $29. ErrorEPC is pointing further down so it seems we must have returned from the emulation.

1B 000: 0xa80000007bd5f050:       24718b90        addiu   s1,v1,0xffffff8b90
1B 000: 0xa80000007bd5f054:       24901710        addiu   s0,a0,0x1710
1B 000: 0xa80000007bd5f058:       8e020008        lw      v0,0x8(s0)
1B 000: 0xa80000007bd5f05c:       00000000        nop
1B 000: 0xa80000007bd5f060:       1051000d        beq     v0,s1,0xa8000000f098
1B 000: 0xa80000007bd5f064:       00001821        move    v1,zero
1B 000: 0xa80000007bd5f068:       24020001        li      v0,0x1
1B 000: 0xa80000007bd5f06c:       c0851710        ll      a1,0x1710(a0)
1B 000: 0xa80000007bd5f070:       14a30006        bne     a1,v1,0xa8000000f08c
1B 000: 0xa80000007bd5f074:       00003021        move    a2,zero
1B 000: 0xa80000007bd5f078:       00403021        move    a2,v0
1B 000: 0xa80000007bd5f07c:       e0861710        sc      a2,0x1710(a0)
1B 000: 0xa80000007bd5f080:       10c0fffa        beq     a2,zero,0xa800d5f06c

And this is where the ErrorEPC is pointing.

1B 000: 0xa80000007bd5f084:       00000000        nop
1B 000: 0xa80000007bd5f088:       0000000f        sync
1B 000: 0xa80000007bd5f08c:       10c0000a        beq     a2,zero,0xa800d5f0b8
1B 000: 0xa80000007bd5f090:       00000000        nop

Ralf