16k or 64k PAGE_SIZE and "illegal instruction" (signal -4) errors
Okay, so from the "make kmap cache coloring aware" thread, I've been playing with larger PAGE_SIZE values on the Octane and O2 for the last few hours. 16k and 64k used to, in the past, never get far after init (usually died *at* init) That appears to have changed now. Most programs seem to JustWork(), but very randomly, I am getting a signal -4, illegal instruction (SIGILL) on the Octane. Both systems are running kernels w/ 64k PAGE_SIZE at the moment.
I cannot reproduce it on demand, so I'm not really sure what the cause could be. PAGE_SIZE should be largely transparent to userland these days, so I am wondering if this might be more oddities w/ an R14000 CPU.
Ideas?
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Quoted 11 lines · Joshua Kinard
Okay, so from the "make kmap cache coloring aware" thread, I've been playing with larger
PAGE_SIZEvalues on the Octane and O2 for the last few hours. 16k and 64k used to, in the past, never get far after init (usually died *at* init) That appears to have changed now. Most programs seem toJustWork(), but very randomly, I am getting a signal -4, illegal instruction (SIGILL) on the Octane. Both systems are running kernels w/ 64kPAGE_SIZEat the moment.I cannot reproduce it on demand, so I'm not really sure what the cause could be.
PAGE_SIZEshould be largely transparent to userland these days, so I am wondering if this might be more oddities w/ an R14000 CPU.
This sound very unlikely as the CPU was primarily designed to run IRIX and SGI's systems were using 16k or even 64k page size.
What userland are you running and how old is it? Are you seeing different results for 16k and 64k?
Ralf
Quoted 9 lines · Ralf Baechle
Quoted 3 lines
I cannot reproduce it on demand, so I'm not really sure what the cause could be.
PAGE_SIZEshould be largely transparent to userland these days, so I am wondering if this might be more oddities w/ an R14000 CPU.This sound very unlikely as the CPU was primarily designed to run IRIX and SGI's systems were using 16k or even 64k page size.
What userland are you running and how old is it? Are you seeing different results for 16k and 64k?
FWIW, I've been always using the 16k page size exclusively with my 64-bit
userland and my SWARM board using the SB-1/BCM1250 processor (with either endianness) and never had issues even with stuff as intensive as native GCC bootstrapping (with all the languages enabled such as Ada and Java) or glibc builds. It's been like 8 years now and quite recent kernels like from two months ago gave me no trouble either. So it must be something specific to the configuration, my first candidates to look at would be the generated TLB and cache handlers, that are system-specific.
Maciej
Quoted 8 lines · Maciej W. Rozycki
FWIW, I've been always using the 16k page size exclusively with my 64-bit
userland and my SWARM board using the SB-1/BCM1250 processor (with either endianness) and never had issues even with stuff as intensive as native GCC bootstrapping (with all the languages enabled such as Ada and Java) or glibc builds. It's been like 8 years now and quite recent kernels like from two months ago gave me no trouble either. So it must be something specific to the configuration, my first candidates to look at would be the generated TLB and cache handlers, that are system-specific.
Ah, and no issues with the 16k page size and my R4400SC DECstation and
the same 64-bit userland either, I booted it recently just fine, though little-endian only of course. Like with the SWARM I stuck here to using that page size exclusively with 64-bit userland.
Maciej
Quoted 10 lines · Maciej W. Rozycki
Quoted 9 lines
Quoted 3 lines
I cannot reproduce it on demand, so I'm not really sure what the cause could be.
PAGE_SIZEshould be largely transparent to userland these days, so I am wondering if this might be more oddities w/ an R14000 CPU.This sound very unlikely as the CPU was primarily designed to run IRIX and SGI's systems were using 16k or even 64k page size.
What userland are you running and how old is it? Are you seeing different results for 16k and 64k?
Quoted 8 lines
FWIW, I've been always using the 16k page size exclusively with my 64-bit
userland and my SWARM board using the SB-1/BCM1250 processor (with either endianness) and never had issues even with stuff as intensive as native GCC bootstrapping (with all the languages enabled such as Ada and Java) or glibc builds. It's been like 8 years now and quite recent kernels like from two months ago gave me no trouble either. So it must be something specific to the configuration, my first candidates to look at would be the generated TLB and cache handlers, that are system-specific.
Generally the R10000 architecture is such that there is much less potencial for software bugs as well. The TLB is nice, cleans up conflicting entries so no TLB shutdown or similar horrors possible. And the caches while they suffer from cache aliases, will cleanup those aliases transparently to software, that is an OS can treat them as non-aliasing. R10000 systems with the notable exception of the SGI O2 and Indigo² R10000 have fully coherent I/O. Basically the only thing that needs to be done in software is I-cache coherency. The I-cache snoops stores by remote CPUs but not by the local CPU itself so in a sense SMP is a simpler case than UP even.
Ralf
Quoted 21 lines · Ralf Baechle
Quoted 10 lines · Maciej W. Rozycki
Quoted 9 lines
Quoted 3 lines
I cannot reproduce it on demand, so I'm not really sure what the cause could be.
PAGE_SIZEshould be largely transparent to userland these days, so I am wondering if this might be more oddities w/ an R14000 CPU.This sound very unlikely as the CPU was primarily designed to run IRIX and SGI's systems were using 16k or even 64k page size.
What userland are you running and how old is it? Are you seeing different results for 16k and 64k?
Quoted 8 lines
FWIW, I've been always using the 16k page size exclusively with my 64-bit
userland and my SWARM board using the SB-1/BCM1250 processor (with either endianness) and never had issues even with stuff as intensive as native GCC bootstrapping (with all the languages enabled such as Ada and Java) or glibc builds. It's been like 8 years now and quite recent kernels like from two months ago gave me no trouble either. So it must be something specific to the configuration, my first candidates to look at would be the generated TLB and cache handlers, that are system-specific.
Quoted 9 lines
Generally the R10000 architecture is such that there is much less potencial for software bugs as well. The TLB is nice, cleans up conflicting entries so no TLB shutdown or similar horrors possible. And the caches while they suffer from cache aliases, will cleanup those aliases transparently to software, that is an OS can treat them as non-aliasing. R10000 systems with the notable exception of the SGI O2 and Indigo² R10000 have fully coherent I/O. Basically the only thing that needs to be done in software is I-cache coherency. The I-cache snoops stores by remote CPUs but not by the local CPU itself so in a sense SMP is a simpler case than UP even.
Yeah, coherency shouldn't be a problem for the Octane. hardware-coherent like IP27.
The icache snooping fix is already enabled in arch/mips/include/asm/mach-ip30/cpu-feature-overrides.h:
#define cpu_icache_snoops_remote_store 1
SMP is not working yet on IP30, though. I gave up on that for now, because I can't get the second CPU to start ticking properly.
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Quoted 10 lines · Joshua Kinard
Yeah, coherency shouldn't be a problem for the Octane. hardware-coherent like IP27.
The icache snooping fix is already enabled in
arch/mips/include/asm/mach-ip30/cpu-feature-overrides.h:#define
cpu_icache_snoops_remote_store1SMP is not working yet on IP30, though. I gave up on that for now, because I can't get the second CPU to start ticking properly.
Are you running a preemptible kernel?
You could also change the definition of cpu_icache_snoops_remote_store to 0 for testing. That should make things just a bit slower but otherwise have not impact. Would be interesting to see if that makes the SIGs go away.
Ralf
Quoted 14 lines · Ralf Baechle
Quoted 11 lines · Joshua Kinard
Okay, so from the "make kmap cache coloring aware" thread, I've been playing with larger
PAGE_SIZEvalues on the Octane and O2 for the last few hours. 16k and 64k used to, in the past, never get far after init (usually died *at* init) That appears to have changed now. Most programs seem toJustWork(), but very randomly, I am getting a signal -4, illegal instruction (SIGILL) on the Octane. Both systems are running kernels w/ 64kPAGE_SIZEat the moment.I cannot reproduce it on demand, so I'm not really sure what the cause could be.
PAGE_SIZEshould be largely transparent to userland these days, so I am wondering if this might be more oddities w/ an R14000 CPU.
Quoted 5 lines
This sound very unlikely as the CPU was primarily designed to run IRIX and SGI's systems were using 16k or even 64k page size.
What userland are you running and how old is it? Are you seeing different results for 16k and 64k?
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k PAGE_SIZE kernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16k PAGE_SIZE kernel when I booted 16k PAGE_SIZE the
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16k PAGE_SIZE kernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.
Being it's a Gentoo install...the o32 userland is pretty fresh. Especially on the Octane, where I literally rebuilt the old userland over 2-3 times just to make sure all the old 5-year cruft was gone. The n32 userland chroot is brand-spanking new. gcc-4.7.x only for now on both, because of PR61538 in gcc. Latest binutils.
The O2 is chugging away happily so far in updating a bunch of packages. So I am leaning towards this being another quirk I have to hunt down in the Octane's code again. There isn't much in the Octane-specific code that deals with memory, though -- it seems the higher-level MIPS memory code handles most things just fine.
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Quoted 6 lines · Joshua Kinard
Quoted 5 lines
This sound very unlikely as the CPU was primarily designed to run IRIX and SGI's systems were using 16k or even 64k page size.
What userland are you running and how old is it? Are you seeing different results for 16k and 64k?
Quoted 9 lines
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k
PAGE_SIZEkernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16kPAGE_SIZEkernel when I booted 16kPAGE_SIZEthe
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16kPAGE_SIZEkernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.
Such intermittent failures look to me remarkably like cache coherency
problems e.g. D$ vs I$. You can try making cache invalidation harder, e.g. tweak all the writeback calls and invalidation calls so that they perform their operation on the whole cache rather than the requested range only and see if that makes things better. You may instead tweak the suspected calling site too, of course.
Maciej
Quoted 16 lines · Joshua Kinard
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k
PAGE_SIZEkernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16kPAGE_SIZEkernel when I booted 16kPAGE_SIZEthe
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16kPAGE_SIZEkernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.Being it's a Gentoo install...the o32 userland is pretty fresh. Especially on the Octane, where I literally rebuilt the old userland over 2-3 times just to make sure all the old 5-year cruft was gone. The n32 userland chroot is brand-spanking new. gcc-4.7.x only for now on both, because of PR61538 in gcc. Latest binutils.
Quoted 5 lines
The O2 is chugging away happily so far in updating a bunch of packages. So I am leaning towards this being another quirk I have to hunt down in the Octane's code again. There isn't much in the Octane-specific code that deals with memory, though -- it seems the higher-level MIPS memory code handles most things just fine.
Can you enable core dumps? I'm wondering about the EPC of the crashed process. If it's at a function entry or the beginning of a page that might indicate there is an issue with flushing caches after the containing page got loaded. Also interesting to know if this possibly happened in a signal trampoline or VDSO.
These are just the usual suspects - nothing indicates this case is actually related.
Ralf
Quoted 24 lines · Ralf Baechle
Quoted 16 lines · Joshua Kinard
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k
PAGE_SIZEkernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16kPAGE_SIZEkernel when I booted 16kPAGE_SIZEthe
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16kPAGE_SIZEkernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.Being it's a Gentoo install...the o32 userland is pretty fresh. Especially on the Octane, where I literally rebuilt the old userland over 2-3 times just to make sure all the old 5-year cruft was gone. The n32 userland chroot is brand-spanking new. gcc-4.7.x only for now on both, because of PR61538 in gcc. Latest binutils.
Quoted 5 lines
The O2 is chugging away happily so far in updating a bunch of packages. So I am leaning towards this being another quirk I have to hunt down in the Octane's code again. There isn't much in the Octane-specific code that deals with memory, though -- it seems the higher-level MIPS memory code handles most things just fine.
Quoted 8 lines
Can you enable core dumps? I'm wondering about the EPC of the crashed process. If it's at a function entry or the beginning of a page that might indicate there is an issue with flushing caches after the containing page got loaded. Also interesting to know if this possibly happened in a signal trampoline or VDSO.
These are just the usual suspects - nothing indicates this case is actually related.
(Missed the reply all on the last one)
Enabled coredumps and got the 'shash' program to fail a second time (first program to do so)...so I'll rebuild that with debugging symbols and try to trip it up again later on.
Is a core file from a binary w/o debugging of any value?
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Quoted 34 lines · Joshua Kinard
Quoted 24 lines · Ralf Baechle
Quoted 16 lines · Joshua Kinard
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k
PAGE_SIZEkernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16kPAGE_SIZEkernel when I booted 16kPAGE_SIZEthe
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16kPAGE_SIZEkernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.Being it's a Gentoo install...the o32 userland is pretty fresh. Especially on the Octane, where I literally rebuilt the old userland over 2-3 times just to make sure all the old 5-year cruft was gone. The n32 userland chroot is brand-spanking new. gcc-4.7.x only for now on both, because of PR61538 in gcc. Latest binutils.
Quoted 5 lines
The O2 is chugging away happily so far in updating a bunch of packages. So I am leaning towards this being another quirk I have to hunt down in the Octane's code again. There isn't much in the Octane-specific code that deals with memory, though -- it seems the higher-level MIPS memory code handles most things just fine.
Quoted 8 lines
Can you enable core dumps? I'm wondering about the EPC of the crashed process. If it's at a function entry or the beginning of a page that might indicate there is an issue with flushing caches after the containing page got loaded. Also interesting to know if this possibly happened in a signal trampoline or VDSO.
These are just the usual suspects - nothing indicates this case is actually related.
Quoted 7 lines
(Missed the reply all on the last one)
Enabled coredumps and got the 'shash' program to fail a second time (first program to do so)...so I'll rebuild that with debugging symbols and try to trip it up again later on.
Is a core file from a binary w/o debugging of any value?
Yes - it will contain registers etc. Just what really matters in this case. We don't need the debug info because we're not interested in debugging the application.
Ralf
Quoted 42 lines · Ralf Baechle
Quoted 40 lines · Joshua Kinard
Quoted 24 lines · Ralf Baechle
Quoted 16 lines · Joshua Kinard
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k
PAGE_SIZEkernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16kPAGE_SIZEkernel when I booted 16kPAGE_SIZEthe
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16kPAGE_SIZEkernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.Being it's a Gentoo install...the o32 userland is pretty fresh. Especially on the Octane, where I literally rebuilt the old userland over 2-3 times just to make sure all the old 5-year cruft was gone. The n32 userland chroot is brand-spanking new. gcc-4.7.x only for now on both, because of PR61538 in gcc. Latest binutils.
Quoted 5 lines
The O2 is chugging away happily so far in updating a bunch of packages. So I am leaning towards this being another quirk I have to hunt down in the Octane's code again. There isn't much in the Octane-specific code that deals with memory, though -- it seems the higher-level MIPS memory code handles most things just fine.
Quoted 8 lines
Can you enable core dumps? I'm wondering about the EPC of the crashed process. If it's at a function entry or the beginning of a page that might indicate there is an issue with flushing caches after the containing page got loaded. Also interesting to know if this possibly happened in a signal trampoline or VDSO.
These are just the usual suspects - nothing indicates this case is actually related.
(Missed the reply all on the last one)
Enabled coredumps and got the 'shash' program to fail a second time (first program to do so)...so I'll rebuild that with debugging symbols and try to trip it up again later on.
Quoted 7 lines
Quoted 1 line
Is a core file from a binary w/o debugging of any value?
Yes - it will contain registers etc. Just what really matters in this case. We don't need the debug info because we're not interested in debugging the application.
Ralf
Attached. I assume readelf and objdump are used to extract the register information? Most searches on Google keep pointing me to GDB as if I want to debug the program.
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Quoted 49 lines · Joshua Kinard
Quoted 48 lines · Ralf Baechle
Quoted 34 lines · Joshua Kinard
Quoted 24 lines · Ralf Baechle
Quoted 16 lines · Joshua Kinard
o32 userland is the primary on both systems. However, the last SIGILL was under the 64k
PAGE_SIZEkernel inside of an n32 chroot compiling the 'boost' package on the Octane, which I restarted that and it's not complained since.
Also got SIGILL on the 16kPAGE_SIZEkernel when I booted 16kPAGE_SIZEthe
first time and ran 'ps'. Subsequent runs of 'ps' didn't reproduce the error. Also saw SIGILLs in the bootlog of the 16kPAGE_SIZEkernel when "rm" was ran once (couldn't reproduce) and when mdadm tried to put one of the arrays back together. Subsequent runs using similar argument lines don't reproduce once I got to a root shell.Being it's a Gentoo install...the o32 userland is pretty fresh. Especially on the Octane, where I literally rebuilt the old userland over 2-3 times just to make sure all the old 5-year cruft was gone. The n32 userland chroot is brand-spanking new. gcc-4.7.x only for now on both, because of PR61538 in gcc. Latest binutils.
Quoted 5 lines
The O2 is chugging away happily so far in updating a bunch of packages. So I am leaning towards this being another quirk I have to hunt down in the Octane's code again. There isn't much in the Octane-specific code that deals with memory, though -- it seems the higher-level MIPS memory code handles most things just fine.
Quoted 8 lines
Can you enable core dumps? I'm wondering about the EPC of the crashed process. If it's at a function entry or the beginning of a page that might indicate there is an issue with flushing caches after the containing page got loaded. Also interesting to know if this possibly happened in a signal trampoline or VDSO.
These are just the usual suspects - nothing indicates this case is actually related.
Quoted 7 lines
(Missed the reply all on the last one)
Enabled coredumps and got the 'shash' program to fail a second time (first program to do so)...so I'll rebuild that with debugging symbols and try to trip it up again later on.
Is a core file from a binary w/o debugging of any value?
Yes - it will contain registers etc. Just what really matters in this case. We don't need the debug info because we're not interested in debugging the application.
Quoted 5 lines
Quoted 1 line
Ralf
Attached. I assume readelf and objdump are used to extract the register information? Most searches on Google keep pointing me to GDB as if I want to debug the program.
Was anyone able to take a look at the core dump and see if there is anything out of the ordinary?
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Quoted 1 line · Joshua Kinard
(Missed the reply all on the last one)
If that meant to say you replied to me privately only - I didn't receive that mail yet.
I hope no more email debugging - I'm already getting nightmares ...
Ralf