HPPA and Squeeze
Hmm...
It's right that some of my comments are rather harsh, though you must know that I'm not speaking from a personal perspective.
Personally (and as Release Manager), I would be very happy to have a good working hppa port for Squeeze and beyond.
I made sure that the hppa port was included into Lenny even when the Debian System Administrators (DSA), package maintainers, release team members an others were shouting to drop it. I thought it was unfair to drop the port just before the release. They accepted my decision as long as I would make it clear that it was a *big* exception not to be taken lightly. At the time java support was completely dropped, buildds were crashing every other day and support for other programing languages looked poor and the port was still using linuxthreads as the latest of all ports.
After the release of Lenny there were still not much signs of improvement to the reliability of the buildds and the move to ntpl (that was going to solve almost all issues the maintainers had) seemed to not happen and not going to solve everything. The only surprising improvement was the regained java support.
It's quite disturbing in my honest opinion that only after us starting a thread like this one that we get to know the status of some of the porting efforts and lots of vocal support, but not many visible improvements. Instead of making sure that there is visibile improvement after that, this pattern seems to repeat itself which is not looking very promising. I'm sorry if my and others' frustration is ventilated in some of the mails. The issues with the buildds are lasting for years already with lots of time spent by DSA and others.
I still hope that the hppa porters can prove us wrong, fix the kernel issues (which are the probable cause of the unreliability of the buildds), finalise the ntpl move and stay on top of porting issues in Debian in the future.
Please let us now focus on improving the status of the hppa port in general and the buildds in particular!
Cheers
Luk
I take a regular look at various arches why packages are not correctly built. hppa is the most annoying arch for me. If you look at the stats you will notice that it's almost always the lowest in the stats. The reason it more or less keeps up is because I put alot of time in looking at the state of the packages and that packages get retried alot by the buildd maintainer.
One of the most annoying things about it is that packages only get uploaded 2 or 3 times a week. This has as effect that alot of packages fail to build because others haven't been uploaded yet. Alot of packages are uninstallable, and are basicly waiting for others to be built/uploaded. With only 2/3 uploads a week this means that packages are uninstallable for _weeks_. By the time the first reason has been fixed, something else already make it uninstallable again. And I guess this is also the reason the release team always needs to take a look at why something is uninstallable on hppa. I can deal with this, but because of this reason, hppa is a low priority port for me.
So it's my understanding that the porters have no idea about the problems. So I will start to mail you about problems as soon as I see them and they look like porting issues specific to hppa.
Here is a list of packages that failed to build because of instability on the buildds today:
package | buildd | error qgit | penalosa | make: *** [install] Segmentation fault acpica-unix | peri | make: *** [install] Segmentation fault
This are probabaly porting issues, they atleast seems to only affect hppa:
package | error rt-tests | PTHREAD_PRIO_INHERIT undeclared (NPTL support?) insighttoolkit | undefined references
axis | #531995: Cannot override the final method from ResourceBundle
petsc | petscvariables: No such file or directory
Logs can be found at
https://buildd.debian.org/build.cgi?arch=hppa;pkg=$package
Kurt
Quoted 50 lines · Luk Claes
Hmm...
It's right that some of my comments are rather harsh, though you must know that I'm not speaking from a personal perspective.
Personally (and as Release Manager), I would be very happy to have a good working hppa port for Squeeze and beyond.
I made sure that the hppa port was included into Lenny even when the Debian System Administrators (DSA), package maintainers, release team members an others were shouting to drop it. I thought it was unfair to drop the port just before the release. They accepted my decision as long as I would make it clear that it was a *big* exception not to be taken lightly. At the time java support was completely dropped, buildds were crashing every other day and support for other programing languages looked poor and the port was still using linuxthreads as the latest of all ports.
After the release of Lenny there were still not much signs of improvement to the reliability of the buildds and the move to ntpl (that was going to solve almost all issues the maintainers had) seemed to not happen and not going to solve everything. The only surprising improvement was the regained java support.
It's quite disturbing in my honest opinion that only after us starting a thread like this one that we get to know the status of some of the porting efforts and lots of vocal support, but not many visible improvements. Instead of making sure that there is visibile improvement after that, this pattern seems to repeat itself which is not looking very promising. I'm sorry if my and others' frustration is ventilated in some of the mails. The issues with the buildds are lasting for years already with lots of time spent by DSA and others.
I still hope that the hppa porters can prove us wrong, fix the kernel issues (which are the probable cause of the unreliability of the buildds), finalise the ntpl move and stay on top of porting issues in Debian in the future.
Please let us now focus on improving the status of the hppa port in general and the buildds in particular!
Cheers
Luk
Signature · 2 lines
To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected]
Quoted 5 lines · Kurt Roeckx
Here is a list of packages that failed to build because of instability on the buildds today:
package | buildd | error qgit | penalosa | make: *** [install] Segmentation fault acpica-unix | peri | make: *** [install] Segmentation fault
I had those random segfaults in make on paer too, until we switched to the UP kernel, at least from what I saw.
Sadly it looks that I forgot the buildd upgrade process on paer some days ago. The buildd will be back building ASAP.
Kind regards,
Philipp Kern
Signature · 4 lines
.''`. Philipp Kern Debian Developer : :' : http://philkern.de Stable Release Manager `. `' xmpp:[email protected] Wanna-Build Admin `- finger pkern/[email protected]
Quoted 4 lines · Kurt Roeckx
So it's my understanding that the porters have no idea about the problems. So I will start to mail you about problems as soon as I see them and they look like porting issues specific to hppa.
netgen fails to built with an internal compiler error since atleast April 2008. Logs are at:
https://buildd.debian.org/build.cgi?pkg=netgen;ver=4.4-15;arch=hppa
https://buildd.debian.org/build.cgi?pkg=netgen;arch=hppa
Kurt
Quoted 4 lines · Kurt Roeckx
So it's my understanding that the porters have no idea about the problems. So I will start to mail you about problems as soon as I see them and they look like porting issues specific to hppa.
Ruby1.9 hangs in test_thread.rb and gets killed after 150 minutes of inactivity. I assume NPTL will fix this.
Kurt
Quoted 8 lines · Kurt Roeckx
Quoted 4 lines · Kurt Roeckx
So it's my understanding that the porters have no idea about the problems. =C2=A0So I will start to mail you about problems as soon as I see them and they look like porting issues specific to hppa.
Ruby1.9 hangs in test_thread.rb and gets killed after 150 minutes of inactivity. =C2=A0I assume NPTL will fix this.
yes, it's a ruby bug, it's been explained on this m-l already. Ruby's implementation cannot handle linuxthreads.
--=20
Thibaut VARENE
http://www.parisc-linux.org/~varenet/
--
To unsubscribe from this list: send the line "unsubscribe linux-parisc"=
in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Kurt Roeckx
Quoted 4 lines
So it's my understanding that the porters have no idea about the problems. So I will start to mail you about problems as soon as I see them and they look like porting issues specific to hppa.
netgen fails to built with an internal compiler error since atleast April 2008. Logs are at:
https://buildd.debian.org/build.cgi?pkg=netgen;ver=4.4-15;arch=hppa
https://buildd.debian.org/build.cgi?pkg=netgen;arch=hppa
g++ -c -I. -I../libsrc/interface -Iinclude -O2 -I/usr/include/GL -I/usr/include/tcl8.4 -I/usr/include/tk8.4 -I/usr/X11R6/include -DLINUX -DOPENGL -DNGSOLVE basiclinalg/cholesky.cpp basiclinalg/calcinverse.cpp basiclinalg/vecmat.cpp basiclinalg/bandmatrix.cpp basiclinalg/eigensystem.cpp linalg/basevector.cpp linalg/vvector.cpp linalg/basematrix.cpp linalg/sparsematrix.cpp linalg/special_matrix.cpp linalg/cg.cpp linalg/chebyshev.cpp linalg/eigen.cpp linalg/order.cpp linalg/sparsecholesky.cpp linalg/pardisoinverse.cpp linalg/superluinverse.cpp linalg/jacobi.cpp linalg/blockjacobi.cpp linalg/commutingAMG.cpp ngstd/exception.cpp ngstd/table.cpp ngstd/bitarray.cpp ngstd/flags.cpp ngstd/symboltable.cpp ngstd/blockalloc.cpp ngstd/evalfunc.cpp ngstd/templates.cpp ngstd/localheap.cpp linalg/basematrix.cpp: In member function 'void ngla::S_BaseMatrix<std::complex<double> >::_ZTv0_n72_NK4ngla12S_BaseMatrixISt7complexIdEE12MultTransAddES2_RKNS_10BaseVectorERS4_(ngbla::Complex, const ngla::BaseVector&, ngla::BaseVector&) const':
linalg/basematrix.cpp:208: internal compiler error: in expand_expr_addr_expr_1, at expr.c:6830
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.3/README.Bugs> for instructions.
When an internal compiler error occurs as above, a GCC bug report with the following information should be filed:
1) failing compiler command,
2) details of the compiler (gcc -v), and
3) the preprocessed source for the failing command.
You can CC me on hppa bugs (danglin at gcc dot gnu dot org).
It takes less than five minutes to generate the above information and file a bug report. The preprocessed source is really important as it is very difficult for others to duplicate the problem otherwise.
In this case, it appears the following assert is triggered.
/* If the DECL isn't in memory, then the DECL wasn't properly
marked TREE_ADDRESSABLE, which will be either a front-end or a tree optimizer bug. */
gcc_assert (MEM_P (result));
There's a fairly high probability that this is a generic bug.
If the bug is a compiler regression, then fixes will applied to all active branches when available. Thus, it is useful to know if an earlier compiler version was able to successfully compile the file.
As things stand, I file more than 90% of hppa compiler bugs based on what I see in my testing. However, I'm not trying to build and maintain 10,000 packages. So, hopefully, the debian build team can help a bit more with problem reporting.
I recognize that not all bugs are easy to classify. However, internal compiler errors are always compiler bugs.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines · John David Anglin
Quoted 8 lines · Kurt Roeckx
Quoted 4 lines
So it's my understanding that the porters have no idea about the problems. So I will start to mail you about problems as soon as I see them and they look like porting issues specific to hppa.
netgen fails to built with an internal compiler error since atleast April 2008. Logs are at:
Quoted 8 lines
When an internal compiler error occurs as above, a GCC bug report with the following information should be filed:
1) failing compiler command,
2) details of the compiler (gcc -v), and
3) the preprocessed source for the failing command.You can CC me on hppa bugs (danglin at gcc dot gnu dot org).
I know how to file gcc bugs. I've just filed one. I didn't have time to try and reduce the testcase, and I can't try different versions of g++ yet. I've asked to have different versions installed on paer.
The bug report is at:
http://gcc.gnu.org/PR40505
Quoted 2 lines
It takes less than five minutes to generate the above information and file a bug report.
I've never been able to file any (non-automated) bug report in 5 minutes. And if you don't even have direct access to the hardware it takes longer.
Kurt
On Sat, Jun 20, 2009 at 07:48:38PM +0200, Kurt Roeckx wrote:
...
Quoted 7 lines
I know how to file gcc bugs. I've just filed one. I didn't have time to try and reduce the testcase, and I can't try different versions of g++ yet. I've asked to have different versions installed on paer.
The bug report is at:
http://gcc.gnu.org/PR40505
Kurt - thanks for filing the bug.
Quoted 6 lines
Quoted 2 lines
It takes less than five minutes to generate the above information and file a bug report.
I've never been able to file any (non-automated) bug report in 5 minutes. And if you don't even have direct access to the hardware it takes longer.
I agree. I'm trying to build netgen here too and if the ICE is easy to reproduce, can make that available to danglin and add to the bugreport.
thanks again,
grant
Quoted 6 lines
Quoted 3 lines
I've never been able to file any (non-automated) bug report in 5 minutes. And if you don't even have direct access to the hardware it takes longer.
I agree. I'm trying to build netgen here too and if the ICE is easy to reproduce, can make that available to danglin and add to the bugreport.
There's no problem reproducing the ICE.
I've pretty much localized the problem. It's a GCC middle-end bug. The problem is in passing a complex double from a thunk. The hppa specification says that values larger than 64 bits are passed by reference in the 32-bit runtime. However, the value is in a pair of registers and not copied to memory. This doesn't happen in calls from normal functions because the value gets copied to a stack slot.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines · Kurt Roeckx
Quoted 5 lines
Here is a list of packages that failed to build because of instability on the buildds today:
package | buildd | error qgit | penalosa | make: *** [install] Segmentation fault acpica-unix | peri | make: *** [install] Segmentation faultI had those random segfaults in make on paer too, until we switched to the UP kernel, at least from what I saw.
Based on my experience with building GCC, a recent UP kernel (e.g., 2.6.30) will avoid the random segfault problem.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 8 lines · John David Anglin
Quoted 6 lines
Quoted 3 lines
I've never been able to file any (non-automated) bug report in 5 minutes. And if you don't even have direct access to the hardware it takes longer.
I agree. I'm trying to build netgen here too and if the ICE is easy to reproduce, can make that available to danglin and add to the bugreport.
There's no problem reproducing the ICE.
Indeed...already failed for me.
Quoted 6 lines
I've pretty much localized the problem. It's a GCC middle-end bug. The problem is in passing a complex double from a thunk. The hppa specification says that values larger than 64 bits are passed by reference in the 32-bit runtime. However, the value is in a pair of registers and not copied to memory. This doesn't happen in calls from normal functions because the value gets copied to a stack slot.
I'm slightly confused about "stack slot". Could the "pass by reference" refer to the address in the stack?
Anyway, thanks for quickly tracking this down.
grant
Quoted 5 lines
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines
Quoted 6 lines
I've pretty much localized the problem. It's a GCC middle-end bug. The problem is in passing a complex double from a thunk. The hppa specification says that values larger than 64 bits are passed by reference in the 32-bit runtime. However, the value is in a pair of registers and not copied to memory. This doesn't happen in calls from normal functions because the value gets copied to a stack slot.
I'm slightly confused about "stack slot". Could the "pass by reference" refer to the address in the stack?
The reference can refer to a location on the stack (region allocated for locals). However, values larger than 64 bits can't be passed in the argument slots.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines · Philipp Kern
Quoted 5 lines · Kurt Roeckx
Here is a list of packages that failed to build because of instability on the buildds today:
package | buildd | error qgit | penalosa | make: *** [install] Segmentation fault acpica-unix | peri | make: *** [install] Segmentation faultI had those random segfaults in make on paer too, until we switched to the UP kernel, at least from what I saw.
Did something change to peri? I'm currently only seeing them on penalosa.
Failed logs the past few days:
Jun 22 | wesnoth | penalosa | quilt segfaults Jun 22 | gitg | penalosa | quilt segfaults Jun 23 | zita-convolver | penalosa | quilt segfaults Jun 25 | autodocksuite | penalosa | make segfaults Jun 26 | mpd | penalosa | find segfaults? Jun 30 | scorched3d | penalosa | make segfaults Jun 30 | libtext-bibtex-perl | penalosa | make segfaults Jun 30 | gnome-chemistry-utils | penalosa | libtool segfaults Jul 01 | openmsx-catapult | penalosa | make segfaults Jul 01 | prima | penalosa | make segfaults Jul 01 | fvwm | penalosa | gcc says as had a segfault Jul 01 | cherokee | penalosa | quilt segfaults Jul 03 | vflib3 | penalosa | make segfaults Jul 03 | rpy2 | penalosa | make segfaults Jul 03 | debian-installer-utils | penalosa | make segfaults
On other that looks weird is, all also on penalosa:
- scid, seems to have been stuck in a "cp".
- postgresql-8.3, not sure what the error is
- building gnome-system-monitor, openssh-client failed to install in it's postinst script calling update-alternatives
And then there is glob2 that fails with:/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot reach 0000f9bf_memcpy@@GLIBC_2.2+0, recompile with -ffunction-sections /usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot handle R_PARISC_PCREL17F for memcpy@@GLIBC_2.2 /usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit status
Kurt
Quoted 2 lines · Kurt Roeckx
Did something change to peri? I'm currently only seeing them on penalosa.
UP kernel, maybe?
Kind regards,
Philipp Kern
Signature · 4 lines
.''`. Philipp Kern Debian Developer : :' : http://philkern.de Stable Release Manager `. `' xmpp:[email protected] Wanna-Build Admin `- finger pkern/[email protected]
Quoted 5 lines · Philipp Kern
Quoted 2 lines · Kurt Roeckx
Did something change to peri? I'm currently only seeing them on penalosa.
UP kernel, maybe?
Both peri and penalosa run 2.6.29-2-parisc64-smp and from what I can tell run on identical hardware.
Kurt
Quoted 5 lines
And then there is glob2 that fails with:
/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot reach 0000f9bf_memcpy@@GLIBC_2.2+0, recompile with -ffunction-sections/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot handleR_PARISC_PCREL17Ffor memcpy@@GLIBC_2.2/usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit status
I couldn't duplicate the problem with binutils 2.19.1-1 and gcc-4.3 4.3.3-10, or with my own binutils build on two different systems.
The above shouldn't happen as the text size of FileManager.o is well below the size where a 17-bit branch can't reach the stub table. Possibly, the stub table is full. On the otherhand, the "0000f9bf_memcpy@@GLIBC_2.2+0" string looks garbled. So, this may be another form of the SMP memory corruption that causes the segvs, particularly if it isn't reproducible on the build system.
The suggestion to "recompile with -ffunction-sections" is somewhat misleading. While -ffunction-sections may help sometimes, in other cases it may be necessary to play with the ld --stub-group-size=N option, or to compile with -mlong-calls.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines · John David Anglin
Quoted 5 lines
And then there is glob2 that fails with:
/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot reach 0000f9bf_memcpy@@GLIBC_2.2+0, recompile with -ffunction-sections/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot handleR_PARISC_PCREL17Ffor memcpy@@GLIBC_2.2/usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit statusI couldn't duplicate the problem with binutils 2.19.1-1 and gcc-4.3 4.3.3-10, or with my own binutils build on two different systems.
Quoted 6 lines
The above shouldn't happen as the text size of FileManager.o is well below the size where a 17-bit branch can't reach the stub table. Possibly, the stub table is full. On the otherhand, the "0000f9bf_memcpy@@GLIBC_2.2+0" string looks garbled. So, this may be another form of the SMP memory corruption that causes the segvs, particularly if it isn't reproducible on the build system.
It actually already failed twice on the same system with the same error. I've just let it retry again, we'll see if it fails again.
The log file show this is with:
gcc-4.3 4.3.3-13 / 4.3.3-11
binutils 2.19.1-1
Kurt
Quoted 17 lines · Kurt Roeckx
Quoted 9 lines · John David Anglin
Quoted 5 lines
And then there is glob2 that fails with:
/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot reach 0000f9bf_memcpy@@GLIBC_2.2+0, recompile with -ffunction-sections/usr/bin/ld: libgag/src/libgag.a(FileManager.o)(.text+0x2fc8): cannot handleR_PARISC_PCREL17Ffor memcpy@@GLIBC_2.2/usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit statusI couldn't duplicate the problem with binutils 2.19.1-1 and gcc-4.3 4.3.3-10, or with my own binutils build on two different systems.
Quoted 6 lines
The above shouldn't happen as the text size of FileManager.o is well below the size where a 17-bit branch can't reach the stub table. Possibly, the stub table is full. On the otherhand, the "0000f9bf_memcpy@@GLIBC_2.2+0" string looks garbled. So, this may be another form of the SMP memory corruption that causes the segvs, particularly if it isn't reproducible on the build system.
Quoted 2 lines
It actually already failed twice on the same system with the same error. I've just let it retry again, we'll see if it fails again.
And it failed again with the same error, on peri now.
Kurt
Quoted 9 lines · Philipp Kern
Quoted 5 lines · Kurt Roeckx
Quoted 2 lines
Did something change to peri? I'm currently only seeing them on penalosa.
UP kernel, maybe?
Both peri and penalosa run 2.6.29-2-parisc64-smp and from what I can tell run on identical hardware.
I tried a SMP kernel (2.6.30.1 with the patch set posted earlier) on my rp34404. Fired up a GCC build. It didn't take more than a few minutes to trigger a couple of segvs in /bin/sh. On the other hand, I ran a UP version of 2.6.30 for more than a week without any major problems.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 5 lines · John David Anglin
Quoted 5 lines · Philipp Kern
Quoted 4 lines · Kurt Roeckx
Quoted 2 lines
Did something change to peri? I'm currently only seeing them on penalosa.
UP kernel, maybe?
Quoted 8 lines
Quoted 2 lines
Both peri and penalosa run 2.6.29-2-parisc64-smp and from what I can tell run on identical hardware.
I tried a SMP kernel (2.6.30.1 with the patch set posted earlier) on my rp34404. Fired up a GCC build. It didn't take more than a few minutes to trigger a couple of segvs in
/bin/sh. On the other hand, I ran a UP version of 2.6.30 for more than a week without any major problems.
So, instead of just complaining, wouldn't it be useful if someone with root access would install a UP kernel (and disable nscd) for the time being on the build servers?
That way we all would avoid debian build problems and could concentrate on solving the issues with the SMP kernel itself.
In the meantime, all (IMHO) major kernel patches have now been included in Linus' 2.6.31-rc2 kernel and I plan to backport and send them to the stable-kernel team for inclusion into 2.6.29 and 2.6.30 stable kernel series.
Helge
Quoted 21 lines · Helge Deller
Quoted 5 lines · John David Anglin
Quoted 5 lines · Philipp Kern
Quoted 4 lines · Kurt Roeckx
Quoted 2 lines
Did something change to peri? I'm currently only seeing them on penalosa.
UP kernel, maybe?
Quoted 8 lines
Quoted 2 lines
Both peri and penalosa run 2.6.29-2-parisc64-smp and from what I can tell run on identical hardware.
I tried a SMP kernel (2.6.30.1 with the patch set posted earlier) on my rp34404. Fired up a GCC build. It didn't take more than a few minutes to trigger a couple of segvs in
/bin/sh. On the other hand, I ran a UP version of 2.6.30 for more than a week without any major problems.So, instead of just complaining, wouldn't it be useful if someone with root access would install a UP kernel (and disable nscd) for the time being on the build servers?
That way we all would avoid debian build problems and could concentrate on solving the issues with the SMP kernel itself.
Quoted 4 lines
In the meantime, all (IMHO) major kernel patches have now been included in Linus' 2.6.31-rc2 kernel and I plan to backport and send them to the stable-kernel team for inclusion into 2.6.29 and 2.6.30 stable kernel series.
HPPA folks, can you please convert at least one buildd to a UP configuration? I have another example of extreme buildd flakiness mentioned in bug #529485. Adam C Powell IV <[email protected]> writes:
That's too bad. I saw 6 attempts to do 3.0.0-X of which two passed the first try and four failed. At that rate (1/3), it would take nine attempts to have a decent chance of passing both rounds (debug static and optimized shared/static).
HPPA is the only arch for which petsc failed to build, and currently this blocks petsc transition.
Best regards,
Signature · 2 lines
Jurij Smakov [email protected] Key: http://www.wooyd.org/pgpkey/ KeyID: C99E03CC
Quoted 3 lines · Jurij Smakov
HPPA folks, can you please convert at least one buildd to a UP configuration? I have another example of extreme buildd flakiness mentioned in bug #529485. Adam C Powell IV <[email protected]> writes:
Well, paer is UP already. It's interesting though that one of the peri/penalosa twins works fine with SMP and the other does not.
Kind regards,
Philipp Kern
Signature · 4 lines
.''`. Philipp Kern Debian Developer : :' : http://philkern.de Stable Release Manager `. `' xmpp:[email protected] Wanna-Build Admin `- finger pkern/[email protected]
Quoted 2 lines
That way we all would avoid debian build problems and could concentrate on solving the issues with the SMP kernel itself.
The problem actually may be present in UP kernels. I had a segv this morning building GCC with a UP 2.6.30.1:
adave@mx3210:~/gnu/gcc/objdir/hppa-linux/libjava$ gdb -c core /bin/sh GNU gdb (GDB) 6.8.50.20090510-cvs Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "hppa-unknown-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... (no debugging symbols found) BFD: Warning: /home/dave/gnu/gcc/objdir/hppa-linux/libjava/core is truncated: expected core file size >= 196608, found: 118784. Reading symbols from /lib/ld.so.1...Reading symbols from /usr/lib/debug/lib/ld-2.9.so...done. done. Loaded symbols for /lib/ld.so.1 Reading symbols from /lib/libncurses.so.5...done. Loaded symbols for /lib/libncurses.so.5 Reading symbols from /lib/libdl.so.2...Reading symbols from /usr/lib/debug/lib/libdl-2.9.so...done. done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debug/lib/libc-2.9.so...done. done. Loaded symbols for /lib/libc.so.6 Core was generated by `/bin/sh -c /home/dave/gnu/gcc/gcc/mkinstalldirs `dirname gnu/java/locale/Locale'. Program terminated with signal 11, Segmentation fault. #0 _dl_map_object_deps (map=0x403d8880, preloads=0x40001398, npreloads=12, trace_mode=0, open_mode=0) at dl-deps.c:224 224 dl-deps.c: No such file or directory. in dl-deps.c (gdb) bt #0 _dl_map_object_deps (map=0x403d8880, preloads=0x40001398, npreloads=12, trace_mode=0, open_mode=0) at dl-deps.c:224 #1 0x403b9bd4 in dl_main (phdr=0x10034, phnum=<value optimized out>, user_entry=<value optimized out>) at rtld.c:1780 #2 0x403cb898 in _dl_sysdep_start (start_argptr=<value optimized out>, dl_main=@0x403d7566: 0x403b8fe4 <dl_main>) at ../elf/dl-sysdep.c:239 #3 0x403b785c in _dl_start_final (arg=0xbfff2020, info=0xbfff2348) at rtld.c:332 #4 0x403b7adc in _dl_start (arg=0xbfff2020) at rtld.c:560 #5 0x403b742c in _start () from /lib/ld.so.1 #6 0x403b742c in _start () from /lib/ld.so.1 #7 0x403b742c in _start () from /lib/ld.so.1 #8 0x403b742c in _start () from /lib/ld.so.1 #9 0x403b742c in _start () from /lib/ld.so.1 #10 0x403b742c in _start () from /lib/ld.so.1 #11 0x403b742c in _start () from /lib/ld.so.1 #12 0x403b742c in _start () from /lib/ld.so.1 #13 0x403b742c in _start () from /lib/ld.so.1 #14 0x403b742c in _start () from /lib/ld.so.1 #15 0x403b742c in _start () from /lib/ld.so.1 #16 0x403b742c in _start () from /lib/ld.so.1 ^CQuit
For some reason, gdb does terminate the backtrace. Recent versions of gdb also complain about core file truncation. Think the full stack region is not being dumped.
As with most of the segvs that I have debugged in the past, the problem occurs in the dynamic loader.
This is the tombstone:
Jul 5 04:07:31 mx3210 kernel: Jul 5 04:07:31 mx3210 kernel: do_page_fault() pid=22068 command='sh' type=15 address=0x00000004 Jul 5 04:07:31 mx3210 kernel: Jul 5 04:07:31 mx3210 kernel: YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI Jul 5 04:07:31 mx3210 kernel: PSW: 00000000000001000000000000001111 Not taintedJul 5 04:07:31 mx3210 kernel: r00-03 000000000004000f 00000000403d76a0 0000000 0403c2c23 00000000bfff2ac0 Jul 5 04:07:31 mx3210 kernel: r04-07 00000000403d76a0 000000000000000c 0000000 040001398 00000000403b0dc0 Jul 5 04:07:31 mx3210 kernel: r08-11 0000000000000000 0000000000000002 0000000 040000e88 00000000bfff2d08 Jul 5 04:07:31 mx3210 kernel: r12-15 000000004037e4b4 00000000bfff2c48 00000000403d76a0 0000000000000004 Jul 5 04:07:31 mx3210 kernel: r16-19 00000000bfff2c08 00000000bfff2bc8 00000000bfff2b88 00000000403d76a0 Jul 5 04:07:31 mx3210 kernel: r20-23 00000000bfff2d0f 0000000000000000 0000000040002000 0000000000000001 Jul 5 04:07:31 mx3210 kernel: r24-27 000000000000000c 0000000040001398 00000000400013a8 00000000000365e8 Jul 5 04:07:31 mx3210 kernel: r28-31 0000000000000000 0000000024242424 00000000bfff2e00 00000000403c45c3 Jul 5 04:07:31 mx3210 kernel: sr00-03 000000000000e800 000000000000e800 0000000000000000 000000000000e800 Jul 5 04:07:31 mx3210 kernel: sr04-07 000000000000e800 000000000000e800 000000000000e800 000000000000e800 Jul 5 04:07:31 mx3210 kernel: Jul 5 04:07:31 mx3210 kernel: VZOUICununcqcqcqcqcqcrmunTDVZOUI Jul 5 04:07:31 mx3210 kernel: FPSR: 00000000000000000000000000000000 Jul 5 04:07:31 mx3210 kernel: FPER1: 00000000 Jul 5 04:07:31 mx3210 kernel: fr00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jul 5 04:07:31 mx3210 kernel: fr04-07 fffffffffffff000 0000000000000000 ffffffffffffff9c bff0000000000000 Jul 5 04:07:31 mx3210 kernel: fr08-11 0000000000000000 000000004055d400 0000000000000802 000000004055d400 Jul 5 04:07:31 mx3210 kernel: fr12-15 00000000401a7d6c 0000000000000000 00000000401a5e94 000000007f40c580 Jul 5 04:07:31 mx3210 kernel: fr16-19 000000007f40ab38 000000007f406c78 000000007f430000 000000007f430000 Jul 5 04:07:31 mx3210 kernel: fr20-23 000000004055d400 00000000404f8bcc 000000000000026a 0000003700000000 Jul 5 04:07:31 mx3210 kernel: fr24-27 0000000000000000 000000007f430000 000000004055d400 0000000000000803 Jul 5 04:07:31 mx3210 kernel: fr28-31 ffffffffffffffe9 000000007ec4bc88 000000004055d400 0000000000000803 Jul 5 04:07:31 mx3210 kernel: Jul 5 04:07:31 mx3210 kernel: IASQ: 000000000000e800 000000000000e800 IAOQ: 00000000403c2b03 00000000403c2b07 Jul 5 04:07:31 mx3210 kernel: IIR: 0f88108c ISR: 000000000000e800 IOR: 0000000000000004 Jul 5 04:07:31 mx3210 kernel: CPU: 0 CR30: 00000002bf8fc000 CR31: 0000000040564000 Jul 5 04:07:31 mx3210 kernel: ORIG_R28: 00000000407a55c8 Jul 5 04:07:31 mx3210 kernel: IAOQ[0]: 00000000403c2b03 Jul 5 04:07:31 mx3210 kernel: IAOQ[1]: 00000000403c2b07 Jul 5 04:07:31 mx3210 kernel: RP(r2): 00000000403c2c23
$ disasm 0x0f88108c 0: 0f 88 10 8c ldw 4(ret0),r12
(gdb) p/x $pc
$1 = 0x403c2b00
(gdb) disass 0x403c2af0 0x403c2b10
Dump of assembler code from 0x403c2af0 to 0x403c2b10:0x403c2af0 <_dl_map_object_deps+396>: cmpib,= 0,ret0,0x403c32c8 <_dl_map_object_deps+2404>0x403c2af4 <_dl_map_object_deps+400>: ldi 0,r110x403c2af8 <_dl_map_object_deps+404>: ldw 34(r10),ret00x403c2afc <_dl_map_object_deps+408>: ldw -30(r3),r210x403c2b00 <_dl_map_object_deps+412>: ldw 4(ret0),r120x403c2b04 <_dl_map_object_deps+416>: stw r21,18(r3)0x403c2b08 <_dl_map_object_deps+420>: ldw -34(r3),ret00x403c2b0c <_dl_map_object_deps+424>: stw r12,20(r3)
End of assembler dump.
(gdb) p/x $r10
$2 = 0x40000e88
(gdb) p/x *($r10 + 0x34)
$3 = 0x0
(gdb) p/x $r10 + 0x34
$4 = 0x40000ebc
(gdb) x/32x $r100x40000e88: 0x4029b000 0x40000e78 0x4029e5fc 0x400011180x40000e98: 0x40000bf0 0x40000e88 0x00000000 0x400010dc0x40000ea8: 0x00000000 0x4029e5fc 0x00000000 0x000000000x40000eb8: 0x00000000 0x00000000 0x00000000 0x000000000x40000ec8: 0x00000000 0x00000000 0x00000000 0x000000000x40000ed8: 0x00000000 0x00000000 0x00000000 0x000000000x40000ee8: 0x00000000 0x00000000 0x00000000 0x000000000x40000ef8: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) p **preloads
$7 = {l_addr = 1077391360, l_name = 0x40000bd8 "/lib/libncurses.so.5",
l_ld = 0x403b0d18, l_next = 0x40000e88, l_prev = 0x403d8180, l_real = 0x40000bf0, l_ns = 0, l_libname = 0x40000e44, l_info = {0x0, 0x403b0d20, 0x403b0d70, 0x403b0d68, 0x403b0d40, 0x403b0d48, 0x403b0d50, 0x403b0d88, 0x403b0d90, 0x403b0d98, 0x403b0d58, 0x403b0d60, 0x403b0d30, 0x403b0d38, 0x403b0d28, 0x0, 0x0, 0x0, 0x0, 0x0, 0x403b0d78, 0x0, 0x0, 0x403b0d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x403b0da8, 0x403b0da0, 0x0, 0x0, 0x0, 0x0, 0x403b0db8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x403b0db0, 0x0 <repeats 26 times>}, l_phdr = 0x4037b034, l_entry = 1077435232, l_phnum = 4, l_ldnum = 26, l_searchlist = { r_list = 0x0, r_nlist = 0}, l_symbolic_searchlist = {r_list = 0x40000e40, r_nlist = 0}, l_loader = 0x403d8880, l_versions = 0x0, l_nversions = 0, l_nbuckets = 521, l_gnu_bitmask_idxbits = 0, l_gnu_shift = 0, l_gnu_bitmask = 0x0, {l_gnu_buckets = 0x4037b8e0, l_chain = 0x4037b8e0}, { l_gnu_chain_zero = 0x4037b0bc, l_buckets = 0x4037b0bc}, l_direct_opencount = 0, l_type = lt_library, l_relocated = 0, l_init_called = 0, l_global = 0, l_reserved = 1, l_phdr_allocated = 0, l_soname_added = 0, l_faked = 0, l_need_tls_init = 0, l_used = 0, l_auditing = 0, l_audit_any_plt = 0, l_removed = 0, l_contiguous = 1, l_rpath_dirs = {dirs = 0x0, malloced = 0}, l_reloc_result = 0x0, l_versyms = 0x0, l_origin = 0x40000e60 "/lib", l_map_start = 1077391360, l_map_end = 1077616976, l_text_end = 1077616640, l_scope_mem = {0x403d89dc, 0x0, 0x0, 0x0}, l_scope_max = 4, l_scope = 0x40000da8, l_local_scope = { 0x40000d4c, 0x0}, l_dev = 536937216, l_ino = 641363, l_runpath_dirs = { dirs = 0x0, malloced = 0}, l_initfini = 0x40001398, l_reldepsmax = 0, l_reldeps = 0x0, l_feature_1 = 0, l_flags_1 = 0, l_flags = 0, l_idx = 0, l_mach = {fptr_table_len = 0, fptr_table = 0x0}, l_lookup_cache = { sym = 0x0, type_class = 0, value = 0x0, ret = 0x0}, l_tls_initimage = 0x0, l_tls_initimage_size = 0, l_tls_blocksize = 0, l_tls_align = 0, l_tls_firstbyte_offset = 0, l_tls_offset = 0, l_tls_modid = 0, l_relro_addr = 0, l_relro_size = 0, l_serial = 2, l_audit = 0x40000e40}
Register $r10 contains the address of the next link map:
(gdb) p (*preloads)->l_next
$10 = (struct link_map *) 0x40000e88
(gdb) p *(*preloads)->l_next
$11 = {l_addr = 1076473856, l_name = 0x40000e78 "/lib/libdl.so.2",
l_ld = 0x4029e5fc, l_next = 0x40001118, l_prev = 0x40000bf0, l_real = 0x40000e88, l_ns = 0, l_libname = 0x400010dc, l_info = {0x0, 0x4029e5fc, 0x0 <repeats 74 times>}, l_phdr = 0x4029b034, l_entry = 1076477148, l_phnum = 7, l_ldnum = 31, l_searchlist = { r_list = 0x0, r_nlist = 0}, l_symbolic_searchlist = {r_list = 0x400010d8, r_nlist = 0}, l_loader = 0x403d8880, l_versions = 0x0, l_nversions = 0, l_nbuckets = 0, l_gnu_bitmask_idxbits = 0, l_gnu_shift = 0, l_gnu_bitmask = 0x0, {l_gnu_buckets = 0x0, l_chain = 0x0}, { l_gnu_chain_zero = 0x0, l_buckets = 0x0}, l_direct_opencount = 0, l_type = lt_library, l_relocated = 0, l_init_called = 0, l_global = 0, l_reserved = 1, l_phdr_allocated = 0, l_soname_added = 0, l_faked = 0, l_need_tls_init = 0, l_used = 0, l_auditing = 0, l_audit_any_plt = 0, l_removed = 0, l_contiguous = 1, l_rpath_dirs = {dirs = 0x0, malloced = 0}, l_reloc_result = 0x0, l_versyms = 0x0, l_origin = 0x400010f8 "/lib", l_map_start = 1076473856, l_map_end = 1076488476, l_text_end = 1076490240, l_scope_mem = {0x403d89dc, 0x0, 0x0, 0x0}, l_scope_max = 4, l_scope = 0x40001040, l_local_scope = {0x40000fe4, 0x0}, l_dev = 536937216, l_ino = 641559, l_runpath_dirs = {dirs = 0x0, malloced = 0}, l_initfini = 0x0, l_reldepsmax = 0, l_reldeps = 0x0, l_feature_1 = 0, l_flags_1 = 0, l_flags = 0, l_idx = 0, l_mach = {fptr_table_len = 0, fptr_table = 0x0}, l_lookup_cache = {sym = 0x0, type_class = 0, value = 0x0, ret = 0x0}, l_tls_initimage = 0x0, l_tls_initimage_size = 0, l_tls_blocksize = 0, l_tls_align = 0, l_tls_firstbyte_offset = 0, l_tls_offset = 0, l_tls_modid = 0, l_relro_addr = 0, l_relro_size = 0, l_serial = 3, l_audit = 0x400010d8}
(gdb) p &(*preloads)->l_next->l_info
$14 = (Elf32_Dyn *(*)[76]) 0x40000ea8
(gdb) p (*preloads)->l_next->l_info
$15 = {0x0, 0x4029e5fc, 0x0 <repeats 74 times>}
(gdb) p/x 0x40000ea8 + 5 * 4
$17 = 0x40000ebc
So, the segmentation fault was caused by a 0x0 in the l_info field of the link map for "/lib/libdl.so.2".
elf.h:#define DT_STRTAB 5 /* Address of string table */
This is the code that causes the segv:
const char *strtab = (const void *) D_PTR (l, l_info[DT_STRTAB]);
$ readelf -a /lib/libdl.so.2 ... Dynamic section at offset 0x25fc contains 27 entries: Tag Type Name/Value 0x00000001 (NEEDED) Shared library: [libc.so.6] 0x00000001 (NEEDED) Shared library: [ld.so.1] 0x0000000e (SONAME) Library soname: [libdl.so.2] 0x0000000c (INIT) 0xc9c 0x0000000d (FINI) 0x20bc 0x00000019 (INIT_ARRAY) 0x3590 0x0000001b (INIT_ARRAYSZ) 4 (bytes) 0x00000004 (HASH) 0x2168 0x6ffffef5 (GNU_HASH) 0x134 0x00000005 (STRTAB) 0x47c
It would appear there should be a string table.
Carlos, what do you think?
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 23 lines
Register $r10 contains the address of the next link map:
(gdb) p (*preloads)->l_next
$10 = (structlink_map*)0x40000e88(gdb) p *(*preloads)->l_next
$11 = {l_addr= 1076473856,l_name=0x40000e78"/lib/libdl.so.2",
l_ld=0x4029e5fc,l_next=0x40001118,l_prev=0x40000bf0,l_real=0x40000e88,l_ns= 0,l_libname=0x400010dc,l_info= {0x0,0x4029e5fc,0x0<repeats 74 times>},l_phdr=0x4029b034,l_entry= 1076477148,l_phnum= 7,l_ldnum= 31,l_searchlist= {r_list=0x0,r_nlist= 0},l_symbolic_searchlist= {r_list=0x400010d8,r_nlist= 0},l_loader=0x403d8880,l_versions=0x0,l_nversions= 0,l_nbuckets= 0,l_gnu_bitmask_idxbits= 0,l_gnu_shift= 0,l_gnu_bitmask=0x0, {l_gnu_buckets=0x0,l_chain=0x0}, {l_gnu_chain_zero=0x0,l_buckets=0x0},l_direct_opencount= 0,l_type=lt_library,l_relocated= 0,l_init_called= 0,l_global= 0,l_reserved= 1,l_phdr_allocated= 0,l_soname_added= 0,l_faked= 0,l_need_tls_init= 0,l_used= 0,l_auditing= 0,l_audit_any_plt= 0,l_removed= 0,l_contiguous= 1,l_rpath_dirs= {dirs =0x0, malloced = 0},l_reloc_result=0x0,l_versyms=0x0,l_origin=0x400010f8"/lib",l_map_start= 1076473856,l_map_end= 1076488476,l_text_end= 1076490240,l_scope_mem= {0x403d89dc,0x0,0x0,0x0},l_scope_max= 4,l_scope=0x40001040,l_local_scope= {0x40000fe4,0x0},l_dev= 536937216,
Quoted 8 lines
l_ino= 641559,l_runpath_dirs= {dirs =0x0, malloced = 0},l_initfini=0x0,l_reldepsmax= 0,l_reldeps=0x0,l_feature_1= 0,l_flags_1= 0,l_flags= 0,l_idx= 0,l_mach= {fptr_table_len= 0,fptr_table=0x0},l_lookup_cache= {sym =0x0,type_class= 0, value =0x0, ret =0x0},l_tls_initimage=0x0,l_tls_initimage_size= 0,l_tls_blocksize= 0,l_tls_align= 0,l_tls_firstbyte_offset= 0,l_tls_offset= 0,l_tls_modid= 0,l_relro_addr= 0,l_relro_size= 0,l_serial= 3,l_audit=0x400010d8}
When I run the command directly under gdb, the info is a different location and the string table entry is not null:
(gdb) bt
#0 _dl_map_object_deps (map=0x403d8880, preloads=0x40001170, npreloads=12,
trace_mode=0, open_mode=0) at dl-deps.c:224
#1 0x403b9bd4 in dl_main (phdr=0x10034, phnum=<value optimized out>,
user_entry=<value optimized out>) at rtld.c:1780
#2 0x403cb898 in _dl_sysdep_start (start_argptr=<value optimized out>,
dl_main=@0x403d7566: 0x403b8fe4 <dl_main>) at ../elf/dl-sysdep.c:239
#3 0x403b785c in _dl_start_final (arg=0xbff01028, info=0xbff01188)
at rtld.c:332#4 0x403b7adc in _dl_start (arg=0xbff01028) at rtld.c:560
(gdb) p *(*preloads)->l_next
$2 = {l_addr = 1076473856, l_name = 0x40000c50 "/lib/libdl.so.2",
l_ld = 0x4029e5fc, l_next = 0x40000ef0, l_prev = 0x400009c8, l_real = 0x40000c60, l_ns = 0, l_libname = 0x40000eb4, l_info = {0x0, 0x4029e604, 0x4029e66c, 0x4029e664, 0x4029e634, 0x4029e644, 0x4029e64c, 0x4029e684, 0x4029e68c, 0x4029e694, 0x4029e654, 0x4029e65c, 0x4029e614, 0x4029e61c, 0x4029e60c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4029e674, 0x0, 0x0, 0x4029e67c, 0x0, 0x4029e624, 0x0, 0x4029e62c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4029e6b4, 0x4029e6ac, 0x4029e6a4, 0x4029e69c, 0x0, 0x0, 0x4029e6c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4029e6bc,
0x0 <repeats 25 times>, 0x4029e63c}, l_phdr = 0x4029b034, l_entry = 1076477148, l_phnum = 7, l_ldnum = 31, l_searchlist = { r_list = 0x0, r_nlist = 0}, l_symbolic_searchlist = {r_list = 0x40000eb0, r_nlist = 0}, l_loader = 0x403d8880, l_versions = 0x0, l_nversions = 0, l_nbuckets = 22, l_gnu_bitmask_idxbits = 3, l_gnu_shift = 7, l_gnu_bitmask = 0x4029b144, {l_gnu_buckets = 0x4029b154, l_chain = 0x4029b154}, {l_gnu_chain_zero = 0x4029b148, l_buckets = 0x4029b148}, l_direct_opencount = 0, l_type = lt_library, l_relocated = 0, l_init_called = 0, l_global = 0, l_reserved = 1, l_phdr_allocated = 0, l_soname_added = 0, l_faked = 0, l_need_tls_init = 0, l_used = 0, l_auditing = 0, l_audit_any_plt = 0, l_removed = 0, l_contiguous = 1, l_rpath_dirs = {dirs = 0x0, malloced = 0}, l_reloc_result = 0x0, l_versyms = 0x0, l_origin = 0x40000ed0 "/lib", l_map_start = 1076473856, l_map_end = 1076488476, l_text_end = 1076490240, l_scope_mem = {0x403d89dc, 0x0, 0x0, 0x0}, l_scope_max = 4, l_scope = 0x40000e18, l_local_scope = {0x40000dbc, 0x0}, l_dev = 536937216, l_ino = 641559, l_runpath_dirs = {dirs = 0x0, malloced = 0}, l_initfini = 0x0, l_reldepsmax = 0, l_reldeps = 0x0, l_feature_1 = 0, l_flags_1 = 0, l_flags = 0, l_idx = 0, l_mach = {fptr_table_len = 0, fptr_table = 0x0}, l_lookup_cache = {sym = 0x0, type_class = 0, value = 0x0, ret = 0x0}, l_tls_initimage = 0x0, l_tls_initimage_size = 0, l_tls_blocksize = 0, l_tls_align = 0, l_tls_firstbyte_offset = 0, l_tls_offset = 0, l_tls_modid = 0, l_relro_addr = 0, l_relro_size = 0, l_serial = 3, l_audit = 0x40000eb0}
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 35 lines
(gdb) p *(*preloads)->l_next
$11 = {l_addr= 1076473856,l_name=0x40000e78"/lib/libdl.so.2",
l_ld=0x4029e5fc,l_next=0x40001118,l_prev=0x40000bf0,l_real=0x40000e88,l_ns= 0,l_libname=0x400010dc,l_info= {0x0,0x4029e5fc,0x0<repeats 74 times>},l_phdr=0x4029b034,l_entry= 1076477148,l_phnum= 7,l_ldnum= 31,l_searchlist= {r_list=0x0,r_nlist= 0},l_symbolic_searchlist= {r_list=0x400010d8,r_nlist= 0},l_loader=0x403d8880,l_versions=0x0,l_nversions= 0,l_nbuckets= 0,l_gnu_bitmask_idxbits= 0,l_gnu_shift= 0,l_gnu_bitmask=0x0, {l_gnu_buckets=0x0,l_chain=0x0}, {l_gnu_chain_zero=0x0,l_buckets=0x0},l_direct_opencount= 0,l_type=lt_library,l_relocated= 0,l_init_called= 0,l_global= 0,l_reserved= 1,l_phdr_allocated= 0,l_soname_added= 0,l_faked= 0,l_need_tls_init= 0,l_used= 0,l_auditing= 0,l_audit_any_plt= 0,l_removed= 0,l_contiguous= 1,l_rpath_dirs= {dirs =0x0, malloced = 0},l_reloc_result=0x0,l_versyms=0x0,l_origin=0x400010f8"/lib",l_map_start= 1076473856,l_map_end= 1076488476,l_text_end= 1076490240,l_scope_mem= {0x403d89dc,0x0,0x0,0x0},l_scope_max= 4,l_scope=0x40001040,l_local_scope= {0x40000fe4,0x0},l_dev= 536937216,l_ino= 641559,l_runpath_dirs= {dirs =0x0, malloced = 0},l_initfini=0x0,l_reldepsmax= 0,l_reldeps=0x0,l_feature_1= 0,l_flags_1= 0,l_flags= 0,l_idx= 0,l_mach= {fptr_table_len= 0,fptr_table=0x0},l_lookup_cache= {sym =0x0,type_class= 0, value =0x0, ret =0x0},l_tls_initimage=0x0,l_tls_initimage_size= 0,l_tls_blocksize= 0,l_tls_align= 0,l_tls_firstbyte_offset= 0,l_tls_offset= 0,l_tls_modid= 0,l_relro_addr= 0,l_relro_size= 0,l_serial= 3,l_audit=0x400010d8}(gdb) p &(*preloads)->l_next->l_info
$14 = (Elf32_Dyn*(*)[76])0x40000ea8
(gdb) p (*preloads)->l_next->l_info
$15 = {0x0,0x4029e5fc,0x0<repeats 74 times>}
(gdb) p/x0x40000ea8+ 5 * 4
$17 =0x40000ebc
Quoted 2 lines
So, the segmentation fault was caused by a
0x0in thel_infofield of the link map for "/lib/libdl.so.2".
After staring at the dynamic loader code for a while, I think the following mmap call in dl-load.c doesn't correctly map the info data for /lib/libdl.so.2.
/* Remember which part of the address space this object uses. */ l->l_map_start = (ElfW(Addr)) __mmap ((void *) mappref, maplength, c->prot, MAP_COPY|MAP_FILE, fd, c->mapoff);
The info data is near the end of the mapped segment. The l_info field is initialized by elf_get_dynamic_info from the dynamic data mapped at l->ld.
I seem to recall that the kernel mmap implementation on hppa is somewhat unique.
In the above call, mappref is NULL. The kernel selects the map location.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 1 line · John David Anglin
Carlos, what do you think?
I think the dynamic linker is always the first to touch the mmap'd file and therefore the most likely to fail if something is wrong in our VM layer.
Cheers,
Carlos.
Quoted 1 line · John David Anglin
After staring at the dynamic loader code for a while, I think the fol=
lowing
Quoted 1 line
mmap call in
dl-load.cdoesn't correctly map the info data for /lib/l=
ibdl.so.2.
Quoted 2 lines
=A0 =A0 =A0 =A0/* Remember which part of the address space this objec=
t uses. =A0*/
Quoted 1 line
=A0 =A0 =A0 =A0l->l_map_start =3D (ElfW(Addr))
__mmap((void *) mappr=
ef, maplength,
Quoted 1 line
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0c->prot,
Quoted 1 line
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0MAP_COPY|MAP_FILE,
Quoted 1 line
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0fd, c->mapoff);
Quoted 2 lines
The info data is near the end of the mapped segment. =A0The
l_infofi=
eld
Quoted 2 lines
is initialized by
elf_get_dynamic_infofrom the dynamic data mapped atl->ld.
Why do you think this is wrong?
Quoted 1 line
I seem to recall that the kernel mmap implementation on hppa is somew=
hat
Quoted 1 line
unique.
I don't recall anything, Kyle?
Quoted 1 line
In the above call, mappref is NULL. =A0The kernel selects the map loc=
ation.
Yes, that's probably correct, the loader is letting the kernel choose the address, at this point we don't care where the library is loaded.
Cheers,
Carlos.
--
To unsubscribe from this list: send the line "unsubscribe linux-parisc"=
in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines
Quoted 3 lines
The info data is near the end of the mapped segment. =A0The
l_infofield is initialized byelf_get_dynamic_infofrom the dynamic data mapped atl->ld.Why do you think this is wrong?
I don't know about the specifics. My supposition is that we may not be copying the entire segment depending on where the map is placed.
Quoted 4 lines
Quoted 2 lines
I seem to recall that the kernel mmap implementation on hppa is somewhat unique.
I don't recall anything, Kyle?
This came up with respect to the GCC PCH implementation for parisc. See comments in host-hpux.h. At the moment, we do have a PCH related bug. See PR 39355. While I know the problem is present in the PCH file, I haven't been able to figure out how wrong data gets in the file.
Quoted 5 lines
Quoted 1 line
In the above call, mappref is NULL. =A0The kernel selects the map locatio=
n.
Yes, that's probably correct, the loader is letting the kernel choose the address, at this point we don't care where the library is loaded.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 8 lines · John David Anglin
Quoted 5 lines
Quoted 3 lines
The info data is near the end of the mapped segment. =A0The
l_infofield is initialized byelf_get_dynamic_infofrom the dynamic data mapped atl->ld.Why do you think this is wrong?
I don't know about the specifics. My supposition is that we may not be copying the entire segment depending on where the map is placed.
Who is supposed to copy the segment? The user or the kernel?
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I seem to recall that the kernel mmap implementation on hppa is somewhat unique.
I don't recall anything, Kyle?
This came up with respect to the GCC PCH implementation for parisc. See comments in
host-hpux.h. At the moment, we do have a PCH related bug. See PR 39355. While I know the problem is present in the PCH file, I haven't been able to figure out how wrong data gets in the file.
Do you have a bugzilla reference so we can take a look ... also, is this likely a tool chain problem or a kernel one? If it's a kernel one could someone provide a description of what they think is going wrong?
James
Quoted 11 lines · John David Anglin
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The info data is near the end of the mapped segment. =A0The
l_infofield is initialized byelf_get_dynamic_infofrom the dynamic data mapped atl->ld.Why do you think this is wrong?
I don't know about the specifics. My supposition is that we may not be copying the entire segment depending on where the map is placed.
Who is supposed to copy the segment? The user or the kernel?
As far as I can tell, the kernel is responsible for mapping the segment. Then, elf_get_dynamic_info processes the mapped data to generate the l_info field in the link map.
The kernel has control over where the segment is mapped. In as much as the dependencies for /bin/sh are processed many times in a GCC build, I have to think the problem relates to the placement or a random problem with mmap. There is a small possibility that the processing of the data by elf_get_dynamic_info has a problem.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
I seem to recall that the kernel mmap implementation on hppa is somewhat unique.
I don't recall anything, Kyle?
Quoted 8 lines
Quoted 4 lines
This came up with respect to the GCC PCH implementation for parisc. See comments in
host-hpux.h. At the moment, we do have a PCH related bug. See PR 39355. While I know the problem is present in the PCH file, I haven't been able to figure out how wrong data gets in the file.Do you have a bugzilla reference so we can take a look ... also, is this likely a tool chain problem or a kernel one? If it's a kernel one could someone provide a description of what they think is going wrong?
The bugzilla reference is <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39355>.
Many of my comments in the PR are wrong. See comment #47. At this time, I know that the data in the PCH file are wrong but I don't know why. In any case, I should not have mentioned the PCH bug. It's probably a tool chain bug. The main point of the comment was that the hppa mmap implementation differs from other implementations (MAP_PRIVATE is not reliable). So, there may be other issues.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines
Quoted 1 line · John David Anglin
Carlos, what do you think?
I think the dynamic linker is always the first to touch the mmap'd file and therefore the most likely to fail if something is wrong in our VM layer.
From the core dump, this is the data supposed processed by
elf_get_dynamic_info:
(gdb) x/64x l->l_ld0x4029e5fc <.LC11+4>: 0x00000001 0x00000167 0x00000001 0x000001710x4029e60c <.LC11+20>: 0x0000000e 0x00000179 0x0000000c 0x00000c9c0x4029e61c <.LC11+36>: 0x0000000d 0x000020bc 0x00000019 0x000035900x4029e62c <.LC11+52>: 0x0000001b 0x00000004 0x00000004 0x000021680x4029e63c <.LC11+68>: 0x6ffffef5 0x00000134 0x00000005 0x0000047c0x4029e64c <.LC11+84>: 0x00000006 0x000001ec 0x0000000a 0x000001c80x4029e65c <.LC11+100>: 0x0000000b 0x00000010 0x00000003 0x000037fc0x4029e66c <.LC11+116>: 0x00000002 0x0000015c 0x00000014 0x000000070x4029e67c <.LC11+132>: 0x00000017 0x00000b40 0x00000007 0x000007b00x4029e68c <.LC11+148>: 0x00000008 0x00000390 0x00000009 0x0000000c0x4029e69c <.LC11+164>: 0x6ffffffc 0x00000698 0x6ffffffd 0x00000006
(gdb) p l->l_ld[9]
$15 = {d_tag = 5, d_un = {d_val = 1148, d_ptr = 1148}} (gdb) p/x 1148
$16 = 0x47c
The numbers seem to match that printed by readelf. So, maybe Carlos is right and this is a VM issue. Is it possible the code saw different data (e.g., the mmap operation was not complete before the mmap call returned)?
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
I seem to recall that the kernel mmap implementation on hppa is somewhat unique.
I don't recall anything, Kyle?
This came up with respect to the GCC PCH implementation for parisc. See comments in
host-hpux.h. At the moment, we do have a PCH related bug. See PR 39355. While I know the problem is present in the PCH file, I haven't been able to figure out how wrong data gets in the file.
There are some limitations on hppa if a file is both opened for reading (via read()) and written to via a mmap'ed mapping. This came up a few years ago.
Does gcc do this?
randolph
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
I seem to recall that the kernel mmap implementation on hppa is somewhat unique.
I don't recall anything, Kyle?
This came up with respect to the GCC PCH implementation for parisc. See comments in
host-hpux.h. At the moment, we do have a PCH related bug. See PR 39355. While I know the problem is present in the PCH file, I haven't been able to figure out how wrong data gets in the file.
Quoted 5 lines
There are some limitations on hppa if a file is both opened for reading (via
read()) and written to via a mmap'ed mapping. This came up a few years ago.Does gcc do this?
Not that I am aware of. The situation is essentially the reverse of the above. Data is written from a region of memory. Then, in another instance of gcc, it needs to be mmap'ed back to the same location in memory. In theory, it could be brought back to a different location but this would require a fairly complex set of relocations.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 5 lines · John David Anglin
Not that I am aware of. The situation is essentially the reverse of the above. Data is written from a region of memory. Then, in another instance of gcc, it needs to be mmap'ed back to the same location in memory. In theory, it could be brought back to a different location but this would require a fairly complex set of relocations.
GCC does not read() and write to the mmap()'d file.
The dynamic loader uses MAP_DENYWRITE to avoid writing into the mmap()'d memory.
I will reiterate my point here that the dynamic linker the first user of mmap in a newly started process, and the first program to read and process data from the mmap'd files. Therefore the dynamic linker is always the first to suffer if a mapped region of memory is not correct.
What we need to do here is create a test case.
I tried this:
1. cp /lib/libc.so.6 original.so
2. cp /lib/libc.so.6 map.so
3. gcc -O2 -g -o test-mmap test-mmap.c
4. while true; do ./test-mmap ./original.so ./map.so; done;
The test mmap's a file and compares it to the original, aborting if the comparison fails. I've yet to see it abort on my a500, and I've run 20-30 instances of the test simultaneously. Then again I don't see any serious segv's like others do (2.6.26-1-parisc64-smp).
What might be a better testcase?
Cheers,
Carlos.
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h> /* mmap */
#include <sys/types.h> /* open */
#include <sys/stat.h> /* open */
#include <fcntl.h> /* open */
#include <unistd.h> /* lseek */
#define BMAX 4096
int
main (int argc, char **argv)
{
void *mappref;
int fd, fdc;
off_t maplength, index, j;
char *original = argv[1], *copy = argv[2];
char buf[BMAX], *mbuf;
ssize_t ret;
/* Open original file to compute size. We open the original
file to simulate having the fd open before mmap as the dynamic loader does. */
fd = open (original, O_RDONLY);
if (fd == -1)
{
perror ("open");
abort ();
}
maplength = lseek (fd, 0, SEEK_END);
if (fd == -1)
{
perror ("lseek");
abort ();
}
/* Now mmap the open file. */
mappref = mmap ((void *)mappref,
maplength,
PROT_READ | PROT_EXEC,
MAP_PRIVATE | MAP_DENYWRITE | MAP_FILE,
fd,
0);
if (mappref == (void *)-1)
{
perror ("mmap");
abort ();
}
mbuf = (char *)mappref;
/* Compare mmap to copy. */
fdc = open (copy, O_RDONLY);
if (fdc == -1)
{
perror ("open #2");
abort ();
}
for (index = 0; index < maplength; index += BMAX)
{
ret = read (fdc, &buf[0], BMAX);
if ((ret != BMAX) && (ret == -1))
{
perror ("read");
abort ();
}
for (j = 0; ((j < BMAX) && ((index + j) < maplength)); j++)
{
if (mbuf[index + j] != buf[j])
{
fprintf(stderr, "Mismatch at %ld, read %d, expected %d\n",
index + j, (unsigned int)mbuf[index + j], (unsigned int)buf[j]);
abort ();
}
if (DEBUG)
printf ("Match at %ld, read %d, expected %d\n",
index + j, (unsigned int)mbuf[index + j], (unsigned int)buf[j]);
}
} return 0;
}
Quoted 5 lines
I will reiterate my point here that the dynamic linker the first user of mmap in a newly started process, and the first program to read and process data from the mmap'd files. Therefore the dynamic linker is always the first to suffer if a mapped region of memory is not correct.
That is true to a certain extent. However, there are large portions of code and initialized data that it doesn't touch. I don't think that I've ever seen an invalid instruction fault. So, I'm not fully convinced that we understand the cause of these segvs.
As far as I can tell, the mmap'd data appears correct (at least as far as what was recorded in the core file). What is wrong is the l_info field in the linker map. Prior to failing on processing libdl.so.2, it had successfully processed itself and libncurses.so.5 (see NEEDED entries for /bin/sh). There isn't a lot that happens between mmap'ing the file and the access to the STRTAB entry in the l_info field. The NEEDED entry at l_info[1] seems ok in the dump.
I doubt this is a TLB issue as the data is a long way from page boundaries. Possibly, there is a cache line issue in the mmap'd file, or as I suggested before a race condition and the file isn't fully mapped when the mmap call returns. In any case, the extraction of the dynamic data failed after doing the first NEEDED entry.
I have to think that this has something to do with the machine being a rp3440 (large memory and cache). I have never seen this on my c3750 with 32-bit UP kernel. Also, this was with a 64-bit UP kernel.
Quoted 9 lines
What we need to do here is create a test case.
I tried this:
1. cp
/lib/libc.so.6original.so
2. cp/lib/libc.so.6map.so
3. gcc -O2 -g -o test-mmaptest-mmap.c
4. while true; do ./test-mmap ./original.so ./map.so; done;
Quoted 6 lines
The test mmap's a file and compares it to the original, aborting if the comparison fails. I've yet to see it abort on my a500, and I've run 20-30 instances of the test simultaneously. Then again I don't see any serious segv's like others do (2.6.26-1-parisc64-smp).
What might be a better testcase?
I typically run my GCC builds with `make -j 4'. So, there's a mix of other stuff actively running at any time.
I'll give the testcase a try tonight.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 5 lines · John David Anglin
I have to think that this has something to do with the machine being a rp3440 (large memory and cache). I have never seen this on my c3750 with 32-bit UP kernel. Also, this was with a 64-bit UP kernel.
If I remember correctly, there's still some issues with the L2 cache on pa8800 that we haven't quite bothered to work out yet, since it's "good enough" for now. James probably knows more. It would be interesting to see if you could reproduce it with a UP 64-bit kernel on your C3750 to discount the L2 problems.
regards, Kyle
Quoted 5 lines
If I remember correctly, there's still some issues with the L2 cache on pa8800 that we haven't quite bothered to work out yet, since it's "good enough" for now. James probably knows more. It would be interesting to see if you could reproduce it with a UP 64-bit kernel on your C3750 to discount the L2 problems.
Googling, I see Grant had trouble with RCU_TORTURE_TEST=y. Probably, I should test current kernel to see if the problem is still present.
I guess you are referring to this change:
http://fossplanet.com/linux.debian.devel.kernel.cvs/thread-4378354-r7141-patches/
I'm thinking we must be missing a flush... Maybe in clear_user_page as for copy_user_page?
Do the problematic debian buildd machines have pa8800/pa8900 processors?
My sense is that some change (probably to the core memory management code) made the coherence issue worse post 2.6.22.x.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 9 lines · John David Anglin
Quoted 5 lines
If I remember correctly, there's still some issues with the L2 cache on pa8800 that we haven't quite bothered to work out yet, since it's "good enough" for now. James probably knows more. It would be interesting to see if you could reproduce it with a UP 64-bit kernel on your C3750 to discount the L2 problems.
Googling, I see Grant had trouble with RCU_TORTURE_TEST=y. Probably, I should test current kernel to see if the problem is still present.
Quoted 7 lines
I guess you are referring to this change:
http://fossplanet.com/linux.debian.devel.kernel.cvs/thread-4378354-r7141-patches/I'm thinking we must be missing a flush... Maybe in
clear_user_pageas forcopy_user_page?Do the problematic debian buildd machines have pa8800/pa8900 processors?
dannf@penalosa:~$ cat /proc/cpuinfo processor : 0 cpu family : PA-RISC 2.0 cpu : PA8700 (PCX-W2) cpu MHz : 750.000000 model : 9000/785/J6700 model name : Duet W2 hversion : 0x00005dd0 sversion : 0x00000491 I-cache : 768 KB D-cache : 1536 KB (WB, direct mapped) ITLB entries : 240 DTLB entries : 240 - shared with ITLB bogomips : 1495.04 software id : 2001606322
Quoted 4 lines
My sense is that some change (probably to the core memory management code) made the coherence issue worse post 2.6.22.x.
Dave
Signature · 1 line
dann frazier
Quoted 7 lines · dann frazier
dannf@penalosa:~$ cat /proc/cpuinfo processor =A0 =A0 =A0 =A0 : 0 cpu family =A0 =A0 =A0 =A0: PA-RISC 2.0 cpu =A0 =A0 =A0 =A0 =A0 =A0 =A0 : PA8700 (PCX-W2) cpu MHz =A0 =A0 =A0 =A0 =A0 =A0 : 750.000000 model =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 : 9000/785/J6700 model name =A0 =A0 =A0 =A0 =A0 =A0: Duet W2
Quoted 8 lines
hversion =A0 =A0 =A0 =A0 =A0 =A0 =A0: 0x00005dd0 sversion =A0 =A0 =A0 =A0 =A0 =A0 =A0: 0x00000491 I-cache =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 : 768 KB D-cache =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 : 1536 KB (WB, direct mapped) ITLB entries =A0 =A0 =A0 =A0 =A0 =A0 =A0: 240 DTLB entries =A0 =A0 =A0 =A0 =A0 =A0 =A0: 240 - shared with ITLB bogomips =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: 1495.04 software id =A0 =A0 =A0 =A0 =A0 =A0 =A0 : 2001606322
This is very similar to my own box. In frustration I've started installing a buildd on my own a500. I might as well run a buildd to load the machine and see if any of the reported package FTBS crop up.
Cheers,
Carlos.
Quoted 6 lines · John David Anglin
Quoted 5 lines
If I remember correctly, there's still some issues with the L2 cache on pa8800 that we haven't quite bothered to work out yet, since it's "good enough" for now. James probably knows more. It would be interesting to see if you could reproduce it with a UP 64-bit kernel on your C3750 to discount the L2 problems.
Quoted 8 lines
Googling, I see Grant had trouble with RCU_TORTURE_TEST=y. Probably, I should test current kernel to see if the problem is still present.
I guess you are referring to this change:
http://fossplanet.com/linux.debian.devel.kernel.cvs/thread-4378354-r7141-patches/I'm thinking we must be missing a flush... Maybe in
clear_user_pageas forcopy_user_page?
Clear user page is very clever and doesn't need a flush. What it does is clear the page through the users cache rather than the kernel's ... what it actually does is place zeros in the cache above the physical page, so the user can only access the zeros ... they eventually get cleaned back to the page as the cache empties.
Quoted 1 line
Do the problematic debian buildd machines have pa8800/pa8900 processors?
No boxes outside of the cupertino test ring and a few giveaways (you, kyle and t-bone, I think) have pa88/8900 cpus.
Quoted 2 lines
My sense is that some change (probably to the core memory management code) made the coherence issue worse post 2.6.22.x.
So if I characterise the problem you think you're seeing: on mmap of a file at a memory location to be determined by the kernel, a sequential set of reads of the mapped location eventually turns up a zero where there should be data? Yes, it does sound like a caching issue.
James
Quoted 7 lines · Kyle McMartin
Quoted 5 lines · John David Anglin
I have to think that this has something to do with the machine being a rp3440 (large memory and cache). I have never seen this on my c3750 with 32-bit UP kernel. Also, this was with a 64-bit UP kernel.
Quoted 5 lines
If I remember correctly, there's still some issues with the L2 cache on pa8800 that we haven't quite bothered to work out yet, since it's "good enough" for now. James probably knows more. It would be interesting to see if you could reproduce it with a UP 64-bit kernel on your C3750 to discount the L2 problems.
The pa8800/8900 are very special systems. They have an L1/L2 VIPT cache but an L2 PIPT one. All other parisc systems have VIPT throughout.
The problem with the VIPT/PIPT combination is that you can get read only stale alias resolution between the VIPT/PIPT hierarchy ... linux just doesn't expect this to happen. We work around this with extra flushes in kmap to prevent the read only alias resolution from happening. There's nothing missing in the logic of this, the only problem is that its operation treats the vast L2 PIPT cache as a useless boat anchor whose only job is to make life harder for us ... so we're getting absolutely no benefit from the 25-50MB of cache there.
James
Quoted 4 lines
So if I characterise the problem you think you're seeing: on mmap of a file at a memory location to be determined by the kernel, a sequential set of reads of the mapped location eventually turns up a zero where there should be data? Yes, it does sound like a caching issue.
Yes. The loop is terminated by a null tag:
while (dyn->d_tag != DT_NULL)
{
...
}However, the core dump doesn't show a null tag before the STRTAB tag that caused the segmentation fault.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)
Quoted 1 line · John David Anglin
Quoted 1 line
So if I characterise the problem you think you're seeing: on mmap of=
a
Quoted 1 line
Quoted 1 line
file at a memory location to be determined by the kernel, a sequenti=
al
Quoted 5 lines
Quoted 2 lines
set of reads of the mapped location eventually turns up a zero where there should be data? =A0Yes, it does sound like a caching issue.
Yes. =A0The loop is terminated by a null tag:
Quoted 7 lines
=A0while (
dyn->d_tag!=3DDT_NULL)
=A0 =A0 =A0{
=A0 =A0 =A0 =A0 ...
=A0 =A0 =A0}However, the core dump doesn't show a null tag before the STRTAB tag that caused the segmentation fault.
Do you mean "after" the STRTAB tag? I assume the library on-disk has a DT_NULL, otherwise it would always fail.
Cheers,
Carlos.
--
To unsubscribe from this list: send the line "unsubscribe linux-parisc"=
in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Carlos O'Donell
Quoted 1 line · John David Anglin
Quoted 1 line
So if I characterise the problem you think you're seeing: on mmap =
of a
Quoted 1 line
Quoted 1 line
Quoted 1 line
file at a memory location to be determined by the kernel, a sequen=
tial
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of reads of the mapped location eventually turns up a zero whe=
re
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
there should be data? =A0Yes, it does sound like a caching issue.
Yes. =A0The loop is terminated by a null tag:
=A0while (
dyn->d_tag!=3DDT_NULL)
=A0 =A0 =A0{
=A0 =A0 =A0 =A0 ...
=A0 =A0 =A0}However, the core dump doesn't show a null tag before the STRTAB ta=
g
Quoted 3 lines
Quoted 1 line
that caused the segmentation fault.
=20
Do you mean "after" the STRTAB tag? I assume the library on-disk has =
a
Quoted 1 line
DT_NULL, otherwise it would always fail.
I'm sure that there is a null tag after the STRTAB. The segmentation fault occurred because the get operation failed after processing the first NEEDED tag and before the STRTAB tag. The loop goes sequentially through the array of DT objects in the recently mmap'd data and inserts pointers to these objects into the dynamic loaders link map for the file (in the l_info field). There were no null tags between the NEEDED entry and the STRTAB entry in the mmap'd data in the core dump. The DT objects are near the end of the mmap'd data.
I would guess that the loop terminated early because the l_info array is all zeros except for the first NEEDED entry. It appears correct. T= he
loop might have terminated early because of a cache issue, or possibly the value loaded from memory somehow got corrupted. Another possibilit= y
would be the mmap operation wasn't complete when the memory was examine= d
by the dynamic loader. When the core dump was done, the operation was complete.
I think it's less likely that a cache issue affected the memory used by the dynamic loader (l_info field) as the data before and after in the map seemed reasonable.
The fact PA8700 processors are also experiencing similar problems would seem to suggest that this isn't a PA8800 L2 issue unless we have multiple problems.
I think we need to try running a recent kernel on gsyprf11 for a while to see if we can capture a similar event.
Dave
--=20
J. David Anglin [email protected]=
c.ca
National Research Council of Canada (613) 990-0752 (FAX: 9=
52-6602)
--
To unsubscribe from this list: send the line "unsubscribe linux-parisc"=
in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · John David Anglin
I would guess that the loop terminated early because the
l_infoarray is all zeros except for the first NEEDED entry. =A0It appears correct=
=2E =A0The
Quoted 1 line
loop might have terminated early because of a cache issue, or possibl=
y
Quoted 1 line
the value loaded from memory somehow got corrupted. =A0Another possib=
ility
Quoted 1 line
would be the mmap operation wasn't complete when the memory was exami=
ned
Quoted 1 line
by the dynamic loader. =A0When the core dump was done, the operation =
was
Quoted 3 lines
complete.
I think it's less likely that a cache issue affected the memory used =
by
Quoted 5 lines
the dynamic loader (
l_infofield) as the data before and after in the map seemed reasonable.The fact PA8700 processors are also experiencing similar problems would seem to suggest that this isn't a PA8800 L2 issue unless we hav=
e
Quoted 3 lines
multiple problems.
I think we need to try running a recent kernel on gsyprf11 for a whil=
e
Quoted 1 line
to see if we can capture a similar event.
This rang a bell...
In glibc/elf/rtld.c we have this:
/* Partly clean the `bootstrap_map' structure up. Don't use
`memset' since it might not be built in or inlined and we cannot make function calls at this point. Use '__builtin_memset' if we know it is available. We do not have to clear the memory if we do not have to use the temporary bootstrap_map. Global variables are initialized to zero by default. */
#ifndef DONT_USE_BOOTSTRAP_MAP
# ifdef HAVE_BUILTIN_MEMSET
__builtin_memset (bootstrap_map.l_info, '\0', sizeof (bootstrap_map.l=
_info));
# else
for (size_t cnt =3D 0;
cnt < sizeof (bootstrap_map.l_info) / sizeof (bootstrap_map.l_in=
fo[0]);
++cnt)
bootstrap_map.l_info[cnt] =3D 0;
# endifOn hppa we don't have builtin memset (probably one of the few arches), so we fall back on this weird loop which I always thought was wrong.
I was seeing problems with l_info having garbage in it, so I had a local hack which cleared the entire bootstrap_map.
Did your l_info come from the bootstrap_map?
Cheers,
Carlos.
--
To unsubscribe from this list: send the line "unsubscribe linux-parisc"=
in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 45 lines
Quoted 2 lines · John David Anglin
I would guess that the loop terminated early because the
l_infoarray is all zeros except for the first NEEDED entry. =A0It appears correct. ==A0The
Quoted 2 lines
loop might have terminated early because of a cache issue, or possibly the value loaded from memory somehow got corrupted. =A0Another possibilit=
y
Quoted 8 lines
would be the mmap operation wasn't complete when the memory was examined by the dynamic loader. =A0When the core dump was done, the operation was complete.
I think it's less likely that a cache issue affected the memory used by the dynamic loader (
l_infofield) as the data before and after in the map seemed reasonable.Quoted 6 lines
The fact PA8700 processors are also experiencing similar problems would seem to suggest that this isn't a PA8800 L2 issue unless we have multiple problems.
I think we need to try running a recent kernel on gsyprf11 for a while to see if we can capture a similar event.
This rang a bell...
In
glibc/elf/rtld.cwe have this:/* Partly clean the `bootstrap_map' structure up. Don't use
`memset' since it might not be built in or inlined and we cannot make function calls at this point. Use '__builtin_memset' if we know it is available. We do not have to clear the memory if we do not have to use the temporarybootstrap_map. Global variables are initialized to zero by default. */
#ifndefDONT_USE_BOOTSTRAP_MAP# ifdef HAVE_BUILTIN_MEMSET __builtin_memset (bootstrap_map.l_info, '\0', sizeof (bootstrap_map.l_inf= o)); # else for (size_t cnt =3D 0; cnt < sizeof (bootstrap_map.l_info) / sizeof (bootstrap_map.l_info[0= ]); ++cnt) bootstrap_map.l_info[cnt] =3D 0; # endif
Quoted 7 lines
On hppa we don't have builtin memset (probably one of the few arches), so we fall back on this weird loop which I always thought was wrong.
I was seeing problems with
l_infohaving garbage in it, so I had a local hack which cleared the entirebootstrap_map.Did your
l_infocome from thebootstrap_map?
No. The l_info fields for the dynamic loader and libncurses.so.5 had already been processed. The segv occurred processing the needed entry for libdl.so.2. The code was processing the needed entries for /bin/sh.
The cause of the corruption that you observed is not obvious to me.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6602)