3.12.8 is doing nasty things
I reported that I saw strange things happening after upgrading to 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I regularly see processes to fail randomly. I can see this as I somethimes get failures in my nagios because the SSH connection to the watched host (only the watched one runs .8) fails. I then see stuff like this in dmesg:
[723945.604000] do_page_fault() pid=30516 command='sshd' type=15 address=0x00000000 [723945.604000] CPU: 1 PID: 30516 Comm: sshd Not tainted 3.12.8 #1 [723945.604000] task: 000000023f0eb7c8 ti: 00000002397ac000 task.ti:
00000002397ac000
[723945.604000] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [723945.604000] PSW: 00000000000001001111111100001111 Not tainted [723945.604000] r00-03 000000ff0004ff0f 0000000040c3dc64 0000000040b63027 0000000000000000 [723945.604000] r04-07 0000000040c3d464 0000000000000000 000000000008d11c 0000000000089ca0 [723945.604000] r08-11 000000000008d11c 000000000008991c 000000000008811c 000000000008811c [723945.604000] r12-15 000000000008811c 0000000000000000 000000000008811c 00000000000888a8 [723945.604000] r16-19 000000000006eb1c 000000000006eb3c 000000000006e440 0000000040c3d464 [723945.604000] r20-23 00000000fb014990 000000004145eb40 0000000000000000 0000000000000095 [723945.604000] r24-27 00000000fffffff5 0000000040002890 00000000fb014988 000000000008711c [723945.604000] r28-31 fffffffffffffffe 0000000000000000 00000000fb014a00 0000000040b644c3 [723945.604000] sr00-03 0000000006807000 0000000000000000 0000000000000000 0000000006807000 [723945.604000] sr04-07 0000000006807000 0000000006807000 0000000006807000
0000000006807000
[723945.604000] VZOUICununcqcqcqcqcqcrmunTDVZOUI [723945.604000] FPSR: 00000000000000000000000000000000 [723945.604000] FPER1: 00000000 [723945.604000] fr00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [723945.604000] fr04-07 41d4b5f44002d0e5 0000000000000000 0000000000000000 0000000000000000 [723945.604000] fr08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [723945.604000] fr12-15 41d4b5f44002d0e5 0000000000000000 0000000000000000 0000000000000000 [723945.604000] fr16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [723945.604000] fr20-23 0000000000000000 0000000000000000 0000000000469971 0000000600000000 [723945.604000] fr24-27 0000000034a94675 0000000000000000 0000000000000000 0000000000000000 [723945.604000] fr28-31 0000000000000000 0000000000000000 0000000000000000
0000000000000000
[723945.604000] IASQ: 0000000006807000 0000000006807000 IAOQ: 0000000040b63033 0000000040b63037 [723945.604000] IIR: 0ca0109c ISR: 0000000006807000 IOR: 0000000000000000 [723945.604000] CPU: 1 CR30: 00000002397ac000 CR31: ffffffffffffffff [723945.604000] ORIG_R28: 0000000000000000 [723945.604000] IAOQ[0]: 0000000040b63033 [723945.604000] IAOQ[1]: 0000000040b63037 [723945.604000] RP(r2): 0000000040b63027
When I upgraded OpenSSL I also saw an ICE during build, which went away on the next try.
I also see a lot of those:
[724507.480000] INEQUIVALENT ALIASES 0x42ed0000 and 0x42ed1000 in file
libwrap.so.0.7.6
That could be the binutils bug that was mentioned, this file is from 2010. I'll rebuild that now to get rid of those warnings.
Next week I'll try 3.13, but since it has the same changes I expect the same results.
Eike
Quoted 5 lines · On Fri, 24 Jan 2014 21:44:51 +0100 Rolf Eike Beer <[email protected]> wrote:
I reported that I saw strange things happening after upgrading to 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I regularly see processes to fail randomly. I can see this as I somethimes get failures in my nagios because the SSH connection to the watched host (only the watched one runs .8) fails.
Confirmed with any kernel above 3.12.2 that I cared to test (I didn't test 3.12.3). The issues vary from illegal instructions to segmentation faults to double frees in userland, so it's basically a "random" thing. I can't say it's become "worse" after 3.12.5 (since I haven't kept per version statistics).
Regards,
jer
Quoted 9 lines · On Fri, 24 Jan 2014 22:07:23 +0100 Jeroen Roovers <[email protected]> wrote:
Quoted 5 lines · On Fri, 24 Jan 2014 21:44:51 +0100 Rolf Eike Beer <[email protected]> wrote:
I reported that I saw strange things happening after upgrading to 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I regularly see processes to fail randomly. I can see this as I somethimes get failures in my nagios because the SSH connection to the watched host (only the watched one runs .8) fails.
Quoted 5 lines
Confirmed with any kernel above 3.12.2 that I cared to test (I didn't test 3.12.3). The issues vary from illegal instructions to segmentation faults to double frees in userland, so it's basically a "random" thing. I can't say it's become "worse" after 3.12.5 (since I haven't kept per version statistics).
Same here, 3.12.5 was not behaving correctly and 3.12.8 was even worse. With the later, I also experienced sshd segfault, python threading segfault/hangs.
I quickly reverted to 3.12.4 which is far more stable.
Guy
Quoted 19 lines · Guy Martin
Quoted 9 lines · On Fri, 24 Jan 2014 22:07:23 +0100 Jeroen Roovers <[email protected]> wrote:
Quoted 5 lines · On Fri, 24 Jan 2014 21:44:51 +0100 Rolf Eike Beer <[email protected]> wrote:
I reported that I saw strange things happening after upgrading to 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I regularly see processes to fail randomly. I can see this as I somethimes get failures in my nagios because the SSH connection to the watched host (only the watched one runs .8) fails.
Quoted 5 lines
Confirmed with any kernel above 3.12.2 that I cared to test (I didn't test 3.12.3). The issues vary from illegal instructions to segmentation faults to double frees in userland, so it's basically a "random" thing. I can't say it's become "worse" after 3.12.5 (since I haven't kept per version statistics).
Quoted 4 lines
Same here, 3.12.5 was not behaving correctly and 3.12.8 was even worse. With the later, I also experienced sshd segfault, python threading segfault/hangs.
I quickly reverted to 3.12.4 which is far more stable.
This all sounds like a flushing problem. I can't confirm with my pa8800 ... but that could be because the additional pa8800 flushes are masking the problem. Could you try bisecting the parisc commits between .4 and .5 and see if there's a particular cause?
Thanks,
James
Am Samstag, 25. Januar 2014, 14:30:39 schrieben Sie:
Quoted 24 lines · Guy Martin
Quoted 18 lines
On Fri, 24 Jan 2014 22:07:23 +0100
Quoted 9 lines · Jeroen Roovers
On Fri, 24 Jan 2014 21:44:51 +0100
Quoted 5 lines · Rolf Eike Beer
I reported that I saw strange things happening after upgrading to 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I regularly see processes to fail randomly. I can see this as I somethimes get failures in my nagios because the SSH connection to the watched host (only the watched one runs .8) fails.
Quoted 5 lines
Confirmed with any kernel above 3.12.2 that I cared to test (I didn't test 3.12.3). The issues vary from illegal instructions to segmentation faults to double frees in userland, so it's basically a "random" thing. I can't say it's become "worse" after 3.12.5 (since I haven't kept per version statistics).
Quoted 4 lines
Same here, 3.12.5 was not behaving correctly and 3.12.8 was even worse. With the later, I also experienced sshd segfault, python threading segfault/hangs.
I quickly reverted to 3.12.4 which is far more stable.
Quoted 4 lines
This all sounds like a flushing problem. I can't confirm with my pa8800 ... but that could be because the additional pa8800 flushes are masking the problem. Could you try bisecting the parisc commits between .4 and .5 and see if there's a particular cause?
It's the one mmap fix, I see .5 being stable after reverting that fix.
Eike
Quoted 2 lines
Next week I'll try 3.13, but since it has the same changes I expect the same results.
I'm now on 3.13.2 and things are similar. While I did an initial test with 100 invocations of ssh which went flawlessly I now see random process errors again. Examples can be found here:
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
Normally the reports should have arrived much earlier, but for an unknown reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP process has failed during while renewing the IP address. The segfaults during the build have not happened with the older kernel.
Greetings,
Eike
Am Dienstag, 11. Februar 2014, 16:30:08 schrieb Rolf Eike Beer:
Quoted 8 lines
Quoted 2 lines
Next week I'll try 3.13, but since it has the same changes I expect the same results.
I'm now on 3.13.2 and things are similar. While I did an initial test with 100 invocations of ssh which went flawlessly I now see random process errors again. Examples can be found here:
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
Quoted 4 lines
Normally the reports should have arrived much earlier, but for an unknown reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP process has failed during while renewing the IP address. The segfaults during the build have not happened with the older kernel.
I can now confirm that the cache coherency fixes are _not_ the reason for those failures I see. I have rebuild 3.13.2 without them and still see random segfaults in my dashboards. I'll try the mmap() patch next.
Eike
Am Donnerstag, 13. Februar 2014, 07:37:55 schrieb Rolf Eike Beer:
Quoted 9 lines
Am Dienstag, 11. Februar 2014, 16:30:08 schrieb Rolf Eike Beer:
Quoted 8 lines
Quoted 2 lines
Next week I'll try 3.13, but since it has the same changes I expect the same results.
I'm now on 3.13.2 and things are similar. While I did an initial test with 100 invocations of ssh which went flawlessly I now see random process errors again. Examples can be found here:
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
Quoted 8 lines
Quoted 4 lines
Normally the reports should have arrived much earlier, but for an unknown reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP process has failed during while renewing the IP address. The segfaults during the build have not happened with the older kernel.
I can now confirm that the cache coherency fixes are _not_ the reason for those failures I see. I have rebuild 3.13.2 without them and still see random segfaults in my dashboards. I'll try the
mmap()patch next.
After 2,5 days with 3.13.2 and 0576da2c08e3d332f1b0653030d28ab804585ab6 I have not yet seen a single random segfault.
Eike
Am Samstag, 15. Februar 2014, 08:47:06 schrieb Rolf Eike Beer:
Quoted 18 lines
Am Donnerstag, 13. Februar 2014, 07:37:55 schrieb Rolf Eike Beer:
Quoted 17 lines
Am Dienstag, 11. Februar 2014, 16:30:08 schrieb Rolf Eike Beer:
Quoted 15 lines
Quoted 3 lines
Next week I'll try 3.13, but since it has the same changes I expect the
same results.I'm now on 3.13.2 and things are similar. While I did an initial test with
100 invocations of ssh which went flawlessly I now see random process errors again. Examples can be found here:
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11Normally the reports should have arrived much earlier, but for an unknown
reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP process has failed during while renewing the IP address. The segfaults during the build have not happened with the older kernel.
Quoted 6 lines
Quoted 3 lines
I can now confirm that the cache coherency fixes are _not_ the reason for those failures I see. I have rebuild 3.13.2 without them and still see random segfaults in my dashboards. I'll try the
mmap()patch next.After 2,5 days with 3.13.2 and
0576da2c08e3d332f1b0653030d28ab804585ab6I have not yet seen a single random segfault.
A short update on this: I am still running on that kernel and have not seen a single random process crash since then. I'm willing to help in debugging this if anyone tells me how, but I will now just revert that patch from every new kernel to get a stable machine.
Greetings,
Eike
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 11 lines
I'm now on 3.13.2 and things are similar. While I did an initial test with
100 invocations of ssh which went flawlessly I now see random process errors again. Examples can be found here:
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11Normally the reports should have arrived much earlier, but for an unknown
reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP process has failed during while renewing the IP address. The segfaults during the build have not happened with the older kernel.I can now confirm that the cache coherency fixes are _not_ the reason for those failures I see. I have rebuild 3.13.2 without them and still see random segfaults in my dashboards. I'll try the
mmap()patch next.
Quoted 7 lines
Quoted 2 lines
After 2,5 days with 3.13.2 and
0576da2c08e3d332f1b0653030d28ab804585ab6I have not yet seen a single random segfault.A short update on this: I am still running on that kernel and have not seen a single random process crash since then. I'm willing to help in debugging this if anyone tells me how, but I will now just revert that patch from every new kernel to get a stable machine.
On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6 which Dave uploaded, I'm now seeing similiar issues which are most likely related:
lsXX:~# dpkg-reconfigure locales Generating locales (this might take a while)... de_DE.ISO-8859-1...cannot map archive header: Invalid argument done de_DE.UTF-8...cannot map archive header: Invalid argument done [email protected] map archive header: Invalid argument done en_US.ISO-8859-1...cannot map archive header: Invalid argument done en_US.ISO-8859-15...cannot map archive header: Invalid argument done en_US.UTF-8...cannot map archive header: Invalid argument done Generation complete.
Helge
Quoted 20 lines · Helge Deller
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
I'm now on 3.13.2 and things are similar. While I did an initial test with
100 invocations of ssh which went flawlessly I now see random process errors again. Examples can be found here:
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11Normally the reports should have arrived much earlier, but for an unknown
reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP process has failed during while renewing the IP address. The segfaults during the build have not happened with the older kernel.I can now confirm that the cache coherency fixes are _not_ the reason for those failures I see. I have rebuild 3.13.2 without them and still see random segfaults in my dashboards. I'll try the
mmap()patch next.Quoted 6 lines
Quoted 2 lines
After 2,5 days with 3.13.2 and
0576da2c08e3d332f1b0653030d28ab804585ab6I have not yet seen a single random segfault.A short update on this: I am still running on that kernel and have not seen a single random process crash since then. I'm willing to help in debugging this if anyone tells me how, but I will now just revert that patch from every new kernel to get a stable machine.
Quoted 6 lines
On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6 which Dave uploaded, I'm now seeing similiar issues which are most likely related:
lsXX:~# dpkg-reconfigure locales Generating locales (this might take a while)... de_DE.ISO-8859-1...cannot map archive header: Invalid argument
In my opinion, this is a glibc bug. The issue has been discussed before.
I'm not seeing random segvs with 3.13.2 and later in general use. I haven't reverted anything.
The only random program failures that I see are thread related. There are a bunch of failures in the glibc testsuite, but these are not new. There was a message on the console this morning about reaching maximum lock depth 1024. This was probably from glibc testsuite. There is a known issue with locking in glib2.0 which sometimes causes dot to segv.
If you are testing programs that use threads intensively, then I could see that you might see problems.
A TLS bug affecting thread local variables was recently fixed in gcc-4.8.
Dave
Signature · 1 line
John David Anglin [email protected]
Am Mittwoch 26 Februar 2014, 11:00:01 schrieben Sie:
Quoted 32 lines · Helge Deller
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
After 2,5 days with 3.13.2 and
0576da2c08e3d332f1b0653030d28ab804585ab6I have not yet seen a single random segfault.A short update on this: I am still running on that kernel and have not seen a single random process crash since then. I'm willing to help in debugging this if anyone tells me how, but I will now just revert that patch from every new kernel to get a stable machine.
Quoted 8 lines
On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6 which Dave uploaded, I'm now seeing similiar issues which are most likely related:
lsXX:~# dpkg-reconfigure locales Generating locales (this might take a while)...de_DE.ISO-8859-1...cannot map archive header: Invalid argument
In my opinion, this is a glibc bug. The issue has been discussed before.
I'm not seeing random segvs with 3.13.2 and later in general use. I haven't reverted anything.
The only random program failures that I see are thread related. There are a bunch of failures
in the glibc testsuite, but these are not new. There was a message on the console this morning about
reaching maximum lock depth 1024. This was probably from glibc testsuite. There is a known
issue with locking in glib2.0 which sometimes causes dot to segv.
Quoted 4 lines
If you are testing programs that use threads intensively, then I could see that you might see problems.
A TLS bug affecting thread local variables was recently fixed in gcc-4.8.
I don't think this is the problem that I'm seeing. What crashed was things like make or ld which don't care for threads.
Eike
Quoted 8 lines · On Wed, 26 Feb 2014 15:26:15 +0100 "Helge Deller" <[email protected]> wrote:
On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6 which Dave uploaded, I'm now seeing similiar issues which are most likely related:
lsXX:~# dpkg-reconfigure locales Generating locales (this might take a while)... de_DE.ISO-8859-1...cannot map archive header: Invalid argument done
That isn't the same problem with the random segmentation faults that this thread documents.
jer
Quoted 7 lines · Jeroen Roovers
Quoted 4 lines · On Wed, 26 Feb 2014 15:26:15 +0100 "Helge Deller" <[email protected]> wrote:
On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6 which Dave uploaded, I'm now seeing similiar issues which are most likely related:
Quoted 6 lines
Quoted 4 lines
lsXX:~# dpkg-reconfigure locales Generating locales (this might take a while)... de_DE.ISO-8859-1...cannot map archive header: Invalid argument doneThat isn't the same problem with the random segmentation faults that this thread documents.
Well the matter is confusing. I may be wrong but Eike seems to be complaining that 0576da2c08e3d332f1b0653030d28ab804585ab6 is related to the random segmentation
fault. This is the change to the mmap handling that causes the invalid argument error above.
Can this be clarified?
Dave
Signature · 1 line
John David Anglin [email protected]
Quoted 6 lines · Helge Deller
On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6 which Dave uploaded, I'm now seeing similiar issues which are most likely related:
lsXX:~# dpkg-reconfigure locales Generating locales (this might take a while)... de_DE.ISO-8859-1...cannot map archive header: Invalid argument
This is caused by this: http://marc.info/?l=linux-parisc&m=138748066325722&w=2
A.