2.6.18-stable release plans?
Quoted 4 lines
Quoted 2 lines
But MY kernel is clearly untainted.
So what other explanation is there apart from a kernel bug?If it's me you're asking: I don't know (overheating, cosmic rays, ...)
I suppose what I'm *really* asking is what the basis is for assuming that this *isn't* a kernel bug and can therefore be safely ignored, seeing as the oops is real, the hardware is fine and the kernel is untainted? That seems to cover the bases from where I'm sitting.
Cheers,
Chris
P.S. No micro-heatwaves have occurred here, either. Do we all need to install muon detectors in our homes before reporting bugs now, so that we can exclude cosmic ray events too?
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
Quoted 4 lines · Chris Rankin
I suppose what I'm *really* asking is what the basis is for assuming that this *isn't* a kernel bug and can therefore be safely ignored, seeing as the oops is real, the hardware is fine and the kernel is untainted? That seems to cover the bases from where I'm sitting.
All I'm claiming is that it's no more a reason to avoid 2.6.19* than to avoid any other release (the kernels before 2.6.7 happened to have no such check, but that doesn't imply they were any safer).
It may indeed be due to a kernel bug, but I can't tell you where: sorry.
Hugh
Quoted 3 lines · --- Hugh Dickins
All I'm claiming is that it's no more a reason to avoid 2.6.19* than to avoid any other release (the kernels before 2.6.7 happened to have no such check, but that doesn't imply they were any safer).
There is *one* reason to avoid 2.6.19.x: it has actually bitten me, while none of the others has. And if I can't trust a kernel to compile something then what good is it?
Cheers,
Chris
___________________________________________________________
New Yahoo! Mail is the ultimate force in competitive emailing. Find out more at the Yahoo! Mail Championships. Plus: play games and win prizes.
http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk
Quoted 9 lines · Chris Rankin
Quoted 4 lines
Quoted 2 lines
But MY kernel is clearly untainted.
So what other explanation is there apart from a kernel bug?If it's me you're asking: I don't know (overheating, cosmic rays, ...)
I suppose what I'm *really* asking is what the basis is for assuming that this *isn't* a kernel bug and can therefore be safely ignored, seeing as the oops is real, the hardware is fine and the kernel is untainted? That seems to cover the bases from where I'm sitting.
Quoted 5 lines
Cheers,
Chris
P.S. No micro-heatwaves have occurred here, either. Do we all need to install muon detectors in our homes before reporting bugs now, so that we can exclude cosmic ray events too?
Well, do you have ECC memory? If not, it is at least possible that that the solar flares that occurred last month may have affected your system. There were three X-class solar flares in the month of December. Even if you have ECC memory, it is still possible to suffer data corruption because many BIOSes do not turn on bus parity error detection. And are the memories in your disk drive controllers ECC or parity protected? I wouldn't bet on it...
All too often, modern PCs are data corruption accidents waiting to happen.
Signature · 1 line
Mark Rustad, [email protected]
Quoted 3 lines · --- Mark Rustad
Well, do you have ECC memory? If not, it is at least possible that that the solar flares that occurred last month may have affected your system.
I am going to assume that you are being facaetious, because it would be the rarified pinnacle of supreme arrogance to suggest that a cosmic ray event is a more likely explanation than a bug in the kernel.
Cheers,
Chris
___________________________________________________________
What kind of emailer are you? Find out today - get a free analysis of your email personality. Take the quiz at the Yahoo! Mail Championship.
http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk
Quoted 3 lines
I am going to assume that you are being facaetious, because it would be the rarified pinnacle of supreme arrogance to suggest that a cosmic ray event is a more likely explanation than a bug in the kernel.
A one off non repeatable error experienced by two people out of the millions using it does fit the cosmic ray description quite well. That's not to say there isn't a bug, but you don't have enough data to even begin debugging it unless its rather more reproducable.
Alan
Quoted 2 lines · --- Alan
A one off non repeatable error experienced by two people out of the millions using it does fit the cosmic ray description quite well.
Actually it's "unrepeated", not "non repeatable". And that's because I switched back to 2.6.18.x immediately since I no longer trusted 2.6.19.x.
Cheers,
Chris
___________________________________________________________
What kind of emailer are you? Find out today - get a free analysis of your email personality. Take the quiz at the Yahoo! Mail Championship.
http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk
Quoted 8 lines · Alan
Quoted 3 lines
I am going to assume that you are being facaetious, because it would be the rarified pinnacle of supreme arrogance to suggest that a cosmic ray event is a more likely explanation than a bug in the kernel.
A one off non repeatable error experienced by two people out of the millions using it does fit the cosmic ray description quite well. That's not to say there isn't a bug, but you don't have enough data to even begin debugging it unless its rather more reproducable.
Exactly. Halting use of a version of the kernel based on a single incident provides no insight to the source of the problem. It could be anything...
Signature · 1 line
Mark Rustad, [email protected]
Quoted 3 lines · --- Mark Rustad
Exactly. Halting use of a version of the kernel based on a single incident provides no insight to the source of the problem. It could be anything...
There is a world of difference between a polite request for more information (although I gave you everything I had), and fobbing someone off with a story about cosmic rays.
Cheers,
Chris
___________________________________________________________
Now you can scan emails quickly with a reading pane. Get the new Yahoo! Mail. http://uk.docs.yahoo.com/nowyoucan.html
Quoted 4 lines · Chris Rankin
There is a world of difference between a polite request for more information (although I gave you everything I had), and fobbing someone off with a story about cosmic rays.
Chris,
I doubt there was a single version of the kernel which ever worked
well for all its users. In a production environment, reverting to an older version may be the best short-term answer, but if nobody recognises the problem you won't get any closer to a proper fix. At the moment, you have a problem that nobody recognises. If you're not willing to test if the problem happens repeatably, (you appear to have had one failure and immediately reverted to an old kernel), who do you think will be able to fix it? And if it turns out it doesn't fail repeatably, maybe the responses you've received could be correct.
The stable team are only there to maintain the current release of
the kernel. There is no maintenance of earlier releases (except Adrian's work on 2.6.16), other than what a distro chooses to do to backport fixes. Of course, if the problem _is_ reproducible on your machine and config, you might get asked to try to identify when it got introduced (e.g. was 2.6.19 itself, or an arbitrary 2.6.19-rc, ok?).
Ken
Signature · 1 line
das eine Mal als Tragödie, das andere Mal als Farce
Quoted 4 lines · --- Ken Moffat
At the moment, you have a problem that nobody recognises. If you're not willing to test if the problem happens repeatably, (you appear to have had one failure and immediately reverted to an old kernel), who do you think will be able to fix it?
This bug seems to be in the kernel's "memory management", and the last memory-related bug I had (caused by a bad DIMM on another machine) caused creeping filesystem corruption. However, this machine is my main desktop, and so I am keen to keep the filesystems intact. So yes, that involves not running a kernel that has shown itself to be unreliable.
I was hoping that someone with a deeper knowledge of the differences between 2.6.18 and 2.6.19 would have an idea of what might have triggered this problem, and yes, I was also thinking that some more people would trip over it and help debug it.
But anyway - can someone please tell me what "Eeek! page_mapcount(page) went negative! (-1)" is *really* saying/implying? Because I am currently translating this as "I WANT TO EAT YOUR FILESYSTEMS".
Cheers,
Chris
___________________________________________________________
New Yahoo! Mail is the ultimate force in competitive emailing. Find out more at the Yahoo! Mail Championships. Plus: play games and win prizes.
http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk
Quoted 5 lines · Chris Rankin
But anyway - can someone please tell me what "Eeek!
page_mapcount(page)went negative! (-1)" is *really* saying/implying? Because I am currently translating this as "I WANT TO EAT YOUR FILESYSTEMS".
I can't, but Dave Jones had a similar problem earlier this month,
archived at http://uwsg.iu.edu/hypermail/linux/kernel/0701.0/1822.html
which I think is a followup from
http://www.mail-archive.com/[email protected]/msg105370.html
- and seems to be a possible hardware failure (bulging capacitors)
becoming apparent under load.
Ken
Signature · 1 line
das eine Mal als Tragödie, das andere Mal als Farce
Quoted 6 lines · --- Ken Moffat
I can't, but Dave Jones had a similar problem earlier this month,
archived at http://uwsg.iu.edu/hypermail/linux/kernel/0701.0/1822.html
which I think is a followup from
http://www.mail-archive.com/[email protected]/msg105370.html
- and seems to be a possible hardware failure (bulging capacitors)
becoming apparent under load.
Interesting, although I don't believe I have a hardware fault. The box is perfectly stable under 2.6.18.x. Anyway, my particular problem happened very quickly last time so I am hoping that recompiling xine from scratch will trigger something again. This time, however, I have built a 2.6.19.x kernel with a few memory debugging options turned on.
Cheers,
Chris
___________________________________________________________
Copy addresses and emails from any email account to Yahoo! Mail - quick, easy and free. http://uk.docs.yahoo.com/trueswitch2.html
Quoted 3 lines · Chris Rankin
But anyway - can someone please tell me what "Eeek!
page_mapcount(page)went negative! (-1)" is *really* saying/implying? Because I am currently translating this as "I WANT TO EAT YOUR FILESYSTEMS".
Hugh already did, multiple times. If there's an external hardware event that corrupts memory, code executing on your CPU is no longer going to behave deterministically. So cases that are typically "impossible" in the design of the code have a chance to trigger.
You can continue to flame 2.6.19, but you're an extreme minority when it comes to this kind of bug and as, again, Hugh already said, almost all of the reports of this and similar other bugs have led to hardware problems that were either unchecked or difficult to detect.
Imagine this scenario. It might seem unrealistic to you, but it's not impossible!
First Use of Linux -> Upgrading to 2.6.19
Undetected hardware error never triggered.
Running 2.6.19
Hardware error triggers. Linux crashes.
Going back to 2.6.18
Hardware error has not yet triggered again.
Will it eat your filesystem? Maybe. But it probably won't, if you claim the memory is tested, it could have been a single bit error, or a cosmic ray event, or a brownout, or anything similar. It's much more likely to simply crash your machine, as it did.
Not running the affected kernel again is a sure way to have _nobody_ listen to your complaints about 2.6.19 having a real software bug, because you're totally unwilling to test the kernel again and see if it triggers. A single report is simply not enough evidence.
Additionally, reports from other users (who may have a million different experimental variables involved) are also insufficient, for reasons which have already been explained (drivers, proprietary code, et cetera).
Signature · 5 lines
Cheers, Alistair. Final year Computer Science undergraduate. 1F2 55 South Clerk Street, Edinburgh, UK.
Quoted 7 lines · Chris Rankin
Quoted 4 lines
--- Mark Rustad <[email protected]> wrote:
Exactly. Halting use of a version of the kernel based on a single incident provides no insight to the source of the problem. It could be anything...There is a world of difference between a polite request for more information (although I gave you everything I had), and fobbing someone off with a story about cosmic rays.
I'm sorry. I didn't mean to imply anything like that. I just happened to notice that the date of the bug report appeared to correlate pretty well with one of the solar flare events last month. I was really trying to share some information that just conceivably might have been related, based on the earlier messages in this thread regarding memory errors.
I don't normally follow solar activity. I have been looking into some system failures that happened last month. The systems had been running with all bus error detection enabled – the hardware set to spontaneously reboot on any uncorrectable error. Since our systems are redundant, performing a reset simply means that the redundant partner will take over, so the reset is the best way to be certain that there is no data corruption. I eventually recalled a radio report last month about a coronal mass ejection on the sun and how things might be disrupted here. I checked out www.spaceweather.com and found that December was a very active month, with three separate X-class flares. I have no way to conclude that the failures that I have seen were influenced by events on the sun, but it seems possible. Compared to our systems, most PCs and even much server-class hardware systems are likely to corrupt a bit just keep on going.
We'll never know if any of these things were correlated with the solar flares because they all seem to be one-off failures. I do find it interesting though. Our systems seem to be doing statistically better this month. What do you think?
Signature · 1 line
Mark Rustad, [email protected]
Quoted 8 lines · Alan
Quoted 3 lines
I am going to assume that you are being facaetious, because it would be the rarified pinnacle of supreme arrogance to suggest that a cosmic ray event is a more likely explanation than a bug in the kernel.
A one off non repeatable error experienced by two people out of the millions using it does fit the cosmic ray description quite well. That's not to say there isn't a bug, but you don't have enough data to even begin debugging it unless its rather more reproducable.
The soft error rate (cosmic rays, alpha decay, etc.) for modern memory at sea level is estimated to be somewhere around 1000 - 5000 FIT/Mbit[1]. FIT is Failures in Time - errors per billion hours of use. If you've got 1GB of memory, you've got 8000Mbits. So you'd expect 8M - 40M errors per billion hours on your machine. Or 8 to 40 errors per 1000 hours. That's about one single-bit error per week to one per day.
Yes, that's a lot. Can it really be that high? Big supercomputer installations actually measure it in errors per day or hour.
Most of these errors will go completely unnoticed because they happen in data structures that aren't revisited (stale cache, unused code, empty memory). The remainder will often look like random disk read or write errors or random application bugs/crashes. Sound familiar? That's why people buy ECC memory.
Now if we say that 10% of of that 1GB of RAM (~100MB) is kernel code/data (not including page cache) and that, say, 1-10% of errors trigger BUG/WARN code, we'll see these bug messages once every 100 days to once every 1000 weeks (per GB per user).
As for the relative error rate vs kernel bugs - there are no shortage of Linux boxes with trouble-free uptimes much longer than the 100 days above.
So yes, if a user reports a bug that's attributable to a single bit memory error that's otherwise unreproduced and unexplained, it's totally reasonable to chalk it up to cosmic rays until some sort of pattern of reports emerges.
As for your particular bug:
Eeek! page_mapcount(page) went negative! (-1)
page->flags = 14
page->count = 0
page->mapping = 00000000
This check occurs whenever the last mapping is removed from a page. It's a very heavily used piece of code. The check is there as sanity-checking from when this logic was introduced. If there were a new bug here that could be triggered by gcc or telnet, odds are very good that it would trigger for TONS of people.
So more likely theories are: a) pointer scribble from something completely unrelated or b) cosmic rays. As the nearby data (flags, count, mapping) doesn't appear to be scribbled on, (a) looks less promising.
[1] http://www.tezzaron.com/about/papers/soft_errors_1_1_secure.pdf
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 8 lines · Chris Rankin
Quoted 3 lines · --- Mark Rustad
Well, do you have ECC memory? If not, it is at least possible that that the solar flares that occurred last month may have affected your system.
I am going to assume that you are being facaetious, because it would be the rarified pinnacle of supreme arrogance to suggest that a cosmic ray event is a more likely explanation than a bug in the kernel.
Sorry for the late reply, but cosmic ray events (actually, self-induced alpha particle events from decays within the chipset itself) *are* a likely explanation:
Most important take-away here:
"With 100 million computers in use today, we should expect roughly 6 million single bit errors per year. Computer hardware and software companies must receive thousands of "side effect" bug reports and support calls due to memory errors alone. The costs of NOT including parity memory must be huge!"
Quoted 4 lines · [email protected] wrote:
"With 100 million computers in use today, we should expect roughly 6 million single bit errors per year. Computer hardware and software companies must receive thousands of "side effect" bug reports and support calls due to memory errors alone. The costs of NOT including parity memory must be huge!"
I must be weird or something, but I often think about this and the sheer number of clock cycles executing at any one time around the world. Have you ever stopped to think how many copies of schedule() (or whatever) are currently running somewhere in the world? It's just nuts :-)
More seriously, if nobody cared about this stuff then we wouldn't have all this MCE reporting and tools to handle differentiating between actual failing DRAMs and temporary bit transitions in ECC memory.
Jon.
Quoted 4 lines · Jon Masters
I must be weird or something, but I often think about this and the sheer number of clock cycles executing at any one time around the world. Have you ever stopped to think how many copies of
schedule()(or whatever) are currently running somewhere in the world? It's just nuts :-)
That's why we count single cycles in fast-path sections of code. :)