2.6.25 released with bug, which leads to XFS crash?
Hi again
I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs), just it is happening even on heavy load after 1-2 days. IMHO such bugs is critical (same as getting kernel panic, and etc),
cause they are unrecoverable, causing minor filesystem corruption, and only way to fix them - wakeup sysadmin. Worst thing, it is hapenning at night, when i restart squid, and probably it is doing agressive unlinking stale cache entries. It doesn't do panic, or even oops, but filesystem will be disconnected, and squid will remain in loop trying to restart. Sure it is easy to restart it, but maybe it has to be OOPS? so at least i can do sysctl -w kernel.panic_on_oops = 1, and FS will be recovered on reboot.
Just want to warn people who is using XFS on loaded servers to keep attention while using 2.6.25, and if you face same bug, report to bugzilla.
Signature · 4 lines
------ Technical Manager Virtual ISP S.A.L. Lebanon
Quoted 3 lines · Denys Fedoryshchenko
Just want to warn people who is using XFS on loaded servers to keep attention while using 2.6.25, and if you face same bug, report to bugzilla.
Thanks for the heads-up!
Signature · 1 line
left blank, right bald
Quoted 6 lines · Denys Fedoryshchenko
Hi again
I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs), just it is happening even on heavy load after 1-2 days. IMHO such bugs is critical (same as getting kernel panic, and etc),
Well, yes, and we treat shutdown bugs as such. A filesystem shutdown is effectively a filesystem panic and is indicative of either a corruption or a bug. The reality is that it takes time to triage such a problem that only occurs on one workload on one set of identical machines once every day or two. This does not make the problem a release blocker, though.
The other side of it is that problems like this in Linux are often the result of a bug in a lower layer and not XFS itself. Given this particular problem seems to be memory corruption it could be anything that is causing it....
Quoted 7 lines
cause they are unrecoverable, causing minor filesystem corruption, and only way to fix them - wakeup sysadmin. Worst thing, it is hapenning at night, when i restart squid, and probably it is doing agressive unlinking stale cache entries. It doesn't do panic, or even oops, but filesystem will be disconnected, > and squid will remain in loop trying to restart. Sure it is easy to restart it, but maybe it has to be OOPS? so at least i can do sysctl -w kernel.panic_on_oops = 1, and FS will be recovered on reboot.
Rather than fearmongering, perhaps you should ask on the XFS list ([email protected]) whether anything like this can be done. Then you might have learnt about Documentation/filesystems/xfs.txt and /proc/sys/fs/xfs/panic_mask:
fs.xfs.panic_mask (Min: 0 Default: 0 Max: 127)
Causes certain error conditions to call BUG(). Value is a bitmask; AND together the tags which represent errors which should cause panics:
Quoted 2 lines
Just want to warn people who is using XFS on loaded servers to keep attention while using 2.6.25, and if you face same bug, report to bugzilla.
Actually, I'd much prefer XFS bug reports to go to [email protected] rather than the kernel bugzilla - that way most of the XFS community will see the bug report and the triage being done and then there's no need for spamming lkml like this....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 3 lines · Denys Fedoryshchenko
Hi again
I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs),
Oh, aufs was not listed in the bug report. that's an out of tree module, right? Is it completely up to date and safe with 2.6.25?
-Eric
Quoted 7 lines · Eric Sandeen
Quoted 3 lines · Denys Fedoryshchenko
Hi again
I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs),
Oh, aufs was not listed in the bug report. that's an out of tree module, right? Is it completely up to date and safe with 2.6.25?
And it's a completely crappy one. I personally would through away every bug report with that piece of junk involved.
AUFS(same as COSS, UFS, NULL) is way of handling with cache storage in squid, and nothing else. It is userspace and not involved deeply with kernel things. Since long long time i didn't had any single problem with it.
Kernel is fully vanilla.
Quoted 13 lines · Christoph Hellwig
Quoted 7 lines · Eric Sandeen
Quoted 3 lines · Denys Fedoryshchenko
Hi again
I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs),
Oh, aufs was not listed in the bug report. that's an out of tree module, right? Is it completely up to date and safe with 2.6.25?
And it's a completely crappy one. I personally would through away every bug report with that piece of junk involved.
Signature · 4 lines
------ Technical Manager Virtual ISP S.A.L. Lebanon
Quoted 3 lines · Denys Fedoryshchenko
AUFS(same as COSS, UFS, NULL)is way of handling with cache storage in squid, and nothing else. It is userspace and not involved deeply with kernel things. Since long long time i didn't had any single problem with it.
Kernel is fully vanilla.
Ah, okay. There's also a kernel filesystem driver doing unioning that's also called aufs that I'm referring to. sorry for the noise.
Quoted 3 lines · Denys Fedoryshchenko
AUFS(same as COSS, UFS, NULL)is way of handling with cache storage in squid, and nothing else. It is userspace and not involved deeply with kernel things. Since long long time i didn't had any single problem with it.
Kernel is fully vanilla.
ah, ok. I had assumed kernel module too, sorry.
-Eric