XFS: corruption detected in 5.9.10, upgrading from 5.9.6: (partial) panic log
So I just tried to reboot my x86 server box from 5.9.6 to 5.9.10 and my system oopsed with an xfs fs corruption message when I kicked up Chromium on another machine which mounted $HOME from the server box (it panicked without logging anything, because the corruption was detected on the rootfs, and it is also the loghost). A subsequent reboot died instantly as soon as it tried to mount root, but the next one got all the way to starting Chromium before dying again the same way.
Rebooting back into 5.9.6 causes everything to work fine again, no reports of corruption and starting Chromium works.
This fs has rmapbt and reflinks enabled, on a filesystem originally created by xfsprogs 4.10.0, but I have never knowingly used them under the Chromium config dirs (or, actually, under that user's $HOME at all). I've used them extensively elsewhere on the fs though. The FS is sitting above a libata -> md-raid6 -> bcache stack. (It is barely possible that bcache is at fault, but bcache has seen no changes since 5.9.6 so I doubt it.)
The relevant bits of the log I could capture -- no console scrollback these days, of course :( and it was a panic anyway so the top is just lost -- is in a photo here:
<http://www.esperi.org.uk/~nix/temporary/xfs-crash.jpg>
The mkfs line used to create this fs was:
mkfs.xfs -m rmapbt=1,reflink=1 -d agcount=17,sunit=$((128*8)),swidth=$((384*8)) -l logdev=/dev/sde3,size=521728b -i sparse=1,maxpct=25 /dev/main/root
(/dev/sde3 is an SSD which also hosts the bcache and RAID journal, though this RAID device is not journalled, and is operating fine.)
I am not using a realtime device.
I have *not* yet run xfs_repair, but just rebooted back into the old kernel, since everything worked there: I'll run xfs_repair over the fs if you think it wise to do so, but right now I have a state which crashes on one kernel and works on another one, which seems useful to not try to fix in case you have some use for it.
Since everything is working fine in 5.9.6 and there were XFS changes after that, I'm hypothesising that this is probably a bug in the post-5.9.6 changes rather than anything xfs_repair should be trying to fix. But I really don't know :)
(I can't help but notice that all these post-5.9.6 XFS changes were sucked in by Sasha's magic regression-hunting stable-tree AI, which I thought wasn't meant to happen -- but I've not been watching closely, and if you changed your minds after the LWN article went in I won't have seen it.)
Quoted 1 line · Nick Alcock
So I just tried to reboot my x86 server box from 5.9.6 to 5.9.10 and my
Sorry about that, there was a bad patch in -rc4 that got sucked into 5.9.9 because it had a fixes tag. The revert is already upstream:
--D
Quoted 49 lines
system oopsed with an xfs fs corruption message when I kicked up Chromium on another machine which mounted $HOME from the server box (it panicked without logging anything, because the corruption was detected on the rootfs, and it is also the loghost). A subsequent reboot died instantly as soon as it tried to mount root, but the next one got all the way to starting Chromium before dying again the same way.
Rebooting back into 5.9.6 causes everything to work fine again, no reports of corruption and starting Chromium works.
This fs has rmapbt and reflinks enabled, on a filesystem originally created by xfsprogs 4.10.0, but I have never knowingly used them under the Chromium config dirs (or, actually, under that user's $HOME at all). I've used them extensively elsewhere on the fs though. The FS is sitting above a libata -> md-raid6 -> bcache stack. (It is barely possible that bcache is at fault, but bcache has seen no changes since 5.9.6 so I doubt it.)
The relevant bits of the log I could capture -- no console scrollback these days, of course :( and it was a panic anyway so the top is just lost -- is in a photo here:
<http://www.esperi.org.uk/~nix/temporary/xfs-crash.jpg>
The mkfs line used to create this fs was:
mkfs.xfs -m rmapbt=1,reflink=1 -d agcount=17,sunit=$((128*8)),swidth=$((384*8)) -l logdev=/dev/sde3,size=521728b -i sparse=1,maxpct=25
/dev/main/root(
/dev/sde3is an SSD which also hosts the bcache and RAID journal, though this RAID device is not journalled, and is operating fine.)I am not using a realtime device.
I have *not* yet run
xfs_repair, but just rebooted back into the old kernel, since everything worked there: I'll runxfs_repairover the fs if you think it wise to do so, but right now I have a state which crashes on one kernel and works on another one, which seems useful to not try to fix in case you have some use for it.Since everything is working fine in 5.9.6 and there were XFS changes after that, I'm hypothesising that this is probably a bug in the post-5.9.6 changes rather than anything
xfs_repairshould be trying to fix. But I really don't know :)(I can't help but notice that all these post-5.9.6 XFS changes were sucked in by Sasha's magic regression-hunting stable-tree AI, which I thought wasn't meant to happen -- but I've not been watching closely, and if you changed your minds after the LWN article went in I won't have seen it.)
On 22 Nov 2020, Darrick J. Wong stated:
Quoted 7 lines · Nick Alcock
Quoted 1 line
So I just tried to reboot my x86 server box from 5.9.6 to 5.9.10 and my
Sorry about that, there was a bad patch in -rc4 that got sucked into 5.9.9 because it had a fixes tag. The revert is already upstream:
Thanks! Will give it a try soon :)
(Solved in a couple of hours on a Sunday and all I had to do was mail off a photo. As I always say about free software, especially while feeling guilty about my own response times... you can't pay for service like this!)
... and no I don't know why I didn't think to check the master branch for obvious related reversions in fs/xfs. I'll do that next time before bothering other people.