Htree ate my hard drive, was: post-halloween 0.2
Quoted 9 lines · Duncan Sands
Quoted 5 lines
I wonder if there is still a bug in the e2fsck code for re-hashing directories? It shouldn't be possible to have e2fsck complete and there still be an error in the filesystem (ok, sometimes it happens, but in those cases it spews a lot of warnings about the filesystem not being fixed yet and to run manually).
It is possible that the filesystem was fine when fsck completed, but was damaged afterwards, i.e. in the time between fsck completing and the reboot.
Just stupid idea. Two or three months ago I complained that if my box crashes shortly after boot, following things happen:
(1) system for some reason reads /var/run directory to page cache
(2) fsck finds that /var/run/* entries points to invalid nodes, and removes them (through block device access)
(4) / is remounted read-write
(5) because of page cache for block device and directory is not coherent (or what...), system still sees /var/run/* populated
(6) rm /var/run/* is run. FS is remounted read-only due to freeing inode already freeed...
(7) Reboot, run fsck again, reboot, fine...
Nobody answered it at that time, and it happened at least 5 times again to me - until I modified initscripts to do unconditional reboot if "fsck /" did ANY modifications to filesystem.
Maybe kernel still uses old directory indexes structure after fsck created new one?
Best regards,
Petr Vandrovec [email protected]
Quoted 24 lines · Petr Vandrovec
Quoted 9 lines · Duncan Sands
Quoted 5 lines
I wonder if there is still a bug in the e2fsck code for re-hashing directories? It shouldn't be possible to have e2fsck complete and there still be an error in the filesystem (ok, sometimes it happens, but in those cases it spews a lot of warnings about the filesystem not being fixed yet and to run manually).
It is possible that the filesystem was fine when fsck completed, but was damaged afterwards, i.e. in the time between fsck completing and the reboot.
Just stupid idea. Two or three months ago I complained that if my box crashes shortly after boot, following things happen:
(1) system for some reason reads
/var/rundirectory to page cache
(2) fsck finds that/var/run/*entries points to invalid nodes, and removes them (through block device access)
(4) / is remounted read-write
(5) because of page cache for block device and directory is not coherent (or what...), system still sees/var/run/*populated
(6) rm/var/run/*is run. FS is remounted read-only due to freeing inode already freeed...
(7) Reboot, run fsck again, reboot, fine...
Quoted 6 lines
Nobody answered it at that time, and it happened at least 5 times again to me - until I modified initscripts to do unconditional reboot if "fsck /" did ANY modifications to filesystem.
Maybe kernel still uses old directory indexes structure after fsck created new one?
File system needs to unmount and remount after e2fsck packed directory index. Kernel need to trash all the dentry cache of that file system. If you pack the "/". It'd better reboot.
Chris
Quoted 5 lines · Petr Vandrovec
Nobody answered it at that time, and it happened at least 5 times again to me - until I modified initscripts to do unconditional reboot if "fsck /" did ANY modifications to filesystem.
In fact, e2fsck should return an exit code which indicates that the systme should be rebooted if an fsck the root filesystem makes any changes to the filesystem. See the man page to fsck(8) for a definition of fsck's exit codes, but if (exit_status & 2) is non-zero, the init scripts **should** reboot.
Unfortunately, not all distributions get this right. However, your analysis is right. If fsck needs to make any modifications to the root filesystem, which is mounted read-only, it is possible for the corrupted filesystem elements to still be cached in memory, and then written back out to disk when the filesystem is remounted read/write.
This is one reason why I normally recommend that / be a small filesystem of approximately 128 megs, with separate partitions for /usr, and either using a separate partition for /var, or using a symlink from /usr/var to /var. (And doing something similar for /home and and /opt, as necessary.) It minimizes the chances that the root filesystem will get corrupted, and makes running fsck on the root filesystem take much less time (obviously, since the root filesystem becomes quite small.)
- Ted