Exaggerated swap usage
Hi Andrea,
Quoted 9 lines
Quoted 8 lines
ok, now it's clear what the problem is. there are inuse-dirty inodes that triggers a deadlock in the schedule-capable
try_to_sync_unused_inodesof 2.4.20rc2aa1 (that avoided me to backout an otherwise corrupt lowlatency fix). It can trigger only in UP, in SMP the other cpu can always run kupdate that will flush all dirty inodes, so it would lockup one cpu as worse for 2.5 sec, this is probably why I couldn't reproduce it, I assume all of you reproducing the deadlock were running on an UP machine (doesn't matter if the kernelCorrect (for me anyways).
ok, deadlock is gone, instead I have EXT3-fs corruption (non data=journal mode, just ordered) like this:
Dec 3 00:25:39 codeman kernel: EXT3-fs error (device ide0(3,9)): ext3_free_blocks: Freeing blocks not in datazone - block = 1530182 , count = 1 Dec 3 00:25:39 codeman kernel: Aborting journal on device ide0(3,9). Dec 3 00:25:39 codeman kernel: ext3_free_blocks: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3 -fs error (device ide0(3,9)) in ext3_free_blocks: Journal has aborted Dec 3 00:25:39 codeman kernel: ext3_reserve_inode_write: aborting transaction: Journal has aborted in __ext3_journal_get_write_acce ss<2>EXT3-fs error (device ide0(3,9)) in ext3_reserve_inode_write: Journal has aborted Dec 3 00:25:39 codeman kernel: EXT3-fs error (device ide0(3,9)) in ext3_truncate: Journal has aborted Dec 3 00:25:39 codeman kernel: ext3_reserve_inode_write: aborting transaction: Journal has aborted in __ext3_journal_get_write_acce ss<2>EXT3-fs error (device ide0(3,9)) in ext3_reserve_inode_write: Journal has aborted Dec 3 00:25:39 codeman kernel: EXT3-fs error (device ide0(3,9)) in ext3_orphan_del: Journal has aborted Dec 3 00:25:39 codeman kernel: ext3_reserve_inode_write: aborting transaction: Journal has aborted in __ext3_journal_get_write_acce ss<2>EXT3-fs error (device ide0(3,9)) in ext3_reserve_inode_write: Journal has aborted Dec 3 00:25:39 codeman kernel: EXT3-fs error (device ide0(3,9)) in ext3_delete_inode: Journal has aborted Dec 3 00:25:39 codeman kernel: ext3_abort called. Dec 3 00:25:39 codeman kernel: EXT3-fs abort (device ide0(3,9)): ext3_journal_start: Detected aborted journal Dec 3 00:25:39 codeman kernel: Remounting filesystem read-only Dec 3 00:25:39 codeman kernel: EXT3-fs error (device ide0(3,9)) in
start_transaction: Journal has aborted
BTW: UP, non-SMP.
ciao, Marc
Quoted 1 line · Marc-Christian Petersen
ext3_free_blocks: Freeing blocks not in datazone - block = 1530182
this is the interesting one. Did you run any unstable kernel/driver software combination recently or maybe you got oopsed or crashes?
journaling sometime gives a false sense of reliability, you've to keep in mind that unless you know why you had to reboot w/o a clean unmount you should always force an e2fsck -f/reiserfsck in single user mode at the next boot, no matter of journaling. If the machine crashed because of a kernel oops or similar skipping the filesystemcheck at the very next boot could left the fs corrupted for a long time until you notice it possibly while running an unrelated kernel. So if you crashed recently and you didn't run any e2fsck -f that could explain it. I doubt there are corruption issues in my current tree (and even the UP deadlock that I fixed couldn't explain the corruption, in this case [the UP deadlock] I'm sure because I know what kind of side effect _this_ bug could generate, for any other crash you cannot tell if e2fsck -f is needed until/unless you know the details of the bug, and most of the time you don't know the details of the bug at the time of the next reboot so normally an e2fsck -f is always required after a kernel crash, this can't be automated simply because if the kernel is crashed we can't write to the superblock to notify e2fsck about it, so at the next boot e2fsck will always think replying the log was enough).
Of course your problem could be explained by a bad cable or whatever else hardware failure too. At the moment I doubt it's a problem in the common code of my tree or mainline.
Andrea
On Tuesday 03 December 2002 01:59, you wrote:
Hi Andrea,
Quoted 2 lines
this is the interesting one. Did you run any unstable kernel/driver software combination recently or maybe you got oopsed or crashes?
nope, no oops, no crash, afaik no unstable kernel/drivers. Kernel is yours ;) and drivers, hmm, just intel i815, eepro100. That happend after some hours of uptime and just doing "rm -rf linux-old"
Quoted 4 lines
journaling sometime gives a false sense of reliability, you've to keep in mind that unless you know why you had to reboot w/o a clean unmount you should always force an e2fsck -f/reiserfsck in single user mode at the next boot, no matter of journaling. If the machine crashed because
Yep, I always do a forced fsck in case of that.
Quoted 4 lines
of a kernel oops or similar skipping the filesystemcheck at the very next boot could left the fs corrupted for a long time until you notice it possibly while running an unrelated kernel. So if you crashed recently and you didn't run any e2fsck -f that could explain it. I doubt
I run e2fsck -fy every time after a crash. Fortunately it doesn't happen so often :-)
Quoted 6 lines
...
don't know the details of the bug at the time of the next reboot so normally an e2fsck -f is always required after a kernel crash, this can't be automated simply because if the kernel is crashed we can't write to the superblock to notify e2fsck about it, so at the next boot e2fsck will always think replying the log was enough).
yep. I tried to remove that 00_umount-against-unused-dirty-inodes-race fix and after that (now 5 hours uptime) doing only copying and deleting, that ext3fs error is away.
Quoted 3 lines
Of course your problem could be explained by a bad cable or whatever else hardware failure too. At the moment I doubt it's a problem in the common code of my tree or mainline.
seems it's a problem in the umount-against-unused-dirty-inodes-race fix or if the fix "is the right way" the problem is located somewhere else what triggers the problem of your patch.
ciao, Marc
Quoted 14 lines · Marc-Christian Petersen
On Tuesday 03 December 2002 01:59, you wrote:
Hi Andrea,
Quoted 2 lines
this is the interesting one. Did you run any unstable kernel/driver software combination recently or maybe you got oopsed or crashes?
nope, no oops, no crash, afaik no unstable kernel/drivers. Kernel is yours ;) and drivers, hmm, just intel i815, eepro100. That happend after some hours of uptime and just doing "rm -rf linux-old"
Quoted 4 lines
journaling sometime gives a false sense of reliability, you've to keep in mind that unless you know why you had to reboot w/o a clean unmount you should always force an e2fsck -f/reiserfsck in single user mode at the next boot, no matter of journaling. If the machine crashed because
Quoted 8 lines
Yep, I always do a forced fsck in case of that.
Quoted 4 lines
of a kernel oops or similar skipping the filesystemcheck at the very next boot could left the fs corrupted for a long time until you notice it possibly while running an unrelated kernel. So if you crashed recently and you didn't run any e2fsck -f that could explain it. I doubt
I run e2fsck -fy every time after a crash. Fortunately it doesn't happen so often :-)
ok ;) I asked just in case.
Quoted 11 lines
Quoted 7 lines
...
don't know the details of the bug at the time of the next reboot so normally an e2fsck -f is always required after a kernel crash, this can't be automated simply because if the kernel is crashed we can't write to the superblock to notify e2fsck about it, so at the next boot e2fsck will always think replying the log was enough).yep. I tried to remove that 00_umount-against-unused-dirty-inodes-race fix and after that (now 5 hours uptime) doing only copying and deleting, that ext3fs error is away.
Quoted 6 lines
Quoted 3 lines
Of course your problem could be explained by a bad cable or whatever else hardware failure too. At the moment I doubt it's a problem in the common code of my tree or mainline.
seems it's a problem in the umount-against-unused-dirty-inodes-race fix or if the fix "is the right way" the problem is located somewhere else what triggers the problem of your patch.
can you reproduce in 2.4.20aa1 too?
Andrea
On Tuesday 03 December 2002 14:59, Andrea Arcangeli wrote:
Hi Andrea,
Quoted 3 lines
Quoted 2 lines
I run e2fsck -fy every time after a crash. Fortunately it doesn't happen so often :-)
ok ;) I asked just in case.
:-) np.
Quoted 4 lines
Quoted 3 lines
seems it's a problem in the umount-against-unused-dirty-inodes-race fix or if the fix "is the right way" the problem is located somewhere else what triggers the problem of your patch.
can you reproduce in 2.4.20aa1 too?
I'll give it a try later this evening.
ciao, Marc
On Tuesday 03 December 2002 19:43, Marc-Christian Petersen wrote:
Hi Andrea,
Quoted 2 lines
Quoted 1 line
can you reproduce in 2.4.20aa1 too?
I'll give it a try later this evening.
nope, I cannot reproduce it. Seems there is another fix in 2.4.20aa1 that fixes it.
ciao, Marc