backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused
A message that is not in the archive (sent off-list or not yet received): 1353851735.22969.18.camel@soupermouf
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The ext4_inode_cache slab is taking almost all my memory (117MB). Please advise!
Thanks,
Dimitris
kmemleak failure (was Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused)
Hello,
I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost within 5 minutes of starting the backup process, I got:
[ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure [ 486.887195] kmemleak: Automatic memory scanning thread ended [ 486.906604] kmemleak: Kernel memory leak detector disabled
Any ideas?
Thanks,
Dimitris
Re: kmemleak failure (was Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused)
Quoted 8 lines · Dimitrios Apostolou
Hello,
I tried running a kmemleak and
DEBUG_SLUBenabled kernel, but almost within 5 minutes of starting the backup process, I got:[ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure [ 486.887195] kmemleak: Automatic memory scanning thread ended [ 486.906604] kmemleak: Kernel memory leak detector disabled
Important detail: I decided to try kmemleak after suspecting a leak because 'echo 3>/proc/sys/vm/drop_caches' left about 90MB out of 118MB of ext4_inode_cache, and I found no way to get rid of that (via remounting filesystems for example).
Quoted 8 lines · On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote:
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
Quoted 6 lines
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The
ext4_inode_cacheslab is taking almost all my memory (117MB). Please advise!
Hello Dimitrios,
I would try a 2.6.27.* kernel, for following reasons:
(1) Kernel development since 2.6.27 achieved significant performance improvements at the cost of exploding memory consumption by the kernel for _internal_ data structures. I am currently using a 3.2.34 kernel on a Notebook with 4 G RAM. 0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this memory cannot be swapped, it cannot be released by other means than rebooting, and there seems to be _no_ adjustment to memory pressure. I am surprised, that you have managed to boot a 3.6.* kernel at all with only 128 M RAM.
(2) At the time, when I used a PIII-Notebook, kernels 2.6.27 to 2.6.29 where current. Thus chances are good, that a 2.6.27.* kernel will support chipset, PCI bus and devices of your notebook. 2.6.27 got longterm maintainance, the latest release in the linux-stable git repository is 2.6.27.62. So Q @ LKML community:
Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
[1] Picture described in my LKML message
Date: Fri, 20 Jan 2012 01:08:00 +0100 Subject: Re: [kmemleak report 1/2] kernel 3.1.6, x86_64: mm, xfs ?, vfs ?
remained the same with _every_ 3.1.* and 3.2.* kernel tried so far.
Signature · 1 line
Roland
Quoted 1 line
Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
Probably not a good idea, there are known exploitable holes in 2.6.27 era kernels and nobody maintains anything that old.
I guess RHEL/Centos might work for you.
I've not had any problems with leaks in 3.6 but there have been a few reports and its clear that some obscure configurations trigger something bad.
Gnome 3 on the other hand leaks like a sieve.
Quoted 4 lines · On 2012-11-25 Sunday at 23:56 +0000 Alan Cox wrote:
Quoted 1 line
Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
Probably not a good idea, there are known exploitable holes in 2.6.27 era kernels and nobody maintains anything that old.
“old” is relative …
cd git/linux-stable && git log -1 --date=iso v2.6.27.62
........................................................
commit bc4e1a77b06519a01e7aed1125695598e27ddeb2
Author: Willy Tarreau <[email protected]> Date: 2012-03-17 14:03:53 +0100
Linux 2.6.27.62
… and surely less exploitable than a system suffering oom-killer actions.
Google "linux-2.6.27 download" gives some Ubuntu hits …
Quoted 7 lines
I guess RHEL/Centos might work for you.
I've not had any problems with leaks in 3.6 but there have been a few reports and its clear that some obscure configurations trigger something bad.
Gnome 3 on the other hand leaks like a sieve.
For a system with 128 M total RAM, Gnome is obviously off topic.
Signature · 1 line
Roland
Jan Kara
reviewerQuoted 8 lines · Dimitrios Apostolou
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
Quoted 6 lines
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The
ext4_inode_cacheslab is taking almost all my memory (117MB). Please advise!
Hmm, it seems commit 4eff96dd5283a102e0c1cac95247090be74a38ed might be
interesting for you. It landed in -stable kernels recently as well if I remember right...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 15 lines · Jan Kara
Quoted 8 lines · Dimitrios Apostolou
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
Quoted 6 lines
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The
ext4_inode_cacheslab is taking almost all my memory (117MB). Please advise!
Quoted 3 lines
Hmm, it seems commit
4eff96dd5283a102e0c1cac95247090be74a38edmight be
interesting for you. It landed in -stable kernels recently as well if I remember right...
Thanks, I appreciate your help as I'm stuck in a dead end now, and I've been trying to write some debug hook that prints all ext4_inodes and the reason they are pinned (is there an easy way to find this out?).
So maybe there is a typo in the SHA1 sum you provided? Gitweb can't find it in Linus' tree.
Thanks,
Dimitris
Jan Kara
reviewerQuoted 20 lines · Dimitrios Apostolou
Quoted 15 lines · Jan Kara
Quoted 8 lines · Dimitrios Apostolou
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
Quoted 6 lines
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The
ext4_inode_cacheslab is taking almost all my memory (117MB). Please advise!Quoted 3 lines
Hmm, it seems commit
4eff96dd5283a102e0c1cac95247090be74a38edmight be interesting for you. It landed in -stable kernels recently as well if I remember right...
Quoted 7 lines
Thanks, I appreciate your help as I'm stuck in a dead end now, and I've been trying to write some debug hook that prints all
ext4_inodesand the reason they are pinned (is there an easy way to find this out?).So maybe there is a typo in the SHA1 sum you provided? Gitweb can't find it in Linus' tree.
Strange. You are right gitweb doesn't show the SHA1 but I can see it in
my git repo I pulled from Linus. Anyway, I've attached the fix for your convenience.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 1 line
From
9501fee10d8594ab8ee7deb749fb48c1d3a7985eMon Sep 17 00:00:00 2001
From: Jan Kara <[email protected]>
Date: Mon, 19 Nov 2012 20:01:16 +0100
Subject: [PATCH v3] writeback: Put unused inodes to LRU after writeback completion
Commit 169ebd90 removed iget-iput pair from inode writeback. As a side effect, inodes that are dirty during iput_final() call won't be ever added to inode LRU (iput_final() doesn't add dirty inodes to LRU and later when the inode is cleaned there's noone to add the inode there). Thus inodes are effectively unreclaimable until someone looks them up again.
Practical effect of this bug is limited by the fact that inodes are pinned by a dentry for long enough that the inode gets cleaned. But still the bug can have nasty consequences leading up to OOM conditions under certain circumstances. Following can easily reproduce the problem:
for (( i = 0; i < 1000; i++ )); do
mkdir $i
for (( j = 0; j < 1000; j++ )); do
touch $i/$j
echo 2 > /proc/sys/vm/drop_caches
done
done
then one needs to run 'sync; ls -lR' to make inodes reclaimable again.
We fix the issue by inserting unused clean inodes into the LRU after writeback finishes in inode_sync_complete().
--- fs/fs-writeback.c | 2 ++ fs/inode.c | 16 ++++++++++++++-- fs/internal.h | 1 + 3 files changed, 17 insertions(+), 2 deletions(-)
fs/fs-writeback.c +2 -0
@@ -228,6 +228,8 @@ static void requeue_io(struct inode *inode, struct bdi_writeback *wb)static void inode_sync_complete (struct inode *inode) { inode->i_state &= ~I_SYNC ; +/* If inode is clean an unused, put it into LRU now... */ +inode_add_lru (inode);/* Waiters must see I_SYNC cleared before being woken up */ smp_mb ();wake_up_bit (&inode->i_state, __I_SYNC);
fs/inode.c +14 -2
@@ -408,6 +408,19 @@ static void inode_lru_list_add(struct inode *inode)spin_unlock (&inode->i_sb->s_inode_lru_lock); } +/* +* Add inode to LRU if needed (inode is unused and clean). +* +* Needs inode->i_lock held. +*/ +void inode_add_lru (struct inode *inode) +{ +if (!(inode->i_state & (I_DIRTY |I_SYNC |I_FREEING |I_WILL_FREE )) && + !atomic_read (&inode->i_count) && inode->i_sb->s_flags &MS_ACTIVE ) +inode_lru_list_add (inode); +} + +static void inode_lru_list_del (struct inode *inode) {spin_lock (&inode->i_sb->s_inode_lru_lock);
@@ -1390,8 +1403,7 @@ static void iput_final(struct inode *inode)if (!drop && (sb->s_flags &MS_ACTIVE )) { inode->i_state |=I_REFERENCED ; -if (!(inode->i_state & (I_DIRTY |I_SYNC ))) -inode_lru_list_add (inode); +inode_add_lru (inode);spin_unlock (&inode->i_lock);return ; }
fs/internal.h +1 -0
@@ -110,6 +110,7 @@ extern int open_check_o_direct(struct file *f);* inode.c */ extern spinlock_t inode_sb_list_lock; +extern void inode_add_lru (struct inode *inode);/* * fs-writeback.c
Signature · 1 line
1.7.1
Hi Jan, thanks for your help, with this commit I can't reproduce the problem. The problematic workload is no more inflating the ext4_inode_cache, in fact I've been able to run even heavier workloads with the ext4_inode_cache never surpassing 2-4MB.
Thanks everyone for helping,
Dimitris
Quoted 38 lines · Jan Kara
Quoted 20 lines · Dimitrios Apostolou
Quoted 15 lines · Jan Kara
Quoted 8 lines · Dimitrios Apostolou
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
Quoted 6 lines
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The
ext4_inode_cacheslab is taking almost all my memory (117MB). Please advise!Quoted 3 lines
Hmm, it seems commit
4eff96dd5283a102e0c1cac95247090be74a38edmight be interesting for you. It landed in -stable kernels recently as well if I remember right...Quoted 7 lines
Thanks, I appreciate your help as I'm stuck in a dead end now, and I've been trying to write some debug hook that prints all
ext4_inodesand the reason they are pinned (is there an easy way to find this out?).So maybe there is a typo in the SHA1 sum you provided? Gitweb can't find it in Linus' tree.
Strange. You are right gitweb doesn't show the SHA1 but I can see it in
my git repo I pulled from Linus. Anyway, I've attached the fix for your convenience.Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR