backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused
Hello list,
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
I managed to get some information by renicing a root console to -20, but even then each keypress showed with a minimum 10s delay! In the attached file you can see the dmesg output, SysRq+{W,T}, ps, vmstat, slabs, meminfo.
I think I'm seeing paging of executable pages because ext4_inode_cache is aggressively using all memory, evicting other pages. However under no condition should realtime processes be unresponsive. What do you think? Please note that I've set vm.swappiness to 0 and gradually increased vm.vfs_cache_pressure to 1000000, but see no difference.
Thanks in advance,
Dimitris
P.S. Please CC me in all replies
Hi, the problem in the quoted message still happens, shouldn't all of ext4_inode_cache slab be emptied after "echo 3
Quoted 1 line
/proc/sys/vm/drop_caches"? In my case slab uses too much memory even
after all processes finish and system is in bad shape due to lack of physical RAM. CC'ing tytso and Catalin Marinas since I've not been able to track any leak with kmemleak.
Kernel is booted with slub_debug=,ext4_inode_cache, as this is the only way to avoid for some time the following message. Nevertheless it has not been able to show any leak.
kmemleak: Cannot allocate a kmemleak_object structure
kmemleak: Automatic memory scanning thread ended
kmemleak: Kernel memory leak detector disabled
Please have a look at the log (archived at [1]).
[1] http://lkml.indiana.edu/hypermail/linux/kernel/1211.3/00183.html
Thanks,
Dimitris
Quoted 28 lines · Dimitrios Apostolou
Hello list,
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
I managed to get some information by renicing a root console to -20, but even then each keypress showed with a minimum 10s delay! In the attached file you can see the dmesg output, SysRq+{W,T}, ps, vmstat, slabs, meminfo.
I think I'm seeing paging of executable pages because
ext4_inode_cacheis aggressively using all memory, evicting other pages. However under no condition should realtime processes be unresponsive. What do you think? Please note that I've set vm.swappiness to 0 and gradually increased vm.vfs_cache_pressure to 1000000, but see no difference.Thanks in advance,
Dimitris
P.S. Please CC me in all replies
Quoted 40 lines · On 2012-11-25 Sun 23:59:55 +0100 Roland Eggner wrote:
Quoted 8 lines · On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote:
Quoted 6 lines · Dimitrios Apostolou
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the system now is unresponsive and is thrashing the disk, but the swapfile is rarely touched.
Quoted 6 lines
I'm now having the same experience even though I replaced xz (which needed ~50MB RAM) with gzip. Even though I feel the realtime root shell is a bit more responsive than before, the OOM killer is out killing small processes like syslog-ng and systemd-logind... The
ext4_inode_cacheslab is taking almost all my memory (117MB). Please advise!Hello Dimitrios,
I would try a 2.6.27.* kernel, for following reasons:
(1) Kernel development since 2.6.27 achieved significant performance improvements at the cost of exploding memory consumption by the kernel for _internal_ data structures. I am currently using a 3.2.34 kernel on a Notebook with 4 G RAM. 0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this memory cannot be swapped, it cannot be released by other means than rebooting, and there seems to be _no_ adjustment to memory pressure. I am surprised, that you have managed to boot a 3.6.* kernel at all with only 128 M RAM.
(2) At the time, when I used a PIII-Notebook, kernels 2.6.27 to 2.6.29 where current. Thus chances are good, that a 2.6.27.* kernel will support chipset, PCI bus and devices of your notebook. 2.6.27 got longterm maintainance, the latest release in the linux-stable git repository is 2.6.27.62. So Q @ LKML community:
Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?[1] Picture described in my LKML message
Date: Fri, 20 Jan 2012 01:08:00 +0100 Subject: Re: [kmemleak report 1/2] kernel 3.1.6, x86_64: mm, xfs ?, vfs ?remained the same with _every_ 3.1.* and 3.2.* kernel tried so far.
Quoted 11 lines · On 2012-12-02 Sunday at 14:44 +0200 Dimitrios Apostolou wrote:
Hi, the problem in the quoted message still happens, shouldn't all of
ext4_inode_cacheslab be emptied after "echo 3Quoted 1 line
/proc/sys/vm/drop_caches"? In my case slab uses too much memory evenafter all processes finish and system is in bad shape due to lack of physical RAM. CC'ing tytso and Catalin Marinas since I've not been able to track any leak with kmemleak.
Kernel is booted with slub_debug=,ext4_inode_cache, as this is the only way to avoid for some time the following message. Nevertheless it has not been able to show any leak.
Quoted 7 lines
kmemleak: Cannot allocate a
kmemleak_objectstructure
kmemleak: Automatic memory scanning thread ended
kmemleak: Kernel memory leak detector disabledPlease have a look at the log (archived at [1]).
[1] http://lkml.indiana.edu/hypermail/linux/kernel/1211.3/00183.html
Hello Dimitrios!
Which part of …
“0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this memory cannot be swapped, it cannot be released by other means than rebooting, and there seems to be _no_ adjustment to memory pressure.”
… should I explain?
When tar or gzip writes to ext4 filesystem on your external disk, the kernel keeps all inode data in slab memory _by design_ , not by memory leaks. Several 100 M slab data cannot be stored within 128 M total RAM. This cannot by solved by usage of /proc/sys/vm/drop_caches.
This cannot by solved by oom-killer-actions.
This cannot be solved by zram tricks.
Unmounting the external disk would release part of slab memory, but then you cannot backup.
Reformatting the ext4 filesystem on your external disk with 128 byte inode size, largest possible blocksize and with extents enabled might mitigate the memory pressure slightly … probably not sufficient to get a working system with 3.6 kernel.
One advantage of Linux compared to other OS is much more support for old hardware, if a _proper_ kernel version is selected. Many years ago I used a notebook with 40 M total RAM, with a 2.4 kernel, Blackbox window manager and Opera web browser … it worked flawlessly … just rather slowly due to swapping. Your notebook has much more RAM than 40 M, thus there is surely a Linux solution for you. Try a 2.6.27.62 kernel, it supports ext4 (“ext4dev”), probably it supports all devices of your notebook, and with a slim window manager e.g. WindowMaker or OpenBox your notebook will probably “fly”.
PS: Please type your reply below the text you are replying to.
Signature · 1 line
Roland
Dear Ronald,
sorry for not replying at your first message but I didn't consider changing kernel as a resolution to my problem.
Quoted 10 lines · Roland Eggner
Hello Dimitrios!
Which part of …
“0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this memory cannot be swapped, it cannot be released by other means than rebooting, and there seems to be _no_ adjustment to memory pressure.”
… should I explain?
Check my initial message. If you see the /proc/meminfo attached you'll see that from all slab memory most is reclaimable, however it's not being reclaimed despite the memory pressure:
Slab: 64880 kB SReclaimable: 60496 kB SUnreclaim: 4384 kB
Also I mentioned that a nice'd -20 (Realtime!) process was unresponsive because of two nice'd + ionice'd processes. This is a bug no matter the memory pressure, IMHO. If not I'd rather hear it from someone more authoritative.
Quoted 2 lines
When tar or gzip writes to ext4 filesystem on your external disk, the kernel keeps all inode data in slab memory
_bydesign_, not by memory leaks.
ext4_inode_cache is a *cache*. So the kernel does not keep /all/ inode data there, but only what was needed recently. And drops it afterwards, which didn't happen after the backup ended.
Quoted 2 lines
Several 100 M slab data cannot be stored within 128 M total RAM. This cannot by solved by usage of
/proc/sys/vm/drop_caches.
Please take a look at [2], it's what is called in that case. Via this call all slabs that have registered a shrinker, get actually shrinked. Other fs do, but I can't find whether ext4 actually registers a shrinker for its slabs.
[2] http://lxr.linux.no/#linux+v3.6.8/mm/vmscan.c#L207
Quoted 4 lines
This cannot by solved by oom-killer-actions.
This cannot be solved by zram tricks.
Unmounting the external disk would release part of slab memory, but then you cannot backup.
Unmounting the back'ed up fs didn't really help in my case. Of course all my filesystems are ext[234], mounted by ext4 code, so they use the same slab for caches. But anyway, the slab didn't shrink significantly as I was expecting.
Quoted 6 lines
Reformatting the ext4 filesystem on your external disk with 128 byte inode size, largest possible blocksize and with extents enabled might mitigate the memory pressure slightly … probably not sufficient to get a working system with 3.6 kernel.
Quoted 8 lines
One advantage of Linux compared to other OS is much more support for old hardware, if a _proper_ kernel version is selected. Many years ago I used a notebook with 40 M total RAM, with a 2.4 kernel, Blackbox window manager and Opera web browser … it worked flawlessly … just rather slowly due to swapping. Your notebook has much more RAM than 40 M, thus there is surely a Linux solution for you. Try a 2.6.27.62 kernel, it supports ext4 (“ext4dev”), probably it supports all devices of your notebook, and with a slim window manager e.g. WindowMaker or OpenBox your notebook will probably “fly”.
I appreciate your advice. I remember when we were running 2.4 much less memory was needed, but still I consider the kernel /fairly/ lean considering the time passed and the bloat in userspace applications. I choose to run latest kernels on old hardware, hopefully this will continue to be the case, and if things deteriorate much then let's hope we'll have enough time to help and fix them! :-)
Thanks,
Dimitris
Quoted 5 lines · Dimitrios Apostolou
Please take a look at [2], it's what is called in that case. Via this call all slabs that have registered a shrinker, get actually shrinked. Other fs do, but I can't find whether ext4 actually registers a shrinker for its slabs.
The ext4 inode cache gets shrunk via the VFS layer, in prune_icache_sb() in fs/inode.c. This is true for all file systems' inode caches.
If you are under heavy memory pressure, or if you run echo 3 > /proc/sys/vm/drop_caches, and the inodes aren't getting dropped, then it's because the inodes are getting pinned for some reason --- i.e., they are referenced via some entry in the dentry cache, perhaps there are files open, or processes are cd'ed into a directory, etc.
So an example of what happens with the ext4_inode_cache before and after running "echo 3 > /proc/sys/vm/drop_caches":
ext4_inode_cache 183379 183379 1872 17 8 : tunables 0 0 0 : slabdata 10787 10787 0
ext4_inode_cache 1595 6120 1872 17 8 : tunables 0 0 0 : slabdata 360 360 0
(What's left is due to all of the executable files, shared libraries, current directories, and open files for all my processes running on my system --- which for context is Xfce plus Chrome, emacs, mutt, and a bunch of terminal windows plus the usual assortment of system daemons.)
If you are seeing a large number of inodes still in the ext4 inode cache after using drop_caches, then I'd look to see whether you have something like SELinux or auditing enabled which is pinning a bunch of dentries or inodes, or whether your backup program (or some other program running on your system) is keeping lots of directories or inodes open for some reason.
Hope this helps,
- Ted
Quoted 4 lines · Theodore Ts'o
If you are seeing a large number of inodes still in the ext4 inode cache after using
drop_caches, then I'd look to see whether you have something like SELinux or auditing enabled which is pinning a bunch of dentries or inodes
You can safely ignore this suggestion as it does make sense. SELinux only grabs a references to dentries during its call to fs_ops->getxattr, which can't last a meaningful length of time (unless the filesystem is busted). It only grabs references to inodes during system initialization, when you couldn't have many in core.
Audit, likewise, only grabs a reference to a dentry during execve() and only long enough to run getxattr and does not grab any reference directly to an inode at all.
Quoted 3 lines
or whether your backup program (or some other
program running on your system) is keeping lots of directories or inodes open for some reason.
Certainly could be this suggestion though..
Quoted 13 lines · Eric Paris
Quoted 4 lines · Theodore Ts'o
If you are seeing a large number of inodes still in the ext4 inode cache after using
drop_caches, then I'd look to see whether you have something like SELinux or auditing enabled which is pinning a bunch of dentries or inodesYou can safely ignore this suggestion as it does make sense. SELinux only grabs a references to dentries during its call to
fs_ops->getxattr, which can't last a meaningful length of time (unless the filesystem is busted). It only grabs references to inodes during system initialization, when you couldn't have many in core.
Quoted 4 lines
Audit, likewise, only grabs a reference to a dentry during
execve()and only long enough to run getxattr and does not grab any reference directly to an inode at all.
AFAICT I use neither SELinux nor audit.
Quoted 6 lines
Quoted 3 lines
or whether your backup program (or some other
program running on your system) is keeping lots of directories or inodes open for some reason.Certainly could be this suggestion though..
I've managed to reproduce the scenario with concurrent "du" commands running on the filesystems. I'll try doing it once more, but it may take a while to get the dmesg/slabinfo etc output, since even a realtime root shell is non-responsive for many minutes.
What other debug output do you suggest to get, to find out why ext4 inodes are pinned?
Thanks,
Dimitris
Quoted 4 lines · Dimitrios Apostolou
I've managed to reproduce the scenario with concurrent "du" commands running on the filesystems. I'll try doing it once more, but it may take a while to get the dmesg/slabinfo etc output, since even a realtime root shell is non-responsive for many minutes.
What other debug output do you suggest to get, to find out why ext4 inodes are pinned?
Important detail: one of the filesystems contains *huge* maildirs, containing hundreds of thousands of messages.
Dimitris
Quoted 1 line · On 2012-12-03 Monday at 01:56 +0200 Dimitrios Apostolou wrote:
Dear Ronald,
Excuse me, my name is Roland.
Quoted 14 lines
sorry for not replying at your first message but I didn't consider changing kernel as a resolution to my problem.
… …
Quoted 9 lines
… …
One advantage of Linux compared to other OS is much more support for old hardware, if a _proper_ kernel version is selected. Many years ago I used a notebook with 40 M total RAM, with a 2.4 kernel, Blackbox window manager and Opera web browser … it worked flawlessly … just rather slowly due to swapping. Your notebook has much more RAM than 40 M, thus there is surely a Linux solution for you. Try a 2.6.27.62 kernel, it supports ext4 (“ext4dev”), probably it supports all devices of your notebook, and with a slim window manager e.g. WindowMaker or OpenBox your notebook will probably “fly”.
Quoted 6 lines
I appreciate your advice. I remember when we were running 2.4 much less memory was needed, but still I consider the kernel /fairly/ lean considering the time passed and the bloat in userspace applications. I choose to run latest kernels on old hardware, hopefully this will continue to be the case, and if things deteriorate much then let's hope we'll have enough time to help and fix them! :-)
Ok, this is another story: You are not looking for a least-effort solution just for your notebook. Instead you want to contribute to the LMKL community solutions how to run current kernel versions under extremly tight memory limitations like with your notebook, right? If so: This is highly appreciated, thank you! I will interestingly read your findings :)
Signature · 1 line
Roland
Quoted 3 lines · Roland Eggner
Quoted 1 line · On 2012-12-03 Monday at 01:56 +0200 Dimitrios Apostolou wrote:
Dear Ronald,
Excuse me, my name is Roland.
Sorry, this was not intentional.
Dimitris