Linux 2.6.31 - very swap-happy with plenty of free RAM
Quoted 10 lines · Norbert Preining
So what normally trashed my system to a quasi halt I tried after a swapoff -a,
*two* svn up of really big repositories (some Gb), plus starting VirtualBox Windows XP with 1Gb virtual RAM on a 2Gb machine.And see hoho, no problem at all. Everything remains responsive and happy. The memory was never above 60% in use, but 40% in cache, and all without any problems.
So that seems to be a real bug.
I am running currently 2.6.32-rc7, but experienced that already in the 31-rc version. Hard to pinpoint exactely where it happens.
Similar here (with 2.6.31.6 kernel).
I have a pretty powerful desktop machine with 8 GB RAM, fast disks with RAID-1 etc.
It runs 5 (mostly idle) KVM machines, Firefox, Thunderbird, KDE4, image editing program, multiseat X session. Around 6 GB of RAM used when caches/buffers are excluded.
Every 10 minutes or so, machine is really unresponsive, load jumps to 10 or 20. Mouse pointer jumps, it's impossible to change between windows etc.
Do a "swapoff -a", and everything is snappy and responsive as it should, there are no more lags.
I noticed that with swap disabled, "Dirty" (in /proc/meminfo) is way below 100 MB (usually, 50 MB or so).
With swap enabled, it "Dirty" is usually around 300-500-700 MB, and from time to time, the system basically thrashes the disks, and everything is unresponsive. Similar to uncompressing a big tar archive - lots of IO, system unresponsive.
BTW, did you try:
echo 0 > /proc/sys/vm/swappiness
swapoff -a
swapon -a
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 5 lines · Tomasz Chmielewski
I have a pretty powerful desktop machine with 8 GB RAM, fast disks with RAID-1 etc.
Every 10 minutes or so, machine is really unresponsive, load jumps to 10 or
20. Mouse pointer jumps, it's impossible to change between windows etc.
This is the type of hefty workstation many of the core developers have, running similar workloads, so I'm somewhat surprised that the default VM settings have this kind of issue.
Quoted 2 lines
Do a "swapoff -a", and everything is snappy and responsive as it should, there are no more lags.
Yes, that's my exact finding.
Is this weird IO storm happening for anyone else with plenty of memory (for their taskload?)
Quoted 12 lines · Dan Merillat
Quoted 5 lines · Tomasz Chmielewski
I have a pretty powerful desktop machine with 8 GB RAM, fast disks with RAID-1 etc.
Every 10 minutes or so, machine is really unresponsive, load jumps to 10 or
20. Mouse pointer jumps, it's impossible to change between windows etc.This is the type of hefty workstation many of the core developers have, running similar workloads, so I'm somewhat surprised that the default VM settings have this kind of issue.
Quoted 7 lines
Quoted 2 lines
Do a "swapoff -a", and everything is snappy and responsive as it should, there are no more lags.
Yes, that's my exact finding.
Is this weird IO storm happening for anyone else with plenty of memory (for their taskload?)
For me it happend on my laptop. 3gb RAM, and a 1gb VMWare Windows-XP instance running, plus the usual like firefox, thunderbird, kde4.
Without running vmware it did not happen. And since I have now disabled barriers on the xfs /home partition (on luks crypto lvm) it also does not happen anymore.
The machine is currently 300mb in swap, together with >2gb shown by free as cached - as I just copied 10gb around nothing I worry about.
For me when it happened it looked like when a big bunch of data is to be commited to disk xfs seeks between superblock/journal and data and syncs/waits between each step - reducing the write speed of the disk from the normal >20mb/s to less than 2mb/s and also delaying all other disk-accesses in the meantime.
c'ya
sven
ps: I know with disabling barriers I sacrifice some data safety, but
better a usable and fast system than everything ultra-safe and slow - for safety in suche rare case there is a backup.
Signature · 1 line
The lights are fading out, once more...
Quoted 22 lines · Sven-Haegar Koch
Quoted 12 lines · Dan Merillat
Quoted 5 lines · Tomasz Chmielewski
I have a pretty powerful desktop machine with 8 GB RAM, fast disks with RAID-1 etc.
Every 10 minutes or so, machine is really unresponsive, load jumps to 10 or
20. Mouse pointer jumps, it's impossible to change between windows etc.This is the type of hefty workstation many of the core developers have, running similar workloads, so I'm somewhat surprised that the default VM settings have this kind of issue.
Quoted 7 lines
Quoted 2 lines
Do a "swapoff -a", and everything is snappy and responsive as it should, there are no more lags.
Yes, that's my exact finding.
Is this weird IO storm happening for anyone else with plenty of memory (for their taskload?)
Quoted 6 lines
For me it happend on my laptop. 3gb RAM, and a 1gb VMWare Windows-XP instance running, plus the usual like firefox, thunderbird, kde4.
Without running vmware it did not happen. And since I have now disabled barriers on the xfs /home partition (on luks crypto lvm) it also does not happen anymore.
Yeah, it looks like dm-crypt recently started supporting barriers in commit 647c7db14ef9cacc4ccb3683e206b61f0de6dc2b. Hence XFS will have detected barriers work at mount time mount and so is now issuing them.
Similarly, raid1 (mirror) has recently gained barrier support so the same issue can be seen there.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 18 lines · Dave Chinner
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I have a pretty powerful desktop machine with 8 GB RAM, fast disks with RAID-1 etc.
Every 10 minutes or so, machine is really unresponsive, load jumps to 10 or
20. Mouse pointer jumps, it's impossible to change between windows etc.This is the type of hefty workstation many of the core developers have, running similar workloads, so I'm somewhat surprised that the default VM settings have this kind of issue.
Quoted 2 lines
Do a "swapoff -a", and everything is snappy and responsive as it should, there are no more lags.
Quoted 7 lines
Quoted 4 lines
Yes, that's my exact finding.
Is this weird IO storm happening for anyone else with plenty of memory (for their taskload?)
For me it happend on my laptop. 3gb RAM, and a 1gb VMWare Windows-XP instance running, plus the usual like firefox, thunderbird, kde4.
Without running vmware it did not happen. And since I have now disabled barriers on the xfs /home partition (on luks crypto lvm) it also does not happen anymore.
Quoted 7 lines
Yeah, it looks like dm-crypt recently started supporting barriers in commit
647c7db14ef9cacc4ccb3683e206b61f0de6dc2b. Hence XFS will have detected barriers work at mount time mount and so is now issuing them.Similarly, raid1 (mirror) has recently gained barrier support so the same issue can be seen there.
What is also interesting, that a normal software RAID-1 sync (i.e. from a degraded state) does not seem to make any visible effect on system responsiveness.
Uncompress a big tar file, or VM writes out lots of data - system becomes really unresponsive.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 6 lines · Tomasz Chmielewski
What is also interesting, that a normal software RAID-1 sync (i.e. from a degraded state) does not seem to make any visible effect on system responsiveness.
Uncompress a big tar file, or VM writes out lots of data - system becomes really unresponsive.
Setup:
XFS -> LV -> md0 (degraded) -> ST3500630AS ver 3.AA (500gb) Basically no other disk activity while doing these tests.
barriers on:
$ time git checkout -f Checking out files: 100% (29108/29108), done.
real 2m51.913s
user 0m3.128s
sys 0m3.004s
$ time rm -r *
real 1m52.562s
user 0m0.072s
sys 0m2.980s
$ sudo mount /usr/src -o remount,nobarrier $ time git checkout -f Checking out files: 100% (29108/29108), done.
real 0m9.782s
user 0m2.944s
sys 0m2.984s
$ time rm -r *
real 0m24.996s
user 0m0.076s
sys 0m2.808s
So XFS + barriers are part of the culprit here, but I was only using xfs for /usr/src. Reformatted that to ext4 after I found that little
nugget of joy. Ext4+barriers isn't anywhere near as dramatic a hit,
it's a much more reasonable speed/safety tradeoff. Again, that's
only for /usr/src, the rest of my system is using ext4, so it doesn't explain any other workload problems. btrfs doesn't seem to have much of a hit at all using barriers but I'd need to test that properly, it's below 6 seconds for checkout and the differences would be in the noise without averaging multiple runs.
Quoted 10 lines · KOSAKI Motohiro
Hi Dan,
Umm, very strange.
I made two debug patch. can you please apply it and post following command output?% cat
/proc/meminfo
% cat/proc/vmstat
% cat/proc/zoneinfo# cat /proc/filecache | sort -nr -k3 |head -30
Unfortunately not, it doesn't compile on 2.6.31 which means I'd have to re-port vmware & fglrx just to test that.
CC fs/proc/filecache.ofs/proc/filecache.c: In function ‘iwin_fill’:fs/proc/filecache.c:108: error: ‘bdi_lock’ undeclared (first use in this function)fs/proc/filecache.c:108: error: (Each undeclared identifier is reported only oncefs/proc/filecache.c:108: error: for each function it appears in.) fs/proc/filecache.c:109: error: ‘struct backing_dev_info’ has no member named ‘bdi_list’fs/proc/filecache.c:109: warning: type defaults to ‘int’ in declaration of ‘__mptr’fs/proc/filecache.c:109: error: ‘bdi_list’ undeclared (first use in this function)fs/proc/filecache.c:109: error: ‘struct backing_dev_info’ has no member named ‘bdi_list’fs/proc/filecache.c:109: error: ‘struct backing_dev_info’ has no member named ‘bdi_list’
And I can't forward port the patch without major work, since the whole bdi_writeback structure was introduced post 2.6.31. The recent-rotated patch works, I'll include that data as soon as I get the memory pressure back up.
Right now it's behaving correctly after the reboot - the usual sign of a problem is free memory going way up while swapping like mad. I'm putting a lot of memory pressure on the kernel post-reboot but swap is
behaving normally. I hope this doesn't take multiple days of uptime
to get back into that state.
Quoted 12 lines · Dan Merillat
Quoted 5 lines · Tomasz Chmielewski
I have a pretty powerful desktop machine with 8 GB RAM, fast disks with RAID-1 etc.
Every 10 minutes or so, machine is really unresponsive, load jumps to 10 or
20. Mouse pointer jumps, it's impossible to change between windows etc.This is the type of hefty workstation many of the core developers have, running similar workloads, so I'm somewhat surprised that the default VM settings have this kind of issue.
Quoted 7 lines
Quoted 2 lines
Do a "swapoff -a", and everything is snappy and responsive as it should, there are no more lags.
Yes, that's my exact finding.
Is this weird IO storm happening for anyone else with plenty of memory (for their taskload?)
My load is virtually the same a Tomasz describes, virtual machines (including this desktop), little backup servers on CentOS-5.3, production Fedora-9 and 10, testing FC11 and 12. I see none of this problem, with the Fedora kernel or my own build of 2.6.31 whatever was stable two weeks ago.
What is not the same is that all are started from CLI and scripts, because much of my VM stuff predates libvirt. So nothing between qemu-kvm and my fingers or boot scripts.
I have no idea if this is useful or significant, just sharing it "in case."
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot