unresponsiveness on linux desktop during file copy
Subsystems (5)
Hello everyone,
I am trying to figure out why the Linux desktop becomes overly unresponsive whenever there is a file copy going on, this happens to a large extent with GUI based file copying, and some lesser, but with command line copying too.
does it have anything to do with the kernel ??,
if it does help, then I am running Fedora 10 on a Hp nx 7400, Centrino Duo processor.
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 10 lines · Satish Eerpini
Hello everyone,
I am trying to figure out why the Linux desktop becomes overly unresponsive whenever there is a file copy going on, this happens to a large extent with GUI based file copying, and some lesser, but with command line copying too.
does it have anything to do with the kernel ??,
if it does help, then I am running Fedora 10 on a Hp nx 7400, Centrino Duo processor.
Quoted 3 lines
Cheers
Satish
--
Quickly, is the system getting good hard drive performance overall?
hdparm -tT /dev/sda
or whatever device you are using. Maybe it's stuck in some sort of programmed IO or something very old?
- Mark
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines
I presume that if you start your machine and don't start X at all, never run Gnome or whatever, that you get similar numbers in your console? Possibly the desktop is using a lot of CPU but I doubt it. I've never seen that. Every time I've had problems like this it's been a newish chipset and the driver for the hard drive interface isn't up to snuff yet.
yeah I get similar numbers even in the absence of the desktop manager , when I tried to copy from a virtual console.
the stats look something like this :
Linux 2.6.29.1 (satish) 05/16/2009
08:01:49 AM CPU %user %nice %system %iowait %steal %idle 08:01:52 AM all 3.68 0.00 6.72 41.60 0.00 48.00 08:01:55 AM all 7.10 0.00 15.00 35.81 0.00 42.10 08:01:58 AM all 10.45 0.00 22.03 51.29 0.00 16.24 08:02:01 AM all 5.37 0.00 9.12 72.48 0.00 13.03 08:02:04 AM all 7.25 0.00 12.40 37.04 0.00 43.32 Average: all 6.77 0.00 13.06 47.58 0.00 32.59
Quoted 1 line
So, maybe something now about the actual hardware? lspci, lsmod, uname
uname -a :
Linux satish 2.6.29.1 #1 SMP Tue Apr 21 01:17:09 IST 2009 i686 i686 i386 GNU/Linux
lspci
00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03)
00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 01)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01)
00:1c.1 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 2 (rev 01)
00:1c.3 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 4 (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e1) 00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge (rev 01)
00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 01)
02:06.0 CardBus bridge: Texas Instruments PCIxx12 Cardbus Controller 02:06.1 FireWire (IEEE 1394): Texas Instruments PCIxx12 OHCI Compliant IEEE 1394 Host Controller
02:0e.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev 02) 10:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection (rev 02)
(sorry , I did not know what to filter out)
Quoted 2 lines
-a. This is your main drive, correct? The driver is built in and not modular, or is it a second drive?
does it make a difference if the driver is built in or a module ??
Satish
Signature · 1 line
http://satish.playdrupal.com
On Fri, May 15, 2009 at 7:38 PM, Satish Eerpini <[email protected]> wrote:
<SNIP>
Quoted 3 lines
yeah I get similar numbers even in the absence of the desktop manager , when I tried to copy from a virtual console.
the stats look something like this :
<SNIP>
Quoted 2 lines
00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 01)
<SNIP>
I googled on "82801GBM/GHM slow" and found a lot to read.
Is DMA for this drive enabled? It seems that some others didn't have it enabled by default, or maybe at all. I didn't real the threads very far.
hdparm -d1 /dev/sda
I think hdparm works for most SATA drivers. man sata to see the other things you can check in terms of configuration.
Clearly, since you see the same thing in a console with X never having been started says it's not a GUI issue.
Hope this helps,
Mark
Quoted 1 line
I googled on "82801GBM/GHM slow" and found a lot to read.
will read through that ,...
Quoted 3 lines
Is DMA for this drive enabled? It seems that some others didn't have it enabled by default, or maybe at all. I didn't real the threads very far.
the udma5 mode seems to be enabled , though I can't really make any sense out of it:
hdparm -i /dev/sda
/dev/sda:
Model=TOSHIBA MK1234GSX , FwRev=AH001H ,
SerialNo= 372HFDVGS
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?16? CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=234441648 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: sdma0 sdma1 sdma2 mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 AdvancedPM=yes: unknown setting WriteCache=enabled Drive conforms to: Unspecified: ATA/ATAPI-3,4,5,6,7
* signifies the current active mode
Quoted 1 line
hdparm -d1
/dev/sda
got the following error when I tried setting the mode as above, I think along with -d1 we need to give some "X" argument( or so reads the man page)..
hdparm -d1 /dev/sda
/dev/sda:
setting using_dma to 1 (on)
HDIO_SET_DMA failed: Inappropriate ioctl for device HDIO_GET_DMA failed: Inappropriate ioctl for device
Quoted 2 lines
Clearly, since you see the same thing in a console with X never having been started says it's not a GUI issue.
yes I suppose, but any idea how I figure out what exactly is causing the IOWAIT, .. and if there are any patches, let me know, I can test them .
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 46 lines · Satish Eerpini
Quoted 1 line
I googled on "82801GBM/GHM slow" and found a lot to read.
will read through that ,...
Quoted 3 lines
Is DMA for this drive enabled? It seems that some others didn't have it enabled by default, or maybe at all. I didn't real the threads very far.
the udma5 mode seems to be enabled , though I can't really make any sense out of it:
hdparm -i
/dev/sda
/dev/sda:Model=TOSHIBA MK1234GSX , FwRev=AH001H , SerialNo= 372HFDVGS
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?16? CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=234441648 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: sdma0 sdma1 sdma2 mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 AdvancedPM=yes: unknown setting WriteCache=enabled Drive conforms to: Unspecified: ATA/ATAPI-3,4,5,6,7* signifies the current active mode
Quoted 1 line
hdparm -d1
/dev/sdagot the following error when I tried setting the mode as above, I think along with -d1 we need to give some "X" argument( or so reads the man page)..
hdparm -d1
/dev/sda
/dev/sda:
settingusing_dmato 1 (on)
HDIO_SET_DMA failed: Inappropriate ioctl for deviceHDIO_GET_DMAfailed: Inappropriate ioctl for deviceQuoted 2 lines
Clearly, since you see the same thing in a console with X never having been started says it's not a GUI issue.
Quoted 6 lines
yes I suppose, but any idea how I figure out what exactly is causing the IOWAIT, .. and if there are any patches, let me know, I can test them .
Cheers
Satish
OK, the solution for this is probably beyond my expertise but this looks identical to when I got a Compaq laptop and the ATI chipset's DMA driver wasn't working right. hdparm said I was using udma but I got error messages like you are when I tried to enable DMA.
Do you believe this was working better on an earlier kernel? If so you might drop back and see if these commands work better. If so that would suggest a regression of some type.
Hopefully someone here can direct you to the right person that supports this.
- Mark
Quoted 3 lines
Do you believe this was working better on an earlier kernel? If so you might drop back and see if these commands work better. If so that would suggest a regression of some type.
I remember good performance with 2.6.26.x series, ... but I could not try that out, ,... things on a 2.6.27.23 kernel seem no better :
Linux 2.6.27.23 (satish) 05/16/2009
12:11:18 PM CPU %user %nice %system %iowait %steal %idle 12:11:21 PM all 0.00 0.00 3.92 75.65 0.00 20.42 12:11:24 PM all 0.00 0.00 3.55 67.85 0.00 28.59 12:11:27 PM all 0.16 0.00 4.51 45.09 0.00 50.24 12:11:30 PM all 0.00 0.00 3.59 44.77 0.00 51.63 12:11:33 PM all 0.16 0.00 5.25 82.51 0.00 12.08 Average: all 0.06 0.00 4.17 63.24 0.00 32.53
the average disk speed was about 10.5 MB/s, indeed the average IOWAIT is more than that in the latest kernel, I will see if I can test it on a 2.6.26.x kernel or earlier one and send you guys the stats.
Cheers
satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 7 lines · Satish Eerpini
Quoted 3 lines
Do you believe this was working better on an earlier kernel? If so you might drop back and see if these commands work better. If so that would suggest a regression of some type.
I remember good performance with 2.6.26.x series, ... but I could not try that out, ,... things on a 2.6.27.23 kernel seem no better :
I think it's critically important that you find a kernel that has good performance. Without it we're trusting a memory. With it developers can bisect and find the solution.
- Mark
Quoted 17 lines · Satish Eerpini
Quoted 3 lines
Do you believe this was working better on an earlier kernel? If so you might drop back and see if these commands work better. If so that would suggest a regression of some type.
I remember good performance with 2.6.26.x series, ... but I could not try that out, ,... things on a 2.6.27.23 kernel seem no better :
Linux 2.6.27.23 (satish) 05/16/2009
12:11:18 PM CPU %user %nice %system %iowait %steal %idle 12:11:21 PM all 0.00 0.00 3.92 75.65 0.00 20.42 12:11:24 PM all 0.00 0.00 3.55 67.85 0.00 28.59 12:11:27 PM all 0.16 0.00 4.51 45.09 0.00 50.24 12:11:30 PM all 0.00 0.00 3.59 44.77 0.00 51.63 12:11:33 PM all 0.16 0.00 5.25 82.51 0.00 12.08 Average: all 0.06 0.00 4.17 63.24 0.00 32.53
Quoted 3 lines
the average disk speed was about 10.5 MB/s, indeed the average IOWAIT is more than that in the latest kernel, I will see if I can test it on a 2.6.26.x kernel or earlier one and send you guys the stats.
High iowait during a file copy with no other activity is entirely normal. If you have a core that has nothing to do but wait for IO to complete, you'll get iowait time. This isn't indicating a problem.
Quoted 3 lines
High iowait during a file copy with no other activity is entirely normal. If you have a core that has nothing to do but wait for IO to complete, you'll get iowait time. This isn't indicating a problem.
then the responsiveness seemed to be better with Wu Fengguang's patches applied, or when I decreased the swappiness by decreasing the value in /proc/sys/vm/swappiness . So I think I will stick with that, and thanks everyone for the help.
thanks
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 12 lines · Mark Knecht
On Fri, May 15, 2009 at 7:38 PM, Satish Eerpini <[email protected]> wrote:
<SNIP>Quoted 3 lines
yeah I get similar numbers even in the absence of the desktop manager , when I tried to copy from a virtual console.
the stats look something like this :<SNIP>
Quoted 2 lines
00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 01)
<SNIP>
I googled on "82801GBM/GHM slow" and found a lot to read.
Quoted 8 lines
Is DMA for this drive enabled? It seems that some others didn't have it enabled by default, or maybe at all. I didn't real the threads very far.
hdparm -d1
/dev/sdaI think hdparm works for most SATA drivers. man sata to see the other things you can check in terms of configuration.
Switching on DMA manually isn't possible with libata through hdparm currently. However, this shouldn't be required with libata.
Quoted 6 lines
Clearly, since you see the same thing in a console with X never having been started says it's not a GUI issue.
Hope this helps,
Mark
Quoted 9 lines · Satish Eerpini
Hello everyone,
I am trying to figure out why the Linux desktop becomes overly unresponsive whenever there is a file copy going on, this happens to a large extent with GUI based file copying, and some lesser, but with command line copying too.
does it have anything to do with the kernel ??,
if it does help, then I am running Fedora 10 on a Hp nx 7400, Centrino Duo processor.
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
Quoted 5 lines · Ray Lee
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
Please keep me in the loop. I'd be interested if those patches work and what effect they have.
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Ray Lee
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
Please keep me in the loop. I'd be interested if those patches work and what effect they have.
Whoops, looks like Wu Fengguang (cc:d), not Rik, has been working on those patches. See the messages starting with "[PATCH 0/3] make mapped executable pages the first class citizen" [
http://lkml.org/lkml/2009/5/16/23 ]
Wu: The original reporter in this thread, Satish Eerpini, is reporting interactivity problems that sound very much like mapped executables are getting paged out while doing large copies. I suggested that perhaps he'd like to try out your patches to see if it addresses his problem. His original message is reproduced below. Later in the thread, he said 2.6.26 was fine, 2.6.27 & .28 were not.
Quoted 14 lines
Hello everyone,
I am trying to figure out why the Linux desktop becomes overly unresponsive whenever there is a file copy going on, this happens to a large extent with GUI based file copying, and some lesser, but with command line copying too.
does it have anything to do with the kernel ??,
if it does help, then I am running Fedora 10 on a Hp nx 7400, Centrino Duo processor.Cheers
Satish
--
http://satish.playdrupal.com
Quoted 9 lines · Ray Lee
Quoted 7 lines · Christoph Lameter
Quoted 5 lines · Ray Lee
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
Quoted 7 lines
Quoted 2 lines
Please keep me in the loop. I'd be interested if those patches work and what effect they have.
Whoops, looks like Wu Fengguang (cc:d), not Rik, has been working on those patches. See the messages starting with "[PATCH 0/3] make mapped executable pages the first class citizen" [
http://lkml.org/lkml/2009/5/16/23 ]
We both have. Wu's patches build on a patch of mine, which Andrew has already merged into -mm.
http://lkml.org/lkml/2009/4/29/489
Signature · 1 line
All rights reversed.
Quoted 18 lines · Rik van Riel
Quoted 9 lines · Ray Lee
Quoted 7 lines · Christoph Lameter
Quoted 5 lines · Ray Lee
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
Quoted 7 lines
Quoted 2 lines
Please keep me in the loop. I'd be interested if those patches work and what effect they have.
Whoops, looks like Wu Fengguang (cc:d), not Rik, has been working on those patches. See the messages starting with "[PATCH 0/3] make mapped executable pages the first class citizen" [
http://lkml.org/lkml/2009/5/16/23 ]
Quoted 4 lines
We both have. Wu's patches build on a patch of mine, which Andrew has already merged into -mm.
Yes, better to apply Rik and mine patches together to get the best cache behavior. Rik's patch is crucial. They are both small patches, so can be backported trivially. Satish, what's your kernel version (or, do you want to try a newer kernel)? And the content of /proc/meminfo?
Thanks,
Fengguang
Quoted 5 lines
Yes, better to apply Rik and mine patches together to get the best cache behavior. Rik's patch is crucial. They are both small patches, so can be backported trivially. Satish, what's your kernel version (or, do you want to try a newer kernel)? And the content of
/proc/meminfo?
I am running 2.6.29.1 right now , just tell me which kernel to try, or to which kernel I have to apply the patches and try them, I will test the patches and let you know if the situation improves.
and the contents of /proc/meminfo are :
MemTotal: 2585724 kB MemFree: 900524 kB Buffers: 197504 kB Cached: 1080332 kB SwapCached: 0 kB Active: 639544 kB Inactive: 933068 kB Active(anon): 307100 kB Inactive(anon): 0 kB Active(file): 332444 kB Inactive(file): 933068 kB Unevictable: 16 kB Mlocked: 16 kB HighTotal: 1707848 kB HighFree: 321868 kB LowTotal: 877876 kB LowFree: 578656 kB SwapTotal: 3526256 kB SwapFree: 3526256 kB Dirty: 1000 kB Writeback: 0 kB AnonPages: 294788 kB Mapped: 80084 kB Slab: 89692 kB SReclaimable: 71436 kB SUnreclaim: 18256 kB PageTables: 6008 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4819116 kB Committed_AS: 796332 kB VmallocTotal: 122880 kB VmallocUsed: 8240 kB VmallocChunk: 96952 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 4096 kB DirectMap4k: 20472 kB DirectMap4M: 884736 kB
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 10 lines · Satish Eerpini
Quoted 5 lines
Yes, better to apply Rik and mine patches together to get the best cache behavior. Rik's patch is crucial. They are both small patches, so can be backported trivially. Satish, what's your kernel version (or, do you want to try a newer kernel)? And the content of
/proc/meminfo?I am running 2.6.29.1 right now , just tell me which kernel to try, or to which kernel I have to apply the patches and try them, I will test the patches and let you know if the situation improves.
and the contents of/proc/meminfoare :
Satish, so you have 2.5GB memory and 80MB mapped pages, 33MB active desktop files to protect.
I've prepared a rolled up patch for you, run time tested. It should protect all of the above pages from being evicted by large file copies.
The attached patch is for 2.6.29.
Thanks,
Fengguang
Quoted 43 lines
MemTotal: 2585724 kB MemFree: 900524 kB Buffers: 197504 kB Cached: 1080332 kB SwapCached: 0 kB Active: 639544 kB Inactive: 933068 kB Active(anon): 307100 kB Inactive(anon): 0 kB Active(file): 332444 kB Inactive(file): 933068 kB Unevictable: 16 kB Mlocked: 16 kB HighTotal: 1707848 kB HighFree: 321868 kB LowTotal: 877876 kB LowFree: 578656 kB SwapTotal: 3526256 kB SwapFree: 3526256 kB Dirty: 1000 kB Writeback: 0 kB AnonPages: 294788 kB Mapped: 80084 kB Slab: 89692 kB SReclaimable: 71436 kB SUnreclaim: 18256 kB PageTables: 6008 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4819116 kB Committed_AS: 796332 kB VmallocTotal: 122880 kB VmallocUsed: 8240 kB VmallocChunk: 96952 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 4096 kB DirectMap4k: 20472 kB DirectMap4M: 884736 kB
Quoted 4 lines
Cheers
SatishSignature · 1 line
http://satish.playdrupal.com
mm/vmscan.c +88 -22
@@ -581,6 +581,7 @@ static unsigned long shrink_page_list(ststruct pagevec freed_pvec;int pgactivate =0 ;unsigned long nr_reclaimed =0 ; +unsigned long vm_flags;cond_resched ();
@@ -631,7 +632,8 @@ static unsigned long shrink_page_list(st goto keep_locked; } - referenced =page_referenced (page,1 , sc->mem_cgroup); + referenced =page_referenced (page,1 , + sc->mem_cgroup, &vm_flags);/* In active use or really unfreeable? Activate it. */ if (sc->order <=PAGE_ALLOC_COSTLY_ORDER && referenced &&page_mapping_inuse (page))
@@ -1210,9 +1212,10 @@ static void shrink_active_list(unsigned struct scan_control *sc, int priority, int file) { unsigned long pgmoved; -int pgdeactivate =0 ;unsigned long pgscanned; +unsigned long vm_flags;LIST_HEAD (l_hold);/* The pages which were snipped off */ +LIST_HEAD (l_active);LIST_HEAD (l_inactive);struct page *page;struct pagevec pvec;
@@ -1239,7 +1242,7 @@ static void shrink_active_list(unsigned__mod_zone_page_state (zone,NR_ACTIVE_ANON , -pgmoved);spin_unlock_irq (&zone->lru_lock); - pgmoved =0 ; + pgmoved =0 ;/* count referenced (mapping) mapped pages */ while (!list_empty (&l_hold)) {cond_resched (); page =lru_to_page (&l_hold);
@@ -1252,28 +1255,41 @@ static void shrink_active_list(unsigned/* page_referenced clears PageReferenced */ if (page_mapping_inuse (page) && -page_referenced (page,0 , sc->mem_cgroup)) +page_referenced (page,0 , sc->mem_cgroup, &vm_flags)) { pgmoved++; +/* +* Identify referenced, file-backed active pages and +* give them one more trip around the active list. So +* that executable code get better chances to stay in +* memory under moderate memory pressure. Anon pages +* are ignored, since JVM can create lots of anon +* VM_EXEC pages. +*/ +if ((vm_flags &VM_EXEC ) && !PageAnon (page)) { +list_add (&page->lru, &l_active); +continue ; + } + }list_add (&page->lru, &l_inactive); }/* -* Move the pages to the [file or anon] inactive list. +* Move pages back to the lru list. */ pagevec_init (&pvec,1 ); - lru =LRU_BASE + file *LRU_FILE ;spin_lock_irq (&zone->lru_lock);/* -* Count referenced pages from currently used mappings as -* rotated, even though they are moved to the inactive list. -* This helps balance scan pressure between file and anonymous -* pages in get_scan_ratio. +* Count referenced pages from currently used mappings as rotated, +* even though only some of them are actually re-activated. This +* helps balance scan pressure between file and anonymous pages in +* get_scan_ratio. */ reclaim_stat->recent_rotated[!!file] += pgmoved; - pgmoved =0 ; + pgmoved =0 ;/* count pages moved to inactive list */ + lru =LRU_BASE + file *LRU_FILE ;while (!list_empty (&l_inactive)) { page =lru_to_page (&l_inactive);prefetchw_prev_lru_page (page, &l_inactive, flags);
@@ -1286,10 +1302,7 @@ static void shrink_active_list(unsignedmem_cgroup_add_lru_list (page, lru); pgmoved++;if (!pagevec_add (&pvec, page)) { -__mod_zone_page_state (zone,NR_LRU_BASE + lru, pgmoved);spin_unlock_irq (&zone->lru_lock); - pgdeactivate += pgmoved; - pgmoved =0 ;if (buffer_heads_over_limit)pagevec_strip (&pvec);__pagevec_release (&pvec);
@@ -1297,14 +1310,31 @@ static void shrink_active_list(unsigned } }__mod_zone_page_state (zone,NR_LRU_BASE + lru, pgmoved); - pgdeactivate += pgmoved; -if (buffer_heads_over_limit) { -spin_unlock_irq (&zone->lru_lock); -pagevec_strip (&pvec); -spin_lock_irq (&zone->lru_lock); - }__count_zone_vm_events (PGREFILL , zone, pgscanned); -__count_vm_events (PGDEACTIVATE , pgdeactivate); +__count_vm_events (PGDEACTIVATE , pgmoved); + + pgmoved =0 ;/* count pages moved back to active list */ + lru =LRU_ACTIVE + file *LRU_FILE ; +while (!list_empty (&l_active)) { + page =lru_to_page (&l_active); +prefetchw_prev_lru_page (page, &l_active, flags); +VM_BUG_ON (PageLRU (page)); +SetPageLRU (page); +VM_BUG_ON (!PageActive (page)); + +list_move (&page->lru, &zone->lru[lru].list); +mem_cgroup_add_lru_list (page, lru); + pgmoved++; +if (!pagevec_add (&pvec, page)) { +spin_unlock_irq (&zone->lru_lock); +if (buffer_heads_over_limit) +pagevec_strip (&pvec); +__pagevec_release (&pvec); +spin_lock_irq (&zone->lru_lock); + } + } +__mod_zone_page_state (zone,NR_LRU_BASE + lru, pgmoved); +spin_unlock_irq (&zone->lru_lock);if (vm_swap_full ())pagevec_swap_free (&pvec);
@@ -1344,12 +1374,48 @@ static int inactive_anon_is_low(struct zreturn low; } +static int inactive_file_is_low_global (struct zone *zone) +{ +unsigned long active, inactive; + + active =zone_page_state (zone,NR_ACTIVE_FILE ); + inactive =zone_page_state (zone,NR_INACTIVE_FILE ); + +return (active > inactive); +} + +/** +* inactive_file_is_low - check if file pages need to be deactivated +* @zone: zone to check +* @sc: scan control of this context +* +* When the system is doing streaming IO, memory pressure here +* ensures that active file pages get deactivated, until more +* than half of the file pages are on the inactive list. +* +* Once we get to that situation, protect the system's working +* set from being evicted by disabling active file page aging. +* +* This uses a different ratio than the anonymous pages, because +* the page cache uses a use-once replacement algorithm. +*/ +static int inactive_file_is_low (struct zone *zone,struct scan_control *sc) +{ +int low; + +if (scanning_global_lru (sc)) + low =inactive_file_is_low_global (zone); +else + low =mem_cgroup_inactive_file_is_low (sc->mem_cgroup); +return low; +} +static unsigned long shrink_list (enum lru_list lru,unsigned long nr_to_scan,struct zone *zone,struct scan_control *sc,int priority) {int file =is_file_lru (lru); -if (lru ==LRU_ACTIVE_FILE ) { +if (lru ==LRU_ACTIVE_FILE &&inactive_file_is_low (zone, sc)) {shrink_active_list (nr_to_scan, zone, sc, priority, file);return 0 ; }
include/linux/memcontrol.h +7 -0
@@ -97,6 +97,7 @@ extern void mem_cgroup_note_reclaim_prioextern void mem_cgroup_record_reclaim_priority (struct mem_cgroup *mem,int priority);int mem_cgroup_inactive_anon_is_low (struct mem_cgroup *memcg); +int mem_cgroup_inactive_file_is_low (struct mem_cgroup *memcg);unsigned long mem_cgroup_zone_nr_pages (struct mem_cgroup *memcg,struct zone *zone,enum lru_list lru);
@@ -250,6 +251,12 @@ mem_cgroup_inactive_anon_is_low(struct mreturn 1 ; } +static inline int +mem_cgroup_inactive_file_is_low (struct mem_cgroup *memcg) +{ +return 1 ; +} +static inline unsigned long mem_cgroup_zone_nr_pages (struct mem_cgroup *memcg,struct zone *zone,enum lru_list lru)
mm/memcontrol.c +11 -0
@@ -536,6 +536,17 @@ int mem_cgroup_inactive_anon_is_low(strureturn 0 ; } +int mem_cgroup_inactive_file_is_low (struct mem_cgroup *memcg) +{ +unsigned long active; +unsigned long inactive; + + inactive =mem_cgroup_get_local_zonestat (memcg,LRU_INACTIVE_FILE ); + active =mem_cgroup_get_local_zonestat (memcg,LRU_ACTIVE_FILE ); + +return (active > inactive); +} +unsigned long mem_cgroup_zone_nr_pages (struct mem_cgroup *memcg,struct zone *zone,enum lru_list lru)
include/linux/rmap.h +3 -2
@@ -83,7 +83,8 @@ static inline void page_dup_rmap(struct/* * Called from mm/vmscan.c to handle paging out */ -int page_referenced (struct page *,int is_locked,struct mem_cgroup *cnt); +int page_referenced (struct page *,int is_locked, +struct mem_cgroup *cnt,unsigned long *vm_flags);int try_to_unmap (struct page *,int ignore_refs);/*
@@ -124,7 +125,7 @@ static inline int try_to_munlock(struct#define anon_vma_prepare (vma) (0)#define anon_vma_link (vma) do {} while (0) -#define page_referenced (page,l,cnt) TestClearPageReferenced(page) +#define page_referenced (page, locked, cnt, flags) TestClearPageReferenced(page)#define try_to_unmap (page, refs) SWAP_FAILstatic inline int page_mkclean (struct page *page)
mm/rmap.c +26 -11
@@ -333,7 +333,9 @@ static int page_mapped_in_vma(struct pag* repeatedly from either page_referenced_anon or page_referenced_file. */ static int page_referenced_one (struct page *page, -struct vm_area_struct *vma,unsigned int *mapcount) +struct vm_area_struct *vma, +unsigned int *mapcount, +unsigned long *vm_flags) {struct mm_struct *mm = vma->vm_mm;unsigned long address;
@@ -381,11 +383,14 @@ out_unmap: (*mapcount)--;pte_unmap_unlock (pte, ptl); out: +if (referenced) + *vm_flags |= vma->vm_flags;return referenced; }static int page_referenced_anon (struct page *page, -struct mem_cgroup *mem_cont) +struct mem_cgroup *mem_cont, +unsigned long *vm_flags) {unsigned int mapcount;struct anon_vma *anon_vma;
@@ -405,7 +410,8 @@ static int page_referenced_anon(struct p*/ if (mem_cont && !mm_match_cgroup (vma->vm_mm, mem_cont))continue ; - referenced +=page_referenced_one (page, vma, &mapcount); + referenced +=page_referenced_one (page, vma, + &mapcount, vm_flags);if (!mapcount)break ; }
@@ -418,6 +424,7 @@ static int page_referenced_anon(struct p* page_referenced_file - referenced check for object-based rmap * @page: the page we're checking references on. * @mem_cont: target memory controller +* @vm_flags: collect encountered vma->vm_flags * * For an object-based mapped page, find all the places it is mapped and * check/clear the referenced flag. This is done by following the page->mapping
@@ -427,7 +434,8 @@ static int page_referenced_anon(struct p* This function is only called from page_referenced for object-based pages. */ static int page_referenced_file (struct page *page, -struct mem_cgroup *mem_cont) +struct mem_cgroup *mem_cont, +unsigned long *vm_flags) {unsigned int mapcount;struct address_space *mapping = page->mapping;
@@ -467,7 +475,8 @@ static int page_referenced_file(struct p*/ if (mem_cont && !mm_match_cgroup (vma->vm_mm, mem_cont))continue ; - referenced +=page_referenced_one (page, vma, &mapcount); + referenced +=page_referenced_one (page, vma, + &mapcount, vm_flags);if (!mapcount)break ; }
@@ -481,29 +490,35 @@ static int page_referenced_file(struct p* @page: the page to test * @is_locked: caller holds lock on the page * @mem_cont: target memory controller +* @vm_flags: collect encountered vma->vm_flags * * Quick test_and_clear_referenced for all mappings to a page, * returns the number of ptes which referenced the page. */ -int page_referenced (struct page *page,int is_locked, -struct mem_cgroup *mem_cont) +int page_referenced (struct page *page, +int is_locked, +struct mem_cgroup *mem_cont, +unsigned long *vm_flags) {int referenced =0 ;if (TestClearPageReferenced (page)) referenced++; + *vm_flags =0 ;if (page_mapped (page) && page->mapping) {if (PageAnon (page)) - referenced +=page_referenced_anon (page, mem_cont); + referenced +=page_referenced_anon (page, mem_cont, + vm_flags);else if (is_locked) - referenced +=page_referenced_file (page, mem_cont); + referenced +=page_referenced_file (page, mem_cont, + vm_flags);else if (!trylock_page (page)) referenced++;else {if (page->mapping) - referenced += -page_referenced_file (page, mem_cont); + referenced +=page_referenced_file (page, + mem_cont, vm_flags);unlock_page (page); } }
Quoted 4 lines
I've prepared a rolled up patch for you, run time tested. It should protect all of the above pages from being evicted by large file copies.
The attached patch is for 2.6.29.
I applied the patch , but the number don seem to show much difference , following are the statistics with the patched kernel, could something else be causing the huge iowait, :
Linux 2.6.29 (satish) 05/17/2009
09:57:14 AM CPU %user %nice %system %iowait %steal %idle
09:57:18 AM all 12.93 0.00 10.25 33.44
0.00 43.38
09:57:21 AM all 13.32 0.00 28.09 24.40
0.00 34.19
09:57:24 AM all 10.79 0.00 4.76 75.56
0.00 8.89
09:57:26 AM all 11.46 0.00 8.01 35.64
0.00 44.90
09:57:30 AM all 11.68 0.00 8.88 35.05
0.00 44.39
Average: all 12.03 0.00 11.94 40.81
0.00 35.22
and
hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 1332 MB in 2.00 seconds = 666.35 MB/sec
Timing buffered disk reads: 24 MB in 3.32 seconds = 7.23 MB/sec
,
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 10 lines · On Sun, 17 May 2009 10:00:28 +0530 Satish Eerpini <[email protected]> wrote:
Quoted 5 lines
I've prepared a rolled up patch for you, run time tested. It should protect all of the above pages from being evicted by large file copies.
The attached patch is for 2.6.29.
I applied the patch , but the number don seem to show much difference , following are the statistics with the patched kernel, could something else be causing the huge iowait, :
for i in pidof kjournald ; do ionice -c1 -p $i ; done
is another thing to try to fix the responsiveness...
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 6 lines · Satish Eerpini
Quoted 4 lines
I've prepared a rolled up patch for you, run time tested. It should protect all of the above pages from being evicted by large file copies.
The attached patch is for 2.6.29.
I applied the patch , but the number don seem to show much difference
The numbers listed in your email? No the most relevant numbers are memory and disk ones.
For my part, the number of mapped pages keeps stable when there is an ongoing background file copy:
% ls -l /b/sparse
-rw-r--r-- 1 root root 98T 2009-04-29 22:38 /b/sparse
% cp /b/sparse /dev/null&
% for i in seq 1 100; do grep Mapped /proc/meminfo; sleep 1; done
Mapped: 22764 kB Mapped: 22796 kB Mapped: 22756 kB Mapped: 22716 kB Mapped: 22800 kB Mapped: 22788 kB Mapped: 22804 kB Mapped: 22808 kB Mapped: 22748 kB Mapped: 22708 kB Mapped: 22916 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22832 kB Mapped: 22812 kB Mapped: 22812 kB Mapped: 22792 kB Mapped: 22772 kB Mapped: 22860 kB Mapped: 22860 kB Mapped: 22748 kB Mapped: 22808 kB Mapped: 22868 kB Mapped: 22956 kB Mapped: 22956 kB Mapped: 22832 kB Mapped: 22832 kB Mapped: 22980 kB Mapped: 22980 kB Mapped: 22980 kB Mapped: 22980 kB Mapped: 22872 kB Mapped: 22872 kB Mapped: 22900 kB Mapped: 22792 kB Mapped: 22920 kB Mapped: 22812 kB Mapped: 22812 kB Mapped: 22752 kB Mapped: 22864 kB Mapped: 22972 kB Mapped: 22860 kB Mapped: 22796 kB Mapped: 22900 kB Mapped: 22888 kB Mapped: 22888 kB Mapped: 22888 kB Mapped: 22764 kB
Quoted 5 lines
, following are the statistics with the patched kernel, could something else be causing the huge iowait, :
Linux 2.6.29 (satish) 05/17/2009
Quoted 7 lines
09:57:14 AM CPU %user %nice %system %iowait %steal %idle 09:57:18 AM all 12.93 0.00 10.25 33.44 0.00 43.38 09:57:21 AM all 13.32 0.00 28.09 24.40 0.00 34.19 09:57:24 AM all 10.79 0.00 4.76 75.56 0.00 8.89 09:57:26 AM all 11.46 0.00 8.01 35.64 0.00 44.90 09:57:30 AM all 11.68 0.00 8.88 35.05 0.00 44.39 Average: all 12.03 0.00 11.94 40.81 0.00 35.22
The iowait is high. Do you have "iostat -x 5' numbers?
Quoted 7 lines
and
hdparm -tT
/dev/sda
/dev/sda:
Timing cached reads: 1332 MB in 2.00 seconds = 666.35 MB/sec
Timing buffered disk reads: 24 MB in 3.32 seconds = 7.23 MB/sec
That's pretty slow numbers. On my laptop:
# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 10266 MB in 1.98 seconds = 5188.46 MB/sec
Timing buffered disk reads: 138 MB in 3.01 seconds = 45.90 MB/sec
Thanks,
Fengguang
Quoted 2 lines
The numbers listed in your email? No the most relevant numbers are memory and disk ones.
the numbers I earlier mentioned were the output of : "sar" for the number of mapped pages stays almost constant on my lappy too :
Mapped: 70840 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70724 kB Mapped: 70860 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70736 kB
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 9 lines · Satish Eerpini
Quoted 2 lines
The numbers listed in your email? No the most relevant numbers are memory and disk ones.
the numbers I earlier mentioned were the output of : "sar" for the number of mapped pages stays almost constant on my lappy too :
Mapped: 70840 kB Mapped: 70860 kB Mapped: 70736 kB Mapped: 70860 kB
Hmm, how do you define/feel the unresponsiveness? Is it related to the currently *running* applications, or opening a *new* application?
Thanks,
Fengguang
Quoted 2 lines
Hmm, how do you define/feel the unresponsiveness? Is it related to the currently *running* applications, or opening a *new* application?
the unresponsiveness is for both the currently running applications and for opening new applications, here are some pointers :
--> during normal operation, swtiching between windows( alt+tab) is simultaneous, while during a file copy it takes around 1 second, which I think is quite a lag.
--> Opening a simple application like gnome-terminal takes a second or more for sure when there is a file copy going on.
-->Also a particular window after gaining focus takes a second or so, to resume what it was doing, .. so if I am copying a file in a session of gnome-terminal and open another session of gnome-terminal and type in "clear" it takes around 2 seconds before the screen is cleared on the second session,
this is the case with almost every application and worser with big ones like firefox and office. This was the kind of unresponsiveness I was talking about.
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 6 lines · Satish Eerpini
Quoted 2 lines
Hmm, how do you define/feel the unresponsiveness? Is it related to the currently *running* applications, or opening a *new* application?
the unresponsiveness is for both the currently running applications and for opening new applications, here are some pointers :
Quoted 10 lines
--> during normal operation, swtiching between windows( alt+tab) is simultaneous, while during a file copy it takes around 1 second, which I think is quite a lag.
--> Opening a simple application like gnome-terminal takes a second or more for sure when there is a file copy going on.
-->Also a particular window after gaining focus takes a second or so, to resume what it was doing, .. so if I am copying a file in a session of gnome-terminal and open another session of gnome-terminal and type in "clear" it takes around 2 seconds before the screen is cleared on the second session,
The clear command can be a big clue. Can you try these commands in the terminal and show us the two log files?
strace -o all.log -T clear
strace -o sum.log -c clear
Quoted 8 lines
this is the case with almost every application and worser with big ones like firefox and office. This was the kind of unresponsiveness I was talking about.
Cheers
SatishSignature · 1 line
http://satish.playdrupal.com
Quoted 6 lines
The clear command can be a big clue. Can you try these commands in the terminal and show us the two log files?
strace -o all.log -T clear
strace -o sum.log -c clear
ok here go the logs :
"all.log"execve("/usr/bin/clear", ["clear"], [/* 47 vars */]) = 0 <0.000432>
brk(0) = 0x981e000 <0.000018>access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
directory) <0.000026>open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000027>fstat64(3, {st_mode=S_IFREG|0644, st_size=140850, ...}) = 0 <0.000019> mmap2(NULL, 140850, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f01000 <0.000023>
close(3) = 0 <0.000018>open("/lib/libtinfo.so.5", O_RDONLY) = 3 <0.000043>
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\332\355\0074\0\0\0D"...,
512) = 512 <0.000028>fstat64(3, {st_mode=S_IFREG|0755, st_size=98036, ...}) = 0 <0.000020> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7f00000 <0.000021>
mmap2(0x7ed8000, 99928, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ed8000 <0.000023> mmap2(0x7eee000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15) = 0x7eee000 <0.000030>
close(3) = 0 <0.000018>open("/lib/libc.so.6", O_RDONLY) = 3 <0.000029>
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@xk\0004\0\0\0P"...,
512) = 512 <0.000021>fstat64(3, {st_mode=S_IFREG|0755, st_size=1809672, ...}) = 0 <0.000019> mmap2(0x6a1000, 1521232, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6a1000 <0.000022>mmap2(0x80f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16e) = 0x80f000 <0.000032> mmap2(0x812000, 9808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x812000 <0.000026>
close(3) = 0 <0.000018>
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7eff000 <0.000024>
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7eff6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 <0.000019>
mprotect(0x80f000, 8192, PROT_READ) = 0 <0.000026>
mprotect(0x69d000, 4096, PROT_READ) = 0 <0.000023>
munmap(0xb7f01000, 140850) = 0 <0.000032>
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo
...}) = 0 <0.000023>
brk(0) = 0x981e000 <0.000019>
brk(0x983f000) = 0x983f000 <0.000022>
stat64("/root/.terminfo", 0xbfd20f24) = -1 ENOENT (No such file ordirectory) <0.000027>stat64("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000025>access("/etc/terminfo/x/xterm", R_OK) = -1 ENOENT (No such file or
directory) <0.000026>stat64("/usr/share/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000026>access("/usr/share/terminfo/x/xterm", R_OK) = 0 <0.000028> open("/usr/share/terminfo/x/xterm", O_RDONLY|O_LARGEFILE) = 3 <0.000031> read(3, "\32\0010\0&\0\17\0\235\1F\5xterm|xterm terminal "..., 4097) = 3220 <0.000035>
close(3) = 0 <0.000022>ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000025>ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000020>ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000021>ioctl(1, TIOCGWINSZ, {ws_row=24, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0 <0.000020>fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0 <0.000020> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7f23000 <0.000025>write(1, "\33[H\33[2J"..., 7) = 7 <0.000027>exit_group(0) = ?
"sum.log"
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
93.27 0.000485 485 1 execve 6.73 0.000035 7 5 ioctl 0.00 0.000000 0 3 read 0.00 0.000000 0 1 write 0.00 0.000000 0 4 open 0.00 0.000000 0 4 close 0.00 0.000000 0 3 2 access 0.00 0.000000 0 3 brk 0.00 0.000000 0 1 munmap 0.00 0.000000 0 2 mprotect 0.00 0.000000 0 9 mmap2 0.00 0.000000 0 3 1 stat64 0.00 0.000000 0 4 fstat64 0.00 0.000000 0 1 set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00 0.000520 44 3 total
I have also attached the logs, in case that is more useful .
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com
execve("/usr/bin/clear", ["clear"], [/* 47 vars */]) = 0 <0.000432>
brk(0) = 0x981e000 <0.000018>
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000026>
open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000027>fstat64(3, {st_mode=S_IFREG|0644, st_size=140850, ...}) = 0 <0.000019>mmap2(NULL, 140850, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f01000 <0.000023>
close(3) = 0 <0.000018>open("/lib/libtinfo.so.5", O_RDONLY) = 3 <0.000043>read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\332\355\0074\0\0\0D"..., 512) = 512 <0.000028>fstat64(3, {st_mode=S_IFREG|0755, st_size=98036, ...}) = 0 <0.000020>mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f00000 <0.000021>mmap2(0x7ed8000, 99928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ed8000 <0.000023>mmap2(0x7eee000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15) = 0x7eee000 <0.000030>
close(3) = 0 <0.000018>open("/lib/libc.so.6", O_RDONLY) = 3 <0.000029>read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@xk\0004\0\0\0P"..., 512) = 512 <0.000021>fstat64(3, {st_mode=S_IFREG|0755, st_size=1809672, ...}) = 0 <0.000019>mmap2(0x6a1000, 1521232, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6a1000 <0.000022>mmap2(0x80f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16e) = 0x80f000 <0.000032>mmap2(0x812000, 9808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x812000 <0.000026>
close(3) = 0 <0.000018>mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7eff000 <0.000024>
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7eff6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 <0.000019>
mprotect(0x80f000, 8192, PROT_READ) = 0 <0.000026>
mprotect(0x69d000, 4096, PROT_READ) = 0 <0.000023>
munmap(0xb7f01000, 140850) = 0 <0.000032>
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000023>
brk(0) = 0x981e000 <0.000019>
brk(0x983f000) = 0x983f000 <0.000022>
stat64("/root/.terminfo", 0xbfd20f24) = -1 ENOENT (No such file or directory) <0.000027>
stat64("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000025>
access("/etc/terminfo/x/xterm", R_OK) = -1 ENOENT (No such file or directory) <0.000026>stat64("/usr/share/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000026> access("/usr/share/terminfo/x/xterm", R_OK) = 0 <0.000028> open("/usr/share/terminfo/x/xterm", O_RDONLY|O_LARGEFILE) = 3 <0.000031>read(3, "\32\0010\0&\0\17\0\235\1F\5xterm|xterm terminal "..., 4097) = 3220 <0.000035>
close(3) = 0 <0.000022>ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000025> ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000020> ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 <0.000021> ioctl(1, TIOCGWINSZ, {ws_row=24, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0 <0.000020> fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0 <0.000020> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f23000 <0.000025>write(1, "\33[H\33[2J"..., 7) = 7 <0.000027>exit_group(0) = ?
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
93.27 0.000485 485 1 execve 6.73 0.000035 7 5 ioctl 0.00 0.000000 0 3 read 0.00 0.000000 0 1 write 0.00 0.000000 0 4 open 0.00 0.000000 0 4 close 0.00 0.000000 0 3 2 access 0.00 0.000000 0 3 brk 0.00 0.000000 0 1 munmap 0.00 0.000000 0 2 mprotect 0.00 0.000000 0 9 mmap2 0.00 0.000000 0 3 1 stat64 0.00 0.000000 0 4 fstat64 0.00 0.000000 0 1 set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00 0.000520 44 3 total
Quoted 8 lines · Satish Eerpini
Quoted 6 lines
The clear command can be a big clue. Can you try these commands in the terminal and show us the two log files?
strace -o all.log -T clear
strace -o sum.log -c clearok here go the logs :
It looks that the "clear" command runs pretty fast in itself. To be sure, can you run this?
strace -o log -tt clear
Or maybe the time is spent in gnome-terminal or bash? What if you change them to xterm and dash?
Quoted 30 lines
"sum.log"
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------93.27 0.000485 485 1 execve 6.73 0.000035 7 5 ioctl 0.00 0.000000 0 3 read 0.00 0.000000 0 1 write 0.00 0.000000 0 4 open 0.00 0.000000 0 4 close 0.00 0.000000 0 3 2 access 0.00 0.000000 0 3 brk 0.00 0.000000 0 1 munmap 0.00 0.000000 0 2 mprotect 0.00 0.000000 0 9 mmap2 0.00 0.000000 0 3 1 stat64 0.00 0.000000 0 4 fstat64 0.00 0.000000 0 1 set_thread_area------ ----------- ----------- --------- --------- ----------------
100.00 0.000520 44 3 totalI have also attached the logs, in case that is more useful .
Cheers
SatishSignature · 1 line
http://satish.playdrupal.com
Quoted 3 lines
Or maybe the time is spent in gnome-terminal or bash? What if you change them to xterm and dash?
ok here you go , I ran "strace -o log -tt clear " on xterm and dash :
"log "
18:31:02.162301 execve("/usr/bin/clear", ["clear"], [/* 49 vars */]) = 0
18:31:02.163558 brk(0) = 0x943b000
18:31:02.163668 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
18:31:02.163779 open("/etc/ld.so.cache", O_RDONLY) = 3 18:31:02.163858 fstat64(3, {st_mode=S_IFREG|0644, st_size=140850, ...}) = 0 18:31:02.163975 mmap2(NULL, 140850, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f16000
18:31:02.164036 close(3) = 0
18:31:02.164122 open("/lib/libtinfo.so.5", O_RDONLY) = 3 18:31:02.164206 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\332\355\0074\0\0\0D"...,
512) = 512
18:31:02.164303 fstat64(3, {st_mode=S_IFREG|0755, st_size=98036, ...}) = 0 18:31:02.164411 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f15000
18:31:02.164481 mmap2(0x7ed8000, 99928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ed8000
18:31:02.164545 mmap2(0x7eee000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15) = 0x7eee000
18:31:02.164620 close(3) = 0
18:31:02.164693 open("/lib/libc.so.6", O_RDONLY) = 3 18:31:02.164777 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@xk\0004\0\0\0P"...,
512) = 512
18:31:02.164862 fstat64(3, {st_mode=S_IFREG|0755, st_size=1809672, ...}) = 0 18:31:02.164972 mmap2(0x6a1000, 1521232, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6a1000
18:31:02.165035 mmap2(0x80f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16e) = 0x80f000 18:31:02.165123 mmap2(0x812000, 9808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x812000
18:31:02.165187 close(3) = 0
18:31:02.165265 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f14000
18:31:02.165327 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7f146c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 18:31:02.165412 mprotect(0x80f000, 8192, PROT_READ) = 0 18:31:02.165472 mprotect(0x69d000, 4096, PROT_READ) = 0 18:31:02.165532 munmap(0xb7f16000, 140850) = 0
18:31:02.165635 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
18:31:02.165816 brk(0) = 0x943b000
18:31:02.165870 brk(0x945c000) = 0x945c000
18:31:02.165966 stat64("/root/.terminfo", 0xbfc37e14) = -1 ENOENT (No such file or directory)
18:31:02.166040 stat64("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
18:31:02.166181 access("/etc/terminfo/x/xterm", R_OK) = -1 ENOENT (No such file or directory)
18:31:02.166256 stat64("/usr/share/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
18:31:02.166379 access("/usr/share/terminfo/x/xterm", R_OK) = 0 18:31:02.166454 open("/usr/share/terminfo/x/xterm", O_RDONLY|O_LARGEFILE) = 3 18:31:02.166531 read(3, "\32\0010\0&\0\17\0\235\1F\5xterm|xterm terminal "..., 4097) = 3220
18:31:02.166646 close(3) = 0
18:31:02.166709 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
18:31:02.166788 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
18:31:02.166866 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
18:31:02.166941 ioctl(1, TIOCGWINSZ, {ws_row=24, ws_col=80, ws_xpixel=499, ws_ypixel=316}) = 0
18:31:02.167010 fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
18:31:02.167134 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f38000
18:31:02.167205 write(1, "\33[H\33[2J"..., 7) = 7
18:31:02.167613 exit_group(0) = ?
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 6 lines · Satish Eerpini
Quoted 3 lines
Or maybe the time is spent in gnome-terminal or bash? What if you change them to xterm and dash?
ok here you go , I ran "strace -o log -tt clear " on xterm and dash :
"log "
Again, it's pretty fast:
begin with:
Quoted 1 line
18:31:02.162301
execve("/usr/bin/clear", ["clear"], [/* 49 vars */])= 0
end with:
Quoted 1 line
18:31:02.167613
exit_group(0)= ?
So the elapsed time for "clear" is about 5ms.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
ok here you go , I ran "strace -o log -tt clear " on xterm and dash :
"log "
Quoted 8 lines
Again, it's pretty fast:
begin with:Quoted 1 line
18:31:02.162301
execve("/usr/bin/clear", ["clear"], [/* 49 vars */])= 0end with:
Quoted 1 line
18:31:02.167613
exit_group(0)= ?So the elapsed time for "clear" is about 5ms.
right, I could not generate the lag in this case, as I said , I am not able to generate the same unresponsiveness deterministically ,
.......
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 93 lines · you
Quoted 6 lines · Satish Eerpini
Quoted 4 lines
I've prepared a rolled up patch for you, run time tested. It should protect all of the above pages from being evicted by large file copies.
The attached patch is for 2.6.29.
I applied the patch , but the number don seem to show much difference
The numbers listed in your email? No the most relevant numbers are memory and disk ones.
For my part, the number of mapped pages keeps stable when there is an ongoing background file copy:
% ls -l /b/sparse
-rw-r--r-- 1 root root 98T 2009-04-29 22:38 /b/sparse
% cp /b/sparse /dev/null&
% for i inseq 1 100; do grep Mapped/proc/meminfo; sleep 1; doneMapped: 22764 kB Mapped: 22796 kB Mapped: 22756 kB Mapped: 22716 kB Mapped: 22800 kB Mapped: 22788 kB Mapped: 22804 kB Mapped: 22808 kB Mapped: 22748 kB Mapped: 22708 kB Mapped: 22916 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22944 kB Mapped: 22832 kB Mapped: 22812 kB Mapped: 22812 kB Mapped: 22792 kB Mapped: 22772 kB Mapped: 22860 kB Mapped: 22860 kB Mapped: 22748 kB Mapped: 22808 kB Mapped: 22868 kB Mapped: 22956 kB Mapped: 22956 kB Mapped: 22832 kB Mapped: 22832 kB Mapped: 22980 kB Mapped: 22980 kB Mapped: 22980 kB Mapped: 22980 kB Mapped: 22872 kB Mapped: 22872 kB Mapped: 22900 kB Mapped: 22792 kB Mapped: 22920 kB Mapped: 22812 kB Mapped: 22812 kB Mapped: 22752 kB Mapped: 22864 kB Mapped: 22972 kB Mapped: 22860 kB Mapped: 22796 kB Mapped: 22900 kB Mapped: 22888 kB Mapped: 22888 kB Mapped: 22888 kB Mapped: 22764 kBQuoted 5 lines
, following are the statistics with the patched kernel, could something else be causing the huge iowait, :
Linux 2.6.29 (satish) 05/17/2009
Quoted 7 lines
09:57:14 AM CPU %user %nice %system %iowait %steal %idle 09:57:18 AM all 12.93 0.00 10.25 33.44 0.00 43.38 09:57:21 AM all 13.32 0.00 28.09 24.40 0.00 34.19 09:57:24 AM all 10.79 0.00 4.76 75.56 0.00 8.89 09:57:26 AM all 11.46 0.00 8.01 35.64 0.00 44.90 09:57:30 AM all 11.68 0.00 8.88 35.05 0.00 44.39 Average: all 12.03 0.00 11.94 40.81 0.00 35.22The iowait is high. Do you have "iostat -x 5' numbers?
Quoted 7 lines
and
hdparm -tT
/dev/sda
/dev/sda:
Timing cached reads: 1332 MB in 2.00 seconds = 666.35 MB/sec
Timing buffered disk reads: 24 MB in 3.32 seconds = 7.23 MB/sec
Quoted 8 lines
That's pretty slow numbers. On my laptop:
# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 10266 MB in 1.98 seconds = 5188.46 MB/sec
Timing buffered disk reads: 138 MB in 3.01 seconds = 45.90 MB/sec
I see (on 2.6.30-rc6, no patches):
# hdparm -tT /dev/sda (The internal, low-power, low performance hdd):
/dev/sda:
Timing cached reads: 492 MB in 2.00 seconds = 245.83 MB/sec
Timing buffered disk reads: 74 MB in 3.05 seconds = 24.26 MB/sec
# hdparm -tT /dev/sdb (A Class-6, SDHC card):
/dev/sdb:
Timing cached reads: 472 MB in 2.01 seconds = 235.28 MB/sec
Timing buffered disk reads: 40 MB in 3.13 seconds = 12.79 MB/sec
And the only thing going on -
(once you deduct the effect of getting this output) - nothing:
top - 08:09:25 up 9 min, 1 user, load average: 0.24, 0.40, 0.25
Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 2.9%us, 4.2%sy, 0.8%ni, 59.9%id, 32.2%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 447040k total, 242996k used, 204044k free, 73516k buffers Swap: 2199324k total, 0k used, 2199324k free, 79868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3359 root 20 0 2444 1048 824 R 3.8 0.2 0:00.04 top
3360 root 20 0 3376 888 744 S 1.9 0.2 0:00.01 less
1 root 20 0 3084 1884 564 S 0.0 0.4 0:01.59 init
2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreaddMike
Quoted 9 lines
Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
this may be the reason, I have seen this lag only with kernel > 2.6.28, currently I am running vanilla kernel 2.6.29.1 .
Satish
Signature · 1 line
http://satish.playdrupal.com
Quoted 5 lines
Which kernel version? If it's something recent, your programs may be getting swapped out due to a flaw in the latest kernels, where it accidentally prefers caching file data over mapped pages. If that's the case, Rik (cc:d) is working on patches to address the issue, and may appreciate another tester.
well I went through http://kerneltrap.org/node/3000 on kerneltrap , which I think is what you are pointing out here. Well I checked the "swappiness" parameter on my system and it is set to 60 , and bravely I changed it to '0', while the system appeared to have become a lot more responsive, I see no difference in the statistics:
Linux 2.6.29.1 (satish) 05/16/2009
07:49:43 PM CPU %user %nice %system %iowait %steal %idle
07:49:46 PM all 7.30 0.00 8.25 40.00
0.00 44.44
07:49:49 PM all 7.21 0.00 9.40 36.36
0.00 47.02
07:49:52 PM all 8.62 0.00 9.40 34.80
0.00 47.18
07:49:55 PM all 7.26 0.00 8.36 37.07
0.00 47.32
07:49:58 PM all 7.89 0.00 9.15 36.59
0.00 46.37Average: all 7.66 0.00 8.92
36.96 0.00 46.47
so there is still a huge IOWAIT , ..
Cheers
Satish
Signature · 1 line
http://satish.playdrupal.com