MM patches against 2.5.31
Subsystems (55)
This seems to have been missed:
Quoted 16 lines · Linus Torvalds
In article <[email protected]>,
Quoted 6 lines · Andrew Morton
What I'm inclined to do there is to change
__page_cache_release()to not attempt to free the page at all. Just let it sit on the LRU until page reclaim encounters it. With the anon-free-via-pagevec patch, very, very, very few pages actually get their final release in__page_cache_release()- zero on uniprocessor, I expect.If you do this, then I would personally suggest a conceptually different
approach: make the LRU list count towards the page count. That will _automatically_ result in what you describe - if a page is on the LRU list, then "freeing" it will always just decrement the count, and the _real_ free comes from walking the LRU list and considering count==1 to be trivially freeable.
Quoted 6 lines
That way you don't have to have separate functions for releasing different kinds of pages (we've seen how nasty that was from a maintainance standpoint already with the "
put_pagevspage_cache_release" thing).Ehh?
If every structure locks before removing its reference (ie before testing and/or removing a lru reference we take zone->lru_lock, for slabs take cachep->spinlock etc) Its a bit of an audit task to make sure the various locks are taken (and documented) though.
By leting the actual free be lazy as Linus suggests things should simplify nicely.
comments,
Ed Tomlinson
Quoted 2 lines · Ed Tomlinson
This seems to have been missed:
Still thinking about it.
Quoted 23 lines
Quoted 21 lines · Linus Torvalds
In article <[email protected]>,
Quoted 6 lines · Andrew Morton
What I'm inclined to do there is to change
__page_cache_release()to not attempt to free the page at all. Just let it sit on the LRU until page reclaim encounters it. With the anon-free-via-pagevec patch, very, very, very few pages actually get their final release in__page_cache_release()- zero on uniprocessor, I expect.If you do this, then I would personally suggest a conceptually different
approach: make the LRU list count towards the page count. That will _automatically_ result in what you describe - if a page is on the LRU list, then "freeing" it will always just decrement the count, and the _real_ free comes from walking the LRU list and considering count==1 to be trivially freeable.That way you don't have to have separate functions for releasing different kinds of pages (we've seen how nasty that was from a maintainance standpoint already with the "
put_pagevspage_cache_release" thing).
Quoted 8 lines
Quoted 1 line
Ehh?
If every structure locks before removing its reference (ie before testing and/or removing a lru reference we take
zone->lru_lock, for slabs takecachep->spinlocketc) Its a bit of an audit task to make sure the various locks are taken (and documented) though.By leting the actual free be lazy as Linus suggests things should simplify nicely.
Well we wouldn't want to leave tons of free pages on the LRU - the VM would needlessly reclaim pagecache before finding the free pages. And higher-order page allocations could suffer.
If we go for explicit lru removal in truncate and zap_pte_range then this approach may be best. Still thinking about it.
Quoted 3 lines · Andrew Morton
Well we wouldn't want to leave tons of free pages on the LRU - the VM would needlessly reclaim pagecache before finding the free pages. And higher-order page allocations could suffer.
We did this with the swap cache in 2.4.<7 and it was an absolute disaster.
regards,
Rik
Signature · 3 lines
Bravely reimplemented by the knights who say "NIH". http://www.surriel.com/ http://distro.conectiva.com/
slablru for 2.5.32-mm1
In reply to Andrew MortonHi Andrew
Here is slablru for 32-mm1. This is based on a version ported to 31ish-mm1. It should be stable. Its been booted as UP (32-mm1) and SMP on UP (31ish-mm1 only) and works as expected.
A typical test cycle involved:
find / -name "*" > /dev/null
edit a large tif with the gimp
run dbench a few times with the dbench dir on tmpfs (trying to use gimp too) run dbench a few times from a reiserfs dir (trying to use gimp too) use the box for news/mail, atp-get update/upgrade etc, wait a few hours and repeat
31ish-mm1 survived a day of this, 32-mm1 is sending this message after one cycle.
Andrew, what do you thing about adding slablru to your experimental dir?
There is also a version for virgin 2.5.32, anyone wanting it should email me - one big patch is eats enough bandwidth.
One interesting change in this version. We only add the first page of a slab to the lru. The reference bit setting logic for slabs has been modified to set the bit on the first page. Pagevec created a little bit of a problem for slablru. How do we know the order of the
slab page when its being freed? My solution is to use 3 bits in page->flags and save the
order there. Then free_pages_ok was modified to take the order from page->flags. This was implement in a minimal fashion. Think Wli is working on a more elaborate version of this - fleshed out, it could be used to support large pages in the vm.
Second topic.
I have also included an optimisation for vmscan. I found that the current code would reduce the inactive list to almost nothing when applications create large numbers of active pages very
quickly run (ie. gimp loading and editing large 20m+ tiffs). This reduces the problem. Always
allowing nr_pages to be scanned caused the active list to be reduced to almost nothing when something like gimp exited and we had another task adding lots to the inactive list. This is fixed here too. I do wonder if zone->refill_counter, as implemented, is a great idea. Do we really need/want to remember to scan the active list if it has massively decreased in size because some app exited? Maybe some sort of decay logic should be used...
Comments?
Ed Tomlinson
---------------
mm/vmscan.c +6 -3
@@ -492,11 +492,14 @@* active list. */ ratio = (unsigned long )nr_pages * zone->nr_active / - ((zone->nr_inactive |1 ) *2 ); -atomic_add (ratio+1 , &zone->refill_counter); -if (atomic_read (&zone->refill_counter) >SWAP_CLUSTER_MAX ) { + ((zone->nr_inactive |1 ) *2 ) +1 ; +atomic_add (ratio, &zone->refill_counter); + max_scan = nr_pages < ratio ? nr_pages : ratio; +while ((atomic_read (&zone->refill_counter) >SWAP_CLUSTER_MAX ) && + (max_scan >0 )) {atomic_sub (SWAP_CLUSTER_MAX , &zone->refill_counter);refill_inactive_zone (zone,SWAP_CLUSTER_MAX ); + max_scan -=SWAP_CLUSTER_MAX ; } max_scan = zone->nr_inactive / priority;
---------------
---------------
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.523 -> 1.524
# include/asm-generic/tlb.h 1.12 -> 1.13
# include/asm-s390x/tlb.h 1.3 -> 1.4
# fs/jfs/jfs_umount.c 1.3 -> 1.4
# include/linux/kernel.h 1.19 -> 1.20
# include/asm-ppc/pgalloc.h 1.8 -> 1.9
# include/asm-ppc/hardirq.h 1.12 -> 1.13
# fs/ext2/dir.c 1.15 -> 1.16
# include/asm-ppc/tlb.h 1.3 -> 1.4
# include/asm-ppc64/page.h 1.10 -> 1.11
# include/linux/pagemap.h 1.24 -> 1.25
# include/linux/mmzone.h 1.13 -> 1.14
# kernel/ksyms.c 1.122 -> 1.123
# include/asm-cris/cache.h 1.1 -> 1.2
# include/asm-ia64/pgalloc.h 1.13 -> 1.14
# include/linux/swap.h 1.53 -> 1.54
# include/asm-arm/tlb.h 1.3 -> 1.4
# include/linux/buffer_head.h 1.26 -> 1.27
# include/linux/ext3_fs.h 1.7 -> 1.8
# include/asm-i386/cache.h 1.1 -> 1.2
# include/asm-ppc64/pgalloc.h 1.6 -> 1.7
# arch/alpha/mm/numa.c 1.5 -> 1.6
# include/linux/mm.h 1.71 -> 1.72
# include/asm-i386/pgalloc.h 1.16 -> 1.17
# include/asm-mips64/mmzone.h 1.3 -> 1.4
# mm/highmem.c 1.30 -> 1.31
# mm/page_alloc.c 1.89 -> 1.90
# include/asm-s390/cache.h 1.3 -> 1.4
# include/asm-sparc/pgalloc.h 1.9 -> 1.10
# arch/arm/mm/init.c 1.15 -> 1.16
# include/asm-ia64/spinlock.h 1.4 -> 1.5
# include/linux/ext3_jbd.h 1.4 -> 1.5
# fs/ext3/Makefile 1.3 -> 1.4
# include/asm-i386/highmem.h 1.7 -> 1.8
# arch/i386/Config.help 1.12 -> 1.13
# include/linux/sched.h 1.84 -> 1.85
# include/linux/writeback.h 1.10 -> 1.11
# include/asm-ppc64/cache.h 1.3 -> 1.4
# fs/driverfs/inode.c 1.48 -> 1.49 # mm/page-writeback.c 1.32 -> 1.33 # fs/smbfs/inode.c 1.27 -> 1.28
# include/asm-s390x/cache.h 1.3 -> 1.4
# mm/vmscan.c 1.96 -> 1.97
# include/asm-x86_64/tlb.h 1.3 -> 1.4
# fs/proc/proc_misc.c 1.34 -> 1.35
# include/asm-s390x/pgalloc.h 1.6 -> 1.7
# include/asm-parisc/cache.h 1.1 -> 1.2
# include/linux/page-flags.h 1.16 -> 1.17
# fs/jfs/jfs_imap.c 1.9 -> 1.10
# include/asm-i386/page.h 1.15 -> 1.16
# arch/i386/kernel/Makefile 1.21 -> 1.22
# include/asm-sparc64/tlb.h 1.5 -> 1.6
# arch/sparc/mm/sun4c.c 1.16 -> 1.17
# arch/i386/mm/init.c 1.23 -> 1.24 # fs/minix/dir.c 1.11 -> 1.12 # fs/mpage.c 1.13 -> 1.14
# include/asm-i386/spinlock.h 1.7 -> 1.8
# include/linux/preempt.h 1.5 -> 1.6
# kernel/suspend.c 1.17 -> 1.18
# fs/jfs/jfs_metapage.c 1.15 -> 1.16
# include/linux/gfp.h 1.2 -> 1.3
# include/linux/fs.h 1.157 -> 1.158
# arch/i386/mm/Makefile 1.3 -> 1.4
# arch/mips64/sgi-ip27/ip27-memory.c 1.3 -> 1.4
# fs/sysv/dir.c 1.12 -> 1.13
# include/asm-i386/semaphore.h 1.5 -> 1.6
# include/asm-alpha/pgtable.h 1.13 -> 1.14
# arch/ppc/mm/fault.c 1.11 -> 1.12
# include/asm-i386/tlb.h 1.5 -> 1.6
# mm/numa.c 1.8 -> 1.9
# include/linux/backing-dev.h 1.1 -> 1.2
# mm/bootmem.c 1.11 -> 1.12
# include/asm-mips/cache.h 1.2 -> 1.3
# arch/i386/config.in 1.46 -> 1.47
# include/linux/rwsem.h 1.8 -> 1.9
# fs/reiserfs/inode.c 1.63 -> 1.64
# fs/ramfs/inode.c 1.24 -> 1.25
# arch/sparc/mm/fault.c 1.8 -> 1.9
# mm/filemap.c 1.127 -> 1.128
# include/asm-ppc64/tlb.h 1.2 -> 1.3
# fs/namei.c 1.54 -> 1.55
# include/asm-alpha/mmzone.h 1.1 -> 1.2
# arch/sparc64/mm/init.c 1.33 -> 1.34
# mm/msync.c 1.7 -> 1.8 # fs/nfsd/vfs.c 1.36 -> 1.37 # mm/swap.c 1.28 -> 1.29
# include/asm-alpha/cache.h 1.3 -> 1.4
# include/asm-m68k/sun3_pgalloc.h 1.6 -> 1.7
# include/asm-ia64/tlb.h 1.7 -> 1.8
# include/asm-i386/pci.h 1.15 -> 1.16
# mm/shmem.c 1.70 -> 1.71
# include/asm-alpha/tlb.h 1.2 -> 1.3
# include/asm-sparc/tlb.h 1.2 -> 1.3
# include/asm-ppc/highmem.h 1.7 -> 1.8
# mm/swap_state.c 1.41 -> 1.42 # fs/super.c 1.82 -> 1.83 # fs/fat/inode.c 1.42 -> 1.43
# include/asm-mips64/cache.h 1.2 -> 1.3
# include/asm-m68k/cache.h 1.2 -> 1.3
# fs/jfs/super.c 1.21 -> 1.22
# arch/cris/mm/init.c 1.9 -> 1.10
# include/asm-ia64/semaphore.h 1.4 -> 1.5
# fs/affs/file.c 1.20 -> 1.21
# include/asm-i386/io.h 1.16 -> 1.17
# arch/i386/kernel/setup.c 1.59 -> 1.60
# kernel/printk.c 1.13 -> 1.14
# include/linux/spinlock.h 1.18 -> 1.19
# include/asm-ia64/cache.h 1.3 -> 1.4
# fs/fs-writeback.c 1.18 -> 1.19
# include/asm-sparc64/cache.h 1.3 -> 1.4
# fs/jfs/jfs_logmgr.c 1.29 -> 1.30
# include/asm-x86_64/pgalloc.h 1.4 -> 1.5
# kernel/acct.c 1.12 -> 1.13
# include/linux/rwsem-spinlock.h 1.7 -> 1.8
# drivers/block/loop.c 1.54 -> 1.55
# fs/buffer.c 1.138 -> 1.139
# arch/sparc/mm/srmmu.c 1.19 -> 1.20
# fs/ext3/namei.c 1.20 -> 1.21 # fs/jfs/jfs_txnmgr.c 1.21 -> 1.22 # fs/jffs/inode-v23.c 1.35 -> 1.36 # drivers/block/rd.c 1.45 -> 1.46 # include/asm-s390/pgalloc.h 1.7 -> 1.8 # include/asm-i386/setup.h 1.1 -> 1.2 # include/asm-sparc/cache.h 1.2 -> 1.3 # fs/nfs/file.c 1.17 -> 1.18 # fs/Config.in 1.33 -> 1.34 # fs/ext3/inode.c 1.32 -> 1.33
# include/asm-sh/cache.h 1.2 -> 1.3
# fs/reiserfs/stree.c 1.31 -> 1.32
# include/asm-i386/pgtable.h 1.17 -> 1.18
# fs/nfs/inode.c 1.47 -> 1.48
# drivers/scsi/scsi_scan.c 1.21 -> 1.22
# arch/sh/mm/init.c 1.9 -> 1.10
# include/asm-s390/tlb.h 1.3 -> 1.4
# fs/jfs/jfs_dmap.c 1.8 -> 1.9
# include/asm-x86_64/cache.h 1.1 -> 1.2
# include/linux/bootmem.h 1.2 -> 1.3
# fs/smbfs/file.c 1.17 -> 1.18 # fs/ext3/super.c 1.29 -> 1.30
# include/asm-arm/memory.h 1.4 -> 1.5
# mm/rmap.c 1.10 -> 1.11
# include/asm-ppc64/mmzone.h 1.2 -> 1.3
# include/asm-m68k/tlb.h 1.3 -> 1.4
# include/asm-ppc/cache.h 1.5 -> 1.6
# drivers/net/ns83820.c 1.14 -> 1.15
# include/asm-mips64/pgtable.h 1.8 -> 1.9
# include/asm-arm/cache.h 1.1 -> 1.2
# include/linux/pagevec.h 1.2 -> 1.3
# fs/inode.c 1.67 -> 1.68
# include/asm-sparc/highmem.h 1.4 -> 1.5
# fs/jffs2/file.c 1.14 -> 1.15
# include/linux/dcache.h 1.14 -> 1.15
# include/asm-sparc/hardirq.h 1.5 -> 1.6
# arch/i386/mm/pgtable.c 1.2 -> 1.3
# include/asm-m68k/motorola_pgalloc.h 1.7 -> 1.8
# fs/reiserfs/tail_conversion.c 1.22 -> 1.23
# fs/udf/inode.c 1.21 -> 1.22
# include/linux/cache.h 1.4 -> 1.5
# (new) -> 1.1 fs/ext3/hash.c # (new) -> 1.1 include/asm-i386/max_numnodes.h # (new) -> 1.1 include/asm-i386/mmzone.h # (new) -> 1.1 include/asm-i386/numaq.h # (new) -> 1.1 arch/i386/kernel/numaq.c # (new) -> 1.1 arch/i386/mm/discontig.c # (new) -> 1.1 include/linux/mm_inline.h
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/08/28 [email protected] 1.524
# 2.5.32-mm1
# --------------------------------------------
#
arch/alpha/mm/numa.c +2 -2
@@ -294,7 +294,7 @@ zones_size[ZONE_DMA ] = dma_local_pfn; zones_size[ZONE_NORMAL ] = (end_pfn - start_pfn) - dma_local_pfn; } -free_area_init_node (nid,NODE_DATA (nid),NULL , zones_size, start_pfn<<PAGE_SHIFT ,NULL ); +free_area_init_node (nid,NODE_DATA (nid),NULL , zones_size, start_pfn,NULL ); lmax_mapnr =PLAT_NODE_DATA_STARTNR (nid) +PLAT_NODE_DATA_SIZE (nid);if (lmax_mapnr > max_mapnr) { max_mapnr = lmax_mapnr;
@@ -371,7 +371,7 @@ totalram_pages +=free_all_bootmem_node (NODE_DATA (nid)); lmem_map =NODE_MEM_MAP (nid); - pfn =NODE_DATA (nid)->node_start_paddr >>PAGE_SHIFT ; + pfn =NODE_DATA (nid)->node_start_pfn;for (i =0 ; i <PLAT_NODE_DATA_SIZE (nid); i++, pfn++)if (page_is_ram (pfn) &&PageReserved (lmem_map+i)) reservedpages++;
arch/arm/mm/init.c +1 -1
@@ -512,7 +512,7 @@arch_adjust_zones (node, zone_size, zhole_size);free_area_init_node (node, pgdat,0 , zone_size, - bdata->node_boot_start, zhole_size); + bdata->node_boot_start >>PAGE_SHIFT , zhole_size); }/*
arch/cris/mm/init.c +1 -1
@@ -345,7 +345,7 @@* mem_map page array. */ -free_area_init_node (0 ,0 ,0 , zones_size,PAGE_OFFSET ,0 ); +free_area_init_node (0 ,0 ,0 , zones_size,PAGE_OFFSET >>PAGE_SHIFT ,0 ); }
arch/i386/Config.help +1 -1
@@ -41,7 +41,7 @@ 486, 586, Pentiums, and various instruction-set-compatible chips by AMD, Cyrix, and others. -CONFIG_MULTIQUAD +CONFIG_X86_NUMAQ This option is used for getting Linux to run on a (IBM/Sequent) NUMA multiquad box. This changes the way that processors are bootstrapped, and uses Clustered Logical APIC addressing mode instead of Flat Logical.
arch/i386/config.in +16 -1
@@ -166,7 +166,22 @@ define_bool CONFIG_X86_IO_APIC y fi else - bool 'Multiquad NUMA system' CONFIG_MULTIQUAD + bool 'Multi-node NUMA system support' CONFIG_X86_NUMA + if [ "$CONFIG_X86_NUMA" = "y" ]; then + #Platform Choices + bool 'Multiquad (IBM/Sequent) NUMAQ support' CONFIG_X86_NUMAQ + if [ "$CONFIG_X86_NUMAQ" = "y" ]; then + define_bool CONFIG_MULTIQUAD y + fi + # Common NUMA Features + if [ "$CONFIG_X86_NUMAQ" = "y" ]; then + bool 'Numa Memory Allocation Support' CONFIG_NUMA + if [ "$CONFIG_NUMA" = "y" ]; then + define_bool CONFIG_DISCONTIGMEM y + define_bool CONFIG_HAVE_ARCH_BOOTMEM_NODE y + fi + fi + fi fi bool 'Machine Check Exception' CONFIG_X86_MCE
arch/i386/kernel/Makefile +1 -0
@@ -25,6 +25,7 @@ obj-$(CONFIG_X86_LOCAL_APIC) += mpparse.o apic.o nmi.o obj-$(CONFIG_X86_IO_APIC) += io_apic.o obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o +obj-$(CONFIG_X86_NUMAQ) += numaq.o ifdef CONFIG_VISWS obj-y += setup-visws.o obj-$(CONFIG_X86_VISWS_APIC) += visws_apic.o
arch/i386/kernel/numaq.c new file +145 -0
145 lines not on this page · raw message
arch/i386/kernel/setup.c +140 -133
341 lines not on this page · raw message
arch/i386/mm/Makefile +1 -0
@@ -10,6 +10,7 @@ O_TARGET := mm.o obj-y := init.o pgtable.o fault.o ioremap.o extable.o pageattr.o +obj-$(CONFIG_DISCONTIGMEM) += discontig.o export-objs := pageattr.o include $(TOPDIR)/Rules.make
arch/i386/mm/discontig.c new file +294 -0
294 lines not on this page · raw message
arch/i386/mm/init.c +48 -27
122 lines not on this page · raw message
arch/i386/mm/pgtable.c +10 -8
@@ -22,24 +22,26 @@void show_mem (void ) { -int i, total =0 , reserved =0 ; +int pfn, total =0 , reserved =0 ;int shared =0 , cached =0 ;int highmem =0 ; +struct page *page;printk ("Mem-info:\n" );show_free_areas ();printk ("Free swap: %6dkB\n" ,nr_swap_pages<<(PAGE_SHIFT -10 )); - i = max_mapnr; -while (i-- >0 ) { + pfn = max_mapnr; +while (pfn-- >0 ) { + page =pfn_to_page (pfn); total++; -if (PageHighMem (mem_map+i)) +if (PageHighMem (page)) highmem++; -if (PageReserved (mem_map+i)) +if (PageReserved (page)) reserved++; -else if (PageSwapCache (mem_map+i)) +else if (PageSwapCache (page)) cached++; -else if (page_count (mem_map+i)) - shared +=page_count (mem_map+i) -1 ; +else if (page_count (page)) + shared +=page_count (page) -1 ; }printk ("%d pages of RAM\n" , total);printk ("%d pages of HIGHMEM\n" ,highmem);
arch/mips64/sgi-ip27/ip27-memory.c +1 -1
@@ -253,7 +253,7 @@ zones_size[ZONE_DMA ] = end_pfn +1 - start_pfn;free_area_init_node (node,NODE_DATA (node),0 , zones_size, - start_pfn <<PAGE_SHIFT ,0 ); + start_pfn,0 );if ((PLAT_NODE_DATA_STARTNR (node) +PLAT_NODE_DATA_SIZE (node)) > pagenr) pagenr = PLAT_NODE_DATA_STARTNR(node) +
arch/ppc/mm/fault.c +1 -1
@@ -102,7 +102,7 @@#endif /* !CONFIG_4xx */ #endif /* CONFIG_XMON || CONFIG_KGDB */ -if (in_interrupt () || mm ==NULL ) { +if (in_atomic () || mm ==NULL ) {bad_page_fault (regs, address,SIGSEGV );return ; }
arch/sh/mm/init.c +2 -2
@@ -123,11 +123,11 @@ zones_size[ZONE_DMA ] = max_dma - start_pfn; zones_size[ZONE_NORMAL ] = low - max_dma; } -free_area_init_node (0 ,NODE_DATA (0 ),0 , zones_size, __MEMORY_START,0 ); +free_area_init_node (0 ,NODE_DATA (0 ),0 , zones_size, __MEMORY_START >>PAGE_SHIFT ,0 );#ifdef CONFIG_DISCONTIGMEM zones_size[ZONE_DMA ] = __MEMORY_SIZE_2ND >>PAGE_SHIFT ; zones_size[ZONE_NORMAL ] =0 ; -free_area_init_node (1 ,NODE_DATA (1 ),0 , zones_size, __MEMORY_START_2ND,0 ); +free_area_init_node (1 ,NODE_DATA (1 ),0 , zones_size, __MEMORY_START_2ND >>PAGE_SHIFT ,0 );#endif } }
arch/sparc/mm/fault.c +1 -1
@@ -233,7 +233,7 @@* If we're in an interrupt or have no user * context, we must not take the fault.. */ -if (in_interrupt () || !mm) +if (in_atomic () || !mm) goto no_context;down_read (&mm->mmap_sem);
arch/sparc/mm/srmmu.c +1 -1
@@ -1311,7 +1311,7 @@ zholes_size[ZONE_HIGHMEM ] = npages -calc_highpages ();free_area_init_node (0 ,NULL ,NULL , zones_size, - phys_base, zholes_size); + phys_base >>PAGE_SHIFT , zholes_size); }/* P3: easy to fix, todo. Current code is utterly broken, though. */
arch/sparc/mm/sun4c.c +1 -1
@@ -2074,7 +2074,7 @@ zholes_size[ZONE_HIGHMEM ] = npages -calc_highpages ();free_area_init_node (0 ,NULL ,NULL , zones_size, - phys_base, zholes_size); + phys_base >>PAGE_SHIFT , zholes_size); } cnt =0 ;
arch/sparc64/mm/init.c +1 -1
@@ -1559,7 +1559,7 @@ zholes_size[ZONE_DMA ] = npages - pages_avail;free_area_init_node (0 ,NULL ,NULL , zones_size, - phys_base, zholes_size); + phys_base >>PAGE_SHIFT , zholes_size); }device_scan ();
drivers/block/loop.c +3 -2
@@ -210,8 +210,7 @@ goto fail;if (aops->prepare_write (file, page, offset, offset+size)) goto unlock; - kaddr =page_address (page); -flush_dcache_page (page); + kaddr =kmap (page); transfer_result =lo_do_transfer (lo,WRITE , kaddr + offset, data, size,IV );if (transfer_result) {/*
@@ -221,6 +220,8 @@printk (KERN_ERR "loop: transfer error block %ld\n" , index);memset (kaddr + offset,0 , size); } +flush_dcache_page (page); +kunmap (page);if (aops->commit_write (file, page, offset, offset+size)) goto unlock;if (transfer_result)
drivers/block/rd.c +99 -113
369 lines not on this page · raw message
drivers/net/ns83820.c +1 -1
@@ -1081,7 +1081,7 @@ frag->page_offset, frag->size, PCI_DMA_TODEVICE);dprintk ("frag: buf=%08Lx page=%08lx offset=%08lx\n" , - (long long )buf, (long )(frag->page - mem_map), + (long long )buf, (long )page_to_pfn (frag->page), frag->page_offset); len = frag->size; frag++;
drivers/scsi/scsi_scan.c +6 -0
@@ -1379,6 +1379,12 @@printk (KERN_INFO "scsi: unknown device type %d\n" , sdev->type); } +/* +* scsi_alloc_sdev did this, but do it again because we can now set +* the bounce limit because the device type is known +*/ +scsi_initialize_merge_fn (sdev); + sdev->random = (sdev->type ==TYPE_TAPE ) ?0 :1 ;print_inquiry (inq_result);
fs/Config.in +1 -0
@@ -32,6 +32,7 @@ # dep_tristate ' Journal Block Device support (JBD for ext3)' CONFIG_JBD $CONFIG_EXT3_FS define_bool CONFIG_JBD $CONFIG_EXT3_FS dep_mbool ' JBD (ext3) debugging support' CONFIG_JBD_DEBUG $CONFIG_JBD +dep_mbool ' Ext3 hashed index (htree) support' CONFIG_EXT3_INDEX $CONFIG_JBD # msdos file systems tristate 'DOS FAT fs support' CONFIG_FAT_FS
fs/affs/file.c +9 -1
@@ -27,6 +27,7 @@#include <linux/fs.h> #include <linux/amigaffs.h> #include <linux/mm.h> +#include <linux/highmem.h> #include <linux/pagemap.h> #include <linux/buffer_head.h>
@@ -518,6 +519,7 @@pr_debug ("AFFS: read_page(%u, %ld, %d, %d)\n" , (u32 )inode->i_ino, page->index, from, to);if (from > to || to >PAGE_CACHE_SIZE )BUG (); +kmap (page); data =page_address (page); bsize =AFFS_SB (sb)->s_data_blksize; tmp = (page->index <<PAGE_CACHE_SHIFT ) + from;
@@ -537,6 +539,8 @@ from += tmp; boff =0 ; } +flush_dcache_page (page); +kunmap (page);return 0 ; }
@@ -656,7 +660,11 @@return err; }if (to <PAGE_CACHE_SIZE ) { -memset (page_address (page) + to,0 ,PAGE_CACHE_SIZE - to); +char *kaddr =kmap_atomic (page,KM_USER0 ); + +memset (kaddr + to,0 ,PAGE_CACHE_SIZE - to); +flush_dcache_page (page); +kunmap_atomic (kaddr,KM_USER0 );if (size > offset + to) {if (size < offset +PAGE_CACHE_SIZE ) tmp = size & ~PAGE_CACHE_MASK ;
fs/buffer.c +123 -53
419 lines not on this page · raw message
fs/driverfs/inode.c +15 -5
@@ -32,6 +32,7 @@#include <linux/namei.h> #include <linux/module.h> #include <linux/slab.h> +#include <linux/backing-dev.h> #include <linux/driverfs_fs.h> #include <asm/uaccess.h>
@@ -56,12 +57,19 @@static spinlock_t mount_lock =SPIN_LOCK_UNLOCKED ;static int mount_count =0 ; +static struct backing_dev_info driverfs_backing_dev_info = { + .ra_pages =0 ,/* No readahead */ + .memory_backed =1 ,/* Does not contribute to dirty memory */ +}; +static int driverfs_readpage (struct file *file,struct page * page) {if (!PageUptodate (page)) { -memset (kmap (page),0 ,PAGE_CACHE_SIZE ); -kunmap (page); +void *kaddr =kmap_atomic (page,KM_USER0 ); + +memset (kaddr,0 ,PAGE_CACHE_SIZE );flush_dcache_page (page); +kunmap_atomic (kaddr,KM_USER0 );SetPageUptodate (page); }unlock_page (page);
@@ -70,10 +78,12 @@static int driverfs_prepare_write (struct file *file,struct page *page,unsigned offset,unsigned to) { -void *addr =kmap (page);if (!PageUptodate (page)) { -memset (addr,0 ,PAGE_CACHE_SIZE ); +void *kaddr =kmap_atomic (page,KM_USER0 ); + +memset (kaddr,0 ,PAGE_CACHE_SIZE );flush_dcache_page (page); +kunmap_atomic (kaddr,KM_USER0 );SetPageUptodate (page); }return 0 ;
@@ -85,7 +95,6 @@loff_t pos = ((loff_t )page->index <<PAGE_CACHE_SHIFT ) + to;set_page_dirty (page); -kunmap (page);if (pos > inode->i_size) inode->i_size = pos;return 0 ;
@@ -105,6 +114,7 @@ inode->i_rdev =NODEV ; inode->i_atime = inode->i_mtime = inode->i_ctime =CURRENT_TIME ; inode->i_mapping->a_ops = &driverfs_aops; + inode->i_mapping->backing_dev_info = &driverfs_backing_dev_info;switch (mode &S_IFMT ) {default :init_special_inode (inode, mode, dev);
fs/ext2/dir.c +5 -6
@@ -571,8 +571,8 @@struct page *page =grab_cache_page (mapping,0 );unsigned chunk_size =ext2_chunk_size (inode);struct ext2_dir_entry_2 * de; -char *base;int err; +void *kaddr;if (!page)return -ENOMEM ;
@@ -581,22 +581,21 @@unlock_page (page); goto fail; } - base =page_address (page); - - de = (struct ext2_dir_entry_2 *) base; + kaddr =kmap_atomic (page,KM_USER0 ); + de = (struct ext2_dir_entry_2 *)kaddr; de->name_len =1 ; de->rec_len =cpu_to_le16 (EXT2_DIR_REC_LEN (1 ));memcpy (de->name,".\0\0" ,4 ); de->inode =cpu_to_le32 (inode->i_ino);ext2_set_de_type (de, inode); - de = (struct ext2_dir_entry_2 *) (base +EXT2_DIR_REC_LEN (1 )); + de = (struct ext2_dir_entry_2 *)(kaddr +EXT2_DIR_REC_LEN (1 )); de->name_len =2 ; de->rec_len =cpu_to_le16 (chunk_size -EXT2_DIR_REC_LEN (1 )); de->inode =cpu_to_le32 (parent->i_ino);memcpy (de->name,"..\0" ,4 );ext2_set_de_type (de, inode); - +kunmap_atomic (kaddr,KM_USER0 ); err =ext2_commit_chunk (page,0 , chunk_size); fail:page_cache_release (page);
fs/ext3/Makefile +1 -1
@@ -5,6 +5,6 @@ obj-$(CONFIG_EXT3_FS) += ext3.o ext3-objs := balloc.o bitmap.o dir.o file.o fsync.o ialloc.o inode.o \ - ioctl.o namei.o super.o symlink.o + ioctl.o namei.o super.o symlink.o hash.o include $(TOPDIR)/Rules.make
fs/ext3/hash.c new file +117 -0
117 lines not on this page · raw message
fs/ext3/inode.c +132 -48
272 lines not on this page · raw message
fs/ext3/namei.c +855 -105
556 lines not on this page · raw message
fs/fs-writeback.c +15 -27
42 lines not on this page · raw message
include/asm-generic/tlb.h +37 -4
41 lines not on this page · raw message
include/asm-i386/max_numnodes.h new file +12 -0
Lines of this file are not shown. View the raw message.
include/asm-i386/mmzone.h new file +78 -0
78 lines not on this page · raw message
include/asm-i386/numaq.h new file +177 -0
177 lines not on this page · raw message
include/linux/mmzone.h +47 -16
63 lines not on this page · raw message
include/linux/page-flags.h +15 -35
50 lines not on this page · raw message
include/linux/swap.h +3 -49
52 lines not on this page · raw message
mm/filemap.c +121 -35
156 lines not on this page · raw message
mm/page-writeback.c +69 -16
85 lines not on this page · raw message
mm/page_alloc.c +81 -55
136 lines not on this page · raw message
mm/swap.c +94 -59
153 lines not on this page · raw message
mm/swap_state.c +57 -23
80 lines not on this page · raw message
mm/vmscan.c +90 -62
152 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
---------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Part of the diff is not shown here. View the raw message.
Re: slablru for 2.5.32-mm1
Quoted 5 lines · Ed Tomlinson
Hi Andrew
Here is slablru for 32-mm1. This is based on a version ported to 31ish-mm1. It should be stable. Its been booted as UP (32-mm1) and SMP on UP (31ish-mm1 only) and works as expected.
Cool. But the diff adds tons of stuff which is already added by -mm1. I suspect you diffed against 2.5.31 base?
Quoted 10 lines
A typical test cycle involved:
find / -name "*" >/dev/null
edit a large tif with the gimp
run dbench a few times with the dbench dir on tmpfs (trying to use gimp too) run dbench a few times from a reiserfs dir (trying to use gimp too) use the box for news/mail, atp-get update/upgrade etc, wait a few hours and repeat31ish-mm1 survived a day of this, 32-mm1 is sending this message after one cycle.
Andrew, what do you thing about adding slablru to your experimental dir?
No probs.
Quoted 10 lines
There is also a version for virgin 2.5.32, anyone wanting it should email me - one big patch is eats enough bandwidth.
One interesting change in this version. We only add the first page of a slab to the lru. The reference bit setting logic for slabs has been modified to set the bit on the first page. Pagevec created a little bit of a problem for slablru. How do we know the order of the
slab page when its being freed? My solution is to use 3 bits inpage->flagsand save the
order there. Thenfree_pages_okwas modified to take the order frompage->flags. This was implement in a minimal fashion. Think Wli is working on a more elaborate version of this - fleshed out, it could be used to support large pages in the vm.
hm. What happened to the idea of walking mem_map[], looking for continuation pages? (This would need to be done via pfn_to_page(), I guess).
Quoted 11 lines
Second topic.
I have also included an optimisation for vmscan. I found that the current code would reduce the inactive list to almost nothing when applications create large numbers of active pages very
quickly run (ie. gimp loading and editing large 20m+ tiffs). This reduces the problem. Always
allowingnr_pagesto be scanned caused the active list to be reduced to almost nothing when something like gimp exited and we had another task adding lots to the inactive list. This is fixed here too. I do wonder ifzone->refill_counter, as implemented, is a great idea. Do we really need/want to remember to scan the active list if it has massively decreased in size because some app exited? Maybe some sort of decay logic should be used...
Well the refill counter thingy is just an optimisation: rather than calling refill_inacative() lots of times to just grab two or three pages, we wait until it builds up to 32, and then go deactivate 32 pages.
But ugh, it's a bit broken. Yup, you're right. Need to s/if/while/ in shrink_zone().
But we do need to slowly sift through the active list even when the inactive list is enormously bigger. Otherwise, completely dead pages will remain in-core forever if there's a lot of pagecache activity going on.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
Quoted 4 lines · Andrew Morton
But we do need to slowly sift through the active list even when the inactive list is enormously bigger. Otherwise, completely dead pages will remain in-core forever if there's a lot of pagecache activity going on.
Doesn't that just indicate we want to get rid of use-once and replace it with something slightly more predictable ?
regards,
Rik
Signature · 8 lines
Bravely reimplemented by the knights who say "NIH". http://www.surriel.com/ http://distro.conectiva.com/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
In reply to Ed TomlinsonEd,
I've taken a pass through this. It's nice and simple. Clever, too.
This is a first pass at changelogging it - I'd appreciate comments
on accuracy, things which should be added. Also a few questions and
comments for you, please.
Ed Tomlinson's patch which puts slab pages on the LRU.
- The patch is huge because it adds another argument to kmem_cache_create(): the address of that slab's application-specific "pruner" function.
- shrink_icache_memory(), shrink_dcache_memory() and shrink_dqcache_memory() are removed. We now have dcache, inode and dquot "pruner" functions which are called from the VM to trim the relevant cache.
- Ed originally had all slab pages on the LRU, including higher-order ones.
We ended up deciding to not do that - there are not really any interesting slab caches which use high-order allocations, and it didn't seem worth the (minor) compexity of managing higher-order pages on the LRU.
If, at some time in the future, the VM at large becomes aware of higher-order pages then we can bring this back. But it doesn't seem justifiable purely for slablru.
The patch which brings back the higher-order slabs is available.
Description of algorithm
========================
The pages which back slab objects may be manually marked as referenced via kmem_touch_page(), which simply sets PG_referenced. It _could_ use mark_page_accessed(), but doesn't. So slab pages will always remain on the inactive list. (what's the thinking here?)
kmem_touch_page() is only ever used when a new object is allocated within a page. It is not used in application-specific places such as dcache lookup. But I think it could be. (what's the thinking here?)
During page reclaim, when a slab page reaches the tail of the LRU we look to see if the page is referenced. If it is not referenced, has no pruner callback, and if it has no active slab entries then we directly reclaim the page. (Why don't we reclaim it even if it is referenced?)
If the page does have some active entries and it has a pruner callback, then we count them up and record the count within the slab structure.
So at the end of a shrink_cache() run, each slab has within it a record of how many live objects-within-pages were encountered at the tail of the LRU.
In shrink_zone(), after running shrink_cache(), we take a pass across all the slab caches, in kmem_do_prunes(). This will call out the the pruner functions and ask them to release a number of objects, where that number is the number of objects which were found at the tail of the LRU.
So the net effect is that the pruning pressure against a particular slab is driven by the number of objects which are encountered at the tail of the LRU.
Comments
========
Ed, this code can be sped up a bit, I think. We can make kmem_count_page() return a boolean back to shrink_cache(), telling it whether it needs to call kmem_do_prunes() at all. Often, there won't be any work to do in there, and taking that semaphore can be quite costly.
The code as-is will even run kmem_do_prunes() when we're examining ZONE_HIGHMEM, which certainly won't have any slab pages. This boolean will fix that too.
I reverted your
BUG_ON(smp_call_function(func, arg, 1, 1));
to
if (smp_call_function(func, arg, 1, 1))BUG();
because BUG_ONs should not have side-effects. Someone may want to compile a kernel which has a stubbed-out BUG_ON()...
__kmem_cache_shrink_locked() is calling kmem_slab_destroy() with local irqs disabled, which is a change from the previous behaviour. Is this deliberate?
The patch does a zillion BUG->BUG_ON conversions in slab.c, which is a bit unfortunate, because it makes it a bit confusing to review. Let's do that in a standalone patch next time ;)
arch/arm/mach-arc/mm.c | 4 arch/cris/drivers/usb-host.c | 2 arch/i386/mm/init.c | 2 drivers/block/ll_rw_blk.c | 2 drivers/ieee1394/eth1394.c | 2 drivers/ieee1394/ieee1394_core.c | 2 drivers/md/raid5.c | 2 drivers/scsi/scsi.c | 2 drivers/usb/host/uhci-hcd.c | 2 fs/adfs/super.c | 2 fs/affs/super.c | 2 fs/aio.c | 4 fs/bfs/inode.c | 1 fs/bio.c | 4 fs/block_dev.c | 4 fs/buffer.c | 2 fs/char_dev.c | 4 fs/coda/inode.c | 3 fs/dcache.c | 46 +---- fs/devfs/base.c | 2 fs/dnotify.c | 2 fs/dquot.c | 20 -- fs/efs/super.c | 1 fs/ext2/super.c | 2 fs/ext3/super.c | 2 fs/fat/inode.c | 1 fs/fcntl.c | 2 fs/freevxfs/vxfs_super.c | 5 fs/hfs/super.c | 1 fs/hpfs/super.c | 2 fs/inode.c | 33 +-- fs/intermezzo/dcache.c | 2 fs/isofs/inode.c | 2 fs/jbd/journal.c | 1 fs/jbd/revoke.c | 4 fs/jffs/inode-v23.c | 4 fs/jffs2/malloc.c | 14 - fs/jffs2/super.c | 2 fs/jfs/jfs_metapage.c | 2 fs/jfs/super.c | 2 fs/locks.c | 2 fs/minix/inode.c | 2 fs/namespace.c | 2 fs/ncpfs/inode.c | 2 fs/nfs/inode.c | 2 fs/nfs/pagelist.c | 2 fs/nfs/read.c | 2 fs/nfs/write.c | 2 fs/ntfs/super.c | 9 - fs/proc/inode.c | 2 fs/proc/proc_misc.c | 2 fs/qnx4/inode.c | 2 fs/reiserfs/super.c | 2 fs/romfs/inode.c | 2 fs/smbfs/inode.c | 2 fs/smbfs/request.c | 2 fs/sysv/inode.c | 2 fs/udf/super.c | 2 fs/ufs/super.c | 2 include/linux/dcache.h | 5 include/linux/page-flags.h | 1 include/linux/slab.h | 26 ++ kernel/fork.c | 12 - kernel/signal.c | 2 kernel/user.c | 2 lib/radix-tree.c | 2 mm/page_alloc.c | 1 mm/rmap.c | 1 mm/shmem.c | 2 mm/slab.c | 287 +++++++++++++++++++++------------ mm/swap.c | 18 +- mm/vmscan.c | 41 +++- net/atm/clip.c | 2 net/bluetooth/af_bluetooth.c | 2 net/core/neighbour.c | 2 net/core/skbuff.c | 2 net/core/sock.c | 2 net/decnet/dn_route.c | 2 net/decnet/dn_table.c | 2 net/ipv4/af_inet.c | 6 net/ipv4/fib_hash.c | 2 net/ipv4/inetpeer.c | 2 net/ipv4/ipmr.c | 2 net/ipv4/netfilter/ip_conntrack_core.c | 2 net/ipv4/route.c | 2 net/ipv4/tcp.c | 6 net/ipv6/af_inet6.c | 6 net/ipv6/ip6_fib.c | 2 net/ipv6/route.c | 2 net/socket.c | 2 net/unix/af_unix.c | 2 91 files changed, 401 insertions(+), 292 deletions(-)
arch/arm/mach-arc/mm.c renamed from arch/arm/mach-arc/mm.c~slablru +2 -2
@@ -173,13 +173,13 @@ void __init pgtable_cache_init(void) { pte_cache =kmem_cache_create ("pte-cache" ,sizeof (pte_t) *PTRS_PER_PTE , -0 ,0 , pte_cache_ctor,NULL ); +0 ,0 ,NULL , pte_cache_ctor,NULL );if (!pte_cache)BUG (); pgd_cache =kmem_cache_create ("pgd-cache" ,MEMC_TABLE_SIZE +sizeof (pgd_t) *PTRS_PER_PGD , -0 ,0 , pgd_cache_ctor,NULL ); +0 ,0 ,NULL , pgd_cache_ctor,NULL );if (!pgd_cache)BUG (); }
arch/cris/drivers/usb-host.c renamed from arch/cris/drivers/usb-host.c~slablru +1 -1
@@ -2330,7 +2330,7 @@ static int __init etrax_usb_hc_init(void hc =kmalloc (sizeof (etrax_hc_t),GFP_KERNEL );/* We use kmem_cache_* to make sure that all DMA desc. are dword aligned */ - usb_desc_cache =kmem_cache_create ("usb_desc_cache" ,sizeof (USB_EP_Desc_t),0 ,0 ,0 ,0 ); + usb_desc_cache =kmem_cache_create ("usb_desc_cache" ,sizeof (USB_EP_Desc_t),0 ,0 ,NULL ,NULL ,NULL );if (!usb_desc_cache) {panic ("USB Desc Cache allocation failed !!!\n" ); }
arch/i386/mm/init.c renamed from arch/i386/mm/init.c~slablru +1 -1
@@ -504,7 +504,7 @@ void __init pgtable_cache_init(void)* PAE pgds must be 16-byte aligned: */ pae_pgd_cachep =kmem_cache_create ("pae_pgd" ,32 ,0 , -SLAB_HWCACHE_ALIGN |SLAB_MUST_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN |SLAB_MUST_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!pae_pgd_cachep)panic ("init_pae(): Cannot alloc pae_pgd SLAB cache" ); }
drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~slablru +1 -1
@@ -2052,7 +2052,7 @@ int __init blk_dev_init(void) request_cachep =kmem_cache_create ("blkdev_requests" ,sizeof (struct request ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!request_cachep)panic ("Can't create request pool slab cache\n" );
drivers/ieee1394/eth1394.c renamed from drivers/ieee1394/eth1394.c~slablru +1 -1
@@ -715,7 +715,7 @@ static struct hpsb_highlevel_ops hl_opsstatic int __init ether1394_init_module (void ) { packet_task_cache =kmem_cache_create ("packet_task" ,sizeof (struct packet_task ), -0 ,0 ,NULL ,NULL ); +0 ,0 ,NULL ,NULL ,NULL );/* Register ourselves as a highlevel driver */ hl_handle =hpsb_register_highlevel (ETHER1394_DRIVER_NAME , &hl_ops);
drivers/ieee1394/ieee1394_core.c renamed from drivers/ieee1394/ieee1394_core.c~slablru +1 -1
@@ -971,7 +971,7 @@ struct proc_dir_entry *ieee1394_procfs_estatic int __init ieee1394_init (void ) { hpsb_packet_cache =kmem_cache_create ("hpsb_packet" ,sizeof (struct hpsb_packet ), -0 ,0 ,NULL ,NULL ); +0 ,0 ,NULL ,NULL ,NULL ); ieee1394_devfs_handle =devfs_mk_dir (NULL ,"ieee1394" ,NULL );
drivers/md/raid5.c renamed from drivers/md/raid5.c~slablru +1 -1
@@ -277,7 +277,7 @@ static int grow_stripes(raid5_conf_t *co sc =kmem_cache_create (conf->cache_name,sizeof (struct stripe_head )+(devs-1 )*sizeof (struct r5dev ), -0 ,0 ,NULL ,NULL ); +0 ,0 ,NULL ,NULL ,NULL );if (!sc)return 1 ; conf->slab_cache = sc;
drivers/scsi/scsi.c renamed from drivers/scsi/scsi.c~slablru +1 -1
@@ -2532,7 +2532,7 @@ static int __init init_scsi(void)struct scsi_host_sg_pool *sgp = scsi_sg_pools + i;int size = sgp->size *sizeof (struct scatterlist ); - sgp->slab =kmem_cache_create (sgp->name, size,0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); + sgp->slab =kmem_cache_create (sgp->name, size,0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!sgp->slab)panic ("SCSI: can't init sg slab\n" );
drivers/usb/host/uhci-hcd.c renamed from drivers/usb/host/uhci-hcd.c~slablru +1 -1
@@ -2512,7 +2512,7 @@ static int __init uhci_hcd_init(void)#endif uhci_up_cachep =kmem_cache_create ("uhci_urb_priv" , -sizeof (struct urb_priv ),0 ,0 ,NULL ,NULL ); +sizeof (struct urb_priv ),0 ,0 ,NULL ,NULL ,NULL );if (!uhci_up_cachep) goto up_failed;
fs/adfs/super.c renamed from fs/adfs/super.c~slablru +1 -1
@@ -234,7 +234,7 @@ static int init_inodecache(void) adfs_inode_cachep =kmem_cache_create ("adfs_inode_cache" ,sizeof (struct adfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (adfs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/affs/super.c renamed from fs/affs/super.c~slablru +1 -1
@@ -117,7 +117,7 @@ static int init_inodecache(void) affs_inode_cachep =kmem_cache_create ("affs_inode_cache" ,sizeof (struct affs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (affs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/aio.c renamed from fs/aio.c~slablru +2 -2
@@ -65,12 +65,12 @@ LIST_HEAD(fput_head);static int __init aio_setup (void ) { kiocb_cachep =kmem_cache_create ("kiocb" ,sizeof (struct kiocb ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!kiocb_cachep)panic ("unable to create kiocb cache\n" ); kioctx_cachep =kmem_cache_create ("kioctx" ,sizeof (struct kioctx ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!kioctx_cachep)panic ("unable to create kioctx cache" );
fs/bfs/inode.c renamed from fs/bfs/inode.c~slablru +1 -0
@@ -240,6 +240,7 @@ static int init_inodecache(void) bfs_inode_cachep =kmem_cache_create ("bfs_inode_cache" ,sizeof (struct bfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , + age_icache_memory, init_once,NULL );if (bfs_inode_cachep ==NULL )return -ENOMEM ;
fs/bio.c renamed from fs/bio.c~slablru +2 -2
@@ -470,7 +470,7 @@ static void __init biovec_init_pool(void bp->size, size); bp->slab =kmem_cache_create (bp->name, size,0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!bp->slab)panic ("biovec: can't init slab cache\n" ); bp->pool = mempool_create(BIO_POOL_SIZE, slab_pool_alloc,
@@ -484,7 +484,7 @@ static void __init biovec_init_pool(voidstatic int __init init_bio (void ) { bio_slab =kmem_cache_create ("bio" ,sizeof (struct bio ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!bio_slab)panic ("bio: can't create slab cache\n" ); bio_pool =mempool_create (BIO_POOL_SIZE , slab_pool_alloc, slab_pool_free, bio_slab);
fs/block_dev.c renamed from fs/block_dev.c~slablru +2 -2
@@ -249,8 +249,8 @@ void __init bdev_cache_init(void) bdev_cachep =kmem_cache_create ("bdev_cache" ,sizeof (struct block_device ), -0 ,SLAB_HWCACHE_ALIGN , init_once, -NULL ); +0 ,SLAB_HWCACHE_ALIGN , +NULL , init_once,NULL );if (!bdev_cachep)panic ("Cannot create bdev_cache SLAB cache" ); err =register_filesystem (&bd_type);
fs/buffer.c renamed from fs/buffer.c~slablru +1 -1
@@ -2645,7 +2645,7 @@ void __init buffer_init(void) bh_cachep =kmem_cache_create ("buffer_head" ,sizeof (struct buffer_head ),0 , -0 , init_buffer_head,NULL ); +0 ,NULL , init_buffer_head,NULL ); bh_mempool =mempool_create (MAX_UNUSED_BUFFERS , bh_mempool_alloc, bh_mempool_free,NULL );for (i =0 ; i <ARRAY_SIZE (bh_wait_queue_heads); i++)
fs/char_dev.c renamed from fs/char_dev.c~slablru +2 -2
@@ -46,8 +46,8 @@ void __init cdev_cache_init(void) cdev_cachep =kmem_cache_create ("cdev_cache" ,sizeof (struct char_device ), -0 ,SLAB_HWCACHE_ALIGN , init_once, -NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL , + init_once,NULL );if (!cdev_cachep)panic ("Cannot create cdev_cache SLAB cache" ); }
fs/coda/inode.c renamed from fs/coda/inode.c~slablru +2 -1
@@ -73,9 +73,10 @@ int coda_init_inodecache(void) coda_inode_cachep =kmem_cache_create ("coda_inode_cache" ,sizeof (struct coda_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (coda_inode_cachep ==NULL )return -ENOMEM ; +return 0 ; }
fs/dcache.c renamed from fs/dcache.c~slablru +16 -30
121 lines not on this page · raw message
fs/devfs/base.c renamed from fs/devfs/base.c~slablru +1 -1
@@ -3455,7 +3455,7 @@ static int __init init_devfs_fs (void) DEVFS_NAME, DEVFS_VERSION); devfsd_buf_cache =kmem_cache_create ("devfsd_event" ,sizeof (struct devfsd_buf_entry ), -0 ,0 ,NULL ,NULL ); +0 ,0 ,NULL ,NULL ,NULL );if (!devfsd_buf_cache)OOPS ("(): unable to allocate event slab\n" );#ifdef CONFIG_DEVFS_DEBUG devfs_debug = devfs_debug_init;
fs/dnotify.c renamed from fs/dnotify.c~slablru +1 -1
@@ -149,7 +149,7 @@ void __inode_dir_notify(struct inode *instatic int __init dnotify_init (void ) { dn_cache =kmem_cache_create ("dnotify cache" , -sizeof (struct dnotify_struct ),0 ,0 ,NULL ,NULL ); +sizeof (struct dnotify_struct ),0 ,0 ,NULL ,NULL ,NULL );if (!dn_cache)panic ("cannot create dnotify slab cache" );return 0 ;
fs/dquot.c renamed from fs/dquot.c~slablru +6 -14
@@ -480,26 +480,18 @@ static void prune_dqcache(int count)/* * This is called from kswapd when we think we need some -* more memory, but aren't really sure how much. So we -* carefully try to free a _bit_ of our dqcache, but not -* too much. -* -* Priority: -* 1 - very urgent: shrink everything -* ... -* 6 - base-level: try to shrink a bit. +* more memory */ -int shrink_dqcache_memory (int priority,unsigned int gfp_mask) +int age_dqcache_memory (kmem_cache_t *cachep,int entries,int gfp_mask) { -int count =0 ; +if (entries > dqstats.free_dquots) + entries = dqstats.free_dquots;lock_kernel (); - count = dqstats.free_dquots / priority; -prune_dqcache (count); +prune_dqcache (entries);unlock_kernel (); -kmem_cache_shrink (dquot_cachep); -return 0 ; +return entries; }/*
fs/efs/super.c renamed from fs/efs/super.c~slablru +1 -0
@@ -58,6 +58,7 @@ static int init_inodecache(void) efs_inode_cachep =kmem_cache_create ("efs_inode_cache" ,sizeof (struct efs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , + age_icache_memory, init_once,NULL );if (efs_inode_cachep ==NULL )return -ENOMEM ;
fs/ext2/super.c renamed from fs/ext2/super.c~slablru +1 -1
@@ -181,7 +181,7 @@ static int init_inodecache(void) ext2_inode_cachep =kmem_cache_create ("ext2_inode_cache" ,sizeof (struct ext2_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (ext2_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/ext3/super.c renamed from fs/ext3/super.c~slablru +1 -1
@@ -480,7 +480,7 @@ static int init_inodecache(void) ext3_inode_cachep =kmem_cache_create ("ext3_inode_cache" ,sizeof (struct ext3_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (ext3_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/fat/inode.c renamed from fs/fat/inode.c~slablru +1 -0
@@ -597,6 +597,7 @@ int __init fat_init_inodecache(void) fat_inode_cachep =kmem_cache_create ("fat_inode_cache" ,sizeof (struct msdos_inode_info ),0 ,SLAB_HWCACHE_ALIGN , + age_icache_memory, init_once,NULL );if (fat_inode_cachep ==NULL )return -ENOMEM ;
fs/fcntl.c renamed from fs/fcntl.c~slablru +1 -1
@@ -559,7 +559,7 @@ void kill_fasync(struct fasync_struct **static int __init fasync_init (void ) { fasync_cache =kmem_cache_create ("fasync_cache" , -sizeof (struct fasync_struct ),0 ,0 ,NULL ,NULL ); +sizeof (struct fasync_struct ),0 ,0 ,NULL ,NULL ,NULL );if (!fasync_cache)panic ("cannot create fasync slab cache" );return 0 ;
fs/freevxfs/vxfs_super.c renamed from fs/freevxfs/vxfs_super.c~slablru +3 -2
@@ -246,9 +246,10 @@ static int __initvxfs_init (void) { vxfs_inode_cachep =kmem_cache_create ("vxfs_inode" , -sizeof (struct vxfs_inode_info ),0 ,0 ,NULL ,NULL ); -if (vxfs_inode_cachep) +sizeof (struct vxfs_inode_info ),0 ,0 , age_icache_memory,NULL ,NULL ); +if (vxfs_inode_cachep) {return (register_filesystem (&vxfs_fs_type)); + }return -ENOMEM ; }
fs/hfs/super.c renamed from fs/hfs/super.c~slablru +1 -0
@@ -72,6 +72,7 @@ static int init_inodecache(void) hfs_inode_cachep =kmem_cache_create ("hfs_inode_cache" ,sizeof (struct hfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , + age_icache_memory, init_once,NULL );if (hfs_inode_cachep ==NULL )return -ENOMEM ;
fs/hpfs/super.c renamed from fs/hpfs/super.c~slablru +1 -1
@@ -186,7 +186,7 @@ static int init_inodecache(void) hpfs_inode_cachep =kmem_cache_create ("hpfs_inode_cache" ,sizeof (struct hpfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (hpfs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/inode.c renamed from fs/inode.c~slablru +12 -21
@@ -388,10 +388,11 @@ void prune_icache(int goal) count =0 ; entry = inode_unused.prev; -while (entry != &inode_unused) - { +for (; goal; goal--) {struct list_head *tmp = entry; +if (entry == &inode_unused) +break ; entry = entry->prev; inode =INODE (tmp);if (inode->i_state & (I_FREEING |I_CLEAR |I_LOCK ))
@@ -405,8 +406,6 @@ void prune_icache(int goal)list_add (tmp, freeable); inode->i_state |=I_FREEING ; count++; -if (!--goal) -break ; } inodes_stat.nr_unused -= count;spin_unlock (&inode_lock);
@@ -416,19 +415,10 @@ void prune_icache(int goal)/* * This is called from kswapd when we think we need some -* more memory, but aren't really sure how much. So we -* carefully try to free a _bit_ of our icache, but not -* too much. -* -* Priority: -* 1 - very urgent: shrink everything -* ... -* 6 - base-level: try to shrink a bit. +* more memory. */ -int shrink_icache_memory (int priority,int gfp_mask) +int age_icache_memory (kmem_cache_t *cachep,int entries,int gfp_mask) { -int count =0 ; -/* * Nasty deadlock avoidance.. *
@@ -439,12 +429,13 @@ int shrink_icache_memory(int priority, iif (!(gfp_mask & __GFP_FS))return 0 ; - count = inodes_stat.nr_unused / priority; +if (entries > inodes_stat.nr_unused) + entries = inodes_stat.nr_unused; -prune_icache (count); -kmem_cache_shrink (inode_cachep); -return 0 ; +prune_icache (entries); +return entries; } +EXPORT_SYMBOL (age_icache_memory);/* * Called with the inode lock held.
@@ -1103,8 +1094,8 @@ void __init inode_init(unsigned long mem/* inode slab cache */ inode_cachep =kmem_cache_create ("inode_cache" ,sizeof (struct inode ), -0 ,SLAB_HWCACHE_ALIGN , init_once, -NULL ); +0 ,SLAB_HWCACHE_ALIGN , age_icache_memory, + init_once,NULL );if (!inode_cachep)panic ("cannot create inode slab cache" ); }
fs/intermezzo/dcache.c renamed from fs/intermezzo/dcache.c~slablru +1 -1
@@ -127,7 +127,7 @@ void presto_init_ddata_cache(void)kmem_cache_create ("presto_cache" ,sizeof (struct presto_dentry_data ),0 ,SLAB_HWCACHE_ALIGN ,NULL , -NULL ); +NULL ,NULL );EXIT ; }
fs/isofs/inode.c renamed from fs/isofs/inode.c~slablru +1 -1
@@ -111,7 +111,7 @@ static int init_inodecache(void) isofs_inode_cachep =kmem_cache_create ("isofs_inode_cache" ,sizeof (struct iso_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (isofs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/jbd/journal.c renamed from fs/jbd/journal.c~slablru +1 -0
@@ -1569,6 +1569,7 @@ static int journal_init_journal_head_cacsizeof (struct journal_head ),0 ,/* offset */ 0 ,/* flags */ +NULL ,/* pruner */ NULL ,/* ctor */ NULL); /* dtor */ retval =0 ;
fs/jbd/revoke.c renamed from fs/jbd/revoke.c~slablru +2 -2
@@ -163,13 +163,13 @@ int __init journal_init_revoke_caches(vo { revoke_record_cache =kmem_cache_create ("revoke_record" ,sizeof (struct jbd_revoke_record_s ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (revoke_record_cache ==0 )return -ENOMEM ; revoke_table_cache =kmem_cache_create ("revoke_table" ,sizeof (struct jbd_revoke_table_s ), -0 ,0 ,NULL ,NULL ); +0 ,0 ,NULL ,NULL ,NULL );if (revoke_table_cache ==0 ) {kmem_cache_destroy (revoke_record_cache); revoke_record_cache =NULL ;
fs/jffs2/malloc.c renamed from fs/jffs2/malloc.c~slablru +7 -7
@@ -37,43 +37,43 @@ int __init jffs2_create_slab_caches(void { full_dnode_slab =kmem_cache_create ("jffs2_full_dnode" ,sizeof (struct jffs2_full_dnode ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (!full_dnode_slab) goto err; raw_dirent_slab =kmem_cache_create ("jffs2_raw_dirent" ,sizeof (struct jffs2_raw_dirent ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (!raw_dirent_slab) goto err; raw_inode_slab =kmem_cache_create ("jffs2_raw_inode" ,sizeof (struct jffs2_raw_inode ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (!raw_inode_slab) goto err; tmp_dnode_info_slab =kmem_cache_create ("jffs2_tmp_dnode" ,sizeof (struct jffs2_tmp_dnode_info ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (!tmp_dnode_info_slab) goto err; raw_node_ref_slab =kmem_cache_create ("jffs2_raw_node_ref" ,sizeof (struct jffs2_raw_node_ref ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (!raw_node_ref_slab) goto err; node_frag_slab =kmem_cache_create ("jffs2_node_frag" ,sizeof (struct jffs2_node_frag ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (!node_frag_slab) goto err; inode_cache_slab =kmem_cache_create ("jffs2_inode_cache" ,sizeof (struct jffs2_inode_cache ), -0 ,JFFS2_SLAB_POISON ,NULL ,NULL ); +0 ,JFFS2_SLAB_POISON ,NULL ,NULL ,NULL );if (inode_cache_slab)return 0 ; err:
fs/jffs2/super.c renamed from fs/jffs2/super.c~slablru +1 -1
@@ -299,7 +299,7 @@ static int __init init_jffs2_fs(void) jffs2_inode_cachep =kmem_cache_create ("jffs2_i" ,sizeof (struct jffs2_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - jffs2_i_init_once,NULL ); + age_icache_memory, jffs2_i_init_once,NULL );if (!jffs2_inode_cachep) {printk (KERN_ERR "JFFS2 error: Failed to initialise inode cache\n" );return -ENOMEM ;
fs/jffs/inode-v23.c renamed from fs/jffs/inode-v23.c~slablru +2 -2
@@ -1799,9 +1799,9 @@ init_jffs_fs(void) jffs_proc_root =proc_mkdir ("jffs" , proc_root_fs);#endif fm_cache =kmem_cache_create ("jffs_fm" ,sizeof (struct jffs_fm ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL ); node_cache =kmem_cache_create ("jffs_node" ,sizeof (struct jffs_node ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );return register_filesystem (&jffs_fs_type); }
fs/jfs/super.c renamed from fs/jfs/super.c~slablru +1 -1
@@ -457,7 +457,7 @@ static int __init init_jfs_fs(void) jfs_inode_cachep =kmem_cache_create ("jfs_ip" ,sizeof (struct jfs_inode_info ),0 ,0 , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (jfs_inode_cachep ==NULL )return -ENOMEM ;
fs/locks.c renamed from fs/locks.c~slablru +1 -1
@@ -1892,7 +1892,7 @@ int lock_may_write(struct inode *inode,static int __init filelock_init (void ) { filelock_cache =kmem_cache_create ("file_lock_cache" , -sizeof (struct file_lock ),0 ,0 , init_once,NULL ); +sizeof (struct file_lock ),0 ,0 ,NULL , init_once,NULL );if (!filelock_cache)panic ("cannot create file lock slab cache" );return 0 ;
fs/minix/inode.c renamed from fs/minix/inode.c~slablru +1 -1
@@ -79,7 +79,7 @@ static int init_inodecache(void) minix_inode_cachep =kmem_cache_create ("minix_inode_cache" ,sizeof (struct minix_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (minix_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/namespace.c renamed from fs/namespace.c~slablru +1 -1
@@ -1046,7 +1046,7 @@ void __init mnt_init(unsigned long mempaint i; mnt_cache =kmem_cache_create ("mnt_cache" ,sizeof (struct vfsmount ), -0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!mnt_cache)panic ("Cannot create vfsmount cache" );
fs/ncpfs/inode.c renamed from fs/ncpfs/inode.c~slablru +1 -1
@@ -69,7 +69,7 @@ static int init_inodecache(void) ncp_inode_cachep =kmem_cache_create ("ncp_inode_cache" ,sizeof (struct ncp_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (ncp_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/nfs/inode.c renamed from fs/nfs/inode.c~slablru +1 -1
@@ -1317,7 +1317,7 @@ int nfs_init_inodecache(void) nfs_inode_cachep =kmem_cache_create ("nfs_inode_cache" ,sizeof (struct nfs_inode ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (nfs_inode_cachep ==NULL )return -ENOMEM ;
fs/nfs/pagelist.c renamed from fs/nfs/pagelist.c~slablru +1 -1
@@ -492,7 +492,7 @@ int nfs_init_nfspagecache(void) nfs_page_cachep =kmem_cache_create ("nfs_page" ,sizeof (struct nfs_page ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (nfs_page_cachep ==NULL )return -ENOMEM ;
fs/nfs/read.c renamed from fs/nfs/read.c~slablru +1 -1
@@ -497,7 +497,7 @@ int nfs_init_readpagecache(void) nfs_rdata_cachep =kmem_cache_create ("nfs_read_data" ,sizeof (struct nfs_read_data ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (nfs_rdata_cachep ==NULL )return -ENOMEM ;
fs/nfs/write.c renamed from fs/nfs/write.c~slablru +1 -1
@@ -1317,7 +1317,7 @@ int nfs_init_writepagecache(void) nfs_wdata_cachep =kmem_cache_create ("nfs_write_data" ,sizeof (struct nfs_write_data ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (nfs_wdata_cachep ==NULL )return -ENOMEM ;
fs/ntfs/super.c renamed from fs/ntfs/super.c~slablru +5 -4
@@ -1667,7 +1667,7 @@ static int __init init_ntfs_fs(void) ntfs_attr_ctx_cache =kmem_cache_create (ntfs_attr_ctx_cache_name,sizeof (attr_search_context),0 /* offset */ , -SLAB_HWCACHE_ALIGN ,NULL /* ctor */ ,NULL /* dtor */ ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL /* ctor */ ,NULL /* dtor */ );if (!ntfs_attr_ctx_cache) {printk (KERN_CRIT "NTFS: Failed to create %s!\n" , ntfs_attr_ctx_cache_name);
@@ -1676,7 +1676,7 @@ static int __init init_ntfs_fs(void) ntfs_name_cache =kmem_cache_create (ntfs_name_cache_name, (NTFS_MAX_NAME_LEN +1 ) *sizeof (uchar_t),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!ntfs_name_cache) {printk (KERN_CRIT "NTFS: Failed to create %s!\n" , ntfs_name_cache_name);
@@ -1684,7 +1684,8 @@ static int __init init_ntfs_fs(void) } ntfs_inode_cache =kmem_cache_create (ntfs_inode_cache_name, -sizeof (ntfs_inode),0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +sizeof (ntfs_inode),0 ,SLAB_HWCACHE_ALIGN , + age_icache_memory,NULL ,NULL );if (!ntfs_inode_cache) {printk (KERN_CRIT "NTFS: Failed to create %s!\n" , ntfs_inode_cache_name);
@@ -1693,7 +1694,7 @@ static int __init init_ntfs_fs(void) ntfs_big_inode_cache =kmem_cache_create (ntfs_big_inode_cache_name,sizeof (big_ntfs_inode),0 ,SLAB_HWCACHE_ALIGN , - ntfs_big_inode_init_once,NULL ); + age_icache_memory, ntfs_big_inode_init_once,NULL );if (!ntfs_big_inode_cache) {printk (KERN_CRIT "NTFS: Failed to create %s!\n" , ntfs_big_inode_cache_name);
fs/proc/inode.c renamed from fs/proc/inode.c~slablru +1 -1
@@ -123,7 +123,7 @@ int __init proc_init_inodecache(void) proc_inode_cachep =kmem_cache_create ("proc_inode_cache" ,sizeof (struct proc_inode ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (proc_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/qnx4/inode.c renamed from fs/qnx4/inode.c~slablru +1 -1
@@ -544,7 +544,7 @@ static int init_inodecache(void) qnx4_inode_cachep =kmem_cache_create ("qnx4_inode_cache" ,sizeof (struct qnx4_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (qnx4_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/reiserfs/super.c renamed from fs/reiserfs/super.c~slablru +1 -1
@@ -434,7 +434,7 @@ static int init_inodecache(void) reiserfs_inode_cachep =kmem_cache_create ("reiser_inode_cache" ,sizeof (struct reiserfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (reiserfs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/romfs/inode.c renamed from fs/romfs/inode.c~slablru +1 -1
@@ -577,7 +577,7 @@ static int init_inodecache(void) romfs_inode_cachep =kmem_cache_create ("romfs_inode_cache" ,sizeof (struct romfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (romfs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/smbfs/inode.c renamed from fs/smbfs/inode.c~slablru +1 -1
@@ -78,7 +78,7 @@ static int init_inodecache(void) smb_inode_cachep =kmem_cache_create ("smb_inode_cache" ,sizeof (struct smb_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (smb_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/smbfs/request.c renamed from fs/smbfs/request.c~slablru +1 -1
@@ -37,7 +37,7 @@ int smb_init_request_cache(void) req_cachep =kmem_cache_create ("smb_request" ,sizeof (struct smb_request ),0 ,SMB_SLAB_DEBUG |SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (req_cachep ==NULL )return -ENOMEM ;
fs/sysv/inode.c renamed from fs/sysv/inode.c~slablru +1 -1
@@ -325,7 +325,7 @@ int __init sysv_init_icache(void) { sysv_inode_cachep =kmem_cache_create ("sysv_inode_cache" ,sizeof (struct sysv_inode_info ),0 , -SLAB_HWCACHE_ALIGN , init_once,NULL ); +SLAB_HWCACHE_ALIGN , age_icache_memory, init_once,NULL );if (!sysv_inode_cachep)return -ENOMEM ;return 0 ;
fs/udf/super.c renamed from fs/udf/super.c~slablru +1 -1
@@ -141,7 +141,7 @@ static int init_inodecache(void) udf_inode_cachep =kmem_cache_create ("udf_inode_cache" ,sizeof (struct udf_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (udf_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/ufs/super.c renamed from fs/ufs/super.c~slablru +1 -1
@@ -1028,7 +1028,7 @@ static int init_inodecache(void) ufs_inode_cachep =kmem_cache_create ("ufs_inode_cache" ,sizeof (struct ufs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (ufs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
include/linux/dcache.h renamed from include/linux/dcache.h~slablru +0 -5
@@ -182,16 +182,11 @@ extern void shrink_dcache_anon(struct liextern int d_invalidate (struct dentry *);/* dcache memory management */ -extern int shrink_dcache_memory (int ,unsigned int );extern void prune_dcache (int );/* icache memory management (defined in linux/fs/inode.c) */ -extern int shrink_icache_memory (int ,int );extern void prune_icache (int ); -/* quota cache memory management (defined in linux/fs/dquot.c) */ -extern int shrink_dqcache_memory (int ,unsigned int ); -/* only used at mount-time */ extern struct dentry *d_alloc_root (struct inode *);
include/linux/page-flags.h renamed from include/linux/page-flags.h~slablru +1 -0
@@ -78,6 +78,7 @@ extern struct page_state {unsigned long nr_pagecache;unsigned long nr_page_table_pages;unsigned long nr_reverse_maps; +unsigned long nr_slab; }____cacheline_aligned_in_smp page_states[NR_CPUS ];extern void get_page_state (struct page_state *ret);
include/linux/slab.h renamed from include/linux/slab.h~slablru +23 -3
@@ -49,12 +49,32 @@ typedef struct kmem_cache_s kmem_cache_textern void kmem_cache_init (void );extern void kmem_cache_sizes_init (void ); +typedef int (*kmem_pruner_t )(kmem_cache_t *,int ,int ); +extern kmem_cache_t *kmem_find_general_cachep (size_t ,int gfpflags); -extern kmem_cache_t *kmem_cache_create (const char *,size_t ,size_t ,unsigned long , -void (*)(void *,kmem_cache_t *,unsigned long ), -void (*)(void *,kmem_cache_t *,unsigned long )); +extern kmem_cache_t *kmem_cache_create (const char *,size_t ,size_t , +unsigned long ,kmem_pruner_t , +void (*)(void *,kmem_cache_t *,unsigned long ), +void (*)(void *,kmem_cache_t *,unsigned long ));extern int kmem_cache_destroy (kmem_cache_t *);extern int kmem_cache_shrink (kmem_cache_t *); + +extern int kmem_do_prunes (int ); +extern int kmem_count_page (struct page *,int ); +#define kmem_touch_page (addr) SetPageReferenced(virt_to_page(addr)); + +/* shrink a slab */ +extern int kmem_shrink_slab (struct page *); + +/* dcache prune ( defined in linux/fs/dcache.c) */ +extern int age_dcache_memory (kmem_cache_t *,int ,int ); + +/* icache prune (defined in linux/fs/inode.c) */ +extern int age_icache_memory (kmem_cache_t *,int ,int ); + +/* quota cache prune (defined in linux/fs/dquot.c) */ +extern int age_dqcache_memory (kmem_cache_t *,int ,int ); +extern void *kmem_cache_alloc (kmem_cache_t *,int );extern void kmem_cache_free (kmem_cache_t *,void *);extern unsigned int kmem_cache_size (kmem_cache_t *);
kernel/fork.c renamed from kernel/fork.c~slablru +6 -6
@@ -112,7 +112,7 @@ void __init fork_init(unsigned long memp task_struct_cachep =kmem_cache_create ("task_struct" ,sizeof (struct task_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!task_struct_cachep)panic ("fork_init(): cannot create task_struct SLAB cache" );
@@ -940,31 +940,31 @@ void __init proc_caches_init(void) { sigact_cachep =kmem_cache_create ("signal_act" ,sizeof (struct signal_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!sigact_cachep)panic ("Cannot create signal action SLAB cache" ); files_cachep =kmem_cache_create ("files_cache" ,sizeof (struct files_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!files_cachep)panic ("Cannot create files SLAB cache" ); fs_cachep =kmem_cache_create ("fs_cache" ,sizeof (struct fs_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!fs_cachep)panic ("Cannot create fs_struct SLAB cache" ); vm_area_cachep =kmem_cache_create ("vm_area_struct" ,sizeof (struct vm_area_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!vm_area_cachep)panic ("vma_init: Cannot alloc vm_area_struct SLAB cache" ); mm_cachep =kmem_cache_create ("mm_struct" ,sizeof (struct mm_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!mm_cachep)panic ("vma_init: Cannot alloc mm_struct SLAB cache" ); }
kernel/signal.c renamed from kernel/signal.c~slablru +1 -1
@@ -43,7 +43,7 @@ void __init signals_init(void)kmem_cache_create ("sigqueue" ,sizeof (struct sigqueue ), __alignof__(struct sigqueue ), -SIG_SLAB_DEBUG ,NULL ,NULL ); +SIG_SLAB_DEBUG ,NULL ,NULL ,NULL );if (!sigqueue_cachep)panic ("signals_init(): cannot create sigqueue SLAB cache" ); }
kernel/user.c renamed from kernel/user.c~slablru +1 -1
@@ -118,7 +118,7 @@ static int __init uid_cache_init(void) uid_cachep =kmem_cache_create ("uid_cache" ,sizeof (struct user_struct ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!uid_cachep)panic ("Cannot create uid taskcount SLAB cache\n" );
lib/radix-tree.c renamed from lib/radix-tree.c~slablru +1 -1
@@ -293,7 +293,7 @@ void __init radix_tree_init(void) { radix_tree_node_cachep =kmem_cache_create ("radix_tree_node" ,sizeof (struct radix_tree_node ),0 , -SLAB_HWCACHE_ALIGN , radix_tree_node_ctor,NULL ); +SLAB_HWCACHE_ALIGN ,NULL , radix_tree_node_ctor,NULL );if (!radix_tree_node_cachep)panic ("Failed to create radix_tree_node cache\n" ); radix_tree_node_pool = mempool_create(512, radix_tree_node_pool_alloc,
mm/page_alloc.c renamed from mm/page_alloc.c~slablru +1 -0
@@ -561,6 +561,7 @@ void get_page_state(struct page_state *r ret->nr_pagecache += ps->nr_pagecache; ret->nr_page_table_pages += ps->nr_page_table_pages; ret->nr_reverse_maps += ps->nr_reverse_maps; + ret->nr_slab += ps->nr_slab; } }
mm/rmap.c renamed from mm/rmap.c~slablru +1 -0
@@ -521,6 +521,7 @@ void __init pte_chain_init(void)sizeof (struct pte_chain ),0 ,0 , +NULL , pte_chain_ctor, NULL);
mm/shmem.c renamed from mm/shmem.c~slablru +1 -1
@@ -1514,7 +1514,7 @@ static int init_inodecache(void) shmem_inode_cachep =kmem_cache_create ("shmem_inode_cache" ,sizeof (struct shmem_inode_info ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); + age_icache_memory, init_once,NULL );if (shmem_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
mm/slab.c renamed from mm/slab.c~slablru +188 -99
551 lines not on this page · raw message
mm/swap.c renamed from mm/swap.c~slablru +14 -4
@@ -46,25 +46,34 @@ void activate_page(struct page *page)/** * lru_cache_add: add a page to the page lists * @page: the page to add +* +* Can be called from interrupt context by slab, so protect against that. */ static struct pagevec lru_add_pvecs[NR_CPUS ];void lru_cache_add (struct page *page) { -struct pagevec *pvec = &lru_add_pvecs[get_cpu ()]; +unsigned long flags; +struct pagevec *pvec;page_cache_get (page); + pvec = &lru_add_pvecs[get_cpu ()]; +local_irq_save (flags);if (!pagevec_add (pvec, page))__pagevec_lru_add (pvec); +local_irq_restore (flags);put_cpu (); }void lru_add_drain (void ) { +unsigned long flags;struct pagevec *pvec = &lru_add_pvecs[get_cpu ()]; +local_irq_save (flags);if (pagevec_count (pvec))__pagevec_lru_add (pvec); +local_irq_restore (flags);put_cpu (); }
@@ -202,6 +211,7 @@ void pagevec_deactivate_inactive(structvoid __pagevec_lru_add (struct pagevec *pvec) {int i; +unsigned long flags =0 ;/* avoid uninitialised var warning */ struct zone *zone =NULL ;for (i =0 ; i <pagevec_count (pvec); i++) {
@@ -210,16 +220,16 @@ void __pagevec_lru_add(struct pagevec *pif (pagezone != zone) {if (zone) -spin_unlock_irq (&zone->lru_lock); +spin_unlock_irqrestore (&zone->lru_lock, flags); zone = pagezone; -spin_lock_irq (&zone->lru_lock); +spin_lock_irqsave (&zone->lru_lock, flags); }if (TestSetPageLRU (page))BUG ();add_page_to_inactive_list (zone, page); }if (zone) -spin_unlock_irq (&zone->lru_lock); +spin_unlock_irqrestore (&zone->lru_lock, flags);pagevec_release (pvec); }
mm/vmscan.c renamed from mm/vmscan.c~slablru +27 -14
79 lines not on this page · raw message
net/atm/clip.c renamed from net/atm/clip.c~slablru +1 -1
@@ -751,5 +751,5 @@ void atm_clip_init(void) { clip_tbl.lock =RW_LOCK_UNLOCKED ; clip_tbl.kmem_cachep =kmem_cache_create (clip_tbl.id, - clip_tbl.entry_size,0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ); + clip_tbl.entry_size,0 ,SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL ); }
net/bluetooth/af_bluetooth.c renamed from net/bluetooth/af_bluetooth.c~slablru +1 -1
@@ -328,7 +328,7 @@ static int __init bluez_init(void)/* Init socket cache */ bluez_sock_cache =kmem_cache_create ("bluez_sock" ,sizeof (struct bluez_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!bluez_sock_cache) {BT_ERR ("BlueZ socket cache creation failed" );
net/core/neighbour.c renamed from net/core/neighbour.c~slablru +1 -1
@@ -1146,7 +1146,7 @@ void neigh_table_init(struct neigh_table (tbl->entry_size +15 ) & ~15 ,0 ,SLAB_HWCACHE_ALIGN , - NULL, NULL); + NULL, NULL, NULL);#ifdef CONFIG_SMP tasklet_init (&tbl->gc_task,SMP_TIMER_NAME (neigh_periodic_timer), (unsigned long )tbl);
net/core/skbuff.c renamed from net/core/skbuff.c~slablru +1 -1
@@ -1204,7 +1204,7 @@ void __init skb_init(void)sizeof (struct sk_buff ),0 ,SLAB_HWCACHE_ALIGN , - skb_headerinit, NULL); + NULL, skb_headerinit, NULL);if (!skbuff_head_cache)panic ("cannot create skbuff cache" );
net/core/sock.c renamed from net/core/sock.c~slablru +1 -1
@@ -633,7 +633,7 @@ void sk_free(struct sock *sk)void __init sk_init (void ) { sk_cachep =kmem_cache_create ("sock" ,sizeof (struct sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!sk_cachep)printk (KERN_CRIT "sk_init: Cannot create sock SLAB cache!" );
net/decnet/dn_route.c renamed from net/decnet/dn_route.c~slablru +1 -1
@@ -1244,7 +1244,7 @@ void __init dn_route_init(void) dn_dst_ops.kmem_cachep =kmem_cache_create ("dn_dst_cache" ,sizeof (struct dn_route ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (!dn_dst_ops.kmem_cachep)panic ("DECnet: Failed to allocate dn_dst_cache\n" );
net/decnet/dn_table.c renamed from net/decnet/dn_table.c~slablru +1 -1
@@ -888,7 +888,7 @@ void __init dn_fib_table_init(void) dn_hash_kmem =kmem_cache_create ("dn_fib_info_cache" ,sizeof (struct dn_fib_info ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL ); } void __exit dn_fib_table_cleanup(void)
net/ipv4/af_inet.c renamed from net/ipv4/af_inet.c~slablru +3 -3
@@ -1142,13 +1142,13 @@ static int __init inet_init(void) tcp_sk_cachep =kmem_cache_create ("tcp_sock" ,sizeof (struct tcp_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL ); udp_sk_cachep =kmem_cache_create ("udp_sock" ,sizeof (struct udp_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL ); raw4_sk_cachep =kmem_cache_create ("raw4_sock" ,sizeof (struct raw_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!tcp_sk_cachep || !udp_sk_cachep || !raw4_sk_cachep)printk (KERN_CRIT "inet_init: Can't create protocol sock SLAB caches!\n" );
net/ipv4/fib_hash.c renamed from net/ipv4/fib_hash.c~slablru +1 -1
@@ -899,7 +899,7 @@ struct fib_table * __init fib_hash_init( fn_hash_kmem =kmem_cache_create ("ip_fib_hash" ,sizeof (struct fib_node ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL ); tb =kmalloc (sizeof (struct fib_table ) +sizeof (struct fn_hash ),GFP_KERNEL );if (tb ==NULL )
net/ipv4/inetpeer.c renamed from net/ipv4/inetpeer.c~slablru +1 -1
@@ -125,7 +125,7 @@ void __init inet_initpeers(void) peer_cachep =kmem_cache_create ("inet_peer_cache" ,sizeof (struct inet_peer ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );/* All the timers, started at system startup tend to synchronize. Perturb it a bit.
net/ipv4/ipmr.c renamed from net/ipv4/ipmr.c~slablru +1 -1
@@ -1750,7 +1750,7 @@ void __init ip_mr_init(void) mrt_cachep =kmem_cache_create ("ip_mrt_cache" ,sizeof (struct mfc_cache ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );init_timer (&ipmr_expire_timer); ipmr_expire_timer.function=ipmr_expire_process;register_netdevice_notifier (&ip_mr_notifier);
net/ipv4/netfilter/ip_conntrack_core.c renamed from net/ipv4/netfilter/ip_conntrack_core.c~slablru +1 -1
@@ -1444,7 +1444,7 @@ int __init ip_conntrack_init(void) ip_conntrack_cachep =kmem_cache_create ("ip_conntrack" ,sizeof (struct ip_conntrack ),0 , -SLAB_HWCACHE_ALIGN ,NULL ,NULL ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!ip_conntrack_cachep) {printk (KERN_ERR "Unable to create ip_conntrack slab cache\n" ); goto err_free_hash;
net/ipv4/route.c renamed from net/ipv4/route.c~slablru +1 -1
@@ -2472,7 +2472,7 @@ void __init ip_rt_init(void) ipv4_dst_ops.kmem_cachep =kmem_cache_create ("ip_dst_cache" ,sizeof (struct rtable ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (!ipv4_dst_ops.kmem_cachep)panic ("IP: failed to allocate ip_dst_cache\n" );
net/ipv4/tcp.c renamed from net/ipv4/tcp.c~slablru +3 -3
@@ -2569,21 +2569,21 @@ void __init tcp_init(void) tcp_openreq_cachep =kmem_cache_create ("tcp_open_request" ,sizeof (struct open_request ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (!tcp_openreq_cachep)panic ("tcp_init: Cannot alloc open_request cache." ); tcp_bucket_cachep =kmem_cache_create ("tcp_bind_bucket" ,sizeof (struct tcp_bind_bucket ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (!tcp_bucket_cachep)panic ("tcp_init: Cannot alloc tcp_bind_bucket cache." ); tcp_timewait_cachep =kmem_cache_create ("tcp_tw_bucket" ,sizeof (struct tcp_tw_bucket ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );if (!tcp_timewait_cachep)panic ("tcp_init: Cannot alloc tcp_tw_bucket cache." );
net/ipv6/af_inet6.c renamed from net/ipv6/af_inet6.c~slablru +3 -3
@@ -655,13 +655,13 @@ static int __init inet6_init(void)/* allocate our sock slab caches */ tcp6_sk_cachep =kmem_cache_create ("tcp6_sock" ,sizeof (struct tcp6_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL ); udp6_sk_cachep =kmem_cache_create ("udp6_sock" ,sizeof (struct udp6_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL ); raw6_sk_cachep =kmem_cache_create ("raw6_sock" ,sizeof (struct raw6_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!tcp6_sk_cachep || !udp6_sk_cachep || !raw6_sk_cachep) printk(KERN_CRIT __FUNCTION__": Can't create protocol sock SLAB caches!\n" );
net/ipv6/ip6_fib.c renamed from net/ipv6/ip6_fib.c~slablru +1 -1
@@ -1218,7 +1218,7 @@ void __init fib6_init(void) fib6_node_kmem =kmem_cache_create ("fib6_nodes" ,sizeof (struct fib6_node ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL ); }#ifdef MODULE
net/ipv6/route.c renamed from net/ipv6/route.c~slablru +1 -1
@@ -1919,7 +1919,7 @@ void __init ip6_route_init(void) ip6_dst_ops.kmem_cachep =kmem_cache_create ("ip6_dst_cache" ,sizeof (struct rt6_info ),0 ,SLAB_HWCACHE_ALIGN , -NULL ,NULL ); +NULL ,NULL ,NULL );fib6_init ();#ifdef CONFIG_PROC_FS proc_net_create ("ipv6_route" ,0 , rt6_proc_info);
net/socket.c renamed from net/socket.c~slablru +1 -1
@@ -305,7 +305,7 @@ static int init_inodecache(void) sock_inode_cachep =kmem_cache_create ("sock_inode_cache" ,sizeof (struct socket_alloc ),0 ,SLAB_HWCACHE_ALIGN , - init_once,NULL ); +NULL , init_once,NULL );if (sock_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
net/unix/af_unix.c renamed from net/unix/af_unix.c~slablru +1 -1
@@ -1893,7 +1893,7 @@ static int __init af_unix_init(void)/* allocate our sock slab cache */ unix_sk_cachep =kmem_cache_create ("unix_sock" ,sizeof (struct unix_sock ),0 , -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN ,NULL ,NULL ,NULL );if (!unix_sk_cachep)printk (KERN_CRIT "af_unix_init: Cannot create unix_sock SLAB cache!\n" );
fs/proc/proc_misc.c renamed from fs/proc/proc_misc.c~slablru +2 -0
@@ -177,6 +177,7 @@ static int meminfo_read_proc(char *page,"SwapFree: %8lu kB\n" "Dirty: %8lu kB\n" "Writeback: %8lu kB\n" +"Slab: %8lu kB\n" "Committed_AS: %8u kB\n" "PageTables: %8lu kB\n" "ReverseMaps: %8lu\n"
@@ -197,6 +198,7 @@ static int meminfo_read_proc(char *page,K (i.freeswap),K (ps.nr_dirty),K (ps.nr_writeback), +K (ps.nr_slab),K (committed),K (ps.nr_page_table_pages), ps.nr_reverse_maps,
fs/jfs/jfs_metapage.c renamed from fs/jfs/jfs_metapage.c~slablru +1 -1
@@ -143,7 +143,7 @@ int __init metapage_init(void)* Allocate the metapage structures */ metapage_cache =kmem_cache_create ("jfs_mp" ,sizeof (metapage_t),0 ,0 , - init_once,NULL ); +NULL , init_once,NULL );if (metapage_cache ==NULL )return -ENOMEM ;
.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
Quoted 9 lines · Andrew Morton
Ed, this code can be sped up a bit, I think. We can make
kmem_count_page()return a boolean back toshrink_cache(), telling it whether it needs to callkmem_do_prunes()at all. Often, there won't be any work to do in there, and taking that semaphore can be quite costly.The code as-is will even run
kmem_do_prunes()when we're examiningZONE_HIGHMEM, which certainly won't have any slab pages. This boolean will fix that too.
How about this? I have modified things so we only try for the sem if there is work to do. It also always uses a down_trylock - if we cannot do the prune now later is ok too...
Lightly tested.
Comments
Ed
-----------
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.531 -> 1.533 # mm/slab.c 1.28 -> 1.30
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/09/02 [email protected] 1.532
# optimization. lets only take the sem if we have work to do. # -------------------------------------------- # 02/09/02 [email protected] 1.533 # more optimizations and a correction # -------------------------------------------- # diff -Nru a/mm/slab.c b/mm/slab.c --- a/mm/slab.c Mon Sep 2 10:54:33 2002 +++ b/mm/slab.c Mon Sep 2 10:54:33 2002 @@ -403,6 +403,9 @@ /* Place maintainer for reaping. */ static kmem_cache_t *clock_searchp = &cache_cache;
+static int pruner_flag;
+#define PRUNE_GATE 0
+
#define cache_chain (cache_cache.next)
#ifdef CONFIG_SMP
@@ -427,6 +430,8 @@
spin_lock_irq(&cachep->spinlock);
if (cachep->pruner != NULL) {
cachep->count += slabp->inuse;
+ if (cachep->count)
+ set_bit(PRUNE_GATE, &pruner_flag);
ret = !slabp->inuse;
} else
ret = !ref && !slabp->inuse;
@@ -441,11 +446,13 @@
struct list_head *p;
int nr;
- if (gfp_mask & __GFP_WAIT)
- down(&cache_chain_sem);
- else
- if (down_trylock(&cache_chain_sem))
- return 0;
+ if (!test_and_clear_bit(PRUNE_GATE, &pruner_flag))
+ return 0;
+
+ if (down_trylock(&cache_chain_sem)) {
+ set_bit(PRUNE_GATE, &pruner_flag);
+ return 0;
+ }
list_for_each(p,&cache_chain) {
kmem_cache_t *cachep = list_entry(p, kmem_cache_t, next);-----------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
Quoted 8 lines · Ed Tomlinson
Quoted 6 lines · Andrew Morton
Ed, this code can be sped up a bit, I think. We can make
kmem_count_page()return a boolean back toshrink_cache(), telling it whether it needs to callkmem_do_prunes()at all. Often, there won't be any work to do in there, and taking that semaphore can be quite costly.
Quoted 8 lines
Quoted 3 lines
The code as-is will even run
kmem_do_prunes()when we're examiningZONE_HIGHMEM, which certainly won't have any slab pages. This boolean will fix that too.How about this? I have modified things so we only try for the sem if there is work to do. It also always uses a
down_trylock- if we cannot do the prune now later is ok too...
well... Using a global like that is a bit un-linuxy. (bitops
are only defined on longs, btw...)
How about this one? It does both: tells the caller whether or not to perform the shrink, and defers the pruning until we have at least a page's worth of objects to be pruned.
Also, make sure that only the CPU which was responsible for the transition-past-threshold is told to do some pruning. Reduces the possibility of two CPUs running the prune.
Also, when we make the sweep across the to-be-pruned caches, only prune the ones which are over threshold.
Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=396, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=66, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=429, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=66, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=264, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=198, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=66, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=429, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=66, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc189f66c, entries=66, gfp_mask=464) at dcache.c:585
It'll make things a bit lumpier. Under high internal fragmentation we'll suddenly release great gobs of pages, but I think it'll average out OK.
What sayest thou?
include/linux/slab.h | 2 +- mm/slab.c | 23 +++++++++++++++++++---- mm/vmscan.c | 22 ++++++++++++---------- 3 files changed, 32 insertions(+), 15 deletions(-)
mm/vmscan.c renamed from mm/vmscan.c~slablru-speedup +12 -10
@@ -95,8 +95,8 @@ static inline int is_page_cache_freeable }static /* inline */ int -shrink_list (struct list_head *page_list,int nr_pages, -unsigned int gfp_mask,int priority,int *max_scan) +shrink_list (struct list_head *page_list,int nr_pages,unsigned int gfp_mask, +int priority,int *max_scan,int *prunes_needed) {struct address_space *mapping;LIST_HEAD (ret_pages);
@@ -124,7 +124,7 @@ shrink_list(struct list_head *page_list,*/ if (PageSlab (page)) {int ref =TestClearPageReferenced (page); -if (kmem_count_page (page, ref)) { +if (kmem_count_page (page, ref, prunes_needed)) {if (kmem_shrink_slab (page)) goto free_ref; }
@@ -292,8 +292,8 @@ keep:* in the kernel (apart from the copy_*_user functions). */ static /* inline */ int -shrink_cache (int nr_pages,struct zone *zone, -unsigned int gfp_mask,int priority,int max_scan) +shrink_cache (int nr_pages,struct zone *zone,unsigned int gfp_mask, +int priority,int max_scan,int *prunes_needed) {LIST_HEAD (page_list);struct pagevec pvec;
@@ -342,8 +342,8 @@ shrink_cache(int nr_pages, struct zone * max_scan -= n;KERNEL_STAT_ADD (pgscan, n); - nr_pages =shrink_list (&page_list, nr_pages, - gfp_mask, priority, &max_scan); + nr_pages =shrink_list (&page_list, nr_pages, gfp_mask, + priority, &max_scan, prunes_needed);if (nr_pages <=0 &&list_empty (&page_list)) goto done;
@@ -489,6 +489,7 @@ shrink_zone(struct zone *zone, int prior {unsigned long ratio;int max_scan; +int prunes_needed =0 ;/* * Try to keep the active list 2/3 of the size of the cache. And
@@ -509,9 +510,10 @@ shrink_zone(struct zone *zone, int prior } max_scan = zone->nr_inactive / priority; - nr_pages =shrink_cache (nr_pages, zone, - gfp_mask, priority, max_scan); -kmem_do_prunes (gfp_mask); + nr_pages =shrink_cache (nr_pages, zone, gfp_mask, + priority, max_scan, &prunes_needed); +if (prunes_needed) +kmem_do_prunes (gfp_mask);if (nr_pages <=0 )return 0 ;
mm/slab.c renamed from mm/slab.c~slablru-speedup +19 -4
@@ -217,7 +217,8 @@ struct kmem_cache_s {unsigned int growing;unsigned int dflags;/* dynamic flags */ kmem_pruner_t pruner;/* shrink callback */ -int count;/* count used to trigger shrink */ +int count;/* nr of objects to be pruned */ +int prune_thresh;/* threshold triggers pruning */ /* constructor func */ void (*ctor)(void *,kmem_cache_t *,unsigned long );
@@ -418,8 +419,11 @@ static void enable_all_cpucaches (void);/* * Used by shrink_cache to determine caches that need pruning. +* +* If this particular call to kmem_count_page takes a slab over its to-be-pruned +* threshold then we tell the caller that kmem_do_prunes() needs to be called. */ -int kmem_count_page (struct page *page,int ref) +int kmem_count_page (struct page *page,int ref,int *prunes_needed) {kmem_cache_t *cachep =GET_PAGE_CACHE (page);slab_t *slabp =GET_PAGE_SLAB (page);
@@ -427,7 +431,12 @@ int kmem_count_page(struct page *page, ispin_lock_irq (&cachep->spinlock);if (cachep->pruner !=NULL ) { +int old_count = cachep->count; + cachep->count += slabp->inuse; +if (old_count < cachep->prune_thresh && + cachep->count >= cachep->prune_thresh) + *prunes_needed =1 ; ret = !slabp->inuse; }else { ret = !ref && !slabp->inuse;
@@ -453,8 +462,11 @@ int kmem_do_prunes(int gfp_mask)kmem_cache_t *cachep =list_entry (p, kmem_cache_t, next);if (cachep->pruner !=NULL ) {spin_lock_irq (&cachep->spinlock); - nr = cachep->count; - cachep->count =0 ; + nr =0 ; +if (cachep->count >= cachep->prune_thresh) { + nr = cachep->count; + cachep->count =0 ; + }spin_unlock_irq (&cachep->spinlock);if (nr >0 ) (*cachep->pruner)(cachep, nr, gfp_mask);
@@ -872,6 +884,9 @@ next: cachep->flags = flags; cachep->pruner = thepruner; cachep->count =0 ; + cachep->prune_thresh =0 ; +if (thepruner) + cachep->prune_thresh =PAGE_SIZE / size; cachep->gfpflags =0 ;if (flags &SLAB_CACHE_DMA ) cachep->gfpflags |=GFP_DMA ;
include/linux/slab.h renamed from include/linux/slab.h~slablru-speedup +1 -1
@@ -60,7 +60,7 @@ extern int kmem_cache_destroy(kmem_cacheextern int kmem_cache_shrink (kmem_cache_t *);extern int kmem_do_prunes (int ); -extern int kmem_count_page (struct page *,int ); +extern int kmem_count_page (struct page *page,int ref,int *prunes_needed);#define kmem_touch_page (addr) SetPageReferenced(virt_to_page(addr));/* shrink a slab */
.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
Quoted 12 lines · Andrew Morton
Quoted 11 lines · Ed Tomlinson
Quoted 9 lines · Andrew Morton
Ed, this code can be sped up a bit, I think. We can make
kmem_count_page()return a boolean back toshrink_cache(), telling it whether it needs to callkmem_do_prunes()at all. Often, there won't be any work to do in there, and taking that semaphore can be quite costly.The code as-is will even run
kmem_do_prunes()when we're examiningZONE_HIGHMEM, which certainly won't have any slab pages. This boolean will fix that too.
Quoted 6 lines
Quoted 3 lines
How about this? I have modified things so we only try for the sem if there is work to do. It also always uses a
down_trylock- if we cannot do the prune now later is ok too...well... Using a global like that is a bit un-linuxy. (bitops
are only defined on longs, btw...
ah. learn something every day.
Quoted 3 lines
How about this one? It does both: tells the caller whether or not to perform the shrink, and defers the pruning until we have at least a page's worth of objects to be pruned.
I thought about doing something like your patch. I wanted to avoid semi-magic numbers (why a page worth of objects? why not two or three...). I would rather see something like my patch, maybe coded in a more stylish way, used. If we want to get bigger batch I would move the kmem_do_prunes up into try_to_free_pages. This way the code is simpler, vmscan changes for slablru are smaller, and nothing magic is involved.
Quoted 3 lines
Also, make sure that only the CPU which was responsible for the transition-past-threshold is told to do some pruning. Reduces the possibility of two CPUs running the prune.
With my code it is possible two cpus could prune but very unlikely.
Quoted 2 lines
Also, when we make the sweep across the to-be-pruned caches, only prune the ones which are over threshold.
If the kmem_do_prunes is moved to try_to_free_pages its not quite as hot a call. Since it now never waits (with my patch) doubt if it is going to show up as something that needs tuning...
How about this? What does it show if you breakpoint it? How would
you make it prettier linux wise? (compiled, untested)
Ed
-----
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.531 -> 1.534 # mm/vmscan.c 1.98 -> 1.99 # mm/slab.c 1.28 -> 1.31
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/09/02 [email protected] 1.532
# optimization. lets only take the sem if we have work to do. # -------------------------------------------- # 02/09/02 [email protected] 1.533 # more optimizations and a correction # -------------------------------------------- # 02/09/02 [email protected] 1.534 # more optimizing # -------------------------------------------- # diff -Nru a/mm/slab.c b/mm/slab.c --- a/mm/slab.c Mon Sep 2 15:05:01 2002 +++ b/mm/slab.c Mon Sep 2 15:05:01 2002 @@ -403,6 +403,9 @@ /* Place maintainer for reaping. */ static kmem_cache_t *clock_searchp = &cache_cache;
+static long pruner_flag;
+#define PRUNE_GATE 0
+
#define cache_chain (cache_cache.next)
#ifdef CONFIG_SMP
@@ -427,6 +430,8 @@
spin_lock_irq(&cachep->spinlock);
if (cachep->pruner != NULL) {
cachep->count += slabp->inuse;
+ if (cachep->count)
+ set_bit(PRUNE_GATE, &pruner_flag);
ret = !slabp->inuse;
} else
ret = !ref && !slabp->inuse;
@@ -441,11 +446,13 @@
struct list_head *p;
int nr;
- if (gfp_mask & __GFP_WAIT)
- down(&cache_chain_sem);
- else
- if (down_trylock(&cache_chain_sem))
- return 0;
+ if (!test_and_clear_bit(PRUNE_GATE, &pruner_flag))
+ return 0;
+
+ if (down_trylock(&cache_chain_sem)) {
+ set_bit(PRUNE_GATE, &pruner_flag);
+ return 0;
+ }
list_for_each(p,&cache_chain) {
kmem_cache_t *cachep = list_entry(p, kmem_cache_t, next);mm/vmscan.c +2 -2
@@ -510,8 +510,6 @@ max_scan = zone->nr_inactive / priority; nr_pages =shrink_cache (nr_pages, zone, gfp_mask, priority, max_scan); -kmem_do_prunes (gfp_mask); -if (nr_pages <=0 )return 0 ;
@@ -549,6 +547,8 @@int nr_pages =SWAP_CLUSTER_MAX ;KERNEL_STAT_INC (pageoutrun); + +kmem_do_prunes (gfp_mask); do { nr_pages = shrink_caches(classzone, priority,
-----
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
Quoted 5 lines · Ed Tomlinson
...
I thought about doing something like your patch. I wanted to avoid semi-magic numbers (why a page worth of objects? why not two or three...).
Well, it's just an efficiency heuristic...
Quoted 5 lines
I would rather see something like my patch, maybe coded
in a more stylish way, used. If we want to get bigger batch I would move thekmem_do_prunesup intotry_to_free_pages. This way the code is simpler, vmscan changes for slablru are smaller, and nothing magic is involved.
Doesn't make much difference, afaict. Generally, the first pass
through shrink_caches() frees a sufficient number of pages, so the before- and after- code are equivalent.
And because there is only one flag, we still attempt to prune all caches which have a pruner, when it's quite possible that just one of them has a decent amount of stuff. Probably a minor issue though.
I wouldn't be too fussed about the extent of changes in vmscan.c. One day, when the kernel is perfect, all of that file will have the inlines turned on and the whole of page reclaim becomes one big function.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Re: slablru for 2.5.32-mm1
In reply to Ed Tomlinsonhm. Doing a bit more testing...
mem=512m, then build the inode and dentry caches up a bit:
ext2_inode_cache: 20483KB 20483KB 100.0
buffer_head: 6083KB 6441KB 94.43
dentry_cache: 4885KB 4885KB 100.0 (using wli's bloatmeter, attached here).
Now,
dd if=/dev/zero of=foo bs=1M count=2000
ext2_inode_cache: 3789KB 8148KB 46.50
buffer_head: 6469KB 6503KB 99.47
size-512: 1450KB 1500KB 96.66this took quite a long time to start dropping, and the machine still has 27 megabytes in slab.
Which kinda surprises me, given my (probably wrong) description of the algorithm. I'd have expected the caches to be pruned a lot faster and further than this. Not that it's necessarily a bad thing, but maybe we should be shrinking a little faster. What are your thoughts on this?
Also, I note that age_dcache_memory is being called for lots of tiny little shrinkings:
Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=1, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=2, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=4, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=12, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=21, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=42, gfp_mask=464) at dcache.c:585 Breakpoint 1, age_dcache_memory (cachep=0xc1911e48, entries=10, gfp_mask=464) at dcache.c:585
I'd suggest that we batch these up a bit: call the pruner less frequently, but with larger request sizes, save a few cycles.
#!/bin/sh
while true
do
clear grep -v '^slabinfo' /proc/slabinfo \ | bloatmon \ | sort -r -n +2 \ | head -22 sleep 5
done
#!/usr/bin/awk -f
BEGIN {
printf "%18s %8s %8s %8s\n", "cache", "active", "alloc", "%util";
}{
if ($3 != 0.0) {
pct = 100.0 * $2 / $3;
frac = (10000.0 * $2 / $3) % 100;
} else {
pct = 100.0;
frac = 0.0;
}
active = ($2 * $4)/1024;
alloc = ($3 * $4)/1024;
if ((alloc - active) < 1.0) {
pct = 100.0;
frac = 0.0;
}
printf "%18s: %8dKB %8dKB %3d.%-2d\n", $1, active, alloc, pct, frac;
}