Swap over Networked storage -v12
Subsystems (26)
cover Swap over Networked storage -v12
There is a fundamental deadlock associated with paging; when writing out a page to free memory requires free memory to complete. The usually solution is to keep a small amount of memory available at all times so we can overcome this problem. This however assumes the amount of memory needed for writeout is (constant and) smaller than the provided reserve.
It is this latter assumption that breaks when doing writeout over network. Network can take up an unspecified amount of memory while waiting for a reply to our write request. This re-introduces the deadlock; we might never complete the writeout, for we might not have enough memory to receive the completion message.
The proposed solution is simple, only allow traffic servicing the VM to make use of the reserves.
This however implies you know what packets are for whom, which generally speaking you don't. Hence we need to receive all packets but discard them as soon as we encounter a non VM bound packet allocated from the reserves.
Also knowing it is headed towards the VM needs a little help, hence we introduce the socket flag SOCK_VMIO to mark sockets with.
Of course, since we are paging all this has to happen in kernel-space, since user-space might just not be there.
Since packet processing might also require memory, this all also implies that those auxiliary allocations may use the reserves when an emergency packet is processed. This is accomplished by using PF_MEMALLOC.
How much memory is to be reserved is also an issue, enough memory to saturate both the route cache and IP fragment reassembly, along with various constants.
This patch-set comes in 6 parts:
1) introduce the memory reserve and make the SLAB allocator play nice with it. patches 01-10
2) add some needed infrastructure to the network code patches 11-13
3) implement the idea outlined above
patches 14-20
4) teach the swap machinery to use generic address_spaces patches 21-24
5) implement swap over NFS using all the new stuff patches 25-31
6) implement swap over iSCSI
patches 32-40
Patches can also be found here:
http://programming.kicks-ass.net/kernel-patches/vm_deadlock/v12/
If I receive no feedback, I will assume the various maintainers do not object and I will respin the series against -mm and submit for inclusion.
There is interest in this feature from the stateless linux world; that is both the virtualization world, and the cluster world.
I have been contacted by various groups, some have just expressed their interest, others have been testing this work in their environments.
Various hardware vendors have also expressed interest, and, of course, my employer finds it important enough to have me work on it.
Also, while it doesn't present a full-fledged reserve-based allocator API yet, it does lay most of the groundwork for it. There is a GFP_NOFAIL elimination project wanting to use this as a foundation. Elimination of GFP_NOFAIL will greatly improve the basic soundness and stability of the code that currently uses that construct - most disk based filesystems.
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/40 mm: page allocation rank
Introduce page allocation rank.
This allocation rank is an measure of the 'hardness' of the page allocation. Where hardness refers to how deep we have to reach (and thereby if reclaim was activated) to obtain the page.
It basically is a mapping from the ALLOC_/gfp flags into a scalar quantity, which allows for comparisons of the kind:
'would this allocation have succeeded using these gfp flags'.
For the gfp -> alloc_flags mapping we use the 'hardest' possible, those used by __alloc_pages() right before going into direct reclaim.
The alloc_flags -> rank mapping is given by: 2*2^wmark - harder - 2*high where wmark = { min = 1, low, high } and harder, high are booleans. This gives:
0 is the hardest possible allocation - ALLOC_NO_WATERMARK, 1 is ALLOC_WMARK_MIN|ALLOC_HARDER|ALLOC_HIGH,
...
15 is ALLOC_WMARK_HIGH|ALLOC_HARDER,
16 is the softest allocation - ALLOC_WMARK_HIGH.
Rank <= 4 will have woke up kswapd and when also > 0 might have ran into direct reclaim.
Rank > 8 rarely happens and means lots of memory free (due to parallel oom kill).
The allocation rank is stored in page->index for successful allocations.
'offline' testing of the rank is made impossible by direct reclaim and fragmentation issues. That is, it is impossible to tell if a given allocation will succeed without actually doing it.
The purpose of this measure is to introduce some fairness into the slab allocator.
--- mm/internal.h | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 58 +++++++++++++--------------------------------- 2 files changed, 87 insertions(+), 41 deletions(-)
mm/internal.h +70 -0
@@ -12,6 +12,7 @@#define __MM_INTERNAL_H#include <linux/mm.h> +#include <linux/hardirq.h> static inline void set_page_count (struct page *page,int v) {
@@ -37,4 +38,73 @@ static inline void __put_page(struct pag extern void fastcall __init __free_pages_bootmem(struct page *page,unsigned int order); +#define ALLOC_HARDER 0x01/* try to alloc harder */ +#define ALLOC_HIGH 0x02/* __GFP_HIGH set */ +#define ALLOC_WMARK_MIN 0x04/* use pages_min watermark */ +#define ALLOC_WMARK_LOW 0x08/* use pages_low watermark */ +#define ALLOC_WMARK_HIGH 0x10/* use pages_high watermark */ +#define ALLOC_NO_WATERMARKS 0x20/* don't check watermarks at all */ +#define ALLOC_CPUSET 0x40/* check for correct cpuset */ + +/* +* get the deepest reaching allocation flags for the given gfp_mask +*/ +static int inline gfp_to_alloc_flags (gfp_t gfp_mask) +{ +struct task_struct *p = current; +int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ; +const gfp_t wait = gfp_mask & __GFP_WAIT; + +/* +* The caller may dip into page reserves a bit more if the caller +* cannot run direct reclaim, or if the caller has realtime scheduling +* policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will +* set both ALLOC_HARDER (!wait) and ALLOC_HIGH (__GFP_HIGH). +*/ +if (gfp_mask & __GFP_HIGH) + alloc_flags |=ALLOC_HIGH ; + +if (!wait) { + alloc_flags |=ALLOC_HARDER ; +/* +* Ignore cpuset if GFP_ATOMIC (!wait) rather than fail alloc. +* See also cpuset_zone_allowed() comment in kernel/cpuset.c. +*/ + alloc_flags &= ~ALLOC_CPUSET ; + }else if (unlikely (rt_task (p)) && !in_interrupt ()) + alloc_flags |=ALLOC_HARDER ; + +if (likely (!(gfp_mask & __GFP_NOMEMALLOC))) { +if (!in_interrupt () && + ((p->flags &PF_MEMALLOC ) || +unlikely (test_thread_flag (TIF_MEMDIE )))) + alloc_flags |=ALLOC_NO_WATERMARKS ; + } + +return alloc_flags; +} + +#define MAX_ALLOC_RANK 16 + +/* +* classify the allocation: 0 is hardest, 16 is easiest. +*/ +static inline int alloc_flags_to_rank (int alloc_flags) +{ +int rank; + +if (alloc_flags &ALLOC_NO_WATERMARKS ) +return 0 ; + + rank = alloc_flags & (ALLOC_WMARK_MIN |ALLOC_WMARK_LOW |ALLOC_WMARK_HIGH ); + rank -= alloc_flags & (ALLOC_HARDER |ALLOC_HIGH ); + +return rank; +} + +static inline int gfp_to_rank (gfp_t gfp_mask) +{ +return alloc_flags_to_rank (gfp_to_alloc_flags (gfp_mask)); +} +#endif
mm/page_alloc.c +17 -41
91 lines not on this page · raw message
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/40 mm: slab allocation fairness
In reply to Peter ZijlstraThe slab allocator has some unfairness wrt gfp flags; when the slab cache is grown the gfp flags are used to allocate more memory, however when there is slab cache available (in partial or free slabs, per cpu caches or otherwise) gfp flags are ignored.
Thus it is possible for less critical slab allocations to succeed and gobble up precious memory when under memory pressure.
This patch solves that by using the newly introduced page allocation rank.
Page allocation rank is a scalar quantity connecting ALLOC_ and gfp flags which represents how deep we had to reach into our reserves when allocating a page. Rank 0 is the deepest we can reach (ALLOC_NO_WATERMARK) and 16 is the most shallow allocation possible (ALLOC_WMARK_HIGH).
When the slab space is grown the rank of the page allocation is stored. For each slab allocation we test the given gfp flags against this rank. Thereby asking the question: would these flags have allowed the slab to grow.
If not so, we need to test the current situation. This is done by forcing the growth of the slab space. (Just testing the free page limits will not work due to direct reclaim) Failing this we need to fail the slab allocation.
Thus if we grew the slab under great duress while PF_MEMALLOC was set and we really did access the memalloc reserve the rank would be set to 0. If the next allocation to that slab would be GFP_NOFS|__GFP_NOMEMALLOC (which ordinarily maps to rank 4 and always > 0) we'd want to make sure that memory pressure has decreased enough to allow an allocation with the given gfp flags.
So in this case we try to force grow the slab cache and on failure we fail the slab allocation. Thus preserving the available slab cache for more pressing allocations.
If this newly allocated slab will be trimmed on the next kmem_cache_free (not unlikely) this is no problem, since 1) it will free memory and 2) the sole purpose of the allocation was to probe the allocation rank, we didn't need the space itself.
[AIM9 results go here]
AIM9 test 2.6.21-rc5 2.6.21-rc5-slab1
CONFIG_SLAB_FAIR=y
54 tcp_test 2124.48 +/- 10.85 2137.43 +/- 9.22 12.95 55 udp_test 5204.43 +/- 45.13 5231.59 +/- 56.66 27.16 56 fifo_test 20991.42 +/- 46.71 19675.97 +/- 56.35 1315.44
57 stream_pipe 10024.16 +/- 119.88 9912.53 +/- 75.52 111.63
58 dgram_pipe 9460.18 +/- 119.50 9502.75 +/- 89.06 42.57
59 pipe_cpy 30719.81 +/- 117.01 27885.52 +/- 46.81 2834.28
2.6.21-rc5-slab2
CONFIG_SLAB_FAIR=n54 tcp_test 2124.48 +/- 10.85 2137.97 +/- 12.85 13.50 55 udp_test 5204.43 +/- 45.13 5268.21 +/- 83.38 63.78 56 fifo_test 20991.42 +/- 46.71 19394.42 +/- 65.15 1596.99
57 stream_pipe 10024.16 +/- 119.88 10042.49 +/- 132.13 18.33
58 dgram_pipe 9460.18 +/- 119.50 9575.97 +/- 111.86 115.80
59 pipe_cpy 30719.81 +/- 117.01 27226.52 +/- 120.15 3493.28
Given that the CONFIG_SLAB_FAIR=n numbers are worse than =y, I'm not sure how to interpret these numbers.
Will work on getting =n equal. Also, will work on a SLUB version of these patches.
--- mm/Kconfig | 3 ++ mm/slab.c | 81 ++++++++++++++++++++++++++++++++++++++++--------------------- 2 files changed, 57 insertions(+), 27 deletions(-)
mm/slab.c +54 -27
@@ -114,6 +114,7 @@#include <asm/cacheflush.h> #include <asm/tlbflush.h> #include <asm/page.h> +#include "internal.h" /* * DEBUG - 1 for kmem_cache_create() to honour; SLAB_DEBUG_INITIAL,
@@ -380,6 +381,7 @@ static void kmem_list3_init(struct kmem_struct kmem_cache {/* 1) per-cpu data, touched during every alloc/free */ +int rank;struct array_cache *array[NR_CPUS ];/* 2) Cache tunables. Protected by cache_chain_mutex */ unsigned int batchcount;
@@ -1023,21 +1025,21 @@ static inline int cache_free_alien(struc }static inline void *alternate_node_alloc (struct kmem_cache *cachep, -gfp_t flags) +gfp_t flags,int rank) {return NULL ; }static inline void *____cache_alloc_node (struct kmem_cache *cachep, -gfp_t flags,int nodeid) +gfp_t flags,int nodeid,int rank) {return NULL ; }#else /* CONFIG_NUMA */ -static void *____cache_alloc_node (struct kmem_cache *,gfp_t ,int ); -static void *alternate_node_alloc (struct kmem_cache *,gfp_t ); +static void *____cache_alloc_node (struct kmem_cache *,gfp_t ,int ,int ); +static void *alternate_node_alloc (struct kmem_cache *,gfp_t ,int );static struct array_cache **alloc_alien_cache (int node,int limit) {
@@ -1628,6 +1630,7 @@ static void *kmem_getpages(struct kmem_cif (!page)return NULL ; + cachep->rank = page->index; nr_pages = (1 << cachep->gfporder);if (cachep->flags &SLAB_RECLAIM_ACCOUNT ) add_zone_page_state(page_zone(page),
@@ -2276,6 +2279,7 @@ kmem_cache_create (const char *name, siz }#endif #endif + cachep->rank =MAX_ALLOC_RANK ;/* * Determine if the slab management is 'on' or 'off' slab.
@@ -2942,7 +2946,7 @@ bad:#define check_slabp (x,y) do { } while(0)#endif -static void *cache_alloc_refill (struct kmem_cache *cachep,gfp_t flags) +static void *cache_alloc_refill (struct kmem_cache *cachep,gfp_t flags,int rank) {int batchcount;struct kmem_list3 *l3;
@@ -2954,6 +2958,8 @@ static void *cache_alloc_refill(struct kcheck_irq_off (); ac =cpu_cache_get (cachep); retry: +if (unlikely (rank > cachep->rank)) + goto force_grow; batchcount = ac->batchcount;if (!ac->touched && batchcount >BATCHREFILL_LIMIT ) {/*
@@ -3009,14 +3015,16 @@ must_grow: l3->free_objects -= ac->avail; alloc_done:spin_unlock (&l3->list_lock); -if (unlikely (!ac->avail)) {int x; +force_grow: x =cache_grow (cachep, flags |GFP_THISNODE , node,NULL );/* cache_grow can reenable interrupts, then ac could change. */ ac =cpu_cache_get (cachep); -if (!x && ac->avail ==0 )/* no objects in sight? abort */ + +/* no objects in sight? abort */ +if (!x && (ac->avail ==0 || rank > cachep->rank))return NULL ;if (!ac->avail)/* objects refilled by interrupt? */
@@ -3173,7 +3181,8 @@ static inline int should_failslab(struct#endif /* CONFIG_FAILSLAB */ -static inline void *____cache_alloc (struct kmem_cache *cachep,gfp_t flags) +static inline void *____cache_alloc (struct kmem_cache *cachep, +gfp_t flags,int rank) {void *objp;struct array_cache *ac;
@@ -3184,17 +3193,29 @@ static inline void *____cache_alloc(strureturn NULL ; ac =cpu_cache_get (cachep); -if (likely (ac->avail)) { +if (likely (ac->avail && rank <= cachep->rank)) {STATS_INC_ALLOCHIT (cachep); ac->touched =1 ; objp = ac->entry[--ac->avail]; }else {STATS_INC_ALLOCMISS (cachep); - objp =cache_alloc_refill (cachep, flags); + objp =cache_alloc_refill (cachep, flags, rank); }return objp; } +#ifdef CONFIG_SLAB_FAIR +static inline int slab_alloc_rank (gfp_t flags) +{ +return gfp_to_rank (flags); +} +#else +static inline int slab_alloc_rank (gfp_t flags) +{ +return 0 ; +} +#endif +#ifdef CONFIG_NUMA /* * Try allocating on another node if PF_SPREAD_SLAB|PF_MEMPOLICY.
@@ -3202,7 +3223,8 @@ static inline void *____cache_alloc(stru* If we are in_interrupt, then process context, including cpusets and * mempolicy, may not apply and should not be used for allocation policy. */ -static void *alternate_node_alloc (struct kmem_cache *cachep,gfp_t flags) +static void *alternate_node_alloc (struct kmem_cache *cachep, +gfp_t flags,int rank) {int nid_alloc, nid_here;
@@ -3214,7 +3236,7 @@ static void *alternate_node_alloc(structelse if (current->mempolicy) nid_alloc =slab_node (current->mempolicy);if (nid_alloc != nid_here) -return ____cache_alloc_node (cachep, flags, nid_alloc); +return ____cache_alloc_node (cachep, flags, nid_alloc, rank);return NULL ; }
@@ -3226,7 +3248,7 @@ static void *alternate_node_alloc(struct* allocator to do its reclaim / fallback magic. We then insert the * slab into the proper nodelist and then allocate from it. */ -static void *fallback_alloc (struct kmem_cache *cache,gfp_t flags) +static void *fallback_alloc (struct kmem_cache *cache,gfp_t flags,int rank) {struct zonelist *zonelist;gfp_t local_flags;
@@ -3253,7 +3275,7 @@ retry: cache->nodelists[nid] && cache->nodelists[nid]->free_objects) obj =____cache_alloc_node (cache, - flags |GFP_THISNODE , nid); + flags |GFP_THISNODE , nid, rank); }if (!obj && !(flags & __GFP_NO_GROW)) {
@@ -3276,7 +3298,7 @@ retry: nid =page_to_nid (virt_to_page (obj));if (cache_grow (cache, flags, nid, obj)) { obj =____cache_alloc_node (cache, - flags |GFP_THISNODE , nid); + flags |GFP_THISNODE , nid, rank);if (!obj)/* * Another processor may allocate the
@@ -3297,7 +3319,7 @@ retry:* A interface to enable slab creation on nodeid */ static void *____cache_alloc_node (struct kmem_cache *cachep,gfp_t flags, -int nodeid) +int nodeid,int rank) {struct list_head *entry;struct slab *slabp;
@@ -3310,6 +3332,8 @@ static void *____cache_alloc_node(struct retry:check_irq_off (); +if (unlikely (rank > cachep->rank)) + goto force_grow;spin_lock (&l3->list_lock); entry = l3->slabs_partial.next;if (entry == &l3->slabs_partial) {
@@ -3345,11 +3369,12 @@ retry: must_grow:spin_unlock (&l3->list_lock); +force_grow: x =cache_grow (cachep, flags |GFP_THISNODE , nodeid,NULL );if (x) goto retry; -return fallback_alloc (cachep, flags); +return fallback_alloc (cachep, flags, rank); done:return obj;
@@ -3373,6 +3398,7 @@ __cache_alloc_node(struct kmem_cache *ca {unsigned long save_flags;void *ptr; +int rank =slab_alloc_rank (flags);cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags);
@@ -3382,7 +3408,7 @@ __cache_alloc_node(struct kmem_cache *caif (unlikely (!cachep->nodelists[nodeid])) {/* Node not bootstrapped yet */ - ptr =fallback_alloc (cachep, flags); + ptr =fallback_alloc (cachep, flags, rank); goto out; }
@@ -3393,12 +3419,12 @@ __cache_alloc_node(struct kmem_cache *ca* to other nodes. It may fail while we still have * objects on other nodes available. */ - ptr =____cache_alloc (cachep, flags); + ptr =____cache_alloc (cachep, flags, rank);if (ptr) goto out; }/* ___cache_alloc_node can fall back to other nodes */ - ptr =____cache_alloc_node (cachep, flags, nodeid); + ptr =____cache_alloc_node (cachep, flags, nodeid, rank); out:local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller);
@@ -3407,23 +3433,23 @@ __cache_alloc_node(struct kmem_cache *ca } static __always_inline void * -__do_cache_alloc (struct kmem_cache *cache,gfp_t flags) +__do_cache_alloc (struct kmem_cache *cache,gfp_t flags,int rank) {void *objp;if (unlikely (current->flags & (PF_SPREAD_SLAB |PF_MEMPOLICY ))) { - objp =alternate_node_alloc (cache, flags); + objp =alternate_node_alloc (cache, flags, rank);if (objp) goto out; } - objp =____cache_alloc (cache, flags); + objp =____cache_alloc (cache, flags, rank);/* * We may just have run out of memory on the local node. * ____cache_alloc_node() knows how to locate memory on other nodes */ if (!objp) - objp =____cache_alloc_node (cache, flags,numa_node_id ()); + objp =____cache_alloc_node (cache, flags,numa_node_id (), rank); out:return objp;
@@ -3431,9 +3457,9 @@ __do_cache_alloc(struct kmem_cache *cach#else static __always_inline void * -__do_cache_alloc (struct kmem_cache *cachep,gfp_t flags) +__do_cache_alloc (struct kmem_cache *cachep,gfp_t flags,int rank) { -return ____cache_alloc (cachep, flags); +return ____cache_alloc (cachep, flags, rank); }#endif /* CONFIG_NUMA */
@@ -3443,10 +3469,11 @@ __cache_alloc(struct kmem_cache *cachep, {unsigned long save_flags;void *objp; +int rank =slab_alloc_rank (flags);cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags); - objp =__do_cache_alloc (cachep, flags); + objp =__do_cache_alloc (cachep, flags, rank);local_irq_restore (save_flags); objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp);
mm/Kconfig +3 -0
@@ -163,3 +163,6 @@ config ZONE_DMA_FLAG default "0" if !ZONE_DMA default "1" +config SLAB_FAIR + def_bool n + depends on SLAB
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 02/40] mm: slab allocation fairness
Quoted 5 lines · Peter Zijlstra
Page allocation rank is a scalar quantity connecting
ALLOC_and gfp flags which represents how deep we had to reach into our reserves when allocating a page. Rank 0 is the deepest we can reach (ALLOC_NO_WATERMARK) and 16 is the most shallow allocation possible (ALLOC_WMARK_HIGH).
Quoted 7 lines
When the slab space is grown the rank of the page allocation is stored. For each slab allocation we test the given gfp flags against this rank. Thereby asking the question: would these flags have allowed the slab to grow.
If not so, we need to test the current situation. This is done by forcing the growth of the slab space. (Just testing the free page limits will not work due to direct reclaim) Failing this we need to fail the slab allocation.
This implies that an allocation at time t2 must be aware of the result of an allocation at time t1. It assumes a linear ordering of allocations that is not possible on large systems. Ordering of events is a very expensive endeavor in particular on NUMA systems given the potentially large latencies between various portions of the system.
Maybe you need to restrict the ordering per cpu or per node? Per zone?
Then we would need to store the ranks somewhere which raises scalability issues if these are global.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/40 mm: allow PF_MEMALLOC from softirq context
In reply to Peter ZijlstraAllow PF_MEMALLOC to be set in softirq context. When running softirqs from a borrowed context save current->flags, ksoftirqd will have its own task_struct.
--- include/linux/sched.h | 4 ++++ kernel/softirq.c | 3 +++ mm/internal.h | 7 ++++--- 3 files changed, 11 insertions(+), 3 deletions(-)
mm/internal.h +4 -3
@@ -75,9 +75,10 @@ static int inline gfp_to_alloc_flags(gfp alloc_flags |=ALLOC_HARDER ;if (likely (!(gfp_mask & __GFP_NOMEMALLOC))) { -if (!in_interrupt () && - ((p->flags &PF_MEMALLOC ) || -unlikely (test_thread_flag (TIF_MEMDIE )))) +if (!in_irq () && (p->flags &PF_MEMALLOC )) + alloc_flags |=ALLOC_NO_WATERMARKS ; +else if (!in_interrupt () && +unlikely (test_thread_flag (TIF_MEMDIE ))) alloc_flags |=ALLOC_NO_WATERMARKS ; }
kernel/softirq.c +3 -0
@@ -210,6 +210,8 @@ asmlinkage void __do_softirq(void)__u32 pending;int max_restart =MAX_SOFTIRQ_RESTART ;int cpu; +unsigned long pflags = current->flags; + current->flags &= ~PF_MEMALLOC ; pending =local_softirq_pending ();account_system_vtime (current);
@@ -248,6 +250,7 @@ restart:account_system_vtime (current);_local_bh_enable (); +tsk_restore_flags (current, pflags,PF_MEMALLOC ); }#ifndef __ARCH_HAS_DO_SOFTIRQ
include/linux/sched.h +4 -0
@@ -1185,6 +1185,10 @@ static inline void put_task_struct(struc#define tsk_used_math (p) ((p)->flags & PF_USED_MATH)#define used_math () tsk_used_math(current) +#define tsk_restore_flags (p, pflags, mask) \ + do { (p)->flags &= ~(mask); \ + (p)->flags |= ((pflags) & (mask)); } while (0) +#ifdef CONFIG_SMP extern int set_cpus_allowed (struct task_struct *p,cpumask_t new_mask);#else
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/40 mm: serialize access to min_free_kbytes
In reply to Peter ZijlstraThere is a small race between the procfs caller and the memory hotplug caller of setup_per_zone_pages_min(). Not a big deal, but the next patch will add yet another caller. Time to close the gap.
--- mm/page_alloc.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
mm/page_alloc.c +13 -3
@@ -95,6 +95,7 @@ static char * const zone_names[MAX_NR_ZO#endif }; +static DEFINE_SPINLOCK (min_free_lock);int min_free_kbytes =1024 ;unsigned long __meminitdata nr_kernel_pages;
@@ -3074,12 +3075,12 @@ static void setup_per_zone_lowmem_reserv }/** -* setup_per_zone_pages_min - called when min_free_kbytes changes. +* __setup_per_zone_pages_min - called when min_free_kbytes changes. * * Ensures that the pages_{min,low,high} values for each zone are set correctly * with respect to min_free_kbytes. */ -void setup_per_zone_pages_min (void ) +static void __setup_per_zone_pages_min (void ) {unsigned long pages_min = min_free_kbytes >> (PAGE_SHIFT -10 );unsigned long lowmem_pages =0 ;
@@ -3133,6 +3134,15 @@ void setup_per_zone_pages_min(void)calculate_totalreserve_pages (); } +void setup_per_zone_pages_min (void ) +{ +unsigned long flags; + +spin_lock_irqsave (&min_free_lock, flags); +__setup_per_zone_pages_min (); +spin_unlock_irqrestore (&min_free_lock, flags); +} +/* * Initialise min_free_kbytes. *
@@ -3168,7 +3178,7 @@ static int __init init_per_zone_pages_mi min_free_kbytes =128 ;if (min_free_kbytes >65536 ) min_free_kbytes =65536 ; -setup_per_zone_pages_min (); +__setup_per_zone_pages_min ();setup_per_zone_lowmem_reserve ();return 0 ; }
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/40 mm: emergency pool
In reply to Peter ZijlstraProvide means to reserve a specific amount of pages.
The emergency pool is separated from the min watermark because ALLOC_HARDER and ALLOC_HIGH modify the watermark in a relative way and thus do not ensure a strict minimum.
--- include/linux/mmzone.h | 3 +- mm/page_alloc.c | 52 ++++++++++++++++++++++++++++++++++++++++--------- mm/vmstat.c | 6 ++--- 3 files changed, 48 insertions(+), 13 deletions(-)
include/linux/mmzone.h +2 -1
@@ -178,7 +178,7 @@ enum zone_type {struct zone {/* Fields commonly accessed by the page allocator */ -unsigned long pages_min, pages_low, pages_high; +unsigned long pages_emerg, pages_min, pages_low, pages_high;/* * We don't know if the memory that we're going to allocate will be freeable * or/and it will be released eventually, so to avoid totally wasting several
@@ -562,6 +562,7 @@ int sysctl_min_unmapped_ratio_sysctl_han struct file *, void __user *, size_t *, loff_t *);int sysctl_min_slab_ratio_sysctl_handler (struct ctl_table *,int ,struct file *,void __user *,size_t *,loff_t *); +void adjust_memalloc_reserve (int pages);#include <linux/topology.h> /* Returns the number of the current Node. */
mm/page_alloc.c +43 -9
116 lines not on this page · raw message
mm/vmstat.c +3 -3
@@ -513,9 +513,9 @@ static int zoneinfo_show(struct seq_file"\n spanned %lu" "\n present %lu" ,zone_page_state (zone,NR_FREE_PAGES ), - zone->pages_min, - zone->pages_low, - zone->pages_high, + zone->pages_emerg + zone->pages_min, + zone->pages_emerg + zone->pages_low, + zone->pages_emerg + zone->pages_high, zone->pages_scanned, zone->nr_scan_active, zone->nr_scan_inactive, zone->spanned_pages,
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/40 mm: __GFP_EMERGENCY
In reply to Peter Zijlstra__GFP_EMERGENCY will allow the allocation to disregard the watermarks, much like PF_MEMALLOC.
--- include/linux/gfp.h | 7 ++++++- mm/internal.h | 10 +++++++--- 2 files changed, 13 insertions(+), 4 deletions(-)
include/linux/gfp.h +6 -1
@@ -35,17 +35,21 @@ struct vm_area_struct;#define __GFP_HIGH ((__force gfp_t)0x20u)/* Should access emergency pools? */ #define __GFP_IO ((__force gfp_t)0x40u)/* Can start physical IO? */ #define __GFP_FS ((__force gfp_t)0x80u)/* Can call down to low-level FS? */ +#define __GFP_COLD ((__force gfp_t)0x100u)/* Cache-cold page required */ #define __GFP_NOWARN ((__force gfp_t)0x200u)/* Suppress page allocation failure warning */ #define __GFP_REPEAT ((__force gfp_t)0x400u)/* Retry the allocation. Might fail */ #define __GFP_NOFAIL ((__force gfp_t)0x800u)/* Retry for ever. Cannot fail */ +#define __GFP_NORETRY ((__force gfp_t)0x1000u)/* Do not retry. Might fail */ #define __GFP_NO_GROW ((__force gfp_t)0x2000u)/* Slab internal usage */ #define __GFP_COMP ((__force gfp_t)0x4000u)/* Add compound page metadata */ #define __GFP_ZERO ((__force gfp_t)0x8000u)/* Return zeroed page on success */ +#define __GFP_NOMEMALLOC ((__force gfp_t)0x10000u)/* Don't use emergency reserves */ #define __GFP_HARDWALL ((__force gfp_t)0x20000u)/* Enforce hardwall cpuset memory allocs */ #define __GFP_THISNODE ((__force gfp_t)0x40000u)/* No fallback, no policies */ +#define __GFP_EMERGENCY ((__force gfp_t)0x80000u)/* Use emergency reserves */ #define __GFP_BITS_SHIFT 20/* Room for 20 __GFP_FOO bits */ #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
@@ -54,7 +58,8 @@ struct vm_area_struct;#define GFP_LEVEL_MASK (__GFP_WAIT|__GFP_HIGH|__GFP_IO|__GFP_FS| \ __GFP_COLD|__GFP_NOWARN|__GFP_REPEAT| \ __GFP_NOFAIL|__GFP_NORETRY|__GFP_NO_GROW|__GFP_COMP| \ - __GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE) + __GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE| \ + __GFP_EMERGENCY)/* This equals 0, but use constants in case they ever change */ #define GFP_NOWAIT (GFP_ATOMIC & ~__GFP_HIGH)
mm/internal.h +3 -1
@@ -75,7 +75,9 @@ static int inline gfp_to_alloc_flags(gfp alloc_flags |=ALLOC_HARDER ;if (likely (!(gfp_mask & __GFP_NOMEMALLOC))) { -if (!in_irq () && (p->flags &PF_MEMALLOC )) +if (gfp_mask & __GFP_EMERGENCY) + alloc_flags |=ALLOC_NO_WATERMARKS ; +else if (!in_irq () && (p->flags &PF_MEMALLOC )) alloc_flags |=ALLOC_NO_WATERMARKS ;else if (!in_interrupt () &&unlikely (test_thread_flag (TIF_MEMDIE )))
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/40 mm: allow mempool to fall back to memalloc reserves
In reply to Peter ZijlstraAllow the mempool to use the memalloc reserves when all else fails and the allocation context would otherwise allow it.
--- mm/mempool.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
mm/mempool.c +10 -0
@@ -14,6 +14,7 @@#include <linux/mempool.h> #include <linux/blkdev.h> #include <linux/writeback.h> +#include "internal.h" static void add_element (mempool_t *pool,void *element) {
@@ -229,6 +230,15 @@ repeat_alloc: }spin_unlock_irqrestore (&pool->lock, flags); +/* if we really had right to the emergency reserves try those */ +if (gfp_to_alloc_flags (gfp_mask) &ALLOC_NO_WATERMARKS ) { +if (gfp_temp & __GFP_NOMEMALLOC) { + gfp_temp &= ~(__GFP_NOMEMALLOC|__GFP_NOWARN); + goto repeat_alloc; + }else + gfp_temp |= __GFP_NOMEMALLOC|__GFP_NOWARN; + } +/* We must not sleep in the GFP_ATOMIC case */ if (!(gfp_mask & __GFP_WAIT))return NULL ;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/40 mm: kmem_cache_objsize
In reply to Peter ZijlstraExpost buffer_size in order to allow fair estimates on the actual space used/needed.
--- include/linux/slab.h | 2 ++ mm/slab.c | 16 ++++++++++++++-- mm/slob.c | 18 ++++++++++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-)
include/linux/slab.h +2 -0
@@ -54,6 +54,7 @@ void *kmem_cache_alloc(struct kmem_cachevoid *kmem_cache_zalloc (struct kmem_cache *,gfp_t );void kmem_cache_free (struct kmem_cache *,void *);unsigned int kmem_cache_size (struct kmem_cache *); +unsigned int kmem_cache_objsize (struct kmem_cache *);const char *kmem_cache_name (struct kmem_cache *);int kmem_ptr_validate (struct kmem_cache *cachep,const void *ptr);
@@ -74,6 +75,7 @@ void *__kmalloc(size_t, gfp_t);void *__kzalloc (size_t ,gfp_t );void kfree (const void *);unsigned int ksize (const void *); +unsigned int kobjsize (size_t );/** * kcalloc - allocate memory for an array. The memory is set to zero.
mm/slab.c +14 -2
@@ -3205,12 +3205,12 @@ static inline void *____cache_alloc(stru }#ifdef CONFIG_SLAB_FAIR -static inline int slab_alloc_rank (gfp_t flags) +static __always_inline int slab_alloc_rank(gfp_t flags) {return gfp_to_rank (flags); }#else -static inline int slab_alloc_rank (gfp_t flags) +static __always_inline int slab_alloc_rank(gfp_t flags) {return 0 ; }
@@ -3815,6 +3815,12 @@ unsigned int kmem_cache_size(struct kmem }EXPORT_SYMBOL (kmem_cache_size); +unsigned int kmem_cache_objsize (struct kmem_cache *cachep) +{ +return cachep->buffer_size; +} +EXPORT_SYMBOL_GPL (kmem_cache_objsize); +const char *kmem_cache_name (struct kmem_cache *cachep) {return cachep->name;
@@ -4512,3 +4518,9 @@ unsigned int ksize(const void *objp)return obj_size (virt_to_cache (objp)); } + +unsigned int kobjsize (size_t size) +{ +return kmem_cache_objsize (kmem_find_general_cachep (size,0 )); +} +EXPORT_SYMBOL_GPL (kobjsize);
mm/slob.c +18 -0
@@ -240,6 +240,15 @@ unsigned int ksize(const void *block)return ((slob_t *)block -1 )->units *SLOB_UNIT ; } +unsigned int kobjsize (size_t size) +{ +if (size <PAGE_SIZE ) +return size; + +return PAGE_SIZE <<find_order (size); +} +EXPORT_SYMBOL_GPL (kobjsize); +struct kmem_cache {unsigned int size, align;const char *name;
@@ -321,6 +330,15 @@ unsigned int kmem_cache_size(struct kmem }EXPORT_SYMBOL (kmem_cache_size); +unsigned int kmem_cache_objsize (struct kmem_cache *c) +{ +if (c->size <PAGE_SIZE ) +return c->size + c->align; + +return PAGE_SIZE <<find_order (c->size); +} +EXPORT_SYMBOL_GPL (kmem_cache_objsize); +const char *kmem_cache_name (struct kmem_cache *c) {return c->name;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.
[snip]
mm/slab.c static inline void *____cache_alloc(stru · line 3216
Quoted 12 lines
#ifdef
CONFIG_SLAB_FAIRmm/slab.c +2
-2-static inline int slab_alloc_rank (gfp_t flags) +static __always_inline int slab_alloc_rank(gfp_t flags) {return gfp_to_rank (flags); }#else -static inline int slab_alloc_rank (gfp_t flags) +static __always_inline int slab_alloc_rank(gfp_t flags) {return 0 ; }
Me thinks this hunk doesn't belong in this patch.
Quoted 14 lines
mm/slab.c +6
-0@@ -3815,6 +3815,12 @@ unsigned int kmem_cache_size(struct kmem }EXPORT_SYMBOL (kmem_cache_size); +unsigned int kmem_cache_objsize (struct kmem_cache *cachep) +{ +return cachep->buffer_size; +} +EXPORT_SYMBOL_GPL (kmem_cache_objsize); +const char *kmem_cache_name (struct kmem_cache *cachep) {return cachep->name;
mm/slab.c unsigned int ksize(const void *objp) · line 4526
Quoted 9 lines
mm/slab.c +6
-0@@ -4512,3 +4518,9 @@ unsigned int ksize(const void *objp)return obj_size (virt_to_cache (objp)); } + +unsigned int kobjsize (size_t size) +{ +return kmem_cache_objsize (kmem_find_general_cachep (size,0 )); +} +EXPORT_SYMBOL_GPL (kobjsize);
Looks good to me. Unfortunately, you need to do SLUB as well. Aah, the wonders of three kernel memory allocators... ;-)
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 3 lines · Pekka Enberg
Quoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.
We already have ksize?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Pekka Enberg
Quoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.We already have ksize?
ksize gives the internal size, whereas these give the external size.
I need to know how much space I need to reserve, hence I need the external size; whereas normally you want to know how much space you have available, which is what ksize gives.
Didn't we have this discussion last time?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Pekka Enberg
Quoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.We already have ksize?
Quoted 7 lines
ksize gives the internal size, whereas these give the external size.
I need to know how much space I need to reserve, hence I need the external size; whereas normally you want to know how much space you have available, which is what ksize gives.
Didn't we have this discussion last time?
I was cced on that as far as I can tell.
The name objsize suggests the size of the object not the slab size. If you want this then maybe call it kmem_cache_slab_size. SLUB distinguishes between obj_size which is the size of the struct that is used and slab_size which is the size of the object after alignment, adding debug information etc etc. See also slabinfo.c for a way to calculate theses sizes from user space.
If we really drop SLAB then we wont need this. SLUBs data structures are not opaque.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 13 lines · Christoph Lameter
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Pekka Enberg
Quoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.We already have ksize?
ksize gives the internal size, whereas these give the external size.
Quoted 7 lines
Quoted 5 lines
I need to know how much space I need to reserve, hence I need the external size; whereas normally you want to know how much space you have available, which is what ksize gives.
Didn't we have this discussion last time?
I was cced on that as far as I can tell.
Ah, that might have been, I was collecting Cc's but must've overlooked you. My bad.
Quoted 6 lines
The name objsize suggests the size of the object not the slab size. If you want this then maybe call it
kmem_cache_slab_size. SLUB distinguishes betweenobj_sizewhich is the size of the struct that is used andslab_sizewhich is the size of the object after alignment, adding debug information etc etc. See alsoslabinfo.cfor a way to calculate theses sizes from user space.
I'm open to renames, this is what Pekka suggested IIRC.
Quoted 2 lines
If we really drop SLAB then we wont need this. SLUBs data structures are not opaque.
Yeah, I know, I still have to add SLUB support, its high on my TODO list though.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
In reply to Peter ZijlstraQuoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.
If its just an estimate that you are after then I think ksize is sufficient.
The buffer size does not include the other per slab overhead that SLAB needs nor the alignment overhead or the padding. For SLUB you'd be more lucky but there it does not include the per slab padding that exist.
Need to check how this is going to be used. It is difficult to estimate slab use because this depends on the availability of object slots in partial slabs.
I could add a function that tells you how many object you could allocate from a slab without the page allocator becoming involved? It would count the object slots available on the partial slabs.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 12 lines · Christoph Lameter
Quoted 2 lines · Peter Zijlstra
Expost
buffer_sizein order to allow fair estimates on the actual space used/needed.If its just an estimate that you are after then I think ksize is sufficient.
The buffer size does not include the other per slab overhead that SLAB needs nor the alignment overhead or the padding. For SLUB you'd be more lucky but there it does not include the per slab padding that exist.
Quoted 7 lines
Need to check how this is going to be used. It is difficult to estimate slab use because this depends on the availability of object slots in partial slabs.
I could add a function that tells you how many object you could allocate from a slab without the page allocator becoming involved? It would count the object slots available on the partial slabs.
I need to know how many pages to reserve to allocate a given number of items from a given slab; assuming the partial slabs are empty. That is, I need a worst case upper bound.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I could add a function that tells you how many object you could allocate from a slab without the page allocator becoming involved? It would count the object slots available on the partial slabs.
I need to know how many pages to reserve to allocate a given number of items from a given slab; assuming the partial slabs are empty. That is, I need a worst case upper bound.
Ok so you really need the number of objects per page? If you know the number of objects then you can calculate the pages needed which would be the maximum memory needed?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 6 lines · Christoph Lameter
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines
I could add a function that tells you how many object you could allocate from a slab without the page allocator becoming involved? It would count the object slots available on the partial slabs.
Quoted 7 lines
Quoted 3 lines
I need to know how many pages to reserve to allocate a given number of items from a given slab; assuming the partial slabs are empty. That is, I need a worst case upper bound.
Ok so you really need the number of objects per page? If you know the number of objects then you can calculate the pages needed which would be the maximum memory needed?
Yes, that would work.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines
Ok so you really need the number of objects per page? If you know the number of objects then you can calculate the pages needed which would be the maximum memory needed?
Yes, that would work.
Hmmm... Maybe lets have
unsigned kmem_estimate_pages(struct kmem_cache *slab_cache, int objects)
which would calculate the worst case memory scenario for allocation the number of indicated objects?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 6 lines · Christoph Lameter
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines
Ok so you really need the number of objects per page? If you know the number of objects then you can calculate the pages needed which would be the maximum memory needed?
Quoted 8 lines
Quoted 1 line
Yes, that would work.
Hmmm... Maybe lets have
unsigned
kmem_estimate_pages(struct kmem_cache *slab_cache, int objects)which would calculate the worst case memory scenario for allocation the number of indicated objects?
Perfectly fine with me, Pekka, any objections?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 5 lines · Christoph Lameter
Quoted 5 lines
Hmmm... Maybe lets have
unsigned
kmem_estimate_pages(struct kmem_cache *slab_cache, int objects)which would calculate the worst case memory scenario for allocation the number of indicated objects?
Quoted 1 line · Peter Zijlstra
Perfectly fine with me, Pekka, any objections?
Again, slab has no way of actually estimating how many pages you need for a given number of objects. So we end up calculating some upper bound which doesn't belong in mm/slab.c. I am perfectly okay with:
(1) kmem_nr_bytes_per_object which is what Peter has now
or alternatively,
(2) kmem_nr_objects_per_page which I think Christoph suggested
Both of them, the slab knows the answer, and doesn't need to guess. It's up to the caller to figure out what the acceptable upper bound is.
Pekka
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 3 lines · Pekka Enberg
Again, slab has no way of actually estimating how many pages you need for a given number of objects. So we end up calculating some upper bound which doesn't belong in
mm/slab.c. I am perfectly okay with:
It can give a worst case number and that is what he wants.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Pekka Enberg
Again, slab has no way of actually estimating how many pages you need for a given number of objects. So we end up calculating some upper bound which doesn't belong in
mm/slab.c. I am perfectly okay with:It can give a worst case number and that is what he wants.
Sure. But he can calculate that elsewhere instead of bringing it in mm/slab.c where it's no use for anyone else...
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 8 lines · Pekka Enberg
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Pekka Enberg
Again, slab has no way of actually estimating how many pages you need for a
given number of objects. So we end up calculating some upper bound which doesn't belong inmm/slab.c. I am perfectly okay with:
Quoted 4 lines
Quoted 1 line
It can give a worst case number and that is what he wants.
Sure. But he can calculate that elsewhere instead of bringing it in
mm/slab.cwhere it's no use for anyone else...
He is not able to calculate it just using the object size since he does not know where the slab put the slab management structure. And in case of SLUB there is no slab management structure... Which means he would have to special case based on the slab allocator selected.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 4 lines · Christoph Lameter
He is not able to calculate it just using the object size since he does not know where the slab put the slab management structure. And in case of SLUB there is no slab management structure... Which means he would have to special case based on the slab allocator selected.
Let me state this once more: he is interested in _rough approximation_. It makes no sense to me to add this kind of fuzzy logic in the slab. Now, as the slab clearly cannot give a _precise number_ either, it shouldn't be added there.
But, if both of you really want to stick it in mm/slab.c, I guess I won't be too violently opposed to it. It just doesn't make any sense to me.
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
In reply to Christoph LameterQuoted 5 lines · Christoph Lameter
Hmmm... Maybe lets have
unsigned
kmem_estimate_pages(struct kmem_cache *slab_cache, int objects)which would calculate the worst case memory scenario for allocation the number of indicated objects?
IIRC this looks more or less what Peter had initially. I don't like the API because there's no way for slab (perhaps this is different for slub) how many pages you really need due to per-node and per-cpu caches, etc.
It's better that the slab tells you what it actually knows and lets the callers figure out what a worst-case upper bound is.
Pekka
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 6 lines · Pekka Enberg
Quoted 2 lines
which would calculate the worst case memory scenario for allocation the number of indicated objects?
IIRC this looks more or less what Peter had initially. I don't like the API because there's no way for slab (perhaps this is different for slub) how many pages you really need due to per-node and per-cpu caches, etc.
SLAB can calculate exactly how many pages are needed. The per cpu and per node stuff is setup at boot and does not change. We are talking about the worst case scenario here. True in case of an off slab we have additional overhead that would also have to go into worst case scenario.
Quoted 2 lines
It's better that the slab tells you what it actually knows and lets the callers figure out what a worst-case upper bound is.
They do not have the data. For that they would need to know how to deal with alignments, (in case of SLAB) the location of the struct slab, the distinction between the differrent sizes, padding etc. I think this has to be done by the allocator. If we ever have another allocator with another structure then this will nicely isolate that functionality. Otherwise we may have to change the callers depending on how the slab organizes its data.
SLUB organizes its data more effectively so SLUB will return a lower number than SLAB f.e.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 2 lines · Christoph Lameter
SLAB can calculate exactly how many pages are needed. The per cpu and per node stuff is setup at boot and does not change. We are talking about the worst case scenario here. True in case of an off slab we have additional overhead that would also have to go into worst case scenario.
Fair enough. But there's no way it can take into account any slab management structures it needs to allocate. The slab simply doesn't know how many pages are needed to _allocate n amount of objects_.
Peter is interested in a _rough estimate_ so I don't see the point of adding that kind of logic in the slab. It's an API that simply cannot satisfy all its callers which is why I suggested exposing buffer size in the first place (the slab certainly knows how many bytes it needs for one object).
Pekka
Re: [PATCH 08/40] mm: kmem_cache_objsize
Quoted 10 lines · Pekka Enberg
Quoted 5 lines · Christoph Lameter
SLAB can calculate exactly how many pages are needed. The per cpu and per node stuff is setup at boot and does not change. We are talking about the worst case scenario here. True in case of an off slab
we have additional overhead that would also have to go into worst case scenario.Fair enough. But there's no way it can take into account any slab management structures it needs to allocate. The slab simply doesn't know how many pages are needed to
_allocaten amount ofobjects_.
In the worst case we will need need nr_objects / nr_object_per_slab off slab management structures. There is one off slab management object per allocated slab.
Quoted 4 lines
Peter is interested in a
_roughestimate_so I don't see the point of adding that kind of logic in the slab. It's an API that simply cannot satisfy all its callers which is why I suggested exposing buffer size in the first place (the slab certainly knows how many bytes it needs for one object).
But the slab size is not useful to the caller since the caller does not know about off slab structures etc. It is only the SLAB that can calculate the worst case.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/40 mm: optimize gfp_to_rank()
In reply to Peter ZijlstraThe gfp_to_rank() call in the slab allocator severely impacts performance. Hence reduce it to the bone, keeping only what is needed to make the reserve work.
[more AIM9 results go here]
AIM9 test 2.6.21-rc5 2.6.21-rc5-slab1
CONFIG_SLAB_FAIR=y
54 tcp_test 2124.48 +/- 10.85 2137.43 +/- 9.22 12.95 55 udp_test 5204.43 +/- 45.13 5231.59 +/- 56.66 27.16 56 fifo_test 20991.42 +/- 46.71 19675.97 +/- 56.35 1315.44
57 stream_pipe 10024.16 +/- 119.88 9912.53 +/- 75.52 111.63
58 dgram_pipe 9460.18 +/- 119.50 9502.75 +/- 89.06 42.57
59 pipe_cpy 30719.81 +/- 117.01 27885.52 +/- 46.81 2834.28
2.6.21-rc5-slab2
CONFIG_SLAB_FAIR=y 54 tcp_test 2124.48 +/- 10.85 2122.80 +/- 4.70 1.68 55 udp_test 5204.43 +/- 45.13 5136.98 +/- 62.31 67.45 56 fifo_test 20991.42 +/- 46.71 19646.81 +/- 53.61 1344.60 57 stream_pipe 10024.16 +/- 119.88 9940.87 +/- 280.73 83.29 58 dgram_pipe 9460.18 +/- 119.50 9432.69 +/- 250.27 27.49
59 pipe_cpy 30719.81 +/- 117.01 27870.70 +/- 65.50 2849.10
--- mm/internal.h | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-)
mm/internal.h +31 -2
@@ -105,9 +105,38 @@ static inline int alloc_flags_to_rank(inreturn rank; } -static inline int gfp_to_rank (gfp_t gfp_mask) +static __always_inline int gfp_to_rank(gfp_t gfp_mask) { -return alloc_flags_to_rank (gfp_to_alloc_flags (gfp_mask)); +/* +* Although correct this full version takes a ~3% performance hit +* on the network test in aim9. +* +* return alloc_flags_to_rank(gfp_to_alloc_flags(gfp_mask)); +* +* So we go cheat a little. We'll only focus on the correctness of +* rank 0. +*/ + +if (likely (!(gfp_mask & __GFP_NOMEMALLOC))) { +if (gfp_mask & __GFP_EMERGENCY) +return 0 ; +else if (!in_irq () && (current->flags &PF_MEMALLOC )) +return 0 ; +/* +* We skip the TIF_MEMDIE test: +* +* if (!in_interrupt() && unlikely(test_thread_flag(TIF_MEMDIE))) +* return 0; +* +* this will force an alloc but since we are allowed the memory +* that will succeed. This will make this very rare occurence +* very expensive when under severe memory pressure, but it +* seems a valid tradeoff. +*/ + } + +/* Cheat by lumping everybody else in rank 1. */ +return 1 ; }#endif
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/40 selinux: tag avc cache alloc as non-critical
In reply to Peter ZijlstraFailing to allocate a cache entry will only harm performance not correctness.
--- security/selinux/avc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
security/selinux/avc.c +1 -1
@@ -332,7 +332,7 @@ static struct avc_node *avc_alloc_node(v {struct avc_node *node; - node =kmem_cache_zalloc (avc_node_cachep,GFP_ATOMIC ); + node =kmem_cache_zalloc (avc_node_cachep,GFP_ATOMIC |__GFP_NOMEMALLOC);if (!node) goto out;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 11/40 net: wrap sk->sk_backlog_rcv()
In reply to Peter ZijlstraWrap calling sk->sk_backlog_rcv() in a function. This will allow extending the generic sk_backlog_rcv behaviour.
--- include/net/sock.h | 5 +++++ net/core/sock.c | 4 ++-- net/ipv4/tcp.c | 2 +- net/ipv4/tcp_timer.c | 2 +- 4 files changed, 9 insertions(+), 4 deletions(-)
include/net/sock.h +5 -0
@@ -480,6 +480,11 @@ static inline void sk_add_backlog(struct skb->next =NULL ; } +static inline int sk_backlog_rcv (struct sock *sk,struct sk_buff *skb) +{ +return sk->sk_backlog_rcv (sk, skb); +} +#define sk_wait_event (__sk, __timeo, __condition) \ ({ int rc; \ release_sock(__sk); \
net/core/sock.c +2 -2
@@ -290,7 +290,7 @@ int sk_receive_skb(struct sock *sk, stru*/ mutex_acquire (&sk->sk_lock.dep_map,0 ,1 , _RET_IP_); - rc = sk->sk_backlog_rcv (sk, skb); + rc =sk_backlog_rcv (sk, skb);mutex_release (&sk->sk_lock.dep_map,1 , _RET_IP_); }else
@@ -1244,7 +1244,7 @@ static void __release_sock(struct sock *struct sk_buff *next = skb->next; skb->next =NULL ; - sk->sk_backlog_rcv (sk, skb); +sk_backlog_rcv (sk, skb);/* * We are in process context here with softirqs
net/ipv4/tcp.c +1 -1
@@ -1002,7 +1002,7 @@ static void tcp_prequeue_process(struct* necessary */ local_bh_disable ();while ((skb =__skb_dequeue (&tp->ucopy.prequeue)) !=NULL ) - sk->sk_backlog_rcv (sk, skb); +sk_backlog_rcv (sk, skb);local_bh_enable ();/* Clear memory counter. */
net/ipv4/tcp_timer.c +1 -1
@@ -198,7 +198,7 @@ static void tcp_delack_timer(unsigned loNET_INC_STATS_BH (LINUX_MIB_TCPSCHEDULERFAILED );while ((skb =__skb_dequeue (&tp->ucopy.prequeue)) !=NULL ) - sk->sk_backlog_rcv (sk, skb); +sk_backlog_rcv (sk, skb); tp->ucopy.memory =0 ; }
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 12/40 net: packet split receive api
In reply to Peter ZijlstraAdd some packet-split receive hooks.
For one this allows to do NUMA node affine page allocs. Later on these hooks will be extended to do emergency reserve allocations for fragments.
--- drivers/net/e1000/e1000_main.c | 8 ++------ drivers/net/sky2.c | 16 ++++++---------- include/linux/skbuff.h | 23 +++++++++++++++++++++++ net/core/skbuff.c | 20 ++++++++++++++++++++ 4 files changed, 51 insertions(+), 16 deletions(-)
drivers/net/e1000/e1000_main.c +2 -6
@@ -4412,12 +4412,8 @@ e1000_clean_rx_irq_ps(struct e1000_adaptpci_unmap_page (pdev, ps_page_dma->ps_page_dma[j],PAGE_SIZE ,PCI_DMA_FROMDEVICE ); ps_page_dma->ps_page_dma[j] =0 ; -skb_fill_page_desc (skb, j, ps_page->ps_page[j],0 , - length); +skb_add_rx_frag (skb, j, ps_page->ps_page[j],0 , length); ps_page->ps_page[j] =NULL ; - skb->len += length; - skb->data_len += length; - skb->truesize += length; }/* strip the ethernet crc, problem is we're using pages now so
@@ -4623,7 +4619,7 @@ e1000_alloc_rx_buffers_ps(struct e1000_aif (j < adapter->rx_ps_pages) {if (likely (!ps_page->ps_page[j])) { ps_page->ps_page[j] = -alloc_page (GFP_ATOMIC ); +netdev_alloc_page (netdev);if (unlikely (!ps_page->ps_page[j])) { adapter->alloc_rx_buff_failed++; goto no_buffers;
include/linux/skbuff.h +23 -0
@@ -813,6 +813,9 @@ static inline void skb_fill_page_desc(stskb_shinfo (skb)->nr_frags = i +1 ; } +extern void skb_add_rx_frag (struct sk_buff *skb,int i,struct page *page, +int off,int size); +#define SKB_PAGE_ASSERT (skb) BUG_ON(skb_shinfo(skb)->nr_frags)#define SKB_FRAG_ASSERT (skb) BUG_ON(skb_shinfo(skb)->frag_list)#define SKB_LINEAR_ASSERT (skb) BUG_ON(skb_is_nonlinear(skb))
@@ -1148,6 +1151,26 @@ static inline struct sk_buff *netdev_allreturn __netdev_alloc_skb (dev, length,GFP_ATOMIC ); } +extern struct page *__netdev_alloc_page (struct net_device *dev,gfp_t gfp_mask); + +/** +* netdev_alloc_page - allocate a page for ps-rx on a specific device +* @dev: network device to receive on +* +* Allocate a new page node local to the specified device. +* +* %NULL is returned if there is no free memory. +*/ +static inline struct page *netdev_alloc_page (struct net_device *dev) +{ +return __netdev_alloc_page (dev,GFP_ATOMIC ); +} + +static inline void netdev_free_page (struct net_device *dev,struct page *page) +{ +__free_page (page); +} +/** * skb_cow - copy header of skb when it is required * @skb: buffer to cow
net/core/skbuff.c +20 -0
@@ -279,6 +279,24 @@ struct sk_buff *__netdev_alloc_skb(strucreturn skb; } +struct page *__netdev_alloc_page (struct net_device *dev,gfp_t gfp_mask) +{ +int node = dev->dev.parent ?dev_to_node (dev->dev.parent) :-1 ; +struct page *page; + + page =alloc_pages_node (node, gfp_mask,0 ); +return page; +} + +void skb_add_rx_frag (struct sk_buff *skb,int i,struct page *page,int off, +int size) +{ +skb_fill_page_desc (skb, i, page, off, size); + skb->len += size; + skb->data_len += size; + skb->truesize += size; +} +static void skb_drop_list (struct sk_buff **listp) {struct sk_buff *list = *listp;
@@ -2066,6 +2084,8 @@ EXPORT_SYMBOL(kfree_skb);EXPORT_SYMBOL (__pskb_pull_tail);EXPORT_SYMBOL (__alloc_skb);EXPORT_SYMBOL (__netdev_alloc_skb); +EXPORT_SYMBOL (__netdev_alloc_page); +EXPORT_SYMBOL (skb_add_rx_frag);EXPORT_SYMBOL (pskb_copy);EXPORT_SYMBOL (pskb_expand_head);EXPORT_SYMBOL (skb_checksum);
drivers/net/sky2.c +6 -10
@@ -1083,7 +1083,7 @@ static struct sk_buff *sky2_rx_alloc(strskb_reserve (skb,ALIGN (p,RX_SKB_ALIGN ) - p);for (i =0 ; i < sky2->rx_nfrags; i++) { -struct page *page =alloc_page (GFP_ATOMIC ); +struct page *page =netdev_alloc_page (sky2->netdev);if (!page) goto free_partial;
@@ -1972,8 +1972,8 @@ static struct sk_buff *receive_copy(stru }/* Adjust length of skb with fragments to match received data */ -static void skb_put_frags (struct sk_buff *skb,unsigned int hdr_space, -unsigned int length) +static void skb_put_frags (struct sky2_port *sky2,struct sk_buff *skb, +unsigned int hdr_space,unsigned int length) {int i, num_frags;unsigned int size;
@@ -1990,15 +1990,11 @@ static void skb_put_frags(struct sk_buffif (length ==0 ) {/* don't need this page */ -__free_page (frag->page); +netdev_free_page (sky2->netdev, frag->page); --skb_shinfo (skb)->nr_frags; }else { size =min (length, (unsigned )PAGE_SIZE ); - - frag->size = size; - skb->data_len += size; - skb->truesize += size; - skb->len += size; +skb_add_rx_frag (skb, i, frag->page,0 , size); length -= size; } }
@@ -2027,7 +2023,7 @@ static struct sk_buff *receive_new(strucsky2_rx_map_skb (sky2->hw->pdev, re, hdr_space);if (skb_shinfo (skb)->nr_frags) -skb_put_frags (skb, hdr_space, length); +skb_put_frags (sky2, skb, hdr_space, length);else skb_put (skb, length);return skb;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 13/40 net: sk_allocation() - concentrate socket related allocations
In reply to Peter ZijlstraIntroduce sk_allocation(), this function allows to inject sock specific flags to each sock related allocation.
--- include/net/sock.h | 7 ++++++- net/ipv4/tcp_output.c | 11 ++++++----- net/ipv6/tcp_ipv6.c | 14 +++++++++----- 3 files changed, 21 insertions(+), 11 deletions(-)
net/ipv4/tcp_output.c +6 -5
@@ -2011,7 +2011,7 @@ void tcp_send_fin(struct sock *sk) }else {/* Socket is locked, keep trying until memory is available. */ for (;;) { - skb =alloc_skb_fclone (MAX_TCP_HEADER ,GFP_KERNEL ); + skb =alloc_skb_fclone (MAX_TCP_HEADER , sk->sk_allocation);if (skb)break ;yield ();
@@ -2044,7 +2044,7 @@ void tcp_send_active_reset(struct sock *struct sk_buff *skb;/* NOTE: No TCP options attached and we never retransmit this. */ - skb =alloc_skb (MAX_TCP_HEADER , priority); + skb =alloc_skb (MAX_TCP_HEADER ,sk_allocation (sk, priority));if (!skb) {NET_INC_STATS (LINUX_MIB_TCPABORTFAILED );return ;
@@ -2117,7 +2117,8 @@ struct sk_buff * tcp_make_synack(struct__u8 *md5_hash_location;#endif - skb =sock_wmalloc (sk,MAX_TCP_HEADER +15 ,1 ,GFP_ATOMIC ); + skb =sock_wmalloc (sk,MAX_TCP_HEADER +15 ,1 , +sk_allocation (sk,GFP_ATOMIC ));if (skb ==NULL )return NULL ;
@@ -2376,7 +2377,7 @@ void tcp_send_ack(struct sock *sk)* tcp_transmit_skb() will set the ownership to this * sock. */ - buff =alloc_skb (MAX_TCP_HEADER ,GFP_ATOMIC ); + buff =alloc_skb (MAX_TCP_HEADER ,sk_allocation (sk,GFP_ATOMIC ));if (buff ==NULL ) {inet_csk_schedule_ack (sk);inet_csk (sk)->icsk_ack.ato =TCP_ATO_MIN ;
@@ -2418,7 +2419,7 @@ static int tcp_xmit_probe_skb(struct socstruct sk_buff *skb;/* We don't queue it, tcp_transmit_skb() sets ownership. */ - skb =alloc_skb (MAX_TCP_HEADER ,GFP_ATOMIC ); + skb =alloc_skb (MAX_TCP_HEADER ,sk_allocation (sk,GFP_ATOMIC ));if (skb ==NULL )return -1 ;
include/net/sock.h +6 -1
@@ -415,6 +415,11 @@ static inline int sock_flag(struct sockreturn test_bit (flag, &sk->sk_flags); } +static inline gfp_t sk_allocation (struct sock *sk,gfp_t gfp_mask) +{ +return gfp_mask; +} +static inline void sk_acceptq_removed (struct sock *sk) { sk->sk_ack_backlog--;
@@ -1207,7 +1212,7 @@ static inline struct sk_buff *sk_stream_int hdr_len; hdr_len =SKB_DATA_ALIGN (sk->sk_prot->max_header); - skb =alloc_skb_fclone (size + hdr_len, gfp); + skb =alloc_skb_fclone (size + hdr_len,sk_allocation (sk, gfp));if (skb) { skb->truesize += mem;if (sk_stream_wmem_schedule (sk, skb->truesize)) {
net/ipv6/tcp_ipv6.c +9 -5
@@ -581,7 +581,8 @@ static int tcp_v6_md5_do_add(struct sock }else {/* reallocate new list if current one is full. */ if (!tp->md5sig_info) { - tp->md5sig_info =kzalloc (sizeof (*tp->md5sig_info),GFP_ATOMIC ); + tp->md5sig_info =kzalloc (sizeof (*tp->md5sig_info), +sk_allocation (sk,GFP_ATOMIC ));if (!tp->md5sig_info) {kfree (newkey);return -ENOMEM ;
@@ -590,7 +591,8 @@ static int tcp_v6_md5_do_add(struct socktcp_alloc_md5sig_pool ();if (tp->md5sig_info->alloced6 == tp->md5sig_info->entries6) { keys =kmalloc ((sizeof (tp->md5sig_info->keys6[0 ]) * - (tp->md5sig_info->entries6 +1 )),GFP_ATOMIC ); + (tp->md5sig_info->entries6 +1 )), +sk_allocation (sk,GFP_ATOMIC ));if (!keys) {tcp_free_md5sig_pool ();
@@ -715,7 +717,7 @@ static int tcp_v6_parse_md5_keys (structstruct tcp_sock *tp =tcp_sk (sk);struct tcp_md5sig_info *p; - p =kzalloc (sizeof (struct tcp_md5sig_info ),GFP_KERNEL ); + p =kzalloc (sizeof (struct tcp_md5sig_info ), sk->allocation);if (!p)return -ENOMEM ;
@@ -1011,7 +1013,7 @@ static void tcp_v6_send_reset(struct soc*/ buff =alloc_skb (MAX_HEADER +sizeof (struct ipv6hdr ) + tot_len, -GFP_ATOMIC ); +sk_allocation (sk,GFP_ATOMIC ));if (buff ==NULL )return ;
@@ -1090,10 +1092,12 @@ static void tcp_v6_send_ack(struct tcp_tstruct tcp_md5sig_key *key;struct tcp_md5sig_key tw_key;#endif +gfp_t gfp_mask =GFP_ATOMIC ;#ifdef CONFIG_TCP_MD5SIG if (!tw && skb->sk) { key =tcp_v6_md5_do_lookup (skb->sk, &ipv6_hdr (skb)->daddr); + gfp_mask =sk_allocation (skb->sk, gfp_mask); }else if (tw && tw->tw_md5_keylen) { tw_key.key = tw->tw_md5_key; tw_key.keylen = tw->tw_md5_keylen;
@@ -1111,7 +1115,7 @@ static void tcp_v6_send_ack(struct tcp_t#endif buff =alloc_skb (MAX_HEADER +sizeof (struct ipv6hdr ) + tot_len, -GFP_ATOMIC ); + gfp_mask);if (buff ==NULL )return ;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 14/40 netvm: link network to vm layer
In reply to Peter ZijlstraHook up networking to the memory reserve.
There are two kinds of reserves: skb and aux.
- skb reserves are used for incomming packets,
- aux reserves are used for processing these packets.
The consumers for these reserves are sockets marked with:
SOCK_VMIO
Such sockets are to be used to service the VM (iow. to swap over). They must be handled kernel side, exposing such a socket to user-space is a BUG.
--- include/net/sock.h | 43 ++++++++++++++++ net/Kconfig | 3 + net/core/sock.c | 135 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 180 insertions(+), 1 deletion(-)
include/net/sock.h +42 -1
@@ -49,6 +49,7 @@#include <linux/skbuff.h> /* struct sk_buff */ #include <linux/mm.h> #include <linux/security.h> +#include <linux/log2.h> #include <linux/filter.h>
@@ -393,6 +394,7 @@ enum sock_flags {SOCK_RCVTSTAMPNS ,/* %SO_TIMESTAMPNS setting */ SOCK_LOCALROUTE ,/* route locally only, %SO_DONTROUTE setting */ SOCK_QUEUE_SHRUNK ,/* write queue has been shrunk recently */ + SOCK_VMIO, /* the VM depends on us - make sure we're serviced */ };static inline void sock_copy_flags (struct sock *nsk,struct sock *osk)
@@ -415,9 +417,48 @@ static inline int sock_flag(struct sockreturn test_bit (flag, &sk->sk_flags); } +static inline int sk_has_vmio (struct sock *sk) +{ +return sock_flag (sk,SOCK_VMIO ); +} + +/* +* Guestimate the per request queue TX upper bound. +* +* Max packet size is 64k, and we need to reserve that much since the data +* might need to bounce it. Double it to be on the safe side. +*/ +#define TX_RESERVE_PAGES DIV_ROUND_UP(2*65536, PAGE_SIZE) + +extern atomic_t vmio_socks; + +static inline int sk_vmio_socks (void ) +{ +return atomic_read (&vmio_socks); +} + +extern int rx_emergency_get (int bytes); +extern int rx_emergency_get_overcommit (int bytes); +extern void rx_emergency_put (int bytes); + +static inline +int guess_kmem_cache_pages (struct kmem_cache *cachep,int nr_objs) +{ +int guess =DIV_ROUND_UP ((kmem_cache_objsize (cachep) * nr_objs), +PAGE_SIZE ); + guess +=ilog2 (guess); +return guess; +} + +extern void sk_adjust_memalloc (int socks,int tx_reserve_pages); +extern void skb_reserve_memory (int skb_reserve_bytes); +extern void aux_reserve_memory (int aux_reserve_pages); +extern int sk_set_vmio (struct sock *sk); +extern int sk_clear_vmio (struct sock *sk); +static inline gfp_t sk_allocation (struct sock *sk,gfp_t gfp_mask) { -return gfp_mask; +return gfp_mask | (sk->sk_allocation & __GFP_EMERGENCY); }static inline void sk_acceptq_removed (struct sock *sk)
net/core/sock.c +135 -0
153 lines not on this page · raw message
net/Kconfig +3 -0
@@ -224,6 +224,9 @@ source "net/ieee80211/Kconfig" endmenu +config NETVM + def_bool n + endif # if NET endmenu # Networking
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 15/40 netvm: INET reserves.
In reply to Peter ZijlstraAdd reserves for INET.
The two big users seem to be the route cache and ip-fragment cache.
Account the route cache to the auxillary reserve. Account the fragments to the skb reserve so that one can at least overflow the fragment cache (avoids fragment deadlocks).
--- net/ipv4/ip_fragment.c | 1 + net/ipv4/route.c | 19 ++++++++++++++++++- net/ipv4/sysctl_net_ipv4.c | 14 +++++++++++++- net/ipv6/reassembly.c | 1 + net/ipv6/route.c | 19 ++++++++++++++++++- net/ipv6/sysctl_net_ipv6.c | 13 ++++++++++++- 6 files changed, 63 insertions(+), 4 deletions(-)
net/ipv4/sysctl_net_ipv4.c +13 -1
@@ -18,6 +18,7 @@#include <net/route.h> #include <net/tcp.h> #include <net/cipso_ipv4.h> +#include <net/sock.h> /* From af_inet.c */ extern int sysctl_ip_nonlocal_bind;
@@ -186,6 +187,17 @@ static int strategy_allowed_congestion_c } +static int proc_dointvec_fragment (ctl_table *table,int write,struct file *filp, +void __user *buffer,size_t *lenp,loff_t *ppos) +{ +int ret; +int old_thresh = *(int *)table->data; + ret =proc_dointvec (table,write,filp,buffer,lenp,ppos); +if (write) +skb_reserve_memory (*(int *)table->data - old_thresh); +return ret; +} +ctl_table ipv4_table[] = { { .ctl_name =NET_IPV4_TCP_TIMESTAMPS ,
@@ -291,7 +303,7 @@ ctl_table ipv4_table[] = { .data = &sysctl_ipfrag_high_thresh, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = &proc_dointvec + .proc_handler = &proc_dointvec_fragment }, { .ctl_name = NET_IPV4_IPFRAG_LOW_THRESH,
net/ipv6/sysctl_net_ipv6.c +12 -1
@@ -15,6 +15,17 @@ #ifdef CONFIG_SYSCTL +static int proc_dointvec_fragment(ctl_table *table, int write, struct file *filp, + void __user *buffer, size_t *lenp, loff_t *ppos) +{ + int ret; + int old_thresh = *(int *)table->data; + ret = proc_dointvec(table,write,filp,buffer,lenp,ppos); + if (write) + skb_reserve_memory(*(int *)table->data - old_thresh); + return ret; +} + static ctl_table ipv6_table[] = { { .ctl_name = NET_IPV6_ROUTE,
@@ -44,7 +55,7 @@ static ctl_table ipv6_table[] = { .data = &sysctl_ip6frag_high_thresh, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = &proc_dointvec + .proc_handler = &proc_dointvec_fragment }, { .ctl_name = NET_IPV6_IP6FRAG_LOW_THRESH,
net/ipv4/ip_fragment.c +1 -0
@@ -743,6 +743,7 @@ void ipfrag_init(void) ipfrag_secret_timer.function = ipfrag_secret_rebuild; ipfrag_secret_timer.expires = jiffies + sysctl_ipfrag_secret_interval;add_timer (&ipfrag_secret_timer); +skb_reserve_memory (sysctl_ipfrag_high_thresh); }EXPORT_SYMBOL (ip_defrag);
net/ipv6/reassembly.c +1 -0
@@ -772,4 +772,5 @@ void __init ipv6_frag_init(void) ip6_frag_secret_timer.function = ip6_frag_secret_rebuild; ip6_frag_secret_timer.expires = jiffies + sysctl_ip6frag_secret_interval;add_timer (&ip6_frag_secret_timer); +skb_reserve_memory (sysctl_ip6frag_high_thresh); }
net/ipv4/route.c +18 -1
@@ -2884,6 +2884,21 @@ static int ipv4_sysctl_rtcache_flush_strreturn 0 ; } +static int proc_dointvec_rt_size (ctl_table *table,int write,struct file *filp, +void __user *buffer,size_t *lenp,loff_t *ppos) +{ +int ret; +int new_pages; +int old_pages =guess_kmem_cache_pages (ipv4_dst_ops.kmem_cachep, + *(int *)table->data); + ret =proc_dointvec (table,write,filp,buffer,lenp,ppos); + new_pages =guess_kmem_cache_pages (ipv4_dst_ops.kmem_cachep, + *(int *)table->data); +if (write && (new_pages - old_pages)) +aux_reserve_memory (new_pages - old_pages); +return ret; +} +ctl_table ipv4_route_table[] = { { .ctl_name =NET_IPV4_ROUTE_FLUSH ,
@@ -2926,7 +2941,7 @@ ctl_table ipv4_route_table[] = { .data = &ip_rt_max_size, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = &proc_dointvec, + .proc_handler = &proc_dointvec_rt_size, }, {/* Deprecated. Use gc_min_interval_ms */
@@ -3153,6 +3168,8 @@ int __init ip_rt_init(void) ipv4_dst_ops.gc_thresh = (rt_hash_mask +1 ); ip_rt_max_size = (rt_hash_mask +1 ) *16 ; +aux_reserve_memory (guess_kmem_cache_pages (ipv4_dst_ops.kmem_cachep, + ip_rt_max_size));devinet_init ();ip_fib_init ();
net/ipv6/route.c +18 -1
@@ -2370,6 +2370,21 @@ int ipv6_sysctl_rtcache_flush(ctl_tablereturn -EINVAL ; } +static int proc_dointvec_rt_size (ctl_table *table,int write,struct file *filp, +void __user *buffer,size_t *lenp,loff_t *ppos) +{ +int ret; +int new_pages; +int old_pages =guess_kmem_cache_pages (ip6_dst_ops.kmem_cachep, + *(int *)table->data); + ret =proc_dointvec (table,write,filp,buffer,lenp,ppos); + new_pages =guess_kmem_cache_pages (ip6_dst_ops.kmem_cachep, + *(int *)table->data); +if (write && (new_pages - old_pages)) +aux_reserve_memory (new_pages - old_pages); +return ret; +} +ctl_table ipv6_route_table[] = { { .ctl_name =NET_IPV6_ROUTE_FLUSH ,
@@ -2393,7 +2408,7 @@ ctl_table ipv6_route_table[] = { .data = &ip6_rt_max_size, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = &proc_dointvec, + .proc_handler = &proc_dointvec_rt_size, }, { .ctl_name = NET_IPV6_ROUTE_GC_MIN_INTERVAL,
@@ -2478,6 +2493,8 @@ void __init ip6_route_init(void)proc_net_fops_create ("rt6_stats" ,S_IRUGO , &rt6_stats_seq_fops);#endif +aux_reserve_memory (guess_kmem_cache_pages (ip6_dst_ops.kmem_cachep, + ip6_rt_max_size));#ifdef CONFIG_XFRM xfrm6_init ();#endif
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 16/40 netvm: hook skb allocation to reserves
In reply to Peter ZijlstraChange the skb allocation api to indicate RX usage and use this to fall back to the reserve when needed. Skbs allocated from the reserve are tagged in skb->emergency.
Teach all other skb ops about emergency skbs and the reserve accounting.
Use the (new) packet split API to allocate and track fragment pages from the emergency reserve. Do this using an atomic counter in page->index. This is needed because the fragments have a different sharing semantic than that indicated by skb_shinfo()->dataref.
(NOTE the extra atomic overhead is only for those pages allocated from the reserves - it does not affect the normal fast path.)
--- include/linux/skbuff.h | 22 +++++- net/core/skbuff.c | 161 ++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 157 insertions(+), 26 deletions(-)
include/linux/skbuff.h +17 -5
@@ -277,7 +277,8 @@ struct sk_buff { nfctinfo:3 ; __u8 pkt_type:3, fclone:2, - ipvs_property:1; + ipvs_property:1, + emergency:1;__be16 protocol;void (*destructor)(struct sk_buff *skb);
@@ -323,10 +324,19 @@ struct sk_buff {#include <asm/system.h> +#define SKB_ALLOC_FCLONE 0x01 +#define SKB_ALLOC_RX 0x02 + +#ifdef CONFIG_NETVM +#define skb_emergency (skb) unlikely((skb)->emergency) +#else +#define skb_emergency (skb) false +#endif +extern void kfree_skb (struct sk_buff *skb);extern void __kfree_skb (struct sk_buff *skb);extern struct sk_buff *__alloc_skb (unsigned int size, -gfp_t priority,int fclone,int node); +gfp_t priority,int flags,int node);static inline struct sk_buff *alloc_skb (unsigned int size,gfp_t priority) {
@@ -336,7 +346,7 @@ static inline struct sk_buff *alloc_skb(static inline struct sk_buff *alloc_skb_fclone (unsigned int size,gfp_t priority) { -return __alloc_skb (size, priority,1 ,-1 ); +return __alloc_skb (size, priority,SKB_ALLOC_FCLONE ,-1 ); }extern void kfree_skbmem (struct sk_buff *skb);
@@ -1279,7 +1289,8 @@ static inline void __skb_queue_purge(strstatic inline struct sk_buff *__dev_alloc_skb (unsigned int length,gfp_t gfp_mask) { -struct sk_buff *skb =alloc_skb (length +NET_SKB_PAD , gfp_mask); +struct sk_buff *skb = +__alloc_skb (length +NET_SKB_PAD , gfp_mask,SKB_ALLOC_RX ,-1 );if (likely (skb))skb_reserve (skb,NET_SKB_PAD );return skb;
@@ -1325,6 +1336,7 @@ static inline struct sk_buff *netdev_all }extern struct page *__netdev_alloc_page (struct net_device *dev,gfp_t gfp_mask); +extern void __netdev_free_page (struct net_device *dev,struct page *page);/** * netdev_alloc_page - allocate a page for ps-rx on a specific device
@@ -1341,7 +1353,7 @@ static inline struct page *netdev_alloc_static inline void netdev_free_page (struct net_device *dev,struct page *page) { -__free_page (page); +__netdev_free_page (dev, page); }/**
net/core/skbuff.c +140 -21
351 lines not on this page · raw message
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 16/40] netvm: hook skb allocation to reserves
Quoted 14 lines · Peter Zijlstra
Change the skb allocation api to indicate RX usage and use this to fall back to the reserve when needed. Skbs allocated from the reserve are tagged in
skb->emergency.Teach all other skb ops about emergency skbs and the reserve accounting.
Use the (new) packet split API to allocate and track fragment pages from the emergency reserve. Do this using an atomic counter in
page->index. This is needed because the fragments have a different sharing semantic than that indicated by skb_shinfo()->dataref.(NOTE the extra atomic overhead is only for those pages allocated from the reserves - it does not affect the normal fast path.)
Quoted 8 lines
Signed-off-by Peter Zijlstra <[email protected]>--- include/linux/skbuff.h | 22 +++++- net/core/skbuff.c | 161 ++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 157 insertions(+), 26 deletions(-)+#define
skb_alloc_rx(skb)(skb_emergency(skb)?SKB_ALLOC_RX: 0)
skb_alloc_rx seems to imply "alloc an skb for rx", not "gimme the right flags to allocate a skb for rx". Can this be changed to "skb_alloc_rx_flag(skb)", similar to the existing sock_flag() for socks?
- Arnaldo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 17/40 netvm: filter emergency skbs.
In reply to Peter ZijlstraToss all emergency packets not for a SOCK_VMIO socket. This ensures our precious memory reserve doesn't get stuck waiting for user-space.
--- include/net/sock.h | 3 +++ 1 file changed, 3 insertions(+)
include/net/sock.h +3 -0
@@ -926,6 +926,9 @@ static inline int sk_filter(struct sock {int err;struct sk_filter *filter; + +if (skb_emergency (skb) && !sk_has_vmio (sk)) +return -EPERM ; err =security_sock_rcv_skb (sk, skb);if (err)
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 18/40 netvm: prevent a TCP specific deadlock
In reply to Peter ZijlstraIt could happen that all !SOCK_VMIO sockets have buffered so much data that we're over the global rmem limit. This will prevent SOCK_VMIO buffers from receiving data, which will prevent userspace from running, which is needed to reduce the buffered data.
--- include/net/sock.h | 7 ++++--- net/core/stream.c | 5 +++-- net/ipv4/tcp_ipv4.c | 8 ++++++++ net/ipv6/tcp_ipv6.c | 8 ++++++++ 4 files changed, 23 insertions(+), 5 deletions(-)
include/net/sock.h +4 -3
@@ -730,7 +730,8 @@ static inline struct inode *SOCK_INODE(s }extern void __sk_stream_mem_reclaim (struct sock *sk); -extern int sk_stream_mem_schedule (struct sock *sk,int size,int kind); +extern int sk_stream_mem_schedule (struct sock *sk,struct sk_buff *skb, +int size,int kind);#define SK_STREAM_MEM_QUANTUM ((int)PAGE_SIZE)
@@ -757,13 +758,13 @@ static inline void sk_stream_writequeue_static inline int sk_stream_rmem_schedule (struct sock *sk,struct sk_buff *skb) {return (int )skb->truesize <= sk->sk_forward_alloc || -sk_stream_mem_schedule (sk, skb->truesize,1 ); +sk_stream_mem_schedule (sk, skb, skb->truesize,1 ); }static inline int sk_stream_wmem_schedule (struct sock *sk,int size) {return size <= sk->sk_forward_alloc || -sk_stream_mem_schedule (sk, size,0 ); +sk_stream_mem_schedule (sk,NULL , size,0 ); }/* Used by processes to "lock" a socket state, so that
net/core/stream.c +3 -2
@@ -207,7 +207,7 @@ void __sk_stream_mem_reclaim(struct sockEXPORT_SYMBOL (__sk_stream_mem_reclaim); -int sk_stream_mem_schedule (struct sock *sk,int size,int kind) +int sk_stream_mem_schedule (struct sock *sk,struct sk_buff *skb,int size,int kind) {int amt =sk_stream_pages (size);
@@ -224,7 +224,8 @@ int sk_stream_mem_schedule(struct sock */* Over hard limit. */ if (atomic_read (sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2 ]) { sk->sk_prot->enter_memory_pressure (); - goto suppress_allocation; +if (!skb || (skb && !skb_emergency (skb))) + goto suppress_allocation; }/* Under pressure. */
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 19/40 netfilter: notify about NF_QUEUE vs emergency skbs
In reply to Peter ZijlstraAvoid memory getting stuck waiting for userspace, drop all emergency packets. This of course requires the regular storage route to not include an NF_QUEUE target ;-)
--- net/netfilter/core.c | 3 +++ 1 file changed, 3 insertions(+)
net/netfilter/core.c +3 -0
@@ -184,9 +184,12 @@ next_hook: ret =1 ; goto unlock; }else if (verdict ==NF_DROP ) { +drop:kfree_skb (*pskb); ret = -EPERM ; }else if ((verdict &NF_VERDICT_MASK ) ==NF_QUEUE ) { +if (skb_emergency (*pskb)) + goto drop;NFDEBUG ("nf_hook: Verdict = QUEUE.\n" );if (!nf_queue (*pskb, elem, pf, hook, indev, outdev, okfn, verdict >>NF_VERDICT_BITS ))
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 20/40 netvm: skb processing
In reply to Peter ZijlstraIn order to make sure emergency packets receive all memory needed to proceed ensure processing of emergency skbs happens under PF_MEMALLOC.
Use the (new) sk_backlog_rcv() wrapper to ensure this for backlog processing.
Skip taps, since those are user-space again.
--- include/net/sock.h | 4 ++++ net/core/dev.c | 42 +++++++++++++++++++++++++++++++++++++----- net/core/sock.c | 19 +++++++++++++++++++ 3 files changed, 60 insertions(+), 5 deletions(-)
net/core/dev.c +37 -5
@@ -1756,10 +1756,23 @@ int netif_receive_skb(struct sk_buff *skstruct net_device *orig_dev;int ret =NET_RX_DROP ;__be16 type; +unsigned long pflags = current->flags; + +/* Emergency skb are special, they should +* - be delivered to SOCK_VMIO sockets only +* - stay away from userspace +* - have bounded memory usage +* +* Use PF_MEMALLOC as a poor mans memory pool - the grouping kind. +* This saves us from propagating the allocation context down to all +* allocation sites. +*/ +if (skb_emergency (skb)) + current->flags |=PF_MEMALLOC ;/* if we've gotten here through NAPI, check netpoll */ if (skb->dev->poll &&netpoll_rx (skb)) -return NET_RX_DROP ; + goto out;if (!skb->tstamp.tv64)net_timestamp (skb);
@@ -1770,7 +1783,7 @@ int netif_receive_skb(struct sk_buff *sk orig_dev =skb_bond (skb);if (!orig_dev) -return NET_RX_DROP ; + goto out;__get_cpu_var (netdev_rx_stat).total++;
@@ -1789,6 +1802,9 @@ int netif_receive_skb(struct sk_buff *sk }#endif +if (skb_emergency (skb)) + goto skip_taps; +list_for_each_entry_rcu (ptype, &ptype_all, list) {if (!ptype->dev || ptype->dev == skb->dev) {if (pt_prev)
@@ -1797,6 +1813,7 @@ int netif_receive_skb(struct sk_buff *sk } } +skip_taps:#ifdef CONFIG_NET_CLS_ACT if (pt_prev) { ret =deliver_skb (skb, pt_prev, orig_dev);
@@ -1809,16 +1826,28 @@ int netif_receive_skb(struct sk_buff *skif (ret ==TC_ACT_SHOT || (ret ==TC_ACT_STOLEN )) {kfree_skb (skb); - goto out; + goto unlock; } skb->tc_verd =0 ; ncls:#endif +if (skb_emergency (skb)) +switch (skb->protocol) { +case __constant_htons (ETH_P_ARP ): +case __constant_htons (ETH_P_IP ): +case __constant_htons (ETH_P_IPV6 ): +case __constant_htons (ETH_P_8021Q ): +break ; + +default : + goto drop; + } + skb =handle_bridge (skb, &pt_prev, &ret, orig_dev);if (!skb) - goto out; + goto unlock; type = skb->protocol;list_for_each_entry_rcu (ptype, &ptype_base[ntohs (type)&15 ], list) {
@@ -1833,6 +1862,7 @@ ncls:if (pt_prev) { ret = pt_prev->func (skb, skb->dev, pt_prev, orig_dev); }else { +drop:kfree_skb (skb);/* Jamal, now you will not able to escape explaining * me how you were going to use this. :-)
@@ -1840,8 +1870,10 @@ ncls: ret =NET_RX_DROP ; } -out: +unlock:rcu_read_unlock (); +out: +tsk_restore_flags (current, pflags,PF_MEMALLOC );return ret; }
include/net/sock.h +4 -0
@@ -527,10 +527,14 @@ static inline void sk_add_backlog(struct skb->next =NULL ; } +#ifndef CONFIG_NETVM static inline int sk_backlog_rcv (struct sock *sk,struct sk_buff *skb) {return sk->sk_backlog_rcv (sk, skb); } +#else +extern int sk_backlog_rcv (struct sock *sk,struct sk_buff *skb); +#endif #define sk_wait_event (__sk, __timeo, __condition) \ ({ int rc; \
net/core/sock.c +19 -0
@@ -332,6 +332,25 @@ int sk_clear_vmio(struct sock *sk) }EXPORT_SYMBOL_GPL (sk_clear_vmio); +#ifdef CONFIG_NETVM +int sk_backlog_rcv (struct sock *sk,struct sk_buff *skb) +{ +if (skb_emergency (skb)) { +int ret; +unsigned long pflags = current->flags; +/* these should have been dropped before queueing */ +BUG_ON (!sk_has_vmio (sk)); + current->flags |=PF_MEMALLOC ; + ret = sk->sk_backlog_rcv (sk, skb); +tsk_restore_flags (current, pflags,PF_MEMALLOC ); +return ret; + } + +return sk->sk_backlog_rcv (sk, skb); +} +EXPORT_SYMBOL (sk_backlog_rcv); +#endif +static int sock_set_timeout (long *timeo_p,char __user *optval,int optlen) {struct timeval tv;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 21/40 uml: rename arch/um remove_mapping()
In reply to Peter ZijlstraWhen 'include/linux/mm.h' includes 'include/linux/swap.h', the global remove_mapping() definition clashes with the arch/um one.
Rename the arch/um one.
--- arch/um/kernel/physmem.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/um/kernel/physmem.c +3 -3
@@ -160,7 +160,7 @@ int physmem_subst_mapping(void *virt, instatic int physmem_fd =-1 ; -static void remove_mapping (struct phys_desc *desc) +static void um_remove_mapping (struct phys_desc *desc) {void *virt = desc->virt;int err;
@@ -184,7 +184,7 @@ int physmem_remove_mapping(void *virt)if (desc ==NULL )return 0 ; -remove_mapping (desc); +um_remove_mapping (desc);return 1 ; }
@@ -205,7 +205,7 @@ void physmem_forget_descriptor(int fd) page = list_entry(ele, struct phys_desc, list); offset = page->offset; addr = page->virt; -remove_mapping (page); +um_remove_mapping (page); err =os_seek_file (fd, offset);if (err) panic("physmem_forget_descriptor - failed to seek "
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 22/40 mm: prepare swap entry methods for use in page methods
In reply to Peter ZijlstraMove around the swap entry methods in preparation for use from page methods.
Also provide a function to obtain the swap_info_struct backing a swap cache page.
--- include/linux/mm.h | 8 ++++++++ include/linux/swap.h | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/swapops.h | 44 -------------------------------------------- mm/swapfile.c | 1 + 4 files changed, 57 insertions(+), 44 deletions(-)
include/linux/mm.h +8 -0
@@ -17,6 +17,7 @@#include <linux/debug_locks.h> #include <linux/backing-dev.h> #include <linux/mm_types.h> +#include <linux/swap.h> struct mempolicy ;struct anon_vma ;
@@ -586,6 +587,13 @@ static inline struct address_space *pagereturn mapping; } +static inline struct swap_info_struct *page_swap_info (struct page *page) +{ +swp_entry_t swap = { .val =page_private (page) }; +BUG_ON (!PageSwapCache (page)); +return get_swap_info_struct (swp_type (swap)); +} +static inline int PageAnon (struct page *page) {return ((unsigned long )page->mapping &PAGE_MAPPING_ANON ) !=0 ;
include/linux/swap.h +48 -0
60 lines not on this page · raw message
include/linux/swapops.h +0 -44
48 lines not on this page · raw message
mm/swapfile.c +1 -0
@@ -1764,6 +1764,7 @@ get_swap_info_struct(unsigned type) {return &swap_info[type]; } +EXPORT_SYMBOL_GPL (get_swap_info_struct);/* * swap_lock prevents swap_map being freed. Don't grab an extra
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 23/40 mm: add support for non block device backed swap files
In reply to Peter ZijlstraA new addres_space_operations method is added:
int swapfile(struct address_space *, int)
When during sys_swapon() this method is found and returns no error the swapper_space.a_ops will proxy to sis->swap_file->f_mapping->a_ops.
The swapfile method will be used to communicate to the address_space that the VM relies on it, and the address_space should take adequate measures (like reserving memory for mempools or the like).
--- Documentation/filesystems/Locking | 9 ++++++ include/linux/fs.h | 1 include/linux/swap.h | 3 ++ mm/Kconfig | 4 ++ mm/page_io.c | 55 ++++++++++++++++++++++++++++++++++++++ mm/swap_state.c | 5 +++ mm/swapfile.c | 22 ++++++++++++++- 7 files changed, 98 insertions(+), 1 deletion(-)
include/linux/swap.h +3 -0
@@ -163,6 +163,7 @@ enum {SWP_USED = (1 <<0 ),/* is slot in swap_info[] used? */ SWP_WRITEOK = (1 <<1 ),/* ok to write to this swap? */ SWP_ACTIVE = (SWP_USED |SWP_WRITEOK ), +SWP_FILE = (1 <<2 ),/* file swap area */ /* add others here before... */ SWP_SCANNING = (1 << 8), /* refcount in scan_swap_map */ };
@@ -264,6 +265,8 @@ extern void swap_unplug_io_fn(struct bac/* linux/mm/page_io.c */ extern int swap_readpage (struct file *,struct page *);extern int swap_writepage (struct page *page,struct writeback_control *wbc); +extern void swap_sync_page (struct page *page); +extern int swap_set_page_dirty (struct page *page);extern int end_swap_bio_read (struct bio *bio,unsigned int bytes_done,int err);/* linux/mm/swap_state.c */
mm/page_io.c +55 -0
79 lines not on this page · raw message
mm/swap_state.c +5 -0
@@ -26,8 +26,13 @@*/ static const struct address_space_operations swap_aops = { .writepage = swap_writepage, +#ifdef CONFIG_SWAP_FILE + .sync_page = swap_sync_page, + .set_page_dirty = swap_set_page_dirty, +#else .sync_page = block_sync_page, .set_page_dirty = __set_page_dirty_nobuffers, +#endif .migratepage = migrate_page, };
mm/swapfile.c +21 -1
@@ -981,6 +981,13 @@ static void destroy_swap_extents(structlist_del (&se->list);kfree (se); } +#ifdef CONFIG_SWAP_FILE +if (sis->flags &SWP_FILE ) { + sis->flags &= ~SWP_FILE ; + sis->swap_file->f_mapping->a_ops-> +swapfile (sis->swap_file->f_mapping,0 ); + } +#endif }/*
@@ -1073,6 +1080,19 @@ static int setup_swap_extents(struct swa goto done; } +#ifdef CONFIG_SWAP_FILE +if (sis->swap_file->f_mapping->a_ops->swapfile) { + ret = sis->swap_file->f_mapping->a_ops-> +swapfile (sis->swap_file->f_mapping,1 ); +if (!ret) { + sis->flags |=SWP_FILE ; + ret =add_swap_extent (sis,0 , sis->max,0 ); + *span = sis->pages; + } + goto done; + } +#endif + blkbits = inode->i_blkbits; blocks_per_page =PAGE_SIZE >> blkbits;
@@ -1640,7 +1660,7 @@ asmlinkage long sys_swapon(const char __mutex_lock (&swapon_mutex);spin_lock (&swap_lock); - p->flags =SWP_ACTIVE ; + p->flags |=SWP_WRITEOK ; nr_swap_pages += nr_good_pages; total_swap_pages += nr_good_pages;
include/linux/fs.h +1 -0
@@ -428,6 +428,7 @@ struct address_space_operations {int (*migratepage) (struct address_space *,struct page *,struct page *);int (*launder_page) (struct page *); +int (*swapfile)(struct address_space *,int ); };struct backing_dev_info ;
Documentation/filesystems/Locking +9 -0
@@ -172,6 +172,7 @@ prototypes: int (*direct_IO)(int, struct kiocb *, const struct iovec *iov, loff_t offset, unsigned long nr_segs); int (*launder_page) (struct page *); + int (*swapfile) (struct address_space *, int); locking rules: All except set_page_dirty may block
@@ -190,6 +191,7 @@ invalidatepage: no yes releasepage: no yes direct_IO: no launder_page: no yes +swapfile no ->prepare_write(), ->commit_write(), ->sync_page() and ->readpage() may be called from the request handler (/dev/loop).
@@ -289,6 +291,13 @@ cleaned, or an error value if not. Note getting mapped back in and redirtied, it needs to be kept locked across the entire operation. + ->swapfile() will be called with a non zero argument on address spaces +backing non block device backed swapfiles. A return value of zero indicates +success. In which case this address space can be used for backing swapspace. +The swapspace operations will be proxied to the address space operations. +Swapoff will call this method with a zero argument to release the address +space. + Note: currently almost all instances of address_space methods are using BKL for internal serialization and that's one of the worst sources of contention. Normally they are calling library functions (in fs/buffer.c)
mm/Kconfig +3 -0
@@ -166,3 +166,6 @@ config ZONE_DMA_FLAG config SLAB_FAIR def_bool n depends on SLAB + +config SWAP_FILE + def_bool n
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 24/40 mm: methods for teaching filesystems about PG_swapcache pages
In reply to Peter ZijlstraIn order to teach filesystems to handle swap cache pages, two new page functions are introduced:
pgoff_t page_file_index(struct page *); struct address_space *page_file_mapping(struct page *);
page_file_index - gives the offset of this page in the file in PAGE_CACHE_SIZE blocks. Like page->index is for mapped pages, this function also gives the correct index for PG_swapcache pages.
page_file_mapping - gives the mapping backing the actual page; that is for swap cache pages it will give swap_file->f_mapping.
page_offset() is modified to use page_file_index(), so that it will give the expected result, even for PG_swapcache pages.
--- include/linux/mm.h | 25 +++++++++++++++++++++++++ include/linux/pagemap.h | 2 +- 2 files changed, 26 insertions(+), 1 deletion(-)
include/linux/mm.h +25 -0
@@ -594,6 +594,16 @@ static inline struct swap_info_struct *preturn get_swap_info_struct (swp_type (swap)); } +static inline +struct address_space *page_file_mapping (struct page *page) +{ +#ifdef CONFIG_SWAP_FILE +if (unlikely (PageSwapCache (page))) +return page_swap_info (page)->swap_file->f_mapping; +#endif +return page->mapping; +} +static inline int PageAnon (struct page *page) {return ((unsigned long )page->mapping &PAGE_MAPPING_ANON ) !=0 ;
@@ -611,6 +621,21 @@ static inline pgoff_t page_index(struct }/* +* Return the file index of the page. Regular pagecache pages use ->index +* whereas swapcache pages use swp_offset(->private) +*/ +static inline pgoff_t page_file_index (struct page *page) +{ +#ifdef CONFIG_SWAP_FILE +if (unlikely (PageSwapCache (page))) { +swp_entry_t swap = { .val =page_private (page) }; +return swp_offset (swap); + } +#endif +return page->index; +} + +/* * The atomic page->_mapcount, like _count, starts from -1: * so that transitions both from it and to it can be tracked, * using atomic_inc_and_test and atomic_add_negative(-1).
include/linux/pagemap.h +1 -1
@@ -120,7 +120,7 @@ extern void __remove_from_page_cache(str*/ static inline loff_t page_offset (struct page *page) { -return ((loff_t )page->index) <<PAGE_CACHE_SHIFT ; +return ((loff_t )page_file_index (page)) <<PAGE_CACHE_SHIFT ; } static inline pgoff_t linear_page_index(struct vm_area_struct *vma,
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 25/40 nfs: remove mempools
In reply to Peter ZijlstraWith the introduction of the shared dirty page accounting in .19, NFS should not be able to surpise the VM with all dirty pages. Thus it should always be able to free some memory. Hence no more need for mempools.
--- fs/nfs/read.c | 15 +++------------ fs/nfs/write.c | 27 +++++---------------------- 2 files changed, 8 insertions(+), 34 deletions(-)
fs/nfs/read.c +3 -12
@@ -33,13 +33,10 @@ static const struct rpc_call_ops nfs_reastatic const struct rpc_call_ops nfs_read_full_ops;static struct kmem_cache *nfs_rdata_cachep; -static mempool_t *nfs_rdata_mempool; - -#define MIN_POOL_READ (32)struct nfs_read_data *nfs_readdata_alloc (unsigned int pagecount) { -struct nfs_read_data *p =mempool_alloc (nfs_rdata_mempool,GFP_NOFS ); +struct nfs_read_data *p =kmem_cache_alloc (nfs_rdata_cachep,GFP_NOFS );if (p) {memset (p,0 ,sizeof (*p));
@@ -50,7 +47,7 @@ struct nfs_read_data *nfs_readdata_allocelse { p->pagevec =kcalloc (pagecount,sizeof (struct page *),GFP_NOFS );if (!p->pagevec) { -mempool_free (p, nfs_rdata_mempool); +kmem_cache_free (nfs_rdata_cachep, p); p =NULL ; } }
@@ -63,7 +60,7 @@ static void nfs_readdata_rcu_free(struct struct nfs_read_data *p = container_of(head, struct nfs_read_data, task.u.tk_rcu);if (p && (p->pagevec != &p->page_array[0 ]))kfree (p->pagevec); -mempool_free (p, nfs_rdata_mempool); +kmem_cache_free (nfs_rdata_cachep, p); }static void nfs_readdata_free (struct nfs_read_data *rdata)
@@ -590,16 +587,10 @@ int __init nfs_init_readpagecache(void)if (nfs_rdata_cachep ==NULL )return -ENOMEM ; - nfs_rdata_mempool =mempool_create_slab_pool (MIN_POOL_READ , - nfs_rdata_cachep); -if (nfs_rdata_mempool ==NULL ) -return -ENOMEM ; -return 0 ; }void nfs_destroy_readpagecache (void ) { -mempool_destroy (nfs_rdata_mempool);kmem_cache_destroy (nfs_rdata_cachep); }
fs/nfs/write.c +5 -22
@@ -29,9 +29,6 @@#define NFSDBG_FACILITY NFSDBG_PAGECACHE -#define MIN_POOL_WRITE (32) -#define MIN_POOL_COMMIT (4) -/* * Local function declarations */
@@ -45,12 +42,10 @@ static const struct rpc_call_ops nfs_wristatic const struct rpc_call_ops nfs_commit_ops;static struct kmem_cache *nfs_wdata_cachep; -static mempool_t *nfs_wdata_mempool; -static mempool_t *nfs_commit_mempool;struct nfs_write_data *nfs_commit_alloc (void ) { -struct nfs_write_data *p =mempool_alloc (nfs_commit_mempool,GFP_NOFS ); +struct nfs_write_data *p =kmem_cache_alloc (nfs_wdata_cachep,GFP_NOFS );if (p) {memset (p,0 ,sizeof (*p));
@@ -64,7 +59,7 @@ void nfs_commit_rcu_free(struct rcu_head struct nfs_write_data *p = container_of(head, struct nfs_write_data, task.u.tk_rcu);if (p && (p->pagevec != &p->page_array[0 ]))kfree (p->pagevec); -mempool_free (p, nfs_commit_mempool); +kmem_cache_free (nfs_wdata_cachep, p); }void nfs_commit_free (struct nfs_write_data *wdata)
@@ -74,7 +69,7 @@ void nfs_commit_free(struct nfs_write_dastruct nfs_write_data *nfs_writedata_alloc (unsigned int pagecount) { -struct nfs_write_data *p =mempool_alloc (nfs_wdata_mempool,GFP_NOFS ); +struct nfs_write_data *p =kmem_cache_alloc (nfs_wdata_cachep,GFP_NOFS );if (p) {memset (p,0 ,sizeof (*p));
@@ -85,7 +80,7 @@ struct nfs_write_data *nfs_writedata_allelse { p->pagevec =kcalloc (pagecount,sizeof (struct page *),GFP_NOFS );if (!p->pagevec) { -mempool_free (p, nfs_wdata_mempool); +kmem_cache_free (nfs_wdata_cachep, p); p =NULL ; } }
@@ -98,7 +93,7 @@ static void nfs_writedata_rcu_free(struc struct nfs_write_data *p = container_of(head, struct nfs_write_data, task.u.tk_rcu);if (p && (p->pagevec != &p->page_array[0 ]))kfree (p->pagevec); -mempool_free (p, nfs_wdata_mempool); +kmem_cache_free (nfs_wdata_cachep, p); }static void nfs_writedata_free (struct nfs_write_data *wdata)
@@ -1465,16 +1460,6 @@ int __init nfs_init_writepagecache(void)if (nfs_wdata_cachep ==NULL )return -ENOMEM ; - nfs_wdata_mempool =mempool_create_slab_pool (MIN_POOL_WRITE , - nfs_wdata_cachep); -if (nfs_wdata_mempool ==NULL ) -return -ENOMEM ; - - nfs_commit_mempool =mempool_create_slab_pool (MIN_POOL_COMMIT , - nfs_wdata_cachep); -if (nfs_commit_mempool ==NULL ) -return -ENOMEM ; -/* * NFS congestion size, scale with available memory. *
@@ -1500,8 +1485,6 @@ int __init nfs_init_writepagecache(void)void nfs_destroy_writepagecache (void ) { -mempool_destroy (nfs_commit_mempool); -mempool_destroy (nfs_wdata_mempool);kmem_cache_destroy (nfs_wdata_cachep); }
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 26/40 nfs: teach the NFS client how to treat PG_swapcache pages
In reply to Peter ZijlstraReplace all relevant occurences of page->index and page->mapping in the NFS client with the new page_file_index() and page_file_mapping() functions.
--- fs/nfs/file.c | 4 ++-- fs/nfs/internal.h | 7 ++++--- fs/nfs/pagelist.c | 6 +++--- fs/nfs/read.c | 6 +++--- fs/nfs/write.c | 36 ++++++++++++++++++------------------ 5 files changed, 30 insertions(+), 29 deletions(-)
fs/nfs/file.c +2 -2
@@ -310,7 +310,7 @@ static void nfs_invalidate_page(struct pif (offset !=0 )return ;/* Cancel any unstarted writes on this page */ -nfs_wb_page_priority (page->mapping->host, page,FLUSH_INVALIDATE ); +nfs_wb_page_priority (page_file_mapping (page)->host, page,FLUSH_INVALIDATE ); }static int nfs_release_page (struct page *page,gfp_t gfp)
@@ -321,7 +321,7 @@ static int nfs_release_page(struct pagestatic int nfs_launder_page (struct page *page) { -return nfs_wb_page (page->mapping->host, page); +return nfs_wb_page (page_file_mapping (page)->host, page); }const struct address_space_operations nfs_file_aops = {
fs/nfs/pagelist.c +3 -3
@@ -78,11 +78,11 @@ nfs_create_request(struct nfs_open_conte* update_nfs_request below if the region is not locked. */ req->wb_page = page;atomic_set (&req->wb_complete,0 ); - req->wb_index = page->index; + req->wb_index =page_file_index (page);page_cache_get (page);BUG_ON (PagePrivate (page));BUG_ON (!PageLocked (page)); -BUG_ON (page->mapping->host != inode); +BUG_ON (page_file_mapping (page)->host != inode); req->wb_offset = offset; req->wb_pgbase = offset; req->wb_bytes = count;
@@ -367,7 +367,7 @@ void nfs_pageio_complete(struct nfs_page* @nfsi: NFS inode * @head: One of the NFS inode request lists * @dst: Destination list -* @idx_start: lower bound of page->index to scan +* @idx_start: lower bound of page_file_index(page) to scan * @npages: idx_start + npages sets the upper bound to scan. * * Moves elements from one of the inode request lists.
fs/nfs/read.c +3 -3
@@ -463,11 +463,11 @@ static const struct rpc_call_ops nfs_reaint nfs_readpage (struct file *file,struct page *page) {struct nfs_open_context *ctx; -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;int error;dprintk ("NFS: nfs_readpage (%p %ld@%lu)\n" , - page,PAGE_CACHE_SIZE , page->index); + page,PAGE_CACHE_SIZE ,page_file_index (page));nfs_inc_stats (inode,NFSIOS_VFSREADPAGE );nfs_add_stats (inode,NFSIOS_READPAGES ,1 );
@@ -514,7 +514,7 @@ static int readpage_async_filler(void *data, struct page *page) {struct nfs_readdesc *desc = (struct nfs_readdesc *)data; -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;struct nfs_page *new;unsigned int len;
fs/nfs/write.c +18 -18
@@ -121,7 +121,7 @@ static struct nfs_page *nfs_page_find_restatic struct nfs_page *nfs_page_find_request (struct page *page) {struct nfs_page *req =NULL ; -spinlock_t *req_lock = &NFS_I (page->mapping->host)->req_lock; +spinlock_t *req_lock = &NFS_I (page_file_mapping (page)->host)->req_lock;spin_lock (req_lock); req =nfs_page_find_request_locked (page);
@@ -132,13 +132,13 @@ static struct nfs_page *nfs_page_find_re/* Adjust the file length if we're writing beyond the end */ static void nfs_grow_file (struct page *page,unsigned int offset,unsigned int count) { -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;loff_t end, i_size =i_size_read (inode);pgoff_t end_index = (i_size -1 ) >>PAGE_CACHE_SHIFT ; -if (i_size >0 && page->index < end_index) +if (i_size >0 &&page_file_index (page) < end_index)return ; - end = ((loff_t )page->index <<PAGE_CACHE_SHIFT ) + ((loff_t )offset+count); + end =page_offset (page) + ((loff_t )offset+count);if (i_size >= end)return ;nfs_inc_stats (inode,NFSIOS_EXTENDWRITE );
@@ -149,7 +149,7 @@ static void nfs_grow_file(struct page *pstatic void nfs_set_pageerror (struct page *page) {SetPageError (page); -nfs_zap_mapping (page->mapping->host, page->mapping); +nfs_zap_mapping (page_file_mapping (page)->host,page_file_mapping (page)); }/* We can set the PG_uptodate flag if we see that a write request
@@ -181,7 +181,7 @@ static int nfs_writepage_setup(struct nf ret =PTR_ERR (req);if (ret != -EBUSY )return ret; - ret =nfs_wb_page (page->mapping->host, page); + ret =nfs_wb_page (page_file_mapping (page)->host, page);if (ret !=0 )return ret; }
@@ -217,7 +217,7 @@ static int nfs_set_page_writeback(structint ret =test_set_page_writeback (page);if (!ret) { -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;struct nfs_server *nfss =NFS_SERVER (inode); if (atomic_inc_return(&nfss->writeback) >
@@ -229,7 +229,7 @@ static int nfs_set_page_writeback(structstatic void nfs_end_page_writeback (struct page *page) { -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;struct nfs_server *nfss =NFS_SERVER (inode);end_page_writeback (page);
@@ -250,7 +250,7 @@ static int nfs_page_async_flush(struct n struct page *page) {struct nfs_page *req; -struct nfs_inode *nfsi =NFS_I (page->mapping->host); +struct nfs_inode *nfsi =NFS_I (page_file_mapping (page)->host);spinlock_t *req_lock = &nfsi->req_lock;int ret;
@@ -303,7 +303,7 @@ static int nfs_writepage_locked(struct p {struct nfs_pageio_descriptor mypgio, *pgio;struct nfs_open_context *ctx; -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;unsigned offset;int err;
@@ -558,7 +558,7 @@ static void nfs_cancel_commit_list(struc* nfs_scan_commit - Scan an inode for commit requests * @inode: NFS inode to scan * @dst: destination list -* @idx_start: lower bound of page->index to scan. +* @idx_start: lower bound of page_file_index(page) to scan. * @npages: idx_start + npages sets the upper bound to scan. * * Moves requests from the inode's 'commit' request list.
@@ -595,7 +595,7 @@ static inline int nfs_scan_commit(structstatic struct nfs_page *nfs_update_request (struct nfs_open_context * ctx,struct page *page,unsigned int offset,unsigned int bytes) { -struct address_space *mapping = page->mapping; +struct address_space *mapping =page_file_mapping (page);struct inode *inode = mapping->host;struct nfs_inode *nfsi =NFS_I (inode);struct nfs_page *req, *new =NULL ;
@@ -698,7 +698,7 @@ int nfs_flush_incompatible(struct file *nfs_release_request (req);if (!do_flush)return 0 ; - status =nfs_wb_page (page->mapping->host, page); + status =nfs_wb_page (page_file_mapping (page)->host, page); }while (status ==0 );return status; }
@@ -713,7 +713,7 @@ int nfs_updatepage(struct file *file, st unsigned int offset, unsigned int count) {struct nfs_open_context *ctx = (struct nfs_open_context *)file->private_data; -struct inode *inode = page->mapping->host; +struct inode *inode =page_file_mapping (page)->host;int status =0 ;nfs_inc_stats (inode,NFSIOS_VFSUPDATEPAGE );
@@ -964,7 +964,7 @@ static void nfs_writeback_done_partial(s }if (nfs_write_need_commit (data)) { -spinlock_t *req_lock = &NFS_I (page->mapping->host)->req_lock; +spinlock_t *req_lock = &NFS_I (page_file_mapping (page)->host)->req_lock;spin_lock (req_lock);if (test_bit (PG_NEED_RESCHED , &req->wb_flags)) {
@@ -1388,7 +1388,7 @@ int nfs_wb_page_priority(struct inode *iloff_t range_start =page_offset (page);loff_t range_end = range_start + (loff_t)(PAGE_CACHE_SIZE -1 );struct writeback_control wbc = { - .bdi = page->mapping->backing_dev_info, + .bdi =page_file_mapping (page)->backing_dev_info, .sync_mode =WB_SYNC_ALL , .nr_to_write =LONG_MAX , .range_start = range_start,
@@ -1404,7 +1404,7 @@ int nfs_wb_page_priority(struct inode *i }if (!PagePrivate (page))return 0 ; - ret =nfs_sync_mapping_wait (page->mapping, &wbc, how); + ret =nfs_sync_mapping_wait (page_file_mapping (page), &wbc, how);if (ret >=0 )return 0 ; out:
@@ -1422,7 +1422,7 @@ int nfs_wb_page(struct inode *inode, strint nfs_set_page_dirty (struct page *page) { -struct address_space *mapping = page->mapping; +struct address_space *mapping =page_file_mapping (page);struct inode *inode;spinlock_t *req_lock;struct nfs_page *req;
fs/nfs/internal.h +4 -3
@@ -220,13 +220,14 @@ void nfs_super_set_maxbytes(struct superstatic inline unsigned int nfs_page_length (struct page *page) { -loff_t i_size =i_size_read (page->mapping->host); +loff_t i_size =i_size_read (page_file_mapping (page)->host);if (i_size >0 ) { +pgoff_t page_index =page_file_index (page);pgoff_t end_index = (i_size -1 ) >>PAGE_CACHE_SHIFT ; -if (page->index < end_index) +if (page_index < end_index)return PAGE_CACHE_SIZE ; -if (page->index == end_index) +if (page_index == end_index)return ((i_size -1 ) & ~PAGE_CACHE_MASK ) +1 ; }return 0 ;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 27/40 nfs: disable data cache revalidation for swapfiles
In reply to Peter ZijlstraDo as Trond suggested:
http://lkml.org/lkml/2006/8/25/348
Disable NFS data cache revalidation on swap files since it doesn't really make sense to have other clients change the file while you are using it.
Thereby we can stop setting PG_private on swap pages, since there ought to be no further races with invalidate_inode_pages2() to deal with.
And since we cannot set PG_private we cannot use page->private (which is already used by PG_swapcache pages anyway) to store the nfs_page. Thus augment the new nfs_page_find_request logic.
--- fs/nfs/inode.c | 6 ++++++ fs/nfs/write.c | 42 ++++++++++++++++++++++++++---------------- 2 files changed, 32 insertions(+), 16 deletions(-)
fs/nfs/inode.c +6 -0
@@ -722,6 +722,12 @@ int nfs_revalidate_mapping_nolock(structstruct nfs_inode *nfsi =NFS_I (inode);int ret =0 ; +/* +* swapfiles are not supposed to be shared. +*/ +if (IS_SWAPFILE (inode)) + goto out; +if ((nfsi->cache_validity &NFS_INO_REVAL_PAGECACHE ) ||nfs_attribute_timeout (inode) ||NFS_STALE (inode)) { ret =__nfs_revalidate_inode (NFS_SERVER (inode), inode);
fs/nfs/write.c +26 -16
96 lines not on this page · raw message
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 28/40 nfs: enable swap on NFS
In reply to Peter ZijlstraProvide an a_ops->swapfile() implementation for NFS. This will set the NFS socket to SOCK_VMIO and run socket reconnect under PF_MEMALLOC as well as reset SOCK_VMIO before engaging the protocol ->connect() method.
PF_MEMALLOC should allow the allocation of struct socket and related objects and the early (re)setting of SOCK_VMIO should allow us to receive the packets required for the TCP connection buildup.
(swapping continues over a server reset during heavy network traffic)
--- fs/Kconfig | 19 +++++++++++++++ fs/nfs/file.c | 10 ++++++++ include/linux/sunrpc/xprt.h | 5 +++- net/sunrpc/sched.c | 7 ++++- net/sunrpc/xprtsock.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 92 insertions(+), 3 deletions(-)
fs/nfs/file.c +10 -0
@@ -324,6 +324,13 @@ static int nfs_launder_page(struct pagereturn nfs_wb_page (page_file_mapping (page)->host, page); } +#ifdef CONFIG_NFS_SWAP +static int nfs_swapfile (struct address_space *mapping,int enable) +{ +return xs_swapper (NFS_CLIENT (mapping->host)->cl_xprt, enable); +} +#endif +const struct address_space_operations nfs_file_aops = { .readpage = nfs_readpage, .readpages = nfs_readpages,
@@ -338,6 +345,9 @@ const struct address_space_operations nf .direct_IO = nfs_direct_IO, #endif .launder_page = nfs_launder_page, +#ifdef CONFIG_NFS_SWAP + .swapfile = nfs_swapfile, +#endif }; static ssize_t nfs_file_write(struct kiocb *iocb, const struct iovec *iov,
include/linux/sunrpc/xprt.h +4 -1
@@ -151,7 +151,9 @@ struct rpc_xprt {unsigned int max_reqs;/* total slots */ unsigned long state;/* transport state */ unsigned char shutdown : 1, /* being shut down */ - resvport : 1; /* use a reserved port */ + resvport : 1, /* use a reserved port */ + swapper : 1; /* we're swapping over this +transport */ unsigned int bind_index;/* bind function index */ /*
@@ -244,6 +246,7 @@ void xprt_disconnect(struct rpc_xprt **/ struct rpc_xprt *xs_setup_udp (struct sockaddr *addr,size_t addrlen,struct rpc_timeout *to);struct rpc_xprt *xs_setup_tcp (struct sockaddr *addr,size_t addrlen,struct rpc_timeout *to); +int xs_swapper (struct rpc_xprt *xprt,int enable);/* * Reserved bit positions in xprt->state
net/sunrpc/sched.c +5 -2
@@ -755,7 +755,10 @@ static void rpc_async_schedule(struct wovoid *rpc_malloc (struct rpc_task *task,size_t size) {size_t *buf; -gfp_t gfp =RPC_IS_SWAPPER (task) ?GFP_ATOMIC :GFP_NOWAIT ; +gfp_t gfp =GFP_NOWAIT ; + +if (RPC_IS_SWAPPER (task)) + gfp |= __GFP_EMERGENCY; size +=sizeof (size_t );if (size <=RPC_BUFFER_MAXSIZE )
@@ -837,7 +840,7 @@ void rpc_init_task(struct rpc_task *taskstatic struct rpc_task *rpc_alloc_task (void ) { -return (struct rpc_task *)mempool_alloc (rpc_task_mempool,GFP_NOFS ); +return (struct rpc_task *)mempool_alloc (rpc_task_mempool,GFP_NOIO ); }static void rpc_free_task (struct rcu_head *rcu)
net/sunrpc/xprtsock.c +54 -0
106 lines not on this page · raw message
fs/Kconfig +19 -0
@@ -1621,6 +1621,18 @@ config NFS_DIRECTIO causes open() to return EINVAL if a file residing in NFS is opened with the O_DIRECT flag. +config NFS_SWAP + bool "Provide swap over NFS support" + default n + depends on NFS_FS + select SUNRPC_SWAP + help + This option enables swapon to work on files located on NFS mounts. + + For more details, see Documentation/vm_deadlock.txt + + If unsure, say N. + config NFSD tristate "NFS server support" depends on INET
@@ -1746,6 +1758,13 @@ config SUNRPC_BIND34 If unsure, say N to get traditional behavior (version 2 rpcbind requests only). +config SUNRPC_SWAP + def_bool n + depends on SUNRPC + select SLAB_FAIR + select NETVM + select SWAP_FILE + config RPCSEC_GSS_KRB5 tristate "Secure RPC: Kerberos V mechanism (EXPERIMENTAL)" depends on SUNRPC && EXPERIMENTAL
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 29/40 nfs: fix various memory recursions possible with swap over NFS.
In reply to Peter ZijlstraGFP_NOFS is not enough, since swap traffic is IO, hence fall back to GFP_NOIO.
--- fs/nfs/pagelist.c | 2 +- fs/nfs/write.c | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
fs/nfs/write.c +3 -3
@@ -45,7 +45,7 @@ static struct kmem_cache *nfs_wdata_cachstruct nfs_write_data *nfs_commit_alloc (void ) { -struct nfs_write_data *p =kmem_cache_alloc (nfs_wdata_cachep,GFP_NOFS ); +struct nfs_write_data *p =kmem_cache_alloc (nfs_wdata_cachep,GFP_NOIO );if (p) {memset (p,0 ,sizeof (*p));
@@ -69,7 +69,7 @@ void nfs_commit_free(struct nfs_write_dastruct nfs_write_data *nfs_writedata_alloc (unsigned int pagecount) { -struct nfs_write_data *p =kmem_cache_alloc (nfs_wdata_cachep,GFP_NOFS ); +struct nfs_write_data *p =kmem_cache_alloc (nfs_wdata_cachep,GFP_NOIO );if (p) {memset (p,0 ,sizeof (*p));
@@ -78,7 +78,7 @@ struct nfs_write_data *nfs_writedata_allif (pagecount <=ARRAY_SIZE (p->page_array)) p->pagevec = p->page_array;else { - p->pagevec =kcalloc (pagecount,sizeof (struct page *),GFP_NOFS ); + p->pagevec =kcalloc (pagecount,sizeof (struct page *),GFP_NOIO );if (!p->pagevec) {kmem_cache_free (nfs_wdata_cachep, p); p =NULL ;
fs/nfs/pagelist.c +1 -1
@@ -28,7 +28,7 @@ static inline struct nfs_page *nfs_page_alloc (void) {struct nfs_page *p; - p =kmem_cache_alloc (nfs_page_cachep,GFP_KERNEL ); + p =kmem_cache_alloc (nfs_page_cachep,GFP_NOIO );if (p) {memset (p,0 ,sizeof (*p));INIT_LIST_HEAD (&p->wb_list);
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 30/40 nfs: fixup missing error code
In reply to Peter ZijlstraCommit 0b67130149b006628389ff3e8f46be9957af98aa lost the setting of tk_status to -EIO when there was no progress with short reads.
--- fs/nfs/read.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfs/read.c +3 -1
@@ -384,8 +384,10 @@ static int nfs_readpage_retry(struct rpc/* This is a short read! */ nfs_inc_stats (data->inode,NFSIOS_SHORTREAD );/* Has the server at least made some progress? */ -if (resp->count ==0 ) +if (resp->count ==0 ) { + task->tk_status = -EIO ;return 0 ; + }/* Yes, so retry the read at the end of the data */ argp->offset += resp->count;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 30/40] nfs: fixup missing error code
Quoted 14 lines · Peter Zijlstra
Commit
0b67130149b006628389ff3e8f46be9957af98aalost the setting oftk_statusto-EIOwhen there was no progress with short reads.Signed-off-by Peter Zijlstra <[email protected]>--- fs/nfs/read.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Index:
linux-2.6-git/fs/nfs/read.c
===================================================================fs/nfs/read.c +0
-0@@ -384,8 +384,10 @@ static int nfs_readpage_retry(struct rpc/* This is a short read! */
fs/nfs/read.c static int nfs_readpage_retry(struct rpc · line 393
Quoted 10 lines
fs/nfs/read.c +3
-1nfs_inc_stats (data->inode,NFSIOS_SHORTREAD );/* Has the server at least made some progress? */ -if (resp->count ==0 ) +if (resp->count ==0 ) { + task->tk_status = -EIO ;return 0 ; + }/* Yes, so retry the read at the end of the data */ argp->offset += resp->count;
This doesn't look right to me. It is not an error for the NFS server to return 0 bytes. It is usually an indication of EOF. If an error occured, then the NFS server would have returned an error.
Thanx...
ps
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 30/40] nfs: fixup missing error code
Quoted 23 lines · Peter Staubach
Quoted 22 lines · Peter Zijlstra
Commit
0b67130149b006628389ff3e8f46be9957af98aalost the setting oftk_statusto-EIOwhen there was no progress with short reads.Signed-off-by Peter Zijlstra <[email protected]>--- fs/nfs/read.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Index:
linux-2.6-git/fs/nfs/read.c
===================================================================fs/nfs/read.c +3
-1@@ -384,8 +384,10 @@ static int nfs_readpage_retry(struct rpc/* This is a short read! */ nfs_inc_stats (data->inode,NFSIOS_SHORTREAD );/* Has the server at least made some progress? */ -if (resp->count ==0 ) +if (resp->count ==0 ) { + task->tk_status = -EIO ;return 0 ; + }
Quoted 6 lines
fs/nfs/read.c static int nfs_readpage_retry(struct rpc · line 393
Quoted 2 lines
fs/nfs/read.c +0
-0/* Yes, so retry the read at the end of the data */ argp->offset += resp->count;This doesn't look right to me. It is not an error for the NFS server to return 0 bytes. It is usually an indication of EOF. If an error occured, then the NFS server would have returned an error.
Ah, ok; I found this when looking through NFS changelogs, and this change was not changelogged. Consider it dropped.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 31/40 mm: balance_dirty_pages() vs throttle_vm_writeout() deadlock
In reply to Peter ZijlstraIf we have a lot of dirty memory and hit the throttle in balance_dirty_pages() we (potentially) generate a lot of writeback and unstable pages, if however during this writeback we need to reclaim a bit, we might hit throttle_vm_writeout(), which might delay us until the combined total of NR_UNSTABLE_NFS + NR_WRITEBACK falls below the dirty limit.
However unstable pages don't go away automagickally, they need a push. While balance_dirty_pages() does this push, throttle_vm_writeout() doesn't. So we can sit here ad infintum.
Hence I propose to remove the NR_UNSTABLE_NFS count from throttle_vm_writeout().
Akpm's recent GFP checks don't much change this picture, any __GFP_IO|__GFP_FS alloc can still get stalled by this. It turns into a deadlock when swapping over NFS.
--- mm/page-writeback.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
mm/page-writeback.c +1 -2
@@ -320,8 +320,7 @@ void throttle_vm_writeout(gfp_t gfp_mask*/ dirty_thresh += dirty_thresh /10 ;/* wheeee... */ -if (global_page_state (NR_UNSTABLE_NFS ) + -global_page_state (NR_WRITEBACK ) <= dirty_thresh) +if (global_page_state (NR_WRITEBACK ) <= dirty_thresh)break ;congestion_wait (WRITE ,HZ /10 ); }
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 32/40 block: add a swapdev callback to the request_queue
In reply to Peter ZijlstraNetworked storage devices need a swap-on/off callback in order to setup some state and reserve memory. Place the block device callback in the request_queue as suggested by James Bottomley.
--- include/linux/blkdev.h | 19 +++++++++++++++++++ mm/swapfile.c | 4 ++++ 2 files changed, 23 insertions(+)
include/linux/blkdev.h +19 -0
@@ -341,6 +341,7 @@ typedef int (merge_bvec_fn) (request_quetypedef int (issue_flush_fn ) (request_queue_t *,struct gendisk *,sector_t *);typedef void (prepare_flush_fn ) (request_queue_t *,struct request *);typedef void (softirq_done_fn )(struct request *); +typedef int (swapdev_fn )(void *,int );enum blk_queue_state { Queue_down,
@@ -379,6 +380,8 @@ struct request_queueissue_flush_fn *issue_flush_fn;prepare_flush_fn *prepare_flush_fn;softirq_done_fn *softirq_done_fn; +swapdev_fn *swapdev_fn; +void *swapdev_obj;/* * Dispatch queue sorting
@@ -766,6 +769,22 @@ request_queue_t *blk_alloc_queue(gfp_t);request_queue_t *blk_alloc_queue_node (gfp_t ,int );extern void blk_put_queue (request_queue_t *); +static inline +void blk_queue_swapdev (struct request_queue *rq, +swapdev_fn *swapdev_fn,void *swapdev_obj) +{ + rq->swapdev_fn = swapdev_fn; + rq->swapdev_obj = swapdev_obj; +} + +static inline +int blk_queue_swapdev_fn (struct request_queue *rq,int enable) +{ +if (rq->swapdev_fn) +return rq->swapdev_fn (rq->swapdev_obj, enable); +return 0 ; +} +/* * tag stuff */
mm/swapfile.c +4 -0
@@ -1305,6 +1305,7 @@ asmlinkage long sys_swapoff(const char _ inode = mapping->host;if (S_ISBLK (inode->i_mode)) {struct block_device *bdev =I_BDEV (inode); +blk_queue_swapdev_fn (bdev->bd_disk->queue,0 );set_blocksize (bdev, p->old_block_size);bd_release (bdev); }else {
@@ -1524,6 +1525,9 @@ asmlinkage long sys_swapon(const char __ error =set_blocksize (bdev,PAGE_SIZE );if (error <0 ) goto bad_swap; + error =blk_queue_swapdev_fn (bdev->bd_disk->queue,1 ); +if (error <0 ) + goto bad_swap; p->bdev = bdev; }else if (S_ISREG (inode->i_mode)) { p->bdev = inode->i_sb->s_bdev;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 33/40 uml: enable scsi and add iscsi config
In reply to Peter ZijlstraEnable (i)SCSI on UML, dunno why SCSI was deemed broken, it works like a charm.
--- arch/um/Kconfig | 16 -------------- arch/um/Kconfig.scsi | 58 --------------------------------------------------- 2 files changed, 1 insertion(+), 73 deletions(-)
arch/um/Kconfig +1 -15
@@ -317,21 +317,7 @@ source "crypto/Kconfig" source "lib/Kconfig" -menu "SCSI support" -depends on BROKEN - -config SCSI - tristate "SCSI support" - -# This gives us free_dma, which scsi.c wants. -config GENERIC_ISA_DMA - bool - depends on SCSI - default y - -source "arch/um/Kconfig.scsi" - -endmenu +source "drivers/scsi/Kconfig" source "drivers/md/Kconfig"
arch/um/Kconfig.scsi deleted +0 -58
58 lines not on this page · raw message
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 34/40 sock: safely expose kernel sockets to userspace
In reply to Peter ZijlstraSOCK_KERNEL - avoids user-space from actually using this socket for anything. This enables sticking kernel sockets into the files_table for identifying and reference counting purposes.
(iSCSI wants to do this)
--- include/net/sock.h | 1 + net/socket.c | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-)
include/net/sock.h +1 -0
@@ -394,6 +394,7 @@ enum sock_flags {SOCK_LOCALROUTE ,/* route locally only, %SO_DONTROUTE setting */ SOCK_QUEUE_SHRUNK ,/* write queue has been shrunk recently */ SOCK_VMIO ,/* the VM depends on us - make sure we're serviced */ + SOCK_KERNEL, /* userspace cannot touch this socket */ };static inline void sock_copy_flags (struct sock *nsk,struct sock *osk)
net/socket.c +9 -1
@@ -353,7 +353,7 @@ static int sock_alloc_fd(struct file **freturn fd; } -static int sock_attach_fd (struct socket *sock,struct file *file) +static noinline int sock_attach_fd(struct socket *sock, struct file *file) {struct qstr this;char name[32 ];
@@ -381,6 +381,10 @@ static int sock_attach_fd(struct socket file->f_op =SOCK_INODE (sock)->i_fop = &socket_file_ops; file->f_mode =FMODE_READ |FMODE_WRITE ; file->f_flags =O_RDWR ; +if (unlikely (sock->sk &&sock_flag (sock->sk,SOCK_KERNEL ))) { + file->f_mode =0 ; + file->f_flags =0 ; + } file->f_pos =0 ; file->private_data = sock;
@@ -806,6 +810,10 @@ static long sock_ioctl(struct file *fileint pid, err; sock = file->private_data; + +if (unlikely (sock_flag (sock->sk,SOCK_KERNEL ))) +return -EBADF ; +if (cmd >=SIOCDEVPRIVATE && cmd <= (SIOCDEVPRIVATE +15 )) { err =dev_ioctl (cmd, argp); }else
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 35/40 From: Mike Christie <[email protected]>
In reply to Peter ZijlstraThis patch has iscsi_tcp implement a ep_connect callback. We only do the connect for now and let userspace do the poll and close. I do not like the lack of symmetry but doing sys_close in iscsi_tcp felt a little creepy.
This patch also fixes a bug where when iscsid restarts while sessions are running, we leak the ep object. This occurs because iscsid, when it restarts, does not know the connection and ep relationship. To fix this, I just export the ep handle sysfs. Or I converted iser in this patch.
--- drivers/infiniband/ulp/iser/iscsi_iser.c | 4 - drivers/scsi/iscsi_tcp.c | 99 +++++++++++++++++-------------- drivers/scsi/libiscsi.c | 8 ++ drivers/scsi/scsi_transport_iscsi.c | 4 - include/scsi/iscsi_if.h | 4 - include/scsi/libiscsi.h | 3 include/scsi/scsi_transport_iscsi.h | 2 7 files changed, 75 insertions(+), 49 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +3 -1
@@ -332,7 +332,8 @@ iscsi_iser_conn_bind(struct iscsi_cls_sestruct iser_conn *ib_conn;int error; - error =iscsi_conn_bind (cls_session, cls_conn, is_leading); + error =iscsi_conn_bind (cls_session, cls_conn, is_leading, + transport_eph);if (error)return error;
@@ -572,6 +573,7 @@ static struct iscsi_transport iscsi_iserISCSI_PDU_INORDER_EN |ISCSI_DATASEQ_INORDER_EN |ISCSI_EXP_STATSN | +ISCSI_PARAM_EP_HANDLE |ISCSI_PERSISTENT_PORT |ISCSI_PERSISTENT_ADDRESS | ISCSI_TARGET_NAME |
drivers/scsi/iscsi_tcp.c +54 -39
160 lines not on this page · raw message
drivers/scsi/libiscsi.c +7 -1
@@ -1793,7 +1793,8 @@ void iscsi_conn_stop(struct iscsi_cls_coEXPORT_SYMBOL_GPL (iscsi_conn_stop);int iscsi_conn_bind (struct iscsi_cls_session *cls_session, -struct iscsi_cls_conn *cls_conn,int is_leading) +struct iscsi_cls_conn *cls_conn,int is_leading, +uint64_t transport_eph) {struct iscsi_session *session =class_to_transport_session (cls_session);struct iscsi_conn *conn = cls_conn->dd_data;
@@ -1803,6 +1804,8 @@ int iscsi_conn_bind(struct iscsi_cls_ses session->leadconn = conn;spin_unlock_bh (&session->lock); + conn->ep_handle = transport_eph; +/* * Unblock xmitworker(), Login Phase will pass through. */
@@ -1983,6 +1986,9 @@ int iscsi_conn_get_param(struct iscsi_clcase ISCSI_PARAM_PERSISTENT_ADDRESS : len =sprintf (buf,"%s\n" , conn->persistent_address);break ; +case ISCSI_PARAM_EP_HANDLE : + len =sprintf (buf,"%llu\n" , conn->ep_handle); +break ;default :return -ENOSYS ; }
drivers/scsi/scsi_transport_iscsi.c +3 -1
@@ -31,7 +31,7 @@#include <scsi/iscsi_if.h> #define ISCSI_SESSION_ATTRS 11 -#define ISCSI_CONN_ATTRS 11 +#define ISCSI_CONN_ATTRS 12#define ISCSI_HOST_ATTRS 0#define ISCSI_TRANSPORT_VERSION "2.0-724"
@@ -1153,6 +1153,7 @@ iscsi_conn_attr(port, ISCSI_PARAM_CONN_Piscsi_conn_attr (exp_statsn,ISCSI_PARAM_EXP_STATSN );iscsi_conn_attr (persistent_address,ISCSI_PARAM_PERSISTENT_ADDRESS );iscsi_conn_attr (address,ISCSI_PARAM_CONN_ADDRESS ); +iscsi_conn_attr (ep_handle,ISCSI_PARAM_EP_HANDLE );#define iscsi_cdev_to_session (_cdev) \ iscsi_dev_to_session(_cdev->dev)
@@ -1343,6 +1344,7 @@ iscsi_register_transport(struct iscsi_trSETUP_CONN_RD_ATTR (exp_statsn,ISCSI_EXP_STATSN );SETUP_CONN_RD_ATTR (persistent_address,ISCSI_PERSISTENT_ADDRESS );SETUP_CONN_RD_ATTR (persistent_port,ISCSI_PERSISTENT_PORT ); +SETUP_CONN_RD_ATTR (ep_handle,ISCSI_EP_HANDLE );BUG_ON (count >ISCSI_CONN_ATTRS ); priv->conn_attrs[count] =NULL ;
include/scsi/iscsi_if.h +3 -1
@@ -219,9 +219,10 @@ enum iscsi_param {ISCSI_PARAM_PERSISTENT_PORT ,ISCSI_PARAM_SESS_RECOVERY_TMO , -/* pased in through bind conn using transport_fd */ +/* pased in through bind or ep callbacks */ ISCSI_PARAM_CONN_PORT ,ISCSI_PARAM_CONN_ADDRESS , +ISCSI_PARAM_EP_HANDLE ,/* must always be last */ ISCSI_PARAM_MAX,
@@ -249,6 +250,7 @@ enum iscsi_param {#define ISCSI_SESS_RECOVERY_TMO (1 << ISCSI_PARAM_SESS_RECOVERY_TMO)#define ISCSI_CONN_PORT (1 << ISCSI_PARAM_CONN_PORT)#define ISCSI_CONN_ADDRESS (1 << ISCSI_PARAM_CONN_ADDRESS) +#define ISCSI_EP_HANDLE (1 << ISCSI_PARAM_EP_HANDLE)#define iscsi_ptr (_handle) ((void*)(unsigned long)_handle)#define iscsi_handle (_ptr) ((uint64_t)(unsigned long)_ptr)
include/scsi/libiscsi.h +2 -1
@@ -123,6 +123,7 @@ struct iscsi_conn {struct iscsi_cls_conn *cls_conn;/* ptr to class connection */ void *dd_data;/* iscsi_transport data */ struct iscsi_session *session;/* parent session */ +uint64_t ep_handle;/* ep handle */ /* * LLDs should set this lock. It protects the transport recv * code
@@ -281,7 +282,7 @@ extern void iscsi_conn_teardown(struct iextern int iscsi_conn_start (struct iscsi_cls_conn *);extern void iscsi_conn_stop (struct iscsi_cls_conn *,int );extern int iscsi_conn_bind (struct iscsi_cls_session *,struct iscsi_cls_conn *, -int ); +int ,uint64_t transport_eph);extern void iscsi_conn_failure (struct iscsi_conn *conn,enum iscsi_err err);extern int iscsi_conn_get_param (struct iscsi_cls_conn *cls_conn,enum iscsi_param param,char *buf);
include/scsi/scsi_transport_iscsi.h +1 -1
@@ -79,7 +79,7 @@ struct iscsi_transport {char *name;unsigned int caps;/* LLD sets this to indicate what values it can export to sysfs */ -unsigned int param_mask; +uint64_t param_mask;struct scsi_host_template *host_template;/* LLD connection data size */ int conndata_size;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 36/40 iscsi: fixup of the ep_connect patch
In reply to Peter ZijlstraMake sure a malicious user-space program cannot crash the kernel module by prematurely closing the filedesc.
--- drivers/scsi/iscsi_tcp.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
drivers/scsi/iscsi_tcp.c +19 -4
@@ -1830,11 +1830,25 @@ tcp_conn_alloc_fail: }static void +iscsi_tcp_release_conn (struct iscsi_conn *conn) +{ +struct iscsi_tcp_conn *tcp_conn = conn->dd_data; + +if (!tcp_conn->sock) +return ; + +sockfd_put (tcp_conn->sock); + tcp_conn->sock =NULL ; + conn->recv_lock =NULL ; +} + +static void iscsi_tcp_conn_destroy (struct iscsi_cls_conn *cls_conn) {struct iscsi_conn *conn = cls_conn->dd_data;struct iscsi_tcp_conn *tcp_conn = conn->dd_data; +iscsi_tcp_release_conn (conn);iscsi_conn_teardown (cls_conn);if (tcp_conn->tx_hash.tfm)crypto_free_hash (tcp_conn->tx_hash.tfm);
@@ -1851,6 +1865,7 @@ iscsi_tcp_conn_stop(struct iscsi_cls_construct iscsi_tcp_conn *tcp_conn = conn->dd_data;iscsi_conn_stop (cls_conn, flag); +iscsi_tcp_release_conn (conn); tcp_conn->hdr_size =sizeof (struct iscsi_hdr ); }
@@ -1873,8 +1888,10 @@ iscsi_tcp_conn_bind(struct iscsi_cls_ses } err =iscsi_conn_bind (cls_session, cls_conn, is_leading, transport_eph); -if (err) - goto done; +if (err) { +sockfd_put (sock); +return err; + }/* bind iSCSI connection and socket */ tcp_conn->sock = sock;
@@ -1898,8 +1915,6 @@ iscsi_tcp_conn_bind(struct iscsi_cls_ses*/ tcp_conn->in_progress =IN_PROGRESS_WAIT_HEADER ; -done: -sockfd_put (sock);return err; }
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 37/40 iscsi: ensure the iscsi kernel fd is not usable in userspace
In reply to Peter ZijlstraWe expose the iSCSI connection fd to userspace for reference tracking, but we do not want userspace to actually have access to the data; mark it with SOCK_KERNEL.
--- drivers/scsi/iscsi_tcp.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/scsi/iscsi_tcp.c +7 -0
@@ -1759,6 +1759,13 @@ iscsi_tcp_ep_connect(struct sockaddr *ds goto release_sock; } +/* +* Even though we're going to expose this socket to user-space +* (as an identifier for the connection and for tracking life times) +* we don't want it used by user-space at all. +*/ +sock_set_flag (sock->sk,SOCK_KERNEL ); + rc = sock->ops->connect (sock, (struct sockaddr *)dst_addr, size,O_NONBLOCK );if (rc == -EINPROGRESS )
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 38/40 netlink: add SOCK_VMIO support to AF_NETLINK
In reply to Peter ZijlstraModify the netlink code so that SOCK_VMIO has the desired effect on the user-space side of the connection.
Modify sys_{send,recv}msg to use sk->sk_allocation instead of GFP_KERNEL, this should not change existing behaviour because the default of sk->sk_allocation is GFP_KERNEL, and no user-space exposed socket would have it any different at this time.
This change allows the system calls to succeed for SOCK_VMIO sockets (who have sk->sk_allocation |= GFP_EMERGENCY) even under extreme memory pressure.
Since netlink_sendmsg is used to transfer msgs from user- to kernel-space treat the skb allocation there as a receive allocation.
Also export netlink_lookup, this is needed to locate the kernel side struct sock object associated with the user-space netlink socket.
--- include/linux/netlink.h | 1 + net/compat.c | 2 +- net/netlink/af_netlink.c | 12 +++++++++--- net/socket.c | 6 +++--- 4 files changed, 14 insertions(+), 7 deletions(-)
net/netlink/af_netlink.c +9 -3
@@ -203,7 +203,7 @@ netlink_unlock_table(void)wake_up (&nl_table_wait); } -static __inline__struct sock *netlink_lookup (int protocol,u32 pid) +struct sock *netlink_lookup (int protocol,u32 pid) {struct nl_pid_hash *hash = &nl_table[protocol].hash;struct hlist_head *head;
@@ -1157,7 +1157,7 @@ static int netlink_sendmsg(struct kiocbif (len > sk->sk_sndbuf -32 ) goto out; err = -ENOBUFS ; - skb =alloc_skb (len,GFP_KERNEL ); + skb =__alloc_skb (len,GFP_KERNEL ,SKB_ALLOC_RX ,-1 );if (skb==NULL ) goto out;
@@ -1186,8 +1186,13 @@ static int netlink_sendmsg(struct kiocb }if (dst_group) { +gfp_t gfp_mask = sk->sk_allocation; + +if (skb_emergency (skb)) + gfp_mask |= __GFP_EMERGENCY; +atomic_inc (&skb->users); -netlink_broadcast (sk, skb, dst_pid, dst_group,GFP_KERNEL ); +netlink_broadcast (sk, skb, dst_pid, dst_group, gfp_mask); } err =netlink_unicast (sk, skb, dst_pid, msg->msg_flags&MSG_DONTWAIT );
@@ -1850,6 +1855,7 @@ panic:core_initcall (netlink_proto_init); +EXPORT_SYMBOL (netlink_lookup);EXPORT_SYMBOL (netlink_ack);EXPORT_SYMBOL (netlink_run_queue);EXPORT_SYMBOL (netlink_broadcast);
net/socket.c +3 -3
@@ -1817,7 +1817,7 @@ asmlinkage long sys_sendmsg(int fd, stru err = -ENOMEM ; iov_size = msg_sys.msg_iovlen *sizeof (struct iovec );if (msg_sys.msg_iovlen >UIO_FASTIOV ) { - iov =sock_kmalloc (sock->sk, iov_size,GFP_KERNEL ); + iov =sock_kmalloc (sock->sk, iov_size, sock->sk->sk_allocation);if (!iov) goto out_put; }
@@ -1846,7 +1846,7 @@ asmlinkage long sys_sendmsg(int fd, stru ctl_len = msg_sys.msg_controllen; }else if (ctl_len) {if (ctl_len >sizeof (ctl)) { - ctl_buf =sock_kmalloc (sock->sk, ctl_len,GFP_KERNEL ); + ctl_buf =sock_kmalloc (sock->sk, ctl_len, sock->sk->sk_allocation);if (ctl_buf ==NULL ) goto out_freeiov; }
@@ -1922,7 +1922,7 @@ asmlinkage long sys_recvmsg(int fd, stru err = -ENOMEM ; iov_size = msg_sys.msg_iovlen *sizeof (struct iovec );if (msg_sys.msg_iovlen >UIO_FASTIOV ) { - iov =sock_kmalloc (sock->sk, iov_size,GFP_KERNEL ); + iov =sock_kmalloc (sock->sk, iov_size, sock->sk->sk_allocation);if (!iov) goto out_put; }
include/linux/netlink.h +1 -0
@@ -157,6 +157,7 @@ struct netlink_skb_parms#define NETLINK_CREDS (skb) (&NETLINK_CB((skb)).creds) +extern struct sock *netlink_lookup (int protocol,__u32 pid); extern struct sock *netlink_kernel_create(int unit, unsigned int groups, void (*input)(struct sock *sk, int len), struct mutex *cb_mutex,
net/compat.c +1 -1
@@ -169,7 +169,7 @@ int cmsghdr_from_user_compat_to_kern(str* from the user. */ if (kcmlen > stackbuf_size) - kcmsg_base = kcmsg =sock_kmalloc (sk, kcmlen,GFP_KERNEL ); + kcmsg_base = kcmsg =sock_kmalloc (sk, kcmlen, sk->sk_allocation);if (kcmsg ==NULL )return -ENOBUFS ;
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 39/40 mm: a process flags to avoid blocking allocations
In reply to Peter ZijlstraPF_MEM_NOWAIT - will make allocations fail before blocking. This is usefull to convert process behaviour to non-blocking.
--- include/linux/sched.h | 1 + kernel/softirq.c | 4 ++-- mm/internal.h | 11 ++++++++++- mm/page_alloc.c | 4 ++-- 4 files changed, 15 insertions(+), 5 deletions(-)
include/linux/sched.h +1 -0
@@ -1158,6 +1158,7 @@ static inline void put_task_struct(struc#define PF_SPREAD_SLAB 0x02000000/* Spread some slab caches over cpuset */ #define PF_MEMPOLICY 0x10000000/* Non-default NUMA mempolicy */ #define PF_MUTEX_TESTER 0x20000000/* Thread belongs to the rt mutex tester */ +#define PF_MEM_NOWAIT 0x40000000/* Make allocations fail instead of block */ /* * Only the _current_ task can read/write to tsk->flags, but other
mm/page_alloc.c +2 -2
@@ -1234,11 +1234,11 @@ struct page * fastcall __alloc_pages(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist) { - const gfp_t wait = gfp_mask & __GFP_WAIT; + struct task_struct *p = current; +const bool wait =gfp_wait (gfp_mask);struct zone **z;struct page *page;struct reclaim_state reclaim_state; -struct task_struct *p = current;int do_retry;int alloc_flags;int did_some_progress;
mm/internal.h +10 -1
@@ -46,6 +46,15 @@ extern void fastcall __init __free_pages#define ALLOC_NO_WATERMARKS 0x20/* don't check watermarks at all */ #define ALLOC_CPUSET 0x40/* check for correct cpuset */ +static bool inline gfp_wait (gfp_t gfp_mask) +{ +bool wait = gfp_mask & __GFP_WAIT; +if (wait && !in_irq () && (current->flags &PF_MEM_NOWAIT )) + wait = false; + +return wait; +} +/* * get the deepest reaching allocation flags for the given gfp_mask */
@@ -53,7 +62,7 @@ static int inline gfp_to_alloc_flags(gfp {struct task_struct *p = current;int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ; -const gfp_t wait = gfp_mask & __GFP_WAIT; +const bool wait =gfp_wait (gfp_mask);/* * The caller may dip into page reserves a bit more if the caller
kernel/softirq.c +2 -2
@@ -211,7 +211,7 @@ asmlinkage void __do_softirq(void)int max_restart =MAX_SOFTIRQ_RESTART ;int cpu;unsigned long pflags = current->flags; - current->flags &= ~PF_MEMALLOC ; + current->flags &= ~(PF_MEMALLOC |PF_MEM_NOWAIT ); pending =local_softirq_pending ();account_system_vtime (current);
@@ -250,7 +250,7 @@ restart:account_system_vtime (current);_local_bh_enable (); -tsk_restore_flags (current, pflags,PF_MEMALLOC ); +tsk_restore_flags (current, pflags, (PF_MEMALLOC |PF_MEM_NOWAIT )); }#ifndef __ARCH_HAS_DO_SOFTIRQ
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 40/40 iscsi: support for swapping over iSCSI.
In reply to Peter ZijlstraSet blk_queue_swapdev for iSCSI. This method takes care of reserving the extra memory needed and marking all relevant sockets with SOCK_VMIO.
When used for swapping, TCP socket creation is done under GFP_MEMALLOC and the TCP connect is done with SOCK_VMIO to ensure their success.
Also the netlink userspace interface is marked SOCK_VMIO, this will ensure that even under pressure we can still communicate with the daemon (which runs as mlockall() and needs no additional memory to operate).
Netlink requests are handled under the new PF_MEM_NOWAIT when a swapper is present. This ensures that the netlink socket will not block. User-space will need to retry failed requests.
The TCP receive path is handled under PF_MEMALLOC for SOCK_VMIO sockets. This makes sure we do not block the critical socket, and that we do not fail to process incoming data.
--- drivers/scsi/Kconfig | 17 ++++++++ drivers/scsi/iscsi_tcp.c | 70 ++++++++++++++++++++++++++++++++--- drivers/scsi/libiscsi.c | 18 ++++++--- drivers/scsi/qla4xxx/ql4_os.c | 2 - drivers/scsi/scsi_transport_iscsi.c | 72 ++++++++++++++++++++++++++++++++---- include/scsi/scsi_transport_iscsi.h | 12 +++++- 6 files changed, 170 insertions(+), 21 deletions(-)
drivers/scsi/iscsi_tcp.c +65 -5
@@ -42,6 +42,7 @@#include <scsi/scsi_host.h> #include <scsi/scsi.h> #include <scsi/scsi_transport_iscsi.h> +#include <scsi/scsi_device.h> #include "iscsi_tcp.h"
@@ -1740,15 +1741,19 @@ iscsi_tcp_ep_connect(struct sockaddr *ds {struct socket *sock;int rc, size; +int swapper =sk_vmio_socks (); +unsigned long pflags = current->flags; + +if (swapper) + pflags |=PF_MEMALLOC ; rc =sock_create_kern (dst_addr->sa_family,SOCK_STREAM ,IPPROTO_TCP , &sock);if (rc <0 ) {printk (KERN_ERR "Could not create socket %d.\n" , rc); -return rc; + goto out; } -/* TODO: test this with GFP_NOIO */ - sock->sk->sk_allocation =GFP_ATOMIC ; + sock->sk->sk_allocation =GFP_NOIO ;if (dst_addr->sa_family ==PF_INET ) size =sizeof (struct sockaddr_in );
@@ -1765,6 +1770,8 @@ iscsi_tcp_ep_connect(struct sockaddr *ds* we don't want it used by user-space at all. */ sock_set_flag (sock->sk,SOCK_KERNEL ); +if (swapper) +sk_set_vmio (sock->sk); rc = sock->ops->connect (sock, (struct sockaddr *)dst_addr, size,O_NONBLOCK );
@@ -1779,11 +1786,14 @@ iscsi_tcp_ep_connect(struct sockaddr *dsif (rc <0 ) goto release_sock; *ep_handle = (uint64_t )rc; -return 0 ; + rc =0 ; +out: + current->flags = pflags; +return rc; release_sock:sock_release (sock); -return rc; + goto out; } static struct iscsi_cls_conn *
@@ -1908,8 +1918,13 @@ iscsi_tcp_conn_bind(struct iscsi_cls_ses sk->sk_reuse =1 ; sk->sk_sndtimeo =15 *HZ ;/* FIXME: make it configurable */ +if (!cls_session->swapper &&sk_has_vmio (sk)) +sk_clear_vmio (sk); +/* FIXME: disable Nagle's algorithm */ +BUG_ON (!sk_has_vmio (sk) && cls_session->swapper); +/* * Intercept TCP callbacks for sendfile like receive * processing.
@@ -2167,6 +2182,50 @@ static void iscsi_tcp_session_destroy(stiscsi_session_teardown (cls_session); } +#ifdef CONFIG_ISCSI_TCP_SWAP + +#define ISCSI_TCP_RESERVE_PAGES (TX_RESERVE_PAGES) + +static int iscsi_tcp_swapdev (void *objp,int enable) +{ +int error =0 ; +struct scsi_device *sdev = objp; +struct Scsi_Host *shost = sdev->host; +struct iscsi_session *session =iscsi_hostdata (shost->hostdata); + +if (enable) { +iscsi_swapdev (session->tt,session_to_cls (session),1 ); +sk_adjust_memalloc (1 ,ISCSI_TCP_RESERVE_PAGES ); + } + +spin_lock (&session->lock); +if (session->leadconn) { +struct iscsi_tcp_conn *tcp_conn = session->leadconn->dd_data; +if (enable) +sk_set_vmio (tcp_conn->sock->sk); +else +sk_clear_vmio (tcp_conn->sock->sk); + } +spin_unlock (&session->lock); + +if (!enable) { +sk_adjust_memalloc (-1 , -ISCSI_TCP_RESERVE_PAGES ); +iscsi_swapdev (session->tt,session_to_cls (session),0 ); + } + +return error; +} +#endif + +static int iscsi_tcp_slave_configure (struct scsi_device *sdev) +{ +#ifdef CONFIG_ISCSI_TCP_SWAP +if (sdev->type ==TYPE_DISK ) +blk_queue_swapdev (sdev->request_queue, iscsi_tcp_swapdev, sdev); +#endif +return 0 ; +} +static struct scsi_host_template iscsi_sht = { .name ="iSCSI Initiator over TCP/IP" , .queuecommand = iscsi_queuecommand,
@@ -2174,6 +2233,7 @@ static struct scsi_host_template iscsi_s .can_queue = ISCSI_XMIT_CMDS_MAX - 1, .sg_tablesize = ISCSI_SG_TABLESIZE, .cmd_per_lun = ISCSI_DEF_CMD_PER_LUN, + .slave_configure = iscsi_tcp_slave_configure, .eh_abort_handler = iscsi_eh_abort, .eh_host_reset_handler = iscsi_eh_host_reset, .use_clustering = DISABLE_CLUSTERING,
drivers/scsi/scsi_transport_iscsi.c +64 -8
146 lines not on this page · raw message
include/scsi/scsi_transport_iscsi.h +10 -2
@@ -137,9 +137,10 @@ extern int iscsi_unregister_transport(st/* * control plane upcalls */ -extern void iscsi_conn_error (struct iscsi_cls_conn *conn,enum iscsi_err error); +extern void iscsi_conn_error (struct iscsi_cls_conn *conn,enum iscsi_err error, +gfp_t gfp_mask);extern int iscsi_recv_pdu (struct iscsi_cls_conn *conn,struct iscsi_hdr *hdr, -char *data,uint32_t data_size); +char *data,uint32_t data_size,gfp_t gfp_mask);/* Connection's states */
@@ -183,6 +184,7 @@ struct iscsi_cls_session {int sid;/* session id */ void *dd_data;/* LLD private data */ struct device dev;/* sysfs transport/container device */ +int swapper;/* we are used to swap on */ };#define iscsi_dev_to_session (_dev) \
@@ -194,6 +196,10 @@ struct iscsi_cls_session {#define starget_to_session (_stgt) \ iscsi_dev_to_session(_stgt->dev.parent) +#define iscsi_session_gfp (_session) \ + ((in_interrupt() ? GFP_ATOMIC : GFP_NOIO) | \ + ((_session)->swapper ? __GFP_EMERGENCY : 0)) +struct iscsi_host {struct list_head sessions;struct mutex mutex;
@@ -217,6 +223,8 @@ extern int iscsi_destroy_session(structextern struct iscsi_cls_conn *iscsi_create_conn (struct iscsi_cls_session *sess,uint32_t cid);extern int iscsi_destroy_conn (struct iscsi_cls_conn *conn); +extern int iscsi_swapdev (struct iscsi_transport *tt,struct iscsi_cls_session *, +int enable);extern void iscsi_unblock_session (struct iscsi_cls_session *session);extern void iscsi_block_session (struct iscsi_cls_session *session);
drivers/scsi/libiscsi.c +13 -5
@@ -361,10 +361,12 @@ int __iscsi_complete_pdu(struct iscsi_co char *data, int datalen) {struct iscsi_session *session = conn->session; +struct iscsi_cls_session *cls_session =session_to_cls (session);int opcode = hdr->opcode &ISCSI_OPCODE_MASK , rc =0 ;struct iscsi_cmd_task *ctask;struct iscsi_mgmt_task *mtask;uint32_t itt; +gfp_t gfp_mask =iscsi_session_gfp (cls_session);if (hdr->itt !=RESERVED_ITT ) itt =get_itt (hdr->itt);
@@ -423,7 +425,8 @@ int __iscsi_complete_pdu(struct iscsi_co* login related PDU's exp_statsn is handled in * userspace */ -if (iscsi_recv_pdu (conn->cls_conn, hdr, data, datalen)) +if (iscsi_recv_pdu (conn->cls_conn, hdr, data, datalen, + gfp_mask)) rc =ISCSI_ERR_CONN_FAILED ;list_del (&mtask->running);if (conn->login_mtask != mtask)
@@ -445,7 +448,8 @@ int __iscsi_complete_pdu(struct iscsi_co } conn->exp_statsn =be32_to_cpu (hdr->statsn) +1 ; -if (iscsi_recv_pdu (conn->cls_conn, hdr, data, datalen)) +if (iscsi_recv_pdu (conn->cls_conn, hdr, data, datalen, + gfp_mask)) rc =ISCSI_ERR_CONN_FAILED ;list_del (&mtask->running);if (conn->login_mtask != mtask)
@@ -472,7 +476,8 @@ int __iscsi_complete_pdu(struct iscsi_coif (hdr->ttt ==cpu_to_be32 (ISCSI_RESERVED_TAG ))break ; -if (iscsi_recv_pdu (conn->cls_conn, hdr,NULL ,0 )) +if (iscsi_recv_pdu (conn->cls_conn, hdr,NULL ,0 , + gfp_mask)) rc =ISCSI_ERR_CONN_FAILED ;break ;case ISCSI_OP_REJECT :
@@ -480,7 +485,8 @@ int __iscsi_complete_pdu(struct iscsi_cobreak ;case ISCSI_OP_ASYNC_EVENT : conn->exp_statsn =be32_to_cpu (hdr->statsn) +1 ; -if (iscsi_recv_pdu (conn->cls_conn, hdr, data, datalen)) +if (iscsi_recv_pdu (conn->cls_conn, hdr, data, datalen, + gfp_mask)) rc =ISCSI_ERR_CONN_FAILED ;break ;default :
@@ -560,7 +566,9 @@ EXPORT_SYMBOL_GPL(iscsi_verify_itt);void iscsi_conn_failure (struct iscsi_conn *conn,enum iscsi_err err) {struct iscsi_session *session = conn->session; +struct iscsi_cls_session *cls_session =session_to_cls (session);unsigned long flags; +gfp_t gfp_mask =iscsi_session_gfp (cls_session);spin_lock_irqsave (&session->lock, flags);if (session->state ==ISCSI_STATE_FAILED ) {
@@ -573,7 +581,7 @@ void iscsi_conn_failure(struct iscsi_conspin_unlock_irqrestore (&session->lock, flags);set_bit (ISCSI_SUSPEND_BIT , &conn->suspend_tx);set_bit (ISCSI_SUSPEND_BIT , &conn->suspend_rx); -iscsi_conn_error (conn->cls_conn, err); +iscsi_conn_error (conn->cls_conn, err, gfp_mask); }EXPORT_SYMBOL_GPL (iscsi_conn_failure);
drivers/scsi/qla4xxx/ql4_os.c +1 -1
@@ -340,7 +340,7 @@ void qla4xxx_mark_device_missing(structDEBUG3 (printk ("scsi%d:%d:%d: index [%d] marked MISSING\n" , ha->host_no, ddb_entry->bus, ddb_entry->target, ddb_entry->fw_ddb_index)); -iscsi_conn_error (ddb_entry->conn,ISCSI_ERR_CONN_FAILED ); +iscsi_conn_error (ddb_entry->conn,ISCSI_ERR_CONN_FAILED ,GFP_ATOMIC ); } static struct srb* qla4xxx_get_new_srb(struct scsi_qla_host *ha,
drivers/scsi/Kconfig +17 -0
@@ -268,6 +268,12 @@ config SCSI_ISCSI_ATTRS each attached iSCSI device to sysfs, say Y. Otherwise, say N. +config ISCSI_SWAP + def_bool n + depends on SCSI_ISCSI_ATTRS + select SLAB_FAIR + select NETVM + config SCSI_SAS_ATTRS tristate "SAS Transport Attributes" depends on SCSI
@@ -306,6 +312,17 @@ config ISCSI_TCP http://linux-iscsi.sf.net +config ISCSI_TCP_SWAP + bool "Provide swap over iSCSI over TCP/IP" + default n + depends on ISCSI_TCP + select ISCSI_SWAP + help + This option enables swapon to savely work with iSCSI over TCP/IP + devices. + + If unsure, say N. + config SGIWD93_SCSI tristate "SGI WD93C93 SCSI Driver" depends on SGI_IP22 && SCSI
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Peter Zijlstra
1) introduce the memory reserve and make the SLAB allocator play nice with it. patches 01-10
2) add some needed infrastructure to the network code patches 11-13
3) implement the idea outlined above
patches 14-20
Quoted 8 lines
4) teach the swap machinery to use generic
address_spacespatches 21-245) implement swap over NFS using all the new stuff patches 25-31
6) implement swap over iSCSI
patches 32-40
This is kind of a lot of patches all at once .. Have you release any of these patch sets prior to this release ?
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · Daniel Walker
Quoted 12 lines · Peter Zijlstra
1) introduce the memory reserve and make the SLAB allocator play nice with it. patches 01-10
2) add some needed infrastructure to the network code patches 11-13
3) implement the idea outlined above
patches 14-204) teach the swap machinery to use generic
address_spacespatches 21-24
Quoted 8 lines
Quoted 5 lines
5) implement swap over NFS using all the new stuff patches 25-31
6) implement swap over iSCSI
patches 32-40This is kind of a lot of patches all at once .. Have you release any of these patch sets prior to this release ?
Like the -v12 suggests, this is the 12th posting of this patch set. Some is the same, some has changed.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines
This is kind of a lot of patches all at once .. Have you release any of these patch sets prior to this release ?
Like the -v12 suggests, this is the 12th posting of this patch set. Some is the same, some has changed.
I can find one prior release with this subject (-v11) , what was the subject prior to that release? It's not a hard rule, but usually >15 patches is too many (check Documentation/SubmittingPatches under references).. You might want to consider submitting a URL instead.
I think it's a benefit to release less since a developer (like myself) might know very little about "Swap over Networked storage", but if you submit 10 patches that developer might still review it, 40 patches they likely wouldn't review it.
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Daniel Walker
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines
This is kind of a lot of patches all at once .. Have you release any of these patch sets prior to this release ?
Quoted 7 lines
Quoted 2 lines
Like the -v12 suggests, this is the 12th posting of this patch set. Some is the same, some has changed.
I can find one prior release with this subject (-v11) , what was the subject prior to that release? It's not a hard rule, but usually >15 patches is too many (check
Documentation/SubmittingPatchesunder references).. You might want to consider submitting a URL instead.
Previous subjects were like:
[PATCH 00/20] vm deadlock avoidance for NFS, NBD and iSCSI (take 7)
A URL doesn't allow for true discussion about a particular patch unless the reviewer takes the initiative to create a new thread to discuss the Nth patch it a patchset; whereby taking on the burden of a structured subject and so on. It would get out of control on a large patchset that actually got a lot of simultaneous feedback... reviewers don't have a forum to talk about each individual change without stepping on each others' toes.
Quoted 4 lines
I think it's a benefit to release less since a developer (like myself) might know very little about "Swap over Networked storage", but if you submit 10 patches that developer might still review it, 40 patches they likely wouldn't review it.
The _suggestions_ in Documentation/SubmittingPatches are nice and all but the quantity of patches shouldn't _really_ matter.
Documentation/SubmittingPatches actually doesn't cover how to post a large change because it first states:
"Separate _logical changes_ into a single patch file."
then:
"If you cannot condense your patch set into a smaller set of patches, then only post say 15 or so at a time and wait for review and integration."
These suggestions conflict in the case of a large patchset: the second can't be met if you honor the first (more important suggestion IMHO). Unless you leave something out... and I can't see the value in leaving out the auxiliary consumers of the core changes.
Reviewing 10 patches that are quite large/overloaded is actually harder than 40 broken-out/well-documented patches. But maybe others disagree.
*shrug*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Mike Snitzer
Quoted 5 lines · Daniel Walker
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines
This is kind of a lot of patches all at once .. Have you release any of these patch sets prior to this release ?
Quoted 7 lines
Quoted 2 lines
Like the -v12 suggests, this is the 12th posting of this patch set. Some is the same, some has changed.
I can find one prior release with this subject (-v11) , what was the subject prior to that release? It's not a hard rule, but usually >15 patches is too many (check
Documentation/SubmittingPatchesunder references).. You might want to consider submitting a URL instead.Previous subjects were like:
[PATCH 00/20] vm deadlock avoidance for NFS, NBD and iSCSI (take 7)
Quoted 7 lines
A URL doesn't allow for true discussion about a particular patch unless the reviewer takes the initiative to create a new thread to discuss the Nth patch it a patchset; whereby taking on the burden of a structured subject and so on. It would get out of control on a large patchset that actually got a lot of simultaneous feedback... reviewers don't have a forum to talk about each individual change without stepping on each others' toes.
True ..
Quoted 7 lines
Quoted 4 lines
I think it's a benefit to release less since a developer (like myself) might know very little about "Swap over Networked storage", but if you submit 10 patches that developer might still review it, 40 patches they likely wouldn't review it.
The _suggestions_ in
Documentation/SubmittingPatchesare nice and all but the quantity of patches shouldn't _really_ matter.
I guess I take the documentation more seriously than your do. It's clearly not mandatory, but for my reviewing I appreciate less then 15 sets of "logical changes".
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Miller
maintainerQuoted 4 lines
These suggestions conflict in the case of a large patchset: the second can't be met if you honor the first (more important suggestion IMHO). Unless you leave something out... and I can't see the value in leaving out the auxiliary consumers of the core changes.
They do not conflict.
If you say you're setting up infrastructure for a well defined purpose, then each and every one of the patches can all stand on their own just fine. You can even post them one at a time and the review process would work just fine.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 18 lines · Daniel Walker
Quoted 15 lines · Peter Zijlstra
1) introduce the memory reserve and make the SLAB allocator play nice with it. patches 01-10
2) add some needed infrastructure to the network code patches 11-13
3) implement the idea outlined above
patches 14-204) teach the swap machinery to use generic
address_spacespatches 21-245) implement swap over NFS using all the new stuff patches 25-31
Quoted 7 lines
Quoted 3 lines
6) implement swap over iSCSI
patches 32-40This is kind of a lot of patches all at once .. Have you release any of these patch sets prior to this release ?
Yes, several times AFAIK.
- Arnaldo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Miller
maintainerQuoted 1 line
There is a fundamental deadlock associated with paging;
I know you'd really like people like myself to review this work, but a set of 40 patches is just too much to try and digest at once especially when I have other things going on. When I have lots of other things already on my plate, when I see a huge patch set like this I have to just say "delete" because I don't kid myself since I know I'll never get to it.
Sorry there's now way I can review this with my current workload.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · David Miller
From: Peter Zijlstra <[email protected]>
Date: Fri, 04 May 2007 12:26:51 +0200Quoted 1 line
There is a fundamental deadlock associated with paging;
I know you'd really like people like myself to review this work, but a set of 40 patches is just too much to try and digest at once especially when I have other things going on.
I realize this, however I expected you to mainly look at the the 10 network related patches, namely: 11/40 - 20/40.
I know they build upon the previous 10 patches, which are mostly VM, and you seem to have an interest in that as well, so that would be 20 patches to look at. Still a sizable set.
How would you prefer I present these?
The other patches are NFS and iSCSI, I'd not expect you to review those in depth.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Miller
maintainerQuoted 1 line
How would you prefer I present these?
How about 8 or 9 at a time? You are building infrastructure and therefore you could post them 1 at a time for review since each patch should be able to stand on it's own.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · David Miller
From: Peter Zijlstra <[email protected]>
Date: Fri, 04 May 2007 21:41:49 +0200Quoted 1 line
How would you prefer I present these?
How about 8 or 9 at a time? You are building infrastructure and therefore you could post them 1 at a time for review since each patch should be able to stand on it's own.
Indeed. Just glancing over the patchset, there are quite a few "easy to apply" cleanup patches that could be fast-forwarded to upstream, without requiring deep thought on the swap-over-storage MM changes or net allocator changes.
Jeff
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · David Miller
From: Peter Zijlstra <[email protected]>
Date: Fri, 04 May 2007 12:26:51 +0200Quoted 1 line
There is a fundamental deadlock associated with paging;
Quoted 8 lines
I know you'd really like people like myself to review this work, but a set of 40 patches is just too much to try and digest at once especially when I have other things going on. When I have lots of other things already on my plate, when I see a huge patch set like this I have to just say "delete" because I don't kid myself since I know I'll never get to it.
Sorry there's now way I can review this with my current workload.
There also quite alot of only semi-related thing in there. It would be much better to only do the network stack and iscsi parts first and leave nfs out for a while. Especially as the former are definitively useful while I strongly doubt that for swap over nfs.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Peter Zijlstra
Quoted 1 line
Quoted 1 line
There is a fundamental deadlock associated with paging;
Quoted 7 lines · David Miller
Quoted 7 lines
I know you'd really like people like myself to review this work, but a set of 40 patches is just too much to try and digest at once especially when I have other things going on. When I have lots of other things already on my plate, when I see a huge patch set like this I have to just say "delete" because I don't kid myself since I know I'll never get to it.
Sorry there's now way I can review this with my current workload.
Quoted 4 lines · Christoph Hellwig
There also quite alot of only semi-related thing in there. It would be much better to only do the network stack and iscsi parts first and leave nfs out for a while. Especially as the former are definitively useful while I strongly doubt that for swap over nfs.
This is backward. As much as we hate it, the common case is swap over nfs, essentially because that is/was how things were commonly set up for other operating systems. I'm not a Solaris administrator, though, so various disclaimers apply.
-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>