Skip to content

Swap over Networked storage -v12

Peter Zijlstra linux-mm lkml netdev 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest40 patches 37 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/40mm: page allocation rankmm/internal.h mm/page_alloc.c+87 -410 replies
2/40mm: slab allocation fairnessmm/Kconfig mm/slab.c+57 -271 reply
3/40mm: allow PF_MEMALLOC from softirq contextinclude/linux/sched.h kernel/softirq.c mm/internal.h+11 -30 replies
4/40mm: serialize access to min_free_kbytesmm/page_alloc.c+13 -30 replies
5/40mm: emergency poolinclude/linux/mmzone.h mm/page_alloc.c mm/vmstat.c+48 -130 replies
6/40mm: __GFP_EMERGENCYinclude/linux/gfp.h mm/internal.h+9 -20 replies
7/40mm: allow mempool to fall back to memalloc reservesmm/mempool.c+10 -00 replies
8/40mm: kmem_cache_objsizeinclude/linux/slab.h mm/slab.c mm/slob.c+34 -220 replies
9/40mm: optimize gfp_to_rank()mm/internal.h+31 -20 replies
10/40selinux: tag avc cache alloc as non-criticalsecurity/selinux/avc.c+1 -10 replies
11/40net: wrap sk->sk_backlog_rcv()include/net/sock.h net/core/sock.c net/ipv4/tcp.c +1 more+9 -40 replies
12/40net: packet split receive apidrivers/net/e1000/e1000_main.c drivers/net/sky2.c include/linux/skbuff.h +1 more+51 -160 replies
13/40net: sk_allocation() - concentrate socket related allocationsinclude/net/sock.h net/ipv4/tcp_output.c net/ipv6/tcp_ipv6.c+21 -110 replies
14/40netvm: link network to vm layerinclude/net/sock.h net/Kconfig net/core/sock.c+180 -10 replies
15/40netvm: INET reserves.net/ipv4/ip_fragment.c net/ipv4/route.c net/ipv4/sysctl_net_ipv4.c +3 more+63 -40 replies
16/40netvm: hook skb allocation to reservesinclude/linux/skbuff.h net/core/skbuff.c+157 -261 reply
17/40netvm: filter emergency skbs.include/net/sock.h+3 -00 replies
18/40netvm: prevent a TCP specific deadlockinclude/net/sock.h net/core/stream.c+7 -50 replies
19/40netfilter: notify about NF_QUEUE vs emergency skbsnet/netfilter/core.c+3 -00 replies
20/40netvm: skb processinginclude/net/sock.h net/core/dev.c net/core/sock.c+60 -50 replies
21/40uml: rename arch/um remove_mapping()arch/um/kernel/physmem.c+3 -30 replies
22/40mm: prepare swap entry methods for use in page methodsinclude/linux/mm.h include/linux/swap.h include/linux/swapops.h +1 more+57 -440 replies
23/40mm: add support for non block device backed swap filesDocumentation/filesystems/Locking include/linux/fs.h include/linux/swap.h +4 more+97 -10 replies
24/40mm: methods for teaching filesystems about PG_swapcache pagesinclude/linux/mm.h include/linux/pagemap.h+26 -10 replies
25/40nfs: remove mempoolsfs/nfs/read.c fs/nfs/write.c+8 -340 replies
26/40nfs: teach the NFS client how to treat PG_swapcache pagesfs/nfs/file.c fs/nfs/internal.h fs/nfs/pagelist.c +2 more+30 -290 replies
27/40nfs: disable data cache revalidation for swapfilesfs/nfs/inode.c fs/nfs/write.c+32 -160 replies
28/40nfs: enable swap on NFSfs/Kconfig fs/nfs/file.c include/linux/sunrpc/xprt.h +2 more+92 -30 replies
29/40nfs: fix various memory recursions possible with swap over NFS.fs/nfs/pagelist.c fs/nfs/write.c+4 -40 replies
30/40nfs: fixup missing error codefs/nfs/read.c+3 -12 replies
31/40mm: balance_dirty_pages() vs throttle_vm_writeout() deadlockmm/page-writeback.c+1 -20 replies
32/40block: add a swapdev callback to the request_queueinclude/linux/blkdev.h mm/swapfile.c+23 -00 replies
33/40uml: enable scsi and add iscsi configarch/um/Kconfig arch/um/Kconfig.scsi+1 -730 replies
34/40sock: safely expose kernel sockets to userspaceinclude/net/sock.h net/socket.c+10 -10 replies
35/40From: Mike Christie <[email protected]>drivers/infiniband/ulp/iser/iscsi_iser.c drivers/scsi/iscsi_tcp.c drivers/scsi/libiscsi.c +4 more+73 -450 replies
36/40iscsi: fixup of the ep_connect patchdrivers/scsi/iscsi_tcp.c+19 -40 replies
37/40iscsi: ensure the iscsi kernel fd is not usable in userspacedrivers/scsi/iscsi_tcp.c+7 -00 replies
38/40netlink: add SOCK_VMIO support to AF_NETLINKinclude/linux/netlink.h net/compat.c net/netlink/af_netlink.c +1 more+14 -70 replies
39/40mm: a process flags to avoid blocking allocationsinclude/linux/sched.h kernel/softirq.c mm/internal.h +1 more+15 -50 replies
40/40iscsi: support for swapping over iSCSI.drivers/scsi/Kconfig drivers/scsi/iscsi_tcp.c drivers/scsi/libiscsi.c +3 more+170 -210 replies

Reviews trailers per patch

4 of 40 patches have a Reviewed-by or Acked-by.

PatchJames MorrisJeff DikeMike Christie
1/40
2/40
3/40
4/40
5/40
6/40
7/40
8/40
9/40
10/40A-b
11/40
12/40
13/40
14/40
15/40
16/40
17/40
18/40
19/40
20/40
21/40A-b
22/40
23/40
24/40
25/40
26/40
27/40
28/40
29/40
30/40
31/40
32/40
33/40A-b
34/40
35/40
36/40A-b
37/40
38/40
39/40
40/40

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v1 thread

Cover letter [PATCH 00/40] Swap over Networked storage -v12

There is a fundamental deadlock associated with paging; when writing out a page to free memory requires free memory to complete. The usually solution is to keep a small amount of memory available at all times so we can overcome this problem. This however assumes the amount of memory needed for writeout is (constant and) smaller than the provided reserve.

It is this latter assumption that breaks when doing writeout over network. Network can take up an unspecified amount of memory while waiting for a reply to our write request. This re-introduces the deadlock; we might never complete the writeout, for we might not have enough memory to receive the completion message.

The proposed solution is simple, only allow traffic servicing the VM to make use of the reserves.

This however implies you know what packets are for whom, which generally speaking you don't. Hence we need to receive all packets but discard them as soon as we encounter a non VM bound packet allocated from the reserves.

Also knowing it is headed towards the VM needs a little help, hence we introduce the socket flag SOCK_VMIO to mark sockets with.

Of course, since we are paging all this has to happen in kernel-space, since user-space might just not be there.

Since packet processing might also require memory, this all also implies that those auxiliary allocations may use the reserves when an emergency packet is processed. This is accomplished by using PF_MEMALLOC.

How much memory is to be reserved is also an issue, enough memory to saturate both the route cache and IP fragment reassembly, along with various constants.

This patch-set comes in 6 parts:

1) introduce the memory reserve and make the SLAB allocator play nice with it. patches 01-10

2) add some needed infrastructure to the network code patches 11-13

3) implement the idea outlined above
patches 14-20

4) teach the swap machinery to use generic address_spaces patches 21-24

5) implement swap over NFS using all the new stuff patches 25-31

6) implement swap over iSCSI
patches 32-40

Patches can also be found here:
http://programming.kicks-ass.net/kernel-patches/vm_deadlock/v12/

If I receive no feedback, I will assume the various maintainers do not object and I will respin the series against -mm and submit for inclusion.

There is interest in this feature from the stateless linux world; that is both the virtualization world, and the cluster world.

I have been contacted by various groups, some have just expressed their interest, others have been testing this work in their environments.

Various hardware vendors have also expressed interest, and, of course, my employer finds it important enough to have me work on it.

Also, while it doesn't present a full-fledged reserve-based allocator API yet, it does lay most of the groundwork for it. There is a GFP_NOFAIL elimination project wanting to use this as a foundation. Elimination of GFP_NOFAIL will greatly improve the basic soundness and stability of the code that currently uses that construct - most disk based filesystems.

--

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>