Return-Path: Received: from mx141.netapp.com ([216.240.21.12]:8396 "EHLO mx141.netapp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753455AbeDYMWH (ORCPT ); Wed, 25 Apr 2018 08:22:07 -0400 Subject: Re: [RFC 1/7] mm: Add new vma flag VM_LOCAL_CPU To: Miklos Szeredi CC: Matthew Wilcox , linux-fsdevel , Ric Wheeler , Steve French , Steven Whitehouse , Jefff moyer , Sage Weil , Jan Kara , Amir Goldstein , Andy Rudof , Anna Schumaker , Amit Golander , Sagi Manole , Shachar Sharon References: <443fea57-f165-6bed-8c8a-0a32f72b9cd2@netapp.com> <20180313185658.GB21538@bombadil.infradead.org> <07cda3e5-c911-a49b-fceb-052f8ca57e66@netapp.com> <9bfa8d53-5693-7953-9dcf-79a8cff0b97f@netapp.com> From: Boaz Harrosh Message-ID: Date: Wed, 25 Apr 2018 06:21:55 -0600 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On 03/15/2018 02:42 PM, Miklos Szeredi wrote: > Ideally most of the complexity would be in the page cache. Not sure > how ready it is to handle pmem pages? > > The general case (non-pmem) will always have to be handled > differently; you've just stated that it's much less latency sensitive > and needs async handling. Basing the design on just trying to make > it use the same mechanism (userspace copy) is flawed in my opinion, > since it's suboptimal for either case. > > Thanks, > Miklos OK So I was thinking hard on all this and am changing my mind and agreeing with all that was said. I want that the usFS plugin will have all the different options and have an easy way to tell Kernel which mode to use. Let me summarize all the options: 1. Sync, userspace copy directly to app-buffers (current implementation) 2. Async block device operation (none pmem)     zuf owns all devices pmem and none pmem at mount time and provides     a very efficient access to both. In the harddisk / ssd case as part of an IO call     the server returns -EWOULD_BLOCK and in the background will issue a     scatter_gather call through zuf.     The memory target for the IO can be pmem, directly to user-buffers (DIO), transient      server buffers.      On completion an up call is made to ZUF to complete the IO operation and      release the waiting application. 3. Splice and R-spilce     In the case that the IO target is not a block-device but an external path like     network / rdma / some none block device.     Zuf already holds an internal object describing the IO context including the     GUP app buffers. This internal object can be made the memory target of a splice     operation. 4. Get-io_map type operation (currently implemented for mmap)     The zus-FS returns a set of dpp_t(s) to kernel and the Kernel does the memcopy     to app buffers. The Server also specifies if those buffers should be cached     on a per inode radix-tree (xarray) and if so at the next access to the same     range Kernel does the copy and never dispatches to user-space     In this mode the Server can also revoke a cached mapping when needed 5. Use of VFS page-cache     For a very slow backing device the FS request the regular VFS page-cache.     On read/write_pages() vector zuf uses option 1. above to read into page-cache     instead of app-buffers directly. Only cache misses dispatch back to user-space Have I forgotten anything? This way the zus-FS is in control and can do the "right thing" depending on target device and FS characteristics. The interface lets us have a rich set of tools to be used. Hope that answers your concerns Boaz