From: Jason Gunthorpe Date: Mon, 16 Mar 2020 20:09:29 +0000 Subject: Re: [PATCH 2/2] mm: remove device private page support from hmm_range_fault Message-Id: <20200316200929.GB20010@ziepe.ca> List-Id: References: <20200316175259.908713-1-hch@lst.de> <20200316175259.908713-3-hch@lst.de> <20200316184935.GA25322@lst.de> In-Reply-To: <20200316184935.GA25322@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Christoph Hellwig Cc: Ralph Campbell , Dan Williams , Bharata B Rao , Christian =?utf-8?B?S8O2bmln?= , Ben Skeggs , Jerome Glisse , kvm-ppc@vger.kernel.org, amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-mm@kvack.org On Mon, Mar 16, 2020 at 07:49:35PM +0100, Christoph Hellwig wrote: > On Mon, Mar 16, 2020 at 11:42:19AM -0700, Ralph Campbell wrote: > > > > On 3/16/20 10:52 AM, Christoph Hellwig wrote: > >> No driver has actually used properly wire up and support this feature. > >> There is various code related to it in nouveau, but as far as I can tell > >> it never actually got turned on, and the only changes since the initial > >> commit are global cleanups. > > > > This is not actually true. OpenCL 2.x does support SVM with nouveau and > > device private memory via clEnqueueSVMMigrateMem(). > > Also, Ben Skeggs has accepted a set of patches to map GPU memory after being > > migrated and this change would conflict with that. > > Can you explain me how we actually invoke this code? > > For that we'd need HMM_PFN_DEVICE_PRIVATE NVIF_VMM_PFNMAP_V0_VRAM > set in ->pfns before calling hmm_range_fault, which isn't happening. Oh, I got tripped on this too The logic is backwards from what you'd think.. If you *set* HMM_PFN_DEVICE_PRIVATE then this triggers: hmm_pte_need_fault(): if ((cpu_flags & range->flags[HMM_PFN_DEVICE_PRIVATE])) { /* Do we fault on device memory ? */ if (pfns & range->flags[HMM_PFN_DEVICE_PRIVATE]) { *write_fault = pfns & range->flags[HMM_PFN_WRITE]; *fault = true; } return; } Ie if the cpu page is a DEVICE_PRIVATE and the caller sets HMM_PFN_DEVICE_PRIVATE in the input flags (pfns) then it always faults it and never sets HMM_PFN_DEVICE_PRIVATE in the output flags. So setting 0 enabled device_private support, and nouveau is Ok. AMDGPU is broken because it can't handle device private and can't set the flag to supress it. I was going to try and sort this out as part of getting rid of range->flags Jason