From: Jerome Glisse Date: Tue, 09 Feb 2021 21:17:38 +0000 Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau Message-Id: <20210209211738.GA834106@redhat.com> List-Id: References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> <20210209133520.GB4718@ziepe.ca> In-Reply-To: <20210209133520.GB4718@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: Jason Gunthorpe Cc: Alistair Popple , Daniel Vetter , Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , John Hubbard , Ralph Campbell On Tue, Feb 09, 2021 at 09:35:20AM -0400, Jason Gunthorpe wrote: > On Tue, Feb 09, 2021 at 11:57:28PM +1100, Alistair Popple wrote: > > On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: > > > > > > > > Recent changes to pin_user_pages() prevent the creation of pinned p= ages in > > > > ZONE_MOVABLE. This series allows pinned pages to be created in=20 > > ZONE_MOVABLE > > > > as attempts to migrate may fail which would be fatal to userspace. > > > > > > > > In this case migration of the pinned page is unnecessary as the pag= e can=20 > > be > > > > unpinned at anytime by having the driver revoke atomic permission a= s it > > > > does for the migrate_to_ram() callback. However a method of calling= this > > > > when memory needs to be moved has yet to be resolved so any discuss= ion is > > > > welcome. > > >=20 > > > Why do we need to pin for gpu atomics? You still have the callback for > > > cpu faults, so you > > > can move the page as needed, and hence a long-term pin sounds like the > > > wrong approach. > >=20 > > Technically a real long term unmoveable pin isn't required, because as = you say=20 > > the page can be moved as needed at any time. However I needed some way = of=20 > > stopping the CPU page from being freed once the userspace mappings for = it had=20 > > been removed.=20 >=20 > The issue is you took the page out of the PTE it belongs to, which > makes it orphaned and unlocatable by the rest of the mm? >=20 > Ideally this would leave the PTE in place so everything continues to > work, just disable CPU access to it. >=20 > Maybe some kind of special swap entry? >=20 > I also don't much like the use of ZONE_DEVICE here, that should only > be used for actual device memory, not as a temporary proxy for CPU > pages.. Having two struct pages refer to the same physical memory is > pretty ugly. >=20 > > The normal solution of registering an MMU notifier to unpin the page wh= en it=20 > > needs to be moved also doesn't work as the CPU page tables now point to= the > > device-private page and hence the migration code won't call any invalid= ate=20 > > notifiers for the CPU page. >=20 > The fact the page is lost from the MM seems to be the main issue here. >=20 > > Yes, I would like to avoid the long term pin constraints as well if pos= sible I=20 > > just haven't found a solution yet. Are you suggesting it might be possi= ble to=20 > > add a callback in the page migration logic to specially deal with movin= g these=20 > > pages? >=20 > How would migration even find the page? Migration can scan memory from physical address (isolate_migratepages_range= ()) So the CPU mapping is not the only path to get to a page. Cheers, J=E9r=F4me