XeKmd basic SVM support
Subsystems (4)
RE: Making drm_gpuvm work across gpu devices
In reply to Zeng, OakHello all,
Since I didn't get a reply for this one, I assume below are agreed. But feel free to let us know if you don't agree.
Thanks,
Oak
-----Original Message-----
From: dri-devel <[email protected]> On Behalf Of Zeng, Oak
Sent: Thursday, February 29, 2024 1:23 PM
To: Christian König <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: Making drm_gpuvm work across gpu devices
Hi Christian/Daniel/Dave/Felix/Thomas, and all,
We have been refining our design internally in the past month. Below is our plan. Please let us know if you have any concern.
1) Remove pseudo /dev/xe-svm device. All system allocator interfaces will be through /dev/dri/render devices. Not global interface.
2) Unify userptr and system allocator codes. We will treat userptr as a speciality of system allocator without migration capability. We will introduce the hmmptr concept for system allocator. We will extend vm_bind API to map a range A..B of process address space to a range C..D of GPU address space for hmmptr. For hmmptr, if gpu program accesses an address which is not backed by core mm vma, it is a fatal error.
3) Multiple device support. We have identified p2p use-cases where we might want to leave memory on a foreign device or direct migrations to a foreign device and therefore might need a global structure that tracks or caches the migration state per process address space. We didn't completely settle down this design. We will come back when we have more details.
4)We will first work on this code on xekmd then look to move some common codes to drm layer so it can also be used by other vendors.
Thomas and me still have open questions to Christian. We will follow up.
Thanks all for this discussion.
Regards,
Oak
Quoted 189 lines
-----Original Message-----
From: Christian König <[email protected]>
Sent: Thursday, February 1, 2024 3:52 AM
To: Zeng, Oak <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Makingdrm_gpuvmwork across gpu devicesHi Oak,
Am 31.01.24 um 21:17 schrieb Zeng, Oak:
Quoted 3 lines
Hi Sima, Dave,
I am well aware nouveau driver is not what Nvidia do with their customer. The
key argument is, can we move forward with the concept shared virtual address space b/t CPU and GPU? This is the foundation of HMM. We already have split address space support with other driver API. SVM, from its name, it means shared address space. Are we allowed to implement another driver model to allow SVM work, along with other APIs supporting split address space? Those two scheme can co-exist in harmony. We actually have real use cases to use both models in one application.
Quoted 4 lines
Hi Christian, Thomas,
In your scheme, GPU VA can != GPU VA. This does introduce some flexibility.
But this scheme alone doesn't solve the problem of the proxy process/para- virtualization. You will still need a second mechanism to partition GPU VA space b/t guest process1 and guest process2 because proxy process (or the host hypervisor whatever you call it) use one single gpu page table for all the guest/client processes. GPU VA for different guest process can't overlap. If this second mechanism exist, we of course can use the same mechanism to partition CPU VA space between guest processes as well, then we can still use shared VA b/t CPU and GPU inside one process, but process1 and process2's address space (for both cpu and gpu) doesn't overlap. This second mechanism is the key to solve the proxy process problem, not the flexibility you introduced.
That approach was suggested before, but it doesn't work. First of all you create a massive security hole when you give the GPU full access to the QEMU CPU process which runs the virtualization.
So even if you say CPU VA == GPU VA you still need some kind of flexibility otherwise you can't implement this use case securely.
Additional to this the CPU VAs are usually controlled by the OS and not some driver, so to make sure that host and guest VAs don't overlap you would need to add some kind of sync between the guest and host OS kernels.
Quoted 1 line
In practice, your scheme also have a risk of running out of process space
because you have to partition whole address space b/t processes. Apparently allowing each guest process to own the whole process space and using separate GPU/CPU page table for different processes is a better solution than using single page table and partition process space b/t processes.
Yeah that you run out of address space is certainly possible. But as I said CPUs are switching to 5 level of pages tables and if you look at for example a "cat maps | cut -c-4 | sort -u" of process you will find that only a handful of 4GiB segments are actually used and thanks to recoverable page faults you can map those between host and client on demand. This gives you at least enough address space to handle a couple of thousand clients.
Quoted 1 line
For Intel GPU, para-virtualization (xenGT, see https://github.com/intel/XenGT-
Preview-kernel. It is similar idea of the proxy process in Flex's email. They are all SW-based GPU virtualization technology) is an old project. It is now replaced with HW accelerated SRIOV/system virtualization. XenGT is abandoned long time ago. So agreed your scheme add some flexibility. The question is, do we have a valid use case to use such flexibility? I don't see a single one ATM.
Yeah, we have SRIOV functionality on AMD hw as well, but for some use cases it's just to inflexible.
Quoted 1 line
I also pictured into how to implement your scheme. You basically rejected the
very foundation of hmm design which is shared address space b/t CPU and GPU. In your scheme, GPU VA = CPU VA + offset. In every single place where driver need to call hmm facilities such as
hmm_range_fault,migrate_vma_setupand in mmu notifier call back, you need to offset the GPU VA to get a CPU VA. From application writer's perspective, whenever he want to use a CPU pointer in his GPU program, he add to add that offset. Do you think this is awkward?What? This flexibility is there to prevent the application writer to change any offset.
Quoted 1 line
Finally, to implement SVM, we need to implement some memory hint API
which applies to a virtual address range across all GPU devices. For example, user would say, for this virtual address range, I prefer the backing store memory to be on GPU deviceX (because user knows deviceX would use this address range much more than other GPU devices or CPU). It doesn't make sense to me to make such API per device based. For example, if you tell device A that the preferred memory location is device B memory, this doesn't sounds correct to me because in your scheme, device A is not even aware of the existence of device B. right?
Correct and while the additional flexibility is somewhat option I strongly think that not having a centralized approach for device driver settings is mandatory.
Going away from the well defined file descriptor based handling of device driver interfaces was one of the worst ideas I've ever seen in roughly thirty years of working with Unixiode operating systems. It basically broke everything, from reverse lockup handling for
mmap()to file system privileges for hardware access.As far as I can see anything which goes into the direction of opening
/dev/kfdor/dev/xe_svmor something similar and saying that this then results into implicit SVM for your render nodes is an absolutely no-go and would required and explicit acknowledgement from Linus on the design to do something like that.What you can do is to have an IOCTL for the render node file descriptor which says this device should do SVM with the current CPU address space and another IOCTL which says range A..B is preferred to migrate to this device for HMM when the device runs into a page fault.
And yes that obviously means shitty performance for device drivers because page play ping/pong if userspace gives contradicting information for migrations, but that is something supposed to be.
Everything else which works over the boarders of a device driver scope should be implemented as system call with the relevant review process around it.
Regards,
Christian.Quoted 15 lines
Regards,
OakQuoted 12 lines
-----Original Message-----
From: Daniel Vetter <[email protected]>
Sent: Wednesday, January 31, 2024 4:15 AM
To: David Airlie <[email protected]>
Cc: Zeng, Oak <[email protected]>; Christian König <[email protected]>; Thomas Hellström <[email protected]>; Daniel Vetter <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; dri- [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhgQuoted 2 lines
Quoted 2 lines
Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah,
Ankur N
Quoted 4 lines
Quoted 4 lines
<[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Makingdrm_gpuvmwork across gpu devicesQuoted 8 lines
Quoted 8 lines · David Airlie
Quoted 6 lines · Oak
Quoted 5 lines
Hi Christian,
Nvidia Nouveau driver uses exactly the same concept of SVM with HMM,
GPU address in the same process is exactly the same with CPU virtual address.
It
Quoted 25 lines
Quoted 25 lines
is already in upstream Linux kernel. We Intel just follow the same direction for our customers. Why we are not allowed?
Quoted 9 lines
Oak, this isn't how upstream works, you don't get to appeal to customer or internal design. nouveau isn't "NVIDIA"'s and it certainly isn't something NVIDIA would ever suggest for their customers. We also likely wouldn't just accept NVIDIA's current solution upstream without some serious discussions. The implementation in nouveau was more of a sample HMM use case rather than a serious implementation. I suspect if we do get down the road of making nouveau an actual compute driver for SVM etc then it would have to severely change.
Yeah on the nouveau hmm code specifically my gut feeling impression is that we didn't really make friends with that among core kernel maintainers. It's a bit too much just a tech demo to be able to merge the hmm core apis for nvidia's out-of-tree driver.
Also, a few years of learning and experience gaining happened meanwhile - you always have to look at an api design in the context of when it was designed, and that context changes all the time.
Cheers, Sima
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
Re: Making drm_gpuvm work across gpu devices
Hi Oak,
sorry the mail sounded like you didn't expected a reply.
And yes, the approaches outlined in the mail sounds really good to me.
Regards,
Christian.
Am 08.03.24 um 05:43 schrieb Zeng, Oak:
Quoted 219 lines
Hello all,
Since I didn't get a reply for this one, I assume below are agreed. But feel free to let us know if you don't agree.
Thanks,
Oak-----Original Message-----
From: dri-devel <[email protected]> On Behalf Of Zeng, Oak
Sent: Thursday, February 29, 2024 1:23 PM
To: Christian König <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: Makingdrm_gpuvmwork across gpu devicesHi Christian/Daniel/Dave/Felix/Thomas, and all,
We have been refining our design internally in the past month. Below is our plan. Please let us know if you have any concern.
1) Remove pseudo
/dev/xe-svmdevice. All system allocator interfaces will be through/dev/dri/renderdevices. Not global interface.2) Unify userptr and system allocator codes. We will treat userptr as a speciality of system allocator without migration capability. We will introduce the hmmptr concept for system allocator. We will extend
vm_bindAPI to map a range A..B of process address space to a range C..D of GPU address space for hmmptr. For hmmptr, if gpu program accesses an address which is not backed by core mm vma, it is a fatal error.3) Multiple device support. We have identified p2p use-cases where we might want to leave memory on a foreign device or direct migrations to a foreign device and therefore might need a global structure that tracks or caches the migration state per process address space. We didn't completely settle down this design. We will come back when we have more details.
4)We will first work on this code on xekmd then look to move some common codes to drm layer so it can also be used by other vendors.
Thomas and me still have open questions to Christian. We will follow up.
Thanks all for this discussion.
Regards,
OakQuoted 186 lines
-----Original Message-----
From: Christian König <[email protected]>
Sent: Thursday, February 1, 2024 3:52 AM
To: Zeng, Oak <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Makingdrm_gpuvmwork across gpu devicesHi Oak,
Am 31.01.24 um 21:17 schrieb Zeng, Oak:
Quoted 3 lines
Hi Sima, Dave,
I am well aware nouveau driver is not what Nvidia do with their customer. The
key argument is, can we move forward with the concept shared virtual address space b/t CPU and GPU? This is the foundation of HMM. We already have split address space support with other driver API. SVM, from its name, it means shared address space. Are we allowed to implement another driver model to allow SVM work, along with other APIs supporting split address space? Those two scheme can co-exist in harmony. We actually have real use cases to use both models in one application.
Quoted 3 lines
Hi Christian, Thomas,
In your scheme, GPU VA can != GPU VA. This does introduce some flexibility.
But this scheme alone doesn't solve the problem of the proxy process/para- virtualization. You will still need a second mechanism to partition GPU VA space b/t guest process1 and guest process2 because proxy process (or the host hypervisor whatever you call it) use one single gpu page table for all the guest/client processes. GPU VA for different guest process can't overlap. If this second mechanism exist, we of course can use the same mechanism to partition CPU VA space between guest processes as well, then we can still use shared VA b/t CPU and GPU inside one process, but process1 and process2's address space (for both cpu and gpu) doesn't overlap. This second mechanism is the key to solve the proxy process problem, not the flexibility you introduced.
That approach was suggested before, but it doesn't work. First of all you create a massive security hole when you give the GPU full access to the QEMU CPU process which runs the virtualization.
So even if you say CPU VA == GPU VA you still need some kind of flexibility otherwise you can't implement this use case securely.
Additional to this the CPU VAs are usually controlled by the OS and not some driver, so to make sure that host and guest VAs don't overlap you would need to add some kind of sync between the guest and host OS kernels.
Quoted 1 line
In practice, your scheme also have a risk of running out of process space
because you have to partition whole address space b/t processes. Apparently allowing each guest process to own the whole process space and using separate GPU/CPU page table for different processes is a better solution than using single page table and partition process space b/t processes.
Yeah that you run out of address space is certainly possible. But as I said CPUs are switching to 5 level of pages tables and if you look at for example a "cat maps | cut -c-4 | sort -u" of process you will find that only a handful of 4GiB segments are actually used and thanks to recoverable page faults you can map those between host and client on demand. This gives you at least enough address space to handle a couple of thousand clients.
Quoted 1 line
For Intel GPU, para-virtualization (xenGT, see https://github.com/intel/XenGT-
Preview-kernel. It is similar idea of the proxy process in Flex's email. They are all SW-based GPU virtualization technology) is an old project. It is now replaced with HW accelerated SRIOV/system virtualization. XenGT is abandoned long time ago. So agreed your scheme add some flexibility. The question is, do we have a valid use case to use such flexibility? I don't see a single one ATM.
Yeah, we have SRIOV functionality on AMD hw as well, but for some use cases it's just to inflexible.
Quoted 1 line
I also pictured into how to implement your scheme. You basically rejected the
very foundation of hmm design which is shared address space b/t CPU and GPU. In your scheme, GPU VA = CPU VA + offset. In every single place where driver need to call hmm facilities such as
hmm_range_fault,migrate_vma_setupand in mmu notifier call back, you need to offset the GPU VA to get a CPU VA. From application writer's perspective, whenever he want to use a CPU pointer in his GPU program, he add to add that offset. Do you think this is awkward?What? This flexibility is there to prevent the application writer to change any offset.
Quoted 1 line
Finally, to implement SVM, we need to implement some memory hint API
which applies to a virtual address range across all GPU devices. For example, user would say, for this virtual address range, I prefer the backing store memory to be on GPU deviceX (because user knows deviceX would use this address range much more than other GPU devices or CPU). It doesn't make sense to me to make such API per device based. For example, if you tell device A that the preferred memory location is device B memory, this doesn't sounds correct to me because in your scheme, device A is not even aware of the existence of device B. right?
Correct and while the additional flexibility is somewhat option I strongly think that not having a centralized approach for device driver settings is mandatory.
Going away from the well defined file descriptor based handling of device driver interfaces was one of the worst ideas I've ever seen in roughly thirty years of working with Unixiode operating systems. It basically broke everything, from reverse lockup handling for
mmap()to file system privileges for hardware access.As far as I can see anything which goes into the direction of opening
/dev/kfdor/dev/xe_svmor something similar and saying that this then results into implicit SVM for your render nodes is an absolutely no-go and would required and explicit acknowledgement from Linus on the design to do something like that.What you can do is to have an IOCTL for the render node file descriptor which says this device should do SVM with the current CPU address space and another IOCTL which says range A..B is preferred to migrate to this device for HMM when the device runs into a page fault.
And yes that obviously means shitty performance for device drivers because page play ping/pong if userspace gives contradicting information for migrations, but that is something supposed to be.
Everything else which works over the boarders of a device driver scope should be implemented as system call with the relevant review process around it.
Regards,
Christian.Quoted 14 lines
Regards,
OakQuoted 12 lines
-----Original Message-----
From: Daniel Vetter <[email protected]>
Sent: Wednesday, January 31, 2024 4:15 AM
To: David Airlie <[email protected]>
Cc: Zeng, Oak <[email protected]>; Christian König <[email protected]>; Thomas Hellström <[email protected]>; Daniel Vetter <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; dri- [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhgQuoted 2 lines
Quoted 2 lines
Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah,
Ankur N
Quoted 4 lines
Quoted 4 lines
<[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Makingdrm_gpuvmwork across gpu devicesQuoted 8 lines
Quoted 8 lines · David Airlie
Quoted 6 lines · Oak
Quoted 5 lines
Hi Christian,
Nvidia Nouveau driver uses exactly the same concept of SVM with HMM,
GPU address in the same process is exactly the same with CPU virtual address.
It
Quoted 24 lines
Quoted 24 lines
is already in upstream Linux kernel. We Intel just follow the same direction for our customers. Why we are not allowed?
Quoted 8 lines
Oak, this isn't how upstream works, you don't get to appeal to customer or internal design. nouveau isn't "NVIDIA"'s and it certainly isn't something NVIDIA would ever suggest for their customers. We also likely wouldn't just accept NVIDIA's current solution upstream without some serious discussions. The implementation in nouveau was more of a sample HMM use case rather than a serious implementation. I suspect if we do get down the road of making nouveau an actual compute driver for SVM etc then it would have to severely change.
Yeah on the nouveau hmm code specifically my gut feeling impression is that we didn't really make friends with that among core kernel maintainers. It's a bit too much just a tech demo to be able to merge the hmm core apis for nvidia's out-of-tree driver.
Also, a few years of learning and experience gaining happened meanwhile - you always have to look at an api design in the context of when it was designed, and that context changes all the time.
Cheers, Sima
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
Thomas Hellström
maintainerRe: Making drm_gpuvm work across gpu devices
In reply to Zeng, OakHi, Oak,
Quoted 15 lines · Oak
The example you used to prove that KFD is a design failure, is against *any* design which utilize system allocator and hmm. The way that one proxy process running on host to handle many guest processes, doesn’t fit into the concept of “share address space b/t cpu and gpu”. The shared address space has to be within one process. Your proxy process represent many guest processes. It is a fundamental conflict.
Also your userptr proposal does’t solve this problem either:
Imagine you have guest process1 mapping CPU address range A…B to GPU address range C…D
And you have guest process 2 mapping CPU address range A…B to GPU address range C…D, since process 1 and 2 are two different process, it is legal for process 2 to do the exact same mapping.
Now when gpu shader access address C…D, a gpu page fault happens, what does your proxy process do? Which guest process will this fault be directed to and handled? Except you have extra information/API to tell proxy process and GPU HW, there is no way to figure out.
Compared to the shared virtual address space concept of HMM, the userptr design is nothing new except it allows CPU and GPU to use different address to access the same object. If you replace above C…D with A…B, above description becomes a description of the “problem” of HMM/shared virtual address design.
Both design has the same difficulty with your example of the special virtualization environment setup.
Quoted 9 lines
As said, we spent effort scoped the userptr solution some time ago. The problem we found enabling userptr with migration were:
1. The user interface of userptr is not as convenient as system allocator. With the userptr solution, user need to call
userptr_ioctlandvm_bindfor *every* single cpu pointer that he want to use in a gpu program. While with system allocator, programmer just use any cpu pointer directly in gpu program without any extra driver ioctls.
No, the augmented userptr (lets call it "hmmptr" to distinguish here) would typically only be bound once when the VM is created. It's just a different way to expose the whole SVM mapping to user-space. It's sparsely populated and is not backed by a bo, and it is per-device so UMD would have to replicate the SVM setup and attribute settings on each device.
Quoted 11 lines
1.
2. We don’t see the real benefit of using a different Gpu address C…D than the A..B, except you can prove my above reasoning is wrong. In most use cases, you can make GPU C…D == CPU A…B, why bother then?
3. Looked into implementation details, since hmm fundamentally assume a shared virtual address space b/t cpu and device, for the userptr solution to leverage hmm, you need perform address space conversion every time you calls into hmm functions.
I think very much focus lands on the A..B -> C..D mapping in the discussion. It's just an added flexibility with little or no implementation cost. Although I must admit I'm not fully clear about the actual use-case. In a para-virtualized environment like virGL or vmware's vmx/renderers I could imagine C..D being the guest virtual addresses including compute kernel pointers, A..B being the host renderer's CPU virtual addresses. (Host creates the VM's, and then this translation is needed. I'm not sure para-virtualized SVM exists ATM, but forcing A==C, B==D in the uAPI would rule out such a beast in the future?)
/Thomas
Quoted 429 lines
1.
In summary, GPU device is just a piece of HW to accelerate your CPU program. If HW allows, it is more convenient to use shared address space b/t cpu and GPU. On old HW (example, no gpu page fault support, or gpu only has a very limited address space), we can disable system allocator/SVM. If you use different address space on modern GPU, why don’t you use different address space on different CPU cores?
Regards,
Oak
*From:*dri-devel <[email protected]> *On Behalf Of *Christian König
*Sent:* Monday, January 29, 2024 5:20 AM
*To:* Zeng, Oak <[email protected]>; Thomas Hellström <[email protected]>; Daniel Vetter <[email protected]>; Dave Airlie <[email protected]>
*Cc:* Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; [email protected]; Danilo Krummrich <[email protected]> *Subject:* Re: Makingdrm_gpuvmwork across gpu devicesWell Daniel and Dave noted it as well, so I'm just repeating it: Your design choices are not an argument to get something upstream.
It's the job of the maintainers and at the end of the Linus to judge of something is acceptable or not.
As far as I can see a good part of this this idea has been exercised lengthy with KFD and it turned out to not be the best approach.
So from what I've seen the design you outlined is extremely unlikely to go upstream.
Regards,
Christian.Am 27.01.24 um 03:21 schrieb Zeng, Oak:
Regarding the idea of expanding userptr to support migration, we explored this idea long time ago. It provides similar functions of the system allocator but its interface is not as convenient as system allocator. Besides the shared virtual address space, another benefit of a system allocator is, you can offload cpu program to gpu easier, you don’t need to call driver specific API (such as
register_userptrandvm_bindin this case) for memory allocation.We also scoped the implementation. It turned out to be big, and not as beautiful as hmm. Why we gave up this approach.
*From:*Christian König <[email protected]> <mailto:[email protected]>
*Sent:* Friday, January 26, 2024 7:52 AM
*To:* Thomas Hellström <[email protected]> <mailto:[email protected]>; Daniel Vetter <[email protected]> <mailto:[email protected]>
*Cc:* Brost, Matthew <[email protected]> <mailto:[email protected]>; Felix Kuehling <[email protected]> <mailto:[email protected]>; Welty, Brian <[email protected]> <mailto:[email protected]>; Ghimiray, Himal Prasad <[email protected]> <mailto:[email protected]>; Zeng, Oak <[email protected]> <mailto:[email protected]>; Gupta, saurabhg <[email protected]> <mailto:[email protected]>; Danilo Krummrich <[email protected]> <mailto:[email protected]>; [email protected] <mailto:[email protected]>; Bommu, Krishnaiah <[email protected]> <mailto:[email protected]>; Dave Airlie <[email protected]> <mailto:[email protected]>; Vishwanathapura, Niranjana <[email protected]> <mailto:[email protected]>; [email protected] <mailto:[email protected]> *Subject:* Re: Makingdrm_gpuvmwork across gpu devicesAm 26.01.24 um 09:21 schrieb Thomas Hellström:
Hi, all
On Thu, 2024-01-25 at 19:32 +0100, Daniel Vetter wrote:
On Wed, Jan 24, 2024 at 09:33:12AM +0100, Christian König wrote:
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
[SNIP]
Yes most API are per device based.
One exception I know is actually the kfd SVM API. If you look at
the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.
Yeah and that was a big mistake in my opinion. We should really not
do that
ever again.
Need to say, kfd SVM represent a shared virtual address space
across CPU and all GPU devices on the system. This is by the
definition of SVM (shared virtual memory). This is very different
from our legacy gpu *device* driver which works for only one
device (i.e., if you want one device to access another device's
memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker
for a
virtualization projects. Having SVM as device independent feature
which
somehow ties to the process address space turned out to be an
extremely bad
idea.
The background is that this only works for some use cases but not
all of
them.
What's working much better is to just have a mirror functionality
which says
that a range A..B of the process address space is mapped into a
range C..D
of the GPU address space.
Those ranges can then be used to implement the SVM feature required
for
higher level APIs and not something you need at the UAPI or even
inside the
low level kernel memory management.
When you talk about migrating memory to a device you also do this
on a per
device basis and *not* tied to the process address space. If you
then get
crappy performance because userspace gave contradicting information
where to
migrate memory then that's a bug in userspace and not something the
kernel
should try to prevent somehow.
[SNIP]
I think if you start using the same
drm_gpuvmfor multipledevices you
will sooner or later start to run into the same mess we have
seen with
KFD, where we moved more and more functionality from the KFD to
the DRM
render node because we found that a lot of the stuff simply
doesn't work
correctly with a single object to maintain the state.
As I understand it, KFD is designed to work across devices. A
single pseudo
/dev/kfddevice represent all hardware gpu devices.That is why during kfd open, many pdd (process device data) is
created, each for one hardware device for this process.
Yes, I'm perfectly aware of that. And I can only repeat myself that
I see
this design as a rather extreme failure. And I think it's one of
the reasons
why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of
extending the
CPU process into the GPUs, but this idea only works for a few use
cases and
is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up
with CPU
address != GPU address because the VAs are actually coming from the
guest VM
and not the host process.
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should
not have
any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need
to get
explicit permission to do this from Dave and Daniel and maybe even
Linus.
I think the one and only one exception where an SVM uapi like in kfd
makes
sense, is if the _hardware_ itself, not the software stack defined
semantics that you've happened to build on top of that hw, enforces a
1:1
mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation,
PCI-ATS
(address translation services) or whatever your hw calls it and has
_no_
device-side pagetables on top. Which from what I've seen all devices
with
device-memory have, simply because they need some place to store
whether
that memory is currently in device memory or should be translated
using
PASID. Currently there's no gpu that works with PASID only, but there
are
some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully
cpu
cache coherent (including atomics) with something like CXL, and the
on-device memory is managed as normal system memory with struct page
as
ZONE_DEVICEand accelerator va -> physical address translation isonly
done with PASID ... but for now I haven't seen that, definitely not
in
upstream drivers.
And the moment you have some per-device pagetables or per-device
memory
management of some sort (like using gpuva mgr) then I'm 100% agreeing
with
Christian that the kfd SVM model is too strict and not a great idea.
Cheers, Sima
I'm trying to digest all the comments here, The end goal is to be able
to support something similar to this here:
Christian, If I understand you correctly, you're strongly suggesting
not to try to manage a common virtual address space across different
devices in the kernel, but merely providing building blocks to do so,
like for example a generalized userptr with migration support using
HMM; That way each "mirror" of the CPU mm would be per device and
inserted into the
gpu_vmjust like any othergpu_vma, and user-spacewould dictate the A..B -> C..D mapping by choosing the
GPU_VAfor thevma.
Exactly that, yes.
Sima, it sounds like you're suggesting to shy away from hmm and not
even attempt to support this except if it can be done using IOMMU sva
on selected hardware?
I think that comment goes more into the direction of: If you have ATS/ATC/PRI capable hardware which exposes the functionality to make memory reads and writes directly into the address space of the CPU then yes an SVM only interface is ok because the hardware can't do anything else. But as long as you have something like GPUVM then please don't restrict yourself.
Which I totally agree on as well. The ATS/ATC/PRI combination doesn't allow using separate page tables device and CPU and so also not separate VAs.
This was one of the reasons why we stopped using this approach for AMD GPUs.
Regards,
Christian.Could you clarify a bit?
Thanks,
Thomas
Re: Making drm_gpuvm work across gpu devices
In reply to Daniel VetterQuoted 76 lines · Daniel Vetter
Quoted 50 lines · Christian König
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.
Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Regards,
Felix
Quoted 2 lines
Cheers, Sima
Re: Making drm_gpuvm work across gpu devices
Am 29.01.24 um 16:03 schrieb Felix Kuehling:
Quoted 77 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.
Quoted 7 lines
Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.
The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.
As far as I can see there is absolutely no need for any special SVM handling.
Regards,
Christian.
Quoted 7 lines
Regards,
FelixQuoted 2 lines
Cheers, Sima
Re: Making drm_gpuvm work across gpu devices
Quoted 84 lines · Christian König
Am 29.01.24 um 16:03 schrieb Felix Kuehling:
Quoted 83 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Quoted 7 lines
Quoted 1 line
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.
The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.
As far as I can see there is absolutely no need for any special SVM handling.
It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.
Regards,
Felix
Quoted 12 lines
Regards,
Christian.Quoted 7 lines
Regards,
FelixQuoted 2 lines
Cheers, Sima
Re: Making drm_gpuvm work across gpu devices
Am 29.01.24 um 17:24 schrieb Felix Kuehling:
Quoted 87 lines · Christian König
Quoted 86 lines
Am 29.01.24 um 16:03 schrieb Felix Kuehling:
Quoted 84 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Quoted 7 lines
Quoted 5 lines
Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.
The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.
As far as I can see there is absolutely no need for any special SVM handling.
It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.
Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.
There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.
So this isn't a blocker, it's just one special use case.
Regards,
Christian.
Quoted 17 lines
Regards,
FelixQuoted 12 lines
Regards,
Christian.Quoted 7 lines
Regards,
FelixQuoted 2 lines
Cheers, Sima
Re: Making drm_gpuvm work across gpu devices
Quoted 90 lines · Christian König
Am 29.01.24 um 17:24 schrieb Felix Kuehling:
Quoted 89 lines · Christian König
Quoted 88 lines
Am 29.01.24 um 16:03 schrieb Felix Kuehling:
Quoted 77 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.Quoted 7 lines
Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.
As far as I can see there is absolutely no need for any special SVM handling.
It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.
Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.
What I want is that address range A..B on the CPU matches A..B on the GPU, because I'm sharing pointers between CPU and GPU. I can't think of any sane user mode using a unified memory programming model, that would ever ask KFD to map unified memory mappints to a different address range on the GPU. Adding such an ioclt is a complete waste of time, and can only serve to add unnecessary complexity.
Regards,
Felix
Quoted 26 lines
There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.
So this isn't a blocker, it's just one special use case.
Regards,
Christian.Quoted 17 lines
Regards,
FelixQuoted 12 lines
Regards,
Christian.Quoted 7 lines
Regards,
FelixQuoted 2 lines
Cheers, Sima
Re: Making drm_gpuvm work across gpu devices
Am 29.01.24 um 18:52 schrieb Felix Kuehling:
Quoted 93 lines · Christian König
Quoted 92 lines
Am 29.01.24 um 17:24 schrieb Felix Kuehling:
Quoted 91 lines · Christian König
Quoted 90 lines
Am 29.01.24 um 16:03 schrieb Felix Kuehling:
Quoted 77 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.Quoted 7 lines
Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.
The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
As far as I can see there is absolutely no need for any special SVM handling.
It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.
Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.
What I want is that address range A..B on the CPU matches A..B on the GPU, because I'm sharing pointers between CPU and GPU. I can't think of any sane user mode using a unified memory programming model, that would ever ask KFD to map unified memory mappints to a different address range on the GPU. Adding such an ioclt is a complete waste of time, and can only serve to add unnecessary complexity.
This is exactly the use case which happens when the submitting process is not the one originally stitching together the command stream.
Basically all native context, virtualization and other proxy use cases work like this.
So that the CPU address doesn't match the GPU address is an absolutely real use case and should be able to be handled by the GPU VA interface.
Regards,
Christian.
Quoted 31 lines
Regards,
FelixQuoted 26 lines
There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.
So this isn't a blocker, it's just one special use case.
Regards,
Christian.Quoted 17 lines
Regards,
FelixQuoted 12 lines
Regards,
Christian.Quoted 7 lines
Regards,
FelixQuoted 2 lines
Cheers, Sima
Re: Making drm_gpuvm work across gpu devices
Quoted 98 lines · Christian König
Am 29.01.24 um 18:52 schrieb Felix Kuehling:
Quoted 97 lines · Christian König
Quoted 96 lines
Am 29.01.24 um 17:24 schrieb Felix Kuehling:
Quoted 94 lines · Christian König
Quoted 84 lines
Am 29.01.24 um 16:03 schrieb Felix Kuehling:
Quoted 83 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
Quoted 4 lines
[SNIP]
Yes most API are per device based.One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Quoted 1 line
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
Quoted 6 lines
Quoted 5 lines
I think if you start using the same
drm_gpuvmfor multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
Quoted 5 lines
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as
ZONE_DEVICEand accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.Quoted 3 lines
And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.
That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.
Quoted 7 lines
Quoted 1 line
Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.
Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.
The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.
As far as I can see there is absolutely no need for any special SVM handling.
It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.
What I want is that address range A..B on the CPU matches A..B on the GPU, because I'm sharing pointers between CPU and GPU. I can't think of any sane user mode using a unified memory programming model, that would ever ask KFD to map unified memory mappints to a different address range on the GPU. Adding such an ioclt is a complete waste of time, and can only serve to add unnecessary complexity.
This is exactly the use case which happens when the submitting process is not the one originally stitching together the command stream.
Basically all native context, virtualization and other proxy use cases work like this.
You cannot use unified memory in this type of environment. A GPU page fault would occur in a GPU virtual address space in the host-side proxy process. That page fault would need to be resolved to map some memory from a process running in a guest? Which guest? Which process? That's anyone's guess. There is no way to annotate that because the pointer in the fault is coming from a shader program that's running in the guest context and competely unaware of the virtualization. There are no API calls from the guest before the fault occurs to establish a meaningful mapping.
The way this virtualization of the GPU is implemented, with out proxy multiplexing multiple clients is just fundamentally incompatible with a unified memory programming model that has to assume a shared virtual address space to make sense. You'd need separate proxy processes on the host side to handle that. You can't blame this on bad design decisions in the SVM API. As I see it, it's just a fundamental limitation of the virtualization approach that cannot handle guest applications that want to use a unified shared memory programming model.
That's why I suggested to completely disable the SVM API in this virtualization scenario when you create a KFD context that's separate from the process address space. It is not essential for any non-unified-memory functionality. ROCm user mode has fallbacks to work without it, because we also needs to support older kernels and GPUs that didn't support this programming model.
Regards,
Felix
Quoted 40 lines
So that the CPU address doesn't match the GPU address is an absolutely real use case and should be able to be handled by the GPU VA interface.
Regards,
Christian.Quoted 31 lines
Regards,
FelixQuoted 26 lines
There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.
So this isn't a blocker, it's just one special use case.
Regards,
Christian.Quoted 17 lines
Regards,
FelixQuoted 12 lines
Regards,
Christian.Quoted 7 lines
Regards,
FelixQuoted 2 lines
Cheers, Sima
RE: Making drm_gpuvm work across gpu devices
In reply to Christian KönigHi Christian,
I go back this old email to ask a question.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.
It looks like you were talking of 1). Were you?
Oak
From: Christian König <[email protected]>
Sent: Wednesday, January 24, 2024 3:33 AM
To: Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>
Cc: Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
Subject: Re: Making drm_gpuvm work across gpu devices
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
[SNIP]
Yes most API are per device based.
One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.
Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
I think if you start using the same drm_gpuvm for multiple devices you
will sooner or later start to run into the same mess we have seen with
KFD, where we moved more and more functionality from the KFD to the DRM
render node because we found that a lot of the stuff simply doesn't work
correctly with a single object to maintain the state.
As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.
Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
Regards,
Christian.
Re: Making drm_gpuvm work across gpu devices
Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Quoted 5 lines
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quoted 27 lines
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.”
“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernelDocumentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.
Quoted 2 lines
It looks like you were talking of 1). Were you?
No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.
And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.
I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.
Regards,
Christian.
Quoted 61 lines
Oak
*From:*Christian König <[email protected]>
*Sent:* Wednesday, January 24, 2024 3:33 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]> *Subject:* Re: Makingdrm_gpuvmwork across gpu devicesAm 23.01.24 um 20:37 schrieb Zeng, Oak:
[SNIP]
Yes most API are per device based.
One exception I know is actually the kfd SVM API. If you look at the
svm_ioctlfunction, it is per-process based. Eachkfd_processrepresent a process across N gpu devices.Yeah and that was a big mistake in my opinion. We should really not do that ever again.
Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.
The background is that this only works for some use cases but not all of them.
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.
[SNIP]
I think if you start using the same
drm_gpuvmfor multiple devices youwill sooner or later start to run into the same mess we have seen with
KFD, where we moved more and more functionality from the KFD to the DRM
render node because we found that a lot of the stuff simply doesn't work
correctly with a single object to maintain the state.
As I understand it, KFD is designed to work across devices. A single pseudo
/dev/kfddevice represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.
Regards,
Christian.
RE: Making drm_gpuvm work across gpu devices
From: Christian König <[email protected]>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
Cc: Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
Subject: Re: Making drm_gpuvm work across gpu devices
Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.
It looks like you were talking of 1). Were you?
No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.
And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.
Lets say we have a hardware platform where both CPU and GPU support 57bit virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space (0~2^57-1), do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program.
I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.
Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we don’t map the whole vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.
The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma.
Regards,
Oak
Regards,
Christian.
RE: Making drm_gpuvm work across gpu devices
The mail wasn’t indent/preface correctly. Manually format it.
From: Christian König <[email protected]<mailto:[email protected]>>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <[email protected]<mailto:[email protected]>>; Danilo Krummrich <[email protected]<mailto:[email protected]>>; Dave Airlie <[email protected]<mailto:[email protected]>>; Daniel Vetter <[email protected]<mailto:[email protected]>>; Felix Kuehling <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>
Cc: Welty, Brian <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; Bommu, Krishnaiah <[email protected]<mailto:[email protected]>>; Ghimiray, Himal Prasad <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; Vishwanathapura, Niranjana <[email protected]<mailto:[email protected]>>; Brost, Matthew <[email protected]<mailto:[email protected]>>; Gupta, saurabhg <[email protected]<mailto:[email protected]>>
Subject: Re: Making drm_gpuvm work across gpu devices
Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.
It looks like you were talking of 1). Were you?
No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.
And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.
[Zeng, Oak]
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?
I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.
[Zeng, Oak]
Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.
The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?
Regards,
Oak
Regards,
Christian.
Re: Making drm_gpuvm work across gpu devices
Am 28.02.24 um 20:51 schrieb Zeng, Oak:
Quoted 54 lines
The mail wasn’t indent/preface correctly. Manually format it.
*From:*Christian König <[email protected]>
*Sent:* Tuesday, February 27, 2024 1:54 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]> *Subject:* Re: Makingdrm_gpuvmwork across gpu devicesHi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.”
“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1.driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2.system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel
Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.It looks like you were talking of 1). Were you?
Quoted 8 lines
No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.
And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.
*/[Zeng, Oak] /*
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?
Well you can perfectly mirror on demand. You just need something similar to userfaultfd() for the GPU. This way you don't need to mirror the full address space, but can rather work with large chunks created on demand, let's say 1GiB or something like that.
The virtual address space is basically just a hardware functionality to route memory accesses. While the mirroring approach is a very common use case for data-centers and high performance computing there are quite a number of different use cases which makes use of virtual address space in a non "standard" fashion. The native context approach for VMs is just one example, databases and emulators are another one.
Quoted 4 lines
I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.
Quoted 8 lines
*/[Zeng, Oak] /*
Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.
*//*
The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic
vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vmain xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?
Yes, exactly that. I mean the splitting and eventually merging of ranges is a standard functionality of the GPUVM code.
So when you need to store additional attributes per range then I would strongly suggest to make use of this splitting and merging functionality as well.
So basically an IOCTL which says range A..B of the GPU address space is mapped to offset X of the CPU address space with parameters Y (caching, migration behavior etc..). That is essentially the same we have for mapping GEM objects, the provider of the backing store is just something different.
Regards,
Christian.
Quoted 9 lines
Regards,
Oak
Regards,
Christian.
RE: Making drm_gpuvm work across gpu devices
Hi Christian,
Can you elaborate the mirror on demand/userfaultfd idea?
userfaultfd is a way for user space to take over page fault handling of a user registered range. From first look, it seems you want a user space page fault handler to mirror a large chunk of memory to GPU. I would imagine this handler is in UMD, because the whole purpose of system svm allocator is to allow user use cpu address (such as malloc’ed) on gpu program without extra driver api call. So the registration and mirroring of this large chunk can’t be in user program. With this, I pictured below sequence:
During process initialization time, umd register a large chunk (lets say 1GiB) of memory using userfaultfd, this include:
1. mem = mmap(NULL, 1GiB, MAP_ANON) 2. register range [mem, mem + 1GiB] through userfaultfd 3. after that, umd can wait on page fault event. When page fault happens, umd call vm_bind to mirror [mem, mem+1GiB] range to GPU
now in a user program:
ptr = malloc(size);
submit a GPU program which uses ptr
This is what I can picture. It doesn’t work because ptr can’t belong to [mem, mem+1GiB] range. So you can’t vm_bind/mirror ptr on demand to GPU.
Also, the page fault event in 3) above can’t happen at all. A page fault only happens when *CPU* access mem but in our case, it could be *only GPU* touch the memory.
The point is, with system svm allocator, user can use *any* valid CPU address for GPU program. This address can be anything in the range of [0~2^57-1]. This design requirement is quite simple and clean. I don’t see how to solve this with userfaultfd/on demand mirroring.
Regards,
Oak
From: Christian König <[email protected]>
Sent: Thursday, February 29, 2024 4:41 AM
To: Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
Cc: Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
Subject: Re: Making drm_gpuvm work across gpu devices
Am 28.02.24 um 20:51 schrieb Zeng, Oak:
The mail wasn’t indent/preface correctly. Manually format it.
From: Christian König <[email protected]<mailto:[email protected]>>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <[email protected]<mailto:[email protected]>>; Danilo Krummrich <[email protected]<mailto:[email protected]>>; Dave Airlie <[email protected]<mailto:[email protected]>>; Daniel Vetter <[email protected]<mailto:[email protected]>>; Felix Kuehling <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>
Cc: Welty, Brian <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; Bommu, Krishnaiah <[email protected]<mailto:[email protected]>>; Ghimiray, Himal Prasad <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; Vishwanathapura, Niranjana <[email protected]<mailto:[email protected]>>; Brost, Matthew <[email protected]<mailto:[email protected]>>; Gupta, saurabhg <[email protected]<mailto:[email protected]>>
Subject: Re: Making drm_gpuvm work across gpu devices
Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.
It looks like you were talking of 1). Were you?
No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.
And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.
[Zeng, Oak]
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?
Well you can perfectly mirror on demand. You just need something similar to userfaultfd() for the GPU. This way you don't need to mirror the full address space, but can rather work with large chunks created on demand, let's say 1GiB or something like that.
The virtual address space is basically just a hardware functionality to route memory accesses. While the mirroring approach is a very common use case for data-centers and high performance computing there are quite a number of different use cases which makes use of virtual address space in a non "standard" fashion. The native context approach for VMs is just one example, databases and emulators are another one.
I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.
[Zeng, Oak]
Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.
The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?
Yes, exactly that. I mean the splitting and eventually merging of ranges is a standard functionality of the GPUVM code.
So when you need to store additional attributes per range then I would strongly suggest to make use of this splitting and merging functionality as well.
So basically an IOCTL which says range A..B of the GPU address space is mapped to offset X of the CPU address space with parameters Y (caching, migration behavior etc..). That is essentially the same we have for mapping GEM objects, the provider of the backing store is just something different.
Regards,
Christian.
Regards,
Oak
Regards,
Christian.
Thomas Hellström
maintainerRe: Making drm_gpuvm work across gpu devices
In reply to Christian KönigHi, Christian.
Quoted 105 lines · Christian König
Am 28.02.24 um 20:51 schrieb Zeng, Oak:
Quoted 88 lines
The mail wasn’t indent/preface correctly. Manually format it.
*From:*Christian König <[email protected]> *Sent:* Tuesday, February 27, 2024 1:54 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad
<[email protected]>; [email protected];
Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
*Subject:* Re: Makingdrm_gpuvmwork across gpu devicesHi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the
UAPI or even inside the low level kernel memory management.”“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”
There are two category of SVM:
1.driver svm allocator: this is implemented in user space, i.g.,
cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd
and our umd implemented clSVMAlloc and zeMemAllocShared on top of
gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you
said.2.system svm allocator: This doesn’t introduce extra driver API
for memory allocation. Any valid CPU virtual address can be used
directly transparently in a GPU program without any extra driver
API call. Quote from kernelDocumentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently”
and “to share the address space by duplicating the CPU page table
in the device page table so the same address points to the same physical memory for any valid main memory address in the process
address space”. In system svm allocator, we don’t need that A..B
C..D mapping.It looks like you were talking of 1). Were you?
No, even when you fully mirror the whole address space from a process
into the GPU you still need to enable this somehow with an IOCTL.And while enabling this you absolutely should specify to which part of
the address space this mirroring applies and where it maps to.*/[Zeng, Oak] /*
Quoted 15 lines
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at
that time, you don’t know which part of the address space will be used
for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a
valid concern?
Quoted 7 lines
Well you can perfectly mirror on demand. You just need something similar
touserfaultfd()for the GPU. This way you don't need to mirror the full
address space, but can rather work with large chunks created on demand,
let's say 1GiB or something like that.
What we're looking at as the current design is an augmented userptr (A..B -> C..D mapping) which is internally sparsely populated in chunks. KMD manages the population using gpu pagefaults. We acknowledge that some parts of this mirror will not have a valid CPU mapping. That is, no vma so a gpu page-fault that resolves to such a mirror address will cause an error. Would you have any concerns / objections against such an approach?
Thanks,
Thomas
Re: Making drm_gpuvm work across gpu devices
Hi Thomas,
Am 29.02.24 um 18:12 schrieb Thomas Hellström:
Quoted 114 lines
Hi, Christian.
Quoted 103 lines · Christian König
Am 28.02.24 um 20:51 schrieb Zeng, Oak:
Quoted 87 lines
The mail wasn’t indent/preface correctly. Manually format it.
*From:*Christian König <[email protected]> *Sent:* Tuesday, February 27, 2024 1:54 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad
<[email protected]>; [email protected];
Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
*Subject:* Re: Makingdrm_gpuvmwork across gpu devicesHi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,
I go back this old email to ask a question.
sorry totally missed that one.
Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at
the
UAPI or even inside the low level kernel memory management.”“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel
UAPI.”There are two category of SVM:
1.driver svm allocator: this is implemented in user space, i.g.,
cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in
xekmd
and our umd implemented clSVMAlloc and zeMemAllocShared on top
of
gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as
you
said.2.system svm allocator: This doesn’t introduce extra driver
API
for memory allocation. Any valid CPU virtual address can be
used
directly transparently in a GPU program without any extra
driver
API call. Quote from kernelDocumentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device
transparently”
and “to share the address space by duplicating the CPU page
table
in the device page table so the same address points to the same physical memory for any valid main memory address in the
process
address space”. In system svm allocator, we don’t need that
A..B
C..D mapping.It looks like you were talking of 1). Were you?
No, even when you fully mirror the whole address space from a process
into the GPU you still need to enable this somehow with an IOCTL.And while enabling this you absolutely should specify to which part of
the address space this mirroring applies and where it maps to.*/[Zeng, Oak] /*
Quoted 15 lines
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at
that time, you don’t know which part of the address space will be used
for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a
valid concern?Quoted 7 lines
Well you can perfectly mirror on demand. You just need something similar
touserfaultfd()for the GPU. This way you don't need to mirror the full
address space, but can rather work with large chunks created on demand,
let's say 1GiB or something like that.
Quoted 7 lines
What we're looking at as the current design is an augmented userptr (A..B -> C..D mapping) which is internally sparsely populated in chunks. KMD manages the population using gpu pagefaults. We acknowledge that some parts of this mirror will not have a valid CPU mapping. That is, no vma so a gpu page-fault that resolves to such a mirror address will cause an error. Would you have any concerns / objections against such an approach?
Nope, as far as I can see that sounds like a perfectly valid design to me.
Regards,
Christian.
Quoted 6 lines
Thanks,
Thomas
PATCH 22/23 drm/xe/svm: Add DRM_XE_SVM kernel config entry
In reply to Oak ZengDRM_XE_SVM kernel config entry is added so
xe svm feature can be configured before kernel
compilation.
--- drivers/gpu/drm/xe/Kconfig | 22 ++++++++++++++++++++++ drivers/gpu/drm/xe/Makefile | 5 +++++ drivers/gpu/drm/xe/xe_mmio.c | 5 +++++ drivers/gpu/drm/xe/xe_vm.c | 2 ++ 4 files changed, 34 insertions(+)
drivers/gpu/drm/xe/Kconfig +22 -0
@@ -83,6 +83,28 @@ config DRM_XE_FORCE_PROBE Use "!*" to block the probe of the driver for all known devices. +config DRM_XE_SVM + bool "Enable Shared Virtual Memory support in xe" + depends on DRM_XE + depends on ARCH_ENABLE_MEMORY_HOTPLUG + depends on ARCH_ENABLE_MEMORY_HOTREMOVE + depends on MEMORY_HOTPLUG + depends on MEMORY_HOTREMOVE + depends on ARCH_HAS_PTE_DEVMAP + depends on SPARSEMEM_VMEMMAP + depends on ZONE_DEVICE + depends on DEVICE_PRIVATE + depends on MMU + select HMM_MIRROR + select MMU_NOTIFIER + default y + help + Choose this option if you want Shared Virtual Memory (SVM) + support in xe. With SVM, virtual address space is shared + between CPU and GPU. This means any virtual address such + as malloc or mmap returns, variables on stack, or global + memory pointers, can be used for GPU transparently. + menu "drm/Xe Debugging" depends on DRM_XE depends on EXPERT
drivers/gpu/drm/xe/Makefile +5 -0
@@ -283,6 +283,11 @@ xe-$(CONFIG_DRM_XE_DISPLAY) += \ i915-display/skl_universal_plane.o \ i915-display/skl_watermark.o +xe-$(CONFIG_DRM_XE_SVM) += xe_svm.o \ + xe_svm_devmem.o \ + xe_svm_range.o \ + xe_svm_migrate.o + ifeq ($(CONFIG_ACPI),y) xe-$(CONFIG_DRM_XE_DISPLAY) += \ i915-display/intel_acpi.o \
drivers/gpu/drm/xe/xe_mmio.c +5 -0
@@ -286,7 +286,9 @@ int xe_mmio_probe_vram(struct xe_device *xe) } io_size -=min_t (u64, tile_size, io_size); +#if IS_ENABLED (CONFIG_DRM_XE_SVM )xe_svm_devm_add (tile, &tile->mem.vram); +#endif } xe->mem.vram.actual_physical_size = total_size;
@@ -361,8 +363,11 @@ static void mmio_fini(struct drm_device *drm, void *arg)pci_iounmap (to_pci_dev (xe->drm.dev), xe->mmio.regs);if (xe->mem.vram.mapping)iounmap (xe->mem.vram.mapping); + +#if IS_ENABLED (CONFIG_DRM_XE_SVM )for_each_tile (tile, xe, id) {xe_svm_devm_remove (xe, &tile->mem.vram); +#endif } }
drivers/gpu/drm/xe/xe_vm.c +2 -0
@@ -1377,7 +1377,9 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags) xe->usm.num_vm_in_non_fault_mode++;mutex_unlock (&xe->usm.lock); +#if IS_ENABLED (CONFIG_DRM_XE_SVM ) vm->svm =xe_create_svm (vm); +#endif trace_xe_vm_create (vm);return vm;
Signature · 1 line
2.26.3
PATCH 23/23 drm/xe/svm: Add svm memory hints interface
In reply to Oak Zeng--- include/uapi/drm/xe_drm.h | 40 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
include/uapi/drm/xe_drm.h +40 -0
@@ -80,6 +80,7 @@ extern "C" {* - &DRM_IOCTL_XE_EXEC_QUEUE_GET_PROPERTY * - &DRM_IOCTL_XE_EXEC * - &DRM_IOCTL_XE_WAIT_USER_FENCE +* - &DRM_IOCTL_XE_SVM */ /*
@@ -100,6 +101,7 @@ extern "C" {#define DRM_XE_EXEC_QUEUE_GET_PROPERTY 0x08#define DRM_XE_EXEC 0x09#define DRM_XE_WAIT_USER_FENCE 0x0a +#define DRM_XE_SVM 0x0b/* Must be kept compact -- no holes */ #define DRM_IOCTL_XE_DEVICE_QUERY DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_DEVICE_QUERY, struct drm_xe_device_query)
@@ -113,6 +115,7 @@ extern "C" {#define DRM_IOCTL_XE_EXEC_QUEUE_GET_PROPERTY DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_EXEC_QUEUE_GET_PROPERTY, struct drm_xe_exec_queue_get_property)#define DRM_IOCTL_XE_EXEC DRM_IOW(DRM_COMMAND_BASE + DRM_XE_EXEC, struct drm_xe_exec)#define DRM_IOCTL_XE_WAIT_USER_FENCE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_WAIT_USER_FENCE, struct drm_xe_wait_user_fence) +#define DRM_IOCTL_XE_SVM DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_SVM, struct drm_xe_svm_args)/** * DOC: Xe IOCTL Extensions
@@ -1339,6 +1342,43 @@ struct drm_xe_wait_user_fence {__u64 reserved[2 ]; }; +enum drm_xe_svm_attr_type { +DRM_XE_SVM_ATTR_PREFERRED_LOC , +DRM_XE_SVM_ATTR_MIGRATION_GRANULARITY , +DRM_XE_SVM_ATTR_ATOMIC , +DRM_XE_SVM_ATTR_CACHE , +DRM_XE_SVM_ATTR_PREFETCH_LOC , +DRM_XE_SVM_ATTR_ACCESS_PATTERN , +}; + +struct drm_xe_svm_attr { +__u32 type; +__u32 value; +}; + +enum drm_xe_svm_op { +DRM_XE_SVM_OP_SET_ATTR , +DRM_XE_SVM_OP_GET_ATTR , +}; + +/** +* struct drm_xe_svm_args - Input of &DRM_IOCTL_XE_SVM +* +* Set or get memory attributes to a virtual address range +*/ +struct drm_xe_svm_args { +/** @start: start of the virtual address range */ +__u64 start; +/** @size: size of the virtual address range */ +__u64 size; +/** @op: operation, either set or get */ +__u32 op; +/** @nattr: number of attributes */ +__u32 nattr; +/** @attrs: An array of attributes */ +struct drm_xe_svm_attr attrs[]; +}; +#if defined(__cplusplus) }#endif
Signature · 1 line
2.26.3
Patchwork
CI✓ CI.Patch_applied: success for XeKmd basic SVM support
In reply to Oak Zeng== Series Details ==
== Series Details ==
Series: XeKmd basic SVM support
URL : https://patchwork.freedesktop.org/series/128910/
State : success
== Summary ==
=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: 2d422dec5 drm-tip: 2024y-01m-17d-23h-44m-17s UTC integration manifest === git am output follows ===
.git/rebase-apply/patch:167: new blank line at EOF.
+
warning: 1 line adds whitespace errors.
Applying: drm/xe/svm: Add SVM document
Applying: drm/xe/svm: Add svm key data structures
Applying: drm/xe/svm: create xe svm during vm creation
Applying: drm/xe/svm: Trace svm creation
Applying: drm/xe/svm: add helper to retrieve svm range from address
Applying: drm/xe/svm: Introduce a helper to build sg table from hmm range
Applying: drm/xe/svm: Add helper for binding hmm range to gpu
Applying: drm/xe/svm: Add helper to invalidate svm range from GPU
Applying: drm/xe/svm: Remap and provide memmap backing for GPU vram
Applying: drm/xe/svm: Introduce svm migration function
Applying: drm/xe/svm: implement functions to allocate and free device memory
Applying: drm/xe/svm: Trace buddy block allocation and free
Applying: drm/xe/svm: Handle CPU page fault
Applying: drm/xe/svm: trace svm range migration
Applying: drm/xe/svm: Implement functions to register and unregister mmu notifier
Applying: drm/xe/svm: Implement the mmu notifier range invalidate callback
Applying: drm/xe/svm: clean up svm range during process exit
Applying: drm/xe/svm: Move a few structures to xe_gt.h
Applying: drm/xe/svm: migrate svm range to vram
Applying: drm/xe/svm: Populate svm range
Applying: drm/xe/svm: GPU page fault support
Applying: drm/xe/svm: Add DRM_XE_SVM kernel config entry
Applying: drm/xe/svm: Add svm memory hints interface
Patchwork
CI✗ CI.checkpatch: warning for XeKmd basic SVM support
In reply to Oak Zeng== Series Details ==
== Series Details ==
Series: XeKmd basic SVM support
URL : https://patchwork.freedesktop.org/series/128910/
State : warning
== Summary ==
+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master e7f730565356938dd6c63cd498218fdc5d6a22b0
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit fca35d34b0e317dbe0033be49616255ba22c100d
Author: Oak Zeng <[email protected]> Date: Wed Jan 17 17:12:23 2024 -0500
drm/xe/svm: Add svm memory hints interface
+ /mt/dim checkpatch 2d422dec58e0dfd61d6c9842de7f83289803dccb drm-intel96a876fc2 drm/xe/svm: Add SVM document
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:28: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #28:
new file mode 100644
-:97: WARNING:TYPO_SPELLING: 'delibrately' may be misspelled - perhaps 'deliberately'? #97: FILE: drivers/gpu/drm/xe/xe_svm_doc.h:51:
+ * in our SVM implementation. We delibrately choose this implementation option
^^^^^^^^^^^
-:149: WARNING:TYPO_SPELLING: 'implemenation' may be misspelled - perhaps 'implementation'? #149: FILE: drivers/gpu/drm/xe/xe_svm_doc.h:103:
+ * implemenation and API" described the locking scheme that driver writer has to
^^^^^^^^^^^^^
-:163: WARNING:REPEATED_WORD: Possible repeated word: 'during' #163: FILE: drivers/gpu/drm/xe/xe_svm_doc.h:117:
+ * during during device page table update, if yes, then retry.
total: 0 errors, 4 warnings, 0 checks, 133 lines checked2743cb454 drm/xe/svm: Add svm key data structures
-:35: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #35:
new file mode 100644
-:40: WARNING:SPDX_LICENSE_TAG: Improper SPDX comment style for 'drivers/gpu/drm/xe/xe_svm.h', please use '/*' instead #40: FILE: drivers/gpu/drm/xe/xe_svm.h:1:
+// SPDX-License-Identifier: MIT
-:40: WARNING:SPDX_LICENSE_TAG: Missing or malformed SPDX-License-Identifier tag in line 1 #40: FILE: drivers/gpu/drm/xe/xe_svm.h:1:
+// SPDX-License-Identifier: MIT
total: 0 errors, 3 warnings, 0 checks, 73 lines checked21034598a drm/xe/svm: create xe svm during vm creation
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:25: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #25:
new file mode 100644
-:65: CHECK:ALLOC_SIZEOF_STRUCT: Prefer kzalloc(sizeof(*svm)...) over kzalloc(sizeof(struct xe_svm)...) #65: FILE: drivers/gpu/drm/xe/xe_svm.c:36:
+ svm = kzalloc(sizeof(struct xe_svm), GFP_KERNEL);
total: 0 errors, 1 warnings, 1 checks, 117 lines checked adb21aa60 drm/xe/svm: Trace svm creation37bbe7f04 drm/xe/svm: add helper to retrieve svm range from address Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:28: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #28: FILE: drivers/gpu/drm/xe/xe_svm.h:71:
+struct xe_svm_range *xe_svm_range_from_addr(struct xe_svm *svm,
+ unsigned long addr);
-:31: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #31:
new file mode 100644
-:56: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #56: FILE: drivers/gpu/drm/xe/xe_svm_range.c:21:
+struct xe_svm_range *xe_svm_range_from_addr(struct xe_svm *svm,
+ unsigned long addr)
total: 0 errors, 1 warnings, 2 checks, 38 lines checked ed971f094 drm/xe/svm: Introduce a helper to build sg table from hmm range -:57: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #57: FILE: drivers/gpu/drm/xe/xe_svm.c:85:
+int xe_svm_build_sg(struct hmm_range *range,
+ struct sg_table *st)
total: 0 errors, 0 warnings, 1 checks, 74 lines checked 989506d3d drm/xe/svm: Add helper for binding hmm range to gpu -:79: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #79: FILE: drivers/gpu/drm/xe/xe_pt.c:1066:
+ if (mmu_interval_read_retry(range->notifier,
+ range->notifier_seq)) {
-:139: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #139: FILE: drivers/gpu/drm/xe/xe_pt.c:1715:
+int xe_bind_svm_range(struct xe_vm *vm, struct xe_tile *tile,
+ struct hmm_range *range, u64 flags)
-:202: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #202: FILE: drivers/gpu/drm/xe/xe_pt.h:51:
+int xe_bind_svm_range(struct xe_vm *vm, struct xe_tile *tile,
+ struct hmm_range *range, u64 flags);
total: 0 errors, 0 warnings, 3 checks, 188 lines checkedc3aaebfaa drm/xe/svm: Add helper to invalidate svm range from GPUcd94b6534 drm/xe/svm: Remap and provide memmap backing for GPU vram
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:102: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #102:
new file mode 100644
-:118: CHECK:LINE_SPACING: Please don't use multiple blank lines #118: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:12:
+
+
-:168: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #168: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:62:
+ drm_err(&tile->xe->drm, "Failed to remap tile %d memory, errno %d\n",
+ tile->id, ret);
-:174: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #174: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:68:
+ drm_info(&tile->xe->drm, "Added tile %d memory [%llx-%llx] to devm, remapped to %pr\n",
+ tile->id, mr->io_start, mr->io_start + mr->usable_size, res);
-:190: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #190: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:84:
+ devm_release_mem_region(dev, mr->pagemap.range.start,
+ mr->pagemap.range.end - mr->pagemap.range.start +1);
-:190: CHECK:SPACING: spaces preferred around that '+' (ctx:WxV) #190: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:84:
+ mr->pagemap.range.end - mr->pagemap.range.start +1);
^
total: 0 errors, 1 warnings, 5 checks, 136 lines checked 6e63cd593 drm/xe/svm: Introduce svm migration function
-:80: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #80: FILE: drivers/gpu/drm/xe/xe_migrate.c:574:
+static void build_pt_update_batch_sram(struct xe_migrate *m,
+ struct xe_bb *bb, u32 pt_offset,
-:151: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #151: FILE: drivers/gpu/drm/xe/xe_migrate.c:955:
+struct dma_fence *xe_migrate_svm(struct xe_migrate *m,
+ u64 src_dpa,
-:192: CHECK:BRACES: braces {} should be used on all arms of this statement #192: FILE: drivers/gpu/drm/xe/xe_migrate.c:996:
+ if (!src_is_vram) {
[...]
+ else
[...]
-:195: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #195: FILE: drivers/gpu/drm/xe/xe_migrate.c:999:
+ build_pt_update_batch_sram(m, bb, pt_slot * XE_PAGE_SIZE,
+ src_dpa, round_update_size);
-:198: ERROR:ELSE_AFTER_BRACE: else should follow close brace '}' #198: FILE: drivers/gpu/drm/xe/xe_migrate.c:1002:
+ }
+ else
-:201: CHECK:BRACES: braces {} should be used on all arms of this statement #201: FILE: drivers/gpu/drm/xe/xe_migrate.c:1005:
+ if (!dst_is_vram) {
[...]
+ else
[...]
-:204: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #204: FILE: drivers/gpu/drm/xe/xe_migrate.c:1008:
+ build_pt_update_batch_sram(m, bb, pt_slot * XE_PAGE_SIZE,
+ dst_dpa, round_update_size);
-:207: ERROR:ELSE_AFTER_BRACE: else should follow close brace '}' #207: FILE: drivers/gpu/drm/xe/xe_migrate.c:1011:
+ }
+ else
-:210: CHECK:LINE_SPACING: Please don't use multiple blank lines #210: FILE: drivers/gpu/drm/xe/xe_migrate.c:1014:
+
+
-:270: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #270: FILE: drivers/gpu/drm/xe/xe_migrate.h:92:
+struct dma_fence *xe_migrate_svm(struct xe_migrate *m,
+ u64 src_dpa,
total: 2 errors, 0 warnings, 8 checks, 244 lines checked 150d7173a drm/xe/svm: implement functions to allocate and free device memory -:40: CHECK:LINE_SPACING: Please don't use multiple blank lines #40: FILE: drivers/gpu/drm/xe/xe_svm.h:78:
+
+
-:42: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #42: FILE: drivers/gpu/drm/xe/xe_svm.h:80:
+int xe_devm_alloc_pages(struct xe_tile *tile,
+ unsigned long npages,
-:116: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #116: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:67:
+int xe_devm_alloc_pages(struct xe_tile *tile,
+ unsigned long npages,
-:126: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #126: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:77:
+ ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE,
+ blocks, DRM_BUDDY_TOPDOWN_ALLOCATION);
-:147: ERROR:SPACING: space required before the open parenthesis '(' #147: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:98:
+ for(i = 0; i < pages_per_block; i++) {
total: 1 errors, 0 warnings, 4 checks, 176 lines checked4e5fce92b drm/xe/svm: Trace buddy block allocation and free
-:10: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one
-:56: ERROR:CODE_INDENT: code indent should use tabs where possible #56: FILE: drivers/gpu/drm/xe/xe_trace.h:605:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$
-:56: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #56: FILE: drivers/gpu/drm/xe/xe_trace.h:605:
+DECLARE_EVENT_CLASS(xe_buddy_block,
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),
-:56: WARNING:LEADING_SPACE: please, no spaces at the start of a line #56: FILE: drivers/gpu/drm/xe/xe_trace.h:605:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$
-:57: ERROR:CODE_INDENT: code indent should use tabs where possible #57: FILE: drivers/gpu/drm/xe/xe_trace.h:606:
+ TP_ARGS(block, size, pfn),$
-:57: WARNING:LEADING_SPACE: please, no spaces at the start of a line #57: FILE: drivers/gpu/drm/xe/xe_trace.h:606:
+ TP_ARGS(block, size, pfn),$
-:59: ERROR:CODE_INDENT: code indent should use tabs where possible #59: FILE: drivers/gpu/drm/xe/xe_trace.h:608:
+ TP_STRUCT__entry($
-:59: WARNING:LEADING_SPACE: please, no spaces at the start of a line #59: FILE: drivers/gpu/drm/xe/xe_trace.h:608:
+ TP_STRUCT__entry($
-:59: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #59: FILE: drivers/gpu/drm/xe/xe_trace.h:608:
+ TP_STRUCT__entry(
-:60: ERROR:CODE_INDENT: code indent should use tabs where possible #60: FILE: drivers/gpu/drm/xe/xe_trace.h:609:
+ __field(u64, block)$
-:60: WARNING:LEADING_SPACE: please, no spaces at the start of a line #60: FILE: drivers/gpu/drm/xe/xe_trace.h:609:
+ __field(u64, block)$
-:61: ERROR:CODE_INDENT: code indent should use tabs where possible #61: FILE: drivers/gpu/drm/xe/xe_trace.h:610:
+ __field(u64, header)$
-:61: WARNING:LEADING_SPACE: please, no spaces at the start of a line #61: FILE: drivers/gpu/drm/xe/xe_trace.h:610:
+ __field(u64, header)$
-:62: ERROR:CODE_INDENT: code indent should use tabs where possible #62: FILE: drivers/gpu/drm/xe/xe_trace.h:611:
+ __field(u64, size)$
-:62: WARNING:LEADING_SPACE: please, no spaces at the start of a line #62: FILE: drivers/gpu/drm/xe/xe_trace.h:611:
+ __field(u64, size)$
-:63: ERROR:CODE_INDENT: code indent should use tabs where possible #63: FILE: drivers/gpu/drm/xe/xe_trace.h:612:
+ __field(u64, pfn)$
-:63: WARNING:LEADING_SPACE: please, no spaces at the start of a line #63: FILE: drivers/gpu/drm/xe/xe_trace.h:612:
+ __field(u64, pfn)$
-:64: ERROR:CODE_INDENT: code indent should use tabs where possible #64: FILE: drivers/gpu/drm/xe/xe_trace.h:613:
+ ),$
-:64: WARNING:LEADING_SPACE: please, no spaces at the start of a line #64: FILE: drivers/gpu/drm/xe/xe_trace.h:613:
+ ),$
-:66: ERROR:CODE_INDENT: code indent should use tabs where possible #66: FILE: drivers/gpu/drm/xe/xe_trace.h:615:
+ TP_fast_assign($
-:66: WARNING:LEADING_SPACE: please, no spaces at the start of a line #66: FILE: drivers/gpu/drm/xe/xe_trace.h:615:
+ TP_fast_assign($
-:66: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #66: FILE: drivers/gpu/drm/xe/xe_trace.h:615:
+ TP_fast_assign(
-:67: ERROR:CODE_INDENT: code indent should use tabs where possible #67: FILE: drivers/gpu/drm/xe/xe_trace.h:616:
+ __entry->block = (u64)block;$
-:67: WARNING:LEADING_SPACE: please, no spaces at the start of a line #67: FILE: drivers/gpu/drm/xe/xe_trace.h:616:
+ __entry->block = (u64)block;$
-:68: ERROR:CODE_INDENT: code indent should use tabs where possible #68: FILE: drivers/gpu/drm/xe/xe_trace.h:617:
+ __entry->header = block->header;$
-:68: WARNING:LEADING_SPACE: please, no spaces at the start of a line #68: FILE: drivers/gpu/drm/xe/xe_trace.h:617:
+ __entry->header = block->header;$
-:69: ERROR:CODE_INDENT: code indent should use tabs where possible #69: FILE: drivers/gpu/drm/xe/xe_trace.h:618:
+ __entry->size = size;$
-:69: WARNING:LEADING_SPACE: please, no spaces at the start of a line #69: FILE: drivers/gpu/drm/xe/xe_trace.h:618:
+ __entry->size = size;$
-:70: ERROR:CODE_INDENT: code indent should use tabs where possible #70: FILE: drivers/gpu/drm/xe/xe_trace.h:619:
+ __entry->pfn = pfn;$
-:70: WARNING:LEADING_SPACE: please, no spaces at the start of a line #70: FILE: drivers/gpu/drm/xe/xe_trace.h:619:
+ __entry->pfn = pfn;$
-:71: ERROR:CODE_INDENT: code indent should use tabs where possible #71: FILE: drivers/gpu/drm/xe/xe_trace.h:620:
+ ),$
-:71: WARNING:LEADING_SPACE: please, no spaces at the start of a line #71: FILE: drivers/gpu/drm/xe/xe_trace.h:620:
+ ),$
-:73: ERROR:CODE_INDENT: code indent should use tabs where possible #73: FILE: drivers/gpu/drm/xe/xe_trace.h:622:
+ TP_printk("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n",$
-:73: WARNING:LEADING_SPACE: please, no spaces at the start of a line #73: FILE: drivers/gpu/drm/xe/xe_trace.h:622:
+ TP_printk("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n",$
-:74: ERROR:CODE_INDENT: code indent should use tabs where possible #74: FILE: drivers/gpu/drm/xe/xe_trace.h:623:
+ __entry->block, __entry->header, __entry->size, __entry->pfn)$
-:74: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #74: FILE: drivers/gpu/drm/xe/xe_trace.h:623:
+ TP_printk("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n",
+ __entry->block, __entry->header, __entry->size, __entry->pfn)
-:74: WARNING:LEADING_SPACE: please, no spaces at the start of a line #74: FILE: drivers/gpu/drm/xe/xe_trace.h:623:
+ __entry->block, __entry->header, __entry->size, __entry->pfn)$
-:77: CHECK:LINE_SPACING: Please don't use multiple blank lines #77: FILE: drivers/gpu/drm/xe/xe_trace.h:626:
+
+
-:79: ERROR:CODE_INDENT: code indent should use tabs where possible #79: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$
-:79: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #79: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+DEFINE_EVENT(xe_buddy_block, xe_buddy_block_alloc,
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),
-:79: WARNING:LEADING_SPACE: please, no spaces at the start of a line #79: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$
-:80: ERROR:CODE_INDENT: code indent should use tabs where possible #80: FILE: drivers/gpu/drm/xe/xe_trace.h:629:
+ TP_ARGS(block, size, pfn)$
-:80: WARNING:LEADING_SPACE: please, no spaces at the start of a line #80: FILE: drivers/gpu/drm/xe/xe_trace.h:629:
+ TP_ARGS(block, size, pfn)$
-:83: CHECK:LINE_SPACING: Please don't use multiple blank lines #83: FILE: drivers/gpu/drm/xe/xe_trace.h:632:
+
+
-:85: ERROR:CODE_INDENT: code indent should use tabs where possible #85: FILE: drivers/gpu/drm/xe/xe_trace.h:634:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$
-:85: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #85: FILE: drivers/gpu/drm/xe/xe_trace.h:634:
+DEFINE_EVENT(xe_buddy_block, xe_buddy_block_free,
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),
-:85: WARNING:LEADING_SPACE: please, no spaces at the start of a line #85: FILE: drivers/gpu/drm/xe/xe_trace.h:634:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$
-:86: ERROR:CODE_INDENT: code indent should use tabs where possible #86: FILE: drivers/gpu/drm/xe/xe_trace.h:635:
+ TP_ARGS(block, size, pfn)$
-:86: WARNING:LEADING_SPACE: please, no spaces at the start of a line #86: FILE: drivers/gpu/drm/xe/xe_trace.h:635:
+ TP_ARGS(block, size, pfn)$
total: 20 errors, 21 warnings, 8 checks, 65 lines checked 92094bcea drm/xe/svm: Handle CPU page fault
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:44: WARNING:LINE_SPACING: Missing a blank line after declarations #44: FILE: drivers/gpu/drm/xe/xe_device_types.h:585:
+ u64 offset = (pfn << PAGE_SHIFT) - mr->hpa_base;
+ dpa = mr->dpa_base + offset;
-:74: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #74: FILE: drivers/gpu/drm/xe/xe_svm.h:78:
+bool xe_svm_range_belongs_to_vma(struct mm_struct *mm,
+ struct xe_svm_range *range,
-:120: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #120:
new file mode 100644
-:147: CHECK:LINE_SPACING: Please don't use multiple blank lines #147: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:23:
+
+
-:169: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #169: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:45:
+static struct page *alloc_host_page(struct device *dev,
+ struct vm_area_struct *vma,
-:220: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #220: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:96:
+static int migrate_page_vram_to_ram(struct vm_area_struct *vma, unsigned long addr,
+ unsigned long src_pfn, unsigned long *dst_pfn)
-:247: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #247: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:123:
+ fence = xe_migrate_svm(tile->migrate, src_dpa, true,
+ dma_addr, false, PAGE_SIZE);
-:317: CHECK:SPACING: spaces preferred around that '*' (ctx:VxW) #317: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:193:
+ buf = kvcalloc(npages, 2* sizeof(*migrate_vma.src), GFP_KERNEL);
^
-:330: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #330: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:206:
+ ret = migrate_page_vram_to_ram(vma, addr, migrate_vma.src[i],
+ migrate_vma.dst + i);
-:386: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #386: FILE: drivers/gpu/drm/xe/xe_svm_range.c:48:
+bool xe_svm_range_belongs_to_vma(struct mm_struct *mm,
+ struct xe_svm_range *range,
total: 0 errors, 2 warnings, 8 checks, 342 lines checked 9520bed78 drm/xe/svm: trace svm range migration -:50: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #50: FILE: drivers/gpu/drm/xe/xe_trace.h:609:
+ TP_STRUCT__entry(
-:55: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #55: FILE: drivers/gpu/drm/xe/xe_trace.h:614:
+ TP_fast_assign(
-:65: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #65: FILE: drivers/gpu/drm/xe/xe_trace.h:624:
+DEFINE_EVENT(xe_svm_migrate, xe_svm_migrate_vram_to_sram,
+ TP_PROTO(struct xe_svm_range *range),
-:69: CHECK:LINE_SPACING: Please don't use multiple blank lines #69: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+
+
-:71: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #71: FILE: drivers/gpu/drm/xe/xe_trace.h:630:
+DEFINE_EVENT(xe_svm_migrate, xe_svm_migrate_sram_to_vram,
+ TP_PROTO(struct xe_svm_range *range),
total: 0 errors, 0 warnings, 5 checks, 49 lines checked93a9ddcb2 drm/xe/svm: Implement functions to register and unregister mmu notifier -:46: WARNING:TYPO_SPELLING: 'splitted' may be misspelled - perhaps 'split'? #46: FILE: drivers/gpu/drm/xe/xe_svm.h:71:
+ * which means one vma can be splitted into multiple
^^^^^^^^
-:143: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #143: FILE: drivers/gpu/drm/xe/xe_svm_range.c:124:
+ ret = mmu_interval_notifier_insert_locked(&range->notifier, vma->vm_mm,
+ start, length, &xe_svm_mni_ops);
total: 0 errors, 1 warnings, 1 checks, 118 lines checked541192d56 drm/xe/svm: Implement the mmu notifier range invalidate callback
-:24: WARNING:COMMIT_LOG_LONG_LINE: Prefer a maximum 75 chars per line (possible unwrapped commit description?) #24:
unregister mmu notifier (calling mmu_interval_notifier_remove) also requires
-:78: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #78: FILE: drivers/gpu/drm/xe/xe_svm_range.c:64:
+static bool xe_svm_range_invalidate(struct mmu_interval_notifier *mni,
+ const struct mmu_notifier_range *range,
-:90: WARNING:REPEATED_WORD: Possible repeated word: 'in' #90: FILE: drivers/gpu/drm/xe/xe_svm_range.c:76:
+ * page table and unregister mmu notifier in in xe_destroy_svm,
total: 0 errors, 2 warnings, 1 checks, 64 lines checked 9ac75e0da drm/xe/svm: clean up svm range during process exit -:37: CHECK:SPACING: spaces preferred around that '*' (ctx:VxV) #37: FILE: drivers/gpu/drm/xe/xe_svm.c:24:
+#define MAX_SVM_RANGE (1024*1024)
^
-:44: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #44: FILE: drivers/gpu/drm/xe/xe_svm.c:31:
+ range_array = kzalloc(sizeof(struct xe_svm_range *) * MAX_SVM_RANGE,
+ GFP_KERNEL);
-:55: ERROR:SPACING: space required before the open parenthesis '(' #55: FILE: drivers/gpu/drm/xe/xe_svm.c:42:
+ for(; i >= 0; i--)
total: 1 errors, 0 warnings, 2 checks, 66 lines checked 41c558827 drm/xe/svm: Move a few structures to xe_gt.h 054dcebaf drm/xe/svm: migrate svm range to vram
-:57: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #57: FILE: drivers/gpu/drm/xe/xe_svm.h:84:
+int svm_migrate_range_to_vram(struct xe_svm_range *range,
+ struct vm_area_struct *vma,
-:71: CHECK:LINE_SPACING: Please don't use multiple blank lines #71: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:233:
+
+
-:83: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #83: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:245:
+int svm_migrate_range_to_vram(struct xe_svm_range *range,
+ struct vm_area_struct *vma,
-:110: CHECK:SPACING: spaces preferred around that '*' (ctx:VxW) #110: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:272:
+ buf = kvcalloc(npages, 2* sizeof(*migrate.src) + sizeof(*src_dma_addr),
^
-:111: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #111: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:273:
+ buf = kvcalloc(npages, 2* sizeof(*migrate.src) + sizeof(*src_dma_addr),
+ GFP_KERNEL);
-:112: ERROR:SPACING: space required before the open parenthesis '(' #112: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:274:
+ if(!buf)
-:116: CHECK:SPACING: No space is necessary after a cast #116: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:278:
+ src_dma_addr = (dma_addr_t *) (migrate.dst + npages);
-:124: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #124: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:286:
+ drm_err(&tile->xe->drm, "vma setup returned %d for range [%lx - %lx]\n",
+ ret, start, end);
-:136: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #136: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:298:
+ drm_warn(&tile->xe->drm, "Partial migration for range [%lx - %lx], range is %ld pages, migrate only %ld pages\n",
+ start, end, npages, migrate.cpages);
-:150: WARNING:LONG_LINE: line length of 101 exceeds 100 columns #150: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:312:
+ drm_warn(&tile->xe->drm, "dma map error for host pfn %lx\n", migrate.src[i]);
-:155: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #155: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:317:
+ fence = xe_migrate_svm(tile->migrate, src_dma_addr[i], false,
+ dst_dpa, true, PAGE_SIZE);
-:158: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #158: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:320:
+ drm_warn(&tile->xe->drm, "migrate host page (pfn: %lx) to vram failed\n",
+ migrate.src[i]);
total: 1 errors, 1 warnings, 10 checks, 145 lines checked ace9fe594 drm/xe/svm: Populate svm range
-:49: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #49: FILE: drivers/gpu/drm/xe/xe_svm.c:166:
+static int svm_populate_range(struct xe_svm_range *svm_range,
+ struct hmm_range *hmm_range, bool write)
total: 0 errors, 0 warnings, 1 checks, 64 lines checked 4d4bd0bf2 drm/xe/svm: GPU page fault support
-:101: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #101: FILE: drivers/gpu/drm/xe/xe_svm.c:236:
+static bool svm_should_migrate(struct xe_svm_range *range,
+ struct xe_mem_region *dst_region, bool is_atomic_fault)
-:119: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #119: FILE: drivers/gpu/drm/xe/xe_svm.c:254:
+int xe_svm_handle_gpu_fault(struct xe_vm *vm,
+ struct xe_gt *gt,
-:143: WARNING:SPACING: space prohibited between function name and open parenthesis '(' #143: FILE: drivers/gpu/drm/xe/xe_svm.c:278:
+ if (!svm_access_allowed (vma, access_type != ACCESS_TYPE_READ)) {
-:158: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #158: FILE: drivers/gpu/drm/xe/xe_svm.c:293:
+ if (svm_should_migrate(range, >->tile->mem.vram,
+ access_type == ACCESS_TYPE_ATOMIC))
-:178: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #178: FILE: drivers/gpu/drm/xe/xe_svm.c:313:
+ ret = xe_bind_svm_range(vm, gt->tile, &hmm_range,
+ !(vma->vm_flags & VM_WRITE) ? DRM_XE_VM_BIND_FLAG_READONLY : 0);
-:207: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #207: FILE: drivers/gpu/drm/xe/xe_svm.h:99:
+struct xe_svm_range *xe_svm_range_create(struct xe_svm *svm,
+ struct vm_area_struct *vma);
-:216: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #216: FILE: drivers/gpu/drm/xe/xe_svm.h:113:
+int xe_svm_handle_gpu_fault(struct xe_vm *vm,
+ struct xe_gt *gt,
-:250: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #250: FILE: drivers/gpu/drm/xe/xe_svm_range.c:208:
+struct xe_svm_range *xe_svm_range_create(struct xe_svm *svm,
+ struct vm_area_struct *vma)
-:262: ERROR:SPACING: space required before the open brace '{' #262: FILE: drivers/gpu/drm/xe/xe_svm_range.c:220:
+ if (xe_svm_range_register_mmu_notifier(range)){
total: 1 errors, 1 warnings, 7 checks, 218 lines checkeda45edccde drm/xe/svm: Add DRM_XE_SVM kernel config entryfca35d34b drm/xe/svm: Add svm memory hints interface
-:7: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one
-:32: WARNING:LONG_LINE: line length of 111 exceeds 100 columns #32: FILE: include/uapi/drm/xe_drm.h:118:
+#define DRM_IOCTL_XE_SVM DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_SVM, struct drm_xe_svm_args)
total: 0 errors, 2 warnings, 0 checks, 64 lines checked
Patchwork
CI✗ CI.KUnit: failure for XeKmd basic SVM support
In reply to Oak Zeng== Series Details ==
== Series Details ==
Series: XeKmd basic SVM support
URL : https://patchwork.freedesktop.org/series/128910/
State : failure
== Summary ==
+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig ERROR:root:../drivers/gpu/drm/xe/xe_mmio.c: In function ‘mmio_fini’:../drivers/gpu/drm/xe/xe_mmio.c:361:5: warning: unused variable ‘id’ [-Wunused-variable]
361 | u8 id;
| ^~../drivers/gpu/drm/xe/xe_mmio.c:360:18: warning: unused variable ‘tile’ [-Wunused-variable]
360 | struct xe_tile *tile;
| ^~~~../drivers/gpu/drm/xe/xe_mmio.c: At top level:../drivers/gpu/drm/xe/xe_mmio.c:372:1: error: expected identifier or ‘(’ before ‘}’ token
372 | }
| ^
make[7]: *** [../scripts/Makefile.build:243: drivers/gpu/drm/xe/xe_mmio.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:480: drivers/gpu/drm/xe] Error 2 make[6]: *** Waiting for unfinished jobs....
make[5]: *** [../scripts/Makefile.build:480: drivers/gpu/drm] Error 2 make[4]: *** [../scripts/Makefile.build:480: drivers/gpu] Error 2 make[3]: *** [../scripts/Makefile.build:480: drivers] Error 2 make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/kernel/Makefile:1911: .] Error 2
make[1]: *** [/kernel/Makefile:234: __sub-make] Error 2
make: *** [Makefile:234: __sub-make] Error 2
[02:46:31] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [02:46:35] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make ARCH=um O=.kunit --jobs=48 + cleanup ++ stat -c %u:%g /kernel + chown -R 1003:1003 /kernel