Skip to contentSkip to conversation

XeKmd basic SVM support

PATCH In review Oak Zeng 13 participants 126 messages Jan 17, 2024 – May 7, 2024 Page 2 of 2

Subsystems (4)
lore
Share & tools
lore
author
#103 lore raw

RE: Making drm_gpuvm work across gpu devices

In reply to Zeng, Oak

Hello all,

Since I didn't get a reply for this one, I assume below are agreed. But feel free to let us know if you don't agree.

Thanks,
Oak

-----Original Message-----
From: dri-devel <[email protected]> On Behalf Of Zeng, Oak
Sent: Thursday, February 29, 2024 1:23 PM
To: Christian König <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: Making drm_gpuvm work across gpu devices

Hi Christian/Daniel/Dave/Felix/Thomas, and all,

We have been refining our design internally in the past month. Below is our plan. Please let us know if you have any concern.

1) Remove pseudo /dev/xe-svm device. All system allocator interfaces will be through /dev/dri/render devices. Not global interface.

2) Unify userptr and system allocator codes. We will treat userptr as a speciality of system allocator without migration capability. We will introduce the hmmptr concept for system allocator. We will extend vm_bind API to map a range A..B of process address space to a range C..D of GPU address space for hmmptr. For hmmptr, if gpu program accesses an address which is not backed by core mm vma, it is a fatal error.

3) Multiple device support. We have identified p2p use-cases where we might want to leave memory on a foreign device or direct migrations to a foreign device and therefore might need a global structure that tracks or caches the migration state per process address space. We didn't completely settle down this design. We will come back when we have more details.

4)We will first work on this code on xekmd then look to move some common codes to drm layer so it can also be used by other vendors.

Thomas and me still have open questions to Christian. We will follow up.

Thanks all for this discussion.

Regards,
Oak

Quoted 189 lines

-----Original Message-----
From: Christian König <[email protected]>
Sent: Thursday, February 1, 2024 3:52 AM
To: Zeng, Oak <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Making drm_gpuvm work across gpu devices

Hi Oak,

Am 31.01.24 um 21:17 schrieb Zeng, Oak:

Quoted 3 lines

Hi Sima, Dave,

I am well aware nouveau driver is not what Nvidia do with their customer. The

key argument is, can we move forward with the concept shared virtual address space b/t CPU and GPU? This is the foundation of HMM. We already have split address space support with other driver API. SVM, from its name, it means shared address space. Are we allowed to implement another driver model to allow SVM work, along with other APIs supporting split address space? Those two scheme can co-exist in harmony. We actually have real use cases to use both models in one application.

Quoted 4 lines

Hi Christian, Thomas,

In your scheme, GPU VA can != GPU VA. This does introduce some flexibility.

But this scheme alone doesn't solve the problem of the proxy process/para- virtualization. You will still need a second mechanism to partition GPU VA space b/t guest process1 and guest process2 because proxy process (or the host hypervisor whatever you call it) use one single gpu page table for all the guest/client processes. GPU VA for different guest process can't overlap. If this second mechanism exist, we of course can use the same mechanism to partition CPU VA space between guest processes as well, then we can still use shared VA b/t CPU and GPU inside one process, but process1 and process2's address space (for both cpu and gpu) doesn't overlap. This second mechanism is the key to solve the proxy process problem, not the flexibility you introduced.

That approach was suggested before, but it doesn't work. First of all you create a massive security hole when you give the GPU full access to the QEMU CPU process which runs the virtualization.

So even if you say CPU VA == GPU VA you still need some kind of flexibility otherwise you can't implement this use case securely.

Additional to this the CPU VAs are usually controlled by the OS and not some driver, so to make sure that host and guest VAs don't overlap you would need to add some kind of sync between the guest and host OS kernels.

Quoted 1 line

In practice, your scheme also have a risk of running out of process space

because you have to partition whole address space b/t processes. Apparently allowing each guest process to own the whole process space and using separate GPU/CPU page table for different processes is a better solution than using single page table and partition process space b/t processes.

Yeah that you run out of address space is certainly possible. But as I said CPUs are switching to 5 level of pages tables and if you look at for example a "cat maps | cut -c-4 | sort -u" of process you will find that only a handful of 4GiB segments are actually used and thanks to recoverable page faults you can map those between host and client on demand. This gives you at least enough address space to handle a couple of thousand clients.

Quoted 1 line

For Intel GPU, para-virtualization (xenGT, see https://github.com/intel/XenGT-

Preview-kernel. It is similar idea of the proxy process in Flex's email. They are all SW-based GPU virtualization technology) is an old project. It is now replaced with HW accelerated SRIOV/system virtualization. XenGT is abandoned long time ago. So agreed your scheme add some flexibility. The question is, do we have a valid use case to use such flexibility? I don't see a single one ATM.

Yeah, we have SRIOV functionality on AMD hw as well, but for some use cases it's just to inflexible.

Quoted 1 line

I also pictured into how to implement your scheme. You basically rejected the

very foundation of hmm design which is shared address space b/t CPU and GPU. In your scheme, GPU VA = CPU VA + offset. In every single place where driver need to call hmm facilities such as hmm_range_fault, migrate_vma_setup and in mmu notifier call back, you need to offset the GPU VA to get a CPU VA. From application writer's perspective, whenever he want to use a CPU pointer in his GPU program, he add to add that offset. Do you think this is awkward?

What? This flexibility is there to prevent the application writer to change any offset.

Quoted 1 line

Finally, to implement SVM, we need to implement some memory hint API

which applies to a virtual address range across all GPU devices. For example, user would say, for this virtual address range, I prefer the backing store memory to be on GPU deviceX (because user knows deviceX would use this address range much more than other GPU devices or CPU). It doesn't make sense to me to make such API per device based. For example, if you tell device A that the preferred memory location is device B memory, this doesn't sounds correct to me because in your scheme, device A is not even aware of the existence of device B. right?

Correct and while the additional flexibility is somewhat option I strongly think that not having a centralized approach for device driver settings is mandatory.

Going away from the well defined file descriptor based handling of device driver interfaces was one of the worst ideas I've ever seen in roughly thirty years of working with Unixiode operating systems. It basically broke everything, from reverse lockup handling for mmap() to file system privileges for hardware access.

As far as I can see anything which goes into the direction of opening /dev/kfd or /dev/xe_svm or something similar and saying that this then results into implicit SVM for your render nodes is an absolutely no-go and would required and explicit acknowledgement from Linus on the design to do something like that.

What you can do is to have an IOCTL for the render node file descriptor which says this device should do SVM with the current CPU address space and another IOCTL which says range A..B is preferred to migrate to this device for HMM when the device runs into a page fault.

And yes that obviously means shitty performance for device drivers because page play ping/pong if userspace gives contradicting information for migrations, but that is something supposed to be.

Everything else which works over the boarders of a device driver scope should be implemented as system call with the relevant review process around it.

Regards,
Christian.

Quoted 15 lines

Regards,
Oak

Quoted 12 lines

-----Original Message-----
From: Daniel Vetter <[email protected]>
Sent: Wednesday, January 31, 2024 4:15 AM
To: David Airlie <[email protected]>
Cc: Zeng, Oak <[email protected]>; Christian König <[email protected]>; Thomas Hellström <[email protected]>; Daniel Vetter <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; dri- [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg

<[email protected]>;

Quoted 2 lines
Quoted 2 lines

Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah,

Ankur N

Quoted 4 lines
Quoted 4 lines

<[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Making drm_gpuvm work across gpu devices

Quoted 8 lines
Quoted 8 lines · David Airlie
Quoted 6 lines · Oak
Quoted 5 lines

Hi Christian,

Nvidia Nouveau driver uses exactly the same concept of SVM with HMM,

GPU address in the same process is exactly the same with CPU virtual address.

It

Quoted 25 lines
Quoted 25 lines

is already in upstream Linux kernel. We Intel just follow the same direction for our customers. Why we are not allowed?

Quoted 9 lines

Oak, this isn't how upstream works, you don't get to appeal to customer or internal design. nouveau isn't "NVIDIA"'s and it certainly isn't something NVIDIA would ever suggest for their customers. We also likely wouldn't just accept NVIDIA's current solution upstream without some serious discussions. The implementation in nouveau was more of a sample HMM use case rather than a serious implementation. I suspect if we do get down the road of making nouveau an actual compute driver for SVM etc then it would have to severely change.

Yeah on the nouveau hmm code specifically my gut feeling impression is that we didn't really make friends with that among core kernel maintainers. It's a bit too much just a tech demo to be able to merge the hmm core apis for nvidia's out-of-tree driver.

Also, a few years of learning and experience gaining happened meanwhile - you always have to look at an api design in the context of when it was designed, and that context changes all the time.

Cheers, Sima
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

#104 lore raw

Re: Making drm_gpuvm work across gpu devices

Hi Oak,

sorry the mail sounded like you didn't expected a reply.

And yes, the approaches outlined in the mail sounds really good to me.

Regards,
Christian.

Am 08.03.24 um 05:43 schrieb Zeng, Oak:

Quoted 219 lines

Hello all,

Since I didn't get a reply for this one, I assume below are agreed. But feel free to let us know if you don't agree.

Thanks,
Oak

-----Original Message-----
From: dri-devel <[email protected]> On Behalf Of Zeng, Oak
Sent: Thursday, February 29, 2024 1:23 PM
To: Christian König <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: Making drm_gpuvm work across gpu devices

Hi Christian/Daniel/Dave/Felix/Thomas, and all,

We have been refining our design internally in the past month. Below is our plan. Please let us know if you have any concern.

1) Remove pseudo /dev/xe-svm device. All system allocator interfaces will be through /dev/dri/render devices. Not global interface.

2) Unify userptr and system allocator codes. We will treat userptr as a speciality of system allocator without migration capability. We will introduce the hmmptr concept for system allocator. We will extend vm_bind API to map a range A..B of process address space to a range C..D of GPU address space for hmmptr. For hmmptr, if gpu program accesses an address which is not backed by core mm vma, it is a fatal error.

3) Multiple device support. We have identified p2p use-cases where we might want to leave memory on a foreign device or direct migrations to a foreign device and therefore might need a global structure that tracks or caches the migration state per process address space. We didn't completely settle down this design. We will come back when we have more details.

4)We will first work on this code on xekmd then look to move some common codes to drm layer so it can also be used by other vendors.

Thomas and me still have open questions to Christian. We will follow up.

Thanks all for this discussion.

Regards,
Oak

Quoted 186 lines

-----Original Message-----
From: Christian König <[email protected]>
Sent: Thursday, February 1, 2024 3:52 AM
To: Zeng, Oak <[email protected]>; Daniel Vetter <[email protected]>; David Airlie <[email protected]>
Cc: Thomas Hellström <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah, Ankur N <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Making drm_gpuvm work across gpu devices

Hi Oak,

Am 31.01.24 um 21:17 schrieb Zeng, Oak:

Quoted 3 lines

Hi Sima, Dave,

I am well aware nouveau driver is not what Nvidia do with their customer. The

key argument is, can we move forward with the concept shared virtual address space b/t CPU and GPU? This is the foundation of HMM. We already have split address space support with other driver API. SVM, from its name, it means shared address space. Are we allowed to implement another driver model to allow SVM work, along with other APIs supporting split address space? Those two scheme can co-exist in harmony. We actually have real use cases to use both models in one application.

Quoted 3 lines

Hi Christian, Thomas,

In your scheme, GPU VA can != GPU VA. This does introduce some flexibility.

But this scheme alone doesn't solve the problem of the proxy process/para- virtualization. You will still need a second mechanism to partition GPU VA space b/t guest process1 and guest process2 because proxy process (or the host hypervisor whatever you call it) use one single gpu page table for all the guest/client processes. GPU VA for different guest process can't overlap. If this second mechanism exist, we of course can use the same mechanism to partition CPU VA space between guest processes as well, then we can still use shared VA b/t CPU and GPU inside one process, but process1 and process2's address space (for both cpu and gpu) doesn't overlap. This second mechanism is the key to solve the proxy process problem, not the flexibility you introduced.

That approach was suggested before, but it doesn't work. First of all you create a massive security hole when you give the GPU full access to the QEMU CPU process which runs the virtualization.

So even if you say CPU VA == GPU VA you still need some kind of flexibility otherwise you can't implement this use case securely.

Additional to this the CPU VAs are usually controlled by the OS and not some driver, so to make sure that host and guest VAs don't overlap you would need to add some kind of sync between the guest and host OS kernels.

Quoted 1 line

In practice, your scheme also have a risk of running out of process space

because you have to partition whole address space b/t processes. Apparently allowing each guest process to own the whole process space and using separate GPU/CPU page table for different processes is a better solution than using single page table and partition process space b/t processes.

Yeah that you run out of address space is certainly possible. But as I said CPUs are switching to 5 level of pages tables and if you look at for example a "cat maps | cut -c-4 | sort -u" of process you will find that only a handful of 4GiB segments are actually used and thanks to recoverable page faults you can map those between host and client on demand. This gives you at least enough address space to handle a couple of thousand clients.

Quoted 1 line

For Intel GPU, para-virtualization (xenGT, see https://github.com/intel/XenGT-

Preview-kernel. It is similar idea of the proxy process in Flex's email. They are all SW-based GPU virtualization technology) is an old project. It is now replaced with HW accelerated SRIOV/system virtualization. XenGT is abandoned long time ago. So agreed your scheme add some flexibility. The question is, do we have a valid use case to use such flexibility? I don't see a single one ATM.

Yeah, we have SRIOV functionality on AMD hw as well, but for some use cases it's just to inflexible.

Quoted 1 line

I also pictured into how to implement your scheme. You basically rejected the

very foundation of hmm design which is shared address space b/t CPU and GPU. In your scheme, GPU VA = CPU VA + offset. In every single place where driver need to call hmm facilities such as hmm_range_fault, migrate_vma_setup and in mmu notifier call back, you need to offset the GPU VA to get a CPU VA. From application writer's perspective, whenever he want to use a CPU pointer in his GPU program, he add to add that offset. Do you think this is awkward?

What? This flexibility is there to prevent the application writer to change any offset.

Quoted 1 line

Finally, to implement SVM, we need to implement some memory hint API

which applies to a virtual address range across all GPU devices. For example, user would say, for this virtual address range, I prefer the backing store memory to be on GPU deviceX (because user knows deviceX would use this address range much more than other GPU devices or CPU). It doesn't make sense to me to make such API per device based. For example, if you tell device A that the preferred memory location is device B memory, this doesn't sounds correct to me because in your scheme, device A is not even aware of the existence of device B. right?

Correct and while the additional flexibility is somewhat option I strongly think that not having a centralized approach for device driver settings is mandatory.

Going away from the well defined file descriptor based handling of device driver interfaces was one of the worst ideas I've ever seen in roughly thirty years of working with Unixiode operating systems. It basically broke everything, from reverse lockup handling for mmap() to file system privileges for hardware access.

As far as I can see anything which goes into the direction of opening /dev/kfd or /dev/xe_svm or something similar and saying that this then results into implicit SVM for your render nodes is an absolutely no-go and would required and explicit acknowledgement from Linus on the design to do something like that.

What you can do is to have an IOCTL for the render node file descriptor which says this device should do SVM with the current CPU address space and another IOCTL which says range A..B is preferred to migrate to this device for HMM when the device runs into a page fault.

And yes that obviously means shitty performance for device drivers because page play ping/pong if userspace gives contradicting information for migrations, but that is something supposed to be.

Everything else which works over the boarders of a device driver scope should be implemented as system call with the relevant review process around it.

Regards,
Christian.

Quoted 14 lines

Regards,
Oak

Quoted 12 lines

-----Original Message-----
From: Daniel Vetter <[email protected]>
Sent: Wednesday, January 31, 2024 4:15 AM
To: David Airlie <[email protected]>
Cc: Zeng, Oak <[email protected]>; Christian König <[email protected]>; Thomas Hellström <[email protected]>; Daniel Vetter <[email protected]>; Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; dri- [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg

<[email protected]>;

Quoted 2 lines
Quoted 2 lines

Vishwanathapura, Niranjana <[email protected]>; intel- [email protected]; Danilo Krummrich <[email protected]>; Shah,

Ankur N

Quoted 4 lines
Quoted 4 lines

<[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: Making drm_gpuvm work across gpu devices

Quoted 8 lines
Quoted 8 lines · David Airlie
Quoted 6 lines · Oak
Quoted 5 lines

Hi Christian,

Nvidia Nouveau driver uses exactly the same concept of SVM with HMM,

GPU address in the same process is exactly the same with CPU virtual address.

It

Quoted 24 lines
Quoted 24 lines

is already in upstream Linux kernel. We Intel just follow the same direction for our customers. Why we are not allowed?

Quoted 8 lines

Oak, this isn't how upstream works, you don't get to appeal to customer or internal design. nouveau isn't "NVIDIA"'s and it certainly isn't something NVIDIA would ever suggest for their customers. We also likely wouldn't just accept NVIDIA's current solution upstream without some serious discussions. The implementation in nouveau was more of a sample HMM use case rather than a serious implementation. I suspect if we do get down the road of making nouveau an actual compute driver for SVM etc then it would have to severely change.

Yeah on the nouveau hmm code specifically my gut feeling impression is that we didn't really make friends with that among core kernel maintainers. It's a bit too much just a tech demo to be able to merge the hmm core apis for nvidia's out-of-tree driver.

Also, a few years of learning and experience gaining happened meanwhile - you always have to look at an api design in the context of when it was designed, and that context changes all the time.

Cheers, Sima
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

#105 lore raw

Re: Making drm_gpuvm work across gpu devices

In reply to Zeng, Oak

Hi, Oak,

Quoted 15 lines · Oak

The example you used to prove that KFD is a design failure, is against *any* design which utilize system allocator and hmm. The way that one proxy process running on host to handle many guest processes, doesn’t fit into the concept of “share address space b/t cpu and gpu”. The shared address space has to be within one process. Your proxy process represent many guest processes. It is a fundamental conflict.

Also your userptr proposal does’t solve this problem either:

Imagine you have guest process1 mapping CPU address range A…B to GPU address range C…D

And you have guest process 2 mapping CPU address range A…B to GPU address range C…D, since process 1 and 2 are two different process, it is legal for process 2 to do the exact same mapping.

Now when gpu shader access address C…D, a gpu page fault happens, what does your proxy process do? Which guest process will this fault be directed to and handled? Except you have extra information/API to tell proxy process and GPU HW, there is no way to figure out.

Compared to the shared virtual address space concept of HMM, the userptr design is nothing new except it allows CPU and GPU to use different address to access the same object. If you replace above C…D with A…B, above description becomes a description of the “problem” of HMM/shared virtual address design.

Both design has the same difficulty with your example of the special virtualization environment setup.

Quoted 9 lines

As said, we spent effort scoped the userptr solution some time ago. The problem we found enabling userptr with migration were:

1. The user interface of userptr is not as convenient as system allocator. With the userptr solution, user need to call userptr_ioctl and vm_bind for *every* single cpu pointer that he want to use in a gpu program. While with system allocator, programmer just use any cpu pointer directly in gpu program without any extra driver ioctls.

No, the augmented userptr (lets call it "hmmptr" to distinguish here) would typically only be bound once when the VM is created. It's just a different way to expose the whole SVM mapping to user-space. It's sparsely populated and is not backed by a bo, and it is per-device so UMD would have to replicate the SVM setup and attribute settings on each device.

Quoted 11 lines

1.

2. We don’t see the real benefit of using a different Gpu address C…D than the A..B, except you can prove my above reasoning is wrong. In most use cases, you can make GPU C…D == CPU A…B, why bother then?
3. Looked into implementation details, since hmm fundamentally assume a shared virtual address space b/t cpu and device, for the userptr solution to leverage hmm, you need perform address space conversion every time you calls into hmm functions.

I think very much focus lands on the A..B -> C..D mapping in the discussion. It's just an added flexibility with little or no implementation cost. Although I must admit I'm not fully clear about the actual use-case. In a para-virtualized environment like virGL or vmware's vmx/renderers I could imagine C..D being the guest virtual addresses including compute kernel pointers, A..B being the host renderer's CPU virtual addresses. (Host creates the VM's, and then this translation is needed. I'm not sure para-virtualized SVM exists ATM, but forcing A==C, B==D in the uAPI would rule out such a beast in the future?)

/Thomas

Quoted 429 lines

1.

In summary, GPU device is just a piece of HW to accelerate your CPU program. If HW allows, it is more convenient to use shared address space b/t cpu and GPU. On old HW (example, no gpu page fault support, or gpu only has a very limited address space), we can disable system allocator/SVM. If you use different address space on modern GPU, why don’t you use different address space on different CPU cores?

Regards,

Oak

*From:*dri-devel <[email protected]> *On Behalf Of *Christian König
*Sent:* Monday, January 29, 2024 5:20 AM
*To:* Zeng, Oak <[email protected]>; Thomas Hellström <[email protected]>; Daniel Vetter <[email protected]>; Dave Airlie <[email protected]>
*Cc:* Brost, Matthew <[email protected]>; Felix Kuehling <[email protected]>; Welty, Brian <[email protected]>; [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; Gupta, saurabhg <[email protected]>; Vishwanathapura, Niranjana <[email protected]>; [email protected]; Danilo Krummrich <[email protected]> *Subject:* Re: Making drm_gpuvm work across gpu devices

Well Daniel and Dave noted it as well, so I'm just repeating it: Your design choices are not an argument to get something upstream.

It's the job of the maintainers and at the end of the Linus to judge of something is acceptable or not.

As far as I can see a good part of this this idea has been exercised lengthy with KFD and it turned out to not be the best approach.

So from what I've seen the design you outlined is extremely unlikely to go upstream.

Regards,
Christian.

Am 27.01.24 um 03:21 schrieb Zeng, Oak:

Regarding the idea of expanding userptr to support migration, we explored this idea long time ago. It provides similar functions of the system allocator but its interface is not as convenient as system allocator. Besides the shared virtual address space, another benefit of a system allocator is, you can offload cpu program to gpu easier, you don’t need to call driver specific API (such as register_userptr and vm_bind in this case) for memory allocation.

We also scoped the implementation. It turned out to be big, and not as beautiful as hmm. Why we gave up this approach.

*From:*Christian König <[email protected]> <mailto:[email protected]>
*Sent:* Friday, January 26, 2024 7:52 AM
*To:* Thomas Hellström <[email protected]> <mailto:[email protected]>; Daniel Vetter <[email protected]> <mailto:[email protected]>
*Cc:* Brost, Matthew <[email protected]> <mailto:[email protected]>; Felix Kuehling <[email protected]> <mailto:[email protected]>; Welty, Brian <[email protected]> <mailto:[email protected]>; Ghimiray, Himal Prasad <[email protected]> <mailto:[email protected]>; Zeng, Oak <[email protected]> <mailto:[email protected]>; Gupta, saurabhg <[email protected]> <mailto:[email protected]>; Danilo Krummrich <[email protected]> <mailto:[email protected]>; [email protected] <mailto:[email protected]>; Bommu, Krishnaiah <[email protected]> <mailto:[email protected]>; Dave Airlie <[email protected]> <mailto:[email protected]>; Vishwanathapura, Niranjana <[email protected]> <mailto:[email protected]>; [email protected] <mailto:[email protected]> *Subject:* Re: Making drm_gpuvm work across gpu devices

Am 26.01.24 um 09:21 schrieb Thomas Hellström:

Hi, all

On Thu, 2024-01-25 at 19:32 +0100, Daniel Vetter wrote:

On Wed, Jan 24, 2024 at 09:33:12AM +0100, Christian König wrote:

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

[SNIP]

Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at

the svm_ioctl function, it is per-process based. Each kfd_process

represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not

do that

ever again.

Need to say, kfd SVM represent a shared virtual address space

across CPU and all GPU devices on the system. This is by the

definition of SVM (shared virtual memory). This is very different

from our legacy gpu *device* driver which works for only one

device (i.e., if you want one device to access another device's

memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker

for a

virtualization projects. Having SVM as device independent feature

which

somehow ties to the process address space turned out to be an

extremely bad

idea.

The background is that this only works for some use cases but not

all of

them.

What's working much better is to just have a mirror functionality

which says

that a range A..B of the process address space is mapped into a

range C..D

of the GPU address space.

Those ranges can then be used to implement the SVM feature required

for

higher level APIs and not something you need at the UAPI or even

inside the

low level kernel memory management.

When you talk about migrating memory to a device you also do this

on a per

device basis and *not* tied to the process address space. If you

then get

crappy performance because userspace gave contradicting information

where to

migrate memory then that's a bug in userspace and not something the

kernel

should try to prevent somehow.

[SNIP]

I think if you start using the same drm_gpuvm for multiple

devices you

will sooner or later start to run into the same mess we have

seen with

KFD, where we moved more and more functionality from the KFD to

the DRM

render node because we found that a lot of the stuff simply

doesn't work

correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A

single pseudo /dev/kfd device represent all hardware gpu devices.

That is why during kfd open, many pdd (process device data) is

created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that

I see

this design as a rather extreme failure. And I think it's one of

the reasons

why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of

extending the

CPU process into the GPUs, but this idea only works for a few use

cases and

is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up

with CPU

address != GPU address because the VAs are actually coming from the

guest VM

and not the host process.

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should

not have

any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need

to get

explicit permission to do this from Dave and Daniel and maybe even

Linus.

I think the one and only one exception where an SVM uapi like in kfd

makes

sense, is if the _hardware_ itself, not the software stack defined

semantics that you've happened to build on top of that hw, enforces a

1:1

mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation,

PCI-ATS

(address translation services) or whatever your hw calls it and has

_no_

device-side pagetables on top. Which from what I've seen all devices

with

device-memory have, simply because they need some place to store

whether

that memory is currently in device memory or should be translated

using

PASID. Currently there's no gpu that works with PASID only, but there

are

some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully

cpu

cache coherent (including atomics) with something like CXL, and the

on-device memory is managed as normal system memory with struct page

as

ZONE_DEVICE and accelerator va -> physical address translation is

only

done with PASID ... but for now I haven't seen that, definitely not

in

upstream drivers.

And the moment you have some per-device pagetables or per-device

memory

management of some sort (like using gpuva mgr) then I'm 100% agreeing

with

Christian that the kfd SVM model is too strict and not a great idea.

Cheers, Sima

I'm trying to digest all the comments here, The end goal is to be able

to support something similar to this here:

https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/

Christian, If I understand you correctly, you're strongly suggesting

not to try to manage a common virtual address space across different

devices in the kernel, but merely providing building blocks to do so,

like for example a generalized userptr with migration support using

HMM; That way each "mirror" of the CPU mm would be per device and

inserted into the gpu_vm just like any other gpu_vma, and user-space

would dictate the A..B -> C..D mapping by choosing the GPU_VA for the

vma.

Exactly that, yes.

Sima, it sounds like you're suggesting to shy away from hmm and not

even attempt to support this except if it can be done using IOMMU sva

on selected hardware?

I think that comment goes more into the direction of: If you have ATS/ATC/PRI capable hardware which exposes the functionality to make memory reads and writes directly into the address space of the CPU then yes an SVM only interface is ok because the hardware can't do anything else. But as long as you have something like GPUVM then please don't restrict yourself.

Which I totally agree on as well. The ATS/ATC/PRI combination doesn't allow using separate page tables device and CPU and so also not separate VAs.

This was one of the reasons why we stopped using this approach for AMD GPUs.

Regards,
Christian.

Could you clarify a bit?

Thanks,

Thomas

#106 lore raw

Re: Making drm_gpuvm work across gpu devices

In reply to Daniel Vetter
Quoted 76 lines · Daniel Vetter
Quoted 50 lines · Christian König

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

#107 lore raw

Re: Making drm_gpuvm work across gpu devices

Am 29.01.24 um 16:03 schrieb Felix Kuehling:

Quoted 77 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 7 lines
Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.

The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.

As far as I can see there is absolutely no need for any special SVM handling.

Regards,
Christian.

Quoted 7 lines

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

#108 lore raw

Re: Making drm_gpuvm work across gpu devices

Quoted 84 lines · Christian König

Am 29.01.24 um 16:03 schrieb Felix Kuehling:

Quoted 83 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Quoted 7 lines
Quoted 1 line

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.

The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.

As far as I can see there is absolutely no need for any special SVM handling.

It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.

Regards,
  Felix

Quoted 12 lines

Regards,
Christian.

Quoted 7 lines

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

#109 lore raw

Re: Making drm_gpuvm work across gpu devices

Am 29.01.24 um 17:24 schrieb Felix Kuehling:

Quoted 87 lines · Christian König
Quoted 86 lines

Am 29.01.24 um 16:03 schrieb Felix Kuehling:

Quoted 84 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Quoted 7 lines
Quoted 5 lines

Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.

The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.

As far as I can see there is absolutely no need for any special SVM handling.

It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.

Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.

There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.

So this isn't a blocker, it's just one special use case.

Regards,
Christian.

Quoted 17 lines

Regards,
  Felix

Quoted 12 lines

Regards,
Christian.

Quoted 7 lines

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

#110 lore raw

Re: Making drm_gpuvm work across gpu devices

Quoted 90 lines · Christian König

Am 29.01.24 um 17:24 schrieb Felix Kuehling:

Quoted 89 lines · Christian König
Quoted 88 lines

Am 29.01.24 um 16:03 schrieb Felix Kuehling:

Quoted 77 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 7 lines
Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.

As far as I can see there is absolutely no need for any special SVM handling.

It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.

Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.

What I want is that address range A..B on the CPU matches A..B on the GPU, because I'm sharing pointers between CPU and GPU. I can't think of any sane user mode using a unified memory programming model, that would ever ask KFD to map unified memory mappints to a different address range on the GPU. Adding such an ioclt is a complete waste of time, and can only serve to add unnecessary complexity.

Regards,
  Felix

Quoted 26 lines

There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.

So this isn't a blocker, it's just one special use case.

Regards,
Christian.

Quoted 17 lines

Regards,
  Felix

Quoted 12 lines

Regards,
Christian.

Quoted 7 lines

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

#111 lore raw

Re: Making drm_gpuvm work across gpu devices

Am 29.01.24 um 18:52 schrieb Felix Kuehling:

Quoted 93 lines · Christian König
Quoted 92 lines

Am 29.01.24 um 17:24 schrieb Felix Kuehling:

Quoted 91 lines · Christian König
Quoted 90 lines

Am 29.01.24 um 16:03 schrieb Felix Kuehling:

Quoted 77 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 7 lines
Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.

The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

As far as I can see there is absolutely no need for any special SVM handling.

It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.

Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.

What I want is that address range A..B on the CPU matches A..B on the GPU, because I'm sharing pointers between CPU and GPU. I can't think of any sane user mode using a unified memory programming model, that would ever ask KFD to map unified memory mappints to a different address range on the GPU. Adding such an ioclt is a complete waste of time, and can only serve to add unnecessary complexity.

This is exactly the use case which happens when the submitting process is not the one originally stitching together the command stream.

Basically all native context, virtualization and other proxy use cases work like this.

So that the CPU address doesn't match the GPU address is an absolutely real use case and should be able to be handled by the GPU VA interface.

Regards,
Christian.

Quoted 31 lines

Regards,
  Felix

Quoted 26 lines

There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.

So this isn't a blocker, it's just one special use case.

Regards,
Christian.

Quoted 17 lines

Regards,
  Felix

Quoted 12 lines

Regards,
Christian.

Quoted 7 lines

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

#112 lore raw

Re: Making drm_gpuvm work across gpu devices

Quoted 98 lines · Christian König

Am 29.01.24 um 18:52 schrieb Felix Kuehling:

Quoted 97 lines · Christian König
Quoted 96 lines

Am 29.01.24 um 17:24 schrieb Felix Kuehling:

Quoted 94 lines · Christian König
Quoted 84 lines

Am 29.01.24 um 16:03 schrieb Felix Kuehling:

Quoted 83 lines · Daniel Vetter
Quoted 76 lines · Christian König
Quoted 50 lines

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

Quoted 4 lines

[SNIP]
Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Quoted 1 line

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

Quoted 6 lines
Quoted 5 lines

I think if you start using the same drm_gpuvm for multiple devices you will sooner or later start to run into the same mess we have seen with KFD, where we moved more and more functionality from the KFD to the DRM render node because we found that a lot of the stuff simply doesn't work correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

Quoted 5 lines

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

I think the one and only one exception where an SVM uapi like in kfd makes sense, is if the _hardware_ itself, not the software stack defined semantics that you've happened to build on top of that hw, enforces a 1:1 mapping with the cpu process address space.

Which means your hardware is using PASID, IOMMU based translation, PCI-ATS (address translation services) or whatever your hw calls it and has _no_ device-side pagetables on top. Which from what I've seen all devices with device-memory have, simply because they need some place to store whether that memory is currently in device memory or should be translated using PASID. Currently there's no gpu that works with PASID only, but there are some on-cpu-die accelerator things that do work like that.

Maybe in the future there will be some accelerators that are fully cpu cache coherent (including atomics) with something like CXL, and the on-device memory is managed as normal system memory with struct page as ZONE_DEVICE and accelerator va -> physical address translation is only done with PASID ... but for now I haven't seen that, definitely not in upstream drivers.

Quoted 3 lines

And the moment you have some per-device pagetables or per-device memory management of some sort (like using gpuva mgr) then I'm 100% agreeing with Christian that the kfd SVM model is too strict and not a great idea.

That basically means, without ATS/PRI+PASID you cannot implement a unified memory programming model, where GPUs or accelerators access virtual addresses without pre-registering them with an SVM API call.

Quoted 7 lines
Quoted 1 line

Unified memory is a feature implemented by the KFD SVM API and used by ROCm. This is used e.g. to implement OpenMP USM (unified shared memory). It's implemented with recoverable GPU page faults. If the page fault interrupt handler cannot assume a shared virtual address space, then implementing this feature isn't possible.

Why not? As far as I can see the OpenMP USM is just another funky way of userptr handling.

The difference is that in an userptr we assume that we always need to request the whole block A..B from a mapping while for page fault based handling it can be just any page in between A and B which is requested and made available to the GPU address space.

As far as I can see there is absolutely no need for any special SVM handling.

It does assume a shared virtual address space between CPU and GPUs. There are no API calls to tell the driver that address A on the CPU maps to address B on the GPU1 and address C on GPU2. The KFD SVM API was designed to work with this programming model, by augmenting the shared virtual address mappings with virtual address range attributes that can modify the migration policy and indicate prefetching, prefaulting, etc. You could think of it as madvise on steroids.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Yeah, so what? In this case you just say through an IOCTL that CPU range A..B should map to GPU range C..D and for A/B and C/D you use the maximum of the address space.

What I want is that address range A..B on the CPU matches A..B on the GPU, because I'm sharing pointers between CPU and GPU. I can't think of any sane user mode using a unified memory programming model, that would ever ask KFD to map unified memory mappints to a different address range on the GPU. Adding such an ioclt is a complete waste of time, and can only serve to add unnecessary complexity.

This is exactly the use case which happens when the submitting process is not the one originally stitching together the command stream.

Basically all native context, virtualization and other proxy use cases work like this.

You cannot use unified memory in this type of environment. A GPU page fault would occur in a GPU virtual address space in the host-side proxy process. That page fault would need to be resolved to map some memory from a process running in a guest? Which guest? Which process? That's anyone's guess. There is no way to annotate that because the pointer in the fault is coming from a shader program that's running in the guest context and competely unaware of the virtualization. There are no API calls from the guest before the fault occurs to establish a meaningful mapping.

The way this virtualization of the GPU is implemented, with out proxy multiplexing multiple clients is just fundamentally incompatible with a unified memory programming model that has to assume a shared virtual address space to make sense. You'd need separate proxy processes on the host side to handle that. You can't blame this on bad design decisions in the SVM API. As I see it, it's just a fundamental limitation of the virtualization approach that cannot handle guest applications that want to use a unified shared memory programming model.

That's why I suggested to completely disable the SVM API in this virtualization scenario when you create a KFD context that's separate from the process address space. It is not essential for any non-unified-memory functionality. ROCm user mode has fallbacks to work without it, because we also needs to support older kernels and GPUs that didn't support this programming model.

Regards,
  Felix

Quoted 40 lines

So that the CPU address doesn't match the GPU address is an absolutely real use case and should be able to be handled by the GPU VA interface.

Regards,
Christian.

Quoted 31 lines

Regards,
  Felix

Quoted 26 lines

There is no restriction that this needs to be accurate in way. It's just the it can be accurate to be more efficient and eventually use only a fraction of the address space instead of all of it for some use cases.

So this isn't a blocker, it's just one special use case.

Regards,
Christian.

Quoted 17 lines

Regards,
  Felix

Quoted 12 lines

Regards,
Christian.

Quoted 7 lines

Regards,
  Felix

Quoted 2 lines

Cheers, Sima

author
#113 lore raw

RE: Making drm_gpuvm work across gpu devices

In reply to Christian König

Hi Christian,

I go back this old email to ask a question.

Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

There are two category of SVM:

1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

It looks like you were talking of 1). Were you?

Oak
From: Christian König <[email protected]>
Sent: Wednesday, January 24, 2024 3:33 AM
To: Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>
Cc: Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
Subject: Re: Making drm_gpuvm work across gpu devices

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

[SNIP]

Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

I think if you start using the same drm_gpuvm for multiple devices you

will sooner or later start to run into the same mess we have seen with

KFD, where we moved more and more functionality from the KFD to the DRM

render node because we found that a lot of the stuff simply doesn't work

correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

Regards,
Christian.

#114 lore raw

Re: Making drm_gpuvm work across gpu devices

Hi Oak,

Am 23.02.24 um 21:12 schrieb Zeng, Oak:

Quoted 5 lines

Hi Christian,

I go back this old email to ask a question.

sorry totally missed that one.

Quoted 27 lines

Quote from your email:

“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.”

“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

There are two category of SVM:

1. driver svm allocator: this is implemented in user space,  i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator:  This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

Quoted 2 lines

It looks like you were talking of 1). Were you?

No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.

I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.

Regards,
Christian.

Quoted 61 lines

Oak

*From:*Christian König <[email protected]>
*Sent:* Wednesday, January 24, 2024 3:33 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]> *Subject:* Re: Making drm_gpuvm work across gpu devices

Am 23.01.24 um 20:37 schrieb Zeng, Oak:

[SNIP]

Yes most API are per device based.

One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.

Yeah and that was a big mistake in my opinion. We should really not do that ever again.

Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).

Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.

The background is that this only works for some use cases but not all of them.

What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.

Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.

When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug in userspace and not something the kernel should try to prevent somehow.

[SNIP]

I think if you start using the same drm_gpuvm for multiple devices you

will sooner or later start to run into the same mess we have seen with

KFD, where we moved more and more functionality from the KFD to the DRM

render node because we found that a lot of the stuff simply doesn't work

correctly with a single object to maintain the state.

As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.

Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.

This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.

A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.

SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.

If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.

Regards,
Christian.

author
#115 lore raw

RE: Making drm_gpuvm work across gpu devices

From: Christian König <[email protected]>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
Cc: Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
Subject: Re: Making drm_gpuvm work across gpu devices

Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,

I go back this old email to ask a question.

sorry totally missed that one.

Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

There are two category of SVM:

1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.
2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

It looks like you were talking of 1). Were you?

No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.

Lets say we have a hardware platform where both CPU and GPU support 57bit virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space (0~2^57-1), do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program.

I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.

Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we don’t map the whole vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.

The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma.

Regards,
Oak

Regards,
Christian.

author
#116 lore raw

RE: Making drm_gpuvm work across gpu devices

The mail wasn’t indent/preface correctly. Manually format it.

From: Christian König <[email protected]<mailto:[email protected]>>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <[email protected]<mailto:[email protected]>>; Danilo Krummrich <[email protected]<mailto:[email protected]>>; Dave Airlie <[email protected]<mailto:[email protected]>>; Daniel Vetter <[email protected]<mailto:[email protected]>>; Felix Kuehling <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>
Cc: Welty, Brian <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; Bommu, Krishnaiah <[email protected]<mailto:[email protected]>>; Ghimiray, Himal Prasad <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; Vishwanathapura, Niranjana <[email protected]<mailto:[email protected]>>; Brost, Matthew <[email protected]<mailto:[email protected]>>; Gupta, saurabhg <[email protected]<mailto:[email protected]>>
Subject: Re: Making drm_gpuvm work across gpu devices

Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,

I go back this old email to ask a question.

sorry totally missed that one.

Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

There are two category of SVM:

1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.

2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

It looks like you were talking of 1). Were you?

No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.

[Zeng, Oak]
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?

I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.

[Zeng, Oak]
Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.

The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?

Regards,
Oak

Regards,
Christian.

#117 lore raw

Re: Making drm_gpuvm work across gpu devices

Am 28.02.24 um 20:51 schrieb Zeng, Oak:

Quoted 54 lines

The mail wasn’t indent/preface correctly. Manually format it.

*From:*Christian König <[email protected]>
*Sent:* Tuesday, February 27, 2024 1:54 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]> *Subject:* Re: Making drm_gpuvm work across gpu devices

Hi Oak,

Am 23.02.24 um 21:12 schrieb Zeng, Oak:

Hi Christian,

I go back this old email to ask a question.

sorry totally missed that one.

Quote from your email:

“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.”

“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

There are two category of SVM:

1.driver svm allocator: this is implemented in user space,  i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.

2.system svm allocator:  This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

It looks like you were talking of 1). Were you?

Quoted 8 lines

No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.

*/[Zeng, Oak] /*

Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program.  If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?

Well you can perfectly mirror on demand. You just need something similar to userfaultfd() for the GPU. This way you don't need to mirror the full address space, but can rather work with large chunks created on demand, let's say 1GiB or something like that.

The virtual address space is basically just a hardware functionality to route memory accesses. While the mirroring approach is a very common use case for data-centers and high performance computing there are quite a number of different use cases which makes use of virtual address space in a non "standard" fashion. The native context approach for VMs is just one example, databases and emulators are another one.

Quoted 4 lines

I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.

Quoted 8 lines

*/[Zeng, Oak] /*

Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.

*//*

The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?

Yes, exactly that. I mean the splitting and eventually merging of ranges is a standard functionality of the GPUVM code.

So when you need to store additional attributes per range then I would strongly suggest to make use of this splitting and merging functionality as well.

So basically an IOCTL which says range A..B of the GPU address space is mapped to offset X of the CPU address space with parameters Y (caching, migration behavior etc..). That is essentially the same we have for mapping GEM objects, the provider of the backing store is just something different.

Regards,
Christian.

Quoted 9 lines

Regards,

Oak

Regards,
Christian.

author
#118 lore raw

RE: Making drm_gpuvm work across gpu devices

Hi Christian,

Can you elaborate the mirror on demand/userfaultfd idea?

userfaultfd is a way for user space to take over page fault handling of a user registered range. From first look, it seems you want a user space page fault handler to mirror a large chunk of memory to GPU. I would imagine this handler is in UMD, because the whole purpose of system svm allocator is to allow user use cpu address (such as malloc’ed) on gpu program without extra driver api call. So the registration and mirroring of this large chunk can’t be in user program. With this, I pictured below sequence:

During process initialization time, umd register a large chunk (lets say 1GiB) of memory using userfaultfd, this include:

  1.  mem = mmap(NULL, 1GiB, MAP_ANON)
  2.  register range [mem, mem + 1GiB] through userfaultfd
  3.  after that, umd can wait on page fault event. When page fault happens, umd call vm_bind to mirror [mem, mem+1GiB] range to GPU

now in a user program:
ptr = malloc(size);
submit a GPU program which uses ptr

This is what I can picture. It doesn’t work because ptr can’t belong to [mem, mem+1GiB] range. So you can’t vm_bind/mirror ptr on demand to GPU.

Also, the page fault event in 3) above can’t happen at all. A page fault only happens when *CPU* access mem but in our case, it could be *only GPU* touch the memory.

The point is, with system svm allocator, user can use *any* valid CPU address for GPU program. This address can be anything in the range of [0~2^57-1]. This design requirement is quite simple and clean. I don’t see how to solve this with userfaultfd/on demand mirroring.

Regards,
Oak

From: Christian König <[email protected]>
Sent: Thursday, February 29, 2024 4:41 AM
To: Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
Cc: Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad <[email protected]>; [email protected]; Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
Subject: Re: Making drm_gpuvm work across gpu devices

Am 28.02.24 um 20:51 schrieb Zeng, Oak:

The mail wasn’t indent/preface correctly. Manually format it.

From: Christian König <[email protected]<mailto:[email protected]>>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <[email protected]<mailto:[email protected]>>; Danilo Krummrich <[email protected]<mailto:[email protected]>>; Dave Airlie <[email protected]<mailto:[email protected]>>; Daniel Vetter <[email protected]<mailto:[email protected]>>; Felix Kuehling <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>
Cc: Welty, Brian <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; Bommu, Krishnaiah <[email protected]<mailto:[email protected]>>; Ghimiray, Himal Prasad <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; Vishwanathapura, Niranjana <[email protected]<mailto:[email protected]>>; Brost, Matthew <[email protected]<mailto:[email protected]>>; Gupta, saurabhg <[email protected]<mailto:[email protected]>>
Subject: Re: Making drm_gpuvm work across gpu devices

Hi Oak,
Am 23.02.24 um 21:12 schrieb Zeng, Oak:
Hi Christian,

I go back this old email to ask a question.

sorry totally missed that one.

Quote from your email:
“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.” “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

There are two category of SVM:

1. driver svm allocator: this is implemented in user space, i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.

2. system svm allocator: This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

It looks like you were talking of 1). Were you?

No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.

[Zeng, Oak]
Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program. If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?

Well you can perfectly mirror on demand. You just need something similar to userfaultfd() for the GPU. This way you don't need to mirror the full address space, but can rather work with large chunks created on demand, let's say 1GiB or something like that.

The virtual address space is basically just a hardware functionality to route memory accesses. While the mirroring approach is a very common use case for data-centers and high performance computing there are quite a number of different use cases which makes use of virtual address space in a non "standard" fashion. The native context approach for VMs is just one example, databases and emulators are another one.

I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.

[Zeng, Oak]
Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.

The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?

Yes, exactly that. I mean the splitting and eventually merging of ranges is a standard functionality of the GPUVM code.

So when you need to store additional attributes per range then I would strongly suggest to make use of this splitting and merging functionality as well.

So basically an IOCTL which says range A..B of the GPU address space is mapped to offset X of the CPU address space with parameters Y (caching, migration behavior etc..). That is essentially the same we have for mapping GEM objects, the provider of the backing store is just something different.

Regards,
Christian.

Regards,
Oak

Regards,
Christian.

#119 lore raw

Re: Making drm_gpuvm work across gpu devices

In reply to Christian König

Hi, Christian.

Quoted 105 lines · Christian König

Am 28.02.24 um 20:51 schrieb Zeng, Oak:

Quoted 88 lines

The mail wasn’t indent/preface correctly. Manually format it.

*From:*Christian König <[email protected]> *Sent:* Tuesday, February 27, 2024 1:54 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad
<[email protected]>; [email protected];
Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
*Subject:* Re: Making drm_gpuvm work across gpu devices

Hi Oak,

Am 23.02.24 um 21:12 schrieb Zeng, Oak:

    Hi Christian,

    I go back this old email to ask a question.

sorry totally missed that one.

    Quote from your email:

    “Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the
    UAPI or even inside the low level kernel memory management.”

    “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

    There are two category of SVM:

    1.driver svm allocator: this is implemented in user space, i.g.,
    cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd
    and our umd implemented clSVMAlloc and zeMemAllocShared on top of
    gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you
    said.

    2.system svm allocator:  This doesn’t introduce extra driver API
    for memory allocation. Any valid CPU virtual address can be used
    directly transparently in a GPU program without any extra driver
    API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently”
    and “to share the address space by duplicating the CPU page table
    in the device page table so the same address points to the same physical memory for any valid main memory address in the process
    address space”. In system svm allocator, we don’t need that A..B
    C..D mapping.

    It looks like you were talking of 1). Were you?

No, even when you fully mirror the whole address space from a process
into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of
the address space this mirroring applies and where it maps to.

*/[Zeng, Oak] /*

Quoted 15 lines

Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at
that time, you don’t know which part of the address space will be used
for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program.  If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a
valid concern?

Quoted 7 lines

Well you can perfectly mirror on demand. You just need something similar
to userfaultfd() for the GPU. This way you don't need to mirror the full
address space, but can rather work with large chunks created on demand,
let's say 1GiB or something like that.

What we're looking at as the current design is an augmented userptr (A..B -> C..D mapping) which is internally sparsely populated in chunks. KMD manages the population using gpu pagefaults. We acknowledge that some parts of this mirror will not have a valid CPU mapping. That is, no vma so a gpu page-fault that resolves to such a mirror address will cause an error. Would you have any concerns / objections against such an approach?

Thanks,
Thomas

#120 lore raw

Re: Making drm_gpuvm work across gpu devices

Hi Thomas,

Am 29.02.24 um 18:12 schrieb Thomas Hellström:

Quoted 114 lines

Hi, Christian.

Quoted 103 lines · Christian König

Am 28.02.24 um 20:51 schrieb Zeng, Oak:

Quoted 87 lines

The mail wasn’t indent/preface correctly. Manually format it.

*From:*Christian König <[email protected]> *Sent:* Tuesday, February 27, 2024 1:54 AM
*To:* Zeng, Oak <[email protected]>; Danilo Krummrich <[email protected]>; Dave Airlie <[email protected]>; Daniel Vetter <[email protected]>; Felix Kuehling <[email protected]>; [email protected]
*Cc:* Welty, Brian <[email protected]>; [email protected]; [email protected]; Bommu, Krishnaiah <[email protected]>; Ghimiray, Himal Prasad
<[email protected]>; [email protected];
Vishwanathapura, Niranjana <[email protected]>; Brost, Matthew <[email protected]>; Gupta, saurabhg <[email protected]>
*Subject:* Re: Making drm_gpuvm work across gpu devices

Hi Oak,

Am 23.02.24 um 21:12 schrieb Zeng, Oak:

    Hi Christian,

    I go back this old email to ask a question.

sorry totally missed that one.

    Quote from your email:

    “Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at
the
    UAPI or even inside the low level kernel memory management.”

    “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel
UAPI.”

    There are two category of SVM:

    1.driver svm allocator: this is implemented in user space, i.g.,
    cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in
xekmd
    and our umd implemented clSVMAlloc and zeMemAllocShared on top
of
    gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as
you
    said.

    2.system svm allocator:  This doesn’t introduce extra driver
API
    for memory allocation. Any valid CPU virtual address can be
used
    directly transparently in a GPU program without any extra
driver
    API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device
transparently”
    and “to share the address space by duplicating the CPU page
table
    in the device page table so the same address points to the same physical memory for any valid main memory address in the
process
    address space”. In system svm allocator, we don’t need that
A..B
    C..D mapping.

    It looks like you were talking of 1). Were you?

No, even when you fully mirror the whole address space from a process
into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of
the address space this mirroring applies and where it maps to.

*/[Zeng, Oak] /*

Quoted 15 lines

Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at
that time, you don’t know which part of the address space will be used
for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program.  If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a
valid concern?

Quoted 7 lines

Well you can perfectly mirror on demand. You just need something similar
to userfaultfd() for the GPU. This way you don't need to mirror the full
address space, but can rather work with large chunks created on demand,
let's say 1GiB or something like that.

Quoted 7 lines

What we're looking at as the current design is an augmented userptr (A..B -> C..D mapping) which is internally sparsely populated in chunks. KMD manages the population using gpu pagefaults. We acknowledge that some parts of this mirror will not have a valid CPU mapping. That is, no vma so a gpu page-fault that resolves to such a mirror address will cause an error. Would you have any concerns / objections against such an approach?

Nope, as far as I can see that sounds like a perfectly valid design to me.

Regards,
Christian.

Quoted 6 lines

Thanks,
Thomas

author
#121 lore raw

PATCH 22/23 drm/xe/svm: Add DRM_XE_SVM kernel config entry

In reply to Oak Zeng

DRM_XE_SVM kernel config entry is added so
xe svm feature can be configured before kernel
compilation.

---
 drivers/gpu/drm/xe/Kconfig   | 22 ++++++++++++++++++++++
 drivers/gpu/drm/xe/Makefile  |  5 +++++
 drivers/gpu/drm/xe/xe_mmio.c |  5 +++++
 drivers/gpu/drm/xe/xe_vm.c   |  2 ++
 4 files changed, 34 insertions(+)
drivers/gpu/drm/xe/Kconfig +22 -0
index 1b57ae38210d..6f498095a915 100644
@@ -83,6 +83,28 @@ config DRM_XE_FORCE_PROBE
 
 	  Use "!*" to block the probe of the driver for all known devices.
 
+config DRM_XE_SVM
+    bool "Enable Shared Virtual Memory support in xe"
+    depends on DRM_XE
+    depends on ARCH_ENABLE_MEMORY_HOTPLUG
+    depends on ARCH_ENABLE_MEMORY_HOTREMOVE
+    depends on MEMORY_HOTPLUG
+    depends on MEMORY_HOTREMOVE
+    depends on ARCH_HAS_PTE_DEVMAP
+    depends on SPARSEMEM_VMEMMAP
+    depends on ZONE_DEVICE
+    depends on DEVICE_PRIVATE
+    depends on MMU
+    select HMM_MIRROR
+    select MMU_NOTIFIER
+    default y
+    help
+      Choose this option if you want Shared Virtual Memory (SVM)
+      support in xe. With SVM, virtual address space is shared
+	  between CPU and GPU. This means any virtual address such
+	  as malloc or mmap returns, variables on stack, or global
+	  memory pointers, can be used for GPU transparently.
+
 menu "drm/Xe Debugging"
 depends on DRM_XE
 depends on EXPERT
drivers/gpu/drm/xe/Makefile +5 -0
index e16b84f79ddf..ae503f7c1f94 100644
@@ -283,6 +283,11 @@ xe-$(CONFIG_DRM_XE_DISPLAY) += \
 	i915-display/skl_universal_plane.o \
 	i915-display/skl_watermark.o
 
+xe-$(CONFIG_DRM_XE_SVM) += xe_svm.o \
+						   xe_svm_devmem.o \
+						   xe_svm_range.o \
+						   xe_svm_migrate.o
+
 ifeq ($(CONFIG_ACPI),y)
 	xe-$(CONFIG_DRM_XE_DISPLAY) += \
 		i915-display/intel_acpi.o \
drivers/gpu/drm/xe/xe_mmio.c +5 -0
index 3d34dcfa3b3a..99810794bd94 100644
@@ -286,7 +286,9 @@ int xe_mmio_probe_vram(struct xe_device *xe)
 		}
 
 		io_size -= min_t(u64, tile_size, io_size);
+#if IS_ENABLED(CONFIG_DRM_XE_SVM)
 		xe_svm_devm_add(tile, &tile->mem.vram);
+#endif
 	}
 
 	xe->mem.vram.actual_physical_size = total_size;
@@ -361,8 +363,11 @@ static void mmio_fini(struct drm_device *drm, void *arg)
 	pci_iounmap(to_pci_dev(xe->drm.dev), xe->mmio.regs);
 	if (xe->mem.vram.mapping)
 		iounmap(xe->mem.vram.mapping);
+
+#if IS_ENABLED(CONFIG_DRM_XE_SVM)
 	for_each_tile(tile, xe, id) {
 		xe_svm_devm_remove(xe, &tile->mem.vram);
+#endif
 	}
 }
 
drivers/gpu/drm/xe/xe_vm.c +2 -0
index 712fe49d8fb2..3bf19c92e01f 100644
@@ -1377,7 +1377,9 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 		xe->usm.num_vm_in_non_fault_mode++;
 	mutex_unlock(&xe->usm.lock);
 
+#if IS_ENABLED(CONFIG_DRM_XE_SVM)
 	vm->svm = xe_create_svm(vm);
+#endif
 	trace_xe_vm_create(vm);
 
 	return vm;
Signature · 1 line
2.26.3
author
#122 lore raw

PATCH 23/23 drm/xe/svm: Add svm memory hints interface

In reply to Oak Zeng
---
 include/uapi/drm/xe_drm.h | 40 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
include/uapi/drm/xe_drm.h +40 -0
index 50bbea0992d9..551ed8706097 100644
@@ -80,6 +80,7 @@ extern "C" {
  *  - &DRM_IOCTL_XE_EXEC_QUEUE_GET_PROPERTY
  *  - &DRM_IOCTL_XE_EXEC
  *  - &DRM_IOCTL_XE_WAIT_USER_FENCE
+ *  - &DRM_IOCTL_XE_SVM
  */
 
 /*
@@ -100,6 +101,7 @@ extern "C" {
 #define DRM_XE_EXEC_QUEUE_GET_PROPERTY	0x08
 #define DRM_XE_EXEC			0x09
 #define DRM_XE_WAIT_USER_FENCE		0x0a
+#define DRM_XE_SVM			0x0b
 /* Must be kept compact -- no holes */
 
 #define DRM_IOCTL_XE_DEVICE_QUERY		DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_DEVICE_QUERY, struct drm_xe_device_query)
@@ -113,6 +115,7 @@ extern "C" {
 #define DRM_IOCTL_XE_EXEC_QUEUE_GET_PROPERTY	DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_EXEC_QUEUE_GET_PROPERTY, struct drm_xe_exec_queue_get_property)
 #define DRM_IOCTL_XE_EXEC			DRM_IOW(DRM_COMMAND_BASE + DRM_XE_EXEC, struct drm_xe_exec)
 #define DRM_IOCTL_XE_WAIT_USER_FENCE		DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_WAIT_USER_FENCE, struct drm_xe_wait_user_fence)
+#define DRM_IOCTL_XE_SVM			DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_SVM, struct drm_xe_svm_args)
 
 /**
  * DOC: Xe IOCTL Extensions
@@ -1339,6 +1342,43 @@ struct drm_xe_wait_user_fence {
 	__u64 reserved[2];
 };
 
+enum drm_xe_svm_attr_type {
+	DRM_XE_SVM_ATTR_PREFERRED_LOC,
+	DRM_XE_SVM_ATTR_MIGRATION_GRANULARITY,
+	DRM_XE_SVM_ATTR_ATOMIC,
+	DRM_XE_SVM_ATTR_CACHE,
+	DRM_XE_SVM_ATTR_PREFETCH_LOC,
+	DRM_XE_SVM_ATTR_ACCESS_PATTERN,
+};
+
+struct drm_xe_svm_attr {
+	__u32 type;
+	__u32 value;
+};
+
+enum drm_xe_svm_op {
+	DRM_XE_SVM_OP_SET_ATTR,
+	DRM_XE_SVM_OP_GET_ATTR,
+};
+
+/**
+ * struct drm_xe_svm_args - Input of &DRM_IOCTL_XE_SVM
+ *
+ * Set or get memory attributes to a virtual address range
+ */
+struct drm_xe_svm_args {
+	/** @start: start of the virtual address range */
+	__u64 start;
+	/** @size: size of the virtual address range */
+	__u64 size;
+	/** @op: operation, either set or get */
+	__u32 op;
+	/** @nattr: number of attributes */
+	__u32 nattr;
+	/** @attrs: An array of attributes */
+	struct drm_xe_svm_attr attrs[];
+};
+
 #if defined(__cplusplus)
 }
 #endif
Signature · 1 line
2.26.3
#123 lore raw

✓ CI.Patch_applied: success for XeKmd basic SVM support

In reply to Oak Zeng
== Series Details ==

== Series Details ==

Series: XeKmd basic SVM support
URL : https://patchwork.freedesktop.org/series/128910/
State : success

== Summary ==

=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: 2d422dec5 drm-tip: 2024y-01m-17d-23h-44m-17s UTC integration manifest === git am output follows ===
.git/rebase-apply/patch:167: new blank line at EOF.
+
warning: 1 line adds whitespace errors.
Applying: drm/xe/svm: Add SVM document
Applying: drm/xe/svm: Add svm key data structures
Applying: drm/xe/svm: create xe svm during vm creation
Applying: drm/xe/svm: Trace svm creation
Applying: drm/xe/svm: add helper to retrieve svm range from address
Applying: drm/xe/svm: Introduce a helper to build sg table from hmm range
Applying: drm/xe/svm: Add helper for binding hmm range to gpu
Applying: drm/xe/svm: Add helper to invalidate svm range from GPU
Applying: drm/xe/svm: Remap and provide memmap backing for GPU vram
Applying: drm/xe/svm: Introduce svm migration function
Applying: drm/xe/svm: implement functions to allocate and free device memory
Applying: drm/xe/svm: Trace buddy block allocation and free
Applying: drm/xe/svm: Handle CPU page fault
Applying: drm/xe/svm: trace svm range migration
Applying: drm/xe/svm: Implement functions to register and unregister mmu notifier
Applying: drm/xe/svm: Implement the mmu notifier range invalidate callback
Applying: drm/xe/svm: clean up svm range during process exit
Applying: drm/xe/svm: Move a few structures to xe_gt.h
Applying: drm/xe/svm: migrate svm range to vram
Applying: drm/xe/svm: Populate svm range
Applying: drm/xe/svm: GPU page fault support
Applying: drm/xe/svm: Add DRM_XE_SVM kernel config entry
Applying: drm/xe/svm: Add svm memory hints interface

#124 lore raw

✗ CI.checkpatch: warning for XeKmd basic SVM support

In reply to Oak Zeng
== Series Details ==

== Series Details ==

Series: XeKmd basic SVM support
URL : https://patchwork.freedesktop.org/series/128910/
State : warning

== Summary ==

+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master e7f730565356938dd6c63cd498218fdc5d6a22b0
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit fca35d34b0e317dbe0033be49616255ba22c100d

Author: Oak Zeng <[email protected]>
Date:   Wed Jan 17 17:12:23 2024 -0500

drm/xe/svm: Add svm memory hints interface

+ /mt/dim checkpatch 2d422dec58e0dfd61d6c9842de7f83289803dccb drm-intel
96a876fc2 drm/xe/svm: Add SVM document
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:28: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #28:
new file mode 100644

-:97: WARNING:TYPO_SPELLING: 'delibrately' may be misspelled - perhaps 'deliberately'? #97: FILE: drivers/gpu/drm/xe/xe_svm_doc.h:51:
+ * in our SVM implementation. We delibrately choose this implementation option

                                  ^^^^^^^^^^^

-:149: WARNING:TYPO_SPELLING: 'implemenation' may be misspelled - perhaps 'implementation'? #149: FILE: drivers/gpu/drm/xe/xe_svm_doc.h:103:
+ * implemenation and API" described the locking scheme that driver writer has to

    ^^^^^^^^^^^^^

-:163: WARNING:REPEATED_WORD: Possible repeated word: 'during' #163: FILE: drivers/gpu/drm/xe/xe_svm_doc.h:117:
+ * during during device page table update, if yes, then retry.

total: 0 errors, 4 warnings, 0 checks, 133 lines checked
2743cb454 drm/xe/svm: Add svm key data structures
-:35: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #35:
new file mode 100644

-:40: WARNING:SPDX_LICENSE_TAG: Improper SPDX comment style for 'drivers/gpu/drm/xe/xe_svm.h', please use '/*' instead #40: FILE: drivers/gpu/drm/xe/xe_svm.h:1:
+// SPDX-License-Identifier: MIT

-:40: WARNING:SPDX_LICENSE_TAG: Missing or malformed SPDX-License-Identifier tag in line 1 #40: FILE: drivers/gpu/drm/xe/xe_svm.h:1:
+// SPDX-License-Identifier: MIT

total: 0 errors, 3 warnings, 0 checks, 73 lines checked
21034598a drm/xe/svm: create xe svm during vm creation
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:25: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #25:
new file mode 100644

-:65: CHECK:ALLOC_SIZEOF_STRUCT: Prefer kzalloc(sizeof(*svm)...) over kzalloc(sizeof(struct xe_svm)...) #65: FILE: drivers/gpu/drm/xe/xe_svm.c:36:
+ svm = kzalloc(sizeof(struct xe_svm), GFP_KERNEL);

total: 0 errors, 1 warnings, 1 checks, 117 lines checked adb21aa60 drm/xe/svm: Trace svm creation
37bbe7f04 drm/xe/svm: add helper to retrieve svm range from address Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:28: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #28: FILE: drivers/gpu/drm/xe/xe_svm.h:71:
+struct xe_svm_range *xe_svm_range_from_addr(struct xe_svm *svm,
+ unsigned long addr);

-:31: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #31:
new file mode 100644

-:56: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #56: FILE: drivers/gpu/drm/xe/xe_svm_range.c:21:
+struct xe_svm_range *xe_svm_range_from_addr(struct xe_svm *svm,
+ unsigned long addr)

total: 0 errors, 1 warnings, 2 checks, 38 lines checked ed971f094 drm/xe/svm: Introduce a helper to build sg table from hmm range -:57: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #57: FILE: drivers/gpu/drm/xe/xe_svm.c:85:
+int xe_svm_build_sg(struct hmm_range *range,
+ struct sg_table *st)

total: 0 errors, 0 warnings, 1 checks, 74 lines checked 989506d3d drm/xe/svm: Add helper for binding hmm range to gpu -:79: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #79: FILE: drivers/gpu/drm/xe/xe_pt.c:1066:
+ if (mmu_interval_read_retry(range->notifier,
+ range->notifier_seq)) {

-:139: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #139: FILE: drivers/gpu/drm/xe/xe_pt.c:1715:
+int xe_bind_svm_range(struct xe_vm *vm, struct xe_tile *tile,
+ struct hmm_range *range, u64 flags)

-:202: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #202: FILE: drivers/gpu/drm/xe/xe_pt.h:51:
+int xe_bind_svm_range(struct xe_vm *vm, struct xe_tile *tile,
+ struct hmm_range *range, u64 flags);

total: 0 errors, 0 warnings, 3 checks, 188 lines checked
c3aaebfaa drm/xe/svm: Add helper to invalidate svm range from GPU
cd94b6534 drm/xe/svm: Remap and provide memmap backing for GPU vram
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:102: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #102:
new file mode 100644

-:118: CHECK:LINE_SPACING: Please don't use multiple blank lines #118: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:12:
+
+

-:168: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #168: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:62:
+ drm_err(&tile->xe->drm, "Failed to remap tile %d memory, errno %d\n",
+ tile->id, ret);

-:174: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #174: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:68:
+ drm_info(&tile->xe->drm, "Added tile %d memory [%llx-%llx] to devm, remapped to %pr\n",
+ tile->id, mr->io_start, mr->io_start + mr->usable_size, res);

-:190: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #190: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:84:
+ devm_release_mem_region(dev, mr->pagemap.range.start,
+ mr->pagemap.range.end - mr->pagemap.range.start +1);

-:190: CHECK:SPACING: spaces preferred around that '+' (ctx:WxV) #190: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:84:
+ mr->pagemap.range.end - mr->pagemap.range.start +1);
^

total: 0 errors, 1 warnings, 5 checks, 136 lines checked 6e63cd593 drm/xe/svm: Introduce svm migration function
-:80: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #80: FILE: drivers/gpu/drm/xe/xe_migrate.c:574:
+static void build_pt_update_batch_sram(struct xe_migrate *m,
+ struct xe_bb *bb, u32 pt_offset,

-:151: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #151: FILE: drivers/gpu/drm/xe/xe_migrate.c:955:
+struct dma_fence *xe_migrate_svm(struct xe_migrate *m,
+ u64 src_dpa,

-:192: CHECK:BRACES: braces {} should be used on all arms of this statement #192: FILE: drivers/gpu/drm/xe/xe_migrate.c:996:
+ if (!src_is_vram) {
[...]
+ else
[...]

-:195: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #195: FILE: drivers/gpu/drm/xe/xe_migrate.c:999:
+ build_pt_update_batch_sram(m, bb, pt_slot * XE_PAGE_SIZE,
+ src_dpa, round_update_size);

-:198: ERROR:ELSE_AFTER_BRACE: else should follow close brace '}' #198: FILE: drivers/gpu/drm/xe/xe_migrate.c:1002:
+ }
+ else

-:201: CHECK:BRACES: braces {} should be used on all arms of this statement #201: FILE: drivers/gpu/drm/xe/xe_migrate.c:1005:
+ if (!dst_is_vram) {
[...]
+ else
[...]

-:204: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #204: FILE: drivers/gpu/drm/xe/xe_migrate.c:1008:
+ build_pt_update_batch_sram(m, bb, pt_slot * XE_PAGE_SIZE,
+ dst_dpa, round_update_size);

-:207: ERROR:ELSE_AFTER_BRACE: else should follow close brace '}' #207: FILE: drivers/gpu/drm/xe/xe_migrate.c:1011:
+ }
+ else

-:210: CHECK:LINE_SPACING: Please don't use multiple blank lines #210: FILE: drivers/gpu/drm/xe/xe_migrate.c:1014:
+
+

-:270: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #270: FILE: drivers/gpu/drm/xe/xe_migrate.h:92:
+struct dma_fence *xe_migrate_svm(struct xe_migrate *m,
+ u64 src_dpa,

total: 2 errors, 0 warnings, 8 checks, 244 lines checked 150d7173a drm/xe/svm: implement functions to allocate and free device memory -:40: CHECK:LINE_SPACING: Please don't use multiple blank lines #40: FILE: drivers/gpu/drm/xe/xe_svm.h:78:
+
+

-:42: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #42: FILE: drivers/gpu/drm/xe/xe_svm.h:80:
+int xe_devm_alloc_pages(struct xe_tile *tile,
+ unsigned long npages,

-:116: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #116: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:67:
+int xe_devm_alloc_pages(struct xe_tile *tile,
+ unsigned long npages,

-:126: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #126: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:77:
+ ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE,
+ blocks, DRM_BUDDY_TOPDOWN_ALLOCATION);

-:147: ERROR:SPACING: space required before the open parenthesis '(' #147: FILE: drivers/gpu/drm/xe/xe_svm_devmem.c:98:
+ for(i = 0; i < pages_per_block; i++) {

total: 1 errors, 0 warnings, 4 checks, 176 lines checked
4e5fce92b drm/xe/svm: Trace buddy block allocation and free
-:10: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one

-:56: ERROR:CODE_INDENT: code indent should use tabs where possible #56: FILE: drivers/gpu/drm/xe/xe_trace.h:605:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$

-:56: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #56: FILE: drivers/gpu/drm/xe/xe_trace.h:605:
+DECLARE_EVENT_CLASS(xe_buddy_block,
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),

-:56: WARNING:LEADING_SPACE: please, no spaces at the start of a line #56: FILE: drivers/gpu/drm/xe/xe_trace.h:605:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$

-:57: ERROR:CODE_INDENT: code indent should use tabs where possible #57: FILE: drivers/gpu/drm/xe/xe_trace.h:606:
+ TP_ARGS(block, size, pfn),$

-:57: WARNING:LEADING_SPACE: please, no spaces at the start of a line #57: FILE: drivers/gpu/drm/xe/xe_trace.h:606:
+ TP_ARGS(block, size, pfn),$

-:59: ERROR:CODE_INDENT: code indent should use tabs where possible #59: FILE: drivers/gpu/drm/xe/xe_trace.h:608:
+ TP_STRUCT__entry($

-:59: WARNING:LEADING_SPACE: please, no spaces at the start of a line #59: FILE: drivers/gpu/drm/xe/xe_trace.h:608:
+ TP_STRUCT__entry($

-:59: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #59: FILE: drivers/gpu/drm/xe/xe_trace.h:608:
+ TP_STRUCT__entry(

-:60: ERROR:CODE_INDENT: code indent should use tabs where possible #60: FILE: drivers/gpu/drm/xe/xe_trace.h:609:
+ __field(u64, block)$

-:60: WARNING:LEADING_SPACE: please, no spaces at the start of a line #60: FILE: drivers/gpu/drm/xe/xe_trace.h:609:
+ __field(u64, block)$

-:61: ERROR:CODE_INDENT: code indent should use tabs where possible #61: FILE: drivers/gpu/drm/xe/xe_trace.h:610:
+ __field(u64, header)$

-:61: WARNING:LEADING_SPACE: please, no spaces at the start of a line #61: FILE: drivers/gpu/drm/xe/xe_trace.h:610:
+ __field(u64, header)$

-:62: ERROR:CODE_INDENT: code indent should use tabs where possible #62: FILE: drivers/gpu/drm/xe/xe_trace.h:611:
+ __field(u64, size)$

-:62: WARNING:LEADING_SPACE: please, no spaces at the start of a line #62: FILE: drivers/gpu/drm/xe/xe_trace.h:611:
+ __field(u64, size)$

-:63: ERROR:CODE_INDENT: code indent should use tabs where possible #63: FILE: drivers/gpu/drm/xe/xe_trace.h:612:
+ __field(u64, pfn)$

-:63: WARNING:LEADING_SPACE: please, no spaces at the start of a line #63: FILE: drivers/gpu/drm/xe/xe_trace.h:612:
+ __field(u64, pfn)$

-:64: ERROR:CODE_INDENT: code indent should use tabs where possible #64: FILE: drivers/gpu/drm/xe/xe_trace.h:613:
+ ),$

-:64: WARNING:LEADING_SPACE: please, no spaces at the start of a line #64: FILE: drivers/gpu/drm/xe/xe_trace.h:613:
+ ),$

-:66: ERROR:CODE_INDENT: code indent should use tabs where possible #66: FILE: drivers/gpu/drm/xe/xe_trace.h:615:
+ TP_fast_assign($

-:66: WARNING:LEADING_SPACE: please, no spaces at the start of a line #66: FILE: drivers/gpu/drm/xe/xe_trace.h:615:
+ TP_fast_assign($

-:66: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #66: FILE: drivers/gpu/drm/xe/xe_trace.h:615:
+ TP_fast_assign(

-:67: ERROR:CODE_INDENT: code indent should use tabs where possible #67: FILE: drivers/gpu/drm/xe/xe_trace.h:616:
+ __entry->block = (u64)block;$

-:67: WARNING:LEADING_SPACE: please, no spaces at the start of a line #67: FILE: drivers/gpu/drm/xe/xe_trace.h:616:
+ __entry->block = (u64)block;$

-:68: ERROR:CODE_INDENT: code indent should use tabs where possible #68: FILE: drivers/gpu/drm/xe/xe_trace.h:617:
+ __entry->header = block->header;$

-:68: WARNING:LEADING_SPACE: please, no spaces at the start of a line #68: FILE: drivers/gpu/drm/xe/xe_trace.h:617:
+ __entry->header = block->header;$

-:69: ERROR:CODE_INDENT: code indent should use tabs where possible #69: FILE: drivers/gpu/drm/xe/xe_trace.h:618:
+ __entry->size = size;$

-:69: WARNING:LEADING_SPACE: please, no spaces at the start of a line #69: FILE: drivers/gpu/drm/xe/xe_trace.h:618:
+ __entry->size = size;$

-:70: ERROR:CODE_INDENT: code indent should use tabs where possible #70: FILE: drivers/gpu/drm/xe/xe_trace.h:619:
+ __entry->pfn = pfn;$

-:70: WARNING:LEADING_SPACE: please, no spaces at the start of a line #70: FILE: drivers/gpu/drm/xe/xe_trace.h:619:
+ __entry->pfn = pfn;$

-:71: ERROR:CODE_INDENT: code indent should use tabs where possible #71: FILE: drivers/gpu/drm/xe/xe_trace.h:620:
+ ),$

-:71: WARNING:LEADING_SPACE: please, no spaces at the start of a line #71: FILE: drivers/gpu/drm/xe/xe_trace.h:620:
+ ),$

-:73: ERROR:CODE_INDENT: code indent should use tabs where possible #73: FILE: drivers/gpu/drm/xe/xe_trace.h:622:
+ TP_printk("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n",$

-:73: WARNING:LEADING_SPACE: please, no spaces at the start of a line #73: FILE: drivers/gpu/drm/xe/xe_trace.h:622:
+ TP_printk("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n",$

-:74: ERROR:CODE_INDENT: code indent should use tabs where possible #74: FILE: drivers/gpu/drm/xe/xe_trace.h:623:
+ __entry->block, __entry->header, __entry->size, __entry->pfn)$

-:74: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #74: FILE: drivers/gpu/drm/xe/xe_trace.h:623:
+ TP_printk("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n",
+ __entry->block, __entry->header, __entry->size, __entry->pfn)

-:74: WARNING:LEADING_SPACE: please, no spaces at the start of a line #74: FILE: drivers/gpu/drm/xe/xe_trace.h:623:
+ __entry->block, __entry->header, __entry->size, __entry->pfn)$

-:77: CHECK:LINE_SPACING: Please don't use multiple blank lines #77: FILE: drivers/gpu/drm/xe/xe_trace.h:626:
+
+

-:79: ERROR:CODE_INDENT: code indent should use tabs where possible #79: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$

-:79: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #79: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+DEFINE_EVENT(xe_buddy_block, xe_buddy_block_alloc,
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),

-:79: WARNING:LEADING_SPACE: please, no spaces at the start of a line #79: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$

-:80: ERROR:CODE_INDENT: code indent should use tabs where possible #80: FILE: drivers/gpu/drm/xe/xe_trace.h:629:
+ TP_ARGS(block, size, pfn)$

-:80: WARNING:LEADING_SPACE: please, no spaces at the start of a line #80: FILE: drivers/gpu/drm/xe/xe_trace.h:629:
+ TP_ARGS(block, size, pfn)$

-:83: CHECK:LINE_SPACING: Please don't use multiple blank lines #83: FILE: drivers/gpu/drm/xe/xe_trace.h:632:
+
+

-:85: ERROR:CODE_INDENT: code indent should use tabs where possible #85: FILE: drivers/gpu/drm/xe/xe_trace.h:634:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$

-:85: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #85: FILE: drivers/gpu/drm/xe/xe_trace.h:634:
+DEFINE_EVENT(xe_buddy_block, xe_buddy_block_free,
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),

-:85: WARNING:LEADING_SPACE: please, no spaces at the start of a line #85: FILE: drivers/gpu/drm/xe/xe_trace.h:634:
+ TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn),$

-:86: ERROR:CODE_INDENT: code indent should use tabs where possible #86: FILE: drivers/gpu/drm/xe/xe_trace.h:635:
+ TP_ARGS(block, size, pfn)$

-:86: WARNING:LEADING_SPACE: please, no spaces at the start of a line #86: FILE: drivers/gpu/drm/xe/xe_trace.h:635:
+ TP_ARGS(block, size, pfn)$

total: 20 errors, 21 warnings, 8 checks, 65 lines checked 92094bcea drm/xe/svm: Handle CPU page fault
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:44: WARNING:LINE_SPACING: Missing a blank line after declarations #44: FILE: drivers/gpu/drm/xe/xe_device_types.h:585:
+ u64 offset = (pfn << PAGE_SHIFT) - mr->hpa_base;
+ dpa = mr->dpa_base + offset;

-:74: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #74: FILE: drivers/gpu/drm/xe/xe_svm.h:78:
+bool xe_svm_range_belongs_to_vma(struct mm_struct *mm,
+ struct xe_svm_range *range,

-:120: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #120:
new file mode 100644

-:147: CHECK:LINE_SPACING: Please don't use multiple blank lines #147: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:23:
+
+

-:169: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #169: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:45:
+static struct page *alloc_host_page(struct device *dev,
+ struct vm_area_struct *vma,

-:220: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #220: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:96:
+static int migrate_page_vram_to_ram(struct vm_area_struct *vma, unsigned long addr,
+ unsigned long src_pfn, unsigned long *dst_pfn)

-:247: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #247: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:123:
+ fence = xe_migrate_svm(tile->migrate, src_dpa, true,
+ dma_addr, false, PAGE_SIZE);

-:317: CHECK:SPACING: spaces preferred around that '*' (ctx:VxW) #317: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:193:
+ buf = kvcalloc(npages, 2* sizeof(*migrate_vma.src), GFP_KERNEL);
^

-:330: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #330: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:206:
+ ret = migrate_page_vram_to_ram(vma, addr, migrate_vma.src[i],
+ migrate_vma.dst + i);

-:386: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #386: FILE: drivers/gpu/drm/xe/xe_svm_range.c:48:
+bool xe_svm_range_belongs_to_vma(struct mm_struct *mm,
+ struct xe_svm_range *range,

total: 0 errors, 2 warnings, 8 checks, 342 lines checked 9520bed78 drm/xe/svm: trace svm range migration -:50: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #50: FILE: drivers/gpu/drm/xe/xe_trace.h:609:
+ TP_STRUCT__entry(

-:55: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #55: FILE: drivers/gpu/drm/xe/xe_trace.h:614:
+ TP_fast_assign(

-:65: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #65: FILE: drivers/gpu/drm/xe/xe_trace.h:624:
+DEFINE_EVENT(xe_svm_migrate, xe_svm_migrate_vram_to_sram,
+ TP_PROTO(struct xe_svm_range *range),

-:69: CHECK:LINE_SPACING: Please don't use multiple blank lines #69: FILE: drivers/gpu/drm/xe/xe_trace.h:628:
+
+

-:71: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #71: FILE: drivers/gpu/drm/xe/xe_trace.h:630:
+DEFINE_EVENT(xe_svm_migrate, xe_svm_migrate_sram_to_vram,
+ TP_PROTO(struct xe_svm_range *range),

total: 0 errors, 0 warnings, 5 checks, 49 lines checked
93a9ddcb2 drm/xe/svm: Implement functions to register and unregister mmu notifier -:46: WARNING:TYPO_SPELLING: 'splitted' may be misspelled - perhaps 'split'? #46: FILE: drivers/gpu/drm/xe/xe_svm.h:71:
+ * which means one vma can be splitted into multiple
^^^^^^^^

-:143: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #143: FILE: drivers/gpu/drm/xe/xe_svm_range.c:124:
+ ret = mmu_interval_notifier_insert_locked(&range->notifier, vma->vm_mm,
+ start, length, &xe_svm_mni_ops);

total: 0 errors, 1 warnings, 1 checks, 118 lines checked
541192d56 drm/xe/svm: Implement the mmu notifier range invalidate callback
-:24: WARNING:COMMIT_LOG_LONG_LINE: Prefer a maximum 75 chars per line (possible unwrapped commit description?) #24:
unregister mmu notifier (calling mmu_interval_notifier_remove) also requires

-:78: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #78: FILE: drivers/gpu/drm/xe/xe_svm_range.c:64:
+static bool xe_svm_range_invalidate(struct mmu_interval_notifier *mni,
+ const struct mmu_notifier_range *range,

-:90: WARNING:REPEATED_WORD: Possible repeated word: 'in' #90: FILE: drivers/gpu/drm/xe/xe_svm_range.c:76:
+ * page table and unregister mmu notifier in in xe_destroy_svm,

total: 0 errors, 2 warnings, 1 checks, 64 lines checked 9ac75e0da drm/xe/svm: clean up svm range during process exit -:37: CHECK:SPACING: spaces preferred around that '*' (ctx:VxV) #37: FILE: drivers/gpu/drm/xe/xe_svm.c:24:
+#define MAX_SVM_RANGE (1024*1024)

                            ^

-:44: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #44: FILE: drivers/gpu/drm/xe/xe_svm.c:31:
+ range_array = kzalloc(sizeof(struct xe_svm_range *) * MAX_SVM_RANGE,
+ GFP_KERNEL);

-:55: ERROR:SPACING: space required before the open parenthesis '(' #55: FILE: drivers/gpu/drm/xe/xe_svm.c:42:
+ for(; i >= 0; i--)

total: 1 errors, 0 warnings, 2 checks, 66 lines checked 41c558827 drm/xe/svm: Move a few structures to xe_gt.h 054dcebaf drm/xe/svm: migrate svm range to vram
-:57: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #57: FILE: drivers/gpu/drm/xe/xe_svm.h:84:
+int svm_migrate_range_to_vram(struct xe_svm_range *range,
+ struct vm_area_struct *vma,

-:71: CHECK:LINE_SPACING: Please don't use multiple blank lines #71: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:233:
+
+

-:83: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #83: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:245:
+int svm_migrate_range_to_vram(struct xe_svm_range *range,
+ struct vm_area_struct *vma,

-:110: CHECK:SPACING: spaces preferred around that '*' (ctx:VxW) #110: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:272:
+ buf = kvcalloc(npages, 2* sizeof(*migrate.src) + sizeof(*src_dma_addr),
^

-:111: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #111: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:273:
+ buf = kvcalloc(npages, 2* sizeof(*migrate.src) + sizeof(*src_dma_addr),
+ GFP_KERNEL);

-:112: ERROR:SPACING: space required before the open parenthesis '(' #112: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:274:
+ if(!buf)

-:116: CHECK:SPACING: No space is necessary after a cast #116: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:278:
+ src_dma_addr = (dma_addr_t *) (migrate.dst + npages);

-:124: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #124: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:286:
+ drm_err(&tile->xe->drm, "vma setup returned %d for range [%lx - %lx]\n",
+ ret, start, end);

-:136: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #136: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:298:
+ drm_warn(&tile->xe->drm, "Partial migration for range [%lx - %lx], range is %ld pages, migrate only %ld pages\n",
+ start, end, npages, migrate.cpages);

-:150: WARNING:LONG_LINE: line length of 101 exceeds 100 columns #150: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:312:
+ drm_warn(&tile->xe->drm, "dma map error for host pfn %lx\n", migrate.src[i]);

-:155: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #155: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:317:
+ fence = xe_migrate_svm(tile->migrate, src_dma_addr[i], false,
+ dst_dpa, true, PAGE_SIZE);

-:158: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #158: FILE: drivers/gpu/drm/xe/xe_svm_migrate.c:320:
+ drm_warn(&tile->xe->drm, "migrate host page (pfn: %lx) to vram failed\n",
+ migrate.src[i]);

total: 1 errors, 1 warnings, 10 checks, 145 lines checked ace9fe594 drm/xe/svm: Populate svm range
-:49: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #49: FILE: drivers/gpu/drm/xe/xe_svm.c:166:
+static int svm_populate_range(struct xe_svm_range *svm_range,
+ struct hmm_range *hmm_range, bool write)

total: 0 errors, 0 warnings, 1 checks, 64 lines checked 4d4bd0bf2 drm/xe/svm: GPU page fault support
-:101: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #101: FILE: drivers/gpu/drm/xe/xe_svm.c:236:
+static bool svm_should_migrate(struct xe_svm_range *range,
+ struct xe_mem_region *dst_region, bool is_atomic_fault)

-:119: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #119: FILE: drivers/gpu/drm/xe/xe_svm.c:254:
+int xe_svm_handle_gpu_fault(struct xe_vm *vm,
+ struct xe_gt *gt,

-:143: WARNING:SPACING: space prohibited between function name and open parenthesis '(' #143: FILE: drivers/gpu/drm/xe/xe_svm.c:278:
+ if (!svm_access_allowed (vma, access_type != ACCESS_TYPE_READ)) {

-:158: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #158: FILE: drivers/gpu/drm/xe/xe_svm.c:293:
+ if (svm_should_migrate(range, &gt->tile->mem.vram,
+ access_type == ACCESS_TYPE_ATOMIC))

-:178: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #178: FILE: drivers/gpu/drm/xe/xe_svm.c:313:
+ ret = xe_bind_svm_range(vm, gt->tile, &hmm_range,
+ !(vma->vm_flags & VM_WRITE) ? DRM_XE_VM_BIND_FLAG_READONLY : 0);

-:207: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #207: FILE: drivers/gpu/drm/xe/xe_svm.h:99:
+struct xe_svm_range *xe_svm_range_create(struct xe_svm *svm,
+ struct vm_area_struct *vma);

-:216: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #216: FILE: drivers/gpu/drm/xe/xe_svm.h:113:
+int xe_svm_handle_gpu_fault(struct xe_vm *vm,
+ struct xe_gt *gt,

-:250: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #250: FILE: drivers/gpu/drm/xe/xe_svm_range.c:208:
+struct xe_svm_range *xe_svm_range_create(struct xe_svm *svm,
+ struct vm_area_struct *vma)

-:262: ERROR:SPACING: space required before the open brace '{' #262: FILE: drivers/gpu/drm/xe/xe_svm_range.c:220:
+ if (xe_svm_range_register_mmu_notifier(range)){

total: 1 errors, 1 warnings, 7 checks, 218 lines checked
a45edccde drm/xe/svm: Add DRM_XE_SVM kernel config entry
fca35d34b drm/xe/svm: Add svm memory hints interface
-:7: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one

-:32: WARNING:LONG_LINE: line length of 111 exceeds 100 columns #32: FILE: include/uapi/drm/xe_drm.h:118:
+#define DRM_IOCTL_XE_SVM DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_SVM, struct drm_xe_svm_args)

total: 0 errors, 2 warnings, 0 checks, 64 lines checked

#125 lore raw

✗ CI.KUnit: failure for XeKmd basic SVM support

In reply to Oak Zeng
== Series Details ==

== Series Details ==

Series: XeKmd basic SVM support
URL : https://patchwork.freedesktop.org/series/128910/
State : failure

== Summary ==

+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig ERROR:root:../drivers/gpu/drm/xe/xe_mmio.c: In function ‘mmio_fini’:
../drivers/gpu/drm/xe/xe_mmio.c:361:5: warning: unused variable ‘id’ [-Wunused-variable]

  361 |  u8 id;
      |     ^~

../drivers/gpu/drm/xe/xe_mmio.c:360:18: warning: unused variable ‘tile’ [-Wunused-variable]
360 | struct xe_tile *tile;
| ^~~~
../drivers/gpu/drm/xe/xe_mmio.c: At top level:
../drivers/gpu/drm/xe/xe_mmio.c:372:1: error: expected identifier or ‘(’ before ‘}’ token
372 | }

      | ^

make[7]: *** [../scripts/Makefile.build:243: drivers/gpu/drm/xe/xe_mmio.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:480: drivers/gpu/drm/xe] Error 2 make[6]: *** Waiting for unfinished jobs....
make[5]: *** [../scripts/Makefile.build:480: drivers/gpu/drm] Error 2 make[4]: *** [../scripts/Makefile.build:480: drivers/gpu] Error 2 make[3]: *** [../scripts/Makefile.build:480: drivers] Error 2 make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/kernel/Makefile:1911: .] Error 2
make[1]: *** [/kernel/Makefile:234: __sub-make] Error 2
make: *** [Makefile:234: __sub-make] Error 2

[02:46:31] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[02:46:35] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel