Skip to contentSkip to conversation

Core Kernel support for Compute-Offload Devices

Ksummit-discuss Joerg Roedel 10 participants 27 messages Jul 30, 2015 – Aug 4, 2015

lore
Share & tools
lore
#0 lore raw

[
The topic is highly technical and could be a tech topic. But it also touches multiple subsystems, so I decided to submit it as a core topic.
]

Across architectures and vendors there are new devices coming up for offloading tasks from the CPUs. Most of these devices are capable to operate on user address spaces.

Besides the commonalities there are important differences in the memory model these devices offer. Some work only on system RAM, others come with their own memory which may or may not be accessible by the CPU.

I'd like to discuss what support we need in the core kernel for these devices. A probably incomplete list of open questions:

(1) Do we need the concept of an off-CPU task in the kernel together with a common interface to create and manage them and probably a (collection of) batch scheduler(s) for these tasks?

(2) Changes in memory management for devices accessing user address spaces:

(2.1) How can we best support the different memory models
these devices support?

(2.2) How do we handle the off-CPU users of an mm_struct?

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

(4) What features do we require in the hardware to support it with a common interface?

I think it would be great if the kernel would have a common interface for these kind of devices. Currently every vendor develops its own interface with various hacks to work around core code behavior.

I am particularily interested in this topic because on PCIe newer IOMMUs are often an integral part in supporting these devices (ARM-SMMUv3, Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also touch the IOMMU code.

Probably (uncomplete list of) interested people:

David Woodhouse
Jesse Barnes
Will Deacon
Paul E. McKenney
Rik van Riel
Mel Gorman
Andrea Arcangeli
Christoph Lameter
Jérôme Glisse

#1 lore raw
Quoted 26 lines · Joerg Roedel

[
The topic is highly technical and could be a tech topic. But it also touches multiple subsystems, so I decided to submit it as a core topic.
]

Across architectures and vendors there are new devices coming up for offloading tasks from the CPUs. Most of these devices are capable to operate on user address spaces.

Besides the commonalities there are important differences in the memory
model these devices offer. Some work only on system RAM, others come with their own memory which may or may not be accessible by the CPU.

I'd like to discuss what support we need in the core kernel for these devices. A probably incomplete list of open questions:

(1) Do we need the concept of an off-CPU task in the kernel together with a common interface to create and manage them and probably a (collection of) batch scheduler(s) for these tasks?

(2) Changes in memory management for devices accessing user address spaces:

Quoted 7 lines

(2.1) How can we best support the different memory models
these devices support?

(2.2) How do we handle the off-CPU users of an mm_struct?

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

And how do we handle the assignment of Address Space IDs? The AMD implementation currently allows the PASID space to be managed per -device, but I understand ARM systems handle the TLB shootdown broadcasts in hardware and need the PASID that the device sees to be identical to the ASID on the CPU's MMU? And there are reasons why we might actually want that model on Intel systems too. I'm working on the Intel SVM right now, and looking at a single-PASID-space model (partly because the PASID tables have to be physically contiguous, and they can be huge!).

Quoted 3 lines

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

This is quite a horrid one, but perhaps ties into generic NUMA considerations — if a memory page is being frequently accessed by something that it's far away from, can we move it to closer memory?

The question is how we handle that. We do have Extended Accessed bits in the Intel implementation of SVM that let us know that a given PTE was used from a device. Although not *which* device, in cases where there might be more than one.

Quoted 6 lines

(4) What features do we require in the hardware to support it with a common interface?

I think it would be great if the kernel would have a common interface for these kind of devices. Currently every vendor develops its own interface with various hacks to work around core code behavior.

Right. For now it's almost all internal on-chip stuff so it's kind of tolerable to have vendor-specific implenentations. But we are starting to see PCIe root ports which support the necessary TLP prefixes to support SVM on discrete devices. And then it'll be really important to have this working cross-platform.

Signature · 1 line
dwmw2
#2 lore raw
Quoted 4 lines · David Woodhouse
Quoted 2 lines · Joerg Roedel

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

Quoted 9 lines

And how do we handle the assignment of Address Space IDs? The AMD implementation currently allows the PASID space to be managed per -device, but I understand ARM systems handle the TLB shootdown broadcasts in hardware and need the PASID that the device sees to be identical to the ASID on the CPU's MMU? And there are reasons why we might actually want that model on Intel systems too. I'm working on the Intel SVM right now, and looking at a single-PASID-space model (partly because the PASID tables have to be physically contiguous, and they can be huge!).

True, ASIDs would be one thing that needs to be attached to a mm_struct, but I am also interested what other platforms might need here. For example, is there a better way to track these off-cpu users than using mmu-notifiers.

Quoted 7 lines
Quoted 3 lines

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

This is quite a horrid one, but perhaps ties into generic NUMA considerations — if a memory page is being frequently accessed by something that it's far away from, can we move it to closer memory?

Yeah, conceptually it is NUMA, so it might fit there. But the difference to the current NUMA handling is that the device memory is not always completly visible to the CPU, so I think quite some significant changes are necessary to make this work.

Another idea is to handle migration like swapping. The difference to real swapping is that it is not relying on the LRU lists but the device access patterns we measure.

Quoted 4 lines

The question is how we handle that. We do have Extended Accessed bits in the Intel implementation of SVM that let us know that a given PTE was used from a device. Although not *which* device, in cases where there might be more than one.

One way would be to use seperate page-tables for the devices (which, on the other side, somehow contradicts the design of the hardware, because its designed to reuse cpu page-tables).

And I don't know which features other devices have (like the CAPI devices on Power that Paul wrote about) to help in this decission.

Joerg

#15 lore raw
In reply to Joerg Roedel
Quoted 9 lines · Joerg Roedel

[
The topic is highly technical and could be a tech topic. But it also touches multiple subsystems, so I decided to submit it as a core topic.
]

Across architectures and vendors there are new devices coming up for offloading tasks from the CPUs. Most of these devices are capable to operate on user address spaces.

There is cross-overs with the proposed FPGA topic as well, for example CAPI is typically FPGA's that can operate on user address spaces ;-)

Quoted 6 lines

Besides the commonalities there are important differences in the memory model these devices offer. Some work only on system RAM, others come with their own memory which may or may not be accessible by the CPU.

I'd like to discuss what support we need in the core kernel for these devices. A probably incomplete list of open questions:

I would definitely like to attend this.

Quoted 4 lines

(1) Do we need the concept of an off-CPU task in the kernel together with a common interface to create and manage them and probably a (collection of) batch scheduler(s) for these tasks?

It might be interesting at least to cleanup how we handle & account page faults for these things. Scheduling is a different matter, for CAPI for example the scheduling is entirely done in HW. For things like GPU, it's a mixture of HW and generally some kind of on-GPU kernel isn't it ? Quite proprietary in any case. Back in the Cell days, the kernel did schedule the SPUs so this would have been a use case of what you propose.

So I'd think that such an off-core scheduler, while a useful thing for some of these devices, should be an optional component, ie, the other functionalities shouldn't necessarily depend on it.

Quoted 10 lines

(2) Changes in memory management for devices accessing user address spaces:

(2.1) How can we best support the different memory models
these devices support?

(2.2) How do we handle the off-CPU users of an mm_struct?

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

Right. Some of this (GPUs, MLX) use the proposed HMM infrastructure that Jerome Glisse have been developing, so he would be an interested party here, which hooks into the existing MM. Some of these like CAPI (or more stuff I can't quite talk about just yet) will just share the MMU data structures (direct access to the host page tables).

The refcounting of mm_struct comes to mind, but also, dealing with the tracking of which CPU accessed a given context (for example, on POWER, with CAPI, we need to "upgrade" to global tlb invalidations even for single threaded apps if the context was used by such an accelerator).

Quoted 3 lines

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

Definitely a hot subject. I don't now if you have seen the "proposal" that Paul McKenney posted a while back. This is in part what HMM does for non-cache-coherent devices. There are lots of open questions for cache-coherent ones, such as should we provide struct page for them, how do we keep normal kernel allocs off the device, etc... ideas like memory-only NUMA nodes with large distance did crop up.

Quoted 14 lines

(4) What features do we require in the hardware to support it with a common interface?

I think it would be great if the kernel would have a common interface for these kind of devices. Currently every vendor develops its own interface with various hacks to work around core code behavior.

I am particularily interested in this topic because on PCIe newer IOMMUs are often an integral part in supporting these devices (ARM-SMMUv3, Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also touch the IOMMU code.

Probably (uncomplete list of) interested people:

Quoted 9 lines

David Woodhouse
Jesse Barnes
Will Deacon
Paul E. McKenney
Rik van Riel
Mel Gorman
Andrea Arcangeli
Christoph Lameter
Jérôme Glisse

Add me :)

Cheers,
Ben.

Quoted 4 lines

_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#17 lore raw
In reply to Joerg Roedel
Quoted 4 lines · Joerg Roedel

(1) Do we need the concept of an off-CPU task in the kernel together with a common interface to create and manage them and probably a (collection of) batch scheduler(s) for these tasks?

Given that some of these compute offload devices share the same address space (mm_struct) as the threads running on CPUs, it would be easiest if there was a reference on the mm_struct for the threads that are running off-CPU.

I do not know if a generic scheduler would work, since it is common to have N threads on compute devices all bound to the same address space, etc.

Different devices might even require different schedulers, but having a common data structure that pins mm_struct, provides for a place to have state (like register content) stored, and has pointers to scheduler, driver, and cleanup functions could be really useful.

Quoted 10 lines

(2) Changes in memory management for devices accessing user address spaces:

(2.1) How can we best support the different memory models
these devices support?

(2.2) How do we handle the off-CPU users of an mm_struct?

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

Jerome has a bunch of code for this already.

Quoted 3 lines

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

I believe he has looked at migration too, but not implemented it yet.

If compute-offload devices are a kernel summit topic this year, it would be useful to invite Jerome Glisse.

Quoted 23 lines

(4) What features do we require in the hardware to support it with a common interface?

I think it would be great if the kernel would have a common interface for these kind of devices. Currently every vendor develops its own interface with various hacks to work around core code behavior.

I am particularily interested in this topic because on PCIe newer IOMMUs are often an integral part in supporting these devices (ARM-SMMUv3, Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also touch the IOMMU code.

Probably (uncomplete list of) interested people:

David Woodhouse
Jesse Barnes
Will Deacon
Paul E. McKenney
Rik van Riel
Mel Gorman
Andrea Arcangeli
Christoph Lameter
Jérôme Glisse

Signature · 1 line
All rights reversed
#18 lore raw
Quoted 16 lines · Rik van Riel
Quoted 4 lines · Joerg Roedel

(1) Do we need the concept of an off-CPU task in the kernel together with a common interface to create and manage them and probably a (collection of) batch scheduler(s) for these tasks?

Given that some of these compute offload devices share the same address space (mm_struct) as the threads running on CPUs, it would be easiest if there was a reference on the mm_struct for the threads that are running off-CPU.

I do not know if a generic scheduler would work, since it is common to have N threads on compute devices all bound to the same address space, etc.

Quoted 5 lines

Different devices might even require different schedulers, but having a common data structure that pins mm_struct, provides for a place to have state (like register content) stored, and has pointers to scheduler, driver, and cleanup functions could be really useful.

Kernel scheduling does not match what hw (today and tomorrow) can do. You have to think 10 000 or 100 000 threads when it comes to GPU (and i would not be surprise that couple years down the road we reach the 1Millions threads).

With so many threads, you do not want to stop them midway, what you really want is rush to completion so you never have to store/save their information.

Hence scheduling here is different, on GPU it is more about a queue of several thousand thread and you just move things up and down on what need to be executed first. Then GPU have hw scheduling that constantly switch btw active thread this why memory latency is so well hidden on GPU.

That being said like Rik said, some common framework would probably make sense, especialy to keep some kind of fairness. But it is definitly not the preempt taks, schedule another one model.

It is the wait current active thread of process A to finish and schedule bunch of thread of process B.

Quoted 7 lines
Quoted 7 lines

(2) Changes in memory management for devices accessing user address spaces:

(2.1) How can we best support the different memory models
these devices support?

Quoted 6 lines
Quoted 4 lines

(2.2) How do we handle the off-CPU users of an mm_struct?

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

Jerome has a bunch of code for this already.

Yes HMM is all about that. It is the first step to provide common framework inside the kernel (not only for GPU but for any device that wish to transparently access process address space.

Quoted 7 lines
Quoted 4 lines

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

I believe he has looked at migration too, but not implemented it yet.

I already implemented several version of it and posted for review couple of them. You do not want automatic migration because kernel as not enough informations here.

HMM design is to let the device driver decide and then device driver can take clue from userspace and use any kind of heuristic to decide what we want to migrate.

Quoted 3 lines

If compute-offload devices are a kernel summit topic this year, it would be useful to invite Jerome Glisse.

I would happy to discuss this topic i have work on GPU open source driver for a long time and last couple year i spend them working on compute and how to integrate this inside the kernel.

Quoted 14 lines
Quoted 14 lines

(4) What features do we require in the hardware to support it with a common interface?

I think it would be great if the kernel would have a common interface for these kind of devices. Currently every vendor develops its own interface with various hacks to work around core code behavior.

I am particularily interested in this topic because on PCIe newer IOMMUs are often an integral part in supporting these devices (ARM-SMMUv3, Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also touch the IOMMU code.

Probably (uncomplete list of) interested people:

Quoted 9 lines
Quoted 9 lines

David Woodhouse
Jesse Barnes
Will Deacon
Paul E. McKenney
Rik van Riel
Mel Gorman
Andrea Arcangeli
Christoph Lameter
Jérôme Glisse

Cheers,
Jérôme

#3 lore raw
In reply to Joerg Roedel
Quoted 15 lines · Joerg Roedel
Quoted 4 lines · David Woodhouse
Quoted 2 lines · Joerg Roedel

(2.3) How can we attach common state for off-CPU tasks to
mm_struct (and what needs to be in there)?

Quoted 9 lines

And how do we handle the assignment of Address Space IDs? The AMD implementation currently allows the PASID space to be managed per -device, but I understand ARM systems handle the TLB shootdown broadcasts in hardware and need the PASID that the device sees to be identical to the ASID on the CPU's MMU? And there are reasons why we might actually want that model on Intel systems too. I'm working on the Intel SVM right now, and looking at a single-PASID-space model (partly because the PASID tables have to be physically contiguous, and they can be huge!).

Quoted 4 lines

True, ASIDs would be one thing that needs to be attached to a mm_struct, but I am also interested what other platforms might need here. For example, is there a better way to track these off-cpu users than using mmu-notifiers.

No the ASID should not be associated with mm_struct. There is to few ASID to have enough of them. I think currently there is only 8bits worth of ASID. So what happen is that the GPU device driver schedule process and recycle ASID as it does.

Which means that ASID really need to be on device driver control as i explained in another mail only device driver knows how to schedule thing for a given device and it is too much hw specific to be move to common code.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

(3) Does it make sense to implement automatic migration of system memory to device memory (when available) and vice versa? How do we decide what and when to migrate?

Quoted 8 lines
Quoted 3 lines

This is quite a horrid one, but perhaps ties into generic NUMA considerations — if a memory page is being frequently accessed by something that it's far away from, can we move it to closer memory?

Yeah, conceptually it is NUMA, so it might fit there. But the difference to the current NUMA handling is that the device memory is not always completly visible to the CPU, so I think quite some significant changes are necessary to make this work.

My HMM patchset already handle all this for anonymous memory, i showed a proof of concept for file back one but i am exploring other method for that.

Quoted 4 lines
Quoted 3 lines

Another idea is to handle migration like swapping. The difference to real swapping is that it is not relying on the LRU lists but the device access patterns we measure.

Quoted 8 lines
Quoted 4 lines

The question is how we handle that. We do have Extended Accessed bits in the Intel implementation of SVM that let us know that a given PTE was used from a device. Although not *which* device, in cases where there might be more than one.

One way would be to use seperate page-tables for the devices (which, on the other side, somehow contradicts the design of the hardware, because its designed to reuse cpu page-tables).

So HMM use a seperate page table for storing information relating to migrated memory. Note that not all hw reuse the CPU page table, some hardware do not and it is very much a platform thing.

Quoted 2 lines

And I don't know which features other devices have (like the CAPI devices on Power that Paul wrote about) to help in this decission.

CAPI would not need special PTE, as on CAPI device memory is accessible by the CPU as regular memory. Only platform that can not offer this need some special handling. AFAICT x86 and ARM have nothing plan to offer such level of integration (thought lately i have not paid close attention to what new features the PCIe consortium is discussing).

Joerg i think you really want to take a look at my patchset to see how i implemented this. I have been discussing this with AMD, Mellanox, NVidia and couple other smaller specialize hw manufacturer.

Cheers,
Jérôme

#19 lore raw
In reply to Jerome Glisse
Quoted 5 lines · Jerome Glisse

Hence scheduling here is different, on GPU it is more about a queue of several thousand thread and you just move things up and down on what need to be executed first. Then GPU have hw scheduling that constantly switch btw active thread this why memory latency is so well hidden on GPU.

Thats why I wrote "batch"-scheduler in the proposal. Its right that it does not make sense to schedule out a GPU process, and some devices do scheduling in hardware anyway.

But the Linux kernel still needs to decide which jobs are sent to the offload device in which order, more like an io-scheduler.

There might be a compute job that only utilizes 60% of the device resources, to the in-kernel scheduler could start another job there to utilize the other 40%.

I think its worth a discussion if some common schedulers (like for blk-io) make sense here too.

Quoted 3 lines

I already implemented several version of it and posted for review couple of them. You do not want automatic migration because kernel as not enough informations here.

Some devices might provide that information, see the extended-access bit of Intel VT-d.

Joerg

#16 lore raw
In reply to Benjamin Herrenschmidt

Hi Ben,

thanks for your thoughts.

Quoted 6 lines · Benjamin Herrenschmidt
Quoted 3 lines

Across architectures and vendors there are new devices coming up for offloading tasks from the CPUs. Most of these devices are capable to operate on user address spaces.

There is cross-overs with the proposed FPGA topic as well, for example CAPI is typically FPGA's that can operate on user address spaces ;-)

True, I was not sure how to put this into the proposal, as FPGAs are a bit different from other compute-offload devices. GPUs take a kernel to execute that is basically a piece of software while FPGAs take a hardware description which in the end might be able to execute its own software. But there is overlap between the topics, thats right.

Quoted 3 lines

So I'd think that such an off-core scheduler, while a useful thing for some of these devices, should be an optional component, ie, the other functionalities shouldn't necessarily depend on it.

Yes, of course. The scheduler(s) could be implemented as a library and optionally be used by the device drivers.

Quoted 5 lines

Right. Some of this (GPUs, MLX) use the proposed HMM infrastructure that Jerome Glisse have been developing, so he would be an interested party here, which hooks into the existing MM. Some of these like CAPI (or more stuff I can't quite talk about just yet) will just share the MMU data structures (direct access to the host page tables).

Everything (what I am aware of), besides of the hardware HMM targets, reuses the CPU MMU structures :) For example all three hardware implementations of ATS/PRI/PASID I am aware of can share them, and as you said, CAPI on Power too.

But they also need to attach some state to mm_struct. As David already said, there will be a need to a global PASID allocation, for example.

Joerg

#20 lore raw
In reply to Joerg Roedel
Quoted 14 lines · Joerg Roedel
Quoted 5 lines · Jerome Glisse

Hence scheduling here is different, on GPU it is more about a queue of several thousand thread and you just move things up and down on what need to be executed first. Then GPU have hw scheduling that constantly switch btw active thread this why memory latency is so well hidden on GPU.

Thats why I wrote "batch"-scheduler in the proposal. Its right that it does not make sense to schedule out a GPU process, and some devices do scheduling in hardware anyway.

But the Linux kernel still needs to decide which jobs are sent to the offload device in which order, more like an io-scheduler.

Quoted 6 lines

There might be a compute job that only utilizes 60% of the device resources, to the in-kernel scheduler could start another job there to utilize the other 40%.

I think its worth a discussion if some common schedulers (like for blk-io) make sense here too.

It is definitly worth a discussion but i fear right now there is little room for anything in the kernel. Hardware scheduling is done is almost 100% hardware. The idea of GPU is that you have 1000 compute unit but the hardware keep track of 10000 threads and at any point in time there is huge probability that 1000 of those 10000 threads are ready to compute something. So if a job is only using 60% of the GPU then the remaining 40% would automaticly be use by the next batch of threads. This is a simplification as the number of thread the hw can keep track of depend of several factor and vary from one model to the other even inside same family of the same manufacturer.

Where kernel have control is which command queue (today GPU have several command queue than run concurently) can spawn threads inside the GPU. Also thing like which queue got priority over another one. You even have mecanism where you can "divide" the GPU among queue (you assign fraction of the GPU compute unit to a particular queue). Thought i expect this last one is vanishing.

Also note that many GPU manufacturer are pushing for userspace queue (i think it is some microsoft requirement) in which case the kernel have even less control.

I agree that blk-io design is probably closest thing that might fit.

Quoted 6 lines
Quoted 3 lines

I already implemented several version of it and posted for review couple of them. You do not want automatic migration because kernel as not enough informations here.

Some devices might provide that information, see the extended-access bit of Intel VT-d.

This would be limited to integrated GPU and so far only on one platform. My point was more that userspace have way more informations to make good decision here. The userspace program is more likely to know what part of the dataset gonna be repeatedly access by the GPU threads.

Cheers,
Jérôme

#23 lore raw
In reply to Joerg Roedel
Quoted 14 lines · On Thursday 30 July 2015 15:00:27 Joerg Roedel wrote:

[
The topic is highly technical and could be a tech topic. But it also touches multiple subsystems, so I decided to submit it as a core topic.
]

Across architectures and vendors there are new devices coming up for offloading tasks from the CPUs. Most of these devices are capable to operate on user address spaces.

Besides the commonalities there are important differences in the memory model these devices offer. Some work only on system RAM, others come with their own memory which may or may not be accessible by the CPU.

Quoted 7 lines

I'd like to discuss what support we need in the core kernel for these devices. A probably incomplete list of open questions:

(1) Do we need the concept of an off-CPU task in the kernel together with a common interface to create and manage them and probably a (collection of) batch scheduler(s) for these tasks?

I think we did this part right with the Cell SPUs 10 years ago: A task is a task, and you just switch between running in user mode and running on the offload engine through some syscall or ioctl.

The part that got us into endless trouble though was trying to satisfy two opposite requirements:

a) having the kernel schedule tasks automatically onto the offload engines and take care of context switches and placement, so you can do multi-user and multi-tasking processing on them.

b) getting most performance out of the of offload engines, by giving a single user total control over the placement and no do any scheduling in the kernel at all.

I would strongly recommend now that any new interface tries to do only one of the two models, but does it right.

Quoted 7 lines

I am particularily interested in this topic because on PCIe newer IOMMUs are often an integral part in supporting these devices (ARM-SMMUv3, Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also touch the IOMMU code.

Probably (uncomplete list of) interested people:

Quoted 9 lines

David Woodhouse
Jesse Barnes
Will Deacon
Paul E. McKenney
Rik van Riel
Mel Gorman
Andrea Arcangeli
Christoph Lameter
Jérôme Glisse

Add me in as well,

Arnd

#21 lore raw
In reply to Jerome Glisse

Hi Jerome,

Quoted 10 lines · Jerome Glisse

It is definitly worth a discussion but i fear right now there is little room for anything in the kernel. Hardware scheduling is done is almost 100% hardware. The idea of GPU is that you have 1000 compute unit but the hardware keep track of 10000 threads and at any point in time there is huge probability that 1000 of those 10000 threads are ready to compute something. So if a job is only using 60% of the GPU then the remaining 40% would automaticly be use by the next batch of threads. This is a simplification as the number of thread the hw can keep track of depend of several factor and vary from one model to the other even inside same family of the same manufacturer.

So the hardware scheduled individual threads, that is right. But still, as you say, there are limits of how many threads the hardware can handle which the device driver needs to take care of, and decide which job will be sent to the offload device next. Same with the priorities for the queues.

Quoted 7 lines
Quoted 2 lines

Some devices might provide that information, see the extended-access bit of Intel VT-d.

This would be limited to integrated GPU and so far only on one platform. My point was more that userspace have way more informations to make good decision here. The userspace program is more likely to know what part of the dataset gonna be repeatedly access by the GPU threads.

Hmm, so what is the point of HMM then? If userspace is going to decide which part of the address space the device needs it could just copy the data over (keeping the address space layout and thus the pointers stable) and you would basically achieve the same without adding a lot of code to memory manangement, no?

Joerg

#24 lore raw
In reply to Arnd Bergmann

Hi Arnd,

Quoted 3 lines · Arnd Bergmann

I think we did this part right with the Cell SPUs 10 years ago: A task is a task, and you just switch between running in user mode and running on the offload engine through some syscall or ioctl.

Do you mean that on Cell the offload is synchronous, so that a task that schedules something to the SPU sleeps until the job there is done?

Quoted 7 lines

The part that got us into endless trouble though was trying to satisfy two opposite requirements:

a) having the kernel schedule tasks automatically onto the offload engines and take care of context switches and placement, so you can do multi-user and multi-tasking processing on them.

Quoted 6 lines

b) getting most performance out of the of offload engines, by giving a single user total control over the placement and no do any scheduling in the kernel at all.

I would strongly recommend now that any new interface tries to do only one of the two models, but does it right.

I think it mostly depends on the use-cases which approach makes more sense. An HPC environment would certainly want to have full control over the placement and scheduling on the offload devices. A desktop environment with typical applications that offload stuff to one or multiple GPUs through optimized libraries would benefit more from automatic scheduling and placement.

It is probably good to hear about the use-cases of the different offload devices to make a good decission here.

Joerg

#22 lore raw
In reply to Joerg Roedel
Quoted 14 lines · Joerg Roedel

Hi Jerome,

Quoted 10 lines · Jerome Glisse

It is definitly worth a discussion but i fear right now there is little room for anything in the kernel. Hardware scheduling is done is almost 100% hardware. The idea of GPU is that you have 1000 compute unit but the hardware keep track of 10000 threads and at any point in time there is huge probability that 1000 of those 10000 threads are ready to compute something. So if a job is only using 60% of the GPU then the remaining 40% would automaticly be use by the next batch of threads. This is a simplification as the number of thread the hw can keep track of depend of several factor and vary from one model to the other even inside same family of the same manufacturer.

Quoted 5 lines

So the hardware scheduled individual threads, that is right. But still, as you say, there are limits of how many threads the hardware can handle which the device driver needs to take care of, and decide which job will be sent to the offload device next. Same with the priorities for the queues.

What i was pointing to is that right now you do not have such granularity of choice from device driver point of view. Right now it is either let command queue spawn thread or not. So it is either stop a command queue or let it run. Thought how and when you can stop a queue vary. In some hw you can only stop it at execution boundary ie you have a packet in a command queue that request 500k thread to be launch, you can only stop that queue once the 500k thread are launch and you can not stop in the middle.

Given that some of those queue a programmed directly from userspace, you can not even force the queue to only schedule small batches of thread (ie something like 1000 thread no more per command packet in the queue).

But newer hw is becoming more capable on that front.

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Some devices might provide that information, see the extended-access bit of Intel VT-d.

This would be limited to integrated GPU and so far only on one platform. My point was more that userspace have way more informations to make good decision here. The userspace program is more likely to know what part of the dataset gonna be repeatedly access by the GPU threads.

Quoted 5 lines

Hmm, so what is the point of HMM then? If userspace is going to decide which part of the address space the device needs it could just copy the data over (keeping the address space layout and thus the pointers stable) and you would basically achieve the same without adding a lot of code to memory manangement, no?

Well no, you can not be "transparent" if you do it in userspace. Let say userspace decide to migrate, then it means CPU will not be able to access that memory, so you have to either PROT_NONE the range or unmap it. This is not what we want. If we get CPU access to memory that is migrated to device memory then we want to migrate it back (at very least one page of it) so CPU can access it. We want this migration back to be transparent from process point of view, like if the memory was swapped on a disk.

Even on hw where the CPU can access device memory properly (maintaining CPU atomic operation for instance which is not the case on PCIe) like with CAPI on powerpc. You have to either have struct page for the device memory of the kernel must know how to handle those special range of memory.

So here HMM never makes any decission, it just leave that with the device driver that can gather more informations from hw and user space to make the best decision. But it might get things wrong or user space program might do stupid thing like trying to access data set with the CPU while the GPU is churning on it. Still we do not want CPU access to be handle as fault or forbidden, when this happen HMM will force migration back to service CPU page fault.

HMM also intends to provide more features that are not doable from user space. Like exclusive write access on a range for the device so that device can perform atomic operation. Again PCIe offer limited capabilities on atomic so only way to provide more advance atomic operations is to map thing read only for CPU and other devices while atomic operation on a device progress.

Another feature is sharing device memory btw different devices. Some devices (not necessarily from the same manufacturer) can communicate to one another and access one another device memory. When a range is migrated on one of such device pair, there must be a way for the other device to find about it. Having userspace device driver try to exchange that kind of information is racy in many way. So easier and better to have it in kernel.

Cheers,
Jérôme

#4 lore raw
In reply to Jerome Glisse
Quoted 4 lines · Jerome Glisse

No the ASID should not be associated with mm_struct. There is to few ASID to have enough of them. I think currently there is only 8bits worth of ASID. So what happen is that the GPU device driver schedule process and recycle ASID as it does.

In PCIe we have 20 bits of PASID. And we are going to expect hardware to implement them all, even if it can only do caching for fewer PASIDs than that.

There is also an expectation that a given MM will have the *same* PASID across all devices.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#5 lore raw
Quoted 10 lines · David Woodhouse
Quoted 4 lines · Jerome Glisse

No the ASID should not be associated with mm_struct. There is to few ASID to have enough of them. I think currently there is only 8bits worth of ASID. So what happen is that the GPU device driver schedule process and recycle ASID as it does.

In PCIe we have 20 bits of PASID. And we are going to expect hardware to implement them all, even if it can only do caching for fewer PASIDs than that.

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

Quoted 2 lines

There is also an expectation that a given MM will have the *same* PASID across all devices.

I understand that this would be prefered. But in case of hw that have only limited number of bit for PASID you surely do not want to starve it ie it would be better to have the device recycle PASID to maximize its usage.

Cheers,
Jérôme

#6 lore raw
Quoted 17 lines · Jerome Glisse
Quoted 10 lines · David Woodhouse
Quoted 4 lines · Jerome Glisse

No the ASID should not be associated with mm_struct. There is to few ASID to have enough of them. I think currently there is only 8bits worth of ASID. So what happen is that the GPU device driver schedule process and recycle ASID as it does.

In PCIe we have 20 bits of PASID. And we are going to expect hardware to implement them all, even if it can only do caching for fewer PASIDs than that.

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

Quoted 7 lines
Quoted 2 lines

There is also an expectation that a given MM will have the *same* PASID across all devices.

I understand that this would be prefered. But in case of hw that have only limited number of bit for PASID you surely do not want to starve it ie it would be better to have the device recycle PASID to maximize its usage.

FWIW, x86 PCID has 12 bits, and, if I ever try to implement support for it, my thought would be to only use 3 or 4 of those bits and aggressively recycle PCIDs.

I have no idea whether ASIC and PCID are supposed to be related at all, given that I don't know anything about how to program these unified memory contraptions.

--Andy

#25 lore raw
In reply to Joerg Roedel
Quoted 9 lines · On Monday 03 August 2015 18:10:27 Joerg Roedel wrote:

Hi Arnd,

Quoted 3 lines · Arnd Bergmann

I think we did this part right with the Cell SPUs 10 years ago: A task is a task, and you just switch between running in user mode and running on the offload engine through some syscall or ioctl.

Do you mean that on Cell the offload is synchronous, so that a task that schedules something to the SPU sleeps until the job there is done?

Correct. The tradeoff here is that accounting in the kernel is greatly simplified, but you need to create a separate thread for each instance of the offload engine that you want to use.

Quoted 21 lines
Quoted 7 lines

The part that got us into endless trouble though was trying to satisfy two opposite requirements:

a) having the kernel schedule tasks automatically onto the offload engines and take care of context switches and placement, so you can do multi-user and multi-tasking processing on them.

Quoted 6 lines

b) getting most performance out of the of offload engines, by giving a single user total control over the placement and no do any scheduling in the kernel at all.

I would strongly recommend now that any new interface tries to do only one of the two models, but does it right.

I think it mostly depends on the use-cases which approach makes more sense. An HPC environment would certainly want to have full control over the placement and scheduling on the offload devices. A desktop environment with typical applications that offload stuff to one or multiple GPUs through optimized libraries would benefit more from automatic scheduling and placement.

Quoted 2 lines

It is probably good to hear about the use-cases of the different offload devices to make a good decission here.

It's also quite likely that not all offload engines fit in one model. E.g. the realtime units on some SoCs should never be scheduled dynamically because that clearly breaks the realtime behavior, while for a lot of others, a scheduler might be the preferred interface.

It may be helpful to have one interface centered around the needs of OpenCL for all those engines that fit into that model, and then do something else for the ones that can't run OpenCL anyway and have some other requirements.

Arnd

#7 lore raw
In reply to Andy Lutomirski
Quoted 26 lines · Andy Lutomirski
Quoted 17 lines · Jerome Glisse
Quoted 10 lines · David Woodhouse
Quoted 4 lines · Jerome Glisse

No the ASID should not be associated with mm_struct. There is to few ASID to have enough of them. I think currently there is only 8bits worth of ASID. So what happen is that the GPU device driver schedule process and recycle ASID as it does.

In PCIe we have 20 bits of PASID. And we are going to expect hardware to implement them all, even if it can only do caching for fewer PASIDs than that.

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

Quoted 7 lines
Quoted 2 lines

There is also an expectation that a given MM will have the *same* PASID across all devices.

I understand that this would be prefered. But in case of hw that have only limited number of bit for PASID you surely do not want to starve it ie it would be better to have the device recycle PASID to maximize its usage.

Quoted 7 lines

FWIW, x86 PCID has 12 bits, and, if I ever try to implement support for it, my thought would be to only use 3 or 4 of those bits and aggressively recycle PCIDs.

I have no idea whether ASIC and PCID are supposed to be related at all, given that I don't know anything about how to program these unified memory contraptions.

So it is even worse than i tought, PASID spec says that the PASID capability register have a field indicating the number of bit for PASID the device supports. Valid value include 1bit which is kind of low, and it does not seems there is any mandatory lower limit.

So if we want one page table (mm) associated with one PASID this might be tedious if not impossible. Let say one device support 12bits and process start using that device, we set PASID to ((1 << 12) - 1), then process want to use another device that only support 8bits, so we would need to switch PASID to something that fit. The first device might not allow to switch PASID without big hammer like fully stoping current job or waiting for current job to complete with no garanty on how long it could takes, minutes, hours, days, ...

I agree and wish we could have one mm one PASID but given hardware, i think we need to do a reality check here :(

Cheers,
Jérôme

#8 lore raw
In reply to Jerome Glisse
Quoted 4 lines · Jerome Glisse

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

AMD hardware currently implements PASIDs with 16 bits. Given that only mm_structs which are used by offload devices get one, this should be enough to put them into a global pool and have one PASID per mm_struct.

Joerg

#9 lore raw
Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Jerome Glisse

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

AMD hardware currently implements PASIDs with 16 bits. Given that only mm_structs which are used by offload devices get one, this should be enough to put them into a global pool and have one PASID per mm_struct.

I think there are many ARM systems which need this model because of the way TLB shootdowns are handled in hardware, and shared with the IOMMU? So we have to use the same ASID for both MMU and IOMMU there, AIUI.

Not that I claim to be an expert on the ARM IOMMUs.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#10 lore raw
Quoted 8 lines · David Woodhouse
Quoted 3 lines · Joerg Roedel

AMD hardware currently implements PASIDs with 16 bits. Given that only mm_structs which are used by offload devices get one, this should be enough to put them into a global pool and have one PASID per mm_struct.

I think there are many ARM systems which need this model because of the way TLB shootdowns are handled in hardware, and shared with the IOMMU? So we have to use the same ASID for both MMU and IOMMU there, AIUI.

Yes, I heard the same, this hardware clearly forces the one-PASID per mm_struct model.

On x86 we probably have to look what offload devices will appear, the ones currently available support 16 bits, which this allows the same model.

Joerg

#11 lore raw
In reply to David Woodhouse
Quoted 11 lines · David Woodhouse
Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Jerome Glisse

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

AMD hardware currently implements PASIDs with 16 bits. Given that only mm_structs which are used by offload devices get one, this should be enough to put them into a global pool and have one PASID per mm_struct.

Quoted 6 lines

I think there are many ARM systems which need this model because of the way TLB shootdowns are handled in hardware, and shared with the IOMMU? So we have to use the same ASID for both MMU and IOMMU there, AIUI.

Not that I claim to be an expert on the ARM IOMMUs.

I see that on some platform the ASID <-> page table must be a 1 to 1 relationship. My experience so far on AMD is that it does not and that while the IOMMU have 16bits their GPU have 9 or 8 bits. Also given that PASID spec says that device can support different number of bits, it seems that this gonna end up being a mess with all of specific arch/device quirks.

Note that i really would like the ASID <-> mm struct 1 to 1 match but i am just fearing this is not something that can be common to all platform.

Cheers,
Jérôme

#12 lore raw
Quoted 3 lines · Jerome Glisse

Note that i really would like the ASID <-> mm struct 1 to 1 match but i am just fearing this is not something that can be common to all platform.

You are quite possibly right. And we don't *have* to force it.

We probably do need to design the core IOMMU interfaces to tolerate either.

The actual PASID allocation wants to be outside the individual IOMMU driver anyway. The main thing we need to do is let the IOMMU know if it can share PASID tables or not.

I'm now pondering a 'pasid_space' object which will contain an IDR or something for allocating PASIDs, and when one of those object is created we can also call into an IOMMU driver function for allocating a set of PASID tables (if it needs to). You then attach a given device to a pasid space.

I suppose it ends up looking at lot like the existing IOMMU domains that you can map multiple devices into.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#14 lore raw
In reply to David Woodhouse
Quoted 6 lines · David Woodhouse

In PCIe we have 20 bits of PASID. And we are going to expect hardware to implement them all, even if it can only do caching for fewer PASIDs than that.

There is also an expectation that a given MM will have the *same* PASID across all devices.

Unless you already have a TLB-coherent fabric keyed on a LPID/PID whose capacity overall is larger than your 20-bit PASID. In that case your IOMMU have remapping facilities between PASIDs and LPID/PID which could in theory provide a different mapping for devices...

Cheers,
Ben.

#26 lore raw
In reply to Arnd Bergmann
Quoted 11 lines · Arnd Bergmann
Quoted 9 lines

David Woodhouse
Jesse Barnes
Will Deacon
Paul E. McKenney
Rik van Riel
Mel Gorman
Andrea Arcangeli
Christoph Lameter
Jérôme Glisse

Add me in as well,

Confirming my interest in the subject matter.

#13 lore raw
In reply to David Woodhouse
Quoted 11 lines · David Woodhouse
Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Jerome Glisse

This is not the case with current AMD hw which IIRC only support 8bits or 9bits for PASID. Dunno if there next hardware will have more bits or not. So i need to check PCIE spec but i do not think the 20bits is a mandatory limit.

AMD hardware currently implements PASIDs with 16 bits. Given that only mm_structs which are used by offload devices get one, this should be enough to put them into a global pool and have one PASID per mm_struct.

Quoted 5 lines

I think there are many ARM systems which need this model because of the way TLB shootdowns are handled in hardware, and shared with the IOMMU? So we have to use the same ASID for both MMU and IOMMU there, AIUI.

Not that I claim to be an expert on the ARM IOMMUs.

Neither am I, cc'ing Will.

As it's the case with the ARM architecture (whether CPU or SMMU/IOMMU), many features are optional. So an implementation may or may not support handling of TLB invalidation broadcasting from the CPU. Even when it does, this is configurable, so it is not forced to share the same ASID space as the CPU.

AFAICT, the ARM SMMU uses StreamID/SubstreamID to map (1:1?) PCIe Request-ID and PASID (when stage 1 translation is supported). The StreamID/SubstreamID can then be mapped onto an ASID via stream tables. Currently the arm-smmu drivers use their own ASID space but, if we are going to support compute-offload devices, they could be made to share the same ASID as the corresponding user processes (with an additional API).

For a more generic compute-offload API, I guess we would need callbacks into the (IOMMU) drivers for page table management, including TLB invalidation and CPU ASID management events (like renewing the CPU ASID for an existing task). An implementation supporting sharing of IOMMU/CPU page tables would just have a minimal implementation of such API. We probably need a (void *)iommu_context pointer in mm_struct that drivers can point to their own context information.

Signature · 1 line
Catalin