Hardware initiated paging of user process pages, hardware access to the CPU page tables of user processes

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

Following is a longer motivation and explanation on the technology presented:

Many application developers would like to be able to be able to communicate directly with the hardware from the userspace.

Use cases for that includes high performance networking API such as InfiniBand, RoCE and iWarp and interfacing with GPUs.

Currently, if the user space application wants to share system memory with the hardware device, the kernel component must pin the memory pages in RAM, using get_user_pages.

This is a hurdle, as it usually makes large portions the application memory unmovable. This pinning also makes the user space development model very complicated – one needs to register memory before using it for communication with the hardware.

We use the mmu-notifiers [1] mechanism to inform the hardware when the mapping of a page is changed. If the hardware tries to access a page which is not yet mapped for the hardware, it requests a resolution for the page address from the kernel.

This mechanism allows the hardware to access the entire address space of the user application, without pinning even a single page.

We would like to use the LSF/MM forum opportunity to discuss open issues we have for further development, such as:

-Allowing the hardware to perform page table walk, similar to get_user_pages_fast to resolve user pages that are already in RAM.

-Batching page eviction by various kernel subsystems (swapper, page-cache) to reduce the amount of communication needed with the hardware in such events

-Hinting from the hardware to the MM regarding page fetches which are speculative, similarly to prefetching done by the page-cache

-Page-in notifications from the kernel to the driver, such that we can keep our secondary TLB in sync with the kernel page table without incurring page faults.

-Allowed and banned actions while in an MMU notifier callback. We have already done some work on making the MMU notifiers sleepable [2], but there might be additional limitations, which we would like to discuss.

-Hinting from the MMU notifiers as for the reason for the notification - for example we would like to react differently if a page was moved by NUMA migration vs. page being swapped out.

[1] http://lwn.net/Articles/266320/

[2] http://comments.gmane.org/gmane.linux.kernel.mm/85002

Thanks,

--Shachar

Quoted 34 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

Following is a longer motivation and explanation on the technology presented:

Many application developers would like to be able to be able to communicate directly with the hardware from the userspace.

Use cases for that includes high performance networking API such as InfiniBand, RoCE and iWarp and interfacing with GPUs.

Currently, if the user space application wants to share system memory with the hardware device, the kernel component must pin the memory pages in RAM, using get_user_pages.

This is a hurdle, as it usually makes large portions the application memory unmovable. This pinning also makes the user space development model very complicated ? one needs to register memory before using it for communication with the hardware.

We use the mmu-notifiers [1] mechanism to inform the hardware when the mapping of a page is changed. If the hardware tries to access a page which is not yet mapped for the hardware, it requests a resolution for the page address from the kernel.

Quoted 8 lines

This mechanism allows the hardware to access the entire address space of the user application, without pinning even a single page.

We would like to use the LSF/MM forum opportunity to discuss open issues we have for further development, such as:

-Allowing the hardware to perform page table walk, similar to get_user_pages_fast to resolve user pages that are already in RAM.

Quoted 20 lines

-Batching page eviction by various kernel subsystems (swapper, page-cache) to reduce the amount of communication needed with the hardware in such events

-Hinting from the hardware to the MM regarding page fetches which are speculative, similarly to prefetching done by the page-cache

-Page-in notifications from the kernel to the driver, such that we can keep our secondary TLB in sync with the kernel page table without incurring page faults.

-Allowed and banned actions while in an MMU notifier callback. We have already done some work on making the MMU notifiers sleepable [2], but there might be additional limitations, which we would like to discuss.

-Hinting from the MMU notifiers as for the reason for the notification - for example we would like to react differently if a page was moved by NUMA migration vs. page being swapped out.

Quoted 7 lines

[1] http://lwn.net/Articles/266320/

[2] http://comments.gmane.org/gmane.linux.kernel.mm/85002

Thanks,

--Shachar

As a GPU driver developer i can say that this is something we want to do in a very near future. Also i think we would like another capabilities :

- hint to mm on memory range that are best not to evict (easier for driver to know what is hot and gonna see activities)

Dunno how big the change to the page eviction path would need to be.

Cheers,
Jerome

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=ilto:"[email protected]"> [email protected] </a>

Hi Jerome,

Quoted 51 lines · Jerome Glisse

Quoted 48 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

Following is a longer motivation and explanation on the technology presented:

Many application developers would like to be able to be able to communicate
directly with the hardware from the userspace.

Use cases for that includes high performance networking API such as InfiniBand, RoCE and iWarp and interfacing with GPUs.

Currently, if the user space application wants to share system memory with
the hardware device, the kernel component must pin the memory pages in RAM,
using get_user_pages.

This is a hurdle, as it usually makes large portions the application memory
unmovable. This pinning also makes the user space development model very complicated – one needs to register memory before using it for communication
with the hardware.

We use the mmu-notifiers [1] mechanism to inform the hardware when the mapping of a page is changed. If the hardware tries to access a page which
is not yet mapped for the hardware, it requests a resolution for the page address from the kernel.

This mechanism allows the hardware to access the entire address space of the
user application, without pinning even a single page.

We would like to use the LSF/MM forum opportunity to discuss open issues we
have for further development, such as:

Quoted 4 lines

Quoted 3 lines

-Allowing the hardware to perform page table walk, similar to get_user_pages_fast to resolve user pages that are already in RAM.

get_user_pages_fast just get page reference count instead of populate the pte to page table, correct? Then how can GPU driver use iommu to access the page?

As i said this is for pre-filling already present entry, ie pte that are present with a valid page (no special bit set). This is an optimization so that the GPU can pre-fill its tlb without having to take any mmap_sem. Hope is that in most common case this will be enough, but in some case you will have to go through the lengthy non fast gup.

Cheers,
Jerome

On 04/17/2013 12:27 AM, Jerome Glisse wrote:

[snip]

Quoted 7 lines

As i said this is for pre-filling already present entry, ie pte that are present with a valid page (no special bit set). This is an optimization so that the GPU can pre-fill its tlb without having to take any mmap_sem. Hope is that in most common case this will be enough, but in some case you will have to go through the lengthy non fast gup.

I know this. What I concern is the pte you mentioned is for normal cpu, correct? How can you pre-fill pte and tlb of GPU?

On Tue, Apr 16, 2013 at 7:50 PM, Simon Jeons <[email protected] <mailto:[email protected]>> wrote:

On 04/17/2013 12:27 AM, Jerome Glisse wrote:

[snip]

As i said this is for pre-filling already present entry, ie pte that are present with a valid page (no special bit set). This is an optimization so that the GPU can pre-fill its tlb without having to take any mmap_sem. Hope is that in most common case this will be enough, but in some case you will have to go through the lengthy non fast gup.

I know this. What I concern is the pte you mentioned is for normal cpu, correct? How can you pre-fill pte and tlb of GPU?

You getting confuse, idea is to look at cpu pte and prefill gpu pte. I do not prefill cpu pte, if a cpu pte is valid then i use the page it point to prefill the GPU pte.

So i don't pre-fill CPU PTE and TLB GPU, i pre-fill GPU PTE from CPU PTE if CPU PTE is valid. Other GPU PTE are marked as invalid and will trigger a fault that will be handle using gup that will fill CPU PTE (if fault happen at a valid address) at which point GPU PTE is updated or error is reported if fault happened at an invalid address.

Cheers,
Jerome

Hi Jerome,

On 04/17/2013 10:01 PM, Jerome Glisse wrote:

Quoted 13 lines · Simon Jeons

On 04/17/2013 12:27 AM, Jerome Glisse wrote:

[snip]

Quoted 7 lines

As i said this is for pre-filling already present entry, ie pte that are present with a valid page (no special bit set). This is an optimization so that the GPU can pre-fill its tlb without having to take any mmap_sem. Hope is that in most common case this will be enough, but in some case you will have to go through the lengthy non fast gup.

Quoted 3 lines

I know this. What I concern is the pte you mentioned is for normal cpu,
correct? How can you pre-fill pte and tlb of GPU?

You getting confuse, idea is to look at cpu pte and prefill gpu pte. I do not prefill cpu pte, if a cpu pte is valid then i use the page it point to prefill the GPU pte.

Yes, confused!

So i don't pre-fill CPU PTE and TLB GPU, i pre-fill GPU PTE from CPU PTE if CPU PTE is valid. Other GPU PTE are marked as invalid and will trigger a fault that will be handle using gup that will fill CPU PTE (if fault happen at a valid address) at which point GPU PTE is updated or error is reported if fault happened at an invalid address.

gup is used to fill CPU PTE, could you point out to me which codes will re-fill GPU PTE? gup fast?
GPU page table is different from CPU?

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space

limit. Lot of device can only address 24bit or 32bit of memory and with computer with several gbytes of memory suddenly lot of the page become unreachable to the hardware. The iommu allow to work around this by remapping those high page into address that the hardware can reach.

The hardware page fault support is a new feature of iommu designed to help the os and driver to reduce memory pinning and also share address space. Thought i am sure there are other motivations that i am not even aware off or would think off.

Btw i won't be at LSF/MM so a free good beer (or other beverage) on me to whoever takes note on this subject in next conf we run into each others.

Cheers,
Jerome

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Jerome,

Quoted 38 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 6 lines

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

Quoted 9 lines

pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space

When need remap address?

Quoted 11 lines

limit. Lot of device can only address 24bit or 32bit of memory and with computer with several gbytes of memory suddenly lot of the page become unreachable to the hardware. The iommu allow to work around this by remapping those high page into address that the hardware can reach.

The hardware page fault support is a new feature of iommu designed to help the os and driver to reduce memory pinning and also share address space. Thought i am sure there are other motivations that i am not even aware off or would think off.

Quoted 4 lines

Btw i won't be at LSF/MM so a free good beer (or other beverage) on me to whoever takes note on this subject in next conf we run into each others.

Quoted 46 lines · Simon Jeons

Hi Jerome,

Quoted 44 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

Quoted 1 line

What's the meaning of hardware pagefault?

It's a PCIE extension (well it's a combination of extension that allow that see http://www.pcisig.com/specifications/iov/ats/). Idea is that the iommu can trigger a regular pagefault inside a process address space on behalf of the hardware. The only iommu supporting that right now is the AMD iommu v2 that you find on recent AMD platform.

Quoted 5 lines

Quoted 5 lines

pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

Quoted 4 lines

Quoted 3 lines

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space

When need remap address?

Some hardware have 24bits or 32bits address limitation, iommu allow to remap memory that are above this range into the working range of the device. Just as i said below. Or are your question different ?

Cheers,
Jerome

Quoted 15 lines

Quoted 15 lines

limit. Lot of device can only address 24bit or 32bit of memory and with computer with several gbytes of memory suddenly lot of the page become unreachable to the hardware. The iommu allow to work around this by remapping those high page into address that the hardware can reach.

The hardware page fault support is a new feature of iommu designed to help the os and driver to reduce memory pinning and also share address space. Thought i am sure there are other motivations that i am not even aware off or would think off.

Btw i won't be at LSF/MM so a free good beer (or other beverage) on me to whoever takes note on this subject in next conf we run into each others.

Hi Jerome,

Quoted 48 lines · Jerome Glisse

Quoted 47 lines · Simon Jeons

Hi Jerome,

Quoted 38 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 6 lines

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

Quoted 5 lines

It's a PCIE extension (well it's a combination of extension that allow that see http://www.pcisig.com/specifications/iov/ats/). Idea is that the iommu can trigger a regular pagefault inside a process address space on behalf of the hardware. The only iommu supporting that right now is the AMD iommu v2 that you find on recent AMD platform.

Why need hardware page fault? regular page fault is trigger by cpu mmu, correct?

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space

Quoted 4 lines

Quoted 1 line

When need remap address?

Some hardware have 24bits or 32bits address limitation, iommu allow to remap memory that are above this range into the working range of the device. Just as i said below. Or are your question different ?

Oh, this method can replace bounce buffer, correct?

Quoted 55 lines · Simon Jeons

Hi Jerome,

Quoted 53 lines · Jerome Glisse

Quoted 40 lines · Simon Jeons

Hi Jerome,

Quoted 38 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 7 lines

Quoted 6 lines

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

It's a PCIE extension (well it's a combination of extension that allow that see http://www.pcisig.com/specifications/iov/ats/). Idea is that the iommu can trigger a regular pagefault inside a process address space on behalf of the hardware. The only iommu supporting that right now is the AMD iommu v2 that you find on recent AMD platform.

Quoted 2 lines

Why need hardware page fault? regular page fault is trigger by cpu mmu, correct?

Well here i abuse regular page fault term. Idea is that with hardware page fault you don't need to pin memory or take reference on page for hardware to use it. So that kernel can free as usual page that would otherwise have been

pinned. If GPU is really using them it will trigger a fault through the iommu driver that call get_user_pages (which can end up calling handle_mm_fault like a regular page fault that happened on the CPU).

One use case is GPU working on BIG dataset (think > GB buffer that can be on disk and just paged in when a chunk is needed). This is one example, but usualy GPU works on very large dataset because that's what they are good at.

Quoted 12 lines

Quoted 12 lines

Quoted 9 lines

Quoted 8 lines

pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space

When need remap address?

Some hardware have 24bits or 32bits address limitation, iommu allow to remap memory that are above this range into the working range of the device. Just as i said below. Or are your question different ?

Quoted 1 line

Oh, this method can replace bounce buffer, correct?

Yes, no bounce buffer, bounce buffer is frowned upon in GPU world because you really really really don't want to use the dma sync buffer API.

Cheers,
Jerome

Hi Jerome,

Quoted 56 lines · Jerome Glisse

Quoted 55 lines · Simon Jeons

Hi Jerome,

Quoted 53 lines · Jerome Glisse

Quoted 40 lines · Simon Jeons

Hi Jerome,

Quoted 38 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 7 lines

Quoted 6 lines

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

It's a PCIE extension (well it's a combination of extension that allow that see http://www.pcisig.com/specifications/iov/ats/). Idea is that the iommu can trigger a regular pagefault inside a process address space on behalf of the hardware. The only iommu supporting that right now is the AMD iommu v2 that you find on recent AMD platform.

Quoted 5 lines

Quoted 2 lines

Why need hardware page fault? regular page fault is trigger by cpu mmu, correct?

Well here i abuse regular page fault term. Idea is that with hardware page fault you don't need to pin memory or take reference on page for hardware to use it. So that kernel can free as usual page that would otherwise have been

For the case when GPU need to pin memory, why GPU need grap the memory of normal process instead of allocating for itself?

Quoted 4 lines

pinned. If GPU is really using them it will trigger a fault through the iommu driver that call get_user_pages (which can end up calling handle_mm_fault like a regular page fault that happened on the CPU).

This time normal process can't use this page, correct? So GPU and normal process both have their own pages?

Quoted 58 lines · Simon Jeons

Hi Jerome,

Quoted 56 lines · Jerome Glisse

Quoted 55 lines · Simon Jeons

Hi Jerome,

Quoted 53 lines · Jerome Glisse

Quoted 40 lines · Simon Jeons

Hi Jerome,

Quoted 38 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 7 lines

Quoted 6 lines

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

It's a PCIE extension (well it's a combination of extension that allow that see http://www.pcisig.com/specifications/iov/ats/). Idea is that the iommu can trigger a regular pagefault inside a process address space on behalf of the hardware. The only iommu supporting that right now is the AMD iommu v2 that you find on recent AMD platform.

Quoted 7 lines

Quoted 5 lines

Quoted 2 lines

Why need hardware page fault? regular page fault is trigger by cpu mmu, correct?

Well here i abuse regular page fault term. Idea is that with hardware page fault you don't need to pin memory or take reference on page for hardware to use it. So that kernel can free as usual page that would otherwise have been

For the case when GPU need to pin memory, why GPU need grap the memory of normal process instead of allocating for itself?

Pin memory is today world where gpu allocate its own memory (GB of memory) that disappear from kernel control ie kernel can no longer reclaim this memory it's lost memory (i had complain about that already from user than saw GB of memory vanish and couldn't understand why the GPU was using so much).

Tomorrow world we want gpu to be able to access memory that the application allocated through a simple malloc and we want the kernel to be able to recycly any page at any time because of memory pressure or because kernel decide to do so.

That's just what we want to do. To achieve so we are getting hw that can do pagefault. No change to kernel core mm code (some improvement might be made).

Quoted 6 lines

Quoted 4 lines

pinned. If GPU is really using them it will trigger a fault through the iommu driver that call get_user_pages (which can end up calling handle_mm_fault like a regular page fault that happened on the CPU).

This time normal process can't use this page, correct? So GPU and normal process both have their own pages?

No, tomorrow world, gpu and cpu both using same page in same address space at the same time. Just like two cpu core each running a different thread of the same process would. Just consider the gpu as a new cpu core working in same address space using the same memory all at the same time as cpu.

Cheers,
Jerome

Hi Jerome,

Quoted 8 lines · Jerome Glisse

On Thu, Apr 11, 2013 at 11:42:05AM +0800, Simon Jeons wrote:

Tomorrow world we want gpu to be able to access memory that the application
allocated through a simple malloc and we want the kernel to be able to recycly any page at any time because of memory pressure or because kernel decide to do so.

Quoted 4 lines

That's just what we want to do. To achieve so we are getting hw that can do
pagefault. No change to kernel core mm code (some improvement might be made).

The memory disappear since you have a reference(gup) against it, correct? Tomorrow world you want the page fault trigger through iommu driver that call get_user_pages, it also will take a reference(since gup is called), isn't it? Anyway, assume tomorrow world doesn't take a reference, we don't need care page which used by GPU is reclaimed?

Hi Jerome,

Quoted 117 lines · Jerome Glisse

Quoted 100 lines · Simon Jeons

Hi Jerome,

Quoted 90 lines · Jerome Glisse

Quoted 85 lines · Simon Jeons

Hi Jerome,

Quoted 73 lines · Jerome Glisse

Quoted 69 lines · Simon Jeons

Hi Jerome,

Quoted 58 lines · Jerome Glisse

Quoted 32 lines · Michel Lespinasse

Quoted 6 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing
memory pages from user space with the hardware, without pinning.

Quoted 7 lines

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such
technology.

I think one issue is that many MM developers are insufficiently aware
of such developments; having a technology presentation would probably
help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology.
I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what
they require of the MM layer) so people can be better prepared.

Quoted 5 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely
solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute
and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of
data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to
achieve this you want the programmer to have a single address space on
the CPU and GPU. So that the same address point to the same object on
GPU as on the CPU. This would also be a tremendous cleaner design from
driver point of view toward memory management.

Quoted 8 lines

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

Quoted 2 lines

What's the meaning of hardware pagefault?

Quoted 9 lines

It's a PCIE extension (well it's a combination of extension that allow
that see http://www.pcisig.com/**specifications/iov/ats/<http://www.pcisig.com/specifications/iov/ats/>).
Idea is that the
iommu can trigger a regular pagefault inside a process address space
on
behalf of the hardware. The only iommu supporting that right now is
the
AMD iommu v2 that you find on recent AMD platform.

Quoted 3 lines

Why need hardware page fault? regular page fault is trigger by cpu mmu, correct?

Quoted 7 lines

Well here i abuse regular page fault term. Idea is that with hardware page
fault you don't need to pin memory or take reference on page for hardware to
use it. So that kernel can free as usual page that would otherwise have been

Quoted 3 lines

For the case when GPU need to pin memory, why GPU need grap the memory of normal process instead of allocating for itself?

Pin memory is today world where gpu allocate its own memory (GB of memory) that disappear from kernel control ie kernel can no longer reclaim this memory it's lost memory (i had complain about that already from user than saw GB of memory vanish and couldn't understand why the GPU was using so much).

Tomorrow world we want gpu to be able to access memory that the application
allocated through a simple malloc and we want the kernel to be able to recycly any page at any time because of memory pressure or because kernel decide to do so.

Quoted 5 lines

That's just what we want to do. To achieve so we are getting hw that can do
pagefault. No change to kernel core mm code (some improvement might be made).

The memory disappear since you have a reference(gup) against it, correct? Tomorrow world you want the page fault trigger through iommu driver that call get_user_pages, it also will take a reference(since gup is called), isn't it? Anyway, assume tomorrow world doesn't take a reference, we don't need care page which used by GPU is reclaimed?

On Thu, Apr 11, 2013 at 9:54 PM, Simon Jeons <[email protected] <mailto:[email protected]>> wrote:

Hi Jerome,

On 04/12/2013 02:38 AM, Jerome Glisse wrote:

On Thu, Apr 11, 2013 at 11:42:05AM +0800, Simon Jeons wrote:

Hi Jerome,
On 04/11/2013 04:45 AM, Jerome Glisse wrote:

On Wed, Apr 10, 2013 at 09:41:57AM +0800, Simon Jeons wrote:

Hi Jerome,
On 04/09/2013 10:21 PM, Jerome Glisse wrote:

On Tue, Apr 09, 2013 at 04:28:09PM +0800, Simon Jeons wrote:

Hi Jerome,
On 02/10/2013 12:29 AM, Jerome Glisse wrote:

On Sat, Feb 9, 2013 at 1:05 AM, Michel Lespinasse <[email protected] <mailto:[email protected]>> wrote:

On Fri, Feb 8, 2013 at 3:18 AM, Shachar Raindel <[email protected] <mailto:[email protected]>> wrote:

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such
technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

It's a PCIE extension (well it's a combination of extension that allow
that see
http://www.pcisig.com/specifications/iov/ats/). Idea is that the
iommu can trigger a regular pagefault inside a process address space on
behalf of the hardware. The only iommu supporting that right now is the
AMD iommu v2 that you find on recent AMD platform.

Why need hardware page fault? regular page fault is trigger by cpu
mmu, correct?

Well here i abuse regular page fault term. Idea is that with hardware page
fault you don't need to pin memory or take reference on page for hardware to
use it. So that kernel can free as usual page that would otherwise have been

For the case when GPU need to pin memory, why GPU need grap the
memory of normal process instead of allocating for itself?

Pin memory is today world where gpu allocate its own memory (GB of memory)
that disappear from kernel control ie kernel can no longer reclaim this
memory it's lost memory (i had complain about that already from user than
saw GB of memory vanish and couldn't understand why the GPU was using so
much).

Tomorrow world we want gpu to be able to access memory that the application
allocated through a simple malloc and we want the kernel to be able to
recycly any page at any time because of memory pressure or because kernel
decide to do so.

That's just what we want to do. To achieve so we are getting hw that can do
pagefault. No change to kernel core mm code (some improvement might be made).

The memory disappear since you have a reference(gup) against it, correct? Tomorrow world you want the page fault trigger through iommu driver that call get_user_pages, it also will take a reference(since gup is called), isn't it? Anyway, assume tomorrow world doesn't take a reference, we don't need care page which used by GPU is reclaimed?

Right now code use gup because it's convenient but it drop the reference right after the fault. So reference is hold only for short period of time.

No you don't need to care about reclaim thanks to mmu notifier, ie before page is remove mmu notifier is call and iommu register a notifier, so it get the invalidate event and invalidate the device tlb and things goes on. If gpu access the page a new pagefault happen and a new page is allocated.

All this code is upstream in linux kernel just read it. There is just no device that use it yet.

That being said we will want improvement so that page that are hot in the device are not reclaimed. But it can work without such improvement.

Cheers,
Jerome

Hi Jerome,

On 04/12/2013 10:57 AM, Jerome Glisse wrote:

Quoted 134 lines · Simon Jeons

Hi Jerome,

Quoted 124 lines · Jerome Glisse

Quoted 106 lines · Simon Jeons

Hi Jerome,

Quoted 96 lines · Jerome Glisse

Quoted 91 lines · Simon Jeons

Hi Jerome,

Quoted 78 lines · Jerome Glisse

Quoted 70 lines · Simon Jeons

Hi Jerome,

Quoted 67 lines · Jerome Glisse

Quoted 32 lines · Michel Lespinasse

Quoted 6 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing
memory pages from user space with the hardware, without pinning.

Quoted 7 lines

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such
technology.

I think one issue is that many MM developers are insufficiently aware
of such developments; having a technology presentation would probably
help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology.
I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what
they require of the MM layer) so people can be better prepared.

Quoted 6 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely
solve this problem ? (probably a dumb question, but that just tells
you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute
and also with newer graphic program we see a massive increase in GPU
memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of
data with the cpu to the gpu buffer. The second and mostly important
to GPU compute is the use of GPU seamlessly with the CPU, in order to
achieve this you want the programmer to have a single address space on
the CPU and GPU. So that the same address point to the same object on
GPU as on the CPU. This would also be a tremendous cleaner design from
driver point of view toward memory management.

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of
the time a small window (every thing is relative the window can be

Quoted 1 line

Quoted 6 lines

Quoted 4 lines

100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to

What's the meaning of hardware pagefault?

Quoted 10 lines

It's a PCIE extension (well it's a combination of extension that allow
that see http://www.pcisig.com/specifications/iov/ats/). Idea is that the
iommu can trigger a regular pagefault inside a process address space on
behalf of the hardware. The only iommu supporting that right now is the
AMD iommu v2 that you find on recent AMD platform.

Quoted 3 lines

Why need hardware page fault? regular page fault is trigger by cpu mmu, correct?

Quoted 7 lines

Well here i abuse regular page fault term. Idea is that with hardware page
fault you don't need to pin memory or take reference on page for hardware to
use it. So that kernel can free as usual page that would otherwise have been

Quoted 3 lines

For the case when GPU need to pin memory, why GPU need grap the memory of normal process instead of allocating for itself?

Pin memory is today world where gpu allocate its own memory (GB of memory)
that disappear from kernel control ie kernel can no longer reclaim this memory it's lost memory (i had complain about that already from user than saw GB of memory vanish and couldn't understand why the GPU was using so much).

Tomorrow world we want gpu to be able to access memory that the application
allocated through a simple malloc and we want the kernel to be able to recycly any page at any time because of memory pressure or because kernel decide to do so.

Quoted 5 lines

That's just what we want to do. To achieve so we are getting hw that can do
pagefault. No change to kernel core mm code (some improvement might be made).

Quoted 7 lines

The memory disappear since you have a reference(gup) against it,
correct? Tomorrow world you want the page fault trigger through iommu driver that call get_user_pages, it also will take a reference(since gup is called), isn't it? Anyway, assume tomorrow world doesn't take a reference, we don't need care page which used by GPU is reclaimed?

Right now code use gup because it's convenient but it drop the reference right after the fault. So reference is hold only for short period of time.

Are you sure gup will drop the reference right after the fault? I redig the codes and fail verify it. Could you point out to me?

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space limit. Lot of device can only address 24bit or 32bit of memory and with computer with several gbytes of memory suddenly lot of the page become unreachable to the hardware. The iommu allow to work around this by remapping those high page into address that the hardware can reach.

The hardware page fault support is a new feature of iommu designed to help the os and driver to reduce memory pinning and also share address space. Thought i am sure there are other motivations that i am not even aware off or would think off.

Btw i won't be at LSF/MM so a free good beer (or other beverage) on me to whoever takes note on this subject in next conf we run into each others.

Cheers,
Jerome

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Jerome,

Quoted 26 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 11 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

When GPU will comsume memory?

The userspace process like mplayer will have video datas and GPU will play this datas and use memory of mplayer since these video datas load in mplayer process's address space? So GPU codes will call gup to take a reference of memory? Please correct me if my understanding is wrong. ;-)

Quoted 39 lines · Simon Jeons

Hi Jerome,

Quoted 26 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 11 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 7 lines

When GPU will comsume memory?

The userspace process like mplayer will have video datas and GPU will play this datas and use memory of mplayer since these video datas load in mplayer process's address space? So GPU codes will call gup to take a reference of memory? Please correct me if my understanding is wrong. ;-)

First target is not thing such as video decompression, however they could too benefit from it given updated driver kernel API. In case of using iommu hardware page fault we don't call get_user_pages (gup) those we don't take a reference on the page. That's the whole point of the hardware pagefault, not taking reference on the page.

Cheers,
Jerome Glisse

Hi Jerome,

Quoted 47 lines · Jerome Glisse

Quoted 39 lines · Simon Jeons

Hi Jerome,

Quoted 26 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 11 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 7 lines

When GPU will comsume memory?

The userspace process like mplayer will have video datas and GPU will play this datas and use memory of mplayer since these video datas load in mplayer process's address space? So GPU codes will call gup to take a reference of memory? Please correct me if my understanding is wrong. ;-)

Quoted 5 lines

First target is not thing such as video decompression, however they could too benefit from it given updated driver kernel API. In case of using iommu hardware page fault we don't call get_user_pages (gup) those we don't take a reference on the page. That's the whole point of the hardware pagefault, not taking reference on the page.

mplayer process is running on normal CPU or GPU? chipset_integrated graphics will use normal memory and discrete graphics will use its own memory, correct? So the memory used by discrete graphics won't need gup, correct?

Quoted 54 lines · Simon Jeons

Hi Jerome,

Quoted 47 lines · Jerome Glisse

Quoted 39 lines · Simon Jeons

Hi Jerome,

Quoted 26 lines · Jerome Glisse

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 11 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 7 lines

When GPU will comsume memory?

The userspace process like mplayer will have video datas and GPU will play this datas and use memory of mplayer since these video datas load in mplayer process's address space? So GPU codes will call gup to take a reference of memory? Please correct me if my understanding is wrong. ;-)

Quoted 5 lines

First target is not thing such as video decompression, however they could too benefit from it given updated driver kernel API. In case of using iommu hardware page fault we don't call get_user_pages (gup) those we don't take a reference on the page. That's the whole point of the hardware pagefault, not taking reference on the page.

Quoted 4 lines

mplayer process is running on normal CPU or GPU? chipset_integrated graphics will use normal memory and discrete graphics will use its own memory, correct? So the memory used by discrete graphics won't need gup, correct?

mplayer can decode video in software an only use the cpu. It can also use one of the accleration API such as VDPAU. In any case mplayer is still opening the video file allocating some memory with malloc, reading from file into this memory eventually do some preprocessing on that memory and then memcpy from this memory to memory allocated by the gpu driver.

No imagine a world where you don't have to memcpy so that the gpu can access it. Even if it's doable today it's really not something you want todo, ie gup on page and not releasing page for minutes.

There is two kind of integrated GPU, on x86 integrated GPU should be considered as discrete GPU because BIOS steal a chunk of system ram and transform it in fake vram. This stolen chunk is never ever under the control of the linux kernel (from mm pov the gpu kernel driver is in charge of it).

In any case both discrete GPU and integrated GPU have their own page table or

memory controller and they map system memory in it or video memory, sometime interleaving (at address 0x100000 64k is in vram but at address 0x10000+64k it's system memory pointing to some pages).

So right now any time we map a normal system ram page we take a reference on it so it does not goes away. We decided to not use gup because it will break several kernel assumption on anonymous memory in case of GPU. But we could use gup for short lived memory transaction like memcpy from system ram to vram (no matter if it's fake vram or real vram).

Cheers,
Jerome

Hi Jerome,

Quoted 91 lines · Jerome Glisse

Quoted 72 lines · Simon Jeons

Hi Jerome,

Quoted 61 lines · Jerome Glisse

Quoted 51 lines · Simon Jeons

Hi Jerome,

Quoted 36 lines · Jerome Glisse

Quoted 29 lines · Michel Lespinasse

Quoted 6 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing
memory pages from user space with the hardware, without pinning.

Quoted 7 lines

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such
technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology.
I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 4 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 12 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

Quoted 8 lines

When GPU will comsume memory?

The userspace process like mplayer will have video datas and GPU will play this datas and use memory of mplayer since these video datas load in mplayer process's address space? So GPU codes will call gup to take a reference of memory? Please correct me if my understanding is wrong. ;-)

Quoted 8 lines

First target is not thing such as video decompression, however they could
too benefit from it given updated driver kernel API. In case of using iommu hardware page fault we don't call get_user_pages (gup) those we don't take a reference on the page. That's the whole point of the hardware
pagefault, not taking reference on the page.

Quoted 5 lines

mplayer process is running on normal CPU or GPU? chipset_integrated graphics will use normal memory and discrete graphics will use its own memory, correct? So the memory used by discrete graphics won't need gup, correct?

mplayer can decode video in software an only use the cpu. It can also use one of the accleration API such as VDPAU. In any case mplayer is still opening
the video file allocating some memory with malloc, reading from file into this memory eventually do some preprocessing on that memory and then memcpy from this memory to memory allocated by the gpu driver.

No imagine a world where you don't have to memcpy so that the gpu can access
it. Even if it's doable today it's really not something you want todo, ie gup on page and not releasing page for minutes.

Quoted 8 lines

There is two kind of integrated GPU, on x86 integrated GPU should be considered
as discrete GPU because BIOS steal a chunk of system ram and transform it in
fake vram. This stolen chunk is never ever under the control of the linux kernel
(from mm pov the gpu kernel driver is in charge of it).

I configure integrated GPU in BIOS during system boot, it's seems that we can preallocate memory for integrated GPU, is this the memory you mentioned ?

In any case both discrete GPU and integrated GPU have their own page table

Quoted 2 lines

or

Discrete GPU will not use normal memory even if their own memory is exhaused, correct?

Quoted 22 lines · Michel Lespinasse

Quoted 10 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 3 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important

to GPU compute is the use of GPU seamlessly with the CPU, in order to achieve this you want the programmer to have a single address space on the CPU and GPU. So that the same address point to the same object on GPU as on the CPU. This would also be a tremendous cleaner design from driver point of view toward memory management.

And last, the most important, with such big buffer (>1gbytes) the memory pinning is becoming way to expensive and also drastically reduce the freedom of the mm to free page for other process. Most of the time a small window (every thing is relative the window can be > 100mbytes not so small :)) of the object will be in use by the hardware. The hardware pagefault support would avoid the necessity to pin memory and thus offer greater flexibility. At the same time the driver wants to avoid page fault as much as possible this is why i would like to be able to give hint to the mm about range of address it should avoid freeing page (swapping them out).

The iommu was designed with other goals, which were first isolate device from one another and restrict device access to allowed memory. Second allow to remap address that are above device address space limit. Lot of device can only address 24bit or 32bit of memory and with computer with several gbytes of memory suddenly lot of the page become unreachable to the hardware. The iommu allow to work around this by remapping those high page into address that the hardware can reach.

The hardware page fault support is a new feature of iommu designed to help the os and driver to reduce memory pinning and also share address space. Thought i am sure there are other motivations that i am not even aware off or would think off.

Btw i won't be at LSF/MM so a free good beer (or other beverage) on me to whoever takes note on this subject in next conf we run into each others.

Cheers,
Jerome

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Jerome,

Quoted 33 lines · Jerome Glisse

Quoted 26 lines · Michel Lespinasse

Quoted 5 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

Quoted 7 lines

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 4 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 7 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important

The pinned memory you mentioned is the memory user allocated or the memory of gpu buffer?

On Mon, Apr 15, 2013 at 4:39 AM, Simon Jeons <[email protected] <mailto:[email protected]>> wrote:

Hi Jerome,
On 02/10/2013 12:29 AM, Jerome Glisse wrote:

On Sat, Feb 9, 2013 at 1:05 AM, Michel Lespinasse <[email protected] <mailto:[email protected]>> wrote:

On Fri, Feb 8, 2013 at 3:18 AM, Shachar Raindel <[email protected] <mailto:[email protected]>> wrote:

Hi,

We would like to present a reference implementation for safely sharing
memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype
implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such
technology.

I think one issue is that many MM developers are insufficiently aware
of such developments; having a technology presentation would probably
help there; but traditionally LSF/MM sessions are more interactive
between developers who are already quite familiar with the technology.
I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what
they require of the MM layer) so people can be better prepared.

And first I'd like to ask, aren't IOMMUs supposed to already largely
solve this problem ? (probably a dumb question, but that just tells
you how much you need to explain :)

For GPU the motivation is three fold. With the advance of GPU compute
and also with newer graphic program we see a massive increase in GPU
memory consumption. We easily can reach buffer that are bigger than
1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of
data with the cpu to the gpu buffer. The second and mostly important

The pinned memory you mentioned is the memory user allocated or the memory of gpu buffer?

Memory user allocated, we don't want to pin this memory.

Cheers,
Jerome

Hi Jerome,

On 04/15/2013 11:38 PM, Jerome Glisse wrote:

Quoted 44 lines · Simon Jeons

Hi Jerome,

Quoted 33 lines · Jerome Glisse

Quoted 26 lines · Michel Lespinasse

Quoted 5 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

Quoted 7 lines

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to
assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

Quoted 4 lines

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Quoted 7 lines

For GPU the motivation is three fold. With the advance of GPU compute and also with newer graphic program we see a massive increase in GPU memory consumption. We easily can reach buffer that are bigger than 1gbytes. So the first motivation is to directly use the memory the user allocated through malloc in the GPU this avoid copying 1gbytes of data with the cpu to the gpu buffer. The second and mostly important

Quoted 3 lines

The pinned memory you mentioned is the memory user allocated or the
memory of gpu buffer?

Memory user allocated, we don't want to pin this memory.

After this idea merged, we don't need to allocate memory for integrated GPU buffer and discrete GPU don't need to have its own memory, correct?

Quoted 5 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

Quoted 5 lines

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

This sounds kinda scary TBH; however I do understand the need for such technology.

I think one issue is that many MM developers are insufficiently aware of such developments; having a technology presentation would probably help there; but traditionally LSF/MM sessions are more interactive between developers who are already quite familiar with the technology. I think it would help if you could send in advance a detailed presentation of the problem and the proposed solutions (and then what they require of the MM layer) so people can be better prepared.

And first I'd like to ask, aren't IOMMUs supposed to already largely solve this problem ? (probably a dumb question, but that just tells you how much you need to explain :)

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

Following is a longer motivation and explanation on the technology presented:

Many application developers would like to be able to be able to communicate directly with the hardware from the userspace.

Use cases for that includes high performance networking API such as InfiniBand, RoCE and iWarp and interfacing with GPUs.

Currently, if the user space application wants to share system memory with the hardware device, the kernel component must pin the memory pages in RAM, using get_user_pages.

This is a hurdle, as it usually makes large portions the application memory unmovable. This pinning also makes the user space development model very complicated ? one needs to register memory before using it for communication with the hardware.

We use the mmu-notifiers [1] mechanism to inform the hardware when the mapping of a page is changed. If the hardware tries to access a page which is not yet mapped for the hardware, it requests a resolution for the page address from the kernel.

This mechanism allows the hardware to access the entire address space of the user application, without pinning even a single page.

We would like to use the LSF/MM forum opportunity to discuss open issues we have for further development, such as:

-Allowing the hardware to perform page table walk, similar to get_user_pages_fast to resolve user pages that are already in RAM.

-Batching page eviction by various kernel subsystems (swapper,
page-cache) to reduce the amount of communication needed with the hardware in such events

-Hinting from the hardware to the MM regarding page fetches which are speculative, similarly to prefetching done by the page-cache

-Page-in notifications from the kernel to the driver, such that we can keep our secondary TLB in sync with the kernel page table without incurring page faults.

-Allowed and banned actions while in an MMU notifier callback. We have
already done some work on making the MMU notifiers sleepable [2], but there might be additional limitations, which we would like to discuss.

-Hinting from the MMU notifiers as for the reason for the notification
- for example we would like to react differently if a page was moved by NUMA migration vs. page being swapped out.

[1] http://lwn.net/Articles/266320/

[2] http://comments.gmane.org/gmane.linux.kernel.mm/85002

Thanks,

--Shachar

Signature · 4 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Simon,

Quoted 12 lines · Shachar Raindel

Hi,

We would like to present a reference implementation for safely sharing memory pages from user space with the hardware, without pinning.

We will be happy to hear the community feedback on our prototype implementation, and suggestions for future improvements.

We would also like to discuss adding features to the core MM subsystem to assist hardware access to user memory without pinning.

Following is a longer motivation and explanation on the technology presented:

Many application developers would like to be able to be able to communicate directly with the hardware from the userspace.

Quoted 7 lines

Use cases for that includes high performance networking API such as InfiniBand, RoCE and iWarp and interfacing with GPUs.

Currently, if the user space application wants to share system memory with the hardware device, the kernel component must pin the memory pages in RAM, using get_user_pages.

This is a hurdle, as it usually makes large portions the application memory unmovable. This pinning also makes the user space development model very complicated ? one needs to register memory before using it for communication with the hardware.

We use the mmu-notifiers [1] mechanism to inform the hardware when the mapping of a page is changed. If the hardware tries to access a page which is not yet mapped for the hardware, it requests a resolution for the page address from the kernel.

mmu_notifiers is used for host notice guest a page changed, is it? Why you said that it is used for informing the hardware when the mapping of a page is changed?

Quoted 29 lines

This mechanism allows the hardware to access the entire address space of the user application, without pinning even a single page.

We would like to use the LSF/MM forum opportunity to discuss open issues we have for further development, such as:

-Allowing the hardware to perform page table walk, similar to get_user_pages_fast to resolve user pages that are already in RAM.

-Batching page eviction by various kernel subsystems (swapper,
page-cache) to reduce the amount of communication needed with the hardware in such events

-Hinting from the hardware to the MM regarding page fetches which are speculative, similarly to prefetching done by the page-cache

-Page-in notifications from the kernel to the driver, such that we
can keep our secondary TLB in sync with the kernel page table without incurring page faults.

-Allowed and banned actions while in an MMU notifier callback. We
have already done some work on making the MMU notifiers sleepable [2], but there might be additional limitations, which we would like to discuss.

-Hinting from the MMU notifiers as for the reason for the
notification - for example we would like to react differently if a page was moved by NUMA migration vs. page being swapped out.

[1] http://lwn.net/Articles/266320/

[2] http://comments.gmane.org/gmane.linux.kernel.mm/85002

Thanks,

Quoted 7 lines

--Shachar

Signature · 4 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>