IOMMU and graphics cards
Hi David,
as I have seen the VT-d code implements a workaround for broken graphics card drivers. What it does, when enabled, is giving each grahics device direct access to all physical memory. I really don't like to implement this but a similar workaround for the AMD IOMMU seems to be necessary. The biggest problem here is that this kind of workaround disables device isolation for graphics cards. Device isolation is the main reason for using an IOMMU in an unvirtualized environment. So if this workaround is enabled it is as good as disable the IOMMU at all.
Thats why I think a kernel compile option for this workaround is not sufficient. Distributors will probably enable this in their kernels which also disables device isolation even if the user don't want to use these broken drivers.
I think we should change that and provide a better way which allows to enable this workaround only if it is required (and it should be transparent to the user which IOMMU is built into the system). We have several options to do this:
* Implement a kernel command line option to enable/disable the workaround (what should be default?)
* Use the IOMMU-API and write a kernel module which creates a direct mapped protection domain and assigns the graphics cards to it (need to be done carefully to not break graphics drivers which do everything right and use the DMA-API)
* Any other great idea?
So what do you (and all the others reading this :-) think? I would prefer the way of implementing a module but there may also be reasons against this. I would like this to be disussed before I implement the workaround for the AMD IOMMU.
Thanks,
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 10 lines · Joerg Roedel
Hi David,
as I have seen the VT-d code implements a workaround for broken graphics card drivers. What it does, when enabled, is giving each grahics device direct access to all physical memory. I really don't like to implement this but a similar workaround for the AMD IOMMU seems to be necessary. The biggest problem here is that this kind of workaround disables device isolation for graphics cards. Device isolation is the main reason for using an IOMMU in an unvirtualized environment. So if this workaround is enabled it is as good as disable the IOMMU at all.
For that device, yes. The IOMMU still catches errors from _other_ devices, of course.
But I agree that it's a crap thing to have to do, and there are probably a number of ways we could make it slightly less crap. For a start, we could at least refrain from mapping the kernel text -- the card has no business DMAing there, whatever happens.
Quoted 4 lines
Thats why I think a kernel compile option for this workaround is not sufficient. Distributors will probably enable this in their kernels which also disables device isolation even if the user don't want to use these broken drivers.
Yes, that makes a lot of sense.
Is the reason you're doing this actually because of broken drivers? Or just because of the performance implications? I've heard both cited as reasons for the 1:1 mapping... and if it's mostly the former, then perhaps the best way for me to help you is to stop enabling the option in Fedora (rawhide, at least), so that the buggy drivers get _fixed_ and you don't get stuck with "it works on Intel, why can't you make it work?" reports?
Quoted 13 lines
I think we should change that and provide a better way which allows to enable this workaround only if it is required (and it should be transparent to the user which IOMMU is built into the system). We have several options to do this:
* Implement a kernel command line option to enable/disable the workaround (what should be default?)
* Use the IOMMU-API and write a kernel module which creates a direct mapped protection domain and assigns the graphics cards to it (need to be done carefully to not break graphics drivers which do everything right and use the DMA-API)
* Any other great idea?
Quoted 4 lines
So what do you (and all the others reading this :-) think? I would prefer the way of implementing a module but there may also be reasons against this. I would like this to be disussed before I implement the workaround for the AMD IOMMU.
I think I also prefer your second option. I don't like having this as a hack in the IOMMU code.
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 4 lines · David Woodhouse
On Tue, 2009-04-28 at 17:05 +0200, Joerg Roedel wrote:
For that device, yes. The IOMMU still catches errors from _other_ devices, of course.
Yes, it is in effect for other devices. But since its a security feature it only makes sense if it covers all devices.
Quoted 7 lines
Is the reason you're doing this actually because of broken drivers? Or just because of the performance implications? I've heard both cited as reasons for the 1:1 mapping... and if it's mostly the former, then perhaps the best way for me to help you is to stop enabling the option in Fedora (rawhide, at least), so that the buggy drivers get _fixed_ and you don't get stuck with "it works on Intel, why can't you make it work?" reports?
Currently some graphics drivers don't work with IOMMU enabled because they don't use the DMA-API. The silently assume that device addess == physical address.
I don't know how it is solved for VT-d but for AMD IOMMU the available DMA memory address space is limited per device. This is a problem for graphics card drivers because, as developers told me, they may need gigabytes of DMA memory. I am currently thinking about ways to fix that without enlarging the DMA address space for each device (which would be a huge waste of memory). But unless this problem isn't solved the drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU enabled?
Quoted 10 lines
Quoted 7 lines
* Implement a kernel command line option to enable/disable the workaround (what should be default?)
* Use the IOMMU-API and write a kernel module which creates a direct mapped protection domain and assigns the graphics cards to it (need to be done carefully to not break graphics drivers which do everything right and use the DMA-API)
* Any other great idea?I think I also prefer your second option. I don't like having this as a hack in the IOMMU code.
Ok, so I will implement such a module and send the code around for discussion.
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 8 lines · On Tue, 28 Apr 2009 18:04:48 +0200 Joerg Roedel <[email protected]> wrote:
Quoted 4 lines · David Woodhouse
On Tue, 2009-04-28 at 17:05 +0200, Joerg Roedel wrote:
For that device, yes. The IOMMU still catches errors from _other_ devices, of course.
Yes, it is in effect for other devices. But since its a security feature it only makes sense if it covers all devices.
For me, IOMMUs are not about security but accessing all memory space (there are still lots of devices that are not capable of 64-bit DMA) but I agree that this workaround is bad.
Quoted 8 lines
Quoted 7 lines
Is the reason you're doing this actually because of broken drivers? Or just because of the performance implications? I've heard both cited as reasons for the 1:1 mapping... and if it's mostly the former, then perhaps the best way for me to help you is to stop enabling the option in Fedora (rawhide, at least), so that the buggy drivers get _fixed_ and you don't get stuck with "it works on Intel, why can't you make it work?" reports?
Quoted 9 lines
Currently some graphics drivers don't work with IOMMU enabled because they don't use the DMA-API. The silently assume that device addess == physical address.
I don't know how it is solved for VT-d but for AMD IOMMU the available DMA memory address space is limited per device. This is a problem for graphics card drivers because, as developers told me, they may need gigabytes of DMA memory. I am currently thinking about ways to fix that without enlarging the DMA address space for each device (which would be a huge waste of memory).
Do you meant that AMD IOMMU code use the bitmap each device to manage address space so enlarging the DMA address space wastes memory?
Quoted 4 lines
But unless this problem isn't solved the
drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU enabled?
Looks like the DRM code uses the DMA API.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
* Implement a kernel command line option to enable/disable the workaround (what should be default?)
* Use the IOMMU-API and write a kernel module which creates a direct mapped protection domain and assigns the graphics cards to it (need to be done carefully to not break graphics drivers which do everything right and use the DMA-API)
* Any other great idea?
Quoted 5 lines
Quoted 2 lines
I think I also prefer your second option. I don't like having this as a hack in the IOMMU code.
Ok, so I will implement such a module and send the code around for discussion.
Yeah, the kernel boot option or the config option don't make sense since distributions always need to enable the workaround.
btw, what graphic drivers are broken? No chance to fix them?
Quoted 9 lines · FUJITA Tomonori
Quoted 3 lines · On Tue, 28 Apr 2009 18:04:48 +0200 Joerg Roedel <[email protected]> wrote:
Yes, it is in effect for other devices. But since its a security feature it only makes sense if it covers all devices.
For me, IOMMUs are not about security but accessing all memory space (there are still lots of devices that are not capable of 64-bit DMA) but I agree that this workaround is bad.
Yeah true. Another major reason for having an IOMMU is to enlarge the physical address range devices can access.
Quoted 2 lines
Do you meant that AMD IOMMU code use the bitmap each device to manage address space so enlarging the DMA address space wastes memory?
Its not only the bitmap. There are also the page tables that would waste memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB. Most devices don't need such a large amount of DMA address space. As I found out VT-d does not have this problem because they implemented another allocator which can cover all possible memory.
For the AMD IOMMU I currently implement a kind of a dynamically growing bitmap allocator to fix this problem.
Quoted 6 lines
Quoted 4 lines
But unless this problem isn't solved the
drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU enabled?Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 5 lines · Joerg Roedel
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I see no harm in "breaking" code which was already broken. If that's the only reason we're enabling the graphics workaround, let's turn it off.
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 9 lines · On Thu, 07 May 2009 11:22:01 +0100 David Woodhouse <[email protected]> wrote:
Quoted 5 lines · Joerg Roedel
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I see no harm in "breaking" code which was already broken. If that's the only reason we're enabling the graphics workaround, let's turn it off.
I also prefer that approach. Adding a workaround for the 'bugs' of the proprietary drivers doesn't sound a nice idea. Let them to fix the bugs. It's not difficult at all to convert a driver to use the DMA API.
Quoted 9 lines · David Woodhouse
Quoted 5 lines · Joerg Roedel
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I see no harm in "breaking" code which was already broken. If that's the only reason we're enabling the graphics workaround, let's turn it off.
Well, it is not that simple. Having no workaround at all will not result in working proprietary drivers. More likely is that distros just disable the IOMMU because their customers want to use these drivers for working 3d acceleration.
As I wrote in the original mail I think we should just make sure that these drivers still work with an IOMMU without disabling device isolation.
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 8 lines · On Thu, 7 May 2009 14:15:35 +0200 Joerg Roedel <[email protected]> wrote:
Quoted 7 lines · David Woodhouse
Quoted 5 lines · Joerg Roedel
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
Quoted 7 lines
Quoted 2 lines
I see no harm in "breaking" code which was already broken. If that's the only reason we're enabling the graphics workaround, let's turn it off.
Well, it is not that simple. Having no workaround at all will not result in working proprietary drivers. More likely is that distros just disable the IOMMU because their customers want to use these drivers for working 3d acceleration.
Have you already discussed this issue with distribution people?
I'm not sure if they would just disable the IOMMU. They want it for exotic virtualization features?
Quoted 3 lines
As I wrote in the original mail I think we should just make sure that these drivers still work with an IOMMU without disabling device isolation.
Quoted 5 lines · FUJITA Tomonori
Have you already discussed this issue with distribution people?
I'm not sure if they would just disable the IOMMU. They want it for exotic virtualization features?
Not yet. But thats true for distributions using KVM as their virtualization technology.
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 2 lines · Joerg Roedel
Well, it is not that simple. Having no workaround at all will not result in working proprietary drivers.
Which could be considered at not too bad thing anyway.
Quoted 3 lines
More likely is that distros just disable
the IOMMU because their customers want to use these drivers for working 3d acceleration.
Maybe Ubuntu :)
From: David Woodhouse <[email protected]>
Date: Thu, 07 May 2009 11:22:01 +0100
Quoted 9 lines · Joerg Roedel
Quoted 5 lines
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I see no harm in "breaking" code which was already broken. If that's the only reason we're enabling the graphics workaround, let's turn it off.
Seriously, this is an enormous price to pay just for one bad apple proprietary driver.
I can't believe the level of concessions some people are suggesting.
Let's not make IOMMU's basically useless by working around one broken driver.
Proper interfaces for what this driver is trying to do have existed in the kernel for 10+ years. There are no excuses, and it's time for Nvidia to get with the game plan.
Quoted 10 lines · David Miller
From: David Woodhouse <[email protected]>
Date: Thu, 07 May 2009 11:22:01 +0100Quoted 7 lines · Joerg Roedel
Quoted 5 lines
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
Quoted 7 lines
Quoted 2 lines
I see no harm in "breaking" code which was already broken. If that's the only reason we're enabling the graphics workaround, let's turn it off.
Seriously, this is an enormous price to pay just for one bad apple proprietary driver.
I can't believe the level of concessions some people are suggesting.
This concession is already in the VT-d code. And this concession is _enabled_ in enterprise distribtions supporting VT-d. I don't like the way it is implemented there and I definitly will not implement a similar way for the AMD IOMMU. What I try to achieve is a consistent solution for both IOMMUs. Having the broken drivers work on VT-d but not on AMD IOMMU is clearly a bad solution.
The other way to achieve consistency is to remove the workaround from VT-d code which significantly increases the chance to get the broken stuff fixed. David?
Quoted 2 lines
Let's not make IOMMU's basically useless by working around one broken driver.
Agreed.
Quoted 3 lines
Proper interfaces for what this driver is trying to do have existed in the kernel for 10+ years. There are no excuses, and it's time for Nvidia to get with the game plan.
Agreed.
Joerg
Quoted 3 lines · Joerg Roedel
The other way to achieve consistency is to remove the workaround from VT-d code which significantly increases the chance to get the broken stuff fixed. David?
It's gone from 2.6.31-rc2, but even the in-tree drivers break, so I think we have to put it back for 2.6.31 (I've sent Linus the pull request already).
I do intend to remove it again for 2.6.32 though, and I filed
http://bugzilla.kernel.org/show_bug.cgi?id=13721
For now it's got a clearer description ("Workaround broken graphics drivers") and defaults to 'n'.
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 9 lines · David Woodhouse
Quoted 3 lines · Joerg Roedel
The other way to achieve consistency is to remove the workaround from VT-d code which significantly increases the chance to get the broken stuff fixed. David?
It's gone from 2.6.31-rc2, but even the in-tree drivers break, so I think we have to put it back for 2.6.31 (I've sent Linus the pull request already).
Quoted 5 lines
I do intend to remove it again for 2.6.32 though, and I filed
http://bugzilla.kernel.org/show_bug.cgi?id=13721For now it's got a clearer description ("Workaround broken graphics drivers") and defaults to 'n'.
Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU for you? I'll may probably add a similar temporary workaround for AMD IOMMU too...
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 3 lines · Joerg Roedel
Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU for you? I'll may probably add a similar temporary workaround for AMD IOMMU too...
The Intel one definitely broke -- I don't know about the others. There are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
which make it look like _all_ AGP drivers are broken.
I wouldn't bother adding the workaround -- as I said, I'm planning to rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do so). Let's just let them fix it.
Signature · 2 lines
David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 5 lines · David Woodhouse
Quoted 3 lines · Joerg Roedel
Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU for you? I'll may probably add a similar temporary workaround for AMD IOMMU too...
Quoted 7 lines
The Intel one definitely broke -- I don't know about the others. There are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
which make it look like _all_ AGP drivers are broken.I wouldn't bother adding the workaround -- as I said, I'm planning to rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do so). Let's just let them fix it.
cc'ing Eric,
My memory of this is graphics becomes totally useless and can be 10x-50x slower. I think ripping this out without the person doing the ripping taking responsiblity for doing speed regression testing is totally insane.
I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit to supply me with any at any point in time, I'm not on the correct gravy train. So I suspect the people with the hw will have to do the work and the regression testing.
Dave.
Quoted 19 lines · Dave Airlie
Quoted 5 lines · David Woodhouse
Quoted 3 lines · Joerg Roedel
Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU for you? I'll may probably add a similar temporary workaround for AMD IOMMU too...
Quoted 7 lines
The Intel one definitely broke -- I don't know about the others. There are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
which make it look like _all_ AGP drivers are broken.I wouldn't bother adding the workaround -- as I said, I'm planning to rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do so). Let's just let them fix it.
cc'ing Eric,
Quoted 8 lines
My memory of this is graphics becomes totally useless and can be 10x-50x slower. I think ripping this out without the person doing the ripping taking responsiblity for doing speed regression testing is totally insane.
I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit to supply me with any at any point in time, I'm not on the correct gravy train. So I suspect the people with the hw will have to do the work and the regression testing.
Could you also enumerate any limitations of the IOMMUs on the amount of memory they can remap per device if any.
Dave
Quoted 6 lines
Dave.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines · Dave Airlie
Could you also enumerate any limitations of the IOMMUs on the amount of memory they can remap per device if any.
The AMD IOMMU driver in 2.6.31-rcX can map up to 4GB per domain. If the gfx card has a domain of its own it can use all of the 4GB. I am not 100% sure about VT-d, but as far as I understood the allocator it doesn't have that 4GB limitation per domain.
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 2 lines · Dave Airlie
Could you also enumerate any limitations of the IOMMUs on the amount of memory they can remap per device if any.
Here's the spec for the AND IOMMU:
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/
34434.pdf
Below is a cut-paste of "Table 3: Device Table Entry Field Definitions". As an example, paging mode 4 supports a 48-bit virtual address space (well beyond 4GB's). In terms of physical address space, "Figure 7: I/O Page Translation Entry (PTE)" shows supports for up-to a 52-bit page address.
Leo.
11:9 Mode: paging mode. Specify how the IOMMU performs page translation on behalf of the device. If
page translation is enabled, the mode specifies the depth of the device's I/O page tables (1 to 6
levels).
000b Translation disabled (Access controlled by IR and IW bits) 001b 1 Level Page Table (provides a 21-bit device virtual address space) 010b 2 Level Page Table (provides a 30-bit device virtual address space) 011b 3 Level Page Table (provides a 39-bit device virtual address space) 100b 4 Level Page Table (provides a 48-bit device virtual address space) 101b 5 Level Page Table (provides a 57-bit device virtual address space) 110b 6 Level Page Table (provides a 64-bit device virtual address space) 111b Reserved
Note: the page table root pointer is ignored when Mode=000b and when Mode=111b.
Note: Mode=111b is reported as an error when V=1 and TV=1.
Here's the spec for the AND IOMMU (with braces):
<http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs
/34434.pdf>
HTH,
Leo.
Quoted 3 lines · Leo
Here's the spec for the AND IOMMU (with braces):
<http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs
/34434.pdf>
It wasn't the lack of braces that was the problem -- it was the presence of a spurious newline in the middle.
Here's a nickel, kid -- get yourself a proper mail program. One that can cope with the 1980s invention known as WYSIWYG, perhaps?
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf
Signature · 1 line
dwmw2
Quoted 17 lines · Dave Airlie
Quoted 5 lines · David Woodhouse
Quoted 3 lines · Joerg Roedel
Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU for you? I'll may probably add a similar temporary workaround for AMD IOMMU too...
Quoted 7 lines
The Intel one definitely broke -- I don't know about the others. There are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
which make it look like _all_ AGP drivers are broken.I wouldn't bother adding the workaround -- as I said, I'm planning to rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do so). Let's just let them fix it.
Quoted 5 lines
cc'ing Eric,
My memory of this is graphics becomes totally useless and can be 10x-50x slower. I think ripping this out without the person doing the ripping taking responsiblity for doing speed regression testing is totally insane.
Are you sure that using the dma-api has such an performance impact? I've heard from other people that switching to dma-api with amd iommu had no significant performance impact.
Quoted 4 lines
I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit to supply me with any at any point in time, I'm not on the correct gravy train. So I suspect the people with the hw will have to do the work and the regression testing.
You just need to switch to dma-api. You don't need an iommu to test. Most bugs in such code can also be found and eliminated using the dma-api debugging interface.
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 8 lines · Joerg Roedel
Quoted 3 lines · Dave Airlie
My memory of this is graphics becomes totally useless and can be 10x-50x slower. I think ripping this out without the person doing the ripping taking responsiblity for doing speed regression testing is totally insane.
Are you sure that using the dma-api has such an performance impact? I've heard from other people that switching to dma-api with amd iommu had no significant performance impact.
The DMA API on its own shouldn't hurt at all. and I'm not sure about the AMD IOMMU.
It's the other IOMMU that hurts -- something about cache-incoherent page tables and having to flush the cache for every change... especially on an architecture which doesn't have any sane cache management (because it was always designed for everything to be cache coherent, dammit!) so you end up _invalidating_ the cache at the same time as you write it back!
Of course, it's a lot saner now that we don't individually flush the cache after _every_ PTE we change, but do it in batches instead. Multi-page mapping/unmapping is now a _lot_ faster than it used to be.
I'm also looking at mapping the page tables uncached, so we don't have to do the cache flushes at all.
Quoted 8 lines
Quoted 4 lines
I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit to supply me with any at any point in time, I'm not on the correct gravy train. So I suspect the people with the hw will have to do the work and the regression testing.
You just need to switch to dma-api. You don't need an iommu to test. Most bugs in such code can also be found and eliminated using the dma-api debugging interface.
Indeed. Just turn on CONFIG_DMA_API_DEBUG (that might have some performance impact, but it's not huge).
Signature · 1 line
dwmw2
Quoted 16 lines · Dave Airlie
Quoted 5 lines · David Woodhouse
Quoted 3 lines · Joerg Roedel
Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU for you? I'll may probably add a similar temporary workaround for AMD IOMMU too...
Quoted 7 lines
The Intel one definitely broke -- I don't know about the others. There are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
which make it look like _all_ AGP drivers are broken.I wouldn't bother adding the workaround -- as I said, I'm planning to rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do so). Let's just let them fix it.
Quoted 5 lines
cc'ing Eric,
My memory of this is graphics becomes totally useless and can be 10x-50x slower. I think ripping this out without the person doing the ripping taking responsiblity for doing speed regression testing is totally insane.
If the IOMMU is absent, disabled or in pass-through mode, then the performance impact should be virtually zero.
It's the case where the IOMMU is in use that matters -- and the choice there is between "slow" and "broken". I'll take "slow", please.
Having said that, I've done a bunch of performance work recently and especially multi-page mappings are a _lot_ faster than they used to be. I'll see if I can do more once I can test it with your actual usage patterns.
We currently have an evil hack which uses a shitload of memory to set up a full set of page tables to map all of physical memory, for every graphics device at boot time whether we need it or not. The AMD folks have so far resisted doing that, partly because of the amount of RAM it would take. I'm getting poked from all sides to _remove_ our hack.
Quoted 4 lines
I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit to supply me with any at any point in time, I'm not on the correct gravy train. So I suspect the people with the hw will have to do the work and the regression testing.
Anything with Cantiga chipset would suffice. Lenovo x200s, T400, etc...
Quoted 2 lines
Could you also enumerate any limitations of the IOMMUs on the amount of memory they can remap per device if any.
The Intel one basically has no relevant limit -- it's 54 bits of virtual address space or something like that.
I think I saw something about the AMD one perhaps being limited to 4GiB per device -- which makes 1:1 mapping of all memory a lot less feasible, but I'll let Jörg answer that question definitively.
Not sure about other platforms off-hand.
Signature · 1 line
dwmw2
Quoted 10 lines · On Thu, 7 May 2009 12:01:14 +0200 Joerg Roedel <[email protected]> wrote:
Quoted 2 lines
Do you meant that AMD IOMMU code use the bitmap each device to manage address space so enlarging the DMA address space wastes memory?
Its not only the bitmap. There are also the page tables that would waste memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB. Most devices don't need such a large amount of DMA address space. As I found out VT-d does not have this problem because they implemented another allocator which can cover all possible memory.
For the AMD IOMMU I currently implement a kind of a dynamically growing bitmap allocator to fix this problem.
Yeah, I know the deference. Well, I like the AMD IOMMU driver's property that dma mapping doesn't fail in OOM but I understand why you want such daynamically growing.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
But unless this problem isn't solved the
drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU enabled?Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I don't know anything about the graphic drivers but are there any other proprietary drivers except for ATI and AMD? Fixing only two drivers to make the majority happy?
Quoted 7 lines · FUJITA Tomonori
Quoted 3 lines
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I don't know anything about the graphic drivers but are there any other proprietary drivers except for ATI and AMD? Fixing only two drivers to make the majority happy?
ATI ~== AMD these days. The problem is Nvidia cards stuck into AMD systems.
I guess refusing those DMA accesses (and printing something meaningful and relentlessly honest so that the user knows where the problem comes from) is the proper solution. There will be some end-user moaning, there might be distros turning off the IOMMU code summarily, but nvidia-in-AMD-box is an unfriendly alliance to begin with.
Ingo
Quoted 7 lines · On Thu, 7 May 2009 13:01:35 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 4 lines · FUJITA Tomonori
Quoted 3 lines
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
Quoted 6 lines
Quoted 3 lines
I don't know anything about the graphic drivers but are there any other proprietary drivers except for ATI and AMD? Fixing only two drivers to make the majority happy?
ATI ~== AMD these days. The problem is Nvidia cards stuck into AMD systems.
Oops, I meant ATI and Nvidia.
Quoted 3 lines
I guess refusing those DMA accesses (and printing something meaningful and relentlessly honest so that the user knows where the problem comes from) is the proper solution.
Refusing DMA means that we will break these broken drivers. That's what David and I like. Yeah, telling users explicitly who is to blame for the problem is even better.
Quoted 4 lines
There will be some
end-user moaning, there might be distros turning off the IOMMU code summarily, but nvidia-in-AMD-box is an unfriendly alliance to begin with.
Quoted 20 lines · FUJITA Tomonori
Quoted 7 lines · On Thu, 7 May 2009 13:01:35 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 4 lines · FUJITA Tomonori
Quoted 3 lines
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
Quoted 6 lines
Quoted 3 lines
I don't know anything about the graphic drivers but are there any other proprietary drivers except for ATI and AMD? Fixing only two drivers to make the majority happy?
ATI ~== AMD these days. The problem is Nvidia cards stuck into AMD systems.
Oops, I meant ATI and Nvidia.
Quoted 7 lines
Quoted 3 lines
I guess refusing those DMA accesses (and printing something meaningful and relentlessly honest so that the user knows where the problem comes from) is the proper solution.
Refusing DMA means that we will break these broken drivers. That's what David and I like. Yeah, telling users explicitly who is to blame for the problem is even better.
If it means a non-working Xorg, then there might be nothing on the screen to report - just a seemingly hung box and people will blame the kernel. It might be better to just turn off all things IOMMU at that point, but still allow things to continue.
... which might break good drivers that relied on the IOMMU sorting out 32-bit DMA space limitations.
So it would be nice to make the failure mode somehow nicer. I.e. test how this affects the nvidia driver. Is there something on the screen to see? Could we print an URL to the Noveau driver perhaps too?
A bad solution in general is to crash/hang a significant proportion of Linux boxes. That is a foot-in-own-mouth masochistic excercise mostly, it creates a stigma for the IOMMU code, not for nvidia. So the failure mode has to be well thought out.
Ingo
Quoted 14 lines · FUJITA Tomonori
Quoted 10 lines · On Thu, 7 May 2009 12:01:14 +0200 Joerg Roedel <[email protected]> wrote:
Quoted 2 lines
Do you meant that AMD IOMMU code use the bitmap each device to manage address space so enlarging the DMA address space wastes memory?
Its not only the bitmap. There are also the page tables that would waste memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB. Most devices don't need such a large amount of DMA address space. As I found out VT-d does not have this problem because they implemented another allocator which can cover all possible memory.
For the AMD IOMMU I currently implement a kind of a dynamically growing bitmap allocator to fix this problem.
Quoted 3 lines
Yeah, I know the deference. Well, I like the AMD IOMMU driver's property that dma mapping doesn't fail in OOM but I understand why you want such daynamically growing.
Well, I will try to minimize the impact. Current plan is to preallocate bitmap and page tables for 128MB of DMA address space. Everything beyond that will be allocated on demand.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
But unless this problem isn't solved the
drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU enabled?Looks like the DRM code uses the DMA API.
Quoted 7 lines
Quoted 3 lines
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
I don't know anything about the graphic drivers but are there any other proprietary drivers except for ATI and AMD? Fixing only two drivers to make the majority happy?
At least Nvidia comes to my mind. Not sure about other graphic card vendors...
Joerg
Signature · 6 lines
| Advanced Micro Devices GmbH
Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
System |
Research | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
Center | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
| Registergericht München, HRB Nr. 43632Quoted 30 lines · Joerg Roedel
Quoted 9 lines · FUJITA Tomonori
Quoted 3 lines · On Tue, 28 Apr 2009 18:04:48 +0200 Joerg Roedel <[email protected]> wrote:
Yes, it is in effect for other devices. But since its a security feature it only makes sense if it covers all devices.
For me, IOMMUs are not about security but accessing all memory space (there are still lots of devices that are not capable of 64-bit DMA) but I agree that this workaround is bad.
Yeah true. Another major reason for having an IOMMU is to enlarge the physical address range devices can access.
Quoted 2 lines
Do you meant that AMD IOMMU code use the bitmap each device to manage address space so enlarging the DMA address space wastes memory?
Its not only the bitmap. There are also the page tables that would waste memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB. Most devices don't need such a large amount of DMA address space. As I found out VT-d does not have this problem because they implemented another allocator which can cover all possible memory.
For the AMD IOMMU I currently implement a kind of a dynamically growing bitmap allocator to fix this problem.Quoted 5 lines
Quoted 4 lines
But unless this problem isn't solved the
drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU enabled?
Quoted 5 lines
Quoted 1 line
Looks like the DRM code uses the DMA API.
The proprietary drivers make problems so far. For the ATI one I am in contact with the developers to try to fix it. But I can't do anything about the other proprietary driver I am aware of :-(
One option would be to enumerate nvidia vga device memory resources, print a nasty warning if they are accessed while not mapped by any in-kernel driver (such as Noveau) and then install a workaround for their resources and exclude them from IOMMU coverage.
Ingo