ehci-hcd: add option to enable 64-bit DMA support

Add a module parameter to allow the user to enable 64-bit DMA support in EHCI, which has been forcibly disabled since 2003 - see:

http://www.mail-archive.com/[email protected]/msg17230.html

At that time the comment was "it'd only matter on a few big Intel boxes anyway", however the situation is much different today when many new machines have 4GB or more of RAM and IOMMU/SWIOTLB are thus needlessly required for USB transfers. For now, the support remains disabled by default and is controlled by an allow_64bit module parameter.

Note that some USB device drivers may require updates to pass the DMA capabilities up to their higher layers to avoid unnecessary IOMMU or bounce- buffer use (i.e. networking layer NETIF_F_HIGHDMA). Some of these checks were disabled by the patch listed above, and more may be required again today. However, those previous checks were done incorrectly using dma_supported, which checks to see whether a device's DMA mask can be validly set to a given mask, not whether its previously set mask will accomodate the mask passed in.

Signed-off-by Robert Hancock <[email protected]>

Quoted 19 lines · Robert Hancock

Add a module parameter to allow the user to enable 64-bit DMA support in EHCI, which has been forcibly disabled since 2003 - see:

http://www.mail-archive.com/[email protected]/msg17230.html

At that time the comment was "it'd only matter on a few big Intel boxes anyway", however the situation is much different today when many new machines have 4GB or more of RAM and IOMMU/SWIOTLB are thus needlessly required for USB transfers. For now, the support remains disabled by default and is controlled by an allow_64bit module parameter.

Note that some USB device drivers may require updates to pass the DMA capabilities up to their higher layers to avoid unnecessary IOMMU or bounce- buffer use (i.e. networking layer NETIF_F_HIGHDMA). Some of these checks were disabled by the patch listed above, and more may be required again today. However, those previous checks were done incorrectly using dma_supported, which checks to see whether a device's DMA mask can be validly set to a given mask, not whether its previously set mask will accomodate the mask passed in.

Quoted 1 line

Signed-off-by Robert Hancock <[email protected]>

What is the "advantage" that setting this option would allow people to do that the code currently does not?  Is such an advantage measurable at the slow rates that the EHCI driver controls?

Is there any way to dynamically figure out if we can enable this or not? Adding module paramaters sucks, as they are hard to configure for most users, and they tend to be ignored.

And are you really ok with enabling this on a system-wide level, and not on a per-controller level?  Does that work properly on all systems?

And if the system does not support it, and a user enables it, who is going to support their broken system?  :)

Quoted 20 lines · Greg KH

Quoted 19 lines · Robert Hancock

Add a module parameter to allow the user to enable 64-bit DMA support in EHCI, which has been forcibly disabled since 2003 - see:

http://www.mail-archive.com/[email protected]/msg17230.html

At that time the comment was "it'd only matter on a few big Intel boxes anyway", however the situation is much different today when many new machines have 4GB or more of RAM and IOMMU/SWIOTLB are thus needlessly required for USB transfers. For now, the support remains disabled by default and is controlled by an allow_64bit module parameter.

Note that some USB device drivers may require updates to pass the DMA capabilities up to their higher layers to avoid unnecessary IOMMU or bounce- buffer use (i.e. networking layer NETIF_F_HIGHDMA). Some of these checks were disabled by the patch listed above, and more may be required again today. However, those previous checks were done incorrectly using dma_supported, which checks to see whether a device's DMA mask can be validly set to a given mask, not whether its previously set mask will accomodate the mask passed in.

Quoted 5 lines

Quoted 1 line

Signed-off-by Robert Hancock <[email protected]>

What is the "advantage" that setting this option would allow people to do that the code currently does not? ?Is such an advantage measurable at the slow rates that the EHCI driver controls?

I expect it would likely be quite system-dependent. However, particularly with devices like high-speed USB storage on systems with lots of RAM, especially 8GB or more (where more of the memory can't be addressed with 32-bit than can), I suspect it would likely be measurable in terms of CPU usage, throughput or both, especially if there's no hardware IOMMU and software bounce-buffering is required.

Quoted 3 lines

Is there any way to dynamically figure out if we can enable this or not? Adding module paramaters sucks, as they are hard to configure for most users, and they tend to be ignored.

Well, the option only has an effect if the controller indicates the 64-bit addressing feature flag in the first place. The only issue would be with potential controllers that report the feature flag but it doesn't work. Based on the libata experience with AHCI which has a similar 64-bit feature flag, there are some controllers that didn't do it properly, but not many (only ATI SB600 it appears, and only on certain boards, as it seems it was a BIOS configuration issue).

Having the ability to turn it on manually for testing would likely be the first step towards turning it on for all controllers that indicate support, not the end goal in itself.

Quoted 2 lines

And are you really ok with enabling this on a system-wide level, and not on a per-controller level? ?Does that work properly on all systems?

As above, the only issue is if the controller reports the capability but it doesn't function properly. If a user runs into the case where one of their controllers doesn't work with it enabled, that likely means that we need to add blacklisting for that device ID, etc. (or blacklist 64-bit DMA for their platform in general).

Quoted 2 lines

And if the system does not support it, and a user enables it, who is going to support their broken system? ?:)

That would be me/us :-) We need those reports though, to know how to proceed, and better to be from those who opted into testing for now..

Quoted 27 lines · Robert Hancock

Quoted 20 lines · Greg KH

Quoted 19 lines · Robert Hancock

Add a module parameter to allow the user to enable 64-bit DMA support in EHCI, which has been forcibly disabled since 2003 - see:

http://www.mail-archive.com/[email protected]/msg17230.html

At that time the comment was "it'd only matter on a few big Intel boxes anyway", however the situation is much different today when many new machines have 4GB or more of RAM and IOMMU/SWIOTLB are thus needlessly required for USB transfers. For now, the support remains disabled by default and is controlled by an allow_64bit module parameter.

Note that some USB device drivers may require updates to pass the DMA capabilities up to their higher layers to avoid unnecessary IOMMU or bounce- buffer use (i.e. networking layer NETIF_F_HIGHDMA). Some of these checks were disabled by the patch listed above, and more may be required again today. However, those previous checks were done incorrectly using dma_supported, which checks to see whether a device's DMA mask can be validly set to a given mask, not whether its previously set mask will accomodate the mask passed in.

Quoted 5 lines

Quoted 1 line

Signed-off-by Robert Hancock <[email protected]>

What is the "advantage" that setting this option would allow people to do that the code currently does not? ?Is such an advantage measurable at the slow rates that the EHCI driver controls?

Quoted 6 lines

I expect it would likely be quite system-dependent. However, particularly with devices like high-speed USB storage on systems with lots of RAM, especially 8GB or more (where more of the memory can't be addressed with 32-bit than can), I suspect it would likely be measurable in terms of CPU usage, throughput or both, especially if there's no hardware IOMMU and software bounce-buffering is required.

So you did not measure it?

Hm, I guess this change must not be necessary :)

Quoted 13 lines

Quoted 4 lines

Is there any way to dynamically figure out if we can enable this or not? Adding module paramaters sucks, as they are hard to configure for most users, and they tend to be ignored.

Well, the option only has an effect if the controller indicates the 64-bit addressing feature flag in the first place. The only issue would be with potential controllers that report the feature flag but it doesn't work. Based on the libata experience with AHCI which has a similar 64-bit feature flag, there are some controllers that didn't do it properly, but not many (only ATI SB600 it appears, and only on certain boards, as it seems it was a BIOS configuration issue).

Quoted 3 lines

Having the ability to turn it on manually for testing would likely be the first step towards turning it on for all controllers that indicate support, not the end goal in itself.

But we disabled it on purpose, because of problems, do we want those problems again?

Quoted 4 lines

Quoted 3 lines

And are you really ok with enabling this on a system-wide level, and not on a per-controller level? ?Does that work properly on all systems?

Quoted 5 lines

As above, the only issue is if the controller reports the capability but it doesn't function properly. If a user runs into the case where one of their controllers doesn't work with it enabled, that likely means that we need to add blacklisting for that device ID, etc. (or blacklist 64-bit DMA for their platform in general).

blacklists are hard to keep up to date, you are always handling bug reports after-the-fact.  That's not something you should add lightly.

Quoted 4 lines

Quoted 3 lines

And if the system does not support it, and a user enables it, who is going to support their broken system? ?:)

Quoted 2 lines

That would be me/us :-) We need those reports though, to know how to proceed, and better to be from those who opted into testing for now..

But we need a _reason_ to enable it.

What user needs this?

And if there is no speed benifit, why take the risk?

Quoted 35 lines · Greg KH

Quoted 33 lines · Robert Hancock

Quoted 20 lines · Greg KH

Quoted 19 lines · Robert Hancock

Add a module parameter to allow the user to enable 64-bit DMA support in EHCI, which has been forcibly disabled since 2003 - see:

http://www.mail-archive.com/[email protected]/msg17230.html

At that time the comment was "it'd only matter on a few big Intel boxes anyway", however the situation is much different today when many new machines have 4GB or more of RAM and IOMMU/SWIOTLB are thus needlessly required for USB transfers. For now, the support remains disabled by default and is controlled by an allow_64bit module parameter.

Note that some USB device drivers may require updates to pass the DMA capabilities up to their higher layers to avoid unnecessary IOMMU or bounce- buffer use (i.e. networking layer NETIF_F_HIGHDMA). Some of these checks were disabled by the patch listed above, and more may be required again today. However, those previous checks were done incorrectly using dma_supported, which checks to see whether a device's DMA mask can be validly set to a given mask, not whether its previously set mask will accomodate the mask passed in.

Quoted 5 lines

Quoted 1 line

Signed-off-by Robert Hancock <[email protected]>

What is the "advantage" that setting this option would allow people to do that the code currently does not? ?Is such an advantage measurable at the slow rates that the EHCI driver controls?

I expect it would likely be quite system-dependent. However, particularly with devices like high-speed USB storage on systems with lots of RAM, especially 8GB or more (where more of the memory can't be addressed with 32-bit than can), I suspect it would likely be measurable in terms of CPU usage, throughput or both, especially if there's no hardware IOMMU and software bounce-buffering is required.

Quoted 3 lines

So you did not measure it?

Hm, I guess this change must not be necessary :)

I'll try and run some tests and see what I can quantify. However, I only have 4GB of RAM on my machine (with a 1GB memory hole) and so a random memory allocation only has a 25% chance of ending up in the area where it would make a difference, so it may take a bit of doing.

Quoted 8 lines

Quoted 3 lines

So you did not measure it?

Hm, I guess this change must not be necessary :)

I'll try and run some tests and see what I can quantify. However, I only have 4GB of RAM on my machine (with a 1GB memory hole) and so a random memory allocation only has a 25% chance of ending up in the area where it would make a difference, so it may take a bit of doing.

Without any good justification, including real tests being run, I can't take this patch, the risk is just too high.

And really, for USB 2.0 speeds, I doubt you are going to even notice this kind of overhead, it's in the noise.  Especially given that almost always the limiting factor is the device itself, not the host.

Quoted 9 lines · Greg KH

Quoted 8 lines

Quoted 3 lines

So you did not measure it?

Hm, I guess this change must not be necessary :)

I'll try and run some tests and see what I can quantify. However, I only have 4GB of RAM on my machine (with a 1GB memory hole) and so a random memory allocation only has a 25% chance of ending up in the area where it would make a difference, so it may take a bit of doing.

Quoted 2 lines

Without any good justification, including real tests being run, I can't take this patch, the risk is just too high.

Again, this particular patch has essentially zero risk for anyone that doesn't choose to experiment with the option. One can hardly say it presents much of a long-term maintenance burden either..

Quoted 3 lines

And really, for USB 2.0 speeds, I doubt you are going to even notice this kind of overhead, it's in the noise.  Especially given that almost always the limiting factor is the device itself, not the host.

Well, I do have some results. This is from running this "dd if=/dev/sdg of=/dev/null bs=3800M iflag=direct" against an OCZ Rally2 USB flash drive, which gets about 30 MB/sec on read, with CPU-burning tasks on all cores in the background. (The huge block size and iflag=direct is to try to force more of the IO to happen to memory above the 4GB mark.) With that workload, swiotlb_bounce shows up as between 1.5 to 4% of the CPU time spent in the kernel according to oprofile. Obviously with the 64-bit DMA enabled, that disappears. Of course, the overall kernel time is only around 2% of the total time, so that's a pretty small overall percentage.

I'll try some tests later with a faster SATA-to-IDE device that should stress things a bit more, but a huge difference doesn't seem likely. One thing that's uncertain is just how much of the IO is needing to be bounced - an even distribution of the buffer across all of physical RAM would suggest 25% in this case, but I don't know an easy way to verify that.

Aside from speed considerations though, I should point out another
factor: IOMMU/SWIOTLB space is in many cases a limited resource for all IO in flight at a particular time (SWIOTLB is typically 64MB). The number of hits when Googling for "Out of IOMMU space" indicates it is a problem that people do hit from time to time. From that perspective, anything that prevents unnecessary use of bounce buffers is a good thing.

Quoted 13 lines · Robert Hancock

Quoted 11 lines · Greg KH

Quoted 8 lines

Quoted 3 lines

So you did not measure it?

Hm, I guess this change must not be necessary :)

I'll try and run some tests and see what I can quantify. However, I only have 4GB of RAM on my machine (with a 1GB memory hole) and so a random memory allocation only has a 25% chance of ending up in the area where it would make a difference, so it may take a bit of doing.

Without any good justification, including real tests being run, I can't take this patch, the risk is just too high.

Quoted 3 lines

Again, this particular patch has essentially zero risk for anyone that doesn't choose to experiment with the option. One can hardly say it presents much of a long-term maintenance burden either..

Then don't give them the option, as it doesn't seem needed :)

Again, it is tough to remove options once you add them, so not adding them at all is the best thing to do.

Quoted 5 lines

Quoted 4 lines

And really, for USB 2.0 speeds, I doubt you are going to even notice this kind of overhead, it's in the noise.  Especially given that almost always the limiting factor is the device itself, not the host.

Quoted 10 lines

Well, I do have some results. This is from running this "dd if=/dev/sdg of=/dev/null bs=3800M iflag=direct" against an OCZ Rally2 USB flash drive, which gets about 30 MB/sec on read, with CPU-burning tasks on all cores in the background. (The huge block size and iflag=direct is to try to force more of the IO to happen to memory above the 4GB mark.) With that workload, swiotlb_bounce shows up as between 1.5 to 4% of the CPU time spent in the kernel according to oprofile. Obviously with the 64-bit DMA enabled, that disappears. Of course, the overall kernel time is only around 2% of the total time, so that's a pretty small overall percentage.

2% is noise, right?  So overall you have not really shown any improvement.

Quoted 8 lines

I'll try some tests later with a faster SATA-to-IDE device that should stress things a bit more, but a huge difference doesn't seem likely. One thing that's uncertain is just how much of the IO is needing to be bounced - an even distribution of the buffer across all of physical RAM would suggest 25% in this case, but I don't know an easy way to verify that.

Quoted 7 lines

Aside from speed considerations though, I should point out another
factor: IOMMU/SWIOTLB space is in many cases a limited resource for all IO in flight at a particular time (SWIOTLB is typically 64MB). The number of hits when Googling for "Out of IOMMU space" indicates it is a problem that people do hit from time to time. From that perspective, anything that prevents unnecessary use of bounce buffers is a good thing.

Sure, but again, for USB 2.0 stuff, we don't have many I/O in flight, as they are pretty slow devices.

USB 3.0 is different, and that's a different driver, and hopefully that is all addressed already :)

Quoted 14 lines · Greg KH

Quoted 13 lines · Robert Hancock

Quoted 11 lines · Greg KH

Quoted 8 lines

Quoted 3 lines

So you did not measure it?

Hm, I guess this change must not be necessary :)

I'll try and run some tests and see what I can quantify. However, I only have 4GB of RAM on my machine (with a 1GB memory hole) and so a random memory allocation only has a 25% chance of ending up in the area where it would make a difference, so it may take a bit of doing.

Without any good justification, including real tests being run, I can't take this patch, the risk is just too high.

Quoted 8 lines

Quoted 3 lines

Again, this particular patch has essentially zero risk for anyone that doesn't choose to experiment with the option. One can hardly say it presents much of a long-term maintenance burden either..

Then don't give them the option, as it doesn't seem needed :)

Again, it is tough to remove options once you add them, so not adding them at all is the best thing to do.

I don't know why you would remove the option. Even if you someday changed the default to 1, it would likely be a still idea to keep it around for debugging purposes at least.

If you're complaining about options, ehci-hcd already has some which are quite a bit more nebulous in usefulness than this one..

Quoted 15 lines

Quoted 14 lines

Quoted 3 lines

And really, for USB 2.0 speeds, I doubt you are going to even notice this kind of overhead, it's in the noise. ?Especially given that almost always the limiting factor is the device itself, not the host.

Well, I do have some results. This is from running this "dd if=/dev/sdg of=/dev/null bs=3800M iflag=direct" against an OCZ Rally2 USB flash drive, which gets about 30 MB/sec on read, with CPU-burning tasks on all cores in the background. (The huge block size and iflag=direct is to try to force more of the IO to happen to memory above the 4GB mark.) With that workload, swiotlb_bounce shows up as between 1.5 to 4% of the CPU time spent in the kernel according to oprofile. Obviously with the 64-bit DMA enabled, that disappears. Of course, the overall kernel time is only around 2% of the total time, so that's a pretty small overall percentage.

Quoted 2 lines

2% is noise, right? ?So overall you have not really shown any improvement.

What threshold of performance improvement would you rather see? It's pretty clear that there will be a performance upside, even if small, and no downside.

I honestly didn't expect as much resistance to a simple hardware feature enablement patch, that has zero impact on anyone that doesn't opt-in..

Quoted 15 lines

Quoted 14 lines

I'll try some tests later with a faster SATA-to-IDE device that should stress things a bit more, but a huge difference doesn't seem likely. One thing that's uncertain is just how much of the IO is needing to be bounced - an even distribution of the buffer across all of physical RAM would suggest 25% in this case, but I don't know an easy way to verify that.

Aside from speed considerations though, I should point out another
factor: IOMMU/SWIOTLB space is in many cases a limited resource for all IO in flight at a particular time (SWIOTLB is typically 64MB). The number of hits when Googling for "Out of IOMMU space" indicates it is a problem that people do hit from time to time. From that perspective, anything that prevents unnecessary use of bounce buffers is a good thing.

Quoted 2 lines

Sure, but again, for USB 2.0 stuff, we don't have many I/O in flight, as they are pretty slow devices.

Think that's a bit simplistic, if you have multiple devices active at once, or multiple controllers (not at all uncommon these days, newer Intel chipset machines have two EHCI controllers, with USB 1.x devices handled through a logical hub with TT connected to each of them) that can chew up more space.

Quoted 2 lines

USB 3.0 is different, and that's a different driver, and hopefully that is all addressed already :)

Doesn't look like it, from the version in current -git anyway - I don't see any calls to set DMA masks in the XHCI code so it will just default to 32-bit. I imagine that'll hurt performance at 4.8 Gbps if you've got lots of RAM..

Quoted 3 lines

But we disabled it on purpose, because of problems, do we want those problems again?

AFAICS, it was disabled because of problems with kernel code, not with hardware (and it appears the issue was with the code that detected the expanded DMA mask in the USB device driver code, not the HCD driver). CCing David Brownell who may know more.

Quoted 8 lines · Robert Hancock

Quoted 3 lines

But we disabled it on purpose, because of problems, do we want those problems again?

AFAICS, it was disabled because of problems with kernel code, not with hardware (and it appears the issue was with the code that detected the expanded DMA mask in the USB device driver code, not the HCD driver). CCing David Brownell who may know more.

That's a good summary of the high points.  Testing was potentially an issue, but it never quite got that far.  So I have no idea if there are systems where EHCI advertises 64-bit DMA support but that support is broken (e.g. "Yet Another Hardware Mechanism MS-Windows Ignores", so that only Linux would ever trip over the relevant BIOS breakage).

I won't attempt to go into details, but I recall a few basic issues:

 * Not all clients or implementors of the "dma mask" mechanism agreed on what it was supposed to achieve.  Few, for example, really used it as a mask ... and it rarely affects allocation of buffers that will later get used for DMA.

 * Confusing semantics for the various types of DMA restriction which hardware may impose, and upper layers in driver stacks would thus need (in some cases) to cope with.

 * How to pass such restrictions up the driver stack ... as for example that NETIF_* flag.  ISTR there was some block layer issue too, but at this remove I can't remember any details at all.  (If networking and the block layer can use 64-bit DMA, I can't imagine many other subsystems would deliver wins as big.)  For example, how would one pass up the knowledge that a driver for a particular USB peripheral across a few hubs) can do DMA to/from address 0x1234567890abcdef, but the same driver can't do that for an otherwise identical peripheral connected through a different HCD?

 * There were probably a few PCI-specific issues too.  I don't think at that time there were many users of 64-bit DMA which weren't specific to PCI.  Wanting to use the generic DMA calls for such stuff wasn't really done back then.  But ... the USB stack uses the generic calls, and drivers sitting on top of usbcore (and its tightly coupled HCDs) will never issue PCI-specific calls, since they need to work on systems that don't even have PCI.

I basically think that if the controller can do 64-bit DMA, it should be enabling it by default ... assuming the software stack can handle that.  (What would be the benefit of adding needless restrictions, and making systems needlessly apply bounce buffering.)  So while I'd like to see the 64-bit DMA working, it should IMO be done without any options to cause trouble/confusion.

But at that time it wasn't straightforward to manage 64-bit DMA except in the very lowest level PCI drivers.  That is, EHCI could do it ... but driver layers on top of it had no good way to do their part.  For example, when they manage DMA mappings themselves.)

- Dave

Quoted 5 lines

That's a good summary of the high points.  Testing was potentially an issue, but it never quite got that far.  So I have no idea if there are systems where EHCI advertises 64-bit DMA support but that support is broken (e.g. "Yet Another Hardware Mechanism MS-Windows Ignores", so that only Linux would ever trip over the relevant BIOS breakage).

According to one Microsoft page I saw, Windows XP did not implement the 64-bit addressing feature in EHCI. I haven't found any information on whether any newer Windows versions do or not.

Quoted 10 lines

I won't attempt to go into details, but I recall a few basic issues:

 * Not all clients or implementors of the "dma mask" mechanism agreed on what it was supposed to achieve.  Few, for example, really used it as a mask ... and it rarely affects allocation of buffers that will later get used for DMA.

 * Confusing semantics for the various types of DMA restriction which hardware may impose, and upper layers in driver stacks would thus need (in some cases) to cope with.

I think this is pretty well nailed down at this point. I suspect the confusion partly comes from the expectation that driver code should be able to use dma_supported to test a particular mask against what a device had been configured for. This function is really meant for use in arch code, not for drivers.

Quoted 9 lines

 * How to pass such restrictions up the driver stack ... as for example that NETIF_* flag.  ISTR there was some block layer issue too, but at this remove I can't remember any details at all.  (If networking and the block layer can use 64-bit DMA, I can't imagine many other subsystems would deliver wins as big.)  For example, how would one pass up the knowledge that a driver for a particular USB peripheral across a few hubs) can do DMA to/from address 0x1234567890abcdef, but the same driver can't do that for an otherwise identical peripheral connected through a different HCD?

I think this logic is also in place, for the most part. The DMA mask from the HCD appears to be propagated into the USB device, and then into the interface objects.

For usb-storage, the SCSI layer
automatically sets the bounce limit based on the device passed into it, so the right thing gets done. The networking layer seems like it would need explicit handling in the drivers - I think basically a check if the device interface's DMA mask was set to DMA_BIT_MASK(64) and if so, set the HIGHDMA flag.

Quoted 9 lines · Robert Hancock

Quoted 5 lines

That's a good summary of the high points.  Testing was potentially an issue, but it never quite got that far.  So I have no idea if there are systems where EHCI advertises 64-bit DMA support but that support is broken (e.g. "Yet Another Hardware Mechanism MS-Windows Ignores", so that only Linux would ever trip over the relevant BIOS breakage).

According to one Microsoft page I saw, Windows XP did not implement the 64-bit addressing feature in EHCI. I haven't found any information on whether any newer Windows versions do or not.

Note that it's pure speculation on my part whether or not any such BIOS setup is needed.  One would hope it wouldn't be required ... but then engineers have been known to create all sorts of options that require tweaking ... and trigger errors when the options aren't stroked in the right way.

Quoted 11 lines

Quoted 10 lines

I won't attempt to go into details, but I recall a few basic issues:

 * Not all clients or implementors of the "dma mask" mechanism agreed on what it was supposed to achieve.  Few, for example, really used it as a mask ... and it rarely affects allocation of buffers that will later get used for DMA.

 * Confusing semantics for the various types of DMA restriction which hardware may impose, and upper layers in driver stacks would thus need (in some cases) to cope with.

Quoted 5 lines

I think this is pretty well nailed down at this point. I suspect the confusion partly comes from the expectation that driver code should be able to use dma_supported to test a particular mask against what a device had been configured for. This function is really meant for use in arch code, not for drivers.

If so, that suggests a minor hole in the DMA interface, since drivers do need such info.

As you note, mask manipulation can be done in drivers ... but on the flip side, such things are a bit error prone and deserve API wrappers.  (Plus, there's the whole confusion about whether it's really a "mask", where a given bit flags whether that address line is valid.  Seems more like using a bitstring of N ones as a representation of N, where only N matters.)

Quoted 12 lines

Quoted 11 lines

 * How to pass such restrictions up the driver stack ... as for example that NETIF_* flag.  ISTR there was some block layer issue too, but at this remove I can't remember any details at all.  (If networking and the block layer can use 64-bit DMA, I can't imagine many other subsystems would deliver wins as big.)  For example, how would one pass up the knowledge that a driver for a particular USB peripheral across a few hubs) can do DMA to/from address 0x1234567890abcdef, but the same driver can't do that for an otherwise identical peripheral connected through a different HCD?

Quoted 3 lines

I think this logic is also in place, for the most part. The DMA mask from the HCD appears to be propagated into the USB device, and then into the interface objects.

Yeah, I recall thinking about that stuff way back when... intended to set that up correctly.  IT was at least partially tested.

Quoted 6 lines

For usb-storage, the SCSI layer
automatically sets the bounce limit based on the device passed into it, so the right thing gets done. The networking layer seems like it would need explicit handling in the drivers - I think basically a check if the device interface's DMA mask was set to DMA_BIT_MASK(64) and if so, set the HIGHDMA flag.

Another example of how roundabout all that stuff is.  "64" being the relevant number, in contrast to something less.  So if for example the DMA address bus width is 48 bits, things will be strange.

I wonder why the two layers don't adopt the same approach ... seemingly they're making different assumptions about driver behavior, suggesting that one of them may well be overly optimistic.

Add a module parameter to allow the user to enable 64-bit DMA support in EHCI, which has been forcibly disabled since 2003 - see:

http://www.mail-archive.com/[email protected]/msg17230.html

At that time the comment was "it'd only matter on a few big Intel boxes anyway",

According to one Microsoft page I saw, Windows XP did not implement the 64-bit addressing feature in EHCI. I haven't found any information on whether any newer Windows versions do or not.

Quoted 4 lines

According to one Microsoft page I saw, Windows XP did not implement the 64-bit addressing feature in EHCI. I haven't found any information on whether any newer Windows versions do or not.

Windows 7 seems to support it, and it seems like they have to already do a workaround for broken NVIDIA chipsets:
http://support.microsoft.com/?kbid=976972
http://www.windows7tipsonline.com/USB_Problem.htm
http://social.technet.microsoft.com/Forums/en-US/w7itprohardware/thread/3aae3b66-6a1a-47e8-ad1b-b20b68eaecf8#79ea3219-d76e-40bc-b910-c7d347002e66
http://social.technet.microsoft.com/Forums/en-US/w7itprohardware/thread/38f25c1d-de67-4c74-8845-2cd3a15d8e41

it doesn't work. Based on the libata experience with AHCI which has a similar 64-bit feature flag, there are some controllers that didn't do it properly, but not many (only ATI SB600 it appears, and only on certain boards, as it seems it was a BIOS configuration issue).

Quoted 5 lines

it doesn't work. Based on the libata experience with AHCI which has a similar 64-bit feature flag, there are some controllers that didn't do it properly, but not many (only ATI SB600 it appears, and only on certain boards, as it seems it was a BIOS configuration issue).

It seems it was solved by a DMI whitelist, but if it is a BIOS configuration issue, I think it would be better if it could be detected directly instead by reading the configuration directly from the device and correcting it if possible. Is that possible?

That would likely be the ideal solution, but only AMD would likely have the info needed to do that, if it's possible..

Quoted 1 line

Quoted 1 line

it doesn't work. Based on the libata experience with AHCI which has

Quoted 1 line

Quoted 1 line

similar 64-bit feature flag, there are some controllers that didn't

Quoted 3 lines

Quoted 2 lines

it properly, but not many (only ATI SB600 it appears, and only on certain boards, as it seems it was a BIOS configuration issue).

It seems it was solved by a DMI whitelist, but if it is a BIOS

instead by reading the configuration directly from the device and

That would likely be the ideal solution, but only AMD would likely have the info needed to do that, if it's possible..

The fact that Windows 7 is now using the feature also means that there aren't likely to be too many machines where the 64-bit addressing is reported but doesn't work. Which means that aside from the NVIDIA quirk, I think enabling 64-bit addressing should be relatively safe.

gained. There just isn't much point in trying to enable a shiny new feature on an aging platform or technology.

Quoted 3 lines

gained. There just isn't much point in trying to enable a shiny new feature on an aging platform or technology.

You mean we should focus on enabling 64-bit DMA for USB 3.0 (xHCI)?

(following thread from the ATA side) I'm with Greg on this. USB2.0 is quite slow on today's standard and most 64bit machines now have IOMMU of some kind anyway.

Quoted 3 lines

(following thread from the ATA side) I'm with Greg on this. USB2.0 is quite slow on today's standard and most 64bit machines now have IOMMU of some kind anyway.

Except that unfortunately, Intel with Lynnfield and Clarkdale decided to do market segmentation on the VT-d feature.
This could be done because both have an integrated northbridge. With Lynnfield, the very common Core i5 750 CPU do not support VT-d, but the Core i7 800 series does.
With Clarkdale, the very common Core i3 500 series do not support VT-d, but the Core i5 600 series does, except the 661.
Not to mention that in the Core 2 generation, VT-d was available only on Q35/Q45/X38/X48 chipsets.

Arghhh.... I hate when intel pulls this type of 'product differentiation' stunts. At any rate, I'm doubtful swiotlb'ing for usb2.0 would be noticeable at all.

Am Donnerstag, 25. Februar 2010 04:29:12 schrieb Tejun Heo:

Quoted 3 lines

Arghhh.... I hate when intel pulls this type of 'product differentiation' stunts. At any rate, I'm doubtful swiotlb'ing for usb2.0 would be noticeable at all.

That can and should be tested.

Quoted 6 lines · Oliver Neukum

Am Donnerstag, 25. Februar 2010 04:29:12 schrieb Tejun Heo:

Quoted 3 lines

Arghhh.... I hate when intel pulls this type of 'product differentiation' stunts. At any rate, I'm doubtful swiotlb'ing for usb2.0 would be noticeable at all.

That can and should be tested.

Robert did already. Given that USB2.0 devices aren't usually used for high-perf storage subsystem, the benefit looks marginal to me but everyone is entitled to own opinion.

Quoted 6 lines · Tejun Heo

Quoted 5 lines · Oliver Neukum

Am Donnerstag, 25. Februar 2010 04:29:12 schrieb Tejun Heo:

Quoted 3 lines

Arghhh.... I hate when intel pulls this type of 'product differentiation' stunts.  At any rate, I'm doubtful swiotlb'ing for usb2.0 would be noticeable at all.

Quoted 5 lines

Quoted 1 line

That can and should be tested.

Robert did already.  Given that USB2.0 devices aren't usually used for high-perf storage subsystem, the benefit looks marginal to me but everyone is entitled to own opinion.

For now, I'd like to not enable this, given the huge risks and no known benifit.

For USB 3.0, we can reconsider it, as it might start to make a difference there.

Quoted 2 lines

For now, I'd like to not enable this, given the huge risks and no known benifit.

"Huge risks" is likely a bit melodramatic

And yes, we can fix things after-the-fact with blacklists, but as Tejun so aptly described, it is a major pain to do so, and lots of users just get frustrated and never tell us that problems happen.