Proposal: Eliminate GFP_DMA
A message that is not in the archive (sent off-list or not yet received): 20030228064631.G23865@parcelfarce.linux.theplanet.co.uk.suse.lists.linux.kernel
Quoted 6 lines · Matthew Wilcox
(quoted diff) +2
-1- +#define GFP_ATOMIC_DMA (GFP_ATOMIC | __GFP_DMA) +#define GFP_KERNEL_DMA (GFP_KERNEL | __GFP_DMA)combined with changing some users to use
__GFP_DMAif they really do mean the bitmask. Comments?
Sounds like a good 2.7.x early project. Currently we still have too much driver breakage for the next release to break even more right now.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Matthew Wilcox
(quoted diff) +2
-1- +#define GFP_ATOMIC_DMA (GFP_ATOMIC | __GFP_DMA) +#define GFP_KERNEL_DMA (GFP_KERNEL | __GFP_DMA)combined with changing some users to use
__GFP_DMAif they really do mean the bitmask. Comments?Sounds like a good 2.7.x early project. Currently we still have too much driver breakage for the next release to break even more right now.
i'm not the kind of person who just changes the header file and breaks all the drivers. plan:
- Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
- Change the drivers
- Delete the GFP_DMA definition
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 6 lines · Matthew Wilcox
i'm not the kind of person who just changes the header file and breaks all the drivers. plan:
- Add the
GFP_ATOMIC_DMA&GFP_KERNEL_DMAdefinitions
- Change the drivers
- Delete theGFP_DMAdefinition
Needless pain for people maintaining cross release drivers. Save it for 2.7 where we should finally do the honourable deed given x86-64 may well be mainstream, and simply remove GFP_DMA and expect people to use pci_*
Quoted 4 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
i'm not the kind of person who just changes the header file and breaks all the drivers. plan:
Quoted 8 lines
Quoted 3 lines
- Add the
GFP_ATOMIC_DMA&GFP_KERNEL_DMAdefinitions
- Change the drivers
- Delete theGFP_DMAdefinitionNeedless pain for people maintaining cross release drivers. Save it for 2.7 where we should finally do the honourable deed given x86-64 may well be mainstream, and simply remove
GFP_DMAand expect people to use pci_*
umm. are you volunteering to convert drivers/net/macmace.c to the pci_* API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 3 lines · Matthew Wilcox
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.
The ia64 is a fine example of how broken it is. People have to hack around with GFP_DMA meaning different things on ia64 to everything else. It needs to die.
Alan
Quoted 8 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.The ia64 is a fine example of how broken it is. People have to hack around with
GFP_DMAmeaning different things on ia64 to everything else. It needs to die.
At least on x86-64 it is still needed when you need have some hardware with address limits < 4GB (e.g. an 24bit soundcard)
pci_* on K8 only allows address mask 0xffffffff or unlimited.
-Andi
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.
Quoted 8 lines
Quoted 3 lines
The ia64 is a fine example of how broken it is. People have to hack around with
GFP_DMAmeaning different things on ia64 to everything else. It needs to die.At least on x86-64 it is still needed when you need have some hardware with address limits < 4GB (e.g. an 24bit soundcard)
pci_* on K8 only allows address mask
0xffffffffor unlimited.
That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit soundcard. pci_set_dma_mask should support 24-bits accordingly, otherwise it's a bug in your platform implementation... Nobody will be able to use certain properly-written drivers on your platform otherwise.
Jeff
Quoted 4 lines
That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit soundcard.
pci_set_dma_maskshould support 24-bits accordingly, otherwise it's a bug in your platform implementation... Nobody will be able to use certain properly-written drivers on your platform otherwise.
It supports it for pci_alloc_consistent (by mapping it to an GFP_DMA allocation)
Just for pci_map_single/pci_map_sg() it is not supported and will error out when the passed address doesn't fit the mask. That behaviour is 100% compatible with IA32 which does the same.
-Andi
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.The ia64 is a fine example of how broken it is. People have to hack around with
GFP_DMAmeaning different things on ia64 to everything else. It needs to die.At least on x86-64 it is still needed when you need have some hardware with address limits < 4GB (e.g. an 24bit soundcard)
Quoted 6 lines
Quoted 1 line
pci_* on K8 only allows address mask
0xffffffffor unlimited.That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit soundcard.
pci_set_dma_maskshould support 24-bits accordingly, otherwise it's a bug in your platform implementation... Nobody will be able to use certain properly-written drivers on your platform otherwise.
If we're going to really sort this out, would be nice to just pass an upper bound for an address to __alloc_pages, instead of a simple bitmask ;-)
M.
Quoted 2 lines · Martin J. Bligh
If we're going to really sort this out, would be nice to just pass an upper bound for an address to
__alloc_pages, instead of a simple bitmask ;-)
Not all systems work out that way. Some ARM for example have a memory window rather than a range. It does solve most platforms so its definitely the right basic feature to have and let awkward boxes overload it.
Quoted 12 lines · Martin J. Bligh
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.The ia64 is a fine example of how broken it is. People have to hack around with
GFP_DMAmeaning different things on ia64 to everything else. It needs to die.At least on x86-64 it is still needed when you need have some hardware with address limits < 4GB (e.g. an 24bit soundcard)
pci_* on K8 only allows address mask
0xffffffffor unlimited.
Quoted 8 lines
Quoted 4 lines
That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit soundcard.
pci_set_dma_maskshould support 24-bits accordingly, otherwise it's a bug in your platform implementation... Nobody will be able to use certain properly-written drivers on your platform otherwise.If we're going to really sort this out, would be nice to just pass an upper bound for an address to
__alloc_pages, instead of a simple bitmask ;-)
To do it properly, I think you'd need to give a range, not just an upper bound. On some ARM / XScale systems, you can specify a window of your RAM that is visible on the PCI bus. That may be a case too odd to care about, but I'm going to have to at some point in the future....
Eli
--------------------. "If it ain't broke now,
Eli Carter \ it will be soon." -- crypto-gram
eli.carter(a)inet.com `-------------------------------------------------
Quoted 4 lines · Eli Carter
To do it properly, I think you'd need to give a range, not just an upper bound. On some ARM / XScale systems, you can specify a window of your RAM that is visible on the PCI bus. That may be a case too odd to care about, but I'm going to have to at some point in the future....
Which may not start at address zero either.
There are even ARM systems where it'd be useful to be able to say "only allocate memory in region N where N = first 1MB of every 2MB region. (Yes, I know, this broken hardware should die, but it just isn't going away thanks to the marketing powers of large IC manufacturers.)
Maybe the generic solution could be something like the resource allocation functions which are passed an alignment function? /me hides.
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.
Quoted 8 lines
Quoted 3 lines
The ia64 is a fine example of how broken it is. People have to hack around with
GFP_DMAmeaning different things on ia64 to everything else. It needs to die.At least on x86-64 it is still needed when you need have some hardware with address limits < 4GB (e.g. an 24bit soundcard)
pci_* on K8 only allows address mask
0xffffffffor unlimited.
for 2.7 the underlying low level budy allocator could (should?) just take a DMA bitmask
Quoted 8 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
umm. are you volunteering to convert
drivers/net/macmace.cto the pci_* API then? also,GFP_DMAis used on, eg, s390 to get memory below 2GB and on ia64 to get memory below 4GB.The ia64 is a fine example of how broken it is. People have to hack around with
GFP_DMAmeaning different things on ia64 to everything else. It needs to die.
All the modifier flags on kmalloc and GFP should be "memory allocation descriptors".
A memory allocation descriptor is a linked list of something like:
<memory pool pointer>
<function to call when out of memory on this pool> <next memory pool>
An "atomic" modifier will not have a function pointer that waits for memory when "out of memory".
A DMA pool descriptor will only point to pools that have that capability.
Drivers will eventually have to start specifying what they really
need: GFP_1M, GFP_16M, GFP_2G, GFP_4G, GFP_36G, GFP_ATOMIC.
The 16M allocation descriptor will also point "back" to the 1M allocation descriptor. Thus drivers requesting memory "below 16M" will prefer to use memory between 1M and 16M because that's first on the list.
I think that this will simplify things in the long run.
Roger.
Signature · 5 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * The Worlds Ecosystem is a stable system. Stable systems may experience * * excursions from the stable situation. We are currently in such an * * excursion: The stable situation does not include humans. ***************
Quoted 4 lines · Rogier Wolff
All the modifier flags on kmalloc and GFP should be "memory allocation descriptors".
A DMA pool descriptor will only point to pools that have that capability.
Much much too simple. DMA to what from where for example ? Post 2.6 its IMHO a case of making the equivalent of pci_alloc_* pci_map_* work with generic device objects now we have them.
Quoted 9 lines · Alan Cox
Quoted 4 lines · Rogier Wolff
All the modifier flags on kmalloc and GFP should be "memory allocation descriptors".
A DMA pool descriptor will only point to pools that have that capability.Much much too simple. DMA to what from where for example ? Post 2.6 its IMHO a case of making the equivalent of pci_alloc_* pci_map_* work with generic device objects now we have them.
I can't think of a case that doesn't fit my model. So, if you're saying that it's simple, that's good.
dev1 +----------+ dev2
| | CPU / | |
--------+--------+ BRIDGE +---------+--------
| | ----> | |
mem1 +----------+ mem2If for example, there are two PCI busses, and a memory controller on both of them, but no DMA cpability in one direction through the PCI bridge (embedded CPU) then there should be two memory pools.
In the picture above: dev1 can reach mem2, but dev2 can't reach mem1.
PCI devices on the restricted PCI bus (dev2) will have to pass a memory allocation descriptor that describes just the memory on that PCI bus, the other one (dev1) can pass a descriptor that prefers the non-shared memory, (leaving as much as possible for the devices on the other bus (bus2)), but
reverts to the memory that the other devices can handle as well.
But if we end up "reading" from dev1, and writing the same data to dev2 we're better off DMAing it to "mem2". So, how to tell the dev1 driver that allocing off the allocation descriptor "mem1, then mem2", is not a good a good idea, and that it should use "mem2, then mem1" is a problem that remains to be solved...
But if I'm missing a case that can't be modelled, please enlighten me as to what I'm missing....
Roger.
Signature · 5 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * The Worlds Ecosystem is a stable system. Stable systems may experience * * excursions from the stable situation. We are currently in such an * * excursion: The stable situation does not include humans. ***************
Quoted 6 lines · Rogier Wolff
PCI devices on the restricted PCI bus (dev2) will have to pass a memory allocation descriptor that describes just the memory on that PCI bus, the other one (dev1) can pass a descriptor that prefers the non-shared memory, (leaving as much as possible for the devices on the other bus (bus2)), but
reverts to the memory that the other devices can handle as well.
Which is actually what I said - you need to have the pci_alloc equivalents for generic devices and pass the device you wish to do the allocation for. How that allocation occurs doesn't matter too much and can be arch specific
Quoted 4 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
i'm not the kind of person who just changes the header file and breaks all the drivers. plan:
Quoted 8 lines
Quoted 3 lines
- Add the
GFP_ATOMIC_DMA&GFP_KERNEL_DMAdefinitions
- Change the drivers
- Delete theGFP_DMAdefinitionNeedless pain for people maintaining cross release drivers. Save it for 2.7 where we should finally do the honourable deed given x86-64 may well be mainstream, and simply remove
GFP_DMAand expect people to use pci_*
But why drivers of ISA bus devices with DMA should use pci_* functions?
I'm personally wouldn't have too much pain with GFP_DMA because I have compatibility headers and proposed change for them is tiny.
Quoted 14 lines · Dmitry A. Fedorov
Quoted 4 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
i'm not the kind of person who just changes the header file and breaks all the drivers. plan:
Quoted 8 lines
Quoted 3 lines
- Add the
GFP_ATOMIC_DMA&GFP_KERNEL_DMAdefinitions
- Change the drivers
- Delete theGFP_DMAdefinitionNeedless pain for people maintaining cross release drivers. Save it for 2.7 where we should finally do the honourable deed given x86-64 may well be mainstream, and simply remove
GFP_DMAand expect people to use pci_*
Quoted 4 lines
But why drivers of ISA bus devices with DMA should use pci_* functions?
I'm personally wouldn't have too much pain with
GFP_DMAbecause I have compatibility headers and proposed change for them is tiny.
Do not let the name "pci_" distract, it works for ISA too :)
We can #define pci_xxx isa_xxx if you like :)
Jeff
On Feb 28 2003, at 10:58, Jeff Garzik was caught saying:
Quoted 12 lines · Dmitry A. Fedorov
Quoted 5 lines
But why drivers of ISA bus devices with DMA should use pci_* functions?
I'm personally wouldn't have too much pain with
GFP_DMAbecause I have compatibility headers and proposed change for them is tiny.Do not let the name "
pci_" distract, it works for ISA too :)We can #define
pci_xxxisa_xxxif you like :)Jeff
This discussion raises an issue that I've been meaning to bring up for a bit. Currently, the DMA-API is defined as returning a cpu physical address [1], but should the API be redefined as returning an address which is valid on the bus which the device sits on? For example, on most xscale systems I deal with, there's not a 1:1 mapping from physical to PCI memory space, so the address which is valid on the bus is not a valid CPU physical address. Imagine an SOC system with on-chip Ethernets and such that that have direct access to the physical bus, plus non-1:1 PCI translation. In such a case, I would need a physical address for the on-chip devices and a PCI bus address for the PCI devices. We can extend this case to a SOC which has on-chip peripherals on some custom FooBus with non-1:1 translation into physical memory. Now, depending on whether my device is on the PCI bus or on said FooBus, my dma_addr needs to have a different meaning.
One easy answer is to provide bus-specific bus_map/unmap/etc functions such as is done with PCI, but this seems rather ugly to me as now every custom bus needs a new set of functions which IMNHO defeats the purpose of a Generic DMA API. I think it would be a much cleaner solution to define the DMA API itself to return an address that is valid on the the bus pointed to by dev->bus. I don't think this would require any implementation changes at the moment, but it would be a more flexible definition of the API allowing for easy addition of new bus types with DMA capabilities. All the bus-specific magic would be hidden in the architecture-specific implementations on platforms that require different mappings for different buses, making the life of driver writers much simpler. On systems in which everything is 1:1 mapped between physical, PCI, and FooBuses, no changes would be required.
~Deepak
[1] From Documentation/DMA-API.txt in the 2.5 BK tree:
dma_addr_t
dma_map_single(struct device *dev, void *cpu_addr, size_t size,
enum dma_data_direction direction)
dma_addr_t
pci_map_single(struct device *dev, void *cpu_addr, size_t size,
int direction)
Maps a piece of processor virtual memory so it can be accessed by the device and returns the _physical_ handle of the memory.
Signature · 2 lines
Deepak Saxena, Code Monkey - Ph:480.517.0372 Fax:480.517.0262 MontaVista Software - Powering the Embedded Revolution - www.mvista.com
I would like to add/clarify that I'm not suggesting this only for the mapping functions, but for the whole DMA API. So for example, dma_alloc_coherent would again return a dma_handle that is valid on the bus that dev->bus points to.
~Deepak
Signature · 2 lines
Deepak Saxena, Code Monkey - Ph:480.517.0372 Fax:480.517.0262 MontaVista Software - Powering the Embedded Revolution - www.mvista.com
Quoted 3 lines · Deepak Saxena
This discussion raises an issue that I've been meaning to bring up for a bit. Currently, the DMA-API is defined as returning a cpu physical address [1]
The DMA-API should reflect the same as the PCI-DMA API, namely:
| pci_alloc_consistent returns two values: the virtual address which you | can use to access it from the CPU and dma_handle which you pass to the | card.
Translated: not a physical address. "Physical" address to me means the address that appears on the outside of the CPU after it has passed through the MMU and is heading for the local CPUs system RAM.
So yes, I think this is a bug that exists in the DMA-API document.
It must be an offsetable cookie that the card on its own bus knows how to deal with to access the memory associated with the mapping. In plain speak, the bus address for that device.
Quoted 4 lines
One easy answer is to provide bus-specific bus_map/unmap/etc functions such as is done with PCI, but this seems rather ugly to me as now every custom bus needs a new set of functions which IMNHO defeats the purpose of a Generic DMA API.
You can do this today with the API - test dev->bus_type to determine the type of the bus, convert to the correct device type and grab the bus information.
The API hasn't really changed - its just that bugs exist in the documentation that need fixing before 2.6 comes out. Submit a patch. 8)
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines
This discussion raises an issue that I've been meaning to bring up for a bit. Currently, the DMA-API is defined as returning a cpu physical address [1], but should the API be redefined as returning an address which is valid on the bus which the device sits on? [...]
dma_addr_t
dma_map_single(struct device *dev, void*cpu_addr,size_tsize,
enumdma_data_directiondirection)
dma_addr_t
pci_map_single(struct device *dev, void*cpu_addr,size_tsize,
int direction)
Quoted 2 lines
Maps a piece of processor virtual memory so it can be accessed by the device and returns the _physical_ handle of the memory.
That's only a sloppy documentation, obviously it returns a bus address, not a physical address. Simply the x86 graduate who wrote it cannot see the difference, that's all. Pay it no mind and implement it properly, that means copy it from sparc64. Who reads documentation anyway when we have the source?
-- Pete
Quoted 2 lines · Pete Zaitcev
Pay it no mind and
implement it properly, that means copy it from sparc64.
and fix the documentation so the next unsuspecting individual doesn't fall into the same trap and ask the same question on the same mailing list. 8)
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
Quoted 14 lines · Dmitry A. Fedorov
Quoted 4 lines · Alan Cox
Quoted 3 lines · Matthew Wilcox
i'm not the kind of person who just changes the header file and breaks all the drivers. plan:
Quoted 8 lines
Quoted 3 lines
- Add the
GFP_ATOMIC_DMA&GFP_KERNEL_DMAdefinitions
- Change the drivers
- Delete theGFP_DMAdefinitionNeedless pain for people maintaining cross release drivers. Save it for 2.7 where we should finally do the honourable deed given x86-64 may well be mainstream, and simply remove
GFP_DMAand expect people to use pci_*
Quoted 4 lines
But why drivers of ISA bus devices with DMA should use pci_* functions?
I'm personally wouldn't have too much pain with
GFP_DMAbecause I have compatibility headers and proposed change for them is tiny.
Umm, question - I've seen ISA bridges with the ability to perform 32-bit DMA using the ISA DMA controllers. AFAIK, Linux doesn't make use of this feature, except on ARM PCI systems with ISA bridges. Is there a reason why this isn't used on x86 hardware?
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
Quoted 3 lines · Russell King
Umm, question - I've seen ISA bridges with the ability to perform 32-bit DMA using the ISA DMA controllers. AFAIK, Linux doesn't make use of this feature, except on ARM PCI systems with ISA bridges.
Alpha uses this from day 1, BTW.
Also, in 2.5 we have "isa_bridge" stuff which was intended exactly for that - it's a pointer to pci device (real ISA bridge with appropriate dma_mask) that can be used by non-busmastering ISA devices as a pci_dev * arg to pci_* mapping functions.
Quoted 2 lines
Is there a reason
why this isn't used on x86 hardware?
Given a huge number of various ISA bridges found in x86 systems, I don't see a generic way to determine which ones can do 32-bit DMA... Maybe kind of white list?
Ivan.
Quoted 4 lines · Dmitry A. Fedorov
But why drivers of ISA bus devices with DMA should use pci_* functions?
I'm personally wouldn't have too much pain with
GFP_DMAbecause I have compatibility headers and proposed change for them is tiny.
ISA devices or ISA like devices on non x86-32 platforms already need to do this to work on non PC boxes. I'm hoping the generic DMA framework stuff will make that all much cleaner for 2.7