kmalloc/pci_alloc and skbuff's
The Markham board (Korva/VR41xx variant) which I'm trying to get submitted, has a PCI problem. It doesn't allow a PCI device to see all the DRAM. DMA-able data must be in a special 4MB range, which is currently set at 0x01c00000 (the last 4MB of a 32MB system). SKB's must be allocated within this area for a (say, Ethernet) PCI device to access. There appears to be no way to tell kmalloc(), used in alloc_skb(), to allocate memory from any special address range.
The pci_{alloc,free}_consistent() routines do the right thing.
Hence the following hack to linux/net/core/skbuff.c.
Any suggestions?
Geoff
Signature · 112 lines
Geoffrey Espin [email protected] -- --- skbuff.c.ORIG Tue Aug 7 08:30:50 2001 +++ skbuff.c Mon Dec 17 13:46:40 2001 @@ -62,6 +62,12 @@ #include <asm/uaccess.h> #include <asm/system.h> +#if defined(CONFIG_NEC_MARKHAM_PCI) +#warning PCI ALLOC: see linux/arch/mips/markham/pci_fixup.c +extern void *pci_alloc_consistent(int, int, dma_addr_t *); +extern void pci_free_consistent(int,int, void *, int); +#endif + int sysctl_hot_list_len = 128; static kmem_cache_t *skbuff_head_cache; @@ -187,6 +193,23 @@ /* Get the DATA. Size must match skb_add_mtu(). */ size = SKB_DATA_ALIGN(size); +#if defined(CONFIG_NEC_MARKHAM_PCI) + /* NEC Markham: see skb_release_data() also */ + + /* + * Markham doesn't allow a PCI device to see all the RAM. + * DMAable data must be on this special 4MB range, which + * is currently set at 0x01c00000. SKB buffers must also + * be allocated within this area for a PCI device to + * access. There is no way to tell kmalloc() to allocate + * memory from any special address range. So we do it on + * our own. + * Dai, Wed Jun 13 19:34:23 PDT 2001 + */ + if (1) + data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL); + else +#endif data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask); if (data == NULL) goto nodata; @@ -286,6 +309,12 @@ if (skb_shinfo(skb)->frag_list) skb_drop_fraglist(skb); +#if defined(CONFIG_NEC_MARKHAM_PCI) + /* NEC Markham: see allock_skb() also */ + if (1) + pci_free_consistent(0,0,skb->head,0); + else +#endif kfree(skb->head); } } @@ -505,6 +534,24 @@ size = (skb->end - skb->head + expand); size = SKB_DATA_ALIGN(size); + +#if defined(CONFIG_NEC_MARKHAM_PCI) + /* NEC Markham: see skb_release_data() also */ + + /* + * Markham doesn't allow a PCI device to see all the RAM. + * DMAable data must be on this special 4MB range, which + * is currently set at 0x01c00000. SKB buffers must also + * be allocated within this area for a PCI device to + * access. There is no way to tell kmalloc() to allocate + * memory from any special address range. So we do it on + * our own. + * Dai, Wed Jun 13 19:34:23 PDT 2001 + */ + if (1) + data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL); + else +#endif data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask); if (data == NULL) return -ENOMEM; @@ -512,7 +559,6 @@ /* Copy entire thing */ if (skb_copy_bits(skb, -headerlen, data, headerlen+skb->len)) BUG(); - /* Offset between the two in bytes */ offset = data - skb->head; @@ -627,6 +673,23 @@ size = SKB_DATA_ALIGN(size); +#if defined(CONFIG_NEC_MARKHAM_PCI) + /* NEC Markham: see skb_release_data() also */ + + /* + * Markham doesn't allow a PCI device to see all the RAM. + * DMAable data must be on this special 4MB range, which + * is currently set at 0x01c00000. SKB buffers must also + * be allocated within this area for a PCI device to + * access. There is no way to tell kmalloc() to allocate + * memory from any special address range. So we do it on + * our own. + * Dai, Wed Jun 13 19:34:23 PDT 2001 + */ + if (1) + data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL); + else +#endif data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask); if (data == NULL) goto nodata;
Quoted 10 lines · Geoffrey Espin
The Markham board (Korva/VR41xx variant) which I'm trying to get submitted, has a PCI problem. It doesn't allow a PCI device to see all the DRAM. DMA-able data must be in a special 4MB range, which is currently set at
0x01c00000(the last 4MB of a 32MB system). SKB's must be allocated within this area for a (say, Ethernet) PCI device to access. There appears to be no way to tellkmalloc(), used inalloc_skb(), to allocate memory from any special address range.
What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?
Regards,
Gleb.
Quoted 2 lines
What's wrong with
GFP_DMA? Doesn't it solve exactly this problem ?
Personally I don't like the hack but you have to ask what he needs. kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory is continues. I think Geoffrey needs to use a specific memory address in PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't like the hack is that skbuff's is bus independent. Not all ethernet cards are PCI based. Please try using GFP_DMA and let us know if it worked.
. ---
|o_o | |:_/ | Give Micro$oft the Bird!!!! // \ \ Use Linux!!!! (| | ) /'_ _/`\
___)=(___/
Quoted 8 lines
Quoted 2 lines
What's wrong with
GFP_DMA? Doesn't it solve exactly this problem ?Personally I don't like the hack but you have to ask what he needs. kmalloc grabs memory from the CPU cache.
GFP_DMAinsures that cache memory is continues. I think Geoffrey needs to use a specific memory address in PCI space. Tho I like Geoffrey to try usingGFP_DMA. The reason I don't like the hack is that skbuff's is bus independent. Not all ethernet cards are PCI based. Please try usingGFP_DMAand let us know if it worked.
Yes, I originally thought this was what addressed it. Is "setting dma_mask" what is meant by "using GFP_DMA"?
The problem is drivers call dev_alloc_skb() which can allocate memory anywhere in (my 32M) memory.
The PCI host controller part of the uPD98052 with its VR4120a core (doc at http://www.idiom.com/~espin/nec/hwdoc/uPD98502-UM.pdf)
allows you to program a 4M window onto DRAM. I use top 4M of 32M, but it's arbitrary. Then only this area can be transferred to by/from the PCI devices. So its not the PCI devices that is the problem, but access to the host-side DRAM.
Currently, my private pci_alloc/free_consistent() routines manage the 4M at top of memory (its not added to kernel with add_memory_region() in prom.c).
With these hacks (including net/core/skbuff.c:alloc_skb->pci_alloc_consistent) I've been successfully using the Tulip Ethernet (LinkSys) card (with no changes to the driver).
Geoff
Signature · 2 lines
Geoffrey Espin [email protected]
Quoted 22 lines · Geoffrey Espin
Quoted 8 lines
Quoted 2 lines
What's wrong with
GFP_DMA? Doesn't it solve exactly this problem ?Personally I don't like the hack but you have to ask what he needs. kmalloc grabs memory from the CPU cache.
GFP_DMAinsures that cache memory is continues. I think Geoffrey needs to use a specific memory address in PCI space. Tho I like Geoffrey to try usingGFP_DMA. The reason I don't like the hack is that skbuff's is bus independent. Not all ethernet cards are PCI based. Please try usingGFP_DMAand let us know if it worked.Yes, I originally thought this was what addressed it. Is "setting
dma_mask" what is meant by "usingGFP_DMA"?The problem is drivers call
dev_alloc_skb()which can allocate memory anywhere in (my 32M) memory.The PCI host controller part of the uPD98052 with its VR4120a core (doc at http://www.idiom.com/~espin/nec/hwdoc/uPD98502-UM.pdf)
allows you to program a 4M window onto DRAM. I use top 4M of 32M, but it's arbitrary. Then only this area can be transferred to by/from the PCI devices. So its not the PCI devices that is the problem, but access to the host-side DRAM.
Quoted 7 lines
Currently, my private pci_alloc/free_consistent() routines manage the 4M at top of memory (its not added to kernel with
add_memory_region()inprom.c).With these hacks (including net/core/skbuff.c:alloc_skb->pci_alloc_consistent) I've been successfully using the Tulip Ethernet (LinkSys) card (with no changes to the driver).
FYI, this is not an isolated issue. We deal with a number of architectures and we've seen this problem with other arches and system controllers as well. A 'generic' solution would be nice and probably necessary at some point. 2.5 would be a good place to do it, if only someone would volunteer ;-)
Pete
Quoted 8 lines
Quoted 2 lines
Currently, my private pci_alloc/free_consistent() routines manage
FYI, this is not an isolated issue. We deal with a number of architectures and we've seen this problem with other arches and system controllers as well. A 'generic' solution would be nice and probably necessary at some point. 2.5 would be a good place to do it, if only
someone would volunteer ;-) [Pete]
Thanks for the reassurance.
Can one include one's own arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a, seems like patching the one file is not feasible.
I tried tweaking $(HEAD) but then stumbled onto this.
How does one package such a work-around? Include a patch file in the LSP, which gets automatically run to munge on kernel sources?
Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?
Geoff
Signature · 1 line
Geoffrey Espin [email protected]
Quoted 11 lines · Geoffrey Espin
Quoted 8 lines
Quoted 2 lines
Currently, my private pci_alloc/free_consistent() routines manage
FYI, this is not an isolated issue. We deal with a number of architectures and we've seen this problem with other arches and system controllers as well. A 'generic' solution would be nice and probably necessary at some point. 2.5 would be a good place to do it, if only
someone would volunteer ;-) [Pete]Thanks for the reassurance.
Quoted 7 lines
Can one include one's own
arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a, seems like patching the one file is not feasible.
I tried tweaking $(HEAD) but then stumbled onto this.How does one package such a work-around? Include a patch file in the LSP, which gets automatically run to munge on kernel sources?
If you mean MontaVista's LSP, they are built from an internal source tree, so in some sense we can do whatever we want -- at least until we implement the right solution. It's not something you want to do though
because carrying patches with you does not scale well. The best
approach in the long run is to do it right and push it out to the community tree asap. The problem, of course, is that if you yourself wanted to do it right, but it was going to take a while, the company you're consulting for might not want to pay for such general linux work.
Quoted 1 line
Or will linux-mips.sf.net accept a patch for
net/core/skbuff.c?
Up the the maintainer, but it's really not the right fix so I think it's best for you or your customer to keep that patch locally.
Another approach would be for you to preallocate your network buffers in your driver, attach them permanently to the rx/tx descriptors, and then copy the data from the buffer to a newly allocated skbuff, when you receive, and the other way when you transmit. There's a performance penalty there, especially for large packets, but you can encapsulate everything within your driver and you won't have to maintain a patch.
Pete
Quoted 2 lines
Quoted 2 lines
Can one include one's own
arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a,
I figured someone might have some magic for hacking arch/mips/Makefile ifdef CONFIG_NEC_KORVA (or any BSP/LSP). I guess I'll have to make the effort. :-)
Quoted 3 lines
Quoted 1 line
Or will linux-mips.sf.net accept a patch for
net/core/skbuff.c?Up the the maintainer, but it's really not the right fix so I think it's best for you or your customer to keep that patch locally.
I talked Jun into checking in the Markham patches... but will have to add a comment to the pci_xxx files that one has to get a separate patch for net/core/skbuff.c if using PCI with a generic driver.
Quoted 2 lines
Another approach would be for you to preallocate your network buffers in your driver, attach them permanently to the rx/tx descriptors, and then
The idea was not to mess with any PCI drivers. 2.5 is not an option for me.
Thanks, Pete.
Geoff
Signature · 1 line
Geoffrey Espin [email protected]
Quoted 16 lines · Geoffrey Espin
Quoted 2 lines
Quoted 2 lines
Can one include one's own
arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a,I figured someone might have some magic for hacking
arch/mips/MakefileifdefCONFIG_NEC_KORVA(or any BSP/LSP). I guess I'll have to make the effort. :-)Quoted 3 lines
Quoted 1 line
Or will linux-mips.sf.net accept a patch for
net/core/skbuff.c?Up the the maintainer, but it's really not the right fix so I think it's best for you or your customer to keep that patch locally.
I talked Jun into checking in the Markham patches... but will have to add a comment to the
pci_xxxfiles that one has to get a separate patch fornet/core/skbuff.cif using PCI with a generic driver.
Quoted 4 lines
Quoted 2 lines
Another approach would be for you to preallocate your network buffers in your driver, attach them permanently to the rx/tx descriptors, and then
The idea was not to mess with any PCI drivers. 2.5 is not an option for me.
Sorry, that was a bad suggestion. I was thinking of an ethernet driver that's part of your SOC and requires a new ethernet driver anyway. Otherwise, messing with all the pci ethernet drivers is definitely not an option.
Pete