Skip to contentSkip to conversation

kmalloc/pci_alloc and skbuff's

DISCUSSION Geoffrey Espin 4 participants 9 messages Dec 18, 2001 – Dec 19, 2001 MIPS

lore
Share & tools
lore
#0 lore raw

The Markham board (Korva/VR41xx variant) which I'm trying to get submitted, has a PCI problem. It doesn't allow a PCI device to see all the DRAM. DMA-able data must be in a special 4MB range, which is currently set at 0x01c00000 (the last 4MB of a 32MB system). SKB's must be allocated within this area for a (say, Ethernet) PCI device to access. There appears to be no way to tell kmalloc(), used in alloc_skb(), to allocate memory from any special address range.

The pci_{alloc,free}_consistent() routines do the right thing.

Hence the following hack to linux/net/core/skbuff.c.

Any suggestions?

Geoff

Signature · 112 lines
Geoffrey Espin [email protected]
--

--- skbuff.c.ORIG	Tue Aug  7 08:30:50 2001
+++ skbuff.c	Mon Dec 17 13:46:40 2001
@@ -62,6 +62,12 @@
 #include <asm/uaccess.h>
 #include <asm/system.h>
 
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+#warning PCI ALLOC: see linux/arch/mips/markham/pci_fixup.c
+extern void *pci_alloc_consistent(int, int, dma_addr_t *);
+extern void pci_free_consistent(int,int, void *, int);
+#endif
+
 int sysctl_hot_list_len = 128;
 
 static kmem_cache_t *skbuff_head_cache;
@@ -187,6 +193,23 @@
 
 	/* Get the DATA. Size must match skb_add_mtu(). */
 	size = SKB_DATA_ALIGN(size);
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+	/* NEC Markham: see skb_release_data() also */
+
+	/*
+	 * Markham doesn't allow a PCI device to see all the RAM.
+	 * DMAable data must be on this special 4MB range, which
+	 * is currently set at 0x01c00000. SKB buffers must also
+	 * be allocated within this area for a PCI device to
+	 * access. There is no way to tell kmalloc() to allocate
+	 * memory from any special address range. So we do it on
+	 * our own.
+	 *                     Dai, Wed Jun 13 19:34:23 PDT 2001
+	 */
+	if (1)
+		data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL);
+	else
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (data == NULL)
 		goto nodata;
@@ -286,6 +309,12 @@
 		if (skb_shinfo(skb)->frag_list)
 			skb_drop_fraglist(skb);
 
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+		/* NEC Markham: see allock_skb() also */
+		if (1)
+			pci_free_consistent(0,0,skb->head,0);
+		else
+#endif
 		kfree(skb->head);
 	}
 }
@@ -505,6 +534,24 @@
 
 	size = (skb->end - skb->head + expand);
 	size = SKB_DATA_ALIGN(size);
+
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+	/* NEC Markham: see skb_release_data() also */
+
+	/*
+	 * Markham doesn't allow a PCI device to see all the RAM.
+	 * DMAable data must be on this special 4MB range, which
+	 * is currently set at 0x01c00000. SKB buffers must also
+	 * be allocated within this area for a PCI device to
+	 * access. There is no way to tell kmalloc() to allocate
+	 * memory from any special address range. So we do it on
+	 * our own.
+	 *                     Dai, Wed Jun 13 19:34:23 PDT 2001
+	 */
+	if (1)
+		data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL);
+	else
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (data == NULL)
 		return -ENOMEM;
@@ -512,7 +559,6 @@
 	/* Copy entire thing */
 	if (skb_copy_bits(skb, -headerlen, data, headerlen+skb->len))
 		BUG();
-
 	/* Offset between the two in bytes */
 	offset = data - skb->head;
 
@@ -627,6 +673,23 @@
 
 	size = SKB_DATA_ALIGN(size);
 
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+	/* NEC Markham: see skb_release_data() also */
+
+	/*
+	 * Markham doesn't allow a PCI device to see all the RAM.
+	 * DMAable data must be on this special 4MB range, which
+	 * is currently set at 0x01c00000. SKB buffers must also
+	 * be allocated within this area for a PCI device to
+	 * access. There is no way to tell kmalloc() to allocate
+	 * memory from any special address range. So we do it on
+	 * our own.
+	 *                     Dai, Wed Jun 13 19:34:23 PDT 2001
+	 */
+	if (1)
+		data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL);
+	else
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (data == NULL)
 		goto nodata;
#1 lore raw
Quoted 10 lines · Geoffrey Espin

The Markham board (Korva/VR41xx variant) which I'm trying to get submitted, has a PCI problem. It doesn't allow a PCI device to see all the DRAM. DMA-able data must be in a special 4MB range, which is currently set at 0x01c00000 (the last 4MB of a 32MB system). SKB's must be allocated within this area for a (say, Ethernet) PCI device to access. There appears to be no way to tell kmalloc(), used in alloc_skb(), to allocate memory from any special address range.

What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?

Regards,
Gleb.

#2 lore raw
Quoted 2 lines

What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?

Personally I don't like the hack but you have to ask what he needs. kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory is continues. I think Geoffrey needs to use a specific memory address in PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't like the hack is that skbuff's is bus independent. Not all ethernet cards are PCI based. Please try using GFP_DMA and let us know if it worked.

. ---

   |o_o |
   |:_/ |   Give Micro$oft the Bird!!!!
  //   \ \  Use Linux!!!!
 (|     | )
 /'_   _/`\

___)=(___/

#3 lore raw
Quoted 8 lines
Quoted 2 lines

What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?

Personally I don't like the hack but you have to ask what he needs. kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory is continues. I think Geoffrey needs to use a specific memory address in PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't like the hack is that skbuff's is bus independent. Not all ethernet cards are PCI based. Please try using GFP_DMA and let us know if it worked.

Yes, I originally thought this was what addressed it. Is "setting dma_mask" what is meant by "using GFP_DMA"?

The problem is drivers call dev_alloc_skb() which can allocate memory anywhere in (my 32M) memory.

The PCI host controller part of the uPD98052 with its VR4120a core (doc at http://www.idiom.com/~espin/nec/hwdoc/uPD98502-UM.pdf)
allows you to program a 4M window onto DRAM. I use top 4M of 32M, but it's arbitrary. Then only this area can be transferred to by/from the PCI devices. So its not the PCI devices that is the problem, but access to the host-side DRAM.

Currently, my private pci_alloc/free_consistent() routines manage the 4M at top of memory (its not added to kernel with add_memory_region() in prom.c).

With these hacks (including net/core/skbuff.c:alloc_skb->pci_alloc_consistent) I've been successfully using the Tulip Ethernet (LinkSys) card (with no changes to the driver).

Geoff

Signature · 2 lines
Geoffrey Espin
[email protected]
#4 lore raw
Quoted 22 lines · Geoffrey Espin
Quoted 8 lines
Quoted 2 lines

What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?

Personally I don't like the hack but you have to ask what he needs. kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory is continues. I think Geoffrey needs to use a specific memory address in PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't like the hack is that skbuff's is bus independent. Not all ethernet cards are PCI based. Please try using GFP_DMA and let us know if it worked.

Yes, I originally thought this was what addressed it. Is "setting dma_mask" what is meant by "using GFP_DMA"?

The problem is drivers call dev_alloc_skb() which can allocate memory anywhere in (my 32M) memory.

The PCI host controller part of the uPD98052 with its VR4120a core (doc at http://www.idiom.com/~espin/nec/hwdoc/uPD98502-UM.pdf)
allows you to program a 4M window onto DRAM. I use top 4M of 32M, but it's arbitrary. Then only this area can be transferred to by/from the PCI devices. So its not the PCI devices that is the problem, but access to the host-side DRAM.

Quoted 7 lines

Currently, my private pci_alloc/free_consistent() routines manage the 4M at top of memory (its not added to kernel with add_memory_region() in prom.c).

With these hacks (including net/core/skbuff.c:alloc_skb->pci_alloc_consistent) I've been successfully using the Tulip Ethernet (LinkSys) card (with no changes to the driver).

FYI, this is not an isolated issue. We deal with a number of architectures and we've seen this problem with other arches and system controllers as well. A 'generic' solution would be nice and probably necessary at some point. 2.5 would be a good place to do it, if only someone would volunteer ;-)

Pete

#5 lore raw
Quoted 8 lines
Quoted 2 lines

Currently, my private pci_alloc/free_consistent() routines manage

FYI, this is not an isolated issue. We deal with a number of architectures and we've seen this problem with other arches and system controllers as well. A 'generic' solution would be nice and probably necessary at some point. 2.5 would be a good place to do it, if only
someone would volunteer ;-) [Pete]

Thanks for the reassurance.

Can one include one's own arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a, seems like patching the one file is not feasible.
I tried tweaking $(HEAD) but then stumbled onto this.

How does one package such a work-around? Include a patch file in the LSP, which gets automatically run to munge on kernel sources?

Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?

Geoff

Signature · 1 line
Geoffrey Espin [email protected]
#6 lore raw
Quoted 11 lines · Geoffrey Espin
Quoted 8 lines
Quoted 2 lines

Currently, my private pci_alloc/free_consistent() routines manage

FYI, this is not an isolated issue. We deal with a number of architectures and we've seen this problem with other arches and system controllers as well. A 'generic' solution would be nice and probably necessary at some point. 2.5 would be a good place to do it, if only
someone would volunteer ;-) [Pete]

Thanks for the reassurance.

Quoted 7 lines

Can one include one's own arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a, seems like patching the one file is not feasible.
I tried tweaking $(HEAD) but then stumbled onto this.

How does one package such a work-around? Include a patch file in the LSP, which gets automatically run to munge on kernel sources?

If you mean MontaVista's LSP, they are built from an internal source tree, so in some sense we can do whatever we want -- at least until we implement the right solution. It's not something you want to do though
because carrying patches with you does not scale well. The best
approach in the long run is to do it right and push it out to the community tree asap. The problem, of course, is that if you yourself wanted to do it right, but it was going to take a while, the company you're consulting for might not want to pay for such general linux work.

Quoted 1 line

Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?

Up the the maintainer, but it's really not the right fix so I think it's best for you or your customer to keep that patch locally.

Another approach would be for you to preallocate your network buffers in your driver, attach them permanently to the rx/tx descriptors, and then copy the data from the buffer to a newly allocated skbuff, when you receive, and the other way when you transmit. There's a performance penalty there, especially for large packets, but you can encapsulate everything within your driver and you won't have to maintain a patch.

Pete

#7 lore raw
Quoted 2 lines
Quoted 2 lines

Can one include one's own arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a,

I figured someone might have some magic for hacking arch/mips/Makefile ifdef CONFIG_NEC_KORVA (or any BSP/LSP). I guess I'll have to make the effort. :-)

Quoted 3 lines
Quoted 1 line

Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?

Up the the maintainer, but it's really not the right fix so I think it's best for you or your customer to keep that patch locally.

I talked Jun into checking in the Markham patches... but will have to add a comment to the pci_xxx files that one has to get a separate patch for net/core/skbuff.c if using PCI with a generic driver.

Quoted 2 lines

Another approach would be for you to preallocate your network buffers in your driver, attach them permanently to the rx/tx descriptors, and then

The idea was not to mess with any PCI drivers. 2.5 is not an option for me.

Thanks, Pete.

Geoff

Signature · 1 line
Geoffrey Espin [email protected]
#8 lore raw
Quoted 16 lines · Geoffrey Espin
Quoted 2 lines
Quoted 2 lines

Can one include one's own arch/mips/korva/skbuff.c? But with network.o being a monolithic blob .o instead of a .a,

I figured someone might have some magic for hacking arch/mips/Makefile ifdef CONFIG_NEC_KORVA (or any BSP/LSP). I guess I'll have to make the effort. :-)

Quoted 3 lines
Quoted 1 line

Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?

Up the the maintainer, but it's really not the right fix so I think it's best for you or your customer to keep that patch locally.

I talked Jun into checking in the Markham patches... but will have to add a comment to the pci_xxx files that one has to get a separate patch for net/core/skbuff.c if using PCI with a generic driver.

Quoted 4 lines
Quoted 2 lines

Another approach would be for you to preallocate your network buffers in your driver, attach them permanently to the rx/tx descriptors, and then

The idea was not to mess with any PCI drivers. 2.5 is not an option for me.

Sorry, that was a bad suggestion. I was thinking of an ethernet driver that's part of your SOC and requires a new ethernet driver anyway. Otherwise, messing with all the pci ethernet drivers is definitely not an option.

Pete