net: add PCINet driver
Subsystems (4)
RFC PATCH v2 1/1 net: add PCINet driver
This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.
I have implemented client support for the Freescale MPC8349EMDS board, which is capable of running in PCI Agent mode (It acts like a PCI card, but is a complete PowerPC computer, running Linux). It is almost certainly trivially ported to any MPC83xx system. It should be a relatively small effort to port it to any chip that can generate PCI interrupts and has at least one PCI accessible scratch register.
It was developed to work in a CompactPCI crate of computers, one of which is a relatively standard x86 system (acting as the host) and many PowerPC systems (acting as clients).
RFC v1 -> RFC v2:
* remove vim modelines
* use net_device->name in request_irq(), for irqbalance
* remove unnecessary wqt_get_stats(), use default get_stats() instead
* use dev_printk() and friends
* add message unit to MPC8349EMDS dts file
---
This is the third posting of this driver. I got some feedback, and have corrected the problems. Stephen, thanks for the review! I also got some off-list feedback from a potential user, so I believe this is worth getting into mainline.
I'll post up a revised version about once a week as long as the changes are minor. If they are more substantial, I'll post them as needed.
GregKH: is this worth putting into the staging tree? (I left you out of the CC list to keep your email traffic down)
The remaining issues I see in this driver:
1) ==== Naming ====
The name wqt originally stood for "workqueue-test" and somewhat evolved over time into the current driver. I'm looking for suggestions for a better name. It should be the same between the host and client drivers, to make porting the code between them easier. The drivers are /very/ similar other than the setup code.
2) ==== IMMR Usage ====
In the Freescale client driver, I use the whole set of board control registers (AKA IMMR registers). I only need a very small subset of them, during startup to set up the DMA window. I used the full set of registers so that I could share the register offsets between the drivers (in pcinet_hw.h)
3) ==== ioremap() of a physical address ====
In the Freescale client driver, I called ioremap() with the physical address of the IMMR registers, 0xe0000000. I don't know a better way to get them. They are somewhat exposed in the Flat Device Tree. Suggestions on how to extract them are welcome.
4) ==== Hardcoded DMA Window Address ====
In the Freescale client driver, I just hardcoded the address of the outbound PCI window into the DMA transfer code. It is 0x80000000. Suggestions on how to get this value at runtime are welcome.
Rationale behind some decisions:
1) ==== Usage of the PCINET_NET_REGISTERS_VALID bit ====
I want to be able to use this driver from U-Boot to tftp a kernel over the PCI backplane, and then boot up the board. This means that the device descriptor memory, which lives in the client RAM, becomes invalid during boot.
2) ==== Buffer Descriptors in client memory ====
I chose to put the buffer descriptors in client memory rather than host memory. It seemed more logical to me at the time. In my application, I'll have 19 boards + 1 host per cPCI chassis. The client -> host direction will see most of the traffic, and so I thought I would cut down on the number of PCI accesses needed. I'm willing to change this.
3) ==== Usage of client DMA controller for all data transfer ==== This was done purely for speed. I tried using the CPU to transfer all data, and it is very slow: ~3MB/sec. Using the DMA controller gets me to ~40MB/sec (as tested with netperf).
4) ==== Static 1GB DMA window ====
Maintaining a window while DMA's in flight, and then changing it seemed too complicated. Also, testing showed that using a static window gave me a ~10MB/sec speedup compared to moving the window for each skb.
5) ==== The serial driver ====
Yes, there are two essentially separate drivers here. I needed a method to communicate with the U-Boot bootloader on these boards without plugging in a serial cable. With 19 clients + 1 host per chassis, the cable clutter is worth avoiding. Since everything is connected via the PCI bus anyway, I used that. A virtual serial port was simple to implement using the messaging unit hardware that I used for the network driver.
I'll post both U-Boot drivers to their mailing list once this driver is finalized.
Thanks,
Ira
arch/powerpc/boot/dts/mpc834x_mds.dts | 7 + drivers/net/Kconfig | 34 + drivers/net/Makefile | 3 + drivers/net/pcinet.h | 75 ++ drivers/net/pcinet_fsl.c | 1351 ++++++++++++++++++++++++++++++++ drivers/net/pcinet_host.c | 1383 +++++++++++++++++++++++++++++++++ drivers/net/pcinet_hw.h | 77 ++ 7 files changed, 2930 insertions(+), 0 deletions(-) create mode 100644 drivers/net/pcinet.h create mode 100644 drivers/net/pcinet_fsl.c create mode 100644 drivers/net/pcinet_host.c create mode 100644 drivers/net/pcinet_hw.h
arch/powerpc/boot/dts/mpc834x_mds.dts +7 -0
@@ -104,6 +104,13 @@ mode = "cpu"; }; + message-unit@8030 { + compatible = "fsl,mpc8349-mu"; + reg = <0x8030 0xd0>; + interrupts = <69 0x8>; + interrupt-parent = <&ipic>; + }; + dma@82a8 { #address-cells = <1>; #size-cells = <1>;
drivers/net/Kconfig +34 -0
@@ -2279,6 +2279,40 @@ config UGETH_TX_ON_DEMAND bool "Transmit on Demand support" depends on UCC_GETH +config PCINET_FSL + tristate "PCINet Virtual Ethernet over PCI support (Freescale)" + depends on MPC834x_MDS && !PCI + select DMA_ENGINE + select FSL_DMA + help + When running as a PCI Agent, this driver will create a virtual + ethernet link running over the PCI bus, allowing simplified + communication with the host system. The host system will need + to use the corresponding driver. + + If in doubt, say N. + +config PCINET_HOST + tristate "PCINet Virtual Ethernet over PCI support (Host)" + depends on PCI + help + This driver will let you communicate with a PCINet client device + using a virtual ethernet link running over the PCI bus. This + allows simplified communication with the client system. + + This is inteded for use in a system that has a crate full of + computers running Linux, all connected by a PCI backplane. + + If in doubt, say N. + +config PCINET_DISABLE_CHECKSUM + bool "Disable packet checksumming" + depends on PCINET_FSL || PCINET_HOST + default n + help + Disable packet checksumming on packets received by the PCINet + driver. This gives a possible speed boost. + config MV643XX_ETH tristate "Marvell Discovery (643XX) and Orion ethernet support" depends on MV64360 || MV64X60 || (PPC_MULTIPLATFORM && PPC32) || PLAT_ORION
drivers/net/Makefile +3 -0
@@ -30,6 +30,9 @@ gianfar_driver-objs := gianfar.o \ obj-$(CONFIG_UCC_GETH) += ucc_geth_driver.o ucc_geth_driver-objs := ucc_geth.o ucc_geth_mii.o ucc_geth_ethtool.o +obj-$(CONFIG_PCINET_FSL) += pcinet_fsl.o +obj-$(CONFIG_PCINET_HOST) += pcinet_host.o + # # link order important here #
drivers/net/pcinet.h new file +75 -0
75 lines not on this page · raw message
drivers/net/pcinet_fsl.c new file +1351 -0
1,351 lines not on this page · raw message
drivers/net/pcinet_host.c new file +1383 -0
1,383 lines not on this page · raw message
drivers/net/pcinet_hw.h new file +77 -0
77 lines not on this page · raw message
Signature · 1 line
1.5.4.3
Quoted 211 lines · On Wed, 29 Oct 2008 13:20:27 -0700 Ira Snyder <[email protected]> wrote:
This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.
I have implemented client support for the Freescale MPC8349EMDS board, which is capable of running in PCI Agent mode (It acts like a PCI card, but is a complete PowerPC computer, running Linux). It is almost certainly trivially ported to any MPC83xx system. It should be a relatively small effort to port it to any chip that can generate PCI interrupts and has at least one PCI accessible scratch register.
It was developed to work in a CompactPCI crate of computers, one of which is a relatively standard x86 system (acting as the host) and many PowerPC systems (acting as clients).
RFC v1 -> RFC v2:
* remove vim modelines
* usenet_device->nameinrequest_irq(), for irqbalance
* remove unnecessarywqt_get_stats(), use defaultget_stats()instead
* usedev_printk()and friends
* add message unit to MPC8349EMDS dts fileSigned-off-by Ira W. Snyder <[email protected]>---
This is the third posting of this driver. I got some feedback, and have corrected the problems. Stephen, thanks for the review! I also got some off-list feedback from a potential user, so I believe this is worth getting into mainline.
I'll post up a revised version about once a week as long as the changes are minor. If they are more substantial, I'll post them as needed.
GregKH: is this worth putting into the staging tree? (I left you out of the CC list to keep your email traffic down)
The remaining issues I see in this driver:
1) ==== Naming ====
The name wqt originally stood for "workqueue-test" and somewhat evolved over time into the current driver. I'm looking for suggestions for a better name. It should be the same between the host and client drivers, to make porting the code between them easier. The drivers are /very/ similar other than the setup code.
2) ==== IMMR Usage ====
In the Freescale client driver, I use the whole set of board control registers (AKA IMMR registers). I only need a very small subset of them, during startup to set up the DMA window. I used the full set of registers so that I could share the register offsets between the drivers (inpcinet_hw.h)
3) ====ioremap()of a physical address ====
In the Freescale client driver, I calledioremap()with the physical address of the IMMR registers,0xe0000000. I don't know a better way to get them. They are somewhat exposed in the Flat Device Tree. Suggestions on how to extract them are welcome.
4) ==== Hardcoded DMA Window Address ====
In the Freescale client driver, I just hardcoded the address of the outbound PCI window into the DMA transfer code. It is0x80000000. Suggestions on how to get this value at runtime are welcome.Rationale behind some decisions:
1) ==== Usage of thePCINET_NET_REGISTERS_VALIDbit ====
I want to be able to use this driver from U-Boot to tftp a kernel over the PCI backplane, and then boot up the board. This means that the device descriptor memory, which lives in the client RAM, becomes invalid during boot.
2) ==== Buffer Descriptors in client memory ====
I chose to put the buffer descriptors in client memory rather than host memory. It seemed more logical to me at the time. In my application, I'll have 19 boards + 1 host per cPCI chassis. The client -> host direction will see most of the traffic, and so I thought I would cut down on the number of PCI accesses needed. I'm willing to change this.
3) ==== Usage of client DMA controller for all data transfer ==== This was done purely for speed. I tried using the CPU to transfer all data, and it is very slow: ~3MB/sec. Using the DMA controller gets me to ~40MB/sec (as tested with netperf).
4) ==== Static 1GB DMA window ====
Maintaining a window while DMA's in flight, and then changing it seemed too complicated. Also, testing showed that using a static window gave me a ~10MB/sec speedup compared to moving the window for each skb.
5) ==== The serial driver ====
Yes, there are two essentially separate drivers here. I needed a method to communicate with the U-Boot bootloader on these boards without plugging in a serial cable. With 19 clients + 1 host per chassis, the cable clutter is worth avoiding. Since everything is connected via the PCI bus anyway, I used that. A virtual serial port was simple to implement using the messaging unit hardware that I used for the network driver.I'll post both U-Boot drivers to their mailing list once this driver is finalized.
Thanks,
Iraarch/powerpc/boot/dts/mpc834x_mds.dts | 7 + drivers/net/Kconfig | 34 + drivers/net/Makefile | 3 + drivers/net/pcinet.h | 75 ++ drivers/net/pcinet_fsl.c | 1351 ++++++++++++++++++++++++++++++++ drivers/net/pcinet_host.c | 1383 +++++++++++++++++++++++++++++++++ drivers/net/pcinet_hw.h | 77 ++ 7 files changed, 2930 insertions(+), 0 deletions(-) create mode 100644 drivers/net/pcinet.h create mode 100644 drivers/net/pcinet_fsl.c create mode 100644 drivers/net/pcinet_host.c create mode 100644 drivers/net/pcinet_hw.harch/powerpc/boot/dts/mpc834x_mds.dts +7
-0@@ -104,6 +104,13 @@ mode = "cpu"; }; + message-unit@8030 { + compatible = "fsl,mpc8349-mu"; + reg = <0x8030 0xd0>; + interrupts = <69 0x8>; + interrupt-parent = <&ipic>; + }; + dma@82a8 { #address-cells = <1>; #size-cells = <1>;drivers/net/Kconfig +34
-0@@ -2279,6 +2279,40 @@ config UGETH_TX_ON_DEMAND bool "Transmit on Demand support" depends on UCC_GETH +config PCINET_FSL + tristate "PCINet Virtual Ethernet over PCI support (Freescale)" + depends on MPC834x_MDS && !PCI + select DMA_ENGINE + select FSL_DMA + help + When running as a PCI Agent, this driver will create a virtual + ethernet link running over the PCI bus, allowing simplified + communication with the host system. The host system will need + to use the corresponding driver. + + If in doubt, say N. + +config PCINET_HOST + tristate "PCINet Virtual Ethernet over PCI support (Host)" + depends on PCI + help + This driver will let you communicate with a PCINet client device + using a virtual ethernet link running over the PCI bus. This + allows simplified communication with the client system. + + This is inteded for use in a system that has a crate full of + computers running Linux, all connected by a PCI backplane. + + If in doubt, say N. + +config PCINET_DISABLE_CHECKSUM + bool "Disable packet checksumming" + depends on PCINET_FSL || PCINET_HOST + default n + help + Disable packet checksumming on packets received by the PCINet + driver. This gives a possible speed boost. + config MV643XX_ETH tristate "Marvell Discovery (643XX) and Orion ethernet support" depends on MV64360 || MV64X60 || (PPC_MULTIPLATFORM && PPC32) || PLAT_ORIONdrivers/net/Makefile +3
-0@@ -30,6 +30,9 @@ gianfar_driver-objs := gianfar.o \ obj-$(CONFIG_UCC_GETH) += ucc_geth_driver.o ucc_geth_driver-objs := ucc_geth.o ucc_geth_mii.o ucc_geth_ethtool.o +obj-$(CONFIG_PCINET_FSL) += pcinet_fsl.o +obj-$(CONFIG_PCINET_HOST) += pcinet_host.o + # # link order important here #drivers/net/pcinet.h new file +19
-0@@ -0,0 +1,75 @@ +/* +* Shared Definitions for the PCINet / PCISerial drivers +* +* Copyright (c) 2008 Ira W. Snyder <[email protected]> +* +* Heavily inspired by the drivers/net/fs_enet driver +* +* This file is licensed under the terms of the GNU General Public License +* version 2. This program is licensed "as is" without any warranty of any +* kind, whether express or implied. +*/ + +#ifndef PCINET_H +#define PCINET_H + +#include <linux/kernel.h> +#include <linux/if_ether.h> + +/* Ring and Frame size -- these must match between the drivers */
drivers/net/pcinet.h line 29
Quoted 10 lines
drivers/net/pcinet.h new file +10
-0+#define PH_RING_SIZE (64) +#define PH_MAX_FRSIZE (64 * 1024) +#define PH_MAX_MTU (PH_MAX_FRSIZE - ETH_HLEN) + +struct circ_buf_desc { +__le32 sc; +__le32 len; +__le32 addr; +} __attribute__((__packed__ )); +typedef struct circ_buf_desc cbd_t ;
Don't use typedef. See chapter 5 of Documentation/CodingStyle
Do you really need packed here, sometime gcc generates much worse code on needlessly packed structures?
Quoted 2 lines · Stephen Hemminger
On Wed, 29 Oct 2008 13:20:27 -0700
Ira Snyder <[email protected]> wrote:
Big snip...
Quoted 29 lines
Quoted 29 lines
drivers/net/pcinet.h new file +23
-0@@ -0,0 +1,75 @@ +/* +* Shared Definitions for the PCINet / PCISerial drivers +* +* Copyright (c) 2008 Ira W. Snyder <[email protected]> +* +* Heavily inspired by the drivers/net/fs_enet driver +* +* This file is licensed under the terms of the GNU General Public License +* version 2. This program is licensed "as is" without any warranty of any +* kind, whether express or implied. +*/ + +#ifndef PCINET_H +#define PCINET_H + +#include <linux/kernel.h> +#include <linux/if_ether.h> + +/* Ring and Frame size -- these must match between the drivers */ +#define PH_RING_SIZE (64) +#define PH_MAX_FRSIZE (64 * 1024) +#define PH_MAX_MTU (PH_MAX_FRSIZE - ETH_HLEN) +
Quoted 9 lines
drivers/net/pcinet.h line 29
Quoted 6 lines
drivers/net/pcinet.h new file +6
-0+struct circ_buf_desc { +__le32 sc; +__le32 len; +__le32 addr; +} __attribute__((__packed__ )); +typedef struct circ_buf_desc cbd_t ;Don't use typedef. See chapter 5 of
Documentation/CodingStyle
I know about this. I was following the example set forth in drivers/net/fs_enet. I tried to make this driver somewhat similar to that driver, but I'm not against changing this to a struct everywhere.
Quoted 2 lines
Do you really need packed here, sometime gcc generates much worse code on needlessly packed structures?
I'm pretty sure I do. I share this structure between both devices over the PCI bus. This is where the physical addresses of the skb's go so that the DMA controller can transfer them.
As an example, let's say I have a machine that places 32bit values on 64bit boundaries. AFAIK, the Freescale places 32bit values on 32bit boundaries. Now the two of them disagree about where the fields of the buffer descriptor are.
Of course, I may be wrong. :) Comments?
Thanks for the review,
Ira
Quoted 2 lines · Ira Snyder
I know about this. I was following the example set forth in
drivers/net/fs_enet.
I recommend against that. :-)
-Scott
Quoted 5 lines · Scott Wood
Quoted 2 lines · Ira Snyder
I know about this. I was following the example set forth in
drivers/net/fs_enet.I recommend against that. :-)
Great, now you tell me :)
I'll go ahead change the typedef to a struct. Hopefully some of the other inspiration I took wasn't too bad. I'm totally open to suggestions on improvements for this driver.
Ira
Quoted 4 lines · Ira Snyder
I'll go ahead change the typedef to a struct. Hopefully some of the other inspiration I took wasn't too bad. I'm totally open to suggestions on improvements for this driver.
Not a suggestion for improvement, but I have to say this is by far the single coolest thing I have EVER seen submitted to mainline :)
Signature · 2 lines
Matt Sealey <[email protected]> Genesi, Manager, Developer Relations
Quoted 10 lines · Matt Sealey
Quoted 4 lines · Ira Snyder
I'll go ahead change the typedef to a struct. Hopefully some of the other inspiration I took wasn't too bad. I'm totally open to suggestions on improvements for this driver.
Not a suggestion for improvement, but I have to say this is by far the single coolest thing I have EVER seen submitted to mainline :)
Awesome, that's what I like to hear!
Here is another quick version of the patch, just for fun. There are only two changes from the original patch, which I didn't think are big enough to start another thread.
1) Use struct circ_buf_desc instead of cbd_t
2) Use get_immrbase() to get the IMMR register address
Ira
======================================================================
[PATCH] net: add PCINet driver
This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.
I have implemented client support for the Freescale MPC8349EMDS board, which is capable of running in PCI Agent mode (It acts like a PCI card, but is a complete PowerPC computer, running Linux). It is almost certainly trivially ported to any MPC83xx system.
It was developed to work in a CompactPCI crate of computers, one of which is a relatively standard x86 system (acting as the host) and many PowerPC systems (acting as clients).
RFC v2 -> RFC v3pre:
* replace typedef cbd_t with struct circ_buf_desc
* use get_immrbase() to get IMMR register offsets
RFC v1 -> RFC v2:
* remove vim modelines
* use net_device->name in request_irq(), for irqbalance
* remove unneccesary wqt_get_stats(), use default get_stats() instead
* use dev_printk() and friends
* add message unit to MPC8349EMDS dts file
--- arch/powerpc/boot/dts/mpc834x_mds.dts | 7 + drivers/net/Kconfig | 34 + drivers/net/Makefile | 3 + drivers/net/pcinet.h | 74 ++ drivers/net/pcinet_fsl.c | 1354 ++++++++++++++++++++++++++++++++ drivers/net/pcinet_host.c | 1383 +++++++++++++++++++++++++++++++++ drivers/net/pcinet_hw.h | 77 ++ 7 files changed, 2932 insertions(+), 0 deletions(-) create mode 100644 drivers/net/pcinet.h create mode 100644 drivers/net/pcinet_fsl.c create mode 100644 drivers/net/pcinet_host.c create mode 100644 drivers/net/pcinet_hw.h
arch/powerpc/boot/dts/mpc834x_mds.dts +7 -0
@@ -104,6 +104,13 @@ mode = "cpu"; }; + message-unit@8030 { + compatible = "fsl,mpc8349-mu"; + reg = <0x8030 0xd0>; + interrupts = <69 0x8>; + interrupt-parent = <&ipic>; + }; + dma@82a8 { #address-cells = <1>; #size-cells = <1>;
drivers/net/Kconfig +34 -0
@@ -2279,6 +2279,40 @@ config UGETH_TX_ON_DEMAND bool "Transmit on Demand support" depends on UCC_GETH +config PCINET_FSL + tristate "PCINet Virtual Ethernet over PCI support (Freescale)" + depends on MPC834x_MDS && !PCI + select DMA_ENGINE + select FSL_DMA + help + When running as a PCI Agent, this driver will create a virtual + ethernet link running over the PCI bus, allowing simplified + communication with the host system. The host system will need + to use the corresponding driver. + + If in doubt, say N. + +config PCINET_HOST + tristate "PCINet Virtual Ethernet over PCI support (Host)" + depends on PCI + help + This driver will let you communicate with a PCINet client device + using a virtual ethernet link running over the PCI bus. This + allows simplified communication with the client system. + + This is inteded for use in a system that has a crate full of + computers running Linux, all connected by a PCI backplane. + + If in doubt, say N. + +config PCINET_DISABLE_CHECKSUM + bool "Disable packet checksumming" + depends on PCINET_FSL || PCINET_HOST + default n + help + Disable packet checksumming on packets received by the PCINet + driver. This gives a possible speed boost. + config MV643XX_ETH tristate "Marvell Discovery (643XX) and Orion ethernet support" depends on MV64360 || MV64X60 || (PPC_MULTIPLATFORM && PPC32) || PLAT_ORION
drivers/net/Makefile +3 -0
@@ -30,6 +30,9 @@ gianfar_driver-objs := gianfar.o \ obj-$(CONFIG_UCC_GETH) += ucc_geth_driver.o ucc_geth_driver-objs := ucc_geth.o ucc_geth_mii.o ucc_geth_ethtool.o +obj-$(CONFIG_PCINET_FSL) += pcinet_fsl.o +obj-$(CONFIG_PCINET_HOST) += pcinet_host.o + # # link order important here #
drivers/net/pcinet.h new file +74 -0
74 lines not on this page · raw message
drivers/net/pcinet_fsl.c new file +1354 -0
1,354 lines not on this page · raw message
drivers/net/pcinet_host.c new file +1383 -0
1,383 lines not on this page · raw message
drivers/net/pcinet_hw.h new file +77 -0
77 lines not on this page · raw message
Signature · 1 line
1.5.4.3
Quoted 3 lines · Ira Snyder
This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.
Very interesting. I have seen multiple such drivers, and still plan to have one as well for the cell/axon platform.
Quoted 4 lines
This is the third posting of this driver. I got some feedback, and have corrected the problems. Stephen, thanks for the review! I also got some off-list feedback from a potential user, so I believe this is worth getting into mainline.
Sorry for not having replied earlier, I did not see first postings. While I did not see any major problems with your code, I think we should actually do it in a different way, which I have talked about at the Linux plumbers conference, and will again at the UKUUG Linux conference this weekend. You can find my slides from the previous talk at
http://userweb.kernel.org/%7Earnd/papers/plumbers08/plumbers-slides.pdf
Since you have code and I only have plans, I suppose I can't make you wait for me to get a driver for your platform, but I hope we can join forces in the future and share code in the future.
My fundamental criticism of your code is that it is not flexible enough. We have a number of platforms that act as a PCI, PCIe or similar kind of device and want to transfer network data. At the same time, some of these want to use the same infrastructure for non-network data like MPI (ibverbs), block device or file system (9p). Even in your limited setup, it seems that you should be able to share more code between the two implementations you have posted here. When looking at the code, I found a lot of duplication between the drivers that you should be able to avoid.
My suggestion is to base this on the virtio infrastructure, and some of my colleagues at IBM already implemented a driver that uses virtio-net on a PCIe connection.
Arnd <><
Now a few comments on your code:
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +0
-0
arch/powerpc/boot/dts/mpc834x_mds.dts line 112
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +6
-0@@ -104,6 +104,13 @@ mode = "cpu"; }; + message-unit@8030 { + compatible = "fsl,mpc8349-mu"; + reg = <0x8030 0xd0>; + interrupts = <69 0x8>; + interrupt-parent = <&ipic>; + };
What is a message unit? Is this the mailbox you use in the driver? We are facing a similar problem on Axon regarding probing of the virtual device because you can't easily tell from the device tree whether the machine is connected to something that is trying to communicate.
The message-unit does not seem to be network-specific, so it's not particularly nice to have the network unit grab that of_device.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 12 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +12
-0+config PCINET_FSL + tristate "PCINet Virtual Ethernet over PCI support (Freescale)" + depends on MPC834x_MDS && !PCI + select DMA_ENGINE + select FSL_DMA + help + When running as a PCI Agent, this driver will create a virtual + ethernet link running over the PCI bus, allowing simplified + communication with the host system. The host system will need + to use the corresponding driver. + + If in doubt, say N.
Why 'depends on !PCI'? This means that you cannot build a kernel that is able to run both as host and endpoint for PCInet, right?
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 12 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +12
-0+config PCINET_HOST + tristate "PCINet Virtual Ethernet over PCI support (Host)" + depends on PCI + help + This driver will let you communicate with a PCINet client device + using a virtual ethernet link running over the PCI bus. This + allows simplified communication with the client system. + + This is inteded for use in a system that has a crate full of + computers running Linux, all connected by a PCI backplane. + + If in doubt, say N.
Is this meant to be portable across different hardware implementations? If the driver can only work with PCINET_FSL on the other side, you should probably mention that in the description.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 7 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +7
-0+config PCINET_DISABLE_CHECKSUM + bool "Disable packet checksumming" + depends on PCINET_FSL || PCINET_HOST + default n + help + Disable packet checksumming on packets received by the PCINet + driver. This gives a possible speed boost.
Why make this one optional? Is there ever a reason to enable checksumming? If there is, how about making it tunable with ethtool?
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +5
-0+struct circ_buf_desc { + __le32 sc; + __le32 len; + __le32 addr; +} __attribute__((__packed__));
It would be useful to always force aligning the desciptors to the whole 32 bit and avoid the packing here. Unaligned accesses are inefficient on many systems.
Quoted 1 line
+typedef struct
circ_buf_desccbd_t;
Also, don't pass structures by value if they don't fit into one or two registers.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 8 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +8
-0+/* Buffer Descriptor Accessors */ +#define CBDW_SC(_cbd, _sc) iowrite32((_sc), &(_cbd)->sc) +#define CBDW_LEN(_cbd, _len) iowrite32((_len), &(_cbd)->len) +#define CBDW_ADDR(_cbd, _addr) iowrite32((_addr), &(_cbd)->addr) + +#define CBDR_SC(_cbd) ioread32(&(_cbd)->sc) +#define CBDR_LEN(_cbd) ioread32(&(_cbd)->len) +#define CBDR_ADDR(_cbd) ioread32(&(_cbd)->addr)
We have found that accessing remote descriptors using mmio read is rather slow, and changed the code to always do local reads and remote writes.
On the host side, I would argue that you should use out_le32 rather than iowrite32, because you are not operating on a PCI device but an OF device.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +5
-0+/* IMMR Accessor Helpers */ +#define IMMR_R32(_off) ioread32(priv->immr+(_off)) +#define IMMR_W32(_off, _val) iowrite32((_val), priv->immr+(_off)) +#define IMMR_R32BE(_off) ioread32be(priv->immr+(_off)) +#define IMMR_W32BE(_off, _val) iowrite32be((_val), priv->immr+(_off))
IIRC, the IMMR is a platform resource, so the system should map those registers already and provide accessor functions for the registers you need. Simply allowing to pass an offset in here does not look clean.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 2 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +2
-0+static void wqtuart_rx_char(struct uart_port *port, const char ch); +static void wqtuart_stop_tx(struct uart_port *port);
You should try to avoid forward declarations for static functions. If you order the function implementation correctly, that will also give you the expected reading order in the driver.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 2 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +2
-0+struct wqt_dev; +typedef void (*wqt_irqhandler_t)(struct wqt_dev *);
Much more importantly, never do forward declarations for global functions!
These belong into a header that is included by both the user and the definition.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 6 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +6
-0+struct wqt_dev { + /*--------------------------------------------------------------------*/ + /* OpenFirmware Infrastructure */ + /*--------------------------------------------------------------------*/ + struct of_device *op; + struct device *dev;
Why the dev? You can always get that from the of_device, right?
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 2 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +2
-0+ int irq; + void __iomem *immr;
I don't think immr is device specific.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 12 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +12
-0+ struct mutex irq_mutex; + int interrupt_count; + + spinlock_t irq_lock; + struct wqt_irqhandlers handlers; + + /*--------------------------------------------------------------------*/ + /* UART Device Infrastructure */ + /*--------------------------------------------------------------------*/ + struct uart_port port; + bool uart_rx_enabled; + bool uart_open;
hmm, if you need a uart, that sounds like it should be a separate driver altogether. What is the relation between the network interface and the UART?
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 1 line
arch/powerpc/boot/dts/mpc834x_mds.dts +1
-0+ struct workqueue_struct *wq;
A per-device pointer to a workqueue_struct is unusual. What are you doing this for? How about using the system work queue?
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ /*--------------------------------------------------------------------*/ + /* Ethernet Device Infrastructure */ + /*--------------------------------------------------------------------*/ + struct net_device *ndev;
Why make this a separate structure? If you have one of these per net_device, you should embed the net_device into your own structure.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 1 line
arch/powerpc/boot/dts/mpc834x_mds.dts +1
-0+ struct tasklet_struct tx_complete_tasklet;
Using a tasklet for tx processing sounds fishy because most of the network code already runs at softirq time. You do not gain anything by another softirq context.
Quoted 13 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +13
-0+/*----------------------------------------------------------------------------*/ +/* Status Register Helper Operations */ +/*----------------------------------------------------------------------------*/ + +static DEFINE_SPINLOCK(status_lock); + +static void wqtstatus_setbit(struct wqt_dev *priv, u32 bit) +{ + unsigned long flags; + + spin_lock_irqsave(&status_lock, flags); + IMMR_W32(OMR1_OFFSET, IMMR_R32(OMR1_OFFSET) | bit); + spin_unlock_irqrestore(&status_lock, flags);
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 10 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +10
-0+} + +static void wqtstatus_clrbit(struct wqt_dev *priv, u32 bit) +{ + unsigned long flags; + + spin_lock_irqsave(&status_lock, flags); + IMMR_W32(OMR1_OFFSET, IMMR_R32(OMR1_OFFSET) & ~bit); + spin_unlock_irqrestore(&status_lock, flags); +}
This looks misplaced, as mentioned before.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+static int wqtstatus_remote_testbit(struct wqt_dev *priv, u32 bit) +{ + return IMMR_R32(IMR1_OFFSET) & bit; +}
Are you sure that do not need a spinlock here? The lock also prevents reordering the device access arount the code using the mmio data.
Quoted 12 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +12
-0+/*----------------------------------------------------------------------------*/ +/* Message Sending and Processing Operations */ +/*----------------------------------------------------------------------------*/ + +static irqreturn_t wqt_interrupt(int irq, void *dev_id) +{ + struct wqt_dev *priv = dev_id; + u32 imisr, idr; + unsigned long flags; + + imisr = IMMR_R32(IMISR_OFFSET); + idr = IMMR_R32(IDR_OFFSET);
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 10 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +10
-0+ + if (!(imisr & 0x8)) + return IRQ_NONE; + + /* Clear all of the interrupt sources, we'll handle them next */ + IMMR_W32(IDR_OFFSET, idr); + + /* Lock over all of the handlers, so they cannot get called when + * the code doesn't expect them to be called */ + spin_lock_irqsave(&priv->irq_lock, flags);
If this is in an interrupt handler, why disable the interrupts again? The same comment applies to many of the other places where you use spin_lock_irqsave rather than spin_lock or spin_lock_irq.
Quoted 39 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +39
-0+/* NOTE: All handlers are called with priv->irq_lock held */ + +static void empty_handler(struct wqt_dev *priv) +{ + /* Intentionally left empty */ +} + +static void net_start_req_handler(struct wqt_dev *priv) +{ + schedule_work(&priv->net_start_work); +} + +static void net_start_ack_handler(struct wqt_dev *priv) +{ + complete(&priv->net_start_completion); +} + +static void net_stop_req_handler(struct wqt_dev *priv) +{ + schedule_work(&priv->net_stop_work); +} + +static void net_stop_ack_handler(struct wqt_dev *priv) +{ + complete(&priv->net_stop_completion); +} + +static void net_tx_complete_handler(struct wqt_dev *priv) +{ + tasklet_schedule(&priv->tx_complete_tasklet); +} + +static void net_rx_packet_handler(struct wqt_dev *priv) +{ + wqtstatus_setbit(priv, PCINET_NET_RXINT_OFF); + netif_rx_schedule(priv->ndev, &priv->napi); +} + +static void uart_rx_ready_handler(struct wqt_dev *priv)
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 10 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +10
-0+{ + wqtuart_rx_char(&priv->port, IMMR_R32(IMR0_OFFSET) & 0xff); + IMMR_W32(ODR_OFFSET, UART_TX_EMPTY_DBELL); +} + +static void uart_tx_empty_handler(struct wqt_dev *priv) +{ + priv->uart_tx_ready = true; + wake_up(&priv->uart_tx_wait); +}
Since these all seem to be trivial functions, why go through the indirect function pointers at all? It seems that wqt_interrupt could do this just as well.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ +static int wqt_request_irq(struct wqt_dev *priv) +{ + int ret = 0;
no point initializing ret.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 1 line
arch/powerpc/boot/dts/mpc834x_mds.dts +1
-0+ mutex_lock(&priv->irq_mutex);
What data does the irq_mutex protect?
Arnd <><
Quoted 13 lines · Arnd Bergmann
Quoted 3 lines · Ira Snyder
This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.
Very interesting. I have seen multiple such drivers, and still plan to have one as well for the cell/axon platform.
Quoted 4 lines
This is the third posting of this driver. I got some feedback, and have corrected the problems. Stephen, thanks for the review! I also got some off-list feedback from a potential user, so I believe this is worth getting into mainline.
Quoted 7 lines
Sorry for not having replied earlier, I did not see first postings. While I did not see any major problems with your code, I think we should actually do it in a different way, which I have talked about at the Linux plumbers conference, and will again at the UKUUG Linux conference this weekend. You can find my slides from the previous talk at
http://userweb.kernel.org/%7Earnd/papers/plumbers08/plumbers-slides.pdf
I'll read through these slides as soon as I'm finished replying to this email :)
Quoted 4 lines
Since you have code and I only have plans, I suppose I can't make you wait for me to get a driver for your platform, but I hope we can join forces in the future and share code in the future.
I hope so too. I wrote this because I needed something working. I'm completely happy to help write code for another implementation.
Quoted 10 lines
My fundamental criticism of your code is that it is not flexible enough. We have a number of platforms that act as a PCI, PCIe or similar kind of device and want to transfer network data. At the same time, some of these want to use the same infrastructure for non-network data like MPI (ibverbs), block device or file system (9p). Even in your limited setup, it seems that you should be able to share more code between the two implementations you have posted here. When looking at the code, I found a lot of duplication between the drivers that you should be able to avoid.
I completely agree. I made the host and client drivers as similar as possible on purpose, to increase my own code sharing. In fact, if you diff them, you'll see that they are almost identical.
I decided that it would be better to post them up for review than try to fix every little problem.
Quoted 4 lines
My suggestion is to base this on the virtio infrastructure, and some of my colleagues at IBM already implemented a driver that uses virtio-net on a PCIe connection.
To be honest, I couldn't get that figured out. I looked at that code for a couple of weeks and couldn't make a lot of sense out of it. I couldn't find much really good documentation, either. I'd love to re-use more code than I have.
Quoted 26 lines
Arnd <><Now a few comments on your code:
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +0
-0arch/powerpc/boot/dts/mpc834x_mds.dts line 112
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +6
-0@@ -104,6 +104,13 @@ mode = "cpu"; }; + message-unit@8030 { + compatible = "fsl,mpc8349-mu"; + reg = <0x8030 0xd0>; + interrupts = <69 0x8>; + interrupt-parent = <&ipic>; + };What is a message unit? Is this the mailbox you use in the driver? We are facing a similar problem on Axon regarding probing of the virtual device because you can't easily tell from the device tree whether the machine is connected to something that is trying to communicate.
Quoted 3 lines
The message-unit does not seem to be network-specific, so it's not particularly nice to have the network unit grab that
of_device.
Yes, the message unit is how the Freescale reference manual describes this hardware. It is the mailbox hardware to generate interrupts, plus a few registers to transfer data.
The registers aren't network specific, but they are needed for this driver. That is how the driver generates PCI interrupts to notify the other side of the connection it has new data, etc.
Quoted 7 lines
Quoted 7 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +7
-0+config PCINET_FSL + tristate "PCINet Virtual Ethernet over PCI support (Freescale)" + depends on MPC834x_MDS && !PCI + select DMA_ENGINE + select FSL_DMA + help + When running as a PCI Agent, this driver will create a virtual
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +5
-0+ ethernet link running over the PCI bus, allowing simplified + communication with the host system. The host system will need + to use the corresponding driver. + + If in doubt, say N.Why 'depends on !PCI'? This means that you cannot build a kernel that is able to run both as host and endpoint for PCInet, right?
Yes, that is correct. I did this because the Linux PCI code does some relatively nasty things in agent mode. One thing that consistently crashed my box was running through the quirks list and trying to re-initialize an e100 that was in my box.
Remember, this is a PCI agent. It shouldn't be re-initializing the other hardware on the PCI bus.
Quoted 8 lines
Quoted 8 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +8
-0+config PCINET_HOST + tristate "PCINet Virtual Ethernet over PCI support (Host)" + depends on PCI + help + This driver will let you communicate with a PCINet client device + using a virtual ethernet link running over the PCI bus. This + allows simplified communication with the client system. +
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ This is inteded for use in a system that has a crate full of + computers running Linux, all connected by a PCI backplane. + + If in doubt, say N.Is this meant to be portable across different hardware implementations? If the driver can only work with
PCINET_FSLon the other side, you should probably mention that in the description.
I had hoped other people would come along and help port it to other hardware, like you yourself are suggesting. :) I'll add to the description.
Quoted 11 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 7 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +7
-0+config PCINET_DISABLE_CHECKSUM + bool "Disable packet checksumming" + depends on PCINET_FSL || PCINET_HOST + default n + help + Disable packet checksumming on packets received by the PCINet + driver. This gives a possible speed boost.Why make this one optional? Is there ever a reason to enable checksumming? If there is, how about making it tunable with ethtool?
I left it optional so I could turn it on and off easily. I have no strong feelings on keeping it optional.
Does the PCI bus reliably transfer data? I'm not sure. I left it there so that we could at least turn on checksumming if there was a problem.
Quoted 10 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +5
-0+struct circ_buf_desc { + __le32 sc; + __le32 len; + __le32 addr; +} __attribute__((__packed__));It would be useful to always force aligning the desciptors to the whole 32 bit and avoid the packing here. Unaligned accesses are inefficient on many systems.
I don't really know how to do that. I got a warning here from sparse telling me something about expensive pointer subtraction. Adding a dummy 32bit padding variable got rid of the warning, but I didn't change the driver.
Quoted 5 lines
Quoted 1 line
+typedef struct
circ_buf_desccbd_t;Also, don't pass structures by value if they don't fit into one or two registers.
These are only used for pointers to the buffer descriptors (in RAM on the Freescale) that hold packet information. I never copy them directly.
Quoted 4 lines
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+/* Buffer Descriptor Accessors */ +#define CBDW_SC(_cbd, _sc) iowrite32((_sc), &(_cbd)->sc) +#define CBDW_LEN(_cbd, _len) iowrite32((_len), &(_cbd)->len) +#define CBDW_ADDR(_cbd, _addr) iowrite32((_addr), &(_cbd)->addr)
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ +#define CBDR_SC(_cbd) ioread32(&(_cbd)->sc) +#define CBDR_LEN(_cbd) ioread32(&(_cbd)->len) +#define CBDR_ADDR(_cbd) ioread32(&(_cbd)->addr)We have found that accessing remote descriptors using mmio read is rather slow, and changed the code to always do local reads and remote writes.
Interesting. I don't know how you would get network speed doing this. X86 systems don't have a DMA conttroller. The entire purpose of making the Freescale do all the copying was to use its DMA controller.
Using the DMA controller to transfer all of the data took my transfer speed from ~3MB/sec to ~45MB/sec. While that is a good increase, it could be better. I should be able to hit close to 133MB/sec (the limit of PCI)
Quoted 4 lines
On the host side, I would argue that you should use
out_le32rather than iowrite32, because you are not operating on a PCI device but an OF device.
Yeah, you're probably correct. I can't remember if I tried this and had problems or not.
Quoted 12 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +5
-0+/* IMMR Accessor Helpers */ +#define IMMR_R32(_off) ioread32(priv->immr+(_off)) +#define IMMR_W32(_off, _val) iowrite32((_val), priv->immr+(_off)) +#define IMMR_R32BE(_off) ioread32be(priv->immr+(_off)) +#define IMMR_W32BE(_off, _val) iowrite32be((_val), priv->immr+(_off))IIRC, the IMMR is a platform resource, so the system should map those registers already and provide accessor functions for the registers you need. Simply allowing to pass an offset in here does not look clean.
Correct. This was done to make both sides as identical as possible. The Freescale exports the entire 1MB block of IMMR registers at PCI BAR0. So I have to use the offsets on the host side.
Quoted 1 line
From the client side, I could just map what I need, but that would make
the two drivers diverge. I was trying to keep them the same.
Quoted 7 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 2 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +2
-0+static void wqtuart_rx_char(struct uart_port *port, const char ch); +static void wqtuart_stop_tx(struct uart_port *port);You should try to avoid forward declarations for static functions. If you order the function implementation correctly, that will also give you the expected reading order in the driver.
Yep, I tried to do this. I couldn't figure out a sane ordering that would work. I tried to keep the network and uart as seperate as possible in the code.
Quoted 10 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 2 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +2
-0+struct wqt_dev; +typedef void (*wqt_irqhandler_t)(struct wqt_dev *);Much more importantly, never do forward declarations for global functions!
These belong into a header that is included by both the user and the definition.
Ok.
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 6 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +6
-0+struct wqt_dev { + /*--------------------------------------------------------------------*/ + /* OpenFirmware Infrastructure */ + /*--------------------------------------------------------------------*/ + struct of_device *op; + struct device *dev;Why the dev? You can always get that from the
of_device, right?
Yes. I stored it there to make it identical to the host driver. By doing this, both drivers have code that says "dev_debug(priv->dev, ...)" rather than:
Host:dev_debug(&priv->pdev->dev, ...)
Freescale:dev_debug(&priv->op->dev, ...)
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 2 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +2
-0+ int irq; + void __iomem *immr;I don't think immr is device specific.
Correct, it is not. Both the PCI subsystem and the OF subsystem will call probe() when a new device is detected. I stored it here so I could ioremap() them at probe() time, and to keep both drivers' code the same.
Quoted 7 lines
Quoted 7 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +7
-0+ struct mutex irq_mutex; + int interrupt_count; + + spinlock_t irq_lock; + struct wqt_irqhandlers handlers; + + /*--------------------------------------------------------------------*/
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +5
-0+ /* UART Device Infrastructure */ + /*--------------------------------------------------------------------*/ + struct uart_port port; + bool uart_rx_enabled; + bool uart_open;hmm, if you need a uart, that sounds like it should be a separate driver altogether. What is the relation between the network interface and the UART?
Yes, I agree. How do you make two Linux drivers that can be loaded for the same hardware at the same time? :) AFAIK, you cannot.
I NEED two functions accessible at the same time, network (to transfer data) and uart (to control my bootloader).
I use the uart to interact with the bootloader (U-Boot) and tell it where to tftp a kernel. I use the network to transfer the kernel.
So you see, I really do need them both at the same time. If you know a better way to do this, please let me know!
It was possible to write seperate U-Boot drivers, but only by being careful to not conflict in my usage of the hardware.
Quoted 5 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 1 line
arch/powerpc/boot/dts/mpc834x_mds.dts +1
-0+ struct workqueue_struct *wq;A per-device pointer to a
workqueue_structis unusual. What are you doing this for? How about using the system work queue?
Look through the code carefully, especially at uart_tx_work_fn(), which calls do_send_message().
do_send_message() can sleep for a long time. If I use the shared workqueue, I block ALL OTHER USERS of the queue for the time I am asleep. They cannot run.
The LDD3 book says not to use the shared workqueue if you are going to sleep for long periods. I followed their advice.
Quoted 8 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ /*--------------------------------------------------------------------*/ + /* Ethernet Device Infrastructure */ + /*--------------------------------------------------------------------*/ + struct net_device *ndev;Why make this a separate structure? If you have one of these per
net_device, you should embed thenet_deviceinto your own structure.
This structure is embedded in struct net_device! Look at how alloc_etherdev() works. You pass it the size of your private data structure and it allocates the space for you.
Quoted 6 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 1 line
arch/powerpc/boot/dts/mpc834x_mds.dts +1
-0+ struct tasklet_struct tx_complete_tasklet;Using a tasklet for tx processing sounds fishy because most of the network code already runs at softirq time. You do not gain anything by another softirq context.
I didn't want to run the TX cleanup routine at hard irq time, because it can potentially take some time to run. I would rather run it with hard interrupts enabled.
This DOES NOT do TX processing, it only frees skbs that have been transferred. I used the network stack to do as much as possible, of course.
Quoted 17 lines
Quoted 17 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +17
-0+/*----------------------------------------------------------------------------*/ +/* Status Register Helper Operations */ +/*----------------------------------------------------------------------------*/ + +static DEFINE_SPINLOCK(status_lock); + +static void wqtstatus_setbit(struct wqt_dev *priv, u32 bit) +{ + unsigned long flags; + + spin_lock_irqsave(&status_lock, flags); + IMMR_W32(OMR1_OFFSET, IMMR_R32(OMR1_OFFSET) | bit); + spin_unlock_irqrestore(&status_lock, flags); +} + +static void wqtstatus_clrbit(struct wqt_dev *priv, u32 bit) +{
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 6 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +6
-0+ unsigned long flags; + + spin_lock_irqsave(&status_lock, flags); + IMMR_W32(OMR1_OFFSET, IMMR_R32(OMR1_OFFSET) & ~bit); + spin_unlock_irqrestore(&status_lock, flags); +}This looks misplaced, as mentioned before.
I used these so I could have some remote status. Remember, this has to work for Linux -> U-Boot, not just Linux <-> Linux.
While the board is booting Linux, the RAM I used to hold buffer descriptors disappears! I need some shared status to make sure no stray accesses happen to random areas of memory.
The case this potentially happens is:
1) Start Freescale board, do NOT install the driver
2) Insert the driver into the host system
Now, if the host tries writing to the buffer descriptors, we've corrupted memory on the Freescale board.
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+static int wqtstatus_remote_testbit(struct wqt_dev *priv, u32 bit) +{ + return IMMR_R32(IMR1_OFFSET) & bit; +}Are you sure that do not need a spinlock here? The lock also prevents reordering the device access arount the code using the mmio data.
I don't think so. Note that this is a status bit on the remote side of the connection. My access ordering means nothing when the remote side is setting and clearing the bit.
Quoted 18 lines
Quoted 18 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +18
-0+/*----------------------------------------------------------------------------*/ +/* Message Sending and Processing Operations */ +/*----------------------------------------------------------------------------*/ + +static irqreturn_t wqt_interrupt(int irq, void *dev_id) +{ + struct wqt_dev *priv = dev_id; + u32 imisr, idr; + unsigned long flags; + + imisr = IMMR_R32(IMISR_OFFSET); + idr = IMMR_R32(IDR_OFFSET); + + if (!(imisr & 0x8)) + return IRQ_NONE; + + /* Clear all of the interrupt sources, we'll handle them next */ + IMMR_W32(IDR_OFFSET, idr);
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ + /* Lock over all of the handlers, so they cannot get called when + * the code doesn't expect them to be called */ + spin_lock_irqsave(&priv->irq_lock, flags);If this is in an interrupt handler, why disable the interrupts again? The same comment applies to many of the other places where you use
spin_lock_irqsaverather thanspin_lockorspin_lock_irq.
I tried to make the locking do only what was needed. I just couldn't get it correct unless I used spin_lock_irqsave(). I was able to get the system to deadlock otherwise. This is why I posted the driver for review, I could use some help here.
It isn't critical anyway. You can always use spin_lock_irqsave(), it is just a little slower, but it will always work :)
Quoted 45 lines
Quoted 45 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +45
-0+/* NOTE: All handlers are called with priv->irq_lock held */ + +static void empty_handler(struct wqt_dev *priv) +{ + /* Intentionally left empty */ +} + +static void net_start_req_handler(struct wqt_dev *priv) +{ + schedule_work(&priv->net_start_work); +} + +static void net_start_ack_handler(struct wqt_dev *priv) +{ + complete(&priv->net_start_completion); +} + +static void net_stop_req_handler(struct wqt_dev *priv) +{ + schedule_work(&priv->net_stop_work); +} + +static void net_stop_ack_handler(struct wqt_dev *priv) +{ + complete(&priv->net_stop_completion); +} + +static void net_tx_complete_handler(struct wqt_dev *priv) +{ + tasklet_schedule(&priv->tx_complete_tasklet); +} + +static void net_rx_packet_handler(struct wqt_dev *priv) +{ + wqtstatus_setbit(priv, PCINET_NET_RXINT_OFF); + netif_rx_schedule(priv->ndev, &priv->napi); +} + +static void uart_rx_ready_handler(struct wqt_dev *priv) +{ + wqtuart_rx_char(&priv->port, IMMR_R32(IMR0_OFFSET) & 0xff); + IMMR_W32(ODR_OFFSET, UART_TX_EMPTY_DBELL); +} + +static void uart_tx_empty_handler(struct wqt_dev *priv)
Quoted 9 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+{ + priv->uart_tx_ready = true; + wake_up(&priv->uart_tx_wait); +}Since these all seem to be trivial functions, why go through the indirect function pointers at all? It seems that
wqt_interruptcould do this just as well.
I did this so they can be disabled independently. It is VERY tricky to get the board started without the chance of corrupting memory. Write the code, you'll see :)
You have to deal with two systems that can read/write each other's memory, but DO NOT have any synchronization primitives.
Quoted 11 lines
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 4 lines
arch/powerpc/boot/dts/mpc834x_mds.dts +4
-0+ +static int wqt_request_irq(struct wqt_dev *priv) +{ + int ret = 0;no point initializing ret.
arch/powerpc/boot/dts/mpc834x_mds.dts
Quoted 1 line
arch/powerpc/boot/dts/mpc834x_mds.dts +1
-0+ mutex_lock(&priv->irq_mutex);What data does the
irq_mutexprotect?
priv->interrupt_count and the indirect handlers.
Note that I have two fundamentally seperate devices here. The network and the uart.
What if noone has brought up the network interface? You'd still expect the uart to work, right? In this case, I need to call request_irq() in TWO places, the network's open() and the uart's open().
There is only a single interrupt for this hardware. When only the uart is open, I only want the uart section of the irq handler enabled. This is where the indirect handlers come in. On uart open(), I call wqt_request_irq(), which will ONLY call request_irq() if it HAS NOT been called yet.
Here are a few examples:
Ex1: uart only
1) uart open() is called
2) wqt_request_irq() is called
3) the uart irq handling functions are enabled
4) request_irq() is called, hooking up the irq
5) the uart is started
The network by itself works exactly the same.
Ex2: uart first, then network while uart is open
1) uart open() is called
2) wqt_request_irq() is called
3) the uart irq handling functions are enabled
4) request_irq() is called, hooking up the irq handler
5) the uart is started
6) time passes
7) network open() is called
8) wqt_request_irq() is called
9) request_irq() is NOT CALLED THIS TIME, because it is already enabled
10) the network irq handling functions are enabled
11) the network is started
Thanks so much for the review! I hope we can work together to get something that can be merged into mainline Linux. I'm willing to write code, I just need some direction from more experienced kernel developers.
Ira
Quoted 6 lines · Ira Snyder
Quoted 4 lines · Arnd Bergmann
Why 'depends on !PCI'? This means that you cannot build a kernel that is able to run both as host and endpoint for PCInet, right?
Quoted 7 lines
Yes, that is correct. I did this because the Linux PCI code does some relatively nasty things in agent mode. One thing that consistently crashed my box was running through the quirks list and trying to re-initialize an e100 that was in my box.
Remember, this is a PCI agent. It shouldn't be re-initializing the other hardware on the PCI bus.
Yes, that makes sense. However, you should still be able to have the PCI code built into the kernel, as long as you prevent it from scanning the bus on the machine that is in agent/endpoint mode.
This should be made clear in the device tree. On the QS22 machine, we remove the "pci" device from the device tree, and add a "pcie-ep" device.
Quoted 5 lines
I left it optional so I could turn it on and off easily. I have no strong feelings on keeping it optional.
Does the PCI bus reliably transfer data? I'm not sure. I left it there so that we could at least turn on checksumming if there was a problem.
Yes, PCI guarantees reliable transfers.
Quoted 11 lines
Quoted 10 lines
drivers/net/pcinet.h line 28
Quoted 5 lines
drivers/net/pcinet.h new file +5
-0+struct circ_buf_desc { +__le32 sc; +__le32 len; +__le32 addr; +} __attribute__((__packed__ ));It would be useful to always force aligning the desciptors to the whole 32 bit and avoid the packing here. Unaligned accesses are inefficient on many systems.
Quoted 4 lines
I don't really know how to do that. I got a warning here from sparse telling me something about expensive pointer subtraction. Adding a dummy 32bit padding variable got rid of the warning, but I didn't change the driver.
Ok, I see. However, adding the packed attribute makes it more expensive to use.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
+typedef struct
circ_buf_desccbd_t;Also, don't pass structures by value if they don't fit into one or two registers.
These are only used for pointers to the buffer descriptors (in RAM on the Freescale) that hold packet information. I never copy them directly.
Ok, then you should not have a typedef.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/pcinet.h new file +4
-0+/* Buffer Descriptor Accessors */ +#define CBDW_SC (_cbd, _sc) iowrite32((_sc), &(_cbd)->sc) +#define CBDW_LEN (_cbd, _len) iowrite32((_len), &(_cbd)->len) +#define CBDW_ADDR (_cbd, _addr) iowrite32((_addr), &(_cbd)->addr)Quoted 9 lines
drivers/net/pcinet.h line 38
Quoted 4 lines
drivers/net/pcinet.h new file +4
-0+ +#define CBDR_SC (_cbd) ioread32(&(_cbd)->sc) +#define CBDR_LEN (_cbd) ioread32(&(_cbd)->len) +#define CBDR_ADDR (_cbd) ioread32(&(_cbd)->addr)We have found that accessing remote descriptors using mmio read is rather slow, and changed the code to always do local reads and remote writes.
Quoted 8 lines
Interesting. I don't know how you would get network speed doing this. X86 systems don't have a DMA conttroller. The entire purpose of making the Freescale do all the copying was to use its DMA controller.
Using the DMA controller to transfer all of the data took my transfer speed from ~3MB/sec to ~45MB/sec. While that is a good increase, it could be better. I should be able to hit close to 133MB/sec (the limit of PCI)
Then I think I misunderstood something about this driver. Are these descriptors accessed by the DMA engine, or by software? If it's the DMA engine accessing them, can you put the descriptors on both sides of the bus rather than just on one side?
Which side allocates them anyway? Since you use ioread32/iowrite32 on the ppc side, it looks like they are on the PCI host, which does not seem to make much sense, because the ppc memory is much closer to the DMA engine?
Obviously, you want the DMA engine to do the data transfers, but here, you use ioread32 for mmio transfers to the descriptors, which is slow.
Quoted 6 lines
Correct. This was done to make both sides as identical as possible. The Freescale exports the entire 1MB block of IMMR registers at PCI BAR0. So I have to use the offsets on the host side.
From the client side, I could just map what I need, but that would make the two drivers diverge. I was trying to keep them the same.
Ah, I see. We had the same problem on Axon, and I'm still looking for a good solution. The best option is probably to abstract the immr access in some way and provide a driver that implements them on top of PCI.
Quoted 9 lines
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+static void wqtuart_rx_char(struct uart_port *port, const char ch); +static void wqtuart_stop_tx(struct uart_port *port);You should try to avoid forward declarations for static functions. If you order the function implementation correctly, that will also give you the expected reading order in the driver.
Quoted 3 lines
Yep, I tried to do this. I couldn't figure out a sane ordering that would work. I tried to keep the network and uart as seperate as possible in the code.
I'd suggest splitting the uart code into a separate driver then.
Quoted 14 lines
Quoted 9 lines
drivers/net/pcinet_fsl.c line 56
Quoted 6 lines
drivers/net/pcinet_fsl.c new file +6
-0+struct wqt_dev { +/*--------------------------------------------------------------------*/ +/* OpenFirmware Infrastructure */ +/*--------------------------------------------------------------------*/ +struct of_device *op; +struct device *dev;Why the dev? You can always get that from the
of_device, right?Yes. I stored it there to make it identical to the host driver. By doing this, both drivers have code that says "
dev_debug(priv->dev, ...)" rather than:
Quoted 5 lines
Host:
dev_debug(&priv->pdev->dev, ...)Freescale:
dev_debug(&priv->op->dev, ...)
Ok. You can just store the dev pointer then, and leave out the op pointer. You can always do a container_of() to get back to it.
Quoted 6 lines
Yes, I agree. How do you make two Linux drivers that can be loaded for the same hardware at the same time? :) AFAIK, you cannot.
I NEED two functions accessible at the same time, network (to transfer data) and uart (to control my bootloader).
Quoted 8 lines
I use the uart to interact with the bootloader (U-Boot) and tell it where to tftp a kernel. I use the network to transfer the kernel.
So you see, I really do need them both at the same time. If you know a better way to do this, please let me know!
It was possible to write seperate U-Boot drivers, but only by being careful to not conflict in my usage of the hardware.
Ok, I see. I fear any nice solution would make the u-boot drivers much more complex.
Quoted 6 lines
Quoted 5 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ struct workqueue_struct *wq;A per-device pointer to a
workqueue_structis unusual. What are you doing this for? How about using the system work queue?
Quoted 6 lines
Look through the code carefully, especially at
uart_tx_work_fn(), which callsdo_send_message().
do_send_message()can sleep for a long time. If I use the shared workqueue, I block ALL OTHER USERS of the queue for the time I am asleep. They cannot run.
ok.
Quoted 2 lines
The LDD3 book says not to use the shared workqueue if you are going to sleep for long periods. I followed their advice.
ok.
Quoted 9 lines
Quoted 8 lines
drivers/net/pcinet_fsl.c line 81
Quoted 4 lines
drivers/net/pcinet_fsl.c new file +4
-0+/*--------------------------------------------------------------------*/ +/* Ethernet Device Infrastructure */ +/*--------------------------------------------------------------------*/ +struct net_device *ndev;Why make this a separate structure? If you have one of these per
net_device, you should embed thenet_deviceinto your own structure.
Quoted 3 lines
This structure is embedded in struct
net_device! Look at howalloc_etherdev()works. You pass it the size of your private data structure and it allocates the space for you.
right, I remember now. Unfortunately, alloc_etherdev is a little bit different from many other kernel interfaces.
Quoted 10 lines
Quoted 6 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ struct tasklet_struct tx_complete_tasklet;Using a tasklet for tx processing sounds fishy because most of the network code already runs at softirq time. You do not gain anything by another softirq context.
I didn't want to run the TX cleanup routine at hard irq time, because it can potentially take some time to run. I would rather run it with hard interrupts enabled.
sure.
Quoted 3 lines
This DOES NOT do TX processing, it only frees skbs that have been transferred. I used the network stack to do as much as possible, of course.
Most drivers now do that from the *rx* poll function, and call netif_rx_schedule when they get a tx interrupt.
Quoted 5 lines
Quoted 4 lines
If this is in an interrupt handler, why disable the interrupts again? The same comment applies to many of the other places where you use
spin_lock_irqsaverather thanspin_lockorspin_lock_irq.
Quoted 7 lines
I tried to make the locking do only what was needed. I just couldn't get it correct unless I used
spin_lock_irqsave(). I was able to get the system to deadlock otherwise. This is why I posted the driver for review, I could use some help here.It isn't critical anyway. You can always use
spin_lock_irqsave(), it is just a little slower, but it will always work :)
I like the documenting character of the spinlock functions. E.g. if you use spin_lock_irq() in a function, it is obvious that interrupts are enabled, and if you use spin_lock() on a lock that requires disabling interrupts, you know that interrupts are already off.
Quoted 4 lines
Thanks so much for the review! I hope we can work together to get something that can be merged into mainline Linux. I'm willing to write code, I just need some direction from more experienced kernel developers.
Great, I can certainly help with that. Please CC me on anything related to this driver.
Arnd <><
Quoted 15 lines · Arnd Bergmann
Quoted 6 lines · Ira Snyder
Quoted 4 lines · Arnd Bergmann
Why 'depends on !PCI'? This means that you cannot build a kernel that is able to run both as host and endpoint for PCInet, right?
Quoted 7 lines
Yes, that is correct. I did this because the Linux PCI code does some relatively nasty things in agent mode. One thing that consistently crashed my box was running through the quirks list and trying to re-initialize an e100 that was in my box.
Remember, this is a PCI agent. It shouldn't be re-initializing the other hardware on the PCI bus.
Quoted 8 lines
Yes, that makes sense. However, you should still be able to have the PCI code built into the kernel, as long as you prevent it from scanning the bus on the machine that is in agent/endpoint mode.
This should be made clear in the device tree. On the QS22 machine, we remove the "pci" device from the device tree, and add a "pcie-ep" device.
Ok, that makes perfect sense. I'll test it at some point and make sure that the kernel doesn't go through the quirks list, but it sounds reasonable to assume it doesn't.
Quoted 8 lines
Quoted 5 lines
I left it optional so I could turn it on and off easily. I have no strong feelings on keeping it optional.
Does the PCI bus reliably transfer data? I'm not sure. I left it there so that we could at least turn on checksumming if there was a problem.
Yes, PCI guarantees reliable transfers.
Great, I didn't know that. I'll turn it off unconditionally. Disabling the checksumming gave me a few extra MB/sec.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
drivers/net/pcinet.h line 28
Quoted 5 lines
drivers/net/pcinet.h new file +5
-0+struct circ_buf_desc { +__le32 sc; +__le32 len; +__le32 addr; +} __attribute__((__packed__ ));It would be useful to always force aligning the desciptors to the whole 32 bit and avoid the packing here. Unaligned accesses are inefficient on many systems.
Quoted 8 lines
Quoted 4 lines
I don't really know how to do that. I got a warning here from sparse telling me something about expensive pointer subtraction. Adding a dummy 32bit padding variable got rid of the warning, but I didn't change the driver.
Ok, I see. However, adding the packed attribute makes it more expensive to use.
Ok. Is there any way to make sure that the structure compiles to the same representation on the host and agent system without using packed?
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
+typedef struct
circ_buf_desccbd_t;Also, don't pass structures by value if they don't fit into one or two registers.
These are only used for pointers to the buffer descriptors (in RAM on the Freescale) that hold packet information. I never copy them directly.
Ok, then you should not have a typedef.
Ok, it is gone in my latest version.
Quoted 22 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/pcinet.h new file +4
-0+/* Buffer Descriptor Accessors */ +#define CBDW_SC (_cbd, _sc) iowrite32((_sc), &(_cbd)->sc) +#define CBDW_LEN (_cbd, _len) iowrite32((_len), &(_cbd)->len) +#define CBDW_ADDR (_cbd, _addr) iowrite32((_addr), &(_cbd)->addr)Quoted 9 lines
drivers/net/pcinet.h line 38
Quoted 4 lines
drivers/net/pcinet.h new file +4
-0+ +#define CBDR_SC (_cbd) ioread32(&(_cbd)->sc) +#define CBDR_LEN (_cbd) ioread32(&(_cbd)->len) +#define CBDR_ADDR (_cbd) ioread32(&(_cbd)->addr)We have found that accessing remote descriptors using mmio read is rather slow, and changed the code to always do local reads and remote writes.
Quoted 8 lines
Interesting. I don't know how you would get network speed doing this. X86 systems don't have a DMA conttroller. The entire purpose of making the Freescale do all the copying was to use its DMA controller.
Using the DMA controller to transfer all of the data took my transfer speed from ~3MB/sec to ~45MB/sec. While that is a good increase, it could be better. I should be able to hit close to 133MB/sec (the limit of PCI)
Quoted 5 lines
Then I think I misunderstood something about this driver. Are these descriptors accessed by the DMA engine, or by software? If it's the DMA engine accessing them, can you put the descriptors on both sides of the bus rather than just on one side?
I access the descriptors in software, and program the DMA controller to transfer the data. They are not directly used by the hardware.
I used the DMAEngine API to interact with the DMA controller. I tried programming them manually, but the DMAEngine API was about 10 MB/sec faster than I could achieve by hand.
See dma_async_copy_raw_to_buf() and dma_async_copy_buf_to_raw() in the PowerPC code.
The basics of the network driver are as follows:
1) PowerPC allocates 4k of RAM for buffer descriptors, and exposes it over PCI in BAR 1
2) Host initializes all buffer descriptors to zero
3) Host allocates RING_SIZE 64K skb's, and puts them in the RX ring
On PowerPC hard_start_xmit():
1) Find the next free buffer in the RX ring, get the address stored inside it
2) DMA the packet given to us by the network stack to that address
3) Mark the buffer descriptor used
4) Interrupt the host
On Host hard_start_xmit():
1) Find the next free buffer descriptor in the TX ring
2) dma_map_single() and put the address into the buffer descriptor
3) Mark the buffer descriptor as used
4) Interrupt the PowerPC
On PowerPC rx_napi(): (scheduled by interrupt)
1) Find the next dirty buffer in the TX ring, get the address and len
2) Allocate an skb of this len
3) DMA the data into the new skb
4) Pass the new skb up into the kernel
5) Mark the buffer as freeable
6) Loop until done
On Host rx_napi():
1) Find the next dirty buffer in the RX ring, get the pointer to it in the list of allocated skbs
2) Allocate a new 64K skb
3) Put the new skb into the buffer descriptors, mark it as clean
4) Push the skb (from the RX ring) into the kernel
5) Loop until done
So, you'll notice that I only copy the data over the PCI bus once, directly into the skb it is supposed to be going into. The buffer descriptors are there so I know where to find the skb in host memory across the PCI bus.
Hopefully that's a good description. :) It seems to me that both sides of the connection need to read the descriptors (to get packet length, clean up dirty packets, etc.) and write them (to set packet length, mark packets dirty, etc.) I just can't come up with something that is local-read / remote-write only.
Quoted 5 lines
Which side allocates them anyway? Since you use ioread32/iowrite32 on the ppc side, it looks like they are on the PCI host, which does not seem to make much sense, because the ppc memory is much closer to the DMA engine?
The PowerPC allocates them. They are accessible via PCI BAR1. They live in regular RAM on the PowerPC. I can't remember why I used ioread32/iowrite32 anymore. I'll try again with in_le32()/out_le32() on the PowerPC system, and see what happens.
Quoted 3 lines
Obviously, you want the DMA engine to do the data transfers, but here, you use ioread32 for mmio transfers to the descriptors, which is slow.
I didn't know it was slow :) Maybe this is why I had to make the MTU very large to get good speed. Using a standard 1500 byte MTU I get <10 MB/sec transfer speed. Using a 64K MTU, I get ~45MB/sec transfer speed.
Do I need to do any sort of flushing to make sure that the read has actually gone out of cache and into memory? When the host accesses the buffer descriptors over PCI, it can only view memory. If a write is still in the PowerPC cache, the host will get stale data.
Quoted 19 lines
Quoted 6 lines
Correct. This was done to make both sides as identical as possible. The Freescale exports the entire 1MB block of IMMR registers at PCI BAR0. So I have to use the offsets on the host side.
From the client side, I could just map what I need, but that would make the two drivers diverge. I was trying to keep them the same.
Ah, I see. We had the same problem on Axon, and I'm still looking for a good solution. The best option is probably to abstract the immr access in some way and provide a driver that implements them on top of PCI.
Quoted 9 lines
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+static void wqtuart_rx_char(struct uart_port *port, const char ch); +static void wqtuart_stop_tx(struct uart_port *port);You should try to avoid forward declarations for static functions. If you order the function implementation correctly, that will also give you the expected reading order in the driver.
Quoted 6 lines
Quoted 3 lines
Yep, I tried to do this. I couldn't figure out a sane ordering that would work. I tried to keep the network and uart as seperate as possible in the code.
I'd suggest splitting the uart code into a separate driver then.
How? In Linux we can only have one driver for a certain set of hardware. I use the messaging unit to do both network (interrupts and status bits) and uart (interrupts and message transfer).
Both the network and uart _must_ run at the same time. This way I can type into the bootloader prompt to start a network transfer, and watch it complete.
Remember, I can't have a real serial console plugged into this board. I'll be using this with about 150 boards in 8 separate chassis, which makes cabling a nightmare. I'm trying to do as much as possible with the PCI backplane.
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
drivers/net/pcinet_fsl.c line 56
Quoted 6 lines
drivers/net/pcinet_fsl.c new file +6
-0+struct wqt_dev { +/*--------------------------------------------------------------------*/ +/* OpenFirmware Infrastructure */ +/*--------------------------------------------------------------------*/ +struct of_device *op; +struct device *dev;Why the dev? You can always get that from the
of_device, right?Yes. I stored it there to make it identical to the host driver. By doing this, both drivers have code that says "
dev_debug(priv->dev, ...)" rather than:
Quoted 9 lines
Quoted 5 lines
Host:
dev_debug(&priv->pdev->dev, ...)Freescale:
dev_debug(&priv->op->dev, ...)Ok. You can just store the dev pointer then, and leave out the op pointer. You can always do a
container_of()to get back to it.
True, I didn't think of that. I'll make that change.
Quoted 9 lines
Quoted 9 lines
Yes, I agree. How do you make two Linux drivers that can be loaded for the same hardware at the same time? :) AFAIK, you cannot.
I NEED two functions accessible at the same time, network (to transfer data) and uart (to control my bootloader).
I use the uart to interact with the bootloader (U-Boot) and tell it where to tftp a kernel. I use the network to transfer the kernel.
Quoted 9 lines
Quoted 5 lines
So you see, I really do need them both at the same time. If you know a better way to do this, please let me know!
It was possible to write seperate U-Boot drivers, but only by being careful to not conflict in my usage of the hardware.
Ok, I see. I fear any nice solution would make the u-boot drivers much more complex.
Perhaps. I'm perfectly willing to port things to U-Boot. Especially if we can make something generic enough to be re-used by many different boards. Recently, another person on the U-Boot list has shown a need for this kind of solution.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
drivers/net/pcinet_fsl.c line 81
Quoted 4 lines
drivers/net/pcinet_fsl.c new file +4
-0+/*--------------------------------------------------------------------*/ +/* Ethernet Device Infrastructure */ +/*--------------------------------------------------------------------*/ +struct net_device *ndev;Why make this a separate structure? If you have one of these per
net_device, you should embed thenet_deviceinto your own structure.
Quoted 7 lines
Quoted 3 lines
This structure is embedded in struct
net_device! Look at howalloc_etherdev()works. You pass it the size of your private data structure and it allocates the space for you.right, I remember now. Unfortunately,
alloc_etherdevis a little bit different from many other kernel interfaces.
Yep. It sure is :)
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ struct tasklet_struct tx_complete_tasklet;Using a tasklet for tx processing sounds fishy because most of the network code already runs at softirq time. You do not gain anything by another softirq context.
I didn't want to run the TX cleanup routine at hard irq time, because it can potentially take some time to run. I would rather run it with hard interrupts enabled.
Quoted 9 lines
sure.
Quoted 3 lines
This DOES NOT do TX processing, it only frees skbs that have been transferred. I used the network stack to do as much as possible, of course.
Most drivers now do that from the *rx* poll function, and call
netif_rx_schedulewhen they get a tx interrupt.
That is an interesting concept. I'll look around the drivers/net tree and try to find one that works this way. It should be pretty easy to implement, though. I'll try it out.
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
If this is in an interrupt handler, why disable the interrupts again? The same comment applies to many of the other places where you use
spin_lock_irqsaverather thanspin_lockorspin_lock_irq.I tried to make the locking do only what was needed. I just couldn't get it correct unless I used
spin_lock_irqsave(). I was able to get the system to deadlock otherwise. This is why I posted the driver for review, I could use some help here.
Quoted 8 lines
Quoted 2 lines
It isn't critical anyway. You can always use
spin_lock_irqsave(), it is just a little slower, but it will always work :)I like the documenting character of the spinlock functions. E.g. if you use
spin_lock_irq()in a function, it is obvious that interrupts are enabled, and if you usespin_lock()on a lock that requires disabling interrupts, you know that interrupts are already off.
True. I just couldn't seem to get it right. I'll try again. Perhaps it was another bug in the driver that I hadn't found at the time.
Quoted 7 lines
Quoted 4 lines
Thanks so much for the review! I hope we can work together to get something that can be merged into mainline Linux. I'm willing to write code, I just need some direction from more experienced kernel developers.
Great, I can certainly help with that. Please CC me on anything related to this driver.
Will do. Please CC me on anything similar that you run across as well. :)
Ira
Quoted 7 lines · Ira Snyder
Quoted 6 lines · Arnd Bergmann
Quoted 4 lines · Ira Snyder
I don't really know how to do that. I got a warning here from sparse telling me something about expensive pointer subtraction. Adding a dummy 32bit padding variable got rid of the warning, but I didn't change the driver.
Quoted 6 lines
Quoted 3 lines
Ok, I see. However, adding the packed attribute makes it more expensive to use.
Ok. Is there any way to make sure that the structure compiles to the same representation on the host and agent system without using packed?
Only knowledge about the alignment on all the possible architectures ;-) As a simplified rule, always pad every struct member to the largest other member in the struct and always use explicitly sized types like __u8 or __le32.
Quoted 5 lines
Hopefully that's a good description. :) It seems to me that both sides of the connection need to read the descriptors (to get packet length, clean up dirty packets, etc.) and write them (to set packet length, mark packets dirty, etc.) I just can't come up with something that is local-read / remote-write only.
If I understand your description correctly, the only remote read is when the host accesses the buffer descriptors to find free space. Avoiding this read access may improve the latency a bit. In our ring buffer concept, both host and endpoint allocate a memory buffer that gets ioremapped into the remote side. Since you always need to read the descriptors from powerpc, you should probably keep them in powerpc memory, but you can change the code so that for finding the next free entry, the host will look in its own memory for the number of the next entry, and the powerpc side will write that when it consumes a descriptor to mark it as free.
Quoted 10 lines
Quoted 5 lines
Which side allocates them anyway? Since you use ioread32/iowrite32 on the ppc side, it looks like they are on the PCI host, which does not seem to make much sense, because the ppc memory is much closer to the DMA engine?
The PowerPC allocates them. They are accessible via PCI BAR1. They live in regular RAM on the PowerPC. I can't remember why I used ioread32/iowrite32 anymore. I'll try again with in_le32()/out_le32() on the PowerPC system, and see what happens.
Actually, if they are in powerpc RAM, you must not neither in_le32 nor ioread32. Both are only well-defined on I/O devices (local bus or PCI, respectively). Instead, you should use directly access the buffer using pointer dereferences, and use rmb()/wmb() to make sure anything you access is synchronized with the host.
Quoted 9 lines
Quoted 3 lines
Obviously, you want the DMA engine to do the data transfers, but here, you use ioread32 for mmio transfers to the descriptors, which is slow.
I didn't know it was slow :) Maybe this is why I had to make the MTU very large to get good speed. Using a standard 1500 byte MTU I get <10 MB/sec transfer speed. Using a 64K MTU, I get ~45MB/sec transfer speed.
Quoted 4 lines
Do I need to do any sort of flushing to make sure that the read has actually gone out of cache and into memory? When the host accesses the buffer descriptors over PCI, it can only view memory. If a write is still in the PowerPC cache, the host will get stale data.
The access over the bus is cache-coherent, unless you are on one of the more obscure powerpc implementations. This means you do not have a problem with data still being in cache. However, you need to make sure that data arrives in the right order. DMA read accesses over PCI may be reordered, and you need a wmb() between two memory stores if you want to be sure that the host sees them in the correct order.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Yep, I tried to do this. I couldn't figure out a sane ordering that would work. I tried to keep the network and uart as seperate as possible in the code.
I'd suggest splitting the uart code into a separate driver then.
How? In Linux we can only have one driver for a certain set of hardware. I use the messaging unit to do both network (interrupts and status bits) and uart (interrupts and message transfer).
Quoted 8 lines
Both the network and uart _must_ run at the same time. This way I can type into the bootloader prompt to start a network transfer, and watch it complete.
Remember, I can't have a real serial console plugged into this board. I'll be using this with about 150 boards in 8 separate chassis, which makes cabling a nightmare. I'm trying to do as much as possible with the PCI backplane.
When splitting out the hardware specific parts, I would write a device driver for the messaging unit that knows about neither the uart nor the network (or any other high-level protocol). It's a bit more complicated to load the two high-level drivers in that case, but one clean way to do it would be to instantiate a new bus-type from the MU driver and have that driver register devices for itself. Then you can load the high-level driver through udev or have them built into the kernel.
To get really fancy, you could find a way for the host to announce what protocols are supported on though the MU. A use case for that, which I have been thinking about before, would be to allow the host to set up direct virtual point-to-point networks between two endpoints, not involving the host at all once the device is up.
Arnd <><
Quoted 12 lines · Arnd Bergmann
Quoted 11 lines · Ira Snyder
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines · Ira Snyder
I don't really know how to do that. I got a warning here from sparse telling me something about expensive pointer subtraction. Adding a dummy 32bit padding variable got rid of the warning, but I didn't change the driver.
Ok, I see. However, adding the packed attribute makes it more expensive to use.
Quoted 8 lines
Quoted 2 lines
Ok. Is there any way to make sure that the structure compiles to the same representation on the host and agent system without using packed?
Only knowledge about the alignment on all the possible architectures ;-) As a simplified rule, always pad every struct member to the largest other member in the struct and always use explicitly sized types like
__u8or__le32.
Ok, I tried using:
struct circ_buf_desc {
__le32 sc;
__le32 len;
__le32 addr;
__le32 padding;
};The structure came out the same size on both x86 and powerpc, and the driver still works. I'll put this change in the driver.
Quoted 6 lines
Quoted 5 lines
Hopefully that's a good description. :) It seems to me that both sides of the connection need to read the descriptors (to get packet length, clean up dirty packets, etc.) and write them (to set packet length, mark packets dirty, etc.) I just can't come up with something that is local-read / remote-write only.
Quoted 11 lines
If I understand your description correctly, the only remote read is when the host accesses the buffer descriptors to find free space. Avoiding this read access may improve the latency a bit. In our ring buffer concept, both host and endpoint allocate a memory buffer that gets ioremapped into the remote side. Since you always need to read the descriptors from powerpc, you should probably keep them in powerpc memory, but you can change the code so that for finding the next free entry, the host will look in its own memory for the number of the next entry, and the powerpc side will write that when it consumes a descriptor to mark it as free.
There are a few remote reads (from the host).
1) in hard_start_xmit() to make sure the queue is stopped
2) in wqt_tx_complete() to find the buffer descriptors that have been consumed
3) in wqt_rx_napi() to find the buffer descriptors that have been dirtied
4) in wqt_rx_napi() to get the actual packet length of a dirty buffer
I /could/ come up with a scheme to use only writes, but it seems much too complicated for a little performance.
I'll keep it in mind, and change it during performance tuning (if needed).
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Which side allocates them anyway? Since you use ioread32/iowrite32 on the ppc side, it looks like they are on the PCI host, which does not seem to make much sense, because the ppc memory is much closer to the DMA engine?
The PowerPC allocates them. They are accessible via PCI BAR1. They live in regular RAM on the PowerPC. I can't remember why I used ioread32/iowrite32 anymore. I'll try again with in_le32()/out_le32() on the PowerPC system, and see what happens.
Quoted 6 lines
Actually, if they are in powerpc RAM, you must not neither
in_le32nor ioread32. Both are only well-defined on I/O devices (local bus or PCI, respectively). Instead, you should use directly access the buffer using pointer dereferences, and use rmb()/wmb() to make sure anything you access is synchronized with the host.
I've changed it to just do pointer dereferences, and it works just fine. I added appropriate wmb() (I think...)
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
Obviously, you want the DMA engine to do the data transfers, but here, you use ioread32 for mmio transfers to the descriptors, which is slow.
I didn't know it was slow :) Maybe this is why I had to make the MTU very large to get good speed. Using a standard 1500 byte MTU I get <10 MB/sec transfer speed. Using a 64K MTU, I get ~45MB/sec transfer speed.
Quoted 4 lines
Do I need to do any sort of flushing to make sure that the read has actually gone out of cache and into memory? When the host accesses the buffer descriptors over PCI, it can only view memory. If a write is still in the PowerPC cache, the host will get stale data.
Quoted 7 lines
The access over the bus is cache-coherent, unless you are on one of the more obscure powerpc implementations. This means you do not have a problem with data still being in cache. However, you need to make sure that data arrives in the right order. DMA read accesses over PCI may be reordered, and you need a
wmb()between two memory stores if you want to be sure that the host sees them in the correct order.
Ok.
Quoted 28 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Yep, I tried to do this. I couldn't figure out a sane ordering that would work. I tried to keep the network and uart as seperate as possible in the code.
I'd suggest splitting the uart code into a separate driver then.
How? In Linux we can only have one driver for a certain set of hardware. I use the messaging unit to do both network (interrupts and status bits) and uart (interrupts and message transfer).
Quoted 8 lines
Both the network and uart _must_ run at the same time. This way I can type into the bootloader prompt to start a network transfer, and watch it complete.
Remember, I can't have a real serial console plugged into this board. I'll be using this with about 150 boards in 8 separate chassis, which makes cabling a nightmare. I'm trying to do as much as possible with the PCI backplane.
When splitting out the hardware specific parts, I would write a device driver for the messaging unit that knows about neither the uart nor the network (or any other high-level protocol). It's a bit more complicated to load the two high-level drivers in that case, but one clean way to do it would be to instantiate a new bus-type from the MU driver and have that driver register devices for itself. Then you can load the high-level driver through udev or have them built into the kernel.
Quoted 6 lines
To get really fancy, you could find a way for the host to announce what protocols are supported on though the MU. A use case for that, which I have been thinking about before, would be to allow the host to set up direct virtual point-to-point networks between two endpoints, not involving the host at all once the device is up.
This is getting pretty complicated, especially for someone just getting started writing drivers. :)
It seems like you'd need a set of functions to access specific interrupts / message boxes, maybe something like the gpio interface (where each gpio pin has an integer number).
Both sides have to agree on what each interrupt means, and which mailbox data transfer happens in. So each driver would still be pretty closely tied to the hardware.
It seems like too much for someone like me to design. I'll leave it up to the pro's.
Ira
On Tue, Nov 04, 2008 at 09:23:03PM +0100, Arnd Bergmann wrote:
Big snip.
Quoted 6 lines
Quoted 6 lines
I tried to make the locking do only what was needed. I just couldn't get it correct unless I used
spin_lock_irqsave(). I was able to get the system to deadlock otherwise. This is why I posted the driver for review, I could use some help here.
Quoted 8 lines
Quoted 2 lines
It isn't critical anyway. You can always use
spin_lock_irqsave(), it is just a little slower, but it will always work :)I like the documenting character of the spinlock functions. E.g. if you use
spin_lock_irq()in a function, it is obvious that interrupts are enabled, and if you usespin_lock()on a lock that requires disabling interrupts, you know that interrupts are already off.
Ok, I've started addressing your comments. I'm having trouble with the locking again. Let me see if my understanding is correct:
spin_lock_irqsave() disables interrupts and saves the interrupt state spin_unlock_irqrestore() MAY re-enable interrupts based on the saved flags
spin_lock_irq() disables interrupts, and always turns them back on when spin_unlock_irq() is called
spin_lock_bh() disables softirqs, spin_unlock_bh() re-enables them
spin_lock() and spin_unlock() are just regular spinlocks
So, since interrupts are disabled while my interrupt handler is running, I think I should be able to use spin_lock() and spin_unlock(), correct?
But sparse gives me the following warning:
wqt.c:185:9: warning: context imbalance in 'wqt_interrupt': wrong count at exit
wqt.c:185:9: context 'lock': wanted 0, got 1
If I'm using spin_lock_irqsave() and spin_lock_irqrestore() I do not get the same warnings. Therefore I must have some misunderstanding :)
Thanks,
Ira
Quoted 2 lines · Ira Snyder
So, since interrupts are disabled while my interrupt handler is running, I think I should be able to use
spin_lock()andspin_unlock(), correct?
yes.
Quoted 6 lines
But sparse gives me the following warning:
wqt.c:185:9: warning: context imbalance in 'wqt_interrupt': wrong count at exit
wqt.c:185:9: context 'lock': wanted 0, got 1If I'm using
spin_lock_irqsave()andspin_lock_irqrestore()I do not get the same warnings. Therefore I must have some misunderstanding :)
I've seen something like that before, I think it was a bug either in sparse or in the powerpc platform code. Try updating both the kernel and sparse and see if it still happens.
Arnd <><
Quoted 11 lines · Arnd Bergmann
Quoted 2 lines · Ira Snyder
So, since interrupts are disabled while my interrupt handler is running, I think I should be able to use
spin_lock()andspin_unlock(), correct?yes.
Quoted 4 lines
But sparse gives me the following warning:
wqt.c:185:9: warning: context imbalance in 'wqt_interrupt': wrong count at exit
wqt.c:185:9: context 'lock': wanted 0, got 1
Quoted 8 lines
Quoted 2 lines
If I'm using
spin_lock_irqsave()andspin_lock_irqrestore()I do not get the same warnings. Therefore I must have some misunderstanding :)I've seen something like that before, I think it was a bug either in sparse or in the powerpc platform code. Try updating both the kernel and sparse and see if it still happens.
Ok, I had the latest sparse, but I've been developing the host driver against Ubuntu 8.04's kernel, which is based on 2.6.24. Using Linus' git tree resolved the errors from sparse.
I ended up finding that the dma_mapping_error() API has changed, and so I updated the driver for that. It should build against 2.6.28-rc now. I'll post it in a little while.
Ira