Skip to contentSkip to conversation

net: add PCINet driver

RFC PATCH Superseded Ira Snyder 6 participants 28 messages Jan 7, 2009 – Jan 16, 2009

Subsystems (4)
lore
Share & tools
lore

Ira Snyder

author
#0 lore raw

RFC PATCH v5 1/1 net: add PCINet driver

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

I have implemented client support for the Freescale MPC8349EMDS board, which is capable of running in PCI Agent mode (It acts like a PCI card, but is a complete PowerPC computer, running Linux). It is almost certainly trivially ported to any MPC83xx system.

It was developed to work in a CompactPCI crate of computers, one of which is a relatively standard x86 system (acting as the host) and many PowerPC systems (acting as clients).

I have tested with two boards in my system. The host is a 1066MHz Pentium3-M. I am able to achieve the following bandwidth, as measured with netperf's TCP_STREAM test. Some improvement would be nice. I'm out of ideas and need suggestions at this point.

Host -> Board1 (Board2 idle): 18 MB/sec
Host -> Board1, Board2: 18 MB/sec
Board1 -> Host (Board2 idle): 32 MB/sec
Board1, Board2 -> Host: 44 MB/sec

RFC v4 -> RFC v5:
* convert to net_device_ops infrastructure
* fix fallout from NAPI API changes
* use seperate DMA channels for RX and TX
* use skb_dma_map()/skb_dma_unmap() infrastructure
* unify the RX and TX DMA routines
* fix uart allocation with multiple boards in a single system
* mask interrupts when not in use
* remove status bits, relying on the state of interrupts instead
* fix circular locking violation reported by the locking validator

RFC v3 -> RFC v4:
* dropped NETIF_F_NO_CSUM from struct net_device->features, it was causing nfsroot to fail due to UDP checksum errors

RFC v2 -> RFC v3:
* use inline functions for accessing struct circ_buf_desc
* use pointer dereferencing on PowerPC local memory instead of ioread32()
* move IMMR and buffer descriptor accessors inside drivers
* update for dma_mapping_error() API changes
* use minimal locking primitives (i.e. spin_lock() instead of _irqsave())
* always disable checksumming, PCI is reliable
* replace typedef cbd_t with struct circ_buf_desc
* use get_immrbase() to get IMMR register offsets

RFC v1 -> RFC v2:
* remove vim modelines
* use net_device->name in request_irq(), for irqbalance
* remove unneccesary wqt_get_stats(), use default get_stats() instead
* use dev_printk() and friends
* add message unit to MPC8349EMDS dts file

---
This is the fifth RFC posting of this driver. I haven't gotten much feedback recently. I'd really like to see some version of this go upstream, so I appreciate your review.

Thanks to all of those who have posted comments about the driver.

Remaining Issues:
1) Naming
The name wqt originally stood for "workqueue test". That test driver eventually evolved into this one. I'm open to suggestions for names.
2) IMMR Usage
In the Freescale client driver, I ioremap() the whole set of board control registers, even though I don't make use of most of them. This was done so that the register offsets could be shared between the drivers in pcinet_hw.h
3) Hardcoded DMA Window Address
In the Freescale client driver, the DMA window address (0x80000000) is hardcoded into the DMA transfer code. Suggestions on how to get this value at runtime are welcome (if it matters).
4) Speed
This driver is relatively fast, but not quite fast enough for my application. I'm out of ideas, and I need some help speeding this up.

Rationale behind some decisions:
1) Buffer Descriptors in client memory
I chose to put the buffer descriptors in client memory rather than host memory. It seemed more logical to me at the time. In my usage of this driver, I'll have 19 boards + 1 host per cPCI chassis. I thought this would cut down on the number of PCI accesses needed. I'm willing to make changes.
2) Usage of client DMA controller for all data transfers
This was done purely for speed. I tried using the CPU to transfer all data, and it is very slow: about 3MB/sec. Using the DMA controller brings the transfer rate up to about 30MB/sec.
3) Static 1GB DMA window
Maintaining changing DMA windows while DMA are in flight seemed like it would overcomplicate the design.
4) Serial Driver
Unfortunately, there are two drivers here. I needed a method to with the U-Boot bootloader on these boards without plugging in a physical cable. With 19 clients + 1 host per chassis, the cable clutter is worth avoiding.

I'll post both U-Boot drivers to their mailing list once this driver is finalized.

Thanks,
Ira

 arch/powerpc/boot/dts/mpc834x_mds.dts |    7 +
 drivers/net/Kconfig                   |   29 +
 drivers/net/Makefile                  |    3 +
 drivers/net/pcinet.h                  |   62 ++
 drivers/net/pcinet_fsl.c              | 1335 +++++++++++++++++++++++++++++++++
 drivers/net/pcinet_host.c             | 1318 ++++++++++++++++++++++++++++++++
 drivers/net/pcinet_hw.h               |   77 ++
 7 files changed, 2831 insertions(+), 0 deletions(-)
 create mode 100644 drivers/net/pcinet.h
 create mode 100644 drivers/net/pcinet_fsl.c
 create mode 100644 drivers/net/pcinet_host.c
 create mode 100644 drivers/net/pcinet_hw.h
arch/powerpc/boot/dts/mpc834x_mds.dts +7 -0
index d9adba0..5c7617d 100644
@@ -104,6 +104,13 @@
 			mode = "cpu";
 		};
 
+		message-unit@8030 {
+			compatible = "fsl,mpc8349-mu";
+			reg = <0x8030 0xd0>;
+			interrupts = <69 0x8>;
+			interrupt-parent = <&ipic>;
+		};
+
 		dma@82a8 {
 			#address-cells = <1>;
 			#size-cells = <1>;
drivers/net/Kconfig +29 -0
index 9a18270..3a8ad4d 100644
@@ -2298,6 +2298,35 @@ config UGETH_TX_ON_DEMAND
 	bool "Transmit on Demand support"
 	depends on UCC_GETH
 
+config PCINET_FSL
+	tristate "PCINet Virtual Ethernet over PCI support (Freescale)"
+	depends on MPC834x_MDS
+	select DMA_ENGINE
+	select FSL_DMA
+	help
+	  When running as a PCI Agent, this driver will create a virtual
+	  ethernet link running over the PCI bus, allowing simplified
+	  communication with the host system. The host system will need
+	  to use the corresponding driver.
+
+	  If in doubt, say N.
+
+config PCINET_HOST
+	tristate "PCINet Virtual Ethernet over PCI support (Host)"
+	depends on PCI
+	help
+	  This driver will let you communicate with a PCINet client device
+	  using a virtual ethernet link running over the PCI bus. This
+	  allows simplified communication with the client system.
+
+	  This is inteded for use in a system that has a crate full of
+	  computers running Linux, all connected by a PCI backplane.
+
+	  The currently, the only board implementing client support is the
+	  Freescale MPC8349EMDS. Enable support with CONFIG_PCINET_FSL=y.
+
+	  If in doubt, say N.
+
 config MV643XX_ETH
 	tristate "Marvell Discovery (643XX) and Orion ethernet support"
 	depends on MV64360 || MV64X60 || (PPC_MULTIPLATFORM && PPC32) || PLAT_ORION
drivers/net/Makefile +3 -0
index e5c34b4..664849a 100644
@@ -30,6 +30,9 @@ gianfar_driver-objs := gianfar.o \
 obj-$(CONFIG_UCC_GETH) += ucc_geth_driver.o
 ucc_geth_driver-objs := ucc_geth.o ucc_geth_mii.o ucc_geth_ethtool.o
 
+obj-$(CONFIG_PCINET_FSL) += pcinet_fsl.o
+obj-$(CONFIG_PCINET_HOST) += pcinet_host.o
+
 #
 # link order important here
 #
drivers/net/pcinet.h new file +62 -0

62 lines not on this page · raw message

drivers/net/pcinet_fsl.c new file +1335 -0

1,335 lines not on this page · raw message

drivers/net/pcinet_host.c new file +1318 -0

1,318 lines not on this page · raw message

drivers/net/pcinet_hw.h new file +77 -0

77 lines not on this page · raw message

Signature · 1 line
1.5.4.3
maintainer
#1 lore raw

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

Quoted 3 lines

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the x86 host and the powerpc nodes.

You haven't even mentioned this UART aspect even indirectly in the commit message.

This just looks like yet another set of virtualization drivers to me. You could have just have easily built this using your own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that will likely improve your throughput, it has console drivers that distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done so much better using facilities we already have that are heavily optimized and userland understands already.

author
#2 lore raw
Quoted 7 lines · David Miller

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

Quoted 3 lines

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

Quoted 7 lines

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the x86 host and the powerpc nodes.

Exactly right. I needed it to tell the U-Boot bootloader to tftp the kernel and boot the board. These boards don't keep their PCI settings (assigned by the BIOS at boot) over a soft or hard reset.

I couldn't think of a better way to get them started.

Quoted 3 lines

You haven't even mentioned this UART aspect even indirectly in the commit message.

Sorry, I'll add something about it.

Quoted 11 lines

This just looks like yet another set of virtualization drivers to me. You could have just have easily built this using your own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that will likely improve your throughput, it has console drivers that distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done so much better using facilities we already have that are heavily optimized and userland understands already.

I've had a really hard time understanding the virtio code. I haven't been able to find much in the way of documentation for it. Can you point me to some code that does anything vaguely similar to what you are suggesting?

Arnd Bergmann said that there is a similar driver (for different hardware) that uses virtio, but it is very slow, because it doesn't use the DMA controller to transfer data. I need at very minimum 40MB/sec of client -> host data transfer.

Thanks for the comments,
Ira

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw
Quoted 45 lines · Ira Snyder
Quoted 7 lines · David Miller

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

Quoted 3 lines

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

Quoted 7 lines

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the x86 host and the powerpc nodes.

Exactly right. I needed it to tell the U-Boot bootloader to tftp the kernel and boot the board. These boards don't keep their PCI settings (assigned by the BIOS at boot) over a soft or hard reset.

I couldn't think of a better way to get them started.

Quoted 3 lines

You haven't even mentioned this UART aspect even indirectly in the commit message.

Sorry, I'll add something about it.

Quoted 11 lines

This just looks like yet another set of virtualization drivers to me. You could have just have easily built this using your own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that will likely improve your throughput, it has console drivers that distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done so much better using facilities we already have that are heavily optimized and userland understands already.

I've had a really hard time understanding the virtio code. I haven't been able to find much in the way of documentation for it. Can you point me to some code that does anything vaguely similar to what you are suggesting?

Quoted 5 lines

Arnd Bergmann said that there is a similar driver (for different hardware) that uses virtio, but it is very slow, because it doesn't use the DMA controller to transfer data. I need at very minimum 40MB/sec of client -> host data transfer.

Rusty, since you wrote the virtio code, can you point me at the things I would need to implement to use virtio over the PCI bus.

The guests (PowerPC computers running Linux) are PCI cards in the host system (an Intel Pentium3-M system). The guest computers can access all of the host's memory. The guests provide a 1MB (movable) window into their memory.

The PowerPC computers also have a DMA controller, which I've used to get better throughput from my driver. I have a way to create interrupts to both the host and guest systems.

I've read your paper titled: "virtio: Towards a De-Facto Standard For Virtual I/O Devices"

If I read that correctly, then I should implement all of the functions in struct virtqueue_ops appropriately, and the existing virtio drivers should just work. The only concern I have there is how to make guest -> host transfer use the DMA controller. I've done all programming of it from the guest kernel, using the Linux DMAEngine API.

Are there any other implementations other than virtio_ring?

I appreciate any input you can give. If I should be CCing someone else, just let me know and I'll drop you from the CC list.

Thanks,
Ira

#4 lore raw
Quoted 8 lines · Ira Snyder

The guests (PowerPC computers running Linux) are PCI cards in the host system (an Intel Pentium3-M system). The guest computers can access all of the host's memory. The guests provide a 1MB (movable) window into their memory.

The PowerPC computers also have a DMA controller, which I've used to get better throughput from my driver. I have a way to create interrupts to both the host and guest systems.

That looks -very- similar to the PCI driver for CAB and Cell triblades that was, I think, submitted a while ago. Arnd what's the status with that driver ?

Cheers,
Ben.

#5 lore raw
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 5 lines · Ira Snyder

The guests (PowerPC computers running Linux) are PCI cards in the host system (an Intel Pentium3-M system). The guest computers can access all of the host's memory. The guests provide a 1MB (movable) window into their memory.

Quoted 7 lines
Quoted 3 lines

The PowerPC computers also have a DMA controller, which I've used to get better throughput from my driver. I have a way to create interrupts to both the host and guest systems.

That looks -very- similar to the PCI driver for CAB and Cell triblades that was, I think, submitted a while ago. Arnd what's the status with that driver ?

Complicated ;-)

There were two device drivers for the Axon PCIe endpoint (from Mercury and IBM), but both were abandoned due to being too complex to get into a mergeable stage, and all the original developers are doing other work now.

I'm maintaining the IBM driver now, but have no plans to submit that for inclusion. I hope that I can soon find more time to work on a replacement driver that will provide the right abstraction through virtio for kernel drivers, plus a ibverbs user interface for direct application programming. It should also have pluggable backends to support not only Cell hardware but anything with a similar hardware implementation.

I did make a presentation about this at the Plumbers Conference, see
http://userweb.kernel.org/%7Earnd/papers/plumbers08/plumbers-slides.pdf.
Since then, I have come to a much clearer idea of what needs to be done, but not more actual code.

	Arnd <><
#6 lore raw
In reply to Ira Snyder
Quoted 12 lines · On Friday 09 January 2009 08:21:27 Ira Snyder wrote:

Rusty, since you wrote the virtio code, can you point me at the things I would need to implement to use virtio over the PCI bus.

The guests (PowerPC computers running Linux) are PCI cards in the host system (an Intel Pentium3-M system). The guest computers can access all of the host's memory. The guests provide a 1MB (movable) window into their memory.

The PowerPC computers also have a DMA controller, which I've used to get better throughput from my driver. I have a way to create interrupts to both the host and guest systems.

Quoted 8 lines

I've read your paper titled: "virtio: Towards a De-Facto Standard For Virtual I/O Devices"

If I read that correctly, then I should implement all of the functions in struct virtqueue_ops appropriately, and the existing virtio drivers should just work. The only concern I have there is how to make guest -> host transfer use the DMA controller. I've done all programming of it from the guest kernel, using the Linux DMAEngine API.

Hi Ira,

Interesting system: the guest being able to access the host's memory but not (fully) vice-versa makes this a little different from the current implementations where that was assumed. virtio assumes that the guest will publish buffers and someone else (ie. the host) will access them.

Quoted 1 line

Are there any other implementations other than virtio_ring?

There were some earlier implementations, but they were trivial. And not particularly helpful in your case.

In summary, yes, it should work quite nicely, but rather than publishing the buffers to the host, the host will need to tell the guest where it wants them transferred (possibly using that 1M window, and a notification mechanism). That will be the complex part I think.

Cheers,
Rusty.

author
#7 lore raw
Quoted 22 lines · Rusty Russell
Quoted 12 lines · On Friday 09 January 2009 08:21:27 Ira Snyder wrote:

Rusty, since you wrote the virtio code, can you point me at the things I would need to implement to use virtio over the PCI bus.

The guests (PowerPC computers running Linux) are PCI cards in the host system (an Intel Pentium3-M system). The guest computers can access all of the host's memory. The guests provide a 1MB (movable) window into their memory.

The PowerPC computers also have a DMA controller, which I've used to get better throughput from my driver. I have a way to create interrupts to both the host and guest systems.

Quoted 8 lines

I've read your paper titled: "virtio: Towards a De-Facto Standard For Virtual I/O Devices"

If I read that correctly, then I should implement all of the functions in struct virtqueue_ops appropriately, and the existing virtio drivers should just work. The only concern I have there is how to make guest -> host transfer use the DMA controller. I've done all programming of it from the guest kernel, using the Linux DMAEngine API.

Quoted 4 lines

Hi Ira,

Interesting system: the guest being able to access the host's memory but not (fully) vice-versa makes this a little different from the current implementations where that was assumed. virtio assumes that the guest will publish buffers and someone else (ie. the host) will access them.

The guest system /could/ publish all of its RAM, but with 256MB per board, 19 boards per cPCI crate, that's way too much for a 32-bit PC to map into it's memory space. That's the real reason I use the 1MB windows. I could make them bigger (16MB would be fine, I think), but I doubt it would make much of a difference to the implementation.

Quoted 6 lines
Quoted 1 line

Are there any other implementations other than virtio_ring?

There were some earlier implementations, but they were trivial. And not particularly helpful in your case.

In summary, yes, it should work quite nicely, but rather than publishing the buffers to the host, the host will need to tell the guest where it wants them transferred (possibly using that 1M window, and a notification mechanism). That will be the complex part I think.

I've got a system like that right now in the driver that started this thread. I tried to mimic the buffer descriptor rings in a network card. The virtio system doesn't seem to be too different. I can probably work something out.

What's keeping me from getting started is what to do with the virtqueue_ops structure once I've got the functions to fill it in. There isn't a register_virtqueue_ops() function. I'm not at all sure when to use a struct virtio_driver or a struct virtio_device. I hunch I need one of these.

If I've understood this whole thing correctly, then I should be able to register a virtio_ops on both sides, and then insmod virtio_net and virtio_console, and have them "just work".

I guess what I'm looking for is something trivial, something that'll insmod and printk() whenever one of the virtqueue_ops pointers is called. Something like a template. That's how I really learned when all of the uart functions are called, for example.

Thanks,
Ira

#8 lore raw
Quoted 8 lines · Ira Snyder
Quoted 6 lines · Rusty Russell

Interesting system: the guest being able to access the host's memory but not (fully) vice-versa makes this a little different from the current implementations where that was assumed. virtio assumes that the guest will publish buffers and someone else (ie. the host) will access them.

Quoted 5 lines

The guest system /could/ publish all of its RAM, but with 256MB per board, 19 boards per cPCI crate, that's way too much for a 32-bit PC to map into it's memory space. That's the real reason I use the 1MB windows. I could make them bigger (16MB would be fine, I think), but I doubt it would make much of a difference to the implementation.

The way we do it in the existing driver for cell, both sides export just a little part of their memory to the other side, and they also both get access to one channel of the DMA engine, which is enough to transfer larger data sections, as the DMA engine has access to all the memory on both sides.

	Arnd <><
author
#9 lore raw
Quoted 15 lines · Arnd Bergmann
Quoted 8 lines · Ira Snyder
Quoted 6 lines · Rusty Russell

Interesting system: the guest being able to access the host's memory but not (fully) vice-versa makes this a little different from the current implementations where that was assumed. virtio assumes that the guest will publish buffers and someone else (ie. the host) will access them.

Quoted 5 lines

The guest system /could/ publish all of its RAM, but with 256MB per board, 19 boards per cPCI crate, that's way too much for a 32-bit PC to map into it's memory space. That's the real reason I use the 1MB windows. I could make them bigger (16MB would be fine, I think), but I doubt it would make much of a difference to the implementation.

Quoted 5 lines

The way we do it in the existing driver for cell, both sides export just a little part of their memory to the other side, and they also both get access to one channel of the DMA engine, which is enough to transfer larger data sections, as the DMA engine has access to all the memory on both sides.

So do you program one channel of the DMA engine from the host side and another channel from the guest side?

I tried to avoid having the host program the DMA controller at all. Using the DMAEngine API on the guest did better than I could achieve by programming the registers manually. I didn't use chaining or any of the fancier features in my tests, though.

Ira

#10 lore raw
Quoted 2 lines · Ira Snyder

So do you program one channel of the DMA engine from the host side and another channel from the guest side?

Yes.

Quoted 4 lines

I tried to avoid having the host program the DMA controller at all. Using the DMAEngine API on the guest did better than I could achieve by programming the registers manually. I didn't use chaining or any of the fancier features in my tests, though.

Our driver unfortunately does not use the DMA API, but it clearly should. What this means in your case is that you would need to port the freescale DMA engine driver to the host side, to operate on the PCI device. Not sure about how specifically this would work on fsl hardware, but it seems generally doable.

	Arnd <><
author
#11 lore raw
Quoted 11 lines · Arnd Bergmann
Quoted 2 lines · Ira Snyder

So do you program one channel of the DMA engine from the host side and another channel from the guest side?

Yes.

Quoted 4 lines

I tried to avoid having the host program the DMA controller at all. Using the DMAEngine API on the guest did better than I could achieve by programming the registers manually. I didn't use chaining or any of the fancier features in my tests, though.

Quoted 6 lines

Our driver unfortunately does not use the DMA API, but it clearly should. What this means in your case is that you would need to port the freescale DMA engine driver to the host side, to operate on the PCI device. Not sure about how specifically this would work on fsl hardware, but it seems generally doable.

The only problem with that is that you cannot route interrupts from the DMA controller over PCI with the PowerPC core running. Which makes it mostly useless for this case.

It /could/ work with polling mmio reads of the DMA controller register, but as you said, mmio reads are slow, and that would be best avoided. I don't think porting the driver is the right way to go.

If I haven't said it before, let me say it now: the DMA controller can access all of the host's memory as well as all of the guest's memory.

I have been studying the virtio and virtio_ring systems, and I now have a good idea about how to implement virtio-over-pci. I'm having trouble coming up with a good scheme to use the DMA controller for all the data transfers, though.

It is fairly easy to use the DMA controller in the virtio_net case, since the virtio_net driver allocates all of the RX buffers ahead of time. The guest can DMA directly from host TX into guest RX, then mark the host TX buffers consumed. The same is true in reverse: the guest can DMA directly from it's TX into the host RX, then mark the host RX buffers consumed.

I'm not sure how to handle the bidirectional virtqueue case, though. Such as the upcoming changes to virtio_net or virtio_blk. It seems like you'd need to copy pages in this case, since you want to use the DMA to transfer them.

I have come up with a nice scheme to maintain the buffer descriptor tables in both host and guest memory without doing anything except mmio writes. The scheme is almost identical to virtio_ring, with the exception that both machines need a copy of vring->desc's flags and next fields, to avoid the reads in virtio_rings detach_buf().

Maybe I'm overoptimistic in thinking that two instances of virtio_net could talk to each other if the RX and TX queues were hooked up accordingly in the find_vq() function. Has anyone actually tried this?

To anyone that is trying to learn virtio, I dummied up this mock driver that registers a virtio_device for every MPC8349EMDS board in the system. The virtqueues don't actually do anything, but they do print messages so you know when they're used. It is almost certainly buggy and horrible code. I'm only reproducing it here because it helped me get started.

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>
#include <linux/pci.h>
#include <linux/virtio.h>
#include <linux/virtio_config.h>
#include <linux/virtio_net.h>

#include "pcinet_hw.h"

static const char driver_name[] = "vdev";

struct vpt_dev {
	struct virtio_device vdev;
	u8 status;

	void __iomem *immr;
	void __iomem *netregs;

	#define NUM_VQS 4
	struct virtqueue *vqs[NUM_VQS];

	unsigned int in;
	unsigned int out;
};

#define vdev_to_vpt(vdev) container_of(vdev, struct vpt_dev, vdev)

/*----------------------------------------------------------------------------*/
/* struct virtqueue_ops infrastructure */
/*----------------------------------------------------------------------------*/

static int vpt_vq_add_buf(struct virtqueue *vq,
struct scatterlist sg[], unsigned int out_num, unsigned int in_num, void *data)

{
	struct virtio_device *vdev = vq->vdev;
	struct vpt_dev *priv = vdev_to_vpt(vdev);

	pr_info("%s: vq=0x%p sg=0x%p, out=%u in=%d data=0x%p\n",
			__func__, vq, sg, out_num, in_num, data);

	priv->in += in_num;
	priv->out += out_num;

	if (priv->in > 10 || priv->out > 10) {
		pr_info("%s: vq=0x%p buffers filled\n", __func__, vq);
		return -ENOMEM;
	}

	return 0;
}

static void *vpt_vq_get_buf(struct virtqueue *vq, unsigned int *len)

{
	pr_info("%s: vq=0x%p\n", __func__, vq);
	return NULL;
}

static void vpt_vq_kick(struct virtqueue *vq)

{
	pr_info("%s: vq=0x%p\n", __func__, vq);
}

static void vpt_vq_disable_cb(struct virtqueue *vq)

{
	pr_info("%s: vq=0x%p\n", __func__, vq);
}

static bool vpt_vq_enable_cb(struct virtqueue *vq)

{
	pr_info("%s: vq=0x%p\n", __func__, vq);
	return true;
}
static struct virtqueue_ops vpt_vq_ops = {
	.add_buf	= vpt_vq_add_buf,
	.get_buf	= vpt_vq_get_buf,
	.kick		= vpt_vq_kick,
	.disable_cb	= vpt_vq_disable_cb,
	.enable_cb	= vpt_vq_enable_cb,
};

/*----------------------------------------------------------------------------*/
/* struct virtio_device infrastructure */
/*----------------------------------------------------------------------------*/

static void vptc_get(struct virtio_device *vdev,
unsigned offset,
void *buf,
unsigned len)

{
	pr_info("%s: vdev=0x%p offset=%u buf=0x%p len=%u\n",
			__func__, vdev, offset, buf, len);
}

static void vptc_set(struct virtio_device *vdev,
unsigned offset,
const void *buf,
unsigned len)

{
	pr_info("%s: vdev=0x%p offset=%u buf=%p len=%u\n",
			__func__, vdev, offset, buf, len);
}

static u8 vptc_get_status(struct virtio_device *vdev)

{
	struct vpt_dev *priv = vdev_to_vpt(vdev);

	pr_info("%s: vdev=0x%p\n", __func__, vdev);

	return priv->status;
}

static void vptc_set_status(struct virtio_device *vdev, u8 status)

{
	struct vpt_dev *priv = vdev_to_vpt(vdev);

	pr_info("%s: vdev=0x%p status=0x%.2x\n", __func__, vdev, status);

	priv->status = status;
}

static void vptc_reset(struct virtio_device *vdev)

{
	struct vpt_dev *priv = vdev_to_vpt(vdev);

	pr_info("%s: vdev=0x%p\n", __func__, vdev);

	priv->status = 0;
}

static struct virtqueue *vptc_find_vq(struct virtio_device *vdev,
unsigned index,
void (*cb)(struct virtqueue *vq))

{
	struct vpt_dev *priv = vdev_to_vpt(vdev);
	struct virtqueue *vq = priv->vqs[index];

	pr_info("%s: vdev=0x%p index=%u cb=0x%p\n", __func__, vdev, index, cb);

	if (vq)
		return vq;

	vq = kzalloc(sizeof(*vq), GFP_KERNEL);
	if (!vq) {
		pr_info("%s: vdev=0x%p unable to allocate vq\n", __func__, vq);
		return ERR_PTR(-ENOMEM);
	}

	vq->callback = cb;
	vq->vdev = vdev;
	vq->vq_ops = &vpt_vq_ops;

	pr_info("%s: vdev=0x%p new vq=0x%p\n", __func__, vdev, vq);

	priv->vqs[index] = vq;
	return vq;
}

static void vptc_del_vq(struct virtqueue *vq)

{
	struct virtio_device *vdev = vq->vdev;
	struct vpt_dev *priv = vdev_to_vpt(vdev);
	int i;

	pr_info("%s: vq=0x%p\n", __func__, vq);

	for (i = 0; i < NUM_VQS; i++) {
		if (priv->vqs[i] == vq) {
			kfree(vq);
			priv->vqs[i] = NULL;
			return;
		}
	}

	pr_err("%s: vdev=0x%p unable to find vq=0x%p\n", __func__, vdev, vq);
}

static u32 vptc_get_features(struct virtio_device *vdev)

{
	pr_info("%s: vdev=0x%p\n", __func__, vdev);
	return 0;
}

static void vptc_finalize_features(struct virtio_device *vdev)

{
	pr_info("%s: vdev=0x%p\n", __func__, vdev);
}
static struct virtio_config_ops vpt_config_ops = {
	.get			= vptc_get,
	.set			= vptc_set,
	.get_status		= vptc_get_status,
	.set_status		= vptc_set_status,
	.reset			= vptc_reset,
	.find_vq		= vptc_find_vq,
	.del_vq			= vptc_del_vq,
	.get_features		= vptc_get_features,
	.finalize_features	= vptc_finalize_features,
};

/*----------------------------------------------------------------------------*/
/* PCI Subsystem */
/*----------------------------------------------------------------------------*/

static int vpt_probe(struct pci_dev *dev, const struct pci_device_id *id)

{
	struct vpt_dev *priv;
	int ret;

	priv = kzalloc(sizeof(*priv), GFP_KERNEL);
	if (!priv) {
		ret = -ENOMEM;
		goto out_return;
	}

	pci_set_drvdata(dev, priv);

	/* Hardware Initialization */
	ret = pci_enable_device(dev);
	if (ret)
		goto out_pci_enable_dev;

	pci_set_master(dev);
	ret = pci_request_regions(dev, driver_name);
	if (ret)
		goto out_pci_request_regions;

	priv->immr = pci_ioremap_bar(dev, 0);
	if (!priv->immr) {
		ret = -ENOMEM;
		goto out_ioremap_immr;
	}

	priv->netregs = pci_ioremap_bar(dev, 1);
	if (!priv->netregs) {
		ret = -ENOMEM;
		goto out_ioremap_netregs;
	}

	ret = dma_set_mask(&dev->dev, DMA_BIT_MASK(30));
	if (ret) {
		dev_err(&dev->dev, "Unable to set DMA mask\n");
		ret = -ENODEV;
		goto out_set_dma_mask;
	}

	/* Mask all of the MBOX interrupts */
	iowrite32(0x1 | 0x2, priv->immr + OMIMR_OFFSET);

	/* Register the virtio_device corresponding to this PCI device */
	priv->vdev.dev.parent = &dev->dev;
	priv->vdev.id.device = VIRTIO_ID_NET;
	priv->vdev.config = &vpt_config_ops;

	ret = register_virtio_device(&priv->vdev);
	if (ret) {
		dev_err(&dev->dev, "Unable to register virtio device\n");
		goto out_register_virtio_device;
	}

	return 0;

out_register_virtio_device:
out_set_dma_mask:

	iounmap(priv->netregs);
out_ioremap_netregs:
	iounmap(priv->immr);
out_ioremap_immr:
	pci_release_regions(dev);
out_pci_request_regions:
	pci_disable_device(dev);
out_pci_enable_dev:
	kfree(priv);
out_return:
	return ret;
}

static void vpt_remove(struct pci_dev *dev)

{
	struct vpt_dev *priv = pci_get_drvdata(dev);

	unregister_virtio_device(&priv->vdev);
	iounmap(priv->netregs);
	iounmap(priv->immr);
	pci_release_regions(dev);
	pci_disable_device(dev);
	kfree(priv);
}

#define PCI_DEVID_FSL_MPC8349EMDS 0x0080

/* The list of devices that this module will support */

static struct pci_device_id vpt_ids[] = {
	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, PCI_DEVID_FSL_MPC8349EMDS), },
	{ 0, }
};

MODULE_DEVICE_TABLE(pci, vpt_ids);

static struct pci_driver vpt_pci_driver = {
.name = (char *)driver_name,
.id_table = vpt_ids,
.probe = vpt_probe,
.remove = vpt_remove,
};

/*----------------------------------------------------------------------------*/
/* Module Init / Exit */
/*----------------------------------------------------------------------------*/

static int __init vpt_init(void)

{
	return pci_register_driver(&vpt_pci_driver);
}

static void __exit vpt_exit(void)

{
	pci_unregister_driver(&vpt_pci_driver);
}

MODULE_AUTHOR("Ira W. Snyder <[email protected]>"); MODULE_DESCRIPTION("Virtio PCI Test Driver"); MODULE_LICENSE("GPL");

module_init(vpt_init);
module_exit(vpt_exit);

#12 lore raw
Quoted 3 lines · Ira Snyder

The only problem with that is that you cannot route interrupts from the DMA controller over PCI with the PowerPC core running. Which makes it mostly useless for this case.

If the host supports MSI, you can simply program the DMA controller to write the correct message to the inbound address of the MSI interrupt controller!

All modern host systems should have MSI, as this is required by the PCIe specification, but it still somewhat limits the choice of your hosts.

	Arnd <><
author
#13 lore raw
Quoted 6 lines · Arnd Bergmann
Quoted 3 lines · Ira Snyder

The only problem with that is that you cannot route interrupts from the DMA controller over PCI with the PowerPC core running. Which makes it mostly useless for this case.

Quoted 8 lines

If the host supports MSI, you can simply program the DMA controller to write the correct message to the inbound address of the MSI interrupt controller!

All modern host systems should have MSI, as this is required by the PCIe specification, but it still somewhat limits the choice of your hosts.

These are PCI boards, not PCIe. The host computers are all Pentium3-M systems. I tried enabling MSI on the Freescale boards in the driver, by calling pci_enable_msi() during probe(), and it failed. I'm pretty sure I can't do MSI.

I'll keep thinking about how to manage the DMA. Feel free to toss around any ideas you have, though.

Ira

#14 lore raw
Quoted 4 lines · Ira Snyder

These are PCI boards, not PCIe. The host computers are all Pentium3-M systems. I tried enabling MSI on the Freescale boards in the driver, by calling pci_enable_msi() during probe(), and it failed.

That doesn't really mean anything, just that the PCI endpoint doesn't announce its capability to do MSI in the config space, or that it does not have an interrupt line. Since you basically implement the device on the FSL board, you should also be able to define the interrupt capabilities by writing to the config space.

Do you know what kind of chipset the host uses? It should be fairly simple to find out whether or not it can do MSI.

	Arnd <><
author
#15 lore raw
Quoted 6 lines · Arnd Bergmann
Quoted 4 lines · Ira Snyder

These are PCI boards, not PCIe. The host computers are all Pentium3-M systems. I tried enabling MSI on the Freescale boards in the driver, by calling pci_enable_msi() during probe(), and it failed.

Quoted 9 lines

That doesn't really mean anything, just that the PCI endpoint doesn't announce its capability to do MSI in the config space, or that it does not have an interrupt line. Since you basically implement the device on the FSL board, you should also be able to define the interrupt capabilities by writing to the config space.

Do you know what kind of chipset the host uses? It should be fairly simple to find out whether or not it can do MSI.

Some sort of Broadcom chipset, I think. Full dmesg and lspci output are appended.

The PCI bridge does mention MSI, so maybe it does support it. Would using the DMA from the host mean that the guest system couldn't use the DMA controller at all? All of the channels share the same interrupt line on the guest. I need one channel on the DMA controller on the guest to do realtime transfers from some data processing FPGAs on the board.

This stuff is never easy, is it? :)

Thanks for the help so far, I really appreciate it. Ira

[    0.000000] Linux version 2.6.28-07812-g5fbbf5f (iws@labslcor1) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #4 Thu Jan 8 15:45:08 PST 2009
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   NSC Geode by NSC
[    0.000000]   Cyrix CyrixInstead
[    0.000000]   Centaur CentaurHauls
[    0.000000]   Transmeta GenuineTMx86
[    0.000000]   Transmeta TransmetaCPU
[    0.000000]   UMC UMC UMC UMC
[    0.000000] PAT WC disabled due to known CPU erratum.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
[    0.000000]  BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000ea800 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 0000000020000000 (usable)
[    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
[    0.000000] DMI present.
[    0.000000] Phoenix BIOS detected: BIOS may corrupt low RAM, working it around.
[    0.000000] last_pfn = 0x20000 max_arch_pfn = 0x100000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] modified physical RAM map:
[    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
[    0.000000]  modified: 0000000000010000 - 000000000009f800 (usable)
[    0.000000]  modified: 000000000009f800 - 00000000000a0000 (reserved)
[    0.000000]  modified: 00000000000ea800 - 0000000000100000 (reserved)
[    0.000000]  modified: 0000000000100000 - 0000000020000000 (usable)
[    0.000000]  modified: 00000000fff80000 - 0000000100000000 (reserved)
[    0.000000] kernel direct mapping tables up to 20000000 @ 10000-15000
[    0.000000] ACPI Error (tbxfroot-0218): A valid RSDP was not found [20080926]
[    0.000000] 512MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 20000000
[    0.000000]   low ram: 00000000 - 20000000
[    0.000000]   bootmap 00011000 - 00015000
[    0.000000] (6 early reservations) ==> bootmem [0000000000 - 0020000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0000100000 - 00007bf49c]    TEXT DATA BSS ==> [0000100000 - 00007bf49c]
[    0.000000]   #2 [00007c0000 - 00007c3000]    INIT_PG_TABLE ==> [00007c0000 - 00007c3000]
[    0.000000]   #3 [000008f400 - 0000100000]    BIOS reserved ==> [000008f400 - 0000100000]
[    0.000000]   #4 [0000010000 - 0000011000]          PGTABLE ==> [0000010000 - 0000011000]
[    0.000000]   #5 [0000011000 - 0000015000]          BOOTMAP ==> [0000011000 - 0000015000]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00020000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00020000
[    0.000000] On node 0 totalpages: 130959
[    0.000000] free_area_init_node: node 0, pgdat c0395f60, node_mem_map c1000200
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 3951 pages, LIFO batch:0
[    0.000000]   Normal zone: 992 pages used for memmap
[    0.000000]   Normal zone: 125984 pages, LIFO batch:31
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] Allocating PCI resources starting at 30000000 (gap: 20000000:dff80000)
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129935
[    0.000000] Kernel command line: console=ttyS0,19200 root=/dev/nfs ide0=noprobe ide1=noprobe nfsroot=/export/netboot/correlator_pvt/%s,tcp ip=::::::dhcp pci=biosirq BOOT_IMAGE=bzImage-labslcor1.correlator.pvt auto
[    0.000000] Enabling fast FPU save and restore... done.
[    0.000000] Enabling unmasked SIMD FPU exception support... done.
[    0.000000] Initializing CPU#0
[    0.000000] CPU 0 irqstacks, hard=c03df000 soft=c03de000
[    0.000000] PID hash table entries: 2048 (order: 11, 8192 bytes)
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 1195.647 MHz processor.
[    0.004000] Console: colour VGA+ 80x25
[    0.004000] console [ttyS0] enabled
[    0.004000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.004000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.004000] ... MAX_LOCK_DEPTH:          48
[    0.004000] ... MAX_LOCKDEP_KEYS:        8191
[    0.004000] ... CLASSHASH_SIZE:          4096
[    0.004000] ... MAX_LOCKDEP_ENTRIES:     8192
[    0.004000] ... MAX_LOCKDEP_CHAINS:      16384
[    0.004000] ... CHAINHASH_SIZE:          8192
[    0.004000]  memory used by lock dependency info: 2335 kB
[    0.004000]  per task-struct memory footprint: 1152 bytes
[    0.004000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.004000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.004000] Memory: 512108k/524288k available (1795k kernel code, 11512k reserved, 890k data, 232k init, 0k highmem)
[    0.004000] virtual kernel memory layout:
[    0.004000]     fixmap  : 0xfffac000 - 0xfffff000   ( 332 kB)
[    0.004000]     vmalloc : 0xe0800000 - 0xfffaa000   ( 503 MB)
[    0.004000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.004000]       .init : 0xc03a1000 - 0xc03db000   ( 232 kB)
[    0.004000]       .data : 0xc02c0cee - 0xc039f7c8   ( 890 kB)
[    0.004000]       .text : 0xc0100000 - 0xc02c0cee   (1795 kB)
[    0.004000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[    0.004000] SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.004025] Calibrating delay loop (skipped), value calculated using timer frequency.. 2391.29 BogoMIPS (lpj=4782588)
[    0.012060] Mount-cache hash table entries: 512
[    0.017091] CPU: L1 I cache: 16K, L1 D cache: 16K
[    0.024006] CPU: L2 cache: 512K
[    0.028010] Intel machine check architecture supported.
[    0.032008] Intel machine check reporting enabled on CPU#0.
[    0.036015] CPU: Intel(R) Pentium(R) III Mobile CPU      1200MHz stepping 01
[    0.048011] Checking 'hlt' instruction... OK.
[    0.070988] Freeing SMP alternatives: 0k freed
[    0.072007] ACPI: Core revision 20080926
[    0.076077] ACPI Exception (tbxface-0627): AE_NO_ACPI_TABLES, While loading namespace from ACPI tables [20080926]
[    0.088006] ACPI: Unable to load the System Description Tables
[    0.093401] net_namespace: 384 bytes
[    0.096363] NET: Registered protocol family 16
[    0.105855] PCI: PCI BIOS revision 2.10 entry at 0xfda10, last bus=2
[    0.108008] PCI: Using configuration type 1 for base access
[    0.120211] bio: create slab <bio-0> at 0
[    0.124230] ACPI: Interpreter disabled.
[    0.132405] usbcore: registered new interface driver usbfs
[    0.136165] usbcore: registered new interface driver hub
[    0.140180] usbcore: registered new device driver usb
[    0.148320] PCI: Probing PCI hardware
[    0.152051] PCI: Probing PCI hardware (bus 00)
[    0.152298] * The chipset may have PM-Timer Bug. Due to workarounds for a bug,
[    0.152302] * this clock source is slow. If you are sure your timer does not have
[    0.152305] * this bug, please use "acpi_pm_good" to disable the workaround
[    0.156074] * The chipset may have PM-Timer Bug. Due to workarounds for a bug,
[    0.156077] * this clock source is slow. If you are sure your timer does not have
[    0.156080] * this bug, please use "acpi_pm_good" to disable the workaround
[    0.160120] pci 0000:00:01.0: reg 10 32bit mmio: [0xf9020000-0xf9020fff]
[    0.160133] pci 0000:00:01.0: reg 14 io port: [0x1080-0x10bf]
[    0.160145] pci 0000:00:01.0: reg 18 32bit mmio: [0xf9000000-0xf901ffff]
[    0.160175] pci 0000:00:01.0: reg 30 32bit mmio: [0x000000-0x0fffff]
[    0.160204] pci 0000:00:01.0: supports D1 D2
[    0.160209] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot
[    0.164011] pci 0000:00:01.0: PME# disabled
[    0.168051] pci 0000:00:02.0: reg 10 32bit mmio: [0xf9021000-0xf9021fff]
[    0.168063] pci 0000:00:02.0: reg 14 io port: [0x10c0-0x10ff]
[    0.168075] pci 0000:00:02.0: reg 18 32bit mmio: [0xf9040000-0xf905ffff]
[    0.168105] pci 0000:00:02.0: reg 30 32bit mmio: [0x000000-0x0fffff]
[    0.168134] pci 0000:00:02.0: supports D1 D2
[    0.168139] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot
[    0.172010] pci 0000:00:02.0: PME# disabled
[    0.176062] pci 0000:00:04.0: reg 10 32bit mmio: [0xf9022000-0xf9022fff]
[    0.176074] pci 0000:00:04.0: reg 14 io port: [0x1000-0x107f]
[    0.176253] pci 0000:00:0f.1: reg 20 io port: [0x1400-0x140f]
[    0.176310] pci 0000:00:0f.2: reg 10 32bit mmio: [0xf9023000-0xf9023fff]
[    0.176491] pci 0000:01:0c.0: reg 10 32bit mmio: [0xf9100000-0xf91fffff]
[    0.176506] pci 0000:01:0c.0: reg 14 32bit mmio: [0xf9700000-0xf9700fff]
[    0.176532] pci 0000:01:0c.0: reg 18 64bit mmio: [0xf9600000-0xf96fffff]
[    0.176556] pci 0000:01:0c.0: reg 20 64bit mmio: [0xf9500000-0xf95fffff]
[    0.176668] pci 0000:01:0e.0: reg 10 32bit mmio: [0xf9200000-0xf92fffff]
[    0.176684] pci 0000:01:0e.0: reg 14 32bit mmio: [0xf9701000-0xf9701fff]
[    0.176708] pci 0000:01:0e.0: reg 18 64bit mmio: [0xf9900000-0xf99fffff]
[    0.176733] pci 0000:01:0e.0: reg 20 64bit mmio: [0xf9800000-0xf98fffff]
[    0.176840] pci 0000:00:04.0: bridge 32bit mmio: [0xf9100000-0xf92fffff]
[    0.176849] pci 0000:00:04.0: bridge 32bit mmio pref: [0xf9500000-0xf99fffff]
[    0.178609] PCI: Discovered peer bus 02
[    0.180189] pci 0000:02:03.0: reg 10 32bit mmio: [0xfa000000-0xfaffffff]
[    0.180235] pci 0000:02:03.0: reg 30 32bit mmio: [0x000000-0x03ffff]
[    0.181688] pnp: PnP ACPI: disabled
[    0.185421] pci 0000:00:04.0: PCI bridge, secondary bus 0000:01
[    0.220873] pci 0000:00:04.0:   IO window: disabled
[    0.250079] pci 0000:00:04.0:   MEM window: 0xf9100000-0xf92fffff
[    0.286569] pci 0000:00:04.0:   PREFETCH window: 0x000000f9500000-0x000000f99fffff
[    0.331918] pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
[    0.331924] pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffff]
[    0.331929] pci_bus 0000:01: resource 0 mem: [0x0-0x0]
[    0.331934] pci_bus 0000:01: resource 1 mem: [0xf9100000-0xf92fffff]
[    0.331940] pci_bus 0000:01: resource 2 mem: [0xf9500000-0xf99fffff]
[    0.331945] pci_bus 0000:01: resource 3 mem: [0x0-0x0]
[    0.331950] pci_bus 0000:02: resource 0 io:  [0x00-0xffff]
[    0.331954] pci_bus 0000:02: resource 1 mem: [0x000000-0xffffffff]
[    0.332010] NET: Registered protocol family 2
[    0.358447] IP route cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.401080] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[    0.446213] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes)
[    0.500128] TCP: Hash tables configured (established 65536 bind 65536)
[    0.539299] TCP reno registered
[    0.558459] NET: Registered protocol family 1
[    0.585465] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.631330] Machine check exception polling timer started.
[    0.665649] Microcode Update Driver: v2.00 <[email protected]>, Peter Oruba
[    0.712519] Scanning for low memory corruption every 60 seconds
[    0.760346] Installing knfsd (copyright (C) 1996 [email protected]).
[    0.799493] fuse init (API version 7.11)
[    0.823571] msgmni has been set to 1000
[    0.847156] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.891466] io scheduler noop registered
[    0.914944] io scheduler deadline registered
[    0.940540] io scheduler cfq registered (default)
[    0.968910] pci 0000:00:02.0: Firmware left e100 interrupts enabled; disabling
[    1.012197] pci 0000:02:03.0: Boot video device
[    1.022963] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.061391] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    1.098302] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.135240] e100: Intel(R) PRO/100 Network Driver, 3.5.23-k6-NAPI
[    1.171721] e100: Copyright(c) 1999-2006 Intel Corporation
[    1.227398] e100 0000:00:01.0: PME# disabled
[    1.253387] e100: eth0: e100_probe: addr 0xf9020000, irq 9, MAC addr 00:80:42:12:ad:ff
[    1.323517] e100 0000:00:02.0: PME# disabled
[    1.349416] e100: eth1: e100_probe: addr 0xf9021000, irq 11, MAC addr 00:80:42:12:ae:00
[    1.397462] console [netcon0] enabled
[    1.419373] netconsole: network logging started
[    1.446687] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.483733] ohci_hcd 0000:00:0f.2: OHCI Host Controller
[    1.516113] ohci_hcd 0000:00:0f.2: new USB bus registered, assigned bus number 1
[    1.560431] ohci_hcd 0000:00:0f.2: irq 10, io mem 0xf9023000
[    1.649658] usb usb1: configuration #1 chosen from 1 choice
[    1.683403] hub 1-0:1.0: USB hub found
[    1.705910] hub 1-0:1.0: 4 ports detected
[    1.731563] PNP: No PS/2 controller found. Probing ports directly.
[    1.771634] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.801376] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.832470] mice: PS/2 mouse device common for all mice
[    1.864545] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    1.902681] rtc0: alarms up to one day, 114 bytes nvram
[    1.934085] cpuidle: using governor ladder
[    1.958642] cpuidle: using governor menu
[    1.984820] usbcore: registered new interface driver usbhid
[    2.018247] usbhid: v2.6:USB HID core driver
[    2.043923] TCP cubic registered
[    2.063273] NET: Registered protocol family 17
[    2.090467] RPC: Registered udp transport module.
[    2.118648] RPC: Registered tcp transport module.
[    2.146845] IO APIC resources could be not be allocated.
[    2.178672] Using IPI Shortcut mode
[    2.200498] rtc_cmos rtc_cmos: setting system clock to 2009-01-15 10:08:28 UTC (1232014108)
[    2.768492] e100 0000:00:01.0: firmware: using built-in firmware e100/d101s_ucode.bin
[    2.835120] e100: eth0 NIC Link is Up 100 Mbps Full Duplex
[    2.876469] e100 0000:00:02.0: firmware: using built-in firmware e100/d101s_ucode.bin
[    3.378827] Clocksource tsc unstable (delta = 115900197 ns)
[    3.970856] Sending DHCP requests ., OK
[    3.997668] IP-Config: Got DHCP answer from 192.168.5.13, my address is 192.168.17.13
[    4.005875] IP-Config: Complete:
[    4.008824]      device=eth0, addr=192.168.17.13, mask=255.255.255.0, gw=192.168.17.1,
[    4.026488]      host=192.168.17.13, domain=correlator.pvt fileserver.pvt ovro.pvt mmarray.org carma.pvt pr, nis-domain=(none),
[    4.028216]      bootserver=192.168.5.13, rootserver=192.168.5.13, rootpath=
[    4.041384] Looking up port of RPC 100003/2 on 192.168.5.13
[    4.050478] Looking up port of RPC 100005/1 on 192.168.5.13
[    4.129452] VFS: Mounted root (nfs filesystem) readonly on device 0:13.
[    4.133518] Freeing unused kernel memory: 232k freed
[    6.834682] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    6.863495] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4

00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
00:01.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09) 00:02.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09) 00:04.0 PCI bridge: Force Computers Device 0001 (rev 01)
00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 50)
00:0f.1 IDE interface: Broadcom OSB4 IDE Controller
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 04)
01:0c.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
01:0e.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
02:03.0 VGA compatible controller: Chips and Technologies F69030 (rev 61)

00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 48, Cache Line Size: 32 bytes

00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64, Cache Line Size: 32 bytes

00:01.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
Subsystem: Intel Corporation Device 100c
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 66 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 9
Region 0: Memory at f9020000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 1080 [size=64]
Region 2: Memory at f9000000 (32-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at 30000000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-
Kernel driver in use: e100

00:02.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
Subsystem: Intel Corporation Device 100c
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 66 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f9021000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 10c0 [size=64]
Region 2: Memory at f9040000 (32-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at 30100000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-
Kernel driver in use: e100

00:04.0 PCI bridge: Force Computers Device 0001 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64, Cache Line Size: 32 bytes
Region 0: Memory at f9022000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 1000 [size=128]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: f9100000-f92fffff
Prefetchable memory behind bridge: f9500000-f99fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [fc] CompactPCI hot-swap <?>
Capabilities: [f0] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
Address: 00000000 Data: 0000
Kernel modules: shpchp

00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 50)
Subsystem: Broadcom OSB4 South Bridge
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0

00:0f.1 IDE interface: Broadcom OSB4 IDE Controller (prog-if 8a [Master SecP PriP])

	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [disabled] [size=8]
	Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [disabled] [size=1]
	Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [disabled] [size=8]
	Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [disabled] [size=1]
	Region 4: I/O ports at 1400 [size=16]

00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 04) (prog-if 10 [OHCI])
Subsystem: Broadcom OSB4/CSB5 OHCI USB Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (20000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at f9023000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci_hcd

01:0c.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)

	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f9100000 (32-bit, non-prefetchable) [size=1M]
	Region 1: Memory at f9700000 (32-bit, prefetchable) [size=4K]
	Region 2: Memory at f9600000 (64-bit, prefetchable) [size=1M]
	Region 4: Memory at f9500000 (64-bit, prefetchable) [size=1M]
	Capabilities: [48] CompactPCI hot-swap <?>

01:0e.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)

	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f9200000 (32-bit, non-prefetchable) [size=1M]
	Region 1: Memory at f9701000 (32-bit, prefetchable) [size=4K]
	Region 2: Memory at f9900000 (64-bit, prefetchable) [size=1M]
	Region 4: Memory at f9800000 (64-bit, prefetchable) [size=1M]
	Capabilities: [48] CompactPCI hot-swap <?>

02:03.0 VGA compatible controller: Chips and Technologies F69030 (rev 61) (prog-if 00 [VGA controller])
Subsystem: Chips and Technologies F69030
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 9
Region 0: Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
[virtual] Expansion ROM at 30200000 [disabled] [size=256K]

#16 lore raw
Quoted 8 lines · Ira Snyder

Some sort of Broadcom chipset, I think. Full dmesg and lspci output are appended.

The PCI bridge does mention MSI, so maybe it does support it. Would using the DMA from the host mean that the guest system couldn't use the DMA controller at all? All of the channels share the same interrupt line on the guest. I need one channel on the DMA controller on the guest to do realtime transfers from some data processing FPGAs on the board.

The PCI-CompactPCI bridge supports MSI, but that does not imply that the host does. My limited understanding of x86 tells me that you need to enable the I/O-APIC in order to use MSI. Your log shows two relavant lines:

[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
and
[    2.146845] IO APIC resources could be not be allocated.

I think you first need to enable the local APIC, then find out what the problem with the IO APIC is. If possible, try enabling ACPI (sic) in the BIOS in order to get the APIC support.

If you get that to work, you can use the interrupt line of the DMA controller to signal interrupts to the FSL machine, while using DMA transfers to the MSI address for sending interrupts to the host machine.

	Arnd <><
author
#17 lore raw
Quoted 19 lines · Arnd Bergmann
Quoted 8 lines · Ira Snyder

Some sort of Broadcom chipset, I think. Full dmesg and lspci output are appended.

The PCI bridge does mention MSI, so maybe it does support it. Would using the DMA from the host mean that the guest system couldn't use the DMA controller at all? All of the channels share the same interrupt line on the guest. I need one channel on the DMA controller on the guest to do realtime transfers from some data processing FPGAs on the board.

The PCI-CompactPCI bridge supports MSI, but that does not imply that the host does. My limited understanding of x86 tells me that you need to enable the I/O-APIC in order to use MSI. Your log shows two relavant lines:

[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
and
[    2.146845] IO APIC resources could be not be allocated.
Quoted 8 lines

I think you first need to enable the local APIC, then find out what the problem with the IO APIC is. If possible, try enabling ACPI (sic) in the BIOS in order to get the APIC support.

If you get that to work, you can use the interrupt line of the DMA controller to signal interrupts to the FSL machine, while using DMA transfers to the MSI address for sending interrupts to the host machine.

I managed to get the local APIC enabled by adding "lapic" to the kernel command line. It didn't seem to change anything.

I am totally unable to get the IO APIC working. I hunch that these boards have totally broken ACPI support, judging from the kernel log. The BIOS doesn't have much in the way of settings, but I tried the ones that made sense. No luck there.

Oh well. As in our further discussion, the mailbox or doorbell registers could be used to trigger DMA interrupts without problems.

Ira

author
#18 lore raw
In reply to Arnd Bergmann
Quoted 6 lines · Arnd Bergmann
Quoted 4 lines · Ira Snyder

These are PCI boards, not PCIe. The host computers are all Pentium3-M systems. I tried enabling MSI on the Freescale boards in the driver, by calling pci_enable_msi() during probe(), and it failed.

Quoted 9 lines

That doesn't really mean anything, just that the PCI endpoint doesn't announce its capability to do MSI in the config space, or that it does not have an interrupt line. Since you basically implement the device on the FSL board, you should also be able to define the interrupt capabilities by writing to the config space.

Do you know what kind of chipset the host uses? It should be fairly simple to find out whether or not it can do MSI.

I have another question for you Arnd.

What did you use as the host and guest drivers when you ran virtio over PCI? Did you use two unmodified instances of virtio_net (one on host, one on guest) for networking, or did you write new virtio drivers for those? How about for virtio_console (if you ran it at all).

Thanks again,
Ira

#19 lore raw
Quoted 6 lines · Ira Snyder

I have another question for you Arnd.

What did you use as the host and guest drivers when you ran virtio over PCI? Did you use two unmodified instances of virtio_net (one on host, one on guest) for networking, or did you write new virtio drivers for those? How about for virtio_console (if you ran it at all).

Jan-Bernd may be able to tell you details about this, and send you the driver code that his interns implemented for it.
This was only doing virtio_net between two machines using MMIO transfers, i.e. the DMA engine was unused, but there was a mailbox interrupt (if you have one of these, you won't need MSI, btw -- just write a DMA to it).

	Arnd <><
author
#20 lore raw
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · Ira Snyder

I have another question for you Arnd.

What did you use as the host and guest drivers when you ran virtio over PCI? Did you use two unmodified instances of virtio_net (one on host, one on guest) for networking, or did you write new virtio drivers for those? How about for virtio_console (if you ran it at all).

Quoted 6 lines

Jan-Bernd may be able to tell you details about this, and send you the driver code that his interns implemented for it.
This was only doing virtio_net between two machines using MMIO transfers, i.e. the DMA engine was unused, but there was a mailbox interrupt (if you have one of these, you won't need MSI, btw -- just write a DMA to it).

Thanks. Jan-Bernd, I'm looking forward to any input you have.

I'd be happy enough to use mmio at this point. My early attempts with this driver only used mmio as well. The DMA was bolted on later.

I do have mailboxes (two inbound, two outbound) which can generate interrupts, as well as doorbell registers (one inbound, one outbound). The doorbell register's bits are "write 1 to clear", and can only be cleared by the opposite side.

All of them can cause interrupts over PCI. I used the doorbell registers to communicate which action needed to be taken in my driver. One doorbell for "receive packet(s)", another for "packet transmission(s) complete", etc. I used the mailboxes to transfer characters for the virtual serial port.

Ira

#21 lore raw
Quoted 5 lines · Ira Snyder

I do have mailboxes (two inbound, two outbound) which can generate interrupts, as well as doorbell registers (one inbound, one outbound). The doorbell register's bits are "write 1 to clear", and can only be cleared by the opposite side.

Ok, in this case, you should be able to use one of these to implement the DMA complete interrupt, but adding an extra DMA transfer to the mailbox after each transfer that you want to get notified for.

	Arnd <><
author
#22 lore raw
Quoted 10 lines · Arnd Bergmann
Quoted 5 lines · Ira Snyder

I do have mailboxes (two inbound, two outbound) which can generate interrupts, as well as doorbell registers (one inbound, one outbound). The doorbell register's bits are "write 1 to clear", and can only be cleared by the opposite side.

Ok, in this case, you should be able to use one of these to implement the DMA complete interrupt, but adding an extra DMA transfer to the mailbox after each transfer that you want to get notified for.

Yep. I wouldn't have come up with that idea on my own, but it seems obvious now :)

It seems a little strange to keep the DMA descriptors host memory (the DMA controller will need to issue a PCI read to get them) but I guess that is how real network cards work.

Thanks,
Ira

#23 lore raw
In reply to Ira Snyder

Hi!

Ira Snyder <[email protected]> wrote on 15.01.2009 22:40:56:

Quoted 5 lines · Arnd Bergmann
Quoted 4 lines · Ira Snyder
Quoted 3 lines

I have another question for you Arnd.

What did you use as the host and guest drivers when you ran virtio

over

Quoted 1 line
Quoted 1 line
Quoted 1 line

PCI? Did you use two unmodified instances of virtio_net (one on

host,

Quoted 1 line
Quoted 1 line
Quoted 1 line

one on guest) for networking, or did you write new virtio drivers

for

Quoted 1 line
Quoted 1 line
Quoted 1 line

those? How about for virtio_console (if you ran it at all).

In addition to what Arnd already told you:
We used two (almost) unmodified instances of virtio_net. When we tested this
driver we modified the change_mtu function slightly as this function was not implemented
properly (for our purpose) at that time.

Quoted 4 lines
Quoted 4 lines

Jan-Bernd may be able to tell you details about this, and send you the driver code that his interns implemented for it.
This was only doing virtio_net between two machines using MMIO

transfers,

Quoted 1 line
Quoted 1 line

i.e. the DMA engine was unused, but there was a mailbox interrupt (if

you

Quoted 5 lines
Quoted 2 lines

have one of these, you won't need MSI, btw -- just write a DMA to it).

Thanks. Jan-Bernd, I'm looking forward to any input you have.

Regards,
Jan-Bernd

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev

#24 lore raw
In reply to Ira Snyder

Hi!

Ira Snyder <[email protected]> wrote on 15.01.2009 22:40:56:

Quoted 5 lines · Arnd Bergmann
Quoted 4 lines · Ira Snyder
Quoted 3 lines

I have another question for you Arnd.

What did you use as the host and guest drivers when you ran virtio

over

Quoted 1 line
Quoted 1 line
Quoted 1 line

PCI? Did you use two unmodified instances of virtio_net (one on

host,

Quoted 1 line
Quoted 1 line
Quoted 1 line

one on guest) for networking, or did you write new virtio drivers

for

Quoted 1 line
Quoted 1 line
Quoted 1 line

those? How about for virtio_console (if you ran it at all).

In addition to what Arnd already told you:
We used two (almost) unmodified instances of virtio_net. When we tested this
driver we modified the change_mtu function slightly as this function was not implemented
properly (for our purpose) at that time.

Quoted 4 lines
Quoted 4 lines

Jan-Bernd may be able to tell you details about this, and send you the driver code that his interns implemented for it.
This was only doing virtio_net between two machines using MMIO

transfers,

Quoted 1 line
Quoted 1 line

i.e. the DMA engine was unused, but there was a mailbox interrupt (if

you

Quoted 5 lines
Quoted 2 lines

have one of these, you won't need MSI, btw -- just write a DMA to it).

Thanks. Jan-Bernd, I'm looking forward to any input you have.

Regards,
Jan-Bernd

maintainer
#25 lore raw
In reply to Ira Snyder

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

Quoted 3 lines

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the x86 host and the powerpc nodes.

You haven't even mentioned this UART aspect even indirectly in the commit message.

This just looks like yet another set of virtualization drivers to me. You could have just have easily built this using your own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that will likely improve your throughput, it has console drivers that distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done so much better using facilities we already have that are heavily optimized and userland understands already.

maintainer
#26 lore raw
In reply to Ira Snyder

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

Quoted 3 lines

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the x86 host and the powerpc nodes.

You haven't even mentioned this UART aspect even indirectly in the commit message.

This just looks like yet another set of virtualization drivers to me. You could have just have easily built this using your own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that will likely improve your throughput, it has console drivers that distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done so much better using facilities we already have that are heavily optimized and userland understands already.

maintainer
#27 lore raw
In reply to Ira Snyder

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

Quoted 3 lines

This adds support to Linux for a virtual ethernet interface which uses the PCI bus as its transport mechanism. It creates a simple, familiar, and fast method of communication for two devices connected by a PCI interface.

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the x86 host and the powerpc nodes.

You haven't even mentioned this UART aspect even indirectly in the commit message.

This just looks like yet another set of virtualization drivers to me. You could have just have easily built this using your own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that will likely improve your throughput, it has console drivers that distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done so much better using facilities we already have that are heavily optimized and userland understands already.