Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1NkAba-0004WB-PM for qemu-devel@nongnu.org; Wed, 24 Feb 2010 01:20:19 -0500 Received: from [199.232.76.173] (port=42986 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NkAbZ-0004Vu-BK for qemu-devel@nongnu.org; Wed, 24 Feb 2010 01:20:17 -0500 Received: from Debian-exim by monty-python.gnu.org with spam-scanned (Exim 4.60) (envelope-from ) id 1NkAbW-0005Dk-1X for qemu-devel@nongnu.org; Wed, 24 Feb 2010 01:20:17 -0500 Received: from mail-iw0-f177.google.com ([209.85.223.177]:35773) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1NkAbV-0005DY-KP for qemu-devel@nongnu.org; Wed, 24 Feb 2010 01:20:13 -0500 Received: by iwn7 with SMTP id 7so3899034iwn.4 for ; Tue, 23 Feb 2010 22:20:11 -0800 (PST) MIME-Version: 1.0 Sender: camm@ualberta.ca In-Reply-To: <4B84432C.6090104@codemonkey.ws> References: <20100223205225.14860.50990.stgit@st-brides.cs.ualberta.ca> <4B84432C.6090104@codemonkey.ws> Date: Tue, 23 Feb 2010 23:20:11 -0700 Message-ID: <8286e4ee1002232220l79bf50e4t70cf7591d798cebe@mail.gmail.com> Subject: Re: [Qemu-devel] [PATCH] VirtIO Support for Memory Regions From: Cam Macdonell Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: qemu-devel@nongnu.org On Tue, Feb 23, 2010 at 2:05 PM, Anthony Liguori wr= ote: > Hi Cam, > > On 02/23/2010 02:52 PM, Cam Macdonell wrote: >> >> Support for passing memory regions via VirtIO to remove need for PCI >> support in the guest. >> >> Adds new vectors to VirtIO config space to pass memory regions similar >> to how virtqueues are passed. >> >> Kernel patch is forthcoming that add device_ops to access the memory >> regions. >> >> I have used this mechanism to implement my host shared memory >> implementation >> and modified Alex's Virtio FB to use it as well. >> > > Virtio is really a DMA engine. =A0One of the nice things about it's desig= n is > that you can do things like transparent bounce buffering if needed. =A0Ad= ding > a mechanism like this breaks this abstract and turns virtio into somethin= g > that's more than I think it should be. > > For guest shared memory, what makes sense to me is to make use of uio_pci > within a guest to map bars in userspace. =A0The device can have the first= bar > map a small MMIO region that is tied to a ioeventfd with KVM. =A0With jus= t > qemu, it can be tied to a normal eventfd. =A0You can use irqfd with KVM t= o tie > an eventfd to the PCI irq. =A0Likewise, you can use a normal eventfd to > emulate that. =A0I'd suggest using a /dev/shm file to act as the shared m= emory > segment. =A0You'll need a little magic within qemu_ram_alloc() to use thi= s > (maybe qemu_ram_alloc() should take a context parameter so this can be > setup). Part of my goal with this patch was to support systems without PCI. My previous patch http://www.mail-archive.com/kvm@vger.kernel.org/msg14284.html is a PCI implementation that uses eventfds (no ioeventfds yet) to support interrupts between guests and a /dev/shm file to share between guests. I'm not sure what you're suggesting the interrupts and eventfds would be used for if the mapping is initiated by the guest in the case. To be clear, this patch is not for shared memory per se, but adds memory regions that have multiple uses and that doesn't rely on PCI to access them. For example, I have modified Alex's virtio frame buffer as a test of this patch of a use that doesn't share memory between guests. But my interest particularly is to share memory using VirtIO. In the response to my initial PCI implementation, you had suggested using VirtIO, but the discussion around virtio not being ideal for importing memory (from another guest or the host) led to Christian's suggestion of the new device_ops which I ran with (slowly). > > You'll need to setup the backend with the name of the shared memory segme= nt > and the two eventfds. =A0Also, a user needs to specify a pci vendor/devic= e id. > =A0It's probably worth having some kind of identifier tag in the PCI devi= ce's > first MMIO region. > > I think there's a couple important characteristics of an implementation l= ike > this. =A0By using a uio_pci, you can implement shared memory in userspace= but > you can also implement a proper kernel driver if you so desired for a > specific device. Agreed, you had suggested uio_pci for my PCI driver and I'd be happy to work on that if we can agree on what the underlying device looks like whether it should be VirtIO or just PCI. Thanks for the feedback, Cam > > Regards, > > Anthony Liguori > >> Comments are welcome, >> >> Thanks, >> Cam >> --- >> =A0hw/virtio-pci.c | =A0 61 >> +++++++++++++++++++++++++++++++++++++++++++++++++++---- >> =A0hw/virtio.c =A0 =A0 | =A0 38 ++++++++++++++++++++++++++++++++++ >> =A0hw/virtio.h =A0 =A0 | =A0 22 ++++++++++++++++++++ >> =A03 files changed, 117 insertions(+), 4 deletions(-) >> >> diff --git a/hw/virtio-pci.c b/hw/virtio-pci.c >> index f3373ae..e71bf64 100644 >> --- a/hw/virtio-pci.c >> +++ b/hw/virtio-pci.c >> @@ -55,13 +55,13 @@ >> >> =A0/* MSI-X registers: only enabled if MSI-X is enabled. */ >> =A0/* A 16-bit vector for configuration changes. */ >> -#define VIRTIO_MSI_CONFIG_VECTOR =A0 =A0 =A0 =A020 >> +#define VIRTIO_MSI_CONFIG_VECTOR =A0 =A0 =A0 =A032 >> =A0/* A 16-bit vector for selected queue notifications. */ >> -#define VIRTIO_MSI_QUEUE_VECTOR =A0 =A0 =A0 =A0 22 >> +#define VIRTIO_MSI_QUEUE_VECTOR =A0 =A0 =A0 =A0 34 >> >> =A0/* Config space size */ >> -#define VIRTIO_PCI_CONFIG_NOMSI =A0 =A0 =A0 =A0 20 >> -#define VIRTIO_PCI_CONFIG_MSI =A0 =A0 =A0 =A0 =A0 24 >> +#define VIRTIO_PCI_CONFIG_NOMSI =A0 =A0 =A0 =A0 32 >> +#define VIRTIO_PCI_CONFIG_MSI =A0 =A0 =A0 =A0 =A0 36 >> =A0#define VIRTIO_PCI_REGION_SIZE(dev) =A0 =A0 (msix_present(dev) ? \ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 VIRTIO_PCI_CONFIG_MSI : \ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 VIRTIO_PCI_CONFIG_NOMSI) >> @@ -72,6 +72,12 @@ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 VIRTIO_PCI_CONFIG_MSI : \ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 VIRTIO_PCI_CONFIG_NOMSI) >> >> +#define VIRTIO_PCI_MEM_SEL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A020 >> + >> +#define VIRTIO_PCI_MEM_SIZE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 24 >> + >> +#define VIRTIO_PCI_MEM_PFN =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A028 >> + >> =A0/* Virtio ABI version, if we increment this, we break the guest drive= r. >> */ >> =A0#define VIRTIO_PCI_ABI_VERSION =A0 =A0 =A0 =A0 =A00 >> >> @@ -227,6 +233,11 @@ static void virtio_ioport_write(void *opaque, >> uint32_t addr, uint32_t val) >> =A0 =A0 =A0 =A0 =A0 =A0 =A0val =3D VIRTIO_NO_VECTOR; >> =A0 =A0 =A0 =A0 =A0virtio_queue_set_vector(vdev, vdev->queue_sel, val); >> =A0 =A0 =A0 =A0 =A0break; >> + =A0 =A0case VIRTIO_PCI_MEM_SEL: >> + =A0 =A0 =A0 =A0fprintf(stderr, "%s: VIRTIO_PCI_MEM_SEL %d\n", __func__= , val); >> + =A0 =A0 =A0 =A0if (val< =A0VIRTIO_PCI_MEM_MAX) >> + =A0 =A0 =A0 =A0 =A0 =A0 vdev->mem_sel =3D val; >> + =A0 =A0 =A0 =A0break; >> =A0 =A0 =A0default: >> =A0 =A0 =A0 =A0 =A0fprintf(stderr, "%s: unexpected address 0x%x value 0x= %x\n", >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0__func__, addr, val); >> @@ -271,6 +282,14 @@ static uint32_t virtio_ioport_read(VirtIOPCIProxy >> *proxy, uint32_t addr) >> =A0 =A0 =A0case VIRTIO_MSI_QUEUE_VECTOR: >> =A0 =A0 =A0 =A0 =A0ret =3D virtio_queue_vector(vdev, vdev->queue_sel); >> =A0 =A0 =A0 =A0 =A0break; >> + =A0 =A0case VIRTIO_PCI_MEM_SIZE: >> + =A0 =A0 =A0 =A0ret =3D virtio_mem_get_size(vdev, vdev->mem_sel); >> + =A0 =A0 =A0 =A0break; >> + =A0 =A0case VIRTIO_PCI_MEM_PFN: >> + =A0 =A0 =A0 =A0ret =3D virtio_mem_get_addr(vdev, vdev->mem_sel) >> +>> =A0VIRTIO_PCI_QUEUE_ADDR_SHIFT; >> + =A0 =A0 =A0 =A0break; >> + >> =A0 =A0 =A0default: >> =A0 =A0 =A0 =A0 =A0break; >> =A0 =A0 =A0} >> @@ -370,6 +389,30 @@ static void virtio_map(PCIDevice *pci_dev, int >> region_num, >> =A0 =A0 =A0 =A0 =A0vdev->get_config(vdev, vdev->config); >> =A0} >> >> + >> +static void virtio_setup_mem(PCIDevice *d, int region_num, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pcibus_t addr, pcibus_t si= ze, int type) >> +{ >> + =A0 =A0VirtIOPCIProxy *proxy =3D DO_UPCAST(VirtIOPCIProxy, pci_dev, d)= ; >> + =A0 =A0VirtIODevice *vdev; >> + >> + =A0 =A0printf("XXX setmem to %#x - %#x\n", (uint32_t)addr, (uint32_t)(= addr + >> size)); >> + =A0 =A0vdev =3D proxy->vdev; >> + >> + =A0 =A0cpu_register_physical_memory(addr, size, IO_MEM_UNASSIGNED); >> + =A0 =A0cpu_register_physical_memory(addr, size, >> vdev->vmem[region_num-2].ram_addr | IO_MEM_RAM); >> + >> + =A0 =A0/* store the address of the region and get a pointer for it >> + =A0 =A0 * that can access the mem region */ >> + =A0 =A0vdev->vmem[region_num-2].guest_phys_addr =3D (target_phys_addr_= t)addr; >> + >> + =A0 =A0if (vdev->vmem[region_num-2].on_map) { >> + =A0 =A0 =A0 =A0vdev->vmem[region_num-2].on_map(vdev, addr, size); >> + =A0 =A0} >> + =A0 =A0return; >> + >> +} >> + >> =A0static void virtio_write_config(PCIDevice *pci_dev, uint32_t address, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0uint3= 2_t val, int len) >> =A0{ >> @@ -406,6 +449,7 @@ static void virtio_init_pci(VirtIOPCIProxy *proxy, >> VirtIODevice *vdev, >> =A0{ >> =A0 =A0 =A0uint8_t *config; >> =A0 =A0 =A0uint32_t size; >> + =A0 =A0int i; >> >> =A0 =A0 =A0proxy->vdev =3D vdev; >> >> @@ -426,6 +470,15 @@ static void virtio_init_pci(VirtIOPCIProxy *proxy, >> VirtIODevice *vdev, >> >> =A0 =A0 =A0config[0x3d] =3D 1; >> >> + =A0 =A0// only BARs that were added with virtio_add_memory will be all= ocated >> here >> + =A0 =A0for (i =3D 0; i< =A0VIRTIO_PCI_QUEUE_MAX; i++) { >> + =A0 =A0 =A0 =A0if (vdev->vmem[i].size> =A00) { >> + =A0 =A0 =A0 =A0 =A0 =A0pci_register_bar(&proxy->pci_dev, 2 + i, vdev->= vmem[i].size, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0PCI_BASE_ADDRESS_SPACE_MEMORY, virtio_s= etup_mem); >> + =A0 =A0 =A0 =A0} >> + =A0 =A0} >> + >> + >> =A0 =A0 =A0if (vdev->nvectors&& =A0!msix_init(&proxy->pci_dev, vdev->nve= ctors, 1, >> 0)) { >> =A0 =A0 =A0 =A0 =A0pci_register_bar(&proxy->pci_dev, 1, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 msix_bar_size(&proxy= ->pci_dev), >> diff --git a/hw/virtio.c b/hw/virtio.c >> index 7c020a3..1f7924e 100644 >> --- a/hw/virtio.c >> +++ b/hw/virtio.c >> @@ -549,6 +549,16 @@ target_phys_addr_t virtio_queue_get_addr(VirtIODevi= ce >> *vdev, int n) >> =A0 =A0 =A0return vdev->vq[n].pa; >> =A0} >> >> +target_phys_addr_t virtio_mem_get_addr(VirtIODevice *vdev, int n) >> +{ >> + =A0 =A0return vdev->vmem[n].guest_phys_addr; >> +} >> + >> +int virtio_mem_get_size(VirtIODevice *vdev, int n) >> +{ >> + =A0 =A0return vdev->vmem[n].size; >> +} >> + >> =A0int virtio_queue_get_num(VirtIODevice *vdev, int n) >> =A0{ >> =A0 =A0 =A0return vdev->vq[n].vring.num; >> @@ -592,6 +602,33 @@ VirtQueue *virtio_add_queue(VirtIODevice *vdev, int >> queue_size, >> =A0 =A0 =A0return&vdev->vq[i]; >> =A0} >> >> +VirtMem *virtio_add_memory(VirtIODevice *vdev, int mem_size, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void (*handle_a= ccess)(VirtIODevice *, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0target_= phys_addr_t, uint32_t)) >> +{ >> + =A0 =A0int i; >> + >> + =A0 =A0/* trace what virtio_fb_setmem does */ >> + >> + =A0 =A0for (i =3D 0; i< =A0VIRTIO_PCI_MEM_MAX; i++) { >> + =A0 =A0 =A0 =A0if (vdev->vmem[i].size =3D=3D 0) { >> + =A0 =A0 =A0 =A0 =A0 =A0/* XXX: check that memory is a power of 2 */ >> + =A0 =A0 =A0 =A0 =A0 =A0vdev->vmem[i].size =3D mem_size * 1024 * 1024; = /* presume size >> is in MBs*/ >> + =A0 =A0 =A0 =A0 =A0 =A0vdev->vmem[i].ram_addr =3D qemu_ram_alloc(vdev-= >vmem[i].size); >> + =A0 =A0 =A0 =A0 =A0 =A0vdev->vmem[i].host_ptr =3D >> qemu_get_ram_ptr(vdev->vmem[i].ram_addr); >> + =A0 =A0 =A0 =A0 =A0 =A0vdev->vmem[i].on_map =3D handle_access; >> + =A0 =A0 =A0 =A0 =A0 =A0break; >> + =A0 =A0 =A0 =A0} >> + =A0 =A0} >> + >> + =A0 =A0return&vdev->vmem[i]; >> +} >> + >> +void * virtio_get_memory_ptr(VirtMem * vmem) >> +{ >> + =A0 =A0return vmem->host_ptr; >> +} >> + >> =A0void virtio_notify(VirtIODevice *vdev, VirtQueue *vq) >> =A0{ >> =A0 =A0 =A0/* Always notify when queue is empty (when feature acknowledg= e) */ >> @@ -714,6 +751,7 @@ VirtIODevice *virtio_common_init(const char *name, >> uint16_t device_id, >> =A0 =A0 =A0vdev->queue_sel =3D 0; >> =A0 =A0 =A0vdev->config_vector =3D VIRTIO_NO_VECTOR; >> =A0 =A0 =A0vdev->vq =3D qemu_mallocz(sizeof(VirtQueue) * VIRTIO_PCI_QUEU= E_MAX); >> + =A0 =A0vdev->vmem =3D qemu_mallocz(sizeof(VirtMem) * VIRTIO_PCI_MEM_MA= X); >> =A0 =A0 =A0for(i =3D 0; i< =A0VIRTIO_PCI_QUEUE_MAX; i++) >> =A0 =A0 =A0 =A0 =A0vdev->vq[i].vector =3D VIRTIO_NO_VECTOR; >> >> diff --git a/hw/virtio.h b/hw/virtio.h >> index 3baa2a3..02beb27 100644 >> --- a/hw/virtio.h >> +++ b/hw/virtio.h >> @@ -82,6 +82,15 @@ typedef struct VirtQueueElement >> =A0 =A0 =A0struct iovec out_sg[VIRTQUEUE_MAX_SIZE]; >> =A0} VirtQueueElement; >> >> +typedef struct VirtMem >> +{ >> + =A0 =A0target_phys_addr_t guest_phys_addr; >> + =A0 =A0ram_addr_t ram_addr; >> + =A0 =A0int size; >> + =A0 =A0uint8_t * host_ptr; >> + =A0 =A0void (*on_map)(VirtIODevice *vdev, target_phys_addr_t, uint32_t= ); >> +} VirtMem; >> + >> =A0typedef struct { >> =A0 =A0 =A0void (*notify)(void * opaque, uint16_t vector); >> =A0 =A0 =A0void (*save_config)(void * opaque, QEMUFile *f); >> @@ -93,6 +102,8 @@ typedef struct { >> >> =A0#define VIRTIO_PCI_QUEUE_MAX 64 >> >> +#define VIRTIO_PCI_MEM_MAX 16 >> + >> =A0#define VIRTIO_NO_VECTOR 0xffff >> >> =A0struct VirtIODevice >> @@ -101,6 +112,7 @@ struct VirtIODevice >> =A0 =A0 =A0uint8_t status; >> =A0 =A0 =A0uint8_t isr; >> =A0 =A0 =A0uint16_t queue_sel; >> + =A0 =A0uint16_t mem_sel; >> =A0 =A0 =A0uint32_t guest_features; >> =A0 =A0 =A0size_t config_len; >> =A0 =A0 =A0void *config; >> @@ -113,6 +125,7 @@ struct VirtIODevice >> =A0 =A0 =A0void (*set_config)(VirtIODevice *vdev, const uint8_t *config)= ; >> =A0 =A0 =A0void (*reset)(VirtIODevice *vdev); >> =A0 =A0 =A0VirtQueue *vq; >> + =A0 =A0VirtMem * vmem; >> =A0 =A0 =A0const VirtIOBindings *binding; >> =A0 =A0 =A0void *binding_opaque; >> =A0 =A0 =A0uint16_t device_id; >> @@ -122,6 +135,12 @@ VirtQueue *virtio_add_queue(VirtIODevice *vdev, int >> queue_size, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void (*handle= _output)(VirtIODevice *, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0VirtQueue *)); >> >> +void * virtio_get_memory_ptr(VirtMem * vmem); >> +VirtMem *virtio_add_memory(VirtIODevice *vdev, int mem_size, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void (*handle_a= ccess)(VirtIODevice *, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0target_phys_addr_t, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0uint32_t)); >> + >> =A0void virtqueue_push(VirtQueue *vq, const VirtQueueElement *elem, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0unsigned int len); >> =A0void virtqueue_flush(VirtQueue *vq, unsigned int count); >> @@ -169,6 +188,9 @@ void virtio_update_irq(VirtIODevice *vdev); >> =A0void virtio_bind_device(VirtIODevice *vdev, const VirtIOBindings >> *binding, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void *opaque); >> >> +target_phys_addr_t virtio_mem_get_addr(VirtIODevice *vdev, int n); >> +int virtio_mem_get_size(VirtIODevice *vdev, int n); >> + >> =A0/* Base devices. =A0*/ >> =A0VirtIODevice *virtio_blk_init(DeviceState *dev, BlockConf *conf); >> =A0VirtIODevice *virtio_net_init(DeviceState *dev, NICConf *conf); >> >> >> > >