Skip to contentSkip to conversation

R/W HG memory mappings with kvm?

PATCH New Stephen Donnelly 3 participants 31 messages Jul 5, 2009 – Sep 9, 2009 1 message not in the archive

Subsystems (2)
KVM Kbuild
lore
Share & tools
lore

Stephen Donnelly

author
#0 lore raw

I am looking at how to do memory mapped IO between host and guests under kvm. I expect to use the PCI emulation layer to present a PCI device to the guest.

I see virtio_pci uses cpu_physical_memory_map() which provides either read or write mappings and notes "Use only for reads OR writes - not for read-modify-write operations."

Is there an alternative method that allows large (Several MB) persistent hg memory mappings that are r/w? I would only be using this under kvm, not kqemu or plain qemu.

Also it appears that PCI IO memory (cpu_register_io_memory) is provided via access functions, like the pci config space? Does this cause a page fault/vm_exit on each read or write, or is it more efficient than that?

Thanks,
Stephen.

#1 lore raw
Quoted 8 lines · Stephen Donnelly

I am looking at how to do memory mapped IO between host and guests under kvm. I expect to use the PCI emulation layer to present a PCI device to the guest.

I see virtio_pci uses cpu_physical_memory_map() which provides either read or write mappings and notes "Use only for reads OR writes - not for read-modify-write operations."

Right, these are for unidirectional transient DMA.

Quoted 4 lines

Is there an alternative method that allows large (Several MB) persistent hg memory mappings that are r/w? I would only be using this under kvm, not kqemu or plain qemu.

All of guest memory is permanently mapped in the host. You can use accessors like cpu_physical_memory_rw() or cpu_physical_memory_map() to access it. What exactly do you need that is not provided by these accessors?

Quoted 2 lines

Also it appears that PCI IO memory (cpu_register_io_memory) is provided via access functions, like the pci config space?

It can also use ordinary RAM (for example, vga maps its framebuffer as a PCI BAR).

Quoted 4 lines

Does this
cause a page fault/vm_exit on each read or write, or is it more efficient than that?

It depends on how you configure it. Look at the vga code (hw/vga.c, hw/cirrus_vga.c). Also Cam (copied) wrote a PCI card that provides shared memory across guests, you may want to look at that.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#2 lore raw
Quoted 5 lines · Avi Kivity
Quoted 3 lines

I see virtio_pci uses cpu_physical_memory_map() which provides either read or write mappings and notes "Use only for reads OR writes - not for read-modify-write operations."

Right, these are for unidirectional transient DMA.

Okay, as I thought. I would rather have 'relatively' persistent mappings, multi-use, and preferably bi-directional.

Quoted 8 lines
Quoted 3 lines

Is there an alternative method that allows large (Several MB) persistent hg memory mappings that are r/w? I would only be using this under kvm, not kqemu or plain qemu.

All of guest memory is permanently mapped in the host.  You can use accessors like cpu_physical_memory_rw() or cpu_physical_memory_map() to access it.  What exactly do you need that is not provided by these accessors?

I have an existing software system that provides high speed communication between processes on a single host using shared memory. I would like to extend the system to provide communication between processes on the host and guest. Unfortunately the transport is optimised for speed and is not highly abstracted so I cannot easily substitute a virtio-ring for example.

The system uses two memory spaces, one is a control area which is register-like and contains R/W values at various offsets. The second area is for data transport and is divided into rings. Each ring is unidirectional so I could map these separately with cpu_physical_memory_map(), but there seems to be no simple solution for the control area. Target ring performance is perhaps 1-2 gigabytes/second with rings approx 32-512MB in size.

Quoted 5 lines
Quoted 2 lines

Also it appears that PCI IO memory (cpu_register_io_memory) is provided via access functions, like the pci config space?

It can also use ordinary RAM (for example, vga maps its framebuffer as a PCI BAR).

So host memory is exported as a PCI_BAR to the guest via cpu_register_physical_memory(). It looks like the code has to explicitly manage marking pages dirty and synchronising at appropriate times. Is the coherency problem bidirectional, e.g. writes from either host or guest to the shared memory need to mark pages dirty, and ensure sync is called before the other side reads those areas?

Quoted 7 lines
Quoted 3 lines

Does this
cause a page fault/vm_exit on each read or write, or is it more efficient than that?

It depends on how you configure it.  Look at the vga code (hw/vga.c, hw/cirrus_vga.c).  Also Cam (copied) wrote a PCI card that provides shared memory across guests, you may want to look at that.

I will look into the vga code and see if I get inspired. The 'copied' driver sounds interesting, the code is not in kvm git?

Thanks for the response!

Stephen.

#3 lore raw
Quoted 8 lines · Stephen Donnelly
Quoted 7 lines
Quoted 4 lines

Also it appears that PCI IO memory (cpu_register_io_memory) is provided via access functions, like the pci config space?

It can also use ordinary RAM (for example, vga maps its framebuffer as a PCI BAR).

Quoted 7 lines

So host memory is exported as a PCI_BAR to the guest via cpu_register_physical_memory(). It looks like the code has to explicitly manage marking pages dirty and synchronising at appropriate times. Is the coherency problem bidirectional, e.g. writes from either host or guest to the shared memory need to mark pages dirty, and ensure sync is called before the other side reads those areas?

Shared memory is fully coherent. You can use the ordinary x86 bus lock operations for concurrent read-modify-write access, and the memory barrier instructions to prevent reordering. Just like ordinary shared memory.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Does this
cause a page fault/vm_exit on each read or write, or is it more efficient than that?

Quoted 8 lines
Quoted 4 lines

It depends on how you configure it. Look at the vga code (hw/vga.c, hw/cirrus_vga.c). Also Cam (copied) wrote a PCI card that provides shared memory across guests, you may want to look at that.

I will look into the vga code and see if I get inspired. The 'copied' driver sounds interesting, the code is not in kvm git?

(copied) means Cam was copied (cc'ed) on the email, not the name of the driver. It hasn't been merged but copies (of the driver, not Cam) are floating around on the Internet.

The relevant parts of cirrus_vga.c are:

static void cirrus_pci_lfb_map(PCIDevice *d, int region_num,
uint32_t addr, uint32_t size, int type)
{

...

    /* XXX: add byte swapping apertures */
    cpu_register_physical_memory(addr, s->vga.vram_size,
                                 s->cirrus_linear_io_addr);

This function is called whenever the guest updates the BAR.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
author
#4 lore raw
Quoted 3 lines

Shared memory is fully coherent.  You can use the ordinary x86 bus lock operations for concurrent read-modify-write access, and the memory barrier instructions to prevent reordering.  Just like ordinary shared memory.

Okay, I think I was confused by the 'dirty' code. Is that just to do with migration?

Quoted 3 lines

(copied) means Cam was copied (cc'ed) on the email, not the name of the driver.  It hasn't been merged but copies (of the driver, not Cam) are floating around on the Internet.

Thanks, I'll ask him for a pointer.

Quoted 6 lines

The relevant parts of cirrus_vga.c are:

static void cirrus_pci_lfb_map(PCIDevice *d, int region_num, uint32_t addr, uint32_t size, int type) {

Quoted 7 lines

...

   /* XXX: add byte swapping apertures */ cpu_register_physical_memory(addr, s->vga.vram_size, s->cirrus_linear_io_addr);

This function is called whenever the guest updates the BAR.

So guest accesses to the LFB PCI_BAR trigger the cirrus_linear functions, which set dirty on writes and allow 'side effect' handling for reads if required? In my case there should be no side effects, so it could be quite simple. I wonder about the cost of the callbacks on each access though, am I still missing something?

Thank you for your patience, I really appreciate the assistance and look forward to using kvm more widely.

Stephen.

#5 lore raw
Quoted 8 lines · Stephen Donnelly
Quoted 4 lines

Shared memory is fully coherent. You can use the ordinary x86 bus lock operations for concurrent read-modify-write access, and the memory barrier instructions to prevent reordering. Just like ordinary shared memory.

Okay, I think I was confused by the 'dirty' code. Is that just to do with migration?

Migration and reducing vga updates.

Quoted 10 lines
Quoted 10 lines

static void cirrus_pci_lfb_map(PCIDevice *d, int region_num,
uint32_t addr, uint32_t size, int type)
{

...

    /* XXX: add byte swapping apertures */
    cpu_register_physical_memory(addr, s->vga.vram_size,
                                 s->cirrus_linear_io_addr);
Quoted 9 lines
Quoted 2 lines

This function is called whenever the guest updates the BAR.

So guest accesses to the LFB PCI_BAR trigger the cirrus_linear functions, which set dirty on writes and allow 'side effect' handling for reads if required? In my case there should be no side effects, so it could be quite simple. I wonder about the cost of the callbacks on each access though, am I still missing something?

Sorry, I quoted the wrong part. vga is complicated because some modes do need callbacks on reads and writes, and others can use normal RAM (with dirty tracking).

The real direct mapping code is:

    static void map_linear_vram(CirrusVGAState *s)
    {
        vga_dirty_log_stop(&s->vga);
        if (!s->vga.map_addr && s->vga.lfb_addr && s->vga.lfb_end) {
            s->vga.map_addr = s->vga.lfb_addr;
            s->vga.map_end = s->vga.lfb_end;
            cpu_register_physical_memory(s->vga.map_addr, 
s->vga.map_end - s->vga.map_addr, s->vga.vram_offset);
        }

s->vga.vram_offset is a ram_addr_t describing the the vga framebuffer. You're much better off reading Cam's code as it's much simpler and closer to what you want to do (possibly you can use it as is).

Signature · 1 line
error compiling committee.c: too many arguments to function
#6 lore raw
In reply to Avi Kivity
Quoted 28 lines · Avi Kivity
Quoted 8 lines · Stephen Donnelly
Quoted 7 lines
Quoted 4 lines

Also it appears that PCI IO memory (cpu_register_io_memory) is provided via access functions, like the pci config space?

It can also use ordinary RAM (for example, vga maps its framebuffer as a PCI BAR).

Quoted 7 lines

So host memory is exported as a PCI_BAR to the guest via cpu_register_physical_memory(). It looks like the code has to explicitly manage marking pages dirty and synchronising at appropriate times. Is the coherency problem bidirectional, e.g. writes from either host or guest to the shared memory need to mark pages dirty, and ensure sync is called before the other side reads those areas?

Shared memory is fully coherent. You can use the ordinary x86 bus lock operations for concurrent read-modify-write access, and the memory barrier instructions to prevent reordering. Just like ordinary shared memory.

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

Does this
cause a page fault/vm_exit on each read or write, or is it more efficient than that?

It depends on how you configure it. Look at the vga code (hw/vga.c, hw/cirrus_vga.c). Also Cam (copied) wrote a PCI card that provides shared memory across guests, you may want to look at that.

Quoted 5 lines
Quoted 3 lines

I will look into the vga code and see if I get inspired. The 'copied' driver sounds interesting, the code is not in kvm git?

(copied) means Cam was copied (cc'ed) on the email, not the name of the driver. It hasn't been merged but copies (of the driver, not Cam) are floating around on the Internet.

Hi Stephen,

Here is the latest patch that supports interrupts. I am currently working on a broadcast mechanism that should be ready fairly soon.

http://patchwork.kernel.org/patch/22368/

I have some test scripts that can demonstrate how to use the memory between guest/host and guest/guest. Let me know if you would like me to send them to you.

Cheers,
Cam

Quoted 15 lines

The relevant parts of cirrus_vga.c are:

static void cirrus_pci_lfb_map(PCIDevice *d, int region_num,
uint32_t addr, uint32_t size, int type)
{

...

    /* XXX: add byte swapping apertures */
    cpu_register_physical_memory(addr, s->vga.vram_size,
                                 s->cirrus_linear_io_addr);

This function is called whenever the guest updates the BAR.

author
#7 lore raw
Quoted 10 lines · Cam Macdonell

Hi Stephen,

Here is the latest patch that supports interrupts.  I am currently working on a broadcast mechanism that should be ready fairly soon.

http://patchwork.kernel.org/patch/22368/

I have some test scripts that can demonstrate how to use the memory between guest/host and guest/guest.  Let me know if you would like me to send them to you.

Hi Cam,

Thanks for the pointer. That makes perfect sense, I'm familiar with PCI drivers so that's fine.

Is there a corresponding qemu patch for the backend to the guest pci driver? I'm curious how the buffer memory is allocated and how BAR accesses are handled from the host side.

Regards,
Stephen.

#8 lore raw
Quoted 12 lines · Stephen Donnelly
Quoted 10 lines · Cam Macdonell

Hi Stephen,

Here is the latest patch that supports interrupts. I am currently working on a broadcast mechanism that should be ready fairly soon.

http://patchwork.kernel.org/patch/22368/

I have some test scripts that can demonstrate how to use the memory between guest/host and guest/guest. Let me know if you would like me to send them to you.

Quoted 7 lines

Hi Cam,

Thanks for the pointer. That makes perfect sense, I'm familiar with PCI drivers so that's fine.

Is there a corresponding qemu patch for the backend to the guest pci driver?

Oops right. For some reason I can't my driver patch in patchwork.

http://kerneltrap.org/mailarchive/linux-kvm/2009/5/7/5665734

Quoted 2 lines

I'm curious how the buffer memory is allocated and how BAR accesses are handled from the host side.

The memory for the device allocated as a POSIX shared memory object and then mmapped on to the allocated BAR region in Qemu's allocated memory. That's actually one spot that needs a bit of fixing by passing the already allocated memory object to qemu instead of mmapping on to it.

Cam

-----------------------------------------------
A. Cameron Macdonell
Ph.D. Student
Department of Computing Science
University of Alberta
[email protected]

author
#9 lore raw
Quoted 6 lines · Cam Macdonell
Quoted 2 lines

Is there a corresponding qemu patch for the backend to the guest pci driver?

Oops right.   For some reason I can't my driver patch in patchwork.

http://kerneltrap.org/mailarchive/linux-kvm/2009/5/7/5665734

Thanks for the link, I have read through the thread now. It seems very relevant to what I am doing. Have you found a link to your qemu-kvm backend patches? Or are you running your own git tree? I don't really know where to look.

Quoted 7 lines
Quoted 2 lines

I'm curious how the buffer memory is allocated and how BAR accesses are handled from the host side.

The memory for the device allocated as a POSIX shared memory object and then mmapped on to the allocated BAR region in Qemu's allocated memory.  That's actually one spot that needs a bit of fixing by passing the already allocated memory object to qemu instead of mmapping on to it.

Right, I would be passing the memory in pre-allocated as well, but should be a relatively simple change.

Regards,
Stephen.

#10 lore raw
Quoted 4 lines · Stephen Donnelly
Quoted 2 lines · Cam Macdonell
Quoted 2 lines

Is there a corresponding qemu patch for the backend to the guest pci driver?

Quoted 8 lines
Quoted 3 lines

Oops right. For some reason I can't my driver patch in patchwork.

http://kerneltrap.org/mailarchive/linux-kvm/2009/5/7/5665734

Thanks for the link, I have read through the thread now. It seems very relevant to what I am doing. Have you found a link to your qemu-kvm backend patches? Or are you running your own git tree? I don't really know where to look.

Oops, I realize now that I passed the driver patch both times. Here is the old patch.

http://patchwork.kernel.org/patch/22363/

What are you compiling against? the git tree or a particular version? The above patch won't compile against the latest git tree due to changes to how BARs are setup in Qemu. I can send you a patch for the latest tree if you need it.

Cam

Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

I'm curious how the buffer memory is allocated and how BAR accesses are handled from the host side.

The memory for the device allocated as a POSIX shared memory object and then mmapped on to the allocated BAR region in Qemu's allocated memory. That's actually one spot that needs a bit of fixing by passing the already allocated memory object to qemu instead of mmapping on to it.

Right, I would be passing the memory in pre-allocated as well, but should be a relatively simple change.

Regards,
Stephen.

author
#11 lore raw
Quoted 9 lines · Cam Macdonell

Oops, I realize now that I passed the driver patch both times.  Here is the old patch.

http://patchwork.kernel.org/patch/22363/

What are you compiling against?  the git tree or a particular version? The above patch won't compile against the latest git tree due to changes to how BARs are setup in Qemu.  I can send you a patch for the latest tree if you need it.

Thanks Cam, I will take a look at this code.

At the moment I have cloned the tree so am intending to work at the tip. If you have a patch for the latest tree that would be great.

Regards,
Stephen.

#12 lore raw

PATCH 1/1 Support shared memory device PCI device

This patch is an updated version of a previous one (http://patchwork.kernel.org/patch/22363/) that supports adding a shared memory PCI device. To be clear, there is no new functionality in this patch, just a fix for changes to the master branch.

The device's memory is mappable into user-level to provide zero-copy messaging between guest and between guest and host. Please see previous patch for a more detailed description.

---
 Makefile.target |    3 +
 hw/ivshmem.c    |  421 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 hw/pc.c         |    6 +
 hw/pc.h         |    3 +
 qemu-options.hx |   14 ++
 sysemu.h        |    8 +
 vl.c            |   14 ++
 7 files changed, 469 insertions(+), 0 deletions(-)
 create mode 100644 hw/ivshmem.c
Makefile.target +3 -0
index 660a855..323a935 100644
@@ -611,6 +611,9 @@ obj-y += pcnet.o
 obj-y += rtl8139.o
 obj-y += e1000.o
 
+# Inter-VM PCI shared memory
+obj-y += ivshmem.o
+
 # Generic watchdog support and some watchdog devices
 obj-y += wdt_ib700.o wdt_i6300esb.o
hw/ivshmem.c new file +421 -0

421 lines not on this page · raw message

hw/pc.c +6 -0
index cf84416..f20dc83 100644
@@ -91,6 +91,8 @@ static void option_rom_setup_reset(target_phys_addr_t addr, unsigned size)
     qemu_register_reset(option_rom_reset, rrd);
 }
 
+extern int ivshmem_enabled;
+
 static void ioport80_write(void *opaque, uint32_t addr, uint32_t data)
 {
 }

@@ -1289,6 +1291,10 @@ static void pc_init1(ram_addr_t ram_size,
         }
     }
 
+    if (pci_enabled && ivshmem_enabled) {
+        pci_ivshmem_init(pci_bus);
+    }
+
     rtc_state = rtc_init(0x70, i8259[8], 2000);
 
     qemu_register_boot_set(pc_boot_set, rtc_state);
hw/pc.h +3 -0
index 10bf002..5d19a0f 100644
@@ -185,4 +185,7 @@ void extboot_init(BlockDriverState *bs, int cmd);
 
 int cpu_is_bsp(CPUState *env);
 
+/* ivshmem.c */
+int pci_ivshmem_init(PCIBus *bus);
+
 #endif
qemu-options.hx +14 -0
index 7e98053..2411372 100644
@@ -1309,6 +1309,20 @@ The default device is @code{vc} in graphical mode and @code{stdio} in
 non graphical mode.
 ETEXI
 
+DEF("ivshmem", HAS_ARG, QEMU_OPTION_ivshmem, \
+    "-ivshmem name,size[,unix:path][,server]  creates or opens a shared file 'name' of size \
+    'size' (in MB) and exposes it as a PCI device in the guest\n")
+STEXI
+@item -ivshmem @var{file},@var{size}
+Creates a POSIX shared file named @var{file} of size @var{size} and creates a
+PCI device of the same size that maps the shared file into the device for guests
+to access.  The created file on the host is located in /dev/shm/
+
+@item unix:@var{path}[,server]
+A unix domain socket is used to send and receive interrupts between VMs.  The unix domain socket
+@var{path} is used for connections.
+ETEXI
+
 DEF("pidfile", HAS_ARG, QEMU_OPTION_pidfile, \
     "-pidfile file   write PID to 'file'\n")
 STEXI
sysemu.h +8 -0
index 5582633..24abda1 100644
@@ -239,6 +239,14 @@ extern CharDriverState *parallel_hds[MAX_PARALLEL_PORTS];
 
 extern CharDriverState *virtcon_hds[MAX_VIRTIO_CONSOLES];
 
+/* inter-VM shared memory devices */
+
+#define MAX_IVSHMEM_DEVICES 1
+
+extern CharDriverState * ivshmem_chardev;
+void ivshmem_init(const char * optarg);
+int ivshmem_get_size(void);
+
 #define TFR(expr) do { if ((expr) != -1) break; } while (errno == EINTR)
 
 #ifdef NEED_CPU_H
vl.c +14 -0
index 5d86e69..553cf5c 100644
@@ -217,6 +217,7 @@ static int rtc_date_offset = -1; /* -1 means no change */
 int cirrus_vga_enabled = 1;
 int std_vga_enabled = 0;
 int vmsvga_enabled = 0;
+int ivshmem_enabled = 0;
 int xenfb_enabled = 0;
 #ifdef TARGET_SPARC
 int graphic_width = 1024;

@@ -235,6 +236,8 @@ int no_quit = 0;
 CharDriverState *serial_hds[MAX_SERIAL_PORTS];
 CharDriverState *parallel_hds[MAX_PARALLEL_PORTS];
 CharDriverState *virtcon_hds[MAX_VIRTIO_CONSOLES];
+CharDriverState *ivshmem_chardev;
+const char * ivshmem_device;
 #ifdef TARGET_I386
 int win2k_install_hack = 0;
 int rtc_td_hack = 0;

@@ -5139,6 +5142,8 @@ int main(int argc, char **argv, char **envp)
     cyls = heads = secs = 0;
     translation = BIOS_ATA_TRANSLATION_AUTO;
     monitor_device = "vc:80Cx24C";
+    ivshmem_device = NULL;
+    ivshmem_chardev = NULL;
 
     serial_devices[0] = "vc:80Cx24C";
     for(i = 1; i < MAX_SERIAL_PORTS; i++)

@@ -5592,6 +5597,10 @@ int main(int argc, char **argv, char **envp)
                 parallel_devices[parallel_device_index] = optarg;
                 parallel_device_index++;
                 break;
+            case QEMU_OPTION_ivshmem:
+                ivshmem_device = optarg;
+                ivshmem_enabled = 1;
+                break;
 	    case QEMU_OPTION_loadvm:
 		loadvm = optarg;
 		break;

@@ -6049,6 +6058,11 @@ int main(int argc, char **argv, char **envp)
 	    }
     }
 
+    if (ivshmem_enabled) {
+        ivshmem_init(ivshmem_device);
+        ram_size += ivshmem_get_size();
+    }
+
 #ifdef CONFIG_KQEMU
     /* FIXME: This is a nasty hack because kqemu can't cope with dynamic
        guest ram allocation.  It needs to go away.  */
Signature · 1 line
1.6.3.2.198.g6096d

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw
Quoted 1 line · Stephen Donnelly

Hi Cam,

Hi Steve,

Sorry I haven't answered your email from last Thursday. I'll answer it shortly.

Quoted 10 lines · Cam Macdonell
Quoted 5 lines

The memory for the device allocated as a POSIX shared memory object and then mmapped on to the allocated BAR region in Qemu's allocated memory. That's actually one spot that needs a bit of fixing by passing the already allocated memory object to qemu instead of mmapping on to it.

If you work out how to use pre-existing host memory rather than allocating it inside qemu I would be interested.

How is the host memory pre-existing?

Quoted 4 lines

I would like to have qemu mmap memory from a host char driver, and then in turn register that mapping as a PCI BAR for the guest device. (I know this sounds like pci pass-through, but it isn't.)

In my setup, qemu just calls mmap on the shared memory object that was opened. So I *think* that switching the shm_open(...) to open("/dev/chardev"), might be all that's necessary as long as your char device handles mmapping.

Quoted 2 lines

What I don't understand is how to turn the host address returned from mmap into a ram_addr_t to pass to pci_register_bar.

Memory must be allocated using the qemu RAM functions. Look at qemu_ram_alloc() and qemu_get_ram_ptr() which are a two step process that allocate the memory. Then notice that the ivshmem_ptr is mmapped on to the memory that is returned from the qemu_get_ram_ptr.

pci_register_bar calls a function (the last parameter passed to it) that in turn calls cpu_register_physical_memory which registers the allocated memory (accessed a s->ivshmem_ptr) as the BAR.

Let me know if you have any more questions,
Cam

author
#15 lore raw

Hi Cam,

Quoted 2 lines

Sorry I haven't answered your email from last Thursday.  I'll answer it shortly.

Thanks, I'm still chipping away at it slowly.

Quoted 9 lines
Quoted 9 lines · Cam Macdonell
Quoted 7 lines

The memory for the device allocated as a POSIX shared memory object and then
mmapped on to the allocated BAR region in Qemu's allocated memory. That's
actually one spot that needs a bit of fixing by passing the already allocated memory object to qemu instead of mmapping on to it.

Quoted 4 lines
Quoted 2 lines

If you work out how to use pre-existing host memory rather than allocating it inside qemu I would be interested.

How is the host memory pre-existing?

It comes from outside qemu, it is mapped in via mmap.

Quoted 8 lines
Quoted 3 lines

I would like to have qemu mmap memory from a host char driver, and then in turn register that mapping as a PCI BAR for the guest device. (I know this sounds like pci pass-through, but it isn't.)

In my setup, qemu just calls mmap on the shared memory object that was opened.  So I *think* that switching the shm_open(...) to open("/dev/chardev"), might be all that's necessary as long as your char device handles mmapping.

It does, but it maps memory into the user program rather than out.

Quoted 4 lines
Quoted 2 lines

What I don't understand is how to turn the host address returned from mmap into a ram_addr_t to pass to pci_register_bar.

Memory must be allocated using the qemu RAM functions.

That seems to be the problem. The memory cannot be allocated by qemu_ram_alloc, because it is coming from the mmap call. The memory is already allocated outside the qemu process. mmap can indicate where in the qemu process address space the local mapping should be, but mapping it 'on top' of memory allocated with qemu_ram_alloc doesn't seem to work (I get a BUG in gfn_to_pfn).

Quoted 8 lines

 Look at
qemu_ram_alloc() and qemu_get_ram_ptr() which are a two step process that allocate the memory.  Then notice that the ivshmem_ptr is mmapped on to the memory that is returned from the qemu_get_ram_ptr.

pci_register_bar calls a function (the last parameter passed to it) that in turn calls cpu_register_physical_memory which registers the allocated memory (accessed a s->ivshmem_ptr) as the BAR.

Right, that seems to make sense for your application where you allocate the memory in qemu and then share it externally via shm.

Have you thought about how to use a shm file that has already been allocated by another application? I think you mentioned this as a feature you were going to look at in one of your list posts.

Regards,
Stephen.

#16 lore raw
Quoted 6 lines · Stephen Donnelly
Quoted 5 lines
Quoted 3 lines

What I don't understand is how to turn the host address returned from mmap into a ram_addr_t to pass to pci_register_bar.

Memory must be allocated using the qemu RAM functions.

Quoted 7 lines

That seems to be the problem. The memory cannot be allocated by qemu_ram_alloc, because it is coming from the mmap call. The memory is already allocated outside the qemu process. mmap can indicate where in the qemu process address space the local mapping should be, but mapping it 'on top' of memory allocated with qemu_ram_alloc doesn't seem to work (I get a BUG in gfn_to_pfn).

You need a variant of qemu_ram_alloc() that accepts an fd and offset and mmaps that. A less intrusive, but uglier, alternative is to call qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#17 lore raw
Quoted 14 lines · Avi Kivity
Quoted 4 lines · Stephen Donnelly
Quoted 4 lines
Quoted 3 lines

What I don't understand is how to turn the host address returned from mmap into a ram_addr_t to pass to pci_register_bar.

Quoted 8 lines
Quoted 1 line

Memory must be allocated using the qemu RAM functions.

That seems to be the problem. The memory cannot be allocated by qemu_ram_alloc, because it is coming from the mmap call. The memory is already allocated outside the qemu process. mmap can indicate where in the qemu process address space the local mapping should be, but mapping it 'on top' of memory allocated with qemu_ram_alloc doesn't seem to work (I get a BUG in gfn_to_pfn).

Quoted 2 lines

You need a variant of qemu_ram_alloc() that accepts an fd and offset and mmaps that.

Okay, it sounds like a function to do this is not currently available. That confirms my understanding at least. I will take a look but I don't think I understand the memory management well enough to write this myself.

Quoted 2 lines

A less intrusive, but uglier, alternative is to call qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.

I did try this, but ended up with a BUG on the host in /var/lib/dkms/kvm/84/build/x86/kvm_main.c:1266 gfn_to_pfn() on the line "BUG_ON(!kvm_is_mmio_pfn(pfn));" when the guest accesses the bar.

[1847926.363458] ------------[ cut here ]------------
[1847926.363464] kernel BUG at /var/lib/dkms/kvm/84/build/x86/kvm_main.c:1266!
[1847926.363466] invalid opcode: 0000 [#1] SMP
[1847926.363470] last sysfs file:
/sys/devices/pci0000:00/0000:00:1c.5/0000:02:00.0/net/eth0/statistics/collisions
[1847926.363473] Dumping ftrace buffer:
[1847926.363476]    (ftrace buffer empty)
[1847926.363478] Modules linked in: softcard_driver(P) nls_iso8859_1

vfat fat usb_storage tun nls_utf8 nls_cp437 cifs nfs lockd nfs_acl sunrpc binfmt_misc ppdev bnep ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm video output input_polldev dm_crypt sbp2 lp parport snd_usb_audio snd_pcm_oss snd_hda_intel snd_mixer_oss snd_pcm snd_seq_dummy snd_usb_lib snd_seq_oss snd_seq_midi snd_seq_midi_event uvcvideo compat_ioctl32 snd_rawmidi snd_seq iTCO_wdt videodev snd_timer snd_seq_device iTCO_vendor_support ftdi_sio usbhid v4l1_compat snd_hwdep intel_agp nvidia(P) usbserial snd soundcore snd_page_alloc agpgart pcspkr ohci1394 ieee1394 atl1 mii floppy fbcon tileblit font bitblit softcursor [last unloaded: softcard_driver]

[1847926.363539]
[1847926.363542] Pid: 31516, comm: qemu-system-x86 Tainted: P
 (2.6.28-13-generic #44-Ubuntu) P5K
[1847926.363544] EIP: 0060:[<f7f5961f>] EFLAGS: 00010246 CPU: 1
[1847926.363556] EIP is at gfn_to_pfn+0xff/0x110 [kvm]
[1847926.363558] EAX: 00000000 EBX: 00000000 ECX: f40d30c8 EDX: 00000000
[1847926.363560] ESI: d0baa000 EDI: 00000001 EBP: f2cddbbc ESP: f2cddbac
[1847926.363562]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[1847926.363564] Process qemu-system-x86 (pid: 31516, ti=f2cdc000
task=f163d7f0 task.ti=f2cdc000)
[1847926.363566] Stack:
[1847926.363567]  f2cddbb0 f2cddbc8 00000000 000f2010 f2cddc7c
f7f65f00 00000004 f2cddbd4
[1847926.363573]  f7f5829f 00000004 f2cddbf4 f7f582ec 00000df4
00000004 d0baa000 f185a370
[1847926.363579]  df402c00 0001f719 f2cddc4c f7f66858 f2cddc40
00000004 0001f95f 00000000
[1847926.363585] Call Trace:
[1847926.363588]  [<f7f65f00>] ? kvm_mmu_pte_write+0x160/0x9a0 [kvm]
[1847926.363598]  [<f7f5829f>] ? kvm_read_guest_page+0x2f/0x40 [kvm]
[1847926.363607]  [<f7f582ec>] ? kvm_read_guest+0x3c/0x70 [kvm]
[1847926.363616]  [<f7f66858>] ? paging32_walk_addr+0x118/0x2d0 [kvm]
[1847926.363625]  [<f7f59360>] ? mark_page_dirty+0x10/0x70 [kvm]
[1847926.363634]  [<f7f59412>] ? kvm_write_guest_page+0x52/0x60 [kvm]
[1847926.363643]  [<f7f5becf>] ? emulator_write_phys+0x4f/0x70 [kvm]
[1847926.363652]  [<f7f5dcc8>] ?
emulator_write_emulated_onepage+0x58/0x130 [kvm]
[1847926.363661]  [<f7f5ddf9>] ? emulator_write_emulated+0x59/0x70 [kvm]
[1847926.363674]  [<f7f69d84>] ? x86_emulate_insn+0x414/0x2650 [kvm]
[1847926.363684]  [<c011f714>] ? handle_vm86_fault+0x4c4/0x740
[1847926.363690]  [<c011f714>] ? handle_vm86_fault+0x4c4/0x740
[1847926.363699]  [<f7f681e6>] ? do_insn_fetch+0x76/0xd0 [kvm]
[1847926.363712]  [<c011f716>] ? handle_vm86_fault+0x4c6/0x740
[1847926.363715]  [<c011f716>] ? handle_vm86_fault+0x4c6/0x740
[1847926.363719]  [<f7f6909a>] ? x86_decode_insn+0x54a/0xe20 [kvm]
[1847926.363732]  [<f7f5ecfc>] ? emulate_instruction+0x12c/0x2a0 [kvm]
[1847926.363741]  [<f7f65988>] ? kvm_mmu_page_fault+0x58/0xa0 [kvm]
[1847926.363750]  [<f7e8797a>] ? handle_exception+0x35a/0x400 [kvm_intel]
[1847926.363755]  [<f7e83e97>] ? handle_interrupt_window+0x27/0xc0 [kvm_intel]
[1847926.363760]  [<c011f714>] ? handle_vm86_fault+0x4c4/0x740
[1847926.363763]  [<f7e864e9>] ? kvm_handle_exit+0xd9/0x270 [kvm_intel]
[1847926.363768]  [<f7e87c87>] ? vmx_vcpu_run+0x137/0xa4a [kvm_intel]
[1847926.363772]  [<f7f6d767>] ? kvm_apic_has_interrupt+0x37/0xb0 [kvm]
[1847926.363781]  [<f7f6c0b7>] ? kvm_cpu_has_interrupt+0x27/0x40 [kvm]
[1847926.363790]  [<f7f61306>] ? kvm_arch_vcpu_ioctl_run+0x626/0xb20 [kvm]
[1847926.363799]  [<c015da68>] ? futex_wait+0x358/0x440
[1847926.363804]  [<f7f576e5>] ? kvm_vcpu_ioctl+0x395/0x490 [kvm]
[1847926.363812]  [<c04fec68>] ? _spin_lock+0x8/0x10
[1847926.363815]  [<c015d508>] ? futex_wake+0xc8/0xf0
[1847926.363819]  [<f7f57350>] ? kvm_vcpu_ioctl+0x0/0x490 [kvm]
[1847926.363827]  [<c01ca1d8>] ? vfs_ioctl+0x28/0x90
[1847926.363831]  [<c01ca6be>] ? do_vfs_ioctl+0x5e/0x200
[1847926.363834]  [<c01ca8c3>] ? sys_ioctl+0x63/0x70
[1847926.363836]  [<c0103f6b>] ? sysenter_do_call+0x12/0x2f
[1847926.363840] Code: 29 d3 c1 eb 0c 03 58 44 64 a1 00 e0 7a c0 8b 80
cc 01 00 00 83 c0 34 e8 b0 9b 1f c8 89 d8 e8 89 fc ff ff 85 c0 0f 85
50 ff ff ff <0f> 0b eb fe 8d b6 00 00 00 00 8d bc 27 00 00 00 00 55 89
e5 e8
[1847926.363873] EIP: [<f7f5961f>] gfn_to_pfn+0xff/0x110 [kvm] SS:ESP
0068:f2cddbac
[1847926.363885] ---[ end trace 314ce851a956cf3c ]---

pseudo code in my pci init function is:
{
offset = qemu_ram_alloc(64*1024);
ptr = qemu_get_ram_ptr(offset);

fd = open(charfile, O_RDWR);

mmap(ptr, 64*1024, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_FIXED, fd, 0))

pci_register_bar((PCIDevice *)d, 0, 1024*64, PCI_ADDRESS_SPACE_MEM, mmio_map); }

mmio_map() {
    cpu_register_physical_memory(addr + 0, 64*1024, offset);
}

Regards,
Stephen.

author
#18 lore raw
Quoted 5 lines · Stephen Donnelly
Quoted 4 lines · Avi Kivity

On 07/28/2009 12:32 AM, Stephen Donnelly wrote:

You need a variant of qemu_ram_alloc() that accepts an fd and offset and mmaps that.

I had a go at this, creating qemu_ram_mmap() using qemu_ram_alloc() as a template, but I'm still seeing the same BUG.

Quoted 6 lines
Quoted 2 lines

A less intrusive, but uglier, alternative is to call qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.

I did try this, but ended up with a BUG on the host in /var/lib/dkms/kvm/84/build/x86/kvm_main.c:1266 gfn_to_pfn() on the line "BUG_ON(!kvm_is_mmio_pfn(pfn));" when the guest accesses the bar.

It looks to me from the call trace like the guest is writing to the memory, gfn_to_pfn() from mmu_guess_page_from_pte_write() gets confused because of the mapping.

Inside gfn_to_pfn:

addr = gfn_to_hva(kvm, gfn); correctly returns the host virtual address of the external memory mapping.

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT, presumably because (vma->vm_flags & (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;

In my case addr == vma->vm_start, and vma->vm_pgoff == 0, so pfn ==0. BUG_ON(!kvm_is_mmio_pfn(pfn)) then triggers.

Instrumenting inside gfn_to_pfn I see:
gfn_to_pfn: gfn f2010 gpte f2010000 hva 7f3eac2b0000 pfn 0 npages -14
gfn_to_pfn: vma ffff88022142af18 start 7f3eac2b0000 pgoff 0

Any suggestions what should be happening here?

[ 1826.807846] ------------[ cut here ]------------
[ 1826.807907] kernel BUG at
/build/buildd/linux-2.6.28/arch/x86/kvm/../../../virt/kvm/kvm_main.c:1001!
[ 1826.807985] invalid opcode: 0000 [#1] SMP
[ 1826.808102] last sysfs file: /sys/module/nf_nat/initstate
[ 1826.808159] Dumping ftrace buffer:
[ 1826.808213]    (ftrace buffer empty)
[ 1826.808266] CPU 3
[ 1826.808347] Modules linked in: tun softcard_driver(P)

ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ip v4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables kvm_intel kvm input_polldev video output
bridge stp lp parport iTCO_wdt iTCO_vendor_support psmouse pcspkr
serio_raw joydev i5000_edac edac_core shpchp e1000 us
bhid usb_storage e1000e floppy raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon til
eblit font bitblit softcursor

[ 1826.810269] Pid: 9353, comm: qemu-system-x86 Tainted: P
2.6.28-13-server #45-Ubuntu
[ 1826.810344] RIP: 0010:[<ffffffffa01da853>]  [<ffffffffa01da853>]
gfn_to_pfn+0x153/0x160 [kvm]
[ 1826.810463] RSP: 0018:ffff88022d857958  EFLAGS: 00010246
[ 1826.810518] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88022d4d32a0
[ 1826.810577] RDX: 0000000000000000 RSI: 0000000000000282 RDI: 0000000000000000
[ 1826.810636] RBP: ffff88022d857978 R08: 0000000000000001 R09: ffff88022d857958
[ 1826.810694] R10: 0000000000000003 R11: 0000000000000001 R12: 00000000000f2010
[ 1826.810753] R13: ffff880212cb0000 R14: ffff880212cb0000 R15: ffff880212cb0000
[ 1826.810812] FS:  00007f5253bfd950(0000) GS:ffff88022f1fa380(0000)
knlGS:0000000000000000
[ 1826.810887] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 1826.810943] CR2: 00000000b7eb2044 CR3: 0000000212cac000 CR4: 00000000000026a0
[ 1826.811002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1826.811061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1826.811120] Process qemu-system-x86 (pid: 9353, threadinfo
ffff88022d856000, task ffff88022e0cd980)
[ 1826.811196] Stack:
[ 1826.811246]  ffff88022d857968 0000000000000004 0000000000000004
0000000000000000
[ 1826.811401]  ffff88022d8579b8 ffffffffa01e7ccf ffff88022d8579b8
00000000f2010073
[ 1826.811634]  0000000000000004 ffff880212cb15b0 000000001f402b00
ffff880212cb0000
[ 1826.811913] Call Trace:
[ 1826.811964]  [<ffffffffa01e7ccf>]
mmu_guess_page_from_pte_write+0xaf/0x190 [kvm]
[ 1826.812076]  [<ffffffffa01e820f>] kvm_mmu_pte_write+0x3f/0x4f0 [kvm]
[ 1826.812172]  [<ffffffffa01da9f1>] ? mark_page_dirty+0x11/0x90 [kvm]
[ 1826.812268]  [<ffffffffa01dabe8>] ? kvm_write_guest+0x48/0x90 [kvm]
[ 1826.812364]  [<ffffffffa01de427>] emulator_write_phys+0x47/0x70 [kvm]
[ 1826.812460]  [<ffffffffa01e0e26>]
emulator_write_emulated_onepage+0x66/0x120 [kvm]
[ 1826.812571]  [<ffffffffa01e0f50>] emulator_write_emulated+0x70/0x90 [kvm]
[ 1826.812668]  [<ffffffffa01eb36f>] x86_emulate_insn+0x4ef/0x32e0 [kvm]
[ 1826.812764]  [<ffffffffa01e950e>] ? do_insn_fetch+0x8e/0x100 [kvm]
[ 1826.812860]  [<ffffffffa01e9454>] ? seg_override_base+0x24/0x50 [kvm]
[ 1826.812955]  [<ffffffffa01eacb0>] ? x86_decode_insn+0x7a0/0x970 [kvm]
[ 1826.813051]  [<ffffffffa01e221f>] emulate_instruction+0x15f/0x2f0 [kvm]
[ 1826.813148]  [<ffffffffa01e7bd5>] kvm_mmu_page_fault+0x65/0xb0 [kvm]
[ 1826.813243]  [<ffffffffa020ac5f>] handle_exception+0x2ef/0x360 [kvm_intel]
[ 1826.813338]  [<ffffffffa01eb0a3>] ? x86_emulate_insn+0x223/0x32e0 [kvm]
[ 1826.813434]  [<ffffffffa0209c25>] kvm_handle_exit+0xb5/0x1d0 [kvm_intel]
[ 1826.813526]  [<ffffffff80699643>] ? __down_read+0xc3/0xce
[ 1826.813618]  [<ffffffffa01dd958>] vcpu_enter_guest+0x1f8/0x400 [kvm]
[ 1826.813714]  [<ffffffffa01dfc29>] __vcpu_run+0x69/0x2d0 [kvm]
[ 1826.813751]  [<ffffffffa01e38ea>] kvm_arch_vcpu_ioctl_run+0x8a/0x1f0 [kvm]
[ 1826.813751]  [<ffffffffa01d8582>] kvm_vcpu_ioctl+0x2e2/0x5a0 [kvm]
[ 1826.813751]  [<ffffffff802f6091>] vfs_ioctl+0x31/0xa0
[ 1826.813751]  [<ffffffff802f6445>] do_vfs_ioctl+0x75/0x230
[ 1826.813751]  [<ffffffff802e8216>] ? generic_file_llseek+0x56/0x70
[ 1826.813751]  [<ffffffff802f6699>] sys_ioctl+0x99/0xa0
[ 1826.813751]  [<ffffffff802e70d2>] ? sys_lseek+0x52/0x90
[ 1826.813751]  [<ffffffff8021253a>] system_call_fastpath+0x16/0x1b
[ 1826.813751] Code: 00 00 65 48 8b 04 25 00 00 00 00 48 8b b8 38 02
00 00 48 83 c7 60 e8 dd 23 09 e0 48 89 df e8 45 fe ff ff 85 c0 0f 85
08 ff ff ff <0f> 0b eb fe 66 0f 1f 84 00 00 00 00 00 55 65 8b 14 25 24
00 00
[ 1826.813751] RIP  [<ffffffffa01da853>] gfn_to_pfn+0x153/0x160 [kvm]
[ 1826.813751]  RSP <ffff88022d857958>
[ 1826.816899] ---[ end trace 2437a1197b66fb45 ]---

Stephen.

#19 lore raw
Quoted 16 lines · Stephen Donnelly
Quoted 7 lines
Quoted 3 lines

A less intrusive, but uglier, alternative is to call qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.

I did try this, but ended up with a BUG on the host in /var/lib/dkms/kvm/84/build/x86/kvm_main.c:1266 gfn_to_pfn() on the line "BUG_ON(!kvm_is_mmio_pfn(pfn));" when the guest accesses the bar.

It looks to me from the call trace like the guest is writing to the memory, gfn_to_pfn() from mmu_guess_page_from_pte_write() gets confused because of the mapping.

Inside gfn_to_pfn:

addr = gfn_to_hva(kvm, gfn); correctly returns the host virtual address of the external memory mapping.

Quoted 7 lines

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT, presumably because (vma->vm_flags& (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

It's calculating the pfn according to pfnmap rules.

Quoted 2 lines

In my case addr == vma->vm_start, and vma->vm_pgoff == 0, so pfn ==0.

How did you set up that vma? It should point to the first pfn of your special memory area.

Quoted 2 lines

BUG_ON(!kvm_is_mmio_pfn(pfn)) then triggers.

That's correct behaviour. We expect a page that is not controlled by the kernel here.

Quoted 6 lines

Instrumenting inside gfn_to_pfn I see:
gfn_to_pfn: gfn f2010 gpte f2010000 hva 7f3eac2b0000 pfn 0 npages -14
gfn_to_pfn: vma ffff88022142af18 start 7f3eac2b0000 pgoff 0

Any suggestions what should be happening here?

Well, we need to understand how that vma came into being and why pgoff == 0.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#20 lore raw
Quoted 10 lines · Avi Kivity
Quoted 7 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT, presumably because (vma->vm_flags&  (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

It's calculating the pfn according to pfnmap rules.

From what I understand this will only work when remapping 'main memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

Quoted 4 lines
Quoted 1 line

In my case addr == vma->vm_start, and vma->vm_pgoff == 0, so pfn ==0.

How did you set up that vma?  It should point to the first pfn of your special memory area.

The vma was created with a remap_pfn_range call from another driver. Because this call sets VM_PFNMAP and VM_IO any get_user_pages(_fast) calls will fail.

In this case the host driver was actually just remapping host memory, so I replaced the remap_pfn_range call with a nopage/fault vm_op. This allows the get_user_pages_fast call to succeed, and the mapping now works as expected. This is sufficient for my work at the moment.

I'm still not sure how genuine IO memory (mapped from a driver to userspace with remap_pfn_range or io_remap_page_range) could be mapped into kvm though.

Regards,
Stephen.

#21 lore raw
Quoted 11 lines · Stephen Donnelly
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT,
presumably because (vma->vm_flags& (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

Quoted 7 lines
Quoted 2 lines

It's calculating the pfn according to pfnmap rules.

From what I understand this will only work when remapping 'main
memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

If so, how do they calculate the pfn when mapping pages? kvm needs to be able to do the same thing.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

In my case addr == vma->vm_start, and vma->vm_pgoff == 0, so pfn ==0.

How did you set up that vma? It should point to the first pfn of your special memory area.

Quoted 10 lines

The vma was created with a remap_pfn_range call from another driver. Because this call sets VM_PFNMAP and VM_IO any get_user_pages(_fast) calls will fail.

In this case the host driver was actually just remapping host memory, so I replaced the remap_pfn_range call with a nopage/fault vm_op. This allows the get_user_pages_fast call to succeed, and the mapping now works as expected. This is sufficient for my work at the moment.

Well if the fix is correct we need it too.

Quoted 4 lines

I'm still not sure how genuine IO memory (mapped from a driver to userspace with remap_pfn_range or io_remap_page_range) could be mapped into kvm though.

If it can be mapped to userspace, it can be mapped to kvm. We just need to synchronize the rules.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#22 lore raw
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Stephen Donnelly
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT,
presumably because (vma->vm_flags& (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

It's calculating the pfn according to pfnmap rules.

From what I understand this will only work when remapping 'main
memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

If so, how do they calculate the pfn when mapping pages? kvm needs to be able to do the same thing.

Maybe the simplest thing is to call vma->vm_ops->fault here. Marcelo/Chris? Context is improving gfn_to_pfn() on the mmio path.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#23 lore raw
Quoted 22 lines · Avi Kivity
Quoted 21 lines · Avi Kivity
Quoted 18 lines · Stephen Donnelly
Quoted 11 lines · Avi Kivity
Quoted 7 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT, presumably because (vma->vm_flags&    (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

It's calculating the pfn according to pfnmap rules.

 From what I understand this will only work when remapping 'main memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

Quoted 5 lines
Quoted 2 lines

If so, how do they calculate the pfn when mapping pages?  kvm needs to be able to do the same thing.

Maybe the simplest thing is to call vma->vm_ops->fault here.  Marcelo/Chris? Context is improving gfn_to_pfn() on the mmio path.

If the mapping is made using remap_pfn_range (or io_remap_pfn_range) then there is are vm_ops attached by default.

gfn_to_pfn: vma 0xffff88022c50d498 start 0x7f4b0de9f000 pgoff 0x0 flags 0x844fb vm_ops 0x0000000000000000 fault 0x0000000000000000 file 0xffff88022e408000 major 250 minor 32

From linux/mm.h:

#define VM_PFNMAP 0x00000400 /* Page-ranges managed without "struct
page", just pure PFN */

Stephen.

author
#24 lore raw
In reply to Avi Kivity
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Stephen Donnelly
Quoted 10 lines · Avi Kivity
Quoted 7 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT, presumably because (vma->vm_flags&    (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

It's calculating the pfn according to pfnmap rules.

Quoted 7 lines
Quoted 4 lines

 From what I understand this will only work when remapping 'main memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

If so, how do they calculate the pfn when mapping pages?  kvm needs to be able to do the same thing.

If the vma->vm_file is /dev/mem, then the pg_off will map to physical addresses directly (at least on x86), and the calculation works. If the vma is remapping io memory from a driver, then vma->vm_file will point to the device node for that driver. Perhaps we can do a check for this at least?

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line

In my case addr == vma->vm_start, and vma->vm_pgoff == 0, so pfn ==0.

How did you set up that vma?  It should point to the first pfn of your special memory area.

The vma was created with a remap_pfn_range call from another driver. Because this call sets VM_PFNMAP and VM_IO any get_user_pages(_fast) calls will fail.

Quoted 6 lines
Quoted 4 lines

In this case the host driver was actually just remapping host memory, so I replaced the remap_pfn_range call with a nopage/fault vm_op. This allows the get_user_pages_fast call to succeed, and the mapping now works as expected. This is sufficient for my work at the moment.

Well if the fix is correct we need it too.

The change is to the external (host) driver. If I submit my device for inclusion upstream then the changes for that driver will be needed as well but would not be part of the qemu-kvm tree.

Quoted 6 lines
Quoted 3 lines

I'm still not sure how genuine IO memory (mapped from a driver to userspace with remap_pfn_range or io_remap_page_range) could be mapped into kvm though.

If it can be mapped to userspace, it can be mapped to kvm.  We just need to synchronize the rules.

We can definitely map it into userspace. The problem seems to be how the kvm kernel module translates the guest pfn back to a host physical address.

Is there a kernel equivalent of mmap?

Stephen.

#25 lore raw
Quoted 22 lines · Stephen Donnelly
Quoted 20 lines · Avi Kivity
Quoted 18 lines · Stephen Donnelly
Quoted 11 lines · Avi Kivity
Quoted 7 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT,
presumably because (vma->vm_flags& (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

It's calculating the pfn according to pfnmap rules.

From what I understand this will only work when remapping 'main
memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

Quoted 9 lines
Quoted 3 lines

If so, how do they calculate the pfn when mapping pages? kvm needs to be able to do the same thing.

If the vma->vm_file is /dev/mem, then the pg_off will map to physical addresses directly (at least on x86), and the calculation works. If the vma is remapping io memory from a driver, then vma->vm_file will point to the device node for that driver. Perhaps we can do a check for this at least?

We can't duplicate mm/ in kvm. However, mm/memory.c says:

* The way we recognize COWed pages within VM_PFNMAP mappings is
through the
* rules set up by "remap_pfn_range()": the vma will have the VM_PFNMAP bit
* set, and the vm_pgoff will point to the first PFN mapped: thus every
special
* mapping will always honor the rule
*
* pfn_of_page == vma->vm_pgoff + ((addr - vma->vm_start) >>
PAGE_SHIFT)
*
* And for normal mappings this is false.

So it seems the kvm calculation is right and you should set vm_pgoff in your driver.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I'm still not sure how genuine IO memory (mapped from a driver to userspace with remap_pfn_range or io_remap_page_range) could be mapped into kvm though.

Quoted 9 lines
Quoted 3 lines

If it can be mapped to userspace, it can be mapped to kvm. We just need to synchronize the rules.

We can definitely map it into userspace. The problem seems to be how the kvm kernel module translates the guest pfn back to a host physical address.

Is there a kernel equivalent of mmap?

do_mmap(), but don't use it. Use mmap() from userspace like everyone else.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
author
#26 lore raw
Quoted 56 lines · Avi Kivity
Quoted 34 lines · Stephen Donnelly
Quoted 21 lines · Avi Kivity
Quoted 18 lines · Stephen Donnelly
Quoted 14 lines · Avi Kivity
Quoted 8 lines · Stephen Donnelly

npages = get_user_pages_fast(addr, 1, 1, page); returns -EFAULT, presumably because (vma->vm_flags&      (VM_IO | VM_PFNMAP)).

It takes then unlikely branch, and checks the vma, but I don't understand what it is doing here: pfn = ((addr - vma->vm_start)>> PAGE_SHIFT) + vma->vm_pgoff;

It's calculating the pfn according to pfnmap rules.

Quoted 9 lines
Quoted 5 lines

 From what I understand this will only work when remapping 'main memory', e.g. where the pgoff is equal to the physical page offset? VMAs that remap IO memory will usually set pgoff to 0 for the start of the mapping.

If so, how do they calculate the pfn when mapping pages?  kvm needs to be able to do the same thing.

Quoted 6 lines

If the vma->vm_file is /dev/mem, then the pg_off will map to physical addresses directly (at least on x86), and the calculation works. If the vma is remapping io memory from a driver, then vma->vm_file will point to the device node for that driver. Perhaps we can do a check for this at least?

We can't duplicate mm/ in kvm.  However, mm/memory.c says:

 * The way we recognize COWed pages within VM_PFNMAP mappings is through the
 * rules set up by "remap_pfn_range()": the vma will have the VM_PFNMAP bit
 * set, and the vm_pgoff will point to the first PFN mapped: thus every special
 * mapping will always honor the rule
 *
 *      pfn_of_page == vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT)
 *
 * And for normal mappings this is false.

Quoted 2 lines

So it seems the kvm calculation is right and you should set vm_pgoff in your driver.

That may be true for COW pages, which are main memory, but I don't think it is true for device drivers.

In a device driver the mmap function receives the vma from the OS. The vm_pgoff field contains the offset area in the file. For drivers this is used to determine where to start the map compared to the io base address.

If the driver is mapping io memory to user space it calls io_remap_pfn_range with the pfn for the io memory. The remap_pfn_range call sets the VM_IO and VM_PFNMAP bits in vm_flags. It does not alter the vm_pgoff value.

A simple example is hpet_mmap() in drivers/char/hpet.c, or mbcs_gscr_mmap() in drivers/char/mbcs.c.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

I'm still not sure how genuine IO memory (mapped from a driver to userspace with remap_pfn_range or io_remap_page_range) could be mapped into kvm though.

If it can be mapped to userspace, it can be mapped to kvm.  We just need to
synchronize the rules.

Quoted 7 lines
Quoted 5 lines

We can definitely map it into userspace. The problem seems to be how the kvm kernel module translates the guest pfn back to a host physical address.

Is there a kernel equivalent of mmap?

do_mmap(), but don't use it.  Use mmap() from userspace like everyone else.

Of course you are right, gfn_to_pfn is in user space. There is already a mapping of the memory to the process (from qemu_ram_mmap), the question is how to look it up.

Regards,
Stephen.

#27 lore raw
Quoted 15 lines · Stephen Donnelly
Quoted 15 lines

We can't duplicate mm/ in kvm. However, mm/memory.c says:

* The way we recognize COWed pages within VM_PFNMAP mappings is through the
* rules set up by "remap_pfn_range()": the vma will have the VM_PFNMAP bit
* set, and the vm_pgoff will point to the first PFN mapped: thus every
special
* mapping will always honor the rule
*
* pfn_of_page == vma->vm_pgoff + ((addr - vma->vm_start)>>
PAGE_SHIFT)
*
* And for normal mappings this is false.

Quoted 6 lines
Quoted 3 lines

So it seems the kvm calculation is right and you should set vm_pgoff in your driver.

That may be true for COW pages, which are main memory, but I don't think it is true for device drivers.

No, COW pages have no linear pfn mapping. It's only true for remap_pfn_range).

Quoted 5 lines

In a device driver the mmap function receives the vma from the OS. The vm_pgoff field contains the offset area in the file. For drivers this is used to determine where to start the map compared to the io base address.

Quoted 8 lines

If the driver is mapping io memory to user space it calls io_remap_pfn_range with the pfn for the io memory. The remap_pfn_range call sets the VM_IO and VM_PFNMAP bits in vm_flags. It does not alter the vm_pgoff value.

A simple example is hpet_mmap() in drivers/char/hpet.c, or mbcs_gscr_mmap() in drivers/char/mbcs.c.

io_remap_pfn_range() is remap_pfn_range(), which has this:

        if (addr == vma->vm_start && end == vma->vm_end) {
                vma->vm_pgoff = pfn;
                vma->vm_flags |= VM_PFN_AT_MMAP;
        }

So remap_pfn_range() will alter the pgoff.

Quoted 6 lines
Quoted 2 lines

do_mmap(), but don't use it. Use mmap() from userspace like everyone else.

Of course you are right, gfn_to_pfn is in user space. There is already a mapping of the memory to the process (from qemu_ram_mmap), the question is how to look it up.

I'm totally confused now.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#28 lore raw
Quoted 43 lines · Avi Kivity
Quoted 16 lines · Stephen Donnelly
Quoted 16 lines

We can't duplicate mm/ in kvm.  However, mm/memory.c says:

 * The way we recognize COWed pages within VM_PFNMAP mappings is through the
 * rules set up by "remap_pfn_range()": the vma will have the VM_PFNMAP bit
 * set, and the vm_pgoff will point to the first PFN mapped: thus every special
 * mapping will always honor the rule
 *
 *      pfn_of_page == vma->vm_pgoff + ((addr - vma->vm_start)>> PAGE_SHIFT)
 *
 * And for normal mappings this is false.

Quoted 8 lines
Quoted 4 lines

So it seems the kvm calculation is right and you should set vm_pgoff in your
driver.

That may be true for COW pages, which are main memory, but I don't think it is true for device drivers.

No, COW pages have no linear pfn mapping.  It's only true for remap_pfn_range).

Quoted 5 lines

In a device driver the mmap function receives the vma from the OS. The vm_pgoff field contains the offset area in the file. For drivers this is used to determine where to start the map compared to the io base address.

Quoted 8 lines

If the driver is mapping io memory to user space it calls io_remap_pfn_range with the pfn for the io memory. The remap_pfn_range call sets the VM_IO and VM_PFNMAP bits in vm_flags. It does not alter the vm_pgoff value.

A simple example is hpet_mmap() in drivers/char/hpet.c, or mbcs_gscr_mmap() in drivers/char/mbcs.c.

Quoted 8 lines

io_remap_pfn_range() is remap_pfn_range(), which has this:

       if (addr == vma->vm_start && end == vma->vm_end) { vma->vm_pgoff = pfn; vma->vm_flags |= VM_PFN_AT_MMAP; }

So remap_pfn_range() will alter the pgoff.

Aha! We are looking at different kernels. I should have mentioned I was looking at 2.6.28. In mm/memory.c remap_pfn_range() this has:

	 * There's a horrible special case to handle copy-on-write
	 * behaviour that some programs depend on. We mark the "original"
	 * un-COW'ed pages by matching them up with "vma->vm_pgoff".
	 */
	if (is_cow_mapping(vma->vm_flags)) {
		if (addr != vma->vm_start || end != vma->vm_end)
			return -EINVAL;
		vma->vm_pgoff = pfn;
	}

The macro is:

static inline int is_cow_mapping(unsigned int flags)

{
	return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
}

Because my vma is marked shared, this clause does not operate and vm_pgoff is not modified (it is still 0).

Quoted 1 line

I'm totally confused now.

Sorry about that. The issue is the BUG in gfn_to_pgn where the pfn is not calculated correctly after looking up the vma.

I still don't see how to get the physical address from the vma, since vm_pgoff is zero, and the vm_ops are not filled. The vma does not seem to store the physical base address.

Regards,
Stephen.

#29 lore raw
Quoted 10 lines · Stephen Donnelly
Quoted 3 lines

I'm totally confused now.

Sorry about that. The issue is the BUG in gfn_to_pgn where the pfn is not calculated correctly after looking up the vma.

I still don't see how to get the physical address from the vma, since vm_pgoff is zero, and the vm_ops are not filled. The vma does not seem to store the physical base address.

So it seems the only place the pfns are stored are in the ptes themselves. Is there an API to recover the ptes from a virtual address? We could use that instead.

Signature · 1 line
error compiling committee.c: too many arguments to function
#30 lore raw
In reply to Avi Kivity
Quoted 15 lines · Avi Kivity
Quoted 6 lines · Stephen Donnelly
Quoted 5 lines
Quoted 3 lines

What I don't understand is how to turn the host address returned from mmap into a ram_addr_t to pass to pci_register_bar.

Memory must be allocated using the qemu RAM functions.

Quoted 7 lines

That seems to be the problem. The memory cannot be allocated by qemu_ram_alloc, because it is coming from the mmap call. The memory is already allocated outside the qemu process. mmap can indicate where in the qemu process address space the local mapping should be, but mapping it 'on top' of memory allocated with qemu_ram_alloc doesn't seem to work (I get a BUG in gfn_to_pfn).

Quoted 1 line

You need a variant of qemu_ram_alloc() that accepts an fd and offset and mmaps that. A less intrusive, but uglier, alternative is to call qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.

Hi Avi,

I noticed that the region of memory being allocated for shared memory using qemu_ram_alloc gets added to the total RAM of the system (according to /proc/meminfo). I'm wondering if this is normal/OK since memory for the shared memory device (and similarly VGA RAM) is not intended to be used as regular RAM.

Should memory of devices be reported as part of MemTotal or is something wrong in my use of qemu_ram_alloc()?

Thanks,
Cam

#31 lore raw
Quoted 6 lines · Cam Macdonell
Quoted 1 line

You need a variant of qemu_ram_alloc() that accepts an fd and offset and mmaps that. A less intrusive, but uglier, alternative is to call qemu_ram_alloc() and them mmap(MAP_FIXED) on top of that.

Hi Avi,

I noticed that the region of memory being allocated for shared memory using qemu_ram_alloc gets added to the total RAM of the system (according to /proc/meminfo). I'm wondering if this is normal/OK since memory for the shared memory device (and similarly VGA RAM) is not intended to be used as regular RAM.

qemu_ram_alloc() and the guets /proc/meminfo are totally disconnected. I don't understand how that happened.

Quoted 2 lines

Should memory of devices be reported as part of MemTotal or is something wrong in my use of qemu_ram_alloc()?

You can call qemu_ram_alloc() all you like. Guest memory is determined by the e820 map, which is in turn determined by the -m parameter.

Signature · 1 line
error compiling committee.c: too many arguments to function