virtio-fs: shared file system for virtual machines3
cover virtio-fs: shared file system for virtual machines3
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.
There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------
The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
This device has an extra BAR that contains (up to) 3 regions:
a) a DAX mapping range ('the cache') - into which QEMU mmap's files on behalf of the external daemon; those files are then directly mapped by the guest in a way similar to a DAX backed file system; one advantage of this is that multiple guests all accessing the same files should all be sharing those pages of host cache.
b) An experimental set of mappings for use by a metadata versioning daemon; this mapping is shared between multiple guests and the daemon, but only contains a set of version counters that allow a guest to quickly tell if its metadata is stale.
TODO
----
This is the first RFC, we know we have a bunch of things to clear up:
a) The virtio device specificiation is still in flux and is expected to change
b) We'd like to find ways of reducing the map/unmap latency for DAX
c) The metadata versioning scheme needs to settle out.
d) mmap'ing host files has some interesting side effects; for example if the file gets truncated by the host and then the guest accesses the mapping, KVM can fail the guest hard.
Dr. David Alan Gilbert (6):
virtio: Add shared memory capability
virtio-fs: Add cache BAR
virtio-fs: Add vhost-user slave commands for mapping
virtio-fs: Fill in slave commands for mapping
virtio-fs: Allow mapping of meta data version table
virtio-fs: Allow mapping of journal
Stefan Hajnoczi (1):
virtio: add vhost-user-fs-pci device
configure | 10 + contrib/libvhost-user/libvhost-user.h | 3 + docs/interop/vhost-user.txt | 35 ++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 517 ++++++++++++++++++++ hw/virtio/vhost-user.c | 16 + hw/virtio/virtio-pci.c | 115 +++++ hw/virtio/virtio-pci.h | 19 + include/hw/pci/pci.h | 1 + include/hw/virtio/vhost-user-fs.h | 79 +++ include/standard-headers/linux/virtio_fs.h | 48 ++ include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_pci.h | 9 + 13 files changed, 854 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.h
Signature · 1 line
2.19.2
RFC PATCH 1/7 virtio: Add shared memory capability
From: "Dr. David Alan Gilbert" <[email protected]>
Define a new capability type 'VIRTIO_PCI_CAP_SHARED_MEMORY_CFG' and the data structure 'virtio_pci_shm_cap' to go with it. They allow defining shared memory regions with sizes and offsets of 2^32 and more.
Multiple instances of the capability are allowed and distinguished by a device-specific 'id'.
--- hw/virtio/virtio-pci.c | 20 ++++++++++++++++++++ include/standard-headers/linux/virtio_pci.h | 9 +++++++++ 2 files changed, 29 insertions(+)
hw/virtio/virtio-pci.c +20 -0
@@ -1163,6 +1163,26 @@ static int virtio_pci_add_mem_cap(VirtIOPCIProxy *proxy,return offset; } +static int virtio_pci_add_shm_cap (VirtIOPCIProxy *proxy, +uint8_t bar, +uint64_t offset,uint64_t length, +uint8_t id) +{ +struct virtio_pci_shm_cap cap = { + .cap.cap_len =sizeof cap, + .cap.cfg_type =VIRTIO_PCI_CAP_SHARED_MEMORY_CFG , + }; +uint32_t mask32 = ~0 ; + + cap.cap.bar = bar; + cap.cap.length =cpu_to_le32 (length & mask32); + cap.length_hi =cpu_to_le32 ((length >>32 ) & mask32); + cap.cap.offset =cpu_to_le32 (offset & mask32); + cap.offset_hi =cpu_to_le32 ((offset >>32 ) & mask32); + cap.id = id; +return virtio_pci_add_mem_cap (proxy, &cap.cap); +} +static uint64_t virtio_pci_common_read (void *opaque,hwaddr addr,unsigned size) {
include/standard-headers/linux/virtio_pci.h +9 -0
@@ -113,6 +113,8 @@#define VIRTIO_PCI_CAP_DEVICE_CFG 4/* PCI configuration access */ #define VIRTIO_PCI_CAP_PCI_CFG 5 +/* Additional shared memory capability */ +#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8/* This is the PCI capability header: */ struct virtio_pci_cap {
@@ -163,6 +165,13 @@ struct virtio_pci_cfg_cap {uint8_t pci_cfg_data[4 ];/* Data for BAR access. */ }; +struct virtio_pci_shm_cap { +struct virtio_pci_cap cap; +uint32_t offset_hi;/* Most sig 32 bits of offset */ +uint32_t length_hi;/* Most sig 32 bits of length */ +uint8_t id;/* To distinguish shm chunks */ +}; +/* Macro versions of offsets for the Old Timers! */ #define VIRTIO_PCI_CAP_VNDR 0#define VIRTIO_PCI_CAP_NEXT 1
Signature · 1 line
2.19.2
Re: [Qemu-devel] [RFC PATCH 1/7] virtio: Add shared memory capability
Quoted 27 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Define a new capability type '
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG' and the data structure 'virtio_pci_shm_cap' to go with it. They allow defining shared memory regions with sizes and offsets of 2^32 and more.
Multiple instances of the capability are allowed and distinguished by a device-specific 'id'.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- hw/virtio/virtio-pci.c | 20 ++++++++++++++++++++ include/standard-headers/linux/virtio_pci.h | 9 +++++++++ 2 files changed, 29 insertions(+)+++
b/include/standard-headers/linux/virtio_pci.hinclude/standard-headers/linux/virtio_pci.h +2
-0@@ -113,6 +113,8 @@#define VIRTIO_PCI_CAP_DEVICE_CFG 4/* PCI configuration access */ #define VIRTIO_PCI_CAP_PCI_CFG 5 +/* Additional shared memory capability */ +#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8/* This is the PCI capability header: */ struct virtio_pci_cap {
include/standard-headers/linux/virtio_pci.h struct virtio_pci_cfg_cap { · line 172
Quoted 8 lines
include/standard-headers/linux/virtio_pci.h +5
-0@@ -163,6 +165,13 @@ struct virtio_pci_cfg_cap {uint8_t pci_cfg_data[4 ];/* Data for BAR access. */ }; +struct virtio_pci_shm_cap { +struct virtio_pci_cap cap; +uint32_t offset_hi;/* Most sig 32 bits of offset */ +uint32_t length_hi;/* Most sig 32 bits of length */ +uint8_t id;/* To distinguish shm chunks */
TAB damage.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [RFC PATCH 1/7] virtio: Add shared memory capability
Quoted 28 lines · * Eric Blake ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Define a new capability type '
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG' and the data structure 'virtio_pci_shm_cap' to go with it. They allow defining shared memory regions with sizes and offsets of 2^32 and more.
Multiple instances of the capability are allowed and distinguished by a device-specific 'id'.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- hw/virtio/virtio-pci.c | 20 ++++++++++++++++++++ include/standard-headers/linux/virtio_pci.h | 9 +++++++++ 2 files changed, 29 insertions(+)+++
b/include/standard-headers/linux/virtio_pci.hinclude/standard-headers/linux/virtio_pci.h +2
-0@@ -113,6 +113,8 @@#define VIRTIO_PCI_CAP_DEVICE_CFG 4/* PCI configuration access */ #define VIRTIO_PCI_CAP_PCI_CFG 5 +/* Additional shared memory capability */ +#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8/* This is the PCI capability header: */ struct virtio_pci_cap {@@ -163,6 +165,13 @@ struct virtio_pci_cfg_cap {uint8_t pci_cfg_data[4 ];/* Data for BAR access. */
Quoted 8 lines
include/standard-headers/linux/virtio_pci.h struct virtio_pci_cfg_cap { · line 172
Quoted 6 lines
include/standard-headers/linux/virtio_pci.h +5
-0}; +struct virtio_pci_shm_cap { +struct virtio_pci_cap cap; +uint32_t offset_hi;/* Most sig 32 bits of offset */ +uint32_t length_hi;/* Most sig 32 bits of length */ +uint8_t id;/* To distinguish shm chunks */ TAB damage.
Thanks, fixed.
Dave
Quoted 4 lines
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
RFC PATCH 2/7 virtio: add vhost-user-fs-pci device
In reply to Dr. David Alan Gilbert (git)From: Stefan Hajnoczi <[email protected]>
The virtio-fs virtio device provides shared file system access. The actual file server is implemented in an external vhost-user-fs device backend process.
Launch QEMU like this:
qemu -chardev socket,path=/tmp/vhost-fs.sock,id=chr0
-device vhost-user-fs-pci,tag=myfs,chardev=chr0
--- configure | 10 + hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 297 ++++++++++++++++++++ hw/virtio/virtio-pci.c | 52 ++++ hw/virtio/virtio-pci.h | 18 ++ include/hw/pci/pci.h | 1 + include/hw/virtio/vhost-user-fs.h | 45 +++ include/standard-headers/linux/virtio_fs.h | 41 +++ include/standard-headers/linux/virtio_ids.h | 1 + 9 files changed, 466 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.h
configure +10 -0
@@ -371,6 +371,7 @@ vhost_crypto="no" vhost_scsi="no" vhost_vsock="no" vhost_user="" +vhost_user_fs="no" kvm="no" hax="no" hvf="no"
@@ -878,6 +879,7 @@ Linux) vhost_crypto="yes" vhost_scsi="yes" vhost_vsock="yes" + vhost_user_fs="yes" QEMU_INCLUDES="-I\$(SRC_PATH)/linux-headers -I$PWD/linux-headers $QEMU_INCLUDES" supported_os="yes" libudev="yes"
@@ -1272,6 +1274,10 @@ for opt do ;; --enable-vhost-vsock) vhost_vsock="yes" ;; + --disable-vhost-user-fs) vhost_user_fs="no" + ;; + --enable-vhost-user-fs) vhost_user_fs="yes" + ;; --disable-opengl) opengl="no" ;; --enable-opengl) opengl="yes"
@@ -6054,6 +6060,7 @@ echo "vhost-crypto support $vhost_crypto" echo "vhost-scsi support $vhost_scsi" echo "vhost-vsock support $vhost_vsock" echo "vhost-user support $vhost_user" +echo "vhost-user-fs support $vhost_user_fs" echo "Trace backends $trace_backends" if have_backend "simple"; then echo "Trace output file $trace_file-<pid>"
@@ -6524,6 +6531,9 @@ fi if test "$vhost_user" = "yes" ; then echo "CONFIG_VHOST_USER=y" >> $config_host_mak fi +if test "$vhost_user_fs" = "yes" ; then + echo "CONFIG_VHOST_USER_FS=y" >> $config_host_mak +fi if test "$blobs" = "yes" ; then echo "INSTALL_BLOBS=yes" >> $config_host_mak fi
hw/virtio/Makefile.objs +1 -0
@@ -11,6 +11,7 @@ obj-$(call land,$(CONFIG_VIRTIO_CRYPTO),$(CONFIG_VIRTIO_PCI)) += virtio-crypto-p obj-$(CONFIG_LINUX) += vhost.o vhost-backend.o vhost-user.o obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock.o +obj-$(CONFIG_VHOST_USER_FS) += vhost-user-fs.o endif common-obj-$(call lnot,$(call land,$(CONFIG_VIRTIO),$(CONFIG_LINUX))) += vhost-stub.o
hw/virtio/vhost-user-fs.c new file +297 -0
297 lines not on this page · raw message
hw/virtio/virtio-pci.c +52 -0
64 lines not on this page · raw message
hw/virtio/virtio-pci.h +18 -0
@@ -40,6 +40,9 @@#ifdef CONFIG_VHOST_VSOCK #include "hw/virtio/vhost-vsock.h" #endif +#ifdef CONFIG_VHOST_USER_FS +#include "hw/virtio/vhost-user-fs.h" +#endif typedef struct VirtIOPCIProxy VirtIOPCIProxy ;typedef struct VirtIOBlkPCI VirtIOBlkPCI ;
@@ -57,6 +60,7 @@ typedef struct VirtIOInputHostPCI VirtIOInputHostPCI;typedef struct VirtIOGPUPCI VirtIOGPUPCI ;typedef struct VHostVSockPCI VHostVSockPCI ;typedef struct VirtIOCryptoPCI VirtIOCryptoPCI ; +typedef struct VHostUserFSPCI VHostUserFSPCI ;/* virtio-pci-bus */
@@ -414,6 +418,20 @@ struct VirtIOCryptoPCI {VirtIOCrypto vdev; }; +#ifdef CONFIG_VHOST_USER_FS +/* +* vhost-user-fs-pci: This extends VirtioPCIProxy. +*/ +#define TYPE_VHOST_USER_FS_PCI "vhost-user-fs-pci" +#define VHOST_USER_FS_PCI (obj) \ + OBJECT_CHECK(VHostUserFSPCI, (obj), TYPE_VHOST_USER_FS_PCI) + +struct VHostUserFSPCI { +VirtIOPCIProxy parent_obj; +VHostUserFS vdev; +}; +#endif +/* Virtio ABI version, if we increment this, we break the guest driver. */ #define VIRTIO_PCI_ABI_VERSION 0
include/hw/pci/pci.h +1 -0
@@ -85,6 +85,7 @@ extern bool pci_available;#define PCI_DEVICE_ID_VIRTIO_RNG 0x1005#define PCI_DEVICE_ID_VIRTIO_9P 0x1009#define PCI_DEVICE_ID_VIRTIO_VSOCK 0x1012 +#define PCI_DEVICE_ID_VIRTIO_FS 0x1019#define PCI_VENDOR_ID_REDHAT 0x1b36#define PCI_DEVICE_ID_REDHAT_BRIDGE 0x0001
include/hw/virtio/vhost-user-fs.h new file +45 -0
45 lines not on this page · raw message
include/standard-headers/linux/virtio_fs.h new file +41 -0
41 lines not on this page · raw message
include/standard-headers/linux/virtio_ids.h +1 -0
@@ -43,5 +43,6 @@#define VIRTIO_ID_INPUT 18/* virtio input */ #define VIRTIO_ID_VSOCK 19/* virtio vsock transport */ #define VIRTIO_ID_CRYPTO 20/* virtio crypto */ +#define VIRTIO_ID_FS 26/* virtio filesystem */ #endif /* _LINUX_VIRTIO_IDS_H */
Signature · 1 line
2.19.2
RFC PATCH 3/7 virtio-fs: Add cache BAR
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add a cache BAR into which files will be directly mapped. The size cacn be set with the cache-size= property, e.g.
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=16G
--- hw/virtio/vhost-user-fs.c | 16 ++++++++++++++++ hw/virtio/virtio-pci.c | 19 +++++++++++++++++++ hw/virtio/virtio-pci.h | 1 + include/hw/virtio/vhost-user-fs.h | 2 ++ include/standard-headers/linux/virtio_fs.h | 5 +++++ 5 files changed, 43 insertions(+)
hw/virtio/vhost-user-fs.c +16 -0
@@ -195,6 +195,21 @@ static void vuf_device_realize(DeviceState *dev, Error **errp) VIRTQUEUE_MAX_SIZE);return ; } +if (!is_power_of_2 (fs->conf.cache_size) || + fs->conf.cache_size <sysconf (_SC_PAGESIZE)) { +error_setg (errp,"cache-size property must be a power of 2 " +"no smaller than the page size" ); +return ; + } +/* We need a region with some host memory, 'ram' is the easiest */ +memory_region_init_ram_nomigrate (&fs->cache,OBJECT (vdev), +"virtio-fs-cache" , + fs->conf.cache_size,NULL ); +/* But we don't actually want anyone reading/writing the raw +* area with no cache data. +*/ +mprotect (memory_region_get_ram_ptr (&fs->cache), fs->conf.cache_size, +PROT_NONE ); fs->vhost_user =vhost_user_init ();if (!fs->vhost_user) {
@@ -263,6 +278,7 @@ static Property vuf_properties[] = {DEFINE_PROP_UINT16 ("num-queues" , VHostUserFS, conf.num_queues,1 ),DEFINE_PROP_UINT16 ("queue-size" , VHostUserFS, conf.queue_size,128 ),DEFINE_PROP_STRING ("vhostfd" , VHostUserFS, conf.vhostfd), +DEFINE_PROP_SIZE ("cache-size" , VHostUserFS, conf.cache_size,1ull <<30 ), DEFINE_PROP_END_OF_LIST(), };
hw/virtio/virtio-pci.c +19 -0
@@ -18,6 +18,7 @@#include "qemu/osdep.h" #include "standard-headers/linux/virtio_pci.h" +#include "standard-headers/linux/virtio_fs.h" #include "hw/virtio/virtio.h" #include "hw/virtio/virtio-blk.h" #include "hw/virtio/virtio-net.h"
@@ -2678,9 +2679,27 @@ static void vhost_user_fs_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) {VHostUserFSPCI *dev =VHOST_USER_FS_PCI (vpci_dev);DeviceState *vdev =DEVICE (&dev->vdev); +uint64_t cachesize;qdev_set_parent_bus (vdev,BUS (&vpci_dev->bus));object_property_set_bool (OBJECT (vdev), true,"realized" , errp); + cachesize =memory_region_size (&dev->vdev.cache); + +/* The bar starts with the data/DAX cache +* Others will be added later. +*/ +memory_region_init (&dev->cachebar,OBJECT (vpci_dev), +"vhost-fs-pci-cachebar" , cachesize); +memory_region_add_subregion (&dev->cachebar,0 , &dev->vdev.cache); +virtio_pci_add_shm_cap (vpci_dev,VIRTIO_FS_PCI_CACHE_BAR ,0 , cachesize, +VIRTIO_FS_PCI_SHMCAP_ID_CACHE ); + +/* After 'realized' so the memory region exists */ +pci_register_bar (&vpci_dev->pci_dev,VIRTIO_FS_PCI_CACHE_BAR , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_PREFETCH | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &dev->cachebar); }static void vhost_user_fs_pci_class_init (ObjectClass *klass,void *data)
hw/virtio/virtio-pci.h +1 -0
@@ -429,6 +429,7 @@ struct VirtIOCryptoPCI {struct VHostUserFSPCI {VirtIOPCIProxy parent_obj;VHostUserFS vdev; +MemoryRegion cachebar; };#endif
include/hw/virtio/vhost-user-fs.h +2 -0
@@ -29,6 +29,7 @@ typedef struct {uint16_t num_queues;uint16_t queue_size;char *vhostfd; +size_t cache_size; } VHostUserFSConf; typedef struct {
@@ -40,6 +41,7 @@ typedef struct {VhostUserState *vhost_user;/*< public >*/ + MemoryRegion cache; } VHostUserFS;#endif /* _QEMU_VHOST_USER_FS_H */
include/standard-headers/linux/virtio_fs.h +5 -0
@@ -38,4 +38,9 @@ struct virtio_fs_config { uint32_t num_queues; }QEMU_PACKED ; +#define VIRTIO_FS_PCI_CACHE_BAR 2 + +/* For the id field in virtio_pci_shm_cap */ +#define VIRTIO_FS_PCI_SHMCAP_ID_CACHE 0 +#endif /* _LINUX_VIRTIO_FS_H */
Signature · 1 line
2.19.2
Re: [Qemu-devel] [RFC PATCH 3/7] virtio-fs: Add cache BAR
Quoted 4 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a cache BAR into which files will be directly mapped. The size cacn be set with the cache-size= property, e.g.
s/cacn/can/
Quoted 4 lines
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=16G
Signed-off-by Dr. David Alan Gilbert <[email protected]>---
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [RFC PATCH 3/7] virtio-fs: Add cache BAR
Quoted 7 lines · * Eric Blake ([email protected]) wrote:
Quoted 4 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a cache BAR into which files will be directly mapped. The size cacn be set with the cache-size= property, e.g.
s/cacn/can/
Thanks, fixed.
Quoted 8 lines
Quoted 4 lines
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=16G
Signed-off-by Dr. David Alan Gilbert <[email protected]>---
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
RFC PATCH 4/7 virtio-fs: Add vhost-user slave commands for mapping
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The daemon may request that fd's be mapped into the virtio-fs cache visible to the guest.
These mappings are triggered by commands sent over the slave fd from the daemon.
--- contrib/libvhost-user/libvhost-user.h | 3 +++ docs/interop/vhost-user.txt | 35 +++++++++++++++++++++++++++ hw/virtio/vhost-user-fs.c | 20 +++++++++++++++ hw/virtio/vhost-user.c | 16 ++++++++++++ include/hw/virtio/vhost-user-fs.h | 24 ++++++++++++++++++ 5 files changed, 98 insertions(+)
contrib/libvhost-user/libvhost-user.h +3 -0
@@ -99,6 +99,9 @@ typedef enum VhostUserSlaveRequest {VHOST_USER_SLAVE_IOTLB_MSG =1 ,VHOST_USER_SLAVE_CONFIG_CHANGE_MSG =2 ,VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG =3 , +VHOST_USER_SLAVE_FS_MAP =4 , +VHOST_USER_SLAVE_FS_UNMAP =5 , +VHOST_USER_SLAVE_FS_SYNC =6 , VHOST_USER_SLAVE_MAX } VhostUserSlaveRequest;
docs/interop/vhost-user.txt +35 -0
@@ -815,6 +815,41 @@ Slave message types This request should be sent only when VHOST_USER_PROTOCOL_F_HOST_NOTIFIER protocol feature has been successfully negotiated. + * VHOST_USER_SLAVE_FS_MAP + + Id: 4 + Equivalent ioctl: N/A + Slave payload: fd + n * (offset + address + len) + Master payload: N/A + + Requests that the QEMU mmap the given fd into the virtio-fs cache; + multiple chunks can be mapped in one command. + A reply is generated indicating whether mapping succeeded. + + * VHOST_USER_SLAVE_FS_UNMAP + + Id: 5 + Equivalent ioctl: N/A + Slave payload: n * (address + len) + Master payload: N/A + + Requests that the QEMU un-mmap the given range in the virtio-fs cache; + multiple chunks can be unmapped in one command. + A reply is generated indicating whether unmapping succeeded. + + * VHOST_USER_SLAVE_FS_SYNC + + Id: 6 + Equivalent ioctl: N/A + Slave payload: n * (address + len) + Master payload: N/A + + Requests that the QEMU causes any changes to the virtio-fs cache to + be synchronised with the backing files. Multiple chunks can be synced + in one command. + A reply is generated indicating whether syncing succeeded. + [Semantic details TBD] + VHOST_USER_PROTOCOL_F_REPLY_ACK: ------------------------------- The original vhost-user specification only demands replies for certain
hw/virtio/vhost-user-fs.c +20 -0
@@ -21,6 +21,26 @@#include "hw/virtio/vhost-user-fs.h" #include "monitor/monitor.h" +int vhost_user_fs_slave_map (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm, +int fd) +{ +/* TODO */ +return -1 ; +} + +int vhost_user_fs_slave_unmap (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm) +{ +/* TODO */ +return -1 ; +} + +int vhost_user_fs_slave_sync (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm) +{ +/* TODO */ +return -1 ; +} + +static void vuf_get_config (VirtIODevice *vdev,uint8_t *config) {VHostUserFS *fs =VHOST_USER_FS (vdev);
hw/virtio/vhost-user.c +16 -0
@@ -12,6 +12,7 @@#include "qapi/error.h" #include "hw/virtio/vhost.h" #include "hw/virtio/vhost-user.h" +#include "hw/virtio/vhost-user-fs.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio.h" #include "hw/virtio/virtio-net.h"
@@ -97,6 +98,9 @@ typedef enum VhostUserSlaveRequest {VHOST_USER_SLAVE_IOTLB_MSG =1 ,VHOST_USER_SLAVE_CONFIG_CHANGE_MSG =2 ,VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG =3 , +VHOST_USER_SLAVE_FS_MAP =4 , +VHOST_USER_SLAVE_FS_UNMAP =5 , +VHOST_USER_SLAVE_FS_SYNC =6 , VHOST_USER_SLAVE_MAX } VhostUserSlaveRequest;
@@ -169,6 +173,7 @@ typedef union {VhostUserConfig config;VhostUserCryptoSession session;VhostUserVringArea area; +VhostUserFSSlaveMsg fs; } VhostUserPayload; typedef struct VhostUserMsg {
@@ -1010,6 +1015,17 @@ static void slave_read(void *opaque) ret =vhost_user_slave_handle_vring_host_notifier (dev, &payload.area, fd[0 ]);break ; +#ifdef CONFIG_VHOST_USER_FS +case VHOST_USER_SLAVE_FS_MAP : + ret =vhost_user_fs_slave_map (dev, &payload.fs, fd[0 ]); +break ; +case VHOST_USER_SLAVE_FS_UNMAP : + ret =vhost_user_fs_slave_unmap (dev, &payload.fs); +break ; +case VHOST_USER_SLAVE_FS_SYNC : + ret =vhost_user_fs_slave_sync (dev, &payload.fs); +break ; +#endif default :error_report ("Received unexpected msg type." ); ret = -EINVAL ;
include/hw/virtio/vhost-user-fs.h +24 -0
@@ -23,6 +23,24 @@#define VHOST_USER_FS (obj) \ OBJECT_CHECK(VHostUserFS, (obj), TYPE_VHOST_USER_FS) +/* Structures carried over the slave channel back to QEMU */ +#define VHOST_USER_FS_SLAVE_ENTRIES 8 + +/* For the flags field of VhostUserFSSlaveMsg */ +#define VHOST_USER_FS_FLAG_MAP_R (1ull << 0) +#define VHOST_USER_FS_FLAG_MAP_W (1ull << 1) + +typedef struct { +/* Offsets within the file being mapped */ +uint64_t fd_offset[VHOST_USER_FS_SLAVE_ENTRIES ]; +/* Offsets within the cache */ +uint64_t c_offset[VHOST_USER_FS_SLAVE_ENTRIES ]; +/* Lengths of sections */ +uint64_t len[VHOST_USER_FS_SLAVE_ENTRIES ]; +/* Flags, from VHOST_USER_FS_FLAG_* */ +uint64_t flags[VHOST_USER_FS_SLAVE_ENTRIES ]; +}VhostUserFSSlaveMsg ; + typedef struct { CharBackend chardev; char *tag;
@@ -44,4 +62,10 @@ typedef struct { MemoryRegion cache; } VHostUserFS; +/* Callbacks from the vhost-user code for slave commands */ +int vhost_user_fs_slave_map (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm, +int fd); +int vhost_user_fs_slave_unmap (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm); +int vhost_user_fs_slave_sync (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm); +#endif /* _QEMU_VHOST_USER_FS_H */
Signature · 1 line
2.19.2
RFC PATCH 5/7 virtio-fs: Fill in slave commands for mapping
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Fill in definitions for map, unmap and sync commands.
--- hw/virtio/vhost-user-fs.c | 129 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 6 deletions(-)
hw/virtio/vhost-user-fs.c +123 -6
@@ -24,20 +24,137 @@int vhost_user_fs_slave_map (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm,int fd) { -/* TODO */ -return -1 ; +VHostUserFS *fs =VHOST_USER_FS (dev->vdev); +size_t cache_size = fs->conf.cache_size; +void *cache_host =memory_region_get_ram_ptr (&fs->cache); + +unsigned int i; +int res =0 ; + +if (fd <0 ) { +fprintf (stderr,"%s: Bad fd for map\n" , __func__); +return -1 ; + } + +for (i =0 ; i <VHOST_USER_FS_SLAVE_ENTRIES ; i++) { +if (sm->len[i] ==0 ) { +continue ; + } + +if ((sm->c_offset[i] + sm->len[i]) < sm->len[i] || + (sm->c_offset[i] + sm->len[i]) > cache_size) { +fprintf (stderr,"%s: Bad offset/len for map [%d] %" + PRIx64"+%" PRIx64"\n" , __func__, + i, sm->c_offset[i], sm->len[i]); + res =-1 ; +break ; + } + +if (mmap (cache_host + sm->c_offset[i], sm->len[i], + ((sm->flags[i] &VHOST_USER_FS_FLAG_MAP_R ) ?PROT_READ :0 ) | + ((sm->flags[i] &VHOST_USER_FS_FLAG_MAP_W ) ?PROT_WRITE :0 ), +MAP_SHARED |MAP_FIXED , + fd, sm->fd_offset[i]) != (cache_host + sm->c_offset[i])) { +fprintf (stderr,"%s: map failed err %d [%d] %" + PRIx64"+%" PRIx64" from %" PRIx64"\n" , __func__, + errno, i, sm->c_offset[i], sm->len[i], + sm->fd_offset[i]); + res =-1 ; +break ; + } + } + +if (res) { +/* Something went wrong, unmap them all */ +vhost_user_fs_slave_unmap (dev, sm); + } +return res; }int vhost_user_fs_slave_unmap (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm) { -/* TODO */ -return -1 ; +VHostUserFS *fs =VHOST_USER_FS (dev->vdev); +size_t cache_size = fs->conf.cache_size; +void *cache_host =memory_region_get_ram_ptr (&fs->cache); + +unsigned int i; +int res =0 ; + +/* Note even if one unmap fails we try the rest, since the effect +* is to clean up as much as possible. +*/ +for (i =0 ; i <VHOST_USER_FS_SLAVE_ENTRIES ; i++) { +void *ptr; +if (sm->len[i] ==0 ) { +continue ; + } + +if (sm->len[i] == ~(uint64_t )0 ) { +/* Special case meaning the whole arena */ + sm->len[i] = cache_size; + } + +if ((sm->c_offset[i] + sm->len[i]) < sm->len[i] || + (sm->c_offset[i] + sm->len[i]) > cache_size) { +fprintf (stderr,"%s: Bad offset/len for unmap [%d] %" + PRIx64"+%" PRIx64"\n" , __func__, + i, sm->c_offset[i], sm->len[i]); + res =-1 ; +continue ; + } + + ptr =mmap (cache_host + sm->c_offset[i], sm->len[i], +PROT_NONE ,MAP_PRIVATE |MAP_ANONYMOUS |MAP_FIXED ,-1 ,0 ); +if (ptr != (cache_host + sm->c_offset[i])) { +fprintf (stderr,"%s: mmap failed (%s) [%d] %" + PRIx64"+%" PRIx64" from %" PRIx64" res: %p\n" , + __func__, +strerror (errno), + i, sm->c_offset[i], sm->len[i], + sm->fd_offset[i], ptr); + res =-1 ; + } + } + +return res; }int vhost_user_fs_slave_sync (struct vhost_dev *dev,VhostUserFSSlaveMsg *sm) { -/* TODO */ -return -1 ; +VHostUserFS *fs =VHOST_USER_FS (dev->vdev); +size_t cache_size = fs->conf.cache_size; +void *cache_host =memory_region_get_ram_ptr (&fs->cache); + +unsigned int i; +int res =0 ; + +/* Note even if one sync fails we try the rest */ +for (i =0 ; i <VHOST_USER_FS_SLAVE_ENTRIES ; i++) { +if (sm->len[i] ==0 ) { +continue ; + } + +if ((sm->c_offset[i] + sm->len[i]) < sm->len[i] || + (sm->c_offset[i] + sm->len[i]) > cache_size) { +fprintf (stderr,"%s: Bad offset/len for sync [%d] %" + PRIx64"+%" PRIx64"\n" , __func__, + i, sm->c_offset[i], sm->len[i]); + res =-1 ; +continue ; + } + +if (msync (cache_host + sm->c_offset[i], sm->len[i], +MS_SYNC /* ?? */ )) { +fprintf (stderr,"%s: msync failed (%s) [%d] %" + PRIx64"+%" PRIx64" from %" PRIx64"\n" , __func__, +strerror (errno), + i, sm->c_offset[i], sm->len[i], + sm->fd_offset[i]); + res =-1 ; + } + } + +return res; }
Signature · 1 line
2.19.2
RFC PATCH 6/7 virtio-fs: Allow mapping of meta data version table
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The 'meta data version table' is a block of shared memory mapped between multiple QEMUs and fuse daemons, so that they can be informed of metadata updates. It's typically a shmfs file, and
it's specified as :
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=1G,versiontable=/dev/shm/mdvt1
It gets mapped into the PCI bar after the data cache; it's read only.
--- hw/virtio/vhost-user-fs.c | 36 ++++++++++++++++++++++ hw/virtio/virtio-pci.c | 16 ++++++++-- include/hw/virtio/vhost-user-fs.h | 4 +++ include/standard-headers/linux/virtio_fs.h | 1 + 4 files changed, 55 insertions(+), 2 deletions(-)
hw/virtio/vhost-user-fs.c +36 -0
@@ -296,6 +296,7 @@ static void vuf_device_realize(DeviceState *dev, Error **errp)unsigned int i;size_t len;int ret; +int mdvtfd =-1 ;if (!fs->conf.chardev.chr) {error_setg (errp,"missing chardev" );
@@ -338,6 +339,31 @@ static void vuf_device_realize(DeviceState *dev, Error **errp) "no smaller than the page size");return ; } + +if (fs->conf.mdvtpath) { +struct stat statbuf; + + mdvtfd =open (fs->conf.mdvtpath,O_RDWR ); +if (mdvtfd <0 ) { +error_setg_errno (errp, errno, +"Failed to open meta-data version table '%s'" , + fs->conf.mdvtpath); + +return ; + } +if (fstat (mdvtfd, &statbuf) ==-1 ) { +error_setg_errno (errp, errno, +"Failed to stat meta-data version table '%s'" , + fs->conf.mdvtpath); +close (mdvtfd); +return ; + } + + fs->mdvt_size = statbuf.st_size; + } +fprintf (stderr,"%s: cachesize=%zd mdvt_size=%zd\n" , __func__, + fs->conf.cache_size, fs->mdvt_size); +/* We need a region with some host memory, 'ram' is the easiest */ memory_region_init_ram_nomigrate(&fs->cache, OBJECT(vdev), "virtio-fs-cache",
@@ -348,6 +374,15 @@ static void vuf_device_realize(DeviceState *dev, Error **errp)mprotect (memory_region_get_ram_ptr (&fs->cache), fs->conf.cache_size,PROT_NONE ); + +if (mdvtfd) { +memory_region_init_ram_from_fd (&fs->mdvt,OBJECT (vdev), +"virtio-fs-mdvt" , + fs->mdvt_size, true, mdvtfd,NULL ); +/* The version table is read-only by the guest */ +memory_region_set_readonly (&fs->mdvt, true); + } + fs->vhost_user =vhost_user_init ();if (!fs->vhost_user) {error_setg (errp,"failed to initialize vhost-user" );
@@ -416,6 +451,7 @@ static Property vuf_properties[] = {DEFINE_PROP_UINT16 ("queue-size" , VHostUserFS, conf.queue_size,128 ),DEFINE_PROP_STRING ("vhostfd" , VHostUserFS, conf.vhostfd),DEFINE_PROP_SIZE ("cache-size" , VHostUserFS, conf.cache_size,1ull <<30 ), +DEFINE_PROP_STRING ("versiontable" , VHostUserFS, conf.mdvtpath), DEFINE_PROP_END_OF_LIST(), };
hw/virtio/virtio-pci.c +14 -2
@@ -2680,20 +2680,32 @@ static void vhost_user_fs_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)VHostUserFSPCI *dev =VHOST_USER_FS_PCI (vpci_dev);DeviceState *vdev =DEVICE (&dev->vdev);uint64_t cachesize; +uint64_t totalsize;qdev_set_parent_bus (vdev,BUS (&vpci_dev->bus));object_property_set_bool (OBJECT (vdev), true,"realized" , errp); cachesize =memory_region_size (&dev->vdev.cache); +/* PCIe bar needs to be a power of 2 */ + totalsize =pow2ceil (cachesize + dev->vdev.mdvt_size); +/* The bar starts with the data/DAX cache -* Others will be added later. +* followed by the metadata cache. */ memory_region_init (&dev->cachebar,OBJECT (vpci_dev), -"vhost-fs-pci-cachebar" , cachesize); +"vhost-fs-pci-cachebar" , totalsize);memory_region_add_subregion (&dev->cachebar,0 , &dev->vdev.cache);virtio_pci_add_shm_cap (vpci_dev,VIRTIO_FS_PCI_CACHE_BAR ,0 , cachesize,VIRTIO_FS_PCI_SHMCAP_ID_CACHE ); +if (dev->vdev.mdvt_size) { +memory_region_add_subregion (&dev->cachebar, cachesize, + &dev->vdev.mdvt); +virtio_pci_add_shm_cap (vpci_dev,VIRTIO_FS_PCI_CACHE_BAR , + cachesize, dev->vdev.mdvt_size, +VIRTIO_FS_PCI_SHMCAP_ID_VERTAB ); + } +/* After 'realized' so the memory region exists */ pci_register_bar(&vpci_dev->pci_dev, VIRTIO_FS_PCI_CACHE_BAR, PCI_BASE_ADDRESS_SPACE_MEMORY |
include/hw/virtio/vhost-user-fs.h +4 -0
@@ -48,6 +48,7 @@ typedef struct {uint16_t queue_size;char *vhostfd;size_t cache_size; +char *mdvtpath; } VHostUserFSConf; typedef struct {
@@ -60,6 +61,9 @@ typedef struct {/*< public >*/ MemoryRegion cache; +/* Metadata version table */ +size_t mdvt_size; +MemoryRegion mdvt; } VHostUserFS;/* Callbacks from the vhost-user code for slave commands */
include/standard-headers/linux/virtio_fs.h +1 -0
@@ -42,5 +42,6 @@ struct virtio_fs_config {/* For the id field in virtio_pci_shm_cap */ #define VIRTIO_FS_PCI_SHMCAP_ID_CACHE 0 +#define VIRTIO_FS_PCI_SHMCAP_ID_VERTAB 1#endif /* _LINUX_VIRTIO_FS_H */
Signature · 1 line
2.19.2
RFC PATCH 7/7 virtio-fs: Allow mapping of journal
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The 'journal' is a shared block of RAM between QEMU and it's fuse daemon. It's typically a shmfs file and it's specified as:
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=1G,versiontable=/dev/shm/mdvt1,journal=/dev/shm/journal1
It gets mapped into the PCI bar after the version table.
--- hw/virtio/vhost-user-fs.c | 35 ++++++++++++++++++++-- hw/virtio/virtio-pci.c | 14 ++++++++- include/hw/virtio/vhost-user-fs.h | 4 +++ include/standard-headers/linux/virtio_fs.h | 1 + 4 files changed, 51 insertions(+), 3 deletions(-)
hw/virtio/vhost-user-fs.c +33 -2
@@ -297,6 +297,7 @@ static void vuf_device_realize(DeviceState *dev, Error **errp)size_t len;int ret;int mdvtfd =-1 ; +int journalfd =-1 ;if (!fs->conf.chardev.chr) {error_setg (errp,"missing chardev" );
@@ -361,8 +362,31 @@ static void vuf_device_realize(DeviceState *dev, Error **errp) fs->mdvt_size = statbuf.st_size; } -fprintf (stderr,"%s: cachesize=%zd mdvt_size=%zd\n" , __func__, - fs->conf.cache_size, fs->mdvt_size); +if (fs->conf.journalpath) { +struct stat statbuf; + + journalfd =open (fs->conf.journalpath,O_RDWR ); +if (journalfd <0 ) { +error_setg_errno (errp, errno, +"Failed to open journal '%s'" , + fs->conf.journalpath); + +close (mdvtfd); +return ; + } +if (fstat (journalfd, &statbuf) ==-1 ) { +error_setg_errno (errp, errno, +"Failed to stat journal '%s'" , + fs->conf.journalpath); +close (mdvtfd); +close (journalfd); +return ; + } + + fs->journal_size = statbuf.st_size; + } +fprintf (stderr,"%s: cachesize=%zd mdvt_size=%zd journal_size=%zd\n" , + __func__, fs->conf.cache_size, fs->mdvt_size, fs->journal_size);/* We need a region with some host memory, 'ram' is the easiest */ memory_region_init_ram_nomigrate(&fs->cache, OBJECT(vdev),
@@ -383,6 +407,12 @@ static void vuf_device_realize(DeviceState *dev, Error **errp)memory_region_set_readonly (&fs->mdvt, true); } +if (journalfd) { +memory_region_init_ram_from_fd (&fs->journal,OBJECT (vdev), +"virtio-fs-journal" , + fs->journal_size, true, journalfd,NULL ); + } + fs->vhost_user =vhost_user_init ();if (!fs->vhost_user) {error_setg (errp,"failed to initialize vhost-user" );
@@ -452,6 +482,7 @@ static Property vuf_properties[] = {DEFINE_PROP_STRING ("vhostfd" , VHostUserFS, conf.vhostfd),DEFINE_PROP_SIZE ("cache-size" , VHostUserFS, conf.cache_size,1ull <<30 ),DEFINE_PROP_STRING ("versiontable" , VHostUserFS, conf.mdvtpath), +DEFINE_PROP_STRING ("journal" , VHostUserFS, conf.journalpath), DEFINE_PROP_END_OF_LIST(), };
hw/virtio/virtio-pci.c +13 -1
@@ -2687,10 +2687,12 @@ static void vhost_user_fs_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) cachesize =memory_region_size (&dev->vdev.cache);/* PCIe bar needs to be a power of 2 */ - totalsize =pow2ceil (cachesize + dev->vdev.mdvt_size); + totalsize =pow2ceil (cachesize + dev->vdev.mdvt_size + + dev->vdev.journal_size);/* The bar starts with the data/DAX cache * followed by the metadata cache. +* followed by the journal */ memory_region_init (&dev->cachebar,OBJECT (vpci_dev),"vhost-fs-pci-cachebar" , totalsize);
@@ -2706,6 +2708,16 @@ static void vhost_user_fs_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) VIRTIO_FS_PCI_SHMCAP_ID_VERTAB); } +if (dev->vdev.journal_size) { +memory_region_add_subregion (&dev->cachebar, + cachesize + dev->vdev.mdvt_size, + &dev->vdev.journal); +virtio_pci_add_shm_cap (vpci_dev,VIRTIO_FS_PCI_CACHE_BAR , + cachesize + dev->vdev.mdvt_size, + dev->vdev.journal_size, +VIRTIO_FS_PCI_SHMCAP_ID_JOURNAL ); + } +/* After 'realized' so the memory region exists */ pci_register_bar(&vpci_dev->pci_dev, VIRTIO_FS_PCI_CACHE_BAR, PCI_BASE_ADDRESS_SPACE_MEMORY |
include/hw/virtio/vhost-user-fs.h +4 -0
@@ -49,6 +49,7 @@ typedef struct {char *vhostfd;size_t cache_size;char *mdvtpath; +char *journalpath; } VHostUserFSConf; typedef struct {
@@ -64,6 +65,9 @@ typedef struct {/* Metadata version table */ size_t mdvt_size;MemoryRegion mdvt; +/* Journal */ +size_t journal_size; +MemoryRegion journal; } VHostUserFS;/* Callbacks from the vhost-user code for slave commands */
include/standard-headers/linux/virtio_fs.h +1 -0
@@ -43,5 +43,6 @@ struct virtio_fs_config {/* For the id field in virtio_pci_shm_cap */ #define VIRTIO_FS_PCI_SHMCAP_ID_CACHE 0#define VIRTIO_FS_PCI_SHMCAP_ID_VERTAB 1 +#define VIRTIO_FS_PCI_SHMCAP_ID_JOURNAL 2#endif /* _LINUX_VIRTIO_FS_H */
Signature · 1 line
2.19.2
Re: [Qemu-devel] [RFC PATCH 7/7] virtio-fs: Allow mapping of journal
Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'journal' is a shared block of RAM between QEMU and it's
s/it's/its/ (here, you want possessive)
Quoted 1 line
fuse daemon. It's typically a shmfs file and it's specified as:
whereas here, both uses of "it's" are correct as contractions for "it is" (although I might use just "is" instead of "it's" for that last instance).
Quoted 8 lines
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=1G,versiontable=/dev/shm/mdvt1,journal=/dev/shm/journal1
It gets mapped into the PCI bar after the version table.
Signed-off-by Dr. David Alan Gilbert <[email protected]>---
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [RFC PATCH 7/7] virtio-fs: Allow mapping of journal
Quoted 6 lines · * Eric Blake ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'journal' is a shared block of RAM between QEMU and it's
s/it's/its/ (here, you want possessive)
Fixed.
Dave
Quoted 6 lines
Quoted 1 line
fuse daemon. It's typically a shmfs file and it's specified as:
whereas here, both uses of "it's" are correct as contractions for "it is" (although I might use just "is" instead of "it's" for that last instance).
Quoted 1 line
Quoted 13 lines
Quoted 7 lines
-device vhost-user-fs-pci,chardev=char0,tag=myfs,cache-size=1G,versiontable=/dev/shm/mdvt1,journal=/dev/shm/journal1
It gets mapped into the PCI bar after the version table.
Signed-off-by Dr. David Alan Gilbert <[email protected]>---
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=8
=== TEST SCRIPT END ===
CC ui/gtk.o
CC chardev/char.o
CC chardev/char-console.o
/tmp/qemu-test/src/hw/virtio/virtio-pci.c:1167:12: error: 'virtio_pci_add_shm_cap' defined but not used [-Werror=unused-function]
static int virtio_pci_add_shm_cap(VirtIOPCIProxy *proxy,
^~~~~~~~~~~~~~~~~~~~~~cc1: all warnings being treated as errors
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]
Quoted 17 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
Quoted 7 lines
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
A draft specification for the virtio-fs device is available here:
https://stefanha.github.io/virtio/virtio-fs.html#x1-38800010 (HTML)
https://github.com/stefanha/virtio/commit/e1cac3777ef03bc9c5c8ee91bcc6ba478272e6b6
Stefan
Quoted 28 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------
Quoted 8 lines
The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
Overall I like the virtio-fs architecture more than the virtio-vsock+NFS approach, as virtio-fs feels simpler and closer to virtio-9p with the latter's proxy backends.
I never really liked the idea of having to mess around with the host NFS server to exposed filesystems to guests, as that's systemwide service. The ability to have an isolated virtio-fs backend process per filesystem share per guest is simpler from a mgmt pov.
One think I would like to see though is a general purpose, production quality backend impl that is shipped by the QEMU project. It is fine if projects like Kata want to write a custom impl tailored to their specific needs, but I think QEMU should have something as standard that isn't just demoware.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 28 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------Quoted 8 lines
The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
Overall I like the virtio-fs architecture more than the virtio-vsock+NFS approach, as virtio-fs feels simpler and closer to virtio-9p with the latter's proxy backends.
I never really liked the idea of having to mess around with the host NFS server to exposed filesystems to guests, as that's systemwide service. The ability to have an isolated virtio-fs backend process per filesystem share per guest is simpler from a mgmt pov.
Quoted 5 lines
One think I would like to see though is a general purpose, production quality backend impl that is shipped by the QEMU project. It is fine if projects like Kata want to write a custom impl tailored to their specific needs, but I think QEMU should have something as standard that isn't just demoware.
Our patches sent to libfuse may provide that - after we tidy them up a bit more; but it is the result of adding the fuse example code to qemu's
contrib vhost-user example code. Given that this is the intersection
of so many projects I'm not sure I care which project distributes a working implementation.
Dave
Quoted 6 lines
Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 54 lines · Dr. David Alan Gilbert
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 28 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------Quoted 8 lines
The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
Overall I like the virtio-fs architecture more than the virtio-vsock+NFS approach, as virtio-fs feels simpler and closer to virtio-9p with the latter's proxy backends.
I never really liked the idea of having to mess around with the host NFS server to exposed filesystems to guests, as that's systemwide service. The ability to have an isolated virtio-fs backend process per filesystem share per guest is simpler from a mgmt pov.
Quoted 5 lines
One think I would like to see though is a general purpose, production quality backend impl that is shipped by the QEMU project. It is fine if projects like Kata want to write a custom impl tailored to their specific needs, but I think QEMU should have something as standard that isn't just demoware.
Quoted 5 lines
Our patches sent to libfuse may provide that - after we tidy them up a bit more; but it is the result of adding the fuse example code to qemu's
contrib vhost-user example code. Given that this is the intersection
of so many projects I'm not sure I care which project distributes a working implementation.
Right, but that's my point - the stuff in QEMU's contrib/ directories is just demoware - not something we actually support as QEMU maintainers, nor expect users to run in production. Likewise for stuff in libfuse example/ directory AFAIK.
IMHO we need something whose support status is on a par with what you'd get if we had the impl in-process for the main QEMU system emulator.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 3 lines · Daniel P. Berrangé
On Wed, Dec 12, 2018 at 01:52:03PM +0000, Dr. David Alan Gilbert wrote:
IMHO we need something whose support status is on a par with what you'd get if we had the impl in-process for the main QEMU system emulator.
I agree. Now that virtio-fs has been released we're working on todo items that will make the libfuse code production-quality, including security auditing and jailing of the process.
Once we're confident that this is a production-quality file server it's a matter of moving it out of example/ or contrib/. We might find that the scope of a production-quality file server exceeds libfuse's example/ anyway and need to move it to a new home.
Stefan
Quoted 87 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
This device has an extra BAR that contains (up to) 3 regions:
a) a DAX mapping range ('the cache') - into which QEMU mmap's files on behalf of the external daemon; those files are then directly mapped by the guest in a way similar to a DAX backed file system; one advantage of this is that multiple guests all accessing the same files should all be sharing those pages of host cache.
b) An experimental set of mappings for use by a metadata versioning daemon; this mapping is shared between multiple guests and the daemon, but only contains a set of version counters that allow a guest to quickly tell if its metadata is stale.
TODO
----This is the first RFC, we know we have a bunch of things to clear up:
a) The virtio device specificiation is still in flux and is expected to change
b) We'd like to find ways of reducing the map/unmap latency for DAX
c) The metadata versioning scheme needs to settle out.
d) mmap'ing host files has some interesting side effects; for example if the file gets truncated by the host and then the guest accesses the mapping, KVM can fail the guest hard.
Dr. David Alan Gilbert (6):
virtio: Add shared memory capability
virtio-fs: Add cache BAR
virtio-fs: Add vhost-user slave commands for mapping
virtio-fs: Fill in slave commands for mapping
virtio-fs: Allow mapping of meta data version table
virtio-fs: Allow mapping of journalStefan Hajnoczi (1):
virtio: add vhost-user-fs-pci deviceconfigure | 10 + contrib/libvhost-user/libvhost-user.h | 3 + docs/interop/vhost-user.txt | 35 ++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 517 ++++++++++++++++++++ hw/virtio/vhost-user.c | 16 + hw/virtio/virtio-pci.c | 115 +++++ hw/virtio/virtio-pci.h | 19 + include/hw/pci/pci.h | 1 +
Quoted 9 lines
include/hw/virtio/vhost-user-fs.h | 79 +++ include/standard-headers/linux/virtio_fs.h | 48 ++ include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_pci.h | 9 + 13 files changed, 854 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.h
Hi Dave,
I encounter a problem after running qemu with virtio-fs,
I find I only can mount virtio-fs using the following command:
mount -t virtio_fs /dev/null /mnt/virtio_fs/ -o tag=myfs,rootmode=040000,user_id=0,group_id=0 or mount -t virtio_fs /dev/null /mnt/virtio_fs/ -o tag=myfs,rootmode=040000,user_id=0,group_id=0,dax
Then, I want to know how to use "cache=always" or "cache=none", even "cache=auto", "cache=writeback"?
Thanks,
Yiwen.
Quoted 102 lines · jiangyiwen
Quoted 87 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
This device has an extra BAR that contains (up to) 3 regions:
a) a DAX mapping range ('the cache') - into which QEMU mmap's files on behalf of the external daemon; those files are then directly mapped by the guest in a way similar to a DAX backed file system; one advantage of this is that multiple guests all accessing the same files should all be sharing those pages of host cache.
b) An experimental set of mappings for use by a metadata versioning daemon; this mapping is shared between multiple guests and the daemon, but only contains a set of version counters that allow a guest to quickly tell if its metadata is stale.
TODO
----This is the first RFC, we know we have a bunch of things to clear up:
a) The virtio device specificiation is still in flux and is expected to change
b) We'd like to find ways of reducing the map/unmap latency for DAX
c) The metadata versioning scheme needs to settle out.
d) mmap'ing host files has some interesting side effects; for example if the file gets truncated by the host and then the guest accesses the mapping, KVM can fail the guest hard.
Dr. David Alan Gilbert (6):
virtio: Add shared memory capability
virtio-fs: Add cache BAR
virtio-fs: Add vhost-user slave commands for mapping
virtio-fs: Fill in slave commands for mapping
virtio-fs: Allow mapping of meta data version table
virtio-fs: Allow mapping of journalStefan Hajnoczi (1):
virtio: add vhost-user-fs-pci deviceconfigure | 10 + contrib/libvhost-user/libvhost-user.h | 3 + docs/interop/vhost-user.txt | 35 ++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 517 ++++++++++++++++++++ hw/virtio/vhost-user.c | 16 + hw/virtio/virtio-pci.c | 115 +++++ hw/virtio/virtio-pci.h | 19 + include/hw/pci/pci.h | 1 +Quoted 9 lines
include/hw/virtio/vhost-user-fs.h | 79 +++ include/standard-headers/linux/virtio_fs.h | 48 ++ include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_pci.h | 9 + 13 files changed, 854 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.hHi Dave,
I encounter a problem after running qemu with virtio-fs,
Quoted 8 lines
I find I only can mount virtio-fs using the following command:
mount -tvirtio_fs/dev/null/mnt/virtio_fs/-o tag=myfs,rootmode=040000,user_id=0,group_id=0 or mount -tvirtio_fs/dev/null/mnt/virtio_fs/-o tag=myfs,rootmode=040000,user_id=0,group_id=0,daxThen, I want to know how to use "cache=always" or "cache=none", even "cache=auto", "cache=writeback"?
Thanks,
Yiwen.
Hi Yiwen,
As of now, cache options are libfuse daemon options. So while starting daemon, specify "-o cache=none" or "-o cache=always" etc. One can not specify caching option at virtio-fs mount time.
Thanks
Vivek
Quoted 114 lines · Vivek Goyal
Quoted 102 lines · jiangyiwen
Quoted 87 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
https://marc.info/?l=linux-kernel&m=154446243024251&w=2
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
QEMU's changes
--------------The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
This device has an extra BAR that contains (up to) 3 regions:
a) a DAX mapping range ('the cache') - into which QEMU mmap's files on behalf of the external daemon; those files are then directly mapped by the guest in a way similar to a DAX backed file system; one advantage of this is that multiple guests all accessing the same files should all be sharing those pages of host cache.
b) An experimental set of mappings for use by a metadata versioning daemon; this mapping is shared between multiple guests and the daemon, but only contains a set of version counters that allow a guest to quickly tell if its metadata is stale.
TODO
----This is the first RFC, we know we have a bunch of things to clear up:
a) The virtio device specificiation is still in flux and is expected to change
b) We'd like to find ways of reducing the map/unmap latency for DAX
c) The metadata versioning scheme needs to settle out.
d) mmap'ing host files has some interesting side effects; for example if the file gets truncated by the host and then the guest accesses the mapping, KVM can fail the guest hard.
Dr. David Alan Gilbert (6):
virtio: Add shared memory capability
virtio-fs: Add cache BAR
virtio-fs: Add vhost-user slave commands for mapping
virtio-fs: Fill in slave commands for mapping
virtio-fs: Allow mapping of meta data version table
virtio-fs: Allow mapping of journalStefan Hajnoczi (1):
virtio: add vhost-user-fs-pci deviceconfigure | 10 + contrib/libvhost-user/libvhost-user.h | 3 + docs/interop/vhost-user.txt | 35 ++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 517 ++++++++++++++++++++ hw/virtio/vhost-user.c | 16 + hw/virtio/virtio-pci.c | 115 +++++ hw/virtio/virtio-pci.h | 19 + include/hw/pci/pci.h | 1 +Quoted 9 lines
include/hw/virtio/vhost-user-fs.h | 79 +++ include/standard-headers/linux/virtio_fs.h | 48 ++ include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_pci.h | 9 + 13 files changed, 854 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.hHi Dave,
I encounter a problem after running qemu with virtio-fs,
Quoted 8 lines
I find I only can mount virtio-fs using the following command:
mount -tvirtio_fs/dev/null/mnt/virtio_fs/-o tag=myfs,rootmode=040000,user_id=0,group_id=0 or mount -tvirtio_fs/dev/null/mnt/virtio_fs/-o tag=myfs,rootmode=040000,user_id=0,group_id=0,daxThen, I want to know how to use "cache=always" or "cache=none", even "cache=auto", "cache=writeback"?
Thanks,
Yiwen.Hi Yiwen,
Quoted 9 lines
As of now, cache options are libfuse daemon options. So while starting daemon, specify "-o cache=none" or "-o cache=always" etc. One can not specify caching option at virtio-fs mount time.
Thanks
Vivek.
Ok, I get it, thanks.
Yiwen.
Quoted 17 lines · On Mon, 10 Dec 2018 17:31:44 +0000 "Dr. David Alan Gilbert (git)"…
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
Quoted 8 lines
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
Hi !
This looks like a very promising replacement for virtio-9p, at least with better chances of reaching a production quality level.
Not sure I'll have enough time to step in, but please Cc me on future posts. As virtio-9p maintainer, I'll be happy to help if I can. Also I'll be happy to get rid of the fsdev proxy backend at some point (which I already wanted to replace with a vhost user based solution :-) ).
Cheers,
--
Greg
Quoted 71 lines
QEMU's changes
--------------The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
This device has an extra BAR that contains (up to) 3 regions:
a) a DAX mapping range ('the cache') - into which QEMU mmap's files on behalf of the external daemon; those files are then directly mapped by the guest in a way similar to a DAX backed file system; one advantage of this is that multiple guests all accessing the same files should all be sharing those pages of host cache.
b) An experimental set of mappings for use by a metadata versioning daemon; this mapping is shared between multiple guests and the daemon, but only contains a set of version counters that allow a guest to quickly tell if its metadata is stale.
TODO
----This is the first RFC, we know we have a bunch of things to clear up:
a) The virtio device specificiation is still in flux and is expected to change
b) We'd like to find ways of reducing the map/unmap latency for DAX
c) The metadata versioning scheme needs to settle out.
d) mmap'ing host files has some interesting side effects; for example if the file gets truncated by the host and then the guest accesses the mapping, KVM can fail the guest hard.
Dr. David Alan Gilbert (6):
virtio: Add shared memory capability
virtio-fs: Add cache BAR
virtio-fs: Add vhost-user slave commands for mapping
virtio-fs: Fill in slave commands for mapping
virtio-fs: Allow mapping of meta data version table
virtio-fs: Allow mapping of journalStefan Hajnoczi (1):
virtio: add vhost-user-fs-pci deviceconfigure | 10 + contrib/libvhost-user/libvhost-user.h | 3 + docs/interop/vhost-user.txt | 35 ++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 517 ++++++++++++++++++++ hw/virtio/vhost-user.c | 16 + hw/virtio/virtio-pci.c | 115 +++++ hw/virtio/virtio-pci.h | 19 + include/hw/pci/pci.h | 1 + include/hw/virtio/vhost-user-fs.h | 79 +++ include/standard-headers/linux/virtio_fs.h | 48 ++ include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_pci.h | 9 + 13 files changed, 854 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.h
Quoted 31 lines · * Greg Kurz ([email protected]) wrote:
Quoted 17 lines · On Mon, 10 Dec 2018 17:31:44 +0000 "Dr. David Alan Gilbert (git)"…
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the first RFC for the QEMU side of 'virtio-fs';
a new mechanism for mounting host directories into the guest in a fast, consistent and secure manner. Our primary use case is kata containers, but it should be usable in other scenarios as well.There are corresponding patches being posted to Linux kernel, libfuse and kata lists.
For a fuller design description, and benchmark numbers, please see Vivek's posting of the kernel set here:
Quoted 8 lines
We've got a small website with instructions on how to use it, here:
and all the code is available on gitlab at:
Hi !
Quoted 8 lines
This looks like a very promising replacement for virtio-9p, at least with better chances of reaching a production quality level.
Not sure I'll have enough time to step in, but please Cc me on future posts. As virtio-9p maintainer, I'll be happy to help if I can. Also I'll be happy to get rid of the fsdev proxy backend at some point (which I already wanted to replace with a vhost user based solution :-) ).
Thanks! We'll try and remember to keep you in the loop. If there are any gotchas that you tripped over in 9p that we should watch out for then please give us a prod.
Dave
Dave
Quoted 77 lines
Cheers,
--
GregQuoted 71 lines
QEMU's changes
--------------The QEMU changes are pretty small;
There's a new vhost-user device, which is used to carry a stream of FUSE messages to an external daemon that actually performs all the file IO. The FUSE daemon is an external process in order to achieve better isolation for security and resource control (e.g. number of file descriptors) and also because it's cleaner than trying to integrate libfuse into QEMU.
This device has an extra BAR that contains (up to) 3 regions:
a) a DAX mapping range ('the cache') - into which QEMU mmap's files on behalf of the external daemon; those files are then directly mapped by the guest in a way similar to a DAX backed file system; one advantage of this is that multiple guests all accessing the same files should all be sharing those pages of host cache.
b) An experimental set of mappings for use by a metadata versioning daemon; this mapping is shared between multiple guests and the daemon, but only contains a set of version counters that allow a guest to quickly tell if its metadata is stale.
TODO
----This is the first RFC, we know we have a bunch of things to clear up:
a) The virtio device specificiation is still in flux and is expected to change
b) We'd like to find ways of reducing the map/unmap latency for DAX
c) The metadata versioning scheme needs to settle out.
d) mmap'ing host files has some interesting side effects; for example if the file gets truncated by the host and then the guest accesses the mapping, KVM can fail the guest hard.
Dr. David Alan Gilbert (6):
virtio: Add shared memory capability
virtio-fs: Add cache BAR
virtio-fs: Add vhost-user slave commands for mapping
virtio-fs: Fill in slave commands for mapping
virtio-fs: Allow mapping of meta data version table
virtio-fs: Allow mapping of journalStefan Hajnoczi (1):
virtio: add vhost-user-fs-pci deviceconfigure | 10 + contrib/libvhost-user/libvhost-user.h | 3 + docs/interop/vhost-user.txt | 35 ++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-user-fs.c | 517 ++++++++++++++++++++ hw/virtio/vhost-user.c | 16 + hw/virtio/virtio-pci.c | 115 +++++ hw/virtio/virtio-pci.h | 19 + include/hw/pci/pci.h | 1 + include/hw/virtio/vhost-user-fs.h | 79 +++ include/standard-headers/linux/virtio_fs.h | 48 ++ include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_pci.h | 9 + 13 files changed, 854 insertions(+) create mode 100644 hw/virtio/vhost-user-fs.c create mode 100644 include/hw/virtio/vhost-user-fs.h create mode 100644 include/standard-headers/linux/virtio_fs.h
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK