virtio-fs: shared file system for virtual machines
Subsystems (5)
v5 cover virtio-fs: shared file system for virtual machines
Git tree for this version is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#virtiofs-v5
Only post patches that actually add virtiofs (virtiofs-v5-base..virtiofs-v5).
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
Thanks,
Miklos
---
Dr. David Alan Gilbert (1):
fuse: reserve values for mapping protocol
Michael S. Tsirkin (1):
fuse: reserve byteswapped init opcodes
Stefan Hajnoczi (2):
virtio-fs: add Documentation/filesystems/virtiofs.rst
virtio-fs: add virtiofs filesystem
Documentation/filesystems/index.rst | 10 + Documentation/filesystems/virtiofs.rst | 60 ++ MAINTAINERS | 11 + fs/fuse/Kconfig | 11 + fs/fuse/Makefile | 1 + fs/fuse/fuse_i.h | 9 + fs/fuse/inode.c | 4 + fs/fuse/virtio_fs.c | 1191 ++++++++++++++++++++++++ include/uapi/linux/fuse.h | 12 +- include/uapi/linux/virtio_fs.h | 19 + include/uapi/linux/virtio_ids.h | 1 + 11 files changed, 1328 insertions(+), 1 deletion(-) create mode 100644 Documentation/filesystems/virtiofs.rst create mode 100644 fs/fuse/virtio_fs.c create mode 100644 include/uapi/linux/virtio_fs.h
Signature · 1 line
2.21.0
PATCH v5 1/4 fuse: reserve byteswapped init opcodes
From: "Michael S. Tsirkin" <[email protected]>
virtio fs tunnels fuse over a virtio channel. One issue is two sides might be speaking different endian-ness. To detects this, host side looks at the opcode value in the FUSE_INIT command. Works fine at the moment but might fail if a future version of fuse will use such an opcode for initialization. Let's reserve this opcode so we remember and don't do this.
Same for CUSE_INIT.
--- include/uapi/linux/fuse.h | 4 ++++ 1 file changed, 4 insertions(+)
include/uapi/linux/fuse.h +4 -0
@@ -425,6 +425,10 @@ enum fuse_opcode {/* CUSE specific operations */ CUSE_INIT =4096 , + +/* Reserved opcodes: helpful to detect structure endian-ness */ +CUSE_INIT_BSWAP_RESERVED =1048576 ,/* CUSE_INIT << 8 */ + FUSE_INIT_BSWAP_RESERVED = 436207616, /* FUSE_INIT << 24 */ }; enum fuse_notify_code {
Signature · 1 line
2.21.0
PATCH v5 2/4 fuse: reserve values for mapping protocol
In reply to Miklos SzerediFrom: "Dr. David Alan Gilbert" <[email protected]>
SETUPMAPPING is a command for use with 'virtiofsd', a fuse-over-virtio implementation; it may find use in other fuse impelementations as well in which the kernel does not have access to the address space of the daemon directly.
A SETUPMAPPING operation causes a section of a file to be mapped into a memory window visible to the kernel. The offsets in the file and the window are defined by the kernel performing the operation.
The daemon may reject the request, for reasons including permissions and limited resources.
When a request perfectly overlaps a previous mapping, the previous mapping is replaced. When a mapping partially overlaps a previous mapping, the previous mapping is split into one or two smaller mappings.
REMOVEMAPPING is the complement to SETUPMAPPING; it unmaps a range of mapped files from the window visible to the kernel.
The map_alignment field communicates the alignment constraint for FUSE_SETUPMAPPING/FUSE_REMOVEMAPPING and allows the daemon to constrain the addresses and file offsets chosen by the kernel.
--- include/uapi/linux/fuse.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
include/uapi/linux/fuse.h +7 -1
@@ -133,6 +133,8 @@* * 7.31 * - add FUSE_WRITE_KILL_PRIV flag +* - add FUSE_SETUPMAPPING and FUSE_REMOVEMAPPING +* - add map_alignment to fuse_init_out, add FUSE_MAP_ALIGNMENT flag */ #ifndef _LINUX_FUSE_H
@@ -274,6 +276,7 @@ struct fuse_file_lock {* FUSE_CACHE_SYMLINKS: cache READLINK responses * FUSE_NO_OPENDIR_SUPPORT: kernel supports zero-message opendir * FUSE_EXPLICIT_INVAL_DATA: only invalidate cached pages on explicit request +* FUSE_MAP_ALIGNMENT: map_alignment field is valid */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)
@@ -301,6 +304,7 @@ struct fuse_file_lock {#define FUSE_CACHE_SYMLINKS (1 << 23)#define FUSE_NO_OPENDIR_SUPPORT (1 << 24)#define FUSE_EXPLICIT_INVAL_DATA (1 << 25) +#define FUSE_MAP_ALIGNMENT (1 << 26)/** * CUSE INIT request/reply flags
@@ -422,6 +426,8 @@ enum fuse_opcode {FUSE_RENAME2 =45 ,FUSE_LSEEK =46 ,FUSE_COPY_FILE_RANGE =47 , +FUSE_SETUPMAPPING =48 , +FUSE_REMOVEMAPPING =49 ,/* CUSE specific operations */ CUSE_INIT = 4096,
@@ -656,7 +662,7 @@ struct fuse_init_out {uint32_t max_write;uint32_t time_gran;uint16_t max_pages; -uint16_t padding; +uint16_t map_alignment;uint32_t unused[8 ]; };
Signature · 1 line
2.21.0
PATCH v5 3/4 virtio-fs: add Documentation/filesystems/virtiofs.rst
In reply to Miklos SzerediFrom: Stefan Hajnoczi <[email protected]>
Add information about the new "virtiofs" file system.
--- Documentation/filesystems/index.rst | 10 +++++ Documentation/filesystems/virtiofs.rst | 60 ++++++++++++++++++++++++++ MAINTAINERS | 11 +++++ 3 files changed, 81 insertions(+) create mode 100644 Documentation/filesystems/virtiofs.rst
Documentation/filesystems/index.rst +10 -0
@@ -32,3 +32,13 @@ filesystem implementations. journalling fscrypt + +Filesystems +=========== + +Documentation for filesystem implementations. + +.. toctree:: + :maxdepth: 2 + + virtiofs
Documentation/filesystems/virtiofs.rst new file +60 -0
60 lines not on this page · raw message
MAINTAINERS +11 -0
@@ -17117,6 +17117,17 @@ S: Supported F: drivers/s390/virtio/ F: arch/s390/include/uapi/asm/virtio-ccw.h +VIRTIO FILE SYSTEM +M: Stefan Hajnoczi <[email protected]> +M: Miklos Szeredi <[email protected]> +L: [email protected] +L: [email protected] +W: https://virtio-fs.gitlab.io/ +S: Supported +F: fs/fuse/virtio_fs.c +F: include/uapi/linux/virtio_fs.h +F: Documentation/filesystems/virtiofs.rst + VIRTIO GPU DRIVER M: David Airlie <[email protected]> M: Gerd Hoffmann <[email protected]>
Signature · 1 line
2.21.0
PATCH v5 4/4 virtio-fs: add virtiofs filesystem
In reply to Miklos SzerediFrom: Stefan Hajnoczi <[email protected]>
Add a basic file system module for virtio-fs. This does not yet contain shared data support between host and guest or metadata coherency speedups. However it is already significantly faster than virtio-9p.
Design Overview
===============
With the goal of designing something with better performance and local file system semantics, a bunch of ideas were proposed.
- Use fuse protocol (instead of 9p) for communication between guest and host. Guest kernel will be fuse client and a fuse server will run on host to serve the requests.
- For data access inside guest, mmap portion of file in QEMU address space and guest accesses this memory using dax. That way guest page cache is bypassed and there is only one copy of data (on host). This will also enable mmap(MAP_SHARED) between guests.
- For metadata coherency, there is a shared memory region which contains version number associated with metadata and any guest changing metadata updates version number and other guests refresh metadata on next access. This is yet to be implemented.
How virtio-fs differs from existing approaches
==============================================
The unique idea behind virtio-fs is to take advantage of the co-location of the virtual machine and hypervisor to avoid communication (vmexits).
DAX allows file contents to be accessed without communication with the hypervisor. The shared memory region for metadata avoids communication in the common case where metadata is unchanged.
By replacing expensive communication with cheaper shared memory accesses, we expect to achieve better performance than approaches based on network file system protocols. In addition, this also makes it easier to achieve local file system semantics (coherency).
These techniques are not applicable to network file system protocols since the communications channel is bypassed by taking advantage of shared memory on a local machine. This is why we decided to build virtio-fs rather than focus on 9P or NFS.
Caching Modes
=============
Like virtio-9p, different caching modes are supported which determine the coherency level as well. The “cache=FOO” and “writeback” options control the level of coherence between the guest and host filesystems.
- cache=none
metadata, data and pathname lookup are not cached in guest. They are always fetched from host and any changes are immediately pushed to host.
- cache=always
metadata, data and pathname lookup are cached in guest and never expire.
- cache=auto
metadata and pathname lookup cache expires after a configured amount of time (default is 1 second). Data is cached while the file is open (close to open consistency).
- writeback/no_writeback
These options control the writeback strategy. If writeback is disabled, then normal writes will immediately be synchronized with the host fs. If writeback is enabled, then writes may be cached in the guest until the file is closed or an fsync(2) performed. This option has no effect on mmap-ed writes or writes going through the DAX mechanism.
--- fs/fuse/Kconfig | 11 + fs/fuse/Makefile | 1 + fs/fuse/fuse_i.h | 9 + fs/fuse/inode.c | 4 + fs/fuse/virtio_fs.c | 1191 +++++++++++++++++++++++++++++++ include/uapi/linux/virtio_fs.h | 19 + include/uapi/linux/virtio_ids.h | 1 + 7 files changed, 1236 insertions(+) create mode 100644 fs/fuse/virtio_fs.c create mode 100644 include/uapi/linux/virtio_fs.h
fs/fuse/Kconfig +11 -0
@@ -27,3 +27,14 @@ config CUSE If you want to develop or use a userspace character device based on CUSE, answer Y or M. + +config VIRTIO_FS + tristate "Virtio Filesystem" + depends on FUSE_FS + select VIRTIO + help + The Virtio Filesystem allows guests to mount file systems from the + host. + + If you want to share files between guests or with the host, answer Y + or M.
fs/fuse/Makefile +1 -0
@@ -5,5 +5,6 @@ obj-$(CONFIG_FUSE_FS) += fuse.o obj-$(CONFIG_CUSE) += cuse.o +obj-$(CONFIG_VIRTIO_FS) += virtio_fs.o fuse-objs := dev.o dir.o file.o inode.o control.o xattr.o acl.o readdir.o
fs/fuse/fuse_i.h +9 -0
@@ -353,6 +353,10 @@ struct fuse_req {/** Used to wake up the task waiting for completion of request*/ wait_queue_head_t waitq; +#if IS_ENABLED (CONFIG_VIRTIO_FS ) +/** virtio-fs's physically contiguous buffer for in and out args */ +void *argbuf; +#endif };struct fuse_iqueue ;
@@ -383,6 +387,11 @@ struct fuse_iqueue_ops {*/ void (*wake_pending_and_unlock)(struct fuse_iqueue *fiq) __releases(fiq->lock); + +/** +* Cleanup up when fuse_iqueue is destroyed +*/ +void (*release)(struct fuse_iqueue *fiq); };/** /dev/fuse input queue operations */
fs/fuse/inode.c +4 -0
@@ -630,6 +630,10 @@ EXPORT_SYMBOL_GPL(fuse_conn_init);void fuse_conn_put (struct fuse_conn *fc) {if (refcount_dec_and_test (&fc->count)) { +struct fuse_iqueue *fiq = &fc->iq; + +if (fiq->ops->release) + fiq->ops->release (fiq);put_pid_ns (fc->pid_ns);put_user_ns (fc->user_ns); fc->release (fc);
fs/fuse/virtio_fs.c new file +1191 -0
1,191 lines not on this page · raw message
include/uapi/linux/virtio_fs.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ + +#ifndef _UAPI_LINUX_VIRTIO_FS_H +#define _UAPI_LINUX_VIRTIO_FS_H + +#include <linux/types.h> +#include <linux/virtio_ids.h> +#include <linux/virtio_config.h> +#include <linux/virtio_types.h> + +struct virtio_fs_config { +/* Filesystem name (UTF-8, not NUL-terminated, padded with NULs) */ +__u8 tag[36 ]; + +/* Number of request queues */ +__u32 num_queues; +} __attribute__((packed)); + +#endif /* _UAPI_LINUX_VIRTIO_FS_H */
include/uapi/linux/virtio_ids.h +1 -0
@@ -44,6 +44,7 @@#define VIRTIO_ID_VSOCK 19/* virtio vsock transport */ #define VIRTIO_ID_CRYPTO 20/* virtio crypto */ #define VIRTIO_ID_IOMMU 23/* virtio IOMMU */ +#define VIRTIO_ID_FS 26/* virtio filesystem */ #define VIRTIO_ID_PMEM 27/* virtio pmem */ #endif /* _LINUX_VIRTIO_IDS_H */
Signature · 1 line
2.21.0
Stefan Hajnoczi
maintainerQuoted 9 lines · Miklos Szeredi
Git tree for this version is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#virtiofs-v5Only post patches that actually add virtiofs (virtiofs-v5-base..virtiofs-v5).
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
Thank you! I am investigating freeze/restore.
Stefan
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Stefan Hajnoczi
maintainerQuoted 9 lines · Miklos Szeredi
Git tree for this version is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#virtiofs-v5Only post patches that actually add virtiofs (virtiofs-v5-base..virtiofs-v5).
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
Thank you! I am investigating freeze/restore.
Stefan
Vivek Goyal
maintainerQuoted 9 lines · Miklos Szeredi
Git tree for this version is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#virtiofs-v5Only post patches that actually add virtiofs (virtiofs-v5-base..virtiofs-v5).
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
Hi Miklos,
We are already handling full virtqueue by waiting a bit and retrying. I think TODO in virtio_fs_enqueue_req() is stale. Caller already handles virtqueue full situation by retrying.
Havind said that, this could be improved by using some sort of wait queue or completion privimitve.
Thanks
Vivek
Vivek Goyal
maintainerQuoted 9 lines · Miklos Szeredi
Git tree for this version is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#virtiofs-v5Only post patches that actually add virtiofs (virtiofs-v5-base..virtiofs-v5).
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
Hi Miklos,
We are already handling full virtqueue by waiting a bit and retrying. I think TODO in virtio_fs_enqueue_req() is stale. Caller already handles virtqueue full situation by retrying.
Havind said that, this could be improved by using some sort of wait queue or completion privimitve.
Thanks
Vivek
Miklos Szeredi
maintainerQuoted 12 lines · Vivek Goyal
Quoted 9 lines · Miklos Szeredi
Git tree for this version is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#virtiofs-v5Only post patches that actually add virtiofs (virtiofs-v5-base..virtiofs-v5).
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
Quoted 5 lines
Hi Miklos,
We are already handling full virtqueue by waiting a bit and retrying. I think TODO in
virtio_fs_enqueue_req()is stale. Caller already handles virtqueue full situation by retrying.
Ah.
Quoted 3 lines
Havind said that, this could be improved by using some sort of wait queue or completion privimitve.
Yeah, the request queuing can be cleaned up in several ways. But I think that we might postpone that till after merging with mainline.
Thanks,
Miklos
Stefan Hajnoczi
maintainerQuoted 3 lines · Miklos Szeredi
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
I have researched freeze/restore and come to the conclusion that it needs to be a future feature. It will probably come together with live migration support for reasons mentioned below.
Most virtio devices have fairly simply power management freeze/restore functions that shut down the device and bring it back to the state held in memory, respectively. virtio-fs, as well as virtio-9p and virtio-gpu, are different because they contain session state. It is not easily possible to bring back the state held in memory after the device has been reset.
The following areas of the FUSE protocol are stateful and need special attention:
* FUSE_INIT - this is pretty easy, we must re-negotiate the same settings as before.
* FUSE_LOOKUP -> fuse_inode (inode_map)
The session contains a set of inode numbers that have been looked up using FUSE_LOOKUP. They are ephemeral in the current virtiofsd implementation and vary across device reset. Therefore we are unable to restore the same inode numbers upon restore.
The solution is persistent inode numbers in virtiofsd. This is also needed to make open_by_handle_at(2) work and probably for live migration.
* FUSE_OPEN -> fh (fd_map)
The session contains FUSE file handles for open files. There is currently no way of re-opening a file so that a specific fh is returned. A mechanism to do so probably isn't necessary if the driver can update the fh to the new one produced by the device for all open files instead.
* FUSE_OPENDIR -> fh (dirp_map)
Same story as for FUSE_OPEN but for open directories.
* FUSE_GETLK/SETLK/SETLKW -> (inode->posix_locks and fcntl(F_OFD_GET/SETLK))
The session contains file locks. The driver must reacquire them upon restore. It's unclear what to do when locking fails.
Live migration has the same problem since the FUSE session will be moved to a new virtio-fs device instance. It makes sense to tackle both features together. This is something that can be implemented in the next year, but it's not a quick fix.
Stefan
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Stefan Hajnoczi
maintainerQuoted 3 lines · Miklos Szeredi
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
I have researched freeze/restore and come to the conclusion that it needs to be a future feature. It will probably come together with live migration support for reasons mentioned below.
Most virtio devices have fairly simply power management freeze/restore functions that shut down the device and bring it back to the state held in memory, respectively. virtio-fs, as well as virtio-9p and virtio-gpu, are different because they contain session state. It is not easily possible to bring back the state held in memory after the device has been reset.
The following areas of the FUSE protocol are stateful and need special attention:
* FUSE_INIT - this is pretty easy, we must re-negotiate the same settings as before.
* FUSE_LOOKUP -> fuse_inode (inode_map)
The session contains a set of inode numbers that have been looked up using FUSE_LOOKUP. They are ephemeral in the current virtiofsd implementation and vary across device reset. Therefore we are unable to restore the same inode numbers upon restore.
The solution is persistent inode numbers in virtiofsd. This is also needed to make open_by_handle_at(2) work and probably for live migration.
* FUSE_OPEN -> fh (fd_map)
The session contains FUSE file handles for open files. There is currently no way of re-opening a file so that a specific fh is returned. A mechanism to do so probably isn't necessary if the driver can update the fh to the new one produced by the device for all open files instead.
* FUSE_OPENDIR -> fh (dirp_map)
Same story as for FUSE_OPEN but for open directories.
* FUSE_GETLK/SETLK/SETLKW -> (inode->posix_locks and fcntl(F_OFD_GET/SETLK))
The session contains file locks. The driver must reacquire them upon restore. It's unclear what to do when locking fails.
Live migration has the same problem since the FUSE session will be moved to a new virtio-fs device instance. It makes sense to tackle both features together. This is something that can be implemented in the next year, but it's not a quick fix.
Stefan
Miklos Szeredi
maintainerQuoted 48 lines · Stefan Hajnoczi
Quoted 3 lines · Miklos Szeredi
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
I have researched freeze/restore and come to the conclusion that it needs to be a future feature. It will probably come together with live migration support for reasons mentioned below.
Most virtio devices have fairly simply power management freeze/restore functions that shut down the device and bring it back to the state held in memory, respectively. virtio-fs, as well as virtio-9p and virtio-gpu, are different because they contain session state. It is not easily possible to bring back the state held in memory after the device has been reset.
The following areas of the FUSE protocol are stateful and need special attention:
*
FUSE_INIT- this is pretty easy, we must re-negotiate the same settings as before.*
FUSE_LOOKUP->fuse_inode(inode_map)The session contains a set of inode numbers that have been looked up using
FUSE_LOOKUP. They are ephemeral in the current virtiofsd implementation and vary across device reset. Therefore we are unable to restore the same inode numbers upon restore.The solution is persistent inode numbers in virtiofsd. This is also needed to make
open_by_handle_at(2)work and probably for live migration.*
FUSE_OPEN-> fh (fd_map)The session contains FUSE file handles for open files. There is currently no way of re-opening a file so that a specific fh is returned. A mechanism to do so probably isn't necessary if the driver can update the fh to the new one produced by the device for all open files instead.
*
FUSE_OPENDIR-> fh (dirp_map)Same story as for
FUSE_OPENbut for open directories.* FUSE_GETLK/SETLK/SETLKW -> (
inode->posix_locksandfcntl(F_OFD_GET/SETLK))
Quoted 7 lines
The session contains file locks. The driver must reacquire them upon restore. It's unclear what to do when locking fails.
Live migration has the same problem since the FUSE session will be moved to a new virtio-fs device instance. It makes sense to tackle both features together. This is something that can be implemented in the next year, but it's not a quick fix.
Right. The question for now is: should the freeze silently succeed
(as it seems to do now) or should it fail instead?
I guess normally freezing should be okay, as long as the virtiofsd remains connected while the system is frozen.
I tried to test this with "echo -n mem > /sys/power/state", which indeed resulted in the virtio_fs_freeze() callback being called. However, I couldn't find a way to wake up the system...
Thanks,
Miklos
Stefan Hajnoczi
maintainerQuoted 60 lines · Miklos Szeredi
Quoted 48 lines · Stefan Hajnoczi
Quoted 3 lines · Miklos Szeredi
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
I have researched freeze/restore and come to the conclusion that it needs to be a future feature. It will probably come together with live migration support for reasons mentioned below.
Most virtio devices have fairly simply power management freeze/restore functions that shut down the device and bring it back to the state held in memory, respectively. virtio-fs, as well as virtio-9p and virtio-gpu, are different because they contain session state. It is not easily possible to bring back the state held in memory after the device has been reset.
The following areas of the FUSE protocol are stateful and need special attention:
*
FUSE_INIT- this is pretty easy, we must re-negotiate the same settings as before.*
FUSE_LOOKUP->fuse_inode(inode_map)The session contains a set of inode numbers that have been looked up using
FUSE_LOOKUP. They are ephemeral in the current virtiofsd implementation and vary across device reset. Therefore we are unable to restore the same inode numbers upon restore.The solution is persistent inode numbers in virtiofsd. This is also needed to make
open_by_handle_at(2)work and probably for live migration.*
FUSE_OPEN-> fh (fd_map)The session contains FUSE file handles for open files. There is currently no way of re-opening a file so that a specific fh is returned. A mechanism to do so probably isn't necessary if the driver can update the fh to the new one produced by the device for all open files instead.
*
FUSE_OPENDIR-> fh (dirp_map)Same story as for
FUSE_OPENbut for open directories.* FUSE_GETLK/SETLK/SETLKW -> (
inode->posix_locksandfcntl(F_OFD_GET/SETLK))Quoted 7 lines
The session contains file locks. The driver must reacquire them upon restore. It's unclear what to do when locking fails.
Live migration has the same problem since the FUSE session will be moved to a new virtio-fs device instance. It makes sense to tackle both features together. This is something that can be implemented in the next year, but it's not a quick fix.
Right. The question for now is: should the freeze silently succeed
(as it seems to do now) or should it fail instead?
Quoted 6 lines
I guess normally freezing should be okay, as long as the virtiofsd remains connected while the system is frozen.
I tried to test this with "echo -n mem >
/sys/power/state", which indeed resulted in thevirtio_fs_freeze()callback being called. However, I couldn't find a way to wake up the system...
The issue occurs only on restore. The core virtio driver code resets the device so we lose state and cannot resume.
virtio-9p and virtio-gpu do not implement the .freeze() callback but this is problematic since the system will think freeze succeeded. It's safer for virtio-fs to implement .freeze() and return -EOPNOTSUPP.
Can you squash in a trivial return -EOPNOTSUPP .freeze() function?
Thanks,
Stefan
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Stefan Hajnoczi
maintainerQuoted 60 lines · Miklos Szeredi
Quoted 48 lines · Stefan Hajnoczi
Quoted 3 lines · Miklos Szeredi
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
I have researched freeze/restore and come to the conclusion that it needs to be a future feature. It will probably come together with live migration support for reasons mentioned below.
Most virtio devices have fairly simply power management freeze/restore functions that shut down the device and bring it back to the state held in memory, respectively. virtio-fs, as well as virtio-9p and virtio-gpu, are different because they contain session state. It is not easily possible to bring back the state held in memory after the device has been reset.
The following areas of the FUSE protocol are stateful and need special attention:
*
FUSE_INIT- this is pretty easy, we must re-negotiate the same settings as before.*
FUSE_LOOKUP->fuse_inode(inode_map)The session contains a set of inode numbers that have been looked up using
FUSE_LOOKUP. They are ephemeral in the current virtiofsd implementation and vary across device reset. Therefore we are unable to restore the same inode numbers upon restore.The solution is persistent inode numbers in virtiofsd. This is also needed to make
open_by_handle_at(2)work and probably for live migration.*
FUSE_OPEN-> fh (fd_map)The session contains FUSE file handles for open files. There is currently no way of re-opening a file so that a specific fh is returned. A mechanism to do so probably isn't necessary if the driver can update the fh to the new one produced by the device for all open files instead.
*
FUSE_OPENDIR-> fh (dirp_map)Same story as for
FUSE_OPENbut for open directories.* FUSE_GETLK/SETLK/SETLKW -> (
inode->posix_locksandfcntl(F_OFD_GET/SETLK))Quoted 7 lines
The session contains file locks. The driver must reacquire them upon restore. It's unclear what to do when locking fails.
Live migration has the same problem since the FUSE session will be moved to a new virtio-fs device instance. It makes sense to tackle both features together. This is something that can be implemented in the next year, but it's not a quick fix.
Right. The question for now is: should the freeze silently succeed
(as it seems to do now) or should it fail instead?
Quoted 6 lines
I guess normally freezing should be okay, as long as the virtiofsd remains connected while the system is frozen.
I tried to test this with "echo -n mem >
/sys/power/state", which indeed resulted in thevirtio_fs_freeze()callback being called. However, I couldn't find a way to wake up the system...
The issue occurs only on restore. The core virtio driver code resets the device so we lose state and cannot resume.
virtio-9p and virtio-gpu do not implement the .freeze() callback but this is problematic since the system will think freeze succeeded. It's safer for virtio-fs to implement .freeze() and return -EOPNOTSUPP.
Can you squash in a trivial return -EOPNOTSUPP .freeze() function?
Thanks,
Stefan
Quoted 69 lines · Stefan Hajnoczi
Quoted 60 lines · Miklos Szeredi
Quoted 48 lines · Stefan Hajnoczi
Quoted 3 lines · Miklos Szeredi
I've folded the series from Vivek and fixed a couple of TODO comments myself. AFAICS two issues remain that need to be resolved in the short term, one way or the other: freeze/restore and full virtqueue.
I have researched freeze/restore and come to the conclusion that it needs to be a future feature. It will probably come together with live migration support for reasons mentioned below.
Most virtio devices have fairly simply power management freeze/restore functions that shut down the device and bring it back to the state held in memory, respectively. virtio-fs, as well as virtio-9p and virtio-gpu, are different because they contain session state. It is not easily possible to bring back the state held in memory after the device has been reset.
The following areas of the FUSE protocol are stateful and need special attention:
*
FUSE_INIT- this is pretty easy, we must re-negotiate the same settings as before.*
FUSE_LOOKUP->fuse_inode(inode_map)The session contains a set of inode numbers that have been looked up using
FUSE_LOOKUP. They are ephemeral in the current virtiofsd implementation and vary across device reset. Therefore we are unable to restore the same inode numbers upon restore.The solution is persistent inode numbers in virtiofsd. This is also needed to make
open_by_handle_at(2)work and probably for live migration.*
FUSE_OPEN-> fh (fd_map)The session contains FUSE file handles for open files. There is currently no way of re-opening a file so that a specific fh is returned. A mechanism to do so probably isn't necessary if the driver can update the fh to the new one produced by the device for all open files instead.
*
FUSE_OPENDIR-> fh (dirp_map)Same story as for
FUSE_OPENbut for open directories.* FUSE_GETLK/SETLK/SETLKW -> (
inode->posix_locksandfcntl(F_OFD_GET/SETLK))Quoted 7 lines
The session contains file locks. The driver must reacquire them upon restore. It's unclear what to do when locking fails.
Live migration has the same problem since the FUSE session will be moved to a new virtio-fs device instance. It makes sense to tackle both features together. This is something that can be implemented in the next year, but it's not a quick fix.
Right. The question for now is: should the freeze silently succeed
(as it seems to do now) or should it fail instead?Quoted 6 lines
I guess normally freezing should be okay, as long as the virtiofsd remains connected while the system is frozen.
I tried to test this with "echo -n mem >
/sys/power/state", which indeed resulted in thevirtio_fs_freeze()callback being called. However, I couldn't find a way to wake up the system...
Quoted 8 lines
The issue occurs only on restore. The core virtio driver code resets the device so we lose state and cannot resume.
virtio-9p and virtio-gpu do not implement the .freeze() callback but this is problematic since the system will think freeze succeeded. It's safer for virtio-fs to implement .freeze() and return
-EOPNOTSUPP.Can you squash in a trivial return
-EOPNOTSUPP.freeze() function?
Sure.
Is this a regression from 9p? How easy would it be to restore state in virtques and reconnect to existing virtiofsd (no saving of FUSE state should be required in this case)?
Thanks,
Miklos
Miklos Szeredi
maintainerQuoted 1 line · Miklos Szeredi
Is this a regression from 9p?
Let me answer myself: 9p seems to behave similarly: after suspend/resume it hangs.
So added -EOPNOTSUPP + pr_warn() to the freeze function and verified that this fixes the bad behavior.
Thanks,
Miklos