Skip to contentSkip to conversation

Enable virtio-fs on s390x

RFC Superseded Marc Hartmayer 9 participants 21 messages Jun 25, 2020 – Jul 2, 2020 Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Enable virtio-fs on s390x

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

libvhost-access.h is based on hw/virtio/virtio-access.h.

How to use?

For general instructions how to use virtio-fs (on x86) please have a look at https://virtio-fs.gitlab.io/howto-qemu.html. Most of the instructions can also be applied on s390x.

In short:

1. Install self-compiled QEMU with this patch series applied
2. Prepare host and guest kernel so they support virtio-fs

Start virtiofsd on the host

 $ virtiofsd -f --socket-path=/tmp/vhostqemu -o source=/tmp/shared

Now you can start QEMU in a separate shell on the host:

 $ qemu-system-s390x -machine type=s390-ccw-virtio,accel=kvm,memory-backend=mem \
   -object memory-backend-file,id=mem,size=2G,mem-path=/dev/shm/virtiofs,share=on,prealloc=on,prealloc-threads=1 \
   -chardev socket,id=char0,path=/tmp/vhostqemu -device vhost-user-fs-ccw,queue-size=1024,chardev=char0,tag=myfs \
   -drive if=virtio,file=disk.qcow2 \
   -m 2G -smp 2 -nographic

Log into the guest and mount it

 $ mount -t virtiofs myfs /mnt

Halil Pasic (1):
virtio: add vhost-user-fs-ccw device

Marc Hartmayer (3):
libvhost-user: print invalid address on vu_panic
libvhost-user: handle endianness as mandated by the spec
HACK: Hard-code the libvhost-user.o-cflags for s390x

 Makefile.objs                           |   1 +
 contrib/libvhost-user/libvhost-access.h |  87 +++++++++++++++++
 contrib/libvhost-user/libvhost-user.c   | 124 ++++++++++++------------
 hw/s390x/Makefile.objs                  |   1 +
 hw/s390x/vhost-user-fs-ccw.c            |  74 ++++++++++++++
 5 files changed, 227 insertions(+), 60 deletions(-)
 create mode 100644 contrib/libvhost-user/libvhost-access.h
 create mode 100644 hw/s390x/vhost-user-fs-ccw.c
Signature · 1 line
2.25.4
#1 lore raw

RFC 1/4 virtio: add vhost-user-fs-ccw device

From: Halil Pasic <[email protected]>

Wire up the CCW device for vhost-user-fs.

---
 hw/s390x/Makefile.objs       |  1 +
 hw/s390x/vhost-user-fs-ccw.c | 74 ++++++++++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)
 create mode 100644 hw/s390x/vhost-user-fs-ccw.c
hw/s390x/Makefile.objs +1 -0
index a46a1c7894e0..c4086ec3171e 100644
@@ -20,6 +20,7 @@ obj-$(CONFIG_VIRTIO_NET) += virtio-ccw-net.o
 obj-$(CONFIG_VIRTIO_BLK) += virtio-ccw-blk.o
 obj-$(call land,$(CONFIG_VIRTIO_9P),$(CONFIG_VIRTFS)) += virtio-ccw-9p.o
 obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock-ccw.o
+obj-$(CONFIG_VHOST_USER_FS) += vhost-user-fs-ccw.o
 endif
 obj-y += css-bridge.o
 obj-y += ccw-device.o
hw/s390x/vhost-user-fs-ccw.c new file +74 -0

74 lines not on this page · raw message

Signature · 1 line
2.25.4
#2 lore raw

Re: [RFC 1/4] virtio: add vhost-user-fs-ccw device

Quoted 10 lines · On Thu, 25 Jun 2020 12:04:27 +0200 Marc Hartmayer <[email protected]>…

From: Halil Pasic <[email protected]>

Wire up the CCW device for vhost-user-fs.

---
 hw/s390x/Makefile.objs       |  1 +
 hw/s390x/vhost-user-fs-ccw.c | 74 ++++++++++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)
 create mode 100644 hw/s390x/vhost-user-fs-ccw.c

(...)

hw/s390x/vhost-user-fs-ccw.c line 45

Quoted 3 lines
hw/s390x/vhost-user-fs-ccw.c new file +3 -0
+static void vhost_user_fs_ccw_instance_init(Object *obj)
+{
+    VHostUserFSCcw *dev = VHOST_USER_FS_CCW(obj);

If it turns out to be a bug that pci is not forcing this to virtio-1, you should add

    ccw_dev->force_revision_1 = true;

hw/s390x/vhost-user-fs-ccw.c line 49

Quoted 4 lines
hw/s390x/vhost-user-fs-ccw.c new file +4 -0
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VHOST_USER_FS);
+}

(...)

LGTM.

#3 lore raw

RFC 2/4 libvhost-user: print invalid address on vu_panic

In reply to Marc Hartmayer

This can be helpful for debugging.

---
 contrib/libvhost-user/libvhost-user.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
contrib/libvhost-user/libvhost-user.c +1 -1
index d315db139606..9e8750a9dabc 100644
@@ -2432,7 +2432,7 @@ virtqueue_map_desc(VuDev *dev,
 
         iov[num_sg].iov_base = vu_gpa_to_va(dev, &len, pa);
         if (iov[num_sg].iov_base == NULL) {
-            vu_panic(dev, "virtio: invalid address for buffers");
+            vu_panic(dev, "virtio: invalid address 0x%lx for buffers", pa);
             return;
         }
         iov[num_sg].iov_len = len;
Signature · 1 line
2.25.4
#4 lore raw

RFC 3/4 libvhost-user: handle endianness as mandated by the spec

In reply to Marc Hartmayer

Since virtio existed even before it got standardized, the virtio standard defines the following types of virtio devices:

+ legacy device (pre-virtio 1.0)
+ non-legacy or VIRTIO 1.0 device
+ transitional device (which can act both as legacy and non-legacy)

Virtio 1.0 defines the fields of the virtqueues as little endian, while legacy uses guest's native endian [1]. Currently libvhost-user does not handle virtio endianness at all, i.e. it works only if the native endianness matches with whatever is actually needed. That means things break spectacularly on big-endian targets. Let us handle virtio endianness as required by the virtio specification [1].

[1] https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-210003

---
 contrib/libvhost-user/libvhost-access.h |  88 +++++++++++++++++
 contrib/libvhost-user/libvhost-user.c   | 122 ++++++++++++------------
 2 files changed, 151 insertions(+), 59 deletions(-)
 create mode 100644 contrib/libvhost-user/libvhost-access.h
contrib/libvhost-user/libvhost-access.h new file +88 -0
new file mode 100644
index 000000000000..a4fc334fe134
@@ -0,0 +1,88 @@
+#ifndef LIBVHOST_ACCESS_H
+
+#include "qemu/osdep.h"
+#include "qemu/bswap.h"
+
+#include "libvhost-user.h"
+
+/* TODO attempt to use poisoned TARGET_PPC64/ARM */
+/* #if defined(TARGET_PPC64) || defined(TARGET_ARM) */
+/* #define LEGACY_VIRTIO_IS_BIENDIAN 1 */
+/* #endif */
+
+static inline bool vu_is_big_endian(VuDev *dev)
+{
+    if (!vu_has_feature(dev, VIRTIO_F_VERSION_1)) {
+        /* TODO there is no `device_endian` attribute for VuDev */
+        /* assert(vdev->device_endian != VIRTIO_DEVICE_ENDIAN_UNKNOWN); */
+        /* return vdev->device_endian == VIRTIO_DEVICE_ENDIAN_BIG; */
+    }
+
+    /* Devices conforming to VIRTIO 1.0 or later are always LE. */
+    return false;
+}
+
+static inline bool vu_access_is_big_endian(VuDev *dev)
+{
+#if defined(LEGACY_VIRTIO_IS_BIENDIAN)
+    return vu_is_big_endian(dev);
+#elif defined(TARGET_WORDS_BIGENDIAN)
+    if (vu_has_feature(dev, VIRTIO_F_VERSION_1)) {
+        /* Devices conforming to VIRTIO 1.0 or later are always LE. */
+        return false;
+    }
+    return true;
+#else
+    return false;
+#endif
+}
+
+static inline void vu_stw_p(VuDev *vdev, void *ptr, uint16_t v)
+{
+    if (vu_access_is_big_endian(vdev)) {
+        stw_be_p(ptr, v);
+    } else {
+        stw_le_p(ptr, v);
+    }
+}
+
+static inline void vu_stl_p(VuDev *vdev, void *ptr, uint32_t v)
+{
+    if (vu_access_is_big_endian(vdev)) {
+        stl_be_p(ptr, v);
+    } else {
+        stl_le_p(ptr, v);
+    }
+}
+
+static inline void vu_stq_p(VuDev *vdev, void *ptr, uint64_t v)
+{
+    if (vu_access_is_big_endian(vdev)) {
+        stq_be_p(ptr, v);
+    } else {
+        stq_le_p(ptr, v);
+    }
+}
+
+static inline int vu_lduw_p(VuDev *vdev, const void *ptr)
+{
+    if (vu_access_is_big_endian(vdev))
+        return lduw_be_p(ptr);
+    return lduw_le_p(ptr);
+}
+
+static inline int vu_ldl_p(VuDev *vdev, const void *ptr)
+{
+    if (vu_access_is_big_endian(vdev))
+        return ldl_be_p(ptr);
+    return ldl_le_p(ptr);
+}
+
+static inline uint64_t vu_ldq_p(VuDev *vdev, const void *ptr)
+{
+    if (vu_access_is_big_endian(vdev))
+        return ldq_be_p(ptr);
+    return ldq_le_p(ptr);
+}
+
+#endif /* LIBVHOST_ACCESS_h */
contrib/libvhost-user/libvhost-user.c +63 -59

304 lines not on this page · raw message

Signature · 1 line
2.25.4
#5 lore raw

RFC 4/4 HACK: Hard-code the libvhost-user.o-cflags for s390x

In reply to Marc Hartmayer

This patch exists only to show the actual problem that libvhost-user and it's users are architecture dependent as soon as we're trying to support legacy virtio.

---
 Makefile.objs                           | 1 +
 contrib/libvhost-user/libvhost-access.h | 7 +++----
 2 files changed, 4 insertions(+), 4 deletions(-)
Makefile.objs +1 -0
index 7ce2588b89a3..abfb1912e456 100644
@@ -111,6 +111,7 @@ elf2dmp-obj-y = contrib/elf2dmp/
 ivshmem-client-obj-$(CONFIG_IVSHMEM) = contrib/ivshmem-client/
 ivshmem-server-obj-$(CONFIG_IVSHMEM) = contrib/ivshmem-server/
 libvhost-user-obj-y = contrib/libvhost-user/
+libvhost-user.o-cflags += -iquote $(SRC_PATH)/s390x-softmmu -DNEED_CPU_H
 vhost-user-scsi.o-cflags := $(LIBISCSI_CFLAGS)
 vhost-user-scsi.o-libs := $(LIBISCSI_LIBS)
 vhost-user-scsi-obj-y = contrib/vhost-user-scsi/
contrib/libvhost-user/libvhost-access.h +3 -4
index a4fc334fe134..e9451ae0fbc6 100644
@@ -5,10 +5,9 @@
 
 #include "libvhost-user.h"
 
-/* TODO attempt to use poisoned TARGET_PPC64/ARM */
-/* #if defined(TARGET_PPC64) || defined(TARGET_ARM) */
-/* #define LEGACY_VIRTIO_IS_BIENDIAN 1 */
-/* #endif */
+#if defined(TARGET_PPC64) || defined(TARGET_ARM)
+#define LEGACY_VIRTIO_IS_BIENDIAN 1
+#endif
 
 static inline bool vu_is_big_endian(VuDev *dev)
 {
Signature · 1 line
2.25.4
#6 lore raw
In reply to Marc Hartmayer
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Subject: [RFC 0/4] Enable virtio-fs on s390x
Type: series
Message-id: [email protected]

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu

 - [tag update]      patchew/[email protected] -> patchew/[email protected]
 * [new tag]         patchew/[email protected] -> patchew/[email protected]
 * [new tag]         patchew/[email protected] -> patchew/[email protected]

Switched to a new branch 'test'
b2cd719 HACK: Hard-code the libvhost-user.o-cflags for s390x f0ba6d5 libvhost-user: handle endianness as mandated by the spec 5f2feba libvhost-user: print invalid address on vu_panic 19994e0 virtio: add vhost-user-fs-ccw device

=== OUTPUT BEGIN ===
1/4 Checking commit 19994e01a67c (virtio: add vhost-user-fs-ccw device)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #25:
new file mode 100644

ERROR: space required after that ',' (ctx:VxV)
#86: FILE: hw/s390x/vhost-user-fs-ccw.c:57:

+    device_class_set_props(dc,vhost_user_fs_ccw_properties);
                              ^

total: 1 errors, 1 warnings, 81 lines checked

Patch 1/4 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

2/4 Checking commit 5f2feba1e3c5 (libvhost-user: print invalid address on vu_panic) 3/4 Checking commit f0ba6d5325c1 (libvhost-user: handle endianness as mandated by the spec)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #27:
new file mode 100644

ERROR: braces {} are necessary for all arms of this statement #100: FILE: contrib/libvhost-user/libvhost-access.h:69:
+ if (vu_access_is_big_endian(vdev))
[...]

ERROR: braces {} are necessary for all arms of this statement #107: FILE: contrib/libvhost-user/libvhost-access.h:76:
+ if (vu_access_is_big_endian(vdev))
[...]

ERROR: braces {} are necessary for all arms of this statement #114: FILE: contrib/libvhost-user/libvhost-access.h:83:
+ if (vu_access_is_big_endian(vdev))
[...]

WARNING: line over 80 characters
#375: FILE: contrib/libvhost-user/libvhost-user.c:2515:
+ vu_ldq_p(dev, &desc[i].addr), vu_ldl_p(dev, &desc[i].len));

WARNING: line over 80 characters
#384: FILE: contrib/libvhost-user/libvhost-user.c:2523:
+ vu_ldq_p(dev, &desc[i].addr), vu_ldl_p(dev, &desc[i].len));

total: 3 errors, 3 warnings, 392 lines checked

Patch 3/4 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

4/4 Checking commit b2cd71997828 (HACK: Hard-code the libvhost-user.o-cflags for s390x) === OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#7 lore raw
In reply to Marc Hartmayer
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===

CC qga/qapi-generated/qga-qapi-commands.o
CC qga/qapi-generated/qga-qapi-init-commands.o
In file included from /tmp/qemu-test/src/contrib/libvhost-user/libvhost-user.c:44:0:
/tmp/qemu-test/src/include/qemu/osdep.h:32:27: fatal error: config-target.h: No such file or directory

 #include "config-target.h"
                           ^
compilation terminated.
---
  BUILD   pc-bios/optionrom/linuxboot_dma.raw
  SIGN    pc-bios/optionrom/pvh.bin
  SIGN    pc-bios/optionrom/linuxboot_dma.bin

make: *** [contrib/libvhost-user/libvhost-user.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
File "./tests/docker/docker.py", line 669, in <module>
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=7638277b753647899fcb0c31ef086305', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-uuens814/src/docker-src.2020-06-25-06.13.41.17712:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=7638277b753647899fcb0c31ef086305
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-uuens814/src'
make: *** [docker-run-test-quick@centos7] Error 2

real 2m49.544s
user 0m8.892s

The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#8 lore raw
In reply to Marc Hartmayer
Quoted 9 lines · On Thu, 25 Jun 2020 12:04:26 +0200 Marc Hartmayer <[email protected]>…

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 6 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

The third option looks the most tempting to me. It is a new device, so I don't think there's much of a case to be made for pre-virtio-1 support?

#9 lore raw
Quoted 17 lines · On Thu, 25 Jun 2020 12:17:55 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 14 lines · On Thu, 25 Jun 2020 12:04:26 +0200 Marc Hartmayer <[email protected]>…

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

Quoted 7 lines
Quoted 1 line

A third option would be to refuse legacy altogether.

The third option looks the most tempting to me. It is a new device, so I don't think there's much of a case to be made for pre-virtio-1 support?

Yes, virtio-fs is a new device, but libvhost-user is not specific to virtio-fs.

Regards,
Halil

#10 lore raw
In reply to Marc Hartmayer
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 4 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

When you talk about big vs little endian, are you referring to TCG scenarios with mixed host/guest arch, or arches which can support either endianess, or both ? i guess it doesn't matter actually, as I think the latter forces a specific answer.

Considering that some architectures allow the guest OS to flip between big & little endian as they boot, libvirt cannot know what endianess the guest is using when it launches virtiofsd. It thus cannot pick between two different endianness builds of virtiofsd automatically. This would force the user to tell libvirt what arch the guest is using at the time they define the guest. This is an undesirable restriction for use cases where the admin of the guest OS has no direct control over the host config.

IOW, I think the only practical answer is to have a single binary that automagically does the right thing at runtime according to guest endianess that currently is in use.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw
Quoted 20 lines · On Thu, 25 Jun 2020 11:19:35 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 4 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

When you talk about big vs little endian, are you referring to TCG scenarios with mixed host/guest arch, or arches which can support either endianess, or both ? i guess it doesn't matter actually, as I think the latter forces a specific answer.

Quoted 8 lines

Considering that some architectures allow the guest OS to flip between big & little endian as they boot, libvirt cannot know what endianess the guest is using when it launches virtiofsd. It thus cannot pick between two different endianness builds of virtiofsd automatically. This would force the user to tell libvirt what arch the guest is using at the time they define the guest. This is an undesirable restriction for use cases where the admin of the guest OS has no direct control over the host config.

Right, but that is in practice only a problem for legacy devices, isn't it? The standard says that non-legacy devices use little-endian everywhere; it's the legacy 'device endian' that is causing us headaches.

Which leads to the question: Do we really need to support legacy virtio-fs devices, or can we just force virtio-1, as many (most?) newer virtio devices do?

Quoted 7 lines

IOW, I think the only practical answer is to have a single binary that automagically does the right thing at runtime according to guest endianess that currently is in use.

Regards,
Daniel

#12 lore raw
Quoted 32 lines · Cornelia Huck
Quoted 20 lines · On Thu, 25 Jun 2020 11:19:35 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 4 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

When you talk about big vs little endian, are you referring to TCG scenarios with mixed host/guest arch, or arches which can support either endianess, or both ? i guess it doesn't matter actually, as I think the latter forces a specific answer.

Quoted 8 lines

Considering that some architectures allow the guest OS to flip between big & little endian as they boot, libvirt cannot know what endianess the guest is using when it launches virtiofsd. It thus cannot pick between two different endianness builds of virtiofsd automatically. This would force the user to tell libvirt what arch the guest is using at the time they define the guest. This is an undesirable restriction for use cases where the admin of the guest OS has no direct control over the host config.

Quoted 8 lines

Right, but that is in practice only a problem for legacy devices, isn't it? The standard says that non-legacy devices use little-endian everywhere; it's the legacy 'device endian' that is causing us headaches.

Which leads to the question: Do we really need to support legacy virtio-fs devices, or can we just force virtio-1, as many (most?) newer virtio devices do?

I'd hope virtio-fs is already forced to modern only, as there's no legacy PCI ID assigned to it in the spec.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#13 lore raw
Quoted 38 lines · On Thu, 25 Jun 2020 11:39:24 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 37 lines · Cornelia Huck
Quoted 20 lines · On Thu, 25 Jun 2020 11:19:35 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 4 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

When you talk about big vs little endian, are you referring to TCG scenarios with mixed host/guest arch, or arches which can support either endianess, or both ? i guess it doesn't matter actually, as I think the latter forces a specific answer.

Quoted 8 lines

Considering that some architectures allow the guest OS to flip between big & little endian as they boot, libvirt cannot know what endianess the guest is using when it launches virtiofsd. It thus cannot pick between two different endianness builds of virtiofsd automatically. This would force the user to tell libvirt what arch the guest is using at the time they define the guest. This is an undesirable restriction for use cases where the admin of the guest OS has no direct control over the host config.

Right, but that is in practice only a problem for legacy devices, isn't it? The standard says that non-legacy devices use little-endian everywhere; it's the legacy 'device endian' that is causing us headaches.

Quoted 6 lines
Quoted 3 lines

Which leads to the question: Do we really need to support legacy virtio-fs devices, or can we just force virtio-1, as many (most?) newer virtio devices do?

I'd hope virtio-fs is already forced to modern only, as there's no legacy PCI ID assigned to it in the spec.

I did not find a call to virtio_pci_force_virtio_1(), so apparently not?

#14 lore raw
In reply to Daniel P. Berrangé
Quoted 29 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 4 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

When you talk about big vs little endian, are you referring to TCG scenarios with mixed host/guest arch, or arches which can support either endianess, or both ? i guess it doesn't matter actually, as I think the latter forces a specific answer.

Considering that some architectures allow the guest OS to flip between big & little endian as they boot, libvirt cannot know what endianess the guest is using when it launches virtiofsd. It thus cannot pick between two different endianness builds of virtiofsd automatically. This would force the user to tell libvirt what arch the guest is using at the time they define the guest. This is an undesirable restriction for use cases where the admin of the guest OS has no direct control over the host config.

Quoted 3 lines

IOW, I think the only practical answer is to have a single binary that automagically does the right thing at runtime according to guest endianess that currently is in use.

Modifying the fuse code in virtiofsd to handle cross-endian would be a big job; IMHO unless you really have a cross-endian need I'd just make sure it deals ok with big<->big and little<->little.

Dave

Quoted 8 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#15 lore raw
In reply to Daniel P. Berrangé
Quoted 15 lines · On Thu, 25 Jun 2020 11:19:35 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 4 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

Quoted 5 lines

When you talk about big vs little endian, are you referring to TCG scenarios with mixed host/guest arch, or arches which can support either endianess, or both ? i guess it doesn't matter actually, as I think the latter forces a specific answer.

Our primary concern is big endian cpu and little endian virtio because virto 1.0 or better. But since we talk about a lib here, everything is possible.

Quoted 9 lines

Considering that some architectures allow the guest OS to flip between big & little endian as they boot, libvirt cannot know what endianess the guest is using when it launches virtiofsd. It thus cannot pick between two different endianness builds of virtiofsd automatically. This would force the user to tell libvirt what arch the guest is using at the time they define the guest. This is an undesirable restriction for use cases where the admin of the guest OS has no direct control over the host config.

Quoted 4 lines

IOW, I think the only practical answer is to have a single binary that automagically does the right thing at runtime according to guest endianess that currently is in use.

The problem is that AFAIK there is no way to figure out what is the right thing at the moment. I guess QEMU should now, but not virtiofsd or whatever userspace virtio device. But I will double check if there is something in the protocol addressing this.

Regards,
Halil

#16 lore raw
In reply to Marc Hartmayer
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 6 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

I suggest the following:

1. Combinations that worked with libvhost-user in the past must not break.

2. New combinations should only support VIRTIO 1.0 and later.

This means continue to allow Legacy mode devices where they already run today but don't add new code for the cases that didn't work.

Stefan

#17 lore raw
Quoted 17 lines · Stefan Hajnoczi
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 6 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

Quoted 8 lines

I suggest the following:

1. Combinations that worked with libvhost-user in the past must not break.

2. New combinations should only support VIRTIO 1.0 and later.

This means continue to allow Legacy mode devices where they already run today but don't add new code for the cases that didn't work.

What I'm missing here is what PCI product ID was being used when the current impl is in legacy/transitional mode ?

Normally legacy and transitional mode devices need an explicit PCI ID reserved, where as modern-only devices have a PCI ID derived from their VirtIO ID + a fixed offset.

Was this mistakenly using a VirtIO ID + fixed offset for the legacy mode too ?

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw
Quoted 30 lines · Daniel P. Berrangé
Quoted 17 lines · Stefan Hajnoczi
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 6 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

Quoted 8 lines

I suggest the following:

1. Combinations that worked with libvhost-user in the past must not break.

2. New combinations should only support VIRTIO 1.0 and later.

This means continue to allow Legacy mode devices where they already run today but don't add new code for the cases that didn't work.

What I'm missing here is what PCI product ID was being used when the current impl is in legacy/transitional mode ?

Quoted 6 lines

Normally legacy and transitional mode devices need an explicit PCI ID reserved, where as modern-only devices have a PCI ID derived from their VirtIO ID + a fixed offset.

Was this mistakenly using a VirtIO ID + fixed offset for the legacy mode too ?

vhost-user-fs-pci does not support Legacy or Transitional mode. See hw/virtio/vhost-user-fs-pci.c:

  static const VirtioPCIDeviceTypeInfo vhost_user_fs_pci_info = {
      .base_name             = TYPE_VHOST_USER_FS_PCI,
      .non_transitional_name = "vhost-user-fs-pci",
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      .instance_size = sizeof(VHostUserFSPCI),
      .instance_init = vhost_user_fs_pci_instance_init,
      .class_init    = vhost_user_fs_pci_class_init,
  };

Stefan

#19 lore raw
Quoted 41 lines · On Tue, 30 Jun 2020 10:04:51 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 30 lines · Daniel P. Berrangé
Quoted 17 lines · Stefan Hajnoczi
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 6 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

Quoted 8 lines

I suggest the following:

1. Combinations that worked with libvhost-user in the past must not break.

2. New combinations should only support VIRTIO 1.0 and later.

This means continue to allow Legacy mode devices where they already run today but don't add new code for the cases that didn't work.

What I'm missing here is what PCI product ID was being used when the current impl is in legacy/transitional mode ?

Quoted 6 lines

Normally legacy and transitional mode devices need an explicit PCI ID reserved, where as modern-only devices have a PCI ID derived from their VirtIO ID + a fixed offset.

Was this mistakenly using a VirtIO ID + fixed offset for the legacy mode too ?

vhost-user-fs-pci does not support Legacy or Transitional mode. See hw/virtio/vhost-user-fs-pci.c:

Quoted 8 lines
  static const VirtioPCIDeviceTypeInfo vhost_user_fs_pci_info = {
      .base_name             = TYPE_VHOST_USER_FS_PCI,
      .non_transitional_name = "vhost-user-fs-pci",
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      .instance_size = sizeof(VHostUserFSPCI),
      .instance_init = vhost_user_fs_pci_instance_init,
      .class_init    = vhost_user_fs_pci_class_init,
  };

This makes it very unlikely that someone accidentally configures non-modern, but does not prevent it AFAICS. See <[email protected]>, which I just sent.

(I may be off, because that is all very confusing...)

#20 lore raw
Quoted 53 lines · Cornelia Huck
Quoted 41 lines · On Tue, 30 Jun 2020 10:04:51 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 30 lines · Daniel P. Berrangé
Quoted 17 lines · Stefan Hajnoczi
Quoted 9 lines · Marc Hartmayer

This RFC is about enabling virtio-fs on s390x. For that we need
+ some shim code (first patch), and we need
+ libvhost-user to deal with virtio endiannes as mandated by the spec.

The second part is trickier, because unlike QEMU we are not certain about the guest's native endianness, which is needed to handle the legacy-interface appropriately. In fact, this is the reason why just RFC.

Quoted 6 lines

One of the open questions is whether to build separate versions, one for guest little endian and one for guest big endian, or do we want something like a command line option? (Digression on the libvirt modeling)

A third option would be to refuse legacy altogether.

Quoted 8 lines

I suggest the following:

1. Combinations that worked with libvhost-user in the past must not break.

2. New combinations should only support VIRTIO 1.0 and later.

This means continue to allow Legacy mode devices where they already run today but don't add new code for the cases that didn't work.

What I'm missing here is what PCI product ID was being used when the current impl is in legacy/transitional mode ?

Quoted 6 lines

Normally legacy and transitional mode devices need an explicit PCI ID reserved, where as modern-only devices have a PCI ID derived from their VirtIO ID + a fixed offset.

Was this mistakenly using a VirtIO ID + fixed offset for the legacy mode too ?

vhost-user-fs-pci does not support Legacy or Transitional mode. See hw/virtio/vhost-user-fs-pci.c:

Quoted 8 lines
  static const VirtioPCIDeviceTypeInfo vhost_user_fs_pci_info = {
      .base_name             = TYPE_VHOST_USER_FS_PCI,
      .non_transitional_name = "vhost-user-fs-pci",
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      .instance_size = sizeof(VHostUserFSPCI),
      .instance_init = vhost_user_fs_pci_instance_init,
      .class_init    = vhost_user_fs_pci_class_init,
  };
Quoted 5 lines

This makes it very unlikely that someone accidentally configures non-modern, but does not prevent it AFAICS. See <[email protected]>, which I just sent.

(I may be off, because that is all very confusing...)

Right. We'll block legacy for modern only devices going forward. Going back to the patchset in question, virtio-fs is modern only, legacy will not be supported.

Signature · 1 line
MST