Skip to contentSkip to conversation

KVM: ioctl for reading/writing guest memory

RFC PATCH In review Thomas Huth 4 participants 15 messages Feb 3, 2015 – Feb 5, 2015

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover KVM: ioctl for reading/writing guest memory

tl;dr:
This patch adds a new ioctl to KVM on s390x for reading and writing from/to virtual guest memory, to take account of the so-called IPTE-lock on s390x (a locking mechanism for the host to walk MMU tables of the guest).

Long story:
Certain instruction interception handlers in QEMU have to access the memory of the guest, either to retrieve additional paramaters/data or to supply results to the guest. On s390x, some of them (e.g. MSCH, SSCH, STSCH, ...) are specified to use logical (i.e. virtual) addresses in memory, i.e. the addresses are subject to MMU translation. The current handlers in target-s390x/ioinst.c just work "by accident" since the Linux kernel on s390x uses a 1:1 MMU mapping for kernel memory, but for correct behaviour we have to do a MMU page table walk in these handlers first.

Now on s390x, there's another specialty for the case the host has to walk the MMU tables of the guest: While doing the page table walk (or while accessing the memory of the guest in bigger, non-atomic chunks on multiple pages), there is a small chance that another CPU might zap or change the MMU mappings inbetween, so in that case an unexpected/undefined behaviour might occur. To avoid such problems, the SIE facility features a locking mechanism, the so called IPTE-lock, which prevents other virtual CPUs from issuing the IPTE (invalidate page table entry) or similar instructions. When the lock is being held, these other instructions are intercepted, so that the execution of the instructions can be delayed until the page table walk / memory operation finished on the locking CPU.

The kernel part of KVM on s390x already uses this locking mechanism for the interception handlers in the kernel (e.g. during the read_guest() and write_guest() functions). For proper MMU page table walk support in QEMU, the IPTE-lock has now somehow to be provided to the userspace, too.

However, providing this lock directly to the userspace would be quite ugly, since we then need to deal with a lot of cumbersome conditions (how should the kernel behave if userspace takes the lock for too long or forgets to free it again etc.). Additionally, there is also another specialty of s390x pending - proper handling of the so-called storage keys when accessing the guest memory - which is also done best in the kernel space instead of user space (I can elaborate more on that topic on request). So I decided to introduce a simple ioctl for reading and writing from/to guest memory instead of exporting the lock itself to userspace.

The userspace (QEMU) then can simply call this ioctl when it wants to read or write from/to virtual guest memory. Then kernel then takes the IPTE-lock, walks the MMU table of the guest to find out the physical address that corresponds to the virtual address, copies the requested amount of bytes from the userspace buffer to guest memory or the other way round, and finally frees the IPTE-lock again.

Does that sound like a viable solution (IMHO it does ;-))? Or should I maybe try to pursue another approach?

Thomas Huth (1):
KVM: s390: Add MEMOP ioctls for reading/writing guest memory

 Documentation/virtual/kvm/api.txt |   44 +++++++++++++++++++++++++
 arch/s390/kvm/gaccess.c           |   22 +++++++++++++
 arch/s390/kvm/gaccess.h           |    2 +
 arch/s390/kvm/kvm-s390.c          |   63 +++++++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h          |   21 ++++++++++++
 5 files changed, 152 insertions(+), 0 deletions(-)
#1 lore raw

RFC PATCH 1/1 KVM: s390: Add MEMOP ioctl for reading/writing guest memory

On s390, we've got to make sure to hold the IPTE lock while accessing virtual memory. So let's add an ioctl for reading and writing virtual memory to provide this feature for userspace, too.

---
 Documentation/virtual/kvm/api.txt |   44 +++++++++++++++++++++++++
 arch/s390/kvm/gaccess.c           |   22 +++++++++++++
 arch/s390/kvm/gaccess.h           |    2 +
 arch/s390/kvm/kvm-s390.c          |   63 +++++++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h          |   21 ++++++++++++
 5 files changed, 152 insertions(+), 0 deletions(-)
Documentation/virtual/kvm/api.txt +44 -0
index b112efc..bf44b53 100644
@@ -2716,6 +2716,50 @@ The fields in each entry are defined as follows:
    eax, ebx, ecx, edx: the values returned by the cpuid instruction for
          this function/index combination
 
+4.89 KVM_GUEST_MEM_OP
+
+Capability: KVM_CAP_MEM_OP
+Architectures: s390
+Type: vcpu ioctl
+Parameters: struct kvm_guest_mem_op (in)
+Returns: = 0 on success,
+         < 0 on generic error (e.g. -EFAULT or -ENOMEM),
+         > 0 if an exception occurred while walking the page tables
+
+Read or write data from/to the virtual memory of a VPCU.
+
+Parameters are specified via the following structure:
+
+struct kvm_guest_mem_op {
+	__u64 gaddr;		/* the guest address */
+	__u64 flags;		/* arch specific flags */
+	__u32 size;		/* amount of bytes */
+	__u32 op;		/* type of operation */
+	__u64 buf;		/* buffer in userspace */
+	__u8 reserved[32];	/* should be set to 0 */
+};
+
+The type of operation is specified in the "op" field, either KVM_MEMOP_VIRTREAD
+for reading from memory, KVM_MEMOP_VIRTWRITE for writing to memory, or
+KVM_MEMOP_CHECKVIRTREAD or KVM_MEMOP_CHECKVIRTWRITE to check whether the
+corresponding memory access would create an access exception (without
+changing the data in the memory at the destination). In case an access
+exception occurred while walking the MMU tables of the guest, the ioctl
+returns a positive error number to indicate the type of exception. The
+exception is raised directly at the corresponding VCPU if the bit
+KVM_MEMOP_F_INJECT_EXC is set in the "flags" field.
+
+The logical (virtual) start address of the memory region has to be specified
+in the "gaddr" field, and the length of the region in the "size" field.
+"buf" is the buffer supplied by the userspace application where the read data
+should be written to for KVM_MEMOP_VIRTREAD, or where the data that should
+be written is stored for a KVM_MEMOP_VIRTWRITE. "buf" can be NULL for both
+CHECK operations.
+
+The "reserved" field is meant for future extensions. It must currently be
+set to 0.
+
+
 5. The kvm_run structure
 ------------------------
 
arch/s390/kvm/gaccess.c +22 -0
index 8a1be90..d912362 100644
@@ -697,6 +697,28 @@ int guest_translate_address(struct kvm_vcpu *vcpu, unsigned long gva,
 }
 
 /**
+ * check_gva_range - test a range of guest virtual addresses for accessibility
+ */
+int check_gva_range(struct kvm_vcpu *vcpu, unsigned long gva,
+		    unsigned long length, int is_write)
+{
+	unsigned long gpa;
+	unsigned long currlen;
+	int rc = 0;
+
+	ipte_lock(vcpu);
+	while (length > 0 && !rc) {
+		currlen = min(length, PAGE_SIZE - (gva % PAGE_SIZE));
+		rc = guest_translate_address(vcpu, gva, &gpa, is_write);
+		gva += currlen;
+		length -= currlen;
+	}
+	ipte_unlock(vcpu);
+
+	return rc;
+}
+
+/**
  * kvm_s390_check_low_addr_protection - check for low-address protection
  * @ga: Guest address
  *
arch/s390/kvm/gaccess.h +2 -0
index 0149cf1..268beb7 100644
@@ -157,6 +157,8 @@ int read_guest_lc(struct kvm_vcpu *vcpu, unsigned long gra, void *data,
 
 int guest_translate_address(struct kvm_vcpu *vcpu, unsigned long gva,
 			    unsigned long *gpa, int write);
+int check_gva_range(struct kvm_vcpu *vcpu, unsigned long gva,
+		    unsigned long length, int is_write);
 
 int access_guest(struct kvm_vcpu *vcpu, unsigned long ga, void *data,
 		 unsigned long len, int write);
arch/s390/kvm/kvm-s390.c +63 -0

81 lines not on this page · raw message

include/uapi/linux/kvm.h +21 -0
index 8055706..528a3d4 100644
@@ -365,6 +365,24 @@ struct kvm_translation {
 	__u8  pad[5];
 };
 
+/* for KVM_GUEST_MEM_OP */
+struct kvm_guest_mem_op {
+	/* in */
+	__u64 gaddr;		/* the guest address */
+	__u64 flags;		/* arch specific flags */
+	__u32 size;		/* amount of bytes */
+	__u32 op;		/* type of operation */
+	__u64 buf;		/* buffer in userspace */
+	__u8 reserved[32];	/* should be set to 0 */
+};
+/* types for kvm_guest_mem_op->op */
+#define KVM_MEMOP_VIRTREAD		0
+#define KVM_MEMOP_VIRTWRITE		1
+#define KVM_MEMOP_CHECKVIRTREAD		2
+#define KVM_MEMOP_CHECKVIRTWRITE	3
+/* flags for kvm_guest_mem_op->flags */
+#define KVM_MEMOP_F_INJECT_EXC	0x0001UL	/* Inject exception on faults */
+
 /* for KVM_INTERRUPT */
 struct kvm_interrupt {
 	/* in */
@@ -760,6 +778,7 @@ struct kvm_ppc_smmu_info {
 #define KVM_CAP_PPC_ENABLE_HCALL 104
 #define KVM_CAP_CHECK_EXTENSION_VM 105
 #define KVM_CAP_S390_USER_SIGP 106
+#define KVM_CAP_MEM_OP 107
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1135,6 +1154,8 @@ struct kvm_s390_ucas_mapping {
 #define KVM_ARM_VCPU_INIT	  _IOW(KVMIO,  0xae, struct kvm_vcpu_init)
 #define KVM_ARM_PREFERRED_TARGET  _IOR(KVMIO,  0xaf, struct kvm_vcpu_init)
 #define KVM_GET_REG_LIST	  _IOWR(KVMIO, 0xb0, struct kvm_reg_list)
+/* Available with KVM_CAP_MEM_OP */
+#define KVM_GUEST_MEM_OP	  _IOW(KVMIO,  0xb1, struct kvm_guest_mem_op)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
Signature · 1 line
1.7.1
maintainer
#2 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Quoted 20 lines · Thomas Huth

On s390, we've got to make sure to hold the IPTE lock while accessing virtual memory. So let's add an ioctl for reading and writing virtual memory to provide this feature for userspace, too.

---
 Documentation/virtual/kvm/api.txt |   44 +++++++++++++++++++++++++
 arch/s390/kvm/gaccess.c           |   22 +++++++++++++
 arch/s390/kvm/gaccess.h           |    2 +
 arch/s390/kvm/kvm-s390.c          |   63 +++++++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h          |   21 ++++++++++++
 5 files changed, 152 insertions(+), 0 deletions(-)
Documentation/virtual/kvm/api.txt +0 -0
index b112efc..bf44b53 100644

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2721

Quoted 6 lines
Documentation/virtual/kvm/api.txt +3 -0
@@ -2716,6 +2716,50 @@ The fields in each entry are defined as follows:
    eax, ebx, ecx, edx: the values returned by the cpuid instruction for
          this function/index combination
 
+4.89 KVM_GUEST_MEM_OP
+
+Capability: KVM_CAP_MEM_OP

Put "virtual" somewhere in the ioctl name and capability?

Quoted 13 lines
Documentation/virtual/kvm/api.txt +13 -0
+Architectures: s390
+Type: vcpu ioctl
+Parameters: struct kvm_guest_mem_op (in)
+Returns: = 0 on success,
+         < 0 on generic error (e.g. -EFAULT or -ENOMEM),
+         > 0 if an exception occurred while walking the page tables
+
+Read or write data from/to the virtual memory of a VPCU.
+
+Parameters are specified via the following structure:
+
+struct kvm_guest_mem_op {
+	__u64 gaddr;		/* the guest address */

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2744

Quoted 10 lines
Documentation/virtual/kvm/api.txt +10 -0
+	__u64 flags;		/* arch specific flags */
+	__u32 size;		/* amount of bytes */
+	__u32 op;		/* type of operation */
+	__u64 buf;		/* buffer in userspace */
+	__u8 reserved[32];	/* should be set to 0 */
+};
+
+The type of operation is specified in the "op" field, either KVM_MEMOP_VIRTREAD
+for reading from memory, KVM_MEMOP_VIRTWRITE for writing to memory, or
+KVM_MEMOP_CHECKVIRTREAD or KVM_MEMOP_CHECKVIRTWRITE to check whether the

Better:

#define KVM_MEMOP_READ 0
#define KVM_MEMOP_WRITE 1

and in the flags field:

#define KVM_MEMOP_F_CHECK_ONLY (1 << 1)

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2750

Quoted 6 lines
Documentation/virtual/kvm/api.txt +6 -0
+corresponding memory access would create an access exception (without
+changing the data in the memory at the destination). In case an access
+exception occurred while walking the MMU tables of the guest, the ioctl
+returns a positive error number to indicate the type of exception. The
+exception is raised directly at the corresponding VCPU if the bit
+KVM_MEMOP_F_INJECT_EXC is set in the "flags" field.

KVM_MEMOP_F_INJECT_EXCEPTION.

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2757

Quoted 6 lines
Documentation/virtual/kvm/api.txt +6 -0
+The logical (virtual) start address of the memory region has to be specified
+in the "gaddr" field, and the length of the region in the "size" field.
+"buf" is the buffer supplied by the userspace application where the read data
+should be written to for KVM_MEMOP_VIRTREAD, or where the data that should
+be written is stored for a KVM_MEMOP_VIRTWRITE. "buf" can be NULL for both
+CHECK operations.

"buf" is unused and can be NULL for both CHECK operations.

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2760

Quoted 2 lines
Documentation/virtual/kvm/api.txt +2 -0
+The "reserved" field is meant for future extensions. It must currently be
+set to 0.

Not really true, as you don't check it. So "It is not used by KVM with the currently defined set of flags" is a better explanation.

Paolo

Quoted 185 lines
Documentation/virtual/kvm/api.txt +1 -0
+
 5. The kvm_run structure
 ------------------------
 
arch/s390/kvm/gaccess.c +22 -0
index 8a1be90..d912362 100644
@@ -697,6 +697,28 @@ int guest_translate_address(struct kvm_vcpu *vcpu, unsigned long gva,
 }
 
 /**
+ * check_gva_range - test a range of guest virtual addresses for accessibility
+ */
+int check_gva_range(struct kvm_vcpu *vcpu, unsigned long gva,
+		    unsigned long length, int is_write)
+{
+	unsigned long gpa;
+	unsigned long currlen;
+	int rc = 0;
+
+	ipte_lock(vcpu);
+	while (length > 0 && !rc) {
+		currlen = min(length, PAGE_SIZE - (gva % PAGE_SIZE));
+		rc = guest_translate_address(vcpu, gva, &gpa, is_write);
+		gva += currlen;
+		length -= currlen;
+	}
+	ipte_unlock(vcpu);
+
+	return rc;
+}
+
+/**
  * kvm_s390_check_low_addr_protection - check for low-address protection
  * @ga: Guest address
  *
arch/s390/kvm/gaccess.h +2 -0
index 0149cf1..268beb7 100644
@@ -157,6 +157,8 @@ int read_guest_lc(struct kvm_vcpu *vcpu, unsigned long gra, void *data,
 
 int guest_translate_address(struct kvm_vcpu *vcpu, unsigned long gva,
 			    unsigned long *gpa, int write);
+int check_gva_range(struct kvm_vcpu *vcpu, unsigned long gva,
+		    unsigned long length, int is_write);
 
 int access_guest(struct kvm_vcpu *vcpu, unsigned long ga, void *data,
 		 unsigned long len, int write);
arch/s390/kvm/kvm-s390.c +63 -0
index b2371c0..c80a640 100644
@@ -168,6 +168,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_VM_ATTRIBUTES:
 	case KVM_CAP_MP_STATE:
 	case KVM_CAP_S390_USER_SIGP:
+	case KVM_CAP_MEM_OP:
 		r = 1;
 		break;
 	case KVM_CAP_NR_VCPUS:
@@ -1886,6 +1887,59 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 	return r;
 }
 
+static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
+				  struct kvm_guest_mem_op *mop)
+{
+	void __user *uaddr = (void __user *)mop->buf;
+	void *tmpbuf = NULL;
+	int r, srcu_idx;
+
+	if ((mop->flags & ~KVM_MEMOP_F_INJECT_EXC) != 0)
+		return -EINVAL;
+
+	if (mop->op == KVM_MEMOP_VIRTREAD || mop->op == KVM_MEMOP_VIRTWRITE) {
+		if (mop->size > 16 * PAGE_SIZE)
+			return -E2BIG;
+		tmpbuf = vmalloc(mop->size);
+		if (!tmpbuf)
+			return -ENOMEM;
+	}
+
+	srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
+
+	switch (mop->op) {
+	case KVM_MEMOP_VIRTREAD:
+		r = read_guest(vcpu, mop->gaddr, tmpbuf, mop->size);
+		if (r == 0) {
+			if (copy_to_user(uaddr, tmpbuf, mop->size))
+				r = -EFAULT;
+		}
+		break;
+	case KVM_MEMOP_VIRTWRITE:
+		if (copy_from_user(tmpbuf, uaddr, mop->size)) {
+			r = -EFAULT;
+			break;
+		}
+		r = write_guest(vcpu, mop->gaddr, tmpbuf, mop->size);
+		break;
+	case KVM_MEMOP_CHECKVIRTREAD:
+	case KVM_MEMOP_CHECKVIRTWRITE:
+		r = check_gva_range(vcpu, mop->gaddr, mop->size,
+				    mop->op == KVM_MEMOP_CHECKVIRTWRITE);
+		break;
+	default:
+		r = -EINVAL;
+	}
+
+	srcu_read_unlock(&vcpu->kvm->srcu, srcu_idx);
+
+	if (r > 0 && (mop->flags & KVM_MEMOP_F_INJECT_EXC) != 0)
+		kvm_s390_inject_prog_irq(vcpu, &vcpu->arch.pgm);
+
+	vfree(tmpbuf);
+	return r;
+}
+
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
@@ -1985,6 +2039,15 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		r = kvm_vcpu_ioctl_enable_cap(vcpu, &cap);
 		break;
 	}
+	case KVM_GUEST_MEM_OP: {
+		struct kvm_guest_mem_op mem_op;
+
+		if (copy_from_user(&mem_op, argp, sizeof(mem_op)) == 0)
+			r = kvm_s390_guest_mem_op(vcpu, &mem_op);
+		else
+			r = -EFAULT;
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
include/uapi/linux/kvm.h +21 -0
index 8055706..528a3d4 100644
@@ -365,6 +365,24 @@ struct kvm_translation {
 	__u8  pad[5];
 };
 
+/* for KVM_GUEST_MEM_OP */
+struct kvm_guest_mem_op {
+	/* in */
+	__u64 gaddr;		/* the guest address */
+	__u64 flags;		/* arch specific flags */
+	__u32 size;		/* amount of bytes */
+	__u32 op;		/* type of operation */
+	__u64 buf;		/* buffer in userspace */
+	__u8 reserved[32];	/* should be set to 0 */
+};
+/* types for kvm_guest_mem_op->op */
+#define KVM_MEMOP_VIRTREAD		0
+#define KVM_MEMOP_VIRTWRITE		1
+#define KVM_MEMOP_CHECKVIRTREAD		2
+#define KVM_MEMOP_CHECKVIRTWRITE	3
+/* flags for kvm_guest_mem_op->flags */
+#define KVM_MEMOP_F_INJECT_EXC	0x0001UL	/* Inject exception on faults */
+
 /* for KVM_INTERRUPT */
 struct kvm_interrupt {
 	/* in */
@@ -760,6 +778,7 @@ struct kvm_ppc_smmu_info {
 #define KVM_CAP_PPC_ENABLE_HCALL 104
 #define KVM_CAP_CHECK_EXTENSION_VM 105
 #define KVM_CAP_S390_USER_SIGP 106
+#define KVM_CAP_MEM_OP 107
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1135,6 +1154,8 @@ struct kvm_s390_ucas_mapping {
 #define KVM_ARM_VCPU_INIT	  _IOW(KVMIO,  0xae, struct kvm_vcpu_init)
 #define KVM_ARM_PREFERRED_TARGET  _IOR(KVMIO,  0xaf, struct kvm_vcpu_init)
 #define KVM_GET_REG_LIST	  _IOWR(KVMIO, 0xb0, struct kvm_reg_list)
+/* Available with KVM_CAP_MEM_OP */
+#define KVM_GUEST_MEM_OP	  _IOW(KVMIO,  0xb1, struct kvm_guest_mem_op)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
#3 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Quoted 24 lines · On Tue, 03 Feb 2015 14:04:43 +0100 Paolo Bonzini <[email protected]> wrote:
Quoted 23 lines · Thomas Huth

On s390, we've got to make sure to hold the IPTE lock while accessing virtual memory. So let's add an ioctl for reading and writing virtual memory to provide this feature for userspace, too.

---
 Documentation/virtual/kvm/api.txt |   44 +++++++++++++++++++++++++
 arch/s390/kvm/gaccess.c           |   22 +++++++++++++
 arch/s390/kvm/gaccess.h           |    2 +
 arch/s390/kvm/kvm-s390.c          |   63 +++++++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h          |   21 ++++++++++++
 5 files changed, 152 insertions(+), 0 deletions(-)
Documentation/virtual/kvm/api.txt +0 -0
index b112efc..bf44b53 100644
@@ -2716,6 +2716,50 @@ The fields in each entry are defined as follows:
    eax, ebx, ecx, edx: the values returned by the cpuid instruction for
          this function/index combination
 
Quoted 5 lines

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2721

Quoted 3 lines
Documentation/virtual/kvm/api.txt +3 -0
+4.89 KVM_GUEST_MEM_OP
+
+Capability: KVM_CAP_MEM_OP

Put "virtual" somewhere in the ioctl name and capability?

Actually, I'd prefer to keep the "virtual" in the defines for the type of operation below: When it comes to s390 storage keys, we likely might need some calls for reading and writing to physical memory, too. Then we could simply extend this ioctl instead of inventing a new one.

Quoted 26 lines
Quoted 13 lines
Documentation/virtual/kvm/api.txt +13 -0
+Architectures: s390
+Type: vcpu ioctl
+Parameters: struct kvm_guest_mem_op (in)
+Returns: = 0 on success,
+         < 0 on generic error (e.g. -EFAULT or -ENOMEM),
+         > 0 if an exception occurred while walking the page tables
+
+Read or write data from/to the virtual memory of a VPCU.
+
+Parameters are specified via the following structure:
+
+struct kvm_guest_mem_op {
+	__u64 gaddr;		/* the guest address */

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2744

Quoted 10 lines
Documentation/virtual/kvm/api.txt +10 -0
+	__u64 flags;		/* arch specific flags */
+	__u32 size;		/* amount of bytes */
+	__u32 op;		/* type of operation */
+	__u64 buf;		/* buffer in userspace */
+	__u8 reserved[32];	/* should be set to 0 */
+};
+
+The type of operation is specified in the "op" field, either KVM_MEMOP_VIRTREAD
+for reading from memory, KVM_MEMOP_VIRTWRITE for writing to memory, or
+KVM_MEMOP_CHECKVIRTREAD or KVM_MEMOP_CHECKVIRTWRITE to check whether the

Better:

Quoted 6 lines

#define KVM_MEMOP_READ 0
#define KVM_MEMOP_WRITE 1

and in the flags field:

#define KVM_MEMOP_F_CHECK_ONLY (1 << 1)

Ok, a flag for the check operations is fine for me, too.

...

Quoted 9 lines

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2757

Quoted 6 lines
Documentation/virtual/kvm/api.txt +6 -0
+The logical (virtual) start address of the memory region has to be specified
+in the "gaddr" field, and the length of the region in the "size" field.
+"buf" is the buffer supplied by the userspace application where the read data
+should be written to for KVM_MEMOP_VIRTREAD, or where the data that should
+be written is stored for a KVM_MEMOP_VIRTWRITE. "buf" can be NULL for both
+CHECK operations.

"buf" is unused and can be NULL for both CHECK operations.

Quoted 5 lines

Documentation/virtual/kvm/api.txt The fields in each entry are defined as follows: · line 2760

Quoted 2 lines
Documentation/virtual/kvm/api.txt +2 -0
+The "reserved" field is meant for future extensions. It must currently be
+set to 0.

Not really true, as you don't check it. So "It is not used by KVM with the currently defined set of flags" is a better explanation.

ok ... and maybe add "should be set to zero" ?

Quoted 1 line

Paolo

Thanks for the review!

Thomas

maintainer
#4 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Quoted 4 lines · Thomas Huth

Actually, I'd prefer to keep the "virtual" in the defines for the type of operation below: When it comes to s390 storage keys, we likely might need some calls for reading and writing to physical memory, too. Then we could simply extend this ioctl instead of inventing a new one.

Can you explain why it is necessary to read/write physical addresses from user space? In the case of QEMU, I'm worried that you would have to invent your own memory read/write APIs that are different from everything else.

On real s390 zPCI, does bus-master DMA update storage keys?

Quoted 4 lines
Quoted 2 lines

Not really true, as you don't check it. So "It is not used by KVM with the currently defined set of flags" is a better explanation.

ok ... and maybe add "should be set to zero" ?

If you don't check it, it is misleading to document this.

Paolo

#5 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Quoted 8 lines · On Tue, 03 Feb 2015 16:22:32 +0100 Paolo Bonzini <[email protected]> wrote:
Quoted 4 lines · Thomas Huth

Actually, I'd prefer to keep the "virtual" in the defines for the type of operation below: When it comes to s390 storage keys, we likely might need some calls for reading and writing to physical memory, too. Then we could simply extend this ioctl instead of inventing a new one.

Quoted 6 lines

Can you explain why it is necessary to read/write physical addresses from user space? In the case of QEMU, I'm worried that you would have to invent your own memory read/write APIs that are different from everything else.

On real s390 zPCI, does bus-master DMA update storage keys?

Ah, I was not thinking about bus-mastering/DMA here: AFAIK there are some CPU instructions that access a parameter block in physical memory, for example the SCLP instruction (see hw/s390x/sclp.c) - it's already doing a cpu_physical_memory_read and ..._write for the parameters. However, I haven't checked yet whether it is also supposed to touch the storage keys, so if not, we also might be fine without the ioctls for reading/writing to physical memory.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Not really true, as you don't check it. So "It is not used by KVM with the currently defined set of flags" is a better explanation.

ok ... and maybe add "should be set to zero" ?

If you don't check it, it is misleading to document this.

True... so I'll omit that.

Thomas

#6 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

In reply to Paolo Bonzini

Am 03.02.2015 um 16:22 schrieb Paolo Bonzini:

Quoted 7 lines · Thomas Huth
Quoted 4 lines

Actually, I'd prefer to keep the "virtual" in the defines for the type of operation below: When it comes to s390 storage keys, we likely might need some calls for reading and writing to physical memory, too. Then we could simply extend this ioctl instead of inventing a new one.

Rereading that. Shall we replace "virtual" with "logical"? That is what is used architecturally when we mean "do whatever is appropriate right now" That can boil down to virtual via DAT, virtual via access register mode, real if DAT is off... and if fact your kernel implementation does that.

Quoted 7 lines

Can you explain why it is necessary to read/write physical addresses from user space? In the case of QEMU, I'm worried that you would have to invent your own memory read/write APIs that are different from everything else.

On real s390 zPCI, does bus-master DMA update storage keys?

the classic channel I/O does set the storage key change/reference and also triggers errors in the storage key protection value mismatches.

The PCI IOTA structure does contain a storage key value for accesses, so I assume its the same here, but I dont know for sure.

Conny:
I am asking myself, if we should explicitly add a comment in the virtio-ccw spec, that all accesses are assumed to be with key 0 and thus never cause key protection. The change/reference bit is set by the underlying I/O or memory copy anyway.
We can then add a ccw later on to set a different key if we ever need that.

Quoted 14 lines
Quoted 5 lines
Quoted 3 lines

Not really true, as you don't check it. So "It is not used by KVM with the currently defined set of flags" is a better explanation.

ok ... and maybe add "should be set to zero" ?

If you don't check it, it is misleading to document this.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#7 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Quoted 11 lines · Christian Borntraeger

Am 03.02.2015 um 16:22 schrieb Paolo Bonzini:

Quoted 5 lines · Thomas Huth
Quoted 4 lines

Actually, I'd prefer to keep the "virtual" in the defines for the type of operation below: When it comes to s390 storage keys, we likely might need some calls for reading and writing to physical memory, too. Then we could simply extend this ioctl instead of inventing a new one.

Rereading that. Shall we replace "virtual" with "logical"? That is what is used architecturally when we mean "do whatever is appropriate right now" That can boil down to virtual via DAT, virtual via access register mode, real if DAT is off... and if fact your kernel implementation does that.

That makes sense.

Quoted 5 lines
Quoted 5 lines

Can you explain why it is necessary to read/write physical addresses from user space? In the case of QEMU, I'm worried that you would have to invent your own memory read/write APIs that are different from everything else.

Quoted 7 lines
Quoted 1 line

On real s390 zPCI, does bus-master DMA update storage keys?

the classic channel I/O does set the storage key change/reference and also triggers errors in the storage key protection value mismatches.

The PCI IOTA structure does contain a storage key value for accesses, so I assume its the same here, but I dont know for sure.

Emulating that in QEMU would be very hard. Every DMA read/write would have to go through a bounce buffer, but QEMU block device models for example try hard to read from host disk directly into guest memory.

Quoted 5 lines

Conny:
I am asking myself, if we should explicitly add a comment in the virtio-ccw spec, that all accesses are assumed to be with key 0 and thus never cause key protection. The change/reference bit is set by the underlying I/O or memory copy anyway.

Can you explain the last sentence? :)

Paolo

Quoted 24 lines

We can then add a ccw later on to set a different key if we ever need that.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Not really true, as you don't check it. So "It is not used by KVM with the currently defined set of flags" is a better explanation.

ok ... and maybe add "should be set to zero" ?

Quoted 8 lines

If you don't check it, it is misleading to document this.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Am 04.02.2015 um 11:39 schrieb Paolo Bonzini:

Quoted 7 lines
Quoted 5 lines

Conny:
I am asking myself, if we should explicitly add a comment in the virtio-ccw spec, that all accesses are assumed to be with key 0 and thus never cause key protection. The change/reference bit is set by the underlying I/O or memory copy anyway.

Can you explain the last sentence? :)

Whenever vhost or qemu or a finished aio request wrote content into a virtio buffer, the HW has set the storage key for that physical page, which makes it automatically dirty/referenced in the guest visible storage key.

For completeness sake:
Now, if the guest does not use the storage key, but instead the new fault based software dirty tracking, it wont notice the change bit. The guest I/O itself when finished will mark the struct page as Dirty, just like on x86.

Makes sense?

Christian

maintainer
#9 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Quoted 4 lines · Christian Borntraeger

Whenever vhost or qemu or a finished aio request wrote content into a virtio buffer, the HW has set the storage key for that physical page, which makes it automatically dirty/referenced in the guest visible storage key.

Ah, I knew the storage keys were per-physical page, but I wasn't sure if they were separate for the host and the guest. That's obvious now.

Can we detect non-zero storage key in emulated zPCI requests, and fail the request somehow?

Paolo

#10 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

Am 04.02.2015 um 12:42 schrieb Paolo Bonzini:

Quoted 10 lines · Christian Borntraeger
Quoted 4 lines

Whenever vhost or qemu or a finished aio request wrote content into a virtio buffer, the HW has set the storage key for that physical page, which makes it automatically dirty/referenced in the guest visible storage key.

Ah, I knew the storage keys were per-physical page, but I wasn't sure if they were separate for the host and the guest. That's obvious now.

Just something on top:
the storage key is per physical page. Just once. It contains C/R/ACC/F (change, reference, access key, fetch protection)

But: there is also the pgste page table extension. That is used to separate both by doing logically ORs. The host and millicode will do the right shifting copying to keep both values separate, but when the physical storage key gets dirty, the host and the guest view is now "changed==yes"

Quoted 3 lines

Can we detect non-zero storage key in emulated zPCI requests, and fail the request somehow?

Not right now. Even the kernel KVM module does not do this for emulated instructions (as Linux has always key 0). Somewhen we might want to add that capability, but its obviously not trivial for I/O like things. It would get easier if we avoid VFIO etc and just had used the hardware support, though. but as far as I can see this is not an option in QEMU.

Christian

#11 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

In reply to Christian Borntraeger
Quoted 9 lines · On Wed, 04 Feb 2015 09:26:11 +0100 Christian Borntraeger…

Am 03.02.2015 um 16:22 schrieb Paolo Bonzini:

Quoted 7 lines · Thomas Huth
Quoted 4 lines

Actually, I'd prefer to keep the "virtual" in the defines for the type of operation below: When it comes to s390 storage keys, we likely might need some calls for reading and writing to physical memory, too. Then we could simply extend this ioctl instead of inventing a new one.

Quoted 4 lines

Rereading that. Shall we replace "virtual" with "logical"? That is what is used architecturally when we mean "do whatever is appropriate right now" That can boil down to virtual via DAT, virtual via access register mode, real if DAT is off... and if fact your kernel implementation does that.

True, but so far I tried to avoid to include s390-only wording into this ioctl in case other architectures might need such a functionality later, too (then this ioctl could be simply used there, too).

OTOH, maybe the memory model on s390 is just too special to try to keep this ioctl generic ... but then I guess I also should rename the define KVM_GUEST_MEM_OP into KVM_S390_GUEST_MEM_OP, for example?

Thomas

#12 lore raw

Re: [PATCH RFC 1/1] KVM: s390: Add MEMOP ioctl for reading/writing guest memory

In reply to Christian Borntraeger
Quoted 2 lines · On Wed, 04 Feb 2015 09:26:11 +0100 Christian Borntraeger…

the classic channel I/O does set the storage key change/reference and also triggers errors in the storage key protection value mismatches.

Just a bit of background for the benefit of innocent bystanders:

When classic channel I/O is initiated, the caller provides a so-called operation request block (orb) which references the actual channel program it wants to start. In this same orb, the caller also provides the storage key to be used for all memory fetches (either of the individual channel commands or of the memory they reference).

qemu interpretation of channel commands currently ignores all of this.

Quoted 7 lines

Conny:
I am asking myself, if we should explicitly add a comment in the virtio-ccw spec, that all accesses are assumed to be with key 0 and thus never cause key protection. The change/reference bit is set by the underlying I/O or memory copy anyway.
We can then add a ccw later on to set a different key if we ever need that.

I don't think we need to clarify anything for the normal channel commands used by virtio: They are treated as any other channel command wrt key protection; and the lack of key checking is a feature of qemu, not of the spec.

For the memory used for the virtqueues, I agree that we should clarify that we assume key 0. I'm not sure whether there's a case for extending this in the future, but who knows.

I'll probably have to open an issue against the virtio spec for this.

maintainer
#13 lore raw
In reply to Thomas Huth
Quoted 9 lines · Thomas Huth

The userspace (QEMU) then can simply call this ioctl when it wants to read or write from/to virtual guest memory. Then kernel then takes the IPTE-lock, walks the MMU table of the guest to find out the physical address that corresponds to the virtual address, copies the requested amount of bytes from the userspace buffer to guest memory or the other way round, and finally frees the IPTE-lock again.

Does that sound like a viable solution (IMHO it does ;-))? Or should I maybe try to pursue another approach?

It looks feasible to me as well.

Paolo

#14 lore raw

Am 03.02.2015 um 13:59 schrieb Paolo Bonzini:

Quoted 10 lines · Thomas Huth
Quoted 7 lines

The userspace (QEMU) then can simply call this ioctl when it wants to read or write from/to virtual guest memory. Then kernel then takes the IPTE-lock, walks the MMU table of the guest to find out the physical address that corresponds to the virtual address, copies the requested amount of bytes from the userspace buffer to guest memory or the other way round, and finally frees the IPTE-lock again.

Quoted 4 lines
Quoted 2 lines

Does that sound like a viable solution (IMHO it does ;-))? Or should I maybe try to pursue another approach?

It looks feasible to me as well.

Yes, we discussed this internally a lot and things are really tricky. The ipte lock could be exported to userspace, but we might also need to handle storage keys (and key protection) in an atomic fashion, so this really looks like the only safe way.
I guess we will give it some more testing, but to me it looks like a good candidate for kvm/next after 3.20-rc1.

Christian