Skip to contentSkip to conversation

kvm-s390: userspace access to guest storage keys

PATCH In review Carsten Otte 4 participants 6 messages May 6, 2011 – May 10, 2011

Subsystems (3)
KVM s390 KVM s390
lore
Share & tools
lore

Carsten Otte

author
#0 lore raw

PATCH 1/1 kvm-s390: userspace access to guest storage keys

From: Carsten Otte <[email protected]>

This patch gives userspace access to the guest visible storage keys. Three operations are supported:
KVM_S390_KEYOP_SSKE for setting storage keys, similar to the set storage key extended (SSKE) instruction.
KVM_S390_KEYOP_ISKE for reading storage key content, similar to the insert storage key extended (ISKE) instruction.
KVM_s390_KEYOP_RRBE for reading and resetting the page reference bit, similar to the reset reference bit extended (RRBE) instruction.
Note that all functions take userspace addresses as input, which typically differ from guest addresses.

This work was requested by Alex Graf for guest live migration: Different from x86 the guest's view of dirty and reference information is not stored in the page table entrys that are part of the guest address space but are stored in the storage key instead. Thus, the storage key needs to be read, transfered, and written back on the migration target side.

Signed-off-by Carsten Otte <[email protected]>
---
 arch/s390/include/asm/kvm_host.h |    4 +
 arch/s390/kvm/kvm-s390.c         |  149 ++++++++++++++++++++++++++++++++++++++-
 include/linux/kvm.h              |    7 +
 3 files changed, 157 insertions(+), 3 deletions(-)
arch/s390/include/asm/kvm_host.h +4 -0
Index: linux-2.6/arch/s390/include/asm/kvm_host.h
===================================================================
@@ -47,6 +47,10 @@ struct sca_block {
 #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
 #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
 
+#define KVM_S390_KEYOP_SSKE 0x01
+#define KVM_S390_KEYOP_ISKE 0x02
+#define KVM_S390_KEYOP_RRBE 0x03
+
 #define CPUSTAT_HOST       0x80000000
 #define CPUSTAT_WAIT       0x10000000
 #define CPUSTAT_ECALL_PEND 0x08000000
arch/s390/kvm/kvm-s390.c +146 -3

159 lines not on this page · raw message

include/linux/kvm.h +7 -0
Index: linux-2.6/include/linux/kvm.h
===================================================================
@@ -432,6 +432,13 @@ struct kvm_ppc_pvinfo {
 #define KVM_GET_MSR_INDEX_LIST    _IOWR(KVMIO, 0x02, struct kvm_msr_list)
 
 #define KVM_S390_ENABLE_SIE       _IO(KVMIO,   0x06)
+
+struct kvm_s390_keyop {
+	__u64 user_addr;
+	__u8  key;
+	__u8  operation;
+};
+#define KVM_S390_KEYOP            _IOWR(KVMIO,   0x09, struct kvm_s390_keyop)
 /*
  * Check if a kvm extension is available.  Argument is extension number,
  * return is 1 (yes) or 0 (no, sorry).
#1 lore raw
Quoted 13 lines · Carsten Otte

From: Carsten Otte<[email protected]>

This patch gives userspace access to the guest visible storage keys. Three operations are supported:
KVM_S390_KEYOP_SSKE for setting storage keys, similar to the set storage key extended (SSKE) instruction.
KVM_S390_KEYOP_ISKE for reading storage key content, similar to the insert storage key extended (ISKE) instruction.
KVM_s390_KEYOP_RRBE for reading and resetting the page reference bit, similar to the reset reference bit extended (RRBE) instruction.
Note that all functions take userspace addresses as input, which typically differ from guest addresses.

Quoted 6 lines

This work was requested by Alex Graf for guest live migration: Different from x86 the guest's view of dirty and reference information is not stored in the page table entrys that are part of the guest address space but are stored in the storage key instead. Thus, the storage key needs to be read, transfered, and written back on the migration target side.

And not in main memory, either?

Quoted 5 lines
Signed-off-by Carsten Otte<[email protected]>
---
  arch/s390/include/asm/kvm_host.h |    4 +
  arch/s390/kvm/kvm-s390.c         |  149 ++++++++++++++++++++++++++++++++++++++-
  include/linux/kvm.h              |    7 +

Documentation/kvm/api.txt ++++++++++++

arch/s390/include/asm/kvm_host.h struct sca_block { · line 52

Quoted 13 lines
  3 files changed, 157 insertions(+), 3 deletions(-)

Index: linux-2.6/arch/s390/include/asm/kvm_host.h
===================================================================

arch/s390/include/asm/kvm_host.h +3 -0
@@ -47,6 +47,10 @@ struct sca_block {
  #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
  #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
 
+#define KVM_S390_KEYOP_SSKE 0x01
+#define KVM_S390_KEYOP_ISKE 0x02
+#define KVM_S390_KEYOP_RRBE 0x03

kvm_host.h is not exported to userspace. Use asm/kvm.h instead.

<snip black magic>

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#2 lore raw
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Carsten Otte

From: Carsten Otte<[email protected]>

This patch gives userspace access to the guest visible storage keys. Three operations are supported:
KVM_S390_KEYOP_SSKE for setting storage keys, similar to the set storage key extended (SSKE) instruction.
KVM_S390_KEYOP_ISKE for reading storage key content, similar to the insert storage key extended (ISKE) instruction.
KVM_s390_KEYOP_RRBE for reading and resetting the page reference bit, similar to the reset reference bit extended (RRBE) instruction.
Note that all functions take userspace addresses as input, which typically differ from guest addresses.

Quoted 8 lines
Quoted 6 lines

This work was requested by Alex Graf for guest live migration: Different from x86 the guest's view of dirty and reference information is not stored in the page table entrys that are part of the guest address space but are stored in the storage key instead. Thus, the storage key needs to be read, transfered, and written back on the migration target side.

And not in main memory, either?

Nope - storage keys are only accessible using special instructions. They're not in RAM (visible to a guest) :).

Alex

#3 lore raw
Quoted 5 lines · Alexander Graf
Quoted 2 lines

And not in main memory, either?

Nope - storage keys are only accessible using special instructions. They're not in RAM (visible to a guest) :).

Interesting, so where are they kept? An on-chip memory? That would limit the amount of main memory to that indexed by the chip. Extra off-chip memory?

Asking purely out of interest, this has no bearing on the patch.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#4 lore raw
Quoted 8 lines · Avi Kivity
Quoted 5 lines · Alexander Graf
Quoted 2 lines

And not in main memory, either?

Nope - storage keys are only accessible using special instructions. They're not in RAM (visible to a guest) :).

Interesting, so where are they kept? An on-chip memory? That would limit the amount of main memory to that indexed by the chip. Extra off-chip memory?

I'll leave the answer of this question to the real experts :). All I know is that the ISA defines special instructions to fetch them, so the question on where they are stored is "implementation dependent". In qemu for example, they are kept in an extra array that's just malloc'ed.

Alex

#5 lore raw
Quoted 10 lines · Alexander Graf
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Alexander Graf
Quoted 2 lines

And not in main memory, either?

Nope - storage keys are only accessible using special instructions. They're not in RAM (visible to a guest) :).

Quoted 6 lines
Quoted 2 lines

Interesting, so where are they kept? An on-chip memory? That would limit the amount of main memory to that indexed by the chip. Extra off-chip memory?

I'll leave the answer of this question to the real experts :). All I know is that the ISA defines special instructions to fetch them, so the question on where they are stored is "implementation dependent". In qemu for example, they are kept in an extra array that's just malloc'ed.

Dont know really - the HW folks probably know it better, so I am just guessing. The final location is probably somewhere in the HSA (hardware system area) which is backed by real memory and hidden by the cpu microcode. (there is no way to access this memory directly). But parts of the storage keys are also mirrored by the different cache levels and TLB (e.g. write protection via page tables).

Christian