Skip to contentSkip to conversation

Documentation/kvm : Add documentation on Hypercalls

PATCH In review Raghavendra K T 3 participants 8 messages Jul 24, 2012 – Aug 2, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Documentation/kvm : Add documentation on Hypercalls

From: Raghavendra K T <[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---
Please have a closer look at Hypercall ABI newly added

Changes since last post:
- Added hypercall ABI (Peter)
- made KVM_HC_VAPIC_POLL_IRQ active explicitly (Randy)

TODO: We need to add history details of each hypercall as suggested by HPA, which I could not trace easily. Hope it is easy for hypercall authors

 Documentation/virtual/hypercalls.txt |   71 ++++++++++++++++++++++++++++++++++
 1 files changed, 71 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/virtual/hypercalls.txt
Documentation/virtual/hypercalls.txt new file +71 -0
new file mode 100644
index 0000000..caffc08
@@ -0,0 +1,71 @@
+Hypercall ABI:
+=============
+A brief look at calling conventions of X86, S390 and PPC
+X86:
+ KVM Hypercalls have a three-byte sequence of either the vmrun or the vmmrun
+ instruction. The hypervisor can replace it with instructions that are
+ guaranteed to be supported.
+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax.  No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.
+
+S390:
+  R2-R7 are used for parameters 1-6. In addition, R1 is used for hypercall
+  number. The return value is written to R2.
+
+  S390 uses diagnose instruction as hypercall (0x500) along with hypercall
+  number in R1.
+
+ PoewerPC:
+  It uses R3-R10 and hypercall number in R11. R4-R11 are used as output registers.
+  Return value is placed in R3.
+
+  KVM hypercalls uses 4 byte opcode, that are patched with 'hypercall-instructions'
+  property inside the device tree's /hypervisor node.
+  For more information refer to Documentation/virtual/kvm/ppc-pv.txt
+
+KVM Hypercalls Documentation
+===========================
+The template for each hypercall is:
+1. Hypercall name, value.
+2. Architecture(s)
+3. Status (deprecated, obsolete, active)
+4. Purpose
+
+1. KVM_HC_VAPIC_POLL_IRQ
+------------------------
+Value: 1
+Architecture: x86
+Status: active
+Purpose: Trigger guest exit so that the host can check for pending
+interrupts on reentry.
+
+2. KVM_HC_MMU_OP
+------------------------
+Value: 2
+Architecture: x86
+Status: deprecated.
+Purpose: Support MMU operations such as writing to PTE,
+flushing TLB, release PT.
+
+3. KVM_HC_FEATURES
+------------------------
+Value: 3
+Architecture: PPC
+Status: active
+Purpose: Expose hypercall availability to the guest. On x86 platforms, cpuid
+used to enumerate which hypercalls are available. On PPC, either device tree
+based lookup ( which is also what EPAPR dictates) OR KVM specific enumeration
+mechanism (which is this hypercall) can be used.
+
+4. KVM_HC_PPC_MAP_MAGIC_PAGE
+------------------------
+Value: 4
+Architecture: PPC
+Status: active
+Purpose: To enable communication between the hypervisor and guest there is a
+shared page that contains parts of supervisor visible register state.
+The guest can map this shared page to access its supervisor register through
+memory using this hypercall.
#1 lore raw
Quoted 37 lines · Raghavendra K T

From: Raghavendra K T <[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---
Please have a closer look at Hypercall ABI newly added

Changes since last post:
- Added hypercall ABI (Peter)
- made KVM_HC_VAPIC_POLL_IRQ active explicitly (Randy)

TODO: We need to add history details of each hypercall as suggested by HPA, which I could not trace easily. Hope it is easy for hypercall authors

  Documentation/virtual/hypercalls.txt |   71 ++++++++++++++++++++++++++++++++++
  1 files changed, 71 insertions(+), 0 deletions(-)

create mode 100644 Documentation/virtual/hypercalls.txt

Documentation/virtual/hypercalls.txt new file +11 -0
new file mode 100644
index 0000000..caffc08
@@ -0,0 +1,71 @@
+Hypercall ABI:
+=============
+A brief look at calling conventions of X86, S390 and PPC
+X86:
+ KVM Hypercalls have a three-byte sequence of either the vmrun or the vmmrun
+ instruction. The hypervisor can replace it with instructions that are
+ guaranteed to be supported.
+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax.  No other registers will be clobbered unless explicitly stated

Documentation/virtual/hypercalls.txt line 21

Quoted 10 lines
Documentation/virtual/hypercalls.txt new file +10 -0
+ by the particular hypercall.
+
+S390:
+  R2-R7 are used for parameters 1-6. In addition, R1 is used for hypercall
+  number. The return value is written to R2.
+
+  S390 uses diagnose instruction as hypercall (0x500) along with hypercall
+  number in R1.
+
+ PoewerPC:

PowerPC

Documentation/virtual/hypercalls.txt line 27

Quoted 6 lines
Documentation/virtual/hypercalls.txt new file +6 -0
+  It uses R3-R10 and hypercall number in R11. R4-R11 are used as output registers.
+  Return value is placed in R3.
+
+  KVM hypercalls uses 4 byte opcode, that are patched with 'hypercall-instructions'
+  property inside the device tree's /hypervisor node.
+  For more information refer to Documentation/virtual/kvm/ppc-pv.txt

What exactly is this document supposed to cover? We have 3 different hypercall ABIs in KVM on PowerPC:

1) KVM hypercalls (ePAPR)

The ePAPR compliant hypercall implementation. This one is used for KVM specific hypercalls. All hypercalls get a KVM vendor prefix (42) and then the hypercall id. We also implement generic hypercalls here, like the ePAPR idle hcall. The instruction sequence for these is listed in the device tree, as you noted above. It's available on all targets.

2) PAPR hypercalls

To run server PowerPC PAPR guests (-M pseries in QEMU), we need to handle PAPR hypercalls. These are the same hypercalls that pHyp, the POWER hypervisor implements. Some of them are handled in the kernel, some are handled in user space. This is only available on book3s_64.

3) OSI hypercalls

In parallel to QEMU, there is another user of KVM on PowerPC: Mac-on-Linux. That project had its own hypercalls long before we came along with KVM, so to maintain compatibility we also support their hypercalls. All of these get forwarded to user space. This is only useful on book3s_32, but can be used with book3s_64 as well.

Documentation/virtual/hypercalls.txt line 39

Quoted 12 lines
Documentation/virtual/hypercalls.txt new file +12 -0
+
+KVM Hypercalls Documentation
+===========================
+The template for each hypercall is:
+1. Hypercall name, value.
+2. Architecture(s)
+3. Status (deprecated, obsolete, active)
+4. Purpose
+
+1. KVM_HC_VAPIC_POLL_IRQ
+------------------------
+Value: 1

What are the value fields supposed to mean?

Alex

Quoted 33 lines
Documentation/virtual/hypercalls.txt new file +32 -0
+Architecture: x86
+Status: active
+Purpose: Trigger guest exit so that the host can check for pending
+interrupts on reentry.
+
+2. KVM_HC_MMU_OP
+------------------------
+Value: 2
+Architecture: x86
+Status: deprecated.
+Purpose: Support MMU operations such as writing to PTE,
+flushing TLB, release PT.
+
+3. KVM_HC_FEATURES
+------------------------
+Value: 3
+Architecture: PPC
+Status: active
+Purpose: Expose hypercall availability to the guest. On x86 platforms, cpuid
+used to enumerate which hypercalls are available. On PPC, either device tree
+based lookup ( which is also what EPAPR dictates) OR KVM specific enumeration
+mechanism (which is this hypercall) can be used.
+
+4. KVM_HC_PPC_MAP_MAGIC_PAGE
+------------------------
+Value: 4
+Architecture: PPC
+Status: active
+Purpose: To enable communication between the hypervisor and guest there is a
+shared page that contains parts of supervisor visible register state.
+The guest can map this shared page to access its supervisor register through
+memory using this hypercall.
#2 lore raw
Quoted 51 lines · Alexander Graf
Quoted 50 lines · Raghavendra K T

From: Raghavendra K T <[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---
Please have a closer look at Hypercall ABI newly added

Changes since last post:
- Added hypercall ABI (Peter)
- made KVM_HC_VAPIC_POLL_IRQ active explicitly (Randy)

TODO: We need to add history details of each hypercall as suggested by HPA,
which I could not trace easily. Hope it is easy for hypercall authors

Documentation/virtual/hypercalls.txt | 71
++++++++++++++++++++++++++++++++++

1 files changed, 71 insertions(+), 0 deletions(-)

create mode 100644 Documentation/virtual/hypercalls.txt

Documentation/virtual/hypercalls.txt new file +5 -0
new file mode 100644
index 0000000..caffc08
@@ -0,0 +1,71 @@
+Hypercall ABI:
+=============
+A brief look at calling conventions of X86, S390 and PPC
+X86:
+ KVM Hypercalls have a three-byte sequence of either the vmrun or the

vmmrun
+ instruction. The hypervisor can replace it with instructions that are
+ guaranteed to be supported.
+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax. No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.

Documentation/virtual/hypercalls.txt new file +3 -0
+
+S390:
+ R2-R7 are used for parameters 1-6. In addition, R1 is used for

hypercall
+ number. The return value is written to R2.

Quoted 8 lines
Quoted 6 lines

+
+ S390 uses diagnose instruction as hypercall (0x500) along with hypercall
+ number in R1.
+
+ PoewerPC:

PowerPC

Sorry for the typo.

Quoted 10 lines
Quoted 9 lines

+ It uses R3-R10 and hypercall number in R11. R4-R11 are used as output registers.
+ Return value is placed in R3.
+
+ KVM hypercalls uses 4 byte opcode, that are patched with 'hypercall-instructions'
+ property inside the device tree's /hypervisor node.
+ For more information refer to Documentation/virtual/kvm/ppc-pv.txt

Quoted 2 lines

What exactly is this document supposed to cover? We have 3 different hypercall ABIs in KVM on PowerPC:

Thanks for the detailed information on PPC hypercalls. My intention was to cover KVM specific hypercalls here.

Patch is wrongly generated against Documentation/virtual/ instead of Documentation/virtual/kvm will correct that.

Information you have provided should be very useful in understanding PPC hypercalls. Do you wish to have (2), (3) below goto some doc?

Quoted 23 lines

1) KVM hypercalls (ePAPR)

The ePAPR compliant hypercall implementation. This one is used for KVM specific hypercalls. All hypercalls get a KVM vendor prefix (42) and then the hypercall id. We also implement generic hypercalls here, like the ePAPR idle hcall. The instruction sequence for these is listed in the device tree, as you noted above. It's available on all targets.

2) PAPR hypercalls

To run server PowerPC PAPR guests (-M pseries in QEMU), we need to handle PAPR hypercalls. These are the same hypercalls that pHyp, the POWER hypervisor implements. Some of them are handled in the kernel, some are handled in user space. This is only available on book3s_64.

3) OSI hypercalls

In parallel to QEMU, there is another user of KVM on PowerPC:
Mac-on-Linux. That project had its own hypercalls long before we came along with KVM, so to maintain compatibility we also support their hypercalls. All of these get forwarded to user space. This is only useful on book3s_32, but can be used with book3s_64 as well.

Quoted 14 lines

Documentation/virtual/hypercalls.txt line 39

Quoted 12 lines
Documentation/virtual/hypercalls.txt new file +12 -0
+
+KVM Hypercalls Documentation
+===========================
+The template for each hypercall is:
+1. Hypercall name, value.
+2. Architecture(s)
+3. Status (deprecated, obsolete, active)
+4. Purpose
+
+1. KVM_HC_VAPIC_POLL_IRQ
+------------------------
+Value: 1

What are the value fields supposed to mean?

Value field is just #define value of the hypercall in header. If that does not make much sense probably it can be dropped.

#3 lore raw
In reply to Raghavendra K T
Quoted 26 lines · Raghavendra K T

From: Raghavendra K T <[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---
Please have a closer look at Hypercall ABI newly added

Changes since last post:
- Added hypercall ABI (Peter)
- made KVM_HC_VAPIC_POLL_IRQ active explicitly (Randy)

TODO: We need to add history details of each hypercall as suggested by HPA, which I could not trace easily. Hope it is easy for hypercall authors

 Documentation/virtual/hypercalls.txt |   71 ++++++++++++++++++++++++++++++++++
 1 files changed, 71 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/virtual/hypercalls.txt
Documentation/virtual/hypercalls.txt new file +0 -0
new file mode 100644
index 0000000..caffc08

Documentation/virtual/hypercalls.txt line 7

Quoted 7 lines
Documentation/virtual/hypercalls.txt new file +7 -0
@@ -0,0 +1,71 @@
+Hypercall ABI:
+=============
+A brief look at calling conventions of X86, S390 and PPC
+X86:
+ KVM Hypercalls have a three-byte sequence of either the vmrun or the vmmrun
+ instruction. The hypervisor can replace it with instructions that are
+ guaranteed to be supported.

vmcall.

Documentation/virtual/hypercalls.txt line 12

Quoted 5 lines
Documentation/virtual/hypercalls.txt new file +5 -0
+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax.  No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.

It depends on the hypercall. It happens that current hypercalls use the four registers, but its not an ABI (hyper-v hypercalls uses r8, for example).

#4 lore raw
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Raghavendra K T

From: Raghavendra K T<[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---
Please have a closer look at Hypercall ABI newly added

Changes since last post:
- Added hypercall ABI (Peter)
- made KVM_HC_VAPIC_POLL_IRQ active explicitly (Randy)

TODO: We need to add history details of each hypercall as suggested by HPA, which I could not trace easily. Hope it is easy for hypercall authors

  Documentation/virtual/hypercalls.txt |   71 ++++++++++++++++++++++++++++++++++
  1 files changed, 71 insertions(+), 0 deletions(-)

create mode 100644 Documentation/virtual/hypercalls.txt

Quoted 15 lines

Documentation/virtual/hypercalls.txt line 7

Quoted 13 lines
Documentation/virtual/hypercalls.txt new file +7 -0
new file mode 100644
index 0000000..caffc08
@@ -0,0 +1,71 @@
+Hypercall ABI:
+=============
+A brief look at calling conventions of X86, S390 and PPC
+X86:
+ KVM Hypercalls have a three-byte sequence of either the vmrun or the vmmrun
+ instruction. The hypervisor can replace it with instructions that are
+ guaranteed to be supported.

vmcall.

How is this:
KVM Hypercalls have a three-byte sequence of vmcall instruction. But
The hypervisor can replace it with the instructions that are guaranteed to be supported.

(Related question: Is it not that vmcall instruction is patched by 0f 01 09 vmmcall instruction sequence for amd?. So I think we need to change documentation I referred from arch/x86/include/asm/kvm_para.h. It should be vmcall / vmmcall instead of vmrun and vmmrun)

Quoted 10 lines

Documentation/virtual/hypercalls.txt line 12

Quoted 6 lines
Documentation/virtual/hypercalls.txt new file +5 -0
+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax.  No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.

It depends on the hypercall. It happens that current hypercalls use the four registers, but its not an ABI (hyper-v hypercalls uses r8, for example).

Okay, agree. I did not know about hyper-v. Above one is for X86 and KVM (and not X86 / hyper-v). So I should remove ABI and probably say,

KVM x86 Hypercall
==================
Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively. The hypercall number should be placed in rax and the return value will be placed in rax. No other registers will be clobbered unless explicitly stated by the particular hypercall.

#5 lore raw
Quoted 38 lines · Raghavendra K T
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Raghavendra K T

From: Raghavendra K T<[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---
Please have a closer look at Hypercall ABI newly added

Changes since last post:
- Added hypercall ABI (Peter)
- made KVM_HC_VAPIC_POLL_IRQ active explicitly (Randy)

TODO: We need to add history details of each hypercall as suggested by HPA, which I could not trace easily. Hope it is easy for hypercall authors

 Documentation/virtual/hypercalls.txt |   71 ++++++++++++++++++++++++++++++++++
 1 files changed, 71 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/virtual/hypercalls.txt
Quoted 15 lines

Documentation/virtual/hypercalls.txt line 7

Quoted 13 lines
Documentation/virtual/hypercalls.txt new file +7 -0
new file mode 100644
index 0000000..caffc08
@@ -0,0 +1,71 @@
+Hypercall ABI:
+=============
+A brief look at calling conventions of X86, S390 and PPC
+X86:
+ KVM Hypercalls have a three-byte sequence of either the vmrun or the vmmrun
+ instruction. The hypervisor can replace it with instructions that are
+ guaranteed to be supported.

vmcall.

Quoted 7 lines

How is this:
KVM Hypercalls have a three-byte sequence of vmcall instruction. But
The hypervisor can replace it with the instructions that are guaranteed to be supported.

(Related question: Is it not that vmcall instruction is patched by 0f 01 09 vmmcall instruction sequence for amd?.

Yes, it is.

Linux guest binary contains the Intel vmcall instruction, which is patched on AMD to vmmcall.

Quoted 3 lines

So I think we need to
change documentation I referred from arch/x86/include/asm/kvm_para.h. It should be vmcall / vmmcall instead of vmrun and vmmrun)

Yes.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Documentation/virtual/hypercalls.txt new file +5 -0
+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax.  No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.
Quoted 6 lines
Quoted 3 lines

It depends on the hypercall. It happens that current hypercalls use the four registers, but its not an ABI (hyper-v hypercalls uses r8, for example).

Okay, agree. I did not know about hyper-v. Above one is for X86 and KVM (and not X86 / hyper-v). So I should remove ABI and probably say,

Just say "Linux x86 Hypercall", yeah.

Quoted 11 lines

KVM x86 Hypercall
==================
Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively. The hypercall number should be placed in rax and the return value will be placed in rax. No other registers will be clobbered unless explicitly stated by the particular hypercall.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 10 lines · Marcelo Tosatti
Quoted 9 lines · Raghavendra K T
Quoted 8 lines · Marcelo Tosatti
Quoted 7 lines · Raghavendra K T

From: Raghavendra K T<[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---

[...]

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax. No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

It depends on the hypercall. It happens that current hypercalls use the four registers, but its not an ABI (hyper-v hypercalls uses r8, for example).

Okay, agree. I did not know about hyper-v. Above one is for X86 and KVM (and not X86 / hyper-v). So I should remove ABI and probably say,

Just say "Linux x86 Hypercall", yeah.

Ok, Thanks Marcelo.will respin patches with the changes you suggested.

Also I would like add Alex's description on ppc hypercalls into Documentation/virtual/kvm/ppc-pv.txt with Alex from/sob or suggested-by in a separate patch unless I see any objection to that.

#7 lore raw
Quoted 25 lines · Raghavendra K T
Quoted 10 lines · Marcelo Tosatti
Quoted 9 lines · Raghavendra K T
Quoted 8 lines · Marcelo Tosatti
Quoted 7 lines · Raghavendra K T

From: Raghavendra K T<[email protected]>

Thanks Alex for KVM_HC_FEATURES inputs and Jan for VAPIC_POLL_IRQ, and Peter (HPA) for suggesting hypercall ABI addition.

---

[...]

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines
Quoted 5 lines

+
+ Up to four arguments may be passed in rbx, rcx, rdx, and rsi respectively.
+ The hypercall number should be placed in rax and the return value will be
+ placed in rax. No other registers will be clobbered unless explicitly stated
+ by the particular hypercall.

It depends on the hypercall. It happens that current hypercalls use the four registers, but its not an ABI (hyper-v hypercalls uses r8, for example).

Okay, agree. I did not know about hyper-v. Above one is for X86 and KVM (and not X86 / hyper-v). So I should remove ABI and probably say,

Quoted 9 lines
Quoted 2 lines

Just say "Linux x86 Hypercall", yeah.

Ok, Thanks Marcelo.will respin patches with the changes you suggested.

Also I would like add Alex's description on ppc hypercalls into Documentation/virtual/kvm/ppc-pv.txt with Alex from/sob or suggested-by in a separate patch unless I see any objection to that.

Yeah, don't stall this on me :)

Alex