hyperv: Fixes for get_vtl(), hv_vtl_apicid_to_vp_id()
Subsystems (3)
v3 cover hyperv: Fixes for get_vtl(), hv_vtl_apicid_to_vp_id()
The get_vtl(void) function
* has got one bug when the code started using a wrong pointer type after refactoring, and also
* the both function in question don't adhere to the requirements of the Hypervisor Top-Level Funactional Specification[1, 2] as the code overlaps the input and output areas for a hypercall.
The first issue leads to a wrong 100% reproducible computation due to reading a byte worth of data at a wrong offset. That in turn leads to using a nonsensical value ("fortunately", could catch it easily!) for the current VTL when initiating VMBus communications. As a repercussion from that, the system wouldn't boot. The fix is straightforward: use the correct pointer type.
The second issue doesn't seem to lead to any reproducible breakage just yet. It is fixed with using the output hypercall pages allocated per-CPU, and that isn't the only or the most obvious choice so let me elaborate why that fix appears to be the best one in my opinion out of the options I could conceive of.
The approach chosen for fixing the second issue makes two things shine through:
* these functions just get a vCPU register, no special treatment needs to be involved,
* VTLs and dom0 can and should share code as both exist to provide services to a guest(s), be that from within the partition or from outside of it.
The projected benefits include replacing the functions in question with a future hv_get_vp_registers one shared between dom0 and VTLs to allow for a better test coverage.
I have validated the fixes by booting the fixed kernel in VTL2 up using OpenVMM and OpenHCL[3, 4].
[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface
[2] https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/main/tlfs
[3] https://openvmm.dev/guide/user_guide/openhcl.html
[4] https://github.com/microsoft/OpenVMM
[v3]
- Added a fix for hv_vtl_apicid_to_vp_id(),
- Split out the patch for enabling the hypercall output page,
- Updated the title of the patch series,
[v2]: https://lore.kernel.org/lkml/[email protected]/
- Used the suggestions to define an additional structure to improve code readability,
- Split out the patch with that definition.
[v1]: https://lore.kernel.org/lkml/[email protected]/
Roman Kisel (5):
hyperv: Define struct hv_output_get_vp_registers
hyperv: Fix pointer type for the output of the hypercall in
get_vtl(void)
hyperv: Enable the hypercall output page for the VTL mode
hyperv: Do not overlap the hvcall IO areas in get_vtl()
hyperv: Do not overlap the hvcall IO areas in hv_vtl_apicid_to_vp_id()
arch/x86/hyperv/hv_init.c | 6 ++-- arch/x86/hyperv/hv_vtl.c | 2 +- drivers/hv/hv_common.c | 6 ++-- include/hyperv/hvgdk_mini.h | 58 +++++++++++++++++++++++++++++++++++-- 4 files changed, 63 insertions(+), 9 deletions(-)
base-commit: 4d4ace979a3066e5c940331571e6c1c3f280d1d3
Signature · 1 line
2.34.1
PATCH v3 1/5 hyperv: Define struct hv_output_get_vp_registers
There is no definition of the output structure for the GetVpRegisters hypercall. Hence, using the hypercall is not possible when the output value has some structure to it. Even getting a datum of a primitive type reads as ad-hoc without that definition.
Define struct hv_output_get_vp_registers to enable using the GetVpRegisters hypercall. Make provisions for all supported architectures. No functional changes.
--- include/hyperv/hvgdk_mini.h | 58 +++++++++++++++++++++++++++++++++++-- 1 file changed, 56 insertions(+), 2 deletions(-)
include/hyperv/hvgdk_mini.h +56 -2
@@ -781,6 +781,8 @@ struct hv_timer_message_payload {__u64 delivery_time;/* When the message was delivered */ } __packed; +#if defined(CONFIG_X86) + struct hv_x64_segment_register { u64 base; u32 limit;
@@ -807,6 +809,8 @@ struct hv_x64_table_register {u64 base; } __packed; +#endif +union hv_input_vtl {u8 as_uint8;struct {
@@ -1068,6 +1072,41 @@ union hv_dispatch_suspend_register { } __packed; }; +#if defined(CONFIG_ARM64 ) + +union hv_arm64_pending_interruption_register { +u64 as_uint64; +struct { +u64 interruption_pending :1 ; +u64 interruption_type :1 ; +u64 reserved :30 ; +u32 error_code; + } __packed; +}; + +union hv_arm64_interrupt_state_register { +u64 as_uint64; +struct { +u64 interrupt_shadow :1 ; +u64 reserved :63 ; + } __packed; +}; + +union hv_arm64_pending_synthetic_exception_event { +u64 as_uint64[2 ]; +struct { +u32 event_pending :1 ; +u32 event_type :3 ; +u32 reserved :4 ; +u32 exception_type; +u64 context; + } __packed; +}; + +#endif + +#if defined(CONFIG_X86) + union hv_x64_interrupt_state_register { u64 as_uint64; struct {
@@ -1091,6 +1130,8 @@ union hv_x64_pending_interruption_register { } __packed; }; +#endif +union hv_register_value {struct hv_u128 reg128;u64 reg64;
@@ -1098,13 +1139,26 @@ union hv_register_value {u16 reg16;u8 reg8; -struct hv_x64_segment_register segment; -struct hv_x64_table_register table;union hv_explicit_suspend_register explicit_suspend;union hv_intercept_suspend_register intercept_suspend;union hv_dispatch_suspend_register dispatch_suspend; +#if defined(CONFIG_X86 ) +struct hv_x64_segment_register segment; +struct hv_x64_table_register table;union hv_x64_interrupt_state_register interrupt_state;union hv_x64_pending_interruption_register pending_interruption; +#elif defined(CONFIG_ARM64 ) +union hv_arm64_pending_interruption_register pending_interruption; +union hv_arm64_interrupt_state_register interrupt_state; +union hv_arm64_pending_synthetic_exception_event pending_synthetic_exception_event; +#else +#error "This architecture is not supported" +#endif +}; + +/* NOTE: Linux helper struct - NOT from Hyper-V code */ +struct hv_output_get_vp_registers { + DECLARE_FLEX_ARRAY(union hv_register_value, values); };#if defined(CONFIG_ARM64 )
Signature · 1 line
2.34.1
PATCH v3 2/5 hyperv: Fix pointer type for the output of the hypercall in get_vtl(void)
In reply to Roman KiselCommit bc905fa8b633 ("hyperv: Switch from hyperv-tlfs.h to hyperv/hvhdk.h") changed the type of the output pointer to struct hv_register_assoc from struct hv_get_vp_registers_output. That leads to an incorrect computation, and leaves the system broken.
Use the correct pointer type for the output of the GetVpRegisters hypercall.
--- arch/x86/hyperv/hv_init.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/hyperv/hv_init.c +3 -3
@@ -416,13 +416,13 @@ static u8 __init get_vtl(void) {u64 control =HV_HYPERCALL_REP_COMP_1 |HVCALL_GET_VP_REGISTERS ;struct hv_input_get_vp_registers *input; -struct hv_register_assoc *output; +struct hv_output_get_vp_registers *output;unsigned long flags;u64 ret;local_irq_save (flags); input = *this_cpu_ptr (hyperv_pcpu_input_arg); - output = (struct hv_register_assoc *)input; + output = (struct hv_output_get_vp_registers *)input;memset (input,0 ,struct_size (input, names,1 )); input->partition_id =HV_PARTITION_ID_SELF ;
@@ -432,7 +432,7 @@ static u8 __init get_vtl(void) ret =hv_do_hypercall (control, input, output);if (hv_result_success (ret)) { - ret = output->value.reg8 &HV_X64_VTL_MASK ; + ret = output->values[0 ].reg8 &HV_X64_VTL_MASK ; }else {pr_err ("Failed to get VTL(error: %lld) exiting...\n" , ret);BUG ();
Signature · 1 line
2.34.1
PATCH v3 3/5 hyperv: Enable the hypercall output page for the VTL mode
In reply to Roman KiselDue to the hypercall page not being allocated in the VTL mode, the code resorts to using a part of the input page.
Allocate the hypercall output page in the VTL mode thus enabling it to use it for output and share code with dom0.
--- drivers/hv/hv_common.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/hv/hv_common.c +3 -3
@@ -340,7 +340,7 @@ int __init hv_common_init(void)BUG_ON (!hyperv_pcpu_input_arg);/* Allocate the per-CPU state for output arg for root */ -if (hv_root_partition) { +if (hv_root_partition ||IS_ENABLED (CONFIG_HYPERV_VTL_MODE )) { hyperv_pcpu_output_arg = alloc_percpu(void *);BUG_ON (!hyperv_pcpu_output_arg); }
@@ -435,7 +435,7 @@ int hv_common_cpu_init(unsigned int cpu)void **inputarg, **outputarg;u64 msr_vp_index;gfp_t flags; -int pgcount = hv_root_partition ?2 :1 ; +const int pgcount = (hv_root_partition ||IS_ENABLED (CONFIG_HYPERV_VTL_MODE )) ?2 :1 ;void *mem;int ret;
@@ -453,7 +453,7 @@ int hv_common_cpu_init(unsigned int cpu)if (!mem)return -ENOMEM ; -if (hv_root_partition) { +if (hv_root_partition ||IS_ENABLED (CONFIG_HYPERV_VTL_MODE )) { outputarg = (void **)this_cpu_ptr (hyperv_pcpu_output_arg); *outputarg = (char *)mem +HV_HYP_PAGE_SIZE ; }
Signature · 1 line
2.34.1
PATCH v3 4/5 hyperv: Do not overlap the hvcall IO areas in get_vtl()
In reply to Roman KiselThe Top-Level Functional Specification for Hyper-V, Section 3.6 [1, 2], disallows overlapping of the input and output hypercall areas, and get_vtl(void) does overlap them.
Use the output hypercall page of the current vCPU for the hypercall.
[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface
[2] https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/main/tlfs
--- arch/x86/hyperv/hv_init.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/hyperv/hv_init.c +1 -1
@@ -422,7 +422,7 @@ static u8 __init get_vtl(void)local_irq_save (flags); input = *this_cpu_ptr (hyperv_pcpu_input_arg); - output = (struct hv_output_get_vp_registers *)input; + output = *this_cpu_ptr (hyperv_pcpu_output_arg);memset (input,0 ,struct_size (input, names,1 )); input->partition_id =HV_PARTITION_ID_SELF ;
Signature · 1 line
2.34.1
PATCH v3 5/5 hyperv: Do not overlap the hvcall IO areas in hv_vtl_apicid_to_vp_id()
In reply to Roman KiselThe Top-Level Functional Specification for Hyper-V, Section 3.6 [1, 2], disallows overlapping of the input and output hypercall areas, and hv_vtl_apicid_to_vp_id() overlaps them.
Use the output hypercall page of the current vCPU for the hypercall.
[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface
[2] https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/main/tlfs
--- arch/x86/hyperv/hv_vtl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/hyperv/hv_vtl.c +1 -1
@@ -189,7 +189,7 @@ static int hv_vtl_apicid_to_vp_id(u32 apic_id) input->partition_id =HV_PARTITION_ID_SELF ; input->apic_ids[0 ] = apic_id; - output = (u32 *)input; + output = (u32 *)*this_cpu_ptr (hyperv_pcpu_output_arg); control =HV_HYPERCALL_REP_COMP_1 |HVCALL_GET_VP_ID_FROM_APIC_ID ; status =hv_do_hypercall (control, input, output);
Signature · 1 line
2.34.1
Re: [PATCH v3 2/5] hyperv: Fix pointer type for the output of the hypercall in get_vtl(void)
In reply to Roman KiselQuoted 11 lines · Roman Kisel
Commit
bc905fa8b633("hyperv: Switch fromhyperv-tlfs.htohyperv/hvhdk.h") changed the type of the output pointer tostruct hv_register_assocfromstruct hv_get_vp_registers_output. That leads to an incorrect computation, and leaves the system broken.Use the correct pointer type for the output of the GetVpRegisters hypercall.
Signed-off-by Roman Kisel <[email protected]>--- arch/x86/hyperv/hv_init.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Please add a Fixes tag, you can just reply to this email, tooling would probably pick it up. Assuming that:
Re: [PATCH v3 3/5] hyperv: Enable the hypercall output page for the VTL mode
In reply to Roman KiselQuoted 25 lines · Roman Kisel
Due to the hypercall page not being allocated in the VTL mode, the code resorts to using a part of the input page.
Allocate the hypercall output page in the VTL mode thus enabling it to use it for output and share code with dom0.
Signed-off-by Roman Kisel <[email protected]>--- drivers/hv/hv_common.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)drivers/hv/hv_common.c +1
-1@@ -340,7 +340,7 @@ int __init hv_common_init(void)BUG_ON (!hyperv_pcpu_input_arg);/* Allocate the per-CPU state for output arg for root */ -if (hv_root_partition) { +if (hv_root_partition ||IS_ENABLED (CONFIG_HYPERV_VTL_MODE )) { hyperv_pcpu_output_arg = alloc_percpu(void *);BUG_ON (!hyperv_pcpu_output_arg); }
drivers/hv/hv_common.c int hv_common_cpu_init(unsigned int cpu) · line 440
Quoted 7 lines
drivers/hv/hv_common.c +1
-1@@ -435,7 +435,7 @@ int hv_common_cpu_init(unsigned int cpu)void **inputarg, **outputarg;u64 msr_vp_index;gfp_t flags; -int pgcount = hv_root_partition ?2 :1 ; +const int pgcount = (hv_root_partition ||IS_ENABLED (CONFIG_HYPERV_VTL_MODE )) ?2 :1 ;void *mem;int ret;
Thanks for the const!
Re: [PATCH v3 4/5] hyperv: Do not overlap the hvcall IO areas in get_vtl()
In reply to Roman KiselQuoted 6 lines · Roman Kisel
The Top-Level Functional Specification for Hyper-V, Section 3.6 [1, 2], disallows overlapping of the input and output hypercall areas, and
get_vtl(void)does overlap them.Use the output hypercall page of the current vCPU for the hypercall.
Quoted 9 lines
[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface
[2] https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/main/tlfsFixes 8387ce06d70b ("x86/hyperv: Set Virtual Trust Level in VMBus init message")Signed-off-by Roman Kisel <[email protected]>--- arch/x86/hyperv/hv_init.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Looks good to me.
Re: [PATCH v3 5/5] hyperv: Do not overlap the hvcall IO areas in hv_vtl_apicid_to_vp_id()
In reply to Roman KiselQuoted 19 lines · Roman Kisel
The Top-Level Functional Specification for Hyper-V, Section 3.6 [1, 2], disallows overlapping of the input and output hypercall areas, and
hv_vtl_apicid_to_vp_id()overlaps them.Use the output hypercall page of the current vCPU for the hypercall.
[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface
[2] https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/main/tlfsSigned-off-by Roman Kisel <[email protected]>--- arch/x86/hyperv/hv_vtl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/hyperv/hv_vtl.c +0
-0
arch/x86/hyperv/hv_vtl.c static int hv_vtl_apicid_to_vp_id(u32 apic_id) · line 192
Quoted 5 lines
arch/x86/hyperv/hv_vtl.c +1
-1@@ -189,7 +189,7 @@ static int hv_vtl_apicid_to_vp_id(u32 apic_id) input->partition_id =HV_PARTITION_ID_SELF ; input->apic_ids[0 ] = apic_id; - output = (u32 *)input; + output = (u32 *)*this_cpu_ptr (hyperv_pcpu_output_arg);
^
Nit: I believe the space is preferred, but I won't insist on respinning it for that.
It's a good idea to give credit to Michael with a Reported-by tag, and maybe a Closes: tag with a link to his email.
As with the Fixes tag for patch 2, you don't need to respin the series and can just reply to this thread.
Otherwise, looks good to me.
Re: [PATCH v3 1/5] hyperv: Define struct hv_output_get_vp_registers
In reply to Roman KiselQuoted 37 lines · Roman Kisel
There is no definition of the output structure for the GetVpRegisters hypercall. Hence, using the hypercall is not possible when the output value has some structure to it. Even getting a datum of a primitive type reads as ad-hoc without that definition.
Define struct
hv_output_get_vp_registersto enable using the GetVpRegisters hypercall. Make provisions for all supported architectures. No functional changes.Signed-off-by Roman Kisel <[email protected]>--- include/hyperv/hvgdk_mini.h | 58 +++++++++++++++++++++++++++++++++++-- 1 file changed, 56 insertions(+), 2 deletions(-)include/hyperv/hvgdk_mini.h +5
-2struct {@@ -1091,6 +1130,8 @@ union hv_x64_pending_interruption_register { } __packed; }; +#endif +union hv_register_value {struct hv_u128 reg128;u64 reg64;@@ -1098,13 +1139,26 @@ union hv_register_value {u16 reg16;u8 reg8; -struct hv_x64_segment_register segment; -struct hv_x64_table_register table;union hv_explicit_suspend_register explicit_suspend;union hv_intercept_suspend_register intercept_suspend;union hv_dispatch_suspend_register dispatch_suspend; +#if defined(CONFIG_X86 ) +struct hv_x64_segment_register segment; +struct hv_x64_table_register table;
include/hyperv/hvgdk_mini.h union hv_register_value { · line 1157
Quoted 10 lines
include/hyperv/hvgdk_mini.h +8
-0union hv_x64_interrupt_state_register interrupt_state;union hv_x64_pending_interruption_register pending_interruption; +#elif defined(CONFIG_ARM64 ) +union hv_arm64_pending_interruption_register pending_interruption; +union hv_arm64_interrupt_state_register interrupt_state; +union hv_arm64_pending_synthetic_exception_event pending_synthetic_exception_event; +#else +#error "This architecture is not supported" +#endif +};
I don't love the #error for unsupported architectures when Kconfig takes care of that for us, but I suppose it's for completeness since the arm64 members have to be conditioned on CONFIG_ARM64?
include/hyperv/hvgdk_mini.h union hv_register_value { · line 1162
Quoted 5 lines
include/hyperv/hvgdk_mini.h +4
-0+ +/* NOTE: Linux helper struct - NOT from Hyper-V code */ +struct hv_output_get_vp_registers { + DECLARE_FLEX_ARRAY(union hv_register_value, values); };
I'm not super familiar with DECLARE_FLEX_ARRAY() but it appears this needs to be wrapped in an anonymous struct at the least per this comment for the definition of DECLARE_FLEX_ARRAY()
Quoted 3 lines
(quoted diff) +0
-0* In order to have a flexible array member [...] alone in a * struct, it needs to be wrapped in an anonymous struct with at least 1* named member, but that member can be empty.
Nuno, since you seem to be more familiar, can you provide some guidance?
Thanks,
Easwar
Re: [PATCH v3 1/5] hyperv: Define struct hv_output_get_vp_registers
Quoted 1 line · Easwar Hariharan
On 12/26/2024 1:31 PM, Roman Kisel wrote:
[...]
Quoted 9 lines
include/hyperv/hvgdk_mini.h union hv_register_value { · line 1157
Quoted 4 lines
include/hyperv/hvgdk_mini.h +4
-0+#else +#error "This architecture is not supported" +#endif +};I don't love the #error for unsupported architectures when
Kconfigtakes care of that for us, but I suppose it's for completeness since the arm64 members have to be conditioned onCONFIG_ARM64?
Felt right to do that, but because it raises questions, it should carry a comment or be removed as, if you pointed out, Kconfig is in charge of that kind of validation. As Kconfig permits Hyper-V on the correct set of arch'es, the best choice would be to remove I think.
Quoted 10 lines
include/hyperv/hvgdk_mini.h union hv_register_value { · line 1162
Quoted 5 lines
include/hyperv/hvgdk_mini.h +4
-0+ +/* NOTE: Linux helper struct - NOT from Hyper-V code */ +struct hv_output_get_vp_registers { + DECLARE_FLEX_ARRAY(union hv_register_value, values); };I'm not super familiar with
DECLARE_FLEX_ARRAY()but it appears this needs to be wrapped in an anonymous struct at the least per this comment for the definition ofDECLARE_FLEX_ARRAY()
Quoted 5 lines
Quoted 3 lines
* In order to have a flexible array member [...] alone in a
* struct, it needs to be wrapped in an anonymous struct with at least 1
* named member, but that member can be empty.Nuno, since you seem to be more familiar, can you provide some guidance?
I will wrap the struct member of the documentation requires, not to make the code look suspicious.
Quoted 3 lines
Thanks,
Easwar
Signature · 2 lines
Thank you, Roman
Re: [PATCH v3 5/5] hyperv: Do not overlap the hvcall IO areas in hv_vtl_apicid_to_vp_id()
In reply to Easwar HariharanQuoted 23 lines · Easwar Hariharan
Quoted 22 lines · Roman Kisel
The Top-Level Functional Specification for Hyper-V, Section 3.6 [1, 2], disallows overlapping of the input and output hypercall areas, and
hv_vtl_apicid_to_vp_id()overlaps them.Use the output hypercall page of the current vCPU for the hypercall.
[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface
[2] https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/main/tlfsSigned-off-by Roman Kisel <[email protected]>--- arch/x86/hyperv/hv_vtl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/hyperv/hv_vtl.c +0
-0@@ -189,7 +189,7 @@ static int hv_vtl_apicid_to_vp_id(u32 apic_id) input->partition_id =HV_PARTITION_ID_SELF ; input->apic_ids[0 ] = apic_id;
Quoted 5 lines
arch/x86/hyperv/hv_vtl.c static int hv_vtl_apicid_to_vp_id(u32 apic_id) · line 192
Quoted 2 lines
arch/x86/hyperv/hv_vtl.c +1
-1- output = (u32 *)input; + output = (u32 *)*this_cpu_ptr (hyperv_pcpu_output_arg);^Nit: I believe the space is preferred, but I won't insist on respinning it for that.
I think we can drop the whole type cast thing (as the other patch does). Type coercion will do what is needed, and the type cast just appears to be noise I guess.
Quoted 3 lines
It's a good idea to give credit to Michael with a Reported-by tag, and maybe a Closes: tag with a link to his email.
My bad, I should have definitely done that! Will certainly do!
Quoted 3 lines
As with the Fixes tag for patch 2, you don't need to respin the series and can just reply to this thread.
"Fixes" means something when the commit is present in the Linus'es tree
as I understood from the Wei's reply (https://lore.kernel.org/lkml/Z2OIsFUXcjVXpqtw@liuwe-devbox-debian-v2/):
" This commit is not in the mainline kernel yet, so this tag is not needed.
It will most likely to be wrong since I will need to rebase the hyperv-next branch.
I can fold this patch into the original patch and leave your Signed-off-by there."
I'll check if the commit has been pulled into the mainline tree.
Quoted 4 lines
Otherwise, looks good to me.
Reviewed-by Easwar Hariharan <[email protected]>
Signature · 2 lines
Thank you, Roman