Skip to contentSkip to conversation

memory: tegra: Skip SID override from Guest VM

PATCH In review Sumit Gupta 6 participants 12 messages Feb 6, 2024 – Feb 7, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 memory: tegra: Skip SID override from Guest VM

MC SID register access is restricted for Guest VM. So, skip the SID override programming from the Guest VM.

---
 drivers/memory/tegra/tegra186.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/memory/tegra/tegra186.c +11 -0
index 1b3183951bfe..df441896b69d 100644
@@ -10,6 +10,7 @@
 #include <linux/of.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
+#include <asm/virt.h>
 
 #include <soc/tegra/mc.h>
 
@@ -118,6 +119,11 @@ static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev)
 	unsigned int i, index = 0;
 	u32 sid;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
 	if (!tegra_dev_iommu_get_stream_id(dev, &sid))
 		return 0;
 
@@ -146,6 +152,11 @@ static int tegra186_mc_resume(struct tegra_mc *mc)
 #if IS_ENABLED(CONFIG_IOMMU_API)
 	unsigned int i;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
 	for (i = 0; i < mc->soc->num_clients; i++) {
 		const struct tegra_mc_client *client = &mc->soc->clients[i];
 
Signature · 1 line
2.17.1
#1 lore raw
Quoted 13 lines · Sumit Gupta

MC SID register access is restricted for Guest VM. So, skip the SID override programming from the Guest VM.

---
 drivers/memory/tegra/tegra186.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/memory/tegra/tegra186.c +0 -0
index 1b3183951bfe..df441896b69d 100644

drivers/memory/tegra/tegra186.c line 13

Quoted 4 lines
drivers/memory/tegra/tegra186.c +1 -0
@@ -10,6 +10,7 @@
 #include <linux/of.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
+#include <asm/virt.h>

Are you sure this still compile tests?

drivers/memory/tegra/tegra186.c static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev) · line 123

Quoted 9 lines
drivers/memory/tegra/tegra186.c +2 -0
 #include <soc/tegra/mc.h>
 
@@ -118,6 +119,11 @@ static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev)
 	unsigned int i, index = 0;
 	u32 sid;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");

Doesn't this depend on hypervisor?

drivers/memory/tegra/tegra186.c static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev) · line 128

Quoted 5 lines
drivers/memory/tegra/tegra186.c +3 -0
+		return 0;
+	}
+
 	if (!tegra_dev_iommu_get_stream_id(dev, &sid))
 		return 0;

Best regards,
Krzysztof

#2 lore raw
In reply to Sumit Gupta
Quoted 4 lines · Sumit Gupta

MC SID register access is restricted for Guest VM. So, skip the SID override programming from the Guest VM.

Surely this is probed from the DT?

Why would the hypervisor put this in the guest's DT if that hypervisor isn't exposing it to the guest?

Mark.

Quoted 43 lines
---
 drivers/memory/tegra/tegra186.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/memory/tegra/tegra186.c +11 -0
index 1b3183951bfe..df441896b69d 100644
@@ -10,6 +10,7 @@
 #include <linux/of.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
+#include <asm/virt.h>
 
 #include <soc/tegra/mc.h>
 
@@ -118,6 +119,11 @@ static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev)
 	unsigned int i, index = 0;
 	u32 sid;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
 	if (!tegra_dev_iommu_get_stream_id(dev, &sid))
 		return 0;
 
@@ -146,6 +152,11 @@ static int tegra186_mc_resume(struct tegra_mc *mc)
 #if IS_ENABLED(CONFIG_IOMMU_API)
 	unsigned int i;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
 	for (i = 0; i < mc->soc->num_clients; i++) {
 		const struct tegra_mc_client *client = &mc->soc->clients[i];
 
Signature · 1 line
2.17.1
#3 lore raw
In reply to Sumit Gupta

Hi Sumit,

Quoted 35 lines · Sumit Gupta

MC SID register access is restricted for Guest VM. So, skip the SID override programming from the Guest VM.

---
 drivers/memory/tegra/tegra186.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/memory/tegra/tegra186.c +6 -0
index 1b3183951bfe..df441896b69d 100644
@@ -10,6 +10,7 @@
 #include <linux/of.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
+#include <asm/virt.h>
 
 #include <soc/tegra/mc.h>
 
@@ -118,6 +119,11 @@ static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev)
 	unsigned int i, index = 0;
 	u32 sid;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
 	if (!tegra_dev_iommu_get_stream_id(dev, &sid))
 		return 0;
 
@@ -146,6 +152,11 @@ static int tegra186_mc_resume(struct tegra_mc *mc)
 #if IS_ENABLED(CONFIG_IOMMU_API)

drivers/memory/tegra/tegra186.c static int tegra186_mc_resume(struct tegra_mc *mc) · line 162

Quoted 10 lines
drivers/memory/tegra/tegra186.c +5 -0
 	unsigned int i;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
 	for (i = 0; i < mc->soc->num_clients; i++) {
 		const struct tegra_mc_client *client = &mc->soc->clients[i];
 

This doesn't look right. Multiple reasons:

- This helper really has nothing to do in a driver. This is architectural stuff that is not intended for use outside of arch core code.

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer
#4 lore raw

Hi Marc,

Quoted 52 lines · Marc Zyngier

Hi Sumit,

Quoted 35 lines · Sumit Gupta

MC SID register access is restricted for Guest VM. So, skip the SID override programming from the Guest VM.

---
  drivers/memory/tegra/tegra186.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
drivers/memory/tegra/tegra186.c +6 -0
index 1b3183951bfe..df441896b69d 100644
@@ -10,6 +10,7 @@
  #include <linux/of.h>
  #include <linux/of_platform.h>
  #include <linux/platform_device.h>
+#include <asm/virt.h>
 
  #include <soc/tegra/mc.h>
 
@@ -118,6 +119,11 @@ static int tegra186_mc_probe_device(struct tegra_mc *mc, struct device *dev)
  	unsigned int i, index = 0;
  	u32 sid;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
  	if (!tegra_dev_iommu_get_stream_id(dev, &sid))
  		return 0;
 
@@ -146,6 +152,11 @@ static int tegra186_mc_resume(struct tegra_mc *mc)
  #if IS_ENABLED(CONFIG_IOMMU_API)

drivers/memory/tegra/tegra186.c static int tegra186_mc_resume(struct tegra_mc *mc) · line 162

Quoted 10 lines
drivers/memory/tegra/tegra186.c +5 -0
  	unsigned int i;
 
+	if (!is_kernel_in_hyp_mode()) {
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
  	for (i = 0; i < mc->soc->num_clients; i++) {
  		const struct tegra_mc_client *client = &mc->soc->clients[i];
 

This doesn't look right. Multiple reasons:

Quoted 3 lines

- This helper really has nothing to do in a driver. This is architectural stuff that is not intended for use outside of arch core code.

We see a few other drivers using this ...

drivers/perf/arm_pmuv3.c drivers/clocksource/arm_arch_timer.c drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h drivers/hwtracing/coresight/coresight-etm4x-core.c drivers/hwtracing/coresight/coresight-etm4x-core.c drivers/irqchip/irq-apple-aic.c

We were looking for a way to determine if the OS is a guest OS or not. However, I can see that this is a ARM64 specific API and so probably the above are only compiled for ARM64. Interestingly, the AMD driver implements the following ...

static inline bool is_virtual_machine(void)
{

#if defined(CONFIG_X86)
        return boot_cpu_has(X86_FEATURE_HYPERVISOR);
#elif defined(CONFIG_ARM64)
        return !is_kernel_in_hyp_mode();
#else
        return false;

#endif
}

Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

I thought this API just checks to see if we are in EL2?

Quoted 4 lines

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Sumit can add some more details on why we don't completely disable the device for guest OSs.

Jon

Signature · 1 line
nvpublic
maintainer
#5 lore raw

On 06/02/2024 12:28, Jon Hunter wrote:

...

Quoted 5 lines
Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

I thought this API just checks to see if we are in EL2?

Sorry to add a bit more info, I see EL2 is used for hypervisor [0], but on my Tegra186 with no hypervisor I see ...

CPU: All CPU(s) started at EL2

Jon

[0] https://developer.arm.com/documentation/102412/0103/Privilege-and-Exception-levels/Exception-levels

Signature · 1 line
nvpublic
#7 lore raw
In reply to Jon Hunter

Hi Jon,

Quoted 53 lines · Jon Hunter

Hi Marc,

Quoted 49 lines · Marc Zyngier

Hi Sumit,

Quoted 36 lines · Sumit Gupta

MC SID register access is restricted for Guest VM. So, skip the SID override programming from the Guest VM.

---
  drivers/memory/tegra/tegra186.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
drivers/memory/tegra/tegra186.c +1 -0
index 1b3183951bfe..df441896b69d 100644
@@ -10,6 +10,7 @@
  #include <linux/of.h>
  #include <linux/of_platform.h>
  #include <linux/platform_device.h>
+#include <asm/virt.h>
    #include <soc/tegra/mc.h>
  @@ -118,6 +119,11 @@ static int tegra186_mc_probe_device(struct
 tegra_mc *mc, struct device *dev)
  	unsigned int i, index = 0;
  	u32 sid;
  +	if (!is_kernel_in_hyp_mode()) {
drivers/memory/tegra/tegra186.c +4 -0
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
  	if (!tegra_dev_iommu_get_stream_id(dev, &sid))
  		return 0;
  @@ -146,6 +152,11 @@ static int tegra186_mc_resume(struct

tegra_mc *mc)
#if IS_ENABLED(CONFIG_IOMMU_API)
unsigned int i;
+ if (!is_kernel_in_hyp_mode()) {

drivers/memory/tegra/tegra186.c static int tegra186_mc_resume(struct tegra_mc *mc) · line 161

Quoted 6 lines
drivers/memory/tegra/tegra186.c +4 -0
+		dev_dbg(mc->dev, "Register access not allowed\n");
+		return 0;
+	}
+
  	for (i = 0; i < mc->soc->num_clients; i++) {
  		const struct tegra_mc_client *client = &mc->soc->clients[i];

This doesn't look right. Multiple reasons:

Quoted 8 lines
Quoted 3 lines

- This helper really has nothing to do in a driver. This is architectural stuff that is not intended for use outside of arch core code.

We see a few other drivers using this ...

drivers/perf/arm_pmuv3.c
drivers/clocksource/arm_arch_timer.c

These two are definitely part of the CPU architecture.

Quoted 1 line

drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h

This is just a bug. Please don't copy this stuff.

Quoted 3 lines

drivers/hwtracing/coresight/coresight-etm4x-core.c drivers/hwtracing/coresight/coresight-etm4x-core.c drivers/irqchip/irq-apple-aic.c

These are also part of the CPU architecture.

Quoted 8 lines

We were looking for a way to determine if the OS is a guest OS or not. However, I can see that this is a ARM64 specific API and so probably the above are only compiled for ARM64. Interestingly, the AMD driver implements the following ...

static inline bool is_virtual_machine(void)
{

Quoted 8 lines
#if defined(CONFIG_X86)
        return boot_cpu_has(X86_FEATURE_HYPERVISOR);
#elif defined(CONFIG_ARM64)
        return !is_kernel_in_hyp_mode();
#else
        return false;

#endif
}

This stuff should simply be ripped out and burned. Whoever wrote it didn't understand how this works.

Quoted 6 lines
Quoted 4 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

I thought this API just checks to see if we are in EL2?

It does. And that's the problem. On ARMv8.0, we run the Linux kernel at EL1. Tegra186 is ARMv8.0 (Denver + A57). So as written, this change breaks the very platform it intends to support.

Quoted 8 lines
Quoted 5 lines

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Sumit can add some more details on why we don't completely disable the device for guest OSs.

It's not about disabling it. It is about correctly supporting it (providing full emulation for it), or advertising it as something different so that SW can handle it differently.

Poking into the internals of how the kernel is booted for a driver that isn't tied to the core architecture (because it would need to access system registers, for example) is not an acceptable outcome.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#6 lore raw
In reply to Jon Hunter
Quoted 10 lines · Jon Hunter

On 06/02/2024 12:28, Jon Hunter wrote:

...

Quoted 4 lines
Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

Quoted 7 lines
Quoted 1 line

I thought this API just checks to see if we are in EL2?

Sorry to add a bit more info, I see EL2 is used for hypervisor [0], but on my Tegra186 with no hypervisor I see ...

CPU: All CPU(s) started at EL2

Yes. and yet the kernel runs at EL1 (on ARMv8.0, we can't run the kernel at EL2 at all).

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#8 lore raw
In reply to Marc Zyngier
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jon Hunter

On 06/02/2024 12:17, Marc Zyngier wrote:

[...]

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

I thought this API just checks to see if we are in EL2?

It does. And that's the problem. On ARMv8.0, we run the Linux kernel at EL1. Tegra186 is ARMv8.0 (Denver + A57). So as written, this change breaks the very platform it intends to support.

To clarify, the code that accesses these registers is shared across Tegra186 and later chips. Tegra194 and later do support ARMv8.1 VHE.

Granted, if it always returns false on Tegra186 that's not what we want.

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Sumit can add some more details on why we don't completely disable the device for guest OSs.

It's not about disabling it. It is about correctly supporting it (providing full emulation for it), or advertising it as something different so that SW can handle it differently.

It's really not a different device. It's exactly the same device except that accessing some registers isn't permitted. We also can't easily remove parts of the register region from device tree because these are intermixed with other registers that we do want access to.

Quoted 3 lines

Poking into the internals of how the kernel is booted for a driver that isn't tied to the core architecture (because it would need to access system registers, for example) is not an acceptable outcome.

So what would be the better option? Use a different compatible string to make the driver handle the device differently? Or adding a custom property to the device tree node to mark this as running in a virtualized environment?

Perhaps we can reuse the top-level hypervisor node? That seems to only ever have been used for Xen on 32-bit ARM, so not sure if that'd still be appropriate.

Thierry

#9 lore raw
Quoted 10 lines · Thierry Reding

[1 <text/plain; UTF-8 (quoted-printable)>]

Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jon Hunter

On 06/02/2024 12:17, Marc Zyngier wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I thought this API just checks to see if we are in EL2?

It does. And that's the problem. On ARMv8.0, we run the Linux kernel at EL1. Tegra186 is ARMv8.0 (Denver + A57). So as written, this change breaks the very platform it intends to support.

To clarify, the code that accesses these registers is shared across Tegra186 and later chips. Tegra194 and later do support ARMv8.1 VHE.

But even on these machines that are VHE-capable, not running at EL2 doesn't mean we're running as a guest. The user can force the kernel to stick to EL1, using a command-line option such as kvm-arm.mode=nvhe which will force the kernel to stay at EL1 while deploying KVM at EL2.

Quoted 2 lines

Granted, if it always returns false on Tegra186 that's not what we want.

I'm glad we agree here.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Sumit can add some more details on why we don't completely disable the device for guest OSs.

Quoted 8 lines
Quoted 3 lines

It's not about disabling it. It is about correctly supporting it (providing full emulation for it), or advertising it as something different so that SW can handle it differently.

It's really not a different device. It's exactly the same device except that accessing some registers isn't permitted. We also can't easily remove parts of the register region from device tree because these are intermixed with other registers that we do want access to.

But that's the definition of being a different device. It has a different programming interface, hence it is different. The fact that it is the same HW block mediated by a hypervisor doesn't really change that.

Quoted 8 lines
Quoted 3 lines

Poking into the internals of how the kernel is booted for a driver that isn't tied to the core architecture (because it would need to access system registers, for example) is not an acceptable outcome.

So what would be the better option? Use a different compatible string to make the driver handle the device differently? Or adding a custom property to the device tree node to mark this as running in a virtualized environment?

A different compatible string would be my preferred option. An extra property would work as well. As far as I am concerned, these two options are the right way to express the fact that you have something that isn't quite like the real thing.

Quoted 3 lines

Perhaps we can reuse the top-level hypervisor node? That seems to only ever have been used for Xen on 32-bit ARM, so not sure if that'd still be appropriate.

I'd shy away from this. You would be deriving properties from a hypervisor implementation, instead of expressing those properties directly. In my experience, the direct method is always preferable.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#10 lore raw
Quoted 48 lines · Marc Zyngier
Quoted 10 lines · Thierry Reding

[1 <text/plain; UTF-8 (quoted-printable)>]

Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jon Hunter

On 06/02/2024 12:17, Marc Zyngier wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I thought this API just checks to see if we are in EL2?

It does. And that's the problem. On ARMv8.0, we run the Linux kernel at EL1. Tegra186 is ARMv8.0 (Denver + A57). So as written, this change breaks the very platform it intends to support.

To clarify, the code that accesses these registers is shared across Tegra186 and later chips. Tegra194 and later do support ARMv8.1 VHE.

But even on these machines that are VHE-capable, not running at EL2 doesn't mean we're running as a guest. The user can force the kernel to stick to EL1, using a command-line option such as kvm-arm.mode=nvhe which will force the kernel to stay at EL1 while deploying KVM at EL2.

Quoted 2 lines

Granted, if it always returns false on Tegra186 that's not what we want.

I'm glad we agree here.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Sumit can add some more details on why we don't completely disable the device for guest OSs.

Quoted 8 lines
Quoted 3 lines

It's not about disabling it. It is about correctly supporting it (providing full emulation for it), or advertising it as something different so that SW can handle it differently.

It's really not a different device. It's exactly the same device except that accessing some registers isn't permitted. We also can't easily remove parts of the register region from device tree because these are intermixed with other registers that we do want access to.

Quoted 4 lines

But that's the definition of being a different device. It has a different programming interface, hence it is different. The fact that it is the same HW block mediated by a hypervisor doesn't really change that.

The programming model isn't really different in these cases, but rather restricted. I think a compatible string is a suboptimal way to describe this.

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Poking into the internals of how the kernel is booted for a driver that isn't tied to the core architecture (because it would need to access system registers, for example) is not an acceptable outcome.

So what would be the better option? Use a different compatible string to make the driver handle the device differently? Or adding a custom property to the device tree node to mark this as running in a virtualized environment?

Quoted 4 lines

A different compatible string would be my preferred option. An extra property would work as well. As far as I am concerned, these two options are the right way to express the fact that you have something that isn't quite like the real thing.

Coincidentally there's another discussion with a lot of similarities regarding simulated platforms. For these it's usually less about the register set being restricted and more about certain quirks that are needed which will not ultimately be necessary for silicon.

This could be a timeout that's longer in simulation, or it could be certain programming that would be needed in silicon but isn't necessary or functional in simulation (think I/O calibration, that sort of thing). One could argue that these are also different devices when in simulation but they really aren't. They're more like an approximation of the actual device that will be in silicon chips.

Another problem that both of the cases have in common is that they are parameters that usually apply to the entire system. For some devices it is easier to parameterize via DT (for example for certain devices we have bindings with special register regions that are only available in host OS mode), but for others this may not be true. Adding extra compatible strings for virtualization/simulation is going to get quite complex very quickly if we need to differentiate between all of these scenarios.

Quoted 7 lines
Quoted 3 lines

Perhaps we can reuse the top-level hypervisor node? That seems to only ever have been used for Xen on 32-bit ARM, so not sure if that'd still be appropriate.

I'd shy away from this. You would be deriving properties from a hypervisor implementation, instead of expressing those properties directly. In my experience, the direct method is always preferable.

I would generally agree. However, I think especially the compatible string solution could turn very ugly for this. If we express these properties via compatible strings we may very well end up with many different compatible strings to cover all cases.

Say you've got one hypervisor that changes the programming model in a certain way and a second hypervisor that constrains in a different way. Do we now need one compatible string for each hypervisor? Do we add compatible strings for each restriction and have potentially very long compatible string lists? Separate properties would work slightly better for this.

There are some cases where we can use register contents to determine what the OS is allowed to do, but these registers don't exist for all HW blocks. We may be able to get more added to new chips, but we obviously can't retroactively add them for existing ones.

A central node or property would at least allow broad parameterization. I would hope that at least hypervisor implementations don't vary too much in terms of what they restrict and what they don't, so perhaps it wouldn't be that bad. Perhaps that's also overly optimistic.

Thierry

#11 lore raw
Quoted 51 lines · Thierry Reding

[1 <text/plain; UTF-8 (quoted-printable)>]

Quoted 48 lines · Marc Zyngier
Quoted 10 lines · Thierry Reding

[1 <text/plain; UTF-8 (quoted-printable)>]

Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jon Hunter

On 06/02/2024 12:17, Marc Zyngier wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

- My own tegra186 HW doesn't have VHE, since it is ARMv8.0, and this helper will always return 'false'. How could this result in something that still works? Can I get a free CPU upgrade?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I thought this API just checks to see if we are in EL2?

It does. And that's the problem. On ARMv8.0, we run the Linux kernel at EL1. Tegra186 is ARMv8.0 (Denver + A57). So as written, this change breaks the very platform it intends to support.

To clarify, the code that accesses these registers is shared across Tegra186 and later chips. Tegra194 and later do support ARMv8.1 VHE.

But even on these machines that are VHE-capable, not running at EL2 doesn't mean we're running as a guest. The user can force the kernel to stick to EL1, using a command-line option such as kvm-arm.mode=nvhe which will force the kernel to stay at EL1 while deploying KVM at EL2.

Quoted 2 lines

Granted, if it always returns false on Tegra186 that's not what we want.

I'm glad we agree here.

Quoted 16 lines
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

- If you assign this device to a VM and that the hypervisor doesn't correctly virtualise it, then it is a different device and you should simply advertise it something else. Or even better, fix your hypervisor.

Sumit can add some more details on why we don't completely disable the device for guest OSs.

It's not about disabling it. It is about correctly supporting it (providing full emulation for it), or advertising it as something different so that SW can handle it differently.

It's really not a different device. It's exactly the same device except that accessing some registers isn't permitted. We also can't easily remove parts of the register region from device tree because these are intermixed with other registers that we do want access to.

Quoted 8 lines
Quoted 4 lines

But that's the definition of being a different device. It has a different programming interface, hence it is different. The fact that it is the same HW block mediated by a hypervisor doesn't really change that.

The programming model isn't really different in these cases, but rather restricted. I think a compatible string is a suboptimal way to describe this.

It *is* different. If it wasn't different, you wouldn't need this patch. I'm puzzled that we have to argue on *that*. You can call it restricted, I call it broken. In both case, it is a *different* programming interface as you can't use existing SW for it.

Quoted 20 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

Poking into the internals of how the kernel is booted for a driver that isn't tied to the core architecture (because it would need to access system registers, for example) is not an acceptable outcome.

So what would be the better option? Use a different compatible string to make the driver handle the device differently? Or adding a custom property to the device tree node to mark this as running in a virtualized environment?

Quoted 4 lines

A different compatible string would be my preferred option. An extra property would work as well. As far as I am concerned, these two options are the right way to express the fact that you have something that isn't quite like the real thing.

Coincidentally there's another discussion with a lot of similarities regarding simulated platforms. For these it's usually less about the register set being restricted and more about certain quirks that are needed which will not ultimately be necessary for silicon.

Quoted 6 lines

This could be a timeout that's longer in simulation, or it could be certain programming that would be needed in silicon but isn't necessary or functional in simulation (think I/O calibration, that sort of thing). One could argue that these are also different devices when in simulation but they really aren't. They're more like an approximation of the actual device that will be in silicon chips.

Simulation/DV environments are a very different kettle of fish. You generally treat passing time with a scaling factor, and you are likely to run very hacked-up SW stack anyway.

In any case, this is not relevant to upstream stuff, unless you plan to ship your emulation environment.

Quoted 8 lines

Another problem that both of the cases have in common is that they are parameters that usually apply to the entire system. For some devices it is easier to parameterize via DT (for example for certain devices we have bindings with special register regions that are only available in host OS mode), but for others this may not be true. Adding extra compatible strings for virtualization/simulation is going to get quite complex very quickly if we need to differentiate between all of these scenarios.

That's the price you pay for these inconsistencies. If your "HW" has a lot of variability and that you can't discover its capabilities from SW, then it either badly designed, badly implemented, badly emulated, or any combination thereof.

In any case, you get to keep the pieces.

Quoted 14 lines
Quoted 8 lines
Quoted 4 lines

Perhaps we can reuse the top-level hypervisor node? That seems to only ever have been used for Xen on 32-bit ARM, so not sure if that'd still be appropriate.

I'd shy away from this. You would be deriving properties from a hypervisor implementation, instead of expressing those properties directly. In my experience, the direct method is always preferable.

I would generally agree. However, I think especially the compatible string solution could turn very ugly for this. If we express these properties via compatible strings we may very well end up with many different compatible strings to cover all cases.

Quoted 6 lines

Say you've got one hypervisor that changes the programming model in a certain way and a second hypervisor that constrains in a different way. Do we now need one compatible string for each hypervisor? Do we add compatible strings for each restriction and have potentially very long compatible string lists? Separate properties would work slightly better for this.

Again, the job of a hypervisor is to offer an architecturally correct view of some HW, emulated or not. If your hypervisors are implementing a large variety of diverging behaviours, SW needs to be able to distinguish between those. You can either add properties, use compat strings, or use a discovery protocol implemented by the device.

In any case, each deviation needs to be uniquely identifiable, and be described either in FW or by the device itself, if only because Linux isn't the only game in town.

Quoted 9 lines

There are some cases where we can use register contents to determine what the OS is allowed to do, but these registers don't exist for all HW blocks. We may be able to get more added to new chips, but we obviously can't retroactively add them for existing ones.

A central node or property would at least allow broad parameterization. I would hope that at least hypervisor implementations don't vary too much in terms of what they restrict and what they don't, so perhaps it wouldn't be that bad. Perhaps that's also overly optimistic.

Top level properties are no good unless what they express is forever immutable and described upfront. Identifying a hypervisor doesn't do that, and most of the time there will be all sorts of *variable* properties that need to be further discovered by a mechanism or another. In my (surely very limited) experience at writing hypervisors for some time, this eventually becomes an unmaintainable mess.

You are of course free to do that in the drivers you maintain as long as you don't break my own toys, but I'd urge you to reconsider this and explore other possibilities.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.