Skip to contentSkip to conversation

Exposing host debug capabilities to userspace

DISCUSSION Alex Bennée 5 participants 26 messages Nov 20, 2014 – Nov 26, 2014 1 message not in the archive KVM

lore
Share & tools
lore
#0 lore raw

Hi,

I've almost finished the ARMv8 guest debug support but I have one problem left to solve. userspace needs to know how many hardware debug registers are available for GDB to use. This information is available from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to fetch it's value however semantically this is poor as it's API is for getting guest state not host state and they could theoretically have different values.

So far the options I've examined are:

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

As explained above, abusing a guest state API for host configuration.

* ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)

This is used by GDB to access the host details in debug-monitors. However the ptrace API really wants you to attach to a process before calling PTRACE_GETREGSET. Currently I've tried attaching to the thread_id of the vCPU but this fails with EPERM, I suspect because attaching to your own threads likely upsets the kernel.

* KVM ioctl KVM_GET_DEBUGREGS

This is currently x86 only and looks like it's more aimed at debug registers than capability stuff. Also I'm not sure what the state of this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or is one an older deprecated x86 only API?

* Export the information via sysfs

I suppose the correct canonical non-subsystem specific way to make this information available it to expose the data in some sort of sysfs node? However I don't see any existing sysfs structure for the CPU.

* Expand /proc/cpuinfo

I suspect adding extra text to be badly parsed by userspace is just horrid and unacceptable behaviour ;-)

* Add another KVM ioctl?

This would have the downside of being specific to KVM and of course proliferating the API space again.

I'm open to any suggestions and look forward to your valued feedback ;-)

Signature · 1 line
Alex Bennée
#1 lore raw
Quoted 10 lines · Alex Bennée

Hi,

I've almost finished the ARMv8 guest debug support but I have one problem left to solve. userspace needs to know how many hardware debug registers are available for GDB to use. This information is available from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to fetch it's value however semantically this is poor as it's API is for getting guest state not host state and they could theoretically have different values.

Quoted 5 lines

So far the options I've examined are:

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

As explained above, abusing a guest state API for host configuration.

It's just wrong, and we should only do this if there's absolutely no other way to do this.

Quoted 8 lines

* ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)

This is used by GDB to access the host details in debug-monitors. However the ptrace API really wants you to attach to a process before calling PTRACE_GETREGSET. Currently I've tried attaching to the thread_id of the vCPU but this fails with EPERM, I suspect because attaching to your own threads likely upsets the kernel.

Can you confirm your suspicion? This seems like a rather good approach so we should really investigate why this doesn't work and explore ways to get it working.

Quoted 7 lines

* KVM ioctl KVM_GET_DEBUGREGS

This is currently x86 only and looks like it's more aimed at debug registers than capability stuff. Also I'm not sure what the state of this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or is one an older deprecated x86 only API?

The API text and a brief glance of the x86 code seems to indicate that this is also the vcpu state...

Quoted 9 lines

* Export the information via sysfs

I suppose the correct canonical non-subsystem specific way to make this information available it to expose the data in some sort of sysfs node? However I don't see any existing sysfs structure for the CPU.

* Expand /proc/cpuinfo

Quoted 8 lines

I suspect adding extra text to be badly parsed by userspace is just horrid and unacceptable behaviour ;-)

* Add another KVM ioctl?

This would have the downside of being specific to KVM and of course proliferating the API space again.

This may not be that bad, for example, could we ever imaging that we'd only want to export a few of the debug registers for host gdbstub usage?

-Christoffer

#2 lore raw
Quoted 8 lines · Christoffer Dall
Quoted 7 lines · Alex Bennée

Hi,

I've almost finished the ARMv8 guest debug support but I have one problem left to solve. userspace needs to know how many hardware debug registers are available for GDB to use.
* KVM ioctl KVM_GET_DEBUGREGS

Quoted 7 lines
Quoted 4 lines

This is currently x86 only and looks like it's more aimed at debug registers than capability stuff. Also I'm not sure what the state of this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or is one an older deprecated x86 only API?

The API text and a brief glance of the x86 code seems to indicate that this is also the vcpu state...

Yeah I was getting confused as to the difference between the two API calls. Is this just an x86 version of what GET/SET_ONE_REG replaced?

Quoted 8 lines
Quoted 5 lines

* Add another KVM ioctl?

This would have the downside of being specific to KVM and of course proliferating the API space again.

This may not be that bad, for example, could we ever imaging that we'd only want to export a few of the debug registers for host gdbstub usage?

However it is general information which might be useful to the whole system (although I suspect KVM and PTRACE are the only two). It would be a shame to have an informational API wrapped up in the extra boiler-plate of a specific API.

Signature · 1 line
Alex Bennée
#3 lore raw
In reply to Christoffer Dall
Quoted 1 line · Christoffer Dall

On Thu, Nov 20, 2014 at 04:55:14PM +0000, Alex Bennée wrote:

<snip>

Quoted 9 lines
Quoted 8 lines

* ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)

This is used by GDB to access the host details in debug-monitors. However the ptrace API really wants you to attach to a process before calling PTRACE_GETREGSET. Currently I've tried attaching to the thread_id of the vCPU but this fails with EPERM, I suspect because attaching to your own threads likely upsets the kernel.

Quoted 3 lines

Can you confirm your suspicion? This seems like a rather good approach so we should really investigate why this doesn't work and explore ways to get it working.

From ptrace_attach:

	retval = -EPERM;
	if (unlikely(task->flags & PF_KTHREAD))
		goto out;
	if (same_thread_group(task, current))
		goto out;

I think this is what is triggering my EPERM. I'm going to dig into the history of code around that bit. While I can see it might be undesirable I'm not sure if it has to be verbotten...

Signature · 1 line
Alex Bennée

A message that is not in the archive (sent off-list or not yet received): 87egssn91o.fsf@zen.linaro.local.i-did-not-set--mail-host-address--so-tickle-me

#5 lore raw

On 24 November 2014 at 11:16, Alex Bennée <alex@zen.linaro.local.i-did-not-set--mail-host-address--so-tickle-me> wrote:

^^^ :-)

Quoted 8 lines
Quoted 6 lines · Alex Bennée

* KVM ioctl KVM_GET_DEBUGREGS

This is currently x86 only and looks like it's more aimed at debug registers than capability stuff. Also I'm not sure what the state of this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or is one an older deprecated x86 only API?

Quoted 4 lines

I'm minded to re-use this ioctl and define it for ARM as reading the host debug architecture state ID_AA64DFR0/1_EL1. Currently for x86 it's used for getting vcpu debug registers which on ARM is handled via the GET/SET one reg interface.

This seems a bit odd. Either the x86 use of this ioctl is for accessing guest state, in which case using it on ARM for host state is a bit weird, or else why is x86 doing its debug via host state and ARM using guest state?

It may well still be the best choice, but it just feels like maybe something isn't lined up right...

-- PMM

#6 lore raw
Quoted 19 lines · Peter Maydell

On 24 November 2014 at 11:16, Alex Bennée <alex@zen.linaro.local.i-did-not-set--mail-host-address--so-tickle-me> wrote:

^^^ :-)

Quoted 8 lines
Quoted 6 lines · Alex Bennée

* KVM ioctl KVM_GET_DEBUGREGS

This is currently x86 only and looks like it's more aimed at debug registers than capability stuff. Also I'm not sure what the state of this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or is one an older deprecated x86 only API?

Quoted 4 lines

I'm minded to re-use this ioctl and define it for ARM as reading the host debug architecture state ID_AA64DFR0/1_EL1. Currently for x86 it's used for getting vcpu debug registers which on ARM is handled via the GET/SET one reg interface.

Quoted 8 lines

This seems a bit odd. Either the x86 use of this ioctl is for accessing guest state, in which case using it on ARM for host state is a bit weird, or else why is x86 doing its debug via host state and ARM using guest state?

It may well still be the best choice, but it just feels like maybe something isn't lined up right...

It seems weird, agreed, but somehow what we're left with except for adding a new ioctl. I think part of the explanation may simply be that x86 solves this problem in an inherently different way.

-Christoffer

#7 lore raw
In reply to a message not in the archive

Fixed CC:kvmarm, Added: Alexander Graf, Fixed: my From:

Replying to myself with additional information on each option

Quoted 10 lines · Alex Bennée

Hi,

I've almost finished the ARMv8 guest debug support but I have one problem left to solve. userspace needs to know how many hardware debug registers are available for GDB to use. This information is available from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to fetch it's value however semantically this is poor as it's API is for getting guest state not host state and they could theoretically have different values.

Quoted 3 lines

So far the options I've examined are:

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

Nope, guest state API

Quoted 1 line

* ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)

Nope, ptrace requires attachment and you can't attach to your own thread group.

Quoted 6 lines

* KVM ioctl KVM_GET_DEBUGREGS

This is currently x86 only and looks like it's more aimed at debug registers than capability stuff. Also I'm not sure what the state of this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or is one an older deprecated x86 only API?

I'm minded to re-use this ioctl and define it for ARM as reading the host debug architecture state ID_AA64DFR0/1_EL1. Currently for x86 it's used for getting vcpu debug registers which on ARM is handled via the GET/SET one reg interface.

Quoted 5 lines

* Export the information via sysfs

I suppose the correct canonical non-subsystem specific way to make this information available it to expose the data in some sort of sysfs node? However I don't see any existing sysfs structure for the CPU.

I suspect this would get complicated depending on the architecture.

Quoted 9 lines

* Expand /proc/cpuinfo

I suspect adding extra text to be badly parsed by userspace is just horrid and unacceptable behaviour ;-)

* Add another KVM ioctl?

This would have the downside of being specific to KVM and of course proliferating the API space again.

So unless there are any objections my intention to re-use the existing API calls for ARM architectures.

Signature · 1 line
Alex Bennée
#8 lore raw
Quoted 17 lines · Alex Bennée

Fixed CC:kvmarm, Added: Alexander Graf, Fixed: my From:

Replying to myself with additional information on each option

Quoted 10 lines · Alex Bennée

Hi,

I've almost finished the ARMv8 guest debug support but I have one problem left to solve. userspace needs to know how many hardware debug registers are available for GDB to use. This information is available from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to fetch it's value however semantically this is poor as it's API is for getting guest state not host state and they could theoretically have different values.

Quoted 4 lines
Quoted 3 lines

So far the options I've examined are:

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

Nope, guest state API

What's the problem with using ONE_REG for this? After all, the total number of available guest debug register is a guest vcpu property of some sort.

Btw, looking through the mess that we have today with asymmetric SET_GUEST_DEBUG and GET_DEBUGREGS ioctls I can't shake off the feeling that we're best off just doing all of the debug register sync via ONE_REGs. That way we at least have a guaranteed symmetric interface that can get and set values, so we won't trip over it on live migration.

Alex

#9 lore raw
Quoted 7 lines · Alexander Graf
Quoted 2 lines · Alex Bennée
Quoted 1 line

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

Nope, guest state API

What's the problem with using ONE_REG for this? After all, the total number of available guest debug register is a guest vcpu property of some sort.

Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

-- PMM

#10 lore raw
Quoted 14 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:

Quoted 8 lines · Alexander Graf
Quoted 2 lines · Alex Bennée
Quoted 1 line

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

Nope, guest state API

What's the problem with using ONE_REG for this? After all, the total number of available guest debug register is a guest vcpu property of some sort.

Quoted 5 lines

Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

Alex

Quoted 2 lines

-- PMM

#11 lore raw
Quoted 9 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

-- PMM

#12 lore raw
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if that's true, but assume it is). Tomorrow there will be a new core - let's call it A67 - with 16 debug registers.

To make sure your legacy, badly written guest behaves exactly the same - especially after live migration - you want to spawn a VM with -cpu A57. That implies you want to expose 8 debug registers into the guest. So debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available to a guest vcpu. That in turn means it's guest state and as such can easily go into ONE_REG.

Alex

#13 lore raw
Quoted 26 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if that's true, but assume it is). Tomorrow there will be a new core - let's call it A67 - with 16 debug registers.

Quoted 9 lines

To make sure your legacy, badly written guest behaves exactly the same - especially after live migration - you want to spawn a VM with -cpu A57. That implies you want to expose 8 debug registers into the guest. So debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available to a guest vcpu. That in turn means it's guest state and as such can easily go into ONE_REG.

We already export this for the guest via ONE_REG.

What we want to do is support gdbstubs in QEMU to debug the guest, and to do this, QEMU needs to know how many hardware registers on the host there is; the guest will never see this information.

So this is really about the host, the guest side is trivially handled through ONE_REG.

-Christoffer

#14 lore raw
Quoted 36 lines · Christoffer Dall
Quoted 26 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if that's true, but assume it is). Tomorrow there will be a new core - let's call it A67 - with 16 debug registers.

Quoted 9 lines

To make sure your legacy, badly written guest behaves exactly the same - especially after live migration - you want to spawn a VM with -cpu A57. That implies you want to expose 8 debug registers into the guest. So debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available to a guest vcpu. That in turn means it's guest state and as such can easily go into ONE_REG.

Quoted 8 lines

We already export this for the guest via ONE_REG.

What we want to do is support gdbstubs in QEMU to debug the guest, and to do this, QEMU needs to know how many hardware registers on the host there is; the guest will never see this information.

So this is really about the host, the guest side is trivially handled through ONE_REG.

That's the cp15 register that happens to get exposed to the guest. You can just add another ONE_REG that does not have a cp15 equivalent to expose the number of the vcpu's actually available debug registers.

Alex

#15 lore raw
Quoted 45 lines · Alexander Graf
Quoted 42 lines · Christoffer Dall
Quoted 26 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if that's true, but assume it is). Tomorrow there will be a new core - let's call it A67 - with 16 debug registers.

Quoted 9 lines

To make sure your legacy, badly written guest behaves exactly the same - especially after live migration - you want to spawn a VM with -cpu A57. That implies you want to expose 8 debug registers into the guest. So debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available to a guest vcpu. That in turn means it's guest state and as such can easily go into ONE_REG.

We already export this for the guest via ONE_REG.

What we want to do is support gdbstubs in QEMU to debug the guest, and to do this, QEMU needs to know how many hardware registers on the host there is; the guest will never see this information.

Quoted 8 lines
Quoted 2 lines

So this is really about the host, the guest side is trivially handled through ONE_REG.

That's the cp15 register that happens to get exposed to the guest. You can just add another ONE_REG that does not have a cp15 equivalent to expose the number of the vcpu's actually available debug registers.

The fact that we currently map the guest vcpu registers to that of the host doesn't mean that will always be the case (which you argued for above).

If you migrate a VM from a CPU with 16 debug registers (or emulate a CPU with 16 debug registers) on a physical CPU with only 8 debug registers, you cannot tell QEMU that it has 16 debug registers on the hardware to configure to debug the guest, which is what ONE_REG would give you.

You could add another set of registers to ONE_REG which says "these are the host versions", or you could say that you don't ever support a setup where the guest number of debug registers is not the same as the host number, but both would be wrong.

-Christoffer

#16 lore raw
Quoted 64 lines · Christoffer Dall
Quoted 45 lines · Alexander Graf
Quoted 42 lines · Christoffer Dall
Quoted 26 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if that's true, but assume it is). Tomorrow there will be a new core - let's call it A67 - with 16 debug registers.

Quoted 9 lines

To make sure your legacy, badly written guest behaves exactly the same - especially after live migration - you want to spawn a VM with -cpu A57. That implies you want to expose 8 debug registers into the guest. So debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available to a guest vcpu. That in turn means it's guest state and as such can easily go into ONE_REG.

We already export this for the guest via ONE_REG.

What we want to do is support gdbstubs in QEMU to debug the guest, and to do this, QEMU needs to know how many hardware registers on the host there is; the guest will never see this information.

Quoted 8 lines
Quoted 2 lines

So this is really about the host, the guest side is trivially handled through ONE_REG.

That's the cp15 register that happens to get exposed to the guest. You can just add another ONE_REG that does not have a cp15 equivalent to expose the number of the vcpu's actually available debug registers.

The fact that we currently map the guest vcpu registers to that of the host doesn't mean that will always be the case (which you argued for above).

If you migrate a VM from a CPU with 16 debug registers (or emulate a CPU with 16 debug registers) on a physical CPU with only 8 debug registers, you cannot tell QEMU that it has 16 debug registers on the hardware to configure to debug the guest, which is what ONE_REG would give you.

Quoted 4 lines

You could add another set of registers to ONE_REG which says "these are the host versions", or you could say that you don't ever support a setup where the guest number of debug registers is not the same as the host number, but both would be wrong.

Why would QEMU ever want to access debug registers that it didn't ask for in the first place? If we simply limit ourselves to the register set the vcpu has we don't have any problems and the complexity matrix shrinks noticably, no?

Alex

#17 lore raw
Quoted 72 lines · Alexander Graf
Quoted 64 lines · Christoffer Dall
Quoted 45 lines · Alexander Graf
Quoted 42 lines · Christoffer Dall
Quoted 26 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf
Quoted 6 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:
Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register by definition is how we tell the guest how many debug registers it has. What we want to know (and perhaps even control) for debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if that's true, but assume it is). Tomorrow there will be a new core - let's call it A67 - with 16 debug registers.

Quoted 9 lines

To make sure your legacy, badly written guest behaves exactly the same - especially after live migration - you want to spawn a VM with -cpu A57. That implies you want to expose 8 debug registers into the guest. So debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available to a guest vcpu. That in turn means it's guest state and as such can easily go into ONE_REG.

We already export this for the guest via ONE_REG.

What we want to do is support gdbstubs in QEMU to debug the guest, and to do this, QEMU needs to know how many hardware registers on the host there is; the guest will never see this information.

Quoted 8 lines
Quoted 2 lines

So this is really about the host, the guest side is trivially handled through ONE_REG.

That's the cp15 register that happens to get exposed to the guest. You can just add another ONE_REG that does not have a cp15 equivalent to expose the number of the vcpu's actually available debug registers.

The fact that we currently map the guest vcpu registers to that of the host doesn't mean that will always be the case (which you argued for above).

If you migrate a VM from a CPU with 16 debug registers (or emulate a CPU with 16 debug registers) on a physical CPU with only 8 debug registers, you cannot tell QEMU that it has 16 debug registers on the hardware to configure to debug the guest, which is what ONE_REG would give you.

Quoted 4 lines

You could add another set of registers to ONE_REG which says "these are the host versions", or you could say that you don't ever support a setup where the guest number of debug registers is not the same as the host number, but both would be wrong.

Quoted 5 lines

Why would QEMU ever want to access debug registers that it didn't ask for in the first place? If we simply limit ourselves to the register set the vcpu has we don't have any problems and the complexity matrix shrinks noticably, no?

(For others following this, not hanging out on #kvm-arm, I provide a summary).

We had a lenghty IRC discussion on this, for the curious, go read it here:
http://irclogs.linaro.org/2014/11/24/%23kvm-arm.html

The point of confusion is that other KVM architectures use the ONE_REG interface to set the debug registers, for both guest and host state, and in fact this can be overlayed. The propose idea of using SET_DEBUGREGS was flawed, because this also pertains to guest state and is a deprecated ABI.

So that left us with two choices:

(1) Implement a new ABI to retrieve and set host debugging independently of the guest state.

(2) Overlay the host debugging of a guest with the guest's internal debugging state (a process inside the guest is debugging a process).

Option (1) has the usual drawback of having to design an ABI, consumes the ioctl number space etc. But it also has (the more severe) drawback that debugging the guest using QEMU gdbstubs breaks guest debugging, because we would keep completely distinct register sets.

Option (2) feels semantically slightly weird in the cases where the number of guest debug regs differs from the number of host debug regs, but as Alex Graf pointed out, we probably wouldn't dream of supporting more guest debug registers than the host has (refuse migration, don't emulate that VCPU on the PCPU etc.), and we don't care deeply about exposing fewer host debug regs than actually available. Migrating a VM that is being debugged by the host is simply not supported, so any state obtained via ONE_REG for migration is for the guest, not the host.

Option (2) has the additional benefit that we can intermix guest and host registers; debug exceptions are in this case always routed to user space and QEMU's magic determines if the exception is for itself or if it should be injected back into the guest. Guest accesses to debug regs are always trapped when the host uses any debug regs, thus not exposing any sensitive state to the guest.

Given this information, I personally lean towards approach (2) since it is similar to what is done for other architectures and caters to the most probable use cases that we would care about.

Feel free to point out omissions or errors in my reasoning.

-Christoffer

maintainer
#18 lore raw
Quoted 4 lines · Christoffer Dall

We had a lenghty IRC discussion on this, for the curious, go read it here:
http://irclogs.linaro.org/2014/11/24/%23kvm-arm.html

Some notes...

Quoted 6 lines

chazy but saying that you want to design an ABI that potentially exposes fewer debug registers to gdbstubs than your hardware supports and breaks gdbstub support if you happen to support emulation of a cpu with more debug registers in the future, because you want to re-use ONE_REG is not a great approach either

agraf chazy: but you wouldn't even remotely think of doing it, no?

agraf chazy: I'm saying that QEMU shouldn't have access to the excess registers

Quoted 7 lines

chazy agraf: I wouldn’t even remotely think of doing nested virtualization, but people have

agraf chazy: QEMU spawns an A57, it gets 8 debug registers

chazy agraf: I understand, but I don’t agree with your rationale

agraf chazy: not "QEMU spawns an A57 on an A67, so it gets 8 real and 8 hidden debug registers that it also needs to be aware of mappings of"

I disagree with Alex. QEMU can use the 16 debug registers as it wishes, since it sets them with KVM_SET_GUEST_DEBUG. The guest's eight can be mapped to the last 8 if those are the required semantics for the architecture. Just exit to QEMU if you do a debug register write while gdbstub debugging is active; QEMU gets the contents of the 8 VCPU-visible registers with ONE_REG (equivalent of x86 GET_DEBUGREGS); calls KVM_SET_GUEST_DEBUG to reflect them in the 16-register state; and restarts.

It works as long as you can trap both reads and writes.

Quoted 6 lines

chazy ajb-linaro: don’t migrate a guest that is getting debugged is the answer to that one I believe

ajb-linaro chazy: at least with the overlay approach that happens automatically

ajb-linaro the overlay isn't migrated and the new host isn't calling SET_GUEST_DEBUG so whatever the debug registers where before are restorerd

Right. The destination will lose the hardware breakpoints/watchpoints because it starts the guest with KVM_SET_GUEST_DEBUG. Apart from losing them, everything should work fine. The dest sets the 8 VCPU-visible registers with ONE_REG, the hypervisor reflects them in a subset of the 16 physical registers, and "that's it".

Paolo

#19 lore raw
In reply to Alexander Graf
Quoted 11 lines · Alexander Graf
Quoted 11 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:

Quoted 8 lines · Alexander Graf
Quoted 2 lines · Alex Bennée
Quoted 1 line

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

Nope, guest state API

What's the problem with using ONE_REG for this? After all, the total number of available guest debug register is a guest vcpu property of some sort.

Quoted 8 lines
Quoted 5 lines

Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

Does the GET/SET_ONE_REG have the concept of a read-only register? I'd be worried by code just blindly iterating over the lists getting thrown when one doesn't work.

Quoted 5 lines

Alex

Quoted 2 lines

-- PMM

Signature · 1 line
Alex Bennée
#20 lore raw
Quoted 20 lines · Alex Bennée
Quoted 17 lines · Alexander Graf
Quoted 16 lines

Am 24.11.2014 um 13:32 schrieb Peter Maydell <[email protected]>:

Quoted 8 lines · Alexander Graf
Quoted 2 lines · Alex Bennée
Quoted 1 line

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

Nope, guest state API

What's the problem with using ONE_REG for this? After all, the total number of available guest debug register is a guest vcpu property of some sort.

Yes, but we don't want to know about properties of the guest vCPU. In an ideal world QEMU could reserve say half the debug registers for debugging the VM on startup and have KVM expose ID registers indicating to the guest that it only had the other half...

Quoted 7 lines
Quoted 2 lines

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

Does the GET/SET_ONE_REG have the concept of a read-only register? I'd be worried by code just blindly iterating over the lists getting thrown when one doesn't work.

yes, we have invariant cp15 registers.

-Christoffer

#21 lore raw
In reply to Alex Bennée

Alex Bennée <[email protected]> writes:

<snip>

Quoted 8 lines
Quoted 6 lines · Alex Bennée

Hi,

I've almost finished the ARMv8 guest debug support but I have one problem left to solve. userspace needs to know how many hardware debug registers are available for GDB to use. This information is available from the ID_AA64DFR0_EL1 register.

<Snip>

Quoted 8 lines
Quoted 8 lines

So far the options I've examined are:

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
* ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)
* KVM ioctl KVM_GET_DEBUGREGS
* Export the information via sysfs
* Expand /proc/cpuinfo
* Add another KVM ioctl?

Alexander Graf pointed out that KVM_CHECK_EXTENSION can return any positive number for success. How about using:

max_hw_bps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS); max_hw_wps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);

Seems pretty sane, doesn't change the semantics of an API and is architecture agnostic if others need the number?

Signature · 1 line
Alex Bennée
maintainer
#22 lore raw
Quoted 8 lines · Alex Bennée

Alexander Graf pointed out that KVM_CHECK_EXTENSION can return any positive number for success. How about using:

max_hw_bps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS); max_hw_wps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);

Seems pretty sane, doesn't change the semantics of an API and is architecture agnostic if others need the number?

Yes, this was going to be my suggestion as well. Just I would use a bitmask in case some register can act as both breakpoint and watchpoint.

On x86, each of the four bp/wp registers (each register can act as both) can be used for either guest or gdbstub debugging. If the KVM_GUESTDBG_USE_HW_BP feature is enabled, the guest is entered with "made up" debug register contents, that we pass via KVM_SET_GUEST_DEBUG's struct kvm_guest_debug_arch. Otherwise, the guest is entered with real debug register contents passed via KVM_SET_DEBUGREGS. Reads/writes of the debug registers trap to KVM (which helps the guest see the expected values of the debug registers in the former case). There is no KVM_GET_GUEST_DEBUG because the corresponding info is passed via struct kvm_debug_exit_arch.

If gdbstub hardware breakpoints are enabled, all hardware breakpoints exit to userspace. QEMU then decides whether the breakpoint came from guest debugging (and then injects an exception), or from gdbstub debugging (and then suspends execution). Same for software breakpoints.
If the total request is >4 hardware breakpoints, someone will have to
lose and some gdbstub breakpoints will be missed.

Paolo

#23 lore raw
Quoted 26 lines · Paolo Bonzini
Quoted 8 lines · Alex Bennée

Alexander Graf pointed out that KVM_CHECK_EXTENSION can return any positive number for success. How about using:

max_hw_bps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS); max_hw_wps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);

Seems pretty sane, doesn't change the semantics of an API and is architecture agnostic if others need the number?

Yes, this was going to be my suggestion as well. Just I would use a bitmask in case some register can act as both breakpoint and watchpoint.

On x86, each of the four bp/wp registers (each register can act as both) can be used for either guest or gdbstub debugging. If the KVM_GUESTDBG_USE_HW_BP feature is enabled, the guest is entered with "made up" debug register contents, that we pass via KVM_SET_GUEST_DEBUG's struct kvm_guest_debug_arch. Otherwise, the guest is entered with real debug register contents passed via KVM_SET_DEBUGREGS. Reads/writes of the debug registers trap to KVM (which helps the guest see the expected values of the debug registers in the former case). There is no KVM_GET_GUEST_DEBUG because the corresponding info is passed via struct kvm_debug_exit_arch.

Quoted 6 lines

If gdbstub hardware breakpoints are enabled, all hardware breakpoints exit to userspace. QEMU then decides whether the breakpoint came from guest debugging (and then injects an exception), or from gdbstub debugging (and then suspends execution). Same for software breakpoints.
If the total request is >4 hardware breakpoints, someone will have to
lose and some gdbstub breakpoints will be missed.

Unfortunately on ARM you also have a few other constraints - the debug register space is partitioned into magic super debug registers at the top (with an implementation specific amount) and normal debug registers in the lower end of the space.

The main pain I have with exposing host information is that it's going to be interesting and challenging enough to get all of this right merely for the guest debug register space. Exposing the host debug register space as well means there is even more space for breakage.

I would just treat gdbstub debugging as the ugly step child that it is and not introduce anything special for it (except for debug event deflection to user space). Then only ever work on guest debug registers and call it a day. Chances are just too high that we get the interfaces wrong and shoot ourselves in the foot.

Alex

maintainer
#24 lore raw
Quoted 4 lines · Alexander Graf

Unfortunately on ARM you also have a few other constraints - the debug register space is partitioned into magic super debug registers at the top (with an implementation specific amount) and normal debug registers in the lower end of the space.

Does the gdbstub ever need to use the magic super debug registers? Even if it does, the logic is the same as x86. On x86 you might run out of breakpoints, on ARM you might run out of breakpoints in general or magic super breakpoints in particular.

Quoted 5 lines

I would just treat gdbstub debugging as the ugly step child that it is and not introduce anything special for it (except for debug event deflection to user space). Then only ever work on guest debug registers and call it a day. Chances are just too high that we get the interfaces wrong and shoot ourselves in the foot.

I agree. But we do need a way to retrieve the number of valid fields in struct kvm_guest_debug_arch, if that is not architecturally defined (e.g. on x86 it's just always 4). A "hidden" ONE_REG, whose existence is determined by (ARM || ARM64) && kvm_check_extension(KVM_CAP_SET_GUEST_DEBUG), is certainly fine and I like it because it doesn't pollute the global KVM_CAP_* namespace.

The alternative is a capability; however, if you start with two capabilities you'll end up needing four, and then realize that returning a struct via ONE_REG would have been a better idea. :) We already have how many breakpoints, how many watchpoints, how many magic super debug registers,...

Paolo

#25 lore raw
Quoted 10 lines · Paolo Bonzini
Quoted 4 lines · Alexander Graf

Unfortunately on ARM you also have a few other constraints - the debug register space is partitioned into magic super debug registers at the top (with an implementation specific amount) and normal debug registers in the lower end of the space.

Does the gdbstub ever need to use the magic super debug registers? Even if it does, the logic is the same as x86. On x86 you might run out of breakpoints, on ARM you might run out of breakpoints in general or magic super breakpoints in particular.

By default gdbstub uses normal software breakpoints unless a) the user asks for a HW one or b) the memory is RO (i.e a ROM). For watchpoints the situation is reversed and they are used by default if available.

Quoted 9 lines
Quoted 5 lines

I would just treat gdbstub debugging as the ugly step child that it is and not introduce anything special for it (except for debug event deflection to user space). Then only ever work on guest debug registers and call it a day. Chances are just too high that we get the interfaces wrong and shoot ourselves in the foot.

I agree. But we do need a way to retrieve the number of valid fields in struct kvm_guest_debug_arch, if that is not architecturally defined (e.g. on x86 it's just always 4).

It's IMPDEF (implementation defined) up to a maximum of 16.

Quoted 4 lines

A "hidden" ONE_REG, whose existence
is determined by (ARM || ARM64) && kvm_check_extension(KVM_CAP_SET_GUEST_DEBUG), is certainly fine and I like it because it doesn't pollute the global KVM_CAP_* namespace.

My original implementation more or less did that but my main worry is migration code tends to iterate over the whole ONE_REG list and introducing a value that doesn't happen to be strictly the guests is semantically impure. Of course if you mean by "hidden" don't export it via GET_REG_LIST then I guess that would work. Does seem a little ugly as the internal KVM code now needs to learn about hidden and non-hidden registers.

Quoted 5 lines

The alternative is a capability; however, if you start with two capabilities you'll end up needing four, and then realize that returning a struct via ONE_REG would have been a better idea. :) We already have how many breakpoints, how many watchpoints, how many magic super debug registers,...

Only two, not sure what the super debug registers are?

Certainly on the QEMU side the capability based approach is beautifully simple:

    max_hw_wp = kvm_check_extension(cs->kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);
    max_hw_bp = kvm_check_extension(cs->kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS);

thanks to kvm_check_extension zeroing failure modes.

And in it's defence it's a generic enough capability to be used across any other architectures that need to expose this information compared to the many PPC specific capabilities. Perhaps a KVM_ARCH_EXTENSION ioctl would be more useful in reducing the growth of the space?

Quoted 2 lines

Paolo

Signature · 1 line
Alex Bennée
#26 lore raw
In reply to Paolo Bonzini
Quoted 9 lines · Paolo Bonzini
Quoted 4 lines · Alexander Graf

Unfortunately on ARM you also have a few other constraints - the debug register space is partitioned into magic super debug registers at the top (with an implementation specific amount) and normal debug registers in the lower end of the space.

Does the gdbstub ever need to use the magic super debug registers?

No. The extra features of the trailing registers in the range are "context matching", which means you can set them up as "breakpoint on this context ID" or "breakpoint on this VMID" [read: breakpoints specific to a particular process ID or particular VM], generally as linked breakpoints adding a constraint to a plain address-match breakpoint register. The gdbstub doesn't use context bps and I don't think Linux does either (though of course it might in future).

-- PMM