RISC-V: support for vector register accesses via ptrace() in RISC-V Linux native
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CADdv1FqjLPZ-eWOKPv0uZxF-u-SYjn0WJGr3KWW9H06-O0L35w@mail.gmail.com
Quoted 10 lines · Greg Savin
The SIGILL guard is being used as a wrapper around determination of the VLENB CSR, which is not part of the
ptrace()payload for vector registers,
at least as it exists at head-of-tree Linux kernel. GDB or gdbserver
needs to know VLENB in order to construct the architectural feature metadata that reports an accurate width for the vector registers. If not for the VLENB determination specifically, and the lack of this information viaptrace(), then there would be no motivation for executing a vector instruction directly. It's a workaround, basically. I guess I could inquire in Linux kernel land regarding whether theNT_RISCV_VECTORptrace()payload could be enhanced to provide VLENB.
I think the kernel interface needs to be clarified first, before we can
proceed with the tools side.
I can see the vector state is carried in a REGSET_V regset, which in turn
corresponds to an NT_RISCV_VECTOR core file note. I can see that besides the vector data registers only the VSTART, VL, VTYPE, and VCSR vector CSRs are provided in that regset, and that vector data registers are assigned a contiguous space of (32 * RISCV_MAX_VLENB) bytes rather than individual slots.
So how are we supposed to determine the width of the vector registers
recorded in a core file? I'd say the RISC-V/Linux kernel regset API is incomplete.
A complete API has to provide `ptrace' and core file access to all the
relevant registers (vector registers in this case) that can be accessed by machine instructions by the debuggee. That includes read-only registers, writes to which via `ptrace' will of course be ignored. If a register is a shadow only and can be reconstructed from another, canonical register (e.g. VXRM vs VCSR) then the shadow register can (and best be) omitted of course. Additional artificial OS registers may also have to be provided that reflect the relevant privileged state made available to the debuggee at run time by OS calls such as prctl(2); this for example might be a mode setting which affects the hardware interpretation of a register set that debug tools may need to take into account or the person debugging may want to check or modify (e.g. REGSET_FP_MODE in the MIPS/Linux port).
I've added the authors of the Linux kernel code and the RISC-V/Linux
mailing list to the list of recipients. Am I missing anything here?
Maciej
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 23 lines · Maciej W. Rozycki
Quoted 10 lines · Greg Savin
The SIGILL guard is being used as a wrapper around determination of the VLENB CSR, which is not part of the
ptrace()payload for vector registers,
at least as it exists at head-of-tree Linux kernel. GDB or gdbserver
needs to know VLENB in order to construct the architectural feature metadata that reports an accurate width for the vector registers. If not for the VLENB determination specifically, and the lack of this information viaptrace(), then there would be no motivation for executing a vector instruction directly. It's a workaround, basically. I guess I could inquire in Linux kernel land regarding whether theNT_RISCV_VECTORptrace()payload could be enhanced to provide VLENB.I think the kernel interface needs to be clarified first, before we can
proceed with the tools side.I can see the vector state is carried in a
REGSET_Vregset, which in turn
corresponds to anNT_RISCV_VECTORcore file note. I can see that besides the vector data registers only the VSTART, VL, VTYPE, and VCSR vector CSRs are provided in that regset, and that vector data registers are assigned a contiguous space of (32 *RISCV_MAX_VLENB) bytes rather than individual slots.
Quoted 3 lines
So how are we supposed to determine the width of the vector registers
recorded in a core file? I'd say the RISC-V/Linux kernel regset API is incomplete.
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for NT_RISCV_VECTOR if we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.
Quoted 14 lines
A complete API has to provide `ptrace' and core file access to all the
relevant registers (vector registers in this case) that can be accessed by machine instructions by the debuggee. That includes read-only registers, writes to which via `ptrace' will of course be ignored. If a register is a shadow only and can be reconstructed from another, canonical register (e.g. VXRM vs VCSR) then the shadow register can (and best be) omitted of course. Additional artificial OS registers may also have to be provided that reflect the relevant privileged state made available to the debuggee at run time by OS calls such as prctl(2); this for example might be a mode setting which affects the hardware interpretation of a register set that debug tools may need to take into account or the person debugging may want to check or modify (e.g.REGSET_FP_MODEin the MIPS/Linux port).
Quoted 4 lines
I've added the authors of the Linux kernel code and the RISC-V/Linux
mailing list to the list of recipients. Am I missing anything here?Maciej
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 25 lines · Andy Chiu
Quoted 21 lines
Quoted 10 lines
The SIGILL guard is being used as a wrapper around determination of the VLENB CSR, which is not part of the
ptrace()payload for vector registers,
at least as it exists at head-of-tree Linux kernel. GDB or gdbserver
needs to know VLENB in order to construct the architectural feature metadata that reports an accurate width for the vector registers. If not for the VLENB determination specifically, and the lack of this information viaptrace(), then there would be no motivation for executing a vector instruction directly. It's a workaround, basically. I guess I could inquire in Linux kernel land regarding whether theNT_RISCV_VECTORptrace()payload could be enhanced to provide VLENB.I think the kernel interface needs to be clarified first, before we can
proceed with the tools side.I can see the vector state is carried in a
REGSET_Vregset, which in turn
corresponds to anNT_RISCV_VECTORcore file note. I can see that besides the vector data registers only the VSTART, VL, VTYPE, and VCSR vector CSRs are provided in that regset, and that vector data registers are assigned a contiguous space of (32 *RISCV_MAX_VLENB) bytes rather than individual slots.Quoted 3 lines
So how are we supposed to determine the width of the vector registers
recorded in a core file? I'd say the RISC-V/Linux kernel regset API is incomplete.
Quoted 5 lines
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for
NT_RISCV_VECTORif we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.
Maciej
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 6 lines · Maciej W. Rozycki
Quoted 5 lines
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for
NT_RISCV_VECTORif we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.
Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.
NB since this stuff went in with v6.5-rc1 and v6.5 hasn't been released
you can still back out the problematic change as no one is expected to use RC stuff in production. Alternatively you can redefine NT_RISCV_VECTOR for a corrected ABI, but I think it shouldn't be necessary. You just need to act quickly as I guess there may be 1-2 further v6.5 RCs only and you have to get with that to Linus right away. We can have a release or two without NT_RISCV_VECTOR support for the otherwise included vector stuff, it shouldn't be a big deal. There just won't be support for the debug API.
CC-ing Linux ptrace/RISC-V maintainers now to bring their attention.
Maciej
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 9 lines · Maciej W. Rozycki
Quoted 6 lines · Maciej W. Rozycki
Quoted 5 lines
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for
NT_RISCV_VECTORif we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.
Quoted 6 lines
Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.
I just took some look into the code and here is what I came up with. Actually, you know VLENB in a core dump file. The size of NT_RISCV_VECTOR in a core dump file just equals sizeof(struct __riscv_v_ext_state), which is 40B, plus VLENB * 32. So, the debugger can actually calculate VLENB and resolve placement of V registers by subtracting 40 from the size of NT_RISCV_VECTOR in a core dump file.
On the other hand, ptrace is not so lucky. The kernel will return the min of either user specified size or the maximum Vector size. It is still safe if we consider SMP with the same VLENB across cores though, which is an assumption made on Linux. We just need a way to get VLENB on the system.
Quoted 11 lines
NB since this stuff went in with v6.5-rc1 and v6.5 hasn't been released
you can still back out the problematic change as no one is expected to use RC stuff in production. Alternatively you can redefineNT_RISCV_VECTORfor a corrected ABI, but I think it shouldn't be necessary. You just need to act quickly as I guess there may be 1-2 further v6.5 RCs only and you have to get with that to Linus right away. We can have a release or two withoutNT_RISCV_VECTORsupport for the otherwise included vector stuff, it shouldn't be a big deal. There just won't be support for the debug API.
Quoted 3 lines
CC-ing Linux ptrace/RISC-V maintainers now to bring their attention.
Maciej
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Palmer Dabbelt
maintainerQuoted 24 lines · On Thu, 10 Aug 2023 10:23:34 PDT (-0700), [email protected] wrote:
Quoted 9 lines · Maciej W. Rozycki
Quoted 6 lines · Maciej W. Rozycki
Quoted 5 lines
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for
NT_RISCV_VECTORif we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.Quoted 6 lines
Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.I just took some look into the code and here is what I came up with. Actually, you know VLENB in a core dump file. The size of
NT_RISCV_VECTORin a core dump file just equalssizeof(struct __riscv_v_ext_state), which is 40B, plus VLENB * 32. So, the debugger can actually calculate VLENB and resolve placement of V registers by subtracting 40 from the size ofNT_RISCV_VECTORin a core dump file.
Quoted 16 lines
On the other hand, ptrace is not so lucky. The kernel will return the min of either user specified size or the maximum Vector size. It is still safe if we consider SMP with the same VLENB across cores though, which is an assumption made on Linux. We just need a way to get VLENB on the system.
Quoted 10 lines
NB since this stuff went in with v6.5-rc1 and v6.5 hasn't been released
you can still back out the problematic change as no one is expected to use RC stuff in production. Alternatively you can redefineNT_RISCV_VECTORfor a corrected ABI, but I think it shouldn't be necessary. You just need to act quickly as I guess there may be 1-2 further v6.5 RCs only and you have to get with that to Linus right away. We can have a release or two withoutNT_RISCV_VECTORsupport for the otherwise included vector stuff, it shouldn't be a big deal. There just won't be support for the debug API.
IMO that's the way to go: given that we're still finding breakagaes this late in the cycle it's likely we've got others. Like Maciej said, we should have gotten the GDB stuff in along with the Linux stuff to find the problems.
So let's just remove the ptrace() and core dump support for vector, it's not been released so it's not stable uABI yet. We'll just get it right before committing it, that can be as simple as just one more release.
Quoted 7 lines
Quoted 4 lines
CC-ing Linux ptrace/RISC-V maintainers now to bring their attention.
Maciej
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 7 lines · Andy Chiu
Quoted 6 lines
Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.
Quoted 6 lines
I just took some look into the code and here is what I came up with. Actually, you know VLENB in a core dump file. The size of
NT_RISCV_VECTORin a core dump file just equalssizeof(struct __riscv_v_ext_state), which is 40B, plus VLENB * 32. So, the debugger can actually calculate VLENB and resolve placement of V registers by subtracting 40 from the size ofNT_RISCV_VECTORin a core dump file.
Fair enough, I didn't dive into Linux code deeply enough to figure out
that the size of an NT_RISCV_VECTOR core file note is indeed dynamically calculated. Most notes are of a fixed size, but we also have generic support for variable-size ones in GDB, so handling this case should be reasonably straightforward.
OTOH VLENB is a program-visible register, so I think it will best be
provided explicitly regardless rather than having to be reconstructed from the size of the note; I would find that awkward.
NB I have been a bit concerned about the unusually huge allocation size
of 256KiB+ for the register buffer required for ptrace(2), but I guess we'll have to live with it, because any solution that makes it dynamic would also complicate the interface. At least we won't waste filesystem space for any extraneous allocation in core dumps.
Maciej
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 23 lines · Maciej W. Rozycki
Quoted 7 lines · Andy Chiu
Quoted 6 lines
Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.Quoted 6 lines
I just took some look into the code and here is what I came up with. Actually, you know VLENB in a core dump file. The size of
NT_RISCV_VECTORin a core dump file just equalssizeof(struct __riscv_v_ext_state), which is 40B, plus VLENB * 32. So, the debugger can actually calculate VLENB and resolve placement of V registers by subtracting 40 from the size ofNT_RISCV_VECTORin a core dump file.Fair enough, I didn't dive into Linux code deeply enough to figure out
that the size of anNT_RISCV_VECTORcore file note is indeed dynamically calculated. Most notes are of a fixed size, but we also have generic support for variable-size ones in GDB, so handling this case should be reasonably straightforward.
Quoted 3 lines
OTOH VLENB is a program-visible register, so I think it will best be
provided explicitly regardless rather than having to be reconstructed from the size of the note; I would find that awkward.
Agreed.
Quoted 6 lines
NB I have been a bit concerned about the unusually huge allocation size
of 256KiB+ for the register buffer required for ptrace(2), but I guess we'll have to live with it, because any solution that makes it dynamic would also complicate the interface. At least we won't waste filesystem space for any extraneous allocation in core dumps.
It is possible to mitigate this consideration with the proposed solution[1], by calling the ptrace twice. First we make a ptrace call to obtain VLENB in struct __riscv_v_ext_state by setting the argument iov.len = sizeof(struct __riscv_v_ext_state). Then, we can allocate a buffer based on the result of the previous ptrace to get the full Vector registers dump.
Quoted 2 lines
Maciej
[1]: https://sourceware.org/pipermail/gdb-patches/2023-August/201507.html
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Hi Maciej,
Quoted 33 lines · Maciej W. Rozycki
Quoted 25 lines · Andy Chiu
Quoted 21 lines
Quoted 10 lines
The SIGILL guard is being used as a wrapper around determination of the VLENB CSR, which is not part of the
ptrace()payload for vector registers,
at least as it exists at head-of-tree Linux kernel. GDB or gdbserver
needs to know VLENB in order to construct the architectural feature metadata that reports an accurate width for the vector registers. If not for the VLENB determination specifically, and the lack of this information viaptrace(), then there would be no motivation for executing a vector instruction directly. It's a workaround, basically. I guess I could inquire in Linux kernel land regarding whether theNT_RISCV_VECTORptrace()payload could be enhanced to provide VLENB.I think the kernel interface needs to be clarified first, before we can
proceed with the tools side.I can see the vector state is carried in a
REGSET_Vregset, which in turn
corresponds to anNT_RISCV_VECTORcore file note. I can see that besides the vector data registers only the VSTART, VL, VTYPE, and VCSR vector CSRs are provided in that regset, and that vector data registers are assigned a contiguous space of (32 *RISCV_MAX_VLENB) bytes rather than individual slots.Quoted 3 lines
So how are we supposed to determine the width of the vector registers
recorded in a core file? I'd say the RISC-V/Linux kernel regset API is incomplete.Quoted 5 lines
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for
NT_RISCV_VECTORif we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.
Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.
Conor just reminded me that we may still have a chance to get it right since 6.5 has not been released yet. I will send a fix patch to address this issue once the discussion settle down. After looking into some code, I think it is possbile to steal the unused space in datap and change the uapi with something like this:
arch/riscv/include/uapi/asm/ptrace.h +6 -3
@@ -97,14 +97,17 @@ struct __riscv_v_ext_state {unsigned long vl;unsigned long vtype;unsigned long vcsr; -void *datap; +union { +void *datap; +unsigned long vlenb; + };/* * In signal handler, datap will be set a correct user stack offset * and vector registers will be copied to the address of datap * pointer. * -* In ptrace syscall, datap will be set to zero and the vector -* registers will be copied to the address right after this +* In ptrace syscall, the space for datap will be set to vlenb and the +* vector registers will be copied to the address right after this * structure. */ };
Now ptrace will have the knowlege of vlen to parse V rsgisters. And this will not cause any size change to the original data structure that is shared by both signal and ptrace because vlenb is XLEN, which has the same size as a pointer in both ilp32/lp64.
Quoted 2 lines
Maciej
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Hi Andy,
Quoted 45 lines
Quoted 6 lines
Quoted 5 lines
Does it make sense to you if we encapsulate this with a hwprobe syscall? e.g provide a hwprobe entry to get system's VLENB. We will have to increase and rearrange the buffer for
NT_RISCV_VECTORif we want to use ptrace as the entry point for this purpose. I am not very sure if it'd be too late to do though.Quoted 6 lines
No, how do you expect it to work with a core dump (that can be examined
on a different system, or with a cross-debugger)? You need to change the API I'm afraid; it's unusable anyway. It's a pity the toolchain community wasn't consulted if you weren't sure how to design the interface. Better yet it would have been to implement the GDB side before the kernel part has been committed.Conor just reminded me that we may still have a chance to get it right since 6.5 has not been released yet. I will send a fix patch to address this issue once the discussion settle down. After looking into some code, I think it is possbile to steal the unused space in datap and change the uapi with something like this:
arch/riscv/include/uapi/asm/ptrace.h +6
-3@@ -97,14 +97,17 @@ struct __riscv_v_ext_state {unsigned long vl;unsigned long vtype;unsigned long vcsr; -void *datap; +union { +void *datap; +unsigned long vlenb; + };/* * In signal handler, datap will be set a correct user stack offset * and vector registers will be copied to the address of datap * pointer. * -* In ptrace syscall, datap will be set to zero and the vector -* registers will be copied to the address right after this +* In ptrace syscall, the space for datap will be set to vlenb and the +* vector registers will be copied to the address right after this * structure. */ };
Quoted 4 lines
Now ptrace will have the knowlege of vlen to parse V rsgisters. And this will not cause any size change to the original data structure that is shared by both signal and ptrace because vlenb is XLEN, which has the same size as a pointer in both ilp32/lp64.
Barring details such as field naming (perhaps `vregp' rather than opaque
`datap'?), or whether we want to have a union embedded such as above or distinct UAPI data types for the two use cases I think your proposal for the updated contents makes sense to me, thanks.
Maciej
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv