Skip to contentSkip to conversation

tools/testing/selftests/kvm/rseq_test and glibc 2.35

DISCUSSION Florian Weimer 3 participants 6 messages Aug 8, 2022 – Aug 9, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

It has come to my attention that the KVM rseq test apparently needs to be ported to glibc 2.35. The background is that on aarch64, rseq is the only way to get a practically useful sched_getcpu. (There's no hidden per-task CPU state the vDSO could reveal as the CPU ID.)

The main rseq tests have already been adjusted via:

commit 233e667e1ae3e348686bd9dd0172e62a09d852e1

Author: Mathieu Desnoyers <[email protected]>
Date:   Mon Jan 24 12:12:45 2022 -0500

selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35

glibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.

The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.

Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:

- librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.

- librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.

- Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.

But I don't see a similar adjustment for tools/testing/selftests/kvm/rseq_test.c. As an additional wrinkle, you'd have to start calling getcpu (glibc function or system call) because comparing rseq.cpu_id against sched_getcpu won't test anything anymore once glibc implements sched_getcpu using rseq.

We noticed this because our downstream glibc version, while based on 2.34, enables rseq registration by default. To facilitate coordination with rseq application usage, we also backported the __rseq_* ABI symbols, so the selftests could use that even in our downstream version. (We enable the glibc tunables downstream, but they are an optional glibc feature, so it's probably better in the long run to fix the kernel selftests rather than using the tunables as a workaround.)

Thanks,
Florian

#1 lore raw

Hi Florian,

Quoted 5 lines · Florian Weimer

It has come to my attention that the KVM rseq test apparently needs to be ported to glibc 2.35. The background is that on aarch64, rseq is the only way to get a practically useful sched_getcpu. (There's no hidden per-task CPU state the vDSO could reveal as the CPU ID.)

Yes, kvm/selftests/rseq needs to support glibc 2.35. The question is about glibc 2.34 or 2.35 because kvm/selftest/rseq fails on glibc 2.34

I would guess upstream-glibc-2.35 feature is enabled on downstream glibc-2.34?

# ./rseq_test
==== Test Assertion Failure ====
  rseq_test.c:60: !r
  pid=112043 tid=112043 errno=22 - Invalid argument
     1	0x0000000000401973: main at rseq_test.c:226
     2	0x0000ffff84b6c79b: ?? ??:0
     3	0x0000ffff84b6c86b: ?? ??:0
     4	0x0000000000401b6f: _start at ??:?
  rseq failed, errno = 22 (Invalid argument)
# rpm -aq | grep glibc-2
glibc-2.34-39.el9.aarch64
Quoted 41 lines

The main rseq tests have already been adjusted via:

commit 233e667e1ae3e348686bd9dd0172e62a09d852e1

Author: Mathieu Desnoyers <[email protected]>
Date:   Mon Jan 24 12:12:45 2022 -0500

selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35

glibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.

The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.

Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:

- librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.

- librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.

- Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.

But I don't see a similar adjustment for tools/testing/selftests/kvm/rseq_test.c. As an additional wrinkle, you'd have to start calling getcpu (glibc function or system call) because comparing rseq.cpu_id against sched_getcpu won't test anything anymore once glibc implements sched_getcpu using rseq.

Quoted 8 lines

We noticed this because our downstream glibc version, while based on 2.34, enables rseq registration by default. To facilitate coordination with rseq application usage, we also backported the __rseq_* ABI symbols, so the selftests could use that even in our downstream version. (We enable the glibc tunables downstream, but they are an optional glibc feature, so it's probably better in the long run to fix the kernel selftests rather than using the tunables as a workaround.)

Thanks for the pointer. It makes sense. So it means rseq registration has been done by glibc TLS? In this case, kvm/selftests/rseq is unable to register again.

I will come up something similiar for kvm/selftest/rseq.

Thanks,
Gavin

#2 lore raw
Quoted 78 lines · ----- Gavin Shan

Hi Florian,

Quoted 5 lines · Florian Weimer

It has come to my attention that the KVM rseq test apparently needs to be ported to glibc 2.35. The background is that on aarch64, rseq is the only way to get a practically useful sched_getcpu. (There's no hidden per-task CPU state the vDSO could reveal as the CPU ID.)

Yes, kvm/selftests/rseq needs to support glibc 2.35. The question is about glibc 2.34 or 2.35 because kvm/selftest/rseq fails on glibc 2.34

I would guess upstream-glibc-2.35 feature is enabled on downstream glibc-2.34?

# ./rseq_test
==== Test Assertion Failure ====
   rseq_test.c:60: !r
   pid=112043 tid=112043 errno=22 - Invalid argument
      1	0x0000000000401973: main at rseq_test.c:226
      2	0x0000ffff84b6c79b: ?? ??:0
      3	0x0000ffff84b6c86b: ?? ??:0
      4	0x0000000000401b6f: _start at ??:?
   rseq failed, errno = 22 (Invalid argument)
# rpm -aq | grep glibc-2
glibc-2.34-39.el9.aarch64
Quoted 41 lines

The main rseq tests have already been adjusted via:

commit 233e667e1ae3e348686bd9dd0172e62a09d852e1

Author: Mathieu Desnoyers <[email protected]>
Date:   Mon Jan 24 12:12:45 2022 -0500

selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35

glibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.

The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.

Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:

- librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.

- librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.

- Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.

But I don't see a similar adjustment for tools/testing/selftests/kvm/rseq_test.c. As an additional wrinkle, you'd have to start calling getcpu (glibc function or system call) because comparing rseq.cpu_id against sched_getcpu won't test anything anymore once glibc implements sched_getcpu using rseq.

Quoted 8 lines

We noticed this because our downstream glibc version, while based on 2.34, enables rseq registration by default. To facilitate coordination with rseq application usage, we also backported the __rseq_* ABI symbols, so the selftests could use that even in our downstream version. (We enable the glibc tunables downstream, but they are an optional glibc feature, so it's probably better in the long run to fix the kernel selftests rather than using the tunables as a workaround.)

Quoted 3 lines

Thanks for the pointer. It makes sense. So it means rseq registration has been done by glibc TLS? In this case, kvm/selftests/rseq is unable to register again.

The registration is done by glibc initialization and thread startup code.

Quoted 2 lines

I will come up something similiar for kvm/selftest/rseq.

Make sure to chech the rseq selftests fixes recently pulled in the current merge window as well. One is relevant:

https://github.com/torvalds/linux/commit/d1a997ba4c1bf65497d956aea90de42a6398f73a

We may want to find a way to remove this duplicated rseq.c code eventually.

Thanks,

Mathieu

Quoted 4 lines

Thanks,
Gavin

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#3 lore raw
Quoted 87 lines · Mathieu Desnoyers
Quoted 78 lines · ----- Gavin Shan

Hi Florian,

Quoted 5 lines · Florian Weimer

It has come to my attention that the KVM rseq test apparently needs to be ported to glibc 2.35. The background is that on aarch64, rseq is the only way to get a practically useful sched_getcpu. (There's no hidden per-task CPU state the vDSO could reveal as the CPU ID.)

Yes, kvm/selftests/rseq needs to support glibc 2.35. The question is about glibc 2.34 or 2.35 because kvm/selftest/rseq fails on glibc 2.34

I would guess upstream-glibc-2.35 feature is enabled on downstream glibc-2.34?

# ./rseq_test
==== Test Assertion Failure ====
    rseq_test.c:60: !r
    pid=112043 tid=112043 errno=22 - Invalid argument
       1	0x0000000000401973: main at rseq_test.c:226
       2	0x0000ffff84b6c79b: ?? ??:0
       3	0x0000ffff84b6c86b: ?? ??:0
       4	0x0000000000401b6f: _start at ??:?
    rseq failed, errno = 22 (Invalid argument)
# rpm -aq | grep glibc-2
glibc-2.34-39.el9.aarch64
Quoted 41 lines

The main rseq tests have already been adjusted via:

commit 233e667e1ae3e348686bd9dd0172e62a09d852e1

Author: Mathieu Desnoyers <[email protected]>
Date:   Mon Jan 24 12:12:45 2022 -0500

selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35

glibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.

The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.

Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:

- librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.

- librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.

- Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.

But I don't see a similar adjustment for tools/testing/selftests/kvm/rseq_test.c. As an additional wrinkle, you'd have to start calling getcpu (glibc function or system call) because comparing rseq.cpu_id against sched_getcpu won't test anything anymore once glibc implements sched_getcpu using rseq.

Quoted 8 lines

We noticed this because our downstream glibc version, while based on 2.34, enables rseq registration by default. To facilitate coordination with rseq application usage, we also backported the __rseq_* ABI symbols, so the selftests could use that even in our downstream version. (We enable the glibc tunables downstream, but they are an optional glibc feature, so it's probably better in the long run to fix the kernel selftests rather than using the tunables as a workaround.)

Quoted 3 lines

Thanks for the pointer. It makes sense. So it means rseq registration has been done by glibc TLS? In this case, kvm/selftests/rseq is unable to register again.

The registration is done by glibc initialization and thread startup code.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

I will come up something similiar for kvm/selftest/rseq.

Make sure to chech the rseq selftests fixes recently pulled in the current merge window as well. One is relevant:

https://github.com/torvalds/linux/commit/d1a997ba4c1bf65497d956aea90de42a6398f73a

We may want to find a way to remove this duplicated rseq.c code eventually.

Thanks, Mathieu. The check for 'rseq-size' will be included either. I almost have something working. I will post the fixes after some tests.

Thanks,
Gavin

#4 lore raw
Quoted 81 lines · Gavin Shan
Quoted 80 lines · Mathieu Desnoyers
Quoted 69 lines · Florian Weimer
Quoted 5 lines

It has come to my attention that the KVM rseq test apparently needs to be ported to glibc 2.35.  The background is that on aarch64, rseq is the only way to get a practically useful sched_getcpu.  (There's no hidden per-task CPU state the vDSO could reveal as the CPU ID.)

Yes, kvm/selftests/rseq needs to support glibc 2.35. The question is about glibc 2.34 or 2.35 because kvm/selftest/rseq fails on glibc 2.34

I would guess upstream-glibc-2.35 feature is enabled on downstream glibc-2.34?

# ./rseq_test
==== Test Assertion Failure ====
    rseq_test.c:60: !r
    pid=112043 tid=112043 errno=22 - Invalid argument
       1    0x0000000000401973: main at rseq_test.c:226
       2    0x0000ffff84b6c79b: ?? ??:0
       3    0x0000ffff84b6c86b: ?? ??:0
       4    0x0000000000401b6f: _start at ??:?
    rseq failed, errno = 22 (Invalid argument)
# rpm -aq | grep glibc-2
glibc-2.34-39.el9.aarch64
Quoted 34 lines

The main rseq tests have already been adjusted via:

commit 233e667e1ae3e348686bd9dd0172e62a09d852e1
Author: Mathieu Desnoyers <[email protected]> Date:   Mon Jan 24 12:12:45 2022 -0500

      selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35 glibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.
      The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.
      Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:
      - librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.
      - librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.
      - Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.

But I don't see a similar adjustment for tools/testing/selftests/kvm/rseq_test.c.  As an additional wrinkle, you'd have to start calling getcpu (glibc function or system call) because comparing rseq.cpu_id against sched_getcpu won't test anything anymore once glibc implements sched_getcpu using rseq.

Quoted 8 lines

We noticed this because our downstream glibc version, while based on 2.34, enables rseq registration by default.  To facilitate coordination with rseq application usage, we also backported the __rseq_* ABI symbols, so the selftests could use that even in our downstream version. (We enable the glibc tunables downstream, but they are an optional glibc feature, so it's probably better in the long run to fix the kernel selftests rather than using the tunables as a workaround.)

Quoted 3 lines

Thanks for the pointer. It makes sense. So it means rseq registration has been done by glibc TLS? In this case, kvm/selftests/rseq is unable to register again.

The registration is done by glibc initialization and thread startup code.

Quoted 2 lines

I will come up something similiar for kvm/selftest/rseq.

Make sure to chech the rseq selftests fixes recently pulled in the current merge window as well. One is relevant:

Quoted 8 lines
Quoted 4 lines

https://github.com/torvalds/linux/commit/d1a997ba4c1bf65497d956aea90de42a6398f73a

We may want to find a way to remove this duplicated rseq.c code eventually.

Thanks, Mathieu. The check for 'rseq-size' will be included either. I almost have something working. I will post the fixes after some tests.

Mathieu and Florian, the fixes have been posted. It would be nice for you to review if you have free cycles :)

https://lore.kernel.org/kvmarm/[email protected]/T/#t

Thanks,
Gavin

#5 lore raw
In reply to Gavin Shan

* Gavin Shan:

Quoted 15 lines

Hi Florian,

Quoted 5 lines · Florian Weimer

It has come to my attention that the KVM rseq test apparently needs to be ported to glibc 2.35. The background is that on aarch64, rseq is the only way to get a practically useful sched_getcpu. (There's no hidden per-task CPU state the vDSO could reveal as the CPU ID.)

Yes, kvm/selftests/rseq needs to support glibc 2.35. The question is about glibc 2.34 or 2.35 because kvm/selftest/rseq fails on glibc 2.34

I would guess upstream-glibc-2.35 feature is enabled on downstream glibc-2.34?

Quoted 11 lines
# ./rseq_test
==== Test Assertion Failure ====
  rseq_test.c:60: !r
  pid=112043 tid=112043 errno=22 - Invalid argument
     1	0x0000000000401973: main at rseq_test.c:226
     2	0x0000ffff84b6c79b: ?? ??:0
     3	0x0000ffff84b6c86b: ?? ??:0
     4	0x0000000000401b6f: _start at ??:?
  rseq failed, errno = 22 (Invalid argument)
# rpm -aq | grep glibc-2
glibc-2.34-39.el9.aarch64

Yes, we have enabled it downstream.

glibc: Restartable sequences interfaces and sched_getcpu accelerated by default
<https://bugzilla.redhat.com/show_bug.cgi?id=2085529>

However,

GLIBC_TUNABLES=glibc.pthread.rseq=0 ./rseq_test

should still work (we added the ability to disable rseq registration precisely to enable scenarios like this), but tunables are an optional glibc feature, so the upstream kernel should probably still be fixed.

Quoted 4 lines

Mathieu and Florian, the fixes have been posted. It would be nice for you to review if you have free cycles :)

https://lore.kernel.org/kvmarm/[email protected]/T/#t

Excellent. I'm going to have a look.

Thanks,
Florian