futex(2) man page update help request

In reply to Carlos O'Donell

Quoted 8 lines · Michael Kerrisk (man-pages)

Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)

I fully agree with Michael here.

The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.

The fact that the futex() syscall is out of date is my fault, is the fault of Linux kernel developers, etc. etc., we should all have reached out to Michael with patches to keep this developer-centric documentation updated.

Quoted 13 lines

Quoted 13 lines
something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inclu
de/futextest.h#n67
Which basically defines:
#define futex(uaddr, op, val, timeout, uaddr2, val3, opflags) \
syscall(SYS_futex, uaddr, op | opflags, val, timeout, uaddr2, val3)

Quoted 8 lines

Quoted 4 lines
Adding Carlos for his perspective.
If I'm wrong, or we can restore the futex() call, great. If not... Should we keep the man-pages and document it as syscall(SYS_futex, ..., op, ...) ?
We should keep the man page ;-). I just need to add some words to point out the use of syscall(). Mostly though, I'm interested in getting documentation of the operations listed below :-).

Agreed. The man page should get expanded and should be as detailed as possible about the interface the kernel is exposing.

There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.

That's what should happen here with futex() (though NOTES does mention this, but it's not called out at the top of the man page).

Cheers,
Carlos.
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Andy Lutomirski

2014-05-14 20:44 UTC #5 lore raw

Quoted 12 lines · Carlos O'Donell

Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)
I fully agree with Michael here.

Quoted 8 lines

The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.
The fact that the futex() syscall is out of date is my fault, is the fault of Linux kernel developers, etc. etc., we should all have reached out to Michael with patches to keep this developer-centric documentation updated.

I realize that this is out of scope for linux-abi, but I *stongly* disagree with this notion. futex() needs to be just as stable as anything else: old glibc versions must continue to work. I just jumped through a bunch of hoops to keep a single glibc patch release in OpenSUSE 9 working in a maintainable way; breaking futex will break far more than that.

Additionally, at least the FUTEX_WAIT and FUTEX_WAKE operations are extremely useful, and they can do things that are tedious at best using mutexes and condvars. It's a simple API to use. I use it, and I've seen plenty of other open-source apps using the futex API directly.

I think the best way forward might be to try to convince the glibc maintainers to add the wrapper.

--Andy

Thomas Gleixner

2014-05-14 23:34 UTC #6 lore raw

Quoted 10 lines · Carlos O'Donell

Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)

Quoted 6 lines

I fully agree with Michael here.
The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.

Nonsense.

If we change that interface (aside of adding functionality or some new error return) it would break the world and some more, simply because out of the blue glibc-2.xx would stop to work on linux-3.yy.

Aside of that the futex syscall is used as a bare interface without any glibc interaction:

- It's handy to implement user space wait queues

- It's (ab)used in very interesting ways by data base apps

- It's (ab)used by some Java monstrosities.

Nothing you care about and you really don't want to see the gory details, but you have to accept that there is an universe which is happy to deal with the raw syscalls instead of going through some ill defined posix interfaces.

Thanks,

tglx

Carlos O'Donell

2014-05-15 03:12 UTC #7 lore raw

Quoted 12 lines · Thomas Gleixner

Quoted 10 lines · Carlos O'Donell
Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)

Quoted 8 lines

Quoted 6 lines
I fully agree with Michael here.
The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.
Nonsense.

What is nonsense?

I do not want to be responsible for the futex API by having glibc provide wrappers. That can't be nonsense since it's a glibc community decision to make.

Perhaps the point at which we disagree is that I said "writers of core runtimes" and you would rather I have said "any application wishing to use raw syscalls." That's fine, I concede that point, I have no right to restrict raw syscall usage.

Quoted 3 lines

If we change that interface (aside of adding functionality or some new error return) it would break the world and some more, simply because out of the blue glibc-2.xx would stop to work on linux-3.yy.

No disagreement from me.

Quoted 5 lines

Aside of that the futex syscall is used as a bare interface without any glibc interaction:
- It's handy to implement user space wait queues

Quoted 8 lines

- It's (ab)used in very interesting ways by data base apps
- It's (ab)used by some Java monstrosities.
Nothing you care about and you really don't want to see the gory details, but you have to accept that there is an universe which is happy to deal with the raw syscalls instead of going through some ill defined posix interfaces.

Sure :-)

Cheers,
Carlos.

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Michael Kerrisk (man-pages)

author

2014-05-15 04:49 UTC #8 lore raw

Quoted 15 lines · Carlos O'Donell

Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Carlos O'Donell
Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)
I fully agree with Michael here.

Quoted 8 lines

Quoted 6 lines
Quoted 4 lines
The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.
Nonsense.
What is nonsense?

I suspect there's a misunderstanding between worlds here. Thomas means that the kernel ABI is stable. You mean, glibc does not want to have to export an ABI that you have to support.

Quoted 3 lines

I do not want to be responsible for the futex API by having glibc provide wrappers. That can't be nonsense since it's a glibc community decision to make.

See my above.

Cheers,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Michael Kerrisk (man-pages)

author

2014-05-15 04:53 UTC #9 lore raw

Hi Thomas,

Quoted 25 lines · Thomas Gleixner

Quoted 10 lines · Carlos O'Donell
Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)
Quoted 6 lines
I fully agree with Michael here.
The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.
Nonsense.
If we change that interface (aside of adding functionality or some new error return) it would break the world and some more, simply because out of the blue glibc-2.xx would stop to work on linux-3.yy.

Quoted 8 lines

Aside of that the futex syscall is used as a bare interface without any glibc interaction:
- It's handy to implement user space wait queues
- It's (ab)used in very interesting ways by data base apps
- It's (ab)used by some Java monstrosities.

Thanks for the education about user-space uses of futexes. I was unaware.

Quoted 4 lines

Nothing you care about and you really don't want to see the gory details, but you have to accept that there is an universe which is happy to deal with the raw syscalls instead of going through some ill defined posix interfaces.

And that universe would love to have your documentation of FUTEX_WAKE_BITSET and FUTEX_WAIT_BITSET ;-),

Cheers,

Michael

Signature · 7 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thomas Gleixner

2014-05-15 14:14 UTC #10 lore raw

Quoted 2 lines · Michael Kerrisk (man-pages)

And that universe would love to have your documentation of FUTEX_WAKE_BITSET and FUTEX_WAIT_BITSET ;-),

I give you almost the full treatment, but I leave REQUEUE_PI to Darren and FUTEX_WAKE_OP to Jakub. :)

FUTEX_WAIT

< Existing blurb seems ok >

Related return values

[EFAULT] Kernel was unable to access the futex value at uaddr.

[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned

[EINVAL] The supplied timeout argument is not normalized.

[EWOULDBLOCK] The atomic enqueueing failed. User space value
at uaddr is not equal val argument.

[ETIMEDOUT] timeout expired

FUTEX_WAKE

< Existing blurb seems ok >

Related return values

[EFAULT] Kernel was unable to access the futex value at uaddr.

[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI

FUTEX_REQUEUE

Existing blurb seems ok , except for this:

The argument val contains the number of waiters on uaddr which are immediately woken up.

The timeout argument is abused to transport the number of waiters which are requeued to the futex at uaddr2. The pointer is typecasted to u32.

[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2

[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

[EINVAL] uaddr equal uaddr2. Requeue to same futex.

FUTEX_REQUEUE_CMP

Existing blurb seems ok , except for this:

The argument val is contains the number of waiters on uaddr which are immediately woken up.

The timeout argument is abused to transport the number of waiters which are requeued to the futex at uaddr2. The pointer is typecasted to u32.

Related return values

[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2

[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned

[EINVAL] uaddr equal uaddr2. Requeue to same futex.

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

[EAGAIN] uaddr1 readout is not equal the compare value in
argument val3

FUTEX_WAKE_OP

Jakub, can you please explain it? I'm lost :)

The argument val contains the maximum number of waiters on uaddr which are immediately woken up.

The timeout argument is abused to transport the maximum number of waiters on uaddr2 which are woken up. The pointer is typecasted to u32.

Related return values

[EFAULT] Kernel was unable to access the futex values at uaddr
or uaddr2

[EINVAL] The supplied uaddr or uaddr2 argument does not point
to a valid object, i.e. pointer is not 4 byte aligned

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

FUTEX_WAIT_BITSET

The same as FUTEX_WAIT except that val3 is used to provide a 32bit bitset to the kernel. This bitset is stored in the kernel internal state of the waiter.

This futex op also allows to have the option bit FUTEX_CLOCK_REALTIME set.

Related return values

[EFAULT] Kernel was unable to access the futex value at uaddr.

[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned

[EINVAL] The supplied bitset is zero.

[EINVAL] The supplied timeout argument is not normalized.

[ETIMEDOUT] timeout expired

FUTEX_WAKE_BITSET

The same as FUTEX_WAKE except that val3 is used to provide a 32bit bitset to the kernel. This bitset is used to select waiters on the futex. The selection is done by a bitwise AND of the wake side supplied bitset and the bitset which is stored in the kernel internal state of the waiters. If the result is non zero, the waiter is woken, otherwise left waiting.

[EFAULT] Kernel was unable to access the futex value at uaddr.

[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned

[EINVAL] The supplied bitset is zero.

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI

FUTEX_LOCK_PI

This operation reads from the futex address provided by the uaddr argument, which contains the namespace specific TID of the lock owner. If the TID is 0, then the kernel tries to set the waiters TID atomically. If the TID is nonzero or the take over fails the kernel sets atomically the FUTEX_WAITERS bit which signals the owner, that it cannot unlock the futex in user space atomically by transitioning from TID to 0. After that the kernel tries to find the task which is associated to the owner TID, creates or reuses kernel state on behalf of the owner and attaches the waiter to it. The enqueing of the waiter is in descending priority order if more than one waiter exists. The owner inherits either the priority or the bandwidth of the waiter. This inheritance follows the lock chain in the case of nested locking and performs deadlock detection.

The timeout argument is handled as described in FUTEX_WAIT. The arguments uaddr2, val, and val3 are ignored.

Related return values

[EFAULT] Kernel was unable to access the futex value at uaddr.

[ENOMEM] Kernel could not allocate state

[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned

[EINVAL] The supplied timeout argument is not normalized.

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state. Thats either state corruption or it found a waiter on uaddr which is waiting on FUTEX_WAIT[_BITSET]

[EPERM] Caller is not allowed to attach itself to the futex.
Can be a legitimate issue or a hint for state corruption in user space

[ESRCH] The TID in the user space value does not exist

[EAGAIN] The futex owner TID is about to exit, but has not yet
handled the internal state cleanup. Try again.

[ETIMEDOUT] timeout expired

[EDEADLOCK] The futex is already locked by the caller or the kernel
detected a deadlock scenario in a nested lock chain

[EOWNERDIED] The owner of the futex died and the kernel made the
caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup

[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)

FUTEX_TRYLOCK_PI

This operation tries to acquire the futex at uaddr. It deals with the situation where the TID value at uaddr is 0, but the FUTEX_HAS_WAITER bit is set. User space cannot handle this race free.

The arguments uaddr2, val, timeout and val3 are ignored.

Return values:

[EFAULT] Kernel was unable to access the futex value at uaddr.

[ENOMEM] Kernel could not allocate state

[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned

[EINVAL] The kernel detected inconsistent state between the user
space state at uaddr and the kernel state

[EPERM] Caller is not allowed to attach itself to the futex.
Can be a legitimate issue or a hint for state corruption in user space

[ESRCH] The TID in the user space value does not exist

[EAGAIN] The futex owner TID is about to exit, but has not yet
handled the internal state cleanup. Try again.

[EDEADLOCK] The futex is already locked by the caller.

[EOWNERDIED] The owner of the futex died and the kernel made the
caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup

[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)

FUTEX_UNLOCK_PI

This operation wakes the top priority waiter which is waiting in FUTEX_LOCK_PI on the futex address provided by the uaddr argument.

This is called when the user space value at uaddr cannot be changed atomically from TID (of the owner) to 0.

The arguments uaddr2, val, timeout and val3 are ignored.

Related return values:

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET].

[EPERM] Caller does not own the futex.

[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)

FUTEX_WAIT_REQUEUE_PI

Wait operation to wait on a non pi futex at uaddr and potentially be requeued on a pi futex at uaddr2. The wait operation on uaddr is the same as FUTEX_WAIT. The waiter can be removed from the wait on uaddr via FUTEX_WAKE without requeuing on uaddr2.

The timeout argument is handled as described in FUTEX_WAIT.

Darren, can you fill in the missing details?

Return values:

[EFAULT] Kernel was unable to access the futex value at uaddr
or uaddr2

[EINVAL] The supplied uaddr or uaddr2 argument does not point
to a valid object, i.e. pointer is not 4 byte aligned

[EINVAL] The supplied timeout argument is not normalized.

[EINVAL] The supplied bitset is zero.

[EWOULDBLOCK] The atomic enqueueing failed. User space value
at uaddr is not equal val argument.

[ETIMEDOUT] timeout expired

[EOWNERDIED] The owner of the PI futex at uaddr2 died and the
kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the uaddr2 futex userspace value. Caller is responsible for cleanup

[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)

FUTEX_CMP_REQUEUE_PI

PI aware variant of FUTEX_CMP_REQUEUE. Inner futex at uaddr is a non PI futex. Outer futex to which is requeued is a PI futex at uaddr2.

The waiters on uaddr must wait in FUTEX_WAIT_REQUEUE_PI.

The argument val is contains the number of waiters on uaddr which are immediately woken up. Must be 1 for this opcode.

The timeout argument is abused to transport the number of waiters which are requeued on to the futex at uaddr2. The pointer is typecasted to u32.

Darren, can you fill in the missing details?

[EFAULT] Kernel was unable to access the futex value at uaddr
or uaddr2

[ENOMEM] Kernel could not allocate state

[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned

[EINVAL] uaddr equal uaddr2. Requeue to same futex.

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET] on uaddr

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr2 and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT on uaddr2.

[EINVAL] The supplied bitset is zero.

[EAGAIN] uaddr1 readout is not equal the compare value in
argument val3

[EAGAIN] The futex owner TID of uaddr2 is about to exit, but
has not yet handled the internal state cleanup. Try again.

[EPERM] Caller is not allowed to attach the waiter to the
futex at uaddr2 Can be a legitimate issue or a hint for state corruption in user space

[ESRCH] The TID in the user space value at uaddr2 does not exist

[EDEADLOCK] The requeuing of a waiter to the kernel representation
of the PI futex at uaddr2 detected a deadlock scenario.

[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)

The various option bits seem to be undocumented as well

FUTEX_PRIVATE_FLAG

This option bit can be ored on all futex ops.

It tells the kernel, that the futex is process private and not shared with another process. That allows the kernel to chose the fast path for validating the user space address and avoids expensive VMA lookup, taking refcounts on file backing store etc.

FUTEX_CLOCK_REALTIME

This option bit can be ored on the futex ops FUTEX_WAIT_BITSET and FUTEX_WAIT_REQUEUE_PI

If set the kernel treats the user space supplied timeout as absolute time based on CLOCK_REALTIME.

If not set the kernel treats the user space supplied timeout as relative time.

If this is set on any other op than the supported ones, kernel returns ENOSYS!

Thanks,

tglx

Michael Kerrisk (man-pages)

author

2014-05-15 20:19 UTC #11 lore raw

Quoted 6 lines · Thomas Gleixner

Quoted 2 lines · Michael Kerrisk (man-pages)
And that universe would love to have your documentation of FUTEX_WAKE_BITSET and FUTEX_WAIT_BITSET ;-),
I give you almost the full treatment, but I leave REQUEUE_PI to Darren and FUTEX_WAKE_OP to Jakub. :)

Thanks Thomas--that's fantastic! Hopefully, Darren and Jakub fill in those missing pieces...

Cheers,

Michael

Quoted 415 lines

FUTEX_WAIT
< Existing blurb seems ok >
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.
[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned
[EINVAL] The supplied timeout argument is not normalized.
[EWOULDBLOCK] The atomic enqueueing failed. User space value
at uaddr is not equal val argument.
[ETIMEDOUT] timeout expired
FUTEX_WAKE
< Existing blurb seems ok >
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.
[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI
FUTEX_REQUEUE
Existing blurb seems ok , except for this:
The argument val contains the number of waiters on uaddr which are immediately woken up.
The timeout argument is abused to transport the number of waiters which are requeued to the futex at uaddr2. The pointer is typecasted to u32.
[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2
[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
FUTEX_REQUEUE_CMP
Existing blurb seems ok , except for this:
The argument val is contains the number of waiters on uaddr which are immediately woken up.
The timeout argument is abused to transport the number of waiters which are requeued to the futex at uaddr2. The pointer is typecasted to u32.
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2
[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr
[EAGAIN] uaddr1 readout is not equal the compare value in
argument val3
FUTEX_WAKE_OP
Jakub, can you please explain it? I'm lost :)
The argument val contains the maximum number of waiters on uaddr which are immediately woken up.
The timeout argument is abused to transport the maximum number of waiters on uaddr2 which are woken up. The pointer is typecasted to u32.
Related return values
[EFAULT] Kernel was unable to access the futex values at uaddr
or uaddr2
[EINVAL] The supplied uaddr or uaddr2 argument does not point
to a valid object, i.e. pointer is not 4 byte aligned
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr
FUTEX_WAIT_BITSET
The same as FUTEX_WAIT except that val3 is used to provide a 32bit bitset to the kernel. This bitset is stored in the kernel internal state of the waiter.
This futex op also allows to have the option bit FUTEX_CLOCK_REALTIME set.
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.
[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned
[EINVAL] The supplied bitset is zero.
[EINVAL] The supplied timeout argument is not normalized.
[ETIMEDOUT] timeout expired
FUTEX_WAKE_BITSET
The same as FUTEX_WAKE except that val3 is used to provide a 32bit bitset to the kernel. This bitset is used to select waiters on the futex. The selection is done by a bitwise AND of the wake side supplied bitset and the bitset which is stored in the kernel internal state of the waiters. If the result is non zero, the waiter is woken, otherwise left waiting.
[EFAULT] Kernel was unable to access the futex value at uaddr.
[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned
[EINVAL] The supplied bitset is zero.
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI
FUTEX_LOCK_PI
This operation reads from the futex address provided by the uaddr argument, which contains the namespace specific TID of the lock owner. If the TID is 0, then the kernel tries to set the waiters TID atomically. If the TID is nonzero or the take over fails the kernel sets atomically the FUTEX_WAITERS bit which signals the owner, that it cannot unlock the futex in user space atomically by transitioning from TID to 0. After that the kernel tries to find the task which is associated to the owner TID, creates or reuses kernel state on behalf of the owner and attaches the waiter to it. The enqueing of the waiter is in descending priority order if more than one waiter exists. The owner inherits either the priority or the bandwidth of the waiter. This inheritance follows the lock chain in the case of nested locking and performs deadlock detection.
The timeout argument is handled as described in FUTEX_WAIT. The arguments uaddr2, val, and val3 are ignored.
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.
[ENOMEM] Kernel could not allocate state
[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned
[EINVAL] The supplied timeout argument is not normalized.
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state. Thats either state corruption or it found a waiter on uaddr which is waiting on FUTEX_WAIT[_BITSET]
[EPERM] Caller is not allowed to attach itself to the futex.
Can be a legitimate issue or a hint for state corruption in user space
[ESRCH] The TID in the user space value does not exist
[EAGAIN] The futex owner TID is about to exit, but has not yet
handled the internal state cleanup. Try again.
[ETIMEDOUT] timeout expired
[EDEADLOCK] The futex is already locked by the caller or the kernel
detected a deadlock scenario in a nested lock chain
[EOWNERDIED] The owner of the futex died and the kernel made the
caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)
FUTEX_TRYLOCK_PI
This operation tries to acquire the futex at uaddr. It deals with the situation where the TID value at uaddr is 0, but the FUTEX_HAS_WAITER bit is set. User space cannot handle this race free.
The arguments uaddr2, val, timeout and val3 are ignored.
Return values:
[EFAULT] Kernel was unable to access the futex value at uaddr.
[ENOMEM] Kernel could not allocate state
[EINVAL] The supplied uaddr argument does not point to a valid
object, i.e. pointer is not 4 byte aligned
[EINVAL] The kernel detected inconsistent state between the user
space state at uaddr and the kernel state
[EPERM] Caller is not allowed to attach itself to the futex.
Can be a legitimate issue or a hint for state corruption in user space
[ESRCH] The TID in the user space value does not exist
[EAGAIN] The futex owner TID is about to exit, but has not yet
handled the internal state cleanup. Try again.
[EDEADLOCK] The futex is already locked by the caller.
[EOWNERDIED] The owner of the futex died and the kernel made the
caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)
FUTEX_UNLOCK_PI
This operation wakes the top priority waiter which is waiting in FUTEX_LOCK_PI on the futex address provided by the uaddr argument.
This is called when the user space value at uaddr cannot be changed atomically from TID (of the owner) to 0.
The arguments uaddr2, val, timeout and val3 are ignored.
Related return values:
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET].
[EPERM] Caller does not own the futex.
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)
FUTEX_WAIT_REQUEUE_PI
Wait operation to wait on a non pi futex at uaddr and potentially be requeued on a pi futex at uaddr2. The wait operation on uaddr is the same as FUTEX_WAIT. The waiter can be removed from the wait on uaddr via FUTEX_WAKE without requeuing on uaddr2.
The timeout argument is handled as described in FUTEX_WAIT.
Darren, can you fill in the missing details?
Return values:
[EFAULT] Kernel was unable to access the futex value at uaddr
or uaddr2
[EINVAL] The supplied uaddr or uaddr2 argument does not point
to a valid object, i.e. pointer is not 4 byte aligned
[EINVAL] The supplied timeout argument is not normalized.
[EINVAL] The supplied bitset is zero.
[EWOULDBLOCK] The atomic enqueueing failed. User space value
at uaddr is not equal val argument.
[ETIMEDOUT] timeout expired
[EOWNERDIED] The owner of the PI futex at uaddr2 died and the
kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the uaddr2 futex userspace value. Caller is responsible for cleanup
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)
FUTEX_CMP_REQUEUE_PI
PI aware variant of FUTEX_CMP_REQUEUE. Inner futex at uaddr is a non PI futex. Outer futex to which is requeued is a PI futex at uaddr2.
The waiters on uaddr must wait in FUTEX_WAIT_REQUEUE_PI.
The argument val is contains the number of waiters on uaddr which are immediately woken up. Must be 1 for this opcode.
The timeout argument is abused to transport the number of waiters which are requeued on to the futex at uaddr2. The pointer is typecasted to u32.
Darren, can you fill in the missing details?
[EFAULT] Kernel was unable to access the futex value at uaddr
or uaddr2
[ENOMEM] Kernel could not allocate state
[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET] on uaddr
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr2 and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT on uaddr2.
[EINVAL] The supplied bitset is zero.
[EAGAIN] uaddr1 readout is not equal the compare value in
argument val3
[EAGAIN] The futex owner TID of uaddr2 is about to exit, but
has not yet handled the internal state cleanup. Try again.
[EPERM] Caller is not allowed to attach the waiter to the
futex at uaddr2 Can be a legitimate issue or a hint for state corruption in user space
[ESRCH] The TID in the user space value at uaddr2 does not exist
[EDEADLOCK] The requeuing of a waiter to the kernel representation
of the PI futex at uaddr2 detected a deadlock scenario.
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)
The various option bits seem to be undocumented as well
FUTEX_PRIVATE_FLAG
This option bit can be ored on all futex ops.
It tells the kernel, that the futex is process private and not shared with another process. That allows the kernel to chose the fast path for validating the user space address and avoids expensive VMA lookup, taking refcounts on file backing store etc.
FUTEX_CLOCK_REALTIME
This option bit can be ored on the futex ops FUTEX_WAIT_BITSET and FUTEX_WAIT_REQUEUE_PI
If set the kernel treats the user space supplied timeout as absolute time based on CLOCK_REALTIME.
If not set the kernel treats the user space supplied timeout as relative time.
If this is set on any other op than the supported ones, kernel returns ENOSYS!
Thanks,
tglx

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Carlos O'Donell

2014-08-04 14:46 UTC #12 lore raw

Quoted 10 lines · Michael Kerrisk (man-pages)

Quoted 6 lines · Thomas Gleixner
Quoted 2 lines · Michael Kerrisk (man-pages)
And that universe would love to have your documentation of FUTEX_WAKE_BITSET and FUTEX_WAIT_BITSET ;-),
I give you almost the full treatment, but I leave REQUEUE_PI to Darren and FUTEX_WAKE_OP to Jakub. :)
Thanks Thomas--that's fantastic! Hopefully, Darren and Jakub fill in those missing pieces...

Michael,

Do you need any help getting these additional futex error codes into the linux kernel man pages project? Thomas provided the missing bits and Darren commented... what else do we need?

I'm asking because I want to point other Red Hat engineers at these pages to say: "these are the canonical error codes."

We're trying to cleanup the userspace side of things.

Cheers,
Carlos.

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Darren Hart

2014-05-15 20:35 UTC #13 lore raw

http://lwn.net/images/conf/rtlws11/papers/proc/p10.pdf

On 5/15/14, 7:14, "Thomas Gleixner" <[email protected]> wrote:

Wow Thomas, I planned to do exactly this and you beat me to it. Again. Thanks for getting this started.

Michael, I imagine you want something more condensed, and I'll add to what tglx posted (inline below) to try and get you that, but if you have questions and need to fill in the gap, the paper I presented at RTLWS11 in '09 covers this particularly nasty OPCODE in detail:

I believe Michael is looking for some higher level documentation, like how to use these and what they are intended for. Probably something more like Ulrich's Futexes are Tricky paper - but let's start with getting the op codes, arguments, and return codes fleshed out.

For all the PI opcodes, we should probably mention something about the futex value scheme (TID), whereas the other opcodes do not require any specific value scheme.

No Owner: 0
Owner: TID
Waiters: TID | FUTEX_WAITERS

This is the relevant section from the referenced paper:

The PI futex operations diverge from the oth-
ers in that they impose a policy describing how
the futex value is to be used. If the lock is un-
owned, the futex value shall be 0. If owned, it
shall be the thread id (tid) of the owning thread.
If there are threads contending for the lock, then
the FUTEX_WAITERS flag is set. With this policy in
place, userspace can atomically acquire an unowned
lock or release an uncontended lock using an atomic
instruction and their own tid. A non-zero futex
value will force waiters into the kernel to lock. The
FUTEX_WAITERS flag forces the owner into the kernel
to unlock. If the callers are forced into the kernel,
they then deal directly with an underlying rt_mutex
which implements the priority inheritance semantics.
After the rt_mutex is acquired, the futex value is up-
dated accordingly, before the calling thread returns
to userspace.

It is important to note that the kernel will update the futex value prior to returning to userspace. Unlike other futex op codes, FUTEX_CMP_REUQUE_PI (and FUTEX_WAIT_REQUEUE_PI, FUTEX_LOCK_PI are designed for the implementation of very specific IPC mechanisms).

Quoted 5 lines

FUTEX_CMP_REQUEUE_PI
PI aware variant of FUTEX_CMP_REQUEUE. Inner futex at uaddr is a non PI futex. Outer futex to which is requeued is a PI futex at uaddr2.

Inner/outer terminology applies specifically to the glibc pthread condition variable and mutex use case, but is overly specific for the man page. Consider:

PI aware variant for FUTEX_CMP_REQUEUE. Requeue tasks blocked on uaddr via FUTEX_WAIT_REQUEUE_PI from a non-PI source futex (uaddr) to a PI target futex (uaddr2).

Quoted 5 lines

The waiters on uaddr must wait in FUTEX_WAIT_REQUEUE_PI.
The argument val is contains the number of waiters on uaddr which are immediately woken up. Must be 1 for this opcode.

Because the point is to avoid the thundering herd in the first place, and other nasty little races and faulting corner cases...

Quoted 4 lines

The timeout argument is abused to transport the number of waiters which are requeued on to the futex at uaddr2. The pointer is typecasted to u32.

val3 contains the expected value of uaddr (same as
FUTEX_CMP_REQUEUE)

Quoted 2 lines

Darren, can you fill in the missing details?

Yup...

Quoted 9 lines

[EFAULT] Kernel was unable to access the futex value at uaddr
or uaddr2
[ENOMEM] Kernel could not allocate state
[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned

Quoted 6 lines

[EINVAL] uaddr equal uaddr2. Requeue to same futex.
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

instead of FUTEX_WAIT_REQUEUE_PI.

Quoted 10 lines

[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET] on uaddr
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr2 and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT on uaddr2.

[EINVAL] The kernel detected the FUTEX_CMP_REQUEUE_PI call is
attempting to requeue a task to a futex other than that specified by the matching FUTEX_WAIT_REQUEUE_PI call for that task.

A number of these EINVALs can probably be combined into "Kernel detected bad state" as far as the C library is concerned, but we can consolidate later. But basically, EINVAL is returned if the non-pi to pi or op pairing semantics are violated.

Quoted 2 lines

[EINVAL] The supplied bitset is zero.

Bitset doesn't apply to FUTEX_CMP_REQUEUE_PI.

          [EINVAL] nr_wake != 1

EAGAIN == EWOULDBLOCK. We use each in the kernel, but will just refer to them here as EAGAIN.

Quoted 7 lines

[EAGAIN] uaddr1 readout is not equal the compare value in
argument val3
[EAGAIN] The futex owner TID of uaddr2 is about to exit, but
has not yet handled the internal state cleanup. Try again.

Quoted 5 lines

[EPERM] Caller is not allowed to attach the waiter to the
futex at uaddr2 Can be a legitimate issue or a hint for state corruption in user space
[ESRCH] The TID in the user space value at uaddr2 does not exist

Hrm, I'm missing ESRCH and EPERM in my state diagrams.... put yes, we can get ESRCH when looking up PI state, and we can return that from futex_requeue.... That needs some time to review...

I'm not seeing the EPERM path, where is that coming from?

Quoted 6 lines

[EDEADLOCK] The requeuing of a waiter to the kernel representation
of the PI futex at uaddr2 detected a deadlock scenario.
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)

Return value >= 0 is successful, indicating the number of of tasks requeued or woken (3 requeued and 1 woken would return 4).

Thanks,

Signature · 2 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

Michael Kerrisk (man-pages)

author

2015-01-15 15:12 UTC #14 lore raw

Hello Darren,

I give you the same apology as to Thomas for the
long-delayed response to your mail.

And I repeat my note to Thomas:
In the next day or two, I hope to send out the new version of the futex(2) page for review. The new draft is a bit bigger (okay -- 4 x bigger) than the current page. And there are a quite number of FIXMEs that I've placed in the page for various points--some minor, but a few major--that need to be checked or fixed. Would you have some time to review that page?

In the meantime, I have a couple of questions, which, if you could answer them, I would work some changes into the page before sending.

1. In various places, distinction is made between non-PI futexs and PI futexes. But what determines that distinction? From the kernel's perspective, hat make a futex one type or another? I presume it is to do with the types of blocking waiters on the futex, but it would be good to have a formal definition.

2. Can you say something about the pairing requirements of FUTEX_WAIT_REQUEUE_PI and FUTEX_CMP_REQUEUE_PI. What is the requirement and why do we need it?

Most of the rest of this mail is just a checklist noting what I did with your comments. No response is needed in most cases, but there is one that I have marked with "???". If you could reply to that. I'd be grateful.

Quoted 10 lines · Darren Hart

On 5/15/14, 7:14, "Thomas Gleixner" <[email protected]> wrote:
Wow Thomas, I planned to do exactly this and you beat me to it. Again. Thanks for getting this started.
Michael, I imagine you want something more condensed, and I'll add to what tglx posted (inline below) to try and get you that, but if you have questions and need to fill in the gap, the paper I presented at RTLWS11 in '09 covers this particularly nasty OPCODE in detail:

Quoted 4 lines

http://lwn.net/images/conf/rtlws11/papers/proc/p10.pdf
I believe Michael is looking for some higher level documentation, like how to use these and what they are intended for.

Yes, that would be good.

Quoted 3 lines

Probably something more like
Ulrich's Futexes are Tricky paper - but let's start with getting the op codes, arguments, and return codes fleshed out.

Okay.

Quoted 29 lines

For all the PI opcodes, we should probably mention something about the futex value scheme (TID), whereas the other opcodes do not require any specific value scheme.
No Owner: 0
Owner: TID
Waiters: TID | FUTEX_WAITERS
This is the relevant section from the referenced paper:
The PI futex operations diverge from the oth- ers in that they impose a policy describing how the futex value is to be used. If the lock is un- owned, the futex value shall be 0. If owned, it shall be the thread id (tid) of the owning thread. If there are threads contending for the lock, then the FUTEX_WAITERS flag is set. With this policy in place, userspace can atomically acquire an unowned lock or release an uncontended lock using an atomic instruction and their own tid. A non-zero futex value will force waiters into the kernel to lock. The FUTEX_WAITERS flag forces the owner into the kernel to unlock. If the callers are forced into the kernel, they then deal directly with an underlying rt_mutex which implements the priority inheritance semantics. After the rt_mutex is acquired, the futex value is up- dated accordingly, before the calling thread returns to userspace.

Quoted 4 lines

It is important to note that the kernel will update the futex value prior to returning to userspace. Unlike other futex op codes, FUTEX_CMP_REUQUE_PI (and FUTEX_WAIT_REQUEUE_PI, FUTEX_LOCK_PI are designed for the implementation of very specific IPC mechanisms).

??? Great text. May I presume that I can take this text and freely adapt it for the man page? (Actually, this is a request for forgiveness, rather than permission :-).)

Quoted 6 lines

Quoted 5 lines
FUTEX_CMP_REQUEUE_PI
PI aware variant of FUTEX_CMP_REQUEUE. Inner futex at uaddr is a non PI futex. Outer futex to which is requeued is a PI futex at uaddr2.

Quoted 7 lines

Inner/outer terminology applies specifically to the glibc pthread condition variable and mutex use case, but is overly specific for the man page. Consider:
PI aware variant for FUTEX_CMP_REQUEUE. Requeue tasks blocked on uaddr via FUTEX_WAIT_REQUEUE_PI from a non-PI source futex (uaddr) to a PI target futex (uaddr2).

Thanks for that text. It is easier to grasp.

Quoted 8 lines

Quoted 5 lines
The waiters on uaddr must wait in FUTEX_WAIT_REQUEUE_PI.
The argument val is contains the number of waiters on uaddr which are immediately woken up. Must be 1 for this opcode.
Because the point is to avoid the thundering herd in the first place, and other nasty little races and faulting corner cases...

I added the piece about "thundering herd".

Quoted 7 lines

Quoted 3 lines
The timeout argument is abused to transport the number of waiters which are requeued on to the futex at uaddr2. The pointer is typecasted to u32.
val3 contains the expected value of uaddr (same as
FUTEX_CMP_REQUEUE)

Yes. (The text now says that 'val3' has the same purpose as for FUTEX_CMP_REQUEUE.)

Quoted 13 lines

Quoted 1 line
Darren, can you fill in the missing details?
Yup...
Quoted 9 lines
[EFAULT] Kernel was unable to access the futex value at uaddr
or uaddr2
[ENOMEM] Kernel could not allocate state
[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a
valid object, i.e. pointer is not 4 byte aligned

Quoted 8 lines

Quoted 6 lines
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr
instead of FUTEX_WAIT_REQUEUE_PI.

Thanks. I added that detail.

Quoted 10 lines

Quoted 9 lines
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET] on uaddr
[EINVAL] The kernel detected inconsistent state between the
user space state at uaddr2 and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT on uaddr2.

Quoted 4 lines

[EINVAL] The kernel detected the FUTEX_CMP_REQUEUE_PI call is
attempting to requeue a task to a futex other than that specified by the matching FUTEX_WAIT_REQUEUE_PI call for that task.

Thanks. Added.

Quoted 4 lines

A number of these EINVALs can probably be combined into "Kernel detected bad state" as far as the C library is concerned, but we can consolidate later. But basically, EINVAL is returned if the non-pi to pi or op pairing semantics are violated.

I think the page probably needs some text to cover that point. I'll add a FIXME for review.

Quoted 3 lines

Quoted 1 line
[EINVAL] The supplied bitset is zero.
Bitset doesn't apply to FUTEX_CMP_REQUEUE_PI.

Thanks.

Quoted 1 line

          [EINVAL] nr_wake != 1

Thanks, I'd already spotted this, but it's good to have confirmation.

Quoted 2 lines

EAGAIN == EWOULDBLOCK. We use each in the kernel, but will just refer to them here as EAGAIN.

Yes. And I've followed that convention now in the man page.

Quoted 11 lines

Quoted 11 lines
[EAGAIN] uaddr1 readout is not equal the compare value in
argument val3
[EAGAIN] The futex owner TID of uaddr2 is about to exit, but
has not yet handled the internal state cleanup. Try again.
[EPERM] Caller is not allowed to attach the waiter to the
futex at uaddr2 Can be a legitimate issue or a hint for state corruption in user space

Quoted 7 lines

Quoted 1 line
[ESRCH] The TID in the user space value at uaddr2 does not exist
Hrm, I'm missing ESRCH and EPERM in my state diagrams.... put yes, we can get ESRCH when looking up PI state, and we can return that from futex_requeue.... That needs some time to review...
I'm not seeing the EPERM path, where is that coming from?

Any further insight on the above?

Quoted 8 lines

Quoted 5 lines
[EDEADLOCK] The requeuing of a waiter to the kernel representation
of the PI futex at uaddr2 detected a deadlock scenario.
[ENOSYS] Not implemented on all architectures and not supported
on some CPU variants (runtime detection)
Return value >= 0 is successful, indicating the number of of tasks requeued or woken (3 requeued and 1 woken would return 4).

Yes. Already noted.

Cheers,

Michael

Signature · 7 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Darren Hart

2015-01-17 01:33 UTC #15 lore raw

Corrected Davidlohr's email address.

Quoted 5 lines · Michael Kerrisk (man-pages)

Hello Darren,
I give you the same apology as to Thomas for the long-delayed response to your mail.

Quoted 8 lines

And I repeat my note to Thomas:
In the next day or two, I hope to send out the new version of the futex(2) page for review. The new draft is a bit bigger (okay -- 4 x bigger) than the current page. And there are a quite number of FIXMEs that I've placed in the page for various points--some minor, but a few major--that need to be checked or fixed. Would you have some time to review that page?

I'll make the time for that. I've wanted to see this for a while, so thank you for working on it!

Quoted 6 lines

In the meantime, I have a couple of questions, which, if you could answer them, I would work some changes into the page before sending.

Quoted 6 lines

1. In various places, distinction is made between non-PI futexs and PI futexes. But what determines that distinction? From the kernel's perspective, hat make a futex one type or another? I presume it is to do with the types of blocking waiters on the futex, but it would be good to have a formal definition.

You're right in that a uaddr is a uaddr is a uaddr. Also "there is no such thing as a futex", it doesn't exist as any kind of identifiable object, so these discussions can get rather confusing :-)

A "futex" becomes a PI futex when it is "created" via a PI futex op code. At that point, the syscall will ensure a pi_state is populated for the futex_q entry. See futex_lock_pi() for example. Before the locks are taken, there is a call to refill_pi_state_cache() which preps a pi_state for assignment later in futex_lock_pi_atomic(). This pi_state provides the necessary linkage to perform the priority boosting in the event of a priority inversion. This is handled externally from the futexes via the rt_mutex construct.

Clear as mud?

Quoted 4 lines

2. Can you say something about the pairing requirements of FUTEX_WAIT_REQUEUE_PI and FUTEX_CMP_REQUEUE_PI. What is the requirement and why do we need it?

Briefly, these op codes exist to support a fairly specific use case:
support for PI aware pthread condvars (glibc patch acceptance STILL PENDING FOR LOVE OF EVERYTHING HOLY WHY?!?!?! But is shipped with various PREEMPT_RT enabled Linux systems. Because these calls are paired, and more of the logic can happen on the kernel side (to preserve ownership of an rt_mutex with waiters), so in order to ensure userspace and kernelspace remain in sync, we pre-specify the target of the requeue in futex_wait_requeue_pi. This also limits the attack surface by only supporting exactly what it was meant to do. The corner cases get insane otherwise.

We could walk through the various ways in which it would break if these pairing restrictions were not in place, but I'll have to take some serious time to page all those into working memory. Let me know if we need more detail here and I will.

Quoted 5 lines

Most of the rest of this mail is just a checklist noting what I did with your comments. No response is needed in most cases, but there is one that I have marked with "???". If you could reply to that. I'd be grateful.

...

Quoted 36 lines

Quoted 29 lines
For all the PI opcodes, we should probably mention something about the futex value scheme (TID), whereas the other opcodes do not require any specific value scheme.
No Owner: 0
Owner: TID
Waiters: TID | FUTEX_WAITERS
This is the relevant section from the referenced paper:
The PI futex operations diverge from the oth- ers in that they impose a policy describing how the futex value is to be used. If the lock is un- owned, the futex value shall be 0. If owned, it shall be the thread id (tid) of the owning thread. If there are threads contending for the lock, then the FUTEX_WAITERS flag is set. With this policy in place, userspace can atomically acquire an unowned lock or release an uncontended lock using an atomic instruction and their own tid. A non-zero futex value will force waiters into the kernel to lock. The FUTEX_WAITERS flag forces the owner into the kernel to unlock. If the callers are forced into the kernel, they then deal directly with an underlying rt_mutex which implements the priority inheritance semantics. After the rt_mutex is acquired, the futex value is up- dated accordingly, before the calling thread returns to userspace.
Quoted 6 lines
It is important to note that the kernel will update the futex value prior
to returning to userspace. Unlike other futex op codes, FUTEX_CMP_REUQUE_PI (and FUTEX_WAIT_REQUEUE_PI, FUTEX_LOCK_PI are designed
for the implementation of very specific IPC mechanisms).

Quoted 3 lines

??? Great text. May I presume that I can take this text and freely adapt it for the man page? (Actually, this is a request for forgiveness, rather than permission :-).)

Thanks, and no objection from me.

--
Darren Hart
Intel Open Source Technology Center

Michael Kerrisk (man-pages)

author

2015-01-17 09:16 UTC #16 lore raw

Hello Darren,

Quoted 1 line · Darren Hart

Corrected Davidlohr's email address.

Thanks!

Quoted 17 lines

Quoted 5 lines · Michael Kerrisk (man-pages)
Hello Darren,
I give you the same apology as to Thomas for the long-delayed response to your mail.
Quoted 8 lines
And I repeat my note to Thomas:
In the next day or two, I hope to send out the new version of the futex(2) page for review. The new draft is a bit bigger (okay -- 4 x bigger) than the current page. And there are a quite number of FIXMEs that I've placed in the page for various points--some minor, but a few major--that need to be checked or fixed. Would you have some time to review that page?

Quoted 2 lines

I'll make the time for that. I've wanted to see this for a while, so thank you for working on it!

Great!

Quoted 11 lines

Quoted 10 lines
In the meantime, I have a couple of questions, which, if you could answer them, I would work some changes into the page before sending.
1. In various places, distinction is made between non-PI futexs and PI futexes. But what determines that distinction? From the kernel's perspective, hat make a futex one type or another? I presume it is to do with the types of blocking waiters on the futex, but it would be good to have a formal definition.

Quoted 3 lines

You're right in that a uaddr is a uaddr is a uaddr. Also "there is no such thing as a futex", it doesn't exist as any kind of identifiable object, so these discussions can get rather confusing :-)

So, I want to make sure that I am clear on what you mean you say this. You say "there is no such thing as a futex" because from the kernel's perspective there is no visible entity in the uncontended case (where everything can be dealt with in user space). And from user-space, in the uncontended case all we're doing is memory operations. Right?

On the other hand, from a kernel perspective, we could say that a futex "exists" in the contended phases, since the kernel has allocated state associated with the uaddr. Right?

Quoted 1 line

A "futex" becomes a PI futex when it is "created" via a PI futex op code.

Precisely which PI op codes? Is it: FUTEX_LOCK_PI, FUTEX_TRYLOCK_PI, and FUTEX_CMP_REQUEUE_PI, and not FUTEX_WAIT_REQUEUE_PI or FUTEX_UNLOCK_PI?

Quoted 9 lines

At that point, the syscall will ensure a pi_state is populated for the futex_q entry. See futex_lock_pi() for example. Before the locks are taken, there is a call to refill_pi_state_cache() which preps a pi_state for assignment later in futex_lock_pi_atomic(). This pi_state provides the necessary linkage to perform the priority boosting in the event of a priority inversion. This is handled externally from the futexes via the rt_mutex construct.
Clear as mud?

Not quite that bad, but... The thing is, still, the man page has text such as the following (based on your wording):

FUTEX_CMP_REQUEUE_PI (since Linux 2.6.31)
This operation is a PI-aware variant of FUTEX_CMP_REQUEUE.
It requeues waiters that are blocked via
FUTEX_WAIT_REQUEUE_PI on uaddr from a non-PI source futex (uaddr) to a PI target futex (uaddr2).

And elsewhere you said

EINVAL is returned if the non-pi to pi or
op pairing semantics are violated.

When someone in user-land (e.g., me) reads pieces like that, they then want to find somewhere in the man page a description of what makes a futex a *PI futex* and probably some statements of the distinction between PI and non-PI futexes. And those statements should be from a perspective that is somewhat comprehensible to user-space. I'm not yet confident that I can do that. Do you care to take a shot at it?

Quoted 7 lines

Quoted 3 lines
2. Can you say something about the pairing requirements of FUTEX_WAIT_REQUEUE_PI and FUTEX_CMP_REQUEUE_PI. What is the requirement and why do we need it?
Briefly, these op codes exist to support a fairly specific use case:
support for PI aware pthread condvars (glibc patch acceptance STILL PENDING FOR LOVE OF EVERYTHING HOLY WHY?!?!?!

Yes, Jan Kiszka recently alerted me to the existence of
https://sourceware.org/bugzilla/show_bug.cgi?id=11588
and I still have some text that you proposed (mail titled ("Pthread Condition Variables and Priority Inversion") quite a long time ago for the pthread_cond_timedwait() page. One day, when that page exists, I'll try to remember to add it.

Quoted 8 lines

But is shipped with various
PREEMPT_RT enabled Linux systems. Because these calls are paired, and more of the logic can happen on the kernel side (to preserve ownership of an rt_mutex with waiters), so in order to ensure userspace and kernelspace remain in sync, we pre-specify the target of the requeue in futex_wait_requeue_pi. This also limits the attack surface by only supporting exactly what it was meant to do. The corner cases get insane otherwise.

Thanks. I've added some text on pairing, based on your text above.

Quoted 4 lines

We could walk through the various ways in which it would break if these pairing restrictions were not in place, but I'll have to take some serious time to page all those into working memory. Let me know if we need more detail here and I will.

I don't think we need that much level of detail.

Quoted 43 lines

Quoted 4 lines
Most of the rest of this mail is just a checklist noting what I did with your comments. No response is needed in most cases, but there is one that I have marked with "???". If you could reply to that. I'd be grateful.
...
Quoted 36 lines
Quoted 35 lines
For all the PI opcodes, we should probably mention something about the futex value scheme (TID), whereas the other opcodes do not require any specific value scheme.
No Owner: 0
Owner: TID
Waiters: TID | FUTEX_WAITERS
This is the relevant section from the referenced paper:
The PI futex operations diverge from the oth- ers in that they impose a policy describing how the futex value is to be used. If the lock is un- owned, the futex value shall be 0. If owned, it shall be the thread id (tid) of the owning thread. If there are threads contending for the lock, then the FUTEX_WAITERS flag is set. With this policy in place, userspace can atomically acquire an unowned lock or release an uncontended lock using an atomic instruction and their own tid. A non-zero futex value will force waiters into the kernel to lock. The FUTEX_WAITERS flag forces the owner into the kernel to unlock. If the callers are forced into the kernel, they then deal directly with an underlying rt_mutex which implements the priority inheritance semantics. After the rt_mutex is acquired, the futex value is up- dated accordingly, before the calling thread returns to userspace.
It is important to note that the kernel will update the futex value prior
to returning to userspace. Unlike other futex op codes, FUTEX_CMP_REUQUE_PI (and FUTEX_WAIT_REQUEUE_PI, FUTEX_LOCK_PI are designed
for the implementation of very specific IPC mechanisms).

Quoted 5 lines

Quoted 3 lines
??? Great text. May I presume that I can take this text and freely adapt it for the man page? (Actually, this is a request for forgiveness, rather than permission :-).)
Thanks, and no objection from me.

Thanks.

Cheers,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Darren Hart

2015-01-17 19:26 UTC #17 lore raw

Quoted 53 lines · Michael Kerrisk (man-pages)

Hello Darren,
Quoted 1 line · Darren Hart
Corrected Davidlohr's email address.
Thanks!
Quoted 17 lines
Quoted 5 lines · Michael Kerrisk (man-pages)
Hello Darren,
I give you the same apology as to Thomas for the long-delayed response to your mail.
Quoted 8 lines
And I repeat my note to Thomas:
In the next day or two, I hope to send out the new version of the futex(2) page for review. The new draft is a bit bigger (okay -- 4 x bigger) than the current page. And there are a quite number of FIXMEs that I've placed in the page for various points--some minor, but a few major--that need to be checked or fixed. Would you have some time to review that page?
Quoted 3 lines
I'll make the time for that. I've wanted to see this for a while, so thank
you for working on it!
Great!
Quoted 11 lines
Quoted 10 lines
In the meantime, I have a couple of questions, which, if you could answer them, I would work some changes into the page before sending.
1. In various places, distinction is made between non-PI futexs and PI futexes. But what determines that distinction? From the kernel's perspective, hat make a futex one type or another? I presume it is to do with the types of blocking waiters on the futex, but it would be good to have a formal definition.
Quoted 5 lines
You're right in that a uaddr is a uaddr is a uaddr. Also "there is no such
thing as a futex", it doesn't exist as any kind of identifiable object, so
these discussions can get rather confusing :-)
So, I want to make sure that I am clear on what you mean you say this. You say "there is no such thing as a futex" because from the kernel's perspective there is no visible entity in the uncontended case (where everything can be dealt with in user space). And from user-space, in the uncontended case all we're doing is memory operations. Right?

Quoted 3 lines

On the other hand, from a kernel perspective, we could say that a futex "exists" in the contended phases, since the kernel has allocated state associated with the uaddr. Right?

Sorry, this was more anecdotal, and probably more of a distraction than constructive. I just meant that unlike other things which you can point to a specific struct for (task, rt_mutex, etc.), a "futex" has it's state distributed across the backing store (uaddr), the queue (futex_q), the pi_state, the rt_mutex, etc, and these span kernel space and userspace. Your description above is correct.

Quoted 6 lines

Quoted 3 lines
A "futex" becomes a PI futex when it is "created" via a PI futex op code.
Precisely which PI op codes? Is it: FUTEX_LOCK_PI, FUTEX_TRYLOCK_PI, and FUTEX_CMP_REQUEUE_PI, and not FUTEX_WAIT_REQUEUE_PI or FUTEX_UNLOCK_PI?

Based on your wording below about taking a user POV on this, I'm going to say "yes" here. These opcodes paired with the PI futex value policy (described below) defines a "futex" as PI aware. These were created very specifically in support of PI pthread_mutexes, so it makes a lot more sense to talk about a PI aware pthread_mutex, than a PI aware futex, since there is a lot of policy and scaffolding that has to be built up around it to use it properly (this is what a PI pthread_mutex is).

Quoted 25 lines

Quoted 10 lines
At that point, the syscall will ensure a pi_state is populated for the futex_q entry. See futex_lock_pi() for example. Before the locks are taken, there is a call to refill_pi_state_cache() which preps a pi_state for assignment later in futex_lock_pi_atomic(). This pi_state provides the
necessary linkage to perform the priority boosting in the event of a priority inversion. This is handled externally from the futexes via the rt_mutex construct.
Clear as mud?
Not quite that bad, but... The thing is, still, the man page has text such as the following (based on your wording):
FUTEX_CMP_REQUEUE_PI (since Linux 2.6.31)
This operation is a PI-aware variant of FUTEX_CMP_REQUEUE.
It requeues waiters that are blocked via
FUTEX_WAIT_REQUEUE_PI on uaddr from a non-PI source futex (uaddr) to a PI target futex (uaddr2).
And elsewhere you said
EINVAL is returned if the non-pi to pi or
op pairing semantics are violated.

Quoted 6 lines

When someone in user-land (e.g., me) reads pieces like that, they then want to find somewhere in the man page a description of what makes a futex a *PI futex* and probably some statements of the distinction between PI and non-PI futexes. And those statements should be from a perspective that is somewhat comprehensible to user-space. I'm not yet confident that I can do that. Do you care to take a shot at it?

Hrm, tricky indeed. From userspace, what makes a "futex" PI is the policy agreement between kernel and userspace (which is the value of the futex:
0, TID, TID|WAITERS, and never just WAITERS, and the use of PI aware futex op codes when making the futex syscalls.

For a longer discussion of this policy, see Documentation/pi-futex.txt. Also note that this policy can be combined with that for robust futexes, adding the OWNERDIED component.

--
Darren Hart
Intel Open Source Technology Center

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Michael Kerrisk (man-pages)

author

2015-01-18 10:18 UTC #18 lore raw

Hello Darren,

Quoted 3 lines · Darren Hart

On 1/17/15, 1:16 AM, "Michael Kerrisk (man-pages)" <[email protected]> wrote:

[...]

Quoted 28 lines

Quoted 23 lines
Quoted 11 lines
Quoted 10 lines
In the meantime, I have a couple of questions, which, if you could answer them, I would work some changes into the page before sending.
1. In various places, distinction is made between non-PI futexs and PI futexes. But what determines that distinction? From the kernel's perspective, hat make a futex one type or another? I presume it is to do with the types of blocking waiters on the futex, but it would be good to have a formal definition.
Quoted 5 lines
You're right in that a uaddr is a uaddr is a uaddr. Also "there is no such
thing as a futex", it doesn't exist as any kind of identifiable object, so
these discussions can get rather confusing :-)
So, I want to make sure that I am clear on what you mean you say this. You say "there is no such thing as a futex" because from the kernel's perspective there is no visible entity in the uncontended case (where everything can be dealt with in user space). And from user-space, in the uncontended case all we're doing is memory operations. Right?
Quoted 3 lines
On the other hand, from a kernel perspective, we could say that a futex "exists" in the contended phases, since the kernel has allocated state associated with the uaddr. Right?

Quoted 6 lines

Sorry, this was more anecdotal, and probably more of a distraction than constructive. I just meant that unlike other things which you can point to a specific struct for (task, rt_mutex, etc.), a "futex" has it's state distributed across the backing store (uaddr), the queue (futex_q), the pi_state, the rt_mutex, etc, and these span kernel space and userspace. Your description above is correct.

Okay. Thanks. I've added a few more words to the page noting that the kernel maintains no state for a futex in the uncontended state.

Quoted 6 lines

Quoted 5 lines
Quoted 2 lines
A "futex" becomes a PI futex when it is "created" via a PI futex op code.
Precisely which PI op codes? Is it: FUTEX_LOCK_PI, FUTEX_TRYLOCK_PI, and FUTEX_CMP_REQUEUE_PI, and not FUTEX_WAIT_REQUEUE_PI or FUTEX_UNLOCK_PI?

Quoted 7 lines

Based on your wording below about taking a user POV on this, I'm going to say "yes" here. These opcodes paired with the PI futex value policy (described below) defines a "futex" as PI aware. These were created very specifically in support of PI pthread_mutexes, so it makes a lot more sense to talk about a PI aware pthread_mutex, than a PI aware futex, since there is a lot of policy and scaffolding that has to be built up around it to use it properly (this is what a PI pthread_mutex is).

See below.

Quoted 32 lines

Quoted 25 lines
Quoted 10 lines
At that point, the syscall will ensure a pi_state is populated for the futex_q entry. See futex_lock_pi() for example. Before the locks are taken, there is a call to refill_pi_state_cache() which preps a pi_state for assignment later in futex_lock_pi_atomic(). This pi_state provides the
necessary linkage to perform the priority boosting in the event of a priority inversion. This is handled externally from the futexes via the rt_mutex construct.
Clear as mud?
Not quite that bad, but... The thing is, still, the man page has text such as the following (based on your wording):
FUTEX_CMP_REQUEUE_PI (since Linux 2.6.31)
This operation is a PI-aware variant of FUTEX_CMP_REQUEUE.
It requeues waiters that are blocked via
FUTEX_WAIT_REQUEUE_PI on uaddr from a non-PI source futex (uaddr) to a PI target futex (uaddr2).
And elsewhere you said
EINVAL is returned if the non-pi to pi or
op pairing semantics are violated.
Quoted 6 lines
When someone in user-land (e.g., me) reads pieces like that, they then want to find somewhere in the man page a description of what makes a futex a *PI futex* and probably some statements of the distinction between PI and non-PI futexes. And those statements should be from a perspective that is somewhat comprehensible to user-space. I'm not yet confident that I can do that. Do you care to take a shot at it?

Quoted 4 lines

Hrm, tricky indeed. From userspace, what makes a "futex" PI is the policy agreement between kernel and userspace (which is the value of the futex:
0, TID, TID|WAITERS, and never just WAITERS, and the use of PI aware futex op codes when making the futex syscalls.

Okay -- I've attempted to capture this in some text that I added to the page.

Quoted 1 line

For a longer discussion of this policy, see Documentation/pi-futex.txt.

Sad to say, that document doesn't supply that much more detail, in my reading of it, at least.

Quoted 2 lines

Also note that this policy can be combined with that for robust futexes, adding the OWNERDIED component.

Now there's two other stories that have yet to be dealt with ;-).

I have a FIXME already in the page regarding OWNERDIED, and get_robust_list(2) is another page that seems like it could do with a fair bit of work, but that's a story for another day.

Cheers,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Michael Kerrisk (man-pages)

author

2015-01-15 15:10 UTC #19 lore raw

[Adding a few people to CC that have expressed interest in the progress of the updates of this page, or who may be able to provide review feedback. Eventually, you'll all get CCed on the new draft of the page.]

Hello Thomas,

Quoted 6 lines · Thomas Gleixner

Quoted 2 lines · Michael Kerrisk (man-pages)
And that universe would love to have your documentation of FUTEX_WAKE_BITSET and FUTEX_WAIT_BITSET ;-),
I give you almost the full treatment, but I leave REQUEUE_PI to Darren and FUTEX_WAKE_OP to Jakub. :)

Thank you for the great effort you put into compiling the text below, and apologies for my long delay in following up.

I've integrated almost all of your suggestions into the manual page. I will shortly send out a new draft of the page that contains various FIXMEs for points that remain unclear.

Most of the rest of this mail is just a checklist noting what I did with your comments. No response is needed in most cases, but there are a very few open questions in this mail that, to help you find them, I have marked with "???". If you (or someone else) could reply to those, I would be grateful.

In the next day or two, I hope to send out the new version of the futex(2) page for review. The new draft is a bit bigger (okay -- 4 x bigger) than the current page. And there are a quite number of FIXMEs that I've placed in the page for various points--some minor, but a few major--that need to be checked or fixed. Would you have some time to review that page?

For that matter, if anyone else would have time for reviewing the page, could they shout out now. It's perhaps unlikely, but I am worried about getting a thundering herd of comments, and bringing the page to the state where I have it now has already been a fairly demanding task.

==========

Quoted 7 lines

FUTEX_WAIT
< Existing blurb seems ok >
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.

Added/reworked.

Quoted 2 lines

[EINVAL] The supplied uaddr argument does not pouint to a valid object, i.e. pointer is not 4 byte aligned

Added.

Quoted 1 line

[EINVAL] The supplied timeout argument is not normalized.

Added, but with more detail.

Quoted 1 line

[EWOULDBLOCK] The atomic enqueueing failed.

Added.

Note, however, that for consistency, I'll use EAGAIN throughout the page.

Quoted 2 lines

User space value at uaddr
is not equal val argument.

Was already present.

Quoted 1 line

[ETIMEDOUT] timeout expired

Was present, but I have now added more detail.

==========

Quoted 7 lines

FUTEX_WAKE
< Existing blurb seems ok >
Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.

Added/reworked.

Quoted 2 lines

[EINVAL] The supplied uaddr argument does not point to a valid object, i.e. pointer is not 4 byte aligned

Added.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI

Added.

==========

Quoted 9 lines

FUTEX_REQUEUE
Existing blurb seems ok , except for this:
The argument val contains the number of waiters on uaddr which are immediately woken up.
The timeout argument is abused to transport the number of waiters which are requeued to the futex at uaddr2. The pointer is typecasted to u32.

What I've actually done with the main text for FUTEX_REQUEUE is defer to the (now-expanded) discussion of FUTEX_CMP_REQUEUE.

Quoted 2 lines

[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2

Added/reworked.

Quoted 2 lines

[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a valid object, i.e. pointer is not 4 byte aligned

Added.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

Added.

Quoted 1 line

[EINVAL] uaddr equal uaddr2. Requeue to same futex.

??? I added this, but does this error not occur only for PI requeues?

==========

Quoted 3 lines

FUTEX_REQUEUE_CMP
Existing blurb seems ok , except for this:

[[

Quoted 6 lines

The argument val is contains the number of waiters on uaddr which are immediately woken up.
The timeout argument is abused to transport the number of waiters which are requeued to the futex at uaddr2. The pointer is typecasted to u32.

]]

Covered now (in more detail).

Quoted 4 lines

Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2

Added/reworked.

Quoted 2 lines

[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a valid object, i.e. pointer is not 4 byte aligned

Added.

Quoted 1 line

[EINVAL] uaddr equal uaddr2. Requeue to same futex.

Added.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

Added

Quoted 2 lines

[EAGAIN] uaddr1 readout is not equal the compare value in argument val3

Was already present.

==========

Quoted 4 lines

FUTEX_WAKE_OP
Jakub, can you please explain it? I'm lost :)

I had a read of Ulrich Drepper's "Futexes are Tricky", and the source code, and took a shot at it. I'd like to have someone check what I wrote though. See the draft that I will soon send out.

Quoted 2 lines

The argument val contains the maximum number of waiters on uaddr which are immediately woken up.

Covered in my new text.

Quoted 3 lines

The timeout argument is abused to transport the maximum number of waiters on uaddr2 which are woken up. The pointer is typecasted to u32.

Covered in my new text.

Quoted 4 lines

Related return values
[EFAULT] Kernel was unable to access the futex values at uaddr or uaddr2

This point was covered already in ERRORS.

Quoted 2 lines

[EINVAL] The supplied uaddr or uaddr2 argument does not point to a valid object, i.e. pointer is not 4 byte aligned

This point was covered already in ERRORS.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

I added this point.

==========

Quoted 5 lines

FUTEX_WAIT_BITSET
The same as FUTEX_WAIT except that val3 is used to provide a 32bit bitset to the kernel. This bitset is stored in the kernel internal state of the waiter.

Added.

Quoted 2 lines

This futex op also allows to have the option bit FUTEX_CLOCK_REALTIME set.

Added.

Quoted 3 lines

Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.

Already covered.

Quoted 2 lines

[EINVAL] The supplied uaddr argument does not point to a valid object, i.e. pointer is not 4 byte aligned

Already covered.

Quoted 1 line

[EINVAL] The supplied bitset is zero.

Added.

Quoted 1 line

[EINVAL] The supplied timeout argument is not normalized.

Already covered.

Quoted 1 line

[ETIMEDOUT] timeout expired

Already covered.

==========

Quoted 8 lines

FUTEX_WAKE_BITSET
The same as FUTEX_WAKE except that val3 is used to provide a 32bit bitset to the kernel. This bitset is used to select waiters on the futex. The selection is done by a bitwise AND of the wake side supplied bitset and the bitset which is stored in the kernel internal state of the waiters. If the result is non zero, the waiter is woken, otherwise left waiting.

Added (along with quite a bit of other detail).

Quoted 1 line

[EFAULT] Kernel was unable to access the futex value at uaddr.

Covered already.

Quoted 2 lines

[EINVAL] The supplied uaddr argument does not point to a valid object, i.e. pointer is not 4 byte aligned

Covered already.

Quoted 1 line

[EINVAL] The supplied bitset is zero.

Added.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI

Added.

==========

Quoted 15 lines

FUTEX_LOCK_PI
This operation reads from the futex address provided by the uaddr argument, which contains the namespace specific TID of the lock owner. If the TID is 0, then the kernel tries to set the waiters TID atomically. If the TID is nonzero or the take over fails the kernel sets atomically the FUTEX_WAITERS bit which signals the owner, that it cannot unlock the futex in user space atomically by transitioning from TID to 0. After that the kernel tries to find the task which is associated to the owner TID, creates or reuses kernel state on behalf of the owner and attaches the waiter to it. The enqueing of the waiter is in descending priority order if more than one waiter exists. The owner inherits either the priority or the bandwidth of the waiter. This inheritance follows the lock chain in the case of nested locking and performs deadlock detection.

Added.

Quoted 2 lines

The timeout argument is handled as described in FUTEX_WAIT. The arguments uaddr2, val, and val3 are ignored.

Added. Note, though, that some crufty code gives the impression that FUTEX_LOCK_PI uses 'val'. I'll send a patch separately.

Quoted 3 lines

Related return values
[EFAULT] Kernel was unable to access the futex value at uaddr.

Already covered.

Quoted 1 line

[ENOMEM] Kernel could not allocate state

Added

Quoted 2 lines

[EINVAL] The supplied uaddr argument does not point to a valid object, i.e. pointer is not 4 byte aligned

Already covered.

Quoted 1 line

[EINVAL] The supplied timeout argument is not normalized.

Already covered.

Quoted 4 lines

[EINVAL]
The kernel detected inconsistent state between the user space state at uaddr and the kernel state. Thats either state corruption or it found a waiter on uaddr which is waiting on FUTEX_WAIT[_BITSET]

Added.

Quoted 2 lines

[EPERM] Caller is not allowed to attach itself to the futex. Can be a legitimate issue or a hint for state corruption in user space

Added.

Quoted 1 line

[ESRCH] The TID in the user space value does not exist

Added.

Quoted 2 lines

[EAGAIN] The futex owner TID is about to exit, but has not yet handled the internal state cleanup. Try again.

Added.

Quoted 1 line

[ETIMEDOUT] timeout expired

Already covered.

Quoted 2 lines

[EDEADLOCK] The futex is already locked by the caller or the kernel detected a deadlock scenario in a nested lock chain

Added.

Quoted 3 lines

[EOWNERDIED] The owner of the futex died and the kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup

There is no such thing as an EOWNERDIED error. I had a look through the kernel source for the FUTEX_OWNER_DIED cases and didn't see an obvious error associated with them. Can you clarify? (I think the point is that this condition, which is described in Documentation/robust-futexes.txt, is not an error as such. However, I'm not yet sure of how to describe it in the man page.)
I will add this point as a FIXME in the new draft man page.

Quoted 2 lines

[ENOSYS] Not implemented on all architectures and not supported on some CPU variants (runtime detection)

Added.

==========

Quoted 5 lines

FUTEX_TRYLOCK_PI
This operation tries to acquire the futex at uaddr. It deals with the situation where the TID value at uaddr is 0, but the FUTEX_HAS_WAITER bit is set. User space cannot handle this race free.

Added.

Quoted 1 line

The arguments uaddr2, val, timeout and val3 are ignored.

??? But the code reads:

        case FUTEX_TRYLOCK_PI:
                return futex_lock_pi(uaddr, flags, 0, timeout, 1);

which momentarily misleads one into thinking that 'timeout' is used.
And: it's not quite ignored, since in futex_lock_pi() a non-NULL 'timeout' is unconditionally dereferenced (meaning you could get an EFAULT error for a bad 'timeout' pointer).
I'm confused....

Maybe the above code should be

        case FUTEX_TRYLOCK_PI:
                return futex_lock_pi(uaddr, flags, 0, NULL, 1);

Quoted 3 lines

Return values:
[EFAULT] Kernel was unable to access the futex value at uaddr.

Already covered.

Quoted 1 line

[ENOMEM] Kernel could not allocate state

Added.

Quoted 2 lines

[EINVAL] The supplied uaddr argument does not point to a valid object, i.e. pointer is not 4 byte aligned

Already covered.

Quoted 2 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state

Added, but with the same text as for FUTEX_LOCK_PI above. I.e., the text "Thats either state corruption or it found a waiter on uaddr which is waiting on FUTEX_WAIT[_BITSET]" is also included.

Quoted 2 lines

[EPERM] Caller is not allowed to attach itself to the futex. Can be a legitimate issue or a hint for state corruption in user space

Added.

Quoted 1 line

[ESRCH] The TID in the user space value does not exist

Added.

Quoted 2 lines

[EAGAIN] The futex owner TID is about to exit, but has not yet handled the internal state cleanup. Try again.

Added.

Quoted 1 line

[EDEADLOCK] The futex is already locked by the caller.

Added.

Quoted 3 lines

[EOWNERDIED] The owner of the futex died and the kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup

See comment above concerning EOWNERDIED for FUTEX_LOCK_PI

Quoted 2 lines

[ENOSYS] Not implemented on all architectures and not supported on some CPU variants (runtime detection)

Added.

==========

Quoted 9 lines

FUTEX_UNLOCK_PI
This operation wakes the top priority waiter which is waiting in FUTEX_LOCK_PI on the futex address provided by the uaddr argument.
This is called when the user space value at uaddr cannot be changed atomically from TID (of the owner) to 0.
The arguments uaddr2, val, timeout and val3 are ignored.

Added.

Quoted 4 lines

Related return values:
[EINVAL] The kernel detected inconsistent
state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET].

Added (but with a question in a FIXME).

Quoted 1 line

[EPERM] Caller does not own the futex.

Added.

Quoted 2 lines

[ENOSYS] Not implemented on all architectures and not supported on some CPU variants (runtime detection)

Added.

==========

Quoted 6 lines

FUTEX_WAIT_REQUEUE_PI
Wait operation to wait on a non pi futex at uaddr and potentially be requeued on a pi futex at uaddr2. The wait operation on uaddr is the same as FUTEX_WAIT. The waiter can be removed from the wait on uaddr via FUTEX_WAKE without requeuing on uaddr2.

Added.

Quoted 1 line

The timeout argument is handled as described in FUTEX_WAIT.

The above seems not to be correct. I've written the discussion of 'timeout' up as I understand it, and added a FIXME to the draft page.

Quoted 6 lines

Darren, can you fill in the missing details?
Return values:
[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2

Already covered.

Quoted 2 lines

[EINVAL] The supplied uaddr or uaddr2 argument does not point to a valid object, i.e. pointer is not 4 byte aligned

Already covered.

Quoted 1 line

[EINVAL] The supplied timeout argument is not normalized.

Already covered.

Quoted 1 line

[EINVAL] The supplied bitset is zero.

??? I don't believe this can happen. 'val3' is internally set to FUTEX_BITSET_MATCH_ANY. Can you confirm?

Quoted 2 lines

[EWOULDBLOCK] The atomic enqueueing failed. User space value at uaddr is not equal val argument.

Added using the same text as FUTEX_WAIT:

EAGAIN (FUTEX_WAIT, FUTEX_WAIT_REQUEUE_PI) The value pointed to
by uaddr was not equal to the expected value val at the time of the call.

Quoted 1 line

[ETIMEDOUT] timeout expired

Already covered.

Quoted 4 lines

[EOWNERDIED] The owner of the PI futex at uaddr2 died and the kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the uaddr2 futex userspace value. Caller is responsible for cleanup

See comment above concerning EOWNERDIED for FUTEX_LOCK_PI

Quoted 2 lines

[ENOSYS] Not implemented on all architectures and not supported on some CPU variants (runtime detection)

Added.

==========

Quoted 4 lines

FUTEX_CMP_REQUEUE_PI
PI aware variant of FUTEX_CMP_REQUEUE. Inner futex at uaddr is a non PI futex. Outer futex to which is requeued is a PI futex at uaddr2.

I instead used Darren's proposed text:

# PI aware variant for FUTEX_CMP_REQUEUE. Requeue tasks blocked on uaddr via # FUTEX_WAIT_REQUEUE_PI from a non-PI source futex (uaddr) to a PI target

# futex (uaddr2).

Quoted 1 line

The waiters on uaddr must wait in FUTEX_WAIT_REQUEUE_PI.

Covered above.

Quoted 2 lines

The argument val is contains the number of waiters on uaddr which are immediately woken up. Must be 1 for this opcode.

Added.

Quoted 3 lines

The timeout argument is abused to transport the number of waiters which are requeued on to the futex at uaddr2. The pointer is typecasted to u32.

Added.

Quoted 4 lines

Darren, can you fill in the missing details?
[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2

Already covered.

Quoted 1 line

[ENOMEM] Kernel could not allocate state

Added.

Quoted 2 lines

[EINVAL] The supplied uaddr/uaddr2 arguments do not point to a valid object, i.e. pointer is not 4 byte aligned

Already covered.

Quoted 1 line

[EINVAL] uaddr equal uaddr2. Requeue to same futex.

Added.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_LOCK_PI on uaddr

Added

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT[_BITSET] on uaddr

Added.

Quoted 3 lines

[EINVAL] The kernel detected inconsistent state between the user space state at uaddr2 and the kernel state, i.e. it detected a waiter which waits in FUTEX_WAIT on uaddr2.

Added.

Quoted 1 line

[EINVAL] The supplied bitset is zero.

Darren Hart noted: Bitset doesn't apply to FUTEX_CMP_REQUEUE_PI.

Quoted 2 lines

[EAGAIN] uaddr1 readout is not equal the compare value in argument val3

Added.

Quoted 2 lines

[EAGAIN] The futex owner TID of uaddr2 is about to exit, but has not yet handled the internal state cleanup. Try again.

Added.

Quoted 3 lines

[EPERM] Caller is not allowed to attach the waiter to the futex at uaddr2 Can be a legitimate issue or a hint for state corruption in user space

Added.

Quoted 1 line

[ESRCH] The TID in the user space value at uaddr2 does not exist

Added.

Quoted 2 lines

[EDEADLOCK] The requeuing of a waiter to the kernel representation of the PI futex at uaddr2 detected a deadlock scenario.

Added.

Quoted 2 lines

[ENOSYS] Not implemented on all architectures and not supported on some CPU variants (runtime detection)

Added.

==========

Quoted 1 line

The various option bits seem to be undocumented as well

Yes, thanks for that.

Quoted 1 line

FUTEX_PRIVATE_FLAG

I've added this one, along with the detail "(since Linux 2.6.22)"

Quoted 8 lines

This option bit can be ored on all futex ops.
It tells the kernel, that the futex is process private and not shared with another process. That allows the kernel to chose the fast path for validating the user space address and avoids expensive VMA lookup, taking refcounts on file backing store etc.
FUTEX_CLOCK_REALTIME

I've added this one, along with the detail "(since Linux 2.6.28)"

Quoted 11 lines

This option bit can be ored on the futex ops FUTEX_WAIT_BITSET and FUTEX_WAIT_REQUEUE_PI
If set the kernel treats the user space supplied timeout as absolute time based on CLOCK_REALTIME.
If not set the kernel treats the user space supplied timeout as relative time.
If this is set on any other op than the supported ones, kernel returns ENOSYS!

The details in the preceding 4 paragraphs have been integrated.

Thanks,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Thomas Gleixner

2015-01-15 22:23 UTC #20 lore raw

Quoted 3 lines · Michael Kerrisk (man-pages)

Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?

It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.

Quoted 4 lines

Quoted 2 lines
[EDEADLOCK] The futex is already locked by the caller or the kernel detected a deadlock scenario in a nested lock chain
Added.

It's actually EDEADLK

Quoted 5 lines

Quoted 4 lines
[EOWNERDIED] The owner of the futex died and the kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup

Quoted 7 lines

There is no such thing as an EOWNERDIED error. I had a look through the kernel source for the FUTEX_OWNER_DIED cases and didn't see an obvious error associated with them. Can you clarify? (I think the point is that this condition, which is described in Documentation/robust-futexes.txt, is not an error as such. However, I'm not yet sure of how to describe it in the man page.)
I will add this point as a FIXME in the new draft man page.

Oops. My bad. That's not the what the kernel does. The kernel merily marks it in the futex itself with FUTEX_OWNER_DIED. User space needs to deal with that and the posix users return EOWNERDEAD (not EOWNERDIED], so it's not part of the futex call itself.

We had discussions about returning EOWNERDEAD in that case, but then glibc with its sophisticated error handling prevented that ....

Quoted 12 lines

Quoted 5 lines
FUTEX_TRYLOCK_PI
This operation tries to acquire the futex at uaddr. It deals with the situation where the TID value at uaddr is 0, but the FUTEX_HAS_WAITER bit is set. User space cannot handle this race free.
Added.
Quoted 1 line
The arguments uaddr2, val, timeout and val3 are ignored.
??? But the code reads:

Quoted 8 lines

        case FUTEX_TRYLOCK_PI:
                return futex_lock_pi(uaddr, flags, 0, timeout, 1);
which momentarily misleads one into thinking that 'timeout' is used.
And: it's not quite ignored, since in futex_lock_pi() a non-NULL 'timeout' is unconditionally dereferenced (meaning you could get an EFAULT error for a bad 'timeout' pointer).
I'm confused....

Indeed. That's just wrong.

Quoted 5 lines

Maybe the above code should be

        case FUTEX_TRYLOCK_PI:
                return futex_lock_pi(uaddr, flags, 0, NULL, 1);

Care to send a patch?

Quoted 28 lines

Quoted 6 lines
FUTEX_WAIT_REQUEUE_PI
Wait operation to wait on a non pi futex at uaddr and potentially be requeued on a pi futex at uaddr2. The wait operation on uaddr is the same as FUTEX_WAIT. The waiter can be removed from the wait on uaddr via FUTEX_WAKE without requeuing on uaddr2.
Added.
Quoted 1 line
The timeout argument is handled as described in FUTEX_WAIT.
The above seems not to be correct. I've written the discussion of 'timeout' up as I understand it, and added a FIXME to the draft page.
Quoted 6 lines
Darren, can you fill in the missing details?
Return values:
[EFAULT] Kernel was unable to access the futex value at uaddr or uaddr2
Already covered.
Quoted 2 lines
[EINVAL] The supplied uaddr or uaddr2 argument does not point to a valid object, i.e. pointer is not 4 byte aligned
Already covered.

Quoted 8 lines

Quoted 1 line
[EINVAL] The supplied timeout argument is not normalized.
Already covered.
Quoted 1 line
[EINVAL] The supplied bitset is zero.
??? I don't believe this can happen. 'val3' is internally set to FUTEX_BITSET_MATCH_ANY. Can you confirm?

Right. We dont support that bitset stuff in requeue_pi ATM.

Thanks,

tglx

Michael Kerrisk (man-pages)

author

2015-01-16 15:17 UTC #21 lore raw

Hello Thomas,

Quoted 7 lines · Thomas Gleixner

Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.

I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:

a) This error text should be there for both normal and PI requeues OR
a) This error text should be there for neither normal nor PI requeues

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
[EDEADLOCK] The futex is already locked by the caller or the kernel detected a deadlock scenario in a nested lock chain
Added.
It's actually EDEADLK

Yes, sorry -- I should have said that I already found and fixed that problem.

Quoted 12 lines

Quoted 11 lines
Quoted 3 lines
[EOWNERDIED] The owner of the futex died and the kernel made the caller the new owner. The kernel sets the FUTEX_OWNER_DIED bit in the futex userspace value. Caller is responsible for cleanup
There is no such thing as an EOWNERDIED error. I had a look through the kernel source for the FUTEX_OWNER_DIED cases and didn't see an obvious error associated with them. Can you clarify? (I think the point is that this condition, which is described in Documentation/robust-futexes.txt, is not an error as such. However, I'm not yet sure of how to describe it in the man page.)
I will add this point as a FIXME in the new draft man page.

Quoted 7 lines

Oops. My bad. That's not the what the kernel does. The kernel merily marks it in the futex itself with FUTEX_OWNER_DIED. User space needs to deal with that and the posix users return EOWNERDEAD (not EOWNERDIED], so it's not part of the futex call itself.
We had discussions about returning EOWNERDEAD in that case, but then glibc with its sophisticated error handling prevented that ....

Okay. I'll add a FIXME to the draft page, to see if we get some good text together to describe FUTEX_OWNER_DIED and how it is used.

Quoted 23 lines

Quoted 12 lines
Quoted 5 lines
FUTEX_TRYLOCK_PI
This operation tries to acquire the futex at uaddr. It deals with the situation where the TID value at uaddr is 0, but the FUTEX_HAS_WAITER bit is set. User space cannot handle this race free.
Added.
Quoted 1 line
The arguments uaddr2, val, timeout and val3 are ignored.
??? But the code reads:
Quoted 8 lines
        case FUTEX_TRYLOCK_PI:
                return futex_lock_pi(uaddr, flags, 0, timeout, 1);
which momentarily misleads one into thinking that 'timeout' is used.
And: it's not quite ignored, since in futex_lock_pi() a non-NULL 'timeout' is unconditionally dereferenced (meaning you could get an EFAULT error for a bad 'timeout' pointer).
I'm confused....
Indeed. That's just wrong.

Quoted 7 lines

Quoted 5 lines
Maybe the above code should be
        case FUTEX_TRYLOCK_PI:
                return futex_lock_pi(uaddr, flags, 0, NULL, 1);
?
Care to send a patch?

Will do.

[...]

Quoted 4 lines

Quoted 2 lines
??? I don't believe this can happen. 'val3' is internally set to FUTEX_BITSET_MATCH_ANY. Can you confirm?
Right. We dont support that bitset stuff in requeue_pi ATM.

Thanks for the confirmation.

Cheers,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Thomas Gleixner

2015-01-16 15:20 UTC #22 lore raw

Quoted 8 lines · Michael Kerrisk (man-pages)

Hello Thomas,
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?

Quoted 7 lines

Quoted 2 lines
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:
a) This error text should be there for both normal and PI requeues

It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.

Thanks,

tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Michael Kerrisk (man-pages)

author

2015-01-16 20:54 UTC #23 lore raw

Quoted 16 lines · Thomas Gleixner

Quoted 14 lines · Michael Kerrisk (man-pages)
Hello Thomas,
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:

Quoted 7 lines

Quoted 1 line
a) This error text should be there for both normal and PI requeues
It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.

Hello Thomas,

Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?

When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.

Thanks,

Michael

Signature · 7 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Darren Hart

2015-01-17 00:46 UTC #24 lore raw

Quoted 25 lines · Michael Kerrisk (man-pages)

Quoted 16 lines · Thomas Gleixner
Quoted 14 lines · Michael Kerrisk (man-pages)
Hello Thomas,
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:
Quoted 7 lines
Quoted 1 line
a) This error text should be there for both normal and PI requeues
It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.

Quoted 7 lines

Hello Thomas,
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.

Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?). Both tests ensuring uaddr1 != uaddr2 are under the requeue_pi conditional block. The second compares the keys in case they are not FUTEX_PRIVATE (uaddrs would be different, but still the same backing store).

Thomas, am I missing a test for this someplace else?

Signature · 2 lines

Darren Hart
Intel Open Source Technology Center

Thomas Gleixner

2015-01-19 10:45 UTC #25 lore raw

Quoted 37 lines · Darren Hart

Quoted 25 lines · Michael Kerrisk (man-pages)
Quoted 16 lines · Thomas Gleixner
Quoted 14 lines · Michael Kerrisk (man-pages)
Hello Thomas,
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:
Quoted 7 lines
Quoted 1 line
a) This error text should be there for both normal and PI requeues
It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.
Quoted 7 lines
Hello Thomas,
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.

Quoted 7 lines

Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?). Both tests ensuring uaddr1 != uaddr2 are under the requeue_pi conditional block. The second compares the keys in case they are not FUTEX_PRIVATE (uaddrs would be different, but still the same backing store).
Thomas, am I missing a test for this someplace else?

No, I had a short look at the code misread it. So, yes, it's a valid operation for the non PI case. Sorry for the confusion.

Thanks,

tglx

Michael Kerrisk (man-pages)

author

2015-01-19 14:07 UTC #26 lore raw

Quoted 44 lines · Thomas Gleixner

Quoted 43 lines · Darren Hart
Quoted 25 lines · Michael Kerrisk (man-pages)
Quoted 16 lines · Thomas Gleixner
Quoted 14 lines · Michael Kerrisk (man-pages)
Hello Thomas,
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:
Quoted 7 lines
Quoted 1 line
a) This error text should be there for both normal and PI requeues
It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.
Quoted 7 lines
Hello Thomas,
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.
Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?). Both tests ensuring uaddr1 != uaddr2 are under the requeue_pi conditional block. The second compares the keys in case they are not FUTEX_PRIVATE (uaddrs would be different, but still the same backing store).

Quoted 4 lines

Quoted 1 line
Thomas, am I missing a test for this someplace else?
No, I had a short look at the code misread it. So, yes, it's a valid operation for the non PI case. Sorry for the confusion.

Thanks for the confirmation, Thomas. Page updated to remove FUTEX_CMP_REQUEUE from that error case.

Cheers,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Torvald Riegel

2015-01-23 18:19 UTC #27 lore raw

Quoted 6 lines · Darren Hart

Quoted 3 lines · Michael Kerrisk (man-pages)
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?

Quoted 6 lines

Quoted 2 lines
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.
Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?).

But we never guarantee a futex is a FIFO, or do we? If we don't, then such a requeue could be implemented as a no-op by the kernel, which would sort of invalidate the use case.

(And I guess we don't want to guarantee FIFO behavior for futexes.)

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thomas Gleixner

2015-01-24 10:05 UTC #28 lore raw

Quoted 11 lines · Torvald Riegel

Quoted 10 lines · Darren Hart
Quoted 5 lines · Michael Kerrisk (man-pages)
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.

Quoted 8 lines

Quoted 2 lines
Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?).
But we never guarantee a futex is a FIFO, or do we? If we don't, then such a requeue could be implemented as a no-op by the kernel, which would sort of invalidate the use case.
(And I guess we don't want to guarantee FIFO behavior for futexes.)

The (current) behaviour is:

real-time threads: FIFO per priority level
sched-other threads: FIFO independent of nice level

The wakeup is priority ordered. Highest priority level first.

Thanks,

tglx

Torvald Riegel

2015-01-24 12:58 UTC #29 lore raw

Quoted 20 lines · Thomas Gleixner

Quoted 18 lines · Torvald Riegel
Quoted 6 lines · Darren Hart
Quoted 3 lines · Michael Kerrisk (man-pages)
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
Quoted 6 lines
Quoted 2 lines
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.
Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?).
But we never guarantee a futex is a FIFO, or do we? If we don't, then such a requeue could be implemented as a no-op by the kernel, which would sort of invalidate the use case.

Quoted 8 lines

Quoted 1 line
(And I guess we don't want to guarantee FIFO behavior for futexes.)
The (current) behaviour is:
real-time threads: FIFO per priority level
sched-other threads: FIFO independent of nice level
The wakeup is priority ordered. Highest priority level first.

OK.

But, just to be clear, do I correctly understand that you do not want to guarantee FIFO behavior in the specified futex semantics? I think there are cases where being able to *rely* on FIFO (now and on all future kernels) would be helpful for users (e.g., on POSIX/C++11 condvars and I assume in other ordered-wakeup cases too) -- but at the same time, this would constrain future futex implementations.

Thomas Gleixner

2015-01-24 16:25 UTC #30 lore raw

Quoted 30 lines · Torvald Riegel

Quoted 20 lines · Thomas Gleixner
Quoted 18 lines · Torvald Riegel
Quoted 6 lines · Darren Hart
Quoted 3 lines · Michael Kerrisk (man-pages)
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
Quoted 6 lines
Quoted 2 lines
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.
Right, in the non-PI case, I believe there are valid use cases: move to the back of the FIFO, for example (OK, maybe the only example?).
But we never guarantee a futex is a FIFO, or do we? If we don't, then such a requeue could be implemented as a no-op by the kernel, which would sort of invalidate the use case.
Quoted 8 lines
Quoted 1 line
(And I guess we don't want to guarantee FIFO behavior for futexes.)
The (current) behaviour is:
real-time threads: FIFO per priority level
sched-other threads: FIFO independent of nice level
The wakeup is priority ordered. Highest priority level first.

Quoted 8 lines

OK.
But, just to be clear, do I correctly understand that you do not want to guarantee FIFO behavior in the specified futex semantics? I think there are cases where being able to *rely* on FIFO (now and on all future kernels) would be helpful for users (e.g., on POSIX/C++11 condvars and I assume in other ordered-wakeup cases too) -- but at the same time, this would constrain future futex implementations.

It would be a constraint, but I don't think it would be a horrible one. Though I have my doubts, that we can actually guarantee it under all circumstances.

One thing comes to my mind right away: spurious wakeups. There is no way that we can guarantee FIFO ordering in the context of those. And we cannot prevent them either.

Thanks,

tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Davidlohr Bueso

2015-01-17 00:56 UTC #31 lore raw

Quoted 25 lines · Michael Kerrisk (man-pages)

Quoted 16 lines · Thomas Gleixner
Quoted 14 lines · Michael Kerrisk (man-pages)
Hello Thomas,
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI requeues?
It's equally wrong for normal futexes. And its actually the same code checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:
Quoted 7 lines
Quoted 1 line
a) This error text should be there for both normal and PI requeues
It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.

Quoted 7 lines

Hello Thomas,
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.

Yeah, its not very straightforward, I was also scratching my head. First we do:

	if (requeue_pi) {
		/*
		 * Requeue PI only works on two distinct uaddrs. This
		 * check is only valid for private futexes. See below.
		 */
		if (uaddr1 == uaddr2)
			return -EINVAL;

Then:

	/*
	 * The check above which compares uaddrs is not sufficient for
	 * shared futexes. We need to compare the keys:
	 */
	if (requeue_pi && match_futex(&key1, &key2)) {
		ret = -EINVAL;
		goto out_put_keys;
	}

I wonder why we're checking for requeue_pi again, when, at least according to the comments, it should be for shared. I guess it would make sense depending on the mappings as the keys are the only true way of determining if both futexes are the same, so perhaps:

	if ((requeue_pi || (flags & FLAGS_SHARED)) && match_futex())

That would also align with the retry labels.

Thanks,
Davidlohr

Darren Hart

2015-01-17 01:11 UTC #32 lore raw

Quoted 39 lines · Davidlohr Bueso

Quoted 27 lines · Michael Kerrisk (man-pages)
Quoted 9 lines · Thomas Gleixner
Quoted 7 lines · Michael Kerrisk (man-pages)
Hello Thomas,
Quoted 4 lines · Thomas Gleixner
Quoted 3 lines · Michael Kerrisk (man-pages)
Quoted 1 line
[EINVAL] uaddr equal uaddr2. Requeue to same futex.
??? I added this, but does this error not occur only for PI
requeues?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
It's equally wrong for normal futexes. And its actually the same
code
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
checking for this for all variants.
I don't understand "equally wrong" in your reply, I'm sorry. Do you mean:
Quoted 7 lines
Quoted 1 line
a) This error text should be there for both normal and PI requeues
It is there for both. The requeue code has that check independent of the requeue type (normal/pi). It never makes sense to requeue something to itself whether normal or pi futex. We added this for PI, because there it is harmful, but we did not special case it. So normal futexes get the same treatment.
Quoted 7 lines
Hello Thomas,
Color me stupid, but I can't see this in futex_requeue(). Where is that check that is "independent of the requeue type (normal/pi)"?
When I look through futex_requeue(), all the likely looking sources of EINVAL are governed by a check on the 'requeue_pi' argument.
Yeah, its not very straightforward, I was also scratching my head. First we do:

Quoted 7 lines

	if (requeue_pi) {
		/*
		 * Requeue PI only works on two distinct uaddrs. This
		 * check is only valid for private futexes. See below.
		 */
		if (uaddr1 == uaddr2)
			return -EINVAL;

We check here to abort as early as possible for the usual security reasons.

Quoted 12 lines

Then:

	/*
	 * The check above which compares uaddrs is not sufficient for
	 * shared futexes. We need to compare the keys:
	 */
	if (requeue_pi && match_futex(&key1, &key2)) {
		ret = -EINVAL;
		goto out_put_keys;
	}

Quoted 6 lines

I wonder why we're checking for requeue_pi again, when, at least according to the comments, it should be for shared. I guess it would make sense depending on the mappings as the keys are the only true way of determining if both futexes are the same, so perhaps:
	if ((requeue_pi || (flags & FLAGS_SHARED)) && match_futex())

No, the rule only applies to requeue_pi. This check is the for-sure version of the uaddr comparison above. We could add if flags & FLAGS_SHARED, but I'm not sure it's worth it.

--
Darren Hart
Intel Open Source Technology Center

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Torvald Riegel

2015-01-23 18:29 UTC #33 lore raw

Quoted 15 lines · Michael Kerrisk (man-pages)

[Adding a few people to CC that have expressed interest in the progress of the updates of this page, or who may be able to provide review feedback. Eventually, you'll all get CCed on the new draft of the page.]
Hello Thomas,
Quoted 6 lines · Thomas Gleixner
Quoted 2 lines · Michael Kerrisk (man-pages)
And that universe would love to have your documentation of FUTEX_WAKE_BITSET and FUTEX_WAIT_BITSET ;-),
I give you almost the full treatment, but I leave REQUEUE_PI to Darren and FUTEX_WAKE_OP to Jakub. :)

Quoted 7 lines

Thank you for the great effort you put into compiling the text below, and apologies for my long delay in following up.
I've integrated almost all of your suggestions into the manual page. I will shortly send out a new draft of the page that contains various FIXMEs for points that remain unclear.

Michael, thanks for working on the draft! I'll review the draft closely once you've sent it (or have I missed it?).

There are a few things that I'd like to see covered.

First, we should discuss that users, until they control all code in the respective process, need to expect futexes to be affected by spurious futex_wake calls; see https://lkml.org/lkml/2014/11/27/472 for background and Linus' choice (AFAIU) to just document this. So, for example, a return code of 0 for FUTEX_WAIT can mean either being woken up by a FUTEX_WAKE intended for this futex, or a stale one intended for another futex used by, for example, glibc internally. (Note that as explained in this thread, this isn't just a glibc artifact, but a result of the general futex design mixed with destruction requirements for mutexes and other constructs in C++11 and POSIX.)
It might also be necessary to further consider this when documenting the errors, because it does affect how to handle them. See this for a glibc perspective:
https://sourceware.org/ml/libc-alpha/2014-09/msg00381.html

Second, the current documentation for EINTR is that it can happen due to receiving a signal *or* due to a spurious wake-up. This is difficult to handle when implementing POSIX semaphores, because they require that EINTR is returned from SEM_WAIT if and only if the interruption was due to a signal. Thus, if FUTEX_WAIT returns EINTR, the semaphore implementation can't return EINTR from sem_wait; see this for more comments, including some discussion why use cases relying on the POSIX requirement around EINTR are borderline timing-dependent:
https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/sem_waitcommon.c;h=96848d7ac5b6f8f1f3099b422deacc09323c796a;hb=HEAD#l282
Others have commented that aio_suspend has a similar issue; if EINTR wouldn't in fact be returned spuriously, the POSIX-implementation-side would get easier.

Third, I think it would be useful to -- somewhere -- explain which behavior the futex operations would have conceptually when expressed by C11 code. We currently say that they wake up, sleep, etc, and which values they return. But we never say how to properly synchronize with them on the userspace side. The C11 memory model is probably the best model to use on the userspace side, so that's why I'm arguing for this. Basically, I think we need to (1) tell people that they should use memory_order_relaxed accesses to the futex variable (ie, the memory location associated with the whole futex construct on the kernel side -- or do we have another name for this?), and (2) give some conceptual guarantees for the kernel-side synchronization so that one use this to derive how to use them correctly in userspace.

The man pages might not be the right place for this, and maybe we just need a revision of "Futexes are tricky". If you have other suggestions for where to document this, or on the content, let me know. (I'm also willing to spend time on this :) ).

Torvald

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thomas Gleixner

2015-01-24 11:35 UTC #34 lore raw

Quoted 2 lines · Torvald Riegel

Second, the current documentation for EINTR is that it can happen due to receiving a signal *or* due to a spurious wake-up. This is difficult to

I don't think so. I went through all callchains again with a fine comb.

futex_wait()
retry:

	ret = futex_wait_setup();
	if (ret) {
		 /*
		  * Possible return codes related to uaddr:
		  * -EINVAL:    Not u32 aligned uaddr
		  * -EFAULT:    No mapping, no RW
		  * -ENOMEM:    Paging ran out of memory
		  * -EHWPOISON: Memory hardware error
		  *
		  * Others:
		  * -EWOULDBLOCK: value at uaddr has changed
		  */
		return ret;
	}

	futex_wait_queue_me();

	if (woken by futex_wake/requeue)
	   	return 0;

	if (timeout)
		return -ETIMEOUT;

	/*
	 * Spurious wakeup, i.e. no signal pending
	 */
	if (!signal_pending())
		goto retry;

	/* Handled in the low level syscall exit code */
	if (!timed_wait)
		return -ERESTARTSYS;
	else
		return -ERESTARTBLOCK;

Now in the low level syscall exit we try to deliver the signal

	if (!signal_delivered())
	      restart_syscall();

	if (sigaction->flags & SA_RESTART)
	      restart_syscall();

	ret_to_userspace -EINTR;

So we should never see -EINTR in the case of a spurious wakeup here.

But, here is the not so good news:

I did some archaeology. The restart handling of futex_wait() got introduced in kernel 2.6.22, so anything older than that will have the spurious -EINTR issues.

futex_wait_pi() always had the restart handling and glibc folks back then (2006) requested that it should never return -EINTR, so it unconditionally restarts the syscall whether a signal had been delivered or not.

So kernels >= 2.6.22 should never return -EINTR spuriously. If that happens it's a bug and needs to be fixed.

Quoted 13 lines

Third, I think it would be useful to -- somewhere -- explain which behavior the futex operations would have conceptually when expressed by C11 code. We currently say that they wake up, sleep, etc, and which values they return. But we never say how to properly synchronize with them on the userspace side. The C11 memory model is probably the best model to use on the userspace side, so that's why I'm arguing for this. Basically, I think we need to (1) tell people that they should use memory_order_relaxed accesses to the futex variable (ie, the memory location associated with the whole futex construct on the kernel side -- or do we have another name for this?), and (2) give some conceptual guarantees for the kernel-side synchronization so that one use this to derive how to use them correctly in userspace.

Quoted 4 lines

The man pages might not be the right place for this, and maybe we just need a revision of "Futexes are tricky". If you have other suggestions for where to document this, or on the content, let me know. (I'm also willing to spend time on this :) ).

The current futex code in the kernel has gained documentation about the required memory ordering recently. That should be a good starting point.

Thanks,

tglx

Torvald Riegel

2015-01-24 13:12 UTC #35 lore raw

Quoted 8 lines · Thomas Gleixner

So we should never see -EINTR in the case of a spurious wakeup here.
But, here is the not so good news:
I did some archaeology. The restart handling of futex_wait() got introduced in kernel 2.6.22, so anything older than that will have the spurious -EINTR issues.

Quoted 7 lines

futex_wait_pi() always had the restart handling and glibc folks back then (2006) requested that it should never return -EINTR, so it unconditionally restarts the syscall whether a signal had been delivered or not.
So kernels >= 2.6.22 should never return -EINTR spuriously. If that happens it's a bug and needs to be fixed.

Thanks for looking into this.

Michael, can you include the above in the documentation please? This is useful for userspace code like glibc that expects a minimum kernel version. Thanks!

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Michael Kerrisk (man-pages)

author

2015-01-27 07:48 UTC #36 lore raw

Hello Torvald,

Quoted 14 lines · Torvald Riegel

Quoted 13 lines · Thomas Gleixner
So we should never see -EINTR in the case of a spurious wakeup here.
But, here is the not so good news:
I did some archaeology. The restart handling of futex_wait() got introduced in kernel 2.6.22, so anything older than that will have the spurious -EINTR issues.
futex_wait_pi() always had the restart handling and glibc folks back then (2006) requested that it should never return -EINTR, so it unconditionally restarts the syscall whether a signal had been delivered or not.

Quoted 8 lines

Quoted 2 lines
So kernels >= 2.6.22 should never return -EINTR spuriously. If that happens it's a bug and needs to be fixed.
Thanks for looking into this.
Michael, can you include the above in the documentation please? This is useful for userspace code like glibc that expects a minimum kernel version. Thanks!

I've added some text to my draft to cover this point.

Cheers,

Michael

Signature · 7 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Darren Hart

2015-02-05 19:57 UTC #37 lore raw

In reply to Carlos O'Donell

Quoted 82 lines · Thomas Gleixner

Quoted 2 lines · Torvald Riegel
Second, the current documentation for EINTR is that it can happen due to receiving a signal *or* due to a spurious wake-up. This is difficult to
I don't think so. I went through all callchains again with a fine comb.
futex_wait()
retry:
	ret = futex_wait_setup();
	if (ret) {
		 /*
		  * Possible return codes related to uaddr:
		  * -EINVAL:    Not u32 aligned uaddr
		  * -EFAULT:    No mapping, no RW
		  * -ENOMEM:    Paging ran out of memory
		  * -EHWPOISON: Memory hardware error
		  *
		  * Others:
		  * -EWOULDBLOCK: value at uaddr has changed
		  */
		return ret;
	}

	futex_wait_queue_me();

	if (woken by futex_wake/requeue)
	   	return 0;

	if (timeout)
		return -ETIMEOUT;

	/*
	 * Spurious wakeup, i.e. no signal pending
	 */
	if (!signal_pending())
		goto retry;

	/* Handled in the low level syscall exit code */
	if (!timed_wait)
		return -ERESTARTSYS;
	else
		return -ERESTARTBLOCK;
Now in the low level syscall exit we try to deliver the signal
	if (!signal_delivered())
	      restart_syscall();

	if (sigaction->flags & SA_RESTART)
	      restart_syscall();

	ret_to_userspace -EINTR;
So we should never see -EINTR in the case of a spurious wakeup here.
But, here is the not so good news:
I did some archaeology. The restart handling of futex_wait() got introduced in kernel 2.6.22, so anything older than that will have the spurious -EINTR issues.
futex_wait_pi() always had the restart handling and glibc folks back then (2006) requested that it should never return -EINTR, so it unconditionally restarts the syscall whether a signal had been delivered or not.
So kernels >= 2.6.22 should never return -EINTR spuriously. If that happens it's a bug and needs to be fixed.
Quoted 13 lines
Third, I think it would be useful to -- somewhere -- explain which behavior the futex operations would have conceptually when expressed by C11 code. We currently say that they wake up, sleep, etc, and which values they return. But we never say how to properly synchronize with them on the userspace side. The C11 memory model is probably the best model to use on the userspace side, so that's why I'm arguing for this. Basically, I think we need to (1) tell people that they should use memory_order_relaxed accesses to the futex variable (ie, the memory location associated with the whole futex construct on the kernel side -- or do we have another name for this?), and (2) give some conceptual guarantees for the kernel-side synchronization so that one use this to derive how to use them correctly in userspace.

Quoted 8 lines

Quoted 4 lines
The man pages might not be the right place for this, and maybe we just need a revision of "Futexes are tricky". If you have other suggestions for where to document this, or on the content, let me know. (I'm also willing to spend time on this :) ).
The current futex code in the kernel has gained documentation about the required memory ordering recently. That should be a good starting point.

Lots of paging in here... If I recall correctly there was something about not being able to return to userspace in these events without owning the lock (waiters but no owner, breaking pi chains and promotion, etc.), so restarting was the preferable path.

Signature · 2 lines

Darren Hart
Intel Open Source Technology Center

Peter Zijlstra

2014-05-15 08:13 UTC #38 lore raw

Quoted 10 lines · Carlos O'Donell

Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)

Quoted 6 lines

I fully agree with Michael here.
The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI. The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.

That ship has sailed.. for one we must always support old glibc which uses the futex() syscall, and secondly there are known other programs that actually use the futex syscall.

So that's really a non-argument, we're hard tied to the ABI.

Darren Hart

2014-05-15 15:42 UTC #39 lore raw

Quoted 19 lines · Peter Zijlstra

Quoted 10 lines · Carlos O'Donell
Quoted 8 lines · Michael Kerrisk (man-pages)
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex(). (BTW, I mispoke in my earlier mail when I said I wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)
Quoted 7 lines
I fully agree with Michael here.
The futex() syscall was never exposed to userspace specifically because it was an interface we did not want to support forever with a stable ABI.
The futex() syscall is an implementation detail that is shared between the kernel and the writers of core runtimes for Linux.

Quoted 5 lines

That ship has sailed.. for one we must always support old glibc which uses the futex() syscall, and secondly there are known other programs that actually use the futex syscall.
So that's really a non-argument, we're hard tied to the ABI.

Indeed. This is specifically why FUTEX_REQUEUE still exists (despite it's bugs) when only FUTEX_CMP_REQUEUE should ever be used in new programs.

Signature · 2 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

Peter Zijlstra

2014-05-15 08:14 UTC #40 lore raw

In reply to Carlos O'Donell

Quoted 2 lines · Carlos O'Donell

There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.

Yes, can we finally fix that please? It gets tedious having to endlessly copy/paste that thing around.

Carlos O'Donell

2014-05-15 13:18 UTC #41 lore raw

Quoted 6 lines · Peter Zijlstra

Quoted 2 lines · Carlos O'Donell
There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.
Yes, can we finally fix that please? It gets tedious having to endlessly copy/paste that thing around.

What exactly would you like fixed?

Cheers,
Carlos.

Peter Zijlstra

2014-05-15 13:22 UTC #42 lore raw

Quoted 9 lines · Carlos O'Donell

Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Carlos O'Donell
There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.
Yes, can we finally fix that please? It gets tedious having to endlessly copy/paste that thing around.
What exactly would you like fixed?

Not having gettid() in glibc.

Michael Kerrisk (man-pages)

author

2014-05-15 13:49 UTC #43 lore raw

Quoted 6 lines · Peter Zijlstra

Quoted 5 lines · Carlos O'Donell
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Carlos O'Donell
There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
Yes, can we finally fix that please? It gets tedious having to endlessly copy/paste that thing around.
What exactly would you like fixed?
Not having gettid() in glibc.

Get in the line ;-).
http://sourceware.org/bugzilla/show_bug.cgi?id=6399

Cheers,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Peter Zijlstra

2014-05-15 13:55 UTC #44 lore raw

Quoted 10 lines · Michael Kerrisk (man-pages)

Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Carlos O'Donell
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Carlos O'Donell
There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.
Yes, can we finally fix that please? It gets tedious having to endlessly copy/paste that thing around.

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
What exactly would you like fixed?
Not having gettid() in glibc.
Get in the line ;-).
http://sourceware.org/bugzilla/show_bug.cgi?id=6399

Oh hey, it moved.. :-) Hadn't seen the comments since the 2008 time-frame.

Carlos O'Donell

2014-05-15 14:39 UTC #45 lore raw

Quoted 10 lines · Michael Kerrisk (man-pages)

Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Carlos O'Donell
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Carlos O'Donell
There are other syscalls like gettid() that have a:
NOTE: There is no glibc wrapper for this system call; see NOTES.
Yes, can we finally fix that please? It gets tedious having to endlessly copy/paste that thing around.

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
What exactly would you like fixed?
Not having gettid() in glibc.
Get in the line ;-).
http://sourceware.org/bugzilla/show_bug.cgi?id=6399

I have no objections to this, but I absolutely object to this without someone documenting and gathering consensus for consistent terminology to be used between the kernel and glibc.

The relevant comment is here:
https://sourceware.org/bugzilla/show_bug.cgi?id=6399#c26

I'd like to see a glibc manual patch for the threads.texi file, which can be completely linux-specific, to document gettid() and nomenclature. It should talk about the nomenclature used to discuss these interfaces and explain when it is or isn't valid to use a task id and with what functions.

For example does gettid *really* return a pid_t as considered by userspace? It's not a full out process...

Cheers,
Carlos.

Peter Zijlstra

2014-05-15 15:11 UTC #46 lore raw

Quoted 2 lines · Carlos O'Donell

For example does gettid *really* return a pid_t as considered by userspace? It's not a full out process...

Yeah, PIDs and TIDs are the same namespace in the kernel. All we have are tasks and each task has an id. gettid() actually returns the id of the current task.

getpid() returns the id of the thread group leader, so for that task gettid() and getpid() return the same id.

Davidlohr Bueso

2014-05-14 20:56 UTC #47 lore raw

In reply to Davidlohr Bueso

Quoted 45 lines · Michael Kerrisk (man-pages)

Hi Darren,
Quoted 34 lines · Darren Hart
Quoted 31 lines · Michael Kerrisk (man-pages)
[So, some futex recent discussions remind me I should make this request]
Hello all (especially those in the To:, namely Thomas, Darren, Ingo, Jakub),
The futex man pages:
http://man7.org/linux/man-pages/man2/futex.2.html
http://man7.org/linux/man-pages/man7/futex.7.html
are currently in a sorry state. I'm by no means convinced that all of the futex operations described there are explained fully and correctly. And probably not all error cases for each operation are properly documented. I'd be very happy if some folk could review those pages and send me corrections (Git:
http://git.kernel.org/pub/scm/docs/man-pages/man-pages).
But worse, a number of futex operations remain undocumented in futex(2) (see the list below).
I am aware of Documentation/pi-futex.txt and Documentation/futex-requeue-pi.txt. However, both of those documents are rather thin on details / explain what certain FUTEX_* operations are used for rather than what they do / focus on the implementation, rather than the semantics.
What I would like is that the futex(2) page documenta each one of these operations with a focus on the semantics in a way that might be useful to writers of library functions or those who simply wish to better understand (from a user-space perspective) what futexes are and how they are used. However, I don't have the knowledge to do this well in any reasonable time.
Quoted 7 lines
Quoted 5 lines
Would the folk in the To: list (or anyone else who is knowledgeable) be willing to write patches
(Git: http://git.kernel.org/pub/scm/docs/man-pages/man-pages ) or just supply me with some raw text that documents these currently undocumented futex operations, in the manner suggested?
Yes, I'll be glad to help.

Quoted 8 lines

Thanks!
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex().

That's actually not quite true. There are plenty of software efforts out there that use futex calls directly to implement userspace serialization mechanisms as an alternative to the bulky sysv semaphores. I worked closely with an in-memory DB project that makes heavy use of them. Not everyone can simply rely on pthreads.

Quoted 3 lines

(BTW, I mispoke in my earlier mail when I said I
wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)

While kind of beyond the scope of documentation like manpages, I would very much welcome this! The internet is full of misleading/false/stupid/incomplete references to futexes and how to use them. Having a best practises helps the kernel directly avoiding users from misusing them.

Thanks,
Davidlohr

Darren Hart

2014-05-14 21:03 UTC #48 lore raw

Quoted 64 lines · Davidlohr Bueso

Quoted 54 lines · Michael Kerrisk (man-pages)
Hi Darren,
Quoted 1 line · Darren Hart
On 5/14/14, 3:35, "Michael Kerrisk (man-pages)"
<[email protected]>
Quoted 3 lines
Quoted 1 line · wrote:
[So, some futex recent discussions remind me I should make this
request]
Quoted 8 lines
Quoted 8 lines
Hello all (especially those in the To:, namely Thomas, Darren, Ingo, Jakub),
The futex man pages:
http://man7.org/linux/man-pages/man2/futex.2.html
http://man7.org/linux/man-pages/man7/futex.7.html
are currently in a sorry state. I'm by no means convinced that all of
the
Quoted 1 line
Quoted 1 line
futex operations described there are explained fully and correctly.
And
Quoted 1 line
Quoted 1 line
probably not all error cases for each operation are properly
documented.
Quoted 5 lines
Quoted 5 lines
I'd be very happy if some folk could review those pages and send me corrections (Git:
http://git.kernel.org/pub/scm/docs/man-pages/man-pages).
But worse, a number of futex operations remain undocumented in
futex(2)
Quoted 5 lines
Quoted 5 lines
(see the list below).
I am aware of Documentation/pi-futex.txt and Documentation/futex-requeue-pi.txt. However, both of those documents are rather thin on details / explain what certain FUTEX_* operations
are
Quoted 1 line
Quoted 1 line
used for rather than what they do / focus on the implementation,
rather
Quoted 9 lines
Quoted 9 lines
than the semantics.
What I would like is that the futex(2) page documenta each one of these operations with a focus on the semantics in a way that might be useful to writers of library functions or those who simply wish to better understand (from a user-space perspective) what futexes are and how they are used. However, I don't have the knowledge to do this well in any reasonable time.
Quoted 7 lines
Quoted 5 lines
Would the folk in the To: list (or anyone else who is knowledgeable) be willing to write patches
(Git: http://git.kernel.org/pub/scm/docs/man-pages/man-pages ) or just supply me with some raw text that documents these currently undocumented futex operations, in the manner suggested?
Yes, I'll be glad to help.
Quoted 8 lines
Thanks!
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex().

Quoted 5 lines

That's actually not quite true. There are plenty of software efforts out there that use futex calls directly to implement userspace serialization mechanisms as an alternative to the bulky sysv semaphores. I worked closely with an in-memory DB project that makes heavy use of them. Not everyone can simply rely on pthreads.

Userspace RCU from Mattieu Desnoyers / Efficios uses it as well - or did.

Quoted 10 lines

Quoted 4 lines
(BTW, I mispoke in my earlier mail when I said I
wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)
While kind of beyond the scope of documentation like manpages, I would very much welcome this! The internet is full of misleading/false/stupid/incomplete references to futexes and how to use them. Having a best practises helps the kernel directly avoiding users from misusing them.

Agreed.

Signature · 2 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

Paul E. McKenney

2014-05-14 22:21 UTC #49 lore raw

Quoted 66 lines · Darren Hart

Quoted 64 lines · Davidlohr Bueso
Quoted 62 lines · Michael Kerrisk (man-pages)
Hi Darren,
Quoted 1 line · Darren Hart
On 5/14/14, 3:35, "Michael Kerrisk (man-pages)"
<[email protected]>
Quoted 3 lines
Quoted 1 line · wrote:
[So, some futex recent discussions remind me I should make this
request]
Quoted 8 lines
Quoted 8 lines
Hello all (especially those in the To:, namely Thomas, Darren, Ingo, Jakub),
The futex man pages:
http://man7.org/linux/man-pages/man2/futex.2.html
http://man7.org/linux/man-pages/man7/futex.7.html
are currently in a sorry state. I'm by no means convinced that all of
the
Quoted 1 line
Quoted 1 line
futex operations described there are explained fully and correctly.
And
Quoted 1 line
Quoted 1 line
probably not all error cases for each operation are properly
documented.
Quoted 5 lines
Quoted 5 lines
I'd be very happy if some folk could review those pages and send me corrections (Git:
http://git.kernel.org/pub/scm/docs/man-pages/man-pages).
But worse, a number of futex operations remain undocumented in
futex(2)
Quoted 5 lines
Quoted 5 lines
(see the list below).
I am aware of Documentation/pi-futex.txt and Documentation/futex-requeue-pi.txt. However, both of those documents are rather thin on details / explain what certain FUTEX_* operations
are
Quoted 1 line
Quoted 1 line
used for rather than what they do / focus on the implementation,
rather
Quoted 9 lines
Quoted 9 lines
than the semantics.
What I would like is that the futex(2) page documenta each one of these operations with a focus on the semantics in a way that might be useful to writers of library functions or those who simply wish to better understand (from a user-space perspective) what futexes are and how they are used. However, I don't have the knowledge to do this well in any reasonable time.
Quoted 7 lines
Quoted 5 lines
Would the folk in the To: list (or anyone else who is knowledgeable) be willing to write patches
(Git: http://git.kernel.org/pub/scm/docs/man-pages/man-pages ) or just supply me with some raw text that documents these currently undocumented futex operations, in the manner suggested?
Yes, I'll be glad to help.
Thanks!
Quoted 2 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex().

Quoted 7 lines

Quoted 5 lines
That's actually not quite true. There are plenty of software efforts out there that use futex calls directly to implement userspace serialization mechanisms as an alternative to the bulky sysv semaphores. I worked closely with an in-memory DB project that makes heavy use of them. Not everyone can simply rely on pthreads.
Userspace RCU from Mattieu Desnoyers / Efficios uses it as well - or did.

It still does. ;-)

Thanx, Paul

Quoted 18 lines

Quoted 9 lines
Quoted 3 lines
(BTW, I mispoke in my earlier mail when I said I
wanted documentation suitable for "writers of library functions" -- I meant suitable for "writers of *C library*".)
While kind of beyond the scope of documentation like manpages, I would very much welcome this! The internet is full of misleading/false/stupid/incomplete references to futexes and how to use them. Having a best practises helps the kernel directly avoiding users from misusing them.
Agreed.
Signature · 2 lines
Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

H. Peter Anvin

2014-05-15 00:28 UTC #50 lore raw

Quoted 8 lines · Davidlohr Bueso

Quoted 7 lines
Quoted 3 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex().

Quoted 6 lines

That's actually not quite true. There are plenty of software efforts out there that use futex calls directly to implement userspace serialization mechanisms as an alternative to the bulky sysv semaphores. I worked closely with an in-memory DB project that makes heavy use of them. Not everyone can simply rely on pthreads.

More fundamentally, futex(2), like clone(2), are things that can be legitimately by user space without automatically breaking all of glibc.
There are some other things where that is *not* true, because glibc
relies on being able to mediate all accesses to a kernel facility, but not here.

-hpa

Andy Lutomirski

2014-05-15 00:35 UTC #51 lore raw

Quoted 16 lines · H. Peter Anvin

Quoted 8 lines · Davidlohr Bueso
Quoted 7 lines
Quoted 3 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex().
Quoted 6 lines
That's actually not quite true. There are plenty of software efforts out there that use futex calls directly to implement userspace serialization mechanisms as an alternative to the bulky sysv semaphores. I worked closely with an in-memory DB project that makes heavy use of them. Not everyone can simply rely on pthreads.

Quoted 2 lines

More fundamentally, futex(2), like clone(2), are things that can be legitimately by user space without automatically breaking all of glibc.

I'm lost -- I think the missing verb is important :)

Quoted 3 lines

There are some other things where that is *not* true, because glibc
relies on being able to mediate all accesses to a kernel facility, but not here.

--Andy

H. Peter Anvin

2014-05-15 00:41 UTC #52 lore raw

Quoted 6 lines · Andy Lutomirski

Quoted 3 lines
More fundamentally, futex(2), like clone(2), are things that can be legitimately by user space without automatically breaking all of glibc.
I'm lost -- I think the missing verb is important :)

... legitimately *used* by user space ...

As in you can use it to implement your own, non-POSIX, synchronization primitives.

-hpa

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Carlos O'Donell

2014-05-15 19:10 UTC #53 lore raw

In reply to H. Peter Anvin

Quoted 16 lines · H. Peter Anvin

Quoted 8 lines · Davidlohr Bueso
Quoted 7 lines
Quoted 3 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe
I don't think futex() ever was in glibc--that's by design, and completely understandable: no user-space application would want to directly use futex().
Quoted 6 lines
That's actually not quite true. There are plenty of software efforts out there that use futex calls directly to implement userspace serialization mechanisms as an alternative to the bulky sysv semaphores. I worked closely with an in-memory DB project that makes heavy use of them. Not everyone can simply rely on pthreads.

Quoted 5 lines

More fundamentally, futex(2), like clone(2), are things that can be legitimately by user space without automatically breaking all of glibc.
There are some other things where that is *not* true, because glibc
relies on being able to mediate all accesses to a kernel facility, but not here.

Careful there. There is *some* danger in using clone(2) because of the coordination required to implement thread-local storage. I'm sure you're aware of this, but I'd like the record to show that we're going to need clear documentation of what's considered safe given the known implementations.

Cheers,
Carlos.

Davidlohr Bueso

2014-05-14 21:05 UTC #54 lore raw

Quoted 47 lines · Darren Hart

Quoted 31 lines · Michael Kerrisk (man-pages)
[So, some futex recent discussions remind me I should make this request]
Hello all (especially those in the To:, namely Thomas, Darren, Ingo, Jakub),
The futex man pages:
http://man7.org/linux/man-pages/man2/futex.2.html
http://man7.org/linux/man-pages/man7/futex.7.html
are currently in a sorry state. I'm by no means convinced that all of the futex operations described there are explained fully and correctly. And probably not all error cases for each operation are properly documented. I'd be very happy if some folk could review those pages and send me corrections (Git:
http://git.kernel.org/pub/scm/docs/man-pages/man-pages).
But worse, a number of futex operations remain undocumented in futex(2) (see the list below).
I am aware of Documentation/pi-futex.txt and Documentation/futex-requeue-pi.txt. However, both of those documents are rather thin on details / explain what certain FUTEX_* operations are used for rather than what they do / focus on the implementation, rather than the semantics.
What I would like is that the futex(2) page documenta each one of these operations with a focus on the semantics in a way that might be useful to writers of library functions or those who simply wish to better understand (from a user-space perspective) what futexes are and how they are used. However, I don't have the knowledge to do this well in any reasonable time.
Quoted 5 lines
Would the folk in the To: list (or anyone else who is knowledgeable) be willing to write patches
(Git: http://git.kernel.org/pub/scm/docs/man-pages/man-pages ) or just supply me with some raw text that documents these currently undocumented futex operations, in the manner suggested?
Yes, I'll be glad to help.
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:

Quoted 8 lines

http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inclu
de/futextest.h#n67
Which basically defines:
#define futex(uaddr, op, val, timeout, uaddr2, val3, opflags) \
syscall(SYS_futex, uaddr, op | opflags, val, timeout, uaddr2, val3)

Yeah, and I actually stole that from you for perf :)

Quoted 5 lines

Adding Carlos for his perspective.
If I'm wrong, or we can restore the futex() call, great. If not... Should we keep the man-pages and document it as syscall(SYS_futex, ..., op, ...) ?

+1, is there anything preventing adding a futex wrapper... glibc folks?

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Joseph S. Myers

2014-05-15 15:15 UTC #55 lore raw

Quoted 4 lines · Davidlohr Bueso

Quoted 2 lines
If I'm wrong, or we can restore the futex() call, great. If not... Should we keep the man-pages and document it as syscall(SYS_futex, ..., op, ...) ?
+1, is there anything preventing adding a futex wrapper... glibc folks?

See what I said at
<https://sourceware.org/bugzilla/show_bug.cgi?id=9712#c4> (with references to previous discussions). Someone needs to take the lead on pushing to consensus the question of what syscalls should have wrappers in glibc, and then implement the conclusions.

Signature · 2 lines

Joseph S. Myers
[email protected]

H. Peter Anvin

2014-05-15 00:18 UTC #56 lore raw

Quoted 9 lines · Darren Hart

However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inclu
de/futextest.h#n67

This really comes down to the fact that we should have a libinux which contains the basic system call wrapper machinery for Linux specific things and nothing else.

syscall(3) is toxic and breaks randomly on some platforms.

-hpa

Darren Hart

2014-05-15 05:21 UTC #57 lore raw

Quoted 13 lines · H. Peter Anvin

Quoted 11 lines · Darren Hart
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inc
lu
de/futextest.h#n67

Quoted 5 lines

This really comes down to the fact that we should have a libinux which contains the basic system call wrapper machinery for Linux specific things and nothing else.
syscall(3) is toxic and breaks randomly on some platforms.

Peter Z and I have had a good time discussing this in the past.... And here it is again. :-)

Signature · 2 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

Peter Zijlstra

2014-05-15 08:23 UTC #58 lore raw

Quoted 15 lines · Darren Hart

Quoted 13 lines · H. Peter Anvin
Quoted 11 lines · Darren Hart
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inc
lu
de/futextest.h#n67

Quoted 8 lines

Quoted 5 lines
This really comes down to the fact that we should have a libinux which contains the basic system call wrapper machinery for Linux specific things and nothing else.
syscall(3) is toxic and breaks randomly on some platforms.
Peter Z and I have had a good time discussing this in the past.... And here it is again. :-)

Oh but we wanted _way_ more than bare syscalls in there ;-)

For a start we wanted to make the vDSO a proper DSO that gets included in the (dynamic) link chain.

/sys/lib/libdso{32,64}.so like

That would also allow all those archs that expose raw dso function pointers for things like cmpxchg or memory barriers to just provide platform functions instead, far more usable.

And yes, we wanted to hijack libpthread in order to finally fix the futex mess :-)

Michael Kerrisk (man-pages)

author

2014-05-15 13:46 UTC #59 lore raw

Quoted 15 lines · Darren Hart

Quoted 13 lines · H. Peter Anvin
Quoted 11 lines · Darren Hart
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inc
lu
de/futextest.h#n67

Quoted 8 lines

Quoted 5 lines
This really comes down to the fact that we should have a libinux which contains the basic system call wrapper machinery for Linux specific things and nothing else.
syscall(3) is toxic and breaks randomly on some platforms.
Peter Z and I have had a good time discussing this in the past.... And here it is again. :-)

People have a number of times noted that there are problems with syscall(), but I'm not knowledgeable on the details. I'd happily take a patch to the man page (which, for historical reasons, is actually syscall(2)) that explains the the problems (and ideally notes those platforms where there are no problems).

Thanks,

Michael

Signature · 3 lines

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

H. Peter Anvin

2014-05-15 14:59 UTC #60 lore raw

Quoted 7 lines · Michael Kerrisk (man-pages)

People have a number of times noted that there are problems with syscall(), but I'm not knowledgeable on the details. I'd happily take a patch to the man page (which, for historical reasons, is actually syscall(2)) that explains the the problems (and ideally notes those platforms where there are no problems).

It has to do with how ABIs deal with doublewidth arguments.

There is a reason why Linux syscall ABIs generally have a 1:1 mapping with the user space ABIs, and why the system call argument is passed not in the first argument but in a different place (usually a separately clobbered register, e.g. %eax on x86-64).

On some platforms, doublewidth registers have to be aligned in register pairs. On some other platforms, enough arguments mean some will be passed in memory, where they are forced to be aligned, or they are not allowed to straddle the register-memory boundary. All of this means that padding words might be introduced, and they will be introduced in the wrong place because of the additional argument introduced at the beginning of the argument sequence.

On the other hand, the old SYSCALL user-space macros just plain didn't handle doubleword arguments.

-hpa

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

[email protected]

2014-05-15 15:42 UTC #61 lore raw

https://github.com/linux-test-project/ltp/commit/04afb02b4280a20c262054e8f99a3fad4ad54916

Hi!

Quoted 5 lines

People have a number of times noted that there are problems with syscall(), but I'm not knowledgeable on the details. I'd happily take a patch to the man page (which, for historical reasons, is actually syscall(2)) that explains the the problems (and ideally notes those platforms where there are no problems).

Have a look at this commit that tries to deal with passing 64 bit numbers to syscalls. On 32 bit ABI (but not on X32) these needs to be split up (accordingly to machine endianity).

Signature · 2 lines

Cyril Hrubis
[email protected]

H. Peter Anvin

2014-05-15 15:52 UTC #62 lore raw

Quoted 7 lines · On 05/15/2014 08:42 AM, [email protected] wrote:

Hi!
Quoted 5 lines
People have a number of times noted that there are problems with syscall(), but I'm not knowledgeable on the details. I'd happily take a patch to the man page (which, for historical reasons, is actually syscall(2)) that explains the the problems (and ideally notes those platforms where there are no problems).

Quoted 6 lines

Have a look at this commit that tries to deal with passing 64 bit numbers to syscalls. On 32 bit ABI (but not on X32) these needs to be split up (accordingly to machine endianity).
https://github.com/linux-test-project/ltp/commit/04afb02b4280a20c262054e8f99a3fad4ad54916

That is wrong, too. That assumes that there will never be padding words, which isn't true in the general case, either.

I really believe the proper fix is to use assembly syscall stubs. In klibc I build a fairly elaborate machinery to autogenerate such syscall stubs for a variety of architectures.

-hpa

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

[email protected]

2014-05-15 16:01 UTC #63 lore raw

Hi!

Quoted 9 lines

Quoted 6 lines
Have a look at this commit that tries to deal with passing 64 bit numbers to syscalls. On 32 bit ABI (but not on X32) these needs to be split up (accordingly to machine endianity).
https://github.com/linux-test-project/ltp/commit/04afb02b4280a20c262054e8f99a3fad4ad54916
That is wrong, too. That assumes that there will never be padding words, which isn't true in the general case, either.

Well, it's still far better than the mess we had previously and it works in most of the cases. However I would love to fix these correctly once for all.

Quoted 3 lines

I really believe the proper fix is to use assembly syscall stubs. In klibc I build a fairly elaborate machinery to autogenerate such syscall stubs for a variety of architectures.

Then it would be nice to share these between klibc and LTP (and possible everybody else).

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

H. Peter Anvin

2014-05-15 16:07 UTC #64 lore raw

Quoted 8 lines · On 05/15/2014 09:01 AM, [email protected] wrote:

Quoted 4 lines
I really believe the proper fix is to use assembly syscall stubs. In klibc I build a fairly elaborate machinery to autogenerate such syscall stubs for a variety of architectures.
Then it would be nice to share these between klibc and LTP (and possible everybody else).

It should be quite easy to extract from klibc.

-hpa

[email protected]

2014-05-15 16:17 UTC #65 lore raw

Hi!

Quoted 9 lines

Quoted 7 lines
Quoted 3 lines
I really believe the proper fix is to use assembly syscall stubs. In klibc I build a fairly elaborate machinery to autogenerate such syscall stubs for a variety of architectures.
Then it would be nice to share these between klibc and LTP (and possible everybody else).
It should be quite easy to extract from klibc.

That is not the main concern here. If I extract the code I would have to watch for any changes manually. If it was in a library or a separate repository all that would be needed is to add it as dependency/git submodule and I would get all updates automatically.

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

H. Peter Anvin

2014-05-15 16:56 UTC #66 lore raw

Quoted 8 lines · On 05/15/2014 09:17 AM, [email protected] wrote:

Quoted 2 lines
It should be quite easy to extract from klibc.
That is not the main concern here. If I extract the code I would have to watch for any changes manually. If it was in a library or a separate repository all that would be needed is to add it as dependency/git submodule and I would get all updates automatically.

Yes, and for that to happen someone needs to do the work to extract it.
I don't have the cycles myself at the moment.

-hpa

[email protected]

2014-05-15 17:06 UTC #67 lore raw

Hi!

Quoted 8 lines

Quoted 5 lines
That is not the main concern here. If I extract the code I would have to watch for any changes manually. If it was in a library or a separate repository all that would be needed is to add it as dependency/git submodule and I would get all updates automatically.
Yes, and for that to happen someone needs to do the work to extract it.
I don't have the cycles myself at the moment.

If that is the only problem, I should be able to allocate some time in order to have a look at it.

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Darren Hart

2014-05-15 15:46 UTC #68 lore raw

In reply to H. Peter Anvin

Quoted 28 lines · Michael Kerrisk (man-pages)

Quoted 18 lines · Darren Hart
Quoted 16 lines · H. Peter Anvin
Quoted 14 lines · Darren Hart
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling
the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/i
nc
lu
de/futextest.h#n67
Quoted 8 lines
Quoted 5 lines
This really comes down to the fact that we should have a libinux which contains the basic system call wrapper machinery for Linux specific things and nothing else.
syscall(3) is toxic and breaks randomly on some platforms.
Peter Z and I have had a good time discussing this in the past.... And here it is again. :-)

Quoted 8 lines

People have a number of times noted that there are problems with syscall(), but I'm not knowledgeable on the details. I'd happily take a patch to the man page (which, for historical reasons, is actually syscall(2)) that explains the the problems (and ideally notes those platforms where there are no problems).
From my perspective, a named interface with specific documented interfaces

is far more usable than a vargs direct syscall. That just leaves all kinds of room for error - which of course is why we all write our own wrappers in our apps rather than use it directly... If we all do it, it seems to me that is a strong indicator we should provide it in some kind of common library. Maybe that's libc... Maybe that's libnix...

Signature · 2 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

[email protected]

2014-05-15 15:35 UTC #69 lore raw

Hi!

Quoted 9 lines

Quoted 8 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inclu
de/futextest.h#n67

Quoted 5 lines

This really comes down to the fact that we should have a libinux which contains the basic system call wrapper machinery for Linux specific things and nothing else.
syscall(3) is toxic and breaks randomly on some platforms.

And while cleaning the LTP[1] testcases, we are slowly extracting the special cases into commont code.

[1] http://linux-test-project.github.io/

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[email protected]

2014-05-15 15:28 UTC #70 lore raw

[1] http://linux-test-project.github.io/

Hi!

Quoted 7 lines

However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inclu

So there actually exists some tests for futexes, I've been asked if we have these as a LTP[1] maintainer several times.

Are these tests executed regulary as a part of some automated framework? If not it would make sense to port them to LTP (looking at the code that would be quite easy task) and get them executed by several QA departments for free. What do you think?

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Steven Rostedt

2014-05-15 15:40 UTC #71 lore raw

Quoted 12 lines · On Thu, 15 May 2014 17:28:35 +0200 [email protected] wrote:

Hi!
Quoted 7 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inclu
So there actually exists some tests for futexes, I've been asked if we have these as a LTP[1] maintainer several times.

Quoted 7 lines

Are these tests executed regulary as a part of some automated framework? If not it would make sense to port them to LTP (looking at the code that would be quite easy task) and get them executed by several QA departments for free. What do you think?
[1] http://linux-test-project.github.io/

I think Thomas may be working on one. If not, I'd be happy to start writing one as well.

-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Darren Hart

2014-05-15 16:13 UTC #72 lore raw

In reply to [email protected]

Quoted 14 lines · On 5/15/14, 8:28, "[email protected]" <[email protected]> wrote:

Hi!
Quoted 9 lines
However, unless I'm sorely mistaken, the larger problem is that glibc removed the futex() call entirely, so these man pages don't describe something users even have access to anymore. I had to revert to calling the syscalls directly in the futextest test suite because of this:
http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git/tree/inc
lu
So there actually exists some tests for futexes, I've been asked if we have these as a LTP[1] maintainer several times.

Quoted 6 lines

Are these tests executed regulary as a part of some automated framework? If not it would make sense to port them to LTP (looking at the code that would be quite easy task) and get them executed by several QA departments for free. What do you think?
[1] http://linux-test-project.github.io/

I've used LTP in the past (quite a bit), and I felt there was some advantage to keeping futextest independent. Perhaps things have changed enough since then (~2009 era) that we should reconsider. We can discuss the pros/cons there if you like. I have agreed to move the performance related tests over to perf, and Davidlohr has added some other such tests to perf. Trinity now covers the planned fuzz testing for futexes (very well... Obviously) so that idea will be dropped, leaving pure functional tests in futextest.

Signature · 2 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation

[email protected]

2014-05-15 16:30 UTC #73 lore raw

Hi!

Quoted 2 lines

I've used LTP in the past (quite a bit), and I felt there was some advantage to keeping futextest independent.

What advantages did you have in mind?

Quoted 2 lines

Perhaps things have changed enough since then (~2009 era) that we should reconsider.

I've been working on LTP for a about three years now and we happen to do quite a lot in that time. The most visible changes would be more proper development practices (git, proper build system, code review, LKML coding style, documentation, ...) and also huge number of fixes. Now we are trying to catch up in coverage too.

Quoted 1 line

We can discuss the pros/cons there if you like.

I would love to :).

Quoted 4 lines

I have agreed to move the performance related tests over to perf, and Davidlohr has added some other such tests to perf. Trinity now covers the planned fuzz testing for futexes (very well... Obviously) so that idea will be dropped, leaving pure functional tests in futextest.

Well LTP mostly consists of functional tests, so that would fit the purpose very well.

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Darren Hart

2014-05-15 18:17 UTC #74 lore raw

Quoted 5 lines · On 5/15/14, 9:30, "[email protected]"…

Hi!
Quoted 2 lines
I've used LTP in the past (quite a bit), and I felt there was some advantage to keeping futextest independent.
What advantages did you have in mind?

Not CVS was a big one at the time ;-)

OK, I don't mean to be disparaging here... But since you asked, back in '09 LTP had some test quality issues and I felt I could maintain futextest to a higher bar independently.

Quoted 10 lines

Quoted 3 lines
Perhaps things have changed enough since then (~2009 era) that we should reconsider.
I've been working on LTP for a about three years now and we happen to do quite a lot in that time. The most visible changes would be more proper development practices (git, proper build system, code review, LKML coding style, documentation, ...) and also huge number of fixes. Now we are trying to catch up in coverage too.

Quoted 3 lines

Quoted 1 line
We can discuss the pros/cons there if you like.
I would love to :).

Does LTP need to own the code, or can it incorporate existing projects and a sort of aggregator?

How much LTP harness type code needs to be used?

Signature · 9 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation



--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[email protected]

2014-05-15 19:05 UTC #75 lore raw

Hi!

Quoted 10 lines

Quoted 4 lines
Quoted 2 lines
I've used LTP in the past (quite a bit), and I felt there was some advantage to keeping futextest independent.
What advantages did you have in mind?
Not CVS was a big one at the time ;-)
OK, I don't mean to be disparaging here... But since you asked, back in '09 LTP had some test quality issues and I felt I could maintain futextest to a higher bar independently.

To be honest LTP was one of the messiest codebases I've seen and it was hacked up by mostly clueless people (there were even tests with race conditions that were carefully disabled in a way that was not easy to see). It took me months to get to a state where it compiled fine on major distributions.

Today we still have quite a bit of legacy code that needs to be cleaned up, however that gets better every day.

And most of the testcases are pretty stable, etc. unfortunatelly LTP has a bad reputation which is lot harder to fix than the code itself.

Quoted 9 lines

Quoted 9 lines
Quoted 2 lines
Perhaps things have changed enough since then (~2009 era) that we should reconsider.
I've been working on LTP for a about three years now and we happen to do quite a lot in that time. The most visible changes would be more proper development practices (git, proper build system, code review, LKML coding style, documentation, ...) and also huge number of fixes. Now we are trying to catch up in coverage too.

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
We can discuss the pros/cons there if you like.
I would love to :).
Does LTP need to own the code, or can it incorporate existing projects and a sort of aggregator?

That is possible as well but not optimal. This approach would need a wrapper script to convert the test exit values to be LTP compatible.

Quoted 1 line

How much LTP harness type code needs to be used?

Not much.

For this complexity of tests you would just need to call the tst_resm() interface to report success/failure and, at the end of the test, tst_exit() to return the stored overall test status.

And ideally call the standard option parsing code and call the test in standard loop so that the test can take advantage of standard options as number of iterations to run, etc.

Have a look at:

https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines

there is simple test example as well as description of the interfaces.

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Darren Hart

2014-05-15 19:38 UTC #76 lore raw

Quoted 58 lines · On 5/15/14, 12:05, "[email protected]"…

Hi!
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
I've used LTP in the past (quite a bit), and I felt there was some advantage to keeping futextest independent.
What advantages did you have in mind?
Not CVS was a big one at the time ;-)
OK, I don't mean to be disparaging here... But since you asked, back in '09 LTP had some test quality issues and I felt I could maintain futextest
to a higher bar independently.
To be honest LTP was one of the messiest codebases I've seen and it was hacked up by mostly clueless people (there were even tests with race conditions that were carefully disabled in a way that was not easy to see). It took me months to get to a state where it compiled fine on major distributions.
Today we still have quite a bit of legacy code that needs to be cleaned up, however that gets better every day.
And most of the testcases are pretty stable, etc. unfortunatelly LTP has a bad reputation which is lot harder to fix than the code itself.
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
Perhaps things have changed enough since then (~2009 era) that we should reconsider.
I've been working on LTP for a about three years now and we happen to
do
Quoted 5 lines
quite a lot in that time. The most visible changes would be more proper development practices (git, proper build system, code review, LKML coding style, documentation, ...) and also huge number of fixes. Now we are trying to catch up in coverage too.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
We can discuss the pros/cons there if you like.
I would love to :).
Does LTP need to own the code, or can it incorporate existing projects and
a sort of aggregator?
That is possible as well but not optimal. This approach would need a wrapper script to convert the test exit values to be LTP compatible.
Quoted 1 line
How much LTP harness type code needs to be used?
Not much.
For this complexity of tests you would just need to call the tst_resm() interface to report success/failure and, at the end of the test, tst_exit() to return the stored overall test status.
And ideally call the standard option parsing code and call the test in standard loop so that the test can take advantage of standard options as number of iterations to run, etc.

Quoted 5 lines

Have a look at:
https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines
there is simple test example as well as description of the interfaces.

Thanks Cyril,

I'll follow up with you in a couple weeks most likely. I have some urgent things that will be taking all my time and then some until then. Feel free to poke me though if I lose track of it :-)

Signature · 9 lines

Darren Hart					Open Source Technology Center
[email protected]				            Intel Corporation



--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[email protected]

2014-08-11 10:19 UTC #77 lore raw

Hi!

Quoted 16 lines

Quoted 16 lines
Quoted 1 line
How much LTP harness type code needs to be used?
Not much.
For this complexity of tests you would just need to call the tst_resm() interface to report success/failure and, at the end of the test, tst_exit() to return the stored overall test status.
And ideally call the standard option parsing code and call the test in standard loop so that the test can take advantage of standard options as number of iterations to run, etc.
Have a look at:
https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines

Quoted 8 lines

Quoted 1 line
there is simple test example as well as description of the interfaces.
Thanks Cyril,
I'll follow up with you in a couple weeks most likely. I have some urgent things that will be taking all my time and then some until then. Feel free to poke me though if I lose track of it :-)

Ping :)

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Cyril Hrubis

2014-11-26 13:41 UTC #78 lore raw

Hi!

Quoted 12 lines

Quoted 12 lines
For this complexity of tests you would just need to call the tst_resm() interface to report success/failure and, at the end of the test, tst_exit() to return the stored overall test status.
And ideally call the standard option parsing code and call the test in standard loop so that the test can take advantage of standard options as number of iterations to run, etc.
Have a look at:
https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines

Quoted 8 lines

Quoted 1 line
there is simple test example as well as description of the interfaces.
Thanks Cyril,
I'll follow up with you in a couple weeks most likely. I have some urgent things that will be taking all my time and then some until then. Feel free to poke me though if I lose track of it :-)

Do you still plan to work on this?

Signature · 6 lines

Cyril Hrubis
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Cyril Hrubis

2015-02-16 13:14 UTC #79 lore raw