Realtime capable userspace?
Hi,
I know this is not a kernel-related question, but I think the amount of people doing realtime here is probably the largest.
As learned the hard-way the glibc is not (yet) realtime capable and first changes are now starting to trickle in. Even if I try to fix known places, who knows what, where and for how long is locked.
What userspace (libc / threads / malloc) are you using for realtime applications? From my quick research:
- glibc:
- priority-inheritance mutex implemented, but never used inside the glibc
- not easy to replace malloc used internally (so that e.g. strdup() uses another one), at least when using shared libraries
- ucLibc:
- threads from glibc, malloc uses non-PI-aware locks
- newlib:
- no PI support, no futexes, mutexes done via kill(treadid, signal)
- tcmalloc (gperftools)
- no PI locks used, although it is probably easily added. Not sure about the state of the software generally - it probably got much less real-world testing than the implementation in glibc
I ended with own low-level implementation of synchronization primitives I needed via futexes (fortunately no full condvars or semaphores needed) and serializing all malloc/free/... cals via a PI-aware mutex (I don't care for performance when contented here, but I do care for priority inversion).
What do you use for non-trivial applications?
Thanks
Signature · 1 line
Stano
Quoted 11 lines · Stanislav Meduna
Hi,
I know this is not a kernel-related question, but I think the amount of people doing realtime here is probably the largest.
As learned the hard-way the glibc is not (yet) realtime capable and first changes are now starting to trickle in. Even if I try to fix known places, who knows what, where and for how long is locked.
What userspace (libc / threads / malloc) are you using for realtime applications? From my quick research:
PI mutexes are not sufficient to make malloc deterministic. I guess if you are looking for a deterministic malloc, you should have a look at TLSF.
You can probably get the replacement malloc be used by glibc by using LD_PRELOAD techniques (but OK, it would be just a hack).
Signature · 1 line
Gilles.
Quoted 2 lines · Gilles Chanteperdrix
PI mutexes are not sufficient to make malloc deterministic. I guess if you are looking for a deterministic malloc, you should have a look at TLSF.
Yup, I know that malloc from glibc is a complicated beast and should be probably not used at all during hard realtime operation.
Right now my biggest problem is the possibility of priority inversion. My time constraints are in the 20 - 50 ms range and I hope the malloc itself - while not deterministic - is not unbounded / won't go to that times. I am of course using mlockall and everything. OTOH a priority inversion case is a sure recipe for missing the deadlines.
Thank you for the pointer to the TLSF, this is very interesting project and I will surely investigate this.
Quoted 2 lines
You can probably get the replacement malloc be used by glibc by using
LD_PRELOADtechniques (but OK, it would be just a hack).
Yes, this is possible but is indeed quite a hack.
Thanks
Signature · 1 line
Stano
Quoted 5 lines · Stanislav Meduna
Quoted 2 lines · Gilles Chanteperdrix
PI mutexes are not sufficient to make malloc deterministic. I guess if you are looking for a deterministic malloc, you should have a look at TLSF.
Quoted 8 lines
Yup, I know that malloc from glibc is a complicated beast and should be probably not used at all during hard realtime operation.
Right now my biggest problem is the possibility of priority inversion. My time constraints are in the 20 - 50 ms range and I hope the malloc itself - while not deterministic - is not unbounded / won't go to that times. I am of course using mlockall and everything. OTOH a priority inversion case is a sure recipe for missing the deadlines.
mlockall does not really help reducing malloc worst case latency.
When malloc needs to grow the memory pool it manages, it will request memory to the kernel, using brk or mmap. Since in the Linux kernel, the memory is shared with the filesystem cache, the worst case latency for satisfying such a memory request is the time of a disk flush, so, given the latencies of good old hard disks, it is well in the milliseconds range. mlockall will simply force the kernel to satisfy the whole request ASAP instead of allocating it little by little, upon memory faults, so, may in fact increase the latency of the malloc service.
Signature · 1 line
Gilles.
Quoted 8 lines · Gilles Chanteperdrix
When malloc needs to grow the memory pool it manages, it will request memory to the kernel, using brk or mmap. Since in the Linux kernel, the memory is shared with the filesystem cache, the worst case latency for satisfying such a memory request is the time of a disk flush, so, given the latencies of good old hard disks, it is well in the milliseconds range. mlockall will simply force the kernel to satisfy the whole request ASAP instead of allocating it little by little, upon memory faults, so, may in fact increase the latency of the malloc service.
Yes, the theoretical possibility is definitely here, but I don't think it can happen in the real life in our environment. Given the processes running there there is nearly no way that the kernel does not have enough non-dirty pages to satisfy a brk or mmap request. It is also highly improbable that the brk/mmap are necessary - during the phase where the deadlines matter the only way they can happen is heap fragmentation causing something to temporarily overrun the brk limit.
You are right, as soon as the kernel has to flush some dirty pages the malloc latency goes through the roof - we have a CF Card instead of a good old hard disk, but tens of ms are definitely possible.
I am using the mlockall() mainly as a guard against throwing non-dirty code and data pages out and then getting a major pagefault and wait for the kernel to read them back. This is a much more probable scenario.
Regards
Signature · 1 line
Stano