Prelink problems -- need help!
I'm cross posting this in an effort to ask for help for the cross-prelink project. (If you are unaware of what the prelinker does, it goes through and adjusts the relocation addresses to predefined values in an attempt to speed up executable load times. For embedded systems, this can be fairly critical in meeting boot and runtime load performance targets.)
As some people might be aware, the upstream 'prelink' project that was maintained by Jakub Jelinek at Red Hat appears to have been abandoned.
I've been the maintainer for the Yocto Project's 'prelink-cross' project for a few years, but my work has been dealing with compilation issues and other things that have to do with using the prelinker in a cross compilation environment.
This means that unfortunately I'm in a situation where I am having problems trying to fix the problems in the prelinker itself, but I am not an ELF expert.
Currently the cross prelinker appears to be broken for pretty much every
architecture and I need help trying to resolve these problems.
Issues I know of right now:
Introduction of RTYPE_CLASS_EXTERN_PROTECTED_DATA appears to be causing failures on ARM32, x86 and x86_64.
Some possibly unrelated issue is causing various items to relocate incorrectly (IFUNCs perhaps) on ARMv7 builds.
Power is no longer working properly, most likely due to an optimization in the way the PPC GOT table(s) are setup.
Power64 hasn't been working properly for a while, possibly due to the switch to ELFv2.
MIPS and MIPS64 also do not appear to be working, and I've not found a possible cause.
As you can see, pretty much everything is currently broken and I'm struggling to find community people with the right skill sets to help me resolve the issues.
The current state of my work in progress is available at:
git://git.yoctoproject.org/prelink-cross
branch cross_prelink_staging
If you think you can help, please let me know.
--Mark
[Cc list trimmed]
Quoted 4 lines · Mark Hatle
As you can see, pretty much everything is currently broken and I'm struggling to
find community people with the right skill sets to help me resolve the issues.
I think it's true that prelink is a technology which has largely had its day, and I could well imagine that upstream Red Hat is no longer very interested in it. https://fedorahosted.org/fesco/ticket/1183 has a summary of some of the reasons why.
Before investing any significant amount of time in fixing up prelink (and from what you've described it sounds like it might indeed be a significant effort to get it working again on all platforms) it might be a good idea to make sure we are still convinced that prelink provides useful and meaningful benefits for OE-core users.
If the answer to that is yes then I probably could spare a bit of time to look at the ARM and MIPS issues at least.
p.
Quoted 13 lines · Phil Blundell
[Cc list trimmed]
Quoted 4 lines · Mark Hatle
As you can see, pretty much everything is currently broken and I'm struggling to
find community people with the right skill sets to help me resolve the issues.I think it's true that prelink is a technology which has largely had its day, and I could well imagine that upstream Red Hat is no longer very interested in it. https://fedorahosted.org/fesco/ticket/1183 has a summary of some of the reasons why.
Quoted 6 lines
Before investing any significant amount of time in fixing up prelink (and from what you've described it sounds like it might indeed be a significant effort to get it working again on all platforms) it might be a good idea to make sure we are still convinced that prelink provides useful and meaningful benefits for OE-core users.
I do think prelink has a purpose. Especially for embedded systems.
The biggest reason I see against prelink is ASLR. I agree 100% that if you want ASLR (Address Randomization) you should not be using the prelinker.
Usually secondary reasons include if you are doing field upgrades at a package level -- it means you also need the ability to prelink on the device which cause have other performance ramifications and such during this upgrade process.
I agree with some of the information on the Fedora hosted link you pointed to, but disagree with others.
While some of the modern hashing techniques and such do improve run-time dynamic link performance, there is still a hit that we must take. For devices that need quick boot times, quick startup, or are memory constrained, the prelinker can still help. (Memory usage on very small systems is a good example. Memory usage can be reduced in larger applications by reducing the number of Copy-on-write pages required to handle the relocation information.)
I probably would not use the prelinker on a workstation/server any longer. I might not use it on a 'larger' embedded system, but on a more traditional embedded system I do think it still has merit.
(Probably a discussion for the future, due to the ASLR, is should the prelinker be enabled by default -- even if it works -- but that is a discussion for later.)
Quoted 2 lines
If the answer to that is yes then I probably could spare a bit of time to look at the ARM and MIPS issues at least.
I can certainly use any help you have. I have someone looking into the ARM issues with me. It APPEARS the primary behavior there has to do with IFUNC behavior to things like memcpy has changed, causing the prelinker to do the wrong thing(s). But I have few additional details at this time.
I've reached out to parts of the MIPS community for help, but we'll see if I get any type of response. (MIPS, especially 32-bit, I'm very surprised has broken recently. I haven't seen many changes in regard to the way binaries are loaded on MIPS -- so the issue must be related to a more fundamental optimization or bug in the prelink.)
Thanks for the offer, let me know if you find anything. I'll certainly respond as I make progress trying to go through the issues.
--Mark
Quoted 2 lines
p.
Quoted 12 lines · Mark Hatle
While some of the modern hashing techniques and such do improve run -time dynamic
link performance, there is still a hit that we must take. For devices that need
quick boot times, quick startup, or are memory constrained, the prelinker can
still help. (Memory usage on very small systems is a good example.
Memory
usage can be reduced in larger applications by reducing the number of Copy-on-write pages required to handle the relocation information.)
Do you have any concrete data on how much of a boot time speedup, and/or what reduction in dirty pages, you see on a modern image with prelink versus the same image without?
p.
Quoted 14 lines · Phil Blundell
Quoted 12 lines · Mark Hatle
While some of the modern hashing techniques and such do improve run -time dynamic
link performance, there is still a hit that we must take. For devices that need
quick boot times, quick startup, or are memory constrained, the prelinker can
still help. (Memory usage on very small systems is a good example.
Memory
usage can be reduced in larger applications by reducing the number of Copy-on-write pages required to handle the relocation information.)
Quoted 3 lines
Do you have any concrete data on how much of a boot time speedup, and/or what reduction in dirty pages, you see on a modern image with prelink versus the same image without?
-last- time I got concrete numbers on a complex boot process. It was on the order of 1-5% boot time.. and a fairly large number of pages saved.. (which reduced fragmentation as well as saved memory.)
We're not talking huge numbers for each, there is a statistics mode in glibc that can be enabled to show the number of relocations and some of the costs involved. Once this is working again, I will be able to run that and give more concrete numbers.
--Mark
Quoted 2 lines
p.
Quoted 3 lines · Mark Hatle
-last- time I got concrete numbers on a complex boot process. It was on the order of 1-5% boot time.. and a fairly large number of pages saved.. (which reduced fragmentation as well as saved memory.)
I'm slightly surprised at the 5% figure. Martin Jansa posted some statistics in another thread earlier today which showed an improvement from prelink of just under 0.5% (0.24 seconds saved out of 49.51 seconds total) and that's more like what I would have expected. What sort of binaries were you running on the system you measured?
I also don't entirely understand the fragmentation argument. We're not talking about a no-mmu configuration here, are we?
p.
Quoted 10 lines · Phil Blundell
Quoted 3 lines · Mark Hatle
-last- time I got concrete numbers on a complex boot process. It was on the order of 1-5% boot time.. and a fairly large number of pages saved.. (which reduced fragmentation as well as saved memory.)
I'm slightly surprised at the 5% figure. Martin Jansa posted some statistics in another thread earlier today which showed an improvement from prelink of just under 0.5% (0.24 seconds saved out of 49.51 seconds total) and that's more like what I would have expected. What sort of binaries were you running on the system you measured?
At the time, it was systemd, X11, a custom control module and a custom UI program. The custom UI program was written in C++ and had a heck of a lot of libraries attached, and lots of RPC via dbus.
Quoted 2 lines
I also don't entirely understand the fragmentation argument. We're not talking about a no-mmu configuration here, are we?
Fragmentation takes time and can slow down memory allocation. We've got both the kernel page allocation, as well as the internal glibc memory allocation. They work together, but are not the same. (glibc will grab pages of memory from the kernel and then dish them out. As this memory fragments, it has to pull more from the kernel to compensate... as well as free blocks to get more contiguous memory allocated).. It's a bit of a viscous cycle of some small systems.
--Mark
Quoted 3 lines
p.
Quoted 55 lines · Mark Hatle
Quoted 13 lines · Phil Blundell
[Cc list trimmed]
Quoted 4 lines · Mark Hatle
As you can see, pretty much everything is currently broken and I'm struggling to
find community people with the right skill sets to help me resolve the issues.I think it's true that prelink is a technology which has largely had its day, and I could well imagine that upstream Red Hat is no longer very interested in it. https://fedorahosted.org/fesco/ticket/1183 has a summary of some of the reasons why.
Quoted 6 lines
Before investing any significant amount of time in fixing up prelink (and from what you've described it sounds like it might indeed be a significant effort to get it working again on all platforms) it might be a good idea to make sure we are still convinced that prelink provides useful and meaningful benefits for OE-core users.
I do think prelink has a purpose. Especially for embedded systems.
The biggest reason I see against prelink is ASLR. I agree 100% that if you want ASLR (Address Randomization) you should not be using the prelinker.
Usually secondary reasons include if you are doing field upgrades at a package level -- it means you also need the ability to prelink on the device which cause have other performance ramifications and such during this upgrade process.
I agree with some of the information on the Fedora hosted link you pointed to, but disagree with others.
While some of the modern hashing techniques and such do improve run-time dynamic link performance, there is still a hit that we must take. For devices that need quick boot times, quick startup, or are memory constrained, the prelinker can still help. (Memory usage on very small systems is a good example. Memory usage can be reduced in larger applications by reducing the number of Copy-on-write pages required to handle the relocation information.)
I probably would not use the prelinker on a workstation/server any longer. I might not use it on a 'larger' embedded system, but on a more traditional embedded system I do think it still has merit.
(Probably a discussion for the future, due to the ASLR, is should the prelinker be enabled by default -- even if it works -- but that is a discussion for later.)
Quoted 2 lines
If the answer to that is yes then I probably could spare a bit of time to look at the ARM and MIPS issues at least.
I can certainly use any help you have. I have someone looking into the ARM issues with me. It APPEARS the primary behavior there has to do with IFUNC behavior to things like memcpy has changed, causing the prelinker to do the wrong thing(s). But I have few additional details at this time.
Quoted 8 lines
I've reached out to parts of the MIPS community for help, but we'll see if I get any type of response. (MIPS, especially 32-bit, I'm very surprised has broken recently. I haven't seen many changes in regard to the way binaries are loaded on MIPS -- so the issue must be related to a more fundamental optimization or bug in the prelink.)
Thanks for the offer, let me know if you find anything. I'll certainly respond as I make progress trying to go through the issues.
I just wanted to update folks on where I'm at.
On the Intel IA32 architecture, the problem appears to have been triggered by the ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA that went in after March of this year.
I've not been able to track down what is causing the bad behavior -- but fully
dropping support for that class type allows the prelink to "mostly work".
On ARM the issues appear to have some problem related to IFUNCs, but I still don't have any data beyond that at this point.
Quoted 5 lines
--Mark
Quoted 2 lines
p.
Quoted 18 lines · Mark Hatle
Quoted 14 lines · Phil Blundell
Quoted 12 lines · Mark Hatle
While some of the modern hashing techniques and such do improve run -time dynamic
link performance, there is still a hit that we must take. For devices that need
quick boot times, quick startup, or are memory constrained, the prelinker can
still help. (Memory usage on very small systems is a good example. Memory
usage can be reduced in larger applications by reducing the number of Copy-on-write pages required to handle the relocation information.)
Quoted 7 lines
Quoted 3 lines
Do you have any concrete data on how much of a boot time speedup, and/or what reduction in dirty pages, you see on a modern image with prelink versus the same image without?
-last- time I got concrete numbers on a complex boot process. It was on the order of 1-5% boot time.. and a fairly large number of pages saved.. (which reduced fragmentation as well as saved memory.)
How is fragmentation getting better ? if dynamic linker was improving the page locality then it would be possible but that does not seem to be the case here.
Quoted 16 lines
We're not talking huge numbers for each, there is a statistics mode in glibc that can be enabled to show the number of relocations and some of the costs involved. Once this is working again, I will be able to run that and give more concrete numbers.
--Mark
Quoted 2 lines
p.
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 22 lines · Khem Raj
Quoted 22 lines · Mark Hatle
Quoted 17 lines · Phil Blundell
Quoted 12 lines · Mark Hatle
While some of the modern hashing techniques and such do improve run -time dynamic
link performance, there is still a hit that we must take. For devices that need
quick boot times, quick startup, or are memory constrained, the prelinker can
still help. (Memory usage on very small systems is a good example. Memory
usage can be reduced in larger applications by reducing the number of Copy-on-write pages required to handle the relocation information.)Do you have any concrete data on how much of a boot time speedup, and/or what reduction in dirty pages, you see on a modern image with prelink versus the same image without?
Quoted 6 lines
Quoted 3 lines
-last- time I got concrete numbers on a complex boot process. It was on the order of 1-5% boot time.. and a fairly large number of pages saved.. (which reduced fragmentation as well as saved memory.)
How is fragmentation getting better ? if dynamic linker was improving the page locality then it would be possible but that does not seem to be the case here.
With no relocations, you don't end up doing COW pages. No COW pages means less memory used for running many of the small quick Linux scripts at easy boot.
These are issues I've seen on older slower systems. It's not that big of a deal with a lot of the newer ones.
However, someone pointed out to me earlier today that with more and more IFUNCs being used across different platforms, that it's likely the COW page savings are reduced from what they used to be.... If the IFUNCs are 'local to the same page', then it should still help reduce the overall count... but I'm not sure that is something that is controlled in the normal case.
--Mark
Quoted 17 lines
Quoted 16 lines
We're not talking huge numbers for each, there is a statistics mode in glibc that can be enabled to show the number of relocations and some of the costs involved. Once this is working again, I will be able to run that and give more concrete numbers.
--Mark
Quoted 2 lines
p.
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: Prelink status
In reply to Mark HatleI just wanted to update everyone on where we are at. (I won't bore anyone with any followups past this.)
IA32 (32-bit and 64-bit):
We believe the problems are resolved. The error was a combination of prior patch merge issues and a bug in glibc 2.22. A patch for the glibc 2.22 has already been sent to the GLIBC community and I have a change ready for OE.
ARM:
The same glibc 2.22 bug affected ARM as well as IA32.
Also it appears that a patch that went in last fall to enable IFUNC on arm somehow had a hunk dropped from it. Now that we've identified and restored this hunk, it appears to be working again.
MIPS32/MIPS64 (o32, n32, n64):
A new section was added, SHT_MIPS_ABIFLAGS, this was confusing the prelinker causing it to refused to continue. A patch has gone in to instruct the prelinker how to handle this new section (ignore it).
Power (32-bit):
We are still investigating changes that have happened in the past year+ to binutils and glibc for Power systems.
Power (64-bit):
It is likely this will not work anytime soon. The ABI has changed and is no longer compatible with the previous methods of prelinking. This is fairly low priority to be fixed, as it's been broken for a while now.
If anyone has any expertise on Power 32-bit or 64-bit ELF, I'm still looking for help there to get these issues resolved.
--Mark
Quoted 47 lines · Mark Hatle
I'm cross posting this in an effort to ask for help for the cross-prelink project. (If you are unaware of what the prelinker does, it goes through and adjusts the relocation addresses to predefined values in an attempt to speed up executable load times. For embedded systems, this can be fairly critical in meeting boot and runtime load performance targets.)
As some people might be aware, the upstream 'prelink' project that was maintained by Jakub Jelinek at Red Hat appears to have been abandoned.
I've been the maintainer for the Yocto Project's 'prelink-cross' project for a few years, but my work has been dealing with compilation issues and other things that have to do with using the prelinker in a cross compilation environment.
This means that unfortunately I'm in a situation where I am having problems trying to fix the problems in the prelinker itself, but I am not an ELF expert.
Currently the cross prelinker appears to be broken for pretty much every
architecture and I need help trying to resolve these problems.Issues I know of right now:
Introduction of
RTYPE_CLASS_EXTERN_PROTECTED_DATAappears to be causing failures on ARM32, x86 andx86_64.Some possibly unrelated issue is causing various items to relocate incorrectly (IFUNCs perhaps) on ARMv7 builds.
Power is no longer working properly, most likely due to an optimization in the way the PPC GOT table(s) are setup.
Power64 hasn't been working properly for a while, possibly due to the switch to ELFv2.
MIPS and MIPS64 also do not appear to be working, and I've not found a possible cause.
As you can see, pretty much everything is currently broken and I'm struggling to find community people with the right skill sets to help me resolve the issues.
The current state of my work in progress is available at:
git://git.yoctoproject.org/prelink-crossbranch
cross_prelink_stagingIf you think you can help, please let me know.
--Mark