makefile: detect corrupted elf files
PATCH 1/1 makefile: detect corrupted elf files
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.
---
Is below useful for others?
Makefile.target | 7 +++++++ 1 file changed, 7 insertions(+)
Makefile.target +7 -0
@@ -191,3 +191,10 @@ endif GENERATED_HEADERS += config-target.h Makefile: $(GENERATED_HEADERS) + +.SECONDEXPANSION: + +.PHONY: CORRUPTBINARY + +$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY) +
Signature · 1 line
MST
Quoted 10 lines · Michael S. Tsirkin
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.Signed-off-by Michael S. Tsirkin <[email protected]>---
Is below useful for others?
Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.
Quoted 1 line
+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)
If we do do this, we probably ought to be running the cross-prefix version of size, not the host version.
thanks
-- PMM
Quoted 7 lines · Peter Maydell
Quoted 6 lines · Michael S. Tsirkin
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.
Quoted 8 lines
Quoted 4 lines
Signed-off-by Michael S. Tsirkin <[email protected]>---
Is below useful for others?
Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.
It can't if it gets killed by kill -9 or e.g. OOM killer (or OS reboot).
Quoted 4 lines
Quoted 1 line
+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)
If we do do this, we probably ought to be running the cross-prefix version of size, not the host version.
Good point, thanks.
Quoted 2 lines
thanks
-- PMM
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 9 lines · Michael S. Tsirkin
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.Signed-off-by Michael S. Tsirkin <[email protected]>---
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Is below useful for others?
Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.It can't if it gets killed by kill -9 or e.g. OOM killer (or OS reboot).
Any generated file could be truncated then, not just objects.
If you abort a build with kill -9 or equivalent, you blow away the build tree and start over. A sufficiently unlucky truncation could still build, but not work.
[...]
Quoted 22 lines · Markus Armbruster
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 9 lines · Michael S. Tsirkin
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.Signed-off-by Michael S. Tsirkin <[email protected]>---
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Is below useful for others?
Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.It can't if it gets killed by kill -9 or e.g. OOM killer (or OS reboot).
Quoted 7 lines
Any generated file could be truncated then, not just objects.
If you abort a build with kill -9 or equivalent, you blow away the build tree and start over. A sufficiently unlucky truncation could still build, but not work.
[...]
At the moment, true.
It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.
Are people interested? If yes I can implement this.
Signature · 1 line
MST
Quoted 5 lines · Michael S. Tsirkin
It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.
I think that would be better implemented in the compiler/linker :-) Alternatively, stop doing compiles on horribly unstable kernels.
-- PMM
Il 22/05/2013 10:38, Peter Maydell ha scritto:
Quoted 9 lines · Michael S. Tsirkin
Quoted 5 lines
It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.I think that would be better implemented in the compiler/linker :-) Alternatively, stop doing compiles on horribly unstable kernels.
Any filesystem with delayed writes can do this if you have a power loss.
But I agree that this patch doesn't solve the problem. For example, if you get stale files in the ccache directory even zapping the build directory won't do.
Paolo
Quoted 11 lines · Paolo Bonzini
Il 22/05/2013 10:38, Peter Maydell ha scritto:
Quoted 9 lines · Michael S. Tsirkin
Quoted 5 lines
It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.I think that would be better implemented in the compiler/linker :-) Alternatively, stop doing compiles on horribly unstable kernels.
Quoted 7 lines
Any filesystem with delayed writes can do this if you have a power loss.
But I agree that this patch doesn't solve the problem. For example, if you get stale files in the ccache directory even zapping the build directory won't do.
Paolo
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Any other examples?
Signature · 1 line
MST
Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:
Quoted 6 lines
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Any other examples?
Testing configure patches should be done (also) from a clean build directory, for example.
Paolo
Quoted 6 lines · Paolo Bonzini
Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:
Quoted 5 lines
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Quoted 6 lines
Quoted 1 line
Any other examples?
Testing configure patches should be done (also) from a clean build directory, for example.
Paolo
In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.
But testing is a completely separate issue IMO, I'm not trying to fix that, just reduce the chance of a failed or corrupted build.
Signature · 1 line
MST
Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:
Quoted 12 lines · Paolo Bonzini
Quoted 11 lines
Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:
Quoted 6 lines
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Any other examples?
Testing configure patches should be done (also) from a clean build directory, for example.
Quoted 6 lines
Quoted 1 line
Paolo
In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.
Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.
Paolo
Quoted 20 lines · Paolo Bonzini
Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:
Quoted 12 lines · Paolo Bonzini
Quoted 11 lines
Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:
Quoted 6 lines
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Any other examples?
Testing configure patches should be done (also) from a clean build directory, for example.
Quoted 6 lines
Quoted 1 line
Paolo
In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.
Quoted 4 lines
Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.
Paolo
I see, this workflow is the exact reverse of mine:
I do as much as possible in a single tree so I rely on the makefile dependencies to be correct to rebuild the right things.
You don't need so many dependencies: just enough to pull the right bits from the cache.
I can see how any patches correcting in-place rebuilds won't scratch any of your itches.
Signature · 1 line
MST
Il 22/05/2013 12:50, Michael S. Tsirkin ha scritto:
Quoted 19 lines · Paolo Bonzini
Quoted 18 lines
Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:
Quoted 16 lines · Paolo Bonzini
Quoted 10 lines
Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:
Quoted 6 lines
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Any other examples?
Testing configure patches should be done (also) from a clean build directory, for example.
In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.
Quoted 8 lines
Quoted 2 lines
Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.
I see, this workflow is the exact reverse of mine:
I do as much as possible in a single tree so I
rely on the makefile dependencies to be correct
to rebuild the right things.
Usually I do the same---I just do slightly more thorough testing for configure patches.
Paolo
Quoted 25 lines · Paolo Bonzini
Il 22/05/2013 12:50, Michael S. Tsirkin ha scritto:
Quoted 24 lines · Paolo Bonzini
Quoted 13 lines
Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:
Quoted 12 lines · Paolo Bonzini
Quoted 10 lines
Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:
Quoted 6 lines
The fix is simple here: don't use ccache. I don't.
In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.
Any other examples?
Testing configure patches should be done (also) from a clean build directory, for example.
Quoted 7 lines
Quoted 4 lines
In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.
Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.
I see, this workflow is the exact reverse of mine:
Quoted 8 lines
Quoted 3 lines
I do as much as possible in a single tree so I
rely on the makefile dependencies to be correct
to rebuild the right things.Usually I do the same---I just do slightly more thorough testing for configure patches.
Paolo
I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.
Signature · 1 line
MST
Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:
Quoted 6 lines
Quoted 2 lines
Usually I do the same---I just do slightly more thorough testing for configure patches.
I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.
It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))
Paolo
Quoted 4 lines · Paolo Bonzini
Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:
Quoted 3 lines
Quoted 2 lines
Usually I do the same---I just do slightly more thorough testing for configure patches.
Quoted 8 lines
Quoted 3 lines
I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.
It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))
Paolo
Well looking at the source, there's quite a bit of handling of renames, so maybe ccache hackers will be interested in fixing this.
Manpage says:
It should be noted that ccache is susceptible to general storage
problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a bad CCACHE_PREFIX command or compiler wrapper.
...
There are no reported issues about ccache producing broken object files reproducibly. That doesn’t mean it can’t happen, so if you find a repeatable case, please report it.
power failure is not listed ...
Signature · 1 line
MST
Quoted 26 lines · Michael S. Tsirkin
Quoted 4 lines · Paolo Bonzini
Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:
Quoted 3 lines
Quoted 2 lines
Usually I do the same---I just do slightly more thorough testing for configure patches.
Quoted 8 lines
Quoted 3 lines
I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.
It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))
Paolo
Well looking at the source, there's quite a bit of handling of renames, so maybe ccache hackers will be interested in fixing this.
Manpage says:
It should be noted that ccache is susceptible to general storage
problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a badCCACHE_PREFIXcommand or compiler wrapper.
Quoted 8 lines
...
There are no reported issues about ccache producing broken object files reproducibly. That doesn’t mean it can’t happen, so if you find a repeatable case, please report it.
power failure is not listed ...
Neither is kill -9 issued by evil BOFH. IIRC I've also had bad builds and weird errors because the disk was almost completely full (not necessarily due to ccache). Once I overclocked a machine but I had to reduce the speed because of random compile errors.
But I think your patch is way too simple to cover possible failure cases when you can't trust the compile environment. Maybe you should build in two separate directories and compare the resulting objects or just the final executables. For added paranoia, build using two machines which have different set of components from different manufacturers, but identical userland.
Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.
Quoted 3 lines
--
MST
Quoted 30 lines · Blue Swirl
Quoted 29 lines · Michael S. Tsirkin
Quoted 4 lines · Paolo Bonzini
Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:
Quoted 3 lines
Quoted 2 lines
Usually I do the same---I just do slightly more thorough testing for configure patches.
Quoted 8 lines
Quoted 3 lines
I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.
It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))
Paolo
Well looking at the source, there's quite a bit of handling of renames, so maybe ccache hackers will be interested in fixing this.
Manpage says:
It should be noted that ccache is susceptible to general storage
problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a badCCACHE_PREFIXcommand or compiler wrapper....
Quoted 7 lines
Quoted 5 lines
There are no reported issues about ccache producing broken object files reproducibly. That doesn’t mean it can’t happen, so if you find a repeatable case, please report it.
power failure is not listed ...
Neither is kill -9 issued by evil BOFH.
So presumably ccache will not fill with junk if you do this.
Quoted 4 lines
IIRC I've also had bad builds
and weird errors because the disk was almost completely full (not necessarily due to ccache). Once I overclocked a machine but I had to reduce the speed because of random compile errors.
This could be a parallel build, and possibly we have some missing dependencies in the makefile.
Quoted 3 lines
But I think your patch is way too simple to cover possible failure cases when you can't trust the compile environment.
That's not the intent. Merely to address the common failure which I personally observe all the time.
Quoted 11 lines
Maybe you should
build in two separate directories and compare the resulting objects or just the final executables. For added paranoia, build using two machines which have different set of components from different manufacturers, but identical userland.Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.
I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.
Quoted 3 lines
Quoted 3 lines
--
MST
Quoted 10 lines · Michael S. Tsirkin
Quoted 5 lines · Blue Swirl
Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.
I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits. Or you could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.
thanks
-- PMM
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines · Blue Swirl
Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.
Quoted 7 lines
Quoted 3 lines
I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems.
Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
Quoted 6 lines
Or you
could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.thanks
-- PMM
We also run lots of scripts in our makefiles. Would you like to also change each of them individually? Add wrapper scripts for e.g. python? What's the benefit as compared to just fixing it all in one place in the makefile?
Signature · 1 line
MST
Am 26.05.2013 14:31, schrieb Michael S. Tsirkin:
Quoted 24 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Blue Swirl
Quoted 5 lines
Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.
Quoted 6 lines
Quoted 3 lines
I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems.
Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.Quoted 4 lines
Or you
could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.
Quoted 6 lines
Quoted 2 lines
thanks
-- PMMWe also run lots of scripts in our makefiles. Would you like to also change each of them individually? Add wrapper scripts for e.g. python? What's the benefit as compared to just fixing it all in one place in the makefile?
GNU make automatically removes .o files which were built because of a Makefile rule if that rule returns an error, so OOM or compiler crashes should not result in corrupted .o files. The same applies to other kinds of files built by make.
If there are corrupted files, we have to look whether the Makefile rules for those files are correct (or exist at all).
Are there other Open Source projects which try to detect corrupted elf files? I know none.
Regards
Stefan W.
Quoted 33 lines · Stefan Weil
Am 26.05.2013 14:31, schrieb Michael S. Tsirkin:
Quoted 24 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Blue Swirl
Quoted 5 lines
Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.
Quoted 6 lines
Quoted 3 lines
I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems.
Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.Quoted 4 lines
Or you
could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.Quoted 6 lines
Quoted 2 lines
thanks
-- PMMWe also run lots of scripts in our makefiles. Would you like to also change each of them individually? Add wrapper scripts for e.g. python? What's the benefit as compared to just fixing it all in one place in the makefile?
Quoted 4 lines
GNU make automatically removes .o files which were built because of a
Makefilerule if that rule returns an error, so OOM or compiler crashes should not result in corrupted .o files.
Not if make itself is killed.
Quoted 2 lines
The same applies to other kinds of files built by make.
Another problem is power failures and other cases of sudden reboot.
Quoted 9 lines
If there are corrupted files, we have to look whether the
Makefilerules for those files are correct (or exist at all). Are there other Open Source projects which try to detect corrupted elf files? I know none.Regards
Stefan W.
It saves me time, at least.
I can keep it out of tree if it rubs others the wrong way for some reason, it's no big deal.
If I have some spare time I might code up the more generic thing with create then rename for all files, if I do we can discuss that.
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
Quoted 2 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
-- PMM
Quoted 6 lines · Peter Maydell
Quoted 5 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
Quoted 7 lines
Quoted 3 lines
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Quoted 8 lines
Quoted 2 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
-- PMM
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems
A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
+1
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
Quoted 4 lines
Quoted 2 lines
-- PMM
Quoted 40 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systemsA generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
Quoted 5 lines
+1
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time. On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.
If others are not interested, I can keep it out of tree.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
-- PMM
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systemsA generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
+1
Quoted 7 lines
Quoted 3 lines
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.
Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?
Don't you have bigger problems with file systems due to the crashes?
Quoted 2 lines
On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.
For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.
Quoted 2 lines
If others are not interested, I can keep it out of tree.
I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
-- PMM
Quoted 48 lines · Blue Swirl
Quoted 47 lines · Michael S. Tsirkin
Quoted 45 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systemsA generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
+1
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
Quoted 7 lines
Quoted 3 lines
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?
What's the question here?
Quoted 1 line
Don't you have bigger problems with file systems due to the crashes?
As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.
Quoted 5 lines
Quoted 2 lines
On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.
For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.
True, I'm not saying it's perfect.
Quoted 5 lines
Quoted 2 lines
If others are not interested, I can keep it out of tree.
I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.
I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.
It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
-- PMM
Quoted 53 lines · Michael S. Tsirkin
Quoted 52 lines · Blue Swirl
Quoted 50 lines · Michael S. Tsirkin
Quoted 40 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systemsA generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
Quoted 5 lines
+1
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.
Quoted 5 lines
Quoted 3 lines
Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?
What's the question here?
The question is if any of the suggestions solves the problem?
Also how about something this: post boot, find -name '*.o' | xargs -iF sh -c 'if test ! -s F; then rm F;fi'
Quoted 20 lines
Quoted 2 lines
Don't you have bigger problems with file systems due to the crashes?
As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.Quoted 5 lines
Quoted 2 lines
On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.
For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.
True, I'm not saying it's perfect.
Quoted 5 lines
Quoted 2 lines
If others are not interested, I can keep it out of tree.
I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.
Quoted 7 lines
I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.
It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?
I'd prefer a more generic solution, like the hook. What you propose wouldn't protect from the disk full scenario.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
-- PMM
Quoted 58 lines · Blue Swirl
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Blue Swirl
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systemsA generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
+1
Quoted 7 lines
Quoted 3 lines
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?
Quoted 6 lines
Quoted 1 line
What's the question here?
The question is if any of the suggestions solves the problem?
Also how about something this: post boot, find -name '*.o' | xargs -iF sh -c 'if test ! -s F; then rm F;fi'
Maybe. I don't know if it's the only kind one typically sees after a power failure. I'll experiment when this happens next.
Quoted 25 lines
Quoted 25 lines
Quoted 2 lines
Don't you have bigger problems with file systems due to the crashes?
As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.Quoted 5 lines
Quoted 2 lines
On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.
For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.
True, I'm not saying it's perfect.
Quoted 5 lines
Quoted 2 lines
If others are not interested, I can keep it out of tree.
I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.
I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.
Quoted 4 lines
Quoted 2 lines
It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?
I'd prefer a more generic solution, like the hook.
What is meant by the hook?
Quoted 2 lines
What you propose
wouldn't protect from the disk full scenario.
Why not?
I think it will - renaming file in same directory doesn't need any space so is almost sure to succeed even on disk full.
What I mean is this
gcc -o a.o a.c -> rm a.o && gcc -o a.tmp.o a.c && mv a.tmp.o a.o
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
-- PMM
Quoted 58 lines · Blue Swirl
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Blue Swirl
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.
This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
Quoted 7 lines
Quoted 3 lines
This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).
You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systemsA generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.
Quoted 3 lines
This means that we would need the fix in our makefiles even if compiler and linker gain this feature.
Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.
I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.
+1
Quoted 7 lines
Quoted 3 lines
Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?
Quoted 6 lines
Quoted 1 line
What's the question here?
The question is if any of the suggestions solves the problem?
Also how about something this: post boot, find -name '*.o' | xargs -iF sh -c 'if test ! -s F; then rm F;fi'
On Linux, even easier:
find -name '*.o' -empty -exec rm '{}' ';'
Seems to be enough here. Thanks, I'll use this hack and leave makefiles alone for now.
Quoted 37 lines
Quoted 20 lines
Quoted 2 lines
Don't you have bigger problems with file systems due to the crashes?
As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.Quoted 5 lines
Quoted 2 lines
On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.
For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.
True, I'm not saying it's perfect.
Quoted 5 lines
Quoted 2 lines
If others are not interested, I can keep it out of tree.
I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.
Quoted 7 lines
I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.
It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?
I'd prefer a more generic solution, like the hook. What you propose wouldn't protect from the disk full scenario.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
-- PMM
Quoted 8 lines · Michael S. Tsirkin
On Sun, May 26, 2013 at 06:20:17PM +0000, Blue Swirl wrote:
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time. On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.If others are not interested, I can keep it out of tree.
You probably should. Trying to be robust here is going to cause more headache than it's worth.
I think your problem has better solutions too. Doing a full build with all optional dependencies enabled really doesn't take that long.
$ time ( ~/git/qemu/configure && CCACHE_DISABLE=1 make -j24) <lots of output> real 2m28.222s user 21m33.763s sys 1m30.721s
I've switched to this as standard practice since it's so quick.
This is a modest two socket system with spinning disks. I'm sure it's even faster with more recent processors and SSDs. With tmpfs as the build directory it would probably fly.
Our build parallelizes very well, even if you only have slow systems, distcc will work wonders.
Regards,
Anthony Liguori
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
-- PMM
Quoted 8 lines · Peter Maydell
Quoted 5 lines · Michael S. Tsirkin
It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.I think that would be better implemented in the compiler/linker :-)
Ow so you mean you want me to run -rc compiler/linker too?
In any case we are generating quite a bit of
code by ourselves, this would need handling,
we can do it uniformly.
Quoted 3 lines
Alternatively, stop doing compiles on horribly unstable kernels.
-- PMM
Well I do lots of kernel development so I dislike the alternatives:
- Compile on a stable kernel, reboot, test, reboot back to stable kernel
- Compile on another box and copy bits over
Signature · 1 line
MST
Quoted 19 lines · Michael S. Tsirkin
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.Signed-off-by Michael S. Tsirkin <[email protected]>---
Is below useful for others?
Makefile.target | 7 +++++++ 1 file changed, 7 insertions(+)Makefile.target +0
-0
Makefile.target endif · line 199
Quoted 9 lines
Makefile.target +6
-0@@ -191,3 +191,10 @@ endif GENERATED_HEADERS += config-target.h Makefile: $(GENERATED_HEADERS) + +.SECONDEXPANSION: + +.PHONY: CORRUPTBINARY + +$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)
How does size(1) establish the validity of the ELF file? Is it possible to sneak past a truncated file (which I think is the only type of corruption you're trying to protect against)?
Stefan
Quoted 35 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.Signed-off-by Michael S. Tsirkin <[email protected]>---
Is below useful for others?
Makefile.target | 7 +++++++ 1 file changed, 7 insertions(+)Makefile.target +0
-0Makefile.target endif · line 199
Quoted 9 lines
Makefile.target +6
-0@@ -191,3 +191,10 @@ endif GENERATED_HEADERS += config-target.h Makefile: $(GENERATED_HEADERS) + +.SECONDEXPANSION: + +.PHONY: CORRUPTBINARY + +$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)How does size(1) establish the validity of the ELF file?
Is it possible
to sneak past a truncated file (which I think is the only type of corruption you're trying to protect against)?
Quoted 1 line
Stefan
It just parses the header, so of course it is.
But it does seem to catch the common failure
scenarios for me.
Signature · 1 line
MST