Skip to contentSkip to conversation

makefile: detect corrupted elf files

PATCH Superseded Michael S. Tsirkin 8 participants 35 messages May 21, 2013 – May 28, 2013 Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 makefile: detect corrupted elf files

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

---

Is below useful for others?

 Makefile.target | 7 +++++++
 1 file changed, 7 insertions(+)
Makefile.target +7 -0
index ce4391f..4dddee5 100644
@@ -191,3 +191,10 @@ endif
 
 GENERATED_HEADERS += config-target.h
 Makefile: $(GENERATED_HEADERS)
+
+.SECONDEXPANSION:
+
+.PHONY: CORRUPTBINARY
+
+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)
+
Signature · 1 line
MST
#1 lore raw
Quoted 10 lines · Michael S. Tsirkin

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

---

Is below useful for others?

Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.

Quoted 1 line

+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)

If we do do this, we probably ought to be running the cross-prefix version of size, not the host version.

thanks
-- PMM

#2 lore raw
Quoted 7 lines · Peter Maydell
Quoted 6 lines · Michael S. Tsirkin

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

Quoted 8 lines
Quoted 4 lines

---

Is below useful for others?

Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.

It can't if it gets killed by kill -9 or e.g. OOM killer (or OS reboot).

Quoted 4 lines
Quoted 1 line

+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)

If we do do this, we probably ought to be running the cross-prefix version of size, not the host version.

Good point, thanks.

Quoted 2 lines

thanks
-- PMM

#3 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 9 lines · Michael S. Tsirkin

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

---

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Is below useful for others?

Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.

It can't if it gets killed by kill -9 or e.g. OOM killer (or OS reboot).

Any generated file could be truncated then, not just objects.

If you abort a build with kill -9 or equivalent, you blow away the build tree and start over. A sufficiently unlucky truncation could still build, but not work.

[...]

#4 lore raw
Quoted 22 lines · Markus Armbruster
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 9 lines · Michael S. Tsirkin

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

---

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Is below useful for others?

Seems to me like this is just working around a make bug:
it is supposed to delete the partial object if it gets killed.

It can't if it gets killed by kill -9 or e.g. OOM killer (or OS reboot).

Quoted 7 lines

Any generated file could be truncated then, not just objects.

If you abort a build with kill -9 or equivalent, you blow away the build tree and start over. A sufficiently unlucky truncation could still build, but not work.

[...]

At the moment, true.
It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.

Are people interested? If yes I can implement this.

Signature · 1 line
MST
#5 lore raw
Quoted 5 lines · Michael S. Tsirkin

It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.

I think that would be better implemented in the compiler/linker :-) Alternatively, stop doing compiles on horribly unstable kernels.

-- PMM

#6 lore raw

Il 22/05/2013 10:38, Peter Maydell ha scritto:

Quoted 9 lines · Michael S. Tsirkin
Quoted 5 lines

It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.

I think that would be better implemented in the compiler/linker :-) Alternatively, stop doing compiles on horribly unstable kernels.

Any filesystem with delayed writes can do this if you have a power loss.

But I agree that this patch doesn't solve the problem. For example, if you get stale files in the ccache directory even zapping the build directory won't do.

Paolo

#7 lore raw
Quoted 11 lines · Paolo Bonzini

Il 22/05/2013 10:38, Peter Maydell ha scritto:

Quoted 9 lines · Michael S. Tsirkin
Quoted 5 lines

It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.

I think that would be better implemented in the compiler/linker :-) Alternatively, stop doing compiles on horribly unstable kernels.

Quoted 7 lines

Any filesystem with delayed writes can do this if you have a power loss.

But I agree that this patch doesn't solve the problem. For example, if you get stale files in the ccache directory even zapping the build directory won't do.

Paolo

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Any other examples?

Signature · 1 line
MST
#8 lore raw

Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:

Quoted 6 lines

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Any other examples?

Testing configure patches should be done (also) from a clean build directory, for example.

Paolo

#9 lore raw
Quoted 6 lines · Paolo Bonzini

Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:

Quoted 5 lines

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Quoted 6 lines
Quoted 1 line

Any other examples?

Testing configure patches should be done (also) from a clean build directory, for example.

Paolo

In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.

But testing is a completely separate issue IMO, I'm not trying to fix that, just reduce the chance of a failed or corrupted build.

Signature · 1 line
MST
#10 lore raw

Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:

Quoted 12 lines · Paolo Bonzini
Quoted 11 lines

Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:

Quoted 6 lines

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Any other examples?

Testing configure patches should be done (also) from a clean build directory, for example.

Quoted 6 lines
Quoted 1 line

Paolo

In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.

Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.

Paolo

#11 lore raw
Quoted 20 lines · Paolo Bonzini

Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:

Quoted 12 lines · Paolo Bonzini
Quoted 11 lines

Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:

Quoted 6 lines

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Any other examples?

Testing configure patches should be done (also) from a clean build directory, for example.

Quoted 6 lines
Quoted 1 line

Paolo

In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.

Quoted 4 lines

Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.

Paolo

I see, this workflow is the exact reverse of mine:

I do as much as possible in a single tree so I rely on the makefile dependencies to be correct to rebuild the right things.

You don't need so many dependencies: just enough to pull the right bits from the cache.

I can see how any patches correcting in-place rebuilds won't scratch any of your itches.

Signature · 1 line
MST
#12 lore raw

Il 22/05/2013 12:50, Michael S. Tsirkin ha scritto:

Quoted 19 lines · Paolo Bonzini
Quoted 18 lines

Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:

Quoted 16 lines · Paolo Bonzini
Quoted 10 lines

Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:

Quoted 6 lines

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Any other examples?

Testing configure patches should be done (also) from a clean build directory, for example.

In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.

Quoted 8 lines
Quoted 2 lines

Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.

I see, this workflow is the exact reverse of mine:

I do as much as possible in a single tree so I
rely on the makefile dependencies to be correct
to rebuild the right things.

Usually I do the same---I just do slightly more thorough testing for configure patches.

Paolo

#13 lore raw
Quoted 25 lines · Paolo Bonzini

Il 22/05/2013 12:50, Michael S. Tsirkin ha scritto:

Quoted 24 lines · Paolo Bonzini
Quoted 13 lines

Il 22/05/2013 11:42, Michael S. Tsirkin ha scritto:

Quoted 12 lines · Paolo Bonzini
Quoted 10 lines

Il 22/05/2013 10:52, Michael S. Tsirkin ha scritto:

Quoted 6 lines

The fix is simple here: don't use ccache. I don't.

In fact, from what I saw people use ccache to work around makefile bugs, so they can do make clean; make and have it finish quickly.

Any other examples?

Testing configure patches should be done (also) from a clean build directory, for example.

Quoted 7 lines
Quoted 4 lines

In fact, relying on make clean for testing the build system is a mistake. It's easy for it to forget to remove some temporary file. You really should do a clean clone.

Yes, I use a clean clone (and a clean build directory for each patch), _hence_ ccache helps reducing test times.

I see, this workflow is the exact reverse of mine:

Quoted 8 lines
Quoted 3 lines

I do as much as possible in a single tree so I
rely on the makefile dependencies to be correct
to rebuild the right things.

Usually I do the same---I just do slightly more thorough testing for configure patches.

Paolo

I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.

Signature · 1 line
MST
#14 lore raw

Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:

Quoted 6 lines
Quoted 2 lines

Usually I do the same---I just do slightly more thorough testing for configure patches.

I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.

It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))

Paolo

#15 lore raw
Quoted 4 lines · Paolo Bonzini

Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:

Quoted 3 lines
Quoted 2 lines

Usually I do the same---I just do slightly more thorough testing for configure patches.

Quoted 8 lines
Quoted 3 lines

I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.

It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))

Paolo

Well looking at the source, there's quite a bit of handling of renames, so maybe ccache hackers will be interested in fixing this.

Manpage says:
It should be noted that ccache is susceptible to general storage
problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a bad CCACHE_PREFIX command or compiler wrapper.

...

There are no reported issues about ccache producing broken object files reproducibly. That doesn’t mean it can’t happen, so if you find a repeatable case, please report it.

power failure is not listed ...

Signature · 1 line
MST
#16 lore raw
Quoted 26 lines · Michael S. Tsirkin
Quoted 4 lines · Paolo Bonzini

Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:

Quoted 3 lines
Quoted 2 lines

Usually I do the same---I just do slightly more thorough testing for configure patches.

Quoted 8 lines
Quoted 3 lines

I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.

It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))

Paolo

Well looking at the source, there's quite a bit of handling of renames, so maybe ccache hackers will be interested in fixing this.

Manpage says:
It should be noted that ccache is susceptible to general storage
problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a bad CCACHE_PREFIX command or compiler wrapper.

Quoted 8 lines

...

There are no reported issues about ccache producing broken object files reproducibly. That doesn’t mean it can’t happen, so if you find a repeatable case, please report it.

power failure is not listed ...

Neither is kill -9 issued by evil BOFH. IIRC I've also had bad builds and weird errors because the disk was almost completely full (not necessarily due to ccache). Once I overclocked a machine but I had to reduce the speed because of random compile errors.

But I think your patch is way too simple to cover possible failure cases when you can't trust the compile environment. Maybe you should build in two separate directories and compare the resulting objects or just the final executables. For added paranoia, build using two machines which have different set of components from different manufacturers, but identical userland.

Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.

Quoted 3 lines

--
MST

#17 lore raw
Quoted 30 lines · Blue Swirl
Quoted 29 lines · Michael S. Tsirkin
Quoted 4 lines · Paolo Bonzini

Il 22/05/2013 13:09, Michael S. Tsirkin ha scritto:

Quoted 3 lines
Quoted 2 lines

Usually I do the same---I just do slightly more thorough testing for configure patches.

Quoted 8 lines
Quoted 3 lines

I've no idea what happens with ccache on a crash by the way. It's possible that it's careful to do renames in order to not leave corrupted output files behind.

It doesn't, it leave 0-sized files. (Or at least it didn't last time power failed during a compilation. :))

Paolo

Well looking at the source, there's quite a bit of handling of renames, so maybe ccache hackers will be interested in fixing this.

Manpage says:
It should be noted that ccache is susceptible to general storage
problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a bad CCACHE_PREFIX command or compiler wrapper.

...

Quoted 7 lines
Quoted 5 lines

There are no reported issues about ccache producing broken object files reproducibly. That doesn’t mean it can’t happen, so if you find a repeatable case, please report it.

power failure is not listed ...

Neither is kill -9 issued by evil BOFH.

So presumably ccache will not fill with junk if you do this.

Quoted 4 lines

IIRC I've also had bad builds
and weird errors because the disk was almost completely full (not necessarily due to ccache). Once I overclocked a machine but I had to reduce the speed because of random compile errors.

This could be a parallel build, and possibly we have some missing dependencies in the makefile.

Quoted 3 lines

But I think your patch is way too simple to cover possible failure cases when you can't trust the compile environment.

That's not the intent. Merely to address the common failure which I personally observe all the time.

Quoted 11 lines

Maybe you should
build in two separate directories and compare the resulting objects or just the final executables. For added paranoia, build using two machines which have different set of components from different manufacturers, but identical userland.

Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.

I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.

Quoted 3 lines
Quoted 3 lines

--
MST

#18 lore raw
Quoted 10 lines · Michael S. Tsirkin
Quoted 5 lines · Blue Swirl

Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.

I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits. Or you could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.

thanks
-- PMM

#19 lore raw
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines · Blue Swirl

Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.

Quoted 7 lines
Quoted 3 lines

I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems.
Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

Quoted 6 lines

Or you
could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.

thanks
-- PMM

We also run lots of scripts in our makefiles. Would you like to also change each of them individually? Add wrapper scripts for e.g. python? What's the benefit as compared to just fixing it all in one place in the makefile?

Signature · 1 line
MST
#20 lore raw

Am 26.05.2013 14:31, schrieb Michael S. Tsirkin:

Quoted 24 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Blue Swirl
Quoted 5 lines

Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.

Quoted 6 lines
Quoted 3 lines

I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems.
Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

Quoted 4 lines

Or you
could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.

Quoted 6 lines
Quoted 2 lines

thanks
-- PMM

We also run lots of scripts in our makefiles. Would you like to also change each of them individually? Add wrapper scripts for e.g. python? What's the benefit as compared to just fixing it all in one place in the makefile?

GNU make automatically removes .o files which were built because of a Makefile rule if that rule returns an error, so OOM or compiler crashes should not result in corrupted .o files. The same applies to other kinds of files built by make.

If there are corrupted files, we have to look whether the Makefile rules for those files are correct (or exist at all).

Are there other Open Source projects which try to detect corrupted elf files? I know none.

Regards
Stefan W.

#21 lore raw
Quoted 33 lines · Stefan Weil

Am 26.05.2013 14:31, schrieb Michael S. Tsirkin:

Quoted 24 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Blue Swirl
Quoted 5 lines

Another way to handle this would be to enhance GCC and linker to use atomic operations when producing or combining object files. The tools could also print a SHA of the object which the next user should verify. Even better, the object files should include a robust checksum to ensure integrity.

Quoted 6 lines
Quoted 3 lines

I think we can make the makefile more robust. It can create a temporary file in same directory and rename when ready. This will prevent corrupted files from appearing in the first place.

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems.
Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.
You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

Quoted 4 lines

Or you
could write yourself a cc wrapper that did the renaming and use configure's --cc= flag.

Quoted 6 lines
Quoted 2 lines

thanks
-- PMM

We also run lots of scripts in our makefiles. Would you like to also change each of them individually? Add wrapper scripts for e.g. python? What's the benefit as compared to just fixing it all in one place in the makefile?

Quoted 4 lines

GNU make automatically removes .o files which were built because of a Makefile rule if that rule returns an error, so OOM or compiler crashes should not result in corrupted .o files.

Not if make itself is killed.

Quoted 2 lines

The same applies to other kinds of files built by make.

Another problem is power failures and other cases of sudden reboot.

Quoted 9 lines

If there are corrupted files, we have to look whether the Makefile rules for those files are correct (or exist at all). Are there other Open Source projects which try to detect corrupted elf files? I know none.

Regards
Stefan W.

It saves me time, at least.
I can keep it out of tree if it rubs others the wrong way for some reason, it's no big deal.
If I have some spare time I might code up the more generic thing with create then rename for all files, if I do we can discuss that.

Signature · 1 line
MST
#22 lore raw
In reply to Michael S. Tsirkin
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

Quoted 2 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

-- PMM

#23 lore raw
Quoted 6 lines · Peter Maydell
Quoted 5 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

Quoted 7 lines
Quoted 3 lines

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Quoted 8 lines
Quoted 2 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

-- PMM

#24 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

+1

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

Quoted 4 lines
Quoted 2 lines

-- PMM

#25 lore raw
Quoted 40 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

Quoted 5 lines

+1

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time. On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

If others are not interested, I can keep it out of tree.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

-- PMM

#26 lore raw
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

+1

Quoted 7 lines
Quoted 3 lines

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.

Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?

Don't you have bigger problems with file systems due to the crashes?

Quoted 2 lines

On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.

Quoted 2 lines

If others are not interested, I can keep it out of tree.

I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

-- PMM

#27 lore raw
Quoted 48 lines · Blue Swirl
Quoted 47 lines · Michael S. Tsirkin
Quoted 45 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

+1

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

Quoted 7 lines
Quoted 3 lines

It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.

Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?

What's the question here?

Quoted 1 line

Don't you have bigger problems with file systems due to the crashes?

As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.

Quoted 5 lines
Quoted 2 lines

On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.

True, I'm not saying it's perfect.

Quoted 5 lines
Quoted 2 lines

If others are not interested, I can keep it out of tree.

I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.

I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.

It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

-- PMM

#28 lore raw
Quoted 53 lines · Michael S. Tsirkin
Quoted 52 lines · Blue Swirl
Quoted 50 lines · Michael S. Tsirkin
Quoted 40 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

Quoted 5 lines

+1

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.

Quoted 5 lines
Quoted 3 lines

Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?

What's the question here?

The question is if any of the suggestions solves the problem?

Also how about something this: post boot, find -name '*.o' | xargs -iF sh -c 'if test ! -s F; then rm F;fi'

Quoted 20 lines
Quoted 2 lines

Don't you have bigger problems with file systems due to the crashes?

As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.

Quoted 5 lines
Quoted 2 lines

On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.

True, I'm not saying it's perfect.

Quoted 5 lines
Quoted 2 lines

If others are not interested, I can keep it out of tree.

I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.

Quoted 7 lines

I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.

It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?

I'd prefer a more generic solution, like the hook. What you propose wouldn't protect from the disk full scenario.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

-- PMM

#29 lore raw
Quoted 58 lines · Blue Swirl
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Blue Swirl
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

+1

Quoted 7 lines
Quoted 3 lines

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.

Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?

Quoted 6 lines
Quoted 1 line

What's the question here?

The question is if any of the suggestions solves the problem?

Also how about something this: post boot, find -name '*.o' | xargs -iF sh -c 'if test ! -s F; then rm F;fi'

Maybe. I don't know if it's the only kind one typically sees after a power failure. I'll experiment when this happens next.

Quoted 25 lines
Quoted 25 lines
Quoted 2 lines

Don't you have bigger problems with file systems due to the crashes?

As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.

Quoted 5 lines
Quoted 2 lines

On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.

True, I'm not saying it's perfect.

Quoted 5 lines
Quoted 2 lines

If others are not interested, I can keep it out of tree.

I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.

I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.

Quoted 4 lines
Quoted 2 lines

It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?

I'd prefer a more generic solution, like the hook.

What is meant by the hook?

Quoted 2 lines

What you propose
wouldn't protect from the disk full scenario.

Why not?
I think it will - renaming file in same directory doesn't need any space so is almost sure to succeed even on disk full.

What I mean is this

gcc -o a.o a.c -> rm a.o && gcc -o a.tmp.o a.c && mv a.tmp.o a.o

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

-- PMM

#30 lore raw
In reply to Blue Swirl
Quoted 58 lines · Blue Swirl
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Blue Swirl
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Blue Swirl
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell

I definitely think individual project makefiles are the wrong place to fix this. If create-as-temp-and-rename is useful functionality it needs to go in the compiler so that everybody benefits.

This will not help users on existing systems. Also it's not just compiler. We'd have to do it in linker, asm, ... lots of work.

Quoted 7 lines
Quoted 3 lines

This is clearly less work than implementing it in the makefile of every single open source project in the world (or even every single open source project in Debian).

You seem to have removed the part that explained that
1. we run scripts in our makefiles so need to handle that anyway
2. we care about users on existing systems

A generic hook (default none, or maybe "test -s") after object production and before linkage should be enough but would scale to SHA producing/verifying tools.

Quoted 3 lines

This means that we would need the fix in our makefiles even if compiler and linker gain this feature.

Depends on the feature. If the object files have robust checksums which are checked after output and before input, this should be transparent to the build system.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You are wellcome to implement this in compiler/linker/etc if you like but we will still want to handle it in our makefile as well.

I specifically don't want it handled in our makefiles because it's the wrong place to fix the problem and it will make our build system more complicated.

+1

Quoted 7 lines
Quoted 3 lines

Also, what is the worst case scenario? The link fails and you have to clean up and rebuild? An automated build system can't produce the expected output if the build machine is unreliable?

It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time.

Compile under a stable kernel and test the bleeding edge kernel only as KVM guest? Get a different box for testing or compiling? Run 'sync' every time gcc finishes?

Quoted 6 lines
Quoted 1 line

What's the question here?

The question is if any of the suggestions solves the problem?

Also how about something this: post boot, find -name '*.o' | xargs -iF sh -c 'if test ! -s F; then rm F;fi'

On Linux, even easier:
find -name '*.o' -empty -exec rm '{}' ';'

Seems to be enough here. Thanks, I'll use this hack and leave makefiles alone for now.

Quoted 37 lines
Quoted 20 lines
Quoted 2 lines

Don't you have bigger problems with file systems due to the crashes?

As it happens, no. Maybe because I'm using ext4.
Maybe I'm lucky.

Quoted 5 lines
Quoted 2 lines

On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

For example on OpenBSD, 'size' does not seem to come from binutils, so there could be portability issues.

True, I'm not saying it's perfect.

Quoted 5 lines
Quoted 2 lines

If others are not interested, I can keep it out of tree.

I've had problems with disk close to full, so I'm semi-interested if the solution does not slow down others and it's not too ugly.

Quoted 7 lines

I think the simplest way to do it is to change makefile to unlike, create then rename. Then you are safe against abrupt killing or crashing make. And with a journaled fs, if you also have e.g. linux ext4 and mount with data=ordered, you are safe against power failures.

It shouldn't be hard to do and I don't expect this to have any measureable speed impact. What do you think?

I'd prefer a more generic solution, like the hook. What you propose wouldn't protect from the disk full scenario.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

-- PMM

#31 lore raw
In reply to Michael S. Tsirkin
Quoted 8 lines · Michael S. Tsirkin

On Sun, May 26, 2013 at 06:20:17PM +0000, Blue Swirl wrote:
It's a simple issue.
Each time I reboot during build, I have to make clean and rebuild. This wastes my time so I looked for ways to save the time. On my system at least, it has no measureable cost, likely also because size only looks at headers and metadata.

If others are not interested, I can keep it out of tree.

You probably should. Trying to be robust here is going to cause more headache than it's worth.

I think your problem has better solutions too. Doing a full build with all optional dependencies enabled really doesn't take that long.

  $ time ( ~/git/qemu/configure && CCACHE_DISABLE=1 make -j24)
  <lots of output>
  real	2m28.222s
  user	21m33.763s
  sys	1m30.721s

I've switched to this as standard practice since it's so quick.

This is a modest two socket system with spinning disks. I'm sure it's even faster with more recent processors and SSDs. With tmpfs as the build directory it would probably fly.

Our build parallelizes very well, even if you only have slow systems, distcc will work wonders.

Regards,

Anthony Liguori

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

-- PMM

#32 lore raw
In reply to Peter Maydell
Quoted 8 lines · Peter Maydell
Quoted 5 lines · Michael S. Tsirkin

It's my fault for running -rc kernels all the time I guess, I get crashes kind of often, and losing more time on make clean
on top of reboot annoys me.
But we actually could make it robust, even against OS crash. Output to a temporary file then rename. This hack won't be needed then.

I think that would be better implemented in the compiler/linker :-)

Ow so you mean you want me to run -rc compiler/linker too?

In any case we are generating quite a bit of
code by ourselves, this would need handling,
we can do it uniformly.

Quoted 3 lines

Alternatively, stop doing compiles on horribly unstable kernels.

-- PMM

Well I do lots of kernel development so I dislike the alternatives:
- Compile on a stable kernel, reboot, test, reboot back to stable kernel
- Compile on another box and copy bits over

Signature · 1 line
MST
#33 lore raw
In reply to Michael S. Tsirkin
Quoted 19 lines · Michael S. Tsirkin

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

---

Is below useful for others?

 Makefile.target | 7 +++++++
 1 file changed, 7 insertions(+)
Makefile.target +0 -0
index ce4391f..4dddee5 100644

Makefile.target endif · line 199

Quoted 9 lines
Makefile.target +6 -0
@@ -191,3 +191,10 @@ endif
 
 GENERATED_HEADERS += config-target.h
 Makefile: $(GENERATED_HEADERS)
+
+.SECONDEXPANSION:
+
+.PHONY: CORRUPTBINARY
+
+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)

How does size(1) establish the validity of the ELF file? Is it possible to sneak past a truncated file (which I think is the only type of corruption you're trying to protect against)?

Stefan

#34 lore raw
Quoted 35 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin

Once in a while make gets killed and doesn't clean up partial object files after it. Result is nasty errors from link.
This hack checks object is well formed before linking, and rebuilds it if not.

---

Is below useful for others?

 Makefile.target | 7 +++++++
 1 file changed, 7 insertions(+)
Makefile.target +0 -0
index ce4391f..4dddee5 100644

Makefile.target endif · line 199

Quoted 9 lines
Makefile.target +6 -0
@@ -191,3 +191,10 @@ endif
 
 GENERATED_HEADERS += config-target.h
 Makefile: $(GENERATED_HEADERS)
+
+.SECONDEXPANSION:
+
+.PHONY: CORRUPTBINARY
+
+$(all-obj-y): % : $$(if $$(shell size %), , CORRUPTBINARY)

How does size(1) establish the validity of the ELF file?
Is it possible
to sneak past a truncated file (which I think is the only type of corruption you're trying to protect against)?

Quoted 1 line

Stefan

It just parses the header, so of course it is.
But it does seem to catch the common failure
scenarios for me.

Signature · 1 line
MST