Skip to contentSkip to conversation

readd build dependency for startup_raw.S

PATCH In review Olaf Hering 4 participants 10 messages Aug 16, 2023 – Aug 18, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 readd build dependency for startup_raw.S

startup_raw.S includes a generated file, and used to have a dependency in the Makefile to make sure the generated file exists prior compilation. This dependency was removed, and as a result building startup_raw.S randomly fails:

../../20230814T181249.4fdcb339b/grub-core/boot/i386/pc/startup_raw.S:129:10: fatal error: rs_decoder.h: No such file or directory
129 | #include <rs_decoder.h>

      |          ^~~~~~~~~~~~~~

compilation terminated.
make: *** [Makefile:34654: boot/i386/pc/lzma_decompress_image-startup_raw.o] Error 1

Fixes cc2841129652 ("Fix make dist on non-pc.")

---

The referenced commit removed also a dependency for mips. I can not test mips, so I left mips as it is.

 grub-core/Makefile.am | 2 ++
 1 file changed, 2 insertions(+)
grub-core/Makefile.am +2 -0
index f0cb2f2cc..a5020c432 100644
@@ -62,6 +62,8 @@ grub_script.yy.c: grub_script.yy.h
 rs_decoder.h: $(srcdir)/lib/reed_solomon.c
 	$(TARGET_CC) $(TARGET_CPPFLAGS) $(TARGET_CFLAGS) -Os -I$(top_builddir) -S -DSTANDALONE -o $@ $< -g0 -mregparm=3 -ffreestanding
 
+boot/i386/pc/startup_raw.S: $(builddir)/rs_decoder.h
+
 CLEANFILES += grub_script.yy.c grub_script.yy.h
 
 include $(srcdir)/Makefile.core.am

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#1 lore raw

Hi Olaf,

There's an extra 'd' in read in the subject line.

Quoted 11 lines · On Wed, 16 Aug 2023 11:55:01 +0200 Olaf Hering <[email protected]> wrote:

startup_raw.S includes a generated file, and used to have a dependency in the Makefile to make sure the generated file exists prior compilation. This dependency was removed, and as a result building startup_raw.S randomly fails:

../../20230814T181249.4fdcb339b/grub-core/boot/i386/pc/startup_raw.S:129:10: fatal error: rs_decoder.h: No such file or directory
129 | #include <rs_decoder.h>

      |          ^~~~~~~~~~~~~~

compilation terminated.
make: *** [Makefile:34654:
boot/i386/pc/lzma_decompress_image-startup_raw.o] Error 1

I don't see this error when building any of the targets that I test. This includes i386-pc, which I'm assuming this error comes from (if not which?). Are you running make if -jN where N > 1? Does rs_decoder.h exist in the fail build directory? (its a generated file) What compiler and version are you using?

When I look at the contents of ./grub-core/boot/i386/pc/.deps-core/lzma_decompress_image-startup_raw.Po in a successful build directory, I see that GCC figured out that rs_decoder.h is a dependency. So I'm not seeing yet why your build is failing, but the below change *should* be unnecessary.

Glenn

Quoted 27 lines

Fixes cc2841129652 ("Fix make dist on non-pc.")

---

The referenced commit removed also a dependency for mips. I can not test mips, so I left mips as it is.

 grub-core/Makefile.am | 2 ++
 1 file changed, 2 insertions(+)
grub-core/Makefile.am +2 -0
index f0cb2f2cc..a5020c432 100644
@@ -62,6 +62,8 @@ grub_script.yy.c: grub_script.yy.h
 rs_decoder.h: $(srcdir)/lib/reed_solomon.c
 	$(TARGET_CC) $(TARGET_CPPFLAGS) $(TARGET_CFLAGS) -Os -I$(top_builddir) -S -DSTANDALONE -o $@ $< -g0 -mregparm=3 -ffreestanding
 
+boot/i386/pc/startup_raw.S: $(builddir)/rs_decoder.h
+
 CLEANFILES += grub_script.yy.c grub_script.yy.h
 
 include $(srcdir)/Makefile.core.am
Quoted 4 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#2 lore raw
Quoted 20 lines · Glenn Washburn

Hi Olaf,

There's an extra 'd' in read in the subject line.

Quoted 11 lines · On Wed, 16 Aug 2023 11:55:01 +0200 Olaf Hering <[email protected]> wrote:

startup_raw.S includes a generated file, and used to have a dependency in the Makefile to make sure the generated file exists prior compilation. This dependency was removed, and as a result building startup_raw.S randomly fails:

../../20230814T181249.4fdcb339b/grub-core/boot/i386/pc/startup_raw.S:129:10: fatal error: rs_decoder.h: No such file or directory
129 | #include <rs_decoder.h>

      |          ^~~~~~~~~~~~~~

compilation terminated.
make: *** [Makefile:34654:
boot/i386/pc/lzma_decompress_image-startup_raw.o] Error 1

Quoted 5 lines

I don't see this error when building any of the targets that I test. This includes i386-pc, which I'm assuming this error comes from (if not which?). Are you running make if -jN where N > 1? Does rs_decoder.h exist in the fail build directory? (its a generated file) What compiler and version are you using?

This is almost certainly a race condition with parallel make jobs. It might also be possible to trigger it with make --shuffle so that the build order of targets is randomized.

Quoted 5 lines

When I look at the contents of ./grub-core/boot/i386/pc/.deps-core/lzma_decompress_image-startup_raw.Po in a successful build directory, I see that GCC figured out that rs_decoder.h is a dependency. So I'm not seeing yet why your build is failing, but the below change *should* be unnecessary.

I think the .Po files are relevant only for incremental builds -- they let the build system detect which files need to be updated when headers are modified.

For an initial clean build, the .Po files will not exist. Any dependency on generated source files would need to be declared explicitly.

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#3 lore raw
Quoted 28 lines · On Wed, 16 Aug 2023 21:43:25 -0400 Mike Gilbert <[email protected]> wrote:
Quoted 20 lines · Glenn Washburn

Hi Olaf,

There's an extra 'd' in read in the subject line.

Quoted 11 lines · On Wed, 16 Aug 2023 11:55:01 +0200 Olaf Hering <[email protected]> wrote:

startup_raw.S includes a generated file, and used to have a dependency in the Makefile to make sure the generated file exists prior compilation. This dependency was removed, and as a result building startup_raw.S randomly fails:

../../20230814T181249.4fdcb339b/grub-core/boot/i386/pc/startup_raw.S:129:10: fatal error: rs_decoder.h: No such file or directory
129 | #include <rs_decoder.h>

      |          ^~~~~~~~~~~~~~

compilation terminated.
make: *** [Makefile:34654:
boot/i386/pc/lzma_decompress_image-startup_raw.o] Error 1

Quoted 5 lines

I don't see this error when building any of the targets that I test. This includes i386-pc, which I'm assuming this error comes from (if not which?). Are you running make if -jN where N > 1? Does rs_decoder.h exist in the fail build directory? (its a generated file) What compiler and version are you using?

Quoted 3 lines

This is almost certainly a race condition with parallel make jobs. It might also be possible to trigger it with make --shuffle so that the build order of targets is randomized.

Looks like I'm on v4.3 and this was introduced in v4.4. Perhaps you can test this? It certainly did sound like a race condition, which is why I was curious about the level of parallelism used. However, it appears as though the default -j1 was used. The two machines I test on use -j9 and -j25 and I've not seen this yet in the hundreds of runs I've done. So I'm skeptical of this. All my testing is done one Debian 11.

Quoted 6 lines
Quoted 5 lines

When I look at the contents of ./grub-core/boot/i386/pc/.deps-core/lzma_decompress_image-startup_raw.Po in a successful build directory, I see that GCC figured out that rs_decoder.h is a dependency. So I'm not seeing yet why your build is failing, but the below change *should* be unnecessary.

Quoted 7 lines

I think the .Po files are relevant only for incremental builds -- they let the build system detect which files need to be updated when headers are modified.

For an initial clean build, the .Po files will not exist. Any dependency on generated source files would need to be declared explicitly.

Yes, I now believe you're right, though its not 100% clear why to me. The .Po files are included via the include make directive, and so I would think make should fail if they do not exist. But it does seem true that the .Po files are only generated after compilation. So perhaps these rules are being included after compilation, which is why a clean build wouldn't see them but also why make doesn't fail[1].

What I'm seeing is that the generated grub-core/Makefile has rs_decoder.h in BUILT_SOURCES (via several levels of variable expansions). This should mean that rs_decoder.h is generated before any sources are compiled[2]. So it looks good to me. I wonder if Olaf has an old version of automake that has a bug in it?

[1] https://www.gnu.org/software/make/manual/html_node/Include.html
[2] https://www.gnu.org/software/automake/manual/html_node/Sources.html

Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#5 lore raw
In reply to Glenn Washburn

Wed, 16 Aug 2023 14:33:43 -0500 Glenn Washburn <[email protected]>:

Quoted 1 line

There's an extra 'd' in read in the subject line.

Perhaps there is a better phrase to describe the act of adding something back?

Quoted 1 line

I don't see this error when building any of the targets that I test.

It is the result of my attempt to build just the required binaries, without docs. I came up with this:
git clean -dffx
bash bootstrap
mkdir .b
cd .b
../configure [options]
make check
make -C grub-core install-platformDATA install-platformPROGRAMS

But thinking further about it, and as others have noted, the real fix would be to wade through all targets and express their dependencies properly.

So, please disregard this change.

Olaf

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#6 lore raw
Quoted 5 lines · On Thu, 17 Aug 2023 11:00:44 +0200 Olaf Hering <[email protected]> wrote:

Wed, 16 Aug 2023 14:33:43 -0500 Glenn Washburn <[email protected]>:

Quoted 1 line

There's an extra 'd' in read in the subject line.

Perhaps there is a better phrase to describe the act of adding something back?

Ah, it needs a dash, "re-add". I misinterpreted your intent.

Quoted 11 lines
Quoted 1 line

I don't see this error when building any of the targets that I test.

It is the result of my attempt to build just the required binaries, without docs. I came up with this:
git clean -dffx
bash bootstrap
mkdir .b
cd .b
../configure [options]
make check
make -C grub-core install-platformDATA install-platformPROGRAMS

And this avoids the compile error for startup_raw.S?

Quoted 3 lines

But thinking further about it, and as others have noted, the real fix would be to wade through all targets and express their dependencies properly.

What version of automake are you using? I'm on 1.16.3-2 from Debian 11. rs_decoder.h is ending up in BUILT_SOURCES, so it should get built before anything is compiled. Can you check BUILT_SOURCES? For me rs_decoder.h is included via a few levels of variable expansion, so it won't be literally in the definition of BUILT_SOURCES. I'm wondering if maybe you're using an older version of automake with a bug.

Glenn

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#7 lore raw
Quoted 6 lines · Glenn Washburn

What version of automake are you using? I'm on 1.16.3-2 from Debian 11. rs_decoder.h is ending up in BUILT_SOURCES, so it should get built before anything is compiled. Can you check BUILT_SOURCES? For me rs_decoder.h is included via a few levels of variable expansion, so it won't be literally in the definition of BUILT_SOURCES. I'm wondering if maybe you're using an older version of automake with a bug.

Ah, I forgot about BUILT_SOURCES. That should cover it indeed.

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#8 lore raw
In reply to Glenn Washburn

Thu, 17 Aug 2023 15:21:44 -0500 Glenn Washburn <[email protected]>:

Quoted 11 lines
Quoted 9 lines · On Thu, 17 Aug 2023 11:00:44 +0200 Olaf Hering <[email protected]> wrote:

It is the result of my attempt to build just the required binaries, without docs. I came up with this:
git clean -dffx
bash bootstrap
mkdir .b
cd .b
../configure [options]
make check
make -C grub-core install-platformDATA install-platformPROGRAMS

Quoted 1 line

And this avoids the compile error for startup_raw.S?

No, the other way around: this triggers it. Previously I had a simple 'make', followed by the install targets shown above. This builds all dependencies, and also the doc target. To get rid of the doc target, I was under the assumption that install-something already has proper dependencies, but only a subset is built. gnulib.a was missing, and likely others. I did not dig deep.

Quoted 9 lines
Quoted 2 lines

But thinking further about it, and as others have noted, the real fix would be to wade through all targets and express their dependencies properly.

What version of automake are you using? I'm on 1.16.3-2 from Debian 11. rs_decoder.h is ending up in BUILT_SOURCES, so it should get built before anything is compiled. Can you check ,? For me rs_decoder.h is included via a few levels of variable expansion, so it won't be literally in the definition of BUILT_SOURCES. I'm wondering if maybe you're using an older version of automake with a bug.

I have seen it with automake 1.15.1 and 1.16.5. "nodist_lzma_decompress_image_SOURCES" should be part of "BUILT_SOURCES", via "am__append_6232". But another local build, without this change, does indeed fail. The 'check' target does have "BUILT_SOURCES" as dependency, not sure why it does not build the target.

With some debug, the content during 'make check' is just:
BUILT_SOURCES grub_fstest_init.c grub_script.tab.c grub_script.tab.h grub_script.yy.c grub_script.yy.h libgrub_a_init.c

And the real reason is an incorrect invocation of make:
'make check' works in the toplevel, while 'make -C grub-core install-something' works in the given directory. So for me the fix (to avoid the docs target) is:
make check
make -C grub-core check
make -C grub-core install-platformDATA install-platformPROGRAMS

Sorry for the noise.

Olaf

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#9 lore raw
Quoted 40 lines · On Fri, 18 Aug 2023 16:33:19 +0200 Olaf Hering <[email protected]> wrote:

Thu, 17 Aug 2023 15:21:44 -0500 Glenn Washburn <[email protected]>:

Quoted 11 lines
Quoted 9 lines · On Thu, 17 Aug 2023 11:00:44 +0200 Olaf Hering <[email protected]> wrote:

It is the result of my attempt to build just the required binaries, without docs. I came up with this:
git clean -dffx
bash bootstrap
mkdir .b
cd .b
../configure [options]
make check
make -C grub-core install-platformDATA install-platformPROGRAMS

Quoted 1 line

And this avoids the compile error for startup_raw.S?

No, the other way around: this triggers it. Previously I had a simple 'make', followed by the install targets shown above. This builds all dependencies, and also the doc target. To get rid of the doc target, I was under the assumption that install-something already has proper dependencies, but only a subset is built. gnulib.a was missing, and likely others. I did not dig deep.

Quoted 9 lines
Quoted 2 lines

But thinking further about it, and as others have noted, the real fix would be to wade through all targets and express their dependencies properly.

What version of automake are you using? I'm on 1.16.3-2 from Debian 11. rs_decoder.h is ending up in BUILT_SOURCES, so it should get built before anything is compiled. Can you check ,? For me rs_decoder.h is included via a few levels of variable expansion, so it won't be literally in the definition of BUILT_SOURCES. I'm wondering if maybe you're using an older version of automake with a bug.

I have seen it with automake 1.15.1 and 1.16.5. "nodist_lzma_decompress_image_SOURCES" should be part of "BUILT_SOURCES", via "am__append_6232". But another local build, without this change, does indeed fail. The 'check' target does have "BUILT_SOURCES" as dependency, not sure why it does not build the target.

With some debug, the content during 'make check' is just:
BUILT_SOURCES grub_fstest_init.c grub_script.tab.c grub_script.tab.h grub_script.yy.c grub_script.yy.h libgrub_a_init.c

Quoted 8 lines

And the real reason is an incorrect invocation of make:
'make check' works in the toplevel, while 'make -C grub-core install-something' works in the given directory. So for me the fix (to avoid the docs target) is:
make check
make -C grub-core check
make -C grub-core install-platformDATA install-platformPROGRAMS

Sorry for the noise.

Ok, I see now the issue. "A source file listed in BUILT_SOURCES is made when ‘make all’, ‘make check’, ‘make install’, ‘make install-exec’ (or make dist) is run, before other targets are processed"[1]. This is why the previous was failing and this one now works (ie. make check uses BUILT_SOURCES while the other targets do not).

Glenn

[1] https://www.gnu.org/software/automake/manual/html_node/Sources.html

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel