Skip to contentSkip to conversation

package/pkg-generic.mk: fix rule order for re{install, build, configure}

PATCH In review James Hilliard 6 participants 13 messages Oct 18, 2022 – Oct 13, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/pkg-generic.mk: fix rule order for re{install, build, configure}

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.

The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
package/pkg-generic.mk +6 -3
index f24e03a325..6cb461af90 100644
@@ -1057,17 +1057,20 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-reinstall:		$(1)-clean-for-reinstall $(1)
+$(1)-reinstall::		$(1)-clean-for-reinstall
+$(1)-reinstall::		$(1)
 
 $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
 			rm -f $$($(2)_TARGET_BUILD)
 
-$(1)-rebuild:		$(1)-clean-for-rebuild $(1)
+$(1)-rebuild::		$(1)-clean-for-rebuild
+$(1)-rebuild::		$(1)
 
 $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
 			rm -f $$($(2)_TARGET_CONFIGURE)
 
-$(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::	$(1)-clean-for-reconfigure
+$(1)-reconfigure::	$(1)
 
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
Signature · 6 lines
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Tested-by on v1 1/1
#1 lore raw
Quoted 6 lines · James Hilliard

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.

The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.

Quoted 8 lines

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

Set LINUX_OVERRIDE_SRCDIR in local.mk
touch arm64/kernel/setup.c
make linux-rebuild
See Building steps with CC on the changed file setup.c

Quoted 35 lines
---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
package/pkg-generic.mk +6 -4
index f24e03a325..6cb461af90 100644
@@ -1057,17 +1057,20 @@ endif
                        rm -f $$($(2)_TARGET_INSTALL_IMAGES)
                        rm -f $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-reinstall:                $(1)-clean-for-reinstall $(1)
+$(1)-reinstall::               $(1)-clean-for-reinstall
+$(1)-reinstall::               $(1)
 
 $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
                        rm -f $$($(2)_TARGET_BUILD)
 
-$(1)-rebuild:          $(1)-clean-for-rebuild $(1)
+$(1)-rebuild::         $(1)-clean-for-rebuild
+$(1)-rebuild::         $(1)
 
 $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
                        rm -f $$($(2)_TARGET_CONFIGURE)
 
-$(1)-reconfigure:      $(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::     $(1)-clean-for-reconfigure
+$(1)-reconfigure::     $(1)
 
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
--

2.34.1

Quoted 5 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#2 lore raw
In reply to James Hilliard

James, All,

Thanks again for working on such grunt work, it's much appreaciated!

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 11 lines

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.

The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

I am not convinced that we should switch over to using double-colons.

As I reviewed your other related patch about legal-info [0], the make manual [1] suggests that double-colons are rally intended for cases where "the method used to update a target differs depending on which prerequisite files caused the update" and "double-colon rules really make sense are those where the order of executing the recipes would not matter".

So, we'd use a mechanism where order should not matter, to solve an issue with ordering. That'd be weird...

Yes, the manual states that "double-colon rules for a target are executed in the order they appear in the makefile", but that really looks like a side-effect of another goal.

Also, we'd need to carefully review all our Makefile to ensure that there are no other cases wehre ordering of prerequisites matter. We'd also have to be careful every time we introduce a new rule and remember to check whether it depends on the ordering prereauisites or not, adn worse yet, also whether we can afford the rule to be parallel-safe or not (see the explanation for that in the legal-info review).

Having colons and double-colons is going to be quite a headache... We already have issues with remembering that we need to use $$ to expand variables in some places, and the rules there are relatively clean and explicit, but for the :: case, I'm afraid that's going to be a bit more complex.

We currently have a single location where we use double-colon:
support/kconfig/Makefile.br

which is added by our patch against kconfig:
support/kconfig/patches/10-br-build-system.patch

which dates back to 2010:
7c524dd0b683 Clean up our patches against kconfig

which got in three in 2007:

    a665ed34960b - pull kconfig from linux-2.6.21.5

So, I'd like that we get another way to fix that, if possible...

Again, thanks a lot for spending efforts in that area, this is really great!

[0] https://lore.kernel.org/buildroot/20230212091914.GJ2796@scaer/
[1] https://www.gnu.org/software/make/manual/make.html#Double_002dColon

Regards,
Yann E. MORIN.

Quoted 40 lines
---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
package/pkg-generic.mk +6 -3
index f24e03a325..6cb461af90 100644
@@ -1057,17 +1057,20 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-reinstall:		$(1)-clean-for-reinstall $(1)
+$(1)-reinstall::		$(1)-clean-for-reinstall
+$(1)-reinstall::		$(1)
 
 $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
 			rm -f $$($(2)_TARGET_BUILD)
 
-$(1)-rebuild:		$(1)-clean-for-rebuild $(1)
+$(1)-rebuild::		$(1)-clean-for-rebuild
+$(1)-rebuild::		$(1)
 
 $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
 			rm -f $$($(2)_TARGET_CONFIGURE)
 
-$(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::	$(1)-clean-for-reconfigure
+$(1)-reconfigure::	$(1)
 
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
Signature · 6 lines
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw
Quoted 18 lines · Yann E. MORIN

James, All,

Thanks again for working on such grunt work, it's much appreaciated!

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 11 lines

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.

The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

Quoted 8 lines

I am not convinced that we should switch over to using double-colons.

As I reviewed your other related patch about legal-info [0], the make manual [1] suggests that double-colons are rally intended for cases where "the method used to update a target differs depending on which prerequisite files caused the update" and "double-colon rules really make sense are those where the order of executing the recipes would not matter".

This appears to be the only documented method of enforcing execution order without disabling parallel builds...that I've come across at least.

Quoted 7 lines

So, we'd use a mechanism where order should not matter, to solve an issue with ordering. That'd be weird...

Yes, the manual states that "double-colon rules for a target are executed in the order they appear in the makefile", but that really looks like a side-effect of another goal.

Yeah, I agree it's not really the intended use case for double-colon rules, however the ordering behavior is documented so I would not expect it to break.

Quoted 7 lines

Also, we'd need to carefully review all our Makefile to ensure that there are no other cases wehre ordering of prerequisites matter. We'd also have to be careful every time we introduce a new rule and remember to check whether it depends on the ordering prereauisites or not, adn worse yet, also whether we can afford the rule to be parallel-safe or not (see the explanation for that in the legal-info review).

So unlike the legal-info issue using .NOTPARALLEL would not work here as these make targets do need to work with parallel builds.

Quoted 6 lines

Having colons and double-colons is going to be quite a headache... We already have issues with remembering that we need to use $$ to expand variables in some places, and the rules there are relatively clean and explicit, but for the :: case, I'm afraid that's going to be a bit more complex.

I don't think we have all that many of these issues, but I could be wrong.

The clean-for-re{install, build, configure} make targets are a bit hacky to begin with so I don't think make was intended to handle those cases in the first place. We're already in unintended feature use case territory here so this doesn't seem to make things much worse.

Quoted 7 lines

We currently have a single location where we use double-colon:
support/kconfig/Makefile.br

which is added by our patch against kconfig:
support/kconfig/patches/10-br-build-system.patch

Quoted 7 lines

which dates back to 2010:
7c524dd0b683 Clean up our patches against kconfig

which got in three in 2007:

    a665ed34960b - pull kconfig from linux-2.6.21.5

So, I'd like that we get another way to fix that, if possible...

I have unfortunately not found any other method to fix the execution ordering...

Quoted 53 lines

Again, thanks a lot for spending efforts in that area, this is really great!

[0] https://lore.kernel.org/buildroot/20230212091914.GJ2796@scaer/
[1] https://www.gnu.org/software/make/manual/make.html#Double_002dColon

Regards,
Yann E. MORIN.

Quoted 36 lines
---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
package/pkg-generic.mk +6 -4
index f24e03a325..6cb461af90 100644
@@ -1057,17 +1057,20 @@ endif
                      rm -f $$($(2)_TARGET_INSTALL_IMAGES)
                      rm -f $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-reinstall:              $(1)-clean-for-reinstall $(1)
+$(1)-reinstall::             $(1)-clean-for-reinstall
+$(1)-reinstall::             $(1)
 
 $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
                      rm -f $$($(2)_TARGET_BUILD)
 
-$(1)-rebuild:                $(1)-clean-for-rebuild $(1)
+$(1)-rebuild::               $(1)-clean-for-rebuild
+$(1)-rebuild::               $(1)
 
 $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
                      rm -f $$($(2)_TARGET_CONFIGURE)
 
-$(1)-reconfigure:    $(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::   $(1)-clean-for-reconfigure
+$(1)-reconfigure::   $(1)
 
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
--

2.34.1

Quoted 4 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

--
.-----------------.--------------------.------------------.--------------------.

Quoted 5 lines
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#4 lore raw

James, All,

On 2023-02-12 03:02 -0700, James Hilliard spake thusly:

Quoted 7 lines · Yann E. MORIN
Quoted 6 lines

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 5 lines

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

[--SNIP--]

Quoted 3 lines
Quoted 1 line

So, I'd like that we get another way to fix that, if possible...

I have unfortunately not found any other method to fix the execution ordering...

What about something along the lines of (untestd!):

    $(1)-rebuild: $(1): $(1)-clean-for-rebuild
    $(1)-rebuild: $(1)

We already use such a construct in package/pkg-kconfig.mk, but only once, so this is not as if it were widespread either...

Still, I think it is more easy to read...

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw
Quoted 12 lines · Yann E. MORIN

James, All,

On 2023-02-12 03:02 -0700, James Hilliard spake thusly:

Quoted 7 lines · Yann E. MORIN
Quoted 6 lines

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 5 lines

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

[--SNIP--]

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

So, I'd like that we get another way to fix that, if possible...

I have unfortunately not found any other method to fix the execution ordering...

What about something along the lines of (untestd!):

    $(1)-rebuild: $(1): $(1)-clean-for-rebuild
    $(1)-rebuild: $(1)

That would appear to be invalid syntax.

Quoted 3 lines

We already use such a construct in package/pkg-kconfig.mk, but only once, so this is not as if it were widespread either...

I don't see anything like that there, just this which isn't doing anything unusual with multiple prerequisites that require a specific execution order like the re{install,build,configure} rules in pkg-generic.mk:

# Force olddefconfig again on -reconfigure $(1)-clean-for-reconfigure: $(1)-clean-kconfig-for-reconfigure

$(1)-clean-kconfig-for-reconfigure:

    rm -f $$($(2)_DIR)/$$($(2)_KCONFIG_STAMP_DOTCONFIG)
Quoted 6 lines

Still, I think it is more easy to read...

Regards,
Yann E. MORIN.

Quoted 7 lines

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#6 lore raw

James, All,

On 2023-02-12 03:22 -0700, James Hilliard spake thusly:

Quoted 22 lines · Yann E. MORIN
Quoted 12 lines

James, All,

On 2023-02-12 03:02 -0700, James Hilliard spake thusly:

Quoted 7 lines · Yann E. MORIN
Quoted 6 lines

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 5 lines

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

[--SNIP--]

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

So, I'd like that we get another way to fix that, if possible...

I have unfortunately not found any other method to fix the execution ordering...

What about something along the lines of (untestd!):

    $(1)-rebuild: $(1): $(1)-clean-for-rebuild
    $(1)-rebuild: $(1)
Quoted 7 lines

That would appear to be invalid syntax.

Quoted 3 lines

We already use such a construct in package/pkg-kconfig.mk, but only once, so this is not as if it were widespread either...

I don't see anything like that there,

    240:  $$(addprefix $(1)-,$$($(2)_KCONFIG_EDITORS)): $(1)-%: $$($(2)_DIR)/.kconfig_editor_%

So, this looks like it is named "target pattern", and it works with a pattern (eh!).

    $ cat Makefile
     1  all: a
     2
     3  a:
     4          @echo a
     5
     6  a-clean:
     7          @sleep 1
     8          @echo a-clean
     9
    10  a-rebuild: %-rebuild: %-clean
    11  a-rebuild: a

However, that does not work with parallel builds:

    $ make
    a
    $ make a
    a
    $ make a-rebuild
    a-clean
    a
    $ make -j a-rebuild
    a
    a-clean

However, you stated in a previous reply:

So unlike the legal-info issue using .NOTPARALLEL would not work here as these make targets do need to work with parallel builds.

I don't think that's true. Indeed, foo-rebuild is usually done after a ful build, when iterating over the foo package, iether to test a new source tree (with ocerride-ssrcdir) or to test fixes in foo.mk. In either case, we do not need a top-level parallel build, because only one package, foo, will be built.

So, even in this case, we should be able to use .NOTPARALLEL, no?

It also means that the kconfig rule I mentionned above, does not work with top-level parallel builds, but that's OK: it's only used to spawn kconfig configurators (menuconfig et al.), so it should not be used with top-level parallel, and thus should require .NOTPARALLEL as well.

Meh, TLPB and shuffling are really uncovering issues everywhere... :-(

Regards,
Yann E. MORIN.

Quoted 23 lines

just this which isn't doing anything
unusual with multiple prerequisites that require a specific execution order like the re{install,build,configure} rules in pkg-generic.mk:

# Force olddefconfig again on -reconfigure $(1)-clean-for-reconfigure: $(1)-clean-kconfig-for-reconfigure

$(1)-clean-kconfig-for-reconfigure:

    rm -f $$($(2)_DIR)/$$($(2)_KCONFIG_STAMP_DOTCONFIG)
Quoted 13 lines

Still, I think it is more easy to read...

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#7 lore raw
Quoted 73 lines · Yann E. MORIN

James, All,

On 2023-02-12 03:22 -0700, James Hilliard spake thusly:

Quoted 22 lines · Yann E. MORIN
Quoted 12 lines

James, All,

On 2023-02-12 03:02 -0700, James Hilliard spake thusly:

Quoted 7 lines · Yann E. MORIN
Quoted 6 lines

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 5 lines

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

[--SNIP--]

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

So, I'd like that we get another way to fix that, if possible...

I have unfortunately not found any other method to fix the execution ordering...

What about something along the lines of (untestd!):

    $(1)-rebuild: $(1): $(1)-clean-for-rebuild
    $(1)-rebuild: $(1)
Quoted 7 lines

That would appear to be invalid syntax.

Quoted 3 lines

We already use such a construct in package/pkg-kconfig.mk, but only once, so this is not as if it were widespread either...

I don't see anything like that there,

    240:  $$(addprefix $(1)-,$$($(2)_KCONFIG_EDITORS)): $(1)-%: $$($(2)_DIR)/.kconfig_editor_%

So, this looks like it is named "target pattern", and it works with a pattern (eh!).

    $ cat Makefile
     1  all: a
     2
     3  a:
     4          @echo a
     5
     6  a-clean:
     7          @sleep 1
     8          @echo a-clean
     9
    10  a-rebuild: %-rebuild: %-clean
    11  a-rebuild: a

However, that does not work with parallel builds:

    $ make
    a
    $ make a
    a
    $ make a-rebuild
    a-clean
    a
    $ make -j a-rebuild
    a
    a-clean

However, you stated in a previous reply:

So unlike the legal-info issue using .NOTPARALLEL would not work here as these make targets do need to work with parallel builds.

Quoted 7 lines

I don't think that's true. Indeed, foo-rebuild is usually done after a ful build, when iterating over the foo package, iether to test a new source tree (with ocerride-ssrcdir) or to test fixes in foo.mk. In either case, we do not need a top-level parallel build, because only one package, foo, will be built.

So, even in this case, we should be able to use .NOTPARALLEL, no?

Breaking parallel build for foo-rebuild would be rather unexpected behavior as foo-rebuild does work for full builds as well and one may default to using foo-rebuild instead of a foo target for various reasons(to ensure that a package is always rsynced from an overridden source directory for example).

I see no good reason to disable parallel build on it when simply ordering the dependency execution order fixes the issue.

Quoted 5 lines

It also means that the kconfig rule I mentionned above, does not work with top-level parallel builds, but that's OK: it's only used to spawn kconfig configurators (menuconfig et al.), so it should not be used with top-level parallel, and thus should require .NOTPARALLEL as well.

Might be better to use double colon rules there as well, they are more explicit than using .NOTPARALLEL IMO as .NOTPARALLEL doesn't specify an execution order, also it can potentially introduce accidental performance regressions.

Quoted 2 lines

Meh, TLPB and shuffling are really uncovering issues everywhere... :-(

Better to make them very obvious so they can be fixed IMO.

Once the known issues are fixed I can run shuffle mode on my autobuilders to uncover remaining issues.

Quoted 28 lines

Regards,
Yann E. MORIN.

Quoted 4 lines

just this which isn't doing anything
unusual with multiple prerequisites that require a specific execution order like the re{install,build,configure} rules in pkg-generic.mk:

Quoted 19 lines

# Force olddefconfig again on -reconfigure $(1)-clean-for-reconfigure: $(1)-clean-kconfig-for-reconfigure

$(1)-clean-kconfig-for-reconfigure:

    rm -f $$($(2)_DIR)/$$($(2)_KCONFIG_STAMP_DOTCONFIG)
Quoted 13 lines

Still, I think it is more easy to read...

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

Quoted 7 lines

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#8 lore raw
In reply to Yann E. MORIN
Quoted 11 lines · Yann E. MORIN

James, All,

Thanks again for working on such grunt work, it's much appreaciated!

On 2022-10-17 21:46 -0600, James Hilliard spake thusly:

Quoted 6 lines

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.

The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.

Quoted 7 lines
Quoted 5 lines

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

I am not convinced that we should switch over to using double-colons.

Me neither.

Quoted 10 lines

As I reviewed your other related patch about legal-info [0], the make manual [1] suggests that double-colons are rally intended for cases where "the method used to update a target differs depending on which prerequisite files caused the update" and "double-colon rules really make sense are those where the order of executing the recipes would not matter".

So, we'd use a mechanism where order should not matter, to solve an issue with ordering. That'd be weird...

Quoted 6 lines

Yes, the manual states that "double-colon rules for a target are executed in the order they appear in the makefile", but that really looks like a side-effect of another goal.

Also, we'd need to carefully review all our Makefile to ensure that there are no other cases wehre ordering of prerequisites matter.

Well, we need to do that regardless of which solution is found, right? Unless
we just disable parallel builds and --shuffle entirely.

Quoted 8 lines

We'd
also have to be careful every time we introduce a new rule and remember to check whether it depends on the ordering prereauisites or not, adn worse yet, also whether we can afford the rule to be parallel-safe or not (see the explanation for that in the legal-info review).

Having colons and double-colons is going to be quite a headache... We already have issues with remembering that we need to use $$ to expand

Yeah, maybe we should have some check-package thing to verify that $$ is used
everywhere in a macro definition...

Quoted 40 lines

variables in some places, and the rules there are relatively clean and explicit, but for the :: case, I'm afraid that's going to be a bit more complex.

We currently have a single location where we use double-colon:
support/kconfig/Makefile.br

which is added by our patch against kconfig:
support/kconfig/patches/10-br-build-system.patch

which dates back to 2010:
7c524dd0b683 Clean up our patches against kconfig

which got in three in 2007:

     a665ed34960b - pull kconfig from linux-2.6.21.5

So, I'd like that we get another way to fix that, if possible...

Again, thanks a lot for spending efforts in that area, this is really great!

[0] https://lore.kernel.org/buildroot/20230212091914.GJ2796@scaer/
[1] https://www.gnu.org/software/make/manual/make.html#Double_002dColon

Regards,
Yann E. MORIN.

Quoted 13 lines
---
  package/pkg-generic.mk | 9 ++++++---
  1 file changed, 6 insertions(+), 3 deletions(-)
package/pkg-generic.mk +0 -0
index f24e03a325..6cb461af90 100644
@@ -1057,17 +1057,20 @@ endif
  			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
  			rm -f $$($(2)_TARGET_INSTALL_HOST)
 
Quoted 3 lines

package/pkg-generic.mk endif · line 1061

Quoted 3 lines
package/pkg-generic.mk +2 -1
-$(1)-reinstall:		$(1)-clean-for-reinstall $(1)
+$(1)-reinstall::		$(1)-clean-for-reinstall
+$(1)-reinstall::		$(1)

I may be stupid, but what we want is to have an ordering between
clean-for-reinstall and $(1), right? So, let's use an ordering dependency?

$(1)-reinstall: $(1)-clean-for-reinstall $(1)
$(1): $(1)-clean-for-reinstall

Regards,
Arnout

Quoted 20 lines
Quoted 20 lines
package/pkg-generic.mk +4 -2
  $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
  			rm -f $$($(2)_TARGET_BUILD)
 
-$(1)-rebuild:		$(1)-clean-for-rebuild $(1)
+$(1)-rebuild::		$(1)-clean-for-rebuild
+$(1)-rebuild::		$(1)
 
  $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
  			rm -f $$($(2)_TARGET_CONFIGURE)
 
-$(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::	$(1)-clean-for-reconfigure
+$(1)-reconfigure::	$(1)
 
  # define the PKG variable for all targets, containing the
  # uppercase package variable prefix
Signature · 1 line
2.34.1
Quoted 5 lines
Quoted 4 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#9 lore raw

On 14/02/2023 22:22, Arnout Vandecappelle wrote:
[1]

Quoted 4 lines

 I may be stupid, but what we want is to have an ordering between clean-for-reinstall and $(1), right? So, let's use an ordering dependency?

$(1)-reinstall:        $(1)-clean-for-reinstall $(1) $(1):             $(1)-clean-for-reinstall

 Gah, I forgot the actual order-only bit...

$(1)-reinstall:   $(1)-clean-for-reinstall $(1)
$(1):           | $(1)-clean-for-reinstall

 Regards,
 Arnout

Quoted 25 lines

 Regards,
 Arnout

Quoted 20 lines
Quoted 19 lines

    $(1)-clean-for-rebuild: $(1)-clean-for-reinstall rm -f $$($(2)_TARGET_BUILD) -$(1)-rebuild:        $(1)-clean-for-rebuild $(1) +$(1)-rebuild::        $(1)-clean-for-rebuild +$(1)-rebuild::        $(1) $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild rm -f $$($(2)_TARGET_CONFIGURE) -$(1)-reconfigure:    $(1)-clean-for-reconfigure $(1) +$(1)-reconfigure::    $(1)-clean-for-reconfigure +$(1)-reconfigure::    $(1)

    # define the PKG variable for all targets, containing the
  # uppercase package variable prefix
Signature · 6 lines
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#10 lore raw
Quoted 7 lines · Arnout Vandecappelle

On 14/02/2023 22:22, Arnout Vandecappelle wrote:
[1]

Quoted 3 lines

I may be stupid, but what we want is to have an ordering between
clean-for-reinstall and $(1), right? So, let's use an ordering dependency?

Quoted 8 lines
Quoted 2 lines

$(1)-reinstall: $(1)-clean-for-reinstall $(1)
$(1): $(1)-clean-for-reinstall

Gah, I forgot the actual order-only bit...

$(1)-reinstall: $(1)-clean-for-reinstall $(1)
$(1): | $(1)-clean-for-reinstall

This doesn't seem to work, it ends up cleaning a bunch of additional targets like the skeleton and toolchain that shouldn't be cleaned when doing a reinstall for say a normal python package.

Quoted 31 lines

Regards,
Arnout

Quoted 25 lines

Regards,
Arnout

Quoted 20 lines
Quoted 19 lines
    $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
              rm -f $$($(2)_TARGET_BUILD)
  -$(1)-rebuild:        $(1)-clean-for-rebuild $(1)
+$(1)-rebuild::        $(1)-clean-for-rebuild
+$(1)-rebuild::        $(1)
    $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
              rm -f $$($(2)_TARGET_CONFIGURE)
  -$(1)-reconfigure:    $(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::    $(1)-clean-for-reconfigure
+$(1)-reconfigure::    $(1)
    # define the PKG variable for all targets, containing the
  # uppercase package variable prefix
--
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#11 lore raw
In reply to James Hilliard

Hi James,

Quoted 11 lines · James Hilliard

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.

The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.

To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:

The double-colon rules for a target are executed in the order they appear in the makefile.

As noted in earlier discussions, this feels a bit hackish and it's not clear
if this behaviour is going to be kept in future versions of make.

Since make 4.4, however, there's a better solution: it introduced a .WAIT
phony target that enforces the ordering of dependencies. So I instead made a commit[1] that uses that approach.

Regards,
Arnout

[1] https://gitlab.com/buildroot.org/buildroot/-/commit/7e3105d5c8bd9ab31e728af9732a67d203c0a2e4

Quoted 24 lines
---
  package/pkg-generic.mk | 9 ++++++---
  1 file changed, 6 insertions(+), 3 deletions(-)
package/pkg-generic.mk +4 -2
index f24e03a325..6cb461af90 100644
@@ -1057,17 +1057,20 @@ endif
  			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
  			rm -f $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-reinstall:		$(1)-clean-for-reinstall $(1)
+$(1)-reinstall::		$(1)-clean-for-reinstall
+$(1)-reinstall::		$(1)
 
  $(1)-clean-for-rebuild: $(1)-clean-for-reinstall
  			rm -f $$($(2)_TARGET_BUILD)
 
-$(1)-rebuild:		$(1)-clean-for-rebuild $(1)
+$(1)-rebuild::		$(1)-clean-for-rebuild
+$(1)-rebuild::		$(1)

package/pkg-generic.mk endif · line 1076

Quoted 10 lines
package/pkg-generic.mk +2 -1
 
  $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
  			rm -f $$($(2)_TARGET_CONFIGURE)
 
-$(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
+$(1)-reconfigure::	$(1)-clean-for-reconfigure
+$(1)-reconfigure::	$(1)
 
  # define the PKG variable for all targets, containing the
  # uppercase package variable prefix

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#12 lore raw
Quoted 13 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle via buildroot <[email protected]> writes:

Hi James,

Quoted 8 lines · James Hilliard

These command rely on the clean operations being first so that the stamp files being deleted will rebuild the targets.
The execution ordering of the clean and rebuild operations may change, for example if --shuffle=reversed is set.
To ensure the evaluation order is always correct use double colon rules to make the evaluation order explicit as per make docs:
The double-colon rules for a target are executed in the order they appear in the makefile.

Quoted 7 lines

As noted in earlier discussions, this feels a bit hackish and it's not clear if this behaviour is going to be kept in future versions of
make.

Since make 4.4, however, there's a better solution: it introduced a .WAIT phony target that enforces the ordering of dependencies. So I
instead made a commit[1] that uses that approach.

Committed to 2023.02.x and 2023.08.x, thanks.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot