classes/waf: Fix builds when B != S

PATCH Superseded Joshua Watt 4 participants 11 messages Nov 30, 2018 – Dec 3, 2018

Share & tools

Joshua Watt

author

2018-11-30 21:55 UTC #0 lore raw

PATCH 1/1 classes/waf: Fix builds when B != S

Waf requires that the current working directory be ${S} (the location of the wscript) when building. Most of the time, this was true only because B defaults to S. However, anything that changed that behavior (notably, using externalsrc) would break the recipe. Remedy this by explicitly changing cwd to ${S} when running waf commands. As a happy side effect, B can be set up for "out of tree" builds to keep the source directory clean.

Signed-off-by Joshua Watt <[email protected]>

---
 meta/classes/waf.bbclass | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

meta/classes/waf.bbclass +5 -3

index 19e93761b39..9cb5b68f449 100644
@@ -1,6 +1,8 @@
 # avoids build breaks when using no-static-libs.inc
 DISABLE_STATIC = ""
 
+B = "${WORKDIR}/${BPN}-${PV}"
+
 EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
 python waf_preconfigure() {

@@ -22,16 +24,16 @@ python waf_preconfigure() {
 do_configure[prefuncs] += "waf_preconfigure"
 
 waf_do_configure() {
-	${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+	(cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
 }
 
 do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
 waf_do_compile()  {
-	${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+	(cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
 }
 
 waf_do_install() {
-	${S}/waf install --destdir=${D}
+	(cd ${S} && ./waf install --destdir=${D})
 }
 
 EXPORT_FUNCTIONS do_configure do_compile do_install

Signature · 1 line

2.19.1

Burton, Ross

2018-11-30 23:00 UTC #1 lore raw

Quoted 1 line · Joshua Watt

+B = "${WORKDIR}/${BPN}-${PV}"

That's the default, so redundant.

Does waf support out-of-tree builds sufficiently well that we can default - or even mandate - out of tree? i.e. follow meson and cmake, and just set B=$WORKDIR/build in the class.

Ross

Joshua Watt

author

2018-12-01 00:01 UTC #2 lore raw

Quoted 9 lines · On Fri, Nov 30, 2018, 5:00 PM Burton, Ross <[email protected] wrote:

Quoted 1 line · Joshua Watt
+B = "${WORKDIR}/${BPN}-${PV}"
That's the default, so redundant.
Does waf support out-of-tree builds sufficiently well that we can default - or even mandate - out of tree? i.e. follow meson and cmake, and just set B=$WORKDIR/build in the class.

Yes. This patch already does that by adding "-o ${B}" in do_configure. I will change it to ${WORKDIR}/build. Out if curiosity, why is that preferred over ${WORKDIR}/${BPN}-${PV} ?

Quoted 3 lines

Ross

Joshua Watt

author

2018-12-01 00:31 UTC #3 lore raw

Quoted 9 lines · On Fri, Nov 30, 2018, 6:01 PM Joshua Watt <[email protected] wrote:

Quoted 5 lines · On Fri, Nov 30, 2018, 5:00 PM Burton, Ross <[email protected] wrote:
Quoted 1 line · Joshua Watt
+B = "${WORKDIR}/${BPN}-${PV}"
That's the default, so redundant.

Quoted 9 lines

Quoted 4 lines
Does waf support out-of-tree builds sufficiently well that we can default - or even mandate - out of tree? i.e. follow meson and cmake, and just set B=$WORKDIR/build in the class.
Yes. This patch already does that by adding "-o ${B}" in do_configure. I will change it to ${WORKDIR}/build. Out if curiosity, why is that preferred over ${WORKDIR}/${BPN}-${PV} ?

Hah, right. {WORKDIR}/${BPN}-${PV} is also the default value for ${S}, so setting to ${WORKDIR}/build ensures they are different, which we want/need for waf.

Quoted 3 lines

Quoted 2 lines
Ross

Joshua Watt

author

2018-12-01 03:01 UTC #4 lore raw

PATCH v2 1/1 classes/waf: Fix builds when B != S

In reply to Joshua Watt

Signed-off-by Joshua Watt <[email protected]>

---
 meta/classes/waf.bbclass | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

meta/classes/waf.bbclass +5 -3

index 19e93761b39..8e6d754c299 100644
@@ -1,6 +1,8 @@
 # avoids build breaks when using no-static-libs.inc
 DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
 EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
 python waf_preconfigure() {

@@ -22,16 +24,16 @@ python waf_preconfigure() {
 do_configure[prefuncs] += "waf_preconfigure"
 
 waf_do_configure() {
-	${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+	(cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
 }
 
 do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
 waf_do_compile()  {
-	${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+	(cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
 }
 
 waf_do_install() {
-	${S}/waf install --destdir=${D}
+	(cd ${S} && ./waf install --destdir=${D})
 }
 
 EXPORT_FUNCTIONS do_configure do_compile do_install

Signature · 1 line

2.19.1

Khem Raj

2018-12-02 00:09 UTC #5 lore raw

Hi Joshua

There is a build failure for a2jmidid see

http://errors.yoctoproject.org/Errors/Details/202833/

Seem to be related can you validate

Quoted 47 lines · Joshua Watt

Waf requires that the current working directory be ${S} (the location of the wscript) when building. Most of the time, this was true only because B defaults to S. However, anything that changed that behavior (notably, using externalsrc) would break the recipe. Remedy this by explicitly changing cwd to ${S} when running waf commands. As a happy side effect, B can be set up for "out of tree" builds to keep the source directory clean.
Signed-off-by Joshua Watt <[email protected]>
---
  meta/classes/waf.bbclass | 8 +++++---
  1 file changed, 5 insertions(+), 3 deletions(-)
meta/classes/waf.bbclass +5 -3
index 19e93761b39..8e6d754c299 100644
@@ -1,6 +1,8 @@
  # avoids build breaks when using no-static-libs.inc
  DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
  EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
  python waf_preconfigure() {
@@ -22,16 +24,16 @@ python waf_preconfigure() {
  do_configure[prefuncs] += "waf_preconfigure"
 
  waf_do_configure() {
-	${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+	(cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
  }
 
  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
  waf_do_compile()  {
-	${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+	(cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
  }
 
  waf_do_install() {
-	${S}/waf install --destdir=${D}
+	(cd ${S} && ./waf install --destdir=${D})
  }
 
  EXPORT_FUNCTIONS do_configure do_compile do_install

Khem Raj

2018-12-03 20:14 UTC #6 lore raw

Quoted 17 lines · Khem Raj

Hi Joshua
There is a build failure for a2jmidid see
http://errors.yoctoproject.org/Errors/Details/202833/
Seem to be related can you validate
Quoted 7 lines · Joshua Watt
Waf requires that the current working directory be ${S} (the location of the wscript) when building. Most of the time, this was true only because B defaults to S. However, anything that changed that behavior (notably, using externalsrc) would break the recipe. Remedy this by explicitly changing cwd to ${S} when running waf commands. As a happy side effect, B can be set up for "out of tree" builds to keep the source directory clean.

I have sent a workaround for a2jmidid to ml which means we can go ahead with this change in OE-Core

Quoted 40 lines

Signed-off-by Joshua Watt <[email protected]>

---
  meta/classes/waf.bbclass | 8 +++++---
  1 file changed, 5 insertions(+), 3 deletions(-)

meta/classes/waf.bbclass +5 -3

index 19e93761b39..8e6d754c299 100644
@@ -1,6 +1,8 @@
  # avoids build breaks when using no-static-libs.inc
  DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
  EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
  python waf_preconfigure() {

@@ -22,16 +24,16 @@ python waf_preconfigure() {
  do_configure[prefuncs] += "waf_preconfigure"
 
  waf_do_configure() {
-     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
  }
 
  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
  waf_do_compile()  {
-     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
  }
 
  waf_do_install() {
-     ${S}/waf install --destdir=${D}
+     (cd ${S} && ./waf install --destdir=${D})
  }
 
  EXPORT_FUNCTIONS do_configure do_compile do_install

Andreas Müller

2018-12-03 21:50 UTC #7 lore raw

Quoted 56 lines · Khem Raj

Quoted 17 lines · Khem Raj
Hi Joshua
There is a build failure for a2jmidid see
http://errors.yoctoproject.org/Errors/Details/202833/
Seem to be related can you validate
Quoted 7 lines · Joshua Watt
Waf requires that the current working directory be ${S} (the location of the wscript) when building. Most of the time, this was true only because B defaults to S. However, anything that changed that behavior (notably, using externalsrc) would break the recipe. Remedy this by explicitly changing cwd to ${S} when running waf commands. As a happy side effect, B can be set up for "out of tree" builds to keep the source directory clean.
I have sent a workaround for a2jmidid to ml which means we can go ahead with this change in OE-Core
Quoted 33 lines
Quoted 33 lines
Signed-off-by Joshua Watt <[email protected]>
---
  meta/classes/waf.bbclass | 8 +++++---
  1 file changed, 5 insertions(+), 3 deletions(-)
meta/classes/waf.bbclass +4 -2
index 19e93761b39..8e6d754c299 100644
@@ -1,6 +1,8 @@
  # avoids build breaks when using no-static-libs.inc
  DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
  EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
  python waf_preconfigure() {
@@ -22,16 +24,16 @@ python waf_preconfigure() {
  do_configure[prefuncs] += "waf_preconfigure"
 
  waf_do_configure() {
-     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
  }
 
  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
  waf_do_compile()  {
-     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
  }
 

Quoted 8 lines

Quoted 7 lines
meta/classes/waf.bbclass python waf_preconfigure() { · line 39
Quoted 7 lines
meta/classes/waf.bbclass +1 -1
  waf_do_install() {
-     ${S}/waf install --destdir=${D}
+     (cd ${S} && ./waf install --destdir=${D})
  }
 
  EXPORT_FUNCTIONS do_configure do_compile do_install
--

Hmm - the a2jmidid shows what's going to happen: All packets shipping older waf need workaround hacks. This should not be the way to go because then we can remove waf.bbclass entirely.

Andreas

Khem Raj

2018-12-03 22:04 UTC #8 lore raw

Quoted 63 lines · Andreas Müller

Quoted 61 lines · Khem Raj
Quoted 17 lines · Khem Raj
Hi Joshua
There is a build failure for a2jmidid see
http://errors.yoctoproject.org/Errors/Details/202833/
Seem to be related can you validate
Quoted 7 lines · Joshua Watt
Waf requires that the current working directory be ${S} (the location of the wscript) when building. Most of the time, this was true only because B defaults to S. However, anything that changed that behavior (notably, using externalsrc) would break the recipe. Remedy this by explicitly changing cwd to ${S} when running waf commands. As a happy side effect, B can be set up for "out of tree" builds to keep the source directory clean.
I have sent a workaround for a2jmidid to ml which means we can go ahead with this change in OE-Core
Quoted 38 lines
Quoted 38 lines
Signed-off-by Joshua Watt <[email protected]>
---
  meta/classes/waf.bbclass | 8 +++++---
  1 file changed, 5 insertions(+), 3 deletions(-)
meta/classes/waf.bbclass +5 -3
index 19e93761b39..8e6d754c299 100644
@@ -1,6 +1,8 @@
  # avoids build breaks when using no-static-libs.inc
  DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
  EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
  python waf_preconfigure() {
@@ -22,16 +24,16 @@ python waf_preconfigure() {
  do_configure[prefuncs] += "waf_preconfigure"
 
  waf_do_configure() {
-     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
  }
 
  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
  waf_do_compile()  {
-     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
  }
 
  waf_do_install() {
-     ${S}/waf install --destdir=${D}
+     (cd ${S} && ./waf install --destdir=${D})
  }
 

Quoted 7 lines

Quoted 3 lines
Quoted 2 lines
meta/classes/waf.bbclass python waf_preconfigure() { · line 39
Quoted 2 lines
meta/classes/waf.bbclass +0 -0
  EXPORT_FUNCTIONS do_configure do_compile do_install
--
Hmm - the a2jmidid shows what's going to happen: All packets shipping older waf need workaround hacks. This should not be the way to go because then we can remove waf.bbclass entirely.

FWIW out of all meta-openembedded layers only 1 recipe was having this issue, while the issue you mention is real, we might have to weigh in if we should not do this just for a handful of recipes. My suggestion would be to change the recipes if its not a cascade

Quoted 1 line

Andreas

Joshua Watt

author

2018-12-03 22:22 UTC #9 lore raw

In reply to Andreas Müller

Quoted 70 lines · Andreas Müller

Quoted 69 lines · Khem Raj
Quoted 22 lines · Khem Raj
Hi Joshua
There is a build failure for a2jmidid see
http://errors.yoctoproject.org/Errors/Details/202833/
Seem to be related can you validate
Quoted 13 lines · Joshua Watt
Waf requires that the current working directory be ${S} (the location of
the wscript) when building. Most of the time, this was true only because
B defaults to S. However, anything that changed that behavior (notably,
using externalsrc) would break the recipe. Remedy this by explicitly
changing cwd to ${S} when running waf commands. As a happy side effect,
B can be set up for "out of tree" builds to keep the source directory
clean.
I have sent a workaround for a2jmidid to ml which means we can go ahead with this change in OE-Core
Quoted 42 lines
Quoted 42 lines
Signed-off-by Joshua Watt <[email protected]>
---
  meta/classes/waf.bbclass | 8 +++++---
  1 file changed, 5 insertions(+), 3 deletions(-)
meta/classes/waf.bbclass +5 -3
index 19e93761b39..8e6d754c299 100644
@@ -1,6 +1,8 @@
  # avoids build breaks when using no-static-libs.inc
  DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
  EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
  python waf_preconfigure() {
@@ -22,16 +24,16 @@ python waf_preconfigure() {
  do_configure[prefuncs] += "waf_preconfigure"
 
  waf_do_configure() {
-     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF}
 ${EXTRA_OECONF}
+     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix}
 ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
  }
 
  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
  waf_do_compile()  {
-     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-
 j%d', limit=64)}
+     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argumen
 t(d, '-j%d', limit=64)})
  }
 
  waf_do_install() {
-     ${S}/waf install --destdir=${D}
+     (cd ${S} && ./waf install --destdir=${D})
  }

Quoted 6 lines

Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
EXPORT_FUNCTIONS do_configure do_compile do_install
--
Hmm - the a2jmidid shows what's going to happen: All packets shipping older waf need workaround hacks. This should not be the way to go because then we can remove waf.bbclass entirely.

Yes, that might actually be the most "waf" answer to that problem... for better or worse waf is designed to be fully self contained, so having a "standard" class that builds it is somewhat problematic. Waf itself could change at any time and render the canned configuration useless. This doesn't really affect any of the projects using waf because they usually just download (or build) the latest when they start development and don't upgrade (or do so infrequently).

The other option that I've though about (not that I really like it) is to have some sort of WAF_VERSION variable that waf.bbclass keys off of to detect what support is available. However, even this isn't fool proof because of add-in modules that you can choose to build into your project's version of waf... these can basically do whatever you want including adding new command line arguments, and adding or modifying existing behaviors. I'm not in favor of going down the road of making variables like WAF_HAS_GNU_DIRS_PLUGIN that each project has to configure.

OTOH, as long as the projects we are building with waf currently use a new enough version (I know 1.7 works and 1.4 is broken), our current bbclass works and maybe that's "good enough" for now?

I'm open to suggestions.

Quoted 2 lines

Andreas

Signature · 1 line

Joshua Watt <[email protected]>

Andreas Müller

2018-12-03 23:18 UTC #10 lore raw

Quoted 22 lines · Joshua Watt

Quoted 3 lines
Hmm - the a2jmidid shows what's going to happen: All packets shipping older waf need workaround hacks. This should not be the way to go because then we can remove waf.bbclass entirely.
Yes, that might actually be the most "waf" answer to that problem... for better or worse waf is designed to be fully self contained, so having a "standard" class that builds it is somewhat problematic. Waf itself could change at any time and render the canned configuration useless. This doesn't really affect any of the projects using waf because they usually just download (or build) the latest when they start development and don't upgrade (or do so infrequently).
The other option that I've though about (not that I really like it) is to have some sort of WAF_VERSION variable that waf.bbclass keys off of to detect what support is available. However, even this isn't fool proof because of add-in modules that you can choose to build into your project's version of waf... these can basically do whatever you want including adding new command line arguments, and adding or modifying existing behaviors. I'm not in favor of going down the road of making variables like WAF_HAS_GNU_DIRS_PLUGIN that each project has to configure.

Quoted 5 lines

OTOH, as long as the projects we are building with waf currently use a new enough version (I know 1.7 works and 1.4 is broken), our current bbclass works and maybe that's "good enough" for now?
I'm open to suggestions.

Have meta-qt5-extra with many recipes on old sources which potentially fail for this change. Will build with your's and Khem's patch and fix affected recipes. Think version magic is not necessary.

Andreas