Skip to contentSkip to conversation

make-mod-scripts: Provide the correct objcopy to kernel make

PATCH In review Nishanth Menon 4 participants 11 messages Mar 26, 2021 – Apr 1, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 make-mod-scripts: Provide the correct objcopy to kernel make

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

---

NOTE:
1. This is a different problem and is not resolved with
https://lists.openembedded.org/g/openembedded-core/message/149943

2. Though reproduced during a dunfell build, this should probably apply else where as well.

3. I have'nt been able to cleanly bisect things, but my best guess was that this might be a consequence of kernel fixup exposing this.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a5b8ca97fbf8300a5e21c393df25ce6f521e7939

 meta/classes/kernel-arch.bbclass                             | 3 +++
 meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
meta/classes/kernel-arch.bbclass +3 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"
 TOOLCHAIN = "gcc"
 
meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb +1 -1
index 87b7d240f51a..2d73e8093c2e 100644
@@ -25,7 +25,7 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
 do_configure() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
 	for t in prepare scripts_basic scripts; do
-		oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
+		oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}"\
 		-C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
 	done
 }
Signature · 1 line
2.31.0
#1 lore raw

Hi, Bruce,

Would you mind taking a look at this? Have you seen any issues building latest kernel 5.12-rc for aarch64 and seeing host objcopy being used instead of cross compile one? Please let us know. Thanks!

Signature · 1 line
Denys
Quoted 62 lines · Nishanth Menon

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

---

NOTE:
1. This is a different problem and is not resolved with
https://lists.openembedded.org/g/openembedded-core/message/149943

2. Though reproduced during a dunfell build, this should probably apply else where as well.

3. I have'nt been able to cleanly bisect things, but my best guess was that this might be a consequence of kernel fixup exposing this.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a5b8ca97fbf8300a5e21c393df25ce6f521e7939

 meta/classes/kernel-arch.bbclass                             | 3 +++
 meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
meta/classes/kernel-arch.bbclass +3 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"
 TOOLCHAIN = "gcc"
 
meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb +1 -1
index 87b7d240f51a..2d73e8093c2e 100644
@@ -25,7 +25,7 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
 do_configure() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
 	for t in prepare scripts_basic scripts; do
-		oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
+		oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}"\
 		-C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
 	done
 }
Signature · 1 line
2.31.0
#2 lore raw
Quoted 6 lines · Denys Dmytriyenko

Hi, Bruce,

Would you mind taking a look at this? Have you seen any issues building latest kernel 5.12-rc for aarch64 and seeing host objcopy being used instead of cross compile one? Please let us know. Thanks!

I've been able to build and boot all arches on v5.12-rc4, and I did run into something similar on 5.4 kernel's, but it was addressed in -stable backports.

That being said, let me test this over the next couple of days and follow up. I may not have had Ross' changes in my tree when I was sanity testing 5.12.

Bruce

Quoted 52 lines

--
Denys

Quoted 46 lines · Nishanth Menon

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

---

NOTE:
1. This is a different problem and is not resolved with
https://lists.openembedded.org/g/openembedded-core/message/149943

2. Though reproduced during a dunfell build, this should probably apply else where as well.

3. I have'nt been able to cleanly bisect things, but my best guess was that this might be a consequence of kernel fixup exposing this.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a5b8ca97fbf8300a5e21c393df25ce6f521e7939

 meta/classes/kernel-arch.bbclass                             | 3 +++
 meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
meta/classes/kernel-arch.bbclass +3 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"
 TOOLCHAIN = "gcc"
 
Quoted 16 lines
Quoted 14 lines
meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb +1 -2
index 87b7d240f51a..2d73e8093c2e 100644
@@ -25,7 +25,7 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
 do_configure() {
      unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
      for t in prepare scripts_basic scripts; do
-             oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
+             oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}"\
              -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
      done
 }
--
Quoted 2 lines

2.31.0

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#3 lore raw
In reply to Nishanth Menon
Quoted 10 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#
Quoted 25 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

---

NOTE:
1. This is a different problem and is not resolved with
https://lists.openembedded.org/g/openembedded-core/message/149943

2. Though reproduced during a dunfell build, this should probably apply else where as well.

3. I have'nt been able to cleanly bisect things, but my best guess was that this might be a consequence of kernel fixup exposing this.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a5b8ca97fbf8300a5e21c393df25ce6f521e7939

 meta/classes/kernel-arch.bbclass                             | 3 +++
 meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
meta/classes/kernel-arch.bbclass +0 -0
index 07ec242e63bb..3d25fc7ac531 100644

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 6 lines
meta/classes/kernel-arch.bbclass +2 -0
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Are you setting them to some value in your builds ?

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 4 lines
meta/classes/kernel-arch.bbclass +1 -0
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

Are you building arm on arm ? or something else like that ?

Bruce

Quoted 21 lines
meta/classes/kernel-arch.bbclass +0 -0
 TOOLCHAIN = "gcc"
 
meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb +1 -2
index 87b7d240f51a..2d73e8093c2e 100644
@@ -25,7 +25,7 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
 do_configure() {
        unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
        for t in prepare scripts_basic scripts; do
-               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
+               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}"\
                -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
        done
 }
--

2.31.0

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#4 lore raw
Quoted 7 lines · Bruce Ashfield
Quoted 5 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

Quoted 9 lines
Quoted 5 lines

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

Sure. understood. Thanks..

Quoted 4 lines

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

Thanks.

Quoted 14 lines

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

Quoted 5 lines

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#

Hmm... only thing I have done is cross compile - see the pastebins.

Quoted 4 lines
Quoted 4 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

[...]

Quoted 17 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 11 lines
meta/classes/kernel-arch.bbclass +2 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Quoted 1 line

Are you setting them to some value in your builds ?

No, I am not. I decided to maintain consistency of usage from LD and AR. I could'nt think of a usage either, true.

Quoted 11 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 5 lines
meta/classes/kernel-arch.bbclass +1 -0
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

OK this is what is confusing me -> I dont see CROSS_COMPILE being set - which is what I had expected in the first place. other than meta/recipes-kernel/perf/perf.bb, I dont see it set else where -> I was really wondering why all the specific usage, when CROSS_COMPILE would have just done the job.. Then I was guessing maybe the rationale is for some ancient kernel where CROSS_COMPILE is'nt set right?

Quoted 11 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

Quoted 2 lines

Are you building arm on arm ? or something else like that ?

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Signature · 3 lines
Regards,
Nishanth Menon
Key (0xDDB5849D1736249D) / Fingerprint: F8A2 8693 54EB 8232 17A3  1A34 DDB5 849D 1736 249D
#5 lore raw
Quoted 73 lines · Nishanth Menon
Quoted 7 lines · Bruce Ashfield
Quoted 5 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

Quoted 9 lines
Quoted 5 lines

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

Sure. understood. Thanks..

Quoted 4 lines

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

Thanks.

Quoted 14 lines

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

Quoted 5 lines

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#

Hmm... only thing I have done is cross compile - see the pastebins.

Quoted 4 lines
Quoted 4 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

[...]

Quoted 17 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 11 lines
meta/classes/kernel-arch.bbclass +2 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Quoted 4 lines
Quoted 1 line

Are you setting them to some value in your builds ?

No, I am not. I decided to maintain consistency of usage from LD and AR. I could'nt think of a usage either, true.

That's what I figured. I was worried I was missing some sort of usecase. No harm in copying the existing pattern, I didn't introduce it either, so I wanted to make sure there wasn't something going on that I didn't see.

Quoted 12 lines
Quoted 12 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 7 lines
meta/classes/kernel-arch.bbclass +1 -0
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Quoted 9 lines
Quoted 1 line

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

OK this is what is confusing me -> I dont see CROSS_COMPILE being set - which is what I had expected in the first place. other than meta/recipes-kernel/perf/perf.bb, I dont see it set else where -> I was really wondering why all the specific usage, when CROSS_COMPILE would have just done the job.. Then I was guessing maybe the rationale is for some ancient kernel where CROSS_COMPILE is'nt set right?

It could be, we also wanted to tack on the CCACHE stuff, but generally speaking .. it is all kind of redundant with new kernels.

Quoted 7 lines
Quoted 7 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

Quoted 7 lines
Quoted 4 lines

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

That is indeed strange.

What does bitbake -e tell you ?

CROSS_COMPILE should be exported by kernel.bbclass itself, it definitely is in my environment dump.

Quoted 8 lines
Quoted 3 lines

Are you building arm on arm ? or something else like that ?

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Indeed. I've done a full set of tests on all the arches for 5.12-rc+ (as you can see by the lttng/perf/devsrc patches that I've been sending in the past week), so something odd is going on here.

Are the layers you are building public ? if so, I can try with your exact setup and see if I can reproduce the issue.

Bruce

Quoted 5 lines

--
Regards,
Nishanth Menon
Key (0xDDB5849D1736249D) / Fingerprint: F8A2 8693 54EB 8232 17A3 1A34 DDB5 849D 1736 249D

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#6 lore raw

On 11:14-20210329, Bruce Ashfield wrote:
[..]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

That is indeed strange.

What does bitbake -e tell you ?

Not sure if it helps.. but..
https://pastebin.ubuntu.com/p/JJHtNkwSwY/

Quoted 3 lines

CROSS_COMPILE should be exported by kernel.bbclass itself, it definitely is in my environment dump.

Hmm.. ok - there is something going on.. I see the export in bbclass..

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Are you building arm on arm ? or something else like that ?

Quoted 6 lines
Quoted 3 lines

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Indeed. I've done a full set of tests on all the arches for 5.12-rc+ (as you can see by the lttng/perf/devsrc patches that I've been sending

Awesome.

Quoted 1 line

in the past week), so something odd is going on here.

Yes, indeed. I think I will need to look closer at what is going on our end.. As you pointed out CROSS_COMPILE should really be available in the environment.. Rest of specific exports tend to be a never-ending patchup job..

Quoted 3 lines

Are the layers you are building public ? if so, I can try with your exact setup and see if I can reproduce the issue.

unfortunately, no. Denys and I are trying to figure our how to make our private layer public in a manner that others can work with as well.. but not yet.. we are still a few weeks or months away..

Let me try the reverse and try and reproduce your setup on my end and see where the deltas are.. That might be more practical, I think..

Thanks for your time and hints.. I need to do more homework here..

Signature · 3 lines
Regards,
Nishanth Menon
Key (0xDDB5849D1736249D) / Fingerprint: F8A2 8693 54EB 8232 17A3  1A34 DDB5 849D 1736 249D
#7 lore raw
In reply to Bruce Ashfield
Quoted 124 lines · Bruce Ashfield
Quoted 73 lines · Nishanth Menon
Quoted 7 lines · Bruce Ashfield
Quoted 5 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

Quoted 9 lines
Quoted 5 lines

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

Sure. understood. Thanks..

Quoted 4 lines

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

Thanks.

Quoted 14 lines

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

Quoted 5 lines

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#

Hmm... only thing I have done is cross compile - see the pastebins.

Quoted 4 lines
Quoted 4 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

[...]

Quoted 17 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 11 lines
meta/classes/kernel-arch.bbclass +2 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Quoted 4 lines
Quoted 1 line

Are you setting them to some value in your builds ?

No, I am not. I decided to maintain consistency of usage from LD and AR. I could'nt think of a usage either, true.

That's what I figured. I was worried I was missing some sort of usecase. No harm in copying the existing pattern, I didn't introduce it either, so I wanted to make sure there wasn't something going on that I didn't see.

Quoted 12 lines
Quoted 12 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 7 lines
meta/classes/kernel-arch.bbclass +1 -0
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Quoted 9 lines
Quoted 1 line

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

OK this is what is confusing me -> I dont see CROSS_COMPILE being set - which is what I had expected in the first place. other than meta/recipes-kernel/perf/perf.bb, I dont see it set else where -> I was really wondering why all the specific usage, when CROSS_COMPILE would have just done the job.. Then I was guessing maybe the rationale is for some ancient kernel where CROSS_COMPILE is'nt set right?

It could be, we also wanted to tack on the CCACHE stuff, but generally speaking .. it is all kind of redundant with new kernels.

Quoted 7 lines
Quoted 7 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

Quoted 7 lines
Quoted 4 lines

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

Quoted 6 lines

That is indeed strange.

What does bitbake -e tell you ?

CROSS_COMPILE should be exported by kernel.bbclass itself, it definitely is in my environment dump.

Bruce,

Looks like make-mod-scripts does not inherit kernel.bbclass, but only kernel-arch.bbclass and hence doesn't do "export CROSS_COMPILE" that is in kernel.bbclass

Since make-mod-scripts is not used for the kernel, only for out-of-tree modules, you have to either build it directly or e.g. cryptodev-module.

I got it easily reproducible with Poky master by adding these to local.conf:

MACHINE = "qemuarm64" PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"

$ bitbake make-mod-scripts

One easy way to fix it is to pass CROSS_COMPILE explicitly in make-mod-scripts. See the patch I just sent.

Quoted 15 lines
Quoted 6 lines
Quoted 1 line

Are you building arm on arm ? or something else like that ?

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Indeed. I've done a full set of tests on all the arches for 5.12-rc+ (as you can see by the lttng/perf/devsrc patches that I've been sending in the past week), so something odd is going on here.

Are the layers you are building public ? if so, I can try with your exact setup and see if I can reproduce the issue.

Bruce

Signature · 4 lines
Regards,
Denys Dmytriyenko <[email protected]>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186  6D76 4209 0272 9A92 C964
#8 lore raw
Quoted 145 lines · Denys Dmytriyenko
Quoted 124 lines · Bruce Ashfield
Quoted 73 lines · Nishanth Menon
Quoted 7 lines · Bruce Ashfield
Quoted 5 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

Quoted 9 lines
Quoted 5 lines

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

Sure. understood. Thanks..

Quoted 4 lines

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

Thanks.

Quoted 14 lines

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

Quoted 5 lines

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#

Hmm... only thing I have done is cross compile - see the pastebins.

Quoted 4 lines
Quoted 4 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

[...]

Quoted 17 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 11 lines
meta/classes/kernel-arch.bbclass +2 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
 HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
 TARGET_AR_KERNEL_ARCH ?= ""
 HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Quoted 4 lines
Quoted 1 line

Are you setting them to some value in your builds ?

No, I am not. I decided to maintain consistency of usage from LD and AR. I could'nt think of a usage either, true.

That's what I figured. I was worried I was missing some sort of usecase. No harm in copying the existing pattern, I didn't introduce it either, so I wanted to make sure there wasn't something going on that I didn't see.

Quoted 12 lines
Quoted 12 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 7 lines
meta/classes/kernel-arch.bbclass +1 -0
 KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
 KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
 KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Quoted 9 lines
Quoted 1 line

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

OK this is what is confusing me -> I dont see CROSS_COMPILE being set - which is what I had expected in the first place. other than meta/recipes-kernel/perf/perf.bb, I dont see it set else where -> I was really wondering why all the specific usage, when CROSS_COMPILE would have just done the job.. Then I was guessing maybe the rationale is for some ancient kernel where CROSS_COMPILE is'nt set right?

It could be, we also wanted to tack on the CCACHE stuff, but generally speaking .. it is all kind of redundant with new kernels.

Quoted 7 lines
Quoted 7 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

Quoted 7 lines
Quoted 4 lines

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

Quoted 6 lines

That is indeed strange.

What does bitbake -e tell you ?

CROSS_COMPILE should be exported by kernel.bbclass itself, it definitely is in my environment dump.

Bruce,

Looks like make-mod-scripts does not inherit kernel.bbclass, but only kernel-arch.bbclass and hence doesn't do "export CROSS_COMPILE" that is in kernel.bbclass

Since make-mod-scripts is not used for the kernel, only for out-of-tree modules, you have to either build it directly or e.g. cryptodev-module.

I got it easily reproducible with Poky master by adding these to local.conf:

Quoted 7 lines

MACHINE = "qemuarm64" PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"

$ bitbake make-mod-scripts

One easy way to fix it is to pass CROSS_COMPILE explicitly in make-mod-scripts. See the patch I just sent.

I agree that make-mod-scripts doesn't have CROSS_COMPILE set (I just dumped the env to confirm).

It is still odd that I can build for that very same machine without any issues though. There's still something lurking that is filling in the required dependencies and making the build not necessary.

I've built and booted qemuarm64 on the tip of poky master with 5.12+ hundreds of times already, and have never seen that.

BUT .. I switched to my second builder, used my exact same configuration and it did in fact show the issue. I have some local patches on my primary builder, so one of them may have been bolting everything together (not relevant here, but I'll go back and look at them later).

I like this simpler fix, and will follow up directly on the patch you sent.

Bruce

Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 3 lines

Are you building arm on arm ? or something else like that ?

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Indeed. I've done a full set of tests on all the arches for 5.12-rc+ (as you can see by the lttng/perf/devsrc patches that I've been sending in the past week), so something odd is going on here.

Are the layers you are building public ? if so, I can try with your exact setup and see if I can reproduce the issue.

Quoted 7 lines
Quoted 1 line

Bruce

--
Regards,
Denys Dmytriyenko <[email protected]>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#9 lore raw
Quoted 154 lines · Bruce Ashfield
Quoted 145 lines · Denys Dmytriyenko
Quoted 124 lines · Bruce Ashfield
Quoted 73 lines · Nishanth Menon
Quoted 7 lines · Bruce Ashfield
Quoted 5 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

Quoted 9 lines
Quoted 5 lines

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

Sure. understood. Thanks..

Quoted 4 lines

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

Thanks.

Quoted 14 lines

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

Quoted 5 lines

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#

Hmm... only thing I have done is cross compile - see the pastebins.

Quoted 4 lines
Quoted 4 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

[...]

Quoted 17 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 11 lines
meta/classes/kernel-arch.bbclass +2 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
  HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
  TARGET_AR_KERNEL_ARCH ?= ""
  HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Quoted 4 lines
Quoted 1 line

Are you setting them to some value in your builds ?

No, I am not. I decided to maintain consistency of usage from LD and AR. I could'nt think of a usage either, true.

That's what I figured. I was worried I was missing some sort of usecase. No harm in copying the existing pattern, I didn't introduce it either, so I wanted to make sure there wasn't something going on that I didn't see.

Quoted 12 lines
Quoted 12 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 7 lines
meta/classes/kernel-arch.bbclass +1 -0
  KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
  KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
  KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Quoted 9 lines
Quoted 1 line

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

OK this is what is confusing me -> I dont see CROSS_COMPILE being set - which is what I had expected in the first place. other than meta/recipes-kernel/perf/perf.bb, I dont see it set else where -> I was really wondering why all the specific usage, when CROSS_COMPILE would have just done the job.. Then I was guessing maybe the rationale is for some ancient kernel where CROSS_COMPILE is'nt set right?

It could be, we also wanted to tack on the CCACHE stuff, but generally speaking .. it is all kind of redundant with new kernels.

Quoted 7 lines
Quoted 7 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

Quoted 7 lines
Quoted 4 lines

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

Quoted 6 lines

That is indeed strange.

What does bitbake -e tell you ?

CROSS_COMPILE should be exported by kernel.bbclass itself, it definitely is in my environment dump.

Bruce,

Looks like make-mod-scripts does not inherit kernel.bbclass, but only kernel-arch.bbclass and hence doesn't do "export CROSS_COMPILE" that is in kernel.bbclass

Since make-mod-scripts is not used for the kernel, only for out-of-tree modules, you have to either build it directly or e.g. cryptodev-module.

I got it easily reproducible with Poky master by adding these to local.conf:

Quoted 7 lines

MACHINE = "qemuarm64" PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"

$ bitbake make-mod-scripts

One easy way to fix it is to pass CROSS_COMPILE explicitly in make-mod-scripts. See the patch I just sent.

Quoted 6 lines

I agree that make-mod-scripts doesn't have CROSS_COMPILE set (I just dumped the env to confirm).

It is still odd that I can build for that very same machine without any issues though. There's still something lurking that is filling in the required dependencies and making the build not necessary.

it will use objcopy from host at this point instead of what you built with binutils, perhaps a host issue creeps in who knows. cross objcopy is not strictly required but its good to use a captured tool.

Quoted 48 lines

I've built and booted qemuarm64 on the tip of poky master with 5.12+ hundreds of times already, and have never seen that.

BUT .. I switched to my second builder, used my exact same configuration and it did in fact show the issue. I have some local patches on my primary builder, so one of them may have been bolting everything together (not relevant here, but I'll go back and look at them later).

I like this simpler fix, and will follow up directly on the patch you sent.

Bruce

Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 3 lines

Are you building arm on arm ? or something else like that ?

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Indeed. I've done a full set of tests on all the arches for 5.12-rc+ (as you can see by the lttng/perf/devsrc patches that I've been sending in the past week), so something odd is going on here.

Are the layers you are building public ? if so, I can try with your exact setup and see if I can reproduce the issue.

Quoted 7 lines
Quoted 1 line

Bruce

--
Regards,
Denys Dmytriyenko <[email protected]>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#10 lore raw
Quoted 161 lines · Khem Raj
Quoted 157 lines · Bruce Ashfield
Quoted 145 lines · Denys Dmytriyenko
Quoted 124 lines · Bruce Ashfield
Quoted 73 lines · Nishanth Menon
Quoted 7 lines · Bruce Ashfield
Quoted 5 lines · Nishanth Menon via lists.openembedded.org

When cross-compiling with v5.12-rc3, prepare fails[1] build of vdso at the objcopy stage since it seems to be using the local host's objcopy rather than the cross-compile version we want it to use.

Quoted 9 lines
Quoted 5 lines

This can be trivially reproduced in a localbuild of the kernel following the build parameters provided in the process[2]

Lets fix this by passing OBJCOPY over to the kernel.

As I mentioned to Denys, I hadn't seen this. Consulting the maintainers file would have found me as a good addition to the cc'.

Sure. understood. Thanks..

Quoted 4 lines

I'm doing some other work on make-mod-scripts dependencies right now, so I've pulled this in and will re-test against all of the active kernel versions in master.

Thanks.

Quoted 14 lines

But I don't think that make-mod-scripts is the only place we'd need this, if it is indeed happening on the tip of all the supported versions. There are routes to have the prepare steps run, without a make-mod-scripts dependency.

We've already made the mistake of letting the make-mod-scrips and kernel build parameters diverge (see AR=${KERNEL_AR}), so I need to spend a few minutes getting them back in sync.

I was just able to build and boot qemuarm64 on 5.12-rc4, so there's something different about your config than my setup. We need to figure that out. It could be a .config triggering different components to be built, but unlikely given vdso being involved.

Quoted 5 lines

qemuarm64 login: root

root@qemuarm64:~# uname -a
Linux qemuarm64 5.12.0-rc4-yoctodev-standard #1 SMP PREEMPT Mon Mar 22
18:46:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~#

Hmm... only thing I have done is cross compile - see the pastebins.

Quoted 4 lines
Quoted 4 lines

[1] https://pastebin.ubuntu.com/p/pNcQtb93wr/
[2] https://pastebin.ubuntu.com/p/vZGqgh9Sq5/

[...]

Quoted 17 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 65

Quoted 11 lines
meta/classes/kernel-arch.bbclass +2 -0
index 07ec242e63bb..3d25fc7ac531 100644
@@ -60,9 +60,12 @@ TARGET_LD_KERNEL_ARCH ?= ""
  HOST_LD_KERNEL_ARCH ?= "${TARGET_LD_KERNEL_ARCH}"
  TARGET_AR_KERNEL_ARCH ?= ""
  HOST_AR_KERNEL_ARCH ?= "${TARGET_AR_KERNEL_ARCH}"
+TARGET_OBJCOPY_KERNEL_ARCH ?= ""
+HOST_OBJCOPY_KERNEL_ARCH ?= "${TARGET_OBJCOPY_KERNEL_ARCH}"
 

We shouldn't need this TARGET_OBJCOPY_KERNEL_ARCH -> HOST_OBJCOPY_KERNEL_ARCH, I can't think of how objcopy would be using them.

Quoted 4 lines
Quoted 1 line

Are you setting them to some value in your builds ?

No, I am not. I decided to maintain consistency of usage from LD and AR. I could'nt think of a usage either, true.

That's what I figured. I was worried I was missing some sort of usecase. No harm in copying the existing pattern, I didn't introduce it either, so I wanted to make sure there wasn't something going on that I didn't see.

Quoted 12 lines
Quoted 12 lines

meta/classes/kernel-arch.bbclass TARGET_LD_KERNEL_ARCH ?= "" · line 69

Quoted 7 lines
meta/classes/kernel-arch.bbclass +1 -0
  KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd ${DEBUG_PREFIX_MAP} -fdebug-prefix-map=${STAGING_KERNEL_DIR}=${KERNEL_SRC_PATH}"
  KERNEL_LD = "${CCACHE}${HOST_PREFIX}ld.bfd ${HOST_LD_KERNEL_ARCH}"
  KERNEL_AR = "${CCACHE}${HOST_PREFIX}ar ${HOST_AR_KERNEL_ARCH}"
+KERNEL_OBJCOPY = "${CCACHE}${HOST_PREFIX}objcopy ${HOST_OBJCOPY_KERNEL_ARCH}"

The main kernel Makefile already defines, and the standard env should already have both CROSS_COMPILE and OBJCOPY defined to be the target version.

Quoted 9 lines
Quoted 1 line

Makefile:OBJCOPY = $(CROSS_COMPILE)objcopy

OK this is what is confusing me -> I dont see CROSS_COMPILE being set - which is what I had expected in the first place. other than meta/recipes-kernel/perf/perf.bb, I dont see it set else where -> I was really wondering why all the specific usage, when CROSS_COMPILE would have just done the job.. Then I was guessing maybe the rationale is for some ancient kernel where CROSS_COMPILE is'nt set right?

It could be, we also wanted to tack on the CCACHE stuff, but generally speaking .. it is all kind of redundant with new kernels.

Quoted 7 lines
Quoted 7 lines

So we really have to pass this, when fundamentally it is the same definition as what the defaults give us.

What does that resolve to in your builds ? In mine, when I dump the objcopy value from within the kernel build, I get:

Quoted 7 lines
Quoted 4 lines

| /opt/poky/build/tmp/work-shared/qemuarm64/kernel-source/Makefile:443:
*** objcopy: aarch64-poky-linux-objcopy. Stop.

Which should be doing the job.

x86's objcopy - which is why I was trying to figure out what the heck went on..

Quoted 6 lines

That is indeed strange.

What does bitbake -e tell you ?

CROSS_COMPILE should be exported by kernel.bbclass itself, it definitely is in my environment dump.

Bruce,

Looks like make-mod-scripts does not inherit kernel.bbclass, but only kernel-arch.bbclass and hence doesn't do "export CROSS_COMPILE" that is in kernel.bbclass

Since make-mod-scripts is not used for the kernel, only for out-of-tree modules, you have to either build it directly or e.g. cryptodev-module.

I got it easily reproducible with Poky master by adding these to local.conf:

Quoted 7 lines

MACHINE = "qemuarm64" PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"

$ bitbake make-mod-scripts

One easy way to fix it is to pass CROSS_COMPILE explicitly in make-mod-scripts. See the patch I just sent.

I agree that make-mod-scripts doesn't have CROSS_COMPILE set (I just dumped the env to confirm).

Quoted 7 lines
Quoted 3 lines

It is still odd that I can build for that very same machine without any issues though. There's still something lurking that is filling in the required dependencies and making the build not necessary.

it will use objcopy from host at this point instead of what you built with binutils, perhaps a host issue creeps in who knows. cross objcopy is not strictly required but its good to use a captured tool.

No it won't.

The host tool is a completely different arch and blows up. I'm saying that with some of the local (to be submitted in the future) patches I have, the cross objtool was picked up in that same dependency chain.

Bruce

Quoted 49 lines
Quoted 49 lines

I've built and booted qemuarm64 on the tip of poky master with 5.12+ hundreds of times already, and have never seen that.

BUT .. I switched to my second builder, used my exact same configuration and it did in fact show the issue. I have some local patches on my primary builder, so one of them may have been bolting everything together (not relevant here, but I'll go back and look at them later).

I like this simpler fix, and will follow up directly on the patch you sent.

Bruce

Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 3 lines

Are you building arm on arm ? or something else like that ?

Nothing fancy.. x86 PC cross compiling for arm. honestly, 5.11 build went fine. What makes me wonder is how does it even build for you? how does OBJCOPY variable even point to aarch64-poky-linux-objcopy

Indeed. I've done a full set of tests on all the arches for 5.12-rc+ (as you can see by the lttng/perf/devsrc patches that I've been sending in the past week), so something odd is going on here.

Are the layers you are building public ? if so, I can try with your exact setup and see if I can reproduce the issue.

Quoted 7 lines
Quoted 1 line

Bruce

--
Regards,
Denys Dmytriyenko <[email protected]>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II