Skip to contentSkip to conversation

kvm: Fix Nix build failure by moving -fno-stack-protector to CFLAGS

PATCH In review Richard Palethorpe via ltp 5 participants 7 messages Sep 19, 2023 – Sep 22, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kvm: Fix Nix build failure by moving -fno-stack-protector to CFLAGS

Nix (and by extension NixOS) uses a binutils wrapper of some sort to allow switching between toolchains. This seems to be causing a problem for the KVM tests although it may not be unique to Nix.

gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -DCOMPILE_PAYLOAD -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx\
-mno-sse -fno-pie -c -o lib_guest.o lib_guest.c
as -c -o bootstrap_x86_64.o bootstrap_x86_64.S
gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -DCOMPILE_PAYLOAD -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx\
-mno-sse -fno-pie -c -o lib_x86.o lib_x86.c
gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -I../../../include -I../../../include -I../../../include/old/ -DCOMPILE_PAYLOAD -ff\
reestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx -mno-sse -fno-pie -nostdlib -Wl,--build-id=none -fno-stack-protector -z noexe\
cstack -no-pie -Wl,-T/home/rich/kernel/ltp/testcases/kernel/kvm/linker/x86_64.lds -o kvm_pagefault01-payload.elf kvm_pagefault01.c lib_guest\
.o bootstrap_x86_64.o lib_x86.o
/nix/store/inq79dwl8sz1ygmfbgsmg77i5cwmdjpz-binutils-2.40/bin/ld: lib_x86.o: in function `kvm_init_guest_vmcb':
lib_x86.c:(.text+0x7b3): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
make: *** [Makefile:58: kvm_pagefault01-payload.o] Error 1

__stack_chk_fail is added to lib_x86.o which means that -fstack-protector somehow gets added to the compilation of lib_x86.o.

Swapping -fno-stack-protector from the LD flags to the C flags ensures it is disabled when compiling lib_x86.o.

---
 testcases/kernel/kvm/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
testcases/kernel/kvm/Makefile +2 -2
index d9eb10728..ce4a5ede2 100644
@@ -8,8 +8,8 @@ include $(top_srcdir)/include/mk/testcases.mk
 ASFLAGS =
 CPPFLAGS += -I$(abs_srcdir)/include
 GUEST_CPPFLAGS = $(CPPFLAGS) -DCOMPILE_PAYLOAD
-GUEST_CFLAGS = -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx -mno-sse
-GUEST_LDFLAGS = -nostdlib -Wl,--build-id=none -fno-stack-protector -z noexecstack
+GUEST_CFLAGS = -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -fno-stack-protector -mno-mmx -mno-sse
+GUEST_LDFLAGS = -nostdlib -Wl,--build-id=none -z noexecstack
 GUEST_LDLIBS =
 KVM_LD ?= $(LD)
 
Signature · 5 lines
2.42.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/1
#1 lore raw

Hi,
pvorel should have a look whether it doesn't break the other systems where -fno-stack-protector is needed, but otherwise:

Quoted 44 lines · Richard Palethorpe

Nix (and by extension NixOS) uses a binutils wrapper of some sort to allow switching between toolchains. This seems to be causing a problem for the KVM tests although it may not be unique to Nix.

gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -DCOMPILE_PAYLOAD -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx\
-mno-sse -fno-pie -c -o lib_guest.o lib_guest.c
as -c -o bootstrap_x86_64.o bootstrap_x86_64.S
gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -DCOMPILE_PAYLOAD -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx\
-mno-sse -fno-pie -c -o lib_x86.o lib_x86.c
gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -I../../../include -I../../../include -I../../../include/old/ -DCOMPILE_PAYLOAD -ff\
reestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx -mno-sse -fno-pie -nostdlib -Wl,--build-id=none -fno-stack-protector -z noexe\
cstack -no-pie -Wl,-T/home/rich/kernel/ltp/testcases/kernel/kvm/linker/x86_64.lds -o kvm_pagefault01-payload.elf kvm_pagefault01.c lib_guest\
.o bootstrap_x86_64.o lib_x86.o
/nix/store/inq79dwl8sz1ygmfbgsmg77i5cwmdjpz-binutils-2.40/bin/ld: lib_x86.o: in function `kvm_init_guest_vmcb':
lib_x86.c:(.text+0x7b3): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
make: *** [Makefile:58: kvm_pagefault01-payload.o] Error 1

__stack_chk_fail is added to lib_x86.o which means that -fstack-protector somehow gets added to the compilation of lib_x86.o.

Swapping -fno-stack-protector from the LD flags to the C flags ensures it is disabled when compiling lib_x86.o.

---
  testcases/kernel/kvm/Makefile | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
testcases/kernel/kvm/Makefile +2 -2
index d9eb10728..ce4a5ede2 100644
@@ -8,8 +8,8 @@ include $(top_srcdir)/include/mk/testcases.mk
  ASFLAGS =
  CPPFLAGS += -I$(abs_srcdir)/include
  GUEST_CPPFLAGS = $(CPPFLAGS) -DCOMPILE_PAYLOAD
-GUEST_CFLAGS = -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx -mno-sse
-GUEST_LDFLAGS = -nostdlib -Wl,--build-id=none -fno-stack-protector -z noexecstack
+GUEST_CFLAGS = -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -fno-stack-protector -mno-mmx -mno-sse
+GUEST_LDFLAGS = -nostdlib -Wl,--build-id=none -z noexecstack
  GUEST_LDLIBS =
  KVM_LD ?= $(LD)
 
Signature · 11 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw

Hello,

Quoted 5 lines · Martin Doucha

Hi,
pvorel should have a look whether it doesn't break the other systems where -fno-stack-protector is needed, but otherwise:

Compilation appears to be fine:
https://github.com/richiejp/ltp/actions/runs/6232857602/job/16916865952

The tests just TCONF on my hardware, but I guess the risk of breaking the test at runtime is low?

Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw

Hi Richie,

Quoted 8 lines

Hello,

Quoted 3 lines · Martin Doucha

Hi,
pvorel should have a look whether it doesn't break the other systems where -fno-stack-protector is needed, but otherwise:

Quoted 7 lines
Quoted 1 line

Compilation appears to be fine:
https://github.com/richiejp/ltp/actions/runs/6232857602/job/16916865952

The tests just TCONF on my hardware, but I guess the risk of breaking the test at runtime is low?

I'll test tonight Buildroot toolchains, which usually find errors which aren't spot by our github CI.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Acked-by
#4 lore raw
In reply to Martin Doucha

Hi all,

[I Cc Fabrice and Buildroot ML]

Quoted 4 lines · Martin Doucha

Hi,
pvorel should have a look whether it doesn't break the other systems where -fno-stack-protector is needed, but otherwise:

Yesterday night I tested the current LTP master with ./utils/test-pkg. Some Bootlin toolchains keep failing (bootlin-x86-64-musl, bootlin-x86-64-glibc):

CC testcases/kernel/kvm/lib_x86.o gbr-test-pkg/bootlin-x86-64-musl/host/opt/ext-toolchain/bin/../lib/gcc/x86_64-buildroot-linux-musl/11.3.0/../../../../x86_64-buildroot-linux-musl/bin/ld:
lib_x86.o: in function `kvm_init_guest_vmcb':
lib_x86.c:(.text+0x7c8): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status

Problems are only on x86_64, other archs (aarch64, riscv64, mipsel, amv7r) are OK.

NOTE: In Buildroot LTP is currently disabled when musl is used due 5.11 headers requirement for musl toolchain [1], but I temporarily remove this requirement to get more testing. That's why there are also musl results.

It looks like this patch actually fixes it, thus:

There are even more kvm test related results [2], but I suspect they are more related to the Buildroot toolchain than LTP itself:

CC testcases/kernel/kvm/lib_guest.o
bootstrap_x86_64.S: Assembler messages:
bootstrap_x86_64.S:16: Error: unknown pseudo-op: .code16' bootstrap_x86_64.S:19: Error: unknown mnemonic cli' -- cli' bootstrap_x86_64.S:21: Error: unknown mnemonic lgdt' -- lgdt kvm_gdt32_desc' bootstrap_x86_64.S:23: Error: operand 1 must be an integer register -- mov $0x11,%eax'
bootstrap_x86_64.S:24: Error: operand 1 must be an integer register -- `mov %eax,%cr0'
bootstrap_x86_64.S:26: Error: unknown mnemonic jmp' -- jmp $3*8,$protected_mode_entry'
bootstrap_x86_64.S:28: Error: unknown pseudo-op: `.code32'

Kind regards,
Petr

[1] https://git.busybox.net/buildroot/commit/?id=5e1c238b7202c3b04d9eab94e429c5580c77444b
[2] http://autobuild.buildroot.net/results/6fe953a989a83da67e7ec228ab1668998928ad11/build-end.log

Quoted 61 lines
Quoted 34 lines · Richard Palethorpe

Nix (and by extension NixOS) uses a binutils wrapper of some sort to allow switching between toolchains. This seems to be causing a problem for the KVM tests although it may not be unique to Nix.

gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -DCOMPILE_PAYLOAD -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx\
-mno-sse -fno-pie -c -o lib_guest.o lib_guest.c
as -c -o bootstrap_x86_64.o bootstrap_x86_64.S
gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -DCOMPILE_PAYLOAD -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx\
-mno-sse -fno-pie -c -o lib_x86.o lib_x86.c
gcc -I/home/rich/kernel/ltp/testcases/kernel/kvm/include -I../../../include -I../../../include -I../../../include/old/ -DCOMPILE_PAYLOAD -ff\
reestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx -mno-sse -fno-pie -nostdlib -Wl,--build-id=none -fno-stack-protector -z noexe\
cstack -no-pie -Wl,-T/home/rich/kernel/ltp/testcases/kernel/kvm/linker/x86_64.lds -o kvm_pagefault01-payload.elf kvm_pagefault01.c lib_guest\
.o bootstrap_x86_64.o lib_x86.o
/nix/store/inq79dwl8sz1ygmfbgsmg77i5cwmdjpz-binutils-2.40/bin/ld: lib_x86.o: in function `kvm_init_guest_vmcb':
lib_x86.c:(.text+0x7b3): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
make: *** [Makefile:58: kvm_pagefault01-payload.o] Error 1

__stack_chk_fail is added to lib_x86.o which means that -fstack-protector somehow gets added to the compilation of lib_x86.o.

Swapping -fno-stack-protector from the LD flags to the C flags ensures it is disabled when compiling lib_x86.o.

---
  testcases/kernel/kvm/Makefile | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
testcases/kernel/kvm/Makefile +0 -0
index d9eb10728..ce4a5ede2 100644

testcases/kernel/kvm/Makefile include $(top_srcdir)/include/mk/testcases.mk · line 15

Quoted 10 lines
testcases/kernel/kvm/Makefile +2 -2
@@ -8,8 +8,8 @@ include $(top_srcdir)/include/mk/testcases.mk
  ASFLAGS =
  CPPFLAGS += -I$(abs_srcdir)/include
  GUEST_CPPFLAGS = $(CPPFLAGS) -DCOMPILE_PAYLOAD
-GUEST_CFLAGS = -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -mno-mmx -mno-sse
-GUEST_LDFLAGS = -nostdlib -Wl,--build-id=none -fno-stack-protector -z noexecstack
+GUEST_CFLAGS = -ffreestanding -O2 -Wall -fno-asynchronous-unwind-tables -fno-stack-protector -mno-mmx -mno-sse
+GUEST_LDFLAGS = -nostdlib -Wl,--build-id=none -z noexecstack
  GUEST_LDLIBS =
  KVM_LD ?= $(LD)
 

--
Martin Doucha [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Hello,

Quoted 13 lines · Petr Vorel

Hi all,

[I Cc Fabrice and Buildroot ML]

Quoted 4 lines · Martin Doucha

Hi,
pvorel should have a look whether it doesn't break the other systems where -fno-stack-protector is needed, but otherwise:

Yesterday night I tested the current LTP master with ./utils/test-pkg. Some Bootlin toolchains keep failing (bootlin-x86-64-musl, bootlin-x86-64-glibc):

Quoted 8 lines

CC testcases/kernel/kvm/lib_x86.o gbr-test-pkg/bootlin-x86-64-musl/host/opt/ext-toolchain/bin/../lib/gcc/x86_64-buildroot-linux-musl/11.3.0/../../../../x86_64-buildroot-linux-musl/bin/ld:
lib_x86.o: in function `kvm_init_guest_vmcb':
lib_x86.c:(.text+0x7c8): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status

Problems are only on x86_64, other archs (aarch64, riscv64, mipsel, amv7r) are OK.

I think only x86_64 is supported.

Quoted 7 lines

NOTE: In Buildroot LTP is currently disabled when musl is used due 5.11 headers requirement for musl toolchain [1], but I temporarily remove this requirement to get more testing. That's why there are also musl results.

It looks like this patch actually fixes it, thus:

Right, it's not just Nix then. I will merge it.

Quoted 4 lines

There are even more kvm test related results [2], but I suspect they are more related to the Buildroot toolchain than LTP itself:

Quoted 12 lines

CC testcases/kernel/kvm/lib_guest.o
bootstrap_x86_64.S: Assembler messages:
bootstrap_x86_64.S:16: Error: unknown pseudo-op: .code16' bootstrap_x86_64.S:19: Error: unknown mnemonic cli' -- cli' bootstrap_x86_64.S:21: Error: unknown mnemonic lgdt' -- lgdt kvm_gdt32_desc' bootstrap_x86_64.S:23: Error: operand 1 must be an integer register -- mov $0x11,%eax'
bootstrap_x86_64.S:24: Error: operand 1 must be an integer register -- `mov %eax,%cr0'
bootstrap_x86_64.S:26: Error: unknown mnemonic jmp' -- jmp $3*8,$protected_mode_entry'
bootstrap_x86_64.S:28: Error: unknown pseudo-op: `.code32'

For me I would at the least we need V=1 enabled or the names and versions of the tools being used to guess what is wrong here.

mov $0x11,%eax looks like a really simple op to put 11 in EAX using AT&T/GAS syntax. If that goes wrong then possibly the wrong syntax is expected by the assembler or else a previous line is confusing it.

Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
Quoted 23 lines

Hello,

Quoted 18 lines · Petr Vorel

Hi all,

[I Cc Fabrice and Buildroot ML]

Quoted 3 lines · Martin Doucha

Hi,
pvorel should have a look whether it doesn't break the other systems where -fno-stack-protector is needed, but otherwise:

Yesterday night I tested the current LTP master with ./utils/test-pkg. Some Bootlin toolchains keep failing (bootlin-x86-64-musl, bootlin-x86-64-glibc):

CC testcases/kernel/kvm/lib_x86.o gbr-test-pkg/bootlin-x86-64-musl/host/opt/ext-toolchain/bin/../lib/gcc/x86_64-buildroot-linux-musl/11.3.0/../../../../x86_64-buildroot-linux-musl/bin/ld:
lib_x86.o: in function `kvm_init_guest_vmcb':
lib_x86.c:(.text+0x7c8): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status

Quoted 4 lines
Quoted 2 lines

Problems are only on x86_64, other archs (aarch64, riscv64, mipsel, amv7r) are OK.

I think only x86_64 is supported.

Yes (+ i386). Thanks, I forgot on it.

Quoted 8 lines
Quoted 6 lines

NOTE: In Buildroot LTP is currently disabled when musl is used due 5.11 headers requirement for musl toolchain [1], but I temporarily remove this requirement to get more testing. That's why there are also musl results.

It looks like this patch actually fixes it, thus:

Right, it's not just Nix then. I will merge it.

+1

Quoted 16 lines
Quoted 15 lines

There are even more kvm test related results [2], but I suspect they are more related to the Buildroot toolchain than LTP itself:

CC testcases/kernel/kvm/lib_guest.o
bootstrap_x86_64.S: Assembler messages:
bootstrap_x86_64.S:16: Error: unknown pseudo-op: .code16' bootstrap_x86_64.S:19: Error: unknown mnemonic cli' -- cli' bootstrap_x86_64.S:21: Error: unknown mnemonic lgdt' -- lgdt kvm_gdt32_desc' bootstrap_x86_64.S:23: Error: operand 1 must be an integer register -- mov $0x11,%eax'
bootstrap_x86_64.S:24: Error: operand 1 must be an integer register -- `mov %eax,%cr0'
bootstrap_x86_64.S:26: Error: unknown mnemonic jmp' -- jmp $3*8,$protected_mode_entry'
bootstrap_x86_64.S:28: Error: unknown pseudo-op: `.code32'

Quoted 2 lines

For me I would at the least we need V=1 enabled or the names and versions of the tools being used to guess what is wrong here.

I'm not sure if I find some spare time in my after-work time. If yes, I'll post logs.

Kind regards,
Petr

Quoted 3 lines

mov $0x11,%eax looks like a really simple op to put 11 in EAX using AT&T/GAS syntax. If that goes wrong then possibly the wrong syntax is expected by the assembler or else a previous line is confusing it.

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp