ppc64le: ftrace self-tests and $(CC_FLAGS_FTRACE) broken?
Hi Steve, Nicholas,
I stumbled across this while working on livepatch self-tests, which like the ftrace self-tests, specify $(CC_FLAGS_FTRACE) to ensure that target functions are indeed trace-able.
On ppc64le, v5.0 fails the self-tests on my machine:
% gcc --version gcc (GCC) 8.2.1 20180905 (Red Hat 8.2.1-3)
% git checkout v5.0
% grep FTRACE_SELFTEST .config
CONFIG_FTRACE_SELFTEST=y
After building the kernel, I noticed no mcount sections and of course the self-tests fail as it can't hook functions:
% readelf --sections kernel/trace/trace_selftest_dynamic.o | grep mcount (nothing)
[ ... snip ... ]
[ 126.147892] Testing all events: OK [ 126.390487] WARNING: CPU: 0 PID: 1 at kernel/trace/trace_events.c:3421 event_trace_self_tests_init+0x94/0xf4 [ 126.390498] Modules linked in: [ 126.390510] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 5.0.0 #10 [ 126.390520] NIP: c000000000f044d4 LR: c000000000f044cc CTR: c000000000b63d70 [ 126.390530] REGS: c0000001fb183910 TRAP: 0700 Tainted: G W (5.0.0) [ 126.390540] MSR: 8000000002029033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 28000222 XER: 20040007 [ 126.390564] CFAR: c00000000025b1f4 IRQMASK: 0 [ 126.390564] GPR00: c000000000f044cc c0000001fb183ba0 c000000001453c00 ffffffffffffffed [ 126.390564] GPR04: 0000000000000000 c0000000017a0aac 000000000000037d c0000001fb142d00 [ 126.390564] GPR08: 0000000000000000 0000000000000001 0000000000000000 0000000000000000 [ 126.390564] GPR12: 0000000000000000 c000000001880000 c000000000010530 0000000000000000 [ 126.390564] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 126.390564] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 126.390564] GPR24: 0000000000000000 0000000000000000 c000000000eb8908 c000000000f4ea68 [ 126.390564] GPR28: c000000000f4eaa8 0000000000000000 c000000000f04440 c000000000f581a8 [ 126.390662] NIP [c000000000f044d4] event_trace_self_tests_init+0x94/0xf4 [ 126.390672] LR [c000000000f044cc] event_trace_self_tests_init+0x8c/0xf4 [ 126.390682] Call Trace: [ 126.390690] [c0000001fb183ba0] [c000000000f044cc] event_trace_self_tests_init+0x8c/0xf4 (unreliable) [ 126.390705] [c0000001fb183c10] [c000000000010134] do_one_initcall+0x64/0x264 [ 126.390719] [c0000001fb183ce0] [c000000000ed443c] kernel_init_freeable+0x36c/0x470 [ 126.390731] [c0000001fb183db0] [c000000000010554] kernel_init+0x2c/0x148 [ 126.390744] [c0000001fb183e20] [c00000000000b65c] ret_from_kernel_thread+0x5c/0x80 [ 126.390755] Instruction dump: [ 126.390765] 3fe2ffb0 f9284578 3bff3d50 794ad182 0b0a0000 2fa90000 419e0058 3bff0858 [ 126.390789] 7fe3fb78 4b356cc1 60000000 54690ffe <0b090000> 2f830000 409c0018 3c62ff96 [ 126.390814] ---[ end trace b5898e7f7f73e163 ]--- [ 126.390822] Failed to enable function tracer for event tests
I git bisected to 6977f95e63b9 ("powerpc: avoid -mno-sched-epilog on GCC 4.9 and newer"). Reverting this change and rebuilding, I could see the mcount sections once again:
% readelf --sections kernel/trace/trace_selftest_dynamic.o | grep mcount
[ 3] __mcount_loc PROGBITS 0000000000000000 00000060
[ 4] .rela__mcount_loc RELA 0000000000000000 000258c8Reboot and the self-test now passes.
Perhaps this is gcc version specific? I didn't see any other reports, so maybe its specific to my config. If I run make V=1, I can see that gcc is called with '-pg -mprofile-kernel', but then the record_mcount script is skipped. Any ideas?
Regards,
-- Joe
Quoted 4 lines · On Sat, 23 Mar 2019 09:19:50 -0400 Joe Lawrence <[email protected]> wrote:
Perhaps this is gcc version specific? I didn't see any other reports, so maybe its specific to my config. If I run make V=1, I can see that gcc is called with '-pg -mprofile-kernel', but then the
record_mcountscript is skipped. Any ideas?
But you see it running the script if you remove that commit? Do you happen to see -mrecord-mcount at all in a V=1 make?
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 4 lines · On Sat, 23 Mar 2019 09:19:50 -0400 Joe Lawrence <[email protected]> wrote:
Perhaps this is gcc version specific? I didn't see any other reports, so maybe its specific to my config. If I run make V=1, I can see that gcc is called with '-pg -mprofile-kernel', but then the
record_mcountscript is skipped. Any ideas?But you see it running the script if you remove that commit? Do you happen to see -mrecord-mcount at all in a V=1 make?
Hey Steve,
Here's a diff of output from 'make V=1 kernel/trace/trace_selftest_dynamic.o' with stock v5.0 vs. v5.0 and that commit reverted. The part that I think is interesting is where it decides to invoke recordmcount.pl
I can send or pastebin the entire output from make if that is easier to read.
-- Joe
v5.0-revert.out renamed from v5.0.out +4 -15
@@ -20,29 +20,18 @@ make -f ./scripts/Makefile.build obj=scr make -f ./scripts/Makefile.build obj=scripts/selinux/genheaders need-builtin= make -f ./scripts/Makefile.build obj=scripts/selinux/mdp need-builtin= make -f ./scripts/Makefile.build obj=scripts/mod - gcc -Wp,-MD,scripts/mod/.empty.o.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"empty"' -DKBUILD_MODNAME='"empty"' -c -o scripts/mod/empty.o scripts/mod/empty.c - if objdump -h scripts/mod/empty.o | grep -q __ksymtab; then gcc -E -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -D__GENKSYMS__ -Wp,-MD,scripts/mod/.empty.o.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"empty"' -DKBUILD_MODNAME='"empty"' scripts/mod/empty.c | scripts/genksyms/genksyms -R -r /dev/null > scripts/mod/.tmp_empty.ver; ld -EL -m elf64lppc -r -o scripts/mod/.tmp_empty.o scripts/mod/empty.o -T scripts/mod/.tmp_empty.ver; mv -f scripts/mod/.tmp_empty.o scripts/mod/empty.o; rm -f scripts/mod/.tmp_empty.ver; fi - if [ "-pg -mprofile-kernel " = "-pg -mprofile-kernel " ]; then perl ./scripts/recordmcount.pl "powerpc" "little" "64" "objdump" "objcopy" "gcc -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned " "ld -EL -m elf64lppc" "nm --synthetic" "" "" "0" "scripts/mod/empty.o"; fi - scripts/mod/mk_elfconfig < scripts/mod/empty.o > scripts/mod/elfconfig.h - gcc -Wp,-MD,scripts/mod/.modpost.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/mod/modpost.o scripts/mod/modpost.c - gcc -Wp,-MD,scripts/mod/.devicetable-offsets.s.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"devicetable_offsets"' -DKBUILD_MODNAME='"devicetable_offsets"' -fverbose-asm -S -o scripts/mod/devicetable-offsets.s scripts/mod/devicetable-offsets.c set -e; mkdir -p scripts/mod/; { echo "#ifndef __DEVICETABLE_OFFSETS_H__"; echo "#define __DEVICETABLE_OFFSETS_H__"; echo "/*"; echo " * DO NOT MODIFY."; echo " *"; echo " * This file was generated by Kbuild"; echo " */"; echo ""; sed -ne 's:^[[:space:]]*\.ascii[[:space:]]*"\(.*\)".*:\1:; /^->/{s:->#\(.*\):/* \1 */:; s:^->\([^ ]*\) [\$#]*\([^ ]*\) \(.*\):#define \1 \2 /* \3 */:; s:->::; p;}' < scripts/mod/devicetable-offsets.s; echo ""; echo "#endif"; } > scripts/mod/devicetable-offsets.h.tmp; if [ -r scripts/mod/devicetable-offsets.h ] && cmp -s scripts/mod/devicetable-offsets.h scripts/mod/devicetable-offsets.h.tmp; then rm -f scripts/mod/devicetable-offsets.h.tmp; else : ' UPD scripts/mod/devicetable-offsets.h'; mv -f scripts/mod/devicetable-offsets.h.tmp scripts/mod/devicetable-offsets.h; fi - gcc -Wp,-MD,scripts/mod/.file2alias.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/mod/file2alias.o scripts/mod/file2alias.c - gcc -Wp,-MD,scripts/mod/.sumversion.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/mod/sumversion.o scripts/mod/sumversion.c - gcc -o scripts/mod/modpost scripts/mod/modpost.o scripts/mod/file2alias.o scripts/mod/sumversion.o make -f ./scripts/Makefile.build obj=. - gcc -Wp,-MD,kernel/.bounds.s.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"bounds"' -DKBUILD_MODNAME='"bounds"' -fverbose-asm -S -o kernel/bounds.s kernel/bounds.c set -e; mkdir -p include/generated/; { echo "#ifndef __LINUX_BOUNDS_H__"; echo "#define __LINUX_BOUNDS_H__"; echo "/*"; echo " * DO NOT MODIFY."; echo " *"; echo " * This file was generated by Kbuild"; echo " */"; echo ""; sed -ne 's:^[[:space:]]*\.ascii[[:space:]]*"\(.*\)".*:\1:; /^->/{s:->#\(.*\):/* \1 */:; s:^->\([^ ]*\) [\$#]*\([^ ]*\) \(.*\):#define \1 \2 /* \3 */:; s:->::; p;}' < kernel/bounds.s; echo ""; echo "#endif"; } > include/generated/bounds.h.tmp; if [ -r include/generated/bounds.h ] && cmp -s include/generated/bounds.h include/generated/bounds.h.tmp; then rm -f include/generated/bounds.h.tmp; else : ' UPD include/generated/bounds.h'; mv -f include/generated/bounds.h.tmp include/generated/bounds.h; fi set -e; mkdir -p include/generated/; { echo 100 | bc -q kernel/time/timeconst.bc; } > include/generated/timeconst.h.tmp; if [ -r include/generated/timeconst.h ] && cmp -s include/generated/timeconst.h include/generated/timeconst.h.tmp; then rm -f include/generated/timeconst.h.tmp; else : ' UPD include/generated/timeconst.h'; mv -f include/generated/timeconst.h.tmp include/generated/timeconst.h; fi - gcc -Wp,-MD,arch/powerpc/kernel/.asm-offsets.s.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"asm_offsets"' -DKBUILD_MODNAME='"asm_offsets"' -fverbose-asm -S -o arch/powerpc/kernel/asm-offsets.s arch/powerpc/kernel/asm-offsets.c set -e; mkdir -p include/generated/; { echo "#ifndef __ASM_OFFSETS_H__"; echo "#define __ASM_OFFSETS_H__"; echo "/*"; echo " * DO NOT MODIFY."; echo " *"; echo " * This file was generated by Kbuild"; echo " */"; echo ""; sed -ne 's:^[[:space:]]*\.ascii[[:space:]]*"\(.*\)".*:\1:; /^->/{s:->#\(.*\):/* \1 */:; s:^->\([^ ]*\) [\$#]*\([^ ]*\) \(.*\):#define \1 \2 /* \3 */:; s:->::; p;}' < arch/powerpc/kernel/asm-offsets.s; echo ""; echo "#endif"; } > include/generated/asm-offsets.h.tmp; if [ -r include/generated/asm-offsets.h ] && cmp -s include/generated/asm-offsets.h include/generated/asm-offsets.h.tmp; then rm -f include/generated/asm-offsets.h.tmp; else : ' UPD include/generated/asm-offsets.h'; mv -f include/generated/asm-offsets.h.tmp include/generated/asm-offsets.h; fi - /bin/sh scripts/checksyscalls.sh gcc -Wp,-MD,./.missing-syscalls.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"missing_syscalls"' -DKBUILD_MODNAME='"missing_syscalls"' + /bin/sh scripts/checksyscalls.sh gcc -Wp,-MD,./.missing-syscalls.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -mno-sched-epilog -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -DKBUILD_BASENAME='"missing_syscalls"' -DKBUILD_MODNAME='"missing_syscalls"' <stdin>:1478:2: warning: #warning syscall pidfd_send_signal not implemented [-Wcpp] <stdin>:1481:2: warning: #warning syscall io_uring_setup not implemented [-Wcpp] <stdin>:1484:2: warning: #warning syscall io_uring_enter not implemented [-Wcpp] <stdin>:1487:2: warning: #warning syscall io_uring_register not implemented [-Wcpp] /bin/sh scripts/atomic/check-atomics.sh make -f ./scripts/Makefile.build obj=kernel/trace kernel/trace/trace_selftest_dynamic.o - gcc -Wp,-MD,kernel/trace/.trace_selftest_dynamic.o.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 -pg -mprofile-kernel -DKBUILD_BASENAME='"trace_selftest_dynamic"' -DKBUILD_MODNAME='"trace_selftest_dynamic"' -c -o kernel/trace/trace_selftest_dynamic.o kernel/trace/trace_selftest_dynamic.c - if objdump -h kernel/trace/trace_selftest_dynamic.o | grep -q __ksymtab; then gcc -E -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 -D__GENKSYMS__ -Wp,-MD,kernel/trace/.trace_selftest_dynamic.o.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 -pg -mprofile-kernel -DKBUILD_BASENAME='"trace_selftest_dynamic"' -DKBUILD_MODNAME='"trace_selftest_dynamic"' kernel/trace/trace_selftest_dynamic.c | scripts/genksyms/genksyms -R -r /dev/null > kernel/trace/.tmp_trace_selftest_dynamic.ver; ld -EL -m elf64lppc -r -o kernel/trace/.tmp_trace_selftest_dynamic.o kernel/trace/trace_selftest_dynamic.o -T kernel/trace/.tmp_trace_selftest_dynamic.ver; mv -f kernel/trace/.tmp_trace_selftest_dynamic.o kernel/trace/trace_selftest_dynamic.o; rm -f kernel/trace/.tmp_trace_selftest_dynamic.ver; fi - if [ "" = "-pg -mprofile-kernel " ]; then perl ./scripts/recordmcount.pl "powerpc" "little" "64" "objdump" "objcopy" "gcc -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 " "ld -EL -m elf64lppc" "nm --synthetic" "" "" "0" "kernel/trace/trace_selftest_dynamic.o"; fi + gcc -Wp,-MD,kernel/trace/.trace_selftest_dynamic.o.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 -pg -mprofile-kernel -mno-sched-epilog -DKBUILD_BASENAME='"trace_selftest_dynamic"' -DKBUILD_MODNAME='"trace_selftest_dynamic"' -c -o kernel/trace/trace_selftest_dynamic.o kernel/trace/trace_selftest_dynamic.c + if objdump -h kernel/trace/trace_selftest_dynamic.o | grep -q __ksymtab; then gcc -E -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -pg -mprofile-kernel -mno-sched-epilog -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 -D__GENKSYMS__ -Wp,-MD,kernel/trace/.trace_selftest_dynamic.o.d -nostdinc -isystem /usr/lib/gcc/ppc64le-redhat-linux/8/include -I./arch/powerpc/include -I./arch/powerpc/include/generated -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 -pg -mprofile-kernel -mno-sched-epilog -DKBUILD_BASENAME='"trace_selftest_dynamic"' -DKBUILD_MODNAME='"trace_selftest_dynamic"' kernel/trace/trace_selftest_dynamic.c | scripts/genksyms/genksyms -R -r /dev/null > kernel/trace/.tmp_trace_selftest_dynamic.ver; ld -EL -m elf64lppc -r -o kernel/trace/.tmp_trace_selftest_dynamic.o kernel/trace/trace_selftest_dynamic.o -T kernel/trace/.tmp_trace_selftest_dynamic.ver; mv -f kernel/trace/.tmp_trace_selftest_dynamic.o kernel/trace/trace_selftest_dynamic.o; rm -f kernel/trace/.tmp_trace_selftest_dynamic.ver; fi + if [ "-pg -mprofile-kernel -mno-sched-epilog" = "-pg -mprofile-kernel -mno-sched-epilog" ]; then perl ./scripts/recordmcount.pl "powerpc" "little" "64" "objdump" "objcopy" "gcc -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mlittle-endian -m64 -msoft-float -pipe -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power8 -mtune=power9 -mno-altivec -mno-vsx -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -Wa,-mpower4 -Wa,-many -mno-strict-align -mlittle-endian -mstack-protector-guard=tls -mstack-protector-guard-reg=r13 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -gdwarf-4 -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -mstack-protector-guard-offset=2936 -D_TASK_CPU=300 " "ld -EL -m elf64lppc" "nm --synthetic" "" "" "0" "kernel/trace/trace_selftest_dynamic.o"; fi
Quoted 11 lines · Steven Rostedt
Quoted 4 lines · On Sat, 23 Mar 2019 09:19:50 -0400 Joe Lawrence <[email protected]> wrote:
Perhaps this is gcc version specific? I didn't see any other reports, so maybe its specific to my config. If I run make V=1, I can see that gcc is called with '-pg -mprofile-kernel', but then the
record_mcountscript is skipped. Any ideas?But you see it running the script if you remove that commit? Do you happen to see -mrecord-mcount at all in a V=1 make?
See entire make output from the two runs here:
http://people.redhat.com/~jolawren/ppc64le-mprofile/
-- Joe
Quoted 9 lines · Joe Lawrence
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Sat, 23 Mar 2019 09:19:50 -0400 Joe Lawrence <[email protected]> wrote:
Perhaps this is gcc version specific? I didn't see any other reports, so maybe its specific to my config. If I run make V=1, I can see that gcc is called with '-pg -mprofile-kernel', but then the
record_mcountscript is skipped. Any ideas?
Quoted 6 lines
Quoted 3 lines
But you see it running the script if you remove that commit? Do you happen to see -mrecord-mcount at all in a V=1 make?
See entire make output from the two runs here:
http://people.redhat.com/~jolawren/ppc64le-mprofile/
Mystery solved! That other commit inadvertently added a trailing space to CC_FLAGS_FTRACE, which then confused the findstring call in the cmd_record_mcount Makefile function.
Workaround patch just posted ... kbuild experts can suggest better fixes if that one is sub optimal.
Thanks,
-- Joe