x86: unify x86 Makefile(s)

Unify the 32 and 64 bit specific Makefiles.
The unification was simplest to do in one step although the readability of the patch suffers a bit from this.

Noteworthy remarks on the unification:
- The 64 bit cpu stuff should be moved to Makefile_32.cpu
but I did not feel confident doing it due to subtle differences
- The use of cflags-y were abandoned since we have seen one bug where we did wrong due to missing assignment to KBUILD_CFLAGS
- The "No need to remake" line are deleted. It caused "make -B" to fail
- For 64 bit the sub architecture stuff is not used but structure is kept to make it easy to introduce.
- The way head64.o is specified could be nicer - but it awaits the introduction of head32.o (which seems like a win to introduce for readability)
- -funit-at-a-time should be easy to unify but it looks like we have a bug in 32 bit. We only enable -funit-at-a-time for gcc less than 0400 if they support it (and I recall it is a gcc 4.00 feature). [-lt -> -gt] Introduced by 9ab34fe76114b9538bfcaf3a9d112dee0feb5f17 "enable unit-at-a-time optimisations for gcc4"
- Patch is checkpatch clean

Patch is tested by doing a defconfig build for i386 and x86_64 and in both cases monitoring that only relevant files were recompiled when applying the patch.

---
Additional unifications are possible I think apart from the cpu stuff. But I do not know enough about the gcc options to judge what happens if we introduce the currently 64 bit specific options in the 32 bit build. (-m64 is obviously not a candidate).

Sam

 arch/x86/Makefile    |  243 +++++++++++++++++++++++++++++++++++++++++++++++++-
 arch/x86/Makefile_32 |  181 -------------------------------------
 arch/x86/Makefile_64 |  141 -----------------------------
 3 files changed, 239 insertions(+), 326 deletions(-)

index 7aa1dc6..b0ab9e7 100644
@@ -7,13 +7,248 @@ else
         KBUILD_DEFCONFIG := $(ARCH)_defconfig
 endif
 
-# No need to remake these files
-$(srctree)/arch/x86/Makefile%: ;
+# BITS is used as extension for files which are available in a 32 bit
+# and a 64 bit version to simplify shared Makefiles.
+# e.g.: obj-y += foo_$(BITS).o
+export BITS
 
 ifeq ($(CONFIG_X86_32),y)
+        BITS := 32
         UTS_MACHINE := i386
-        include $(srctree)/arch/x86/Makefile_32
+
+        HAS_BIARCH      := $(call cc-option-yn, -m32)
+        ifeq ($(HAS_BIARCH),y)
+                AS              := $(AS) --32
+                LD              := $(LD) -m elf_i386
+                CC              := $(CC) -m32
+        endif
+
+        ifdef CONFIG_RELOCATABLE
+                LDFLAGS_vmlinux := --emit-relocs
+        endif
+        CHECKFLAGS += -D__i386__
+        KBUILD_CFLAGS += -msoft-float -mregparm=3 -freg-struct-return
 else
+        BITS := 64
         UTS_MACHINE := x86_64
-        include $(srctree)/arch/x86/Makefile_64
+        CHECKFLAGS += -D__x86_64__ -m64
 endif
+
+LDFLAGS         := -m elf_$(UTS_MACHINE)
+OBJCOPYFLAGS    := -O binary -R .note -R .comment -S
+KBUILD_CFLAGS += -pipe
+
+
+ifeq ($(CONFIG_X86_32),y)
+        # prevent gcc from keeping the stack 16 byte aligned
+        KBUILD_CFLAGS += $(call cc-option,-mpreferred-stack-boundary=2)
+
+        # Disable unit-at-a-time mode on pre-gcc-4.0 compilers, it makes gcc use
+        # a lot more stack due to the lack of sharing of stacklots:
+        KBUILD_CFLAGS += $(shell if [ $(call cc-version) -lt 0400 ] ; then \
+                echo $(call cc-option,-fno-unit-at-a-time); fi ;)
+
+        # CPU-specific tuning. Anything which can be shared with UML should go here.
+        include $(srctree)/arch/x86/Makefile_32.cpu
+        KBUILD_CFLAGS += $(cflags-y)
+
+        # temporary until string.h is fixed
+        KBUILD_CFLAGS += -ffreestanding
+
+else
+        KBUILD_AFLAGS += -m64
+        KBUILD_CFLAGS += -m64
+        # FIXME - should be integrated in Makefile.cpu
+        cflags-$(CONFIG_MK8) += $(call cc-option,-march=k8)
+        cflags-$(CONFIG_MPSC) += $(call cc-option,-march=nocona)
+        # gcc doesn't support -march=core2 yet as of gcc 4.3, but I hope it
+        # will eventually. Use -mtune=generic as fallback
+        cflags-$(CONFIG_MCORE2) += \
+                $(call cc-option,-march=core2,$(call cc-option,-mtune=generic))
+        cflags-$(CONFIG_GENERIC_CPU) += $(call cc-option,-mtune=generic)
+        KBUILD_CFLAGS += $(cflags-y)
+
+        KBUILD_CFLAGS += -mno-red-zone
+        KBUILD_CFLAGS += -mcmodel=kernel
+
+        KBUILD_CFLAGS += -Wno-sign-compare
+        KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
+        ifneq ($(CONFIG_DEBUG_INFO),y)
+                # -fweb shrinks the kernel a bit, but the difference is very small
+                # it also messes up debugging, so don't use it for now.
+                #KBUILD_CFLAGS += $(call cc-option,-fweb)
+        endif
+        # -funit-at-a-time shrinks the kernel .text considerably
+        # unfortunately it makes reading oopses harder.
+        KBUILD_CFLAGS += $(call cc-option,-funit-at-a-time)
+
+        # prevent gcc from generating any FP code by mistake
+        KBUILD_CFLAGS += $(call cc-option,-mno-sse -mno-mmx -mno-sse2 -mno-3dnow,)
+        # this works around some issues with generating unwind tables in older gccs
+        # newer gccs do it by default
+        KBUILD_CFLAGS += -maccumulate-outgoing-args
+
+        stackp := $(CONFIG_SHELL) $(srctree)/scripts/gcc-x86_64-has-stack-protector.sh
+        stackp-$(CONFIG_CC_STACKPROTECTOR) := $(shell $(stackp) \
+                "$(CC)" -fstack-protector )
+        stackp-$(CONFIG_CC_STACKPROTECTOR_ALL) += $(shell $(stackp) \
+                 "$(CC)" -fstack-protector-all )
+
+        KBUILD_CFLAGS += $(stackp-y)
+endif
+
+
+
+# Stackpointer is addressed different for 32 bit and 64 bit x86
+sp-$(CONFIG_X86_32) := esp
+sp-$(CONFIG_X86_64) := rsp
+
+# do binutils support CFI?
+cfi := $(call as-instr,.cfi_startproc\n.cfi_rel_offset $(sp-y)$(comma)0\n.cfi_endproc,-DCONFIG_AS_CFI=1)
+# is .cfi_signal_frame supported too?
+cfi-sigframe := $(call as-instr,.cfi_startproc\n.cfi_signal_frame\n.cfi_endproc,-DCONFIG_AS_CFI_SIGNAL_FRAME=1)
+KBUILD_AFLAGS += $(cfi) $(cfi-sigframe)
+KBUILD_CFLAGS += $(cfi) $(cfi-sigframe)
+
+
+###
+# Sub architecture support
+
+# Default subarch .c files
+mcore-y  := arch/x86/mach-default
+
+# Voyager subarch support
+mflags-$(CONFIG_X86_VOYAGER)	:= -Iinclude/asm-x86/mach-voyager
+mcore-$(CONFIG_X86_VOYAGER)	:= arch/x86/mach-voyager
+
+# VISWS subarch support
+mflags-$(CONFIG_X86_VISWS)	:= -Iinclude/asm-x86/mach-visws
+mcore-$(CONFIG_X86_VISWS)	:= arch/x86/mach-visws
+
+# NUMAQ subarch support
+mflags-$(CONFIG_X86_NUMAQ)	:= -Iinclude/asm-x86/mach-numaq
+mcore-$(CONFIG_X86_NUMAQ)	:= arch/x86/mach-default
+
+# BIGSMP subarch support
+mflags-$(CONFIG_X86_BIGSMP)	:= -Iinclude/asm-x86/mach-bigsmp
+mcore-$(CONFIG_X86_BIGSMP)	:= arch/x86/mach-default
+
+# RDC R-321x subarch support
+mflags-$(CONFIG_X86_RDC321X)	:= -Iinclude/asm-x86/mach-rdc321x
+mcore-$(CONFIG_X86_RDC321X)	:= arch/x86/mach-default
+core-$(CONFIG_X86_RDC321X)	+= arch/x86/mach-rdc321x/
+
+#Summit subarch support
+mflags-$(CONFIG_X86_SUMMIT) := -Iinclude/asm-x86/mach-summit
+mcore-$(CONFIG_X86_SUMMIT)  := arch/x86/mach-default
+
+# generic subarchitecture
+mflags-$(CONFIG_X86_GENERICARCH) := -Iinclude/asm-x86/mach-generic
+mcore-$(CONFIG_X86_GENERICARCH) := arch/x86/mach-default
+core-$(CONFIG_X86_GENERICARCH) += arch/x86/mach-generic/
+
+# ES7000 subarch support
+mflags-$(CONFIG_X86_ES7000)	:= -Iinclude/asm-x86/mach-es7000
+mcore-$(CONFIG_X86_ES7000)	:= arch/x86/mach-default
+core-$(CONFIG_X86_ES7000)	:= arch/x86/mach-es7000/
+
+# default subarch .h files
+mflags-y += -Iinclude/asm-x86/mach-default
+
+# 64 bit does not use subarch support - clear relevant variables
+core-$(CONFIG_X86_64)   :=
+mcore-$(CONFIG_X86_64)  :=
+mflags-$(CONFIG_X86_64) :=
+
+KBUILD_CFLAGS += $(mflags-y)
+KBUILD_AFLAGS += $(mflags-y)
+
+
+###
+# Kernel objects
+
+head-y                := arch/x86/kernel/head_$(BITS).o
+head-$(CONFIG_X86_64) += arch/x86/kernel/head64.o
+head-y                += arch/x86/kernel/init_task.o
+
+libs-y  += arch/x86/lib/
+
+# Xen paravirtualization support
+core-$(CONFIG_XEN) += arch/x86/xen/
+
+# lguest paravirtualization support
+core-$(CONFIG_LGUEST_GUEST) += arch/x86/lguest/
+
+core-y += arch/x86/kernel/
+core-y += arch/x86/mm/
+core-y += $(mcore-y)/
+core-y += arch/x86/crypto/
+core-y += arch/x86/vdso/
+core-$(CONFIG_IA32_EMULATION) += arch/x86/ia32/
+
+# drivers-y are linked after core-y
+drivers-$(CONFIG_MATH_EMULATION) += arch/x86/math-emu/
+drivers-$(CONFIG_PCI)            += arch/x86/pci/
+
+# must be linked after kernel/
+drivers-$(CONFIG_OPROFILE) += arch/x86/oprofile/
+
+ifeq ($(CONFIG_X86_32),y)
+drivers-$(CONFIG_PM) += arch/x86/power/
+drivers-$(CONFIG_FB) += arch/x86/video/
+endif
+
+boot := arch/x86/boot
+
+PHONY += zImage bzImage compressed zlilo bzlilo \
+         zdisk bzdisk fdimage fdimage144 fdimage288 isoimage install
+
+# Default kernel to build
+all: bzImage
+
+# KBUILD_IMAGE specify target image being built
+                    KBUILD_IMAGE := $(boot)/bzImage
+zImage zlilo zdisk: KBUILD_IMAGE := arch/x86/boot/zImage
+
+zImage bzImage: vmlinux
+	$(Q)$(MAKE) $(build)=$(boot) $(KBUILD_IMAGE)
+	$(Q)mkdir -p $(objtree)/arch/$(UTS_MACHINE)/boot
+	$(Q)ln -fsn ../../x86/boot/bzImage $(objtree)/arch/$(UTS_MACHINE)/boot/bzImage
+
+compressed: zImage
+
+zlilo bzlilo: vmlinux
+	$(Q)$(MAKE) $(build)=$(boot) BOOTIMAGE=$(KBUILD_IMAGE) zlilo
+
+zdisk bzdisk: vmlinux
+	$(Q)$(MAKE) $(build)=$(boot) BOOTIMAGE=$(KBUILD_IMAGE) zdisk
+
+fdimage fdimage144 fdimage288 isoimage: vmlinux
+	$(Q)$(MAKE) $(build)=$(boot) BOOTIMAGE=$(KBUILD_IMAGE) $@
+
+install: vdso_install
+	$(Q)$(MAKE) $(build)=$(boot) BOOTIMAGE=$(KBUILD_IMAGE) install
+
+PHONY += vdso_install
+vdso_install:
+	$(Q)$(MAKE) $(build)=arch/x86/vdso $@
+
+archclean:
+	$(Q)rm -rf $(objtree)/arch/i386
+	$(Q)rm -rf $(objtree)/arch/x86_64
+	$(Q)$(MAKE) $(clean)=$(boot)
+
+define archhelp
+  echo  '* bzImage	- Compressed kernel image (arch/x86/boot/bzImage)'
+  echo  '  install	- Install kernel using'
+  echo  '		   (your) ~/bin/installkernel or'
+  echo  '		   (distribution) /sbin/installkernel or'
+  echo  '		   install to $$(INSTALL_PATH) and run lilo'
+  echo  '  bzdisk       - Create a boot floppy in /dev/fd0'
+  echo  '  fdimage      - Create a boot floppy image'
+  echo  '  isoimage     - Create a boot CD-ROM image'
+endef
+
+CLEAN_FILES += arch/x86/boot/fdimage \
+	       arch/x86/boot/image.iso \
+	       arch/x86/boot/mtools.conf

No, we _dis_able it on 32bit if a gcc < 4.0 supports it.

unit-at-a-time was introduced in upstream gcc 3.4 and backported to some popular 3.3 x86_64 branch.

i386 and x86_64 go in exactly opposite directions regarding when to use unit-at-a-time, but that was intentional and we should keep it that way.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

i386 disabled it because there were claims that it increased stack size too much, but I haven't seen any real evidence for that. That is why I never added it to x86-64. On the other hand it tends to shrink text size considerably.

I think it should be reevaluated and enabled by default on i386 again.

BTW there is a patch pending for gcc 4.3+ that prevents optimizations that increase stack size unduly. Not sure it has been added there yet, but that was the right fix for these problems anyways.

-Andi

We are talking only about gcc 3.4 on i386 - for gcc >= 4.0 or gcc 3.4 on other architectures we always leave unit-at-a-time enabled.

Considering that:
- there were some problems with gcc 3.4 and i386 4kB stacks,
- gcc 3.4 is ancient (and therefore nothing people who want the fastest or smallest code should use) and
- the number of i386 gcc 3.4 users is quite low today
we should simply leave it as it is, and all this will anyway automatically vanish when we'll drop support for gcc < 4.0 a few years from now.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

I have:

$ gcc --version
gcc (GCC) 4.1.2 20070925 (Red Hat 4.1.2-33)

And with the current code in arch/x86/Makefile_32 I do not have -funit-at-a-time.

Sample gcc cammandline:
gcc -m32 -Wp,-MD,init/.do_mounts_rd.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include -D__KERNEL__ -Iinclude -include include/linux/autoconf.h -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -pipe -msoft-float -mregparm=3 -freg-struct-return -mpreferred-stack-boundary=2 -march=i686 -mtune=pentium3 -mtune=generic -ffreestanding -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -Iinclude/asm-x86/mach-generic -Iinclude/asm-x86/mach-default -fomit-frame-pointer -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(do_mounts_rd)" -D"KBUILD_MODNAME=KBUILD_STR(mounts)" -c -o init/do_mounts_rd.o init/do_mounts_rd.c

This is the relevant snippet from the Makefile:
KBUILD_CFLAGS += $(shell if [ $(call cc-version) -lt 0400 ] ; then echo $(call cc-option,-fno-unit-at-a-time); fi ;)

It reads:
If gcc version is less than 4.00 then enable -funit-at-a-time only for gcc that supports it. -funit-at-a-time is NOT enabled for gcc >= 4.00

So it will enable -funit-at-a-time for gcc 3.4 as it is now.
Which was what we wanted to avoid..
So I see no reason not to enable it uncondintionally (if gcc supports it) as we have used it for 3.4 for as long as we have introduced -funit-at-a-time.

Sam

You do not have this option, but you do not need it since it's enabled with -O in all gcc versions that support it except for some vendor backports to gcc 3.3.

It is not _explicitely_ enabled for gcc >= 4.0 since it's anyway enabled.

We have never used it with gcc 3.4 on i386.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

If -funit-at-a-time really increases stack size too much on some compiler version the right fix would be to check where it does that using make checkstack and then add "noinline" attributes there to prevent the compiler from inlining. That would prevent them.

Globally disabling it is too big a hammer.

e.g. I know XFS did it in a similar way to prevent this problem.

So I would reenable it for now and if you know it causes problems on specific compiler versions, Adrian, you could watch make checkstack there and submit noinline patches as needed.

-Andi

The main point is that we are _only_ talking about gcc 3.4 on i386 - for more recent compilers we do not disable unit-at-a-time.

First of all our user - and therefore tester - base with this compiler has become quite small. And checkstack alone doesn't help that much with finding the problems since it only lists per-function stack usage, not the stack usage of the complete call chain.

People who want maximum performance and/or minimum code size anyway won't use a more than 3 years old compiler.

If we were talking about gcc 4.2 I would agree with you, but I simply do not see the point in risking regressions for gcc 3.4 users when the only benefit would be better code with an ancient compiler.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

I took a second look and realized that Adrian is right about the disabling. I had overlooked the 'no' part in -fno-unit-at-a-time.

So to try it out I build a gcc with crosstool:
gcc --version:
i686-unknown-linux-gnu-gcc (GCC) 3.4.5

This is the version that should exhibit the behaviour with increased text size. Build two kernels - one without and one with -funit-at-a-time:

size output:
text data bss dec hex filename

4878494  471580  610304 5960378  5af2ba o-i386/vmlinux
4955008  472934  614400 6042342  5c32e6 o-i386-no-funit-at-a-time/vmlinux

[Last line is with unit-at-a-time disabled as per the filename.]

With a i386 defconfig we see an *increase* in text size if we disable -funit-at-a-time. That is we see the opposite of the expected.

Now the bug in gcc may have been fixed between 3.4 and 3.4.5 - I dunno.

So this simple test favour to always enable unit-at-a-time.

Sam

The issue was not text size (u-a-a-t generally decreases that) but maximum stack usage. You need to check make checkstack output.

-Andi

gcc 3.4.5 with: -fno-unit-at-a-time:
0xc0173b9a do_sys_poll [vmlinux]: 852
0xc0173cc2 do_sys_poll [vmlinux]: 852
0xc0206f8e nlmclnt_reclaim [vmlinux]: 704
0xc020701e nlmclnt_reclaim [vmlinux]: 704
0xc01a2c83 reiserfs_rename [vmlinux]: 668
0xc01a34a0 reiserfs_rename [vmlinux]: 668
0xc0173052 do_select [vmlinux]: 656
0xc0173320 do_select [vmlinux]: 656
0xc0417ceb do_ipv6_setsockopt [vmlinux]: 608
0xc0418746 do_ipv6_setsockopt [vmlinux]: 608

gcc 3.4.5 with: -funit-at-a-time:
0xc02926e8 lo_ioctl [vmlinux]: 1336
0xc0293055 lo_ioctl [vmlinux]: 1336
0xc03bf222 dev_ethtool [vmlinux]: 908
0xc03c00e5 dev_ethtool [vmlinux]: 908
0xc036adf9 cdrom_open [vmlinux]: 892
0xc036b78d cdrom_open [vmlinux]: 892
0xc0172445 do_sys_poll [vmlinux]: 860
0xc01726e9 do_sys_poll [vmlinux]: 860
0xc020fca3 sys_semctl [vmlinux]: 796
0xc0210447 sys_semctl [vmlinux]: 796
0xc02058af nlmclnt_reclaim [vmlinux]: 704
0xc020593f nlmclnt_reclaim [vmlinux]: 704
0xc01719d4 do_select [vmlinux]: 668
0xc0171d59 do_select [vmlinux]: 668
0xc01a121a reiserfs_rename [vmlinux]: 668
0xc01a1a54 reiserfs_rename [vmlinux]: 668
0xc0412e39 ipv6_setsockopt [vmlinux]: 612
0xc04138ca ipv6_setsockopt [vmlinux]: 612

So here we indeed see the expected (bad) behaviour. unit-at-a-time causes much higher stack usage in some spots. Above is listed only the hits above 600 bytes.

Until we are on par with stack usage I recommend to keep -fno-unit-at-a-time disabled for gcc less than 4.00 as suggested by Adrian (as is what we have today).

Sam

Again the correct fix is to add "noinline" to the functions that are getting inlined here in 3.4 but not 4+.

-Andi

The problem isn't that gcc 3.4 and gcc 4.x inlined different functions, the problem is that with gcc 3.4 the stack usage was higher.

AFAIR, the problem was roughly something like:

<-- snip -->

static void foo(void)
{
...
}

static void bar(void)
{
...
}

void foobar(void)

{
	foo();
	bar();
}

<-- snip -->

Without inlining the maxmimum stack usage inside foobar() is max(stack usage foo(), stack usage bar()). [1]

With foo() and bar() inlined (-funit-at-a-time also enables -finline-functions-called-once), the maxmimum stack usage inside foobar() is sum(stack usage foo(), stack usage bar()). And this worst case is the area where gcc 4 is much better than gcc 3.4.

We have cases in the kernel were a switch handles a dozen cases (e.g. ioctl handling) and each results in a call to a different static function.

cu
Adrian

[1] this ignores function call stack usages, but these are low compared to the interesting ones

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

It's a little more complicated. gcc 4.x (not sure which x, might 0) is clever enough to not use max() stack, but only use the stack for the different scopes as needed similar as when the calls weren't inlined. But gcc 3 didn't do that.

Yes exactly. If the functions weren't inlined the problem wouldn't occur because the stack sizes do not add up in the same dynamic call chain. Thus a few statetic noinlines will fix it.

-Andi

That's roughly what I wanted to say.

And we are back at my main point that risking regressions for getting better code with some ancient compiler isn't worth it.

Plus the fact that noinline's might result in slightly worse code with current compilers.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

I hope not. subarch is one of the main disaster areas in the i386 port.

And that too should be on for i386 too.

The Makefile part looks reasonably ok - the implementation I have not looked at.

OK

OK

OK

OK - I was tempted to do so as it is a noop anyway.

OK

OK

Thanks for the feedback Andi.
I will prepare a small patch serie so we have the additional options applied for i386 in individual patches and the original patch will only be slightly updated.

Sam