2.6.0-tiny1 tree for small systems

This is the second release of the -tiny kernel tree. The aim of this tree is to collect patches that reduce kernel disk and memory footprint as well as tools for working on small systems. Target users are things like embedded systems, small or legacy desktop folks, and handhelds.

Latest release includes:
- sync against 2.6.0
- fix for CONFIG_SYSCTL compilation
- Jeff Garzik's netdrvr patchkit, along with latest netpoll/netconsole code
- kgdb and kgdb-over-ethernet debugging support
- "make checkstack" to find largest stack users
- script to estimate actual code size of inline functions (experimental)
- some other minor new tweaks

Thanks to Holger Shuring, Bill Irwin, and Zwane Mwaikamba for sending me various tweaks.

The patch, currently against 2.6.0, can be found at:

http://selenic.com/tiny/2.6.0-tiny1.patch.bz2
http://selenic.com/tiny/2.6.0-tiny1-broken-out.tar.bz2

And while I'm at it, I'll include some demo logs:

[my tiny test config: ide + vt + ext2 + net ]

$ size vmlinux
   text    data     bss     dec     hex filename
 861316   76979   20524  958819   ea163 vmlinux

LILO boot: tiny mem=4m
Loading tiny.........
Uncompressing Linux... Ok, booting the kernel.

# cat /proc/meminfo
MemTotal:         2772 kB
MemFree:          1096 kB
Buffers:            64 kB
Cached:            896 kB
SwapCached:          0 kB
Active:            788 kB
Inactive:          392 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:         2772 kB
LowFree:          1096 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:            616 kB
Slab:              356 kB
Committed_AS:      280 kB
PageTables:         48 kB
VmallocTotal:  1032168 kB
VmallocUsed:         0 kB
VmallocChunk:  1032168 kB
#

[rebooting with kmalloc auditing and kallsyms]

# cat /proc/kmalloc
total bytes allocated:   266848
slack bytes allocated:    37774
net bytes allocated:     145568
number of allocs:           732
number of frees:            282
number of callers:           71
lost callers:                 0
lost allocs:                  0
unknown frees:                0

   total    slack      net alloc/free  caller
     256      203      256     8/0     alloc_vfsmnt+0x73
    8192     3648     4096     2/1     atkbd_connect+0x1b
     192       48       64     3/2     seq_open+0x10
   12288        0     4096     3/2     seq_read+0x53
    8192        0        0     2/2     alloc_skb+0x3b
     960        0        0    10/10    load_elf_interp+0xa1
    1920      288        0    10/10    load_elf_binary+0x100
     320      130        0    10/10    load_elf_binary+0x1d8
     192       48       96     6/3     request_irq+0x22
    7200     1254     7200    75/0     proc_create+0x74
      64       43       64     2/0     proc_symlink+0x40
    4096      984        0     1/1     check_partition+0x1b
   69632        0    45056    17/6     dup_task_struct+0x38
     128       48      128     2/0     netlink_create+0x84
     128       20      128     1/0     ext2_fill_super+0x2f
      32       28       32     1/0     ext2_fill_super+0x385
      32       31       32     1/0     ext2_fill_super+0x3b6
     608       76      384    19/7     __request_region+0x18
      64       32       64     2/0     rand_initialize_disk+0xd
    8192     2016     8192     2/0     alloc_tty_struct+0x10
     128       56      128     2/0     init_dev+0xba
     128       56      128     2/0     init_dev+0xf3
     128        0      128     2/0     create_workqueue+0x28
    8960     1680     8960    70/0     tty_add_class_device+0x20
    2048      960     2048     4/0     alloc_tty_driver+0x10
    9280     2332     9280     4/0     tty_register_driver+0x2d
     288        0      288     9/0     mempool_create+0x16
    1280      196     1280     9/0     mempool_create+0x41
    1536      384     1536     8/0     mempool_create+0x8f
      64       28       64     1/0     kbd_connect+0x3e
     928      348        0    29/29    kmem_cache_create+0x235
   28288     1448    28288    81/0     do_tune_cpucache+0x2c

...

[counting inline function calls]

$ make 2>../inline-info
  CHK     include/linux/version.h
  SPLIT   include/linux/autoconf.h -> include/config/*
make[1]: `arch/i386/kernel/asm-offsets.s' is up to date.
  CHK     include/asm-i386/asm_offsets.h
  CC      init/main.o
...
  LD      arch/i386/boot/compressed/vmlinux
  OBJCOPY arch/i386/boot/vmlinux.bin
  BUILD   arch/i386/boot/bzImage
Root device is (3, 3)
Boot sector 512 bytes.
Setup is 2538 bytes.
System is 485 kB
Kernel: arch/i386/boot/bzImage is ready
$ scripts/count-inlines ../inline-info
2682  current_thread_info (485 in *.c)
calls:
callers: get_current(1764) <other>(276) copy_from_user(135)
copy_to_user(131) test_thread_flag(36) tcp_put_port(23)
tcp_set_state(22) verify_area(21) TLan_SetTimer(10)
tcp_enter_memory_pressure(6) set_thread_flag(6) sock_orphan(5)
inet_putpeer(5) dev_kfree_skb_any(5) clear_thread_flag(5)
inet_getid(3) sock_graft(2) igmp_mcf_get_first(2) icmp_xmit_lock(2)
csum_and_copy_to_user(2) test_and_clear_thread_flag(1)
tcp_v4_lookup(1) neigh_update_hhs(1) ip_finish_output2(1) gfp_any(1)
fn_flush_list(1) do_kdsk_ioctl(1) do_kdgkb_ioctl(1)
do_kbkeycode_ioctl(1) do_getname(1) alloc_buf(1) activate_task(1)

1764 get_current (1494 in *.c)
calls: current_thread_info(2682)
callers: <other>(417) capable(159) flush_tlb_page(11) flush_tlb_mm(10) do_trap(10) on_sig_stack(9) sas_ss_flags(8) find_process_by_pid(6) check_sticky(5) flush_tlb_range(4) walk_init_root(3) scm_send(3) current_is_pdflush(3) current_is_kswapd(3) rwsem_down_failed_common(2) kmem_freepages(2) get_sigframe(2) follow_dotdot(2) do_mmap2(2) do_follow_link(2) __vfs_follow_link(2) __exit_mm(2) scm_check_creds(1) save_i387_fsave(1) restore_i387_fsave(1) read_zero_pagealigned(1) handle_group_stop(1) good_sigevent(1) get_close_on_exec(1) fork_traceflag(1) ext2_init_acl(1) exec_permission_lite(1) dup_mmap(1) do_tty_write(1) de_thread(1) copy_signal(1) copy_sighand(1) copy_fs(1) cap_set_all(1) cap_netlink_send(1) cap_emulate_setxuid(1) arch_get_unmapped_area(1)

...

[top stack users]

$ make checkstack
objdump -d vmlinux | perl scripts/checkstack.pl i386
 1492 ide_unregister
 1428 huft_build
 1320 inflate_dynamic
 1168 inflate_fixed
  552 ip_setsockopt
  460 tcp_v4_conn_request
  456 tcp_check_req
  448 tcp_timewait_state_process
  440 ip_getsockopt
  368 extract_entropy
  320 do_execve
  288 icmp_send
...

Matt Mackall : http://www.selenic.com : Linux development and consulting

Hi Matt,

This looks very interesting. I could produce a small "bloated" kernel which I use for kexec and remote recovery on a VIA C3-based system :

text data bss dec hex filename
1328734 118955 52016 1499705 16e239 vmlinux

This was with gcc-3.3.1 which is more efficient than its predecessors on -Os.

I changed -march=c3 to -march=i386, which generally gives me better numbers :

text data bss dec hex filename
1287578 118955 52016 1458549 164175 vmlinux

=> this is 40 kB saved. I simply changed this :

@@ -44,7 +44,7 @@
 cflags-$(CONFIG_MWINCHIPC6)	+= $(call check_gcc,-march=winchip-c6,-march=i586)
 cflags-$(CONFIG_MWINCHIP2)	+= $(call check_gcc,-march=winchip2,-march=i586)
 cflags-$(CONFIG_MWINCHIP3D)	+= $(call check_gcc,-march=winchip2,-march=i586)
-cflags-$(CONFIG_MCYRIXIII)	+= $(call check_gcc,-march=c3,-march=i486) $(align)-functions=0 $(align)-jumps=0 $(align)-loops=0
+cflags-$(CONFIG_MCYRIXIII)	+= -march=i386 $(align)-functions=0 $(align)-jumps=0 $(align)-loops=0
 cflags-$(CONFIG_MVIAC3_2)	+= $(call check_gcc,-march=c3-2,-march=i686)
 
 CFLAGS += $(cflags-y)

So it might be interesting to have an option to drop back to the smallest common arch during compilation. Note that this is not the same as choosing i386 as the target system, since this kernel still has C3 features. Perhaps the simplest and most portable way would be to create a new config entry which would override default cflags ?

Cheers,
Willy

Ok, I added the ability to override the arch-default CFLAGS, but sadly I'm unable to reproduce your space savings here with gcc 3.3.2. The default -march=i586 seems to produce the smallest code for me.

 tiny-mpm/arch/i386/Makefile |    4 ++++
 tiny-mpm/init/Kconfig       |   14 ++++++++++++++
 2 files changed, 18 insertions(+)

diff -puN arch/i386/Makefile~tiny-cflags arch/i386/Makefile
@@ -26,6 +26,9 @@ CFLAGS += $(call check_gcc,-mpreferred-s
 
 align := $(subst -functions=0,,$(call check_gcc,-falign-functions=0,-malign-functions=0))
 
+ifdef CONFIG_TINY_CFLAGS
+cflags-y += $(CONFIG_TINY_CFLAGS_VAL)
+else
 cflags-$(CONFIG_M386)		+= -march=i386
 cflags-$(CONFIG_M486)		+= -march=i486
 cflags-$(CONFIG_M586)		+= -march=i586

@@ -46,6 +49,7 @@ cflags-$(CONFIG_MWINCHIP2)	+= $(call che
 cflags-$(CONFIG_MWINCHIP3D)	+= $(call check_gcc,-march=winchip2,-march=i586)
 cflags-$(CONFIG_MCYRIXIII)	+= $(call check_gcc,-march=c3,-march=i486) $(align)-functions=0 $(align)-jumps=0 $(align)-loops=0
 cflags-$(CONFIG_MVIAC3_2)	+= $(call check_gcc,-march=c3-2,-march=i686)
+endif
 
 CFLAGS += $(cflags-y)
 

diff -puN init/Kconfig~tiny-cflags init/Kconfig
@@ -446,6 +446,20 @@ config OOL_THREADINFO
 	  This simplifies the code for finding information about the current
           thread. Saves about 4k on small kernels.
 
+config TINY_CFLAGS
+	default n
+	bool "Set compiler arch flags for small 386 code" if EMBEDDED
+	help
+	  This allows user to replace the architecture CFLAGS despite the arch
+	  setting in an attempt to build smaller code.
+
+config TINY_CFLAGS_VAL
+	depends TINY_CFLAGS
+	default "-march=i386"
+	string "Arch CFLAGS"
+	help
+	  Enter CFLAGS to override ARCH defaults.
+
 endmenu		# General setup
 
 config SLAB

_

Matt Mackall : http://www.selenic.com : Linux development and consulting

Hi Matt,

Same results here. I was a bit stumped at first, but finally found the reason :
by changing CFLAGS, both you and me have implictly removed '-falign-functions=0', which is what grows up the kernel image ! I really don't understand what happens in the kernel, because on many other programs, i386 gives me smaller progs than i586 for -Os optimization, and -falign-functions=0 is *always* smaller. In the kernel, i386, i586 an c3 are identical, but -falign-functions=0 is bigger by 50 kB :

-Os -mpreferred-stack-boundary=2 -march=c3 -falign-functions=0 -falign-loops=0 -falign-jumps=0 :

text data bss dec hex filename
1306682 111227 50896 1468805 166985 vmlinux

-Os -mpreferred-stack-boundary=2 -march=c3 -falign-loops=0 -falign-jumps=0 :

text data bss dec hex filename
1256604 111227 50896 1418727 15a5e7 vmlinux

I have tested several values and found that the second compilation above is equivalent to the default gcc setting of -falign-functions=1. And using 1 instead of 0 in -falign-{loops,jumps,labels} makes not difference at all. So I would suggest adding -falign-functions=1 everywhere.

BTW, I slightly changed your patch to be able to specify several options. I did it the dirty way because I don't know how to split the string into several words :

@@ -27,7 +27,7 @@
 align := $(subst -functions=0,,$(call check_gcc,-falign-functions=0,-malign-functions=0))
 
 ifdef CONFIG_TINY_CFLAGS
-cflags-y += $(CONFIG_TINY_CFLAGS_VAL)
+cflags-y += $(shell echo $(CONFIG_TINY_CFLAGS_VAL))
 else
 cflags-$(CONFIG_M386)		+= -march=i386
 cflags-$(CONFIG_M486)		+= -march=i486

Other than that, I've compiled 2.6.1-rc1-tiny1. It's slightly smaller than 2.6.0-tiny1 here. But I had to keep CONFIG_INETPEER=y, CONFIG_DNOTIFY=y, CONFIG_PTRACE=y, and CONFIG_CPU_SUP_* otherwise it would not link. I can give you the error reports and .config in case you're interested.

Cheers,
Willy

Doh, I fixed that but it didn't make it into the release:

+ifdef CONFIG_TINY_CFLAGS
+cflags-y += $(shell echo $(CONFIG_TINY_CFLAGS_VAL) | sed -e 's/"//g') +else

Hmmm, works for me, error messages appreciated.

Matt Mackall : http://www.selenic.com : Linux development and consulting

Maybe wli will be interested in this one since he has some stack shrinking patches in his tree...

I'm already following this in general. I contributed a number of fixes I've done for the 4K stack code over time at the time mpm originally put -tiny together, though I think he's rearranged various things (e.g. re-split the thing into 3 pieces) I can't be arsed to deal with.

-- wli