2.5.65-mjb1 (scalability / NUMA patchset)

The patchset contains mainly scalability and NUMA stuff, and anything else that stops things from irritating me. It's meant to be pretty stable, not so much a testing ground for new stuff.

I'd be very interested in feedback from anyone willing to test on any platform, however large or small.

ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/2.5.65/patch-2.5.65-mjb1.bz2

additional:

http://www.aracnet.com/~fletch/linux/2.5.59/pidmaps_nodepages

Since 2.5.64-mjb5 (~ = changed, + = added, - = dropped)

Notes: just a sync with 2.5.65 (dispose of -bk snapshot). Very, very boring.

Now in Linus' tree:

New:

Pending:
Hyperthreaded scheduler (Ingo Molnar)
objrmap bugfixes for nonlinear vma's (Dave McCracken) Seperate kernel PMDs per process (Dave Hansen) Non-PAE aligned kernel splits (Dave Hansen) scheduler callers profiling (Anton or Bill Hartner) PPC64 NUMA patches (Anton)
Child runs first (akpm)
Kexec
e1000 fixes
Update the lost timer ticks code

Present in this patch:

doaction Martin J. Bligh
Fix cruel torture of macros and small furry animals in io_apic.c

early_printk Dave Hansen et al.
Allow printk before console_init

confighz Andrew Morton / Dave Hansen
Make HZ a config option of 100 Hz or 1000 Hz

config_page_offset Dave Hansen / Andrea
Make PAGE_OFFSET a config option

vmalloc_stats Dave Hansen
Expose useful vmalloc statistics

numameminfo Martin Bligh / Keith Mannthey
Expose NUMA meminfo information under /proc/meminfo.numa

schedstat Rick Lindsley
Provide stats about the scheduler under /proc/schedstat

schedstat2 Rick Lindsley
Provide more stats about the scheduler under /proc/schedstat

schedstat-scripts Rick Lindsley
Provide some scripts for schedstat analysis under scripts/

sched_tunables Robert Love
Provide tunable parameters for the scheduler (+ NUMA scheduler)

irq_affinity Martin J. Bligh
Workaround for irq_affinity on clustered apic mode systems (eg x440)

cleaner_inodes Andrew Morton
Make noatime filesystems more efficient

partial_objrmap Dave McCracken
Object based rmap for filebacked pages.

objrmap_fix Dave McCracken
Fix detection of anon pages

objrmap_fixes Dave McCracken / Hugh Dickins
Fix up some mapped sizing bugs in objrmap

objrmap_mapcount Dave McCracken
Fix up some mapped sizing bugs in objrmap

kgdb Andrew Morton / Various People
The older version of kgdb, synched with 2.5.54-mm1

kprobes Vamsi Krishna S
Add kernel probes hooks to the kernel

thread_info_cleanup (4K stacks pt 1) Dave Hansen / Ben LaHaise
Prep work to reduce kernel stacks to 4K

interrupt_stacks (4K stacks pt 2) Dave Hansen / Ben LaHaise
Create a per-cpu interrupt stack.

stack_usage_check (4K stacks pt 3) Dave Hansen / Ben LaHaise
Check for kernel stack overflows.

4k_stack (4K stacks pt 4) Dave Hansen
Config option to reduce kernel stacks to 4K

fix_kgdb Dave Hansen
Fix interaction between kgdb and 4K stacks

stacks_from_slab William Lee Irwin
Take kernel stacks from the slab cache, not page allocation.

thread_under_page William Lee Irwin
Fix THREAD_SIZE < PAGE_SIZE case

lkcd LKCD team
Linux kernel crash dump support

percpu_loadavg Martin J. Bligh
Provide per-cpu loadaverages, and real load averages

get_empty_filp Manfred Spraul
Kill the lock contention on files_lock from get_empty_filp ...

files_lock_goodness Andrew Morton
... and drive a silver stake through it's heart.

spinlock_inlining Andrew Morton
Inline spinlocks for profiling. Made into a ugly config option by me.

summit_pcimap Matt Dobson

	Provide pci bus -> node mapping for x440

shpte Dave McCracken
Shared pagetables

reiserfs_dio Mingming Cao
DIO for Reiserfs

concurrent_balloc Alex Tomas
Concurrent ext2 block allocation - makes SDET & dbench go whizzy fast.

concurrent_inode Alex Tomas
Concurrent ext2 inode allocation - makes SDET & dbench go whizzy fast.

debkl_ext2_readdir Alex Tomas
Don't take the BKL in ext2_readdir

sched_interactive Ingo Molnar
Bugfix for interactive scheduler

kgdb_cleanup Martin J. Bligh
Stop kgdb renaming schedule to do_schedule when it's not even enabled

numa_protector Martin J. Bligh / Dave Hansen
Stop people shooting themselves in the foot with CONFIG_NUMA

-mjb Martin J. Bligh
Add a tag to the makefile

kernel.org seems to be very sleepy right now:

http://www.aracnet.com/~fletch/linux/2.5.65/patch-2.5.65-mjb1.bz2

I got the following compile error for !CONFIG_SMP:

<-- snip -->

...
gcc-2.95 -Wp,-MD,drivers/dump/.dump_fmt.o.d -D__KERNEL__ -Iinclude
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=k6 -Iinclude/asm-i386/mach-default -fomit-frame-pointer -nostdinc
-iwithprefix include -DKBUILD_BASENAME=dump_fmt -DKBUILD_MODNAME=dump
-c -o drivers/dump/dump_fmt.o drivers/dump/dump_fmt.c drivers/dump/dump_fmt.c:171: macro `__dump_save_other_cpus' used without args make[2]: *** [drivers/dump/dump_fmt.o] Error 1

<-- snip -->

Trivial fix:

@@ -343,7 +343,7 @@
 #ifdef CONFIG_SMP
 extern void 	__dump_save_other_cpus(void);
 #else
-#define 	__dump_save_other_cpus(void)
+#define 	__dump_save_other_cpus()
 #endif
 
 #endif /* __KERNEL__ */

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Errrm ... oops. Sorry about that. Thanks for all the testing, and to you, Sam, and Christoph for suggested fixes. I've rolled up these, and a couple of other issues I found below ... could you try this? Compiles for NUMA-Q and UP at least. Any feedback on the fixes welcome too ...

Thanks,

M.

diff -urpN -X /home/fletch/.diff.exclude 900-mjb1/drivers/dump/Makefile 901-lkcd_fixups/drivers/dump/Makefile
@@ -1,7 +1,6 @@
 #
 # Makefile for the dump device drivers.
 #
-export-objs				:= dump_setup.o
 
 dump-y					:= dump_setup.o dump_fmt.o dump_filters.o dump_scheme.o dump_execute.o
 dump-$(CONFIG_X86)			+= dump_i386.o

diff -urpN -X /home/fletch/.diff.exclude 900-mjb1/drivers/dump/dump_netdev.c 901-lkcd_fixups/drivers/dump/dump_netdev.c
@@ -556,9 +556,9 @@ do_netdump(struct dump_dev *net_dev, con
 		case COMM_GET_NR_PAGES:
 			reply.code = REPLY_NR_PAGES;
 			reply.nr = req.nr;
-			reply.info = max_mapnr;
+			reply.info = num_physpages;
                         reply.info = page_counter;
-			sprintf(tmp, "Number of pages: %ld\n", max_mapnr);
+			sprintf(tmp, "Number of pages: %ld\n", num_physpages);
 			dump_send_skb(dump_ndev, tmp, strlen(tmp), &reply);
 			break;
 

diff -urpN -X /home/fletch/.diff.exclude 900-mjb1/drivers/dump/dump_scheme.c 901-lkcd_fixups/drivers/dump/dump_scheme.c
@@ -292,7 +292,7 @@ int dump_generic_configure(unsigned long
 	for (filter = dump_config.dumper->filter ;filter->selector; filter++) {
 		if (!filter->start && !filter->end) {
 			filter->start = 0;
-			filter->end = max_mapnr << PAGE_SHIFT;
+			filter->end = num_physpages << PAGE_SHIFT;
 		}
 	}
 

diff -urpN -X /home/fletch/.diff.exclude 900-mjb1/include/asm-i386/pgtable.h 901-lkcd_fixups/include/asm-i386/pgtable.h
@@ -15,6 +15,7 @@
 #ifndef __ASSEMBLY__
 #include <asm/processor.h>
 #include <asm/fixmap.h>
+#include <asm/system.h>
 #include <linux/threads.h>
 
 #ifndef _I386_BITOPS_H

diff -urpN -X /home/fletch/.diff.exclude 900-mjb1/include/linux/dump.h 901-lkcd_fixups/include/linux/dump.h
@@ -343,7 +343,7 @@ extern int	__dump_page_valid(unsigned lo
 #ifdef CONFIG_SMP
 extern void 	__dump_save_other_cpus(void);
 #else
-#define 	__dump_save_other_cpus(void)
+#define 	__dump_save_other_cpus()
 #endif
 
 #endif /* __KERNEL__ */

diff -urpN -X /home/fletch/.diff.exclude 900-mjb1/init/Makefile 901-lkcd_fixups/init/Makefile
@@ -1,16 +1,15 @@
 #
 # Makefile for the linux kernel.
 #
-ifdef CONFIG_CRASH_DUMP
-EXTRA_TARGETS := kerntypes.o
-CFLAGS_kerntypes.o := -gstabs
-endif
 
 obj-y				:= main.o version.o mounts.o initramfs.o
 mounts-y			:= do_mounts.o
 mounts-$(CONFIG_DEVFS_FS)	+= do_mounts_devfs.o
 mounts-$(CONFIG_BLK_DEV_RAM)	+= do_mounts_rd.o
 mounts-$(CONFIG_BLK_DEV_MD)	+= do_mounts_md.o
+
+extra-$(CONFIG_CRASH_DUMP)	+= kerntypes.o
+CFLAGS_kerntypes.o		:= -gstabs
 
 # files to be removed upon make clean
 clean-files := ../include/linux/compile.h

<-- snip -->

...
gcc-2.95 -Wp,-MD,drivers/dump/.dump_filters.o.d -D__KERNEL__ -Iinclude
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=k6 -Iinclude/asm-i386/mach-default -fomit-frame-pointer -nostdinc
-iwithprefix include -DKBUILD_BASENAME=dump_filters
-DKBUILD_MODNAME=dump -c -o drivers/dump/dump_filters.o drivers/dump/dump_filters.c
In file included from include/asm/pgtable.h:43,
from include/linux/bootmem.h:7,
from drivers/dump/dump_filters.c:24:
include/asm/pgtable-3level.h: In function `set_pte':
include/asm/pgtable-3level.h:49: warning: implicit declaration of function `smp_wmb'
include/asm/pgtable-3level.h: In function `ptep_get_and_clear':
include/asm/pgtable-3level.h:79: warning: implicit declaration of function `xchg'
...

<-- snip -->

I don't know the correct fix, my solution was:

@@ -21,6 +21,7 @@
  */
 
 #include <linux/kernel.h>
+#include <asm/system.h>
 #include <linux/bootmem.h>
 #include <linux/mm.h>
 #include <linux/slab.h>

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

EXTRA_TARGETS is obsolete in 2.5.

The following should do the same:

@@ -1,10 +1,6 @@
 #
 # Makefile for the linux kernel.
 #
-ifdef CONFIG_CRASH_DUMP
-EXTRA_TARGETS := kerntypes.o
-CFLAGS_kerntypes.o := -gstabs
-endif
 
 obj-y				:= main.o version.o mounts.o initramfs.o
 mounts-y			:= do_mounts.o

@@ -12,6 +8,9 @@
 mounts-$(CONFIG_BLK_DEV_RAM)	+= do_mounts_rd.o
 mounts-$(CONFIG_BLK_DEV_MD)	+= do_mounts_md.o
 
+obj-$(CONFIG_CRASH_DUMP)	+= kerntypes.o
+CFLAGS_kerntypes.o		:= -gstabs
+
 # files to be removed upon make clean
 clean-files := ../include/linux/compile.h
 

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

No, kerntypes.o should not be linked into the kernel image.

Ups, sorry, my bad.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

As Andrew pointed out this is wrong.
Use the following notation:
+extra-$(CONFIG_CRASH_DUMP) += kerntypes.o

This way of selecting extra .o files is the reason to have the "-y" in extra-y.

Sam

When you have the final patch done, send it along so I can incorporate it into our main CVS tree. We went down this kerntypes.o issue a while ago when trying to get into the tree the first time. Thanks.

--Matt