2.5 merge candidate list, final version. (End of "crunch time" series.)

Hi Linus.

This list is the result of a week of scouring linux-kernel and posting more or less daily versions soliciting feedback from everybody seriously trying to get a patch into 2.5. This is the ninth and final posting of this list.

Previous versions, and the discussion they spawned, were here:
1.0) http://lists.insecure.org/lists/linux-kernel/2002/Oct/7006.html
1.1) http://lists.insecure.org/lists/linux-kernel/2002/Oct/7051.html
1.2) http://lists.insecure.org/lists/linux-kernel/2002/Oct/7363.html
1.3) http://lists.insecure.org/lists/linux-kernel/2002/Oct/7452.html
1.4) http://lists.insecure.org/lists/linux-kernel/2002/Oct/7827.html
1.5) http://lists.insecure.org/lists/linux-kernel/2002/Oct/8174.html
1.6) http://lists.insecure.org/lists/linux-kernel/2002/Oct/8787.html
1.7) http://lists.insecure.org/lists/linux-kernel/2002/Oct/9130.html

The vast majority of the 30 items in this list will probably be dropped, but if this list results in explicit rejections instead of patches getting missed or lost in the shuffle, it will have served its purpose.

On a side note, Dave Jones has started a 2.5 series development summary:
http://www.codemonkey.org.uk/post-halloween-2.5.txt

================================= Intro ====================================

The following features aim to be ready for submission to Linus by Monday, October 28th, to be considered for inclusion (in 2.5.45) before the feature freeze on Thursday, October 31 (halloween).

This list is just pending features trying to get in before feature freeze. It's primarily for features that need more testing, or might otherwise get forgotten in the rush. If you want to know what's already gone in, or what's being worked on for the next development cycle, check out:
http://kernelnewbies.org/status

Thanks to Rusty Russell and Guillaume Boissiere, whose respective 2.5 merge candidate lists have been ruthlessly strip-mined in the process of assembling this. And to everybody who's emailed stuff.

============================ Pending features: =============================

1) Andrew Morton's -mm tree. (Andrew Morton, editor.)

Andrew Morton's -mm tree collates several other projects, including:

The ext2/ext3 Extended Attributes and Access Control Lists patch from Ted Tso and Andreas Gruenbacher (ext23-*.patch), Page Table Sharing from Daniel Phillips and Dave McCracken (shpte-ng.patch), a bunch of huge page upgrades from Bill Irwin (hugetlb*.patch), the orlov allocator, Ingo's generic nonlinear mappings...

Stuff. Lots of stuff.

You can get Andrew Morton's MM tree from the following URL, including a broken-out patches directory and a description file. (The latest version as of this writing is -mm6.)

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.44

----------------------------------------------------------------------------

2) Device mapper for Logical Volume Manager (LVM2) (LVM2 team) (in -ac)

Announce:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103536883428443&w=2

Download:
http://people.sistina.com/~thornber/patches/2.5-stable/

Home page:
http://www.sistina.com/products_lvm.htm

Note: this is in the 2.5-ac tree, available at:
http://www.kernel.org/pub/linux/kernel/people/alan/

----------------------------------------------------------------------------

3) EVMS (Enterprise Volume Management System) (IBM, Contact: Kevin Corry)

Fighting with LVM2 for a place in the tree, a bigger solution to a bigger set of problems:

Home page:
http://sourceforge.net/projects/evms

Home page:
http://evms.sourceforge.net

Download:
http://evms.sourceforge.net/patches/

Some related discussions:
http://marc.theaimsgroup.com/?t=103359686900003&r=1&w=2
http://marc.theaimsgroup.com/?t=103439913000001&r=1&w=2
http://marc.theaimsgroup.com/?w=2&r=1&s=%5Bpatch%5D+evms+core&q=t

----------------------------------------------------------------------------

4) New kernel configuration system (Roman Zippel)

Announcement:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/9043.html

Download:
http://www.xs4all.nl/~zippel/lc/

Linus has actually looked fairly favorably on this one so far:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/3250.html

And an AOL for it:

http://lists.insecure.org/lists/linux-kernel/2002/Oct/8255.html

----------------------------------------------------------------------------

5) Linux Trace Toolkit (LTT) (Karim Yaghmour)

Announce:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/7016.html

Download:
http://opersys.com/ftp/pub/LTT/ExtraPatches/patch-ltt-linux-2.5.44-vanilla-021026-2.2.bz2

User tools:
http://opersys.com/ftp/pub/LTT/TraceToolkit-0.9.6pre2.tgz

----------------------------------------------------------------------------

6) Kernel Probes (IBM, contact: Vamsi Krishna S)

Kprobes announcement:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103528410215211&w=2

Download Base Kprobes Patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103528425615302&q=raw

KProbes->DProbes support patches:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103528454215523&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103528454015520&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103528485415813&q=raw

Download gzipped tarball patches from official IBM site:
http://www-124.ibm.com/linux/patches/?project_id=141

KProbes home page:
http://www-124.ibm.com/linux/projects/kprobes

A good explanation of the difference between kprobes, dprobes, and kernel hooks:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103532874900445&w=2

Clarification: just kprobes is being submitted for 2.5.45, (and optionally some basic dprobes support,) but not the whole of dprobes:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103536827928012&w=2
And also:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/9160.html

----------------------------------------------------------------------------

7) Posix support and High Resolution timers (George Anzinger)

George has a patch that provides posix support, and on top of that a patch to provide high resolution timers. He talks about it here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103562196700924&w=2

The project's home page is here:
http://high-res-timers.sourceforge.net/

Downloadable patches may be found here:
http://sourceforge.net/project/showfiles.php?group_id=20460&release_id=118345

And descriptions for each patch (in the order they should be applied) are available here (although in this case the archive truncates the patches themselves, use the download link above):
http://marc.theaimsgroup.com/?l=linux-kernel&m=103553654329827&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=103557676007653&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=103557677207693&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=103558349714128&w=2

Linus had concerns with this one a while back:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/3463.html

----------------------------------------------------------------------------

8) Posix timers alternate implementation (Jim Houston)

Jim Houston has an alternate patch to provide posix support (but not high resolution timers on top of it, yet).

http://marc.theaimsgroup.com/?l=linux-kernel&m=103549000027416&q=raw

----------------------------------------------------------------------------

9) Linux Kernel Crash Dumps (Matt Robinson, LKCD team)

Announce:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103553563728914&w=2
And again:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103536576625905&w=2

Download:
http://lkcd.sourceforge.net/download/latest/

----------------------------------------------------------------------------

10) Rewrite of the console layer (James Simmons)

Announcement:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103487329526903&w=2

Home page:
http://linuxconsole.sourceforge.net/

Downloadable patch:
http://phoenix.infradead.org/~jsimmons/fbdev.diff.gz

Bitkeeper tree:
bk://fbdev.bkbits.net/fbdev-2.5

----------------------------------------------------------------------------

11) IPv6 upgrades and crypto API (Yoshifuji Hideyaki)

The Usagi ipv6 upgrades have been available for a while, and their author would like to see them in 2.5:

README:
ftp://ftp.linux-ipv6.org/pub/usagi/patch/ipsec/README.IPSEC

Downloadable patch here:
ftp://ftp.linux-ipv6.org/pub/usagi/patch/ipsec/ipsec-2.5.43-ALL-03.patch.gz

Dave Miller is doing a major ipv6 layer rewrite, but no patch has been sent to the list yet. Ironically, James Morris has a Crypto API patch on top of Dave Miller's tree:

Announce:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103559983324080&w=2

Download:
http://samba.org/~jamesm/crypto/

----------------------------------------------------------------------------

12) MMU-less processor support (Greg Ungerer)

Most recent announcement (with links):
http://marc.theaimsgroup.com/?l=linux-kernel&m=103578189421588&w=2

A version of this is in the 2.5-ac tree. An announcement of a patch against that is here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103578338922236&w=2

----------------------------------------------------------------------------

13) sys_epoll (I.E. /dev/poll) (Davide Libenzi)

Announce:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103542994232004&w=2

Homepage:
http://www.xmailserver.org/linux-patches/nio-improve.html

Download:
http://www.xmailserver.org/linux-patches/sys_epoll-2.5.44-last.diff

Linus participated repeatedly in a thread on this one too, expressing concerns which (hopefully) have been addressed. See:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/6428.html

----------------------------------------------------------------------------

14) CD Recording/sgio patches (Jens Axboe)

Announce:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/8060.html

Download patch:
http://www.kernel.org/pub/linux/kernel/people/axboe/patches/v2.5/2.5.44/sgio-15.bz2

This should be in Alan Cox's tree as of 2.4.44-ac4.

----------------------------------------------------------------------------

15) In-kernel module loader (Rusty Russell.)

Announce:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/6214.html

Download patch:
http://www.kernel.org/pub/linux/kernel/people/rusty/patches/module-x86-18-10-2002.2.5.43.diff.gz

----------------------------------------------------------------------------

16) Unlimited groups patch (Tim Hockin.)

Announce:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103524761319825&w=2

Download patch set from:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103524717119443&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103524761819834&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103524761619831&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103524761519829&q=raw

----------------------------------------------------------------------------

17) Initramfs (Al Viro)

Way back when, Al said:
http://www.cs.helsinki.fi/linux/linux-kernel/2001-30/0110.html

Download (most recent patch so far):
ftp://ftp.math.psu.edu/pub/viro/N0-initramfs-C40

And Linus recently made happy noises about the idea:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/1110.html

----------------------------------------------------------------------------

18) Kernel Hooks (IBM contact: Vamsi Krishna S.)

Website:
http://www-124.ibm.com/linux/projects/kernelhooks/

Download site:
http://www-124.ibm.com/linux/patches/?patch_id=595

Posted patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103364774926440&w=2

----------------------------------------------------------------------------

19) NMI request/release interface (Corey Minyard)

He says:

...

There was a lot of back and forth, resulting in the latest patch (version 8):
http://home.attbi.com/~minyard/linux-nmi-v8.diff

----------------------------------------------------------------------------

20) DriverFS Topology (Matthew Dobson)

Announcement:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103523702710396&w=2

Patches:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103540707113401&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103540757613962&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103540758013984&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103540757513957&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103540757813966&q=raw

----------------------------------------------------------------------------

21) Advanced TCA Disk Hotswap (Steven Dake)

Announcement of most recent patch, with links:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103558466315221&w=2

Steven's comments:

----------------------------------------------------------------------------

22) Mobile IPV6 (contact: Antti Tuominen)

Antti Tuominen says:

----------------------------------------------------------------------------

23) SCSI multi-path I/O (Patrick Mansfield)

Announcement with URLs:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/8736.html

) VFS Intent Lookup Patch (Peter Braam)

Download:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103552797823568&q=raw

----------------------------------------------------------------------------

24) NUMA Scheduler Upgrade

Erich Focht and Michael Hohnbaum have two different NUMA scheduler patches.

Michael has a stripped down NUMA scheduler, which he says was created because the full Node Affine NUMA Scheduler didn't look like it would be ready for 2.5. He talks about it here, with links to patches:

http://marc.theaimsgroup.com/?l=linux-kernel&m=103548635122591&w=2

Meanwhile, Erich Focht says the full Node Affine Numa Scheduler is indeed ready for 2.5, and already in use at customer sites. He makes his case here, with links to patches, home page, LWN review, etc:

http://marc.theaimsgroup.com/?l=linux-kernel&m=103549657202782&w=2

Here's Erich's scheduler's home page:
http://home.arcor.de/efocht/sched/

The most current version of the patches are downloadable from:

http://home.arcor.de/efocht/sched/01-numa_sched_core-2.5.44-10a.patch
http://home.arcor.de/efocht/sched/02-numa_sched_ilb-2.5.44-10.patch

Martin J. Bligh has been testing both NUMA schedulers, and wandering back and forth in his endorsement. At first he was leaning towards Erich's patch, now he seems to be leaning towards Michael's.

That thread is here:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/8904.html

----------------------------------------------------------------------------

25) ptrace over fork (Daniel Jacobowitz)

At the last possible second, this was submitted:

http://marc.theaimsgroup.com/?l=linux-kernel&m=103574480632057&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103574480232051&q=raw
http://marc.theaimsgroup.com/?l=linux-kernel&m=103574511932225&q=raw

----------------------------------------------------------------------------

26) Kexec, launch new linux kernel from Linux (Eric W. Biederman)

Announcement, description, links, and patch. All in one:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103579334328023&q=raw

P.S. This thread is just too brazen not to include:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/7952.html

----------------------------------------------------------------------------

27) Nanosecond support in stat.

Discussion thread:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/8983.html

Download:
ftp://ftp.firstfloor.org/pub/ak/v2.5/nsec-2.5.44-2.bz2

----------------------------------------------------------------------------

28) Digital Video Broadcasting Layer (LinuxTV team)

Home page:
http://www.linuxtv.org:81/dvb/

Download:
http://www.linuxtv.org:81/download/dvb/

This is also in Alan Cox's 2.5 tree.

----------------------------------------------------------------------------

29) Hotplug CPU Removal (Rusty Russell)

On sunday, Rusty announced the submission of hotplug against 2.5.44:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/9141.html

(Unfortunately, he notes the patch conflicts with the -mm5 tree.)

Download:
http://www.kernel.org/pub/linux/kernel/people/rusty/patches/Hotcpu/hotcpu-cpudown.patch.gz

----------------------------------------------------------------------------

30) Reiser4.

I don't have a patch yet, but Hans Reiser is very insistent that this will be ready by halloween. (VERY insistent.) I'll let him speak for himself:
http://lists.insecure.org/lists/linux-kernel/2002/Oct/8793.html

And again (promises, promises):
http://lists.insecure.org/lists/linux-kernel/2002/Oct/9082.html

Still no patch at the time of this writing, though. In theory it should show up here:
http://namesys.com/download.html

Or perhaps here:
ftp://ftp.lugoj.org/pub/reiserfs/devlinux.com/pub/namesys/reiserfs-for-2.5

In the meantime, all I can find on Reiser4 is some kind of hybrid marketing brochure/design document thing:
http://www.reiserfs.org/v4/v4.html

Did I mention Hans was insistent? The man can make puppy eyes through email. It's quite impressive.

http://penguicon.sf.net - Terry Pratchett, Eric Raymond, Pete Abrams, Illiad, 
CmdrTaco, liquid nitrogen ice cream, and caffienated jello.  Well why not?

[...]

Snapshot is available at http://www.namesys.com/snapshots/:

reiser4 proper:
http://www.namesys.com/snapshots/reiser4-2002.10.24.tar.gz

necessary changes to the core kernel, plus some UML patches, plus some patches for debugging:
http://www.namesys.com/snapshots/reiser4-core-2002.10.24.diff

utils:
http://www.namesys.com/snapshots/reiser4progs-2002.10.24.tar.gz

I shall produce newer snapshot to-morrow.

It does work, but code is not production quality yet. Do *NOT* put anything close to critical data on it.

Works without email too. :-)

Nikita.

2.5.44:
Producing a module I get a couple unresolved symbols, and trying to build directly into the kernel results in

...
ld -m elf_i386 -e stext -T arch/i386/vmlinux.lds.s arch/i386/kernel/head.o arch/i386/kernel/init_task.o init/built-in.o --start-group arch/i386/kernel/built-in.o arch/i386/mm/built-in.o arch/i386/mach-generic/built-in.o kernel/built-in.o mm/built-in.o fs/built-in.o ipc/built-in.o security/built-in.o lib/lib.a arch/i386/lib/lib.a drivers/built-in.o sound/built-in.o arch/i386/pci/built-in.o net/built-in.o --end-group -o vmlinux fs/built-in.o(.init.text+0x1381): In function `init_reiser4':
: undefined reference to `local symbols in discarded section .exit.text'
make[1]: *** [vmlinux] Error 1
make: *** [vmlinux] Error 2

... is missing the "configure" script.
Generating one with autoconf 2.53 doesn't seem to work, either.

T.

We are going to submit a patch appropriate for inclusion as an experimental FS on Halloween. I hope you will forgive our pushing the limit timewise, it is not by choice, but the algorithms we used to more than double reiserfs V3 performance were, quite frankly, hard to code.

A description of it can be found at www.namesys.com/v4/fast_reiser4.html.

I am going to ask persons to volunteer to produce independent benchmarks. I am told by Oleg and Zam that it is now faster than in the benchmarks in the above draft paper, and I hope to see this verified by persons outside Namesys.

I hope you will also find our atomic transactions infrastructure interesting. It allows us to guarantee each system call to reiserfs is fully atomic, and in later versions we hope to use it to make atomicity available to user space so that multiple fs operations can be guaranteed atomic. This will hopefully help to eliminate a major class of security holes, as well as provide significantly better protection against user data corruption by crashes than any current filesystem. As you can see, it does not seem to substantially harm performance, or at least we assume that our more than doubling performance means that performance was not harmed.;-)

Finally, and I think in the longterm most importantly, we have implemented a full plugin infrastructure for reiser4. This will make it possible for us to implement database and search engine functionality one plugin at a time. (See www.namesys.com/whitepaper.html for a description of those semantics.) We hope you will find the semantics much more fun than tacking SQL onto the FS like OFS is going to do.;-)

We don't know how fast OFS will be, but we hope you will enjoy watching us raise the bar a few feet. We are very excited by more than doubling performance. The read performance improvements are more than I expected.

(If any of you attempt to replicate the benchmarks, please be sure to use reiser4 readdir order for writes to reiser4 (that means don't use tarballs made from ext2), and to use the latest hard drives and fast processors with udma 5 turned on. We are quite sensitive to transfer speed since we do a good job of avoiding seeks. We are sensitive to readdir order because we sort directory entries (which is necessary for having efficient large directory lookups)). In reiser4.1 we will ship a repacker, and then it won't matter what order you do writes in so long as the repacker gets a chance to run at night.

Cheers,

Hans

Hans

Does your merge change the core code at all? Does it add new syscalls?

Their changes are tiny, and sensible. See
http://www.namesys.com/snapshots/2002.10.29/

But I'd like to ask about the status of reiser3 support.

Chris had patches *ages* ago to convert it to use direct-to-BIO for reads, and writes should be done as well. reiserfs3 is still using buffer-head-based IO for bulk reads and writes. That's a 25-30% hit in CPU cost, and all the old ZONE_NORMAL-full-of-buffer_heads problems.

Any plans to get that work finished off?

Are you talking about buffer_heads_over_limit magic inside vmscan.c? Is it to avoid deadlock with buffer_head allocation in ->vm_writeback?

Chris had done great work indeed (plus data logging, plus quotas, plus ACLs by Jeff Mahoney, etc). Unfortunately we were too busy with reiser4 here lately. Hopefully, a lot of stuff will be merged into reiserfs3.x after Halloween.

Nikita.

Very much so, but I wanted it on top of the data journaling stuff to make porting that easier. But, in the interest in getting something more stable and faster out the door asap, I'll pull just the direct-to-BIO stuff out.

-chris

Here is a list of things reiser4 needs from the core:

(1) export of generic_forget_inode()

(2) export of page_cache_readahead()

(3) export of remove_from_page_cache()

(4) export of fsync_super()

(5) patch to allow safe sharing of ->journal_info pointer in task_struct by several users

(1)-(4) are only necessary to compile reiser4 as module.

(5) is good not only for reiser4. Without it different file systems using ->journal_info cannot "co-exist peacefully".

No new system calls are introduced (yet).

Nikita.