May 1, 2002

This week's status update is available at the usual place:
http://kernelnewbies.org/status/

A couple items have been merged since last week, including the new NTFS driver, the fast walk dcache patch, the start of the new framebuffer layer, as well as some more delayed disk block allocation bits.

There has also been a lot of work done by various people to remove the BKL from many places, which leads to my question:
does anyone has a URL with a list of all the places where the BKL should eventually be removed and who's working on it?

It seems like it would be most useful if someone was willing to maintain something like this, but it might be a lot of work - I don't know how long the list would be...

Anyway, as usual, if I missed anything, let me know! Enjoy,

-- Guillaume

--------------------------------------------------
Kernel 2.5 status - May 1st, 2002
(Latest kernel release is 2.5.12)

Features:

Merged

o in 2.5.1+   Rewrite of the block IO (bio) layer             (Jens Axboe)
o in 2.5.2    Initial support for USB 2.0                     (David 
Brownell, Greg Kroah-Hartman, etc.)
o in 2.5.2    Per-process namespaces, late-boot cleanups      (Al Viro, 
Manfred Spraul)
o in 2.5.2+   New scheduler for improved scalability          (Ingo Molnar)
o in 2.5.2+   New kernel device structure (kdev_t)            (Linus 

Torvalds, etc.)
o in 2.5.3 IDE layer update (Andre
Hedrick)
o in 2.5.3 Support reiserfs external journal (Reiserfs
team)

o in 2.5.3    Generic ACL (Access Control List) support       (Nathan Scott)
o in 2.5.3    PnP BIOS driver                                 (Alan Cox, 

Thomas Hood, Dave Jones, etc.)
o in 2.5.3+ New driver model & unified device tree (Patrick
Mochel)
o in 2.5.4 Add preempt kernel option (Robert Love,
MontaVista team)

o in 2.5.4    Support for Next Generation POSIX Threading     (NGPT team)
o in 2.5.4+   Porting all input devices over to input API     (Vojtech 
Pavlik, James Simmons)
o in 2.5.5    Add ALSA (Advanced Linux Sound Architecture)    (ALSA team)
o in 2.5.5    Pagetables in highmem support                   (Ingo Molnar, 

Arjan van de Ven)
o in 2.5.5 New architecture: AMD 64-bit (x86-64) (Andi Kleen,
x86-64 Linux team)
o in 2.5.5 New architecture: PowerPC 64-bit (ppc64) (Anton
Blanchard, ppc64 team)
o in 2.5.5+ IDE subsystem major cleanup (Martin
Dalecki, Vojtech Pavlik)

o in 2.5.6    Add JFS (Journaling FileSystem from IBM)        (JFS team)
o in 2.5.6    per_cpu infrastructure                          (Rusty 

Russell)
o in 2.5.6 HDLC (High-level Data Link Control) update (Krzysztof
Halasa)
o in 2.5.6 smbfs Unicode and large file support (Urban
Widmark)
o in 2.5.7 New driver API for Wireless Extensions (Jean
Tourrilhes)
o in 2.5.7 Video for Linux (V4L) redesign (Gerd Knorr)
o in 2.5.7 Futexes (Fast Lightweight Userspace Semaphores) (Rusty
Russell, etc.)
o in 2.5.7+ NAPI network interrupt mitigation (Jamal Hadi
Salim, Robert Olsson, Alexey Kuznetsov)
o in 2.5.7+ ACPI (Advanced Configuration & Power Interface) (Andy Grover,
ACPI team)

o in 2.5.8    Syscall interface for CPU task affinity         (Robert Love)
o in 2.5.8    Radix-tree pagecache                            (Momchil 
Velikov, Christoph Hellwig)
o in 2.5.8+   Delayed disk block allocation                   (Andrew 
Morton)
o in 2.5.9    Smarter IRQ balancing                           (Ingo Molnar)
o in 2.5.11   Replace old NTFS driver with NTFS TNG driver    (Anton 
Altaparmakov)
o in 2.5.11   Fast walk dcache                                (Hanna Linder)
o in 2.5.11+  Rewrite of the framebuffer layer                (James 

Simmons)

o in -dj Rewrite of the console layer (James
Simmons)

o in -ac      Strict address space accounting                 (Alan Cox)
o in -ac      PCMCIA Zoom video support                       (Alan Cox)
o in -ac      More complete IEEE 802.2 stack                  (Arnaldo, Jay 

Schullist, from Procom donated code)

o Ready Better event logging for enterprise systems (Larry
Kessler, evlog team)

o Ready       New quota system supporting plugins             (Jan Kara)
o Ready       Linux booting ELF images                        (Eric 

Biederman)
o Ready First pass at LinuxBIOS support (Eric
Biederman)
o Ready Build option for Linux Trace Toolkit (LTT) (Karim
Yaghmour)

o Beta        New kernel build system (kbuild 2.5)            (Keith Owens)
o Beta        Add support for CPU clock/voltage scaling       (Erik Mouw, 
Dave Jones, Russell King, Arjan van de Ven)
o Beta        Serial driver restructure                       (Russell King)
o Beta        New IO scheduler                                (Jens Axboe)
o Beta        Add XFS (A journaling filesystem from SGI)      (XFS team)
o Beta        New VM with reverse mappings                    (Rik van Riel)
o Beta        Fix long-held locks for low scheduling latency  (Andrew 
Morton, Robert Love, etc.)
o Beta        Add Linux Security Module (LSM)                 (LSM team)
o Beta        Hotplug CPU support                             (Rusty 
Russell)
o Beta        Per-mountpoint read-only, union-mounts, unionfs (Al Viro)
o Beta        EVMS (Enterprise Volume Management System)      (EVMS team)
o Beta        LVM (Logical Volume Manager) v2.0               (LVM team)
o Beta        Dynamic Probes                                  (Suparna 
Bhattacharya, dprobes team)
o Beta        Scalable CPU bitmasks                           (Russ Weight)
o Beta        Page table sharing                              (Daniel 
Phillips)
o Beta        ext2/ext3 online resize support                 (Andreas 
Dilger)
o Beta        Add User-Mode Linux (UML)                       (Jeff Dike)
o Beta        UDF Write support for CD-R/RW (packet writing)  (Jens Axboe, 
Peter Osterlund)
o Beta        Add hardware sensors drivers                    (lm_sensors 
team)
o Beta        New kernel config system: CML2                  (Eric Raymond)
o Beta        Read-Copy Update Mutual Exclusion               (Dipankar 

Sarma, Rusty Russell, Andrea Arcangeli, LSE Team)
o Beta USB device (not host) support (Stuart
Lynne, Greg Kroah-Hartman)
o Beta Support for IDE TCQ (Tagged Command Queueing) (Jens Axboe)

o Alpha       Better support of high-end NUMA machines        (NUMA team)
o Alpha       Add Asynchronous IO (aio) support               (Ben LaHaise)
o Alpha       Overhaul PCMCIA support                         (David 
Woodhouse, David Hinds)
o Alpha       Full compliance with IPv6                       (Alexey 
Kuznetzov, Jun Murai, Yoshifuji Hideaki, USAGI team)
o Alpha       UMSDOS (Unix under MS-DOS) Rewrite              (Al Viro)
o Alpha       Scalable Statistics Counter                     (Ravikiran 
Thirumalai)
o Alpha       Linux Kernel Crash Dumps                        (Matt 
Robinson, LKCD team)
o Alpha       Add support for NFS v4                          (NFS v4 team)
o Alpha       ext2/ext3 large directory support: HTree index  (Daniel 
Phillips, Christopher Li, Ted Ts'o)
o Alpha       Improved i2o (Intelligent Input/Ouput) layer    (Alan Cox)
o Alpha       New MTRR (Memory Type Range Register) driver    (Patrick 

Mochel)
o Alpha Remove use of the BKL (Big Kernel Lock) (Alan Cox,
Robert Love, Neil Brown, Dave Hansen, etc.)
o Alpha Zerocopy NFS (Hirokazu
Takahashi)

o Started More complete NetBEUI stack (Arnaldo
Carvalho de Melo, from Procom donated code)
o Started Change all drivers to new driver model (All
maintainers)
o Started Reiserfs v4 (Reiserfs
team)
o Started Move ISDN4Linux to CAPI based interface (ISDN4Linux
team)

o Draft #2 New lightweight library (klibc) (Greg Kroah-
Hartman)
o Draft #3 Replace initrd by initramfs (H. Peter
Anvin, Al Viro)

o Planning    Add thrashing control                           (Rik van Riel)
o Planning    Remove all hardwired drivers from kernel        (Alan Cox, 
etc.)
o Planning    Generic parameter/command line interface        (Keith Owens)
o Planning    New mount API                                   (Al Viro)
o Planning    Implement new device naming convention          (Device 

naming team)

Cleanups:

Merged
o in 2.5.3 Break Configure.help into multiple files (Linus
Torvalds)
o in 2.5.3 Untangle include file dependancies (Dave Jones,
Roman Zippel)
o in 2.5.4 Per network protocol slabcache & sock.h (Arnaldo
Carvalho de Melo)
o in 2.5.4 Per filesystem slabcache & fs.h (Daniel
Phillips, Jeff Garzik, Al Viro)
o in 2.5.6 Killing kdev_t for block devices (Al Viro)

o Ready       Switch to ->get_super() for file_system_type    (Al Viro)
o Ready       ->getattr() ->setattr() ->permission() changes  (Al Viro)

o Beta file.h and INIT_TASK (Benjamin
LaHaise)

o Beta        Proper UFS fixes, ext2 and locking cleanups     (Al Viro)
o Beta        Lifting limitations on mount(2)                 (Al Viro)
o Beta        Remove dcache_lock                              (Maneesh 

Soni, IBM team)

o Alpha Split up x86 setup.c into managable pieces (Patrick
Mochel)

o Started Reorder x86 initialization (Dave Jones,
Randy Dunlap)

Have some free time and want to help? Check out the Kernel Janitor TO DO list for a list of source code cleanups you can work on. A great place to start learning more about kernel internals!

I may not be the leading BKL expert, but I play one on TV :)

Perhaps one of the kernel-janitor people would like to assist me with this (cc'ing that list). I'd be willing to keep a web page to list all current BKL uses and keep track of them as they are removed/added Perhaps a set of web pages which resemble the directory structure of the kernel tree would be helpful??

Here's a good question for kernel-janitor, and anyone else who's interested, what format describing BKL use would most encourage you to go and remove it? We already have Rick Lindsley's Global spinlock list:
http://prdownloads.sourceforge.net/lse/locking_doc-2.4.16 . The BKL
use in there is somewhat dated, but might be a good start.

I have some awk scripts that I use on each new kernel release to check for new and removed uses of the BKL. I can adapt these to start checking new BK changesets for BKL changes.

Dave Hansen
[email protected]

Actually Andrews work on address_space based writeback is related somewhat, but really it's a rewrite/cleanup of the buffer layer. Delayed block alocation is helped alot by this, and almost depends on it IIRC.

One vote for a seperate listing in the status for "Address Space based Writeback / Buffer layer cleanup".

Well the next major step here is going direct pagecache<->BIO, bypassing the intermediate submit_bh for most I/O.

Probably that will make most of the performance benefits of delayed-allocate go away.

There are other reasons for implementing delalloc (XFS, improved layout, ...). So it ain't dead yet.

At 48 bytes, 2.5's buffer_head is now precisely half the size of 2.4's. I'm hoping to be able to shed another eight bytes yet.

With the pagecache<->BIO change, the buffer_head will most definitely become "per-page metadata which describes the state of sub-page segments" and not "something which is used for performing I/O".

-

Most of the performance benefits of delayed allocate are that you do not the hard work of allocating the disk space in each write call, you get to do it once, in potentially larger chunks, and often not in the user's context.

Steve

Except for moving the work out of the users context, ext2 gets a similar benefit by reserving disk space ahead of time. So it isn't clear that you need to have a delayed allocation to achieve this.

Eric

Doesn't everyone reserve more than the write asks for?

Delayed allocation as implemented in XFS means that during a write call which has to make more room in a file, a super block counter is changed, and the extent information is recorded in an in memory structure associated with the inode.

Yes, if you have all your metadata buffers sitting in memory and are not journalling changes then all you are doing is a different set of memory manipulations which are noise on modern cpus when compared with actually moving the file data out to disk.

Steve

I'm by no means a BKL expert, not even on TV :-) , but here is a suggestion.
What about a listing sorted by how the BKL should be replaced, something like this:

BKL --> dma_spin_lock
- alpha/kernel/ptrace.c Held while doing a ptrace (John Doe)
- drivers/block/rd.c Held while doing whatever (?)
- ...

BKL --> xprt_lock
- ...

BKL --> no lock required
- ...

BKL instances that will NOT be removed
- ...

The advantage is that it should be easier to group together instances that require the same type of work.
Comments?

-- Guillaume