Integration branch pushed out to btrfs-unstable

Hi everyone,

I've pushed out my current kernel git tree to a new branch called integration-test. This is meant for integration testing only and should not be run by anyone who doesn't love crashes.

I've pulled together a lot of important work from a lot of different people. It includes:

The new inode number allocator
Delayed inode and directory item updates
Scrub, chunk allocator fixes
Races in device addition and removal
Josef's performance fixes
A large series of cleanups and fixes

Most of the integration work was just pushing the cleanups into the new code bits. I still want to integrate Hugo Mills' balance progress ioctls (really nice). But it was a lot of work to get this all working together and I wanted to let everyone retest the result.

Miao, I didn't include your patch to delay inode item insertion because I was worried about interactions with the caching thread of the new inode number allocator. I did have to make a small change to make sure the inode-map cache didn't go through the delayed item routines.

Li, one thing I haven't done yet is a lot of benchmarking on the performance hit from waiting for the cache to fill on large existing filesystems. Was there any particular reason you didn't setup the caching to pre-cache the from root->highest_objectid to BTRFS_LAST_FREE_OBJECTID?

diffstat for my current tree:

-chris

]# git diff --stat v2.6.39...HEAD

 fs/btrfs/Makefile           |    2 +-
 fs/btrfs/acl.c              |    2 +-
 fs/btrfs/btrfs_inode.h      |   14 +
 fs/btrfs/compression.c      |   47 +-
 fs/btrfs/compression.h      |    2 +-
 fs/btrfs/ctree.c            |   51 +-
 fs/btrfs/ctree.h            |  199 ++---
 fs/btrfs/delayed-inode.c    | 1695 ++++++++++++++++++++++++++++++++++++
 fs/btrfs/delayed-inode.h    |  141 +++
 fs/btrfs/delayed-ref.c      |  114 ---
 fs/btrfs/delayed-ref.h      |    6 -
 fs/btrfs/dir-item.c         |   39 +-
 fs/btrfs/disk-io.c          |  202 +++---
 fs/btrfs/disk-io.h          |   19 +-
 fs/btrfs/export.c           |   25 +-
 fs/btrfs/extent-tree.c      | 2014 ++++---------------------------------------
 fs/btrfs/extent_io.c        |  327 ++------
 fs/btrfs/extent_io.h        |   40 +-
 fs/btrfs/extent_map.c       |    8 +-
 fs/btrfs/extent_map.h       |    4 +-
 fs/btrfs/file-item.c        |   38 +-
 fs/btrfs/file.c             |   45 +-
 fs/btrfs/free-space-cache.c |  993 +++++++++++++---------
 fs/btrfs/free-space-cache.h |   48 +-
 fs/btrfs/inode-item.c       |    2 -
 fs/btrfs/inode-map.c        |  428 +++++++++-
 fs/btrfs/inode-map.h        |   13 +
 fs/btrfs/inode.c            |  688 ++++++---------
 fs/btrfs/ioctl.c            |  176 ++++-
 fs/btrfs/ioctl.h            |   76 ++
 fs/btrfs/locking.c          |   25 -
 fs/btrfs/locking.h          |    2 -
 fs/btrfs/ref-cache.c        |  164 ----
 fs/btrfs/ref-cache.h        |   24 -
 fs/btrfs/relocation.c       |   67 +-
 fs/btrfs/root-tree.c        |   61 +--
 fs/btrfs/scrub.c            | 1368 +++++++++++++++++++++++++++++
 fs/btrfs/super.c            |   44 +-
 fs/btrfs/sysfs.c            |   77 --
 fs/btrfs/transaction.c      |  196 ++---
 fs/btrfs/transaction.h      |    5 +-
 fs/btrfs/tree-defrag.c      |    2 +-
 fs/btrfs/tree-log.c         |  208 +++--
 fs/btrfs/tree-log.h         |    1 -
 fs/btrfs/version.sh         |   43 -
 fs/btrfs/volumes.c          |  657 ++++++---------
 fs/btrfs/volumes.h          |   27 +-
 fs/btrfs/xattr.c            |   12 +-
 48 files changed, 5902 insertions(+), 4539 deletions(-)

please note that in order to test scrub functionality you need an updated btrfs utility. You can pull it from:

git://git.kernel.org/pub/scm/linux/kernel/git/arne/btrfs-progs-unstable-arne.git
scrub

Thanks,
Arne

I'll cook up a patch for this kind of pre-cache.

The commit

commit c4f675cd40d955d539180506c09515c90169b15b

Author: Sergei Trofimovich <[email protected]>
Date:   Fri May 20 20:20:30 2011 +0000

btrfs: don't spin in shrink_delalloc if there is nothing to free

gives me very early ENOSPC in my tests on a small volume (7GB).

-Arne

What king of test is that? I'd like to reproduce it. Does it do a lot of parallel I/O?

May I ask you to try to reproduce the problem with this sole patch on top of linus master?

  Sergei

With Arne's params to fs_mark it triggers for 2.6.39+this patch. I took 2GB loop device and the script [1].

$ ./make_it.sh
...
#  /home/slyfox/dev/fs_mark/fs_mark  -D  512  -t  16  -n  4096  -s  51200  -L7  -S0  -d  /mnt/ut 
#       Version 3.3, 16 thread(s) starting at Wed May 25 23:35:06 2011
#       Sync method: NO SYNC: Test does not issue sync() or fsync() calls.
#       Directories:  Time based hash between directories across 512 subdirectories with 180 seconds per subdirectory.
#       File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name)
#       Files info: size 51200 bytes, written with an IO size of 16384 bytes per write
#       App overhead is time in microseconds spent in the test not doing file writing related system calls.

FSUse% Count Size Files/sec App Overhead
fs_mark: write_file write failed: -1 No space left on device
fs_mark: write_file write failed: -1 No space left on device
fs_mark: write_file write failed: -1 No space left on device ...^C

$ .../btrfs fi df /mnt/ut/
Data: total=622.25MB, used=434.61MB
System: total=4.00MB, used=4.00KB
Metadata: total=417.50MB, used=6.69MB

$ sudo .../btrfs fi show /dev/loop0
Label: none  uuid: ccb18c12-4cbe-458c-835a-30e767939121
        Total devices 1 FS bytes used 441.31MB
        devid    1 size 2.00GB used 1.02GB path /dev/loop0

[1] 'make_it.sh' script exposing bug:

    export LANG=C

    FS_MARK=$HOME/dev/fs_mark/fs_mark
    MKFS_BTRFS=$HOME/dev/git/btrfs-progs-unstable/mkfs.btrfs

    sz=2
    the_file=${sz}G.img
    mount_point=/mnt/ut

    dd if=/dev/zero of="${the_file}" bs=1 count=1 seek=$((${sz}*1024*1024*1024-1))
    $MKFS_BTRFS -d raid0 -m raid0 "${the_file}"
    sudo mount -oloop "${the_file}" "${mount_point}"
    sudo chown slyfox:users "${mount_point}"
    chmod u+w "${mount_point}"
    cd /tmp
    $FS_MARK \
        -D  512  -t  16  -n  4096  -s  51200  -L7  -S0 \
        -d "${mount_point}"
    sudo umount "${mount_point}"

  Sergei