Skip to contentSkip to conversation

Btrfs updates

GIT PULL Chris Mason 4 participants 6 messages May 27, 2010 – Sep 12, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi everyone,

The master branch of the btrfs-unstable tree:

git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git master

Has the first round of btrfs updates for 2.6.35-rc. I still have some pending fixes and patches queued up from other people, but this pull request has the two major features for this release.

First we've got a large ENOSPC improvement from Yan Zheng. He hit on all the difficult corners including space balancing, drive management, fsync logging and many others. This has been in the works for sometime and it has held up very well in testing.

The second batch is O_DIRECT work from Josef Bacik. We've had a few different iterations on ENOSPC floating on the btrfs list, and this one takes a much less complex road by hooking into the generic fs/direct-io.c code. Josef added a hook for bio submission to fs/direct-io.c so that we could do proper block number transaction and properly do multi-device IO.

There are small changes in here to fs/direct-io.c, mm/filemap.c and include/linux/fs.h. These have been sent out for review a few times and commented on. The filemap.c change allows O_DIRECT reads to fall back to buffered IO, which we need to support compression and do checksumming IOs < 1 page.

My commits here are just integrating the two and fixing a few bugs in the result.

Zheng Yan (13) commits (+4076/-2679):
Btrfs: Integrate metadata reservation with start_transaction (+678/-528)
Btrfs: Update metadata reservation for delayed allocation (+232/-415)
Btrfs: Shrink delay allocated space in a synchronized (+88/-121)
Btrfs: Introduce contexts for metadata reservation (+853/-385)
Btrfs: Link block groups of different raid types (+121/-55)
Btrfs: Metadata ENOSPC handling for balance (+1163/-747)
Btrfs: Metadata ENOSPC handling for tree log (+156/-128)
Btrfs: Metadata reservation for orphan inodes (+365/-66)
Btrfs: Pre-allocate space for data relocation (+92/-45)
Btrfs: Introduce global metadata reservation (+241/-76)
Btrfs: Fix block generation verification race (+1/-1)
Btrfs: Kill allocate_wait in space_info (+58/-76)
Btrfs: Kill init_btrfs_i() (+28/-36)

Chris Mason (9) commits (+314/-80):
Btrfs: fix preallocation and nodatacow checks in O_DIRECT (+140/-16)
Btrfs: move O_DIRECT space reservation to btrfs_direct_IO (+8/-6)
Btrfs: don't walk around with task->state != TASK_RUNNING (+4/-3)
Btrfs: use async helpers for DIO write checksumming (+52/-17)
Btrfs: add more error checking to btrfs_dirty_inode (+13/-2)
Btrfs: avoid ENOSPC errors in btrfs_dirty_inode (+11/-3)
Btrfs: rework O_DIRECT enospc handling (+49/-30)
Btrfs: drop verbose enospc printk (+2/-0)
Btrfs: allow unaligned DIO (+35/-3)

Josef Bacik (5) commits (+829/-134):
direct-io: add a hook for the fs to provide its own submit_bio function (+45/-8)
fs: allow short direct-io reads to be completed via buffered IO (+31/-5)
direct-io: do not merge logically non-contiguous requests (+18/-2)
Btrfs: add basic DIO read/write support (+631/-36)
Btrfs: do aio_write instead of write (+104/-83)

Total: (27) commits

 fs/btrfs/extent-tree.c  | 2317 ++++++++++++++++++++++++++++--------------------
 fs/btrfs/relocation.c   | 1991 +++++++++++++++++++++++++----------------
 fs/btrfs/inode.c        | 1797 +++++++++++++++++++++++++++++--------
 fs/btrfs/file.c         |  304 +++---
 fs/btrfs/tree-log.c     |  241 +++-
 fs/btrfs/transaction.c  |  232 +++-
 fs/btrfs/ioctl.c        |  206 ++--
 fs/btrfs/disk-io.c      |  171 +--
 fs/btrfs/ctree.h        |  163 ++-
 fs/btrfs/ctree.c        |  109 +-
 fs/btrfs/delayed-ref.c  |  101 --
 fs/btrfs/extent_io.c    |   85 -
 fs/btrfs/ordered-data.c |   82 +
 fs/direct-io.c          |   62 +
 mm/filemap.c            |   36 
 fs/btrfs/super.c        |   30 
 fs/btrfs/file-item.c    |   28 
 fs/btrfs/inode-item.c   |   27 
 fs/btrfs/transaction.h  |   24 
 fs/btrfs/root-tree.c    |   23 
 fs/btrfs/volumes.c      |   17 
 fs/btrfs/extent_io.h    |   14 
 fs/btrfs/xattr.c        |   12 
 include/linux/fs.h      |   11 
 fs/btrfs/ordered-data.h |    9 
 fs/btrfs/tree-defrag.c  |    7 
 fs/btrfs/disk-io.h      |    4 
 fs/btrfs/delayed-ref.h  |    3 
 fs/btrfs/btrfs_inode.h  |    3 
 fs/btrfs/tree-log.h     |    2 
 fs/btrfs/async-thread.c |    1 
 31 files changed, 5219 insertions(+), 2893 deletions(-)
#1 lore raw
Quoted 7 lines · Chris Mason
 fs/btrfs/extent-tree.c  | 2317 ++++++++++++++++++++++++++++--------------------
 fs/btrfs/relocation.c   | 1991 +++++++++++++++++++++++++----------------
 fs/btrfs/inode.c        | 1797 +++++++++++++++++++++++++++++--------
 fs/btrfs/file.c         |  304 +++---
 fs/btrfs/tree-log.c     |  241 +++-

...

What kind of bogus diffstat is this?

Don't do that. I cannot compare your bogus diffstat with what I get, because it's just random noise. You've apparently sorted it by size of the damage, but the numbers are total crap too. That is _not_ the actual size of the changes at all.

I suspect you have some script that adds up all the patches, but that's wrong. If a subsequent patch changes the things that an earlier patch has done, then the numbers don't just add up.

And if the diffstat doesn't match what I get when I pull, then my reaction inevitably is "ok, that's not what they asked me to pull, so I'll just reject it out-of-hand". So don't play games with diffstats - that just means that things won't get pulled.

Linus

#2 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 7 lines · Chris Mason
 fs/btrfs/extent-tree.c  | 2317 ++++++++++++++++++++++++++++--------------------
 fs/btrfs/relocation.c   | 1991 +++++++++++++++++++++++++----------------
 fs/btrfs/inode.c        | 1797 +++++++++++++++++++++++++++++--------
 fs/btrfs/file.c         |  304 +++---
 fs/btrfs/tree-log.c     |  241 +++-

...

Quoted 6 lines

What kind of bogus diffstat is this?

Don't do that. I cannot compare your bogus diffstat with what I get, because it's just random noise. You've apparently sorted it by size of the damage, but the numbers are total crap too. That is _not_ the actual size of the changes at all.

My the btrfs-unstable repo is against v2.6.34, here is a 'pure' diffstat. My script ends up piping the git log -p output through diffstat, which does give different numbers. I'll use the git diff variant from now on.

# git diff v2.6.34 HEAD | diffstat
 fs/btrfs/async-thread.c |    1 
 fs/btrfs/btrfs_inode.h  |    3 
 fs/btrfs/ctree.c        |  109 +-
 fs/btrfs/ctree.h        |  163 ++-
 fs/btrfs/delayed-ref.c  |  101 --
 fs/btrfs/delayed-ref.h  |    3 
 fs/btrfs/disk-io.c      |  169 +--
 fs/btrfs/disk-io.h      |    4 
 fs/btrfs/extent-tree.c  | 2255 ++++++++++++++++++++++++++++--------------------
 fs/btrfs/extent_io.c    |   85 -
 fs/btrfs/extent_io.h    |   14 
 fs/btrfs/file-item.c    |   28 
 fs/btrfs/file.c         |  166 ++-
 fs/btrfs/inode-item.c   |   27 
 fs/btrfs/inode.c        | 1713 ++++++++++++++++++++++++++++--------
 fs/btrfs/ioctl.c        |  206 ++--
 fs/btrfs/ordered-data.c |   82 +
 fs/btrfs/ordered-data.h |    9 
 fs/btrfs/relocation.c   | 1971 +++++++++++++++++++++++++----------------
 fs/btrfs/root-tree.c    |   23 
 fs/btrfs/super.c        |   30 
 fs/btrfs/transaction.c  |  232 +++-
 fs/btrfs/transaction.h  |   24 
 fs/btrfs/tree-defrag.c  |    7 
 fs/btrfs/tree-log.c     |  241 +++--
 fs/btrfs/tree-log.h     |    2 
 fs/btrfs/volumes.c      |   17 
 fs/btrfs/xattr.c        |   12 
 fs/direct-io.c          |   62 +
 include/linux/fs.h      |   11 
 mm/filemap.c            |   36 
 31 files changed, 5066 insertions(+), 2740 deletions(-)

-chris

#3 lore raw
Quoted 2 lines · Chris Mason
# git diff v2.6.34 HEAD | diffstat

That still has the potential to be wrong (but got the numbers I expected this time). It will be wrong in several cases:

- "diffstat" has some random common prefix removal logic that I've never figured out the exact rules for.

You don't happen to see it, because you actually had changes outside of fs/btrfs (so there was no common prefix to worry about), but if everything had been inside fs/btrfs, then at least some versions of diffstat will just remove that whole thing as common, and talk about changes to 'ioctl.c' rather than 'fs/btrfs/ioctl.c'

(And no, I'm not entirely sure what triggers it. It might only happen for certain patterns and/or certain versions of diffstat, but it's a reason to avoid diffstat in general, or at least use "-p1" to make it reliable)

- it will do the wrong thing for renames and copies.

- it doesn't give the summary that pull-request does, which talks about new, deleted and file-mode-changed files.

So just do

	git diff --stat --summary -M v2.6.34..HEAD

instead, which gets all the above cases right. Also, you don't even need to remember where you started - you might as well use git to do that too, and write it as (assuming you have an 'origin' branch that points to the upstream tree):

	git diff --stat --summary -M origin...HEAD

(note the *three* dots: that says that you should diff against the common ancestor, so git will just figure out where you started on its own).

Linus

#4 lore raw
In reply to Chris Mason

Hi, Chris

Quoted 29 lines · Chris Mason

My commits here are just integrating the two and fixing a few bugs in the result.

Zheng Yan (13) commits (+4076/-2679):
Btrfs: Integrate metadata reservation with start_transaction (+678/-528)
Btrfs: Update metadata reservation for delayed allocation (+232/-415)
Btrfs: Shrink delay allocated space in a synchronized (+88/-121)
Btrfs: Introduce contexts for metadata reservation (+853/-385)
Btrfs: Link block groups of different raid types (+121/-55)
Btrfs: Metadata ENOSPC handling for balance (+1163/-747)
Btrfs: Metadata ENOSPC handling for tree log (+156/-128)
Btrfs: Metadata reservation for orphan inodes (+365/-66)
Btrfs: Pre-allocate space for data relocation (+92/-45)
Btrfs: Introduce global metadata reservation (+241/-76)
Btrfs: Fix block generation verification race (+1/-1)
Btrfs: Kill allocate_wait in space_info (+58/-76)
Btrfs: Kill init_btrfs_i() (+28/-36)

Chris Mason (9) commits (+314/-80):
Btrfs: fix preallocation and nodatacow checks in O_DIRECT (+140/-16)
Btrfs: move O_DIRECT space reservation to btrfs_direct_IO (+8/-6)
Btrfs: don't walk around with task->state != TASK_RUNNING (+4/-3)
Btrfs: use async helpers for DIO write checksumming (+52/-17)
Btrfs: add more error checking to btrfs_dirty_inode (+13/-2)
Btrfs: avoid ENOSPC errors in btrfs_dirty_inode (+11/-3)
Btrfs: rework O_DIRECT enospc handling (+49/-30)
Btrfs: drop verbose enospc printk (+2/-0)
Btrfs: allow unaligned DIO (+35/-3)

Quoted 9 lines

Josef Bacik (5) commits (+829/-134):
direct-io: add a hook for the fs to provide its own submit_bio function (+45/-8)
fs: allow short direct-io reads to be completed via buffered IO (+31/-5)
direct-io: do not merge logically non-contiguous requests (+18/-2)
Btrfs: add basic DIO read/write support (+631/-36)
Btrfs: do aio_write instead of write (+104/-83)

Total: (27) commits

In the following mail, You said that you had put the cleanups which Zhao Lei and I contributed into a .35 queue, but I cannot find those patches have been merged into the .35 kernel...
http://marc.info/?l=linux-btrfs&m=127049622906168&w=2

Is there anything wrong with those patches?

Regards
Miao

#5 lore raw
In reply to Chris Mason

On Thu, May 27, 2010 at 6:15 PM, Chris Mason <[email protected]> w= rote:

Quoted 1 line

=C2=A0 =C2=A0Btrfs: Fix block generation verification race (+1/-1)

This seems to cause a hang when there are "parent transid verify failed= " errors.

Somebody on #btrfs told me to revert it, and at least the system doesn't hang any more.

--=20
=46elipe Contreras
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel= " in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html