Btrfs: a copy on write, snapshotting FS
Hello everyone,
After the last FS summit, I started working on a new filesystem that maintains checksums of all file data and metadata. Many thanks to Zach Brown for his ideas, and to Dave Chinner for his help on benchmarking analysis.
The basic list of features looks like this:
* Extent based file storage (2^64 max file size)
* Space efficient packing of small files
* Space efficient indexed directories
* Dynamic inode allocation
* Writable snapshots
* Subvolumes (separate internal filesystem roots)
- Object level mirroring and striping
* Checksums on data and metadata (multiple algorithms available)
- Strong integration with device mapper for multiple device support
- Online filesystem check
* Very fast offline filesystem check
- Efficient incremental backup and FS mirroring
The ones with marked with * are mostly working, and the others are on my todo list. There are more details on the FS design, some benchmarks and download links here:
http://oss.oracle.com/~mason/btrfs/
The current status is a very early alpha state, and the kernel code weighs in at a sparsely commented 10,547 lines. I'm releasing now in hopes of finding people interested in testing, benchmarking, documenting, and contributing to the code.
I've gotten this far pretty quickly, and plan on continuing to knock off the features as fast as I can. Hopefully I'll manage a release every few weeks or so. The disk format will probably change in some major way every couple of releases.
The TODO list has some critical stuff:
* Ability to return -ENOSPC instead of oopsing
* mmap()ed writes
* Fault tolerance, (EIO, bad metadata etc)
* Concurrency. I use one mutex for all operations today
* ACLs and extended attributes
* Reclaim dead roots after a crash
* Various other bits from the feature list above
And finally, here's a quick and dirty summary of the FS design points:
* One large Btree per subvolume
* Copy on write logging for all data and metadata
* Reference count snapshots are the basis of the transaction system. A transaction is just a snapshot where the old root is immediately deleted on commit
* Subvolumes can be snapshotted any number of times
* Snapshots are read/write and can be snapshotted again
* Directories are doubly indexed to improve readdir speeds
So, please give it a try or a look and let me know what you think.
-chris
Quoted 59 lines · Chris Mason
Hello everyone,
After the last FS summit, I started working on a new filesystem that maintains checksums of all file data and metadata. Many thanks to Zach Brown for his ideas, and to Dave Chinner for his help on benchmarking analysis.
The basic list of features looks like this:
* Extent based file storage (2^64 max file size)
* Space efficient packing of small files
* Space efficient indexed directories
* Dynamic inode allocation
* Writable snapshots
* Subvolumes (separate internal filesystem roots)
- Object level mirroring and striping
* Checksums on data and metadata (multiple algorithms available)
- Strong integration with device mapper for multiple device support
- Online filesystem check
* Very fast offline filesystem check
- Efficient incremental backup and FS mirroringThe ones with marked with * are mostly working, and the others are on my todo list. There are more details on the FS design, some benchmarks and download links here:
http://oss.oracle.com/~mason/btrfs/
The current status is a very early alpha state, and the kernel code weighs in at a sparsely commented 10,547 lines. I'm releasing now in hopes of finding people interested in testing, benchmarking, documenting, and contributing to the code.
I've gotten this far pretty quickly, and plan on continuing to knock off the features as fast as I can. Hopefully I'll manage a release every few weeks or so. The disk format will probably change in some major way every couple of releases.
The TODO list has some critical stuff:
* Ability to return
-ENOSPCinstead of oopsing
* mmap()ed writes
* Fault tolerance, (EIO, bad metadata etc)
* Concurrency. I use one mutex for all operations today
* ACLs and extended attributes
* Reclaim dead roots after a crash
* Various other bits from the feature list aboveAnd finally, here's a quick and dirty summary of the FS design points:
* One large Btree per subvolume
* Copy on write logging for all data and metadata
* Reference count snapshots are the basis of the transaction system. A transaction is just a snapshot where the old root is immediately deleted on commit
* Subvolumes can be snapshotted any number of times
* Snapshots are read/write and can be snapshotted again
* Directories are doubly indexed to improve readdir speeds
Quoted 1 line
So, please give it a try or a look and let me know what you think.
Chris,
Given the substantial work that you've already put into btrfs and the direction you're Todo list details; it feels as though Btrfs will quickly provide the features that only Sun's ZFS provides.
Looking at your Btrfs benchmark and design pages it is clear that you're motivation is a filesystem that addresses modern concerns (performance that doesn't degrade over time, online fsck, fast offline fsck, data/metadata checksums, unlimited snapshots, efficient remote mirroring, etc). There is still much "Todo" but you've made very impressive progress for the first announcement!
I have some management oriented questions/comments.
1)
Regarding the direction of Btrfs as it relates to integration with DM. The allocation policies, the ease of configuring DM-based striping/mirroring, management of large pools of storage all seems to indicate that Btrfs will manage the physical spindles internally. This is very ZFS-ish (ZFS pools) so I'd like to understand where you see Btrfs going in this area.
Your initial benchmarks were all done ontop of a single disk with an LVM stack yet your roadmap/todo and design speaks to a tighter integration of the volume management features. So long term is traditional LVM/MD functionality to be pulled directly into Btrfs?
2)
The Btrfs notion of subvolumes and snapshots is very elegant and provides for a fluid management of the filesystem system data. It feels as though each subvolume/snapshot is just folded into the parent Btrfs volumes' namespace. Was there any particular reason you elected to do this? I can see that it lends itself to allowing snapshots of snapshots. If you could elaborate I'd appreciate it.
In practice subvolumes and/or snapshots appear to be implicitly mounted upon creation (refcount of parent is incremented). Is this correct? For snapshots, this runs counter to mapping the snapshots' data into the namespace of the origin Btrfs (e.g. with a .snapshot dir, but this is only useful for read-only snaps). Having snapshot namespaces in terms of monolithic subvolumes puts a less intuitive face on N Btrfs snapshots. The history of a given file/dir feels to be lost with this model.
Aside from folding snapshot history into the origin's namespace... It could be possible to have a mount.btrfs that allows subvolumes and/or snapshot volumes to be mounted as unique roots? I'd imagine a bind mount _could_ provide this too? Anyway, I'm just interested in understanding the vision for managing the potentially complex nature of a Btrfs namespace.
Thanks for doing all this work; I think the Linux community got a much needed shot in the arm with this Btrfs announcement.
regards,
Mike
Quoted 23 lines · Mike Snitzer
Quoted 6 lines · Chris Mason
Hello everyone,
After the last FS summit, I started working on a new filesystem that maintains checksums of all file data and metadata. Many thanks to Zach Brown for his ideas, and to Dave Chinner for his help on benchmarking analysis.
Chris,
Given the substantial work that you've already put into btrfs and the direction you're Todo list details; it feels as though Btrfs will quickly provide the features that only Sun's ZFS provides.
Looking at your Btrfs benchmark and design pages it is clear that you're motivation is a filesystem that addresses modern concerns (performance that doesn't degrade over time, online fsck, fast offline fsck, data/metadata checksums, unlimited snapshots, efficient remote mirroring, etc). There is still much "Todo" but you've made very impressive progress for the first announcement!
I have some management oriented questions/comments.
Quoted 7 lines
1)
Regarding the direction of Btrfs as it relates to integration with DM. The allocation policies, the ease of configuring DM-based striping/mirroring, management of large pools of storage all seems to indicate that Btrfs will manage the physical spindles internally. This is very ZFS-ish (ZFS pools) so I'd like to understand where you see Btrfs going in this area.
There's quite a lot of hand waving in that section. What I'd like to do is work closely with the LVM/DM/MD maintainers and come up with something that leverages what linux already does. I don't want to rewrite LVM into the FS, but I do want to make better use of info about the underlying storage.
Quoted 6 lines
Your initial benchmarks were all done ontop of a single disk with an LVM stack yet your roadmap/todo and design speaks to a tighter integration of the volume management features. So long term is traditional LVM/MD functionality to be pulled directly into Btrfs?
Quoted 8 lines
2)
The Btrfs notion of subvolumes and snapshots is very elegant and provides for a fluid management of the filesystem system data. It feels as though each subvolume/snapshot is just folded into the parent Btrfs volumes' namespace. Was there any particular reason you elected to do this? I can see that it lends itself to allowing snapshots of snapshots. If you could elaborate I'd appreciate it.
Yes, I wanted snapshots to be writable and resnapshottable. It also lowers the complexity to keep each snapshot as a subvolume/tree.
subvolumes are only slightly more expensive than a directory. So, even though a subvolume is a large grained unit for a snapshot, you can get around this by just making more subvolumes.
Quoted 8 lines
In practice subvolumes and/or snapshots appear to be implicitly mounted upon creation (refcount of parent is incremented). Is this correct? For snapshots, this runs counter to mapping the snapshots' data into the namespace of the origin Btrfs (e.g. with a .snapshot dir, but this is only useful for read-only snaps). Having snapshot namespaces in terms of monolithic subvolumes puts a less intuitive face on N Btrfs snapshots. The history of a given file/dir feels to be lost with this model.
That's somewhat true, the disk format does have enough information to show you that history, but cleanly expressing it to the user is a daunting task.
Quoted 7 lines
Aside from folding snapshot history into the origin's namespace... It could be possible to have a mount.btrfs that allows subvolumes and/or snapshot volumes to be mounted as unique roots? I'd imagine a bind mount _could_ provide this too? Anyway, I'm just interested in understanding the vision for managing the potentially complex nature of a Btrfs namespace.
One option is to put the real btrfs root into some directory in (/sys/fs/btrfs/$device?) and then use tools in userland to mount -o bind outside of that. I wanted to wait to get fancy until I had a better idea of how people would use the feature.
Quoted 4 lines
Thanks for doing all this work; I think the Linux community got a much needed shot in the arm with this Btrfs announcement.
Thanks for the comments.
-chris
Quoted 11 lines · Chris Mason
Quoted 6 lines
Aside from folding snapshot history into the origin's namespace... It could be possible to have a mount.btrfs that allows subvolumes and/or snapshot volumes to be mounted as unique roots? I'd imagine a bind mount _could_ provide this too? Anyway, I'm just interested in understanding the vision for managing the potentially complex nature of a Btrfs namespace.
One option is to put the real btrfs root into some directory in (
/sys/fs/btrfs/$device?) and then use tools in userland to mount -o bind outside of that. I wanted to wait to get fancy until I had a better idea of how people would use the feature.
We already support mounting into subdirectories of a filesystem for nfs connection sharing. The patch below makes use of this to allow mounting any subdirectory of a btrfs filesystem by specifying it in the form of /dev/somedevice:directory and when no subdirectory is specified uses 'default'. To make this more useful btrfs directories should grow some way to be marked head of a subvolume, and we'd need a more useful way to actually create subvolumes and snapshots without fugly ioctls.
Btw, do create a subvolume with my patch you need to escape back to the global root which is done by specifing /dev/somedevice:. as the device on the mount command line.
super.c +108 -3
@@ -17,6 +17,7 @@*/ #include <linux/module.h> +#include <linux/blkdev.h> #include <linux/buffer_head.h> #include <linux/fs.h> #include <linux/pagemap.h>
@@ -26,6 +27,7 @@#include <linux/string.h> #include <linux/smp_lock.h> #include <linux/backing-dev.h> +#include <linux/mount.h> #include <linux/mpage.h> #include <linux/swap.h> #include <linux/writeback.h>
@@ -135,11 +137,114 @@ static void btrfs_write_super(struct sup sb->s_dirt =0 ; } +/* +* This is almost a copy of get_sb_bdev in fs/super.c. +* We need the local copy to allow direct mounting of +* subvolumes, but this could be easily integrated back +* into the generic version. --hch +*/ + +/* start copy & paste */ +static int set_bdev_super (struct super_block *s,void *data) +{ + s->s_bdev = data; + s->s_dev = s->s_bdev->bd_dev; +return 0 ; +} + +static int test_bdev_super (struct super_block *s,void *data) +{ +return (void *)s->s_bdev == data; +} + +int btrfs_get_sb_bdev (struct file_system_type *fs_type, +int flags,const char *dev_name,void *data, +int (*fill_super)(struct super_block *,void *,int ), +struct vfsmount *mnt,const char *subvol) +{ +struct block_device *bdev =NULL ; +struct super_block *s; +struct dentry *root; +int error =0 ; + + bdev =open_bdev_excl (dev_name, flags, fs_type); +if (IS_ERR (bdev)) +return PTR_ERR (bdev); + +/* +* once the super is inserted into the list by sget, s_umount +* will protect the lockfs code from trying to start a snapshot +* while we are mounting +*/ +down (&bdev->bd_mount_sem); + s =sget (fs_type, test_bdev_super, set_bdev_super, bdev); +up (&bdev->bd_mount_sem); +if (IS_ERR (s)) + goto error_s; + +if (s->s_root) { +if ((flags ^ s->s_flags) &MS_RDONLY ) { +up_write (&s->s_umount); +deactivate_super (s); + error = -EBUSY ; + goto error_bdev; + } + +close_bdev_excl (bdev); + bdev =NULL ; + }else { +char b[BDEVNAME_SIZE ]; + + s->s_flags = flags; +strlcpy (s->s_id,bdevname (bdev, b),sizeof (s->s_id)); +sb_set_blocksize (s,block_size (bdev)); + error =fill_super (s, data, flags &MS_SILENT ?1 :0 ); +if (error) { +up_write (&s->s_umount); +deactivate_super (s); + goto error; + } + + s->s_flags |=MS_ACTIVE ; + } + +if (subvol) { + root =lookup_one_len (subvol, s->s_root,strlen (subvol)); +if (!root) { + error = -ENXIO ; + goto error_bdev; + } + }else { + root =dget (s->s_root); + } + + mnt->mnt_sb = s; + mnt->mnt_root = root; +return 0 ; + +error_s: + error =PTR_ERR (s); +error_bdev: +if (bdev) +close_bdev_excl (bdev); +error: +return error; +} +/* end copy & paste */ +static int btrfs_get_sb (struct file_system_type *fs_type, -int flags,const char *dev_name,void *data,struct vfsmount *mnt) +int flags,const char *identifier,void *data,struct vfsmount *mnt) { -return get_sb_bdev (fs_type, flags, dev_name, data, - btrfs_fill_super, mnt); +char *_identifier =kstrdup (identifier,GFP_KERNEL ); +const char *dev_name; + + dev_name =strsep (&_identifier,":" ); +if (!dev_name) +return -ENOMEM ; + +return btrfs_get_sb_bdev (fs_type, flags, dev_name, data, + btrfs_fill_super, mnt, + _identifier ? _identifier :"default" ); }static int btrfs_statfs (struct dentry *dentry,struct kstatfs *buf)
Quoted 13 lines · Christoph Hellwig
Quoted 11 lines · Chris Mason
Quoted 6 lines
Aside from folding snapshot history into the origin's namespace... It could be possible to have a mount.btrfs that allows subvolumes and/or snapshot volumes to be mounted as unique roots? I'd imagine a bind mount _could_ provide this too? Anyway, I'm just interested in understanding the vision for managing the potentially complex nature of a Btrfs namespace.
One option is to put the real btrfs root into some directory in (
/sys/fs/btrfs/$device?) and then use tools in userland to mount -o bind outside of that. I wanted to wait to get fancy until I had a better idea of how people would use the feature.
Quoted 5 lines
We already support mounting into subdirectories of a filesystem for nfs connection sharing. The patch below makes use of this to allow mounting any subdirectory of a btrfs filesystem by specifying it in the form of /dev/somedevice:directory and when no subdirectory is specified uses 'default'.
Neat, thanks Christoph, this will be much nicer longer term. I'll integrate it after I finish off -enospc.
Quoted 2 lines
To make this more useful btrfs directories should grow some way to be marked head of a subvolume,
They are already different in the btree, but maybe I'm not 100% sure what you mean by marked as the head of a subvolume?
Quoted 3 lines
and we'd need
a more useful way to actually create subvolumes and snapshots without fugly ioctls.
One way I can think of that doesn't involve an ioctl is to have a special subdir at the root of the subvolume:
cd/mnt/default/.snaps
mkdir new_snapshot
rmdir old_snapshot
cd /mnt
mkdir new_subvol
rmdir old_subvol
-chris
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
Chris> After the last FS summit, I started working on a new filesystem Chris> that maintains checksums of all file data and metadata. Many Chris> thanks to Zach Brown for his ideas, and to Dave Chinner for his Chris> help on benchmarking analysis.
Chris> The basic list of features looks like this:
Chris> * Extent based file storage (2^64 max file size)
Chris> * Space efficient packing of small files
Chris> * Space efficient indexed directories
Chris> * Dynamic inode allocation
Chris> * Writable snapshots
Chris> * Subvolumes (separate internal filesystem roots)
Chris> - Object level mirroring and striping
Chris> * Checksums on data and metadata (multiple algorithms available)
Chris> - Strong integration with device mapper for multiple device support
Chris> - Online filesystem check
Chris> * Very fast offline filesystem check
Chris> - Efficient incremental backup and FS mirroring
So, can you resize a filesystem both bigger and smaller? Or is that implicit in the Object level mirroring and striping?
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
Ie. being able to setup two pools, one being RAID6, the other being RAID1, where all currently accessed files are in the RAID1 setup, but if un-used get migrated to the RAID6 area.
And of course some way for efficient backups and more importantly RESTORES of data which is segregated like this.
John
Quoted 22 lines · John Stoffel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
Chris> After the last FS summit, I started working on a new filesystem Chris> that maintains checksums of all file data and metadata. Many Chris> thanks to Zach Brown for his ideas, and to Dave Chinner for his Chris> help on benchmarking analysis.
Chris> The basic list of features looks like this:
Chris> * Extent based file storage (2^64 max file size)
Chris> * Space efficient packing of small files
Chris> * Space efficient indexed directories
Chris> * Dynamic inode allocation
Chris> * Writable snapshots
Chris> * Subvolumes (separate internal filesystem roots)
Chris> - Object level mirroring and striping
Chris> * Checksums on data and metadata (multiple algorithms available)
Chris> - Strong integration with device mapper for multiple device support
Chris> - Online filesystem check
Chris> * Very fast offline filesystem check
Chris> - Efficient incremental backup and FS mirroring
Quoted 2 lines
So, can you resize a filesystem both bigger and smaller? Or is that implicit in the Object level mirroring and striping?
Growing the FS is just either extending or adding a new extent tree. Shrinking is more complex. The extent trees do have back pointers to the objectids that own the extent, but snapshotting makes that a little non-deterministic. The good news is there are no fixed locations for any of the metadata. So it is at least possible to shrink and pop out arbitrary chunks.
Quoted 4 lines
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
So far, I'm not planning quotas beyond the subvolume level.
Quoted 4 lines
Ie. being able to setup two pools, one being RAID6, the other being RAID1, where all currently accessed files are in the RAID1 setup, but if un-used get migrated to the RAID6 area.
HSM in general is definitely interesting. I'm afraid it is a long ways off, but it could be integrated into the scrubber that wanders the trees in the background.
-chris
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
So, can you resize a filesystem both bigger and smaller? Or is that implicit in the Object level mirroring and striping?
Chris> Growing the FS is just either extending or adding a new extent Chris> tree. Shrinking is more complex. The extent trees do have Chris> back pointers to the objectids that own the extent, but Chris> snapshotting makes that a little non-deterministic. The good Chris> news is there are no fixed locations for any of the metadata. Chris> So it is at least possible to shrink and pop out arbitrary Chris> chunks.
That's good to know. Being able to grow (online of course!) is great,
but so would shrinking as well. It makes life so much more flexible
for the SysAdmins, which is my particular focus... since it's my day job.
Quoted 3 lines
Quoted 3 lines
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
Chris> So far, I'm not planning quotas beyond the subvolume level.
So let me get this straight. Are you saying that quotas would only be on the volume level, and for the initial level of sub-volumes below that level? Or would *all* sub-volumes have quota support? And does that include snapshots as well?
Quoted 3 lines
Quoted 3 lines
Ie. being able to setup two pools, one being RAID6, the other being RAID1, where all currently accessed files are in the RAID1 setup, but if un-used get migrated to the RAID6 area.
Chris> HSM in general is definitely interesting. I'm afraid it is a Chris> long ways off, but it could be integrated into the scrubber Chris> that wanders the trees in the background.
Neat. As long as the idea is kept around a bit, that would be nice for an eventual addition. Or maybe someone needs to come up with a stackable filesystems to take care of this...
John
Quoted 11 lines · John Stoffel
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
Chris> So far, I'm not planning quotas beyond the subvolume level.
So let me get this straight. Are you saying that quotas would only be on the volume level, and for the initial level of sub-volumes below that level? Or would *all* sub-volumes have quota support? And does that include snapshots as well?
On disk, snapshots and subvolumes are identical...the only difference is their starting state (sorry, it's confusing, and it doesn't help that I interchange the terms when describing features).
Every subvolume will have a quota on the number of blocks it can consume. I haven't yet decided on the best way to account for blocks that are actually shared between snapshots, but it'll be in there somehow. So if you wanted to make a snapshot readonly, you just set the quota to 1 block.
But, I'm not planning on adding a way to say user X in subvolume Y has quota Z. I'll just be: this subvolume can't get bigger than a given size. (at least for version 1.0).
-chris
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
Quoted 5 lines · John Stoffel
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
Chris> So far, I'm not planning quotas beyond the subvolume level.
Quoted 5 lines
Quoted 5 lines
So let me get this straight. Are you saying that quotas would only be on the volume level, and for the initial level of sub-volumes below that level? Or would *all* sub-volumes have quota support? And does that include snapshots as well?
Chris> On disk, snapshots and subvolumes are identical...the only Chris> difference is their starting state (sorry, it's confusing, and Chris> it doesn't help that I interchange the terms when describing Chris> features).
Ok, that's fine. A sub-volume is the unit and depending on it's state, it's either a snapshot of an existing volume, or it's a volume on it's own, though it still has a parent (?) which it is mounted below? Do I have it right now?
Chris> Every subvolume will have a quota on the number of blocks it Chris> can consume. I haven't yet decided on the best way to account Chris> for blocks that are actually shared between snapshots, but Chris> it'll be in there somehow. So if you wanted to make a snapshot Chris> readonly, you just set the quota to 1 block.
Ok, so you really aren't talking about Quotas here, but space reservations instead.
Also, I think you're wrong here when you state that making a snapshot (sub-volume?) RO just requires you to set the quota to 1 block. What is to stop me from writing 1 block to a random file that already exists?
Chris> But, I'm not planning on adding a way to say user X in Chris> subvolume Y has quota Z. I'll just be: this subvolume can't Chris> get bigger than a given size. (at least for version 1.0).
Ok, so version 1.0 isn't as interesting to me in a production environment, since we pretty much need quotas (or a quick way to monitor how much space a user has been allocated on a volume.
But for a home system, it's certainly looking interesting as well, since I could give each home directory it's own sub-volume and just grow/shrink them as needed.
Maybe. :]
Thanks for your work on this.
John
Quoted 1 line · John Stoffel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Mason <[email protected]> writes:
[ nod ]
Quoted 4 lines
Also, I think you're wrong here when you state that making a snapshot (sub-volume?) RO just requires you to set the quota to 1 block. What is to stop me from writing 1 block to a random file that already exists?
It's copy on write, so changing one block means allocating a new one and putting the new contents there. The old blocks don't become available for reuse until the transaction commits.
-chris
Quoted 3 lines · Chris Mason
But, I'm not planning on adding a way to say user X in subvolume Y has quota Z. I'll just be: this subvolume can't get bigger than a given size. (at least for version 1.0).
I am affraid that this one is a major stopper for any production usage. Think about OpenVZ (or similar) VPSes. Of course having each VPS in own subvolume on the same device and being able to limit each subvolume is more than cool but on the other hand admin in VPS really needs to be able to set normal quotas for his users.
Other than that your project looks really good and interesting.
I also wonder if it is (would be) possible to set per-tree quotas like this:
/a - 20GB
/a/b - 10GB
/a/b/c - 2GB
/a/d - 5GB
/e - 30GB
meaning that whole subtree under /a is limited to 20GB, whole tree under /a/b is limited to both 20GB of /a and also by 10GB of /a/b, tree under /a/b/c is limited by 20GB of /a, 10GB of /a/b and 2GB of /a/b/c and so on? Or only /a and /e could be limited?
Thanks,
Grzegorz Kulewski
Hi Chris-
Quoted 10 lines · John Stoffel
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
Ie. being able to setup two pools, one being RAID6, the other being RAID1, where all currently accessed files are in the RAID1 setup, but if un-used get migrated to the RAID6 area.
And of course some way for efficient backups and more importantly RESTORES of data which is segregated like this.
I like the way dump and restore was handled in AFS (and now ZFS and NetApp). There is a simple command to flatten a file system and send it to another system, which can receive it and re-expand it. The dump/restore process uses snapshots and can easily send incremental backups which are significantly smaller than 0-level. This is somewhat better than rsync, because you don't need checksums to discover what data has changed -- you already have the new data segregated into copied-on-write blocks.
NetApp happens to use the standard NDMP protocol for sending the flattened file system. NetApp uses it for synchronous replication, volume migration, and back up to nearline storage and tape. AFS used "vol dump" and "vol restore" for migration, replication, and back-up. ZFS has the "zfs send" and "zfs receive" commands that do basically the same (Eric Kustarz recently published a blog entry that described how these work). And of course, all file system objects are able to be sent this way: streams, xattrs, ACLs, and so on are all supported.
Note also that NFSv4 supports the idea of migrated or replicated file objects. All that is needed to support it is a mechanism on the servers to actually move the data.
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA email;internet:chuck dot lever at nospam oracle dot com title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
Quoted 14 lines · Chuck Lever
Hi Chris-
Quoted 10 lines · John Stoffel
As a user of Netapps, having quotas (if only for reporting purposes) and some way to migrate non-used files to slower/cheaper storage would be great.
Ie. being able to setup two pools, one being RAID6, the other being RAID1, where all currently accessed files are in the RAID1 setup, but if un-used get migrated to the RAID6 area.
And of course some way for efficient backups and more importantly RESTORES of data which is segregated like this.
Quoted 8 lines
I like the way dump and restore was handled in AFS (and now ZFS and NetApp). There is a simple command to flatten a file system and send it to another system, which can receive it and re-expand it. The dump/restore process uses snapshots and can easily send incremental backups which are significantly smaller than 0-level. This is somewhat better than rsync, because you don't need checksums to discover what data has changed -- you already have the new data segregated into copied-on-write blocks.
The planned scheme in btrfs involves storing the transaction id in the header of every btree block, the inodes and the file extents. So, the only info you need to generate the incremental is the transaction id of the last time you sync'd. The tree would get walked and everything with a higher transaction id sent down the pipe.
Since there have back pointers from inode to directory, it also be able to generate a list of filenames that have changed, along with a list of extents that were modified.
Between the two schemes, you would be able to do metadata aware (ie storing exact duplicates) syncs between two boxes, or just have a faster rsync that doesn't need to walk the directory structure to find changes.
I will probably code the rsync version first, just because it allows you to backup to anything, not just a btrfs specific target. It is somewhat safer for an experimental FS (metadata errors are not duplicated) and will hopefully be easier to code.
Quoted 10 lines
NetApp happens to use the standard NDMP protocol for sending the flattened file system. NetApp uses it for synchronous replication, volume migration, and back up to nearline storage and tape. AFS used "vol dump" and "vol restore" for migration, replication, and back-up. ZFS has the "zfs send" and "zfs receive" commands that do basically the same (Eric Kustarz recently published a blog entry that described how these work). And of course, all file system objects are able to be sent this way: streams, xattrs, ACLs, and so on are all supported.
Quoted 3 lines
Note also that NFSv4 supports the idea of migrated or replicated file objects. All that is needed to support it is a mechanism on the servers to actually move the data.
Stringing the replication together with the underlying FS would be neat. Is there a way to deal with a master/slave setup, where the slave may be out of date?
-chris
Quoted 8 lines · Chris Mason
Quoted 3 lines · Chuck Lever
NetApp happens to use the standard NDMP protocol for sending the flattened file system. NetApp uses it for synchronous replication, volume migration, and back up to nearline storage and tape. AFS used "vol dump" and "vol restore" for migration, replication, and back-up. ZFS has the "zfs send" and "zfs receive" commands that do basically the same (Eric Kustarz recently published a blog entry that described how these work). And of course, all file system objects are able to be sent this way: streams, xattrs, ACLs, and so on are all supported.
Note also that NFSv4 supports the idea of migrated or replicated file objects. All that is needed to support it is a mechanism on the servers to actually move the data.
Stringing the replication together with the underlying FS would be neat. Is there a way to deal with a master/slave setup, where the slave may be out of date?
Among the implementations I'm aware of, there is a varying degree of integration into the physical file system. In general, it depends on how far out of date the slave is, and how closely the slave is supposed to be synchronized to the master.
A hot backup file system, for example, should be data-consistent within a few seconds of the master. A snapshot is used to initialize a slave, followed by a live stream of updates to the master being sent to slaves.
Such a mechanism already exists on NetApp filers because they gather
changes in NVRAM before committing them to the local file system. Simply put, these changes can also be bundled and sent to a local hot backup filer that is attached via Infiniband, or over the network to a remote hot backup filer.
For AFS, replication is done by maintaining a rw and ro copy of a volume on the designated master server. Changes are made to the rw copy over time. When admins want to push out a new version to replicas on another server, the ro copy on the master is replaced with a new snapshot, then this is pushed to the slaves. The replicas are always ro and are used mostly for load balancing; clients contact the closest or fastest server containing a replica of the volume they want to access. They always have a complete copy of the volume (ie no COW on the slaves).
I think you have designed into btrfs a lot of opportunity to implement this kind of data virtualization and management... I'm excited to see what can be done.
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel/
version:2.1
end:vcard
Quoted 1 line · Chris Mason
The basic list of features looks like this:
[amazing stuff snipped]
Quoted 4 lines
The current status is a very early alpha state, and the kernel code weighs in at a sparsely commented 10,547 lines. I'm releasing now in hopes of finding people interested in testing, benchmarking, documenting, and contributing to the code.
ok, what kind of benchmarks would help you most? bonnie? compilebench? sth. other?
is it possible to test it on top of LVM2 on RAID at this stage?
thanks,
florian
Quoted 10 lines · Florian D.
Quoted 1 line · Chris Mason
The basic list of features looks like this:
[amazing stuff snipped]
Quoted 4 lines
The current status is a very early alpha state, and the kernel code weighs in at a sparsely commented 10,547 lines. I'm releasing now in hopes of finding people interested in testing, benchmarking, documenting, and contributing to the code.
ok, what kind of benchmarks would help you most? bonnie? compilebench? sth. other?
Thanks! Lets start with a list of the things I know will go badly:
O_SYNC (not implemented)O_DIRECT (not implemented)
aio (not implemented)
multi-threaded (brain dead tree locking)
things that fill the drive (will oops)mmap() writes (not supported, mmap reads are ok)
Also, overlapping writes are not that well supported. For example, tar by default will write in 10k chunks, and btrfs_file_write currently cows on every single write. So, if your tar file has a bunch of 16k files, it'll go much faster if you tell tar to use 16k (or 8k) buffers.
In general, I was hoping for a generic delayed allocation facility to magically appear in the kernel, and so I haven't spent a lot of time tuning btrfs_file_write for this yet.
Any other workload is fair game, and I'm especially interested in seeing how badly the COW hurts. For example, on a big file, I'd like to see how much slower big sequential reads are after small random writes (fio is good for this). Or, writing to every file on the FS in random order and then seeing how much slower we are at reading.
Benchmarks that stress the directory structure are interesting too, huge numbers of files + directories etc. Ric Wheeler's fs_mark has a lot of options and output.
But, that's just my list, you can pick anything that you find interesting ;) Please try btrfsck after the run to see how well it keeps up. If you use blktrace to generate io traces, graphs can be generated:
http://oss.oracle.com/~mason/seekwatcher/
Not that well documented, but drop me a line if you need help running it. btt is a good alternative to the graphs too, and easier to run.
Quoted 2 lines
is it possible to test it on top of LVM2 on RAID at this stage?
Yes, I haven't done much multi-spindle testing yet, so I'm definitely interested in these numbers.
-chris
Quoted 9 lines · Chris Mason
Quoted 2 lines
is it possible to test it on top of LVM2 on RAID at this stage?
Yes, I haven't done much multi-spindle testing yet, so I'm definitely interested in these numbers.
-chris
I did not get very far:
# insmod btrfs.ko # mkfs.btrfs /dev/brain_volume_group/btrfstest on close 0 blocks are allocated fs created on /dev/brain_volume_group/btrfstest blocksize 4096 blocks 4980736
(/dev/brain_volume_group/btrfstest is a 20GB logical volume on top of RAID6)
# mount /dev/brain_volume_group/btrfstest /mnt/temp/ (this gives these kernel-msgs: [ 385.980358] btrfs: dm-6 checksum verify failed on 4 [ 385.980462] btrfs: dm-6 checksum verify failed on 12 [ 385.980559] btrfs: dm-6 checksum verify failed on 11 )
# touch /mnt/temp/default/testfile.txt [ 445.445638] btrfs: dm-6 checksum verify failed on 10
# umount /mnt/temp/
[ 457.980372] ------------[ cut here ]------------ [ 457.980377] kernel BUG at fs/buffer.c:2644! [ 457.980379] invalid opcode: 0000 [1] PREEMPT [ 457.980382] CPU 0 [ 457.980384] Modules linked in: btrfs snd_seq_midi cx88_dvb
cx88_vp3054_i2c video_buf_dvb snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi cx8800 cx8802 cx88xx ir_common tveeprom btcx_risc video_buf uhci_hcd
[ 457.980397] Pid: 6040, comm: btrfs/0 Not tainted 2.6.21.5 #50 [ 457.980400] RIP: 0010:[<ffffffff8021996c>] [<ffffffff8021996c>] submit_bh+0xf/0x102 [ 457.980408] RSP: 0018:ffff81000bab7d30 EFLAGS: 00010246 [ 457.980411] RAX: 000000000000a829 RBX: ffff81000ac207b0 RCX: ffff81005f0458c8 [ 457.980414] RDX: 0000000000000033 RSI: ffff81000ac207b0 RDI: 0000000000000001 [ 457.980418] RBP: 0000000000000001 R08: ffff81000ccdd3f8 R09: ffff81005fe78d50 [ 457.980422] R10: 00000000025fffe0 R11: ffffffff802407c7 R12: 0000000000000000 [ 457.980426] R13: ffff81001c16f480 R14: ffff81000ccdd3f8 R15: ffff81000bab7d88 [ 457.980430] FS: 00002b7554d54050(0000) GS:ffffffff80728000(0000) knlGS:00000000f7e3f6b0 [ 457.980434] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 457.980437] CR2: 00002aaaaabaf000 CR3: 0000000003b06000 CR4: 00000000000006e0 [ 457.980441] Process btrfs/0 (pid: 6040, threadinfo ffff81000bab6000, task ffff81005dba8480) [ 457.980443] Stack: ffff81000ac207b0 ffff81001c16f480 0000000000000000 ffffffff880988bb [ 457.980450] ffff81001c16f480 ffff81000bab7d80 ffff81000fee76e0 ffffffff88099eb1 [ 457.980455] 0000000100000000 ffff81001b318c10 ffff81001c16f180 0000000000000050 [ 457.980459] Call Trace: [ 457.980471] [<ffffffff880988bb>] :btrfs:write_ctree_super+0xd3/0x11f [ 457.980480] [<ffffffff88099eb1>] :btrfs:btrfs_commit_transaction+0x43e/0x5c0 [ 457.980486] [<ffffffff80257e4b>] cache_alloc_refill+0x2a3/0x4f7 [ 457.980491] [<ffffffff802873fb>] autoremove_wake_function+0x0/0x2e [ 457.980501] [<ffffffff8809a033>] :btrfs:btrfs_transaction_cleaner+0x0/0x141 [ 457.980510] [<ffffffff8809a0e0>] :btrfs:btrfs_transaction_cleaner+0xad/0x141 [ 457.980515] [<ffffffff8024869c>] run_workqueue+0xb5/0x18e [ 457.980519] [<ffffffff80245499>] worker_thread+0x0/0x145 [ 457.980523] [<ffffffff80287256>] keventd_create_kthread+0x0/0x89 [ 457.980526] [<ffffffff802455a8>] worker_thread+0x10f/0x145 [ 457.980531] [<ffffffff80277d4f>] default_wake_function+0x0/0xe [ 457.980535] [<ffffffff80287256>] keventd_create_kthread+0x0/0x89 [ 457.980540] [<ffffffff802302cb>] kthread+0xca/0xfb [ 457.980545] [<ffffffff80259318>] child_rip+0xa/0x12 [ 457.980549] [<ffffffff80287256>] keventd_create_kthread+0x0/0x89 [ 457.980555] [<ffffffff80230201>] kthread+0x0/0xfb [ 457.980558] [<ffffffff8025930e>] child_rip+0x0/0x12 [ 457.980561] [ 457.980562] [ 457.980563] Code: 0f 0b eb fe 8b 06 a8 20 75 04 0f 0b eb fe 48 83 7e 38 00 75 [ 457.980571] RIP [<ffffffff8021996c>] submit_bh+0xf/0x102 [ 457.980576] RSP <ffff81000bab7d30>
Linux localhost 2.6.21.5 #51 Fri Jun 15 20:53:36 CEST 2007 x86_64 AMD Athlon(tm) 64 Processor 3200+ AuthenticAMD GNU/Linux
Quoted 20 lines · Florian D.
Quoted 9 lines · Chris Mason
Quoted 2 lines
is it possible to test it on top of LVM2 on RAID at this stage?
Yes, I haven't done much multi-spindle testing yet, so I'm definitely interested in these numbers.
-chris
I did not get very far:
# insmod btrfs.ko # mkfs.btrfs /dev/brain_volume_group/btrfstest on close 0 blocks are allocated fs created on /dev/brain_volume_group/btrfstest blocksize 4096 blocks 4980736
Quoted 8 lines
(
/dev/brain_volume_group/btrfstestis a 20GB logical volume on top of RAID6)# mount /dev/brain_volume_group/btrfstest /mnt/temp/ (this gives these kernel-msgs: [ 385.980358] btrfs: dm-6 checksum verify failed on 4 [ 385.980462] btrfs: dm-6 checksum verify failed on 12 [ 385.980559] btrfs: dm-6 checksum verify failed on 11
These are normal on the first mount, the mkfs doesn't set the csums on the blocks it creates (will fix ;)
Quoted 10 lines
)
# touch /mnt/temp/default/testfile.txt [ 445.445638] btrfs: dm-6 checksum verify failed on 10# umount /mnt/temp/[ 457.980372] ------------[ cut here ]------------ [ 457.980377] kernel BUG at fs/buffer.c:2644!
Whoops. Please try this:
disk-io.c +1 -0
@@ -541,6 +541,7 @@ int write_ctree_super(struct btrfs_transelse ret =submit_bh (WRITE , bh);if (ret == -EOPNOTSUPP ) { +lock_buffer (bh);set_buffer_uptodate (bh); root->fs_info->do_barriers =0 ; ret =submit_bh (WRITE , bh);
Quoted 12 lines · Chris Mason
Quoted 4 lines
# umount /mnt/temp/[ 457.980372] ------------[ cut here ]------------ [ 457.980377] kernel BUG at fs/buffer.c:2644!Whoops. Please try this:
diff -r
38b36aaaa731disk-io.c
disk-io.c int write_ctree_super(struct btrfs_trans · line 547
Quoted 7 lines
disk-io.c +1
-0@@ -541,6 +541,7 @@ int write_ctree_super(struct btrfs_transelse ret =submit_bh (WRITE , bh);if (ret == -EOPNOTSUPP ) { +lock_buffer (bh);set_buffer_uptodate (bh); root->fs_info->do_barriers =0 ; ret =submit_bh (WRITE , bh);
sorry, with the patch applied:
[ 147.475077] BUG: at /home/florian/system/btrfs_test/btrfs-0.2/disk-io.c:534 write_ctree_super() [ 147.475082] [ 147.475083] Call Trace: [ 147.475096] [<ffffffff880957f7>] :btrfs:write_ctree_super+0x70/0x140 [ 147.475106] [<ffffffff88096ec5>] :btrfs:btrfs_commit_transaction+0x43e/0x5c0 [ 147.475112] [<ffffffff8022a2a6>] __writeback_single_inode+0x34f/0x361 [ 147.475121] [<ffffffff88096fec>] :btrfs:btrfs_commit_transaction+0x565/0x5c0 [ 147.475126] [<ffffffff8027b4eb>] autoremove_wake_function+0x0/0x2e [ 147.475136] [<ffffffff88095915>] :btrfs:close_ctree+0x4e/0x191 [ 147.475141] [<ffffffff8022e22e>] dispose_list+0xad/0xc9 [ 147.475146] [<ffffffff8029fd1a>] invalidate_inodes+0xc3/0xd5 [ 147.475155] [<ffffffff8808d170>] :btrfs:btrfs_put_super+0x10/0x31 [ 147.475159] [<ffffffff80299849>] generic_shutdown_super+0x5b/0xd2 [ 147.475163] [<ffffffff802998e6>] kill_block_super+0x26/0x3b [ 147.475167] [<ffffffff80299971>] deactivate_super+0x3d/0x55 [ 147.475172] [<ffffffff802a0e4b>] sys_umount+0x1ca/0x1f1 [ 147.475177] [<ffffffff8021fd18>] sys_newstat+0x19/0x31 [ 147.475184] [<ffffffff80250d5e>] system_call+0x7e/0x83 [ 147.475188] [ 147.476020] BUG: at /home/florian/system/btrfs_test/btrfs-0.2/disk-io.c:534 write_ctree_super() [ 147.476023] [ 147.476024] Call Trace: [ 147.476033] [<ffffffff880957f7>] :btrfs:write_ctree_super+0x70/0x140 [ 147.476042] [<ffffffff88096ec5>] :btrfs:btrfs_commit_transaction+0x43e/0x5c0 [ 147.476048] [<ffffffff8022a2a6>] __writeback_single_inode+0x34f/0x361 [ 147.476057] [<ffffffff88096fec>] :btrfs:btrfs_commit_transaction+0x565/0x5c0 [ 147.476061] [<ffffffff8027b4eb>] autoremove_wake_function+0x0/0x2e [ 147.476066] [<ffffffff802554d9>] mutex_lock+0xd/0x1d [ 147.476075] [<ffffffff8809592d>] :btrfs:close_ctree+0x66/0x191 [ 147.476080] [<ffffffff8022e22e>] dispose_list+0xad/0xc9 [ 147.476085] [<ffffffff8029fd1a>] invalidate_inodes+0xc3/0xd5 [ 147.476096] [<ffffffff8808d170>] :btrfs:btrfs_put_super+0x10/0x31 [ 147.476100] [<ffffffff80299849>] generic_shutdown_super+0x5b/0xd2 [ 147.476104] [<ffffffff802998e6>] kill_block_super+0x26/0x3b [ 147.476108] [<ffffffff80299971>] deactivate_super+0x3d/0x55 [ 147.476112] [<ffffffff802a0e4b>] sys_umount+0x1ca/0x1f1 [ 147.476118] [<ffffffff8021fd18>] sys_newstat+0x19/0x31 [ 147.476124] [<ffffffff80250d5e>] system_call+0x7e/0x83 [ 147.476128] [ 147.482579] BUG: at /home/florian/system/btrfs_test/btrfs-0.2/disk-io.c:534 write_ctree_super() [ 147.482582] [ 147.482583] Call Trace: [ 147.482592] [<ffffffff880957f7>] :btrfs:write_ctree_super+0x70/0x140 [ 147.482601] [<ffffffff88095949>] :btrfs:close_ctree+0x82/0x191 [ 147.482605] [<ffffffff8022e22e>] dispose_list+0xad/0xc9 [ 147.482611] [<ffffffff8029fd1a>] invalidate_inodes+0xc3/0xd5 [ 147.482619] [<ffffffff8808d170>] :btrfs:btrfs_put_super+0x10/0x31 [ 147.482623] [<ffffffff80299849>] generic_shutdown_super+0x5b/0xd2 [ 147.482627] [<ffffffff802998e6>] kill_block_super+0x26/0x3b [ 147.482631] [<ffffffff80299971>] deactivate_super+0x3d/0x55 [ 147.482636] [<ffffffff802a0e4b>] sys_umount+0x1ca/0x1f1 [ 147.482641] [<ffffffff8021fd18>] sys_newstat+0x19/0x31 [ 147.482648] [<ffffffff80250d5e>] system_call+0x7e/0x83 [ 147.482652] [ 147.483066] VFS: brelse: Trying to free free buffer [ 147.483069] BUG: at fs/buffer.c:1164 __brelse() [ 147.483071] [ 147.483072] Call Trace: [ 147.483081] [<ffffffff88095982>] :btrfs:close_ctree+0xbb/0x191 [ 147.483086] [<ffffffff8022e22e>] dispose_list+0xad/0xc9 [ 147.483091] [<ffffffff8029fd1a>] invalidate_inodes+0xc3/0xd5 [ 147.483099] [<ffffffff8808d170>] :btrfs:btrfs_put_super+0x10/0x31 [ 147.483103] [<ffffffff80299849>] generic_shutdown_super+0x5b/0xd2 [ 147.483107] [<ffffffff802998e6>] kill_block_super+0x26/0x3b [ 147.483111] [<ffffffff80299971>] deactivate_super+0x3d/0x55 [ 147.483116] [<ffffffff802a0e4b>] sys_umount+0x1ca/0x1f1 [ 147.483121] [<ffffffff8021fd18>] sys_newstat+0x19/0x31 [ 147.483127] [<ffffffff80250d5e>] system_call+0x7e/0x83
Quoted 7 lines · Florian D.
Quoted 6 lines · Chris Mason
Quoted 4 lines
# umount /mnt/temp/[ 457.980372] ------------[ cut here ]------------ [ 457.980377] kernel BUG at fs/buffer.c:2644!Whoops. Please try this:
[ bad patch ]
Quoted 4 lines
sorry, with the patch applied:
[ 147.475077] BUG: at
/home/florian/system/btrfs_test/btrfs-0.2/disk-io.c:534
Well, apparently I get get the silly stuff wrong an infinite number of times. Sorry, lets try again:
disk-io.c +2 -0
@@ -541,6 +541,8 @@ int write_ctree_super(struct btrfs_transelse ret =submit_bh (WRITE , bh);if (ret == -EOPNOTSUPP ) { +get_bh (bh); +lock_buffer (bh);set_buffer_uptodate (bh); root->fs_info->do_barriers =0 ; ret =submit_bh (WRITE , bh);
Quoted 8 lines · Chris Mason
Well, apparently I get get the silly stuff wrong an infinite number of times. Sorry, lets try again:
diff -r
38b36aaaa731disk-io.c
disk-io.c int write_ctree_super(struct btrfs_trans · line 548
Quoted 8 lines
disk-io.c +2
-0@@ -541,6 +541,8 @@ int write_ctree_super(struct btrfs_transelse ret =submit_bh (WRITE , bh);if (ret == -EOPNOTSUPP ) { +get_bh (bh); +lock_buffer (bh);set_buffer_uptodate (bh); root->fs_info->do_barriers =0 ; ret =submit_bh (WRITE , bh);
ha! it is working now. some numbers from here(with the fio-tool):
1. sequential read
2. random writes
3. sequential read again
filesize:300MB, bs:4K
btrfs reiserfs ext3 usr% sys% bw sec. usr% sys% bw sec. usr% sys% bw sec. 1 5 51 68.3 4.6 1 17 67.4 4.6 5 24 68.0 4.6 2 0 1 0.7 431 2 21 29.8 10.5 3 18 29.0 10.8 3 0 1 2.3 133 1 19 70.5 4.4 5 24 68.6 4.5
bw: MB/sec.
ext3: -o data=writeback,barrier=1
20GB LVM2 partition on a RAID6 (4 SATA-disks)
cheers,
florian
Quoted 4 lines · Florian D.
Quoted 3 lines · Chris Mason
Well, apparently I get get the silly stuff wrong an infinite number of times. Sorry, lets try again:
Quoted 15 lines
disk-io.c int write_ctree_super(struct btrfs_trans · line 548
Quoted 13 lines
diff -r
38b36aaaa731disk-io.cdisk-io.c +2
-0@@ -541,6 +541,8 @@ int write_ctree_super(struct btrfs_transelse ret =submit_bh (WRITE , bh);if (ret == -EOPNOTSUPP ) { +get_bh (bh); +lock_buffer (bh);set_buffer_uptodate (bh); root->fs_info->do_barriers =0 ; ret =submit_bh (WRITE , bh);ha! it is working now. some numbers from here(with the fio-tool):
Great, I'll have a v0.3 out on Monday with that fix rolled in.
Quoted 13 lines
1. sequential read
2. random writes
3. sequential read againfilesize:300MB, bs:4K
btrfs reiserfs ext3 usr% sys% bw sec. usr% sys% bw sec. usr% sys% bw sec. 1 5 51 68.3 4.6 1 17 67.4 4.6 5 24 68.0 4.6 2 0 1 0.7 431 2 21 29.8 10.5 3 18 29.0 10.8 3 0 1 2.3 133 1 19 70.5 4.4 5 24 68.6 4.5
Quoted 4 lines
bw: MB/sec.
ext3: -o data=writeback,barrier=120GB LVM2 partition on a RAID6 (4 SATA-disks)
Strange, these numbers are not quite what I was expecting ;) Could you please post your fio job files? Also, how much ram does the machine have? Only writing doesn't seem like enough to fill the ram.
-chris
Quoted 7 lines · Chris Mason
Strange, these numbers are not quite what I was expecting ;) Could you please post your fio job files? Also, how much ram does the machine have? Only writing doesn't seem like enough to fill the ram.
-chris
Sure:
Quoted 17 lines
[global]
directory=/mnt/temp/default
filename=testfile
size=300m
randrepeat=1
overwrite=1
end_fsync=1[job1]
description="sequ. read"
rw=read[job2]
stonewall
description="rand. write"
rw=randwrite
Quoted 4 lines
[job3]
stonewall
description="sequ. read"
rw=read
I have 1.5 GB RAM. I can easily rerun the tests with modified jobs, just tell me how to modify it
florian
Quoted 6 lines · Florian D.
Quoted 4 lines · Chris Mason
Strange, these numbers are not quite what I was expecting ;) Could you please post your fio job files? Also, how much ram does the machine have? Only writing doesn't seem like enough to fill the ram.
Quoted 12 lines
Quoted 3 lines
-chris
Sure:
Quoted 7 lines
[global]
directory=/mnt/temp/default
filename=testfile
size=300m
randrepeat=1
overwrite=1
end_fsync=1
[ very bad results on btrfs with these parameters ]
Ok, the numbers make more sense now. Basically what is happening is that during the random IO phase, fio is hitting every single block in the file. Btrfs will allocate new blocks in a sequential fashion, but the fsync does writeback in page order. So, the fsync sees completely random block ordering, and then we see it again on the reads.
In ext3 even though the writes are random, the fsync uses the original (sequential) ordering of the blocks, and everything works nicely.
The fix is either delayed allocation or defrag-on-writeback. Another option (which I'll have to do for O_SYNC performance) is to leave space in the blocks allocated to the file for COWs (basically strides of allocated blocks).
I'll do the defrag-on-writeback right after enospc.
-chris
Quoted 6 lines · Chris Mason
Hello everyone,
After the last FS summit, I started working on a new filesystem that maintains checksums of all file data and metadata. Many thanks to Zach Brown for his ideas, and to Dave Chinner for his help on benchmarking analysis.
Thanks to everyone that tried out btrfs. Most of those that did managed to hit problems with apps that did writes via mmap. So, I added a page_mkwrite call and worked out cow safe mmap writes.
Terje Røsten sent along a btrfsprogs patch so that it compiles properly on FC7 (thanks!).
I'll have a site up on oss.oracle.com/projects with mailing lists and links to my HG trees shortly. Until then:
http://oss.oracle.com/~mason/btrfs/ has the latest tar balls.
-chris
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Chris Mason
Hello everyone,
After the last FS summit, I started working on a new filesystem that maintains checksums of all file data and metadata. Many thanks to Zach Brown for his ideas, and to Dave Chinner for his help on benchmarking analysis.
The basic list of features looks like this:
Quoted 12 lines
* Extent based file storage (2^64 max file size)
* Space efficient packing of small files
* Space efficient indexed directories
* Dynamic inode allocation
* Writable snapshots
* Subvolumes (separate internal filesystem roots)
- Object level mirroring and striping
* Checksums on data and metadata (multiple algorithms available)
- Strong integration with device mapper for multiple device support
- Online filesystem check
* Very fast offline filesystem check
- Efficient incremental backup and FS mirroring
I would also suggest one more feature: support for block level de-duplication. I mean:
1. Ability for Btrfs to have blocks in several files to point to the same block on disk
2. Support for new syscall or IOCTL to de-duplicate as a single transaction two or more blocks on disk, i.e. link them to one of them and free others
3. De-de-duplicate blocks on disk, i.e. copy them on write
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Vlad
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Vladislav" == Vladislav Bolkhovitin <[email protected]> writes:
The basic list of features looks like this:
Quoted 12 lines
Quoted 4 lines
* Extent based file storage (2^64 max file size)
* Space efficient packing of small files
* Space efficient indexed directories
* Dynamic inode allocationQuoted 8 lines
* Writable snapshots
* Subvolumes (separate internal filesystem roots)
- Object level mirroring and striping
* Checksums on data and metadata (multiple algorithms available)
- Strong integration with device mapper for multiple device support
- Online filesystem check
* Very fast offline filesystem check
- Efficient incremental backup and FS mirroring
Vladislav> I would also suggest one more feature: support for block Vladislav> level de-duplication. I mean:
Vladislav> 1. Ability for Btrfs to have blocks in several files to Vladislav> point to the same block on disk
Yikes! I'd be *very* wary of this feature. It's going to be computationally expensive to do, and it's going to make the entire filesystem more fragile, since now one bit of corruption means that possibly all files with that block are now toast.
There would need to be some serious parity checking done here to make me feel confortable with such a system.
Vladislav> 2. Support for new syscall or IOCTL to de-duplicate as a Vladislav> single transaction two or more blocks on disk, i.e. link Vladislav> them to one of them and free others
Do you want it per-block, or per-file?
Vladislav> 3. De-de-duplicate blocks on disk, i.e. copy them on write
You mean that if two files are sharing blocks and one gets written to, then the changes are copied to a new block and the pointers updated? Sure, that's a integral part of the concest.
Vladislav> I suppose that de-duplication itself would be done by some Vladislav> user space process that would scan files, determine blocks Vladislav> with the same data and then de-duplicate them by using Vladislav> syscall or IOCTL (2).
Lots of work... where do you keep the metadata then? In the filesystem?
Vladislav> That would be very usable feature, which in most cases Vladislav> would allow to shrink occupied disk space on 50-90%.
Maybe. It's reliability I'd be concerned about here.
John
Quoted 11 lines · Vladislav Bolkhovitin
I would also suggest one more feature: support for block level de-duplication. I mean:
1. Ability for Btrfs to have blocks in several files to point to the same block on disk
2. Support for new syscall or IOCTL to de-duplicate as a single transaction two or more blocks on disk, i.e. link them to one of them and free others
Quoted 8 lines
3. De-de-duplicate blocks on disk, i.e. copy them on write
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Have you references for this number?
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.
Note a checksum stored in file metadata, that is automatically invalidated on write would speed up user space file de duplification, and rsync, etc....
Pádraig.
Quoted 14 lines · Pádraig Brady
Quoted 12 lines · Vladislav Bolkhovitin
I would also suggest one more feature: support for block level de-duplication. I mean:
1. Ability for Btrfs to have blocks in several files to point to the same block on disk
2. Support for new syscall or IOCTL to de-duplicate as a single transaction two or more blocks on disk, i.e. link them to one of them and free others
3. De-de-duplicate blocks on disk, i.e. copy them on write
Quoted 8 lines
Quoted 6 lines
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Have you references for this number?
No, I've seen it somewhere and it well confirms with my own observations.
Quoted 2 lines
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.
Yes, sure, de-duplication on files level brings its benefits, but on FS blocks level it would bring ever more benefits, because there are many more or less big files, which are different as a whole, but with a lot of the same blocks. Simple example of such files is UNIX-style mail boxes on a mail server.
Quoted 4 lines
Note a checksum stored in file metadata,
that is automatically invalidated on write would
speed up user space file de duplification,
and rsync, etc....
Sure, file level deduplication is simpler to implement, but it is generally less powerful, than FS block level one. And seems it should not be hard to add the above (1)-(3) features in the existing Btrfs structure, especially on this stage, when the disk format isn't stabilized yet.
Quoted 7 lines
Pádraig.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Vladislav Bolkhovitin
Quoted 9 lines
Quoted 8 lines
3. De-de-duplicate blocks on disk, i.e. copy them on write
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Quoted 8 lines
Quoted 1 line
Have you references for this number?
No, I've seen it somewhere and it well confirms with my own observations.
Quoted 2 lines
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.
Yes, sure, de-duplication on files level brings its benefits, but on FS blocks level it would bring ever more benefits, because there are many more or less big files, which are different as a whole, but with a lot of the same blocks. Simple example of such files is UNIX-style mail boxes on a mail server.
unix style mail boxes would not be a good example of wins for sector-based de-duplication since the duplicate mail is not going to be sector aligned.
David Lang
Quoted 14 lines · [email protected] wrote:
Quoted 14 lines
Quoted 10 lines
Quoted 2 lines
3. De-de-duplicate blocks on disk, i.e. copy them on write
Quoted 1 line
I suppose that de-duplication itself would be done by some user
space
Quoted 3 lines
process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
Quoted 1 line
That would be very usable feature, which in most cases would
allow to
Quoted 1 line
shrink occupied disk space on 50-90%.
Have you references for this number?
No, I've seen it somewhere and it well confirms with my own observations.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.
Yes, sure, de-duplication on files level brings its benefits, but on FS blocks level it would bring ever more benefits, because there are many more or less big files, which are different as a whole, but with a lot of the same blocks. Simple example of such files is UNIX-style mail boxes on a mail server.
unix style mail boxes would not be a good example of wins for sector-based de-duplication since the duplicate mail is not going to be sector aligned.
Yes, I realized that after I sent the e-mail. Handling of the same, but not aligned, data in different files would need more complex logic. Maybe too complex.
Vlad
Quoted 18 lines · Pádraig Brady
Quoted 17 lines · Vladislav Bolkhovitin
I would also suggest one more feature: support for block level de-duplication. I mean:
1. Ability for Btrfs to have blocks in several files to point to the same block on disk
2. Support for new syscall or IOCTL to de-duplicate as a single transaction two or more blocks on disk, i.e. link them to one of them and free others
3. De-de-duplicate blocks on disk, i.e. copy them on write
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
Quoted 6 lines
Quoted 2 lines
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Have you references for this number?
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.
Yes, I would expect simple hard links to be a better solution for this, but the feature request is not that out of line. I actually had plans on implementing auto duplicate block reuse earlier in btrfs.
Snapshots already share duplicate blocks between files, and so all of the reference counting needed to implement this already exists. Snapshots are writable, and data mods are copy on write, and in general things work.
But, to help fsck, the extent allocation tree has a back pointer to the inode that owns an extent. If you're doing snapshots, all of the owners
of the extent have the same inode number. If you're sharing duplicate
blocks, the owners can have any inode number, and fsck becomes much more complex.
In general, when I have to decide between fsck and a feature, I'm going to pick fsck. The features are much more fun, but fsck is one of the main motivations for doing this work.
Thanks for the input,
Chris
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 23 lines · Chris Mason
Quoted 21 lines · Pádraig Brady
Quoted 18 lines · Vladislav Bolkhovitin
I would also suggest one more feature: support for block level de-duplication. I mean:
1. Ability for Btrfs to have blocks in several files to point to the same block on disk
2. Support for new syscall or IOCTL to de-duplicate as a single transaction two or more blocks on disk, i.e. link them to one of them and free others
3. De-de-duplicate blocks on disk, i.e. copy them on write
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Quoted 7 lines
Quoted 3 lines
Have you references for this number?
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.Yes, I would expect simple hard links to be a better solution for this, but the feature request is not that out of line.
From effort POV hard links could be a better solution, but from effectiveness POV I can't agree with you.
Quoted 14 lines
I actually had plans
on implementing auto duplicate block reuse earlier in btrfs.Snapshots already share duplicate blocks between files, and so all of the reference counting needed to implement this already exists. Snapshots are writable, and data mods are copy on write, and in general things work.
But, to help fsck, the extent allocation tree has a back pointer to the inode that owns an extent. If you're doing snapshots, all of the owners
of the extent have the same inode number. If you're sharing duplicate
blocks, the owners can have any inode number, and fsck becomes much more complex.
Quoted 3 lines
In general, when I have to decide between fsck and a feature, I'm going to pick fsck. The features are much more fun, but fsck is one of the main motivations for doing this work.
I see. Thanks for explaining your position.
Vlad
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Chris Mason
Quoted 9 lines
Quoted 8 lines
3. De-de-duplicate blocks on disk, i.e. copy them on write
I suppose that de-duplication itself would be done by some user space process that would scan files, determine blocks with the same data and then de-duplicate them by using syscall or IOCTL (2).
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Quoted 7 lines
Quoted 3 lines
Have you references for this number?
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.Yes, I would expect simple hard links to be a better solution for this, but the feature request is not that out of line. I actually had plans on implementing auto duplicate block reuse earlier in btrfs.
with COW de-duplication you can merge things that have vastly different permissions. hard-links can't be used if different people have write permission.
David Lang
Hello!
Quoted 5 lines · Chris Mason
Quoted 4 lines · Pádraig Brady
Quoted 3 lines · Vladislav Bolkhovitin
I would also suggest one more feature: support for block level de-duplication. I mean:
...
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Have you references for this number?
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.Yes, I would expect simple hard links to be a better solution for this, but the feature request is not that out of line. I actually had plans on implementing auto duplicate block reuse earlier in btrfs.
One problem with hard-links for me is, they also share the meta-data, especially file permissions and owners.
Take a Subversion checkout for example: For each file "$A" Subversion saves a backup under ".svm/text-base/$A.svn-base" for file comparison and diff generation. The user controls the file permissions of "$A", Subversion protects its backup with 0444. You can't hard-link them, because than "svn diff" doesn't work anymore if your editor doesn't break the hard-link, or worse, your permissions can get wrong.
If previous versions Subversion also had an extra file for file attributes (mime-type, permissions, to-be-ignored, etc.) Since most files had no special attributes, each had a file only containing "END". Those you could hard-link by hand to save space.
If somebody want to research this further:
There is this nice little package called "perforate", which contains "finddup" to find duplicate files. Run it two times, once with "-i" to ignore permissions while comparing file contents
finddup -i -d /
and once without "-i" for "content and permissions must match"
finddup -d /
This will give you a hint on how many files you could hard-link or how many files share their content.
BYtE
Philipp
Signature · 7 lines
/ / (_)__ __ ____ __ Philipp Hahn / /__/ / _ \/ // /\ \/ / /____/_/_//_/\_,_/ /_/\_\ [email protected] - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 34 lines · Philipp Matthias Hahn
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I would also suggest one more feature: support for block level de-duplication. I mean:
...
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
That would be very usable feature, which in most cases would allow to shrink occupied disk space on 50-90%.
Have you references for this number?
In my experience one gets a lot of benefit from the much simpler process of "de-duplication" of files.Yes, I would expect simple hard links to be a better solution for this, but the feature request is not that out of line. I actually had plans on implementing auto duplicate block reuse earlier in btrfs.
One problem with hard-links for me is, they also share the meta-data, especially file permissions and owners.
Take a Subversion checkout for example: For each file "$A" Subversion saves a backup under ".svm/text-base/$A.svn-base" for file comparison and diff generation. The user controls the file permissions of "$A", Subversion protects its backup with 0444. You can't hard-link them, because than "svn diff" doesn't work anymore if your editor doesn't break the hard-link, or worse, your permissions can get wrong.
If previous versions Subversion also had an extra file for file attributes (mime-type, permissions, to-be-ignored, etc.) Since most files had no special attributes, each had a file only containing "END". Those you could hard-link by hand to save space.
If somebody want to research this further:
Quoted 8 lines
There is this nice little package called "perforate", which contains "finddup" to find duplicate files. Run it two times, once with "-i" to ignore permissions while comparing file contents
finddup -i -d /
and once without "-i" for "content and permissions must match"
finddup -d /
This will give you a hint on how many files you could hard-link or how many files share their content.
So, seems ever for file based de-duplication some support from the FS, including some kind of ability for different inodes point to the same data blocks to store the meta-data, would be needed anyway.
Vlad
Quoted 6 lines · Vladislav Bolkhovitin
Quoted 2 lines · Philipp Matthias Hahn
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I would also suggest one more feature: support for block level de-duplication. I mean:
So, seems ever for file based de-duplication some support from the FS, including some kind of ability for different inodes point to the same data blocks to store the meta-data, would be needed anyway.
The easy way is to have the inode point to a (shared, reference counted) data storage, which lists the data - then inodes can share the data, but have different meta-data.
Ever since I read about filesystems using the files' hash as addressing mechanism (per some Linus mail on LKML, about 10 years ago) and manber hashes
(http://citeseer.ist.psu.edu/manber94finding.html) I'm thinking about various ways to use both in a filesystem.
Manber-Hashes allow to split data into similar chunks, which could be addresses per some cryptographic checksum, and used by several files.
*But*: The boundaries are not at power-of-2 addresses, so the data for read()/mmap() would have to be rebuild somehow. (Would eg. be necessary anyway if the data block itself is stored compressed).
The other question I'm still pondering ... File sizes vary very much. If I have a large project, eg. the kernel, with many thousand files of some 10 kB, some data could be shared - GPL licenses in files, #include lists, and some others.
If I have some other data, with files of several hundred megabytes, sharing gets more interesting ... but what should the right block size (for manber hashes) be? If it's small, we have to concatenate a lot of blocks to reconstruct data - if it's big, we might lose many chances for sharing.
And getting good performance, when blocks in the middle of a file have to be re-splitted, might be a bit of a problem, too ...
Regards,
Phil