LVM vs btrfs as a "volume manager" for SANs
Right now, the majority of Linux users probably have LVM on their SAN devices (i.e those being iSCSI targets).
Using LVM on a SAN device is easy: just create a new logical volume or its snapshot, make it a target to iSCSI initiators, done.
I was wondering how btrfs would fit here and if it could replace LVM.
I see the following benefits of using btrfs instead of LVM:
- you can create sparse files which would grow as iSCSI initators use more space (you can do it with ext3 now as well)
- you can use btrfs compression, to further reduce used space and perhaps increase speed (SANs are mostly IO bound, not CPU bound)
- LVM has a big performance hit when using snapshots; btrfs doesn't
However, with btrfs, I'm not sure about:
- what happens if SAN machine crashes while the iSCSI file images were being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessary
- taking snapshots of individual files (file images on SAN) is not possible with btrfs? Probably they would have to be placed in separate directories first to make snapshots - some minor manageability issue
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 3 lines · Tomasz Chmielewski
- what happens if SAN machine crashes while the iSCSI file images were being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessary
I would hope that COW applies to such a setting. It certainly does for ZFS, making it an excellent backend for iSCSI. At least, having it on btrfs shouldn't make it any less reliable than on LVM, as long as btrfs does its job correctly.
Signature · 5 lines
Dmitri Nikulin Centre for Synchrotron Science Monash University Victoria 3800, Australia
Quoted 6 lines · Tomasz Chmielewski
Right now, the majority of Linux users probably have LVM on their SAN devices (i.e those being iSCSI targets).
Using LVM on a SAN device is easy: just create a new logical volume or its snapshot, make it a target to iSCSI initiators, done.
This is definitely one of the target use cases for btrfs.
Quoted 15 lines
I was wondering how btrfs would fit here and if it could replace LVM.
I see the following benefits of using btrfs instead of LVM:
- you can create sparse files which would grow as iSCSI initators use more space (you can do it with ext3 now as well)
- you can use btrfs compression, to further reduce used space and perhaps increase speed (SANs are mostly IO bound, not CPU bound)
- LVM has a big performance hit when using snapshots; btrfs doesn't
Quoted 5 lines
However, with btrfs, I'm not sure about:
- what happens if SAN machine crashes while the iSCSI file images were being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessary
If iscsi is writing with O_DIRECT|O_SYNC it should work. But, tuning for this config is something we have to concentrate more on.
Quoted 4 lines
- taking snapshots of individual files (file images on SAN) is not possible with btrfs? Probably they would have to be placed in separate directories first to make snapshots - some minor manageability issue
Btrfs can't snapshot a single dir, but it can snapshot a single file. See the bcp command included with btrfs-progs.
I'd also suggest using preallocated files (via fallocate) instead of sparse files. I will perform better in general.
-chris
Quoted 6 lines · Chris Mason
Quoted 4 lines
However, with btrfs, I'm not sure about:
- what happens if SAN machine crashes while the iSCSI file images were
being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessaryIf iscsi is writing with O_DIRECT|O_SYNC it should work.
What if it doesn't?
Quoted 11 lines
But, tuning
for this config is something we have to concentrate more on.Quoted 2 lines
- taking snapshots of individual files (file images on SAN) is not
possible with btrfs? Probably they would have to be placed in separate directories first to make snapshots - some minor manageability issueBtrfs can't snapshot a single dir, but it can snapshot a single file. See the bcp command included with btrfs-progs.
I'd also suggest using preallocated files (via fallocate) instead of sparse files. I will perform better in general.
When performance is needed - sure.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 4 lines · Tomasz Chmielewski
Quoted 2 lines · Chris Mason
Quoted 2 lines
However, with btrfs, I'm not sure about:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
- what happens if SAN machine crashes while the iSCSI file images were being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessary
If iscsi is writing with O_DIRECT|O_SYNC it should work.
What if it doesn't?
Writes would go to the page cache only (not O_DIRECT) and the metadata wouldn't get flushed with each write (not O_SYNC).
-chris
Quoted 5 lines · Chris Mason
Quoted 4 lines · Tomasz Chmielewski
Quoted 2 lines · Chris Mason
Quoted 2 lines
However, with btrfs, I'm not sure about:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
- what happens if SAN machine crashes while the iSCSI file images were
being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessaryIf iscsi is writing with O_DIRECT|O_SYNC it should work.
What if it doesn't?
Writes would go to the page cache only (not
O_DIRECT) and the metadata wouldn't get flushed with each write (notO_SYNC).
What about the "administrative" part of using btrfs as a volume manager?
For example, I can partition (fdisk + kpartx), extend volumes in LVM easily, on the host.
Are there any tools to "partition" or "extend" file images?
qemu-nbd and nbd-client come to mind, but it's a bit of an overhead (in typing, I mean).
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 10 lines · Tomasz Chmielewski
Quoted 9 lines · Chris Mason
Quoted 8 lines · Tomasz Chmielewski
Quoted 6 lines · Chris Mason
Quoted 6 lines
However, with btrfs, I'm not sure about:
- what happens if SAN machine crashes while the iSCSI file images were being written to; with LVM and its block devices, I'm somehow more confident it wouldn't make more data loss than necessary
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
If iscsi is writing with O_DIRECT|O_SYNC it should work.
What if it doesn't?
Writes would go to the page cache only (not
O_DIRECT) and the metadata wouldn't get flushed with each write (notO_SYNC).What about the "administrative" part of using btrfs as a volume manager?
In this case it depends on exactly how you do it. If you're just using files and putting filesystems on those files (not partitioning inside the image), then you can use truncate to shrink the image and fallocate (or truncate if you want it sparse) to extend it.
If you want to make partition tables inside the volumes, then you need to do the truncate/fallocate game and also change the partition table.
-chris