"not enough space" with "data raid0"
Hallo, linux-btrfs,
I've (once more) created my test system:
mkfs.btrfs -d raid0 -m raid1 /dev/sdb1 /dev/sdc1
73 GB + 146 GB.
Then I mounted it and copied about 150 GByte onto it. But copying was incomplete, the job ended with "no space on ..."
# btrfs fi show
Label: 'Scsi' uuid: e30586e9-a903-4d17-8ec0-1781457212c6
Total devices 2 FS bytes used 134.86GB devid 1 size 68.37GB used 68.37GB path /dev/sdb1 devid 2 size 136.73GB used 68.35GB path /dev/sdc1
Btrfs Btrfs v0.19
# btrfs fi df <mountpoint>
Data, RAID0: total=134.68GB, used=134.67GB
Data: total=8.00MB, used=8.00MB
System, RAID1: total=8.00MB, used=16.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.00GB, used=185.39MB
Metadata: total=8.00MB, used=0.00
# df <mountpoint>
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sdc1 215059324 141597364 8724 100% /mnt/btr
# fdisk -l
Disk /dev/sdb: 73.4 GB, 73407868928 bytes
Disk /dev/sdc: 146.8 GB, 146814976000 bytes
-----------------------------------
Where is the problem, how can I use the full space?
Viele Gruesse!
Helmut
Quoted 1 line · On Saturday 17 March 2012 23:01:00 Helmut Hullen wrote:
Where is the problem, how can I use the full space?
Which kernel was this with Helmut?
cheers,
Chris
Signature · 4 lines
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP
Hallo, Chris,
Du meintest am 17.03.12:
Quoted 3 lines
Quoted 1 line
Where is the problem, how can I use the full space?
Which kernel was this with Helmut?
Kernel 3.2.9 (self made)
btrfs-progs-20111030
Viele Gruesse!
Helmut
Quoted 35 lines · Helmut Hullen
Hallo, linux-btrfs,
I've (once more) created my test system:
mkfs.btrfs -d raid0 -m raid1
/dev/sdb1/dev/sdc173 GB + 146 GB.
Then I mounted it and copied about 150 GByte onto it. But copying was incomplete, the job ended with "no space on ..."
# btrfs fi showLabel: 'Scsi' uuid: e30586e9-a903-4d17-8ec0-1781457212c6
Total devices 2 FS bytes used 134.86GB devid 1 size 68.37GB used 68.37GB path /dev/sdb1 devid 2 size 136.73GB used 68.35GB path /dev/sdc1Btrfs Btrfs v0.19
# btrfs fi df <mountpoint>Data, RAID0: total=134.68GB, used=134.67GB
Data: total=8.00MB, used=8.00MB
System, RAID1: total=8.00MB, used=16.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.00GB, used=185.39MB
Metadata: total=8.00MB, used=0.00# df <mountpoint>Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdc1215059324 141597364 8724 100%/mnt/btr
Quoted 8 lines
# fdisk -lDisk
/dev/sdb: 73.4 GB, 73407868928 bytes
Disk/dev/sdc: 146.8 GB, 146814976000 bytes-----------------------------------
Where is the problem, how can I use the full space?
You can't. btrfs requires RAID-0 to be at least two devices wide
(otherwise it's not striped at all, which is the point of RAID-0). If you want to use the full capacity of both disks and don't care about the performance gain from striping, use -d single (which is the default). If you do care about the performance gain from striping, then you're going to have to lose some usable space.
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- I am but mad north-north-west: when the wind is southerly, I ---
know a hawk from a handsaw. Quoted 8 lines · Hugo Mills
Quoted 1 line
Where is the problem, how can I use the full space?
You can't. btrfs requires RAID-0 to be at least two devices wide
(otherwise it's not striped at all, which is the point of RAID-0). If you want to use the full capacity of both disks and don't care about the performance gain from striping, use -d single (which is the default). If you do care about the performance gain from striping, then you're going to have to lose some usable space.
So currently btrfs's concept of raid0 is "stripe across as many disks as possible, with a minimum of 2 disks". Is there any reason for that minimum? I don't see why it can't allow only one if that's the best it can manage.
Quoted 6 lines · Phillip Susi <psusi <at> ubuntu.com> writes:
So currently btrfs's concept of raid0 is "stripe across as many disks as possible, with a minimum of 2 disks". Is there any reason for that minimum? I don't see why it can't allow only one if that's the best it can manage.
That's called "Single". http://btrfs.ipv5.de/index.php?title=Mkfs.btrfs
Alex posted on Wed, 28 Mar 2012 22:11:01 +0000 as excerpted:
Quoted 9 lines
Quoted 5 lines · Phillip Susi <psusi <at> ubuntu.com> writes:
So currently btrfs's concept of raid0 is "stripe across as many disks as possible, with a minimum of 2 disks". Is there any reason for that minimum? I don't see why it can't allow only one if that's the best it can manage.
That's called "Single". http://btrfs.ipv5.de/index.php?title=Mkfs.btrfs
It's called "single" if you deliberately create it that way, yes.
However, Phillip's point was, I believe, that if a striped set can start with say five devices of varying size and drop one at a time as the smallest devices run out of space, there's no reason it should stop at two devices. If the goal is maximum performance, it should insist on using only the space available in parallel across all devices and should refuse to drop even a single one. If OTOH, the goal is maximum utilization of available space, with a secondary goal of performance if at all possible, then it should drop devices all the way down to a single device, not stopping at two.
Since btrfs apparently is already content to drop from say five devices to two despite the performance drop on the way, there's no fundamental reason it should stop there; it should continue all the way down to a single device, even tho when it gets to that point it's no longer actually striped.
Of course, that's defining the behavior I'd expect of a mature btrfs. Right now, it's still experimental, with many missing features. If at this point there's and arbitrary line drawn at two devices minimum, that's just the way it is. But I'd not expect that line to be there when the filesystem is declared mature and fully ready for use.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 7 lines · Helmut Hullen <Hullen <at> t-online.de> writes:
Where is the problem, how can I use the full space?
Viele Gruesse!
Helmut
Effectively it's missing the trigger to rebalance when the 'primary' device starts to get full, or just to randomly spread the data between the devices.
Isn't this going to be a problem for anyone restoring from a backup? Lots of data heading to btrfs without a pause.
Helmut, can you tell everyone which version of the kernel & patches you're running pls.
Quoted 5 lines · Alex
Quoted 3 lines · Helmut Hullen <Hullen <at> t-online.de> writes:
Quoted 7 lines
Quoted 4 lines
Where is the problem, how can I use the full space?
Viele Gruesse!
HelmutEffectively it's missing the trigger to rebalance when the 'primary' device starts to get full, or just to randomly spread the data between the devices.
No, a balance isn't going to help here. RAID-0 requires a minimum
of 2 chunks in a block group. With two disks, you're only going to be able to fill the smallest one before you run out of space.
Quoted 2 lines
Isn't this going to be a problem for anyone restoring from a backup? Lots of data heading to btrfs without a pause.
Again, not the issue here.
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- I am but mad north-north-west: when the wind is southerly, I ---
know a hawk from a handsaw. Quoted 6 lines
No, a balance isn't going to help here. RAID-0 requires a minimum
of 2 chunks in a block group. With two disks, you're only going to be able to fill the smallest one before you run out of space.
Quoted 7 lines
Quoted 2 lines
Isn't this going to be a problem for anyone restoring from a backup? Lots of data heading to btrfs without a pause.
Again, not the issue here.
Hugo.
Thank you HH for the explanation.
What's the 'solution' though to Hugo's situation?
By 'solution' I mean the highest-utility way of dealing with unequal devices problem in a two or more -up setting.
(I understand that we're violating the expectations of RAID0 and have read the
http://btrfs.ipv5.de/index.php?title=FAQ#How_much_space_do_I_get_with_unequal_devices_in_RAID-1_mode.3F
about RAID1.)
Disks are (were) getting cheaper per MB faster than I could run out of space! So I have increasing size array of drives - apparently the similar situation as Hugo (I have three btrfs-ready drives).
I'm struggling how to mkfs.btrfs with this new knowledge whilst resolving it with the btrfs storage pool concept.
I stress there is a difficulty with my understanding not btrfs.
Kind regards.
Al.
Quoted 7 lines · Alex
Quoted 7 lines
No, a balance isn't going to help here. RAID-0 requires a minimum
of 2 chunks in a block group. With two disks, you're only going to be able to fill the smallest one before you run out of space.Quoted 2 lines
Isn't this going to be a problem for anyone restoring from a backup? Lots of data heading to btrfs without a pause.
Quoted 7 lines
Quoted 1 line
Again, not the issue here.
Thank you HH for the explanation.
What's the 'solution' though to Hugo's situation?
By 'solution' I mean the highest-utility way of dealing with unequal devices problem in a two or more -up setting.
Use mkfs.btrfs -d single, as I said in another part of this thread.
We have a bit of a problem right now with adding a second device to
an existing FS with -d single, where the FS "upgrades" the single to RAID-0. However, there's patches from Ilya that stop that behaviour, and the restriper can be used to switch back to single after the "upgrade".
Quoted 4 lines
(I understand that we're violating the expectations of RAID0 and have read the
http://btrfs.ipv5.de/index.php?title=FAQ#How_much_space_do_I_get_with_unequal_devices_in_RAID-1_mode.3F
about RAID1.)
I'm working on a comprehensive explanation of the limits on btrfs's
space usage, and a little JavaScript tool to help plan/explain disk usage.
Quoted 3 lines
Disks are (were) getting cheaper per MB faster than I could run out of space! So I have increasing size array of drives - apparently the similar situation as Hugo (I have three btrfs-ready drives).
I don't have this problem personally. I just know something about
the way btrfs allocates chunks. I think you meant Helmut has the problem. :)
Quoted 4 lines
I'm struggling how to mkfs.btrfs with this new knowledge whilst resolving it with the btrfs storage pool concept.
I stress there is a difficulty with my understanding not btrfs.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- UNIX: Spanish manufacturer of fire extinguishers. --- Hallo, Hugo,
Du meintest am 17.03.12:
Quoted 6 lines
Quoted 3 lines
What's the 'solution' though to Hugo's situation?
By 'solution' I mean the highest-utility way of dealing with unequal devices problem in a two or more -up setting.Use mkfs.btrfs -d single, as I said in another part of this
thread.
Does "single" allow adding new (bigger) disks and removing old (smaller) disks?
Viele Gruesse!
Helmut
Quoted 8 lines · Helmut Hullen
Hallo, Hugo,
Du meintest am 17.03.12:
Quoted 3 lines
Quoted 3 lines
What's the 'solution' though to Hugo's situation?
By 'solution' I mean the highest-utility way of dealing with unequal devices problem in a two or more -up setting.
Quoted 5 lines
Quoted 2 lines
Use mkfs.btrfs -d single, as I said in another part of this
thread.Does "single" allow adding new (bigger) disks and removing old (smaller) disks?
Yes. "single" is effectively "RAID linear" -- just adding the disks
together, effectively linearly(*). It's called "single" because it's single (i.e. no) redundancy.
Hugo.
(*) actually the allocator will kind of "stripe" the data in 1GiB-wide stripes, but that's incidental really.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- A diverse working environment: Di longer you vork here, di ---
verse it gets. Quoted 8 lines · Hugo Mills <hugo <at> carfax.org.uk> writes:
On Sat, Mar 17, 2012 at 01:36:04PM +0000, Alex wrote:
I don't have this problem personally. I just know something about
the way btrfs allocates chunks. I think you meant Helmut has the problem. :)
Quoted 7 lines
Quoted 4 lines
I'm struggling how to mkfs.btrfs with this new knowledge whilst resolving it with the btrfs storage pool concept.
I stress there is a difficulty with my understanding not btrfs.
Hugo.
Yes, I meant Hugo .. damn .. Helmut, sorry! However deed-poll is a cost effective measure which would allow you to avoid the coincident first letter with Helmut. ;-)
Since we're still experimental in the non-SUSE and Oracle world and the space meta demands sb lower can I squeak by:
mkfs.btrfs -d single -m raid1 ?
Don't laugh.
Quoted 4 lines
I'm working on a comprehensive explanation of the limits on btrfs's
space usage, and a little JavaScript tool to help plan/explain disk usage.
Thank you Hugo.
Given the kernel 3.3 release yesterday [1] we've gained the ability to restripe to different raid levels after the fact. So I guess I don't have too much to worry about with decent backups.
The captcha word for this post is "baptizes". Quite apt (no pun intended) really!
Quoted 10 lines · Alex
Quoted 4 lines
I'm working on a comprehensive explanation of the limits on btrfs's
space usage, and a little JavaScript tool to help plan/explain disk usage.Thank you Hugo.
Given the kernel 3.3 release yesterday [1] we've gained the ability to restripe to different raid levels after the fact. So I guess I don't have too much to worry about with decent backups.
Yes, you do. RAID is not a backup: What happens if you accidentally
write zeroes over all your superblocks? Or rm -rf /home? RAID will not help you here.
Hugo.
Signature · 5 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- In the future, terrorists won't be carrying their ID cards. ---
They'll be carrying yours. -- Henry
Porter, Suspect Nation SGkgSHVnbywgDQoNCkkgZGlkIG1lbnRpb24gYmFja3VwcyBzaXIgaW4gbXkgbGFzdCBzZW50ZW5j ZSwgYnV0IHRoYW5rIHlvdSBhbnl3YXkuICBObyBoYXJtIGluIHJlcGVhdGluZyB0aGUgcG9pbnQu IA0KDQoNCg0KQmxhY2tiZXJyeSBQSU46MjY4YTI5NTMNClNlbnQgZnJvbSBteSBCbGFja0JlcnJ5 rg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogSHVnbyBNaWxscyA8aHVnb0Bj YXJmYXgub3JnLnVrPg0KRGF0ZTogTW9uLCAxOSBNYXIgMjAxMiAxNzo1ODo1MCANClRvOiBBbGV4 PGFsZXhAYnBtaXQuY29tPg0KQ2M6IDxsaW51eC1idHJmc0B2Z2VyLmtlcm5lbC5vcmc+DQpTdWJq ZWN0OiBSZTogIm5vdCBlbm91Z2ggc3BhY2UiIHdpdGggImRhdGEgcmFpZDAiDQoNCk9uIE1vbiwg TWFyIDE5LCAyMDEyIGF0IDAzOjE0OjE5UE0gKzAwMDAsIEFsZXggd3JvdGU6DQo+IA0KPiA+ICAg IEknbSB3b3JraW5nIG9uIGEgY29tcHJlaGVuc2l2ZSBleHBsYW5hdGlvbiBvZiB0aGUgbGltaXRz IG9uIGJ0cmZzJ3MNCj4gPiBzcGFjZSB1c2FnZSwgYW5kIGEgbGl0dGxlIEphdmFTY3JpcHQgdG9v bCB0byBoZWxwIHBsYW4vZXhwbGFpbiBkaXNrDQo+ID4gdXNhZ2UuDQo+IA0KPiBUaGFuayB5b3Ug SHVnby4NCj4gDQo+IEdpdmVuIHRoZSBrZXJuZWwgMy4zIHJlbGVhc2UgeWVzdGVyZGF5IFsxXSB3 ZSd2ZSBnYWluZWQgdGhlIGFiaWxpdHkgdG8gcmVzdHJpcGUNCj4gdG8gZGlmZmVyZW50IHJhaWQg bGV2ZWxzIGFmdGVyIHRoZSBmYWN0LiBTbyBJIGd1ZXNzIEkgZG9uJ3QgaGF2ZSB0b28gbXVjaCB0 bw0KPiB3b3JyeSBhYm91dCB3aXRoIGRlY2VudCBiYWNrdXBzLg0KDQogICBZZXMsIHlvdSBkby4g UkFJRCBpcyBub3QgYSBiYWNrdXA6IFdoYXQgaGFwcGVucyBpZiB5b3UgYWNjaWRlbnRhbGx5DQp3 cml0ZSB6ZXJvZXMgb3ZlciBhbGwgeW91ciBzdXBlcmJsb2Nrcz8gT3Igcm0gLXJmIC9ob21lPyBS QUlEIHdpbGwgbm90DQpoZWxwIHlvdSBoZXJlLg0KDQogICBIdWdvLg0KDQotLSANCj09PSBIdWdv IE1pbGxzOiBodWdvQC4uLiBjYXJmYXgub3JnLnVrIHwgZGFya3NhdGFuaWMubmV0IHwgbHVnLm9y Zy51ayA9PT0NCiAgUEdQIGtleTogNTE1QzIzOEQgZnJvbSB3d3drZXlzLmV1LnBncC5uZXQgb3Ig aHR0cDovL3d3dy5jYXJmYXgub3JnLnVrDQogICAtLS0gSW4gdGhlIGZ1dHVyZSwgdGVycm9yaXN0 cyB3b24ndCBiZSBjYXJyeWluZyB0aGVpciBJRCBjYXJkcy4gLS0tICAgDQogICAgICAgICAgICAg ICAgICAgVGhleSdsbCBiZSBjYXJyeWluZyB5b3Vycy4gLS0gSGVucnkgICAgICAgICAgICAgICAg ICAgDQogICAgICAgICAgICAgICAgICAgICAgICAgUG9ydGVyLCAgU3VzcGVjdCBOYXRpb24gICAg ICAgICAgICAgICAgICAgICAgICAgDQoNCg==
Hallo, Hugo,
Du meintest am 17.03.12:
[no space left on device ...]
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Where is the problem, how can I use the full space?
Effectively it's missing the trigger to rebalance when the 'primary' device starts to get full, or just to randomly spread the data between the devices.
No, a balance isn't going to help here. RAID-0 requires a minimum
of 2 chunks in a block group. With two disks, you're only going to be able to fill the smallest one before you run out of space.
Ok - it happens only with 2 disks/Partitions?
I've continued playing; added a 3rd partition/device and then balanced:
# btrfs device add /dev/sdd1 /mnt/btr ## 73 + 146 + 146 GByte
# df
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sdc1 358432300 225400136 59842800 80% /mnt/btr
# added about 70 GByte (all together more than 3 times the smallest partition)
# btrfs fi show
Label: 'Scsi' uuid: e30586e9-a903-4d17-8ec0-1781457212c6
Total devices 3 FS bytes used 214.67GB devid 1 size 68.37GB used 68.37GB path /dev/sdb1 devid 3 size 136.73GB used 41.01GB path /dev/sdd1 devid 2 size 136.73GB used 109.36GB path /dev/sdc1
Btrfs Btrfs v0.19
# btrfs fi df /mnt/btr
Data, RAID0: total=216.71GB, used=214.38GB
System, RAID1: total=8.00MB, used=24.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.00GB, used=295.13MB
=================================================================
# btrfs fi balance /mnt/btr
# btrfs fi show
Label: 'Scsi' uuid: e30586e9-a903-4d17-8ec0-1781457212c6
Total devices 3 FS bytes used 214.67GB devid 1 size 68.37GB used 67.34GB path /dev/sdb1 devid 3 size 136.73GB used 40.85GB path /dev/sdd1 devid 2 size 136.73GB used 107.85GB path /dev/sdc1
Btrfs Btrfs v0.19
# btrfs fi df /mnt/btr
Data, RAID0: total=215.01GB, used=214.38GB
System, RAID1: total=8.00MB, used=24.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=512.00MB, used=294.88MB
----------------------------------------------------------------------
Looks as desired, the 3-disks-system contains more than 3 times the smallest disk.
Balancing hasn't redistributed the contents - no problem.
By the way: you should name the prefixes in the NIST way for powers of 2: KiB, MiB, GiB. Or change to decimal prefixes.
Viele Gruesse!
Helmut
Quoted 12 lines · Helmut Hullen
Hallo, Hugo,
Du meintest am 17.03.12:
[no space left on device ...]
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Where is the problem, how can I use the full space?
Effectively it's missing the trigger to rebalance when the 'primary' device starts to get full, or just to randomly spread the data between the devices.
Quoted 5 lines
Quoted 3 lines
No, a balance isn't going to help here. RAID-0 requires a minimum
of 2 chunks in a block group. With two disks, you're only going to be able to fill the smallest one before you run out of space.Ok - it happens only with 2 disks/Partitions?
Not quite. With the current RAID-0 implementation, you will lose
the difference between the largest disk and the second-largest. There's a proposal from November that may help with that, but I haven't had a chance to look at it yet to see what the implications are.
Quoted 5 lines
I've continued playing; added a 3rd partition/device and then balanced:
# btrfs device add /dev/sdd1 /mnt/btr ## 73 + 146 + 146 GByte
OK, so in this case the largest and second-largest devices are the
same, so you will lose no space.
[snip]
Quoted 7 lines
Looks as desired, the 3-disks-system contains more than 3 times the smallest disk.
Balancing hasn't redistributed the contents - no problem.
By the way: you should name the prefixes in the NIST way for powers of 2: KiB, MiB, GiB. Or change to decimal prefixes.
I know. It's something I submitted patches for a long time ago, and
they never got taken up. It's on my list of things to fix.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- UNIX: Spanish manufacturer of fire extinguishers. ---