Btrfs raid allocator
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
Cheers
Hendrik
Quoted 9 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
For (current) RAID-0 allocation, the block group allocator will use
as many chunks as there are devices with free space (down to a minimum of 2). Data is then striped across those chunks in 64 KiB stripes. Thus, the first block group will be N GiB of usable space, striped across N devices.
There's a second level of allocation (which I haven't looked at at
all), which is how the FS decides where to put data within the allocated block groups. I think it will almost certainly be beneficial in your case to use prealloc extents, which will turn your continuous write into large contiguous sections of striping.
I would recommend thoroughly benchmarking your application with the
FS first though, just to see how it's going to behave for you.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Ceci n'est pas une pipe: | --- Quoted 11 lines · Hugo Mills
Quoted 9 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
Quoted 5 lines
For (current) RAID-0 allocation, the block group allocator will use
as many chunks as there are devices with free space (down to a minimum of 2). Data is then striped across those chunks in 64 KiB stripes. Thus, the first block group will be N GiB of usable space, striped across N devices.
So do I understand this correctly that (assuming we have enough space) data will be spread equally between the disks independend of write speeds? So one slow device would slow down the whole raid?
Quoted 5 lines
There's a second level of allocation (which I haven't looked at at
all), which is how the FS decides where to put data within the allocated block groups. I think it will almost certainly be beneficial in your case to use prealloc extents, which will turn your continuous write into large contiguous sections of striping.
Why does prealloc change anything? For me latency does not matter, only continuous troughput!
Quoted 5 lines
I would recommend thoroughly benchmarking your application with the
FS first though, just to see how it's going to behave for you.Hugo.
Of course - it's just that I do not yet have the hardware, but I plan to test with a small model - I just try to find out how it actually works first, so I know what look out for.
Hendrik
Quoted 18 lines · Hendrik Siedelmann
Quoted 11 lines · Hugo Mills
Quoted 9 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
Quoted 5 lines
For (current) RAID-0 allocation, the block group allocator will use
as many chunks as there are devices with free space (down to a minimum of 2). Data is then striped across those chunks in 64 KiB stripes. Thus, the first block group will be N GiB of usable space, striped across N devices.
Quoted 3 lines
So do I understand this correctly that (assuming we have enough space) data will be spread equally between the disks independend of write speeds? So one slow device would slow down the whole raid?
Yes. Exactly the same as it would be with DM RAID-0 on the same
configuration. There's not a lot we can do about that at this point.
Quoted 8 lines
Quoted 5 lines
There's a second level of allocation (which I haven't looked at at
all), which is how the FS decides where to put data within the allocated block groups. I think it will almost certainly be beneficial in your case to use prealloc extents, which will turn your continuous write into large contiguous sections of striping.Why does prealloc change anything? For me latency does not matter, only continuous troughput!
It makes the extent allocation algorithm much simpler, because it
can then allocate in larger chunks and do more linear writes
Quoted 9 lines
Quoted 5 lines
I would recommend thoroughly benchmarking your application with the
FS first though, just to see how it's going to behave for you.Hugo.
Of course - it's just that I do not yet have the hardware, but I plan to test with a small model - I just try to find out how it actually works first, so I know what look out for.
Good luck. :)
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- "I am the author. You are the audience. I outrank you!" --- Quoted 19 lines · Hugo Mills
Quoted 18 lines · Hendrik Siedelmann
Quoted 16 lines · Hugo Mills
Quoted 9 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
For (current) RAID-0 allocation, the block group allocator will use
as many chunks as there are devices with free space (down to a minimum of 2). Data is then striped across those chunks in 64 KiB stripes. Thus, the first block group will be N GiB of usable space, striped across N devices.
Quoted 6 lines
Quoted 3 lines
So do I understand this correctly that (assuming we have enough space) data will be spread equally between the disks independend of write speeds? So one slow device would slow down the whole raid?
Yes. Exactly the same as it would be with DM RAID-0 on the same
configuration. There's not a lot we can do about that at this point.
So striping is fixed but which disk takes part with a chunk is dynamic? But for large workloads slower disks could 'skip a chunk' as chunk allocation is dynamic, correct?
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
There's a second level of allocation (which I haven't looked at at
all), which is how the FS decides where to put data within the allocated block groups. I think it will almost certainly be beneficial in your case to use prealloc extents, which will turn your continuous write into large contiguous sections of striping.Why does prealloc change anything? For me latency does not matter, only continuous troughput!
It makes the extent allocation algorithm much simpler, because it
can then allocate in larger chunks and do more linear writes
Is this still true if I do very large writes? Or do those get broken down by the kernel somewhere?
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I would recommend thoroughly benchmarking your application with the
FS first though, just to see how it's going to behave for you.Hugo.
Quoted 8 lines
Quoted 3 lines
Of course - it's just that I do not yet have the hardware, but I plan to test with a small model - I just try to find out how it actually works first, so I know what look out for.
Good luck. :)
Hugo.
Thanks!
Hendrik
Quoted 24 lines · Hendrik Siedelmann
Quoted 23 lines · Hugo Mills
Quoted 21 lines · Hendrik Siedelmann
Quoted 11 lines · Hugo Mills
Quoted 9 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
Quoted 5 lines
For (current) RAID-0 allocation, the block group allocator will use
as many chunks as there are devices with free space (down to a minimum of 2). Data is then striped across those chunks in 64 KiB stripes. Thus, the first block group will be N GiB of usable space, striped across N devices.So do I understand this correctly that (assuming we have enough space) data will be spread equally between the disks independend of write speeds? So one slow device would slow down the whole raid?
Quoted 6 lines
Quoted 2 lines
Yes. Exactly the same as it would be with DM RAID-0 on the same
configuration. There's not a lot we can do about that at this point.So striping is fixed but which disk takes part with a chunk is dynamic? But for large workloads slower disks could 'skip a chunk' as chunk allocation is dynamic, correct?
You'd have to rewrite the chunk allocator to do this, _and_ provide
different RAID levels for different subvolumes. The chunk/block group allocator right now uses only one rule for allocating data, and one for allocating metadata. Now, both of these are planned, and _might_ between them possibly cover the use-case you're talking about, but I'm not certain it's necessarily a sensible thing to do in this case.
My question is, if you actually care about the performance of this
system, why are you buying some slow devices to drag the performance of your fast devices down? It seems like a recipe for disaster...
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
There's a second level of allocation (which I haven't looked at at
all), which is how the FS decides where to put data within the allocated block groups. I think it will almost certainly be beneficial in your case to use prealloc extents, which will turn your continuous write into large contiguous sections of striping.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Why does prealloc change anything? For me latency does not matter, only continuous troughput!
It makes the extent allocation algorithm much simpler, because it
can then allocate in larger chunks and do more linear writesIs this still true if I do very large writes? Or do those get broken down by the kernel somewhere?
I guess it'll depend on the approach you use to do these "very
large" writes, and on the exact definition of "very large". This is not an area I know a huge amount about.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- "I am the author. You are the audience. I outrank you!" --- Quoted 32 lines · Hugo Mills
Quoted 24 lines · Hendrik Siedelmann
Quoted 23 lines · Hugo Mills
Quoted 21 lines · Hendrik Siedelmann
Quoted 11 lines · Hugo Mills
Quoted 9 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk. So my question is, how is the block allocator deciding on which device to write, can this decision be dynamic and could it incorporate timing/troughput decisions? I'm willing to write code, I just have no clue as to how this works right now. I read somewhere that the decision is based on free space, is this still true?
Quoted 5 lines
For (current) RAID-0 allocation, the block group allocator will use
as many chunks as there are devices with free space (down to a minimum of 2). Data is then striped across those chunks in 64 KiB stripes. Thus, the first block group will be N GiB of usable space, striped across N devices.So do I understand this correctly that (assuming we have enough space) data will be spread equally between the disks independend of write speeds? So one slow device would slow down the whole raid?
Quoted 6 lines
Quoted 2 lines
Yes. Exactly the same as it would be with DM RAID-0 on the same
configuration. There's not a lot we can do about that at this point.So striping is fixed but which disk takes part with a chunk is dynamic? But for large workloads slower disks could 'skip a chunk' as chunk allocation is dynamic, correct?
Quoted 6 lines
You'd have to rewrite the chunk allocator to do this, _and_ provide
different RAID levels for different subvolumes. The chunk/block group allocator right now uses only one rule for allocating data, and one for allocating metadata. Now, both of these are planned, and _might_ between them possibly cover the use-case you're talking about, but I'm not certain it's necessarily a sensible thing to do in this case.
But what does the allocator currently do when one disk runs out of space? I thought those disks do not get used but we can still write data. So the mechanism is already there, it just needs to be invoked when a drive is too busy instead of too full.
Quoted 3 lines
My question is, if you actually care about the performance of this
system, why are you buying some slow devices to drag the performance of your fast devices down? It seems like a recipe for disaster...
Even the speed of a single hdd varies depending on where I write the data. So actually there is not much choice :-D.
I'm aware that this could be a case of overengineering. Actually my first thought was to write a simple fuse module which only handles data and puts metadata on a regular filesystem. But then I thought that it would be nice to have this in btrfs - and not just for raid0.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
There's a second level of allocation (which I haven't looked at at
all), which is how the FS decides where to put data within the allocated block groups. I think it will almost certainly be beneficial in your case to use prealloc extents, which will turn your continuous write into large contiguous sections of striping.Why does prealloc change anything? For me latency does not matter, only continuous troughput!
It makes the extent allocation algorithm much simpler, because it
can then allocate in larger chunks and do more linear writes
Quoted 9 lines
Quoted 2 lines
Is this still true if I do very large writes? Or do those get broken down by the kernel somewhere?
I guess it'll depend on the approach you use to do these "very
large" writes, and on the exact definition of "very large". This is not an area I know a huge amount about.Hugo.
Never mind I'll just try it out!
Hendrik
Hendrik Siedelmann posted on Tue, 06 May 2014 12:41:38 +0200 as excerpted:
Quoted 3 lines
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk.
If flexible parallelization is all you're worried about, not data integrity or the other things btrfs does, I'd suggest looking at a more mature solution such as md- or dm-raid. They're more mature and less complex than btrfs, and if you're not using the other features of btrfs anyway, they should simply work better for your use-case.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 3 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk.
I think the only way to know what works best for your workload is to test configurations with the actual workload. For optimization of multiple device file systems, it's hard to beat XFS on raid0 or even linear/concat due to its parallelization, if you have more than one stream (or a stream that produces a lot of files that XFS can allocate into separate allocation groups). Also mdadm supports use specified strip/chunk sizes, whereas currently on Btrfs this is fixed to 64KiB. Depending on the file size for your workload, it's possible a much larger strip will yield better performance.
Another optimization is hardware RAID with a battery backed write cache (the drives' write cache are disabled) and using nobarrier mount option. If your workload supports linear/concat then it's fine to use md linear for this. What I'm not sure of is if it's an OK practice to disable barriers if the system is on a UPS (rather than a battery backed hardware RAID cache). You should post the workload and hardware details on the XFS list to get suggestions about such things. They'll also likely recommend the deadline scheduler over cfq.
Unless you have a workload really familiar to the responder, they'll tell you any benchmarking you do needs to approximate the actual workflow. A mismatched benchmark to the workload will lead you to the wrong conclusions. Typically when you optimize for a particular workload, other workloads suffer.
Chris Murphy
Quoted 10 lines · Chris Murphy
Quoted 5 lines · Hendrik Siedelmann
Hello all!
I would like to use btrfs (or anyting else actually) to maximize raid0 performance. Basically I have a relatively constant stream of data that simply has to be written out to disk.
Quoted 9 lines
I think the only way to know what works best for your workload is to test configurations with the actual workload. For optimization of multiple device file systems, it's hard to beat XFS on raid0 or even linear/concat due to its parallelization, if you have more than one stream (or a stream that produces a lot of files that XFS can allocate into separate allocation groups). Also mdadm supports use specified strip/chunk sizes, whereas currently on Btrfs this is fixed to 64KiB. Depending on the file size for your workload, it's possible a much larger strip will yield better performance.
Thanks, that's quite a few knobs I can try out - I just have a lot of data - with a rate up to 450MB/s that I want to write out in time, preferably without having to rely on too expensive hardware.
Quoted 9 lines
Another optimization is hardware RAID with a battery backed write cache (the drives' write cache are disabled) and using nobarrier mount option. If your workload supports linear/concat then it's fine to use md linear for this. What I'm not sure of is if it's an OK practice to disable barriers if the system is on a UPS (rather than a battery backed hardware RAID cache). You should post the workload and hardware details on the XFS list to get suggestions about such things. They'll also likely recommend the deadline scheduler over cfq.
Actually data integrity does not matter for the workload. If everything is succesfull the result will be backed up - before that full filesystem corruption is acceptable as a failure mode.
Quoted 8 lines
Unless you have a workload really familiar to the responder, they'll tell you any benchmarking you do needs to approximate the actual workflow. A mismatched benchmark to the workload will lead you to the wrong conclusions. Typically when you optimize for a particular workload, other workloads suffer.
Chris Murphy
Thanks again for all the infos! I'll get back if everything works fine - or if it doesn't ;-)
Cheers
Hendrik