Multi-device update
Hello everyone,
I've pushed out another set of changes to the unstable trees, and these include:
* RAID 1+0 support
mkfs.btrfs -d raid10 -m raid10 /dev/sd...
4 or more drives required.
* async work queues for checksumming writes
* Better back references in the multi-device data structs
The async work queues include code to checksum data pages without the FS mutex held, greatly increasing streaming write throughput. On my 4 drive system, I was getting around 120MB/s writes with checksumming on. Now I get 180MB/s, which is disk speed.
The rest of the week will be spent doing hot add/remove of devices. Happy testing ;)
-chris
Quoted 2 lines · Chris Mason
The async work queues include code to checksum data pages without the FS mutex
Are they able to distribute work to other cores?
-Andi
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Chris Mason
The async work queues include code to checksum data pages without the FS mutex
Are they able to distribute work to other cores?
Yes, it just uses a workqueue. The current implemention is pretty simple, it surely could be more effective at spreading the work around.
I'm testing a variant that only tosses over to the async queue for pdflush, inline reclaim should stay inline.
-chris
Quoted 8 lines · Chris Mason
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Chris Mason
The async work queues include code to checksum data pages without the FS mutex
Are they able to distribute work to other cores?
Yes, it just uses a workqueue.
Unfortunately work queues don't do that by default currently. They tend to process on the current CPU only.
Quoted 5 lines
The current implemention is pretty simple, it surely could be more effective at spreading the work around.
I'm testing a variant that only tosses over to the async queue for pdflush, inline reclaim should stay inline.
Longer term I would hope that write checksum will be basically free by doing csum-copy at write() time. The only problem is just where to store the checksum between the write and the final IO? There's no space in struct page.
The same could be also done for read() but that might be a little more tricky because it would require delayed error reporting and it might be difficult to do this for partial blocks?
-Andi
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Chris Mason
Quoted 4 lines · Andi Kleen
Quoted 2 lines · Chris Mason
The async work queues include code to checksum data pages without the FS mutex
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Are they able to distribute work to other cores?
Yes, it just uses a workqueue.
Unfortunately work queues don't do that by default currently. They tend to process on the current CPU only.
Well, I see multiple work queue threads using CPU time, but I haven't spent much time optimizing it. There's definitely room for improvement.
Quoted 11 lines
Quoted 6 lines
The current implemention is pretty simple, it surely could be more effective at spreading the work around.
I'm testing a variant that only tosses over to the async queue for pdflush, inline reclaim should stay inline.
Longer term I would hope that write checksum will be basically free by doing csum-copy at
write()time. The only problem is just where to store the checksum between the write and the final IO? There's no space in struct page.
At write time is easier (except for mmap) because I can toss the csum directly into the btree inside btrfs_file_write. The current code avoids that complexity and does it all at writeout.
One advantage to the current code is that I'm able to optimize tree searches away but checksumming a bunch of pages at a time. Multiple pages worth of checksums get stored in a single btree item, so at least for btree operations the current code is fairly optimal.
Quoted 4 lines
The same could be also done for
read()but that might be a little more tricky because it would require delayed error reporting and it might be difficult to do this for partial blocks?
Yeah, it doesn't quite fit with how the kernel does reads. For now it is much easier if the retry-other-mirror operation happens long before copy_to_user.
-chris
Quoted 6 lines · Chris Mason
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Chris Mason
Quoted 3 lines · Andi Kleen
Quoted 2 lines · Chris Mason
The async work queues include code to checksum data pages without the FS mutex
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
Are they able to distribute work to other cores?
Yes, it just uses a workqueue.
Unfortunately work queues don't do that by default currently. They tend to process on the current CPU only.
Well, I see multiple work queue threads using CPU time, but I haven't spent much time optimizing it. There's definitely room for improvement.
That's likely because you submit from multiple CPUs. But with a single submitter running on a single CPU there shouldn't be any load balancing currently.
-Andi
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Chris Mason
Quoted 9 lines · Andi Kleen
Quoted 6 lines · Chris Mason
Quoted 4 lines · Andi Kleen
Quoted 2 lines · Chris Mason
The async work queues include code to checksum data pages without the FS mutex
Are they able to distribute work to other cores?
Yes, it just uses a workqueue.
Unfortunately work queues don't do that by default currently. They tend to process on the current CPU only.
Quoted 6 lines
Quoted 2 lines
Well, I see multiple work queue threads using CPU time, but I haven't spent much time optimizing it. There's definitely room for improvement.
That's likely because you submit from multiple CPUs. But with a single submitter running on a single CPU there shouldn't be any load balancing currently.
There have been various implementations of queue_work_on() posted through the years, I've had one version that I've used off and on for a long time:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=c68c42fd6df96f5b3fb5b8b47c571f233d054c71
then you need some balancing decider on top of that, of course.
Signature · 1 line
Jens Axboe
Quoted 4 lines
There have been various implementations of
queue_work_on()posted through the years, I've had one version that I've used off and on for a long time:
queue_work_on is the wrong interface I think. You rather want a pool of non pinned threads that are then load balanced by the scheduler (who knows best what cpus have cycles available)
-Andi
Quoted 8 lines · Andi Kleen
Quoted 4 lines
There have been various implementations of
queue_work_on()posted through the years, I've had one version that I've used off and on for a long time:
queue_work_onis the wrong interface I think. You rather want a pool of non pinned threads that are then load balanced by the scheduler (who knows best what cpus have cycles available)
Yeah, that actually sounds like the best interface. What I described typically ends up trying to be too clever, you really want to leave any scheduling decisions to the scheduler.
Signature · 1 line
Jens Axboe
Quoted 7 lines · Andi Kleen
Quoted 3 lines
There have been various implementations of
queue_work_on()posted through the years, I've had one version that I've used off and on for a long time:
queue_work_onis the wrong interface I think. You rather want a pool of non pinned threads that are then load balanced by the scheduler (who knows best what cpus have cycles available)
Fair enough, I'll tune things a bit.
-chris