Some very basic questions
Quoted 3 lines
question is: if you had such an implementation, are there drawbacks expectable for the single-mount case? If not I'd vote for it because there are not really many alternatives "on the market".
As I understand it, the largest issue is in locking and boundaries. Two different systems could mount a filesystem, and try to use some sort of on-disk markers to keep from writing to the same area at the same time... but there is often some bit of time between when a system sends data to the disk and when it would become available to read from the disk, and little or no guarantee about the order in which the data is written. All the work that goes into making transactions atomic depends on there only being a single path to the disk - through the code that handles transactions. If data can arrive on the disk without being managed by that code, all bets are off.
Quoted 5 lines · calin
Quoted 3 lines
question is: if you had such an implementation, are there drawbacks expectable for the single-mount case? If not I'd vote for it because there are not really many alternatives "on the market".
As I understand it, the largest issue is in locking and boundaries.
Correct, that is the first big issue. As soon as 2 machines can access the same device, you must design for distributed locking. And that means a lot more code, lower performance, and a lot of things a local-only filesystem could do that must be disallowed.
The second issue is what is the purpose of more than 1 host accessing the data directly from the device. There are cases where this is a good thing because the application is designed with data partitioning and multi-instance coordination. It is a bad thing for random uncoordinated use like backups or fsck.
Remember that the device bandwidth is the limiter so even when each host has a dedicated path to the device (as in dual port SAS or FC), that 2nd host cuts the throughput by more than 1/2 with uncoordinated seeks and transfers.
And if the host device drivers are not designed for multiple host sharing, this can cause timeouts, resets, and false device-failed states.
And yes... even read-only access from a 2nd host is trouble in many parts of the design and does not come for free.
jim
Quoted 5 lines · jim owens
Remember that the device bandwidth is the limiter so even when each host has a dedicated path to the device (as in dual port SAS or FC), that 2nd host cuts the throughput by more than 1/2 with uncoordinated seeks and transfers.
That's only a problem if there is a single shared device. Since btrfs supports multiple devices, each host could own a device set and access from other hosts would be through the owner. You would need RDMA to get reasonable performance and some kind of dual-porting to get high availability. Each host could control the allocation tree for its devices.
Of course, this doesn't solve the other problems with parallel mounts.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 8 lines · Avi Kivity
Quoted 5 lines · jim owens
Remember that the device bandwidth is the limiter so even when each host has a dedicated path to the device (as in dual port SAS or FC), that 2nd host cuts the throughput by more than 1/2 with uncoordinated seeks and transfers.
That's only a problem if there is a single shared device. Since btrfs supports multiple devices, each host could own a device set and access from other hosts would be through the owner. You would need RDMA to get reasonable performance and some kind of dual-porting to get high availability. Each host could control the allocation tree for its devices.
No. Every device including a monster $$$ array has the problem.
As I said before, unless the application is partitioned there is always data host2 needs from host1's disk and that slows down host1.
If host2 seldom needs any host1 data, then you are describing a configuration that can be done easily by each host having a separate filesystem for the device it owns by default. Each host nfs mounts the other host's data and if host1 fails, host2 can direct mount host1-fs from the shared array.
Even with multiple disks under the same filesystem as separate allocated storage there is still the problem of shared namespace metadata that slows down both hosts. If you don't need shared namespaces then you absolutely don't want a cluster fs.
A cluster fs is useful, but the cost can be high so using it for a single-host fs is not a good idea.
jim
jim
Quoted 8 lines · jim owens
Quoted 7 lines · Avi Kivity
Quoted 5 lines · jim owens
Remember that the device bandwidth is the limiter so even when each host has a dedicated path to the device (as in dual port SAS or FC), that 2nd host cuts the throughput by more than 1/2 with uncoordinated seeks and transfers.
Quoted 7 lines
Quoted 1 line
That's only a problem if there is a single shared device. Since btrfs supports multiple devices, each host could own a device set and access from other hosts would be through the owner. You would need RDMA to get reasonable performance and some kind of dual-porting to get high availability. Each host could control the allocation tree for its devices.
No. Every device including a monster $$$ array has the problem.
As I said before, unless the application is partitioned there is always data host2 needs from host1's disk and that slows down host1.
The CPU load should not be significant if you have RDMA. Or are you talking about the seek load? Since host1's load should be distributed over all devices in the system, overall seek capacity increases as you add more nodes.
Quoted 7 lines
If host2 seldom needs any host1 data, then you are describing a configuration that can be done easily by each host having a separate filesystem for the device it owns by default. Each host nfs mounts the other host's data and if host1 fails, host2 can direct mount host1-fs from the shared array.
Separate namespaces are uninteresting to me. That's just pushing back the problem to the user.
Quoted 5 lines
Even with multiple disks under the same filesystem as separate allocated storage there is still the problem of shared namespace metadata that slows down both hosts. If you don't need shared namespaces then you absolutely don't want a cluster fs.
If you separate the allocation metadata to the storage owning node, and the file metadata to the actively using node, the slowdown should be low in most cases. Problems begin when all nodes access the same file, but that's relatively rare. Even then, when the file size does not change and when the data is preallocated, it's possible to achieve acceptable overhead.
Quoted 2 lines
A cluster fs is useful, but the cost can be high so using it for a single-host fs is not a good idea.
Development costs, yes. But I don't see why the runtime overhead can't disappear when running on a single host. Sort of like running an smp kernel on uniprocessor (I agree the fs problem is much bigger).
Signature · 1 line
error compiling committee.c: too many arguments to function