Cross-subvolume rename behavior
Hello, all. I'm currently tracking down the source of some strange behavior in my setup. I recognize that this isn't strictly a btrfs issue, but I figured I'd start at the bottom of the stack and work my way up.
I have a server with a btrfs filesystem on it that I remotely access on several systems via an sshfs mount. For the most part this works perfectly, but I just discovered that moving files between subvolumes on that mount fails with a confusing "Operation not permitted" error.
After doing some digging, it turns out it's not actually a permissions error. If I do the same operation locally on the server, it succeeds, but an strace of the mv reveals that the rename() syscall returns EXDEV. The mv util takes this as a sign to fall back on the copy-and-delete routine, so the move succeeds. Unfortunately, it seems that somewhere in sshfs, sftp, or fuse, the EXDEV is getting turned into a generic failure, which mv apparently interprets as "permission denied".
So my question for the btrfs devs: is rename()-ing across subvolumes not feasible, or is this simply a case of no one has implemented that yet?
Thanks for any insights,
--Sean
Quoted 18 lines · Sean Greenslade
Hello, all. I'm currently tracking down the source of some strange behavior in my setup. I recognize that this isn't strictly a btrfs issue, but I figured I'd start at the bottom of the stack and work my way up.
I have a server with a btrfs filesystem on it that I remotely access on several systems via an sshfs mount. For the most part this works perfectly, but I just discovered that moving files between subvolumes on that mount fails with a confusing "Operation not permitted" error.
After doing some digging, it turns out it's not actually a permissions error. If I do the same operation locally on the server, it succeeds, but an strace of the mv reveals that the
rename()syscall returns EXDEV. The mv util takes this as a sign to fall back on the copy-and-delete routine, so the move succeeds. Unfortunately, it seems that somewhere in sshfs, sftp, or fuse, the EXDEV is getting turned into a generic failure, which mv apparently interprets as "permission denied".
Quoted 3 lines
So my question for the btrfs devs: is rename()-ing across subvolumes not feasible, or is this simply a case of no one has implemented that yet?
Direct rename (using rename(2)) isn't possible across subvols,
which is what the EXDEV result indicates. The solution is exactly what mv does, which is reflink-and-delete (which is cheaper than copy-and-delete, because no data is moved). In theory, you probably could implement rename across subvolumes in the FS, but it would just be moving the exact same operations from userspace to kernel space.
I think that the solution here is for the sshfs stack to be fixed
so that it passes the EXDEV up to the mv command properly, and passes the subsequent server-side copy (reflink) back down correctly.
Hugo.
Signature · 4 lines
Hugo Mills | I'm on a 30-day diet. So far I've lost 18 days. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 23 lines · Hugo Mills
Quoted 18 lines · Sean Greenslade
Hello, all. I'm currently tracking down the source of some strange behavior in my setup. I recognize that this isn't strictly a btrfs issue, but I figured I'd start at the bottom of the stack and work my way up.
I have a server with a btrfs filesystem on it that I remotely access on several systems via an sshfs mount. For the most part this works perfectly, but I just discovered that moving files between subvolumes on that mount fails with a confusing "Operation not permitted" error.
After doing some digging, it turns out it's not actually a permissions error. If I do the same operation locally on the server, it succeeds, but an strace of the mv reveals that the
rename()syscall returns EXDEV. The mv util takes this as a sign to fall back on the copy-and-delete routine, so the move succeeds. Unfortunately, it seems that somewhere in sshfs, sftp, or fuse, the EXDEV is getting turned into a generic failure, which mv apparently interprets as "permission denied".Quoted 3 lines
So my question for the btrfs devs: is rename()-ing across subvolumes not feasible, or is this simply a case of no one has implemented that yet?
Quoted 6 lines
Direct rename (using rename(2)) isn't possible across subvols,
which is what the EXDEV result indicates. The solution is exactly what mv does, which is reflink-and-delete (which is cheaper than copy-and-delete, because no data is moved). In theory, you probably could implement rename across subvolumes in the FS, but it would just be moving the exact same operations from userspace to kernel space.
Doing so though would have the advantage that it theoretically could be made (almost) atomic like a normal rename is, whereas the fallback in mv is absolutely not atomic.
Quoted 4 lines
I think that the solution here is for the sshfs stack to be fixed
so that it passes the EXDEV up to the mv command properly, and passes the subsequent server-side copy (reflink) back down correctly.
This would be wonderful in theory, but it can't pass down the reflink, because the SFTP protocol (which is what sshfs uses) doesn't even have the concept of reflinks, so implementing this completely would require a revision to the SFTP protocol, which I don't see as likely to happen.
Quoted 12 lines · Austin S. Hemmelgarn
Quoted 7 lines · Hugo Mills
Direct rename (using rename(2)) isn't possible across subvols,
which is what the EXDEV result indicates. The solution is exactly what mv does, which is reflink-and-delete (which is cheaper than copy-and-delete, because no data is moved). In theory, you probably could implement rename across subvolumes in the FS, but it would just be moving the exact same operations from userspace to kernel space.Doing so though would have the advantage that it theoretically could be made (almost) atomic like a normal rename is, whereas the fallback in mv is absolutely not atomic.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
I think that the solution here is for the sshfs stack to be fixed
so that it passes the EXDEV up to the mv command properly, and passes the subsequent server-side copy (reflink) back down correctly.This would be wonderful in theory, but it can't pass down the reflink, because the SFTP protocol (which is what sshfs uses) doesn't even have the concept of reflinks, so implementing this completely would require a revision to the SFTP protocol, which I don't see as likely to happen.
Thanks for the insights, Hugo and Austin. That's more or less what I was expecting, so I guess my next stop will be with the openssh folks. Though I was looking over the SFTP protocol, and none of the SSH_FX_* errors seem suitable for this purpose. This might require a workaround in sshfs...
--Sean