buffered write path without inode lock (for bcachefs)
cover buffered write path without inode lock (for bcachefs)
this is going in my for-next branch - it's tested and I think all the corner cases are handled to my satisfaction (there are some fun ones!)
Kent Overstreet (2):
fs: file_remove_privs_flags()
bcachefs: Buffered write path now can avoid the inode lock
fs/bcachefs/errcode.h | 3 +- fs/bcachefs/fs-io-buffered.c | 145 +++++++++++++++++++++++++---------- fs/inode.c | 7 +- include/linux/fs.h | 1 + 4 files changed, 112 insertions(+), 44 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/2 fs: file_remove_privs_flags()
Rename and export __file_remove_privs(); for a buffered write path that doesn't take the inode lock we need to be able to check if the operation needs to do work first.
--- fs/inode.c | 7 ++++--- include/linux/fs.h | 1 + 2 files changed, 5 insertions(+), 3 deletions(-)
fs/inode.c +4 -3
@@ -2031,7 +2031,7 @@ static int __remove_privs(struct mnt_idmap *idmap,return notify_change (idmap, dentry, &newattrs,NULL ); } -static int __file_remove_privs (struct file *file,unsigned int flags) +int file_remove_privs_flags (struct file *file,unsigned int flags) {struct dentry *dentry =file_dentry (file);struct inode *inode =file_inode (file);
@@ -2056,6 +2056,7 @@ static int __file_remove_privs(struct file *file, unsigned int flags)inode_has_no_xattr (inode);return error; } +EXPORT_SYMBOL_GPL (file_remove_privs_flags);/** * file_remove_privs - remove special file privileges (suid, capabilities)
@@ -2068,7 +2069,7 @@ static int __file_remove_privs(struct file *file, unsigned int flags)*/ int file_remove_privs (struct file *file) { -return __file_remove_privs (file,0 ); +return file_remove_privs_flags (file,0 ); }EXPORT_SYMBOL (file_remove_privs);
@@ -2161,7 +2162,7 @@ static int file_modified_flags(struct file *file, int flags)* Clear the security bits if the process is not being run by root. * This keeps people from modifying setuid and setgid binaries. */ - ret =__file_remove_privs (file, flags); + ret =file_remove_privs_flags (file, flags);if (ret)return ret;
include/linux/fs.h +1 -0
@@ -3004,6 +3004,7 @@ extern struct inode *new_inode_pseudo(struct super_block *sb);extern struct inode *new_inode (struct super_block *sb);extern void free_inode_nonrcu (struct inode *inode);extern int setattr_should_drop_suidgid (struct mnt_idmap *,struct inode *); +extern int file_remove_privs_flags (struct file *file,unsigned int flags);extern int file_remove_privs (struct file *);int setattr_should_drop_sgid (struct mnt_idmap *idmap,const struct inode *inode);
Signature · 1 line
2.43.0
PATCH 2/2 bcachefs: Buffered write path now can avoid the inode lock
In reply to Kent OverstreetNon append, non extending buffered writes can now avoid taking the inode lock.
To ensure atomicity of writes w.r.t. other writes, we lock every folio that we'll be writing to, and if this fails we fall back to taking the inode lock.
Extensive comments are provided as to corner cases.
--- fs/bcachefs/errcode.h | 3 +- fs/bcachefs/fs-io-buffered.c | 145 +++++++++++++++++++++++++---------- 2 files changed, 107 insertions(+), 41 deletions(-)
fs/bcachefs/errcode.h +2 -1
@@ -247,7 +247,8 @@ x(BCH_ERR_nopromote, nopromote_congested) \ x(BCH_ERR_nopromote, nopromote_in_flight) \ x(BCH_ERR_nopromote, nopromote_no_writes) \ - x(BCH_ERR_nopromote, nopromote_enomem) + x(BCH_ERR_nopromote, nopromote_enomem) \ + x(0, need_inode_lock)enum bch_errcode {BCH_ERR_START =2048 ,
fs/bcachefs/fs-io-buffered.c +105 -40
197 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH 2/2] bcachefs: Buffered write path now can avoid the inode lock
Quoted 3 lines · Kent Overstreet
Non append, non extending buffered writes can now avoid taking the inode lock.
I think this is buggy.
I think you still need to take the inode lock *shared* for the writes, because otherwise you can have somebody else that truncates the file and now you will do a write past the end of the size of the file. That will cause a lot of issues.
So it's not a "inode_lock or not" situation. I think it's a "inode_lock vs inode_locks_shared" situation.
Note that the reading side isn't all that critical - if a read races with a truncate, at worst it will read some zeroes because we used the old length and the page cache got cleared in the meantime.
But the writing side ends up having actual consistency issues on disk. You don't want to have a truncate that removes the pages past the end of the new size and clears the end of the new last page, and race with another write that used the old size and *thought* it was writing to the middle of the file, but is now actually accessing a folio that is past the end of the whole file and writing to it.
There may be some reason that I'm missing that would make this a non-issue, but I really think you want to get the inode lock at least shared for the duration of the write.
Also note that for similar reasons, you can't just look at "will I extend the file" and take the lock non-shared. No, in order to actually trust the size, you need to *hold* the lock, so the logic needs to be something like
- take the lock exclusively if O_APPEND or if it *looks* like you
might extend the file size.
- otherwise, take the shared lock, and THEN RE-CHECK. The file size
might have changed, so now you need to double-check that you're really not going to extend the size of the file, and if you are, you need to go back and take the inode lock exclusively after all.
Again - I haven't thought a ton about this, so maybe there's some trick to it, but the above is what my naive thinking says the rule has to be. Writes are different from reads.
Linus
Re: [PATCH 2/2] bcachefs: Buffered write path now can avoid the inode lock
Quoted 9 lines · Linus Torvalds
- take the lock exclusively if
O_APPENDor if it *looks* like you
might extend the file size.- otherwise, take the shared lock, and THEN RE-CHECK. The file size
might have changed, so now you need to double-check that you're really not going to extend the size of the file, and if you are, you need to go back and take the inode lock exclusively after all.
Same goes for the suid/sgid checks. You need to take the inode lock shared to even have the i_mode be stable, and at that point you might decide "oh, I need to clear suid/sgid after all" and have to go drop the lock and retake it for exclusive access after all.
(And yes, we *really* should have a rwsem_try_upgrade() operation and at least avoid the "read_unlock -> write_lock" dance), but we don't. See a comment in mmap_upgrade_trylock() about it. It should be reasonably easy to add, but we've never really had enough reason to. Maybe somebody decides it's worth the effort)
Linus
Re: [PATCH 2/2] bcachefs: Buffered write path now can avoid the inode lock
Quoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
- take the lock exclusively if
O_APPENDor if it *looks* like you
might extend the file size.- otherwise, take the shared lock, and THEN RE-CHECK. The file size
might have changed, so now you need to double-check that you're really not going to extend the size of the file, and if you are, you need to go back and take the inode lock exclusively after all.
Quoted 4 lines
Same goes for the suid/sgid checks. You need to take the inode lock shared to even have the
i_modebe stable, and at that point you might decide "oh, I need to clear suid/sgid after all" and have to go drop the lock and retake it for exclusive access after all.
Do we though? Yes i_mode can change mid write, but if userspace issues a chmod() while we're in a write() - the end result has to be consistent with either "chmod(); write();" or 'write(); chmod();"; meaning as long as there's been a barrier in the syscall path so that we can't have seen a value of i_mode from before the chmod returned we're good.
Christian Brauner
maintainerRe: [PATCH 2/2] bcachefs: Buffered write path now can avoid the inode lock
In reply to Linus TorvaldsQuoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
- take the lock exclusively if
O_APPENDor if it *looks* like you
might extend the file size.- otherwise, take the shared lock, and THEN RE-CHECK. The file size
might have changed, so now you need to double-check that you're really not going to extend the size of the file, and if you are, you need to go back and take the inode lock exclusively after all.
Quoted 4 lines
Same goes for the suid/sgid checks. You need to take the inode lock shared to even have the
i_modebe stable, and at that point you might decide "oh, I need to clear suid/sgid after all" and have to go drop the lock and retake it for exclusive access after all.
I agree. And this is how we've done it in xfs as well. The inode lock is taken shared, then check for IS_NOSEC(inode) and if that's true keep the shared lock, otherwise upgrade to an exclusive lock. Note that this whole check also checks filesystem capability xattrs.
I think you'd also get potential security issues or at least very subtle behavior. Someone could make a binary setuid and a concurrent write happens. chmod is taking the exclusive lock and there's a concurrent write going on changing the contents to something unsafe without being forced to remove the set*id bits.
Similar for {g,u}id changes. Someone starts a chown() taking inode lock while someone issues a concurrent write. The chown changes the group ownership so that a writer would be forced to drop the sgid bit. But the writer proceeds keeping that bit.
So that's all potentially pretty subtle side-effects we'd allow. And I've removed so many bugs in that area the last couple of years as well. So let's please use the shared lock.
Re: [PATCH 2/2] bcachefs: Buffered write path now can avoid the inode lock
Quoted 29 lines · Christian Brauner
Quoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
- take the lock exclusively if
O_APPENDor if it *looks* like you
might extend the file size.- otherwise, take the shared lock, and THEN RE-CHECK. The file size
might have changed, so now you need to double-check that you're really not going to extend the size of the file, and if you are, you need to go back and take the inode lock exclusively after all.Quoted 4 lines
Same goes for the suid/sgid checks. You need to take the inode lock shared to even have the
i_modebe stable, and at that point you might decide "oh, I need to clear suid/sgid after all" and have to go drop the lock and retake it for exclusive access after all.I agree. And this is how we've done it in xfs as well. The inode lock is taken shared, then check for
IS_NOSEC(inode)and if that's true keep the shared lock, otherwise upgrade to an exclusive lock. Note that this whole check also checks filesystem capability xattrs.I think you'd also get potential security issues or at least very subtle behavior. Someone could make a binary setuid and a concurrent write happens. chmod is taking the exclusive lock and there's a concurrent write going on changing the contents to something unsafe without being forced to remove the set*id bits.
Quoted 4 lines
Similar for {g,u}id changes. Someone starts a
chown()taking inode lock while someone issues a concurrent write. The chown changes the group ownership so that a writer would be forced to drop the sgid bit. But the writer proceeds keeping that bit.
That's the kind of race that's not actually a race - if the syscalls being invoked at the same time, it's fine if the result is as if the write happened first, then the chown.
For there to be an actual race one of two things would have to be the case:
- the chown/chmod path would have to be dependent on the on the data the write path is changing, or
- there'd have to be a problem wnith the write path seeing inconsistent state mid modification
But I can't claim 100% the second case isn't a factor, I don't know the security side of things as well, and the XFS way sounds nice and clean.
Re: [PATCH 2/2] bcachefs: Buffered write path now can avoid the inode lock
In reply to Linus TorvaldsQuoted 26 lines · Linus Torvalds
Quoted 3 lines · Kent Overstreet
Non append, non extending buffered writes can now avoid taking the inode lock.
I think this is buggy.
I think you still need to take the inode lock *shared* for the writes, because otherwise you can have somebody else that truncates the file and now you will do a write past the end of the size of the file. That will cause a lot of issues.
So it's not a "
inode_lockor not" situation. I think it's a "inode_lockvsinode_locks_shared" situation.Note that the reading side isn't all that critical - if a read races with a truncate, at worst it will read some zeroes because we used the old length and the page cache got cleared in the meantime.
But the writing side ends up having actual consistency issues on disk. You don't want to have a truncate that removes the pages past the end of the new size and clears the end of the new last page, and race with another write that used the old size and *thought* it was writing to the middle of the file, but is now actually accessing a folio that is past the end of the whole file and writing to it.
Quoted 3 lines
There may be some reason that I'm missing that would make this a non-issue, but I really think you want to get the inode lock at least shared for the duration of the write.
It's even mentioned in one of the comments - bcachefs's pagecache add lock guards against that. The rules for that lock are
- things that add to the pagecache take the add side of that lock
- things that remove the pagecache take the block side of that lock
I added that so that we wouldn't have pagecache inconsistency issues with dio and mmap'd IO - without it anything that needs to shoot down the pagecache while it's doing IO that bypasses the pagecache is buggy (fpunch, fcollapse...).
Quoted 5 lines
Also note that for similar reasons, you can't just look at "will I extend the file" and take the lock non-shared. No, in order to actually trust the size, you need to *hold* the lock, so the logic needs to be something like
Quoted 7 lines
- take the lock exclusively if
O_APPENDor if it *looks* like you
might extend the file size.- otherwise, take the shared lock, and THEN RE-CHECK. The file size
might have changed, so now you need to double-check that you're really not going to extend the size of the file, and if you are, you need to go back and take the inode lock exclusively after all.
That one - yes.
pagecache add lock was also supposed to handle that because anything that changes i_size downward needs pagecache block, but I moved where we take that lock for lock ordering reasons, and I really didn't need too...
I'm undecided on that one. I dislike using pagecache add lock to guard i_size because that's really not what it's for, but I also hate hitting the inode lock if we don't actually need it.
Kinda waiting for Al to drop in and mention the other super obscure reason the inode lock actually is needed here...