vfs.git
Al Viro
maintainerFWIW, I considered sending that pile in several pull requests, but for some reason git request-pull v4.6 vfs work.lookups spews something very odd into diffstat - files that have never been touched by it and, in fact, doing merge with mainline does *not* end up with those files anywhere in the diff. Full pile doesn't produce any oddities of that sort, so...
Several series here:
* constified struct path * in LSM arguments (me)
* acl and xattr cleanups (some from me, most from Andreas)
* parallel lookups/readdir/atomic_open (me). ->i_mutex replaced with
rwsem, pure lookups take it shared. Exclusion is per-name - no parallel lookups on the same name in the same parent at the same time. ->atomic_open() without O_CREAT is also called with parent locked shared. ->iterate() is being replaced by a new method (->iterate_shared()), which is called with directory being locked only shared. Most of the filesystems switched to it. All of them (switched or not) get per-struct file exclusion for readdir and lseek. Incidentally, do_last()/lookup_open()/atomic_open() got cleaned up quite a bit.
* preadv2 updates (Christoph)
* rlimit vs coredumping stuff (Omar Sandoval)
* cifs finally getting rid of copying iovecs, manually draining them,
etc. - sock_sendmsg() and sock_recvmsg() allow to simplify things quite a bit (me).
* assorted cleanups and fixes
If you prefer that stuff to go in separate pulls, please say so. I've no idea what's triggering the junk in git request-pull for work.lookups - looks like a merge from -rc1-based branch into -rc3-based one has caused that somehow (commit 84695ffee). Affected files are the ones changed in mainline between -rc1 and -rc3 and they *are* identical to their -rc3 state after that merge commit...
Anyway, sane-looking git request-pull for the whole pile follows:
The following changes since commit 38b78a5f18584db6fa7441e0f4531b283b0e6725:
ovl: ignore permissions on underlying lookup (2016-05-10 23:58:18 -0400)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git for-next
for you to fetch changes up to 16a9fad4ef80ba18d84615eec1a6286b9d71e909:
Merge branch 'sendmsg.cifs' into for-next (2016-05-12 22:31:59 -0400)
----------------------------------------------------------------
Al Viro (113):
[apparmor] constify struct path * in a bunch of helpers
mtd: switch open_mtd_by_chdev() to use of vfs_stat()
mtd: switch ubi_open_volume_path() to vfs_stat()
__d_alloc(): treat NULL name as QSTR("/", 1)
bpf: reject invalid names right in ->lookup()
constify security_path_truncate()
constify vfs_truncate()
apparmor_path_truncate(): path->mnt is never NULL
tomoyo: constify assorted struct path *
constify chown_common/security_path_chown
constify security_sb_mount()
constify chmod_common/security_path_chmod
apparmor: new helper - common_path_perm()
apparmor: constify aa_path_link()
apparmor: constify common_perm_...()
constify security_path_{unlink,rmdir}
constify security_path_{mkdir,mknod,symlink}
apparmor: remove useless checks for NULL ->mnt
constify security_path_{link,rename}
constify security_path_chroot()
constify security_sb_pivotroot()
constify ima_d_path()
reiserfs_cache_default_acl(): use get_acl()
[net] drop 'size' argument of sock_recvmsg()
cifs: merge the hash calculation helpers
cifs: quit playing games with draining iovecs
cifs: no need to wank with copying and advancing iovec on recvmsg side either
cifs_readv_receive: use cifs_read_from_socket()
cifs: don't bother with kmap on read_pages side
ecryptfs: avoid multiple aliases for directories
ecryptfs_lookup(): try either only encrypted or plaintext name
aio: remove a pointless assignment
rw_verify_area(): saner calling conventions
Merge branch 'for-linus' into work.iov_iter
don't bother with ->d_inode->i_sb - it's always equal to ->d_sb
cifs: kill more bogus checks in ->...xattr() methods
reiserfs: switch to generic_{get,set,remove}xattr()
xattr_handler: pass dentry and inode as separate arguments of ->get()
->getxattr(): pass dentry and inode as separate arguments
Merge getxattr prototype change into work.lookups
security_d_instantiate(): move to the point prior to attaching dentry to inode
kernfs: use lookup_one_len_unlocked()
configfs_detach_prep(): make sure that wait_mutex won't go away
ocfs2: don't open-code inode_lock/inode_unlock
orangefs: don't open-code inode_lock/inode_unlock
reiserfs: open-code reiserfs_mutex_lock_safe() in reiserfs_unpack()
reconnect_one(): use lookup_one_len_unlocked()
ovl_lookup_real(): use lookup_one_len_unlocked()
make ext2_get_page() and friends work without external serialization
nfs: missing wakeup in nfs_unblock_sillyrename()
lookup_slow(): bugger off on IS_DEADDIR() from the very beginning
__d_add(): don't drop/regain ->d_lock
beginning of transition to parallel lookups - marking in-lookup dentries
parallel lookups machinery, part 2
parallel lookups machinery, part 3
parallel lookups machinery, part 4 (and last)
parallel lookups: actual switch to rwsem
give readdir(2)/getdents(2)/etc. uniform exclusion with lseek()
introduce a parallel variant of ->iterate()
proc_fill_cache(): switch to d_alloc_parallel()
proc_sys_fill_cache(): switch to d_alloc_parallel()
switch all procfs directories ->iterate_shared()
fuse: switch to ->iterate_shared()
cifs: switch to ->iterate_shared()
dcache_{readdir,dir_lseek}() users: switch to ->iterate_shared
simple local filesystems: switch to ->iterate_shared()
path_openat(): take O_PATH handling out of do_last()
lookup_open(): expand the call of vfs_create()
Merge branch 'for-linus' into work.lookups
atomic_open(): don't bother with EEXIST check - it's done in do_last()
atomic_open(): consolidate "overridden ENOENT" in open-yourself cases
atomic_open(): massage the create_error logics a bit
do_last(): get rid of duplicate ELOOP check
do_last(): take fput() on error after opening to out:
atomic_open(): delay open_to_namei_flags() until the method call
atomic_open(): be paranoid about may_open() return value
lookup_open(): lift the "fallback to !O_CREAT" logics from atomic_open()
atomic_open(): reorder and clean up a bit
lookup_open(): expand the call of real_lookup()
lookup_open(): put the dentry fed to ->lookup() or ->atomic_open() into in-lookup hash lookup_open(): lock the parent shared unless O_CREAT is given
nfs: switch to ->iterate_shared()
nfs: per-name sillyunlink exclusion
configfs_readdir(): make safe under shared lock
kernfs: no point locking directory around that generic_file_llseek()
lustre: don't need to lock inode in directory lseek
more trivial ->iterate_shared conversions
romfs, squashfs: switch to ->iterate_shared()
fat: switch to ->iterate_shared()
9p: switch to ->iterate_shared()
Merge branch 'for-linus' into work.lookups
switch ecryptfs to ->iterate_shared
logfs: no need to lock directory in lseek
btrfs: switch to ->iterate_shared()
get_acorn_filename(): deobfuscate a bit
isofs: switch to ->iterate_shared()
fold checks into iterate_and_advance()
befs: constify stuff a bit
befs: switch to ->iterate_shared()
afs: switch to ->iterate_shared()
f2fs: switch to ->iterate_shared()
gfs2: switch to ->iterate_shared()
hpfs: handle allocation failures in hpfs_add_pos()
hpfs: switch to ->iterate_shared()
hostfs: switch to ->iterate_shared()
hfsplus: switch to ->iterate_shared()
hfs: switch to ->iterate_shared()
ext4: switch to ->iterate_shared()
gfs2: Switch to generic xattr handlers
Merge branch 'for-cifs' into work.xattr
Merge branches 'work.xattr', 'work.preadv2', 'work.iov_iter', 'work.const-path' and 'work.misc' into for-next Merge branch 'ovl-fixes' into for-next
Merge branch 'sendmsg.cifs' into for-next
Andreas Gruenbacher (13):
jfs: Remove unnecessary code in jfs_get_acl
posix_acl: Inode acl caching fixes
posix_acl: Unexport acl_by_type and make it static
cifs: Fix xattr name checks
cifs: Check for equality with ACL_TYPE_ACCESS and ACL_TYPE_DEFAULT
cifs: Fix removexattr for os2.* xattrs
cifs: Switch to generic xattr handlers
ceph: Get rid of d_find_alias in ceph_set_acl
ceph: Switch to generic xattr handlers
jfs: Clean up xattr name mapping
jfs: Switch to generic xattr handlers
ubifs: Switch to generic xattr handlers
btrfs: Switch to generic xattr handlers
Christoph Hellwig (10):
filemap: remove pos variables in generic_file_read_iter
filemap: remove the pos argument to generic_file_direct_write
xfs: eliminate the pos variable in xfs_file_dio_aio_write
direct-io: eliminate the offset argument to ->direct_IO
direct-io: remove the offset argument to dio_complete
fs: add IOCB_SYNC and IOCB_DSYNC
fs: simplify the generic_write_sync prototype
ceph: use generic_write_sync
fs: add RWF_DSYNC aand RWF_SYNC
nfsd: use RWF_SYNC
Omar Sandoval (2):
coredump: get rid of coredump_params->written
coredump: only charge written data against RLIMIT_CORE
Yan, Zheng (1):
ceph: kill __ceph_removexattr()
Documentation/filesystems/Locking | 2 +- Documentation/filesystems/porting | 53 +++ Documentation/filesystems/vfs.txt | 2 +- arch/alpha/kernel/osf_sys.c | 4 +- arch/powerpc/platforms/cell/spufs/coredump.c | 5 +- arch/powerpc/platforms/cell/spufs/inode.c | 2 +- block/blk-map.c | 47 +-- drivers/mtd/ubi/build.c | 13 +- drivers/mtd/ubi/kapi.c | 19 +- drivers/staging/lustre/lustre/llite/dir.c | 4 +- .../staging/lustre/lustre/llite/llite_internal.h | 4 +- drivers/staging/lustre/lustre/llite/rw26.c | 4 +- drivers/staging/lustre/lustre/llite/xattr.c | 6 +- drivers/target/iscsi/iscsi_target_util.c | 5 +- fs/9p/acl.c | 8 +- fs/9p/vfs_addr.c | 3 +- fs/9p/vfs_dir.c | 4 +- fs/9p/vfs_inode.c | 2 +- fs/9p/xattr.c | 4 +- fs/affs/dir.c | 2 +- fs/affs/file.c | 5 +- fs/afs/dir.c | 16 +- fs/aio.c | 2 - fs/autofs4/root.c | 4 +- fs/bad_inode.c | 4 +- fs/befs/befs.h | 4 +- fs/befs/btree.c | 16 +- fs/befs/btree.h | 4 +- fs/befs/datastream.c | 26 +- fs/befs/datastream.h | 11 +- fs/befs/linuxvfs.c | 6 +- fs/bfs/dir.c | 2 +- fs/binfmt_elf.c | 2 +- fs/binfmt_elf_fdpic.c | 2 +- fs/block_dev.c | 14 +- fs/btrfs/acl.c | 3 - fs/btrfs/file.c | 16 +- fs/btrfs/inode.c | 24 +- fs/btrfs/ioctl.c | 18 +- fs/btrfs/tree-log.c | 6 +- fs/btrfs/xattr.c | 28 +- fs/btrfs/xattr.h | 3 - fs/ceph/acl.c | 16 +- fs/ceph/addr.c | 3 +- fs/ceph/dir.c | 7 +- fs/ceph/file.c | 11 +- fs/ceph/inode.c | 29 +- fs/ceph/super.h | 8 +- fs/ceph/xattr.c | 192 ++-------- fs/cifs/Makefile | 3 +- fs/cifs/cifs_dfs_ref.c | 2 +- fs/cifs/cifsencrypt.c | 97 ++--- fs/cifs/cifsfs.c | 28 +- fs/cifs/cifsfs.h | 12 +- fs/cifs/cifsglob.h | 2 - fs/cifs/cifsproto.h | 12 +- fs/cifs/cifssmb.c | 15 +- fs/cifs/connect.c | 127 ++----- fs/cifs/file.c | 62 +--- fs/cifs/inode.c | 3 +- fs/cifs/readdir.c | 57 +-- fs/cifs/smb2transport.c | 107 +----- fs/cifs/transport.c | 141 +++----- fs/cifs/xattr.c | 386 ++++++++------------ fs/coda/dir.c | 18 +- fs/compat.c | 12 +- fs/configfs/dir.c | 37 +- fs/configfs/inode.c | 2 +- fs/coredump.c | 5 +- fs/cramfs/inode.c | 2 +- fs/dax.c | 4 +- fs/dcache.c | 280 ++++++++++++-- fs/direct-io.c | 34 +- fs/ecryptfs/crypto.c | 5 +- fs/ecryptfs/ecryptfs_kernel.h | 4 +- fs/ecryptfs/file.c | 73 +++- fs/ecryptfs/inode.c | 108 +++--- fs/ecryptfs/mmap.c | 3 +- fs/efs/dir.c | 3 +- fs/efs/namei.c | 2 +- fs/exofs/dir.c | 16 +- fs/exofs/inode.c | 3 +- fs/exofs/super.c | 2 +- fs/exportfs/expfs.c | 12 +- fs/ext2/acl.c | 3 - fs/ext2/dir.c | 16 +- fs/ext2/inode.c | 8 +- fs/ext2/namei.c | 2 +- fs/ext2/xattr_security.c | 6 +- fs/ext2/xattr_trusted.c | 6 +- fs/ext2/xattr_user.c | 8 +- fs/ext4/acl.c | 3 - fs/ext4/dir.c | 4 +- fs/ext4/ext4.h | 3 +- fs/ext4/file.c | 9 +- fs/ext4/indirect.c | 12 +- fs/ext4/inode.c | 18 +- fs/ext4/namei.c | 4 +- fs/ext4/xattr_security.c | 6 +- fs/ext4/xattr_trusted.c | 6 +- fs/ext4/xattr_user.c | 8 +- fs/f2fs/acl.c | 3 - fs/f2fs/data.c | 6 +- fs/f2fs/dir.c | 2 +- fs/f2fs/file.c | 9 +- fs/f2fs/namei.c | 2 +- fs/f2fs/xattr.c | 14 +- fs/fat/dir.c | 6 +- fs/fat/inode.c | 6 +- fs/file.c | 5 + fs/freevxfs/vxfs_lookup.c | 2 +- fs/fuse/dir.c | 99 +++-- fs/fuse/file.c | 5 +- fs/gfs2/acl.c | 58 ++- fs/gfs2/acl.h | 1 + fs/gfs2/aops.c | 6 +- fs/gfs2/file.c | 9 +- fs/gfs2/inode.c | 83 +---- fs/gfs2/ops_fstype.c | 4 +- fs/gfs2/super.c | 2 +- fs/gfs2/xattr.c | 48 ++- fs/hfs/attr.c | 5 +- fs/hfs/catalog.c | 3 + fs/hfs/dir.c | 12 +- fs/hfs/hfs_fs.h | 5 +- fs/hfs/inode.c | 9 +- fs/hfsplus/catalog.c | 3 + fs/hfsplus/dir.c | 12 +- fs/hfsplus/hfsplus_fs.h | 1 + fs/hfsplus/inode.c | 8 +- fs/hfsplus/posix_acl.c | 3 - fs/hfsplus/super.c | 1 + fs/hfsplus/xattr.c | 10 +- fs/hfsplus/xattr.h | 2 +- fs/hfsplus/xattr_security.c | 6 +- fs/hfsplus/xattr_trusted.c | 6 +- fs/hfsplus/xattr_user.c | 6 +- fs/hostfs/hostfs_kern.c | 2 +- fs/hpfs/dir.c | 12 +- fs/hpfs/dnode.c | 8 +- fs/hpfs/hpfs_fn.h | 2 +- fs/inode.c | 17 +- fs/isofs/dir.c | 4 +- fs/isofs/rock.c | 13 +- fs/jffs2/acl.c | 2 - fs/jffs2/dir.c | 4 +- fs/jffs2/security.c | 6 +- fs/jffs2/super.c | 2 +- fs/jffs2/xattr_trusted.c | 6 +- fs/jffs2/xattr_user.c | 6 +- fs/jfs/acl.c | 6 - fs/jfs/file.c | 6 +- fs/jfs/inode.c | 7 +- fs/jfs/jfs_xattr.h | 6 +- fs/jfs/namei.c | 8 +- fs/jfs/symlink.c | 12 +- fs/jfs/xattr.c | 224 +++++------- fs/kernfs/dir.c | 17 +- fs/kernfs/inode.c | 6 +- fs/kernfs/kernfs-internal.h | 4 +- fs/kernfs/mount.c | 5 +- fs/libfs.c | 11 +- fs/logfs/dir.c | 4 +- fs/minix/dir.c | 2 +- fs/namei.c | 401 ++++++++++----------- fs/nfs/dir.c | 80 ++-- fs/nfs/direct.c | 23 +- fs/nfs/file.c | 2 +- fs/nfs/inode.c | 4 +- fs/nfs/nfs3acl.c | 43 ++- fs/nfs/nfs4proc.c | 14 +- fs/nfs/nfstrace.h | 2 +- fs/nfs/unlink.c | 192 +++------- fs/nfsd/nfs3proc.c | 4 +- fs/nfsd/nfs3xdr.c | 2 +- fs/nfsd/nfsfh.c | 2 +- fs/nfsd/vfs.c | 18 +- fs/nilfs2/dir.c | 16 +- fs/nilfs2/inode.c | 4 +- fs/nilfs2/namei.c | 2 +- fs/ntfs/file.c | 7 +- fs/ocfs2/aops.c | 13 +- fs/ocfs2/dlmglue.c | 3 + fs/ocfs2/file.c | 2 +- fs/ocfs2/inode.c | 2 +- fs/ocfs2/xattr.c | 20 +- fs/omfs/dir.c | 2 +- fs/open.c | 8 +- fs/openpromfs/inode.c | 2 +- fs/orangefs/file.c | 4 +- fs/orangefs/orangefs-kernel.h | 4 +- fs/orangefs/xattr.c | 10 +- fs/overlayfs/inode.c | 4 +- fs/overlayfs/overlayfs.h | 4 +- fs/overlayfs/readdir.c | 4 +- fs/overlayfs/super.c | 2 +- fs/posix_acl.c | 116 +++--- fs/proc/base.c | 35 +- fs/proc/fd.c | 8 +- fs/proc/generic.c | 2 +- fs/proc/namespaces.c | 3 +- fs/proc/proc_net.c | 2 +- fs/proc/proc_sysctl.c | 17 +- fs/proc/root.c | 4 +- fs/qnx4/dir.c | 2 +- fs/qnx6/dir.c | 2 +- fs/read_write.c | 51 +-- fs/readdir.c | 37 +- fs/reiserfs/dir.c | 2 +- fs/reiserfs/file.c | 6 +- fs/reiserfs/inode.c | 7 +- fs/reiserfs/ioctl.c | 6 +- fs/reiserfs/namei.c | 18 +- fs/reiserfs/xattr.c | 54 --- fs/reiserfs/xattr.h | 9 +- fs/reiserfs/xattr_acl.c | 8 +- fs/reiserfs/xattr_security.c | 19 +- fs/reiserfs/xattr_trusted.c | 19 +- fs/reiserfs/xattr_user.c | 19 +- fs/romfs/super.c | 4 +- fs/splice.c | 3 + fs/squashfs/dir.c | 4 +- fs/squashfs/xattr.c | 6 +- fs/sysv/dir.c | 2 +- fs/ubifs/dir.c | 8 +- fs/ubifs/file.c | 12 +- fs/ubifs/super.c | 1 + fs/ubifs/ubifs.h | 7 +- fs/ubifs/xattr.c | 145 ++++---- fs/udf/dir.c | 2 +- fs/udf/file.c | 7 +- fs/udf/inode.c | 7 +- fs/udf/namei.c | 2 +- fs/ufs/dir.c | 16 +- fs/ufs/super.c | 2 +- fs/xattr.c | 12 +- fs/xfs/xfs_acl.c | 20 +- fs/xfs/xfs_aops.c | 7 +- fs/xfs/xfs_file.c | 25 +- fs/xfs/xfs_xattr.c | 6 +- include/linux/dax.h | 2 +- include/linux/dcache.h | 26 +- include/linux/file.h | 13 + include/linux/fs.h | 94 ++++- include/linux/lsm_hooks.h | 28 +- include/linux/net.h | 3 +- include/linux/nfs_fs.h | 16 +- include/linux/nfs_xdr.h | 4 +- include/linux/posix_acl.h | 1 - include/linux/security.h | 58 +-- include/linux/uio.h | 1 + include/linux/xattr.h | 5 +- include/trace/events/ext4.h | 6 +- include/uapi/linux/fs.h | 2 + kernel/audit_watch.c | 2 +- kernel/bpf/inode.c | 37 +- lib/iov_iter.c | 123 +++---- mm/filemap.c | 30 +- mm/page_io.c | 2 +- mm/shmem.c | 9 +- net/socket.c | 25 +- net/unix/af_unix.c | 2 +- security/apparmor/file.c | 4 +- security/apparmor/include/file.h | 4 +- security/apparmor/include/path.h | 2 +- security/apparmor/lsm.c | 83 ++--- security/apparmor/path.c | 8 +- security/commoncap.c | 6 +- security/integrity/evm/evm_main.c | 6 +- security/integrity/ima/ima.h | 2 +- security/integrity/ima/ima_api.c | 2 +- security/security.c | 28 +- security/selinux/hooks.c | 13 +- security/smack/smack_lsm.c | 6 +- security/tomoyo/common.h | 12 +- security/tomoyo/file.c | 10 +- security/tomoyo/mount.c | 4 +- security/tomoyo/tomoyo.c | 28 +- 278 files changed, 2654 insertions(+), 3037 deletions(-)
Quoted 5 lines · Al Viro
FWIW, I considered sending that pile in several pull requests, but for some reason git request-pull v4.6 vfs work.lookups spews something very odd into diffstat - files that have never been touched by it and, in fact, doing merge with mainline does *not* end up with those files anywhere in the diff.
This is "normal" if you have multiple merge bases (which in turn happens various ways, but all of them involved the branch having merges in itself, either back-merges or just merging two or more topic branches that had different starting points).
What happens is that request-pull will just pick the first merge base and then do a diff against that - which can happen to work, but most of the time you'll get part of the diff being from the differences that happened betweeen the different merge bases, rather than just the code that is in "your" branch.
The only way to get a reliably correct diff is to actually do the merge (which is a much more complex operation when there are multiple merge bases - git will actually do intermediate merges between the merge bases and then do a merge based on that - it's why the default git merge is called a "recursive" merge).
Some maintainers actually do that merge (into a private branch) to get that reliable end result and also to get a heads-up about any merge conflicts I would see when pulling. But I don't actually require it, it's ok if the pull request ends up looking messy due to just using the plain request-pull machinery without any merging.
The shortlog is always reliable, because that is not a "diff" operation (which is inherently about two end-points - and with multiple merge bases there are more than two end-points so "diff" is inevitably too weak an operation to show the differences). The shortlog is about the set of commits, and git will do the proper set operation to see "commits in A but not in B". So if you get surprising results in the shortlog, then that means that something is actually wrong in your tree and you should look out.
Quoted 1 line
Full pile doesn't produce any oddities of that sort, so...
The fact that the full pile doesn't is likely just because the merges you added, which brought in a new merge base and it may now be unique (because you merged with my tree as you did them) or it just happens to be one of the merge points that works by luck for "diff".
Quoted 1 line
If you prefer that stuff to go in separate pulls, please say so.
This time I'd really prefer it, especially that parallel lookup branch. That's such a big fundamental change that it definitely merits being merged separately rather than as part of "here's all the vfs changes for 4.7"..
As mentioned, you *can* get the diff right by testing a pre-merge, but you don't need to, and I'll take the messy diff.
Even when it doesn't match the diff after-the-merge, I can trivially verify _why_ the pull-request diff differs. That happens not just because of the multiple merge-bases issue, but you get differences with merge diffs in other cases too (ie any case where merging had to do any non-trivial merging action - perhaps due to part of a patch already being applied in my tree for other reasons etc etc).
Linus
Al Viro
maintainerQuoted 11 lines · Linus Torvalds
Quoted 5 lines · Al Viro
FWIW, I considered sending that pile in several pull requests, but for some reason git request-pull v4.6 vfs work.lookups spews something very odd into diffstat - files that have never been touched by it and, in fact, doing merge with mainline does *not* end up with those files anywhere in the diff.
This is "normal" if you have multiple merge bases (which in turn happens various ways, but all of them involved the branch having merges in itself, either back-merges or just merging two or more topic branches that had different starting points).
OK... What happened is that I started these branches off -rc1, then by -rc3 realized that PAGE_CACHE_SIZE->PAGE_SIZE stuff was going to create one hell of merge noise and rebased work.lookups to it. I would've done the same to work.xattr, but by then I had a never-rebase branch on top of it (cifs xattr stuff) and that meant that work.xattr (including its initial segment needed in work.lookups) was stuck at -rc1. Oh, well...
Quoted 6 lines
Quoted 1 line
If you prefer that stuff to go in separate pulls, please say so.
This time I'd really prefer it, especially that parallel lookup branch. That's such a big fundamental change that it definitely merits being merged separately rather than as part of "here's all the vfs changes for 4.7"..
OK. What I've got looks so:
#work.const-path - rc1-based, no conflicts with anything, independent from everything else
#work.misc - ditto
#work.iov_iter - ditto, but with merge from #for-linus at some point. No conflicts either (and that #for-linus had also been rc1-based).
#work.preadv2 - rc3-based, no conflicts with anything, independent from everything else
#sendmsg.cifs - rc1-based, trims quite a bit of now-unneeded crap from cifs. One obvious conflict with PAGE_CACHE_SIZE->PAGE_SIZE in there. Independent from everything else.
#work.xattr - rc1-based, some starts with some acl fixes, then switches ->getxattr to passing inode and dentry separately. This is the point where the things start to get tricky - that got merged into the very beginning of the -rc3-based #work.lookups, to allow untangling the security_d_instantiate() mess. That merge actually got one of the PAGE_CACHE_SIZE conflicts resolved. #work.xattr itself proceeds to switch a lot of filesystems to generic_...xattr(); no complications there.
#work.lookups, after that initial merge from #work.xattr does the following:
* untangle security_d_instantiate()
* convert a bunch of open-coded lookup_one_len_unlocked() to calls
of that thing; one such place (in overlayfs) actually yields a trivial conflict with overlayfs fixes later in the cycle - overlayfs ended up switching to a variant of lookup_one_len_unlocked() sans the permission checks. I would've dropped that commit (it gets overridden on merge from #ovl-fixes in #for-next; proper resolution is to use the variant in mainline fs/overlayfs/super.c), but I didn't want to rebase the damn thing - it was fairly late in the cycle...
* some filesystems had managed to depend on lookup/lookup exclusion
for *fs-internal* data structures in a way that would break if we relaxed the VFS exclusion. Fixing hadn't been hard, fortunately.
* core of that series - parallel lookup machinery, replacing ->i_mutex
with rwsem, making lookup_slow() take it only shared. At that point lookups happen in parallel; lookups on the same name wait for the in-progress one to be done with that dentry. Surprisingly little code, at that - almost all of it is in fs/dcache.c, with fs/namei.c changes limited to lookup_slow() - making it use the new primitive and actually switching to locking shared.
* parallel readdir stuff - first of all, we provide the exclusion
on per-struct file basis, same as we do for read() vs. lseek() for regular files. That takes care of most of the needed exclusion in readdir/readdir; however, these guys are trickier than lookups, so I went for switching them one-by-one - new method (->iterate_shared()) is added and filesystems are switched to it as they are either confirmed to be OK with shared lock on directory or fixed to be OK with that. I hope to kill the original method come next cycle (almost all in-tree filesystems are switched already), but it's still not quite finished.
* several filesystems get switched to parallel readdir. The
interesting part here is dealing with dcache preseeding by readdir; that needs minor adjustment to be safe with directory locked only shared. Most of the filesystems doing that got switched to in those commits. Important exception: NFS. Turns out that NFS folks, with their, er, insistence on VFS getting the fuck out of the way of the Smart Filesystem Code That Knows How And What To Lock(tm) have grown the locking of their own. Homegrown rwsem, with lookup/readdir/atomic_open being *writers* (sillyunlink is the reader there). Of course, with VFS getting the fuck out of the way, as requested, the actual smarts of the smart filesystem code etc. had become exposed...
* do_last/lookup_open/atomic_open cleanups. As the result,open() without O_CREAT locks the directory only shared. Including the ->atomic_open() case. Backmerge from #for-linus in the middle of that - atomic_open() fix got brought in.
* then comes NFS switch to saner (VFS-based ;-) locking, killing
the homegrown "lookup and readdir are writers" kinda-sorta rwsem. All exclusion for sillyunlink/lookup is done by the parallel lookups mechanism. Exclusion between sillyunlink and rmdir is a real rwsem now - rmdir being the writer. Result: NFS lookups/readdirs/O_CREAT-less opens happen in parallel now.
* the rest of the series consists of switching a lot of filesystems
to parallel readdir; in a lot of cases ->llseek() gets simplified as well. One backmerge in there (again, #for-linus - rockridge fix).
That's it for #work.lookups. There's one conflict in that pile (overlayfs one mentioned above) and there's a conflict resolved in the first merge (rc3 vs. beginning of #work.xattr).
FWIW, I think the least PITA would be if I send #work.lookups + backmerge from #ovl-fixes to resolve the fs/overlay/super.c conflict. If you think that it needs to done in even smaller steps (after all, it's about 3/4 of the entire pile), please say so. Otherwise, see below:
The following changes since commit 38b78a5f18584db6fa7441e0f4531b283b0e6725:
ovl: ignore permissions on underlying lookup (2016-05-10 23:58:18 -0400)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git for-linus
for you to fetch changes up to 0e0162bb8c008fa7742f69d4d4982c8a37b88f95:
Merge branch 'ovl-fixes' into for-linus (2016-05-17 02:17:59 -0400)
----------------------------------------------------------------
Al Viro (75):
reiserfs_cache_default_acl(): use get_acl()
don't bother with ->d_inode->i_sb - it's always equal to ->d_sb
cifs: kill more bogus checks in ->...xattr() methods
reiserfs: switch to generic_{get,set,remove}xattr()
xattr_handler: pass dentry and inode as separate arguments of ->get() ->getxattr(): pass dentry and inode as separate arguments
Merge getxattr prototype change into work.lookups security_d_instantiate(): move to the point prior to attaching dentry to inode
kernfs: use lookup_one_len_unlocked()
configfs_detach_prep(): make sure that wait_mutex won't go away
ocfs2: don't open-code inode_lock/inode_unlock
orangefs: don't open-code inode_lock/inode_unlock
reiserfs: open-code reiserfs_mutex_lock_safe() in reiserfs_unpack() reconnect_one(): use lookup_one_len_unlocked()
ovl_lookup_real(): use lookup_one_len_unlocked()
make ext2_get_page() and friends work without external serialization
nfs: missing wakeup in nfs_unblock_sillyrename()
lookup_slow(): bugger off on IS_DEADDIR() from the very beginning __d_add(): don't drop/regain ->d_lock
beginning of transition to parallel lookups - marking in-lookup dentries parallel lookups machinery, part 2
parallel lookups machinery, part 3
parallel lookups machinery, part 4 (and last)
parallel lookups: actual switch to rwsem
give readdir(2)/getdents(2)/etc. uniform exclusion with lseek() introduce a parallel variant of ->iterate()
proc_fill_cache(): switch to d_alloc_parallel()
proc_sys_fill_cache(): switch to d_alloc_parallel()
switch all procfs directories ->iterate_shared()
fuse: switch to ->iterate_shared()
cifs: switch to ->iterate_shared()
dcache_{readdir,dir_lseek}() users: switch to ->iterate_shared
simple local filesystems: switch to ->iterate_shared()
path_openat(): take O_PATH handling out of do_last()
lookup_open(): expand the call of vfs_create()
Merge branch 'for-linus' into work.lookups
atomic_open(): don't bother with EEXIST check - it's done in do_last() atomic_open(): consolidate "overridden ENOENT" in open-yourself cases atomic_open(): massage the create_error logics a bit
do_last(): get rid of duplicate ELOOP check
do_last(): take fput() on error after opening to out:
atomic_open(): delay open_to_namei_flags() until the method call atomic_open(): be paranoid about may_open() return value
lookup_open(): lift the "fallback to !O_CREAT" logics from atomic_open() atomic_open(): reorder and clean up a bit
lookup_open(): expand the call of real_lookup()
lookup_open(): put the dentry fed to ->lookup() or ->atomic_open() into in-lookup hash lookup_open(): lock the parent shared unless O_CREAT is given
nfs: switch to ->iterate_shared()
nfs: per-name sillyunlink exclusion
configfs_readdir(): make safe under shared lock
kernfs: no point locking directory around that generic_file_llseek()
lustre: don't need to lock inode in directory lseek
more trivial ->iterate_shared conversions
romfs, squashfs: switch to ->iterate_shared()
fat: switch to ->iterate_shared()
9p: switch to ->iterate_shared()
Merge branch 'for-linus' into work.lookups
switch ecryptfs to ->iterate_shared
logfs: no need to lock directory in lseek
btrfs: switch to ->iterate_shared()
get_acorn_filename(): deobfuscate a bit
isofs: switch to ->iterate_shared()
befs: constify stuff a bit
befs: switch to ->iterate_shared()
afs: switch to ->iterate_shared()
f2fs: switch to ->iterate_shared()
gfs2: switch to ->iterate_shared()
hpfs: handle allocation failures in hpfs_add_pos()
hpfs: switch to ->iterate_shared()
hostfs: switch to ->iterate_shared()
hfsplus: switch to ->iterate_shared()
hfs: switch to ->iterate_shared()
ext4: switch to ->iterate_shared()
Merge branch 'ovl-fixes' into for-linus
Andreas Gruenbacher (3):
jfs: Remove unnecessary code in jfs_get_acl
posix_acl: Inode acl caching fixes
posix_acl: Unexport acl_by_type and make it static
Documentation/filesystems/porting | 53 +++ arch/alpha/kernel/osf_sys.c | 4 +- arch/powerpc/platforms/cell/spufs/inode.c | 2 +- block/blk-map.c | 47 +-- drivers/staging/lustre/lustre/llite/dir.c | 4 +- .../staging/lustre/lustre/llite/llite_internal.h | 4 +- drivers/staging/lustre/lustre/llite/xattr.c | 6 +- fs/9p/acl.c | 8 +- fs/9p/vfs_dir.c | 4 +- fs/9p/vfs_inode.c | 2 +- fs/9p/xattr.c | 4 +- fs/affs/dir.c | 2 +- fs/afs/dir.c | 16 +- fs/autofs4/root.c | 4 +- fs/bad_inode.c | 4 +- fs/befs/befs.h | 4 +- fs/befs/btree.c | 16 +- fs/befs/btree.h | 4 +- fs/befs/datastream.c | 26 +- fs/befs/datastream.h | 11 +- fs/befs/linuxvfs.c | 6 +- fs/bfs/dir.c | 2 +- fs/btrfs/acl.c | 3 - fs/btrfs/inode.c | 2 +- fs/btrfs/ioctl.c | 18 +- fs/btrfs/tree-log.c | 6 +- fs/btrfs/xattr.c | 6 +- fs/ceph/acl.c | 2 + fs/ceph/super.h | 2 +- fs/ceph/xattr.c | 8 +- fs/cifs/cifs_dfs_ref.c | 2 +- fs/cifs/cifsfs.c | 2 +- fs/cifs/cifsfs.h | 2 +- fs/cifs/inode.c | 3 +- fs/cifs/readdir.c | 57 +-- fs/cifs/xattr.c | 56 +-- fs/coda/dir.c | 18 +- fs/compat.c | 12 +- fs/configfs/dir.c | 37 +- fs/configfs/inode.c | 2 +- fs/cramfs/inode.c | 2 +- fs/dcache.c | 267 ++++++++++++-- fs/ecryptfs/crypto.c | 5 +- fs/ecryptfs/ecryptfs_kernel.h | 4 +- fs/ecryptfs/file.c | 73 +++- fs/ecryptfs/inode.c | 23 +- fs/ecryptfs/mmap.c | 3 +- fs/efs/dir.c | 3 +- fs/efs/namei.c | 2 +- fs/exofs/dir.c | 16 +- fs/exofs/super.c | 2 +- fs/exportfs/expfs.c | 12 +- fs/ext2/acl.c | 3 - fs/ext2/dir.c | 16 +- fs/ext2/namei.c | 2 +- fs/ext2/xattr_security.c | 6 +- fs/ext2/xattr_trusted.c | 6 +- fs/ext2/xattr_user.c | 8 +- fs/ext4/acl.c | 3 - fs/ext4/dir.c | 4 +- fs/ext4/namei.c | 4 +- fs/ext4/xattr_security.c | 6 +- fs/ext4/xattr_trusted.c | 6 +- fs/ext4/xattr_user.c | 8 +- fs/f2fs/acl.c | 3 - fs/f2fs/dir.c | 2 +- fs/f2fs/namei.c | 2 +- fs/f2fs/xattr.c | 14 +- fs/fat/dir.c | 6 +- fs/file.c | 5 + fs/freevxfs/vxfs_lookup.c | 2 +- fs/fuse/dir.c | 99 +++-- fs/gfs2/file.c | 4 +- fs/gfs2/inode.c | 9 +- fs/gfs2/ops_fstype.c | 4 +- fs/gfs2/super.c | 2 +- fs/gfs2/xattr.c | 6 +- fs/hfs/attr.c | 5 +- fs/hfs/catalog.c | 3 + fs/hfs/dir.c | 12 +- fs/hfs/hfs_fs.h | 5 +- fs/hfs/inode.c | 2 + fs/hfsplus/catalog.c | 3 + fs/hfsplus/dir.c | 12 +- fs/hfsplus/hfsplus_fs.h | 1 + fs/hfsplus/inode.c | 1 + fs/hfsplus/posix_acl.c | 3 - fs/hfsplus/super.c | 1 + fs/hfsplus/xattr.c | 10 +- fs/hfsplus/xattr.h | 2 +- fs/hfsplus/xattr_security.c | 6 +- fs/hfsplus/xattr_trusted.c | 6 +- fs/hfsplus/xattr_user.c | 6 +- fs/hostfs/hostfs_kern.c | 2 +- fs/hpfs/dir.c | 12 +- fs/hpfs/dnode.c | 8 +- fs/hpfs/hpfs_fn.h | 2 +- fs/inode.c | 17 +- fs/isofs/dir.c | 4 +- fs/isofs/rock.c | 13 +- fs/jffs2/acl.c | 2 - fs/jffs2/dir.c | 4 +- fs/jffs2/security.c | 6 +- fs/jffs2/super.c | 2 +- fs/jffs2/xattr_trusted.c | 6 +- fs/jffs2/xattr_user.c | 6 +- fs/jfs/acl.c | 6 - fs/jfs/jfs_xattr.h | 2 +- fs/jfs/namei.c | 2 +- fs/jfs/xattr.c | 8 +- fs/kernfs/dir.c | 17 +- fs/kernfs/inode.c | 6 +- fs/kernfs/kernfs-internal.h | 4 +- fs/kernfs/mount.c | 5 +- fs/libfs.c | 11 +- fs/logfs/dir.c | 4 +- fs/minix/dir.c | 2 +- fs/namei.c | 399 ++++++++++----------- fs/nfs/dir.c | 80 +++-- fs/nfs/direct.c | 2 +- fs/nfs/inode.c | 4 +- fs/nfs/nfs3acl.c | 43 ++- fs/nfs/nfs4proc.c | 14 +- fs/nfs/nfstrace.h | 2 +- fs/nfs/unlink.c | 192 +++------- fs/nfsd/nfs3proc.c | 4 +- fs/nfsd/nfs3xdr.c | 2 +- fs/nfsd/nfsfh.c | 2 +- fs/nilfs2/dir.c | 16 +- fs/nilfs2/namei.c | 2 +- fs/ocfs2/aops.c | 4 +- fs/ocfs2/dlmglue.c | 3 + fs/ocfs2/file.c | 2 +- fs/ocfs2/inode.c | 2 +- fs/ocfs2/xattr.c | 20 +- fs/omfs/dir.c | 2 +- fs/open.c | 2 +- fs/openpromfs/inode.c | 2 +- fs/orangefs/file.c | 4 +- fs/orangefs/orangefs-kernel.h | 4 +- fs/orangefs/xattr.c | 10 +- fs/overlayfs/inode.c | 4 +- fs/overlayfs/overlayfs.h | 4 +- fs/overlayfs/readdir.c | 4 +- fs/overlayfs/super.c | 2 +- fs/posix_acl.c | 116 +++--- fs/proc/base.c | 35 +- fs/proc/fd.c | 8 +- fs/proc/generic.c | 2 +- fs/proc/namespaces.c | 3 +- fs/proc/proc_net.c | 2 +- fs/proc/proc_sysctl.c | 17 +- fs/proc/root.c | 4 +- fs/qnx4/dir.c | 2 +- fs/qnx6/dir.c | 2 +- fs/read_write.c | 12 - fs/readdir.c | 37 +- fs/reiserfs/dir.c | 2 +- fs/reiserfs/file.c | 6 +- fs/reiserfs/ioctl.c | 6 +- fs/reiserfs/namei.c | 18 +- fs/reiserfs/xattr.c | 54 --- fs/reiserfs/xattr.h | 9 +- fs/reiserfs/xattr_acl.c | 8 +- fs/reiserfs/xattr_security.c | 19 +- fs/reiserfs/xattr_trusted.c | 19 +- fs/reiserfs/xattr_user.c | 19 +- fs/romfs/super.c | 4 +- fs/splice.c | 3 + fs/squashfs/dir.c | 4 +- fs/squashfs/xattr.c | 6 +- fs/sysv/dir.c | 2 +- fs/ubifs/dir.c | 2 +- fs/ubifs/ubifs.h | 4 +- fs/ubifs/xattr.c | 6 +- fs/udf/dir.c | 2 +- fs/udf/namei.c | 2 +- fs/ufs/dir.c | 16 +- fs/ufs/super.c | 2 +- fs/xattr.c | 12 +- fs/xfs/xfs_acl.c | 20 +- fs/xfs/xfs_file.c | 2 +- fs/xfs/xfs_xattr.c | 6 +- include/linux/dcache.h | 26 +- include/linux/file.h | 13 + include/linux/fs.h | 51 ++- include/linux/nfs_fs.h | 11 +- include/linux/nfs_xdr.h | 4 +- include/linux/posix_acl.h | 1 - include/linux/uio.h | 1 + include/linux/xattr.h | 5 +- include/trace/events/ext4.h | 6 +- kernel/audit_watch.c | 2 +- lib/iov_iter.c | 19 + mm/shmem.c | 9 +- net/socket.c | 2 +- security/commoncap.c | 6 +- security/integrity/evm/evm_main.c | 6 +- security/selinux/hooks.c | 11 +- security/smack/smack_lsm.c | 6 +- 200 files changed, 1545 insertions(+), 1300 deletions(-)
Quoted 5 lines · Al Viro
FWIW, I think the least PITA would be if I send #work.lookups + backmerge from #ovl-fixes to resolve the
fs/overlay/super.cconflict. If you think that it needs to done in even smaller steps (after all, it's about 3/4 of the entire pile), please say so.
That looked ok and at least separates out the other fluff from the real meat.
I would probably have preferred to get the "lookup" part separately as a first merge, and then do the readdir work as an independent pull a day or two later, just because they are reasonably independent.
But I took it this way as one bigger pull instead. With the lookup_open() work being in the middle there, it wasn't a black-and-white totally independent series anyway.
I tried to turn your description of the work into a reasonable merge message, because I think that's a useful explanation of what went on if there are issues. And while I hope that everything "just works", this is definitely conceptually one of the biggest vfs changes we've had in a long time. That directory semaphore serialization has been there since pretty much day one. I'm happy with how relatively small the actual changes end up being, but still..
I'll push out after my test-build, can you please then double-check that everything looks good?
Thanks,
Linus
Al Viro
maintainerQuoted 10 lines · Linus Torvalds
I tried to turn your description of the work into a reasonable merge message, because I think that's a useful explanation of what went on if there are issues. And while I hope that everything "just works", this is definitely conceptually one of the biggest vfs changes we've had in a long time. That directory semaphore serialization has been there since pretty much day one. I'm happy with how relatively small the actual changes end up being, but still..
I'll push out after my test-build, can you please then double-check that everything looks good?
The merge looks sane...