Pull request for FS-Cache, including NFS patches

Hi Stephen,

Can you try pulling the master branch of this tree:

git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-2.6-nfs-fscache.git

into linux-next, please?

This tree includes the following:

(1) The 'next' branch of the security tree, which you already have.

(2) The 'linux-next' branch of the NFS tree, which you already have.

(3) My FS-Cache, CacheFiles and AFS patches, and associated enablement patches.

(4) My patches to enable NFS to use FS-Cache.

I've tried merging into next-20081217, and it just applied and the tests worked upon it.

David
---
The following changes since commit 1bda71282ded6a2e09a2db7c8884542fb46bfd4f:
Linus Torvalds (1):

        Merge branch 'for-linus' of git://git.kernel.org/.../ieee1394/linux1394-2.6

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-2.6-nfs-fscache.git master

Al Viro (1):
Audit: Log TIOCSTI

David Howells (123):
CRED: Wrap task credential accesses in the IA64 arch
CRED: Wrap task credential accesses in the MIPS arch
CRED: Wrap task credential accesses in the PA-RISC arch
CRED: Wrap task credential accesses in the PowerPC arch
CRED: Wrap task credential accesses in the S390 arch
CRED: Wrap task credential accesses in the x86 arch
CRED: Wrap task credential accesses in the block loopback driver
CRED: Wrap task credential accesses in the tty driver
CRED: Wrap task credential accesses in the ISDN drivers
CRED: Wrap task credential accesses in the network device drivers
CRED: Wrap task credential accesses in the USB driver
CRED: Wrap task credential accesses in 9P2000 filesystem
CRED: Wrap task credential accesses in the AFFS filesystem
CRED: Wrap task credential accesses in the autofs filesystem
CRED: Wrap task credential accesses in the autofs4 filesystem
CRED: Wrap task credential accesses in the BFS filesystem
CRED: Wrap task credential accesses in the CIFS filesystem
CRED: Wrap task credential accesses in the Coda filesystem
CRED: Wrap task credential accesses in the devpts filesystem
CRED: Wrap task credential accesses in the eCryptFS filesystem
CRED: Wrap task credential accesses in the Ext2 filesystem
CRED: Wrap task credential accesses in the Ext3 filesystem
CRED: Wrap task credential accesses in the Ext4 filesystem
CRED: Wrap task credential accesses in the FAT filesystem
CRED: Wrap task credential accesses in the FUSE filesystem
CRED: Wrap task credential accesses in the GFS2 filesystem
CRED: Wrap task credential accesses in the HFS filesystem
CRED: Wrap task credential accesses in the HFSplus filesystem
CRED: Wrap task credential accesses in the HPFS filesystem
CRED: Wrap task credential accesses in the hugetlbfs filesystem
CRED: Wrap task credential accesses in the JFS filesystem
CRED: Wrap task credential accesses in the Minix filesystem
CRED: Wrap task credential accesses in the NCPFS filesystem
CRED: Wrap task credential accesses in the NFS daemon
CRED: Wrap task credential accesses in the OCFS2 filesystem
CRED: Wrap task credential accesses in the OMFS filesystem
CRED: Wrap task credential accesses in the RAMFS filesystem
CRED: Wrap task credential accesses in the ReiserFS filesystem
CRED: Wrap task credential accesses in the SMBFS filesystem
CRED: Wrap task credential accesses in the SYSV filesystem
CRED: Wrap task credential accesses in the UBIFS filesystem
CRED: Wrap task credential accesses in the UDF filesystem
CRED: Wrap task credential accesses in the UFS filesystem
CRED: Wrap task credential accesses in the XFS filesystem
CRED: Wrap task credential accesses in the filesystem subsystem
CRED: Wrap task credential accesses in the SYSV IPC subsystem
CRED: Wrap task credential accesses in the AX25 protocol
CRED: Wrap task credential accesses in the IPv6 protocol
CRED: Wrap task credential accesses in the netrom protocol
CRED: Wrap task credential accesses in the ROSE protocol
CRED: Wrap task credential accesses in the SunRPC protocol
CRED: Wrap task credential accesses in the UNIX socket protocol
CRED: Wrap task credential accesses in the networking subsystem
CRED: Wrap task credential accesses in the key management code
CRED: Wrap task credential accesses in the capabilities code
CRED: Wrap task credential accesses in the core kernel
KEYS: Disperse linux/key_ui.h
KEYS: Alter use of key instantiation link-to-keyring argument
CRED: Neuter sys_capset()
CRED: Constify the kernel_cap_t arguments to the capset LSM hooks
CRED: Separate task security context from task_struct
CRED: Detach the credentials from task_struct
CRED: Wrap current->cred and a few other accessors
CRED: Use RCU to access another task's creds and to release a task's own creds
CRED: Wrap access to SELinux's task SID
CRED: Separate per-task-group keyrings from signal_struct
CRED: Rename is_single_threaded() to is_wq_single_threaded()
CRED: Make inode_has_perm() and file_has_perm() take a cred pointer
CRED: Pass credentials through dentry_open()
CRED: Inaugurate COW credentials
CRED: Make execve() take advantage of copy-on-write credentials
CRED: Prettify commoncap.c
CRED: Use creds in file structs
CRED: Documentation
CRED: Differentiate objective and effective subjective credentials on a task
CRED: Add a kernel_service object class to SELinux
CRED: Allow kernel services to override LSM settings for task actions Merge branch 'master' of /warthog/git/security-testing-2.6-next Create a dynamically sized pool of threads for doing very slow work items Make slow-work thread pool actually dynamic
Make the slow work pool configurable
Document the slow work thread pool
FS-Cache: Release page->private after failed readahead
FS-Cache: Recruit a couple of page flags for cache management
FS-Cache: Add the FS-Cache netfs API and documentation
FS-Cache: Add the FS-Cache cache backend API and documentation
FS-Cache: Add main configuration option, module entry points and debugging
FS-Cache: Add use of /proc and presentation of statistics
FS-Cache: Root index definition
FS-Cache: Add cache tag handling
FS-Cache: Add cache management
FS-Cache: Provide a slab for cookie allocation
FS-Cache: Add netfs registration
FS-Cache: Bit waiting helpers
FS-Cache: Object management state machine
FS-Cache: Implement the cookie management part of the netfs API
FS-Cache: Add and document asynchronous operation handling
FS-Cache: Implement data I/O part of netfs API
CacheFiles: Be consistent about the use of mapping vs file->f_mapping in Ext3
CacheFiles: Add a hook to write a single page of data to an inode
CacheFiles: Permit the page lock state to be monitored
CacheFiles: Export things for CacheFiles
CacheFiles: A cache that backs onto a mounted filesystem
FS-Cache: Make kAFS use FS-Cache
NFS: Add comment banners to some NFS functions
NFS: Add FS-Cache option bit and debug bit
NFS: Permit local filesystem caching to be enabled for NFS
NFS: Register NFS for caching and retrieve the top-level index
NFS: Define and create server-level objects
NFS: Define and create superblock-level objects
NFS: Define and create inode-level cache objects
NFS: Use local disk inode cache
NFS: Invalidate FsCache page flags when cache removed
NFS: Add some new I/O counters for FS-Cache doing things for NFS
NFS: FS-Cache page management
NFS: Add read context retention for FS-Cache to call back with
NFS: nfs_readpage_async() needs to be accessible as a fallback for local caching
NFS: Read pages from FS-Cache into an NFS inode
NFS: Store pages from an NFS inode into a local cache
NFS: Display local caching state
NFS: Add mount options to enable local caching on NFS
Merge branch 'master' of /warthog/git/security-testing-2.6-next Merge branch 'master' of /warthog/nfs/linux-2.6-fscache

Eric Paris (13):
SELinux: check open perms in dentry_open not inode_permission
SELinux: hold tasklist_lock and siglock while waking wait_chldexit
SELinux: Use unknown perm handling to handle unknown netlink msg types Document the order of arguments for cap_issubset. It's not instantly clear This patch add a generic cpu endian caps structure and externally available This patch will print cap_permitted and cap_inheritable data in the PATH Any time fcaps or a setuid app under SECURE_NOROOT is used to result in a When the capset syscall is used it is not possible for audit to record the
Capabilities: BUG when an invalid capability is requested
Add a new capable interface that will be used by systems that use audit to The oomkiller calculations make decisions based on capabilities. Since Currently SELinux jumps through some ugly hoops to not audit a capbility
capabilities: define get_vfs_caps_from_disk when file caps are not enabled

Hannes Eder (1):
CRED: fix sparse warnings

James Morris (6):
Merge branch 'master' into next
security: remove broken and useless declarations Merge branch 'master' into next
Merge branch 'master' into next
Merge branch 'serge-next' into next Merge branch 'master' into next

Michal Schmidt (1):
selinux: recognize netlink messages for 'ip addrlabel'

Prarit Bhargava (1):
CacheFiles: Add missing copy_page export for ia64

Randy Dunlap (2):
coda: fix creds reference
nfsctl: add headers for credentials

Serge E. Hallyn (4):
file capabilities: add no_file_caps switch (v4) user namespaces: let user_ns be cloned with fairsched user namespaces: require cap_set{ug}id for CLONE_NEWUSER user namespaces: document CFS behavior

Serge Hallyn (2):
User namespaces: set of cleanups (v2) User namespaces: use the current_user_ns() macro

 Documentation/credentials.txt                     |  582 ++++++++++
 Documentation/filesystems/caching/backend-api.txt |  658 +++++++++++
 Documentation/filesystems/caching/cachefiles.txt  |  501 +++++++++
 Documentation/filesystems/caching/fscache.txt     |  333 ++++++
 Documentation/filesystems/caching/netfs-api.txt   |  800 +++++++++++++
 Documentation/filesystems/caching/object.txt      |  313 ++++++
 Documentation/filesystems/caching/operations.txt  |  213 ++++
 Documentation/kernel-parameters.txt               |    4 +
 Documentation/scheduler/sched-design-CFS.txt      |   21 +
 Documentation/slow-work.txt                       |  174 +++
 arch/alpha/kernel/asm-offsets.c                   |   11 +-
 arch/alpha/kernel/entry.S                         |   10 +-
 arch/ia64/ia32/sys_ia32.c                         |    7 +-
 arch/ia64/kernel/ia64_ksyms.c                     |    1 +
 arch/ia64/kernel/mca_drv.c                        |    2 +-
 arch/ia64/kernel/perfmon.c                        |   43 +-
 arch/ia64/kernel/signal.c                         |    4 +-
 arch/mips/kernel/kspd.c                           |    4 +-
 arch/mips/kernel/mips-mt-fpaff.c                  |    5 +-
 arch/mips/kernel/vpe.c                            |    4 +-
 arch/parisc/kernel/signal.c                       |    2 +-
 arch/powerpc/mm/fault.c                           |    2 +-
 arch/powerpc/platforms/cell/spufs/inode.c         |    8 +-
 arch/s390/hypfs/inode.c                           |    4 +-
 arch/s390/kernel/compat_linux.c                   |   28 +-
 arch/um/drivers/mconsole_kern.c                   |    3 +-
 arch/x86/ia32/ia32_aout.c                         |    2 +-
 arch/x86/mm/fault.c                               |    2 +-
 drivers/block/loop.c                              |    6 +-
 drivers/char/tty_audit.c                          |   76 +-
 drivers/char/tty_io.c                             |    1 +
 drivers/connector/cn_proc.c                       |   16 +-
 drivers/isdn/capi/capifs.c                        |    4 +-
 drivers/isdn/hysdn/hysdn_procconf.c               |    6 +-
 drivers/net/tun.c                                 |    8 +-
 drivers/usb/core/devio.c                          |   10 +-
 drivers/usb/core/inode.c                          |    4 +-
 fs/9p/fid.c                                       |    2 +-
 fs/9p/vfs_inode.c                                 |    4 +-
 fs/9p/vfs_super.c                                 |    4 +-
 fs/Kconfig                                        |   24 +
 fs/Makefile                                       |    2 +
 fs/affs/inode.c                                   |    4 +-
 fs/affs/super.c                                   |    4 +-
 fs/afs/Makefile                                   |    3 +
 fs/afs/cache.c                                    |  503 ++++++---
 fs/afs/cache.h                                    |   15 +-
 fs/afs/cell.c                                     |   16 +-
 fs/afs/file.c                                     |  212 +++--
 fs/afs/inode.c                                    |   25 +-
 fs/afs/internal.h                                 |   53 +-
 fs/afs/main.c                                     |   27 +-
 fs/afs/mntpt.c                                    |    4 +-
 fs/afs/vlocation.c                                |   23 +-
 fs/afs/volume.c                                   |   14 +-
 fs/afs/write.c                                    |   19 +
 fs/anon_inodes.c                                  |    4 +-
 fs/attr.c                                         |    4 +-
 fs/autofs/inode.c                                 |    4 +-
 fs/autofs4/dev-ioctl.c                            |    3 +-
 fs/autofs4/inode.c                                |    4 +-
 fs/autofs4/waitq.c                                |    4 +-
 fs/bfs/dir.c                                      |    4 +-
 fs/binfmt_aout.c                                  |    2 +-
 fs/binfmt_elf.c                                   |   20 +-
 fs/binfmt_elf_fdpic.c                             |   19 +-
 fs/binfmt_flat.c                                  |    2 +-
 fs/binfmt_som.c                                   |    2 +-
 fs/cachefiles/Kconfig                             |   39 +
 fs/cachefiles/Makefile                            |   18 +
 fs/cachefiles/cf-bind.c                           |  286 +++++
 fs/cachefiles/cf-daemon.c                         |  754 +++++++++++++
 fs/cachefiles/cf-interface.c                      |  449 ++++++++
 fs/cachefiles/cf-internal.h                       |  360 ++++++
 fs/cachefiles/cf-key.c                            |  159 +++
 fs/cachefiles/cf-main.c                           |  106 ++
 fs/cachefiles/cf-namei.c                          |  772 +++++++++++++
 fs/cachefiles/cf-proc.c                           |  134 +++
 fs/cachefiles/cf-rdwr.c                           |  853 ++++++++++++++
 fs/cachefiles/cf-security.c                       |  116 ++
 fs/cachefiles/cf-xattr.c                          |  291 +++++
 fs/cifs/cifs_fs_sb.h                              |    2 +-
 fs/cifs/cifsproto.h                               |    2 +-
 fs/cifs/connect.c                                 |    4 +-
 fs/cifs/dir.c                                     |   12 +-
 fs/cifs/inode.c                                   |    8 +-
 fs/cifs/ioctl.c                                   |    2 +-
 fs/cifs/misc.c                                    |    4 +-
 fs/coda/cache.c                                   |    6 +-
 fs/coda/file.c                                    |    3 +-
 fs/coda/upcall.c                                  |    2 +-
 fs/compat.c                                       |   42 +-
 fs/devpts/inode.c                                 |    4 +-
 fs/dquot.c                                        |    4 +-
 fs/ecryptfs/ecryptfs_kernel.h                     |    3 +-
 fs/ecryptfs/kthread.c                             |    9 +-
 fs/ecryptfs/main.c                                |    3 +-
 fs/ecryptfs/messaging.c                           |   27 +-
 fs/ecryptfs/miscdev.c                             |   27 +-
 fs/exec.c                                         |  183 ++--
 fs/exportfs/expfs.c                               |    4 +-
 fs/ext2/balloc.c                                  |    2 +-
 fs/ext2/ialloc.c                                  |    4 +-
 fs/ext2/inode.c                                   |    2 +
 fs/ext3/balloc.c                                  |    2 +-
 fs/ext3/ialloc.c                                  |    4 +-
 fs/ext3/inode.c                                   |    9 +-
 fs/ext4/balloc.c                                  |    2 +-
 fs/ext4/ialloc.c                                  |    4 +-
 fs/fat/file.c                                     |    2 +-
 fs/fat/inode.c                                    |    4 +-
 fs/fcntl.c                                        |   18 +-
 fs/file_table.c                                   |   10 +-
 fs/fscache/Kconfig                                |   56 +
 fs/fscache/Makefile                               |   19 +
 fs/fscache/fsc-cache.c                            |  415 +++++++
 fs/fscache/fsc-cookie.c                           |  498 ++++++++
 fs/fscache/fsc-fsdef.c                            |  144 +++
 fs/fscache/fsc-histogram.c                        |  109 ++
 fs/fscache/fsc-internal.h                         |  380 +++++++
 fs/fscache/fsc-main.c                             |  124 ++
 fs/fscache/fsc-netfs.c                            |  103 ++
 fs/fscache/fsc-object.c                           |  810 +++++++++++++
 fs/fscache/fsc-operation.c                        |  459 ++++++++
 fs/fscache/fsc-page.c                             |  771 +++++++++++++
 fs/fscache/fsc-proc.c                             |   68 ++
 fs/fscache/fsc-stats.c                            |  212 ++++
 fs/fuse/dev.c                                     |    4 +-
 fs/fuse/dir.c                                     |   25 +-
 fs/gfs2/inode.c                                   |   10 +-
 fs/hfs/inode.c                                    |    4 +-
 fs/hfs/super.c                                    |    4 +-
 fs/hfsplus/inode.c                                |    4 +-
 fs/hfsplus/options.c                              |    4 +-
 fs/hpfs/namei.c                                   |   24 +-
 fs/hpfs/super.c                                   |    4 +-
 fs/hppfs/hppfs.c                                  |    6 +-
 fs/hugetlbfs/inode.c                              |   21 +-
 fs/inotify_user.c                                 |    2 +-
 fs/internal.h                                     |    6 +
 fs/ioprio.c                                       |   18 +-
 fs/jfs/jfs_inode.c                                |    4 +-
 fs/locks.c                                        |    2 +-
 fs/minix/bitmap.c                                 |    4 +-
 fs/namei.c                                        |   10 +-
 fs/namespace.c                                    |    2 +-
 fs/ncpfs/ioctl.c                                  |   91 +-
 fs/nfs/Makefile                                   |    1 +
 fs/nfs/client.c                                   |   14 +-
 fs/nfs/file.c                                     |   37 +-
 fs/nfs/fscache-index.c                            |  337 ++++++
 fs/nfs/fscache.c                                  |  521 +++++++++
 fs/nfs/fscache.h                                  |  208 ++++
 fs/nfs/inode.c                                    |   17 +-
 fs/nfs/internal.h                                 |    4 +
 fs/nfs/iostat.h                                   |   18 +
 fs/nfs/read.c                                     |   27 +-
 fs/nfs/super.c                                    |   44 +-
 fs/nfsctl.c                                       |    5 +-
 fs/nfsd/auth.c                                    |   95 +-
 fs/nfsd/nfs4recover.c                             |   72 +-
 fs/nfsd/nfsfh.c                                   |   11 +-
 fs/nfsd/vfs.c                                     |    9 +-
 fs/ocfs2/dlm/dlmfs.c                              |    8 +-
 fs/ocfs2/namei.c                                  |    4 +-
 fs/omfs/inode.c                                   |    8 +-
 fs/open.c                                         |   59 +-
 fs/pipe.c                                         |    4 +-
 fs/posix_acl.c                                    |    4 +-
 fs/proc/array.c                                   |   32 +-
 fs/proc/base.c                                    |   32 +-
 fs/quota.c                                        |    4 +-
 fs/ramfs/inode.c                                  |    4 +-
 fs/reiserfs/namei.c                               |    4 +-
 fs/smbfs/dir.c                                    |    3 +-
 fs/smbfs/inode.c                                  |    2 +-
 fs/smbfs/proc.c                                   |    2 +-
 fs/splice.c                                       |    3 +-
 fs/super.c                                        |    1 +
 fs/sysv/ialloc.c                                  |    4 +-
 fs/ubifs/budget.c                                 |    2 +-
 fs/ubifs/dir.c                                    |    4 +-
 fs/udf/ialloc.c                                   |    4 +-
 fs/udf/namei.c                                    |    2 +-
 fs/ufs/ialloc.c                                   |    4 +-
 fs/xfs/linux-2.6/xfs_cred.h                       |    6 +-
 fs/xfs/linux-2.6/xfs_globals.h                    |    2 +-
 fs/xfs/linux-2.6/xfs_ioctl.c                      |    5 +-
 fs/xfs/xfs_acl.c                                  |    6 +-
 fs/xfs/xfs_inode.h                                |    2 +-
 fs/xfs/xfs_vnodeops.h                             |   10 +-
 include/keys/keyring-type.h                       |   31 +
 include/linux/audit.h                             |   26 +
 include/linux/binfmts.h                           |   16 +-
 include/linux/capability.h                        |   25 +-
 include/linux/cred.h                              |  342 ++++++-
 include/linux/fs.h                                |   15 +-
 include/linux/fscache-cache.h                     |  504 +++++++++
 include/linux/fscache.h                           |  592 ++++++++++
 include/linux/init_task.h                         |   14 +-
 include/linux/key-ui.h                            |   66 --
 include/linux/key.h                               |   32 +-
 include/linux/keyctl.h                            |    4 +-
 include/linux/nfs_fs.h                            |   13 +
 include/linux/nfs_fs_sb.h                         |   11 +
 include/linux/nfs_iostat.h                        |   12 +
 include/linux/nsproxy.h                           |    1 -
 include/linux/page-flags.h                        |   44 +-
 include/linux/pagemap.h                           |   21 +
 include/linux/sched.h                             |   65 +-
 include/linux/securebits.h                        |    2 +-
 include/linux/security.h                          |  338 +++---
 include/linux/slow-work.h                         |   95 ++
 include/linux/tty.h                               |    4 +
 include/linux/user_namespace.h                    |   13 +-
 include/net/scm.h                                 |    4 +-
 init/Kconfig                                      |   12 +
 init/main.c                                       |    1 +
 ipc/mqueue.c                                      |   19 +-
 ipc/shm.c                                         |    9 +-
 ipc/util.c                                        |   18 +-
 kernel/Makefile                                   |    3 +-
 kernel/acct.c                                     |    7 +-
 kernel/auditsc.c                                  |  255 ++++-
 kernel/capability.c                               |  288 +----
 kernel/cgroup.c                                   |   17 +-
 kernel/cred-internals.h                           |   21 +
 kernel/cred.c                                     |  588 ++++++++++
 kernel/exit.c                                     |   23 +-
 kernel/fork.c                                     |   62 +-
 kernel/futex.c                                    |   20 +-
 kernel/futex_compat.c                             |    7 +-
 kernel/kmod.c                                     |   30 +-
 kernel/nsproxy.c                                  |   15 +-
 kernel/ptrace.c                                   |   29 +-
 kernel/sched.c                                    |   26 +-
 kernel/signal.c                                   |   60 +-
 kernel/slow-work.c                                |  617 ++++++++++
 kernel/sys.c                                      |  586 ++++++-----
 kernel/sysctl.c                                   |   11 +-
 kernel/timer.c                                    |    8 +-
 kernel/trace/trace.c                              |    2 +-
 kernel/tsacct.c                                   |    6 +-
 kernel/uid16.c                                    |   31 +-
 kernel/user.c                                     |   96 +--
 kernel/user_namespace.c                           |   65 +-
 kernel/workqueue.c                                |    8 +-
 lib/Makefile                                      |    2 +-
 lib/is_single_threaded.c                          |   45 +
 mm/filemap.c                                      |   99 ++
 mm/mempolicy.c                                    |    9 +-
 mm/migrate.c                                      |   19 +-
 mm/oom_kill.c                                     |   12 +-
 mm/readahead.c                                    |   40 +-
 mm/shmem.c                                        |    8 +-
 mm/swap.c                                         |    4 +-
 mm/swap_state.c                                   |    4 +-
 mm/swapfile.c                                     |    4 +-
 mm/truncate.c                                     |   10 +-
 mm/vmscan.c                                       |    6 +-
 net/9p/client.c                                   |    2 +-
 net/ax25/af_ax25.c                                |    2 +-
 net/ax25/ax25_route.c                             |    2 +-
 net/core/dev.c                                    |    8 +-
 net/core/scm.c                                    |   10 +-
 net/ipv4/netfilter/ipt_LOG.c                      |    4 +-
 net/ipv6/ip6_flowlabel.c                          |    2 +-
 net/ipv6/netfilter/ip6t_LOG.c                     |    4 +-
 net/netfilter/nfnetlink_log.c                     |    5 +-
 net/netfilter/xt_owner.c                          |   16 +-
 net/netrom/af_netrom.c                            |    4 +-
 net/rose/af_rose.c                                |    4 +-
 net/rxrpc/ar-key.c                                |    6 +-
 net/sched/cls_flow.c                              |    4 +-
 net/socket.c                                      |    4 +-
 net/sunrpc/auth.c                                 |   14 +-
 net/unix/af_unix.c                                |   11 +-
 security/capability.c                             |   56 +-
 security/commoncap.c                              |  830 +++++++++-----
 security/keys/internal.h                          |   49 +-
 security/keys/key.c                               |   25 +-
 security/keys/keyctl.c                            |  210 +++--
 security/keys/keyring.c                           |   15 +-
 security/keys/permission.c                        |   29 +-
 security/keys/proc.c                              |    8 +-
 security/keys/process_keys.c                      |  469 ++++-----
 security/keys/request_key.c                       |  135 ++-
 security/keys/request_key_auth.c                  |   46 +-
 security/root_plug.c                              |   13 +-
 security/security.c                               |  101 +-
 security/selinux/exports.c                        |    8 +-
 security/selinux/hooks.c                          | 1246 +++++++++++----------
 security/selinux/include/av_perm_to_string.h      |    2 +
 security/selinux/include/av_permissions.h         |    2 +
 security/selinux/include/class_to_string.h        |    5 +
 security/selinux/include/flask.h                  |    1 +
 security/selinux/include/objsec.h                 |   11 -
 security/selinux/nlmsgtab.c                       |    3 +
 security/selinux/selinuxfs.c                      |   13 +-
 security/selinux/xfrm.c                           |    6 +-
 security/smack/smack_access.c                     |    4 +-
 security/smack/smack_lsm.c                        |  173 ++-
 security/smack/smackfs.c                          |    6 +-
 303 files changed, 21049 insertions(+), 3726 deletions(-)
 create mode 100644 Documentation/credentials.txt
 create mode 100644 Documentation/filesystems/caching/backend-api.txt
 create mode 100644 Documentation/filesystems/caching/cachefiles.txt
 create mode 100644 Documentation/filesystems/caching/fscache.txt
 create mode 100644 Documentation/filesystems/caching/netfs-api.txt
 create mode 100644 Documentation/filesystems/caching/object.txt
 create mode 100644 Documentation/filesystems/caching/operations.txt
 create mode 100644 Documentation/slow-work.txt
 create mode 100644 fs/cachefiles/Kconfig
 create mode 100644 fs/cachefiles/Makefile
 create mode 100644 fs/cachefiles/cf-bind.c
 create mode 100644 fs/cachefiles/cf-daemon.c
 create mode 100644 fs/cachefiles/cf-interface.c
 create mode 100644 fs/cachefiles/cf-internal.h
 create mode 100644 fs/cachefiles/cf-key.c
 create mode 100644 fs/cachefiles/cf-main.c
 create mode 100644 fs/cachefiles/cf-namei.c
 create mode 100644 fs/cachefiles/cf-proc.c
 create mode 100644 fs/cachefiles/cf-rdwr.c
 create mode 100644 fs/cachefiles/cf-security.c
 create mode 100644 fs/cachefiles/cf-xattr.c
 create mode 100644 fs/fscache/Kconfig
 create mode 100644 fs/fscache/Makefile
 create mode 100644 fs/fscache/fsc-cache.c
 create mode 100644 fs/fscache/fsc-cookie.c
 create mode 100644 fs/fscache/fsc-fsdef.c
 create mode 100644 fs/fscache/fsc-histogram.c
 create mode 100644 fs/fscache/fsc-internal.h
 create mode 100644 fs/fscache/fsc-main.c
 create mode 100644 fs/fscache/fsc-netfs.c
 create mode 100644 fs/fscache/fsc-object.c
 create mode 100644 fs/fscache/fsc-operation.c
 create mode 100644 fs/fscache/fsc-page.c
 create mode 100644 fs/fscache/fsc-proc.c
 create mode 100644 fs/fscache/fsc-stats.c
 create mode 100644 fs/nfs/fscache-index.c
 create mode 100644 fs/nfs/fscache.c
 create mode 100644 fs/nfs/fscache.h
 create mode 100644 include/keys/keyring-type.h
 create mode 100644 include/linux/fscache-cache.h
 create mode 100644 include/linux/fscache.h
 delete mode 100644 include/linux/key-ui.h
 create mode 100644 include/linux/slow-work.h
 create mode 100644 kernel/cred-internals.h
 create mode 100644 kernel/cred.c
 create mode 100644 kernel/slow-work.c
 create mode 100644 lib/is_single_threaded.c

Hi David,

Added from today.

Usual spiel: all patches in that branch must have been
posted to a relevant mailing list
reviewed
unit tested
destined for the next merge window (or the current release)
*before* they are included.

Cheers,
Stephen Rothwell                    [email protected]
http://www.canb.auug.org.au/~sfr/

I don't think we want fscache for .29 yet. I'd rather let the credential code settle for one release, and have more time for actually reviewing it properly and have it 100% ready for .30.

I don't believe that it has yet been convincingly demonstrated that we want to merge it at all.

It's a huuuuuuuuge lump of new code, so it really needs to provide decent value. Can we revisit this? Yet again? What do we get from all this?

I really don't understand why fs-cache is always rejected. Actually it is the perfect solution for NFS booted systems - you have a big cluster of nodes and in order to minimize administration overhead the nodes are booted over NFS from one common chroot. With unionfs (preferred solution here is unionfs-fuse) one then maintains files required to be differently by different clients.

Caching files on the local disk minimized the network access and boosts the performance, so at least for this usage example fs-cache would be great. (Actually I have been thinking about to implement a caching branch into unionfs-fuse, but if the kernel can do it on its own, it is also fine.)

In the past David already posted many benchmarks and just a few weeks ago again:

http://lkml.indiana.edu/hypermail/linux/kernel/0811.3/00584.html

Cheers,
Bernd

It's never been rejected. For a long time it has been in a state where we're looking for the data which would allow us to agree that its benefits are worth its costs. AFAIK that has never really been convincingly demonstrated. Nor has the converse case been demonstrated, so it floats in limbo.

Have you actually used cachefs/cachefiles for this?

OK, benchmarks are good. But look:

 303 files changed, 21049 insertions(+), 3726 deletions(-)

it's an enormous hunk of code. That will be in the kernel for ever and ever, needing maintenance, adding additional burden to our effort to evolve the kernel, etc.

Are any distros pushing for this? Or shipping it? If so, are they able to weigh in and help us with this quite difficult decision?

Hi David,

Given the ongoing discussions around FS-Cache, I have removed it from linux-next. Please ask me to include it again (if sensible) once some decision has been reached about its future.

Cheers,
Stephen Rothwell                    [email protected]
http://www.canb.auug.org.au/~sfr/

I should tell you to go and reread LKML at this point.

But... What can FS-Cache do for you? Well, in your specific case, probably nothing. If all your computers are local to your normal desktop box and are connected by sufficiently fast network and you have sufficiently few of them, or you don't use any of NFS, AFS, CIFS, Lustre, CRFS, CD-ROMs then it is likely that won't gain you anything.

Even if you do use some of those "netfs's", it won't get you anything yet because I haven't included patches to support anything other than NFS and the in-kernel AFS client yet.

However, if you do use NFS (or my AFS client), and you are accessing computers via slow networks, or you have lots of machines spamming your NFS server, then it might avail you.

It's a compromise: a trade-off between the loading and latencies of your network vs the loading and latencies of your disk; you sacrifice disk space to make up for the deficiencies of your network. The worst bit is that the latencies are additive under some circumstances (when doing a page read you may have to check disk and then go to the network).

So, FS-Cache can do the following for you:

(1) Allow you to reduce network loading by diverting repeat reads to local storage.

(2) Allow you to reduce the latency of slow network links by diverting repeat reads to local storage.

(3) Allow you to reduce the effect of uncontactable servers by serving data out of local storage.

(4) Allows you to reduce the latency of slow rotating media (such as CDROM and CD-changers).

(5) Allow you to implement disconnected operation, partly by (3), but also by caching changes for later syncing.

Now, (1) and (2) are readily demonstrable. I have posted benchmarks to do this. (3) to (5) are not yet implemented; these have to be mostly implemented in the filesystems that use FS-Cache rather than FS-Cache itself. FS-Cache currently has sufficient functionality to do (3) and (4), but needs some extra bits to do (5).

I've tried to implement just the minimal useful functionality for persistent caching. There is more to be added, but it's not immediately necessary.

FS-Cache tries to make its facilities as simple and as general as possible so that _any_ filesystem or blockdev can use it. I have patches here to make NFS and AFS use it. I know someone is working on getting Lustre to use it, and other filesystem maintainers have expressed interest, subject to the code getting upstream.

Furthermore, FS-Cache hides the implementation of the cache from the netfs. Not only that, it hides I/O errors in the cache from the netfs. Why should the netfs have to deal with such things?

Another way to look at things is to look at other cases of cached netfs's. OpenAFS for example. It has a local cache of its own. Solaris has local NFS caching. Windows has local caching for NFS and CIFS, I think. Even web browsers have local caching.

A big chunk of that, particularly the deletions, is the creds patches. Excluding the stuff pulled from the security and NFS trees, the combined FS-Cache, CacheFiles and AFS+ and NFS+FS-Cache patches are, in fact:

	86 files changed, 15385 insertions(+), 413 deletions(-)

and over 19% of the insertions is documentation. Most of the deletions (373) are in AFS.

We (Red Hat) have shipped it in RHEL-5 and some Fedora releases. Doing so is quite an effort, though, precisely because the code is not yet upstream. We have customers using it and are gaining more customers who want it. There even appear to be CentOS users using it (or at least complaining when it breaks).

I don't know what will convince you. I've given you theoretical reasons why caching ought to be useful; I've backed up the ones I've implemented with benchmarks; I've given you examples of what our customers are doing with it or want to do with it. Please help me understand what else you want.

Do you perhaps want the netfs maintainers (such as Trond) to say that it's necessary?

David

That's useful news.

Was that information captured/maintained somewhere? It really is important (I think) for something of this magnitude.

I want to be able to have an answer when someone asks me "why was all that stuff merged". One which I can believe.

Of course, their opinions (and supporting explanations) would be valuable.

The users that I spoke to from the financial sector that tried it are still quite interested. One simple use case for them is a very large cluster of NFS clients for read-mostly workloads (say, 1000 or more NFS clients for shared system partitions). They like the ability to do persistent caching across reboots which allows them to have less (and less beefy) NFS servers for all of those boxes trying to reboot at once.

The other use case was for the large rendering customers, but I don't have first hand knowledge of the details...

Regards,

Ric

Local disk cache was great for AFS back around 1992. Typical networks were 10 or 100Mbps (slower than disk access at the time), and memories were small (typical 16MB). FS-Cache appears to help only with read traffic--one reason why the web loves caching--and only for reads that would miss the buffer/page cache (memories are now "large"). Solaris has had CacheFS since ~1995, HPUX had a port of it since ~1997. I'd be interested in evidence of even a small fraction of Solaris and/or HPUX shops using CacheFS. I am aware of customers who thought it sounded like a good idea, but ended up ditching it for various reasons (e.g., CacheFS just adds overhead if you almost always hit your local mem cache).

One argument in favor that I don't see here is that local disk cache is persistent (I'm assuming it is in your implementation).

Addressing 1 and 2 in your list, I'd be curious how often a miss in core is a hit on disk.
Number 3 scares me. How does this play with the expected semantics of NFS?
Number 5 is hard, if not provably requiring human intervention to do syncs when writes are involved (see Disconnected AFS work by UM/CITI/Huston, and work at CMU).
Add persistence as number 6. This may be the best reason to have it, imho.

-Dan

-----Original Message-----
From: David Howells [mailto:[email protected]]
Sent: Thursday, December 18, 2008 6:27 PM
To: Andrew Morton
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Pull request for FS-Cache, including NFS patches

I should tell you to go and reread LKML at this point.

But... What can FS-Cache do for you? Well, in your specific case, probably nothing. If all your computers are local to your normal desktop box and are connected by sufficiently fast network and you have sufficiently few of them, or you don't use any of NFS, AFS, CIFS, Lustre, CRFS, CD-ROMs then it is likely that won't gain you anything.

Even if you do use some of those "netfs's", it won't get you anything yet because I haven't included patches to support anything other than NFS and the in-kernel AFS client yet.

However, if you do use NFS (or my AFS client), and you are accessing computers via slow networks, or you have lots of machines spamming your NFS server, then it might avail you.

It's a compromise: a trade-off between the loading and latencies of your network vs the loading and latencies of your disk; you sacrifice disk space to make up for the deficiencies of your network. The worst bit is that the latencies are additive under some circumstances (when doing a page read you may have to check disk and then go to the network).

So, FS-Cache can do the following for you:

(1) Allow you to reduce network loading by diverting repeat reads to
local
storage.

(2) Allow you to reduce the latency of slow network links by diverting
repeat
reads to local storage.

(3) Allow you to reduce the effect of uncontactable servers by serving
data
out of local storage.

(4) Allows you to reduce the latency of slow rotating media (such as
CDROM
and CD-changers).

(5) Allow you to implement disconnected operation, partly by (3), but
also by
caching changes for later syncing.

Now, (1) and (2) are readily demonstrable. I have posted benchmarks to do this. (3) to (5) are not yet implemented; these have to be mostly implemented in the filesystems that use FS-Cache rather than FS-Cache itself. FS-Cache currently has sufficient functionality to do (3) and (4), but needs some extra bits to do (5).

I've tried to implement just the minimal useful functionality for persistent caching. There is more to be added, but it's not immediately necessary.

FS-Cache tries to make its facilities as simple and as general as possible so that _any_ filesystem or blockdev can use it. I have patches here to make NFS and AFS use it. I know someone is working on getting Lustre to use it, and other filesystem maintainers have expressed interest, subject to the code getting upstream.

Furthermore, FS-Cache hides the implementation of the cache from the netfs.
Not only that, it hides I/O errors in the cache from the netfs. Why should the netfs have to deal with such things?

Another way to look at things is to look at other cases of cached netfs's.
OpenAFS for example. It has a local cache of its own. Solaris has local NFS caching. Windows has local caching for NFS and CIFS, I think. Even web browsers have local caching.

A big chunk of that, particularly the deletions, is the creds patches. Excluding the stuff pulled from the security and NFS trees, the combined FS-Cache, CacheFiles and AFS+ and NFS+FS-Cache patches are, in fact:

	86 files changed, 15385 insertions(+), 413 deletions(-)

and over 19% of the insertions is documentation. Most of the deletions
(373) are in AFS.

We (Red Hat) have shipped it in RHEL-5 and some Fedora releases. Doing so is quite an effort, though, precisely because the code is not yet upstream. We have customers using it and are gaining more customers who want it. There even appear to be CentOS users using it (or at least complaining when it breaks).

I don't know what will convince you. I've given you theoretical reasons why caching ought to be useful; I've backed up the ones I've implemented with benchmarks; I've given you examples of what our customers are doing with it or want to do with it. Please help me understand what else you want.

Do you perhaps want the netfs maintainers (such as Trond) to say that it's necessary?

David
_______________________________________________
NFSv4 mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Would a disk cache on SSD make any sense? Seems like it'd change the latency tradeoff.

More details on the experiences of RHEL/Fedora users might be interesting. My (vague, mostly uniformed) impression is that the group of people who think they need it is indeed a lot larger than the group who really do need it--but that the latter group still exists.

--b.

OK. I do agree that persistent caches can be (sometimes very!) useful for certain workloads. As far as I'm concerned, it is mainly a tool to help scale up the number of clients per server.

One interesting use case that I didn't see David mention is for cluster boot up. In a lot of the HPC clustered set-ups there tend be a number of 'hot' files that all clients need to access at roughly the same time in the boot cycle. Pre-loading these files into the persistent cache before booting the cluster is one way to solve this problem. Server replication and/or copying the files to local storage on the clients are other solutions.
The advantage the persistent cache confers over the two other solutions is that it simplifies the data management: if you do need to change one or two of those hot files on the server, then the clients will automatically update their caches (both the page cache and the persistent cache) without requiring any further action from the cluster administrator. The disadvantage is that cachefs doesn't yet appear to have a tool to select those files that are hot and are therefore best suited to cache (please correct me if I'm wrong, David). The fact that a file is 'hot' on some given client is not necessarily equivalent to saying that it is hot on the server and vice versa. Are there any plans to at some point introduce a tool to manage the persistent cache?

Cheers
Trond

What customers are doing with it, you mean?

That is slightly tricky, as it involves customers, most of whom don't want their names to appear in public communications about their infrastructure.

However, I can give you some examples of how it is being used or being considered for use:

(1) We have a number of customers in the entertainment industry who use or would like to use this caching infrastructure in their render farms. They use NFS to distribute textures (say a million and a quarter files) to the individual rendering units. FS-Cache allows them to reduce the network load by satisfying subsequent NFS READ requests from each rendering unit's local cache rather than having to go to the network again.

(2) We have another customer who is looking to roll out bunches of systems, where each bunch has an NFS file server and a few tens of processing units with their root filesystems NFS-mounted from the server. They are willing to include storage per processing unit for swap and local caching (e.g. FS-Cache). This deployment is on the order of ten thousand machines.

(3) Similar to 3, people are looking at 'diskless' workstations that do actually have a disk, but only for swap and local caching. All O/S files are read by NFS.

(4) Yet others are using or looking at using FS-Cache to accelerate distributed web servers.

(5) Although it's not someone using FS-Cache directly, I do have an anecdote from a customer using a very large quantity of AFS clients (it may have been OpenAFS), where /usr was distributed by AFS. Before they had persistent local caches, the power went out in their building and came back on, and then all their machines in the process of booting spammed the AFS servers wanting their /usr again.

David

Yes. If you connect an NFS client to an NFS server by GigE and have no other load on the net, and the NFS server can keep the whole dataset in RAM, then the CacheFiles backend kills your performance. I have pointed out that you need to consider carefully whether a local cache is right for you.

FS-Cache itself doesn't care whether you're trying to cache read traffic or write traffic. The in-kernel AFS filesystem will cache reads and writes, but the NFS fs will only cache reads as I have currently implemented it.

The reason I did this is that if you make a write on an AFS file, you can immediately tell if it overlapped with a write from another client, and you can then nuke your caches.

On NFS, you can't, at least not without proper change attribute support in NFSv4. This is not so much a problem with the NFS protocol as a problem with the fact that the backing filesystems that NFS uses don't or didn't support it.

However, the same problem affects the NFS pagecache. If you do a write to an NFS server, *that* can then be undetectably out of sync with the server. If you don't mind that state persisting on disk, FS-Cache presents no barrier to NFS files that are open for writing being cached too.

I know that Solaris and Windows have it; I don't have any facts about how widely it is used.

It can be. As I said in my email:

I've tried to implement just the minimal useful functionality for persistent caching. There is more to be added, but it's not immediately necessary.

The CacheFiles backend is persistent, but you could write a backend that isn't, if you, for example, wanted to make use of the >4G of RAM permitted to some Pentium motherboards as a huge ramdisk.

Not having fully persistent caching allows you to skimp on what metadata you store to disk.

It depends on what you're doing. I'm not sure I can give you a better answer than that.

I don't know. Yet it's something you want to do for AFS, I think.

Yeah, I appreciate that it's, erm, 'yucky'. You can aliken it to doing a GIT pull or CVS update into your tree and then having to manually fix up the conflicts.

David

It could, as could using a battery backed RAM cache.

David

I've come across something similar, where a large company was distributing /usr to its UNIX/Linux workstations by AFS without persistent local caching. Cue a powercut, that took away the power from a large quantity of machines and then brought it back again, at which point all the machines tried to boot...

Yes. I have plans for tools to pin and unpin files, introduce culling priorities, make space reservations in the cache, and cache readahead.

These aren't, however, immediately necessary to make local caching useful.

To do this, ideally I want a set of ioctl, fcntl or fadvise commands that are common to all filesystems that will just be ignored if the filesystem isn't currently doing caching.

Our customers also want to be able to configure this statically, perhaps in some /etc file. Something like, on NFS mount X from server Y, fully readahead all files in or under directory Z. I have an idea on how to do this, but I need to thrash it out with Al.

David

Not just boot up. Consider a room full of thin clients using nfsroot and the lecturer saying "Now everybody open a browser" or "Now everybody open Openoffice". With just NFS, it takes ages (there is a bottleneck of a single gigabit link between the clients and the NFS server even though the server itself has a 10gig card). If we redirect most of /usr/lib to a small local flash with some LD_LIBRARY_PATH and bind mount trickery we get an acceptable startup time. The flash is too small to hold even /usr/lib (flash size: 500M, /usr/lib is: 927M) so it is not possible to keep everything locally.

It would be really nice if the local caching could be handled automatically and we would not need so many hacks, so I really look forward trying FS-Cache if I have time. I used cachefs on Solaris ages ago and I had good experiences back then; it would be really nice if Linux would catch up.

Gabor

     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

AFS was designed to support local disk cache, so with callbacks you can get a consistent system. I'll ass-u-me that Linux flavors of AFS have callbacks. It should be possible to *integrate* NFSv4.x with FS-Cache similarly (i.e., I don't think you could drop it in as a 'black-box' without breaking something, unless you explicitly build an independent proxy cache server).

-Dan

-----Original Message-----
From: David Howells [mailto:[email protected]]
Sent: Friday, December 19, 2008 5:21 AM
To: Muntz, Daniel
Cc: [email protected]; Andrew Morton; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Pull request for FS-Cache, including NFS patches

NFS?

I don't know. Yet it's something you want to do for AFS, I think.

David

It's less the callbacks and more the data version number that's important.

They do.

NFSv4 has equivalents of both the data version number and callbacks.

David

important.

Maybe for consistency, but for the performance benefits of local disk caching, I believe the callbacks are pretty important. I say that because I regularly use an NFS 3 filesystem on the IBM internal network that is painfully slow on Linux and fine on AIX with CacheFS. It was also fine when this data was in AFS instead, and if I copy all the files to a local disk filesystem.

In this case, the files are _all in local page cache_, so I assume the waiting is for the constant stream of transactions the client uses to make sure the data in the cache is still current every time I open a file. AIX CacheFS doesn't sweat the consistency, so makes these queries only periodically. AFS had the callbacks, so rather than the client asking every time if the data had changed, the server just told it when it did.

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Storage Systems

Indeed, but he said "so with callbacks you can get a _consistent_ system".

David

Callbacks, instead of assuming your data is good for some short period of time. It's all well and good that there are mechanisms to know that data has changed, but without some leasing/callback/(ugh)mandatory locking, you're just estimating consistency.

-Dan

-----Original Message-----
From: David Howells [mailto:[email protected]]
Sent: Friday, December 19, 2008 5:20 PM
To: Bryan Henderson
Cc: [email protected]; Andrew Morton; Muntz, Daniel; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Pull request for FS-Cache, including NFS patches

important.

Indeed, but he said "so with callbacks you can get a _consistent_ system".

David

Hi David,

What was the result of discussions around FS-Cache? I ask because it reappeared in linux-next today via the nfs tree (merged into that on Dec 24 and 25) ...

Cheers,
Stephen Rothwell                    [email protected]
http://www.canb.auug.org.au/~sfr/

There was none.

Dan Muntz's question:

Solaris has had CacheFS since ~1995, HPUX had a port of it since ~1997. I'd be interested in evidence of even a small fraction of Solaris and/or HPUX shops using CacheFS. I am aware of customers who thought it sounded like a good idea, but ended up ditching it for various reasons (e.g., CacheFS just adds overhead if you almost always hit your local mem cache).

was an very very good one.

Seems that instead of answering it, we've decided to investigate the fate of those who do not learn from history.

oh.

And that of course means that many many 2.6.28 patches which I am maintaining will need significant rework to apply on top of linux-next, and then they won't apply to mainline. Or that linux-next will not apply on top of those patches. Mainly memory management.

Please drop the NFS tree until after -rc1.

Guys, this: http://lkml.org/lkml/2008/12/27/173

Hi Andrew, Trond,

OK, it is dropped for now (including today's tree).

Cheers,
Stephen Rothwell                    [email protected]
http://www.canb.auug.org.au/~sfr/

_______________________________________________
NFSv4 mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

That is the result of discussions during the kernel summit in Portland. The discussion here is about whether Andrew agrees with adding the patches or not, as far as I can tell. There are a number of people/companies who want them; there is Andrew who does not.

David

David has given you plenty of arguments for why it helps scale the server (including specific workloads), has given you numbers validating his claim, and has presented claims that Red Hat has customers using cachefs in RHEL-5.
The arguments I've seen against it, have so far been:

1. Solaris couldn't sell their implementation
2. It's too big
3. It's intrusive

Argument (1) has so far appeared to be pure FUD. In order to discuss the lessons of history, you need to first do the work of analysing and understanding it first. I really don't see how it is relevant to Linux whether or not the Solaris and HPUX cachefs implementations worked out unless you can demonstrate that that their experience shows some fatal flaw in the arguments and numbers that David presented, and that his customers are deluded.
If you want examples of permanent caches that clearly do help servers scale, then look no further than the on-disk caches used in almost all http browser implemantations. Alternatively, as David mentioned, there are the on-disk caches used by AFS/DFS/coda.

(2) may be valid, but I have yet to see specifics for where you'd like to see the cachefs code slimmed down. Did I miss them?

(3) was certainly true 3 years ago, when the code was first presented for review, and so we did a review and critique then. The NFS specific changes have improved greatly as a result, and as far as I know, the security folks are happy too. If you're not happy with the parts that affect the memory management code then, again, it would be useful to see specifics that what you want changed.

If there is still controversy concerning this, then I can temporarily remove cachefs from the nfs linux-next branch, but I'm definitely keeping it in the linux-mm branch until someone gives me a reason for why it shouldn't be merged in its current state.

Trond

I can add that our Red Hat customers who tried the cachefs preview did find it useful for their workloads (and, by the way, also use the Solaris cachefs on solaris boxes if I remember correctly). They have been nagging me and others at Red Hat about getting it into supported state for quite a while :-)

As you point out, this is all about getting more clients to be driven by a set of NFS servers.

Regards,

Ric

I disagree with your assertion that there was no result. Various people, beside myself, have weighed in with situations where FS-Cache is or may be useful. You've been presented with benchmarks showing that it can make a difference.

However, *you* are the antagonist, as strictly defined in the dictionary; we were trying to convince *you*, so a result has to come from *you*. I feel that you are completely against it and that we've no hope of shifting you.

And to a large extent irrelevant. Yes, we know caching adds overhead; I've never tried to pretend otherwise. It's an exercise in compromise. You don't just go and slap a cache on everything. There *are* situations in which a cache will help. We have customers who know about them and are willing to live with the overhead.

What I have done is to ensure that, even if caching is compiled in, then the overhead is minimal if there is _no_ cache present. That is requirement #1 on my list.

Assuming I understand what he said correctly, I've avoided the main issue listed by Dan because I don't do as Solaris does and interpolate the cache between the user and NFS. Of course, that probably buys me other issues (FS design is an exercise in compromise too).

Sigh.

The main point is that caching _is_ useful, even with its drawbacks. Dan may be aware of customers of Sun/HP who thought caching sounds like a good idea, but then ended up ditching it. I can well believe it. But I am also aware of customers of Red Hat who are actively using the caching we put in RHEL-5 and customers who really want caching available in future RHEL and Fedora versions for various reasons.

To sum up:

(1) Overhead is minimal if there is no cache.

(2) Benchmarks show that the cache can be effective.

(3) People are already using it and finding it useful.

(4) There are people who want it for various projects.

(5) The use of a cache does not automatically buy you an improvement in
performance: it's a matter of compromise.

(6) The performance improvement may be in the network or the servers, not the client that is actually doing the caching.

David

Significant rework to many many patches? The FS-Cache patches don't have all that much impact outside of their own directories, AFS and NFS.

Okay, that's a reasonable request.

David

Before throwing the 'FUD' acronym around, maybe you should re-read the details. My point was that there were few users of cachefs even when the technology had the potential for greater benefit (slower networks, less powerful servers, smaller memory caches). Obviously cachefs can improve performance--it's simply a function of workload and the assumptions made about server/disk/network bandwidth. However, I would expect the real benefits and real beneficiaries to be fewer than in the past. HOWEVER^2 I did provide some argument(s) in favor of adding cachefs, and look forward to extensions to support delayed write, offline operation, and NFSv4 support with real consistency checking (as long as I don't have to take the customer calls ;-). BTW, animation/video shops were one group that did benefit, and I imagine they still could today (the one I had in mind did work across Britain, the US, and Asia and relied on cachefs for overcoming slow network connections). Wonder if the same company is a RH customer...

All the comparisons to HTTP browser implementations are, imho, absurd. It's fine to keep a bunch of http data around on disk because a) it's RO data, b) correctness is not terribly important, and c) a human is generally the consumer and can manually request non-cached data if things look wonky. It is a trivial case of caching.

As for security, look at what MIT had to do to prevent local disk caching from breaking the security guarantees of AFS.

Customers (deluded or otherwise) are still customers. No one is forced to compile it into their kernel. Ship it.

-Dan

-----Original Message-----
From: Trond Myklebust [mailto:[email protected]]
Sent: Monday, December 29, 2008 6:31 AM
To: Andrew Morton
Cc: Stephen Rothwell; Bernd Schubert; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Pull request for FS-Cache, including NFS patches

who

David has given you plenty of arguments for why it helps scale the server (including specific workloads), has given you numbers validating his claim, and has presented claims that Red Hat has customers using cachefs in RHEL-5.
The arguments I've seen against it, have so far been:

1. Solaris couldn't sell their implementation
2. It's too big
3. It's intrusive

Argument (1) has so far appeared to be pure FUD. In order to discuss the lessons of history, you need to first do the work of analysing and understanding it first. I really don't see how it is relevant to Linux whether or not the Solaris and HPUX cachefs implementations worked out unless you can demonstrate that that their experience shows some fatal flaw in the arguments and numbers that David presented, and that his customers are deluded.
If you want examples of permanent caches that clearly do help servers scale, then look no further than the on-disk caches used in almost all http browser implemantations. Alternatively, as David mentioned, there are the on-disk caches used by AFS/DFS/coda.

(2) may be valid, but I have yet to see specifics for where you'd like to see the cachefs code slimmed down. Did I miss them?

(3) was certainly true 3 years ago, when the code was first presented for review, and so we did a review and critique then. The NFS specific changes have improved greatly as a result, and as far as I know, the security folks are happy too. If you're not happy with the parts that affect the memory management code then, again, it would be useful to see specifics that what you want changed.

If there is still controversy concerning this, then I can temporarily remove cachefs from the nfs linux-next branch, but I'm definitely keeping it in the linux-mm branch until someone gives me a reason for why it shouldn't be merged in its current state.

Trond

_______________________________________________
NFSv4 mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

I did read your argument. My point is that although the argument sounds reasonable, it ignores the fact that the customer bases are completely different. The people asking for cachefs on Linux typically run a cluster of 2000+ clients all accessing the same read-only data from just a handful of servers. They're primarily looking to improve the performance and stability of the _servers_, since those are the single point of failure of the cluster.

As far as I know, historically there has never been a market for 2000+ HP-UX, or even Solaris based clusters, and unless the HP and Sun product plans change drastically, then simple economics dictates that nor will there ever be such a market, whether or not they have cachefs support.

OpenSolaris is a different kettle of fish since it has cachefs, and does run on COTS hardware, but there are other reasons why that hasn't yet penetrated the HPC market.

See above. The majority of people I'm aware of that have been asking for this are interested mainly in improving read-only workloads for data that changes infrequently. Correctness tends to be important, but the requirements are no different from those that apply to the page cache. You mentioned the animation industry: they are prime example of an industry that satisfies (a), (b), and (c). Ditto the oil and gas exploration industry, as well as pretty much all scientific computing, to mention only a few examples...

See what David has added to the LSM code to provide the same guarantees for cachefs...

Trond

for cachefs...

Unless it (at least) leverages TPM, the issues I had in mind can't really be addressed in code. One requirement is to prevent a local root user from accessing fs information without appropriate permissions. This leads to unwieldly requirements such as allowing only one user on a machine at a time, blowing away the cache on logout, validating (e.g., refreshing) the kernel on each boot, etc. Sure, some applications won't care, but you're also potentially opening holes that users may not consider.

-Dan

-----Original Message-----
From: Trond Myklebust [mailto:[email protected]]
Sent: Tuesday, December 30, 2008 10:45 AM
To: Muntz, Daniel
Cc: Andrew Morton; Stephen Rothwell; Bernd Schubert; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: Pull request for FS-Cache, including NFS patches

I did read your argument. My point is that although the argument sounds reasonable, it ignores the fact that the customer bases are completely different. The people asking for cachefs on Linux typically run a cluster of 2000+ clients all accessing the same read-only data from just a handful of servers. They're primarily looking to improve the performance and stability of the _servers_, since those are the single point of failure of the cluster.

As far as I know, historically there has never been a market for 2000+ HP-UX, or even Solaris based clusters, and unless the HP and Sun product plans change drastically, then simple economics dictates that nor will there ever be such a market, whether or not they have cachefs support.

OpenSolaris is a different kettle of fish since it has cachefs, and does run on COTS hardware, but there are other reasons why that hasn't yet penetrated the HPC market.

See above. The majority of people I'm aware of that have been asking for this are interested mainly in improving read-only workloads for data that changes infrequently. Correctness tends to be important, but the requirements are no different from those that apply to the page cache. You mentioned the animation industry: they are prime example of an industry that satisfies (a), (b), and (c). Ditto the oil and gas exploration industry, as well as pretty much all scientific computing, to mention only a few examples...

See what David has added to the LSM code to provide the same guarantees for cachefs...

Trond

You can't prevent a local root user from accessing cached data: that's true with or without cachefs. root can typically access the data using /dev/kmem, swap, intercepting tty traffic, spoofing user creds,... If root can't be trusted, then find another machine.

The worry is rather that privileged daemons may be tricked into revealing said data to unprivileged users, or that unprivileged users may attempt to read data from files to which they have no rights using the cachefs itself. That is a problem that is addressable by means of LSM, and is what David has attempted to solve.

Trond

Yes, and if you have a single user on the machine at a time (with cache flushed inbetween, kernel refreshed), root can read /dev/kmem, swap, intercept traffic and read cachefs data to its heart's content--hence, those requirements.

-Dan

-----Original Message-----
From: Trond Myklebust [mailto:[email protected]]
Sent: Tuesday, December 30, 2008 2:36 PM
To: Muntz, Daniel
Cc: Andrew Morton; Stephen Rothwell; Bernd Schubert; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: Pull request for FS-Cache, including NFS patches

permissions.

You can't prevent a local root user from accessing cached data: that's true with or without cachefs. root can typically access the data using /dev/kmem, swap, intercepting tty traffic, spoofing user creds,... If root can't be trusted, then find another machine.

The worry is rather that privileged daemons may be tricked into revealing said data to unprivileged users, or that unprivileged users may attempt to read data from files to which they have no rights using the cachefs itself. That is a problem that is addressable by means of LSM, and is what David has attempted to solve.

Trond

Unless you _are_ root and can check every executable, after presumably rebooting into your own trusted kernel, then those requirements won't mean squat. If you're that paranoid, then you will presumably also be using a cryptfs-encrypted partition for cachefs, which you unmount when you're not logged in.

That said, most cluster environments will tend to put most of their security resources into keeping untrusted users out altogether. The client nodes tend to be a homogeneous lot with presumably only a trusted few sysadmins...

Trond

we're talking about NFS here (but also local CDs and potentially CIFS etc). The level of security you're talking about is going to be the same before or after cachefs.... very little against local root.

Frankly, any networking filesystem just trusts that the connection is authenticated... eg there is SOMEONE on the machine who has the right credentials.

Cachefs doesn't change that; it still validates with the server before giving userspace the data.

Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Actually... Cachefiles could fairly trivially add encryption. It would have to be simple encryption but you wouldn't have to store any keys locally.

Currently cachefiles _copies_ data between the backingfs and the netfs pages because the direct-IO code is only usable to/from userspace. Rather than copying, encrypt/decrypt could be called.

A key could be constructed at the point a cache file is looked up. It could be constructed from the coherency data. In the case of NFS that would be mtime, ctime, isize and change_attr. The coherency data would be encrypted with this key and then stored on disk, as would the contents of the file.

It might be possible to chuck the cache key (NFS fh) into the encryption key too and also encrypt the cache key before it is turned into a filename, though we'd have to be careful to avoid collisions if each filename is encrypted with a different key.

We'd probably have to be careful about the coherency data decrypting with a different key showing up as the wrong but valid thing.

The nice thing about this is that the key need not be retained locally since it's entirely constructed from data fetched from the netfs.

David

Sure, trusted kernel and trusted executables, but it's easier than it sounds. If you start with a "clean" system, you don't need to verify excutables _if_ they're coming from the secured file server (by
induction: if you started out secure, the executables on the file server will remain secure). You simply can't trust the local disk from one user to the next. Following the protocol, a student can log into a machine, su to do their OS homework, but not compromise the security of the distributed file system.

If I can su while another user is logged in, or the kernel/cmds are not validated between users, cryptfs isn't safe either.

If you're following the protocol, it doesn't even matter if a bad guy ("untrusted user"?) gets root on the client--they still can't gain inappropriate access to the file server. OTOH, if my security plan is simply to not allow root access to untrusted users, history says I'm going to lose.

-Dan

-----Original Message-----
From: Trond Myklebust [mailto:[email protected]]
Sent: Tuesday, December 30, 2008 3:18 PM
To: Muntz, Daniel
Cc: Andrew Morton; Stephen Rothwell; Bernd Schubert; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: Pull request for FS-Cache, including NFS patches

Unless you _are_ root and can check every executable, after presumably rebooting into your own trusted kernel, then those requirements won't mean squat. If you're that paranoid, then you will presumably also be using a cryptfs-encrypted partition for cachefs, which you unmount when you're not logged in.

That said, most cluster environments will tend to put most of their security resources into keeping untrusted users out altogether. The client nodes tend to be a homogeneous lot with presumably only a trusted few sysadmins...

Trond

On Wed, 31 Dec 2008 20:11:13 -0800
"Muntz, Daniel" <[email protected]> wrote:

please don't top post.

if you have a user, history says you're going to lose.

you can make your system as secure as you want, with physical access all bets are off.
keyboard sniffer.. easy.
special dimms that mirror data... not even all THAT hard, just takes a bit of cash.
running the user in a VM without him noticing.. not too hard either. etc.

Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Yeah... this is precisely the reason that the security-test-plan and system-design-document for any really security sensitive system starts with:

[  ]  The system is in a locked rack
[  ]  The rack is in a locked server room with detailed access logs
[  ]  The server room is in a locked and secured building with 24-hour

camera surveillance and armed guards

I've spent a little time looking into the security guarantees provided by DAC and by the FS-Cache LSM hooks, and it is possible to reasonably guarantee that no *REMOTE* user will be able to compromise the contents of the cache using a combination of DAC (file permissions, etc) and MAC (SELinux, etc) controls. As previously mentioned, local users (with physical hardware access) are an entirely different story.

As far as performance considerations for the merge... FS-cache on flash-based storage also has very different performance tradeoffs from traditional rotating media. Specifically I have some sample 32GB SATA-based flash media here with ~230Mbyte/sec sustained read and ~200Mbyte/sec sustained write and with a 75usec read latency. It doesn't take much link latency at all to completely dwarf that kind of access time.

Cheers,
Kyle Moffett