Skip to contentSkip to conversation

fs: truncate introduce new sequence

PATCH In review Nick Piggin 3 participants 16 messages Dec 8, 2009 – Dec 10, 2009

Subsystems (9)
lore
Share & tools
lore
#0 lore raw

PATCH 1/5 fs: truncate introduce new sequence

Changes: avoid .truncate_kludge_to_be_killed hack by implementing additional fs helper functions that do not call vmtruncate.

So far have converted ext2 and fat, tested with fsx-linux and the recent writev truncate bug tester. The reset of the patches that were already made are pretty simple to convert but I will likely prefer to send those via their maintainers after this initial set is agreed and merged.

The new helpers, *_newtrunc, are a bit ugly, but we can do a rename back to the old names after all support for old sequence is removed.

--
Introduce a new truncate calling sequence into fs/mm subsystems. Rather than setattr > vmtruncate > truncate, have filesystems call their truncate sequence from ->setattr if filesystem specific operations are required. vmtruncate is deprecated, and truncate_pagecache and inode_newsize_ok helpers introduced previously should be used.

simple_setattr is introduced for simple in-ram filesystems to implement the new truncate sequence. Eventually all filesystems should be converted to implement a setattr, and the default code in notify_change should go away.

simple_setsize is also introduced to perform just the ATTR_SIZE portion of simple_setattr (ie. changing i_size and trimming pagecache).

To implement the new truncate sequence:
- filesystem specific manipulations (eg freeing blocks) must be done in the setattr method rather than ->truncate.
- vmtruncate can not be used by core code to trim blocks past i_size in the event of write failure after allocation, so this must be performed in the fs code.
- convert usage of helpers block_write_begin, nobh_write_begin, cont_write_begin, and *blockdev_direct_IO* to use _newtrunc postfixed variants. These avoid calling vmtruncate to trim blocks (see previous).
- inode_setattr should not be used. generic_setattr is a new function to be used to copy simple attributes into the generic inode.
- make use of the better opportunity to handle errors with the new sequence.

Big problem with the previous calling sequence: the filesystem is not called until i_size has already changed. This means it is not allowed to fail the call, and also it does not know what the previous i_size was. Also, generic code calling vmtruncate to truncate allocated blocks in case of error had no good way to return a meaningful error (or, for example, atomically handle block deallocation).

Cc Christoph Hellwig <[email protected]>
---
 Documentation/filesystems/vfs.txt |    7 +-
 fs/attr.c                         |   50 ++++++++++++---
 fs/buffer.c                       |  123 ++++++++++++++++++++++++++++++--------
 fs/direct-io.c                    |   85 ++++++++++++++++----------
 fs/libfs.c                        |   83 +++++++++++++++++++++++++
 include/linux/buffer_head.h       |    9 ++
 include/linux/fs.h                |   36 ++++++++++-
 mm/truncate.c                     |   10 +--
 8 files changed, 328 insertions(+), 75 deletions(-)
Documentation/filesystems/vfs.txt +6 -1
Index: linux-2.6/Documentation/filesystems/vfs.txt
===================================================================
@@ -401,11 +401,16 @@ otherwise noted.
   	started might not be in the page cache at the end of the
   	walk).
 
-  truncate: called by the VFS to change the size of a file.  The
+  truncate: Deprecated. This will not be called if ->setsize is defined.
+	Called by the VFS to change the size of a file.  The
  	i_size field of the inode is set to the desired size by the
  	VFS before this method is called.  This method is called by
  	the truncate(2) system call and related functionality.
 
+	Note: ->truncate and vmtruncate are deprecated. Do not add new
+	instances/calls of these. Filesystems shoud be converted to do their
+	truncate sequence via ->setattr().
+
   permission: called by the VFS to check for access rights on a POSIX-like
   	filesystem.
 
fs/attr.c +40 -10

77 lines not on this page · raw message

fs/buffer.c +98 -25

183 lines not on this page · raw message

fs/direct-io.c +52 -33

103 lines not on this page · raw message

fs/libfs.c +83 -0

95 lines not on this page · raw message

include/linux/fs.h +35 -1
Index: linux-2.6/include/linux/fs.h
===================================================================
@@ -2259,6 +2259,10 @@ static inline int xip_truncate_page(stru
 #endif
 
 #ifdef CONFIG_BLOCK
+ssize_t __blockdev_direct_IO_newtrunc(int rw, struct kiocb *iocb, struct inode *inode,
+	struct block_device *bdev, const struct iovec *iov, loff_t offset,
+	unsigned long nr_segs, get_block_t get_block, dio_iodone_t end_io,
+	int lock_type);
 ssize_t __blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
 	struct block_device *bdev, const struct iovec *iov, loff_t offset,
 	unsigned long nr_segs, get_block_t get_block, dio_iodone_t end_io,
@@ -2270,6 +2274,33 @@ enum {
 	DIO_OWN_LOCKING, /* filesystem locks buffered and direct internally */
 };
 
+static inline ssize_t blockdev_direct_IO_newtrunc(int rw, struct kiocb *iocb,
+	struct inode *inode, struct block_device *bdev, const struct iovec *iov,
+	loff_t offset, unsigned long nr_segs, get_block_t get_block,
+	dio_iodone_t end_io)
+{
+	return __blockdev_direct_IO_newtrunc(rw, iocb, inode, bdev, iov, offset,
+				nr_segs, get_block, end_io, DIO_LOCKING);
+}
+
+static inline ssize_t blockdev_direct_IO_no_locking_newtrunc(int rw, struct kiocb *iocb,
+	struct inode *inode, struct block_device *bdev, const struct iovec *iov,
+	loff_t offset, unsigned long nr_segs, get_block_t get_block,
+	dio_iodone_t end_io)
+{
+	return __blockdev_direct_IO_newtrunc(rw, iocb, inode, bdev, iov, offset,
+				nr_segs, get_block, end_io, DIO_NO_LOCKING);
+}
+
+static inline ssize_t blockdev_direct_IO_own_locking_newtrunc(int rw, struct kiocb *iocb,
+	struct inode *inode, struct block_device *bdev, const struct iovec *iov,
+	loff_t offset, unsigned long nr_segs, get_block_t get_block,
+	dio_iodone_t end_io)
+{
+	return __blockdev_direct_IO_newtrunc(rw, iocb, inode, bdev, iov, offset,
+				nr_segs, get_block, end_io, DIO_OWN_LOCKING);
+}
+
 static inline ssize_t blockdev_direct_IO(int rw, struct kiocb *iocb,
 	struct inode *inode, struct block_device *bdev, const struct iovec *iov,
 	loff_t offset, unsigned long nr_segs, get_block_t get_block,
@@ -2347,12 +2378,14 @@ extern int dcache_dir_open(struct inode
 extern int dcache_dir_close(struct inode *, struct file *);
 extern loff_t dcache_dir_lseek(struct file *, loff_t, int);
 extern int dcache_readdir(struct file *, void *, filldir_t);
+extern int simple_setattr(struct dentry *dentry, struct iattr *attr);
 extern int simple_getattr(struct vfsmount *, struct dentry *, struct kstat *);
 extern int simple_statfs(struct dentry *, struct kstatfs *);
 extern int simple_link(struct dentry *, struct inode *, struct dentry *);
 extern int simple_unlink(struct inode *, struct dentry *);
 extern int simple_rmdir(struct inode *, struct dentry *);
 extern int simple_rename(struct inode *, struct dentry *, struct inode *, struct dentry *);
+extern int simple_setsize(struct inode *inode, loff_t newsize);
 extern int simple_sync_file(struct file *, struct dentry *, int);
 extern int simple_empty(struct dentry *);
 extern int simple_readpage(struct file *file, struct page *page);
@@ -2389,7 +2422,8 @@ extern int buffer_migrate_page(struct ad
 
 extern int inode_change_ok(const struct inode *, struct iattr *);
 extern int inode_newsize_ok(const struct inode *, loff_t offset);
-extern int __must_check inode_setattr(struct inode *, struct iattr *);
+extern int __must_check inode_setattr(struct inode *, const struct iattr *);
+extern void generic_setattr(struct inode *inode, const struct iattr *attr);
 
 extern void file_update_time(struct file *file);
 
mm/truncate.c +5 -5
Index: linux-2.6/mm/truncate.c
===================================================================
@@ -543,18 +543,18 @@ EXPORT_SYMBOL(truncate_pagecache);
  * NOTE! We have to be ready to update the memory sharing
  * between the file and the memory map for a potential last
  * incomplete page.  Ugly, but necessary.
+ *
+ * This function is deprecated and simple_setsize or truncate_pagecache
+ * should be used instead.
  */
 int vmtruncate(struct inode *inode, loff_t offset)
 {
-	loff_t oldsize;
 	int error;
 
-	error = inode_newsize_ok(inode, offset);
+	error = simple_setsize(inode, offset);
 	if (error)
 		return error;
-	oldsize = inode->i_size;
-	i_size_write(inode, offset);
-	truncate_pagecache(inode, oldsize, offset);
+
 	if (inode->i_op->truncate)
 		inode->i_op->truncate(inode);
 
include/linux/buffer_head.h +9 -0
Index: linux-2.6/include/linux/buffer_head.h
===================================================================
@@ -203,6 +203,9 @@ int block_write_full_page_endio(struct p
 int block_read_full_page(struct page*, get_block_t*);
 int block_is_partially_uptodate(struct page *page, read_descriptor_t *desc,
 				unsigned long from);
+int block_write_begin_newtrunc(struct file *, struct address_space *,
+				loff_t, unsigned, unsigned,
+				struct page **, void **, get_block_t*);
 int block_write_begin(struct file *, struct address_space *,
 				loff_t, unsigned, unsigned,
 				struct page **, void **, get_block_t*);
@@ -214,6 +217,9 @@ int generic_write_end(struct file *, str
 				struct page *, void *);
 void page_zero_new_buffers(struct page *page, unsigned from, unsigned to);
 int block_prepare_write(struct page*, unsigned, unsigned, get_block_t*);
+int cont_write_begin_newtrunc(struct file *, struct address_space *, loff_t,
+			unsigned, unsigned, struct page **, void **,
+			get_block_t *, loff_t *);
 int cont_write_begin(struct file *, struct address_space *, loff_t,
 			unsigned, unsigned, struct page **, void **,
 			get_block_t *, loff_t *);
@@ -225,6 +231,9 @@ void block_sync_page(struct page *);
 sector_t generic_block_bmap(struct address_space *, sector_t, get_block_t *);
 int block_truncate_page(struct address_space *, loff_t, get_block_t *);
 int file_fsync(struct file *, struct dentry *, int);
+int nobh_write_begin_newtrunc(struct file *, struct address_space *,
+				loff_t, unsigned, unsigned,
+				struct page **, void **, get_block_t*);
 int nobh_write_begin(struct file *, struct address_space *,
 				loff_t, unsigned, unsigned,
 				struct page **, void **, get_block_t*);
#1 lore raw

PATCH 2/5 fs: convert simple fs to new truncate

Convert simple filesystems: ramfs, configfs, sysfs, block_dev to new truncate sequence.

---
 fs/block_dev.c        |    9 +++++----
 fs/configfs/inode.c   |   10 +++++++---
 fs/ramfs/file-mmu.c   |    1 +
 fs/ramfs/file-nommu.c |    7 ++++---
 fs/sysfs/inode.c      |    6 +-----
 5 files changed, 18 insertions(+), 15 deletions(-)
fs/configfs/inode.c +7 -3
Index: linux-2.6/fs/configfs/inode.c
===================================================================
@@ -77,9 +77,13 @@ int configfs_setattr(struct dentry * den
 	if (error)
 		return error;
 
-	error = inode_setattr(inode, iattr);
-	if (error)
-		return error;
+	if (iattr->ia_valid & ATTR_SIZE) {
+		error = simple_setsize(inode, iattr->ia_size);
+		if (error)
+			return error;
+	}
+
+	generic_setattr(inode, iattr);
 
 	if (!sd_iattr) {
 		/* setting attributes for the first time, allocate now */
fs/ramfs/file-mmu.c +1 -0
Index: linux-2.6/fs/ramfs/file-mmu.c
===================================================================
@@ -50,5 +50,6 @@ const struct file_operations ramfs_file_
 };
 
 const struct inode_operations ramfs_file_inode_operations = {
+	.setattr	= simple_setattr,
 	.getattr	= simple_getattr,
 };
fs/ramfs/file-nommu.c +4 -3
Index: linux-2.6/fs/ramfs/file-nommu.c
===================================================================
@@ -169,7 +169,7 @@ static int ramfs_nommu_resize(struct ino
 			return ret;
 	}
 
-	ret = vmtruncate(inode, newsize);
+	ret = simple_setsize(inode, newsize);
 
 	return ret;
 }
@@ -192,7 +192,8 @@ static int ramfs_nommu_setattr(struct de
 
 	/* pick out size-changing events */
 	if (ia->ia_valid & ATTR_SIZE) {
-		loff_t size = i_size_read(inode);
+		loff_t size = inode->i_size;
+
 		if (ia->ia_size != size) {
 			ret = ramfs_nommu_resize(inode, ia->ia_size, size);
 			if (ret < 0 || ia->ia_valid == ATTR_SIZE)
@@ -205,7 +206,7 @@ static int ramfs_nommu_setattr(struct de
 		}
 	}
 
-	ret = inode_setattr(inode, ia);
+	generic_setattr(inode, ia);
  out:
 	ia->ia_valid = old_ia_valid;
 	return ret;
fs/sysfs/inode.c +1 -5
Index: linux-2.6/fs/sysfs/inode.c
===================================================================
@@ -82,11 +82,7 @@ int sysfs_setattr(struct dentry * dentry
 	if (error)
 		return error;
 
-	iattr->ia_valid &= ~ATTR_SIZE; /* ignore size changes */
-
-	error = inode_setattr(inode, iattr);
-	if (error)
-		return error;
+	generic_setattr(inode, iattr);
 
 	if (!sd_attrs) {
 		/* setting attributes for the first time, allocate now */
fs/block_dev.c +5 -4
Index: linux-2.6/fs/block_dev.c
===================================================================
@@ -172,8 +172,9 @@ blkdev_direct_IO(int rw, struct kiocb *i
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file->f_mapping->host;
 
-	return blockdev_direct_IO_no_locking(rw, iocb, inode, I_BDEV(inode),
-				iov, offset, nr_segs, blkdev_get_blocks, NULL);
+	return blockdev_direct_IO_no_locking_newtrunc(rw, iocb, inode,
+				I_BDEV(inode), iov, offset, nr_segs,
+				blkdev_get_blocks, NULL);
 }
 
 int __sync_blockdev(struct block_device *bdev, int wait)
@@ -349,8 +350,8 @@ static int blkdev_write_begin(struct fil
 			struct page **pagep, void **fsdata)
 {
 	*pagep = NULL;
-	return block_write_begin(file, mapping, pos, len, flags, pagep, fsdata,
-				blkdev_get_block);
+	return block_write_begin_newtrunc(file, mapping, pos, len, flags,
+				pagep, fsdata, blkdev_get_block);
 }
 
 static int blkdev_write_end(struct file *file, struct address_space *mapping,
#3 lore raw

Re: [patch 2/5] fs: convert simple fs to new truncate

Quoted 7 lines · Jan Kara
Quoted 4 lines · Nick Piggin

Convert simple filesystems: ramfs, configfs, sysfs, block_dev to new truncate sequence.

Looks good.

Thanks.

Al, any chance to merge patches 1 and 2 upstream early in this round? They should be fairly non intrusive and back compatible. Then I can get the filesystem specific patches to their maintainers and you won't have to bother with them.

ecryptfs is being fixed to use notify_change rather than vmtruncate (which is anyway buggy today), so that won't be a problem.

I think we'll be able to get rid of old protocol completely within just a couple of releases. Jan and Christoph have made pretty good progress there.

#4 lore raw

Re: [patch 2/5] fs: convert simple fs to new truncate

Quoted 6 lines · Nick Piggin

Al, any chance to merge patches 1 and 2 upstream early in this round? They should be fairly non intrusive and back compatible. Then I can get the filesystem specific patches to their maintainers and you won't have to bother with them.

Quoted 7 lines

ecryptfs is being fixed to use notify_change rather than vmtruncate (which is anyway buggy today), so that won't be a problem.

I think we'll be able to get rid of old protocol completely within just a couple of releases. Jan and Christoph have made pretty good progress there.

Nick Hi.

I want to do the exofs conversion. Is there a public git-tree with all this that I can base my work, and run tests?

Thanks
Boaz

#5 lore raw

Re: [patch 2/5] fs: convert simple fs to new truncate

Quoted 15 lines · Boaz Harrosh
Quoted 6 lines · Nick Piggin

Al, any chance to merge patches 1 and 2 upstream early in this round? They should be fairly non intrusive and back compatible. Then I can get the filesystem specific patches to their maintainers and you won't have to bother with them.

Quoted 7 lines

ecryptfs is being fixed to use notify_change rather than vmtruncate (which is anyway buggy today), so that won't be a problem.

I think we'll be able to get rid of old protocol completely within just a couple of releases. Jan and Christoph have made pretty good progress there.

Quoted 7 lines

Nick Hi.

I want to do the exofs conversion. Is there a public git-tree with all this that I can base my work, and run tests?

Thanks
Boaz

Hi Boaz,

I don't have any git trees published. Apart from merging the patch by hand, I should hope it goes upstream with Al's vfs merge.

If you've got patches based on the previous patchset in Al's tree, then it is identical except that we don't switch those helper functions with the .truncate_kludge = 1 flag but instead use _newtrunc variants.

Thanks,
Nick

#6 lore raw

PATCH 3/5 tmpfs: convert to use the new truncate convention

In reply to Nick Piggin
---
 mm/shmem.c |   43 ++++++++++++++++++++++---------------------
 1 file changed, 22 insertions(+), 21 deletions(-)
mm/shmem.c +22 -21
Index: linux-2.6/mm/shmem.c
===================================================================
@@ -729,10 +729,11 @@ done2:
 	if (inode->i_mapping->nrpages && (info->flags & SHMEM_PAGEIN)) {
 		/*
 		 * Call truncate_inode_pages again: racing shmem_unuse_inode
-		 * may have swizzled a page in from swap since vmtruncate or
-		 * generic_delete_inode did it, before we lowered next_index.
-		 * Also, though shmem_getpage checks i_size before adding to
-		 * cache, no recheck after: so fix the narrow window there too.
+		 * may have swizzled a page in from swap since
+		 * truncate_pagecache or generic_delete_inode did it, before we
+		 * lowered next_index.  Also, though shmem_getpage checks
+		 * i_size before adding to cache, no recheck after: so fix the
+		 * narrow window there too.
 		 *
 		* Recalling truncate_inode_pages_range and unmap_mapping_range
 		 * every time for punch_hole (which never got a chance to clear
@@ -762,19 +763,16 @@ done2:
 	}
 }
 
-static void shmem_truncate(struct inode *inode)
-{
-	shmem_truncate_range(inode, inode->i_size, (loff_t)-1);
-}
-
 static int shmem_notify_change(struct dentry *dentry, struct iattr *attr)
 {
 	struct inode *inode = dentry->d_inode;
-	struct page *page = NULL;
 	int error;
 
 	if (S_ISREG(inode->i_mode) && (attr->ia_valid & ATTR_SIZE)) {
-		if (attr->ia_size < inode->i_size) {
+		loff_t newsize = attr->ia_size;
+		struct page *page = NULL;
+
+		if (newsize < inode->i_size) {
 			/*
 			 * If truncating down to a partial page, then
 			 * if that page is already allocated, hold it
@@ -782,9 +780,9 @@ static int shmem_notify_change(struct de
 			 * truncate_partial_page cannnot miss it were
 			 * it assigned to swap.
 			 */
-			if (attr->ia_size & (PAGE_CACHE_SIZE-1)) {
+			if (newsize & (PAGE_CACHE_SIZE-1)) {
 				(void) shmem_getpage(inode,
-					attr->ia_size>>PAGE_CACHE_SHIFT,
+					newsize >> PAGE_CACHE_SHIFT,
 						&page, SGP_READ, NULL);
 				if (page)
 					unlock_page(page);
@@ -796,24 +794,29 @@ static int shmem_notify_change(struct de
 			 * if it's being fully truncated to zero-length: the
 			 * nrpages check is efficient enough in that case.
 			 */
-			if (attr->ia_size) {
+			if (newsize) {
 				struct shmem_inode_info *info = SHMEM_I(inode);
 				spin_lock(&info->lock);
 				info->flags &= ~SHMEM_PAGEIN;
 				spin_unlock(&info->lock);
 			}
 		}
+
+		error = simple_setsize(inode, newsize);
+		if (page)
+			page_cache_release(page);
+		if (error)
+			return error;
+		shmem_truncate_range(inode, newsize, (loff_t)-1);
 	}
 
 	error = inode_change_ok(inode, attr);
 	if (!error)
-		error = inode_setattr(inode, attr);
+		generic_setattr(inode, attr);
 #ifdef CONFIG_TMPFS_POSIX_ACL
 	if (!error && (attr->ia_valid & ATTR_MODE))
 		error = generic_acl_chmod(inode, &shmem_acl_ops);
 #endif
-	if (page)
-		page_cache_release(page);
 	return error;
 }
 
@@ -821,11 +824,11 @@ static void shmem_delete_inode(struct in
 {
 	struct shmem_inode_info *info = SHMEM_I(inode);
 
-	if (inode->i_op->truncate == shmem_truncate) {
+	if (inode->i_mapping->a_ops == &shmem_aops) {
 		truncate_inode_pages(inode->i_mapping, 0);
 		shmem_unacct_size(info->flags, inode->i_size);
 		inode->i_size = 0;
-		shmem_truncate(inode);
+		shmem_truncate_range(inode, 0, (loff_t)-1);
 		if (!list_empty(&info->swaplist)) {
 			mutex_lock(&shmem_swaplist_mutex);
 			list_del_init(&info->swaplist);
@@ -2022,7 +2025,6 @@ static const struct inode_operations shm
 };
 
 static const struct inode_operations shmem_symlink_inode_operations = {
-	.truncate	= shmem_truncate,
 	.readlink	= generic_readlink,
 	.follow_link	= shmem_follow_link,
 	.put_link	= shmem_put_link,
@@ -2439,7 +2441,6 @@ static const struct file_operations shme
 };
 
 static const struct inode_operations shmem_inode_operations = {
-	.truncate	= shmem_truncate,
 	.setattr	= shmem_notify_change,
 	.truncate_range	= shmem_truncate_range,
 #ifdef CONFIG_TMPFS_POSIX_ACL
maintainer
#7 lore raw

Re: [patch 3/5] tmpfs: convert to use the new truncate convention

Quoted 3 lines · Nick Piggin

...

mm/shmem.c static void shmem_delete_inode(struct in · line 827

Quoted 6 lines
mm/shmem.c +1 -1
@@ -821,11 +824,11 @@ static void shmem_delete_inode(struct in
 {
 	struct shmem_inode_info *info = SHMEM_I(inode);
 
-	if (inode->i_op->truncate == shmem_truncate) {
+	if (inode->i_mapping->a_ops == &shmem_aops) {

How about symlinks here? Originally we've truncated inode for them as
well... BTW the check is really strange. Shouldn't we rather make it something like S_ISLNK || S_ISREG?

mm/shmem.c static void shmem_delete_inode(struct in · line 834

Quoted 8 lines
mm/shmem.c +1 -1
 		truncate_inode_pages(inode->i_mapping, 0);
 		shmem_unacct_size(info->flags, inode->i_size);
 		inode->i_size = 0;
-		shmem_truncate(inode);
+		shmem_truncate_range(inode, 0, (loff_t)-1);
 		if (!list_empty(&info->swaplist)) {
 			mutex_lock(&shmem_swaplist_mutex);
 			list_del_init(&info->swaplist);

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#8 lore raw

Re: [patch 3/5] tmpfs: convert to use the new truncate convention

Quoted 9 lines · Jan Kara
Quoted 3 lines · Nick Piggin

...

Quoted 4 lines
mm/shmem.c +0 -0
@@ -821,11 +824,11 @@ static void shmem_delete_inode(struct in
 {
 	struct shmem_inode_info *info = SHMEM_I(inode);
 
Quoted 5 lines

mm/shmem.c static void shmem_delete_inode(struct in · line 827

Quoted 2 lines
mm/shmem.c +1 -1
-	if (inode->i_op->truncate == shmem_truncate) {
+	if (inode->i_mapping->a_ops == &shmem_aops) {

How about symlinks here? Originally we've truncated inode for them as
well... BTW the check is really strange. Shouldn't we rather make it something like S_ISLNK || S_ISREG?

Yes you're right. I don't see why we can't use the mode check, Hugh?

#9 lore raw

PATCH 4/5 ext2: convert to use the new truncate convention

In reply to Nick Piggin

I also have commented a possible bug in existing ext2 code, marked with XXX.

---
 fs/ext2/ext2.h  |    1 
 fs/ext2/file.c  |    1 
 fs/ext2/inode.c |  153 +++++++++++++++++++++++++++++++++++++++++++-------------
 3 files changed, 119 insertions(+), 36 deletions(-)
fs/ext2/ext2.h +0 -1
Index: linux-2.6/fs/ext2/ext2.h
===================================================================
@@ -122,7 +122,6 @@ extern int ext2_write_inode (struct inod
 extern void ext2_delete_inode (struct inode *);
 extern int ext2_sync_inode (struct inode *);
 extern int ext2_get_block(struct inode *, sector_t, struct buffer_head *, int);
-extern void ext2_truncate (struct inode *);
 extern int ext2_setattr (struct dentry *, struct iattr *);
 extern void ext2_set_inode_flags(struct inode *inode);
 extern void ext2_get_inode_flags(struct ext2_inode_info *);
fs/ext2/file.c +0 -1
Index: linux-2.6/fs/ext2/file.c
===================================================================
@@ -77,7 +77,6 @@ const struct file_operations ext2_xip_fi
 #endif
 
 const struct inode_operations ext2_file_inode_operations = {
-	.truncate	= ext2_truncate,
 #ifdef CONFIG_EXT2_FS_XATTR
 	.setxattr	= generic_setxattr,
 	.getxattr	= generic_getxattr,
fs/ext2/inode.c +119 -34

232 lines not on this page · raw message

maintainer
#10 lore raw

Re: [patch 4/5] ext2: convert to use the new truncate convention

Quoted 5 lines · Nick Piggin

I also have commented a possible bug in existing ext2 code, marked with XXX.

...

fs/ext2/inode.c int __ext2_write_begin(struct file *file · line 769

Quoted 9 lines
fs/ext2/inode.c +2 -2
@@ -752,8 +764,8 @@ int __ext2_write_begin(struct file *file
 		loff_t pos, unsigned len, unsigned flags,
 		struct page **pagep, void **fsdata)
 {
-	return block_write_begin(file, mapping, pos, len, flags, pagep, fsdata,
-							ext2_get_block);
+	return block_write_begin_newtrunc(file, mapping, pos, len, flags,
+					pagep, fsdata, ext2_get_block);
 }

OK, but you should update the code in dir.c using __ext2_write_begin,
shouldn't you?

fs/ext2/inode.c ext2_write_begin(struct file *file, stru · line 795

Quoted 11 lines
fs/ext2/inode.c +10 -0
+static int ext2_write_end(struct file *file, struct address_space *mapping,
+			loff_t pos, unsigned len, unsigned copied,
+			struct page *page, void *fsdata)
+{
+	int ret;
+
+	ret = generic_write_end(file, mapping, pos, len, copied, page, fsdata);
+	if (ret < len)
+		ext2_write_failed(mapping, pos + len);
+	return ret;
 }

OK, when doing this, please also update ext2_commit_chunk in dir.c...

Quoted 7 lines
fs/ext2/inode.c +7 -0
+static void ext2_truncate_blocks(struct inode *inode, loff_t offset)
+{
+	/*
+	 * XXX: it seems like a bug here that we don't allow
+	 * IS_APPEND inode to have blocks-past-i_size trimmed off.
+	 * review and fix this.
+	 *

fs/ext2/inode.c do_indirects: · line 1163

Quoted 10 lines
fs/ext2/inode.c +10 -0
+	 * Also would be nice to be able to handle IO errors and such,
+	 * but that's probably too much to ask.
+	 */
+	if (!(S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
+	    S_ISLNK(inode->i_mode)))
+		return;
+	if (ext2_inode_is_fast_symlink(inode))
+		return;
+	if (IS_APPEND(inode) || IS_IMMUTABLE(inode))
+		return;

Yes, I'd remove IS_APPEND check from here.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw

Re: [patch 4/5] ext2: convert to use the new truncate convention

Quoted 7 lines · Jan Kara
Quoted 5 lines · Nick Piggin

I also have commented a possible bug in existing ext2 code, marked with XXX.

...

Quoted 11 lines

fs/ext2/inode.c int __ext2_write_begin(struct file *file · line 769

Quoted 9 lines
fs/ext2/inode.c +2 -2
@@ -752,8 +764,8 @@ int __ext2_write_begin(struct file *file
 		loff_t pos, unsigned len, unsigned flags,
 		struct page **pagep, void **fsdata)
 {
-	return block_write_begin(file, mapping, pos, len, flags, pagep, fsdata,
-							ext2_get_block);
+	return block_write_begin_newtrunc(file, mapping, pos, len, flags,
+					pagep, fsdata, ext2_get_block);
 }

OK, but you should update the code in dir.c using __ext2_write_begin,
shouldn't you?

To trim off blocks past i_size on failure? Yes I guess it should. In fact, the ext2_write_failed call could just go into here I think. I'll take a look.

Quoted 4 lines
Quoted 4 lines
fs/ext2/inode.c +4 -0
+static int ext2_write_end(struct file *file, struct address_space *mapping,
+			loff_t pos, unsigned len, unsigned copied,
+			struct page *page, void *fsdata)
+{
Quoted 8 lines

fs/ext2/inode.c ext2_write_begin(struct file *file, stru · line 795

Quoted 7 lines
fs/ext2/inode.c +6 -0
+	int ret;
+
+	ret = generic_write_end(file, mapping, pos, len, copied, page, fsdata);
+	if (ret < len)
+		ext2_write_failed(mapping, pos + len);
+	return ret;
 }

OK, when doing this, please also update ext2_commit_chunk in dir.c...

I think commit_chunk is OK. Because block_write_end does not fail and the only reason for checking failure here is in case of a short copy (which won't happen in dir.c code).

Quoted 10 lines
Quoted 10 lines
fs/ext2/inode.c +10 -0
+static void ext2_truncate_blocks(struct inode *inode, loff_t offset)
+{
+	/*
+	 * XXX: it seems like a bug here that we don't allow
+	 * IS_APPEND inode to have blocks-past-i_size trimmed off.
+	 * review and fix this.
+	 *
+	 * Also would be nice to be able to handle IO errors and such,
+	 * but that's probably too much to ask.
+	 */
Quoted 8 lines

fs/ext2/inode.c do_indirects: · line 1163

Quoted 7 lines
fs/ext2/inode.c +7 -0
+	if (!(S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
+	    S_ISLNK(inode->i_mode)))
+		return;
+	if (ext2_inode_is_fast_symlink(inode))
+		return;
+	if (IS_APPEND(inode) || IS_IMMUTABLE(inode))
+		return;

Yes, I'd remove IS_APPEND check from here.

Didn't want to change that in this patch, but I'm happy for someone to fix it (and in ext3/4 etc).

The checks probably should be done at a different level anyway: by the time that this code gets called, the pagecache is truncated and i_size modified anyway so it seems buggy if this made any difference.

maintainer
#12 lore raw

Re: [patch 4/5] ext2: convert to use the new truncate convention

Quoted 20 lines · Nick Piggin
Quoted 7 lines · Jan Kara
Quoted 5 lines · Nick Piggin

I also have commented a possible bug in existing ext2 code, marked with XXX.

...

Quoted 11 lines

fs/ext2/inode.c int __ext2_write_begin(struct file *file · line 769

Quoted 9 lines
fs/ext2/inode.c +2 -2
@@ -752,8 +764,8 @@ int __ext2_write_begin(struct file *file
 		loff_t pos, unsigned len, unsigned flags,
 		struct page **pagep, void **fsdata)
 {
-	return block_write_begin(file, mapping, pos, len, flags, pagep, fsdata,
-							ext2_get_block);
+	return block_write_begin_newtrunc(file, mapping, pos, len, flags,
+					pagep, fsdata, ext2_get_block);
 }

OK, but you should update the code in dir.c using __ext2_write_begin,
shouldn't you?

Quoted 3 lines

To trim off blocks past i_size on failure? Yes I guess it should. In fact, the ext2_write_failed call could just go into here I think. I'll take a look.

Yes, that would be probably the easiest.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
fs/ext2/inode.c +4 -0
+static int ext2_write_end(struct file *file, struct address_space *mapping,
+			loff_t pos, unsigned len, unsigned copied,
+			struct page *page, void *fsdata)
+{
Quoted 8 lines

fs/ext2/inode.c ext2_write_begin(struct file *file, stru · line 795

Quoted 7 lines
fs/ext2/inode.c +6 -0
+	int ret;
+
+	ret = generic_write_end(file, mapping, pos, len, copied, page, fsdata);
+	if (ret < len)
+		ext2_write_failed(mapping, pos + len);
+	return ret;
 }

OK, when doing this, please also update ext2_commit_chunk in dir.c...

Quoted 3 lines

I think commit_chunk is OK. Because block_write_end does not fail and the only reason for checking failure here is in case of a short copy (which won't happen in dir.c code).

Ah, right.

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
fs/ext2/inode.c +10 -0
+static void ext2_truncate_blocks(struct inode *inode, loff_t offset)
+{
+	/*
+	 * XXX: it seems like a bug here that we don't allow
+	 * IS_APPEND inode to have blocks-past-i_size trimmed off.
+	 * review and fix this.
+	 *
+	 * Also would be nice to be able to handle IO errors and such,
+	 * but that's probably too much to ask.
+	 */
Quoted 8 lines

fs/ext2/inode.c do_indirects: · line 1163

Quoted 7 lines
fs/ext2/inode.c +7 -0
+	if (!(S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
+	    S_ISLNK(inode->i_mode)))
+		return;
+	if (ext2_inode_is_fast_symlink(inode))
+		return;
+	if (IS_APPEND(inode) || IS_IMMUTABLE(inode))
+		return;

Yes, I'd remove IS_APPEND check from here.

Quoted 2 lines

Didn't want to change that in this patch, but I'm happy for someone to fix it (and in ext3/4 etc).

OK.

Quoted 3 lines

The checks probably should be done at a different level anyway: by the time that this code gets called, the pagecache is truncated and i_size modified anyway so it seems buggy if this made any difference.

It depends. The first if can be WARN_ON as well - we shouldn't really get
different inode types here. For fast symlinks we have nothing to truncate so we want to immediately return doing nothing. For IMMUTABLE inode it is again a bug if we get to this function and for APPEND inode the only acceptable way to get here is the error recovery. We can clean that up when your patch series goes in.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#13 lore raw

PATCH 5/5 fat: convert to use the new truncate convention

In reply to Nick Piggin
Cc OGAWA Hirofumi <[email protected]>
Cc Christoph Hellwig <[email protected]>
---
 fs/fat/fat.h   |    3 ++-
 fs/fat/file.c  |   34 ++++++++++++++++++++++++++--------
 fs/fat/inode.c |   35 +++++++++++++++++++++++++++++------
 3 files changed, 57 insertions(+), 15 deletions(-)
fs/fat/fat.h +2 -1
Index: linux-2.6/fs/fat/fat.h
===================================================================
@@ -302,7 +302,8 @@ extern int fat_generic_ioctl(struct inod
 extern const struct file_operations fat_file_operations;
 extern const struct inode_operations fat_file_inode_operations;
 extern int fat_setattr(struct dentry * dentry, struct iattr * attr);
-extern void fat_truncate(struct inode *inode);
+extern int fat_setsize(struct inode *inode, loff_t offset);
+extern void fat_truncate_blocks(struct inode *inode, loff_t offset);
 extern int fat_getattr(struct vfsmount *mnt, struct dentry *dentry,
 		       struct kstat *stat);
 extern int fat_file_fsync(struct file *file, struct dentry *dentry,
fs/fat/file.c +26 -8
Index: linux-2.6/fs/fat/file.c
===================================================================
@@ -270,7 +270,7 @@ static int fat_free(struct inode *inode,
 	return fat_free_clusters(inode, free_start);
 }
 
-void fat_truncate(struct inode *inode)
+void fat_truncate_blocks(struct inode *inode, loff_t offset)
 {
 	struct msdos_sb_info *sbi = MSDOS_SB(inode->i_sb);
 	const unsigned int cluster_size = sbi->cluster_size;
@@ -280,10 +280,10 @@ void fat_truncate(struct inode *inode)
 	 * This protects against truncating a file bigger than it was then
 	 * trying to write into the hole.
 	 */
-	if (MSDOS_I(inode)->mmu_private > inode->i_size)
-		MSDOS_I(inode)->mmu_private = inode->i_size;
+	if (MSDOS_I(inode)->mmu_private > offset)
+		MSDOS_I(inode)->mmu_private = offset;
 
-	nr_clusters = (inode->i_size + (cluster_size - 1)) >> sbi->cluster_bits;
+	nr_clusters = (offset + (cluster_size - 1)) >> sbi->cluster_bits;
 
 	fat_free(inode, nr_clusters);
 	fat_flush_inodes(inode->i_sb, inode, NULL);
@@ -351,6 +351,18 @@ static int fat_allow_set_time(struct msd
 	return 0;
 }
 
+int fat_setsize(struct inode *inode, loff_t offset)
+{
+	int error;
+
+	error = simple_setsize(inode, offset);
+	if (error)
+		return error;
+	fat_truncate_blocks(inode, offset);
+
+	return error;
+}
+
 #define TIMES_SET_FLAGS	(ATTR_MTIME_SET | ATTR_ATIME_SET | ATTR_TIMES_SET)
 /* valid file mode bits */
 #define FAT_VALID_MODE	(S_IFREG | S_IFDIR | S_IRWXUGO)
@@ -365,7 +377,8 @@ int fat_setattr(struct dentry *dentry, s
 	/*
 	 * Expand the file. Since inode_setattr() updates ->i_size
 	 * before calling the ->truncate(), but FAT needs to fill the
-	 * hole before it.
+	 * hole before it. XXX: this is no longer true with new truncate
+	 * sequence.
 	 */
 	if (attr->ia_valid & ATTR_SIZE) {
 		if (attr->ia_size > inode->i_size) {
@@ -414,15 +427,20 @@ int fat_setattr(struct dentry *dentry, s
 			attr->ia_valid &= ~ATTR_MODE;
 	}
 
-	if (attr->ia_valid)
-		error = inode_setattr(inode, attr);
+	if (attr->ia_valid & ATTR_SIZE) {
+		error = fat_setsize(inode, attr->ia_size);
+		if (error)
+			goto out;
+	}
+
+	generic_setattr(inode, attr);
+	mark_inode_dirty(inode);
 out:
 	return error;
 }
 EXPORT_SYMBOL_GPL(fat_setattr);
 
 const struct inode_operations fat_file_inode_operations = {
-	.truncate	= fat_truncate,
 	.setattr	= fat_setattr,
 	.getattr	= fat_getattr,
 };
fs/fat/inode.c +29 -6
Index: linux-2.6/fs/fat/inode.c
===================================================================
@@ -142,14 +142,29 @@ static int fat_readpages(struct file *fi
 	return mpage_readpages(mapping, pages, nr_pages, fat_get_block);
 }
 
+static void fat_write_failed(struct address_space *mapping, loff_t to)
+{
+	struct inode *inode = mapping->host;
+
+	if (to > inode->i_size) {
+		truncate_pagecache(inode, to, inode->i_size);
+		fat_truncate_blocks(inode, inode->i_size);
+	}
+}
+
 static int fat_write_begin(struct file *file, struct address_space *mapping,
 			loff_t pos, unsigned len, unsigned flags,
 			struct page **pagep, void **fsdata)
 {
+	int err;
+
 	*pagep = NULL;
-	return cont_write_begin(file, mapping, pos, len, flags, pagep, fsdata,
-				fat_get_block,
+	err = cont_write_begin_newtrunc(file, mapping, pos, len, flags,
+				pagep, fsdata, fat_get_block,
 				&MSDOS_I(mapping->host)->mmu_private);
+	if (err < 0)
+		fat_write_failed(mapping, pos + len);
+	return err;
 }
 
 static int fat_write_end(struct file *file, struct address_space *mapping,
@@ -159,6 +174,8 @@ static int fat_write_end(struct file *fi
 	struct inode *inode = mapping->host;
 	int err;
 	err = generic_write_end(file, mapping, pos, len, copied, pagep, fsdata);
+	if (err < len)
+		fat_write_failed(mapping, pos + len);
 	if (!(err < 0) && !(MSDOS_I(inode)->i_attrs & ATTR_ARCH)) {
 		inode->i_mtime = inode->i_ctime = CURRENT_TIME_SEC;
 		MSDOS_I(inode)->i_attrs |= ATTR_ARCH;
@@ -172,7 +189,9 @@ static ssize_t fat_direct_IO(int rw, str
 			     loff_t offset, unsigned long nr_segs)
 {
 	struct file *file = iocb->ki_filp;
-	struct inode *inode = file->f_mapping->host;
+	struct address_space *mapping = file->f_mapping;
+	struct inode *inode = mapping->host;
+	ssize_t ret;
 
 	if (rw == WRITE) {
 		/*
@@ -193,8 +212,12 @@ static ssize_t fat_direct_IO(int rw, str
 	 * FAT need to use the DIO_LOCKING for avoiding the race
 	 * condition of fat_get_block() and ->truncate().
 	 */
-	return blockdev_direct_IO(rw, iocb, inode, inode->i_sb->s_bdev, iov,
-				  offset, nr_segs, fat_get_block, NULL);
+	ret = blockdev_direct_IO_newtrunc(rw, iocb, inode, inode->i_sb->s_bdev,
+				iov, offset, nr_segs, fat_get_block, NULL);
+	if (ret < 0 && (rw & WRITE))
+		fat_write_failed(mapping, offset + iov_length(iov, nr_segs));
+
+	return ret;
 }
 
 static sector_t _fat_bmap(struct address_space *mapping, sector_t block)
@@ -429,7 +452,7 @@ static void fat_delete_inode(struct inod
 {
 	truncate_inode_pages(&inode->i_data, 0);
 	inode->i_size = 0;
-	fat_truncate(inode);
+	fat_truncate_blocks(inode, 0);
 	clear_inode(inode);
 }
 
maintainer Acked-by on v1 5/5
#14 lore raw

Re: [patch 5/5] fat: convert to use the new truncate convention

Quoted 3 lines · Nick Piggin
Cc OGAWA Hirofumi <[email protected]>
Cc Christoph Hellwig <[email protected]>

Looks good except that the code in fat_setattr could be simplified I
think.

fs/fat/file.c int fat_setattr(struct dentry *dentry, s · line 384

Quoted 10 lines
fs/fat/file.c +2 -1
@@ -365,7 +377,8 @@ int fat_setattr(struct dentry *dentry, s
 	/*
 	 * Expand the file. Since inode_setattr() updates ->i_size
 	 * before calling the ->truncate(), but FAT needs to fill the
-	 * hole before it.
+	 * hole before it. XXX: this is no longer true with new truncate
+	 * sequence.
 	 */
 	if (attr->ia_valid & ATTR_SIZE) {
 		if (attr->ia_size > inode->i_size) {

I think the content of this condition can be moved inside the ATTR_SIZE
check you've added below and the comment can be removed...

Quoted 4 lines
fs/fat/file.c +0 -0
@@ -414,15 +427,20 @@ int fat_setattr(struct dentry *dentry, s
 			attr->ia_valid &= ~ATTR_MODE;
 	}
 

fs/fat/file.c int fat_setattr(struct dentry *dentry, s · line 437

Quoted 10 lines
fs/fat/file.c +8 -2
-	if (attr->ia_valid)
-		error = inode_setattr(inode, attr);
+	if (attr->ia_valid & ATTR_SIZE) {
+		error = fat_setsize(inode, attr->ia_size);
+		if (error)
+			goto out;
+	}
+
+	generic_setattr(inode, attr);
+	mark_inode_dirty(inode);

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer Acked-by on v1 1/5
#15 lore raw
In reply to Nick Piggin

Hi,

Quoted 10 lines · Nick Piggin

Changes: avoid .truncate_kludge_to_be_killed hack by implementing additional fs helper functions that do not call vmtruncate.

So far have converted ext2 and fat, tested with fsx-linux and the recent writev truncate bug tester. The reset of the patches that were already made are pretty simple to convert but I will likely prefer to send those via their maintainers after this initial set is agreed and merged.

The new helpers, *_newtrunc, are a bit ugly, but we can do a rename back to the old names after all support for old sequence is removed.

Yeah. I'm not sure what Al dislikes about the original temporary hack.
Maybe he's afraid that the hack will last longer than we'd like... What I dislike about this approach is that we'll have to change every filesystem in several places also in the second patch. Anyway I'd like to get this change merged and I don't care too much about details of the intermediate period...

Quoted 35 lines

--
Introduce a new truncate calling sequence into fs/mm subsystems. Rather than setattr > vmtruncate > truncate, have filesystems call their truncate sequence from ->setattr if filesystem specific operations are required. vmtruncate is deprecated, and truncate_pagecache and inode_newsize_ok helpers introduced previously should be used.

simple_setattr is introduced for simple in-ram filesystems to implement the new truncate sequence. Eventually all filesystems should be converted to implement a setattr, and the default code in notify_change should go away.

simple_setsize is also introduced to perform just the ATTR_SIZE portion of simple_setattr (ie. changing i_size and trimming pagecache).

To implement the new truncate sequence:
- filesystem specific manipulations (eg freeing blocks) must be done in the setattr method rather than ->truncate.
- vmtruncate can not be used by core code to trim blocks past i_size in the event of write failure after allocation, so this must be performed in the fs code.
- convert usage of helpers block_write_begin, nobh_write_begin, cont_write_begin, and *blockdev_direct_IO* to use _newtrunc postfixed variants. These avoid calling vmtruncate to trim blocks (see previous).
- inode_setattr should not be used. generic_setattr is a new function to be used to copy simple attributes into the generic inode.
- make use of the better opportunity to handle errors with the new sequence.

Big problem with the previous calling sequence: the filesystem is not called until i_size has already changed. This means it is not allowed to fail the call, and also it does not know what the previous i_size was. Also, generic code calling vmtruncate to truncate allocated blocks in case of error had no good way to return a meaningful error (or, for example, atomically handle block deallocation).

Quoted 2 lines
Cc Christoph Hellwig <[email protected]>

The patch looks fine except for a typo below.

Honza

Quoted 18 lines
---
 Documentation/filesystems/vfs.txt |    7 +-
 fs/attr.c                         |   50 ++++++++++++---
 fs/buffer.c                       |  123 ++++++++++++++++++++++++++++++--------
 fs/direct-io.c                    |   85 ++++++++++++++++----------
 fs/libfs.c                        |   83 +++++++++++++++++++++++++
 include/linux/buffer_head.h       |    9 ++
 include/linux/fs.h                |   36 ++++++++++-
 mm/truncate.c                     |   10 +--
 8 files changed, 328 insertions(+), 75 deletions(-)

Index: linux-2.6/Documentation/filesystems/vfs.txt
===================================================================

Documentation/filesystems/vfs.txt +0 -0
@@ -401,11 +401,16 @@ otherwise noted.
   	started might not be in the page cache at the end of the
   	walk).

Documentation/filesystems/vfs.txt otherwise noted. · line 411

Quoted 10 lines
Documentation/filesystems/vfs.txt +4 -1
 
-  truncate: called by the VFS to change the size of a file.  The
+  truncate: Deprecated. This will not be called if ->setsize is defined.
+	Called by the VFS to change the size of a file.  The
  	i_size field of the inode is set to the desired size by the
  	VFS before this method is called.  This method is called by
  	the truncate(2) system call and related functionality.
 
+	Note: ->truncate and vmtruncate are deprecated. Do not add new
+	instances/calls of these. Filesystems shoud be converted to do their
                                              ^^^^^ should
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR