Skip to contentSkip to conversation

Btrfs: add interface for writing compressed extent directly

RFC PATCH In review Omar Sandoval 4 participants 23 messages Aug 15, 2019 – Sep 3, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Btrfs: add interface for writing compressed extent directly

From: Omar Sandoval <[email protected]>

Hello,

This series adds a way to write compressed data directly to Btrfs. The intended use case is making send/receive on compressed file systems more efficient; however, the interface is general enough that it could be used in other scenarios. Patch 5 is the main change; see that for more details.

Patches 1-3 are small fixes/cleanups that I ran into while implementing this; they should go in regardless of the remainder of the series. Patch 4 exports a required VFS interface.

An example program and test case are available at [1].

To preemptively address a few concerns:

- Writing arbitrary, untrusted data which we feed to the decompression algorithm can be a security risk. For that reason, the ioctl is restricted to CAP_SYS_ADMIN. The Btrfs code is properly hardened against invalid compressed data/incorrect lengths, and the compression libraries are mature, but better safe than sorry for now.
- If the user is writing their own compressed data rather than just blindly feeding in something from btrfs send, they need to know some implementation details about the compression format. For zlib, there are no special requirements. For zstd, a non-default compression parameter must be used. For lzo, we have our own wrapper format since lzo doesn't have a standard wrapper format. It feels a little wrong to expose these details, but they are part of the on-disk format, so they must be stable regardless.
- The permissions checks duplicated from the VFS code are fairly minimal.

This series is based on misc-next.

This is an RFC, so please, comment away.

Thanks!

1: https://github.com/osandov/xfstests/tree/btrfs-compressed-write

Omar Sandoval (5):
Btrfs: use correct count in btrfs_file_write_iter()
Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs
Btrfs: stop clearing EXTENT_DIRTY in inode I/O tree
fs: export rw_verify_area()
Btrfs: add ioctl for directly writing compressed data

 fs/btrfs/compression.c       |   6 +-
 fs/btrfs/compression.h       |  14 +--
 fs/btrfs/ctree.h             |  12 ++
 fs/btrfs/extent_io.c         |   6 +-
 fs/btrfs/file.c              |  22 ++--
 fs/btrfs/free-space-cache.c  |   9 +-
 fs/btrfs/inode.c             | 232 +++++++++++++++++++++++++++++++----
 fs/btrfs/ioctl.c             | 101 ++++++++++++++-
 fs/btrfs/tests/inode-tests.c |  12 +-
 fs/internal.h                |   5 -
 fs/read_write.c              |   1 +
 include/linux/fs.h           |   1 +
 include/uapi/linux/btrfs.h   |  63 ++++++++++
 13 files changed, 415 insertions(+), 69 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/5 Btrfs: use correct count in btrfs_file_write_iter()

From: Omar Sandoval <[email protected]>

generic_write_checks() may modify iov_iter_count(), so we must get the count after the call, not before. Using the wrong one has a couple of consequences:

1. We check a longer range in check_can_nocow() for nowait than we're actually writing.
2. We create extra hole extent maps in btrfs_cont_expand(). As far as I can tell, this is harmless, but I might be missing something.

These issues are pretty minor, but let's fix it before something more important trips on it.

Fixes edf064e7c6fe ("btrfs: nowait aio support")
---
 fs/btrfs/file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/btrfs/file.c +2 -1
index b31991f0f440..4393b6b24e02 100644
@@ -1885,7 +1885,7 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	bool sync = (file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host);
 	ssize_t err;
 	loff_t pos;
-	size_t count = iov_iter_count(from);
+	size_t count;
 	loff_t oldsize;
 	int clean_page = 0;
 
@@ -1906,6 +1906,7 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	}
 
 	pos = iocb->ki_pos;
+	count = iov_iter_count(from);
 	if (iocb->ki_flags & IOCB_NOWAIT) {
 		/*
 		 * We will allocate space in case nodatacow is not set,
Signature · 1 line
2.17.1
#3 lore raw

PATCH 2/5 Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs

In reply to Omar Sandoval

From: Omar Sandoval <[email protected]>

In btrfs_file_write_iter(), we treat a write as synchrononous if the file is marked as synchronous. However, with pwritev2(), a write with RWF_SYNC or RWF_DSYNC is also synchronous even if the file isn't by default. Make sure we bump the sync_writers counter in that case, too, so that we'll do the CRCs synchronously.

---
 fs/btrfs/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/file.c +1 -1
index 4393b6b24e02..27223753da7b 100644
@@ -1882,7 +1882,7 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	u64 start_pos;
 	u64 end_pos;
 	ssize_t num_written = 0;
-	bool sync = (file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host);
+	bool sync = iocb->ki_flags & IOCB_DSYNC;
 	ssize_t err;
 	loff_t pos;
 	size_t count;
Signature · 1 line
2.17.1
#8 lore raw

PATCH 3/5 Btrfs: stop clearing EXTENT_DIRTY in inode I/O tree

In reply to Omar Sandoval

From: Omar Sandoval <[email protected]>

Since commit fee187d9d9dd ("Btrfs: do not set EXTENT_DIRTY along with EXTENT_DELALLOC"), we never set EXTENT_DIRTY in inode->io_tree, so we can simplify and stop trying to clear it.

---
 fs/btrfs/extent_io.c         |  6 ++----
 fs/btrfs/file.c              |  4 ++--
 fs/btrfs/free-space-cache.c  |  9 ++++----
 fs/btrfs/inode.c             | 41 ++++++++++++++----------------------
 fs/btrfs/ioctl.c             |  5 ++---
 fs/btrfs/tests/inode-tests.c | 12 ++++-------
 6 files changed, 30 insertions(+), 47 deletions(-)
fs/btrfs/extent_io.c +2 -4
index bac59d721b54..4dc5e6939856 100644
@@ -4322,10 +4322,8 @@ int extent_invalidatepage(struct extent_io_tree *tree,
 
 	lock_extent_bits(tree, start, end, &cached_state);
 	wait_on_page_writeback(page);
-	clear_extent_bit(tree, start, end,
-			 EXTENT_LOCKED | EXTENT_DIRTY | EXTENT_DELALLOC |
-			 EXTENT_DO_ACCOUNTING,
-			 1, 1, &cached_state);
+	clear_extent_bit(tree, start, end, EXTENT_LOCKED | EXTENT_DELALLOC |
+			 EXTENT_DO_ACCOUNTING, 1, 1, &cached_state);
 	return 0;
 }
 
fs/btrfs/file.c +2 -2
index 27223753da7b..c080fbcbda11 100644
@@ -537,8 +537,8 @@ int btrfs_dirty_pages(struct inode *inode, struct page **pages,
 	 * we can set things up properly
 	 */
 	clear_extent_bit(&BTRFS_I(inode)->io_tree, start_pos, end_of_last_block,
-			 EXTENT_DIRTY | EXTENT_DELALLOC |
-			 EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG, 0, 0, cached);
+			 EXTENT_DELALLOC | EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG,
+			 0, 0, cached);
 
 	if (!btrfs_is_free_space_inode(BTRFS_I(inode))) {
 		if (start_pos >= isize &&
fs/btrfs/free-space-cache.c +4 -5
index faaf57a7c289..96cf1e2dc388 100644
@@ -1005,7 +1005,7 @@ update_cache_item(struct btrfs_trans_handle *trans,
 	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
 	if (ret < 0) {
 		clear_extent_bit(&BTRFS_I(inode)->io_tree, 0, inode->i_size - 1,
-				 EXTENT_DIRTY | EXTENT_DELALLOC, 0, 0, NULL);
+				 EXTENT_DELALLOC, 0, 0, NULL);
 		goto fail;
 	}
 	leaf = path->nodes[0];
@@ -1017,9 +1017,8 @@ update_cache_item(struct btrfs_trans_handle *trans,
 		if (found_key.objectid != BTRFS_FREE_SPACE_OBJECTID ||
 		    found_key.offset != offset) {
 			clear_extent_bit(&BTRFS_I(inode)->io_tree, 0,
-					 inode->i_size - 1,
-					 EXTENT_DIRTY | EXTENT_DELALLOC, 0, 0,
-					 NULL);
+					 inode->i_size - 1, EXTENT_DELALLOC, 0,
+					 0, NULL);
 			btrfs_release_path(path);
 			goto fail;
 		}
@@ -1115,7 +1114,7 @@ static int flush_dirty_cache(struct inode *inode)
 	ret = btrfs_wait_ordered_range(inode, 0, (u64)-1);
 	if (ret)
 		clear_extent_bit(&BTRFS_I(inode)->io_tree, 0, inode->i_size - 1,
-				 EXTENT_DIRTY | EXTENT_DELALLOC, 0, 0, NULL);
+				 EXTENT_DELALLOC, 0, 0, NULL);
 
 	return ret;
 }
fs/btrfs/inode.c +16 -25

97 lines not on this page · raw message

fs/btrfs/ioctl.c +2 -3
index 4eabd419aaca..4b383811a7d2 100644
@@ -1333,9 +1333,8 @@ static int cluster_pages_for_defrag(struct inode *inode,
 	lock_extent_bits(&BTRFS_I(inode)->io_tree,
 			 page_start, page_end - 1, &cached_state);
 	clear_extent_bit(&BTRFS_I(inode)->io_tree, page_start,
-			  page_end - 1, EXTENT_DIRTY | EXTENT_DELALLOC |
-			  EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG, 0, 0,
-			  &cached_state);
+			  page_end - 1, EXTENT_DELALLOC | EXTENT_DO_ACCOUNTING |
+			  EXTENT_DEFRAG, 0, 0, &cached_state);
 
 	if (i_done != page_cnt) {
 		spin_lock(&BTRFS_I(inode)->lock);
fs/btrfs/tests/inode-tests.c +4 -8
index b363fb990cec..09ecf7dc7b08 100644
@@ -988,8 +988,7 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize)
 	ret = clear_extent_bit(&BTRFS_I(inode)->io_tree,
 			       BTRFS_MAX_EXTENT_SIZE >> 1,
 			       (BTRFS_MAX_EXTENT_SIZE >> 1) + sectorsize - 1,
-			       EXTENT_DELALLOC | EXTENT_DIRTY |
-			       EXTENT_UPTODATE, 0, 0, NULL);
+			       EXTENT_DELALLOC | EXTENT_UPTODATE, 0, 0, NULL);
 	if (ret) {
 		test_err("clear_extent_bit returned %d", ret);
 		goto out;
@@ -1056,8 +1055,7 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize)
 	ret = clear_extent_bit(&BTRFS_I(inode)->io_tree,
 			       BTRFS_MAX_EXTENT_SIZE + sectorsize,
 			       BTRFS_MAX_EXTENT_SIZE + 2 * sectorsize - 1,
-			       EXTENT_DIRTY | EXTENT_DELALLOC |
-			       EXTENT_UPTODATE, 0, 0, NULL);
+			       EXTENT_DELALLOC | EXTENT_UPTODATE, 0, 0, NULL);
 	if (ret) {
 		test_err("clear_extent_bit returned %d", ret);
 		goto out;
@@ -1089,8 +1087,7 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize)
 
 	/* Empty */
 	ret = clear_extent_bit(&BTRFS_I(inode)->io_tree, 0, (u64)-1,
-			       EXTENT_DIRTY | EXTENT_DELALLOC |
-			       EXTENT_UPTODATE, 0, 0, NULL);
+			       EXTENT_DELALLOC | EXTENT_UPTODATE, 0, 0, NULL);
 	if (ret) {
 		test_err("clear_extent_bit returned %d", ret);
 		goto out;
@@ -1105,8 +1102,7 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize)
 out:
 	if (ret)
 		clear_extent_bit(&BTRFS_I(inode)->io_tree, 0, (u64)-1,
-				 EXTENT_DIRTY | EXTENT_DELALLOC |
-				 EXTENT_UPTODATE, 0, 0, NULL);
+				 EXTENT_DELALLOC | EXTENT_UPTODATE, 0, 0, NULL);
 	iput(inode);
 	btrfs_free_dummy_root(root);
 	btrfs_free_dummy_fs_info(fs_info);
Signature · 1 line
2.17.1
#12 lore raw

RFC PATCH 5/5 Btrfs: add ioctl for directly writing compressed data

In reply to Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

---
 fs/btrfs/compression.c     |   6 +-
 fs/btrfs/compression.h     |  14 +--
 fs/btrfs/ctree.h           |  12 +++
 fs/btrfs/file.c            |  13 ++-
 fs/btrfs/inode.c           | 191 ++++++++++++++++++++++++++++++++++++-
 fs/btrfs/ioctl.c           |  96 +++++++++++++++++++
 include/uapi/linux/btrfs.h |  63 ++++++++++++
 7 files changed, 380 insertions(+), 15 deletions(-)
fs/btrfs/compression.c +4 -2
index 60c47b417a4b..50e3a9a7e829 100644
@@ -275,7 +275,8 @@ static void end_compressed_bio_write(struct bio *bio)
 			bio->bi_status == BLK_STS_OK);
 	cb->compressed_pages[0]->mapping = NULL;
 
-	end_compressed_writeback(inode, cb);
+	if (cb->writeback)
+		end_compressed_writeback(inode, cb);
 	/* note, our inode could be gone now */
 
 	/*
@@ -310,7 +311,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,
 				 unsigned long compressed_len,
 				 struct page **compressed_pages,
 				 unsigned long nr_pages,
-				 unsigned int write_flags)
+				 unsigned int write_flags, bool writeback)
 {
 	struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
 	struct bio *bio = NULL;
@@ -335,6 +336,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,
 	cb->mirror_num = 0;
 	cb->compressed_pages = compressed_pages;
 	cb->compressed_len = compressed_len;
+	cb->writeback = writeback;
 	cb->orig_bio = NULL;
 	cb->nr_pages = nr_pages;
 
fs/btrfs/compression.h +5 -9
index 2035b8eb1290..f39b69e8fbd7 100644
@@ -6,6 +6,7 @@
 #ifndef BTRFS_COMPRESSION_H
 #define BTRFS_COMPRESSION_H
 
+#include <linux/btrfs.h>
 #include <linux/sizes.h>
 
 /*
@@ -47,6 +48,9 @@ struct compressed_bio {
 	/* the compression algorithm for this bio */
 	int compress_type;
 
+	/* Whether this is a write for writeback. */
+	bool writeback;
+
 	/* number of compressed pages in the array */
 	unsigned long nr_pages;
 
@@ -93,20 +97,12 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,
 				  unsigned long compressed_len,
 				  struct page **compressed_pages,
 				  unsigned long nr_pages,
-				  unsigned int write_flags);
+				  unsigned int write_flags, bool writeback);
 blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,
 				 int mirror_num, unsigned long bio_flags);
 
 unsigned int btrfs_compress_str2level(unsigned int type, const char *str);
 
-enum btrfs_compression_type {
-	BTRFS_COMPRESS_NONE  = 0,
-	BTRFS_COMPRESS_ZLIB  = 1,
-	BTRFS_COMPRESS_LZO   = 2,
-	BTRFS_COMPRESS_ZSTD  = 3,
-	BTRFS_COMPRESS_TYPES = 3,
-};
-
 struct workspace_manager {
 	const struct btrfs_compress_op *ops;
 	struct list_head idle_ws;
fs/btrfs/ctree.h +12 -0
index 85b808e3ea42..e2854345a3a6 100644
@@ -2985,6 +2985,16 @@ int btrfs_run_delalloc_range(struct inode *inode, struct page *locked_page,
 int btrfs_writepage_cow_fixup(struct page *page, u64 start, u64 end);
 void btrfs_writepage_endio_finish_ordered(struct page *page, u64 start,
 					  u64 end, int uptodate);
+
+struct btrfs_compressed_write {
+	void __user *buf;
+	unsigned long compressed_len;
+	unsigned long orig_len;
+	int compress_type;
+};
+ssize_t btrfs_compressed_write(struct kiocb *iocb, struct iov_iter *from,
+			       struct btrfs_compressed_write *compressed);
+
 extern const struct dentry_operations btrfs_dentry_operations;
 
 /* ioctl.c */
@@ -3008,6 +3018,8 @@ int btrfs_add_inode_defrag(struct btrfs_trans_handle *trans,
 			   struct btrfs_inode *inode);
 int btrfs_run_defrag_inodes(struct btrfs_fs_info *fs_info);
 void btrfs_cleanup_defrag_inodes(struct btrfs_fs_info *fs_info);
+ssize_t btrfs_do_write_iter(struct kiocb *iocb, struct iov_iter *from,
+			    struct btrfs_compressed_write *compressed);
 int btrfs_sync_file(struct file *file, loff_t start, loff_t end, int datasync);
 void btrfs_drop_extent_cache(struct btrfs_inode *inode, u64 start, u64 end,
 			     int skip_pinned);
fs/btrfs/file.c +10 -3
index c080fbcbda11..1fcaa338baf5 100644
@@ -1872,8 +1872,8 @@ static void update_time_for_write(struct inode *inode)
 		inode_inc_iversion(inode);
 }
 
-static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
-				    struct iov_iter *from)
+ssize_t btrfs_do_write_iter(struct kiocb *iocb, struct iov_iter *from,
+			    struct btrfs_compressed_write *compressed)
 {
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file_inode(file);
@@ -1965,7 +1965,9 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	if (sync)
 		atomic_inc(&BTRFS_I(inode)->sync_writers);
 
-	if (iocb->ki_flags & IOCB_DIRECT) {
+	if (compressed) {
+		num_written = btrfs_compressed_write(iocb, from, compressed);
+	} else if (iocb->ki_flags & IOCB_DIRECT) {
 		num_written = __btrfs_direct_write(iocb, from);
 	} else {
 		num_written = btrfs_buffered_write(iocb, from);
@@ -1996,6 +1998,11 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	return num_written ? num_written : err;
 }
 
+static ssize_t btrfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
+{
+	return btrfs_do_write_iter(iocb, from, NULL);
+}
+
 int btrfs_release_file(struct inode *inode, struct file *filp)
 {
 	struct btrfs_file_private *private = filp->private_data;
fs/btrfs/inode.c +190 -1

203 lines not on this page · raw message

fs/btrfs/ioctl.c +96 -0

120 lines not on this page · raw message

include/uapi/linux/btrfs.h +63 -0

74 lines not on this page · raw message

Signature · 1 line
2.17.1
#10 lore raw

RFC PATCH 4/5 fs: export rw_verify_area()

In reply to Omar Sandoval

From: Omar Sandoval <[email protected]>

I'm adding a Btrfs ioctl to write compressed data, and rather than duplicating the checks in rw_verify_area(), let's just export it.

---
 fs/internal.h      | 5 -----
 fs/read_write.c    | 1 +
 include/linux/fs.h | 1 +
 3 files changed, 2 insertions(+), 5 deletions(-)
fs/internal.h +0 -5
index 315fcd8d237c..94e1831d4c95 100644
@@ -160,11 +160,6 @@ extern char *simple_dname(struct dentry *, char *, int);
 extern void dput_to_list(struct dentry *, struct list_head *);
 extern void shrink_dentry_list(struct list_head *);
 
-/*
- * read_write.c
- */
-extern int rw_verify_area(int, struct file *, const loff_t *, size_t);
-
 /*
  * pipe.c
  */
fs/read_write.c +1 -0
index 1f5088dec566..9d95491ce9ab 100644
@@ -399,6 +399,7 @@ int rw_verify_area(int read_write, struct file *file, const loff_t *ppos, size_t
 	return security_file_permission(file,
 				read_write == READ ? MAY_READ : MAY_WRITE);
 }
+EXPORT_SYMBOL(rw_verify_area);
 
 static ssize_t new_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos)
 {
include/linux/fs.h +1 -0
index 997a530ff4e9..a9a1884768e4 100644
@@ -3082,6 +3082,7 @@ extern loff_t fixed_size_llseek(struct file *file, loff_t offset,
 		int whence, loff_t size);
 extern loff_t no_seek_end_llseek_size(struct file *, loff_t, int, loff_t);
 extern loff_t no_seek_end_llseek(struct file *, loff_t, int);
+extern int rw_verify_area(int, struct file *, const loff_t *, size_t);
 extern int generic_file_open(struct inode * inode, struct file * filp);
 extern int nonseekable_open(struct inode * inode, struct file * filp);
 extern int stream_open(struct inode * inode, struct file * filp);
Signature · 1 line
2.17.1
#21 lore raw
In reply to Omar Sandoval
Quoted 49 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

Hello,

This series adds a way to write compressed data directly to Btrfs. The intended use case is making send/receive on compressed file systems more efficient; however, the interface is general enough that it could be used in other scenarios. Patch 5 is the main change; see that for more details.

Patches 1-3 are small fixes/cleanups that I ran into while implementing this; they should go in regardless of the remainder of the series. Patch 4 exports a required VFS interface.

An example program and test case are available at [1].

To preemptively address a few concerns:

- Writing arbitrary, untrusted data which we feed to the decompression algorithm can be a security risk. For that reason, the ioctl is restricted to CAP_SYS_ADMIN. The Btrfs code is properly hardened against invalid compressed data/incorrect lengths, and the compression libraries are mature, but better safe than sorry for now.
- If the user is writing their own compressed data rather than just blindly feeding in something from btrfs send, they need to know some implementation details about the compression format. For zlib, there are no special requirements. For zstd, a non-default compression parameter must be used. For lzo, we have our own wrapper format since lzo doesn't have a standard wrapper format. It feels a little wrong to expose these details, but they are part of the on-disk format, so they must be stable regardless.
- The permissions checks duplicated from the VFS code are fairly minimal.

This series is based on misc-next.

This is an RFC, so please, comment away.

Thanks!

1: https://github.com/osandov/xfstests/tree/btrfs-compressed-write

Omar Sandoval (5):
Btrfs: use correct count in btrfs_file_write_iter()
Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs
Btrfs: stop clearing EXTENT_DIRTY in inode I/O tree
fs: export rw_verify_area()
Btrfs: add ioctl for directly writing compressed data

Quoted 14 lines
 fs/btrfs/compression.c       |   6 +-
 fs/btrfs/compression.h       |  14 +--
 fs/btrfs/ctree.h             |  12 ++
 fs/btrfs/extent_io.c         |   6 +-
 fs/btrfs/file.c              |  22 ++--
 fs/btrfs/free-space-cache.c  |   9 +-
 fs/btrfs/inode.c             | 232 +++++++++++++++++++++++++++++++----
 fs/btrfs/ioctl.c             | 101 ++++++++++++++-
 fs/btrfs/tests/inode-tests.c |  12 +-
 fs/internal.h                |   5 -
 fs/read_write.c              |   1 +
 include/linux/fs.h           |   1 +
 include/uapi/linux/btrfs.h   |  63 ++++++++++
 13 files changed, 415 insertions(+), 69 deletions(-)

I forgot to CC fsdevel. I'll do that for v2.

Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH 1/5] Btrfs: use correct count in btrfs_file_write_iter()

In reply to Omar Sandoval
Quoted 11 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

generic_write_checks() may modify iov_iter_count(), so we must get the count after the call, not before. Using the wrong one has a couple of consequences:

1. We check a longer range in check_can_nocow() for nowait than we're actually writing.
2. We create extra hole extent maps in btrfs_cont_expand(). As far as I can tell, this is harmless, but I might be missing something.

Quoted 5 lines

These issues are pretty minor, but let's fix it before something more important trips on it.

Fixes edf064e7c6fe ("btrfs: nowait aio support")

Thanks,

Josef

Reviewed-by on v1 2/5
#4 lore raw

Re: [PATCH 2/5] Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs

In reply to Omar Sandoval
Quoted 9 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

In btrfs_file_write_iter(), we treat a write as synchrononous if the file is marked as synchronous. However, with pwritev2(), a write with RWF_SYNC or RWF_DSYNC is also synchronous even if the file isn't by default. Make sure we bump the sync_writers counter in that case, too, so that we'll do the CRCs synchronously.

Thanks,

Josef

Reviewed-by on v1 3/5
#9 lore raw

Re: [PATCH 3/5] Btrfs: stop clearing EXTENT_DIRTY in inode I/O tree

In reply to Omar Sandoval
Quoted 7 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

Since commit fee187d9d9dd ("Btrfs: do not set EXTENT_DIRTY along with EXTENT_DELALLOC"), we never set EXTENT_DIRTY in inode->io_tree, so we can simplify and stop trying to clear it.

Ship this, dear lord,

Thanks,

Josef

#13 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

In reply to Omar Sandoval
Quoted 16 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

Quoted 9 lines

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

I've looked at this a few times, the locking and space reservation stuff look right. What about encrypted send/recieve? Are we going to want to use this to just blind copy encrypted data without having to decrypt/re-encrypt? Should this be taken into consideration for this interface? I'll think more about it, but I can't really see any better option than this. Thanks,

Josef

#14 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

Quoted 27 lines · Josef Bacik
Quoted 16 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

Quoted 9 lines

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

Quoted 5 lines

I've looked at this a few times, the locking and space reservation stuff look right. What about encrypted send/recieve? Are we going to want to use this to just blind copy encrypted data without having to decrypt/re-encrypt? Should this be taken into consideration for this interface? I'll think more about it, but I can't really see any better option than this. Thanks,

The main problem is we don't have encryption implemented. And one of the larger aspects of the encryption support is going to be how we are storing the encryption keys. E.g. should they be part of the send format? Or are we going to limit send/receive based on whether the source/dest have transferred encryption keys out of line?

Quoted 3 lines

Josef

#15 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

Quoted 36 lines · Nikolay Borisov
Quoted 27 lines · Josef Bacik
Quoted 16 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

Quoted 9 lines

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

Quoted 5 lines

I've looked at this a few times, the locking and space reservation stuff look right. What about encrypted send/recieve? Are we going to want to use this to just blind copy encrypted data without having to decrypt/re-encrypt? Should this be taken into consideration for this interface? I'll think more about it, but I can't really see any better option than this. Thanks,

Quoted 6 lines

The main problem is we don't have encryption implemented. And one of the larger aspects of the encryption support is going to be how we are storing the encryption keys. E.g. should they be part of the send format? Or are we going to limit send/receive based on whether the source/dest have transferred encryption keys out of line?

Subvolume encryption will be coming soon, but I'm less worried about the mechanics of how that will be used and more worried about making this interface work for that eventual future. I assume we'll want to be able to just blind copy the encrypted data instead of decrypting into the send stream and then re-encrypting on the other side. Which means we'll have two uses for this interface, and I want to make sure we're happy with it before it gets merged. Thanks,

Josef

#5 lore raw

Re: [PATCH 2/5] Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs

In reply to Omar Sandoval
Quoted 18 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

In btrfs_file_write_iter(), we treat a write as synchrononous if the file is marked as synchronous. However, with pwritev2(), a write with RWF_SYNC or RWF_DSYNC is also synchronous even if the file isn't by default. Make sure we bump the sync_writers counter in that case, too, so that we'll do the CRCs synchronously.

---
 fs/btrfs/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/file.c +0 -0
index 4393b6b24e02..27223753da7b 100644

fs/btrfs/file.c static ssize_t btrfs_file_write_iter(struct kiocb *iocb, · line 1885

Quoted 5 lines
fs/btrfs/file.c +1 -1
@@ -1882,7 +1882,7 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	u64 start_pos;
 	u64 end_pos;
 	ssize_t num_written = 0;
-	bool sync = (file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host);
+	bool sync = iocb->ki_flags & IOCB_DSYNC;

I'd like to merge the patches 1-3, but have hard time matching the changelog to the change here. It's from one set of sync flags to another, mentioning pwritev2 but that's a syscall and the function itself does not use the sync flags at all. That's probably somewhere deep in the vfs calls but that's what I'd appreciate stated explicitly in the changelog as I was not able to find it out in a reasonable time.

#6 lore raw

Re: [PATCH 2/5] Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs

Quoted 23 lines · David Sterba
Quoted 22 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

In btrfs_file_write_iter(), we treat a write as synchrononous if the file is marked as synchronous. However, with pwritev2(), a write with RWF_SYNC or RWF_DSYNC is also synchronous even if the file isn't by default. Make sure we bump the sync_writers counter in that case, too, so that we'll do the CRCs synchronously.

---
 fs/btrfs/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/file.c +0 -1
index 4393b6b24e02..27223753da7b 100644
@@ -1882,7 +1882,7 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	u64 start_pos;
 	u64 end_pos;
 	ssize_t num_written = 0;
-	bool sync = (file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host);
Quoted 8 lines

fs/btrfs/file.c static ssize_t btrfs_file_write_iter(struct kiocb *iocb, · line 1885

Quoted 1 line
fs/btrfs/file.c +1 -0
+	bool sync = iocb->ki_flags & IOCB_DSYNC;

I'd like to merge the patches 1-3, but have hard time matching the changelog to the change here. It's from one set of sync flags to another, mentioning pwritev2 but that's a syscall and the function itself does not use the sync flags at all. That's probably somewhere deep in the vfs calls but that's what I'd appreciate stated explicitly in the changelog as I was not able to find it out in a reasonable time.

You're right, there are a few layers here. How about this for the changelog:

The VFS indicates a synchronous write to ->write_iter() via iocb->ki_flags. The IOCB_{,D}SYNC flags may be set based on the file (see iocb_flags()) or the RWF_* flags passed to a syscall like pwritev2() (see kiocb_set_rw_flags()). However, in btrfs_file_write_iter(), we're checking if a write is synchronous based only on the file; we use this to decide when to bump the sync_writers counter and thus do CRCs synchronously. Make sure we do this for all synchronous writes as determined by the VFS.

Let me know if you want me to resend with the new changelog.

#16 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

In reply to Josef Bacik
Quoted 52 lines · Josef Bacik
Quoted 36 lines · Nikolay Borisov
Quoted 27 lines · Josef Bacik
Quoted 16 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

Quoted 9 lines

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

Quoted 5 lines

I've looked at this a few times, the locking and space reservation stuff look right. What about encrypted send/recieve? Are we going to want to use this to just blind copy encrypted data without having to decrypt/re-encrypt? Should this be taken into consideration for this interface? I'll think more about it, but I can't really see any better option than this. Thanks,

Quoted 6 lines

The main problem is we don't have encryption implemented. And one of the larger aspects of the encryption support is going to be how we are storing the encryption keys. E.g. should they be part of the send format? Or are we going to limit send/receive based on whether the source/dest have transferred encryption keys out of line?

Subvolume encryption will be coming soon, but I'm less worried about the mechanics of how that will be used and more worried about making this interface work for that eventual future. I assume we'll want to be able to just blind copy the encrypted data instead of decrypting into the send stream and then re-encrypting on the other side. Which means we'll have two uses for this interface, and I want to make sure we're happy with it before it gets merged. Thanks,

Quoted 1 line

Josef

Right, I think the only way to do this would be to blindly send encrypted data, and leave the key management to a higher layer.

Looking at the ioctl definition:

struct btrfs_ioctl_compressed_pwrite_args {
        __u64 offset;           /* in */
        __u32 orig_len;         /* in */
        __u32 compressed_len;   /* in */
        __u32 compress_type;    /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));

I think there are enough reserved fields in there for, e.g., encryption type, any key management-related things we might need to stuff in, etc. But the naming would be pretty bad if we extended it this way. Maybe compressed write -> raw write, orig_len -> num_bytes, compressed_len -> disk_num_bytes?

struct btrfs_ioctl_raw_pwrite_args {
        __u64 offset;           /* in */
        __u32 num_bytes;        /* in */
        __u32 disk_num_bytes;   /* in */
        __u32 compress_type;    /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));

Besides the naming, I don't think anything else would need to change for now. And if we decide that we don't want encrypted send/receive, then fine, this naming is still okay.

#7 lore raw

Re: [PATCH 2/5] Btrfs: treat RWF_{,D}SYNC writes as sync for CRCs

In reply to Omar Sandoval
Quoted 37 lines · Omar Sandoval
Quoted 23 lines · David Sterba
Quoted 22 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

In btrfs_file_write_iter(), we treat a write as synchrononous if the file is marked as synchronous. However, with pwritev2(), a write with RWF_SYNC or RWF_DSYNC is also synchronous even if the file isn't by default. Make sure we bump the sync_writers counter in that case, too, so that we'll do the CRCs synchronously.

---
 fs/btrfs/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/file.c +0 -1
index 4393b6b24e02..27223753da7b 100644
@@ -1882,7 +1882,7 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
 	u64 start_pos;
 	u64 end_pos;
 	ssize_t num_written = 0;
-	bool sync = (file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host);
Quoted 8 lines

fs/btrfs/file.c static ssize_t btrfs_file_write_iter(struct kiocb *iocb, · line 1885

Quoted 1 line
fs/btrfs/file.c +1 -0
+	bool sync = iocb->ki_flags & IOCB_DSYNC;

I'd like to merge the patches 1-3, but have hard time matching the changelog to the change here. It's from one set of sync flags to another, mentioning pwritev2 but that's a syscall and the function itself does not use the sync flags at all. That's probably somewhere deep in the vfs calls but that's what I'd appreciate stated explicitly in the changelog as I was not able to find it out in a reasonable time.

You're right, there are a few layers here. How about this for the changelog:

Quoted 8 lines

The VFS indicates a synchronous write to ->write_iter() via iocb->ki_flags. The IOCB_{,D}SYNC flags may be set based on the file (see iocb_flags()) or the RWF_* flags passed to a syscall like pwritev2() (see kiocb_set_rw_flags()). However, in btrfs_file_write_iter(), we're checking if a write is synchronous based only on the file; we use this to decide when to bump the sync_writers counter and thus do CRCs synchronously. Make sure we do this for all synchronous writes as determined by the VFS.

That's great, thanks, no need to resend.

#17 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

In reply to Omar Sandoval
Quoted 84 lines · Omar Sandoval
Quoted 52 lines · Josef Bacik
Quoted 36 lines · Nikolay Borisov
Quoted 27 lines · Josef Bacik
Quoted 16 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

Quoted 9 lines

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

Quoted 5 lines

I've looked at this a few times, the locking and space reservation stuff look right. What about encrypted send/recieve? Are we going to want to use this to just blind copy encrypted data without having to decrypt/re-encrypt? Should this be taken into consideration for this interface? I'll think more about it, but I can't really see any better option than this. Thanks,

Quoted 6 lines

The main problem is we don't have encryption implemented. And one of the larger aspects of the encryption support is going to be how we are storing the encryption keys. E.g. should they be part of the send format? Or are we going to limit send/receive based on whether the source/dest have transferred encryption keys out of line?

Subvolume encryption will be coming soon, but I'm less worried about the mechanics of how that will be used and more worried about making this interface work for that eventual future. I assume we'll want to be able to just blind copy the encrypted data instead of decrypting into the send stream and then re-encrypting on the other side. Which means we'll have two uses for this interface, and I want to make sure we're happy with it before it gets merged. Thanks,

Quoted 1 line

Josef

Right, I think the only way to do this would be to blindly send encrypted data, and leave the key management to a higher layer.

Looking at the ioctl definition:

struct btrfs_ioctl_compressed_pwrite_args {
        __u64 offset;           /* in */
        __u32 orig_len;         /* in */
        __u32 compressed_len;   /* in */
        __u32 compress_type;    /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));

I think there are enough reserved fields in there for, e.g., encryption type, any key management-related things we might need to stuff in, etc. But the naming would be pretty bad if we extended it this way. Maybe compressed write -> raw write, orig_len -> num_bytes, compressed_len -> disk_num_bytes?

struct btrfs_ioctl_raw_pwrite_args {
        __u64 offset;           /* in */
        __u32 num_bytes;        /* in */
        __u32 disk_num_bytes;   /* in */
        __u32 compress_type;    /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));
Quoted 3 lines

Besides the naming, I don't think anything else would need to change for now. And if we decide that we don't want encrypted send/receive, then fine, this naming is still okay.

Oh, and at this again, compression and encryption are only u8 in the extent item, and we have an extra u16 for "other_encoding", so it'd probably be safe to make it:

struct btrfs_ioctl_raw_pwrite_args {
        __u64 offset;           /* in */
        __u32 num_bytes;        /* in */
        __u32 disk_num_bytes;   /* in */
        __u8 compression;       /* in */
        __u8 encryption;        /* in */
	__u16 other_encoding;   /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));
#18 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

Quoted 93 lines · Omar Sandoval
Quoted 84 lines · Omar Sandoval
Quoted 52 lines · Josef Bacik
Quoted 36 lines · Nikolay Borisov
Quoted 27 lines · Josef Bacik
Quoted 16 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

This adds an API for writing compressed data directly to the filesystem. The use case that I have in mind is send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The send part will be implemented in a separate series, as this ioctl can stand alone.

The interface is essentially pwrite(2) with some extra information:

- The input buffer contains the compressed data.
- Both the compressed and decompressed sizes of the data are given.
- The compression type (zlib, lzo, or zstd) is given.

Quoted 9 lines

A more detailed description of the interface, including restrictions and edge cases, is included in include/uapi/linux/btrfs.h.

The implementation is similar to direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the new ioctl from writeback.

Quoted 5 lines

I've looked at this a few times, the locking and space reservation stuff look right. What about encrypted send/recieve? Are we going to want to use this to just blind copy encrypted data without having to decrypt/re-encrypt? Should this be taken into consideration for this interface? I'll think more about it, but I can't really see any better option than this. Thanks,

Quoted 6 lines

The main problem is we don't have encryption implemented. And one of the larger aspects of the encryption support is going to be how we are storing the encryption keys. E.g. should they be part of the send format? Or are we going to limit send/receive based on whether the source/dest have transferred encryption keys out of line?

Subvolume encryption will be coming soon, but I'm less worried about the mechanics of how that will be used and more worried about making this interface work for that eventual future. I assume we'll want to be able to just blind copy the encrypted data instead of decrypting into the send stream and then re-encrypting on the other side. Which means we'll have two uses for this interface, and I want to make sure we're happy with it before it gets merged. Thanks,

Quoted 1 line

Josef

Right, I think the only way to do this would be to blindly send encrypted data, and leave the key management to a higher layer.

Looking at the ioctl definition:

struct btrfs_ioctl_compressed_pwrite_args {
        __u64 offset;           /* in */
        __u32 orig_len;         /* in */
        __u32 compressed_len;   /* in */
        __u32 compress_type;    /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));

I think there are enough reserved fields in there for, e.g., encryption type, any key management-related things we might need to stuff in, etc. But the naming would be pretty bad if we extended it this way. Maybe compressed write -> raw write, orig_len -> num_bytes, compressed_len -> disk_num_bytes?

struct btrfs_ioctl_raw_pwrite_args {
        __u64 offset;           /* in */
        __u32 num_bytes;        /* in */
        __u32 disk_num_bytes;   /* in */
        __u32 compress_type;    /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));
Quoted 3 lines

Besides the naming, I don't think anything else would need to change for now. And if we decide that we don't want encrypted send/receive, then fine, this naming is still okay.

Oh, and at this again, compression and encryption are only u8 in the extent item, and we have an extra u16 for "other_encoding", so it'd probably be safe to make it:

Quoted 10 lines
struct btrfs_ioctl_raw_pwrite_args {
        __u64 offset;           /* in */
        __u32 num_bytes;        /* in */
        __u32 disk_num_bytes;   /* in */
        __u8 compression;       /* in */
        __u8 encryption;        /* in */
	__u16 other_encoding;   /* in */
        __u32 reserved[9];
        void __user *buf;       /* in */
} __attribute__ ((__packed__));

I like this, then just adjust the patches to utilize the generic naming convention instead of "compression" and I think it's good to go. Thanks,

Josef

#22 lore raw
In reply to Omar Sandoval
Quoted 4 lines · Omar Sandoval

From: Omar Sandoval <[email protected]>

Hello,

Quoted 8 lines

This series adds a way to write compressed data directly to Btrfs. The intended use case is making send/receive on compressed file systems more efficient; however, the interface is general enough that it could be used in other scenarios. Patch 5 is the main change; see that for more details.

Patches 1-3 are small fixes/cleanups that I ran into while implementing this; they should go in regardless of the remainder of the series.

1-3 added to misc-next, thanks. I haven't looked at the rest yet.

#19 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

In reply to Omar Sandoval
Quoted 4 lines · Omar Sandoval

#define BTRFS_IOC_SEND_32 _IOW(BTRFS_IOCTL_MAGIC, 38, \

fs/btrfs/ioctl.c +2 -0
 			       struct btrfs_ioctl_send_args_32)
+
+struct btrfs_ioctl_compressed_pwrite_args_32 {

fs/btrfs/ioctl.c struct btrfs_ioctl_send_args_32 { · line 99

Quoted 10 lines
fs/btrfs/ioctl.c +10 -0
+	__u64 offset;		/* in */
+	__u32 compressed_len;	/* in */
+	__u32 orig_len;		/* in */
+	__u32 compress_type;	/* in */
+	__u32 reserved[9];
+	compat_uptr_t buf;	/* in */
+} __attribute__ ((__packed__));
+
+#define BTRFS_IOC_COMPRESSED_PWRITE_32 _IOW(BTRFS_IOCTL_MAGIC, 63, \
+				 struct btrfs_ioctl_compressed_pwrite_args_32)

Note that the _32 is a workaround for a mistake in the send ioctl definitions that slipped trhough. Any pointer in the structure changes the ioctl number on 32bit and 64bit.

But as the raw data ioctl is new there's point to copy the mistake. The alignment and width can be forced eg. like

include/uapi/linux/btrfs.h struct btrfs_ioctl_get_subvol_rootref_args { · line 883

Quoted 1 line
include/uapi/linux/btrfs.h +1 -0
+	void __user *buf;	/* in */
	union {
		void __user *buf;
		__u64 __buf_alignment;
	};

This allows to user buf as a buffer without casts to a intermediate type.

#20 lore raw

Re: [RFC PATCH 5/5] Btrfs: add ioctl for directly writing compressed data

Quoted 30 lines · David Sterba
Quoted 4 lines · Omar Sandoval

#define BTRFS_IOC_SEND_32 _IOW(BTRFS_IOCTL_MAGIC, 38, \

fs/btrfs/ioctl.c +2 -0
 			       struct btrfs_ioctl_send_args_32)
+
+struct btrfs_ioctl_compressed_pwrite_args_32 {

fs/btrfs/ioctl.c struct btrfs_ioctl_send_args_32 { · line 99

Quoted 10 lines
fs/btrfs/ioctl.c +10 -0
+	__u64 offset;		/* in */
+	__u32 compressed_len;	/* in */
+	__u32 orig_len;		/* in */
+	__u32 compress_type;	/* in */
+	__u32 reserved[9];
+	compat_uptr_t buf;	/* in */
+} __attribute__ ((__packed__));
+
+#define BTRFS_IOC_COMPRESSED_PWRITE_32 _IOW(BTRFS_IOCTL_MAGIC, 63, \
+				 struct btrfs_ioctl_compressed_pwrite_args_32)

Note that the _32 is a workaround for a mistake in the send ioctl definitions that slipped trhough. Any pointer in the structure changes the ioctl number on 32bit and 64bit.

But as the raw data ioctl is new there's point to copy the mistake. The alignment and width can be forced eg. like

include/uapi/linux/btrfs.h struct btrfs_ioctl_get_subvol_rootref_args { · line 883

Quoted 1 line
include/uapi/linux/btrfs.h +1 -0
+	void __user *buf;	/* in */
	union {
		void __user *buf;
		__u64 __buf_alignment;
	};
Quoted 2 lines

This allows to user buf as a buffer without casts to a intermediate type.

I don't think this works on big-endian architectures. Let's say a 32-bit application does:

struct btrfs_ioctl_compressed_pwrite_args_32 {
	.buf = 0x12345678,
};

The pointer will be in the first 4 bytes of the 8-byte union:

0 1 2 3 4 5 6 7
0x12 0x34 0x56 0x78 0x00 0x00 0x00 0x00

But, the 64-bit kernel will read buf as 0x1234567800000000. Let me know if I messed up my analysis, but I think we need the compat stuff.