fs: interface for directly reading/writing compressed data
Subsystems (14)
v3 cover fs: interface for directly reading/writing compressed data
From: Omar Sandoval <[email protected]>
Hello,
This series adds an API for reading compressed data on a filesystem without decompressing it as well as support for writing compressed data directly to the filesystem. As with the previous submissions, I've included a man page patch describing the API, and test cases and example programs are available [1].
This version reworks the VFS interface to be backward and forward compatible and support for writing inline and bookend extents to the Btrfs implementation.
Patches 1-3 add the VFS support. Patches 4-7 are Btrfs cleanups necessary for the encoded I/O support that can go in independently of this series. Patches 8-10 are Btrfs prep patches. Patch 11 adds Btrfs encoded read support and patch 12 adds Btrfs encoded write support.
A few TODOs remain:
- Once we've settled on the interface, I'll add RWF_ENCODED support to fsstress and friends and send up the xfstests patches in [1].
- btrfs_encoded_read() still doesn't implement repair.
Changes from v2 [2]:
- Rebase on v5.4-rc8
- Add patch 1 introducing copy_struct_from_iter() as suggested by Aleksa
- Rename O_ENCODED to O_ALLOW_ENCODED as suggested by Amir
- Add arch-specific definitions of O_ALLOW_ENCODED for alpha, parisc, and sparc
- Rework the VFS interface to be backward and forward compatible
- Document the VFS interface as requested by Dave
- Use __aligned_u64 for struct encoded_iov as noted by Aleksa
- Fix len/unencoded_len mixup in mm/filemap.c as noted by Nikolay
- Add support for writing inline and bookend extents to Btrfs
- Use ENOBUFS for "buffers not big enough for encoded extent" case and E2BIG for "encoded_iov has unsupported fields" case
Please share any comments on the API or implementation. Thanks!
1: https://github.com/osandov/xfstests/tree/rwf-encoded
2: https://lore.kernel.org/linux-btrfs/[email protected]/
Omar Sandoval (12):
iov_iter: add copy_struct_from_iter()
fs: add O_ALLOW_ENCODED open flag
fs: add RWF_ENCODED for reading/writing compressed data
btrfs: get rid of trivial __btrfs_lookup_bio_sums() wrappers
btrfs: don't advance offset for compressed bios in
btrfs_csum_one_bio()
btrfs: remove dead snapshot-aware defrag code
btrfs: make btrfs_ordered_extent naming consistent with
btrfs_file_extent_item
btrfs: add ram_bytes and offset to btrfs_ordered_extent
btrfs: support different disk extent size for delalloc
btrfs: optionally extend i_size in cow_file_range_inline()
btrfs: implement RWF_ENCODED reads
btrfs: implement RWF_ENCODED writes
Documentation/filesystems/encoded_io.rst | 79 + Documentation/filesystems/index.rst | 1 + arch/alpha/include/uapi/asm/fcntl.h | 1 + arch/parisc/include/uapi/asm/fcntl.h | 1 + arch/sparc/include/uapi/asm/fcntl.h | 1 + fs/btrfs/compression.c | 15 +- fs/btrfs/compression.h | 5 +- fs/btrfs/ctree.h | 13 +- fs/btrfs/delalloc-space.c | 38 +- fs/btrfs/delalloc-space.h | 4 +- fs/btrfs/file-item.c | 54 +- fs/btrfs/file.c | 61 +- fs/btrfs/inode.c | 2463 +++++++++++----------- fs/btrfs/ordered-data.c | 106 +- fs/btrfs/ordered-data.h | 28 +- fs/btrfs/relocation.c | 9 +- fs/fcntl.c | 10 +- fs/namei.c | 4 + include/linux/fcntl.h | 2 +- include/linux/fs.h | 16 + include/linux/uio.h | 2 + include/trace/events/btrfs.h | 6 +- include/uapi/asm-generic/fcntl.h | 4 + include/uapi/linux/fs.h | 33 +- lib/iov_iter.c | 82 + mm/filemap.c | 165 +- 26 files changed, 1807 insertions(+), 1396 deletions(-) create mode 100644 Documentation/filesystems/encoded_io.rst
Signature · 1 line
2.24.0
PATCH v2 1/1 Document encoded I/O
From: Omar Sandoval <[email protected]>
This adds a new page, encoded_io(7), providing an overview of encoded I/O and updates fcntl(2), open(2), and preadv2(2)/pwritev2(2) to reference it.
Signed-off-by Omar Sandoval <[email protected]>
--- man2/fcntl.2 | 10 +- man2/open.2 | 13 ++ man2/readv.2 | 64 ++++++++++ man7/encoded_io.7 | 308 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 394 insertions(+), 1 deletion(-) create mode 100644 man7/encoded_io.7
man2/fcntl.2 +9 -1
@@ -222,8 +222,9 @@ On Linux, this command can change only the .BR O_ASYNC , .BR O_DIRECT , .BR O_NOATIME , +.BR O_NONBLOCK , and -.B O_NONBLOCK +.B O_ALLOW_ENCODED flags. It is not possible to change the .BR O_DSYNC
@@ -1803,6 +1804,13 @@ Attempted to clear the flag on a file that has the append-only attribute set. .TP .B EPERM +Attempted to set the +.B O_ALLOW_ENCODED +flag and the calling process did not have the +.B CAP_SYS_ADMIN +capability. +.TP +.B EPERM .I cmd was .BR F_ADD_SEALS ,
man2/open.2 +13 -0
@@ -421,6 +421,14 @@ was followed by a call to .BR fdatasync (2)). .IR "See NOTES below" . .TP +.B O_ALLOW_ENCODED +Open the file with encoded I/O permissions; +see +.BR encoded_io (7). +The caller must have the +.B CAP_SYS_ADMIN +capability. +.TP .B O_EXCL Ensure that this call creates the file: if this flag is specified in conjunction with
@@ -1168,6 +1176,11 @@ did not match the owner of the file and the caller was not privileged. The operation was prevented by a file seal; see .BR fcntl (2). .TP +.B EPERM +The +.B O_ALLOW_ENCODED +flag was specified, but the caller was not privileged. +.TP .B EROFS .I pathname refers to a file on a read-only filesystem and write access was
man2/readv.2 +64 -0
82 lines not on this page · raw message
man7/encoded_io.7 new file +308 -0
308 lines not on this page · raw message
Signature · 1 line
2.24.0
RFC PATCH v3 1/12 iov_iter: add copy_struct_from_iter()
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
This is essentially copy_struct_from_user() but for an iov_iter.
Suggested-by Aleksa Sarai <[email protected]>
Signed-off-by Omar Sandoval <[email protected]>
--- include/linux/uio.h | 2 ++ lib/iov_iter.c | 82 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+)
include/linux/uio.h +2 -0
@@ -122,6 +122,8 @@ size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i);size_t copy_page_from_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i); +int copy_struct_from_iter (void *dst,size_t ksize,struct iov_iter *i, +size_t usize);size_t _copy_to_iter (const void *addr,size_t bytes,struct iov_iter *i);size_t _copy_from_iter (void *addr,size_t bytes,struct iov_iter *i);
lib/iov_iter.c +82 -0
88 lines not on this page · raw message
Signature · 1 line
2.24.0
RFC PATCH v3 2/12 fs: add O_ALLOW_ENCODED open flag
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
The upcoming RWF_ENCODED operation introduces some security concerns:
1. Compressed writes will pass arbitrary data to decompression algorithms in the kernel.
2. Compressed reads can leak truncated/hole punched data.
Therefore, we need to require privilege for RWF_ENCODED. It's not possible to do the permissions checks at the time of the read or write because, e.g., io_uring submits IO from a worker thread. So, add an open flag which requires CAP_SYS_ADMIN. It can also be set and cleared with fcntl(). The flag is not cleared in any way on fork or exec; it should probably be used with O_CLOEXEC in most cases.
Note that the usual issue that unknown open flags are ignored doesn't really matter for O_ALLOW_ENCODED; if the kernel doesn't support O_ALLOW_ENCODED, then it doesn't support RWF_ENCODED, either.
Signed-off-by Omar Sandoval <[email protected]>
--- arch/alpha/include/uapi/asm/fcntl.h | 1 + arch/parisc/include/uapi/asm/fcntl.h | 1 + arch/sparc/include/uapi/asm/fcntl.h | 1 + fs/fcntl.c | 10 ++++++++-- fs/namei.c | 4 ++++ include/linux/fcntl.h | 2 +- include/uapi/asm-generic/fcntl.h | 4 ++++ 7 files changed, 20 insertions(+), 3 deletions(-)
arch/alpha/include/uapi/asm/fcntl.h +1 -0
@@ -34,6 +34,7 @@#define O_PATH 040000000#define __O_TMPFILE 0100000000 +#define O_ALLOW_ENCODED 0200000000#define F_GETLK 7#define F_SETLK 8
arch/parisc/include/uapi/asm/fcntl.h +1 -0
@@ -22,6 +22,7 @@#define O_PATH 020000000#define __O_TMPFILE 040000000 +#define O_ALLOW_ENCODED 100000000#define F_GETLK64 8#define F_SETLK64 9
arch/sparc/include/uapi/asm/fcntl.h +1 -0
@@ -37,6 +37,7 @@#define O_PATH 0x1000000#define __O_TMPFILE 0x2000000 +#define O_ALLOW_ENCODED 0x8000000#define F_GETOWN 5/* for sockets. */ #define F_SETOWN 6/* for sockets. */
fs/fcntl.c +8 -2
@@ -30,7 +30,8 @@#include <asm/siginfo.h> #include <linux/uaccess.h> -#define SETFL_MASK (O_APPEND | O_NONBLOCK | O_NDELAY | O_DIRECT | O_NOATIME) +#define SETFL_MASK (O_APPEND | O_NONBLOCK | O_NDELAY | O_DIRECT | O_NOATIME | \ + O_ALLOW_ENCODED)static int setfl (int fd,struct file * filp,unsigned long arg) {
@@ -49,6 +50,11 @@ static int setfl(int fd, struct file * filp, unsigned long arg)if (!inode_owner_or_capable (inode))return -EPERM ; +/* O_ALLOW_ENCODED can only be set by superuser */ +if ((arg &O_ALLOW_ENCODED ) && !(filp->f_flags &O_ALLOW_ENCODED ) && + !capable (CAP_SYS_ADMIN )) +return -EPERM ; +/* required for strict SunOS emulation */ if (O_NONBLOCK !=O_NDELAY )if (arg &O_NDELAY )
@@ -1031,7 +1037,7 @@ static int __init fcntl_init(void)* Exceptions: O_NONBLOCK is a two bit define on parisc; O_NDELAY * is defined as O_NONBLOCK on some platforms and not on others. */ -BUILD_BUG_ON (21 -1 /* for O_RDONLY being 0 */ != +BUILD_BUG_ON (22 -1 /* for O_RDONLY being 0 */ !=HWEIGHT32 ( (VALID_OPEN_FLAGS & ~(O_NONBLOCK |O_NDELAY )) | __FMODE_EXEC | __FMODE_NONOTIFY));
fs/namei.c +4 -0
@@ -2978,6 +2978,10 @@ static int may_open(const struct path *path, int acc_mode, int flag)if (flag &O_NOATIME && !inode_owner_or_capable (inode))return -EPERM ; +/* O_ALLOW_ENCODED can only be set by superuser */ +if ((flag &O_ALLOW_ENCODED ) && !capable (CAP_SYS_ADMIN )) +return -EPERM ; +return 0 ; }
include/linux/fcntl.h +1 -1
@@ -9,7 +9,7 @@ (O_RDONLY |O_WRONLY |O_RDWR |O_CREAT |O_EXCL |O_NOCTTY |O_TRUNC | \O_APPEND |O_NDELAY |O_NONBLOCK |O_NDELAY | __O_SYNC |O_DSYNC | \FASYNC |O_DIRECT |O_LARGEFILE |O_DIRECTORY |O_NOFOLLOW | \ - O_NOATIME | O_CLOEXEC | O_PATH | __O_TMPFILE) + O_NOATIME | O_CLOEXEC | O_PATH | __O_TMPFILE | O_ALLOW_ENCODED)#ifndef force_o_largefile#define force_o_largefile () (!IS_ENABLED(CONFIG_ARCH_32BIT_OFF_T))
include/uapi/asm-generic/fcntl.h +4 -0
@@ -89,6 +89,10 @@#define __O_TMPFILE 020000000#endif +#ifndef O_ALLOW_ENCODED +#define O_ALLOW_ENCODED 040000000 +#endif +/* a horrid kludge trying to make sure that this will fail on old kernels */ #define O_TMPFILE (__O_TMPFILE | O_DIRECTORY)#define O_TMPFILE_MASK (__O_TMPFILE | O_DIRECTORY | O_CREAT)
Signature · 1 line
2.24.0
RFC PATCH v3 3/12 fs: add RWF_ENCODED for reading/writing compressed data
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Btrfs supports transparent compression: data written by the user can be compressed when written to disk and decompressed when read back. However, we'd like to add an interface to write pre-compressed data directly to the filesystem, and the matching interface to read compressed data without decompressing it. This adds support for so-called "encoded I/O" via preadv2() and pwritev2().
A new RWF_ENCODED flags indicates that a read or write is "encoded". If this flag is set, iov[0].iov_base points to a struct encoded_iov which is used for metadata: namely, the compression algorithm, unencoded (i.e., decompressed) length, and what subrange of the unencoded data should be used (needed for truncated or hole-punched extents and when reading in the middle of an extent). For reads, the filesystem returns this information; for writes, the caller provides it to the filesystem. iov[0].iov_len must be set to sizeof(struct encoded_iov), which can be used to extend the interface in the future a la copy_struct_from_user(). The remaining iovecs contain the encoded extent.
This adds the VFS helpers for supporting encoded I/O and documentation for filesystem support.
Signed-off-by Omar Sandoval <[email protected]>
--- Documentation/filesystems/encoded_io.rst | 79 +++++++++++ Documentation/filesystems/index.rst | 1 + include/linux/fs.h | 16 +++ include/uapi/linux/fs.h | 33 ++++- mm/filemap.c | 165 +++++++++++++++++++++-- 5 files changed, 280 insertions(+), 14 deletions(-) create mode 100644 Documentation/filesystems/encoded_io.rst
Documentation/filesystems/encoded_io.rst new file +79 -0
@@ -0,0 +1,79 @@ +=========== +Encoded I/O +=========== + +Encoded I/O is a mechanism for reading and writing encoded (e.g., compressed +and/or encrypted) data directly from/to the filesystem. The userspace interface +is thoroughly described in the :manpage:`encoded_io(7)` man page; this document +describes the requirements for filesystem support. + +First of all, a filesystem supporting encoded I/O must indicate this by setting +the ``FMODE_ENCODED_IO`` flag in its ``file_open`` file operation:: + + static int foo_file_open(struct inode *inode, struct file *filp) + { + ... + filep->f_mode |= FMODE_ENCODED_IO; + ... + } + +Encoded I/O goes through ``read_iter`` and ``write_iter``, designated by the +``IOCB_ENCODED`` flag in ``kiocb->ki_flags``. + +Reads +===== + +Encoded ``read_iter`` should: + +1. Call ``generic_encoded_read_checks()`` to validate the file and buffers + provided by userspace. +2. Initialize the ``encoded_iov`` appropriately. +3. Copy it to the user with ``copy_encoded_iov_to_iter()``. +4. Copy the encoded data to the user. +5. Advance ``kiocb->ki_pos`` by ``encoded_iov->len``. +6. Return the size of the encoded data read, not including the ``encoded_iov``. + +There are a few details to be aware of: + +* Encoded ``read_iter`` should support reading unencoded data if the extent is + not encoded. +* If the buffers provided by the user are not large enough to contain an entire + encoded extent, then ``read_iter`` should return ``-ENOBUFS``. This is to + avoid confusing userspace with truncated data that cannot be properly + decoded. +* Reads in the middle of an encoded extent can be returned by setting + ``encoded_iov->unencoded_offset`` to non-zero. +* Truncated unencoded data (e.g., because the file does not end on a block + boundary) may be returned by setting ``encoded_iov->len`` to a value smaller + value than ``encoded_iov->unencoded_len - encoded_iov->unencoded_offset``. +* If ``encoded_iov->len`` is greater than ``encoded_iov->unencoded_len - + encoded_iov->unencoded_offset``, then the user will treat the remainder as + zero-filled. Therefore, holes may be returned by setting ``encoded_iov->len`` + to the size of the hole, setting ``encoded_iov->unencoded_len`` to zero, and + copying no data to the user. + +Writes +====== + +Encoded ``write_iter`` should (in addition to the usual accounting/checks done +by ``write_iter``): + +1. Call ``copy_encoded_iov_from_iter()`` to get and validate the + ``encoded_iov``. +2. Call ``generic_encoded_write_checks()`` instead of + ``generic_write_checks()``. +3. Check that the provided encoding in ``encoded_iov`` is supported. +4. Advance ``kiocb->ki_pos`` by ``encoded_iov->len``. +5. Return the size of the encoded data written. + +Again, there are a few details: + +* Encoded ``write_iter`` doesn't need to support writing unencoded data. +* ``write_iter`` should either write all of the encoded data or none of it; it + must not do partial writes. +* ``write_iter`` doesn't need to validate the encoded data; a subsequent read + may return, e.g., ``-EIO`` if the data is not valid. +* The user may lie about the unencoded size of the data; a subsequent read + should truncate or zero-extend the unencoded data rather than returning an + error. +* Be careful of page cache coherency.
Documentation/filesystems/index.rst +1 -0
@@ -37,6 +37,7 @@ filesystem implementations. journalling fscrypt fsverity + encoded_io Filesystems ===========
include/linux/fs.h +16 -0
@@ -175,6 +175,9 @@ typedef int (dio_iodone_t)(struct kiocb *iocb, loff_t offset,/* File does not contribute to nr_files count */ #define FMODE_NOACCOUNT ((__force fmode_t)0x20000000) +/* File supports encoded IO */ +#define FMODE_ENCODED_IO ((__force fmode_t)0x40000000) +/* * Flag for rw_copy_check_uvector and compat_rw_copy_check_uvector * that indicates that they should check the contents of the iovec are
@@ -314,6 +317,7 @@ enum rw_hint {#define IOCB_SYNC (1 << 5)#define IOCB_WRITE (1 << 6)#define IOCB_NOWAIT (1 << 7) +#define IOCB_ENCODED (1 << 8)struct kiocb {struct file *ki_filp;
@@ -3088,6 +3092,13 @@ extern int sb_min_blocksize(struct super_block *, int);extern int generic_file_mmap (struct file *,struct vm_area_struct *);extern int generic_file_readonly_mmap (struct file *,struct vm_area_struct *);extern ssize_t generic_write_checks (struct kiocb *,struct iov_iter *); +struct encoded_iov ; +extern int generic_encoded_write_checks (struct kiocb *, +const struct encoded_iov *); +extern int copy_encoded_iov_from_iter (struct encoded_iov *,struct iov_iter *); +extern ssize_t generic_encoded_read_checks (struct kiocb *,struct iov_iter *); +extern int copy_encoded_iov_to_iter (const struct encoded_iov *, +struct iov_iter *);extern int generic_remap_checks (struct file *file_in,loff_t pos_in,struct file *file_out,loff_t pos_out,loff_t *count,unsigned int remap_flags);
@@ -3403,6 +3414,11 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags)return -EOPNOTSUPP ; ki->ki_flags |=IOCB_NOWAIT ; } +if (flags &RWF_ENCODED ) { +if (!(ki->ki_filp->f_mode &FMODE_ENCODED_IO )) +return -EOPNOTSUPP ; + ki->ki_flags |=IOCB_ENCODED ; + }if (flags &RWF_HIPRI ) ki->ki_flags |=IOCB_HIPRI ;if (flags &RWF_DSYNC )
include/uapi/linux/fs.h +32 -1
@@ -284,6 +284,34 @@ struct fsxattr {typedef int __bitwise __kernel_rwf_t ; +enum { +ENCODED_IOV_COMPRESSION_NONE , +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_ZLIB, +#define ENCODED_IOV_COMPRESSION_ZLIB ENCODED_IOV_COMPRESSION_ZLIB + ENCODED_IOV_COMPRESSION_LZO, +#define ENCODED_IOV_COMPRESSION_LZO ENCODED_IOV_COMPRESSION_LZO + ENCODED_IOV_COMPRESSION_ZSTD, +#define ENCODED_IOV_COMPRESSION_ZSTD ENCODED_IOV_COMPRESSION_ZSTD + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_ZSTD, +}; + +enum { +ENCODED_IOV_ENCRYPTION_NONE , +#define ENCODED_IOV_ENCRYPTION_NONE ENCODED_IOV_ENCRYPTION_NONE + ENCODED_IOV_ENCRYPTION_TYPES = ENCODED_IOV_ENCRYPTION_NONE, +}; + +struct encoded_iov { +__aligned_u64 len; +__aligned_u64 unencoded_len; +__aligned_u64 unencoded_offset; +__u32 compression; +__u32 encryption; +}; + +#define ENCODED_IOV_SIZE_VER0 32 +/* high priority request, poll if possible */ #define RWF_HIPRI ((__force __kernel_rwf_t)0x00000001)
@@ -299,8 +327,11 @@ typedef int __bitwise __kernel_rwf_t;/* per-IO O_APPEND */ #define RWF_APPEND ((__force __kernel_rwf_t)0x00000010) +/* encoded (e.g., compressed and/or encrypted) IO */ +#define RWF_ENCODED ((__force __kernel_rwf_t)0x00000020) +/* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND) + RWF_APPEND | RWF_ENCODED)#endif /* _UAPI_LINUX_FS_H */
mm/filemap.c +152 -13
190 lines not on this page · raw message
Signature · 1 line
2.24.0
Re: [RFC PATCH v3 03/12] fs: add RWF_ENCODED for reading/writing compressed data
Quoted 1 line · Omar Sandoval
From: Omar Sandoval <[email protected]>
<snip>
include/uapi/linux/fs.h struct fsxattr { · line 296
Quoted 11 lines
include/uapi/linux/fs.h +10
-0+enum { +ENCODED_IOV_COMPRESSION_NONE , +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_ZLIB, +#define ENCODED_IOV_COMPRESSION_ZLIB ENCODED_IOV_COMPRESSION_ZLIB + ENCODED_IOV_COMPRESSION_LZO, +#define ENCODED_IOV_COMPRESSION_LZO ENCODED_IOV_COMPRESSION_LZO + ENCODED_IOV_COMPRESSION_ZSTD, +#define ENCODED_IOV_COMPRESSION_ZSTD ENCODED_IOV_COMPRESSION_ZSTD + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_ZSTD,
This looks very dodgy, what am I missing?
<snip>
Re: [RFC PATCH v3 03/12] fs: add RWF_ENCODED for reading/writing compressed data
Quoted 8 lines · Nikolay Borisov
Quoted 1 line · Omar Sandoval
From: Omar Sandoval <[email protected]>
<snip>
Quoted 1 line
Quoted 12 lines
include/uapi/linux/fs.h struct fsxattr { · line 296
Quoted 10 lines
include/uapi/linux/fs.h +10
-0+enum { +ENCODED_IOV_COMPRESSION_NONE , +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_ZLIB, +#define ENCODED_IOV_COMPRESSION_ZLIB ENCODED_IOV_COMPRESSION_ZLIB + ENCODED_IOV_COMPRESSION_LZO, +#define ENCODED_IOV_COMPRESSION_LZO ENCODED_IOV_COMPRESSION_LZO + ENCODED_IOV_COMPRESSION_ZSTD, +#define ENCODED_IOV_COMPRESSION_ZSTD ENCODED_IOV_COMPRESSION_ZSTD + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_ZSTD,This looks very dodgy, what am I missing?
This is a somewhat common trick so that enum values can be checked for with ifdef/ifndef. See include/uapi/linux.in.h, for example.
Re: [RFC PATCH v3 03/12] fs: add RWF_ENCODED for reading/writing compressed data
Quoted 22 lines · Omar Sandoval
Quoted 8 lines · Nikolay Borisov
Quoted 1 line · Omar Sandoval
From: Omar Sandoval <[email protected]>
<snip>
Quoted 1 line
Quoted 12 lines
include/uapi/linux/fs.h struct fsxattr { · line 296
Quoted 10 lines
include/uapi/linux/fs.h +10
-0+enum { +ENCODED_IOV_COMPRESSION_NONE , +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_ZLIB, +#define ENCODED_IOV_COMPRESSION_ZLIB ENCODED_IOV_COMPRESSION_ZLIB + ENCODED_IOV_COMPRESSION_LZO, +#define ENCODED_IOV_COMPRESSION_LZO ENCODED_IOV_COMPRESSION_LZO + ENCODED_IOV_COMPRESSION_ZSTD, +#define ENCODED_IOV_COMPRESSION_ZSTD ENCODED_IOV_COMPRESSION_ZSTD + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_ZSTD,This looks very dodgy, what am I missing?
Quoted 2 lines
This is a somewhat common trick so that enum values can be checked for with ifdef/ifndef. See
include/uapi/linux.in.h, for example.
I cannot seem to have this file on my system (or any .in.h file for that matter in the kernel source dir).
Quoted 1 line
Re: [RFC PATCH v3 03/12] fs: add RWF_ENCODED for reading/writing compressed data
Quoted 25 lines · Nikolay Borisov
Quoted 22 lines · Omar Sandoval
Quoted 20 lines · Nikolay Borisov
Quoted 1 line · Omar Sandoval
From: Omar Sandoval <[email protected]>
<snip>
include/uapi/linux/fs.h struct fsxattr { · line 296
Quoted 11 lines
include/uapi/linux/fs.h +10
-0+enum { +ENCODED_IOV_COMPRESSION_NONE , +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_ZLIB, +#define ENCODED_IOV_COMPRESSION_ZLIB ENCODED_IOV_COMPRESSION_ZLIB + ENCODED_IOV_COMPRESSION_LZO, +#define ENCODED_IOV_COMPRESSION_LZO ENCODED_IOV_COMPRESSION_LZO + ENCODED_IOV_COMPRESSION_ZSTD, +#define ENCODED_IOV_COMPRESSION_ZSTD ENCODED_IOV_COMPRESSION_ZSTD + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_ZSTD,This looks very dodgy, what am I missing?
Quoted 5 lines
Quoted 2 lines
This is a somewhat common trick so that enum values can be checked for with ifdef/ifndef. See
include/uapi/linux.in.h, for example.I cannot seem to have this file on my system (or any
.in.hfile for that matter in the kernel source dir).
Whoops, that should be include/uapi/linux/in.h
RFC PATCH v3 4/12 btrfs: get rid of trivial __btrfs_lookup_bio_sums() wrappers
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Currently, we have two wrappers for __btrfs_lookup_bio_sums():btrfs_lookup_bio_sums_dio(), which is used for direct I/O, and btrfs_lookup_bio_sums(), which is used everywhere else. The only difference is that the _dio variant looks up csums starting at the given offset instead of using the page index, which isn't actually direct I/O-specific. Let's clean up the signature and return value of __btrfs_lookup_bio_sums(), rename it to btrfs_lookup_bio_sums(), and get rid of the trivial helpers.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/compression.c | 4 ++-- fs/btrfs/ctree.h | 4 +--- fs/btrfs/file-item.c | 35 +++++++++++++++++------------------ fs/btrfs/inode.c | 6 +++--- 4 files changed, 23 insertions(+), 26 deletions(-)
fs/btrfs/compression.c +2 -2
@@ -660,7 +660,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,if (!(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM )) { ret =btrfs_lookup_bio_sums (inode, comp_bio, - sums); + false,0 , sums);BUG_ON (ret);/* -ENOMEM */ }
@@ -689,7 +689,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,BUG_ON (ret);/* -ENOMEM */ if (!(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM )) { - ret =btrfs_lookup_bio_sums (inode, comp_bio, sums); + ret =btrfs_lookup_bio_sums (inode, comp_bio, false,0 , sums);BUG_ON (ret);/* -ENOMEM */ }
fs/btrfs/ctree.h +1 -3
@@ -2787,9 +2787,7 @@ struct btrfs_dio_private;int btrfs_del_csums (struct btrfs_trans_handle *trans,struct btrfs_fs_info *fs_info,u64 bytenr,u64 len);blk_status_t btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u8 *dst); -blk_status_t btrfs_lookup_bio_sums_dio (struct inode *inode,struct bio *bio, -u64 logical_offset); +bool at_offset,u64 offset,u8 *dst); int btrfs_insert_file_extent(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 objectid, u64 pos,
fs/btrfs/file-item.c +17 -18
@@ -148,8 +148,21 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,return ret; } -static blk_status_t __btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u64 logical_offset,u8 *dst,int dio) +/** +* btrfs_lookup_bio_sums - Look up checksums for a bio. +* @inode: inode that the bio is for. +* @bio: bio embedded in btrfs_io_bio. +* @at_offset: If true, look up checksums for the extent at @c offset. +* If false, use the page offsets from the bio. +* @offset: If @at_offset is true, offset in file to look up checksums for. +* Ignored otherwise. +* @dst: Buffer of size btrfs_super_csum_size() used to return checksum. If +* NULL, the checksum is returned in btrfs_io_bio(bio)->csum instead. +* +* Return: BLK_STS_RESOURCE if allocating memory fails, BLK_STS_OK otherwise. +*/ +blk_status_t btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, +bool at_offset,u64 offset,u8 *dst) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct bio_vec bvec;
@@ -159,7 +172,6 @@ static blk_status_t __btrfs_lookup_bio_sums(struct inode *inode, struct bio *biostruct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct btrfs_path *path;u8 *csum; -u64 offset =0 ;u64 item_start_offset =0 ;u64 item_last_offset =0 ;u64 disk_bytenr;
@@ -205,15 +217,13 @@ static blk_status_t __btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio } disk_bytenr = (u64 )bio->bi_iter.bi_sector <<9 ; -if (dio) - offset = logical_offset;bio_for_each_segment (bvec, bio, iter) { page_bytes_left = bvec.bv_len;if (count) goto next; -if (!dio) +if (!at_offset) offset =page_offset (bvec.bv_page) + bvec.bv_offset; count =btrfs_find_ordered_sum (inode, offset, disk_bytenr, csum, nblocks);
@@ -285,18 +295,7 @@ static blk_status_t __btrfs_lookup_bio_sums(struct inode *inode, struct bio *bioWARN_ON_ONCE (count);btrfs_free_path (path); -return 0 ; -} - -blk_status_t btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u8 *dst) -{ -return __btrfs_lookup_bio_sums (inode, bio,0 , dst,0 ); -} - -blk_status_t btrfs_lookup_bio_sums_dio (struct inode *inode,struct bio *bio,u64 offset) -{ -return __btrfs_lookup_bio_sums (inode, bio, offset,NULL ,1 ); +return BLK_STS_OK ; } int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end,
fs/btrfs/inode.c +3 -3
@@ -2090,7 +2090,7 @@ static blk_status_t btrfs_submit_bio_hook(struct inode *inode, struct bio *bio, bio_flags); goto out; }else if (!skip_sum) { - ret =btrfs_lookup_bio_sums (inode, bio,NULL ); + ret =btrfs_lookup_bio_sums (inode, bio, false,0 ,NULL );if (ret) goto out; }
@@ -8332,8 +8332,8 @@ static inline blk_status_t btrfs_lookup_and_bind_dio_csum(struct inode *inode,* contention. */ if (dip->logical_offset == file_offset) { - ret =btrfs_lookup_bio_sums_dio (inode, dip->orig_bio, - file_offset); + ret =btrfs_lookup_bio_sums (inode, dip->orig_bio, true, + file_offset,NULL );if (ret)return ret; }
Signature · 1 line
2.24.0
Re: [RFC PATCH v3 04/12] btrfs: get rid of trivial __btrfs_lookup_bio_sums() wrappers
Quoted 11 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Currently, we have two wrappers for
__btrfs_lookup_bio_sums():btrfs_lookup_bio_sums_dio(), which is used for direct I/O, andbtrfs_lookup_bio_sums(), which is used everywhere else. The only difference is that the_diovariant looks up csums starting at the given offset instead of using the page index, which isn't actually direct I/O-specific. Let's clean up the signature and return value of__btrfs_lookup_bio_sums(), rename it tobtrfs_lookup_bio_sums(), and get rid of the trivial helpers.
Quoted 1 line
Signed-off-by Omar Sandoval <[email protected]>
Overall looks good but 2 nits, see below.
In any case:
Reviewed-by Nikolay Borisov <[email protected]>
Quoted 49 lines
--- fs/btrfs/compression.c | 4 ++-- fs/btrfs/ctree.h | 4 +--- fs/btrfs/file-item.c | 35 +++++++++++++++++------------------ fs/btrfs/inode.c | 6 +++--- 4 files changed, 23 insertions(+), 26 deletions(-)fs/btrfs/compression.c +2
-2@@ -660,7 +660,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,if (!(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM )) { ret =btrfs_lookup_bio_sums (inode, comp_bio, - sums); + false,0 , sums);BUG_ON (ret);/* -ENOMEM */ }@@ -689,7 +689,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,BUG_ON (ret);/* -ENOMEM */ if (!(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM )) { - ret =btrfs_lookup_bio_sums (inode, comp_bio, sums); + ret =btrfs_lookup_bio_sums (inode, comp_bio, false,0 , sums);BUG_ON (ret);/* -ENOMEM */ }fs/btrfs/ctree.h +1
-3@@ -2787,9 +2787,7 @@ struct btrfs_dio_private;int btrfs_del_csums (struct btrfs_trans_handle *trans,struct btrfs_fs_info *fs_info,u64 bytenr,u64 len);blk_status_t btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u8 *dst); -blk_status_t btrfs_lookup_bio_sums_dio (struct inode *inode,struct bio *bio, -u64 logical_offset); +bool at_offset,u64 offset,u8 *dst); int btrfs_insert_file_extent(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 objectid, u64 pos,fs/btrfs/file-item.c +0
-0
fs/btrfs/file-item.c int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans, · line 155
Quoted 10 lines
fs/btrfs/file-item.c +5
-2@@ -148,8 +148,21 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,return ret; } -static blk_status_t __btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u64 logical_offset,u8 *dst,int dio) +/** +* btrfs_lookup_bio_sums - Look up checksums for a bio. +* @inode: inode that the bio is for. +* @bio: bio embedded in btrfs_io_bio. +* @at_offset: If true, look up checksums for the extent at @c offset.
nit: that @c is an editing artifact? On the other hand rather than having an explicit bool signifying whether we want a specific offset can't we simply check if offset is != 0 ?
<snip>
Re: [RFC PATCH v3 04/12] btrfs: get rid of trivial __btrfs_lookup_bio_sums() wrappers
Quoted 14 lines · Nikolay Borisov
Quoted 11 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Currently, we have two wrappers for
__btrfs_lookup_bio_sums():btrfs_lookup_bio_sums_dio(), which is used for direct I/O, andbtrfs_lookup_bio_sums(), which is used everywhere else. The only difference is that the_diovariant looks up csums starting at the given offset instead of using the page index, which isn't actually direct I/O-specific. Let's clean up the signature and return value of__btrfs_lookup_bio_sums(), rename it tobtrfs_lookup_bio_sums(), and get rid of the trivial helpers.
Quoted 7 lines
Quoted 1 line
Signed-off-by Omar Sandoval <[email protected]>Overall looks good but 2 nits, see below.
In any case:
Reviewed-by Nikolay Borisov <[email protected]>
Thanks!
Quoted 52 lines
Quoted 52 lines
--- fs/btrfs/compression.c | 4 ++-- fs/btrfs/ctree.h | 4 +--- fs/btrfs/file-item.c | 35 +++++++++++++++++------------------ fs/btrfs/inode.c | 6 +++--- 4 files changed, 23 insertions(+), 26 deletions(-)fs/btrfs/compression.c +2
-2@@ -660,7 +660,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,if (!(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM )) { ret =btrfs_lookup_bio_sums (inode, comp_bio, - sums); + false,0 , sums);BUG_ON (ret);/* -ENOMEM */ }@@ -689,7 +689,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,BUG_ON (ret);/* -ENOMEM */ if (!(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM )) { - ret =btrfs_lookup_bio_sums (inode, comp_bio, sums); + ret =btrfs_lookup_bio_sums (inode, comp_bio, false,0 , sums);BUG_ON (ret);/* -ENOMEM */ }fs/btrfs/ctree.h +1
-3@@ -2787,9 +2787,7 @@ struct btrfs_dio_private;int btrfs_del_csums (struct btrfs_trans_handle *trans,struct btrfs_fs_info *fs_info,u64 bytenr,u64 len);blk_status_t btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u8 *dst); -blk_status_t btrfs_lookup_bio_sums_dio (struct inode *inode,struct bio *bio, -u64 logical_offset); +bool at_offset,u64 offset,u8 *dst); int btrfs_insert_file_extent(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 objectid, u64 pos,fs/btrfs/file-item.c +0
-0@@ -148,8 +148,21 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,return ret; }
Quoted 9 lines
fs/btrfs/file-item.c int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans, · line 155
Quoted 7 lines
fs/btrfs/file-item.c +5
-2-static blk_status_t __btrfs_lookup_bio_sums (struct inode *inode,struct bio *bio, -u64 logical_offset,u8 *dst,int dio) +/** +* btrfs_lookup_bio_sums - Look up checksums for a bio. +* @inode: inode that the bio is for. +* @bio: bio embedded in btrfs_io_bio. +* @at_offset: If true, look up checksums for the extent at @c offset. nit: that @c is an editing artifact?
Oops, I mixed up kernel-doc with Doxygen. Fixed, thanks.
Quoted 3 lines
On the other hand rather than
having an explicit bool signifying whether we want a specific offset can't we simply check if offset is != 0 ?
Zero is a perfectly valid offset to have an extent at, but we could do (u64)-1 instead. I'm not sure what's cleaner.
RFC PATCH v3 5/12 btrfs: don't advance offset for compressed bios in btrfs_csum_one_bio()
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
btrfs_csum_one_bio() loops over each sector in the bio while keeping a cursor of its current logical position in the file in order to look up the ordered extent to add the checksums to. However, this doesn't make much sense for compressed extents, as a sector on disk does not correspond to a sector of decompressed file data. It happens to work because 1) the compressed bio always covers one ordered extent and 2) the size of the bio is always less than the size of the ordered extent. However, the second point will not always be true for encoded writes.
Let's add a boolean parameter to btrfs_csum_one_bio() to indicate that it can assume that the bio only covers one ordered extent. Since we're already changing the signature, let's make contig bool instead of int, too.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/compression.c | 5 +++-- fs/btrfs/ctree.h | 2 +- fs/btrfs/file-item.c | 19 +++++++++++-------- fs/btrfs/inode.c | 8 ++++---- 4 files changed, 19 insertions(+), 15 deletions(-)
fs/btrfs/compression.c +3 -2
@@ -374,7 +374,8 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,BUG_ON (ret);/* -ENOMEM */ if (!skip_sum) { - ret =btrfs_csum_one_bio (inode, bio, start,1 ); + ret =btrfs_csum_one_bio (inode, bio, start, + true, true);BUG_ON (ret);/* -ENOMEM */ }
@@ -405,7 +406,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,BUG_ON (ret);/* -ENOMEM */ if (!skip_sum) { - ret =btrfs_csum_one_bio (inode, bio, start,1 ); + ret =btrfs_csum_one_bio (inode, bio, start, true, true);BUG_ON (ret);/* -ENOMEM */ }
fs/btrfs/ctree.h +1 -1
@@ -2802,7 +2802,7 @@ int btrfs_csum_file_blocks(struct btrfs_trans_handle *trans,struct btrfs_root *root, struct btrfs_ordered_sum *sums);blk_status_t btrfs_csum_one_bio (struct inode *inode,struct bio *bio, -u64 file_start,int contig); +u64 file_start,bool contig,bool one_ordered);int btrfs_lookup_csums_range (struct btrfs_root *root,u64 start,u64 end,struct list_head *list,int search_commit); void btrfs_extent_item_to_extent_map(struct btrfs_inode *inode,
fs/btrfs/file-item.c +11 -8
@@ -423,13 +423,14 @@ int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end,* @inode: Owner of the data inside the bio * @bio: Contains the data to be checksummed * @file_start: offset in file this bio begins to describe -* @contig: Boolean. If true/1 means all bio vecs in this bio are -* contiguous and they begin at @file_start in the file. False/0 -* means this bio can contains potentially discontigous bio vecs -* so the logical offset of each should be calculated separately. +* @contig: If true, all bio vecs in @bio are contiguous and they begin at +* @file_start in the file. If false, @bio may contain +* discontigous bio vecs, so the logical offset of each should be +* calculated separately (@file_start is ignored). +* @one_ordered: If true, @bio only refers to one ordered extent. */ blk_status_t btrfs_csum_one_bio (struct inode *inode,struct bio *bio, -u64 file_start,int contig) +u64 file_start,bool contig,bool one_ordered) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);
@@ -482,8 +483,9 @@ blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio, - 1);for (i =0 ; i < nr_sectors; i++) { -if (offset >= ordered->file_offset + ordered->len || - offset < ordered->file_offset) { +if (!one_ordered && + (offset >= ordered->file_offset + ordered->len || + offset < ordered->file_offset)) {unsigned long bytes_left; sums->len = this_sum_bytes;
@@ -515,7 +517,8 @@ blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio,kunmap_atomic (data);crypto_shash_final (shash, (char *)(sums->sums + index)); index += csum_size; - offset += fs_info->sectorsize; +if (!one_ordered) + offset += fs_info->sectorsize; this_sum_bytes += fs_info->sectorsize; total_bytes += fs_info->sectorsize; }
fs/btrfs/inode.c +4 -4
@@ -2039,7 +2039,7 @@ static blk_status_t btrfs_submit_bio_start(void *private_data, struct bio *bio,struct inode *inode = private_data;blk_status_t ret =0 ; - ret =btrfs_csum_one_bio (inode, bio,0 ,0 ); + ret =btrfs_csum_one_bio (inode, bio,0 , false, false);BUG_ON (ret);/* -ENOMEM */ return 0 ; }
@@ -2104,7 +2104,7 @@ static blk_status_t btrfs_submit_bio_hook(struct inode *inode, struct bio *bio, 0, inode, btrfs_submit_bio_start); goto out; }else if (!skip_sum) { - ret =btrfs_csum_one_bio (inode, bio,0 ,0 ); + ret =btrfs_csum_one_bio (inode, bio,0 , false, false);if (ret) goto out; }
@@ -8272,7 +8272,7 @@ static blk_status_t btrfs_submit_bio_start_direct_io(void *private_data, {struct inode *inode = private_data;blk_status_t ret; - ret =btrfs_csum_one_bio (inode, bio, offset,1 ); + ret =btrfs_csum_one_bio (inode, bio, offset, true, false);BUG_ON (ret);/* -ENOMEM */ return 0 ; }
@@ -8379,7 +8379,7 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,* If we aren't doing async submit, calculate the csum of the * bio now. */ - ret =btrfs_csum_one_bio (inode, bio, file_offset,1 ); + ret =btrfs_csum_one_bio (inode, bio, file_offset, true, false);if (ret) goto err; }else {
Signature · 1 line
2.24.0
Re: [RFC PATCH v3 05/12] btrfs: don't advance offset for compressed bios in btrfs_csum_one_bio()
Quoted 3 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
btrfs_csum_one_bio()loops over each sector in the bio while keeping a
'sector' here is ambiguous it really loops over every fs block (which in btrfs is also known as sector). SO perhaps change the wording in the changelog but also in the function instead of nr_sectors perhahps it could be renamed to blockcount?
Quoted 144 lines
cursor of its current logical position in the file in order to look up the ordered extent to add the checksums to. However, this doesn't make much sense for compressed extents, as a sector on disk does not correspond to a sector of decompressed file data. It happens to work because 1) the compressed bio always covers one ordered extent and 2) the size of the bio is always less than the size of the ordered extent. However, the second point will not always be true for encoded writes.
Let's add a boolean parameter to
btrfs_csum_one_bio()to indicate that it can assume that the bio only covers one ordered extent. Since we're already changing the signature, let's make contig bool instead of int, too.Signed-off-by Omar Sandoval <[email protected]>--- fs/btrfs/compression.c | 5 +++-- fs/btrfs/ctree.h | 2 +- fs/btrfs/file-item.c | 19 +++++++++++-------- fs/btrfs/inode.c | 8 ++++---- 4 files changed, 19 insertions(+), 15 deletions(-)fs/btrfs/compression.c +3
-2@@ -374,7 +374,8 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,BUG_ON (ret);/* -ENOMEM */ if (!skip_sum) { - ret =btrfs_csum_one_bio (inode, bio, start,1 ); + ret =btrfs_csum_one_bio (inode, bio, start, + true, true);BUG_ON (ret);/* -ENOMEM */ }@@ -405,7 +406,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,BUG_ON (ret);/* -ENOMEM */ if (!skip_sum) { - ret =btrfs_csum_one_bio (inode, bio, start,1 ); + ret =btrfs_csum_one_bio (inode, bio, start, true, true);BUG_ON (ret);/* -ENOMEM */ }fs/btrfs/ctree.h +1
-1@@ -2802,7 +2802,7 @@ int btrfs_csum_file_blocks(struct btrfs_trans_handle *trans,struct btrfs_root *root, struct btrfs_ordered_sum *sums);blk_status_t btrfs_csum_one_bio (struct inode *inode,struct bio *bio, -u64 file_start,int contig); +u64 file_start,bool contig,bool one_ordered);int btrfs_lookup_csums_range (struct btrfs_root *root,u64 start,u64 end,struct list_head *list,int search_commit); void btrfs_extent_item_to_extent_map(struct btrfs_inode *inode,fs/btrfs/file-item.c +11
-8@@ -423,13 +423,14 @@ int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end,* @inode: Owner of the data inside the bio * @bio: Contains the data to be checksummed * @file_start: offset in file this bio begins to describe -* @contig: Boolean. If true/1 means all bio vecs in this bio are -* contiguous and they begin at @file_start in the file. False/0 -* means this bio can contains potentially discontigous bio vecs -* so the logical offset of each should be calculated separately. +* @contig: If true, all bio vecs in @bio are contiguous and they begin at +* @file_start in the file. If false, @bio may contain +* discontigous bio vecs, so the logical offset of each should be +* calculated separately (@file_start is ignored). +* @one_ordered: If true, @bio only refers to one ordered extent. */ blk_status_t btrfs_csum_one_bio (struct inode *inode,struct bio *bio, -u64 file_start,int contig) +u64 file_start,bool contig,bool one_ordered) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);@@ -482,8 +483,9 @@ blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio, - 1);for (i =0 ; i < nr_sectors; i++) { -if (offset >= ordered->file_offset + ordered->len || - offset < ordered->file_offset) { +if (!one_ordered && + (offset >= ordered->file_offset + ordered->len || + offset < ordered->file_offset)) {unsigned long bytes_left; sums->len = this_sum_bytes;@@ -515,7 +517,8 @@ blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio,kunmap_atomic (data);crypto_shash_final (shash, (char *)(sums->sums + index)); index += csum_size; - offset += fs_info->sectorsize; +if (!one_ordered) + offset += fs_info->sectorsize; this_sum_bytes += fs_info->sectorsize; total_bytes += fs_info->sectorsize; }fs/btrfs/inode.c +4
-4@@ -2039,7 +2039,7 @@ static blk_status_t btrfs_submit_bio_start(void *private_data, struct bio *bio,struct inode *inode = private_data;blk_status_t ret =0 ; - ret =btrfs_csum_one_bio (inode, bio,0 ,0 ); + ret =btrfs_csum_one_bio (inode, bio,0 , false, false);BUG_ON (ret);/* -ENOMEM */ return 0 ; }@@ -2104,7 +2104,7 @@ static blk_status_t btrfs_submit_bio_hook(struct inode *inode, struct bio *bio, 0, inode, btrfs_submit_bio_start); goto out; }else if (!skip_sum) { - ret =btrfs_csum_one_bio (inode, bio,0 ,0 ); + ret =btrfs_csum_one_bio (inode, bio,0 , false, false);if (ret) goto out; }@@ -8272,7 +8272,7 @@ static blk_status_t btrfs_submit_bio_start_direct_io(void *private_data, {struct inode *inode = private_data;blk_status_t ret; - ret =btrfs_csum_one_bio (inode, bio, offset,1 ); + ret =btrfs_csum_one_bio (inode, bio, offset, true, false);BUG_ON (ret);/* -ENOMEM */ return 0 ; }@@ -8379,7 +8379,7 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,* If we aren't doing async submit, calculate the csum of the * bio now. */ - ret =btrfs_csum_one_bio (inode, bio, file_offset,1 ); + ret =btrfs_csum_one_bio (inode, bio, file_offset, true, false);if (ret) goto err; }else {
Re: [RFC PATCH v3 05/12] btrfs: don't advance offset for compressed bios in btrfs_csum_one_bio()
Quoted 11 lines · Nikolay Borisov
Quoted 3 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
btrfs_csum_one_bio()loops over each sector in the bio while keeping a'sector' here is ambiguous it really loops over every fs block (which in btrfs is also known as sector). SO perhaps change the wording in the changelog but also in the function instead of
nr_sectorsperhahps it could be renamed to blockcount?
Fixed, thanks.
RFC PATCH v3 6/12 btrfs: remove dead snapshot-aware defrag code
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Snapshot-aware defrag has been disabled since commit 8101c8dbf624 ("Btrfs: disable snapshot aware defrag for now") almost 6 years ago. Let's remove the dead code. If someone is up to the task of bringing it back, they can dig it up from git.
This is logically a revert of commit 38c227d87c49 ("Btrfs:
snapshot-aware defrag") except that now we have to clear the EXTENT_DEFRAG bit to avoid need_force_cow() returning true forever.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/inode.c | 695 +---------------------------------------------- 1 file changed, 11 insertions(+), 684 deletions(-)
fs/btrfs/inode.c +11 -684
@@ -44,7 +44,6 @@#include "locking.h" #include "free-space-cache.h" #include "inode-map.h" -#include "backref.h" #include "props.h" #include "qgroup.h" #include "delalloc-space.h"
@@ -2353,649 +2352,6 @@ static int insert_reserved_file_extent(struct btrfs_trans_handle *trans,return ret; } -/* snapshot-aware defrag */ -struct sa_defrag_extent_backref { -struct rb_node node; -struct old_sa_defrag_extent *old; -u64 root_id; -u64 inum; -u64 file_pos; -u64 extent_offset; -u64 num_bytes; -u64 generation; -}; - -struct old_sa_defrag_extent { -struct list_head list; -struct new_sa_defrag_extent *new; - -u64 extent_offset; -u64 bytenr; -u64 offset; -u64 len; -int count; -}; - -struct new_sa_defrag_extent { -struct rb_root root; -struct list_head head; -struct btrfs_path *path; -struct inode *inode; -u64 file_pos; -u64 len; -u64 bytenr; -u64 disk_len; -u8 compress_type; -}; - -static int backref_comp (struct sa_defrag_extent_backref *b1, -struct sa_defrag_extent_backref *b2) -{ -if (b1->root_id < b2->root_id) -return -1 ; -else if (b1->root_id > b2->root_id) -return 1 ; - -if (b1->inum < b2->inum) -return -1 ; -else if (b1->inum > b2->inum) -return 1 ; - -if (b1->file_pos < b2->file_pos) -return -1 ; -else if (b1->file_pos > b2->file_pos) -return 1 ; - -/* -* [------------------------------] ===> (a range of space) -* |<--->| |<---->| =============> (fs/file tree A) -* |<---------------------------->| ===> (fs/file tree B) -* -* A range of space can refer to two file extents in one tree while -* refer to only one file extent in another tree. -* -* So we may process a disk offset more than one time(two extents in A) -* and locate at the same extent(one extent in B), then insert two same -* backrefs(both refer to the extent in B). -*/ -return 0 ; -} - -static void backref_insert (struct rb_root *root, -struct sa_defrag_extent_backref *backref) -{ -struct rb_node **p = &root->rb_node; -struct rb_node *parent =NULL ; -struct sa_defrag_extent_backref *entry; -int ret; - -while (*p) { - parent = *p; - entry = rb_entry(parent, struct sa_defrag_extent_backref, node); - - ret =backref_comp (backref, entry); -if (ret <0 ) - p = &(*p)->rb_left; -else - p = &(*p)->rb_right; - } - -rb_link_node (&backref->node, parent, p); -rb_insert_color (&backref->node, root); -} - -/* -* Note the backref might has changed, and in this case we just return 0. -*/ -static noinline int record_one_backref(u64 inum, u64 offset, u64 root_id, -void *ctx) -{ -struct btrfs_file_extent_item *extent; -struct old_sa_defrag_extent *old = ctx; -struct new_sa_defrag_extent *new = old->new; -struct btrfs_path *path = new->path; -struct btrfs_key key; -struct btrfs_root *root; -struct sa_defrag_extent_backref *backref; -struct extent_buffer *leaf; -struct inode *inode = new->inode; -struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb); -int slot; -int ret; -u64 extent_offset; -u64 num_bytes; - -if (BTRFS_I (inode)->root->root_key.objectid == root_id && - inum ==btrfs_ino (BTRFS_I (inode))) -return 0 ; - - key.objectid = root_id; - key.type =BTRFS_ROOT_ITEM_KEY ; - key.offset = (u64)-1 ; - - root =btrfs_read_fs_root_no_name (fs_info, &key); -if (IS_ERR (root)) { -if (PTR_ERR (root) == -ENOENT ) -return 0 ; -WARN_ON (1 ); -btrfs_debug (fs_info,"inum=%llu, offset=%llu, root_id=%llu" , - inum, offset, root_id); -return PTR_ERR (root); - } - - key.objectid = inum; - key.type =BTRFS_EXTENT_DATA_KEY ; -if (offset > (u64)-1 <<32 ) - key.offset =0 ; -else - key.offset = offset; - - ret =btrfs_search_slot (NULL , root, &key, path,0 ,0 ); -if (WARN_ON (ret <0 )) -return ret; - ret =0 ; - -while (1 ) { -cond_resched (); - - leaf = path->nodes[0 ]; - slot = path->slots[0 ]; - -if (slot >=btrfs_header_nritems (leaf)) { - ret =btrfs_next_leaf (root, path); -if (ret <0 ) { - goto out; - }else if (ret >0 ) { - ret =0 ; - goto out; - } -continue ; - } - - path->slots[0 ]++; - -btrfs_item_key_to_cpu (leaf, &key, slot); - -if (key.objectid > inum) - goto out; - -if (key.objectid < inum || key.type !=BTRFS_EXTENT_DATA_KEY ) -continue ; - - extent =btrfs_item_ptr (leaf, slot, - struct btrfs_file_extent_item); - -if (btrfs_file_extent_disk_bytenr (leaf, extent) != old->bytenr) -continue ; - -/* -* 'offset' refers to the exact key.offset, -* NOT the 'offset' field in btrfs_extent_data_ref, ie. -* (key.offset - extent_offset). -*/ -if (key.offset != offset) -continue ; - - extent_offset =btrfs_file_extent_offset (leaf, extent); - num_bytes =btrfs_file_extent_num_bytes (leaf, extent); - -if (extent_offset >= old->extent_offset + old->offset + - old->len || extent_offset + num_bytes <= - old->extent_offset + old->offset) -continue ; -break ; - } - - backref =kmalloc (sizeof (*backref),GFP_NOFS ); -if (!backref) { - ret = -ENOENT ; - goto out; - } - - backref->root_id = root_id; - backref->inum = inum; - backref->file_pos = offset; - backref->num_bytes = num_bytes; - backref->extent_offset = extent_offset; - backref->generation =btrfs_file_extent_generation (leaf, extent); - backref->old = old; -backref_insert (&new->root, backref); - old->count++; -out: -btrfs_release_path (path); -WARN_ON (ret); -return ret; -} - -static noinline bool record_extent_backrefs(struct btrfs_path *path, -struct new_sa_defrag_extent *new) -{ -struct btrfs_fs_info *fs_info =btrfs_sb (new->inode->i_sb); -struct old_sa_defrag_extent *old, *tmp; -int ret; - - new->path = path; - -list_for_each_entry_safe (old, tmp, &new->head, list) { - ret =iterate_inodes_from_logical (old->bytenr + - old->extent_offset, fs_info, - path, record_one_backref, - old, false); -if (ret <0 && ret != -ENOENT ) -return false; - -/* no backref to be processed for this extent */ -if (!old->count) { -list_del (&old->list); -kfree (old); - } - } - -if (list_empty (&new->head)) -return false; - -return true; -} - -static int relink_is_mergable (struct extent_buffer *leaf, -struct btrfs_file_extent_item *fi, -struct new_sa_defrag_extent *new) -{ -if (btrfs_file_extent_disk_bytenr (leaf, fi) != new->bytenr) -return 0 ; - -if (btrfs_file_extent_type (leaf, fi) !=BTRFS_FILE_EXTENT_REG ) -return 0 ; - -if (btrfs_file_extent_compression (leaf, fi) != new->compress_type) -return 0 ; - -if (btrfs_file_extent_encryption (leaf, fi) || -btrfs_file_extent_other_encoding (leaf, fi)) -return 0 ; - -return 1 ; -} - -/* -* Note the backref might has changed, and in this case we just return 0. -*/ -static noinline int relink_extent_backref(struct btrfs_path *path, -struct sa_defrag_extent_backref *prev, -struct sa_defrag_extent_backref *backref) -{ -struct btrfs_file_extent_item *extent; -struct btrfs_file_extent_item *item; -struct btrfs_ordered_extent *ordered; -struct btrfs_trans_handle *trans; -struct btrfs_ref ref = {0 }; -struct btrfs_root *root; -struct btrfs_key key; -struct extent_buffer *leaf; -struct old_sa_defrag_extent *old = backref->old; -struct new_sa_defrag_extent *new = old->new; -struct btrfs_fs_info *fs_info =btrfs_sb (new->inode->i_sb); -struct inode *inode; -struct extent_state *cached =NULL ; -int ret =0 ; -u64 start; -u64 len; -u64 lock_start; -u64 lock_end; -bool merge = false; -int index; - -if (prev && prev->root_id == backref->root_id && - prev->inum == backref->inum && - prev->file_pos + prev->num_bytes == backref->file_pos) - merge = true; - -/* step 1: get root */ - key.objectid = backref->root_id; - key.type =BTRFS_ROOT_ITEM_KEY ; - key.offset = (u64)-1 ; - - index =srcu_read_lock (&fs_info->subvol_srcu); - - root =btrfs_read_fs_root_no_name (fs_info, &key); -if (IS_ERR (root)) { -srcu_read_unlock (&fs_info->subvol_srcu, index); -if (PTR_ERR (root) == -ENOENT ) -return 0 ; -return PTR_ERR (root); - } - -if (btrfs_root_readonly (root)) { -srcu_read_unlock (&fs_info->subvol_srcu, index); -return 0 ; - } - -/* step 2: get inode */ - key.objectid = backref->inum; - key.type =BTRFS_INODE_ITEM_KEY ; - key.offset =0 ; - - inode =btrfs_iget (fs_info->sb, &key, root,NULL ); -if (IS_ERR (inode)) { -srcu_read_unlock (&fs_info->subvol_srcu, index); -return 0 ; - } - -srcu_read_unlock (&fs_info->subvol_srcu, index); - -/* step 3: relink backref */ - lock_start = backref->file_pos; - lock_end = backref->file_pos + backref->num_bytes -1 ; -lock_extent_bits (&BTRFS_I (inode)->io_tree, lock_start, lock_end, - &cached); - - ordered =btrfs_lookup_first_ordered_extent (inode, lock_end); -if (ordered) { -btrfs_put_ordered_extent (ordered); - goto out_unlock; - } - - trans =btrfs_join_transaction (root); -if (IS_ERR (trans)) { - ret =PTR_ERR (trans); - goto out_unlock; - } - - key.objectid = backref->inum; - key.type =BTRFS_EXTENT_DATA_KEY ; - key.offset = backref->file_pos; - - ret =btrfs_search_slot (NULL , root, &key, path,0 ,0 ); -if (ret <0 ) { - goto out_free_path; - }else if (ret >0 ) { - ret =0 ; - goto out_free_path; - } - - extent =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], - struct btrfs_file_extent_item); - -if (btrfs_file_extent_generation (path->nodes[0 ], extent) != - backref->generation) - goto out_free_path; - -btrfs_release_path (path); - - start = backref->file_pos; -if (backref->extent_offset < old->extent_offset + old->offset) - start += old->extent_offset + old->offset - - backref->extent_offset; - - len =min (backref->extent_offset + backref->num_bytes, - old->extent_offset + old->offset + old->len); - len -=max (backref->extent_offset, old->extent_offset + old->offset); - - ret =btrfs_drop_extents (trans, root, inode, start, - start + len,1 ); -if (ret) - goto out_free_path; -again: - key.objectid =btrfs_ino (BTRFS_I (inode)); - key.type =BTRFS_EXTENT_DATA_KEY ; - key.offset = start; - - path->leave_spinning =1 ; -if (merge) { -struct btrfs_file_extent_item *fi; -u64 extent_len; -struct btrfs_key found_key; - - ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); -if (ret <0 ) - goto out_free_path; - - path->slots[0 ]--; - leaf = path->nodes[0 ]; -btrfs_item_key_to_cpu (leaf, &found_key, path->slots[0 ]); - - fi =btrfs_item_ptr (leaf, path->slots[0 ], - struct btrfs_file_extent_item); - extent_len =btrfs_file_extent_num_bytes (leaf, fi); - -if (extent_len + found_key.offset == start && -relink_is_mergable (leaf, fi, new)) { -btrfs_set_file_extent_num_bytes (leaf, fi, - extent_len + len); -btrfs_mark_buffer_dirty (leaf); -inode_add_bytes (inode, len); - - ret =1 ; - goto out_free_path; - }else { - merge = false; -btrfs_release_path (path); - goto again; - } - } - - ret =btrfs_insert_empty_item (trans, root, path, &key, -sizeof (*extent)); -if (ret) { -btrfs_abort_transaction (trans, ret); - goto out_free_path; - } - - leaf = path->nodes[0 ]; - item =btrfs_item_ptr (leaf, path->slots[0 ], - struct btrfs_file_extent_item); -btrfs_set_file_extent_disk_bytenr (leaf, item, new->bytenr); -btrfs_set_file_extent_disk_num_bytes (leaf, item, new->disk_len); -btrfs_set_file_extent_offset (leaf, item, start - new->file_pos); -btrfs_set_file_extent_num_bytes (leaf, item, len); -btrfs_set_file_extent_ram_bytes (leaf, item, new->len); -btrfs_set_file_extent_generation (leaf, item, trans->transid); -btrfs_set_file_extent_type (leaf, item,BTRFS_FILE_EXTENT_REG ); -btrfs_set_file_extent_compression (leaf, item, new->compress_type); -btrfs_set_file_extent_encryption (leaf, item,0 ); -btrfs_set_file_extent_other_encoding (leaf, item,0 ); - -btrfs_mark_buffer_dirty (leaf); -inode_add_bytes (inode, len); -btrfs_release_path (path); - -btrfs_init_generic_ref (&ref,BTRFS_ADD_DELAYED_REF , new->bytenr, - new->disk_len,0 ); -btrfs_init_data_ref (&ref, backref->root_id, backref->inum, - new->file_pos);/* start - extent_offset */ - ret =btrfs_inc_extent_ref (trans, &ref); -if (ret) { -btrfs_abort_transaction (trans, ret); - goto out_free_path; - } - - ret =1 ; -out_free_path: -btrfs_release_path (path); - path->leave_spinning =0 ; -btrfs_end_transaction (trans); -out_unlock: -unlock_extent_cached (&BTRFS_I (inode)->io_tree, lock_start, lock_end, - &cached); -iput (inode); -return ret; -} - -static void free_sa_defrag_extent (struct new_sa_defrag_extent *new) -{ -struct old_sa_defrag_extent *old, *tmp; - -if (!new) -return ; - -list_for_each_entry_safe (old, tmp, &new->head, list) { -kfree (old); - } -kfree (new); -} - -static void relink_file_extents (struct new_sa_defrag_extent *new) -{ -struct btrfs_fs_info *fs_info =btrfs_sb (new->inode->i_sb); -struct btrfs_path *path; -struct sa_defrag_extent_backref *backref; -struct sa_defrag_extent_backref *prev =NULL ; -struct rb_node *node; -int ret; - - path =btrfs_alloc_path (); -if (!path) -return ; - -if (!record_extent_backrefs (path, new)) { -btrfs_free_path (path); - goto out; - } -btrfs_release_path (path); - -while (1 ) { - node =rb_first (&new->root); -if (!node) -break ; -rb_erase (node, &new->root); - - backref = rb_entry(node, struct sa_defrag_extent_backref, node); - - ret =relink_extent_backref (path, prev, backref); -WARN_ON (ret <0 ); - -kfree (prev); - -if (ret ==1 ) - prev = backref; -else - prev =NULL ; -cond_resched (); - } -kfree (prev); - -btrfs_free_path (path); -out: -free_sa_defrag_extent (new); - -atomic_dec (&fs_info->defrag_running); -wake_up (&fs_info->transaction_wait); -} - -static struct new_sa_defrag_extent * -record_old_file_extents (struct inode *inode, -struct btrfs_ordered_extent *ordered) -{ -struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb); -struct btrfs_root *root =BTRFS_I (inode)->root; -struct btrfs_path *path; -struct btrfs_key key; -struct old_sa_defrag_extent *old; -struct new_sa_defrag_extent *new; -int ret; - - new =kmalloc (sizeof (*new),GFP_NOFS ); -if (!new) -return NULL ; - - new->inode = inode; - new->file_pos = ordered->file_offset; - new->len = ordered->len; - new->bytenr = ordered->start; - new->disk_len = ordered->disk_len; - new->compress_type = ordered->compress_type; - new->root =RB_ROOT ; -INIT_LIST_HEAD (&new->head); - - path =btrfs_alloc_path (); -if (!path) - goto out_kfree; - - key.objectid =btrfs_ino (BTRFS_I (inode)); - key.type =BTRFS_EXTENT_DATA_KEY ; - key.offset = new->file_pos; - - ret =btrfs_search_slot (NULL , root, &key, path,0 ,0 ); -if (ret <0 ) - goto out_free_path; -if (ret >0 && path->slots[0 ] >0 ) - path->slots[0 ]--; - -/* find out all the old extents for the file range */ -while (1 ) { -struct btrfs_file_extent_item *extent; -struct extent_buffer *l; -int slot; -u64 num_bytes; -u64 offset; -u64 end; -u64 disk_bytenr; -u64 extent_offset; - - l = path->nodes[0 ]; - slot = path->slots[0 ]; - -if (slot >=btrfs_header_nritems (l)) { - ret =btrfs_next_leaf (root, path); -if (ret <0 ) - goto out_free_path; -else if (ret >0 ) -break ; -continue ; - } - -btrfs_item_key_to_cpu (l, &key, slot); - -if (key.objectid !=btrfs_ino (BTRFS_I (inode))) -break ; -if (key.type !=BTRFS_EXTENT_DATA_KEY ) -break ; -if (key.offset >= new->file_pos + new->len) -break ; - - extent = btrfs_item_ptr(l, slot, struct btrfs_file_extent_item); - - num_bytes =btrfs_file_extent_num_bytes (l, extent); -if (key.offset + num_bytes < new->file_pos) - goto next; - - disk_bytenr =btrfs_file_extent_disk_bytenr (l, extent); -if (!disk_bytenr) - goto next; - - extent_offset =btrfs_file_extent_offset (l, extent); - - old =kmalloc (sizeof (*old),GFP_NOFS ); -if (!old) - goto out_free_path; - - offset =max (new->file_pos, key.offset); - end =min (new->file_pos + new->len, key.offset + num_bytes); - - old->bytenr = disk_bytenr; - old->extent_offset = extent_offset; - old->offset = offset - key.offset; - old->len = end - offset; - old->new = new; - old->count =0 ; -list_add_tail (&old->list, &new->head); -next: - path->slots[0 ]++; -cond_resched (); - } - -btrfs_free_path (path); -atomic_inc (&fs_info->defrag_running); - -return new; - -out_free_path: -btrfs_free_path (path); -out_kfree: -free_sa_defrag_extent (new); -return NULL ; -} -static void btrfs_release_delalloc_bytes (struct btrfs_fs_info *fs_info,u64 start,u64 len) {
@@ -3023,7 +2379,6 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)struct btrfs_trans_handle *trans =NULL ;struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct extent_state *cached_state =NULL ; -struct new_sa_defrag_extent *new =NULL ;int compress_type =0 ;int ret =0 ;u64 logical_len = ordered_extent->len;
@@ -3032,6 +2387,7 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)bool range_locked = false;bool clear_new_delalloc_bytes = false;bool clear_reserved_extent = true; +unsigned int clear_bits; if (!test_bit(BTRFS_ORDERED_NOCOW, &ordered_extent->flags) && !test_bit(BTRFS_ORDERED_PREALLOC, &ordered_extent->flags) &&
@@ -3090,20 +2446,6 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent) ordered_extent->file_offset + ordered_extent->len -1 , &cached_state); - ret =test_range_bit (io_tree, ordered_extent->file_offset, - ordered_extent->file_offset + ordered_extent->len -1 , -EXTENT_DEFRAG ,0 , cached_state); -if (ret) { -u64 last_snapshot =btrfs_root_last_snapshot (&root->root_item); -if (0 && last_snapshot >=BTRFS_I (inode)->generation) -/* the inode is shared */ - new =record_old_file_extents (inode, ordered_extent); - -clear_extent_bit (io_tree, ordered_extent->file_offset, - ordered_extent->file_offset + ordered_extent->len -1 , -EXTENT_DEFRAG ,0 ,0 , &cached_state); - } -if (nolock) trans =btrfs_join_transaction_nolock (root);else
@@ -3164,21 +2506,16 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent) } ret =0 ; out: -if (range_locked || clear_new_delalloc_bytes) { -unsigned int clear_bits =0 ; - -if (range_locked) - clear_bits |=EXTENT_LOCKED ; -if (clear_new_delalloc_bytes) - clear_bits |=EXTENT_DELALLOC_NEW ; -clear_extent_bit (&BTRFS_I (inode)->io_tree, - ordered_extent->file_offset, - ordered_extent->file_offset + - ordered_extent->len -1 , - clear_bits, - (clear_bits &EXTENT_LOCKED ) ?1 :0 , -0 , &cached_state); - } + clear_bits =EXTENT_DEFRAG ; +if (range_locked) + clear_bits |=EXTENT_LOCKED ; +if (clear_new_delalloc_bytes) + clear_bits |=EXTENT_DELALLOC_NEW ; +clear_extent_bit (&BTRFS_I (inode)->io_tree, + ordered_extent->file_offset, + ordered_extent->file_offset + ordered_extent->len -1 , + clear_bits, (clear_bits &EXTENT_LOCKED ) ?1 :0 ,0 , + &cached_state);if (trans)btrfs_end_transaction (trans);
@@ -3222,16 +2559,6 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)*/ btrfs_remove_ordered_extent (inode, ordered_extent); -/* for snapshot-aware defrag */ -if (new) { -if (ret) { -free_sa_defrag_extent (new); -atomic_dec (&fs_info->defrag_running); - }else { -relink_file_extents (new); - } - } -/* once for us */ btrfs_put_ordered_extent (ordered_extent);/* once for the tree */
Signature · 1 line
2.24.0
Re: [RFC PATCH v3 06/12] btrfs: remove dead snapshot-aware defrag code
Quoted 7 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Snapshot-aware defrag has been disabled since commit
8101c8dbf624("Btrfs: disable snapshot aware defrag for now") almost 6 years ago. Let's remove the dead code. If someone is up to the task of bringing it back, they can dig it up from git.
Quoted 5 lines
This is logically a revert of commit
38c227d87c49("Btrfs:
snapshot-aware defrag") except that now we have to clear theEXTENT_DEFRAGbit to avoidneed_force_cow()returning true forever.Signed-off-by Omar Sandoval <[email protected]>
Reviewed-by Nikolay Borisov <[email protected]>
RFC PATCH v3 7/12 btrfs: make btrfs_ordered_extent naming consistent with btrfs_file_extent_item
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
ordered->start, ordered->len, and ordered->disk_len correspond to fi->disk_bytenr, fi->num_bytes, and fi->disk_num_bytes, respectively. It's confusing to translate between the two naming schemes. Since a btrfs_ordered_extent is basically a pending btrfs_file_extent_item, let's make the former use the naming from the latter.
Note that I didn't touch the names in tracepoints just in case there are scripts depending on the current naming.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/file-item.c | 2 +- fs/btrfs/file.c | 6 ++-- fs/btrfs/inode.c | 67 ++++++++++++++++------------------ fs/btrfs/ordered-data.c | 69 ++++++++++++++++++------------------ fs/btrfs/ordered-data.h | 26 +++++++------- fs/btrfs/relocation.c | 5 +-- include/trace/events/btrfs.h | 6 ++-- 7 files changed, 89 insertions(+), 92 deletions(-)
fs/btrfs/file-item.c +1 -1
@@ -484,7 +484,7 @@ blk_status_t btrfs_csum_one_bio(struct inode *inode, struct bio *bio,for (i =0 ; i < nr_sectors; i++) {if (!one_ordered && - (offset >= ordered->file_offset + ordered->len || + (offset >= ordered->file_offset + ordered->num_bytes || offset < ordered->file_offset)) {unsigned long bytes_left;
fs/btrfs/file.c +3 -3
@@ -1503,7 +1503,7 @@ lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages, ordered =btrfs_lookup_ordered_range (inode, start_pos, last_pos - start_pos +1 );if (ordered && - ordered->file_offset + ordered->len > start_pos && + ordered->file_offset + ordered->num_bytes > start_pos && ordered->file_offset <= last_pos) {unlock_extent_cached (&inode->io_tree, start_pos, last_pos, cached_state);
@@ -2428,7 +2428,7 @@ static int btrfs_punch_hole_lock_range(struct inode *inode,* we need to try again. */ if ((!ordered || - (ordered->file_offset + ordered->len <= lockstart || + (ordered->file_offset + ordered->num_bytes <= lockstart || ordered->file_offset > lockend)) && !filemap_range_has_page (inode->i_mapping, lockstart, lockend)) {
@@ -3250,7 +3250,7 @@ static long btrfs_fallocate(struct file *file, int mode, ordered =btrfs_lookup_first_ordered_extent (inode, locked_end);if (ordered && - ordered->file_offset + ordered->len > alloc_start && + ordered->file_offset + ordered->num_bytes > alloc_start && ordered->file_offset < alloc_end) {btrfs_put_ordered_extent (ordered); unlock_extent_cached(&BTRFS_I(inode)->io_tree,
fs/btrfs/inode.c +31 -36
156 lines not on this page · raw message
fs/btrfs/ordered-data.c +34 -35
160 lines not on this page · raw message
fs/btrfs/ordered-data.h +14 -12
@@ -67,14 +67,13 @@ struct btrfs_ordered_extent {/* logical offset in the file */ u64 file_offset; -/* disk byte number */ -u64 start; - -/* ram length of the extent in bytes */ -u64 len; - -/* extent length on disk */ -u64 disk_len; +/* +* These fields directly correspond to the same fields in +* btrfs_file_extent_item. +*/ +u64 disk_bytenr; +u64 num_bytes; +u64 disk_num_bytes;/* number of bytes that still need writing */ u64 bytes_left;
@@ -161,12 +160,15 @@ int btrfs_dec_test_first_ordered_pending(struct inode *inode, u64 *file_offset, u64 io_size, int uptodate);int btrfs_add_ordered_extent (struct inode *inode,u64 file_offset, -u64 start,u64 len,u64 disk_len,int type); +u64 disk_bytenr,u64 num_bytes,u64 disk_num_bytes, +int type);int btrfs_add_ordered_extent_dio (struct inode *inode,u64 file_offset, -u64 start,u64 len,u64 disk_len,int type); +u64 disk_bytenr,u64 num_bytes, +u64 disk_num_bytes,int type);int btrfs_add_ordered_extent_compress (struct inode *inode,u64 file_offset, -u64 start,u64 len,u64 disk_len, -int type,int compress_type); +u64 disk_bytenr,u64 num_bytes, +u64 disk_num_bytes,int type, +int compress_type);void btrfs_add_ordered_sum (struct btrfs_ordered_extent *entry,struct btrfs_ordered_sum *sum); struct btrfs_ordered_extent *btrfs_lookup_ordered_extent(struct inode *inode,
fs/btrfs/relocation.c +3 -2
@@ -4617,7 +4617,7 @@ int btrfs_reloc_clone_csums(struct inode *inode, u64 file_pos, u64 len)LIST_HEAD (list); ordered =btrfs_lookup_ordered_extent (inode, file_pos); -BUG_ON (ordered->file_offset != file_pos || ordered->len != len); +BUG_ON (ordered->file_offset != file_pos || ordered->num_bytes != len); disk_bytenr = file_pos +BTRFS_I (inode)->index_cnt; ret = btrfs_lookup_csums_range(fs_info->csum_root, disk_bytenr,
@@ -4641,7 +4641,8 @@ int btrfs_reloc_clone_csums(struct inode *inode, u64 file_pos, u64 len)* disk_len vs real len like with real inodes since it's all * disk length. */ - new_bytenr = ordered->start + (sums->bytenr - disk_bytenr); + new_bytenr = (ordered->disk_bytenr + + (sums->bytenr - disk_bytenr)); sums->bytenr = new_bytenr;btrfs_add_ordered_sum (ordered, sums);
include/trace/events/btrfs.h +3 -3
@@ -496,9 +496,9 @@ DECLARE_EVENT_CLASS(btrfs__ordered_extent,TP_fast_assign_btrfs (btrfs_sb (inode->i_sb), __entry->ino = btrfs_ino(BTRFS_I(inode)); __entry->file_offset = ordered->file_offset; - __entry->start = ordered->start; - __entry->len = ordered->len; - __entry->disk_len = ordered->disk_len; + __entry->start = ordered->disk_bytenr; + __entry->len = ordered->num_bytes; + __entry->disk_len = ordered->disk_num_bytes; __entry->bytes_left = ordered->bytes_left; __entry->flags = ordered->flags; __entry->compress_type = ordered->compress_type;
Signature · 1 line
2.24.0
RFC PATCH v3 8/12 btrfs: add ram_bytes and offset to btrfs_ordered_extent
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Currently, we only create ordered extents when ram_bytes == num_bytes and offset == 0. However, RWF_ENCODED writes may create extents which only refer to a subset of the full unencoded extent, so we need to plumb these fields through the ordered extent infrastructure and pass them down to insert_reserved_file_extent().
Since we're changing the btrfs_add_ordered_extent* signature, let's get rid of the trivial wrappers and add a kernel-doc.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/inode.c | 65 +++++++++++++++++++++++------------------ fs/btrfs/ordered-data.c | 65 +++++++++++++++-------------------------- fs/btrfs/ordered-data.h | 16 ++++------ 3 files changed, 67 insertions(+), 79 deletions(-)
fs/btrfs/inode.c +37 -28
@@ -846,13 +846,12 @@ static noinline void submit_compressed_extents(struct async_chunk *async_chunk) goto out_free_reserve;free_extent_map (em); - ret =btrfs_add_ordered_extent_compress (inode, - async_extent->start, - ins.objectid, - async_extent->ram_size, - ins.offset, -BTRFS_ORDERED_COMPRESSED , - async_extent->compress_type); + ret =btrfs_add_ordered_extent (inode, async_extent->start, + async_extent->ram_size, + async_extent->ram_size, + ins.objectid, ins.offset,0 , +1 <<BTRFS_ORDERED_COMPRESSED , + async_extent->compress_type);if (ret) { btrfs_drop_extent_cache(BTRFS_I(inode), async_extent->start,
@@ -1046,8 +1045,9 @@ static noinline int cow_file_range(struct inode *inode, }free_extent_map (em); - ret =btrfs_add_ordered_extent (inode, start, ins.objectid, - ram_size, cur_alloc_size,0 ); + ret =btrfs_add_ordered_extent (inode, start, ram_size, ram_size, + ins.objectid, cur_alloc_size,0 , +0 ,BTRFS_COMPRESS_NONE );if (ret) goto out_drop_extent_cache;
@@ -1584,10 +1584,11 @@ static noinline int run_delalloc_nocow(struct inode *inode, goto error; }free_extent_map (em); - ret =btrfs_add_ordered_extent (inode, cur_offset, - disk_bytenr, num_bytes, - num_bytes, -BTRFS_ORDERED_PREALLOC ); + ret =btrfs_add_ordered_extent (inode, + cur_offset, num_bytes, num_bytes, + disk_bytenr, num_bytes,0 , +1 <<BTRFS_ORDERED_PREALLOC , +BTRFS_COMPRESS_NONE );if (ret) { btrfs_drop_extent_cache(BTRFS_I(inode), cur_offset,
@@ -1597,9 +1598,11 @@ static noinline int run_delalloc_nocow(struct inode *inode, } }else { ret =btrfs_add_ordered_extent (inode, cur_offset, + num_bytes, num_bytes, disk_bytenr, num_bytes, - num_bytes, -BTRFS_ORDERED_NOCOW ); +0 , +1 <<BTRFS_ORDERED_NOCOW , +BTRFS_COMPRESS_NONE );if (ret) goto error; }
@@ -2269,7 +2272,7 @@ int btrfs_writepage_cow_fixup(struct page *page, u64 start, u64 end)static int insert_reserved_file_extent (struct btrfs_trans_handle *trans,struct inode *inode,u64 file_pos,u64 disk_bytenr,u64 disk_num_bytes, -u64 num_bytes,u64 ram_bytes, +u64 offset,u64 num_bytes,u64 ram_bytes,u8 compression,u8 encryption,u16 other_encoding,int extent_type) {
@@ -2319,7 +2322,7 @@ static int insert_reserved_file_extent(struct btrfs_trans_handle *trans,btrfs_set_file_extent_type (leaf, fi, extent_type);btrfs_set_file_extent_disk_bytenr (leaf, fi, disk_bytenr);btrfs_set_file_extent_disk_num_bytes (leaf, fi, disk_num_bytes); -btrfs_set_file_extent_offset (leaf, fi,0 ); +btrfs_set_file_extent_offset (leaf, fi, offset);btrfs_set_file_extent_num_bytes (leaf, fi, num_bytes);btrfs_set_file_extent_ram_bytes (leaf, fi, ram_bytes);btrfs_set_file_extent_compression (leaf, fi, compression);
@@ -2345,7 +2348,8 @@ static int insert_reserved_file_extent(struct btrfs_trans_handle *trans, qg_released = ret; ret =btrfs_alloc_reserved_file_extent (trans, root,btrfs_ino (BTRFS_I (inode)), - file_pos, qg_released, &ins); + file_pos - offset, qg_released, + &ins); out:btrfs_free_path (path);
@@ -2382,7 +2386,8 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)u64 start, end;int compress_type =0 ;int ret =0 ; -u64 logical_len = ordered_extent->num_bytes; +u64 num_bytes = ordered_extent->num_bytes; +u64 ram_bytes = ordered_extent->ram_bytes;bool nolock;bool truncated = false;bool range_locked = false;
@@ -2409,9 +2414,9 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)if (test_bit (BTRFS_ORDERED_TRUNCATED , &ordered_extent->flags)) { truncated = true; - logical_len = ordered_extent->truncated_len; + num_bytes = ram_bytes = ordered_extent->truncated_len;/* Truncated the entire extent, don't bother adding */ -if (!logical_len) +if (!num_bytes) goto out; }
@@ -2466,13 +2471,14 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent) ret =btrfs_mark_extent_written (trans,BTRFS_I (inode), ordered_extent->file_offset, ordered_extent->file_offset + - logical_len); + num_bytes); }else {BUG_ON (root == fs_info->tree_root); ret =insert_reserved_file_extent (trans, inode, start, ordered_extent->disk_bytenr, ordered_extent->disk_num_bytes, - logical_len, logical_len, + ordered_extent->offset, + num_bytes, ram_bytes, compress_type,0 ,0 ,BTRFS_FILE_EXTENT_REG );if (!ret) {
@@ -2520,7 +2526,7 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)u64 unwritten_start = start;if (truncated) - unwritten_start += logical_len; + unwritten_start += num_bytes;clear_extent_uptodate (io_tree, unwritten_start, end,NULL );/* Drop the cache for the part of the extent we didn't write. */
@@ -2537,7 +2543,7 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)* errored out then we don't need to do this as the accounting * has already been done. */ -if ((ret || !logical_len) && +if ((ret || !num_bytes) && clear_reserved_extent && !test_bit (BTRFS_ORDERED_NOCOW , &ordered_extent->flags) && !test_bit (BTRFS_ORDERED_PREALLOC , &ordered_extent->flags))
@@ -6609,8 +6615,11 @@ static struct extent_map *btrfs_create_dio_extent(struct inode *inode,if (IS_ERR (em)) goto out; } - ret =btrfs_add_ordered_extent_dio (inode, start, block_start, - len, block_len, type); + ret =btrfs_add_ordered_extent (inode, start, len, len, block_start, + block_len,0 , + (1 << type) | + (1 <<BTRFS_ORDERED_DIRECT ), +BTRFS_COMPRESS_NONE );if (ret) {if (em) {free_extent_map (em);
@@ -9743,7 +9752,7 @@ static int __btrfs_prealloc_file_range(struct inode *inode, int mode, last_alloc = ins.offset; ret =insert_reserved_file_extent (trans, inode, cur_offset, ins.objectid, - ins.offset, ins.offset, + ins.offset,0 , ins.offset, ins.offset,0 ,0 ,0 ,BTRFS_FILE_EXTENT_PREALLOC );if (ret) {
fs/btrfs/ordered-data.c +24 -41
93 lines not on this page · raw message
fs/btrfs/ordered-data.h +6 -10
@@ -71,9 +71,11 @@ struct btrfs_ordered_extent {* These fields directly correspond to the same fields in * btrfs_file_extent_item. */ -u64 disk_bytenr;u64 num_bytes; +u64 ram_bytes; +u64 disk_bytenr;u64 disk_num_bytes; +u64 offset;/* number of bytes that still need writing */ u64 bytes_left;
@@ -160,15 +162,9 @@ int btrfs_dec_test_first_ordered_pending(struct inode *inode, u64 *file_offset, u64 io_size, int uptodate);int btrfs_add_ordered_extent (struct inode *inode,u64 file_offset, -u64 disk_bytenr,u64 num_bytes,u64 disk_num_bytes, -int type); -int btrfs_add_ordered_extent_dio (struct inode *inode,u64 file_offset, -u64 disk_bytenr,u64 num_bytes, -u64 disk_num_bytes,int type); -int btrfs_add_ordered_extent_compress (struct inode *inode,u64 file_offset, -u64 disk_bytenr,u64 num_bytes, -u64 disk_num_bytes,int type, -int compress_type); +u64 num_bytes,u64 ram_bytes,u64 disk_bytenr, +u64 disk_num_bytes,u64 offset,int flags, +int compress_type);void btrfs_add_ordered_sum (struct btrfs_ordered_extent *entry,struct btrfs_ordered_sum *sum); struct btrfs_ordered_extent *btrfs_lookup_ordered_extent(struct inode *inode,
Signature · 1 line
2.24.0
Re: [RFC PATCH v3 08/12] btrfs: add ram_bytes and offset to btrfs_ordered_extent
Quoted 8 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Currently, we only create ordered extents when
ram_bytes==num_bytesand offset == 0. However,RWF_ENCODEDwrites may create extents which only refer to a subset of the full unencoded extent, so we need to plumb these fields through the ordered extent infrastructure and pass them down toinsert_reserved_file_extent().
Quoted 4 lines
Since we're changing the btrfs_add_ordered_extent* signature, let's get rid of the trivial wrappers and add a kernel-doc.
Signed-off-by Omar Sandoval <[email protected]>
Reviewed-by Nikolay Borisov <[email protected]>
RFC PATCH v3 9/12 btrfs: support different disk extent size for delalloc
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Currently, we always reserve the same extent size in the file and extent size on disk for delalloc because the former is the worst case for the latter. For RWF_ENCODED writes, we know the exact size of the extent on disk, which may be less than or greater than (for bookends) the size in the file. Add a disk_num_bytes parameter to btrfs_delalloc_reserve_metadata() so that we can reserve the correct amount of csum bytes. Additionally, make btrfs_free_reserve_data_space_noquota() take a number of bytes instead of a range, as it refers to the extent size on disk, not in the file. No functional change.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/ctree.h | 3 ++- fs/btrfs/delalloc-space.c | 38 +++++++++++++++++--------------------- fs/btrfs/delalloc-space.h | 4 ++-- fs/btrfs/file.c | 3 ++- fs/btrfs/inode.c | 7 ++----- fs/btrfs/relocation.c | 4 ++-- 6 files changed, 27 insertions(+), 32 deletions(-)
fs/btrfs/ctree.h +2 -1
@@ -2489,7 +2489,8 @@ void btrfs_subvolume_release_metadata(struct btrfs_fs_info *fs_info, struct btrfs_block_rsv *rsv);void btrfs_delalloc_release_extents (struct btrfs_inode *inode,u64 num_bytes); -int btrfs_delalloc_reserve_metadata (struct btrfs_inode *inode,u64 num_bytes); +int btrfs_delalloc_reserve_metadata (struct btrfs_inode *inode,u64 num_bytes, +u64 disk_num_bytes);u64 btrfs_account_ro_block_groups_free_space (struct btrfs_space_info *sinfo);int btrfs_error_unpin_extent_range (struct btrfs_fs_info *fs_info,u64 start,u64 end);
fs/btrfs/delalloc-space.c +17 -21
@@ -153,34 +153,28 @@ int btrfs_check_data_free_space(struct inode *inode,/* Use new btrfs_qgroup_reserve_data to reserve precious data space. */ ret =btrfs_qgroup_reserve_data (inode, reserved, start, len);if (ret <0 ) -btrfs_free_reserved_data_space_noquota (inode, start, len); +btrfs_free_reserved_data_space_noquota (fs_info, len);else ret =0 ;return ret; }/* -* Called if we need to clear a data reservation for this inode -* Normally in a error case. +* Called if we need to clear a data reservation, normally in an error case. * * This one will *NOT* use accurate qgroup reserved space API, just for case * which we can't sleep and is sure it won't affect qgroup reserved space. * Like clear_bit_hook(). */ -void btrfs_free_reserved_data_space_noquota (struct inode *inode,u64 start, -u64 len) +void btrfs_free_reserved_data_space_noquota (struct btrfs_fs_info *fs_info, +u64 num_bytes) { -struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct btrfs_space_info *data_sinfo; -/* Make sure the range is aligned to sectorsize */ - len =round_up (start + len, fs_info->sectorsize) - -round_down (start, fs_info->sectorsize); - start =round_down (start, fs_info->sectorsize); - + num_bytes =ALIGN (num_bytes, fs_info->sectorsize); data_sinfo = fs_info->data_sinfo;spin_lock (&data_sinfo->lock); -btrfs_space_info_update_bytes_may_use (fs_info, data_sinfo, -len); +btrfs_space_info_update_bytes_may_use (fs_info, data_sinfo, -num_bytes);spin_unlock (&data_sinfo->lock); }
@@ -201,7 +195,7 @@ void btrfs_free_reserved_data_space(struct inode *inode,round_down (start, root->fs_info->sectorsize); start =round_down (start, root->fs_info->sectorsize); -btrfs_free_reserved_data_space_noquota (inode, start, len); +btrfs_free_reserved_data_space_noquota (root->fs_info, len);btrfs_qgroup_free_data (inode, reserved, start, len); }
@@ -280,11 +274,11 @@ static void btrfs_calculate_inode_block_rsv_size(struct btrfs_fs_info *fs_info, }static void calc_inode_reservations (struct btrfs_fs_info *fs_info, -u64 num_bytes,u64 *meta_reserve, -u64 *qgroup_reserve) +u64 num_bytes,u64 disk_num_bytes, +u64 *meta_reserve,u64 *qgroup_reserve) {u64 nr_extents =count_max_extents (num_bytes); -u64 csum_leaves =btrfs_csum_bytes_to_leaves (fs_info, num_bytes); +u64 csum_leaves =btrfs_csum_bytes_to_leaves (fs_info, disk_num_bytes);u64 inode_update =btrfs_calc_metadata_size (fs_info,1 ); *meta_reserve = btrfs_calc_insert_metadata_size(fs_info,
@@ -298,7 +292,8 @@ static void calc_inode_reservations(struct btrfs_fs_info *fs_info, *qgroup_reserve = nr_extents * fs_info->nodesize; } -int btrfs_delalloc_reserve_metadata (struct btrfs_inode *inode,u64 num_bytes) +int btrfs_delalloc_reserve_metadata (struct btrfs_inode *inode,u64 num_bytes, +u64 disk_num_bytes) {struct btrfs_root *root = inode->root;struct btrfs_fs_info *fs_info = root->fs_info;
@@ -333,6 +328,7 @@ int btrfs_delalloc_reserve_metadata(struct btrfs_inode *inode, u64 num_bytes)mutex_lock (&inode->delalloc_mutex); num_bytes =ALIGN (num_bytes, fs_info->sectorsize); + disk_num_bytes =ALIGN (disk_num_bytes, fs_info->sectorsize);/* * We always want to do it this way, every other way is wrong and ends
@@ -344,8 +340,8 @@ int btrfs_delalloc_reserve_metadata(struct btrfs_inode *inode, u64 num_bytes)* everything out and try again, which is bad. This way we just * over-reserve slightly, and clean up the mess when we are done. */ -calc_inode_reservations (fs_info, num_bytes, &meta_reserve, - &qgroup_reserve); +calc_inode_reservations (fs_info, num_bytes, disk_num_bytes, + &meta_reserve, &qgroup_reserve); ret =btrfs_qgroup_reserve_meta_prealloc (root, qgroup_reserve, true);if (ret) goto out_fail;
@@ -362,7 +358,7 @@ int btrfs_delalloc_reserve_metadata(struct btrfs_inode *inode, u64 num_bytes)spin_lock (&inode->lock); nr_extents =count_max_extents (num_bytes);btrfs_mod_outstanding_extents (inode, nr_extents); - inode->csum_bytes += num_bytes; + inode->csum_bytes += disk_num_bytes;btrfs_calculate_inode_block_rsv_size (fs_info, inode);spin_unlock (&inode->lock);
@@ -474,7 +470,7 @@ int btrfs_delalloc_reserve_space(struct inode *inode, ret =btrfs_check_data_free_space (inode, reserved, start, len);if (ret <0 )return ret; - ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), len); + ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), len, len);if (ret <0 )btrfs_free_reserved_data_space (inode, *reserved, start, len);return ret;
fs/btrfs/delalloc-space.h +2 -2
@@ -13,8 +13,8 @@ void btrfs_free_reserved_data_space(struct inode *inode,void btrfs_delalloc_release_space (struct inode *inode,struct extent_changeset *reserved,u64 start,u64 len,bool qgroup_free); -void btrfs_free_reserved_data_space_noquota (struct inode *inode,u64 start, -u64 len); +void btrfs_free_reserved_data_space_noquota (struct btrfs_fs_info *fs_info, +u64 num_bytes);void btrfs_delalloc_release_metadata (struct btrfs_inode *inode,u64 num_bytes,bool qgroup_free); int btrfs_delalloc_reserve_space(struct inode *inode,
fs/btrfs/file.c +2 -1
@@ -1669,7 +1669,8 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb,WARN_ON (reserve_bytes ==0 ); ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), - reserve_bytes); + reserve_bytes, + reserve_bytes);if (ret) {if (!only_release_metadata) btrfs_free_reserved_data_space(inode,
fs/btrfs/inode.c +2 -5
@@ -1963,9 +1963,7 @@ void btrfs_clear_delalloc_extent(struct inode *vfs_inode,if (root->root_key.objectid !=BTRFS_DATA_RELOC_TREE_OBJECTID && do_list && !(state->state &EXTENT_NORESERVE ) && (*bits &EXTENT_CLEAR_DATA_RESV )) -btrfs_free_reserved_data_space_noquota ( - &inode->vfs_inode, - state->start, len); +btrfs_free_reserved_data_space_noquota (fs_info, len);percpu_counter_add_batch (&fs_info->delalloc_bytes, -len, fs_info->delalloc_batch);
@@ -7025,8 +7023,7 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map,* use the existing or preallocated extent, so does not * need to adjust btrfs_space_info's bytes_may_use. */ -btrfs_free_reserved_data_space_noquota (inode, start, - len); +btrfs_free_reserved_data_space_noquota (fs_info, len); goto skip_cow; } }
fs/btrfs/relocation.c +2 -2
@@ -3262,8 +3262,8 @@ static int relocate_file_extent_cluster(struct inode *inode, index = (cluster->start - offset) >>PAGE_SHIFT ; last_index = (cluster->end - offset) >>PAGE_SHIFT ;while (index <= last_index) { - ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), -PAGE_SIZE ); + ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode),PAGE_SIZE , +PAGE_SIZE );if (ret) goto out;
Signature · 1 line
2.24.0
Re: [RFC PATCH v3 09/12] btrfs: support different disk extent size for delalloc
Quoted 13 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Currently, we always reserve the same extent size in the file and extent size on disk for delalloc because the former is the worst case for the latter. For
RWF_ENCODEDwrites, we know the exact size of the extent on disk, which may be less than or greater than (for bookends) the size in the file. Add adisk_num_bytesparameter tobtrfs_delalloc_reserve_metadata()so that we can reserve the correct amount of csum bytes. Additionally, makebtrfs_free_reserve_data_space_noquota()take a number of bytes instead of a range, as it refers to the extent size on disk, not in the file. No functional change.
Quoted 1 line
Signed-off-by Omar Sandoval <[email protected]>
Reviewed-by Nikolay Borisov <[email protected]>
RFC PATCH v3 10/12 btrfs: optionally extend i_size in cow_file_range_inline()
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Currently, an inline extent is always created after i_size is extended from btrfs_dirty_pages(). However, for encoded writes, we only want to update i_size after we successfully created the inline extent. Add an update_i_size parameter to cow_file_range_inline() and insert_inline_extent() and pass in the size of the extent rather than determining it from i_size. Since the start parameter is always passed as 0, get rid of it and simplify the logic in these two functions. While we're here, let's document the requirements for creating an inline extent.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/inode.c | 94 +++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 50 deletions(-)
fs/btrfs/inode.c +44 -50
@@ -165,9 +165,10 @@ static int btrfs_init_inode_security(struct btrfs_trans_handle *trans,static int insert_inline_extent (struct btrfs_trans_handle *trans,struct btrfs_path *path,int extent_inserted,struct btrfs_root *root,struct inode *inode, -u64 start,size_t size,size_t compressed_size, +size_t size,size_t compressed_size,int compress_type, -struct page **compressed_pages) +struct page **compressed_pages, +bool update_i_size) {struct extent_buffer *leaf;struct page *page =NULL ;
@@ -176,7 +177,7 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans,struct btrfs_file_extent_item *ei;int ret;size_t cur_size = size; -unsigned long offset; +u64 i_size;ASSERT ((compressed_size >0 && compressed_pages) || (compressed_size ==0 && !compressed_pages));
@@ -191,7 +192,7 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans,size_t datasize; key.objectid =btrfs_ino (BTRFS_I (inode)); - key.offset = start; + key.offset =0 ; key.type =BTRFS_EXTENT_DATA_KEY ; datasize =btrfs_file_extent_calc_inline_size (cur_size);
@@ -230,12 +231,10 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans,btrfs_set_file_extent_compression (leaf, ei, compress_type); }else { - page =find_get_page (inode->i_mapping, - start >>PAGE_SHIFT ); + page =find_get_page (inode->i_mapping,0 );btrfs_set_file_extent_compression (leaf, ei,0 ); kaddr =kmap_atomic (page); - offset =offset_in_page (start); -write_extent_buffer (leaf, kaddr + offset, ptr, size); +write_extent_buffer (leaf, kaddr, ptr, size);kunmap_atomic (kaddr);put_page (page); }
@@ -251,7 +250,12 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans,* before we unlock the pages. Otherwise we * could end up racing with unlink. */ -BTRFS_I (inode)->disk_i_size = inode->i_size; + i_size =i_size_read (inode); +if (update_i_size && size > i_size) { +i_size_write (inode, size); + i_size = size; + } +BTRFS_I (inode)->disk_i_size = i_size; ret =btrfs_update_inode (trans, root, inode); fail:
@@ -264,36 +268,31 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans,* does the checks required to make sure the data is small enough * to fit as an inline extent. */ -static noinline int cow_file_range_inline(struct inode *inode, u64 start, -u64 end,size_t compressed_size, +static noinline int cow_file_range_inline(struct inode *inode, u64 size, +size_t compressed_size,int compress_type, -struct page **compressed_pages) +struct page **compressed_pages, +bool update_i_size) {struct btrfs_root *root =BTRFS_I (inode)->root;struct btrfs_fs_info *fs_info = root->fs_info;struct btrfs_trans_handle *trans; -u64 isize =i_size_read (inode); -u64 actual_end =min (end +1 , isize); -u64 inline_len = actual_end - start; -u64 aligned_end =ALIGN (end, fs_info->sectorsize); -u64 data_len = inline_len; +u64 data_len = compressed_size ? compressed_size : size;int ret;struct btrfs_path *path;int extent_inserted =0 ;u32 extent_item_size; -if (compressed_size) - data_len = compressed_size; - -if (start >0 || - actual_end > fs_info->sectorsize || +/* +* We can create an inline extent if it ends at or beyond the current +* i_size, is no larger than a sector (decompressed), and the (possibly +* compressed) data fits in a leaf and the configured maximum inline +* size. +*/ +if (size <i_size_read (inode) || size > fs_info->sectorsize || data_len >BTRFS_MAX_INLINE_DATA_SIZE (fs_info) || - (!compressed_size && - (actual_end & (fs_info->sectorsize -1 )) ==0 ) || - end +1 < isize || - data_len > fs_info->max_inline) { + data_len > fs_info->max_inline)return 1 ; - } path =btrfs_alloc_path ();if (!path)
@@ -306,27 +305,18 @@ static noinline int cow_file_range_inline(struct inode *inode, u64 start, } trans->block_rsv = &BTRFS_I (inode)->block_rsv; -if (compressed_size && compressed_pages) - extent_item_size =btrfs_file_extent_calc_inline_size ( - compressed_size); -else - extent_item_size =btrfs_file_extent_calc_inline_size ( - inline_len); - - ret =__btrfs_drop_extents (trans, root, inode, path, - start, aligned_end,NULL , -1 ,1 , extent_item_size, &extent_inserted); + extent_item_size =btrfs_file_extent_calc_inline_size (data_len); + ret =__btrfs_drop_extents (trans, root, inode, path,0 , + fs_info->sectorsize,NULL ,1 ,1 , + extent_item_size, &extent_inserted);if (ret) {btrfs_abort_transaction (trans, ret); goto out; } -if (isize > actual_end) - inline_len =min_t (u64, isize, actual_end); - ret =insert_inline_extent (trans, path, extent_inserted, - root, inode, start, - inline_len, compressed_size, - compress_type, compressed_pages); + ret =insert_inline_extent (trans, path, extent_inserted, root, inode, + size, compressed_size, compress_type, + compressed_pages, update_i_size);if (ret && ret != -ENOSPC ) {btrfs_abort_transaction (trans, ret); goto out;
@@ -336,7 +326,7 @@ static noinline int cow_file_range_inline(struct inode *inode, u64 start, }set_bit (BTRFS_INODE_NEEDS_FULL_SYNC , &BTRFS_I (inode)->runtime_flags); -btrfs_drop_extent_cache (BTRFS_I (inode), start, aligned_end -1 ,0 ); +btrfs_drop_extent_cache (BTRFS_I (inode),0 , fs_info->sectorsize -1 ,0 ); out:/* * Don't forget to free the reserved space, as for inlined extent
@@ -605,13 +595,15 @@ static noinline int compress_file_range(struct async_chunk *async_chunk)/* we didn't compress the entire range, try * to make an uncompressed inline extent. */ - ret =cow_file_range_inline (inode, start, end,0 , -BTRFS_COMPRESS_NONE ,NULL ); + ret =cow_file_range_inline (inode, actual_end,0 , +BTRFS_COMPRESS_NONE ,NULL , + false); }else {/* try making a compressed inline extent */ - ret =cow_file_range_inline (inode, start, end, + ret =cow_file_range_inline (inode, actual_end, total_compressed, - compress_type, pages); + compress_type, pages, + false); }if (ret <=0 ) { unsigned long clear_flags = EXTENT_DELALLOC |
@@ -991,9 +983,11 @@ static noinline int cow_file_range(struct inode *inode,inode_should_defrag (BTRFS_I (inode), start, end, num_bytes,SZ_64K );if (start ==0 ) { +u64 actual_end =min_t (u64,i_size_read (inode), end +1 ); +/* lets try to make an inline extent */ - ret =cow_file_range_inline (inode, start, end,0 , -BTRFS_COMPRESS_NONE ,NULL ); + ret =cow_file_range_inline (inode, actual_end,0 , +BTRFS_COMPRESS_NONE ,NULL , false);if (ret ==0 ) {/* * We use DO_ACCOUNTING here because we need the
Signature · 1 line
2.24.0
RFC PATCH v3 11/12 btrfs: implement RWF_ENCODED reads
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
There are 4 main cases:
1. Inline extents: we copy the data straight out of the extent buffer.
2. Hole/preallocated extents: we indicate the size of the extent starting from the read position; we don't need to copy zeroes.
3. Regular, uncompressed extents: we read the sectors we need directly from disk.
4. Regular, compressed extents: we read the entire compressed extent from disk and indicate what subset of the decompressed extent is in the file.
This initial implementation simplifies a few things that can be improved in the future:
- We hold the inode lock during the operation.
- Cases 1, 3, and 4 allocate temporary memory to read into before copying out to userspace.
- Cases 3 and 4 do not implement repair yet.
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/ctree.h | 2 + fs/btrfs/file.c | 12 +- fs/btrfs/inode.c | 454 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 467 insertions(+), 1 deletion(-)
fs/btrfs/ctree.h +2 -0
@@ -2901,6 +2901,8 @@ int btrfs_run_delalloc_range(struct inode *inode, struct page *locked_page,int btrfs_writepage_cow_fixup (struct page *page,u64 start,u64 end);void btrfs_writepage_endio_finish_ordered (struct page *page,u64 start,u64 end,int uptodate); +ssize_t btrfs_encoded_read (struct kiocb *iocb,struct iov_iter *iter); +extern const struct dentry_operations btrfs_dentry_operations;/* ioctl.c */
fs/btrfs/file.c +11 -1
@@ -390,6 +390,16 @@ int btrfs_run_defrag_inodes(struct btrfs_fs_info *fs_info)return 0 ; } +static ssize_t btrfs_file_read_iter (struct kiocb *iocb,struct iov_iter *iter) +{ +if (iocb->ki_flags &IOCB_ENCODED ) { +if (iocb->ki_flags &IOCB_NOWAIT ) +return -EOPNOTSUPP ; +return btrfs_encoded_read (iocb, iter); + } +return generic_file_read_iter (iocb, iter); +} +/* simple helper to fault in pages and copy. This should go away * and be replaced with calls into generic code. */
@@ -3455,7 +3465,7 @@ static int btrfs_file_open(struct inode *inode, struct file *filp)const struct file_operations btrfs_file_operations = { .llseek = btrfs_file_llseek, - .read_iter = generic_file_read_iter, + .read_iter = btrfs_file_read_iter, .splice_read = generic_file_splice_read, .write_iter = btrfs_file_write_iter, .mmap = btrfs_file_mmap,
fs/btrfs/inode.c +454 -0
460 lines not on this page · raw message
Signature · 1 line
2.24.0
RFC PATCH v3 12/12 btrfs: implement RWF_ENCODED writes
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
The implementation resembles direct I/O: we have to flush any ordered extents, invalidate the page cache, and do the io tree/delalloc/extent map/ordered extent dance. From there, we can reuse the compression code with a minor modification to distinguish the write from writeback. This also creates inline extents when possible.
Now that read and write are implemented, this also sets the FMODE_ENCODED_IO flag in btrfs_file_open().
Signed-off-by Omar Sandoval <[email protected]>
--- fs/btrfs/compression.c | 6 +- fs/btrfs/compression.h | 5 +- fs/btrfs/ctree.h | 2 + fs/btrfs/file.c | 40 +++++-- fs/btrfs/inode.c | 243 +++++++++++++++++++++++++++++++++++++++- fs/btrfs/ordered-data.c | 12 +- fs/btrfs/ordered-data.h | 2 + 7 files changed, 293 insertions(+), 17 deletions(-)
fs/btrfs/compression.c +4 -2
@@ -276,7 +276,8 @@ static void end_compressed_bio_write(struct bio *bio) bio->bi_status == BLK_STS_OK); cb->compressed_pages[0 ]->mapping =NULL ; -end_compressed_writeback (inode, cb); +if (cb->writeback) +end_compressed_writeback (inode, cb);/* note, our inode could be gone now */ /*
@@ -311,7 +312,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,unsigned long compressed_len, struct page **compressed_pages, unsigned long nr_pages, - unsigned int write_flags) + unsigned int write_flags, bool writeback) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);struct bio *bio =NULL ;
@@ -336,6 +337,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start, cb->mirror_num =0 ; cb->compressed_pages = compressed_pages; cb->compressed_len = compressed_len; + cb->writeback = writeback; cb->orig_bio =NULL ; cb->nr_pages = nr_pages;
fs/btrfs/compression.h +4 -1
@@ -47,6 +47,9 @@ struct compressed_bio {/* the compression algorithm for this bio */ int compress_type; +/* Whether this is a write for writeback. */ +bool writeback; +/* number of compressed pages in the array */ unsigned long nr_pages;
@@ -93,7 +96,7 @@ blk_status_t btrfs_submit_compressed_write(struct inode *inode, u64 start,unsigned long compressed_len, struct page **compressed_pages, unsigned long nr_pages, - unsigned int write_flags); + unsigned int write_flags, bool writeback);blk_status_t btrfs_submit_compressed_read (struct inode *inode,struct bio *bio,int mirror_num,unsigned long bio_flags);
fs/btrfs/ctree.h +2 -0
@@ -2902,6 +2902,8 @@ int btrfs_writepage_cow_fixup(struct page *page, u64 start, u64 end);void btrfs_writepage_endio_finish_ordered (struct page *page,u64 start,u64 end,int uptodate);ssize_t btrfs_encoded_read (struct kiocb *iocb,struct iov_iter *iter); +ssize_t btrfs_encoded_write (struct kiocb *iocb,struct iov_iter *from, +struct encoded_iov *encoded);extern const struct dentry_operations btrfs_dentry_operations;
fs/btrfs/file.c +31 -9
@@ -1893,8 +1893,7 @@ static void update_time_for_write(struct inode *inode)inode_inc_iversion (inode); } -static ssize_t btrfs_file_write_iter (struct kiocb *iocb, -struct iov_iter *from) +static ssize_t btrfs_file_write_iter (struct kiocb *iocb,struct iov_iter *from) {struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);
@@ -1904,30 +1903,51 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,u64 end_pos;ssize_t num_written =0 ;const bool sync = iocb->ki_flags &IOCB_DSYNC ; +struct encoded_iov encoded;ssize_t err;loff_t pos;size_t count;loff_t oldsize;int clean_page =0 ; -if (!(iocb->ki_flags &IOCB_DIRECT ) && - (iocb->ki_flags &IOCB_NOWAIT )) +if ((iocb->ki_flags &IOCB_NOWAIT ) && + (!(iocb->ki_flags &IOCB_DIRECT ) || + (iocb->ki_flags &IOCB_ENCODED )))return -EOPNOTSUPP ; +if (iocb->ki_flags &IOCB_ENCODED ) { + err =copy_encoded_iov_from_iter (&encoded, from); +if (err) +return err; + } +if (!inode_trylock (inode)) {if (iocb->ki_flags &IOCB_NOWAIT )return -EAGAIN ;inode_lock (inode); } - err =generic_write_checks (iocb, from); -if (err <=0 ) { +if (iocb->ki_flags &IOCB_ENCODED ) { + err =generic_encoded_write_checks (iocb, &encoded); +if (err) { +inode_unlock (inode); +return err; + } + count = encoded.len; + }else { + err =generic_write_checks (iocb, from); +if (err <0 ) { +inode_unlock (inode); +return err; + } + count =iov_iter_count (from); + } +if (count ==0 ) {inode_unlock (inode);return err; } pos = iocb->ki_pos; - count =iov_iter_count (from);if (iocb->ki_flags &IOCB_NOWAIT ) {/* * We will allocate space in case nodatacow is not set,
@@ -1986,7 +2006,9 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,if (sync)atomic_inc (&BTRFS_I (inode)->sync_writers); -if (iocb->ki_flags &IOCB_DIRECT ) { +if (iocb->ki_flags &IOCB_ENCODED ) { + num_written =btrfs_encoded_write (iocb, from, &encoded); + }else if (iocb->ki_flags &IOCB_DIRECT ) { num_written =__btrfs_direct_write (iocb, from); }else { num_written =btrfs_buffered_write (iocb, from);
@@ -3459,7 +3481,7 @@ static loff_t btrfs_file_llseek(struct file *file, loff_t offset, int whence)static int btrfs_file_open (struct inode *inode,struct file *filp) { - filp->f_mode |=FMODE_NOWAIT ; + filp->f_mode |=FMODE_NOWAIT |FMODE_ENCODED_IO ;return generic_file_open (inode, filp); }
fs/btrfs/inode.c +241 -2
261 lines not on this page · raw message
fs/btrfs/ordered-data.c +9 -3
@@ -451,9 +451,15 @@ void btrfs_remove_ordered_extent(struct inode *inode,spin_lock (&btrfs_inode->lock);btrfs_mod_outstanding_extents (btrfs_inode,-1 );spin_unlock (&btrfs_inode->lock); -if (root != fs_info->tree_root) -btrfs_delalloc_release_metadata (btrfs_inode, entry->num_bytes, - false); +if (root != fs_info->tree_root) { +u64 release; + +if (test_bit (BTRFS_ORDERED_ENCODED , &entry->flags)) + release = entry->disk_num_bytes; +else + release = entry->num_bytes; +btrfs_delalloc_release_metadata (btrfs_inode, release, false); + }if (test_bit (BTRFS_ORDERED_DIRECT , &entry->flags)) percpu_counter_add_batch(&fs_info->dio_bytes, -entry->num_bytes,
fs/btrfs/ordered-data.h +2 -0
@@ -61,6 +61,8 @@ enum {BTRFS_ORDERED_TRUNCATED ,/* Regular IO for COW */ BTRFS_ORDERED_REGULAR , +/* RWF_ENCODED I/O */ + BTRFS_ORDERED_ENCODED, }; struct btrfs_ordered_extent {
Signature · 1 line
2.24.0
Quoted 81 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Hello,
This series adds an API for reading compressed data on a filesystem without decompressing it as well as support for writing compressed data directly to the filesystem. As with the previous submissions, I've included a man page patch describing the API, and test cases and example programs are available [1].
This version reworks the VFS interface to be backward and forward compatible and support for writing inline and bookend extents to the Btrfs implementation.
Patches 1-3 add the VFS support. Patches 4-7 are Btrfs cleanups necessary for the encoded I/O support that can go in independently of this series. Patches 8-10 are Btrfs prep patches. Patch 11 adds Btrfs encoded read support and patch 12 adds Btrfs encoded write support.
A few TODOs remain:
- Once we've settled on the interface, I'll add
RWF_ENCODEDsupport to fsstress and friends and send up the xfstests patches in [1].
-btrfs_encoded_read()still doesn't implement repair.Changes from v2 [2]:
- Rebase on v5.4-rc8
- Add patch 1 introducingcopy_struct_from_iter()as suggested by Aleksa
- RenameO_ENCODEDtoO_ALLOW_ENCODEDas suggested by Amir
- Add arch-specific definitions ofO_ALLOW_ENCODEDfor alpha, parisc, and sparc
- Rework the VFS interface to be backward and forward compatible
- Document the VFS interface as requested by Dave
- Use__aligned_u64for structencoded_iovas noted by Aleksa
- Fix len/unencoded_len mixup inmm/filemap.cas noted by Nikolay
- Add support for writing inline and bookend extents to Btrfs
- Use ENOBUFS for "buffers not big enough for encoded extent" case and E2BIG for "encoded_iovhas unsupported fields" casePlease share any comments on the API or implementation. Thanks!
1: https://github.com/osandov/xfstests/tree/rwf-encoded
2: https://lore.kernel.org/linux-btrfs/[email protected]/Omar Sandoval (12):
iov_iter: addcopy_struct_from_iter()
fs: addO_ALLOW_ENCODEDopen flag
fs: addRWF_ENCODEDfor reading/writing compressed data
btrfs: get rid of trivial__btrfs_lookup_bio_sums()wrappers
btrfs: don't advance offset for compressed bios in
btrfs_csum_one_bio()
btrfs: remove dead snapshot-aware defrag code
btrfs: makebtrfs_ordered_extentnaming consistent with
btrfs_file_extent_item
btrfs: addram_bytesand offset tobtrfs_ordered_extent
btrfs: support different disk extent size for delalloc
btrfs: optionally extendi_sizeincow_file_range_inline()
btrfs: implementRWF_ENCODEDreads
btrfs: implementRWF_ENCODEDwritesDocumentation/filesystems/encoded_io.rst | 79 + Documentation/filesystems/index.rst | 1 + arch/alpha/include/uapi/asm/fcntl.h | 1 + arch/parisc/include/uapi/asm/fcntl.h | 1 + arch/sparc/include/uapi/asm/fcntl.h | 1 + fs/btrfs/compression.c | 15 +- fs/btrfs/compression.h | 5 +- fs/btrfs/ctree.h | 13 +- fs/btrfs/delalloc-space.c | 38 +- fs/btrfs/delalloc-space.h | 4 +- fs/btrfs/file-item.c | 54 +- fs/btrfs/file.c | 61 +- fs/btrfs/inode.c | 2463 +++++++++++----------- fs/btrfs/ordered-data.c | 106 +- fs/btrfs/ordered-data.h | 28 +- fs/btrfs/relocation.c | 9 +- fs/fcntl.c | 10 +- fs/namei.c | 4 + include/linux/fcntl.h | 2 +- include/linux/fs.h | 16 +
Quoted 8 lines
include/linux/uio.h | 2 + include/trace/events/btrfs.h | 6 +- include/uapi/asm-generic/fcntl.h | 4 + include/uapi/linux/fs.h | 33 +- lib/iov_iter.c | 82 + mm/filemap.c | 165 +- 26 files changed, 1807 insertions(+), 1396 deletions(-) create mode 100644 Documentation/filesystems/encoded_io.rst
Ping. Al, would you mind taking a look at the generic bits/interface?