Btrfs for 4.13, part 1

Hi,

this is the main batch for 4.13. There are some user visible changes, see below. The core updates improve error handling (mostly related to bios), with the usual incremental work on the GFP_NOFS (mis)use removal. All patches have been in for-next for an extensive amount of time.

Thre will be followups but I want push the series (111 patches) forward. There are also some updates to adjacent subsystems (writeback and blocklayer), so I want to give some stable point for merging in the upcoming weeks.

User visible changes:
* quota override tunable
* improved compression thresholds
* statx support
* obsoleted mount option alloc_start

Core updates:
* bio-related updates
- faster bio cloning
- no allocation failures
- preallocated flush bios
* more kvzalloc use, memalloc_nofs protections, GFP_NOFS updates
* prep work for btree_inode removal
* dir-item validation
* qgoup fixes and updates
* cleanups
- removed unused struct members, unused code, refactoring
- argument refactoring (fs_info/root, caller -> callee sink)
- SEARCH_TREE docs

----------------------------------------------------------------
The following changes since commit 41f1830f5a7af77cf5c86359aba3cbd706687e52:

Linux 4.12-rc6 (2017-06-19 22:19:37 +0800)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-4.13-part1

for you to fetch changes up to f3f000297be88b1b75fde5027d660a8d8a44de14:

btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges (2017-06-21 20:56:14 +0200)

----------------------------------------------------------------
Anand Jain (7):
btrfs: cleanup unused qgroup trace event
btrfs: add framework to handle device flush error as a volume
btrfs: btrfs_decompress_bio() could accept compressed_bio instead
btrfs: reduce arguments for decompress_bio ops
btrfs: write_dev_flush does not return ENOMEM anymore
btrfs: remove redundant null bdev counting during flush submission
btrfs: wait part of the write_dev_flush() can be separated out

Arnd Bergmann (1):
Btrfs: work around maybe-uninitialized warning

Daichou (1):
Btrfs: remove obsolete FIXMEs in qgroup ioctls

Dan Carpenter (1):
Btrfs: remove an unused variable

David Sterba (39):
btrfs: remove unused member err from reada_extent
btrfs: remove unused member list from async_submit_bio
btrfs: remove unused members dir_path from recorded_ref
btrfs: remove unused member list from btrfs_end_io_wq
btrfs: fix bool type in btrfs_page_exists_in_range
btrfs: scrub: inline helper scrub_setup_wr_ctx
btrfs: scrub: inline helper scrub_free_wr_ctx
btrfs: scrub: simplify cleanup of wr_ctx in scrub_free_ctx
btrfs: scrub: use fs_info::sectorsize and drop it from scrub context
btrfs: scrub: embed scrub_wr_ctx into scrub context
btrfs: use generic slab for for btrfs_transaction
btrfs: replace opencoded kvzalloc with the helper
btrfs: send: use kvmalloc in iterate_dir_item
btrfs: scrub: add memalloc_nofs protection around init_ipath
btrfs: use GFP_KERNEL in init_ipath
btrfs: adjust includes after vmalloc removal
btrfs: add memalloc_nofs protections around alloc_workspace callback
btrfs: switch kmallocs to GFP_KERNEL in lzo/zlib alloc_workspace
btrfs: switch to kvmalloc and GFP_KERNEL in lzo/zlib alloc_workspace
btrfs: bioset allocations will never fail, adapt our helpers
btrfs: btrfs_bio_alloc never fails, skip error handling
btrfs: btrfs_bio_clone never fails, skip error handling
btrfs: btrfs_io_bio_alloc never fails, skip error handling
btrfs: sink gfp parameter to btrfs_bio_clone
btrfs: remove redundant parameters from btrfs_bio_alloc
btrfs: opencode trivial compressed_bio_alloc, simplify error handling
btrfs: pass bytes to btrfs_bio_alloc
btrfs: document mandatory order of bio in btrfs_io_bio
btrfs: add helper to initialize the non-bio part of btrfs_io_bio
btrfs: sink gfp parameter to btrfs_io_bio_alloc
btrfs: use GFP_KERNEL in btrfs_calc_avail_data_space
btrfs: use GFP_KERNEL in btrfs_init_dev_replace_tgtdev
btrfs: cleanup duplicate return value in insert_inline_extent
btrfs: move fs_info::fs_frozen to the flags
btrfs: obsolete and remove mount option alloc_start
btrfs: preallocate device flush bio
btrfs: account as waiting for IO, while waiting fot the flush bio completion
btrfs: move dev stats accounting out of wait_dev_flush
btrfs: fix validation of XATTR_ITEM dir items

Fabian Frederick (1):
btrfs: kmap() can't fail

Filipe Manana (3):
Btrfs: fix invalid extent maps due to hole punching
Btrfs: send, fix invalid path after renaming and linking file
Btrfs: incremental send, fix invalid path for unlink commands

Guoqing Jiang (1):
btrfs: simplify code with bio_io_error

Hans van Kranenburg (1):
Btrfs: btrfs_ioctl_search_key documentation

Jeff Layton (1):
btrfs: btrfs_wait_tree_block_writeback can be void return

Jeff Mahoney (3):
btrfs: cleanup root usage by btrfs_get_alloc_profile
btrfs: remove root usage from can_overcommit
btrfs: add cond_resched to btrfs_qgroup_trace_leaf_items

Josef Bacik (3):
Btrfs: replace tree->mapping with tree->private_data
btrfs: remove inode argument from repair_io_failure
Btrfs: don't pass the inode through clean_io_failure

Liu Bo (12):
Btrfs: use bio_clone_fast to clone our bio
Btrfs: new helper btrfs_bio_clone_partial
Btrfs: use bio_clone_bioset_partial to simplify DIO submit
Btrfs: change how we iterate bios in endio
Btrfs: record error if one block has failed to retry
Btrfs: check-integrity use bvec_iter
Btrfs: unify naming of btrfs_io_bio
Btrfs: hardcode GFP_NOFS for btrfs_bio_clone_partial
Btrfs: skip commit transaction if we don't have enough pinned bytes
Btrfs: let btrfs_print_leaf print more about block group
Btrfs: tolerate errors if we have retried successfully
Btrfs: skip checksum verification if IO error occurs

Nikolay Borisov (8):
btrfs: Convert fs_info->free_chunk_space to atomic64_t
btrfs: Separate space_info create/update
btrfs: Refactor update_space_info
btrfs: rename btrfs_leaf_data to BTRFS_LEAF_DATA_OFFSET
btrfs: remove __BTRFS_LEAF_DATA_SIZE
btrfs: Use btrfs_space_info_used instead of opencoding it
btrfs: Manually implement device_total_bytes getter/setter
btrfs: Round down values which are written for total_bytes_size

Omar Sandoval (7):
Btrfs: use memalloc_nofs and kvzalloc() for free space tree bitmaps
Btrfs: make add_pinned_bytes() take an s64 num_bytes instead of u64
Btrfs: make BUG_ON() in add_pinned_bytes() an ASSERT()
Btrfs: update total_bytes_pinned when pinning down extents
Btrfs: always account pinned bytes when dropping a tree block ref
Btrfs: return old and new total ref mods when adding delayed refs
Btrfs: rework delayed ref total_bytes_pinned accounting

Qu Wenruo (6):
btrfs: qgroup: Add quick exit for non-fs extents
btrfs: qgroup: Cleanup btrfs_qgroup_prepare_account_extents function
btrfs: qgroup: Return actually freed bytes for qgroup release or free data
btrfs: qgroup: Fix qgroup reserved space underflow caused by buffered write and quotas being enabled
btrfs: qgroup: Introduce extent changeset for qgroup reserve functions
btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges

Sahil Kang (1):
btrfs: Remove unnecessary branching in free-space-tree.c

Sargun Dhillon (2):
btrfs: add quota override flag to enable quota override for CAP_SYS_RESOURCE
btrfs: Add quota_override knob into sysfs

Su Yue (9):
btrfs: Introduce btrfs_is_name_len_valid to avoid reading beyond boundary
btrfs: Check name_len with boundary in verify dir_item
btrfs: Check name_len on add_inode_ref call path
btrfs: Verify dir_item in replay_xattr_deletes
btrfs: Check name_len in btrfs_check_ref_name_override
btrfs: Check name_len before read in iterate_dir_item
btrfs: Check name_len before reading btrfs_get_name
btrfs: Check name_len before in btrfs_del_root_ref
btrfs: Verify dir_item in iterate_object_props

Timofey Titovets (3):
Btrfs: lzo: fix typo in error message after failed deflate
Btrfs: lzo: compressed data size must be less then input size
Btrfs: compression must free at least one sector size

Yonghong Song (1):
Btrfs: add statx support

 fs/btrfs/backref.c               |  10 +-
 fs/btrfs/check-integrity.c       |  53 ++---
 fs/btrfs/compression.c           |  94 ++------
 fs/btrfs/compression.h           |  44 +++-
 fs/btrfs/ctree.c                 |  42 ++--
 fs/btrfs/ctree.h                 |  84 ++++---
 fs/btrfs/delayed-ref.c           |  29 ++-
 fs/btrfs/delayed-ref.h           |   6 +-
 fs/btrfs/dir-item.c              |  94 +++++++-
 fs/btrfs/disk-io.c               | 179 +++++++--------
 fs/btrfs/disk-io.h               |   8 +-
 fs/btrfs/export.c                |   5 +
 fs/btrfs/extent-tree.c           | 481 +++++++++++++++++++++------------------
 fs/btrfs/extent_io.c             | 217 ++++++++----------
 fs/btrfs/extent_io.h             |  82 +++++--
 fs/btrfs/file-item.c             |  31 ++-
 fs/btrfs/file.c                  |  46 ++--
 fs/btrfs/free-space-tree.c       |  38 ++--
 fs/btrfs/inode-map.c             |   4 +-
 fs/btrfs/inode.c                 | 449 ++++++++++++++++++++----------------
 fs/btrfs/ioctl.c                 |  16 +-
 fs/btrfs/lzo.c                   |  33 +--
 fs/btrfs/print-tree.c            |   7 +-
 fs/btrfs/props.c                 |   7 +
 fs/btrfs/qgroup.c                | 223 +++++++++++++-----
 fs/btrfs/qgroup.h                |   9 +-
 fs/btrfs/raid56.c                |  16 +-
 fs/btrfs/reada.c                 |   1 -
 fs/btrfs/relocation.c            |  15 +-
 fs/btrfs/root-tree.c             |   7 +
 fs/btrfs/scrub.c                 | 209 +++++++----------
 fs/btrfs/send.c                  | 112 ++++++---
 fs/btrfs/super.c                 |  72 +-----
 fs/btrfs/sysfs.c                 |  41 ++++
 fs/btrfs/tests/extent-io-tests.c |   2 +-
 fs/btrfs/transaction.c           |  23 +-
 fs/btrfs/tree-log.c              |  44 +++-
 fs/btrfs/volumes.c               |  74 +++---
 fs/btrfs/volumes.h               |   7 +
 fs/btrfs/xattr.c                 |   2 +-
 fs/btrfs/zlib.c                  |  20 +-
 include/trace/events/btrfs.h     |  36 ---
 include/uapi/linux/btrfs.h       |  63 +++--
 43 files changed, 1682 insertions(+), 1353 deletions(-)

Sorry for bikeshedding.

<..SNIP..>

Hmm..add those check might be expensive for metadata operations, especially in hot path, i could see similar behavior in Ext4 for ext4 dentry check.

Could we run some metadata tests to confirm？it makes sense to add check but with min affects to performace.

Thanks,
Shilong

Agreed. XFS does this at read/write and results in invalid entries never even making it to the consumer. That's the approach I had partially written up. This approach has the advantage of being more fine-grained so we don't end up dropping e.g. an entire node's worth of entries but is more expensive to maintain and at runtime.

-Jeff

Jeff Mahoney
SUSE Labs

IIRC there is a backtrace in one of the patches.
If needed, we can also upload the fuzzed image.
Although we don't have good enough test suite which accepts fuzzed image for kernel.

We already have tons of runtime check at chunk/super block read time.

This new check is not as comprehensive as what we did in chunk/superblock read time, but only to ensure that item with variable length doesn't cross its boundary. So performance wise it should not be a problem.

Although it's expensive to maintain, as for each structure with variable length, we need to call verification function every time.

But we can also extract the check to leaf reading time, this should reduce the effort to maintain it and make it easier to expand (or even make it optional if it really affects performance).

Thanks,
Qu

Well, ideally, we'd do complete checks for any type that we have enough information to check at read or write time. If a check is important enough to be done at a consumer site, it's important enough to be done at read/write. Then we can make the consumer code much simpler and catch some forms of corruption before it hits the disk.

-Jeff

Jeff Mahoney
SUSE Labs

As we have the fuzzed images in progs, I started prototyping a way to export just the testsuite from the git repository, to a tarball. The test would then iterate over the stored images and attempt to mount them. So any images you'd like mount-tested, also submit to btrfs-progs.

No problem, I won't read the rest of the mail then. (IOW, don't write such things if you want people to read your mail.)

The checks are really simple, comparing a few numbers, and performed in places that do not seem to be performance critical. The validation is supposed to happen after the blocks are read from the disk (or using the interface that does that, the page might be actually cached).

We can run such tests, I expect that the effect won't be measurable.

On Fri, Jun 23, 2017 at 05:16:46PM +0200, David Sterba wrote:

Two more patches added to the branch

Chris Mason (1):
btrfs: fix integer overflow in calc_reclaim_items_nr

David Sterba (1):
btrfs: scrub: fix target device intialization while setting up scrub context

Updated branch and tag:

----------------------------------------------------------------
The following changes since commit 41f1830f5a7af77cf5c86359aba3cbd706687e52:

Linux 4.12-rc6 (2017-06-19 22:19:37 +0800)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-4.13-part1

for you to fetch changes up to 8399f53f0c7450ab050b1b0ffee4e2c1ddd2a3e0:

btrfs: fix integer overflow in calc_reclaim_items_nr (2017-06-26 15:33:42 +0200)
----------------------------------------------------------------

Previous:

Thanks Dave, I ran this (along with the updates we added) through a long stress and the usual xfstests.

For everyone else on the list, since I'm heading off to vacation until ~July 9th, Dave is sending this off to Linus once the merge window starts.

-chris

Have any of you run xfstests without failures?

I'm finding it hard to believe, as some of these changes cause problems all the time, and at least one of them deterministically ("btrfs: fix validation of XATTR_ITEM dir items"), as I've just noted in the respective thread for that patch.

I haven't traced which change is causing a BUG_ON in the raid56 code, due to lack of time, but running btrfs tests from 060 to 074 triggers it very easily, for example:

[ 6086.912567] run fstests btrfs/064 at 2017-06-26 21:10:55
[ 6089.821472] BTRFS: device fsid 26f67507-c01d-4101-8435-2299cddab311
devid 1 transid 5 /dev/sdc
[ 6089.913286] BTRFS: device fsid 26f67507-c01d-4101-8435-2299cddab311
devid 2 transid 5 /dev/sdd
[ 6089.966903] BTRFS: device fsid 26f67507-c01d-4101-8435-2299cddab311
devid 3 transid 5 /dev/sde
[ 6090.065205] BTRFS: device fsid 26f67507-c01d-4101-8435-2299cddab311
devid 4 transid 5 /dev/sdf
[ 6090.873020] BTRFS info (device sdf): turning on discard
[ 6090.874452] BTRFS info (device sdf): disk space caching is enabled
[ 6090.893146] BTRFS info (device sdf): has skinny extents
[ 6091.110544] BTRFS info (device sdf): creating UUID tree
[ 6093.459619] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg started
[ 6099.046000] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg finished
[ 6101.221892] BTRFS info (device sdf): dev_replace from /dev/sde
(devid 3) to /dev/sdd started
[ 6104.224275] BTRFS info (device sdf): dev_replace from /dev/sde
(devid 3) to /dev/sdd finished
[ 6104.610774] BTRFS info (device sdf): dev_replace from /dev/sdf
(devid 4) to /dev/sde started
[ 6105.621356] BTRFS info (device sdf): dev_replace from /dev/sdf
(devid 4) to /dev/sde finished
[ 6111.693426] BTRFS info (device sdf): relocating block group
3242196992 flags data
[ 6111.731322] BTRFS info (device sdf): found 1 extents
[ 6111.762907] BTRFS info (device sdf): found 1 extents
[ 6111.785554] BTRFS info (device sdf): relocating block group
2168455168 flags metadata
[ 6111.823810] BTRFS info (device sdf): relocating block group
1094713344 flags data
[ 6111.974995] BTRFS info (device sdf): found 209 extents
[ 6112.471755] BTRFS info (device sdf): found 209 extents
[ 6112.485584] BTRFS info (device sdf): relocating block group
20971520 flags metadata
[ 6112.527924] BTRFS info (device sdf): found 2 extents
[ 6112.535532] BTRFS info (device sdf): relocating block group
12582912 flags data
[ 6112.585868] BTRFS info (device sdf): found 92 extents
[ 6112.838224] BTRFS info (device sdf): found 92 extents
[ 6112.847285] BTRFS info (device sdf): relocating block group 4194304
flags metadata
[ 6112.963176] BTRFS info (device sdf): found 116 extents
[ 6112.986034] BTRFS info (device sdf): relocating block group 0 flags system
[ 6113.013839] BTRFS info (device sdf): found 1 extents
[ 6114.496170] blk_update_request: I/O error, dev fd0, sector 0
[ 6114.497343] floppy: error -5 while reading block 0
[ 6114.700151] blk_update_request: I/O error, dev fd0, sector 0
[ 6114.701047] floppy: error -5 while reading block 0
[ 6114.975828] BTRFS: device fsid 7ea53fb0-9bc6-48e7-90ed-3d75bf00c180
devid 3 transid 5 /dev/sde
[ 6115.047448] BTRFS: device fsid 7ea53fb0-9bc6-48e7-90ed-3d75bf00c180
devid 1 transid 5 /dev/sdc
[ 6115.080295] BTRFS: device fsid 7ea53fb0-9bc6-48e7-90ed-3d75bf00c180
devid 2 transid 5 /dev/sdd
[ 6115.225944] BTRFS: device fsid 7ea53fb0-9bc6-48e7-90ed-3d75bf00c180
devid 4 transid 5 /dev/sdf
[ 6115.525122] BTRFS info (device sdf): turning on discard
[ 6115.525122] BTRFS info (device sdf): disk space caching is enabled
[ 6115.541244] BTRFS info (device sdf): has skinny extents
[ 6115.653520] BTRFS info (device sdf): creating UUID tree
[ 6115.744243] systemd-journald[618]: Data hash table of
/run/log/journal/bba104ad3dc06c1bd2dc3d7c546a5b43/system.journal has a
fill level at 75.0 (26843 of 35790 items, 16777216 file size, 625
bytes per hash table item), suggesting rotation.
[ 6115.813742] systemd-journald[618]:
/run/log/journal/bba104ad3dc06c1bd2dc3d7c546a5b43/system.journal:
Journal header limits reached or header out-of-date, rotating.
[ 6115.820095] systemd-journald[618]: Rotating...
[ 6115.841292] systemd-journald[618]: Reserving 35790 entries in hash table.
[ 6115.854029] systemd-journald[618]: Vacuuming...
[ 6115.855641] systemd-journald[618]: Vacuuming done, freed 0B of
archived journals on disk.
[ 6116.428897] blk_update_request: I/O error, dev fd0, sector 0
[ 6116.432844] floppy: error -5 while reading block 0
[ 6121.422099] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg started
[ 6131.042683] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg finished
[ 6136.197456] BTRFS info (device sdf): relocating block group
9768534016 flags data|raid0
[ 6136.338019] BTRFS info (device sdf): found 4 extents
[ 6136.444110] BTRFS info (device sdf): found 4 extents
[ 6136.599240] BTRFS info (device sdf): relocating block group
8694792192 flags metadata|raid0
[ 6136.766634] BTRFS info (device sdf): found 43 extents
[ 6136.854219] BTRFS info (device sdf): relocating block group
8627683328 flags system|raid0
[ 6136.956588] BTRFS info (device sdf): relocating block group
4332716032 flags data|raid0
[ 6143.223627] BTRFS info (device sdf): found 268 extents
[ 6143.225358] systemd-journald[618]: Sent WATCHDOG=1 notification.
[ 6143.657919] BTRFS info (device sdf): found 268 extents
[ 6143.744224] BTRFS info (device sdf): relocating block group
37748736 flags metadata|raid0
[ 6143.867552] BTRFS info (device sdf): found 102 extents
[ 6143.924525] BTRFS info (device sdf): relocating block group
20971520 flags system|raid0
[ 6143.983050] BTRFS info (device sdf): found 1 extents
[ 6145.424161] blk_update_request: I/O error, dev fd0, sector 0
[ 6145.425595] floppy: error -5 while reading block 0
[ 6145.628137] blk_update_request: I/O error, dev fd0, sector 0
[ 6145.628960] floppy: error -5 while reading block 0
[ 6147.044939] BTRFS: device fsid 94157252-d25f-4c1c-9d26-8b7729d7bc5a
devid 1 transid 5 /dev/sdc
[ 6147.049400] BTRFS: device fsid 94157252-d25f-4c1c-9d26-8b7729d7bc5a
devid 2 transid 5 /dev/sdd
[ 6147.086296] BTRFS: device fsid 94157252-d25f-4c1c-9d26-8b7729d7bc5a
devid 3 transid 5 /dev/sde
[ 6147.130074] BTRFS: device fsid 94157252-d25f-4c1c-9d26-8b7729d7bc5a
devid 4 transid 5 /dev/sdf
[ 6147.853052] BTRFS info (device sdf): turning on discard
[ 6147.854682] BTRFS info (device sdf): disk space caching is enabled
[ 6147.873477] BTRFS info (device sdf): has skinny extents
[ 6148.285495] BTRFS info (device sdf): creating UUID tree
[ 6149.101071] blk_update_request: I/O error, dev fd0, sector 0
[ 6149.104974] floppy: error -5 while reading block 0
[ 6151.577159] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg started
[ 6241.821804] systemd-journald[618]: Sent WATCHDOG=1 notification.
[ 6306.238550] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg finished
[ 6320.201287] systemd-journald[618]: Sent WATCHDOG=1 notification.
[ 6334.496060] blk_update_request: I/O error, dev fd0, sector 0
[ 6334.496849] floppy: error -5 while reading block 0
[ 6368.344060] blk_update_request: I/O error, dev fd0, sector 0
[ 6368.344850] floppy: error -5 while reading block 0
[ 6405.872088] BTRFS: device fsid 3b456cae-f946-4422-8b69-3e562660d3ca
devid 1 transid 5 /dev/sdc
[ 6407.056284] BTRFS: device fsid 3b456cae-f946-4422-8b69-3e562660d3ca
devid 2 transid 5 /dev/sdd
[ 6409.036854] BTRFS: device fsid 3b456cae-f946-4422-8b69-3e562660d3ca
devid 3 transid 5 /dev/sde
[ 6409.061008] systemd-journald[618]: Sent WATCHDOG=1 notification.
[ 6410.988164] BTRFS: device fsid 3b456cae-f946-4422-8b69-3e562660d3ca
devid 4 transid 5 /dev/sdf
[ 6414.503001] BTRFS info (device sdf): turning on discard
[ 6414.542595] BTRFS info (device sdf): disk space caching is enabled
[ 6414.570512] BTRFS info (device sdf): has skinny extents
[ 6415.488940] BTRFS info (device sdf): creating UUID tree
[ 6420.048032] blk_update_request: I/O error, dev fd0, sector 0
[ 6420.051396] floppy: error -5 while reading block 0
[ 6466.781846] BTRFS info (device sdf): relocating block group
1103101952 flags data|raid1
[ 6499.163148] systemd-journald[618]: Sent WATCHDOG=1 notification.
[ 6500.988823] BTRFS info (device sdf): found 163 extents
[ 6520.228862] BTRFS info (device sdf): found 155 extents
[ 6526.150033] BTRFS info (device sdf): relocating block group
29360128 flags metadata|raid1
[ 6543.788137] BTRFS info (device sdf): found 94 extents
[ 6548.960587] BTRFS info (device sdf): relocating block group
20971520 flags system|raid1
[ 6554.786983] BTRFS info (device sdf): found 1 extents
[ 6577.584717] blk_update_request: I/O error, dev fd0, sector 0
[ 6577.588657] floppy: error -5 while reading block 0
[ 6601.128050] blk_update_request: I/O error, dev fd0, sector 0
[ 6601.128840] floppy: error -5 while reading block 0
[ 6601.140652] systemd-journald[618]: Sent WATCHDOG=1 notification.
[ 6625.542882] BTRFS: device fsid e3828911-137a-48d1-9be3-c3b7ec16f953
devid 1 transid 5 /dev/sdc
[ 6625.581187] BTRFS: device fsid e3828911-137a-48d1-9be3-c3b7ec16f953
devid 2 transid 5 /dev/sdd
[ 6625.601238] BTRFS: device fsid e3828911-137a-48d1-9be3-c3b7ec16f953
devid 3 transid 5 /dev/sde
[ 6625.723378] BTRFS: device fsid e3828911-137a-48d1-9be3-c3b7ec16f953
devid 4 transid 5 /dev/sdf
[ 6626.304067] BTRFS info (device sdf): turning on discard
[ 6626.304830] BTRFS info (device sdf): disk space caching is enabled
[ 6626.305520] BTRFS info (device sdf): has skinny extents
[ 6626.471590] BTRFS info (device sdf): creating UUID tree
[ 6628.368615] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg started
[ 6641.236372] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg finished
[ 6647.060891] BTRFS info (device sdf): relocating block group
5473566720 flags data|raid10
[ 6647.164149] BTRFS info (device sdf): found 2 extents
[ 6647.259690] BTRFS info (device sdf): found 2 extents
[ 6647.330143] BTRFS info (device sdf): relocating block group
4399824896 flags metadata|raid10
[ 6647.458031] BTRFS info (device sdf): found 34 extents
[ 6647.523545] BTRFS info (device sdf): relocating block group
4332716032 flags system|raid10
[ 6647.603603] BTRFS info (device sdf): relocating block group
2185232384 flags data|raid10
[ 6651.461293] BTRFS info (device sdf): found 245 extents
[ 6651.803813] BTRFS info (device sdf): found 245 extents
[ 6651.872335] BTRFS info (device sdf): relocating block group
37748736 flags metadata|raid10
[ 6652.067394] BTRFS info (device sdf): found 102 extents
[ 6652.124856] BTRFS info (device sdf): relocating block group
20971520 flags system|raid10
[ 6652.199854] BTRFS info (device sdf): found 1 extents
[ 6661.700121] blk_update_request: I/O error, dev fd0, sector 0
[ 6661.701526] floppy: error -5 while reading block 0
[ 6662.052252] blk_update_request: I/O error, dev fd0, sector 0
[ 6662.053296] floppy: error -5 while reading block 0
[ 6663.040606] BTRFS: device fsid fac7c5d7-9979-4280-9ced-6adf1baf673d
devid 1 transid 5 /dev/sdc
[ 6663.224627] BTRFS: device fsid fac7c5d7-9979-4280-9ced-6adf1baf673d
devid 2 transid 5 /dev/sdd
[ 6663.266865] BTRFS: device fsid fac7c5d7-9979-4280-9ced-6adf1baf673d
devid 3 transid 5 /dev/sde
[ 6663.280780] BTRFS: device fsid fac7c5d7-9979-4280-9ced-6adf1baf673d
devid 4 transid 5 /dev/sdf
[ 6663.784636] BTRFS info (device sdf): turning on discard
[ 6663.811720] BTRFS info (device sdf): disk space caching is enabled
[ 6663.824680] BTRFS info (device sdf): has skinny extents
[ 6664.005123] BTRFS info (device sdf): creating UUID tree
[ 6664.254147] btrfs_print_data_csum_error: 11 callbacks suppressed
[ 6664.292699] BTRFS warning (device sdf): csum failed root 5 ino 284
off 294912 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6664.344601] BTRFS warning (device sdf): csum failed root 5 ino 284
off 299008 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6664.376676] BTRFS warning (device sdf): csum failed root 5 ino 284
off 303104 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6664.391328] BTRFS warning (device sdf): csum failed root 5 ino 284
off 294912 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6664.401801] BTRFS warning (device sdf): csum failed root 5 ino 284
off 299008 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6664.440441] BTRFS warning (device sdf): csum failed root 5 ino 284
off 303104 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6664.540695] BTRFS warning (device sdf): csum failed root 5 ino 284
off 294912 csum 0x98f94189 expected csum 0xf2364862 mirror 2
[ 6664.542010] BTRFS warning (device sdf): csum failed root 5 ino 284
off 299008 csum 0x98f94189 expected csum 0xf2364862 mirror 2
[ 6664.543258] BTRFS warning (device sdf): csum failed root 5 ino 284
off 303104 csum 0x98f94189 expected csum 0xf2364862 mirror 2
[ 6664.672878] BTRFS warning (device sdf): csum failed root 5 ino 284
off 303104 csum 0x98f94189 expected csum 0xf2364862 mirror 1
[ 6665.232628] blk_update_request: I/O error, dev fd0, sector 0
[ 6665.233526] floppy: error -5 while reading block 0
[ 6669.256609] btrfs_print_data_csum_error: 297 callbacks suppressed
[ 6669.291821] BTRFS warning (device sdf): csum failed root 5 ino 300
off 684032 csum 0x98f94189 expected csum 0x42186b7f mirror 1
[ 6669.319412] BTRFS warning (device sdf): csum failed root 5 ino 300
off 688128 csum 0x98f94189 expected csum 0x42186b7f mirror 1
[ 6669.332644] BTRFS warning (device sdf): csum failed root 5 ino 298
off 892928 csum 0x98f94189 expected csum 0x28d76294 mirror 2
[ 6669.356673] BTRFS warning (device sdf): csum failed root 5 ino 298
off 897024 csum 0x98f94189 expected csum 0x28d76294 mirror 2
[ 6669.376583] BTRFS warning (device sdf): csum failed root 5 ino 300
off 692224 csum 0x98f94189 expected csum 0x42186b7f mirror 1
[ 6669.447063] BTRFS warning (device sdf): csum failed root 5 ino 298
off 901120 csum 0x98f94189 expected csum 0x28d76294 mirror 2
[ 6669.500703] BTRFS warning (device sdf): csum failed root 5 ino 298
off 905216 csum 0x98f94189 expected csum 0x28d76294 mirror 2
[ 6669.505183] BTRFS warning (device sdf): csum failed root 5 ino 298
off 929792 csum 0x98f94189 expected csum 0x28d76294 mirror 2
[ 6669.505238] BTRFS warning (device sdf): csum failed root 5 ino 281
off 901120 csum 0x98f94189 expected csum 0x27a85bb4 mirror 2
[ 6669.508819] BTRFS warning (device sdf): csum failed root 5 ino 300
off 696320 csum 0x98f94189 expected csum 0x42186b7f mirror 1
[ 6669.920580] BTRFS info (device sdf): dev_replace from /dev/sdd
(devid 2) to /dev/sdg started
[ 6674.297803] btrfs_print_data_csum_error: 138 callbacks suppressed
[ 6674.317501] BTRFS warning (device sdf): csum failed root 5 ino 402
off 831488 csum 0x98f94189 expected csum 0x4d67525f mirror 2
[ 6674.348626] BTRFS warning (device sdf): csum failed root 5 ino 311
off 200704 csum 0x98f94189 expected csum 0x4d67525f mirror 1
[ 6674.380668] BTRFS warning (device sdf): csum failed root 5 ino 311
off 204800 csum 0x98f94189 expected csum 0x4d67525f mirror 1
[ 6674.389074] BTRFS warning (device sdf): csum failed root 5 ino 388
off 192512 csum 0x98f94189 expected csum 0xfd497142 mirror 1
[ 6674.407222] BTRFS warning (device sdf): csum failed root 5 ino 402
off 729088 csum 0x42186b7f expected csum 0x4d67525f mirror 2
[ 6674.416111] BTRFS warning (device sdf): csum failed root 5 ino 388
off 147456 csum 0x98f94189 expected csum 0xfd497142 mirror 2
[ 6674.416156] ------------[ cut here ]------------
[ 6674.416157] kernel BUG at fs/btrfs/raid56.c:1897!
[ 6674.416159] invalid opcode: 0000 [#1] PREEMPT SMP
[ 6674.416160] Modules linked in: dm_flakey dm_mod dax ppdev tpm_tis

parport_pc tpm_tis_core evdev tpm psmouse sg i2c_piix4 pcspkr parport i2c_core serio_raw button sunrpc loop autofs4 ext4 crc16 jbd2 mbcache btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod sr_mod cdrom ata_generic sd_mod virtio_scsi ata_piix virtio_pci virtio_ring libata scsi_mod e1000 virtio floppy

[ 6674.416184] CPU: 3 PID: 19236 Comm: kworker/u32:10 Not tainted
4.12.0-rc6-btrfs-next-44+ #1
[ 6674.416185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
[ 6674.416210] Workqueue: btrfs-endio btrfs_endio_helper [btrfs]
[ 6674.416211] task: ffff880147f6c740 task.stack: ffffc90001fb8000
[ 6674.416229] RIP: 0010:__raid_recover_end_io+0x1ac/0x370 [btrfs]
[ 6674.416230] RSP: 0018:ffffc90001fbbb90 EFLAGS: 00010217
[ 6674.416231] RAX: ffff8801ff4b4f00 RBX: 0000000000000002 RCX: 0000000000000001
[ 6674.416232] RDX: ffff880099b045d8 RSI: ffffffff81a5f6e0 RDI: 0000000000000004
[ 6674.416232] RBP: ffffc90001fbbbc8 R08: 0000000000000001 R09: 0000000000000001
[ 6674.416233] R10: ffffc90001fbbac8 R11: 0000000000001000 R12: 0000000000000002
[ 6674.416234] R13: ffff880099b045c0 R14: 0000000000000004 R15: ffff88012bff2000
[ 6674.416235] FS:  0000000000000000(0000) GS:ffff88023f2c0000(0000)
knlGS:0000000000000000
[ 6674.416235] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6674.416236] CR2: 00007f28cf282000 CR3: 00000001000c6000 CR4: 00000000000006e0
[ 6674.416239] Call Trace:
[ 6674.416259]  __raid56_parity_recover+0xfc/0x16e [btrfs]
[ 6674.416276]  raid56_parity_recover+0x157/0x16b [btrfs]
[ 6674.416293]  btrfs_map_bio+0xe0/0x259 [btrfs]
[ 6674.416310]  btrfs_submit_bio_hook+0xbf/0x147 [btrfs]
[ 6674.416327]  end_bio_extent_readpage+0x27b/0x4a0 [btrfs]
[ 6674.416331]  bio_endio+0x17d/0x1b3
[ 6674.416346]  end_workqueue_fn+0x3c/0x3f [btrfs]
[ 6674.416362]  btrfs_scrubparity_helper+0x1aa/0x3b8 [btrfs]
[ 6674.416379]  btrfs_endio_helper+0xe/0x10 [btrfs]
[ 6674.416381]  process_one_work+0x276/0x4b6
[ 6674.416384]  worker_thread+0x1ac/0x266
[ 6674.416386]  ? rescuer_thread+0x278/0x278
[ 6674.416387]  kthread+0x106/0x10e
[ 6674.416389]  ? __list_del_entry+0x22/0x22
[ 6674.416391]  ret_from_fork+0x27/0x40
[ 6674.416395] Code: 44 89 e2 be 00 10 00 00 ff 15 b0 ab ef ff eb 72
4d 89 e8 89 d9 44 89 e2 be 00 10 00 00 ff 15 a3 ab ef ff eb 5d 41 83
fc ff 74 02 <0f> 0b 49 63 97 0c 01 00 00 48 63 c3 b9 00 04 00 00 49 8d
44 c5
[ 6674.416432] RIP: __raid_recover_end_io+0x1ac/0x370 [btrfs] RSP:
ffffc90001fbbb90
[ 6674.416434] ---[ end trace 74d56ebe7489dd6a ]---

Scrub also finds lots of errors in those tests, causing them to fail.

btrfs/048 also fails due to this new dir item length validation function....

And so on...

Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Without any failures, no, with acceptable failures at this point in development cycle, yes. The dir-item validation failures are fixable and will come next. I haven't seen the crashes in raid56, thanks for the report.

Yeah, that was an incomplete fix for a bug that I had reported but did not receive a fix in time to send the pull request. I want the dir-item validation patches merged now, we have enough time to fix the failures.

The merge window is due in the next weeks, major part of the patches be ready earlier, so I'm deciding what's applicable even it's not perfect. Known and fixable failures are just fine right now, theres whole rc1-rc5 period.

You might disagree with this approach. The quality dropped, somebody did not verify the code, run the tests or check the results. Fill in the names. I'm not developers' QA, I put patches together and push further upstream. Reviews cost a lot of time, driving multiple patchsets forward is exhausting. So I do not test each and every commit or address all failures immediatelly, I count on the time where this can happen. And also on the rest of the developer crew.

(Note for casual readers of this mail: I want to set the expectations right, there's nothing personal.)

Dave I run bisection of all failures between -rc6 and your pull request and here are the results:

btrfs/048 - e79a33270d05 ("btrfs: Check name_len with boundary in verify dir_item") btrfs/053 - b780e22768d3 ("btrfs: fix validation of XATTR_ITEM dir items") generic/273 - 28785f70ef88 ("Btrfs: skip commit transaction if we don't have enough pinned bytes") generic/337 - e79a33270d05 ("btrfs: Check name_len with boundary in verify dir_item") shared/002 - e79a33270d05 ("btrfs: Check name_len with boundary in verify dir_item")

So those are tests which are failing as a direct of the listed patches.

Thanks for the list. I'm not sure the bisection helps here, the bug is in the validation function, other patches call it in various places. But I've used the list of tests for verifying my fix I sent yesterday. Overnight run of all fstests still found one instance that fails, test generic/066 so I'll look at it today.