xfs: free space defrag and autonomous self healing
Subsystems (5)
Hi everyone,
Thank you all for helping get online repair, parent pointers, and metadata directory trees, and realtime allocation groups merged this year! We got a lot done in 2024.
Having sent pull requests to Carlos for the last pieces of the realtime modernization project, I have exactly two worthwhile projects left in my development trees! The stuff here isn't necessarily in mergeable state yet, but I still believe everyone ought to know what I'm up to.
The first project implements (somewhat buggily; I never quite got back to dealing with moving eof blocks) free space defragmentation so that we can meaningfully shrink filesystems; garbage collect regions of the filesystem; or prepare for large allocations. There's not much new kernel code other than exporting refcounts and gaining the ability to map free space.
The second project initiates filesystem self healing routines whenever problems start to crop up, which means that it can run fully autonomously in the background. The monitoring system uses some pseudo-file and seqbuf tricks that I lifted from kmo last winter.
Both of these projects are largely userspace code.
Also I threw in some xfs_repair code to do dangerous fs upgrades. Nobody should use these, ever.
Maintainers: please do not merge, this is a dog-and-pony show to attract developer attention.
--D
PS: I'll be back after the holidays to look at the zoned/atomic/fsverity patches. And finally rebase fstests to 2024-12-08.
PATCH 1/5 xfs: improve post-close eofblocks gc behavior
Hi all,
Here's a few patches mostly from Dave to make XFS more aggressive about keeping post-eof speculative preallocations when closing files.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=reduce-eofblocks-gc-on-close
---
Commits in this patchset:
* xfs: Don't free EOF blocks on close when extent size hints are set
--- fs/xfs/xfs_bmap_util.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
PATCH 1/1 xfs: Don't free EOF blocks on close when extent size hints are set
From: Dave Chinner <[email protected]>
When we have a workload that does open/write/close on files with extent size hints set in parallel with other allocation, the file becomes rapidly fragmented. This is due to close() calling xfs_release() and removing the preallocated extent beyond EOF. This occurs for both buffered and direct writes that append to files with extent size hints.
The existing open/write/close hueristic in xfs_release() does not catch this as writes to files using extent size hints do not use delayed allocation and hence do not leave delayed allocation blocks allocated on the inode that can be detected in xfs_release(). Hence XFS_IDIRTY_RELEASE never gets set.
In xfs_file_release(), we can tell whether the inode has extent size hints set and skip EOF block truncation. We add this check to xfs_can_free_eofblocks() so that we treat the post-EOF preallocated extent like intentional preallocation and so are persistent unless directly removed by userspace.
Before:
Test 2: Extent size hint fragmentation counts
/mnt/scratch/file.0: 1002/mnt/scratch/file.1: 1002/mnt/scratch/file.2: 1002/mnt/scratch/file.3: 1002/mnt/scratch/file.4: 1002/mnt/scratch/file.5: 1002/mnt/scratch/file.6: 1002/mnt/scratch/file.7: 1002
After:
Test 2: Extent size hint fragmentation counts
/mnt/scratch/file.0: 4/mnt/scratch/file.1: 4/mnt/scratch/file.2: 4/mnt/scratch/file.3: 4/mnt/scratch/file.4: 4/mnt/scratch/file.5: 4/mnt/scratch/file.6: 4/mnt/scratch/file.7: 4
--- fs/xfs/xfs_bmap_util.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/xfs/xfs_bmap_util.c +3 -2
@@ -527,8 +527,9 @@ xfs_can_free_eofblocks(* Do not free real extents in preallocated files unless the file has * delalloc blocks and we are forced to remove them. */ -if ((ip->i_diflags &XFS_DIFLAG_PREALLOC ) && !ip->i_delayed_blks) -return false; +if (xfs_get_extsz_hint (ip) || (ip->i_diflags &XFS_DIFLAG_APPEND )) +if (ip->i_delayed_blks ==0 ) +return false;/* * Do not try to free post-EOF blocks if EOF is beyond the end of the
RFC PATCH 1/5 xfsprogs: noalloc allocation groups
In reply to Darrick J. WongHi all,
This series creates a new NOALLOC flag for allocation groups that causes the block and inode allocators to look elsewhere when trying to allocate resources. This is either the first part of a patchset to implement online shrinking (set noalloc on the last AGs, run fsr to move the files and directories) or freeze-free rmapbt rebuilding (set noalloc to prevent creation of new mappings, then hook deletion of old mappings). This is still totally a research project.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=noalloc-ags
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=noalloc-ags
---
Commits in this patchset:
* xfs: track deferred ops statistics
* xfs: create a noalloc mode for allocation groups
* xfs: enable userspace to hide an AG from allocation
* xfs: apply noalloc mode to inode allocations too
* xfs_io: enhance the aginfo command to control the noalloc flag
--- include/xfs_trace.h | 2 + include/xfs_trans.h | 4 ++ io/aginfo.c | 45 ++++++++++++++++++-- libxfs/xfs_ag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_ag.h | 8 ++++ libxfs/xfs_ag_resv.c | 28 +++++++++++- libxfs/xfs_defer.c | 18 +++++++- libxfs/xfs_fs.h | 5 ++ libxfs/xfs_ialloc.c | 3 + man/man8/xfs_io.8 | 6 ++- 10 files changed, 223 insertions(+), 10 deletions(-)
PATCH 1/5 xfs: track deferred ops statistics
From: Darrick J. Wong <[email protected]>
Track some basic statistics on how hard we're pushing the defer ops.
--- include/xfs_trans.h | 4 ++++ libxfs/xfs_defer.c | 18 +++++++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-)
include/xfs_trans.h +4 -0
@@ -82,6 +82,10 @@ typedef struct xfs_trans {long t_frextents_delta;/* superblock freextents chg*/ struct list_head t_items;/* log item descriptors */ struct list_head t_dfops;/* deferred operations */ + +unsigned int t_dfops_nr; +unsigned int t_dfops_nr_max; +unsigned int t_dfops_finished; } xfs_trans_t;void xfs_trans_init (struct xfs_mount *);
libxfs/xfs_defer.c +17 -1
@@ -611,6 +611,8 @@ xfs_defer_finish_one(/* Done with the dfp, free it. */ list_del (&dfp->dfp_list);kmem_cache_free (xfs_defer_pending_cache, dfp); + tp->t_dfops_nr--; + tp->t_dfops_finished++; out:if (ops->finish_cleanup) ops->finish_cleanup (tp, state, error);
@@ -673,6 +675,9 @@ xfs_defer_finish_noroll(list_splice_init (&(*tp)->t_dfops, &dop_pending); + (*tp)->t_dfops_nr_max =max ((*tp)->t_dfops_nr, + (*tp)->t_dfops_nr_max); +if (has_intents <0 ) { error = has_intents; goto out_shutdown;
@@ -714,6 +719,7 @@ xfs_defer_finish_noroll(xfs_force_shutdown ((*tp)->t_mountp,SHUTDOWN_CORRUPT_INCORE );trace_xfs_defer_finish_error (*tp, error);xfs_defer_cancel_list ((*tp)->t_mountp, &dop_pending); + (*tp)->t_dfops_nr =0 ;xfs_defer_cancel (*tp);return error; }
@@ -761,6 +767,7 @@ xfs_defer_cancel(trace_xfs_defer_cancel (tp, _RET_IP_);xfs_defer_trans_abort (tp, &tp->t_dfops);xfs_defer_cancel_list (mp, &tp->t_dfops); + tp->t_dfops_nr =0 ; }/*
@@ -846,8 +853,10 @@ xfs_defer_add( } dfp =xfs_defer_find_last (tp, ops); -if (!dfp || !xfs_defer_can_append (dfp, ops)) +if (!dfp || !xfs_defer_can_append (dfp, ops)) { dfp =xfs_defer_alloc (&tp->t_dfops, ops); + tp->t_dfops_nr++; + }xfs_defer_add_item (dfp, li);trace_xfs_defer_add_item (tp->t_mountp, dfp, li);
@@ -872,6 +881,7 @@ xfs_defer_add_barrier(return ;xfs_defer_alloc (&tp->t_dfops, &xfs_barrier_defer_type); + tp->t_dfops_nr++;trace_xfs_defer_add_item (tp->t_mountp, dfp,NULL ); }
@@ -932,6 +942,12 @@ xfs_defer_move( struct xfs_trans *stp) {list_splice_init (&stp->t_dfops, &dtp->t_dfops); + dtp->t_dfops_nr += stp->t_dfops_nr; + dtp->t_dfops_nr_max = stp->t_dfops_nr_max; + dtp->t_dfops_finished = stp->t_dfops_finished; + stp->t_dfops_nr =0 ; + stp->t_dfops_nr_max =0 ; + stp->t_dfops_finished =0 ;/* * Low free space mode was historically controlled by a dfops field.
PATCH 2/5 xfs: create a noalloc mode for allocation groups
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a new noalloc state for the per-AG structure that will disable block allocation in this AG. We accomplish this by subtracting from fdblocks all the free blocks in this AG, hiding those blocks from the allocator, and preventing freed blocks from updating fdblocks until we're ready to lift noalloc mode.
Note that we reduce the free block count of the filesystem so that we can prevent transactions from entering the allocator looking for "free" space that we've turned off incore.
--- include/xfs_trace.h | 2 ++ libxfs/xfs_ag.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_ag.h | 8 +++++++ libxfs/xfs_ag_resv.c | 28 +++++++++++++++++++++-- 4 files changed, 95 insertions(+), 3 deletions(-)
include/xfs_trace.h +2 -0
@@ -13,6 +13,8 @@#define trace_xfbtree_trans_cancel_buf (...) ((void) 0)#define trace_xfbtree_trans_commit_buf (...) ((void) 0) +#define trace_xfs_ag_clear_noalloc (a) ((void) 0) +#define trace_xfs_ag_set_noalloc (a) ((void) 0)#define trace_xfs_agfl_reset (a,b,c,d) ((void) 0)#define trace_xfs_agfl_free_defer (...) ((void) 0)#define trace_xfs_alloc_cur_check (...) ((void) 0)
libxfs/xfs_ag.c +60 -0
63 lines not on this page · raw message
libxfs/xfs_ag.h +8 -0
@@ -120,6 +120,7 @@ static inline xfs_agnumber_t pag_agno(const struct xfs_perag *pag)#define XFS_AGSTATE_PREFERS_METADATA 2#define XFS_AGSTATE_ALLOWS_INODES 3#define XFS_AGSTATE_AGFL_NEEDS_RESET 4 +#define XFS_AGSTATE_NOALLOC 5#define __XFS_AG_OPSTATE (name,NAME ) \ static inline bool xfs_perag_ ## name (struct xfs_perag *pag) \
@@ -132,6 +133,7 @@ __XFS_AG_OPSTATE(initialised_agi, AGI_INIT) __XFS_AG_OPSTATE(prefers_metadata, PREFERS_METADATA) __XFS_AG_OPSTATE(allows_inodes, ALLOWS_INODES) __XFS_AG_OPSTATE(agfl_needs_reset, AGFL_NEEDS_RESET) +__XFS_AG_OPSTATE(prohibits_alloc, NOALLOC) int xfs_initialize_perag(struct xfs_mount *mp, xfs_agnumber_t orig_agcount, xfs_agnumber_t new_agcount, xfs_rfsblock_t dcount,
@@ -164,6 +166,7 @@ xfs_perag_put(xfs_group_put (pag_group (pag)); } +/* Active AG references */ static inline struct xfs_perag * xfs_perag_grab(
@@ -208,6 +211,11 @@ xfs_perag_next(return xfs_perag_next_from (mp, pag,0 ); } +/* Enable or disable allocation from an AG */ +xfs_extlen_t xfs_ag_fdblocks (struct xfs_perag *pag); +int xfs_ag_set_noalloc (struct xfs_perag *pag); +void xfs_ag_clear_noalloc (struct xfs_perag *pag); +/* * Per-ag geometry infomation and validation */
libxfs/xfs_ag_resv.c +25 -3
@@ -20,6 +20,7 @@#include "xfs_ialloc_btree.h" #include "xfs_ag.h" #include "xfs_ag_resv.h" +#include "xfs_ag.h" /* * Per-AG Block Reservations
@@ -73,6 +74,13 @@ xfs_ag_resv_critical(xfs_extlen_t avail;xfs_extlen_t orig; +/* +* Pretend we're critically low on reservations in this AG to scare +* everyone else away. +*/ +if (xfs_perag_prohibits_alloc (pag)) +return true; +switch (type) {case XFS_AG_RESV_METADATA : avail = pag->pagf_freeblks - pag->pag_rmapbt_resv.ar_reserved;
@@ -115,7 +123,12 @@ xfs_ag_resv_needed(break ;case XFS_AG_RESV_METAFILE :case XFS_AG_RESV_NONE : -/* empty */ +/* +* In noalloc mode, we pretend that all the free blocks in this +* AG have been allocated. Make this AG look full. +*/ +if (xfs_perag_prohibits_alloc (pag)) + len +=xfs_ag_fdblocks (pag);break ;default :ASSERT (0 );
@@ -343,6 +356,8 @@ xfs_ag_resv_alloc_extent(xfs_extlen_t len;uint field; +ASSERT (type !=XFS_AG_RESV_NONE || !xfs_perag_prohibits_alloc (pag)); +trace_xfs_ag_resv_alloc_extent (pag, type, args->len);switch (type) {
@@ -400,7 +415,14 @@ xfs_ag_resv_free_extent(ASSERT (0 ); fallthrough;case XFS_AG_RESV_NONE : -xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , (int64_t )len); +/* +* Normally we put freed blocks back into fdblocks. In noalloc +* mode, however, we pretend that there are no fdblocks in the +* AG, so don't put them back. +*/ +if (!xfs_perag_prohibits_alloc (pag)) +xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , + (int64_t )len); fallthrough;case XFS_AG_RESV_IGNORE :return ;
@@ -413,6 +435,6 @@ xfs_ag_resv_free_extent(/* Freeing into the reserved pool only requires on-disk update... */ xfs_trans_mod_sb (tp,XFS_TRANS_SB_RES_FDBLOCKS , len);/* ...but freeing beyond that requires in-core and on-disk update. */ -if (len > leftover) +if (len > leftover && !xfs_perag_prohibits_alloc (pag))xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , len - leftover); }
PATCH 3/5 xfs: enable userspace to hide an AG from allocation
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add an administrative interface so that userspace can hide an allocation group from block allocation.
--- libxfs/xfs_ag.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_fs.h | 5 +++++ 2 files changed, 59 insertions(+)
libxfs/xfs_ag.c +54 -0
@@ -930,6 +930,54 @@ xfs_ag_extend_space(return 0 ; } +/* Compute the AG geometry flags. */ +static inline uint32_t +xfs_ag_calc_geoflags ( +struct xfs_perag *pag) +{ +uint32_t ret =0 ; + +if (xfs_perag_prohibits_alloc (pag)) + ret |=XFS_AG_FLAG_NOALLOC ; + +return ret; +} + +/* +* Compare the current AG geometry flags against the flags in the AG geometry +* structure and update the AG state to reflect any changes, then update the +* struct to reflect the current status. +*/ +static inline int +xfs_ag_update_geoflags ( +struct xfs_perag *pag, +struct xfs_ag_geometry *ageo, +uint32_t new_flags) +{ +uint32_t old_flags =xfs_ag_calc_geoflags (pag); +int error; + +if (!(new_flags &XFS_AG_FLAG_UPDATE )) { + ageo->ag_flags = old_flags; +return 0 ; + } + +if ((old_flags &XFS_AG_FLAG_NOALLOC ) && + !(new_flags &XFS_AG_FLAG_NOALLOC )) { +xfs_ag_clear_noalloc (pag); + } + +if (!(old_flags &XFS_AG_FLAG_NOALLOC ) && + (new_flags &XFS_AG_FLAG_NOALLOC )) { + error =xfs_ag_set_noalloc (pag); +if (error) +return error; + } + + ageo->ag_flags =xfs_ag_calc_geoflags (pag); +return 0 ; +} +/* Retrieve AG geometry. */ int xfs_ag_get_geometry(
@@ -941,6 +989,7 @@ xfs_ag_get_geometry(struct xfs_agi *agi;struct xfs_agf *agf;unsigned int freeblks; +uint32_t inflags = ageo->ag_flags;int error;/* Lock the AG headers. */
@@ -951,6 +1000,10 @@ xfs_ag_get_geometry(if (error) goto out_agi; + error =xfs_ag_update_geoflags (pag, ageo, inflags); +if (error) + goto out; +/* Fill out form. */ memset (ageo,0 ,sizeof (*ageo)); ageo->ag_number =pag_agno (pag);
@@ -968,6 +1021,7 @@ xfs_ag_get_geometry( ageo->ag_freeblks = freeblks;xfs_ag_geom_health (pag, ageo); +out:/* Release resources. */ xfs_buf_relse (agf_bp); out_agi:
libxfs/xfs_fs.h +5 -0
@@ -307,6 +307,11 @@ struct xfs_ag_geometry {#define XFS_AG_GEOM_SICK_REFCNTBT (1 << 9)/* reference counts */ #define XFS_AG_GEOM_SICK_INODES (1 << 10)/* bad inodes were seen */ +#define XFS_AG_FLAG_UPDATE (1 << 0)/* update flags */ +#define XFS_AG_FLAG_NOALLOC (1 << 1)/* do not allocate from this AG */ +#define XFS_AG_FLAG_ALL (XFS_AG_FLAG_UPDATE | \ + XFS_AG_FLAG_NOALLOC) +/* * Structures for XFS_IOC_FSGROWFSDATA, XFS_IOC_FSGROWFSLOG & XFS_IOC_FSGROWFSRT */
PATCH 4/5 xfs: apply noalloc mode to inode allocations too
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Don't allow inode allocations from this group if it's marked noalloc.
--- libxfs/xfs_ialloc.c | 3 +++ 1 file changed, 3 insertions(+)
libxfs/xfs_ialloc.c +3 -0
@@ -1102,6 +1102,7 @@ xfs_dialloc_ag_inobt(ASSERT (xfs_perag_initialised_agi (pag));ASSERT (xfs_perag_allows_inodes (pag)); +ASSERT (!xfs_perag_prohibits_alloc (pag));ASSERT (pag->pagi_freecount >0 ); restart_pagno:
@@ -1730,6 +1731,8 @@ xfs_dialloc_good_ag(return false;if (!xfs_perag_allows_inodes (pag))return false; +if (xfs_perag_prohibits_alloc (pag)) +return false;if (!xfs_perag_initialised_agi (pag)) { error =xfs_ialloc_read_agi (pag, tp,0 ,NULL );
PATCH 5/5 xfs_io: enhance the aginfo command to control the noalloc flag
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Augment the aginfo command to be able to set and clear the noalloc state for an AG.
--- io/aginfo.c | 45 ++++++++++++++++++++++++++++++++++++++++----- man/man8/xfs_io.8 | 6 +++++- 2 files changed, 45 insertions(+), 6 deletions(-)
io/aginfo.c +40 -5
@@ -19,9 +19,11 @@ static cmdinfo_t rginfo_cmd;static int report_aginfo (struct xfs_fd *xfd, -xfs_agnumber_t agno) +xfs_agnumber_t agno, +int oflag) {struct xfs_ag_geometry ageo = {0 }; +bool update = false;int ret; ret = -xfrog_ag_geometry (xfd->fd, agno, &ageo);
@@ -30,6 +32,26 @@ report_aginfo(return 1 ; } +switch (oflag) { +case 0 : + ageo.ag_flags |=XFS_AG_FLAG_UPDATE ; + ageo.ag_flags &= ~XFS_AG_FLAG_NOALLOC ; + update = true; +break ; +case 1 : + ageo.ag_flags |= (XFS_AG_FLAG_UPDATE |XFS_AG_FLAG_NOALLOC ); + update = true; +break ; + } + +if (update) { + ret = -xfrog_ag_geometry (xfd->fd, agno, &ageo); +if (ret) { +xfrog_perror (ret,"aginfo update" ); +return 1 ; + } + } +printf (_ ("AG: %u\n" ), ageo.ag_number);printf (_ ("Blocks: %u\n" ), ageo.ag_length);printf (_ ("Free Blocks: %u\n" ), ageo.ag_freeblks);
@@ -51,6 +73,7 @@ aginfo_f(struct xfs_fd xfd =XFS_FD_INIT (file->fd);unsigned long long x;xfs_agnumber_t agno =NULLAGNUMBER ; +int oflag =-1 ;int c;int ret =0 ;
@@ -61,7 +84,7 @@ aginfo_f(return 1 ; } -while ((c =getopt (argc, argv,"a:" )) !=EOF ) { +while ((c =getopt (argc, argv,"a:o:" )) !=EOF ) {switch (c) {case 'a' : errno =0 ;
@@ -74,16 +97,27 @@ aginfo_f( } agno = x;break ; +case 'o' : + errno =0 ; + x =strtoll (optarg,NULL ,10 ); +if (!errno && x !=0 && x !=1 ) + errno =ERANGE ; +if (errno) { +perror ("aginfo" ); +return 1 ; + } + oflag = x; +break ;default :return command_usage (&aginfo_cmd); } }if (agno !=NULLAGNUMBER ) { - ret =report_aginfo (&xfd, agno); + ret =report_aginfo (&xfd, agno, oflag); }else {for (agno =0 ; !ret && agno < xfd.fsgeom.agcount; agno++) { - ret =report_aginfo (&xfd, agno); + ret =report_aginfo (&xfd, agno, oflag); } }
@@ -98,6 +132,7 @@ aginfo_help(void)"Report allocation group geometry.\n" "\n" " -a agno -- Report on the given allocation group.\n" +" -o state -- Change the NOALLOC state for this allocation group.\n" "\n")); }
@@ -107,7 +142,7 @@ static cmdinfo_t aginfo_cmd = { .cfunc = aginfo_f, .argmin = 0, .argmax = -1, - .args = "[-a agno]", + .args = "[-a agno] [-o state]", .flags = CMD_NOMAP_OK, .help = aginfo_help, };
man/man8/xfs_io.8 +5 -1
@@ -1243,7 +1243,7 @@ .SH MEMORY MAPPED I/O COMMANDS .SH FILESYSTEM COMMANDS .TP -.BI "aginfo [ \-a " agno " ]" +.BI "aginfo [ \-a " agno " ] [ \-o " nr " ]" Show information about or update the state of allocation groups. .RE .RS 1.0i
@@ -1251,6 +1251,10 @@ .SH FILESYSTEM COMMANDS .TP .BI \-a Act only on a specific allocation group. +.TP +.BI \-o +If 0, clear the NOALLOC flag. +If 1, set the NOALLOC flag. .PD .RE
PATCH 1/5 fstests: functional test for refcount reporting
In reply to Darrick J. WongHi all,
Add a short functional test for the new GETFSREFCOUNTS ioctl that allows userspace to query reference count information for a given range of physical blocks.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=report-refcounts
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=report-refcounts
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=report-refcounts
---
Commits in this patchset:
* xfs: test output of new FSREFCOUNTS ioctl
--- common/rc | 4 + doc/group-names.txt | 1 tests/xfs/1921 | 164 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1921.out | 4 + 4 files changed, 171 insertions(+), 2 deletions(-) create mode 100755 tests/xfs/1921 create mode 100644 tests/xfs/1921.out
PATCH 1/1 xfs: test output of new FSREFCOUNTS ioctl
From: Darrick J. Wong <[email protected]>
Make sure the cursors work properly and that refcounts are correct.
--- common/rc | 4 + doc/group-names.txt | 1 tests/xfs/1921 | 164 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1921.out | 4 + 4 files changed, 171 insertions(+), 2 deletions(-) create mode 100755 tests/xfs/1921 create mode 100644 tests/xfs/1921.out
common/rc +2 -2
@@ -2811,8 +2811,8 @@ _require_xfs_io_command() echo $testio | grep -q "Operation not supported" && \ _notrun "O_TMPFILE is not supported" ;; - "fsmap") - testio=`$XFS_IO_PROG -f -c "fsmap" $testfile 2>&1` + "fsmap"|"fsrefcounts") + testio=`$XFS_IO_PROG -f -c "$command" $testfile 2>&1` echo $testio | grep -q "Inappropriate ioctl" && \ _notrun "xfs_io $command support is missing" ;;
doc/group-names.txt +1 -0
@@ -58,6 +58,7 @@ fsck general fsck tests fsmap FS_IOC_GETFSMAP ioctl fsproperties Filesystem properties fsr XFS free space reorganizer +fsrefcounts FS_IOC_GETFSREFCOUNTS ioctl fuzzers filesystem fuzz tests growfs increasing the size of a filesystem hardlink hardlinks
tests/xfs/1921 new file +164 -0
164 lines not on this page · raw message
tests/xfs/1921.out new file +4 -0
@@ -0,0 +1,4 @@ +QA output created by 1921 +Format and mount +Compare fsrefcounts +Compare fsrefcounts to fsmap
PATCH 2/5 xfsprogs: report refcount information to userspace
In reply to Darrick J. WongHi all,
Create a new ioctl to report the number of owners of each disk block so that reflink-aware defraggers can make better decisions about which extents to target.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=report-refcounts
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=report-refcounts
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=report-refcounts
---
Commits in this patchset:
* xfs: export reference count information to userspace
* xfs_io: dump reference count information
--- io/Makefile | 1 io/fsrefcounts.c | 476 +++++++++++++++++++++++++++++++++++ io/init.c | 1 io/io.h | 1 libxfs/xfs_fs.h | 80 ++++++ man/man2/ioctl_xfs_getfsrefcounts.2 | 237 +++++++++++++++++ man/man8/xfs_io.8 | 88 ++++++ 7 files changed, 884 insertions(+) create mode 100644 io/fsrefcounts.c create mode 100644 man/man2/ioctl_xfs_getfsrefcounts.2
PATCH 1/2 xfs: export reference count information to userspace
From: Darrick J. Wong <[email protected]>
Export refcount info to userspace so we can prototype a sharing-aware defrag/fs rearranging tool.
--- libxfs/xfs_fs.h | 80 ++++++++++++ man/man2/ioctl_xfs_getfsrefcounts.2 | 237 +++++++++++++++++++++++++++++++++++ 2 files changed, 317 insertions(+) create mode 100644 man/man2/ioctl_xfs_getfsrefcounts.2
libxfs/xfs_fs.h +80 -0
@@ -1008,6 +1008,85 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +/* +* Structure for XFS_IOC_GETFSREFCOUNTS. +* +* The memory layout for this call are the scalar values defined in struct +* xfs_getfsrefs_head, followed by two struct xfs_getfsrefs that describe +* the lower and upper bound of mappings to return, followed by an array +* of struct xfs_getfsrefs mappings. +* +* fch_iflags control the output of the call, whereas fch_oflags report +* on the overall record output. fch_count should be set to the length +* of the fch_recs array, and fch_entries will be set to the number of +* entries filled out during each call. If fch_count is zero, the number +* of refcount mappings will be returned in fch_entries, though no +* mappings will be returned. fch_reserved must be set to zero. +* +* The two elements in the fch_keys array are used to constrain the +* output. The first element in the array should represent the lowest +* disk mapping ("low key") that the user wants to learn about. If this +* value is all zeroes, the filesystem will return the first entry it +* knows about. For a subsequent call, the contents of +* fsrefs_head.fch_recs[fsrefs_head.fch_count - 1] should be copied into +* fch_keys[0] to have the kernel start where it left off. +* +* The second element in the fch_keys array should represent the highest +* disk mapping ("high key") that the user wants to learn about. If this +* value is all ones, the filesystem will not stop until it runs out of +* mapping to return or runs out of space in fch_recs. +* +* fcr_device can be either a 32-bit cookie representing a device, or a +* 32-bit dev_t if the FCH_OF_DEV_T flag is set. fcr_physical and +* fcr_length are expressed in units of bytes. fcr_owners is the number +* of owners. +*/ +struct xfs_getfsrefs { +__u32 fcr_device;/* device id */ +__u32 fcr_flags;/* mapping flags */ +__u64 fcr_physical;/* device offset of segment */ +__u64 fcr_owners;/* number of owners */ +__u64 fcr_length;/* length of segment */ +__u64 fcr_reserved[4 ];/* must be zero */ +}; + +struct xfs_getfsrefs_head { +__u32 fch_iflags;/* control flags */ +__u32 fch_oflags;/* output flags */ +__u32 fch_count;/* # of entries in array incl. input */ +__u32 fch_entries;/* # of entries filled in (output). */ +__u64 fch_reserved[6 ];/* must be zero */ + +struct xfs_getfsrefs fch_keys[2 ];/* low and high keys for the mapping search */ +struct xfs_getfsrefs fch_recs[];/* returned records */ +}; + +/* Size of an fsrefs_head with room for nr records. */ +static inline unsigned long long +xfs_getfsrefs_sizeof ( +unsigned int nr) +{ +return sizeof (struct xfs_getfsrefs_head ) + + (nr *sizeof (struct xfs_getfsrefs )); +} + +/* Start the next fsrefs query at the end of the current query results. */ +static inline void +xfs_getfsrefs_advance ( +struct xfs_getfsrefs_head *head) +{ + head->fch_keys[0 ] = head->fch_recs[head->fch_entries -1 ]; +} + +/* fch_iflags values - set by XFS_IOC_GETFSREFCOUNTS caller in the header. */ +#define FCH_IF_VALID 0 + +/* fch_oflags values - returned in the header segment only. */ +#define FCH_OF_DEV_T (1U << 0)/* fcr_device values will be dev_t */ + +/* fcr_flags values - returned for each non-header segment */ +#define FCR_OF_LAST (1U << 0)/* last record in the dataset */ +/* * ioctl commands that are used by Linux filesystems */
@@ -1047,6 +1126,7 @@ struct xfs_rtgroup_geometry {#define XFS_IOC_GETPARENTS_BY_HANDLE _IOWR('X', 63, struct xfs_getparents_by_handle)#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry) +#define XFS_IOC_GETFSREFCOUNTS _IOWR('X', 66, struct xfs_getfsrefs_head)/* * ioctl commands that replace IRIX syssgi()'s
man/man2/ioctl_xfs_getfsrefcounts.2 new file +237 -0
237 lines not on this page · raw message
PATCH 2/2 xfs_io: dump reference count information
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Dump refcount info from the kernel so we can prototype a sharing-aware defrag/fs rearranging tool.
--- io/Makefile | 1 io/fsrefcounts.c | 476 +++++++++++++++++++++++++++++++++++++++++++++++++++++ io/init.c | 1 io/io.h | 1 man/man8/xfs_io.8 | 88 ++++++++++ 5 files changed, 567 insertions(+) create mode 100644 io/fsrefcounts.c
io/Makefile +1 -0
@@ -22,6 +22,7 @@ CFILES = \ file.c \ freeze.c \ fsproperties.c \ + fsrefcounts.c \ fsuuid.c \ fsync.c \ getrusage.c \
io/fsrefcounts.c new file +476 -0
476 lines not on this page · raw message
io/init.c +1 -0
@@ -58,6 +58,7 @@ init_commands(void)freeze_init ();fsmap_init ();fsuuid_init (); +fsrefcounts_init ();fsync_init ();getrusage_init ();help_init ();
io/io.h +1 -0
@@ -156,3 +156,4 @@ extern void bulkstat_init(void);void exchangerange_init (void );void fsprops_init (void );void aginfo_init (void ); +void fsrefcounts_init (void );
man/man8/xfs_io.8 +88 -0
94 lines not on this page · raw message
PATCH 2/5 fstests: defragment free space
In reply to Darrick J. WongHi all,
These patches contain experimental code to enable userspace to defragment the free space in a filesystem. Two purposes are imagined for this
functionality: clearing space at the end of a filesystem before shrinking it, and clearing free space in anticipation of making a large allocation.
The first patch adds a new fallocate mode that allows userspace to allocate free space from the filesystem into a file. The goal here is to allow the filesystem shrink process to prevent allocation from a certain part of the filesystem while a free space defragmenter evacuates all the files from the doomed part of the filesystem.
The second patch amends the online repair system to allow the sysadmin to forcibly rebuild metadata structures, even if they're not corrupt. Without adding an ioctl to move metadata btree blocks, this is the only way to dislodge metadata.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=defrag-freespace
---
Commits in this patchset:
* xfs: test clearing of free space
--- common/rc | 5 ++++ tests/xfs/1400 | 52 +++++++++++++++++++++++++++++++++++++++ tests/xfs/1400.out | 2 + tests/xfs/1401 | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1401.out | 2 + 5 files changed, 131 insertions(+) create mode 100755 tests/xfs/1400 create mode 100644 tests/xfs/1400.out create mode 100755 tests/xfs/1401 create mode 100644 tests/xfs/1401.out
PATCH 1/1 xfs: test clearing of free space
From: Darrick J. Wong <[email protected]>
Simple regression test for the spaceman clearspace command, which tries to free all the used space in some part of the filesystem.
--- common/rc | 5 ++++ tests/xfs/1400 | 52 +++++++++++++++++++++++++++++++++++++++ tests/xfs/1400.out | 2 + tests/xfs/1401 | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1401.out | 2 + 5 files changed, 131 insertions(+) create mode 100755 tests/xfs/1400 create mode 100644 tests/xfs/1400.out create mode 100755 tests/xfs/1401 create mode 100644 tests/xfs/1401.out
common/rc +5 -0
@@ -2786,6 +2786,11 @@ _require_xfs_io_command() -c "fsync" -c "$command $blocksize $((2 * $blocksize))" \ $testfile 2>&1` ;; + "fmapfree") + local blocksize=$(_get_file_block_size $TEST_DIR) + testio=`$XFS_IO_PROG -F -f -c "$command $blocksize $((2 * $blocksize))" \ + $testfile 2>&1` + ;; "fiemap") # If 'ranged' is passed as argument then we check to see if fiemap supports # ranged query params
tests/xfs/1400 new file +52 -0
52 lines not on this page · raw message
tests/xfs/1400.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1400 +mapfree space used is in range
tests/xfs/1401 new file +70 -0
70 lines not on this page · raw message
tests/xfs/1401.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1401 +Silence is golden
RFC PATCH 2/5 xfs: noalloc allocation groups
In reply to Darrick J. WongHi all,
This series creates a new NOALLOC flag for allocation groups that causes the block and inode allocators to look elsewhere when trying to allocate resources. This is either the first part of a patchset to implement online shrinking (set noalloc on the last AGs, run fsr to move the files and directories) or freeze-free rmapbt rebuilding (set noalloc to prevent creation of new mappings, then hook deletion of old mappings). This is still totally a research project.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=noalloc-ags
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=noalloc-ags
---
Commits in this patchset:
* xfs: track deferred ops statistics
* xfs: whine to dmesg when we encounter errors
* xfs: create a noalloc mode for allocation groups
* xfs: enable userspace to hide an AG from allocation
* xfs: apply noalloc mode to inode allocations too
--- fs/xfs/Kconfig | 13 +++++ fs/xfs/libxfs/xfs_ag.c | 114 +++++++++++++++++++++++++++++++++++++++++++ fs/xfs/libxfs/xfs_ag.h | 8 +++ fs/xfs/libxfs/xfs_ag_resv.c | 27 +++++++++- fs/xfs/libxfs/xfs_defer.c | 18 ++++++- fs/xfs/libxfs/xfs_fs.h | 5 ++ fs/xfs/libxfs/xfs_ialloc.c | 3 + fs/xfs/scrub/btree.c | 89 +++++++++++++++++++++++++++++++++- fs/xfs/scrub/common.c | 107 ++++++++++++++++++++++++++++++++++++++++ fs/xfs/scrub/common.h | 1 fs/xfs/scrub/dabtree.c | 24 +++++++++ fs/xfs/scrub/fscounters.c | 3 + fs/xfs/scrub/inode.c | 4 ++ fs/xfs/scrub/scrub.c | 40 +++++++++++++++ fs/xfs/scrub/trace.c | 22 ++++++++ fs/xfs/scrub/trace.h | 2 + fs/xfs/xfs_fsops.c | 10 +++- fs/xfs/xfs_globals.c | 5 ++ fs/xfs/xfs_ioctl.c | 4 +- fs/xfs/xfs_super.c | 1 fs/xfs/xfs_sysctl.h | 1 fs/xfs/xfs_sysfs.c | 32 ++++++++++++ fs/xfs/xfs_trace.h | 65 +++++++++++++++++++++++++ fs/xfs/xfs_trans.c | 3 + fs/xfs/xfs_trans.h | 7 +++ 25 files changed, 599 insertions(+), 9 deletions(-)
PATCH 1/5 xfs: track deferred ops statistics
From: Darrick J. Wong <[email protected]>
Track some basic statistics on how hard we're pushing the defer ops.
--- fs/xfs/libxfs/xfs_defer.c | 18 +++++++++++++++++- fs/xfs/xfs_trace.h | 19 +++++++++++++++++++ fs/xfs/xfs_trans.c | 3 +++ fs/xfs/xfs_trans.h | 7 +++++++ 4 files changed, 46 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_defer.c +17 -1
@@ -618,6 +618,8 @@ xfs_defer_finish_one(/* Done with the dfp, free it. */ list_del (&dfp->dfp_list);kmem_cache_free (xfs_defer_pending_cache, dfp); + tp->t_dfops_nr--; + tp->t_dfops_finished++; out:if (ops->finish_cleanup) ops->finish_cleanup (tp, state, error);
@@ -680,6 +682,9 @@ xfs_defer_finish_noroll(list_splice_init (&(*tp)->t_dfops, &dop_pending); + (*tp)->t_dfops_nr_max =max ((*tp)->t_dfops_nr, + (*tp)->t_dfops_nr_max); +if (has_intents <0 ) { error = has_intents; goto out_shutdown;
@@ -721,6 +726,7 @@ xfs_defer_finish_noroll(xfs_force_shutdown ((*tp)->t_mountp,SHUTDOWN_CORRUPT_INCORE );trace_xfs_defer_finish_error (*tp, error);xfs_defer_cancel_list ((*tp)->t_mountp, &dop_pending); + (*tp)->t_dfops_nr =0 ;xfs_defer_cancel (*tp);return error; }
@@ -768,6 +774,7 @@ xfs_defer_cancel(trace_xfs_defer_cancel (tp, _RET_IP_);xfs_defer_trans_abort (tp, &tp->t_dfops);xfs_defer_cancel_list (mp, &tp->t_dfops); + tp->t_dfops_nr =0 ; }/*
@@ -853,8 +860,10 @@ xfs_defer_add( } dfp =xfs_defer_find_last (tp, ops); -if (!dfp || !xfs_defer_can_append (dfp, ops)) +if (!dfp || !xfs_defer_can_append (dfp, ops)) { dfp =xfs_defer_alloc (&tp->t_dfops, ops); + tp->t_dfops_nr++; + }xfs_defer_add_item (dfp, li);trace_xfs_defer_add_item (tp->t_mountp, dfp, li);
@@ -879,6 +888,7 @@ xfs_defer_add_barrier(return ;xfs_defer_alloc (&tp->t_dfops, &xfs_barrier_defer_type); + tp->t_dfops_nr++;trace_xfs_defer_add_item (tp->t_mountp, dfp,NULL ); }
@@ -939,6 +949,12 @@ xfs_defer_move( struct xfs_trans *stp) {list_splice_init (&stp->t_dfops, &dtp->t_dfops); + dtp->t_dfops_nr += stp->t_dfops_nr; + dtp->t_dfops_nr_max = stp->t_dfops_nr_max; + dtp->t_dfops_finished = stp->t_dfops_finished; + stp->t_dfops_nr =0 ; + stp->t_dfops_nr_max =0 ; + stp->t_dfops_finished =0 ;/* * Low free space mode was historically controlled by a dfops field.
fs/xfs/xfs_trace.h +19 -0
@@ -2880,6 +2880,25 @@ TRACE_EVENT(xfs_btree_free_block,/* deferred ops */ struct xfs_defer_pending ; +TRACE_EVENT (xfs_defer_stats, + TP_PROTO(struct xfs_trans *tp), +TP_ARGS (tp), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(unsigned int, max) + __field(unsigned int, finished) + ), +TP_fast_assign ( + __entry->dev = tp->t_mountp->m_super->s_dev; + __entry->max = tp->t_dfops_nr_max; + __entry->finished = tp->t_dfops_finished; + ), +TP_printk ("dev %d:%d max %u finished %u" , +MAJOR (__entry->dev),MINOR (__entry->dev), + __entry->max, + __entry->finished) +) + DECLARE_EVENT_CLASS(xfs_defer_class, TP_PROTO(struct xfs_trans *tp, unsigned long caller_ip), TP_ARGS(tp, caller_ip),
fs/xfs/xfs_trans.c +3 -0
@@ -71,6 +71,9 @@ xfs_trans_free(xfs_extent_busy_sort (&tp->t_busy);xfs_extent_busy_clear (&tp->t_busy, false); +if (tp->t_dfops_finished >0 ) +trace_xfs_defer_stats (tp); +trace_xfs_trans_free (tp, _RET_IP_);xfs_trans_clear_context (tp);if (!(tp->t_flags &XFS_TRANS_NO_WRITECOUNT ))
fs/xfs/xfs_trans.h +7 -0
@@ -153,6 +153,13 @@ typedef struct xfs_trans {struct list_head t_busy;/* list of busy extents */ struct list_head t_dfops;/* deferred operations */ unsigned long t_pflags;/* saved process flags state */ + +/* Count of deferred ops attached to transaction. */ +unsigned int t_dfops_nr; +/* Maximum t_dfops_nr seen in a loop. */ +unsigned int t_dfops_nr_max; +/* Number of dfops finished. */ +unsigned int t_dfops_finished; } xfs_trans_t;/*
PATCH 2/5 xfs: whine to dmesg when we encounter errors
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Forward everything scrub whines about to dmesg.
--- fs/xfs/Kconfig | 13 ++++++ fs/xfs/scrub/btree.c | 89 +++++++++++++++++++++++++++++++++++++++- fs/xfs/scrub/common.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/scrub/common.h | 1 fs/xfs/scrub/dabtree.c | 24 +++++++++++ fs/xfs/scrub/inode.c | 4 ++ fs/xfs/scrub/scrub.c | 40 ++++++++++++++++++ fs/xfs/scrub/trace.c | 22 ++++++++++ fs/xfs/scrub/trace.h | 2 + fs/xfs/xfs_globals.c | 5 ++ fs/xfs/xfs_sysctl.h | 1 fs/xfs/xfs_sysfs.c | 32 ++++++++++++++ 12 files changed, 338 insertions(+), 2 deletions(-)
fs/xfs/Kconfig +13 -0
@@ -172,6 +172,19 @@ config XFS_ONLINE_SCRUB_STATS If unsure, say N. +config XFS_ONLINE_SCRUB_WHINE + bool "XFS online metadata verbose logging by default" + default n + depends on XFS_ONLINE_SCRUB + help + If you say Y here, the kernel will by default log the outcomes of all + scrub and repair operations, as well as any corruptions found. This + may slow down scrub due to printk logging overhead timers. + + This value can be changed by editing /sys/fs/xfs/debug/scrub_whine + + If unsure, say N. + config XFS_ONLINE_REPAIR bool "XFS online metadata repair support" default n
fs/xfs/scrub/btree.c +87 -2
120 lines not on this page · raw message
fs/xfs/scrub/common.c +107 -0
206 lines not on this page · raw message
fs/xfs/scrub/common.h +1 -0
@@ -179,6 +179,7 @@ bool xchk_ilock_nowait(struct xfs_scrub *sc, unsigned int ilock_flags);void xchk_iunlock (struct xfs_scrub *sc,unsigned int ilock_flags);void xchk_buffer_recheck (struct xfs_scrub *sc,struct xfs_buf *bp); +void xchk_whine (const struct xfs_mount *mp,const char *fmt, ...);/* * Grab the inode at @inum. The caller must have created a scrub transaction
fs/xfs/scrub/dabtree.c +24 -0
@@ -47,9 +47,26 @@ xchk_da_process_error(case -EFSCORRUPTED :/* Note the badness but don't abort. */ sc->sm->sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ; +xchk_whine (sc->mp,"ino 0x%llx fork %d type %s dablk 0x%llx error %d ret_ip %pS" , + sc->ip->i_ino, + ds->dargs.whichfork, +xchk_type_string (sc->sm->sm_type), +xfs_dir2_da_to_db (ds->dargs.geo, + ds->state->path.blk[level].blkno), + *error, + __return_address); *error =0 ; fallthrough;default : +if (*error) +xchk_whine (sc->mp,"ino 0x%llx fork %d type %s dablk 0x%llx error %d ret_ip %pS" , + sc->ip->i_ino, + ds->dargs.whichfork, +xchk_type_string (sc->sm->sm_type), +xfs_dir2_da_to_db (ds->dargs.geo, + ds->state->path.blk[level].blkno), + *error, + __return_address); trace_xchk_file_op_error(sc, ds->dargs.whichfork, xfs_dir2_da_to_db(ds->dargs.geo, ds->state->path.blk[level].blkno),
@@ -72,6 +89,13 @@ xchk_da_set_corrupt( sc->sm->sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ; +xchk_whine (sc->mp,"ino 0x%llx fork %d type %s dablk 0x%llx ret_ip %pS" , + sc->ip->i_ino, + ds->dargs.whichfork, +xchk_type_string (sc->sm->sm_type), +xfs_dir2_da_to_db (ds->dargs.geo, + ds->state->path.blk[level].blkno), + __return_address); trace_xchk_fblock_error(sc, ds->dargs.whichfork, xfs_dir2_da_to_db(ds->dargs.geo, ds->state->path.blk[level].blkno),
fs/xfs/scrub/inode.c +4 -0
@@ -218,6 +218,10 @@ xchk_setup_inode( out_cancel:xchk_trans_cancel (sc); out_error: +xchk_whine (mp,"type %s agno 0x%x agbno 0x%x error %d ret_ip %pS" , +xchk_type_string (sc->sm->sm_type), agno, +XFS_INO_TO_AGBNO (mp, sc->sm->sm_ino), error, + __return_address);trace_xchk_op_error (sc, agno,XFS_INO_TO_AGBNO (mp, sc->sm->sm_ino), error, __return_address);return error;
fs/xfs/scrub/scrub.c +40 -0
@@ -639,6 +639,45 @@ xchk_scrub_create_subord(return sub; } +static inline void +repair_outcomes (struct xfs_scrub *sc,int error) +{ +struct xfs_scrub_metadata *sm = sc->sm; +const char *wut =NULL ; + +if (!xfs_globals.scrub_whine) +return ; + +if (sc->flags &XREP_ALREADY_FIXED ) { + wut ="*** REPAIR SUCCESS" ; + error =0 ; + }else if (error == -EBUSY ) { + wut ="??? FILESYSTEM BUSY" ; + }else if (error == -EAGAIN ) { + wut ="??? REPAIR DEFERRED" ; + }else if (error == -ECANCELED ) { + wut ="??? REPAIR CANCELLED" ; + }else if (error == -EINTR ) { + wut ="??? REPAIR INTERRUPTED" ; + }else if (error != -EOPNOTSUPP && error != -ENOENT ) { + wut ="!!! REPAIR FAILED" ; +xfs_info (sc->mp, +"%s ino 0x%llx type %s agno 0x%x inum 0x%llx gen 0x%x flags 0x%x error %d" , + wut,XFS_I (file_inode (sc->file))->i_ino, +xchk_type_string (sm->sm_type), sm->sm_agno, + sm->sm_ino, sm->sm_gen, sm->sm_flags, error); +return ; + }else { +return ; + } + +xfs_info_ratelimited (sc->mp, +"%s ino 0x%llx type %s agno 0x%x inum 0x%llx gen 0x%x flags 0x%x error %d" , + wut,XFS_I (file_inode (sc->file))->i_ino, +xchk_type_string (sm->sm_type), sm->sm_agno, sm->sm_ino, + sm->sm_gen, sm->sm_flags, error); +} +/* Dispatch metadata scrubbing. */ STATIC int xfs_scrub_metadata(
@@ -735,6 +774,7 @@ xfs_scrub_metadata(* already tried to fix it, then attempt a repair. */ error =xrep_attempt (sc, &run); +repair_outcomes (sc, error);if (error == -EAGAIN ) {/* * Either the repair function succeeded or it couldn't
fs/xfs/scrub/trace.c +22 -0
@@ -58,3 +58,25 @@ xchk_btree_cur_fsbno(*/ #define CREATE_TRACE_POINTS #include "scrub/trace.h" + +/* xchk_whine stuff */ +struct xchk_tstr { +unsigned int type; +const char *tag; +}; + +static const struct xchk_tstr xchk_tstr_tags[] = {XFS_SCRUB_TYPE_STRINGS }; + +const char * +xchk_type_string ( +unsigned int type) +{ +unsigned int i; + +for (i =0 ; i <ARRAY_SIZE (xchk_tstr_tags); i++) { +if (xchk_tstr_tags[i].type == type) +return xchk_tstr_tags[i].tag; + } + +return "???" ; +}
fs/xfs/scrub/trace.h +2 -0
@@ -115,6 +115,8 @@ TRACE_DEFINE_ENUM(XFS_SCRUB_TYPE_RTREFCBT); { XFS_SCRUB_TYPE_RTRMAPBT, "rtrmapbt" }, \ { XFS_SCRUB_TYPE_RTREFCBT, "rtrefcountbt" } +const char *xchk_type_string (unsigned int type); +#define XFS_SCRUB_FLAG_STRINGS \ { XFS_SCRUB_IFLAG_REPAIR, "repair" }, \ { XFS_SCRUB_OFLAG_CORRUPT, "corrupt" }, \
fs/xfs/xfs_globals.c +5 -0
@@ -44,6 +44,11 @@ struct xfs_globals xfs_globals = { .pwork_threads = -1, /* automatic thread detection */ .larp = false, /* log attribute replay */#endif +#ifdef CONFIG_XFS_ONLINE_SCRUB_WHINE + .scrub_whine = true, +#else + .scrub_whine = false, +#endif /* * Leave this many record slots empty when bulk loading btrees. By
fs/xfs/xfs_sysctl.h +1 -0
@@ -91,6 +91,7 @@ struct xfs_globals {int mount_delay;/* mount setup delay (secs) */ bool bug_on_assert;/* BUG() the kernel on assert failure */ bool always_cow;/* use COW fork for all overwrites */ +bool scrub_whine;/* noisier output from scrub */ };extern struct xfs_globals xfs_globals;
fs/xfs/xfs_sysfs.c +32 -0
@@ -260,6 +260,37 @@ larp_show( }XFS_SYSFS_ATTR_RW (larp); +/* Logging of the outcomes of everything that scrub does */ +STATIC ssize_t +scrub_whine_store ( +struct kobject *kobject, +const char *buf, +size_t count) +{ +int ret; +int val; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + +if (val <-1 || val >num_possible_cpus ()) +return -EINVAL ; + + xfs_globals.scrub_whine = val; + +return count; +} + +STATIC ssize_t +scrub_whine_show ( +struct kobject *kobject, +char *buf) +{ +return sysfs_emit (buf,"%d\n" , xfs_globals.scrub_whine); +} +XFS_SYSFS_ATTR_RW (scrub_whine); + STATIC ssize_t bload_leaf_slack_store( struct kobject *kobject,
@@ -319,6 +350,7 @@ static struct attribute *xfs_dbg_attrs[] = {ATTR_LIST (always_cow),ATTR_LIST (pwork_threads),ATTR_LIST (larp), +ATTR_LIST (scrub_whine),ATTR_LIST (bload_leaf_slack),ATTR_LIST (bload_node_slack), NULL,
PATCH 3/5 xfs: create a noalloc mode for allocation groups
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a new noalloc state for the per-AG structure that will disable block allocation in this AG. We accomplish this by subtracting from fdblocks all the free blocks in this AG, hiding those blocks from the allocator, and preventing freed blocks from updating fdblocks until we're ready to lift noalloc mode.
Note that we reduce the free block count of the filesystem so that we can prevent transactions from entering the allocator looking for "free" space that we've turned off incore.
--- fs/xfs/libxfs/xfs_ag.c | 60 +++++++++++++++++++++++++++++++++++++++++++ fs/xfs/libxfs/xfs_ag.h | 8 ++++++ fs/xfs/libxfs/xfs_ag_resv.c | 27 +++++++++++++++++-- fs/xfs/scrub/fscounters.c | 3 +- fs/xfs/xfs_fsops.c | 10 ++++++- fs/xfs/xfs_super.c | 1 + fs/xfs/xfs_trace.h | 46 +++++++++++++++++++++++++++++++++ 7 files changed, 150 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_ag.c +60 -0
@@ -976,3 +976,63 @@ xfs_ag_get_geometry(xfs_buf_relse (agi_bp);return error; } + +/* How many blocks does this AG contribute to fdblocks? */ +xfs_extlen_t +xfs_ag_fdblocks ( +struct xfs_perag *pag) +{ +xfs_extlen_t ret; + +ASSERT (xfs_perag_initialised_agf (pag)); + + ret = pag->pagf_freeblks + pag->pagf_flcount + pag->pagf_btreeblks; + ret -= pag->pag_meta_resv.ar_reserved; + ret -= pag->pag_rmapbt_resv.ar_orig_reserved; +return ret; +} + +/* +* Hide all the free space in this AG. Caller must hold both the AGI and the +* AGF buffers or have otherwise prevented concurrent access. +*/ +int +xfs_ag_set_noalloc ( +struct xfs_perag *pag) +{ +struct xfs_mount *mp =pag_mount (pag); +int error; + +ASSERT (xfs_perag_initialised_agf (pag)); +ASSERT (xfs_perag_initialised_agi (pag)); + +if (xfs_perag_prohibits_alloc (pag)) +return 0 ; + + error =xfs_dec_fdblocks (mp,xfs_ag_fdblocks (pag), false); +if (error) +return error; + +trace_xfs_ag_set_noalloc (pag); +set_bit (XFS_AGSTATE_NOALLOC , &pag->pag_opstate); +return 0 ; +} + +/* +* Unhide all the free space in this AG. Caller must hold both the AGI and +* the AGF buffers or have otherwise prevented concurrent access. +*/ +void +xfs_ag_clear_noalloc ( +struct xfs_perag *pag) +{ +struct xfs_mount *mp =pag_mount (pag); + +if (!xfs_perag_prohibits_alloc (pag)) +return ; + +xfs_add_fdblocks (mp,xfs_ag_fdblocks (pag)); + +trace_xfs_ag_clear_noalloc (pag); +clear_bit (XFS_AGSTATE_NOALLOC , &pag->pag_opstate); +}
fs/xfs/libxfs/xfs_ag.h +8 -0
@@ -120,6 +120,7 @@ static inline xfs_agnumber_t pag_agno(const struct xfs_perag *pag)#define XFS_AGSTATE_PREFERS_METADATA 2#define XFS_AGSTATE_ALLOWS_INODES 3#define XFS_AGSTATE_AGFL_NEEDS_RESET 4 +#define XFS_AGSTATE_NOALLOC 5#define __XFS_AG_OPSTATE (name,NAME ) \ static inline bool xfs_perag_ ## name (struct xfs_perag *pag) \
@@ -132,6 +133,7 @@ __XFS_AG_OPSTATE(initialised_agi, AGI_INIT) __XFS_AG_OPSTATE(prefers_metadata, PREFERS_METADATA) __XFS_AG_OPSTATE(allows_inodes, ALLOWS_INODES) __XFS_AG_OPSTATE(agfl_needs_reset, AGFL_NEEDS_RESET) +__XFS_AG_OPSTATE(prohibits_alloc, NOALLOC) int xfs_initialize_perag(struct xfs_mount *mp, xfs_agnumber_t orig_agcount, xfs_agnumber_t new_agcount, xfs_rfsblock_t dcount,
@@ -164,6 +166,7 @@ xfs_perag_put(xfs_group_put (pag_group (pag)); } +/* Active AG references */ static inline struct xfs_perag * xfs_perag_grab(
@@ -208,6 +211,11 @@ xfs_perag_next(return xfs_perag_next_from (mp, pag,0 ); } +/* Enable or disable allocation from an AG */ +xfs_extlen_t xfs_ag_fdblocks (struct xfs_perag *pag); +int xfs_ag_set_noalloc (struct xfs_perag *pag); +void xfs_ag_clear_noalloc (struct xfs_perag *pag); +/* * Per-ag geometry infomation and validation */
fs/xfs/libxfs/xfs_ag_resv.c +24 -3
@@ -74,6 +74,13 @@ xfs_ag_resv_critical(xfs_extlen_t avail;xfs_extlen_t orig; +/* +* Pretend we're critically low on reservations in this AG to scare +* everyone else away. +*/ +if (xfs_perag_prohibits_alloc (pag)) +return true; +switch (type) {case XFS_AG_RESV_METADATA : avail = pag->pagf_freeblks - pag->pag_rmapbt_resv.ar_reserved;
@@ -116,7 +123,12 @@ xfs_ag_resv_needed(break ;case XFS_AG_RESV_METAFILE :case XFS_AG_RESV_NONE : -/* empty */ +/* +* In noalloc mode, we pretend that all the free blocks in this +* AG have been allocated. Make this AG look full. +*/ +if (xfs_perag_prohibits_alloc (pag)) + len +=xfs_ag_fdblocks (pag);break ;default :ASSERT (0 );
@@ -344,6 +356,8 @@ xfs_ag_resv_alloc_extent(xfs_extlen_t len;uint field; +ASSERT (type !=XFS_AG_RESV_NONE || !xfs_perag_prohibits_alloc (pag)); +trace_xfs_ag_resv_alloc_extent (pag, type, args->len);switch (type) {
@@ -401,7 +415,14 @@ xfs_ag_resv_free_extent(ASSERT (0 ); fallthrough;case XFS_AG_RESV_NONE : -xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , (int64_t )len); +/* +* Normally we put freed blocks back into fdblocks. In noalloc +* mode, however, we pretend that there are no fdblocks in the +* AG, so don't put them back. +*/ +if (!xfs_perag_prohibits_alloc (pag)) +xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , + (int64_t )len); fallthrough;case XFS_AG_RESV_IGNORE :return ;
@@ -414,6 +435,6 @@ xfs_ag_resv_free_extent(/* Freeing into the reserved pool only requires on-disk update... */ xfs_trans_mod_sb (tp,XFS_TRANS_SB_RES_FDBLOCKS , len);/* ...but freeing beyond that requires in-core and on-disk update. */ -if (len > leftover) +if (len > leftover && !xfs_perag_prohibits_alloc (pag))xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , len - leftover); }
fs/xfs/scrub/fscounters.c +2 -1
@@ -337,7 +337,8 @@ xchk_fscount_aggregate_agcounts(*/ fsc->fdblocks -= pag->pag_meta_resv.ar_reserved; fsc->fdblocks -= pag->pag_rmapbt_resv.ar_orig_reserved; - +if (xfs_perag_prohibits_alloc (pag)) + fsc->fdblocks -=xfs_ag_fdblocks (pag); }if (pag)xfs_perag_rele (pag);
fs/xfs/xfs_fsops.c +9 -1
@@ -592,6 +592,14 @@ xfs_fs_unreserve_ag_blocks(if (xfs_has_realtime (mp))xfs_rt_resv_free (mp); -while ((pag =xfs_perag_next (mp, pag))) +while ((pag =xfs_perag_next (mp, pag))) { +/* +* Bring the AG back online because our AG hiding only exists +* in-core and we need the superblock to be written out with +* the super fdblocks reflecting the AGF freeblks. Do this +* before adding the per-AG reservations back to fdblocks. +*/ +xfs_ag_clear_noalloc (pag);xfs_ag_resv_free (pag); + } }
fs/xfs/xfs_super.c +1 -0
@@ -336,6 +336,7 @@ xfs_set_inode_alloc( pag =xfs_perag_get (mp, index);if (xfs_set_inode_alloc_perag (pag, ino, max_metadata)) maxagi++; +clear_bit (XFS_AGSTATE_NOALLOC , &pag->pag_opstate);xfs_perag_put (pag); }
fs/xfs/xfs_trace.h +46 -0
52 lines not on this page · raw message
PATCH 4/5 xfs: enable userspace to hide an AG from allocation
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add an administrative interface so that userspace can hide an allocation group from block allocation.
--- fs/xfs/libxfs/xfs_ag.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/libxfs/xfs_fs.h | 5 ++++ fs/xfs/xfs_ioctl.c | 4 +++- 3 files changed, 62 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_ag.c +54 -0
@@ -932,6 +932,54 @@ xfs_ag_extend_space(return 0 ; } +/* Compute the AG geometry flags. */ +static inline uint32_t +xfs_ag_calc_geoflags ( +struct xfs_perag *pag) +{ +uint32_t ret =0 ; + +if (xfs_perag_prohibits_alloc (pag)) + ret |=XFS_AG_FLAG_NOALLOC ; + +return ret; +} + +/* +* Compare the current AG geometry flags against the flags in the AG geometry +* structure and update the AG state to reflect any changes, then update the +* struct to reflect the current status. +*/ +static inline int +xfs_ag_update_geoflags ( +struct xfs_perag *pag, +struct xfs_ag_geometry *ageo, +uint32_t new_flags) +{ +uint32_t old_flags =xfs_ag_calc_geoflags (pag); +int error; + +if (!(new_flags &XFS_AG_FLAG_UPDATE )) { + ageo->ag_flags = old_flags; +return 0 ; + } + +if ((old_flags &XFS_AG_FLAG_NOALLOC ) && + !(new_flags &XFS_AG_FLAG_NOALLOC )) { +xfs_ag_clear_noalloc (pag); + } + +if (!(old_flags &XFS_AG_FLAG_NOALLOC ) && + (new_flags &XFS_AG_FLAG_NOALLOC )) { + error =xfs_ag_set_noalloc (pag); +if (error) +return error; + } + + ageo->ag_flags =xfs_ag_calc_geoflags (pag); +return 0 ; +} +/* Retrieve AG geometry. */ int xfs_ag_get_geometry(
@@ -943,6 +991,7 @@ xfs_ag_get_geometry(struct xfs_agi *agi;struct xfs_agf *agf;unsigned int freeblks; +uint32_t inflags = ageo->ag_flags;int error;/* Lock the AG headers. */
@@ -953,6 +1002,10 @@ xfs_ag_get_geometry(if (error) goto out_agi; + error =xfs_ag_update_geoflags (pag, ageo, inflags); +if (error) + goto out; +/* Fill out form. */ memset (ageo,0 ,sizeof (*ageo)); ageo->ag_number =pag_agno (pag);
@@ -970,6 +1023,7 @@ xfs_ag_get_geometry( ageo->ag_freeblks = freeblks;xfs_ag_geom_health (pag, ageo); +out:/* Release resources. */ xfs_buf_relse (agf_bp); out_agi:
fs/xfs/libxfs/xfs_fs.h +5 -0
@@ -307,6 +307,11 @@ struct xfs_ag_geometry {#define XFS_AG_GEOM_SICK_REFCNTBT (1 << 9)/* reference counts */ #define XFS_AG_GEOM_SICK_INODES (1 << 10)/* bad inodes were seen */ +#define XFS_AG_FLAG_UPDATE (1 << 0)/* update flags */ +#define XFS_AG_FLAG_NOALLOC (1 << 1)/* do not allocate from this AG */ +#define XFS_AG_FLAG_ALL (XFS_AG_FLAG_UPDATE | \ + XFS_AG_FLAG_NOALLOC) +/* * Structures for XFS_IOC_FSGROWFSDATA, XFS_IOC_FSGROWFSLOG & XFS_IOC_FSGROWFSRT */
fs/xfs/xfs_ioctl.c +3 -1
@@ -385,10 +385,12 @@ xfs_ioc_ag_geometry(if (copy_from_user (&ageo, arg,sizeof (ageo)))return -EFAULT ; -if (ageo.ag_flags) +if (ageo.ag_flags & ~XFS_AG_FLAG_ALL )return -EINVAL ;if (memchr_inv (&ageo.ag_reserved,0 ,sizeof (ageo.ag_reserved)))return -EINVAL ; +if ((ageo.ag_flags &XFS_AG_FLAG_UPDATE ) && !capable (CAP_SYS_ADMIN )) +return -EPERM ; pag =xfs_perag_get (mp, ageo.ag_number);if (!pag)
PATCH 5/5 xfs: apply noalloc mode to inode allocations too
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Don't allow inode allocations from this group if it's marked noalloc.
--- fs/xfs/libxfs/xfs_ialloc.c | 3 +++ 1 file changed, 3 insertions(+)
fs/xfs/libxfs/xfs_ialloc.c +3 -0
@@ -1107,6 +1107,7 @@ xfs_dialloc_ag_inobt(ASSERT (xfs_perag_initialised_agi (pag));ASSERT (xfs_perag_allows_inodes (pag)); +ASSERT (!xfs_perag_prohibits_alloc (pag));ASSERT (pag->pagi_freecount >0 ); restart_pagno:
@@ -1735,6 +1736,8 @@ xfs_dialloc_good_ag(return false;if (!xfs_perag_allows_inodes (pag))return false; +if (xfs_perag_prohibits_alloc (pag)) +return false;if (!xfs_perag_initialised_agi (pag)) { error =xfs_ialloc_read_agi (pag, tp,0 ,NULL );
PATCH 3/5 xfs: report refcount information to userspace
In reply to Darrick J. WongHi all,
Create a new ioctl to report the number of owners of each disk block so that reflink-aware defraggers can make better decisions about which extents to target.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=report-refcounts
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=report-refcounts
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=report-refcounts
---
Commits in this patchset:
* xfs: export reference count information to userspace
--- fs/xfs/Makefile | 1 fs/xfs/libxfs/xfs_fs.h | 80 +++++ fs/xfs/xfs_fsrefs.c | 777 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_fsrefs.h | 45 +++ fs/xfs/xfs_ioctl.c | 4 fs/xfs/xfs_trace.c | 1 fs/xfs/xfs_trace.h | 125 ++++++++ 7 files changed, 1033 insertions(+) create mode 100644 fs/xfs/xfs_fsrefs.c create mode 100644 fs/xfs/xfs_fsrefs.h
PATCH 1/1 xfs: export reference count information to userspace
From: Darrick J. Wong <[email protected]>
Export refcount info to userspace so we can prototype a sharing-aware defrag/fs rearranging tool.
--- fs/xfs/Makefile | 1 fs/xfs/libxfs/xfs_fs.h | 80 +++++ fs/xfs/xfs_fsrefs.c | 777 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_fsrefs.h | 45 +++ fs/xfs/xfs_ioctl.c | 4 fs/xfs/xfs_trace.c | 1 fs/xfs/xfs_trace.h | 125 ++++++++ 7 files changed, 1033 insertions(+) create mode 100644 fs/xfs/xfs_fsrefs.c create mode 100644 fs/xfs/xfs_fsrefs.h
fs/xfs/Makefile +1 -0
@@ -85,6 +85,7 @@ xfs-y += xfs_aops.o \ xfs_filestream.o \ xfs_fsmap.o \ xfs_fsops.o \ + xfs_fsrefs.o \ xfs_globals.o \ xfs_handle.o \ xfs_health.o \
fs/xfs/libxfs/xfs_fs.h +80 -0
92 lines not on this page · raw message
fs/xfs/xfs_fsrefs.c new file +777 -0
777 lines not on this page · raw message
fs/xfs/xfs_fsrefs.h new file +45 -0
45 lines not on this page · raw message
fs/xfs/xfs_ioctl.c +4 -0
@@ -29,6 +29,7 @@#include "xfs_btree.h" #include <linux/fsmap.h> #include "xfs_fsmap.h" +#include "xfs_fsrefs.h" #include "scrub/xfs_scrub.h" #include "xfs_sb.h" #include "xfs_ag.h"
@@ -1266,6 +1267,9 @@ xfs_file_ioctl(case FS_IOC_GETFSMAP :return xfs_ioc_getfsmap (ip, arg); +case XFS_IOC_GETFSREFCOUNTS : +return xfs_ioc_getfsrefcounts (ip, arg); +case XFS_IOC_SCRUBV_METADATA :return xfs_ioc_scrubv_metadata (filp, arg);case XFS_IOC_SCRUB_METADATA :
fs/xfs/xfs_trace.c +1 -0
@@ -51,6 +51,7 @@#include "xfs_rtgroup.h" #include "xfs_zone_alloc.h" #include "xfs_zone_priv.h" +#include "xfs_fsrefs.h" /* * We include this last to have the helpers above available for the trace
fs/xfs/xfs_trace.h +125 -0
137 lines not on this page · raw message
PATCH 3/5 xfsprogs: defragment free space
In reply to Darrick J. WongHi all,
These patches contain experimental code to enable userspace to defragment the free space in a filesystem. Two purposes are imagined for this
functionality: clearing space at the end of a filesystem before shrinking it, and clearing free space in anticipation of making a large allocation.
The first patch adds a new fallocate mode that allows userspace to allocate free space from the filesystem into a file. The goal here is to allow the filesystem shrink process to prevent allocation from a certain part of the filesystem while a free space defragmenter evacuates all the files from the doomed part of the filesystem.
The second patch amends the online repair system to allow the sysadmin to forcibly rebuild metadata structures, even if they're not corrupt. Without adding an ioctl to move metadata btree blocks, this is the only way to dislodge metadata.
This patchset also includes a separate inode migration tool as prototyped by Dave Chinner in 2020.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=defrag-freespace
---
Commits in this patchset:
* xfs_io: display rtgroup number in verbose fsrefs output
* xfs: add an ioctl to map free space into a file
* xfs_io: support using XFS_IOC_MAP_FREESP to map free space
* xfs_db: get and put blocks on the AGFL
* xfs_spaceman: implement clearing free space
* spaceman: physically move a regular inode
* spaceman: find owners of space in an AG
* xfs_spaceman: wrap radix tree accesses in find_owner.c
* xfs_spaceman: port relocation structure to 32-bit systems
* spaceman: relocate the contents of an AG
* spaceman: move inodes with hardlinks
--- configure.ac | 1 db/agfl.c | 297 +++- include/builddefs.in | 1 include/xfs_trace.h | 4 io/fsrefcounts.c | 22 io/prealloc.c | 35 libfrog/Makefile | 5 libfrog/clearspace.c | 3294 +++++++++++++++++++++++++++++++++++++++ libfrog/clearspace.h | 79 + libfrog/fsgeom.h | 29 libfrog/radix-tree.c | 2 libfrog/radix-tree.h | 2 libxfs/libxfs_api_defs.h | 4 libxfs/libxfs_priv.h | 9 libxfs/xfs_alloc.c | 88 + libxfs/xfs_alloc.h | 3 libxfs/xfs_fs.h | 14 m4/package_libcdev.m4 | 20 man/man2/ioctl_xfs_map_freesp.2 | 76 + man/man8/xfs_db.8 | 11 man/man8/xfs_io.8 | 8 man/man8/xfs_spaceman.8 | 40 spaceman/Makefile | 11 spaceman/clearfree.c | 171 ++ spaceman/find_owner.c | 442 +++++ spaceman/init.c | 7 spaceman/move_inode.c | 662 ++++++++ spaceman/relocation.c | 566 +++++++ spaceman/relocation.h | 53 + spaceman/space.h | 6 30 files changed, 5953 insertions(+), 9 deletions(-) create mode 100644 libfrog/clearspace.c create mode 100644 libfrog/clearspace.h create mode 100644 man/man2/ioctl_xfs_map_freesp.2 create mode 100644 spaceman/clearfree.c create mode 100644 spaceman/find_owner.c create mode 100644 spaceman/move_inode.c create mode 100644 spaceman/relocation.c create mode 100644 spaceman/relocation.h
PATCH 1/11 xfs_io: display rtgroup number in verbose fsrefs output
From: Darrick J. Wong <[email protected]>
Display the rtgroup number in the verbose fsrefcounts output.
--- io/fsrefcounts.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
io/fsrefcounts.c +21 -1
@@ -13,6 +13,7 @@static cmdinfo_t fsrefcounts_cmd;static dev_t xfs_data_dev; +static dev_t xfs_rt_dev;static void fsrefcounts_help (void )
@@ -119,7 +120,7 @@ dump_refcounts_verbose(unsigned long long i;struct xfs_getfsrefs *p;int agno; -off_t agoff, bperag; +off_t agoff, bperag, bperrtg;int boff_w, aoff_w, tot_w, agno_w, own_w;int nr_w, dev_w;char bbuf[40 ], abuf[40 ], obuf[40 ];
@@ -132,6 +133,7 @@ dump_refcounts_verbose( nr_w =4 ; tot_w =MINTOT_WIDTH ; bperag = (off_t )fsgeo->agblocks * (off_t )fsgeo->blocksize; + bperrtg =bytes_per_rtgroup (fsgeo); sunit = (fsgeo->sunit * fsgeo->blocksize); swidth = (fsgeo->swidth * fsgeo->blocksize);
@@ -173,6 +175,13 @@ dump_refcounts_verbose("(%lld..%lld)" , (long long )BTOBBT (agoff), (long long)BTOBBT(agoff + p->fcr_length - 1)); + }else if (p->fcr_device == xfs_rt_dev && fsgeo->rgcount >0 ) { + agno = p->fcr_physical / bperrtg; + agoff = p->fcr_physical - (agno * bperrtg); +snprintf (abuf,sizeof (abuf), +"(%lld..%lld)" , + (long long )BTOBBT (agoff), + (long long )BTOBBT (agoff + p->fcr_length -1 )); }else abuf[0 ] =0 ; aoff_w =max (aoff_w,strlen (abuf));
@@ -231,6 +240,16 @@ dump_refcounts_verbose(snprintf (gbuf,sizeof (gbuf),"%lld" , (long long )agno); + }else if (p->fcr_device == xfs_rt_dev && fsgeo->rgcount >0 ) { + agno = p->fcr_physical / bperrtg; + agoff = p->fcr_physical - (agno * bperrtg); +snprintf (abuf,sizeof (abuf), +"(%lld..%lld)" , + (long long )BTOBBT (agoff), + (long long )BTOBBT (agoff + p->fcr_length -1 )); +snprintf (gbuf,sizeof (gbuf), +"%lld" , + (long long )agno); }else { abuf[0 ] =0 ; gbuf[0 ] =0 ;
@@ -420,6 +439,7 @@ fsrefcounts_f( } fs =fs_table_lookup (file->name,FS_MOUNT_POINT ); xfs_data_dev = fs ? fs->fs_datadev :0 ; + xfs_rt_dev = fs ? fs->fs_rtdev :0 ; head->fch_count = map_size; do {
PATCH 2/11 xfs: add an ioctl to map free space into a file
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new ioctl to map free physical space into a file, at the same file offset as if the file were a sparse image of the physical device backing the filesystem. The intent here is to use this to prototype a free space defragmentation tool.
--- include/xfs_trace.h | 4 ++ libxfs/libxfs_priv.h | 9 ++++ libxfs/xfs_alloc.c | 88 +++++++++++++++++++++++++++++++++++++++ libxfs/xfs_alloc.h | 3 + libxfs/xfs_fs.h | 14 ++++++ man/man2/ioctl_xfs_map_freesp.2 | 76 ++++++++++++++++++++++++++++++++++ 6 files changed, 194 insertions(+) create mode 100644 man/man2/ioctl_xfs_map_freesp.2
include/xfs_trace.h +4 -0
@@ -26,6 +26,8 @@#define trace_xfs_alloc_exact_done (a) ((void) 0)#define trace_xfs_alloc_exact_notfound (a) ((void) 0)#define trace_xfs_alloc_exact_error (a) ((void) 0) +#define trace_xfs_alloc_find_freesp (...) ((void) 0) +#define trace_xfs_alloc_find_freesp_done (...) ((void) 0)#define trace_xfs_alloc_near_first (a) ((void) 0)#define trace_xfs_alloc_near_greater (a) ((void) 0)#define trace_xfs_alloc_near_lesser (a) ((void) 0)
@@ -197,6 +199,8 @@#define trace_xfs_bmap_pre_update (a,b,c,d) ((void) 0)#define trace_xfs_bmap_post_update (a,b,c,d) ((void) 0) +#define trace_xfs_bmapi_freesp (...) ((void) 0) +#define trace_xfs_bmapi_freesp_done (...) ((void) 0)#define trace_xfs_bunmap (a,b,c,d,e) ((void) 0)#define trace_xfs_read_extent (a,b,c,d) ((void) 0)
libxfs/libxfs_priv.h +9 -0
@@ -446,6 +446,15 @@ xfs_buf_readahead(#define xfs_filestream_new_ag (ip,ag) (0)#define xfs_filestream_select_ag (...) (-ENOSYS) +struct xfs_trans ; + +static inline int +xfs_rtallocate_extent (struct xfs_trans *tp,xfs_rtxnum_t start, +xfs_rtxlen_t maxlen,xfs_rtxlen_t *len,xfs_rtxnum_t *rtx) +{ +return -EOPNOTSUPP ; +} +#define xfs_trans_inode_buf (tp, bp) ((void) 0)/* quota bits */
libxfs/xfs_alloc.c +88 -0
91 lines not on this page · raw message
libxfs/xfs_alloc.h +3 -0
@@ -286,5 +286,8 @@ void xfs_extfree_intent_destroy_cache(void);xfs_failaddr_t xfs_validate_ag_length (struct xfs_buf *bp,uint32_t seqno,uint32_t length); +int xfs_alloc_find_freesp (struct xfs_trans *tp,struct xfs_perag *pag, +xfs_agblock_t *agbno,xfs_agblock_t end_agbno, +xfs_extlen_t *len);#endif /* __XFS_ALLOC_H__ */
libxfs/xfs_fs.h +14 -0
@@ -1087,6 +1087,19 @@ xfs_getfsrefs_advance(/* fcr_flags values - returned for each non-header segment */ #define FCR_OF_LAST (1U << 0)/* last record in the dataset */ +/* map free space to file */ + +/* +* XFS_IOC_MAP_FREESP maps all the free physical space in the filesystem into +* the file at the same offsets. This ioctl requires CAP_SYS_ADMIN. +*/ +struct xfs_map_freesp { +__s64 offset;/* disk address to map, in bytes */ +__s64 len;/* length in bytes */ +__u64 flags;/* must be zero */ +__u64 pad;/* must be zero */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1127,6 +1140,7 @@ xfs_getfsrefs_advance(#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_GETFSREFCOUNTS _IOWR('X', 66, struct xfs_getfsrefs_head) +#define XFS_IOC_MAP_FREESP _IOW ('X', 67, struct xfs_map_freesp)/* * ioctl commands that replace IRIX syssgi()'s
man/man2/ioctl_xfs_map_freesp.2 new file +76 -0
76 lines not on this page · raw message
PATCH 3/11 xfs_io: support using XFS_IOC_MAP_FREESP to map free space
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a command to call XFS_IOC_MAP_FREESP. This is experimental code to see if we can build a free space defragmenter out of this.
--- io/prealloc.c | 35 +++++++++++++++++++++++++++++++++++ man/man8/xfs_io.8 | 8 +++++++- 2 files changed, 42 insertions(+), 1 deletion(-)
io/prealloc.c +35 -0
@@ -41,6 +41,7 @@ static cmdinfo_t fcollapse_cmd;static cmdinfo_t finsert_cmd;static cmdinfo_t fzero_cmd;static cmdinfo_t funshare_cmd; +static cmdinfo_t fmapfree_cmd;static int offset_length(
@@ -377,6 +378,30 @@ funshare_f(return 0 ; } +static int +fmapfree_f ( +int argc, +char **argv) +{ +struct xfs_flock64 segment; +struct xfs_map_freesp args = { }; + +if (!offset_length (argv[1 ], argv[2 ], &segment)) { + exitcode =1 ; +return 0 ; + } + + args.offset = segment.l_start; + args.len = segment.l_len; + +if (ioctl (file->fd,XFS_IOC_MAP_FREESP , &args)) { +perror ("XFS_IOC_MAP_FREESP" ); + exitcode =1 ; +return 0 ; + } +return 0 ; +} +void prealloc_init (void ) {
@@ -489,4 +514,14 @@ prealloc_init(void) funshare_cmd.oneline =_ ("unshares shared blocks within the range" );add_command (&funshare_cmd); + + fmapfree_cmd.name ="fmapfree" ; + fmapfree_cmd.cfunc = fmapfree_f; + fmapfree_cmd.argmin =2 ; + fmapfree_cmd.argmax =2 ; + fmapfree_cmd.flags =CMD_NOMAP_OK |CMD_FOREIGN_OK ; + fmapfree_cmd.args =_ ("off len" ); + fmapfree_cmd.oneline = +_ ("maps free space into a file" ); +add_command (&fmapfree_cmd); }
man/man8/xfs_io.8 +7 -1
@@ -519,8 +519,14 @@ .SH FILE I/O COMMANDS .BR fallocate (2) manual page to create the hole by shifting data blocks. .TP +.BI fmapfree " offset length" +Maps free physical space into the file by calling XFS_IOC_MAP_FREESP as +described in the +.BR XFS_IOC_MAP_FREESP (2) +manual page. +.TP .BI fpunch " offset length" -Punches (de-allocates) blocks in the file by calling fallocate with +Punches (de-allocates) blocks in the file by calling fallocate with the FALLOC_FL_PUNCH_HOLE flag as described in the .BR fallocate (2) manual page.
PATCH 4/11 xfs_db: get and put blocks on the AGFL
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new xfs_db command to let people add and remove blocks from an AGFL. This isn't really related to rmap btree reconstruction, other than enabling debugging code to mess around with the AGFL to exercise various odd scenarios.
--- db/agfl.c | 297 ++++++++++++++++++++++++++++++++++++++++++++++ libxfs/libxfs_api_defs.h | 4 + man/man8/xfs_db.8 | 11 ++ 3 files changed, 308 insertions(+), 4 deletions(-)
db/agfl.c +294 -3
@@ -15,13 +15,14 @@#include "output.h" #include "init.h" #include "agfl.h" +#include "libfrog/bitmap.h" static int agfl_bno_size (void *obj,int startoff);static int agfl_f (int argc,char **argv);static void agfl_help (void );static const cmdinfo_t agfl_cmd = - {"agfl" ,NULL , agfl_f,0 ,1 ,1 ,N_ ("[agno]" ), + {"agfl" ,NULL , agfl_f,0 ,-1 ,1 ,N_ ("[agno] [-g nr] [-p nr]" ),N_ ("set address to agfl block" ), agfl_help };const field_t agfl_hfld[] = { {
@@ -77,10 +78,280 @@ agfl_help(void)" for each allocation group. This acts as a reserved pool of space\n" " separate from the general filesystem freespace (not used for user data).\n" "\n" +" -g quantity\tRemove this many blocks from the AGFL.\n" +" -p quantity\tAdd this many blocks to the AGFL.\n" +"\n" )); } +struct dump_info { +struct xfs_perag *pag; +bool leak; +}; + +/* Return blocks freed from the AGFL to the free space btrees. */ +static int +free_grabbed ( +uint64_t start, +uint64_t length, +void *data) +{ +struct dump_info *di = data; +struct xfs_perag *pag = di->pag; +struct xfs_mount *mp =pag_mount (pag); +struct xfs_trans *tp; +struct xfs_buf *agf_bp; +int error; + + error = -libxfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate,0 ,0 ,0 , + &tp); +if (error) +return error; + + error = -libxfs_alloc_read_agf (pag, tp,0 , &agf_bp); +if (error) + goto out_cancel; + + error = -libxfs_free_extent (tp, pag, start, length, &XFS_RMAP_OINFO_AG , +XFS_AG_RESV_AGFL ); +if (error) + goto out_cancel; + +return -libxfs_trans_commit (tp); + +out_cancel: +libxfs_trans_cancel (tp); +return error; +} + +/* Report blocks freed from the AGFL. */ +static int +dump_grabbed ( +uint64_t start, +uint64_t length, +void *data) +{ +struct dump_info *di = data; +const char *fmt; + +if (length ==1 ) + fmt = di->leak ?_ ("agfl %u: leaked agbno %u\n" ) : +_ ("agfl %u: removed agbno %u\n" ); +else + fmt = di->leak ?_ ("agfl %u: leaked agbno %u-%u\n" ) : +_ ("agfl %u: removed agbno %u-%u\n" ); + +printf (fmt,pag_agno (di->pag), (unsigned int )start, + (unsigned int )(start + length -1 )); +return 0 ; +} + +/* Remove blocks from the AGFL. */ +static int +agfl_get ( +struct xfs_perag *pag, +int quantity) +{ +struct dump_info di = { + .pag = pag, + .leak = quantity <0 , + }; +struct xfs_agf *agf; +struct xfs_buf *agf_bp; +struct xfs_trans *tp; +struct bitmap *grabbed; +const unsigned int agfl_size =libxfs_agfl_size (pag_mount (pag)); +unsigned int i; +int error; + +if (!quantity) +return 0 ; + +if (di.leak) + quantity = -quantity; + quantity =min (quantity, agfl_size); + + error =bitmap_alloc (&grabbed); +if (error) + goto out; + + error = -libxfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate, quantity,0 , +0 , &tp); +if (error) + goto out_bitmap; + + error = -libxfs_alloc_read_agf (pag, tp,0 , &agf_bp); +if (error) + goto out_cancel; + + agf = agf_bp->b_addr; + quantity =min (quantity,be32_to_cpu (agf->agf_flcount)); + +for (i =0 ; i < quantity; i++) { +xfs_agblock_t agbno; + + error = -libxfs_alloc_get_freelist (pag, tp, agf_bp, &agbno,0 ); +if (error) + goto out_cancel; + +if (agbno ==NULLAGBLOCK ) { + error =ENOSPC ; + goto out_cancel; + } + + error =bitmap_set (grabbed, agbno,1 ); +if (error) + goto out_cancel; + } + + error = -libxfs_trans_commit (tp); +if (error) + goto out_bitmap; + + error =bitmap_iterate (grabbed, dump_grabbed, &di); +if (error) + goto out_bitmap; + +if (!di.leak) { + error =bitmap_iterate (grabbed, free_grabbed, &di); +if (error) + goto out_bitmap; + } + +bitmap_free (&grabbed); +return 0 ; + +out_cancel: +libxfs_trans_cancel (tp); +out_bitmap: +bitmap_free (&grabbed); +out: +if (error) +printf (_ ("agfl %u: %s\n" ),pag_agno (pag),strerror (error)); +return error; +} + +/* Add blocks to the AGFL. */ +static int +agfl_put ( +struct xfs_perag *pag, +int quantity) +{ +struct xfs_alloc_arg args = { + .mp =pag_mount (pag), + .alignment =1 , + .minlen =1 , + .prod =1 , + .resv =XFS_AG_RESV_AGFL , + .oinfo =XFS_RMAP_OINFO_AG , + }; +struct xfs_buf *agfl_bp; +struct xfs_agf *agf; +struct xfs_trans *tp; +xfs_fsblock_t target; +const unsigned int agfl_size =libxfs_agfl_size (pag_mount (pag)); +unsigned int i; +bool eoag = quantity <0 ; +int error; + +if (!quantity) +return 0 ; + +if (eoag) + quantity = -quantity; + quantity =min (quantity, agfl_size); + + error = -libxfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate, quantity,0 , +0 , &tp); +if (error) +return error; + args.tp = tp; + + error = -libxfs_alloc_read_agf (pag, tp,0 , &args.agbp); +if (error) + goto out_cancel; + + agf = args.agbp->b_addr; + args.maxlen =min (quantity, agfl_size -be32_to_cpu (agf->agf_flcount)); + +if (eoag) + target =xfs_agbno_to_fsb (pag, +be32_to_cpu (agf->agf_length) -1 ); +else + target =xfs_agbno_to_fsb (pag,0 ); + + error = -libxfs_alloc_read_agfl (pag, tp, &agfl_bp); +if (error) + goto out_cancel; + + error = -libxfs_alloc_vextent_near_bno (&args, target); +if (error) + goto out_cancel; + +if (args.agbno ==NULLAGBLOCK ) { + error =ENOSPC ; + goto out_cancel; + } + +for (i =0 ; i < args.len; i++) { + error = -libxfs_alloc_put_freelist (pag, tp, args.agbp, + agfl_bp, args.agbno + i,0 ); +if (error) + goto out_cancel; + } + +if (i ==1 ) +printf (_ ("agfl %u: added agbno %u\n" ),pag_agno (pag), + args.agbno); +else if (i >1 ) +printf (_ ("agfl %u: added agbno %u-%u\n" ),pag_agno (pag), + args.agbno, args.agbno + i -1 ); + + error = -libxfs_trans_commit (tp); +if (error) + goto out; + +return 0 ; + +out_cancel: +libxfs_trans_cancel (tp); +out: +if (error) +printf (_ ("agfl %u: %s\n" ),pag_agno (pag),strerror (error)); +return error; +} + +static void +agfl_adjust ( +struct xfs_mount *mp, +xfs_agnumber_t agno, +int gblocks, +int pblocks) +{ +struct xfs_perag *pag; +int error; + +if (!expert_mode) { +printf (_ ("AGFL get/put only supported in expert mode.\n" )); + exitcode =1 ; +return ; + } + + pag =libxfs_perag_get (mp, agno); + + error =agfl_get (pag, gblocks); +if (error) + goto out_pag; + + error =agfl_put (pag, pblocks); + +out_pag: +libxfs_perag_put (pag); +if (error) + exitcode =1 ; +} + static int agfl_f( int argc,
@@ -88,9 +359,25 @@ agfl_f( {xfs_agnumber_t agno;char *p; +int c; +int gblocks =0 , pblocks =0 ; -if (argc >1 ) { - agno = (xfs_agnumber_t )strtoul (argv[1 ], &p,0 ); +while ((c =getopt (argc, argv,"g:p:" )) !=-1 ) { +switch (c) { +case 'g' : + gblocks =atoi (optarg); +break ; +case 'p' : + pblocks =atoi (optarg); +break ; +default : +agfl_help (); +return 1 ; + } + } + +if (argc > optind) { + agno = (xfs_agnumber_t )strtoul (argv[optind], &p,0 );if (*p !='\0' || agno >= mp->m_sb.sb_agcount) {dbprintf (_ ("bad allocation group number %s\n" ), argv[1 ]);return 0 ;
@@ -98,6 +385,10 @@ agfl_f( cur_agno = agno; }else if (cur_agno ==NULLAGNUMBER ) cur_agno =0 ; + +if (gblocks || pblocks) +agfl_adjust (mp, cur_agno, gblocks, pblocks); +ASSERT (typtab[TYP_AGFL ].typnm ==TYP_AGFL ); set_cur(&typtab[TYP_AGFL], XFS_AG_DADDR(mp, cur_agno, XFS_AGFL_DADDR(mp)),
libxfs/libxfs_api_defs.h +4 -0
@@ -31,8 +31,12 @@#define xfs_allocbt_maxrecs libxfs_allocbt_maxrecs#define xfs_allocbt_stage_cursor libxfs_allocbt_stage_cursor#define xfs_alloc_fix_freelist libxfs_alloc_fix_freelist +#define xfs_alloc_get_freelist libxfs_alloc_get_freelist#define xfs_alloc_min_freelist libxfs_alloc_min_freelist +#define xfs_alloc_put_freelist libxfs_alloc_put_freelist#define xfs_alloc_read_agf libxfs_alloc_read_agf +#define xfs_alloc_read_agfl libxfs_alloc_read_agfl +#define xfs_alloc_vextent_near_bno libxfs_alloc_vextent_near_bno#define xfs_alloc_vextent_start_ag libxfs_alloc_vextent_start_ag#define xfs_ascii_ci_hashname libxfs_ascii_ci_hashname
man/man8/xfs_db.8 +10 -1
@@ -182,10 +182,19 @@ .SH COMMANDS .IR agno . If no argument is given, use the current allocation group. .TP -.BI "agfl [" agno ] +.BI "agfl [" agno "] [\-g " " quantity" "] [\-p " quantity ] Set current address to the AGFL block for allocation group .IR agno . If no argument is given, use the current allocation group. +If the +.B -g +option is specified with a positive quantity, remove that many blocks from the +AGFL and put them in the free space btrees. +If the quantity is negative, remove the blocks and leak them. +If the +.B -p +option is specified, add that many blocks to the AGFL. +If the quantity is negative, the blocks are selected from the end of the AG. .TP .BI "agi [" agno ] Set current address to the AGI block for allocation group
PATCH 5/11 xfs_spaceman: implement clearing free space
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
First attempt at evacuating all the used blocks from part of a filesystem.
--- libfrog/Makefile | 5 libfrog/clearspace.c | 3294 +++++++++++++++++++++++++++++++++++++++++++++++ libfrog/clearspace.h | 79 + man/man8/xfs_spaceman.8 | 17 spaceman/Makefile | 2 spaceman/clearfree.c | 171 ++ spaceman/init.c | 1 spaceman/space.h | 2 8 files changed, 3570 insertions(+), 1 deletion(-) create mode 100644 libfrog/clearspace.c create mode 100644 libfrog/clearspace.h create mode 100644 spaceman/clearfree.c
libfrog/Makefile +5 -0
@@ -65,6 +65,11 @@ workqueue.h LSRCFILES += gen_crc32table.c +ifeq ($(HAVE_GETFSMAP),yes) +CFILES+=clearspace.c +HFILES+=clearspace.h +endif + LDIRT = gen_crc32table crc32table.h default: ltdepend $(LTLIBRARY)
libfrog/clearspace.c new file +3294 -0
2,989 lines not on this page · raw message
libfrog/clearspace.h new file +79 -0
79 lines not on this page · raw message
spaceman/clearfree.c new file +171 -0
171 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
PATCH 6/11 spaceman: physically move a regular inode
In reply to Darrick J. WongFrom: Dave Chinner <[email protected]>
To be able to shrink a filesystem, we need to be able to physically move an inode and all it's data and metadata from it's current location to a new AG. Add a command to spaceman to allow an inode to be moved to a new AG.
This new command is not intended to be a perfect solution. I am not trying to handle atomic movement of open files - this is intended to be run as a maintenance operation on idle filesystem. If root filesystems are the target, then this should be run via a rescue environment that is not executing directly on the root fs. With those caveats in place, we can do the entire inode move as a set of non-destructive operations finalised by an atomic inode swap without any needing special kernel support.
To ensure we move metadata such as BMBT blocks even if we don't need to move data, we clone the data to a new inode that we've allocated in the destination AG. This will result in new bmbt blocks being allocated in the new location even though the data is not copied. Attributes need to be copied one at a time from the original inode.
If data needs to be moved, then we use fallocate(UNSHARE) to create a private copy of the range of data that needs to be moved in the new inode. This will be allocated in the destination AG by normal allocation policy.
Once the new inode has been finalised, use RENAME_EXCHANGE to swap it into place and unlink the original inode to free up all the resources it still pins.
There are many optimisations still possible to speed this up, but the goal here is "functional" rather than "optimal". Performance can be optimised once all the parts for a "empty the tail of the filesystem before shrink" operation are implemented and solidly tested.
This functionality has been smoke tested by creating a 32MB data file with 4k extents and several hundred attributes:
$ cat test.sh fname=/mnt/scratch/foo xfs_io -f -c "pwrite 0 32m" -c sync $fname for (( i=0; i < 4096 ; i++ )); do xfs_io -c "fpunch $((i * 8))k 4k" $fname done
for (( i=0; i < 100 ; i++ )); do
setfattr -n user.blah.$i.$i.blah -v blah.$i.$i.blah $fname setfattr -n user.foo.$i.$i.foo -v $i.cantbele.$i.ve.$i.tsnotbutter $fname
done
for (( i=0; i < 100 ; i++ )); do
setfattr -n security.baz.$i.$i.baz -v wotchul$i$iookinat $fname
done
xfs_io -c stat -c "bmap -vp" -c "bmap -avp" $fname xfs_spaceman -c "move_inode -a 22" /mnt/scratch/foo xfs_io -c stat -c "bmap -vp" -c "bmap -avp" $fname $
and the output looks something like:
$ sudo ./test.sh .... fd.path = "/mnt/scratch/foo" fd.flags = non-sync,non-direct,read-write stat.ino = 133 /mnt/scratch/foo: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..7]: hole 8 1: [8..15]: 208..215 0 (208..215) 8 000000 2: [16..23]: hole 8 3: [24..31]: 224..231 0 (224..231) 8 000000 .... 8189: [65512..65519]: 65712..65719 0 (65712..65719) 8 000000 8190: [65520..65527]: hole 8 8191: [65528..65535]: 65728..65735 0 (65728..65735) 8 000000 mnt/scratch/foo: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..7]: 392..399 0 (392..399) 8 000000 1: [8..15]: 408..415 0 (408..415) 8 000000 2: [16..23]: 424..431 0 (424..431) 8 000000 3: [24..31]: 456..463 0 (456..463) 8 000000 move mnt /mnt/scratch, path /mnt/scratch/foo, agno 22 fd.path = "/mnt/scratch/foo" fd.flags = non-sync,non-direct,read-write stat.ino = 47244651475 .... /mnt/scratch/foo: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..7]: hole 8 1: [8..15]: 47244763192..47244763199 22 (123112..123119) 8 000000 2: [16..23]: hole 8 3: [24..31]: 47244763208..47244763215 22 (123128..123135) 8 000000 .... 8189: [65512..65519]: 47244828808..47244828815 22 (188728..188735) 8 000000 8190: [65520..65527]: hole 8 8191: [65528..65535]: 47244828824..47244828831 22 (188744..188751) 8 000000 /mnt/scratch/foo: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..7]: 47244763176..47244763183 22 (123096..123103) 8 000000 $
--- man/man8/xfs_spaceman.8 | 4 spaceman/Makefile | 3 spaceman/init.c | 1 spaceman/move_inode.c | 562 +++++++++++++++++++++++++++++++++++++++++++++++ spaceman/space.h | 1 5 files changed, 570 insertions(+), 1 deletion(-) create mode 100644 spaceman/move_inode.c
man/man8/xfs_spaceman.8 +4 -0
@@ -146,6 +146,10 @@ .SH COMMANDS .TP .BR "help [ " command " ]" Display a brief description of one or all commands. +.TP +.BI "move_inode \-a agno" +Move the currently open file into the specified allocation group. + .TP .BI "prealloc [ \-u id ] [ \-g id ] [ -p id ] [ \-m minlen ] [ \-s ]" Removes speculative preallocation.
spaceman/Makefile +2 -1
@@ -14,11 +14,12 @@ CFILES = \ health.c \ info.c \ init.c \ + move_inode.c \ prealloc.c \ trim.c LSRCFILES = xfs_info.sh -LLDLIBS = $(LIBHANDLE) $(LIBXCMD) $(LIBFROG) +LLDLIBS = $(LIBHANDLE) $(LIBXCMD) $(LIBFROG) $(LIBHANDLE) LTDEPENDENCIES = $(LIBHANDLE) $(LIBXCMD) $(LIBFROG) LLDFLAGS = -static
spaceman/init.c +1 -0
@@ -36,6 +36,7 @@ init_commands(void)freesp_init ();health_init ();clearfree_init (); +move_inode_init (); } static int
spaceman/move_inode.c new file +562 -0
562 lines not on this page · raw message
spaceman/space.h +1 -0
@@ -38,5 +38,6 @@ extern void clearfree_init(void);#endif extern void info_init (void );extern void health_init (void ); +void move_inode_init (void );#endif /* XFS_SPACEMAN_SPACE_H_ */
PATCH 7/11 spaceman: find owners of space in an AG
In reply to Darrick J. WongFrom: Dave Chinner <[email protected]>
Before we can move inodes for a shrink operation, we have to find all the inodes that own space in the AG(s) we want to empty.
This implementation uses FS_IOC_GETFSMAP on the assumption that filesystems to be shrunk have reverse mapping enabled as it is the only way to identify inode related metadata that userspace is unable to see or influence (e.g. BMBT blocks) that may be located in the specific AG. We can use GETFSMAP to identify both inodes to be moved (via XFS_FMR_OWN_INODES records) and inodes with just data and/or metadata to be moved.
Once we have identified all the inodes to be moved, we have to map them to paths so that we can use renameat2() to exchange the directory entries pointing at the moved inode atomically. We also need to record inodes with hard links and all of the paths to the inode so that hard links can be recreated appropriately.
This requires a directory tree walk to discover the paths (until parent pointers are a thing). Hence for filesystems that aren't reverse mapping enabled, we can eventually use this pass to discover inodes with visible data and metadata that need to be moved.
As we resolve the paths to the inodes to be moved, output the information to stdout so that it can be acted upon by other utilities. This results in a command that acts similar to find but with a physical location filter rather than an inode metadata filter.
Again, this is not meant to be an optimal implementation. It shouldn't suck, but there is plenty of scope for performance optimisation, especially with a multithreaded and/or async directory traversal/parent pointer path resolution process to hide access latencies.
--- libfrog/fsgeom.h | 19 ++ libfrog/radix-tree.c | 2 libfrog/radix-tree.h | 2 man/man8/xfs_spaceman.8 | 11 + spaceman/Makefile | 1 spaceman/find_owner.c | 481 +++++++++++++++++++++++++++++++++++++++++++++++ spaceman/init.c | 4 spaceman/space.h | 2 8 files changed, 521 insertions(+), 1 deletion(-) create mode 100644 spaceman/find_owner.c
libfrog/fsgeom.h +19 -0
@@ -97,6 +97,25 @@ cvt_ino_to_agino(return ino & ((1ULL << xfd->aginolog) -1 ); } +/* Convert an AG block to an AG inode number. */ +static inline uint32_t +cvt_agbno_to_agino ( +const struct xfs_fd *xfd, +xfs_agblock_t agbno) +{ +return agbno << xfd->inopblog; +} + +/* Calculate the number of inodes in a byte range */ +static inline uint32_t +cvt_b_to_inode_count ( +const struct xfs_fd *xfd, +uint64_t bytes) +{ +return (bytes >> xfd->blocklog) << xfd->inopblog; +} + +/* * Convert a linear fs block offset number into bytes. This is the runtime * equivalent of XFS_FSB_TO_B, which means that it is /not/ for segmented fsbno
libfrog/radix-tree.c +2 -0
@@ -377,6 +377,8 @@ void *radix_tree_tag_set(struct radix_tree_root *root,unsigned int height, shift;struct radix_tree_node *slot; +ASSERT (tag <RADIX_TREE_MAX_TAGS ); + height = root->height;if (index >radix_tree_maxindex (height))return NULL ;
libfrog/radix-tree.h +1 -1
@@ -28,7 +28,7 @@ do { \ } while (0)#ifdef RADIX_TREE_TAGS -#define RADIX_TREE_MAX_TAGS 2 +#define RADIX_TREE_MAX_TAGS 3#endif int radix_tree_insert (struct radix_tree_root *,unsigned long ,void *);
man/man8/xfs_spaceman.8 +11 -0
@@ -41,6 +41,14 @@ .SH COMMANDS If the .B -v option is given, print what's happening every step of the way. +.TP +.BI "find_owner \-a agno" +Create an internal structure to map physical space in the given allocation +group to file paths. +This enables space reorganization on a mounted filesystem by enabling +us to find files. +Unclear why we can't just use FSMAP and BULKSTAT to open by handle. + .TP .BI "freesp [ \-dgrs ] [-a agno]... [ \-b | \-e bsize | \-h bsize | \-m factor ]" With no arguments,
@@ -195,6 +203,9 @@ .SH COMMANDS .B print Display a list of all open files. .TP +.B resolve_owner +Resolves space in the filesystem to file paths, maybe? +.TP .B quit Exit .BR xfs_spaceman .
spaceman/Makefile +1 -0
@@ -11,6 +11,7 @@ HFILES = \ space.h CFILES = \ file.c \ + find_owner.c \ health.c \ info.c \ init.c \
spaceman/find_owner.c new file +481 -0
481 lines not on this page · raw message
spaceman/init.c +4 -0
@@ -10,6 +10,7 @@#include "input.h" #include "init.h" #include "libfrog/paths.h" +#include "libfrog/radix-tree.h" #include "space.h" char *progname;
@@ -37,6 +38,8 @@ init_commands(void)health_init ();clearfree_init ();move_inode_init (); +find_owner_init (); +resolve_owner_init (); } static int
@@ -71,6 +74,7 @@ init(setlocale (LC_ALL ,"" );bindtextdomain (PACKAGE ,LOCALEDIR );textdomain (PACKAGE ); +radix_tree_init ();fs_table_initialise (0 ,NULL ,0 ,NULL );while ((c =getopt (argc, argv,"c:p:V" )) !=EOF ) {
spaceman/space.h +2 -0
@@ -39,5 +39,7 @@ extern void clearfree_init(void);extern void info_init (void );extern void health_init (void );void move_inode_init (void ); +void find_owner_init (void ); +void resolve_owner_init (void );#endif /* XFS_SPACEMAN_SPACE_H_ */
PATCH 8/11 xfs_spaceman: wrap radix tree accesses in find_owner.c
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Wrap the raw radix tree accesses here so that we can provide an alternate implementation on platforms where radix tree indices cannot store a full 64-bit inode number.
--- spaceman/Makefile | 1 spaceman/find_owner.c | 76 +++++++++------------------------ spaceman/relocation.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++ spaceman/relocation.h | 46 ++++++++++++++++++++ 4 files changed, 183 insertions(+), 54 deletions(-) create mode 100644 spaceman/relocation.c create mode 100644 spaceman/relocation.h
spaceman/Makefile +1 -0
@@ -17,6 +17,7 @@ CFILES = \ init.c \ move_inode.c \ prealloc.c \ + relocation.c \ trim.c LSRCFILES = xfs_info.sh
spaceman/find_owner.c +22 -54
171 lines not on this page · raw message
spaceman/relocation.c new file +114 -0
114 lines not on this page · raw message
spaceman/relocation.h new file +46 -0
46 lines not on this page · raw message
PATCH 9/11 xfs_spaceman: port relocation structure to 32-bit systems
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
We can't use the radix tree to store relocation information on 32-bit systems because unsigned longs are not large enough to hold 64-bit inodes. Use an avl64 tree instead.
--- configure.ac | 1 include/builddefs.in | 1 m4/package_libcdev.m4 | 20 +++++ spaceman/Makefile | 4 + spaceman/relocation.c | 203 +++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 229 insertions(+)
configure.ac +1 -0
@@ -212,6 +212,7 @@ fi AC_MANUAL_FORMAT AC_HAVE_LIBURCU_ATOMIC64 +AC_USE_RADIX_TREE_FOR_INUMS AC_CONFIG_FILES([include/builddefs]) AC_OUTPUT
include/builddefs.in +1 -0
@@ -114,6 +114,7 @@ CROND_DIR = @crond_dir@ HAVE_UDEV = @have_udev@ UDEV_RULE_DIR = @udev_rule_dir@ HAVE_LIBURCU_ATOMIC64 = @have_liburcu_atomic64@ +USE_RADIX_TREE_FOR_INUMS = @use_radix_tree_for_inums@ GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall # -Wbitwise -Wno-transparent-union -Wno-old-initializer -Wno-decl
m4/package_libcdev.m4 +20 -0
@@ -255,3 +255,23 @@ AC_DEFUN([AC_PACKAGE_CHECK_LTO], AC_SUBST(lto_cflags) AC_SUBST(lto_ldflags) ]) + +# +# Check if the radix tree index (unsigned long) is large enough to hold a +# 64-bit inode number +# +AC_DEFUN([AC_USE_RADIX_TREE_FOR_INUMS], + [ AC_MSG_CHECKING([if radix tree can store XFS inums]) + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#include <sys/param.h> +#include <stdint.h> +#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)])) + ]], [[ + typedef uint64_t xfs_ino_t; + + BUILD_BUG_ON(sizeof(unsigned long) < sizeof(xfs_ino_t)); + return 0; + ]])],[use_radix_tree_for_inums=yes + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) + AC_SUBST(use_radix_tree_for_inums) + ])
spaceman/Makefile +4 -0
@@ -33,6 +33,10 @@ ifeq ($(HAVE_GETFSMAP),yes) CFILES += freesp.c clearfree.c endif +ifeq ($(USE_RADIX_TREE_FOR_INUMS),yes) +LCFLAGS += -DUSE_RADIX_TREE_FOR_INUMS +endif + default: depend $(LTCOMMAND) include $(BUILDRULES)
spaceman/relocation.c +203 -0
219 lines not on this page · raw message
PATCH 10/11 spaceman: relocate the contents of an AG
In reply to Darrick J. WongFrom: Dave Chinner <[email protected]>
Shrinking a filesystem needs to first remove all the active user data and metadata from the AGs that are going to be lopped off the filesystem. Before we can do this, we have to relocate this information to a region of the filesystem that is going to be retained.
We have a function to move an inode and all it's related information to a specific AG, we have functions to find the owners of all the information in an AG and we can find their paths. This gives us all the information we need to relocate all the objects in an AG we are going to remove via shrinking.
Firstly we scan the AG to be emptied to find the inodes that need to be relocated, then we scan the directory structure to find all the paths to those inodes that need to be moved. Then we iterate over all the inodes to be moved attempting to move them to the lowest numbers AGs.
When the destination AG fills up, we'll get ENOSPC from the moving code and this is a trigger to bump the destination AG and retry the move. If we haven't moved all the inodes and their data by the time the destination reaches the source AG, then the entire operation will fail with ENOSPC - there is not enough room in the filesystem to empty the selected AG in preparation for a shrink.
This, once again, is not intended as an optimal or even guaranteed way of emptying an AG for shrink. It simply provides the basic algorithm and mechanisms we need to perform a shrink operation. Improvements and optimisations will come in time, but we can't get to an optimal solution without first having basic functionality in place.
--- libfrog/fsgeom.h | 10 ++ man/man8/xfs_spaceman.8 | 8 ++ spaceman/find_owner.c | 32 +++--- spaceman/init.c | 1 spaceman/move_inode.c | 7 + spaceman/relocation.c | 234 +++++++++++++++++++++++++++++++++++++++++++++++ spaceman/relocation.h | 5 + spaceman/space.h | 1 8 files changed, 280 insertions(+), 18 deletions(-)
libfrog/fsgeom.h +10 -0
@@ -196,6 +196,16 @@ cvt_daddr_to_agno(return cvt_bb_to_off_fsbt (xfd, daddr) / xfd->fsgeom.agblocks; } +/* Convert sparse filesystem block to AG Number */ +static inline uint32_t +cvt_fsb_to_agno ( +struct xfs_fd *xfd, +uint64_t fsbno) +{ +return fsbno >> xfd->agblklog; +} + +/* Convert sector number to AG block number. */ static inline uint32_t cvt_daddr_to_agbno(
man/man8/xfs_spaceman.8 +8 -0
@@ -202,9 +202,17 @@ .SH COMMANDS .TP .B print Display a list of all open files. +.TP +.BI "relocate \-a agno [ \-h agno ]" +Empty out the given allocation group by moving file data elsewhere. +The +.B -h +option specifies the highest allocation group into which we can move data. + .TP .B resolve_owner Resolves space in the filesystem to file paths, maybe? + .TP .B quit Exit
spaceman/find_owner.c +17 -15
@@ -9,10 +9,10 @@#include <linux/fiemap.h> #include "libfrog/fsgeom.h" #include "libfrog/radix-tree.h" -#include "command.h" -#include "init.h" #include "libfrog/paths.h" #include <linux/fsmap.h> +#include "command.h" +#include "init.h" #include "space.h" #include "input.h" #include "relocation.h"
@@ -65,8 +65,8 @@ track_inode(set_reloc_iflag (owner,MOVE_BLOCKS ); } -static void -scan_ag ( +int +find_relocation_targets (xfs_agnumber_t agno) {struct fsmap_head *fsmap;
@@ -80,8 +80,7 @@ scan_ag( fsmap =malloc (fsmap_sizeof (NR_EXTENTS ));if (!fsmap) {fprintf (stderr,_ ("%s: fsmap malloc failed.\n" ), progname); - exitcode =1 ; -return ; +return -ENOMEM ; }memset (fsmap,0 ,sizeof (*fsmap));
@@ -102,8 +101,7 @@ scan_ag(fprintf (stderr,_ ("%s: FS_IOC_GETFSMAP [\"%s\"]: %s\n" ), progname, file->name,strerror (errno));free (fsmap); - exitcode =1 ; -return ; +return -errno; }/* No more extents to map, exit */
@@ -148,6 +146,7 @@ scan_ag( }free (fsmap); +return 0 ; }/*
@@ -159,6 +158,7 @@ find_owner_f( char **argv) {xfs_agnumber_t agno =-1 ; +int ret;int c;while ((c =getopt (argc, argv,"a:" )) !=EOF ) {
@@ -198,7 +198,9 @@ _("Filesystem at %s does not have reverse mapping enabled. Aborting.\n"),return 0 ; } -scan_ag (agno); + ret =find_relocation_targets (agno); +if (ret) + exitcode =1 ;return 0 ; }
@@ -299,8 +301,8 @@ _("Aborting: Storing path %s for inode 0x%lx failed: %s\n"),* This should be parallelised - pass subdirs off to a work queue, have the * work queue processes subdirs, queueing more subdirs to work on. */ -static int -walk_mount ( +int +resolve_target_paths (const char *mntpt) {int ret;
@@ -361,9 +363,9 @@ list_inode_paths(void)/* * Any inodes remaining in the tree at this point indicate inodes whose -* paths were not found. This will be unlinked but still open inodes or -* lost inodes due to corruptions. Either way, a shrink will not succeed -* until these inodes are removed from the filesystem. +* paths were not found. This will be free inodes or unlinked but still +* open inodes. Either way, a shrink will not succeed until these inodes +* are removed from the filesystem. */ idx =0 ; do {
@@ -400,7 +402,7 @@ _("Inode list has not been populated. No inodes to resolve.\n"));return 0 ; } - ret =walk_mount (file->fs_path.fs_dir); + ret =resolve_target_paths (file->fs_path.fs_dir);if (ret) { fprintf(stderr, _("Failed to resolve all paths from mount point %s: %s\n"),
spaceman/init.c +1 -0
@@ -40,6 +40,7 @@ init_commands(void)move_inode_init ();find_owner_init ();resolve_owner_init (); +relocate_init (); } static int
spaceman/move_inode.c +4 -3
@@ -12,6 +12,7 @@#include "space.h" #include "input.h" #include "handle.h" +#include "relocation.h" #include <linux/fiemap.h> #include <linux/falloc.h>
@@ -404,8 +405,8 @@ exchange_inodes(return 0 ; } -static int -move_file_to_ag( +int +relocate_file_to_ag( const char *mnt, const char *path, struct xfs_fd *xfd,
@@ -511,7 +512,7 @@ _("Destination AG %d does not exist. Filesystem only has %d AGs\n"), }if (S_ISREG (st.st_mode)) { - ret =move_file_to_ag (file->fs_path.fs_dir, file->name, + ret =relocate_file_to_ag (file->fs_path.fs_dir, file->name, &file->xfd, agno); }else { fprintf(stderr, _("Unsupported: %s is not a regular file.\n"),
spaceman/relocation.c +234 -0
237 lines not on this page · raw message
spaceman/relocation.h +5 -0
@@ -43,4 +43,9 @@ struct inode_path {*/ #define UNLINKED_IPATH ((struct inode_path *)1) +int find_relocation_targets (xfs_agnumber_t agno); +int relocate_file_to_ag (const char *mnt,const char *path,struct xfs_fd *xfd, +xfs_agnumber_t agno); +int resolve_target_paths (const char *mntpt); +#endif /* XFS_SPACEMAN_RELOCATION_H_ */
spaceman/space.h +1 -0
@@ -41,5 +41,6 @@ extern void health_init(void);void move_inode_init (void );void find_owner_init (void );void resolve_owner_init (void ); +void relocate_init (void );#endif /* XFS_SPACEMAN_SPACE_H_ */
PATCH 11/11 spaceman: move inodes with hardlinks
In reply to Darrick J. WongFrom: Dave Chinner <[email protected]>
When a inode to be moved to a different AG has multiple hard links, we need to "move" all the hard links, too. To do this, we need to create temporary hardlinks to the new file, and then use rename exchange to swap all the hardlinks that point to the old inode with new hardlinks that point to the new inode.
We already know that an inode has hard links via the path discovery, and we can check it against the link count that is reported for the inode before we start building the link farm.
--- spaceman/find_owner.c | 13 +---- spaceman/move_inode.c | 119 +++++++++++++++++++++++++++++++++++++++++++++---- spaceman/relocation.c | 35 ++++++++++---- spaceman/relocation.h | 6 ++ 4 files changed, 140 insertions(+), 33 deletions(-)
spaceman/find_owner.c +2 -11
@@ -240,7 +240,6 @@ resolve_owner_cb( struct FTW *data) {struct inode_path *ipath, *slot_ipath; -int pathlen;struct inode_path **slot;/*
@@ -260,17 +259,9 @@ _("Failed to obtain stat(2) information from path %s. Aborting\n"), }/* Allocate a new inode path and record the path in it. */ - pathlen =strlen (path); - ipath =calloc (1 ,sizeof (*ipath) + pathlen +1 ); -if (!ipath) { -fprintf (stderr, -_ ("Aborting: Storing path %s for inode 0x%lx failed: %s\n" ), - path, stat->st_ino,strerror (ENOMEM )); + ipath =ipath_alloc (path, stat); +if (!ipath)return -ENOMEM ; - } -INIT_LIST_HEAD (&ipath->path_list); -memcpy (&ipath->path[0 ], path, pathlen); - ipath->ino = stat->st_ino;/* * If the slot contains the inode number we just looked up, then we
spaceman/move_inode.c +109 -10
203 lines not on this page · raw message
spaceman/relocation.c +25 -10
@@ -318,6 +318,30 @@ forget_reloc_ino(static struct cmdinfo relocate_cmd; +struct inode_path * +ipath_alloc ( +const char *path, +const struct stat *stat) +{ +struct inode_path *ipath; +int pathlen =strlen (path); + +/* Allocate a new inode path and record the path in it. */ + ipath =calloc (1 ,sizeof (*ipath) + pathlen +1 ); +if (!ipath) { +fprintf (stderr, +_ ("Failed to allocate ipath %s for inode 0x%llx failed: %s\n" ), + path, (unsigned long long )stat->st_ino, +strerror (-errno)); +return NULL ; + } +INIT_LIST_HEAD (&ipath->path_list); +memcpy (&ipath->path[0 ], path, pathlen); + ipath->ino = stat->st_ino; + +return ipath; +} + static int relocate_targets_to_ag( const char *mnt,
@@ -336,15 +360,6 @@ relocate_targets_to_ag(if (!ipath)break ; -/* XXX: don't handle hard link cases yet */ -if (ipath->link_count >1 ) { -fprintf (stderr, -"FIXME! Skipping hardlinked inode at path %s\n" , - ipath->path); - goto next; - } - - ret =stat (ipath->path, &st);if (ret) { fprintf(stderr, _("stat(%s) failed: %s\n"),
@@ -367,7 +382,7 @@ relocate_targets_to_ag( }/* move to destination AG */ - ret =relocate_file_to_ag (mnt, ipath->path, &xfd, dst_agno); + ret =relocate_file_to_ag (mnt, ipath, &xfd, dst_agno);xfd_close (&xfd);/*
spaceman/relocation.h +4 -2
@@ -43,9 +43,11 @@ struct inode_path {*/ #define UNLINKED_IPATH ((struct inode_path *)1) +struct inode_path *ipath_alloc (const char *path,const struct stat *st); +int find_relocation_targets (xfs_agnumber_t agno); -int relocate_file_to_ag (const char *mnt,const char *path,struct xfs_fd *xfd, -xfs_agnumber_t agno); +int relocate_file_to_ag (const char *mnt,struct inode_path *ipath, +struct xfs_fd *xfd,xfs_agnumber_t agno);int resolve_target_paths (const char *mntpt);#endif /* XFS_SPACEMAN_RELOCATION_H_ */
PATCH 3/5 fstests: capture logs from mount failures
In reply to Darrick J. WongHi all,
Whenever a mount fails, we should capture the kernel logs for the last few seconds before the failure. If the test fails, retain the log contents for further analysis.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=capture-mount-failures
---
Commits in this patchset:
* treewide: convert all $MOUNT_PROG to _mount
* check: capture dmesg of mount failures if test fails
--- check | 22 +++++++++++++++++++++- common/btrfs | 4 ++-- common/dmdelay | 2 +- common/dmerror | 2 +- common/dmlogwrites | 2 +- common/overlay | 6 +++--- common/rc | 26 +++++++++++++++++++++++++- common/report | 8 ++++++++ tests/btrfs/075 | 2 +- tests/btrfs/208 | 2 +- tests/ext4/032 | 2 +- tests/generic/067 | 6 +++--- tests/generic/085 | 2 +- tests/generic/361 | 2 +- tests/generic/373 | 2 +- tests/generic/374 | 2 +- tests/generic/409 | 6 +++--- tests/generic/410 | 8 ++++---- tests/generic/411 | 8 ++++---- tests/generic/589 | 8 ++++---- tests/overlay/005 | 4 ++-- tests/overlay/025 | 2 +- tests/overlay/035 | 2 +- tests/overlay/062 | 2 +- tests/overlay/083 | 6 +++--- tests/overlay/086 | 12 ++++++------ tests/selftest/008 | 20 ++++++++++++++++++++ tests/selftest/008.out | 1 + tests/xfs/078 | 2 +- tests/xfs/149 | 4 ++-- tests/xfs/289 | 4 ++-- tests/xfs/544 | 2 +- 32 files changed, 128 insertions(+), 55 deletions(-) create mode 100755 tests/selftest/008 create mode 100644 tests/selftest/008.out
PATCH 1/2 treewide: convert all $MOUNT_PROG to _mount
From: Darrick J. Wong <[email protected]>
Going to add some new log scraping functionality when mount failures occur, so we need everyone to use _mount instead of $MOUNT_PROG.
--- common/btrfs | 4 ++-- common/dmdelay | 2 +- common/dmerror | 2 +- common/dmlogwrites | 2 +- common/overlay | 6 +++--- tests/btrfs/075 | 2 +- tests/btrfs/208 | 2 +- tests/ext4/032 | 2 +- tests/generic/067 | 6 +++--- tests/generic/085 | 2 +- tests/generic/361 | 2 +- tests/generic/373 | 2 +- tests/generic/374 | 2 +- tests/generic/409 | 6 +++--- tests/generic/410 | 8 ++++---- tests/generic/411 | 8 ++++---- tests/generic/589 | 8 ++++---- tests/overlay/005 | 4 ++-- tests/overlay/025 | 2 +- tests/overlay/035 | 2 +- tests/overlay/062 | 2 +- tests/overlay/083 | 6 +++--- tests/overlay/086 | 12 ++++++------ tests/xfs/078 | 2 +- tests/xfs/149 | 4 ++-- tests/xfs/289 | 4 ++-- tests/xfs/544 | 2 +- 27 files changed, 53 insertions(+), 53 deletions(-)
common/btrfs +2 -2
@@ -351,7 +351,7 @@ _btrfs_stress_subvolume() mkdir -p $subvol_mnt while [ ! -e $stop_file ]; do $BTRFS_UTIL_PROG subvolume create $btrfs_mnt/$subvol_name - $MOUNT_PROG -o subvol=$subvol_name $btrfs_dev $subvol_mnt + _mount -o subvol=$subvol_name $btrfs_dev $subvol_mnt $UMOUNT_PROG $subvol_mnt $BTRFS_UTIL_PROG subvolume delete $btrfs_mnt/$subvol_name done
@@ -437,7 +437,7 @@ _btrfs_stress_remount_compress() local btrfs_mnt=$1 while true; do for algo in no zlib lzo; do - $MOUNT_PROG -o remount,compress=$algo $btrfs_mnt + _mount -o remount,compress=$algo $btrfs_mnt done done }
common/dmdelay +1 -1
@@ -20,7 +20,7 @@ _init_delay() _mount_delay() { _scratch_options mount - $MOUNT_PROG -t $FSTYP `_common_dev_mount_options` $SCRATCH_OPTIONS \ + _mount -t $FSTYP `_common_dev_mount_options` $SCRATCH_OPTIONS \ $DELAY_DEV $SCRATCH_MNT }
common/dmerror +1 -1
@@ -91,7 +91,7 @@ _dmerror_init() _dmerror_mount() { _scratch_options mount - $MOUNT_PROG -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \ + _mount -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \ $DMERROR_DEV $SCRATCH_MNT }
common/dmlogwrites +1 -1
@@ -139,7 +139,7 @@ _log_writes_mkfs() _log_writes_mount() { _scratch_options mount - $MOUNT_PROG -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \ + _mount -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \ $LOGWRITES_DMDEV $SCRATCH_MNT }
common/overlay +3 -3
@@ -29,13 +29,13 @@ _overlay_mount_dirs() [ -n "$upperdir" ] && [ "$upperdir" != "-" ] && \ diropts+=",upperdir=$upperdir,workdir=$workdir" - $MOUNT_PROG -t overlay $diropts `_common_dev_mount_options $*` + _mount -t overlay $diropts `_common_dev_mount_options $*` } # Mount with mnt/dev of scratch mount and custom mount options _overlay_scratch_mount_opts() { - $MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $* + _mount -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $* } # Mount with same options/mnt/dev of scratch mount, but optionally
@@ -127,7 +127,7 @@ _overlay_base_scratch_mount() _overlay_scratch_mount() { if echo "$*" | grep -q remount; then - $MOUNT_PROG $SCRATCH_MNT $* + _mount $SCRATCH_MNT $* return fi
tests/btrfs/075 +1 -1
@@ -37,7 +37,7 @@ _scratch_mount subvol_mnt=$TEST_DIR/$seq.mnt mkdir -p $subvol_mnt $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol >>$seqres.full 2>&1 -$MOUNT_PROG -o subvol=subvol $SELINUX_MOUNT_OPTIONS $SCRATCH_DEV $subvol_mnt +_mount -o subvol=subvol $SELINUX_MOUNT_OPTIONS $SCRATCH_DEV $subvol_mnt status=$? exit
tests/btrfs/208 +1 -1
@@ -45,7 +45,7 @@ _scratch_unmount # Now we mount the subvol2, which makes subvol3 not accessible for this mount # point, but we should be able to delete it using it's subvolume id -$MOUNT_PROG -o subvol=subvol2 $SCRATCH_DEV $SCRATCH_MNT +_mount -o subvol=subvol2 $SCRATCH_DEV $SCRATCH_MNT _delete_and_list subvol3 "Last remaining subvolume:" _scratch_unmount
tests/ext4/032 +1 -1
@@ -48,7 +48,7 @@ ext4_online_resize() $seqres.full 2>&1 || _fail "mkfs failed" echo "+++ mount image file" | tee -a $seqres.full - $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \ + _mount -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \ /dev/null 2>&1 || _fail "mount failed" echo "+++ resize fs to $final_size" | tee -a $seqres.full
tests/generic/067 +3 -3
@@ -34,7 +34,7 @@ mount_nonexistent_mnt() { echo "# mount to nonexistent mount point" >>$seqres.full rm -rf $TEST_DIR/nosuchdir - $MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1 + _mount $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1 } # fs driver should be able to handle mounting a free loop device gracefully
@@ -43,7 +43,7 @@ mount_free_loopdev() { echo "# mount a free loop device" >>$seqres.full loopdev=`losetup -f` - $MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1 + _mount -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1 } # mount with wrong fs type specified.
@@ -55,7 +55,7 @@ mount_wrong_fstype() fs=xfs fi echo "# mount with wrong fs type" >>$seqres.full - $MOUNT_PROG -t $fs $SCRATCH_DEV $SCRATCH_MNT >>$seqres.full 2>&1 + _mount -t $fs $SCRATCH_DEV $SCRATCH_MNT >>$seqres.full 2>&1 } # umount a symlink to device, which is not mounted.
tests/generic/085 +1 -1
@@ -69,7 +69,7 @@ for ((i=0; i<100; i++)); do done & pid=$! for ((i=0; i<100; i++)); do - $MOUNT_PROG $lvdev $SCRATCH_MNT >/dev/null 2>&1 + _mount $lvdev $SCRATCH_MNT >/dev/null 2>&1 $UMOUNT_PROG $lvdev >/dev/null 2>&1 done & pid="$pid $!"
tests/generic/361 +1 -1
@@ -52,7 +52,7 @@ fi $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1 # remount should not hang -$MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1 +_mount -o remount,ro $fs_mnt >>$seqres.full 2>&1 # success, all done echo "Silence is golden"
tests/generic/373 +1 -1
@@ -42,7 +42,7 @@ blksz=65536 sz=$((blksz * blocks)) echo "Mount otherdir" -$MOUNT_PROG --bind $SCRATCH_MNT $otherdir +_mount --bind $SCRATCH_MNT $otherdir echo "Create file" _pwrite_byte 0x61 0 $sz $testdir/file >> $seqres.full
tests/generic/374 +1 -1
@@ -41,7 +41,7 @@ blksz=65536 sz=$((blocks * blksz)) echo "Mount otherdir" -$MOUNT_PROG --bind $SCRATCH_MNT $otherdir +_mount --bind $SCRATCH_MNT $otherdir echo "Create file" _pwrite_byte 0x61 0 $sz $testdir/file >> $seqres.full
tests/generic/409 +3 -3
@@ -87,7 +87,7 @@ start_test() _scratch_mkfs >$seqres.full 2>&1 _get_mount -t $FSTYP $SCRATCH_DEV $MNTHEAD - $MOUNT_PROG --make-"${type}" $MNTHEAD + _mount --make-"${type}" $MNTHEAD mkdir $mpA $mpB $mpC $mpD }
@@ -107,9 +107,9 @@ bind_run() echo "bind $source on $dest" _get_mount -t $FSTYP $SCRATCH_DEV $mpA mkdir -p $mpA/dir 2>/dev/null - $MOUNT_PROG --make-shared $mpA + _mount --make-shared $mpA _get_mount --bind $mpA $mpB - $MOUNT_PROG --make-"$source" $mpB + _mount --make-"$source" $mpB # maybe unbindable at here _get_mount --bind $mpB $mpC 2>/dev/null if [ $? -ne 0 ]; then
tests/generic/410 +4 -4
@@ -93,7 +93,7 @@ start_test() _scratch_mkfs >>$seqres.full 2>&1 _get_mount -t $FSTYP $SCRATCH_DEV $MNTHEAD - $MOUNT_PROG --make-"${type}" $MNTHEAD + _mount --make-"${type}" $MNTHEAD mkdir $mpA $mpB $mpC }
@@ -117,14 +117,14 @@ run() echo "make-$cmd a $orgs mount" _get_mount -t $FSTYP $SCRATCH_DEV $mpA mkdir -p $mpA/dir 2>/dev/null - $MOUNT_PROG --make-shared $mpA + _mount --make-shared $mpA # prepare the original status on mpB _get_mount --bind $mpA $mpB # shared&slave status need to do make-slave then make-shared # two operations. for t in $orgs; do - $MOUNT_PROG --make-"$t" $mpB + _mount --make-"$t" $mpB done # "before" for prepare and check original status
@@ -145,7 +145,7 @@ run() _put_mount # umount C fi if [ "$i" = "before" ];then - $MOUNT_PROG --make-"${cmd}" $mpB + _mount --make-"${cmd}" $mpB fi done
tests/generic/411 +4 -4
@@ -76,7 +76,7 @@ start_test() _scratch_mkfs >$seqres.full 2>&1 _get_mount -t $FSTYP $SCRATCH_DEV $MNTHEAD - $MOUNT_PROG --make-"${type}" $MNTHEAD + _mount --make-"${type}" $MNTHEAD mkdir $mpA $mpB $mpC }
@@ -99,11 +99,11 @@ crash_test() _get_mount -t $FSTYP $SCRATCH_DEV $mpA mkdir $mpA/mnt1 - $MOUNT_PROG --make-shared $mpA + _mount --make-shared $mpA _get_mount --bind $mpA $mpB _get_mount --bind $mpA $mpC - $MOUNT_PROG --make-slave $mpB - $MOUNT_PROG --make-slave $mpC + _mount --make-slave $mpB + _mount --make-slave $mpC _get_mount -t $FSTYP $SCRATCH_DEV $mpA/mnt1 mkdir $mpA/mnt1/mnt2
tests/generic/589 +4 -4
@@ -80,12 +80,12 @@ start_test() _get_mount -t $FSTYP $SCRATCH_DEV $SRCHEAD # make sure $SRCHEAD is private - $MOUNT_PROG --make-private $SRCHEAD + _mount --make-private $SRCHEAD _get_mount -t $FSTYP $SCRATCH_DEV $DSTHEAD # test start with a bind, then make-shared $DSTHEAD _get_mount --bind $DSTHEAD $DSTHEAD - $MOUNT_PROG --make-"${type}" $DSTHEAD + _mount --make-"${type}" $DSTHEAD mkdir $mpA $mpB $mpC $mpD }
@@ -105,10 +105,10 @@ move_run() echo "move $source to $dest" _get_mount -t $FSTYP $SCRATCH_DEV $mpA mkdir -p $mpA/dir 2>/dev/null - $MOUNT_PROG --make-shared $mpA + _mount --make-shared $mpA # need a peer for slave later _get_mount --bind $mpA $mpB - $MOUNT_PROG --make-"$source" $mpB + _mount --make-"$source" $mpB # maybe unbindable at here _get_mount --move $mpB $mpC 2>/dev/null if [ $? -ne 0 ]; then
tests/overlay/005 +2 -2
@@ -50,8 +50,8 @@ $MKFS_XFS_PROG -f -n ftype=1 $upper_loop_dev >>$seqres.full 2>&1 # mount underlying xfs mkdir -p ${OVL_BASE_SCRATCH_MNT}/lowermnt mkdir -p ${OVL_BASE_SCRATCH_MNT}/uppermnt -$MOUNT_PROG $fs_loop_dev ${OVL_BASE_SCRATCH_MNT}/lowermnt -$MOUNT_PROG $upper_loop_dev ${OVL_BASE_SCRATCH_MNT}/uppermnt +_mount $fs_loop_dev ${OVL_BASE_SCRATCH_MNT}/lowermnt +_mount $upper_loop_dev ${OVL_BASE_SCRATCH_MNT}/uppermnt # prepare dirs mkdir -p ${OVL_BASE_SCRATCH_MNT}/lowermnt/lower
tests/overlay/025 +1 -1
@@ -36,7 +36,7 @@ _require_extra_fs tmpfs # create a tmpfs in $TEST_DIR tmpfsdir=$TEST_DIR/tmpfs mkdir -p $tmpfsdir -$MOUNT_PROG -t tmpfs tmpfs $tmpfsdir +_mount -t tmpfs tmpfs $tmpfsdir mkdir -p $tmpfsdir/{lower,upper,work,mnt} mkdir -p -m 0 $tmpfsdir/upper/testd
tests/overlay/035 +1 -1
@@ -42,7 +42,7 @@ mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir # Verify that overlay is mounted read-only and that it cannot be remounted rw. _overlay_scratch_mount_opts -o"lowerdir=$lowerdir2:$lowerdir1" touch $SCRATCH_MNT/foo 2>&1 | _filter_scratch -$MOUNT_PROG -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount +_mount -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount $UMOUNT_PROG $SCRATCH_MNT # Make workdir immutable to prevent workdir re-create on mount
tests/overlay/062 +1 -1
@@ -60,7 +60,7 @@ lowertestdir=$lower2/testdir create_test_files $lowertestdir # bind mount to pin lower test dir dentry to dcache -$MOUNT_PROG --bind $lowertestdir $lowertestdir +_mount --bind $lowertestdir $lowertestdir # For non-upper overlay mount, nfs_export requires disabling redirect_dir. _overlay_scratch_mount_opts \
tests/overlay/083 +3 -3
@@ -40,14 +40,14 @@ mkdir -p "$lowerdir_spaces" "$lowerdir_colons" "$lowerdir_commas" # _overlay_mount_* helpers do not handle special chars well, so execute mount directly. # if escaped colons are not parsed correctly, mount will fail. -$MOUNT_PROG -t overlay ovl_esc_test $SCRATCH_MNT \ +_mount -t overlay ovl_esc_test $SCRATCH_MNT \ -o"upperdir=$upperdir,workdir=$workdir" \ -o"lowerdir=$lowerdir_colons_esc:$lowerdir_spaces" \ 2>&1 | tee -a $seqres.full # if spaces are not escaped when showing mount options, # mount command will not show the word 'spaces' after the spaces -$MOUNT_PROG -t overlay | grep ovl_esc_test | tee -a $seqres.full | grep -v spaces && \ +_mount -t overlay | grep ovl_esc_test | tee -a $seqres.full | grep -v spaces && \ echo "ERROR: escaped spaces truncated from lowerdir mount option" # Re-create the upper/work dirs to mount them with a different lower
@@ -65,7 +65,7 @@ mkdir -p "$upperdir" "$workdir" # and this test will fail, but the failure would indicate a libmount issue, not # a kernel issue. Therefore, force libmount to use mount(2) syscall, so we only # test the kernel fix. -LIBMOUNT_FORCE_MOUNT2=always $MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_DEV $SCRATCH_MNT \ +LIBMOUNT_FORCE_MOUNT2=always _mount -t overlay $OVL_BASE_SCRATCH_DEV $SCRATCH_MNT \ -o"upperdir=$upperdir,workdir=$workdir,lowerdir=$lowerdir_commas_esc" 2>> $seqres.full || \ echo "ERROR: incorrect parsing of escaped comma in lowerdir mount option"
tests/overlay/086 +6 -6
@@ -33,21 +33,21 @@ mkdir -p "$lowerdir_spaces" "$lowerdir_colons" # _overlay_mount_* helpers do not handle lowerdir+,datadir+, so execute mount directly. # check illegal combinations and order of lowerdir,lowerdir+,datadir+ -$MOUNT_PROG -t overlay none $SCRATCH_MNT \ +_mount -t overlay none $SCRATCH_MNT \ -o"lowerdir=$lowerdir,lowerdir+=$lowerdir_colons" \ 2>> $seqres.full && \ echo "ERROR: invalid combination of lowerdir and lowerdir+ mount options" $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null -$MOUNT_PROG -t overlay none $SCRATCH_MNT \ +_mount -t overlay none $SCRATCH_MNT \ -o"lowerdir=$lowerdir,datadir+=$lowerdir_colons" \ -o redirect_dir=follow,metacopy=on 2>> $seqres.full && \ echo "ERROR: invalid combination of lowerdir and datadir+ mount options" $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null -$MOUNT_PROG -t overlay none $SCRATCH_MNT \ +_mount -t overlay none $SCRATCH_MNT \ -o"datadir+=$lowerdir,lowerdir+=$lowerdir_colons" \ -o redirect_dir=follow,metacopy=on 2>> $seqres.full && \ echo "ERROR: invalid order of lowerdir+ and datadir+ mount options"
@@ -55,7 +55,7 @@ $MOUNT_PROG -t overlay none $SCRATCH_MNT \ $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null # mount is expected to fail with escaped colons. -$MOUNT_PROG -t overlay none $SCRATCH_MNT \ +_mount -t overlay none $SCRATCH_MNT \ -o"lowerdir+=$lowerdir_colons_esc" \ 2>> $seqres.full && \ echo "ERROR: incorrect parsing of escaped colons in lowerdir+ mount option"
@@ -63,14 +63,14 @@ $MOUNT_PROG -t overlay none $SCRATCH_MNT \ $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null # mount is expected to succeed without escaped colons. -$MOUNT_PROG -t overlay ovl_esc_test $SCRATCH_MNT \ +_mount -t overlay ovl_esc_test $SCRATCH_MNT \ -o"lowerdir+=$lowerdir_colons,datadir+=$lowerdir_spaces" \ -o redirect_dir=follow,metacopy=on \ 2>&1 | tee -a $seqres.full # if spaces are not escaped when showing mount options, # mount command will not show the word 'spaces' after the spaces -$MOUNT_PROG -t overlay | grep ovl_esc_test | tee -a $seqres.full | \ +_mount -t overlay | grep ovl_esc_test | tee -a $seqres.full | \ grep -q 'datadir+'.*spaces || \ echo "ERROR: escaped spaces truncated from datadir+ mount option"
tests/xfs/078 +1 -1
@@ -75,7 +75,7 @@ _grow_loop() $XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io LOOP_DEV=`_create_loop_device $LOOP_IMG` echo "*** mount loop filesystem" - $MOUNT_PROG -t xfs $LOOP_DEV $LOOP_MNT + _mount -t xfs $LOOP_DEV $LOOP_MNT echo "*** grow loop filesystem" $XFS_GROWFS_PROG $LOOP_MNT 2>&1 | _filter_growfs 2>&1
tests/xfs/149 +2 -2
@@ -64,7 +64,7 @@ $XFS_GROWFS_PROG $loop_symlink 2>&1 | sed -e s:$loop_symlink:LOOPSYMLINK: # These mounted operations should pass echo "=== mount ===" -$MOUNT_PROG $loop_dev $mntdir || _fail "!!! failed to loopback mount" +_mount $loop_dev $mntdir || _fail "!!! failed to loopback mount" echo "=== xfs_growfs - check device node ===" $XFS_GROWFS_PROG -D 8192 $loop_dev > /dev/null
@@ -76,7 +76,7 @@ echo "=== unmount ===" $UMOUNT_PROG $mntdir || _fail "!!! failed to unmount" echo "=== mount device symlink ===" -$MOUNT_PROG $loop_symlink $mntdir || _fail "!!! failed to loopback mount" +_mount $loop_symlink $mntdir || _fail "!!! failed to loopback mount" echo "=== xfs_growfs - check device symlink ===" $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
tests/xfs/289 +2 -2
@@ -56,7 +56,7 @@ echo "=== xfs_growfs - plain file - should be rejected ===" $XFS_GROWFS_PROG $tmpfile 2>&1 | _filter_test_dir echo "=== mount ===" -$MOUNT_PROG -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount" +_mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount" echo "=== xfs_growfs - mounted - check absolute path ===" $XFS_GROWFS_PROG -D 8192 $tmpdir | _filter_test_dir > /dev/null
@@ -79,7 +79,7 @@ $XFS_GROWFS_PROG -D 28672 tmpsymlink.$$ > /dev/null echo "=== xfs_growfs - bind mount ===" mkdir $tmpbind -$MOUNT_PROG -o bind $tmpdir $tmpbind +_mount -o bind $tmpdir $tmpbind $XFS_GROWFS_PROG -D 32768 $tmpbind | _filter_test_dir > /dev/null echo "=== xfs_growfs - bind mount - relative path ==="
tests/xfs/544 +1 -1
@@ -35,7 +35,7 @@ mkdir $TEST_DIR/dest.$seq # Test echo "*** dump with bind-mounted test ***" >> $seqres.full -$MOUNT_PROG --bind $TEST_DIR/src.$seq $TEST_DIR/dest.$seq || _fail "Bind mount failed" +_mount --bind $TEST_DIR/src.$seq $TEST_DIR/dest.$seq || _fail "Bind mount failed" $XFSDUMP_PROG -L session -M test -f $tmp.dump $TEST_DIR/dest.$seq \ >> $seqres.full 2>&1 && echo "dump with bind-mounted should be failed, but passed."
PATCH 2/2 check: capture dmesg of mount failures if test fails
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Capture the kernel output after a mount failure occurs. If the test itself fails, then keep the logging output for further diagnosis.
--- check | 22 +++++++++++++++++++++- common/rc | 26 +++++++++++++++++++++++++- common/report | 8 ++++++++ tests/selftest/008 | 20 ++++++++++++++++++++ tests/selftest/008.out | 1 + 5 files changed, 75 insertions(+), 2 deletions(-) create mode 100755 tests/selftest/008 create mode 100644 tests/selftest/008.out
check +21 -1
@@ -614,7 +614,7 @@ _stash_fail_loop_files() { local seq_prefix="${REPORT_DIR}/${1}" local cp_suffix="$2" - for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"; do + for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints" ".mountfail"; do rm -f "${seq_prefix}${i}${cp_suffix}" if [ -f "${seq_prefix}${i}" ]; then cp "${seq_prefix}${i}" "${seq_prefix}${i}${cp_suffix}"
@@ -994,6 +994,7 @@ function run_section() echo -n " $seqnum -- " cat $seqres.notrun tc_status="notrun" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" # Unmount the scratch fs so that we can wipe the scratch
@@ -1053,6 +1054,7 @@ function run_section() if [ ! -f $seq.out ]; then _dump_err "no qualified output" tc_status="fail" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" continue; fi
@@ -1089,6 +1091,24 @@ function run_section() rm -f $seqres.hints fi fi + + if [ -f "$seqres.mountfail?" ]; then + if [ "$tc_status" = "fail" ]; then + # Let the user know if there were mount + # failures on a test that failed because that + # could be interesting. + mv "$seqres.mountfail?" "$seqres.mountfail" + _dump_err "check: possible mount failures (see $seqres.mountfail)" + test -f $seqres.mountfail && \ + maybe_compress_logfile $seqres.mountfail $MAX_MOUNTFAIL_SIZE + else + # Don't retain mount failure logs for tests + # that pass or were skipped because some tests + # intentionally drive mount failures. + rm -f "$seqres.mountfail?" + fi + fi + _stash_test_status "$seqnum" "$tc_status" done
common/rc +25 -1
@@ -204,9 +204,33 @@ _get_hugepagesize() awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo } +# Does dmesg have a --since flag? +_dmesg_detect_since() +{ + if [ -z "$DMESG_HAS_SINCE" ]; then + test "$DMESG_HAS_SINCE" = "yes" + return + elif dmesg --help | grep -q -- --since; then + DMESG_HAS_SINCE=yes + else + DMESG_HAS_SINCE=no + fi +} + _mount() { - $MOUNT_PROG $* + $MOUNT_PROG $* + ret=$? + if [ "$ret" -ne 0 ]; then + echo "\"$MOUNT_PROG $*\" failed at $(date)" >> "$seqres.mountfail?" + if _dmesg_detect_since; then + dmesg --since '30s ago' >> "$seqres.mountfail?" + else + dmesg | tail -n 100 >> "$seqres.mountfail?" + fi + fi + + return $ret } # Call _mount to do mount operation but also save mountpoint to
common/report +8 -0
@@ -199,6 +199,7 @@ _xunit_make_testcase_report() local out_src="${SRC_DIR}/${test_name}.out" local full_file="${REPORT_DIR}/${test_name}.full" local dmesg_file="${REPORT_DIR}/${test_name}.dmesg" + local mountfail_file="${REPORT_DIR}/${test_name}.mountfail" local outbad_file="${REPORT_DIR}/${test_name}.out.bad" if [ -z "$_err_msg" ]; then _err_msg="Test $test_name failed, reason unknown"
@@ -225,6 +226,13 @@ _xunit_make_testcase_report() printf ']]>\n' >>$report echo -e "\t\t</system-err>" >> $report fi + if [ -z "$quiet" -a -f "$mountfail_file" ]; then + echo -e "\t\t<mount-failure>" >> $report + printf '<![CDATA[\n' >>$report + cat "$mountfail_file" | tr -dc '[:print:][:space:]' | encode_cdata >>$report + printf ']]>\n' >>$report + echo -e "\t\t</mount-failure>" >> $report + fi ;; *) echo -e "\t\t<failure message=\"Unknown test_status=$test_status\" type=\"TestFail\"/>" >> $report
tests/selftest/008 new file +20 -0
@@ -0,0 +1,20 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# +# FS QA Test 008 +# +# Test mount failure capture. +# +. ./common/preamble +_begin_fstest selftest + +_require_command "$WIPEFS_PROG" wipefs +_require_scratch + +$WIPEFS_PROG -a $SCRATCH_DEV +_scratch_mount &>> $seqres.full + +# success, all done +status=0 +exit
tests/selftest/008.out new file +1 -0
@@ -0,0 +1 @@ +QA output created by 008
Re: [PATCH 2/2] check: capture dmesg of mount failures if test fails
Quoted 110 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Capture the kernel output after a mount failure occurs. If the test itself fails, then keep the logging output for further diagnosis.
Signed-off-by "Darrick J. Wong" <[email protected]>--- check | 22 +++++++++++++++++++++- common/rc | 26 +++++++++++++++++++++++++- common/report | 8 ++++++++ tests/selftest/008 | 20 ++++++++++++++++++++ tests/selftest/008.out | 1 + 5 files changed, 75 insertions(+), 2 deletions(-) create mode 100755 tests/selftest/008 create mode 100644 tests/selftest/008.outcheck +0
-1@@ -614,7 +614,7 @@ _stash_fail_loop_files() { local seq_prefix="${REPORT_DIR}/${1}" local cp_suffix="$2" - for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core"".hints"; do
check +1
-0+ for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"".mountfail"; do
check +0
-0rm -f "${seq_prefix}${i}${cp_suffix}" if [ -f "${seq_prefix}${i}" ]; then cp "${seq_prefix}${i}""${seq_prefix}${i}${cp_suffix}"
check +1
-0@@ -994,6 +994,7 @@ function run_section() echo -n " $seqnum -- " cat $seqres.notrun tc_status="notrun" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" # Unmount the scratch fs so that we can wipethe scratch
check +6
-0@@ -1053,6 +1054,7 @@ function run_section() if [ ! -f $seq.out ]; then _dump_err "no qualified output" tc_status="fail" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" continue; fi@@ -1089,6 +1091,24 @@ function run_section() rm -f $seqres.hints fi fi + + if [ -f "$seqres.mountfail?" ]; then + if [ "$tc_status" = "fail" ]; then + # Let the user know if there were mount + # failures on a test that failedbecause that
check +2
-0+ # could be interesting. + mv "$seqres.mountfail?""$seqres.mountfail"
check +1
-0+ _dump_err "check: possible mountfailures (see $seqres.mountfail)"
check +2
-0+ test -f $seqres.mountfail && \ + maybe_compress_logfile$seqres.mountfail $MAX_MOUNTFAIL_SIZE
check +2
-0+ else + # Don't retain mount failure logs fortests
check +1
-0+ # that pass or were skipped becausesome tests
check +5
-0+ # intentionally drive mount failures. + rm -f "$seqres.mountfail?" + fi + fi + _stash_test_status "$seqnum" "$tc_status" donecommon/rc +17
-1@@ -204,9 +204,33 @@ _get_hugepagesize() awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo } +# Does dmesg have a --since flag? +_dmesg_detect_since() +{ + if [ -z "$DMESG_HAS_SINCE" ]; then + test "$DMESG_HAS_SINCE" = "yes" + return + elif dmesg --help | grep -q -- --since; then + DMESG_HAS_SINCE=yes + else + DMESG_HAS_SINCE=no + fi +} + _mount() { - $MOUNT_PROG $* + $MOUNT_PROG $* + ret=$? + if [ "$ret" -ne 0 ]; then + echo "\"$MOUNT_PROG $*\" failed at $(date)" >>"$seqres.mountfail?"
common/rc _get_hugepagesize() · line 229
Quoted 4 lines
common/rc +4
-0+ if _dmesg_detect_since; then + dmesg --since '30s ago' >> "$seqres.mountfail?" + else + dmesg | tail -n 100 >> "$seqres.mountfail?"
Is it possible to grep for a mount failure message in dmesg and then capture the last n lines? Do you think that will be more accurate?
Also, do you think it is useful to make this 100 configurable instead of hardcoding?
Quoted 49 lines
common/rc +4
-0+ fi + fi + + return $ret } # Call _mount to do mount operation but also save mountpoint tocommon/report +1
-0@@ -199,6 +199,7 @@ _xunit_make_testcase_report() local out_src="${SRC_DIR}/${test_name}.out" local full_file="${REPORT_DIR}/${test_name}.full" local dmesg_file="${REPORT_DIR}/${test_name}.dmesg" + localmountfail_file="${REPORT_DIR}/${test_name}.mountfail"
common/report +0
-0local outbad_file="${REPORT_DIR}/${test_name}.out.bad" if [ -z "$_err_msg" ]; then _err_msg="Test $test_name failed, reasonunknown"
common/report +4
-0@@ -225,6 +226,13 @@ _xunit_make_testcase_report() printf ']]>\n' >>$report echo -e "\t\t</system-err>" >> $report fi + if [ -z "$quiet" -a -f "$mountfail_file" ]; then + echo -e "\t\t<mount-failure>" >> $report + printf '<![CDATA[\n' >>$report + cat "$mountfail_file" | tr -dc'[:print:][:space:]' |
encode_cdata>>$reportcommon/report +3
-0+ printf ']]>\n' >>$report + echo -e "\t\t</mount-failure>" >> $report + fi ;; *) echo -e "\t\t<failure message=\"Unknowntest_status=$test_status\" type=\"TestFail\"/>" >> $report
tests/selftest/008 new file +6
-0@@ -0,0 +1,20 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# +# FS QA Test 008 +#
tests/selftest/008 line 16
Quoted 10 lines
tests/selftest/008 new file +10
-0+# Test mount failure capture. +# +. ./common/preamble +_begin_fstest selftest + +_require_command "$WIPEFS_PROG" wipefs +_require_scratch + +$WIPEFS_PROG -a $SCRATCH_DEV +_scratch_mount &>> $seqres.full
Minor: Do you think adding some filtered messages from the captured dmesg logs in the output will be helpful?
Quoted 12 lines
tests/selftest/008 new file +4
-0+ +# success, all done +status=0 +exittests/selftest/008.out new file +1
-0@@ -0,0 +1 @@ +QA output created by 008
Re: [PATCH 2/2] check: capture dmesg of mount failures if test fails
Quoted 109 lines · Nirjhar Roy
Quoted 108 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Capture the kernel output after a mount failure occurs. If the test itself fails, then keep the logging output for further diagnosis.
Signed-off-by "Darrick J. Wong" <[email protected]>--- check | 22 +++++++++++++++++++++- common/rc | 26 +++++++++++++++++++++++++- common/report | 8 ++++++++ tests/selftest/008 | 20 ++++++++++++++++++++ tests/selftest/008.out | 1 + 5 files changed, 75 insertions(+), 2 deletions(-) create mode 100755 tests/selftest/008 create mode 100644 tests/selftest/008.outcheck +0
-1@@ -614,7 +614,7 @@ _stash_fail_loop_files() { local seq_prefix="${REPORT_DIR}/${1}" local cp_suffix="$2" - for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core"".hints"; do
check +1
-0+ for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"".mountfail"; do
check +0
-0rm -f "${seq_prefix}${i}${cp_suffix}" if [ -f "${seq_prefix}${i}" ]; then cp "${seq_prefix}${i}""${seq_prefix}${i}${cp_suffix}"
check +1
-0@@ -994,6 +994,7 @@ function run_section() echo -n " $seqnum -- " cat $seqres.notrun tc_status="notrun" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" # Unmount the scratch fs so that we can wipethe scratch
check +6
-0@@ -1053,6 +1054,7 @@ function run_section() if [ ! -f $seq.out ]; then _dump_err "no qualified output" tc_status="fail" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" continue; fi@@ -1089,6 +1091,24 @@ function run_section() rm -f $seqres.hints fi fi + + if [ -f "$seqres.mountfail?" ]; then + if [ "$tc_status" = "fail" ]; then + # Let the user know if there were mount + # failures on a test that failedbecause that
check +2
-0+ # could be interesting. + mv "$seqres.mountfail?""$seqres.mountfail"
check +1
-0+ _dump_err "check: possible mountfailures (see $seqres.mountfail)"
check +2
-0+ test -f $seqres.mountfail && \ + maybe_compress_logfile$seqres.mountfail $MAX_MOUNTFAIL_SIZE
check +2
-0+ else + # Don't retain mount failure logs fortests
check +1
-0+ # that pass or were skipped becausesome tests
check +5
-0+ # intentionally drive mount failures. + rm -f "$seqres.mountfail?" + fi + fi + _stash_test_status "$seqnum" "$tc_status" donecommon/rc +16
-1@@ -204,9 +204,33 @@ _get_hugepagesize() awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo } +# Does dmesg have a --since flag? +_dmesg_detect_since() +{ + if [ -z "$DMESG_HAS_SINCE" ]; then + test "$DMESG_HAS_SINCE" = "yes" + return + elif dmesg --help | grep -q -- --since; then + DMESG_HAS_SINCE=yes + else + DMESG_HAS_SINCE=no + fi +} + _mount() { - $MOUNT_PROG $* + $MOUNT_PROG $* + ret=$? + if [ "$ret" -ne 0 ]; then
Quoted 8 lines
common/rc _get_hugepagesize() · line 229
Quoted 6 lines
common/rc +1
-0+ echo "\"$MOUNT_PROG $*\" failed at $(date)" >>"$seqres.mountfail?"
common/rc +4
-0+ if _dmesg_detect_since; then + dmesg --since '30s ago' >> "$seqres.mountfail?" + else + dmesg | tail -n 100 >> "$seqres.mountfail?"Is it possible to grep for a mount failure message in dmesg and then capture the last n lines? Do you think that will be more accurate?
Alas no, because there's no standard mount failure log message for us to latch onto.
Quoted 2 lines
Also, do you think it is useful to make this 100 configurable instead of hardcoding?
I suppose, but why do you need more than 100?
Quoted 53 lines
Quoted 53 lines
common/rc +4
-0+ fi + fi + + return $ret } # Call _mount to do mount operation but also save mountpoint tocommon/report +1
-0@@ -199,6 +199,7 @@ _xunit_make_testcase_report() local out_src="${SRC_DIR}/${test_name}.out" local full_file="${REPORT_DIR}/${test_name}.full" local dmesg_file="${REPORT_DIR}/${test_name}.dmesg" + localmountfail_file="${REPORT_DIR}/${test_name}.mountfail"
common/report +0
-0local outbad_file="${REPORT_DIR}/${test_name}.out.bad" if [ -z "$_err_msg" ]; then _err_msg="Test $test_name failed, reasonunknown"
common/report +4
-0@@ -225,6 +226,13 @@ _xunit_make_testcase_report() printf ']]>\n' >>$report echo -e "\t\t</system-err>" >> $report fi + if [ -z "$quiet" -a -f "$mountfail_file" ]; then + echo -e "\t\t<mount-failure>" >> $report + printf '<![CDATA[\n' >>$report + cat "$mountfail_file" | tr -dc'[:print:][:space:]' |
encode_cdata>>$reportcommon/report +3
-0+ printf ']]>\n' >>$report + echo -e "\t\t</mount-failure>" >> $report + fi ;; *) echo -e "\t\t<failure message=\"Unknowntest_status=$test_status\" type=\"TestFail\"/>" >> $report
tests/selftest/008 new file +10
-0@@ -0,0 +1,20 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# +# FS QA Test 008 +# +# Test mount failure capture. +# +. ./common/preamble +_begin_fstest selftest
Quoted 8 lines
tests/selftest/008 line 16
Quoted 6 lines
tests/selftest/008 new file +6
-0+ +_require_command "$WIPEFS_PROG" wipefs +_require_scratch + +$WIPEFS_PROG -a $SCRATCH_DEV +_scratch_mount &>> $seqres.fullMinor: Do you think adding some filtered messages from the captured dmesg logs in the output will be helpful?
No, this test exists to make sure that the dmesg log is captured in $RESULT_DIR. We don't care about the mount(8) output.
--D
Quoted 14 lines
Quoted 12 lines
tests/selftest/008 new file +4
-0+ +# success, all done +status=0 +exittests/selftest/008.out new file +1
-0@@ -0,0 +1 @@ +QA output created by 008
Re: [PATCH 2/2] check: capture dmesg of mount failures if test fails
Quoted 121 lines · Darrick J. Wong
Quoted 111 lines · Nirjhar Roy
Quoted 110 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Capture the kernel output after a mount failure occurs. If the test
itself fails, then keep the logging output for further diagnosis.Signed-off-by "Darrick J. Wong" <[email protected]>--- check | 22 +++++++++++++++++++++- common/rc | 26 +++++++++++++++++++++++++- common/report | 8 ++++++++ tests/selftest/008 | 20 ++++++++++++++++++++ tests/selftest/008.out | 1 + 5 files changed, 75 insertions(+), 2 deletions(-) create mode 100755 tests/selftest/008 create mode 100644 tests/selftest/008.outcheck +0
-1@@ -614,7 +614,7 @@ _stash_fail_loop_files() { local seq_prefix="${REPORT_DIR}/${1}" local cp_suffix="$2" - for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core"".hints"; do
check +1
-0+ for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"".mountfail"; do
check +0
-0rm -f "${seq_prefix}${i}${cp_suffix}" if [ -f "${seq_prefix}${i}" ]; then cp "${seq_prefix}${i}""${seq_prefix}${i}${cp_suffix}"
check +1
-0@@ -994,6 +994,7 @@ function run_section() echo -n " $seqnum -- " cat $seqres.notrun tc_status="notrun" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" # Unmount the scratch fs so that we can wipethe scratch
check +6
-0@@ -1053,6 +1054,7 @@ function run_section() if [ ! -f $seq.out ]; then _dump_err "no qualified output" tc_status="fail" + rm -f "$seqres.mountfail?" _stash_test_status "$seqnum" "$tc_status" continue; fi@@ -1089,6 +1091,24 @@ function run_section() rm -f $seqres.hints fi fi + + if [ -f "$seqres.mountfail?" ]; then + if [ "$tc_status" = "fail" ]; then + # Let the user know if there were mount + # failures on a test that failedbecause that
check +2
-0+ # could be interesting. + mv "$seqres.mountfail?""$seqres.mountfail"
check +1
-0+ _dump_err "check: possible mountfailures (see $seqres.mountfail)"
check +2
-0+ test -f $seqres.mountfail && \ + maybe_compress_logfile$seqres.mountfail $MAX_MOUNTFAIL_SIZE
check +2
-0+ else + # Don't retain mount failure logs fortests
check +1
-0+ # that pass or were skipped becausesome tests
check +5
-0+ # intentionally drive mount failures. + rm -f "$seqres.mountfail?" + fi + fi + _stash_test_status "$seqnum" "$tc_status" donecommon/rc +17
-1@@ -204,9 +204,33 @@ _get_hugepagesize() awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo } +# Does dmesg have a --since flag? +_dmesg_detect_since() +{ + if [ -z "$DMESG_HAS_SINCE" ]; then + test "$DMESG_HAS_SINCE" = "yes" + return + elif dmesg --help | grep -q -- --since; then + DMESG_HAS_SINCE=yes + else + DMESG_HAS_SINCE=no + fi +} + _mount() { - $MOUNT_PROG $* + $MOUNT_PROG $* + ret=$? + if [ "$ret" -ne 0 ]; then + echo "\"$MOUNT_PROG $*\" failed at $(date)" >>Quoted 8 lines
common/rc _get_hugepagesize() · line 229
Quoted 5 lines
"$seqres.mountfail?"
common/rc +4
-0+ if _dmesg_detect_since; then + dmesg --since '30s ago' >> "$seqres.mountfail?" + else + dmesg | tail -n 100 >> "$seqres.mountfail?"Is it possible to grep for a mount failure message in dmesg and then
capture the last n lines? Do you think that will be more accurate?
Quoted 3 lines
Alas no, because there's no standard mount failure log message for us to
latch onto.
Okay makes sense.
Quoted 6 lines
Quoted 4 lines
Also, do you think it is useful to make this 100 configurable instead
of hardcoding?I suppose, but why do you need more than 100?
So my thought behind this is that in case, the dmesg gets cluttered with noisy logs from other processes. No hard preferences though.
Quoted 63 lines
Quoted 54 lines
Quoted 54 lines
common/rc +4
-0+ fi + fi + + return $ret } # Call _mount to do mount operation but also save mountpoint tocommon/report +1
-0@@ -199,6 +199,7 @@ _xunit_make_testcase_report() local out_src="${SRC_DIR}/${test_name}.out" local full_file="${REPORT_DIR}/${test_name}.full" local dmesg_file="${REPORT_DIR}/${test_name}.dmesg" + localmountfail_file="${REPORT_DIR}/${test_name}.mountfail"
common/report +0
-0local outbad_file="${REPORT_DIR}/${test_name}.out.bad" if [ -z "$_err_msg" ]; then _err_msg="Test $test_name failed, reasonunknown"
common/report +4
-0@@ -225,6 +226,13 @@ _xunit_make_testcase_report() printf ']]>\n' >>$report echo -e "\t\t</system-err>" >> $report fi + if [ -z "$quiet" -a -f "$mountfail_file" ]; then + echo -e "\t\t<mount-failure>" >> $report + printf '<![CDATA[\n' >>$report + cat "$mountfail_file" | tr -dc'[:print:][:space:]' |
encode_cdata>>$reportcommon/report +3
-0+ printf ']]>\n' >>$report + echo -e "\t\t</mount-failure>" >> $report + fi ;; *) echo -e "\t\t<failure message=\"Unknowntest_status=$test_status\" type=\"TestFail\"/>" >> $report
tests/selftest/008 new file +10
-0@@ -0,0 +1,20 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# +# FS QA Test 008 +# +# Test mount failure capture. +# +. ./common/preamble +_begin_fstest selftestQuoted 8 lines
tests/selftest/008 line 16
Quoted 6 lines
tests/selftest/008 new file +6
-0+ +_require_command "$WIPEFS_PROG" wipefs +_require_scratch + +$WIPEFS_PROG -a $SCRATCH_DEV +_scratch_mount &>> $seqres.fullMinor: Do you think adding some filtered messages from the captured dmesg logs in the output will be helpful?
Quoted 4 lines
No, this test exists to make sure that the dmesg log is captured in $RESULT_DIR. We don't care about the mount(8) output.
--D
Okay, got it.
--NR
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
tests/selftest/008 new file +4
-0+ +# success, all done +status=0 +exittests/selftest/008.out new file +1
-0@@ -0,0 +1 @@ +QA output created by 008
PATCH 4/5 xfs: defragment free space
In reply to Darrick J. WongHi all,
These patches contain experimental code to enable userspace to defragment the free space in a filesystem. Two purposes are imagined for this
functionality: clearing space at the end of a filesystem before shrinking it, and clearing free space in anticipation of making a large allocation.
The first patch adds a new fallocate mode that allows userspace to allocate free space from the filesystem into a file. The goal here is to allow the filesystem shrink process to prevent allocation from a certain part of the filesystem while a free space defragmenter evacuates all the files from the doomed part of the filesystem.
The second patch amends the online repair system to allow the sysadmin to forcibly rebuild metadata structures, even if they're not corrupt. Without adding an ioctl to move metadata btree blocks, this is the only way to dislodge metadata.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=defrag-freespace
---
Commits in this patchset:
* xfs: export realtime refcount information
* xfs: capture the offset and length in fallocate tracepoints
* xfs: add an ioctl to map free space into a file
* xfs: implement FALLOC_FL_MAP_FREE for realtime files
--- fs/xfs/libxfs/xfs_alloc.c | 88 ++++++++ fs/xfs/libxfs/xfs_alloc.h | 3 fs/xfs/libxfs/xfs_bmap.c | 1 fs/xfs/libxfs/xfs_fs.h | 14 + fs/xfs/xfs_bmap_util.c | 513 +++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_bmap_util.h | 3 fs/xfs/xfs_file.c | 143 ++++++++++++- fs/xfs/xfs_file.h | 2 fs/xfs/xfs_fsrefs.c | 405 ++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 5 fs/xfs/xfs_rtalloc.c | 108 +++++++++ fs/xfs/xfs_rtalloc.h | 7 + fs/xfs/xfs_trace.h | 86 +++++++- 13 files changed, 1368 insertions(+), 10 deletions(-)
PATCH 1/4 xfs: export realtime refcount information
From: Darrick J. Wong <[email protected]>
Add support for reporting space refcount information from the realtime volume.
--- fs/xfs/xfs_fsrefs.c | 405 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 405 insertions(+)
fs/xfs/xfs_fsrefs.c +405 -0
@@ -478,6 +478,395 @@ xfs_fsrefs_logdev(return xfs_fsrefs_helper (tp, info, &frec); } +#ifdef CONFIG_XFS_RT +/* Synthesize fsrefs records from rtbitmap records. */ +STATIC int +xfs_fsrefs_rtdev_bitmap_helper ( +struct xfs_rtgroup *rtg, +struct xfs_trans *tp, +const struct xfs_rtalloc_rec *rec, +void *priv) +{ +struct xfs_fsrefs_irec frec = { + .refcount =1 , + }; +struct xfs_mount *mp =rtg_mount (rtg); +struct xfs_fsrefs_info *info = priv; +xfs_rtblock_t next_rtb, rec_rtb, rtb; +xfs_rgnumber_t next_rgno; +xfs_rgblock_t next_rgbno; +xfs_rgblock_t rec_rgbno; + +/* Translate the free space record to group and block number. */ + rec_rtb =xfs_rtx_to_rtb (rtg, rec->ar_startext); + rec_rgbno =xfs_rtb_to_rgbno (mp, rec_rtb); + +/* +* Figure out if there's a gap between the last fsrefs record we +* emitted and this free extent. If there is, report the gap as a +* refcount==1 record. +*/ + next_rtb =xfs_daddr_to_rtb (mp, info->next_daddr); + next_rgno =xfs_rtb_to_rgno (mp, next_rtb); + next_rgbno =xfs_rtb_to_rgbno (mp, next_rtb); + +ASSERT (next_rgno >= info->group->xg_gno); +ASSERT (rec_rgbno >= next_rgbno); + +/* +* If we've already moved on to the next rtgroup, we don't have any +* fsrefs records to synthesize. +*/ +if (next_rgno > info->group->xg_gno) +return 0 ; + + rtb =xfs_rtx_to_rtb (rtg, rec->ar_startext + rec->ar_extcount); + info->next_daddr =xfs_rtb_to_daddr (mp, rtb); + +if (rec_rtb == next_rtb) +return 0 ; + +/* Emit a record for the in-use space. */ + frec.start_daddr =xfs_rtb_to_daddr (mp, next_rtb); + frec.len_daddr =XFS_FSB_TO_BB (mp, rec_rgbno - next_rgbno); + frec.rec_key = next_rgbno; +return xfs_fsrefs_helper (tp, info, &frec); +} + +/* Emit records to fill a gap in the refcount btree with singly-owned blocks. */ +STATIC int +xfs_fsrefs_rtdev_fill_refcount_gap ( +struct xfs_trans *tp, +struct xfs_fsrefs_info *info, +xfs_rgblock_t rgbno) +{ +struct xfs_rtalloc_rec high = {0 }; +struct xfs_mount *mp = tp->t_mountp; +struct xfs_rtgroup *rtg =to_rtg (info->group); +xfs_rtblock_t start_rtbno = +xfs_daddr_to_rtb (mp, info->next_daddr); +xfs_rtblock_t end_rtbno = +xfs_rgbno_to_rtb (rtg, rgbno); +xfs_rtxnum_t low_rtx; +xfs_daddr_t rec_daddr; +int error; + +ASSERT (xfs_rtb_to_rgno (mp, start_rtbno) == info->group->xg_gno); + + low_rtx =xfs_rtb_to_rtx (mp, start_rtbno); +if (rgbno ==-1U ) { +/* +* If the caller passes in an all 1s high key to signify the +* end of the group, set the extent to all 1s as well. +*/ + high.ar_startext =-1ULL ; + }else { + high.ar_startext =xfs_rtb_to_rtx (mp, + end_rtbno + mp->m_sb.sb_rextsize -1 ); + } +if (low_rtx >= high.ar_startext) +return 0 ; + + error =xfs_rtalloc_query_range (rtg, tp, low_rtx, high.ar_startext, + xfs_fsrefs_rtdev_bitmap_helper, info); +if (error) +return error; + +/* +* Synthesize records for single-owner extents between the last +* fsrefcount record emitted and the end of the query range. +*/ + high.ar_startext =min (high.ar_startext, rtg->rtg_extents); + rec_daddr =xfs_rtb_to_daddr (mp,xfs_rtx_to_rtb (rtg, high.ar_startext)); +if (info->next_daddr > rec_daddr) +return 0 ; + + info->last = true; +return xfs_fsrefs_rtdev_bitmap_helper (rtg, tp, &high, info); +} + +/* Transform a absolute-startblock refcount (rtdev, logdev) into a fsrefs */ +STATIC int +xfs_fsrefs_rtdev_refcountbt_helper ( +struct xfs_btree_cur *cur, +const struct xfs_refcount_irec *rec, +void *priv) +{ +struct xfs_fsrefs_irec frec = { + .refcount = rec->rc_refcount, + .rec_key = rec->rc_startblock, + }; +struct xfs_mount *mp = cur->bc_mp; +struct xfs_fsrefs_info *info = priv; +struct xfs_rtgroup *rtg =to_rtg (info->group); +xfs_rtblock_t rec_rtbno; +int error; + +/* +* Stop once we get to the CoW staging extents; they're all shoved to +* the right side of the btree and were already covered by the rtbitmap +* scan. +*/ +if (rec->rc_domain !=XFS_REFC_DOMAIN_SHARED ) +return -ECANCELED ; + +/* Report on any gaps first */ + error =xfs_fsrefs_rtdev_fill_refcount_gap (cur->bc_tp, info, + rec->rc_startblock); +if (error) +return error; + +/* Report the refcount record from the refcount btree. */ + rec_rtbno =xfs_rgbno_to_rtb (rtg, rec->rc_startblock); + frec.start_daddr =xfs_rtb_to_daddr (mp, rec_rtbno); + frec.len_daddr =XFS_FSB_TO_BB (mp, rec->rc_blockcount); + info->next_daddr =xfs_rtb_to_daddr (mp, rec_rtbno + rec->rc_blockcount); +return xfs_fsrefs_helper (cur->bc_tp, info, &frec); +} + +#define XFS_RTGLOCK_FSREFS (XFS_RTGLOCK_BITMAP | XFS_RTGLOCK_REFCOUNT) + +/* Execute a getfsrefs query against the realtime device. */ +STATIC int +xfs_fsrefs_rtdev ( +struct xfs_trans *tp, +const struct xfs_fsrefs *keys, +struct xfs_fsrefs_info *info) +{ +struct xfs_mount *mp = tp->t_mountp; +struct xfs_rtgroup *rtg =NULL , *locked_rtg =NULL ; +xfs_rtblock_t start_rtbno; +xfs_rtblock_t end_rtbno; +xfs_rgnumber_t start_rg; +xfs_rgnumber_t end_rg; +uint64_t eofs; +int error =0 ; + + eofs =XFS_FSB_TO_BB (mp, mp->m_sb.sb_rblocks); +if (keys[0 ].fcr_physical >= eofs) +return 0 ; + start_rtbno =xfs_daddr_to_rtb (mp, keys[0 ].fcr_physical); + end_rtbno =xfs_daddr_to_rtb (mp,min (eofs -1 , keys[1 ].fcr_physical)); + + info->refc_cur = info->bno_cur =NULL ; + +/* +* Convert the fsrefs low/high keys to rtgroup based keys. Initialize +* low to the fsrefs low key and max out the high key to the end of the +* rtgroup. +*/ + info->low.rc_startblock =xfs_rtb_to_rgbno (mp, start_rtbno); + info->low.rc_blockcount =XFS_BB_TO_FSBT (mp, keys[0 ].fcr_length); + info->low.rc_refcount =0 ; + info->low.rc_domain =XFS_REFC_DOMAIN_SHARED ; + +/* Adjust the low key if we are continuing from where we left off. */ +if (info->low.rc_blockcount >0 ) { + info->low.rc_startblock += info->low.rc_blockcount; + + start_rtbno += info->low.rc_blockcount; +if (xfs_rtb_to_daddr (mp, start_rtbno) >= eofs) +return 0 ; + } + + info->high.rc_startblock =-1U ; + info->high.rc_blockcount =0 ; + info->high.rc_refcount =0 ; + info->high.rc_domain =XFS_REFC_DOMAIN_SHARED ; + + start_rg =xfs_rtb_to_rgno (mp, start_rtbno); + end_rg =xfs_rtb_to_rgno (mp, end_rtbno); + +/* Query each rtgroup */ +while ((rtg =xfs_rtgroup_next_range (mp, rtg, start_rg, end_rg))) { + info->group =rtg_group (rtg); + +/* +* Set the rtgroup high key from the fsrefs high key if this +* is the last rtgroup that we're querying. +*/ +if (rtg_rgno (rtg) == end_rg) + info->high.rc_startblock =xfs_rtb_to_rgbno (mp, + end_rtbno); + +if (info->refc_cur) { +xfs_btree_del_cursor (info->refc_cur,XFS_BTREE_NOERROR ); + info->refc_cur =NULL ; + } +if (locked_rtg) +xfs_rtgroup_unlock (locked_rtg,XFS_RTGLOCK_FSREFS ); + +trace_xfs_fsrefs_low_group_key (mp, info->dev, info->group, + &info->low); +trace_xfs_fsrefs_high_group_key (mp, info->dev, info->group, + &info->high); + +xfs_rtgroup_lock (rtg,XFS_RTGLOCK_FSREFS ); + locked_rtg = rtg; + +/* +* Fill the query with refcount records and synthesize +* singly-owned block records from free space data. +*/ +if (xfs_has_rtreflink (mp)) { + info->refc_cur =xfs_rtrefcountbt_init_cursor (tp, rtg); + + error =xfs_refcount_query_range (info->refc_cur, + &info->low, &info->high, + xfs_fsrefs_rtdev_refcountbt_helper, + info); +if (error && error != -ECANCELED ) +break ; + } + +/* +* Synthesize refcount==1 records from the free space data +* between the end of the last fsrefs record reported and the +* end of the range. If we don't have refcount support, the +* starting point will be the start of the query range. +*/ + error =xfs_fsrefs_rtdev_fill_refcount_gap (tp, info, + info->high.rc_startblock); +if (error) +break ; + +/* +* Set the rtgroup low key to the start of the rtgroup prior to +* moving on to the next rtgroup. +*/ +if (rtg_rgno (rtg) == start_rg) +memset (&info->low,0 ,sizeof (info->low)); + info->group =NULL ; + } + +if (info->refc_cur) { +xfs_btree_del_cursor (info->refc_cur, error); + info->refc_cur =NULL ; + } +if (locked_rtg) +xfs_rtgroup_unlock (locked_rtg,XFS_RTGLOCK_FSREFS ); +if (info->group) { +xfs_rtgroup_rele (rtg); + info->group =NULL ; + }else if (rtg) { +/* loop termination case */ +xfs_rtgroup_rele (rtg); + } + +return error; +} + +/* Synthesize fsrefs records from 64-bit rtbitmap records. */ +STATIC int +xfs_fsrefs_rtdev_nogroups_helper ( +struct xfs_rtgroup *rtg, +struct xfs_trans *tp, +const struct xfs_rtalloc_rec *rec, +void *priv) +{ +struct xfs_fsrefs_irec frec = { + .refcount =1 , + }; +struct xfs_mount *mp =rtg_mount (rtg); +struct xfs_fsrefs_info *info = priv; +xfs_rtblock_t next_rtb, rec_rtb, rtb; + +/* Translate the free space record to group and block number. */ + rec_rtb =xfs_rtx_to_rtb (rtg, rec->ar_startext); + +/* +* Figure out if there's a gap between the last fsrefs record we +* emitted and this free extent. If there is, report the gap as a +* refcount==1 record. +*/ + next_rtb =xfs_daddr_to_rtb (mp, info->next_daddr); + +ASSERT (rec_rtb >= next_rtb); + + rtb =xfs_rtx_to_rtb (rtg, rec->ar_startext + rec->ar_extcount); + info->next_daddr =xfs_rtb_to_daddr (mp, rtb); + +if (rec_rtb == next_rtb) +return 0 ; + +/* Emit records for the in-use space. */ + frec.start_daddr =xfs_rtb_to_daddr (mp, next_rtb); + frec.len_daddr =xfs_rtb_to_daddr (mp, rec_rtb - next_rtb); +return xfs_fsrefs_helper (tp, info, &frec); +} + +/* +* Synthesize refcount information from the rtbitmap for a pre-rtgroups +* filesystem. +*/ +STATIC int +xfs_fsrefs_rtdev_nogroups ( +struct xfs_trans *tp, +const struct xfs_fsrefs *keys, +struct xfs_fsrefs_info *info) +{ +struct xfs_mount *mp = tp->t_mountp; +struct xfs_rtgroup *rtg =NULL ; +xfs_rtblock_t start_rtbno; +xfs_rtblock_t end_rtbno; +xfs_rtxnum_t low_rtx; +xfs_rtxnum_t high_rtx; +uint64_t eofs; +int error =0 ; + + eofs =XFS_FSB_TO_BB (mp, mp->m_sb.sb_rblocks); +if (keys[0 ].fcr_physical >= eofs) +return 0 ; + start_rtbno =xfs_daddr_to_rtb (mp, keys[0 ].fcr_physical); + end_rtbno =xfs_daddr_to_rtb (mp,min (eofs -1 , keys[1 ].fcr_physical)); + + info->refc_cur = info->bno_cur =NULL ; + +/* +* Convert the fsrefs low/high keys to rtgroup based keys. Initialize +* low to the fsrefs low key and max out the high key to the end of the +* rtgroup. +*/ + info->low_daddr = keys[0 ].fcr_physical; + +/* Adjust the low key if we are continuing from where we left off. */ +if (keys[0 ].fcr_length >0 ) { + info->low_daddr += keys[0 ].fcr_length; +if (info->low_daddr >= eofs) +return 0 ; + } + + rtg =xfs_rtgroup_grab (mp,0 ); +if (!rtg) +return -EFSCORRUPTED ; + + info->group =rtg_group (rtg); + +trace_xfs_fsrefs_low_linear_key (mp, info->dev, start_rtbno); +trace_xfs_fsrefs_high_linear_key (mp, info->dev, end_rtbno); + +xfs_rtgroup_lock (rtg,XFS_RTGLOCK_BITMAP ); + +/* +* Walk the whole rtbitmap. Without rtgroups, the startext values can +* be more than 32-bits wide, which is why we need this separate +* implementation. +*/ + low_rtx =xfs_rtb_to_rtx (mp, start_rtbno); + high_rtx =xfs_rtb_to_rtx (mp, end_rtbno + mp->m_sb.sb_rextsize -1 ); +if (low_rtx < high_rtx) + error =xfs_rtalloc_query_range (rtg, tp, low_rtx, high_rtx, + xfs_fsrefs_rtdev_nogroups_helper, info); + + info->group =NULL ; + +xfs_rtgroup_unlock (rtg,XFS_RTGLOCK_BITMAP ); +xfs_rtgroup_rele (rtg); + +return error; +} +#endif +/* Do we recognize the device? */ STATIC bool xfs_fsrefs_is_valid_device(
@@ -515,7 +904,14 @@ xfs_fsrefs_check_keys(return false; } +/* +* There are only two devices if we didn't configure RT devices at build time. +*/ +#ifdef CONFIG_XFS_RT +#define XFS_GETFSREFS_DEVS 3 +#else #define XFS_GETFSREFS_DEVS 2 +#endif /* CONFIG_XFS_RT */ /* * Get filesystem's extent refcounts as described in head, and format for
@@ -569,6 +965,15 @@ xfs_getfsrefs( handlers[1 ].dev =new_encode_dev (mp->m_logdev_targp->bt_dev); handlers[1 ].fn = xfs_fsrefs_logdev; } +#ifdef CONFIG_XFS_RT +if (mp->m_rtdev_targp) { + handlers[2 ].dev =new_encode_dev (mp->m_rtdev_targp->bt_dev); +if (xfs_has_rtgroups (mp)) + handlers[2 ].fn = xfs_fsrefs_rtdev; +else + handlers[2 ].fn = xfs_fsrefs_rtdev_nogroups; + } +#endif /* CONFIG_XFS_RT */ xfs_sort (handlers,XFS_GETFSREFS_DEVS ,sizeof (struct xfs_fsrefs_dev ), xfs_fsrefs_dev_compare);
PATCH 2/4 xfs: capture the offset and length in fallocate tracepoints
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Change the class of the fallocate tracepoints to capture the offset and length of the requested operation.
--- fs/xfs/xfs_bmap_util.c | 8 ++++---- fs/xfs/xfs_file.c | 2 +- fs/xfs/xfs_trace.h | 10 +++++----- 3 files changed, 10 insertions(+), 10 deletions(-)
fs/xfs/xfs_bmap_util.c +4 -4
@@ -652,7 +652,7 @@ xfs_alloc_file_space(if (xfs_is_always_cow_inode (ip))return 0 ; -trace_xfs_alloc_file_space (ip); +trace_xfs_alloc_file_space (ip, offset, len);if (xfs_is_shutdown (mp))return -EIO ;
@@ -839,7 +839,7 @@ xfs_free_file_space(xfs_fileoff_t endoffset_fsb;int done =0 , error; -trace_xfs_free_file_space (ip); +trace_xfs_free_file_space (ip, offset, len); error =xfs_qm_dqattach (ip);if (error)
@@ -987,7 +987,7 @@ xfs_collapse_file_space(xfs_assert_ilocked (ip,XFS_IOLOCK_EXCL |XFS_MMAPLOCK_EXCL ); -trace_xfs_collapse_file_space (ip); +trace_xfs_collapse_file_space (ip, offset, len); error =xfs_free_file_space (ip, offset, len, ac);if (error)
@@ -1056,7 +1056,7 @@ xfs_insert_file_space(xfs_assert_ilocked (ip,XFS_IOLOCK_EXCL |XFS_MMAPLOCK_EXCL ); -trace_xfs_insert_file_space (ip); +trace_xfs_insert_file_space (ip, offset, len); error =xfs_bmap_can_insert_extents (ip, stop_fsb, shift_fsb);if (error)
fs/xfs/xfs_file.c +1 -1
@@ -1362,7 +1362,7 @@ xfs_falloc_zero_range(loff_t new_size =0 ;int error; -trace_xfs_zero_file_space (XFS_I (inode)); +trace_xfs_zero_file_space (XFS_I (inode), offset, len); error =xfs_falloc_newsize (file, mode, offset, len, &new_size);if (error)
fs/xfs/xfs_trace.h +5 -5
@@ -928,11 +928,6 @@ DEFINE_INODE_EVENT(xfs_getattr);DEFINE_INODE_EVENT (xfs_setattr);DEFINE_INODE_EVENT (xfs_readlink);DEFINE_INODE_EVENT (xfs_inactive_symlink); -DEFINE_INODE_EVENT (xfs_alloc_file_space); -DEFINE_INODE_EVENT (xfs_free_file_space); -DEFINE_INODE_EVENT (xfs_zero_file_space); -DEFINE_INODE_EVENT (xfs_collapse_file_space); -DEFINE_INODE_EVENT (xfs_insert_file_space);DEFINE_INODE_EVENT (xfs_readdir);#ifdef CONFIG_XFS_POSIX_ACL DEFINE_INODE_EVENT (xfs_get_acl);
@@ -1732,6 +1727,11 @@ DEFINE_SIMPLE_IO_EVENT(xfs_end_io_direct_write_unwritten);DEFINE_SIMPLE_IO_EVENT (xfs_end_io_direct_write_append);DEFINE_SIMPLE_IO_EVENT (xfs_file_splice_read);DEFINE_SIMPLE_IO_EVENT (xfs_zoned_map_blocks); +DEFINE_SIMPLE_IO_EVENT (xfs_alloc_file_space); +DEFINE_SIMPLE_IO_EVENT (xfs_free_file_space); +DEFINE_SIMPLE_IO_EVENT (xfs_zero_file_space); +DEFINE_SIMPLE_IO_EVENT (xfs_collapse_file_space); +DEFINE_SIMPLE_IO_EVENT (xfs_insert_file_space); DECLARE_EVENT_CLASS(xfs_itrunc_class, TP_PROTO(struct xfs_inode *ip, xfs_fsize_t new_size),
PATCH 3/4 xfs: add an ioctl to map free space into a file
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new ioctl to map free physical space into a file, at the same file offset as if the file were a sparse image of the physical device backing the filesystem. The intent here is to use this to prototype a free space defragmentation tool.
--- fs/xfs/libxfs/xfs_alloc.c | 88 +++++++++++++ fs/xfs/libxfs/xfs_alloc.h | 3 fs/xfs/libxfs/xfs_bmap.c | 1 fs/xfs/libxfs/xfs_fs.h | 14 ++ fs/xfs/xfs_bmap_util.c | 303 +++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_bmap_util.h | 1 fs/xfs/xfs_file.c | 139 +++++++++++++++++++++ fs/xfs/xfs_file.h | 2 fs/xfs/xfs_ioctl.c | 5 + fs/xfs/xfs_trace.h | 35 +++++ 10 files changed, 591 insertions(+)
fs/xfs/libxfs/xfs_alloc.c +88 -0
@@ -4168,3 +4168,91 @@ xfs_extfree_intent_destroy_cache(void)kmem_cache_destroy (xfs_extfree_item_cache); xfs_extfree_item_cache =NULL ; } + +/* +* Find the next chunk of free space in @pag starting at @agbno and going no +* higher than @end_agbno. Set @agbno and @len to whatever free space we find, +* or to @end_agbno if we find no space. +*/ +int +xfs_alloc_find_freesp ( +struct xfs_trans *tp, +struct xfs_perag *pag, +xfs_agblock_t *agbno, +xfs_agblock_t end_agbno, +xfs_extlen_t *len) +{ +struct xfs_mount *mp =pag_mount (pag); +struct xfs_btree_cur *cur; +struct xfs_buf *agf_bp =NULL ; +xfs_agblock_t found_agbno; +xfs_extlen_t found_len; +int found; +int error; + +trace_xfs_alloc_find_freesp (pag_group (pag), *agbno, + end_agbno - *agbno); + + error =xfs_alloc_read_agf (pag, tp,0 , &agf_bp); +if (error) +return error; + + cur =xfs_bnobt_init_cursor (mp, tp, agf_bp, pag); + +/* Try to find a free extent that starts before here. */ + error =xfs_alloc_lookup_le (cur, *agbno,0 , &found); +if (error) + goto out_cur; +if (found) { + error =xfs_alloc_get_rec (cur, &found_agbno, &found_len, + &found); +if (error) + goto out_cur; +if (XFS_IS_CORRUPT (mp, !found)) { +xfs_btree_mark_sick (cur); + error = -EFSCORRUPTED ; + goto out_cur; + } + +if (found_agbno + found_len > *agbno) + goto found; + } + +/* Examine the next record if free extent not in range. */ + error =xfs_btree_increment (cur,0 , &found); +if (error) + goto out_cur; +if (!found) + goto next_ag; + + error =xfs_alloc_get_rec (cur, &found_agbno, &found_len, &found); +if (error) + goto out_cur; +if (XFS_IS_CORRUPT (mp, !found)) { +xfs_btree_mark_sick (cur); + error = -EFSCORRUPTED ; + goto out_cur; + } + +if (found_agbno >= end_agbno) + goto next_ag; + +found: +/* Found something, so update the mapping. */ +trace_xfs_alloc_find_freesp_done (pag_group (pag), found_agbno, + found_len); +if (found_agbno < *agbno) { + found_len -= *agbno - found_agbno; + found_agbno = *agbno; + } + *len = found_len; + *agbno = found_agbno; + goto out_cur; +next_ag: +/* Found nothing, so advance the cursor beyond the end of the range. */ + *agbno = end_agbno; + *len =0 ; +out_cur: +xfs_btree_del_cursor (cur, error); +return error; +}
fs/xfs/libxfs/xfs_alloc.h +3 -0
@@ -286,5 +286,8 @@ void xfs_extfree_intent_destroy_cache(void);xfs_failaddr_t xfs_validate_ag_length (struct xfs_buf *bp,uint32_t seqno,uint32_t length); +int xfs_alloc_find_freesp (struct xfs_trans *tp,struct xfs_perag *pag, +xfs_agblock_t *agbno,xfs_agblock_t end_agbno, +xfs_extlen_t *len);#endif /* __XFS_ALLOC_H__ */
fs/xfs/libxfs/xfs_bmap.c +1 -0
@@ -41,6 +41,7 @@#include "xfs_inode_util.h" #include "xfs_rtgroup.h" #include "xfs_zone_alloc.h" +#include "xfs_rtalloc.h" struct kmem_cache *xfs_bmap_intent_cache;
fs/xfs/libxfs/xfs_fs.h +14 -0
@@ -1087,6 +1087,19 @@ xfs_getfsrefs_advance(/* fcr_flags values - returned for each non-header segment */ #define FCR_OF_LAST (1U << 0)/* last record in the dataset */ +/* map free space to file */ + +/* +* XFS_IOC_MAP_FREESP maps all the free physical space in the filesystem into +* the file at the same offsets. This ioctl requires CAP_SYS_ADMIN. +*/ +struct xfs_map_freesp { +__s64 offset;/* disk address to map, in bytes */ +__s64 len;/* length in bytes */ +__u64 flags;/* must be zero */ +__u64 pad;/* must be zero */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1127,6 +1140,7 @@ xfs_getfsrefs_advance(#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_GETFSREFCOUNTS _IOWR('X', 66, struct xfs_getfsrefs_head) +#define XFS_IOC_MAP_FREESP _IOW ('X', 67, struct xfs_map_freesp)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_bmap_util.c +303 -0
312 lines not on this page · raw message
fs/xfs/xfs_bmap_util.h +1 -0
@@ -63,6 +63,7 @@ int xfs_collapse_file_space(struct xfs_inode *, xfs_off_t offset, xfs_off_t len, struct xfs_zone_alloc_ctx *ac);int xfs_insert_file_space (struct xfs_inode *,xfs_off_t offset,xfs_off_t len); +int xfs_map_free_space (struct xfs_inode *ip,xfs_off_t off,xfs_off_t len);/* EOF block manipulation functions */ bool xfs_can_free_eofblocks (struct xfs_inode *ip);
fs/xfs/xfs_file.c +139 -0
151 lines not on this page · raw message
fs/xfs/xfs_file.h +2 -0
@@ -15,4 +15,6 @@ bool xfs_is_falloc_aligned(struct xfs_inode *ip, loff_t pos,bool xfs_truncate_needs_cow_around (struct xfs_inode *ip,loff_t pos);int xfs_file_unshare_at (struct xfs_inode *ip,loff_t pos); +long xfs_ioc_map_freesp (struct file *file,struct xfs_map_freesp __user *argp); +#endif /* __XFS_FILE_H__ */
fs/xfs/xfs_ioctl.c +5 -0
@@ -45,6 +45,8 @@#include <linux/mount.h> #include <linux/fileattr.h> +#include <linux/security.h> +#include <linux/fsnotify.h> /* Return 0 on success or positive error */ int
@@ -1429,6 +1431,9 @@ xfs_file_ioctl(case XFS_IOC_COMMIT_RANGE :return xfs_ioc_commit_range (filp, arg); +case XFS_IOC_MAP_FREESP : +return xfs_ioc_map_freesp (filp, arg); +default :return -ENOTTY ; }
fs/xfs/xfs_trace.h +35 -0
@@ -1732,6 +1732,7 @@ DEFINE_SIMPLE_IO_EVENT(xfs_free_file_space);DEFINE_SIMPLE_IO_EVENT (xfs_zero_file_space);DEFINE_SIMPLE_IO_EVENT (xfs_collapse_file_space);DEFINE_SIMPLE_IO_EVENT (xfs_insert_file_space); +DEFINE_SIMPLE_IO_EVENT (xfs_map_free_space); DECLARE_EVENT_CLASS(xfs_itrunc_class, TP_PROTO(struct xfs_inode *ip, xfs_fsize_t new_size),
@@ -1821,6 +1822,36 @@ TRACE_EVENT(xfs_bunmap, ); +DECLARE_EVENT_CLASS (xfs_map_free_extent_class, + TP_PROTO(struct xfs_inode *ip, xfs_fileoff_t bno, xfs_extlen_t len), +TP_ARGS (ip, bno, len), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(xfs_ino_t, ino) + __field(xfs_fsize_t, size) + __field(xfs_fileoff_t, bno) +__field (xfs_extlen_t, len) + ), +TP_fast_assign ( + __entry->dev = VFS_I(ip)->i_sb->s_dev; + __entry->ino = ip->i_ino; + __entry->size = ip->i_disk_size; + __entry->bno = bno; + __entry->len = len; + ), +TP_printk ("dev %d:%d ino 0x%llx disize 0x%llx fileoff 0x%llx fsbcount 0x%x" , +MAJOR (__entry->dev),MINOR (__entry->dev), + __entry->ino, + __entry->size, + __entry->bno, + __entry->len) +); +#define DEFINE_MAP_FREE_EXTENT_EVENT (name) \ +DEFINE_EVENT(xfs_map_free_extent_class, name, \ + TP_PROTO(struct xfs_inode *ip, xfs_fileoff_t bno, xfs_extlen_t len), \ + TP_ARGS(ip, bno, len)) +DEFINE_MAP_FREE_EXTENT_EVENT (xfs_map_free_ag_extent); + DECLARE_EVENT_CLASS(xfs_extent_busy_class, TP_PROTO(const struct xfs_group *xg, xfs_agblock_t agbno, xfs_extlen_t len),
@@ -1856,6 +1887,8 @@ DEFINE_BUSY_EVENT(xfs_extent_busy);DEFINE_BUSY_EVENT (xfs_extent_busy_force);DEFINE_BUSY_EVENT (xfs_extent_busy_reuse);DEFINE_BUSY_EVENT (xfs_extent_busy_clear); +DEFINE_BUSY_EVENT (xfs_alloc_find_freesp); +DEFINE_BUSY_EVENT (xfs_alloc_find_freesp_done); TRACE_EVENT(xfs_extent_busy_trim, TP_PROTO(const struct xfs_group *xg, xfs_agblock_t agbno,
@@ -3962,6 +3995,7 @@ DECLARE_EVENT_CLASS(xfs_inode_irec_class,DEFINE_EVENT (xfs_inode_irec_class, name, \ TP_PROTO(struct xfs_inode *ip, struct xfs_bmbt_irec *irec), \ TP_ARGS(ip, irec)) +DEFINE_INODE_IREC_EVENT (xfs_map_free_ag_extent_done);/* inode iomap invalidation events */ DECLARE_EVENT_CLASS(xfs_wb_invalid_class,
@@ -4096,6 +4130,7 @@ DEFINE_INODE_ERROR_EVENT(xfs_reflink_remap_blocks_error);DEFINE_INODE_ERROR_EVENT (xfs_reflink_remap_extent_error);DEFINE_INODE_IREC_EVENT (xfs_reflink_remap_extent_src);DEFINE_INODE_IREC_EVENT (xfs_reflink_remap_extent_dest); +DEFINE_INODE_ERROR_EVENT (xfs_map_free_reserve_more_fail);/* dedupe tracepoints */ DEFINE_DOUBLE_IO_EVENT (xfs_reflink_compare_extents);
PATCH 4/4 xfs: implement FALLOC_FL_MAP_FREE for realtime files
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Implement mapfree for realtime space.
--- fs/xfs/xfs_bmap_util.c | 202 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_bmap_util.h | 2 fs/xfs/xfs_file.c | 14 ++- fs/xfs/xfs_rtalloc.c | 108 ++++++++++++++++++++++++++ fs/xfs/xfs_rtalloc.h | 7 ++ fs/xfs/xfs_trace.h | 41 ++++++++++ 6 files changed, 368 insertions(+), 6 deletions(-)
fs/xfs/xfs_bmap_util.c +202 -0
@@ -2219,3 +2219,205 @@ xfs_map_free_space(return 0 ;return error; } + +#ifdef CONFIG_XFS_RT +/* +* Given a file and a free rt extent, map it into the file at the same offset +* if the file were a sparse image of the physical device. Set @mval to +* whatever mapping we added to the file. +*/ +STATIC int +xfs_map_free_rtgroup_extent ( +struct xfs_trans *tp, +struct xfs_inode *ip, +struct xfs_rtgroup *rtg, +xfs_rtxnum_t rtx, +xfs_rtxlen_t rtxlen, +struct xfs_bmbt_irec *mval) +{ +struct xfs_bmbt_irec irec; +struct xfs_mount *mp = ip->i_mount; +xfs_fsblock_t fsbno =xfs_rtx_to_rtb (rtg, rtx); +xfs_fileoff_t startoff = fsbno; +xfs_extlen_t len =xfs_rtbxlen_to_blen (mp, rtxlen); +int nimaps; +int error; + +ASSERT (XFS_IS_REALTIME_INODE (ip)); + +trace_xfs_map_free_rt_extent (ip, fsbno, len); + +/* Make sure the entire range is a hole. */ + nimaps =1 ; + error =xfs_bmapi_read (ip, startoff, len, &irec, &nimaps,0 ); +if (error) +return error; + +if (irec.br_startoff != startoff || + irec.br_startblock !=HOLESTARTBLOCK || + irec.br_blockcount < len) +return -EINVAL ; + + error =xfs_iext_count_extend (tp, ip,XFS_DATA_FORK , +XFS_IEXT_ADD_NOSPLIT_CNT ); +if (error) +return error; + +/* +* Allocate the physical extent. We should not have dropped the lock +* since the scan of the free space metadata, so this should work, +* though the length may be adjusted to play nicely with metadata space +* reservations. +*/ + error =xfs_rtallocate_exact (tp, rtg, rtx, rtxlen); +if (error) +return error; + +/* Map extent into file, update quota. */ + mval->br_blockcount = len; + mval->br_startblock = fsbno; + mval->br_startoff = startoff; + mval->br_state =XFS_EXT_UNWRITTEN ; + +trace_xfs_map_free_rt_extent_done (ip, mval); + +xfs_bmap_map_extent (tp, ip,XFS_DATA_FORK , mval); +xfs_trans_mod_dquot_byino (tp, ip,XFS_TRANS_DQ_RTBCOUNT , + mval->br_blockcount); + +return 0 ; +} + +/* Find a free extent in this rtgroup and map it into the file. */ +STATIC int +xfs_map_free_rt_extent ( +struct xfs_inode *ip, +struct xfs_rtgroup *rtg, +xfs_rtxnum_t *cursor, +xfs_rtxnum_t end_rtx) +{ +struct xfs_bmbt_irec irec; +struct xfs_mount *mp = ip->i_mount; +struct xfs_trans *tp; +loff_t endpos; +xfs_rtxlen_t len_rtx; +xfs_extlen_t free_len; +int error; + +if (fatal_signal_pending (current)) +return -EINTR ; + + error =xfs_trans_alloc_inode (ip, &M_RES (mp)->tr_write,0 ,0 , false, + &tp); +if (error) +return error; + +xfs_rtgroup_lock (rtg,XFS_RTGLOCK_BITMAP ); + + error =xfs_rtallocate_find_freesp (tp, rtg, cursor, end_rtx, &len_rtx); +if (error) + goto out_rtglock; + +/* +* If off_rtx is beyond the end of the rt device or is past what the +* user asked for, bail out. +*/ +if (*cursor >= end_rtx) + goto out_rtglock; + + free_len =xfs_rtxlen_to_extlen (mp, len_rtx); + error =xfs_map_free_reserve_more (tp, ip, &free_len); +if (error) + goto out_rtglock; + + error =xfs_map_free_rtgroup_extent (tp, ip, rtg, *cursor, len_rtx, + &irec); +if (error == -EAGAIN ) { +/* +* The allocator was busy and told us to try again. The +* transaction could be dirty due to a nrext64 upgrade, so +* commit the transaction and try again without advancing +* the cursor. +* +* XXX do we fail to unlock something here? +*/ +xfs_rtgroup_unlock (rtg,XFS_RTGLOCK_BITMAP ); + error =xfs_trans_commit (tp); +xfs_iunlock (ip,XFS_ILOCK_EXCL ); +return error; + } +if (error) + goto out_cancel; + +/* Update isize if needed. */ + endpos =XFS_FSB_TO_B (mp, irec.br_startoff + irec.br_blockcount); +if (endpos >i_size_read (VFS_I (ip))) { +i_size_write (VFS_I (ip), endpos); + ip->i_disk_size = endpos; +xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); + } + + error =xfs_trans_commit (tp); +xfs_iunlock (ip,XFS_ILOCK_EXCL ); +if (error) +return error; + +ASSERT (xfs_blen_to_rtxoff (mp, irec.br_blockcount) ==0 ); + *cursor +=xfs_extlen_to_rtxlen (mp, irec.br_blockcount); +return 0 ; +out_rtglock: +xfs_rtgroup_unlock (rtg,XFS_RTGLOCK_BITMAP ); +out_cancel: +xfs_trans_cancel (tp); +xfs_iunlock (ip,XFS_ILOCK_EXCL ); +return error; +} + +/* +* Allocate all free physical space between off and len and map it to this +* regular realtime file. +*/ +int +xfs_map_free_rt_space ( +struct xfs_inode *ip, +xfs_off_t off, +xfs_off_t len) +{ +struct xfs_mount *mp = ip->i_mount; +struct xfs_rtgroup *rtg =NULL ; +xfs_daddr_t off_daddr =BTOBB (off); +xfs_daddr_t end_daddr =BTOBBT (off + len); +xfs_rtblock_t off_rtb =xfs_daddr_to_rtb (mp, off_daddr); +xfs_rtblock_t end_rtb =xfs_daddr_to_rtb (mp, end_daddr); +xfs_rgnumber_t off_rgno =xfs_rtb_to_rgno (mp, off_rtb); +xfs_rgnumber_t end_rgno =xfs_rtb_to_rgno (mp, end_rtb); +int error =0 ; + +trace_xfs_map_free_rt_space (ip, off, len); + +while ((rtg =xfs_rtgroup_next_range (mp, rtg, off_rgno, + mp->m_sb.sb_rgcount))) { +xfs_rtxnum_t off_rtx =0 ; +xfs_rtxnum_t end_rtx = rtg->rtg_extents; + +if (rtg_rgno (rtg) == off_rgno) + off_rtx =xfs_rtb_to_rtx (mp, off_rtb); +if (rtg_rgno (rtg) == end_rgno) + end_rtx =min (end_rtx,xfs_rtb_to_rtx (mp, end_rtb)); + +while (off_rtx < end_rtx) { + error =xfs_map_free_rt_extent (ip, rtg, &off_rtx, + end_rtx); +if (error) + goto out; + } + } + +out: +if (rtg) +xfs_rtgroup_rele (rtg); +if (error == -ENOSPC ) +return 0 ; +return error; +} +#endif
fs/xfs/xfs_bmap_util.h +2 -0
@@ -85,8 +85,10 @@ int xfs_flush_unmap_range(struct xfs_inode *ip, xfs_off_t offset,#ifdef CONFIG_XFS_RT int xfs_convert_rtbigalloc_file_space (struct xfs_inode *ip,loff_t pos,uint64_t len); +int xfs_map_free_rt_space (struct xfs_inode *ip,xfs_off_t off,xfs_off_t len);#else # define xfs_convert_rtbigalloc_file_space (ip, pos, len) (-EOPNOTSUPP) +# define xfs_map_free_rt_space (ip, off, len) (-EOPNOTSUPP)#endif #endif /* __XFS_BMAP_UTIL_H__ */
fs/xfs/xfs_file.c +8 -6
@@ -1580,11 +1580,10 @@ xfs_file_map_freesp(if (error) goto out_unlock; -if (XFS_IS_REALTIME_INODE (ip)) { - error = -EOPNOTSUPP ; - goto out_unlock; - } - device_size =XFS_FSB_TO_B (mp, mp->m_sb.sb_dblocks); +if (XFS_IS_REALTIME_INODE (ip)) + device_size =XFS_FSB_TO_B (mp, mp->m_sb.sb_rblocks); +else + device_size =XFS_FSB_TO_B (mp, mp->m_sb.sb_dblocks);/* * Bail out now if we aren't allowed to make the file size the
@@ -1597,7 +1596,10 @@ xfs_file_map_freesp( goto out_unlock; } - error =xfs_map_free_space (ip, mf->offset, mf->len); +if (XFS_IS_REALTIME_INODE (ip)) + error =xfs_map_free_rt_space (ip, mf->offset, mf->len); +else + error =xfs_map_free_space (ip, mf->offset, mf->len);if (error) {if (error == -ECANCELED ) error =0 ;
fs/xfs/xfs_rtalloc.c +108 -0
111 lines not on this page · raw message
fs/xfs/xfs_rtalloc.h +7 -0
@@ -10,6 +10,7 @@struct xfs_mount ;struct xfs_trans ; +struct xfs_rtgroup ;#ifdef CONFIG_XFS_RT /* rtgroup superblock initialization */
@@ -48,6 +49,10 @@ xfs_growfs_rt(int xfs_rtalloc_reinit_frextents (struct xfs_mount *mp);int xfs_growfs_check_rtgeom (const struct xfs_mount *mp,xfs_rfsblock_t dblocks,xfs_rfsblock_t rblocks,xfs_agblock_t rextsize); +int xfs_rtallocate_find_freesp (struct xfs_trans *tp,struct xfs_rtgroup *rtg, +xfs_rtxnum_t *rtx,xfs_rtxnum_t end_rtx,xfs_rtxlen_t *len_rtx); +int xfs_rtallocate_exact (struct xfs_trans *tp,struct xfs_rtgroup *rtg, +xfs_rtxnum_t rtx,xfs_rtxlen_t rtxlen);#else # define xfs_growfs_rt (mp,in) (-ENOSYS)# define xfs_rtalloc_reinit_frextents (m) (0)
@@ -67,6 +72,8 @@ xfs_rtmount_init(# define xfs_rtunmount_inodes (m)# define xfs_rt_resv_free (mp) ((void)0)# define xfs_rt_resv_init (mp) (0) +# define xfs_rtallocate_find_freesp (...) (-EOPNOTSUPP) +# define xfs_rtallocate_exact (...) (-EOPNOTSUPP) static inline int xfs_growfs_check_rtgeom(const struct xfs_mount *mp,
fs/xfs/xfs_trace.h +41 -0
71 lines not on this page · raw message
PATCH 4/5 xfsprogs: live health monitoring of filesystems
In reply to Darrick J. WongHi all,
This patchset builds off of Kent Overstreet's thread_with_file code to deliver live information about filesystem health events to userspace. This is done by creating a twf file and hooking internal operations so that the event information can be queued to the twf without stalling the kernel if the twf client program is nonresponsive. This is a private ioctl, so events are expressed using simple json objects so that we can enrich the output later on without having to rev a ton of C structs.
In userspace, we create a new daemon program that will read the json event objects and initiate repairs automatically. This daemon is managed entirely by systemd and will not block unmounting of the filesystem unless repairs are ongoing. It is autostarted via some horrible udev rules.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* xfs: create hooks for monitoring health updates
* xfs: create a special file to pass filesystem health to userspace
* xfs: create event queuing, formatting, and discovery infrastructure
* xfs: report metadata health events through healthmon
* xfs: report shutdown events through healthmon
* xfs: report media errors through healthmon
* xfs: report file io errors through healthmon
* xfs: add media error reporting ioctl
* xfs_io: monitor filesystem health events
* xfs_io: add a media error reporting command
* xfs_scrubbed: create daemon to listen for health events
* xfs_scrubbed: check events against schema
* xfs_scrubbed: enable repairing filesystems
* xfs_scrubbed: check for fs features needed for effective repairs
* xfs_scrubbed: use getparents to look up file names
* builddefs: refactor udev directory specification
* xfs_scrubbed: create a background monitoring service
* xfs_scrubbed: don't start service if kernel support unavailable
* xfs_scrubbed: use the autofsck fsproperty to select mode
* xfs_scrub: report media scrub failures to the kernel
* debian: enable xfs_scrubbed on the root filesystem by default
--- configure.ac | 2 debian/control | 2 debian/postinst | 8 debian/prerm | 13 include/builddefs.in | 3 io/Makefile | 1 io/healthmon.c | 183 ++++++ io/init.c | 1 io/io.h | 1 io/shutdown.c | 113 ++++ libxfs/Makefile | 10 libxfs/xfs_fs.h | 31 + libxfs/xfs_health.h | 47 ++ libxfs/xfs_healthmon.schema.json | 595 ++++++++++++++++++++ m4/package_services.m4 | 30 + man/man8/xfs_io.8 | 46 ++ scrub/Makefile | 34 + scrub/phase6.c | 25 + scrub/xfs_scrubbed.in | 1106 ++++++++++++++++++++++++++++++++++++++ scrub/xfs_scrubbed.rules | 7 scrub/[email protected] | 104 ++++ scrub/xfs_scrubbed_start | 17 + 22 files changed, 2354 insertions(+), 25 deletions(-) create mode 100644 debian/prerm create mode 100644 io/healthmon.c create mode 100644 libxfs/xfs_healthmon.schema.json create mode 100644 scrub/xfs_scrubbed.in create mode 100644 scrub/xfs_scrubbed.rules create mode 100644 scrub/[email protected] create mode 100755 scrub/xfs_scrubbed_start
PATCH 1/21 xfs: create hooks for monitoring health updates
From: Darrick J. Wong <[email protected]>
Create hooks for monitoring health events.
--- libxfs/xfs_health.h | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
libxfs/xfs_health.h +47 -0
@@ -289,4 +289,51 @@ void xfs_bulkstat_health(struct xfs_inode *ip, struct xfs_bulkstat *bs);#define xfs_metadata_is_sick (error) \ (unlikely((error) == -EFSCORRUPTED || (error) == -EFSBADCRC)) +/* +* Parameters for tracking health updates. The enum below is passed as the +* hook function argument. +*/ +enum xfs_health_update_type { +XFS_HEALTHUP_SICK =1 ,/* runtime corruption observed */ +XFS_HEALTHUP_CORRUPT ,/* fsck reported corruption */ +XFS_HEALTHUP_HEALTHY ,/* fsck reported healthy structure */ +XFS_HEALTHUP_UNMOUNT ,/* filesystem is unmounting */ +}; + +/* Where in the filesystem was the event observed? */ +enum xfs_health_update_domain { +XFS_HEALTHUP_FS =1 ,/* main filesystem */ +XFS_HEALTHUP_AG ,/* allocation group */ +XFS_HEALTHUP_INODE ,/* inode */ +XFS_HEALTHUP_RTGROUP ,/* realtime group */ +}; + +struct xfs_health_update_params { +/* XFS_HEALTHUP_INODE */ +xfs_ino_t ino; +uint32_t gen; + +/* XFS_HEALTHUP_AG/RTGROUP */ +uint32_t group; + +/* XFS_SICK_* flags */ +unsigned int old_mask; +unsigned int new_mask; + +enum xfs_health_update_domain domain; +}; + +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_health_hook { +struct xfs_hook health_hook; +}; + +void xfs_health_hook_disable (void ); +void xfs_health_hook_enable (void ); + +int xfs_health_hook_add (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_del (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn); +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_HEALTH_H__ */
PATCH 2/21 xfs: create a special file to pass filesystem health to userspace
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create an ioctl that installs a file descriptor backed by an anon_inode file that will convey filesystem health events to userspace.
--- libxfs/xfs_fs.h | 8 ++++++++ 1 file changed, 8 insertions(+)
libxfs/xfs_fs.h +8 -0
@@ -1100,6 +1100,13 @@ struct xfs_map_freesp {__u64 pad;/* must be zero */ }; +struct xfs_health_monitor { +__u64 flags;/* flags */ +__u8 format;/* output format */ +__u8 pad1[7 ];/* zeroes */ +__u64 pad2[2 ];/* zeroes */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1141,6 +1148,7 @@ struct xfs_map_freesp {#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_GETFSREFCOUNTS _IOWR('X', 66, struct xfs_getfsrefs_head)#define XFS_IOC_MAP_FREESP _IOW ('X', 67, struct xfs_map_freesp) +#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 3/21 xfs: create event queuing, formatting, and discovery infrastructure
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create the basic infrastructure that we need to report health events to userspace. We need a compact form for recording critical information about an event and queueing them; a means to notice that we've lost some events; and a means to format the events into something that userspace can handle.
Here, we've chosen json to export information to userspace. The structured key-value nature of json gives us enormous flexibility to modify the schema of what we'll send to userspace because we can add new keys at any time. Userspace can use whatever json parsers are available to consume the events and will not be confused by keys they don't recognize.
Note that we do NOT allow sending json back to the kernel, nor is there any intent to do that.
--- libxfs/xfs_fs.h | 8 +++++ libxfs/xfs_healthmon.schema.json | 63 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 libxfs/xfs_healthmon.schema.json
libxfs/xfs_fs.h +8 -0
@@ -1107,6 +1107,14 @@ struct xfs_health_monitor {__u64 pad2[2 ];/* zeroes */ }; +/* Return all health status events, not just deltas */ +#define XFS_HEALTH_MONITOR_VERBOSE (1ULL << 0) + +#define XFS_HEALTH_MONITOR_ALL (XFS_HEALTH_MONITOR_VERBOSE) + +/* Return events in JSON format */ +#define XFS_HEALTH_MONITOR_FMT_JSON (1) +/* * ioctl commands that are used by Linux filesystems */
libxfs/xfs_healthmon.schema.json new file +63 -0
63 lines not on this page · raw message
PATCH 4/21 xfs: report metadata health events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a metadata health event hook so that we can send events to userspace as we collect information. The unmount hook severs the weak reference between the health monitor and the filesystem it's monitoring; when this happens, we stop reporting events because there's no longer any point.
--- libxfs/xfs_healthmon.schema.json | 328 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 328 insertions(+)
libxfs/xfs_healthmon.schema.json +328 -0
@@ -18,6 +18,18 @@ "oneOf": [ { "$ref": "#/$events/lost" + }, + { + "$ref": "#/$events/fs_metadata" + }, + { + "$ref": "#/$events/rtgroup_metadata" + }, + { + "$ref": "#/$events/perag_metadata" + }, + { + "$ref": "#/$events/inode_metadata" } ],
@@ -27,6 +39,169 @@ "title": "Time of Event", "description": "Timestamp of the event, in nanoseconds since the Unix epoch.", "type": "integer" + }, + "xfs_agnumber_t": { + "description": "Allocation group number", + "type": "integer", + "minimum": 0, + "maximum": 2147483647 + }, + "xfs_rgnumber_t": { + "description": "Realtime allocation group number", + "type": "integer", + "minimum": 0, + "maximum": 2147483647 + }, + "xfs_ino_t": { + "description": "Inode number", + "type": "integer", + "minimum": 1 + }, + "i_generation": { + "description": "Inode generation number", + "type": "integer" + } + }, + + "$comment": "Filesystem metadata event data are defined here.", + "$metadata": { + "status": { + "description": "Metadata health status", + "$comment": [ + "One of:", + "", + " * sick: metadata corruption discovered", + " during a runtime operation.", + " * corrupt: corruption discovered during", + " an xfs_scrub run.", + " * healthy: metadata object was found to be", + " ok by xfs_scrub." + ], + "enum": [ + "sick", + "corrupt", + "healthy" + ] + }, + "fs": { + "description": [ + "Metadata structures that affect the entire", + "filesystem. Options include:", + "", + " * fscounters: summary counters", + " * usrquota: user quota records", + " * grpquota: group quota records", + " * prjquota: project quota records", + " * quotacheck: quota counters", + " * nlinks: file link counts", + " * metadir: metadata directory", + " * metapath: metadata inode paths" + ], + "enum": [ + "fscounters", + "grpquota", + "metadir", + "metapath", + "nlinks", + "prjquota", + "quotacheck", + "usrquota" + ] + }, + "perag": { + "description": [ + "Metadata structures owned by allocation", + "groups on the data device. Options include:", + "", + " * agf: group space header", + " * agfl: per-group free block list", + " * agi: group inode header", + " * bnobt: free space by position btree", + " * cntbt: free space by length btree", + " * finobt: free inode btree", + " * inobt: inode btree", + " * rmapbt: reverse mapping btree", + " * refcountbt: reference count btree", + " * inodes: problems were recorded for", + " this group's inodes, but the", + " inodes themselves had to be", + " reclaimed.", + " * super: superblock" + ], + "enum": [ + "agf", + "agfl", + "agi", + "bnobt", + "cntbt", + "finobt", + "inobt", + "inodes", + "refcountbt", + "rmapbt", + "super" + ] + }, + "rtgroup": { + "description": [ + "Metadata structures owned by allocation", + "groups on the realtime volume. Options", + "include:", + "", + " * bitmap: free space bitmap contents", + " for this group", + " * summary: realtime free space summary file", + " * rmapbt: reverse mapping btree", + " * refcountbt: reference count btree", + " * super: group superblock" + ], + "enum": [ + "bitmap", + "summary", + "refcountbt", + "rmapbt", + "super" + ] + }, + "inode": { + "description": [ + "Metadata structures owned by file inodes.", + "Options include:", + "", + " * bmapbta: attr fork", + " * bmapbtc: cow fork", + " * bmapbtd: data fork", + " * core: inode record", + " * directory: directory entries", + " * dirtree: directory tree problems detected", + " * parent: directory parent pointer", + " * symlink: symbolic link target", + " * xattr: extended attributes", + "", + "These are set when an inode record repair had", + "to drop the corresponding data structure to", + "get the inode back to a consistent state.", + "", + " * bmapbtd_zapped", + " * bmapbta_zapped", + " * directory_zapped", + " * symlink_zapped" + ], + "enum": [ + "bmapbta", + "bmapbta_zapped", + "bmapbtc", + "bmapbtd", + "bmapbtd_zapped", + "core", + "directory", + "directory_zapped", + "dirtree", + "parent", + "symlink", + "symlink_zapped", + "xattr" + ] } },
@@ -58,6 +233,159 @@ "time_ns", "domain" ] + }, + "fs_metadata": { + "title": "Filesystem-wide metadata event", + "description": [ + "Health status updates for filesystem-wide", + "metadata objects." + ], + "type": "object", + + "properties": { + "type": { + "$ref": "#/$metadata/status" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "fs" + }, + "structures": { + "type": "array", + "items": { + "$ref": "#/$metadata/fs" + }, + "minItems": 1 + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "structures" + ] + }, + "perag_metadata": { + "title": "Data device allocation group metadata event", + "description": [ + "Health status updates for data device ", + "allocation group metadata." + ], + "type": "object", + + "properties": { + "type": { + "$ref": "#/$metadata/status" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "perag" + }, + "group": { + "$ref": "#/$defs/xfs_agnumber_t" + }, + "structures": { + "type": "array", + "items": { + "$ref": "#/$metadata/perag" + }, + "minItems": 1 + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "group", + "structures" + ] + }, + "rtgroup_metadata": { + "title": "Realtime allocation group metadata event", + "description": [ + "Health status updates for realtime allocation", + "group metadata." + ], + "type": "object", + + "properties": { + "type": { + "$ref": "#/$metadata/status" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "rtgroup" + }, + "group": { + "$ref": "#/$defs/xfs_rgnumber_t" + }, + "structures": { + "type": "array", + "items": { + "$ref": "#/$metadata/rtgroup" + }, + "minItems": 1 + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "group", + "structures" + ] + }, + "inode_metadata": { + "title": "Inode metadata event", + "description": [ + "Health status updates for inode metadata.", + "The inode and generation number describe the", + "file that is affected by the change." + ], + "type": "object", + + "properties": { + "type": { + "$ref": "#/$metadata/status" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "inode" + }, + "inumber": { + "$ref": "#/$defs/xfs_ino_t" + }, + "generation": { + "$ref": "#/$defs/i_generation" + }, + "structures": { + "type": "array", + "items": { + "$ref": "#/$metadata/inode" + }, + "minItems": 1 + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "inumber", + "generation", + "structures" + ] } } }
PATCH 5/21 xfs: report shutdown events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a shutdown hook so that we can send notifications to userspace.
--- libxfs/xfs_healthmon.schema.json | 62 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+)
libxfs/xfs_healthmon.schema.json +62 -0
@@ -30,6 +30,9 @@ }, { "$ref": "#/$events/inode_metadata" + }, + { + "$ref": "#/$events/shutdown" } ],
@@ -205,6 +208,31 @@ } }, + "$comment": "Shutdown event data are defined here.", + "$shutdown": { + "reason": { + "description": [ + "Reason for a filesystem to shut down.", + "Options include:", + "", + " * corrupt_incore: in-memory corruption", + " * corrupt_ondisk: on-disk corruption", + " * device_removed: device removed", + " * force_umount: userspace asked for it", + " * log_ioerr: log write IO error", + " * meta_ioerr: metadata writeback IO error" + ], + "enum": [ + "corrupt_incore", + "corrupt_ondisk", + "device_removed", + "force_umount", + "log_ioerr", + "meta_ioerr" + ] + } + }, + "$comment": "Event types are defined here.", "$events": { "lost": {
@@ -386,6 +414,40 @@ "generation", "structures" ] + }, + "shutdown": { + "title": "Abnormal Shutdown Event", + "description": [ + "The filesystem went offline due to", + "unrecoverable errors." + ], + "type": "object", + + "properties": { + "type": { + "const": "shutdown" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "mount" + }, + "reasons": { + "type": "array", + "items": { + "$ref": "#/$shutdown/reason" + }, + "minItems": 1 + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "reasons" + ] } } }
PATCH 6/21 xfs: report media errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we have hooks to report media errors, connect this to the health monitor as well.
--- libxfs/xfs_healthmon.schema.json | 65 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+)
libxfs/xfs_healthmon.schema.json +65 -0
@@ -33,6 +33,9 @@ }, { "$ref": "#/$events/shutdown" + }, + { + "$ref": "#/$events/media_error" } ],
@@ -63,6 +66,31 @@ "i_generation": { "description": "Inode generation number", "type": "integer" + }, + "storage_devs": { + "description": "Storage devices in a filesystem", + "_comment": [ + "One of:", + "", + " * datadev: filesystem device", + " * logdev: external log device", + " * rtdev: realtime volume" + ], + "enum": [ + "datadev", + "logdev", + "rtdev" + ] + }, + "xfs_daddr_t": { + "description": "Storage device address, in units of 512-byte blocks", + "type": "integer", + "minimum": 0 + }, + "bbcount": { + "description": "Storage space length, in units of 512-byte blocks", + "type": "integer", + "minimum": 1 } },
@@ -448,6 +476,43 @@ "domain", "reasons" ] + }, + "media_error": { + "title": "Media Error", + "description": [ + "A storage device reported a media error.", + "The domain element tells us which storage", + "device reported the media failure. The", + "daddr and bbcount elements tell us where", + "inside that device the failure was observed." + ], + "type": "object", + + "properties": { + "type": { + "const": "media" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "$ref": "#/$defs/storage_devs" + }, + "daddr": { + "$ref": "#/$defs/xfs_daddr_t" + }, + "bbcount": { + "$ref": "#/$defs/bbcount" + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "daddr", + "bbcount" + ] } } }
PATCH 7/21 xfs: report file io errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a file io error event hook so that we can send events about read errors, writeback errors, and directio errors to userspace.
--- libxfs/xfs_healthmon.schema.json | 77 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+)
libxfs/xfs_healthmon.schema.json +77 -0
@@ -36,6 +36,9 @@ }, { "$ref": "#/$events/media_error" + }, + { + "$ref": "#/$events/file_ioerror" } ],
@@ -67,6 +70,16 @@ "description": "Inode generation number", "type": "integer" }, + "off_t": { + "description": "File position, in bytes", + "type": "integer", + "minimum": 0 + }, + "size_t": { + "description": "File operation length, in bytes", + "type": "integer", + "minimum": 1 + }, "storage_devs": { "description": "Storage devices in a filesystem", "_comment": [
@@ -261,6 +274,26 @@ } }, + "$comment": "File IO event data are defined here.", + "$fileio": { + "types": { + "description": [ + "File I/O operations. One of:", + "", + " * readahead: reads into the page cache.", + " * writeback: writeback of dirty page cache.", + " * dioread: O_DIRECT reads.", + " * diowrite: O_DIRECT writes." + ], + "enum": [ + "readahead", + "writeback", + "dioread", + "diowrite" + ] + } + }, + "$comment": "Event types are defined here.", "$events": { "lost": {
@@ -513,6 +546,50 @@ "daddr", "bbcount" ] + }, + "file_ioerror": { + "title": "File I/O error", + "description": [ + "A read or a write to a file failed. The", + "inode, generation, pos, and len fields", + "describe the range of the file that is", + "affected." + ], + "type": "object", + + "properties": { + "type": { + "$ref": "#/$fileio/types" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "filerange" + }, + "inumber": { + "$ref": "#/$defs/xfs_ino_t" + }, + "generation": { + "$ref": "#/$defs/i_generation" + }, + "pos": { + "$ref": "#/$defs/off_t" + }, + "len": { + "$ref": "#/$defs/size_t" + } + }, + + "required": [ + "type", + "time_ns", + "domain", + "inumber", + "generation", + "pos", + "len" + ] } } }
PATCH 8/21 xfs: add media error reporting ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new privileged ioctl so that xfs_scrub can report media errors to the kernel for further processing.
--- libxfs/xfs_fs.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
libxfs/xfs_fs.h +15 -0
@@ -1115,6 +1115,20 @@ struct xfs_health_monitor {/* Return events in JSON format */ #define XFS_HEALTH_MONITOR_FMT_JSON (1) +struct xfs_media_error { +__u64 flags;/* flags */ +__u64 daddr;/* disk address of range */ +__u64 bbcount;/* length, in 512b blocks */ +__u64 pad;/* zero */ +}; + +#define XFS_MEDIA_ERROR_DATADEV (1)/* data device */ +#define XFS_MEDIA_ERROR_LOGDEV (2)/* external log device */ +#define XFS_MEDIA_ERROR_RTDEV (3)/* realtime device */ + +/* bottom byte of flags is the device code */ +#define XFS_MEDIA_ERROR_DEVMASK (0xFF) +/* * ioctl commands that are used by Linux filesystems */
@@ -1157,6 +1171,7 @@ struct xfs_health_monitor {#define XFS_IOC_GETFSREFCOUNTS _IOWR('X', 66, struct xfs_getfsrefs_head)#define XFS_IOC_MAP_FREESP _IOW ('X', 67, struct xfs_map_freesp)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor) +#define XFS_IOC_MEDIA_ERROR _IOW ('X', 69, struct xfs_media_error)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 9/21 xfs_io: monitor filesystem health events
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a subcommand to monitor for health events generated by the kernel.
--- io/Makefile | 1 io/healthmon.c | 183 +++++++++++++++++++++++++++++++++++++++++++++++++++++ io/init.c | 1 io/io.h | 1 man/man8/xfs_io.8 | 25 +++++++ 5 files changed, 211 insertions(+) create mode 100644 io/healthmon.c
io/Makefile +1 -0
@@ -26,6 +26,7 @@ CFILES = \ fsuuid.c \ fsync.c \ getrusage.c \ + healthmon.c \ imap.c \ init.c \ inject.c \
io/healthmon.c new file +183 -0
183 lines not on this page · raw message
io/init.c +1 -0
@@ -92,6 +92,7 @@ init_commands(void)crc32cselftest_init ();exchangerange_init ();fsprops_init (); +healthmon_init (); }/*
io/io.h +1 -0
@@ -157,3 +157,4 @@ void exchangerange_init(void);void fsprops_init (void );void aginfo_init (void );void fsrefcounts_init (void ); +void healthmon_init (void );
man/man8/xfs_io.8 +25 -0
@@ -1419,6 +1419,31 @@ .SH FILESYSTEM COMMANDS .RE .PD +.TP +.BI "healthmon [ \-c " bufsize " ] [ \-c ] [ \-d " delay_ms " ] [ \-p ] [ \-v ]" +Watch for filesystem health events and write them to the console. +.RE +.RS 1.0i +.PD 0 +.TP +.BI "\-b " bufsize +Use a buffer of this size to read events from the kernel. +.TP +.BI \-c +Close the open file and replace it with the monitor file. +.TP +.BI "\-d " delay_ms +Sleep for this long between read attempts. +.TP +.B \-p +Probe for the existence of the functionality by opening the monitoring fd and +closing it immediately. +.TP +.BI \-v +Request all health events, even if nothing changed. +.PD +.RE + .TP .BI "inject [ " tag " ]" Inject errors into a filesystem to observe filesystem behavior at
PATCH 10/21 xfs_io: add a media error reporting command
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a subcommand to invoke the media error ioctl to make sure it works.
--- io/shutdown.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++++ man/man8/xfs_io.8 | 21 ++++++++++ 2 files changed, 133 insertions(+), 1 deletion(-)
io/shutdown.c +112 -1
@@ -53,6 +53,115 @@ shutdown_help(void) "\n")); } +static void +mediaerror_help (void ) +{ +printf (_ ( +"\n" +" Report a media error on the data device to the filesystem.\n" +"\n" +" -l -- Report against the log device.\n" +" -r -- Report against the realtime device.\n" +"\n" +" offset is the byte offset of the start of the failed range. If offset is\n" +" specified, mapping length may (optionally) be specified as well." +"\n" +" length is the byte length of the failed range.\n" +"\n" +" If neither offset nor length are specified, the media error report will\n" +" be made against the entire device." +"\n" )); +} + +static int +mediaerror_f ( +int argc, +char **argv) +{ +struct xfs_media_error me = { + .daddr =0 , + .bbcount =-1ULL , + .flags =XFS_MEDIA_ERROR_DATADEV , + }; +long long l; +size_t fsblocksize, fssectsize; +int c, ret; + +init_cvtnum (&fsblocksize, &fssectsize); + +while ((c =getopt (argc, argv,"lr" )) !=EOF ) { +switch (c) { +case 'l' : + me.flags = (me.flags & ~XFS_MEDIA_ERROR_DEVMASK ) | +XFS_MEDIA_ERROR_LOGDEV ; +break ; +case 'r' : + me.flags = (me.flags & ~XFS_MEDIA_ERROR_DEVMASK ) | +XFS_MEDIA_ERROR_RTDEV ; +break ; +default : +mediaerror_help (); + exitcode =1 ; +return 0 ; + } + } + +/* Range start (optional) */ +if (optind < argc) { + l =cvtnum (fsblocksize, fssectsize, argv[optind]); +if (l <0 ) { +printf ("non-numeric offset argument -- %s\n" , + argv[optind]); + exitcode =1 ; +return 0 ; + } + + me.daddr = l /512 ; + optind++; + } + +/* Range length (optional if range start was specified) */ +if (optind < argc) { + l =cvtnum (fsblocksize, fssectsize, argv[optind]); +if (l <0 ) { +printf ("non-numeric len argument -- %s\n" , + argv[optind]); + exitcode =1 ; +return 0 ; + } + + me.bbcount =howmany (l,512 ); + optind++; + } + +if (optind < argc) { +printf ("too many arguments -- %s\n" , argv[optind]); + exitcode =1 ; +return 0 ; + } + + ret =ioctl (file->fd,XFS_IOC_MEDIA_ERROR , &me); +if (ret) { +fprintf (stderr, +"%s: ioctl(XFS_IOC_MEDIA_ERROR) [\"%s\"]: %s\n" , + progname, file->name,strerror (errno)); + exitcode =1 ; +return 0 ; + } + +return 0 ; +} + +static struct cmdinfo mediaerror_cmd = { + .name ="mediaerror" , + .cfunc = mediaerror_f, + .argmin =0 , + .argmax =-1 , + .flags =CMD_FLAG_ONESHOT |CMD_NOMAP_OK , + .args ="[-lr] [offset [length]]" , + .help = mediaerror_help, +}; +void shutdown_init (void ) {
@@ -66,6 +175,8 @@ shutdown_init(void) shutdown_cmd.oneline =_ ("shuts down the filesystem where the current file resides" ); -if (expert) +if (expert) {add_command (&shutdown_cmd); +add_command (&mediaerror_cmd); + } }
man/man8/xfs_io.8 +21 -0
@@ -1452,6 +1452,27 @@ .SH FILESYSTEM COMMANDS argument, displays the list of error tags available. Only available in expert mode and requires privileges. +.TP +.BI "mediaerror [ \-lr ] [ " offset " [ " length " ]]" +Report a media error against the data device of an XFS filesystem. +The +.I offset +and +.I length +parameters are specified in units of bytes. +If neither are specified, the entire device will be reported. +.RE +.RS 1.0i +.PD 0 +.TP +.BI \-l +Report against the log device instead of the data device. +.TP +.BI \-r +Report against the realtime device instead of the data device. +.PD +.RE + .TP .BI "rginfo [ \-r " rgno " ]" Show information about or update the state of realtime allocation groups.
PATCH 11/21 xfs_scrubbed: create daemon to listen for health events
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a daemon program that can listen for and log health events.
--- scrub/Makefile | 15 ++- scrub/xfs_scrubbed.in | 287 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 300 insertions(+), 2 deletions(-) create mode 100644 scrub/xfs_scrubbed.in
scrub/Makefile +13 -2
@@ -18,6 +18,7 @@ XFS_SCRUB_ALL_PROG = xfs_scrub_all XFS_SCRUB_FAIL_PROG = xfs_scrub_fail XFS_SCRUB_ARGS = -p XFS_SCRUB_SERVICE_ARGS = -b -o autofsck +XFS_SCRUBBED_PROG = xfs_scrubbed ifeq ($(HAVE_SYSTEMD),yes) INSTALL_SCRUB += install-systemd SYSTEMD_SERVICES=\
@@ -108,9 +109,9 @@ endif # Automatically trigger a media scan once per month XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL=1mo -LDIRT = $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) *.service *.cron +LDIRT = $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) $(XFS_SCRUBBED_PROG) *.service *.cron -default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) $(OPTIONAL_TARGETS) +default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) $(XFS_SCRUBBED_PROG) $(OPTIONAL_TARGETS) xfs_scrub_all: xfs_scrub_all.in $(builddefs) @echo " [SED] $@"
@@ -123,6 +124,14 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ $(Q)chmod a+x $@ +xfs_scrubbed: xfs_scrubbed.in $(builddefs) + @echo " [SED] $@" + $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ + -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + < $< > $@ + $(Q)chmod a+x $@ + xfs_scrub_fail: xfs_scrub_fail.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
@@ -165,6 +174,8 @@ install-scrub: default $(INSTALL) -m 755 -d $(PKG_SBIN_DIR) $(LTINSTALL) -m 755 $(LTCOMMAND) $(PKG_SBIN_DIR) $(INSTALL) -m 755 $(XFS_SCRUB_ALL_PROG) $(PKG_SBIN_DIR) + $(INSTALL) -m 755 -d $(PKG_LIBEXEC_DIR) + $(INSTALL) -m 755 $(XFS_SCRUBBED_PROG) $(PKG_LIBEXEC_DIR) $(INSTALL) -m 755 -d $(PKG_STATE_DIR) install-udev: $(UDEV_RULES)
scrub/xfs_scrubbed.in new file +287 -0
287 lines not on this page · raw message
PATCH 12/21 xfs_scrubbed: check events against schema
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Validate that the event objects that we get from the kernel actually obey the schema that the kernel publishes.
--- libxfs/Makefile | 10 ++++++-- scrub/Makefile | 1 + scrub/xfs_scrubbed.in | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+), 3 deletions(-)
libxfs/Makefile +7 -3
@@ -151,6 +151,8 @@ EXTRA_OBJECTS=\ LDIRT += $(EXTRA_OBJECTS) +JSON_SCHEMAS=xfs_healthmon.schema.json + # # Tracing flags: # -DMEM_DEBUG all zone memory use
@@ -174,7 +176,7 @@ LTLIBS = $(LIBPTHREAD) $(LIBRT) # don't try linking xfs_repair with a debug libxfs. DEBUG = -DNDEBUG -default: ltdepend $(LTLIBRARY) $(EXTRA_OBJECTS) +default: ltdepend $(LTLIBRARY) $(EXTRA_OBJECTS) $(JSON_SCHEMAS) %dummy.o: %dummy.cpp @echo " [CXXD] $@"
@@ -196,14 +198,16 @@ MAKECXXDEP := $(MAKEDEPEND) $(CXXFLAGS) include $(BUILDRULES) install: default - $(INSTALL) -m 755 -d $(PKG_INC_DIR) + $(INSTALL) -m 755 -d $(PKG_DATA_DIR) + $(INSTALL) -m 644 $(JSON_SCHEMAS) $(PKG_DATA_DIR) install-headers: $(addsuffix -hdrs, $(PKGHFILES)) %-hdrs: $(Q)$(LN_S) -f $(CURDIR)/$* $(TOPDIR)/include/xfs/$* -install-dev: install +install-dev: default + $(INSTALL) -m 755 -d $(PKG_INC_DIR) $(INSTALL) -m 644 $(PKGHFILES) $(PKG_INC_DIR) # We need to install the headers before building the dependencies. If we
scrub/Makefile +1 -0
@@ -129,6 +129,7 @@ xfs_scrubbed: xfs_scrubbed.in $(builddefs) $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e "s|@pkg_data_dir@|$(PKG_DATA_DIR)|g" \ < $< > $@ $(Q)chmod a+x $@
scrub/xfs_scrubbed.in +62 -0
92 lines not on this page · raw message
PATCH 13/21 xfs_scrubbed: enable repairing filesystems
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make it so that our health monitoring daemon can initiate repairs. Because repairs can take a while to run, so we don't actually want to be doing that work in the event thread because the kernel queue can drop events if userspace doesn't respond in time.
Therefore, create a subprocess executor to run the repairs in the background, and do the repairs from there. The subprocess executor is similar in concept to what a libfrog workqueue does, but the workers do not share address space, which eliminates GIL contention.
--- scrub/xfs_scrubbed.in | 366 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 360 insertions(+), 6 deletions(-)
scrub/xfs_scrubbed.in +360 -6
@@ -17,6 +17,7 @@ import errno import ctypes import gc from concurrent.futures import ProcessPoolExecutor +import ctypes.util try: # Not all systems will have this json schema validation libarary,
@@ -37,7 +38,7 @@ try: vcls.check_schema(schema_js) validator = vcls(schema_js) except jsonschema.exceptions.SchemaError as e: - print(f"{args.event_schema}: invalid event data, {e.message}", + print(f"{args.event_schema}: invalid event data: {e.message}", file = sys.stderr) return except Exception as e:
@@ -69,6 +70,9 @@ log = False everything = False debug_fast = False printf_prefix = '' +want_repair = False +libhandle = None +repair_queue = None # placeholder for event queue worker # ioctl encoding stuff _IOC_NRBITS = 8
@@ -112,6 +116,9 @@ def _IOW(type, number, size): def _IOWR(type, number, size): return _IOC(_IOC_READ | _IOC_WRITE, type, number, size) +def _IOWR(type, number, size): + return _IOC(_IOC_READ | _IOC_WRITE, type, number, size) + # xfs health monitoring ioctl stuff XFS_HEALTH_MONITOR_FMT_JSON = 1 XFS_HEALTH_MONITOR_VERBOSE = 1 << 0
@@ -139,9 +146,206 @@ def open_health_monitor(fd, verbose = False): ret = fcntl.ioctl(fd, XFS_IOC_HEALTH_MONITOR, arg) return ret +# libhandle stuff +class xfs_fsid(ctypes.Structure): + _fields_ = [ + ("_val0", ctypes.c_uint), + ("_val1", ctypes.c_uint) + ] + +class xfs_fid(ctypes.Structure): + _fields_ = [ + ("fid_len", ctypes.c_ushort), + ("fid_pad", ctypes.c_ushort), + ("fid_gen", ctypes.c_uint), + ("fid_ino", ctypes.c_ulonglong) + ] + +class xfs_handle(ctypes.Structure): + _fields_ = [ + ("_ha_fsid", xfs_fsid), + ("ha_fid", xfs_fid) + ] +assert ctypes.sizeof(xfs_handle) == 24 + +class fshandle(object): + def __init__(self, fd, mountpoint): + global libhandle + global printf_prefix + + self.handle = xfs_handle() + + if mountpoint is None: + raise Exception('fshandle needs a mountpoint') + + self.mountpoint = mountpoint + + # Create the file and fs handles for the open mountpoint + # so that we can compare them later + buf = ctypes.c_void_p() + buflen = ctypes.c_size_t() + ret = libhandle.fd_to_handle(fd, buf, buflen) + if ret < 0: + errcode = ctypes.get_errno() + raise OSError(errcode, + f'cannot create handle: {os.strerror(errcode)}', + printf_prefix) + if buflen.value != ctypes.sizeof(xfs_handle): + libhandle.free_handle(buf, buflen.value) + raise Exception(f"fshandle expected {ctypes.sizeof(xfs_handle)} bytes, got {buflen.value}.") + + hanp = ctypes.cast(buf, ctypes.POINTER(xfs_handle)) + self.handle = hanp.contents + + def open(self): + '''Reopen a file handle obtained via weak reference.''' + global libhandle + global printf_prefix + + buf = ctypes.c_void_p() + buflen = ctypes.c_size_t() + + fd = os.open(self.mountpoint, os.O_RDONLY) + + # Create the file and fs handles for the open mountpoint + # so that we can compare them later + ret = libhandle.fd_to_handle(fd, buf, buflen) + if ret < 0: + errcode = ctypes.get_errno() + os.close(fd) + raise OSError(errcode, + f'resampling handle: {os.strerror(errcode)}', + printf_prefix) + + hanp = ctypes.cast(buf, ctypes.POINTER(xfs_handle)) + + # Did we get the same handle? + if buflen.value != ctypes.sizeof(xfs_handle) or \ + bytes(hanp.contents) != bytes(self.handle): + os.close(fd) + libhandle.free_handle(buf, buflen) + raise OSError(errno.ESTALE, + os.strerror(errno.ESTALE), + printf_prefix) + + libhandle.free_handle(buf, buflen) + return fd + +def libhandle_load(): + '''Load libhandle and set things up.''' + global libhandle + + soname = ctypes.util.find_library('handle') + if soname is None: + raise OSError(errno.ENOENT, + f'while finding library: {os.strerror(errno.ENOENT)}', + 'libhandle') + + libhandle = ctypes.CDLL(soname, use_errno = True) + libhandle.fd_to_handle.argtypes = ( + ctypes.c_int, + ctypes.POINTER(ctypes.c_void_p), + ctypes.POINTER(ctypes.c_size_t)) + libhandle.handle_to_fshandle.argtypes = ( + ctypes.c_void_p, + ctypes.c_size_t, + ctypes.POINTER(ctypes.c_void_p), + ctypes.POINTER(ctypes.c_size_t)) + libhandle.path_to_fshandle.argtypes = ( + ctypes.c_char_p, + ctypes.c_void_p, + ctypes.c_size_t) + libhandle.free_handle.argtypes = ( + ctypes.c_void_p, + ctypes.c_size_t) + +# metadata scrubbing stuff +XFS_SCRUB_TYPE_PROBE = 0 +XFS_SCRUB_TYPE_SB = 1 +XFS_SCRUB_TYPE_AGF = 2 +XFS_SCRUB_TYPE_AGFL = 3 +XFS_SCRUB_TYPE_AGI = 4 +XFS_SCRUB_TYPE_BNOBT = 5 +XFS_SCRUB_TYPE_CNTBT = 6 +XFS_SCRUB_TYPE_INOBT = 7 +XFS_SCRUB_TYPE_FINOBT = 8 +XFS_SCRUB_TYPE_RMAPBT = 9 +XFS_SCRUB_TYPE_REFCNTBT = 10 +XFS_SCRUB_TYPE_INODE = 11 +XFS_SCRUB_TYPE_BMBTD = 12 +XFS_SCRUB_TYPE_BMBTA = 13 +XFS_SCRUB_TYPE_BMBTC = 14 +XFS_SCRUB_TYPE_DIR = 15 +XFS_SCRUB_TYPE_XATTR = 16 +XFS_SCRUB_TYPE_SYMLINK = 17 +XFS_SCRUB_TYPE_PARENT = 18 +XFS_SCRUB_TYPE_RTBITMAP = 19 +XFS_SCRUB_TYPE_RTSUM = 20 +XFS_SCRUB_TYPE_UQUOTA = 21 +XFS_SCRUB_TYPE_GQUOTA = 22 +XFS_SCRUB_TYPE_PQUOTA = 23 +XFS_SCRUB_TYPE_FSCOUNTERS = 24 +XFS_SCRUB_TYPE_QUOTACHECK = 25 +XFS_SCRUB_TYPE_NLINKS = 26 +XFS_SCRUB_TYPE_HEALTHY = 27 +XFS_SCRUB_TYPE_DIRTREE = 28 +XFS_SCRUB_TYPE_METAPATH = 29 +XFS_SCRUB_TYPE_RGSUPER = 30 +XFS_SCRUB_TYPE_RGBITMAP = 31 +XFS_SCRUB_TYPE_RTRMAPBT = 32 +XFS_SCRUB_TYPE_RTREFCBT = 33 + +XFS_SCRUB_IFLAG_REPAIR = 1 << 0 +XFS_SCRUB_OFLAG_CORRUPT = 1 << 1 +XFS_SCRUB_OFLAG_PREEN = 1 << 2 +XFS_SCRUB_OFLAG_XFAIL = 1 << 3 +XFS_SCRUB_OFLAG_XCORRUPT = 1 << 4 +XFS_SCRUB_OFLAG_INCOMPLETE = 1 << 5 +XFS_SCRUB_OFLAG_WARNING = 1 << 6 +XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED = 1 << 7 +XFS_SCRUB_IFLAG_FORCE_REBUILD = 1 << 8 + +class xfs_scrub_metadata(ctypes.Structure): + _fields_ = [ + ('sm_type', ctypes.c_uint), + ('sm_flags', ctypes.c_uint), + ('sm_ino', ctypes.c_ulonglong), + ('sm_gen', ctypes.c_uint), + ('sm_agno', ctypes.c_uint), + ('_pad', ctypes.c_ulonglong * 5), + ] +assert ctypes.sizeof(xfs_scrub_metadata) == 64 + +XFS_IOC_SCRUB_METADATA = _IOWR(0x58, 60, xfs_scrub_metadata) + +def __xfs_repair_metadata(fd, type, group, ino, gen): + '''Call the kernel to repair some inode metadata.''' + + arg = xfs_scrub_metadata() + arg.sm_type = type + arg.sm_flags = XFS_SCRUB_IFLAG_REPAIR + arg.sm_ino = ino + arg.sm_gen = gen + arg.sm_agno = group + + fcntl.ioctl(fd, XFS_IOC_SCRUB_METADATA, arg) + return arg.sm_flags + +def xfs_repair_fs_metadata(fd, type): + '''Call the kernel to repair some whole-fs metadata.''' + return __xfs_repair_metadata(fd, type, 0, 0, 0) + +def xfs_repair_group_metadata(fd, type, group): + '''Call the kernel to repair some group metadata.''' + return __xfs_repair_metadata(fd, type, group, 0, 0) + +def xfs_repair_inode_metadata(fd, type, ino, gen): + '''Call the kernel to repair some inode metadata.''' + return __xfs_repair_metadata(fd, type, 0, ino, gen) + # main program -def health_reports(mon_fp): +def health_reports(mon_fp, fh): '''Generate python objects describing health events.''' global debug global printf_prefix
@@ -164,7 +368,7 @@ def health_reports(mon_fp): if debug: print(f'new event: {s}') try: - yield json.loads(s) + yield (json.loads(s), fh) except json.decoder.JSONDecodeError as e: print(f"{printf_prefix}: {e} from {s}", file = sys.stderr)
@@ -208,7 +412,7 @@ def report_shutdown(event): print(f"{printf_prefix}: Filesystem shut down due to {', '.join(reasons)}.") sys.stdout.flush() -def handle_event(event): +def handle_event(e): '''Handle an event asynchronously.''' def stringify_timestamp(event): '''Try to convert a timestamp to something human readable.'''
@@ -222,6 +426,17 @@ def handle_event(event): print(f'{printf_prefix}: bad timestamp: {e}', file = sys.stderr) global log + global repair_queue + + # Use a separate subprocess to handle the repairs so that the event + # processing worker does not block on the GIL of the repair workers. + # The downside is that we cannot pass function pointers and all data + # must be pickleable; the upside is that we don't stall processing of + # non-sickness events while repairs are in progress. + if want_repair and not repair_queue: + repair_queue = ProcessPoolExecutor(max_workers = 1) + + event, fh = e # Ignore any event that doesn't pass our schema. This program must # not try to handle a newer kernel that say things that it is not
@@ -236,13 +451,21 @@ def handle_event(event): report_lost(event) elif event['type'] == 'shutdown': report_shutdown(event) + elif want_repair and event['type'] == 'sick': + repair_queue.submit(repair_metadata, event, fh) def monitor(mountpoint, event_queue, **kwargs): '''Monitor the given mountpoint for health events.''' global everything + global log + global printf_prefix + global want_repair + fh = None fd = os.open(mountpoint, os.O_RDONLY) try: + if want_repair: + fh = fshandle(fd, mountpoint) mon_fd = open_health_monitor(fd, verbose = everything) except OSError as e: if e.errno != errno.ENOTTY and e.errno != errno.EOPNOTSUPP:
@@ -251,14 +474,15 @@ def monitor(mountpoint, event_queue, **kwargs): file = sys.stderr) return 1 finally: - # Close the mountpoint if opening the health monitor fails + # Close the mountpoint if opening the health monitor fails; + # the handle object will free its own memory. os.close(fd) # Ownership of mon_fd (and hence responsibility for closing it) is # transferred to the mon_fp object. with os.fdopen(mon_fd) as mon_fp: nr = 0 - for e in health_reports(mon_fp): + for e in health_reports(mon_fp, fh): event_queue.submit(handle_event, e) # Periodically run the garbage collector to constrain
@@ -271,6 +495,125 @@ def monitor(mountpoint, event_queue, **kwargs): return 0 +def __scrub_type(code): + '''Convert a "structures" json list to a scrub type code.''' + SCRUB_TYPES = { + "probe": XFS_SCRUB_TYPE_PROBE, + "sb": XFS_SCRUB_TYPE_SB, + "agf": XFS_SCRUB_TYPE_AGF, + "agfl": XFS_SCRUB_TYPE_AGFL, + "agi": XFS_SCRUB_TYPE_AGI, + "bnobt": XFS_SCRUB_TYPE_BNOBT, + "cntbt": XFS_SCRUB_TYPE_CNTBT, + "inobt": XFS_SCRUB_TYPE_INOBT, + "finobt": XFS_SCRUB_TYPE_FINOBT, + "rmapbt": XFS_SCRUB_TYPE_RMAPBT, + "refcountbt": XFS_SCRUB_TYPE_REFCNTBT, + "inode": XFS_SCRUB_TYPE_INODE, + "bmapbtd": XFS_SCRUB_TYPE_BMBTD, + "bmapbta": XFS_SCRUB_TYPE_BMBTA, + "bmapbtc": XFS_SCRUB_TYPE_BMBTC, + "directory": XFS_SCRUB_TYPE_DIR, + "xattr": XFS_SCRUB_TYPE_XATTR, + "symlink": XFS_SCRUB_TYPE_SYMLINK, + "parent": XFS_SCRUB_TYPE_PARENT, + "rtbitmap": XFS_SCRUB_TYPE_RTBITMAP, + "rtsummary": XFS_SCRUB_TYPE_RTSUM, + "usrquota": XFS_SCRUB_TYPE_UQUOTA, + "grpquota": XFS_SCRUB_TYPE_GQUOTA, + "prjquota": XFS_SCRUB_TYPE_PQUOTA, + "fscounters": XFS_SCRUB_TYPE_FSCOUNTERS, + "quotacheck": XFS_SCRUB_TYPE_QUOTACHECK, + "nlinks": XFS_SCRUB_TYPE_NLINKS, + "healthy": XFS_SCRUB_TYPE_HEALTHY, + "dirtree": XFS_SCRUB_TYPE_DIRTREE, + "metapath": XFS_SCRUB_TYPE_METAPATH, + "rgsuper": XFS_SCRUB_TYPE_RGSUPER, + "rgbitmap": XFS_SCRUB_TYPE_RGBITMAP, + "rtrmapbt": XFS_SCRUB_TYPE_RTRMAPBT, + "rtrefcountbt": XFS_SCRUB_TYPE_RTREFCBT, + } + + if code not in SCRUB_TYPES: + return None + + return SCRUB_TYPES[code] + +def report_outcome(oflags): + if oflags & (XFS_SCRUB_OFLAG_CORRUPT | \ + XFS_SCRUB_OFLAG_CORRUPT | \ + XFS_SCRUB_OFLAG_INCOMPLETE): + return "Repair unsuccessful; offline repair required." + + if oflags & XFS_SCRUB_OFLAG_XFAIL: + return "Seems correct but cross-referencing failed; offline repair recommended." + + if oflags & XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED: + return "No modification needed." + + return "Repairs successful." + +def repair_wholefs(event, fd): + '''React to a fs-domain corruption event by repairing it.''' + for s in event['structures']: + type = __scrub_type(s) + if type is None: + continue + try: + oflags = xfs_repair_fs_metadata(fd, type) + print(f"{printf_prefix}: {s}: {report_outcome(oflags)}") + sys.stdout.flush() + except Exception as e: + print(f"{printf_prefix}: {s}: {e}", file = sys.stderr) + +def repair_group(event, fd, group_type): + '''React to a group-domain corruption event by repairing it.''' + for s in event['structures']: + type = __scrub_type(s) + if type is None: + continue + try: + oflags = xfs_repair_group_metadata(fd, type, event['group']) + print(f"{printf_prefix}: {s}: {report_outcome(oflags)}") + sys.stdout.flush() + except Exception as e: + print(f"{printf_prefix}: {s}: {e}", file = sys.stderr) + +def repair_inode(event, fd): + '''React to a inode-domain corruption event by repairing it.''' + for s in event['structures']: + type = __scrub_type(s) + if type is None: + continue + try: + oflags = xfs_repair_inode_metadata(fd, type, + event['inumber'], event['generation']) + print(f"{printf_prefix}: {s}: {report_outcome(oflags)}") + sys.stdout.flush() + except Exception as e: + print(f"{printf_prefix}: {s}: {e}", file = sys.stderr) + +def repair_metadata(event, fh): + '''Repair a metadata corruption.''' + global debug + global printf_prefix + + if debug: + print(f'repair {event}') + + fd = fh.open() + try: + if event['domain'] in ['fs', 'realtime']: + repair_wholefs(event, fd) + elif event['domain'] in ['perag', 'rtgroup']: + repair_group(event, fd, event['domain']) + elif event['domain'] == 'inode': + repair_inode(event, fd) + else: + raise Exception(f"{printf_prefix}: Unknown metadata domain \"{event['domain']}\".") + finally: + os.close(fd) + def main(): global debug global log
@@ -278,6 +621,7 @@ def main(): global everything global debug_fast global validator_fn + global want_repair parser = argparse.ArgumentParser( \ description = "XFS filesystem health monitoring demon.")
@@ -287,6 +631,8 @@ def main(): action = "store_true") parser.add_argument("--everything", help = "Capture all events.", \ action = "store_true") + parser.add_argument("--repair", help = "Automatically repair corrupt metadata.", \ + action = "store_true") parser.add_argument("-V", help = "Report version and exit.", \ action = "store_true") parser.add_argument('mountpoint', default = None, nargs = '?',
@@ -312,6 +658,12 @@ def main(): if not validator_fn: return 1 + try: + libhandle_load() + except OSError as e: + print(f"libhandle: {e}", file = sys.stderr) + return 1 + if args.debug: debug = True if args.log:
@@ -320,6 +672,8 @@ def main(): everything = True if args.debug_fast: debug_fast = True + if args.repair: + want_repair = True # Use a separate subprocess to handle the events so that the main event # reading process does not block on the GIL of the event handling
PATCH 14/21 xfs_scrubbed: check for fs features needed for effective repairs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Online repair relies heavily on back references such as reverse mappings and directory parent pointers to add redundancy to the filesystem. Check for these two features and whine a bit if they are missing.
--- scrub/xfs_scrubbed.in | 72 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+)
scrub/xfs_scrubbed.in +72 -0
@@ -71,6 +71,8 @@ everything = False debug_fast = False printf_prefix = '' want_repair = False +has_parent = False +has_rmapbt = False libhandle = None repair_queue = None # placeholder for event queue worker
@@ -343,6 +345,57 @@ def xfs_repair_inode_metadata(fd, type, ino, gen): '''Call the kernel to repair some inode metadata.''' return __xfs_repair_metadata(fd, type, 0, ino, gen) +# fsgeometry ioctl +class xfs_fsop_geom(ctypes.Structure): + _fields_ = [ + ("blocksize", ctypes.c_uint), + ("rtextesize", ctypes.c_uint), + ("agblocks", ctypes.c_uint), + ("agcount", ctypes.c_uint), + ("logblocks", ctypes.c_uint), + ("sectsize", ctypes.c_uint), + ("inodesize", ctypes.c_uint), + ("imaxpct", ctypes.c_uint), + ("datablocks", ctypes.c_ulonglong), + ("rtblocks", ctypes.c_ulonglong), + ("rtextents", ctypes.c_ulonglong), + ("logstart", ctypes.c_ulonglong), + ("uuid", ctypes.c_ubyte * 16), + ("sunit", ctypes.c_uint), + ("swidth", ctypes.c_uint), + ("version", ctypes.c_uint), + ("flags", ctypes.c_uint), + ("logsectsize", ctypes.c_uint), + ("rtsectsize", ctypes.c_uint), + ("dirblocksize", ctypes.c_uint), + ("logsunit", ctypes.c_uint), + ("sick", ctypes.c_uint), + ("checked", ctypes.c_uint), + ("rgblocks", ctypes.c_uint), + ("rgcount", ctypes.c_uint), + ("_pad", ctypes.c_ulonglong * 16), + ] +assert ctypes.sizeof(xfs_fsop_geom) == 256 + +XFS_FSOP_GEOM_FLAGS_RMAPBT = 1 << 19 +XFS_FSOP_GEOM_FLAGS_PARENT = 1 << 25 + +XFS_IOC_FSGEOMETRY = _IOR (0x58, 126, xfs_fsop_geom) + +def xfs_has_parent(fd): + '''Does this filesystem have parent pointers?''' + + arg = xfs_fsop_geom() + fcntl.ioctl(fd, XFS_IOC_FSGEOMETRY, arg) + return arg.flags & XFS_FSOP_GEOM_FLAGS_PARENT != 0 + +def xfs_has_rmapbt(fd): + '''Does this filesystem have reverse mapping?''' + + arg = xfs_fsop_geom() + fcntl.ioctl(fd, XFS_IOC_FSGEOMETRY, arg) + return arg.flags & XFS_FSOP_GEOM_FLAGS_RMAPBT != 0 + # main program def health_reports(mon_fp, fh):
@@ -460,9 +513,28 @@ def monitor(mountpoint, event_queue, **kwargs): global log global printf_prefix global want_repair + global has_parent + global has_rmapbt fh = None fd = os.open(mountpoint, os.O_RDONLY) + try: + has_parent = xfs_has_parent(fd) + has_rmapbt = xfs_has_rmapbt(fd) + except Exception as e: + # Don't care if we can't detect parent pointers or rmap + print(f'{printf_prefix}: detecting fs features: {e}', file = sys.stderr) + + # Check for the backref metadata that makes repair effective. + if want_repair: + if not has_rmapbt: + print(f"{mountpoint}: XFS online repair is less effective without rmap btrees.") + if not has_parent: + print(f"{mountpoint}: XFS online repair is less effective without parent pointers.") + + # Flush anything that we may have printed about operational state. + sys.stdout.flush() + try: if want_repair: fh = fshandle(fd, mountpoint)
PATCH 15/21 xfs_scrubbed: use getparents to look up file names
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the kernel tells about something that happened to a file, use the GETPARENTS ioctl to try to look up the path to that file for more ergonomic reporting.
--- scrub/xfs_scrubbed.in | 235 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 230 insertions(+), 5 deletions(-)
scrub/xfs_scrubbed.in +230 -5
@@ -18,6 +18,7 @@ import ctypes import gc from concurrent.futures import ProcessPoolExecutor import ctypes.util +import collections try: # Not all systems will have this json schema validation libarary,
@@ -171,12 +172,18 @@ class xfs_handle(ctypes.Structure): assert ctypes.sizeof(xfs_handle) == 24 class fshandle(object): - def __init__(self, fd, mountpoint): + def __init__(self, fd, mountpoint = None): global libhandle global printf_prefix self.handle = xfs_handle() + if isinstance(fd, fshandle): + # copy an existing fshandle + self.mountpoint = fd.mountpoint + ctypes.pointer(self.handle)[0] = fd.handle + return + if mountpoint is None: raise Exception('fshandle needs a mountpoint')
@@ -233,6 +240,11 @@ class fshandle(object): libhandle.free_handle(buf, buflen) return fd + def subst(self, ino, gen): + '''Substitute the inode and generation components of a handle.''' + self.handle.ha_fid.fid_ino = ino + self.handle.ha_fid.fid_gen = gen + def libhandle_load(): '''Load libhandle and set things up.''' global libhandle
@@ -396,6 +408,170 @@ def xfs_has_rmapbt(fd): fcntl.ioctl(fd, XFS_IOC_FSGEOMETRY, arg) return arg.flags & XFS_FSOP_GEOM_FLAGS_RMAPBT != 0 +# getparents ioctl +class xfs_attrlist_cursor(ctypes.Structure): + _fields_ = [ + ("_opaque0", ctypes.c_uint), + ("_opaque1", ctypes.c_uint), + ("_opaque2", ctypes.c_uint), + ("_opaque3", ctypes.c_uint) + ] + +class xfs_getparents_rec(ctypes.Structure): + _fields_ = [ + ("gpr_parent", xfs_handle), + ("gpr_reclen", ctypes.c_uint), + ("_gpr_reserved", ctypes.c_uint), + ] + +xfs_getparents_tuple = collections.namedtuple('xfs_getparents_tuple', \ + ['gpr_parent', 'gpr_reclen', 'gpr_name']) + +class xfs_getparents_rec_array(object): + def __init__(self, nr_bytes): + self.nr_bytes = nr_bytes + self.bytearray = (ctypes.c_byte * int(nr_bytes))() + + def __slice_to_record(self, bufslice): + '''Compute the number of bytes in a getparents record that contain a null-terminated directory entry name.''' + rec = ctypes.cast(bytes(bufslice), \ + ctypes.POINTER(xfs_getparents_rec)) + fixedlen = ctypes.sizeof(xfs_getparents_rec) + namelen = rec.contents.gpr_reclen - fixedlen + + for i in range(0, namelen): + if bufslice[fixedlen + i] == 0: + namelen = i + break + + if namelen == 0: + return + + return xfs_getparents_tuple( + gpr_parent = rec.contents.gpr_parent, + gpr_reclen = rec.contents.gpr_reclen, + gpr_name = bufslice[fixedlen:fixedlen + namelen]) + + def get_buffer(self): + '''Return a pointer to the bytearray masquerading as an int.''' + return ctypes.addressof(self.bytearray) + + def __iter__(self): + '''Walk the getparents records in this array.''' + off = 0 + nr = 0 + buf = bytes(self.bytearray) + while off < self.nr_bytes: + bufslice = buf[off:] + t = self.__slice_to_record(bufslice) + if t is None: + break + yield t + off += t.gpr_reclen + nr += 1 + +class xfs_getparents(ctypes.Structure): + _fields_ = [ + ("_gp_cursor", xfs_attrlist_cursor), + ("gp_iflags", ctypes.c_ushort), + ("gp_oflags", ctypes.c_ushort), + ("gp_bufsize", ctypes.c_uint), + ("_pad", ctypes.c_ulonglong), + ("gp_buffer", ctypes.c_ulonglong) + ] + + def __init__(self, fd, nr_bytes): + self.fd = fd + self.records = xfs_getparents_rec_array(nr_bytes) + self.gp_buffer = self.records.get_buffer() + self.gp_bufsize = nr_bytes + + def __call_kernel(self): + if self.gp_oflags & XFS_GETPARENTS_OFLAG_DONE: + return False + + ret = fcntl.ioctl(self.fd, XFS_IOC_GETPARENTS, self) + if ret != 0: + return False + + return self.gp_oflags & XFS_GETPARENTS_OFLAG_ROOT == 0 + + def __iter__(self): + ctypes.memset(ctypes.pointer(self._gp_cursor), 0, \ + ctypes.sizeof(xfs_attrlist_cursor)) + + while self.__call_kernel(): + for i in self.records: + yield i + +class xfs_getparents_by_handle(ctypes.Structure): + _fields_ = [ + ("gph_handle", xfs_handle), + ("gph_request", xfs_getparents) + ] + + def __init__(self, fd, fh, nr_bytes): + self.fd = fd + self.records = xfs_getparents_rec_array(nr_bytes) + self.gph_request.gp_buffer = self.records.get_buffer() + self.gph_request.gp_bufsize = nr_bytes + self.gph_handle = fh.handle + + def __call_kernel(self): + if self.gph_request.gp_oflags & XFS_GETPARENTS_OFLAG_DONE: + return False + + ret = fcntl.ioctl(self.fd, XFS_IOC_GETPARENTS_BY_HANDLE, self) + if ret != 0: + return False + + return self.gph_request.gp_oflags & XFS_GETPARENTS_OFLAG_ROOT == 0 + + def __iter__(self): + ctypes.memset(ctypes.pointer(self.gph_request._gp_cursor), 0, \ + ctypes.sizeof(xfs_attrlist_cursor)) + while self.__call_kernel(): + for i in self.records: + yield i + +assert ctypes.sizeof(xfs_getparents) == 40 +assert ctypes.sizeof(xfs_getparents_by_handle) == 64 +assert ctypes.sizeof(xfs_getparents_rec) == 32 + +XFS_GETPARENTS_OFLAG_ROOT = 1 << 0 +XFS_GETPARENTS_OFLAG_DONE = 1 << 1 + +XFS_IOC_GETPARENTS = _IOWR(0x58, 62, xfs_getparents) +XFS_IOC_GETPARENTS_BY_HANDLE = _IOWR(0x58, 63, xfs_getparents_by_handle) + +def fgetparents(fd, fh = None, bufsize = 1024): + '''Return all the parent pointers for a given fd and/or handle.''' + + if fh is not None: + return xfs_getparents_by_handle(fd, fh, bufsize) + return xfs_getparents(fd, bufsize) + +def fgetpath(fd, fh = None, mountpoint = None): + '''Return a list of path components up to the root dir of the filesystem for a given fd.''' + ret = [] + if fh is None: + nfh = fshandle(fd, mountpoint) + else: + # Don't subst into the caller's handle + nfh = fshandle(fh) + + while True: + added = False + for pptr in fgetparents(fd, nfh): + ret.insert(0, pptr.gpr_name) + nfh.subst(pptr.gpr_parent.ha_fid.fid_ino, \ + pptr.gpr_parent.ha_fid.fid_gen) + added = True + break + if not added: + break + return ret + # main program def health_reports(mon_fp, fh):
@@ -429,11 +605,23 @@ def health_reports(mon_fp, fh): lines = [] buf = mon_fp.readline() +def inode_printf_prefix(event): + '''Compute the logging prefix for this event.''' + global printf_prefix + + if 'path' not in event: + return printf_prefix + + if printf_prefix.endswith(os.sep): + return f"{printf_prefix}{event['path']}" + + return f"{printf_prefix}{os.sep}{event['path']}" + def log_event(event): '''Log a monitoring event to stdout.''' global printf_prefix - print(f"{printf_prefix}: {event}") + print(f"{inode_printf_prefix(event)}: {event}") sys.stdout.flush() def report_lost(event):
@@ -480,6 +668,39 @@ def handle_event(e): global log global repair_queue + global has_parent + + def pathify_event(event, fh): + '''Come up with a directory tree path for a file event.''' + try: + path_fd = fh.open() + except Exception as e: + # Not the end of the world if we get nothing + if e.errno != errno.EOPNOTSUPP and e.errno != errno.ENOTTY: + print(f'{printf_prefix}: opening file handle: {e}', file = sys.stderr) + return + + try: + fh2 = fshandle(fh) + except OSError as e: + if e.errno != errno.EOPNOTSUPP: + print(f'{printf_prefix}: making new file handle: {e}', file = sys.stderr) + os.close(path_fd) + return + except Exception as e: + print(f'{printf_prefix}: making new file handle: {e}', file = sys.stderr) + os.close(path_fd) + return + + try: + fh2.subst(event['inumber'], event['generation']) + components = [x.decode('utf-8') for x in fgetpath(path_fd, fh2)] + event['path'] = os.sep.join(components) + except OSError as e: + if e.errno != errno.EOPNOTSUPP: + print(f'{printf_prefix}: constructing path: {e}', file = sys.stderr) + finally: + os.close(path_fd) # Use a separate subprocess to handle the repairs so that the event # processing worker does not block on the GIL of the repair workers.
@@ -498,6 +719,8 @@ def handle_event(e): return stringify_timestamp(event) + if event['domain'] == 'inode' and has_parent and not debug_fast: + pathify_event(event, fh) if log: log_event(event) if event['type'] == 'lost':
@@ -536,7 +759,7 @@ def monitor(mountpoint, event_queue, **kwargs): sys.stdout.flush() try: - if want_repair: + if want_repair or has_parent: fh = fshandle(fd, mountpoint) mon_fd = open_health_monitor(fd, verbose = everything) except OSError as e:
@@ -653,6 +876,8 @@ def repair_group(event, fd, group_type): def repair_inode(event, fd): '''React to a inode-domain corruption event by repairing it.''' + ipp = inode_printf_prefix(event) + for s in event['structures']: type = __scrub_type(s) if type is None:
@@ -660,10 +885,10 @@ def repair_inode(event, fd): try: oflags = xfs_repair_inode_metadata(fd, type, event['inumber'], event['generation']) - print(f"{printf_prefix}: {s}: {report_outcome(oflags)}") + print(f"{ipp}: {s}: {report_outcome(oflags)}") sys.stdout.flush() except Exception as e: - print(f"{printf_prefix}: {s}: {e}", file = sys.stderr) + print(f"{ipp}: {s}: {e}", file = sys.stderr) def repair_metadata(event, fh): '''Repair a metadata corruption.'''
PATCH 16/21 builddefs: refactor udev directory specification
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Refactor the code that finds the udev rules directory to detect the location of the parent udev directory instead. IOWs, we go from:
UDEV_RULE_DIR=/foo/bar/rules.d
to:
UDEV_DIR=/foo/bar
UDEV_RULE_DIR=/foo/bar/rules.d
This is needed by the next patch, which adds a helper script.
--- configure.ac | 2 +- include/builddefs.in | 3 ++- m4/package_services.m4 | 30 +++++++++++++++--------------- 3 files changed, 18 insertions(+), 17 deletions(-)
configure.ac +1 -1
@@ -175,7 +175,7 @@ if test "$enable_scrub" = "yes"; then fi AC_CONFIG_SYSTEMD_SYSTEM_UNIT_DIR AC_CONFIG_CROND_DIR -AC_CONFIG_UDEV_RULE_DIR +AC_CONFIG_UDEV_DIR AC_HAVE_BLKID_TOPO if test "$enable_ubsan" = "yes" || test "$enable_ubsan" = "probe"; then
include/builddefs.in +2 -1
@@ -112,7 +112,8 @@ SYSTEMD_SYSTEM_UNIT_DIR = @systemd_system_unit_dir@ HAVE_CROND = @have_crond@ CROND_DIR = @crond_dir@ HAVE_UDEV = @have_udev@ -UDEV_RULE_DIR = @udev_rule_dir@ +UDEV_DIR = @udev_dir@ +UDEV_RULE_DIR = @udev_dir@/rules.d HAVE_LIBURCU_ATOMIC64 = @have_liburcu_atomic64@ USE_RADIX_TREE_FOR_INUMS = @use_radix_tree_for_inums@
m4/package_services.m4 +15 -15
@@ -77,33 +77,33 @@ AC_DEFUN([AC_CONFIG_CROND_DIR], ]) # -# Figure out where to put udev rule files +# Figure out where to put udev files # -AC_DEFUN([AC_CONFIG_UDEV_RULE_DIR], +AC_DEFUN([AC_CONFIG_UDEV_DIR], [ AC_REQUIRE([PKG_PROG_PKG_CONFIG]) - AC_ARG_WITH([udev_rule_dir], - [AS_HELP_STRING([--with-udev-rule-dir@<:@=DIR@:>@], - [Install udev rules into DIR.])], + AC_ARG_WITH([udev_dir], + [AS_HELP_STRING([--with-udev-dir@<:@=DIR@:>@], + [Install udev files underneath DIR.])], [], - [with_udev_rule_dir=yes]) - AS_IF([test "x${with_udev_rule_dir}" != "xno"], + [with_udev_dir=yes]) + AS_IF([test "x${with_udev_dir}" != "xno"], [ - AS_IF([test "x${with_udev_rule_dir}" = "xyes"], + AS_IF([test "x${with_udev_dir}" = "xyes"], [ PKG_CHECK_MODULES([udev], [udev], [ - with_udev_rule_dir="$($PKG_CONFIG --variable=udev_dir udev)/rules.d" + with_udev_dir="$($PKG_CONFIG --variable=udev_dir udev)" ], [ - with_udev_rule_dir="" + with_udev_dir="" ]) m4_pattern_allow([^PKG_(MAJOR|MINOR|BUILD|REVISION)$]) ]) - AC_MSG_CHECKING([for udev rule dir]) - udev_rule_dir="${with_udev_rule_dir}" - AS_IF([test -n "${udev_rule_dir}"], + AC_MSG_CHECKING([for udev dir]) + udev_dir="${with_udev_dir}" + AS_IF([test -n "${udev_dir}"], [ - AC_MSG_RESULT(${udev_rule_dir}) + AC_MSG_RESULT(${udev_dir}) have_udev="yes" ], [
@@ -115,5 +115,5 @@ AC_DEFUN([AC_CONFIG_UDEV_RULE_DIR], have_udev="disabled" ]) AC_SUBST(have_udev) - AC_SUBST(udev_rule_dir) + AC_SUBST(udev_dir) ])
PATCH 17/21 xfs_scrubbed: create a background monitoring service
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a systemd service and activate it automatically.
--- scrub/Makefile | 18 +++++++ scrub/xfs_scrubbed.in | 9 +++ scrub/xfs_scrubbed.rules | 7 +++ scrub/[email protected] | 103 ++++++++++++++++++++++++++++++++++++++++ scrub/xfs_scrubbed_start | 17 +++++++ 5 files changed, 153 insertions(+), 1 deletion(-) create mode 100644 scrub/xfs_scrubbed.rules create mode 100644 scrub/[email protected] create mode 100755 scrub/xfs_scrubbed_start
scrub/Makefile +17 -1
@@ -29,8 +29,16 @@ SYSTEMD_SERVICES=\ xfs_scrub_all.service \ xfs_scrub_all_fail.service \ xfs_scrub_all.timer \ - system-xfs_scrub.slice + system-xfs_scrub.slice \ + [email protected] OPTIONAL_TARGETS += $(SYSTEMD_SERVICES) + +ifeq ($(HAVE_UDEV),yes) + XFS_SCRUBBED_UDEV_RULES = xfs_scrubbed.rules + XFS_SCRUBBED_HELPER = xfs_scrubbed_start + INSTALL_SCRUB += install-udev-scrubbed + OPTIONAL_TARGETS += $(XFS_SCRUBBED_HELPER) +endif endif ifeq ($(HAVE_CROND),yes) INSTALL_SCRUB += install-crond
@@ -185,6 +193,14 @@ install-udev: $(UDEV_RULES) $(INSTALL) -m 644 $$i $(UDEV_RULE_DIR)/64-$$i; \ done +install-udev-scrubbed: $(XFS_SCRUBBED_HELPER) + $(INSTALL) -m 755 -d $(UDEV_DIR) + $(INSTALL) -m 755 $(XFS_SCRUBBED_HELPER) $(UDEV_DIR) + $(INSTALL) -m 755 -d $(UDEV_RULE_DIR) + for i in $(XFS_SCRUBBED_UDEV_RULES); do \ + $(INSTALL) -m 644 $$i $(UDEV_RULE_DIR)/64-$$i; \ + done + install-dev: -include .dep
scrub/xfs_scrubbed.in +9 -0
@@ -19,6 +19,7 @@ import gc from concurrent.futures import ProcessPoolExecutor import ctypes.util import collections +import time try: # Not all systems will have this json schema validation libarary,
@@ -994,6 +995,14 @@ def main(): pass args.event_queue.shutdown() + + # See the service mode comments in xfs_scrub.c for why we sleep and + # compress all nonzero exit codes to 1. + if 'SERVICE_MODE' in os.environ: + time.sleep(2) + if ret != 0: + ret = 1 + return ret if __name__ == '__main__':
scrub/xfs_scrubbed.rules new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (c) 2024-2025 Oracle. All rights reserved. +# Author: Darrick J. Wong <[email protected]> +# +# Start autonomous self healing automatically +ACTION=="add", SUBSYSTEM=="xfs", ENV{TYPE}=="mount", RUN+="xfs_scrubbed_start"
scrub/[email protected] new file +103 -0
103 lines not on this page · raw message
scrub/xfs_scrubbed_start new file +17 -0
@@ -0,0 +1,17 @@ +#!/bin/sh + +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# Author: Darrick J. Wong <[email protected]> + +# Start the xfs_scrubbed service when the filesystem is mounted + +command -v systemctl || exit 0 + +grep "^$SOURCE[[:space:]]" /proc/mounts | while read source mntpt therest; do + inst="$(systemd-escape --path "$mntpt")" + systemctl restart --no-block "xfs_scrubbed@$inst" && break +done + +exit 0
PATCH 18/21 xfs_scrubbed: don't start service if kernel support unavailable
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use ExecCondition= in the system service to check if kernel support for the health monitor is available. If not, we don't want to run the service, have it fail, and generate a bunch of silly log messages.
--- scrub/xfs_scrubbed.in | 39 ++++++++++++++++++++++++++++++++++++++- scrub/[email protected] | 1 + 2 files changed, 39 insertions(+), 1 deletion(-)
scrub/xfs_scrubbed.in +38 -1
@@ -791,6 +791,38 @@ def monitor(mountpoint, event_queue, **kwargs): return 0 +def check_monitor(mountpoint): + '''Check if the kernel can send us health events for the given mountpoint.''' + global log + global printf_prefix + global everything + global want_repair + global has_parent + + try: + fd = os.open(mountpoint, os.O_RDONLY) + except OSError as e: + # Can't open mountpoint; monitor not available. + print(f"{mountpoint}: {e}", file = sys.stderr) + return 1 + + try: + mon_fd = open_health_monitor(fd, verbose = everything) + except OSError as e: + # Error opening monitor (or it's simply not there); monitor + # not available. + if e.errno == errno.ENOTTY or e.errno == errno.EOPNOTSUPP: + print(f"{mountpoint}: XFS health monitoring not supported.", + file = sys.stderr) + return 1 + finally: + # Close the mountpoint if opening the health monitor fails; + # the handle object will free its own memory. + os.close(fd) + + # Monitor available; success! + return 0 + def __scrub_type(code): '''Convert a "structures" json list to a scrub type code.''' SCRUB_TYPES = {
@@ -923,6 +955,8 @@ def main(): parser = argparse.ArgumentParser( \ description = "XFS filesystem health monitoring demon.") + parser.add_argument("--check", help = "Check presense of health monitor.", \ + action = "store_true") parser.add_argument("--debug", help = "Enabling debugging messages.", \ action = "store_true") parser.add_argument("--log", help = "Log health events to stdout.", \
@@ -989,7 +1023,10 @@ def main(): printf_prefix = args.mountpoint ret = 0 try: - ret = monitor(**vars(args)) + if args.check: + ret = check_monitor(args.mountpoint) + else: + ret = monitor(**vars(args)) except KeyboardInterrupt: # Consider SIGINT to be a clean exit. pass
scrub/[email protected] +1 -0
@@ -18,6 +18,7 @@ RequiresMountsFor=%f [Service] Type=exec Environment=SERVICE_MODE=1 +ExecCondition=@pkg_libexec_dir@/xfs_scrubbed --check %f ExecStart=@pkg_libexec_dir@/xfs_scrubbed --log %f SyslogIdentifier=%N
PATCH 19/21 xfs_scrubbed: use the autofsck fsproperty to select mode
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make the xfs_scrubbed background service query the autofsck filesystem property to figure out which operating mode it should use.
--- scrub/xfs_scrubbed.in | 62 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-)
scrub/xfs_scrubbed.in +61 -1
@@ -573,6 +573,21 @@ def fgetpath(fd, fh = None, mountpoint = None): break return ret +# Filesystem properties + +FSPROP_NAMESPACE = "trusted." +FSPROP_NAME_PREFIX = "xfs:" +FSPROP_AUTOFSCK_NAME = "autofsck" + +def fsprop_attrname(n): + '''Construct the xattr name for a filesystem property.''' + return f"{FSPROP_NAMESPACE}{FSPROP_NAME_PREFIX}{n}" + +def fsprop_getstr(fd, n): + '''Return the value of a filesystem property as a string.''' + attrname = fsprop_attrname(n) + return os.getxattr(fd, attrname).decode('utf-8') + # main program def health_reports(mon_fp, fh):
@@ -731,6 +746,31 @@ def handle_event(e): elif want_repair and event['type'] == 'sick': repair_queue.submit(repair_metadata, event, fh) +def want_repair_from_autofsck(fd): + '''Determine want_repair from the autofsck filesystem property.''' + global has_parent + global has_rmapbt + + try: + advice = fsprop_getstr(fd, FSPROP_AUTOFSCK_NAME) + if advice == "repair": + return True + if advice == "check" or advice == "optimize": + return False + if advice == "none": + return None + except: + # Any OS error (including ENODATA) or string parsing error is + # treated the same as an unrecognized value. + pass + + # For an unrecognized value, log but do not fix runtime corruption if + # backref metadata are enabled. If no backref metadata are available, + # the fs is too old so don't run at all. + if has_rmapbt or has_parent: + return False + return None + def monitor(mountpoint, event_queue, **kwargs): '''Monitor the given mountpoint for health events.''' global everything
@@ -749,6 +789,20 @@ def monitor(mountpoint, event_queue, **kwargs): # Don't care if we can't detect parent pointers or rmap print(f'{printf_prefix}: detecting fs features: {e}', file = sys.stderr) + # Does the sysadmin have any advice for us about whether or not to + # background scrub? + if want_repair is None: + want_repair = want_repair_from_autofsck(fd) + if want_repair is None: + print(f"{mountpoint}: Disabling daemon per autofsck directive.") + os.close(fd) + return 0 + elif want_repair: + print(f"{mountpoint}: Automatically repairing per autofsck directive.") + else: + print(f"{mountpoint}: Only logging errors per autofsck directive.") + + # Check for the backref metadata that makes repair effective. if want_repair: if not has_rmapbt:
@@ -963,7 +1017,11 @@ def main(): action = "store_true") parser.add_argument("--everything", help = "Capture all events.", \ action = "store_true") - parser.add_argument("--repair", help = "Automatically repair corrupt metadata.", \ + action_group = parser.add_mutually_exclusive_group() + action_group.add_argument("--repair", \ + help = "Automatically repair corrupt metadata.", \ + action = "store_true") + action_group.add_argument("--autofsck", help = argparse.SUPPRESS, \ action = "store_true") parser.add_argument("-V", help = "Report version and exit.", \ action = "store_true")
@@ -1004,6 +1062,8 @@ def main(): everything = True if args.debug_fast: debug_fast = True + if args.autofsck: + want_repair = None if args.repair: want_repair = True
PATCH 20/21 xfs_scrub: report media scrub failures to the kernel
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the media scan finds that media have been lost, report this to the kernel so that the healthmon code can pass that along to xfs_scrubbed.
--- scrub/phase6.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
scrub/phase6.c +25 -0
@@ -671,6 +671,29 @@ clean_pool(return ret; } +static void +report_ioerr_to_kernel ( +struct scrub_ctx *ctx, +struct disk *disk, +uint64_t start, +uint64_t length) +{ +struct xfs_media_error me = { + .daddr = start, + .bbcount = length, + }; +dev_t dev =disk_to_dev (ctx, disk); + +if (dev == ctx->fsinfo.fs_datadev) + me.flags |=XFS_MEDIA_ERROR_DATADEV ; +else if (dev == ctx->fsinfo.fs_rtdev) + me.flags |=XFS_MEDIA_ERROR_RTDEV ; +else if (dev == ctx->fsinfo.fs_logdev) + me.flags |=XFS_MEDIA_ERROR_LOGDEV ; + +ioctl (ctx->mnt.fd,XFS_IOC_MEDIA_ERROR , &me); +} +/* Remember a media error for later. */ static void remember_ioerr(
@@ -695,6 +718,8 @@ remember_ioerr(return ; } +report_ioerr_to_kernel (ctx, disk, start, length); + tree =bitmap_for_disk (ctx, disk, vs);if (!tree) {str_liberror (ctx,ENOENT ,_ ("finding bad block bitmap" ));
PATCH 21/21 debian: enable xfs_scrubbed on the root filesystem by default
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we're finished building autonomous repair, enable the service on the root filesystem by default. The root filesystem is mounted by the initrd prior to starting systemd, which is why the udev rule cannot autostart the service for the root filesystem.
dh_installsystemd won't activate a template service (aka one with an at-sign in the name) even if it provides a DefaultInstance directive to make that possible. Use a fugly shim for this.
--- debian/control | 2 +- debian/postinst | 8 ++++++++ debian/prerm | 13 +++++++++++++ scrub/[email protected] | 2 +- 4 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 debian/prerm
debian/control +1 -1
@@ -10,7 +10,7 @@ Homepage: https://xfs.wiki.kernel.org/ Package: xfsprogs Depends: ${shlibs:Depends}, ${misc:Depends}, python3-dbus, python3:any Provides: fsck-backend -Suggests: xfsdump, acl, attr, quota +Suggests: xfsdump, acl, attr, quota, python3-jsonschema Breaks: xfsdump (<< 3.0.0) Replaces: xfsdump (<< 3.0.0) Architecture: linux-any
debian/postinst +8 -0
@@ -24,5 +24,13 @@ case "${1}" in esac #DEBHELPER# +# +# dh_installsystemd doesn't handle template services even if we supply a +# default instance, so we'll install it here. +if [ -z "${DPKG_ROOT:-}" ] && [ -d /run/systemd/system ] ; then + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + /bin/systemctl enable [email protected] || true + fi +fi exit 0
debian/prerm new file +13 -0
@@ -0,0 +1,13 @@ +#!/bin/sh + +set -e + +# dh_installsystemd doesn't handle template services even if we supply a +# default instance, so we'll install it here. +if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = remove ] && [ -d /run/systemd/system ] ; then + /bin/systemctl disable [email protected] || true +fi + +#DEBHELPER# + +exit 0
scrub/[email protected] +1 -1
@@ -19,7 +19,7 @@ RequiresMountsFor=%f Type=exec Environment=SERVICE_MODE=1 ExecCondition=@pkg_libexec_dir@/xfs_scrubbed --check %f -ExecStart=@pkg_libexec_dir@/xfs_scrubbed --log %f +ExecStart=@pkg_libexec_dir@/xfs_scrubbed --autofsck --log %f SyslogIdentifier=%N # Run scrub with minimal CPU and IO priority so that nothing else will starve.
PATCH 4/5 fstests: live health monitoring of filesystems
In reply to Darrick J. WongHi all,
This patchset builds off of Kent Overstreet's thread_with_file code to deliver live information about filesystem health events to userspace. This is done by creating a twf file and hooking internal operations so that the event information can be queued to the twf without stalling the kernel if the twf client program is nonresponsive. This is a private ioctl, so events are expressed using simple json objects so that we can enrich the output later on without having to rev a ton of C structs.
In userspace, we create a new daemon program that will read the json event objects and initiate repairs automatically. This daemon is managed entirely by systemd and will not block unmounting of the filesystem unless repairs are ongoing. It is autostarted via some horrible udev rules.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* misc: convert all $UMOUNT_PROG to a _umount helper
* misc: convert all umount(1) invocations to _umount
* xfs: test health monitoring code
* xfs: test for metadata corruption error reporting via healthmon
* xfs: test io error reporting via healthmon
* xfs: test new xfs_scrubbed daemon
--- common/btrfs | 2 + common/config | 6 +++ common/dmdelay | 4 +- common/dmdust | 4 +- common/dmerror | 4 +- common/dmflakey | 4 +- common/dmhugedisk | 2 + common/dmlogwrites | 4 +- common/dmthin | 4 +- common/overlay | 10 +++--- common/populate | 8 ++--- common/quota | 2 + common/rc | 47 ++++++++++++++++++--------- common/systemd | 9 +++++ common/xfs | 18 ++++++++++ doc/group-names.txt | 1 + tests/btrfs/012 | 2 + tests/btrfs/020 | 2 + tests/btrfs/029 | 2 + tests/btrfs/031 | 2 + tests/btrfs/060 | 2 + tests/btrfs/065 | 2 + tests/btrfs/066 | 2 + tests/btrfs/067 | 2 + tests/btrfs/068 | 2 + tests/btrfs/075 | 2 + tests/btrfs/089 | 2 + tests/btrfs/124 | 2 + tests/btrfs/125 | 2 + tests/btrfs/185 | 4 +- tests/btrfs/197 | 4 +- tests/btrfs/199 | 2 + tests/btrfs/219 | 12 +++---- tests/btrfs/254 | 2 + tests/btrfs/291 | 2 + tests/btrfs/298 | 4 +- tests/ext4/006 | 4 +- tests/ext4/007 | 4 +- tests/ext4/008 | 4 +- tests/ext4/009 | 8 ++--- tests/ext4/010 | 6 ++- tests/ext4/011 | 2 + tests/ext4/012 | 2 + tests/ext4/013 | 6 ++- tests/ext4/014 | 6 ++- tests/ext4/015 | 6 ++- tests/ext4/016 | 6 ++- tests/ext4/017 | 6 ++- tests/ext4/018 | 6 ++- tests/ext4/019 | 6 ++- tests/ext4/032 | 4 +- tests/ext4/033 | 2 + tests/ext4/052 | 4 +- tests/ext4/053 | 32 +++++++++--------- tests/ext4/056 | 2 + tests/generic/042 | 4 +- tests/generic/067 | 6 ++- tests/generic/081 | 2 + tests/generic/085 | 4 +- tests/generic/108 | 2 + tests/generic/171 | 2 + tests/generic/172 | 2 + tests/generic/173 | 2 + tests/generic/174 | 2 + tests/generic/306 | 2 + tests/generic/330 | 2 + tests/generic/332 | 2 + tests/generic/361 | 2 + tests/generic/373 | 2 + tests/generic/374 | 2 + tests/generic/395 | 2 + tests/generic/459 | 2 + tests/generic/563 | 4 +- tests/generic/604 | 2 + tests/generic/631 | 2 + tests/generic/648 | 6 ++- tests/generic/698 | 4 +- tests/generic/699 | 8 ++--- tests/generic/704 | 2 + tests/generic/717 | 2 + tests/generic/730 | 2 + tests/generic/731 | 2 + tests/generic/732 | 4 +- tests/generic/746 | 8 ++--- tests/overlay/003 | 2 + tests/overlay/004 | 2 + tests/overlay/005 | 6 ++- tests/overlay/014 | 4 +- tests/overlay/022 | 2 + tests/overlay/025 | 4 +- tests/overlay/029 | 6 ++- tests/overlay/031 | 8 ++--- tests/overlay/035 | 2 + tests/overlay/036 | 8 ++--- tests/overlay/037 | 6 ++- tests/overlay/040 | 2 + tests/overlay/041 | 2 + tests/overlay/042 | 2 + tests/overlay/043 | 2 + tests/overlay/044 | 2 + tests/overlay/048 | 4 +- tests/overlay/049 | 2 + tests/overlay/050 | 2 + tests/overlay/051 | 4 +- tests/overlay/052 | 2 + tests/overlay/053 | 4 +- tests/overlay/054 | 2 + tests/overlay/055 | 4 +- tests/overlay/056 | 2 + tests/overlay/057 | 4 +- tests/overlay/059 | 2 + tests/overlay/060 | 2 + tests/overlay/062 | 2 + tests/overlay/063 | 2 + tests/overlay/065 | 22 ++++++------- tests/overlay/067 | 2 + tests/overlay/068 | 4 +- tests/overlay/069 | 6 ++- tests/overlay/070 | 6 ++- tests/overlay/071 | 6 ++- tests/overlay/076 | 2 + tests/overlay/077 | 2 + tests/overlay/078 | 2 + tests/overlay/079 | 2 + tests/overlay/080 | 2 + tests/overlay/081 | 14 ++++---- tests/overlay/083 | 2 + tests/overlay/084 | 10 +++--- tests/overlay/085 | 2 + tests/overlay/086 | 8 ++--- tests/xfs/014 | 4 +- tests/xfs/049 | 8 ++--- tests/xfs/073 | 8 ++--- tests/xfs/074 | 4 +- tests/xfs/078 | 4 +- tests/xfs/083 | 6 ++- tests/xfs/085 | 4 +- tests/xfs/086 | 8 ++--- tests/xfs/087 | 6 ++- tests/xfs/088 | 8 ++--- tests/xfs/089 | 8 ++--- tests/xfs/091 | 8 ++--- tests/xfs/093 | 6 ++- tests/xfs/097 | 6 ++- tests/xfs/098 | 4 +- tests/xfs/099 | 6 ++- tests/xfs/100 | 6 ++- tests/xfs/101 | 6 ++- tests/xfs/102 | 6 ++- tests/xfs/105 | 6 ++- tests/xfs/112 | 8 ++--- tests/xfs/113 | 6 ++- tests/xfs/117 | 6 ++- tests/xfs/120 | 6 ++- tests/xfs/123 | 6 ++- tests/xfs/124 | 6 ++- tests/xfs/125 | 6 ++- tests/xfs/126 | 6 ++- tests/xfs/130 | 2 + tests/xfs/148 | 6 ++- tests/xfs/149 | 4 +- tests/xfs/152 | 2 + tests/xfs/169 | 6 ++- tests/xfs/186 | 4 +- tests/xfs/1878 | 80 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1878.out | 10 ++++++ tests/xfs/1879 | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1879.out | 12 +++++++ tests/xfs/1882 | 64 +++++++++++++++++++++++++++++++++++++ tests/xfs/1882.out | 2 + tests/xfs/1883 | 75 +++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1883.out | 2 + tests/xfs/1884 | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1884.out | 2 + tests/xfs/1885 | 53 ++++++++++++++++++++++++++++++ tests/xfs/1885.out | 5 +++ tests/xfs/206 | 2 + tests/xfs/216 | 2 + tests/xfs/217 | 2 + tests/xfs/235 | 6 ++- tests/xfs/236 | 6 ++- tests/xfs/239 | 2 + tests/xfs/241 | 2 + tests/xfs/250 | 4 +- tests/xfs/265 | 6 ++- tests/xfs/289 | 4 +- tests/xfs/310 | 4 +- tests/xfs/507 | 2 + tests/xfs/513 | 4 +- tests/xfs/544 | 2 + tests/xfs/716 | 4 +- tests/xfs/806 | 4 +- 192 files changed, 921 insertions(+), 391 deletions(-) create mode 100755 tests/xfs/1878 create mode 100644 tests/xfs/1878.out create mode 100755 tests/xfs/1879 create mode 100644 tests/xfs/1879.out create mode 100755 tests/xfs/1882 create mode 100644 tests/xfs/1882.out create mode 100755 tests/xfs/1883 create mode 100644 tests/xfs/1883.out create mode 100755 tests/xfs/1884 create mode 100644 tests/xfs/1884.out create mode 100755 tests/xfs/1885 create mode 100644 tests/xfs/1885.out
PATCH 1/6 misc: convert all $UMOUNT_PROG to a _umount helper
From: Darrick J. Wong <[email protected]>
We're going to start collecting ephemeral(ish) filesystem stats in the next patch, so switch all the $UMOUNT_PROG to a helper.
sed -e 's/$UMOUNT_PROG/_umount/g' -i $(git ls-files common tests check)
--- common/btrfs | 2 +- common/dmdelay | 4 ++-- common/dmdust | 4 ++-- common/dmerror | 2 +- common/dmflakey | 4 ++-- common/dmhugedisk | 2 +- common/dmlogwrites | 4 ++-- common/dmthin | 4 ++-- common/overlay | 10 +++++----- common/rc | 33 +++++++++++++++++++-------------- tests/btrfs/020 | 2 +- tests/btrfs/029 | 2 +- tests/btrfs/031 | 2 +- tests/btrfs/060 | 2 +- tests/btrfs/065 | 2 +- tests/btrfs/066 | 2 +- tests/btrfs/067 | 2 +- tests/btrfs/068 | 2 +- tests/btrfs/075 | 2 +- tests/btrfs/089 | 2 +- tests/btrfs/124 | 2 +- tests/btrfs/125 | 2 +- tests/btrfs/185 | 4 ++-- tests/btrfs/197 | 4 ++-- tests/btrfs/219 | 12 ++++++------ tests/btrfs/254 | 2 +- tests/ext4/032 | 4 ++-- tests/ext4/052 | 4 ++-- tests/ext4/053 | 32 ++++++++++++++++---------------- tests/ext4/056 | 2 +- tests/generic/042 | 4 ++-- tests/generic/067 | 6 +++--- tests/generic/081 | 2 +- tests/generic/085 | 4 ++-- tests/generic/108 | 2 +- tests/generic/361 | 2 +- tests/generic/373 | 2 +- tests/generic/374 | 2 +- tests/generic/459 | 2 +- tests/generic/604 | 2 ++ tests/generic/648 | 6 +++--- tests/generic/698 | 4 ++-- tests/generic/699 | 8 ++++---- tests/generic/704 | 2 +- tests/generic/730 | 2 +- tests/generic/731 | 2 +- tests/generic/732 | 4 ++-- tests/generic/746 | 8 ++++---- tests/overlay/003 | 2 +- tests/overlay/004 | 2 +- tests/overlay/005 | 6 +++--- tests/overlay/014 | 4 ++-- tests/overlay/022 | 2 +- tests/overlay/025 | 4 ++-- tests/overlay/029 | 6 +++--- tests/overlay/031 | 8 ++++---- tests/overlay/035 | 2 +- tests/overlay/036 | 8 ++++---- tests/overlay/037 | 6 +++--- tests/overlay/040 | 2 +- tests/overlay/041 | 2 +- tests/overlay/042 | 2 +- tests/overlay/043 | 2 +- tests/overlay/044 | 2 +- tests/overlay/048 | 4 ++-- tests/overlay/049 | 2 +- tests/overlay/050 | 2 +- tests/overlay/051 | 4 ++-- tests/overlay/052 | 2 +- tests/overlay/053 | 4 ++-- tests/overlay/054 | 2 +- tests/overlay/055 | 4 ++-- tests/overlay/056 | 2 +- tests/overlay/057 | 4 ++-- tests/overlay/059 | 2 +- tests/overlay/060 | 2 +- tests/overlay/062 | 2 +- tests/overlay/063 | 2 +- tests/overlay/065 | 22 +++++++++++----------- tests/overlay/067 | 2 +- tests/overlay/068 | 4 ++-- tests/overlay/069 | 6 +++--- tests/overlay/070 | 6 +++--- tests/overlay/071 | 6 +++--- tests/overlay/076 | 2 +- tests/overlay/077 | 2 +- tests/overlay/078 | 2 +- tests/overlay/079 | 2 +- tests/overlay/080 | 2 +- tests/overlay/081 | 14 +++++++------- tests/overlay/083 | 2 +- tests/overlay/084 | 10 +++++----- tests/overlay/085 | 2 +- tests/overlay/086 | 8 ++++---- tests/xfs/078 | 4 ++-- tests/xfs/148 | 6 +++--- tests/xfs/149 | 4 ++-- tests/xfs/186 | 4 ++-- tests/xfs/289 | 4 ++-- tests/xfs/507 | 2 +- tests/xfs/513 | 4 ++-- tests/xfs/544 | 2 +- tests/xfs/806 | 4 ++-- 103 files changed, 226 insertions(+), 219 deletions(-)
common/btrfs +1 -1
@@ -352,7 +352,7 @@ _btrfs_stress_subvolume() while [ ! -e $stop_file ]; do $BTRFS_UTIL_PROG subvolume create $btrfs_mnt/$subvol_name _mount -o subvol=$subvol_name $btrfs_dev $subvol_mnt - $UMOUNT_PROG $subvol_mnt + _umount $subvol_mnt $BTRFS_UTIL_PROG subvolume delete $btrfs_mnt/$subvol_name done }
common/dmdelay +2 -2
@@ -26,7 +26,7 @@ _mount_delay() _unmount_delay() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } _cleanup_delay()
@@ -34,7 +34,7 @@ _cleanup_delay() # If dmsetup load fails then we need to make sure to do resume here # otherwise the umount will hang $DMSETUP_PROG resume delay-test > /dev/null 2>&1 - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _dmsetup_remove delay-test }
common/dmdust +2 -2
@@ -22,7 +22,7 @@ _mount_dust() _unmount_dust() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } _cleanup_dust()
@@ -30,6 +30,6 @@ _cleanup_dust() # If dmsetup load fails then we need to make sure to do resume here # otherwise the umount will hang $DMSETUP_PROG resume dust-test > /dev/null 2>&1 - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _dmsetup_remove dust-test }
common/dmerror +1 -1
@@ -106,7 +106,7 @@ _dmerror_cleanup() test -n "$NON_ERROR_RTDEV" && $DMSETUP_PROG resume error-rttest &>/dev/null $DMSETUP_PROG resume error-test > /dev/null 2>&1 - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 test -n "$NON_ERROR_LOGDEV" && _dmsetup_remove error-logtest test -n "$NON_ERROR_RTDEV" && _dmsetup_remove error-rttest
common/dmflakey +2 -2
@@ -67,7 +67,7 @@ _mount_flakey() _unmount_flakey() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } _cleanup_flakey()
@@ -78,7 +78,7 @@ _cleanup_flakey() test -n "$NON_FLAKEY_RTDEV" && $DMSETUP_PROG resume flakey-rttest &> /dev/null $DMSETUP_PROG resume flakey-test > /dev/null 2>&1 - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _dmsetup_remove flakey-test test -n "$NON_FLAKEY_LOGDEV" && _dmsetup_remove flakey-logtest
common/dmhugedisk +1 -1
@@ -39,7 +39,7 @@ _dmhugedisk_init() _dmhugedisk_cleanup() { - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _dmsetup_remove huge-test _dmsetup_remove huge-test-zero }
common/dmlogwrites +2 -2
@@ -145,7 +145,7 @@ _log_writes_mount() _log_writes_unmount() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # _log_writes_replay_log <mark>
@@ -177,7 +177,7 @@ _log_writes_remove() _log_writes_cleanup() { - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _log_writes_remove }
common/dmthin +2 -2
@@ -23,7 +23,7 @@ DMTHIN_VOL_DEV="/dev/mapper/$DMTHIN_VOL_NAME" _dmthin_cleanup() { - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _dmsetup_remove $DMTHIN_VOL_NAME _dmsetup_remove $DMTHIN_POOL_NAME _dmsetup_remove $DMTHIN_META_NAME
@@ -32,7 +32,7 @@ _dmthin_cleanup() _dmthin_check_fs() { - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 _check_scratch_fs $DMTHIN_VOL_DEV }
common/overlay +5 -5
@@ -142,18 +142,18 @@ _overlay_base_unmount() [ -n "$dev" -a -n "$mnt" ] || return 0 - $UMOUNT_PROG $mnt + _umount $mnt } _overlay_test_unmount() { - $UMOUNT_PROG $TEST_DIR + _umount $TEST_DIR _overlay_base_unmount "$OVL_BASE_TEST_DEV" "$OVL_BASE_TEST_DIR" } _overlay_scratch_unmount() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_base_unmount "$OVL_BASE_SCRATCH_DEV" "$OVL_BASE_SCRATCH_MNT" }
@@ -342,7 +342,7 @@ _overlay_check_scratch_dirs() # Need to umount overlay for scratch dir check local ovl_mounted=`_is_dir_mountpoint $SCRATCH_MNT` - [ -z "$ovl_mounted" ] || $UMOUNT_PROG $SCRATCH_MNT + [ -z "$ovl_mounted" ] || _umount $SCRATCH_MNT # Check dirs with extra overlay options _overlay_check_dirs $lowerdir $upperdir $workdir $*
@@ -387,7 +387,7 @@ _overlay_check_fs() else # Check and umount overlay for dir check ovl_mounted=`_is_dir_mountpoint $ovl_mnt` - [ -z "$ovl_mounted" ] || $UMOUNT_PROG $ovl_mnt + [ -z "$ovl_mounted" ] || _umount $ovl_mnt fi _overlay_check_dirs $base_mnt/$OVL_LOWER $base_mnt/$OVL_UPPER \
common/rc +19 -14
@@ -233,6 +233,11 @@ _mount() return $ret } +_umount() +{ + $UMOUNT_PROG $* +} + # Call _mount to do mount operation but also save mountpoint to # MOUNTED_POINT_STACK. Note that the mount point must be the last parameter _get_mount()
@@ -266,7 +271,7 @@ _put_mount() local last_mnt=`echo $MOUNTED_POINT_STACK | awk '{print $1}'` if [ -n "$last_mnt" ]; then - $UMOUNT_PROG $last_mnt + _umount $last_mnt fi MOUNTED_POINT_STACK=`echo $MOUNTED_POINT_STACK | cut -d\ -f2-` }
@@ -275,7 +280,7 @@ _put_mount() _clear_mount_stack() { if [ -n "$MOUNTED_POINT_STACK" ]; then - $UMOUNT_PROG $MOUNTED_POINT_STACK + _umount $MOUNTED_POINT_STACK fi MOUNTED_POINT_STACK="" }
@@ -420,20 +425,20 @@ _scratch_unmount() _overlay_scratch_unmount ;; btrfs) - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT ;; tmpfs) $UMOUNT_PROG $SCRATCH_MNT ;; *) - $UMOUNT_PROG $SCRATCH_DEV + _umount $SCRATCH_DEV ;; esac } _scratch_umount_idmapped() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } _scratch_remount()
@@ -457,7 +462,7 @@ _scratch_cycle_mount() ;; overlay) if [ "$OVL_BASE_FSTYP" = tmpfs ]; then - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT unmounted=true fi ;;
@@ -505,9 +510,9 @@ _move_mount() # Replace $mnt with $tmp. Use a temporary bind-mount because # mount --move will fail with certain mount propagation layouts. - $UMOUNT_PROG $mnt || _fail "Failed to unmount $mnt" + _umount $mnt || _fail "Failed to unmount $mnt" _mount --bind $tmp $mnt || _fail "Failed to bind-mount $tmp to $mnt" - $UMOUNT_PROG $tmp || _fail "Failed to unmount $tmp" + _umount $tmp || _fail "Failed to unmount $tmp" rmdir $tmp }
@@ -573,7 +578,7 @@ _test_unmount() if [ "$FSTYP" == "overlay" ]; then _overlay_test_unmount else - $UMOUNT_PROG $TEST_DEV + _umount $TEST_DEV fi }
@@ -587,7 +592,7 @@ _test_cycle_mount() ;; overlay) if [ "$OVL_BASE_FSTYP" = tmpfs ]; then - $UMOUNT_PROG $TEST_DIR + _umount $TEST_DIR unmounted=true fi ;;
@@ -1375,7 +1380,7 @@ _repair_scratch_fs() # Fall through to repair base fs dev=$OVL_BASE_SCRATCH_DEV fstyp=$OVL_BASE_FSTYP - $UMOUNT_PROG $OVL_BASE_SCRATCH_MNT + _umount $OVL_BASE_SCRATCH_MNT fi # Let's hope fsck -y suffices... fsck -t $fstyp -y $dev 2>&1
@@ -2189,7 +2194,7 @@ _require_logdev() _notrun "This test requires USE_EXTERNAL to be enabled" # ensure its not mounted - $UMOUNT_PROG $SCRATCH_LOGDEV 2>/dev/null + _umount $SCRATCH_LOGDEV 2>/dev/null } # This test requires that an external log device is not in use
@@ -3281,7 +3286,7 @@ _umount_or_remount_ro() local mountpoint=`_is_dev_mounted $device` if [ $USE_REMOUNT -eq 0 ]; then - $UMOUNT_PROG $device + _umount $device else _remount $device ro fi
@@ -3799,7 +3804,7 @@ _require_scratch_dev_pool() _notrun "$i is part of TEST_DEV, this test requires unique disks" fi if _mount | grep -q $i; then - if ! $UMOUNT_PROG $i; then + if ! _umount $i; then echo "failed to unmount $i - aborting" exit 1 fi
tests/btrfs/020 +1 -1
@@ -17,7 +17,7 @@ _cleanup() { cd / rm -f $tmp.* - $UMOUNT_PROG $loop_mnt + _umount $loop_mnt _destroy_loop_device $loop_dev1 losetup -d $loop_dev2 >/dev/null 2>&1 _destroy_loop_device $loop_dev3
tests/btrfs/029 +1 -1
@@ -74,7 +74,7 @@ cp --reflink=always $orig_file $copy_file >> $seqres.full 2>&1 || echo "cp refli md5sum $orig_file | _filter_testdir_and_scratch md5sum $copy_file | _filter_testdir_and_scratch -$UMOUNT_PROG $reflink_test_dir +_umount $reflink_test_dir # success, all done status=0
tests/btrfs/031 +1 -1
@@ -99,7 +99,7 @@ mv $testdir2/file* $subvol2/ echo "Verify the file contents:" _checksum_files -$UMOUNT_PROG $cross_mount_test_dir +_umount $cross_mount_test_dir # success, all done status=0
tests/btrfs/060 +1 -1
@@ -82,7 +82,7 @@ run_test() fi # in case the subvolume is still mounted - $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _umount $subvol_mnt >/dev/null 2>&1 _scratch_unmount # we called _require_scratch_nocheck instead of _require_scratch # do check after test for each profile config
tests/btrfs/065 +1 -1
@@ -90,7 +90,7 @@ run_test() fi # in case the subvolume is still mounted - $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _umount $subvol_mnt >/dev/null 2>&1 _scratch_unmount # we called _require_scratch_nocheck instead of _require_scratch # do check after test for each profile config
tests/btrfs/066 +1 -1
@@ -82,7 +82,7 @@ run_test() fi # in case the subvolume is still mounted - $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _umount $subvol_mnt >/dev/null 2>&1 _scratch_unmount # we called _require_scratch_nocheck instead of _require_scratch # do check after test for each profile config
tests/btrfs/067 +1 -1
@@ -83,7 +83,7 @@ run_test() fi # in case the subvolume is still mounted - $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _umount $subvol_mnt >/dev/null 2>&1 _scratch_unmount # we called _require_scratch_nocheck instead of _require_scratch # do check after test for each profile config
tests/btrfs/068 +1 -1
@@ -83,7 +83,7 @@ run_test() fi # in case the subvolume is still mounted - $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _umount $subvol_mnt >/dev/null 2>&1 _scratch_unmount # we called _require_scratch_nocheck instead of _require_scratch # do check after test for each profile config
tests/btrfs/075 +1 -1
@@ -15,7 +15,7 @@ _cleanup() { cd / rm -f $tmp.* - $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _umount $subvol_mnt >/dev/null 2>&1 } . ./common/filter
tests/btrfs/089 +1 -1
@@ -35,7 +35,7 @@ mount --bind "$SCRATCH_MNT/testvol/testdir" "$SCRATCH_MNT/testvol/mnt" $BTRFS_UTIL_PROG subvolume delete "$SCRATCH_MNT/testvol" >>$seqres.full 2>&1 # Unmount the bind mount, which should still be alive. -$UMOUNT_PROG "$SCRATCH_MNT/testvol/mnt" +_umount "$SCRATCH_MNT/testvol/mnt" echo "Silence is golden" status=0
tests/btrfs/124 +1 -1
@@ -132,7 +132,7 @@ if [ "$checkpoint1" != "$checkpoint3" ]; then echo "Inital sum does not match with data on dev2 written by balance" fi -$UMOUNT_PROG $dev2 +_umount $dev2 _scratch_dev_pool_put _btrfs_rescan_devices _test_mount
tests/btrfs/125 +1 -1
@@ -144,7 +144,7 @@ if [ "$checkpoint1" != "$checkpoint3" ]; then echo "Inital sum does not match with data on dev2 written by balance" fi -$UMOUNT_PROG $dev2 +_umount $dev2 _scratch_dev_pool_put _btrfs_rescan_devices _test_mount
tests/btrfs/185 +2 -2
@@ -15,7 +15,7 @@ mnt=$TEST_DIR/$seq.mnt # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $mnt > /dev/null 2>&1 + _umount $mnt > /dev/null 2>&1 rm -rf $mnt > /dev/null 2>&1 cd / rm -f $tmp.*
@@ -62,7 +62,7 @@ $BTRFS_UTIL_PROG device scan $device_1 >> $seqres.full 2>&1 _fail "if it fails here, then it means subvolume mount at boot may fail "\ "in some configs." -$UMOUNT_PROG $mnt > /dev/null 2>&1 +_umount $mnt > /dev/null 2>&1 _scratch_dev_pool_put # success, all done
tests/btrfs/197 +2 -2
@@ -15,7 +15,7 @@ _begin_fstest auto quick volume # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $TEST_DIR/$seq.mnt >/dev/null 2>&1 + _umount $TEST_DIR/$seq.mnt >/dev/null 2>&1 rm -rf $TEST_DIR/$seq.mnt cd / rm -f $tmp.*
@@ -67,7 +67,7 @@ workout() grep -q "${SCRATCH_DEV_NAME[1]}" $tmp.output && _fail "found stale device" $BTRFS_UTIL_PROG device remove "${SCRATCH_DEV_NAME[1]}" "$TEST_DIR/$seq.mnt" - $UMOUNT_PROG $TEST_DIR/$seq.mnt + _umount $TEST_DIR/$seq.mnt _scratch_unmount _spare_dev_put _scratch_dev_pool_put
tests/btrfs/219 +6 -6
@@ -21,8 +21,8 @@ _cleanup() rm -f $tmp.* # The variables are set before the test case can fail. - $UMOUNT_PROG ${loop_mnt1} &> /dev/null - $UMOUNT_PROG ${loop_mnt2} &> /dev/null + _umount ${loop_mnt1} &> /dev/null + _umount ${loop_mnt2} &> /dev/null rm -rf $loop_mnt1 rm -rf $loop_mnt2
@@ -66,7 +66,7 @@ loop_dev2=`_create_loop_device $fs_img2` # Normal single device case, should pass just fine _mount $loop_dev1 $loop_mnt1 > /dev/null 2>&1 || \ _fail "Couldn't do initial mount" -$UMOUNT_PROG $loop_mnt1 +_umount $loop_mnt1 _btrfs_forget_or_module_reload
@@ -75,15 +75,15 @@ _btrfs_forget_or_module_reload # measure. _mount $loop_dev1 $loop_mnt1 > /dev/null 2>&1 || \ _fail "Failed to mount the second time" -$UMOUNT_PROG $loop_mnt1 +_umount $loop_mnt1 _mount $loop_dev2 $loop_mnt2 > /dev/null 2>&1 || \ _fail "We couldn't mount the old generation" -$UMOUNT_PROG $loop_mnt2 +_umount $loop_mnt2 _mount $loop_dev1 $loop_mnt1 > /dev/null 2>&1 || \ _fail "Failed to mount the second time" -$UMOUNT_PROG $loop_mnt1 +_umount $loop_mnt1 # Now try mount them at the same time, if kernel does not support # temp-fsid feature then mount will fail.
tests/btrfs/254 +1 -1
@@ -96,7 +96,7 @@ test_add_device() $BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT | \ _filter_btrfs_filesystem_show - $UMOUNT_PROG $seq_mnt + _umount $seq_mnt _scratch_unmount cleanup_dmdev }
tests/ext4/032 +2 -2
@@ -63,7 +63,7 @@ ext4_online_resize() fi cat $tmp.resize2fs >> $seqres.full echo "+++ umount fs" | tee -a $seqres.full - $UMOUNT_PROG ${IMG_MNT} + _umount ${IMG_MNT} echo "+++ check fs" | tee -a $seqres.full _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
@@ -77,7 +77,7 @@ _cleanup() cd / [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1 rm -f $tmp.* - $UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1 + _umount ${IMG_MNT} > /dev/null 2>&1 rm -f ${IMG_FILE} > /dev/null 2>&1 }
tests/ext4/052 +2 -2
@@ -18,7 +18,7 @@ _cleanup() cd / rm -r -f $tmp.* if [ ! -z "$loop_mnt" ]; then - $UMOUNT_PROG $loop_mnt + _umount $loop_mnt rm -rf $loop_mnt fi [ ! -z "$fs_img" ] && rm -rf $fs_img
@@ -63,7 +63,7 @@ then status=1 fi -$UMOUNT_PROG $loop_mnt || _fail "umount failed" +_umount $loop_mnt || _fail "umount failed" loop_mnt= $E2FSCK_PROG -fn $fs_img >> $seqres.full 2>&1 || _fail "file system corrupted"
tests/ext4/053 +16 -16
@@ -20,7 +20,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _cleanup() { cd / - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 if [ -n "$LOOP_LOGDEV" ];then _destroy_loop_device $LOOP_LOGDEV 2>/dev/null fi
@@ -237,7 +237,7 @@ not_mnt() { if simple_mount -o $1 $SCRATCH_DEV $SCRATCH_MNT; then print_log "(mount unexpectedly succeeded)" fail - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT return fi ok
@@ -248,7 +248,7 @@ not_mnt() { return fi not_remount $1 - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } mnt_only() {
@@ -270,7 +270,7 @@ mnt() { fi mnt_only $* - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null [ "$t2fs" -eq 0 ] && return
@@ -289,7 +289,7 @@ mnt() { -e 's/data=writeback/journal_data_writeback/') $TUNE2FS_PROG -o $op_set $SCRATCH_DEV > /dev/null 2>&1 mnt_only "defaults" $check - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null if [ "$op_set" = ^* ]; then op_set=${op_set#^} else
@@ -309,12 +309,12 @@ remount() { do_mnt remount,$2 $3 if [ $? -ne 0 ]; then fail - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null return else ok fi - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null # Now just specify mnt print_log "mounting $fstype \"$1\" "
@@ -328,7 +328,7 @@ remount() { ok fi - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null } # Test that the filesystem cannot be remounted with option(s) $1 (meaning that
@@ -364,7 +364,7 @@ mnt_then_not_remount() { return fi not_remount $2 - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT }
@@ -400,8 +400,8 @@ LOGDEV_DEVNUM=`echo "${majmin%:*}*2^8 + ${majmin#*:}" | bc` fstype= for fstype in ext2 ext3 ext4; do - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null - $UMOUNT_PROG $SCRATCH_DEV 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_DEV 2> /dev/null do_mkfs $SCRATCH_DEV ${SIZE}k
@@ -418,7 +418,7 @@ for fstype in ext2 ext3 ext4; do continue fi - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null not_mnt failme mnt
@@ -552,7 +552,7 @@ for fstype in ext2 ext3 ext4; do # dax mount options simple_mount -o dax=always $SCRATCH_DEV $SCRATCH_MNT > /dev/null 2>&1 if [ $? -eq 0 ]; then - $UMOUNT_PROG $SCRATCH_MNT 2> /dev/null + _umount $SCRATCH_MNT 2> /dev/null mnt dax mnt dax=always mnt dax=never
@@ -633,7 +633,7 @@ for fstype in ext2 ext3 ext4; do not_remount jqfmt=vfsv1 not_remount noquota mnt_only remount,usrquota,grpquota ^usrquota,^grpquota - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 # test clearing/changing quota when enabled do_mkfs -E quotatype=^prjquota $SCRATCH_DEV ${SIZE}k
@@ -654,7 +654,7 @@ for fstype in ext2 ext3 ext4; do mnt_only remount,usrquota,grpquota usrquota,grpquota quotaoff -f $SCRATCH_MNT >> $seqres.full 2>&1 mnt_only remount,noquota ^usrquota,^grpquota,quota - $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 + _umount $SCRATCH_MNT > /dev/null 2>&1 # Quota feature echo "== Testing quota feature " >> $seqres.full
@@ -696,7 +696,7 @@ for fstype in ext2 ext3 ext4; do done #for fstype in ext2 ext3 ext4; do -$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1 +_umount $SCRATCH_MNT > /dev/null 2>&1 echo "$ERR errors encountered" >> $seqres.full status=$ERR
tests/ext4/056 +1 -1
@@ -70,7 +70,7 @@ do_resize() # delay sleep 0.2 _scratch_unmount >> $seqres.full 2>&1 \ - || _fail "$UMOUNT_PROG failed. Exiting" + || _fail "_umount failed. Exiting" } run_test()
tests/generic/042 +2 -2
@@ -44,7 +44,7 @@ _crashtest() _filter_xfs_io $here/src/godown -f $mnt - $UMOUNT_PROG $mnt + _umount $mnt _mount $img $mnt # We should /never/ see 0xCD in the file, because we wrote that pattern
@@ -54,7 +54,7 @@ _crashtest() _hexdump $file fi - $UMOUNT_PROG $mnt + _umount $mnt } # Modify as appropriate.
tests/generic/067 +3 -3
@@ -66,7 +66,7 @@ umount_symlink_device() rm -f $symlink echo "# umount symlink to device, which is not mounted" >>$seqres.full ln -s $SCRATCH_DEV $symlink - $UMOUNT_PROG $symlink >>$seqres.full 2>&1 + _umount $symlink >>$seqres.full 2>&1 } # umount a path name that is 256 bytes long, this should fail gracefully,
@@ -78,7 +78,7 @@ umount_toolong_name() _scratch_mount 2>&1 | tee -a $seqres.full echo "# umount a too-long name" >>$seqres.full - $UMOUNT_PROG $longname >>$seqres.full 2>&1 + _umount $longname >>$seqres.full 2>&1 _scratch_unmount 2>&1 | tee -a $seqres.full }
@@ -93,7 +93,7 @@ lazy_umount_symlink() rm -f $symlink ln -s $SCRATCH_MNT/testdir $symlink - $UMOUNT_PROG -l $symlink >>$seqres.full 2>&1 + _umount -l $symlink >>$seqres.full 2>&1 # _scratch_unmount should not be blocked _scratch_unmount 2>&1 | tee -a $seqres.full }
tests/generic/081 +1 -1
@@ -32,7 +32,7 @@ _cleanup() # other tests to fail. while test -e /dev/mapper/$vgname-$snapname || \ test -e /dev/mapper/$vgname-$lvname; do - $UMOUNT_PROG $mnt >> $seqres.full 2>&1 + _umount $mnt >> $seqres.full 2>&1 $LVM_PROG lvremove -f $vgname/$snapname >>$seqres.full 2>&1 $LVM_PROG lvremove -f $vgname/$lvname >>$seqres.full 2>&1 $LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
tests/generic/085 +2 -2
@@ -27,7 +27,7 @@ cleanup_dmdev() $DMSETUP_PROG resume $lvdev >/dev/null 2>&1 [ -n "$pid" ] && kill -9 $pid 2>/dev/null wait $pid - $UMOUNT_PROG $lvdev >/dev/null 2>&1 + _umount $lvdev >/dev/null 2>&1 _dmsetup_remove $node }
@@ -70,7 +70,7 @@ done & pid=$! for ((i=0; i<100; i++)); do _mount $lvdev $SCRATCH_MNT >/dev/null 2>&1 - $UMOUNT_PROG $lvdev >/dev/null 2>&1 + _umount $lvdev >/dev/null 2>&1 done & pid="$pid $!"
tests/generic/108 +1 -1
@@ -18,7 +18,7 @@ _cleanup() { cd / echo running > /sys/block/`_short_dev $SCSI_DEBUG_DEV`/device/state - $UMOUNT_PROG $SCRATCH_MNT >>$seqres.full 2>&1 + _umount $SCRATCH_MNT >>$seqres.full 2>&1 $LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1 $LVM_PROG pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1 $UDEV_SETTLE_PROG
tests/generic/361 +1 -1
@@ -16,7 +16,7 @@ _begin_fstest auto quick # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $fs_mnt + _umount $fs_mnt _destroy_loop_device $loop_dev cd / rm -f $tmp.*
tests/generic/373 +1 -1
@@ -60,7 +60,7 @@ md5sum $testdir/file | _filter_scratch md5sum $othertestdir/otherfile | filter_otherdir echo "Unmount otherdir" -$UMOUNT_PROG $otherdir +_umount $otherdir rm -rf $otherdir # success, all done
tests/generic/374 +1 -1
@@ -59,7 +59,7 @@ echo "Check output" md5sum $testdir/file $othertestdir/otherfile | filter_md5 echo "Unmount otherdir" -$UMOUNT_PROG $otherdir +_umount $otherdir rm -rf $otherdir # success, all done
tests/generic/459 +1 -1
@@ -28,7 +28,7 @@ _cleanup() xfs_freeze -u $SCRATCH_MNT 2>/dev/null cd / rm -f $tmp.* - $UMOUNT_PROG $SCRATCH_MNT >>$seqres.full 2>&1 + _umount $SCRATCH_MNT >>$seqres.full 2>&1 $LVM_PROG vgremove -ff $vgname >>$seqres.full 2>&1 $LVM_PROG pvremove -ff $SCRATCH_DEV >>$seqres.full 2>&1 $UDEV_SETTLE_PROG
tests/generic/604 +2 -0
@@ -26,6 +26,8 @@ done # mount the base fs. Delay the mount attempt by a small amount in the hope # that the mount() call will try to lock s_umount /after/ umount has already # taken it. +# This is the /one/ place in fstests where we need to call the umount binary +# directly. $UMOUNT_PROG $SCRATCH_MNT & sleep 0.01s ; _scratch_mount wait
tests/generic/648 +3 -3
@@ -20,7 +20,7 @@ _cleanup() $KILLALL_PROG -9 fsstress > /dev/null 2>&1 wait if [ -n "$loopmnt" ]; then - $UMOUNT_PROG $loopmnt 2>/dev/null + _umount $loopmnt 2>/dev/null rm -r -f $loopmnt fi rm -f $tmp.*
@@ -111,7 +111,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do # Mount again to replay log after loading working table, so we have a # consistent fs after test. - $UMOUNT_PROG $loopmnt + _umount $loopmnt is_unmounted=1 # We must unmount dmerror at here, or whole later testing will crash. # So try to umount enough times, before we have no choice.
@@ -137,7 +137,7 @@ done # Make sure the fs image file is ok if [ -f "$loopimg" ]; then if _mount $loopimg $loopmnt -o loop; then - $UMOUNT_PROG $loopmnt &> /dev/null + _umount $loopmnt &> /dev/null else _metadump_dev $DMERROR_DEV $seqres.scratch.final.md echo "final scratch mount failed"
tests/generic/698 +2 -2
@@ -17,8 +17,8 @@ _begin_fstest auto quick perms attr idmapped mount _cleanup() { cd / - $UMOUNT_PROG $SCRATCH_MNT/target-mnt 2>/dev/null - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT/target-mnt 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null rm -r -f $tmp.* }
tests/generic/699 +4 -4
@@ -15,9 +15,9 @@ _begin_fstest auto quick perms attr idmapped mount _cleanup() { cd / - $UMOUNT_PROG $SCRATCH_MNT/target-mnt - $UMOUNT_PROG $SCRATCH_MNT/ovl-merge 2>/dev/null - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT/target-mnt + _umount $SCRATCH_MNT/ovl-merge 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null rm -r -f $tmp.* }
@@ -113,7 +113,7 @@ setup_overlayfs_idmapped_lower_metacopy_on() reset_overlayfs() { - $UMOUNT_PROG $SCRATCH_MNT/ovl-merge 2>/dev/null + _umount $SCRATCH_MNT/ovl-merge 2>/dev/null rm -rf $upper $work $merge }
tests/generic/704 +1 -1
@@ -14,7 +14,7 @@ _cleanup() { cd / rm -r -f $tmp.* - [ -d "$SCSI_DEBUG_MNT" ] && $UMOUNT_PROG $SCSI_DEBUG_MNT 2>/dev/null + [ -d "$SCSI_DEBUG_MNT" ] && _umount $SCSI_DEBUG_MNT 2>/dev/null _put_scsi_debug_dev }
tests/generic/730 +1 -1
@@ -12,7 +12,7 @@ _begin_fstest auto quick _cleanup() { cd / - $UMOUNT_PROG $SCSI_DEBUG_MNT >>$seqres.full 2>&1 + _umount $SCSI_DEBUG_MNT >>$seqres.full 2>&1 _put_scsi_debug_dev rm -f $tmp.* }
tests/generic/731 +1 -1
@@ -13,7 +13,7 @@ _begin_fstest auto quick _cleanup() { cd / - $UMOUNT_PROG $SCSI_DEBUG_MNT >>$seqres.full 2>&1 + _umount $SCSI_DEBUG_MNT >>$seqres.full 2>&1 _put_scsi_debug_dev rm -f $tmp.* }
tests/generic/732 +2 -2
@@ -15,8 +15,8 @@ _begin_fstest auto quick rename # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $testdir1 2>/dev/null - $UMOUNT_PROG $testdir2 2>/dev/null + _umount $testdir1 2>/dev/null + _umount $testdir2 2>/dev/null cd / rm -r -f $tmp.* }
tests/generic/746 +4 -4
@@ -38,7 +38,7 @@ esac # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $loop_dev &> /dev/null + _umount $loop_dev &> /dev/null _destroy_loop_device $loop_dev if [ $status -eq 0 ]; then rm -rf $tmp
@@ -53,7 +53,7 @@ get_holes() # in-core state that will perturb the free space map on umount. Stick # to established convention which requires the filesystem to be # unmounted while we probe the underlying file. - $UMOUNT_PROG $loop_mnt + _umount $loop_mnt # FIEMAP only works on regular files, so call it on the backing file # and not the loop device like everything else
@@ -66,7 +66,7 @@ get_free_sectors() { case $FSTYP in ext4) - $UMOUNT_PROG $loop_mnt + _umount $loop_mnt $DUMPE2FS_PROG $loop_dev 2>&1 | grep " Free blocks" | cut -d ":" -f2- | \ tr ',' '\n' | $SED_PROG 's/^ //' | \ $AWK_PROG -v spb=$sectors_per_block 'BEGIN{FS="-"};
@@ -80,7 +80,7 @@ get_free_sectors() xfs) agsize=`$XFS_INFO_PROG $loop_mnt | $SED_PROG -n 's/.*agsize=\(.*\) blks.*/\1/p'` # Convert free space (agno, block, length) to (start sector, end sector) - $UMOUNT_PROG $loop_mnt + _umount $loop_mnt $XFS_DB_PROG -r -c "freesp -d" $loop_dev | $SED_PROG '/^.*from/,$d'| \ $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \ '{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }'
tests/overlay/003 +1 -1
@@ -56,7 +56,7 @@ rm -rf ${SCRATCH_MNT}/* ls ${SCRATCH_MNT}/ # unmount overlayfs but not base fs -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT echo "Silence is golden" # success, all done
tests/overlay/004 +1 -1
@@ -53,7 +53,7 @@ _user_do "chmod u-X ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1" stat -c %a ${SCRATCH_MNT}/attr_file2 # unmount overlayfs but not base fs -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # check mode bits of the file that has been copied up, and # the file that should not have been copied up.
tests/overlay/005 +3 -3
@@ -75,14 +75,14 @@ $XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \ >>$seqres.full 2>&1 # unmount overlayfs -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # check overlayfs _overlay_check_scratch_dirs $lowerd $upperd $workd # unmount undelying xfs, this tiggers panic if memleak happens -$UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/uppermnt -$UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/lowermnt +_umount ${OVL_BASE_SCRATCH_MNT}/uppermnt +_umount ${OVL_BASE_SCRATCH_MNT}/lowermnt # success, all done echo "Silence is golden"
tests/overlay/014 +2 -2
@@ -46,7 +46,7 @@ _overlay_scratch_mount_dirs $lowerdir1 $lowerdir2 $workdir2 rm -rf $SCRATCH_MNT/testdir mkdir -p $SCRATCH_MNT/testdir/visibledir # unmount overlayfs but not base fs -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # check overlayfs _overlay_check_scratch_dirs $lowerdir1 $lowerdir2 $workdir2
@@ -59,7 +59,7 @@ touch $SCRATCH_MNT/testdir/visiblefile # umount and mount overlay again, buggy kernel treats the copied-up dir as # opaque, visibledir is not seen in merged dir. -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT _overlay_scratch_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir ls $SCRATCH_MNT/testdir
tests/overlay/022 +1 -1
@@ -17,7 +17,7 @@ _begin_fstest auto quick mount nested _cleanup() { cd / - $UMOUNT_PROG $tmp/mnt > /dev/null 2>&1 + _umount $tmp/mnt > /dev/null 2>&1 rm -rf $tmp rm -f $tmp.* }
tests/overlay/025 +2 -2
@@ -19,8 +19,8 @@ _begin_fstest auto quick attr _cleanup() { cd / - $UMOUNT_PROG $tmpfsdir/mnt - $UMOUNT_PROG $tmpfsdir + _umount $tmpfsdir/mnt + _umount $tmpfsdir rm -rf $tmpfsdir rm -f $tmp.* }
tests/overlay/029 +3 -3
@@ -22,7 +22,7 @@ _begin_fstest auto quick nested _cleanup() { cd / - $UMOUNT_PROG $tmp/mnt + _umount $tmp/mnt rm -rf $tmp rm -f $tmp.* }
@@ -56,7 +56,7 @@ _overlay_mount_dirs $SCRATCH_MNT/up $tmp/{upper,work} \ overlay $tmp/mnt # accessing file in the second mount cat $tmp/mnt/foo -$UMOUNT_PROG $tmp/mnt +_umount $tmp/mnt # re-create upper/work to avoid ovl_verify_origin() mount failure # when index is enabled
@@ -66,7 +66,7 @@ mkdir -p $tmp/{upper,work} _overlay_mount_dirs $SCRATCH_MNT/low $tmp/{upper,work} \ overlay $tmp/mnt cat $tmp/mnt/bar -$UMOUNT_PROG $tmp/mnt +_umount $tmp/mnt rm -rf $tmp/{upper,work} mkdir -p $tmp/{upper,work}
tests/overlay/031 +4 -4
@@ -28,7 +28,7 @@ create_whiteout() rm -f $SCRATCH_MNT/testdir/$file - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Import common functions.
@@ -68,7 +68,7 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch # umount overlay again, create a new file with the same name and # mount overlay again. -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT touch $lowerdir1/testdir _overlay_scratch_mount_dirs $lowerdir1 $upperdir $workdir
@@ -77,7 +77,7 @@ _overlay_scratch_mount_dirs $lowerdir1 $upperdir $workdir # it will not clean up the dir and lead to residue. rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # let lower dir have invalid whiteouts, repeat ls and rmdir test again. rm -rf $lowerdir1/testdir
@@ -92,7 +92,7 @@ _overlay_scratch_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir ls $SCRATCH_MNT/testdir rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again. rm -rf $lowerdir1/testdir
tests/overlay/035 +1 -1
@@ -43,7 +43,7 @@ mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir _overlay_scratch_mount_opts -o"lowerdir=$lowerdir2:$lowerdir1" touch $SCRATCH_MNT/foo 2>&1 | _filter_scratch _mount -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Make workdir immutable to prevent workdir re-create on mount $CHATTR_PROG +i $workdir
tests/overlay/036 +4 -4
@@ -34,8 +34,8 @@ _cleanup() cd / rm -f $tmp.* # unmount the two extra mounts in case they did not fail - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null } # Import common functions.
@@ -66,13 +66,13 @@ _overlay_mount_dirs $lowerdir $upperdir $workdir \ # with index=off - expect success _overlay_mount_dirs $lowerdir $upperdir $workdir2 \ overlay0 $SCRATCH_MNT -oindex=off && \ - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT # Try to mount another overlay with the same workdir # with index=off - expect success _overlay_mount_dirs $lowerdir2 $upperdir2 $workdir \ overlay1 $SCRATCH_MNT -oindex=off && \ - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT # Try to mount another overlay with the same upperdir # with index=on - expect EBUSY
tests/overlay/037 +3 -3
@@ -39,17 +39,17 @@ mkdir -p $lowerdir $lowerdir2 $upperdir $upperdir2 $workdir # Mount overlay with lowerdir, upperdir, workdir and index=on # to store the file handles of lowerdir and upperdir in overlay.origin xattr _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -oindex=on -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Try to mount an overlay with the same upperdir and different lowerdir - expect ESTALE _overlay_scratch_mount_dirs $lowerdir2 $upperdir $workdir -oindex=on \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null # Try to mount an overlay with the same workdir and different upperdir - expect ESTALE _overlay_scratch_mount_dirs $lowerdir $upperdir2 $workdir -oindex=on \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null # Mount overlay with original lowerdir, upperdir, workdir and index=on - expect success _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -oindex=on
tests/overlay/040 +1 -1
@@ -48,7 +48,7 @@ _scratch_mount # modify lower origin file. $CHATTR_PROG +i $SCRATCH_MNT/foo > /dev/null 2>&1 -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # touching origin file in lower, should succeed touch $lowerdir/foo
tests/overlay/041 +1 -1
@@ -142,7 +142,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino) [[ $subdir_d == "subdir d" ]] || \ echo "Merged dir: Invalid d_ino reported for subdir" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # check overlayfs _overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o xino=on
tests/overlay/042 +1 -1
@@ -45,7 +45,7 @@ _scratch_mount -o index=off # Copy up lower and create upper hardlink with no index ln $SCRATCH_MNT/0 $SCRATCH_MNT/1 -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Add lower hardlinks while overlay is offline ln $lowerdir/0 $lowerdir/2
tests/overlay/043 +1 -1
@@ -126,7 +126,7 @@ echo 3 > /proc/sys/vm/drop_caches check_inode_numbers $testdir $tmp.after_copyup $tmp.after_move # Verify that the inode numbers survive a mount cycle -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on # Compare inode numbers before/after mount cycle
tests/overlay/044 +1 -1
@@ -99,7 +99,7 @@ cat $FILES check_ino_nlink $SCRATCH_MNT $tmp.before $tmp.after_one # Verify that the hardlinks survive a mount cycle -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT _overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o index=on,xino=on _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
tests/overlay/048 +2 -2
@@ -32,7 +32,7 @@ report_nlink() _ls_l $SCRATCH_MNT/$f | awk '{ print $2, $9 }' | _filter_scratch done - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Create lower hardlinks
@@ -101,7 +101,7 @@ touch $SCRATCH_MNT/1 touch $SCRATCH_MNT/2 # Perform the rest of the changes offline -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT test_hardlinks_offline
tests/overlay/049 +1 -1
@@ -32,7 +32,7 @@ create_redirect() touch $SCRATCH_MNT/origin/file mv $SCRATCH_MNT/origin $SCRATCH_MNT/$redirect - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Import common functions.
tests/overlay/050 +1 -1
@@ -76,7 +76,7 @@ mount_dirs() # Unmount the overlay without unmounting base fs unmount_dirs() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Check non-stale file handles of lower/upper files and verify
tests/overlay/051 +2 -2
@@ -28,7 +28,7 @@ _cleanup() # Cleanup overlay scratch mount that is holding base test mount # to prevent _check_test_fs and _test_umount from failing before # _check_scratch_fs _scratch_umount - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null } # Import common functions.
@@ -103,7 +103,7 @@ mount_dirs() # underlying dirs unmount_dirs() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_check_scratch_dirs $middle:$lower $upper $work \ -o "index=on,nfs_export=on"
tests/overlay/052 +1 -1
@@ -73,7 +73,7 @@ mount_dirs() # Unmount the overlay without unmounting base fs unmount_dirs() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Check non-stale file handles of lower/upper moved files
tests/overlay/053 +2 -2
@@ -30,7 +30,7 @@ _cleanup() # Cleanup overlay scratch mount that is holding base test mount # to prevent _check_test_fs and _test_umount from failing before # _check_scratch_fs _scratch_umount - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null } # Import common functions.
@@ -99,7 +99,7 @@ mount_dirs() # underlying dirs unmount_dirs() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_check_scratch_dirs $middle:$lower $upper $work \ -o "index=on,nfs_export=on,redirect_dir=on"
tests/overlay/054 +1 -1
@@ -87,7 +87,7 @@ mount_dirs() # Unmount the overlay without unmounting base fs unmount_dirs() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Check encode/decode/read file handles of dir with non-indexed ancestor
tests/overlay/055 +2 -2
@@ -37,7 +37,7 @@ _cleanup() # Cleanup overlay scratch mount that is holding base test mount # to prevent _check_test_fs and _test_umount from failing before # _check_scratch_fs _scratch_umount - $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null + _umount $SCRATCH_MNT 2>/dev/null } # Import common functions.
@@ -109,7 +109,7 @@ mount_dirs() # underlying dirs unmount_dirs() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_check_scratch_dirs $middle:$lower $upper $work \ -o "index=on,nfs_export=on,redirect_dir=on"
tests/overlay/056 +1 -1
@@ -73,7 +73,7 @@ mkdir $lowerdir/testdir2/subdir _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir touch $SCRATCH_MNT/testdir1/foo touch $SCRATCH_MNT/testdir2/subdir -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT remove_impure $upperdir/testdir1 remove_impure $upperdir/testdir2
tests/overlay/057 +2 -2
@@ -48,7 +48,7 @@ _overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on # Create opaque parent with absolute redirect child in middle layer mkdir $SCRATCH_MNT/pure mv $SCRATCH_MNT/origin $SCRATCH_MNT/pure/redirect -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT _overlay_scratch_mount_dirs $lowerdir2:$lowerdir $upperdir $workdir -o redirect_dir=on mv $SCRATCH_MNT/pure/redirect $SCRATCH_MNT/redirect # List content of renamed merge dir before mount cycle
@@ -56,7 +56,7 @@ ls $SCRATCH_MNT/redirect/ # Verify that redirects are followed by listing content of renamed merge dir # after mount cycle -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT _overlay_scratch_mount_dirs $lowerdir2:$lowerdir $upperdir $workdir -o redirect_dir=on ls $SCRATCH_MNT/redirect/
tests/overlay/059 +1 -1
@@ -33,7 +33,7 @@ create_origin_ref() _scratch_mount -o redirect_dir=on mv $SCRATCH_MNT/origin $SCRATCH_MNT/$ref - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Import common functions.
tests/overlay/060 +1 -1
@@ -130,7 +130,7 @@ mount_ro_overlay() umount_overlay() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } # Assumes it is called with overlay mounted.
tests/overlay/062 +1 -1
@@ -18,7 +18,7 @@ _cleanup() { cd / rm -f $tmp.* - $UMOUNT_PROG $lowertestdir + _umount $lowertestdir } # Import common functions.
tests/overlay/063 +1 -1
@@ -40,7 +40,7 @@ rm ${upperdir}/file mkdir ${SCRATCH_MNT}/file > /dev/null 2>&1 # unmount overlayfs -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT echo "Silence is golden" # success, all done
tests/overlay/065 +11 -11
@@ -30,7 +30,7 @@ _cleanup() { cd / rm -f $tmp.* - $UMOUNT_PROG $mnt2 2>/dev/null + _umount $mnt2 2>/dev/null } # Import common functions.
@@ -63,7 +63,7 @@ mkdir -p $lowerdir/lower $upperdir $workdir echo Conflicting upperdir/lowerdir _overlay_scratch_mount_dirs $upperdir $upperdir $workdir \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null # Use new upper/work dirs for each test to avoid ESTALE errors # on mismatch lowerdir/upperdir (see test overlay/037)
@@ -75,7 +75,7 @@ mkdir $upperdir $workdir echo Conflicting workdir/lowerdir _overlay_scratch_mount_dirs $workdir $upperdir $workdir \ -oindex=off 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir $workdir mkdir -p $upperdir/lower $workdir
@@ -85,7 +85,7 @@ mkdir -p $upperdir/lower $workdir echo Overlapping upperdir/lowerdir _overlay_scratch_mount_dirs $upperdir/lower $upperdir $workdir \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir $workdir mkdir $upperdir $workdir
@@ -94,7 +94,7 @@ mkdir $upperdir $workdir echo Conflicting lower layers _overlay_scratch_mount_dirs $lowerdir:$lowerdir $upperdir $workdir \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir $workdir mkdir $upperdir $workdir
@@ -103,7 +103,7 @@ mkdir $upperdir $workdir echo Overlapping lower layers below _overlay_scratch_mount_dirs $lowerdir:$lowerdir/lower $upperdir $workdir \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir $workdir mkdir $upperdir $workdir
@@ -112,7 +112,7 @@ mkdir $upperdir $workdir echo Overlapping lower layers above _overlay_scratch_mount_dirs $lowerdir/lower:$lowerdir $upperdir $workdir \ 2>&1 | _filter_error_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir $workdir mkdir -p $upperdir/upper $workdir $mnt2
@@ -129,14 +129,14 @@ mkdir -p $upperdir2 $workdir2 $mnt2 echo "Overlapping with upperdir of another instance (index=on)" _overlay_scratch_mount_dirs $upperdir/upper $upperdir2 $workdir2 \ -oindex=on 2>&1 | _filter_busy_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir2 $workdir2 mkdir -p $upperdir2 $workdir2 echo "Overlapping with upperdir of another instance (index=off)" _overlay_scratch_mount_dirs $upperdir/upper $upperdir2 $workdir2 \ - -oindex=off && $UMOUNT_PROG $SCRATCH_MNT + -oindex=off && _umount $SCRATCH_MNT rm -rf $upperdir2 $workdir2 mkdir -p $upperdir2 $workdir2
@@ -146,14 +146,14 @@ mkdir -p $upperdir2 $workdir2 echo "Overlapping with workdir of another instance (index=on)" _overlay_scratch_mount_dirs $workdir/work $upperdir2 $workdir2 \ -oindex=on 2>&1 | _filter_busy_mount -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null rm -rf $upperdir2 $workdir2 mkdir -p $upperdir2 $workdir2 echo "Overlapping with workdir of another instance (index=off)" _overlay_scratch_mount_dirs $workdir/work $upperdir2 $workdir2 \ - -oindex=off && $UMOUNT_PROG $SCRATCH_MNT + -oindex=off && _umount $SCRATCH_MNT # Move upper layer root into lower layer after mount echo Overlapping upperdir/lowerdir after mount
tests/overlay/067 +1 -1
@@ -70,7 +70,7 @@ stat $testfile >>$seqres.full diff -q $realfile $testfile >>$seqres.full && echo "diff with middle layer file doesn't know right from wrong! (cold cache)" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # check overlayfs _overlay_check_scratch_dirs $middle:$lower $upper $work -o xino=off
tests/overlay/068 +2 -2
@@ -28,7 +28,7 @@ _cleanup() cd / rm -f $tmp.* # Unmount the nested overlay mount - $UMOUNT_PROG $mnt2 2>/dev/null + _umount $mnt2 2>/dev/null } # Import common functions.
@@ -100,7 +100,7 @@ mount_dirs() unmount_dirs() { # unmount & check nested overlay - $UMOUNT_PROG $mnt2 + _umount $mnt2 _overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \ -o "index=on,nfs_export=on,redirect_dir=on"
tests/overlay/069 +3 -3
@@ -28,7 +28,7 @@ _cleanup() cd / rm -f $tmp.* # Unmount the nested overlay mount - $UMOUNT_PROG $mnt2 2>/dev/null + _umount $mnt2 2>/dev/null } # Import common functions.
@@ -108,12 +108,12 @@ mount_dirs() unmount_dirs() { # unmount & check nested overlay - $UMOUNT_PROG $mnt2 + _umount $mnt2 _overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \ -o "index=on,nfs_export=on,redirect_dir=on" # unmount & check underlying overlay - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_check_dirs $lower $upper $work \ -o "index=on,nfs_export=on,redirect_dir=on" }
tests/overlay/070 +3 -3
@@ -26,7 +26,7 @@ _cleanup() cd / rm -f $tmp.* # Unmount the nested overlay mount - $UMOUNT_PROG $mnt2 2>/dev/null + _umount $mnt2 2>/dev/null [ -z "$loopdev" ] || _destroy_loop_device $loopdev }
@@ -93,12 +93,12 @@ mount_dirs() unmount_dirs() { # unmount & check nested overlay - $UMOUNT_PROG $mnt2 + _umount $mnt2 _overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \ -o "redirect_dir=on,index=on,xino=on" # unmount & check underlying overlay - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_check_scratch_dirs $lower $upper $work \ -o "index=on,nfs_export=on" }
tests/overlay/071 +3 -3
@@ -29,7 +29,7 @@ _cleanup() cd / rm -f $tmp.* # Unmount the nested overlay mount - $UMOUNT_PROG $mnt2 2>/dev/null + _umount $mnt2 2>/dev/null [ -z "$loopdev" ] || _destroy_loop_device $loopdev }
@@ -103,12 +103,12 @@ mount_dirs() unmount_dirs() { # unmount & check nested overlay - $UMOUNT_PROG $mnt2 + _umount $mnt2 _overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \ -o "redirect_dir=on,index=on,xino=on" # unmount & check underlying overlay - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT _overlay_check_dirs $lower $upper $work \ -o "index=on,nfs_export=on" }
tests/overlay/076 +1 -1
@@ -47,7 +47,7 @@ _scratch_mount # on kernel v5.10..v5.10.14. Anything but hang is considered a test success. $CHATTR_PROG +i $SCRATCH_MNT/foo > /dev/null 2>&1 -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # success, all done echo "Silence is golden"
tests/overlay/077 +1 -1
@@ -65,7 +65,7 @@ mv $SCRATCH_MNT/f100 $SCRATCH_MNT/former/ # Remove the lower directory and mount overlay again to create # a "former merge dir" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT rm -rf $lowerdir/former _scratch_mount
tests/overlay/078 +1 -1
@@ -61,7 +61,7 @@ do_check() echo "Test chattr +$1 $2" >> $seqres.full - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT # Add attribute to lower file $CHATTR_PROG +$attr $lowertestfile
tests/overlay/079 +1 -1
@@ -156,7 +156,7 @@ mount_ro_overlay() umount_overlay() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } test_no_access()
tests/overlay/080 +1 -1
@@ -264,7 +264,7 @@ mount_overlay() umount_overlay() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT }
tests/overlay/081 +7 -7
@@ -46,7 +46,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir) echo "Overlayfs (uuid=null) and upper fs fsid differ" # Keep base fs mounted in case it has a volatile fsid (e.g. tmpfs) -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Test legacy behavior is preserved by default for existing "impure" overlayfs _scratch_mount
@@ -55,7 +55,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir) [[ "$ovl_fsid" == "$upper_fsid" ]] || \ echo "Overlayfs (after uuid=null) and upper fs fsid differ" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Test unique fsid on explicit opt-in for existing "impure" overlayfs _scratch_mount -o uuid=on
@@ -65,7 +65,7 @@ ovl_unique_fsid=$ovl_fsid [[ "$ovl_fsid" != "$upper_fsid" ]] || \ echo "Overlayfs (uuid=on) and upper fs fsid are the same" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Test unique fsid is persistent by default after it was created _scratch_mount
@@ -74,7 +74,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir) [[ "$ovl_fsid" == "$ovl_unique_fsid" ]] || \ echo "Overlayfs (after uuid=on) unique fsid is not persistent" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Test ignore existing persistent fsid on explicit opt-out _scratch_mount -o uuid=null
@@ -83,7 +83,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir) [[ "$ovl_fsid" == "$upper_fsid" ]] || \ echo "Overlayfs (uuid=null) and upper fs fsid differ" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Test fallback to uuid=null with non-upper ovelray _overlay_scratch_mount_dirs "$upperdir:$lowerdir" "-" "-" -o ro,uuid=on
@@ -110,7 +110,7 @@ ovl_unique_fsid=$ovl_fsid [[ "$ovl_fsid" != "$upper_fsid" ]] || \ echo "Overlayfs (new) and upper fs fsid are the same" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT # Test unique fsid is persistent by default after it was created _scratch_mount -o uuid=on
@@ -119,7 +119,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir) [[ "$ovl_fsid" == "$ovl_unique_fsid" ]] || \ echo "Overlayfs (uuid=on) unique fsid is not persistent" -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT echo "Silence is golden" status=0
tests/overlay/083 +1 -1
@@ -52,7 +52,7 @@ _mount -t overlay | grep ovl_esc_test | tee -a $seqres.full | grep -v spaces && # Re-create the upper/work dirs to mount them with a different lower # This is required in case index feature is enabled -$UMOUNT_PROG $SCRATCH_MNT +_umount $SCRATCH_MNT rm -rf "$upperdir" "$workdir" mkdir -p "$upperdir" "$workdir"
tests/overlay/084 +5 -5
@@ -15,7 +15,7 @@ _cleanup() { cd / # Unmount nested mounts if things fail - $UMOUNT_PROG $OVL_BASE_SCRATCH_MNT/nested 2>/dev/null + _umount $OVL_BASE_SCRATCH_MNT/nested 2>/dev/null rm -rf $tmp }
@@ -44,7 +44,7 @@ nesteddir=$OVL_BASE_SCRATCH_MNT/nested umount_overlay() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } test_escape()
@@ -88,12 +88,12 @@ test_escape() echo "nested xattr mount with trusted.overlay" _overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - overlayfs $nesteddir stat $nesteddir/dir/file 2>&1 | _filter_scratch - $UMOUNT_PROG $nesteddir + _umount $nesteddir echo "nested xattr mount with user.overlay" _overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - -o userxattr overlayfs $nesteddir stat $nesteddir/dir/file 2>&1 | _filter_scratch - $UMOUNT_PROG $nesteddir + _umount $nesteddir # Also ensure propagate the escaped xattr when we copy-up layer2/dir echo "copy-up of escaped xattrs"
@@ -164,7 +164,7 @@ test_escaped_xwhiteout() do_test_xwhiteout $prefix $nesteddir - $UMOUNT_PROG $nesteddir + _umount $nesteddir } test_escaped_xwhiteout trusted
tests/overlay/085 +1 -1
@@ -157,7 +157,7 @@ mount_ro_overlay() umount_overlay() { - $UMOUNT_PROG $SCRATCH_MNT + _umount $SCRATCH_MNT } test_no_access()
tests/overlay/086 +4 -4
@@ -38,21 +38,21 @@ _mount -t overlay none $SCRATCH_MNT \ 2>> $seqres.full && \ echo "ERROR: invalid combination of lowerdir and lowerdir+ mount options" -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null _mount -t overlay none $SCRATCH_MNT \ -o"lowerdir=$lowerdir,datadir+=$lowerdir_colons" \ -o redirect_dir=follow,metacopy=on 2>> $seqres.full && \ echo "ERROR: invalid combination of lowerdir and datadir+ mount options" -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null _mount -t overlay none $SCRATCH_MNT \ -o"datadir+=$lowerdir,lowerdir+=$lowerdir_colons" \ -o redirect_dir=follow,metacopy=on 2>> $seqres.full && \ echo "ERROR: invalid order of lowerdir+ and datadir+ mount options" -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null # mount is expected to fail with escaped colons. _mount -t overlay none $SCRATCH_MNT \
@@ -60,7 +60,7 @@ _mount -t overlay none $SCRATCH_MNT \ 2>> $seqres.full && \ echo "ERROR: incorrect parsing of escaped colons in lowerdir+ mount option" -$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null +_umount $SCRATCH_MNT 2>/dev/null # mount is expected to succeed without escaped colons. _mount -t overlay ovl_esc_test $SCRATCH_MNT \
tests/xfs/078 +2 -2
@@ -16,7 +16,7 @@ _cleanup() { cd / rm -f $tmp.* - $UMOUNT_PROG $LOOP_MNT 2>/dev/null + _umount $LOOP_MNT 2>/dev/null [ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null # try to keep the image file if test fails [ $status -eq 0 ] && rm -f $LOOP_IMG
@@ -81,7 +81,7 @@ _grow_loop() $XFS_GROWFS_PROG $LOOP_MNT 2>&1 | _filter_growfs 2>&1 echo "*** unmount" - $UMOUNT_PROG -d $LOOP_MNT && LOOP_DEV= + _umount -d $LOOP_MNT && LOOP_DEV= # Large grows takes forever to check.. if [ "$check" -gt "0" ]
tests/xfs/148 +3 -3
@@ -14,7 +14,7 @@ _begin_fstest auto quick fuzzers _cleanup() { cd / - $UMOUNT_PROG $mntpt > /dev/null 2>&1 + _umount $mntpt > /dev/null 2>&1 _destroy_loop_device $loopdev > /dev/null 2>&1 rm -r -f $tmp.* }
@@ -90,7 +90,7 @@ cat $tmp.log >> $seqres.full cat $tmp.log | _filter_test_dir # Corrupt the entries -$UMOUNT_PROG $mntpt +_umount $mntpt _destroy_loop_device $loopdev cp $imgfile $imgfile.old sed -b \
@@ -121,7 +121,7 @@ fi echo "does repair complain?" >> $seqres.full # Does repair complain about this? -$UMOUNT_PROG $mntpt +_umount $mntpt $XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1 res=$? test $res -eq 1 || \
tests/xfs/149 +2 -2
@@ -22,7 +22,7 @@ loop_symlink=$TEST_DIR/loop_symlink.$$ # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $mntdir + _umount $mntdir [ -n "$loop_dev" ] && _destroy_loop_device $loop_dev rmdir $mntdir rm -f $loop_symlink
@@ -73,7 +73,7 @@ echo "=== xfs_growfs - check device symlink ===" $XFS_GROWFS_PROG -D 12288 $loop_symlink > /dev/null echo "=== unmount ===" -$UMOUNT_PROG $mntdir || _fail "!!! failed to unmount" +_umount $mntdir || _fail "!!! failed to unmount" echo "=== mount device symlink ===" _mount $loop_symlink $mntdir || _fail "!!! failed to loopback mount"
tests/xfs/186 +2 -2
@@ -87,7 +87,7 @@ _do_eas() _create_eas $2 $3 fi echo "" - cd /; $UMOUNT_PROG $SCRATCH_MNT + cd /; _umount $SCRATCH_MNT _print_inode }
@@ -99,7 +99,7 @@ _do_dirents() echo "" _scratch_mount _create_dirents $1 $2 - cd /; $UMOUNT_PROG $SCRATCH_MNT + cd /; _umount $SCRATCH_MNT _print_inode }
tests/xfs/289 +2 -2
@@ -13,8 +13,8 @@ _begin_fstest growfs auto quick # Override the default cleanup function. _cleanup() { - $UMOUNT_PROG $tmpdir - $UMOUNT_PROG $tmpbind + _umount $tmpdir + _umount $tmpbind rmdir $tmpdir rm -f $tmpsymlink rmdir $tmpbind
tests/xfs/507 +1 -1
@@ -22,7 +22,7 @@ _register_cleanup "_cleanup" BUS _cleanup() { cd / - test -n "$loop_mount" && $UMOUNT_PROG $loop_mount > /dev/null 2>&1 + test -n "$loop_mount" && _umount $loop_mount > /dev/null 2>&1 test -n "$loop_dev" && _destroy_loop_device $loop_dev rm -rf $tmp.* }
tests/xfs/513 +2 -2
@@ -14,7 +14,7 @@ _cleanup() { cd / rm -f $tmp.* - $UMOUNT_PROG $LOOP_MNT 2>/dev/null + _umount $LOOP_MNT 2>/dev/null if [ -n "$LOOP_DEV" ];then _destroy_loop_device $LOOP_DEV 2>/dev/null fi
@@ -89,7 +89,7 @@ get_mount_info() force_unmount() { - $UMOUNT_PROG $LOOP_MNT >/dev/null 2>&1 + _umount $LOOP_MNT >/dev/null 2>&1 } # _do_test <mount options> <should be mounted?> [<key string> <key should be found?>]
tests/xfs/544 +1 -1
@@ -15,7 +15,7 @@ _cleanup() _cleanup_dump cd / rm -r -f $tmp.* - $UMOUNT_PROG $TEST_DIR/dest.$seq 2> /dev/null + _umount $TEST_DIR/dest.$seq 2> /dev/null rmdir $TEST_DIR/src.$seq 2> /dev/null rmdir $TEST_DIR/dest.$seq 2> /dev/null }
tests/xfs/806 +2 -2
@@ -23,7 +23,7 @@ _cleanup() { cd / rm -r -f $tmp.* - umount $dummymnt &>/dev/null + _umount $dummymnt &>/dev/null rmdir $dummymnt &>/dev/null rm -f $dummyfile }
@@ -46,7 +46,7 @@ testme() { XFS_SCRUB_PHASE=7 $XFS_SCRUB_PROG -d -o autofsck $dummymnt 2>&1 | \ grep autofsck | _filter_test_dir | \ sed -e 's/\(directive.\).*$/\1/g' - umount $dummymnt + _umount $dummymnt } # We don't test the absence of an autofsck directive because xfs_scrub behaves