xfs: the rest of online repair part 1
Subsystems (3)
Re: [PATCH 1/1] xfs: map xfile pages directly into xfs_buf
In reply to Christoph HellwigThis comment should have been for the kernel version and not the userspace artefact patch.
PATCH 25/40 xfsprogs: inode-related repair fixes
In reply to Darrick J. WongHi all,
While doing QA of the online fsck code, I made a few observations:
First, nobody was checking that the di_onlink field is actually zero; Second, that allocating a temporary file for repairs can fail (and thus bring down the entire fs) if the inode cluster is corrupt; and Third, that file link counts do not pin at ~0U to prevent integer overflows.
This scattered patchset fixes those three problems.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=inode-repair-improvements
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=inode-repair-improvements
--- include/xfs_inode.h | 2 ++ libxfs/util.c | 24 ++++++++++++++++++++++++ libxfs/xfs_format.h | 6 ++++++ libxfs/xfs_ialloc.c | 40 ++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_inode_buf.c | 8 ++++++++ mkfs/proto.c | 4 ++-- repair/incore_ino.c | 3 ++- repair/phase6.c | 10 +++++----- 8 files changed, 89 insertions(+), 8 deletions(-)
PATCH 1/4 xfs: check unused nlink fields in the ondisk inode
From: Darrick J. Wong <[email protected]>
v2/v3 inodes use di_nlink and not di_onlink; and v1 inodes use di_onlink and not di_nlink. Whichever field is not in use, make sure its contents are zero, and teach xfs_scrub to fix that if it is.
This clears a bunch of missing scrub failure errors in xfs/385 for core.onlink.
--- libxfs/xfs_inode_buf.c | 8 ++++++++ 1 file changed, 8 insertions(+)
libxfs/xfs_inode_buf.c +8 -0
@@ -488,6 +488,14 @@ xfs_dinode_verify(return __this_address; } +if (dip->di_version >1 ) { +if (dip->di_onlink) +return __this_address; + }else { +if (dip->di_nlink) +return __this_address; + } +/* don't allow invalid i_size */ di_size =be64_to_cpu (dip->di_size);if (di_size & (1ULL <<63 ))
PATCH 2/4 xfs: try to avoid allocating from sick inode clusters
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
I noticed that xfs/413 and xfs/375 occasionally failed while fuzzing core.mode of an inode. The root cause of these problems is that the field we fuzzed (core.mode or core.magic, typically) causes the entire inode cluster buffer verification to fail, which affects several inodes at once. The repair process tries to create either a /lost+found or a temporary repair file, but regrettably it picks the same inode cluster that we just corrupted, with the result that repair triggers the demise of the filesystem.
Try avoid this by making the inode allocation path detect when the perag health status indicates that someone has found bad inode cluster buffers, and try to read the inode cluster buffer. If the cluster buffer fails the verifiers, try another AG. This isn't foolproof and can result in premature ENOSPC, but that might be better than shutting down.
--- libxfs/util.c | 6 ++++++ libxfs/xfs_ialloc.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
libxfs/util.c +6 -0
@@ -732,6 +732,12 @@ void xfs_fs_mark_sick(struct xfs_mount *mp, unsigned int mask) { }void xfs_agno_mark_sick (struct xfs_mount *mp,xfs_agnumber_t agno,unsigned int mask) { }void xfs_ag_mark_sick (struct xfs_perag *pag,unsigned int mask) { } +void xfs_ag_measure_sickness (struct xfs_perag *pag,unsigned int *sick, +unsigned int *checked) +{ + *sick =0 ; + *checked =0 ; +}void xfs_bmap_mark_sick (struct xfs_inode *ip,int whichfork) { }void xfs_btree_mark_sick (struct xfs_btree_cur *cur) { }void xfs_dirattr_mark_sick (struct xfs_inode *ip,int whichfork) { }
libxfs/xfs_ialloc.c +40 -0
@@ -1007,6 +1007,33 @@ xfs_inobt_first_free_inode(return xfs_lowbit64 (realfree); } +/* +* If this AG has corrupt inodes, check if allocating this inode would fail +* with corruption errors. Returns 0 if we're clear, or EAGAIN to try again +* somewhere else. +*/ +static int +xfs_dialloc_check_ino ( +struct xfs_perag *pag, +struct xfs_trans *tp, +xfs_ino_t ino) +{ +struct xfs_imap imap; +struct xfs_buf *bp; +int error; + + error =xfs_imap (pag, tp, ino, &imap,0 ); +if (error) +return -EAGAIN ; + + error =xfs_imap_to_bp (pag->pag_mount, tp, &imap, &bp); +if (error) +return -EAGAIN ; + +xfs_trans_brelse (tp, bp); +return 0 ; +} +/* * Allocate an inode using the inobt-only algorithm. */
@@ -1259,6 +1286,13 @@ xfs_dialloc_ag_inobt(ASSERT ((XFS_AGINO_TO_OFFSET (mp, rec.ir_startino) %XFS_INODES_PER_CHUNK ) ==0 ); ino =XFS_AGINO_TO_INO (mp, pag->pag_agno, rec.ir_startino + offset); + +if (xfs_ag_has_sickness (pag,XFS_SICK_AG_INODES )) { + error =xfs_dialloc_check_ino (pag, tp, ino); +if (error) + goto error0; + } + rec.ir_free &= ~XFS_INOBT_MASK (offset); rec.ir_freecount--; error =xfs_inobt_update (cur, &rec);
@@ -1534,6 +1568,12 @@ xfs_dialloc_ag( XFS_INODES_PER_CHUNK) == 0); ino =XFS_AGINO_TO_INO (mp, pag->pag_agno, rec.ir_startino + offset); +if (xfs_ag_has_sickness (pag,XFS_SICK_AG_INODES )) { + error =xfs_dialloc_check_ino (pag, tp, ino); +if (error) + goto error_cur; + } +/* * Modify or remove the finobt record. */
PATCH 3/4 libxfs: port the bumplink function from the kernel
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port the xfs_bumplink function from the kernel and use it to replace raw calls to inc_nlink. The next patch will need this common function to prevent integer overflows in the link count.
--- include/xfs_inode.h | 2 ++ libxfs/util.c | 17 +++++++++++++++++ mkfs/proto.c | 4 ++-- repair/phase6.c | 10 +++++----- 4 files changed, 26 insertions(+), 7 deletions(-)
include/xfs_inode.h +2 -0
@@ -348,6 +348,8 @@ extern void libxfs_trans_ichgtime(struct xfs_trans *, struct xfs_inode *, int);extern int libxfs_iflush_int (struct xfs_inode *,struct xfs_buf *); +void libxfs_bumplink (struct xfs_trans *tp,struct xfs_inode *ip); +/* Inode Cache Interfaces */ extern int libxfs_iget (struct xfs_mount *,struct xfs_trans *,xfs_ino_t ,uint ,struct xfs_inode **);
libxfs/util.c +17 -0
@@ -240,6 +240,23 @@ xfs_inode_propagate_flags( ip->i_diflags |= di_flags; } +/* +* Increment the link count on an inode & log the change. +*/ +void +libxfs_bumplink ( +struct xfs_trans *tp, +struct xfs_inode *ip) +{ +struct inode *inode =VFS_I (ip); + +xfs_trans_ichgtime (tp, ip,XFS_ICHGTIME_CHG ); + +inc_nlink (inode); + +xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); +} +/* * Initialise a newly allocated inode and return the in-core inode to the * caller locked exclusively.
mkfs/proto.c +2 -2
@@ -590,7 +590,7 @@ parseproto( &creds, fsxp, &ip);if (error)fail (_ ("Inode allocation failed" ), error); -inc_nlink (VFS_I (ip));/* account for . */ +libxfs_bumplink (tp, ip);/* account for . */ if (!pip) { pip = ip; mp->m_sb.sb_rootino = ip->i_ino;
@@ -600,7 +600,7 @@ parseproto(libxfs_trans_ijoin (tp, pip,0 ); xname.type =XFS_DIR3_FT_DIR ;newdirent (mp, tp, pip, &xname, ip->i_ino); -inc_nlink (VFS_I (pip)); +libxfs_bumplink (tp, pip);libxfs_trans_log_inode (tp, pip,XFS_ILOG_CORE ); }newdirectory (mp, tp, ip, pip);
repair/phase6.c +5 -5
@@ -944,7 +944,7 @@ mk_orphanage(xfs_mount_t *mp)do_error (_ ("%s inode allocation failed %d\n" ),ORPHANAGE , error); } -inc_nlink (VFS_I (ip));/* account for . */ +libxfs_bumplink (tp, ip);/* account for . */ ino = ip->i_ino; irec = find_inode_rec(mp,
@@ -996,7 +996,7 @@ mk_orphanage(xfs_mount_t *mp)* for .. in the new directory, and update the irec copy of the * on-disk nlink so we don't fail the link count check later. */ -inc_nlink (VFS_I (pip)); +libxfs_bumplink (tp, pip); irec =find_inode_rec (mp,XFS_INO_TO_AGNO (mp, mp->m_sb.sb_rootino),XFS_INO_TO_AGINO (mp, mp->m_sb.sb_rootino));add_inode_ref (irec,0 );
@@ -1090,7 +1090,7 @@ mv_orphanage(if (irec)add_inode_ref (irec, ino_offset);else -inc_nlink (VFS_I (orphanage_ip)); +libxfs_bumplink (tp, orphanage_ip);libxfs_trans_log_inode (tp, orphanage_ip,XFS_ILOG_CORE ); err = -libxfs_dir_createname(tp, ino_p, &xfs_name_dotdot,
@@ -1099,7 +1099,7 @@ mv_orphanage(do_error (_ ("creation of .. entry failed (%d)\n" ), err); -inc_nlink (VFS_I (ino_p)); +libxfs_bumplink (tp, ino_p);libxfs_trans_log_inode (tp, ino_p,XFS_ILOG_CORE ); err = -libxfs_trans_commit (tp);if (err)
@@ -1124,7 +1124,7 @@ mv_orphanage(if (irec)add_inode_ref (irec, ino_offset);else -inc_nlink (VFS_I (orphanage_ip)); +libxfs_bumplink (tp, orphanage_ip);libxfs_trans_log_inode (tp, orphanage_ip,XFS_ILOG_CORE );/*
PATCH 4/4 xfs: pin inodes that would otherwise overflow link count
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
The VFS inc_nlink function does not explicitly check for integer overflows in the i_nlink field. Instead, it checks the link count against s_max_links in the vfs_{link,create,rename} functions. XFS sets the maximum link count to 2.1 billion, so integer overflows should not be a problem.
However. It's possible that online repair could find that a file has more than four billion links, particularly if the link count got corrupted while creating hardlinks to the file. The di_nlinkv2 field is not large enough to store a value larger than 2^32, so we ought to define a magic pin value of ~0U which means that the inode never gets deleted. This will prevent a UAF error if the repair finds this situation and users begin deleting links to the file.
--- libxfs/util.c | 3 ++- libxfs/xfs_format.h | 6 ++++++ repair/incore_ino.c | 3 ++- 3 files changed, 10 insertions(+), 2 deletions(-)
libxfs/util.c +2 -1
@@ -252,7 +252,8 @@ libxfs_bumplink(xfs_trans_ichgtime (tp, ip,XFS_ICHGTIME_CHG ); -inc_nlink (inode); +if (inode->i_nlink !=XFS_NLINK_PINNED ) +inc_nlink (inode);xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); }
libxfs/xfs_format.h +6 -0
@@ -912,6 +912,12 @@ static inline uint xfs_dinode_size(int version)*/ #define XFS_MAXLINK ((1U << 31) - 1U) +/* +* Any file that hits the maximum ondisk link count should be pinned to avoid +* a use-after-free situation. +*/ +#define XFS_NLINK_PINNED (~0U) +/* * Values for di_format *
repair/incore_ino.c +2 -1
@@ -108,7 +108,8 @@ void add_inode_ref(struct ino_tree_node *irec, int ino_offset)nlink_grow_16_to_32 (irec);/*FALLTHRU*/ case sizeof (uint32_t ): - irec->ino_un.ex_data->counted_nlinks.un32[ino_offset]++; +if (irec->ino_un.ex_data->counted_nlinks.un32[ino_offset] !=XFS_NLINK_PINNED ) + irec->ino_un.ex_data->counted_nlinks.un32[ino_offset]++;break ;default :ASSERT (0 );
PATCH 26/40 xfs_scrub: fixes to the repair code
In reply to Darrick J. WongHi all,
Now that we've landed the new kernel code, it's time to reorganize the xfs_scrub code that handles repairs. Clean up various naming warts and misleading error messages. Move the repair code to scrub/repair.c as the first step. Then, fix various issues in the repair code before we start reorganizing things.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-fixes
--- scrub/phase1.c | 2 scrub/phase2.c | 3 - scrub/phase3.c | 2 scrub/phase4.c | 22 ++++- scrub/phase5.c | 2 scrub/phase6.c | 13 +++ scrub/phase7.c | 2 scrub/repair.c | 177 ++++++++++++++++++++++++++++++++++++++++++- scrub/repair.h | 16 +++- scrub/scrub.c | 204 +------------------------------------------------ scrub/scrub.h | 16 ---- scrub/scrub_private.h | 55 +++++++++++++ scrub/xfs_scrub.c | 2 13 files changed, 283 insertions(+), 233 deletions(-) create mode 100644 scrub/scrub_private.h
PATCH 1/7 xfs_scrub: flush stdout after printing to it
From: Darrick J. Wong <[email protected]>
Make sure we flush stdout after printf'ing to it, especially before we start any operation that could take a while to complete. Most of scrub already does this, but we missed a couple of spots.
--- scrub/xfs_scrub.c | 2 ++ 1 file changed, 2 insertions(+)
scrub/xfs_scrub.c +2 -0
@@ -535,6 +535,7 @@ _("%s: repairs made: %llu.\n"),fprintf (stdout,_ ("%s: optimizations made: %llu.\n" ), ctx->mntpoint, ctx->preens); +fflush (stdout); } static void
@@ -620,6 +621,7 @@ main(int error;fprintf (stdout,"EXPERIMENTAL xfs_scrub program in use! Use at your own risk!\n" ); +fflush (stdout); progname =basename (argv[0 ]);setlocale (LC_ALL ,"" );
Re: [PATCH 1/7] xfs_scrub: flush stdout after printing to it
Quoted 5 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Make sure we flush stdout after printf'ing to it, especially before we start any operation that could take a while to complete. Most of scrub already does this, but we missed a couple of spots.
Looks good:
PATCH 2/7 xfs_scrub: don't report media errors for space with unknowable owner
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
On filesystems that don't have the reverse mapping feature enabled, the GETFSMAP call cannot tell us much about the owner of a space extent -- we're limited to static fs metadata, free space, or "unknown". In this case, nothing is corrupt, so str_corrupt is not an appropriate logging function. Relax this to str_info so that the user sees a notice that media errors have been found so that the user knows something bad happened even if the directory tree walker cannot find the file owning the space where the media error was found.
Filesystems with rmap enabled are never supposed to return OWN_UNKNOWN from a GETFSMAP report, so continue to report that as a corruption. This fixes a regression in xfs/556.
--- scrub/phase6.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
scrub/phase6.c +12 -1
@@ -397,7 +397,18 @@ report_ioerr_fsmap(snprintf (buf,DESCR_BUFSZ ,_ ("disk offset %" PRIu64), (uint64_t )map->fmr_physical + err_off); type =decode_special_owner (map->fmr_owner); -str_corrupt (ctx, buf,_ ("media error in %s." ), type); +/* +* On filesystems that don't store reverse mappings, the +* GETFSMAP call returns OWNER_UNKNOWN for allocated space. +* We'll have to let the directory tree walker find the file +* that lost data. +*/ +if (!(ctx->mnt.fsgeom.flags &XFS_FSOP_GEOM_FLAGS_RMAPBT ) && + map->fmr_owner ==XFS_FMR_OWN_UNKNOWN ) { +str_info (ctx, buf,_ ("media error detected." )); + }else { +str_corrupt (ctx, buf,_ ("media error in %s." ), type); + } }/* Report extent maps */
Re: [PATCH 2/7] xfs_scrub: don't report media errors for space with unknowable owner
Looks good:
PATCH 3/7 xfs_scrub: remove ALP_* flags namespace
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
In preparation to move all the repair code to repair.[ch], remove the ALP_* flags namespace since it mostly overlaps with XRM_*. Rename the clunky "COMPLAIN_IF_UNFIXED" flag to "FINAL_WARNING", because that's what it really means.
--- scrub/phase3.c | 2 +- scrub/phase4.c | 2 +- scrub/phase5.c | 2 +- scrub/phase7.c | 2 +- scrub/repair.c | 4 ++-- scrub/repair.h | 16 ++++++++++++---- scrub/scrub.c | 10 +++++----- scrub/scrub.h | 10 ---------- 8 files changed, 23 insertions(+), 25 deletions(-)
scrub/phase3.c +1 -1
@@ -88,7 +88,7 @@ try_inode_repair(return 0 ; ret =action_list_process (ictx->ctx, fd, alist, -ALP_REPAIR_ONLY |ALP_NOPROGRESS ); +XRM_REPAIR_ONLY |XRM_NOPROGRESS );if (ret)return ret;
scrub/phase4.c +1 -1
@@ -54,7 +54,7 @@ repair_ag( }while (unfixed >0 );/* Try once more, but this time complain if we can't fix things. */ - flags |=ALP_COMPLAIN_IF_UNFIXED ; + flags |=XRM_FINAL_WARNING ; ret =action_list_process (ctx,-1 , alist, flags);if (ret) *aborted = true;
scrub/phase5.c +1 -1
@@ -422,7 +422,7 @@ fs_scan_worker( } ret =action_list_process (ctx, ctx->mnt.fd, &item->alist, -ALP_COMPLAIN_IF_UNFIXED |ALP_NOPROGRESS ); +XRM_FINAL_WARNING |XRM_NOPROGRESS );if (ret) {str_liberror (ctx, ret,_ ("repairing fs scan metadata" )); *item->abortedp = true;
scrub/phase7.c +1 -1
@@ -122,7 +122,7 @@ phase7_func(if (error)return error; error =action_list_process (ctx,-1 , &alist, -ALP_COMPLAIN_IF_UNFIXED |ALP_NOPROGRESS ); +XRM_FINAL_WARNING |XRM_NOPROGRESS );if (error)return error;
scrub/repair.c +2 -2
@@ -274,7 +274,7 @@ action_list_process( fix =xfs_repair_metadata (ctx, xfdp, aitem, repair_flags);switch (fix) {case CHECK_DONE : -if (!(repair_flags &ALP_NOPROGRESS )) +if (!(repair_flags &XRM_NOPROGRESS ))progress_add (1 ); alist->nr--;list_del (&aitem->list);
@@ -316,7 +316,7 @@ action_list_process_or_defer(int ret; ret =action_list_process (ctx,-1 , alist, -ALP_REPAIR_ONLY |ALP_NOPROGRESS ); +XRM_REPAIR_ONLY |XRM_NOPROGRESS );if (ret)return ret;
scrub/repair.h +12 -4
@@ -32,10 +32,18 @@ void action_list_find_mustfix(struct action_list *actions,unsigned long long *broken_primaries, unsigned long long *broken_secondaries); -/* Passed through to xfs_repair_metadata() */ -#define ALP_REPAIR_ONLY (XRM_REPAIR_ONLY) -#define ALP_COMPLAIN_IF_UNFIXED (XRM_COMPLAIN_IF_UNFIXED) -#define ALP_NOPROGRESS (1U << 31) +/* +* Only ask the kernel to repair this object if the kernel directly told us it +* was corrupt. Objects that are only flagged as having cross-referencing +* errors or flagged as eligible for optimization are left for later. +*/ +#define XRM_REPAIR_ONLY (1U << 0) + +/* This is the last repair attempt; complain if still broken even after fix. */ +#define XRM_FINAL_WARNING (1U << 1) + +/* Don't call progress_add after repairing an item. */ +#define XRM_NOPROGRESS (1U << 2)int action_list_process (struct scrub_ctx *ctx,int fd,struct action_list *alist,unsigned int repair_flags);
scrub/scrub.c +5 -5
@@ -743,7 +743,7 @@ _("Filesystem is shut down, aborting."));* could fix this, it's at least worth trying the scan * again to see if another repair fixed it. */ -if (!(repair_flags &XRM_COMPLAIN_IF_UNFIXED )) +if (!(repair_flags &XRM_FINAL_WARNING ))return CHECK_RETRY ; fallthrough;case EINVAL :
@@ -773,13 +773,13 @@ _("Read-only filesystem; cannot make changes."));* to requeue the repair for later and don't say a * thing. Otherwise, print error and bail out. */ -if (!(repair_flags &XRM_COMPLAIN_IF_UNFIXED )) +if (!(repair_flags &XRM_FINAL_WARNING ))return CHECK_RETRY ;str_liberror (ctx, error,descr_render (&dsc));return CHECK_DONE ; } -if (repair_flags &XRM_COMPLAIN_IF_UNFIXED ) +if (repair_flags &XRM_FINAL_WARNING )scrub_warn_incomplete_scrub (ctx, &dsc, &meta);if (needs_repair (&meta)) {/*
@@ -787,7 +787,7 @@ _("Read-only filesystem; cannot make changes."));* just requeue this and try again later. Otherwise we * log the error loudly and don't try again. */ -if (!(repair_flags &XRM_COMPLAIN_IF_UNFIXED )) +if (!(repair_flags &XRM_FINAL_WARNING ))return CHECK_RETRY ;str_corrupt (ctx,descr_render (&dsc),_ ("Repair unsuccessful; offline repair required." ));
@@ -799,7 +799,7 @@ _("Repair unsuccessful; offline repair required."));* caller to run xfs_repair; otherwise, we'll keep trying to * reverify the cross-referencing as repairs progress. */ -if (repair_flags &XRM_COMPLAIN_IF_UNFIXED ) { +if (repair_flags &XRM_FINAL_WARNING ) {str_info (ctx,descr_render (&dsc),_ ("Seems correct but cross-referencing failed; offline repair recommended." )); }else {
scrub/scrub.h +0 -10
@@ -54,16 +54,6 @@ struct action_item {__u32 agno; }; -/* -* Only ask the kernel to repair this object if the kernel directly told us it -* was corrupt. Objects that are only flagged as having cross-referencing -* errors or flagged as eligible for optimization are left for later. -*/ -#define XRM_REPAIR_ONLY (1U << 0) - -/* Complain if still broken even after fix. */ -#define XRM_COMPLAIN_IF_UNFIXED (1U << 1) -enum check_outcome xfs_repair_metadata (struct scrub_ctx *ctx,struct xfs_fd *xfdp,struct action_item *aitem,unsigned int repair_flags);
Re: [PATCH 3/7] xfs_scrub: remove ALP_* flags namespace
Looks good:
PATCH 4/7 xfs_scrub: move repair functions to repair.c
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Move all the repair functions to repair.c.
--- scrub/phase1.c | 2 scrub/repair.c | 169 +++++++++++++++++++++++++++++++++++++++++ scrub/scrub.c | 204 +------------------------------------------------ scrub/scrub.h | 6 - scrub/scrub_private.h | 55 +++++++++++++ 5 files changed, 230 insertions(+), 206 deletions(-) create mode 100644 scrub/scrub_private.h
scrub/phase1.c +1 -1
@@ -210,7 +210,7 @@ _("Kernel metadata scrubbing facility is not available.")); }/* Do we need kernel-assisted metadata repair? */ -if (ctx->mode !=SCRUB_MODE_DRY_RUN && !xfs_can_repair (ctx)) { +if (ctx->mode !=SCRUB_MODE_DRY_RUN && !can_repair (ctx)) {str_error (ctx, ctx->mntpoint,_ ("Kernel metadata repair facility is not available. Use -n to scrub." ));return ECANCELED ;
scrub/repair.c +169 -0
180 lines not on this page · raw message
scrub/scrub.c +4 -200
230 lines not on this page · raw message
scrub/scrub.h +1 -5
@@ -38,7 +38,7 @@ bool can_scrub_dir(struct scrub_ctx *ctx);bool can_scrub_attr (struct scrub_ctx *ctx);bool can_scrub_symlink (struct scrub_ctx *ctx);bool can_scrub_parent (struct scrub_ctx *ctx); -bool xfs_can_repair (struct scrub_ctx *ctx); +bool can_repair (struct scrub_ctx *ctx);bool can_force_rebuild (struct scrub_ctx *ctx); int scrub_file(struct scrub_ctx *ctx, int fd, const struct xfs_bulkstat *bstat,
@@ -54,8 +54,4 @@ struct action_item { __u32 agno; }; -enum check_outcome xfs_repair_metadata (struct scrub_ctx *ctx, -struct xfs_fd *xfdp,struct action_item *aitem, -unsigned int repair_flags); -#endif /* XFS_SCRUB_SCRUB_H_ */
scrub/scrub_private.h new file +55 -0
55 lines not on this page · raw message
Re: [PATCH 4/7] xfs_scrub: move repair functions to repair.c
Looks good:
PATCH 5/7 xfs_scrub: log when a repair was unnecessary
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the kernel tells us that a filesystem object didn't need repairs, we should log that with a message specific to that outcome.
--- scrub/repair.c | 4 ++++ 1 file changed, 4 insertions(+)
scrub/repair.c +4 -0
@@ -167,6 +167,10 @@ _("Repair unsuccessful; offline repair required.")); _("Seems correct but cross-referencing failed; will keep checking."));return CHECK_RETRY ; } + }else if (meta.sm_flags &XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED ) { +if (verbose) +str_info (ctx,descr_render (&dsc), +_ ("No modification needed." )); }else {/* Clean operation, no corruption detected. */ if (is_corrupt (&oldm))
Re: [PATCH 5/7] xfs_scrub: log when a repair was unnecessary
Looks good:
PATCH 6/7 xfs_scrub: require primary superblock repairs to complete before proceeding
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Phase 2 of the xfs_scrub program calls the kernel to check the primary superblock before scanning the rest of the filesystem. Though doing so is a no-op now (since the primary super must pass all checks as a prerequisite for mounting), the goal of this code is to enable future kernel code to intercept an xfs_scrub run before it actually does anything. If this some day involves fixing the primary superblock, it seems reasonable to require that /all/ repairs complete successfully before moving on to the rest of the filesystem.
Unfortunately, that's not what xfs_scrub does now -- primary super repairs that fail are theoretically deferred to phase 4! So make this mandatory.
--- scrub/phase2.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scrub/phase2.c +2 -1
@@ -174,7 +174,8 @@ phase2_func( ret =scrub_primary_super (ctx, &alist);if (ret) goto out_wq; - ret =action_list_process_or_defer (ctx,0 , &alist); + ret =action_list_process (ctx,-1 , &alist, +XRM_FINAL_WARNING |XRM_NOPROGRESS );if (ret) goto out_wq;
Re: [PATCH 6/7] xfs_scrub: require primary superblock repairs to complete before proceeding
Looks good:
PATCH 7/7 xfs_scrub: actually try to fix summary counters ahead of repairs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
A while ago, I decided to make phase 4 check the summary counters before it starts any other repairs, having observed that repairs of primary metadata can fail because the summary counters (incorrectly) claim that there aren't enough free resources in the filesystem. However, if problems are found in the summary counters, the repair work will be run as part of the AG 0 repairs, which means that it runs concurrently with other scrubbers. This doesn't quite get us to the intended goal, so try to fix the scrubbers ahead of time. If that fails, tough, we'll get back to it in phase 7 if scrub gets that far.
--- scrub/phase4.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
scrub/phase4.c +15 -5
@@ -129,6 +129,7 @@ phase4_func( struct scrub_ctx *ctx) {struct xfs_fsop_geom fsgeom; +struct action_list alist;int ret;if (!have_action_items (ctx))
@@ -136,11 +137,13 @@ phase4_func(/* * Check the summary counters early. Normally we do this during phase -* seven, but some of the cross-referencing requires fairly-accurate -* counters, so counter repairs have to be put on the list now so that -* they get fixed before we stop retrying unfixed metadata repairs. +* seven, but some of the cross-referencing requires fairly accurate +* summary counters. Check and try to repair them now to minimize the +* chance that repairs of primary metadata fail due to secondary +* metadata. If repairs fails, we'll come back during phase 7. */ - ret =scrub_fs_counters (ctx, &ctx->action_lists[0 ]); +action_list_init (&alist); + ret =scrub_fs_counters (ctx, &alist);if (ret)return ret;
@@ -155,11 +158,18 @@ phase4_func(return ret;if (fsgeom.sick &XFS_FSOP_GEOM_SICK_QUOTACHECK ) { - ret =scrub_quotacheck (ctx, &ctx->action_lists[0 ]); + ret =scrub_quotacheck (ctx, &alist);if (ret)return ret; } +/* Repair counters before starting on the rest. */ + ret =action_list_process (ctx,-1 , &alist, +XRM_REPAIR_ONLY |XRM_NOPROGRESS ); +if (ret) +return ret; +action_list_discard (&alist); + ret =repair_everything (ctx);if (ret)return ret;
Re: [PATCH 7/7] xfs_scrub: actually try to fix summary counters ahead of repairs
Looks good:
PATCH 27/40 xfs_scrub: improve warnings about difficult repairs
In reply to Darrick J. WongHi all,
While I was poking through the QA results for xfs_scrub, I noticed that it doesn't warn the user when the primary and secondary realtime metadata are so out of whack that the chances of a successful repair are not so high. I decided that it was worth refactoring the scrub code a bit so that we could warn the user about these types of things, and ended up refactoring unnecessary helpers out of existence and fixing other reporting gaps.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings
--- man/man8/xfs_scrub.8 | 19 ++++++++++++++++++ scrub/common.c | 2 ++ scrub/phase1.c | 2 +- scrub/phase2.c | 53 +++++++++++++++++++++++++++++++------------------- scrub/phase3.c | 21 ++++++++++++++++---- scrub/phase4.c | 9 +++++--- scrub/phase5.c | 15 +++++++------- scrub/repair.c | 47 ++++++++++++++++++++++++++++++++++---------- scrub/repair.h | 10 +++++++-- scrub/scrub.c | 52 +------------------------------------------------ scrub/scrub.h | 7 ++----- scrub/xfs_scrub.c | 45 ++++++++++++++++++++++++++++++++++++++++++ scrub/xfs_scrub.h | 1 + 13 files changed, 175 insertions(+), 108 deletions(-)
PATCH 1/8 xfs_scrub: fix missing scrub coverage for broken inodes
From: Darrick J. Wong <[email protected]>
If INUMBERS says that an inode is allocated, but BULKSTAT skips over the inode and BULKSTAT_SINGLE errors out when loading the inumber, there are two possibilities: One, we're racing with ifree; or two, the inode is corrupt and iget failed.
When this happens, the scrub_scan_all_inodes code will insert a dummy bulkstat record with all fields zeroed except bs_ino and bs_blksize. Hence the use of i_mode switches in phase3 to schedule file content scrubbing are not entirely correct -- bs_mode==0 means "type unknown", which ought to mean "schedule all scrubbers".
Unfortunately, the current code doesn't do that, so instead we schedule no content scrubs. If the broken file was actually a directory, we fail to check the directory contents for further corruptions.
Found by using fuzzing with xfs/385 and core.format = 0.
--- scrub/phase3.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)
scrub/phase3.c +17 -4
@@ -166,16 +166,29 @@ scrub_inode(if (error) goto out; -if (S_ISLNK (bstat->bs_mode)) { +/* +* Check file data contents, e.g. symlink and directory entries. +* +* Note: bs_mode==0 occurs when inumbers says an inode is allocated, +* bulkstat skips the inode, and bulkstat_single errors out when +* loading the inode. This could be due to racing with ifree, but it +* could be a corrupt inode. Either way, schedule all the data fork +* content scrubbers. Better to have them return -ENOENT than miss +* some coverage. +*/ +if (S_ISLNK (bstat->bs_mode) || !bstat->bs_mode) {/* Check symlink contents. */ error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_SYMLINK , &alist); - }else if (S_ISDIR (bstat->bs_mode)) { +if (error) + goto out; + } +if (S_ISDIR (bstat->bs_mode) || !bstat->bs_mode) {/* Check the directory entries. */ error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_DIR , &alist); +if (error) + goto out; } -if (error) - goto out;/* Check all the extended attributes. */ error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_XATTR , &alist);
Re: [PATCH 1/8] xfs_scrub: fix missing scrub coverage for broken inodes
Looks good:
PATCH 2/8 xfs_scrub: collapse trivial superblock scrub helpers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Remove the trivial primary super scrub helper function since it makes tracing code paths difficult and will become annoying in the patches that follow.
--- scrub/phase2.c | 9 +++++---- scrub/scrub.c | 16 +--------------- scrub/scrub.h | 3 ++- 3 files changed, 8 insertions(+), 20 deletions(-)
scrub/phase2.c +5 -4
@@ -166,12 +166,13 @@ phase2_func( }/* -* In case we ever use the primary super scrubber to perform fs -* upgrades (followed by a full scrub), do that before we launch -* anything else. +* Scrub primary superblock. This will be useful if we ever need to +* hook a filesystem-wide pre-scrub activity (e.g. enable filesystem +* upgrades) off of the sb 0 scrubber (which currently does nothing). +* If errors occur, this function will log them and return nonzero. */ action_list_init (&alist); - ret =scrub_primary_super (ctx, &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB ,0 , &alist);if (ret) goto out_wq; ret = action_list_process(ctx, -1, &alist,
scrub/scrub.c +1 -15
@@ -259,7 +259,7 @@ scrub_save_repair(* Returns 0 for success. If errors occur, this function will log them and * return a positive error code. */ -static int +int scrub_meta_type( struct scrub_ctx *ctx, unsigned int type,
@@ -325,20 +325,6 @@ scrub_group(return 0 ; } -/* -* Scrub primary superblock. This will be useful if we ever need to hook -* a filesystem-wide pre-scrub activity off of the sb 0 scrubber (which -* currently does nothing). If errors occur, this function will log them and -* return nonzero. -*/ -int -scrub_primary_super ( -struct scrub_ctx *ctx, -struct action_list *alist) -{ -return scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB ,0 , alist); -} -/* Scrub each AG's header blocks. */ int scrub_ag_headers(
scrub/scrub.h +2 -1
@@ -17,7 +17,6 @@ enum check_outcome {struct action_item ;void scrub_report_preen_triggers (struct scrub_ctx *ctx); -int scrub_primary_super (struct scrub_ctx *ctx,struct action_list *alist);int scrub_ag_headers (struct scrub_ctx *ctx,xfs_agnumber_t agno,struct action_list *alist); int scrub_ag_metadata(struct scrub_ctx *ctx, xfs_agnumber_t agno,
@@ -30,6 +29,8 @@ int scrub_fs_counters(struct scrub_ctx *ctx, struct action_list *alist);int scrub_quotacheck (struct scrub_ctx *ctx,struct action_list *alist);int scrub_nlinks (struct scrub_ctx *ctx,struct action_list *alist);int scrub_clean_health (struct scrub_ctx *ctx,struct action_list *alist); +int scrub_meta_type (struct scrub_ctx *ctx,unsigned int type, +xfs_agnumber_t agno,struct action_list *alist);bool can_scrub_fs_metadata (struct scrub_ctx *ctx);bool can_scrub_inode (struct scrub_ctx *ctx);
Re: [PATCH 2/8] xfs_scrub: collapse trivial superblock scrub helpers
Looks good:
PATCH 3/8 xfs_scrub: get rid of trivial fs metadata scanner helpers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Get rid of these pointless wrappers.
--- scrub/phase1.c | 2 +- scrub/phase4.c | 9 +++++---- scrub/phase5.c | 15 +++++++-------- scrub/scrub.c | 36 ------------------------------------ scrub/scrub.h | 4 ---- 5 files changed, 13 insertions(+), 53 deletions(-)
scrub/phase1.c +1 -1
@@ -61,7 +61,7 @@ report_to_kernel(return 0 ;action_list_init (&alist); - ret =scrub_clean_health (ctx, &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY ,0 , &alist);if (ret)return ret;
scrub/phase4.c +5 -4
@@ -136,14 +136,14 @@ phase4_func( goto maybe_trim;/* -* Check the summary counters early. Normally we do this during phase -* seven, but some of the cross-referencing requires fairly accurate +* Check the resource usage counters early. Normally we do this during +* phase 7, but some of the cross-referencing requires fairly accurate * summary counters. Check and try to repair them now to minimize the * chance that repairs of primary metadata fail due to secondary * metadata. If repairs fails, we'll come back during phase 7. */ action_list_init (&alist); - ret =scrub_fs_counters (ctx, &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS ,0 , &alist);if (ret)return ret;
@@ -158,7 +158,8 @@ phase4_func(return ret;if (fsgeom.sick &XFS_FSOP_GEOM_SICK_QUOTACHECK ) { - ret =scrub_quotacheck (ctx, &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_QUOTACHECK ,0 , + &alist);if (ret)return ret; }
scrub/phase5.c +7 -8
@@ -384,12 +384,10 @@ check_fs_label(return error; } -typedef int (*fs_scan_item_fn )(struct scrub_ctx *,struct action_list *); -struct fs_scan_item {struct action_list alist;bool *abortedp; -fs_scan_item_fn scrub_fn; +unsigned int scrub_type; };/* Run one full-fs scan scrubber in this thread. */
@@ -414,7 +412,7 @@ fs_scan_worker(nanosleep (&tv,NULL ); } - ret = item->scrub_fn (ctx, &item->alist); + ret =scrub_meta_type (ctx, item->scrub_type,0 , &item->alist);if (ret) {str_liberror (ctx, ret,_ ("checking fs scan metadata" )); *item->abortedp = true;
@@ -440,7 +438,7 @@ queue_fs_scan(struct workqueue *wq, bool *abortedp, xfs_agnumber_t nr, - fs_scan_item_fn scrub_fn) + unsigned int scrub_type) { struct fs_scan_item *item;struct scrub_ctx *ctx = wq->wq_ctx;
@@ -453,7 +451,7 @@ queue_fs_scan(return ret; }action_list_init (&item->alist); - item->scrub_fn = scrub_fn; + item->scrub_type = scrub_type; item->abortedp = abortedp; ret = -workqueue_add (wq, fs_scan_worker, nr, item);
@@ -485,14 +483,15 @@ run_kernel_fs_scan_scrubbers(* The nlinks scanner is much faster than quotacheck because it only * walks directories, so we start it first. */ - ret =queue_fs_scan (&wq_fs_scan, &aborted, nr, scrub_nlinks); + ret =queue_fs_scan (&wq_fs_scan, &aborted, nr,XFS_SCRUB_TYPE_NLINKS );if (ret) goto wait;if (nr_threads >1 ) nr++; - ret =queue_fs_scan (&wq_fs_scan, &aborted, nr, scrub_quotacheck); + ret =queue_fs_scan (&wq_fs_scan, &aborted, nr, +XFS_SCRUB_TYPE_QUOTACHECK );if (ret) goto wait;
scrub/scrub.c +0 -36
@@ -366,42 +366,6 @@ scrub_summary_metadata(return scrub_group (ctx,XFROG_SCRUB_GROUP_SUMMARY ,0 , alist); } -/* Scrub /only/ the superblock summary counters. */ -int -scrub_fs_counters ( -struct scrub_ctx *ctx, -struct action_list *alist) -{ -return scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS ,0 , alist); -} - -/* Scrub /only/ the quota counters. */ -int -scrub_quotacheck ( -struct scrub_ctx *ctx, -struct action_list *alist) -{ -return scrub_meta_type (ctx,XFS_SCRUB_TYPE_QUOTACHECK ,0 , alist); -} - -/* Scrub /only/ the file link counters. */ -int -scrub_nlinks ( -struct scrub_ctx *ctx, -struct action_list *alist) -{ -return scrub_meta_type (ctx,XFS_SCRUB_TYPE_NLINKS ,0 , alist); -} - -/* Update incore health records if we were clean. */ -int -scrub_clean_health ( -struct scrub_ctx *ctx, -struct action_list *alist) -{ -return scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY ,0 , alist); -} -/* How many items do we have to check? */ unsigned int scrub_estimate_ag_work(
scrub/scrub.h +0 -4
@@ -25,10 +25,6 @@ int scrub_fs_metadata(struct scrub_ctx *ctx, unsigned int scrub_type, struct action_list *alist);int scrub_iscan_metadata (struct scrub_ctx *ctx,struct action_list *alist);int scrub_summary_metadata (struct scrub_ctx *ctx,struct action_list *alist); -int scrub_fs_counters (struct scrub_ctx *ctx,struct action_list *alist); -int scrub_quotacheck (struct scrub_ctx *ctx,struct action_list *alist); -int scrub_nlinks (struct scrub_ctx *ctx,struct action_list *alist); -int scrub_clean_health (struct scrub_ctx *ctx,struct action_list *alist);int scrub_meta_type (struct scrub_ctx *ctx,unsigned int type,xfs_agnumber_t agno,struct action_list *alist);
Re: [PATCH 3/8] xfs_scrub: get rid of trivial fs metadata scanner helpers
Looks good:
PATCH 4/8 xfs_scrub: split up the mustfix repairs and difficulty assessment functions
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, action_list_find_mustfix does two things -- it figures out which repairs must be tried during phase 2 to enable the inode scan in phase 3; and it figures out if xfs_scrub should warn about secondary and primary metadata corruption that might make repair difficult.
Split these into separate functions to make each more coherent. A long time from now we'll need this to enable warnings about difficult rt repairs, but for now this is merely a code cleanup.
--- scrub/phase2.c | 15 +++++++-------- scrub/repair.c | 38 +++++++++++++++++++++++++++----------- scrub/repair.h | 10 +++++++--- 3 files changed, 41 insertions(+), 22 deletions(-)
scrub/phase2.c +7 -8
@@ -42,9 +42,8 @@ scan_ag_metadata(struct scan_ctl *sctl = arg;struct action_list alist;struct action_list immediate_alist; -unsigned long long broken_primaries; -unsigned long long broken_secondaries;char descr[DESCR_BUFSZ ]; +unsigned int difficulty;int ret;if (sctl->aborted)
@@ -79,12 +78,12 @@ scan_ag_metadata(* the inobt from rmapbt data, but if the rmapbt is broken even * at this early phase then we are sunk. */ - broken_secondaries =0 ; - broken_primaries =0 ; -action_list_find_mustfix (&alist, &immediate_alist, - &broken_primaries, &broken_secondaries); -if (broken_secondaries && !debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) { -if (broken_primaries) + difficulty =action_list_difficulty (&alist); +action_list_find_mustfix (&alist, &immediate_alist); + +if ((difficulty &REPAIR_DIFFICULTY_SECONDARY ) && + !debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) { +if (difficulty &REPAIR_DIFFICULTY_PRIMARY )str_info (ctx, descr,_ ("Corrupt primary and secondary block mapping metadata." ));else
scrub/repair.c +27 -11
@@ -290,9 +290,7 @@ xfs_action_item_compare(void action_list_find_mustfix (struct action_list *alist, -struct action_list *immediate_alist, -unsigned long long *broken_primaries, -unsigned long long *broken_secondaries) +struct action_list *immediate_alist) {struct action_item *n;struct action_item *aitem;
@@ -301,25 +299,43 @@ action_list_find_mustfix(if (!(aitem->flags &XFS_SCRUB_OFLAG_CORRUPT ))continue ;switch (aitem->type) { -case XFS_SCRUB_TYPE_RMAPBT : - (*broken_secondaries)++; -break ;case XFS_SCRUB_TYPE_FINOBT :case XFS_SCRUB_TYPE_INOBT : alist->nr--;list_move_tail (&aitem->list, &immediate_alist->list); immediate_alist->nr++; - fallthrough; +break ; + } + } +} + +/* Determine if primary or secondary metadata are inconsistent. */ +unsigned int +action_list_difficulty ( +const struct action_list *alist) +{ +struct action_item *aitem, *n; +unsigned int ret =0 ; + +list_for_each_entry_safe (aitem, n, &alist->list, list) { +if (!(aitem->flags &XFS_SCRUB_OFLAG_CORRUPT )) +continue ; + +switch (aitem->type) { +case XFS_SCRUB_TYPE_RMAPBT : + ret |=REPAIR_DIFFICULTY_SECONDARY ; +break ; +case XFS_SCRUB_TYPE_FINOBT : +case XFS_SCRUB_TYPE_INOBT :case XFS_SCRUB_TYPE_BNOBT :case XFS_SCRUB_TYPE_CNTBT :case XFS_SCRUB_TYPE_REFCNTBT : - (*broken_primaries)++; -break ; -default : -abort (); + ret |=REPAIR_DIFFICULTY_PRIMARY ;break ; } } + +return ret; }/*
scrub/repair.h +7 -3
@@ -28,9 +28,13 @@ void action_list_discard(struct action_list *alist);void action_list_splice (struct action_list *dest,struct action_list *src);void action_list_find_mustfix (struct action_list *actions, -struct action_list *immediate_alist, -unsigned long long *broken_primaries, -unsigned long long *broken_secondaries); +struct action_list *immediate_alist); + +/* Primary metadata is corrupt */ +#define REPAIR_DIFFICULTY_PRIMARY (1U << 0) +/* Secondary metadata is corrupt */ +#define REPAIR_DIFFICULTY_SECONDARY (1U << 1) +unsigned int action_list_difficulty (const struct action_list *actions);/* * Only ask the kernel to repair this object if the kernel directly told us it
Re: [PATCH 4/8] xfs_scrub: split up the mustfix repairs and difficulty assessment functions
Looks good:
PATCH 5/8 xfs_scrub: add missing repair types to the mustfix and difficulty assessment
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a few scrub types that ought to trigger a mustfix (such as AGI corruption) and all the AG space metadata to the repair difficulty assessment.
--- scrub/repair.c | 7 +++++++ 1 file changed, 7 insertions(+)
scrub/repair.c +7 -0
@@ -299,6 +299,7 @@ action_list_find_mustfix(if (!(aitem->flags &XFS_SCRUB_OFLAG_CORRUPT ))continue ;switch (aitem->type) { +case XFS_SCRUB_TYPE_AGI :case XFS_SCRUB_TYPE_FINOBT :case XFS_SCRUB_TYPE_INOBT : alist->nr--;
@@ -325,11 +326,17 @@ action_list_difficulty(case XFS_SCRUB_TYPE_RMAPBT : ret |=REPAIR_DIFFICULTY_SECONDARY ;break ; +case XFS_SCRUB_TYPE_SB : +case XFS_SCRUB_TYPE_AGF : +case XFS_SCRUB_TYPE_AGFL : +case XFS_SCRUB_TYPE_AGI :case XFS_SCRUB_TYPE_FINOBT :case XFS_SCRUB_TYPE_INOBT :case XFS_SCRUB_TYPE_BNOBT :case XFS_SCRUB_TYPE_CNTBT :case XFS_SCRUB_TYPE_REFCNTBT : +case XFS_SCRUB_TYPE_RTBITMAP : +case XFS_SCRUB_TYPE_RTSUM : ret |=REPAIR_DIFFICULTY_PRIMARY ;break ; }
Re: [PATCH 5/8] xfs_scrub: add missing repair types to the mustfix and difficulty assessment
Looks good:
PATCH 6/8 xfs_scrub: any inconsistency in metadata should trigger difficulty warnings
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Any inconsistency in the space metadata can be a sign that repairs will be difficult, so set off the warning if there were cross referencing problems too.
--- scrub/repair.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scrub/repair.c +3 -1
@@ -319,7 +319,9 @@ action_list_difficulty(unsigned int ret =0 ;list_for_each_entry_safe (aitem, n, &alist->list, list) { -if (!(aitem->flags &XFS_SCRUB_OFLAG_CORRUPT )) +if (!(aitem->flags & (XFS_SCRUB_OFLAG_CORRUPT | +XFS_SCRUB_OFLAG_XCORRUPT | +XFS_SCRUB_OFLAG_XFAIL )))continue ;switch (aitem->type) {
Re: [PATCH 6/8] xfs_scrub: any inconsistency in metadata should trigger difficulty warnings
Looks good:
PATCH 7/8 xfs_scrub: warn about difficult repairs to rt and quota metadata
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Warn the user if there are problems with the rt or quota metadata that might make repairs difficult. For now there aren't any corruption conditions that would trigger this, but we don't want to leave a gap.
--- scrub/phase2.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-)
scrub/phase2.c +25 -12
@@ -31,6 +31,25 @@ struct scan_ctl {bool aborted; }; +/* Warn about the types of mutual inconsistencies that may make repairs hard. */ +static inline void +warn_repair_difficulties ( +struct scrub_ctx *ctx, +unsigned int difficulty, +const char *descr) +{ +if (!(difficulty &REPAIR_DIFFICULTY_SECONDARY )) +return ; +if (debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) +return ; + +if (difficulty &REPAIR_DIFFICULTY_PRIMARY ) +str_info (ctx, descr,_ ("Corrupt primary and secondary metadata." )); +else +str_info (ctx, descr,_ ("Corrupt secondary metadata." )); +str_info (ctx, descr,_ ("Filesystem might not be repairable." )); +} +/* Scrub each AG's metadata btrees. */ static void scan_ag_metadata(
@@ -80,18 +99,7 @@ scan_ag_metadata(*/ difficulty =action_list_difficulty (&alist);action_list_find_mustfix (&alist, &immediate_alist); - -if ((difficulty &REPAIR_DIFFICULTY_SECONDARY ) && - !debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) { -if (difficulty &REPAIR_DIFFICULTY_PRIMARY ) -str_info (ctx, descr, -_ ("Corrupt primary and secondary block mapping metadata." )); -else -str_info (ctx, descr, -_ ("Corrupt secondary block mapping metadata." )); -str_info (ctx, descr, -_ ("Filesystem might not be repairable." )); - } +warn_repair_difficulties (ctx, difficulty, descr);/* Repair (inode) btree damage. */ ret =action_list_process_or_defer (ctx, agno, &immediate_alist);
@@ -115,6 +123,7 @@ scan_fs_metadata(struct action_list alist;struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx;struct scan_ctl *sctl = arg; +unsigned int difficulty;int ret;if (sctl->aborted)
@@ -127,6 +136,10 @@ scan_fs_metadata( goto out; } +/* Complain about metadata corruptions that might not be fixable. */ + difficulty =action_list_difficulty (&alist); +warn_repair_difficulties (ctx, difficulty, xfrog_scrubbers[type].descr); +action_list_defer (ctx,0 , &alist); out:
Re: [PATCH 7/8] xfs_scrub: warn about difficult repairs to rt and quota metadata
Looks good:
PATCH 8/8 xfs_scrub: enable users to bump information messages to warnings
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a -o iwarn option that enables users to specify that informational messages (such as incomplete scans, or confusing names) should be treated as warnings.
--- man/man8/xfs_scrub.8 | 19 +++++++++++++++++++ scrub/common.c | 2 ++ scrub/xfs_scrub.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- scrub/xfs_scrub.h | 1 + 4 files changed, 66 insertions(+), 1 deletion(-)
man/man8/xfs_scrub.8 +19 -0
@@ -85,6 +85,25 @@ Search this file for mounted filesystems instead of /etc/mtab. .B \-n Only check filesystem metadata. Do not repair or optimize anything. +.HP +.B \-o +.I subopt\c +[\c +.B =\c +.IR value ] +.BR +Override what the program might conclude about the filesystem +if left to its own devices. +.IP +The +.IR subopt ions +supported are: +.RS 1.0i +.TP +.BI iwarn +Treat informational messages as warnings. +This will result in a nonzero return code, and a higher logging level. +.RE .TP .BI \-T Print timing and memory usage information for each phase.
scrub/common.c +2 -0
@@ -110,6 +110,8 @@ __str_out(/* print strerror or format of choice but not both */ assert (!(error && format)); +if (level ==S_INFO && info_is_warning) + level =S_WARN ;if (level >=S_INFO ) stream = stdout;
scrub/xfs_scrub.c +44 -1
69 lines not on this page · raw message
scrub/xfs_scrub.h +1 -0
@@ -22,6 +22,7 @@ extern bool stderr_isatty;extern bool stdout_isatty;extern bool is_service;extern bool use_force_rebuild; +extern bool info_is_warning;enum scrub_mode {SCRUB_MODE_DRY_RUN ,
Re: [PATCH 8/8] xfs_scrub: enable users to bump information messages to warnings
Looks good:
PATCH 28/40 xfs_scrub: track data dependencies for repairs
In reply to Darrick J. WongHi all,
Certain kinds of XFS metadata depend on the correctness of lower level metadata. For example, directory indexes depends on the directory data fork, which in turn depend on the directory inode to be correct. The current scrub code does not strictly preserve these dependencies if it has to defer a repair until phase 4, because phase 4 prioritizes repairs by type (corruption, then cross referencing, and then preening) and loses the ordering of in the previous phases. This leads to absurd things like trying to repair a directory before repairing its corrupted fork, which is absurd.
To solve this problem, introduce a repair ticket structure to track all the repairs pending for a principal object (inode, AG, etc). This reduces memory requirements if an object requires more than one type of repair and makes it very easy to track the data dependencies between sub-objects of a principal object. Repair dependencies between object types (e.g. bnobt before inodes) must still be encoded statically into phase 4.
A secondary benefit of this new ticket structure is that we can decide to attempt a repair of an object A that was flagged for a cross referencing error during the scan if a different object B depends on A but only B showed definitive signs of corruption.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps
--- libfrog/scrub.c | 1 scrub/phase1.c | 9 - scrub/phase2.c | 46 ++-- scrub/phase3.c | 77 ++++--- scrub/phase4.c | 17 +- scrub/phase5.c | 9 - scrub/phase7.c | 9 - scrub/repair.c | 530 +++++++++++++++++++++++++++++++++---------------- scrub/repair.h | 47 +++- scrub/scrub.c | 136 ++++++------- scrub/scrub.h | 108 ++++++++-- scrub/scrub_private.h | 37 +++ 12 files changed, 664 insertions(+), 362 deletions(-)
PATCH 1/9 xfs_scrub: track repair items by principal, not by individual repairs
From: Darrick J. Wong <[email protected]>
Create a new structure to track scrub and repair state by principal filesystem object (e.g. ag number or inode number/generation) so that we can more easily examine and ensure that we satisfy repair order dependencies. This transposition will eventually enable bulk scrub operations and will also save a lot of memory if a given object needs a lot of work.
--- scrub/phase1.c | 4 ++ scrub/phase2.c | 14 ++++++-- scrub/phase3.c | 19 ++++++----- scrub/phase4.c | 6 ++-- scrub/phase5.c | 5 ++- scrub/phase7.c | 4 ++ scrub/scrub.c | 68 ++++++++++++++++++++++++++++++++-------- scrub/scrub.h | 83 +++++++++++++++++++++++++++++++++++++++++++++---- scrub/scrub_private.h | 19 +++++++++++ 9 files changed, 185 insertions(+), 37 deletions(-)
scrub/phase1.c +3 -1
@@ -52,6 +52,7 @@ static intreport_to_kernel ( struct scrub_ctx *ctx) { +struct scrub_item sri;struct action_list alist;int ret;
@@ -60,8 +61,9 @@ report_to_kernel( ctx->warnings_found)return 0 ; +scrub_item_init_fs (&sri);action_list_init (&alist); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY ,0 , &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY ,0 , &alist, &sri);if (ret)return ret;
scrub/phase2.c +10 -4
@@ -57,6 +57,7 @@ scan_ag_metadata(xfs_agnumber_t agno, void *arg) { + struct scrub_item sri;struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx;struct scan_ctl *sctl = arg;struct action_list alist;
@@ -68,6 +69,7 @@ scan_ag_metadata(if (sctl->aborted)return ; +scrub_item_init_ag (&sri, agno);action_list_init (&alist);action_list_init (&immediate_alist);snprintf (descr,DESCR_BUFSZ ,_ ("AG %u" ), agno);
@@ -76,7 +78,7 @@ scan_ag_metadata(* First we scrub and fix the AG headers, because we need * them to work well enough to check the AG btrees. */ - ret =scrub_ag_headers (ctx, agno, &alist); + ret =scrub_ag_headers (ctx, agno, &alist, &sri);if (ret) goto err;
@@ -86,7 +88,7 @@ scan_ag_metadata( goto err;/* Now scrub the AG btrees. */ - ret =scrub_ag_metadata (ctx, agno, &alist); + ret =scrub_ag_metadata (ctx, agno, &alist, &sri);if (ret) goto err;
@@ -120,6 +122,7 @@ scan_fs_metadata(xfs_agnumber_t type, void *arg) { + struct scrub_item sri;struct action_list alist;struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx;struct scan_ctl *sctl = arg;
@@ -129,8 +132,9 @@ scan_fs_metadata(if (sctl->aborted) goto out; +scrub_item_init_fs (&sri);action_list_init (&alist); - ret =scrub_fs_metadata (ctx, type, &alist); + ret =scrub_fs_metadata (ctx, type, &alist, &sri);if (ret) { sctl->aborted = true; goto out;
@@ -162,6 +166,7 @@ phase2_func( .rbm_done = false, };struct action_list alist; +struct scrub_item sri;const struct xfrog_scrub_descr *sc = xfrog_scrubbers;xfs_agnumber_t agno;unsigned int type;
@@ -183,8 +188,9 @@ phase2_func(* upgrades) off of the sb 0 scrubber (which currently does nothing). * If errors occur, this function will log them and return nonzero. */ +scrub_item_init_ag (&sri,0 );action_list_init (&alist); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB ,0 , &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB ,0 , &alist, &sri);if (ret) goto out_wq; ret = action_list_process(ctx, -1, &alist,
scrub/phase3.c +11 -8
@@ -105,12 +105,14 @@ scrub_inode( void *arg) {struct action_list alist; +struct scrub_item sri;struct scrub_inode_ctx *ictx = arg;struct ptcounter *icount = ictx->icount;xfs_agnumber_t agno;int fd =-1 ;int error; +scrub_item_init_file (&sri, bstat);action_list_init (&alist); agno =cvt_ino_to_agno (&ctx->mnt, bstat->bs_ino);background_sleep ();
@@ -143,7 +145,7 @@ scrub_inode( fd =scrub_open_handle (handle);/* Scrub the inode. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_INODE , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_INODE , &alist, &sri);if (error) goto out;
@@ -152,13 +154,13 @@ scrub_inode( goto out;/* Scrub all block mappings. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTD , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTD , &alist, &sri);if (error) goto out; - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTA , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTA , &alist, &sri);if (error) goto out; - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTC , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTC , &alist, &sri);if (error) goto out;
@@ -179,24 +181,25 @@ scrub_inode(if (S_ISLNK (bstat->bs_mode) || !bstat->bs_mode) {/* Check symlink contents. */ error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_SYMLINK , - &alist); + &alist, &sri);if (error) goto out; }if (S_ISDIR (bstat->bs_mode) || !bstat->bs_mode) {/* Check the directory entries. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_DIR , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_DIR , &alist, + &sri);if (error) goto out; }/* Check all the extended attributes. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_XATTR , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_XATTR , &alist, &sri);if (error) goto out;/* Check parent pointers. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_PARENT , &alist); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_PARENT , &alist, &sri);if (error) goto out;
scrub/phase4.c +4 -2
@@ -130,6 +130,7 @@ phase4_func( {struct xfs_fsop_geom fsgeom;struct action_list alist; +struct scrub_item sri;int ret;if (!have_action_items (ctx))
@@ -142,8 +143,9 @@ phase4_func(* chance that repairs of primary metadata fail due to secondary * metadata. If repairs fails, we'll come back during phase 7. */ +scrub_item_init_fs (&sri);action_list_init (&alist); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS ,0 , &alist); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS ,0 , &alist, &sri);if (ret)return ret;
@@ -159,7 +161,7 @@ phase4_func(if (fsgeom.sick &XFS_FSOP_GEOM_SICK_QUOTACHECK ) { ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_QUOTACHECK ,0 , - &alist); + &alist, &sri);if (ret)return ret; }
scrub/phase5.c +4 -1
@@ -385,6 +385,7 @@ check_fs_label( }struct fs_scan_item { +struct scrub_item sri;struct action_list alist;bool *abortedp;unsigned int scrub_type;
@@ -412,7 +413,8 @@ fs_scan_worker(nanosleep (&tv,NULL ); } - ret =scrub_meta_type (ctx, item->scrub_type,0 , &item->alist); + ret =scrub_meta_type (ctx, item->scrub_type,0 , &item->alist, + &item->sri);if (ret) {str_liberror (ctx, ret,_ ("checking fs scan metadata" )); *item->abortedp = true;
@@ -450,6 +452,7 @@ queue_fs_scan(str_liberror (ctx, ret,_ ("setting up fs scan" ));return ret; } +scrub_item_init_fs (&item->sri);action_list_init (&item->alist); item->scrub_type = scrub_type; item->abortedp = abortedp;
scrub/phase7.c +3 -1
@@ -99,6 +99,7 @@ phase7_func( struct scrub_ctx *ctx) {struct summary_counts totalcount = {0 }; +struct scrub_item sri;struct action_list alist;struct ptvar *ptvar;unsigned long long used_data;
@@ -117,8 +118,9 @@ phase7_func(int error;/* Check and fix the summary metadata. */ +scrub_item_init_fs (&sri);action_list_init (&alist); - error =scrub_summary_metadata (ctx, &alist); + error =scrub_summary_metadata (ctx, &alist, &sri);if (error)return error; error = action_list_process(ctx, -1, &alist,
scrub/scrub.c +55 -13
144 lines not on this page · raw message
scrub/scrub.h +76 -7
100 lines not on this page · raw message
scrub/scrub_private.h +19 -0
@@ -52,4 +52,23 @@ static inline bool needs_repair(struct xfs_scrub_metadata *sm)void scrub_warn_incomplete_scrub (struct scrub_ctx *ctx,struct descr *dsc,struct xfs_scrub_metadata *meta); +/* Scrub item functions */ + +static inline void +scrub_item_save_state ( +struct scrub_item *sri, +unsigned int scrub_type, +unsigned int scrub_flags) +{ + sri->sri_state[scrub_type] = scrub_flags &SCRUB_ITEM_REPAIR_ANY ; +} + +static inline void +scrub_item_clean_state ( +struct scrub_item *sri, +unsigned int scrub_type) +{ + sri->sri_state[scrub_type] =0 ; +} +#endif /* XFS_SCRUB_SCRUB_PRIVATE_H_ */
Re: [PATCH 1/9] xfs_scrub: track repair items by principal, not by individual repairs
Looks good:
PATCH 2/9 xfs_scrub: use repair_item to direct repair activities
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that the new scrub_item tracks the state of any filesystem object needing any kind of repair, use it to drive filesystem repairs and updates to the in-kernel health status when repair finishes.
--- scrub/phase1.c | 2 scrub/phase2.c | 24 ++-- scrub/phase3.c | 57 ++++---- scrub/phase4.c | 7 - scrub/phase5.c | 2 scrub/phase7.c | 3 scrub/repair.c | 381 +++++++++++++++++++++++++++++++------------------------- scrub/repair.h | 45 +++++-- scrub/scrub.c | 44 ------ scrub/scrub.h | 12 -- 10 files changed, 298 insertions(+), 279 deletions(-)
scrub/phase1.c +1 -1
@@ -71,7 +71,7 @@ report_to_kernel(* Complain if we cannot fail the clean bill of health, unless we're * just testing repairs. */ -if (action_list_length (&alist) >0 && +if (repair_item_count_needsrepair (&sri) !=0 && !debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) {str_info (ctx,_ ("Couldn't upload clean bill of health." ),NULL );action_list_discard (&alist);
scrub/phase2.c +15 -9
@@ -58,6 +58,7 @@ scan_ag_metadata( void *arg) {struct scrub_item sri; +struct scrub_item fix_now;struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx;struct scan_ctl *sctl = arg;struct action_list alist;
@@ -83,7 +84,7 @@ scan_ag_metadata( goto err;/* Repair header damage. */ - ret =action_list_process_or_defer (ctx, agno, &alist); + ret =repair_item_corruption (ctx, &sri);if (ret) goto err;
@@ -99,17 +100,19 @@ scan_ag_metadata(* the inobt from rmapbt data, but if the rmapbt is broken even * at this early phase then we are sunk. */ - difficulty =action_list_difficulty (&alist); -action_list_find_mustfix (&alist, &immediate_alist); + difficulty =repair_item_difficulty (&sri); +repair_item_mustfix (&sri, &fix_now);warn_repair_difficulties (ctx, difficulty, descr);/* Repair (inode) btree damage. */ - ret =action_list_process_or_defer (ctx, agno, &immediate_alist); + ret =repair_item_corruption (ctx, &fix_now);if (ret) goto err;/* Everything else gets fixed during phase 4. */ -action_list_defer (ctx, agno, &alist); + ret =repair_item_defer (ctx, &sri); +if (ret) + goto err;return ; err: sctl->aborted = true;
@@ -141,10 +144,14 @@ scan_fs_metadata( }/* Complain about metadata corruptions that might not be fixable. */ - difficulty =action_list_difficulty (&alist); + difficulty =repair_item_difficulty (&sri);warn_repair_difficulties (ctx, difficulty, xfrog_scrubbers[type].descr); -action_list_defer (ctx,0 , &alist); + ret =repair_item_defer (ctx, &sri); +if (ret) { + sctl->aborted = true; + goto out; + } out:if (type ==XFS_SCRUB_TYPE_RTBITMAP ) {
@@ -193,8 +200,7 @@ phase2_func( ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB ,0 , &alist, &sri);if (ret) goto out_wq; - ret =action_list_process (ctx,-1 , &alist, -XRM_FINAL_WARNING |XRM_NOPROGRESS ); + ret =repair_item_completely (ctx, &sri);if (ret) goto out_wq;
scrub/phase3.c +29 -28
115 lines not on this page · raw message
scrub/phase4.c +3 -4
@@ -40,7 +40,7 @@ repair_ag(/* Repair anything broken until we fail to make progress. */ do { - ret = action_list_process(ctx, -1, alist, flags); + ret = action_list_process(ctx, alist, flags); if (ret) { *aborted = true; return;
@@ -55,7 +55,7 @@ repair_ag(/* Try once more, but this time complain if we can't fix things. */ flags |=XRM_FINAL_WARNING ; - ret =action_list_process (ctx,-1 , alist, flags); + ret =action_list_process (ctx, alist, flags);if (ret) *aborted = true; }
@@ -167,8 +167,7 @@ phase4_func( }/* Repair counters before starting on the rest. */ - ret =action_list_process (ctx,-1 , &alist, -XRM_REPAIR_ONLY |XRM_NOPROGRESS ); + ret =repair_item_corruption (ctx, &sri);if (ret)return ret;action_list_discard (&alist);
scrub/phase5.c +1 -1
@@ -421,7 +421,7 @@ fs_scan_worker( goto out; } - ret =action_list_process (ctx, ctx->mnt.fd, &item->alist, + ret =action_list_process (ctx, &item->alist,XRM_FINAL_WARNING |XRM_NOPROGRESS );if (ret) {str_liberror (ctx, ret,_ ("repairing fs scan metadata" ));
scrub/phase7.c +1 -2
@@ -123,8 +123,7 @@ phase7_func( error =scrub_summary_metadata (ctx, &alist, &sri);if (error)return error; - error =action_list_process (ctx,-1 , &alist, -XRM_FINAL_WARNING |XRM_NOPROGRESS ); + error =repair_item_completely (ctx, &sri);if (error)return error;
scrub/repair.c +211 -170
529 lines not on this page · raw message
scrub/repair.h +35 -10
68 lines not on this page · raw message
scrub/scrub.c +2 -42
68 lines not on this page · raw message
scrub/scrub.h +0 -12
@@ -14,8 +14,6 @@ enum check_outcome { CHECK_RETRY, /* repair failed, try again later */ }; -struct action_item ; -/* * These flags record the metadata object state that the kernel returned. * We want to remember if the object was corrupt, if the cross-referencing
@@ -110,14 +108,4 @@ int scrub_file(struct scrub_ctx *ctx, int fd, const struct xfs_bulkstat *bstat, unsigned int type, struct action_list *alist, struct scrub_item *sri); -/* Repair parameters are the scrub inputs and retry count. */ -struct action_item { -struct list_head list; -__u64 ino; -__u32 type; -__u32 flags; -__u32 gen; -__u32 agno; -}; -#endif /* XFS_SCRUB_SCRUB_H_ */
Re: [PATCH 2/9] xfs_scrub: use repair_item to direct repair activities
Looks good:
PATCH 3/9 xfs_scrub: remove action lists from phaseX code
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we track repair schedules by filesystem object (and not individual repairs) we can get rid of all the onstack list heads and whatnot in the phaseX code.
--- scrub/phase1.c | 5 +---- scrub/phase2.c | 16 ++++------------ scrub/phase3.c | 19 ++++++++----------- scrub/phase4.c | 8 ++------ scrub/phase5.c | 8 ++------ scrub/phase7.c | 4 +--- scrub/scrub.c | 37 ++++++++++++++++++++----------------- scrub/scrub.h | 16 +++++----------- 8 files changed, 43 insertions(+), 70 deletions(-)
scrub/phase1.c +1 -4
@@ -53,7 +53,6 @@ report_to_kernel( struct scrub_ctx *ctx) {struct scrub_item sri; -struct action_list alist;int ret; if (!ctx->scrub_setup_succeeded || ctx->corruptions_found ||
@@ -62,8 +61,7 @@ report_to_kernel(return 0 ;scrub_item_init_fs (&sri); -action_list_init (&alist); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY ,0 , &alist, &sri); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY , &sri);if (ret)return ret;
@@ -74,7 +72,6 @@ report_to_kernel(if (repair_item_count_needsrepair (&sri) !=0 && !debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) {str_info (ctx,_ ("Couldn't upload clean bill of health." ),NULL ); -action_list_discard (&alist); }return 0 ;
scrub/phase2.c +4 -12
@@ -61,8 +61,6 @@ scan_ag_metadata(struct scrub_item fix_now;struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx;struct scan_ctl *sctl = arg; -struct action_list alist; -struct action_list immediate_alist;char descr[DESCR_BUFSZ ];unsigned int difficulty;int ret;
@@ -71,15 +69,13 @@ scan_ag_metadata(return ;scrub_item_init_ag (&sri, agno); -action_list_init (&alist); -action_list_init (&immediate_alist);snprintf (descr,DESCR_BUFSZ ,_ ("AG %u" ), agno);/* * First we scrub and fix the AG headers, because we need * them to work well enough to check the AG btrees. */ - ret =scrub_ag_headers (ctx, agno, &alist, &sri); + ret =scrub_ag_headers (ctx, &sri);if (ret) goto err;
@@ -89,7 +85,7 @@ scan_ag_metadata( goto err;/* Now scrub the AG btrees. */ - ret =scrub_ag_metadata (ctx, agno, &alist, &sri); + ret =scrub_ag_metadata (ctx, &sri);if (ret) goto err;
@@ -126,7 +122,6 @@ scan_fs_metadata( void *arg) {struct scrub_item sri; -struct action_list alist;struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx;struct scan_ctl *sctl = arg;unsigned int difficulty;
@@ -136,8 +131,7 @@ scan_fs_metadata( goto out;scrub_item_init_fs (&sri); -action_list_init (&alist); - ret =scrub_fs_metadata (ctx, type, &alist, &sri); + ret =scrub_fs_metadata (ctx, type, &sri);if (ret) { sctl->aborted = true; goto out;
@@ -172,7 +166,6 @@ phase2_func( .aborted = false, .rbm_done = false, }; -struct action_list alist;struct scrub_item sri;const struct xfrog_scrub_descr *sc = xfrog_scrubbers;xfs_agnumber_t agno;
@@ -196,8 +189,7 @@ phase2_func(* If errors occur, this function will log them and return nonzero. */ scrub_item_init_ag (&sri,0 ); -action_list_init (&alist); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB ,0 , &alist, &sri); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB , &sri);if (ret) goto out_wq; ret =repair_item_completely (ctx, &sri);
scrub/phase3.c +8 -11
@@ -107,7 +107,6 @@ scrub_inode(struct xfs_bulkstat *bstat, void *arg) { - struct action_list alist; struct scrub_item sri;struct scrub_inode_ctx *ictx = arg;struct ptcounter *icount = ictx->icount;
@@ -115,7 +114,6 @@ scrub_inode(int error;scrub_item_init_file (&sri, bstat); -action_list_init (&alist);background_sleep ();/*
@@ -146,7 +144,7 @@ scrub_inode( fd =scrub_open_handle (handle);/* Scrub the inode. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_INODE , &alist, &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_INODE , &sri);if (error) goto out;
@@ -155,13 +153,13 @@ scrub_inode( goto out;/* Scrub all block mappings. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTD , &alist, &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTD , &sri);if (error) goto out; - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTA , &alist, &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTA , &sri);if (error) goto out; - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTC , &alist, &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_BMBTC , &sri);if (error) goto out;
@@ -182,25 +180,24 @@ scrub_inode(if (S_ISLNK (bstat->bs_mode) || !bstat->bs_mode) {/* Check symlink contents. */ error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_SYMLINK , - &alist, &sri); + &sri);if (error) goto out; }if (S_ISDIR (bstat->bs_mode) || !bstat->bs_mode) {/* Check the directory entries. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_DIR , &alist, - &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_DIR , &sri);if (error) goto out; }/* Check all the extended attributes. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_XATTR , &alist, &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_XATTR , &sri);if (error) goto out;/* Check parent pointers. */ - error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_PARENT , &alist, &sri); + error =scrub_file (ctx, fd, bstat,XFS_SCRUB_TYPE_PARENT , &sri);if (error) goto out;
scrub/phase4.c +2 -6
@@ -129,7 +129,6 @@ phase4_func( struct scrub_ctx *ctx) {struct xfs_fsop_geom fsgeom; -struct action_list alist;struct scrub_item sri;int ret;
@@ -144,8 +143,7 @@ phase4_func(* metadata. If repairs fails, we'll come back during phase 7. */ scrub_item_init_fs (&sri); -action_list_init (&alist); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS ,0 , &alist, &sri); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS , &sri);if (ret)return ret;
@@ -160,8 +158,7 @@ phase4_func(return ret;if (fsgeom.sick &XFS_FSOP_GEOM_SICK_QUOTACHECK ) { - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_QUOTACHECK ,0 , - &alist, &sri); + ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_QUOTACHECK , &sri);if (ret)return ret; }
@@ -170,7 +167,6 @@ phase4_func( ret =repair_item_corruption (ctx, &sri);if (ret)return ret; -action_list_discard (&alist); ret =repair_everything (ctx);if (ret)
scrub/phase5.c +2 -6
@@ -386,7 +386,6 @@ check_fs_label(struct fs_scan_item {struct scrub_item sri; -struct action_list alist;bool *abortedp;unsigned int scrub_type; };
@@ -413,16 +412,14 @@ fs_scan_worker(nanosleep (&tv,NULL ); } - ret =scrub_meta_type (ctx, item->scrub_type,0 , &item->alist, - &item->sri); + ret =scrub_meta_type (ctx, item->scrub_type, &item->sri);if (ret) {str_liberror (ctx, ret,_ ("checking fs scan metadata" )); *item->abortedp = true; goto out; } - ret =action_list_process (ctx, &item->alist, -XRM_FINAL_WARNING |XRM_NOPROGRESS ); + ret =repair_item_completely (ctx, &item->sri);if (ret) {str_liberror (ctx, ret,_ ("repairing fs scan metadata" )); *item->abortedp = true;
@@ -453,7 +450,6 @@ queue_fs_scan(return ret; }scrub_item_init_fs (&item->sri); -action_list_init (&item->alist); item->scrub_type = scrub_type; item->abortedp = abortedp;
scrub/phase7.c +1 -3
@@ -100,7 +100,6 @@ phase7_func( {struct summary_counts totalcount = {0 };struct scrub_item sri; -struct action_list alist;struct ptvar *ptvar;unsigned long long used_data;unsigned long long used_rt;
@@ -119,8 +118,7 @@ phase7_func(/* Check and fix the summary metadata. */ scrub_item_init_fs (&sri); -action_list_init (&alist); - error =scrub_summary_metadata (ctx, &alist, &sri); + error =scrub_summary_metadata (ctx, &sri);if (error)return error; error =repair_item_completely (ctx, &sri);
scrub/scrub.c +20 -17
@@ -219,6 +219,7 @@ _("Optimizations of %s are possible."), _(xfrog_scrubbers[i].descr));/* * Scrub a single XFS_SCRUB_TYPE_*, saving corruption reports for later. +* Do not call this function to repair file metadata. * * Returns 0 for success. If errors occur, this function will log them and * return a positive error code.
@@ -227,18 +228,29 @@ intscrub_meta_type ( struct scrub_ctx *ctx, unsigned int type, - xfs_agnumber_t agno, - struct action_list *alist, struct scrub_item *sri) { struct xfs_scrub_metadata meta = { .sm_type = type, - .sm_agno = agno, };enum check_outcome fix;background_sleep (); +switch (xfrog_scrubbers[type].group) { +case XFROG_SCRUB_GROUP_AGHEADER : +case XFROG_SCRUB_GROUP_PERAG : + meta.sm_agno = sri->sri_agno; +break ; +case XFROG_SCRUB_GROUP_FS : +case XFROG_SCRUB_GROUP_SUMMARY : +case XFROG_SCRUB_GROUP_NONE : +break ; +default : +assert (0 ); +break ; + } +/* Check the item. */ fix =xfs_check_metadata (ctx, &ctx->mnt, &meta, false);progress_add (1 );
@@ -267,8 +279,6 @@ static boolscrub_group ( struct scrub_ctx *ctx, enum xfrog_scrub_group group, - xfs_agnumber_t agno, - struct action_list *alist, struct scrub_item *sri) { const struct xfrog_scrub_descr *sc;
@@ -281,7 +291,7 @@ scrub_group(if (sc->group != group)continue ; - ret =scrub_meta_type (ctx, type, agno, alist, sri); + ret =scrub_meta_type (ctx, type, sri);if (ret)return ret; }
@@ -293,22 +303,18 @@ scrub_group(int scrub_ag_headers (struct scrub_ctx *ctx, -xfs_agnumber_t agno, -struct action_list *alist,struct scrub_item *sri) { -return scrub_group (ctx,XFROG_SCRUB_GROUP_AGHEADER , agno, alist, sri); +return scrub_group (ctx,XFROG_SCRUB_GROUP_AGHEADER , sri); }/* Scrub each AG's metadata btrees. */ int scrub_ag_metadata (struct scrub_ctx *ctx, -xfs_agnumber_t agno, -struct action_list *alist,struct scrub_item *sri) { -return scrub_group (ctx,XFROG_SCRUB_GROUP_PERAG , agno, alist, sri); +return scrub_group (ctx,XFROG_SCRUB_GROUP_PERAG , sri); }/* Scrub whole-filesystem metadata. */
@@ -316,22 +322,20 @@ intscrub_fs_metadata ( struct scrub_ctx *ctx, unsigned int type, - struct action_list *alist, struct scrub_item *sri) { ASSERT(xfrog_scrubbers[type].group == XFROG_SCRUB_GROUP_FS); -return scrub_meta_type (ctx, type,0 , alist, sri); +return scrub_meta_type (ctx, type, sri); }/* Scrub all FS summary metadata. */ int scrub_summary_metadata (struct scrub_ctx *ctx, -struct action_list *alist,struct scrub_item *sri) { -return scrub_group (ctx,XFROG_SCRUB_GROUP_SUMMARY ,0 , alist, sri); +return scrub_group (ctx,XFROG_SCRUB_GROUP_SUMMARY , sri); }/* How many items do we have to check? */
@@ -393,7 +397,6 @@ scrub_file(int fd, const struct xfs_bulkstat *bstat, unsigned int type, - struct action_list *alist, struct scrub_item *sri) { struct xfs_scrub_metadata meta = {0};
scrub/scrub.h +5 -11
@@ -80,18 +80,13 @@ void scrub_item_dump(struct scrub_item *sri, unsigned int group_mask, const char *tag);void scrub_report_preen_triggers (struct scrub_ctx *ctx); -int scrub_ag_headers (struct scrub_ctx *ctx,xfs_agnumber_t agno, -struct action_list *alist,struct scrub_item *sri); -int scrub_ag_metadata (struct scrub_ctx *ctx,xfs_agnumber_t agno, -struct action_list *alist,struct scrub_item *sri); +int scrub_ag_headers (struct scrub_ctx *ctx,struct scrub_item *sri); +int scrub_ag_metadata (struct scrub_ctx *ctx,struct scrub_item *sri);int scrub_fs_metadata (struct scrub_ctx *ctx,unsigned int scrub_type, -struct action_list *alist,struct scrub_item *sri); -int scrub_iscan_metadata (struct scrub_ctx *ctx,struct action_list *alist, -struct scrub_item *sri); -int scrub_summary_metadata (struct scrub_ctx *ctx,struct action_list *alist,struct scrub_item *sri); +int scrub_iscan_metadata (struct scrub_ctx *ctx,struct scrub_item *sri); +int scrub_summary_metadata (struct scrub_ctx *ctx,struct scrub_item *sri);int scrub_meta_type (struct scrub_ctx *ctx,unsigned int type, -xfs_agnumber_t agno,struct action_list *alist,struct scrub_item *sri);bool can_scrub_fs_metadata (struct scrub_ctx *ctx);
@@ -105,7 +100,6 @@ bool can_repair(struct scrub_ctx *ctx);bool can_force_rebuild (struct scrub_ctx *ctx);int scrub_file (struct scrub_ctx *ctx,int fd,const struct xfs_bulkstat *bstat, -unsigned int type,struct action_list *alist, -struct scrub_item *sri); +unsigned int type,struct scrub_item *sri);#endif /* XFS_SCRUB_SCRUB_H_ */
Re: [PATCH 3/9] xfs_scrub: remove action lists from phaseX code
Looks good:
PATCH 4/9 xfs_scrub: remove scrub_metadata_file
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Collapse this function with scrub_meta_type.
--- scrub/phase2.c | 2 +- scrub/scrub.c | 12 ------------ scrub/scrub.h | 2 -- 3 files changed, 1 insertion(+), 15 deletions(-)
scrub/phase2.c +1 -1
@@ -131,7 +131,7 @@ scan_fs_metadata( goto out;scrub_item_init_fs (&sri); - ret =scrub_fs_metadata (ctx, type, &sri); + ret =scrub_meta_type (ctx, type, &sri);if (ret) { sctl->aborted = true; goto out;
scrub/scrub.c +0 -12
@@ -317,18 +317,6 @@ scrub_ag_metadata(return scrub_group (ctx,XFROG_SCRUB_GROUP_PERAG , sri); } -/* Scrub whole-filesystem metadata. */ -int -scrub_fs_metadata ( -struct scrub_ctx *ctx, -unsigned int type, -struct scrub_item *sri) -{ -ASSERT (xfrog_scrubbers[type].group ==XFROG_SCRUB_GROUP_FS ); - -return scrub_meta_type (ctx, type, sri); -} -/* Scrub all FS summary metadata. */ int scrub_summary_metadata(
scrub/scrub.h +0 -2
@@ -82,8 +82,6 @@ void scrub_item_dump(struct scrub_item *sri, unsigned int group_mask,void scrub_report_preen_triggers (struct scrub_ctx *ctx);int scrub_ag_headers (struct scrub_ctx *ctx,struct scrub_item *sri);int scrub_ag_metadata (struct scrub_ctx *ctx,struct scrub_item *sri); -int scrub_fs_metadata (struct scrub_ctx *ctx,unsigned int scrub_type, -struct scrub_item *sri);int scrub_iscan_metadata (struct scrub_ctx *ctx,struct scrub_item *sri);int scrub_summary_metadata (struct scrub_ctx *ctx,struct scrub_item *sri); int scrub_meta_type(struct scrub_ctx *ctx, unsigned int type,
Re: [PATCH 4/9] xfs_scrub: remove scrub_metadata_file
Looks good:
PATCH 5/9 xfs_scrub: boost the repair priority of dependencies of damaged items
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
In XFS, certain types of metadata objects depend on the correctness of lower level metadata objects. For example, directory blocks are stored in the data fork of directory files, which means that any issues with the inode core and the data fork should be dealt with before we try to repair a directory.
xfs_scrub prioritises repairs by the severity of what the kernel scrub function reports -- anything directly observed to be corrupt get repaired first, then anything that had trouble with cross referencing, and finally anything that was correct but could be further optimised. Returning to the above example, if a directory data fork mapping offset is off by a bit flip, scrub will mark that as failing cross referencing, but it'll mark the directory as corrupt. Repair should check out the mapping problem before it tackles the directory.
Do this by embedding a dependency table and using it to boost the priority of the repair_item fields as needed.
--- libfrog/scrub.c | 1 scrub/repair.c | 99 ++++++++++++++++++++++++++++++++++++++++++++++++- scrub/scrub.h | 12 ++++++ scrub/scrub_private.h | 8 ++++ 4 files changed, 117 insertions(+), 3 deletions(-)
libfrog/scrub.c +1 -0
@@ -150,6 +150,7 @@ const struct xfrog_scrub_descr xfrog_scrubbers[XFS_SCRUB_TYPE_NR] = { .group = XFROG_SCRUB_GROUP_NONE, }, }; +#undef DEP/* Invoke the scrub ioctl. Returns zero or negative error code. */ int
scrub/repair.c +96 -3
152 lines not on this page · raw message
scrub/scrub.h +12 -0
@@ -14,6 +14,14 @@ enum check_outcome { CHECK_RETRY, /* repair failed, try again later */ }; +/* +* This flag boosts the repair priority of a scrub item when a dependent scrub +* item is scheduled for repair. Use a separate flag to preserve the +* corruption state that we got from the kernel. Priority boost is cleared the +* next time xfs_repair_metadata is called. +*/ +#define SCRUB_ITEM_BOOST_REPAIR (1 << 0) +/* * These flags record the metadata object state that the kernel returned. * We want to remember if the object was corrupt, if the cross-referencing
@@ -31,6 +39,10 @@ enum check_outcome { SCRUB_ITEM_XFAIL | \ SCRUB_ITEM_XCORRUPT) +/* Cross-referencing failures only. */ +#define SCRUB_ITEM_REPAIR_XREF (SCRUB_ITEM_XFAIL | \ + SCRUB_ITEM_XCORRUPT) +struct scrub_item {/* * Information we need to call the scrub and repair ioctls. Per-AG
scrub/scrub_private.h +8 -0
@@ -71,4 +71,12 @@ scrub_item_clean_state( sri->sri_state[scrub_type] = 0; } +static inline bool +scrub_item_type_boosted ( +struct scrub_item *sri, +unsigned int scrub_type) +{ +return sri->sri_state[scrub_type] &SCRUB_ITEM_BOOST_REPAIR ; +} +#endif /* XFS_SCRUB_SCRUB_PRIVATE_H_ */
Re: [PATCH 5/9] xfs_scrub: boost the repair priority of dependencies of damaged items
Looks good:
PATCH 6/9 xfs_scrub: clean up repair_item_difficulty a little
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Document the flags handling in repair_item_difficulty.
--- scrub/repair.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
scrub/repair.c +13 -3
@@ -340,6 +340,15 @@ repair_item_mustfix( } } +/* +* These scrub item states correspond to metadata that is inconsistent in some +* way and must be repaired. If too many metadata objects share these states, +* this can make repairs difficult. +*/ +#define HARDREPAIR_STATES (SCRUB_ITEM_CORRUPT | \ + SCRUB_ITEM_XCORRUPT | \ + SCRUB_ITEM_XFAIL) +/* Determine if primary or secondary metadata are inconsistent. */ unsigned int repair_item_difficulty(
@@ -349,9 +358,10 @@ repair_item_difficulty(unsigned int ret =0 ;foreach_scrub_type (scrub_type) { -if (!(sri->sri_state[scrub_type] & (XFS_SCRUB_OFLAG_CORRUPT | -XFS_SCRUB_OFLAG_XCORRUPT | -XFS_SCRUB_OFLAG_XFAIL ))) +unsigned int state; + + state = sri->sri_state[scrub_type] &HARDREPAIR_STATES ; +if (!state)continue ;switch (scrub_type) {
Re: [PATCH 6/9] xfs_scrub: clean up repair_item_difficulty a little
Looks good:
PATCH 7/9 xfs_scrub: check dependencies of a scrub type before repairing
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we have a map of a scrub type to its dependent scrub types, use this information to avoid trying to fix higher level metadata before the lower levels have passed.
--- scrub/repair.c | 32 ++++++++++++++++++++++++++++++++ scrub/scrub.h | 5 +++++ 2 files changed, 37 insertions(+)
scrub/repair.c +32 -0
@@ -497,6 +497,29 @@ action_list_process(return ret; } +/* Decide if the dependent scrub types of the given scrub type are ok. */ +static bool +repair_item_dependencies_ok ( +const struct scrub_item *sri, +unsigned int scrub_type) +{ +unsigned int dep_mask = repair_deps[scrub_type]; +unsigned int b; + +for (b =0 ; dep_mask && b <XFS_SCRUB_TYPE_NR ; b++, dep_mask >>=1 ) { +if (!(dep_mask &1 )) +continue ; +/* +* If this lower level object also needs repair, we can't fix +* the higher level item. +*/ +if (sri->sri_state[b] &SCRUB_ITEM_NEEDSREPAIR ) +return false; + } + +return true; +} +/* * For a given filesystem object, perform all repairs of a given class * (corrupt, xcorrupt, xfail, preen) if the repair item says it's needed.
@@ -536,6 +559,15 @@ repair_item_class(if (!(sri->sri_state[scrub_type] & repair_mask))continue ; +/* +* Don't try to repair higher level items if their lower-level +* dependencies haven't been verified, unless this is our last +* chance to fix things without complaint. +*/ +if (!(flags &XRM_FINAL_WARNING ) && + !repair_item_dependencies_ok (sri, scrub_type)) +continue ; + fix =xfs_repair_metadata (ctx, xfdp, scrub_type, sri, flags);switch (fix) {case CHECK_DONE :
scrub/scrub.h +5 -0
@@ -43,6 +43,11 @@ enum check_outcome {#define SCRUB_ITEM_REPAIR_XREF (SCRUB_ITEM_XFAIL | \ SCRUB_ITEM_XCORRUPT) +/* Mask of bits signalling that a piece of metadata requires attention. */ +#define SCRUB_ITEM_NEEDSREPAIR (SCRUB_ITEM_CORRUPT | \ + SCRUB_ITEM_XFAIL | \ + SCRUB_ITEM_XCORRUPT) +struct scrub_item {/* * Information we need to call the scrub and repair ioctls. Per-AG
Re: [PATCH 7/9] xfs_scrub: check dependencies of a scrub type before repairing
Looks good:
PATCH 8/9 xfs_scrub: retry incomplete repairs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If a repair says it didn't do anything on account of not being able to complete a scan of the metadata, retry the repair a few times; if even that doesn't work, we can delay it to phase 4.
--- scrub/repair.c | 15 ++++++++++++++- scrub/scrub.c | 3 +-- scrub/scrub_private.h | 10 ++++++++++ 3 files changed, 25 insertions(+), 3 deletions(-)
scrub/repair.c +14 -1
@@ -58,6 +58,7 @@ xfs_repair_metadata(struct xfs_scrub_metadata oldm;DEFINE_DESCR (dsc, ctx, format_scrub_descr);bool repair_only; +unsigned int tries =0 ;int error;/*
@@ -99,6 +100,7 @@ xfs_repair_metadata(str_info (ctx,descr_render (&dsc),_ ("Attempting optimization." )); +retry: error = -xfrog_scrub_metadata (xfdp, &meta);switch (error) {case 0 :
@@ -179,9 +181,20 @@ _("Read-only filesystem; cannot make changes."));return CHECK_DONE ; } +/* +* If the kernel says the repair was incomplete or that there was a +* cross-referencing discrepancy but no obvious corruption, we'll try +* the repair again, just in case the fs was busy. Only retry so many +* times. +*/ +if (want_retry (&meta) && tries <10 ) { + tries++; + goto retry; + } +if (repair_flags &XRM_FINAL_WARNING )scrub_warn_incomplete_scrub (ctx, &dsc, &meta); -if (needs_repair (&meta)) { +if (needs_repair (&meta) ||is_incomplete (&meta)) {/* * Still broken; if we've been told not to complain then we * just requeue this and try again later. Otherwise we
scrub/scrub.c +1 -2
@@ -137,8 +137,7 @@ _("Filesystem is shut down, aborting."));* we'll try the scan again, just in case the fs was busy. * Only retry so many times. */ -if (tries <10 && (is_incomplete (meta) || - (xref_disagrees (meta) && !is_corrupt (meta)))) { +if (want_retry (meta) && tries <10 ) { tries++; goto retry; }
scrub/scrub_private.h +10 -0
@@ -49,6 +49,16 @@ static inline bool needs_repair(struct xfs_scrub_metadata *sm)return is_corrupt (sm) ||xref_disagrees (sm); } +/* +* We want to retry an operation if the kernel says it couldn't complete the +* scan/repair; or if there were cross-referencing problems but the object was +* not obviously corrupt. +*/ +static inline bool want_retry (struct xfs_scrub_metadata *sm) +{ +return is_incomplete (sm) || (xref_disagrees (sm) && !is_corrupt (sm)); +} +void scrub_warn_incomplete_scrub (struct scrub_ctx *ctx,struct descr *dsc,struct xfs_scrub_metadata *meta);
Re: [PATCH 8/9] xfs_scrub: retry incomplete repairs
Looks good:
PATCH 9/9 xfs_scrub: remove unused action_list fields
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Remove some fields since we don't need them anymore.
--- scrub/repair.c | 5 ----- scrub/repair.h | 2 -- 2 files changed, 7 deletions(-)
scrub/repair.c +0 -5
@@ -432,7 +432,6 @@ action_list_discard(struct action_item *n;list_for_each_entry_safe (aitem, n, &alist->list, list) { - alist->nr--;list_del (&aitem->list);free (aitem); }
@@ -453,8 +452,6 @@ action_list_init( struct action_list *alist) {INIT_LIST_HEAD (&alist->list); - alist->nr =0 ; - alist->sorted = false; }/* Number of pending repairs in this list. */
@@ -478,8 +475,6 @@ action_list_add( struct action_item *aitem) {list_add_tail (&aitem->list, &alist->list); - alist->nr++; - alist->sorted = false; }/* Repair everything on this list. */
scrub/repair.h +0 -2
@@ -8,8 +8,6 @@struct action_list {struct list_head list; -unsigned long long nr; -bool sorted; };struct action_item ;
Re: [PATCH 9/9] xfs_scrub: remove unused action_list fields
Looks good:
PATCH 29/40 xfs_scrub: use scrub_item to track check progress
In reply to Darrick J. WongHi all,
Now that we've introduced tickets to track the status of repairs to a specific principal XFS object (fs, ag, file), use them to track the scrub state of those same objects. Ultimately, we want to make it easy to introduce vectorized repair, where we send a batch of repair requests to the kernel instead of making millions of ioctl calls. For now, however, we'll settle for easier bookkeepping.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking
--- scrub/phase1.c | 3 scrub/phase2.c | 12 +- scrub/phase3.c | 41 ++---- scrub/phase4.c | 16 +- scrub/phase5.c | 5 - scrub/phase7.c | 5 + scrub/repair.c | 71 +++++------ scrub/scrub.c | 321 ++++++++++++++++++++++--------------------------- scrub/scrub.h | 40 ++++-- scrub/scrub_private.h | 14 ++ 10 files changed, 257 insertions(+), 271 deletions(-)
PATCH 1/5 xfs_scrub: start tracking scrub state in scrub_item
From: Darrick J. Wong <[email protected]>
Start using the scrub_item to track which metadata objects need checking by adding a new flag to the scrub_item state set.
--- scrub/phase1.c | 3 + scrub/phase2.c | 12 +++-- scrub/phase3.c | 41 +++++----------- scrub/phase4.c | 16 +++--- scrub/phase5.c | 5 +- scrub/phase7.c | 5 ++ scrub/scrub.c | 147 +++++++++++++++++++------------------------------------- scrub/scrub.h | 28 ++++++++--- 8 files changed, 108 insertions(+), 149 deletions(-)
scrub/phase1.c +2 -1
@@ -61,7 +61,8 @@ report_to_kernel(return 0 ;scrub_item_init_fs (&sri); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_HEALTHY , &sri); +scrub_item_schedule (&sri,XFS_SCRUB_TYPE_HEALTHY ); + ret =scrub_item_check (ctx, &sri);if (ret)return ret;
scrub/phase2.c +8 -4
@@ -75,7 +75,8 @@ scan_ag_metadata(* First we scrub and fix the AG headers, because we need * them to work well enough to check the AG btrees. */ - ret =scrub_ag_headers (ctx, &sri); +scrub_item_schedule_group (&sri,XFROG_SCRUB_GROUP_AGHEADER ); + ret =scrub_item_check (ctx, &sri);if (ret) goto err;
@@ -85,7 +86,8 @@ scan_ag_metadata( goto err;/* Now scrub the AG btrees. */ - ret =scrub_ag_metadata (ctx, &sri); +scrub_item_schedule_group (&sri,XFROG_SCRUB_GROUP_PERAG ); + ret =scrub_item_check (ctx, &sri);if (ret) goto err;
@@ -131,7 +133,8 @@ scan_fs_metadata( goto out;scrub_item_init_fs (&sri); - ret =scrub_meta_type (ctx, type, &sri); +scrub_item_schedule (&sri, type); + ret =scrub_item_check (ctx, &sri);if (ret) { sctl->aborted = true; goto out;
@@ -189,7 +192,8 @@ phase2_func(* If errors occur, this function will log them and return nonzero. */ scrub_item_init_ag (&sri,0 ); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_SB , &sri); +scrub_item_schedule (&sri,XFS_SCRUB_TYPE_SB ); + ret =scrub_item_check (ctx, &sri);if (ret) goto out_wq; ret =repair_item_completely (ctx, &sri);
scrub/phase3.c +13 -28
60 lines not on this page · raw message
scrub/phase4.c +7 -9
@@ -143,9 +143,7 @@ phase4_func(* metadata. If repairs fails, we'll come back during phase 7. */ scrub_item_init_fs (&sri); - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_FSCOUNTERS , &sri); -if (ret) -return ret; +scrub_item_schedule (&sri,XFS_SCRUB_TYPE_FSCOUNTERS );/* * Repair possibly bad quota counts before starting other repairs,
@@ -157,13 +155,13 @@ phase4_func(if (ret)return ret; -if (fsgeom.sick &XFS_FSOP_GEOM_SICK_QUOTACHECK ) { - ret =scrub_meta_type (ctx,XFS_SCRUB_TYPE_QUOTACHECK , &sri); -if (ret) -return ret; - } +if (fsgeom.sick &XFS_FSOP_GEOM_SICK_QUOTACHECK ) +scrub_item_schedule (&sri,XFS_SCRUB_TYPE_QUOTACHECK ); -/* Repair counters before starting on the rest. */ +/* Check and repair counters before starting on the rest. */ + ret =scrub_item_check (ctx, &sri); +if (ret) +return ret; ret =repair_item_corruption (ctx, &sri);if (ret)return ret;
scrub/phase5.c +2 -3
@@ -387,7 +387,6 @@ check_fs_label(struct fs_scan_item {struct scrub_item sri;bool *abortedp; -unsigned int scrub_type; };/* Run one full-fs scan scrubber in this thread. */
@@ -412,7 +411,7 @@ fs_scan_worker(nanosleep (&tv,NULL ); } - ret =scrub_meta_type (ctx, item->scrub_type, &item->sri); + ret =scrub_item_check (ctx, &item->sri);if (ret) {str_liberror (ctx, ret,_ ("checking fs scan metadata" )); *item->abortedp = true;
@@ -450,7 +449,7 @@ queue_fs_scan(return ret; }scrub_item_init_fs (&item->sri); - item->scrub_type = scrub_type; +scrub_item_schedule (&item->sri, scrub_type); item->abortedp = abortedp; ret = -workqueue_add (wq, fs_scan_worker, nr, item);
scrub/phase7.c +4 -1
@@ -10,6 +10,8 @@#include <linux/fsmap.h> #include "libfrog/paths.h" #include "libfrog/ptvar.h" +#include "libfrog/fsgeom.h" +#include "libfrog/scrub.h" #include "list.h" #include "xfs_scrub.h" #include "common.h"
@@ -118,7 +120,8 @@ phase7_func(/* Check and fix the summary metadata. */ scrub_item_init_fs (&sri); - error =scrub_summary_metadata (ctx, &sri); +scrub_item_schedule_group (&sri,XFROG_SCRUB_GROUP_SUMMARY ); + error =scrub_item_check (ctx, &sri);if (error)return error; error =repair_item_completely (ctx, &sri);
scrub/scrub.c +50 -97
205 lines not on this page · raw message
scrub/scrub.h +22 -6
@@ -6,6 +6,8 @@#ifndef XFS_SCRUB_SCRUB_H_ #define XFS_SCRUB_SCRUB_H_ +enum xfrog_scrub_group ; +/* Online scrub and repair. */ enum check_outcome { CHECK_DONE, /* no further processing needed */
@@ -33,6 +35,9 @@ enum check_outcome {#define SCRUB_ITEM_XFAIL (XFS_SCRUB_OFLAG_XFAIL)/* (1 << 3) */ #define SCRUB_ITEM_XCORRUPT (XFS_SCRUB_OFLAG_XCORRUPT)/* (1 << 4) */ +/* This scrub type needs to be checked. */ +#define SCRUB_ITEM_NEEDSCHECK (1 << 5) +/* All of the state flags that we need to prioritize repair work. */ #define SCRUB_ITEM_REPAIR_ANY (SCRUB_ITEM_CORRUPT | \ SCRUB_ITEM_PREEN | \
@@ -96,13 +101,24 @@ scrub_item_init_file(struct scrub_item *sri, const struct xfs_bulkstat *bstat)void scrub_item_dump (struct scrub_item *sri,unsigned int group_mask,const char *tag); +static inline void +scrub_item_schedule (struct scrub_item *sri,unsigned int scrub_type) +{ + sri->sri_state[scrub_type] =SCRUB_ITEM_NEEDSCHECK ; +} + +void scrub_item_schedule_group (struct scrub_item *sri, +enum xfrog_scrub_group group); +int scrub_item_check_file (struct scrub_ctx *ctx,struct scrub_item *sri, +int override_fd); + +static inline int +scrub_item_check (struct scrub_ctx *ctx,struct scrub_item *sri) +{ +return scrub_item_check_file (ctx, sri,-1 ); +} +void scrub_report_preen_triggers (struct scrub_ctx *ctx); -int scrub_ag_headers (struct scrub_ctx *ctx,struct scrub_item *sri); -int scrub_ag_metadata (struct scrub_ctx *ctx,struct scrub_item *sri); -int scrub_iscan_metadata (struct scrub_ctx *ctx,struct scrub_item *sri); -int scrub_summary_metadata (struct scrub_ctx *ctx,struct scrub_item *sri); -int scrub_meta_type (struct scrub_ctx *ctx,unsigned int type, -struct scrub_item *sri);bool can_scrub_fs_metadata (struct scrub_ctx *ctx);bool can_scrub_inode (struct scrub_ctx *ctx);
Re: [PATCH 1/5] xfs_scrub: start tracking scrub state in scrub_item
Looks good:
PATCH 2/5 xfs_scrub: remove enum check_outcome
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Get rid of this enumeration, and just do what we will directly.
--- scrub/repair.c | 56 ++++++++++++++++++++++---------------------------- scrub/scrub.c | 63 +++++++++++++++++++++++++++++--------------------------- scrub/scrub.h | 8 ------- 3 files changed, 58 insertions(+), 69 deletions(-)
scrub/repair.c +25 -31
@@ -46,7 +46,7 @@ static const unsigned int repair_deps[XFS_SCRUB_TYPE_NR] = {#undef DEP/* Repair some metadata. */ -static enum check_outcome +static int xfs_repair_metadata( struct scrub_ctx *ctx, struct xfs_fd *xfdp,
@@ -88,7 +88,7 @@ xfs_repair_metadata( }if (!is_corrupt (&meta) && repair_only) -return CHECK_RETRY ; +return 0 ;memcpy (&oldm, &meta,sizeof (oldm)); oldm.sm_flags = sri->sri_state[scrub_type] &SCRUB_ITEM_REPAIR_ANY ;
@@ -112,12 +112,12 @@ xfs_repair_metadata(if (debug || verbose)str_info (ctx,descr_render (&dsc),_ ("Filesystem is busy, deferring repair." )); -return CHECK_RETRY ; +return 0 ;case ESHUTDOWN :/* Filesystem is already shut down, abort. */ str_error (ctx,descr_render (&dsc),_ ("Filesystem is shut down, aborting." )); -return CHECK_ABORT ; +return ECANCELED ;case ENOTTY :case EOPNOTSUPP :/*
@@ -129,7 +129,7 @@ _("Filesystem is shut down, aborting."));if (is_unoptimized (&oldm) ||debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) {scrub_item_clean_state (sri, scrub_type); -return CHECK_DONE ; +return 0 ; }/* * If we're in no-complain mode, requeue the check for
@@ -140,30 +140,30 @@ _("Filesystem is shut down, aborting."));* again to see if another repair fixed it. */ if (!(repair_flags &XRM_FINAL_WARNING )) -return CHECK_RETRY ; +return 0 ; fallthrough;case EINVAL :/* Kernel doesn't know how to repair this? */ str_corrupt (ctx,descr_render (&dsc),_ ("Don't know how to fix; offline repair required." ));scrub_item_clean_state (sri, scrub_type); -return CHECK_DONE ; +return 0 ;case EROFS :/* Read-only filesystem, can't fix. */ if (verbose || debug ||needs_repair (&oldm))str_error (ctx,descr_render (&dsc),_ ("Read-only filesystem; cannot make changes." )); -return CHECK_ABORT ; +return ECANCELED ;case ENOENT :/* Metadata not present, just skip it. */ scrub_item_clean_state (sri, scrub_type); -return CHECK_DONE ; +return 0 ;case ENOMEM :case ENOSPC :/* Don't care if preen fails due to low resources. */ if (is_unoptimized (&oldm) && !needs_repair (&oldm)) {scrub_item_clean_state (sri, scrub_type); -return CHECK_DONE ; +return 0 ; } fallthrough;default :
@@ -175,10 +175,10 @@ _("Read-only filesystem; cannot make changes."));* trying to repair it, and bail out. */ if (!(repair_flags &XRM_FINAL_WARNING )) -return CHECK_RETRY ; +return 0 ;str_liberror (ctx, error,descr_render (&dsc));scrub_item_clean_state (sri, scrub_type); -return CHECK_DONE ; +return 0 ; }/*
@@ -201,7 +201,7 @@ _("Read-only filesystem; cannot make changes."));* log the error loudly and don't try again. */ if (!(repair_flags &XRM_FINAL_WARNING )) -return CHECK_RETRY ; +return 0 ;str_corrupt (ctx,descr_render (&dsc),_ ("Repair unsuccessful; offline repair required." )); }else if (xref_failed (&meta)) {
@@ -219,7 +219,7 @@ _("Repair unsuccessful; offline repair required."));if (verbose)str_info (ctx,descr_render (&dsc),_ ("Seems correct but cross-referencing failed; will keep checking." )); -return CHECK_RETRY ; +return 0 ; } }else if (meta.sm_flags &XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED ) {if (verbose)
@@ -242,7 +242,7 @@ _("Repair unsuccessful; offline repair required.")); }scrub_item_clean_state (sri, scrub_type); -return CHECK_DONE ; +return 0 ; }/*
@@ -543,6 +543,7 @@ repair_item_class(struct xfs_fd xfd;struct xfs_fd *xfdp = &ctx->mnt;unsigned int scrub_type; +int error =0 ;if (ctx->mode <SCRUB_MODE_REPAIR )return 0 ;
@@ -559,8 +560,6 @@ repair_item_class( }foreach_scrub_type (scrub_type) { -enum check_outcome fix; -if (scrub_excessive_errors (ctx))return ECANCELED ;
@@ -576,22 +575,17 @@ repair_item_class( !repair_item_dependencies_ok(sri, scrub_type)) continue; - fix =xfs_repair_metadata (ctx, xfdp, scrub_type, sri, flags); -switch (fix) { -case CHECK_DONE : -if (!(flags &XRM_NOPROGRESS )) -progress_add (1 ); -continue ; -case CHECK_ABORT : -return ECANCELED ; -case CHECK_RETRY : -continue ; -case CHECK_REPAIR : -abort (); - } + error =xfs_repair_metadata (ctx, xfdp, scrub_type, sri, flags); +if (error) +break ; + +/* Maybe update progress if we fixed the problem. */ +if (!(flags &XRM_NOPROGRESS ) && + !(sri->sri_state[scrub_type] &SCRUB_ITEM_REPAIR_ANY )) +progress_add (1 ); } -return 0 ; +return error; }/*
scrub/scrub.c +33 -30
145 lines not on this page · raw message
scrub/scrub.h +0 -8
@@ -8,14 +8,6 @@enum xfrog_scrub_group ; -/* Online scrub and repair. */ -enum check_outcome { -CHECK_DONE ,/* no further processing needed */ -CHECK_REPAIR ,/* schedule this for repairs */ -CHECK_ABORT ,/* end program */ -CHECK_RETRY ,/* repair failed, try again later */ -}; -/* * This flag boosts the repair priority of a scrub item when a dependent scrub * item is scheduled for repair. Use a separate flag to preserve the
Re: [PATCH 2/5] xfs_scrub: remove enum check_outcome
Looks good:
PATCH 3/5 xfs_scrub: refactor scrub_meta_type out of existence
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Remove this helper function since it's trivial now.
--- scrub/scrub.c | 124 ++++++++++++++++++++++++--------------------------------- 1 file changed, 53 insertions(+), 71 deletions(-)
scrub/scrub.c +53 -71
@@ -82,31 +82,51 @@ static intxfs_check_metadata ( struct scrub_ctx *ctx, struct xfs_fd *xfdp, - struct xfs_scrub_metadata *meta, + unsigned int scrub_type, struct scrub_item *sri) {DEFINE_DESCR (dsc, ctx, format_scrub_descr); +struct xfs_scrub_metadata meta = { };enum xfrog_scrub_group group;unsigned int tries =0 ;int error; - group = xfrog_scrubbers[meta->sm_type].group; +background_sleep (); + + group = xfrog_scrubbers[scrub_type].group; + meta.sm_type = scrub_type; +switch (group) { +case XFROG_SCRUB_GROUP_AGHEADER : +case XFROG_SCRUB_GROUP_PERAG : + meta.sm_agno = sri->sri_agno; +break ; +case XFROG_SCRUB_GROUP_FS : +case XFROG_SCRUB_GROUP_SUMMARY : +case XFROG_SCRUB_GROUP_ISCAN : +case XFROG_SCRUB_GROUP_NONE : +break ; +case XFROG_SCRUB_GROUP_INODE : + meta.sm_ino = sri->sri_ino; + meta.sm_gen = sri->sri_gen; +break ; + } +assert (!debug_tweak_on ("XFS_SCRUB_NO_KERNEL" )); -assert (meta->sm_type <XFS_SCRUB_TYPE_NR ); -descr_set (&dsc, meta); +assert (scrub_type <XFS_SCRUB_TYPE_NR ); +descr_set (&dsc, &meta); -dbg_printf ("check %s flags %xh\n" ,descr_render (&dsc), meta->sm_flags); +dbg_printf ("check %s flags %xh\n" ,descr_render (&dsc), meta.sm_flags); retry: - error = -xfrog_scrub_metadata (xfdp, meta); + error = -xfrog_scrub_metadata (xfdp, &meta);if (debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" ) && !error) - meta->sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ; + meta.sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ;switch (error) {case 0 :/* No operational errors encountered. */ break ;case ENOENT :/* Metadata not present, just skip it. */ -scrub_item_clean_state (sri, meta->sm_type); +scrub_item_clean_state (sri, scrub_type);return 0 ;case ESHUTDOWN :/* FS already crashed, give up. */
@@ -128,12 +148,12 @@ _("Filesystem is shut down, aborting."));* Log it and move on. */ str_liberror (ctx, error,_ ("Kernel bug" )); -scrub_item_clean_state (sri, meta->sm_type); +scrub_item_clean_state (sri, scrub_type);return 0 ;default :/* Operational error. Log it and move on. */ str_liberror (ctx, error,descr_render (&dsc)); -scrub_item_clean_state (sri, meta->sm_type); +scrub_item_clean_state (sri, scrub_type);return 0 ; }
@@ -143,29 +163,29 @@ _("Filesystem is shut down, aborting."));* we'll try the scan again, just in case the fs was busy. * Only retry so many times. */ -if (want_retry (meta) && tries <10 ) { +if (want_retry (&meta) && tries <10 ) { tries++; goto retry; }/* Complain about incomplete or suspicious metadata. */ -scrub_warn_incomplete_scrub (ctx, &dsc, meta); +scrub_warn_incomplete_scrub (ctx, &dsc, &meta);/* * If we need repairs or there were discrepancies, schedule a * repair if desired, otherwise complain. */ -if (is_corrupt (meta) ||xref_disagrees (meta)) { +if (is_corrupt (&meta) ||xref_disagrees (&meta)) {if (ctx->mode <SCRUB_MODE_REPAIR ) {/* Dry-run mode, so log an error and forget it. */ str_corrupt (ctx,descr_render (&dsc),_ ("Repairs are required." )); -scrub_item_clean_state (sri, meta->sm_type); +scrub_item_clean_state (sri, scrub_type);return 0 ; }/* Schedule repairs. */ -scrub_item_save_state (sri, meta->sm_type, meta->sm_flags); +scrub_item_save_state (sri, scrub_type, meta.sm_flags);return 0 ; }
@@ -173,26 +193,26 @@ _("Repairs are required."));* If we could optimize, schedule a repair if desired, * otherwise complain. */ -if (is_unoptimized (meta)) { +if (is_unoptimized (&meta)) {if (ctx->mode !=SCRUB_MODE_REPAIR ) {/* Dry-run mode, so log an error and forget it. */ if (group !=XFROG_SCRUB_GROUP_INODE ) {/* AG or FS metadata, always warn. */ str_info (ctx,descr_render (&dsc),_ ("Optimization is possible." )); - }else if (!ctx->preen_triggers[meta->sm_type]) { + }else if (!ctx->preen_triggers[scrub_type]) {/* File metadata, only warn once per type. */ pthread_mutex_lock (&ctx->lock); -if (!ctx->preen_triggers[meta->sm_type]) - ctx->preen_triggers[meta->sm_type] = true; +if (!ctx->preen_triggers[scrub_type]) + ctx->preen_triggers[scrub_type] = true;pthread_mutex_unlock (&ctx->lock); } -scrub_item_clean_state (sri, meta->sm_type); +scrub_item_clean_state (sri, scrub_type);return 0 ; }/* Schedule optimizations. */ -scrub_item_save_state (sri, meta->sm_type, meta->sm_flags); +scrub_item_save_state (sri, scrub_type, meta.sm_flags);return 0 ; }
@@ -203,13 +223,13 @@ _("Optimization is possible."));* re-examine the object as repairs progress to see if the kernel will * deem it completely consistent at some point. */ -if (xref_failed (meta) && ctx->mode ==SCRUB_MODE_REPAIR ) { -scrub_item_save_state (sri, meta->sm_type, meta->sm_flags); +if (xref_failed (&meta) && ctx->mode ==SCRUB_MODE_REPAIR ) { +scrub_item_save_state (sri, scrub_type, meta.sm_flags);return 0 ; }/* Everything is ok. */ -scrub_item_clean_state (sri, meta->sm_type); +scrub_item_clean_state (sri, scrub_type);return 0 ; }
@@ -233,52 +253,6 @@ _("Optimizations of %s are possible."), _(xfrog_scrubbers[i].descr)); } } -/* -* Scrub a single XFS_SCRUB_TYPE_*, saving corruption reports for later. -* Do not call this function to repair file metadata. -* -* Returns 0 for success. If errors occur, this function will log them and -* return a positive error code. -*/ -static int -scrub_meta_type ( -struct scrub_ctx *ctx, -struct xfs_fd *xfdp, -unsigned int type, -struct scrub_item *sri) -{ -struct xfs_scrub_metadata meta = { - .sm_type = type, - }; -int error; - -background_sleep (); - -switch (xfrog_scrubbers[type].group) { -case XFROG_SCRUB_GROUP_AGHEADER : -case XFROG_SCRUB_GROUP_PERAG : - meta.sm_agno = sri->sri_agno; -break ; -case XFROG_SCRUB_GROUP_FS : -case XFROG_SCRUB_GROUP_SUMMARY : -case XFROG_SCRUB_GROUP_ISCAN : -case XFROG_SCRUB_GROUP_NONE : -break ; -case XFROG_SCRUB_GROUP_INODE : - meta.sm_ino = sri->sri_ino; - meta.sm_gen = sri->sri_gen; -break ; - } - -/* Check the item. */ - error =xfs_check_metadata (ctx, xfdp, &meta, sri); - -if (xfrog_scrubbers[type].group !=XFROG_SCRUB_GROUP_INODE ) -progress_add (1 ); - -return error; -} -/* Schedule scrub for all metadata of a given group. */ void scrub_item_schedule_group(
@@ -321,7 +295,15 @@ scrub_item_check_file(if (!(sri->sri_state[scrub_type] &SCRUB_ITEM_NEEDSCHECK ))continue ; - error =scrub_meta_type (ctx, xfdp, scrub_type, sri); + error =xfs_check_metadata (ctx, xfdp, scrub_type, sri); + +/* +* Progress is counted by the inode for inode metadata; for +* everything else, it's counted for each scrub call. +*/ +if (sri->sri_ino ==-1ULL ) +progress_add (1 ); +if (error)break ; }
Re: [PATCH 3/5] xfs_scrub: refactor scrub_meta_type out of existence
Looks good:
PATCH 4/5 xfs_scrub: hoist repair retry loop to repair_item_class
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
For metadata repair calls, move the ioctl retry and freeze permission tracking into scrub_item. This enables us to move the repair retry loop out of xfs_repair_metadata and into its caller to remove a long backwards jump, and gets us closer to vectorizing scrub calls.
--- scrub/repair.c | 21 ++++++++++++--------- scrub/scrub.c | 32 ++++++++++++++++++++++++++++++-- scrub/scrub.h | 6 ++++++ scrub/scrub_private.h | 14 ++++++++++++++ 4 files changed, 62 insertions(+), 11 deletions(-)
scrub/repair.c +12 -9
@@ -58,7 +58,6 @@ xfs_repair_metadata(struct xfs_scrub_metadata oldm;DEFINE_DESCR (dsc, ctx, format_scrub_descr);bool repair_only; -unsigned int tries =0 ;int error;/*
@@ -100,7 +99,6 @@ xfs_repair_metadata(str_info (ctx,descr_render (&dsc),_ ("Attempting optimization." )); -retry: error = -xfrog_scrub_metadata (xfdp, &meta);switch (error) {case 0 :
@@ -187,10 +185,8 @@ _("Read-only filesystem; cannot make changes."));* the repair again, just in case the fs was busy. Only retry so many * times. */ -if (want_retry (&meta) && tries <10 ) { - tries++; - goto retry; - } +if (want_retry (&meta) &&scrub_item_schedule_retry (sri, scrub_type)) +return 0 ;if (repair_flags &XRM_FINAL_WARNING )scrub_warn_incomplete_scrub (ctx, &dsc, &meta);
@@ -541,6 +537,7 @@ repair_item_class( unsigned int flags) {struct xfs_fd xfd; +struct scrub_item old_sri;struct xfs_fd *xfdp = &ctx->mnt;unsigned int scrub_type;int error =0 ;
@@ -575,9 +572,15 @@ repair_item_class( !repair_item_dependencies_ok(sri, scrub_type)) continue; - error =xfs_repair_metadata (ctx, xfdp, scrub_type, sri, flags); -if (error) -break ; + sri->sri_tries[scrub_type] =SCRUB_ITEM_MAX_RETRIES ; +do { +memcpy (&old_sri, sri,sizeof (old_sri)); + error =xfs_repair_metadata (ctx, xfdp, scrub_type, sri, + flags); +if (error) +return error; + }while (scrub_item_call_kernel_again (sri, scrub_type, + repair_mask, &old_sri));/* Maybe update progress if we fixed the problem. */ if (!(flags & XRM_NOPROGRESS) &&
scrub/scrub.c +30 -2
@@ -268,6 +268,34 @@ scrub_item_schedule_group( } } +/* Decide if we call the kernel again to finish scrub/repair activity. */ +bool +scrub_item_call_kernel_again ( +struct scrub_item *sri, +unsigned int scrub_type, +uint8_t work_mask, +const struct scrub_item *old) +{ +uint8_t statex; + +/* If there's nothing to do, we're done. */ +if (!(sri->sri_state[scrub_type] & work_mask)) +return false; + +/* +* We are willing to go again if the last call had any effect on the +* state of the scrub item that the caller cares about, if the freeze +* flag got set, or if the kernel asked us to try again... +*/ + statex = sri->sri_state[scrub_type] ^ old->sri_state[scrub_type]; +if (statex & work_mask) +return true; +if (sri->sri_tries[scrub_type] != old->sri_tries[scrub_type]) +return true; + +return false; +} +/* Run all the incomplete scans on this scrub principal. */ int scrub_item_check_file(
@@ -383,9 +411,9 @@ scrub_item_dump(unsigned int g =1U << xfrog_scrubbers[i].group;if (g & group_mask) -printf ("[%u]: type '%s' state 0x%x\n" , i, +printf ("[%u]: type '%s' state 0x%x tries %u\n" , i, xfrog_scrubbers[i].name, - sri->sri_state[i]); + sri->sri_state[i], sri->sri_tries[i]); }fflush (stdout); }
scrub/scrub.h +6 -0
@@ -45,6 +45,9 @@ enum xfrog_scrub_group; SCRUB_ITEM_XFAIL | \ SCRUB_ITEM_XCORRUPT) +/* Maximum number of times we'll retry a scrub ioctl call. */ +#define SCRUB_ITEM_MAX_RETRIES 10 +struct scrub_item {/* * Information we need to call the scrub and repair ioctls. Per-AG
@@ -58,6 +61,9 @@ struct scrub_item {/* Scrub item state flags, one for each XFS_SCRUB_TYPE. */ __u8 sri_state[XFS_SCRUB_TYPE_NR ]; + +/* Track scrub and repair call retries for each scrub type. */ +__u8 sri_tries[XFS_SCRUB_TYPE_NR ]; };#define foreach_scrub_type (loopvar) \
scrub/scrub_private.h +14 -0
@@ -89,4 +89,18 @@ scrub_item_type_boosted( return sri->sri_state[scrub_type] & SCRUB_ITEM_BOOST_REPAIR; } +/* Decide if we want to retry this operation and update bookkeeping if yes. */ +static inline bool +scrub_item_schedule_retry (struct scrub_item *sri,unsigned int scrub_type) +{ +if (sri->sri_tries[scrub_type] ==0 ) +return false; + sri->sri_tries[scrub_type]--; +return true; +} + +bool scrub_item_call_kernel_again (struct scrub_item *sri, +unsigned int scrub_type,uint8_t work_mask, +const struct scrub_item *old); +#endif /* XFS_SCRUB_SCRUB_PRIVATE_H_ */
Re: [PATCH 4/5] xfs_scrub: hoist repair retry loop to repair_item_class
Looks good:
PATCH 5/5 xfs_scrub: hoist scrub retry loop to scrub_item_check_file
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
For metadata check calls, use the ioctl retry and freeze permission tracking in scrub_item that we created in the last patch. This enables us to move the check retry loop out of xfs_scrub_metadata and into its caller to remove a long backwards jump, and gets us closer to vectorizing scrub calls.
--- scrub/scrub.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
scrub/scrub.c +12 -7
@@ -88,7 +88,6 @@ xfs_check_metadata(DEFINE_DESCR (dsc, ctx, format_scrub_descr);struct xfs_scrub_metadata meta = { };enum xfrog_scrub_group group; -unsigned int tries =0 ;int error;background_sleep ();
@@ -116,7 +115,7 @@ xfs_check_metadata(descr_set (&dsc, &meta);dbg_printf ("check %s flags %xh\n" ,descr_render (&dsc), meta.sm_flags); -retry: + error = -xfrog_scrub_metadata (xfdp, &meta);if (debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" ) && !error) meta.sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ;
@@ -163,10 +162,8 @@ _("Filesystem is shut down, aborting."));* we'll try the scan again, just in case the fs was busy. * Only retry so many times. */ -if (want_retry (&meta) && tries <10 ) { - tries++; - goto retry; - } +if (want_retry (&meta) &&scrub_item_schedule_retry (sri, scrub_type)) +return 0 ;/* Complain about incomplete or suspicious metadata. */ scrub_warn_incomplete_scrub (ctx, &dsc, &meta);
@@ -304,6 +301,7 @@ scrub_item_check_file( int override_fd) {struct xfs_fd xfd; +struct scrub_item old_sri;struct xfs_fd *xfdp = &ctx->mnt;unsigned int scrub_type;int error;
@@ -323,7 +321,14 @@ scrub_item_check_file(if (!(sri->sri_state[scrub_type] &SCRUB_ITEM_NEEDSCHECK ))continue ; - error =xfs_check_metadata (ctx, xfdp, scrub_type, sri); + sri->sri_tries[scrub_type] =SCRUB_ITEM_MAX_RETRIES ; +do { +memcpy (&old_sri, sri,sizeof (old_sri)); + error =xfs_check_metadata (ctx, xfdp, scrub_type, sri); +if (error) +return error; + }while (scrub_item_call_kernel_again (sri, scrub_type, +SCRUB_ITEM_NEEDSCHECK , &old_sri));/* * Progress is counted by the inode for inode metadata; for
Re: [PATCH 5/5] xfs_scrub: hoist scrub retry loop to scrub_item_check_file
Looks good:
PATCH 30/40 xfs_scrub: improve scheduling of repair items
In reply to Darrick J. WongHi all,
Currently, phase 4 of xfs_scrub uses per-AG repair item lists to schedule repair work across a thread pool. This scheme is suboptimal when most of the repairs involve a single AG because all the work gets dumped on a single pool thread.
Instead, we should create a thread pool with the same number of workers as CPUs, and dispatch individual repair tickets as separate work items to maximize parallelization.
However, we also need to ensure that repairs to space metadata and file metadata are kept in separate queues because file repairs generally depend on correctness of space metadata.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling
--- libfrog/ptvar.c | 9 ++ libfrog/ptvar.h | 4 + scrub/counter.c | 2 scrub/descr.c | 2 scrub/phase1.c | 15 ++- scrub/phase2.c | 23 ++++- scrub/phase3.c | 106 ++++++++++++++------- scrub/phase4.c | 244 ++++++++++++++++++++++++++++++++++++------------- scrub/phase7.c | 2 scrub/read_verify.c | 2 scrub/repair.c | 172 ++++++++++++++++++++++------------- scrub/repair.h | 37 ++++++- scrub/scrub.c | 5 + scrub/scrub.h | 10 ++ scrub/scrub_private.h | 2 scrub/xfs_scrub.h | 3 - 16 files changed, 455 insertions(+), 183 deletions(-)
PATCH 1/4 libfrog: enhance ptvar to support initializer functions
From: Darrick J. Wong <[email protected]>
Modify the per-thread variable code to support passing in an initializer function that will set up each thread's variable space when it is claimed.
--- libfrog/ptvar.c | 9 ++++++++- libfrog/ptvar.h | 4 +++- scrub/counter.c | 2 +- scrub/descr.c | 2 +- scrub/phase7.c | 2 +- scrub/read_verify.c | 2 +- 6 files changed, 15 insertions(+), 6 deletions(-)
libfrog/ptvar.c +8 -1
@@ -26,6 +26,7 @@struct ptvar {pthread_key_t key;pthread_mutex_t lock; +ptvar_init_fn init_fn;size_t nr_used;size_t nr_counters;size_t data_size;
@@ -38,6 +39,7 @@ intptvar_alloc ( size_t nr, size_t size, + ptvar_init_fn init_fn, struct ptvar **pptv) { struct ptvar *ptv;
@@ -58,6 +60,7 @@ ptvar_alloc( ptv->data_size = size; ptv->nr_counters = nr; ptv->nr_used =0 ; + ptv->init_fn = init_fn;memset (ptv->data,0 , nr * size); ret = -pthread_mutex_init (&ptv->lock,NULL );if (ret)
@@ -98,11 +101,15 @@ ptvar_get(if (!p) {pthread_mutex_lock (&ptv->lock);assert (ptv->nr_used < ptv->nr_counters); - p = &ptv->data[(ptv->nr_used++) * ptv->data_size]; + p = &ptv->data[ptv->nr_used * ptv->data_size]; ret = -pthread_setspecific (ptv->key, p);if (ret) goto out_unlock; + ptv->nr_used++;pthread_mutex_unlock (&ptv->lock); + +if (ptv->init_fn) + ptv->init_fn (p); } *retp =0 ;return p;
libfrog/ptvar.h +3 -1
@@ -8,7 +8,9 @@struct ptvar ; -int ptvar_alloc (size_t nr,size_t size,struct ptvar **pptv); +typedef void (*ptvar_init_fn )(void *data); +int ptvar_alloc (size_t nr,size_t size,ptvar_init_fn init_fn, +struct ptvar **pptv);void ptvar_free (struct ptvar *ptv);void *ptvar_get (struct ptvar *ptv,int *ret);
scrub/counter.c +1 -1
@@ -38,7 +38,7 @@ ptcounter_alloc( p =malloc (sizeof (struct ptcounter ));if (!p)return errno; - ret = -ptvar_alloc (nr,sizeof (uint64_t ), &p->var); + ret = -ptvar_alloc (nr,sizeof (uint64_t ),NULL , &p->var);if (ret) {free (p);return ret;
scrub/descr.c +1 -1
@@ -89,7 +89,7 @@ descr_init_phase(int ret;assert (descr_ptvar ==NULL ); - ret = -ptvar_alloc (nr_threads,DESCR_BUFSZ , &descr_ptvar); + ret = -ptvar_alloc (nr_threads,DESCR_BUFSZ ,NULL , &descr_ptvar);if (ret)str_liberror (ctx, ret,_ ("creating description buffer" ));
scrub/phase7.c +1 -1
@@ -136,7 +136,7 @@ phase7_func( } error = -ptvar_alloc (scrub_nproc (ctx),sizeof (struct summary_counts ), - &ptvar); +NULL , &ptvar);if (error) {str_liberror (ctx, error,_ ("setting up block counter" ));return error;
scrub/read_verify.c +1 -1
@@ -120,7 +120,7 @@ read_verify_pool_alloc( rvp->disk = disk; rvp->ioerr_fn = ioerr_fn; ret = -ptvar_alloc (submitter_threads,sizeof (struct read_verify ), - &rvp->rvstate); +NULL , &rvp->rvstate);if (ret) goto out_counter; ret = -workqueue_create(&rvp->wq, (struct xfs_mount *)rvp,
Re: [PATCH 1/4] libfrog: enhance ptvar to support initializer functions
Looks good:
PATCH 2/4 xfs_scrub: improve thread scheduling repair items during phase 4
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
As it stands, xfs_scrub doesn't do a good job of scheduling repair items during phase 4. The repair lists are sharded by AG, and one repair worker is started for each per-AG repair list. Consequently, if one AG requires considerably more work than the others (e.g. inodes are not spread evenly among the AGs) then phase 4 can stall waiting for that one worker thread when there's still plenty of CPU power available.
While our initial assumptions were that repairs would be vanishingly scarce, the reality is that "repairs" can be triggered for optimizations like gaps in the xattr structures, or clearing the inode reflink flag on inodes that no longer share data. In real world testing scenarios, the lack of balance leads to complaints about excessive runtime of xfs_scrub.
To fix these balance problems, we replace the per-AG repair item lists in the scrub context with a single repair item list. Phase 4 will be redesigned as follows:
The repair worker will grab a repair item from the main list, try to repair it, record whether the repair attempt made any progress, and requeue the item if it was not fully fixed. A separate repair scheduler function starts the repair workers, and waits for them all to complete. Requeued repairs are merged back into the main repair list. If we made any forward progress, we'll start another round of repairs with the repair workers. Phase 4 retains the behavior that if the pool stops making forward progress, it will try all the repairs one last time, serially.
To facilitate this new design, phase 2 will queue repairs of space metadata items directly to the main list. Phase 3's worker threads will queue repair items to per-thread lists and splice those lists into the main list at the end.
On a filesystem crafted to put all the inodes in a single AG, this restores xfs_scrub's ability to parallelize repairs. There seems to be a slight performance hit for the evenly-spread case, but avoiding a performance cliff due to an unbalanced fs is more important here.
--- scrub/phase1.c | 8 +- scrub/phase2.c | 23 +++++ scrub/phase3.c | 106 ++++++++++++++++-------- scrub/phase4.c | 230 ++++++++++++++++++++++++++++++++++++++--------------- scrub/repair.c | 135 +++++++++++++++++-------------- scrub/repair.h | 37 +++++++-- scrub/xfs_scrub.h | 2 7 files changed, 367 insertions(+), 174 deletions(-)
scrub/phase1.c +4 -4
@@ -89,7 +89,8 @@ scrub_cleanup(if (error)return error; -action_lists_free (&ctx->action_lists); +action_list_free (&ctx->action_list); +if (ctx->fshandle)free_handle (ctx->fshandle, ctx->fshandle_len);if (ctx->rtdev)
@@ -185,10 +186,9 @@ _("Not an XFS filesystem."));return error; } - error =action_lists_alloc (ctx->mnt.fsgeom.agcount, - &ctx->action_lists); + error =action_list_alloc (&ctx->action_list);if (error) { -str_liberror (ctx, error,_ ("allocating action lists" )); +str_liberror (ctx, error,_ ("allocating repair list" ));return error; }
scrub/phase2.c +21 -2
@@ -50,6 +50,25 @@ warn_repair_difficulties(str_info (ctx, descr,_ ("Filesystem might not be repairable." )); } +/* Add a scrub item that needs more work to fs metadata repair list. */ +static int +defer_fs_repair ( +struct scrub_ctx *ctx, +const struct scrub_item *sri) +{ +struct action_item *aitem =NULL ; +int error; + + error =repair_item_to_action_item (ctx, sri, &aitem); +if (error || !aitem) +return error; + +pthread_mutex_lock (&ctx->lock); +action_list_add (ctx->action_list, aitem); +pthread_mutex_unlock (&ctx->lock); +return 0 ; +} +/* Scrub each AG's metadata btrees. */ static void scan_ag_metadata(
@@ -108,7 +127,7 @@ scan_ag_metadata( goto err;/* Everything else gets fixed during phase 4. */ - ret =repair_item_defer (ctx, &sri); + ret =defer_fs_repair (ctx, &sri);if (ret) goto err;return ;
@@ -144,7 +163,7 @@ scan_fs_metadata( difficulty =repair_item_difficulty (&sri);warn_repair_difficulties (ctx, difficulty, xfrog_scrubbers[type].descr); - ret =repair_item_defer (ctx, &sri); + ret =defer_fs_repair (ctx, &sri);if (ret) { sctl->aborted = true; goto out;
scrub/phase3.c +70 -36
202 lines not on this page · raw message
scrub/phase4.c +167 -63
284 lines not on this page · raw message
scrub/repair.c +74 -61
174 lines not on this page · raw message
scrub/repair.h +30 -7
@@ -12,19 +12,43 @@ struct action_list {struct action_item ; -int action_lists_alloc (size_t nr,struct action_list **listsp); -void action_lists_free (struct action_list **listsp); +int action_list_alloc (struct action_list **listp); +void action_list_free (struct action_list **listp); +static inline void action_list_init (struct action_list *alist) +{ +INIT_LIST_HEAD (&alist->list); +} -void action_list_init (struct action_list *alist); +unsigned long long action_list_length (struct action_list *alist); + +/* Move all the items of @src to the tail of @dst, and reinitialize @src. */ +static inline void +action_list_merge ( +struct action_list *dst, +struct action_list *src) +{ +list_splice_tail_init (&src->list, &dst->list); +} + +struct action_item *action_list_pop (struct action_list *alist); +void action_list_add (struct action_list *alist,struct action_item *aitem);static inline bool action_list_empty (const struct action_list *alist) {return list_empty (&alist->list); } -unsigned long long action_list_length (struct action_list *alist); -void action_list_add (struct action_list *dest,struct action_item *item); -void action_list_discard (struct action_list *alist); +enum tryrepair_outcome { +/* No progress was made on repairs at all. */ +TR_NOPROGRESS =0 , +/* Some progress was made on repairs; try again soon. */ +TR_REQUEUE , +/* Repairs completely successful. */ +TR_REPAIRED , +}; + +int action_item_try_repair (struct scrub_ctx *ctx,struct action_item *aitem, +enum tryrepair_outcome *outcome);void repair_item_mustfix (struct scrub_item *sri,struct scrub_item *fix_now);
@@ -56,7 +80,6 @@ int repair_item(struct scrub_ctx *ctx, struct scrub_item *sri, unsigned int repair_flags);int repair_item_to_action_item (struct scrub_ctx *ctx,const struct scrub_item *sri,struct action_item **aitemp); -int repair_item_defer (struct scrub_ctx *ctx,const struct scrub_item *sri);static inline unsigned int repair_item_count_needsrepair(
scrub/xfs_scrub.h +1 -1
@@ -72,7 +72,7 @@ struct scrub_ctx {/* Mutable scrub state; use lock. */ pthread_mutex_t lock; -struct action_list *action_lists; +struct action_list *action_list;unsigned long long max_errors;unsigned long long runtime_errors;unsigned long long corruptions_found;
Re: [PATCH 2/4] xfs_scrub: improve thread scheduling repair items during phase 4
Looks good:
PATCH 3/4 xfs_scrub: recheck entire metadata objects after corruption repairs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
When we've finished making repairs to some domain of filesystem metadata (file, AG, etc.) to correct an inconsistency, we should recheck all the other metadata types within that domain to make sure that we neither made things worse nor introduced more cross-referencing problems. If we did, requeue the item to make the repairs. If the only changes we made were optimizations, don't bother.
The XFS_SCRUB_TYPE_ values are getting close to the max for a u32, so I chose u64 for sri_selected.
--- scrub/repair.c | 37 +++++++++++++++++++++++++++++++++++++ scrub/scrub.c | 5 +++-- scrub/scrub.h | 10 ++++++++++ scrub/scrub_private.h | 2 ++ 4 files changed, 52 insertions(+), 2 deletions(-)
scrub/repair.c +37 -0
@@ -485,8 +485,10 @@ action_item_try_repair( {struct scrub_item *sri = &aitem->sri;unsigned int before, after; +unsigned int scrub_type;int ret; +BUILD_BUG_ON (sizeof (sri->sri_selected) *NBBY <XFS_SCRUB_TYPE_NR ); before =repair_item_count_needsrepair (sri); ret =repair_item (ctx, sri,0 );
@@ -507,6 +509,41 @@ action_item_try_repair(return 0 ; } +/* +* Nothing in this fs object was marked inconsistent. This means we +* were merely optimizing metadata and there is no revalidation work to +* be done. +*/ +if (!sri->sri_inconsistent) { + *outcome =TR_REPAIRED ; +return 0 ; + } + +/* +* We fixed inconsistent metadata, so reschedule the entire object for +* immediate revalidation to see if anything else went wrong. +*/ + foreach_scrub_type(scrub_type) + if (sri->sri_selected & (1ULL << scrub_type)) + sri->sri_state[scrub_type] =SCRUB_ITEM_NEEDSCHECK ; + sri->sri_inconsistent = false; + sri->sri_revalidate = true; + + ret =scrub_item_check (ctx, sri); +if (ret) +return ret; + + after =repair_item_count_needsrepair (sri); +if (after >0 ) { +/* +* Uhoh, we found something else broken. Tell the caller that +* this item needs to be queued for more repairs. +*/ + sri->sri_revalidate = false; + *outcome =TR_REQUEUE ; +return 0 ; + } +/* Repairs complete. */ *outcome =TR_REPAIRED ;return 0 ;
scrub/scrub.c +3 -2
@@ -117,11 +117,12 @@ xfs_check_metadata(dbg_printf ("check %s flags %xh\n" ,descr_render (&dsc), meta.sm_flags); error = -xfrog_scrub_metadata (xfdp, &meta); -if (debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" ) && !error) - meta.sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ;switch (error) {case 0 :/* No operational errors encountered. */ +if (!sri->sri_revalidate && +debug_tweak_on ("XFS_SCRUB_FORCE_REPAIR" )) + meta.sm_flags |=XFS_SCRUB_OFLAG_CORRUPT ;break ;case ENOENT :/* Metadata not present, just skip it. */
scrub/scrub.h +10 -0
@@ -59,11 +59,20 @@ struct scrub_item {__u32 sri_gen;__u32 sri_agno; +/* Bitmask of scrub types that were scheduled here. */ +__u64 sri_selected; +/* Scrub item state flags, one for each XFS_SCRUB_TYPE. */ __u8 sri_state[XFS_SCRUB_TYPE_NR ];/* Track scrub and repair call retries for each scrub type. */ __u8 sri_tries[XFS_SCRUB_TYPE_NR ]; + +/* Were there any corruption repairs needed? */ + bool sri_inconsistent:1; + +/* Are we revalidating after repairs? */ + bool sri_revalidate:1; };#define foreach_scrub_type (loopvar) \
@@ -103,6 +112,7 @@ static inline void scrub_item_schedule(struct scrub_item *sri, unsigned int scrub_type) { sri->sri_state[scrub_type] = SCRUB_ITEM_NEEDSCHECK; + sri->sri_selected |= (1ULL << scrub_type); } void scrub_item_schedule_group(struct scrub_item *sri,
scrub/scrub_private.h +2 -0
@@ -71,6 +71,8 @@ scrub_item_save_state( unsigned int scrub_flags) { sri->sri_state[scrub_type] = scrub_flags &SCRUB_ITEM_REPAIR_ANY ; +if (scrub_flags &SCRUB_ITEM_NEEDSREPAIR ) + sri->sri_inconsistent = true; } static inline void
Re: [PATCH 3/4] xfs_scrub: recheck entire metadata objects after corruption repairs
Looks good:
PATCH 4/4 xfs_scrub: try to repair space metadata before file metadata
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Phase 4 (metadata repairs) of xfs_scrub has suffered a mild race condition since the beginning of its existence. Repair functions for higher level metadata such as directories build the new directory blocks in an unlinked temporary file and use atomic extent swapping to commit the corrected directory contents into the existing directory. Atomic extent swapping requires consistent filesystem space metadata, but phase 4 has never enforced correctness dependencies between space and file metadata repairs.
Before the previous patch eliminated the per-AG repair lists, this error was not often hit in testing scenarios because the allocator generally succeeds in placing file data blocks in the same AG as the inode. With pool threads now able to pop file repairs from the repair list before space repairs complete, this error became much more obvious.
Fortunately, the new phase 4 design makes it easy to try to enforce the consistency requirements of higher level file metadata repairs. Split the repair list into one for space metadata and another for file metadata. Phase 4 will now try to fix the space metadata until it stops making progress on that, and only then will it try to fix file metadata.
--- scrub/phase1.c | 13 ++++++++++--- scrub/phase2.c | 2 +- scrub/phase3.c | 4 ++-- scrub/phase4.c | 22 +++++++++++++++++----- scrub/xfs_scrub.h | 3 ++- 5 files changed, 32 insertions(+), 12 deletions(-)
scrub/phase1.c +10 -3
@@ -89,7 +89,8 @@ scrub_cleanup(if (error)return error; -action_list_free (&ctx->action_list); +action_list_free (&ctx->file_repair_list); +action_list_free (&ctx->fs_repair_list);if (ctx->fshandle)free_handle (ctx->fshandle, ctx->fshandle_len);
@@ -186,9 +187,15 @@ _("Not an XFS filesystem."));return error; } - error =action_list_alloc (&ctx->action_list); + error =action_list_alloc (&ctx->fs_repair_list);if (error) { -str_liberror (ctx, error,_ ("allocating repair list" )); +str_liberror (ctx, error,_ ("allocating fs repair list" )); +return error; + } + + error =action_list_alloc (&ctx->file_repair_list); +if (error) { +str_liberror (ctx, error,_ ("allocating file repair list" ));return error; }
scrub/phase2.c +1 -1
@@ -64,7 +64,7 @@ defer_fs_repair(return error;pthread_mutex_lock (&ctx->lock); -action_list_add (ctx->action_list, aitem); +action_list_add (ctx->fs_repair_list, aitem);pthread_mutex_unlock (&ctx->lock);return 0 ; }
scrub/phase3.c +2 -2
@@ -234,7 +234,7 @@ collect_repairs(struct scrub_ctx *ctx = foreach_arg;struct action_list *alist = data; -action_list_merge (ctx->action_list, alist); +action_list_merge (ctx->file_repair_list, alist);return 0 ; }
@@ -278,7 +278,7 @@ phase3_func(* to repair the space metadata. */ for (agno =0 ; agno < ctx->mnt.fsgeom.agcount; agno++) { -if (!action_list_empty (ctx->action_list)) +if (!action_list_empty (ctx->fs_repair_list)) ictx.always_defer_repairs = true; }
scrub/phase4.c +17 -5
@@ -198,7 +198,13 @@ repair_everything( do { fixed_anything = 0; - ret = repair_list_schedule(ctx, &wq, ctx->action_list); + ret = repair_list_schedule(ctx, &wq, ctx->fs_repair_list); + if (ret < 0) + break; + if (ret == 1) + fixed_anything++; + + ret = repair_list_schedule(ctx, &wq, ctx->file_repair_list); if (ret < 0) break; if (ret == 1)
@@ -213,8 +219,12 @@ repair_everything(if (ret <0 )return ret; -/* Repair everything serially. Last chance to fix things. */ -return action_list_process (ctx, ctx->action_list,XRM_FINAL_WARNING ); +/* +* Combine both repair lists and repair everything serially. This is +* the last chance to fix things. +*/ +action_list_merge (ctx->fs_repair_list, ctx->file_repair_list); +return action_list_process (ctx, ctx->fs_repair_list,XRM_FINAL_WARNING ); }/* Trim the unused areas of the filesystem if the caller asked us to. */
@@ -236,7 +246,8 @@ phase4_func(struct scrub_item sri;int ret; -if (action_list_empty (ctx->action_list)) +if (action_list_empty (ctx->fs_repair_list) && +action_list_empty (ctx->file_repair_list)) goto maybe_trim;/*
@@ -297,7 +308,8 @@ phase4_estimate(unsigned long long need_fixing;/* Everything on the repair list plus FSTRIM. */ - need_fixing =action_list_length (ctx->action_list); + need_fixing =action_list_length (ctx->fs_repair_list) + +action_list_length (ctx->file_repair_list); need_fixing++; *items = need_fixing;
scrub/xfs_scrub.h +2 -1
@@ -72,7 +72,8 @@ struct scrub_ctx {/* Mutable scrub state; use lock. */ pthread_mutex_t lock; -struct action_list *action_list; +struct action_list *fs_repair_list; +struct action_list *file_repair_list;unsigned long long max_errors;unsigned long long runtime_errors;unsigned long long corruptions_found;
Re: [PATCH 4/4] xfs_scrub: try to repair space metadata before file metadata
Looks good:
PATCH 31/40 xfs_scrub: detect deceptive filename extensions
In reply to Darrick J. WongHi all,
In early 2023, malware researchers disclosed a phishing attack that was targeted at people running Linux workstations. The attack vector involved the use of filenames containing what looked like a file extension but instead contained a lookalike for the full stop (".") and a common extension ("pdf"). Enhance xfs_scrub phase 5 to detect these types of attacks and warn the system administrator.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-detect-deceptive-extensions
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-detect-deceptive-extensions
--- scrub/unicrash.c | 530 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 424 insertions(+), 106 deletions(-)
PATCH 1/13 xfs_scrub: use proper UChar string iterators
From: Darrick J. Wong <[email protected]>
For code that wants to examine a UChar string, use libicu's string iterators to walk UChar strings, instead of the open-coded U16_NEXT* macros that perform no typechecking.
--- scrub/unicrash.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
scrub/unicrash.c +3 -4
@@ -330,13 +330,12 @@ name_entry_examine(struct name_entry *entry, unsigned int *badflags) { + UCharIterator uiter;UChar32 uchr; -int32_t i;uint8_t mask =0 ; -for (i =0 ; i < entry->normstrlen;) { -U16_NEXT_UNSAFE (entry->normstr, i, uchr); - +uiter_setString (&uiter, entry->normstr, entry->normstrlen); +while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) {/* zero width character sequences */ switch (uchr) {case 0x200B :/* zero width space */
PATCH 2/13 xfs_scrub: hoist code that removes ignorable characters
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Hoist the loop that removes "ignorable" code points from the skeleton string into a separate function and give the UChar cursors names that are easier to understand. Convert the code to use the safe versions of the U16_ accessor functions.
--- scrub/unicrash.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)
scrub/unicrash.c +26 -13
@@ -145,6 +145,31 @@ is_utf8_locale(void)return answer; } +/* +* Remove control/formatting characters from this string and return its new +* length. UChar32 is required for U16_NEXT, despite the name. +*/ +static int32_t +remove_ignorable ( +UChar *ustr, +int32_t ustrlen) +{ +UChar32 uchr; +int32_t src, dest; + +for (src =0 , dest =0 ; src < ustrlen; dest = src) { +U16_NEXT (ustr, src, ustrlen, uchr); +if (!u_isIDIgnorable (uchr)) +continue ; +memmove (&ustr[dest], &ustr[src], + (ustrlen - src +1 ) *sizeof (UChar)); + ustrlen -= (src - dest); + src = dest; + } + +return dest; +} +/* * Generate normalized form and skeleton of the name. If this fails, just * forget everything and return false; this is an advisory checker.
@@ -160,9 +185,6 @@ name_entry_compute_checknames(int32_t normstrlen;int32_t unistrlen;int32_t skelstrlen; -UChar32 uchr; -int32_t i, j; -UErrorCode uerr =U_ZERO_ERROR ;/* Convert bytestr to unistr for normalization */
@@ -206,16 +228,7 @@ name_entry_compute_checknames(if (U_FAILURE (uerr)) goto out_skelstr; -/* Remove control/formatting characters from skeleton. */ -for (i =0 , j =0 ; i < skelstrlen; j = i) { -U16_NEXT_UNSAFE (skelstr, i, uchr); -if (!u_isIDIgnorable (uchr)) -continue ; -memmove (&skelstr[j], &skelstr[i], - (skelstrlen - i +1 ) *sizeof (UChar)); - skelstrlen -= (i - j); - i = j; - } + skelstrlen =remove_ignorable (skelstr, skelstrlen); entry->skelstr = skelstr; entry->skelstrlen = skelstrlen;
PATCH 3/13 xfs_scrub: add a couple of omitted invisible code points
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
I missed a few non-rendering code points in the "zero width" classification code. Add them now, and sort the list.
$ wget https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt $ grep -E '(zero width|invisible|joiner|application)' -i UnicodeData.txt
--- scrub/unicrash.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scrub/unicrash.c +3 -1
@@ -351,15 +351,17 @@ name_entry_examine(while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) {/* zero width character sequences */ switch (uchr) { +case 0x034F :/* combining grapheme joiner */ case 0x200B :/* zero width space */ case 0x200C :/* zero width non-joiner */ case 0x200D :/* zero width joiner */ -case 0xFEFF :/* zero width non breaking space */ case 0x2060 :/* word joiner */ case 0x2061 :/* function application */ case 0x2062 :/* invisible times (multiply) */ case 0x2063 :/* invisible separator (comma) */ case 0x2064 :/* invisible plus (addition) */ +case 0x2D7F :/* tifinagh consonant joiner */ +case 0xFEFF :/* zero width non breaking space */ *badflags |=UNICRASH_ZERO_WIDTH ;break ; }
PATCH 4/13 xfs_scrub: avoid potential UAF after freeing a duplicate name entry
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Change the function declaration of unicrash_add to set the caller's @new_entry to NULL if we detect an updated name entry and do not wish to continue processing. This avoids a theoretical UAF if the unicrash_add caller were to accidentally continue using the pointer.
This isn't an /actual/ UAF because the function formerly set @badflags to zero, but let's be a little defensive.
--- scrub/unicrash.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
scrub/unicrash.c +5 -4
@@ -626,10 +626,11 @@ _("Unicode name \"%s\" in %s could be confused with \"%s\"."),static void unicrash_add (struct unicrash *uc, -struct name_entry *new_entry, +struct name_entry **new_entryp,unsigned int *badflags,struct name_entry **existing_entry) { +struct name_entry *new_entry = *new_entryp;struct name_entry *entry;size_t bucket;xfs_dahash_t hash;
@@ -652,7 +653,7 @@ unicrash_add( entry->ino = new_entry->ino; uc->buckets[bucket] = new_entry->next;name_entry_free (new_entry); - *badflags =0 ; + *new_entryp =NULL ;return ; }
@@ -695,8 +696,8 @@ __unicrash_check_name(return 0 ;name_entry_examine (new_entry, &badflags); -unicrash_add (uc, new_entry, &badflags, &dup_entry); -if (badflags) +unicrash_add (uc, &new_entry, &badflags, &dup_entry); +if (new_entry && badflags)unicrash_complain (uc, dsc, namedescr, new_entry, badflags, dup_entry);
PATCH 5/13 xfs_scrub: guard against libicu returning negative buffer lengths
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
The libicu functions u_strFromUTF8, unorm2_normalize, and uspoof_getSkeleton return int32_t values. Guard against negative return values, even though the library itself never does this.
--- scrub/unicrash.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
scrub/unicrash.c +3 -3
@@ -189,7 +189,7 @@ name_entry_compute_checknames(/* Convert bytestr to unistr for normalization */ u_strFromUTF8 (NULL ,0 , &unistrlen, entry->name, entry->namelen, &uerr); -if (uerr !=U_BUFFER_OVERFLOW_ERROR ) +if (uerr !=U_BUFFER_OVERFLOW_ERROR || unistrlen <0 )return false; uerr =U_ZERO_ERROR ; unistr =calloc (unistrlen +1 ,sizeof (UChar));
@@ -203,7 +203,7 @@ name_entry_compute_checknames(/* Normalize the string. */ normstrlen =unorm2_normalize (uc->normalizer, unistr, unistrlen,NULL ,0 , &uerr); -if (uerr !=U_BUFFER_OVERFLOW_ERROR ) +if (uerr !=U_BUFFER_OVERFLOW_ERROR || normstrlen <0 ) goto out_unistr; uerr =U_ZERO_ERROR ; normstr =calloc (normstrlen +1 ,sizeof (UChar));
@@ -217,7 +217,7 @@ name_entry_compute_checknames(/* Compute skeleton. */ skelstrlen =uspoof_getSkeleton (uc->spoof,0 , unistr, unistrlen,NULL ,0 , &uerr); -if (uerr !=U_BUFFER_OVERFLOW_ERROR ) +if (uerr !=U_BUFFER_OVERFLOW_ERROR || skelstrlen <0 ) goto out_normstr; uerr =U_ZERO_ERROR ; skelstr =calloc (skelstrlen +1 ,sizeof (UChar));
PATCH 6/13 xfs_scrub: hoist non-rendering character predicate
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Hoist this predicate code into its own function; we're going to use it elsewhere later on. While we're at it, document how we generated this list in the first place.
--- scrub/unicrash.c | 45 ++++++++++++++++++++++++++++++--------------- 1 file changed, 30 insertions(+), 15 deletions(-)
scrub/unicrash.c +30 -15
@@ -170,6 +170,34 @@ remove_ignorable(return dest; } +/* +* Certain unicode codepoints are formatting hints that are not themselves +* supposed to be rendered by a display system. These codepoints can be +* encoded in file names to try to confuse users. +* +* Download https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt and +* $ grep -E '(zero width|invisible|joiner|application)' -i UnicodeData.txt +*/ +static inline bool is_nonrendering (UChar32 uchr) +{ +switch (uchr) { +case 0x034F :/* combining grapheme joiner */ +case 0x200B :/* zero width space */ +case 0x200C :/* zero width non-joiner */ +case 0x200D :/* zero width joiner */ +case 0x2060 :/* word joiner */ +case 0x2061 :/* function application */ +case 0x2062 :/* invisible times (multiply) */ +case 0x2063 :/* invisible separator (comma) */ +case 0x2064 :/* invisible plus (addition) */ +case 0x2D7F :/* tifinagh consonant joiner */ +case 0xFEFF :/* zero width non breaking space */ +return true; + } + +return false; +} +/* * Generate normalized form and skeleton of the name. If this fails, just * forget everything and return false; this is an advisory checker.
@@ -349,22 +377,9 @@ name_entry_examine(uiter_setString (&uiter, entry->normstr, entry->normstrlen);while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) { -/* zero width character sequences */ -switch (uchr) { -case 0x034F :/* combining grapheme joiner */ -case 0x200B :/* zero width space */ -case 0x200C :/* zero width non-joiner */ -case 0x200D :/* zero width joiner */ -case 0x2060 :/* word joiner */ -case 0x2061 :/* function application */ -case 0x2062 :/* invisible times (multiply) */ -case 0x2063 :/* invisible separator (comma) */ -case 0x2064 :/* invisible plus (addition) */ -case 0x2D7F :/* tifinagh consonant joiner */ -case 0xFEFF :/* zero width non breaking space */ +/* characters are invisible */ +if (is_nonrendering (uchr)) *badflags |=UNICRASH_ZERO_WIDTH ; -break ; - }/* control characters */ if (u_iscntrl (uchr))
PATCH 7/13 xfs_scrub: store bad flags with the name entry
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
When scrub is checking unicode names, there are certain properties of the directory/attribute/label name itself that it can complain about. Store these in struct name_entry so that the confusable names detector can pick this up later.
This restructuring enables a subsequent patch to detect suspicious sequences in the NFC normalized form of the name without needing to hang on to that NFC form until the end of processing. IOWs, it's a memory usage optimization.
--- scrub/unicrash.c | 122 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 64 insertions(+), 58 deletions(-)
scrub/unicrash.c +64 -58
@@ -69,6 +69,9 @@ struct name_entry {xfs_ino_t ino; +/* Everything that we don't like about this name. */ +unsigned int badflags; +/* Raw dirent name */ size_t namelen;char name[0 ];
@@ -274,6 +277,55 @@ name_entry_compute_checknames(return false; } +/* +* Check a name for suspicious elements that have appeared in filename +* spoofing attacks. This includes names that mixed directions or contain +* direction overrides control characters, both of which have appeared in +* filename spoofing attacks. +*/ +static unsigned int +name_entry_examine ( +const struct name_entry *entry) +{ +UCharIterator uiter; +UChar32 uchr; +uint8_t mask =0 ; +unsigned int ret =0 ; + +uiter_setString (&uiter, entry->normstr, entry->normstrlen); +while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) { +/* characters are invisible */ +if (is_nonrendering (uchr)) + ret |=UNICRASH_ZERO_WIDTH ; + +/* control characters */ +if (u_iscntrl (uchr)) + ret |=UNICRASH_CONTROL_CHAR ; + +switch (u_charDirection (uchr)) { +case U_LEFT_TO_RIGHT : + mask |=0x01 ; +break ; +case U_RIGHT_TO_LEFT : + mask |=0x02 ; +break ; +case U_RIGHT_TO_LEFT_OVERRIDE : + ret |=UNICRASH_BIDI_OVERRIDE ; +break ; +case U_LEFT_TO_RIGHT_OVERRIDE : + ret |=UNICRASH_BIDI_OVERRIDE ; +break ; +default : +break ; + } + } + +/* mixing left-to-right and right-to-left chars */ +if (mask ==0x3 ) + ret |=UNICRASH_BIDI_MIXED ; +return ret; +} +/* Create a new name entry, returns false if we could not succeed. */ static bool name_entry_create(
@@ -299,6 +351,7 @@ name_entry_create(if (!name_entry_compute_checknames (uc, new_entry)) goto out; + new_entry->badflags =name_entry_examine (new_entry); *entry = new_entry;return true;
@@ -360,54 +413,6 @@ name_entry_hash( } } -/* -* Check a name for suspicious elements that have appeared in filename -* spoofing attacks. This includes names that mixed directions or contain -* direction overrides control characters, both of which have appeared in -* filename spoofing attacks. -*/ -static void -name_entry_examine ( -struct name_entry *entry, -unsigned int *badflags) -{ -UCharIterator uiter; -UChar32 uchr; -uint8_t mask =0 ; - -uiter_setString (&uiter, entry->normstr, entry->normstrlen); -while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) { -/* characters are invisible */ -if (is_nonrendering (uchr)) - *badflags |=UNICRASH_ZERO_WIDTH ; - -/* control characters */ -if (u_iscntrl (uchr)) - *badflags |=UNICRASH_CONTROL_CHAR ; - -switch (u_charDirection (uchr)) { -case U_LEFT_TO_RIGHT : - mask |=0x01 ; -break ; -case U_RIGHT_TO_LEFT : - mask |=0x02 ; -break ; -case U_RIGHT_TO_LEFT_OVERRIDE : - *badflags |=UNICRASH_BIDI_OVERRIDE ; -break ; -case U_LEFT_TO_RIGHT_OVERRIDE : - *badflags |=UNICRASH_BIDI_OVERRIDE ; -break ; -default : -break ; - } - } - -/* mixing left-to-right and right-to-left chars */ -if (mask ==0x3 ) - *badflags |=UNICRASH_BIDI_MIXED ; -} -/* Initialize the collision detector. */ static int unicrash_init(
@@ -638,17 +643,17 @@ _("Unicode name \"%s\" in %s could be confused with \"%s\"."),* must be skeletonized according to Unicode TR39 to detect names that * could be visually confused with each other. */ -static void +static unsigned int unicrash_add (struct unicrash *uc,struct name_entry **new_entryp, -unsigned int *badflags,struct name_entry **existing_entry) {struct name_entry *new_entry = *new_entryp;struct name_entry *entry;size_t bucket;xfs_dahash_t hash; +unsigned int badflags = new_entry->badflags;/* Store name in hashtable. */ hash =name_entry_hash (new_entry);
@@ -669,28 +674,30 @@ unicrash_add( uc->buckets[bucket] = new_entry->next;name_entry_free (new_entry); *new_entryp =NULL ; -return ; +return 0 ; }/* Same normalization? */ if (new_entry->normstrlen == entry->normstrlen && !u_strcmp (new_entry->normstr, entry->normstr) && (uc->compare_ino ? entry->ino != new_entry->ino : true)) { - *badflags |=UNICRASH_NOT_UNIQUE ; + badflags |=UNICRASH_NOT_UNIQUE ; *existing_entry = entry; -return ; +break ; }/* Confusable? */ if (new_entry->skelstrlen == entry->skelstrlen && !u_strcmp (new_entry->skelstr, entry->skelstr) && (uc->compare_ino ? entry->ino != new_entry->ino : true)) { - *badflags |=UNICRASH_CONFUSABLE ; + badflags |=UNICRASH_CONFUSABLE ; *existing_entry = entry; -return ; +break ; } entry = entry->next; } + +return badflags; }/* Check a name for unicode normalization problems or collisions. */
@@ -704,14 +711,13 @@ __unicrash_check_name( {struct name_entry *dup_entry =NULL ;struct name_entry *new_entry =NULL ; -unsigned int badflags =0 ; +unsigned int badflags;/* If we can't create entry data, just skip it. */ if (!name_entry_create (uc, name, ino, &new_entry))return 0 ; -name_entry_examine (new_entry, &badflags); -unicrash_add (uc, &new_entry, &badflags, &dup_entry); + badflags =unicrash_add (uc, &new_entry, &dup_entry);if (new_entry && badflags)unicrash_complain (uc, dsc, namedescr, new_entry, badflags, dup_entry);
PATCH 8/13 xfs_scrub: rename UNICRASH_ZERO_WIDTH to UNICRASH_INVISIBLE
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
"Zero width" doesn't fully describe what the flag represents -- it gets set for any codepoint that doesn't render. Rename it accordingly.
--- scrub/unicrash.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
scrub/unicrash.c +3 -3
@@ -109,7 +109,7 @@ struct unicrash {#define UNICRASH_CONTROL_CHAR (1 << 3)/* Invisible characters. Only a problem if we have collisions. */ -#define UNICRASH_ZERO_WIDTH (1 << 4) +#define UNICRASH_INVISIBLE (1 << 4)/* Multiple names resolve to the same skeleton string. */ #define UNICRASH_CONFUSABLE (1 << 5)
@@ -296,7 +296,7 @@ name_entry_examine(while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) {/* characters are invisible */ if (is_nonrendering (uchr)) - ret |=UNICRASH_ZERO_WIDTH ; + ret |=UNICRASH_INVISIBLE ;/* control characters */ if (u_iscntrl (uchr))
@@ -580,7 +580,7 @@ _("Unicode name \"%s\" in %s renders identically to \"%s\"."),* confused with another name as a result, we should complain. * "moo<zerowidthspace>cow" and "moocow" are misleading. */ -if ((badflags &UNICRASH_ZERO_WIDTH ) && +if ((badflags &UNICRASH_INVISIBLE ) && (badflags &UNICRASH_CONFUSABLE )) { str_warn(uc->ctx, descr_render(dsc), _("Unicode name \"%s\" in %s could be confused with '%s' due to invisible characters."),
PATCH 9/13 xfs_scrub: type-coerce the UNICRASH_* flags
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Promote this type to something that we can type-check.
--- scrub/unicrash.c | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-)
scrub/unicrash.c +18 -12
@@ -4,6 +4,7 @@* Author: Darrick J. Wong <[email protected]> */ #include "xfs.h" +#include "xfs_arch.h" #include <stdint.h> #include <stdlib.h> #include <dirent.h>
@@ -56,6 +57,8 @@* In other words, skel = remove_invisible(nfd(remap_confusables(nfd(name)))). */ +typedef unsigned int __bitwise badname_t ; +struct name_entry {struct name_entry *next;
@@ -70,7 +73,7 @@ struct name_entry {xfs_ino_t ino;/* Everything that we don't like about this name. */ -unsigned int badflags; +badname_t badflags;/* Raw dirent name */ size_t namelen;
@@ -93,26 +96,29 @@ struct unicrash {/* Things to complain about in Unicode naming. */ +/* Everything is ok */ +#define UNICRASH_OK ((__force badname_t)0) +/* * Multiple names resolve to the same normalized string and therefore render * identically. */ -#define UNICRASH_NOT_UNIQUE (1 << 0) +#define UNICRASH_NOT_UNIQUE ((__force badname_t)(1U << 0))/* Name contains directional overrides. */ -#define UNICRASH_BIDI_OVERRIDE (1 << 1) +#define UNICRASH_BIDI_OVERRIDE ((__force badname_t)(1U << 1))/* Name mixes left-to-right and right-to-left characters. */ -#define UNICRASH_BIDI_MIXED (1 << 2) +#define UNICRASH_BIDI_MIXED ((__force badname_t)(1U << 2))/* Control characters in name. */ -#define UNICRASH_CONTROL_CHAR (1 << 3) +#define UNICRASH_CONTROL_CHAR ((__force badname_t)(1U << 3))/* Invisible characters. Only a problem if we have collisions. */ -#define UNICRASH_INVISIBLE (1 << 4) +#define UNICRASH_INVISIBLE ((__force badname_t)(1U << 4))/* Multiple names resolve to the same skeleton string. */ -#define UNICRASH_CONFUSABLE (1 << 5) +#define UNICRASH_CONFUSABLE ((__force badname_t)(1U << 5))/* * We only care about validating utf8 collisions if the underlying
@@ -540,7 +546,7 @@ unicrash_complain(struct descr *dsc, const char *what, struct name_entry *entry, - unsigned int badflags, + badname_t badflags, struct name_entry *dup_entry) { char *bad1 = NULL;
@@ -643,7 +649,7 @@ _("Unicode name \"%s\" in %s could be confused with \"%s\"."),* must be skeletonized according to Unicode TR39 to detect names that * could be visually confused with each other. */ -static unsigned int +static badname_t unicrash_add( struct unicrash *uc, struct name_entry **new_entryp,
@@ -653,7 +659,7 @@ unicrash_add(struct name_entry *entry;size_t bucket;xfs_dahash_t hash; -unsigned int badflags = new_entry->badflags; +badname_t badflags = new_entry->badflags;/* Store name in hashtable. */ hash =name_entry_hash (new_entry);
@@ -711,14 +717,14 @@ __unicrash_check_name( {struct name_entry *dup_entry =NULL ;struct name_entry *new_entry =NULL ; -unsigned int badflags; +badname_t badflags;/* If we can't create entry data, just skip it. */ if (!name_entry_create (uc, name, ino, &new_entry))return 0 ; badflags =unicrash_add (uc, &new_entry, &dup_entry); -if (new_entry && badflags) +if (new_entry && badflags !=UNICRASH_OK )unicrash_complain (uc, dsc, namedescr, new_entry, badflags, dup_entry);
PATCH 10/13 xfs_scrub: reduce size of struct name_entry
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
libicu doesn't support processing strings longer than 2GB in length, and we never feed the unicrash code a name longer than about 300 bytes. Rearrange the structure to reduce the head structure size from 56 bytes to 44 bytes.
--- scrub/unicrash.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
scrub/unicrash.c +12 -4
@@ -57,18 +57,20 @@* In other words, skel = remove_invisible(nfd(remap_confusables(nfd(name)))). */ -typedef unsigned int __bitwise badname_t ; +typedef uint16_t __bitwise badname_t ;struct name_entry {struct name_entry *next;/* NFKC normalized name */ UChar *normstr; -size_t normstrlen;/* Unicode skeletonized name */ UChar *skelstr; -size_t skelstrlen; + +/* Lengths for normstr and skelstr */ +int32_t normstrlen; +int32_t skelstrlen;xfs_ino_t ino;
@@ -76,7 +78,7 @@ struct name_entry {badname_t badflags;/* Raw dirent name */ -size_t namelen; +uint16_t namelen;char name[0 ]; };#define NAME_ENTRY_SZ (nl) (sizeof(struct name_entry) + 1 + \
@@ -343,6 +345,12 @@ name_entry_create(struct name_entry *new_entry;size_t namelen =strlen (name); +/* should never happen */ +if (namelen >UINT16_MAX ) { +ASSERT (namelen <=UINT16_MAX ); +return false; + } +/* Create new entry */ new_entry =calloc (NAME_ENTRY_SZ (namelen),1 );if (!new_entry)
PATCH 11/13 xfs_scrub: rename struct unicrash.normalizer
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
We're about to introduce a second normalizer, so change the name of the existing one to reflect the algorithm that you'll get if you use it.
--- scrub/unicrash.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scrub/unicrash.c +4 -4
@@ -87,7 +87,7 @@ struct name_entry {struct unicrash {struct scrub_ctx *ctx;USpoofChecker *spoof; -const UNormalizer2 *normalizer; +const UNormalizer2 *nfkc;bool compare_ino;bool is_only_root_writeable;size_t nr_buckets;
@@ -240,7 +240,7 @@ name_entry_compute_checknames( goto out_unistr;/* Normalize the string. */ - normstrlen =unorm2_normalize (uc->normalizer, unistr, unistrlen,NULL , + normstrlen =unorm2_normalize (uc->nfkc, unistr, unistrlen,NULL ,0 , &uerr);if (uerr !=U_BUFFER_OVERFLOW_ERROR || normstrlen <0 ) goto out_unistr;
@@ -248,7 +248,7 @@ name_entry_compute_checknames( normstr =calloc (normstrlen +1 ,sizeof (UChar));if (!normstr) goto out_unistr; -unorm2_normalize (uc->normalizer, unistr, unistrlen, normstr, normstrlen, +unorm2_normalize (uc->nfkc, unistr, unistrlen, normstr, normstrlen, &uerr);if (U_FAILURE (uerr)) goto out_normstr;
@@ -455,7 +455,7 @@ unicrash_init( p->ctx = ctx; p->nr_buckets = nr_buckets; p->compare_ino = compare_ino; - p->normalizer =unorm2_getNFKCInstance (&uerr); + p->nfkc =unorm2_getNFKCInstance (&uerr);if (U_FAILURE (uerr)) goto out_free; p->spoof =uspoof_open (&uerr);
PATCH 12/13 xfs_scrub: report deceptive file extensions
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Earlier this year, ESET revealed that Linux users had been tricked into opening executables containing malware payloads. The trickery came in the form of a malicious zip file containing a filename with the string "job offer․pdf". Note that the filename does *not* denote a real pdf file, since the last four codepoints in the file name are "ONE DOT LEADER", p, d, and f. Not period (ok, FULL STOP), p, d, f like you'd normally expect.
Teach xfs_scrub to look for codepoints that could be confused with a period followed by alphanumerics.
--- scrub/unicrash.c | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+), 1 deletion(-)
scrub/unicrash.c +214 -1
@@ -88,6 +88,7 @@ struct unicrash {struct scrub_ctx *ctx;USpoofChecker *spoof;const UNormalizer2 *nfkc; +const UNormalizer2 *nfc;bool compare_ino;bool is_only_root_writeable;size_t nr_buckets;
@@ -122,6 +123,12 @@ struct unicrash {/* Multiple names resolve to the same skeleton string. */ #define UNICRASH_CONFUSABLE ((__force badname_t)(1U << 5)) +/* Possible phony file extension. */ +#define UNICRASH_PHONY_EXTENSION ((__force badname_t)(1U << 6)) + +/* FULL STOP (aka period), 0x2E */ +#define UCHAR_PERIOD ((UChar32)'.') +/* * We only care about validating utf8 collisions if the underlying * system configuration says we're using utf8. If the language
@@ -209,6 +216,193 @@ static inline bool is_nonrendering(UChar32 uchr)return false; } +/* +* Decide if this unicode codepoint looks similar enough to a period (".") +* to fool users into thinking that any subsequent alphanumeric sequence is +* the file extension. Most of the fullstop characters do not do this. +* +* $ grep -i 'full stop' UnicodeData.txt +*/ +static inline bool is_fullstop_lookalike (UChar32 uchr) +{ +switch (uchr) { +case 0x0701 :/* syriac supralinear full stop */ +case 0x0702 :/* syriac sublinear full stop */ +case 0x2024 :/* one dot leader */ +case 0xA4F8 :/* lisu letter tone mya ti */ +case 0xFE52 :/* small full stop */ +case 0xFF61 :/* haflwidth ideographic full stop */ +case 0xFF0E :/* fullwidth full stop */ +return true; + } + +return false; +} + +/* How many UChar do we need to fit a full UChar32 codepoint? */ +#define UCHAR_PER_UCHAR32 2 + +/* Format this UChar32 into a UChar buffer. */ +static inline int32_t +uchar32_to_uchar ( +UChar32 uchr, +UChar *buf) +{ +int32_t i =0 ; +bool err = false; + +U16_APPEND (buf, i,UCHAR_PER_UCHAR32 , uchr, err); +if (err) +return 0 ; +return i; +} + +/* Extract a single UChar32 code point from this UChar string. */ +static inline UChar32 +uchar_to_uchar32 ( +UChar *buf, +int32_t buflen) +{ +UChar32 ret; +int32_t i =0 ; + +U16_NEXT (buf, i, buflen, ret); +return ret; +} + +/* +* For characters that are not themselves a full stop (0x2E), let's see if the +* compatibility normalization (NFKC) will turn it into a full stop. If so, +* then this could be the start of a phony file extension. +*/ +static bool +is_period_lookalike ( +struct unicrash *uc, +UChar32 uchr) +{ +UChar uchrstr[UCHAR_PER_UCHAR32 ]; +UChar nfkcstr[UCHAR_PER_UCHAR32 ]; +int32_t uchrstrlen, nfkcstrlen; +UChar32 nfkc_uchr; +UErrorCode uerr =U_ZERO_ERROR ; + +if (uchr ==UCHAR_PERIOD ) +return false; + + uchrstrlen =uchar32_to_uchar (uchr, uchrstr); +if (!uchrstrlen) +return false; + +/* +* Normalize the UChar string to NFKC form, which does all the +* compatibility transformations. +*/ + nfkcstrlen =unorm2_normalize (uc->nfkc, uchrstr, uchrstrlen,NULL , +0 , &uerr); +if (uerr ==U_BUFFER_OVERFLOW_ERROR ) +return false; + + uerr =U_ZERO_ERROR ; +unorm2_normalize (uc->nfkc, uchrstr, uchrstrlen, nfkcstr, nfkcstrlen, + &uerr); +if (U_FAILURE (uerr)) +return false; + + nfkc_uchr =uchar_to_uchar32 (nfkcstr, nfkcstrlen); +return nfkc_uchr ==UCHAR_PERIOD ; +} + +/* +* Detect directory entry names that contain deceptive sequences that look like +* file extensions but are not. This we define as a sequence that begins with +* a code point that renders like a period ("full stop" in unicode parlance) +* but is not actually a period, followed by any number of alphanumeric code +* points or a period, all the way to the end. +* +* The 3cx attack used a zip file containing an executable file named "job +* offer․pdf". Note that the dot mark in the extension is /not/ a period but +* the Unicode codepoint "leader dot". The file was also marked executable +* inside the zip file, which meant that naïve file explorers could inflate +* the file and restore the execute bit. If a user double-clicked on the file, +* the binary would open a decoy pdf while infecting the system. +* +* For this check, we need to normalize with canonical (and not compatibility) +* decomposition, because compatibility mode will turn certain code points +* (e.g. one dot leader, 0x2024) into actual periods (0x2e). The NFC +* composition is not needed after this, so we save some memory by keeping this +* a separate function from name_entry_examine. +*/ +static badname_t +name_entry_phony_extension ( +struct unicrash *uc, +const UChar *unistr, +int32_t unistrlen) +{ +UCharIterator uiter; +UChar *nfcstr; +int32_t nfcstrlen; +UChar32 uchr; +bool maybe_phony_extension = false; +badname_t ret =UNICRASH_OK ; +UErrorCode uerr =U_ZERO_ERROR ; + +/* Normalize with NFC. */ + nfcstrlen =unorm2_normalize (uc->nfc, unistr, unistrlen,NULL , +0 , &uerr); +if (uerr !=U_BUFFER_OVERFLOW_ERROR || nfcstrlen <0 ) +return ret; + uerr =U_ZERO_ERROR ; + nfcstr =calloc (nfcstrlen +1 ,sizeof (UChar)); +if (!nfcstr) +return ret; +unorm2_normalize (uc->nfc, unistr, unistrlen, nfcstr, nfcstrlen, + &uerr); +if (U_FAILURE (uerr)) + goto out_nfcstr; + +/* Examine the NFC normalized string... */ +uiter_setString (&uiter, nfcstr, nfcstrlen); +while ((uchr =uiter_next32 (&uiter)) !=U_SENTINEL ) { +/* +* If this *looks* like, but is not, a full stop (0x2E), this +* could be the start of a phony file extension. +*/ +if (is_period_lookalike (uc, uchr)) { + maybe_phony_extension = true; +continue ; + } + +if (is_fullstop_lookalike (uchr)) { +/* +* The normalizer above should catch most of these +* codepoints that look like periods, but record the +* ones known to have been used in attacks. +*/ + maybe_phony_extension = true; + }else if (uchr ==UCHAR_PERIOD ) { +/* +* Due to the propensity of file explores to obscure +* file extensions in the name of "user friendliness", +* this classifier ignores periods. +*/ + }else { +/* +* File extensions (as far as the author knows) tend +* only to use ascii alphanumerics. +*/ +if (maybe_phony_extension && + !u_isalnum (uchr) && !is_nonrendering (uchr)) + maybe_phony_extension = false; + } + } +if (maybe_phony_extension) + ret |=UNICRASH_PHONY_EXTENSION ; + +out_nfcstr: +free (nfcstr); +return ret; +} +/* * Generate normalized form and skeleton of the name. If this fails, just * forget everything and return false; this is an advisory checker.
@@ -269,6 +463,11 @@ name_entry_compute_checknames( skelstrlen =remove_ignorable (skelstr, skelstrlen); +/* Check for deceptive file extensions in directory entry names. */ +if (entry->ino) + entry->badflags |=name_entry_phony_extension (uc, unistr, + unistrlen); + entry->skelstr = skelstr; entry->skelstrlen = skelstrlen; entry->normstr = normstr;
@@ -365,7 +564,7 @@ name_entry_create(if (!name_entry_compute_checknames (uc, new_entry)) goto out; - new_entry->badflags =name_entry_examine (new_entry); + new_entry->badflags |=name_entry_examine (new_entry); *entry = new_entry;return true;
@@ -456,6 +655,9 @@ unicrash_init( p->nr_buckets = nr_buckets; p->compare_ino = compare_ino; p->nfkc =unorm2_getNFKCInstance (&uerr); +if (U_FAILURE (uerr)) + goto out_free; + p->nfc =unorm2_getNFCInstance (&uerr);if (U_FAILURE (uerr)) goto out_free; p->spoof =uspoof_open (&uerr);
@@ -602,6 +804,17 @@ _("Unicode name \"%s\" in %s could be confused with '%s' due to invisible charac goto out; } +/* +* Fake looking file extensions have tricked Linux users into thinking +* that an executable is actually a pdf. See Lazarus 3cx attack. +*/ +if (badflags &UNICRASH_PHONY_EXTENSION ) { +str_warn (uc->ctx,descr_render (dsc), +_ ("Unicode name \"%s\" in %s contains a possibly deceptive file extension." ), + bad1, what); + goto out; + } +/* * Unfiltered control characters can mess up your terminal and render * invisibly in filechooser UIs.
PATCH 13/13 xfs_scrub: dump unicode points
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add some debug functions to make it easier to query unicode character properties.
--- scrub/unicrash.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-)
scrub/unicrash.c +57 -2
@@ -5,6 +5,7 @@*/ #include "xfs.h" #include "xfs_arch.h" +#include "list.h" #include <stdint.h> #include <stdlib.h> #include <dirent.h>
@@ -1001,14 +1002,68 @@ unicrash_check_fs_label( label, 0); } +/* Dump a unicode code point and its properties. */ +static inline void dump_uchar32 (UChar32 c) +{ +UChar uchrstr[UCHAR_PER_UCHAR32 ]; +const char *descr; +char buf[16 ]; +int32_t uchrstrlen, buflen; +UProperty p; +UErrorCode uerr =U_ZERO_ERROR ; + +printf ("Unicode point 0x%x:" , c); + +/* Convert UChar32 to UTF8 representation. */ + uchrstrlen =uchar32_to_uchar (c, uchrstr); +if (!uchrstrlen) +return ; + +u_strToUTF8 (buf,sizeof (buf), &buflen, uchrstr, uchrstrlen, &uerr); +if (!U_FAILURE (uerr) && buflen >0 ) { +int32_t i; + +printf (" \"" ); +for (i =0 ; i < buflen; i++) +printf ("\\x%02x" , buf[i]); +printf ("\"" ); + } +printf ("\n" ); + +for (p =0 ; p <UCHAR_BINARY_LIMIT ; p++) { +int has; + + descr =u_getPropertyName (p,U_LONG_PROPERTY_NAME ); +if (!descr) + descr =u_getPropertyName (p,U_SHORT_PROPERTY_NAME ); + + has =u_hasBinaryProperty (c, p) ?1 :0 ; +if (descr) { +printf (" %s(%u) = %d\n" , descr, p, has); + }else { +printf (" ?(%u) = %d\n" , p, has); + } + } +} +/* Load libicu and initialize it. */ bool unicrash_load (void ) { -UErrorCode uerr =U_ZERO_ERROR ; +char *dbgstr; +UChar32 uchr; +UErrorCode uerr =U_ZERO_ERROR ;u_init (&uerr); -return U_FAILURE (uerr); +if (U_FAILURE (uerr)) +return true; + + dbgstr =getenv ("XFS_SCRUB_DUMP_CHAR" ); +if (dbgstr) { + uchr =strtol (dbgstr,NULL ,0 ); +dump_uchar32 (uchr); + } +return false; }/* Unload libicu once we're done with it. */
PATCH 32/40 xfs_scrub: move fstrim to a separate phase
In reply to Darrick J. WongHi all,
Back when I originally designed xfs_scrub, all filesystem metadata checks were complete by the end of phase 3, and phase 4 was where all the metadata repairs occurred. On the grounds that the filesystem should be fully consistent by then, I made a call to FITRIM at the end of phase 4 to discard empty space in the filesystem.
Unfortunately, that's no longer the case -- summary counters, link counts, and quota counters are not checked until phase 7. It's not safe to instruct the storage to unmap "empty" areas if we don't know where those empty areas are, so we need to create a phase 8 to trim the fs. While we're at it, make it more obvious that fstrim only gets to run if there are no unfixed corruptions and no other runtime errors have occurred.
Finally, reduce the latency impacts on the rest of the system by breaking up the fstrim work into a loop that targets only 16GB per call. This enables better progress reporting for interactive runs and cgroup based resource constraints for background runs.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-fstrim-phase
--- scrub/Makefile | 1 scrub/phase4.c | 30 +---------- scrub/phase8.c | 151 +++++++++++++++++++++++++++++++++++++++++++++++++++++ scrub/vfs.c | 22 +++++--- scrub/vfs.h | 2 - scrub/xfs_scrub.c | 11 ++++ scrub/xfs_scrub.h | 3 + 7 files changed, 183 insertions(+), 37 deletions(-) create mode 100644 scrub/phase8.c
PATCH 1/8 xfs_scrub: move FITRIM to phase 8
From: Darrick J. Wong <[email protected]>
Issuing discards against the filesystem should be the *last* thing that xfs_scrub does, after everything else has been checked, repaired, and found to be clean. If we can't satisfy all those conditions, we have no business telling the storage to discard itself.
--- scrub/Makefile | 1 + scrub/phase4.c | 30 ++---------------------- scrub/phase8.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++ scrub/xfs_scrub.h | 3 ++ 4 files changed, 73 insertions(+), 27 deletions(-) create mode 100644 scrub/phase8.c
scrub/Makefile +1 -0
@@ -65,6 +65,7 @@ phase4.c \ phase5.c \ phase6.c \ phase7.c \ +phase8.c \ progress.c \ read_verify.c \ repair.c \
scrub/phase4.c +3 -27
@@ -227,16 +227,6 @@ repair_everything(return action_list_process (ctx, ctx->fs_repair_list,XRM_FINAL_WARNING ); } -/* Trim the unused areas of the filesystem if the caller asked us to. */ -static void -trim_filesystem ( -struct scrub_ctx *ctx) -{ -if (want_fstrim) -fstrim (ctx); -progress_add (1 ); -} -/* Fix everything that needs fixing. */ int phase4_func(
@@ -248,7 +238,7 @@ phase4_func(if (action_list_empty (ctx->fs_repair_list) &&action_list_empty (ctx->file_repair_list)) - goto maybe_trim; +return 0 ;/* * Check the resource usage counters early. Normally we do this during
@@ -281,20 +271,7 @@ phase4_func(if (ret)return ret; - ret =repair_everything (ctx); -if (ret) -return ret; - -/* -* If errors remain on the filesystem, do not trim anything. We don't -* have any threads running, so it's ok to skip the ctx lock here. -*/ -if (ctx->corruptions_found || ctx->unfixable_errors !=0 ) -return 0 ; - -maybe_trim: -trim_filesystem (ctx); -return 0 ; +return repair_everything (ctx); }/* Estimate how much work we're going to do. */
@@ -307,10 +284,9 @@ phase4_estimate( {unsigned long long need_fixing; -/* Everything on the repair list plus FSTRIM. */ +/* Everything on the repair lis. */ need_fixing =action_list_length (ctx->fs_repair_list) +action_list_length (ctx->file_repair_list); - need_fixing++; *items = need_fixing; *nr_threads =scrub_nproc (ctx) +1 ;
scrub/phase8.c new file +66 -0
66 lines not on this page · raw message
scrub/xfs_scrub.h +3 -0
@@ -98,6 +98,7 @@ int phase4_func(struct scrub_ctx *ctx);int phase5_func (struct scrub_ctx *ctx);int phase6_func (struct scrub_ctx *ctx);int phase7_func (struct scrub_ctx *ctx); +int phase8_func (struct scrub_ctx *ctx);/* Progress estimator functions */ unsigned int scrub_estimate_ag_work (struct scrub_ctx *ctx);
@@ -112,5 +113,7 @@ int phase5_estimate(struct scrub_ctx *ctx, uint64_t *items, unsigned int *nr_threads, int *rshift);int phase6_estimate (struct scrub_ctx *ctx,uint64_t *items,unsigned int *nr_threads,int *rshift); +int phase8_estimate (struct scrub_ctx *ctx,uint64_t *items, +unsigned int *nr_threads,int *rshift);#endif /* XFS_SCRUB_XFS_SCRUB_H_ */
PATCH 2/8 xfs_scrub: ignore phase 8 if the user disabled fstrim
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the user told us to skip trimming the filesystem, don't run the phase at all.
--- scrub/xfs_scrub.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
scrub/xfs_scrub.c +10 -1
@@ -249,6 +249,7 @@ struct phase_rusage {/* Operations for each phase. */ #define DATASCAN_DUMMY_FN ((void *)1)#define REPAIR_DUMMY_FN ((void *)2) +#define FSTRIM_DUMMY_FN ((void *)3)struct phase_ops {char *descr;int (*fn)(struct scrub_ctx *ctx);
@@ -429,6 +430,11 @@ run_scrub_phases( .fn = phase7_func, .must_run = true, }, + { + .descr = _("Trim filesystem storage."), + .fn = FSTRIM_DUMMY_FN, + .estimate_work = phase8_estimate, + }, { NULL },
@@ -449,6 +455,8 @@ run_scrub_phases(/* Turn on certain phases if user said to. */ if (sp->fn ==DATASCAN_DUMMY_FN && scrub_data) { sp->fn = phase6_func; + }else if (sp->fn ==FSTRIM_DUMMY_FN && want_fstrim) { + sp->fn = phase8_func; }else if (sp->fn ==REPAIR_DUMMY_FN && ctx->mode ==SCRUB_MODE_REPAIR ) { sp->descr =_ ("Repair filesystem." );
@@ -458,7 +466,8 @@ run_scrub_phases(/* Skip certain phases unless they're turned on. */ if (sp->fn ==REPAIR_DUMMY_FN || - sp->fn ==DATASCAN_DUMMY_FN ) + sp->fn ==DATASCAN_DUMMY_FN || + sp->fn ==FSTRIM_DUMMY_FN )continue ;/* Allow debug users to force a particular phase. */
PATCH 3/8 xfs_scrub: collapse trim_filesystem
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Collapse this two-line helper into the main function since it's trivial.
--- scrub/phase8.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
scrub/phase8.c +2 -10
@@ -21,15 +21,6 @@/* Phase 8: Trim filesystem. */ -/* Trim the unused areas of the filesystem if the caller asked us to. */ -static void -trim_filesystem ( -struct scrub_ctx *ctx) -{ -fstrim (ctx); -progress_add (1 ); -} -/* Trim the filesystem, if desired. */ int phase8_func(
@@ -47,7 +38,8 @@ phase8_func(return 0 ; maybe_trim: -trim_filesystem (ctx); +fstrim (ctx); +progress_add (1 );return 0 ; }
PATCH 4/8 xfs_scrub: fix the work estimation for phase 8
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If there are latent errors on the filesystem, we aren't going to do any work during phase 8 and it makes no sense to add that into the work estimate for the progress bar.
--- scrub/phase8.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)
scrub/phase8.c +26 -10
@@ -21,23 +21,35 @@/* Phase 8: Trim filesystem. */ -/* Trim the filesystem, if desired. */ -int -phase8_func ( +static inline bool +fstrim_ok (struct scrub_ctx *ctx) { -if (action_list_empty (ctx->fs_repair_list) && -action_list_empty (ctx->file_repair_list)) - goto maybe_trim; -/* * If errors remain on the filesystem, do not trim anything. We don't * have any threads running, so it's ok to skip the ctx lock here. */ -if (ctx->corruptions_found || ctx->unfixable_errors !=0 ) +if (!action_list_empty (ctx->fs_repair_list)) +return false; +if (!action_list_empty (ctx->file_repair_list)) +return false; + +if (ctx->corruptions_found !=0 ) +return false; +if (ctx->unfixable_errors !=0 ) +return false; + +return true; +} + +/* Trim the filesystem, if desired. */ +int +phase8_func ( +struct scrub_ctx *ctx) +{ +if (!fstrim_ok (ctx))return 0 ; -maybe_trim:fstrim (ctx);progress_add (1 );return 0 ;
@@ -51,7 +63,11 @@ phase8_estimate(unsigned int *nr_threads, int *rshift) { - *items =1 ; + *items =0 ; + +if (fstrim_ok (ctx)) + *items =1 ; + *nr_threads =1 ; *rshift =0 ;return 0 ;
PATCH 5/8 xfs_scrub: report FITRIM errors properly
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Move the error reporting for the FITRIM ioctl out of vfs.c and into phase8.c. This makes it so that IO errors encountered during trim are counted as runtime errors instead of being dropped silently.
--- scrub/phase8.c | 12 +++++++++++- scrub/vfs.c | 12 +++++++----- scrub/vfs.h | 2 +- 3 files changed, 19 insertions(+), 7 deletions(-)
scrub/phase8.c +11 -1
@@ -47,10 +47,20 @@ intphase8_func ( struct scrub_ctx *ctx) { +int error; +if (!fstrim_ok (ctx))return 0 ; -fstrim (ctx); + error =fstrim (ctx); +if (error ==EOPNOTSUPP ) +return 0 ; + +if (error) { +str_liberror (ctx, error,_ ("fstrim" )); +return error; + } +progress_add (1 );return 0 ; }
scrub/vfs.c +7 -5
@@ -296,15 +296,17 @@ struct fstrim_range {#endif /* Call FITRIM to trim all the unused space in a filesystem. */ -void +int fstrim (struct scrub_ctx *ctx) {struct fstrim_range range = {0 }; -int error; range.len =ULLONG_MAX ; - error =ioctl (ctx->mnt.fd,FITRIM , &range); -if (error && errno !=EOPNOTSUPP && errno !=ENOTTY ) -perror (_ ("fstrim" )); +if (ioctl (ctx->mnt.fd,FITRIM , &range) ==0 ) +return 0 ; +if (errno ==EOPNOTSUPP || errno ==ENOTTY ) +return EOPNOTSUPP ; + +return errno; }
scrub/vfs.h +1 -1
@@ -24,6 +24,6 @@ typedef int (*scan_fs_tree_dirent_fn)(struct scrub_ctx *, const char *,int scan_fs_tree (struct scrub_ctx *ctx,scan_fs_tree_dir_fn dir_fn,scan_fs_tree_dirent_fn dirent_fn,void *arg); -void fstrim (struct scrub_ctx *ctx); +int fstrim (struct scrub_ctx *ctx);#endif /* XFS_SCRUB_VFS_H_ */
PATCH 6/8 xfs_scrub: don't call FITRIM after runtime errors
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Don't call FITRIM if there have been runtime errors -- we don't want to touch anything after any kind of unfixable problem.
--- scrub/phase8.c | 3 +++ 1 file changed, 3 insertions(+)
scrub/phase8.c +3 -0
@@ -39,6 +39,9 @@ fstrim_ok(if (ctx->unfixable_errors !=0 )return false; +if (ctx->runtime_errors !=0 ) +return false; +return true; }
PATCH 7/8 xfs_scrub: don't trim the first agbno of each AG for better performance
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
XFS issues discard IOs while holding the free space btree and the AGF buffers locked. If the discard IOs are slow or the free space is extremely fragmented, this can lead to long stalls for every other thread trying to access that AG. On a 10TB high performance flash storage device with a severely fragmented free space btree in every AG, this results in many threads tripping the hangcheck warnings while waiting for the AGF. This happens even after we've run fstrim a few times and waited for the nvme namespace utilization counters to stabilize.
Strace for the entire 10TB looks like:ioctl(3, FITRIM, {start=0x0, len=10995116277760, minlen=0}) = 0 <686.209839>
Reducing the size of the FITRIM requests to a single AG at a time produces lower times for each individual call, but even this isn't quite acceptable, because the lock hold times are still high enough to cause stall warnings:
Strace for the first 4x 1TB AGs looks like (2):ioctl(3, FITRIM, {start=0x0, len=1099511627776, minlen=0}) = 0 <68.352033> ioctl(3, FITRIM, {start=0x10000000000, len=1099511627776, minlen=0}) = 0 <68.760323> ioctl(3, FITRIM, {start=0x20000000000, len=1099511627776, minlen=0}) = 0 <67.235226> ioctl(3, FITRIM, {start=0x30000000000, len=1099511627776, minlen=0}) = 0 <69.465744>
The fstrim code has to synchronize discards with block allocations, so we must hold the AGF lock while issuing discard IOs. Breaking up the calls into smaller start/len segments ought to reduce the lock hold time and allow other threads a chance to make progress. Unfortunately, the current fstrim implementation handles this poorly because it walks the entire free space by length index (cntbt) and it's not clear if we can cycle the AGF periodically to reduce latency because there's no less-than btree lookup.
The first solution I thought of was to limit latency by scanning parts of an AG at a time, but this doesn't solve the stalling problem when the free space is heavily fragmented because each sub-AG scan has to walk the entire cntbt to find free space that fits within the given range. In fact, this dramatically increases the runtime!
Ultimately, I forked the kernel implementations -- for full AG fstrims, it still trims by length. However, for a sub-AG scan, it will walk the bnobt and perform the trims in block number order. Since the cursor has an obviously monotonically increasing value, it is easy to cycle the AGF periodically to allow other threads to do work. This implementation avoids the worst problems of the original code, though it lacks the desirable attribute of freeing the biggest chunks first.
This second algorithm is what we want for xfs_scrub, which generally runs as a background service. Skip the first block of each AG to ensure that we get the sub-AG algorithm,
--- scrub/phase8.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++--------- scrub/vfs.c | 10 ++++++--- scrub/vfs.h | 2 +- 3 files changed, 61 insertions(+), 14 deletions(-)
scrub/phase8.c +53 -10
@@ -45,29 +45,72 @@ fstrim_ok(return true; } -/* Trim the filesystem, if desired. */ -int -phase8_func ( -struct scrub_ctx *ctx) +/* Trim a certain range of the filesystem. */ +static int +fstrim_fsblocks ( +struct scrub_ctx *ctx, +uint64_t start_fsb, +uint64_t fsbcount) { +uint64_t start =cvt_off_fsb_to_b (&ctx->mnt, start_fsb); +uint64_t len =cvt_off_fsb_to_b (&ctx->mnt, fsbcount);int error; -if (!fstrim_ok (ctx)) -return 0 ; - - error =fstrim (ctx); + error =fstrim (ctx, start, len);if (error ==EOPNOTSUPP )return 0 ; -if (error) { -str_liberror (ctx, error,_ ("fstrim" )); +char descr[DESCR_BUFSZ ]; + +snprintf (descr,sizeof (descr) -1 , +_ ("fstrim start 0x%llx len 0x%llx" ), + (unsigned long long )start, + (unsigned long long )len); +str_liberror (ctx, error, descr);return error; } +return 0 ; +} + +/* Trim each AG on the data device. */ +static int +fstrim_datadev ( +struct scrub_ctx *ctx) +{ +struct xfs_fsop_geom *geo = &ctx->mnt.fsgeom; +uint64_t fsbno; +int error; + +for (fsbno =0 ; fsbno < geo->datablocks; fsbno += geo->agblocks) { +uint64_t fsbcount; + +/* +* Skip the first block of each AG to ensure that we get the +* partial-AG discard implementation, which cycles the AGF lock +* to prevent foreground threads from stalling. +*/ + fsbcount =min (geo->datablocks - fsbno +1 , geo->agblocks); + error =fstrim_fsblocks (ctx, fsbno +1 , fsbcount); +if (error) +return error; + } +progress_add (1 );return 0 ; } +/* Trim the filesystem, if desired. */ +int +phase8_func ( +struct scrub_ctx *ctx) +{ +if (!fstrim_ok (ctx)) +return 0 ; + +return fstrim_datadev (ctx); +} +/* Estimate how much work we're going to do. */ int phase8_estimate(
scrub/vfs.c +7 -3
@@ -298,11 +298,15 @@ struct fstrim_range {/* Call FITRIM to trim all the unused space in a filesystem. */ int fstrim ( -struct scrub_ctx *ctx) +struct scrub_ctx *ctx, +uint64_t start, +uint64_t len) { -struct fstrim_range range = {0 }; +struct fstrim_range range = { + .start = start, + .len = len, + }; - range.len =ULLONG_MAX ;if (ioctl (ctx->mnt.fd,FITRIM , &range) ==0 )return 0 ;if (errno ==EOPNOTSUPP || errno ==ENOTTY )
scrub/vfs.h +1 -1
@@ -24,6 +24,6 @@ typedef int (*scan_fs_tree_dirent_fn)(struct scrub_ctx *, const char *,int scan_fs_tree (struct scrub_ctx *ctx,scan_fs_tree_dir_fn dir_fn,scan_fs_tree_dirent_fn dirent_fn,void *arg); -int fstrim (struct scrub_ctx *ctx); +int fstrim (struct scrub_ctx *ctx,uint64_t start,uint64_t len);#endif /* XFS_SCRUB_VFS_H_ */
PATCH 8/8 xfs_scrub: improve progress meter for phase 8 fstrimming
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, progress reporting in phase 8 is awful, because we stall at 0% until jumping to 100%. Since we're now performing sub-AG fstrim calls to limit the latency impacts to the rest of the system, we might as well limit the FSTRIM scan size so that we can report status updates to the user more regularly. Doing so also facilitates CPU usage control during phase 8.
--- scrub/phase8.c | 59 ++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 19 deletions(-)
scrub/phase8.c +40 -19
@@ -45,6 +45,13 @@ fstrim_ok(return true; } +/* +* Limit the amount of fstrim scanning that we let the kernel do in a single +* call so that we can implement decent progress reporting and CPU resource +* control. Pick a prime number of gigabytes for interest. +*/ +#define FSTRIM_MAX_BYTES (11ULL << 30) +/* Trim a certain range of the filesystem. */ static int fstrim_fsblocks(
@@ -56,18 +63,31 @@ fstrim_fsblocks(uint64_t len =cvt_off_fsb_to_b (&ctx->mnt, fsbcount);int error; - error =fstrim (ctx, start, len); -if (error ==EOPNOTSUPP ) -return 0 ; -if (error) { -char descr[DESCR_BUFSZ ]; - -snprintf (descr,sizeof (descr) -1 , -_ ("fstrim start 0x%llx len 0x%llx" ), - (unsigned long long )start, - (unsigned long long )len); -str_liberror (ctx, error, descr); -return error; +while (len >0 ) { +uint64_t run; + + run =min (len,FSTRIM_MAX_BYTES ); + + error =fstrim (ctx, start, run); +if (error ==EOPNOTSUPP ) { +/* Pretend we finished all the work. */ +progress_add (len); +return 0 ; + } +if (error) { +char descr[DESCR_BUFSZ ]; + +snprintf (descr,sizeof (descr) -1 , +_ ("fstrim start 0x%llx run 0x%llx" ), + (unsigned long long )start, + (unsigned long long )run); +str_liberror (ctx, error, descr); +return error; + } + +progress_add (run); + len -= run; + start += run; }return 0 ;
@@ -90,13 +110,13 @@ fstrim_datadev(* partial-AG discard implementation, which cycles the AGF lock * to prevent foreground threads from stalling. */ +progress_add (geo->blocksize); fsbcount =min (geo->datablocks - fsbno +1 , geo->agblocks); error =fstrim_fsblocks (ctx, fsbno +1 , fsbcount);if (error)return error; } -progress_add (1 );return 0 ; }
@@ -119,12 +139,13 @@ phase8_estimate(unsigned int *nr_threads, int *rshift) { - *items =0 ; - -if (fstrim_ok (ctx)) - *items =1 ; - + if (fstrim_ok(ctx)) { + *items =cvt_off_fsb_to_b (&ctx->mnt, + ctx->mnt.fsgeom.datablocks); + } else { + *items =0 ; + } *nr_threads =1 ; - *rshift =0 ; + *rshift =30 ;/* GiB */ return 0 ; }
PATCH 33/40 xfs_scrub: use free space histograms to reduce fstrim runtime
In reply to Darrick J. WongHi all,
This patchset dramatically reduces the runtime of the FITRIM calls made during phase 8 of xfs_scrub. It turns out that phase 8 can really get bogged down if the free space contains a large number of very small extents. In these cases, the runtime can increase by an order of magnitude to free less than 1% of the free space. This is not worth the time, since we're spending a lot of time to do very little work. The FITRIM ioctl allows us to specify a minimum extent length, so we can use statistical methods to compute a minlen parameter.
It turns out xfs_db/spaceman already have the code needed to create histograms of free space extent lengths. We add the ability to compute a CDF of the extent lengths, which make it easy to pick a minimum length corresponding to 99% of the free space. In most cases, this results in dramatic reductions in phase 8 runtime. Hence, move the histogram code to libfrog, and wire up xfs_scrub, since phase 7 already walks the fsmap.
We also add a new -o suboption to xfs_scrub so that people who /do/ want to examine every free extent can do so.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
--- db/freesp.c | 83 +++------------- libfrog/Makefile | 2 libfrog/histogram.c | 252 ++++++++++++++++++++++++++++++++++++++++++++++++++ libfrog/histogram.h | 54 +++++++++++ man/man8/xfs_scrub.8 | 16 +++ scrub/phase7.c | 47 +++++++++ scrub/phase8.c | 75 ++++++++++++++- scrub/spacemap.c | 11 +- scrub/vfs.c | 4 + scrub/vfs.h | 2 scrub/xfs_scrub.c | 45 +++++++++ scrub/xfs_scrub.h | 16 +++ spaceman/freesp.c | 93 +++++------------- 13 files changed, 544 insertions(+), 156 deletions(-) create mode 100644 libfrog/histogram.c create mode 100644 libfrog/histogram.h
PATCH 1/7 libfrog: hoist free space histogram code
From: Darrick J. Wong <[email protected]>
Combine the two free space histograms in xfs_db and xfs_spaceman into a single implementation.
--- db/freesp.c | 83 +++++-------------------------- libfrog/Makefile | 2 + libfrog/histogram.c | 135 +++++++++++++++++++++++++++++++++++++++++++++++++++ libfrog/histogram.h | 50 +++++++++++++++++++ spaceman/freesp.c | 93 +++++++++-------------------------- 5 files changed, 225 insertions(+), 138 deletions(-) create mode 100644 libfrog/histogram.c create mode 100644 libfrog/histogram.h
db/freesp.c +14 -69
@@ -12,14 +12,7 @@#include "output.h" #include "init.h" #include "malloc.h" - -typedef struct histent -{ -int low; -int high; -long long count; -long long blocks; -}histent_t ; +#include "libfrog/histogram.h" static void addhistent (int h); static void addtohist(xfs_agnumber_t agno, xfs_agblock_t agbno,
@@ -46,13 +39,10 @@ static int alignment;static int countflag;static int dumpflag;static int equalsize; -static histent_t *hist; -static int histcount; +static struct histogram freesp_hist;static int multsize;static int seen1;static int summaryflag; -static long long totblocks; -static long long totexts;static const cmdinfo_t freesp_cmd = {"freesp" ,NULL , freesp_f,0 ,-1 ,0 ,
@@ -93,18 +83,13 @@ freesp_f(if (inaglist (agno))scan_ag (agno); } -if (histcount) +if (hist_buckets (&freesp_hist))printhist (); -if (summaryflag) { -dbprintf (_ ("total free extents %lld\n" ), totexts); -dbprintf (_ ("total free blocks %lld\n" ), totblocks); -dbprintf (_ ("average free extent size %g\n" ), - (double )totblocks / (double )totexts); - } +if (summaryflag) +hist_summarize (&freesp_hist);if (aglist)xfree (aglist); -if (hist) -xfree (hist); +hist_free (&freesp_hist);return 0 ; }
@@ -132,10 +117,9 @@ init(int speced =0 ; agcount = countflag = dumpflag = equalsize = multsize = optind =0 ; - histcount = seen1 = summaryflag =0 ; - totblocks = totexts =0 ; + seen1 = summaryflag =0 ; aglist =NULL ; - hist =NULL ; +while ((c =getopt (argc, argv,"A:a:bcde:h:m:s" )) !=EOF ) {switch (c) {case 'A' :
@@ -163,7 +147,7 @@ init( speced =1 ;break ;case 'h' : -if (speced && !histcount) +if (speced &&hist_buckets (&freesp_hist) ==0 )return usage ();addhistent (atoi (optarg)); speced =1 ;
@@ -339,14 +323,7 @@ static voidaddhistent ( int h) { - hist =xrealloc (hist, (histcount +1 ) *sizeof (*hist)); -if (h ==0 ) - h =1 ; - hist[histcount].low = h; - hist[histcount].count = hist[histcount].blocks =0 ; - histcount++; -if (h ==1 ) - seen1 =1 ; +hist_add_bucket (&freesp_hist, h); } static void
@@ -355,30 +332,12 @@ addtohist( xfs_agblock_t agbno, xfs_extlen_t len) { - int i; -if (alignment && (XFS_AGB_TO_FSB (mp,agno,agbno) % alignment))return ;if (dumpflag)dbprintf ("%8d %8d %8d\n" , agno, agbno, len); - totexts++; - totblocks += len; -for (i =0 ; i < histcount; i++) { -if (hist[i].high >= len) { - hist[i].count++; - hist[i].blocks += len; -break ; - } - } -} - -static int -hcmp ( -const void *a, -const void *b) -{ -return ((histent_t *)a)->low - ((histent_t *)b)->low; +hist_add (&freesp_hist, len); } static void
@@ -387,6 +346,7 @@ histinit( {int i; +hist_init (&freesp_hist);if (equalsize) {for (i =1 ; i < maxlen; i += equalsize)addhistent (i);
@@ -396,27 +356,12 @@ histinit( }else {if (!seen1)addhistent (1 ); -qsort (hist, histcount,sizeof (*hist), hcmp); - } -for (i =0 ; i < histcount; i++) { -if (i < histcount -1 ) - hist[i].high = hist[i +1 ].low -1 ; -else - hist[i].high = maxlen; } +hist_prepare (&freesp_hist, maxlen); }static void printhist (void ) { -int i; - -dbprintf ("%7s %7s %7s %7s %6s\n" , -_ ("from" ),_ ("to" ),_ ("extents" ),_ ("blocks" ),_ ("pct" )); -for (i =0 ; i < histcount; i++) { -if (hist[i].count) -dbprintf ("%7d %7d %7lld %7lld %6.2f\n" , hist[i].low, - hist[i].high, hist[i].count, hist[i].blocks, - hist[i].blocks *100.0 / totblocks); - } +hist_print (&freesp_hist); }
libfrog/Makefile +2 -0
@@ -20,6 +20,7 @@ convert.c \ crc32.c \ file_exchange.c \ fsgeom.c \ +histogram.c \ list_sort.c \ linux.c \ logging.c \
@@ -45,6 +46,7 @@ dahashselftest.h \ div64.h \ file_exchange.h \ fsgeom.h \ +histogram.h \ logging.h \ paths.h \ projects.h \
libfrog/histogram.c new file +135 -0
135 lines not on this page · raw message
libfrog/histogram.h new file +50 -0
50 lines not on this page · raw message
spaceman/freesp.c +24 -69
170 lines not on this page · raw message
PATCH 2/7 libfrog: print wider columns for free space histogram
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
The values reported here can reach very large values, so compute the column width dynamically.
--- libfrog/histogram.c | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-)
libfrog/histogram.c +29 -5
@@ -108,17 +108,41 @@ voidhist_print ( const struct histogram *hs) { +unsigned int from_w, to_w, extents_w, blocks_w;unsigned int i; -printf ("%7s %7s %7s %7s %6s\n" , -_ ("from" ),_ ("to" ),_ ("extents" ),_ ("blocks" ),_ ("pct" )); + from_w = to_w = extents_w = blocks_w =7 ; +for (i =0 ; i < hs->nr_buckets; i++) { +char buf[256 ]; + +if (!hs->buckets[i].count) +continue ; + +snprintf (buf,sizeof (buf) -1 ,"%lld" , hs->buckets[i].low); + from_w =max (from_w,strlen (buf)); + +snprintf (buf,sizeof (buf) -1 ,"%lld" , hs->buckets[i].high); + to_w =max (to_w,strlen (buf)); + +snprintf (buf,sizeof (buf) -1 ,"%lld" , hs->buckets[i].count); + extents_w =max (extents_w,strlen (buf)); + +snprintf (buf,sizeof (buf) -1 ,"%lld" , hs->buckets[i].blocks); + blocks_w =max (blocks_w,strlen (buf)); + } + +printf ("%*s %*s %*s %*s %6s\n" , + from_w,_ ("from" ), to_w,_ ("to" ), extents_w,_ ("extents" ), + blocks_w,_ ("blocks" ),_ ("pct" ));for (i =0 ; i < hs->nr_buckets; i++) {if (hs->buckets[i].count ==0 )continue ; -printf ("%7lld %7lld %7lld %7lld %6.2f\n" , - hs->buckets[i].low, hs->buckets[i].high, - hs->buckets[i].count, hs->buckets[i].blocks, +printf ("%*lld %*lld %*lld %*lld %6.2f\n" , + from_w, hs->buckets[i].low, + to_w, hs->buckets[i].high, + extents_w, hs->buckets[i].count, + blocks_w, hs->buckets[i].blocks, hs->buckets[i].blocks *100.0 / hs->totblocks); } }
PATCH 3/7 libfrog: print cdf of free space buckets
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Print the cumulative distribution function of the free space buckets in reverse order.
--- libfrog/histogram.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 59 insertions(+), 4 deletions(-)
libfrog/histogram.c +59 -4
@@ -103,13 +103,64 @@ hist_free(memset (hs,0 ,sizeof (struct histogram )); } +/* +* Compute the CDF of the free space in decreasing order of extent length. +* This enables users to determine how much free space is not in the long tail +* of small extents, e.g. 98% of the free space extents are larger than 31 +* blocks. +*/ +static int +hist_cdf ( +const struct histogram *hs, +struct histogram *cdf) +{ +struct histent *buckets; +int i = hs->nr_buckets -1 ; + +ASSERT (cdf->nr_buckets ==0 ); +ASSERT (hs->nr_buckets <INT_MAX ); + +if (hs->nr_buckets ==0 ) +return 0 ; + + buckets =calloc (hs->nr_buckets,sizeof (struct histent )); +if (!buckets) +return errno; + +memset (cdf,0 ,sizeof (struct histogram )); + cdf->buckets = buckets; + + cdf->buckets[i].count = hs->buckets[i].count; + cdf->buckets[i].blocks = hs->buckets[i].blocks; + i--; + +while (i >=0 ) { + cdf->buckets[i].count = hs->buckets[i].count + + cdf->buckets[i +1 ].count; + + cdf->buckets[i].blocks = hs->buckets[i].blocks + + cdf->buckets[i +1 ].blocks; + i--; + } + +return 0 ; +} +/* Dump a histogram to stdout. */ void hist_print (const struct histogram *hs) { +struct histogram cdf = { };unsigned int from_w, to_w, extents_w, blocks_w;unsigned int i; +int error; + + error =hist_cdf (hs, &cdf); +if (error) { +printf (_ ("histogram cdf: %s\n" ),strerror (error)); +return ; + } from_w = to_w = extents_w = blocks_w =7 ;for (i =0 ; i < hs->nr_buckets; i++) {
@@ -131,20 +182,24 @@ hist_print( blocks_w =max (blocks_w,strlen (buf)); } -printf ("%*s %*s %*s %*s %6s\n" , +printf ("%*s %*s %*s %*s %6s %6s %6s\n" , from_w,_ ("from" ), to_w,_ ("to" ), extents_w,_ ("extents" ), - blocks_w,_ ("blocks" ),_ ("pct" )); + blocks_w,_ ("blocks" ),_ ("pct" ),_ ("blkcdf" ),_ ("extcdf" ));for (i =0 ; i < hs->nr_buckets; i++) {if (hs->buckets[i].count ==0 )continue ; -printf ("%*lld %*lld %*lld %*lld %6.2f\n" , +printf ("%*lld %*lld %*lld %*lld %6.2f %6.2f %6.2f\n" , from_w, hs->buckets[i].low, to_w, hs->buckets[i].high, extents_w, hs->buckets[i].count, blocks_w, hs->buckets[i].blocks, - hs->buckets[i].blocks *100.0 / hs->totblocks); + hs->buckets[i].blocks *100.0 / hs->totblocks, + cdf.buckets[i].blocks *100.0 / hs->totblocks, + cdf.buckets[i].count *100.0 / hs->totexts); } + +hist_free (&cdf); }/* Summarize the contents of the histogram. */
PATCH 4/7 xfs_scrub: don't close stdout when closing the progress bar
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
When we're tearing down the progress bar file stream, check that it's not an alias of stdout before closing it.
--- scrub/xfs_scrub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scrub/xfs_scrub.c +1 -1
@@ -878,7 +878,7 @@ main(if (ctx.runtime_errors) ret |=SCRUB_RET_OPERROR ;phase_end (&all_pi,0 ); -if (progress_fp) +if (progress_fp &&fileno (progress_fp) !=1 )fclose (progress_fp);unicrash_unload ();
PATCH 5/7 xfs_scrub: remove pointless spacemap.c arguments
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Remove unused parameters from the full-device spacemap scan functions.
--- scrub/spacemap.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
scrub/spacemap.c +4 -7
@@ -132,7 +132,6 @@ scan_ag_rmaps(static void scan_dev_rmaps (struct scrub_ctx *ctx, -int idx,dev_t dev,struct scan_blocks *sbx) {
@@ -170,7 +169,7 @@ scan_rt_rmaps( {struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx; -scan_dev_rmaps (ctx, agno, ctx->fsinfo.fs_rtdev, arg); +scan_dev_rmaps (ctx, ctx->fsinfo.fs_rtdev, arg); }/* Iterate all the reverse mappings of the log device. */
@@ -182,7 +181,7 @@ scan_log_rmaps( {struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx; -scan_dev_rmaps (ctx, agno, ctx->fsinfo.fs_logdev, arg); +scan_dev_rmaps (ctx, ctx->fsinfo.fs_logdev, arg); }/*
@@ -210,8 +209,7 @@ scrub_scan_all_spacemaps(return ret; }if (ctx->fsinfo.fs_rt) { - ret = -workqueue_add (&wq, scan_rt_rmaps, - ctx->mnt.fsgeom.agcount +1 , &sbx); + ret = -workqueue_add (&wq, scan_rt_rmaps,0 , &sbx);if (ret) { sbx.aborted = true;str_liberror (ctx, ret,_ ("queueing rtdev fsmap work" ));
@@ -219,8 +217,7 @@ scrub_scan_all_spacemaps( } }if (ctx->fsinfo.fs_log) { - ret = -workqueue_add (&wq, scan_log_rmaps, - ctx->mnt.fsgeom.agcount +2 , &sbx); + ret = -workqueue_add (&wq, scan_log_rmaps,0 , &sbx);if (ret) { sbx.aborted = true;str_liberror (ctx, ret,_ ("queueing logdev fsmap work" ));
PATCH 6/7 xfs_scrub: collect free space histograms during phase 7
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Collect a histogram of free space observed during phase 7. We'll put this information to use in the next patch.
--- libfrog/histogram.c | 38 ++++++++++++++++++++++++++++++++++++++ libfrog/histogram.h | 3 +++ scrub/phase7.c | 47 +++++++++++++++++++++++++++++++++++++++++++++-- scrub/xfs_scrub.c | 5 +++++ scrub/xfs_scrub.h | 4 ++++ 5 files changed, 95 insertions(+), 2 deletions(-)
libfrog/histogram.c +38 -0
@@ -212,3 +212,41 @@ hist_summarize(printf (_ ("average free extent size %g\n" ), (double )hs->totblocks / (double )hs->totexts); } + +/* Copy the contents of src to dest. */ +void +hist_import ( +struct histogram *dest, +const struct histogram *src) +{ +unsigned int i; + +ASSERT (dest->nr_buckets == src->nr_buckets); + + dest->totblocks += src->totblocks; + dest->totexts += src->totexts; + +for (i =0 ; i < dest->nr_buckets; i++) { +ASSERT (dest->buckets[i].low == src->buckets[i].low); +ASSERT (dest->buckets[i].high == src->buckets[i].high); + + dest->buckets[i].count += src->buckets[i].count; + dest->buckets[i].blocks += src->buckets[i].blocks; + } +} + +/* +* Move the contents of src to dest and reinitialize src. dst must not +* contain any observations or buckets. +*/ +void +hist_move ( +struct histogram *dest, +struct histogram *src) +{ +ASSERT (dest->nr_buckets ==0 ); +ASSERT (dest->totexts ==0 ); + +memcpy (dest, src,sizeof (struct histogram )); +hist_init (src); +}
libfrog/histogram.h +3 -0
@@ -47,4 +47,7 @@ static inline unsigned int hist_buckets(const struct histogram *hs) return hs->nr_buckets; } +void hist_import (struct histogram *dest,const struct histogram *src); +void hist_move (struct histogram *dest,struct histogram *src); +#endif /* __LIBFROG_HISTOGRAM_H__ */
scrub/phase7.c +45 -2
92 lines not on this page · raw message
scrub/xfs_scrub.c +5 -0
@@ -18,6 +18,7 @@#include "descr.h" #include "unicrash.h" #include "progress.h" +#include "libfrog/histogram.h" /* * XFS Online Metadata Scrub (and Repair)
@@ -669,6 +670,8 @@ main(int ret =SCRUB_RET_SUCCESS ;int error; +hist_init (&ctx.datadev_hist); +fprintf (stdout,"EXPERIMENTAL xfs_scrub program in use! Use at your own risk!\n" );fflush (stdout);
@@ -882,6 +885,8 @@ main(fclose (progress_fp);unicrash_unload (); +hist_free (&ctx.datadev_hist); +/* * If we're being run as a service, the return code must fit the LSB * init script action error guidelines, which is to say that we
scrub/xfs_scrub.h +4 -0
@@ -7,6 +7,7 @@#define XFS_SCRUB_XFS_SCRUB_H_ #include "libfrog/fsgeom.h" +#include "libfrog/histogram.h" extern char *progname;
@@ -86,6 +87,9 @@ struct scrub_ctx {unsigned long long preens;bool scrub_setup_succeeded;bool preen_triggers[XFS_SCRUB_TYPE_NR ]; + +/* Free space histograms, in fsb */ +struct histogram datadev_hist; };/* Phase helper functions */
PATCH 7/7 xfs_scrub: tune fstrim minlen parameter based on free space histograms
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, phase 8 runs very slowly on filesystems with a lot of small free space extents. To reduce the amount of time spent on fstrim activities during phase 8, we want to balance estimated runtime against completeness of the trim. In short, the goal is to reduce runtime by avoiding small trim requests.
At the start of phase 8, a CDF is computed in decreasing order of extent length from the histogram buckets created during the fsmap scan in phase
7. A point corresponding to the fstrim percentage target is chosen from the CDF and mapped back to a histogram bucket, and free space extents smaller than that amount are ommitted from fstrim.
On my aging /home filesystem, the free space histogram reported by xfs_spaceman looks like this:
from to extents blocks pct blkcdf extcdf
1 1 121953 121953 0.04 100.00 100.00
2 3 124741 299694 0.09 99.96 81.16
4 7 113492 593763 0.18 99.87 61.89
8 15 109215 1179524 0.36 99.69 44.36
16 31 76972 1695455 0.52 99.33 27.48
32 63 48655 2219667 0.68 98.82 15.59
64 127 31398 2876898 0.88 98.14 8.08
128 255 8014 1447920 0.44 97.27 3.23
256 511 4142 1501758 0.46 96.82 1.99
512 1023 2433 1768732 0.54 96.37 1.35
1024 2047 1795 2648460 0.81 95.83 0.97
2048 4095 1429 4206103 1.28 95.02 0.69
4096 8191 1045 6162111 1.88 93.74 0.47
8192 16383 791 9242745 2.81 91.87 0.31
16384 32767 473 10883977 3.31 89.06 0.19
32768 65535 272 12385566 3.77 85.74 0.12
65536 131071 192 18098739 5.51 81.98 0.07
131072 262143 108 20675199 6.29 76.47 0.04
262144 524287 80 29061285 8.84 70.18 0.03
524288 1048575 39 29002829 8.83 61.33 0.02
1048576 2097151 25 36824985 11.21 52.51 0.01
2097152 4194303 32 101727192 30.95 41.30 0.01
4194304 8388607 7 34007410 10.35 10.35 0.00From this table, we see that free space extents that are 16 blocks or longer constitute 99.3% of the free space in the filesystem but only 27.5% of the extents. If we set the fstrim minlen parameter to 16 blocks, that means that we can trim over 99% of the space in one third of the time it would take to trim everything.
Add a new -o fstrim_pct= option to xfs_scrub just in case there are users out there who want a different percentage. For example, accepting a 95% trim would net us a speed increase of nearly two orders of magnitude, ignoring system call overhead. Setting it to 100% will trim everything, just like fstrim(8).
--- libfrog/histogram.c | 2 + libfrog/histogram.h | 1 + man/man8/xfs_scrub.8 | 16 +++++++++++ scrub/phase8.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++--- scrub/vfs.c | 4 ++- scrub/vfs.h | 2 + scrub/xfs_scrub.c | 38 +++++++++++++++++++++++++ scrub/xfs_scrub.h | 12 ++++++++ 8 files changed, 141 insertions(+), 9 deletions(-)
libfrog/histogram.c +1 -1
@@ -109,7 +109,7 @@ hist_free(* of small extents, e.g. 98% of the free space extents are larger than 31 * blocks. */ -static int +int hist_cdf (const struct histogram *hs,struct histogram *cdf)
libfrog/histogram.h +1 -0
@@ -39,6 +39,7 @@ void hist_add(struct histogram *hs, long long len);void hist_init (struct histogram *hs);void hist_prepare (struct histogram *hs,long long maxlen);void hist_free (struct histogram *hs); +int hist_cdf (const struct histogram *hs,struct histogram *cdf);void hist_print (const struct histogram *hs);void hist_summarize (const struct histogram *hs);
man/man8/xfs_scrub.8 +16 -0
@@ -100,6 +100,22 @@ The supported are: .RS 1.0i .TP +.BI fstrim_pct= percentage +To constrain the amount of time spent on fstrim activities during phase 8, +this program tries to balance estimated runtime against completeness of the +trim. +In short, the program avoids small trim requests to save time. + +During phase 7, a log-scale histogram of free space extents is constructed. +At the start of phase 8, a CDF is computed in decreasing order of extent +length from the histogram buckets. +A point corresponding to the fstrim percentage target is chosen from the CDF +and mapped back to a histogram bucket. +Free space extents at least as long as the bucket size are trimmed. +Smaller extents are ignored. + +By default, the percentage threshold is 99%. +.TP .BI iwarn Treat informational messages as warnings. This will result in a nonzero return code, and a higher logging level.
scrub/phase8.c +70 -5
123 lines not on this page · raw message
scrub/vfs.c +3 -1
@@ -300,11 +300,13 @@ intfstrim ( struct scrub_ctx *ctx, uint64_t start, - uint64_t len) + uint64_t len, + uint64_t minlen) { struct fstrim_range range = { .start = start, .len = len, + .minlen = minlen, };if (ioctl (ctx->mnt.fd,FITRIM , &range) ==0 )
scrub/vfs.h +1 -1
@@ -24,6 +24,6 @@ typedef int (*scan_fs_tree_dirent_fn)(struct scrub_ctx *, const char *,int scan_fs_tree (struct scrub_ctx *ctx,scan_fs_tree_dir_fn dir_fn,scan_fs_tree_dirent_fn dirent_fn,void *arg); -int fstrim (struct scrub_ctx *ctx,uint64_t start,uint64_t len); +int fstrim (struct scrub_ctx *ctx,uint64_t start,uint64_t len,uint64_t minlen);#endif /* XFS_SCRUB_VFS_H_ */
scrub/xfs_scrub.c +37 -1
@@ -622,11 +622,13 @@ report_outcome(*/ enum o_opt_nums {IWARN =0 , +FSTRIM_PCT ,O_MAX_OPTS , };static char *o_opts[] = { [IWARN ] ="iwarn" , + [FSTRIM_PCT ] ="fstrim_pct" , [O_MAX_OPTS ] =NULL , };
@@ -635,8 +637,11 @@ parse_o_opts(struct scrub_ctx *ctx, char *p) { + double dval; +while (*p !='\0' ) {char *val; +char *endp;switch (getsubopt (&p, o_opts, &val)) {case IWARN :
@@ -647,6 +652,35 @@ parse_o_opts( } info_is_warning = true;break ; +case FSTRIM_PCT : +if (!val) { +fprintf (stderr, +_ ("-o fstrim_pct requires a parameter\n" )); +usage (); + } + + errno =0 ; + dval =strtod (val, &endp); + +if (*endp) { +fprintf (stderr, +_ ("-o fstrim_pct must be a floating point number\n" )); +usage (); + } +if (errno) { +fprintf (stderr, +_ ("-o fstrim_pct: %s\n" ), +strerror (errno)); +usage (); + } +if (dval <=0 || dval >100 ) { +fprintf (stderr, +_ ("-o fstrim_pct must be larger than 0 and less than 100\n" )); +usage (); + } + + ctx->fstrim_block_pct = dval /100.0 ; +break ;default :usage ();break ;
@@ -659,7 +693,9 @@ main(int argc, char **argv) { - struct scrub_ctx ctx = {0}; + struct scrub_ctx ctx = { + .fstrim_block_pct =FSTRIM_BLOCK_PCT_DEFAULT , + };struct phase_rusage all_pi;char *mtab =NULL ;FILE *progress_fp =NULL ;
scrub/xfs_scrub.h +12 -0
@@ -90,8 +90,20 @@ struct scrub_ctx {/* Free space histograms, in fsb */ struct histogram datadev_hist; + +/* +* Pick the largest value for fstrim minlen such that we trim at least +* this much space per volume. +*/ +double fstrim_block_pct; }; +/* +* Trim only enough free space extents (in order of decreasing length) to +* ensure that this percentage of the free space is trimmed. +*/ +#define FSTRIM_BLOCK_PCT_DEFAULT (99.0 / 100.0) +/* Phase helper functions */ void xfs_shutdown_fs (struct scrub_ctx *ctx);int scrub_cleanup (struct scrub_ctx *ctx);
PATCH 34/40 xfs_scrub: fixes for systemd services
In reply to Darrick J. WongHi all,
This series fixes deficiencies in the systemd services that were created to manage background scans. First, improve the debian packaging so that services get installed at package install time. Next, fix copyright and spdx header omissions.
Finally, fix bugs in the mailer scripts so that scrub failures are reported effectively. Finally, fix xfs_scrub_all to deal with systemd restarts causing it to think that a scrub has finished before the service actually finishes.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-service-fixes
--- debian/rules | 1 + include/builddefs.in | 2 +- scrub/Makefile | 26 ++++++++++++++------ scrub/[email protected] | 6 ++--- scrub/xfs_scrub_all.in | 49 ++++++++++++++++---------------------- scrub/xfs_scrub_fail.in | 12 ++++++++- scrub/[email protected] | 4 ++- 7 files changed, 55 insertions(+), 45 deletions(-) rename scrub/{xfs_scrub_fail => xfs_scrub_fail.in} (62%)
Re: [PATCHSET v29.0 34/40] xfs_scrub: fixes for systemd services
Quoted 23 lines · Darrick J. Wong
Hi all,
This series fixes deficiencies in the systemd services that were created to manage background scans. First, improve the debian packaging so that services get installed at package install time. Next, fix copyright and spdx header omissions.
Finally, fix bugs in the mailer scripts so that scrub failures are reported effectively. Finally, fix
xfs_scrub_allto deal with systemd restarts causing it to think that a scrub has finished before the service actually finishes.If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-service-fixes
Quoted 11 lines
--- debian/rules | 1 + include/builddefs.in | 2 +- scrub/Makefile | 26 ++++++++++++++------ scrub/[email protected] | 6 ++--- scrub/xfs_scrub_all.in | 49 ++++++++++++++++---------------------- scrub/xfs_scrub_fail.in | 12 ++++++++- scrub/[email protected] | 4 ++- 7 files changed, 55 insertions(+), 45 deletions(-) rename scrub/{xfs_scrub_fail => xfs_scrub_fail.in} (62%)
In your Makefile changes, you should be able to drop PKG_LIB_SCRIPT_DIR entirely from your Makefiles since it should be unused now, can you fold that into
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/commit/?h=scrub-service-fixes&id=1e0dce5c54270f1813f5661c266989917f08baf8
?
Signature · 1 line
真実はいつも一つ!/ Always, there's only one truth!
Re: [PATCHSET v29.0 34/40] xfs_scrub: fixes for systemd services
Quoted 36 lines · Neal Gompa
Quoted 23 lines · Darrick J. Wong
Hi all,
This series fixes deficiencies in the systemd services that were created to manage background scans. First, improve the debian packaging so that services get installed at package install time. Next, fix copyright and spdx header omissions.
Finally, fix bugs in the mailer scripts so that scrub failures are reported effectively. Finally, fix
xfs_scrub_allto deal with systemd restarts causing it to think that a scrub has finished before the service actually finishes.If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-service-fixesQuoted 11 lines
--- debian/rules | 1 + include/builddefs.in | 2 +- scrub/Makefile | 26 ++++++++++++++------ scrub/[email protected] | 6 ++--- scrub/xfs_scrub_all.in | 49 ++++++++++++++++---------------------- scrub/xfs_scrub_fail.in | 12 ++++++++- scrub/[email protected] | 4 ++- 7 files changed, 55 insertions(+), 45 deletions(-) rename scrub/{xfs_scrub_fail => xfs_scrub_fail.in} (62%)
Quoted 5 lines
In your
Makefilechanges, you should be able to dropPKG_LIB_SCRIPT_DIRentirely from your Makefiles since it should be unused now, can you fold that into
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/commit/?h=scrub-service-fixes&id=1e0dce5c54270f1813f5661c266989917f08baf8
?
Already done in:
https://lore.kernel.org/linux-xfs/170405001964.1800712.10514067731814883862.stgit@frogsfrogsfrogs/
Sorry I forgot to cc you there.
--D
Quoted 5 lines
Signature · 2 lines
-- 真実はいつも一つ!/ Always, there's only one truth!
PATCH 1/9 debian: install scrub services with dh_installsystemd
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use dh_installsystemd to handle the installation and activation of the scrub systemd services. This requires bumping the compat version to 11. Note that the services are /not/ activated on installation.
--- debian/rules | 1 + 1 file changed, 1 insertion(+)
debian/rules +1 -0
@@ -108,6 +108,7 @@ binary-arch: checkroot built dh_compress dh_fixperms dh_makeshlibs + dh_installsystemd -p xfsprogs --no-enable --no-start --no-restart-after-upgrade --no-stop-on-upgrade dh_installdeb dh_shlibdeps dh_gencontrol
PATCH 2/9 xfs_scrub_all: escape service names consistently
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
This program is not consistent as to whether or not it escapes the pathname that is being used as the xfs_scrub service instance name. Fix it to be consistent, and to fall back to direct invocation if escaping doesn't work. The escaping itself is also broken, but we'll fix that in the next patch.
--- scrub/xfs_scrub_all.in | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-)
scrub/xfs_scrub_all.in +16 -14
@@ -93,19 +93,19 @@ def run_killable(cmd, stdout, killfuncs, kill_fn): # that log messages from the service units preserve the full path and users can # look up log messages using full paths. However, for "/" the escaping rules # do /not/ drop the initial slash, so we have to special-case that here. -def systemd_escape(path): +def path_to_service(path): '''Escape a path to avoid mangled systemd mangling.''' if path == '/': - return '-' + return 'xfs_scrub@-' cmd = ['systemd-escape', '--path', path] try: proc = subprocess.Popen(cmd, stdout = subprocess.PIPE) proc.wait() for line in proc.stdout: - return '-' + line.decode(sys.stdout.encoding).strip() + return 'xfs_scrub@-%s' % line.decode(sys.stdout.encoding).strip() except: - return path + return None def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): '''Run a scrub process.'''
@@ -119,17 +119,19 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): return # Try it the systemd way - cmd=['systemctl', 'start', 'xfs_scrub@%s' % systemd_escape(mnt)] - ret = run_killable(cmd, DEVNULL(), killfuncs, \ - lambda proc: kill_systemd('xfs_scrub@%s' % mnt, proc)) - if ret == 0 or ret == 1: - print("Scrubbing %s done, (err=%d)" % (mnt, ret)) - sys.stdout.flush() - retcode |= ret - return + svcname = path_to_service(path) + if svcname is not None: + cmd=['systemctl', 'start', svcname] + ret = run_killable(cmd, DEVNULL(), killfuncs, \ + lambda proc: kill_systemd(svcname, proc)) + if ret == 0 or ret == 1: + print("Scrubbing %s done, (err=%d)" % (mnt, ret)) + sys.stdout.flush() + retcode |= ret + return - if terminate: - return + if terminate: + return # Invoke xfs_scrub manually cmd=['@sbindir@/xfs_scrub', '@scrub_args@', mnt]
PATCH 3/9 xfs_scrub: fix pathname escaping across all service definitions
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
systemd services provide an "instance name" that can be associated with a particular invocation of a service. This allows service users to invoke multiple copies of a service, each with a unique string. For xfs_scrub, we pass the mountpoint of the filesystem as the instance name. However, systemd services aren't supposed to have slashes in them, so we're supposed to escape them.
The canonical escaping scheme for pathnames is defined by the systemd-escape --path command. Unfortunately, we've been adding our own opinionated sauce for years, to work around the fact that --path didn't exist in systemd before January 2017. The special sauce is incorrect, and we no longer care about systemd of 7 years past.
Clean up this mess by following the systemd escaping scheme throughout the service units. Now we can use the '%f' specifier in them, which makes things a lot less complicated.
--- scrub/Makefile | 19 +++++++++++++++---- scrub/[email protected] | 6 +++--- scrub/xfs_scrub_all.in | 33 +++++++++++---------------------- scrub/xfs_scrub_fail.in | 5 ++++- scrub/[email protected] | 4 ++-- 5 files changed, 35 insertions(+), 32 deletions(-) rename scrub/{xfs_scrub_fail => xfs_scrub_fail.in} (75%)
scrub/Makefile +15 -4
@@ -8,14 +8,17 @@ include $(builddefs) SCRUB_PREREQS=$(HAVE_OPENAT)$(HAVE_FSTATAT)$(HAVE_GETFSMAP) [email protected] + ifeq ($(SCRUB_PREREQS),yesyesyes) LTCOMMAND = xfs_scrub INSTALL_SCRUB = install-scrub XFS_SCRUB_ALL_PROG = xfs_scrub_all +XFS_SCRUB_FAIL_PROG = xfs_scrub_fail XFS_SCRUB_ARGS = -b -n ifeq ($(HAVE_SYSTEMD),yes) INSTALL_SCRUB += install-systemd -SYSTEMD_SERVICES = [email protected] xfs_scrub_all.service xfs_scrub_all.timer [email protected] +SYSTEMD_SERVICES = $(scrub_svcname) xfs_scrub_all.service xfs_scrub_all.timer [email protected] OPTIONAL_TARGETS += $(SYSTEMD_SERVICES) endif ifeq ($(HAVE_CROND),yes)
@@ -108,17 +111,25 @@ ifeq ($(HAVE_HDIO_GETGEO),yes) LCFLAGS += -DHAVE_HDIO_GETGEO endif -LDIRT = $(XFS_SCRUB_ALL_PROG) *.service *.cron +LDIRT = $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) *.service *.cron -default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(OPTIONAL_TARGETS) +default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) $(OPTIONAL_TARGETS) xfs_scrub_all: xfs_scrub_all.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ $(Q)chmod a+x $@ +xfs_scrub_fail: xfs_scrub_fail.in $(builddefs) + @echo " [SED] $@" + $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ + -e "s|@pkg_version@|$(PKG_VERSION)|g" < $< > $@ + $(Q)chmod a+x $@ + phase5.o unicrash.o xfs.o: $(builddefs) include $(BUILDRULES)
@@ -141,7 +152,7 @@ install-systemd: default $(SYSTEMD_SERVICES) $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR) $(INSTALL) -m 644 $(SYSTEMD_SERVICES) $(SYSTEMD_SYSTEM_UNIT_DIR) $(INSTALL) -m 755 -d $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) - $(INSTALL) -m 755 xfs_scrub_fail $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) + $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) install-crond: default $(CRONTABS) $(INSTALL) -m 755 -d $(CROND_DIR)
scrub/[email protected] +3 -3
@@ -4,7 +4,7 @@ # Author: Darrick J. Wong <[email protected]> [Unit] -Description=Online XFS Metadata Check for %I +Description=Online XFS Metadata Check for %f OnFailure=xfs_scrub_fail@%i.service Documentation=man:xfs_scrub(8)
@@ -13,7 +13,7 @@ Type=oneshot PrivateNetwork=true ProtectSystem=full ProtectHome=read-only -# Disable private /tmp just in case %i is a path under /tmp. +# Disable private /tmp just in case %f is a path under /tmp. PrivateTmp=no AmbientCapabilities=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO NoNewPrivileges=yes
@@ -21,5 +21,5 @@ User=nobody IOSchedulingClass=idle CPUSchedulingPolicy=idle Environment=SERVICE_MODE=1 -ExecStart=@sbindir@/xfs_scrub @scrub_args@ %I +ExecStart=@sbindir@/xfs_scrub @scrub_args@ %f SyslogIdentifier=%N
scrub/xfs_scrub_all.in +11 -22
@@ -81,29 +81,18 @@ def run_killable(cmd, stdout, killfuncs, kill_fn): return -1 # systemd doesn't like unit instance names with slashes in them, so it -# replaces them with dashes when it invokes the service. However, it's not -# smart enough to convert the dashes to something else, so when it unescapes -# the instance name to feed to xfs_scrub, it turns all dashes into slashes. -# "/moo-cow" becomes "-moo-cow" becomes "/moo/cow", which is wrong. systemd -# actually /can/ escape the dashes correctly if it is told that this is a path -# (and not a unit name), but it didn't do this prior to January 2017, so fix -# this for them. -# -# systemd path escaping also drops the initial slash so we add that back in so -# that log messages from the service units preserve the full path and users can -# look up log messages using full paths. However, for "/" the escaping rules -# do /not/ drop the initial slash, so we have to special-case that here. -def path_to_service(path): - '''Escape a path to avoid mangled systemd mangling.''' +# replaces them with dashes when it invokes the service. Filesystem paths +# need a special --path argument so that dashes do not get mangled. +def path_to_serviceunit(path): + '''Convert a pathname into a systemd service unit name.''' - if path == '/': - return 'xfs_scrub@-' - cmd = ['systemd-escape', '--path', path] + cmd = ['systemd-escape', '--template', '@scrub_svcname@', + '--path', path] try: proc = subprocess.Popen(cmd, stdout = subprocess.PIPE) proc.wait() for line in proc.stdout: - return 'xfs_scrub@-%s' % line.decode(sys.stdout.encoding).strip() + return line.decode(sys.stdout.encoding).strip() except: return None
@@ -119,11 +108,11 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): return # Try it the systemd way - svcname = path_to_service(path) - if svcname is not None: - cmd=['systemctl', 'start', svcname] + unitname = path_to_serviceunit(path) + if unitname is not None: + cmd=['systemctl', 'start', unitname] ret = run_killable(cmd, DEVNULL(), killfuncs, \ - lambda proc: kill_systemd(svcname, proc)) + lambda proc: kill_systemd(unitname, proc)) if ret == 0 or ret == 1: print("Scrubbing %s done, (err=%d)" % (mnt, ret)) sys.stdout.flush()
scrub/xfs_scrub_fail.in renamed from scrub/xfs_scrub_fail +4 -1
@@ -19,6 +19,9 @@ if [ ! -x "${mailer}" ]; then exit 1 fi +# Turn the mountpoint into a properly escaped systemd instance name +scrub_svc="$(systemd-escape --template "@scrub_svcname@" --path "${mntpoint}")" + (cat << ENDL To: $1 From: <xfs_scrub@${hostname}>
@@ -28,4 +31,4 @@ So sorry, the automatic xfs_scrub of ${mntpoint} on ${hostname} failed. A log of what happened follows: ENDL -systemctl status --full --lines 4294967295 "xfs_scrub@${mntpoint}") | "${mailer}" -t -i +systemctl status --full --lines 4294967295 "${scrub_svc}") | "${mailer}" -t -i
scrub/[email protected] +2 -2
@@ -4,13 +4,13 @@ # Author: Darrick J. Wong <[email protected]> [Unit] -Description=Online XFS Metadata Check Failure Reporting for %I +Description=Online XFS Metadata Check Failure Reporting for %f Documentation=man:xfs_scrub(8) [Service] Type=oneshot Environment=EMAIL_ADDR=root -ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_fail "${EMAIL_ADDR}" %I +ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_fail "${EMAIL_ADDR}" %f User=mail Group=mail SupplementaryGroups=systemd-journal
PATCH 4/9 xfs_scrub_fail: fix sendmail detection
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
This script emails the results of failed scrub runs to root. We shouldn't be hardcoding the path to the mailer program because distros can change the path according to their whim. Modify this script to use command -v to find the program.
--- scrub/xfs_scrub_fail.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scrub/xfs_scrub_fail.in +2 -1
@@ -7,13 +7,14 @@ # Email logs of failed xfs_scrub unit runs -mailer=/usr/sbin/sendmail recipient="$1" test -z "${recipient}" && exit 0 mntpoint="$2" test -z "${mntpoint}" && exit 0 hostname="$(hostname -f 2>/dev/null)" test -z "${hostname}" && hostname="${HOSTNAME}" + +mailer="$(command -v sendmail)" if [ ! -x "${mailer}" ]; then echo "${mailer}: Mailer program not found." exit 1
PATCH 5/9 xfs_scrub_fail: return the failure status of the mailer program
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
We should return the exit code of the mailer program sending the scrub failure reports, since that's much more important to anyone watching the system.
--- scrub/xfs_scrub_fail.in | 1 + 1 file changed, 1 insertion(+)
scrub/xfs_scrub_fail.in +1 -0
@@ -33,3 +33,4 @@ So sorry, the automatic xfs_scrub of ${mntpoint} on ${hostname} failed. A log of what happened follows: ENDL systemctl status --full --lines 4294967295 "${scrub_svc}") | "${mailer}" -t -i +exit "${PIPESTATUS[1]}"
PATCH 6/9 xfs_scrub_fail: add content type header to failure emails
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add content type and encoding metadata so that these emails display correctly.
--- scrub/xfs_scrub_fail.in | 2 ++ 1 file changed, 2 insertions(+)
scrub/xfs_scrub_fail.in +2 -0
@@ -27,6 +27,8 @@ scrub_svc="$(systemd-escape --template "@scrub_svcname@" --path "${mntpoint}")" To: $1 From: <xfs_scrub@${hostname}> Subject: xfs_scrub failure on ${mntpoint} +Content-Transfer-Encoding: 8bit +Content-Type: text/plain; charset=UTF-8 So sorry, the automatic xfs_scrub of ${mntpoint} on ${hostname} failed.
Re: [PATCH 6/9] xfs_scrub_fail: add content type header to failure emails
Looks good:
PATCH 7/9 xfs_scrub_fail: advise recipients not to reply
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Advise recipients of the service failure emails that they should not try to reply to the automated service message.
--- scrub/xfs_scrub_fail.in | 1 + 1 file changed, 1 insertion(+)
scrub/xfs_scrub_fail.in +1 -0
@@ -31,6 +31,7 @@ Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 So sorry, the automatic xfs_scrub of ${mntpoint} on ${hostname} failed. +Please do not reply to this mesage. A log of what happened follows: ENDL
Re: [PATCH 7/9] xfs_scrub_fail: advise recipients not to reply
Looks good:
PATCH 8/9 xfs_scrub_fail: move executable script to /usr/libexec
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Per FHS 3.0, non-PATH executable binaries are supposed to live under /usr/libexec, not /usr/lib. xfs_scrub_fail is an executable script, so move it to libexec in case some distro some day tries to mount /usr/lib as noexec or something.
--- include/builddefs.in | 1 + scrub/Makefile | 7 +++---- scrub/[email protected] | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
include/builddefs.in +1 -0
@@ -52,6 +52,7 @@ PKG_ROOT_SBIN_DIR = @root_sbindir@ PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@ PKG_LIB_DIR = @libdir@@libdirsuffix@ PKG_LIB_SCRIPT_DIR = @libdir@ +PKG_LIBEXEC_DIR = @libexecdir@/@pkg_name@ PKG_INC_DIR = @includedir@/xfs DK_INC_DIR = @includedir@/disk PKG_MAN_DIR = @mandir@
scrub/Makefile +3 -4
@@ -140,8 +140,7 @@ install: $(INSTALL_SCRUB) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ - -e "s|@pkg_lib_dir@|$(PKG_LIB_SCRIPT_DIR)|g" \ - -e "s|@pkg_name@|$(PKG_NAME)|g" \ + -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs)
@@ -151,8 +150,8 @@ install: $(INSTALL_SCRUB) install-systemd: default $(SYSTEMD_SERVICES) $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR) $(INSTALL) -m 644 $(SYSTEMD_SERVICES) $(SYSTEMD_SYSTEM_UNIT_DIR) - $(INSTALL) -m 755 -d $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) - $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) + $(INSTALL) -m 755 -d $(PKG_LIBEXEC_DIR) + $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIBEXEC_DIR) install-crond: default $(CRONTABS) $(INSTALL) -m 755 -d $(CROND_DIR)
scrub/[email protected] +1 -1
@@ -10,7 +10,7 @@ Documentation=man:xfs_scrub(8) [Service] Type=oneshot Environment=EMAIL_ADDR=root -ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_fail "${EMAIL_ADDR}" %f +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" %f User=mail Group=mail SupplementaryGroups=systemd-journal
Re: [PATCH 8/9] xfs_scrub_fail: move executable script to /usr/libexec
Quoted 61 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Per FHS 3.0, non-PATH executable binaries are supposed to live under
/usr/libexec, not/usr/lib.xfs_scrub_failis an executable script, so move it to libexec in case some distro some day tries to mount/usr/libas noexec or something.Cc Neal Gompa <[email protected]>Signed-off-by Darrick J. Wong <[email protected]>--- include/builddefs.in | 1 + scrub/Makefile | 7 +++---- scrub/[email protected] | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)include/builddefs.in +1
-0@@ -52,6 +52,7 @@ PKG_ROOT_SBIN_DIR = @root_sbindir@ PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@ PKG_LIB_DIR = @libdir@@libdirsuffix@ PKG_LIB_SCRIPT_DIR = @libdir@ +PKG_LIBEXEC_DIR = @libexecdir@/@pkg_name@ PKG_INC_DIR = @includedir@/xfs DK_INC_DIR = @includedir@/disk PKG_MAN_DIR = @mandir@scrub/Makefile +3
-4@@ -140,8 +140,7 @@ install: $(INSTALL_SCRUB) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ - -e "s|@pkg_lib_dir@|$(PKG_LIB_SCRIPT_DIR)|g" \ - -e "s|@pkg_name@|$(PKG_NAME)|g" \ + -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs)@@ -151,8 +150,8 @@ install: $(INSTALL_SCRUB) install-systemd: default $(SYSTEMD_SERVICES) $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR) $(INSTALL) -m 644 $(SYSTEMD_SERVICES) $(SYSTEMD_SYSTEM_UNIT_DIR) - $(INSTALL) -m 755 -d $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) - $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) + $(INSTALL) -m 755 -d $(PKG_LIBEXEC_DIR) + $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIBEXEC_DIR) install-crond: default $(CRONTABS) $(INSTALL) -m 755 -d $(CROND_DIR)scrub/[email protected] +0
-0
scrub/[email protected] Documentation=man:xfs_scrub(8) · line 16
Quoted 8 lines
scrub/[email protected] +1
-1@@ -10,7 +10,7 @@ Documentation=man:xfs_scrub(8) [Service] Type=oneshot Environment=EMAIL_ADDR=root -ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_fail "${EMAIL_ADDR}" %f +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" %f User=mail Group=mail SupplementaryGroups=systemd-journal
Looks great to me.
--
真実はいつも一つ!/ Always, there's only one truth!
Re: [PATCH 8/9] xfs_scrub_fail: move executable script to /usr/libexec
Quoted 73 lines · Neal Gompa
Quoted 61 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Per FHS 3.0, non-PATH executable binaries are supposed to live under
/usr/libexec, not/usr/lib.xfs_scrub_failis an executable script, so move it to libexec in case some distro some day tries to mount/usr/libas noexec or something.Cc Neal Gompa <[email protected]>Signed-off-by Darrick J. Wong <[email protected]>--- include/builddefs.in | 1 + scrub/Makefile | 7 +++---- scrub/[email protected] | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)include/builddefs.in +1
-0@@ -52,6 +52,7 @@ PKG_ROOT_SBIN_DIR = @root_sbindir@ PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@ PKG_LIB_DIR = @libdir@@libdirsuffix@ PKG_LIB_SCRIPT_DIR = @libdir@ +PKG_LIBEXEC_DIR = @libexecdir@/@pkg_name@ PKG_INC_DIR = @includedir@/xfs DK_INC_DIR = @includedir@/disk PKG_MAN_DIR = @mandir@scrub/Makefile +3
-4@@ -140,8 +140,7 @@ install: $(INSTALL_SCRUB) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ - -e "s|@pkg_lib_dir@|$(PKG_LIB_SCRIPT_DIR)|g" \ - -e "s|@pkg_name@|$(PKG_NAME)|g" \ + -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs)@@ -151,8 +150,8 @@ install: $(INSTALL_SCRUB) install-systemd: default $(SYSTEMD_SERVICES) $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR) $(INSTALL) -m 644 $(SYSTEMD_SERVICES) $(SYSTEMD_SYSTEM_UNIT_DIR) - $(INSTALL) -m 755 -d $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) - $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) + $(INSTALL) -m 755 -d $(PKG_LIBEXEC_DIR) + $(INSTALL) -m 755 $(XFS_SCRUB_FAIL_PROG) $(PKG_LIBEXEC_DIR) install-crond: default $(CRONTABS) $(INSTALL) -m 755 -d $(CROND_DIR)scrub/[email protected] +0
-0scrub/[email protected] Documentation=man:xfs_scrub(8) · line 16
Quoted 8 lines
scrub/[email protected] +1
-1@@ -10,7 +10,7 @@ Documentation=man:xfs_scrub(8) [Service] Type=oneshot Environment=EMAIL_ADDR=root -ExecStart=@pkg_lib_dir@/@pkg_name@/xfs_scrub_fail "${EMAIL_ADDR}" %f +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" %f User=mail Group=mail SupplementaryGroups=systemd-journalLooks great to me.
Quoted 1 line
Reviewed-by Neal Gompa <[email protected]>
Thanks!
--D
Quoted 6 lines
--
真実はいつも一つ!/ Always, there's only one truth!
Re: [PATCH 8/9] xfs_scrub_fail: move executable script to /usr/libexec
In reply to Darrick J. WongOh, libexec is back, that gives me strong 4.4-BSD vibes..
PATCH 9/9 xfs_scrub_all.cron: move to package data directory
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
cron jobs don't belong in /usr/lib. Since the cron job is also secondary to the systemd timer, it's really only provided as a courtesy for distributions that don't use systemd. Move it to @datadir@, aka /usr/share/xfsprogs.
--- include/builddefs.in | 1 - scrub/Makefile | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-)
include/builddefs.in +0 -1
@@ -51,7 +51,6 @@ PKG_SBIN_DIR = @sbindir@ PKG_ROOT_SBIN_DIR = @root_sbindir@ PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@ PKG_LIB_DIR = @libdir@@libdirsuffix@ -PKG_LIB_SCRIPT_DIR = @libdir@ PKG_LIBEXEC_DIR = @libexecdir@/@pkg_name@ PKG_INC_DIR = @includedir@/xfs DK_INC_DIR = @includedir@/disk
scrub/Makefile +1 -1
@@ -26,7 +26,7 @@ INSTALL_SCRUB += install-crond CRONTABS = xfs_scrub_all.cron OPTIONAL_TARGETS += $(CRONTABS) # Don't enable the crontab by default for now -CROND_DIR = $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) +CROND_DIR = $(PKG_DATA_DIR) endif endif # scrub_prereqs
Re: [PATCH 9/9] xfs_scrub_all.cron: move to package data directory
Quoted 34 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
cron jobs don't belong in
/usr/lib. Since the cron job is also secondary to the systemd timer, it's really only provided as a courtesy for distributions that don't use systemd. Move it to @datadir@, aka/usr/share/xfsprogs.Signed-off-by Darrick J. Wong <[email protected]>--- include/builddefs.in | 1 - scrub/Makefile | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-)include/builddefs.in +0
-1@@ -51,7 +51,6 @@ PKG_SBIN_DIR = @sbindir@ PKG_ROOT_SBIN_DIR = @root_sbindir@ PKG_ROOT_LIB_DIR= @root_libdir@@libdirsuffix@ PKG_LIB_DIR = @libdir@@libdirsuffix@ -PKG_LIB_SCRIPT_DIR = @libdir@ PKG_LIBEXEC_DIR = @libexecdir@/@pkg_name@ PKG_INC_DIR = @includedir@/xfs DK_INC_DIR = @includedir@/diskscrub/Makefile +0
-0
scrub/Makefile INSTALL_SCRUB += install-crond · line 32
Quoted 8 lines
scrub/Makefile +1
-1@@ -26,7 +26,7 @@ INSTALL_SCRUB += install-crond CRONTABS = xfs_scrub_all.cron OPTIONAL_TARGETS += $(CRONTABS) # Don't enable the crontab by default for now -CROND_DIR = $(PKG_LIB_SCRIPT_DIR)/$(PKG_NAME) +CROND_DIR = $(PKG_DATA_DIR) endif endif # scrub_prereqs
Looks good to me.
Signature · 1 line
真実はいつも一つ!/ Always, there's only one truth!
Re: [PATCH 9/9] xfs_scrub_all.cron: move to package data directory
In reply to Darrick J. WongQuoted 6 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
cron jobs don't belong in
/usr/lib. Since the cron job is also secondary to the systemd timer, it's really only provided as a courtesy for distributions that don't use systemd. Move it to @datadir@, aka/usr/share/xfsprogs.
Looks good:
Re: [PATCHSET v29.0 34/40] xfs_scrub: fixes for systemd services
In reply to Darrick J. WongCan we somehow expedite these plumbing fixes for the next xfsprogs release instead of just hiding them in the giant patchbomb?
Re: [PATCHSET v29.0 34/40] xfs_scrub: fixes for systemd services
Quoted 2 lines · Christoph Hellwig
Can we somehow expedite these plumbing fixes for the next xfsprogs release instead of just hiding them in the giant patchbomb?
I was planning to do that, though I don't think cem has merged any of the 4 pull requests I've already sent him for xfsprogs 6.6. :/
(Granted everyone has been on vacation for weeks, myself included, so I wasn't expecting much progress...)
--D
PATCH 35/40 xfs_scrub_all: fixes for systemd services
In reply to Darrick J. WongHi all,
This patchset ties up some problems in the xfs_scrub_all program and service, which are essential for finding mounted filesystems to scrub and creating the background service instances that do the scrub.
First, we need to fix various errors in pathname escaping, because systemd does /not/ like slashes in service names. Then, teach xfs_scrub_all to deal with systemd restarts causing it to think that a scrub has finished before the service actually finishes. Finally, implement a signal handler so that SIGINT (console ^C) and SIGTERM (systemd stopping the service) shut down the xfs_scrub@ services correctly.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scruball-service-fixes
--- scrub/xfs_scrub_all.in | 157 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 125 insertions(+), 32 deletions(-)
PATCH 1/4 xfs_scrub_all: fix argument passing when invoking xfs_scrub manually
From: Darrick J. Wong <[email protected]>
Currently, xfs_scrub_all will try to invoke xfs_scrub with argv[1] being "-n -x". This of course is recognized by C getopt as a weird looking string, not two individual arguments, and causes the child process to exit with complaints about CLI usage.
What we really want is to split the string into a proper array and then add them to the xfs_scrub command line. The code here isn't strictly correct, but as @scrub_args@ is controlled by us in the Makefile, it'll do for now.
--- scrub/xfs_scrub_all.in | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scrub/xfs_scrub_all.in +3 -1
@@ -123,7 +123,9 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): return # Invoke xfs_scrub manually - cmd=['@sbindir@/xfs_scrub', '@scrub_args@', mnt] + cmd = ['@sbindir@/xfs_scrub'] + cmd += '@scrub_args@'.split() + cmd += [mnt] ret = run_killable(cmd, None, killfuncs, \ lambda proc: proc.terminate()) if ret >= 0:
PATCH 2/4 xfs_scrub_all: survive systemd restarts when waiting for services
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If xfs_scrub_all detects a running systemd, it will use it to invoke xfs_scrub subprocesses in a sandboxed and resource-controlled environment. Unfortunately, if you happen to restart dbus or systemd while it's running, you get this:
systemd[1]: Reexecuting.
xfs_scrub_all[9958]: Warning! D-Bus connection terminated. xfs_scrub_all[9956]: Warning! D-Bus connection terminated. xfs_scrub_all[9956]: Failed to wait for response: Connection reset by peer xfs_scrub_all[9958]: Failed to wait for response: Connection reset by peer xfs_scrub_all[9930]: Scrubbing / done, (err=1) xfs_scrub_all[9930]: Scrubbing /storage done, (err=1)
The xfs_scrub units themselves are still running, it's just that the `systemctl start' command that xfs_scrub_all uses to start and wait for the unit lost its connection to dbus and hence is no longer monitoring sub-services.
When this happens, we don't have great options -- systemctl doesn't have a command to wait on an activating (aka running) unit. Emulate the functionality we normally get by polling the failed/active statuses.
--- scrub/xfs_scrub_all.in | 78 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 65 insertions(+), 13 deletions(-)
scrub/xfs_scrub_all.in +65 -13
@@ -14,6 +14,7 @@ import time import sys import os import argparse +from io import TextIOWrapper retcode = 0 terminate = False
@@ -58,12 +59,18 @@ def find_mounts(): return fs -def kill_systemd(unit, proc): - '''Kill systemd unit.''' - proc.terminate() - cmd=['systemctl', 'stop', unit] - x = subprocess.Popen(cmd) - x.wait() +def backtick(cmd): + '''Generator function that yields lines of a program's stdout.''' + p = subprocess.Popen(cmd, stdout = subprocess.PIPE) + for line in TextIOWrapper(p.stdout, encoding="utf-8"): + yield line.strip() + +def remove_killfunc(killfuncs, fn): + '''Ensure fn is not in killfuncs.''' + try: + killfuncs.remove(fn) + except: + pass def run_killable(cmd, stdout, killfuncs, kill_fn): '''Run a killable program. Returns program retcode or -1 if we can't start it.'''
@@ -72,10 +79,7 @@ def run_killable(cmd, stdout, killfuncs, kill_fn): real_kill_fn = lambda: kill_fn(proc) killfuncs.add(real_kill_fn) proc.wait() - try: - killfuncs.remove(real_kill_fn) - except: - pass + remove_killfunc(killfuncs, real_kill_fn) return proc.returncode except: return -1
@@ -96,6 +100,56 @@ def path_to_serviceunit(path): except: return None +def systemctl_stop(unitname): + '''Stop a systemd unit.''' + cmd = ['systemctl', 'stop', unitname] + x = subprocess.Popen(cmd) + x.wait() + +def systemctl_start(unitname, killfuncs): + '''Start a systemd unit and wait for it to complete.''' + stop_fn = None + cmd = ['systemctl', 'start', unitname] + try: + proc = subprocess.Popen(cmd, stdout = DEVNULL()) + stop_fn = lambda: systemctl_stop(unitname) + killfuncs.add(stop_fn) + proc.wait() + ret = proc.returncode + except: + if stop_fn is not None: + remove_killfunc(killfuncs, stop_fn) + return -1 + + if ret != 1: + remove_killfunc(killfuncs, stop_fn) + return ret + + # If systemctl-start returns 1, it's possible that the service failed + # or that dbus/systemd restarted and the client program lost its + # connection -- according to the systemctl man page, 1 means "unit not + # failed". + # + # Either way, we switch to polling the service status to try to wait + # for the service to end. As of systemd 249, the is-active command + # returns any of the following states: active, reloading, inactive, + # failed, activating, deactivating, or maintenance. Apparently these + # strings are not localized. + while True: + try: + for l in backtick(['systemctl', 'is-active', unitname]): + if l == 'failed': + remove_killfunc(killfuncs, stop_fn) + return 1 + if l == 'inactive': + remove_killfunc(killfuncs, stop_fn) + return 0 + except: + remove_killfunc(killfuncs, stop_fn) + return -1 + + time.sleep(1) + def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): '''Run a scrub process.''' global retcode, terminate
@@ -110,9 +164,7 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): # Try it the systemd way unitname = path_to_serviceunit(path) if unitname is not None: - cmd=['systemctl', 'start', unitname] - ret = run_killable(cmd, DEVNULL(), killfuncs, \ - lambda proc: kill_systemd(unitname, proc)) + ret = systemctl_start(unitname, killfuncs) if ret == 0 or ret == 1: print("Scrubbing %s done, (err=%d)" % (mnt, ret)) sys.stdout.flush()
PATCH 3/4 xfs_scrub_all: simplify cleanup of run_killable
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Get rid of the nested lambda functions to simplify the code.
--- scrub/xfs_scrub_all.in | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
scrub/xfs_scrub_all.in +6 -7
@@ -72,14 +72,14 @@ def remove_killfunc(killfuncs, fn): except: pass -def run_killable(cmd, stdout, killfuncs, kill_fn): - '''Run a killable program. Returns program retcode or -1 if we can't start it.''' +def run_killable(cmd, stdout, killfuncs): + '''Run a killable program. Returns program retcode or -1 if we can't + start it.''' try: proc = subprocess.Popen(cmd, stdout = stdout) - real_kill_fn = lambda: kill_fn(proc) - killfuncs.add(real_kill_fn) + killfuncs.add(proc.terminate) proc.wait() - remove_killfunc(killfuncs, real_kill_fn) + remove_killfunc(killfuncs, proc.terminate) return proc.returncode except: return -1
@@ -178,8 +178,7 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): cmd = ['@sbindir@/xfs_scrub'] cmd += '@scrub_args@'.split() cmd += [mnt] - ret = run_killable(cmd, None, killfuncs, \ - lambda proc: proc.terminate()) + ret = run_killable(cmd, None, killfuncs) if ret >= 0: print("Scrubbing %s done, (err=%d)" % (mnt, ret)) sys.stdout.flush()
PATCH 4/4 xfs_scrub_all: fix termination signal handling
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, xfs_scrub_all does not handle termination signals well. SIGTERM and SIGINT are left to their default handlers, which are immediate termination of the process group in the case of SIGTERM and raising KeyboardInterrupt in the case of SIGINT.
Terminating the process group is fine when the xfs_scrub processes are direct children, but this completely doesn't work if we're farming the work out to systemd services since we don't terminate the child service. Instead, they keep going.
Raising KeyboardInterrupt doesn't work because once the main thread calls sys.exit at the bottom of main(), it blocks in the python runtime waiting for child threads to terminate. There's no longer any context to handle an exception, so the signal is ignored and no child processes are killed.
In other words, if you try to kill a running xfs_scrub_all, chances are good it won't kill the child xfs_scrub processes. This is undesirable and egregious since we actually have the ability to track and kill all the subprocesses that we create.
Solve the subproblem of getting stuck in the python runtime by calling it repeatedly until we no longer have subprocesses. This means that the main thread loops until all threads have exited.
Solve the subproblem of the signals doing the wrong thing by setting up our own signal handler that can wake up the main thread and initiate subprocess shutdown, no matter whether the subprocesses are systemd services or directly fork/exec'd.
--- scrub/xfs_scrub_all.in | 64 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 52 insertions(+), 12 deletions(-)
scrub/xfs_scrub_all.in +52 -12
@@ -14,6 +14,7 @@ import time import sys import os import argparse +import signal from io import TextIOWrapper retcode = 0
@@ -196,6 +197,45 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): cond.notify() cond.release() +def signal_scrubs(signum, cond): + '''Handle termination signals by killing xfs_scrub children.''' + global debug, terminate + + if debug: + print('Signal handler called with signal', signum) + sys.stdout.flush() + + terminate = True + cond.acquire() + cond.notify() + cond.release() + +def wait_for_termination(cond, killfuncs): + '''Wait for a child thread to terminate. Returns True if we should + abort the program, False otherwise.''' + global debug, terminate + + if debug: + print('waiting for threads to terminate') + sys.stdout.flush() + + cond.acquire() + try: + cond.wait() + except KeyboardInterrupt: + terminate = True + cond.release() + + if not terminate: + return False + + print("Terminating...") + sys.stdout.flush() + while len(killfuncs) > 0: + fn = killfuncs.pop() + fn() + return True + def main(): '''Find mounts, schedule scrub runs.''' def thr(mnt, devs):
@@ -231,6 +271,10 @@ def main(): running_devs = set() killfuncs = set() cond = threading.Condition() + + signal.signal(signal.SIGINT, lambda s, f: signal_scrubs(s, cond)) + signal.signal(signal.SIGTERM, lambda s, f: signal_scrubs(s, cond)) + while len(fs) > 0: if len(running_devs) == 0: mnt, devs = fs.popitem()
@@ -250,18 +294,14 @@ def main(): thr(mnt, devs) for p in poppers: fs.pop(p) - cond.acquire() - try: - cond.wait() - except KeyboardInterrupt: - terminate = True - print("Terminating...") - sys.stdout.flush() - while len(killfuncs) > 0: - fn = killfuncs.pop() - fn() - fs = [] - cond.release() + + # Wait for one thread to finish + if wait_for_termination(cond, killfuncs): + break + + # Wait for the rest of the threads to finish + while len(killfuncs) > 0: + wait_for_termination(cond, killfuncs) if journalthread is not None: journalthread.terminate()
PATCH 36/40 xfs_scrub: tighten security of systemd services
In reply to Darrick J. WongHi all,
To reduce the risk of the online fsck service suffering some sort of catastrophic breach that results in attackers reconfiguring the running system, I embarked on a security audit of the systemd service files. The result should be that all elements of the background service (individual scrub jobs, the scrub_all initiator, and the failure reporting) run with as few privileges and within as strong of a sandbox as possible.
Granted, this does nothing about the potential for the /kernel/ screwing up, but at least we could prevent obvious container escapes.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-service-security
--- man/man8/xfs_scrub.8 | 9 +++- scrub/Makefile | 7 ++- scrub/phase1.c | 4 +- scrub/system-xfs_scrub.slice | 30 ++++++++++++ scrub/vfs.c | 2 - scrub/xfs_scrub.c | 11 +++- scrub/xfs_scrub.h | 5 ++ scrub/[email protected] | 97 ++++++++++++++++++++++++++++++++++---- scrub/xfs_scrub_all.service.in | 66 ++++++++++++++++++++++++++ scrub/[email protected] | 59 +++++++++++++++++++++++ 10 files changed, 270 insertions(+), 20 deletions(-) create mode 100644 scrub/system-xfs_scrub.slice
PATCH 1/6 xfs_scrub: allow auxiliary pathnames for sandboxing
From: Darrick J. Wong <[email protected]>
In the next patch, we'll tighten up the security on the xfs_scrub service so that it can't escape. However, sandboxing the service involves making the host filesystem as inaccessible as possible, with the filesystem to scrub bind mounted onto a known location within the sandbox. Hence we need one path for reporting and a new -M argument to tell scrub what it should actually be trying to open.
--- man/man8/xfs_scrub.8 | 9 ++++++++- scrub/phase1.c | 4 ++-- scrub/vfs.c | 2 +- scrub/xfs_scrub.c | 11 ++++++++--- scrub/xfs_scrub.h | 5 ++++- 5 files changed, 23 insertions(+), 8 deletions(-)
man/man8/xfs_scrub.8 +8 -1
@@ -4,7 +4,7 @@ xfs_scrub \- check and repair the contents of a mounted XFS filesystem .SH SYNOPSIS .B xfs_scrub [ -.B \-abCemnTvx +.B \-abCeMmnTvx ] .I mount-point .br
@@ -79,6 +79,13 @@ behavior. .B \-k Do not call TRIM on the free space. .TP +.BI \-M " real-mount-point" +Open the this path for issuing scrub system calls to the kernel. +The positional +.I mount-point +parameter will be used for displaying informational messages and logging. +This parameter exists to enable process sandboxing for service mode. +.TP .BI \-m " file" Search this file for mounted filesystems instead of /etc/mtab. .TP
scrub/phase1.c +2 -2
@@ -146,7 +146,7 @@ phase1_func(* CAP_SYS_ADMIN, which we probably need to do anything fancy * with the (XFS driver) kernel. */ - error = -xfd_open (&ctx->mnt, ctx->mntpoint, + error = -xfd_open (&ctx->mnt, ctx->actual_mntpoint,O_RDONLY |O_NOATIME |O_DIRECTORY );if (error) {if (error ==EPERM )
@@ -199,7 +199,7 @@ _("Not an XFS filesystem."));return error; } - error =path_to_fshandle (ctx->mntpoint, &ctx->fshandle, + error =path_to_fshandle (ctx->actual_mntpoint, &ctx->fshandle, &ctx->fshandle_len);if (error) {str_errno (ctx,_ ("getting fshandle" ));
scrub/vfs.c +1 -1
@@ -249,7 +249,7 @@ scan_fs_tree( goto out_cond; } - ret =queue_subdir (ctx, &sft, &wq, ctx->mntpoint, true); + ret =queue_subdir (ctx, &sft, &wq, ctx->actual_mntpoint, true);if (ret) {str_liberror (ctx, ret,_ ("queueing directory scan" )); goto out_wq;
scrub/xfs_scrub.c +8 -3
@@ -725,7 +725,7 @@ main(pthread_mutex_init (&ctx.lock,NULL ); ctx.mode =SCRUB_MODE_REPAIR ; ctx.error_action =ERRORS_CONTINUE ; -while ((c =getopt (argc, argv,"a:bC:de:km:no:TvxV" )) !=EOF ) { +while ((c =getopt (argc, argv,"a:bC:de:kM:m:no:TvxV" )) !=EOF ) {switch (c) {case 'a' : ctx.max_errors =cvt_u64 (optarg,10 );
@@ -769,6 +769,9 @@ main(case 'k' : want_fstrim = false;break ; +case 'M' : + ctx.actual_mntpoint = optarg; +break ;case 'm' : mtab = optarg;break ;
@@ -823,6 +826,8 @@ main(usage (); ctx.mntpoint = argv[optind]; +if (!ctx.actual_mntpoint) + ctx.actual_mntpoint = ctx.mntpoint; stdout_isatty =isatty (STDOUT_FILENO ); stderr_isatty =isatty (STDERR_FILENO );
@@ -840,7 +845,7 @@ main(return SCRUB_RET_OPERROR ;/* Find the mount record for the passed-in argument. */ -if (stat (argv[optind], &ctx.mnt_sb) <0 ) { +if (stat (ctx.actual_mntpoint, &ctx.mnt_sb) <0 ) {fprintf (stderr,_ ("%s: could not stat: %s: %s\n" ), progname, argv[optind],strerror (errno));
@@ -863,7 +868,7 @@ main( }fs_table_initialise (0 ,NULL ,0 ,NULL ); - fsp =fs_table_lookup_mount (ctx.mntpoint); + fsp =fs_table_lookup_mount (ctx.actual_mntpoint);if (!fsp) {fprintf (stderr,_ ("%s: Not a XFS mount point.\n" ), ctx.mntpoint);
scrub/xfs_scrub.h +4 -1
@@ -38,9 +38,12 @@ enum error_action {struct scrub_ctx {/* Immutable scrub state. */ -/* Strings we need for presentation */ +/* Mountpoint we use for presentation */ char *mntpoint; +/* Actual VFS path to the filesystem */ +char *actual_mntpoint; +/* Mountpoint info */ struct stat mnt_sb;struct statvfs mnt_sv;
PATCH 2/6 xfs_scrub.service: reduce CPU usage to 60% when possible
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, the xfs_scrub background service is configured to use -b, which means that the program runs completely serially. However, even using 100% of one CPU with idle priority may be enough to cause thermal throttling and unwanted fan noise on smaller systems (e.g. laptops) with fast IO systems.
Let's try to avoid this (at least on systemd) by using cgroups to limit the program's usage to 60% of one CPU and lowering the nice priority in the scheduler. What we /really/ want is to run steadily on an efficiency core, but there doesn't seem to be a means to ask the scheduler not to ramp up the CPU frequency for a particular task.
While we're at it, group the resource limit directives together.
--- scrub/Makefile | 7 ++++++- scrub/system-xfs_scrub.slice | 30 ++++++++++++++++++++++++++++++ scrub/[email protected] | 12 ++++++++++-- scrub/xfs_scrub_all.service.in | 4 ++++ scrub/[email protected] | 4 ++++ 5 files changed, 54 insertions(+), 3 deletions(-) create mode 100644 scrub/system-xfs_scrub.slice
scrub/Makefile +6 -1
@@ -18,7 +18,12 @@ XFS_SCRUB_FAIL_PROG = xfs_scrub_fail XFS_SCRUB_ARGS = -b -n ifeq ($(HAVE_SYSTEMD),yes) INSTALL_SCRUB += install-systemd -SYSTEMD_SERVICES = $(scrub_svcname) xfs_scrub_all.service xfs_scrub_all.timer [email protected] +SYSTEMD_SERVICES=\ + $(scrub_svcname) \ + [email protected] \ + xfs_scrub_all.service \ + xfs_scrub_all.timer \ + system-xfs_scrub.slice OPTIONAL_TARGETS += $(SYSTEMD_SERVICES) endif ifeq ($(HAVE_CROND),yes)
scrub/system-xfs_scrub.slice new file +30 -0
@@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (c) 2022-2024 Oracle. All Rights Reserved. +# Author: Darrick J. Wong <[email protected]> + +[Unit] +Description=xfs_scrub background service slice +Before=slices.target + +[Slice] + +# If the CPU usage cgroup controller is available, don't use more than 60% of a +# single core for all background processes. +CPUQuota=60% +CPUAccounting=true + +[Install] +# As of systemd 249, the systemd cgroupv2 configuration code will drop resource +# controllers from the root and system.slice cgroups at startup if it doesn't +# find any direct dependencies that require a given controller. Newly +# activated units with resource control directives are created under the system +# slice but do not cause a reconfiguration of the slice's resource controllers. +# Hence we cannot put CPUQuota= into the xfs_scrub service units directly. +# +# For the CPUQuota directive to have any effect, we must therefore create an +# explicit definition file for the slice that systemd creates to contain the +# xfs_scrub instance units (e.g. [email protected]) and we must configure this +# slice as a dependency of the system slice to establish the direct dependency +# relation. +WantedBy=system.slice
scrub/[email protected] +10 -2
@@ -18,8 +18,16 @@ PrivateTmp=no AmbientCapabilities=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO NoNewPrivileges=yes User=nobody -IOSchedulingClass=idle -CPUSchedulingPolicy=idle Environment=SERVICE_MODE=1 ExecStart=@sbindir@/xfs_scrub @scrub_args@ %f SyslogIdentifier=%N + +# Run scrub with minimal CPU and IO priority so that nothing else will starve. +IOSchedulingClass=idle +CPUSchedulingPolicy=idle +CPUAccounting=true +Nice=19 + +# Create the service underneath the scrub background service slice so that we +# can control resource usage. +Slice=system-xfs_scrub.slice
scrub/xfs_scrub_all.service.in +4 -0
@@ -14,3 +14,7 @@ Type=oneshot Environment=SERVICE_MODE=1 ExecStart=@sbindir@/xfs_scrub_all SyslogIdentifier=xfs_scrub_all + +# Create the service underneath the scrub background service slice so that we +# can control resource usage. +Slice=system-xfs_scrub.slice
scrub/[email protected] +4 -0
@@ -14,3 +14,7 @@ ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" %f User=mail Group=mail SupplementaryGroups=systemd-journal + +# Create the service underneath the scrub background service slice so that we +# can control resource usage. +Slice=system-xfs_scrub.slice
PATCH 3/6 xfs_scrub: use dynamic users when running as a systemd service
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Five years ago, systemd introduced the DynamicUser directive that allocates a new unique user/group id, runs a service with those ids, and deletes them after the service exits. This is a good replacement for User=nobody, since it eliminates the threat of nobody-services messing with each other.
Make this transition ahead of all the other security tightenings that will land in the next few patches, and add credits for the people who suggested the change and reviewed it.
--- scrub/[email protected] | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scrub/[email protected] +3 -1
@@ -17,7 +17,6 @@ ProtectHome=read-only PrivateTmp=no AmbientCapabilities=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO NoNewPrivileges=yes -User=nobody Environment=SERVICE_MODE=1 ExecStart=@sbindir@/xfs_scrub @scrub_args@ %f SyslogIdentifier=%N
@@ -31,3 +30,6 @@ Nice=19 # Create the service underneath the scrub background service slice so that we # can control resource usage. Slice=system-xfs_scrub.slice + +# Dynamically create a user that isn't root +DynamicUser=true
PATCH 4/6 xfs_scrub: tighten up the security on the background systemd service
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, xfs_scrub has to run with some elevated privileges. Minimize the risk of xfs_scrub escaping its service container or contaminating the rest of the system by using systemd's sandboxing controls to prohibit as much access as possible.
The directives added by this patch were recommended by the command 'systemd-analyze security [email protected]' in systemd 249.
--- scrub/[email protected] | 81 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 73 insertions(+), 8 deletions(-)
scrub/[email protected] +73 -8
@@ -8,17 +8,21 @@ Description=Online XFS Metadata Check for %f OnFailure=xfs_scrub_fail@%i.service Documentation=man:xfs_scrub(8) +# Explicitly require the capabilities that this program needs +ConditionCapability=CAP_SYS_ADMIN +ConditionCapability=CAP_FOWNER +ConditionCapability=CAP_DAC_OVERRIDE +ConditionCapability=CAP_DAC_READ_SEARCH +ConditionCapability=CAP_SYS_RAWIO + +# Must be a mountpoint +ConditionPathIsMountPoint=%f +RequiresMountsFor=%f + [Service] Type=oneshot -PrivateNetwork=true -ProtectSystem=full -ProtectHome=read-only -# Disable private /tmp just in case %f is a path under /tmp. -PrivateTmp=no -AmbientCapabilities=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO -NoNewPrivileges=yes Environment=SERVICE_MODE=1 -ExecStart=@sbindir@/xfs_scrub @scrub_args@ %f +ExecStart=@sbindir@/xfs_scrub @scrub_args@ -M /tmp/scrub/ %f SyslogIdentifier=%N # Run scrub with minimal CPU and IO priority so that nothing else will starve.
@@ -31,5 +35,66 @@ Nice=19 # can control resource usage. Slice=system-xfs_scrub.slice +# No realtime CPU scheduling +RestrictRealtime=true + # Dynamically create a user that isn't root DynamicUser=true + +# Make the entire filesystem readonly and /home inaccessible, then bind mount +# the filesystem we're supposed to be checking into our private /tmp dir. +# 'norbind' means that we don't bind anything under that original mount. +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=true +BindPaths=%f:/tmp/scrub:norbind + +# Don't let scrub complain about paths in /etc/projects that have been hidden +# by our sandboxing. scrub doesn't care about project ids anyway. +InaccessiblePaths=-/etc/projects + +# No network access +PrivateNetwork=true +ProtectHostname=true +RestrictAddressFamilies=none +IPAddressDeny=any + +# Don't let the program mess with the kernel configuration at all +ProtectKernelLogs=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectControlGroups=true +ProtectProc=invisible +RestrictNamespaces=true + +# Hide everything in /proc, even /proc/mounts +ProcSubset=pid + +# Only allow the default personality Linux +LockPersonality=true + +# No writable memory pages +MemoryDenyWriteExecute=true + +# Don't let our mounts leak out to the host +PrivateMounts=true + +# Restrict system calls to the native arch and only enough to get things going +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged +SystemCallFilter=~@resources +SystemCallFilter=~@mount + +# xfs_scrub needs these privileges to run, and no others +CapabilityBoundingSet=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO +AmbientCapabilities=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO +NoNewPrivileges=true + +# xfs_scrub doesn't create files +UMask=7777 + +# No access to hardware /dev files except for block devices +ProtectClock=true +DevicePolicy=closed +DeviceAllow=block-*
PATCH 5/6 xfs_scrub_fail: tighten up the security on the background systemd service
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, xfs_scrub_fail has to run with enough privileges to access the journal contents for a given scrub run and to send a report via email. Minimize the risk of xfs_scrub_fail escaping its service container or contaminating the rest of the system by using systemd's sandboxing controls to prohibit as much access as possible.
The directives added by this patch were recommended by the command 'systemd-analyze security [email protected]' in systemd 249.
--- scrub/[email protected] | 55 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
scrub/[email protected] +55 -0
@@ -18,3 +18,58 @@ SupplementaryGroups=systemd-journal # Create the service underneath the scrub background service slice so that we # can control resource usage. Slice=system-xfs_scrub.slice + +# No realtime scheduling +RestrictRealtime=true + +# Make the entire filesystem readonly and /home inaccessible. +ProtectSystem=full +ProtectHome=yes +PrivateTmp=true +RestrictSUIDSGID=true + +# Emailing reports requires network access, but not the ability to change the +# hostname. +ProtectHostname=true + +# Don't let the program mess with the kernel configuration at all +ProtectKernelLogs=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectControlGroups=true +ProtectProc=invisible +RestrictNamespaces=true + +# Can't hide /proc because journalctl needs it to find various pieces of log +# information +#ProcSubset=pid + +# Only allow the default personality Linux +LockPersonality=true + +# No writable memory pages +MemoryDenyWriteExecute=true + +# Don't let our mounts leak out to the host +PrivateMounts=true + +# Restrict system calls to the native arch and only enough to get things going +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged +SystemCallFilter=~@resources +SystemCallFilter=~@mount + +# xfs_scrub needs these privileges to run, and no others +CapabilityBoundingSet= +NoNewPrivileges=true + +# Failure reporting shouldn't create world-readable files +UMask=0077 + +# Clean up any IPC objects when this unit stops +RemoveIPC=true + +# No access to hardware device files +PrivateDevices=true +ProtectClock=true
PATCH 6/6 xfs_scrub_all: tighten up the security on the background systemd service
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Currently, xfs_scrub_all has to run with enough privileges to find mounted XFS filesystems and the device associated with that mount and to start xfs_scrub@<mountpoint> sub-services. Minimize the risk of xfs_scrub_all escaping its service container or contaminating the rest of the system by using systemd's sandboxing controls to prohibit as much access as possible.
The directives added by this patch were recommended by the command 'systemd-analyze security xfs_scrub_all.service' in systemd 249.
--- scrub/xfs_scrub_all.service.in | 62 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+)
scrub/xfs_scrub_all.service.in +62 -0
@@ -18,3 +18,65 @@ SyslogIdentifier=xfs_scrub_all # Create the service underneath the scrub background service slice so that we # can control resource usage. Slice=system-xfs_scrub.slice + +# Run scrub_all with minimal CPU and IO priority so that nothing will starve. +IOSchedulingClass=idle +CPUSchedulingPolicy=idle +CPUAccounting=true +Nice=19 + +# No realtime scheduling +RestrictRealtime=true + +# No special privileges, but we still have to run as root so that we can +# contact the service manager to start the sub-units. +CapabilityBoundingSet= +NoNewPrivileges=true +RestrictSUIDSGID=true + +# Make the entire filesystem readonly. We don't want to hide anything because +# we need to find all mounted XFS filesystems in the host. +ProtectSystem=strict +ProtectHome=read-only +PrivateTmp=false + +# No network access except to the systemd control socket +PrivateNetwork=true +ProtectHostname=true +RestrictAddressFamilies=AF_UNIX +IPAddressDeny=any + +# Don't let the program mess with the kernel configuration at all +ProtectKernelLogs=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectControlGroups=true +ProtectProc=invisible +RestrictNamespaces=true + +# Hide everything in /proc, even /proc/mounts +ProcSubset=pid + +# Only allow the default personality Linux +LockPersonality=true + +# No writable memory pages +MemoryDenyWriteExecute=true + +# Don't let our mounts leak out to the host +PrivateMounts=true + +# Restrict system calls to the native arch and only enough to get things going +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged +SystemCallFilter=~@resources +SystemCallFilter=~@mount + +# Media scan stamp file shouldn't be readable by regular users +UMask=0077 + +# lsblk ignores mountpoints if it can't find the device files, so we cannot +# hide them +#ProtectClock=true +#PrivateDevices=true
PATCH 37/40 xfs_scrub_all: automatic media scan service
In reply to Darrick J. WongHi all,
Now that we've completed the online fsck functionality, there are a few things that could be improved in the automatic service. Specifically, we would like to perform a more intensive metadata + media scan once per month, to give the user confidence that the filesystem isn't losing data silently. To accomplish this, enhance xfs_scrub_all to be able to trigger media scans. Next, add a duplicate set of system services that start the media scans automatically.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service
--- debian/rules | 3 + include/builddefs.in | 3 + man/man8/Makefile | 7 ++ man/man8/xfs_scrub_all.8.in | 20 +++++ scrub/Makefile | 21 +++++- scrub/[email protected] | 2 - scrub/xfs_scrub_all.cron.in | 2 - scrub/xfs_scrub_all.in | 122 ++++++++++++++++++++++++++------ scrub/xfs_scrub_all.service.in | 9 ++ scrub/xfs_scrub_all_fail.service.in | 71 +++++++++++++++++++ scrub/xfs_scrub_fail.in | 46 +++++++++--- scrub/[email protected] | 2 - scrub/[email protected] | 100 ++++++++++++++++++++++++++ scrub/[email protected] | 76 ++++++++++++++++++++ 14 files changed, 439 insertions(+), 45 deletions(-) rename man/man8/{xfs_scrub_all.8 => xfs_scrub_all.8.in} (59%) create mode 100644 scrub/xfs_scrub_all_fail.service.in create mode 100644 scrub/[email protected] create mode 100644 scrub/[email protected]
PATCH 1/6 xfs_scrub_all: only use the xfs_scrub@ systemd services in service mode
From: Darrick J. Wong <[email protected]>
Since the per-mount [email protected] definition includes a bunch of resource usage constraints, we no longer want to use those services if xfs_scrub_all is being run directly by the sysadmin (aka not in service mode) on the presumption that sysadmins want answers as quickly as possible.
Therefore, only try to call the systemd service from xfs_scrub_all if SERVICE_MODE is set in the environment. If reaching out to systemd fails and we're in service mode, we still want to run xfs_scrub directly. Split the makefile variables as necessary so that we only pass -b to xfs_scrub in service mode.
--- scrub/Makefile | 5 ++++- scrub/[email protected] | 2 +- scrub/xfs_scrub_all.in | 11 ++++++++--- 3 files changed, 13 insertions(+), 5 deletions(-)
scrub/Makefile +4 -1
@@ -15,7 +15,8 @@ LTCOMMAND = xfs_scrub INSTALL_SCRUB = install-scrub XFS_SCRUB_ALL_PROG = xfs_scrub_all XFS_SCRUB_FAIL_PROG = xfs_scrub_fail -XFS_SCRUB_ARGS = -b -n +XFS_SCRUB_ARGS = -n +XFS_SCRUB_SERVICE_ARGS = -b ifeq ($(HAVE_SYSTEMD),yes) INSTALL_SCRUB += install-systemd SYSTEMD_SERVICES=\
@@ -125,6 +126,7 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ $(Q)chmod a+x $@
@@ -144,6 +146,7 @@ install: $(INSTALL_SCRUB) %.service: %.service.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ < $< > $@
scrub/[email protected] +1 -1
@@ -22,7 +22,7 @@ RequiresMountsFor=%f [Service] Type=oneshot Environment=SERVICE_MODE=1 -ExecStart=@sbindir@/xfs_scrub @scrub_args@ -M /tmp/scrub/ %f +ExecStart=@sbindir@/xfs_scrub @scrub_service_args@ @scrub_args@ -M /tmp/scrub/ %f SyslogIdentifier=%N # Run scrub with minimal CPU and IO priority so that nothing else will starve.
scrub/xfs_scrub_all.in +8 -3
@@ -162,9 +162,10 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): if terminate: return - # Try it the systemd way + # Run per-mount systemd xfs_scrub service only if we ourselves + # are running as a systemd service. unitname = path_to_serviceunit(path) - if unitname is not None: + if unitname is not None and 'SERVICE_MODE' in os.environ: ret = systemctl_start(unitname, killfuncs) if ret == 0 or ret == 1: print("Scrubbing %s done, (err=%d)" % (mnt, ret))
@@ -175,8 +176,12 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): if terminate: return - # Invoke xfs_scrub manually + # Invoke xfs_scrub manually if we're running in the foreground. + # We also permit this if we're running as a cronjob where + # systemd services are unavailable. cmd = ['@sbindir@/xfs_scrub'] + if 'SERVICE_MODE' in os.environ: + cmd += '@scrub_service_args@'.split() cmd += '@scrub_args@'.split() cmd += [mnt] ret = run_killable(cmd, None, killfuncs)
PATCH 2/6 xfs_scrub_all: remove journalctl background process
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we only start systemd services if we're running in service mode, there's no need for the background journalctl process that only ran if we had started systemd services in non-service mode.
--- scrub/xfs_scrub_all.in | 14 -------------- 1 file changed, 14 deletions(-)
scrub/xfs_scrub_all.in +0 -14
@@ -261,17 +261,6 @@ def main(): fs = find_mounts() - # Tail the journal if we ourselves aren't a service... - journalthread = None - if 'SERVICE_MODE' not in os.environ: - try: - cmd=['journalctl', '--no-pager', '-q', '-S', 'now', \ - '-f', '-u', 'xfs_scrub@*', '-o', \ - 'cat'] - journalthread = subprocess.Popen(cmd) - except: - pass - # Schedule scrub jobs... running_devs = set() killfuncs = set()
@@ -308,9 +297,6 @@ def main(): while len(killfuncs) > 0: wait_for_termination(cond, killfuncs) - if journalthread is not None: - journalthread.terminate() - # See the service mode comments in xfs_scrub.c for why we do this. if 'SERVICE_MODE' in os.environ: time.sleep(2)
PATCH 3/6 xfs_scrub_all: support metadata+media scans of all filesystems
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add the necessary systemd services and control bits so that xfs_scrub_all can kick off a metadata+media scan of a filesystem.
--- man/man8/xfs_scrub_all.8 | 5 +- scrub/Makefile | 4 + scrub/xfs_scrub_all.in | 23 +++++-- scrub/xfs_scrub_fail.in | 13 +++- scrub/[email protected] | 2 - scrub/[email protected] | 100 ++++++++++++++++++++++++++++++++ scrub/[email protected] | 76 ++++++++++++++++++++++++ 7 files changed, 210 insertions(+), 13 deletions(-) create mode 100644 scrub/[email protected] create mode 100644 scrub/[email protected]
man/man8/xfs_scrub_all.8 +4 -1
@@ -4,7 +4,7 @@ xfs_scrub_all \- scrub all mounted XFS filesystems .SH SYNOPSIS .B xfs_scrub_all [ -.B \-hV +.B \-hxV ] .SH DESCRIPTION .B xfs_scrub_all
@@ -21,6 +21,9 @@ the same device simultaneously. .B \-h Display help. .TP +.B \-x +Read all file data extents to look for disk errors. +.TP .B \-V Prints the version number and exits. .SH EXIT CODE
scrub/Makefile +4 -0
@@ -9,6 +9,7 @@ include $(builddefs) SCRUB_PREREQS=$(HAVE_OPENAT)$(HAVE_FSTATAT)$(HAVE_GETFSMAP) [email protected] [email protected] ifeq ($(SCRUB_PREREQS),yesyesyes) LTCOMMAND = xfs_scrub
@@ -22,6 +23,8 @@ INSTALL_SCRUB += install-systemd SYSTEMD_SERVICES=\ $(scrub_svcname) \ [email protected] \ + $(scrub_media_svcname) \ + [email protected] \ xfs_scrub_all.service \ xfs_scrub_all.timer \ system-xfs_scrub.slice
@@ -125,6 +128,7 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ + -e "s|@scrub_media_svcname@|$(scrub_media_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@
scrub/xfs_scrub_all.in +17 -6
@@ -19,6 +19,7 @@ from io import TextIOWrapper retcode = 0 terminate = False +scrub_media = False def DEVNULL(): '''Return /dev/null in subprocess writable format.'''
@@ -88,11 +89,15 @@ def run_killable(cmd, stdout, killfuncs): # systemd doesn't like unit instance names with slashes in them, so it # replaces them with dashes when it invokes the service. Filesystem paths # need a special --path argument so that dashes do not get mangled. -def path_to_serviceunit(path): +def path_to_serviceunit(path, scrub_media): '''Convert a pathname into a systemd service unit name.''' - cmd = ['systemd-escape', '--template', '@scrub_svcname@', - '--path', path] + if scrub_media: + svcname = '@scrub_media_svcname@' + else: + svcname = '@scrub_svcname@' + cmd = ['systemd-escape', '--template', svcname, '--path', path] + try: proc = subprocess.Popen(cmd, stdout = subprocess.PIPE) proc.wait()
@@ -153,7 +158,7 @@ def systemctl_start(unitname, killfuncs): def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): '''Run a scrub process.''' - global retcode, terminate + global retcode, terminate, scrub_media print("Scrubbing %s..." % mnt) sys.stdout.flush()
@@ -164,7 +169,7 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): # Run per-mount systemd xfs_scrub service only if we ourselves # are running as a systemd service. - unitname = path_to_serviceunit(path) + unitname = path_to_serviceunit(path, scrub_media) if unitname is not None and 'SERVICE_MODE' in os.environ: ret = systemctl_start(unitname, killfuncs) if ret == 0 or ret == 1:
@@ -183,6 +188,8 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): if 'SERVICE_MODE' in os.environ: cmd += '@scrub_service_args@'.split() cmd += '@scrub_args@'.split() + if scrub_media: + cmd += '-x' cmd += [mnt] ret = run_killable(cmd, None, killfuncs) if ret >= 0:
@@ -247,18 +254,22 @@ def main(): a = (mnt, cond, running_devs, devs, killfuncs) thr = threading.Thread(target = run_scrub, args = a) thr.start() - global retcode, terminate + global retcode, terminate, scrub_media parser = argparse.ArgumentParser( \ description = "Scrub all mounted XFS filesystems.") parser.add_argument("-V", help = "Report version and exit.", \ action = "store_true") + parser.add_argument("-x", help = "Scrub file data after filesystem metadata.", \ + action = "store_true") args = parser.parse_args() if args.V: print("xfs_scrub_all version @pkg_version@") sys.exit(0) + scrub_media = args.x + fs = find_mounts() # Schedule scrub jobs...
scrub/xfs_scrub_fail.in +8 -5
@@ -9,8 +9,11 @@ recipient="$1" test -z "${recipient}" && exit 0 -mntpoint="$2" +service="$2" +test -z "${service}" && exit 0 +mntpoint="$3" test -z "${mntpoint}" && exit 0 + hostname="$(hostname -f 2>/dev/null)" test -z "${hostname}" && hostname="${HOSTNAME}"
@@ -21,16 +24,16 @@ if [ ! -x "${mailer}" ]; then fi # Turn the mountpoint into a properly escaped systemd instance name -scrub_svc="$(systemd-escape --template "@scrub_svcname@" --path "${mntpoint}")" +scrub_svc="$(systemd-escape --template "${service}@.service" --path "${mntpoint}")" (cat << ENDL To: $1 -From: <xfs_scrub@${hostname}> -Subject: xfs_scrub failure on ${mntpoint} +From: <${service}@${hostname}> +Subject: ${service} failure on ${mntpoint} Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 -So sorry, the automatic xfs_scrub of ${mntpoint} on ${hostname} failed. +So sorry, the automatic ${service} of ${mntpoint} on ${hostname} failed. Please do not reply to this mesage. A log of what happened follows:
scrub/[email protected] +1 -1
@@ -10,7 +10,7 @@ Documentation=man:xfs_scrub(8) [Service] Type=oneshot Environment=EMAIL_ADDR=root -ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" %f +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" xfs_scrub %f User=mail Group=mail SupplementaryGroups=systemd-journal
scrub/[email protected] new file +100 -0
100 lines not on this page · raw message
scrub/[email protected] new file +76 -0
76 lines not on this page · raw message
PATCH 4/6 xfs_scrub_all: enable periodic file data scrubs automatically
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Enhance xfs_scrub_all with the ability to initiate a file data scrub periodically. The user must specify the period, and they may optionally specify the path to a file that will record the last time the file data was scrubbed.
---
debian/rules | 3 +-
include/builddefs.in | 3 ++
man/man8/Makefile | 7 +++-
man/man8/xfs_scrub_all.8.in | 15 ++++++++
scrub/Makefile | 3 ++
scrub/xfs_scrub_all.in | 76 +++++++++++++++++++++++++++++++++++++++-
scrub/xfs_scrub_all.service.in | 6 ++-
7 files changed, 108 insertions(+), 5 deletions(-)
rename man/man8/{xfs_scrub_all.8 => xfs_scrub_all.8.in} (63%)debian/rules +2 -1
@@ -34,7 +34,8 @@ configure_options = \ --disable-ubsan \ --disable-addrsan \ --disable-threadsan \ - --enable-lto + --enable-lto \ + --localstatedir=/var options = export DEBUG=-DNDEBUG DISTRIBUTION=debian \ INSTALL_USER=root INSTALL_GROUP=root \
include/builddefs.in +3 -0
@@ -59,6 +59,9 @@ PKG_DOC_DIR = @datadir@/doc/@pkg_name@ PKG_LOCALE_DIR = @datadir@/locale PKG_DATA_DIR = @datadir@/@pkg_name@ MKFS_CFG_DIR = @datadir@/@pkg_name@/mkfs +PKG_STATE_DIR = @localstatedir@/lib/@pkg_name@ + +XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP=$(PKG_STATE_DIR)/xfs_scrub_all_media.stamp CC = @cc@ BUILD_CC = @BUILD_CC@
man/man8/Makefile +6 -1
@@ -11,11 +11,12 @@ ifneq ("$(ENABLE_SCRUB)","yes") MAN_PAGES = $(filter-out xfs_scrub%,$(shell echo *.$(MAN_SECTION))) else MAN_PAGES = $(shell echo *.$(MAN_SECTION)) + MAN_PAGES += xfs_scrub_all.8 endif MAN_PAGES += mkfs.xfs.8 MAN_DEST = $(PKG_MAN_DIR)/man$(MAN_SECTION) LSRCFILES = $(MAN_PAGES) -DIRT = mkfs.xfs.8 +DIRT = mkfs.xfs.8 xfs_scrub_all.8 default : $(MAN_PAGES)
@@ -29,4 +30,8 @@ mkfs.xfs.8: mkfs.xfs.8.in @echo " [SED] $@" $(Q)$(SED) -e 's|@mkfs_cfg_dir@|$(MKFS_CFG_DIR)|g' < $^ > $@ +xfs_scrub_all.8: xfs_scrub_all.8.in + @echo " [SED] $@" + $(Q)$(SED) -e 's|@stampfile@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP)|g' < $^ > $@ + install-dev :
man/man8/xfs_scrub_all.8.in renamed from man/man8/xfs_scrub_all.8 +15 -0
@@ -18,6 +18,21 @@ operations can be run in parallel so long as no two scrubbers access the same device simultaneously. .SH OPTIONS .TP +.B \--auto-media-scan-interval +Automatically enable the file data scan (i.e. the +.B -x +flag) if it has not been run in the specified interval. +The interval must be a floating point number with an optional unit suffix. +Supported unit suffixes are +.IR y ", " q ", " mo ", " w ", " d ", " h ", " m ", and " s +for years, 90-day quarters, 30-day months, weeks, days, hours, minutes, and +seconds, respectively. +If no units are specified, the default is seconds. +.TP +.B \--auto-media-scan-stamp +Path to a file that will record the last time the media scan was run. +Defaults to @stampfile@. +.TP .B \-h Display help. .TP
scrub/Makefile +3 -0
@@ -130,6 +130,7 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ -e "s|@scrub_media_svcname@|$(scrub_media_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e "s|@stampfile@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP)|g" \ -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ $(Q)chmod a+x $@
@@ -153,6 +154,7 @@ install: $(INSTALL_SCRUB) -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ + -e "s|@pkg_state_dir@|$(PKG_STATE_DIR)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs)
@@ -173,6 +175,7 @@ install-scrub: default $(INSTALL) -m 755 -d $(PKG_SBIN_DIR) $(LTINSTALL) -m 755 $(LTCOMMAND) $(PKG_SBIN_DIR) $(INSTALL) -m 755 $(XFS_SCRUB_ALL_PROG) $(PKG_SBIN_DIR) + $(INSTALL) -m 755 -d $(PKG_STATE_DIR) install-udev: $(UDEV_RULES) $(INSTALL) -m 755 -d $(UDEV_RULE_DIR)
scrub/xfs_scrub_all.in +75 -1
100 lines not on this page · raw message
scrub/xfs_scrub_all.service.in +4 -2
@@ -34,11 +34,13 @@ CapabilityBoundingSet= NoNewPrivileges=true RestrictSUIDSGID=true -# Make the entire filesystem readonly. We don't want to hide anything because -# we need to find all mounted XFS filesystems in the host. +# Make the entire filesystem readonly except for the media scan stamp file +# directory. We don't want to hide anything because we need to find all +# mounted XFS filesystems in the host. ProtectSystem=strict ProtectHome=read-only PrivateTmp=false +BindPaths=@pkg_state_dir@ # No network access except to the systemd control socket PrivateNetwork=true
PATCH 5/6 xfs_scrub_all: trigger automatic media scans once per month
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Teach the xfs_scrub_all background service to trigger an automatic scan of all file data once per month.
--- scrub/Makefile | 8 +++++++- scrub/xfs_scrub_all.cron.in | 2 +- scrub/xfs_scrub_all.service.in | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-)
scrub/Makefile +7 -1
@@ -120,6 +120,9 @@ ifeq ($(HAVE_HDIO_GETGEO),yes) LCFLAGS += -DHAVE_HDIO_GETGEO endif +# Automatically trigger a media scan once per month +XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL=1mo + LDIRT = $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) *.service *.cron default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) $(OPTIONAL_TARGETS)
@@ -155,11 +158,14 @@ install: $(INSTALL_SCRUB) -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ -e "s|@pkg_state_dir@|$(PKG_STATE_DIR)|g" \ + -e "s|@media_scan_interval@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs) @echo " [SED] $@" - $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" < $< > $@ + $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@media_scan_interval@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL)|g" \ + < $< > $@ install-systemd: default $(SYSTEMD_SERVICES) $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR)
scrub/xfs_scrub_all.cron.in +1 -1
@@ -3,4 +3,4 @@ # Copyright (C) 2018-2024 Oracle. All Rights Reserved. # Author: Darrick J. Wong <[email protected]> # -10 3 * * 0 root test -e /run/systemd/system || @sbindir@/xfs_scrub_all +10 3 * * 0 root test -e /run/systemd/system || @sbindir@/xfs_scrub_all --auto-media-scan-interval @media_scan_interval@
scrub/xfs_scrub_all.service.in +1 -1
@@ -12,7 +12,7 @@ After=paths.target multi-user.target network.target network-online.target system [Service] Type=oneshot Environment=SERVICE_MODE=1 -ExecStart=@sbindir@/xfs_scrub_all +ExecStart=@sbindir@/xfs_scrub_all --auto-media-scan-interval @media_scan_interval@ SyslogIdentifier=xfs_scrub_all # Create the service underneath the scrub background service slice so that we
PATCH 6/6 xfs_scrub_all: failure reporting for the xfs_scrub_all job
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a failure reporting service for when xfs_scrub_all fails. This shouldn't happen often, but let's report anyways.
--- scrub/Makefile | 1 scrub/xfs_scrub_all.service.in | 1 scrub/xfs_scrub_all_fail.service.in | 71 +++++++++++++++++++++++++++++++++++ scrub/xfs_scrub_fail.in | 35 ++++++++++++++--- 4 files changed, 101 insertions(+), 7 deletions(-) create mode 100644 scrub/xfs_scrub_all_fail.service.in
scrub/Makefile +1 -0
@@ -26,6 +26,7 @@ SYSTEMD_SERVICES=\ $(scrub_media_svcname) \ [email protected] \ xfs_scrub_all.service \ + xfs_scrub_all_fail.service \ xfs_scrub_all.timer \ system-xfs_scrub.slice OPTIONAL_TARGETS += $(SYSTEMD_SERVICES)
scrub/xfs_scrub_all.service.in +1 -0
@@ -5,6 +5,7 @@ [Unit] Description=Online XFS Metadata Check for All Filesystems +OnFailure=xfs_scrub_all_fail.service ConditionACPower=true Documentation=man:xfs_scrub_all(8) After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service
scrub/xfs_scrub_all_fail.service.in new file +71 -0
71 lines not on this page · raw message
scrub/xfs_scrub_fail.in +28 -7
@@ -5,14 +5,13 @@ # Copyright (C) 2018-2024 Oracle. All Rights Reserved. # Author: Darrick J. Wong <[email protected]> -# Email logs of failed xfs_scrub unit runs +# Email logs of failed xfs_scrub and xfs_scrub_all unit runs recipient="$1" test -z "${recipient}" && exit 0 service="$2" test -z "${service}" && exit 0 mntpoint="$3" -test -z "${mntpoint}" && exit 0 hostname="$(hostname -f 2>/dev/null)" test -z "${hostname}" && hostname="${HOSTNAME}"
@@ -23,11 +22,13 @@ if [ ! -x "${mailer}" ]; then exit 1 fi -# Turn the mountpoint into a properly escaped systemd instance name -scrub_svc="$(systemd-escape --template "${service}@.service" --path "${mntpoint}")" +fail_mail_mntpoint() { + local scrub_svc -(cat << ENDL -To: $1 + # Turn the mountpoint into a properly escaped systemd instance name + scrub_svc="$(systemd-escape --template "${service}@.service" --path "${mntpoint}")" + cat << ENDL +To: ${recipient} From: <${service}@${hostname}> Subject: ${service} failure on ${mntpoint} Content-Transfer-Encoding: 8bit
@@ -38,5 +39,25 @@ Please do not reply to this mesage. A log of what happened follows: ENDL -systemctl status --full --lines 4294967295 "${scrub_svc}") | "${mailer}" -t -i + systemctl status --full --lines 4294967295 "${scrub_svc}" +} + +fail_mail() { + cat << ENDL +To: ${recipient} +From: <${service}@${hostname}> +Subject: ${service} failure + +So sorry, the automatic ${service} on ${hostname} failed. + +A log of what happened follows: +ENDL + systemctl status --full --lines 4294967295 "${service}" +} + +if [ -n "${mntpoint}" ]; then + fail_mail_mntpoint | "${mailer}" -t -i +else + fail_mail | "${mailer}" -t -i +fi exit "${PIPESTATUS[1]}"
PATCH 38/40 xfs_scrub_all: improve systemd handling
In reply to Darrick J. WongHi all,
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-all-improve-systemd-handling
--- debian/control | 2 scrub/xfs_scrub_all.in | 279 ++++++++++++++++++++++++++++++++++++------------ 2 files changed, 209 insertions(+), 72 deletions(-)
PATCH 1/5 xfs_scrub_all: encapsulate all the subprocess code in an object
From: Darrick J. Wong <[email protected]>
Move all the xfs_scrub subprocess handling code to an object so that we can contain all the details in a single place. This also simplifies the background state management.
--- scrub/xfs_scrub_all.in | 68 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 54 insertions(+), 14 deletions(-)
scrub/xfs_scrub_all.in +54 -14
@@ -78,15 +78,62 @@ def remove_killfunc(killfuncs, fn): except: pass -def run_killable(cmd, stdout, killfuncs): +class scrub_control(object): + '''Control object for xfs_scrub.''' + def __init__(self): + pass + + def start(self): + '''Start scrub and wait for it to complete. Returns -1 if the + service was not started, 0 if it succeeded, or 1 if it + failed.''' + assert False + + def stop(self): + '''Stop scrub.''' + assert False + +class scrub_subprocess(scrub_control): + '''Control object for xfs_scrub subprocesses.''' + def __init__(self, mnt, scrub_media): + cmd = ['@sbindir@/xfs_scrub'] + if 'SERVICE_MODE' in os.environ: + cmd += '@scrub_service_args@'.split() + cmd += '@scrub_args@'.split() + if scrub_media: + cmd += '-x' + cmd += [mnt] + self.cmdline = cmd + self.proc = None + + def start(self): + '''Start xfs_scrub and wait for it to complete. Returns -1 if + the service was not started, 0 if it succeeded, or 1 if it + failed.''' + try: + self.proc = subprocess.Popen(self.cmdline) + self.proc.wait() + except: + return -1 + + proc = self.proc + self.proc = None + return proc.returncode + + def stop(self): + '''Stop xfs_scrub.''' + if self.proc is not None: + self.proc.terminate() + +def run_subprocess(mnt, scrub_media, killfuncs): '''Run a killable program. Returns program retcode or -1 if we can't start it.''' try: - proc = subprocess.Popen(cmd, stdout = stdout) - killfuncs.add(proc.terminate) - proc.wait() - remove_killfunc(killfuncs, proc.terminate) - return proc.returncode + p = scrub_subprocess(mnt, scrub_media) + killfuncs.add(p.stop) + ret = p.start() + remove_killfunc(killfuncs, p.stop) + return ret except: return -1
@@ -188,14 +235,7 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): # Invoke xfs_scrub manually if we're running in the foreground. # We also permit this if we're running as a cronjob where # systemd services are unavailable. - cmd = ['@sbindir@/xfs_scrub'] - if 'SERVICE_MODE' in os.environ: - cmd += '@scrub_service_args@'.split() - cmd += '@scrub_args@'.split() - if scrub_media: - cmd += '-x' - cmd += [mnt] - ret = run_killable(cmd, None, killfuncs) + ret = run_subprocess(mnt, scrub_media, killfuncs) if ret >= 0: print("Scrubbing %s done, (err=%d)" % (mnt, ret)) sys.stdout.flush()
PATCH 2/5 xfs_scrub_all: encapsulate all the systemctl code in an object
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Move all the systemd service handling code to an object so that we can contain all the insanity^Wdetails in a single place. This also makes the killfuncs handling similar to starting background processes.
--- scrub/xfs_scrub_all.in | 113 ++++++++++++++++++++++++++---------------------- 1 file changed, 61 insertions(+), 52 deletions(-)
scrub/xfs_scrub_all.in +61 -52
@@ -149,63 +149,73 @@ def path_to_serviceunit(path, scrub_media): svcname = '@scrub_svcname@' cmd = ['systemd-escape', '--template', svcname, '--path', path] - try: - proc = subprocess.Popen(cmd, stdout = subprocess.PIPE) - proc.wait() - for line in proc.stdout: - return line.decode(sys.stdout.encoding).strip() - except: - return None + proc = subprocess.Popen(cmd, stdout = subprocess.PIPE) + proc.wait() + for line in proc.stdout: + return line.decode(sys.stdout.encoding).strip() -def systemctl_stop(unitname): - '''Stop a systemd unit.''' - cmd = ['systemctl', 'stop', unitname] - x = subprocess.Popen(cmd) - x.wait() +class scrub_service(scrub_control): + '''Control object for xfs_scrub systemd service.''' + def __init__(self, mnt, scrub_media): + self.unitname = path_to_serviceunit(mnt, scrub_media) -def systemctl_start(unitname, killfuncs): - '''Start a systemd unit and wait for it to complete.''' - stop_fn = None - cmd = ['systemctl', 'start', unitname] - try: - proc = subprocess.Popen(cmd, stdout = DEVNULL()) - stop_fn = lambda: systemctl_stop(unitname) - killfuncs.add(stop_fn) - proc.wait() - ret = proc.returncode - except: - if stop_fn is not None: - remove_killfunc(killfuncs, stop_fn) - return -1 + def wait(self, interval = 1): + '''Wait until the service finishes.''' - if ret != 1: - remove_killfunc(killfuncs, stop_fn) - return ret + # As of systemd 249, the is-active command returns any of the + # following states: active, reloading, inactive, failed, + # activating, deactivating, or maintenance. Apparently these + # strings are not localized. + while True: + try: + for l in backtick(['systemctl', 'is-active', self.unitname]): + if l == 'failed': + return 1 + if l == 'inactive': + return 0 + except: + return -1 - # If systemctl-start returns 1, it's possible that the service failed - # or that dbus/systemd restarted and the client program lost its - # connection -- according to the systemctl man page, 1 means "unit not - # failed". - # - # Either way, we switch to polling the service status to try to wait - # for the service to end. As of systemd 249, the is-active command - # returns any of the following states: active, reloading, inactive, - # failed, activating, deactivating, or maintenance. Apparently these - # strings are not localized. - while True: + time.sleep(interval) + + def start(self): + '''Start the service and wait for it to complete. Returns -1 + if the service was not started, 0 if it succeeded, or 1 if it + failed.''' + cmd = ['systemctl', 'start', self.unitname] try: - for l in backtick(['systemctl', 'is-active', unitname]): - if l == 'failed': - remove_killfunc(killfuncs, stop_fn) - return 1 - if l == 'inactive': - remove_killfunc(killfuncs, stop_fn) - return 0 + proc = subprocess.Popen(cmd, stdout = DEVNULL()) + proc.wait() + ret = proc.returncode except: - remove_killfunc(killfuncs, stop_fn) return -1 - time.sleep(1) + if ret != 1: + return ret + + # If systemctl-start returns 1, it's possible that the service + # failed or that dbus/systemd restarted and the client program + # lost its connection -- according to the systemctl man page, 1 + # means "unit not failed". + return self.wait() + + def stop(self): + '''Stop the service.''' + cmd = ['systemctl', 'stop', self.unitname] + x = subprocess.Popen(cmd) + x.wait() + +def run_service(mnt, scrub_media, killfuncs): + '''Run scrub as a service.''' + try: + svc = scrub_service(mnt, scrub_media) + except: + return -1 + + killfuncs.add(svc.stop) + retcode = svc.start() + remove_killfunc(killfuncs, svc.stop) + return retcode def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): '''Run a scrub process.'''
@@ -220,9 +230,8 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): # Run per-mount systemd xfs_scrub service only if we ourselves # are running as a systemd service. - unitname = path_to_serviceunit(path, scrub_media) - if unitname is not None and 'SERVICE_MODE' in os.environ: - ret = systemctl_start(unitname, killfuncs) + if 'SERVICE_MODE' in os.environ: + ret = run_service(mnt, scrub_media, killfuncs) if ret == 0 or ret == 1: print("Scrubbing %s done, (err=%d)" % (mnt, ret)) sys.stdout.flush()
PATCH 3/5 xfs_scrub_all: add CLI option for easier debugging
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new CLI argument to make it easier to figure out what exactly the program is doing.
--- scrub/xfs_scrub_all.in | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
scrub/xfs_scrub_all.in +24 -1
@@ -24,6 +24,7 @@ from datetime import timezone retcode = 0 terminate = False scrub_media = False +debug = False def DEVNULL(): '''Return /dev/null in subprocess writable format.'''
@@ -110,6 +111,11 @@ class scrub_subprocess(scrub_control): '''Start xfs_scrub and wait for it to complete. Returns -1 if the service was not started, 0 if it succeeded, or 1 if it failed.''' + global debug + + if debug: + print('run ', ' '.join(self.cmdline)) + try: self.proc = subprocess.Popen(self.cmdline) self.proc.wait()
@@ -122,6 +128,10 @@ class scrub_subprocess(scrub_control): def stop(self): '''Stop xfs_scrub.''' + global debug + + if debug: + print('kill ', ' '.join(self.cmdline)) if self.proc is not None: self.proc.terminate()
@@ -182,8 +192,12 @@ class scrub_service(scrub_control): '''Start the service and wait for it to complete. Returns -1 if the service was not started, 0 if it succeeded, or 1 if it failed.''' + global debug + cmd = ['systemctl', 'start', self.unitname] try: + if debug: + print(' '.join(cmd)) proc = subprocess.Popen(cmd, stdout = DEVNULL()) proc.wait() ret = proc.returncode
@@ -201,7 +215,11 @@ class scrub_service(scrub_control): def stop(self): '''Stop the service.''' + global debug + cmd = ['systemctl', 'stop', self.unitname] + if debug: + print(' '.join(cmd)) x = subprocess.Popen(cmd) x.wait()
@@ -366,10 +384,12 @@ def main(): a = (mnt, cond, running_devs, devs, killfuncs) thr = threading.Thread(target = run_scrub, args = a) thr.start() - global retcode, terminate, scrub_media + global retcode, terminate, scrub_media, debug parser = argparse.ArgumentParser( \ description = "Scrub all mounted XFS filesystems.") + parser.add_argument("--debug", help = "Enabling debugging messages.", \ + action = "store_true") parser.add_argument("-V", help = "Report version and exit.", \ action = "store_true") parser.add_argument("-x", help = "Scrub file data after filesystem metadata.", \
@@ -384,6 +404,9 @@ def main(): print("xfs_scrub_all version @pkg_version@") sys.exit(0) + if args.debug: + debug = True + if args.auto_media_scan_interval is not None: try: scrub_media = enable_automatic_media_scan(args)
PATCH 4/5 xfs_scrub_all: convert systemctl calls to dbus
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Convert the systemctl invocations to direct dbus calls, which decouples us from the CLI in favor of direct API calls. This spares us from some of the insanity of divining service state from program outputs.
--- debian/control | 2 + scrub/xfs_scrub_all.in | 96 +++++++++++++++++++++++++++++++----------------- 2 files changed, 63 insertions(+), 35 deletions(-)
debian/control +1 -1
@@ -8,7 +8,7 @@ Standards-Version: 4.0.0 Homepage: https://xfs.wiki.kernel.org/ Package: xfsprogs -Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any +Depends: ${shlibs:Depends}, ${misc:Depends}, python3-dbus, python3:any Provides: fsck-backend Suggests: xfsdump, acl, attr, quota Breaks: xfsdump (<< 3.0.0)
scrub/xfs_scrub_all.in +62 -34
127 lines not on this page · raw message
PATCH 5/5 xfs_scrub_all: implement retry and backoff for dbus calls
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Calls to systemd across dbus are remote procedure calls, which means that they're subject to transitory connection failures (e.g. systemd re-exec itself). We don't want to fail at the *first* sign of what could be temporary trouble, so implement a limited retry with fibonacci backoff before we resort to invoking xfs_scrub as a subprocess.
--- scrub/xfs_scrub_all.in | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-)
scrub/xfs_scrub_all.in +40 -3
@@ -165,6 +165,22 @@ def path_to_serviceunit(path, scrub_media): for line in proc.stdout: return line.decode(sys.stdout.encoding).strip() +def fibonacci(max_ret): + '''Yield fibonacci sequence up to but not including max_ret.''' + if max_ret < 1: + return + + x = 0 + y = 1 + yield 1 + + z = x + y + while z <= max_ret: + yield z + x = y + y = z + z = x + y + class scrub_service(scrub_control): '''Control object for xfs_scrub systemd service.''' def __init__(self, mnt, scrub_media):
@@ -188,6 +204,25 @@ class scrub_service(scrub_control): self.unit = dbus.Interface(svc_obj, 'org.freedesktop.systemd1.Unit') + def __dbusrun(self, lambda_fn): + '''Call the lambda function to execute something on dbus. dbus + exceptions result in retries with Fibonacci backoff, and the + bindings will be rebuilt every time.''' + global debug + + fatal_ex = None + + for i in fibonacci(30): + try: + return lambda_fn() + except dbus.exceptions.DBusException as e: + if debug: + print(e) + fatal_ex = e + time.sleep(i) + self.bind() + raise fatal_ex + def state(self): '''Retrieve the active state for a systemd service. As of systemd 249, this is supposed to be one of the following:
@@ -195,8 +230,10 @@ class scrub_service(scrub_control): or "deactivating". These strings are not localized.''' global debug + l = lambda: self.prop.Get('org.freedesktop.systemd1.Unit', + 'ActiveState') try: - return self.prop.Get('org.freedesktop.systemd1.Unit', 'ActiveState') + return self.__dbusrun(l) except Exception as e: if debug: print(e, file = sys.stderr)
@@ -231,7 +268,7 @@ class scrub_service(scrub_control): print('starting %s' % self.unitname) try: - self.unit.Start('replace') + self.__dbusrun(lambda: self.unit.Start('replace')) return self.wait() except Exception as e: print(e, file = sys.stderr)
@@ -245,7 +282,7 @@ class scrub_service(scrub_control): print('stopping %s' % self.unitname) try: - self.unit.Stop('replace') + self.__dbusrun(lambda: self.unit.Stop('replace')) return self.wait() except Exception as e: print(e, file = sys.stderr)
PATCH 39/40 xfs_scrub: automatic optimization by default
In reply to Darrick J. WongHi all,
This final patchset in the online fsck series enables the background service to optimize filesystems by default. This is the first step towards enabling repairs by default.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-optimize-by-default
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-optimize-by-default
--- debian/rules | 2 +- man/man8/xfs_scrub.8 | 6 +++++- scrub/Makefile | 2 +- scrub/phase1.c | 13 +++++++++++++ scrub/phase4.c | 6 ++++++ scrub/repair.c | 37 ++++++++++++++++++++++++++++++++++++- scrub/repair.h | 2 ++ scrub/scrub.c | 4 ++-- scrub/xfs_scrub.c | 21 +++++++++++++++++++-- scrub/xfs_scrub.h | 1 + 10 files changed, 86 insertions(+), 8 deletions(-)
PATCH 1/3 xfs_scrub: automatic downgrades to dry-run mode in service mode
From: Darrick J. Wong <[email protected]>
When service mode is enabled, xfs_scrub is being run within the context of a systemd service. The service description language doesn't have any particularly good constructs for adding in a '-n' argument if the filesystem is readonly, which means that xfs_scrub is passed a path, and needs to switch to dry-run mode on its own if the fs is mounted readonly or the kernel doesn't support repairs.
--- scrub/phase1.c | 13 +++++++++++++ scrub/repair.c | 33 +++++++++++++++++++++++++++++++++ scrub/repair.h | 2 ++ 3 files changed, 48 insertions(+)
scrub/phase1.c +13 -0
@@ -216,6 +216,19 @@ _("Kernel metadata scrubbing facility is not available."));return ECANCELED ; } +/* +* Normally, callers are required to pass -n if the provided path is a +* readonly filesystem or the kernel wasn't built with online repair +* enabled. However, systemd services are not scripts and cannot +* determine either of these conditions programmatically. Change the +* behavior to dry-run mode if either condition is detected. +*/ +if (repair_want_service_downgrade (ctx)) { +str_info (ctx, ctx->mntpoint, +_ ("Filesystem cannot be repaired in service mode, downgrading to dry-run mode." )); + ctx->mode =SCRUB_MODE_DRY_RUN ; + } +/* Do we need kernel-assisted metadata repair? */ if (ctx->mode !=SCRUB_MODE_DRY_RUN && !can_repair (ctx)) { str_error(ctx, ctx->mntpoint,
scrub/repair.c +33 -0
@@ -45,6 +45,39 @@ static const unsigned int repair_deps[XFS_SCRUB_TYPE_NR] = { };#undef DEP +/* +* Decide if we want an automatic downgrade to dry-run mode. This is only +* for service mode, where we are fed a path and have to figure out if the fs +* is repairable or not. +*/ +bool +repair_want_service_downgrade ( +struct scrub_ctx *ctx) +{ +struct xfs_scrub_metadata meta = { + .sm_type =XFS_SCRUB_TYPE_PROBE , + .sm_flags =XFS_SCRUB_IFLAG_REPAIR , + }; +int error; + +if (ctx->mode ==SCRUB_MODE_DRY_RUN ) +return false; +if (!is_service) +return false; +if (debug_tweak_on ("XFS_SCRUB_NO_KERNEL" )) +return false; + + error = -xfrog_scrub_metadata (&ctx->mnt, &meta); +switch (error) { +case EROFS : +case ENOTRECOVERABLE : +case EOPNOTSUPP : +return true; + } + +return false; +} +/* Repair some metadata. */ static int xfs_repair_metadata(
scrub/repair.h +2 -0
@@ -102,4 +102,6 @@ repair_item_completely( return repair_item(ctx, sri, XRM_FINAL_WARNING | XRM_NOPROGRESS); } +bool repair_want_service_downgrade (struct scrub_ctx *ctx); +#endif /* XFS_SCRUB_REPAIR_H_ */